What is the limit size of tmpfs /dev/shm ?

What is the limit size of tmpfs /dev/shm ?

[Tomasz Chmielewski]

> Hello Kernel Users,
> 
> is there a size limit for tmpfs for the /dev/shm filesystem?
> Normally its default size is set to 2 GB. Is it possible to create a 2 
> TB (Terrabyte) filesystem with tmpfs?
> Or is there a maximum size defined in the linux kernel?

Depends on your arch.

If it's 32 bit, it's limited to 16TB:

# mount -o size=16383G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                  16T     0   16T   0% /mnt/2

# umount /mnt/2

# mount -o size=16385G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                 1.0G     0  1.0G   0% /mnt/2


So 16384G would mean the same as 0.


If you're 64 bit, you need to have really loads of storage and/or RAM to 
accumulate 16EB:

# mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
# df -h
(...)
tmpfs                  16E     0   16E   0% /mnt/2


-- 
Tomasz Chmielewski
http://lists.wpkg.org

Re: What is the limit size of tmpfs /dev/shm ?

[rzryyvzy]

Tomasz Chmielewski schrieb:
>> Hello Kernel Users,
>>
>> is there a size limit for tmpfs for the /dev/shm filesystem?
>> Normally its default size is set to 2 GB. Is it possible to create a 
>> 2 TB (Terrabyte) filesystem with tmpfs?
>> Or is there a maximum size defined in the linux kernel?
>
> Depends on your arch.
I have a 64 Bits architecture (amd64/x86_64).

> If you're 64 bit, you need to have really loads of storage and/or RAM 
> to accumulate 16EB:
>
> # mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
> # df -h
> (...)
> tmpfs                  16E     0   16E   0% /mnt/2
Nice, I will try this out.
I have not the money for 16E of memory of RAM! :-) lol
Already to provide 1 TB it would cost at least for the memory about 20 
000 EUR + the server costs which support this.

-- 
Best regards

Re: What is the limit size of tmpfs /dev/shm ?

[Hugh Dickins]

On Wed, 6 Feb 2008, Tomasz Chmielewski wrote:
> > Hello Kernel Users,
> > 
> > is there a size limit for tmpfs for the /dev/shm filesystem?

There shouldn't be any artificial size limit on the /dev/shm filesystem,
it's an "internal" mount, and those are unlimited by default.  Which is
not to say that you magically receive unlimited memory along with it!

The practical limit would be somewhere round about the total size of
your swap plus, ooh, finger in the air, 75% of your RAM - though that
all depends on what else you'd be wanting to use RAM+swap for.

> > Normally its default size is set to 2 GB.

Hmm, where did that number come from, maybe I'm forgetting something.
The user-visible mounts are limited by default to half of ram, so if
you've 4GB of ram, then 2GB would be the default for those tmpfs mounts.

> > Is it possible to create a 2 TB (Terrabyte) filesystem with tmpfs?

Yes.

> > Or is there a maximum size defined in the linux kernel?
> 
> Depends on your arch.
> 
> If it's 32 bit, it's limited to 16TB:
> 
> # mount -o size=16383G -t tmpfs tmpfs /mnt/2
> # df -h
> (...)
> tmpfs                  16T     0   16T   0% /mnt/2
> 
> # umount /mnt/2
> 
> # mount -o size=16385G -t tmpfs tmpfs /mnt/2
> # df -h
> (...)
> tmpfs                 1.0G     0  1.0G   0% /mnt/2
> 
> So 16384G would mean the same as 0.

Hah!  Nice investigation.  That's more of a bug than anything,
though not one I feel urgently compelled to fix.

> If you're 64 bit, you need to have really loads of storage and/or RAM to
> accumulate 16EB:
> 
> # mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
> # df -h
> (...)
> tmpfs                  16E     0   16E   0% /mnt/2

Mika, you answered:

> Nice, I will try this out.
> I have not the money for 16E of memory of RAM! :-) lol
> Already to provide 1 TB it would cost at least for the memory about 20 000
> EUR + the server costs which support this.

Don't forget that tmpfs overflows into swap, so you could save money
by adding adding more swap and cutting down on the RAM: though of
course that will perform very poorly once it's actually using the
swap, probably not the direction you want to go in.

Hugh

Re: What is the limit size of tmpfs /dev/shm ?

[H. Peter Anvin]

Hugh Dickins wrote:
> 
> Don't forget that tmpfs overflows into swap, so you could save money
> by adding adding more swap and cutting down on the RAM: though of
> course that will perform very poorly once it's actually using the
> swap, probably not the direction you want to go in.
> 

It can still outperform conventional disk filesystems, however, mostly 
because it doesn't have to worry one iota about consistency.

	-hpa

Re: What is the limit size of tmpfs /dev/shm ?

[Hugh Dickins]

On Wed, 6 Feb 2008, H. Peter Anvin wrote:
> Hugh Dickins wrote:
> > 
> > Don't forget that tmpfs overflows into swap, so you could save money
> > by adding adding more swap and cutting down on the RAM: though of
> > course that will perform very poorly once it's actually using the
> > swap, probably not the direction you want to go in.
> 
> It can still outperform conventional disk filesystems, however, mostly because
> it doesn't have to worry one iota about consistency.

In theory, yes, and should be true in practice before it hits swap.
But I think you'll find our swap handling is too primitive for tmpfs
to perform well once we hit swap.  Most filesystems pay considerable
attention to good performance within their constraints of correctness.
Whereas with tmpfs we've just never worried about the performance once
swapping.  It's used so you don't lose your data, but if you're really
expecting to be going to disk very much, better start with a filesystem
really designed for that.

Hugh

Re: What is the limit size of tmpfs /dev/shm ?

[H. Peter Anvin]

Hugh Dickins wrote:
> 
> In theory, yes, and should be true in practice before it hits swap.
> But I think you'll find our swap handling is too primitive for tmpfs
> to perform well once we hit swap.  Most filesystems pay considerable
> attention to good performance within their constraints of correctness.
> Whereas with tmpfs we've just never worried about the performance once
> swapping.  It's used so you don't lose your data, but if you're really
> expecting to be going to disk very much, better start with a filesystem
> really designed for that.
> 

That sounds like a problem in our overall swap handling, not 
specifically in tmpfs.  Now, I can't say anything concrete about heavy 
swap conditions, but in light swap conditions I have measured a 20x 
performance improvement(!) over ext3 on real workloads.

	-hpa

Re: What is the limit size of tmpfs /dev/shm ?

[Hugh Dickins]

On Wed, 6 Feb 2008, H. Peter Anvin wrote:
> 
> That sounds like a problem in our overall swap handling, not specifically in
> tmpfs.  Now, I can't say anything concrete about heavy swap conditions, but in
> light swap conditions I have measured a 20x performance improvement(!) over
> ext3 on real workloads.

Wow, I'm surprised.  I suppose I do jump to thinking of heavy swapping
when light swapping won't be so bad; but even so, 20x ext3 astonishes
me - ext3 wouldn't be anyone's choice for fastest, but even so...

I certainly guess too much and measure too little:
is there a useful test you could point me to?  TIA

Hugh

Re: What is the limit size of tmpfs /dev/shm ?

[H. Peter Anvin]

Hugh Dickins wrote:
> On Wed, 6 Feb 2008, H. Peter Anvin wrote:
>> That sounds like a problem in our overall swap handling, not specifically in
>> tmpfs.  Now, I can't say anything concrete about heavy swap conditions, but in
>> light swap conditions I have measured a 20x performance improvement(!) over
>> ext3 on real workloads.
> 
> Wow, I'm surprised.  I suppose I do jump to thinking of heavy swapping
> when light swapping won't be so bad; but even so, 20x ext3 astonishes
> me - ext3 wouldn't be anyone's choice for fastest, but even so...
> 
> I certainly guess too much and measure too little:
> is there a useful test you could point me to?  TIA
> 

The specific application was this:

- extract a kernel tarball
- "make distclean"
- cp -al the resulting tree
- apply a patch to each tree
- do a diff between the trees
- delete all files

... repeat something like 20,000 times.

	-hpa

Re: What is the limit size of tmpfs /dev/shm ?

[Hugh Dickins]

On Wed, 6 Feb 2008, H. Peter Anvin wrote:
> 
> The specific application was this:
> 
> - extract a kernel tarball
> - "make distclean"
> - cp -al the resulting tree
> - apply a patch to each tree
> - do a diff between the trees
> - delete all files
> 
> ... repeat something like 20,000 times.

Right, thanks.

Certainly extracting a tarball can be a surprising challenge
for some filesystems.

And you remind me that of course, to the extent that a load is
about metadata (lots of creates for example), tmpfs can hardly
lose since its metadata never goes out to swap.  But there is
plenty of data in your test above.

Hugh

What is the limit size of tmpfs /dev/shm ?

[Mika Lawando]

Hello Kernel Users,

is there a size limit for tmpfs for the /dev/shm filesystem?
Normally its default size is set to 2 GB. Is it possible to create a 2 
TB (Terrabyte) filesystem with tmpfs?
Or is there a maximum size defined in the linux kernel?

-- 
Best regards,
Mika Lawando