jffs2: -ENOSPC when truncating file?!

jffs2: -ENOSPC when truncating file?!

[Pavel Machek]

Hi!

I'm trying to free space by truncating big file, and I get:

root@fic-gta01:~# ls -al gps.nmea 
-rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
root@fic-gta01:~# > gps.nmea 
-sh: cannot create gps.nmea: No space left on device
root@fic-gta01:~# rm gps.nmea
root@fic-gta01:~# > gps.nmea
root@fic-gta01:~# 

:-(

Kernel is not-so-ancient, on openmoko:

root@fic-gta01:~# uname -a
Linux fic-gta01 2.6.24 #1 PREEMPT Mon Feb 18 21:30:53 GMT 2008 armv4tl
unknown
root@fic-gta01:~# 


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: jffs2: -ENOSPC when truncating file?!

[David Woodhouse]

On Sun, 2008-02-24 at 00:57 +0100, Pavel Machek wrote:
> Hi!
> 
> I'm trying to free space by truncating big file, and I get:
> 
> root@fic-gta01:~# ls -al gps.nmea 
> -rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
> root@fic-gta01:~# > gps.nmea 
> -sh: cannot create gps.nmea: No space left on device
> root@fic-gta01:~# rm gps.nmea
> root@fic-gta01:~# > gps.nmea
> root@fic-gta01:~# 

You need to write a log entry indicating the new length of the file.
There is no space for new log entries.

There is a special case for removal -- 'rm gps.nmea' would work. Perhaps
we should add a special case for truncation too, so that it can also use
the extra pool of free space.

-- 
dwmw2

Re: jffs2: -ENOSPC when truncating file?!

[Jörn Engel]

On Sun, 24 February 2008 09:36:07 +0900, David Woodhouse wrote:
> On Sun, 2008-02-24 at 00:57 +0100, Pavel Machek wrote:
> > 
> > I'm trying to free space by truncating big file, and I get:
> > 
> > root@fic-gta01:~# ls -al gps.nmea 
> > -rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
> > root@fic-gta01:~# > gps.nmea 
> > -sh: cannot create gps.nmea: No space left on device
> > root@fic-gta01:~# rm gps.nmea
> > root@fic-gta01:~# > gps.nmea
> > root@fic-gta01:~# 
> 
> You need to write a log entry indicating the new length of the file.
> There is no space for new log entries.
> 
> There is a special case for removal -- 'rm gps.nmea' would work. Perhaps
> we should add a special case for truncation too, so that it can also use
> the extra pool of free space.

Could a naïve implementation of this get exploited by doing a large
number of truncates that just shave single bytes off various files?

Looks like the safe way to do it would be to write out a replacement
node for the truncated node, if the special case ever triggers.

Jörn

-- 
"[One] doesn't need to know [...] how to cause a headache in order
to take an aspirin."
-- Scott Culp, Manager of the Microsoft Security Response Center, 2001

Re: jffs2: -ENOSPC when truncating file?!

[Pavel Machek]

On Sun 2008-02-24 09:36:07, David Woodhouse wrote:
> 
> On Sun, 2008-02-24 at 00:57 +0100, Pavel Machek wrote:
> > Hi!
> > 
> > I'm trying to free space by truncating big file, and I get:
> > 
> > root@fic-gta01:~# ls -al gps.nmea 
> > -rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
> > root@fic-gta01:~# > gps.nmea 
> > -sh: cannot create gps.nmea: No space left on device
> > root@fic-gta01:~# rm gps.nmea
> > root@fic-gta01:~# > gps.nmea
> > root@fic-gta01:~# 
> 
> You need to write a log entry indicating the new length of the file.
> There is no space for new log entries.
> 
> There is a special case for removal -- 'rm gps.nmea' would work. Perhaps
> we should add a special case for truncation too, so that it can also use
> the extra pool of free space.

Yes, that would be nice. I somehow assumed that truncate can't fail
for -ENOSPC ... I was trying to actually free some space on the
filesystem...
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: jffs2: -ENOSPC when truncating file?!

[David Woodhouse]

On Sun, 2008-02-24 at 07:57 +0100, Jörn Engel wrote:
> Could a naïve implementation of this get exploited by doing a large
> number of truncates that just shave single bytes off various files?

Yeah, which is why _my_ naïve implementation would do it for
truncate-to-zero instead of just _any_ truncate (which could even be
truncate-to-larger).

A more complex version might allow _any_ transaction to eat into the
ALLOC_DELETION pool if it is ultimately going to reduce the amount of
space taken on the file system -- even overwriting 'real' data with
zeroes which compress better. That's going to be hard to calculate in
the general case though.

If allowing only truncate-to-zero isn't good enough, perhaps we could
allow truncation to use the ALLOC_DELETION pool when it's going to
obsolete at least one full data node. That's not so hard to check.

-- 
dwmw2

Re: jffs2: -ENOSPC when truncating file?!

[Jörn Engel]

On Sun, 24 February 2008 18:57:32 +0800, David Woodhouse wrote:
> On Sun, 2008-02-24 at 07:57 +0100, Jörn Engel wrote:
> > Could a naïve implementation of this get exploited by doing a large
> > number of truncates that just shave single bytes off various files?
> 
> Yeah, which is why _my_ naïve implementation would do it for
> truncate-to-zero instead of just _any_ truncate (which could even be
> truncate-to-larger).

Truncate-to-larger is trivial to check.  Almost every filesystem does it
somewhere, including JFFS2. ;)

But yeah, truncate-to-zero should catch the common case.

> If allowing only truncate-to-zero isn't good enough, perhaps we could
> allow truncation to use the ALLOC_DELETION pool when it's going to
> obsolete at least one full data node. That's not so hard to check.

I would simply always write out a "full" replacement node, i.e. the
complete tail page for the file.  No need to check if an old node gets
obsoleted, we just made sure it does.  Anyway, it is your baby, so you
get to change the dirty diapers and pick your favorite pair of clean
ones.

Jörn

-- 
Joern's library part 3:
http://inst.eecs.berkeley.edu/~cs152/fa05/handouts/clark-test.pdf

Re: jffs2: -ENOSPC when truncating file?!

[Jamie Lokier]

Pavel Machek wrote:
> > You need to write a log entry indicating the new length of the file.
> > There is no space for new log entries.
> > 
> > There is a special case for removal -- 'rm gps.nmea' would work. Perhaps
> > we should add a special case for truncation too, so that it can also use
> > the extra pool of free space.
> 
> Yes, that would be nice. I somehow assumed that truncate can't fail
> for -ENOSPC ... I was trying to actually free some space on the
> filesystem...

Same here!  When I got ENOSPC from truncate, trying to free some
space, I was so surprised (and a bit disappointed) that I assumed
removal could fail too.  So now I'm pleasantly surprised to learn I
can at least remove a file.

It does seem odd that truncate to zero length can fail.  It is
guaranteed to free up one or more data nodes, so there should be
enough space for the size change node, provided GC is invoked when
necessary and the node sizes are compatible for this in corner cases.

-- Jamie

Re: jffs2: -ENOSPC when truncating file?!

[Pavel Machek]

On Sun 2008-02-24 18:57:32, David Woodhouse wrote:
> 
> On Sun, 2008-02-24 at 07:57 +0100, Jörn Engel wrote:
> > Could a na??ve implementation of this get exploited by doing a large
> > number of truncates that just shave single bytes off various files?
> 
> Yeah, which is why _my_ na??ve implementation would do it for
> truncate-to-zero instead of just _any_ truncate (which could even be
> truncate-to-larger).
> 
> A more complex version might allow _any_ transaction to eat into the
> ALLOC_DELETION pool if it is ultimately going to reduce the amount of
> space taken on the file system -- even overwriting 'real' data with
> zeroes which compress better. That's going to be hard to calculate in
> the general case though.
> 
> If allowing only truncate-to-zero isn't good enough, perhaps we could
> allow truncation to use the ALLOC_DELETION pool when it's going to
> obsolete at least one full data node. That's not so hard to check.

I believe truncate-to-zero is good enough.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: jffs2: -ENOSPC when truncating file?!

[Alexander Belyakov]

On Sun, Feb 24, 2008 at 2:57 AM, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>
>  I'm trying to free space by truncating big file, and I get:
>
>  root@fic-gta01:~# ls -al gps.nmea
>  -rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
>  root@fic-gta01:~# > gps.nmea
>  -sh: cannot create gps.nmea: No space left on device
>  root@fic-gta01:~# rm gps.nmea
>  root@fic-gta01:~# > gps.nmea
>  root@fic-gta01:~#
>
>  :-(
>
>  Kernel is not-so-ancient, on openmoko:

Hi!

JFFS2 has known bug in GC leading to -ENOSPC upon file removal. Time
ago I've sent a patch with detailed explanations of what is going on.
Probably your case is the same.

Here is the patch
http://lists.infradead.org/pipermail/linux-mtd/2007-December/020091.html
(though infradead mailing list archive is missing most of messages
from this thread)

And here are some details
http://lists.infradead.org/pipermail/linux-mtd/2007-June/018756.html

Alexander

Re: jffs2: -ENOSPC when truncating file?!

[Pavel Machek]

Hi!

> >  I'm trying to free space by truncating big file, and I get:
> >
> >  root@fic-gta01:~# ls -al gps.nmea
> >  -rw-r--r--    1 root     root      2332070 Feb 19 22:13 gps.nmea
> >  root@fic-gta01:~# > gps.nmea
> >  -sh: cannot create gps.nmea: No space left on device
> >  root@fic-gta01:~# rm gps.nmea
> >  root@fic-gta01:~# > gps.nmea
> >  root@fic-gta01:~#
> >
> >  :-(
> >
> >  Kernel is not-so-ancient, on openmoko:
> 
> Hi!
> 
> JFFS2 has known bug in GC leading to -ENOSPC upon file removal. Time
> ago I've sent a patch with detailed explanations of what is going on.
> Probably your case is the same.
> 
> Here is the patch
> http://lists.infradead.org/pipermail/linux-mtd/2007-December/020091.html
> (though infradead mailing list archive is missing most of messages
> from this thread)
> 
> And here are some details
> http://lists.infradead.org/pipermail/linux-mtd/2007-June/018756.html

Patch is here (I whitespace-damaged it, sorry], and I've seen no
discussion.. should it just be applied?
								Pavel

diff -uNr a/fs/jffs2/wbuf.c b/fs/jffs2/wbuf.c
--- a/fs/jffs2/wbuf.c	2007-05-25 01:22:47.000000000 +0400
+++ b/fs/jffs2/wbuf.c	2007-06-20 18:45:09.000000000 +0400
@@ -117,7 +117,7 @@

 		D1(printk(KERN_DEBUG "Removing eraseblock at 0x%08x
from erasable_pending_wbuf_list...\n", jeb->offset));
 		list_del(this);
-		if ((jiffies + (n++)) & 127) {
+		if (((jiffies + (n++)) & 127) &&
!list_empty(&c->erasable_list)) {
 			/* Most of the time, we just erase it immediately. Otherwise we
 			   spend ages scanning it on mount, etc. */
 			D1(printk(KERN_DEBUG "...and adding to
erase_pending_list\n"));



-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: jffs2: -ENOSPC when truncating file?!

[Jörn Engel]

On Mon, 3 March 2008 13:15:11 +0100, Pavel Machek wrote:
> 
> Patch is here (I whitespace-damaged it, sorry], and I've seen no
> discussion.. should it just be applied?

There was discussion.  The patch is a pure band-aid.  It should not make
a difference at all, so clearly something is very wrong and we should
dig deeper to find the real problem.  So far noone has done it.

Jörn

-- 
All models are wrong. Some models are useful.
-- George Box

Re: jffs2: -ENOSPC when truncating file?!

[Alexander Belyakov]

On Mon, Mar 3, 2008 at 3:15 PM, Pavel Machek <pavel@ucw.cz> wrote:
>
>  Patch is here (I whitespace-damaged it, sorry], and I've seen no
>  discussion.. should it just be applied?
>                                                                 Pavel

Being very simple workaround, this patch helps to avoid -ENOSPC bug
until better (proper) solution is found. IMO it is worth to try this
patch (at least to make sure that you are indeed dealing with the same
bug).

-a