UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Tomasz Chmielewski]

Artem Bityutskiy wrote:

> I've renamed the thread because I do not like this flamish discussion
> to me mixed with the technical one.
> 
> Jörn Engel wrote:
>> Shiny numbers!  Performance has improved significantly in the last six
>> month.  Still worth a closer look.
> We'll re-run them. Does logfs support write-back? Does it support compression?

For me, the motivators to wait for LogFS are mainly the facts that it
can work on traditional block devices, and not only on pure flash:

1. It works on normal block devices and it supports transparent compression

Today, a 64 GB SSD/flash-based media costs ~about the same as a 1 TB
hard disk. This makes flash very expensive to use; compression can
compensate that cost a bit (will depend on the usage, of course).

I believe there is no other Linux filesystem which can do transparent
compression on block devices.


2. It does wear-levelling also on normal block devices

Although it doesn't sound normal to do wear-levelling twice (most
flash-based block devices do wear-levelling on their own), I had a flash
corruption after just ~one month of using RAID bitmap on a IDE-flash
disk formatted with ext3. Apparently, device-level wear-levelling wasn't
spreading updates of RAID bitmap file well enough.

(...)

> This basically means it is unfinished. Handling dynamic bad blocks is a *must*
> if you are going to work on NAND, especially on MLC NAND which are not as
> reliable as SLC.
> I think you should bluntly say about this when you submit patches to prevent
> people from starting using it in production.

I too wouldn't use LogFS today in a production environment - it is still 
not feature complete and not widely tested.
I wouldn't use btrfs or ext4 today for the very same reason.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Tomasz Chmielewski wrote:
> For me, the motivators to wait for LogFS are mainly the facts that it
> can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file
systems for block devices. UBIFS may work on top of a block device as
well (just needs few hacks to make it possible) - it is not a problem
at all, it is just _senseless_.

JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed
for _flash_, which has completely different work model then block device.
They are _native_ flash file systems.
Here are more details: http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd

The traditional FSes _cannot_ work on top of flash. The solution for this
is using FTL, which emulates a block device on top of flash. It _hides_ the
real device, and fakes a block device for you. And you can use traditional
FSes on top of that fake block device.

The whole _point_ of this separate class of FSes is because we believe
we may do much _better_ job if we use flash _natively_, instead of using
FTL. FTL is the place where you loose performance, reliability, and so on.

And you are saying about using a native flash FS on top of a block device
like an SD card. This is just not sane: SD card first emulates a block device
for you, looses performance at this point, then you again emulate a flash
on top of this, and suffer from this again.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 11:45:56 +0300, Artem Bityutskiy wrote:
> Tomasz Chmielewski wrote:
> >For me, the motivators to wait for LogFS are mainly the facts that it
> >can work on traditional block devices, and not only on pure flash:
> 
> Sorry Thomasz, for me this makes zero sense. There are _much_ better file
> systems for block devices. UBIFS may work on top of a block device as
> well (just needs few hacks to make it possible) - it is not a problem
> at all, it is just _senseless_.
> 
> JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed
> for _flash_, which has completely different work model then block device.
> They are _native_ flash file systems.
> Here are more details: 
> http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd
> 
> The traditional FSes _cannot_ work on top of flash. The solution for this
> is using FTL, which emulates a block device on top of flash. It _hides_ the
> real device, and fakes a block device for you. And you can use traditional
> FSes on top of that fake block device.
> 
> The whole _point_ of this separate class of FSes is because we believe
> we may do much _better_ job if we use flash _natively_, instead of using
> FTL. FTL is the place where you loose performance, reliability, and so on.
> 
> And you are saying about using a native flash FS on top of a block device
> like an SD card. This is just not sane: SD card first emulates a block 
> device
> for you, looses performance at this point, then you again emulate a flash
> on top of this, and suffer from this again.

About a year ago I would have used roughly the same argument.  But in
the meanwhile I have accepted the fact that any piece of flash I'll use
in a notebook in the next five years will likely use SATA or USB as a
transport medium.

Jörn

-- 
...one more straw can't possibly matter...
-- Kirby Bakken

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Jörn Engel wrote:
> About a year ago I would have used roughly the same argument.  But in
> the meanwhile I have accepted the fact that any piece of flash I'll use
> in a notebook in the next five years will likely use SATA or USB as a
> transport medium.

Well, Thomasz was talking not about the transport I believe. If you have a
raw flash access via SATA or USB, then this is handled on the driver level
and does not any require FS support. JFFS2 or UBIFS may also work with such a
device, because it is just raw flash. But I know that you know that most of
the FTL-enabled stuff does not give use raw flash access.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 12:09:35 +0300, Artem Bityutskiy wrote:
> Jörn Engel wrote:
> >About a year ago I would have used roughly the same argument.  But in
> >the meanwhile I have accepted the fact that any piece of flash I'll use
> >in a notebook in the next five years will likely use SATA or USB as a
> >transport medium.
> 
> Well, Thomasz was talking not about the transport I believe.

When the day comes and I can actually use raw flash with a SATA or USB
transport medium, I will be more careful and not imply that they all
expose a block device interface.  Today they do.

[ With the exception of Alauda, which has a userbase of about 1 - me. ]

Jörn

-- 
It does not matter how slowly you go, so long as you do not stop.
-- Confucius

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Tomasz Chmielewski]

Artem Bityutskiy schrieb:

> Tomasz Chmielewski wrote:
>> For me, the motivators to wait for LogFS are mainly the facts that it
>> can work on traditional block devices, and not only on pure flash:
> 
> Sorry Thomasz, for me this makes zero sense. There are _much_ better file
> systems for block devices.

Such as?
Flash (also on block devices) is slow and expensive (when compared to 
modern hard disks) and therefore compression is *very* useful here.
Not only it can potentially save you money; reads and writes will be 
smaller/faster (unless you're editing music and videos, where one won't 
use flash anyway), flash will wear out slower etc.

Is there a filesystem for Linux which can provide transparent 
compression and works for block devices (other than user-space NTFS or 
some outdated ext2 patches)? I don't think so.


> UBIFS may work on top of a block device as
> well (just needs few hacks to make it possible) - it is not a problem
> at all, it is just _senseless_.

Do you mean using hacks like block2mtd? It's hacky, and pretty hard to 
boot a system this way (need to build own initramfs, with a static 
block2mtd or loads of libraries - not something an average user would 
like to do; no distro supports it; updating a kernel would be a pain etc.).


> JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed
> for _flash_, which has completely different work model then block device.
> They are _native_ flash file systems.
> Here are more details: 
> http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd

It's a good comparison, but it doesn't show disadvantages of using 
traditional filesystems on flash-based block devices.

I just mentioned the reasons why a filesystem like LogFS would be useful 
on block devices: there are valid reasons to do it.


(...)

> The whole _point_ of this separate class of FSes is because we believe
> we may do much _better_ job if we use flash _natively_, instead of using
> FTL. FTL is the place where you loose performance, reliability, and so on.

True.
Unfortunately, there is no way to access flash directly on flash-based 
block devices (USB-sticks, IDE-flash disks, SSD disks etc.).

Therefore, could an answer be: use a traditional filesystem?

Unfortunately, traditional filesystems were rather designed for rotating 
media / cheap disks (no transparent compression; tend to accumulate 
writes in one area of the disk - more on that - below).


> And you are saying about using a native flash FS on top of a block device
> like an SD card. This is just not sane: SD card first emulates a block 
> device
> for you, looses performance at this point, then you again emulate a flash
> on top of this, and suffer from this again.

Performance is only one factor in the equation. Other factors are: cost 
and reliability.

I speak from experience: flash-based block devices tend to have poor 
wear-levelling (at least Transcend IDE-flash disks).
To reproduce:
- format a 2 GB Transcend IDE-flash disk with ext3
- write a small file (50-100 kB)
- update that file ~several hundred thousand times - as you finish, 
IDE-flash disk will have 200-300 badblocks

If wear-levelling on the underlying IDE-flash device was any decent, 
writes would be spread on the whole ~2GB surface, totalling in many more 
successful writes.


Again: this is my experience, although it may contradict the theory 
underlined in the link you gave earlier.
You have much more experience with flash file systems: correct me where 
I'm wrong.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Tomasz Chmielewski wrote:
> Such as?

ext3 for example.

> Flash (also on block devices) is slow and expensive (when compared to 
> modern hard disks) and therefore compression is *very* useful here.
Well, if you are ready to trade performance to compression, then well,
go ahead :-) May be I used too strong wording, but I wanted to say then
use raw flash then. But I'd also consider implementing compression support
for a block based FS. Reiser4 claimed to have it for example.

> Do you mean using hacks like block2mtd? It's hacky, and pretty hard to 
> boot a system this way (need to build own initramfs, with a static 
> block2mtd or loads of libraries - not something an average user would 
> like to do; no distro supports it; updating a kernel would be a pain etc.).
Well, ok, it still sounds strange for me, but you may use JFFS2 and UBIFS
with block2mtd as well if you really want to.

> True.
> Unfortunately, there is no way to access flash directly on flash-based 
> block devices (USB-sticks, IDE-flash disks, SSD disks etc.).
Yeah, that's a pity :-(

> Unfortunately, traditional filesystems were rather designed for rotating 
> media / cheap disks (no transparent compression; tend to accumulate 
> writes in one area of the disk - more on that - below).
Sure.

> Performance is only one factor in the equation. Other factors are: cost 
> and reliability.
> 
> I speak from experience: flash-based block devices tend to have poor 
> wear-levelling (at least Transcend IDE-flash disks).
> To reproduce:
> - format a 2 GB Transcend IDE-flash disk with ext3
> - write a small file (50-100 kB)
> - update that file ~several hundred thousand times - as you finish, 
> IDE-flash disk will have 200-300 badblocks
Yeah, that's bad. But if you have a bad FTL, surely there is not guarantee
a flash FS will help? Isn't it better to use better hardware?

We did some experiments with MMC cards and we were unable to wear them
out with re-writing the same sectors again and again. This suggests there
_is_ better FTL hardware then that USB stick you was using.

Anyway, your original mail said Logfs can work with block devices. My answer -
UBIFS too, but this is very strange to do this IMO. But OK, it might is not
senseless, sorry for the wording. :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Andi Kleen]

Artem Bityutskiy <dedekind@yandex.ru> writes:

> Tomasz Chmielewski wrote:
> > For me, the motivators to wait for LogFS are mainly the facts that it
> > can work on traditional block devices, and not only on pure flash:
> 
> Sorry Thomasz, for me this makes zero sense. There are _much_ better file
> systems for block devices.

I think he refers to flash disks appearing as block devices, like
usb sticks or similar.

-Andi

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Andi Kleen wrote:
> Artem Bityutskiy <dedekind@yandex.ru> writes:
> 
>> Tomasz Chmielewski wrote:
>>> For me, the motivators to wait for LogFS are mainly the facts that it
>>> can work on traditional block devices, and not only on pure flash:
>> Sorry Thomasz, for me this makes zero sense. There are _much_ better file
>> systems for block devices.
> 
> I think he refers to flash disks appearing as block devices, like
> usb sticks or similar.

Right, I also meant that in my opinion it makes more sense to use traditional
file-systems like ext3 on USB-key/MMC and the like stuff (which I confusingly
referred as "block devices"), or may be something more "heavy-weight" like
XFS or JFS (never tried them, though).

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[H. Peter Anvin]

Artem Bityutskiy wrote:
> Andi Kleen wrote:
>> Artem Bityutskiy <dedekind@yandex.ru> writes:
>>
>>> Tomasz Chmielewski wrote:
>>>> For me, the motivators to wait for LogFS are mainly the facts that it
>>>> can work on traditional block devices, and not only on pure flash:
>>> Sorry Thomasz, for me this makes zero sense. There are _much_ better 
>>> file
>>> systems for block devices.
>>
>> I think he refers to flash disks appearing as block devices, like
>> usb sticks or similar.
> 
> Right, I also meant that in my opinion it makes more sense to use 
> traditional
> file-systems like ext3 on USB-key/MMC and the like stuff (which I 
> confusingly
> referred as "block devices"), or may be something more "heavy-weight" like
> XFS or JFS (never tried them, though).
> 

Well, even auto-levelling storage should benefit from a filesystem which 
minimizes the total number of flash sectors churned, which means doing 
as few writes as possible and to large, contiguous sections.

	-hpa

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Willy Tarreau]

On Tue, Apr 01, 2008 at 09:27:36AM -0700, H. Peter Anvin wrote:
> Artem Bityutskiy wrote:
> >Andi Kleen wrote:
> >>Artem Bityutskiy <dedekind@yandex.ru> writes:
> >>
> >>>Tomasz Chmielewski wrote:
> >>>>For me, the motivators to wait for LogFS are mainly the facts that it
> >>>>can work on traditional block devices, and not only on pure flash:
> >>>Sorry Thomasz, for me this makes zero sense. There are _much_ better 
> >>>file
> >>>systems for block devices.
> >>
> >>I think he refers to flash disks appearing as block devices, like
> >>usb sticks or similar.
> >
> >Right, I also meant that in my opinion it makes more sense to use 
> >traditional
> >file-systems like ext3 on USB-key/MMC and the like stuff (which I 
> >confusingly
> >referred as "block devices"), or may be something more "heavy-weight" like
> >XFS or JFS (never tried them, though).
> >
> 
> Well, even auto-levelling storage should benefit from a filesystem which 
> minimizes the total number of flash sectors churned, which means doing 
> as few writes as possible and to large, contiguous sections.

Exactly. At exosec, we ship one appliance which writes statistics to a
partition on a compactflash every 5 minutes. We preferred to go with JFFS2
exactly because of this reason. We never had any problem proceeding this way.
I'm not sure if it would have been the same with ext2 though.

Willy

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Willy Tarreau wrote:
>> Well, even auto-levelling storage should benefit from a filesystem which 
>> minimizes the total number of flash sectors churned, which means doing 
>> as few writes as possible and to large, contiguous sections.
> 
> Exactly. At exosec, we ship one appliance which writes statistics to a
> partition on a compactflash every 5 minutes. We preferred to go with JFFS2
> exactly because of this reason. We never had any problem proceeding this way.
> I'm not sure if it would have been the same with ext2 though.
>

Yes, as I agreed in a previous mail this may make sense in some cases.

But in general it is not a good approach. Basically, it is wastage of resources.
Indeed, first the firmware on MMC/SD/etc makes efforts to make flash look
like a block device. It gives you in-place updates, but by cost of performance
and reliability. Then you just drop this nice property, and use JFFS2, which
assumes it has only out-of-place updates. But if this solves the task you have
- fine!

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Willy Tarreau]

On Wed, Apr 02, 2008 at 07:47:25AM +0300, Artem Bityutskiy wrote:
> Willy Tarreau wrote:
> >>Well, even auto-levelling storage should benefit from a filesystem which 
> >>minimizes the total number of flash sectors churned, which means doing 
> >>as few writes as possible and to large, contiguous sections.
> >
> >Exactly. At exosec, we ship one appliance which writes statistics to a
> >partition on a compactflash every 5 minutes. We preferred to go with JFFS2
> >exactly because of this reason. We never had any problem proceeding this 
> >way.
> >I'm not sure if it would have been the same with ext2 though.
> >
> 
> Yes, as I agreed in a previous mail this may make sense in some cases.
> 
> But in general it is not a good approach. Basically, it is wastage of 
> resources.
> Indeed, first the firmware on MMC/SD/etc makes efforts to make flash look
> like a block device. It gives you in-place updates,

One of the problem is that unless you crash-test your flash cards, you will
never know if their wear-leveling algorithm is fine or not. And I suspect
that nowadays, due to very large consumer demand, flash cards price drop
at the cost of reliability. I think that most of those not flagged
"industrial-grade" do absolutely zero wear-leveling, because they are sold
to people using them in digital cameras, and they will never kill their
device with such a usage.

> but by cost of performance
> and reliability. Then you just drop this nice property, and use JFFS2, which
> assumes it has only out-of-place updates. But if this solves the task you 
> have

I'm certainly not the only one with this requirement. A lot of embedded
motherboards come with IDE compactflash connectors. This is very convenient,
but if you need to keep informations between reboots, you have to write to
the device anyway. If you need to do that very often, either you pray for
the device to be very reliable, or you take all the chances on your side
by adding your own wear-leveling "just in case".

Cheers,
Willy

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Willy Tarreau wrote:
> One of the problem is that unless you crash-test your flash cards, you will
> never know if their wear-leveling algorithm is fine or not. And I suspect
> that nowadays, due to very large consumer demand, flash cards price drop
> at the cost of reliability. I think that most of those not flagged
> "industrial-grade" do absolutely zero wear-leveling, because they are sold
> to people using them in digital cameras, and they will never kill their
> device with such a usage.

Sure, I know about this problem. My point was that in this case it is wiser
to use bare flash and put JFFS2 on it, instead of using this black box
MMC/etc and then put JFFS2 on it.

> I'm certainly not the only one with this requirement. A lot of embedded
> motherboards come with IDE compactflash connectors. This is very convenient,
> but if you need to keep informations between reboots, you have to write to
> the device anyway. If you need to do that very often, either you pray for
> the device to be very reliable, or you take all the chances on your side
> by adding your own wear-leveling "just in case".

OK. Fair enough. Although stuff exists, but this does not necessarily mean
this a good design :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Tomasz Chmielewski]

Artem Bityutskiy schrieb:
> Tomasz Chmielewski wrote:

(...)

>> Performance is only one factor in the equation. Other factors are: 
>> cost and reliability.
>>
>> I speak from experience: flash-based block devices tend to have poor 
>> wear-levelling (at least Transcend IDE-flash disks).
>> To reproduce:
>> - format a 2 GB Transcend IDE-flash disk with ext3
>> - write a small file (50-100 kB)
>> - update that file ~several hundred thousand times - as you finish, 
>> IDE-flash disk will have 200-300 badblocks
> Yeah, that's bad. But if you have a bad FTL, surely there is not guarantee
> a flash FS will help? Isn't it better to use better hardware?
> 
> We did some experiments with MMC cards and we were unable to wear them
> out with re-writing the same sectors again and again. This suggests there
> _is_ better FTL hardware then that USB stick you was using.
> 
> Anyway, your original mail said Logfs can work with block devices. My 
> answer -
> UBIFS too, but this is very strange to do this IMO. But OK, it might is not
> senseless, sorry for the wording. :-)

I was thinking why my IDE-flash disk died so soon[1] and how efficient 
can an internal wear-levelling be in devices which hide its "flashiness" 
(USB-sticks, IDE-flash disks etc.).

Internal wear-levelling mechanism doesn't have a clue about free space 
on the filesystem - it that case, how can it do any efficient 
wear-levelling?



[1] Well, it didn't die, really. Once I removed the file which was 
showing I/O errors and did "dd if=/dev/zero of=bigfile", there are no 
badblocks anymore - probably remapped.



-- 
Tomasz Chmielewski
http://wpkg.org

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Wed, 2 April 2008 16:17:25 +0200, Tomasz Chmielewski wrote:
> 
> [1] Well, it didn't die, really. Once I removed the file which was 
> showing I/O errors and did "dd if=/dev/zero of=bigfile", there are no 
> badblocks anymore - probably remapped.

Could have been soft errors.  Heaven knows how those devices report
uncorrectable errors, but claiming the block to be bad is a good guess.
Once blocks with uncorrectable errors are rewritten (the physical
blocks, not what the device reports), they are perfectly usable again.

Jörn

-- 
They laughed at Galileo.  They laughed at Copernicus.  They laughed at
Columbus. But remember, they also laughed at Bozo the Clown.
-- unknown

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Pavel Machek]

Hi!

> For me, the motivators to wait for LogFS are mainly the 
> facts that it
> can work on traditional block devices, and not only on 
> pure flash:
> 
> 1. It works on normal block devices and it supports 
> transparent compression
> 
> Today, a 64 GB SSD/flash-based media costs ~about the 
> same as a 1 TB
> hard disk. This makes flash very expensive to use; 
> compression can
> compensate that cost a bit (will depend on the usage, of 
> course).
> 
> I believe there is no other Linux filesystem which can 
> do transparent
> compression on block devices.

I'd like compressed filesystem for maps and lingvistic data... but
will the flash flesystems have 'reasonable' performance when used on
harddrive?
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Wed, 9 April 2008 23:09:07 +0200, Pavel Machek wrote:
> 
> I'd like compressed filesystem for maps and lingvistic data... but
> will the flash flesystems have 'reasonable' performance when used on
> harddrive?

If someone implemented readpages(), that might be possible - depending
on your definition of 'reasonable' and your workload.

Writes will hit the disk roughly in order.  When readahead requests a
bunch of pages, there is some chance of them being adjacent and having
the block layer combine most of the bios into a few large ones.

The main drawbacks are:
- No reservations.  If data is written in random order or several
  writers chew away in parallel, write order will be fairly pessimal.
- No readahead yet.  Wouldn't be hard to do.
- Garbage collection completely ignores fragmentation.  If segments are
  needed and one contains a nice long extend from a single file, that
  data will be written elsewhere, often split between two segments.
  Rinse, repeat and fragmentation will increase over time.
- File creation/deletion currently will cause disk heads to jump in
  triangles.  This hurts write performance on most flash media as well,
  so it will get changed reasonably soon.

All of those are solvable.  Some will definitely be solved because the
help performance on flash media as well.  Other may or may not.

Jörn

-- 
When people work hard for you for a pat on the back, you've got
to give them that pat.
-- Robert Heinlein

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Jörn Engel wrote:
> Fair enough.
> 
> The obvious downside of all this is depending on UBI, which has a linear
> scan.  My goal was to avoid the linear scan completely.  It is a harder
> goal and I haven't reached it yet.  Imo it is reachable and I will
> continue going in that direction.

Yes, it was our core design decision. One of the reasons, we were not sure
this is technically possible to do on bare flashes. I mean, it just looked
so complex to have all in one, so we figured that was a good split, where
you can cut on big work on two smaller separate ones. The benefit of this
is obvious - we have created a complete system, which is not perfect though
and have scalability issues.

Our point is that UBI is scalable enough for the time being.

I wrote some documentation about this in UBI FAQ and UBIFS FAQ:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_scalability

We can now improve scalability of UBI without affecting UBIFS - it has some
potential. And we may develop UBI2 which would be more much more scalable,
but this is a big project and we are not planning to do this so far. Others
could do.

So in other words, using UBI allowed us to get a finished system faster. I
meets our's and many other people's requirements, although it has issues if
you try to use it on really huge flashes, like 64GiB. That's a drawback.
But the good thing is that this would require re-working UBI layer, without
complete re-working of UBIFS. 

> You picked the route of using UBI, which makes a lot of stuff easier.
> It is a fair approach and I don't mind you taking it.  It has drawbacks,
> but so has everything else.
Agree.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 12:46:12 +0300, Artem Bityutskiy wrote:
> Jörn Engel wrote:
> >Write-back of data: currently not.
> 
> Taking into account this, bad block handling and other stuff I kindly ask 
> you
> not to do these "tricky" things like comparing amount of code lines.
> I believe if you implement all the things, you'll have _way_ more code. I
> believe you'll even have to re-design many things because write-back really
> affects the design a lot. Let's be fair.

Take those numbers with as much salt as you like.  Implementing
write-back caching for metadata actually reduced the line count by 100
or so.  And it did affect the design a lot, I agree.

Jörn

-- 
The wise man seeks everything in himself; the ignorant man tries to get
everything from somebody else.
-- unknown

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 12:39:25 +0300, Artem Bityutskiy wrote:
> 
> I'm not sure what you mean. In UBIFS we have lprops area, where we store
> per-LEB accounting. UBI lets it possible to have it on a fixed position.
> The accounting is a separate B-tree. Lprops area has its own independent
> garbage collector. You should probably refer the white paper for more
> information.

Fair enough.

The obvious downside of all this is depending on UBI, which has a linear
scan.  My goal was to avoid the linear scan completely.  It is a harder
goal and I haven't reached it yet.  Imo it is reachable and I will
continue going in that direction.

You picked the route of using UBI, which makes a lot of stuff easier.
It is a fair approach and I don't mind you taking it.  It has drawbacks,
but so has everything else.

Jörn

-- 
Anything that can go wrong, will.
-- Finagle's Law

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Jörn Engel wrote:
> Write-back of data: currently not.

Taking into account this, bad block handling and other stuff I kindly ask you
not to do these "tricky" things like comparing amount of code lines.
I believe if you implement all the things, you'll have _way_ more code. I
believe you'll even have to re-design many things because write-back really
affects the design a lot. Let's be fair.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Jörn Engel wrote:
> Correct.  I have a patch that caches this information in RAM, so the
> worst-case scenario is to do the full scan once - if the filesystem is
> near 100% full and performance is down the drain anyway.  The goal is
> obviously to store that information on the medium.

Anyway, if you do not store this information on the flash, you are going
to scan to acquire it. You may have caches, but they do not have to have
information (may be empty), then you scan. This will lead to scalability
problems at some point anyway.

>> UBIFS stores per-eraseblock information on the media in a B-tree, and
>> it also has lists of empty/dirty eraseblocks, which allow to very quickly
>> find the best eraseblock to garbage-collect or to write to.
> 
> So how do you solve the catch-22?

I'm not sure what you mean. In UBIFS we have lprops area, where we store
per-LEB accounting. UBI lets it possible to have it on a fixed position.
The accounting is a separate B-tree. Lprops area has its own independent
garbage collector. You should probably refer the white paper for more
information.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 08:56:45 +0300, Artem Bityutskiy wrote:
>
> I asked you some time ago to describe how you maintain per-eraseblock
> space accounting [1]. E.g., how you select an eraseblock for garbage
> collection, how do you store the accounting information.
> 
> You said you find eraseblocks by scanning. This means logfs is not
> really scalable because you may spend ages before you find anything
> appropriate. When the FS is almost full, yo need to scan nearly
> whole flash to find an eraseblock? So if I mount a nearly full 
> FS, and start writing, I'll get my request handled when nearly whole
> media is scanned?

Correct.  I have a patch that caches this information in RAM, so the
worst-case scenario is to do the full scan once - if the filesystem is
near 100% full and performance is down the drain anyway.  The goal is
obviously to store that information on the medium.

And you get the obvious catch-22.  When writing out how much free space
exists in which blocks, you occupy some free space in one of those
blocks and obsolete some in another.  So by writing the data you have
just written changes and should be written again.

Takes a bit of thought to solve.

> UBIFS stores per-eraseblock information on the media in a B-tree, and
> it also has lists of empty/dirty eraseblocks, which allow to very quickly
> find the best eraseblock to garbage-collect or to write to.

So how do you solve the catch-22?

Jörn

-- 
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it.
-- Brian W. Kernighan

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Jörn Engel]

On Tue, 1 April 2008 08:26:37 +0300, Artem Bityutskiy wrote:
> Jörn Engel wrote:
> >Shiny numbers!  Performance has improved significantly in the last six
> >month.  Still worth a closer look.
> We'll re-run them. Does logfs support write-back? Does it support 
> compression?

Write-back of metadata: yes.  Write-back of data: currently not.
Compression: yes.

> Is this patch the right one we should look at:
> http://logfs.org/logfs/patches

Yes.

> >>3. logfs does not seem to have bad-block handling.
> >
> >Bad blocks at mkfs time are handled, blocks turning bad later on aren't
> >yet.
> 
> This basically means it is unfinished. Handling dynamic bad blocks is a 
> *must*
> if you are going to work on NAND, especially on MLC NAND which are not as
> reliable as SLC.
> 
> I think you should bluntly say about this when you submit patches to prevent
> people from starting using it in production.
> 
> UBI handles I/O errors and gracefully recovers the data. And it will not be
> easy to do this in LogFS at all. If you have write error in the middle of an
> eraseblock, you have to recover data which means correcting all the indexing
> data structures which refer the data you recover. Correcting them may 
> require
> garbage collection, because you have to update them out of place. And what
> do you do if you are already in the middle of doing garbage-collection?
> 
> In other words, I believe it will take a lot of time and efforts to 
> implement
> this. And any speculation about the number of lines of code makes no sense
> before you finish your work.

You are right, bad-block handling is easier to implement in ubi than in
a filesystem.  That is one of the real benefits from using ubi.  And if
my other arguments about ubi would have been handled instead of answered
with flames, I would likely be using it as well.

> >>4. logfs does not seem to have wear-leveling.
> >It does.
> Could you please point the core functions which implement this and shortly
> describe the algorithm?
> 
> I grep'ed for "wear" and "leveling" and found only one match. Where should
> I look at?

fs/logfs/gc.c
Grep for s_ec_list.  It is currently quite aggressive and keeps the
spread between maximum erasecount and minimum erasecount at about 40.
Should become an mkfs option one of these days.

There is no dedicated subsystem for wear leveling.  There simply isn't
that much to do.

Jörn

-- 
Premature optimization is the root of all evil.
-- Donald Knuth

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Jörn Engel wrote:
> Code size difference is huge.  Ubi weighs some 11kloc, ubifs some 30,
> logfs some 8.
Please, lets refrain form unfair comparisons like this before logfs is
finished. Also, when you compare, please, take into account that UBI/UBIFS
has many times more commentaries in the code.

> Ubi scales linearly, as it does a large scan at init time.  It is still
> reasonably fast, as it reads just a few bytes worth of header per block.
> Logfs mounts in O(1) but will currently become mindbogglingly slow when
> the filesystem nears 100% full and write are purely random.  Not that
> any other flash filesystem would perform well under these conditions -
> it is the known worst case scenario.
I asked you some time ago to describe how you maintain per-eraseblock
space accounting [1]. E.g., how you select an eraseblock for garbage
collection, how do you store the accounting information.

You said you find eraseblocks by scanning. This means logfs is not
really scalable because you may spend ages before you find anything
appropriate. When the FS is almost full, yo need to scan nearly
whole flash to find an eraseblock? So if I mount a nearly full 
FS, and start writing, I'll get my request handled when nearly whole
media is scanned?

UBIFS stores per-eraseblock information on the media in a B-tree, and
it also has lists of empty/dirty eraseblocks, which allow to very quickly
find the best eraseblock to garbage-collect or to write to.

[1] http://lkml.org/lkml/2007/8/8/333

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

Sorry, forgot to paste the link :-)

Artem Bityutskiy wrote:
> We'll re-run them. Does logfs support write-back? Does it support 
> compression?
> Is this patch the right one we should look at:
http://www.lazybastard.org/logfs/patches

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

[Artem Bityutskiy]

I've renamed the thread because I do not like this flamish discussion
to me mixed with the technical one.

Jörn Engel wrote:
> Shiny numbers!  Performance has improved significantly in the last six
> month.  Still worth a closer look.
We'll re-run them. Does logfs support write-back? Does it support compression?
Is this patch the right one we should look at:

>> 3. logfs does not seem to have bad-block handling.
> 
> Bad blocks at mkfs time are handled, blocks turning bad later on aren't
> yet.

This basically means it is unfinished. Handling dynamic bad blocks is a *must*
if you are going to work on NAND, especially on MLC NAND which are not as
reliable as SLC.

I think you should bluntly say about this when you submit patches to prevent
people from starting using it in production.

UBI handles I/O errors and gracefully recovers the data. And it will not be
easy to do this in LogFS at all. If you have write error in the middle of an
eraseblock, you have to recover data which means correcting all the indexing
data structures which refer the data you recover. Correcting them may require
garbage collection, because you have to update them out of place. And what
do you do if you are already in the middle of doing garbage-collection?

In other words, I believe it will take a lot of time and efforts to implement
this. And any speculation about the number of lines of code makes no sense
before you finish your work.

>> 4. logfs does not seem to have wear-leveling.
> It does.
Could you please point the core functions which implement this and shortly
describe the algorithm?

I grep'ed for "wear" and "leveling" and found only one match. Where should
I look at?

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)