Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Indan Zupancic]

On Mon, February 26, 2007 15:45, Jens Axboe wrote:
> Test case is doing random reads from /dev/sdb, in chunks of 64kb:
>
> Engine          Depth   Processes       Bw (KiB/sec)
> ----------------------------------------------------
> libaio           200       100            2813
> syslet           200       100            3944
> libaio         20000         1            2793
> syslet         20000         1            3854
> sync (*)       20000         1            2866

If you have time, could you please add Threadlet results?

Considering how awful the syslet API is, and how nice the threadlet
one, it would be great if it turned out that Syslets aren't worth it.
If they are, a better API/ABI should be thought of.

Greetings,

Indan

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Wed, Feb 28 2007, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > Engine                Depth      Batch	Bw (KiB/sec)
> > libaio		20000	   8		21,125
> > syslet		20000	   8		19,610
> 
> i'd like to do something more about this to be more in line with libaio 
> - if nothing else then for the bragging rights ;-) It seems to me that a 
> drop of ~7% in throughput cannot be explained with any CPU overhead, it 
> must be some sort of queueing + IO scheduling effect - right?

syslet shows a slightly higher overhead, but nothing that will account
for any bandwidth change in this test. The box is obviously mostly idle
when running this test, it's not very CPU consuming. The IO pattern
issued is not the same, since libaio would commit IO [0..7], then
[8..15] and so on, where syslet would expose [0,8,16,24,32,40,48,56] and
then [1,9,17,25,33,41,49,57] etc. If iodepth_batch is set to 1 you'd
get a closer match wrt io pattern, but at a higher cost (increased
system calls, and 8 times as many pending async threads). That gets it
to 20,253KiB/s here with ~1000 as many context switches.

So in short, it's harder to compare with real storage, as access
patterns don't translate very easily

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> Engine                Depth      Batch	Bw (KiB/sec)
> libaio		20000	   8		21,125
> syslet		20000	   8		19,610

i'd like to do something more about this to be more in line with libaio 
- if nothing else then for the bragging rights ;-) It seems to me that a 
drop of ~7% in throughput cannot be explained with any CPU overhead, it 
must be some sort of queueing + IO scheduling effect - right?

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Suparna Bhattacharya wrote:
> > It's not bad for such a high depth/batch setting, but I still wonder why
> > are results are so different. I'll look around for an x86 box with some
> > TCQ/NCQ enabled storage attached for testing. Can you pass me your
> > command line or job file (whatever you use) so we are on the same page?
> 
> Sure - I used variations of the following job file (e.g. engine=syslet-rw,
> iodepth=20000).
> 
> Also the io scheduler on my system is set to Anticipatory by default.
> FWIW it is a 4 way SMP (PIII, 700MHz)
> 
> ; aio-stress -l -O -o3 <1GB file>
> [global]
> ioengine=libaio
> buffered=0
> rw=randread
> bs=64k
> size=1024m
> directory=/kdump/suparna
> 
> [testfile2]
> iodepth=64
> iodepth_batch=8

Ok, now that I can run this on more than x86, I gave it a spin on a box
with a little more potent storage. This is a core 2 quad, disks are
7200rpm sata (with NCQ) and a 15krpm SCSI disk. IO scheduler is
deadline.

SATA disk:

Engine          Depth      Batch	Bw (KiB/sec)
----------------------------------------------------
libaio		64	   8		17,486
syslet		64	   8		17,357
libaio		20000	   8		17,625
syslet		20000	   8		16,526
sync            1          1             7,529


SCSI disk:

Engine          Depth      Batch	Bw (KiB/sec)
----------------------------------------------------
libaio		64	   8		20,723
syslet		64	   8		20,742
libaio		20000	   8		21,125
syslet		20000	   8		19,610
sync            1          1            16,659


> > > Engine          Depth      Batch	Bw (KiB/sec)
> > > ----------------------------------------------------
> > > libaio		64	   default	17,429
> > > syslet		64	   default	16,155
> > > libaio		20000	   default	15,494
> > > syslet		20000	   default	7,971
> > >
> > If iodepth_batch isn't set, the syslet queued io will be serialized and
> 
> I see, so then this particular setting is not very meaningful

Not if you want to take advantage of hw queuing, as in this random
workload. fio being a test tool, it's important to be able to control as
many aspects of what happens as possible. That means you can also do
things that you do not want to do in real life, having a pending list of
20000 serialized requests is indeed one of them. It also means you
pretty much have to know what you are doing, when testing little details
like this.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Evgeniy Polyakov wrote:
> On Tue, Feb 27, 2007 at 07:45:41PM +0100, Jens Axboe (jens.axboe@oracle.com) wrote:
> > > Deadline shows this:
> > > 
> > > sync:
> > > READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s,
> > > maxb=38,212KiB/s, mint=28099msec, maxt=28099msec
> > > 
> > > libaio:
> > > READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s,
> > > maxb=37,933KiB/s, mint=28306msec, maxt=28306msec
> > > 
> > > syslet-rw:
> > > READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s,
> > > maxb=34,759KiB/s, mint=30891msec, maxt=30891msec
> > > 
> > > There were about 10k async schedulings.
> > 
> > I think the issue here is pretty simple - when fio gets a queue full
> > like condition (it reaches the depth you set, 32), it commits them and
> > starts queuing again. Since that'll likely block, it'll get issued by
> > another process. So you suddenly have a nice sequence of reads from one
> > process (pending, only one is actually committed since it's serialized),
> > and then a read further down the line that goes behind those you already
> > committed. Then result is seeky, where it should have been sequential.
> > 
> > Do you get expected results if you set iodepth_low=1? That'll make fio
> > drain the queue before building it up again, should get you a sequential
> > access pattern with syslets.
> 
> With such a change results should be better - not only because seek is
> removed with sequential read, but also number of working threads
> decreases with time - until queue is filled again.

Yep, although it probably doesn't matter for such a low bandwidth test
anyway.

> So, syslet-rw has increased to 37mb/sec out of 39/sync and 38/libaio,
> the latter two did not changed.

I wonder why all three aren't doing 39mb/sec flat here, it's a pretty
trivial case...

> With iodepth of 10k, I get the same performance for
> libaio and syslets - about 36mb/sec, it does not depend on iodepth_low
> being set to 1 or default (full).

Yep, the larger the iodepth, the less costly a seek on new queue buildup
gets. So that is as expected.

> So syslets have small problems with small number of iodepth - its
> performance is about 34mb/sec and then increases to 36 with iodepth
> grow. While libaio decreases from 38 down to 36 mb/sec.

Using your job file and fio HEAD (forces iodepth_low=1 for syslet if
iodepth_low isn't specified), I get:

Engine          Depth   Bw (kb/sec)
-----------------------------------
syslet              1      37163
syslet             32      37197
syslet          10000      36577
libaio              1      37144
libaio             32      37159
libaio          10000      36463
sync                1      37154

Results are highly stable. Note that this test case isn't totally fair,
since libaio isn't really async when you do buffered file IO.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Evgeniy Polyakov]

On Tue, Feb 27, 2007 at 07:45:41PM +0100, Jens Axboe (jens.axboe@oracle.com) wrote:
> > Deadline shows this:
> > 
> > sync:
> > READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s,
> > maxb=38,212KiB/s, mint=28099msec, maxt=28099msec
> > 
> > libaio:
> > READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s,
> > maxb=37,933KiB/s, mint=28306msec, maxt=28306msec
> > 
> > syslet-rw:
> > READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s,
> > maxb=34,759KiB/s, mint=30891msec, maxt=30891msec
> > 
> > There were about 10k async schedulings.
> 
> I think the issue here is pretty simple - when fio gets a queue full
> like condition (it reaches the depth you set, 32), it commits them and
> starts queuing again. Since that'll likely block, it'll get issued by
> another process. So you suddenly have a nice sequence of reads from one
> process (pending, only one is actually committed since it's serialized),
> and then a read further down the line that goes behind those you already
> committed. Then result is seeky, where it should have been sequential.
> 
> Do you get expected results if you set iodepth_low=1? That'll make fio
> drain the queue before building it up again, should get you a sequential
> access pattern with syslets.

With such a change results should be better - not only because seek is
removed with sequential read, but also number of working threads
decreases with time - until queue is filled again.

So, syslet-rw has increased to 37mb/sec out of 39/sync and 38/libaio,
the latter two did not changed.

With iodepth of 10k, I get the same performance for
libaio and syslets - about 36mb/sec, it does not depend on iodepth_low
being set to 1 or default (full).

So syslets have small problems with small number of iodepth - its
performance is about 34mb/sec and then increases to 36 with iodepth
grow. While libaio decreases from 38 down to 36 mb/sec.

iodepth_low=1 helps syslets to have 37mb/sec with iodepth=32, with 3200
and 10k it does not play any role.

> -- 
> Jens Axboe

-- 
	Evgeniy Polyakov

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Avi Kivity wrote:
> Ingo Molnar wrote:
> >* Avi Kivity <avi@argo.co.il> wrote:
> >
> >  
> >>But what about cpu usage?  At these low levels, the cpu is probably 
> >>underutilized.  It would be interesting to measure cpu time per I/O 
> >>request (or, alternatively, use an I/O subsystem that can saturate the 
> >>processors).
> >>    
> >
> >yeah - that's what testing on ramdisk (Jens') or on a loopback block 
> >device (mine) approximates to a certain degree.
> >
> >  
> 
> Ramdisks or fully cached loopback return immediately, so cache thrashing 
> effects don't show up.
> 
> Maybe a device mapper delay target or nbd + O_DIRECT can insert delays 
> to make the workload more disk-like.

Take a look at scsi-debug, it can do at least some of that.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Evgeniy Polyakov wrote:
> On Tue, Feb 27, 2007 at 12:29:08PM +0100, Jens Axboe (jens.axboe@oracle.com) wrote:
> > On Tue, Feb 27 2007, Evgeniy Polyakov wrote:
> > > My two coins:
> > > # cat job
> > > [global]
> > > bs=8k
> > > size=1g
> > > direct=0
> > > ioengine=sync
> > > iodepth=32
> > > rw=read
> > > 
> > > [file]
> > > filename=/home/user/test
> > > 
> > > sync: 
> > > READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s,
> > > maxb=39,329KiB/s, mint=27301msec, maxt=27301msec
> > > 
> > > libaio:
> > > READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s,
> > > maxb=39,435KiB/s, mint=27228msec, maxt=27228msec
> > > 
> > > syslet-rw:
> > > READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s,
> > > maxb=29,567KiB/s, mint=36315msec, maxt=36315msec
> > > 
> > > During syslet-rw test about 9500 async schduledes happend.
> > > I use fio-git-20070226150114.tar.gz
> > 
> > That looks pretty pathetic :-). What IO scheduler did you use? syslets
> > will confuse CFQ currently, so you want to compare with using eg
> > deadline or as. That is one of the downsides of this approach.
> 
> Deadline shows this:
> 
> sync:
> READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s,
> maxb=38,212KiB/s, mint=28099msec, maxt=28099msec
> 
> libaio:
> READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s,
> maxb=37,933KiB/s, mint=28306msec, maxt=28306msec
> 
> syslet-rw:
> READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s,
> maxb=34,759KiB/s, mint=30891msec, maxt=30891msec
> 
> There were about 10k async schedulings.

I think the issue here is pretty simple - when fio gets a queue full
like condition (it reaches the depth you set, 32), it commits them and
starts queuing again. Since that'll likely block, it'll get issued by
another process. So you suddenly have a nice sequence of reads from one
process (pending, only one is actually committed since it's serialized),
and then a read further down the line that goes behind those you already
committed. Then result is seeky, where it should have been sequential.

Do you get expected results if you set iodepth_low=1? That'll make fio
drain the queue before building it up again, should get you a sequential
access pattern with syslets.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Avi Kivity]

Ingo Molnar wrote:
>   
>> Maybe a device mapper delay target or nbd + O_DIRECT can insert delays 
>> to make the workload more disk-like.
>>     
>
> yeah. I'll hack a small timeout into loopback requests i think. But then 
> real disk-platter effects are left out ... so it all comes down to 
> eventually having to try it on real disks too :)
>   

Having a random component in the timeout may increase realism.

hundred-disk boxes are noisy, though the blinkenlights are nice :)


-- 
error compiling committee.c: too many arguments to function

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Avi Kivity <avi@argo.co.il> wrote:

> > yeah - that's what testing on ramdisk (Jens') or on a loopback block 
> > device (mine) approximates to a certain degree.
> 
> Ramdisks or fully cached loopback return immediately, so cache 
> thrashing effects don't show up.

even fully cached loopback schedules the loopback kernel thread - but i 
agree that it's inaccurate: hence the 'approximates to a certain 
degree'.

> Maybe a device mapper delay target or nbd + O_DIRECT can insert delays 
> to make the workload more disk-like.

yeah. I'll hack a small timeout into loopback requests i think. But then 
real disk-platter effects are left out ... so it all comes down to 
eventually having to try it on real disks too :)

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Avi Kivity]

Ingo Molnar wrote:
> * Avi Kivity <avi@argo.co.il> wrote:
>
>   
>> But what about cpu usage?  At these low levels, the cpu is probably 
>> underutilized.  It would be interesting to measure cpu time per I/O 
>> request (or, alternatively, use an I/O subsystem that can saturate the 
>> processors).
>>     
>
> yeah - that's what testing on ramdisk (Jens') or on a loopback block 
> device (mine) approximates to a certain degree.
>
>   

Ramdisks or fully cached loopback return immediately, so cache thrashing 
effects don't show up.

Maybe a device mapper delay target or nbd + O_DIRECT can insert delays 
to make the workload more disk-like.




-- 
error compiling committee.c: too many arguments to function

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Avi Kivity <avi@argo.co.il> wrote:

> But what about cpu usage?  At these low levels, the cpu is probably 
> underutilized.  It would be interesting to measure cpu time per I/O 
> request (or, alternatively, use an I/O subsystem that can saturate the 
> processors).

yeah - that's what testing on ramdisk (Jens') or on a loopback block 
device (mine) approximates to a certain degree.

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Avi Kivity]

Suparna Bhattacharya wrote:
> I tried the latest fio code with syslet v4, and my results are a little
> different - have yet to figure out why or what to make of it.
> I hope I have all the right pieces now.
>
> This is an ext2 filesystem, SCSI AIC7xxx.
>
> I used an iodepth_batch size of 8 to limit the number of ios in a single
> io_submit (thanks for adding that parameter to fio !), like we did in
> aio-stress.
>
> Engine          Depth      Batch	Bw (KiB/sec)
> ----------------------------------------------------
> libaio		64	   8		17,226
> syslet		64	   8		17,620
> libaio		20000	   8		18,552
> syslet		20000	   8		14,935
>
>
> Which is not bad, actually.
>
> If I do not specify the iodepth_batch (i.e. default to depth), then the
> difference becomes more pronounced at higher depths. However, I doubt
> whether anyone would be using such high batch sizes in practice ...
>
> Engine          Depth      Batch	Bw (KiB/sec)
> ----------------------------------------------------
> libaio		64	   default	17,429
> syslet		64	   default	16,155
> libaio		20000	   default	15,494
> syslet		20000	   default	7,971
>   

But what about cpu usage?  At these low levels, the cpu is probably 
underutilized.  It would be interesting to measure cpu time per I/O 
request (or, alternatively, use an I/O subsystem that can saturate the 
processors).


-- 
error compiling committee.c: too many arguments to function

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Tue, Feb 27, 2007 at 10:42:11AM +0100, Jens Axboe wrote:
> On Tue, Feb 27 2007, Suparna Bhattacharya wrote:
> > On Mon, Feb 26, 2007 at 03:45:48PM +0100, Jens Axboe wrote:
> > > On Mon, Feb 26 2007, Suparna Bhattacharya wrote:
> > > > On Mon, Feb 26, 2007 at 02:57:36PM +0100, Jens Axboe wrote:
> > > > > 
> > > > > Some more results, using a larger number of processes and io depths. A
> > > > > repeat of the tests from friday, with added depth 20000 for syslet and
> > > > > libaio:
> > > > > 
> > > > > Engine          Depth   Processes       Bw (MiB/sec)
> > > > > ----------------------------------------------------
> > > > > libaio            1         1            602
> > > > > syslet            1         1            759
> > > > > sync              1         1            776
> > > > > libaio           32         1            832
> > > > > syslet           32         1            898
> > > > > libaio        20000         1            581
> > > > > syslet        20000         1            609
> > > > > 
> > > > > syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
> > > > > with 100 processes each with a depth of 200, reading a per-process
> > > > > private file of 10mb (need to fit in my ram...) 10 times each. IOW,
> > > > > doing 10,000MiB of IO in total:
> > > > 
> > > > But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually
> > > > blocks ? Or am I missing something here ?
> > > 
> > > Just overhead numbers for that test case, lets try something like your
> > > described job.
> > > 
> > > Test case is doing random reads from /dev/sdb, in chunks of 64kb:
> > > 
> > > Engine          Depth   Processes       Bw (KiB/sec)
> > > ----------------------------------------------------
> > > libaio           200       100            2813
> > > syslet           200       100            3944
> > > libaio         20000         1            2793
> > > syslet         20000         1            3854
> > > sync (*)       20000         1            2866
> > > 
> > > deadline was used for IO scheduling, to minimize impact. Not sure why
> > > syslet actually does so much better here, looing at vmstat the rate is
> > > steady and all runs are basically 50/50 idle/wait. One difference is
> > > that the submission itself takes a long time on libaio, since the
> > > io_submit() will block on request allocation.  The generated IO pattern
> > > from each process is the same for all runs. The drive is a lousy sata
> > > that doesn't even do queuing, FWIW.
> > 
> > 
> > I tried the latest fio code with syslet v4, and my results are a little
> > different - have yet to figure out why or what to make of it.
> > I hope I have all the right pieces now.
> > 
> > This is an ext2 filesystem, SCSI AIC7xxx.
> > 
> > I used an iodepth_batch size of 8 to limit the number of ios in a single
> > io_submit (thanks for adding that parameter to fio !), like we did in
> > aio-stress.
> > 
> > Engine          Depth      Batch	Bw (KiB/sec)
> > ----------------------------------------------------
> > libaio		64	   8		17,226
> > syslet		64	   8		17,620
> > libaio		20000	   8		18,552
> > syslet		20000	   8		14,935
> > 
> > 
> > Which is not bad, actually.
> 
> It's not bad for such a high depth/batch setting, but I still wonder why
> are results are so different. I'll look around for an x86 box with some
> TCQ/NCQ enabled storage attached for testing. Can you pass me your
> command line or job file (whatever you use) so we are on the same page?

Sure - I used variations of the following job file (e.g. engine=syslet-rw,
iodepth=20000).

Also the io scheduler on my system is set to Anticipatory by default.
FWIW it is a 4 way SMP (PIII, 700MHz)

; aio-stress -l -O -o3 <1GB file>
[global]
ioengine=libaio
buffered=0
rw=randread
bs=64k
size=1024m
directory=/kdump/suparna

[testfile2]
iodepth=64
iodepth_batch=8

> 
> > If I do not specify the iodepth_batch (i.e. default to depth), then the
> > difference becomes more pronounced at higher depths. However, I doubt
> > whether anyone would be using such high batch sizes in practice ...
> >
> > Engine          Depth      Batch	Bw (KiB/sec)
> > ----------------------------------------------------
> > libaio		64	   default	17,429
> > syslet		64	   default	16,155
> > libaio		20000	   default	15,494
> > syslet		20000	   default	7,971
> >
> If iodepth_batch isn't set, the syslet queued io will be serialized and

I see, so then this particular setting is not very meaningful

> not take advantage of queueing. How does the job file perform with
> ioengine=sync?

Just tried it now : 9,027KiB/s

> 
> > Often times it is the application tuning that makes all the difference,
> > so am not really sure how much to read into these results.
> > That's always been the hard part of async io ...
> 
> Yes I agree, it's handy to get an overview though.

True, at least some of this helps us gain a little more understanding
about the boundaries and how to tune it to be most effective.

Regards
Suparna

> 
> -- 
> Jens Axboe

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Evgeniy Polyakov]

On Tue, Feb 27, 2007 at 12:29:08PM +0100, Jens Axboe (jens.axboe@oracle.com) wrote:
> On Tue, Feb 27 2007, Evgeniy Polyakov wrote:
> > My two coins:
> > # cat job
> > [global]
> > bs=8k
> > size=1g
> > direct=0
> > ioengine=sync
> > iodepth=32
> > rw=read
> > 
> > [file]
> > filename=/home/user/test
> > 
> > sync: 
> > READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s,
> > maxb=39,329KiB/s, mint=27301msec, maxt=27301msec
> > 
> > libaio:
> > READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s,
> > maxb=39,435KiB/s, mint=27228msec, maxt=27228msec
> > 
> > syslet-rw:
> > READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s,
> > maxb=29,567KiB/s, mint=36315msec, maxt=36315msec
> > 
> > During syslet-rw test about 9500 async schduledes happend.
> > I use fio-git-20070226150114.tar.gz
> 
> That looks pretty pathetic :-). What IO scheduler did you use? syslets
> will confuse CFQ currently, so you want to compare with using eg
> deadline or as. That is one of the downsides of this approach.

Deadline shows this:

sync:
READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s,
maxb=38,212KiB/s, mint=28099msec, maxt=28099msec

libaio:
READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s,
maxb=37,933KiB/s, mint=28306msec, maxt=28306msec

syslet-rw:
READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s,
maxb=34,759KiB/s, mint=30891msec, maxt=30891msec

There were about 10k async schedulings.

-- 
	Evgeniy Polyakov

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Evgeniy Polyakov wrote:
> My two coins:
> # cat job
> [global]
> bs=8k
> size=1g
> direct=0
> ioengine=sync
> iodepth=32
> rw=read
> 
> [file]
> filename=/home/user/test
> 
> sync: 
> READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s,
> maxb=39,329KiB/s, mint=27301msec, maxt=27301msec
> 
> libaio:
> READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s,
> maxb=39,435KiB/s, mint=27228msec, maxt=27228msec
> 
> syslet-rw:
> READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s,
> maxb=29,567KiB/s, mint=36315msec, maxt=36315msec
> 
> During syslet-rw test about 9500 async schduledes happend.
> I use fio-git-20070226150114.tar.gz

That looks pretty pathetic :-). What IO scheduler did you use? syslets
will confuse CFQ currently, so you want to compare with using eg
deadline or as. That is one of the downsides of this approach.

I'll try your test as soon as this bisect series is done.

> P.S. Jens, fio_latest.tar.gz has wrong permissions, it can not be
> opened.

Oh thanks, indeed. It was disabled symlinks that broke it. Fixed now.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Evgeniy Polyakov]

My two coins:
# cat job
[global]
bs=8k
size=1g
direct=0
ioengine=sync
iodepth=32
rw=read

[file]
filename=/home/user/test

sync: 
READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s,
maxb=39,329KiB/s, mint=27301msec, maxt=27301msec

libaio:
READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s,
maxb=39,435KiB/s, mint=27228msec, maxt=27228msec

syslet-rw:
READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s,
maxb=29,567KiB/s, mint=36315msec, maxt=36315msec

During syslet-rw test about 9500 async schduledes happend.
I use fio-git-20070226150114.tar.gz

P.S. Jens, fio_latest.tar.gz has wrong permissions, it can not be
opened.

-- 
	Evgeniy Polyakov

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Tue, Feb 27 2007, Suparna Bhattacharya wrote:
> On Mon, Feb 26, 2007 at 03:45:48PM +0100, Jens Axboe wrote:
> > On Mon, Feb 26 2007, Suparna Bhattacharya wrote:
> > > On Mon, Feb 26, 2007 at 02:57:36PM +0100, Jens Axboe wrote:
> > > > 
> > > > Some more results, using a larger number of processes and io depths. A
> > > > repeat of the tests from friday, with added depth 20000 for syslet and
> > > > libaio:
> > > > 
> > > > Engine          Depth   Processes       Bw (MiB/sec)
> > > > ----------------------------------------------------
> > > > libaio            1         1            602
> > > > syslet            1         1            759
> > > > sync              1         1            776
> > > > libaio           32         1            832
> > > > syslet           32         1            898
> > > > libaio        20000         1            581
> > > > syslet        20000         1            609
> > > > 
> > > > syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
> > > > with 100 processes each with a depth of 200, reading a per-process
> > > > private file of 10mb (need to fit in my ram...) 10 times each. IOW,
> > > > doing 10,000MiB of IO in total:
> > > 
> > > But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually
> > > blocks ? Or am I missing something here ?
> > 
> > Just overhead numbers for that test case, lets try something like your
> > described job.
> > 
> > Test case is doing random reads from /dev/sdb, in chunks of 64kb:
> > 
> > Engine          Depth   Processes       Bw (KiB/sec)
> > ----------------------------------------------------
> > libaio           200       100            2813
> > syslet           200       100            3944
> > libaio         20000         1            2793
> > syslet         20000         1            3854
> > sync (*)       20000         1            2866
> > 
> > deadline was used for IO scheduling, to minimize impact. Not sure why
> > syslet actually does so much better here, looing at vmstat the rate is
> > steady and all runs are basically 50/50 idle/wait. One difference is
> > that the submission itself takes a long time on libaio, since the
> > io_submit() will block on request allocation.  The generated IO pattern
> > from each process is the same for all runs. The drive is a lousy sata
> > that doesn't even do queuing, FWIW.
> 
> 
> I tried the latest fio code with syslet v4, and my results are a little
> different - have yet to figure out why or what to make of it.
> I hope I have all the right pieces now.
> 
> This is an ext2 filesystem, SCSI AIC7xxx.
> 
> I used an iodepth_batch size of 8 to limit the number of ios in a single
> io_submit (thanks for adding that parameter to fio !), like we did in
> aio-stress.
> 
> Engine          Depth      Batch	Bw (KiB/sec)
> ----------------------------------------------------
> libaio		64	   8		17,226
> syslet		64	   8		17,620
> libaio		20000	   8		18,552
> syslet		20000	   8		14,935
> 
> 
> Which is not bad, actually.

It's not bad for such a high depth/batch setting, but I still wonder why
are results are so different. I'll look around for an x86 box with some
TCQ/NCQ enabled storage attached for testing. Can you pass me your
command line or job file (whatever you use) so we are on the same page?

> If I do not specify the iodepth_batch (i.e. default to depth), then the
> difference becomes more pronounced at higher depths. However, I doubt
> whether anyone would be using such high batch sizes in practice ...
>
> Engine          Depth      Batch	Bw (KiB/sec)
> ----------------------------------------------------
> libaio		64	   default	17,429
> syslet		64	   default	16,155
> libaio		20000	   default	15,494
> syslet		20000	   default	7,971
>
If iodepth_batch isn't set, the syslet queued io will be serialized and
not take advantage of queueing. How does the job file perform with
ioengine=sync?

> Often times it is the application tuning that makes all the difference,
> so am not really sure how much to read into these results.
> That's always been the hard part of async io ...

Yes I agree, it's handy to get an overview though.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Mon, Feb 26, 2007 at 03:45:48PM +0100, Jens Axboe wrote:
> On Mon, Feb 26 2007, Suparna Bhattacharya wrote:
> > On Mon, Feb 26, 2007 at 02:57:36PM +0100, Jens Axboe wrote:
> > > 
> > > Some more results, using a larger number of processes and io depths. A
> > > repeat of the tests from friday, with added depth 20000 for syslet and
> > > libaio:
> > > 
> > > Engine          Depth   Processes       Bw (MiB/sec)
> > > ----------------------------------------------------
> > > libaio            1         1            602
> > > syslet            1         1            759
> > > sync              1         1            776
> > > libaio           32         1            832
> > > syslet           32         1            898
> > > libaio        20000         1            581
> > > syslet        20000         1            609
> > > 
> > > syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
> > > with 100 processes each with a depth of 200, reading a per-process
> > > private file of 10mb (need to fit in my ram...) 10 times each. IOW,
> > > doing 10,000MiB of IO in total:
> > 
> > But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually
> > blocks ? Or am I missing something here ?
> 
> Just overhead numbers for that test case, lets try something like your
> described job.
> 
> Test case is doing random reads from /dev/sdb, in chunks of 64kb:
> 
> Engine          Depth   Processes       Bw (KiB/sec)
> ----------------------------------------------------
> libaio           200       100            2813
> syslet           200       100            3944
> libaio         20000         1            2793
> syslet         20000         1            3854
> sync (*)       20000         1            2866
> 
> deadline was used for IO scheduling, to minimize impact. Not sure why
> syslet actually does so much better here, looing at vmstat the rate is
> steady and all runs are basically 50/50 idle/wait. One difference is
> that the submission itself takes a long time on libaio, since the
> io_submit() will block on request allocation.  The generated IO pattern
> from each process is the same for all runs. The drive is a lousy sata
> that doesn't even do queuing, FWIW.


I tried the latest fio code with syslet v4, and my results are a little
different - have yet to figure out why or what to make of it.
I hope I have all the right pieces now.

This is an ext2 filesystem, SCSI AIC7xxx.

I used an iodepth_batch size of 8 to limit the number of ios in a single
io_submit (thanks for adding that parameter to fio !), like we did in
aio-stress.

Engine          Depth      Batch	Bw (KiB/sec)
----------------------------------------------------
libaio		64	   8		17,226
syslet		64	   8		17,620
libaio		20000	   8		18,552
syslet		20000	   8		14,935


Which is not bad, actually.

If I do not specify the iodepth_batch (i.e. default to depth), then the
difference becomes more pronounced at higher depths. However, I doubt
whether anyone would be using such high batch sizes in practice ...

Engine          Depth      Batch	Bw (KiB/sec)
----------------------------------------------------
libaio		64	   default	17,429
syslet		64	   default	16,155
libaio		20000	   default	15,494
syslet		20000	   default	7,971

Often times it is the application tuning that makes all the difference,
so am not really sure how much to read into these results.
That's always been the hard part of async io ...

Regards
Suparna

> 
> [*] Just for comparison, the depth is obviously really 1 at the kernel
>     side since it's sync.
> 
> -- 
> Jens Axboe

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Davide Libenzi]

On Mon, 26 Feb 2007, Jens Axboe wrote:

> 
> Some more results, using a larger number of processes and io depths. A
> repeat of the tests from friday, with added depth 20000 for syslet and
> libaio:
> 
> Engine          Depth   Processes       Bw (MiB/sec)
> ----------------------------------------------------
> libaio            1         1            602
> syslet            1         1            759
> sync              1         1            776
> libaio           32         1            832
> syslet           32         1            898
> libaio        20000         1            581
> syslet        20000         1            609

That looks great! IMO there may be a little higher cost associated with 
the syslets thread switches, that we currently do not perform 100% 
cleanly, but results look nevertheless awesome.



- Davide

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Mon, Feb 26 2007, Suparna Bhattacharya wrote:
> On Mon, Feb 26, 2007 at 02:57:36PM +0100, Jens Axboe wrote:
> > 
> > Some more results, using a larger number of processes and io depths. A
> > repeat of the tests from friday, with added depth 20000 for syslet and
> > libaio:
> > 
> > Engine          Depth   Processes       Bw (MiB/sec)
> > ----------------------------------------------------
> > libaio            1         1            602
> > syslet            1         1            759
> > sync              1         1            776
> > libaio           32         1            832
> > syslet           32         1            898
> > libaio        20000         1            581
> > syslet        20000         1            609
> > 
> > syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
> > with 100 processes each with a depth of 200, reading a per-process
> > private file of 10mb (need to fit in my ram...) 10 times each. IOW,
> > doing 10,000MiB of IO in total:
> 
> But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually
> blocks ? Or am I missing something here ?

Just overhead numbers for that test case, lets try something like your
described job.

Test case is doing random reads from /dev/sdb, in chunks of 64kb:

Engine          Depth   Processes       Bw (KiB/sec)
----------------------------------------------------
libaio           200       100            2813
syslet           200       100            3944
libaio         20000         1            2793
syslet         20000         1            3854
sync (*)       20000         1            2866

deadline was used for IO scheduling, to minimize impact. Not sure why
syslet actually does so much better here, looing at vmstat the rate is
steady and all runs are basically 50/50 idle/wait. One difference is
that the submission itself takes a long time on libaio, since the
io_submit() will block on request allocation.  The generated IO pattern
from each process is the same for all runs. The drive is a lousy sata
that doesn't even do queuing, FWIW.

[*] Just for comparison, the depth is obviously really 1 at the kernel
    side since it's sync.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Suparna Bhattacharya <suparna@in.ibm.com> wrote:

> > syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs 
> > with 100 processes each with a depth of 200, reading a per-process 
> > private file of 10mb (need to fit in my ram...) 10 times each. IOW, 
> > doing 10,000MiB of IO in total:
> 
> But, why ramfs ? Don't we want to exercise the case where O_DIRECT 
> actually blocks ? Or am I missing something here ?

ramfs is just the easiest way to measure the pure CPU overhead of a 
workload without real IO delays (and resulting idle time) getting in the 
way. It's certainly not the same thing, but still it's pretty useful 
most of the time. I used a different method, loopback block device, and 
got similar results. [ Real IO shows similar results as well, but is a 
lot more noisy and hence harder to interpret (and thus easier to get 
wrong). ]

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Mon, Feb 26, 2007 at 02:57:36PM +0100, Jens Axboe wrote:
> 
> Some more results, using a larger number of processes and io depths. A
> repeat of the tests from friday, with added depth 20000 for syslet and
> libaio:
> 
> Engine          Depth   Processes       Bw (MiB/sec)
> ----------------------------------------------------
> libaio            1         1            602
> syslet            1         1            759
> sync              1         1            776
> libaio           32         1            832
> syslet           32         1            898
> libaio        20000         1            581
> syslet        20000         1            609
> 
> syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
> with 100 processes each with a depth of 200, reading a per-process
> private file of 10mb (need to fit in my ram...) 10 times each. IOW,
> doing 10,000MiB of IO in total:

But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually
blocks ? Or am I missing something here ?

Regards
Suparna

> 
> Engine          Depth   Processes       Bw (MiB/sec)
> ----------------------------------------------------
> libaio           200       100            1488
> syslet           200       100            1714
> 
> Results are stable to within approx +/- 10MiB/sec. The syslet case
> completes a whole second faster than libaio (~6 vs ~7 seconds). Testing
> was done with fio HEAD eb7c8ae27bc301b77490b3586dd5ccab7c95880a, and it
> uses the v4 patch series.
> 
> Engine          Depth   Processes       Bw (MiB/sec)
> ----------------------------------------------------
> libaio            200       100            1488
> syslet            200       100            1714
> sync              200       100            1843
> 
> -- 
> Jens Axboe

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

Some more results, using a larger number of processes and io depths. A
repeat of the tests from friday, with added depth 20000 for syslet and
libaio:

Engine          Depth   Processes       Bw (MiB/sec)
----------------------------------------------------
libaio            1         1            602
syslet            1         1            759
sync              1         1            776
libaio           32         1            832
syslet           32         1            898
libaio        20000         1            581
syslet        20000         1            609

syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs
with 100 processes each with a depth of 200, reading a per-process
private file of 10mb (need to fit in my ram...) 10 times each. IOW,
doing 10,000MiB of IO in total:

Engine          Depth   Processes       Bw (MiB/sec)
----------------------------------------------------
libaio           200       100            1488
syslet           200       100            1714

Results are stable to within approx +/- 10MiB/sec. The syslet case
completes a whole second faster than libaio (~6 vs ~7 seconds). Testing
was done with fio HEAD eb7c8ae27bc301b77490b3586dd5ccab7c95880a, and it
uses the v4 patch series.

Engine          Depth   Processes       Bw (MiB/sec)
----------------------------------------------------
libaio            200       100            1488
syslet            200       100            1714
sync              200       100            1843

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Fri, Feb 23 2007, Joel Becker wrote:
> On Fri, Feb 23, 2007 at 01:52:47PM +0100, Jens Axboe wrote:
> > Results:
> > 
> > Engine          Depth           Bw (MiB/sec)
> > --------------------------------------------
> > libaio            1             441
> > syslet            1             574
> > sync              1             589
> > libaio           32             613
> > syslet           32             681
> 
> 	Can we get runs with large I/Os, large I/O depths, and most
> importantly tons of processes?  I can absolutely believe that syslets
> would compete well with one process on the system.  But with 1000
> processes doing 1000s of blocking I/Os, I'd really be interested to see
> how that plays out.

Sure, I'll add this to the testing list for monday.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Joel Becker]

On Fri, Feb 23, 2007 at 01:52:47PM +0100, Jens Axboe wrote:
> Results:
> 
> Engine          Depth           Bw (MiB/sec)
> --------------------------------------------
> libaio            1             441
> syslet            1             574
> sync              1             589
> libaio           32             613
> syslet           32             681

	Can we get runs with large I/Os, large I/O depths, and most
importantly tons of processes?  I can absolutely believe that syslets
would compete well with one process on the system.  But with 1000
processes doing 1000s of blocking I/Os, I'd really be interested to see
how that plays out.

Joel

-- 

 Joel's Second Law:

	If a code change requires additional user setup, it is wrong.

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Fri, Feb 23 2007, Suparna Bhattacharya wrote:
> On Fri, Feb 23, 2007 at 05:25:08PM +0100, Jens Axboe wrote:
> > On Fri, Feb 23 2007, Suparna Bhattacharya wrote:
> > > On Fri, Feb 23, 2007 at 03:58:26PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Suparna Bhattacharya <suparna@in.ibm.com> wrote:
> > > > 
> > > > > As a really crude (and not very realistic) example of the potential 
> > > > > impact of large numbers of outstanding IOs, I tried some quick direct 
> > > > > IO comparisons using fio:
> > > > > 
> > > > > [global]
> > > > > ioengine=syslet-rw
> > > > > buffered=0
> > > > > rw=randread
> > > > > bs=64k
> > > > > size=1024m
> > > > > iodepth=64
> > > > 
> > > > could you please try those iodepth=20000 tests with the latest 
> > > > fio-testing branch of fio as well? Jens wrote a new, smarter syslet 
> > > > plugin for FIO. You'll need the v3 syslet kernel plus:
> > > > 
> > > >  git-clone git://git.kernel.dk/data/git/fio.git
> > > >  cd fio
> > > >  git-checkout syslet-testing
> > > > 
> > > > my expectation is that it should behave better with iodepth=20000 
> > > > (although i havent tried that yet).
> > > 
> > > I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz)
> > > and used the v3 syslet patches from your web-site.
> > > 
> > > Do I still need to get something more recent ?
> > 
> > Yes, you need to test the syslet+testing branch that Ingo referenced.
> > Your test above is not totally fair right now, since you are doing
> > significantly less system calls with libaio. So to compare apples with
> > apples, try the syslet-testing branch. If you can't get it because of
> > firewall problems, check http://brick.kernel.dk/snaps/ for the latest
> > fio snapshot. If it has the syslet-testing branch, then that is
> > recent enough.
> 
> I have a feeling this is getting to be a little more bleeding edge than
> I had anticipated :), so will just hold off for a bit until this
> crystallizes a bit.

Fair enough, I'll try your test with a huge number of pending requests
and see how it fares.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Fri, Feb 23, 2007 at 05:25:08PM +0100, Jens Axboe wrote:
> On Fri, Feb 23 2007, Suparna Bhattacharya wrote:
> > On Fri, Feb 23, 2007 at 03:58:26PM +0100, Ingo Molnar wrote:
> > > 
> > > * Suparna Bhattacharya <suparna@in.ibm.com> wrote:
> > > 
> > > > As a really crude (and not very realistic) example of the potential 
> > > > impact of large numbers of outstanding IOs, I tried some quick direct 
> > > > IO comparisons using fio:
> > > > 
> > > > [global]
> > > > ioengine=syslet-rw
> > > > buffered=0
> > > > rw=randread
> > > > bs=64k
> > > > size=1024m
> > > > iodepth=64
> > > 
> > > could you please try those iodepth=20000 tests with the latest 
> > > fio-testing branch of fio as well? Jens wrote a new, smarter syslet 
> > > plugin for FIO. You'll need the v3 syslet kernel plus:
> > > 
> > >  git-clone git://git.kernel.dk/data/git/fio.git
> > >  cd fio
> > >  git-checkout syslet-testing
> > > 
> > > my expectation is that it should behave better with iodepth=20000 
> > > (although i havent tried that yet).
> > 
> > I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz)
> > and used the v3 syslet patches from your web-site.
> > 
> > Do I still need to get something more recent ?
> 
> Yes, you need to test the syslet+testing branch that Ingo referenced.
> Your test above is not totally fair right now, since you are doing
> significantly less system calls with libaio. So to compare apples with
> apples, try the syslet-testing branch. If you can't get it because of
> firewall problems, check http://brick.kernel.dk/snaps/ for the latest
> fio snapshot. If it has the syslet-testing branch, then that is
> recent enough.

I have a feeling this is getting to be a little more bleeding edge than
I had anticipated :), so will just hold off for a bit until this
crystallizes a bit.

Regards
Suparna

> 
> -- 
> Jens Axboe

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Suparna Bhattacharya <suparna@in.ibm.com> wrote:

> > my expectation is that it should behave better with iodepth=20000 
> > (although i havent tried that yet).
> 
> I picked up the fio snapshot from 22nd Feb 
> (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from 
> your web-site.
> 
> Do I still need to get something more recent ?

yeah, there's something more recent. Please do this:

  yum install git
  git-clone git://git.kernel.dk/data/git/fio.git
  cd fio
  git-branch syslet-testing
  git-checkout

this should give you the latest version of the v3 based FIO code. It's 
one generation newer than the one you tried. I mean the snapshot you 
used is meanwhile a /whole/ day old, so it's truly ancient stuff! ;-)

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Fri, Feb 23 2007, Suparna Bhattacharya wrote:
> On Fri, Feb 23, 2007 at 03:58:26PM +0100, Ingo Molnar wrote:
> > 
> > * Suparna Bhattacharya <suparna@in.ibm.com> wrote:
> > 
> > > As a really crude (and not very realistic) example of the potential 
> > > impact of large numbers of outstanding IOs, I tried some quick direct 
> > > IO comparisons using fio:
> > > 
> > > [global]
> > > ioengine=syslet-rw
> > > buffered=0
> > > rw=randread
> > > bs=64k
> > > size=1024m
> > > iodepth=64
> > 
> > could you please try those iodepth=20000 tests with the latest 
> > fio-testing branch of fio as well? Jens wrote a new, smarter syslet 
> > plugin for FIO. You'll need the v3 syslet kernel plus:
> > 
> >  git-clone git://git.kernel.dk/data/git/fio.git
> >  cd fio
> >  git-checkout syslet-testing
> > 
> > my expectation is that it should behave better with iodepth=20000 
> > (although i havent tried that yet).
> 
> I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz)
> and used the v3 syslet patches from your web-site.
> 
> Do I still need to get something more recent ?

Yes, you need to test the syslet+testing branch that Ingo referenced.
Your test above is not totally fair right now, since you are doing
significantly less system calls with libaio. So to compare apples with
apples, try the syslet-testing branch. If you can't get it because of
firewall problems, check http://brick.kernel.dk/snaps/ for the latest
fio snapshot. If it has the syslet-testing branch, then that is
recent enough.

-- 
Jens Axboe

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Fri, Feb 23, 2007 at 03:58:26PM +0100, Ingo Molnar wrote:
> 
> * Suparna Bhattacharya <suparna@in.ibm.com> wrote:
> 
> > As a really crude (and not very realistic) example of the potential 
> > impact of large numbers of outstanding IOs, I tried some quick direct 
> > IO comparisons using fio:
> > 
> > [global]
> > ioengine=syslet-rw
> > buffered=0
> > rw=randread
> > bs=64k
> > size=1024m
> > iodepth=64
> 
> could you please try those iodepth=20000 tests with the latest 
> fio-testing branch of fio as well? Jens wrote a new, smarter syslet 
> plugin for FIO. You'll need the v3 syslet kernel plus:
> 
>  git-clone git://git.kernel.dk/data/git/fio.git
>  cd fio
>  git-checkout syslet-testing
> 
> my expectation is that it should behave better with iodepth=20000 
> (although i havent tried that yet).

I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz)
and used the v3 syslet patches from your web-site.

Do I still need to get something more recent ?

Regards
Suparna


> 
> 	Ingo

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Ingo Molnar]

* Suparna Bhattacharya <suparna@in.ibm.com> wrote:

> As a really crude (and not very realistic) example of the potential 
> impact of large numbers of outstanding IOs, I tried some quick direct 
> IO comparisons using fio:
> 
> [global]
> ioengine=syslet-rw
> buffered=0
> rw=randread
> bs=64k
> size=1024m
> iodepth=64

could you please try those iodepth=20000 tests with the latest 
fio-testing branch of fio as well? Jens wrote a new, smarter syslet 
plugin for FIO. You'll need the v3 syslet kernel plus:

 git-clone git://git.kernel.dk/data/git/fio.git
 cd fio
 git-checkout syslet-testing

my expectation is that it should behave better with iodepth=20000 
(although i havent tried that yet).

	Ingo

Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Suparna Bhattacharya]

On Fri, Feb 23, 2007 at 01:52:47PM +0100, Jens Axboe wrote:
> On Wed, Feb 21 2007, Ingo Molnar wrote:
> > this is the v3 release of the syslet/threadlet subsystem:
> > 
> >    http://redhat.com/~mingo/syslet-patches/
> 
> [snip]
> 
> Ingo, some testing of the experimental syslet queueing stuff, in the
> syslet-testing branch of fio.
> 
> Fio job file:
> 
> [global]
> bs=8k
> size=1g
> direct=0
> ioengine=syslet-rw
> iodepth=32
> rw=read
> 
> [file]
> filename=/ramfs/testfile
> 
> Only changes between runs was changing ioengine and iodepth as indicated
> in the table below.
> 
> Results:
> 
> Engine          Depth           Bw (MiB/sec)
> --------------------------------------------
> libaio            1             441
> syslet            1             574
> sync              1             589
> libaio           32             613
> syslet           32             681
> 
> Results are stable to within +/- 1MiB/sec. So you can see that syslet
> are still a bit slower than sync for depth 1, but beats the pants off
> libaio for equal depths. Note that this is buffered IO, I'll be out for
> the weekend, but I'll hack some direct IO testing up next week to
> compare "real" queuing.
> 
> Just a quick microbenchmark to gauge current overhead...

This is just ramfs, to gauge pure overheads, is that correct ? 

BTW, I'm not surprised at Ingo's initial results of syslet vs libaio
overheads, for aio-stress/fio type streaming io runs, because these cases
do not involve large numbers of outstanding ios. So the overhead of
thread creation with syslets is amortized across the entire run of io
submissions because of the reuse of already created async threads. While
in the libaio case there is a setup and teardown of kiocbs per request.

What I have been concerned about instead in the past when considering
thread based AIO implementations is the resource(memory) consumption impact
on overall system performance and adaptability to varying loads. It is nice
that we can avoid that for the cached cases, but for the general blocking
cases, it is still not clear to me whether we have addressed this well
enough yet. I used to think that even the kiocb was too heavyweight for its
purpose ... especially in terms of scaling to larger loads.

As a really crude (and not very realistic) example of the potential impact
of large numbers of outstanding IOs, I tried some quick direct IO comparisons
using fio:

[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64

Engine		Depth		Bw (MiB/sec)

libaio		64		17.323
syslet		64		17.524
libaio		20000		15.226
syslet		20000		11.015


Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)

[Jens Axboe]

On Wed, Feb 21 2007, Ingo Molnar wrote:
> this is the v3 release of the syslet/threadlet subsystem:
> 
>    http://redhat.com/~mingo/syslet-patches/

[snip]

Ingo, some testing of the experimental syslet queueing stuff, in the
syslet-testing branch of fio.

Fio job file:

[global]
bs=8k
size=1g
direct=0
ioengine=syslet-rw
iodepth=32
rw=read

[file]
filename=/ramfs/testfile

Only changes between runs was changing ioengine and iodepth as indicated
in the table below.

Results:

Engine          Depth           Bw (MiB/sec)
--------------------------------------------
libaio            1             441
syslet            1             574
sync              1             589
libaio           32             613
syslet           32             681

Results are stable to within +/- 1MiB/sec. So you can see that syslet
are still a bit slower than sync for depth 1, but beats the pants off
libaio for equal depths. Note that this is buffered IO, I'll be out for
the weekend, but I'll hack some direct IO testing up next week to
compare "real" queuing.

Just a quick microbenchmark to gauge current overhead...

-- 
Jens Axboe