LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* block layer copying user io vectors
@ 2015-01-22 21:33 Andy Falanga (afalanga)
  2015-01-23 17:06 ` Christoph Hellwig
  0 siblings, 1 reply; 4+ messages in thread
From: Andy Falanga (afalanga) @ 2015-01-22 21:33 UTC (permalink / raw)
  To: linux-kernel

Please CC me directly.

I am working in kernel 2.6.32 (CentOS 6).  To increase the upper limit
of sg from 4mb to at least 128mb in a single SCSI command.  At first I
thought this issue was in sg, but have tracked the issue to the block
layer.

Thinking I could solve this issue by using scatter/gather lists, I
increased the size from 32k to 4mb of each vector.  This did work
until I tried to send 8mb.  When I do so, I get errno EINVAL.  After
some tracing, I tracked the problem into bio_copy_user_iov().

This function does something that seems rather strange.  On line 859,
a for loop determines the number of pages needed for the copying of the
user data to kernel space.  Then the memory is allocated (line
886 bio_kmalloc()).  Then, strangely, on line 895, there is this
conditional:

if (map_data) {
    nr_pages = 1 << map_data->page_order;
    i = map_data->offset / PAGE_SIZE;
}

This effectively ignores the number of pages counted earlier (in this
case which applies to me), and then apparently disregards whatever
memory may have been allocated earlier.  Thinking that this was the
root I tried to correct this by commenting that simple branch from
bio_copy_user_iov, but still had the same result.  Can someone
help me understand what is happening in the block layer?

Thanks

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: block layer copying user io vectors
  2015-01-22 21:33 block layer copying user io vectors Andy Falanga (afalanga)
@ 2015-01-23 17:06 ` Christoph Hellwig
  2015-01-30 17:43   ` Andy Falanga (afalanga)
  0 siblings, 1 reply; 4+ messages in thread
From: Christoph Hellwig @ 2015-01-23 17:06 UTC (permalink / raw)
  To: Andy Falanga (afalanga); +Cc: linux-kernel, linux-scsi, Doug Gilbert

On Thu, Jan 22, 2015 at 09:33:08PM +0000, Andy Falanga (afalanga) wrote:
> Please CC me directly.
> 
> I am working in kernel 2.6.32 (CentOS 6).  To increase the upper limit
> of sg from 4mb to at least 128mb in a single SCSI command.  At first I
> thought this issue was in sg, but have tracked the issue to the block
> layer.

2.6.32 is fairly old, but fortunately for you not too many things should
have changed in this area.

> 
> Thinking I could solve this issue by using scatter/gather lists, I
> increased the size from 32k to 4mb of each vector.  This did work
> until I tried to send 8mb.  When I do so, I get errno EINVAL.  After
> some tracing, I tracked the problem into bio_copy_user_iov().
> 
> This function does something that seems rather strange.  On line 859,
> a for loop determines the number of pages needed for the copying of the
> user data to kernel space.  Then the memory is allocated (line
> 886 bio_kmalloc()).  Then, strangely, on line 895, there is this
> conditional:

This is because the function can also be used with preallocated pages,
a feature only used by the sg and tape drivers.

Make sure your user memory is 4k aligned, and you should be able
to avoid the copy entirely (1).

(1) except that the sg driver disables the direct mapping of user pages
    when using readv/writev.  I can't really see why and it should be
    fixable by just removign that condition from the if in sg_start_req.
    Alternatively use the SG_IO ioctl directly on the disk device node,
    which neither has the read/writev limitation, nor does it use
    a fixed upper bound preallocated page pool.

> 
> if (map_data) {
>     nr_pages = 1 << map_data->page_order;
>     i = map_data->offset / PAGE_SIZE;
> }
> 
> This effectively ignores the number of pages counted earlier (in this
> case which applies to me), and then apparently disregards whatever
> memory may have been allocated earlier.  Thinking that this was the
> root I tried to correct this by commenting that simple branch from
> bio_copy_user_iov, but still had the same result.  Can someone
> help me understand what is happening in the block layer?

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: block layer copying user io vectors
  2015-01-23 17:06 ` Christoph Hellwig
@ 2015-01-30 17:43   ` Andy Falanga (afalanga)
  2015-02-02 10:00     ` Christoph Hellwig
  0 siblings, 1 reply; 4+ messages in thread
From: Andy Falanga (afalanga) @ 2015-01-30 17:43 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: linux-kernel, linux-scsi, Doug Gilbert

> > This function does something that seems rather strange.  On line 859,
> > a for loop determines the number of pages needed for the copying of
> > the user data to kernel space.  Then the memory is allocated (line
> > 886 bio_kmalloc()).  Then, strangely, on line 895, there is this
> > conditional:
> 
> This is because the function can also be used with preallocated pages,
> a feature only used by the sg and tape drivers.
> 
> Make sure your user memory is 4k aligned, and you should be able to
> avoid the copy entirely (1).

Where is this 4k alignment being enforced?  When sg_start_req calls to
blk_rq_map_user_iov, the only check for alignment is that the data buffers
are 4-byte aligned (q->dma_alignment == 3).  I have verified that they are.

> 
> (1) except that the sg driver disables the direct mapping of user pages
>     when using readv/writev.  I can't really see why and it should be
>     fixable by just removign that condition from the if in
> sg_start_req.

I am investigating this now.

>     Alternatively use the SG_IO ioctl directly on the disk device node,
>     which neither has the read/writev limitation, nor does it use
>     a fixed upper bound preallocated page pool.

Unfortunately, the write/read interface of sg is what I need to use for
asynchronous reasons.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: block layer copying user io vectors
  2015-01-30 17:43   ` Andy Falanga (afalanga)
@ 2015-02-02 10:00     ` Christoph Hellwig
  0 siblings, 0 replies; 4+ messages in thread
From: Christoph Hellwig @ 2015-02-02 10:00 UTC (permalink / raw)
  To: Andy Falanga (afalanga); +Cc: linux-kernel, linux-scsi, Doug Gilbert

On Fri, Jan 30, 2015 at 05:43:04PM +0000, Andy Falanga (afalanga) wrote:
> > > This function does something that seems rather strange.  On line 859,
> > > a for loop determines the number of pages needed for the copying of
> > > the user data to kernel space.  Then the memory is allocated (line
> > > 886 bio_kmalloc()).  Then, strangely, on line 895, there is this
> > > conditional:
> > 
> > This is because the function can also be used with preallocated pages,
> > a feature only used by the sg and tape drivers.
> > 
> > Make sure your user memory is 4k aligned, and you should be able to
> > avoid the copy entirely (1).
> 
> Where is this 4k alignment being enforced?  When sg_start_req calls to
> blk_rq_map_user_iov, the only check for alignment is that the data buffers
> are 4-byte aligned (q->dma_alignment == 3).  I have verified that they are.

Indeed, I though we had the more strict direct I/O alignment.

Still doesn't help because the sg driver refuses to directly map user
pages for vectored I/O.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2015-02-02 10:00 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-22 21:33 block layer copying user io vectors Andy Falanga (afalanga)
2015-01-23 17:06 ` Christoph Hellwig
2015-01-30 17:43   ` Andy Falanga (afalanga)
2015-02-02 10:00     ` Christoph Hellwig

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).