Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Shirley Ma]

On Tue, 2008-04-01 at 13:02 -0700, Roland Dreier wrote:
> - Multiple CQ event vector support.  I still haven't seen any
>    discussions about how ULPs or userspace apps should decide which
>    vector to use, and hence no progress has been made since we
>    deferred this during the 2.6.23 merge window. 

I did some prototype for IPoIB to enable multiple CQ event support. I
did see the approach improved multiple links aggregation performance. I
also see some customers' requirements in userspace. I will start the
discussion as soon as possible. But it would most likely miss 2.6.26
window.

Thanks
Shirley

InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

The 2.6.26 will open soon, so it's time to review what my plans are
for the merge window opens.

As usual, patch review by non-me people is always welcome.

Anyway, here are all the pending things that I'm aware of.  As usual,
if something isn't already in my tree and isn't listed below, I
probably missed it or dropped it by mistake.  Please remind me again
in that case.

Core:

 - I did a bunch of cleanups all over drivers/infiniband and the
   gcc and sparse warning noise is down to a pretty reasonable level.
   Further cleanups welcome of course.

ULPs:

 - I merged Eli's IPoIB stateless offload changes for checksum
   offload and LSO changes.  The interrupt moderation changes are
   next, and should not be a problem to merge.  Please test IPoIB
   on all sorts of hardware!

 - Shirley's IPoIB 4 KB MTU changes.  I expect these to make it in,
   although I would certainly appreciate review from Eli or anyone else.

HW specific:

 - Vlad's mlx4 resize CQ support.  Looks basically OK, so I think we
   should be able to get it in.

 - ipath support for 7220 HCAs.  I don't expect any issues here once
   the patches appear.

Here are a few topics that I believe will not be ready in time for the
2.6.26 window and will need to wait for 2.6.27 at least:

 - XRC.  I still don't have a good feeling that we have settled on all
   the nuances of the ABI we want to expose to userspace for this, and
   ideally I would like to understand how ehca LL QPs fit into the
   picture as well.

 - Remove LLTX from IPoIB.  I haven't had time to finish this yet, so
   I guess it will probably wait for 2.6.27 now...

 - Multiple CQ event vector support.  I still haven't seen any
   discussions about how ULPs or userspace apps should decide which
   vector to use, and hence no progress has been made since we
   deferred this during the 2.6.23 merge window.

Here all the patches I already have in my for-2.6.26 branch:

Arthur Jones (4):
      IB/ipath: Fix sparse warning about pointer signedness
      IB/ipath: Misc sparse warning cleanup
      IB/ipath: Provide I/O bus speeds for diagnostic purposes
      IB/ipath: Fix link up LED display

Dave Olson (4):
      IB/ipath: Make some constants chip-specific, related cleanup
      IB/ipath: Shared context code needs to be sure device is usable
      IB/ipath: Enable 4KB MTU
      IB/ipath: HW workaround for case where chip can send but not receive

David Dillow (1):
      IB/srp: Enforce protocol limit on srp_sg_tablesize

Eli Cohen (7):
      IPoIB: Use checksum offload support if available
      IB/mlx4: Add IPoIB checksum offload support
      IB/mthca: Add IPoIB checksum offload support
      IB/core: Add creation flags to struct ib_qp_init_attr
      IB/core: Add IPoIB UD LSO support
      IPoIB: Add LSO support
      IB/mlx4: Add IPoIB LSO support

Harvey Harrison (1):
      IB: Replace remaining __FUNCTION__ occurrences with __func__

Hoang-Nam Nguyen (1):
      IB/ehca: Remove tgid checking

John Gregor (1):
      IB/ipath: Head of Line blocking vs forward progress of user apps

Julia Lawall (1):
      RDMA/iwcm: Test rdma_create_id() for IS_ERR rather than 0

Michael Albaugh (2):
      IB/ipath: Prevent link-recovery code from negating admin disable
      IB/ipath: EEPROM support for 7220 devices, robustness improvements, cleanup

Ralph Campbell (11):
      IB/ipath: Fix byte order of pioavail in handle_errors()
      IB/ipath: Fix error recovery for send buffer status after chip freeze mode
      IB/ipath: Don't try to handle freeze mode HW errors if diagnostic mode
      IB/ipath: Make debug error message match the constraint that is checked for
      IB/ipath: Add code to support multiple link speeds and widths
      IB/ipath: Remove useless comments
      IB/ipath: Fix sanity checks on QP number of WRs and SGEs
      IB/ipath: Change the module author
      IB/ipath: Remove some useless (void) casts
      IB/ipath: Make send buffers available for kernel if not allocated to user
      IB/ipath: Use PIO buffer for RC ACKs

Robert P. J. Day (2):
      IB: Use shorter list_splice_init() for brevity
      RDMA/nes: Use more concise list_for_each_entry()

Roland Dreier (28):
      IB/mthca: Formatting cleanups
      IB/mlx4: Convert "if(foo)" to "if (foo)"
      mlx4_core: Move opening brace of function onto a new line
      RDMA/amso1100: Don't use 0UL as a NULL pointer
      RDMA/cxgb3: IDR IDs are signed
      IB: Make struct ib_uobject.id a signed int
      IB/ipath: Fix sparse warning about shadowed symbol
      IB/mlx4: Endianness annotations
      IB/cm: Endianness annotations
      RDMA/ucma: Endian annotation
      RDMA/nes: Trivial endianness annotations
      RDMA/nes: Delete unused variables
      RDMA/amso1100: Start of endianness annotation
      RDMA/amso1100: Endian annotate mqsq allocator
      mlx4_core: Fix confusion between mlx4_event and mlx4_dev_event enums
      IB/uverbs: Don't store struct file * for event files
      IB/uverbs: Use alloc_file() instead of get_empty_filp()
      RDMA/nes: Remove redundant NULL check in nes_unregister_ofa_device()
      RDMA/nes: Remove unused nes_netdev_exit() function
      RDMA/nes: Use proper format and cast to print dma_addr_t
      RDMA/nes: Make symbols used only in a single source file static
      IB/ehca: Make symbols used only in a single source file static
      IB/core: Add support for "send with invalidate" work requests
      RDMA/amso1100: Add support for "send with invalidate" work requests
      IB/mthca: Avoid integer overflow when dealing with profile size
      IB/mthca: Avoid integer overflow when allocating huge ICM table
      IB/ipath: Fix PCI config write size used to clear linkctrl error bits
      RDMA/nes: Remove session_id from nes_cm stuff

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Shirley Ma]

What's the status of RDS?

Thanks
Shirley

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Tziporet Koren]

Roland Dreier wrote:
> Core:
>
>  - I did a bunch of cleanups all over drivers/infiniband and the
>    gcc and sparse warning noise is down to a pretty reasonable level.
>    Further cleanups welcome of course.
>   
We want to add send with invalidate & mask compare and swap.
Eli will be able to send the patches next week and since they are small 
I think they can be in for 2.6.26
> ULPs:
>
>  - I merged Eli's IPoIB stateless offload changes for checksum
>    offload and LSO changes.  The interrupt moderation changes are
>    next, and should not be a problem to merge.  Please test IPoIB
>    on all sorts of hardware!
>   
What about the split CQ for UD mode? It's improved the IPoIB performance 
for small messages significantly.
>
> HW specific:
>
>   
mlx4- we plan to send patches for the low level driver only to enable 
mlx4_en. These only affect our low level driver.
Should be ready next week. I hope these can get in too.
> Here are a few topics that I believe will not be ready in time for the
> 2.6.26 window and will need to wait for 2.6.27 at least:
>
>  - XRC.  I still don't have a good feeling that we have settled on all
>    the nuances of the ABI we want to expose to userspace for this, and
>    ideally I would like to understand how ehca LL QPs fit into the
>    picture as well.
>   
I think we should try to push for XEC in 2.6.26 since there are already 
MPI implementation that use it and this ties them to use OFED only.
Also this feature is stable and now being defined in IBTA
Not taking it causing changes between OFED and the kernel and your 
libibverbs and we wish to avoid such gaps.
Is there any thing we can do to help and make it into 2.6.26?

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > What's the status of RDS?

I've never seen any patches.  I guess ask the RDS guys if/when they want
to start working on getting RDS merged.

 - R.

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > What is the work we need to do here - I was thinking RDS should just work ?

Stuff doesn't get merged into the kernel on its own.  If you want RDS
upstream then the first step is to post patches in a form suitable for
reviewing.  Then respond to the review comments.

The files Documentation/SubmittingPatches and to some extent
Documentation/SubmittingDrivers in the kernel source have more info.

 - R.

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > We want to add send with invalidate & mask compare and swap.
 > Eli will be able to send the patches next week and since they are
 > small I think they can be in for 2.6.26

Send with invalidate should be OK.  Let's see about the masked atomics
stuff -- we have a ton of new verbs and I think we might want to slow
down and make sure it all makes sense.

 > What about the split CQ for UD mode? It's improved the IPoIB
 > performance for small messages significantly.

Oh yeah... I'll try to get that in too.

 > mlx4- we plan to send patches for the low level driver only to enable
 > mlx4_en. These only affect our low level driver.

No problem in principle, let's see the actual patches.

 > I think we should try to push for XEC in 2.6.26 since there are
 > already MPI implementation that use it and this ties them to use OFED
 > only.
 > Also this feature is stable and now being defined in IBTA
 > Not taking it causing changes between OFED and the kernel and your
 > libibverbs and we wish to avoid such gaps.
 > Is there any thing we can do to help and make it into 2.6.26?

I don't have a good feeling that the user-kernel interface is well
thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs
and make sure we have something that makes sense for the future.

 - R.

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > Yes, I see this is for pushing RDS upstream - but what about running
 > RDS as is over IWARP NICs - that should just work right ?

No idea.  It depends on whether you took into account the differences
between IB and iWARP.  Anyway that's not really what this thread was about.

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

[Scott Weitzenkamp (sweitzen)]

> WRT to merging RDS into the kernel - our current plans are to wait to 
> see RDS adopted by more than Oracle - before approaching the kernel 
> community about inclusion of RDS.

I've seen statements before from someone from Oracle that RDS was only
for Oracle's use, for example, that person did not want netperf changed
to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

[Scott Weitzenkamp (sweitzen)]

Rich,

On Nov 1, 2007, you wrote this to rds-devel:

  "Netperf is too simplistic in that all it seems to do is stream data
in a 
  simple loop. This is not how Oracle uses the IPC and again does not 
  reflect what it would take to make UDP reliable.

  For this reason we are not interested in having Netperf support RDS
and 
  or seeing Netperf data."

I would like to see RDS supported by existing common tools like netperf,
iperf, etc. so we can easily compare how RDS performs to UDP for IPC
models other than Oracle.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems


 

> -----Original Message-----
> From: Richard Frank [mailto:richard.frank@oracle.com] 
> Sent: Wednesday, April 02, 2008 10:38 AM
> To: Scott Weitzenkamp (sweitzen)
> Cc: Roland Dreier (rdreier); rds-devel@oss.oracle.com; 
> linux-kernel@vger.kernel.org; general@lists.openfabrics.org
> Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans 
> for 2.6.26 (what'sin infiniband.git)
> 
> I believe there is a patch for NetPerf which supports RDS - 
> although it 
> may need to be updated - and submitted.
> 
> The only prior discussion I can think of - was whether or not NetPerf 
> exercises RDS as Oracle would.
> 
> I'm not proposing that we should enhance NetPerf to do that 
> (but that's 
> OK with me).
> 
> We created a tool rds-stress which does that.
> 
> Scott Weitzenkamp (sweitzen) wrote:
> >> WRT to merging RDS into the kernel - our current plans are 
> to wait to 
> >> see RDS adopted by more than Oracle - before approaching 
> the kernel 
> >> community about inclusion of RDS.
> >>     
> >
> > I've seen statements before from someone from Oracle that 
> RDS was only
> > for Oracle's use, for example, that person did not want 
> netperf changed
> > to support RDS.
> >
> > Scott Weitzenkamp
> > SQA and Release Manager
> > Data Center Access Engineering
> > Cisco Systems
> >   
> 

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

[Scott Weitzenkamp (sweitzen)]

I'd like to see netperf comparisions of UDP_STREAM/UDP_RR vs
RDS_STREAM/RDS_RR, does anyone have a patch that will apply cleanly to a
recent netperf?

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems


 

> -----Original Message-----
> From: Richard Frank [mailto:richard.frank@oracle.com] 
> Sent: Wednesday, April 02, 2008 11:00 AM
> To: Scott Weitzenkamp (sweitzen)
> Cc: Roland Dreier (rdreier); rds-devel@oss.oracle.com; 
> linux-kernel@vger.kernel.org; general@lists.openfabrics.org
> Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans 
> for 2.6.26 (what'sin infiniband.git)
> 
> OK - and the conversation was about using NetPerf to compare 
> performance 
> of RDS to UDP relative to suitability for Oracle use ... so I think 
> those statements still illustrate my points...
> 
> 1) NetPerf does not do what Oracle does - and hence is not 
> useful from 
> Oracle's perspective in comparing ULPs.
> 2) For some metrics - it's not valid to compare a 
> non-reliable IPC to a 
> reliable IPC -  it's not an apples to apples comparison. 
> Especially when 
> the app is considered and what the app must do to use UDP vs RDS.
> 
> I did not say that NetPerf should not be extended to support 
> RDS - just 
> that using it to do a comparison of ULPs to determine how well Oracle 
> would run - is not what we (Oracle) would want - at least that was my 
> intention..
> 
> Scott Weitzenkamp (sweitzen) wrote:
> > Rich,
> >
> > On Nov 1, 2007, you wrote this to rds-devel:
> >
> >   "Netperf is too simplistic in that all it seems to do is 
> stream data
> > in a 
> >   simple loop. This is not how Oracle uses the IPC and 
> again does not 
> >   reflect what it would take to make UDP reliable.
> >
> >   For this reason we are not interested in having Netperf 
> support RDS
> > and 
> >   or seeing Netperf data."
> >
> > I would like to see RDS supported by existing common tools 
> like netperf,
> > iperf, etc. so we can easily compare how RDS performs to UDP for IPC
> > models other than Oracle.
> >
> > Scott Weitzenkamp
> > SQA and Release Manager
> > Data Center Access Engineering
> > Cisco Systems
> >
> >
> >  
> >
> >   
> >> -----Original Message-----
> >> From: Richard Frank [mailto:richard.frank@oracle.com] 
> >> Sent: Wednesday, April 02, 2008 10:38 AM
> >> To: Scott Weitzenkamp (sweitzen)
> >> Cc: Roland Dreier (rdreier); rds-devel@oss.oracle.com; 
> >> linux-kernel@vger.kernel.org; general@lists.openfabrics.org
> >> Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans 
> >> for 2.6.26 (what'sin infiniband.git)
> >>
> >> I believe there is a patch for NetPerf which supports RDS - 
> >> although it 
> >> may need to be updated - and submitted.
> >>
> >> The only prior discussion I can think of - was whether or 
> not NetPerf 
> >> exercises RDS as Oracle would.
> >>
> >> I'm not proposing that we should enhance NetPerf to do that 
> >> (but that's 
> >> OK with me).
> >>
> >> We created a tool rds-stress which does that.
> >>
> >> Scott Weitzenkamp (sweitzen) wrote:
> >>     
> >>>> WRT to merging RDS into the kernel - our current plans are 
> >>>>         
> >> to wait to 
> >>     
> >>>> see RDS adopted by more than Oracle - before approaching 
> >>>>         
> >> the kernel 
> >>     
> >>>> community about inclusion of RDS.
> >>>>     
> >>>>         
> >>> I've seen statements before from someone from Oracle that 
> >>>       
> >> RDS was only
> >>     
> >>> for Oracle's use, for example, that person did not want 
> >>>       
> >> netperf changed
> >>     
> >>> to support RDS.
> >>>
> >>> Scott Weitzenkamp
> >>> SQA and Release Manager
> >>> Data Center Access Engineering
> >>> Cisco Systems
> >>>   
> >>>       
> 

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

What is the work we need to do here - I was thinking RDS should just work ?

Roland Dreier wrote:
>  > What's the status of RDS?
>
> I've never seen any patches.  I guess ask the RDS guys if/when they want
> to start working on getting RDS merged.
>
>  - R.
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

Yes, I see this is for pushing RDS upstream - but what about running RDS 
as is over IWARP NICs - that should just work right ?

Roland Dreier wrote:
>  > What is the work we need to do here - I was thinking RDS should just work ?
>
> Stuff doesn't get merged into the kernel on its own.  If you want RDS
> upstream then the first step is to post patches in a form suitable for
> reviewing.  Then respond to the review comments.
>
> The files Documentation/SubmittingPatches and to some extent
> Documentation/SubmittingDrivers in the kernel source have more info.
>
>  - R.
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

WRT to merging RDS into the kernel - our current plans are to wait to 
see RDS adopted by more than Oracle - before approaching the kernel 
community about inclusion of RDS.

Roland Dreier wrote:
>  > What is the work we need to do here - I was thinking RDS should just work ?
>
> Stuff doesn't get merged into the kernel on its own.  If you want RDS
> upstream then the first step is to post patches in a form suitable for
> reviewing.  Then respond to the review comments.
>
> The files Documentation/SubmittingPatches and to some extent
> Documentation/SubmittingDrivers in the kernel source have more info.
>
>  - R.
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

got it...

Roland Dreier wrote:
>  > Yes, I see this is for pushing RDS upstream - but what about running
>  > RDS as is over IWARP NICs - that should just work right ?
>
> No idea.  It depends on whether you took into account the differences
> between IB and iWARP.  Anyway that's not really what this thread was about.
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

[Richard Frank]

I believe there is a patch for NetPerf which supports RDS - although it 
may need to be updated - and submitted.

The only prior discussion I can think of - was whether or not NetPerf 
exercises RDS as Oracle would.

I'm not proposing that we should enhance NetPerf to do that (but that's 
OK with me).

We created a tool rds-stress which does that.

Scott Weitzenkamp (sweitzen) wrote:
>> WRT to merging RDS into the kernel - our current plans are to wait to 
>> see RDS adopted by more than Oracle - before approaching the kernel 
>> community about inclusion of RDS.
>>     
>
> I've seen statements before from someone from Oracle that RDS was only
> for Oracle's use, for example, that person did not want netperf changed
> to support RDS.
>
> Scott Weitzenkamp
> SQA and Release Manager
> Data Center Access Engineering
> Cisco Systems
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

[Richard Frank]

OK - and the conversation was about using NetPerf to compare performance 
of RDS to UDP relative to suitability for Oracle use ... so I think 
those statements still illustrate my points...

1) NetPerf does not do what Oracle does - and hence is not useful from 
Oracle's perspective in comparing ULPs.
2) For some metrics - it's not valid to compare a non-reliable IPC to a 
reliable IPC -  it's not an apples to apples comparison. Especially when 
the app is considered and what the app must do to use UDP vs RDS.

I did not say that NetPerf should not be extended to support RDS - just 
that using it to do a comparison of ULPs to determine how well Oracle 
would run - is not what we (Oracle) would want - at least that was my 
intention..

Scott Weitzenkamp (sweitzen) wrote:
> Rich,
>
> On Nov 1, 2007, you wrote this to rds-devel:
>
>   "Netperf is too simplistic in that all it seems to do is stream data
> in a 
>   simple loop. This is not how Oracle uses the IPC and again does not 
>   reflect what it would take to make UDP reliable.
>
>   For this reason we are not interested in having Netperf support RDS
> and 
>   or seeing Netperf data."
>
> I would like to see RDS supported by existing common tools like netperf,
> iperf, etc. so we can easily compare how RDS performs to UDP for IPC
> models other than Oracle.
>
> Scott Weitzenkamp
> SQA and Release Manager
> Data Center Access Engineering
> Cisco Systems
>
>
>  
>
>   
>> -----Original Message-----
>> From: Richard Frank [mailto:richard.frank@oracle.com] 
>> Sent: Wednesday, April 02, 2008 10:38 AM
>> To: Scott Weitzenkamp (sweitzen)
>> Cc: Roland Dreier (rdreier); rds-devel@oss.oracle.com; 
>> linux-kernel@vger.kernel.org; general@lists.openfabrics.org
>> Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans 
>> for 2.6.26 (what'sin infiniband.git)
>>
>> I believe there is a patch for NetPerf which supports RDS - 
>> although it 
>> may need to be updated - and submitted.
>>
>> The only prior discussion I can think of - was whether or not NetPerf 
>> exercises RDS as Oracle would.
>>
>> I'm not proposing that we should enhance NetPerf to do that 
>> (but that's 
>> OK with me).
>>
>> We created a tool rds-stress which does that.
>>
>> Scott Weitzenkamp (sweitzen) wrote:
>>     
>>>> WRT to merging RDS into the kernel - our current plans are 
>>>>         
>> to wait to 
>>     
>>>> see RDS adopted by more than Oracle - before approaching 
>>>>         
>> the kernel 
>>     
>>>> community about inclusion of RDS.
>>>>     
>>>>         
>>> I've seen statements before from someone from Oracle that 
>>>       
>> RDS was only
>>     
>>> for Oracle's use, for example, that person did not want 
>>>       
>> netperf changed
>>     
>>> to support RDS.
>>>
>>> Scott Weitzenkamp
>>> SQA and Release Manager
>>> Data Center Access Engineering
>>> Cisco Systems
>>>   
>>>       

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Tziporet Koren]

Roland Dreier wrote:
> Send with invalidate should be OK.  Let's see about the masked atomics
> stuff -- we have a ton of new verbs and I think we might want to slow
> down and make sure it all makes sense.
>   
OK - will send and then we will see what will come out.

>  > What about the split CQ for UD mode? It's improved the IPoIB
>  > performance for small messages significantly.
>
> Oh yeah... I'll try to get that in too.
>   
thanks
>  > mlx4- we plan to send patches for the low level driver only to enable
>  > mlx4_en. These only affect our low level driver.
>
> No problem in principle, let's see the actual patches.
>   
Sure
>  > I think we should try to push for XEC in 2.6.26 since there are
>  > already MPI implementation that use it and this ties them to use OFED
>  > only.
>  > Also this feature is stable and now being defined in IBTA
>  > Not taking it causing changes between OFED and the kernel and your
>  > libibverbs and we wish to avoid such gaps.
>  > Is there any thing we can do to help and make it into 2.6.26?
>
> I don't have a good feeling that the user-kernel interface is well
> thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs
> and make sure we have something that makes sense for the future.
>
>   
I see - but can't we figure this all for the 2.6.26 window?

Tziporet

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Or Gerlitz]

On Wed, Apr 2, 2008 at 3:31 PM, Tziporet Koren
<tziporet@dev.mellanox.co.il> wrote:

>  We want to add send with invalidate
>  Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

Does send with invalidate applies to rkeys generated through the
proprietary FMR API?
if not, what usage you envision to the new verb under nowadays IB devices?

Or.

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > We are very interested in these new operations and are moving in the
 > direction of tightly integrating RDMA along with atomics (if
 > available) into Oracle.  We plan on testing some early prototypes of
 > the these in the few months.

And you need the ConnectX-only masked atomics?  Or do the standard IB
atomic operations work for you?  Of course using atomics at all means
that things don't work on iWARP.

 > Send with invalidate is an exact match for our current RDS V3 rdma
 > driver - and should be more efficient than the current background
 > syncing of the tpt  to ensure keys are invalidated.

How does send with invalidate interact with the current IB FMR stuff?
Seems that you would run into trouble keeping the state of the FMR
straight if the remote side is invalidating them.

Also I would think that send-with-invalidate would be much more
expensive than the current FMR method of batching up the invalidates,
since you don't get to amortize the cost of syncing up all the internal
HCA state.

 - R.

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

 > We want to add send with invalidate & mask compare and swap.
 > Eli will be able to send the patches next week and since they are
 > small I think they can be in for 2.6.26

We are very interested in these new operations and are moving in the 
direction of tightly integrating RDMA along with atomics (if available) 
into Oracle.  We plan on testing some early prototypes of the these in 
the few months.

Send with invalidate is an exact match for our current RDS V3 rdma 
driver - and should be more efficient than the current background 
syncing of the tpt  to ensure keys are invalidated.

We intend on exposing the atomics via the RDS driver along with simple 
low level rdma operations to Oracle's internal clients. If Oracle is 
running over a transport which exports atomics and rdma - Oracle will 
see a dramatic performance boost for several database operations.

Roland Dreier wrote:
>  > We want to add send with invalidate & mask compare and swap.
>  > Eli will be able to send the patches next week and since they are
>  > small I think they can be in for 2.6.26
>
> Send with invalidate should be OK.  Let's see about the masked atomics
> stuff -- we have a ton of new verbs and I think we might want to slow
> down and make sure it all makes sense.
>
>  > What about the split CQ for UD mode? It's improved the IPoIB
>  > performance for small messages significantly.
>
> Oh yeah... I'll try to get that in too.
>
>  > mlx4- we plan to send patches for the low level driver only to enable
>  > mlx4_en. These only affect our low level driver.
>
> No problem in principle, let's see the actual patches.
>
>  > I think we should try to push for XEC in 2.6.26 since there are
>  > already MPI implementation that use it and this ties them to use OFED
>  > only.
>  > Also this feature is stable and now being defined in IBTA
>  > Not taking it causing changes between OFED and the kernel and your
>  > libibverbs and we wish to avoid such gaps.
>  > Is there any thing we can do to help and make it into 2.6.26?
>
> I don't have a good feeling that the user-kernel interface is well
> thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs
> and make sure we have something that makes sense for the future.
>
>  - R.
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Richard Frank]

Roland Dreier wrote:
>  > We are very interested in these new operations and are moving in the
>  > direction of tightly integrating RDMA along with atomics (if
>  > available) into Oracle.  We plan on testing some early prototypes of
>  > the these in the few months.
>
> And you need the ConnectX-only masked atomics?  Or do the standard IB
> atomic operations work for you?  Of course using atomics at all means
> that things don't work on iWARP.
>
>   
We specifically asked for the masked operations.

Yes, this means Oracle will not get the performance boost of atomics on 
IWARP - but we still get rdma - and that's a real win / benefit for 
Oracle today - and more so over the next few months.

>  > Send with invalidate is an exact match for our current RDS V3 rdma
>  > driver - and should be more efficient than the current background
>  > syncing of the tpt  to ensure keys are invalidated.
>
> How does send with invalidate interact with the current IB FMR stuff?
> Seems that you would run into trouble keeping the state of the FMR
> straight if the remote side is invalidating them.
>
>   
The model we implement is based on "use once" keys - we issue the key to 
the rdma server and want to toss it as soon as the rdma is complete. 
Today, we explicitly free the key after the rdma completes and we get a 
message from the rdma server - saying rdma is complete. If the key is 
auto invalidated by the recv'ing HCA then we do not need to do it in the 
driver... which also meanswe do not need to issue the sync tpts to force 
the HCA to be update its cache.

At least this is how I think it works - Olaf is the divine source here.

> Also I would think that send-with-invalidate would be much more
> expensive than the current FMR method of batching up the invalidates,
> since you don't get to amortize the cost of syncing up all the internal
> HCA state.
>
>   
This is the one piece we do not know - our plans are to test this and 
see where the trade offs are. We will keep the current design / 
implementation to run over NICs that do not support send-with-invalidate.
>  - R.
>   

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

[Roland Dreier]

 > Can the maintainer submit RDS patch for mainline kernel, in 2.6.26 or
 > 2.6.27 window? It's hard for Distros pick this feature without mainline
 > kernel acceptance.

At least as a first order approximation, there is no chance of RDS being
merged for 2.6.26 even if patches appear right this second...