n
> > > from OFED.
> > > 
> > > Intel put efforts into the iWARP, which is sort of alternative for RoCEv2.
> > > It has RFCs and works over TCP AFAIR, which causes problems for it.
> > Heh, heh. I'm trying to avoid the iWARP vs RoCE wars.;-)
> > (I did see a Mellanox white paper with graphs showing how RoCE outperforms
> > iWARP.)
> > Intel currently claims to support RoCE on its 810 and 820 NICs.
> > Broadcom also claims to support RoCE, but doesn't mention FreeBSD
> > drivers and Chelsio does iWARP, afaik.
> > 
> > For some reason, at the last NFSv4 Bakeathon, Chuck was testing with
> > iWARP and not RoCE? (I haven't asked Chuck why he chose that. It
> > might just be more convenient to set up the siw driver in Linux vs the
> > rxe one? He is the main author of RFC-8166, so he's the NFS-over-RDMA guy.)
> > 
> > But it does look like a fun project for the next year. (I recall jhb@ mentioning
> > that NFS-over-TLS wouldn't be easy and it turned out to be a fun
> > little project.)
> 
> Konstantin is right though that sys/ofed is Linux OpenIB and has an interface
> that should let you do RDMA (both ROCEv2 and iWARP).  I'm hoping to use the APIs
> in sys/ofed to support NVMe over RDMA (both ROCEv2 and iWARP) at some point as
> well.
> > rick
> > 
> > > 
> > > > 
> > > > Btw, if anyone is interested in taking a more active involvement in this,
> > > > they are more than welcome to do so. (I'm going to be starting where I
> > > > understand things in the krpc/nfs. I'm not looking forward to porting rxe,
> > > > but will probably end up there. I have already had one offer w.r.t. access
> > > > to a lab that includes Mellanox hardware, but I don't know if remote
> > > > debugging will be practical yet.)
> > > > 
> > > > rick
> > > > 
> > > > > 
> > > > > The IB implementation for us is still called OFED for historical reasons,
> > > > > and it is located in sys/ofed.
> > > > > 
> > > > > > 
> > > > > > As for testing, I am planning on hacking away at one of the RDMA
> > > > > > in software drivers in Linux to get it working well enough to use for
> > > > > > testing. Whatever seems to be easiest to get kinda working.
> > > > > Yes rxe driver is the sw RoCE v2 implementation.  We looked at the
> > > > > amount of work to port it.  Its size is ~12 kLoC, which is compatible
> > > > > with libibverbs (userspace core infiniband interface).
> 
> Interesting.  I'm currently working on merging back several OFED commits from
> Linux to sys/ofed (currently I have about 30 commits merged, some older than
> Hans' last merge, and some newer, some of the newer ones should permit removing
> compat stubs for some of the newer APIs that are duplicated in bnxt, irdma, and
> mlx*).  When I get a bit further along I'll post the branch I have for more
> testing (it is a bunch of individual cherry-picks rather than a giant merge).

Talking about merges.  There is a long-time rotten series of patches
from supposedly Juniper that added some features to our IB.  Largest
features seems to be tunnelling modes, but there are a lot of scattered
bug fixes as well.  This was advertised as necessary for native DPDK 
support on mlx5 (PMD).

See the patches starting at https://reviews.freebsd.org/D32176.
I revived them and rebased on top of main, see
https://people.freebsd.org/~kib/git/deviant3.git/
branch devx.

I was unable to contact the author of the reviews, and we (NVidia) are not
sure how to proceed with this stuff.  We do not want to allow it to rot
further.

If anybody has the right contact either for author or for the team that
uses this series of patches, it would be helpful.

> 
> Porting over rxe could be useful for me as well for some work I am doing.

Thank you for noting this.