Date: Tue, 8 May 2012 12:11:20 +0200 From: Gergely CZUCZY <gergely.czuczy@harmless.hu> To: net@freebsd.org Cc: jeff@freebsd.org, jhb@freebsd.org Subject: OFED stack, RDMA, ipoib help needed Message-ID: <20120508121120.00003ea3@unknown>
next in thread | raw e-mail | index | archive | help
Hello,
I'd like to ask a few question in order to get some hardware to work
we've got recently.
The hardwares are the following:
- 2x dualport Mellanox ConnectX-3 VPI cards, with 56Gbps ports
- 4 computing modules with a singleport Mellanox MT27500-family
ConnectX-3 port.
The 2 dualport cards are in a storage box, and the 4 singleport ones
are integrated on blade-like computing nodes (4 boxes in 2U). The
storage is running FreeBSD 9-STABLE, 2012-05-07 cvsup, and the
computing nodes are running linux.
So far we had been able to bring up the subnet-manager on the FreeBSD
node, and one of the links got into Active state, which is quite good.
We had been able to ibping between the nodes. The FreeBSD kernel
config, in addition to GENERIC, is the following:
options OFED
options SDP
device ipoib
options IPOIB_CM
device mlx4ib
device mthca
device mlxen
Right now we're having problems with the following issues, situations:
1) we assigned IP addresses to both ib interfaces (fbsd, linux side),
but weren't able to ping over IP. We've seen icmp-echo-requests leaving
the box on the linux box, but haven't seen any incoming traffic on the
freebsd one. On the freebsd side, we had several issues:
- no incoming packets seen by tcpdump on the ib interface
- when trying to ping the other side, we've got "no route to host",
but the routing entry existed in the routing table.
- we had a few of these messages in our messages: "ib2: timing out; 0
sends N recieves not completed", where started at 22,34 and was
growing.
2) We're unable to find any resources on how to do RDMA on the FreeBSD
side. We'd like to use SRP (SCSI RDMA Protocol) communication, and/or
NFS-over-RDMA for our storage link between the boxes. Where could we
find any info on this?
3) Enabling connected-mode, we weren't able to find a way to specify or
query the port that connected mode is using. Could someone please point
us to the right direction regarding this minor issue?
4) We were also unable to find how to switch these dual-personality
cards between infiniband and ethernet modes. Could we also get some
pointers regarding this please?
Basically any help would be welcome which could help making infiniband
work.
As a side question, I've seen a comming for OFED in HEAD by jhb, fixing
a few things, may I ask when will that get MFC'd to RELENG-9?
Thanks in advance.
Best regards,
Gergely CZUCZY
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120508121120.00003ea3>