Date: Tue, 15 May 2012 09:44:39 -0400 From: gnn@freebsd.org To: Gergely CZUCZY <gergely.czuczy@harmless.hu> Cc: jeff@freebsd.org, jhb@freebsd.org, net@freebsd.org Subject: Re: OFED stack, RDMA, ipoib help needed Message-ID: <8662bxa794.wl%gnn@neville-neil.com> In-Reply-To: <20120508121120.00003ea3@unknown> References: <20120508121120.00003ea3@unknown>
next in thread | previous in thread | raw e-mail | index | archive | help
At Tue, 8 May 2012 12:11:20 +0200, Gergely CZUCZY wrote: > > Hello, > > I'd like to ask a few question in order to get some hardware to work > we've got recently. > > The hardwares are the following: > - 2x dualport Mellanox ConnectX-3 VPI cards, with 56Gbps ports > - 4 computing modules with a singleport Mellanox MT27500-family > ConnectX-3 port. > > The 2 dualport cards are in a storage box, and the 4 singleport ones > are integrated on blade-like computing nodes (4 boxes in 2U). The > storage is running FreeBSD 9-STABLE, 2012-05-07 cvsup, and the > computing nodes are running linux. > > So far we had been able to bring up the subnet-manager on the FreeBSD > node, and one of the links got into Active state, which is quite good. > We had been able to ibping between the nodes. The FreeBSD kernel > config, in addition to GENERIC, is the following: > > options OFED > options SDP > device ipoib > options IPOIB_CM > device mlx4ib > device mthca > device mlxen > > Right now we're having problems with the following issues, situations: > > 1) we assigned IP addresses to both ib interfaces (fbsd, linux side), > but weren't able to ping over IP. We've seen icmp-echo-requests leaving > the box on the linux box, but haven't seen any incoming traffic on the > freebsd one. On the freebsd side, we had several issues: > - no incoming packets seen by tcpdump on the ib interface > - when trying to ping the other side, we've got "no route to host", > but the routing entry existed in the routing table. > - we had a few of these messages in our messages: "ib2: timing out; 0 > sends N recieves not completed", where started at 22,34 and was > growing. > Have you looked at your arp tables? (arp -a) Do you have any messages in dmesg on the FreeBSD side? Can you show us the output of ifconfig on the FreeBSD side? > 2) We're unable to find any resources on how to do RDMA on the FreeBSD > side. We'd like to use SRP (SCSI RDMA Protocol) communication, and/or > NFS-over-RDMA for our storage link between the boxes. Where could we > find any info on this? Sorry but I can't help you with this one. > 3) Enabling connected-mode, we weren't able to find a way to specify or > query the port that connected mode is using. Could someone please point > us to the right direction regarding this minor issue? This ought to work in FreeBSD as it does in Linux, but I've not personally tried it. > 4) We were also unable to find how to switch these dual-personality > cards between infiniband and ethernet modes. Could we also get some > pointers regarding this please? > It usually depends on what cable you're using, what it's plugged into, and what driver you bring up. The mlx4 driver should be able to give you an Ethernet device with the Connect X-3 cards. > Basically any help would be welcome which could help making infiniband > work. > > As a side question, I've seen a comming for OFED in HEAD by jhb, fixing > a few things, may I ask when will that get MFC'd to RELENG-9? > This I don't know about. Best, George
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8662bxa794.wl%gnn>