From owner-freebsd-net@FreeBSD.ORG Tue May 8 11:07:25 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A20BF1065670; Tue, 8 May 2012 11:07:25 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2A1348FC08; Tue, 8 May 2012 11:07:25 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAAP+qE+DaFvO/2dsb2JhbABEhXKuOoIMAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEcBIdoBQunc5MggS+JbQiEUIEYBJNPgi+BEY8xgwWBOwc X-IronPort-AV: E=Sophos;i="4.75,550,1330923600"; d="scan'208";a="168354586" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 08 May 2012 07:07:24 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 6998DB4059; Tue, 8 May 2012 07:07:24 -0400 (EDT) Date: Tue, 8 May 2012 07:07:24 -0400 (EDT) From: Rick Macklem To: Gergely CZUCZY Message-ID: <1217430168.59507.1336475244392.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20120508121120.00003ea3@unknown> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: jeff@freebsd.org, jhb@freebsd.org, net@freebsd.org Subject: Re: OFED stack, RDMA, ipoib help needed X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2012 11:07:25 -0000 Gergely CZUCZY wrote: > Hello, > > I'd like to ask a few question in order to get some hardware to work > we've got recently. > > The hardwares are the following: > - 2x dualport Mellanox ConnectX-3 VPI cards, with 56Gbps ports > - 4 computing modules with a singleport Mellanox MT27500-family > ConnectX-3 port. > > The 2 dualport cards are in a storage box, and the 4 singleport ones > are integrated on blade-like computing nodes (4 boxes in 2U). The > storage is running FreeBSD 9-STABLE, 2012-05-07 cvsup, and the > computing nodes are running linux. > > So far we had been able to bring up the subnet-manager on the FreeBSD > node, and one of the links got into Active state, which is quite good. > We had been able to ibping between the nodes. The FreeBSD kernel > config, in addition to GENERIC, is the following: > > options OFED > options SDP > device ipoib > options IPOIB_CM > device mlx4ib > device mthca > device mlxen > > Right now we're having problems with the following issues, situations: > > 1) we assigned IP addresses to both ib interfaces (fbsd, linux side), > but weren't able to ping over IP. We've seen icmp-echo-requests > leaving > the box on the linux box, but haven't seen any incoming traffic on the > freebsd one. On the freebsd side, we had several issues: > - no incoming packets seen by tcpdump on the ib interface > - when trying to ping the other side, we've got "no route to host", > but the routing entry existed in the routing table. > - we had a few of these messages in our messages: "ib2: timing out; 0 > sends N recieves not completed", where started at 22,34 and was > growing. > > 2) We're unable to find any resources on how to do RDMA on the FreeBSD > side. We'd like to use SRP (SCSI RDMA Protocol) communication, and/or > NFS-over-RDMA for our storage link between the boxes. Where could we > find any info on this? > NFS-over-RDMA requires sessions, which are a part of NFSv4.1. There is no NFSv4.1 server support at this time. I know diddly about infiniband, so I can't help w.r.t. the rest. Good luck with it, rick > 3) Enabling connected-mode, we weren't able to find a way to specify > or > query the port that connected mode is using. Could someone please > point > us to the right direction regarding this minor issue? > > 4) We were also unable to find how to switch these dual-personality > cards between infiniband and ethernet modes. Could we also get some > pointers regarding this please? > > Basically any help would be welcome which could help making infiniband > work. > > As a side question, I've seen a comming for OFED in HEAD by jhb, > fixing > a few things, may I ask when will that get MFC'd to RELENG-9? > > Thanks in advance. > > Best regards, > Gergely CZUCZY > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"