From owner-freebsd-net@FreeBSD.ORG Tue May 8 10:11:23 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01397106566C; Tue, 8 May 2012 10:11:23 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id B5E0D8FC12; Tue, 8 May 2012 10:11:22 +0000 (UTC) Received: from mail.qualysoft.com ([213.253.200.123] helo=unknown) by marvin.harmless.hu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.75 (FreeBSD)) (envelope-from ) id 1SRhO4-000FjH-PM; Tue, 08 May 2012 12:11:20 +0200 Date: Tue, 8 May 2012 12:11:20 +0200 From: Gergely CZUCZY To: net@freebsd.org Message-ID: <20120508121120.00003ea3@unknown> Organization: Harmless Digital X-Mailer: Claws Mail 3.7.6 (GTK+ 2.16.0; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: jeff@freebsd.org, jhb@freebsd.org Subject: OFED stack, RDMA, ipoib help needed X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2012 10:11:23 -0000 Hello, I'd like to ask a few question in order to get some hardware to work we've got recently. The hardwares are the following: - 2x dualport Mellanox ConnectX-3 VPI cards, with 56Gbps ports - 4 computing modules with a singleport Mellanox MT27500-family ConnectX-3 port. The 2 dualport cards are in a storage box, and the 4 singleport ones are integrated on blade-like computing nodes (4 boxes in 2U). The storage is running FreeBSD 9-STABLE, 2012-05-07 cvsup, and the computing nodes are running linux. So far we had been able to bring up the subnet-manager on the FreeBSD node, and one of the links got into Active state, which is quite good. We had been able to ibping between the nodes. The FreeBSD kernel config, in addition to GENERIC, is the following: options OFED options SDP device ipoib options IPOIB_CM device mlx4ib device mthca device mlxen Right now we're having problems with the following issues, situations: 1) we assigned IP addresses to both ib interfaces (fbsd, linux side), but weren't able to ping over IP. We've seen icmp-echo-requests leaving the box on the linux box, but haven't seen any incoming traffic on the freebsd one. On the freebsd side, we had several issues: - no incoming packets seen by tcpdump on the ib interface - when trying to ping the other side, we've got "no route to host", but the routing entry existed in the routing table. - we had a few of these messages in our messages: "ib2: timing out; 0 sends N recieves not completed", where started at 22,34 and was growing. 2) We're unable to find any resources on how to do RDMA on the FreeBSD side. We'd like to use SRP (SCSI RDMA Protocol) communication, and/or NFS-over-RDMA for our storage link between the boxes. Where could we find any info on this? 3) Enabling connected-mode, we weren't able to find a way to specify or query the port that connected mode is using. Could someone please point us to the right direction regarding this minor issue? 4) We were also unable to find how to switch these dual-personality cards between infiniband and ethernet modes. Could we also get some pointers regarding this please? Basically any help would be welcome which could help making infiniband work. As a side question, I've seen a comming for OFED in HEAD by jhb, fixing a few things, may I ask when will that get MFC'd to RELENG-9? Thanks in advance. Best regards, Gergely CZUCZY