From owner-freebsd-infiniband@FreeBSD.ORG Sat Mar 22 23:15:50 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB78937C for ; Sat, 22 Mar 2014 23:15:50 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7F6771E3 for ; Sat, 22 Mar 2014 23:15:50 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s2MNFfI6057606 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 22 Mar 2014 23:15:42 GMT Date: Sat, 22 Mar 2014 23:15:41 +0000 From: Karl Pielorz To: freebsd-infiniband@freebsd.org Subject: Mellanox MT25418 performance IPOIB? Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Mar 2014 23:15:50 -0000 Hi All, I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to use them as a cheap alternative to 10Gbps LAN cards for a point to point link to run HAST over. I've remade/installed the world with OFED support - and compiled OFED, SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel. I've setup/run opensm - and the cards appear to be active OK - ibstat output seems OK, and ibportstate shows the cabled port is 'Active' / 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to each other). I've put an IP on both ends - and can ping between the cards, but TCP/IP performance seems very poor - e.g. a test 'ftp' between the cards only nets around 381Kbyte/s? [iperf shows similarly poor results] I can't see any errors logged anywhere - and 'netstat -b -i' doesn't show any errors either. Anyone else running these cards successfully? My opensm config consists of just 'PORTS="ALL"' - I was unable to find a lot of example opensm file configs. Also I get some chatter dumped on the console (presumably from the drivers) most of it looks just like some kind of debug output [can't see any obvious errors]. fwiw both machines are 1230v3 Xeon based. Can anyone suggest what might be wrong? / What to look at? Thanks, -Karl