Date: Wed, 16 Apr 2014 14:28:37 +0200 From: =?ISO-8859-1?Q?Stig_Inge_Lea_Bj=F8rnsen?= <stiginge@pvv.org> To: freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <lilt1l$15v$1@ger.gmane.org> In-Reply-To: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk> References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, I'm experimenting with a similar setup for a direct link using two HP MT25418 cards. One machine runs FreeBSD 10.0-RELEASE-p1 and the other runs Debian Unstable jessie/sid. You could try to use Infiniband in datagram mode, recompile without the IPOIB_CM option. The maximum MTU will then be 4092, but it might default to 2044. I run OpenSM on both machines and set the MTU explicitly: $ cat /etc/opensm/partitions.conf # mtu=5 for 4092, 4 for 2044 Default=0x7fff, ipoib, mtu=5 : ALL=full ; For iperf, I found that the performance when using FreeBSD as the client improved dramatically when I set the TCP window size equal to what the iperf client on Linux use: $ iperf -c linux-host -w 165K The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions. Regards, Stig Inge Lea Bjørnsen On 2014-03-23 00:15, Karl Pielorz wrote: > > Hi All, > > I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / > firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to > use them as a cheap alternative to 10Gbps LAN cards for a point to point > link to run HAST over. > > I've remade/installed the world with OFED support - and compiled OFED, > SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel. > > I've setup/run opensm - and the cards appear to be active OK - ibstat > output seems OK, and ibportstate shows the cabled port is 'Active' / > 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to > each other). > > I've put an IP on both ends - and can ping between the cards, but TCP/IP > performance seems very poor - e.g. a test 'ftp' between the cards only > nets around 381Kbyte/s? [iperf shows similarly poor results] > > I can't see any errors logged anywhere - and 'netstat -b -i' doesn't > show any errors either. > > Anyone else running these cards successfully? > > My opensm config consists of just 'PORTS="ALL"' - I was unable to find a > lot of example opensm file configs. > > Also I get some chatter dumped on the console (presumably from the > drivers) most of it looks just like some kind of debug output [can't see > any obvious errors]. > > fwiw both machines are 1230v3 Xeon based. > > Can anyone suggest what might be wrong? / What to look at? > > Thanks, > > -Karl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?lilt1l$15v$1>