Date: Sun, 23 Mar 2014 08:27:47 -0500 From: Jason Bacon <jwbacon@tds.net> To: freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <532EE153.3010903@tds.net> In-Reply-To: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk> References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Coincidentally, I had just run similar tests on our cluster using a
FreeBSD file server and CentOS compute nodes.
I reported the results in to this list. If you didn't see the thread
already, you should be able to find it in the archives at:
http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
Basically, iperf reported excellent incoming speed from the CentOS
nodes, but outgoing speed was terrible.
In response, I was informed by a Mellanox engineer that the ipoib layer
is currently being overhauled and we should see improvement later this year.
Regards,
JB
On 03/22/14 18:15, Karl Pielorz wrote:
>
> Hi All,
>
> I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 /
> firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping
> to use them as a cheap alternative to 10Gbps LAN cards for a point to
> point link to run HAST over.
>
> I've remade/installed the world with OFED support - and compiled OFED,
> SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel.
>
> I've setup/run opensm - and the cards appear to be active OK - ibstat
> output seems OK, and ibportstate shows the cabled port is 'Active' /
> 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to
> each other).
>
> I've put an IP on both ends - and can ping between the cards, but
> TCP/IP performance seems very poor - e.g. a test 'ftp' between the
> cards only nets around 381Kbyte/s? [iperf shows similarly poor results]
>
> I can't see any errors logged anywhere - and 'netstat -b -i' doesn't
> show any errors either.
>
> Anyone else running these cards successfully?
>
> My opensm config consists of just 'PORTS="ALL"' - I was unable to find
> a lot of example opensm file configs.
>
> Also I get some chatter dumped on the console (presumably from the
> drivers) most of it looks just like some kind of debug output [can't
> see any obvious errors].
>
> fwiw both machines are 1230v3 Xeon based.
>
> Can anyone suggest what might be wrong? / What to look at?
>
> Thanks,
>
> -Karl
> _______________________________________________
> freebsd-infiniband@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to
> "freebsd-infiniband-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532EE153.3010903>