Date: Thu, 17 Apr 2014 08:01:47 -0500 From: Jason Bacon <jwbacon@tds.net> To: Karl Pielorz <kpielorz_lst@tdx.co.uk>, =?UTF-8?B?U3RpZyBJbmdlIExlYSBC?= =?UTF-8?B?asO4cm5zZW4=?= <stiginge@pvv.org>, freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <534FD0BB.1070308@tds.net> In-Reply-To: <830453C43410411669F120CC@study64.tdx.co.uk> References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk> <lilt1l$15v$1@ger.gmane.org> <830453C43410411669F120CC@study64.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Karl, What type of network are you running? DDR? QDR? FDR? Switch and HCA models? We're purchasing a new cluster with Mellanox ConnectX-3 FDR cards. I'll report back some results with and without IPOIB once I have my hands on it. FYI, with some tuning effort, I was able to get up to 600 megabytes/sec both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a PERC H710, ext4, RAID 6. Might be able to do a little better with FreeBSD and ZFS where IB is providing 7.5gb/sec. Regards, JB On 4/17/14 4:09 AM, Karl Pielorz wrote: > > --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bjørnsen > <stiginge@pvv.org> wrote: > >> You could try to use Infiniband in datagram mode, recompile without the >> IPOIB_CM option. > > Ok, same boxes - I just removed IPOIB_CM out of the kernel config and > recompiled / installed it. > > iperf now consistently returns: > > [ 3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 8.71 GBytes 7.46 Gbits/sec > >> The maximum MTU will then be 4092, but it might default to 2044. I run >> OpenSM on both machines and set the MTU explicitly: > > Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss > of .5-1Gbit with iperf (obviously not 'real world') - but at leas I > can now try and use the stuff with real world (now it's returning more > than a few hundred kilobytes a second!). > >> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions. > > Same as here then. > > Thanks for your email! > > Regards, > > -Karl > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?534FD0BB.1070308>