Date: Thu, 17 Apr 2014 08:01:47 -0500 From: Jason Bacon <jwbacon@tds.net> To: Karl Pielorz <kpielorz_lst@tdx.co.uk>, =?UTF-8?B?U3RpZyBJbmdlIExlYSBC?= =?UTF-8?B?asO4cm5zZW4=?= <stiginge@pvv.org>, freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <534FD0BB.1070308@tds.net> In-Reply-To: <830453C43410411669F120CC@study64.tdx.co.uk> References: <DF61C9D7E8748BF2824DDFED@study64.tdx.co.uk> <lilt1l$15v$1@ger.gmane.org> <830453C43410411669F120CC@study64.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Karl,
What type of network are you running? DDR? QDR? FDR? Switch and HCA
models?
We're purchasing a new cluster with Mellanox ConnectX-3 FDR cards. I'll
report back some results with and without IPOIB once I have my hands on it.
FYI, with some tuning effort, I was able to get up to 600 megabytes/sec
both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a
PERC H710, ext4, RAID 6. Might be able to do a little better with
FreeBSD and ZFS where IB is providing 7.5gb/sec.
Regards,
JB
On 4/17/14 4:09 AM, Karl Pielorz wrote:
>
> --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bjørnsen
> <stiginge@pvv.org> wrote:
>
>> You could try to use Infiniband in datagram mode, recompile without the
>> IPOIB_CM option.
>
> Ok, same boxes - I just removed IPOIB_CM out of the kernel config and
> recompiled / installed it.
>
> iperf now consistently returns:
>
> [ 3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 8.71 GBytes 7.46 Gbits/sec
>
>> The maximum MTU will then be 4092, but it might default to 2044. I run
>> OpenSM on both machines and set the MTU explicitly:
>
> Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss
> of .5-1Gbit with iperf (obviously not 'real world') - but at leas I
> can now try and use the stuff with real world (now it's returning more
> than a few hundred kilobytes a second!).
>
>> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions.
>
> Same as here then.
>
> Thanks for your email!
>
> Regards,
>
> -Karl
> _______________________________________________
> freebsd-infiniband@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to
> "freebsd-infiniband-unsubscribe@freebsd.org"
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jason W. Bacon
jwbacon@tds.net
Circumstances don't make a man:
They reveal him.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?534FD0BB.1070308>