Date: Wed, 19 Mar 2014 12:26:16 -0500 From: Jason Bacon <bacon@uwm.edu> To: Oded Shanoon <odeds@mellanox.com> Cc: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: Supported NICs Message-ID: <5329D338.3060103@uwm.edu> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> References: <52361685.8000901@tds.net> <CAAoDvRxtKZ4AH5MTC5xi9TrXgcFr7MWGpJ3ODa-Vz0WCANkC_A@mail.gmail.com> <CAAoDvRwaDT6d8goNpvSp4jVYcSm4cZZLLqa2gCxn8EjSiLFHpA@mail.gmail.com> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com>
next in thread | previous in thread | raw e-mail | index | archive | help
We finally got a new server in with a ConnectX-3 card. It seems to work
reliably with our SilverStorm DDR network using a CX4 - QSFP cable.
Below are specs and iperf benchmarks between the FreeBSD server and a
CentOS 6.5 compute node with a Qlogic DDR card. Connected mode with MTU
65520. FreeBSD 10.0-RELEASE on a Dell R720xd.
Wondering if anyone has any suggestions about what might cause the poor
output performance. Any sysctls or loader.conf settings I should look
into? None of the kern.conftext flags mentioned below are present.
Any pointers on how to diagnose the issue would be appreciated as well.
Thanks,
Jason
Input throughput is excellent - over 11 gb/sec. Linux to Linux on the
same network does about 10 gb/sec.
[root@compute-5-33 ~]# iperf -c 10.1.2.250
------------------------------------------------------------
Client connecting to 10.1.2.250, TCP port 5001
TCP window size: 645 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.2.140 port 54223 connected with 10.1.2.250 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 13.0 GBytes 11.2 Gbits/sec
bacon@raid1:~ % netstat 1
input (Total) output
packets errs idrops bytes packets errs bytes colls
10631 0 0 685149597 5390 0 409806 0
22281 0 0 1452318244 11136 0 846730 0
23776 0 0 1557833160 11884 0 903654 0
23891 0 0 1565368420 11952 0 907986 0
23328 0 0 1528478408 11663 0 886554 0
Output throughput is terrible:
[root@compute-5-33 ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.1.2.140 port 5001 connected with 10.1.2.250 port 50207
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.9 sec 1.50 MBytes 1.16 Mbits/sec
bacon@raid1:~ % netstat 1
input (Total) output
packets errs idrops bytes packets errs bytes colls
20 0 0 1444 27 1 124684 0
5 0 0 284 1 4 393718 0
5 0 0 284 1 4 393718 0
6 0 0 344 1 4 393718 0
6 0 0 344 18 4 426562 0
5 0 0 284 1 4 393718 0
root@raid1:/home/bacon # ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0xf4521403007d2c10
System image GUID: 0xf4521403007d2c13
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 163
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0xf4521403007d2c11
root@raid1:/home/bacon # ibportstate -G 0xf4521403007d2c11 query
PortInfo:
# Port info: Lid 163 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkWidthSupported:..............4X (IBA extension)
LinkWidthEnabled:................4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................5.0 Gbps
On 10/16/13 10:23, Oded Shanoon wrote:
> Please run sysctl kern.conftxt and grep for the following options:
>
> OFED_DEBUG_INIT
> IPOIB_DEBUG
> DEADLKRES
> INVARIANTS
> INVARIANT_SUPPORT
> WITNESS
> WITNESS_SKIPSPIN
>
> These could result in performance degredation
>
> Regards,
>
> Oded Shanoon
> OFED-FreeBSD Team Leader
> Mellanox Technologies, Raanana
>
>
> -----Original Message-----
> From: Jason Bacon [mailto:bacon@uwm.edu]
> Sent: Monday, October 14, 2013 9:37 PM
> To: Oded Shanoon
> Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org
> Subject: Re: Supported NICs
>
>
> Some initial test results...
>
> I installed an old Infinihost DDR HCA in one of our compute nodes running FreeBSD 9.1.
>
> RHEL nodes are using qlogic IB HCAs.
>
> 10.1.1 is gigabit Ethernet, 10.1.2 is IB.
>
> Running iperf server on FreeBSD and client on one of our RHEL nodes shows OK performance:
>
> bacon@infinibsd:/home/bacon % iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 64.0 KByte (default)
> ------------------------------------------------------------
> [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947
> [ ID] Interval Transfer Bandwidth
> [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec
>
> [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec
>
> RHEL to RHEL gives us about 8 Gbits/sec.
>
> Iperf server on RHEL and client on FreeBSD shows very poor performance for IB, while GigE is fine:
>
> bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39
> ------------------------------------------------------------
> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
>
> bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39
> ------------------------------------------------------------
> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec
>
> Any ideas what might cause this?
>
> I'm going to install 9.2-RELEASE and retest in any case, plus explore the ib config tools, but any feedback in the meantime would be appreciated.
>
> Thanks,
>
> JB
>
> On 10/07/13 01:51, Oded Shanoon wrote:
>> Hi Jason,
>>
>> IB support in 9.2 is supposed to be stable enough.
>> Please note that Mellanox started supporting FreeBSD only recently. The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5.3.
>> Since we started our involvement we entered some fixes to that driver (that were submitted into 9.2).
>> We also mapped various issues which needs to be fixed in the future.
>> We are now working on a major "face lift" to the driver - making it much more stable and with improved performance and features.
>>
>> Regards,
>>
>> Oded Shanoon
>> OFED-FreeBSD Team Leader
>> Mellanox Technologies, Raanana
>>
>>
>> -----Original Message-----
>> From: owner-freebsd-infiniband@freebsd.org
>> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon
>> Sent: Monday, September 16, 2013 4:32 PM
>> To: Anthony Cornehl
>> Cc: freebsd-infiniband@freebsd.org
>> Subject: Re: Supported NICs
>>
>>
>> Thanks Anthony& Oded!
>>
>> Would you say that IB support in 9.2 is stable enough for a production HPC file server?
>>
>> Will do plenty of my own testing first, of course.
>>
>> Regards,
>>
>> Jason
>>
>> On 09/15/13 22:38, Anthony Cornehl wrote:
>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"<accornehl@gmail.com
>>> <mailto:accornehl@gmail.com>> wrote:
>>>> On Sep 15, 2013 1:20 PM, "Jason bacon"<jwbacon@tds.net
>>> <mailto:jwbacon@tds.net>> wrote:
>>>>> Is there a list of supported IB NICs out there somewhere?
>>>>>
>>>>> I followed the wiki instructions for rebuilding with IB support
>>> and now have mlx4ib, mlxen, etc.
>>>>> Was hoping there would be man pages for the drivers that list
>>> known working cards, but there don't seem to be. I'm hoping to test
>>> a file server using IPOIB and possible roll a FreeNAS ISO with IB
>>> support if it works out.
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> Jason W. Bacon
>>>>> jwbacon@tds.net<mailto:jwbacon@tds.net>
>>>>>
>>>>> Circumstances don't make a man:
>>>>> They reveal him.
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-infiniband@freebsd.org
>>> <mailto:freebsd-infiniband@freebsd.org> mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>>>>> To unsubscribe, send any mail to
>>> "freebsd-infiniband-unsubscribe@freebsd.org
>>> <mailto:freebsd-infiniband-unsubscribe@freebsd.org>"
>>>> Hey Jason,
>>>>
>>>> Mellanox ConnectX series cards are the only ones supported currently...
>>>>
>>>> http://www.mellanox.com/page/infiniband_cards_overview
>>>>
>>>> Don't forget to compile with IPoIB as well, since the IPoIB support
>>> is compiled into the driver, unlike in Linux.
>>>> Just be aware that IPoIB performance is also improved by enabling
>>> connected mode when you compile the kernel module. The IB code in
>>> FreeBSD is a few years older the what is in Linux, but, the following
>>> forum thread is probably relevant...
>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo
>>> r
>>> mance-problems.html
>>>> Cheers!
>>> It also appears that Jeff fixed SDP a few weeks ago, which is more
>>> capable of reaching line-speed for IB-connected devices.
>>>
>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html
>>>
>>> Cheers!
>>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Jason W. Bacon
>> jwbacon@tds.net
>>
>> Circumstances don't make a man:
>> They reveal him.
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> _______________________________________________
>> freebsd-infiniband@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5329D338.3060103>