Date: Wed, 16 Oct 2013 15:23:28 +0000 From: Oded Shanoon <odeds@mellanox.com> To: Jason Bacon <bacon@uwm.edu> Cc: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: RE: Supported NICs Message-ID: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> In-Reply-To: <525C39C0.6050004@uwm.edu> References: <52361685.8000901@tds.net> <CAAoDvRxtKZ4AH5MTC5xi9TrXgcFr7MWGpJ3ODa-Vz0WCANkC_A@mail.gmail.com> <CAAoDvRwaDT6d8goNpvSp4jVYcSm4cZZLLqa2gCxn8EjSiLFHpA@mail.gmail.com> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Please run sysctl kern.conftxt and grep for the following options:
OFED_DEBUG_INIT
IPOIB_DEBUG
DEADLKRES
INVARIANTS
INVARIANT_SUPPORT
WITNESS
WITNESS_SKIPSPIN
These could result in performance degredation
Regards,
Oded Shanoon
OFED-FreeBSD Team Leader
Mellanox Technologies, Raanana
-----Original Message-----
From: Jason Bacon [mailto:bacon@uwm.edu]=20
Sent: Monday, October 14, 2013 9:37 PM
To: Oded Shanoon
Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org
Subject: Re: Supported NICs
Some initial test results...
I installed an old Infinihost DDR HCA in one of our compute nodes running F=
reeBSD 9.1.
RHEL nodes are using qlogic IB HCAs.
10.1.1 is gigabit Ethernet, 10.1.2 is IB.
Running iperf server on FreeBSD and client on one of our RHEL nodes shows O=
K performance:
bacon@infinibsd:/home/bacon % iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec
[ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] =
0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec
RHEL to RHEL gives us about 8 Gbits/sec.
Iperf server on RHEL and client on FreeBSD shows very poor performance for =
IB, while GigE is fine:
bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39
------------------------------------------------------------
Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (=
default)
------------------------------------------------------------
[ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39
------------------------------------------------------------
Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (=
default)
------------------------------------------------------------
[ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec
Any ideas what might cause this?
I'm going to install 9.2-RELEASE and retest in any case, plus explore the i=
b config tools, but any feedback in the meantime would be appreciated.
Thanks,
JB
On 10/07/13 01:51, Oded Shanoon wrote:
> Hi Jason,
>
> IB support in 9.2 is supposed to be stable enough.
> Please note that Mellanox started supporting FreeBSD only recently. The d=
river in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5=
.3.
> Since we started our involvement we entered some fixes to that driver (th=
at were submitted into 9.2).
> We also mapped various issues which needs to be fixed in the future.
> We are now working on a major "face lift" to the driver - making it much =
more stable and with improved performance and features.
>
> Regards,
>
> Oded Shanoon
> OFED-FreeBSD Team Leader
> Mellanox Technologies, Raanana
>
>
> -----Original Message-----
> From: owner-freebsd-infiniband@freebsd.org=20
> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon
> Sent: Monday, September 16, 2013 4:32 PM
> To: Anthony Cornehl
> Cc: freebsd-infiniband@freebsd.org
> Subject: Re: Supported NICs
>
>
> Thanks Anthony& Oded!
>
> Would you say that IB support in 9.2 is stable enough for a production HP=
C file server?
>
> Will do plenty of my own testing first, of course.
>
> Regards,
>
> Jason
>
> On 09/15/13 22:38, Anthony Cornehl wrote:
>>
>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"<accornehl@gmail.com=20
>> <mailto:accornehl@gmail.com>> wrote:
>>>
>>> On Sep 15, 2013 1:20 PM, "Jason bacon"<jwbacon@tds.net
>> <mailto:jwbacon@tds.net>> wrote:
>>>>
>>>> Is there a list of supported IB NICs out there somewhere?
>>>>
>>>> I followed the wiki instructions for rebuilding with IB support
>> and now have mlx4ib, mlxen, etc.
>>>> Was hoping there would be man pages for the drivers that list
>> known working cards, but there don't seem to be. I'm hoping to test=20
>> a file server using IPOIB and possible roll a FreeNAS ISO with IB=20
>> support if it works out.
>>>> Thanks,
>>>>
>>>> --
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> Jason W. Bacon
>>>> jwbacon@tds.net<mailto:jwbacon@tds.net>
>>>>
>>>> Circumstances don't make a man:
>>>> They reveal him.
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>
>>>> _______________________________________________
>>>> freebsd-infiniband@freebsd.org
>> <mailto:freebsd-infiniband@freebsd.org> mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>>>> To unsubscribe, send any mail to
>> "freebsd-infiniband-unsubscribe@freebsd.org
>> <mailto:freebsd-infiniband-unsubscribe@freebsd.org>"
>>> Hey Jason,
>>>
>>> Mellanox ConnectX series cards are the only ones supported currently...
>>>
>>> http://www.mellanox.com/page/infiniband_cards_overview
>>>
>>> Don't forget to compile with IPoIB as well, since the IPoIB support
>> is compiled into the driver, unlike in Linux.
>>> Just be aware that IPoIB performance is also improved by enabling
>> connected mode when you compile the kernel module. The IB code in=20
>> FreeBSD is a few years older the what is in Linux, but, the following=20
>> forum thread is probably relevant...
>>>
>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo
>> r
>> mance-problems.html
>>> Cheers!
>> It also appears that Jeff fixed SDP a few weeks ago, which is more=20
>> capable of reaching line-speed for IB-connected devices.
>>
>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html
>>
>> Cheers!
>>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Jason W. Bacon
> jwbacon@tds.net
>
> Circumstances don't make a man:
> They reveal him.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> _______________________________________________
> freebsd-infiniband@freebsd.org mailing list=20
> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.=
org"
--=20
-------------------------------------
Jason W. Bacon
Systems Programmer
Research Computing Support
University of Wisconsin Milwaukee
bacon@uwm.edu
-------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?058596FC127BDB469D7D60F04E37C63F73526997>