Date: Wed, 16 Oct 2013 15:23:28 +0000 From: Oded Shanoon <odeds@mellanox.com> To: Jason Bacon <bacon@uwm.edu> Cc: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: RE: Supported NICs Message-ID: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> In-Reply-To: <525C39C0.6050004@uwm.edu> References: <52361685.8000901@tds.net> <CAAoDvRxtKZ4AH5MTC5xi9TrXgcFr7MWGpJ3ODa-Vz0WCANkC_A@mail.gmail.com> <CAAoDvRwaDT6d8goNpvSp4jVYcSm4cZZLLqa2gCxn8EjSiLFHpA@mail.gmail.com> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Please run sysctl kern.conftxt and grep for the following options: OFED_DEBUG_INIT IPOIB_DEBUG DEADLKRES INVARIANTS INVARIANT_SUPPORT WITNESS WITNESS_SKIPSPIN These could result in performance degredation Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: Jason Bacon [mailto:bacon@uwm.edu]=20 Sent: Monday, October 14, 2013 9:37 PM To: Oded Shanoon Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org Subject: Re: Supported NICs Some initial test results... I installed an old Infinihost DDR HCA in one of our compute nodes running F= reeBSD 9.1. RHEL nodes are using qlogic IB HCAs. 10.1.1 is gigabit Ethernet, 10.1.2 is IB. Running iperf server on FreeBSD and client on one of our RHEL nodes shows O= K performance: bacon@infinibsd:/home/bacon % iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] = 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec RHEL to RHEL gives us about 8 Gbits/sec. Iperf server on RHEL and client on FreeBSD shows very poor performance for = IB, while GigE is fine: bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 ------------------------------------------------------------ Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (= default) ------------------------------------------------------------ [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 ------------------------------------------------------------ Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (= default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec Any ideas what might cause this? I'm going to install 9.2-RELEASE and retest in any case, plus explore the i= b config tools, but any feedback in the meantime would be appreciated. Thanks, JB On 10/07/13 01:51, Oded Shanoon wrote: > Hi Jason, > > IB support in 9.2 is supposed to be stable enough. > Please note that Mellanox started supporting FreeBSD only recently. The d= river in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5= .3. > Since we started our involvement we entered some fixes to that driver (th= at were submitted into 9.2). > We also mapped various issues which needs to be fixed in the future. > We are now working on a major "face lift" to the driver - making it much = more stable and with improved performance and features. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org=20 > [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Monday, September 16, 2013 4:32 PM > To: Anthony Cornehl > Cc: freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > > Thanks Anthony& Oded! > > Would you say that IB support in 9.2 is stable enough for a production HP= C file server? > > Will do plenty of my own testing first, of course. > > Regards, > > Jason > > On 09/15/13 22:38, Anthony Cornehl wrote: >> >> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"<accornehl@gmail.com=20 >> <mailto:accornehl@gmail.com>> wrote: >>> >>> On Sep 15, 2013 1:20 PM, "Jason bacon"<jwbacon@tds.net >> <mailto:jwbacon@tds.net>> wrote: >>>> >>>> Is there a list of supported IB NICs out there somewhere? >>>> >>>> I followed the wiki instructions for rebuilding with IB support >> and now have mlx4ib, mlxen, etc. >>>> Was hoping there would be man pages for the drivers that list >> known working cards, but there don't seem to be. I'm hoping to test=20 >> a file server using IPOIB and possible roll a FreeNAS ISO with IB=20 >> support if it works out. >>>> Thanks, >>>> >>>> -- >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> Jason W. Bacon >>>> jwbacon@tds.net<mailto:jwbacon@tds.net> >>>> >>>> Circumstances don't make a man: >>>> They reveal him. >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> _______________________________________________ >>>> freebsd-infiniband@freebsd.org >> <mailto:freebsd-infiniband@freebsd.org> mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>> To unsubscribe, send any mail to >> "freebsd-infiniband-unsubscribe@freebsd.org >> <mailto:freebsd-infiniband-unsubscribe@freebsd.org>" >>> Hey Jason, >>> >>> Mellanox ConnectX series cards are the only ones supported currently... >>> >>> http://www.mellanox.com/page/infiniband_cards_overview >>> >>> Don't forget to compile with IPoIB as well, since the IPoIB support >> is compiled into the driver, unlike in Linux. >>> Just be aware that IPoIB performance is also improved by enabling >> connected mode when you compile the kernel module. The IB code in=20 >> FreeBSD is a few years older the what is in Linux, but, the following=20 >> forum thread is probably relevant... >>> >> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >> r >> mance-problems.html >>> Cheers! >> It also appears that Jeff fixed SDP a few weeks ago, which is more=20 >> capable of reaching line-speed for IB-connected devices. >> >> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >> >> Cheers! >> > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Jason W. Bacon > jwbacon@tds.net > > Circumstances don't make a man: > They reveal him. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.= org" --=20 ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu -------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?058596FC127BDB469D7D60F04E37C63F73526997>