From owner-freebsd-infiniband@FreeBSD.ORG Thu Oct 17 09:01:06 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 70EA578B for ; Thu, 17 Oct 2013 09:01:06 +0000 (UTC) (envelope-from odeds@mellanox.com) Received: from eu1sys200aog125.obsmtp.com (eu1sys200aog125.obsmtp.com [207.126.144.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 464932E37 for ; Thu, 17 Oct 2013 09:01:04 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob125.postini.com ([207.126.147.11]) with SMTP ID DSNKUl+nSlLRWqabL4iipoPN/IjB5+B0NPui@postini.com; Thu, 17 Oct 2013 09:01:05 UTC Received: from MTLDAG02.mtl.com ([10.0.8.76]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Thu, 17 Oct 2013 11:49:11 +0300 From: Oded Shanoon To: Jason Bacon , Anthony Cornehl , "freebsd-infiniband@freebsd.org" Subject: RE: Supported NICs Thread-Topic: Supported NICs Thread-Index: AQHOslEQ1rGFr8lSzka7+8vcVkQcF5nHf7gAgAAFq4CAAKXOgIAgwecwgAuUqQCAA0Ip94ABAqwQ Date: Thu, 17 Oct 2013 08:49:11 +0000 Message-ID: <058596FC127BDB469D7D60F04E37C63F73527093@MTLDAG02.mtl.com> References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <525ECB42.3070908@uwm.edu> In-Reply-To: <525ECB42.3070908@uwm.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 09:01:06 -0000 Are you sure the card is connected in the slot related to the same NUMA as = cpu0? How many cpus per core do you have? Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Jason Bacon Sent: Wednesday, October 16, 2013 8:22 PM To: Anthony Cornehl; freebsd-infiniband@freebsd.org Subject: Re: Supported NICs On 10/14/13 22:53, Anthony Cornehl wrote: > > > On Oct 14, 2013 11:36 AM, "Jason Bacon" > wrote: > > > > > > Some initial test results... > > > > I installed an old Infinihost DDR HCA in one of our compute nodes > running FreeBSD 9.1. > > > > RHEL nodes are using qlogic IB HCAs. > > > > 10.1.1 is gigabit Ethernet, 10.1.2 is IB. > > > > Running iperf server on FreeBSD and client on one of our RHEL nodes > shows OK performance: > > > > bacon@infinibsd:/home/bacon % iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec > > > > [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090=20 > > [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec > > > > RHEL to RHEL gives us about 8 Gbits/sec. > > > > Iperf server on RHEL and client on FreeBSD shows very poor > performance for IB, while GigE is fine: > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8=20 > > KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0=20 > > KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > > > Any ideas what might cause this? > > > > - Does the performance change when you pin iperf to cpu0 on the=20 > FreeBSD host? > Unfortunately, no: FreeBSD infinibsd bacon ~ 39: cpuset -l 0 iperf -c 10.1.2.116 ------------------------------------------------------------ Client connecting to 10.1.2.116, TCP port 5001 TCP window size: 88.0 KByte = (default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 50193 connected with 10.1.2.116 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec For verification, "top" showed iperf consistently on CPU 0. > > (Assuming NUMA) > - Is there a MTU mismatch between the interfaces? (Assuming=20 > connected-mode is broken) > Nope: FreeBSD: ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.4.4.fe.80.0.0.0.0.0.0.0.2.c9.2.0.23.15.d1 inet 10.1.2.140 netmask 0xffffff00 broadcast 10.1.2.255 inet6 fe80::226:b9ff:fe2e:207e%ib0 prefixlen 64 scopeid 0xc nd6 options=3D29 RHEL: ib0 Link encap:InfiniBand HWaddr=20 80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:10.1.2.116 Bcast:10.1.2.255 Mask:255.255.255.0 inet6 addr: fe80::211:7500:ff:5f80/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:656228332 errors:0 dropped:0 overruns:0 frame:0 TX packets:631393406 errors:0 dropped:5026 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:1096675565577 (1021.3 GiB) TX bytes:4942093939442 (4.4 TiB) Also verified using -m flag with iperf server and client. > - What is the profile when going from FreeBSD to FreeBSD? (Assuming=20 > there's some cross-IB stack discrepancies) > Can't tell right now. We only have the one Mellanox card at the moment, so= I can only configure one FreeBSD host with IB. Our RHEL nodes are using Q= logic cards. I'll try it out as soon as I get my hands on another usable H= CA. I get better performance out of an scp transfer, so this appears to be some= sort of interaction between iperf and the IB stack. I'll play around with= some other benchmarks and report my findings. Thanks for the feedback, JB > > > I'm going to install 9.2-RELEASE and retest in any case, plus > explore the ib config tools, but any feedback in the meantime would be=20 > appreciated. > > > > Thanks, > > > > JB > > > > > > On 10/07/13 01:51, Oded Shanoon wrote: > >> > >> Hi Jason, > >> > >> IB support in 9.2 is supposed to be stable enough. > >> Please note that Mellanox started supporting FreeBSD only recently.=20 > The driver in 9.2 was ported by someone from Isilon (Jeff Roberson)=20 > from OFA-1.5.3. > >> Since we started our involvement we entered some fixes to that > driver (that were submitted into 9.2). > >> We also mapped various issues which needs to be fixed in the future. > >> We are now working on a major "face lift" to the driver - making it > much more stable and with improved performance and features. > >> > >> Regards, > >> > >> Oded Shanoon > >> OFED-FreeBSD Team Leader > >> Mellanox Technologies, Raanana > >> > >> > >> -----Original Message----- > >> From: owner-freebsd-infiniband@freebsd.org > > [mailto:owner-freebsd-infiniband@freebsd.org > ] On Behalf Of Jason=20 > Bacon > >> Sent: Monday, September 16, 2013 4:32 PM > >> To: Anthony Cornehl > >> Cc: freebsd-infiniband@freebsd.org > > >> Subject: Re: Supported NICs > >> > >> > >> Thanks Anthony& Oded! > >> > >> > >> Would you say that IB support in 9.2 is stable enough for a > production HPC file server? > >> > >> Will do plenty of my own testing first, of course. > >> > >> Regards, > >> > >> Jason > >> > >> On 09/15/13 22:38, Anthony Cornehl wrote: > >>> > >>> > >>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl" > >>> >> wrote: > >>>> > >>>> > >>>> On Sep 15, 2013 1:20 PM, "Jason bacon" > >>> > >>> >> wrote: > >>>>> > >>>>> > >>>>> Is there a list of supported IB NICs out there somewhere? > >>>>> > >>>>> I followed the wiki instructions for rebuilding with IB support > >>> > >>> and now have mlx4ib, mlxen, etc. > >>>>> > >>>>> Was hoping there would be man pages for the drivers that list > >>> > >>> known working cards, but there don't seem to be. I'm hoping to=20 > >>> test a file server using IPOIB and possible roll a FreeNAS ISO=20 > >>> with IB support if it works out. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> -- > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> Jason W. Bacon > >>>>> jwbacon@tds.net > > >>>>> > >>>>> Circumstances don't make a man: > >>>>> They reveal him. > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> > >>>>> _______________________________________________ > >>>>> freebsd-infiniband@freebsd.org > > >>> > >>> > mailing list > >>>>> > >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >>>>> To unsubscribe, send any mail to > >>> > >>> "freebsd-infiniband-unsubscribe@freebsd.org > > >>> >" > >>>> > >>>> Hey Jason, > >>>> > >>>> Mellanox ConnectX series cards are the only ones supported > currently... > >>>> > >>>> http://www.mellanox.com/page/infiniband_cards_overview > >>>> > >>>> Don't forget to compile with IPoIB as well, since the IPoIB=20 > >>>> support > >>> > >>> is compiled into the driver, unlike in Linux. > >>>> > >>>> Just be aware that IPoIB performance is also improved by enabling > >>> > >>> connected mode when you compile the kernel module. The IB code in=20 > >>> FreeBSD is a few years older the what is in Linux, but, the=20 > >>> following forum thread is probably relevant... > >>>> > >>>> > >>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-pe > >>> rfor > >>> mance-problems.html > >>>> > >>>> Cheers! > >>> > >>> It also appears that Jeff fixed SDP a few weeks ago, which is more=20 > >>> capable of reaching line-speed for IB-connected devices. > >>> > >>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > >>> > >>> Cheers! > >>> > >> > >> -- > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> Jason W. Bacon > >> jwbacon@tds.net > >> > >> Circumstances don't make a man: > >> They reveal him. > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> > >> _______________________________________________ > >> freebsd-infiniband@freebsd.org > mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >> To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org > " > > > > > > > > -- > > > > ------------------------------------- > > Jason W. Bacon > > Systems Programmer > > Research Computing Support > > University of Wisconsin Milwaukee > > bacon@uwm.edu > > ------------------------------------- > > > > > --=20 ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g"