From owner-freebsd-infiniband@FreeBSD.ORG Mon Oct 14 18:38:04 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4EA53727 for ; Mon, 14 Oct 2013 18:38:04 +0000 (UTC) (envelope-from prvs=992e47200=bacon@uwm.edu) Received: from ip4mta.uwm.edu (ip4mta.uwm.edu [129.89.7.194]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 95E7F2D93 for ; Mon, 14 Oct 2013 18:38:03 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAL04XFJFg2Tv/2dsb2JhbABZgz/CAUuBPHSCJQEBAQQBAQE1FiAKAQwECxEEAQEBCRYIBwkDAgECAQ8GAQUEFQkIBggCAwEBBAEBAQEZBIdTAxuzYQ1XiRSMY4E1gTkHBoQdA4k8i35igxiLHYh4T4ED X-UWM-First: True Message-ID: <525C39C0.6050004@uwm.edu> Date: Mon, 14 Oct 2013 13:36:48 -0500 From: Jason Bacon Organization: UW - Milwaukee User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.3) Gecko/20120327 Thunderbird/10.0.3 MIME-Version: 1.0 To: Oded Shanoon Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 18:38:04 -0000 Some initial test results... I installed an old Infinihost DDR HCA in one of our compute nodes running FreeBSD 9.1. RHEL nodes are using qlogic IB HCAs. 10.1.1 is gigabit Ethernet, 10.1.2 is IB. Running iperf server on FreeBSD and client on one of our RHEL nodes shows OK performance: bacon@infinibsd:/home/bacon % iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec RHEL to RHEL gives us about 8 Gbits/sec. Iperf server on RHEL and client on FreeBSD shows very poor performance for IB, while GigE is fine: bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 ------------------------------------------------------------ Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (default) ------------------------------------------------------------ [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 ------------------------------------------------------------ Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec Any ideas what might cause this? I'm going to install 9.2-RELEASE and retest in any case, plus explore the ib config tools, but any feedback in the meantime would be appreciated. Thanks, JB On 10/07/13 01:51, Oded Shanoon wrote: > Hi Jason, > > IB support in 9.2 is supposed to be stable enough. > Please note that Mellanox started supporting FreeBSD only recently. The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5.3. > Since we started our involvement we entered some fixes to that driver (that were submitted into 9.2). > We also mapped various issues which needs to be fixed in the future. > We are now working on a major "face lift" to the driver - making it much more stable and with improved performance and features. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Monday, September 16, 2013 4:32 PM > To: Anthony Cornehl > Cc: freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > > Thanks Anthony& Oded! > > Would you say that IB support in 9.2 is stable enough for a production HPC file server? > > Will do plenty of my own testing first, of course. > > Regards, > > Jason > > On 09/15/13 22:38, Anthony Cornehl wrote: >> >> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"> > wrote: >>> >>> On Sep 15, 2013 1:20 PM, "Jason bacon"> > wrote: >>>> >>>> Is there a list of supported IB NICs out there somewhere? >>>> >>>> I followed the wiki instructions for rebuilding with IB support >> and now have mlx4ib, mlxen, etc. >>>> Was hoping there would be man pages for the drivers that list >> known working cards, but there don't seem to be. I'm hoping to test a >> file server using IPOIB and possible roll a FreeNAS ISO with IB >> support if it works out. >>>> Thanks, >>>> >>>> -- >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> Jason W. Bacon >>>> jwbacon@tds.net >>>> >>>> Circumstances don't make a man: >>>> They reveal him. >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> _______________________________________________ >>>> freebsd-infiniband@freebsd.org >> mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>> To unsubscribe, send any mail to >> "freebsd-infiniband-unsubscribe@freebsd.org >> " >>> Hey Jason, >>> >>> Mellanox ConnectX series cards are the only ones supported currently... >>> >>> http://www.mellanox.com/page/infiniband_cards_overview >>> >>> Don't forget to compile with IPoIB as well, since the IPoIB support >> is compiled into the driver, unlike in Linux. >>> Just be aware that IPoIB performance is also improved by enabling >> connected mode when you compile the kernel module. The IB code in >> FreeBSD is a few years older the what is in Linux, but, the following >> forum thread is probably relevant... >>> >> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfor >> mance-problems.html >>> Cheers! >> It also appears that Jeff fixed SDP a few weeks ago, which is more >> capable of reaching line-speed for IB-connected devices. >> >> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >> >> Cheers! >> > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Jason W. Bacon > jwbacon@tds.net > > Circumstances don't make a man: > They reveal him. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" -- ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- From owner-freebsd-infiniband@FreeBSD.ORG Wed Oct 16 15:23:43 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E94C973C for ; Wed, 16 Oct 2013 15:23:43 +0000 (UTC) (envelope-from odeds@mellanox.com) Received: from eu1sys200aog125.obsmtp.com (eu1sys200aog125.obsmtp.com [207.126.144.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 904322666 for ; Wed, 16 Oct 2013 15:23:39 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob125.postini.com ([207.126.147.11]) with SMTP ID DSNKUl6vclexq9DuWBbRZ3+vgW7oKkJWApqM@postini.com; Wed, 16 Oct 2013 15:23:39 UTC Received: from MTLDAG02.mtl.com ([10.0.8.76]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Wed, 16 Oct 2013 18:23:28 +0300 From: Oded Shanoon To: Jason Bacon Subject: RE: Supported NICs Thread-Topic: Supported NICs Thread-Index: AQHOslEQ1rGFr8lSzka7+8vcVkQcF5nHf7gAgAAFq4CAAKXOgIAgwecwgAuUqQCAAyCJ0A== Date: Wed, 16 Oct 2013 15:23:28 +0000 Message-ID: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> In-Reply-To: <525C39C0.6050004@uwm.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 15:23:44 -0000 Please run sysctl kern.conftxt and grep for the following options: OFED_DEBUG_INIT IPOIB_DEBUG DEADLKRES INVARIANTS INVARIANT_SUPPORT WITNESS WITNESS_SKIPSPIN These could result in performance degredation Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: Jason Bacon [mailto:bacon@uwm.edu]=20 Sent: Monday, October 14, 2013 9:37 PM To: Oded Shanoon Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org Subject: Re: Supported NICs Some initial test results... I installed an old Infinihost DDR HCA in one of our compute nodes running F= reeBSD 9.1. RHEL nodes are using qlogic IB HCAs. 10.1.1 is gigabit Ethernet, 10.1.2 is IB. Running iperf server on FreeBSD and client on one of our RHEL nodes shows O= K performance: bacon@infinibsd:/home/bacon % iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] = 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec RHEL to RHEL gives us about 8 Gbits/sec. Iperf server on RHEL and client on FreeBSD shows very poor performance for = IB, while GigE is fine: bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 ------------------------------------------------------------ Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (= default) ------------------------------------------------------------ [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 ------------------------------------------------------------ Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (= default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec Any ideas what might cause this? I'm going to install 9.2-RELEASE and retest in any case, plus explore the i= b config tools, but any feedback in the meantime would be appreciated. Thanks, JB On 10/07/13 01:51, Oded Shanoon wrote: > Hi Jason, > > IB support in 9.2 is supposed to be stable enough. > Please note that Mellanox started supporting FreeBSD only recently. The d= river in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5= .3. > Since we started our involvement we entered some fixes to that driver (th= at were submitted into 9.2). > We also mapped various issues which needs to be fixed in the future. > We are now working on a major "face lift" to the driver - making it much = more stable and with improved performance and features. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org=20 > [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Monday, September 16, 2013 4:32 PM > To: Anthony Cornehl > Cc: freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > > Thanks Anthony& Oded! > > Would you say that IB support in 9.2 is stable enough for a production HP= C file server? > > Will do plenty of my own testing first, of course. > > Regards, > > Jason > > On 09/15/13 22:38, Anthony Cornehl wrote: >> >> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"> > wrote: >>> >>> On Sep 15, 2013 1:20 PM, "Jason bacon"> > wrote: >>>> >>>> Is there a list of supported IB NICs out there somewhere? >>>> >>>> I followed the wiki instructions for rebuilding with IB support >> and now have mlx4ib, mlxen, etc. >>>> Was hoping there would be man pages for the drivers that list >> known working cards, but there don't seem to be. I'm hoping to test=20 >> a file server using IPOIB and possible roll a FreeNAS ISO with IB=20 >> support if it works out. >>>> Thanks, >>>> >>>> -- >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> Jason W. Bacon >>>> jwbacon@tds.net >>>> >>>> Circumstances don't make a man: >>>> They reveal him. >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> _______________________________________________ >>>> freebsd-infiniband@freebsd.org >> mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>> To unsubscribe, send any mail to >> "freebsd-infiniband-unsubscribe@freebsd.org >> " >>> Hey Jason, >>> >>> Mellanox ConnectX series cards are the only ones supported currently... >>> >>> http://www.mellanox.com/page/infiniband_cards_overview >>> >>> Don't forget to compile with IPoIB as well, since the IPoIB support >> is compiled into the driver, unlike in Linux. >>> Just be aware that IPoIB performance is also improved by enabling >> connected mode when you compile the kernel module. The IB code in=20 >> FreeBSD is a few years older the what is in Linux, but, the following=20 >> forum thread is probably relevant... >>> >> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >> r >> mance-problems.html >>> Cheers! >> It also appears that Jeff fixed SDP a few weeks ago, which is more=20 >> capable of reaching line-speed for IB-connected devices. >> >> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >> >> Cheers! >> > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Jason W. Bacon > jwbacon@tds.net > > Circumstances don't make a man: > They reveal him. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.= org" --=20 ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- From owner-freebsd-infiniband@FreeBSD.ORG Wed Oct 16 16:47:13 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DF5495F9 for ; Wed, 16 Oct 2013 16:47:13 +0000 (UTC) (envelope-from prvs=994424ef5=bacon@uwm.edu) Received: from ip4mta.uwm.edu (ip4mta.uwm.edu [129.89.7.194]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8A0902B65 for ; Wed, 16 Oct 2013 16:47:13 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAOrBXlKBWSZL/2dsb2JhbABagz/CFkuBNHSCJQEBAQQBAQE1FiAKAQwECxEEAQEBCRYIBwkDAgECAQ8GAQkVCQgGCAIDAQEEAQEBARkEh1MDG7VGDVeJFIxjgTWBOQcGhB8DiTyLfWKDGIsdiHhPgQM X-UWM-First: True Message-ID: <525EC30D.6070107@uwm.edu> Date: Wed, 16 Oct 2013 11:47:09 -0500 From: Jason Bacon Organization: UW - Milwaukee User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:10.0.3) Gecko/20120327 Thunderbird/10.0.3 MIME-Version: 1.0 To: Oded Shanoon Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 16:47:13 -0000 Good timing: just finished a new round of testing on 9.2-RELEASE. No discernible difference from 9.1. Throughput was a bit slower both BSD-Linux and Linux-Linux, probably due to the high load on the cluster today. FreeBSD infinibsd bacon ~ 36: sysctl kern.conftxt | egrep 'OFED|IPOIB|DEAD|INVARIANT|WITNESS' options IPOIB_CM options OFED Thanks, JB On 10/16/13 10:23, Oded Shanoon wrote: > Please run sysctl kern.conftxt and grep for the following options: > > OFED_DEBUG_INIT > IPOIB_DEBUG > DEADLKRES > INVARIANTS > INVARIANT_SUPPORT > WITNESS > WITNESS_SKIPSPIN > > These could result in performance degredation > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: Jason Bacon [mailto:bacon@uwm.edu] > Sent: Monday, October 14, 2013 9:37 PM > To: Oded Shanoon > Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > > Some initial test results... > > I installed an old Infinihost DDR HCA in one of our compute nodes running FreeBSD 9.1. > > RHEL nodes are using qlogic IB HCAs. > > 10.1.1 is gigabit Ethernet, 10.1.2 is IB. > > Running iperf server on FreeBSD and client on one of our RHEL nodes shows OK performance: > > bacon@infinibsd:/home/bacon % iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec > > [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec > > RHEL to RHEL gives us about 8 Gbits/sec. > > Iperf server on RHEL and client on FreeBSD shows very poor performance for IB, while GigE is fine: > > bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 > ------------------------------------------------------------ > Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec > > bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 > ------------------------------------------------------------ > Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > Any ideas what might cause this? > > I'm going to install 9.2-RELEASE and retest in any case, plus explore the ib config tools, but any feedback in the meantime would be appreciated. > > Thanks, > > JB > > On 10/07/13 01:51, Oded Shanoon wrote: >> Hi Jason, >> >> IB support in 9.2 is supposed to be stable enough. >> Please note that Mellanox started supporting FreeBSD only recently. The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5.3. >> Since we started our involvement we entered some fixes to that driver (that were submitted into 9.2). >> We also mapped various issues which needs to be fixed in the future. >> We are now working on a major "face lift" to the driver - making it much more stable and with improved performance and features. >> >> Regards, >> >> Oded Shanoon >> OFED-FreeBSD Team Leader >> Mellanox Technologies, Raanana >> >> >> -----Original Message----- >> From: owner-freebsd-infiniband@freebsd.org >> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon >> Sent: Monday, September 16, 2013 4:32 PM >> To: Anthony Cornehl >> Cc: freebsd-infiniband@freebsd.org >> Subject: Re: Supported NICs >> >> >> Thanks Anthony& Oded! >> >> Would you say that IB support in 9.2 is stable enough for a production HPC file server? >> >> Will do plenty of my own testing first, of course. >> >> Regards, >> >> Jason >> >> On 09/15/13 22:38, Anthony Cornehl wrote: >>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl">> > wrote: >>>> On Sep 15, 2013 1:20 PM, "Jason bacon">> > wrote: >>>>> Is there a list of supported IB NICs out there somewhere? >>>>> >>>>> I followed the wiki instructions for rebuilding with IB support >>> and now have mlx4ib, mlxen, etc. >>>>> Was hoping there would be man pages for the drivers that list >>> known working cards, but there don't seem to be. I'm hoping to test >>> a file server using IPOIB and possible roll a FreeNAS ISO with IB >>> support if it works out. >>>>> Thanks, >>>>> >>>>> -- >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> Jason W. Bacon >>>>> jwbacon@tds.net >>>>> >>>>> Circumstances don't make a man: >>>>> They reveal him. >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> _______________________________________________ >>>>> freebsd-infiniband@freebsd.org >>> mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>>> To unsubscribe, send any mail to >>> "freebsd-infiniband-unsubscribe@freebsd.org >>> " >>>> Hey Jason, >>>> >>>> Mellanox ConnectX series cards are the only ones supported currently... >>>> >>>> http://www.mellanox.com/page/infiniband_cards_overview >>>> >>>> Don't forget to compile with IPoIB as well, since the IPoIB support >>> is compiled into the driver, unlike in Linux. >>>> Just be aware that IPoIB performance is also improved by enabling >>> connected mode when you compile the kernel module. The IB code in >>> FreeBSD is a few years older the what is in Linux, but, the following >>> forum thread is probably relevant... >>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >>> r >>> mance-problems.html >>>> Cheers! >>> It also appears that Jeff fixed SDP a few weeks ago, which is more >>> capable of reaching line-speed for IB-connected devices. >>> >>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >>> >>> Cheers! >>> >> -- >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Jason W. Bacon >> jwbacon@tds.net >> >> Circumstances don't make a man: >> They reveal him. >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> _______________________________________________ >> freebsd-infiniband@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" > -- ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- From owner-freebsd-infiniband@FreeBSD.ORG Wed Oct 16 17:22:08 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D854F19F for ; Wed, 16 Oct 2013 17:22:08 +0000 (UTC) (envelope-from prvs=994424ef5=bacon@uwm.edu) Received: from ip4mta.uwm.edu (ip4mta.uwm.edu [129.89.7.194]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 593922D99 for ; Wed, 16 Oct 2013 17:22:07 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEABzKXlKBWSZL/2dsb2JhbABagz+De71vS4E0dIIlAQEBBAEBASAVFhkHCg0ECw4DBAEBAQICBRYIAwICCQMCAQIBDwYBCRUJCAYBBwIDAgQBAQEBGQSHUwMbrHqIVw1XiRSBKYs6gTWBQAaCZIE7A4k8i31igxiLHYh4T4ED X-UWM-First: True Message-ID: <525ECB42.3070908@uwm.edu> Date: Wed, 16 Oct 2013 12:22:10 -0500 From: Jason Bacon Organization: UW - Milwaukee User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:10.0.3) Gecko/20120327 Thunderbird/10.0.3 MIME-Version: 1.0 To: Anthony Cornehl , "freebsd-infiniband@freebsd.org" Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 17:22:08 -0000 On 10/14/13 22:53, Anthony Cornehl wrote: > > > On Oct 14, 2013 11:36 AM, "Jason Bacon" > wrote: > > > > > > Some initial test results... > > > > I installed an old Infinihost DDR HCA in one of our compute nodes > running FreeBSD 9.1. > > > > RHEL nodes are using qlogic IB HCAs. > > > > 10.1.1 is gigabit Ethernet, 10.1.2 is IB. > > > > Running iperf server on FreeBSD and client on one of our RHEL nodes > shows OK performance: > > > > bacon@infinibsd:/home/bacon % iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec > > > > [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 > > [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec > > > > RHEL to RHEL gives us about 8 Gbits/sec. > > > > Iperf server on RHEL and client on FreeBSD shows very poor > performance for IB, while GigE is fine: > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.1.39, TCP port 5001 > > TCP window size: 32.8 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.2.39, TCP port 5001 > > TCP window size: 32.0 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > > > Any ideas what might cause this? > > > > - Does the performance change when you pin iperf to cpu0 on the > FreeBSD host? > Unfortunately, no: FreeBSD infinibsd bacon ~ 39: cpuset -l 0 iperf -c 10.1.2.116 ------------------------------------------------------------ Client connecting to 10.1.2.116, TCP port 5001 TCP window size: 88.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 50193 connected with 10.1.2.116 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec For verification, "top" showed iperf consistently on CPU 0. > > (Assuming NUMA) > - Is there a MTU mismatch between the interfaces? (Assuming > connected-mode is broken) > Nope: FreeBSD: ib0: flags=8043 metric 0 mtu 65520 options=80018 lladdr 80.0.4.4.fe.80.0.0.0.0.0.0.0.2.c9.2.0.23.15.d1 inet 10.1.2.140 netmask 0xffffff00 broadcast 10.1.2.255 inet6 fe80::226:b9ff:fe2e:207e%ib0 prefixlen 64 scopeid 0xc nd6 options=29 RHEL: ib0 Link encap:InfiniBand HWaddr 80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:10.1.2.116 Bcast:10.1.2.255 Mask:255.255.255.0 inet6 addr: fe80::211:7500:ff:5f80/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:656228332 errors:0 dropped:0 overruns:0 frame:0 TX packets:631393406 errors:0 dropped:5026 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:1096675565577 (1021.3 GiB) TX bytes:4942093939442 (4.4 TiB) Also verified using -m flag with iperf server and client. > - What is the profile when going from FreeBSD to FreeBSD? (Assuming > there's some cross-IB stack discrepancies) > Can't tell right now. We only have the one Mellanox card at the moment, so I can only configure one FreeBSD host with IB. Our RHEL nodes are using Qlogic cards. I'll try it out as soon as I get my hands on another usable HCA. I get better performance out of an scp transfer, so this appears to be some sort of interaction between iperf and the IB stack. I'll play around with some other benchmarks and report my findings. Thanks for the feedback, JB > > > I'm going to install 9.2-RELEASE and retest in any case, plus > explore the ib config tools, but any feedback in the meantime would be > appreciated. > > > > Thanks, > > > > JB > > > > > > On 10/07/13 01:51, Oded Shanoon wrote: > >> > >> Hi Jason, > >> > >> IB support in 9.2 is supposed to be stable enough. > >> Please note that Mellanox started supporting FreeBSD only recently. > The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) > from OFA-1.5.3. > >> Since we started our involvement we entered some fixes to that > driver (that were submitted into 9.2). > >> We also mapped various issues which needs to be fixed in the future. > >> We are now working on a major "face lift" to the driver - making it > much more stable and with improved performance and features. > >> > >> Regards, > >> > >> Oded Shanoon > >> OFED-FreeBSD Team Leader > >> Mellanox Technologies, Raanana > >> > >> > >> -----Original Message----- > >> From: owner-freebsd-infiniband@freebsd.org > > [mailto:owner-freebsd-infiniband@freebsd.org > ] On Behalf Of Jason Bacon > >> Sent: Monday, September 16, 2013 4:32 PM > >> To: Anthony Cornehl > >> Cc: freebsd-infiniband@freebsd.org > > >> Subject: Re: Supported NICs > >> > >> > >> Thanks Anthony& Oded! > >> > >> > >> Would you say that IB support in 9.2 is stable enough for a > production HPC file server? > >> > >> Will do plenty of my own testing first, of course. > >> > >> Regards, > >> > >> Jason > >> > >> On 09/15/13 22:38, Anthony Cornehl wrote: > >>> > >>> > >>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl" > >>> >> wrote: > >>>> > >>>> > >>>> On Sep 15, 2013 1:20 PM, "Jason bacon" > >>> > >>> >> wrote: > >>>>> > >>>>> > >>>>> Is there a list of supported IB NICs out there somewhere? > >>>>> > >>>>> I followed the wiki instructions for rebuilding with IB support > >>> > >>> and now have mlx4ib, mlxen, etc. > >>>>> > >>>>> Was hoping there would be man pages for the drivers that list > >>> > >>> known working cards, but there don't seem to be. I'm hoping to test a > >>> file server using IPOIB and possible roll a FreeNAS ISO with IB > >>> support if it works out. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> -- > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> Jason W. Bacon > >>>>> jwbacon@tds.net > > >>>>> > >>>>> Circumstances don't make a man: > >>>>> They reveal him. > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> > >>>>> _______________________________________________ > >>>>> freebsd-infiniband@freebsd.org > > >>> > >>> > mailing list > >>>>> > >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >>>>> To unsubscribe, send any mail to > >>> > >>> "freebsd-infiniband-unsubscribe@freebsd.org > > >>> >" > >>>> > >>>> Hey Jason, > >>>> > >>>> Mellanox ConnectX series cards are the only ones supported > currently... > >>>> > >>>> http://www.mellanox.com/page/infiniband_cards_overview > >>>> > >>>> Don't forget to compile with IPoIB as well, since the IPoIB support > >>> > >>> is compiled into the driver, unlike in Linux. > >>>> > >>>> Just be aware that IPoIB performance is also improved by enabling > >>> > >>> connected mode when you compile the kernel module. The IB code in > >>> FreeBSD is a few years older the what is in Linux, but, the following > >>> forum thread is probably relevant... > >>>> > >>>> > >>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfor > >>> mance-problems.html > >>>> > >>>> Cheers! > >>> > >>> It also appears that Jeff fixed SDP a few weeks ago, which is more > >>> capable of reaching line-speed for IB-connected devices. > >>> > >>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > >>> > >>> Cheers! > >>> > >> > >> -- > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> Jason W. Bacon > >> jwbacon@tds.net > >> > >> Circumstances don't make a man: > >> They reveal him. > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> > >> _______________________________________________ > >> freebsd-infiniband@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >> To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org > " > > > > > > > > -- > > > > ------------------------------------- > > Jason W. Bacon > > Systems Programmer > > Research Computing Support > > University of Wisconsin Milwaukee > > bacon@uwm.edu > > ------------------------------------- > > > > > -- ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- From owner-freebsd-infiniband@FreeBSD.ORG Thu Oct 17 09:01:06 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 70EA578B for ; Thu, 17 Oct 2013 09:01:06 +0000 (UTC) (envelope-from odeds@mellanox.com) Received: from eu1sys200aog125.obsmtp.com (eu1sys200aog125.obsmtp.com [207.126.144.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 464932E37 for ; Thu, 17 Oct 2013 09:01:04 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob125.postini.com ([207.126.147.11]) with SMTP ID DSNKUl+nSlLRWqabL4iipoPN/IjB5+B0NPui@postini.com; Thu, 17 Oct 2013 09:01:05 UTC Received: from MTLDAG02.mtl.com ([10.0.8.76]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Thu, 17 Oct 2013 11:49:11 +0300 From: Oded Shanoon To: Jason Bacon , Anthony Cornehl , "freebsd-infiniband@freebsd.org" Subject: RE: Supported NICs Thread-Topic: Supported NICs Thread-Index: AQHOslEQ1rGFr8lSzka7+8vcVkQcF5nHf7gAgAAFq4CAAKXOgIAgwecwgAuUqQCAA0Ip94ABAqwQ Date: Thu, 17 Oct 2013 08:49:11 +0000 Message-ID: <058596FC127BDB469D7D60F04E37C63F73527093@MTLDAG02.mtl.com> References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <525ECB42.3070908@uwm.edu> In-Reply-To: <525ECB42.3070908@uwm.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 09:01:06 -0000 Are you sure the card is connected in the slot related to the same NUMA as = cpu0? How many cpus per core do you have? Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Jason Bacon Sent: Wednesday, October 16, 2013 8:22 PM To: Anthony Cornehl; freebsd-infiniband@freebsd.org Subject: Re: Supported NICs On 10/14/13 22:53, Anthony Cornehl wrote: > > > On Oct 14, 2013 11:36 AM, "Jason Bacon" > wrote: > > > > > > Some initial test results... > > > > I installed an old Infinihost DDR HCA in one of our compute nodes > running FreeBSD 9.1. > > > > RHEL nodes are using qlogic IB HCAs. > > > > 10.1.1 is gigabit Ethernet, 10.1.2 is IB. > > > > Running iperf server on FreeBSD and client on one of our RHEL nodes > shows OK performance: > > > > bacon@infinibsd:/home/bacon % iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec > > > > [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090=20 > > [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec > > > > RHEL to RHEL gives us about 8 Gbits/sec. > > > > Iperf server on RHEL and client on FreeBSD shows very poor > performance for IB, while GigE is fine: > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8=20 > > KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec > > > > bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 > > ------------------------------------------------------------ > > Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0=20 > > KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > > > Any ideas what might cause this? > > > > - Does the performance change when you pin iperf to cpu0 on the=20 > FreeBSD host? > Unfortunately, no: FreeBSD infinibsd bacon ~ 39: cpuset -l 0 iperf -c 10.1.2.116 ------------------------------------------------------------ Client connecting to 10.1.2.116, TCP port 5001 TCP window size: 88.0 KByte = (default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 50193 connected with 10.1.2.116 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec For verification, "top" showed iperf consistently on CPU 0. > > (Assuming NUMA) > - Is there a MTU mismatch between the interfaces? (Assuming=20 > connected-mode is broken) > Nope: FreeBSD: ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.4.4.fe.80.0.0.0.0.0.0.0.2.c9.2.0.23.15.d1 inet 10.1.2.140 netmask 0xffffff00 broadcast 10.1.2.255 inet6 fe80::226:b9ff:fe2e:207e%ib0 prefixlen 64 scopeid 0xc nd6 options=3D29 RHEL: ib0 Link encap:InfiniBand HWaddr=20 80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:10.1.2.116 Bcast:10.1.2.255 Mask:255.255.255.0 inet6 addr: fe80::211:7500:ff:5f80/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:656228332 errors:0 dropped:0 overruns:0 frame:0 TX packets:631393406 errors:0 dropped:5026 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:1096675565577 (1021.3 GiB) TX bytes:4942093939442 (4.4 TiB) Also verified using -m flag with iperf server and client. > - What is the profile when going from FreeBSD to FreeBSD? (Assuming=20 > there's some cross-IB stack discrepancies) > Can't tell right now. We only have the one Mellanox card at the moment, so= I can only configure one FreeBSD host with IB. Our RHEL nodes are using Q= logic cards. I'll try it out as soon as I get my hands on another usable H= CA. I get better performance out of an scp transfer, so this appears to be some= sort of interaction between iperf and the IB stack. I'll play around with= some other benchmarks and report my findings. Thanks for the feedback, JB > > > I'm going to install 9.2-RELEASE and retest in any case, plus > explore the ib config tools, but any feedback in the meantime would be=20 > appreciated. > > > > Thanks, > > > > JB > > > > > > On 10/07/13 01:51, Oded Shanoon wrote: > >> > >> Hi Jason, > >> > >> IB support in 9.2 is supposed to be stable enough. > >> Please note that Mellanox started supporting FreeBSD only recently.=20 > The driver in 9.2 was ported by someone from Isilon (Jeff Roberson)=20 > from OFA-1.5.3. > >> Since we started our involvement we entered some fixes to that > driver (that were submitted into 9.2). > >> We also mapped various issues which needs to be fixed in the future. > >> We are now working on a major "face lift" to the driver - making it > much more stable and with improved performance and features. > >> > >> Regards, > >> > >> Oded Shanoon > >> OFED-FreeBSD Team Leader > >> Mellanox Technologies, Raanana > >> > >> > >> -----Original Message----- > >> From: owner-freebsd-infiniband@freebsd.org > > [mailto:owner-freebsd-infiniband@freebsd.org > ] On Behalf Of Jason=20 > Bacon > >> Sent: Monday, September 16, 2013 4:32 PM > >> To: Anthony Cornehl > >> Cc: freebsd-infiniband@freebsd.org > > >> Subject: Re: Supported NICs > >> > >> > >> Thanks Anthony& Oded! > >> > >> > >> Would you say that IB support in 9.2 is stable enough for a > production HPC file server? > >> > >> Will do plenty of my own testing first, of course. > >> > >> Regards, > >> > >> Jason > >> > >> On 09/15/13 22:38, Anthony Cornehl wrote: > >>> > >>> > >>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl" > >>> >> wrote: > >>>> > >>>> > >>>> On Sep 15, 2013 1:20 PM, "Jason bacon" > >>> > >>> >> wrote: > >>>>> > >>>>> > >>>>> Is there a list of supported IB NICs out there somewhere? > >>>>> > >>>>> I followed the wiki instructions for rebuilding with IB support > >>> > >>> and now have mlx4ib, mlxen, etc. > >>>>> > >>>>> Was hoping there would be man pages for the drivers that list > >>> > >>> known working cards, but there don't seem to be. I'm hoping to=20 > >>> test a file server using IPOIB and possible roll a FreeNAS ISO=20 > >>> with IB support if it works out. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> -- > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> Jason W. Bacon > >>>>> jwbacon@tds.net > > >>>>> > >>>>> Circumstances don't make a man: > >>>>> They reveal him. > >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>> > >>>>> _______________________________________________ > >>>>> freebsd-infiniband@freebsd.org > > >>> > >>> > mailing list > >>>>> > >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >>>>> To unsubscribe, send any mail to > >>> > >>> "freebsd-infiniband-unsubscribe@freebsd.org > > >>> >" > >>>> > >>>> Hey Jason, > >>>> > >>>> Mellanox ConnectX series cards are the only ones supported > currently... > >>>> > >>>> http://www.mellanox.com/page/infiniband_cards_overview > >>>> > >>>> Don't forget to compile with IPoIB as well, since the IPoIB=20 > >>>> support > >>> > >>> is compiled into the driver, unlike in Linux. > >>>> > >>>> Just be aware that IPoIB performance is also improved by enabling > >>> > >>> connected mode when you compile the kernel module. The IB code in=20 > >>> FreeBSD is a few years older the what is in Linux, but, the=20 > >>> following forum thread is probably relevant... > >>>> > >>>> > >>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-pe > >>> rfor > >>> mance-problems.html > >>>> > >>>> Cheers! > >>> > >>> It also appears that Jeff fixed SDP a few weeks ago, which is more=20 > >>> capable of reaching line-speed for IB-connected devices. > >>> > >>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > >>> > >>> Cheers! > >>> > >> > >> -- > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> Jason W. Bacon > >> jwbacon@tds.net > >> > >> Circumstances don't make a man: > >> They reveal him. > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> > >> _______________________________________________ > >> freebsd-infiniband@freebsd.org > mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > >> To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org > " > > > > > > > > -- > > > > ------------------------------------- > > Jason W. Bacon > > Systems Programmer > > Research Computing Support > > University of Wisconsin Milwaukee > > bacon@uwm.edu > > ------------------------------------- > > > > > --=20 ------------------------------------- Jason W. Bacon Systems Programmer Research Computing Support University of Wisconsin Milwaukee bacon@uwm.edu ------------------------------------- _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@FreeBSD.ORG Thu Oct 17 13:25:21 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6C4D3452 for ; Thu, 17 Oct 2013 13:25:21 +0000 (UTC) (envelope-from jwbacon@tds.net) Received: from mail-ie0-f170.google.com (mail-ie0-f170.google.com [209.85.223.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3574B2F5B for ; Thu, 17 Oct 2013 13:25:17 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id at1so3983008iec.29 for ; Thu, 17 Oct 2013 06:25:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=MYNpXGxzgb/tUF10w2+0zSznUPttVftOMhR8bjuACeI=; b=OSzsoh4q3huh3A7BTVBSz7xKzklguGWPK2H7Ppy9xH/knwQujSS1c9WvNQ9qPSkoYk R1zvRWOgf0EL/3Vp1n8C9cosDL2AM/2lB8ajiRDpXIM4dcbiJ/fslE/GwPdruH7lcNzf DYgORH7w0lLMAkiA/EPFI5K9lqrpBKOeOWTk75GHzGzD6kcwUkKPbIOekXWYj2wfJ0D+ fmF1nve34fu1/iR0ugfrukInxoytRKhj90oWIoRIPB9ArdnK2ubtNo+zyFNZbfyXKfPY afy8Zg8JghzdMIqCmaJ8Wgw9BVX6mBMIrmxnPualXBasRcU2KN6xZ6OzIqbVPfNEpJRK P8Lg== X-Gm-Message-State: ALoCoQkIUrlMrx1GqK1YHLVhsHTirw1evRK/wrQtit5s/qF7o5qIYgiP3UScSgNmHNbml2kqZELP X-Received: by 10.50.36.5 with SMTP id m5mr26060673igj.3.1382016310894; Thu, 17 Oct 2013 06:25:10 -0700 (PDT) Received: from [192.168.0.52] (h69-131-100-239.nwblwi.broadband.dynamic.tds.net. [69.131.100.239]) by mx.google.com with ESMTPSA id m1sm8532148igj.10.2013.10.17.06.25.09 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Oct 2013 06:25:10 -0700 (PDT) Message-ID: <525FE535.6000603@tds.net> Date: Thu, 17 Oct 2013 08:25:09 -0500 From: Jason bacon User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.20) Gecko/20110804 Thunderbird/3.1.12 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <525ECB42.3070908@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73527093@MTLDAG02.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F73527093@MTLDAG02.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 13:25:21 -0000 It's a Dell Poweredge r410, which is 1U. Only one PCIe slot. I'd have to go into the BIOS settings to see if NUMA is even enabled. There's a setting that toggles between enabling interleaving and NUMA. I suspect we have interleaving enabled. In any case, I configured and tuned an NFS server on the BSD machine and was able to saturate the local hard disk on NFS reads from a CentOS client, so I know ib0 can do at least 1.2 gigabit/sec outgoing, much better than it did with iperf. I'm going to repeat the test with a RAM disk and eventually with a good RAID to see what it can really do for NFS. That's what we're ultimately interested in. I'll still try and figure out what's going on with iperf. It seems to be an anomaly, but it must be indicating *something* that should probably be addressed. I also plan to try a better HCA than this old Infinihost I scrounged up. Motorola is giving us a rack full of servers with IB in the next couple of weeks. I don't know the details yet, but I'm hoping it has ConnectX cards... Thanks again for your assistance, JB On 10/17/13 3:49 AM, Oded Shanoon wrote: > Are you sure the card is connected in the slot related to the same NUMA as cpu0? > How many cpus per core do you have? > > > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Wednesday, October 16, 2013 8:22 PM > To: Anthony Cornehl; freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > On 10/14/13 22:53, Anthony Cornehl wrote: >> >> On Oct 14, 2013 11:36 AM, "Jason Bacon"> > wrote: >>> >>> Some initial test results... >>> >>> I installed an old Infinihost DDR HCA in one of our compute nodes >> running FreeBSD 9.1. >>> RHEL nodes are using qlogic IB HCAs. >>> >>> 10.1.1 is gigabit Ethernet, 10.1.2 is IB. >>> >>> Running iperf server on FreeBSD and client on one of our RHEL nodes >> shows OK performance: >>> bacon@infinibsd:/home/bacon % iperf -s >>> ------------------------------------------------------------ >>> Server listening on TCP port 5001 >>> TCP window size: 64.0 KByte (default) >>> ------------------------------------------------------------ >>> [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 >>> [ ID] Interval Transfer Bandwidth >>> [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec >>> >>> [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 >>> [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec >>> >>> RHEL to RHEL gives us about 8 Gbits/sec. >>> >>> Iperf server on RHEL and client on FreeBSD shows very poor >> performance for IB, while GigE is fine: >>> bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 >>> ------------------------------------------------------------ >>> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 >>> KByte (default) >>> ------------------------------------------------------------ >>> [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 >>> [ ID] Interval Transfer Bandwidth >>> [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec >>> >>> bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 >>> ------------------------------------------------------------ >>> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 >>> KByte (default) >>> ------------------------------------------------------------ >>> [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 >>> [ ID] Interval Transfer Bandwidth >>> [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec >>> >>> Any ideas what might cause this? >>> >> - Does the performance change when you pin iperf to cpu0 on the >> FreeBSD host? >> > Unfortunately, no: > > FreeBSD infinibsd bacon ~ 39: cpuset -l 0 iperf -c 10.1.2.116 > ------------------------------------------------------------ > Client connecting to 10.1.2.116, TCP port 5001 TCP window size: 88.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.2.140 port 50193 connected with 10.1.2.116 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > For verification, "top" showed iperf consistently on CPU 0. >> (Assuming NUMA) >> - Is there a MTU mismatch between the interfaces? (Assuming >> connected-mode is broken) >> > Nope: > > FreeBSD: > > ib0: flags=8043 metric 0 mtu 65520 > options=80018 > lladdr 80.0.4.4.fe.80.0.0.0.0.0.0.0.2.c9.2.0.23.15.d1 > inet 10.1.2.140 netmask 0xffffff00 broadcast 10.1.2.255 > inet6 fe80::226:b9ff:fe2e:207e%ib0 prefixlen 64 scopeid 0xc > nd6 options=29 > > RHEL: > > ib0 Link encap:InfiniBand HWaddr > 80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 > inet addr:10.1.2.116 Bcast:10.1.2.255 Mask:255.255.255.0 > inet6 addr: fe80::211:7500:ff:5f80/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 > RX packets:656228332 errors:0 dropped:0 overruns:0 frame:0 > TX packets:631393406 errors:0 dropped:5026 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:1096675565577 (1021.3 GiB) TX bytes:4942093939442 > (4.4 TiB) > > Also verified using -m flag with iperf server and client. > >> - What is the profile when going from FreeBSD to FreeBSD? (Assuming >> there's some cross-IB stack discrepancies) >> > Can't tell right now. We only have the one Mellanox card at the moment, so I can only configure one FreeBSD host with IB. Our RHEL nodes are using Qlogic cards. I'll try it out as soon as I get my hands on another usable HCA. > > I get better performance out of an scp transfer, so this appears to be some sort of interaction between iperf and the IB stack. I'll play around with some other benchmarks and report my findings. > > Thanks for the feedback, > > JB >>> I'm going to install 9.2-RELEASE and retest in any case, plus >> explore the ib config tools, but any feedback in the meantime would be >> appreciated. >>> Thanks, >>> >>> JB >>> >>> >>> On 10/07/13 01:51, Oded Shanoon wrote: >>>> Hi Jason, >>>> >>>> IB support in 9.2 is supposed to be stable enough. >>>> Please note that Mellanox started supporting FreeBSD only recently. >> The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) >> from OFA-1.5.3. >>>> Since we started our involvement we entered some fixes to that >> driver (that were submitted into 9.2). >>>> We also mapped various issues which needs to be fixed in the future. >>>> We are now working on a major "face lift" to the driver - making it >> much more stable and with improved performance and features. >>>> Regards, >>>> >>>> Oded Shanoon >>>> OFED-FreeBSD Team Leader >>>> Mellanox Technologies, Raanana >>>> >>>> >>>> -----Original Message----- >>>> From: owner-freebsd-infiniband@freebsd.org >> >> [mailto:owner-freebsd-infiniband@freebsd.org >> ] On Behalf Of Jason >> Bacon >>>> Sent: Monday, September 16, 2013 4:32 PM >>>> To: Anthony Cornehl >>>> Cc: freebsd-infiniband@freebsd.org >> >>>> Subject: Re: Supported NICs >>>> >>>> >>>> Thanks Anthony& Oded! >>>> >>>> >>>> Would you say that IB support in 9.2 is stable enough for a >> production HPC file server? >>>> Will do plenty of my own testing first, of course. >>>> >>>> Regards, >>>> >>>> Jason >>>> >>>> On 09/15/13 22:38, Anthony Cornehl wrote: >>>>> >>>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"> >>>>> >> wrote: >>>>>> >>>>>> On Sep 15, 2013 1:20 PM, "Jason bacon"> >>>>> >> wrote: >>>>>>> >>>>>>> Is there a list of supported IB NICs out there somewhere? >>>>>>> >>>>>>> I followed the wiki instructions for rebuilding with IB support >>>>> and now have mlx4ib, mlxen, etc. >>>>>>> Was hoping there would be man pages for the drivers that list >>>>> known working cards, but there don't seem to be. I'm hoping to >>>>> test a file server using IPOIB and possible roll a FreeNAS ISO >>>>> with IB support if it works out. >>>>>>> Thanks, >>>>>>> >>>>>>> -- >>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> Jason W. Bacon >>>>>>> jwbacon@tds.net> > >>>>>>> Circumstances don't make a man: >>>>>>> They reveal him. >>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> freebsd-infiniband@freebsd.org >> >>>>> > > mailing list >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>>>>> To unsubscribe, send any mail to >>>>> "freebsd-infiniband-unsubscribe@freebsd.org >> >>>>> > >" >>>>>> Hey Jason, >>>>>> >>>>>> Mellanox ConnectX series cards are the only ones supported >> currently... >>>>>> http://www.mellanox.com/page/infiniband_cards_overview >>>>>> >>>>>> Don't forget to compile with IPoIB as well, since the IPoIB >>>>>> support >>>>> is compiled into the driver, unlike in Linux. >>>>>> Just be aware that IPoIB performance is also improved by enabling >>>>> connected mode when you compile the kernel module. The IB code in >>>>> FreeBSD is a few years older the what is in Linux, but, the >>>>> following forum thread is probably relevant... >>>>>> >>>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-pe >>>>> rfor >>>>> mance-problems.html >>>>>> Cheers! >>>>> It also appears that Jeff fixed SDP a few weeks ago, which is more >>>>> capable of reaching line-speed for IB-connected devices. >>>>> >>>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >>>>> >>>>> Cheers! >>>>> >>>> -- >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> Jason W. Bacon >>>> jwbacon@tds.net >>>> >>>> Circumstances don't make a man: >>>> They reveal him. >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> _______________________________________________ >>>> freebsd-infiniband@freebsd.org >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>> To unsubscribe, send any mail to >> "freebsd-infiniband-unsubscribe@freebsd.org >> " >>> >>> >>> -- >>> >>> ------------------------------------- >>> Jason W. Bacon >>> Systems Programmer >>> Research Computing Support >>> University of Wisconsin Milwaukee >>> bacon@uwm.edu >>> ------------------------------------- >>> >>> > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~