From owner-freebsd-infiniband@FreeBSD.ORG Wed Mar 19 17:27:27 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB095E97 for ; Wed, 19 Mar 2014 17:27:27 +0000 (UTC) Received: from ip3mta.uwm.edu (ip3mta.uwm.edu [129.89.7.192]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4E56034B for ; Wed, 19 Mar 2014 17:27:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEALrSKVOBWSZL/2dsb2JhbABag0G7aYZiUYExdIIlAQEBBAEBATUWIAoBDAQLEQQBAQEJFggHCQMCAQIBDwYBCRUJCAYIAgMBAQQBAQEBGQSHRAMeyE8NV4ZOF4xNgUcBAU8HBoQyBIlSjB5qgx+LNokTUIEE X-UWM-First: True Message-ID: <5329D338.3060103@uwm.edu> Date: Wed, 19 Mar 2014 12:26:16 -0500 From: Jason Bacon Organization: UW - Milwaukee User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Oded Shanoon Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2014 17:27:27 -0000 We finally got a new server in with a ConnectX-3 card. It seems to work reliably with our SilverStorm DDR network using a CX4 - QSFP cable. Below are specs and iperf benchmarks between the FreeBSD server and a CentOS 6.5 compute node with a Qlogic DDR card. Connected mode with MTU 65520. FreeBSD 10.0-RELEASE on a Dell R720xd. Wondering if anyone has any suggestions about what might cause the poor output performance. Any sysctls or loader.conf settings I should look into? None of the kern.conftext flags mentioned below are present. Any pointers on how to diagnose the issue would be appreciated as well. Thanks, Jason Input throughput is excellent - over 11 gb/sec. Linux to Linux on the same network does about 10 gb/sec. [root@compute-5-33 ~]# iperf -c 10.1.2.250 ------------------------------------------------------------ Client connecting to 10.1.2.250, TCP port 5001 TCP window size: 645 KByte (default) ------------------------------------------------------------ [ 3] local 10.1.2.140 port 54223 connected with 10.1.2.250 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 13.0 GBytes 11.2 Gbits/sec bacon@raid1:~ % netstat 1 input (Total) output packets errs idrops bytes packets errs bytes colls 10631 0 0 685149597 5390 0 409806 0 22281 0 0 1452318244 11136 0 846730 0 23776 0 0 1557833160 11884 0 903654 0 23891 0 0 1565368420 11952 0 907986 0 23328 0 0 1528478408 11663 0 886554 0 Output throughput is terrible: [root@compute-5-33 ~]# iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.1.2.140 port 5001 connected with 10.1.2.250 port 50207 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.9 sec 1.50 MBytes 1.16 Mbits/sec bacon@raid1:~ % netstat 1 input (Total) output packets errs idrops bytes packets errs bytes colls 20 0 0 1444 27 1 124684 0 5 0 0 284 1 4 393718 0 5 0 0 284 1 4 393718 0 6 0 0 344 1 4 393718 0 6 0 0 344 18 4 426562 0 5 0 0 284 1 4 393718 0 root@raid1:/home/bacon # ibstat CA 'mlx4_0' CA type: MT4099 Number of ports: 1 Firmware version: 2.11.500 Hardware version: 0 Node GUID: 0xf4521403007d2c10 System image GUID: 0xf4521403007d2c13 Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 163 LMC: 0 SM lid: 1 Capability mask: 0x02514868 Port GUID: 0xf4521403007d2c11 root@raid1:/home/bacon # ibportstate -G 0xf4521403007d2c11 query PortInfo: # Port info: Lid 163 port 0 LinkState:.......................Active PhysLinkState:...................LinkUp LinkWidthSupported:..............4X (IBA extension) LinkWidthEnabled:................4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................5.0 Gbps On 10/16/13 10:23, Oded Shanoon wrote: > Please run sysctl kern.conftxt and grep for the following options: > > OFED_DEBUG_INIT > IPOIB_DEBUG > DEADLKRES > INVARIANTS > INVARIANT_SUPPORT > WITNESS > WITNESS_SKIPSPIN > > These could result in performance degredation > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > > -----Original Message----- > From: Jason Bacon [mailto:bacon@uwm.edu] > Sent: Monday, October 14, 2013 9:37 PM > To: Oded Shanoon > Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org > Subject: Re: Supported NICs > > > Some initial test results... > > I installed an old Infinihost DDR HCA in one of our compute nodes running FreeBSD 9.1. > > RHEL nodes are using qlogic IB HCAs. > > 10.1.1 is gigabit Ethernet, 10.1.2 is IB. > > Running iperf server on FreeBSD and client on one of our RHEL nodes shows OK performance: > > bacon@infinibsd:/home/bacon % iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec > > [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec > > RHEL to RHEL gives us about 8 Gbits/sec. > > Iperf server on RHEL and client on FreeBSD shows very poor performance for IB, while GigE is fine: > > bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 > ------------------------------------------------------------ > Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec > > bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 > ------------------------------------------------------------ > Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec > > Any ideas what might cause this? > > I'm going to install 9.2-RELEASE and retest in any case, plus explore the ib config tools, but any feedback in the meantime would be appreciated. > > Thanks, > > JB > > On 10/07/13 01:51, Oded Shanoon wrote: >> Hi Jason, >> >> IB support in 9.2 is supposed to be stable enough. >> Please note that Mellanox started supporting FreeBSD only recently. The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5.3. >> Since we started our involvement we entered some fixes to that driver (that were submitted into 9.2). >> We also mapped various issues which needs to be fixed in the future. >> We are now working on a major "face lift" to the driver - making it much more stable and with improved performance and features. >> >> Regards, >> >> Oded Shanoon >> OFED-FreeBSD Team Leader >> Mellanox Technologies, Raanana >> >> >> -----Original Message----- >> From: owner-freebsd-infiniband@freebsd.org >> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon >> Sent: Monday, September 16, 2013 4:32 PM >> To: Anthony Cornehl >> Cc: freebsd-infiniband@freebsd.org >> Subject: Re: Supported NICs >> >> >> Thanks Anthony& Oded! >> >> Would you say that IB support in 9.2 is stable enough for a production HPC file server? >> >> Will do plenty of my own testing first, of course. >> >> Regards, >> >> Jason >> >> On 09/15/13 22:38, Anthony Cornehl wrote: >>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl">> > wrote: >>>> On Sep 15, 2013 1:20 PM, "Jason bacon">> > wrote: >>>>> Is there a list of supported IB NICs out there somewhere? >>>>> >>>>> I followed the wiki instructions for rebuilding with IB support >>> and now have mlx4ib, mlxen, etc. >>>>> Was hoping there would be man pages for the drivers that list >>> known working cards, but there don't seem to be. I'm hoping to test >>> a file server using IPOIB and possible roll a FreeNAS ISO with IB >>> support if it works out. >>>>> Thanks, >>>>> >>>>> -- >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> Jason W. Bacon >>>>> jwbacon@tds.net >>>>> >>>>> Circumstances don't make a man: >>>>> They reveal him. >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> _______________________________________________ >>>>> freebsd-infiniband@freebsd.org >>> mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>>> To unsubscribe, send any mail to >>> "freebsd-infiniband-unsubscribe@freebsd.org >>> " >>>> Hey Jason, >>>> >>>> Mellanox ConnectX series cards are the only ones supported currently... >>>> >>>> http://www.mellanox.com/page/infiniband_cards_overview >>>> >>>> Don't forget to compile with IPoIB as well, since the IPoIB support >>> is compiled into the driver, unlike in Linux. >>>> Just be aware that IPoIB performance is also improved by enabling >>> connected mode when you compile the kernel module. The IB code in >>> FreeBSD is a few years older the what is in Linux, but, the following >>> forum thread is probably relevant... >>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >>> r >>> mance-problems.html >>>> Cheers! >>> It also appears that Jeff fixed SDP a few weeks ago, which is more >>> capable of reaching line-speed for IB-connected devices. >>> >>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >>> >>> Cheers! >>> >> -- >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Jason W. Bacon >> jwbacon@tds.net >> >> Circumstances don't make a man: >> They reveal him. >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> _______________________________________________ >> freebsd-infiniband@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" > From owner-freebsd-infiniband@FreeBSD.ORG Wed Mar 19 20:22:03 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 471EFBC5 for ; Wed, 19 Mar 2014 20:22:03 +0000 (UTC) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0C606981 for ; Wed, 19 Mar 2014 20:22:02 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id lx4so9216282iec.24 for ; Wed, 19 Mar 2014 13:21:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=JRr0TvwUyOTY0JvKdaD63CGIfxqhJTcBFDtISGYpTyk=; b=V/IOCD1lA4/wITcWzjkYGEg4i/BAwyDzKHD0vDyKb0P+xeX6Nj0W7WgS+g0+xgu3/P inVff3T4iIaSIPUtFjtOGokPvUfJyLLggGld2QO6DUQ+u+sE9YsDDd9jMT8YnIqW3v/Y 2aOa6LybvzkBfjEwNg5wJSROaqmJ5TE2TdxEQaOiExbKunjii8X2LnsmSs5DUmKniemb dITxku+CtjprHWiEQDQ7meYW/KIS/YLF5Zh0Z8oVNaOC5A7h1Q9Sc+53u8qMjOWTU+eQ a1zSpX2hLIsAnNyp15s4+AYJHhC9l2MgXWVAbMC+gh/vF7ckMQ9Oj8NrJ0XyRqG+rlkd pLng== X-Gm-Message-State: ALoCoQncWahDS/nWnjWwMOA5I78lUqQ/Oa7fclDhS2uknFAVhEPwKTmMfobabIyUaSWff+jiz0NX X-Received: by 10.43.106.137 with SMTP id du9mr74959icc.93.1395260516421; Wed, 19 Mar 2014 13:21:56 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id mi2sm21959igb.3.2014.03.19.13.21.53 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 19 Mar 2014 13:21:55 -0700 (PDT) Message-ID: <5329FC5E.4080305@tds.net> Date: Wed, 19 Mar 2014 15:21:50 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Jason Bacon , Oded Shanoon Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> <5329D338.3060103@uwm.edu> In-Reply-To: <5329D338.3060103@uwm.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2014 20:22:03 -0000 Also, ibping is very slow in both directions: root@raid1:~ # ibping 28 Pong from compute-1-15.(none) (Lid 28): time 7.481 ms Pong from compute-1-15.(none) (Lid 28): time 4.465 ms Pong from compute-1-15.(none) (Lid 28): time 11.933 ms Pong from compute-1-15.(none) (Lid 28): time 21.255 ms Pong from compute-1-15.(none) (Lid 28): time 8.672 ms Pong from compute-1-15.(none) (Lid 28): time 13.633 ms [root@compute-1-15 ~]# /usr/sbin/ibping 163 Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 4.035 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.690 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.721 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.776 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 24.457 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.723 ms On 03/19/14 12:26, Jason Bacon wrote: > > We finally got a new server in with a ConnectX-3 card. It seems to > work reliably with our SilverStorm DDR network using a CX4 - QSFP cable. > > Below are specs and iperf benchmarks between the FreeBSD server and a > CentOS 6.5 compute node with a Qlogic DDR card. Connected mode with > MTU 65520. FreeBSD 10.0-RELEASE on a Dell R720xd. > > Wondering if anyone has any suggestions about what might cause the > poor output performance. Any sysctls or loader.conf settings I should > look into? None of the kern.conftext flags mentioned below are present. > > Any pointers on how to diagnose the issue would be appreciated as well. > > Thanks, > > Jason > > Input throughput is excellent - over 11 gb/sec. Linux to Linux on the > same network does about 10 gb/sec. > > [root@compute-5-33 ~]# iperf -c 10.1.2.250 > ------------------------------------------------------------ > Client connecting to 10.1.2.250, TCP port 5001 > TCP window size: 645 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.2.140 port 54223 connected with 10.1.2.250 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 13.0 GBytes 11.2 Gbits/sec > > > bacon@raid1:~ % netstat 1 > input (Total) output > packets errs idrops bytes packets errs bytes colls > 10631 0 0 685149597 5390 0 409806 0 > 22281 0 0 1452318244 11136 0 846730 0 > 23776 0 0 1557833160 11884 0 903654 0 > 23891 0 0 1565368420 11952 0 907986 0 > 23328 0 0 1528478408 11663 0 886554 0 > > Output throughput is terrible: > > [root@compute-5-33 ~]# iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.1.2.140 port 5001 connected with 10.1.2.250 port 50207 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.9 sec 1.50 MBytes 1.16 Mbits/sec > > > bacon@raid1:~ % netstat 1 > input (Total) output > packets errs idrops bytes packets errs bytes colls > 20 0 0 1444 27 1 124684 0 > 5 0 0 284 1 4 393718 0 > 5 0 0 284 1 4 393718 0 > 6 0 0 344 1 4 393718 0 > 6 0 0 344 18 4 426562 0 > 5 0 0 284 1 4 393718 0 > > root@raid1:/home/bacon # ibstat > CA 'mlx4_0' > CA type: MT4099 > Number of ports: 1 > Firmware version: 2.11.500 > Hardware version: 0 > Node GUID: 0xf4521403007d2c10 > System image GUID: 0xf4521403007d2c13 > Port 1: > State: Active > Physical state: LinkUp > Rate: 20 > Base lid: 163 > LMC: 0 > SM lid: 1 > Capability mask: 0x02514868 > Port GUID: 0xf4521403007d2c11 > > root@raid1:/home/bacon # ibportstate -G 0xf4521403007d2c11 query > PortInfo: > # Port info: Lid 163 port 0 > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkWidthSupported:..............4X (IBA extension) > LinkWidthEnabled:................4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps > LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps > LinkSpeedActive:.................5.0 Gbps > > > > > On 10/16/13 10:23, Oded Shanoon wrote: >> Please run sysctl kern.conftxt and grep for the following options: >> >> OFED_DEBUG_INIT >> IPOIB_DEBUG >> DEADLKRES >> INVARIANTS >> INVARIANT_SUPPORT >> WITNESS >> WITNESS_SKIPSPIN >> >> These could result in performance degredation >> >> Regards, >> >> Oded Shanoon >> OFED-FreeBSD Team Leader >> Mellanox Technologies, Raanana >> >> >> -----Original Message----- >> From: Jason Bacon [mailto:bacon@uwm.edu] >> Sent: Monday, October 14, 2013 9:37 PM >> To: Oded Shanoon >> Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org >> Subject: Re: Supported NICs >> >> >> Some initial test results... >> >> I installed an old Infinihost DDR HCA in one of our compute nodes >> running FreeBSD 9.1. >> >> RHEL nodes are using qlogic IB HCAs. >> >> 10.1.1 is gigabit Ethernet, 10.1.2 is IB. >> >> Running iperf server on FreeBSD and client on one of our RHEL nodes >> shows OK performance: >> >> bacon@infinibsd:/home/bacon % iperf -s >> ------------------------------------------------------------ >> Server listening on TCP port 5001 >> TCP window size: 64.0 KByte (default) >> ------------------------------------------------------------ >> [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 >> [ ID] Interval Transfer Bandwidth >> [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec >> >> [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 >> [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec >> >> RHEL to RHEL gives us about 8 Gbits/sec. >> >> Iperf server on RHEL and client on FreeBSD shows very poor >> performance for IB, while GigE is fine: >> >> bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 >> ------------------------------------------------------------ >> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 >> KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec >> >> bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 >> ------------------------------------------------------------ >> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 >> KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec >> >> Any ideas what might cause this? >> >> I'm going to install 9.2-RELEASE and retest in any case, plus explore >> the ib config tools, but any feedback in the meantime would be >> appreciated. >> >> Thanks, >> >> JB >> >> On 10/07/13 01:51, Oded Shanoon wrote: >>> Hi Jason, >>> >>> IB support in 9.2 is supposed to be stable enough. >>> Please note that Mellanox started supporting FreeBSD only recently. >>> The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) >>> from OFA-1.5.3. >>> Since we started our involvement we entered some fixes to that >>> driver (that were submitted into 9.2). >>> We also mapped various issues which needs to be fixed in the future. >>> We are now working on a major "face lift" to the driver - making it >>> much more stable and with improved performance and features. >>> >>> Regards, >>> >>> Oded Shanoon >>> OFED-FreeBSD Team Leader >>> Mellanox Technologies, Raanana >>> >>> >>> -----Original Message----- >>> From: owner-freebsd-infiniband@freebsd.org >>> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon >>> Sent: Monday, September 16, 2013 4:32 PM >>> To: Anthony Cornehl >>> Cc: freebsd-infiniband@freebsd.org >>> Subject: Re: Supported NICs >>> >>> >>> Thanks Anthony& Oded! >>> >>> Would you say that IB support in 9.2 is stable enough for a >>> production HPC file server? >>> >>> Will do plenty of my own testing first, of course. >>> >>> Regards, >>> >>> Jason >>> >>> On 09/15/13 22:38, Anthony Cornehl wrote: >>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl">>> > wrote: >>>>> On Sep 15, 2013 1:20 PM, "Jason bacon">>> > wrote: >>>>>> Is there a list of supported IB NICs out there somewhere? >>>>>> >>>>>> I followed the wiki instructions for rebuilding with IB support >>>> and now have mlx4ib, mlxen, etc. >>>>>> Was hoping there would be man pages for the drivers that list >>>> known working cards, but there don't seem to be. I'm hoping to test >>>> a file server using IPOIB and possible roll a FreeNAS ISO with IB >>>> support if it works out. >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> Jason W. Bacon >>>>>> jwbacon@tds.net >>>>>> >>>>>> Circumstances don't make a man: >>>>>> They reveal him. >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> _______________________________________________ >>>>>> freebsd-infiniband@freebsd.org >>>> mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>>>> To unsubscribe, send any mail to >>>> "freebsd-infiniband-unsubscribe@freebsd.org >>>> " >>>>> Hey Jason, >>>>> >>>>> Mellanox ConnectX series cards are the only ones supported >>>>> currently... >>>>> >>>>> http://www.mellanox.com/page/infiniband_cards_overview >>>>> >>>>> Don't forget to compile with IPoIB as well, since the IPoIB support >>>> is compiled into the driver, unlike in Linux. >>>>> Just be aware that IPoIB performance is also improved by enabling >>>> connected mode when you compile the kernel module. The IB code in >>>> FreeBSD is a few years older the what is in Linux, but, the following >>>> forum thread is probably relevant... >>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >>>> r >>>> mance-problems.html >>>>> Cheers! >>>> It also appears that Jeff fixed SDP a few weeks ago, which is more >>>> capable of reaching line-speed for IB-connected devices. >>>> >>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >>>> >>>> Cheers! >>>> >>> -- >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Jason W. Bacon >>> jwbacon@tds.net >>> >>> Circumstances don't make a man: >>> They reveal him. >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> _______________________________________________ >>> freebsd-infiniband@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>> To unsubscribe, send any mail to >>> "freebsd-infiniband-unsubscribe@freebsd.org" >> > From owner-freebsd-infiniband@FreeBSD.ORG Sat Mar 22 23:15:50 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB78937C for ; Sat, 22 Mar 2014 23:15:50 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7F6771E3 for ; Sat, 22 Mar 2014 23:15:50 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s2MNFfI6057606 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 22 Mar 2014 23:15:42 GMT Date: Sat, 22 Mar 2014 23:15:41 +0000 From: Karl Pielorz To: freebsd-infiniband@freebsd.org Subject: Mellanox MT25418 performance IPOIB? Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Mar 2014 23:15:50 -0000 Hi All, I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to use them as a cheap alternative to 10Gbps LAN cards for a point to point link to run HAST over. I've remade/installed the world with OFED support - and compiled OFED, SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel. I've setup/run opensm - and the cards appear to be active OK - ibstat output seems OK, and ibportstate shows the cabled port is 'Active' / 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to each other). I've put an IP on both ends - and can ping between the cards, but TCP/IP performance seems very poor - e.g. a test 'ftp' between the cards only nets around 381Kbyte/s? [iperf shows similarly poor results] I can't see any errors logged anywhere - and 'netstat -b -i' doesn't show any errors either. Anyone else running these cards successfully? My opensm config consists of just 'PORTS="ALL"' - I was unable to find a lot of example opensm file configs. Also I get some chatter dumped on the console (presumably from the drivers) most of it looks just like some kind of debug output [can't see any obvious errors]. fwiw both machines are 1230v3 Xeon based. Can anyone suggest what might be wrong? / What to look at? Thanks, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Sun Mar 23 13:54:59 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A6F6AB9F for ; Sun, 23 Mar 2014 13:54:59 +0000 (UTC) Received: from mail-ie0-f172.google.com (mail-ie0-f172.google.com [209.85.223.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7238DBD2 for ; Sun, 23 Mar 2014 13:54:58 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id as1so4411946iec.3 for ; Sun, 23 Mar 2014 06:54:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=a8VyE76u8hHR+f62NqJpoPWQ0BMNk1K0CYY1cgsznvw=; b=l0IuZw5uiB2MgaB+M5fsp8M7Bb0/08HpfblSYSTgLbZyXyxtVL+FS09xNlLlEHidoK pLbRp5IT5xmMAALdFvqvzA2WABEvvCBSmbFnx8Bt1UCbeKBhWewZHL+ulFDywFVa1moV 8KZirYv+T4Z3yCwYLB35t/FXwGVorufouQbCVAyGXCZ7Bbt7L9RrUJzAqWH9lbGBfvSr 268ySCzqltjI7r0amsF8X3M4wjkbpgm3VZ+TxhYrHmeW7y81+I18u535DwxbpZENbOlT wtJIU9UBg0MxRDhAnMNXYgxMNxg+SbemQXXCRcdx4dKZLmRBrsn+c1p+MkKcW4fbGpft ZPow== X-Gm-Message-State: ALoCoQnfj+SdP/t+UaTyh2T7lD6kXMK2I02bk01oQerFTrtJKRw7NQ/Ba7nCvQnXT3Mck/eOTnUY X-Received: by 10.51.17.104 with SMTP id gd8mr6893944igd.48.1395581273869; Sun, 23 Mar 2014 06:27:53 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id v2sm16895349igk.7.2014.03.23.06.27.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 23 Mar 2014 06:27:51 -0700 (PDT) Message-ID: <532EE153.3010903@tds.net> Date: Sun, 23 Mar 2014 08:27:47 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2014 13:54:59 -0000 Coincidentally, I had just run similar tests on our cluster using a FreeBSD file server and CentOS compute nodes. I reported the results in to this list. If you didn't see the thread already, you should be able to find it in the archives at: http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband Basically, iperf reported excellent incoming speed from the CentOS nodes, but outgoing speed was terrible. In response, I was informed by a Mellanox engineer that the ipoib layer is currently being overhauled and we should see improvement later this year. Regards, JB On 03/22/14 18:15, Karl Pielorz wrote: > > Hi All, > > I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / > firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping > to use them as a cheap alternative to 10Gbps LAN cards for a point to > point link to run HAST over. > > I've remade/installed the world with OFED support - and compiled OFED, > SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel. > > I've setup/run opensm - and the cards appear to be active OK - ibstat > output seems OK, and ibportstate shows the cabled port is 'Active' / > 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to > each other). > > I've put an IP on both ends - and can ping between the cards, but > TCP/IP performance seems very poor - e.g. a test 'ftp' between the > cards only nets around 381Kbyte/s? [iperf shows similarly poor results] > > I can't see any errors logged anywhere - and 'netstat -b -i' doesn't > show any errors either. > > Anyone else running these cards successfully? > > My opensm config consists of just 'PORTS="ALL"' - I was unable to find > a lot of example opensm file configs. > > Also I get some chatter dumped on the console (presumably from the > drivers) most of it looks just like some kind of debug output [can't > see any obvious errors]. > > fwiw both machines are 1230v3 Xeon based. > > Can anyone suggest what might be wrong? / What to look at? > > Thanks, > > -Karl > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" From owner-freebsd-infiniband@FreeBSD.ORG Mon Mar 24 11:15:01 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 97C94B85 for ; Mon, 24 Mar 2014 11:15:01 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 39BDF37B for ; Mon, 24 Mar 2014 11:15:00 +0000 (UTC) Received: from Mail-PC.tdx.co.uk (storm.tdx.co.uk [62.13.130.251]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s2OBErd6044566 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Mar 2014 11:14:53 GMT Date: Mon, 24 Mar 2014 11:14:52 +0000 From: Karl Pielorz To: Jason Bacon , freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <64FD6E0C96BA7D6812B5E7FF@Mail-PC.tdx.co.uk> In-Reply-To: <532EE153.3010903@tds.net> References: <532EE153.3010903@tds.net> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 11:15:01 -0000 --On 23 March 2014 08:27 -0500 Jason Bacon wrote: > > Coincidentally, I had just run similar tests on our cluster using a > FreeBSD file server and CentOS compute nodes. > > I reported the results in to this list. If you didn't see the thread > already, you should be able to find it in the archives at: Yeah, that was before I joined the list - I missed it on Google as well, sorry :( > In response, I was informed by a Mellanox engineer that the ipoib layer > is currently being overhauled and we should see improvement later this > year. That'd be great - as it's pretty useless at the moment [380Kbyte/sec] :( - I guess I'll go back to regular NICs in the meantime, and keep an eye on this list, Regards, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Wed Apr 16 12:45:12 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C315507 for ; Wed, 16 Apr 2014 12:45:12 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1D7FA1AC1 for ; Wed, 16 Apr 2014 12:45:10 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1WaPD5-0001Wc-5j for freebsd-infiniband@freebsd.org; Wed, 16 Apr 2014 14:45:03 +0200 Received: from 139.81-166-165.customer.lyse.net ([81.166.165.139]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2014 14:45:03 +0200 Received: from stiginge by 139.81-166-165.customer.lyse.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2014 14:45:03 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-infiniband@freebsd.org From: =?ISO-8859-1?Q?Stig_Inge_Lea_Bj=F8rnsen?= Subject: Re: Mellanox MT25418 performance IPOIB? Date: Wed, 16 Apr 2014 14:28:37 +0200 Lines: 69 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 139.81-166-165.customer.lyse.net User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Icedove/24.4.0 In-Reply-To: X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2014 12:45:12 -0000 Hi, I'm experimenting with a similar setup for a direct link using two HP MT25418 cards. One machine runs FreeBSD 10.0-RELEASE-p1 and the other runs Debian Unstable jessie/sid. You could try to use Infiniband in datagram mode, recompile without the IPOIB_CM option. The maximum MTU will then be 4092, but it might default to 2044. I run OpenSM on both machines and set the MTU explicitly: $ cat /etc/opensm/partitions.conf # mtu=5 for 4092, 4 for 2044 Default=0x7fff, ipoib, mtu=5 : ALL=full ; For iperf, I found that the performance when using FreeBSD as the client improved dramatically when I set the TCP window size equal to what the iperf client on Linux use: $ iperf -c linux-host -w 165K The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions. Regards, Stig Inge Lea Bjørnsen On 2014-03-23 00:15, Karl Pielorz wrote: > > Hi All, > > I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / > firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to > use them as a cheap alternative to 10Gbps LAN cards for a point to point > link to run HAST over. > > I've remade/installed the world with OFED support - and compiled OFED, > SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel. > > I've setup/run opensm - and the cards appear to be active OK - ibstat > output seems OK, and ibportstate shows the cabled port is 'Active' / > 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to > each other). > > I've put an IP on both ends - and can ping between the cards, but TCP/IP > performance seems very poor - e.g. a test 'ftp' between the cards only > nets around 381Kbyte/s? [iperf shows similarly poor results] > > I can't see any errors logged anywhere - and 'netstat -b -i' doesn't > show any errors either. > > Anyone else running these cards successfully? > > My opensm config consists of just 'PORTS="ALL"' - I was unable to find a > lot of example opensm file configs. > > Also I get some chatter dumped on the console (presumably from the > drivers) most of it looks just like some kind of debug output [can't see > any obvious errors]. > > fwiw both machines are 1230v3 Xeon based. > > Can anyone suggest what might be wrong? / What to look at? > > Thanks, > > -Karl From owner-freebsd-infiniband@FreeBSD.ORG Thu Apr 17 09:10:43 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A3773FE7 for ; Thu, 17 Apr 2014 09:10:43 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 4A0A7105C for ; Thu, 17 Apr 2014 09:10:42 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3H99l3Q072150 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Apr 2014 10:09:47 +0100 (BST) Date: Thu, 17 Apr 2014 10:09:46 +0100 From: Karl Pielorz To: =?UTF-8?Q?Stig_Inge_Lea_Bj=C3=B8rnsen?= , freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <830453C43410411669F120CC@study64.tdx.co.uk> In-Reply-To: References: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 09:10:43 -0000 --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bj=C3=B8rnsen=20 wrote: > You could try to use Infiniband in datagram mode, recompile without the > IPOIB_CM option. Ok, same boxes - I just removed IPOIB_CM out of the kernel config and=20 recompiled / installed it. iperf now consistently returns: [ 3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 8.71 GBytes 7.46 Gbits/sec > The maximum MTU will then be 4092, but it might default to 2044. I run > OpenSM on both machines and set the MTU explicitly: Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss of=20 .5-1Gbit with iperf (obviously not 'real world') - but at leas I can now=20 try and use the stuff with real world (now it's returning more than a few=20 hundred kilobytes a second!). > The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions. Same as here then. Thanks for your email! Regards, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Thu Apr 17 13:01:51 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE481188 for ; Thu, 17 Apr 2014 13:01:50 +0000 (UTC) Received: from mail-ie0-f181.google.com (mail-ie0-f181.google.com [209.85.223.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6AE618E1 for ; Thu, 17 Apr 2014 13:01:50 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id tp5so324616ieb.12 for ; Thu, 17 Apr 2014 06:01:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=TnO2//UvNB6YpEsLqFMQbLmt4fCZ1bS1XPxM359jxxQ=; b=QtX1KtNkx6bEG9P47X6X2giScBlgk2LRiYOKXyy+WhARmTh7eRKBo2OTxi8JRvOJ49 /Rpp//QF31VdrzlwuU2LLO7rXLiQ61X30iOxft9BEeueV0O4Bcg5UoWtlkSBhx3iy+9V TVtlDabJjyi2/B/NRHYJAykiD2Ykmots1QxOiMBLisPfpTj48VLMC/KcTdG5M4IkOOZu wWJ1wA1kwGO0cnv+qiVrHxCqIl/zhnqa0FyaBOV7QMaEbfT4FCGNg9P9m0OJ6IXXYhMz 6HRlYg1lTRFUw0BrNohoHEfz6tHndsyuhR1WJlWt+5jaGIkjm3cIbZsgShG8qxDDvD6b 3l2w== X-Gm-Message-State: ALoCoQnxOSQyF+Wp9uKGBH2qfwXzkUWPoNVnFokDWU7bYOS9zzpis6i5s1UfdmEuuwdZKb9aBvjK X-Received: by 10.42.41.212 with SMTP id q20mr491030ice.80.1397739709681; Thu, 17 Apr 2014 06:01:49 -0700 (PDT) Received: from [192.168.0.52] (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id h7sm6261068igy.2.2014.04.17.06.01.48 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Apr 2014 06:01:49 -0700 (PDT) Message-ID: <534FD0BB.1070308@tds.net> Date: Thu, 17 Apr 2014 08:01:47 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Karl Pielorz , =?UTF-8?B?U3RpZyBJbmdlIExlYSBC?= =?UTF-8?B?asO4cm5zZW4=?= , freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? References: <830453C43410411669F120CC@study64.tdx.co.uk> In-Reply-To: <830453C43410411669F120CC@study64.tdx.co.uk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 13:01:51 -0000 Hi Karl, What type of network are you running? DDR? QDR? FDR? Switch and HCA models? We're purchasing a new cluster with Mellanox ConnectX-3 FDR cards. I'll report back some results with and without IPOIB once I have my hands on it. FYI, with some tuning effort, I was able to get up to 600 megabytes/sec both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a PERC H710, ext4, RAID 6. Might be able to do a little better with FreeBSD and ZFS where IB is providing 7.5gb/sec. Regards, JB On 4/17/14 4:09 AM, Karl Pielorz wrote: > > --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bjørnsen > wrote: > >> You could try to use Infiniband in datagram mode, recompile without the >> IPOIB_CM option. > > Ok, same boxes - I just removed IPOIB_CM out of the kernel config and > recompiled / installed it. > > iperf now consistently returns: > > [ 3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 8.71 GBytes 7.46 Gbits/sec > >> The maximum MTU will then be 4092, but it might default to 2044. I run >> OpenSM on both machines and set the MTU explicitly: > > Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss > of .5-1Gbit with iperf (obviously not 'real world') - but at leas I > can now try and use the stuff with real world (now it's returning more > than a few hundred kilobytes a second!). > >> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions. > > Same as here then. > > Thanks for your email! > > Regards, > > -Karl > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-freebsd-infiniband@FreeBSD.ORG Thu Apr 17 17:22:13 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D93F9B8 for ; Thu, 17 Apr 2014 17:22:13 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id A771919F2 for ; Thu, 17 Apr 2014 17:22:12 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3HHM9Yf017862 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Apr 2014 18:22:10 +0100 (BST) Date: Thu, 17 Apr 2014 18:22:08 +0100 From: Karl Pielorz To: Jason Bacon , =?UTF-8?Q?Stig_Inge_Lea_Bj=C3=B8rnsen?= , freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? Message-ID: <2E4C44F3E209188F46C55C4F@study64.tdx.co.uk> In-Reply-To: <534FD0BB.1070308@tds.net> References: <830453C43410411669F120CC@study64.tdx.co.uk> <534FD0BB.1070308@tds.net> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 17:22:13 -0000 --On 17 April 2014 08:01:47 -0500 Jason Bacon wrote: > Hi Karl, > > What type of network are you running? DDR? QDR? FDR? Switch and HCA > models? Ok, still 'new' to IB - I've basically just got two MT25418 cards running back to back in two machines. 'ibportstate' seems to tell me I'm running LinkWidthActive 4X, LinkSpeedActive 5.0Gbps. > FYI, with some tuning effort, I was able to get up to 600 megabytes/sec > both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a > PERC H710, ext4, RAID 6. Might be able to do a little better with > FreeBSD and ZFS where IB is providing 7.5gb/sec. I'm planning on using these cards for HAST between two boxes running ZFS (so they're just doing TCP/IP duties). Though at the moment I've not been overly impressed with HAST's performance (even running with 'none' as the secondary) - it seems to dull the IO performance of any disks it's using even when just reading. I'm going to give it the benefit of doubt though and just build the boxes up, sling a bunch of drives in and see what the performance comes out as. -Karl From owner-freebsd-infiniband@FreeBSD.ORG Thu Apr 17 18:08:02 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0924A548 for ; Thu, 17 Apr 2014 18:08:02 +0000 (UTC) Received: from mail-ie0-f173.google.com (mail-ie0-f173.google.com [209.85.223.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C62171FA7 for ; Thu, 17 Apr 2014 18:08:00 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id rl12so722955iec.18 for ; Thu, 17 Apr 2014 11:07:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=TgnUV2v39gq5AXNOYlkWXWjEBzsyh/v5CIyQ+Q6ZxQk=; b=coG5uHNAfZs5wv/0fxLVSXgFe3LDpLC3ieSm7llRKxb1XX4kwCHVZhX9fNaQ1RMMYf JEzmnk/aDSsXxTz3EzYVXaSSTJIV0hf7lvRtZfvaJ5zVJMl7C1GLXpOjYK1gPxLSVk93 Y8Ppvlo6sfFYLXC1XUy/LNvE1YEQnAheFf3mO0HlrVK+VyS6jE5Y4oatjjJlJLH3hKrl 8LzYovjdcxzQTs4KDld1eAbPNOCn3+09BIgA2VurteOKCVuuYZ778Mcmdrc7JMMErzyo ITvnogZXVTzmIrZVwnHPwFS5MrS4uMYWZjEH62dwM1Ht0K4jxDXR+b/qAubMebJ1O6VP DyFQ== X-Gm-Message-State: ALoCoQmcc0KWAv2H1uKyMRQ+MrVFdiT9az25Ux1wGvt17IdIJAUuovklT1H+RkWkuuoxsgM5Mkck X-Received: by 10.42.129.9 with SMTP id o9mr10768176ics.38.1397757621706; Thu, 17 Apr 2014 11:00:21 -0700 (PDT) Received: from tocino.cs.uwm.edu (tocino.cs.uwm.edu. [129.89.38.75]) by mx.google.com with ESMTPSA id s9sm7988153igw.16.2014.04.17.11.00.20 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Apr 2014 11:00:21 -0700 (PDT) Message-ID: <535016B7.2060905@tds.net> Date: Thu, 17 Apr 2014 13:00:23 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Karl Pielorz , =?ISO-8859-1?Q?Stig_Inge_L?= =?ISO-8859-1?Q?ea_Bj=F8rnsen?= , freebsd-infiniband@freebsd.org Subject: Re: Mellanox MT25418 performance IPOIB? References: <830453C43410411669F120CC@study64.tdx.co.uk> <534FD0BB.1070308@tds.net> <2E4C44F3E209188F46C55C4F@study64.tdx.co.uk> In-Reply-To: <2E4C44F3E209188F46C55C4F@study64.tdx.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 18:08:02 -0000 OK, so 4X 5.0Gbps would imply a DDR network, for which the best throughput I've seen on TCP/IP connections is around 12Gbps. I get around 11 both ways CentOS to CentOS. OpenMPI will reach about 16. Given that, I'm hopeful that an FDR network will be able to exceed the local throughput of our 12-disk RAIDs even without IPOIB. Regards, JB On 04/17/14 12:22, Karl Pielorz wrote: > > --On 17 April 2014 08:01:47 -0500 Jason Bacon wrote: > >> Hi Karl, >> >> What type of network are you running? DDR? QDR? FDR? Switch and HCA >> models? > > Ok, still 'new' to IB - I've basically just got two MT25418 cards > running back to back in two machines. > > 'ibportstate' seems to tell me I'm running LinkWidthActive 4X, > LinkSpeedActive 5.0Gbps. > >> FYI, with some tuning effort, I was able to get up to 600 megabytes/sec >> both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a >> PERC H710, ext4, RAID 6. Might be able to do a little better with >> FreeBSD and ZFS where IB is providing 7.5gb/sec. > > I'm planning on using these cards for HAST between two boxes running > ZFS (so they're just doing TCP/IP duties). Though at the moment I've > not been overly impressed with HAST's performance (even running with > 'none' as the secondary) - it seems to dull the IO performance of any > disks it's using even when just reading. > > I'm going to give it the benefit of doubt though and just build the > boxes up, sling a bunch of drives in and see what the performance > comes out as. > > -Karl > From owner-freebsd-infiniband@FreeBSD.ORG Tue May 6 14:10:42 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BDC1414 for ; Tue, 6 May 2014 14:10:42 +0000 (UTC) Received: from mail-ig0-f181.google.com (mail-ig0-f181.google.com [209.85.213.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A29F606 for ; Tue, 6 May 2014 14:10:41 +0000 (UTC) Received: by mail-ig0-f181.google.com with SMTP id h3so3008120igd.2 for ; Tue, 06 May 2014 07:10:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=hHlDw2UaBWXS8zPHOqZRgGTwHvqcC7zfOWpksWOqpAs=; b=h2C3RmPaZrmxAJ+z/yATRD1BSVaFczZ57XjK9rD6xNYWcekVQGDeaJXE7X1MUTiKkt SF8fKn4kRtu3RfD/henNyQ2btiEWsnKswq5FszuWnyKDgsq+Rd+kkommPqD8pkysDWUj mIqB80hcM9FWhpkLGOd1ST12iyqR60cL2R8TQqlatGk4AQJViqB7JVSvEbcaI4Lv0zTS KJXPi+1ZnjJAKKE/mSXr6F5AbmTUA6ppOz1lILxUmU++AfJ7y48KpfiD5OoQqAUIBm+L 8QvsopfDsNqDhbCifyhTkVhZ0URuhmEZaqXyE5pzyFIA3ZOKTAosBk0doiG2Jifsm2nR jHYg== X-Gm-Message-State: ALoCoQmLwxSqUBFzxFknSkzpYdlbo8tNFmQsq/bA2MJ1UGKDpg/tZrXpITpgqwkAU4PnZr70p7Ku X-Received: by 10.43.151.7 with SMTP id kq7mr10671642icc.78.1399385435402; Tue, 06 May 2014 07:10:35 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id sb11sm39164783igb.0.2014.05.06.07.10.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 06 May 2014 07:10:34 -0700 (PDT) Message-ID: <5368ED57.5090701@tds.net> Date: Tue, 06 May 2014 09:10:31 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: "freebsd-infiniband@freebsd.org" Subject: Kernel module? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2014 14:10:42 -0000 Does anyone know if are any plans for converting the IB stack to a loadable kernel module? At present, freebsd-update will remove IB support if it updates the kernel, making it necessary to do another buildkernel following the updates. Certain other updates might necessitate a buildworld as well, which would also be nice to eliminate. Thanks, Jason From owner-freebsd-infiniband@FreeBSD.ORG Wed May 7 04:39:41 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2555F37C for ; Wed, 7 May 2014 04:39:41 +0000 (UTC) Received: from eu1sys200aog119.obsmtp.com (eu1sys200aog119.obsmtp.com [207.126.144.147]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52298EC5 for ; Wed, 7 May 2014 04:39:39 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob119.postini.com ([207.126.147.11]) with SMTP ID DSNKU2m46l4euvU34y2KGWlhWmPtYEqan/BQ@postini.com; Wed, 07 May 2014 04:39:40 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Wed, 7 May 2014 07:39:05 +0300 From: Oded Shanoon To: Jason Bacon , "freebsd-infiniband@freebsd.org" Subject: RE: Kernel module? Thread-Topic: Kernel module? Thread-Index: AQHPaTVML0ctwXpmjku8PvvKg34KAJs0iRUA Date: Wed, 7 May 2014 04:39:04 +0000 Message-ID: <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> References: <5368ED57.5090701@tds.net> In-Reply-To: <5368ED57.5090701@tds.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 04:39:41 -0000 Hi, We (Mellanox) already did that in FreeBSD-10.0. You can go to sys/modules and compile + kldload the following modules (in t= hat specific order): mlx4, ibcore, mlx4ib, ipoib. We are planning on refining it and break ib_core into smaller modules (ib_s= a, ib_mad...) on our next commit. Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Jason Bacon Sent: Tuesday, May 06, 2014 5:13 PM To: freebsd-infiniband@freebsd.org Subject: Kernel module? Does anyone know if are any plans for converting the IB stack to a loadable= kernel module? At present, freebsd-update will remove IB support if it up= dates the kernel, making it necessary to do another buildkernel following t= he updates. Certain other updates might necessitate a buildworld as well, which would a= lso be nice to eliminate. Thanks, Jason _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@FreeBSD.ORG Wed May 7 13:20:31 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4E22930 for ; Wed, 7 May 2014 13:20:31 +0000 (UTC) Received: from mail-ig0-f181.google.com (mail-ig0-f181.google.com [209.85.213.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F76E894 for ; Wed, 7 May 2014 13:20:31 +0000 (UTC) Received: by mail-ig0-f181.google.com with SMTP id h3so1075270igd.2 for ; Wed, 07 May 2014 06:20:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=syAWne7wi19FnG9fGSQn7ykZlISAOTcb4PPclJWr8fc=; b=XT51GoIuQfM8d8QZuyLx+lh3IR8CDSlgiwRyakeqfA/+Wh6F1eduuXDRpyQysxSO4B dTg3GMgACOGwudPRGv63SXRugYkg45syBSai11Ttuw8YT6QGkibEiALXUAj/8rCN224+ pIMWF45wycHbNm9wKlVgmCepp2vSr+XfvukHF95S1MAAQrSRL0E7Ie2zEku2Ncqi9m1T yXGm4M76cgTjZePwcWUCg4jHHvcc38Qyo2gWOBSuURYnsEwGy11kzTRTlLZS4vTJbWg9 RDAOoNeMR1KBPp4ETowUmjNvQkQvR31rM1giNEDPHwDbAqFywH3DBCrrTr4uDeQDJx+6 77ZA== X-Gm-Message-State: ALoCoQl78wPZNsmlhcBxrGX9ohSAlUos9VduAU0SkKt454i72pmvsZs8WHP8armDZ4sg9vqv668X X-Received: by 10.50.111.138 with SMTP id ii10mr43438508igb.34.1399468830643; Wed, 07 May 2014 06:20:30 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id e5sm4338866igl.20.2014.05.07.06.20.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 07 May 2014 06:20:30 -0700 (PDT) Message-ID: <536A331C.50808@tds.net> Date: Wed, 07 May 2014 08:20:28 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Oded Shanoon , "freebsd-infiniband@freebsd.org" Subject: Re: Kernel module? References: <5368ED57.5090701@tds.net> <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 13:20:31 -0000 Now I see, thanks. It might be good to update the wiki with all the recent developments: https://wiki.freebsd.org/InfiniBand It looks pretty outdated, and it's the first thing that comes up on a Google search for "freebsd infiniband". If nobody else is inclined to do so, I'd be happy to do it myself assuming I can get write access. Regards, JB On 05/06/14 23:39, Oded Shanoon wrote: > Hi, > > We (Mellanox) already did that in FreeBSD-10.0. > You can go to sys/modules and compile + kldload the following modules (in that specific order): mlx4, ibcore, mlx4ib, ipoib. > > We are planning on refining it and break ib_core into smaller modules (ib_sa, ib_mad...) on our next commit. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Tuesday, May 06, 2014 5:13 PM > To: freebsd-infiniband@freebsd.org > Subject: Kernel module? > > > Does anyone know if are any plans for converting the IB stack to a loadable kernel module? At present, freebsd-update will remove IB support if it updates the kernel, making it necessary to do another buildkernel following the updates. > > Certain other updates might necessitate a buildworld as well, which would also be nice to eliminate. > > Thanks, > > Jason > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" From owner-freebsd-infiniband@FreeBSD.ORG Wed May 7 16:48:57 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3080992B for ; Wed, 7 May 2014 16:48:57 +0000 (UTC) Received: from mail-ie0-f181.google.com (mail-ie0-f181.google.com [209.85.223.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE959FAD for ; Wed, 7 May 2014 16:48:56 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id y20so1208838ier.26 for ; Wed, 07 May 2014 09:48:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=jDqa+tqj6pAQ0+Sk/pW6nvJ5kO4zWT06uC1mE6yuhow=; b=AR9IIJ8+v9xLXJPg4cSzo0ALpjrKk2AXVLMBNwEG72SQYRLWDNLOakwCKexulg/xb7 okEKc++l3v5AbBTSjGMnMqKNGidGzcKAcwLnFuJBkusGmKSVe4m8PnpzlHFjYRP1Vnl9 e0p3xcWYQTxbbJJ+VzthG3vsFaOGHqhWe8CWC18v6Thyn8ytd2G2gTtSya2c5a9mSfio bSQm6xmM6prBmg4Ni0HhmcpMk+GM65CB+FY5EYQZX5jkTbOKK0pjJXwpq3uA5FyoP7mE BzPtk2LdP6y//cjM72qdWX0aKMRA4kj8SOs2UQ/yr30hT5mLahE1PjcG5qOSS2cZaYJ6 fPyw== X-Gm-Message-State: ALoCoQliFd9wCy2zsbUcD7uzzrQ9P7LOI9XZ0I4M/ITPQjGxgWO5nbs3BHogMou2mM0smc5k36N3 X-Received: by 10.50.143.34 with SMTP id sb2mr45003471igb.48.1399479626547; Wed, 07 May 2014 09:20:26 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id ie20sm48505918igb.10.2014.05.07.09.20.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 07 May 2014 09:20:25 -0700 (PDT) Message-ID: <536A5D46.5050304@tds.net> Date: Wed, 07 May 2014 11:20:22 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Oded Shanoon , "freebsd-infiniband@freebsd.org" Subject: Re: Kernel module? References: <5368ED57.5090701@tds.net> <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 16:48:57 -0000 For userland tools, is it still recommended to do a buildworld, or is a more targeted approach possible? E.g., I'm thinking it might suffice to set WITH_OFED='yes' in src.conf and just rebuild select tools/libs. There appear to be things outside contrib/ofed and sys/modules that need to be rebuilt, but I wouldn't want to try and determine what they are on my own. Thanks, JB On 05/06/14 23:39, Oded Shanoon wrote: > Hi, > > We (Mellanox) already did that in FreeBSD-10.0. > You can go to sys/modules and compile + kldload the following modules (in that specific order): mlx4, ibcore, mlx4ib, ipoib. > > We are planning on refining it and break ib_core into smaller modules (ib_sa, ib_mad...) on our next commit. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Tuesday, May 06, 2014 5:13 PM > To: freebsd-infiniband@freebsd.org > Subject: Kernel module? > > > Does anyone know if are any plans for converting the IB stack to a loadable kernel module? At present, freebsd-update will remove IB support if it updates the kernel, making it necessary to do another buildkernel following the updates. > > Certain other updates might necessitate a buildworld as well, which would also be nice to eliminate. > > Thanks, > > Jason > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" From owner-freebsd-infiniband@FreeBSD.ORG Tue May 13 07:42:03 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A74F2DDA for ; Tue, 13 May 2014 07:42:03 +0000 (UTC) Received: from eu1sys200aog101.obsmtp.com (eu1sys200aog101.obsmtp.com [207.126.144.111]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 868D42784 for ; Tue, 13 May 2014 07:42:02 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob101.postini.com ([207.126.147.11]) with SMTP ID DSNKU3HMx95OgW6J6JLl8FwYtzwAxnQRWVbJ@postini.com; Tue, 13 May 2014 07:42:02 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Tue, 13 May 2014 10:37:57 +0300 From: Meny Yossefi To: "jwbacon@tds.net" , "freebsd-infiniband@freebsd.org" Subject: RE: Kernel module? Thread-Topic: Kernel module? Thread-Index: AQHPaTVML0ctwXpmjku8PvvKg34KAJs0iRUAgACSswCAB7eS0IABVIMA Date: Tue, 13 May 2014 07:37:56 +0000 Message-ID: References: <5368ED57.5090701@tds.net> <058596FC127BDB469D7D60F04E37C63F95E8B917@MTLDAG01.mtl.com> <536A5D46.5050304@tds.net> <058596FC127BDB469D7D60F04E37C63F95E978B1@MTLDAG01.mtl.com> In-Reply-To: <058596FC127BDB469D7D60F04E37C63F95E978B1@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Meny Yossefi X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2014 07:42:03 -0000 Hi, Yes, it is still recommended to do a buildworld for userland tools, adding = 'WITH_OFED=3D1' to /etc/make.conf. Another way to go to install ibutils for instance: cd /usr/src/contrib/ofed && make includes cd /usr/src/contrib/ofed /usr.lib && make depend all install cd /usr/src/contrib/ofed /usr.bin && make depend all install -Meny -----Original Message----- From: Jason Bacon [mailto:jwbacon@tds.net] Sent: Wednesday, May 07, 2014 7:21 PM To: Oded Shanoon; freebsd-infiniband@freebsd.org Subject: Re: Kernel module? For userland tools, is it still recommended to do a buildworld, or is a mor= e targeted approach possible? E.g., I'm thinking it might suffice to set WITH_OFED=3D'yes' in src.conf an= d just rebuild select tools/libs. There appear to be things outside contri= b/ofed and sys/modules that need to be rebuilt, but I wouldn't want to try = and determine what they are on my own. Thanks, JB On 05/06/14 23:39, Oded Shanoon wrote: > Hi, > > We (Mellanox) already did that in FreeBSD-10.0. > You can go to sys/modules and compile + kldload the following modules (in= that specific order): mlx4, ibcore, mlx4ib, ipoib. > > We are planning on refining it and break ib_core into smaller modules (ib= _sa, ib_mad...) on our next commit. > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org > [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon > Sent: Tuesday, May 06, 2014 5:13 PM > To: freebsd-infiniband@freebsd.org > Subject: Kernel module? > > > Does anyone know if are any plans for converting the IB stack to a loadab= le kernel module? At present, freebsd-update will remove IB support if it = updates the kernel, making it necessary to do another buildkernel following= the updates. > > Certain other updates might necessitate a buildworld as well, which would= also be nice to eliminate. > > Thanks, > > Jason > > _______________________________________________ > freebsd-infiniband@freebsd.org mai= ling list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.= org" From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 15:00:25 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38676C5F for ; Thu, 15 May 2014 15:00:25 +0000 (UTC) Received: from morbid.purplehat.org (morbid.purplehat.org [206.225.82.173]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1AA00210E for ; Thu, 15 May 2014 15:00:24 +0000 (UTC) Received: from localhost (morbid.purplehat.org [206.225.82.173]) by morbid.purplehat.org (Postfix) with ESMTP id 3A355D8CCE1 for ; Thu, 15 May 2014 07:50:50 -0700 (MST) Received: from morbid.purplehat.org ([206.225.82.173]) by localhost (morbid.purplehat.org [206.225.82.173]) (maiad, port 10024) with ESMTP id 82615-08 for ; Thu, 15 May 2014 07:50:49 -0700 (MST) Received: from [172.17.0.60] (fw1.aspsys.com [173.14.5.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jankyj@unfs.us) by morbid.purplehat.org (Postfix) with ESMTPSA id DAE97D8CCE0 for ; Thu, 15 May 2014 07:50:48 -0700 (MST) Message-ID: <5374D431.5020501@unfs.us> Date: Thu, 15 May 2014 08:50:25 -0600 From: "Janky Jay, III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: FBSD to FBSD NFS Mounts over IB. Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 15:00:25 -0000 Hello All, We're recently been poking around with IB support in FBSD because we would really like to use ZFS for cluster storage. We're currently testing between two FBSD (10.0-R-p2) systems and two Linux (CentOS 6.5 2.6.32) systems and we're running into something a little strange... I have set up one of the FBSD systems to run OpenSM and also be an NFS server which all the systems seem to be able to mount over the IB devices without any issue at all. Small reads and writes to and from the NFS server to all the other nodes also seems to work without any issue. However, if I try to dump large amounts of data using "dd" (in order to test speeds and stability), the FBSD NFS client craps out immediately. I just get the following message(s) over and over: newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding newnfs server 10.11.1.1:/data: not responding Strangely enough, though, the Linux nodes doesn't seem to have a problem at all. I'm unsure if anyone else has seen this problem or if it's even an IB issue at all (could be an NFS issue). There just doesn't seem to be much (if any) documentation about NFS over IB on FBSD any so I'm just curious if someone here may be able to point me in the right direction. The output of "ibstat" for both FBSD nodes is as follows: node1 (NFS Server) CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.9.1000 Hardware version: b0 Node GUID: 0x002590ffff16a3cc System image GUID: 0x002590ffff16a3cf Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0251086a Port GUID: 0x002590ffff16a3cd node2 (NFS Client) CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.9.1000 Hardware version: b0 Node GUID: 0x002590ffff16b3a0 System image GUID: 0x002590ffff16b3a3 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 3 LMC: 0 SM lid: 1 Capability mask: 0x02510868 Port GUID: 0x002590ffff16b3a1 If anyone has any ideas as to what I can try, it would be greatly appreciated! Regards, Janky Jay, III From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 15:46:01 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95954C69 for ; Thu, 15 May 2014 15:46:01 +0000 (UTC) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 60BE9252F for ; Thu, 15 May 2014 15:46:01 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id tp5so1185755ieb.41 for ; Thu, 15 May 2014 08:46:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=b5bfjrjMY357tWq7TxA7ZzYTp0ykrSCRvUNTki3F4IE=; b=BtQ7XZqfLRRZ4h+rB/eXtx/61r0pbSTbXu2RMzKi+aAASoJLdqfkhB+JrNlkElFEc9 tMXYVBMT47uv4AsvTex5M2KEtxbTZtbWVdnVV59sqndCn+b1TwvUjSBIh0y5nljrznto t/B9d57yLp+IMc/ARyVV1qEXMfDtWFxmJIdx2DhqJ9wAwNDwCQST5+PnsNS6G3BZTtJp YtIp5hz5gQQPQT5MQubOuKlCdLZUsaaWZ/1v1L329JsLxPT4hKckFF+g7d0KYzGaqyA/ JcuNsQ6YseO4NMAumjm/sDV4UBOU6eXhatXpfSff9v+nPPlrjeAvHeP4vl7FRbJnC9sB W5Eg== X-Gm-Message-State: ALoCoQmuCzgIVoX/2alCFbVN9zebhcMYBrMafw8itY7En9OguRD463p2HpVI2Se/5JeB0X3K4vpZ X-Received: by 10.50.118.69 with SMTP id kk5mr13308465igb.10.1400168760246; Thu, 15 May 2014 08:46:00 -0700 (PDT) Received: from [192.168.0.52] (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id d10sm13925867igc.8.2014.05.15.08.45.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 May 2014 08:45:58 -0700 (PDT) Message-ID: <5374E134.5080809@tds.net> Date: Thu, 15 May 2014 10:45:56 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: "Janky Jay, III" , freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. References: <5374D431.5020501@unfs.us> In-Reply-To: <5374D431.5020501@unfs.us> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 15:46:01 -0000 Are you using connected mode? I've seen severe performance issues with IPOIB in connected mode. In one direction, it outperformed CentOS 6.5, but in the other it was abysmal. Another list member recently reported that disabling connected mode made things much more stable, although you won't get the throughput that you would from a stable connected mode. A Mellanox engineer also told me that they're in the process of revamping the IPOIB layer, due out sometime later this year. As for documentation, the archives of this list are probably the best source right now. I keep my progress and tools posted here: http://acadix.biz/infiniband.php Regards, Jason On 5/15/14 9:50 AM, Janky Jay, III wrote: > Hello All, > > We're recently been poking around with IB support in FBSD because > we would really like to use ZFS for cluster storage. We're currently > testing between two FBSD (10.0-R-p2) systems and two Linux (CentOS 6.5 > 2.6.32) systems and we're running into something a little strange... > > I have set up one of the FBSD systems to run OpenSM and also be an > NFS server which all the systems seem to be able to mount over the IB > devices without any issue at all. Small reads and writes to and from > the NFS server to all the other nodes also seems to work without any > issue. However, if I try to dump large amounts of data using "dd" (in > order to test speeds and stability), the FBSD NFS client craps out > immediately. I just get the following message(s) over and over: > > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding > > Strangely enough, though, the Linux nodes doesn't seem to have a > problem at all. I'm unsure if anyone else has seen this problem or if > it's even an IB issue at all (could be an NFS issue). There just > doesn't seem to be much (if any) documentation about NFS over IB on > FBSD any so I'm just curious if someone here may be able to point me > in the right direction. The output of "ibstat" for both FBSD nodes is > as follows: > > node1 (NFS Server) > CA 'mlx4_0' > CA type: MT26428 > Number of ports: 1 > Firmware version: 2.9.1000 > Hardware version: b0 > Node GUID: 0x002590ffff16a3cc > System image GUID: 0x002590ffff16a3cf > Port 1: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 1 > LMC: 0 > SM lid: 1 > Capability mask: 0x0251086a > Port GUID: 0x002590ffff16a3cd > > node2 (NFS Client) > CA 'mlx4_0' > CA type: MT26428 > Number of ports: 1 > Firmware version: 2.9.1000 > Hardware version: b0 > Node GUID: 0x002590ffff16b3a0 > System image GUID: 0x002590ffff16b3a3 > Port 1: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 3 > LMC: 0 > SM lid: 1 > Capability mask: 0x02510868 > Port GUID: 0x002590ffff16b3a1 > > If anyone has any ideas as to what I can try, it would be greatly > appreciated! > > Regards, > Janky Jay, III > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 16:50:13 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E404CB58 for ; Thu, 15 May 2014 16:50:13 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 8BF8E2ACD for ; Thu, 15 May 2014 16:50:12 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s4FGo3mp059805 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 May 2014 17:50:04 +0100 (BST) Date: Thu, 15 May 2014 17:50:03 +0100 From: Karl Pielorz To: "Janky Jay, III" , freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. Message-ID: <55BC554716A7EA5C54F5DD02@study64.tdx.co.uk> In-Reply-To: <5374D431.5020501@unfs.us> References: <5374D431.5020501@unfs.us> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 16:50:14 -0000 --On 15 May 2014 08:50:25 -0600 "Janky Jay, III" wrote: > I have set up one of the FBSD systems to run OpenSM and also be an NFS > server which all the systems seem to be able to mount over the IB devices > without any issue at all. Small reads and writes to and from the NFS > server to all the other nodes also seems to work without any issue. > However, if I try to dump large amounts of data using "dd" (in order to > test speeds and stability), the FBSD NFS client craps out immediately. I > just get the following message(s) over and over: > > newnfs server 10.11.1.1:/data: not responding > newnfs server 10.11.1.1:/data: not responding Can both sides 'ping' each other when this happens? The reason I ask is I've hit a similar issue setting up ZFS over iSCSI on IPOIB (I'm not running connected mode). At my end it looks like an ARP expires or something so the sides 'lose site' of each other. If 'A' can't see 'B' - a ping from 'B' to 'A' usually restores the connection. Maybe make sure they can both still see each other outside of nfs - I can temporarily 'fix' the issue here by leaving both sides pinging each other - I've not really had a chance to look at it much recently... -Karl From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 17:30:24 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 75414795 for ; Thu, 15 May 2014 17:30:24 +0000 (UTC) Received: from morbid.purplehat.org (morbid.purplehat.org [206.225.82.173]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43EBB2EEE for ; Thu, 15 May 2014 17:30:24 +0000 (UTC) Received: from localhost (morbid.purplehat.org [206.225.82.173]) by morbid.purplehat.org (Postfix) with ESMTP id B0660D8CCEE; Thu, 15 May 2014 10:23:19 -0700 (MST) Received: from morbid.purplehat.org ([206.225.82.173]) by localhost (morbid.purplehat.org [206.225.82.173]) (maiad, port 10024) with ESMTP id 85263-04; Thu, 15 May 2014 10:23:14 -0700 (MST) Received: from [172.17.0.60] (fw1.aspsys.com [173.14.5.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jankyj@unfs.us) by morbid.purplehat.org (Postfix) with ESMTPSA id 84014D8CCF6; Thu, 15 May 2014 10:23:14 -0700 (MST) Message-ID: <5374F7EA.6060505@unfs.us> Date: Thu, 15 May 2014 11:22:50 -0600 From: "Janky Jay, III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Karl Pielorz , freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. References: <5374D431.5020501@unfs.us> <55BC554716A7EA5C54F5DD02@study64.tdx.co.uk> In-Reply-To: <55BC554716A7EA5C54F5DD02@study64.tdx.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 17:30:24 -0000 Hello Karl, On 05/15/2014 10:50 AM, Karl Pielorz wrote: > > > --On 15 May 2014 08:50:25 -0600 "Janky Jay, III" wrote: > >> I have set up one of the FBSD systems to run OpenSM and also be an >> NFS >> server which all the systems seem to be able to mount over the IB devices >> without any issue at all. Small reads and writes to and from the NFS >> server to all the other nodes also seems to work without any issue. >> However, if I try to dump large amounts of data using "dd" (in order to >> test speeds and stability), the FBSD NFS client craps out immediately. I >> just get the following message(s) over and over: >> >> newnfs server 10.11.1.1:/data: not responding >> newnfs server 10.11.1.1:/data: not responding > > Can both sides 'ping' each other when this happens? > I just tested this while node2 was hanging with another NFS transfer (just a "cp /home/file /data/file") and both nodes (1 and 2) can ping each other without any issues. > The reason I ask is I've hit a similar issue setting up ZFS over iSCSI > on IPOIB (I'm not running connected mode). > > At my end it looks like an ARP expires or something so the sides 'lose > site' of each other. If 'A' can't see 'B' - a ping from 'B' to 'A' > usually restores the connection. > I've actually seen this a lot with OpenVPN as well. For some reason, the ARP does seem to expire or something and I can no longer reach other systems on the LAN. And, like you, simply pinging the IP/hostname resolves the issue and I am able to connect to services on the LAN again... Very strange. > Maybe make sure they can both still see each other outside of nfs - I > can temporarily 'fix' the issue here by leaving both sides pinging each > other - I've not really had a chance to look at it much recently... > I'll leave them both pinging each other for a while and see if the transfer ends up finishing. Thanks for the info! Also, I found a lot of the below in the /var/log/messages log files on both FBSD systems over and over but they seem very random (time-wise): May 15 03:14:26 node1 kernel: "received MAD: slid:4 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 15\n" May 15 03:17:43 node1 kernel: "received MAD: slid:3 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 15\n" May 15 03:17:50 node1 kernel: ib0: timing out; 2 sends not completed May 15 03:18:22 node1 kernel: "received MAD: slid:4 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 10\n"<7>"received MAD: slid:4 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 14\n" May 15 03:19:26 node1 kernel: "received MAD: slid:4 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 15\n" May 15 03:26:10 node1 kernel: "received MAD: slid:4 sqpn:1 " "dlid_bits:0 dqpn:1 wc_flags:0x0, cls 7, mtd 3, atr 15\n" Regards, Janky Jay, III From owner-freebsd-infiniband@FreeBSD.ORG Thu May 15 20:34:10 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6785DE7 for ; Thu, 15 May 2014 20:34:10 +0000 (UTC) Received: from morbid.purplehat.org (morbid.purplehat.org [206.225.82.173]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47D0E20B5 for ; Thu, 15 May 2014 20:34:09 +0000 (UTC) Received: from localhost (morbid.purplehat.org [206.225.82.173]) by morbid.purplehat.org (Postfix) with ESMTP id 230DBD8CCB4 for ; Thu, 15 May 2014 13:34:08 -0700 (MST) Received: from morbid.purplehat.org ([206.225.82.173]) by localhost (morbid.purplehat.org [206.225.82.173]) (maiad, port 10024) with ESMTP id 86642-06 for ; Thu, 15 May 2014 13:34:03 -0700 (MST) Received: from [172.17.0.60] (fw1.aspsys.com [173.14.5.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jankyj@unfs.us) by morbid.purplehat.org (Postfix) with ESMTPSA id 05F86D8CCB3 for ; Thu, 15 May 2014 13:34:02 -0700 (MST) Message-ID: <537524A2.1010901@unfs.us> Date: Thu, 15 May 2014 14:33:38 -0600 From: "Janky Jay, III" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Re: FBSD to FBSD NFS Mounts over IB. References: <5374D431.5020501@unfs.us> <55BC554716A7EA5C54F5DD02@study64.tdx.co.uk> <5374F7EA.6060505@unfs.us> In-Reply-To: <5374F7EA.6060505@unfs.us> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2014 20:34:10 -0000 On 05/15/2014 11:22 AM, Janky Jay, III wrote: > Hello Karl, > > On 05/15/2014 10:50 AM, Karl Pielorz wrote: >> >> >> --On 15 May 2014 08:50:25 -0600 "Janky Jay, III" wrote: >> >>> I have set up one of the FBSD systems to run OpenSM and also be an >>> NFS >>> server which all the systems seem to be able to mount over the IB >>> devices >>> without any issue at all. Small reads and writes to and from the NFS >>> server to all the other nodes also seems to work without any issue. >>> However, if I try to dump large amounts of data using "dd" (in order to >>> test speeds and stability), the FBSD NFS client craps out immediately. I >>> just get the following message(s) over and over: >>> >>> newnfs server 10.11.1.1:/data: not responding >>> newnfs server 10.11.1.1:/data: not responding >> >> Can both sides 'ping' each other when this happens? >> > > I just tested this while node2 was hanging with another NFS > transfer (just a "cp /home/file /data/file") and both nodes (1 and 2) > can ping each other without any issues. > Quick update: I rebuild the kernel without the IPOIB_CM to get rid of connected mode and everything seems to be working fine now. Hopefully this will turn out to be as stable as I need and we can use ZFS for cluster storage. Thanks again for the replies and help, Karl! Regards, Janky Jay, III From owner-freebsd-infiniband@FreeBSD.ORG Tue May 27 13:46:45 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDB4FA27 for ; Tue, 27 May 2014 13:46:45 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CBC72BA8 for ; Tue, 27 May 2014 13:46:45 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id rp18so7255242iec.40 for ; Tue, 27 May 2014 06:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=WRF0l8c4KCT7KjqLU24u0ItiRvUbjQSBojnmqntPmNY=; b=mLZsW3T6u3rb6JtXU0I5D8eXIklrbiPX4aFZdyV0zDIf2+M0fFujyv+h0mjJX91cnB 6f9HWu9d9z+XlFyDQ2Ai6jIc36nOP/tkOGnq9Rl34rgbkhKkcAFRSaQMhQ6bJz1Qe4v3 TcaJ9sSNDFK8OB0WL/sjhWQrnWk2L81BTyUCX4yFoaLz0bN++9ZR0GfA5t2hi9j/i4fu aBpryGv3l03FdCaRzMyxk9YfHLbShupK7tEL1Mm3H66WELeZMB9a6ww3+pvKm+97Yx6F u6tBeDWsrrruSoLZXMTVwzIlX5uaFpZFjeNzyzHxrgQMQcDtOZTg70wR+hNp3YWteHd3 8dYA== X-Received: by 10.50.147.101 with SMTP id tj5mr34074468igb.27.1401198405056; Tue, 27 May 2014 06:46:45 -0700 (PDT) Received: from [10.1.68.236] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id ri2sm7858423igc.1.2014.05.27.06.46.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 27 May 2014 06:46:41 -0700 (PDT) Message-ID: <5384973F.8030304@gmail.com> Date: Tue, 27 May 2014 09:46:39 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: basic questions on using Mellanox IB/40GbE converged cards Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2014 13:46:45 -0000 I'm not quite sure if this belongs in freebsd-infiniband or another list, such as freebsd-net or -questions. If it should be filed elsewhere, my apologies. I am attempting to prototype a 40GbE router/packet filter, using FreeBSD 10.0 as the base operating system, with the test platform being a Dell R820. I have three dual port Mellanox cards in the system, one a ConnectX-3, the others OEM-branded; and now attempting performance tests. I'm hoping to get answers to the following questions: A) As the cards come up, I see, on each interface, "64B EQEs/CQEs supported by the device but not enabled". My googling has not yielded FreeBSD-specific results on this. Can 64B EQEs and CQEs be enabled under FreeBSD? Will it help with performance across multiple cards? If so, how can 64B EQE and CQEs be enabled? B) I also see "Using 5 TX rings" and "Using 2 RX rings" as the cards come up. Is this tunable under FreeBSD? If it is tunable, should it be tuned for a router? Maybe 4 TX and 3 RX, perhaps? And, if it is tunable and should be tuned, how is this accomplished? It appears hw.mlxen$N.conf.rx_rings and hw.mlxen$N.conf.tx_rings are read-only sysctls. Does that mean /boot/loader.conf? C) network tuning guides for FreeBSD are easy to find. Ones discussing 40GbE are not. Are there specific recommendations to obtain higher performance out of these cards, either general or specific to the mlx modules, that I should be using? D) In my tests, I am seeing various TCP retransmissions, which decrease throughput. What can be done to mitigate those? Thanks in advance for any assistance! -- John Jasen (jjasen@gmail.com) From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 4 19:46:05 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B80BC5F for ; Wed, 4 Jun 2014 19:46:05 +0000 (UTC) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AB4828CC for ; Wed, 4 Jun 2014 19:46:05 +0000 (UTC) Received: by mail-ig0-f179.google.com with SMTP id hn18so1475944igb.0 for ; Wed, 04 Jun 2014 12:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=QDCK6mf/62leAxUyR99FkaZAe3Wf5DHTwOTacFgsbLA=; b=boSQoEL7pRLAqrqd3jENUbP6oR5B4EevO8wn/ExOQGidOG0JeVwEkgMHx8PhoqiofZ o/uoInj2yVRztenRLzh/+zPabbcou9Q1mhHi//K4vxPNoLu9Kr0xpS4TRihgOz5+ChEp PKlg6SvT4COByPAik7kkfWZbVmCzx57Qwt7dhNzVqKIkWm6OLQcyySM7WS78XRYfv8nb Rjb2X20mLzBnAsCQ8Pf5WQf0nhyOIVhSkCcODTgq996m4JcRi2/oAHdZBDHFuTnxOO2O KcCFAz79K3oDOMH89zSxQXJ8WeQ6NmFmazZDexnjVYWt6AXyLD4BNWYLJkbBrVAiyu+Z IJjg== X-Received: by 10.50.46.35 with SMTP id s3mr10884839igm.40.1401911164633; Wed, 04 Jun 2014 12:46:04 -0700 (PDT) Received: from [10.1.68.236] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id j1sm12690869ige.0.2014.06.04.12.46.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 04 Jun 2014 12:46:03 -0700 (PDT) Message-ID: <538F7775.8040701@gmail.com> Date: Wed, 04 Jun 2014 15:45:57 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: side note: sysctl with mlxen devices can cause kernel panics Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jun 2014 19:46:05 -0000 Forgive the short report. I imagine this is easily reproduced if its a software error -- less so if the nut on the end of the keyboard needs adjustment. I was trying to loop through the mlxen cards I have on the system, basically tweaking hw.mlxenNUM.conf.rx_size, tx_size, pkt_rate_low and pkt_rate_high. It seems doing this too often or perhaps doing it in the middle of several data streams can be hazardous -- namely that it will panic the box. There have been times I've been able to tickle those sysctl settings multiple times without issue, and there have been times like today, where the system panicked on my first loop. I can provide info from /var/crash on it, for anyone who wants it. Thanks! From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 4 19:56:32 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98F39F70 for ; Wed, 4 Jun 2014 19:56:32 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 668CD29C9 for ; Wed, 4 Jun 2014 19:56:32 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id rp18so7404289iec.26 for ; Wed, 04 Jun 2014 12:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=LjP1R/maDESsc1ccnSq/Y+mZTPbJ14ziuq5MsX0QLPE=; b=Xh0F60IQVtGGB2B7Lt0c5EUg6R/AlHa1lH8mv7QBs36ddORjFq/8K6R3FN70K0tpC+ lc7UTaDbCtHt4Txv5ktwhXt26UkJIy9g0WKTQAh37NZnL+vhWPC1yeNNeJjo2Wb5x09U u/w8mTbDInpNLaIV7D6Vbp3MzLyZG2zHkuDNPjL3DMzYSH9bF5T18F3GV/UfooFBsbSq DYTepx560DxpRh1pVMgc2LafL+qb2HrV8jCyzhLm0vba1RZ8NlYqO8nh0+cqTaOdH3YH JRNSTyqucENpDWEcsOdvAcjb6LzFtuXklu7NEGP0WR9xzOaC9UXjNqnfNSdgaGqQsjCT xS6w== X-Received: by 10.51.17.66 with SMTP id gc2mr6508230igd.5.1401911791369; Wed, 04 Jun 2014 12:56:31 -0700 (PDT) Received: from [10.1.68.236] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id ie13sm12734366igb.10.2014.06.04.12.56.30 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 04 Jun 2014 12:56:30 -0700 (PDT) Message-ID: <538F79EC.6050900@gmail.com> Date: Wed, 04 Jun 2014 15:56:28 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Re: basic questions on using Mellanox IB/40GbE converged cards References: <5384973F.8030304@gmail.com> In-Reply-To: <5384973F.8030304@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jun 2014 19:56:32 -0000 On 05/27/2014 09:46 AM, John Jasen wrote: > B) I also see "Using 5 TX rings" and "Using 2 RX rings" as the cards > come up. > > Is this tunable under FreeBSD? I imagine it would be good if these were tunable. From reading the sysctl output on the cards (hw.mlxenNUM.stat.{rx|tx}_ringNUM), tx_ring0 and 1 are heavily used, sometimes 4, but 2 and 3 are empty. rx_ring0 and 1 are both heavily used. I have not found a way to tune it, either via sysctl or loader.conf. Thanks! From owner-freebsd-infiniband@FreeBSD.ORG Tue Jun 10 14:17:58 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 755F5A95 for ; Tue, 10 Jun 2014 14:17:58 +0000 (UTC) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 412872AF3 for ; Tue, 10 Jun 2014 14:17:58 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id at1so3683782iec.28 for ; Tue, 10 Jun 2014 07:17:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=fJoq5oCXz+CDP7Am1v0qcdYImqfHCzxBdg+iKkDSguM=; b=LETKGVpEcOIzlcztAKK+pcUlzS7JY6Nz4hHIC6ghwJ//ap5l60wAcj2MY1vsaS41/N pH+p2c8+z54yiw/pOEaOL/SXt3iWAUVD1GpEOyF8ZBVGiL510FEdVhyMEvztEE+NRZF/ 6WWTor8+d1Ss978CRMSeJSSww5LN89fic+3KBb8zzeeUYjdgSTE1RTOb3TdzEZx4s8Ji iQ/kt+3UVMT827PvJ5lBifLQvAWCcDRdp/LFUcT6pQ8DOdrHByVwu7vt3Kn1JR0tIhZv pVDwE+Vv+rwpMfY70EASSaLd9rdNwoqoTXAg8whbsgbnlXxf8pKxUxpQxx5X2lNAlqJ6 cgyQ== X-Received: by 10.42.201.84 with SMTP id ez20mr38438033icb.13.1402409877688; Tue, 10 Jun 2014 07:17:57 -0700 (PDT) Received: from [10.1.68.236] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id ng14sm65646119igb.19.2014.06.10.07.17.53 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 10 Jun 2014 07:17:53 -0700 (PDT) Message-ID: <53971390.4070305@gmail.com> Date: Tue, 10 Jun 2014 10:17:52 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: mlxen drivers/adapters: potential problem: inactivity requires ifconfig nudge to wake up? X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2014 14:17:58 -0000 I came back to my test environment after a long weekend of it being idle, and resumed some bandwidth/packets-per-second testing where I left off last week. The tests all failed, with errors similar to 'unable to connect'. I could generally ping through the interfaces, but trying to tickle a TCP port timed out. The first interface, I did a manual ifconfig down and ifconfig up to bring it back to life. I looped through most of the interfaces; adjusting the mlxenSOMENUM tx_size and rx_size (which brings the interface down and up when changed), and now my tests are working again. I've encountered this before if the test equipment was left alone overnight or so. Any pointers as to what could be the cause? From owner-freebsd-infiniband@FreeBSD.ORG Tue Jul 8 12:24:17 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 742E4294 for ; Tue, 8 Jul 2014 12:24:17 +0000 (UTC) Received: from mail-oa0-f41.google.com (mail-oa0-f41.google.com [209.85.219.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C8512CD2 for ; Tue, 8 Jul 2014 12:24:13 +0000 (UTC) Received: by mail-oa0-f41.google.com with SMTP id l6so6326435oag.0 for ; Tue, 08 Jul 2014 05:24:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=CsAn4ZVDpJg/8EOBmEeEc2ThxwX4qA45OFzuaGzhq1g=; b=MY6p/n8GvNVB6ZXPwFIXQUdsLblo/gvRMTyVOWbegOSPDRe34LpKBF5fQiHpSWVLvu fJxgp0fkQUn7IeKYtHvp3cpIbIEaEhkQ0Pwf0n65MZD21eDVwNVILm/O4pXdre9AVV7u KGKCnDwU7R8htvKKkQk2+efMHz6WotBoAQdMaKge+9nVGUnsUsSsyugZ0bytgYqQDohZ 9MAlcoOrQBtKJljBUbhdmXufnMejZHNCCyR8M917DRq4D4WqoBd3MQ819Kg5EfyH2Asx /DiuiZLS1JVh0/MgjDyH/PDVt3jAmOBU2HAl8ctEUZ60PDOh7ryoUazoPBXEUnXPLraS /NZQ== X-Gm-Message-State: ALoCoQmCBT5VAniRQehVacFe5VzkeCxkGaSfRvd54iwUYz5AmLvFJffoIv7Bo7r8XWH8vdsgS9E0 MIME-Version: 1.0 X-Received: by 10.60.97.66 with SMTP id dy2mr19448433oeb.50.1404822247646; Tue, 08 Jul 2014 05:24:07 -0700 (PDT) Received: by 10.76.93.106 with HTTP; Tue, 8 Jul 2014 05:24:07 -0700 (PDT) Date: Tue, 8 Jul 2014 17:54:07 +0530 Message-ID: Subject: rping, krping and libibverbs From: Sourish Mazumder To: freebsd-infiniband@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 12:24:17 -0000 Does rping and krping work with freebsd9.2, mellanox connectX3 and OFED? What is the extent of support for libibverbs in freebsd. I see that userspace rdma binaries are not working because the library is not able to find the device - /sys/class/infiniband/uverbs0. The libibverbs README also mentions about a device /dev/infiniband/uverbs0. In my freebsd9.2, i only see the /dev/uverbs0 device. I see that the code which creates the device - /sys/class/infiniband/uverbs0, is commented in the kernel. Are userspace rdma verbs, well supported on freebsd? If not, what is the best way to implement rdma data transfer in the kernel? Write your own ULP? -- Sourish Mazumder CloudByte Inc. From owner-freebsd-infiniband@FreeBSD.ORG Wed Jul 9 09:10:44 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C134E82; Wed, 9 Jul 2014 09:10:44 +0000 (UTC) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3lp0076.outbound.protection.outlook.com [213.199.154.76]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DF5552502; Wed, 9 Jul 2014 09:10:42 +0000 (UTC) Received: from DB3PR05MB202.eurprd05.prod.outlook.com (10.242.133.13) by DB3PR05MB025.eurprd05.prod.outlook.com (10.242.137.12) with Microsoft SMTP Server (TLS) id 15.0.980.8; Wed, 9 Jul 2014 09:10:28 +0000 Received: from DB3PR05MB202.eurprd05.prod.outlook.com ([169.254.6.186]) by DB3PR05MB202.eurprd05.prod.outlook.com ([169.254.6.186]) with mapi id 15.00.0980.000; Wed, 9 Jul 2014 09:10:28 +0000 From: Shahar Klein To: "owner-freebsd-infiniband@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: RE: rping, krping and libibverbs Thread-Topic: rping, krping and libibverbs Thread-Index: AQHPmqeQvf25VRrrXkutPZ2y4tLuzpuWNjIggAABGrCAAABwAIAAAGqg Date: Wed, 9 Jul 2014 09:10:27 +0000 Message-ID: <05b7a9ef6b1142d3a1eee3a3363049ab@DB3PR05MB202.eurprd05.prod.outlook.com> References: <477a09cbf2ff449698b47f9bb364e1de@DB3PR05MB202.eurprd05.prod.outlook.com> <54f761f1e7cb41389c9407a55787819f@DB3PR05MB026.eurprd05.prod.outlook.com> In-Reply-To: <54f761f1e7cb41389c9407a55787819f@DB3PR05MB026.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [193.47.165.251] x-microsoft-antispam: BCL:0;PCL:0;RULEID: x-forefront-prvs: 0267E514F9 x-forefront-antispam-report: SFV:NSPM; SFS:(6009001)(189002)(199002)(377454003)(13464003)(20776003)(92566001)(64706001)(74662001)(77982001)(79102001)(15975445006)(99396002)(31966008)(76482001)(33646001)(80022001)(76176999)(86362001)(74502001)(77096002)(101416001)(107046002)(54356999)(83322001)(46102001)(50986999)(81342001)(93886003)(81542001)(85306003)(66066001)(74316001)(19580395003)(4396001)(95666004)(105586002)(106116001)(21056001)(76576001)(19580405001)(15202345003)(87936001)(106356001)(2656002)(85852003)(83072002)(108616002)(24736002); DIR:OUT; SFP:; SCL:1; SRVR:DB3PR05MB025; H:DB3PR05MB202.eurprd05.prod.outlook.com; FPR:; MLV:sfv; PTR:InfoNoRecords; MX:1; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2014 09:10:44 -0000 Sourish, Please try this: 1. cd $SRC/contrib/ofed/libibverbs/examples 2. create a Makefile.fbsd: libibverbs/examples >diff Makefile Makefile.fbsd=20 1c1 < CFLAGS=3D -I../../../../sys/ofed/include -libverbs -lmlx4 -lmthca -pthrea= d --- > CFLAGS=3D -libverbs -lmlx4 -lmthca -pthread 3. edit the files: asyncwatch.c device_list.c devinfo.c and comment out the= se lines: a. //#include b. //#include 4. Run make -f Makefie.fbsd -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Sourish Mazumder Sent: Tuesday, July 08, 2014 3:24 PM To: freebsd-infiniband@freebsd.org Subject: rping, krping and libibverbs Does rping and krping work with freebsd9.2, mellanox connectX3 and OFED? What is the extent of support for libibverbs in freebsd. I see that userspa= ce rdma binaries are not working because the library is not able to find th= e device - /sys/class/infiniband/uverbs0. The libibverbs README also mentio= ns about a device /dev/infiniband/uverbs0. In my freebsd9.2, i only see the /dev/uverbs0 device. I see that the code which creates the device - /sys/class/infiniband/uverbs= 0, is commented in the kernel. Are userspace rdma verbs, well supported on freebsd? If not, what is the be= st way to implement rdma data transfer in the kernel? Write your own ULP? -- Sourish Mazumder CloudByte Inc. _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g"