From owner-freebsd-infiniband@FreeBSD.ORG Wed Mar 19 20:22:03 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 471EFBC5 for ; Wed, 19 Mar 2014 20:22:03 +0000 (UTC) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0C606981 for ; Wed, 19 Mar 2014 20:22:02 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id lx4so9216282iec.24 for ; Wed, 19 Mar 2014 13:21:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=JRr0TvwUyOTY0JvKdaD63CGIfxqhJTcBFDtISGYpTyk=; b=V/IOCD1lA4/wITcWzjkYGEg4i/BAwyDzKHD0vDyKb0P+xeX6Nj0W7WgS+g0+xgu3/P inVff3T4iIaSIPUtFjtOGokPvUfJyLLggGld2QO6DUQ+u+sE9YsDDd9jMT8YnIqW3v/Y 2aOa6LybvzkBfjEwNg5wJSROaqmJ5TE2TdxEQaOiExbKunjii8X2LnsmSs5DUmKniemb dITxku+CtjprHWiEQDQ7meYW/KIS/YLF5Zh0Z8oVNaOC5A7h1Q9Sc+53u8qMjOWTU+eQ a1zSpX2hLIsAnNyp15s4+AYJHhC9l2MgXWVAbMC+gh/vF7ckMQ9Oj8NrJ0XyRqG+rlkd pLng== X-Gm-Message-State: ALoCoQncWahDS/nWnjWwMOA5I78lUqQ/Oa7fclDhS2uknFAVhEPwKTmMfobabIyUaSWff+jiz0NX X-Received: by 10.43.106.137 with SMTP id du9mr74959icc.93.1395260516421; Wed, 19 Mar 2014 13:21:56 -0700 (PDT) Received: from freebie.acadix.biz (h69-131-101-133.nwblwi.broadband.dynamic.tds.net. [69.131.101.133]) by mx.google.com with ESMTPSA id mi2sm21959igb.3.2014.03.19.13.21.53 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 19 Mar 2014 13:21:55 -0700 (PDT) Message-ID: <5329FC5E.4080305@tds.net> Date: Wed, 19 Mar 2014 15:21:50 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Jason Bacon , Oded Shanoon Subject: Re: Supported NICs References: <52361685.8000901@tds.net> <52370835.7020901@tds.net> <058596FC127BDB469D7D60F04E37C63F7351B162@MTLDAG02.mtl.com> <525C39C0.6050004@uwm.edu> <058596FC127BDB469D7D60F04E37C63F73526997@MTLDAG02.mtl.com> <5329D338.3060103@uwm.edu> In-Reply-To: <5329D338.3060103@uwm.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2014 20:22:03 -0000 Also, ibping is very slow in both directions: root@raid1:~ # ibping 28 Pong from compute-1-15.(none) (Lid 28): time 7.481 ms Pong from compute-1-15.(none) (Lid 28): time 4.465 ms Pong from compute-1-15.(none) (Lid 28): time 11.933 ms Pong from compute-1-15.(none) (Lid 28): time 21.255 ms Pong from compute-1-15.(none) (Lid 28): time 8.672 ms Pong from compute-1-15.(none) (Lid 28): time 13.633 ms [root@compute-1-15 ~]# /usr/sbin/ibping 163 Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 4.035 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.690 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.721 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.776 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 24.457 ms Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.723 ms On 03/19/14 12:26, Jason Bacon wrote: > > We finally got a new server in with a ConnectX-3 card. It seems to > work reliably with our SilverStorm DDR network using a CX4 - QSFP cable. > > Below are specs and iperf benchmarks between the FreeBSD server and a > CentOS 6.5 compute node with a Qlogic DDR card. Connected mode with > MTU 65520. FreeBSD 10.0-RELEASE on a Dell R720xd. > > Wondering if anyone has any suggestions about what might cause the > poor output performance. Any sysctls or loader.conf settings I should > look into? None of the kern.conftext flags mentioned below are present. > > Any pointers on how to diagnose the issue would be appreciated as well. > > Thanks, > > Jason > > Input throughput is excellent - over 11 gb/sec. Linux to Linux on the > same network does about 10 gb/sec. > > [root@compute-5-33 ~]# iperf -c 10.1.2.250 > ------------------------------------------------------------ > Client connecting to 10.1.2.250, TCP port 5001 > TCP window size: 645 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.1.2.140 port 54223 connected with 10.1.2.250 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 13.0 GBytes 11.2 Gbits/sec > > > bacon@raid1:~ % netstat 1 > input (Total) output > packets errs idrops bytes packets errs bytes colls > 10631 0 0 685149597 5390 0 409806 0 > 22281 0 0 1452318244 11136 0 846730 0 > 23776 0 0 1557833160 11884 0 903654 0 > 23891 0 0 1565368420 11952 0 907986 0 > 23328 0 0 1528478408 11663 0 886554 0 > > Output throughput is terrible: > > [root@compute-5-33 ~]# iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.1.2.140 port 5001 connected with 10.1.2.250 port 50207 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.9 sec 1.50 MBytes 1.16 Mbits/sec > > > bacon@raid1:~ % netstat 1 > input (Total) output > packets errs idrops bytes packets errs bytes colls > 20 0 0 1444 27 1 124684 0 > 5 0 0 284 1 4 393718 0 > 5 0 0 284 1 4 393718 0 > 6 0 0 344 1 4 393718 0 > 6 0 0 344 18 4 426562 0 > 5 0 0 284 1 4 393718 0 > > root@raid1:/home/bacon # ibstat > CA 'mlx4_0' > CA type: MT4099 > Number of ports: 1 > Firmware version: 2.11.500 > Hardware version: 0 > Node GUID: 0xf4521403007d2c10 > System image GUID: 0xf4521403007d2c13 > Port 1: > State: Active > Physical state: LinkUp > Rate: 20 > Base lid: 163 > LMC: 0 > SM lid: 1 > Capability mask: 0x02514868 > Port GUID: 0xf4521403007d2c11 > > root@raid1:/home/bacon # ibportstate -G 0xf4521403007d2c11 query > PortInfo: > # Port info: Lid 163 port 0 > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkWidthSupported:..............4X (IBA extension) > LinkWidthEnabled:................4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps > LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps > LinkSpeedActive:.................5.0 Gbps > > > > > On 10/16/13 10:23, Oded Shanoon wrote: >> Please run sysctl kern.conftxt and grep for the following options: >> >> OFED_DEBUG_INIT >> IPOIB_DEBUG >> DEADLKRES >> INVARIANTS >> INVARIANT_SUPPORT >> WITNESS >> WITNESS_SKIPSPIN >> >> These could result in performance degredation >> >> Regards, >> >> Oded Shanoon >> OFED-FreeBSD Team Leader >> Mellanox Technologies, Raanana >> >> >> -----Original Message----- >> From: Jason Bacon [mailto:bacon@uwm.edu] >> Sent: Monday, October 14, 2013 9:37 PM >> To: Oded Shanoon >> Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband@freebsd.org >> Subject: Re: Supported NICs >> >> >> Some initial test results... >> >> I installed an old Infinihost DDR HCA in one of our compute nodes >> running FreeBSD 9.1. >> >> RHEL nodes are using qlogic IB HCAs. >> >> 10.1.1 is gigabit Ethernet, 10.1.2 is IB. >> >> Running iperf server on FreeBSD and client on one of our RHEL nodes >> shows OK performance: >> >> bacon@infinibsd:/home/bacon % iperf -s >> ------------------------------------------------------------ >> Server listening on TCP port 5001 >> TCP window size: 64.0 KByte (default) >> ------------------------------------------------------------ >> [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947 >> [ ID] Interval Transfer Bandwidth >> [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec >> >> [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 >> [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec >> >> RHEL to RHEL gives us about 8 Gbits/sec. >> >> Iperf server on RHEL and client on FreeBSD shows very poor >> performance for IB, while GigE is fine: >> >> bacon@infinibsd:/home/bacon % iperf -c 10.1.1.39 >> ------------------------------------------------------------ >> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 >> KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec >> >> bacon@infinibsd:/home/bacon % iperf -c 10.1.2.39 >> ------------------------------------------------------------ >> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 >> KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec >> >> Any ideas what might cause this? >> >> I'm going to install 9.2-RELEASE and retest in any case, plus explore >> the ib config tools, but any feedback in the meantime would be >> appreciated. >> >> Thanks, >> >> JB >> >> On 10/07/13 01:51, Oded Shanoon wrote: >>> Hi Jason, >>> >>> IB support in 9.2 is supposed to be stable enough. >>> Please note that Mellanox started supporting FreeBSD only recently. >>> The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) >>> from OFA-1.5.3. >>> Since we started our involvement we entered some fixes to that >>> driver (that were submitted into 9.2). >>> We also mapped various issues which needs to be fixed in the future. >>> We are now working on a major "face lift" to the driver - making it >>> much more stable and with improved performance and features. >>> >>> Regards, >>> >>> Oded Shanoon >>> OFED-FreeBSD Team Leader >>> Mellanox Technologies, Raanana >>> >>> >>> -----Original Message----- >>> From: owner-freebsd-infiniband@freebsd.org >>> [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Jason Bacon >>> Sent: Monday, September 16, 2013 4:32 PM >>> To: Anthony Cornehl >>> Cc: freebsd-infiniband@freebsd.org >>> Subject: Re: Supported NICs >>> >>> >>> Thanks Anthony& Oded! >>> >>> Would you say that IB support in 9.2 is stable enough for a >>> production HPC file server? >>> >>> Will do plenty of my own testing first, of course. >>> >>> Regards, >>> >>> Jason >>> >>> On 09/15/13 22:38, Anthony Cornehl wrote: >>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl">>> > wrote: >>>>> On Sep 15, 2013 1:20 PM, "Jason bacon">>> > wrote: >>>>>> Is there a list of supported IB NICs out there somewhere? >>>>>> >>>>>> I followed the wiki instructions for rebuilding with IB support >>>> and now have mlx4ib, mlxen, etc. >>>>>> Was hoping there would be man pages for the drivers that list >>>> known working cards, but there don't seem to be. I'm hoping to test >>>> a file server using IPOIB and possible roll a FreeNAS ISO with IB >>>> support if it works out. >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> Jason W. Bacon >>>>>> jwbacon@tds.net >>>>>> >>>>>> Circumstances don't make a man: >>>>>> They reveal him. >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> _______________________________________________ >>>>>> freebsd-infiniband@freebsd.org >>>> mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>>>>> To unsubscribe, send any mail to >>>> "freebsd-infiniband-unsubscribe@freebsd.org >>>> " >>>>> Hey Jason, >>>>> >>>>> Mellanox ConnectX series cards are the only ones supported >>>>> currently... >>>>> >>>>> http://www.mellanox.com/page/infiniband_cards_overview >>>>> >>>>> Don't forget to compile with IPoIB as well, since the IPoIB support >>>> is compiled into the driver, unlike in Linux. >>>>> Just be aware that IPoIB performance is also improved by enabling >>>> connected mode when you compile the kernel module. The IB code in >>>> FreeBSD is a few years older the what is in Linux, but, the following >>>> forum thread is probably relevant... >>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo >>>> r >>>> mance-problems.html >>>>> Cheers! >>>> It also appears that Jeff fixed SDP a few weeks ago, which is more >>>> capable of reaching line-speed for IB-connected devices. >>>> >>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html >>>> >>>> Cheers! >>>> >>> -- >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Jason W. Bacon >>> jwbacon@tds.net >>> >>> Circumstances don't make a man: >>> They reveal him. >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> _______________________________________________ >>> freebsd-infiniband@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>> To unsubscribe, send any mail to >>> "freebsd-infiniband-unsubscribe@freebsd.org" >> >