From owner-freebsd-infiniband@freebsd.org Thu May 26 19:55:54 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 033D7B4B6CF for ; Thu, 26 May 2016 19:55:54 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail2.jnielsen.NET (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D0F2B11CC for ; Thu, 26 May 2016 19:55:53 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from [10.180.26.108] (mobile-166-171-249-213.mycingular.net [166.171.249.213]) (authenticated bits=0) by webmail2.jnielsen.NET (8.15.2/8.15.2) with ESMTPSA id u4QJtnhb054591 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 26 May 2016 13:55:52 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.NET: Host mobile-166-171-249-213.mycingular.net [166.171.249.213] claimed to be [10.180.26.108] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: InfiniBand supported hw From: John Nielsen X-Mailer: iPhone Mail (13F69) In-Reply-To: <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> Date: Thu, 26 May 2016 13:55:47 -0600 Cc: freebsd-infiniband@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <7C7D1835-52DD-4F27-B0CA-A59650E59639@jnielsen.net> References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> To: Grzegorz Junka X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 19:55:54 -0000 > On May 26, 2016, at 12:33 PM, Grzegorz Junka wrote: >=20 >=20 >> On 26/05/2016 17:44, John Nielsen wrote: >>> On May 26, 2016, at 11:07 AM, Justin Clift wrote= : >>>=20 >>>> On 26 May 2016, at 14:00, Grzegorz Junka wrote: >>>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version an= d it is the same as Mellanox ConnectX, which supposed to be supported. >>> Yep, they should work. You may need to flash them with their Mellanox e= quivalent >>> firmware, but I'm not sure. (doing that flash isn't very hard btw) >>>=20 >>>> Also, most InfiniBand cards provide two ports. When connecting two comp= uters together (IPoIB) is it possible to use both ports to double the speed?= >>> Not sure, as I've not needed to personally. None of my storage (or othe= r uses) >>> even fills one DDR port. Yet. :) >>>=20 >>> I *think* the answer to that is yes, >> It depends on what you mean by "use both ports." You can of course use bo= th ports at the same time, but if you're envisioning something like an IPoIB= version of an Ethernet LAGG I don't think that is supported. >>=20 >> If you are using the IB for storage protocols, then active/active multipa= thing is one way to utilize the full bandwidth of multiple ports. >>=20 >>>> Lastly, is NFS RDMA properly supported on FreeBSD? >>> Again not sure. Other people here would likely know. :) >=20 > Yes, that's exactly what I was hoping for, a sort of LAGG to double the sp= eed of the link between computer by utilizing both ports together. LAGG is i= ndependent from the network interface, if I could set up an IPoIB NIC for ea= ch port separately what could prevent me from creating a LAGG interface on t= op of them? And if that's not supported, what do people usually do with the o= ther interface? Just leave it unconnected? LAGG is designed to work with Ethernet. InfiniBand is a very different layer= 2 technology with its own advantages and disadvantages. I don't want to get= too far out of my depth so I'll just suggest you do your own research on th= e subject. How you use the second port is entirely up to you. You can use it for separa= te applications, for redundancy or higher throughput. In my experience it us= ually makes sense to give it its own IP. If you want higher bandwidth than a= single port can provide then you should at layer 3 or higher for ways to ag= gregate bandwidth. (You should also do some research on PCI-e throughput. A x= 8 PCI-e 3.0 slot maxes out well before 112Gbit/s.) As I mentioned previously you can use active/active multipath to get more th= roughput for block storage applications. You could also run a routing daemon= and do ECMP for any layer 3 traffic. Or your particular application may hav= e other ways to intelligently use two distinct IPoIB paths.