Date: Thu, 26 May 2016 13:55:47 -0600 From: John Nielsen <lists@jnielsen.net> To: Grzegorz Junka <list1@gjunka.com> Cc: freebsd-infiniband@freebsd.org Subject: Re: InfiniBand supported hw Message-ID: <7C7D1835-52DD-4F27-B0CA-A59650E59639@jnielsen.net> In-Reply-To: <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <A69E3E1D-002C-47CA-A898-0ABF67F64D5B@postgresql.org> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On May 26, 2016, at 12:33 PM, Grzegorz Junka <list1@gjunka.com> wrote: >=20 >=20 >> On 26/05/2016 17:44, John Nielsen wrote: >>> On May 26, 2016, at 11:07 AM, Justin Clift <justin@postgresql.org> wrote= : >>>=20 >>>> On 26 May 2016, at 14:00, Grzegorz Junka <list1@gjunka.com> wrote: >>>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version an= d it is the same as Mellanox ConnectX, which supposed to be supported. >>> Yep, they should work. You may need to flash them with their Mellanox e= quivalent >>> firmware, but I'm not sure. (doing that flash isn't very hard btw) >>>=20 >>>> Also, most InfiniBand cards provide two ports. When connecting two comp= uters together (IPoIB) is it possible to use both ports to double the speed?= >>> Not sure, as I've not needed to personally. None of my storage (or othe= r uses) >>> even fills one DDR port. Yet. :) >>>=20 >>> I *think* the answer to that is yes, >> It depends on what you mean by "use both ports." You can of course use bo= th ports at the same time, but if you're envisioning something like an IPoIB= version of an Ethernet LAGG I don't think that is supported. >>=20 >> If you are using the IB for storage protocols, then active/active multipa= thing is one way to utilize the full bandwidth of multiple ports. >>=20 >>>> Lastly, is NFS RDMA properly supported on FreeBSD? >>> Again not sure. Other people here would likely know. :) >=20 > Yes, that's exactly what I was hoping for, a sort of LAGG to double the sp= eed of the link between computer by utilizing both ports together. LAGG is i= ndependent from the network interface, if I could set up an IPoIB NIC for ea= ch port separately what could prevent me from creating a LAGG interface on t= op of them? And if that's not supported, what do people usually do with the o= ther interface? Just leave it unconnected? LAGG is designed to work with Ethernet. InfiniBand is a very different layer= 2 technology with its own advantages and disadvantages. I don't want to get= too far out of my depth so I'll just suggest you do your own research on th= e subject. How you use the second port is entirely up to you. You can use it for separa= te applications, for redundancy or higher throughput. In my experience it us= ually makes sense to give it its own IP. If you want higher bandwidth than a= single port can provide then you should at layer 3 or higher for ways to ag= gregate bandwidth. (You should also do some research on PCI-e throughput. A x= 8 PCI-e 3.0 slot maxes out well before 112Gbit/s.) As I mentioned previously you can use active/active multipath to get more th= roughput for block storage applications. You could also run a routing daemon= and do ECMP for any layer 3 traffic. Or your particular application may hav= e other ways to intelligently use two distinct IPoIB paths.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7C7D1835-52DD-4F27-B0CA-A59650E59639>