From owner-freebsd-net@freebsd.org Fri May 11 19:58:59 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 685E8FC93B7 for ; Fri, 11 May 2018 19:58:59 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DB69A81006 for ; Fri, 11 May 2018 19:58:58 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w4BJwooE083993 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 11 May 2018 19:58:51 GMT (envelope-from list1@gjunka.com) Subject: Re: Unrecognized Inifiniband HCA To: freebsd-net@freebsd.org References: <169976f0-2159-9ef0-d956-b13e717ef783@gjunka.com> <7218e2c4b23ebf4553a352633fe49e72@postgresql.org> Cc: justin@postgresql.org From: Grzegorz Junka Message-ID: <1ed53554-a512-e4f3-3477-aed1d7860126@gjunka.com> Date: Fri, 11 May 2018 19:58:50 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB-large X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2018 19:58:59 -0000 On 10/05/2018 15:35, Justin Clift wrote: > On 2018-05-10 10:21, Grzegorz Junka wrote: >> On 08/05/2018 23:41, Justin Clift wrote: > >>> That's probably a ConnectX (series 1) Mellanox card.  Those can >>> operate in >>> either Infiniband mode, or Ethernet mode. >>> >>> Which mode are you wanting it to run in? :) >>> >>> As a thought, the FreeBSD wiki page has a bit of info: >>> >>>   https://wiki.freebsd.org/InfiniBand >>> >>> For that card to be recognised at all, it'll need the mlx4 driver(s) >>> to load. >>> >>> I don't remember the exact one off hand (it's been a while), but >>> some searching >>> online for mlx4 and FreeBSD should turn up the right bits. >> >> Many thanks Justin. This is the first time I am hearing about an >> Infiniband card operating in Ethernet mode. These cards come with two >> CX4/SFF 8470 ports. It's not possible to connect standard Ethernet >> cables that I know of (not even SFP modules). Do you mean that they >> can operate in Ethernet mode over the CX4 cable? > > Yep. :) > > Back in the day when these cards were current tech, CX4 was an acceptable > connector for 10GbE.  The Infiniband switches from that era (that I had > access to) were mostly Infiniband only though. > > But 10GbE CX4 switches did exist, and can still be found reasonably > cheaply > on Ebay.  eg: > >   * HP ProCurve 6 port CX4 10GBe switch >     https://www.ebay.com/itm/152232294328 > >   * HP ProCurve 48 port 1GbE switch, with 2x 10GbE CX4 ports on the back >     https://www.ebay.com/itm/281899832599 > > One of the good things about those HP ProCurve switches... being > enterprise > gear they just keep working.  For Years. > > From memory, HP still releases firmware security updates for them (for > free) to this day. Unlike (say) Cisco. ;) > > Note - As a data point, FreeNAS (based on FreeBSD) includes the > 10/40GbE driver > for these cards by default.  With FreeNAS they work as 10GbE "out of the > box". :D > > Hopefully that helps. :) Thanks Justin. That's an amazing piece of tech history :) GrzegorzJ