From owner-freebsd-net@freebsd.org Thu May 10 15:35:12 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DAD8EFCDED1 for ; Thu, 10 May 2018 15:35:12 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "meldrar.postgresql.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6BB6E74CAA for ; Thu, 10 May 2018 15:35:12 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org ([87.238.57.231] helo=webmail.postgresql.org) by meldrar.postgresql.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fGnb1-0004gL-Hg; Thu, 10 May 2018 15:35:10 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 10 May 2018 16:35:06 +0100 From: Justin Clift To: Grzegorz Junka Cc: freebsd-net@freebsd.org Subject: Re: Unrecognized Inifiniband HCA In-Reply-To: References: <169976f0-2159-9ef0-d956-b13e717ef783@gjunka.com> <7218e2c4b23ebf4553a352633fe49e72@postgresql.org> Message-ID: X-Sender: justin@postgresql.org User-Agent: Roundcube Webmail/1.2.3 X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2018 15:35:13 -0000 On 2018-05-10 10:21, Grzegorz Junka wrote: > On 08/05/2018 23:41, Justin Clift wrote: >> That's probably a ConnectX (series 1) Mellanox card.  Those can >> operate in >> either Infiniband mode, or Ethernet mode. >> >> Which mode are you wanting it to run in? :) >> >> As a thought, the FreeBSD wiki page has a bit of info: >> >>   https://wiki.freebsd.org/InfiniBand >> >> For that card to be recognised at all, it'll need the mlx4 driver(s) >> to load. >> >> I don't remember the exact one off hand (it's been a while), but some >> searching >> online for mlx4 and FreeBSD should turn up the right bits. > > Many thanks Justin. This is the first time I am hearing about an > Infiniband card operating in Ethernet mode. These cards come with two > CX4/SFF 8470 ports. It's not possible to connect standard Ethernet > cables that I know of (not even SFP modules). Do you mean that they > can operate in Ethernet mode over the CX4 cable? Yep. :) Back in the day when these cards were current tech, CX4 was an acceptable connector for 10GbE. The Infiniband switches from that era (that I had access to) were mostly Infiniband only though. But 10GbE CX4 switches did exist, and can still be found reasonably cheaply on Ebay. eg: * HP ProCurve 6 port CX4 10GBe switch https://www.ebay.com/itm/152232294328 * HP ProCurve 48 port 1GbE switch, with 2x 10GbE CX4 ports on the back https://www.ebay.com/itm/281899832599 One of the good things about those HP ProCurve switches... being enterprise gear they just keep working. For Years. From memory, HP still releases firmware security updates for them (for free) to this day. Unlike (say) Cisco. ;) Note - As a data point, FreeNAS (based on FreeBSD) includes the 10/40GbE driver for these cards by default. With FreeNAS they work as 10GbE "out of the box". :D Hopefully that helps. :) + Justin