From owner-freebsd-infiniband@FreeBSD.ORG Mon Jan 5 23:15:17 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B16ABA6C for ; Mon, 5 Jan 2015 23:15:17 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7D6C36424C for ; Mon, 5 Jan 2015 23:15:16 +0000 (UTC) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t05NCL42076462 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 5 Jan 2015 23:12:22 GMT Date: Mon, 05 Jan 2015 23:12:21 +0000 From: Karl Pielorz To: freebsd-infiniband@freebsd.org Subject: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jan 2015 23:15:17 -0000 Hi, I just built a couple of 11.0-CURRENT r275655M amd64 boxes up, both with: " MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] " Cards in that appear after boot as 'mlxen0' and 'mlxen1' (they're dual port cards). I'd previously run these cards with FreeBSD 10.0. I noticed the Infiniband options seem to be in GENERIC now, and opensm/ibstat appears to be in base. But neither work, i.e. " #ibstat ibpanic: [1343] main: stat of IB device 'mthca0' failed: (m) " And, " #opensm ... No local ports detected! Error from osm_opensm_bind (0x2A) Perhaps another instance of OpenSM is already running Exiting SM " Things have obviously changed since 10.0 (and I realise this is work in progress) - is there a setup guide or anywhere I can lookup how to get this working? [presuming it does work :)] Thanks, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Tue Jan 6 06:47:43 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4F333EB for ; Tue, 6 Jan 2015 06:47:43 +0000 (UTC) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1395464CB0 for ; Tue, 6 Jan 2015 06:47:43 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 602171FE022; Tue, 6 Jan 2015 07:47:40 +0100 (CET) Message-ID: <54AB853C.7070300@selasky.org> Date: Tue, 06 Jan 2015 07:48:28 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Karl Pielorz , freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 06:47:43 -0000 On 01/06/15 00:12, Karl Pielorz wrote: > > Hi, > > I just built a couple of 11.0-CURRENT r275655M amd64 boxes up, both with: > > " > MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] > " > > Cards in that appear after boot as 'mlxen0' and 'mlxen1' (they're dual > port cards). > > I'd previously run these cards with FreeBSD 10.0. I noticed the > Infiniband options seem to be in GENERIC now, and opensm/ibstat appears > to be in base. > > But neither work, i.e. > > " > #ibstat > ibpanic: [1343] main: stat of IB device 'mthca0' failed: (m) > " > > And, > > " > #opensm > ... > No local ports detected! > > Error from osm_opensm_bind (0x2A) > Perhaps another instance of OpenSM is already running > Exiting SM > " > > > Things have obviously changed since 10.0 (and I realise this is work in > progress) - is there a setup guide or anywhere I can lookup how to get > this working? [presuming it does work :)] > Hi, Did you set the following sysctl(s) to specify the port mode: sysctl -a | grep eth sys.device.mlx4_core0.mlx4_port1: eth sys.device.mlx4_core0.mlx4_port2: eth sys.device.mlx4_core1.mlx4_port1: eth sys.device.mlx4_core1.mlx4_port2: eth --HPS From owner-freebsd-infiniband@FreeBSD.ORG Tue Jan 6 06:48:09 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C20B7F6 for ; Tue, 6 Jan 2015 06:48:09 +0000 (UTC) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 831A764CB2 for ; Tue, 6 Jan 2015 06:48:09 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id A348F1FE022; Tue, 6 Jan 2015 07:48:07 +0100 (CET) Message-ID: <54AB8557.4020607@selasky.org> Date: Tue, 06 Jan 2015 07:48:55 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Karl Pielorz , freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... References: <54AB853C.7070300@selasky.org> In-Reply-To: <54AB853C.7070300@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 06:48:09 -0000 On 01/06/15 07:48, Hans Petter Selasky wrote: > On 01/06/15 00:12, Karl Pielorz wrote: >> >> Hi, >> >> I just built a couple of 11.0-CURRENT r275655M amd64 boxes up, both with: >> >> " >> MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] >> " >> >> Cards in that appear after boot as 'mlxen0' and 'mlxen1' (they're dual >> port cards). >> >> I'd previously run these cards with FreeBSD 10.0. I noticed the >> Infiniband options seem to be in GENERIC now, and opensm/ibstat appears >> to be in base. >> >> But neither work, i.e. >> >> " >> #ibstat >> ibpanic: [1343] main: stat of IB device 'mthca0' failed: (m) >> " >> >> And, >> >> " >> #opensm >> ... >> No local ports detected! >> >> Error from osm_opensm_bind (0x2A) >> Perhaps another instance of OpenSM is already running >> Exiting SM >> " >> >> >> Things have obviously changed since 10.0 (and I realise this is work in >> progress) - is there a setup guide or anywhere I can lookup how to get >> this working? [presuming it does work :)] >> > > Hi, > > Did you set the following sysctl(s) to specify the port mode: > > sysctl -a | grep eth > > sys.device.mlx4_core0.mlx4_port1: eth > sys.device.mlx4_core0.mlx4_port2: eth > sys.device.mlx4_core1.mlx4_port1: eth > sys.device.mlx4_core1.mlx4_port2: eth > > --HPS Valid options are: eth and ib --HPS From owner-freebsd-infiniband@FreeBSD.ORG Tue Jan 6 23:22:33 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEDF3963 for ; Tue, 6 Jan 2015 23:22:33 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id A9A57FA4 for ; Tue, 6 Jan 2015 23:22:33 +0000 (UTC) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t06NMP8X004771 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jan 2015 23:22:25 GMT Date: Tue, 06 Jan 2015 23:22:25 +0000 From: Karl Pielorz To: Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> In-Reply-To: <54AB853C.7070300@selasky.org> References: <54AB853C.7070300@selasky.org> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 23:22:34 -0000 --On 6 January 2015 07:48:28 +0100 Hans Petter Selasky wrote: > Hi, > > Did you set the following sysctl(s) to specify the port mode: > > sysctl -a | grep eth > > sys.device.mlx4_core0.mlx4_port1: eth > sys.device.mlx4_core0.mlx4_port2: eth > sys.device.mlx4_core1.mlx4_port1: eth > sys.device.mlx4_core1.mlx4_port2: eth > > --HPS If I run the above sysctl on my system, I get: sys.device.mlx4_core0.mlx4_port1: eth sys.device.mlx4_core0.mlx4_port2: eth A lot seems to have changed infiniband wise from FBSD 10 to 11 - is there any updated 'getting started' guide or notes or anything? Is it still the same thing of installing the cards (everything seems to be compiled up already OFED / ibstat / opensm way) and running up opensm? -Karl From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 07:55:48 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 84672B7A for ; Wed, 7 Jan 2015 07:55:48 +0000 (UTC) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4439464099 for ; Wed, 7 Jan 2015 07:55:48 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 772A61FE022; Wed, 7 Jan 2015 08:55:45 +0100 (CET) Message-ID: <54ACE6B1.1060705@selasky.org> Date: Wed, 07 Jan 2015 08:56:33 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Karl Pielorz , freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> In-Reply-To: <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 07:55:48 -0000 On 01/07/15 00:22, Karl Pielorz wrote: > > --On 6 January 2015 07:48:28 +0100 Hans Petter Selasky > wrote: > >> Hi, >> >> Did you set the following sysctl(s) to specify the port mode: >> >> sysctl -a | grep eth >> >> sys.device.mlx4_core0.mlx4_port1: eth >> sys.device.mlx4_core0.mlx4_port2: eth >> sys.device.mlx4_core1.mlx4_port1: eth >> sys.device.mlx4_core1.mlx4_port2: eth >> >> --HPS > > If I run the above sysctl on my system, I get: > > sys.device.mlx4_core0.mlx4_port1: eth > sys.device.mlx4_core0.mlx4_port2: eth > > A lot seems to have changed infiniband wise from FBSD 10 to 11 - is > there any updated 'getting started' guide or notes or anything? > > Is it still the same thing of installing the cards (everything seems to > be compiled up already OFED / ibstat / opensm way) and running up opensm? > > -Karl > Hi, From what I know everything should work like ususal. In between you maybe want to grab this patch aswell, hence it plugs a file descriptor memory leak related to infiniband: https://svnweb.freebsd.org/changeset/base/276749 --HPS From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 13:26:05 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73D78873 for ; Wed, 7 Jan 2015 13:26:05 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 3DADC66FE3 for ; Wed, 7 Jan 2015 13:26:04 +0000 (UTC) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t07DQ2ZE079897 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Jan 2015 13:26:03 GMT Date: Wed, 07 Jan 2015 13:26:03 +0000 From: Karl Pielorz To: Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: <62BC1791A40C62A1836B5368@[10.12.30.106]> In-Reply-To: <54ACE6B1.1060705@selasky.org> References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> <54ACE6B1.1060705@selasky.org> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 13:26:05 -0000 --On 07 January 2015 08:56 +0100 Hans Petter Selasky wrote: > Hi, > > From what I know everything should work like ususal. In between you > maybe want to grab this patch aswell, hence it plugs a file descriptor > memory leak related to infiniband: > > https://svnweb.freebsd.org/changeset/base/276749 Thanks. I'll try updating to HEAD tonight and see if that makes any difference. The cards are 'seen' [e.g. by ifconfig] - but none of the utils (ibstat/opensm) can see them. If I try to ifconfig one of the interfaces I get an instant kernel panic as well... Like I said I'll try with HEAD and post back if I've still got issues... Cheers, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 13:45:37 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82C35CFC for ; Wed, 7 Jan 2015 13:45:37 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 4A1D6148D for ; Wed, 7 Jan 2015 13:45:36 +0000 (UTC) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t07DjZ7F081937 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Jan 2015 13:45:35 GMT Date: Wed, 07 Jan 2015 13:45:36 +0000 From: Karl Pielorz To: Oded Shanoon , Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: RE: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: <640D4900BB006C2D80760288@[10.12.30.106]> In-Reply-To: References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> <54ACE6B1.1060705@selasky.org> <62BC1791A40C62A1836B5368@[10.12.30.106]> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 13:45:37 -0000 --On 07 January 2015 13:34 +0000 Oded Shanoon wrote: > Did you change the ports to 'ib' instead of 'eth' in the sysctl? Sorry - I took the original email to mean 'run the sysctl to check they are in eth mode' - not, 'see if they're in eth mode then flip them to ib mode if needed' :) > Also, please verify the relevant modules are loaded: > Kldstat -v | grep 'mlx4\|mlx4ib\|ibcore\|ipoib' Ok, I'll set them to ib - and check the relevant modules are loaded, Thanks, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 14:24:06 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A31C60C for ; Wed, 7 Jan 2015 14:24:06 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id C5FE164B0B for ; Wed, 7 Jan 2015 14:24:05 +0000 (UTC) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t07EO3jP085699 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Jan 2015 14:24:04 GMT Date: Wed, 07 Jan 2015 14:24:04 +0000 From: Karl Pielorz To: Oded Shanoon , freebsd-infiniband@freebsd.org Subject: RE: Infiniband under FreeBSD 11.0-CURRENT... Message-ID: <922BDD3D8725118FA5C28939@[10.12.30.106]> In-Reply-To: <640D4900BB006C2D80760288@[10.12.30.106]> References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> <54ACE6B1.1060705@selasky.org> <62BC1791A40C62A1836B5368@[10.12.30.106]> <640D4900BB006C2D80760288@[10.12.30.106]> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 14:24:06 -0000 --On 07 January 2015 13:45 +0000 Karl Pielorz wrote: > Ok, I'll set them to ib - and check the relevant modules are loaded, Ok - sysctl'ing the ports to 'ib' - and opensm / ibstat now see them. >> Also, please verify the relevant modules are loaded: >> Kldstat -v | grep 'mlx4\|mlx4ib\|ibcore\|ipoib' This gives: 421 ipoib 420 ibcore 423 mlx4 422 mlx4ib Having configured the interfaces, and run the obligatory 'iperf' on them - the results aren't outstanding: " [ 4] local 10.90.90.2 port 5001 connected with 10.90.90.1 port 64249 [ ID] Interval Transfer Bandwidth [ 4] 0.0-12.2 sec 1.12 MBytes 771 Kbits/sec ... [ 3] local 10.90.90.1 port 64249 connected with 10.90.90.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.7 sec 1.12 MBytes 879 Kbits/sec " Under 10.x I know it was better to disable connected mode - is there a way this can be done under 11.x - or should I see better performance than that using CM? 'ibstat' on the system returns: " CA 'mlx4_0' CA type: MT25418 Number of ports: 2 Firmware version: 2.6.0 Hardware version: a0 Node GUID: 0x001a4bffff0c328c System image GUID: 0x001a4bffff0c328f Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0251086a Port GUID: 0x001a4bffff0c328d Port 2: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x001a4bffff0c328e " (Second port is 'Initialising' as I only have one instance of opensm running bound to the first port). Thanks, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Wed Jan 7 15:09:33 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD48ECF8 for ; Wed, 7 Jan 2015 15:09:33 +0000 (UTC) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0637.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::637]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 415DC66FB1 for ; Wed, 7 Jan 2015 15:09:33 +0000 (UTC) Received: from DB3PR05MB026.eurprd05.prod.outlook.com (10.242.137.13) by DB3PR05MB026.eurprd05.prod.outlook.com (10.242.137.13) with Microsoft SMTP Server (TLS) id 15.1.49.12; Wed, 7 Jan 2015 13:34:07 +0000 Received: from DB3PR05MB026.eurprd05.prod.outlook.com ([169.254.2.224]) by DB3PR05MB026.eurprd05.prod.outlook.com ([169.254.2.224]) with mapi id 15.01.0049.002; Wed, 7 Jan 2015 13:34:07 +0000 From: Oded Shanoon To: Karl Pielorz , Hans Petter Selasky , "freebsd-infiniband@freebsd.org" Subject: RE: Infiniband under FreeBSD 11.0-CURRENT... Thread-Topic: Infiniband under FreeBSD 11.0-CURRENT... Thread-Index: AQHQKT180SUBwbVUTUuFlcDbYowXe5yypxIAgAEVtYCAAI+lgIAAXBCAgAAAjwA= Date: Wed, 7 Jan 2015 13:34:07 +0000 Message-ID: References: <54AB853C.7070300@selasky.org> <9406DDD429A698CCFE8EA77B@Karls-Mac-mini.local> <54ACE6B1.1060705@selasky.org> <62BC1791A40C62A1836B5368@[10.12.30.106]> In-Reply-To: <62BC1791A40C62A1836B5368@[10.12.30.106]> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [193.47.165.251] authentication-results: spf=none (sender IP is ) smtp.mailfrom=odeds@mellanox.com; x-dmarcaction: None x-microsoft-antispam: BCL:0;PCL:0;RULEID:(3005003);SRVR:DB3PR05MB026; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:;SRVR:DB3PR05MB026; x-forefront-prvs: 044968D9E1 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(24454002)(13464003)(189002)(377454003)(51704005)(199003)(102836002)(76576001)(2656002)(20776003)(64706001)(87936001)(74316001)(101416001)(46102003)(33656002)(21056001)(97736003)(1720100001)(62966003)(77156002)(122556002)(40100003)(15975445007)(93886004)(68736005)(2900100001)(105586002)(106356001)(106116001)(54206007)(107046002)(107886001)(99396003)(92566001)(50986999)(76176999)(54356999)(54606007)(86362001)(66066001)(19580405001)(19580395003)(4396001)(2950100001)(31966008)(120916001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR05MB026; H:DB3PR05MB026.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jan 2015 13:34:07.6448 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR05MB026 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 15:09:33 -0000 Did you change the ports to 'ib' instead of 'eth' in the sysctl? Also, please verify the relevant modules are loaded: Kldstat -v | grep 'mlx4\|mlx4ib\|ibcore\|ipoib' Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Karl Pielorz Sent: Wednesday, January 07, 2015 3:26 PM To: Hans Petter Selasky; freebsd-infiniband@freebsd.org Subject: Re: Infiniband under FreeBSD 11.0-CURRENT... --On 07 January 2015 08:56 +0100 Hans Petter Selasky wrote: > Hi, > > From what I know everything should work like ususal. In between you=20 > maybe want to grab this patch aswell, hence it plugs a file descriptor=20 > memory leak related to infiniband: > > https://svnweb.freebsd.org/changeset/base/276749 Thanks. I'll try updating to HEAD tonight and see if that makes any difference. The= cards are 'seen' [e.g. by ifconfig] - but none of the utils (ibstat/opensm= ) can see them. If I try to ifconfig one of the interfaces I get an instant kernel panic as= well... Like I said I'll try with HEAD and post back if I've still got issues... Cheers, -Karl _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@FreeBSD.ORG Fri Jan 9 02:45:50 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AD04A275 for ; Fri, 9 Jan 2015 02:45:50 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0088.outbound.protection.outlook.com [157.56.111.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 625C7F9F for ; Fri, 9 Jan 2015 02:45:49 +0000 (UTC) Received: from BN3PR0801MB1028.namprd08.prod.outlook.com (25.161.218.14) by BN3PR0801MB1028.namprd08.prod.outlook.com (25.161.218.14) with Microsoft SMTP Server (TLS) id 15.1.53.17; Fri, 9 Jan 2015 02:11:43 +0000 Received: from BN3PR0801MB1028.namprd08.prod.outlook.com ([25.161.218.14]) by BN3PR0801MB1028.namprd08.prod.outlook.com ([25.161.218.14]) with mapi id 15.01.0053.000; Fri, 9 Jan 2015 02:11:43 +0000 From: "Lewis, Fred" To: "freebsd-infiniband@freebsd.org" Subject: Question On Supported Mellanox Adapters Thread-Topic: Question On Supported Mellanox Adapters Thread-Index: AQHQK7GcJR6uol/N40mHfKNCVJ9i9g== Date: Fri, 9 Jan 2015 02:11:43 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [66.31.107.140] authentication-results: spf=none (sender IP is ) smtp.mailfrom=flewis@panasas.com; x-dmarcaction: None x-microsoft-antispam: BCL:0;PCL:0;RULEID:(3005003);SRVR:BN3PR0801MB1028; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:; SRVR:BN3PR0801MB1028; x-forefront-prvs: 04519BA941 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(199003)(164054003)(189002)(20776003)(36756003)(68736005)(2351001)(107046002)(107886001)(66066001)(64706001)(102836002)(40100003)(2656002)(97736003)(122556002)(229853001)(106116001)(105586002)(2900100001)(86362001)(50986999)(99286002)(4396001)(106356001)(54356999)(31966008)(21056001)(110136001)(87936001)(46102003)(99396003)(62966003)(120916001)(101416001)(450100001)(16236675004)(92566001)(77156002)(94096001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0801MB1028; H:BN3PR0801MB1028.namprd08.prod.outlook.com; FPR:; SPF:None; MLV:sfv; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: panasas.com does not designate permitted sender hosts) MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jan 2015 02:11:43.0181 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0801MB1028 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 02:45:50 -0000 Hello, FreeBSD IB folks. I am working off of a FreeBSD 10.1 RC4 code base and I'm investigating InfiniBand running on FreeBSD as part of a prototyping effort. The prototype hardware is an Intel FDR InfiniBand* ConnectX-3* I/O Module A= XX2FDRIBIOM (Dual Port). According to the documentation , the Module is based on the Mellanox MT27508A1-FCCR-FV Chip. >From the Mellanox documentation the PCI Device IDs for the adapter are: 0x1003 MT27500 Family [ConnectX-3] 0x1004 MT27500 Family [ConnectX-3 Virtual Function] >From looking at the FreeBSD driver source code it appears that the MT27500 family is not supported: #define PCI_DEVICE_ID_MELLANOX_TAVOR 0x5a44 #define PCI_DEVICE_ID_MELLANOX_ARBEL_COMPAT 0x6278 #define PCI_DEVICE_ID_MELLANOX_ARBEL 0x6282 #define PCI_DEVICE_ID_MELLANOX_SINAI_OLD 0x5e8c #define PCI_DEVICE_ID_MELLANOX_SINAI 0x6274 /* Types of supported HCA */ enum { TAVOR, /* MT23108 */ ARBEL_COMPAT, /* MT25208 in Tavor compat mode */ ARBEL_NATIVE, /* MT25208 with extended features */ SINAI /* MT25204 */ }; Does anyone know if support is planned for the MT27500 (ConnectX-3) Family of adapters. Any info will be appreciated. Thanks, -Fred Lewis From owner-freebsd-infiniband@FreeBSD.ORG Fri Jan 9 06:43:15 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AEE7E594 for ; Fri, 9 Jan 2015 06:43:15 +0000 (UTC) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 72EE8993 for ; Fri, 9 Jan 2015 06:43:15 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 49EF51FE022; Fri, 9 Jan 2015 07:43:12 +0100 (CET) Message-ID: <54AF78B0.8000800@selasky.org> Date: Fri, 09 Jan 2015 07:44:00 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: "Lewis, Fred" , "freebsd-infiniband@freebsd.org" Subject: Re: Question On Supported Mellanox Adapters References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 06:43:15 -0000 On 01/09/15 03:11, Lewis, Fred wrote: >>From the Mellanox documentation the PCI Device IDs for the adapter are: > 0x1003 MT27500 Family [ConnectX-3] > 0x1004 MT27500 Family [ConnectX-3 Virtual Function] > >>From looking at the FreeBSD driver source code it appears that the MT27500 > family is not supported: Hi, These are supported in 9/10-stable and 11-current. Just grep for the number in the code: > sys/ofed/drivers/net/mlx4/main.c: if (dev->pdev->device == 0x1003) > sys/ofed/drivers/net/mlx4/main.c: { PCI_VDEVICE(MELLANOX, 0x1003), 0 }, --HPS From owner-freebsd-infiniband@FreeBSD.ORG Sat Jan 10 18:00:07 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BF08A94 for ; Sat, 10 Jan 2015 18:00:07 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id CC115EDA for ; Sat, 10 Jan 2015 18:00:06 +0000 (UTC) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t0AHxvH6022182 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 10 Jan 2015 17:59:59 GMT Date: Sat, 10 Jan 2015 17:59:57 +0000 From: Karl Pielorz To: freebsd-infiniband@freebsd.org Subject: FreeBSD 11 Infiniband performance / usable? Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jan 2015 18:00:07 -0000 Hi, I've now got my MT25418's running 'back to back' on a couple of FreeBSD 11 boxes [amd64 Haswell based] - but the performance seems rather poor using them for IPoIB? (iperf sites around 2Mbyte/sec on defaults - rising to 98Mbyte/sec if I set an MTU of 4k) - default on the card was 64k. Disabling IBOIB_CM in the kernel doesn't seem to make any difference (under 10.0 this made a huge difference). Also - the two boxes have lost site of each other now [I had this problem previously with 10.0 - one of the reasons why I was looking at 11]. What's the current state of IB support in FreeBSD 11? - I know a lot of work has been done on this - but is it usable yet? The two machines both have MT25418's fitted running 'back to back' with one CX4 cable between them, and opensm running on them (after running the kernel sysctl's to switch the ports from 'eth' mode to 'ib'). Cheers, -Karl From owner-freebsd-infiniband@FreeBSD.ORG Sat Jan 10 23:17:05 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B10AA93F for ; Sat, 10 Jan 2015 23:17:05 +0000 (UTC) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FAB5150 for ; Sat, 10 Jan 2015 23:17:05 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id A2A911FE022; Sun, 11 Jan 2015 00:17:01 +0100 (CET) Message-ID: <54B1B31C.2040901@selasky.org> Date: Sun, 11 Jan 2015 00:17:48 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Karl Pielorz , freebsd-infiniband@freebsd.org Subject: Re: FreeBSD 11 Infiniband performance / usable? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jan 2015 23:17:05 -0000 On 01/10/15 18:59, Karl Pielorz wrote: > > Hi, > > I've now got my MT25418's running 'back to back' on a couple of FreeBSD > 11 boxes [amd64 Haswell based] - but the performance seems rather poor > using them for IPoIB? (iperf sites around 2Mbyte/sec on defaults - > rising to 98Mbyte/sec if I set an MTU of 4k) - default on the card was 64k. > > Disabling IBOIB_CM in the kernel doesn't seem to make any difference > (under 10.0 this made a huge difference). > > Also - the two boxes have lost site of each other now [I had this > problem previously with 10.0 - one of the reasons why I was looking at 11]. > > > What's the current state of IB support in FreeBSD 11? - I know a lot of > work has been done on this - but is it usable yet? > > The two machines both have MT25418's fitted running 'back to back' with > one CX4 cable between them, and opensm running on them (after running > the kernel sysctl's to switch the ports from 'eth' mode to 'ib'). > > Cheers, Hi, IPoIB needs to be optimized to shuffle more packets per request to and from the kernel basically. Currently that is not done. Are you planning to use IPoIB for real workloads? --HPS