From owner-freebsd-infiniband@FreeBSD.ORG Wed Feb 4 23:26:18 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6DBD93BA for ; Wed, 4 Feb 2015 23:26:18 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0081.outbound.protection.outlook.com [65.55.169.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E74358FF for ; Wed, 4 Feb 2015 23:26:17 +0000 (UTC) Received: from DM2PR0801MB1037.namprd08.prod.outlook.com (25.160.133.25) by DM2PR0801MB1086.namprd08.prod.outlook.com (25.160.134.144) with Microsoft SMTP Server (TLS) id 15.1.75.20; Wed, 4 Feb 2015 23:26:08 +0000 Received: from DM2PR0801MB1039.namprd08.prod.outlook.com (25.160.133.27) by DM2PR0801MB1037.namprd08.prod.outlook.com (25.160.133.25) with Microsoft SMTP Server (TLS) id 15.1.81.19; Wed, 4 Feb 2015 23:26:06 +0000 Received: from DM2PR0801MB1039.namprd08.prod.outlook.com ([25.160.133.27]) by DM2PR0801MB1039.namprd08.prod.outlook.com ([25.160.133.27]) with mapi id 15.01.0081.018; Wed, 4 Feb 2015 23:26:06 +0000 From: "Lewis, Fred" To: "freebsd-infiniband@freebsd.org" Subject: Does mlxen work on FreeBSD 10.1 with ConnectX-3 Thread-Topic: Does mlxen work on FreeBSD 10.1 with ConnectX-3 Thread-Index: AQHQQNHyTj4Ou0Fn6kKQiYrjjngl2Q== Date: Wed, 4 Feb 2015 23:26:06 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [64.80.217.3] authentication-results: freebsd.org; dkim=none (message not signed) header.d=none; x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR0801MB1037;UriScan:; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:; SRVR:DM2PR0801MB1037; x-forefront-prvs: 04772EA191 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(164054003)(36756003)(229853001)(19580395003)(19580405001)(106116001)(122556002)(110136001)(107886001)(2351001)(2900100001)(86362001)(77156002)(92566002)(450100001)(16236675004)(46102003)(62966003)(102836002)(87936001)(50986999)(54356999)(40100003)(66066001)(2656002)(94096001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM2PR0801MB1037; H:DM2PR0801MB1039.namprd08.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Feb 2015 23:26:06.1091 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0801MB1037 X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR0801MB1086; X-OriginatorOrg: panasas.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2015 23:26:18 -0000 Hi FreeBSD IB folks, I'm trying to bring up a Mellanox Ethernert Interface and it is showing as = "no carrier." The cables were swapped out with a known good one and it didn't seem to mak= e any difference. The remote peer port is admin UP. I've built the KLDs using the latest FreeBSD OFED stuff (v2.1.5). The build machine is different from the machine that I'm loading the KLDs o= n. The build machine does not have any OFED options or devices installed. [mlxen]$ sysctl kern.conftxt | grep -i ofed [mlxen]$ I built the KLDs as follows: cd mlx4 ; make clean cleandepend; make cd mlxen; make clean cleandepend; make I then copied the KLDs (mlx4.ko and mlxen.ko) to /boot/kernel on the target= machine and did a kldload. The modules loaded fine as far as I can tell and mlxen0 = shows up in the interface list. mlx4_core0@pci0:5:0:0: class=3D0x028000 card=3D0x100315b3 chip=3D0x100315b= 3 rev=3D0x00 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT27500 Family [ConnectX-3]' class =3D network root@mau-da-27-4-3:/boot/kernel # kldload mlx4.ko; kldload mlxen.ko mlx4_core0: mem 0x93a00000-0x93afffff,0x383fff000000-0x383fff7f= ffff irq 42 at device 0.0 on pci5 <6>mlx4_core: Mellanox ConnectX core driver v2.1 (Feb 4 2015) <6>mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1 (Feb 4 2015) mlxen0: Ethernet address: 00:1e:67:81:40:61 mau-da-27-4-3 kemlrnel: mlx4_en: mx4lx4_core0: Port _e1:= Using 24 TX rn:ings mlx4_core0: Port 1: Using 24 TX rings <4>mlx4_en: mlx4_core0: Port 1: Using 16 RX rings mau-da-27-4-3 kemlrnel: mlx4_en: mx4lx4_core0: Port _e1:= Using 16 RX rn:ings Ub 4 14 m:23:18 mau-da-27lx-4-3 kernel: mlxen4_en: mlxen0: Us0:ing 24 TX = rings sing 24 TX rings <4>mlx4_en: mlxen0: Using 16 RX rings Feb 4 14:23:18 <4mau-da-27-4-3 ke>mrnel: mlx4_en: mlxlxen0: Using 16 4_RX = rings en: mlxen0: Initializing port Feb 4 14:23:19 mau-da-27-4-3 kernel: mlx4_en: mlxen0: Initializing port root@mau-da-27-4-3:/boot/kernel # kldstat -v | grep 'mlx4\|mlxen' 9 2 0xffffffff81c6c000 37816 mlx4.ko (/boot/kernel/mlx4.ko) 494 mlx4 10 1 0xffffffff81ca4000 125e2 mlxen.ko (/boot/kernel/mlxen.ko) 495 mlxen ifconfig -a mlxen0: flags=3D8843 metric 0 mtu 1= 500 options=3Dd07bb ether 00:1e:67:81:40:61 nd6 options=3D29 media: Ethernet autoselect status: no carrier root@mau-da-27-4-3:/boot/kernel # sysctl -a | grep mlx4 hw.mlxen0.conf.device_name: mlx4_core0 dev.mlx4_core.%parent: dev.mlx4_core.0.%desc: mlx4_core dev.mlx4_core.0.%driver: mlx4_core dev.mlx4_core.0.%location: slot=3D0 function=3D0 handle=3D\_SB_.PCI0.BR3C.H= 000 dev.mlx4_core.0.%pnpinfo: vendor=3D0x15b3 device=3D0x1003 subvendor=3D0x15b= 3 subdevice=3D0x1003 class=3D0x028000 dev.mlx4_core.0.%parent: pci5 sys.device.mlx4_core0.rate_limit_caps.different_rates_num: 0 sys.device.mlx4_core0.mlx4_port1: eth If anyone can see anything I am missing or doing wrong, I'd be glad to know= . Thanks, -Fred From owner-freebsd-infiniband@FreeBSD.ORG Thu Feb 5 05:39:42 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C9812C7 for ; Thu, 5 Feb 2015 05:39:42 +0000 (UTC) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CEB7EA2F for ; Thu, 5 Feb 2015 05:39:40 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id D590C1FE023; Thu, 5 Feb 2015 06:39:31 +0100 (CET) Message-ID: <54D30246.5090508@selasky.org> Date: Thu, 05 Feb 2015 06:40:22 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: "Lewis, Fred" , "freebsd-infiniband@freebsd.org" Subject: Re: Does mlxen work on FreeBSD 10.1 with ConnectX-3 References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 05:39:42 -0000 On 02/05/15 00:26, Lewis, Fred wrote: > Hi FreeBSD IB folks, > > I'm trying to bring up a Mellanox Ethernert Interface and it is showing as "no carrier." > The cables were swapped out with a known good one and it didn't seem to make any difference. > The remote peer port is admin UP. > > I've built the KLDs using the latest FreeBSD OFED stuff (v2.1.5). > The build machine is different from the machine that I'm loading the KLDs on. > The build machine does not have any OFED options or devices installed. > Hi, Try setting: sysctl sys.device.mlx4_core0.mlx4_port1=ib --HPS From owner-freebsd-infiniband@FreeBSD.ORG Thu Feb 5 15:41:12 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EA8355A3 for ; Thu, 5 Feb 2015 15:41:12 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0082.outbound.protection.outlook.com [157.56.110.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F5FADA9 for ; Thu, 5 Feb 2015 15:41:11 +0000 (UTC) Received: from DM2PR0801MB1039.namprd08.prod.outlook.com (25.160.133.27) by DM2PR0801MB1039.namprd08.prod.outlook.com (25.160.133.27) with Microsoft SMTP Server (TLS) id 15.1.81.19; Thu, 5 Feb 2015 15:41:03 +0000 Received: from DM2PR0801MB1039.namprd08.prod.outlook.com ([25.160.133.27]) by DM2PR0801MB1039.namprd08.prod.outlook.com ([25.160.133.27]) with mapi id 15.01.0081.018; Thu, 5 Feb 2015 15:41:03 +0000 From: "Lewis, Fred" To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" Subject: Re: Does mlxen work on FreeBSD 10.1 with ConnectX-3 Thread-Topic: Does mlxen work on FreeBSD 10.1 with ConnectX-3 Thread-Index: AQHQQNHyTj4Ou0Fn6kKQiYrjjngl2ZzhitMAgABT/wA= Date: Thu, 5 Feb 2015 15:41:03 +0000 Message-ID: References: <54D30246.5090508@selasky.org> In-Reply-To: <54D30246.5090508@selasky.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [64.80.217.3] authentication-results: selasky.org; dkim=none (message not signed) header.d=none; x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR0801MB1039; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:; SRVR:DM2PR0801MB1039; x-forefront-prvs: 0478C23FE0 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(479174004)(24454002)(51704005)(377454003)(54356999)(87936001)(66066001)(62966003)(77156002)(122556002)(92566002)(40100003)(107886001)(102836002)(76176999)(50986999)(19580405001)(2656002)(86362001)(106116001)(2900100001)(36756003)(19580395003)(46102003)(2950100001)(99286002)(94096001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM2PR0801MB1039; H:DM2PR0801MB1039.namprd08.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-ID: <1BEC4212B57B184B8012660CBB915360@namprd08.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Feb 2015 15:41:03.0605 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0801MB1039 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 15:41:13 -0000 Thanks Hans, Actually, I want to use the port as an Ethernet port. If my understanding is correct that means the port setting should be "=3Deth" -Fred On 2/5/15 12:40 AM, "Hans Petter Selasky" wrote: >On 02/05/15 00:26, Lewis, Fred wrote: >> Hi FreeBSD IB folks, >> >> I'm trying to bring up a Mellanox Ethernert Interface and it is showing >>as "no carrier." >> The cables were swapped out with a known good one and it didn't seem to >>make any difference. >> The remote peer port is admin UP. >> >> I've built the KLDs using the latest FreeBSD OFED stuff (v2.1.5). >> The build machine is different from the machine that I'm loading the >>KLDs on. >> The build machine does not have any OFED options or devices installed. >> > >Hi, > >Try setting: > >sysctl sys.device.mlx4_core0.mlx4_port1=3Dib > >--HPS > From owner-freebsd-infiniband@FreeBSD.ORG Thu Feb 5 16:17:51 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E45B7303 for ; Thu, 5 Feb 2015 16:17:51 +0000 (UTC) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A7B041F4 for ; Thu, 5 Feb 2015 16:17:51 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id EC3751FE023; Thu, 5 Feb 2015 17:17:48 +0100 (CET) Message-ID: <54D397DE.2060001@selasky.org> Date: Thu, 05 Feb 2015 17:18:38 +0100 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: "Lewis, Fred" , "freebsd-infiniband@freebsd.org" Subject: Re: Does mlxen work on FreeBSD 10.1 with ConnectX-3 References: <54D30246.5090508@selasky.org> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 16:17:52 -0000 On 02/05/15 16:41, Lewis, Fred wrote: > Thanks Hans, > > Actually, I want to use the port as an Ethernet port. > If my understanding is correct that means the port setting should be > "=eth" > -Fred > Yes, that's correctly guessed! --HPS