From owner-freebsd-current@FreeBSD.ORG Mon Sep 24 16:39:37 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 591341065673; Mon, 24 Sep 2012 16:39:37 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from ns1.jnielsen.net (secure.freebsdsolutions.net [69.55.234.48]) by mx1.freebsd.org (Postfix) with ESMTP id 3B8B08FC17; Mon, 24 Sep 2012 16:39:36 +0000 (UTC) Received: from [10.10.1.32] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by ns1.jnielsen.net (8.14.4/8.14.4) with ESMTP id q8OGbMQh073787 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 24 Sep 2012 12:37:23 -0400 (EDT) (envelope-from lists@jnielsen.net) From: John Nielsen Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 24 Sep 2012 10:37:30 -0600 Message-Id: <3A359B33-380C-4230-A62C-623765E9376A@jnielsen.net> To: freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) X-Mailer: Apple Mail (2.1498) X-DCC-sonic.net-Metrics: ns1.jnielsen.net 1117; Body=2 Fuz1=2 Fuz2=2 X-Virus-Scanned: clamav-milter 0.97.5 at ns1.jnielsen.net X-Virus-Status: Clean Cc: Subject: using ConnectX card as Ethernet (mlxen) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 16:39:37 -0000 I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with = two ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv = below. The cards are connected directly to a 10GB Ethernet switch so I = need to run them in "eth" mode rather than "ib". Unfortunately they come = up in "ib" mode and I don't know how to change it. The same hardware works fine under CentOS 6.3, though I need to manually = set the cards to 'eth' there as well (which I do using a = 'connectx_port_config script from Mellanox that twiddles the mlx4_port1 = entries under /sys (sysfs). Under FreeBSD I see these sysctls but I = can't set them to 'eth' either via /boot/loader.conf or by sysctl after = boot, with or without mlxen and/or mlx4ib loaded: sys.device.mlx4_core0.mlx4_port1: ib sys.device.mlx4_core1.mlx4_port1: ib Assuming mlxen is actually supported, how do I configure the card so it = will attach? mlx4_core0: mem 0xdfa00000-0xdfafffff,0xdd800000-0xddffffff = irq 32 at device 0.0 on pci4 mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2 (August 4, 2010) mlx4_core: Initializing mlx4_core mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010) mlx4_en mlx4_core0: UDP RSS is not supported on this device. mlx4_core1: mem 0xdf900000-0xdf9fffff,0xdd000000-0xdd7fffff = irq 42 at device 0.0 on pci7 mlx4_core: Initializing mlx4_core mlx4_core0@pci0:4:0:0: class=3D0x0c0600 card=3D0x002215b3 = chip=3D0x673c15b3 rev=3D0xb0 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / = 10GigE]' class =3D serial bus mlx4_core1@pci0:7:0:0: class=3D0x028000 card=3D0x001715b3 = chip=3D0x100315b3 rev=3D0x00 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT27500 Family [ConnectX-3]' class =3D network Thanks, JN