From owner-freebsd-infiniband@FreeBSD.ORG Tue May 28 11:43:37 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A83E86C8; Tue, 28 May 2013 11:43:37 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog113.obsmtp.com (eu1sys200aog113.obsmtp.com [207.126.144.135]) by mx1.freebsd.org (Postfix) with ESMTP id 2221A1AF; Tue, 28 May 2013 11:43:35 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob113.postini.com ([207.126.147.11]) with SMTP ID DSNKUaSYTLVCQei4GSKePqg2j6sAR5pgng6D@postini.com; Tue, 28 May 2013 11:43:36 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Tue, 28 May 2013 14:35:46 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" , "freebsd-net@freebsd.org" , "freebsd-questions@freebsd.org" Subject: Create bond on Infiniband ports Thread-Topic: Create bond on Infiniband ports Thread-Index: Ac5blHiC28mlGzpUR72ew+cW7MoT7Q== Date: Tue, 28 May 2013 11:35:45 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF65A33@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 11:43:37 -0000 Hi. I use FreeBSD 9.1 with OFED compiled on it. There is a Mellanox adapter: [root@qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3 mlx4_core0@pci0:0:5:0: class=3D0x028000 card=3D0x005015b3 chip=3D0x100315b= 3 rev=3D0x00 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT27500 Family [ConnectX-3]' class =3D network I want to create Bond on the two ports (ib0 and ib1) of this device: [root@qa-h-vrt-030-006 ~]# ifconfig em0: flags=3D8843 metric 0 mtu 1500 options=3D209b ether 00:50:56:23:1e:06 inet6 fe80::250:56ff:fe23:1e06%em0 prefixlen 64 scopeid 0x2 inet 10.195.30.6 netmask 0xffff0000 broadcast 10.195.255.255 nd6 options=3D23 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D23 ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.51 inet 11.195.30.1 netmask 0xffff0000 broadcast 11.195.255.255 inet6 fe80::250:56ff:fe23:1e06%ib0 prefixlen 64 scopeid 0x4 nd6 options=3D23 ib1: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.52 inet 12.195.30.1 netmask 0xffff0000 broadcast 12.195.255.255 inet6 fe80::250:56ff:fe23:1e06%ib1 prefixlen 64 scopeid 0x5 nd6 options=3D23 1. Is there "bond" in FreeBSD or its "Lagg"? 2. I tried to create this way, but failed: [root@qa-h-vrt-030-006 ~]# ifconfig lagg0 create root@qa-h-vrt-031-005 conf]# ifconfig lagg0 laggproto failover laggport ib0= laggport ib1 ifconfig: SIOCSLAGGPORT: Protocol not supported Any ideas? Is it supported on Infiniband ports? Regards, Alex Liptsin From owner-freebsd-infiniband@FreeBSD.ORG Tue May 28 12:09:00 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D3DACF1B for ; Tue, 28 May 2013 12:09:00 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog116.obsmtp.com (eu1sys200aog116.obsmtp.com [207.126.144.141]) by mx1.freebsd.org (Postfix) with ESMTP id C8CB1340 for ; Tue, 28 May 2013 12:08:59 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob116.postini.com ([207.126.147.11]) with SMTP ID DSNKUaSeWpfLK/KHx3/uQa7jN7djrn5a467D@postini.com; Tue, 28 May 2013 12:08:59 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Tue, 28 May 2013 15:06:59 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" Subject: How to chnge Datagram/Connected modes at FreeBSD? Thread-Topic: How to chnge Datagram/Connected modes at FreeBSD? Thread-Index: Ac5bm9dhXi+Z7pUEQvya+AFBf1W5yw== Date: Tue, 28 May 2013 12:06:57 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF65A87@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 12:09:00 -0000 Hello. I work with FreeBSD 9.1 and Mellanox devices. How can I configure MTU in connected mode on FreeBSD 9.1? In Linux to enable connected mode for interface ib0, I enter: echo connected > /sys/class/net/ib0/mode Switching between CM and UD mode can be done in run time: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM There is no such directories at FreeBSD. Wat shall I do? Datagram vs Connected modes The IPoIB driver supports two modes of operation: datagram and connected. The mode is set and read through an interface's /sys/class/net//mode file. In datagram mode, the IB UD (Unreliable Datagram) transport is used and so the interface MTU has is equal to the IB L2 MTU minus the IPoIB encapsulation header (4 bytes). For example, in a typical IB fabric with a 2K MTU, the IPoIB MTU will be 2048 - 4 =3D 2044 bytes. In connected mode, the IB RC (Reliable Connected) transport is used. Connected mode takes advantage of the connected nature of the IB transport and allows an MTU up to the maximal IP packet size of 64K, which reduces the number of IP packets needed for handling large UDP datagrams, TCP segments, etc and increases the performance for large messages. In connected mode, the interface's UD QP is still used for multicast and communication with peers that don't support connected mode. In this case, RX emulation of ICMP PMTU packets is used to cause the networking stack to use the smaller UD MTU for these neighbours. Thanks a lot Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael From owner-freebsd-infiniband@FreeBSD.ORG Tue May 28 12:13:46 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 31D121B9 for ; Tue, 28 May 2013 12:13:46 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog102.obsmtp.com (eu1sys200aog102.obsmtp.com [207.126.144.113]) by mx1.freebsd.org (Postfix) with ESMTP id A6D103A4 for ; Tue, 28 May 2013 12:13:44 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob102.postini.com ([207.126.147.11]) with SMTP ID DSNKUaSfaihau6xP4cPioWjZBmKtT2kRdhKC@postini.com; Tue, 28 May 2013 12:13:44 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Tue, 28 May 2013 15:10:59 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" Subject: How can I create 'pkey' on mellanox IB port? Thread-Topic: How can I create 'pkey' on mellanox IB port? Thread-Index: Ac5bnGmt1FujSxuQTFy5OFl7a4bgoQ== Date: Tue, 28 May 2013 12:10:59 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF65A9A@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 12:13:46 -0000 Hello. I have FreeBSD 9.1 installed. There is mellanox adapter inside. [root@qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3 mlx4_core0@pci0:0:5:0: class=3D0x028000 card=3D0x005015b3 chip=3D0x100315b= 3 rev=3D0x00 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT27500 Family [ConnectX-3]' class =3D network OFED support is already compiled. I try to add pkeys on ib0 port. Usually in Linux I did: echo 0x800c > /sys/class/net/ib0/create_child ifconfig -a To Make sure you see a new interface: ib0.800c How can I do it on FreeBSD? There is no "/sys/class/net/ib0/create_child" d= irectory. Regards, Alex Liptsin From owner-freebsd-infiniband@FreeBSD.ORG Tue May 28 15:45:23 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D88F742D; Tue, 28 May 2013 15:45:23 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-bk0-x231.google.com (mail-bk0-x231.google.com [IPv6:2a00:1450:4008:c01::231]) by mx1.freebsd.org (Postfix) with ESMTP id 44380326; Tue, 28 May 2013 15:45:23 +0000 (UTC) Received: by mail-bk0-f49.google.com with SMTP id na1so2955623bkb.8 for ; Tue, 28 May 2013 08:45:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=GZHpXy+9ypF7/5f3kvEEOKNDVj0419mHaXfQl6MInys=; b=IYknWmpC7XUfiVgUwCtAPekf8uc/EFq8EapmcGmhWY4/NFdmh30/espMyJuckm/Zo1 ozTl1DDZgbf5SkkOp114M0bJEV6BN+fbW7ed1fYF1bY4xrDe4XGBHanOR84HjH4MeGrj qzwvAj9Tf9ulB7hQu/AFoS0IsWKzsVfbQjSPwyIQfxVU6UcnXBp6wUS9vGbE/UeLMS5k VPJKwAnUtCcG3s2H7OpntBGtv/T6ypHjaJ0lDdFFf/lqyJmu1EWSYl7dF9wimZOiDopH /GfrXn93mcFpESoXxj2GbHTu7Zx0KvIc9pvOykaz59scO4/qnKmlWiCfZ+hr2/Jct/uM fabA== MIME-Version: 1.0 X-Received: by 10.204.174.143 with SMTP id t15mr13494019bkz.37.1369755922390; Tue, 28 May 2013 08:45:22 -0700 (PDT) Received: by 10.205.40.129 with HTTP; Tue, 28 May 2013 08:45:22 -0700 (PDT) In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF65A9A@MTLDAG01.mtl.com> References: <64DAB3164E410447932305F50F896D8D6AF65A9A@MTLDAG01.mtl.com> Date: Tue, 28 May 2013 08:45:22 -0700 Message-ID: Subject: Re: How can I create 'pkey' on mellanox IB port? From: Garrett Cooper To: Alex Liptsin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-infiniband@freebsd.org" , Jeff Roberson , Xin LI X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 15:45:23 -0000 On Tue, May 28, 2013 at 5:10 AM, Alex Liptsin wrote: > Hello. > > I have FreeBSD 9.1 installed. > There is mellanox adapter inside. > > [root@qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3 > mlx4_core0@pci0:0:5:0: class=0x028000 card=0x005015b3 chip=0x100315b3 > rev=0x00 hdr=0x00 > vendor = 'Mellanox Technologies' > device = 'MT27500 Family [ConnectX-3]' > class = network > > > OFED support is already compiled. > > I try to add pkeys on ib0 port. > > Usually in Linux I did: > > echo 0x800c > /sys/class/net/ib0/create_child > > ifconfig -a > To Make sure you see a new interface: ib0.800c > > How can I do it on FreeBSD? There is no "/sys/class/net/ib0/create_child" > directory. > Hi Alex! For all sysfs entries there *should* be a corresponding sysctl for it. Do sysctl -Na | grep create_child and you might find the node that you seek. After that all you need to do is call sysctl =0x800c and it should achieve the same thing as setting the sysfs node. Please note that you must be the patch attached to http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174213 in order for things to function properly as the sysfs handling code is still broken on HEAD. I've CCed Jeff Roberson about it a few times, and asked for assistance from Xin as well as I haven't gotten any feedback from Jeff. Thanks, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Tue May 28 15:56:52 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6CF4990E; Tue, 28 May 2013 15:56:52 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-bk0-x22b.google.com (mail-bk0-x22b.google.com [IPv6:2a00:1450:4008:c01::22b]) by mx1.freebsd.org (Postfix) with ESMTP id 9D6CF3F3; Tue, 28 May 2013 15:56:51 +0000 (UTC) Received: by mail-bk0-f43.google.com with SMTP id jm2so2730422bkc.30 for ; Tue, 28 May 2013 08:56:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=uL2xTPn1L5HiYo/HocjOlHx4E7saC7x/Rg5jgaQEpI4=; b=VFJobJWgLI8bu0X7yxVPjS7ki9SelCq9R96CS/TsywqV+WnVBSUzr2kPmSvRmuK4cA pOwbO3dn4e9Slie7HAyWjRUUTflSzmeSS0XwrEEK9dvFsrHqnKuPl/wNsiBHcRLgo+J9 G+lVabbCy6KY5DACqHHFCsRuSLMrl6TlWzBWBzjVGzhQckisIjLQBVpq++FLdN+vUpk4 STQSYto+rzotBV3eb+G773XWDdU/hoEeoar6ZTFPpOX1FdzOMddJWwB6kqgs/YXwfmeR hAJhjRPVo5Yp0PxZkoDS/L36z3nIWjSaj3gYRWQuyH/wIwuChjs5agSNwM8rJgEPd23d 33Og== MIME-Version: 1.0 X-Received: by 10.204.235.129 with SMTP id kg1mr13508098bkb.28.1369756610195; Tue, 28 May 2013 08:56:50 -0700 (PDT) Received: by 10.205.40.129 with HTTP; Tue, 28 May 2013 08:56:50 -0700 (PDT) In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF65A33@MTLDAG01.mtl.com> References: <64DAB3164E410447932305F50F896D8D6AF65A33@MTLDAG01.mtl.com> Date: Tue, 28 May 2013 08:56:50 -0700 Message-ID: Subject: Re: Create bond on Infiniband ports From: Garrett Cooper To: Alex Liptsin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-infiniband@freebsd.org" , Regev Lev , "freebsd-questions@freebsd.org" , "freebsd-net@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 15:56:52 -0000 On Tue, May 28, 2013 at 4:35 AM, Alex Liptsin wrote: > Hi. > > I use FreeBSD 9.1 with OFED compiled on it. > > There is a Mellanox adapter: > > [root@qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3 > mlx4_core0@pci0:0:5:0: class=0x028000 card=0x005015b3 chip=0x100315b3 > rev=0x00 hdr=0x00 > vendor = 'Mellanox Technologies' > device = 'MT27500 Family [ConnectX-3]' > class = network > > I want to create Bond on the two ports (ib0 and ib1) of this device: > > [root@qa-h-vrt-030-006 ~]# ifconfig > em0: flags=8843 metric 0 mtu 1500 > > options=209b > ether 00:50:56:23:1e:06 > inet6 fe80::250:56ff:fe23:1e06%em0 prefixlen 64 scopeid 0x2 > inet 10.195.30.6 netmask 0xffff0000 broadcast 10.195.255.255 > nd6 options=23 > media: Ethernet autoselect (1000baseT ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=23 > ib0: flags=8043 metric 0 mtu 65520 > options=80018 > lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.51 > inet 11.195.30.1 netmask 0xffff0000 broadcast 11.195.255.255 > inet6 fe80::250:56ff:fe23:1e06%ib0 prefixlen 64 scopeid 0x4 > nd6 options=23 > ib1: flags=8043 metric 0 mtu 65520 > options=80018 > lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.52 > inet 12.195.30.1 netmask 0xffff0000 broadcast 12.195.255.255 > inet6 fe80::250:56ff:fe23:1e06%ib1 prefixlen 64 scopeid 0x5 > nd6 options=23 > > > 1. Is there "bond" in FreeBSD or its "Lagg"? > It's lagg. Setting up bonded interfaces is different (Anthony C. or someone else can provide more details about this). failover mode via lagg is completely different from failover mode with IB IIRC. > 2. I tried to create this way, but failed: > > [root@qa-h-vrt-030-006 ~]# ifconfig lagg0 create > > root@qa-h-vrt-031-005 conf]# ifconfig lagg0 laggproto failover laggport > ib0 laggport ib1 > > ifconfig: SIOCSLAGGPORT: Protocol not supported > > Any ideas? > Is it supported on Infiniband ports? > The media setting pieces are look to only be supported with mlx4 in en mode: 1076 static int mlx4_en_ioctl(struct ifnet *dev, u_long command, caddr_t data) 1077 { ... 1115 case SIOCGIFMEDIA: 1116 error = ifmedia_ioctl(dev, ifr, &priv->media, command); 1117 break; If I remember correctly, IB ports with mlx4 default to ib mode, not en mode (it also helps to have the right drivers loaded for this). Cheers, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Wed May 29 07:48:17 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 039C85DE; Wed, 29 May 2013 07:48:17 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog113.obsmtp.com (eu1sys200aog113.obsmtp.com [207.126.144.135]) by mx1.freebsd.org (Postfix) with ESMTP id B14631C3; Wed, 29 May 2013 07:48:15 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob113.postini.com ([207.126.147.11]) with SMTP ID DSNKUaWyvjGZQqf2VFxarAVtZStQk95kIhPF@postini.com; Wed, 29 May 2013 07:48:16 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Wed, 29 May 2013 10:46:31 +0300 From: Alex Liptsin To: Garrett Cooper Subject: RE: How can I create 'pkey' on mellanox IB port? Thread-Topic: How can I create 'pkey' on mellanox IB port? Thread-Index: Ac5bnGmt1FujSxuQTFy5OFl7a4bgoQABM3cAACfNlkA= Date: Wed, 29 May 2013 07:46:30 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF65E71@MTLDAG01.mtl.com> References: <64DAB3164E410447932305F50F896D8D6AF65A9A@MTLDAG01.mtl.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-infiniband@freebsd.org" , Jeff Roberson , Xin LI , Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 07:48:17 -0000 Hi Garrett. Thanks a lot. Where can I download the updated "sysfs.h" file? Regards, Alex Liptsin From: Garrett Cooper [mailto:yanegomi@gmail.com] Sent: Tuesday, May 28, 2013 6:45 PM To: Alex Liptsin Cc: freebsd-infiniband@freebsd.org; Jeff Roberson; Xin LI Subject: Re: How can I create 'pkey' on mellanox IB port? On Tue, May 28, 2013 at 5:10 AM, Alex Liptsin > wrote: Hello. I have FreeBSD 9.1 installed. There is mellanox adapter inside. [root@qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3 mlx4_core0@pci0:0:5:0: class=3D0x028000 card=3D0x005015b3 chip=3D0x100315b= 3 rev=3D0x00 hdr=3D0x00 vendor =3D 'Mellanox Technologies' device =3D 'MT27500 Family [ConnectX-3]' class =3D network OFED support is already compiled. I try to add pkeys on ib0 port. Usually in Linux I did: echo 0x800c > /sys/class/net/ib0/create_child ifconfig -a To Make sure you see a new interface: ib0.800c How can I do it on FreeBSD? There is no "/sys/class/net/ib0/create_child" d= irectory. Hi Alex! For all sysfs entries there *should* be a corresponding sysctl for it. = Do sysctl -Na | grep create_child and you might find the node that you seek= . After that all you need to do is call sysctl =3D0x800c and it shoul= d achieve the same thing as setting the sysfs node. Please note that you must be the patch attached to http://www.freebsd.o= rg/cgi/query-pr.cgi?pr=3Dkern/174213 in order for things to function proper= ly as the sysfs handling code is still broken on HEAD. I've CCed Jeff Rober= son about it a few times, and asked for assistance from Xin as well as I ha= ven't gotten any feedback from Jeff. Thanks, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Thu May 30 07:30:21 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BC7FDB2B; Thu, 30 May 2013 07:30:21 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog102.obsmtp.com (eu1sys200aog102.obsmtp.com [207.126.144.113]) by mx1.freebsd.org (Postfix) with ESMTP id 1ACE8775; Thu, 30 May 2013 07:30:18 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob102.postini.com ([207.126.147.11]) with SMTP ID DSNKUab/7/twKWXQ/hCqjiExEtNY7t/mtVy/@postini.com; Thu, 30 May 2013 07:30:20 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Thu, 30 May 2013 10:29:47 +0300 From: Alex Liptsin To: John Baldwin , "freebsd-net@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: RE: Create pkey on FreeBSD 9.1 Thread-Topic: Create pkey on FreeBSD 9.1 Thread-Index: Ac5Xj0ChbAPWD1OURoemsmYlNeqHFQAPAgSAASseiQAAI6cDkA== Date: Thu, 30 May 2013 07:29:46 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF665F7@MTLDAG01.mtl.com> References: <64DAB3164E410447932305F50F896D8D6AF63DB3@MTLDAG01.mtl.com> <201305291321.08657.jhb@freebsd.org> In-Reply-To: <201305291321.08657.jhb@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Ryan Stone X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 07:30:21 -0000 Hi John. I did it, but there is no ping between the vlans. Ping without VLANs on th= at ports pass. Host1: [root@qa-h-vrt-030-006 ~]# ifconfig ib0.100 create [root@qa-h-vrt-030-006 ~]# ifconfig ib0.100 11.195.30.1/16 up [root@qa-h-vrt-030-006 ~]# ifconfig em0: flags=3D8843 metric 0 mtu 1500 options=3D209b ether 00:50:56:23:1e:06 inet6 fe80::250:56ff:fe23:1e06%em0 prefixlen 64 scopeid 0x2 inet 10.195.30.6 netmask 0xffff0000 broadcast 10.195.255.255 nd6 options=3D23 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D23 ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.51 inet6 fe80::250:56ff:fe23:1e06%ib0 prefixlen 64 scopeid 0x4 nd6 options=3D23 ib1: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.52 nd6 options=3D23 ib0.100: flags=3D8003 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.4a.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.51 inet6 fe80::8200:4a:fe80:0%ib0.100 prefixlen 64 scopeid 0x6 inet 11.195.30.1 netmask 0xffff0000 broadcast 11.195.255.255 nd6 options=3D23 vlan: 100 parent interface: ib0 Host2: [root@qa-h-vrt-031-005 ~]# ifconfig ib0.100 create [root@qa-h-vrt-031-005 ~]# ifconfig ib0.100 11.195.31.1/16 up [root@qa-h-vrt-031-005 ~]# ifconfig em0: flags=3D8843 metric 0 mtu 1500 options=3D209b ether 00:50:56:23:1f:05 inet6 fe80::250:56ff:fe23:1f05%em0 prefixlen 64 scopeid 0x2 inet 10.195.31.5 netmask 0xffff0000 broadcast 10.195.255.255 nd6 options=3D23 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D23 ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.a0.65.91 inet6 fe80::250:56ff:fe23:1f05%ib0 prefixlen 64 scopeid 0x4 nd6 options=3D23 ib1: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.a0.65.92 nd6 options=3D23 ib0.100: flags=3D8003 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.4a.fe.80.0.0.0.0.0.0.0.2.c9.3.0.a0.65.91 inet6 fe80::8200:4a:fe80:0%ib0.100 prefixlen 64 scopeid 0x6 inet 11.195.31.1 netmask 0xffff0000 broadcast 11.195.255.255 nd6 options=3D23 vlan: 100 parent interface: ib0 [root@qa-h-vrt-031-005 ~]# ping 11.195.30.1 PING 11.195.30.1 (11.195.30.1): 56 data bytes ping: sendto: Network is down ping: sendto: Network is down ping: sendto: Network is down Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161=20 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org]=20 Sent: Wednesday, May 29, 2013 9:17 PM To: freebsd-net@freebsd.org Cc: Ryan Stone; Alex Liptsin Subject: Re: Create pkey on FreeBSD 9.1 On Thursday, May 23, 2013 2:36:25 pm Ryan Stone wrote: > On Thu, May 23, 2013 at 4:32 AM, Alex Liptsin wrote: >=20 > > Hello. > > > > I have FreeBSD 9.1 installed. > > There is mellanox adapter inside. > > OFED support is already installed. > > > > I try to add pkeys on ib0 port. > > > > Usually in Linux I did: > > > > echo 0x800c > /sys/class/net/ib0/create_child > > > > ifconfig -a > > To Make sure you see a new interface: ib0.800c > > > > How can I do it on FreeBSD? There is no "/sys/class/net/ib0/create_chil= d" > > directory. > > > > Regards, > > Alex Liptsin > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list=20 > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > >=20 > From reading the source it looks like this is done by attaching a vlan=20 > interface to the interface. So try: >=20 > ifconfig vlan create vlandev ib0 vlan 0xc >=20 > This will create a new vlanX interface (ifconfig will its precise name=20 > with its unit number to stdout). Simpler though is just 'ifconfig ib0.12 create' (and how most folks expect = subinterfaces to be named). -- John Baldwin From owner-freebsd-infiniband@FreeBSD.ORG Thu May 30 12:18:55 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D45D17C9; Thu, 30 May 2013 12:18:55 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id AEF35A7F; Thu, 30 May 2013 12:18:55 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0AD2EB9A9; Thu, 30 May 2013 08:18:55 -0400 (EDT) From: John Baldwin To: Alex Liptsin Subject: Re: Create pkey on FreeBSD 9.1 Date: Thu, 30 May 2013 08:16:17 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <64DAB3164E410447932305F50F896D8D6AF63DB3@MTLDAG01.mtl.com> <201305291321.08657.jhb@freebsd.org> <64DAB3164E410447932305F50F896D8D6AF665F7@MTLDAG01.mtl.com> In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF665F7@MTLDAG01.mtl.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201305300816.17676.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 30 May 2013 08:18:55 -0400 (EDT) Cc: "freebsd-net@freebsd.org" , Ryan Stone , "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 12:18:55 -0000 On Thursday, May 30, 2013 3:29:46 am Alex Liptsin wrote: > Hi John. > > I did it, but there is no ping between the vlans. Ping without VLANs on that ports pass. Unfortunately I do not have an IB setup to test this. I also don't know how IB treats vlans (e.g. does it use an 802.1(q) type header?). Can you tcpdump on the ib0 interface and see if your pings on ib0.100 show up and if they have the appropriate headers? -- John Baldwin