From owner-freebsd-net@FreeBSD.ORG Tue Jul 10 17:15:41 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89C6B106564A for ; Tue, 10 Jul 2012 17:15:41 +0000 (UTC) (envelope-from adarsh.joshi@qlogic.com) Received: from va3outboundpool.messaging.microsoft.com (va3ehsobe004.messaging.microsoft.com [216.32.180.14]) by mx1.freebsd.org (Postfix) with ESMTP id 286038FC14 for ; Tue, 10 Jul 2012 17:15:40 +0000 (UTC) Received: from mail82-va3-R.bigfish.com (10.7.14.250) by VA3EHSOBE004.bigfish.com (10.7.40.24) with Microsoft SMTP Server id 14.1.225.23; Tue, 10 Jul 2012 17:13:20 +0000 Received: from mail82-va3 (localhost [127.0.0.1]) by mail82-va3-R.bigfish.com (Postfix) with ESMTP id DBE7B42008C; Tue, 10 Jul 2012 17:13:20 +0000 (UTC) X-Forefront-Antispam-Report: CIP:198.70.193.61; KIP:(null); UIP:(null); IPV:NLI; H:avexcashub1.qlogic.com; RD:avexcashub1.qlogic.com; EFVD:NLI X-SpamScore: -4 X-BigFish: VPS-4(zz98dI9371I542M1432I1447I853kzz1202hzz8275dh74efjz2fh2a8h668h839h944hd25hf0ah107ah) Received-SPF: pass (mail82-va3: domain of qlogic.com designates 198.70.193.61 as permitted sender) client-ip=198.70.193.61; envelope-from=adarsh.joshi@qlogic.com; helo=avexcashub1.qlogic.com ; 1.qlogic.com ; Received: from mail82-va3 (localhost.localdomain [127.0.0.1]) by mail82-va3 (MessageSwitch) id 1341940399772526_27550; Tue, 10 Jul 2012 17:13:19 +0000 (UTC) Received: from VA3EHSMHS009.bigfish.com (unknown [10.7.14.240]) by mail82-va3.bigfish.com (Postfix) with ESMTP id B79514E0051; Tue, 10 Jul 2012 17:13:19 +0000 (UTC) Received: from avexcashub1.qlogic.com (198.70.193.61) by VA3EHSMHS009.bigfish.com (10.7.99.19) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 10 Jul 2012 17:13:17 +0000 Received: from avexmb1.qlogic.org ([fe80::9545:3a4f:c131:467d]) by avexcashub1.qlogic.org ([::1]) with mapi; Tue, 10 Jul 2012 10:15:34 -0700 From: Adarsh Joshi To: Jason Hellenthal Date: Tue, 10 Jul 2012 10:15:32 -0700 Thread-Topic: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Thread-Index: Ac1eaxJj5XIwqx19SUajOXKTV+ranwAVBkpw Message-ID: <5E4F49720D0BAD499EE1F01232234BA877435B2F38@AVEXMB1.qlogic.org> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <20120710071011.GB91639@DataIX.net> In-Reply-To: <20120710071011.GB91639@DataIX.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: qlogic.com Cc: "freebsd-net@freebsd.org" Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2012 17:15:41 -0000 Jason, Thanks for the reply. Why do you say - " Given this is LAgg and LACP you will see some variation = regardless " Can you please throw some more light on this? I understand if there is a slight variation in the throughput when LAGG and= LACP is configured but in my case, I do not see a single packet on 1 of th= e interface except for LACPDUs and other LACP control packets. Thanks Adarsh -----Original Message----- From: Jason Hellenthal [mailto:jhellenthal@dataix.net] Sent: Tuesday, July 10, 2012 12:10 AM To: Adarsh Joshi Cc: freebsd-net@freebsd.org Subject: Re: lacp lagg port flags do not show correctly resulting in poor t= raffic distribution/performance On Mon, Jul 09, 2012 at 05:38:24PM -0700, Adarsh Joshi wrote: > Hi, > > I am trying to configure lacp lagg interfaces with 2 systems connected ba= ck to back as follows: > > Ifconfig lagg0 create > Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 > netmask 255.255.255.0 > > Sometimes, the lag interface comes up correctly but sometimes the laggpor= t flags do not show properly. Instead of 1c= , it shows values of 18. I have seen similar issues reported on various for= ums with no solution. > Looking at the lagg driver code and reading the standard, I thought the l= aggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in f= ile ieee8023ad_lacp.h. But the following ifconfig -v output does not make a= ny sense to me. > > My concern is that when all the interfaces show flags as 1c, the traffic = is distributed across both the interfaces uniformly and I get aggregated th= roughput. If not, the traffic flows only on 1 interface. > > Is this a bug? How do I solve this? Or am I doing something wrong? > > I am using Free-BSD 9.0 release. > > System 1: > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:08:05:20 > inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c state=3D= 3D > [(8000,00-0E-1E-08-05-20,0213,8000,000E), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] > > System 2: > > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:04:2c:f0 > inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c state=3D= 7D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,0213,8000,000E)] > > Just for reference ... (stable/8 @ r238264) lagg0: flags=3D8843 metric 0 mtu 15= 00 options=3D80048 ether 00:0c:41:21:1d:b5 inet 192.168.XX.X netmask 0xffffff00 broadcast 192.168.XX.XXX media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 lag id: [(8000,00-0C-41-21-1D-B5,00E6,0000,0000), (FFFF,00-00-00-00-00-00,0000,0000,0000)] laggport: dc1 flags=3D1c state=3D7D [(8000,00-0C-41-21-1D-B5,00E6,8000,0002), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: dc0 flags=3D1c state=3D7D [(8000,00-0C-41-21-1D-B5,00E6,8000,0001), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] They have had flags =3D 1c for quite some time and state =3D 7D And just to show the variation ... dc0: yesterday 8.53 MiB / 2.61 MiB / 11.14 MiB today 693 KiB / 156 KiB / 849 KiB dc1: yesterday 19.00 MiB / 1.79 MiB / 20.78 MiB today 496 KiB / 103 KiB / 599 KiB lagg0: yesterday 27.53 MiB / 3.71 MiB / 31.24 MiB today 1.16 MiB / 172 KiB / 1.33 MiB I believe (know) there has been some changes in the LAgg code in stable/9 and stable/8 recently so you may want to check into that. Given this is LAgg and LACP you will see some variation regardless but I re= call a point that it seemed like one interface was being favored over the o= ther quite repeatedly or obsessively that had me second guessing whether it= was doing the right thing. LACP in Cisco is quite different than how we treat it here in FreeBSD as it= tends to use the interfaces quite evenly all the time so that also has me = second guessing whether the right thing is happening here. ( in PAgP and LA= CP modes ). These tunables (sysctl)'s may also lead you in direction you want to look i= n. net.link.lagg.default_use_flowid: 1 net.link.lagg.failover_rx_all: 0 net.link.lagg.0.use_flowid: 1 -rxcsum & -txcsum for lo0 and ql0 ql1 might be of benefit too though I only= turn them off on lo0. Good luck -- - (2^(N-1)) This message and any attached documents contain information from QLogic Cor= poration or its wholly-owned subsidiaries that may be confidential. If you = are not the intended recipient, you may not read, copy, distribute, or use = this information. If you have received this transmission in error, please n= otify the sender immediately by reply e-mail and then delete this message.