From owner-freebsd-net@FreeBSD.ORG Tue Jul 10 17:23:26 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6B9E1065670 for ; Tue, 10 Jul 2012 17:23:26 +0000 (UTC) (envelope-from adarsh.joshi@qlogic.com) Received: from va3outboundpool.messaging.microsoft.com (va3ehsobe004.messaging.microsoft.com [216.32.180.14]) by mx1.freebsd.org (Postfix) with ESMTP id 5C9848FC25 for ; Tue, 10 Jul 2012 17:23:26 +0000 (UTC) Received: from mail10-va3-R.bigfish.com (10.7.14.239) by VA3EHSOBE001.bigfish.com (10.7.40.21) with Microsoft SMTP Server id 14.1.225.23; Tue, 10 Jul 2012 17:05:56 +0000 Received: from mail10-va3 (localhost [127.0.0.1]) by mail10-va3-R.bigfish.com (Postfix) with ESMTP id C65F52A031D; Tue, 10 Jul 2012 17:05:56 +0000 (UTC) X-Forefront-Antispam-Report: CIP:198.70.193.61; KIP:(null); UIP:(null); IPV:NLI; H:avexcashub1.qlogic.com; RD:avexcashub1.qlogic.com; EFVD:NLI X-SpamScore: -6 X-BigFish: VPS-6(zz98dI9371Ic89bhc85dh1447I328cMzz1202hzz8275bh8275dh74efjz2fh2a8h668h839hd25hf0ah107ah) Received-SPF: pass (mail10-va3: domain of qlogic.com designates 198.70.193.61 as permitted sender) client-ip=198.70.193.61; envelope-from=adarsh.joshi@qlogic.com; helo=avexcashub1.qlogic.com ; 1.qlogic.com ; Received: from mail10-va3 (localhost.localdomain [127.0.0.1]) by mail10-va3 (MessageSwitch) id 1341939952606608_17899; Tue, 10 Jul 2012 17:05:52 +0000 (UTC) Received: from VA3EHSMHS018.bigfish.com (unknown [10.7.14.245]) by mail10-va3.bigfish.com (Postfix) with ESMTP id 807244E0128; Tue, 10 Jul 2012 17:05:52 +0000 (UTC) Received: from avexcashub1.qlogic.com (198.70.193.61) by VA3EHSMHS018.bigfish.com (10.7.99.28) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 10 Jul 2012 17:05:49 +0000 Received: from avexmb1.qlogic.org ([fe80::9545:3a4f:c131:467d]) by avexcashub1.qlogic.org ([::1]) with mapi; Tue, 10 Jul 2012 10:08:07 -0700 From: Adarsh Joshi To: Andrew Boyer Date: Tue, 10 Jul 2012 10:08:05 -0700 Thread-Topic: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Thread-Index: Ac1em3iDmAWR/fFfTBmYMQ9wIwfjmAAINpAg Message-ID: <5E4F49720D0BAD499EE1F01232234BA877435B2F2E@AVEXMB1.qlogic.org> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <4C5FC147-2F49-4AE0-ADF3-C5381DE6580F@averesystems.com> In-Reply-To: <4C5FC147-2F49-4AE0-ADF3-C5381DE6580F@averesystems.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-OriginatorOrg: qlogic.com Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-net@freebsd.org" Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2012 17:23:26 -0000 Andrew, Thanks for the reply. The reason for my suspicion on the portflags is thus (extracted from the if= config output in my previous mail): System 1: Laggport: ql1 flags =3D 18 state =3D 7D Laggport: ql0 flags =3D 1c state =3D 3D System 2: Laggport: ql1 flags =3D 1c state =3D 7D Laggport: ql0 flags =3D 18 state =3D 3D I should have explained my setup to you before. Here it is. Both the ql0 interfaces of the 2 systems are connected using a single cable= and ql1 interfaces of the 2 systems are connected using a single cable. System 1 System 2 ql0 <=3D=3D=3D=3D=3D=3D=3D> ql0 ql1 <=3D=3D=3D=3D=3D=3D=3D> ql1 With this setup, I don't think it is possible for ports ql0 to talk to thei= r partners (each other) and ql1 ports not getting a response from their par= tner and still get the lagg configuration I have posted. I thought the portflags are dependent on the LACP state. But I see differen= t flags for the same LACP state (For the state 7D, ql1 on system 1 shows fl= ags =3D 18 and ql1 on system 2 shows flags =3D 1c). Or is my understanding totally wrong? I will send the LACP_DEBUG logs within the hour. Thanks Adarsh From: Andrew Boyer [mailto:aboyer@averesystems.com] Sent: Tuesday, July 10, 2012 5:57 AM To: Adarsh Joshi Cc: freebsd-net@freebsd.org Subject: Re: lacp lagg port flags do not show correctly resulting in poor t= raffic distribution/performance On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote: Hi, I am trying to configure lacp lagg interfaces with 2 systems connected back= to back as follows: Ifconfig lagg0 create Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netma= sk 255.255.255.0 Sometimes, the lag interface comes up correctly but sometimes the laggport = flags do not show properly. Instead of 1c, = it shows values of 18. I have seen similar issues reported on various forum= s with no solution. Looking at the lagg driver code and reading the standard, I thought the lag= gport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in fil= e ieee8023ad_lacp.h. But the following ifconfig -v output does not make any= sense to me. My concern is that when all the interfaces show flags as 1c, the traffic is= distributed across both the interfaces uniformly and I get aggregated thro= ughput. If not, the traffic flows only on 1 interface. Is this a bug? How do I solve this? Or am I doing something wrong? I am using Free-BSD 9.0 release. System 1: # ifconfig -v lagg0 lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] laggport: ql1 flags=3D18 state=3D7D [(8000,00-0E-1E-08-05-20,0213,8000,000F), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=3D1c state=3D3D [(8000,00-0E-1E-08-05-20,0213,8000,000E), (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] System 2: # ifconfig -v lagg0 lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), (FFFF,00-00-00-00-00-00,0000,0000,0000)] laggport: ql1 flags=3D1c state=3D7D [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=3D18 state=3D3D [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), (8000,00-0E-1E-08-05-20,0213,8000,000E)] thanks Adarsh I don't think you have a port flags problem per se; the flags are correctly= displaying the state of the lagg. Your problem is that your systems aren'= t negotiating the correct lagg configuration. Each tuple after the laggpor= t represents the [(actor state),(partner state)]. Ports ql0 have been able= to talk to their partners (each other). Neither ql1 port has seen a respo= nse from a partner, though. You could try restarting the state machine on one box with 'ifconfig lagg0 = laggproto lacp'. To see the negotiation you'll need to rebuild your kernel= with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp.c= . Or upgrade to a newer stable snapshot that has the net.lacp_debug sysctl= and turn it on. Or just turn off LACP. What does it get you in this configuration? Hope this helps, Andrew -------------------------------------------------- Andrew Boyer aboyer@averesystems.com ________________________________ This message and any attached documents contain information from QLogic Cor= poration or its wholly-owned subsidiaries that may be confidential. If you = are not the intended recipient, you may not read, copy, distribute, or use = this information. If you have received this transmission in error, please n= otify the sender immediately by reply e-mail and then delete this message.