From owner-freebsd-net@FreeBSD.ORG Fri Jul 27 16:16:46 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB220106566B for ; Fri, 27 Jul 2012 16:16:46 +0000 (UTC) (envelope-from adarsh.joshi@qlogic.com) Received: from co1outboundpool.messaging.microsoft.com (co1ehsobe003.messaging.microsoft.com [216.32.180.186]) by mx1.freebsd.org (Postfix) with ESMTP id 757AE8FC1B for ; Fri, 27 Jul 2012 16:16:46 +0000 (UTC) Received: from mail42-co1-R.bigfish.com (10.243.78.231) by CO1EHSOBE011.bigfish.com (10.243.66.74) with Microsoft SMTP Server id 14.1.225.23; Fri, 27 Jul 2012 16:16:40 +0000 Received: from mail42-co1 (localhost [127.0.0.1]) by mail42-co1-R.bigfish.com (Postfix) with ESMTP id 02C845C028F; Fri, 27 Jul 2012 16:16:40 +0000 (UTC) X-Forefront-Antispam-Report: CIP:198.70.193.64; KIP:(null); UIP:(null); IPV:NLI; H:avexcashub1.qlogic.com; RD:avexcashub2.qlogic.com; EFVD:NLI X-SpamScore: -18 X-BigFish: VPS-18(zz98dI9371I936eI542M1432I1453M1447I328cMzz1202hzz8275bh8275dh74efjz2fh2a8h668h839h944hd25hf0ah107ah) Received-SPF: pass (mail42-co1: domain of qlogic.com designates 198.70.193.64 as permitted sender) client-ip=198.70.193.64; envelope-from=adarsh.joshi@qlogic.com; helo=avexcashub1.qlogic.com ; 1.qlogic.com ; Received: from mail42-co1 (localhost.localdomain [127.0.0.1]) by mail42-co1 (MessageSwitch) id 1343405797471223_8429; Fri, 27 Jul 2012 16:16:37 +0000 (UTC) Received: from CO1EHSMHS023.bigfish.com (unknown [10.243.78.249]) by mail42-co1.bigfish.com (Postfix) with ESMTP id 7107C6C028A; Fri, 27 Jul 2012 16:16:37 +0000 (UTC) Received: from avexcashub1.qlogic.com (198.70.193.64) by CO1EHSMHS023.bigfish.com (10.243.66.33) with Microsoft SMTP Server (TLS) id 14.1.225.23; Fri, 27 Jul 2012 16:16:36 +0000 Received: from avexmb1.qlogic.org ([fe80::9545:3a4f:c131:467d]) by avexcashub2.qlogic.org ([::1]) with mapi; Fri, 27 Jul 2012 09:16:35 -0700 From: Adarsh Joshi To: Andrew Boyer Date: Fri, 27 Jul 2012 09:14:55 -0700 Thread-Topic: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Thread-Index: Ac1sCiabdQPmokAcR5SWpvdfGzlB3wACM8L/ Message-ID: <5E4F49720D0BAD499EE1F01232234BA87742F8ABF8@AVEXMB1.qlogic.org> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <4C5FC147-2F49-4AE0-ADF3-C5381DE6580F@averesystems.com> <5E4F49720D0BAD499EE1F01232234BA877435B2F2E@AVEXMB1.qlogic.org> <5E4F49720D0BAD499EE1F01232234BA877435B2F66@AVEXMB1.qlogic.org>, <9BC66C40-6F4B-48A0-AEC0-DCAD0C6863EF@averesystems.com> In-Reply-To: <9BC66C40-6F4B-48A0-AEC0-DCAD0C6863EF@averesystems.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: qlogic.com Cc: "freebsd-net@freebsd.org" Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2012 16:16:46 -0000 Andrew, Thanks for the response. Yes, the link is working fine. I am able to see LA= CPDUs on tcpdump. regards Adarsh ________________________________________ From: Andrew Boyer [aboyer@averesystems.com] Sent: Friday, July 27, 2012 8:11 AM To: Adarsh Joshi Cc: freebsd-net@freebsd.org Subject: Re: lacp lagg port flags do not show correctly resulting in poor t= raffic distribution/performance Adarsh, Sorry for the delay. I'm not an LACP protocol expert, but looking at your logs I don't see ql1 o= n either node receiving a lacpdu response. Are you certain that link is wo= rking? -Andrew On Jul 10, 2012, at 1:53 PM, Adarsh Joshi wrote: > Andrew, > > Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c, > > after typing > > Ifconfig lagg0 create > ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 net= mask 255.255.255.0 > > I compiled it as a standalone driver by the way. > > System 1: > > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:08:05:20 > inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,01D3,8000,000D), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c state=3D3= D > [(8000,00-0E-1E-08-05-20,01D3,8000,000C), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] > > > System 2: > > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:04:2c:f0 > inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c state=3D7= D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,01D3,8000,000C)] > > > System 1 logs : > > Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached > Jul 10 10:38:49 bsd-14 kernel: lacp_attach[740] : lacp_defined > Jul 10 10:38:49 bsd-14 kernel: lagg0: link state changed to UP > Jul 10 10:38:49 bsd-14 kernel: ql0: media changed 0x0 -> 0x100033, ether = =3D 1, fdx =3D 1, link =3D 1 > Jul 10 10:38:49 bsd-14 kernel: ql0: -> UNSELECTED > Jul 10 10:38:49 bsd-14 kernel: ql1: media changed 0x0 -> 0x100033, ether = =3D 1, fdx =3D 1, link =3D 1 > Jul 10 10:38:49 bsd-14 kernel: ql1: -> UNSELECTED > Jul 10 10:38:49 bsd-14 kernel: lacp_select_tx_port: no active aggregator > Jul 10 10:38:50 bsd-14 kernel: ql1: port lagid=3D[(8000,00-0E-1E-08-05-20= ,01D3,8000,000D),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created > Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-08= -05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1 > Jul 10 10:38:50 bsd-14 kernel: ql0: port lagid=3D[(8000,00-0E-1E-08-05-20= ,01D3,8000,000C),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created > Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-08= -05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1 > Jul 10 10:38:51 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000D) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: partner=3D(0000,00-00-00-00-00-00,0000,000= 0,0000) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: partner=3D(0000,00-00-00-00-00-00,0000,000= 0,0000) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:51 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: old pstate 2 > Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5 > Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout changed > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:51 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTE= D > Jul 10 10:38:52 bsd-14 kernel: ql1: partner timeout changed > Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)],= pending 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: collecting disabled > Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=3D[(8000,00-= 0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], re= fcnt 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: mux_state 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:52 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000D) > Jul 10 10:38:52 bsd-14 kernel: actor.state=3D45 > Jul 10 10:38:52 bsd-14 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFF= F,0000) > Jul 10 10:38:52 bsd-14 kernel: partner.state=3D3c > Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:52 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)],= pending 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: collecting disabled > Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=3D[(8000,00-= 0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], re= fcnt 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: mux_state 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:52 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 10:38:52 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:52 bsd-14 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,800= 0,000E) > Jul 10 10:38:52 bsd-14 kernel: partner.state=3D5 > Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:53 bsd-14 kernel: ql1: port lagid=3D[(8000,00-0E-1E-08-05-20= ,01D3,8000,000D),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created > Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-08= -05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1 > Jul 10 10:38:53 bsd-14 kernel: ql0: port lagid=3D[(8000,00-0E-1E-08-05-20= ,01D3,8000,000C),(8000,00-0E-1E-04-2C-F0,0213,8000,000E)] > Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created > Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-08= -05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1 > Jul 10 10:38:54 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:54 bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 10:38:54 bsd-14 kernel: actor.state=3Dd > Jul 10 10:38:54 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 10:38:54 bsd-14 kernel: partner.state=3D5 > Jul 10 10:38:54 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:54 bsd-14 kernel: ql0: old pstate 5 > Jul 10 10:38:54 bsd-14 kernel: ql0: new pstate d > Jul 10 10:38:55 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)],= pending 1 -> 0 > Jul 10 10:38:55 bsd-14 kernel: ql1: collecting disabled > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 1 -> 2 > Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3 > Jul 10 10:38:55 bsd-14 kernel: ql1: enable distributing on aggregator [(8= 000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,000= 0)], nports 0 -> 1 > Jul 10 10:38:55 bsd-14 kernel: lacp_select_active_aggregator: > Jul 10 10:38:55 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(= FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:55 bsd-14 kernel: active aggregator changed > Jul 10 10:38:55 bsd-14 kernel: old (none) > Jul 10 10:38:55 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,000= 0),(FFFF,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports > Jul 10 10:38:55 bsd-14 kernel: lacp_suppress_distributing > Jul 10 10:38:55 bsd-14 kernel: ql1: marker transmit, port=3D13, sys=3D00:= 0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: ql0: marker transmit, port=3D12, sys=3D00:= 0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4 > Jul 10 10:38:55 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:55 bsd-14 kernel: ql0: marker response, port=3D12, sys=3D00:= 0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000D) > Jul 10 10:38:55 bsd-14 kernel: actor.state=3D7d > Jul 10 10:38:55 bsd-14 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFF= F,0000) > Jul 10 10:38:55 bsd-14 kernel: partner.state=3D3c > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)],= pending 1 -> 0 > Jul 10 10:38:55 bsd-14 kernel: ql0: collecting disabled > Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 1 -> 2 > Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled > Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:55 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 10:38:55 bsd-14 kernel: actor.state=3D1d > Jul 10 10:38:55 bsd-14 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,800= 0,000E) > Jul 10 10:38:55 bsd-14 kernel: partner.state=3Dd > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:55 bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 10:38:55 bsd-14 kernel: actor.state=3D3d > Jul 10 10:38:55 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 10:38:55 bsd-14 kernel: partner.state=3D1d > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: old pstate d > Jul 10 10:38:55 bsd-14 kernel: ql0: new pstate 3d > Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator [(8= 000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,000= 0)], nports 0 -> 1 > Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregator: > Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(= FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(= 8000,00-0E-1E-04-2C-F0,0213,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:56 bsd-14 kernel: active aggregator changed > Jul 10 10:38:56 bsd-14 kernel: old [(8000,00-0E-1E-08-05-20,01D3,0000,000= 0),(FFFF,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 10:38:56 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,000= 0),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports > Jul 10 10:38:56 bsd-14 kernel: lacp_suppress_distributing > Jul 10 10:38:56 bsd-14 kernel: ql1: marker transmit, port=3D13, sys=3D00:= 0e:1e:08:05:20, id=3D2 > Jul 10 10:38:56 bsd-14 kernel: ql0: marker transmit, port=3D12, sys=3D00:= 0e:1e:08:05:20, id=3D2 > Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4 > Jul 10 10:38:56 bsd-14 kernel: ql0: marker response, port=3D12, sys=3D00:= 0e:1e:08:05:20, id=3D2 > Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire > ^C > > > > > System 2 logs : > > Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached > Jul 10 02:38:24 bsd-15 kernel: lacp_attach[740] : lacp_defined > Jul 10 02:38:24 bsd-15 kernel: lagg0: link state changed to UP > Jul 10 02:38:24 bsd-15 kernel: ql0: media changed 0x0 -> 0x100033, ether = =3D 1, fdx =3D 1, link =3D 1 > Jul 10 02:38:24 bsd-15 kernel: ql0: -> UNSELECTED > Jul 10 02:38:24 bsd-15 kernel: ql1: media changed 0x0 -> 0x100033, ether = =3D 1, fdx =3D 1, link =3D 1 > Jul 10 02:38:24 bsd-15 kernel: ql1: -> UNSELECTED > Jul 10 02:38:24 bsd-15 kernel: lacp_select_tx_port: no active aggregator > Jul 10 02:38:25 bsd-15 kernel: ql1: port lagid=3D[(8000,00-0E-1E-04-2C-F0= ,0213,8000,000F),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created > Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-04= -2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1 > Jul 10 02:38:25 bsd-15 kernel: ql0: port lagid=3D[(8000,00-0E-1E-04-2C-F0= ,0213,8000,000E),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created > Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-04= -2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: partner=3D(0000,00-00-00-00-00-00,0000,000= 0,0000) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_update_ntt: assert ntt > Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2 > Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate 85 > Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:26 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000F) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: partner=3D(0000,00-00-00-00-00-00,0000,000= 0,0000) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql0: collecting disabled > Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: lagid=3D[(8000,00-= 0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], r= efcnt 1 -> 0 > > Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:26 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:27 bsd-15 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 02:38:27 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:27 bsd-15 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,800= 0,000E) > Jul 10 02:38:27 bsd-15 kernel: partner.state=3D5 > Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: old pstate 85 > Jul 10 02:38:27 bsd-15 kernel: ql0: new pstate 5 > Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTE= D > Jul 10 02:38:27 bsd-15 kernel: ql1: partner timeout changed > Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-04-2C-F0, 0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]= , pending 1 -> 0 > Jul 10 02:38:27 bsd-15 kernel: ql1: collecting disabled > Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_delref: lagid=3D[(8000,00-= 0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], r= efcnt 1 -> 0 > > Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0 > Jul 10 02:38:27 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:27 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000F) > Jul 10 02:38:27 bsd-15 kernel: actor.state=3D45 > Jul 10 02:38:27 bsd-15 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFF= F,0000) > Jul 10 02:38:27 bsd-15 kernel: partner.state=3D3c > Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: port lagid=3D[(8000,00-0E-1E-04-2C-F0= ,0213,8000,000E),(8000,00-0E-1E-08-05-20,01D3,8000,000C)] > Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created > Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-04= -2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)] > Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1 > Jul 10 02:38:28 bsd-15 kernel: ql1: port lagid=3D[(8000,00-0E-1E-04-2C-F0= ,0213,8000,000F),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created > Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-04= -2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1 > Jul 10 02:38:29 bsd-15 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-04-2C-F0, 0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)]= , pending 1 -> 0 > Jul 10 02:38:29 bsd-15 kernel: ql0: collecting disabled > Jul 10 02:38:29 bsd-15 kernel: ql0: mux_state 1 -> 2 > Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:29 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 02:38:29 bsd-15 kernel: actor.state=3Dd > Jul 10 02:38:29 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 02:38:29 bsd-15 kernel: partner.state=3D5 > Jul 10 02:38:29 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:30 bsd-15 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,= 000C) > Jul 10 02:38:30 bsd-15 kernel: actor.state=3D1d > Jul 10 02:38:30 bsd-15 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,800= 0,000E) > Jul 10 02:38:30 bsd-15 kernel: partner.state=3Dd > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: old pstate 5 > Jul 10 02:38:30 bsd-15 kernel: ql0: new pstate 1d > Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,= 00-0E-1E-04-2C-F0, 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]= , pending 1 -> 0 > Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2 > Jul 10 02:38:30 bsd-15 kernel: ql1: collecting enabled > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 2 -> 3 > Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregator [(8= 000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,000= 0)], nports 0 -> 1 > Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: > Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(= FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: active aggregator changed > Jul 10 02:38:30 bsd-15 kernel: old (none) > Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,000= 0),(FFFF,00-00-00 00-00-00,0000,0000,0000)] > Jul 10 02:38:30 bsd-15 kernel: Set table 1 with 1 ports > Jul 10 02:38:30 bsd-15 kernel: lacp_suppress_distributing > Jul 10 02:38:30 bsd-15 kernel: ql1: marker transmit, port=3D15, sys=3D00:= 0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: ql0: marker transmit, port=3D14, sys=3D00:= 0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4 > Jul 10 02:38:30 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:30 bsd-15 kernel: ql0: marker response, port=3D14, sys=3D00:= 0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000F) > Jul 10 02:38:30 bsd-15 kernel: actor.state=3D7d > Jul 10 02:38:30 bsd-15 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFF= F,0000) > Jul 10 02:38:30 bsd-15 kernel: partner.state=3D3c > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: collecting enabled > Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> 3 > Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator [(8= 000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,000= 0)], nports 0 -> 1 > Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: > Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(= 8000,00-0E-1E-08-05-20,01D3,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(= FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: active aggregator not changed > Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,000= 0),(FFFF,00-00-00-00-00-00,0000,0000,0000)] > Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4 > Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:30 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,= 000E) > Jul 10 02:38:30 bsd-15 kernel: actor.state=3D3d > Jul 10 02:38:30 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,800= 0,000C) > Jul 10 02:38:30 bsd-15 kernel: partner.state=3D1d > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:33 bsd-15 kernel: lacp_transit_expire > ^C > > Let me know if you need more info. > > Thanks > Adarsh > > > -----Original Message----- > From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org= ] On Behalf Of Adarsh Joshi > Sent: Tuesday, July 10, 2012 10:08 AM > To: Andrew Boyer > Cc: freebsd-net@freebsd.org > Subject: RE: lacp lagg port flags do not show correctly resulting in poor= traffic distribution/performance > > Andrew, > > Thanks for the reply. > > The reason for my suspicion on the portflags is thus (extracted from the = ifconfig output in my previous mail): > > System 1: > Laggport: ql1 flags =3D 18 state =3D 7D > Laggport: ql0 flags =3D 1c state =3D 3D > > System 2: > Laggport: ql1 flags =3D 1c state =3D 7D > Laggport: ql0 flags =3D 18 state =3D 3D > > I should have explained my setup to you before. Here it is. > Both the ql0 interfaces of the 2 systems are connected using a single cab= le and ql1 interfaces of the 2 systems are connected using a single cable. > > System 1 System 2 > ql0 <=3D=3D=3D=3D=3D=3D=3D> ql0 > ql1 <=3D=3D=3D=3D=3D=3D=3D> ql1 > > With this setup, I don't think it is possible for ports ql0 to talk to th= eir partners (each other) and ql1 ports not getting a response from their p= artner and still get the lagg configuration I have posted. > > I thought the portflags are dependent on the LACP state. But I see differ= ent flags for the same LACP state (For the state 7D, ql1 on system 1 shows = flags =3D 18 and ql1 on system 2 shows flags =3D 1c). > > Or is my understanding totally wrong? > > I will send the LACP_DEBUG logs within the hour. > > Thanks > Adarsh > > From: Andrew Boyer [mailto:aboyer@averesystems.com] > Sent: Tuesday, July 10, 2012 5:57 AM > To: Adarsh Joshi > Cc: freebsd-net@freebsd.org > Subject: Re: lacp lagg port flags do not show correctly resulting in poor= traffic distribution/performance > > > On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote: > > > Hi, > > I am trying to configure lacp lagg interfaces with 2 systems connected ba= ck to back as follows: > > Ifconfig lagg0 create > Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 net= mask 255.255.255.0 > > Sometimes, the lag interface comes up correctly but sometimes the laggpor= t flags do not show properly. Instead of 1c= , it shows values of 18. I have seen similar issues reported on various for= ums with no solution. > Looking at the lagg driver code and reading the standard, I thought the l= aggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in f= ile ieee8023ad_lacp.h. But the following ifconfig -v output does not make a= ny sense to me. > > My concern is that when all the interfaces show flags as 1c, the traffic = is distributed across both the interfaces uniformly and I get aggregated th= roughput. If not, the traffic flows only on 1 interface. > > Is this a bug? How do I solve this? Or am I doing something wrong? > > I am using Free-BSD 9.0 release. > > System 1: > # ifconfig -v lagg0 > lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c state=3D3D > [(8000,00-0E-1E-08-05-20,0213,8000,000E), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] > > System 2: > > # ifconfig -v lagg0 > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c state=3D7D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,0213,8000,000E)] > > > thanks > Adarsh > > I don't think you have a port flags problem per se; the flags are correct= ly displaying the state of the lagg. Your problem is that your systems are= n't negotiating the correct lagg configuration. Each tuple after the laggp= ort represents the [(actor state),(partner state)]. Ports ql0 have been ab= le to talk to their partners (each other). Neither ql1 port has seen a res= ponse from a partner, though. > > You could try restarting the state machine on one box with 'ifconfig lagg= 0 laggproto lacp'. To see the negotiation you'll need to rebuild your kern= el with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp= .c. Or upgrade to a newer stable snapshot that has the net.lacp_debug sysc= tl and turn it on. > > Or just turn off LACP. What does it get you in this configuration? > > Hope this helps, > Andrew > > -------------------------------------------------- > Andrew Boyer aboyer@averesystems.com > > > > > > ________________________________ > This message and any attached documents contain information from QLogic C= orporation or its wholly-owned subsidiaries that may be confidential. If yo= u are not the intended recipient, you may not read, copy, distribute, or us= e this information. If you have received this transmission in error, please= notify the sender immediately by reply e-mail and then delete this message= . > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > This message and any attached documents contain information from QLogic C= orporation or its wholly-owned subsidiaries that may be confidential. If yo= u are not the intended recipient, you may not read, copy, distribute, or us= e this information. If you have received this transmission in error, please= notify the sender immediately by reply e-mail and then delete this message= . > -------------------------------------------------- Andrew Boyer aboyer@averesystems.com This message and any attached documents contain information from QLogic Cor= poration or its wholly-owned subsidiaries that may be confidential. If you = are not the intended recipient, you may not read, copy, distribute, or use = this information. If you have received this transmission in error, please n= otify the sender immediately by reply e-mail and then delete this message.