From owner-freebsd-net@FreeBSD.ORG Fri Jul 27 15:11:45 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 43E31106566B for ; Fri, 27 Jul 2012 15:11:45 +0000 (UTC) (envelope-from aboyer@averesystems.com) Received: from mail.averesystems.com (mail.averesystems.com [208.70.68.85]) by mx1.freebsd.org (Postfix) with ESMTP id 0D1568FC0C for ; Fri, 27 Jul 2012 15:11:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.averesystems.com (Postfix) with ESMTP id 912E34808CC; Fri, 27 Jul 2012 11:11:47 -0400 (EDT) X-Virus-Scanned: amavisd-new at mail.averesystems.com Received: from mail.averesystems.com ([127.0.0.1]) by localhost (mail.averesystems.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M+Zd+0zXUVDA; Fri, 27 Jul 2012 11:11:44 -0400 (EDT) Received: from riven.arriad.com (206.193.225.214.nauticom.net [206.193.225.214]) by mail.averesystems.com (Postfix) with ESMTPSA id B2D4C4808C5; Fri, 27 Jul 2012 11:11:44 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: Andrew Boyer In-Reply-To: <5E4F49720D0BAD499EE1F01232234BA877435B2F66@AVEXMB1.qlogic.org> Date: Fri, 27 Jul 2012 11:11:41 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <9BC66C40-6F4B-48A0-AEC0-DCAD0C6863EF@averesystems.com> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <4C5FC147-2F49-4AE0-ADF3-C5381DE6580F@averesystems.com> <5E4F49720D0BAD499EE1F01232234BA877435B2F2E@AVEXMB1.qlogic.org> <5E4F49720D0BAD499EE1F01232234BA877435B2F66@AVEXMB1.qlogic.org> To: Adarsh Joshi X-Mailer: Apple Mail (2.1278) Cc: "freebsd-net@freebsd.org" Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2012 15:11:45 -0000 Adarsh, Sorry for the delay. I'm not an LACP protocol expert, but looking at your logs I don't see = ql1 on either node receiving a lacpdu response. Are you certain that = link is working? -Andrew On Jul 10, 2012, at 1:53 PM, Adarsh Joshi wrote: > Andrew, >=20 > Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c, >=20 > after typing >=20 > Ifconfig lagg0 create > ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 = netmask 255.255.255.0 >=20 > I compiled it as a standalone driver by the way. >=20 > System 1: >=20 > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 = mtu 1500 > = options=3D13b > ether 00:0e:1e:08:05:20 > inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,01D3,8000,000D), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c = state=3D3D > [(8000,00-0E-1E-08-05-20,01D3,8000,000C), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] >=20 >=20 > System 2: >=20 > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 = mtu 1500 > = options=3D13b > ether 00:0e:1e:04:2c:f0 > inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c = state=3D7D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,01D3,8000,000C)] >=20 >=20 > System 1 logs : >=20 > Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached > Jul 10 10:38:49 bsd-14 kernel: lacp_attach[740] : lacp_defined > Jul 10 10:38:49 bsd-14 kernel: lagg0: link state changed to UP > Jul 10 10:38:49 bsd-14 kernel: ql0: media changed 0x0 -> 0x100033, = ether =3D 1, fdx =3D 1, link =3D 1 > Jul 10 10:38:49 bsd-14 kernel: ql0: -> UNSELECTED > Jul 10 10:38:49 bsd-14 kernel: ql1: media changed 0x0 -> 0x100033, = ether =3D 1, fdx =3D 1, link =3D 1 > Jul 10 10:38:49 bsd-14 kernel: ql1: -> UNSELECTED > Jul 10 10:38:49 bsd-14 kernel: lacp_select_tx_port: no active = aggregator > Jul 10 10:38:50 bsd-14 kernel: ql1: port = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created > Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1 > Jul 10 10:38:50 bsd-14 kernel: ql0: port = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created > Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1 > Jul 10 10:38:51 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:51 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000D) > Jul 10 10:38:51 bsd-14 kernel: = actor.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: = partner=3D(0000,00-00-00-00-00-00,0000,0000,0000) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:51 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:51 bsd-14 kernel: = actor.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: = partner=3D(0000,00-00-00-00-00-00,0000,0000,0000) > Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:51 bsd-14 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:51 bsd-14 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:51 bsd-14 kernel: = partner.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:51 bsd-14 kernel: ql0: old pstate 2 > Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5 > Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout changed > Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:51 bsd-14 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:51 bsd-14 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:51 bsd-14 kernel: = partner.state=3D85 > Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> = DEFAULTED > Jul 10 10:38:52 bsd-14 kernel: ql1: partner timeout changed > Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000= ,0000)], pending 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: collecting disabled > Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)], refcnt 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: mux_state 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:52 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000D) > Jul 10 10:38:52 bsd-14 kernel: = actor.state=3D45 > Jul 10 10:38:52 bsd-14 kernel: = partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,0000) > Jul 10 10:38:52 bsd-14 kernel: = partner.state=3D3c > Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:52 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000= ,0000)], pending 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: collecting disabled > Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)], refcnt 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: mux_state 1 -> 0 > Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:52 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:52 bsd-14 kernel: actor.state=3D5 > Jul 10 10:38:52 bsd-14 kernel: = partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:52 bsd-14 kernel: partner.state=3D5 > Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:53 bsd-14 kernel: ql1: port = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(FFFF,00-00-00-00-00-00,0= 000,FFFF,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created > Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1 > Jul 10 10:38:53 bsd-14 kernel: ql0: port = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(8000,00-0E-1E-04-2C-F0,0= 213,8000,000E)] > Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created > Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator = lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0= 213,0000,0000)] > Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1 > Jul 10 10:38:54 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:54 bsd-14 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:54 bsd-14 kernel: = actor.state=3Dd > Jul 10 10:38:54 bsd-14 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:54 bsd-14 kernel: partner.state=3D5 > Jul 10 10:38:54 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:54 bsd-14 kernel: ql0: old pstate 5 > Jul 10 10:38:54 bsd-14 kernel: ql0: new pstate = d > Jul 10 10:38:55 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], pending 1 -> 0 > Jul 10 10:38:55 bsd-14 kernel: ql1: collecting disabled > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 1 -> 2 > Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3 > Jul 10 10:38:55 bsd-14 kernel: ql1: enable distributing on aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], nports 0 -> 1 > Jul 10 10:38:55 bsd-14 kernel: lacp_select_active_aggregator: > Jul 10 10:38:55 bsd-14 kernel: = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:55 bsd-14 kernel: active aggregator changed > Jul 10 10:38:55 bsd-14 kernel: old (none) > Jul 10 10:38:55 bsd-14 kernel: new = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)] > Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports > Jul 10 10:38:55 bsd-14 kernel: lacp_suppress_distributing > Jul 10 10:38:55 bsd-14 kernel: ql1: marker transmit, port=3D13, = sys=3D00:0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: ql0: marker transmit, port=3D12, = sys=3D00:0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4 > Jul 10 10:38:55 bsd-14 kernel: ql1: lacpdu transmit > Jul 10 10:38:55 bsd-14 kernel: ql0: marker response, port=3D12, = sys=3D00:0e:1e:08:05:20, id=3D1 > Jul 10 10:38:55 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000D) > Jul 10 10:38:55 bsd-14 kernel: = actor.state=3D7d > Jul 10 10:38:55 bsd-14 kernel: = partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,0000) > Jul 10 10:38:55 bsd-14 kernel: = partner.state=3D3c > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000= ,0000)], pending 1 -> 0 > Jul 10 10:38:55 bsd-14 kernel: ql0: collecting disabled > Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 1 -> 2 > Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled > Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu transmit > Jul 10 10:38:55 bsd-14 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:55 bsd-14 kernel: = actor.state=3D1d > Jul 10 10:38:55 bsd-14 kernel: = partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:55 bsd-14 kernel: = partner.state=3Dd > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu receive > Jul 10 10:38:55 bsd-14 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 10:38:55 bsd-14 kernel: = actor.state=3D3d > Jul 10 10:38:55 bsd-14 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 10:38:55 bsd-14 kernel: = partner.state=3D1d > Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 > Jul 10 10:38:55 bsd-14 kernel: ql0: old pstate = d > Jul 10 10:38:55 bsd-14 kernel: ql0: new pstate = 3d > Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000= ,0000)], nports 0 -> 1 > Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregator: > Jul 10 10:38:56 bsd-14 kernel: = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:56 bsd-14 kernel: = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 10:38:56 bsd-14 kernel: active aggregator changed > Jul 10 10:38:56 bsd-14 kernel: old = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)] > Jul 10 10:38:56 bsd-14 kernel: new = [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000= ,0000)] > Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports > Jul 10 10:38:56 bsd-14 kernel: lacp_suppress_distributing > Jul 10 10:38:56 bsd-14 kernel: ql1: marker transmit, port=3D13, = sys=3D00:0e:1e:08:05:20, id=3D2 > Jul 10 10:38:56 bsd-14 kernel: ql0: marker transmit, port=3D12, = sys=3D00:0e:1e:08:05:20, id=3D2 > Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4 > Jul 10 10:38:56 bsd-14 kernel: ql0: marker response, port=3D12, = sys=3D00:0e:1e:08:05:20, id=3D2 > Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire > ^C >=20 >=20 >=20 >=20 > System 2 logs : >=20 > Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached > Jul 10 02:38:24 bsd-15 kernel: lacp_attach[740] : lacp_defined > Jul 10 02:38:24 bsd-15 kernel: lagg0: link state changed to UP > Jul 10 02:38:24 bsd-15 kernel: ql0: media changed 0x0 -> 0x100033, = ether =3D 1, fdx =3D 1, link =3D 1 > Jul 10 02:38:24 bsd-15 kernel: ql0: -> UNSELECTED > Jul 10 02:38:24 bsd-15 kernel: ql1: media changed 0x0 -> 0x100033, = ether =3D 1, fdx =3D 1, link =3D 1 > Jul 10 02:38:24 bsd-15 kernel: ql1: -> UNSELECTED > Jul 10 02:38:24 bsd-15 kernel: lacp_select_tx_port: no active = aggregator > Jul 10 02:38:25 bsd-15 kernel: ql1: port = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created > Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1 > Jul 10 02:38:25 bsd-15 kernel: ql0: port = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created > Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:26 bsd-15 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:26 bsd-15 kernel: = actor.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: = partner=3D(0000,00-00-00-00-00-00,0000,0000,0000) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_update_ntt: assert ntt > Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2 > Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate = 85 > Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:26 bsd-15 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:26 bsd-15 kernel: = partner.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000F) > Jul 10 02:38:26 bsd-15 kernel: = actor.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: = partner=3D(0000,00-00-00-00-00-00,0000,0000,0000) > Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:26 bsd-15 kernel: ql0: collecting disabled > Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213, = 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 >=20 > Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0 > Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:26 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:26 bsd-15 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:26 bsd-15 kernel: = partner.state=3D85 > Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:27 bsd-15 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:27 bsd-15 kernel: actor.state=3D5 > Jul 10 02:38:27 bsd-15 kernel: = partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:27 bsd-15 kernel: partner.state=3D5 > Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: old pstate = 85 > Jul 10 02:38:27 bsd-15 kernel: ql0: new pstate 5 > Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> = DEFAULTED > Jul 10 02:38:27 bsd-15 kernel: ql1: partner timeout changed > Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-04-2C-F0, = 0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 > Jul 10 02:38:27 bsd-15 kernel: ql1: collecting disabled > Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_delref: = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213, = 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 >=20 > Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0 > Jul 10 02:38:27 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:27 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000F) > Jul 10 02:38:27 bsd-15 kernel: = actor.state=3D45 > Jul 10 02:38:27 bsd-15 kernel: = partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,0000) > Jul 10 02:38:27 bsd-15 kernel: = partner.state=3D3c > Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:27 bsd-15 kernel: ql0: port = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(8000,00-0E-1E-08-05-20,0= 1D3,8000,000C)] > Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created > Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,0= 1D3,0000,0000)] > Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1 > Jul 10 02:38:28 bsd-15 kernel: ql1: port = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(FFFF,00-00-00-00-00-00,0= 000,FFFF,0000)] > Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created > Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator = lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0= 000,0000,0000)] > Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1 > Jul 10 02:38:29 bsd-15 kernel: ql0: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-04-2C-F0, = 0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], pending 1 -> 0 > Jul 10 02:38:29 bsd-15 kernel: ql0: collecting disabled > Jul 10 02:38:29 bsd-15 kernel: ql0: mux_state 1 -> 2 > Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:29 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:29 bsd-15 kernel: = actor.state=3Dd > Jul 10 02:38:29 bsd-15 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:29 bsd-15 kernel: partner.state=3D5 > Jul 10 02:38:29 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu receive > Jul 10 02:38:30 bsd-15 kernel: = actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:30 bsd-15 kernel: = actor.state=3D1d > Jul 10 02:38:30 bsd-15 kernel: = partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:30 bsd-15 kernel: = partner.state=3Dd > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: old pstate 5 > Jul 10 02:38:30 bsd-15 kernel: ql0: new pstate = 1d > Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator = [(8000,00-0E-1E-04-2C-F0, = 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 > Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2 > Jul 10 02:38:30 bsd-15 kernel: ql1: collecting enabled > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 2 -> 3 > Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregator = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], nports 0 -> 1 > Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: > Jul 10 02:38:30 bsd-15 kernel: = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: active aggregator changed > Jul 10 02:38:30 bsd-15 kernel: old (none) > Jul 10 02:38:30 bsd-15 kernel: new = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00 = 00-00-00,0000,0000,0000)] > Jul 10 02:38:30 bsd-15 kernel: Set table 1 with 1 ports > Jul 10 02:38:30 bsd-15 kernel: lacp_suppress_distributing > Jul 10 02:38:30 bsd-15 kernel: ql1: marker transmit, port=3D15, = sys=3D00:0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: ql0: marker transmit, port=3D14, = sys=3D00:0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4 > Jul 10 02:38:30 bsd-15 kernel: ql1: lacpdu transmit > Jul 10 02:38:30 bsd-15 kernel: ql0: marker response, port=3D14, = sys=3D00:0e:1e:04:2c:f0, id=3D1 > Jul 10 02:38:30 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000F) > Jul 10 02:38:30 bsd-15 kernel: = actor.state=3D7d > Jul 10 02:38:30 bsd-15 kernel: = partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,0000) > Jul 10 02:38:30 bsd-15 kernel: = partner.state=3D3c > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:30 bsd-15 kernel: ql0: collecting enabled > Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> 3 > Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000= ,0000)], nports 0 -> 1 > Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: > Jul 10 02:38:30 bsd-15 kernel: = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)], speed=3D10000000000, nports=3D1 > Jul 10 02:38:30 bsd-15 kernel: active aggregator not changed > Jul 10 02:38:30 bsd-15 kernel: new = [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000= ,0000)] > Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4 > Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu transmit > Jul 10 02:38:30 bsd-15 kernel: = actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E) > Jul 10 02:38:30 bsd-15 kernel: = actor.state=3D3d > Jul 10 02:38:30 bsd-15 kernel: = partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,000C) > Jul 10 02:38:30 bsd-15 kernel: = partner.state=3D1d > Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 > Jul 10 02:38:33 bsd-15 kernel: lacp_transit_expire > ^C >=20 > Let me know if you need more info. >=20 > Thanks > Adarsh >=20 >=20 > -----Original Message----- > From: owner-freebsd-net@freebsd.org = [mailto:owner-freebsd-net@freebsd.org] On Behalf Of Adarsh Joshi > Sent: Tuesday, July 10, 2012 10:08 AM > To: Andrew Boyer > Cc: freebsd-net@freebsd.org > Subject: RE: lacp lagg port flags do not show correctly resulting in = poor traffic distribution/performance >=20 > Andrew, >=20 > Thanks for the reply. >=20 > The reason for my suspicion on the portflags is thus (extracted from = the ifconfig output in my previous mail): >=20 > System 1: > Laggport: ql1 flags =3D 18 state =3D 7D > Laggport: ql0 flags =3D 1c state =3D 3D >=20 > System 2: > Laggport: ql1 flags =3D 1c state =3D 7D > Laggport: ql0 flags =3D 18 state =3D 3D >=20 > I should have explained my setup to you before. Here it is. > Both the ql0 interfaces of the 2 systems are connected using a single = cable and ql1 interfaces of the 2 systems are connected using a single = cable. >=20 > System 1 System 2 > ql0 <=3D=3D=3D=3D=3D=3D=3D> ql0 > ql1 <=3D=3D=3D=3D=3D=3D=3D> ql1 >=20 > With this setup, I don't think it is possible for ports ql0 to talk to = their partners (each other) and ql1 ports not getting a response from = their partner and still get the lagg configuration I have posted. >=20 > I thought the portflags are dependent on the LACP state. But I see = different flags for the same LACP state (For the state 7D, ql1 on system = 1 shows flags =3D 18 and ql1 on system 2 shows flags =3D 1c). >=20 > Or is my understanding totally wrong? >=20 > I will send the LACP_DEBUG logs within the hour. >=20 > Thanks > Adarsh >=20 > From: Andrew Boyer [mailto:aboyer@averesystems.com] > Sent: Tuesday, July 10, 2012 5:57 AM > To: Adarsh Joshi > Cc: freebsd-net@freebsd.org > Subject: Re: lacp lagg port flags do not show correctly resulting in = poor traffic distribution/performance >=20 >=20 > On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote: >=20 >=20 > Hi, >=20 > I am trying to configure lacp lagg interfaces with 2 systems connected = back to back as follows: >=20 > Ifconfig lagg0 create > Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 = netmask 255.255.255.0 >=20 > Sometimes, the lag interface comes up correctly but sometimes the = laggport flags do not show properly. Instead of = 1c, it shows values of 18. I have seen = similar issues reported on various forums with no solution. > Looking at the lagg driver code and reading the standard, I thought = the laggport flags ( defined in if_lagg.h) are based on the = LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v = output does not make any sense to me. >=20 > My concern is that when all the interfaces show flags as 1c, the = traffic is distributed across both the interfaces uniformly and I get = aggregated throughput. If not, the traffic flows only on 1 interface. >=20 > Is this a bug? How do I solve this? Or am I doing something wrong? >=20 > I am using Free-BSD 9.0 release. >=20 > System 1: > # ifconfig -v lagg0 > lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c = state=3D3D > [(8000,00-0E-1E-08-05-20,0213,8000,000E), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] >=20 > System 2: >=20 > # ifconfig -v lagg0 > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c = state=3D7D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,0213,8000,000E)] >=20 >=20 > thanks > Adarsh >=20 > I don't think you have a port flags problem per se; the flags are = correctly displaying the state of the lagg. Your problem is that your = systems aren't negotiating the correct lagg configuration. Each tuple = after the laggport represents the [(actor state),(partner state)]. = Ports ql0 have been able to talk to their partners (each other). = Neither ql1 port has seen a response from a partner, though. >=20 > You could try restarting the state machine on one box with 'ifconfig = lagg0 laggproto lacp'. To see the negotiation you'll need to rebuild = your kernel with '#define LACP_DEBUG 1' added to the top of = sys/net/ieee802.3ad_lacp.c. Or upgrade to a newer stable snapshot that = has the net.lacp_debug sysctl and turn it on. >=20 > Or just turn off LACP. What does it get you in this configuration? >=20 > Hope this helps, > Andrew >=20 > -------------------------------------------------- > Andrew Boyer = aboyer@averesystems.com >=20 >=20 >=20 >=20 >=20 > ________________________________ > This message and any attached documents contain information from = QLogic Corporation or its wholly-owned subsidiaries that may be = confidential. If you are not the intended recipient, you may not read, = copy, distribute, or use this information. If you have received this = transmission in error, please notify the sender immediately by reply = e-mail and then delete this message. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 >=20 > This message and any attached documents contain information from = QLogic Corporation or its wholly-owned subsidiaries that may be = confidential. If you are not the intended recipient, you may not read, = copy, distribute, or use this information. If you have received this = transmission in error, please notify the sender immediately by reply = e-mail and then delete this message. >=20 -------------------------------------------------- Andrew Boyer aboyer@averesystems.com