Date: Wed, 11 Jul 2012 10:59:42 -0700 From: Adarsh Joshi <adarsh.joshi@qlogic.com> To: Andrew Boyer <aboyer@averesystems.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Message-ID: <5E4F49720D0BAD499EE1F01232234BA8774378F76D@AVEXMB1.qlogic.org> In-Reply-To: <5E4F49720D0BAD499EE1F01232234BA877435B2F66@AVEXMB1.qlogic.org> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <4C5FC147-2F49-4AE0-ADF3-C5381DE6580F@averesystems.com> <5E4F49720D0BAD499EE1F01232234BA877435B2F2E@AVEXMB1.qlogic.org> <5E4F49720D0BAD499EE1F01232234BA877435B2F66@AVEXMB1.qlogic.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I tried to configure lagg with the latest source from SVN head and it did n=
ot help.
Regards
Adarsh
-----Original Message-----
From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] =
On Behalf Of Adarsh Joshi
Sent: Tuesday, July 10, 2012 10:53 AM
To: Andrew Boyer
Cc: freebsd-net@freebsd.org
Subject: RE: lacp lagg port flags do not show correctly resulting in poor t=
raffic distribution/performance
Andrew,
Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c,
after typing
Ifconfig lagg0 create
ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netma=
sk 255.255.255.0
I compiled it as a standalone driver by the way.
System 1:
# ifconfig -v lagg0
lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15=
00
options=3D13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
ether 00:0e:1e:08:05:20
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp
lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),
(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
laggport: ql1 flags=3D18<COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0E-1E-08-05-20,01D3,8000,000D),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D3D
[(8000,00-0E-1E-08-05-20,01D3,8000,000C),
(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
System 2:
# ifconfig -v lagg0
lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15=
00
options=3D13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
ether 00:0e:1e:04:2c:f0
inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp
lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
(FFFF,00-00-00-00-00-00,0000,0000,0000)]
laggport: ql1 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=3D18<COLLECTING,DISTRIBUTING> state=3D3D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
(8000,00-0E-1E-08-05-20,01D3,8000,000C)]
System 1 logs :
Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached Jul 10 10:3=
8:49 bsd-14 kernel: lacp_attach[740] : lacp_defined Jul 10 10:38:49 bsd-14 =
kernel: lagg0: link state changed to UP Jul 10 10:38:49 bsd-14 kernel: ql0:=
media changed 0x0 -> 0x100033, ether =3D 1, fdx =3D 1, link =3D 1 Jul 10 1=
0:38:49 bsd-14 kernel: ql0: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: ql=
1: media changed 0x0 -> 0x100033, ether =3D 1, fdx =3D 1, link =3D 1 Jul 10=
10:38:49 bsd-14 kernel: ql1: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: =
lacp_select_tx_port: no active aggregator Jul 10 10:38:50 bsd-14 kernel: ql=
1: port lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(0000,00-00-00-00-=
00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:50 bsd-=
14 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:50 bsd-14=
kernel: ql0: port lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(0000,0=
0-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:50 bsd-=
14 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:51 bsd-14=
kernel: ql1: lacpdu transmit Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,=
00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:51 bsd-14 kernel: actor.state=3D85<ACTIVITY,AGGREGATION,EXPIRE=
D>
Jul 10 10:38:51 bsd-14 kernel: partner=3D(0000,00-00-00-00-00-00,0000,0000,=
0000)
Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2<TIMEOUT> Jul 10 10:38:51 b=
sd-14 kernel: maxdelay=3D0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu trans=
mit Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,800=
0,000C)
Jul 10 10:38:51 bsd-14 kernel: actor.state=3D85<ACTIVITY,AGGREGATION,EXPIRE=
D>
Jul 10 10:38:51 bsd-14 kernel: partner=3D(0000,00-00-00-00-00-00,0000,0000,=
0000)
Jul 10 10:38:51 bsd-14 kernel: partner.state=3D2<TIMEOUT> Jul 10 10:38:51 b=
sd-14 kernel: maxdelay=3D0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu recei=
ve Jul 10 10:38:51 bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000=
,000E)
Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 10:38:51 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 10:38:51 bsd-14 kernel: partner.state=3D85<ACTIVITY,AGGREGATION,EXPI=
RED>
Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:51 bsd-14 kernel: =
ql0: old pstate 2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5=
<ACTIVITY,AGGREGATION> Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout =
changed Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:51 =
bsd-14 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:51 bsd-14 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 10:38:51 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 10:38:51 bsd-14 kernel: partner.state=3D85<ACTIVITY,AGGREGATION,EXPI=
RED>
Jul 10 10:38:51 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:52 bsd-14 kernel: =
ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 10:38:52 bsd-14 kernel: =
ql1: partner timeout changed Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mu=
x_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00=
-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: q=
l1: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delr=
ef: lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-0=
0,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: mux_s=
tate 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:3=
8:52 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:52 bsd-14 kernel: actor.state=3D45<ACTIVITY,AGGREGATION,DEFAUL=
TED>
Jul 10 10:38:52 bsd-14 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,=
0000)
Jul 10 10:38:52 bsd-14 kernel: partner.state=3D3c<AGGREGATION,SYNC,COLLECTI=
NG,DISTRIBUTING>
Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:52 bsd-14 kernel: =
ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 b=
sd-14 kernel: ql0: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_=
aggregator_delref: lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,0=
0-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 ker=
nel: ql0: mux_state 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu trans=
mit Jul 10 10:38:52 bsd-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,800=
0,000C)
Jul 10 10:38:52 bsd-14 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 10:38:52 bsd-14 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,=
000E)
Jul 10 10:38:52 bsd-14 kernel: partner.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 10:38:52 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:53 bsd-14 kernel: =
ql1: port lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(FFFF,00-00-00-0=
0-00-00,0000,FFFF,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:53 bsd-=
14 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:53 bsd-14=
kernel: ql0: port lagid=3D[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(8000,0=
0-0E-1E-04-2C-F0,0213,8000,000E)]
Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:53 bsd-=
14 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:54 bsd-14=
kernel: ql0: lacpdu receive Jul 10 10:38:54 bsd-14 kernel: actor=3D(8000,0=
0-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:54 bsd-14 kernel: actor.state=3Dd<ACTIVITY,AGGREGATION,SYNC>
Jul 10 10:38:54 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 10:38:54 bsd-14 kernel: partner.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 10:38:54 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:54 bsd-14 kernel: =
ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 10:38:54 bsd-14 kernel: ql0:=
new pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: ql1=
: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(F=
FFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 bsd-=
14 kernel: ql1: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql1: mux=
_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled Jul 10=
10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 kerne=
l: ql1: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,000=
0,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1 Jul 10 10:3=
8:55 bsd-14 kernel: lacp_select_active_aggregator:
Jul 10 10:38:55 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FF=
FF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 Jul =
10 10:38:55 bsd-14 kernel: active aggregator changed Jul 10 10:38:55 bsd-14=
kernel: old (none) Jul 10 10:38:55 bsd-14 kernel: new [(8000,00-0E-1E-08-0=
5-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports Jul 10 10:38:55 bsd=
-14 kernel: lacp_suppress_distributing Jul 10 10:38:55 bsd-14 kernel: ql1: =
marker transmit, port=3D13, sys=3D00:0e:1e:08:05:20, id=3D1 Jul 10 10:38:55=
bsd-14 kernel: ql0: marker transmit, port=3D12, sys=3D00:0e:1e:08:05:20, i=
d=3D1 Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4 Jul 10 10:38:55 =
bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: ql0: mar=
ker response, port=3D12, sys=3D00:0e:1e:08:05:20, id=3D1 Jul 10 10:38:55 bs=
d-14 kernel: actor=3D(8000,00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:55 bsd-14 kernel: actor.state=3D7d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING,DISTRIBUTING,DEFAULTED>
Jul 10 10:38:55 bsd-14 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,=
0000)
Jul 10 10:38:55 bsd-14 kernel: partner.state=3D3c<AGGREGATION,SYNC,COLLECTI=
NG,DISTRIBUTING>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:55 bsd-14 kernel: =
ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 b=
sd-14 kernel: ql0: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql0: =
mux_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled Jul=
10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 ke=
rnel: ql0: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: actor=3D(8000,00-=
0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:55 bsd-14 kernel: actor.state=3D1d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING>
Jul 10 10:38:55 bsd-14 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,=
000E)
Jul 10 10:38:55 bsd-14 kernel: partner.state=3Dd<ACTIVITY,AGGREGATION,SYNC>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:55 bsd-14 kernel: =
ql0: lacpdu receive Jul 10 10:38:55 bsd-14 kernel: actor=3D(8000,00-0E-1E-0=
4-2C-F0,0213,8000,000E)
Jul 10 10:38:55 bsd-14 kernel: actor.state=3D3d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING,DISTRIBUTING>
Jul 10 10:38:55 bsd-14 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 10:38:55 bsd-14 kernel: partner.state=3D1d<ACTIVITY,AGGREGATION,SYNC=
,COLLECTING>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=3D0 Jul 10 10:38:55 bsd-14 kernel: =
ql0: old pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel:=
ql0: new pstate 3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator [(800=
0,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)=
], nports 0 -> 1 Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregat=
or:
Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FF=
FF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 Jul =
10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,0=
0-0E-1E-04-2C-F0,0213,0000,0000)], speed=3D10000000000, nports=3D1 Jul 10 1=
0:38:56 bsd-14 kernel: active aggregator changed Jul 10 10:38:56 bsd-14 ker=
nel: old [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0=
000,0000,0000)]
Jul 10 10:38:56 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000)=
,(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports Jul 10 10:38:56 bsd=
-14 kernel: lacp_suppress_distributing Jul 10 10:38:56 bsd-14 kernel: ql1: =
marker transmit, port=3D13, sys=3D00:0e:1e:08:05:20, id=3D2 Jul 10 10:38:56=
bsd-14 kernel: ql0: marker transmit, port=3D12, sys=3D00:0e:1e:08:05:20, i=
d=3D2 Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4 Jul 10 10:38:56 =
bsd-14 kernel: ql0: marker response, port=3D12, sys=3D00:0e:1e:08:05:20, id=
=3D2 Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire ^C
System 2 logs :
Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached Jul 10 02:3=
8:24 bsd-15 kernel: lacp_attach[740] : lacp_defined Jul 10 02:38:24 bsd-15 =
kernel: lagg0: link state changed to UP Jul 10 02:38:24 bsd-15 kernel: ql0:=
media changed 0x0 -> 0x100033, ether =3D 1, fdx =3D 1, link =3D 1 Jul 10 0=
2:38:24 bsd-15 kernel: ql0: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: ql=
1: media changed 0x0 -> 0x100033, ether =3D 1, fdx =3D 1, link =3D 1 Jul 10=
02:38:24 bsd-15 kernel: ql1: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: =
lacp_select_tx_port: no active aggregator Jul 10 02:38:25 bsd-15 kernel: ql=
1: port lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(0000,00-00-00-00-=
00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:25 bsd-=
15 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000)=
,(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:25 bsd-15=
kernel: ql0: port lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(0000,0=
0-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:25 bsd-=
15 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000)=
,(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:26 bsd-15=
kernel: ql0: lacpdu receive Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,0=
0-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:26 bsd-15 kernel: actor.state=3D85<ACTIVITY,AGGREGATION,EXPIRE=
D>
Jul 10 02:38:26 bsd-15 kernel: partner=3D(0000,00-00-00-00-00-00,0000,0000,=
0000)
Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2<TIMEOUT> Jul 10 02:38:26 b=
sd-15 kernel: maxdelay=3D0 Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_u=
pdate_ntt: assert ntt Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2<TIME=
OUT> Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate 85<ACTIVITY,AGGREGATION=
,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed Jul 1=
0 02:38:26 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kerne=
l: actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 02:38:26 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 02:38:26 bsd-15 kernel: partner.state=3D85<ACTIVITY,AGGREGATION,EXPI=
RED>
Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:26 bsd-15 kernel: =
ql1: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,00-0E-1E-=
04-2C-F0,0213,8000,000F)
Jul 10 02:38:26 bsd-15 kernel: actor.state=3D85<ACTIVITY,AGGREGATION,EXPIRE=
D>
Jul 10 02:38:26 bsd-15 kernel: partner=3D(0000,00-00-00-00-00-00,0000,0000,=
0000)
Jul 10 02:38:26 bsd-15 kernel: partner.state=3D2<TIMEOUT> Jul 10 02:38:26 b=
sd-15 kernel: maxdelay=3D0 Jul 10 02:38:26 bsd-15 kernel: ql0: collecting d=
isabled Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: lagid=3D[(80=
00,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,000=
0)], refcnt 1 -> 0
Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0 Jul 10 02:38:26 bsd-15=
kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=3D(8000,=
00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:26 bsd-15 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 02:38:26 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 02:38:26 bsd-15 kernel: partner.state=3D85<ACTIVITY,AGGREGATION,EXPI=
RED>
Jul 10 02:38:26 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:27 bsd-15 kernel: =
ql0: lacpdu receive Jul 10 02:38:27 bsd-15 kernel: actor=3D(8000,00-0E-1E-0=
8-05-20,01D3,8000,000C)
Jul 10 02:38:27 bsd-15 kernel: actor.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 02:38:27 bsd-15 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,=
000E)
Jul 10 02:38:27 bsd-15 kernel: partner.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:27 bsd-15 kernel: =
ql0: old pstate 85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:27 bsd-15 ker=
nel: ql0: new pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:27 bsd-15 kernel:=
ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 02:38:27 bsd-15 kernel:=
ql1: partner timeout changed Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_m=
ux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(0000,00-00-=
00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 02:38:27 bsd-15 kernel:=
ql1: collecting disabled Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_de=
lref: lagid=3D[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-0=
0-00,0000,0000,0000)], refcnt 1 -> 0
Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0 Jul 10 02:38:27 bsd-15=
kernel: ql1: lacpdu transmit Jul 10 02:38:27 bsd-15 kernel: actor=3D(8000,=
00-0E-1E-04-2C-F0,0213,8000,000F)
Jul 10 02:38:27 bsd-15 kernel: actor.state=3D45<ACTIVITY,AGGREGATION,DEFAUL=
TED>
Jul 10 02:38:27 bsd-15 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,=
0000)
Jul 10 02:38:27 bsd-15 kernel: partner.state=3D3c<AGGREGATION,SYNC,COLLECTI=
NG,DISTRIBUTING>
Jul 10 02:38:27 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:27 bsd-15 kernel: =
ql0: port lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(8000,00-0E-1E-0=
8-05-20,01D3,8000,000C)]
Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:27 bsd-=
15 kernel: ql0: aggregator lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000)=
,(8000,00-0E-1E-08-05-20,01D3,0000,0000)]
Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:28 bsd-15=
kernel: ql1: port lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(FFFF,0=
0-00-00-00-00-00,0000,FFFF,0000)]
Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:28 bsd-=
15 kernel: ql1: aggregator lagid=3D[(8000,00-0E-1E-04-2C-F0,0213,0000,0000)=
,(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:29 bsd-15=
kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,=
0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], pending 1 -> 0 Jul 10 =
02:38:29 bsd-15 kernel: ql0: collecting disabled Jul 10 02:38:29 bsd-15 ker=
nel: ql0: mux_state 1 -> 2 Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu trans=
mit Jul 10 02:38:29 bsd-15 kernel: actor=3D(8000,00-0E-1E-04-2C-F0,0213,800=
0,000E)
Jul 10 02:38:29 bsd-15 kernel: actor.state=3Dd<ACTIVITY,AGGREGATION,SYNC>
Jul 10 02:38:29 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 02:38:29 bsd-15 kernel: partner.state=3D5<ACTIVITY,AGGREGATION>
Jul 10 02:38:29 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:30 bsd-15 kernel: =
ql0: lacpdu receive Jul 10 02:38:30 bsd-15 kernel: actor=3D(8000,00-0E-1E-0=
8-05-20,01D3,8000,000C)
Jul 10 02:38:30 bsd-15 kernel: actor.state=3D1d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING>
Jul 10 02:38:30 bsd-15 kernel: partner=3D(8000,00-0E-1E-04-2C-F0,0213,8000,=
000E)
Jul 10 02:38:30 bsd-15 kernel: partner.state=3Dd<ACTIVITY,AGGREGATION,SYNC>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:30 bsd-15 kernel: =
ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:30 bsd-15 kernel: ql0:=
new pstate 1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00=
-0E-1E-04-2C-F0, 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], =
pending 1 -> 0 Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled Jul =
10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2 Jul 10 02:38:30 bsd-15 ker=
nel: ql1: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state =
2 -> 3 Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregato=
r [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,000=
0,0000)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_a=
ggregator:
Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FF=
FF,00-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 Jul =
10 02:38:30 bsd-15 kernel: active aggregator changed Jul 10 02:38:30 bsd-15=
kernel: old (none) Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2=
C-F0,0213,0000,0000),(FFFF,00-00-00 00-00-00,0000,0000,0000)] Jul 10 02:38:=
30 bsd-15 kernel: Set table 1 with 1 ports Jul 10 02:38:30 bsd-15 kernel: l=
acp_suppress_distributing Jul 10 02:38:30 bsd-15 kernel: ql1: marker transm=
it, port=3D15, sys=3D00:0e:1e:04:2c:f0, id=3D1 Jul 10 02:38:30 bsd-15 kerne=
l: ql0: marker transmit, port=3D14, sys=3D00:0e:1e:04:2c:f0, id=3D1 Jul 10 =
02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15 kernel=
: ql1: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: ql0: marker response,=
port=3D14, sys=3D00:0e:1e:04:2c:f0, id=3D1 Jul 10 02:38:30 bsd-15 kernel: =
actor=3D(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
Jul 10 02:38:30 bsd-15 kernel: actor.state=3D7d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING,DISTRIBUTING,DEFAULTED>
Jul 10 02:38:30 bsd-15 kernel: partner=3D(FFFF,00-00-00-00-00-00,0000,FFFF,=
0000)
Jul 10 02:38:30 bsd-15 kernel: partner.state=3D3c<AGGREGATION,SYNC,COLLECTI=
NG,DISTRIBUTING>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:30 bsd-15 kernel: =
ql0: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> =
3 Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator [(8=
000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,000=
0)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggreg=
ator:
Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(80=
00,00-0E-1E-08-05-20,01D3,0000,0000)], speed=3D10000000000, nports=3D1 Jul =
10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,0=
0-00-00-00-00-00,0000,0000,0000)], speed=3D10000000000, nports=3D1 Jul 10 0=
2:38:30 bsd-15 kernel: active aggregator not changed Jul 10 02:38:30 bsd-15=
kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-=
00,0000,0000,0000)]
Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15=
kernel: ql0: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: actor=3D(8000,=
00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:30 bsd-15 kernel: actor.state=3D3d<ACTIVITY,AGGREGATION,SYNC,C=
OLLECTING,DISTRIBUTING>
Jul 10 02:38:30 bsd-15 kernel: partner=3D(8000,00-0E-1E-08-05-20,01D3,8000,=
000C)
Jul 10 02:38:30 bsd-15 kernel: partner.state=3D1d<ACTIVITY,AGGREGATION,SYNC=
,COLLECTING>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=3D0 Jul 10 02:38:33 bsd-15 kernel: =
lacp_transit_expire ^C
Let me know if you need more info.
Thanks
Adarsh
-----Original Message-----
From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] =
On Behalf Of Adarsh Joshi
Sent: Tuesday, July 10, 2012 10:08 AM
To: Andrew Boyer
Cc: freebsd-net@freebsd.org
Subject: RE: lacp lagg port flags do not show correctly resulting in poor t=
raffic distribution/performance
Andrew,
Thanks for the reply.
The reason for my suspicion on the portflags is thus (extracted from the if=
config output in my previous mail):
System 1:
Laggport: ql1 flags =3D 18 state =3D 7D
Laggport: ql0 flags =3D 1c state =3D 3D
System 2:
Laggport: ql1 flags =3D 1c state =3D 7D
Laggport: ql0 flags =3D 18 state =3D 3D
I should have explained my setup to you before. Here it is.
Both the ql0 interfaces of the 2 systems are connected using a single cable=
and ql1 interfaces of the 2 systems are connected using a single cable.
System 1 System 2
ql0 <=3D=3D=3D=3D=3D=3D=3D> ql0
ql1 <=3D=3D=3D=3D=3D=3D=3D> ql1
With this setup, I don't think it is possible for ports ql0 to talk to thei=
r partners (each other) and ql1 ports not getting a response from their par=
tner and still get the lagg configuration I have posted.
I thought the portflags are dependent on the LACP state. But I see differen=
t flags for the same LACP state (For the state 7D, ql1 on system 1 shows fl=
ags =3D 18 and ql1 on system 2 shows flags =3D 1c).
Or is my understanding totally wrong?
I will send the LACP_DEBUG logs within the hour.
Thanks
Adarsh
From: Andrew Boyer [mailto:aboyer@averesystems.com]
Sent: Tuesday, July 10, 2012 5:57 AM
To: Adarsh Joshi
Cc: freebsd-net@freebsd.org
Subject: Re: lacp lagg port flags do not show correctly resulting in poor t=
raffic distribution/performance
On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote:
Hi,
I am trying to configure lacp lagg interfaces with 2 systems connected back=
to back as follows:
Ifconfig lagg0 create
Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netma=
sk 255.255.255.0
Sometimes, the lag interface comes up correctly but sometimes the laggport =
flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>, =
it shows values of 18. I have seen similar issues reported on various forum=
s with no solution.
Looking at the lagg driver code and reading the standard, I thought the lag=
gport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in fil=
e ieee8023ad_lacp.h. But the following ifconfig -v output does not make any=
sense to me.
My concern is that when all the interfaces show flags as 1c, the traffic is=
distributed across both the interfaces uniformly and I get aggregated thro=
ughput. If not, the traffic flows only on 1 interface.
Is this a bug? How do I solve this? Or am I doing something wrong?
I am using Free-BSD 9.0 release.
System 1:
# ifconfig -v lagg0
lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000),
(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
laggport: ql1 flags=3D18<COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0E-1E-08-05-20,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D3D
[(8000,00-0E-1E-08-05-20,0213,8000,000E),
(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
System 2:
# ifconfig -v lagg0
lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
(FFFF,00-00-00-00-00-00,0000,0000,0000)]
laggport: ql1 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=3D18<COLLECTING,DISTRIBUTING> state=3D3D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
(8000,00-0E-1E-08-05-20,0213,8000,000E)]
thanks
Adarsh
I don't think you have a port flags problem per se; the flags are correctly=
displaying the state of the lagg. Your problem is that your systems aren'=
t negotiating the correct lagg configuration. Each tuple after the laggpor=
t represents the [(actor state),(partner state)]. Ports ql0 have been able=
to talk to their partners (each other). Neither ql1 port has seen a respo=
nse from a partner, though.
You could try restarting the state machine on one box with 'ifconfig lagg0 =
laggproto lacp'. To see the negotiation you'll need to rebuild your kernel=
with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp.c=
. Or upgrade to a newer stable snapshot that has the net.lacp_debug sysctl=
and turn it on.
Or just turn off LACP. What does it get you in this configuration?
Hope this helps,
Andrew
--------------------------------------------------
Andrew Boyer aboyer@averesystems.com<mailto:aboyer@averesystems.com>
________________________________
This message and any attached documents contain information from QLogic Cor=
poration or its wholly-owned subsidiaries that may be confidential. If you =
are not the intended recipient, you may not read, copy, distribute, or use =
this information. If you have received this transmission in error, please n=
otify the sender immediately by reply e-mail and then delete this message.
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
This message and any attached documents contain information from QLogic Cor=
poration or its wholly-owned subsidiaries that may be confidential. If you =
are not the intended recipient, you may not read, copy, distribute, or use =
this information. If you have received this transmission in error, please n=
otify the sender immediately by reply e-mail and then delete this message.
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
This message and any attached documents contain information from QLogic Cor=
poration or its wholly-owned subsidiaries that may be confidential. If you =
are not the intended recipient, you may not read, copy, distribute, or use =
this information. If you have received this transmission in error, please n=
otify the sender immediately by reply e-mail and then delete this message.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5E4F49720D0BAD499EE1F01232234BA8774378F76D>