Date: Tue, 10 Jul 2012 03:10:11 -0400 From: Jason Hellenthal <jhellenthal@dataix.net> To: Adarsh Joshi <adarsh.joshi@qlogic.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Message-ID: <20120710071011.GB91639@DataIX.net> In-Reply-To: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--huq684BweRXVnRxX
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Jul 09, 2012 at 05:38:24PM -0700, Adarsh Joshi wrote:
> Hi,
>=20
> I am trying to configure lacp lagg interfaces with 2 systems connected ba=
ck to back as follows:
>=20
> Ifconfig lagg0 create
> Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 net=
mask 255.255.255.0
>=20
> Sometimes, the lag interface comes up correctly but sometimes the laggpor=
t flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>=
, it shows values of 18. I have seen similar issues reported on various for=
ums with no solution.
> Looking at the lagg driver code and reading the standard, I thought the l=
aggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in f=
ile ieee8023ad_lacp.h. But the following ifconfig -v output does not make a=
ny sense to me.
>=20
> My concern is that when all the interfaces show flags as 1c, the traffic =
is distributed across both the interfaces uniformly and I get aggregated th=
roughput. If not, the traffic flows only on 1 interface.
>=20
> Is this a bug? How do I solve this? Or am I doing something wrong?
>=20
> I am using Free-BSD 9.0 release.
>=20
> System 1:
> # ifconfig -v lagg0
> lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu =
1500
> options=3D13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO=
4>
> ether 00:0e:1e:08:05:20
> inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: lagg
> laggproto lacp
> lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000),
> (8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
> laggport: ql1 flags=3D18<COLLECTING,DISTRIBUTING> state=3D7D
> [(8000,00-0E-1E-08-05-20,0213,8000,000F),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D=
3D
> [(8000,00-0E-1E-08-05-20,0213,8000,000E),
> (8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
>=20
> System 2:
>=20
> # ifconfig -v lagg0
> lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu =
1500
> options=3D13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO=
4>
> ether 00:0e:1e:04:2c:f0
> inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: lagg
> laggproto lacp
> lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
> (FFFF,00-00-00-00-00-00,0000,0000,0000)]
> laggport: ql1 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D=
7D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=3D18<COLLECTING,DISTRIBUTING> state=3D3D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
> (8000,00-0E-1E-08-05-20,0213,8000,000E)]
>=20
>=20
Just for reference ... (stable/8 @ r238264)
lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15=
00
options=3D80048<VLAN_MTU,POLLING,LINKSTATE>
ether 00:0c:41:21:1d:b5
inet 192.168.XX.X netmask 0xffffff00 broadcast 192.168.XX.XXX
media: Ethernet autoselect
status: active
groups: lagg=20
laggproto lacp lagghash l2,l3,l4
lag id: [(8000,00-0C-41-21-1D-B5,00E6,0000,0000),
(FFFF,00-00-00-00-00-00,0000,0000,0000)]
laggport: dc1 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0C-41-21-1D-B5,00E6,8000,0002),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: dc0 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D7D
[(8000,00-0C-41-21-1D-B5,00E6,8000,0001),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
They have had flags =3D 1c for quite some time and state =3D 7D
And just to show the variation ...
dc0:
yesterday 8.53 MiB / 2.61 MiB / 11.14 MiB
today 693 KiB / 156 KiB / 849 KiB
dc1:
yesterday 19.00 MiB / 1.79 MiB / 20.78 MiB
today 496 KiB / 103 KiB / 599 KiB
lagg0:
yesterday 27.53 MiB / 3.71 MiB / 31.24 MiB
today 1.16 MiB / 172 KiB / 1.33 MiB
I believe (know) there has been some changes in the LAgg code in
stable/9 and stable/8 recently so you may want to check into that.
Given this is LAgg and LACP you will see some variation regardless but I
recall a point that it seemed like one interface was being favored over
the other quite repeatedly or obsessively that had me second guessing
whether it was doing the right thing.
LACP in Cisco is quite different than how we treat it here in FreeBSD as
it tends to use the interfaces quite evenly all the time so that also
has me second guessing whether the right thing is happening here. ( in
PAgP and LACP modes ).
These tunables (sysctl)'s may also lead you in direction you want to
look in.
net.link.lagg.default_use_flowid: 1
net.link.lagg.failover_rx_all: 0
net.link.lagg.0.use_flowid: 1
-rxcsum & -txcsum for lo0 and ql0 ql1 might be of benefit too though I
only turn them off on lo0.
Good luck
--=20
- (2^(N-1))
--huq684BweRXVnRxX
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
iQEcBAEBAgAGBQJP+9VSAAoJEBSh2Dr1DU7WHzEH/2PBC6Iz1BOoL8PPAEfzb/Mz
llMEDATV45W+JVWyAeWaG2GH+o7KaQPfp/Z5qTH/dcgqA3r7fiCqrRZpd61DhbGF
GlzryhUJKY9V/FDJXCSaevyhFdjtosvZQ8qEchA18FsDGEX2wA/4dXv/BZs1YQQx
yhXgVd3gIjrqa7bt0sqOqWdngizVHaEt7oDyFvl2I4kpTbd1TETUOCXT2IMEGNPQ
uVghGx3sQSiAePXV1IjizMcW0QGOTX+bUmJLlvQXSOPjxkXIRYnn2xSIh8WGMZT+
iK2UiuvzY8mIPBTVw+yvst954pxrscWPNXPhH8nqAudp3SHpSlWPszEJwxN3Ypo=
=pvae
-----END PGP SIGNATURE-----
--huq684BweRXVnRxX--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120710071011.GB91639>