Date: Mon, 16 Mar 2015 10:50:04 +0300 From: Konstantin Kulikov <k.kulikov2@gmail.com> To: Vitalii Duk <mlevel.ars@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Unbalanced LACP link Message-ID: <CAD%2BeXGRB4BD7xRL14KkLo-7uMWWEwZx97Y01g4dywR3-KGmTuA@mail.gmail.com> In-Reply-To: <CAH6gFOnpQ9LrpuWOe6w2nQ42Yz%2BhKSoy4WcBOTSEOkMMxnV8sA@mail.gmail.com> References: <CAH6gFOnpQ9LrpuWOe6w2nQ42Yz%2BhKSoy4WcBOTSEOkMMxnV8sA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, Vitalii. Make sure following sysctl are set to zero: sysctl net.link.lagg.default_use_flowid=0 sysctl net.link.lagg.0.use_flowid=0 sysctl net.link.lagg.1.use_flowid=0 Then adjust lagghash depending on your configuration. For example if you freebsd machine acts as router with single, default, route, you should set lagghash to l3, since for l2 both src and dst MACs will always be same and you'd get unbalanced load. On Sun, Mar 15, 2015 at 1:38 PM, Vitalii Duk <mlevel.ars@gmail.com> wrote: > Hello, guys. > I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link > DGS-3610-26G switch. This link worked fine last two years on FreeBSD > 9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic > started to go mostly through one interface. > > lagg0 - input interface (no IP-address, only VLANs). > lagg1 - output interface. > > [router]# ifconfig lagg0 > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> > ether 90:e2:ba:02:d0:ae > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2 > laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > [#router]# ifconfig lagg1 > lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> > ether 00:1b:21:93:52:d4 > inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2 > laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > [router]# netstat -hw1 -I igb0 > input igb0 output > packets errs idrops bytes packets errs bytes colls > 27K 0 0 12M 5 0 350 0 > 26K 0 0 12M 7 0 693 0 > 29K 0 0 12M 8 0 838 0 > 29K 0 0 11M 10 0 1.0K 0 > 27K 0 0 10M 7 0 770 0 > > As you can see most traffic is going through igb1 (lagg0) and igb3 (lagg1) > interfaces. In peak times there are more that 1 Gbps of traffic mostly > through one 1GbE interface, and it cause an errors. So this link is > assymetric. > > [router]# netstat -hw1 -I igb1 > input igb1 output > packets errs idrops bytes packets errs bytes colls > 22K 0 0 8.5M 63K 0 69M 0 > 20K 0 0 7.7M 58K 0 62M 0 > 20K 0 0 7.3M 57K 0 61M 0 > 19K 0 0 6.8M 56K 0 61M 0 > 18K 0 0 6.5M 57K 0 62M 0 > > [router]# netstat -hw1 -I igb2 > input igb2 output > packets errs idrops bytes packets errs bytes colls > 40K 0 0 49M 1 0 93 0 > 41K 0 0 51M 1 0 88 0 > 38K 0 0 45M 1 0 88 0 > 33K 0 0 39M 2 0 174 0 > 31K 0 0 35M 1 0 89 0 > > [router]# netstat -hw1 -I igb3 > input igb3 output > packets errs idrops bytes packets errs bytes colls > 32K 0 0 34M 44K 0 18M 0 > 29K 0 0 30M 45K 0 18M 0 > 32K 0 0 34M 48K 0 17M 0 > 28K 0 0 29M 42K 0 15M 0 > 25K 0 0 26M 40K 0 16M 0 > > [router]# netstat -hw1 -I lagg0 > input lagg0 output > packets errs idrops bytes packets errs bytes colls > 45K 0 0 17M 62K 0 69M 0 > 46K 0 0 17M 66K 0 75M 0 > 43K 0 0 17M 62K 0 69M 0 > 50K 0 0 19M 71K 1 82M 0 > 46K 0 0 18M 66K 0 75M 0 > > [router]# netstat -hw1 -I lagg1 > input lagg1 output > packets errs idrops bytes packets errs bytes colls > 70K 0 0 79M 49K 0 18M 0 > 71K 0 0 79M 49K 0 19M 0 > 79K 0 0 91M 54K 0 20M 0 > 65K 0 0 72M 45K 0 17M 0 > 65K 0 0 72M 45K 0 17M 0 > > All igb interfaces are the same: > igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = '82576 Gigabit Network Connection' > class = network > subclass = ethernet > > I've tried to disable/enable strict_mode, put switch into active/passive > mode and trying to set duplex/speed manyally. Nothing helps. > Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because > switch supports only l2 load-balancing, but it also doesn't helps. > > net.link.lagg.lacp.debug: 0 > net.link.lagg.0.lacp.lacp_strict_mode: 0 > net.link.lagg.0.lacp.debug.rx_test: 0 > net.link.lagg.0.lacp.debug.tx_test: 0 > net.link.lagg.1.lacp.lacp_strict_mode: 0 > net.link.lagg.1.lacp.debug.rx_test: 0 > net.link.lagg.1.lacp.debug.tx_test: 0 > > I've read that there were some problems with LACP on FreeBSD 10. > > Information from the switch: > > # show lacp summary > Aggregate port 6: > > Local information: > LACP port Oper Port Port > Port Flags State Priority Key Number State > ---------------------------------------------------------------------- > Gi0/18 SP bndl 4096 0x6 0x12 0x3c > Gi0/22 SP bndl 4096 0x6 0x16 0x3c > > Partner information: > LACP port Oper Port Port > Port Flags Priority Dev ID Key Number State > --------------------------------------------------------------------- > Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d > Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d > > Aggregate port 7: > > Local information: > LACP port Oper Port Port > Port Flags State Priority Key Number State > ---------------------------------------------------------------------- > Gi0/19 SP bndl 4096 0x7 0x13 0x3c > Gi0/21 SP bndl 4096 0x7 0x15 0x3c > > Partner information: > LACP port Oper Port Port > Port Flags Priority Dev ID Key Number State > --------------------------------------------------------------------- > Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d > Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD%2BeXGRB4BD7xRL14KkLo-7uMWWEwZx97Y01g4dywR3-KGmTuA>