From owner-freebsd-net@FreeBSD.ORG Sun Mar 15 10:38:44 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 691C9E for ; Sun, 15 Mar 2015 10:38:44 +0000 (UTC) Received: from mail-qc0-x236.google.com (mail-qc0-x236.google.com [IPv6:2607:f8b0:400d:c01::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23532DAC for ; Sun, 15 Mar 2015 10:38:44 +0000 (UTC) Received: by qcbkw5 with SMTP id kw5so22172769qcb.2 for ; Sun, 15 Mar 2015 03:38:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=s7SLogciVE3FGA2UbJaJ336wPDYk9J+lqFR25JHYsow=; b=w1OGsX2k8rlys3bK3H0xKSCJRvUunkP0fMh+i+ithE0ribluGjiCKilFHkYb0JY4hA osSHkR/y6kCLVa2QovNwCw9KjvR8d6e0tbLklysUb4VZPvmwko0Puk6isRb1Y9MxXyaG SLMs7w/QJMQHp6ABKrrBTkSPTDKERxU0CqZV720nz0Ev4OtH0OTcw3ja0NORZfkyxTWv Cjw87YZrnB4pbNw3YkAv1Q6NAPE/LNKBU+u+8/nAC2lh/sNqUO4LEjYLCaq+BiK9wPE0 Ms4SPPMnZZ3PNCCjYWAs13YXeTqnSOAfZ2KFEeQumerCbKjhvwq3pxViaBfV+L0gyo1d Ec3Q== MIME-Version: 1.0 X-Received: by 10.140.145.9 with SMTP id 9mr69161053qhr.104.1426415922899; Sun, 15 Mar 2015 03:38:42 -0700 (PDT) Received: by 10.229.226.1 with HTTP; Sun, 15 Mar 2015 03:38:42 -0700 (PDT) Date: Sun, 15 Mar 2015 12:38:42 +0200 Message-ID: Subject: Unbalanced LACP link From: Vitalii Duk To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Mar 2015 10:38:44 -0000 Hello, guys. I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link DGS-3610-26G switch. This link worked fine last two years on FreeBSD 9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic started to go mostly through one interface. lagg0 - input interface (no IP-address, only VLANs). lagg1 - output interface. [router]# ifconfig lagg0 lagg0: flags=8843 metric 0 mtu 1500 options=400b8 ether 90:e2:ba:02:d0:ae nd6 options=29 media: Ethernet autoselect status: active laggproto lacp lagghash l2 laggport: igb1 flags=1c laggport: igb0 flags=1c [#router]# ifconfig lagg1 lagg1: flags=8843 metric 0 mtu 1500 options=400b8 ether 00:1b:21:93:52:d4 inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15 nd6 options=29 media: Ethernet autoselect status: active laggproto lacp lagghash l2 laggport: igb3 flags=1c laggport: igb2 flags=1c [router]# netstat -hw1 -I igb0 input igb0 output packets errs idrops bytes packets errs bytes colls 27K 0 0 12M 5 0 350 0 26K 0 0 12M 7 0 693 0 29K 0 0 12M 8 0 838 0 29K 0 0 11M 10 0 1.0K 0 27K 0 0 10M 7 0 770 0 As you can see most traffic is going through igb1 (lagg0) and igb3 (lagg1) interfaces. In peak times there are more that 1 Gbps of traffic mostly through one 1GbE interface, and it cause an errors. So this link is assymetric. [router]# netstat -hw1 -I igb1 input igb1 output packets errs idrops bytes packets errs bytes colls 22K 0 0 8.5M 63K 0 69M 0 20K 0 0 7.7M 58K 0 62M 0 20K 0 0 7.3M 57K 0 61M 0 19K 0 0 6.8M 56K 0 61M 0 18K 0 0 6.5M 57K 0 62M 0 [router]# netstat -hw1 -I igb2 input igb2 output packets errs idrops bytes packets errs bytes colls 40K 0 0 49M 1 0 93 0 41K 0 0 51M 1 0 88 0 38K 0 0 45M 1 0 88 0 33K 0 0 39M 2 0 174 0 31K 0 0 35M 1 0 89 0 [router]# netstat -hw1 -I igb3 input igb3 output packets errs idrops bytes packets errs bytes colls 32K 0 0 34M 44K 0 18M 0 29K 0 0 30M 45K 0 18M 0 32K 0 0 34M 48K 0 17M 0 28K 0 0 29M 42K 0 15M 0 25K 0 0 26M 40K 0 16M 0 [router]# netstat -hw1 -I lagg0 input lagg0 output packets errs idrops bytes packets errs bytes colls 45K 0 0 17M 62K 0 69M 0 46K 0 0 17M 66K 0 75M 0 43K 0 0 17M 62K 0 69M 0 50K 0 0 19M 71K 1 82M 0 46K 0 0 18M 66K 0 75M 0 [router]# netstat -hw1 -I lagg1 input lagg1 output packets errs idrops bytes packets errs bytes colls 70K 0 0 79M 49K 0 18M 0 71K 0 0 79M 49K 0 19M 0 79K 0 0 91M 54K 0 20M 0 65K 0 0 72M 45K 0 17M 0 65K 0 0 72M 45K 0 17M 0 All igb interfaces are the same: igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82576 Gigabit Network Connection' class = network subclass = ethernet I've tried to disable/enable strict_mode, put switch into active/passive mode and trying to set duplex/speed manyally. Nothing helps. Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because switch supports only l2 load-balancing, but it also doesn't helps. net.link.lagg.lacp.debug: 0 net.link.lagg.0.lacp.lacp_strict_mode: 0 net.link.lagg.0.lacp.debug.rx_test: 0 net.link.lagg.0.lacp.debug.tx_test: 0 net.link.lagg.1.lacp.lacp_strict_mode: 0 net.link.lagg.1.lacp.debug.rx_test: 0 net.link.lagg.1.lacp.debug.tx_test: 0 I've read that there were some problems with LACP on FreeBSD 10. Information from the switch: # show lacp summary Aggregate port 6: Local information: LACP port Oper Port Port Port Flags State Priority Key Number State ---------------------------------------------------------------------- Gi0/18 SP bndl 4096 0x6 0x12 0x3c Gi0/22 SP bndl 4096 0x6 0x16 0x3c Partner information: LACP port Oper Port Port Port Flags Priority Dev ID Key Number State --------------------------------------------------------------------- Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d Aggregate port 7: Local information: LACP port Oper Port Port Port Flags State Priority Key Number State ---------------------------------------------------------------------- Gi0/19 SP bndl 4096 0x7 0x13 0x3c Gi0/21 SP bndl 4096 0x7 0x15 0x3c Partner information: LACP port Oper Port Port Port Flags Priority Dev ID Key Number State --------------------------------------------------------------------- Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d