From owner-freebsd-net@FreeBSD.ORG Mon Mar 16 09:37:18 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A1FCC41 for ; Mon, 16 Mar 2015 09:37:18 +0000 (UTC) Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com [IPv6:2607:f8b0:400d:c01::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C3251EEE for ; Mon, 16 Mar 2015 09:37:17 +0000 (UTC) Received: by qcaz10 with SMTP id z10so38276176qca.1 for ; Mon, 16 Mar 2015 02:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=lmMsSFyw87/suiV1i8R0Am7WC++Jyf1bjugrdxMF8Fo=; b=m3ZeLytONiGcNA/426eNxc51cGTTWo1dyUeGRqQ/Dk0ccLikFJmAp3pycvaBwFu9g+ jSY4CcxHyxAm8/9w0UEu1ztABaFwPtHFc0HBIQep5uNkdPd6QoAHrfYZ5SW8TlMrboI2 uTIDtjtLWfDNuFN06DefZdxXNWWwH3/1tp6CCl2aux4+92tvPSq2YMxnFoBC5HQ+YhGe GMA43MJEpIzHPbqnQW70Sj2cFDtI8PppVUK0JkFSdPIruCZjQCC5y+fMGS669HnJwJMu rrtd1BW/A5Kjw1Zaw/5pdHPFj4Nv1F942MKAp3UH3ANMVZBzx/YMhu603q0Hu+eIDoef 03cw== MIME-Version: 1.0 X-Received: by 10.55.40.9 with SMTP id o9mr89737682qkh.55.1426498636875; Mon, 16 Mar 2015 02:37:16 -0700 (PDT) Received: by 10.229.226.1 with HTTP; Mon, 16 Mar 2015 02:37:16 -0700 (PDT) In-Reply-To: References: Date: Mon, 16 Mar 2015 11:37:16 +0200 Message-ID: Subject: Re: Unbalanced LACP link From: Vitalii Duk To: Konstantin Kulikov Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Mar 2015 09:37:18 -0000 I've changed use_flowid to 0 and it helped! But isn't it setting significant? In a description it says "Shift flowid bits to prevent multiqueue collisions". On 16 March 2015 at 09:50, Konstantin Kulikov wrote: > Hello, Vitalii. > > Make sure following sysctl are set to zero: > sysctl net.link.lagg.default_use_flowid=0 > sysctl net.link.lagg.0.use_flowid=0 > sysctl net.link.lagg.1.use_flowid=0 > > Then adjust lagghash depending on your configuration. > For example if you freebsd machine acts as router with single, default, > route, > you should set lagghash to l3, since for l2 both src and dst MACs will > always be same and you'd get unbalanced load. > > On Sun, Mar 15, 2015 at 1:38 PM, Vitalii Duk wrote: > > Hello, guys. > > I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link > > DGS-3610-26G switch. This link worked fine last two years on FreeBSD > > 9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic > > started to go mostly through one interface. > > > > lagg0 - input interface (no IP-address, only VLANs). > > lagg1 - output interface. > > > > [router]# ifconfig lagg0 > > lagg0: flags=8843 metric 0 mtu > 1500 > > > > options=400b8 > > ether 90:e2:ba:02:d0:ae > > nd6 options=29 > > media: Ethernet autoselect > > status: active > > laggproto lacp lagghash l2 > > laggport: igb1 flags=1c > > laggport: igb0 flags=1c > > > > [#router]# ifconfig lagg1 > > lagg1: flags=8843 metric 0 mtu > 1500 > > > > options=400b8 > > ether 00:1b:21:93:52:d4 > > inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15 > > nd6 options=29 > > media: Ethernet autoselect > > status: active > > laggproto lacp lagghash l2 > > laggport: igb3 flags=1c > > laggport: igb2 flags=1c > > > > [router]# netstat -hw1 -I igb0 > > input igb0 output > > packets errs idrops bytes packets errs bytes colls > > 27K 0 0 12M 5 0 350 0 > > 26K 0 0 12M 7 0 693 0 > > 29K 0 0 12M 8 0 838 0 > > 29K 0 0 11M 10 0 1.0K 0 > > 27K 0 0 10M 7 0 770 0 > > > > As you can see most traffic is going through igb1 (lagg0) and igb3 > (lagg1) > > interfaces. In peak times there are more that 1 Gbps of traffic mostly > > through one 1GbE interface, and it cause an errors. So this link is > > assymetric. > > > > [router]# netstat -hw1 -I igb1 > > input igb1 output > > packets errs idrops bytes packets errs bytes colls > > 22K 0 0 8.5M 63K 0 69M 0 > > 20K 0 0 7.7M 58K 0 62M 0 > > 20K 0 0 7.3M 57K 0 61M 0 > > 19K 0 0 6.8M 56K 0 61M 0 > > 18K 0 0 6.5M 57K 0 62M 0 > > > > [router]# netstat -hw1 -I igb2 > > input igb2 output > > packets errs idrops bytes packets errs bytes colls > > 40K 0 0 49M 1 0 93 0 > > 41K 0 0 51M 1 0 88 0 > > 38K 0 0 45M 1 0 88 0 > > 33K 0 0 39M 2 0 174 0 > > 31K 0 0 35M 1 0 89 0 > > > > [router]# netstat -hw1 -I igb3 > > input igb3 output > > packets errs idrops bytes packets errs bytes colls > > 32K 0 0 34M 44K 0 18M 0 > > 29K 0 0 30M 45K 0 18M 0 > > 32K 0 0 34M 48K 0 17M 0 > > 28K 0 0 29M 42K 0 15M 0 > > 25K 0 0 26M 40K 0 16M 0 > > > > [router]# netstat -hw1 -I lagg0 > > input lagg0 output > > packets errs idrops bytes packets errs bytes colls > > 45K 0 0 17M 62K 0 69M 0 > > 46K 0 0 17M 66K 0 75M 0 > > 43K 0 0 17M 62K 0 69M 0 > > 50K 0 0 19M 71K 1 82M 0 > > 46K 0 0 18M 66K 0 75M 0 > > > > [router]# netstat -hw1 -I lagg1 > > input lagg1 output > > packets errs idrops bytes packets errs bytes colls > > 70K 0 0 79M 49K 0 18M 0 > > 71K 0 0 79M 49K 0 19M 0 > > 79K 0 0 91M 54K 0 20M 0 > > 65K 0 0 72M 45K 0 17M 0 > > 65K 0 0 72M 45K 0 17M 0 > > > > All igb interfaces are the same: > > igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 > > rev=0x01 hdr=0x00 > > vendor = 'Intel Corporation' > > device = '82576 Gigabit Network Connection' > > class = network > > subclass = ethernet > > > > I've tried to disable/enable strict_mode, put switch into active/passive > > mode and trying to set duplex/speed manyally. Nothing helps. > > Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because > > switch supports only l2 load-balancing, but it also doesn't helps. > > > > net.link.lagg.lacp.debug: 0 > > net.link.lagg.0.lacp.lacp_strict_mode: 0 > > net.link.lagg.0.lacp.debug.rx_test: 0 > > net.link.lagg.0.lacp.debug.tx_test: 0 > > net.link.lagg.1.lacp.lacp_strict_mode: 0 > > net.link.lagg.1.lacp.debug.rx_test: 0 > > net.link.lagg.1.lacp.debug.tx_test: 0 > > > > I've read that there were some problems with LACP on FreeBSD 10. > > > > Information from the switch: > > > > # show lacp summary > > Aggregate port 6: > > > > Local information: > > LACP port Oper Port Port > > Port Flags State Priority Key Number State > > ---------------------------------------------------------------------- > > Gi0/18 SP bndl 4096 0x6 0x12 0x3c > > Gi0/22 SP bndl 4096 0x6 0x16 0x3c > > > > Partner information: > > LACP port Oper Port Port > > Port Flags Priority Dev ID Key Number State > > --------------------------------------------------------------------- > > Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d > > Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d > > > > Aggregate port 7: > > > > Local information: > > LACP port Oper Port Port > > Port Flags State Priority Key Number State > > ---------------------------------------------------------------------- > > Gi0/19 SP bndl 4096 0x7 0x13 0x3c > > Gi0/21 SP bndl 4096 0x7 0x15 0x3c > > > > Partner information: > > LACP port Oper Port Port > > Port Flags Priority Dev ID Key Number State > > --------------------------------------------------------------------- > > Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d > > Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >