From owner-freebsd-net@FreeBSD.ORG Tue Jul 10 07:10:16 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 19A5F106566B for ; Tue, 10 Jul 2012 07:10:16 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: from mail-gg0-f182.google.com (mail-gg0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id A97788FC17 for ; Tue, 10 Jul 2012 07:10:15 +0000 (UTC) Received: by ggnm2 with SMTP id m2so12609577ggn.13 for ; Tue, 10 Jul 2012 00:10:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dataix.net; s=rsa; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=Yrxrz70WqBEtArNBG/eRbNrxvhdSqocZ5B5zzaS9G9M=; b=LaVFObOhJCBITtlTWrjqG1DcWw/ifseVEgQm8yO5C+wBeA6+8Pn7FRtche2YD0q8aa nfAlxn7ZuHboN8KpNv7lNH1DIbLbrjFc4FPg2k3uHcqXVEfdSTBO6aUemito7NjERfrW 6zYqQEMLQPB0AkyDlohAJ1TCpvttLu0ZqGrpY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-gm-message-state; bh=Yrxrz70WqBEtArNBG/eRbNrxvhdSqocZ5B5zzaS9G9M=; b=cGJY3zL2nbLYtq62HZ53JjsmZC6JdhyAt9YU2tTwNu2pRP+ntA8ULCek5pZfGyHAP4 /ZGRUgZLxFTo/YwM/HJ/a+M1h1y2orsyVM/xJ2peDNEIiF3YRz3MvH8MRDUI1GX49dLg NM3os25POtXQy345OAg32kZhwD2WablUZExrnqhCGaY4MTGR/uQns8LioL995I0V7J7k 1+O0zi3sajZAwZwCOsH3co4g+CLCcULIKEVFHOAUW4PpLD86VLJcCGtGR/b7EcxsBGYh HZw0mTrvLlrV2YUBVTYs8H0W4RZFQf9xgk8gbDJ1ufWaIhlIOoJEPPMoJHa0JuTtu8TK Wqfg== Received: by 10.50.41.201 with SMTP id h9mr604487igl.37.1341904214979; Tue, 10 Jul 2012 00:10:14 -0700 (PDT) Received: from DataIX.net (adsl-99-181-136-60.dsl.klmzmi.sbcglobal.net. [99.181.136.60]) by mx.google.com with ESMTPS id if4sm11032084igc.10.2012.07.10.00.10.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 10 Jul 2012 00:10:14 -0700 (PDT) Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id q6A7ABSW094638 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 10 Jul 2012 03:10:12 -0400 (EDT) (envelope-from jhellenthal@DataIX.net) Received: (from jh@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id q6A7ABfH094637; Tue, 10 Jul 2012 03:10:11 -0400 (EDT) (envelope-from jhellenthal@DataIX.net) Date: Tue, 10 Jul 2012 03:10:11 -0400 From: Jason Hellenthal To: Adarsh Joshi Message-ID: <20120710071011.GB91639@DataIX.net> References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline In-Reply-To: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> X-Gm-Message-State: ALoCoQnIWrAibjDkdSfJSInEf33XrfLVsIIKJdS3dYfiJkiQukDWsUuz2kbdLf2DoELnhA0C5s3+ Cc: "freebsd-net@freebsd.org" Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2012 07:10:16 -0000 --huq684BweRXVnRxX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 09, 2012 at 05:38:24PM -0700, Adarsh Joshi wrote: > Hi, >=20 > I am trying to configure lacp lagg interfaces with 2 systems connected ba= ck to back as follows: >=20 > Ifconfig lagg0 create > Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 net= mask 255.255.255.0 >=20 > Sometimes, the lag interface comes up correctly but sometimes the laggpor= t flags do not show properly. Instead of 1c= , it shows values of 18. I have seen similar issues reported on various for= ums with no solution. > Looking at the lagg driver code and reading the standard, I thought the l= aggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in f= ile ieee8023ad_lacp.h. But the following ifconfig -v output does not make a= ny sense to me. >=20 > My concern is that when all the interfaces show flags as 1c, the traffic = is distributed across both the interfaces uniformly and I get aggregated th= roughput. If not, the traffic flows only on 1 interface. >=20 > Is this a bug? How do I solve this? Or am I doing something wrong? >=20 > I am using Free-BSD 9.0 release. >=20 > System 1: > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:08:05:20 > inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), > (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] > laggport: ql1 flags=3D18 state=3D7D > [(8000,00-0E-1E-08-05-20,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D1c state=3D= 3D > [(8000,00-0E-1E-08-05-20,0213,8000,000E), > (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] >=20 > System 2: >=20 > # ifconfig -v lagg0 > lagg0: flags=3D8843 metric 0 mtu = 1500 > options=3D13b > ether 00:0e:1e:04:2c:f0 > inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=3D29 > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp > lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: ql1 flags=3D1c state=3D= 7D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: ql0 flags=3D18 state=3D3D > [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), > (8000,00-0E-1E-08-05-20,0213,8000,000E)] >=20 >=20 Just for reference ... (stable/8 @ r238264) lagg0: flags=3D8843 metric 0 mtu 15= 00 options=3D80048 ether 00:0c:41:21:1d:b5 inet 192.168.XX.X netmask 0xffffff00 broadcast 192.168.XX.XXX media: Ethernet autoselect status: active groups: lagg=20 laggproto lacp lagghash l2,l3,l4 lag id: [(8000,00-0C-41-21-1D-B5,00E6,0000,0000), (FFFF,00-00-00-00-00-00,0000,0000,0000)] laggport: dc1 flags=3D1c state=3D7D [(8000,00-0C-41-21-1D-B5,00E6,8000,0002), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: dc0 flags=3D1c state=3D7D [(8000,00-0C-41-21-1D-B5,00E6,8000,0001), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] They have had flags =3D 1c for quite some time and state =3D 7D And just to show the variation ... dc0: yesterday 8.53 MiB / 2.61 MiB / 11.14 MiB today 693 KiB / 156 KiB / 849 KiB dc1: yesterday 19.00 MiB / 1.79 MiB / 20.78 MiB today 496 KiB / 103 KiB / 599 KiB lagg0: yesterday 27.53 MiB / 3.71 MiB / 31.24 MiB today 1.16 MiB / 172 KiB / 1.33 MiB I believe (know) there has been some changes in the LAgg code in stable/9 and stable/8 recently so you may want to check into that. Given this is LAgg and LACP you will see some variation regardless but I recall a point that it seemed like one interface was being favored over the other quite repeatedly or obsessively that had me second guessing whether it was doing the right thing. LACP in Cisco is quite different than how we treat it here in FreeBSD as it tends to use the interfaces quite evenly all the time so that also has me second guessing whether the right thing is happening here. ( in PAgP and LACP modes ). These tunables (sysctl)'s may also lead you in direction you want to look in. net.link.lagg.default_use_flowid: 1 net.link.lagg.failover_rx_all: 0 net.link.lagg.0.use_flowid: 1 -rxcsum & -txcsum for lo0 and ql0 ql1 might be of benefit too though I only turn them off on lo0. Good luck --=20 - (2^(N-1)) --huq684BweRXVnRxX Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJP+9VSAAoJEBSh2Dr1DU7WHzEH/2PBC6Iz1BOoL8PPAEfzb/Mz llMEDATV45W+JVWyAeWaG2GH+o7KaQPfp/Z5qTH/dcgqA3r7fiCqrRZpd61DhbGF GlzryhUJKY9V/FDJXCSaevyhFdjtosvZQ8qEchA18FsDGEX2wA/4dXv/BZs1YQQx yhXgVd3gIjrqa7bt0sqOqWdngizVHaEt7oDyFvl2I4kpTbd1TETUOCXT2IMEGNPQ uVghGx3sQSiAePXV1IjizMcW0QGOTX+bUmJLlvQXSOPjxkXIRYnn2xSIh8WGMZT+ iK2UiuvzY8mIPBTVw+yvst954pxrscWPNXPhH8nqAudp3SHpSlWPszEJwxN3Ypo= =pvae -----END PGP SIGNATURE----- --huq684BweRXVnRxX--