From owner-freebsd-net@FreeBSD.ORG Tue Jul 10 08:35:19 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B2ABB106564A for ; Tue, 10 Jul 2012 08:35:19 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 36F158FC0C for ; Tue, 10 Jul 2012 08:35:19 +0000 (UTC) Received: by eabm6 with SMTP id m6so4805259eab.13 for ; Tue, 10 Jul 2012 01:35:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=z876bEE4+CxbrwwIP8AYrQOqbaXgJH8QuaCDVhfXoDE=; b=iC7quxPLcjSTH+dDrTjiLXg6UCowbeQZMn1m6xoGCb+rWdOHK0Shy3ILK7ev3Py7JL EKqqqP9nsqCDxH0h/dDourO5OpsuIC07lgxZpb9a0Jq3YzL2li7JM/ZkDgRauJgpyTM0 355DPqKb1XMb69FrVb0xkeNs9W+zaBoRfUUpQr/faPFsnd6rPGv5Dx1398t0vQERxGfg e1VISBDDRaPko+4aPMg2B3q5Mcb0HBRmOz8clB1SqfLnH5tpfg7hO0xO0axHualzbjm4 XGXmCWIbBGQx3FypCZcnDZMQ+PIIVrTjLr4R/Z1khjmeOR9mhS2+mLmGE6nBZu2Gtz8w Ek/g== Received: by 10.14.94.204 with SMTP id n52mr10306484eef.200.1341909311500; Tue, 10 Jul 2012 01:35:11 -0700 (PDT) Received: from dfleuriot-at-hi-media.com ([83.167.62.196]) by mx.google.com with ESMTPS id q53sm100458608eef.8.2012.07.10.01.35.10 (version=SSLv3 cipher=OTHER); Tue, 10 Jul 2012 01:35:10 -0700 (PDT) Message-ID: <4FFBE93C.3080808@my.gd> Date: Tue, 10 Jul 2012 10:35:08 +0200 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <5E4F49720D0BAD499EE1F01232234BA877435B2E28@AVEXMB1.qlogic.org> <20120710071011.GB91639@DataIX.net> In-Reply-To: <20120710071011.GB91639@DataIX.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQlguwduqvlMRgQ8buY253rBr2CfrKpMcE4ldOufj8HjizX9cpiy7wUdzRnZXSlnDIDqiMY+ Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2012 08:35:19 -0000 On 7/10/12 9:10 AM, Jason Hellenthal wrote: > > > On Mon, Jul 09, 2012 at 05:38:24PM -0700, Adarsh Joshi wrote: >> Hi, >> >> I am trying to configure lacp lagg interfaces with 2 systems connected back to back as follows: >> >> Ifconfig lagg0 create >> Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0 >> >> Sometimes, the lag interface comes up correctly but sometimes the laggport flags do not show properly. Instead of 1c, it shows values of 18. I have seen similar issues reported on various forums with no solution. >> Looking at the lagg driver code and reading the standard, I thought the laggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v output does not make any sense to me. >> >> My concern is that when all the interfaces show flags as 1c, the traffic is distributed across both the interfaces uniformly and I get aggregated throughput. If not, the traffic flows only on 1 interface. >> >> Is this a bug? How do I solve this? Or am I doing something wrong? >> >> I am using Free-BSD 9.0 release. >> >> System 1: >> # ifconfig -v lagg0 >> lagg0: flags=8843 metric 0 mtu 1500 >> options=13b >> ether 00:0e:1e:08:05:20 >> inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 >> nd6 options=29 >> media: Ethernet autoselect >> status: active >> groups: lagg >> laggproto lacp >> lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), >> (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] >> laggport: ql1 flags=18 state=7D >> [(8000,00-0E-1E-08-05-20,0213,8000,000F), >> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] >> laggport: ql0 flags=1c state=3D >> [(8000,00-0E-1E-08-05-20,0213,8000,000E), >> (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] >> >> System 2: >> >> # ifconfig -v lagg0 >> lagg0: flags=8843 metric 0 mtu 1500 >> options=13b >> ether 00:0e:1e:04:2c:f0 >> inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 >> nd6 options=29 >> media: Ethernet autoselect >> status: active >> groups: lagg >> laggproto lacp >> lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), >> (FFFF,00-00-00-00-00-00,0000,0000,0000)] >> laggport: ql1 flags=1c state=7D >> [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), >> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] >> laggport: ql0 flags=18 state=3D >> [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), >> (8000,00-0E-1E-08-05-20,0213,8000,000E)] >> >> > > Just for reference ... (stable/8 @ r238264) > > lagg0: flags=8843 metric 0 mtu 1500 > options=80048 > ether 00:0c:41:21:1d:b5 > inet 192.168.XX.X netmask 0xffffff00 broadcast 192.168.XX.XXX > media: Ethernet autoselect > status: active > groups: lagg > laggproto lacp lagghash l2,l3,l4 > lag id: [(8000,00-0C-41-21-1D-B5,00E6,0000,0000), > (FFFF,00-00-00-00-00-00,0000,0000,0000)] > laggport: dc1 flags=1c state=7D > [(8000,00-0C-41-21-1D-B5,00E6,8000,0002), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > laggport: dc0 flags=1c state=7D > [(8000,00-0C-41-21-1D-B5,00E6,8000,0001), > (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] > > > They have had flags = 1c for quite some time and state = 7D > > And just to show the variation ... > > > dc0: > yesterday 8.53 MiB / 2.61 MiB / 11.14 MiB > today 693 KiB / 156 KiB / 849 KiB > > dc1: > yesterday 19.00 MiB / 1.79 MiB / 20.78 MiB > today 496 KiB / 103 KiB / 599 KiB > > lagg0: > yesterday 27.53 MiB / 3.71 MiB / 31.24 MiB > today 1.16 MiB / 172 KiB / 1.33 MiB > > > I believe (know) there has been some changes in the LAgg code in > stable/9 and stable/8 recently so you may want to check into that. > > Given this is LAgg and LACP you will see some variation regardless but I > recall a point that it seemed like one interface was being favored over > the other quite repeatedly or obsessively that had me second guessing > whether it was doing the right thing. > > LACP in Cisco is quite different than how we treat it here in FreeBSD as > it tends to use the interfaces quite evenly all the time so that also > has me second guessing whether the right thing is happening here. ( in > PAgP and LACP modes ). > Note that you can configure the way the switch load balances traffic thus: switch(config)#port-channel load-balance ? dst-ip Dst IP Addr dst-mac Dst Mac Addr src-dst-ip Src XOR Dst IP Addr src-dst-mac Src XOR Dst Mac Addr src-ip Src IP Addr src-mac Src Mac Addr You may also run simulations on the switch to see what interface from a port-channel would be used: switch#test etherchannel load-balance interface po48 ip 10.1.2.3 88.190.45.1 Would select Te1/1/2 of Po48 switch#test etherchannel load-balance interface po48 ip 10.1.2.3 88.190.45.9 Would select Te2/1/2 of Po48 switch#test etherchannel load-balance interface po48 ip 10.1.2.3 88.190.45.10 Would select Te2/1/2 of Po48 Hope this helps.