From owner-freebsd-net@FreeBSD.ORG Fri Aug 2 23:37:17 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 868CC8C7 for ; Fri, 2 Aug 2013 23:37:17 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm22-vm2.bullet.mail.ne1.yahoo.com (nm22-vm2.bullet.mail.ne1.yahoo.com [98.138.91.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 460E02F2E for ; Fri, 2 Aug 2013 23:37:16 +0000 (UTC) Received: from [98.138.101.131] by nm22.bullet.mail.ne1.yahoo.com with NNFMP; 02 Aug 2013 23:35:28 -0000 Received: from [98.138.89.240] by tm19.bullet.mail.ne1.yahoo.com with NNFMP; 02 Aug 2013 23:35:28 -0000 Received: from [127.0.0.1] by omp1013.mail.ne1.yahoo.com with NNFMP; 02 Aug 2013 23:35:28 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 144754.30943.bm@omp1013.mail.ne1.yahoo.com Received: (qmail 43048 invoked by uid 60001); 2 Aug 2013 23:35:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1375486527; bh=PfKSdRFw3zd1GlDbck01yFM1pel2W/mpBJjjIZt49Tc=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=H5SJ0tOtqK+B6VGuMUvYDt3pp3TjDLpnWtLt8sx0eN2Lgs4dXXHFAPddLW/aWtUadhIBJE7Sbv1aWJLkHBbUZDiMepdW51Mykfvhb4s4oA+xxKLPRBuWL3JS9dvr3lIR4PSt271XWBqtztXAAohrejirLLUzl4tJcmhV5pKIHzQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ZBs7bA3LSzPEbYE0eFH5ehzNb0JBEzWDmelrhiHkvqJLq3h+iOSzeB3/Z1qhL5dR/0lwJTjvSPmBohF7xDGqRghNFtMP/aW4kzPRErGG5P6dtcVXX8VO87WtHwzL/nr5zP3hypmF0ct93rCSPILpnwdr0A/1Jm6b+t+aUxkPGfc=; X-YMail-OSG: iQGy7GkVM1kS1XuPuyGsdjuY._u9RSo6K__jysJgN4Q3BsH 16j46AtW0Xcwk6AkuVd9ZDVRvGgwpN6VS13wknddIF4vgk5TavExYLU1M2CS f0Aulf864eAbBeaFAhLWPZAn6hrgNTCvSRjhsxTaGbbjqa7pxh_e2GZ.ZxNy EilIQfHup4p4M2IBQqmsbSoxfeIoxH4KkJO6faHU9jFmIj7fFDQ2sAkaMX7. dGzZbqzsmUcX6hHZGVkHxPBJwgBlaCtWcCN9WK0vcWPxeg2KAXUwO_SswQ__ znFy8sADXjTtlGvOIUw3eN8FefvgokvEZPMVJMTR7C3lnE6jcTmgwll1Y6qf 1yNqch2SoX.6SdkvqUdGQLPuth3myvhbyksjCT90fo5_o6urj700_mMTkXuf q3fpc.1QuFLC7ye.blYo07q5Y7Y0QJmkjylXJZygLkq.Jr94ZtsfknUh1krk 2QqT.3LUM4VOHovWwB0ErLI7Q8hLTSrcmDG6q7VFRTCd2AR3JKK7xlk4wdtW n2lkhceluaNQ9A_O7WNUyPQSyG08ppcahJ1UCbePbwVXoFDqjVf2CbUn.o1e 9YRQtJ0hiyox4CIjt96n4m15CPZzoWfnGiR8hOmr6_UKNHg0pOqGQfJtOXIp jqFYFqvcPDLCXrp4uNZbqRlc.8D6fwiwwEJheXm7lDbk- Received: from [98.203.118.124] by web121603.mail.ne1.yahoo.com via HTTP; Fri, 02 Aug 2013 16:35:27 PDT X-Rocket-MIMEInfo: 002.001, VGhlIHN0b2NrIGlnYiBkcml2ZXIgYmluZHMgdG8gYWxsIGNvcmVzLCBzbyB3aXRoIG11bHRpcGxlIGlnYnMgeW91IGhhdmUgbXVsdGlwbGUKbmljcyBiaW5kaW5nIHRvIHRoZSBzYW1lIGNvcmVzLiBJIHN1cHBvc2UgdGhhdCBtaWdodCBjcmVhdGUgaXNzdWVzIGluIGEgbGFnZyBzZXR1cC4KVHJ5IDEgcXVldWUgwqBhbmQvb3IgY29tbWVudCBvdXQgdGhlIGJpbmQgY29kZS4KCkJDCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KIEZyb206IFphcGhvZCBCZWVibGVicm94IDx6YmVlYmxlQGdtYWkBMAEBAQE- X-Mailer: YahooMailWebService/0.8.152.567 References: <51FB617E.2090904@netasq.com> Message-ID: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 2 Aug 2013 16:35:27 -0700 (PDT) From: Barney Cordoba Subject: Re: Intel 4-port ethernet adaptor link aggregation issue To: Zaphod Beeblebrox , Freddie Cash In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Steve Read , freebsd-net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 23:37:17 -0000 The stock igb driver binds to all cores, so with multiple igbs you have mul= tiple=0Anics binding to the same cores. I suppose that might create issues = in a lagg setup.=0ATry 1 queue =A0and/or comment out the bind code.=0A=0ABC= =0A=0A=0A________________________________=0A From: Zaphod Beeblebrox =0ATo: Freddie Cash =0ACc: Steve Read ; freebsd-net =0ASent: Friday,= August 2, 2013 5:41 PM=0ASubject: Re: Intel 4-port ethernet adaptor link a= ggregation issue=0A =0A=0AOn several machines with large numbers of IGBx in= terfaces, I've found that=0Ahw.igb.enable_msix=3D0 is necessary to ensure p= roper operation.=0A=0A=0AOn Fri, Aug 2, 2013 at 11:49 AM, Freddie Cash wrote:=0A=0A> On Fri, Aug 2, 2013 at 12:36 AM, Steve Read <= steve.read@netasq.com> wrote:=0A>=0A> > On 01.08.2013 20:07, Joe Moog wrote= :=0A> >=0A> >> We have an iXsystems 1U server (E5) with an Intel 4-port eth= ernet NIC=0A> >> installed, model I350-T4 (manufactured May of 2013). We're= trying to=0A> bind=0A> >> the 4 ports on this NIC together into a single l= agg port, connected=0A> LACP to=0A> >> a distribution switch (Cisco 4900-se= ries). We are able to successfully=0A> bind=0A> >> the 2 on-board ethernet = ports to a single lagg, however the NIC is not=0A> so=0A> >> cooperative. A= t first we thought we had a bad NIC, but a replacement has=0A> >> not fixed= the issue. We are thinking there may be a driver limitation=0A> with=0A> >= > these Intel ethernet NICs when attempting to bind more than 2 ports to a= =0A> >> lagg.=0A> >>=0A> >> FreeBSD version:=0A> >> FreeBSD 9.1-PRERELEASE = #0 r244125: Wed Dec 12 11:47:47 CST 2012=0A> >>=0A> >> rc.conf:=0A> >> # LI= NK AGGREGATION=0A> >> ifconfig_igb2=3D"UP"=0A> >> ifconfig_igb3=3D"UP"=0A> = >> ifconfig_igb4=3D"UP"=0A> >> ifconfig_igb5=3D"UP"=0A> >> cloned_interface= s=3D"lagg0"=0A> >> ifconfig_lagg0=3D"laggproto lacp laggport igb2 laggport = igb3 laggport igb4=0A> >> laggport igb5"=0A> >> ifconfig_lagg0=3D"inet 192.= 168.1.14=A0 netmask 255.255.255.0"=0A> >>=0A> >=0A>=0A> > Am I the only one= who noticed that you replaced the value of=0A> > $ifconfig_lagg0 that spec= ifies the proto and the ports with one that=0A> > specifies just the addres= s?=0A> >=0A>=0A> Good catch!=0A>=0A>=0A> > Merge the two ifconfig_lagg0 lin= es into one, and it will work infinitely=0A> > better, or at least no worse= .=0A> >=0A> > ifconfig_lagg0=3D"laggproto lacp laggport igb2 laggport igb3 = laggport igb4=0A> > laggport igb5 inet 192.168.1.14=A0 netmask 255.255.255.= 0"=0A> >=0A> > Or, if you want to keep them split into two parts (initialis= e lagg0, then=0A> add IP):=0A>=0A> create_args_lagg0=3D"laggproto lacp lagg= port igb2 laggport igb3 laggport igb4=0A> laggport igb5"=0A>=0A> ifconfig_l= agg0=3D"inet 192.168.1.14=A0 netmask 255.255.255.0"=0A>=0A> create_args_* a= re run first, then ifconfig_* are run.=A0 I like this setup,=0A> as it sepa= rates "create and initialise" from "configure" for cloned/virtual=0A> inter= faces like vlans, laggs, etc.=0A>=0A>=0A> --=0A> Freddie Cash=0A> fjwcash@g= mail.com=0A> _______________________________________________=0A> freebsd-ne= t@freebsd.org mailing list=0A> http://lists.freebsd.org/mailman/listinfo/fr= eebsd-net=0A> To unsubscribe, send any mail to "freebsd-net-unsubscribe@fre= ebsd.org"=0A>=0A_______________________________________________=0Afreebsd-n= et@freebsd.org mailing list=0Ahttp://lists.freebsd.org/mailman/listinfo/fre= ebsd-net=0ATo unsubscribe, send any mail to "freebsd-net-unsubscribe@freebs= d.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 3 04:21:11 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DC3FC91F for ; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70F68271E for ; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id n11so64437wgh.0 for ; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=5DGFt85uIBlAKRTv4+LbgV08TrqISup/JA7E8IB/KI8=; b=awbPxtrTUsqc9OKXMugJWGiYLV0V4XPlCMbF2OCUMrGYFqHurs1Pjm+P/xzsCzb9FL lFjypRQB0ABJuHHsjHn9bxHffUM14S7DMqFbFnH+gRRDhBhfMZm5b5UXV95bxy0SnrIZ 2tHAyB6m+ueYMWbpT6mNeCKxNe5p8aoB9IDEVZ4hgv/efdvxINzwgmhxZUOQzQxRlA60 WrBCy0xDfMWidTUjORWCymoSxaiDzMylCeOGVtll3aL9CJ8YKN2Yj08G6beTI7RR8NIJ xFTV2I+FIKWOnbntMvZSz3pD/JLSRkXhwlqiZNXHy6SV9Vd+vKLtwFvC34HjjIPaZ/pg NkUQ== MIME-Version: 1.0 X-Received: by 10.194.11.67 with SMTP id o3mr7093699wjb.0.1375503669850; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Fri, 2 Aug 2013 21:21:09 -0700 (PDT) In-Reply-To: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> References: <51FB617E.2090904@netasq.com> <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 2 Aug 2013 21:21:09 -0700 X-Google-Sender-Auth: hsEA0hut4Bn947cA9Kr4fiWaaGM Message-ID: Subject: Re: Intel 4-port ethernet adaptor link aggregation issue From: Adrian Chadd To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 Cc: Steve Read , Zaphod Beeblebrox , Freddie Cash , freebsd-net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 04:21:11 -0000 On 2 August 2013 16:35, Barney Cordoba wrote: > The stock igb driver binds to all cores, so with multiple igbs you have multiple > nics binding to the same cores. I suppose that might create issues in a lagg setup. > Try 1 queue and/or comment out the bind code. > I have thrashed the hell out of 2-port ixgbe and 4-port chelsio (cxgbe) on 4-core device all with lagg. All is great. There's apparently some more igb improvements coming in the pipeline. Fear not! -adrian