Date: Fri, 2 Aug 2013 16:35:27 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Zaphod Beeblebrox <zbeeble@gmail.com>, Freddie Cash <fjwcash@gmail.com> Cc: Steve Read <steve.read@netasq.com>, freebsd-net <freebsd-net@freebsd.org> Subject: Re: Intel 4-port ethernet adaptor link aggregation issue Message-ID: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> In-Reply-To: <CACpH0McBFG5qYNZDrp=1ubW49bnj%2BPsFPBbhUDJpO4Hgsccs2A@mail.gmail.com> References: <B966242F-A52D-43F7-A001-99942D53339E@ebureau.com> <51FB617E.2090904@netasq.com> <CAOjFWZ4B2RLSaWTkriCB3imis0KL1FzOBbnQerJ1GnO2dc5YdA@mail.gmail.com> <CACpH0McBFG5qYNZDrp=1ubW49bnj%2BPsFPBbhUDJpO4Hgsccs2A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The stock igb driver binds to all cores, so with multiple igbs you have mul= tiple=0Anics binding to the same cores. I suppose that might create issues = in a lagg setup.=0ATry 1 queue =A0and/or comment out the bind code.=0A=0ABC= =0A=0A=0A________________________________=0A From: Zaphod Beeblebrox <zbeeb= le@gmail.com>=0ATo: Freddie Cash <fjwcash@gmail.com> =0ACc: Steve Read <ste= ve.read@netasq.com>; freebsd-net <freebsd-net@freebsd.org> =0ASent: Friday,= August 2, 2013 5:41 PM=0ASubject: Re: Intel 4-port ethernet adaptor link a= ggregation issue=0A =0A=0AOn several machines with large numbers of IGBx in= terfaces, I've found that=0Ahw.igb.enable_msix=3D0 is necessary to ensure p= roper operation.=0A=0A=0AOn Fri, Aug 2, 2013 at 11:49 AM, Freddie Cash <fjw= cash@gmail.com> wrote:=0A=0A> On Fri, Aug 2, 2013 at 12:36 AM, Steve Read <= steve.read@netasq.com> wrote:=0A>=0A> > On 01.08.2013 20:07, Joe Moog wrote= :=0A> >=0A> >> We have an iXsystems 1U server (E5) with an Intel 4-port eth= ernet NIC=0A> >> installed, model I350-T4 (manufactured May of 2013). We're= trying to=0A> bind=0A> >> the 4 ports on this NIC together into a single l= agg port, connected=0A> LACP to=0A> >> a distribution switch (Cisco 4900-se= ries). We are able to successfully=0A> bind=0A> >> the 2 on-board ethernet = ports to a single lagg, however the NIC is not=0A> so=0A> >> cooperative. A= t first we thought we had a bad NIC, but a replacement has=0A> >> not fixed= the issue. We are thinking there may be a driver limitation=0A> with=0A> >= > these Intel ethernet NICs when attempting to bind more than 2 ports to a= =0A> >> lagg.=0A> >>=0A> >> FreeBSD version:=0A> >> FreeBSD 9.1-PRERELEASE = #0 r244125: Wed Dec 12 11:47:47 CST 2012=0A> >>=0A> >> rc.conf:=0A> >> # LI= NK AGGREGATION=0A> >> ifconfig_igb2=3D"UP"=0A> >> ifconfig_igb3=3D"UP"=0A> = >> ifconfig_igb4=3D"UP"=0A> >> ifconfig_igb5=3D"UP"=0A> >> cloned_interface= s=3D"lagg0"=0A> >> ifconfig_lagg0=3D"laggproto lacp laggport igb2 laggport = igb3 laggport igb4=0A> >> laggport igb5"=0A> >> ifconfig_lagg0=3D"inet 192.= 168.1.14=A0 netmask 255.255.255.0"=0A> >>=0A> >=0A>=0A> > Am I the only one= who noticed that you replaced the value of=0A> > $ifconfig_lagg0 that spec= ifies the proto and the ports with one that=0A> > specifies just the addres= s?=0A> >=0A>=0A> Good catch!=0A>=0A>=0A> > Merge the two ifconfig_lagg0 lin= es into one, and it will work infinitely=0A> > better, or at least no worse= .=0A> >=0A> > ifconfig_lagg0=3D"laggproto lacp laggport igb2 laggport igb3 = laggport igb4=0A> > laggport igb5 inet 192.168.1.14=A0 netmask 255.255.255.= 0"=0A> >=0A> > Or, if you want to keep them split into two parts (initialis= e lagg0, then=0A> add IP):=0A>=0A> create_args_lagg0=3D"laggproto lacp lagg= port igb2 laggport igb3 laggport igb4=0A> laggport igb5"=0A>=0A> ifconfig_l= agg0=3D"inet 192.168.1.14=A0 netmask 255.255.255.0"=0A>=0A> create_args_* a= re run first, then ifconfig_* are run.=A0 I like this setup,=0A> as it sepa= rates "create and initialise" from "configure" for cloned/virtual=0A> inter= faces like vlans, laggs, etc.=0A>=0A>=0A> --=0A> Freddie Cash=0A> fjwcash@g= mail.com=0A> _______________________________________________=0A> freebsd-ne= t@freebsd.org mailing list=0A> http://lists.freebsd.org/mailman/listinfo/fr= eebsd-net=0A> To unsubscribe, send any mail to "freebsd-net-unsubscribe@fre= ebsd.org"=0A>=0A_______________________________________________=0Afreebsd-n= et@freebsd.org mailing list=0Ahttp://lists.freebsd.org/mailman/listinfo/fre= ebsd-net=0ATo unsubscribe, send any mail to "freebsd-net-unsubscribe@freebs= d.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 3 04:21:11 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DC3FC91F for <freebsd-net@freebsd.org>; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70F68271E for <freebsd-net@freebsd.org>; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id n11so64437wgh.0 for <freebsd-net@freebsd.org>; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=5DGFt85uIBlAKRTv4+LbgV08TrqISup/JA7E8IB/KI8=; b=awbPxtrTUsqc9OKXMugJWGiYLV0V4XPlCMbF2OCUMrGYFqHurs1Pjm+P/xzsCzb9FL lFjypRQB0ABJuHHsjHn9bxHffUM14S7DMqFbFnH+gRRDhBhfMZm5b5UXV95bxy0SnrIZ 2tHAyB6m+ueYMWbpT6mNeCKxNe5p8aoB9IDEVZ4hgv/efdvxINzwgmhxZUOQzQxRlA60 WrBCy0xDfMWidTUjORWCymoSxaiDzMylCeOGVtll3aL9CJ8YKN2Yj08G6beTI7RR8NIJ xFTV2I+FIKWOnbntMvZSz3pD/JLSRkXhwlqiZNXHy6SV9Vd+vKLtwFvC34HjjIPaZ/pg NkUQ== MIME-Version: 1.0 X-Received: by 10.194.11.67 with SMTP id o3mr7093699wjb.0.1375503669850; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Fri, 2 Aug 2013 21:21:09 -0700 (PDT) In-Reply-To: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> References: <B966242F-A52D-43F7-A001-99942D53339E@ebureau.com> <51FB617E.2090904@netasq.com> <CAOjFWZ4B2RLSaWTkriCB3imis0KL1FzOBbnQerJ1GnO2dc5YdA@mail.gmail.com> <CACpH0McBFG5qYNZDrp=1ubW49bnj+PsFPBbhUDJpO4Hgsccs2A@mail.gmail.com> <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 2 Aug 2013 21:21:09 -0700 X-Google-Sender-Auth: hsEA0hut4Bn947cA9Kr4fiWaaGM Message-ID: <CAJ-Vmok=dhaA+TMJCgzck5Nhgi-qMTLtiQvrYXN0UPbVGvjHcw@mail.gmail.com> Subject: Re: Intel 4-port ethernet adaptor link aggregation issue From: Adrian Chadd <adrian@freebsd.org> To: Barney Cordoba <barney_cordoba@yahoo.com> Content-Type: text/plain; charset=ISO-8859-1 Cc: Steve Read <steve.read@netasq.com>, Zaphod Beeblebrox <zbeeble@gmail.com>, Freddie Cash <fjwcash@gmail.com>, freebsd-net <freebsd-net@freebsd.org> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 03 Aug 2013 04:21:11 -0000 On 2 August 2013 16:35, Barney Cordoba <barney_cordoba@yahoo.com> wrote: > The stock igb driver binds to all cores, so with multiple igbs you have multiple > nics binding to the same cores. I suppose that might create issues in a lagg setup. > Try 1 queue and/or comment out the bind code. > I have thrashed the hell out of 2-port ixgbe and 4-port chelsio (cxgbe) on 4-core device all with lagg. All is great. There's apparently some more igb improvements coming in the pipeline. Fear not! -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1375486527.35304.YahooMailNeo>