Date: Fri, 2 Aug 2013 16:35:27 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Zaphod Beeblebrox <zbeeble@gmail.com>, Freddie Cash <fjwcash@gmail.com> Cc: Steve Read <steve.read@netasq.com>, freebsd-net <freebsd-net@freebsd.org> Subject: Re: Intel 4-port ethernet adaptor link aggregation issue Message-ID: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> In-Reply-To: <CACpH0McBFG5qYNZDrp=1ubW49bnj%2BPsFPBbhUDJpO4Hgsccs2A@mail.gmail.com> References: <B966242F-A52D-43F7-A001-99942D53339E@ebureau.com> <51FB617E.2090904@netasq.com> <CAOjFWZ4B2RLSaWTkriCB3imis0KL1FzOBbnQerJ1GnO2dc5YdA@mail.gmail.com> <CACpH0McBFG5qYNZDrp=1ubW49bnj%2BPsFPBbhUDJpO4Hgsccs2A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The stock igb driver binds to all cores, so with multiple igbs you have multiple nics binding to the same cores. I suppose that might create issues in a lagg setup. Try 1 queue and/or comment out the bind code. BC ________________________________ From: Zaphod Beeblebrox <zbeeble@gmail.com> To: Freddie Cash <fjwcash@gmail.com> Cc: Steve Read <steve.read@netasq.com>; freebsd-net <freebsd-net@freebsd.org> Sent: Friday, August 2, 2013 5:41 PM Subject: Re: Intel 4-port ethernet adaptor link aggregation issue On several machines with large numbers of IGBx interfaces, I've found that hw.igb.enable_msix=0 is necessary to ensure proper operation. On Fri, Aug 2, 2013 at 11:49 AM, Freddie Cash <fjwcash@gmail.com> wrote: > On Fri, Aug 2, 2013 at 12:36 AM, Steve Read <steve.read@netasq.com> wrote: > > > On 01.08.2013 20:07, Joe Moog wrote: > > > >> We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC > >> installed, model I350-T4 (manufactured May of 2013). We're trying to > bind > >> the 4 ports on this NIC together into a single lagg port, connected > LACP to > >> a distribution switch (Cisco 4900-series). We are able to successfully > bind > >> the 2 on-board ethernet ports to a single lagg, however the NIC is not > so > >> cooperative. At first we thought we had a bad NIC, but a replacement has > >> not fixed the issue. We are thinking there may be a driver limitation > with > >> these Intel ethernet NICs when attempting to bind more than 2 ports to a > >> lagg. > >> > >> FreeBSD version: > >> FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012 > >> > >> rc.conf: > >> # LINK AGGREGATION > >> ifconfig_igb2="UP" > >> ifconfig_igb3="UP" > >> ifconfig_igb4="UP" > >> ifconfig_igb5="UP" > >> cloned_interfaces="lagg0" > >> ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 > >> laggport igb5" > >> ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0" > >> > > > > > Am I the only one who noticed that you replaced the value of > > $ifconfig_lagg0 that specifies the proto and the ports with one that > > specifies just the address? > > > > Good catch! > > > > Merge the two ifconfig_lagg0 lines into one, and it will work infinitely > > better, or at least no worse. > > > > ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 > > laggport igb5 inet 192.168.1.14 netmask 255.255.255.0" > > > > Or, if you want to keep them split into two parts (initialise lagg0, then > add IP): > > create_args_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 > laggport igb5" > > ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0" > > create_args_* are run first, then ifconfig_* are run. I like this setup, > as it separates "create and initialise" from "configure" for cloned/virtual > interfaces like vlans, laggs, etc. > > > -- > Freddie Cash > fjwcash@gmail.com > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 3 04:21:11 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DC3FC91F for <freebsd-net@freebsd.org>; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70F68271E for <freebsd-net@freebsd.org>; Sat, 3 Aug 2013 04:21:11 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id n11so64437wgh.0 for <freebsd-net@freebsd.org>; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s 120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh]GFt85uIBlAKRTv4+LbgV08TrqISup/JA7E8IB/KI8=; b=awbPxtrTUsqc9OKXMugJWGiYLV0V4XPlCMbF2OCUMrGYFqHurs1Pjm+P/xzsCzb9FL lFjypRQB0ABJuHHsjHn9bxHffUM14S7DMqFbFnH+gRRDhBhfMZm5b5UXV95bxy0SnrIZ 2tHAyB6m+ueYMWbpT6mNeCKxNe5p8aoB9IDEVZ4hgv/efdvxINzwgmhxZUOQzQxRlA60 WrBCy0xDfMWidTUjORWCymoSxaiDzMylCeOGVtll3aL9CJ8YKN2Yj08G6beTI7RR8NIJ xFTV2I+FIKWOnbntMvZSz3pD/JLSRkXhwlqiZNXHy6SV9Vd+vKLtwFvC34HjjIPaZ/pg NkUQ=MIME-Version: 1.0 X-Received: by 10.194.11.67 with SMTP id o3mr7093699wjb.0.1375503669850; Fri, 02 Aug 2013 21:21:09 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Fri, 2 Aug 2013 21:21:09 -0700 (PDT) In-Reply-To: <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> References: <B966242F-A52D-43F7-A001-99942D53339E@ebureau.com> <51FB617E.2090904@netasq.com> <CAOjFWZ4B2RLSaWTkriCB3imis0KL1FzOBbnQerJ1GnO2dc5YdA@mail.gmail.com> <CACpH0McBFG5qYNZDrp=1ubW49bnj+PsFPBbhUDJpO4Hgsccs2A@mail.gmail.com> <1375486527.35304.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 2 Aug 2013 21:21:09 -0700 X-Google-Sender-Auth: hsEA0hut4Bn947cA9Kr4fiWaaGM Message-ID: <CAJ-Vmok=dhaA+TMJCgzck5Nhgi-qMTLtiQvrYXN0UPbVGvjHcw@mail.gmail.com> Subject: Re: Intel 4-port ethernet adaptor link aggregation issue From: Adrian Chadd <adrian@freebsd.org> To: Barney Cordoba <barney_cordoba@yahoo.com> Content-Type: text/plain; charset=ISO-8859-1 Cc: Steve Read <steve.read@netasq.com>, Zaphod Beeblebrox <zbeeble@gmail.com>, Freddie Cash <fjwcash@gmail.com>, freebsd-net <freebsd-net@freebsd.org> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 03 Aug 2013 04:21:11 -0000 On 2 August 2013 16:35, Barney Cordoba <barney_cordoba@yahoo.com> wrote: > The stock igb driver binds to all cores, so with multiple igbs you have multiple > nics binding to the same cores. I suppose that might create issues in a lagg setup. > Try 1 queue and/or comment out the bind code. > I have thrashed the hell out of 2-port ixgbe and 4-port chelsio (cxgbe) on 4-core device all with lagg. All is great. There's apparently some more igb improvements coming in the pipeline. Fear not! -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1375486527.35304.YahooMailNeo>