From owner-freebsd-net@FreeBSD.ORG Mon Aug 5 19:33:34 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2118E8C9 for ; Mon, 5 Aug 2013 19:33:34 +0000 (UTC) (envelope-from joemoog@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) by mx1.freebsd.org (Postfix) with ESMTP id EAE732299 for ; Mon, 5 Aug 2013 19:33:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id A8F8737C6177; Mon, 5 Aug 2013 14:33:23 -0500 (CDT) X-Virus-Scanned: amavisd-new at ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L1nrVzYuschY; Mon, 5 Aug 2013 14:33:20 -0500 (CDT) Received: from nail.office.ebureau.com (nail.office.ebureau.com [10.10.20.23]) by internet06.ebureau.com (Postfix) with ESMTPSA id 7511B37C6104; Mon, 5 Aug 2013 14:32:49 -0500 (CDT) Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1786.1\)) Subject: Re: Intel 4-port ethernet adaptor link aggregation issue From: Joe Moog In-Reply-To: <20130801231643.GB94127@funkthat.com> Date: Mon, 5 Aug 2013 14:32:48 -0500 Message-Id: References: <2A0C085A-1AAF-42D7-867B-6CDD1143B4AC@ebureau.com> <20130801231643.GB94127@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1786.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net , Ryan Stone X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Aug 2013 19:33:34 -0000 On Aug 1, 2013, at 6:16 PM, John-Mark Gurney wrote: > Joe Moog wrote this message on Thu, Aug 01, 2013 at 17:14 -0500: >> On Aug 1, 2013, at 4:27 PM, Joe Moog wrote: >>=20 >>> On Aug 1, 2013, at 3:55 PM, Ryan Stone wrote: >>>=20 >>>> Have you tried using only two ports, but both from the NIC? My = suspicion would be that the problem is in the lagg's handling of more = than 2 ports rather than the driver, especially given that it is the igb = driver in all cases. >>>=20 >>> Ryan: >>>=20 >>> We have done this successfully with two ports on the NIC, on another = hardware-identical host. That said, it is entirely possible that this is = a shortcoming of lagg.=20 >>>=20 >>> Can you think of any sort of workaround? Our desired implementation = really requires the inclusion of all 4 ports in the lagg. Failing this = we're looking at the likelihood of 10G ethernet, but with that comes = significant overhead, both cost and administration (before anybody tries = to force the cost debate, remember that there are 10G router modules and = 10G-capable distribution switches involved, never mind the cabling and = SFPs -- it's not just a $600 10G card for the host). I'd like to defer = that requirement as long as possible. 4 aggregated gig ports would serve = us perfectly well for the near-term. >>>=20 >>> Thanks >>>=20 >>> Joe >>=20 >> UPDATE: After additional testing, I'm beginning to suspect the igb = driver. With our setup, ifconfig identifies all the ethernet ports as = igb(0-5). I configured igb0 with a single static IP address (say, = 192.168.1.10), and was able to connect to the host administratively. = While connected, I enabled another port as a second standalone port, = again with a unique address (say, 192.168.1.20), and was able to access = the host via that interface as well. The problem arises when we attempt = to similarly add a third interface to the mix -- and it doesn't seem to = matter what interface(s) we use, or in what order we activate them. = Always on the third interface, that third interface fails to respond = despite showing "active" both in ifconfig and on the switch. >=20 > Can you show an ifconfig -au from the host when it fails, and which = was > the third interface that you added? Above, you talk about adding ips = in > the same subnet to different interfaces, which with modern switchs can > cause issues with which port to deliver packets, etc. >=20 > Do you have any firewalling enabled on the host? >=20 There are no firewalls enabled on the host. I don't know that I see the switch as being the weak point in this setup = as we have been very successful multihoming boxes with these switches = for a variety of other purposes. I will collect and forward "ifconfig = -au" output from the host in a couple of days, as we have had to fall = back on the 2-port lagg to get this particular host in service until = such time the 4-port lagg issue can be resolved. We will be setting up = another hardware-identical host in a lab for further testing and info = gathering. Thanks Joe