From owner-freebsd-net@FreeBSD.ORG Fri Feb 28 02:05:06 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B29B9A3 for ; Fri, 28 Feb 2014 02:05:06 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70A1C10BB for ; Fri, 28 Feb 2014 02:05:06 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1WJCox-0007pn-Um for freebsd-net@freebsd.org; Fri, 28 Feb 2014 03:05:03 +0100 Received: from tempe0.bbox.io ([24.249.180.233]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 28 Feb 2014 03:05:03 +0100 Received: from kevin.bowling by tempe0.bbox.io with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 28 Feb 2014 03:05:03 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-net@freebsd.org From: Kevin Bowling Subject: Re: FreeBSD 10 network flapping, ix driver unreliable? Date: Thu, 27 Feb 2014 19:02:01 -0700 Lines: 63 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: tempe0.bbox.io User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Thunderbird/27.0 In-Reply-To: X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 02:05:06 -0000 On 2/15/2014 1:14 PM, Kevin Bowling wrote: > Hi, > > I have FreeBSD 10.0-RELEASE installed on two Dell C6100 nodes. Each > node has an Intel X520-DA2 dual port 10gig card. One of the ports on > each go to a switch using direct attach coaxial cables. The other port > is directly connected between the two nodes (think crossover in twisted > pair terminology) again using direct attach coaxial cables. > > On both machines, and on both ports (including the "crossover"), the > links flap several times per day. > > I've pasted the output of lspci -vv and dmesg here: > https://gist.github.com/kev009/9024442 > > There's nothing outstanding about the setup otherwise. I suspected some > interaction with the switch initially but the "crossover" has eliminated > that suspicion. > > It seems the ix driver is not very reliable under common conditions, > i.e. https://forums.freebsd.org/viewtopic.php?f=7&t=44570 and a search > of this list. Any recommendations or tests? > > Regards, > Kevin Bowling > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > After descending a rather dark rabbit hole, I'm pleased to have found a simple solution! On some of these cards, there are known firmware problems. The driver sometimes tries to compensate, but these code paths probably receive far less testing and look pretty fragile. It seems some version of NIC firmware are particularly flaky with DA cables. Some spam in the Linux dmesg led me to this, which does not appear to be in the FreeBSD ixgbe driver, although I did not stay in Linux long enough to see if it fully fixed the problem: http://markmail.org/message/ivsjxoyfbvzv7mvo Instead, I found a way to update the microcode. My card and server are Dell and I was able to use this live cd to do the firmware upgrade: http://linux.dell.com/files/openmanage-contributions/om-firmware-live/ which applies a NIC firmware package like http://www.dell.com/support/drivers/us/en/19/driverdetails?driverid=HKK1W Both of my systems appear to be stable; I'll comment if there are issues over the next few days. Intel is less than forthcoming about these microcode updates; I'm not sure if the preboot code (https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3591&DwnldID=19186&ProductFamily=Network+Connectivity&ProductLine=Intel%C2%AE+Server+Adapters&ProductProduct=Intel%C2%AE+Ethernet+Converged+Network+Adapter+X520+Series&lang=eng) contains the NIC microcode or if you must obtain it from a vendor. Regards, Kevin Bowling