From owner-freebsd-net@FreeBSD.ORG Fri May 22 16:48:14 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 29F2A1D4; Fri, 22 May 2015 16:48:14 +0000 (UTC) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E19231D87; Fri, 22 May 2015 16:48:13 +0000 (UTC) Received: by iepj10 with SMTP id j10so35338287iep.3; Fri, 22 May 2015 09:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=yHntIgapTmjXTMIdN9mfQCX3bIfmnwkTzUdsKkKvvBI=; b=jR8M2pf5BeTHKdpr1q09V4+QXSFieYU/ibSKO1w+lKpDbskRnN/3F28OA3+QItu4a2 NL76h2seDvUjtsMg/JwFYAw7R6wS1O2tJg99OHdreqYiyrOCi2A4aBhk3NAwRev1Oksh HIC+qk5tSd2wqqYUug/dgQbLuk9xpfNIHQbMqbWmvIikBbKSPvuwQv1omjyAM5I/0KEN z4ta4mPtVBf2eLlmILTIwG6j3u5XrDoNoMpvLO2ke7s9BCoTlrXp8lnwtfv4U4v7ML90 s3zpuiSgPpYHJM4FiYiT82OGYXbbLEIk/jTDMA2qedN+780nW2YgqwERJ5AUMiGaLq4n /o2A== X-Received: by 10.107.34.140 with SMTP id i134mr10136539ioi.88.1432313293176; Fri, 22 May 2015 09:48:13 -0700 (PDT) Received: from [172.22.132.117] ([192.119.231.58]) by mx.google.com with ESMTPSA id k37sm2131599iod.39.2015.05.22.09.48.11 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 22 May 2015 09:48:12 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: FreeBSD 9.3 - Intel X520-SR2 stops passing packets From: Guy Helmer In-Reply-To: Date: Fri, 22 May 2015 11:48:23 -0500 Cc: Christopher Forgeron , FreeBSD Net Message-Id: References: To: Eric Joyner X-Mailer: Apple Mail (2.2098) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 16:48:14 -0000 > On May 22, 2015, at 10:21 AM, Guy Helmer wrote: >=20 >=20 >> On May 21, 2015, at 6:09 PM, Eric Joyner > wrote: >>=20 >> Are there any log messages printed out by the driver? The sysctls = don't really look out of the ordinary, other than the number of sub-64 = byte packets. >=20 > Not that I could tell =E2=80=94 grep of /var/log/messages and the text = from the rotated messages.*.gz logs yielded nothing. >=20 > Guy This is interesting - I was not looking at the Ierrs or Idrop columns of = netstat before, but as the problem has recurred, I=E2=80=99ve found = it=E2=80=99s dropping nearly all packets received on the ix interfaces: > netstat -i Name Mtu Network Address Ipkts Ierrs Idrop = Opkts Oerrs Coll ix0 1500 90:e2:ba:35:aa:8c 8310484539 3913 16255003933 = 0 0 0 ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - = 0 - - ix1 1500 90:e2:ba:35:aa:8d 6897310372 58387379 = 8207969523 0 0 0 ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - = 0 - - > netstat -i Name Mtu Network Address Ipkts Ierrs Idrop = Opkts Oerrs Coll ix0 1500 90:e2:ba:35:aa:8c 8310484538 3920 16299121753 = 0 0 0 ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - = 0 - - ix1 1500 90:e2:ba:35:aa:8d 6897310369 58470675 = 8237532419 0 0 0 ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - = 0 - - # vmstat -z =E2=80=94 related to mbufs: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 12982305, 38834, = 5203,9171090979,196374, 0 mbuf: 256, 12982305, 3, 167430,8518885166, 0, = 0 mbuf_cluster: 2048, 262144, 44037, = 9849,70443362,399118,892109 mbuf_jumbo_page: 4096, 1014242, 0, 4619, 9721759, 0, = 0 mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 # vmstat -z | grep mbuf mbuf_packet: 256, 12982305, 38834, = 5203,9171103748,196374, 0 mbuf: 256, 12982305, 5, 167428,8518902143, 0, = 0 mbuf_cluster: 2048, 262144, 44037, = 9849,70443362,399118,892109 mbuf_jumbo_page: 4096, 1014242, 0, 4747, 9721896, 0, = 0 mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 No interface errors logging in /var/log/messages=E2=80=A6 ifconfig down / ifconfig up on the ix0 and ix1 interfaces gets traffic = flowing again. Guy >=20 >>=20 >> On Thu, May 21, 2015 at 9:10 AM Guy Helmer > wrote: >>=20 >> > On May 21, 2015, at 8:52 AM, Christopher Forgeron = > wrote: >> > >> > A few things: >> > >> > 1) How long before you have this behaviour? >> > >> > 2) What's the output of 'netstat -m' when you have the problem? >> > >> > 3) What is your MTU set to, and do you have TSO on or off? >> > >> > On Thu, May 21, 2015 at 10:33 AM, Guy Helmer >> wrote: >> > I=E2=80=99ve noticed that there have been reports of problems with = Intel X520-SR2 network interfaces stopping working. I think I=E2=80=99m = seeing a similar issue where the 10Gb interfaces stop receiving traffic = (they=E2=80=99re being used in promiscuous mode to sniff traffic from a = tap). ifconfig shows the interfaces are still active and the links are = OK. ifconfig down/up restores activity. I=E2=80=99ve changed = hw.intr_storm_threshold=3D8000 but I couldn=E2=80=99t tell if the = interrupt storm threshold had been triggered at the time the interfaces = stopped passing traffic. >>=20 >> It seems to run from hours to days without problems. >>=20 >> I don=E2=80=99t have the output of =E2=80=9Cnetstat -m=E2=80=9D = available, but it did not indicate any mbuf or cluster allocation = failures. No jumbo clusters (4k, 9k, or 16k) were allocated. >>=20 >> MTU is 1500. TSO is =E2=80=9Con=E2=80=9D but would seem to be = irrelevant =E2=80=94 no packets are transmitted out of these interfaces = (verified using =E2=80=9Cnetstat -i=E2=80=9D). >>=20 >> Thanks, >> Guy >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net = >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org = " >=20