Date: Fri, 22 May 2015 11:48:23 -0500 From: Guy Helmer <guy.helmer@gmail.com> To: Eric Joyner <erj@freebsd.org> Cc: Christopher Forgeron <csforgeron@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: FreeBSD 9.3 - Intel X520-SR2 stops passing packets Message-ID: <A4442208-0711-4799-9483-44C87FC6412F@gmail.com> In-Reply-To: <F545D806-70F7-4B7F-826F-55BDB65BD129@gmail.com> References: <C8B78CF4-184C-4BFE-B848-37A91A3B696B@gmail.com> <CAB2_NwBmpHZZ03wht55=kVHdGk4mVKpFCcmcT=sJ12y7OJSXtA@mail.gmail.com> <BD7871A4-69C2-49EB-89D4-9EB741E2EB2B@gmail.com> <CA%2Bb0zg_0gvmHbVjLyBhA-yiX83ecbDLZz2nmqSLbDwVuXOURDQ@mail.gmail.com> <F545D806-70F7-4B7F-826F-55BDB65BD129@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On May 22, 2015, at 10:21 AM, Guy Helmer <guy.helmer@gmail.com> wrote: > > >> On May 21, 2015, at 6:09 PM, Eric Joyner <erj@freebsd.org <mailto:erj@freebsd.org>> wrote: >> >> Are there any log messages printed out by the driver? The sysctls don't really look out of the ordinary, other than the number of sub-64 byte packets. > > Not that I could tell — grep of /var/log/messages and the text from the rotated messages.*.gz logs yielded nothing. > > Guy This is interesting - I was not looking at the Ierrs or Idrop columns of netstat before, but as the problem has recurred, I’ve found it’s dropping nearly all packets received on the ix interfaces: > netstat -i Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 1500 <Link#3> 90:e2:ba:35:aa:8c 8310484539 3913 16255003933 0 0 0 ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - 0 - - ix1 1500 <Link#4> 90:e2:ba:35:aa:8d 6897310372 58387379 8207969523 0 0 0 ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - 0 - - > netstat -i Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 1500 <Link#3> 90:e2:ba:35:aa:8c 8310484538 3920 16299121753 0 0 0 ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - 0 - - ix1 1500 <Link#4> 90:e2:ba:35:aa:8d 6897310369 58470675 8237532419 0 0 0 ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - 0 - - # vmstat -z — related to mbufs: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 12982305, 38834, 5203,9171090979,196374, 0 mbuf: 256, 12982305, 3, 167430,8518885166, 0, 0 mbuf_cluster: 2048, 262144, 44037, 9849,70443362,399118,892109 mbuf_jumbo_page: 4096, 1014242, 0, 4619, 9721759, 0, 0 mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 # vmstat -z | grep mbuf mbuf_packet: 256, 12982305, 38834, 5203,9171103748,196374, 0 mbuf: 256, 12982305, 5, 167428,8518902143, 0, 0 mbuf_cluster: 2048, 262144, 44037, 9849,70443362,399118,892109 mbuf_jumbo_page: 4096, 1014242, 0, 4747, 9721896, 0, 0 mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 No interface errors logging in /var/log/messages… ifconfig down / ifconfig up on the ix0 and ix1 interfaces gets traffic flowing again. Guy > >> >> On Thu, May 21, 2015 at 9:10 AM Guy Helmer <guy.helmer@gmail.com <mailto:guy.helmer@gmail.com>> wrote: >> >> > On May 21, 2015, at 8:52 AM, Christopher Forgeron <csforgeron@gmail.com <mailto:csforgeron@gmail.com>> wrote: >> > >> > A few things: >> > >> > 1) How long before you have this behaviour? >> > >> > 2) What's the output of 'netstat -m' when you have the problem? >> > >> > 3) What is your MTU set to, and do you have TSO on or off? >> > >> > On Thu, May 21, 2015 at 10:33 AM, Guy Helmer <guy.helmer@gmail.com <mailto:guy.helmer@gmail.com> <mailto:guy.helmer@gmail.com <mailto:guy.helmer@gmail.com>>> wrote: >> > I’ve noticed that there have been reports of problems with Intel X520-SR2 network interfaces stopping working. I think I’m seeing a similar issue where the 10Gb interfaces stop receiving traffic (they’re being used in promiscuous mode to sniff traffic from a tap). ifconfig shows the interfaces are still active and the links are OK. ifconfig down/up restores activity. I’ve changed hw.intr_storm_threshold=8000 but I couldn’t tell if the interrupt storm threshold had been triggered at the time the interfaces stopped passing traffic. >> >> It seems to run from hours to days without problems. >> >> I don’t have the output of “netstat -m” available, but it did not indicate any mbuf or cluster allocation failures. No jumbo clusters (4k, 9k, or 16k) were allocated. >> >> MTU is 1500. TSO is “on” but would seem to be irrelevant — no packets are transmitted out of these interfaces (verified using “netstat -i”). >> >> Thanks, >> Guy >> _______________________________________________ >> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net <http://lists.freebsd.org/mailman/listinfo/freebsd-net>; >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org <mailto:freebsd-net-unsubscribe@freebsd.org>" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A4442208-0711-4799-9483-44C87FC6412F>