Date: Fri, 22 May 2015 11:48:23 -0500 From: Guy Helmer <guy.helmer@gmail.com> To: Eric Joyner <erj@freebsd.org> Cc: Christopher Forgeron <csforgeron@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: FreeBSD 9.3 - Intel X520-SR2 stops passing packets Message-ID: <A4442208-0711-4799-9483-44C87FC6412F@gmail.com> In-Reply-To: <F545D806-70F7-4B7F-826F-55BDB65BD129@gmail.com> References: <C8B78CF4-184C-4BFE-B848-37A91A3B696B@gmail.com> <CAB2_NwBmpHZZ03wht55=kVHdGk4mVKpFCcmcT=sJ12y7OJSXtA@mail.gmail.com> <BD7871A4-69C2-49EB-89D4-9EB741E2EB2B@gmail.com> <CA%2Bb0zg_0gvmHbVjLyBhA-yiX83ecbDLZz2nmqSLbDwVuXOURDQ@mail.gmail.com> <F545D806-70F7-4B7F-826F-55BDB65BD129@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On May 22, 2015, at 10:21 AM, Guy Helmer <guy.helmer@gmail.com> wrote:
>=20
>=20
>> On May 21, 2015, at 6:09 PM, Eric Joyner <erj@freebsd.org =
<mailto:erj@freebsd.org>> wrote:
>>=20
>> Are there any log messages printed out by the driver? The sysctls =
don't really look out of the ordinary, other than the number of sub-64 =
byte packets.
>=20
> Not that I could tell =E2=80=94 grep of /var/log/messages and the text =
from the rotated messages.*.gz logs yielded nothing.
>=20
> Guy
This is interesting - I was not looking at the Ierrs or Idrop columns of =
netstat before, but as the problem has recurred, I=E2=80=99ve found =
it=E2=80=99s dropping nearly all packets received on the ix interfaces:
> netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop =
Opkts Oerrs Coll
ix0 1500 <Link#3> 90:e2:ba:35:aa:8c 8310484539 3913 16255003933 =
0 0 0
ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - =
0 - -
ix1 1500 <Link#4> 90:e2:ba:35:aa:8d 6897310372 58387379 =
8207969523 0 0 0
ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - =
0 - -
> netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop =
Opkts Oerrs Coll
ix0 1500 <Link#3> 90:e2:ba:35:aa:8c 8310484538 3920 16299121753 =
0 0 0
ix0 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - =
0 - -
ix1 1500 <Link#4> 90:e2:ba:35:aa:8d 6897310369 58470675 =
8237532419 0 0 0
ix1 1500 fe80::92e2:ba fe80::92e2:baff:f 0 - - =
0 - -
# vmstat -z =E2=80=94 related to mbufs:
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
mbuf_packet: 256, 12982305, 38834, =
5203,9171090979,196374, 0
mbuf: 256, 12982305, 3, 167430,8518885166, 0, =
0
mbuf_cluster: 2048, 262144, 44037, =
9849,70443362,399118,892109
mbuf_jumbo_page: 4096, 1014242, 0, 4619, 9721759, 0, =
0
mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
# vmstat -z | grep mbuf
mbuf_packet: 256, 12982305, 38834, =
5203,9171103748,196374, 0
mbuf: 256, 12982305, 5, 167428,8518902143, 0, =
0
mbuf_cluster: 2048, 262144, 44037, =
9849,70443362,399118,892109
mbuf_jumbo_page: 4096, 1014242, 0, 4747, 9721896, 0, =
0
mbuf_jumbo_9k: 9216, 300516, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 169040, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
No interface errors logging in /var/log/messages=E2=80=A6
ifconfig down / ifconfig up on the ix0 and ix1 interfaces gets traffic =
flowing again.
Guy
>=20
>>=20
>> On Thu, May 21, 2015 at 9:10 AM Guy Helmer <guy.helmer@gmail.com =
<mailto:guy.helmer@gmail.com>> wrote:
>>=20
>> > On May 21, 2015, at 8:52 AM, Christopher Forgeron =
<csforgeron@gmail.com <mailto:csforgeron@gmail.com>> wrote:
>> >
>> > A few things:
>> >
>> > 1) How long before you have this behaviour?
>> >
>> > 2) What's the output of 'netstat -m' when you have the problem?
>> >
>> > 3) What is your MTU set to, and do you have TSO on or off?
>> >
>> > On Thu, May 21, 2015 at 10:33 AM, Guy Helmer <guy.helmer@gmail.com =
<mailto:guy.helmer@gmail.com> <mailto:guy.helmer@gmail.com =
<mailto:guy.helmer@gmail.com>>> wrote:
>> > I=E2=80=99ve noticed that there have been reports of problems with =
Intel X520-SR2 network interfaces stopping working. I think I=E2=80=99m =
seeing a similar issue where the 10Gb interfaces stop receiving traffic =
(they=E2=80=99re being used in promiscuous mode to sniff traffic from a =
tap). ifconfig shows the interfaces are still active and the links are =
OK. ifconfig down/up restores activity. I=E2=80=99ve changed =
hw.intr_storm_threshold=3D8000 but I couldn=E2=80=99t tell if the =
interrupt storm threshold had been triggered at the time the interfaces =
stopped passing traffic.
>>=20
>> It seems to run from hours to days without problems.
>>=20
>> I don=E2=80=99t have the output of =E2=80=9Cnetstat -m=E2=80=9D =
available, but it did not indicate any mbuf or cluster allocation =
failures. No jumbo clusters (4k, 9k, or 16k) were allocated.
>>=20
>> MTU is 1500. TSO is =E2=80=9Con=E2=80=9D but would seem to be =
irrelevant =E2=80=94 no packets are transmitted out of these interfaces =
(verified using =E2=80=9Cnetstat -i=E2=80=9D).
>>=20
>> Thanks,
>> Guy
>> _______________________________________________
>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net =
<http://lists.freebsd.org/mailman/listinfo/freebsd-net>;
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org =
<mailto:freebsd-net-unsubscribe@freebsd.org>"
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A4442208-0711-4799-9483-44C87FC6412F>