Date: Sat, 1 Sep 2012 22:44:00 -0400 From: Andy Young <ayoung@mosaicarchive.com> To: freebsd-hardware@freebsd.org Subject: Re: Load testing knocks out network Message-ID: <CAHMRaQdv0Wd3u79cxcNBa=aC%2Bv3L3XMEirDtxQ7aKFF0JxxSag@mail.gmail.com> In-Reply-To: <CAHMRaQfLGY%2BYeDkG7K1GJQ-pmAi6rgT6-gthKQ3j7rSyzr7qVA@mail.gmail.com> References: <CAHMRaQfLGY%2BYeDkG7K1GJQ-pmAi6rgT6-gthKQ3j7rSyzr7qVA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I read through the driver man page, which is a great source of information.
I see I'm using the Intel igb driver and it supports three tunables. Could
I have exceeded the number of receive descriptors? What would the effect of
this number being too low be? What about the Adaptive Interrupt Moderation?
To clarify, I was simulating about 800 users simultaneously uploading files
when the crash occurred.
Thanks for any help or insights!!
Andy
NAME
igb -- Intel(R) PRO/1000 PCI Express Gigabit Ethernet adapter driver
LOADER TUNABLES
Tunables can be set at the loader(8) prompt before booting the kernel
or
stored in loader.conf(5).
hw.igb.rxd
Number of receive descriptors allocated by the driver. The
default value is 256. The minimum is 80, and the maximum is
4096.
hw.igb.txd
Number of transmit descriptors allocated by the driver. The
default value is 256. The minimum is 80, and the maximum is
4096.
hw.igb.enable_aim
If set to 1, enable Adaptive Interrupt Moderation. The default
is to enable Adaptive Interrupt Moderation.
On Sat, Sep 1, 2012 at 4:14 PM, Andy Young <ayoung@mosaicarchive.com> wrote:
> Last night one our servers went offline while I was load testing it. When
> I got to the datacenter to check on it, the server seemed perfectly fine.
> Everything was running on it, there were no panics or any other sign of a
> hard crash. The only problem is the network was unreachable. I couldn't
> connect to the box even from a laptop directly attached to the ethernet
> port. I couldn't connect to anything from the box either. It was if the
> network controller had seized up. I restarted netif and it didn't make a
> difference. Rebooting the machine however, solved the issue and everything
> went back to working great. I restarted the load testing and reproduced the
> problem twice more this morning so at least its repeatable. It feels like a
> network controller / driver issue to me for a couple reasons. First, the
> problem affects the entire system. We're running FreeBSD 9 with about a
> half dozen jails. Most of the jails are running Apache but the one I was
> load testing was running Jetty. However, if it was my application code
> crashing I would expect the problem to at least be isolated to the jail
> that hosts it. Instead, the entire machine and all jails in it lose access
> to the network.
>
> Apart from not being able to access the network, I don't see any other
> signs of problems. This is the first major problem I've had to debug in
> FreeBSD so I'm not a debugging expert by any means. There are no error
> messages in /var/log/messages or dmesg apart from syslogd not being able to
> reach the network. If anyone has ideas on where I can look for more
> evidence of what is going wrong, I would really appreciate it.
>
> We're running FreeBSD 9.0-RELEASE-p3. The network controller is a Intel(R)
> PRO/1000 Network Connection version - 2.2.5 configured with 6 ips using
> aliases, five of which are used for jails.
>
> Thank you for the help!!
>
> Andy
>
>
>
--
Andrew Young
Mosaic Storage Systems, Inc
http://www.mosaicarchive.com/
Follow us on:
Twitter <https://twitter.com/#!/MosaicArchive>,
Facebook<http://www.facebook.com/MosaicArchive>;
, Google Plus<https://plus.google.com/b/102077382489657821832/https://plus.google.com/b/104681960235222388167/104681960235222388167/posts>;
, Pinterest <http://pinterest.com/mosaicarchive/>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHMRaQdv0Wd3u79cxcNBa=aC%2Bv3L3XMEirDtxQ7aKFF0JxxSag>