Date: Fri, 13 Jan 2012 10:04:08 -0800 From: Jack Vogel <jfvogel@gmail.com> To: Iordan Iordanov <iordan@cdf.toronto.edu> Cc: freebsd-bugs@freebsd.org, CDF Admin <admin@cdf.toronto.edu> Subject: Re: Intel x520-t2: interrupt storm detected + throttling Message-ID: <CAFOYbckTT4Nz_fghCGuYqGz7hL82EepjACAJ2XUdxUrqDPN2Qg@mail.gmail.com> In-Reply-To: <4F106EA6.3000003@cdf.toronto.edu> References: <4F106EA6.3000003@cdf.toronto.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
10G pushes interrupts into a realm that the defaults are unprepared for... specifically there is a storm threshold that you will exceed: do a `sysctl hw.intr_storm_threshold` its too low. My suggestion is to set that to 0, this disables the test. But otherwise you will need to increase it high enough to eliminate the throttling. Jack On Fri, Jan 13, 2012 at 9:49 AM, Iordan Iordanov <iordan@cdf.toronto.edu>wrote: > Hello, > > I decided to send a separate message about another problem that we > witnessed while testing 10GbaseT connectivity between a pair of Intel > x520-t2 interconnected with cat6 cable. > > While we were testing we got a multitude of kernel messages of this form: > > interrupt storm detected on "irq271:"; throttling interrupt source > interrupt storm detected on "irq271:"; throttling interrupt source > interrupt storm detected on "irq271:"; throttling interrupt source > > An easy way for us to reproduce the problem was this. Start an nc process > listening on one machine: > > fw4# nc -l 8080 > /dev/null > > Then, start piping from /dev/zero into nc from the other machine: > > fw3# dd if=/dev/zero bs=4096k | nc 192.168.1.200 8080 > > Almost immediately, on the server-side, we start getting the messages. The > client-side is silent. When I reversed the client/server roles, the > scenario was repeated the other way around to prove to myself it's the > server that reports the issue. The transfer speed for a single process over > TCP appears to be about 240 megabytes/s or about 2 gigabits/s. However, > with multiple nc processes we are able to push this to about 4 gigabits/s > (similar to our iperf tests). > > A curious, and probably easy to explain fact is that when doing transfers > over UDP (the -u flag to nc), we do not get the "interrupt storm" messages, > and indeed, the interrupt rate in "top" is 1/2 that of when we do the TCP > test under the same conditions. > > More information about our hardware is: > > Motherboard: Supermicro H8DGU-F > CPU: AMD 6128 8-core > Memory: 16GB DDR3 > NICs: Intel x520-t2 10GbaseT adapters > > An ifconfig ix1 output for one of the machines: > > fw4# ifconfig ix1 > ix1: flags=8843<UP,BROADCAST,**RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > options=401bb<RXCSUM,TXCSUM,**VLAN_MTU,VLAN_HWTAGGING,JUMBO_** > MTU,VLAN_HWCSUM,TSO4,VLAN_**HWTSO> > ether 00:1b:21:d6:19:51 > inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255 > nd6 options=29<PERFORMNUD,**IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect (autoselect <full-duplex>) > status: active > > > Many thanks for any help/action on this! > > Sincerely, > > Iordan Iordanov > CDF System Administrator > University of Toronto > ______________________________**_________________ > freebsd-bugs@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-bugs<http://lists.freebsd.org/mailman/listinfo/freebsd-bugs>; > To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@**freebsd.org<freebsd-bugs-unsubscribe@freebsd.org> > " >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbckTT4Nz_fghCGuYqGz7hL82EepjACAJ2XUdxUrqDPN2Qg>