Date: Fri, 13 Jan 2012 12:49:26 -0500 From: Iordan Iordanov <iordan@cdf.toronto.edu> To: freebsd-bugs@freebsd.org Cc: CDF Admin <admin@cdf.toronto.edu> Subject: Intel x520-t2: interrupt storm detected + throttling Message-ID: <4F106EA6.3000003@cdf.toronto.edu>
next in thread | raw e-mail | index | archive | help
Hello,
I decided to send a separate message about another problem that we
witnessed while testing 10GbaseT connectivity between a pair of Intel
x520-t2 interconnected with cat6 cable.
While we were testing we got a multitude of kernel messages of this form:
interrupt storm detected on "irq271:"; throttling interrupt source
interrupt storm detected on "irq271:"; throttling interrupt source
interrupt storm detected on "irq271:"; throttling interrupt source
An easy way for us to reproduce the problem was this. Start an nc
process listening on one machine:
fw4# nc -l 8080 > /dev/null
Then, start piping from /dev/zero into nc from the other machine:
fw3# dd if=/dev/zero bs=4096k | nc 192.168.1.200 8080
Almost immediately, on the server-side, we start getting the messages.
The client-side is silent. When I reversed the client/server roles, the
scenario was repeated the other way around to prove to myself it's the
server that reports the issue. The transfer speed for a single process
over TCP appears to be about 240 megabytes/s or about 2 gigabits/s.
However, with multiple nc processes we are able to push this to about 4
gigabits/s (similar to our iperf tests).
A curious, and probably easy to explain fact is that when doing
transfers over UDP (the -u flag to nc), we do not get the "interrupt
storm" messages, and indeed, the interrupt rate in "top" is 1/2 that of
when we do the TCP test under the same conditions.
More information about our hardware is:
Motherboard: Supermicro H8DGU-F
CPU: AMD 6128 8-core
Memory: 16GB DDR3
NICs: Intel x520-t2 10GbaseT adapters
An ifconfig ix1 output for one of the machines:
fw4# ifconfig ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:d6:19:51
inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
Many thanks for any help/action on this!
Sincerely,
Iordan Iordanov
CDF System Administrator
University of Toronto
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F106EA6.3000003>