From owner-freebsd-bugs@FreeBSD.ORG Fri Jan 13 17:49:28 2012 Return-Path: Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 445571065732 for ; Fri, 13 Jan 2012 17:49:28 +0000 (UTC) (envelope-from iordan@cdf.toronto.edu) Received: from smtp.cdf.toronto.edu (smtp.cdf.utoronto.ca [128.100.31.106]) by mx1.freebsd.org (Postfix) with SMTP id 929F48FC1E for ; Fri, 13 Jan 2012 17:49:27 +0000 (UTC) Received: (qmail 874 invoked from network); 13 Jan 2012 12:49:27 -0500 Received: from australopithecus.cdf.toronto.edu (HELO ?128.100.31.7?) (iordan@128.100.31.7) by penguin.cdf.toronto.edu with SMTP; 13 Jan 2012 12:49:27 -0500 Message-ID: <4F106EA6.3000003@cdf.toronto.edu> Date: Fri, 13 Jan 2012 12:49:26 -0500 From: Iordan Iordanov Organization: University Of Toronto User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.25) Gecko/20111213 Thunderbird/3.1.17 MIME-Version: 1.0 To: freebsd-bugs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: CDF Admin Subject: Intel x520-t2: interrupt storm detected + throttling X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2012 17:49:28 -0000 Hello, I decided to send a separate message about another problem that we witnessed while testing 10GbaseT connectivity between a pair of Intel x520-t2 interconnected with cat6 cable. While we were testing we got a multitude of kernel messages of this form: interrupt storm detected on "irq271:"; throttling interrupt source interrupt storm detected on "irq271:"; throttling interrupt source interrupt storm detected on "irq271:"; throttling interrupt source An easy way for us to reproduce the problem was this. Start an nc process listening on one machine: fw4# nc -l 8080 > /dev/null Then, start piping from /dev/zero into nc from the other machine: fw3# dd if=/dev/zero bs=4096k | nc 192.168.1.200 8080 Almost immediately, on the server-side, we start getting the messages. The client-side is silent. When I reversed the client/server roles, the scenario was repeated the other way around to prove to myself it's the server that reports the issue. The transfer speed for a single process over TCP appears to be about 240 megabytes/s or about 2 gigabits/s. However, with multiple nc processes we are able to push this to about 4 gigabits/s (similar to our iperf tests). A curious, and probably easy to explain fact is that when doing transfers over UDP (the -u flag to nc), we do not get the "interrupt storm" messages, and indeed, the interrupt rate in "top" is 1/2 that of when we do the TCP test under the same conditions. More information about our hardware is: Motherboard: Supermicro H8DGU-F CPU: AMD 6128 8-core Memory: 16GB DDR3 NICs: Intel x520-t2 10GbaseT adapters An ifconfig ix1 output for one of the machines: fw4# ifconfig ix1 ix1: flags=8843 metric 0 mtu 1500 options=401bb ether 00:1b:21:d6:19:51 inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (autoselect ) status: active Many thanks for any help/action on this! Sincerely, Iordan Iordanov CDF System Administrator University of Toronto