From owner-freebsd-net@FreeBSD.ORG Tue Jan 24 17:17:15 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06AF41065672 for ; Tue, 24 Jan 2012 17:17:15 +0000 (UTC) (envelope-from mmarkowski@leon.pl) Received: from kameleon.leon.pl (kameleon.leon.pl [IPv6:2a02:c40:0:10::5bc3:8704]) by mx1.freebsd.org (Postfix) with ESMTP id BA35A8FC0A for ; Tue, 24 Jan 2012 17:17:14 +0000 (UTC) Received: by kameleon.leon.pl (Postfix, from userid 1080) id 2DABC358C5C; Tue, 24 Jan 2012 18:17:13 +0100 (CET) To: X-PHP-Script: mail.leon.pl/index.php for 188.137.111.246 X-PHP-Originating-Script: 1080:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 24 Jan 2012 18:17:13 +0100 From: Marcin Markowski In-Reply-To: <529374128DC1B04D9D037911B8E8F0531095BC48@Exchange26.EDU.epsb.ca> References: <159d4fbce722663a84f3cea12da828a5@leon.pl><6A254A40-7DA5-4EFE-93C5-4E084F33B78A@gmail.com> <86d5416f7f55a71fb01fd86e9051d678@leon.pl> <529374128DC1B04D9D037911B8E8F0531095BC48@Exchange26.EDU.epsb.ca> Message-ID: X-Sender: mmarkowski@leon.pl User-Agent: Roundcube Webmail/0.6 Subject: RE: Performance problem using Intel X520-DA2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2012 17:17:15 -0000 On 24.01.2012 16:07, Kirk Davis wrote: >>-----Original Message----- >>On 24.01.2012 09:18, Nikolay Denev wrote: >>> On Jan 23, 2012, at 11:39 PM, Marcin Markowski wrote: >>> >>>> Hello, >>>> >>>> This message has been sent to freebsd-performance@ but got the >>>> information that should contact also with freebsd-net@. >>>> >>>> We use FreeBSD as sniffer (libpcap programs) and we experience >>>> performance problems when incoming traffic is greater than >>>> 7.5Gbps/s. >>>> If we check 'top' we see that first irq from network card is using >>>> 100% CPU. I've tested this on FreeBSD 8.2-RELEASE and 9.0-RELEASE >>>> (on >>>> 9.0 we can see also kernel thread named {ix0 que} using 100% CPU), >>>> and both systems behave the same. In logs we see also: >>>> interrupt storm detected on "irq268:"; throttling interrupt source >>>> >>>> Our server platform is Intel SR2600URBRP, 2x Xeon X5650, 6GB RAM >>>> and >>>> NIC Intel X520-DA2. >>>> >>>> I'm not sure if problem is with NIC or motherboard in SR2600URBRP, >>>> because everything is fine when we use other server configuration: >>>> Intel SR1630GP, 1x Xeon X3450, 8GB RAM, NIC X520-DA2 >>>> >>>> My /boot/loader.conf: >>>> kern.ipc.nmbclusters=262144 >>>> hw.ixgbe.rxd=2048 >>>> hw.ixgbe.txd=2048 >>>> hw.ixgbe.num_queues=16 >>>> >>>> /etc/sysctl.conf >>>> hw.intr_storm_threshold=10000 >>>> > > I just finished a bunch of performance tests on this card. In my > case I was trying to get as close to a full 10Gb/s as possible on a > Dell R710 but I haven't yet tried any sniffing. > > Have you tried turning on LRO on the interface ( ifconfig lro ). In > my case this made a big difference and I can now get 9.41Gb/s without > high CPU or interrupt storms. I am also using Jack's latest driver > downloaded from intel (version 2.4.4). Even the driver in 9.0 was > older. > > Here is what I have > /etc/sysctl.conf > # Increase the network buffers > kern.ipc.nmbclusters=262144 > kern.ipc.maxsockbuf=4194304 > hw.intr_storm_threshold=9000 > kern.ipc.nmbjumbop=262144 > > /boot/loader.conf > ixgbe_load="YES" > hw.ixgbe.txd=4096 > hw.ixgbe.rxd=4096 > > ---- Kirk Hi Kirk, I did not notice any change after turning on LRO. When checking stats I see that LRO is not used (perhaps because the interface is receiving traffic from port mirror on the switch). sysctl output from dev.ix.0: http://pastebin.com/fkRp7Py5 -- Marcin Markowski