From owner-freebsd-net@FreeBSD.ORG Thu Mar 11 18:24:00 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41223106564A for ; Thu, 11 Mar 2010 18:24:00 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id EF41C8FC14 for ; Thu, 11 Mar 2010 18:23:59 +0000 (UTC) Received: from [217.25.27.27] (port=13308 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1Npn37-0004J7-00; Thu, 11 Mar 2010 21:23:57 +0300 Message-ID: <4B993542.2070001@mail.ru> Date: Thu, 11 Mar 2010 22:24:02 +0400 From: rihad User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: Jack Vogel References: <4B99114E.7060909@mail.ru> <2a41acea1003110942u717e2222hd984bd2859c3e477@mail.gmail.com> In-Reply-To: <2a41acea1003110942u717e2222hd984bd2859c3e477@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: ixgbe input errors at high data rates X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 18:24:00 -0000 Jack Vogel wrote: > The 1.3.3 driver is two years old, and your OS is older. I would > respectfully suggest > that you update to 8.0 where lots of effort was put to make 10G hardware > perform > up to its capabilities. Similarly, I have done lots of work in two years > to the ixgbe > driver, I would even suggest that once you have 8 installed you get the > driver from > HEAD. > Thanks, I will certainly make the move to 8.0, but on a new box yet to be chosen and ordered. It will take some time before it ships to us. I don't think there's a temporary solution, is there? BTW, slightly off topic: 8.0 seems to be based on libc.so.7, same as FreeBSD 7.1. Does this mean I won't be having problems running the current ports on 8.0? Unless I run make delete-old-libs, of course :-/ > Regards, > > Jack > > On Thu, Mar 11, 2010 at 7:50 AM, rihad > wrote: > > Hi, our Intel 10 GigE cards are finally here, identified as > > with the driver ixgbe-1.3.3 off the CD-ROM. > One card is used for input, the other for output, doing traffic > limiting (dummynet) and accounting in between. > At data rates of about 700-1000 mbps netstat -i shows many Input > errors on ix0 at a rate of 10-20K per second :( > > top -HS: > CPU: 1.3% user, 0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle > Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf, > 239M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 32 root -68 - 0K 16K CPU3 3 460:56 100.00% > irq258: ix0 > 33 root -68 - 0K 16K CPU7 7 143:14 100.00% ix0 rxq > 13 root 171 ki31 0K 16K RUN 5 574:39 93.65% idle: cpu5 > 12 root 171 ki31 0K 16K RUN 6 507:08 88.33% idle: cpu6 > 14 root 171 ki31 0K 16K CPU4 4 424:04 80.37% idle: cpu4 > 18 root 171 ki31 0K 16K CPU0 0 395:34 75.00% idle: cpu0 > 16 root 171 ki31 0K 16K RUN 2 433:10 70.21% idle: cpu2 > 700 root -68 - 0K 16K - 2 292:19 56.64% dummynet > 17 root 171 ki31 0K 16K CPU1 1 399:02 50.39% idle: cpu1 > 37 root -68 - 0K 16K CPU1 1 196:19 39.50% ix1 rxq > 11 root 171 ki31 0K 16K RUN 7 510:39 14.79% idle: cpu7 > 36 root -68 - 0K 16K WAIT 5 36:36 8.64% > irq260: ix1 > 19 root -32 - 0K 16K CPU6 6 36:52 5.08% swi4: > clock sio > > > Turning dummynet off (by short-circuiting the IPFW rule "allow ip > from any to any" before the "pipe tablearg") doesn't eliminate the > input errors. Turning ip.fastfowarding off (see below) doesn't help > either (why would it), only this time "swi" is chewing up the CPU > time instead of "irq". Are we hitting the CPU core limits here? It's > a dual cpu quad-core Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (Dell > PowerEdge 2950). > Shouldn't this $2.5K expensive card have decently-sized hardware > buffers to prevent any overruns? > > Some custom settings: > kern.hz=4000 > net.inet.ip.fastforwarding=1 > kern.ipc.nmbclusters=111111 > net.inet.ip.dummynet.io_fast=1 > net.isr.direct=0 > net.inet.ip.intr_queue_maxlen=5000 > hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs > > FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't > used. Should I nonetheless recompile without it? I heard the mere > existence of DEVICE_POLLING affects some cards' performance. > > Thanks for any tips. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > >