From owner-freebsd-net@FreeBSD.ORG Thu Mar 11 17:42:06 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7789106564A for ; Thu, 11 Mar 2010 17:42:06 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f54.google.com (mail-ww0-f54.google.com [74.125.82.54]) by mx1.freebsd.org (Postfix) with ESMTP id 789338FC26 for ; Thu, 11 Mar 2010 17:42:06 +0000 (UTC) Received: by wwb28 with SMTP id 28so225124wwb.13 for ; Thu, 11 Mar 2010 09:42:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=Oh0NnddPmOztI9TKdiwKRkicsXHcPVduXBZeX9q/ZeM=; b=wHPLJMLjbpzC2r5cqaAeIQDw6BQ6DiVJi2tI7sagpvAx5MzsIV5aSwOmxrHynnQwyd BoHZepTIwqNesMbO5jRuUYhR2Ib2BjX116YJzq2QY1N0qCZjNLGdhYpBWAh6bpDEbDIf 5mph4iZIQ9w3jJ3pq/4KIX9n+janeJdi4wTRE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=hddNafbA6mAKUpvv7914ltZTDBsIN41IZp1qegNGlccH2weWnp4ZYToN4AiJBhiiEC CrckN7hgxLWBXFqwUswqcyvOtV3fEIFof5/NYo8r1VfvrwK6NqX2I0xhjYda2ssNupps Od7fOt2RXYgAs8+gwhFzTF5+UbLNMlKEBYa70= MIME-Version: 1.0 Received: by 10.216.87.143 with SMTP id y15mr2346611wee.42.1268329325464; Thu, 11 Mar 2010 09:42:05 -0800 (PST) In-Reply-To: <4B99114E.7060909@mail.ru> References: <4B99114E.7060909@mail.ru> Date: Thu, 11 Mar 2010 09:42:05 -0800 Message-ID: <2a41acea1003110942u717e2222hd984bd2859c3e477@mail.gmail.com> From: Jack Vogel To: rihad Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: ixgbe input errors at high data rates X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 17:42:07 -0000 The 1.3.3 driver is two years old, and your OS is older. I would respectfully suggest that you update to 8.0 where lots of effort was put to make 10G hardware perform up to its capabilities. Similarly, I have done lots of work in two years to the ixgbe driver, I would even suggest that once you have 8 installed you get the driver from HEAD. Regards, Jack On Thu, Mar 11, 2010 at 7:50 AM, rihad wrote: > Hi, our Intel 10 GigE cards are finally here, identified as > > with the driver ixgbe-1.3.3 off the CD-ROM. > One card is used for input, the other for output, doing traffic limiting > (dummynet) and accounting in between. > At data rates of about 700-1000 mbps netstat -i shows many Input errors on > ix0 at a rate of 10-20K per second :( > > top -HS: > CPU: 1.3% user, 0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle > Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf, 239M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 32 root -68 - 0K 16K CPU3 3 460:56 100.00% irq258: ix0 > 33 root -68 - 0K 16K CPU7 7 143:14 100.00% ix0 rxq > 13 root 171 ki31 0K 16K RUN 5 574:39 93.65% idle: cpu5 > 12 root 171 ki31 0K 16K RUN 6 507:08 88.33% idle: cpu6 > 14 root 171 ki31 0K 16K CPU4 4 424:04 80.37% idle: cpu4 > 18 root 171 ki31 0K 16K CPU0 0 395:34 75.00% idle: cpu0 > 16 root 171 ki31 0K 16K RUN 2 433:10 70.21% idle: cpu2 > 700 root -68 - 0K 16K - 2 292:19 56.64% dummynet > 17 root 171 ki31 0K 16K CPU1 1 399:02 50.39% idle: cpu1 > 37 root -68 - 0K 16K CPU1 1 196:19 39.50% ix1 rxq > 11 root 171 ki31 0K 16K RUN 7 510:39 14.79% idle: cpu7 > 36 root -68 - 0K 16K WAIT 5 36:36 8.64% irq260: ix1 > 19 root -32 - 0K 16K CPU6 6 36:52 5.08% swi4: clock > sio > > > Turning dummynet off (by short-circuiting the IPFW rule "allow ip from any > to any" before the "pipe tablearg") doesn't eliminate the input errors. > Turning ip.fastfowarding off (see below) doesn't help either (why would it), > only this time "swi" is chewing up the CPU time instead of "irq". Are we > hitting the CPU core limits here? It's a dual cpu quad-core Intel(R) Xeon(R) > CPU E5410 @ 2.33GHz (Dell PowerEdge 2950). > Shouldn't this $2.5K expensive card have decently-sized hardware buffers to > prevent any overruns? > > Some custom settings: > kern.hz=4000 > net.inet.ip.fastforwarding=1 > kern.ipc.nmbclusters=111111 > net.inet.ip.dummynet.io_fast=1 > net.isr.direct=0 > net.inet.ip.intr_queue_maxlen=5000 > hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs > > FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't used. > Should I nonetheless recompile without it? I heard the mere existence of > DEVICE_POLLING affects some cards' performance. > > Thanks for any tips. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >