Date: Thu, 11 Mar 2010 19:50:38 +0400 From: rihad <rihad@mail.ru> To: freebsd-net@freebsd.org Subject: ixgbe input errors at high data rates Message-ID: <4B99114E.7060909@mail.ru>
next in thread | raw e-mail | index | archive | help
Hi, our Intel 10 GigE cards are finally here, identified as
<Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.3.3>
with the driver ixgbe-1.3.3 off the CD-ROM.
One card is used for input, the other for output, doing traffic limiting
(dummynet) and accounting in between.
At data rates of about 700-1000 mbps netstat -i shows many Input errors
on ix0 at a rate of 10-20K per second :(
top -HS:
CPU: 1.3% user, 0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle
Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf, 239M Free
Swap: 2048M Total, 2048M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
32 root -68 - 0K 16K CPU3 3 460:56 100.00% irq258: ix0
33 root -68 - 0K 16K CPU7 7 143:14 100.00% ix0 rxq
13 root 171 ki31 0K 16K RUN 5 574:39 93.65% idle: cpu5
12 root 171 ki31 0K 16K RUN 6 507:08 88.33% idle: cpu6
14 root 171 ki31 0K 16K CPU4 4 424:04 80.37% idle: cpu4
18 root 171 ki31 0K 16K CPU0 0 395:34 75.00% idle: cpu0
16 root 171 ki31 0K 16K RUN 2 433:10 70.21% idle: cpu2
700 root -68 - 0K 16K - 2 292:19 56.64% dummynet
17 root 171 ki31 0K 16K CPU1 1 399:02 50.39% idle: cpu1
37 root -68 - 0K 16K CPU1 1 196:19 39.50% ix1 rxq
11 root 171 ki31 0K 16K RUN 7 510:39 14.79% idle: cpu7
36 root -68 - 0K 16K WAIT 5 36:36 8.64% irq260: ix1
19 root -32 - 0K 16K CPU6 6 36:52 5.08% swi4:
clock sio
Turning dummynet off (by short-circuiting the IPFW rule "allow ip from
any to any" before the "pipe tablearg") doesn't eliminate the input
errors. Turning ip.fastfowarding off (see below) doesn't help either
(why would it), only this time "swi" is chewing up the CPU time instead
of "irq". Are we hitting the CPU core limits here? It's a dual cpu
quad-core Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (Dell PowerEdge 2950).
Shouldn't this $2.5K expensive card have decently-sized hardware buffers
to prevent any overruns?
Some custom settings:
kern.hz=4000
net.inet.ip.fastforwarding=1
kern.ipc.nmbclusters=111111
net.inet.ip.dummynet.io_fast=1
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000
hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs
FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't used.
Should I nonetheless recompile without it? I heard the mere existence of
DEVICE_POLLING affects some cards' performance.
Thanks for any tips.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B99114E.7060909>