Date: Sun, 17 Apr 2005 22:17:48 -0500 (CDT) From: Joe Greco <jgreco@ns.sol.net> To: paul@xciv.org, stable@freebsd.org Subject: Re: Tuning for router performance Message-ID: <200504180317.j3I3Hmdo068332@aurora.sol.net>
next in thread | raw e-mail | index | archive | help
> RELENG_5 as of Apr 03 21:33
>
> * P4 3.0GHz / 2GB DDR400 RAM
> * Supermicro P4SCi with 2 on-board Intel gige
> * Additional 2-port Intel gige on 64-bit PCI-X
>
> With this many interfaces I have disabled unnecessary devices in the
> BIOS such as USB to reduce IRQ sharing.
Interesting.
I'm using 4.11-RELEASE on a P4 3.0G / 1GB DDR400, on a SuperMicro P4SC8
(which is basically the P4SCi with SCSI). No additional 2-port card.
It's a fairly similar setup.
No problems with high traffic rates. (For the purposes of this discussion,
all traffic except ssh to the router is traffic /thru/ the router).
input (Total) output
packets errs bytes packets errs bytes colls
107439 0 72162014 144813 0 142032833 0
101704 0 68046409 135931 0 128359873 0
104645 0 65768204 141990 0 129334101 0
112468 0 75986846 150024 0 144615912 0
106497 0 72030483 142558 0 141768086 0
99765 0 70045449 133374 0 132466526 0
96067 0 63798208 125844 0 125487982 0
101886 0 70674519 134926 0 134179234 0
103760 0 69877737 135941 0 137523002 0
112911 0 79450478 154102 0 151303406 0
116146 0 77471891 156882 0 152534206 0
The apparent high output rate is due to vlan devices, where the output is
attributed to both vlan* and em1 (dedicated to vlans). The input and the
output are actually both approximately the input numbers.
We had noticed at one point that we were starting to see em1 report drops
which were in turn being reported as errs by the vlan devices. We were
pushing right near a gigabit of traffic at the time, so I had written it
off as just pushing the router to the edge. But let's re-examine that.
Now, if I start pumping more PPS, I do indeed see errs:
input (Total) output
packets errs bytes packets errs bytes colls drops
97267 0 66675536 122086 0 131168641 0 0
104247 0 75413955 130001 0 142561178 0 0
95617 0 64612833 121289 0 127119179 0 0
253471 0 77582829 397208 10000 139671386 0 10000
289454 0 71304600 483657 9443 129348488 0 9443
288084 0 71948183 486654 8725 135213379 0 8725
287875 0 68385123 491482 7460 123001416 0 7460
290068 0 72583882 487009 10140 135995424 0 10140
283690 0 66749355 489184 5460 120351717 0 5460
287656 0 71174008 494372 6546 135130910 0 6546
294036 0 73220341 485374 12531 131770769 0 12531
292414 0 74153414 489359 10575 139045664 0 10575
293152 0 72187276 485446 11945 129875438 0 11945
288516 0 73740228 484401 8261 138960063 0 8261
286625 0 70340602 476896 11569 126852274 0 11569
226024 0 70356111 374375 3985 135568639 0 3985
97423 0 67559069 123148 0 131424813 0 0
99617 0 70842045 126110 0 134722149 0 0
109440 0 75528823 139232 0 148738405 0 0
99994 0 69608728 128823 0 130991145 0 0
This is running a little "udpblast" program that sends min size packets
in addition to the production traffic load on the router.
I find this interesting, because the aggregate traffic through the router
is clearly not anywhere near a gigabit. So it does appear that there is
some sort of inadvertent cap on PPS here.
Now, a few notes.
1) I notice that /your/ errors are showing as input errors. Mine seem to
show up as output errors.
2) We need to remember that the design of the P4SC{8,i} is a bit crappy,
in that the onboard ports consist of one CSA port (no problem here)
and one PCI port - which Supermicro wisely placed on the 32-bit, 33 MHz
legacy PCI bus. This could potentially limit the throughput on that
port.
3) The driver tunable EM_TIDV looks quite promising as a first guess as to
what to tune for /my/ issue.
4) I think playing with EM_MAX_TXD and EM_MAX_RXD may be useful to both you
and I as well.
5) Don't know about FreeBSD 5. You're on your own there. :-)
... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200504180317.j3I3Hmdo068332>