Date: Sat, 27 Sep 2008 03:00:18 +0200 From: Ivan Voras <ivoras@freebsd.org> To: freebsd-net@freebsd.org Subject: Optimizing for high PPS, Intel NICs Message-ID: <gbk0j9$cpj$1@ger.gmane.org>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
I have an application that consists of a server and clients which
communicate via TCP. The communication consists of exchanging many small
packets - around 96 bytes of payload (non-uniform). I'm testing it in
8-CURRENT with two machines connected directly via a patch cable. The
server is a 8-core Xeon system with 5000X chipset and embedded Intel
PRO/1000 EB NIC (card=0x109615d9 chip=0x10968086) and the client is a
4-core desktop Core2 also with Intel's embedded NIC, 82566DM-2. The
client and server applications are multithreaded and verified that they
scale well in environments like this. I've verified with iperf that the
NICs and the cable are ok with gigabit traffic.
Here's an example netstat trace from a test run on the client:
input (Total) output
packets errs bytes packets errs bytes colls
161700 0 29374199 161768 0 16290562 0
158320 0 28741763 158405 0 15962986 0
157617 0 28614088 157696 0 15889426 0
157569 0 28618951 157674 0 15884576 0
(i.e. the client receives about 2x the data it sends)
I've noticed something strange: the server is bottlenecked with "em1
taskq" kernel thread taking 100% of a CPU core, while the global CPU
utilization is around 50%, but the client's em0 taskq thread for this
same load is ~~ 10% (with > 30% idle). The client CPU is a bit faster
then the server (2.4 GHz vs 2.0 GHz) but I don't think this can account
for such a big difference. Toggling TSO on the server doesn't help.
This difference in taskq CPU load between the client and the server
machine looks wrong to me. Also, I'd expect more PPS here. Can someone
comment on this? Are there any known issues with the server NIC I have
there?
(Both machines run amd64 kernels, WITNESS & INVARIANTS are disabled on
both machines).
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkjdhagACgkQldnAQVacBch/OwCgxnZmPkgGh36h2LVHQQ+daR8P
HUwAoNhdnsrfN83uUMrGT4dwlVf6tIV9
=tBdN
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?gbk0j9$cpj$1>