Date: Sat, 27 Sep 2008 03:00:18 +0200 From: Ivan Voras <ivoras@freebsd.org> To: freebsd-net@freebsd.org Subject: Optimizing for high PPS, Intel NICs Message-ID: <gbk0j9$cpj$1@ger.gmane.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigAB5470830777A1634945BAD2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I have an application that consists of a server and clients which communicate via TCP. The communication consists of exchanging many small packets - around 96 bytes of payload (non-uniform). I'm testing it in 8-CURRENT with two machines connected directly via a patch cable. The server is a 8-core Xeon system with 5000X chipset and embedded Intel PRO/1000 EB NIC (card=3D0x109615d9 chip=3D0x10968086) and the client is a= 4-core desktop Core2 also with Intel's embedded NIC, 82566DM-2. The client and server applications are multithreaded and verified that they scale well in environments like this. I've verified with iperf that the NICs and the cable are ok with gigabit traffic. Here's an example netstat trace from a test run on the client: input (Total) output packets errs bytes packets errs bytes colls 161700 0 29374199 161768 0 16290562 0 158320 0 28741763 158405 0 15962986 0 157617 0 28614088 157696 0 15889426 0 157569 0 28618951 157674 0 15884576 0 (i.e. the client receives about 2x the data it sends) I've noticed something strange: the server is bottlenecked with "em1 taskq" kernel thread taking 100% of a CPU core, while the global CPU utilization is around 50%, but the client's em0 taskq thread for this same load is ~~ 10% (with > 30% idle). The client CPU is a bit faster then the server (2.4 GHz vs 2.0 GHz) but I don't think this can account for such a big difference. Toggling TSO on the server doesn't help. This difference in taskq CPU load between the client and the server machine looks wrong to me. Also, I'd expect more PPS here. Can someone comment on this? Are there any known issues with the server NIC I have there? (Both machines run amd64 kernels, WITNESS & INVARIANTS are disabled on both machines). --------------enigAB5470830777A1634945BAD2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjdhagACgkQldnAQVacBch/OwCgxnZmPkgGh36h2LVHQQ+daR8P HUwAoNhdnsrfN83uUMrGT4dwlVf6tIV9 =tBdN -----END PGP SIGNATURE----- --------------enigAB5470830777A1634945BAD2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?gbk0j9$cpj$1>