Date: Sun, 05 Apr 2009 14:20:25 +0200 From: Ivan Voras <ivoras@freebsd.org> To: freebsd-net@freebsd.org Subject: Advice on a multithreaded netisr patch? Message-ID: <gra7mq$ei8$1@ger.gmane.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig9D3AA7C6A7FB08F179C61F87 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I'm developing an application that needs a high rate of small TCP transactions on multi-core systems, and I'm hitting a limit where a kernel task, usually swi:net (but it depends on the driver) hits 100% of a CPU at some transactions/s rate and blocks further performance increase even though other cores are 100% idle. So I've got an idea and tested it out, but it fails in an unexpected way. I'm not very familiar with the network code so I'm probably missing something obvious. The idea was to locate where the packet processing takes place and offload packets to several new kernel threads. I see this can happen in several places - netisr, ip_input and tcp_input, and I chose netisr because I thought maybe it would also help other uses (routing?). Here's a patch against CURRENT: http://people.freebsd.org/~ivoras/diffs/mpip.patch It's fairly simple - starts a configurable number of threads in start_netisr(), assigns circular queues to each, and modifies what I think are entry points for packets in the non-netisr.direct case. I also try to have TCP and UDP traffic from the same host+port processed by the same thread. It has some rough edges but I think this is enough to test the idea. I know that there are several people officially working in this area and I'm not an expert in it so think of it as a weekend hack for learning purposes :) These parameters are needed in loader.conf to test it: net.isr.direct=3D0 net.isr.mtdispatch_n_threads=3D2 I expected things like the contention in upper layers (TCP) leading to not improving performance one bit, but I can't explain what I'm getting here. While testing the application on a plain kernel, I get approx. 100,000 - 120,000 packets/s per direction (by looking at "netstat 1") and a similar number of transactions/s in the application. With the patch I get up to 250,000 packets/s in netstat (3 mtdispatch threads), but for some weird reason the actual number of transactions processed by the application drops to less than 1,000 at the beginning (~~ 30 seconds), then jumps to close to 100,000 transactions/s, with netstat also showing a drop this number of packets. In the first phase, the new threads (netd0..3) are using CPU time almost 100%, in the second phase I can't see where the CPU time is going (using top). I thought this has something to deal with NIC moderation (em) but can't really explain it. The bad performance part (not the jump) is also visible over the loopback interface. Any ideas? --------------enig9D3AA7C6A7FB08F179C61F87 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknYogoACgkQldnAQVacBcg0rwCeK5aaPe2Al0xFoelvU1IyJXup 9DQAmwRr/BgW8/Q/sBkNmlrJqtJtmvci =KeAh -----END PGP SIGNATURE----- --------------enig9D3AA7C6A7FB08F179C61F87--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?gra7mq$ei8$1>