Date: Sun, 18 Mar 2012 22:37:49 +0100 From: Christof Meerwald <cmeerw@cmeerw.org> To: freebsd-threads@freebsd.org Subject: kevent and multiple worker threads Message-ID: <20120318213749.GQ843@edge.cmeerw.net>
next in thread | raw e-mail | index | archive | help
Hi, I am trying to figure out how kqueue/kevent works in a multi-threaded environment. So I have created a small test program (http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates a socketpair and send 4-byte messages between the 2 sockets. There is only ever 1 message in transit, but there can be multiple threads waiting for event notification using kevent. Note that I have only been able to test this inside a KVM virtual machine with FreeBSD 9.0 on a dual-core AMD64 laptop. But what I find extremely interesting is that the time seems to increase exponentially with the number of worker threads, e.g. I get (the first argument is the number of worker threads and the second is the number of iterations): $ ./kqtest 1 100000 0.508513 $ ./kqtest 2 100000 1.377444 $ ./kqtest 4 100000 2.533485 $ ./kqtest 8 100000 12.305741 BTW, I have also tested in with NetBSD 6.0 BETA on the same machine (also inside a KVM virtual machine) and there the results look different: $ ./kqtest 1 100000 0.480187 $ ./kqtest 2 100000 1.875658 $ ./kqtest 4 100000 1.552574 $ ./kqtest 8 100000 2.024548 $ ./kqtest 16 100000 1.951622 While there is a big difference on NetBSD between 1 and 2 worker threads, the run time then is roughly stable. There is also a Linux version (using epoll) of this test program (http://svn.cmeerw.net/src/nginetd/trunk/test/eptest.cc). There the timings are much more stable (but this is the actual host operating system, so no virtual machine) although the single-threaded case is actually slower than on NetBSD/FreeBSD: $ ./eptest 1 100000 0.635127 $ ./eptest 2 100000 0.827467 $ ./eptest 4 100000 0.909760 $ ./eptest 8 100000 0.929008 $ ./eptest 16 100000 1.010422 So, I guess my first question is: is this just caused by running it inside a KVM virtual machine or could someone be so nice and try to reproduce this behaviour on actual hardware? Am I doing something completely stupid in the source code? As far as I can tell, when data is available on one of the sockets, only one threads gets woken up (so at least from the user space side the behaviour looks reasonable). Christof -- http://cmeerw.org sip:cmeerw at cmeerw.org mailto:cmeerw at cmeerw.org xmpp:cmeerw at cmeerw.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120318213749.GQ843>