Date: Wed, 3 Jun 2020 16:55:27 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Christof Meerwald <cmeerw@cmeerw.org> Cc: freebsd-threads@freebsd.org Subject: Re: kevent and multiple worker threads Message-ID: <20200603235527.GG4213@funkthat.com> In-Reply-To: <20120318213749.GQ843@edge.cmeerw.net> References: <20120318213749.GQ843@edge.cmeerw.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Christof Meerwald wrote this message on Sun, Mar 18, 2012 at 22:37 +0100: > I am trying to figure out how kqueue/kevent works in a multi-threaded > environment. So I have created a small test program > (http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates > a socketpair and send 4-byte messages between the 2 sockets. There is > only ever 1 message in transit, but there can be multiple threads > waiting for event notification using kevent. > > Note that I have only been able to test this inside a KVM virtual > machine with FreeBSD 9.0 on a dual-core AMD64 laptop. > > But what I find extremely interesting is that the time seems to > increase exponentially with the number of worker threads, e.g. I get > (the first argument is the number of worker threads and the second is > the number of iterations): > > $ ./kqtest 1 100000 > 0.508513 > $ ./kqtest 2 100000 > 1.377444 > $ ./kqtest 4 100000 > 2.533485 > $ ./kqtest 8 100000 > 12.305741 I believe this is because you are not using _ONESHOT or _DISPATCH to remove/disable the event from the queue such that only one thread will service the event... Then once the event has been services, you can re-add or re-enable the event to keep getting events... Make sure that only one thread is being woken up and run for each even that happens... You can use ktrace(1) to see what threads are doing what syscalls to ensure that only one thread wakes up... kevent is a level triggered system, so when an event comes in, a thread will wake up, get a copy of the event, but that event is still active, so the next thread will get it because the first hasn't serviced it yet, and so on, but only one thread will end up servicing the event despite all the threads waking up... The reason clear does not work for sockets is that the data, gets "cleared", but then the socket is rechecked if there's data available, and since the first thread hasn't processed it yet, there's still data... > BTW, I have also tested in with NetBSD 6.0 BETA on the same machine > (also inside a KVM virtual machine) and there the results look > different: w/o a ktrace to compare the two, not sure why there's a difference... Also, I do not know if NetBSD has chosen the same behavior wrt _CLEAR as FreeBSD, or if they are different. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200603235527.GG4213>