From owner-freebsd-threads@FreeBSD.ORG Sun Mar 18 21:37:52 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11BAA106564A for ; Sun, 18 Mar 2012 21:37:52 +0000 (UTC) (envelope-from cmeerw@cmeerw.org) Received: from edge.cmeerw.net (edge.cmeerw.net [IPv6:2001:1608:10:3::7]) by mx1.freebsd.org (Postfix) with ESMTP id 9C4508FC12 for ; Sun, 18 Mar 2012 21:37:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cmeerw.org; s=dkim; h=Content-Type:MIME-Version:Message-ID:Subject:To:From:Date; bh=tUWQ8E5bY6XMDqx7ig8CmC49JKMrgIWEvK3pDifhWQE=; b=UIbAPzVSa65EljvdidRRGgWtVldePC36/ZI3tvv7K0mUOL7M7ReY/BZ+wyRFKmkolCLuA9vMoBMTeVw4+jTWrgi5ZIG1mSBtynqfqEBrpSpjU4+UJ/O1KCmYQVs5eqkkoBlkTrxsvkNhURU4wst1+p9FFpN+mnd7jQMoJ2OVaLk=; Received: from cmeerw by edge.cmeerw.net with local (Exim 4.77) (envelope-from ) id 1S9NnR-0006CO-Gm; Sun, 18 Mar 2012 22:37:49 +0100 Date: Sun, 18 Mar 2012 22:37:49 +0100 From: Christof Meerwald To: freebsd-threads@freebsd.org Message-ID: <20120318213749.GQ843@edge.cmeerw.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-PGP-Key: 1024D/2B10BE68, 1998-06-29 X-PGP-Fingerprint: 0289 5466 C1F5 B03C DBA7 6304 8CAF 9782 2B10 BE68 User-Agent: Mutt/1.5.20 (2009-06-14) Subject: kevent and multiple worker threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Mar 2012 21:37:52 -0000 Hi, I am trying to figure out how kqueue/kevent works in a multi-threaded environment. So I have created a small test program (http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates a socketpair and send 4-byte messages between the 2 sockets. There is only ever 1 message in transit, but there can be multiple threads waiting for event notification using kevent. Note that I have only been able to test this inside a KVM virtual machine with FreeBSD 9.0 on a dual-core AMD64 laptop. But what I find extremely interesting is that the time seems to increase exponentially with the number of worker threads, e.g. I get (the first argument is the number of worker threads and the second is the number of iterations): $ ./kqtest 1 100000 0.508513 $ ./kqtest 2 100000 1.377444 $ ./kqtest 4 100000 2.533485 $ ./kqtest 8 100000 12.305741 BTW, I have also tested in with NetBSD 6.0 BETA on the same machine (also inside a KVM virtual machine) and there the results look different: $ ./kqtest 1 100000 0.480187 $ ./kqtest 2 100000 1.875658 $ ./kqtest 4 100000 1.552574 $ ./kqtest 8 100000 2.024548 $ ./kqtest 16 100000 1.951622 While there is a big difference on NetBSD between 1 and 2 worker threads, the run time then is roughly stable. There is also a Linux version (using epoll) of this test program (http://svn.cmeerw.net/src/nginetd/trunk/test/eptest.cc). There the timings are much more stable (but this is the actual host operating system, so no virtual machine) although the single-threaded case is actually slower than on NetBSD/FreeBSD: $ ./eptest 1 100000 0.635127 $ ./eptest 2 100000 0.827467 $ ./eptest 4 100000 0.909760 $ ./eptest 8 100000 0.929008 $ ./eptest 16 100000 1.010422 So, I guess my first question is: is this just caused by running it inside a KVM virtual machine or could someone be so nice and try to reproduce this behaviour on actual hardware? Am I doing something completely stupid in the source code? As far as I can tell, when data is available on one of the sockets, only one threads gets woken up (so at least from the user space side the behaviour looks reasonable). Christof -- http://cmeerw.org sip:cmeerw at cmeerw.org mailto:cmeerw at cmeerw.org xmpp:cmeerw at cmeerw.org