From owner-freebsd-threads@freebsd.org Wed Jun 3 23:55:50 2020 Return-Path: Delivered-To: freebsd-threads@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 59C0C2F7A33 for ; Wed, 3 Jun 2020 23:55:50 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49cm5133Xdz45XN for ; Wed, 3 Jun 2020 23:55:49 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 053NtRZF017107 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 3 Jun 2020 16:55:27 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 053NtRjC017106; Wed, 3 Jun 2020 16:55:27 -0700 (PDT) (envelope-from jmg) Date: Wed, 3 Jun 2020 16:55:27 -0700 From: John-Mark Gurney To: Christof Meerwald Cc: freebsd-threads@freebsd.org Subject: Re: kevent and multiple worker threads Message-ID: <20200603235527.GG4213@funkthat.com> Mail-Followup-To: Christof Meerwald , freebsd-threads@freebsd.org References: <20120318213749.GQ843@edge.cmeerw.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120318213749.GQ843@edge.cmeerw.net> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Wed, 03 Jun 2020 16:55:27 -0700 (PDT) X-Rspamd-Queue-Id: 49cm5133Xdz45XN X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [0.97 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.45)[-0.454]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.01)[0.011]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.22)[0.218]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2020 23:55:50 -0000 Christof Meerwald wrote this message on Sun, Mar 18, 2012 at 22:37 +0100: > I am trying to figure out how kqueue/kevent works in a multi-threaded > environment. So I have created a small test program > (http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates > a socketpair and send 4-byte messages between the 2 sockets. There is > only ever 1 message in transit, but there can be multiple threads > waiting for event notification using kevent. > > Note that I have only been able to test this inside a KVM virtual > machine with FreeBSD 9.0 on a dual-core AMD64 laptop. > > But what I find extremely interesting is that the time seems to > increase exponentially with the number of worker threads, e.g. I get > (the first argument is the number of worker threads and the second is > the number of iterations): > > $ ./kqtest 1 100000 > 0.508513 > $ ./kqtest 2 100000 > 1.377444 > $ ./kqtest 4 100000 > 2.533485 > $ ./kqtest 8 100000 > 12.305741 I believe this is because you are not using _ONESHOT or _DISPATCH to remove/disable the event from the queue such that only one thread will service the event... Then once the event has been services, you can re-add or re-enable the event to keep getting events... Make sure that only one thread is being woken up and run for each even that happens... You can use ktrace(1) to see what threads are doing what syscalls to ensure that only one thread wakes up... kevent is a level triggered system, so when an event comes in, a thread will wake up, get a copy of the event, but that event is still active, so the next thread will get it because the first hasn't serviced it yet, and so on, but only one thread will end up servicing the event despite all the threads waking up... The reason clear does not work for sockets is that the data, gets "cleared", but then the socket is rechecked if there's data available, and since the first thread hasn't processed it yet, there's still data... > BTW, I have also tested in with NetBSD 6.0 BETA on the same machine > (also inside a KVM virtual machine) and there the results look > different: w/o a ktrace to compare the two, not sure why there's a difference... Also, I do not know if NetBSD has chosen the same behavior wrt _CLEAR as FreeBSD, or if they are different. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."