From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 12 08:49:11 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 220F7D32; Wed, 12 Nov 2014 08:49:11 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EB62CF12; Wed, 12 Nov 2014 08:49:10 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id sAC8n91c025370 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 12 Nov 2014 00:49:09 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id sAC8n9qv025369; Wed, 12 Nov 2014 00:49:09 -0800 (PST) (envelope-from jmg) Date: Wed, 12 Nov 2014 00:49:09 -0800 From: John-Mark Gurney To: J David Subject: Re: How thread-friendly is kevent? Message-ID: <20141112084909.GV24601@funkthat.com> Mail-Followup-To: J David , "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" References: <20141110071353.GO24601@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 12 Nov 2014 00:49:10 -0800 (PST) Cc: "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2014 08:49:11 -0000 J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500: > On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney wrote: > > you > > really need to use one of _DISPATCH or _ONESHOT to ensure that the > > event only gets delivered to a single thread.... > > That's what one would expect, which is why the observed behavior was > so surprising. After increasing the testing load considerably, it did > behave as expected (waking more than one thread for one event). But > even so, the occurrences were very rare. It would wake up at most one > "extra" thread in slightly less than 1 out of 100,000 events. This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH to be delivered many times... Is it possible you have locks in your userland side of things that make this less likely? > > Though if you mean how many threads will be woken up in the kernel > > and find that there are no events remaining as one of the other kernel > > threads has delivered the event, then yes, I have looked at the code, > > and there will be a thundering herd problem... > > Thanks for that, that's exactly the kind of information I was hoping to find. > > Is that something that can happen without any usermode-visible > effects? I.e. all the threads wake up, but they almost all go back to > sleep without leaving the kevent() syscall since they can see there's > nothing to do anymore. If so, that would match the observed behavior, > but could add up to a lot of hidden overhead. I have an idea that should only be a few lines of changes that would prevent all the threads waking up... As we lock the kq before doing the wakeup, we can change KQ_SLEEP from a flag to a count for how many threads are sleeping for an event, and if non-zero, do a wakeup_one... Then when kqueue_scan is about to exit, check to see if there are still events and threads waiting, and then do another wakeup_one... Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make sure everyone wakes up... Well, if you don't have _ONESHOT and _DISPATCH, any changes I make should make it more reliable that all threads get the events dispatched to them... :) > > And if you do, it would make more sense to > > use the recent RSS work that Adrian has been working on, and have one > > kq per CPU w/ the proper cpu binding for that set of sockets... > > The most recent information I was able to find: > > http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html > > suggests that this work, while admirable and important, is quite some > ways away from being production-stable for usermode code: > > "hopefully I can get my network / rss library up and running enough to > prototype an RSS-aware memcached and see if it'll handle this > particular workload." > > It's definitely something to keep an eye on, but probably not a viable > approach for us right now. True... But some of this is making sure you only run enough threads as necessary... As the kq lock is a single lock, having extra threads that don't really do much work only increasing contention and other issues... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."