From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 13 08:25:53 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 204DF532; Thu, 13 Nov 2014 08:25:53 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E9A48262; Thu, 13 Nov 2014 08:25:52 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id sAD8PoGO040027 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 13 Nov 2014 00:25:51 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id sAD8PoEC040026; Thu, 13 Nov 2014 00:25:50 -0800 (PST) (envelope-from jmg) Date: Thu, 13 Nov 2014 00:25:50 -0800 From: John-Mark Gurney To: J David Subject: Re: How thread-friendly is kevent? Message-ID: <20141113082550.GG24601@funkthat.com> Mail-Followup-To: J David , "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" References: <20141110071353.GO24601@funkthat.com> <20141112084909.GV24601@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 13 Nov 2014 00:25:51 -0800 (PST) Cc: "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 08:25:53 -0000 J David wrote this message on Wed, Nov 12, 2014 at 17:24 -0500: > On Wed, Nov 12, 2014 at 3:49 AM, John-Mark Gurney wrote: > > This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH > > to be delivered many times... Is it possible you have locks in your > > userland side of things that make this less likely? > > Nope, the test code is (intentionally) entirely lock-free in userland. > > > I have an idea that should only be a few lines of changes that would > > prevent all the threads waking up... As we lock the kq before doing > > the wakeup, we can change KQ_SLEEP from a flag to a count for how many > > threads are sleeping for an event, and if non-zero, do a wakeup_one... > > Then when kqueue_scan is about to exit, check to see if there are > > still events and threads waiting, and then do another wakeup_one... > > This sounds like it could optimize some workloads at substantial > penalties for others. If pursued, maybe it needs its own flag. It really wouldn't be a penalty as the other thread couldn't make progress since the kq lock was held and would be waiting for the kq lock anyways... The only penalty might be the delay in waking up, but that'd be minor... But there was already a penalty for the cross processor read to find out that the lock is held... > > Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make > > sure everyone wakes up... > > > > Well, if you don't have _ONESHOT and _DISPATCH, any changes I make > > should make it more reliable that all threads get the events dispatched > > to them... :) > > Using _DISPATCH is no problem, although a solution that didn't require > two kevent()-calls per event would obviously be better when every > syscall matters. Albeit that is largely an issue on VM's where the > syscall penalty is artificially large. In production, this will of > course run on bare metal. If you care about that, I'd recommend you have a thread local buffer that you add enable events to, and then when you get back to your main loop, you add all of these waiting events.. > The other option is do wrap kevent() with a mutex on the user side. > That's what Apache does with accept(), IIRC. kevent effectively provides that lock internally... > > But some of this is making sure you only run enough threads as > > necessary... > > That's almost always true. But, almost always, determining the > correct value of "enough" requires a blood sacrifice. :) :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."