From owner-freebsd-questions@FreeBSD.ORG Wed Nov 12 22:14:06 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC3FD134; Wed, 12 Nov 2014 22:14:06 +0000 (UTC) Received: from mail-yk0-x22b.google.com (mail-yk0-x22b.google.com [IPv6:2607:f8b0:4002:c07::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6C3F95; Wed, 12 Nov 2014 22:14:06 +0000 (UTC) Received: by mail-yk0-f171.google.com with SMTP id 142so1491760ykq.2 for ; Wed, 12 Nov 2014 14:14:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=Y+2x2nwM6zHvfeRMvEQbfoM1ZLkwmYkR0UEflLr3zmQ=; b=FzRnJSP7K+4DiyQMEQyz3eY0fa4I50YjcSl054EgoZay8FvjMWcSy8Zrqfi3Er+IIG IJ8NtvUn143sfdGk+rNNuGf3XgT20EOLkFFWIOMrlDtfTkjqV1l9E7Pbi9MhHV8xQyJJ j+jz877M5GeiSL0sFa4cbVhQWU09wCVhDGXK/Hwj+OV3mQMa2qdWQPbx7cvLuLbBbdkW Ok05rjvpulb8OhBpW94zPVvuf6vxzsMe4OnkI2LegPbuizW00t44MwvKTWORVKuqJuN2 +ABKswyCUWdwPaJVvc34WFnY25gbPQD0eiy34qMm0JjF+SBVtM6ctLPuxcLDNmka5GOV shEA== MIME-Version: 1.0 X-Received: by 10.170.204.23 with SMTP id v23mr2471923yke.115.1415830445720; Wed, 12 Nov 2014 14:14:05 -0800 (PST) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Wed, 12 Nov 2014 14:14:05 -0800 (PST) In-Reply-To: <20141112084909.GV24601@funkthat.com> References: <20141110071353.GO24601@funkthat.com> <20141112084909.GV24601@funkthat.com> Date: Wed, 12 Nov 2014 14:14:05 -0800 X-Google-Sender-Auth: tnQpuFxg5QTBjCubGUuVR910Z5E Message-ID: Subject: Re: How thread-friendly is kevent? From: "K. Macy" To: J David , "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2014 22:14:07 -0000 This may not be related, but it's interesting: a client I worked for many years ago switched from kqueue to poll for their web server because kqueue did not distribute the workload evenly. I imagine this is unchanged. -K On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney wrote: > J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500: >> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney wrote: >> > you >> > really need to use one of _DISPATCH or _ONESHOT to ensure that the >> > event only gets delivered to a single thread.... >> >> That's what one would expect, which is why the observed behavior was >> so surprising. After increasing the testing load considerably, it did >> behave as expected (waking more than one thread for one event). But >> even so, the occurrences were very rare. It would wake up at most one >> "extra" thread in slightly less than 1 out of 100,000 events. > > This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH > to be delivered many times... Is it possible you have locks in your > userland side of things that make this less likely? > >> > Though if you mean how many threads will be woken up in the kernel >> > and find that there are no events remaining as one of the other kernel >> > threads has delivered the event, then yes, I have looked at the code, >> > and there will be a thundering herd problem... >> >> Thanks for that, that's exactly the kind of information I was hoping to find. >> >> Is that something that can happen without any usermode-visible >> effects? I.e. all the threads wake up, but they almost all go back to >> sleep without leaving the kevent() syscall since they can see there's >> nothing to do anymore. If so, that would match the observed behavior, >> but could add up to a lot of hidden overhead. > > I have an idea that should only be a few lines of changes that would > prevent all the threads waking up... As we lock the kq before doing > the wakeup, we can change KQ_SLEEP from a flag to a count for how many > threads are sleeping for an event, and if non-zero, do a wakeup_one... > Then when kqueue_scan is about to exit, check to see if there are > still events and threads waiting, and then do another wakeup_one... > > Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make > sure everyone wakes up... > > Well, if you don't have _ONESHOT and _DISPATCH, any changes I make > should make it more reliable that all threads get the events dispatched > to them... :) > >> > And if you do, it would make more sense to >> > use the recent RSS work that Adrian has been working on, and have one >> > kq per CPU w/ the proper cpu binding for that set of sockets... >> >> The most recent information I was able to find: >> >> http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html >> >> suggests that this work, while admirable and important, is quite some >> ways away from being production-stable for usermode code: >> >> "hopefully I can get my network / rss library up and running enough to >> prototype an RSS-aware memcached and see if it'll handle this >> particular workload." >> >> It's definitely something to keep an eye on, but probably not a viable >> approach for us right now. > > True... But some of this is making sure you only run enough threads as > necessary... As the kq lock is a single lock, having extra threads that > don't really do much work only increasing contention and other issues... > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"