Date: Thu, 13 Nov 2014 00:34:26 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: "K. Macy" <kmacy@freebsd.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, J David <j.david.lists@gmail.com>, "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Re: How thread-friendly is kevent? Message-ID: <20141113083426.GH24601@funkthat.com> In-Reply-To: <CAHM0Q_MJjH=Cq0AKTn64RkzgwPBMoKnDVu-GvcX6qBcFYDMpOg@mail.gmail.com> References: <CABXB=RQWxu-d30raZ%2BFcrnrGsr5gG2Za_=cx8-jCnLSgJDSF=Q@mail.gmail.com> <20141110071353.GO24601@funkthat.com> <CABXB=RStLz6J9L3--KsM308-0h0N5ZeZZvw1GbDi%2BZvKO4U64g@mail.gmail.com> <20141112084909.GV24601@funkthat.com> <CAHM0Q_MJjH=Cq0AKTn64RkzgwPBMoKnDVu-GvcX6qBcFYDMpOg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
K. Macy wrote this message on Wed, Nov 12, 2014 at 14:14 -0800: > This may not be related, but it's interesting: a client I worked for > many years ago switched from kqueue to poll for their web server > because kqueue did not distribute the workload evenly. I imagine this > is unchanged. I've heard this before... But no one has been able to say that this caused a performance issue... If they were able to switch to poll w/o significant performance overhead, then they probably weren't dealing w/ thousands upon thousands of fd's, or they were already doing thread balancing by having each thread wait on a subset of fd's... It's also the standard problem of batching vs. single processing... If you call kevent w/ room for 8 threads, the first thread will try to take them all if it can... If that same thread finishes before more events come in, then you really don't have that much more work to do for the other threads... You can obviously decrease this batching, but then other issues come into play... If someone has some solid numbers and can test patches, I'd be interested in helping them work on it, but until then... > On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney <jmg@funkthat.com> wrote: > > J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500: > >> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg@funkthat.com> wrote: > >> > you > >> > really need to use one of _DISPATCH or _ONESHOT to ensure that the > >> > event only gets delivered to a single thread.... > >> > >> That's what one would expect, which is why the observed behavior was > >> so surprising. After increasing the testing load considerably, it did > >> behave as expected (waking more than one thread for one event). But > >> even so, the occurrences were very rare. It would wake up at most one > >> "extra" thread in slightly less than 1 out of 100,000 events. > > > > This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH > > to be delivered many times... Is it possible you have locks in your > > userland side of things that make this less likely? > > > >> > Though if you mean how many threads will be woken up in the kernel > >> > and find that there are no events remaining as one of the other kernel > >> > threads has delivered the event, then yes, I have looked at the code, > >> > and there will be a thundering herd problem... > >> > >> Thanks for that, that's exactly the kind of information I was hoping to find. > >> > >> Is that something that can happen without any usermode-visible > >> effects? I.e. all the threads wake up, but they almost all go back to > >> sleep without leaving the kevent() syscall since they can see there's > >> nothing to do anymore. If so, that would match the observed behavior, > >> but could add up to a lot of hidden overhead. > > > > I have an idea that should only be a few lines of changes that would > > prevent all the threads waking up... As we lock the kq before doing > > the wakeup, we can change KQ_SLEEP from a flag to a count for how many > > threads are sleeping for an event, and if non-zero, do a wakeup_one... > > Then when kqueue_scan is about to exit, check to see if there are > > still events and threads waiting, and then do another wakeup_one... > > > > Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make > > sure everyone wakes up... > > > > Well, if you don't have _ONESHOT and _DISPATCH, any changes I make > > should make it more reliable that all threads get the events dispatched > > to them... :) > > > >> > And if you do, it would make more sense to > >> > use the recent RSS work that Adrian has been working on, and have one > >> > kq per CPU w/ the proper cpu binding for that set of sockets... > >> > >> The most recent information I was able to find: > >> > >> http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html > >> > >> suggests that this work, while admirable and important, is quite some > >> ways away from being production-stable for usermode code: > >> > >> "hopefully I can get my network / rss library up and running enough to > >> prototype an RSS-aware memcached and see if it'll handle this > >> particular workload." > >> > >> It's definitely something to keep an eye on, but probably not a viable > >> approach for us right now. > > > > True... But some of this is making sure you only run enough threads as > > necessary... As the kq lock is a single lock, having extra threads that > > don't really do much work only increasing contention and other issues... > > > > -- > > John-Mark Gurney Voice: +1 415 225 5579 > > > > "All that I will do, has been done, All that I have, has not." > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141113083426.GH24601>