Date: Wed, 27 Sep 2006 20:16:45 +0400 (MSD) From: Igor Sysoev <is@rambler-co.ru> To: John-Mark Gurney <gurney_j@resnet.uoregon.edu> Cc: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: kqueue disable on delivery... Message-ID: <20060927200042.N4274@is.park.rambler.ru> In-Reply-To: <20060923185727.GW23915@funkthat.com> References: <20060917210426.GI9421@funkthat.com> <20060922171542.G17859@is.park.rambler.ru> <20060922165848.GS23915@funkthat.com> <20060923105426.B20782@is.park.rambler.ru> <20060923185727.GW23915@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Sat, 23 Sep 2006, John-Mark Gurney wrote: > Igor Sysoev wrote this message on Sat, Sep 23, 2006 at 11:40 +0400: >> On Fri, 22 Sep 2006, John-Mark Gurney wrote: >> >>> Igor Sysoev wrote this message on Fri, Sep 22, 2006 at 17:25 +0400: >>>> On Sun, 17 Sep 2006, John-Mark Gurney wrote: >>>> >>>>> I have implemented a couple additional features to kqueue. These allow >>>>> kqueue to be a multithreaded event delivery system that can guarantee >>>>> that the event will only be active in one thread at any time. >>>>> >>>>> The first is EV_DOD, aka disable on delivery. When the event will be >>>>> delivered to userland, the knote is marked disabled so we don't >>>>> have to go through the expense of reallocing the knote each time. >>>>> (Reallocation of the knote is also lock intensive, and disabling is >>>>> cheap.) >>>> >>>> In my opinion, it's too implementation specific flag. >>> >>> How else are you doing to solve having multiple threads servicing >>> the same queue at the same time? Also, Apple is planing on having >>> a similar flag to EV_DOD, but I don't know what they are naming it.. >>> I've tried for a while to find out, but haven't been able to... >> >> As I understand EV_DOD or EV_CLEAR|EV_DOD are like simple EV_ONESHOT, >> except the filter is not deleted on delivery, but is disabled skipping >> some in-kernel lock overhead. That's I'd named it too implementation >> specific. >> >> Yes, the EV_CLEAR|EV_DOD guarantees that the event will be active >> in one thread only at any time. But in my practice I saw there is >> necessity to guarantee that the socket (both events - EVFILT_READ >> and EVFILT_WRITE) will be active in one thread only at any time. >> It seems that is the reason why heavy threaded Solaris 10 event ports >> use the oneshot only model where a socket is deleted from port on delivery. > > Only if you need to both read and write active on the socket at once... > In some/many servers, you only need one or the other, such as file > transfer servers like http and ftp... I thought about you flags and believe it would be usefull. Also I think that the cheapness of EV_DISABLE and EV_DOD should be documented. >>>>> Even though this means that the event will only ever be active in a >>>>> thread at a time, (when you're done handling the event, you reenable >>>>> it), removing the event from the queue outside the event handler (say >>>>> a timeout handler for the connection) poses to be a problem. If you >>>>> simply close the socket, the event disappears, but then there is a >>>>> race between another event being created with the same socket, and >>>>> notification of the handler that you want the event to stop. >>>>> >>>>> In order to handle that situation, I have come up w/ EV_FORCEOS, aka >>>>> FORCE ONE_SHOT. EV_ONESHOT events have the advantage that once queued, >>>>> they don't care if they have been activated or not, they will be returned >>>>> the next round. This means that the timeout handler can safely set >>>>> EV_FORCEOS on the handler, and either if it's _DISABLED (handler running >>>>> and will reenable it), or it's _ENABLED, it will get dispatched, allowing >>>>> the handler to detect the EV_FORCEOS flag and teardown the connection. >>>> >>>> I think it should be EVFILT_USER event, allowing to >>>> EV_SET(&kev, fd, EVFILT_USER, 0, 0, 0, udata); >>>> and the event should automatically sets the EV_ONESHOT flag internally. >>> >>> I'll agree EV_FORCEOS is open for discussion, but you did see how much >>> code it adds right? I was surprised at how small the patch was for the >>> additional functionality.. >> >> Yes, EV_FORCEOS is small patch. However, EVFILT_USER is more generic >> (by the way, Solaris 10 event ports allow to send user-specific >> PORT_SOURCE_USER notification). > > I agree EVFILT_USER would be a useful thing, but it is still different > from EV_FORCEOS... Would you like to contribute some the to > EVFILT_USER? I'll look at integrating it... Here is patch and test program. The patch is against 6.2-PRERELEASE. On 7.0 the EVFILT_LIO should be taked into account. test program should show oneshot user event: >./t n: 1, id: 0x55, filt: -10, fl: 0x0010, ff:0, data:0x0, udata: 0x5678 n: 0, id: 0x0, filt: 0, fl: 0x0000, ff:0, data:0x0, udata: 0x0 >> Two years ago I was implementing threads for my server nginx >> on FreeBSD 4.x, using rfork(). In the absence of EVFILT_USER I made >> the condition variables using kill() and EV_SIGNAL and this user-level >> code may panic kernel. > > Does it still? It seems it was fixed in 1.80, 1.79.2.1, and 1.2.2.11 revisions of src/sys/kern/kern_event.c, but there is report that is not so: http://freebsd.rambler.ru/bsdmail/cvs-src_2005/msg04709.html Currently nginx has threads disabled, so I could not test it. Igor Sysoev http://sysoev.ru/en/ [-- Attachment #2 --] --- src/sys/sys/event.h Fri Jul 1 20:28:32 2005 +++ src/sys/sys/event.h Wed Sep 27 17:35:09 2006 @@ -38,8 +38,9 @@ #define EVFILT_TIMER (-7) /* timers */ #define EVFILT_NETDEV (-8) /* network devices */ #define EVFILT_FS (-9) /* filesystem events */ +#define EVFILT_USER (-10) /* user events */ -#define EVFILT_SYSCOUNT 9 +#define EVFILT_SYSCOUNT 10 #define EV_SET(kevp_, a, b, c, d, e, f) do { \ struct kevent *kevp = (kevp_); \ --- src/sys/kern/kern_event.c Mon Sep 4 21:17:25 2006 +++ src/sys/kern/kern_event.c Wed Sep 27 18:53:25 2006 @@ -132,6 +132,9 @@ static int filt_timerattach(struct knote *kn); static void filt_timerdetach(struct knote *kn); static int filt_timer(struct knote *kn, long hint); +static int filt_userattach(struct knote *kn); +static void filt_userdetach(struct knote *kn); +static int filt_user(struct knote *kn, long hint); static struct filterops file_filtops = { 1, filt_fileattach, NULL, NULL }; @@ -142,6 +145,8 @@ { 0, filt_procattach, filt_procdetach, filt_proc }; static struct filterops timer_filtops = { 0, filt_timerattach, filt_timerdetach, filt_timer }; +static struct filterops user_filtops = + { 0, filt_userattach, filt_userdetach, filt_user }; static uma_zone_t knote_zone; static int kq_ncallouts = 0; @@ -247,6 +252,7 @@ { &timer_filtops }, /* EVFILT_TIMER */ { &file_filtops }, /* EVFILT_NETDEV */ { &fs_filtops }, /* EVFILT_FS */ + { &user_filtops }, /* EVFILT_USER */ }; /* @@ -495,6 +501,27 @@ { return (kn->kn_data != 0); +} + +static int +filt_userattach(struct knote *kn) +{ + kn->kn_flags |= EV_ONESHOT; /* automatically set */ + kn->kn_status &= ~KN_DETACHED; /* knlist_add usually sets it */ + return (0); +} + +static void +filt_userdetach(struct knote *kn) +{ + kn->kn_status |= KN_DETACHED; /* knlist_remove usually clears it */ +} + +static int +filt_user(struct knote *kn, long hint) +{ + + return (1); } /*
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060927200042.N4274>