Date: Wed, 27 Sep 2006 20:16:45 +0400 (MSD) From: Igor Sysoev <is@rambler-co.ru> To: John-Mark Gurney <gurney_j@resnet.uoregon.edu> Cc: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: kqueue disable on delivery... Message-ID: <20060927200042.N4274@is.park.rambler.ru> In-Reply-To: <20060923185727.GW23915@funkthat.com> References: <20060917210426.GI9421@funkthat.com> <20060922171542.G17859@is.park.rambler.ru> <20060922165848.GS23915@funkthat.com> <20060923105426.B20782@is.park.rambler.ru> <20060923185727.GW23915@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-1588627711-1159373805=:4274 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Sat, 23 Sep 2006, John-Mark Gurney wrote: > Igor Sysoev wrote this message on Sat, Sep 23, 2006 at 11:40 +0400: >> On Fri, 22 Sep 2006, John-Mark Gurney wrote: >> >>> Igor Sysoev wrote this message on Fri, Sep 22, 2006 at 17:25 +0400: >>>> On Sun, 17 Sep 2006, John-Mark Gurney wrote: >>>> >>>>> I have implemented a couple additional features to kqueue. These allow >>>>> kqueue to be a multithreaded event delivery system that can guarantee >>>>> that the event will only be active in one thread at any time. >>>>> >>>>> The first is EV_DOD, aka disable on delivery. When the event will be >>>>> delivered to userland, the knote is marked disabled so we don't >>>>> have to go through the expense of reallocing the knote each time. >>>>> (Reallocation of the knote is also lock intensive, and disabling is >>>>> cheap.) >>>> >>>> In my opinion, it's too implementation specific flag. >>> >>> How else are you doing to solve having multiple threads servicing >>> the same queue at the same time? Also, Apple is planing on having >>> a similar flag to EV_DOD, but I don't know what they are naming it.. >>> I've tried for a while to find out, but haven't been able to... >> >> As I understand EV_DOD or EV_CLEAR|EV_DOD are like simple EV_ONESHOT, >> except the filter is not deleted on delivery, but is disabled skipping >> some in-kernel lock overhead. That's I'd named it too implementation >> specific. >> >> Yes, the EV_CLEAR|EV_DOD guarantees that the event will be active >> in one thread only at any time. But in my practice I saw there is >> necessity to guarantee that the socket (both events - EVFILT_READ >> and EVFILT_WRITE) will be active in one thread only at any time. >> It seems that is the reason why heavy threaded Solaris 10 event ports >> use the oneshot only model where a socket is deleted from port on delivery. > > Only if you need to both read and write active on the socket at once... > In some/many servers, you only need one or the other, such as file > transfer servers like http and ftp... I thought about you flags and believe it would be usefull. Also I think that the cheapness of EV_DISABLE and EV_DOD should be documented. >>>>> Even though this means that the event will only ever be active in a >>>>> thread at a time, (when you're done handling the event, you reenable >>>>> it), removing the event from the queue outside the event handler (say >>>>> a timeout handler for the connection) poses to be a problem. If you >>>>> simply close the socket, the event disappears, but then there is a >>>>> race between another event being created with the same socket, and >>>>> notification of the handler that you want the event to stop. >>>>> >>>>> In order to handle that situation, I have come up w/ EV_FORCEOS, aka >>>>> FORCE ONE_SHOT. EV_ONESHOT events have the advantage that once queued, >>>>> they don't care if they have been activated or not, they will be returned >>>>> the next round. This means that the timeout handler can safely set >>>>> EV_FORCEOS on the handler, and either if it's _DISABLED (handler running >>>>> and will reenable it), or it's _ENABLED, it will get dispatched, allowing >>>>> the handler to detect the EV_FORCEOS flag and teardown the connection. >>>> >>>> I think it should be EVFILT_USER event, allowing to >>>> EV_SET(&kev, fd, EVFILT_USER, 0, 0, 0, udata); >>>> and the event should automatically sets the EV_ONESHOT flag internally. >>> >>> I'll agree EV_FORCEOS is open for discussion, but you did see how much >>> code it adds right? I was surprised at how small the patch was for the >>> additional functionality.. >> >> Yes, EV_FORCEOS is small patch. However, EVFILT_USER is more generic >> (by the way, Solaris 10 event ports allow to send user-specific >> PORT_SOURCE_USER notification). > > I agree EVFILT_USER would be a useful thing, but it is still different > from EV_FORCEOS... Would you like to contribute some the to > EVFILT_USER? I'll look at integrating it... Here is patch and test program. The patch is against 6.2-PRERELEASE. On 7.0 the EVFILT_LIO should be taked into account. test program should show oneshot user event: >./t n: 1, id: 0x55, filt: -10, fl: 0x0010, ff:0, data:0x0, udata: 0x5678 n: 0, id: 0x0, filt: 0, fl: 0x0000, ff:0, data:0x0, udata: 0x0 >> Two years ago I was implementing threads for my server nginx >> on FreeBSD 4.x, using rfork(). In the absence of EVFILT_USER I made >> the condition variables using kill() and EV_SIGNAL and this user-level >> code may panic kernel. > > Does it still? It seems it was fixed in 1.80, 1.79.2.1, and 1.2.2.11 revisions of src/sys/kern/kern_event.c, but there is report that is not so: http://freebsd.rambler.ru/bsdmail/cvs-src_2005/msg04709.html Currently nginx has threads disabled, so I could not test it. Igor Sysoev http://sysoev.ru/en/ --0-1588627711-1159373805=:4274 Content-Type: TEXT/plain; charset=US-ASCII; name="patch.evfilt_user.txt" Content-Transfer-Encoding: BASE64 Content-ID: <20060927201645.D4274@is.park.rambler.ru> Content-Description: Content-Disposition: attachment; filename="patch.evfilt_user.txt" LS0tIHNyYy9zeXMvc3lzL2V2ZW50LmgJRnJpIEp1bCAgMSAyMDoyODozMiAy MDA1DQorKysgc3JjL3N5cy9zeXMvZXZlbnQuaAlXZWQgU2VwIDI3IDE3OjM1 OjA5IDIwMDYNCkBAIC0zOCw4ICszOCw5IEBADQogI2RlZmluZSBFVkZJTFRf VElNRVIJCSgtNykJLyogdGltZXJzICovDQogI2RlZmluZSBFVkZJTFRfTkVU REVWCQkoLTgpCS8qIG5ldHdvcmsgZGV2aWNlcyAqLw0KICNkZWZpbmUgRVZG SUxUX0ZTCQkoLTkpCS8qIGZpbGVzeXN0ZW0gZXZlbnRzICovDQorI2RlZmlu ZSBFVkZJTFRfVVNFUgkJKC0xMCkJLyogdXNlciBldmVudHMgKi8NCiANCi0j ZGVmaW5lIEVWRklMVF9TWVNDT1VOVAkJOQ0KKyNkZWZpbmUgRVZGSUxUX1NZ U0NPVU5UCQkxMA0KIA0KICNkZWZpbmUgRVZfU0VUKGtldnBfLCBhLCBiLCBj LCBkLCBlLCBmKSBkbyB7CVwNCiAJc3RydWN0IGtldmVudCAqa2V2cCA9IChr ZXZwXyk7CQlcDQotLS0gc3JjL3N5cy9rZXJuL2tlcm5fZXZlbnQuYwlNb24g U2VwICA0IDIxOjE3OjI1IDIwMDYNCisrKyBzcmMvc3lzL2tlcm4va2Vybl9l dmVudC5jCVdlZCBTZXAgMjcgMTg6NTM6MjUgMjAwNg0KQEAgLTEzMiw2ICsx MzIsOSBAQA0KIHN0YXRpYyBpbnQJZmlsdF90aW1lcmF0dGFjaChzdHJ1Y3Qg a25vdGUgKmtuKTsNCiBzdGF0aWMgdm9pZAlmaWx0X3RpbWVyZGV0YWNoKHN0 cnVjdCBrbm90ZSAqa24pOw0KIHN0YXRpYyBpbnQJZmlsdF90aW1lcihzdHJ1 Y3Qga25vdGUgKmtuLCBsb25nIGhpbnQpOw0KK3N0YXRpYyBpbnQJZmlsdF91 c2VyYXR0YWNoKHN0cnVjdCBrbm90ZSAqa24pOw0KK3N0YXRpYyB2b2lkCWZp bHRfdXNlcmRldGFjaChzdHJ1Y3Qga25vdGUgKmtuKTsNCitzdGF0aWMgaW50 CWZpbHRfdXNlcihzdHJ1Y3Qga25vdGUgKmtuLCBsb25nIGhpbnQpOw0KIA0K IHN0YXRpYyBzdHJ1Y3QgZmlsdGVyb3BzIGZpbGVfZmlsdG9wcyA9DQogCXsg MSwgZmlsdF9maWxlYXR0YWNoLCBOVUxMLCBOVUxMIH07DQpAQCAtMTQyLDYg KzE0NSw4IEBADQogCXsgMCwgZmlsdF9wcm9jYXR0YWNoLCBmaWx0X3Byb2Nk ZXRhY2gsIGZpbHRfcHJvYyB9Ow0KIHN0YXRpYyBzdHJ1Y3QgZmlsdGVyb3Bz IHRpbWVyX2ZpbHRvcHMgPQ0KIAl7IDAsIGZpbHRfdGltZXJhdHRhY2gsIGZp bHRfdGltZXJkZXRhY2gsIGZpbHRfdGltZXIgfTsNCitzdGF0aWMgc3RydWN0 IGZpbHRlcm9wcyB1c2VyX2ZpbHRvcHMgPQ0KKwl7IDAsIGZpbHRfdXNlcmF0 dGFjaCwgZmlsdF91c2VyZGV0YWNoLCBmaWx0X3VzZXIgfTsNCiANCiBzdGF0 aWMgdW1hX3pvbmVfdAlrbm90ZV96b25lOw0KIHN0YXRpYyBpbnQgCQlrcV9u Y2FsbG91dHMgPSAwOw0KQEAgLTI0Nyw2ICsyNTIsNyBAQA0KIAl7ICZ0aW1l cl9maWx0b3BzIH0sCQkJLyogRVZGSUxUX1RJTUVSICovDQogCXsgJmZpbGVf ZmlsdG9wcyB9LAkJCS8qIEVWRklMVF9ORVRERVYgKi8NCiAJeyAmZnNfZmls dG9wcyB9LAkJCS8qIEVWRklMVF9GUyAqLw0KKwl7ICZ1c2VyX2ZpbHRvcHMg fSwJCQkvKiBFVkZJTFRfVVNFUiAqLw0KIH07DQogDQogLyoNCkBAIC00OTUs NiArNTAxLDI3IEBADQogew0KIA0KIAlyZXR1cm4gKGtuLT5rbl9kYXRhICE9 IDApOw0KK30NCisNCitzdGF0aWMgaW50DQorZmlsdF91c2VyYXR0YWNoKHN0 cnVjdCBrbm90ZSAqa24pDQorew0KKwlrbi0+a25fZmxhZ3MgfD0gRVZfT05F U0hPVDsJCS8qIGF1dG9tYXRpY2FsbHkgc2V0ICovDQorCWtuLT5rbl9zdGF0 dXMgJj0gfktOX0RFVEFDSEVEOwkJLyoga25saXN0X2FkZCB1c3VhbGx5IHNl dHMgaXQgKi8NCisJcmV0dXJuICgwKTsNCit9DQorDQorc3RhdGljIHZvaWQN CitmaWx0X3VzZXJkZXRhY2goc3RydWN0IGtub3RlICprbikNCit7DQorCWtu LT5rbl9zdGF0dXMgfD0gS05fREVUQUNIRUQ7CS8qIGtubGlzdF9yZW1vdmUg dXN1YWxseSBjbGVhcnMgaXQgKi8NCit9DQorDQorc3RhdGljIGludA0KK2Zp bHRfdXNlcihzdHJ1Y3Qga25vdGUgKmtuLCBsb25nIGhpbnQpDQorew0KKw0K KwlyZXR1cm4gKDEpOw0KIH0NCiANCiAvKg0K --0-1588627711-1159373805=:4274--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060927200042.N4274>