From owner-freebsd-current@FreeBSD.ORG Fri Sep 22 16:58:51 2006 Return-Path: X-Original-To: freebsd-current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 180FF16A403; Fri, 22 Sep 2006 16:58:51 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id A209D43D53; Fri, 22 Sep 2006 16:58:50 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (eln3rdzzyu8mwswk@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k8MGwo0Z014737; Fri, 22 Sep 2006 09:58:50 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k8MGwn8F014736; Fri, 22 Sep 2006 09:58:49 -0700 (PDT) (envelope-from jmg) Date: Fri, 22 Sep 2006 09:58:49 -0700 From: John-Mark Gurney To: Igor Sysoev Message-ID: <20060922165848.GS23915@funkthat.com> Mail-Followup-To: Igor Sysoev , freebsd-arch@FreeBSD.org, freebsd-current@FreeBSD.org References: <20060917210426.GI9421@funkthat.com> <20060922171542.G17859@is.park.rambler.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060922171542.G17859@is.park.rambler.ru> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: kqueue disable on delivery... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Sep 2006 16:58:51 -0000 Igor Sysoev wrote this message on Fri, Sep 22, 2006 at 17:25 +0400: > On Sun, 17 Sep 2006, John-Mark Gurney wrote: > > >I have implemented a couple additional features to kqueue. These allow > >kqueue to be a multithreaded event delivery system that can guarantee > >that the event will only be active in one thread at any time. > > > >The first is EV_DOD, aka disable on delivery. When the event will be > >delivered to userland, the knote is marked disabled so we don't > >have to go through the expense of reallocing the knote each time. > >(Reallocation of the knote is also lock intensive, and disabling is > >cheap.) > > In my opinion, it's too implementation specific flag. How else are you doing to solve having multiple threads servicing the same queue at the same time? Also, Apple is planing on having a similar flag to EV_DOD, but I don't know what they are naming it.. I've tried for a while to find out, but haven't been able to... > >Even though this means that the event will only ever be active in a > >thread at a time, (when you're done handling the event, you reenable > >it), removing the event from the queue outside the event handler (say > >a timeout handler for the connection) poses to be a problem. If you > >simply close the socket, the event disappears, but then there is a > >race between another event being created with the same socket, and > >notification of the handler that you want the event to stop. > > > >In order to handle that situation, I have come up w/ EV_FORCEOS, aka > >FORCE ONE_SHOT. EV_ONESHOT events have the advantage that once queued, > >they don't care if they have been activated or not, they will be returned > >the next round. This means that the timeout handler can safely set > >EV_FORCEOS on the handler, and either if it's _DISABLED (handler running > >and will reenable it), or it's _ENABLED, it will get dispatched, allowing > >the handler to detect the EV_FORCEOS flag and teardown the connection. > > I think it should be EVFILT_USER event, allowing to > EV_SET(&kev, fd, EVFILT_USER, 0, 0, 0, udata); > and the event should automatically sets the EV_ONESHOT flag internally. I'll agree EV_FORCEOS is open for discussion, but you did see how much code it adds right? I was surprised at how small the patch was for the additional functionality.. What happens if you are in the process of tearing down udata when this happens, but you haven't gotten far enough to drop it? Then you'd have to deal w/ possible lock inversions between the timeout list and your object lock, deal w/ flags on the object and ref counts.. With _DOD and _FORCEOS, you are able to continue to not require special state flags, locks nor reference counting on your objects serviced by kqueue... I wrote this code in anticipation of supporting sun4v boxes where it'd be useful to have 32 threads (or more) servicing a single kqueue... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."