From owner-freebsd-arch@freebsd.org Tue Oct 31 10:05:15 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C657E5415D for ; Tue, 31 Oct 2017 10:05:15 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-qt0-x22a.google.com (mail-qt0-x22a.google.com [IPv6:2607:f8b0:400d:c0d::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 513A16C819; Tue, 31 Oct 2017 10:05:15 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-qt0-x22a.google.com with SMTP id f8so19944442qta.5; Tue, 31 Oct 2017 03:05:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=klfrzuCLyyxZegIauwCuagWJDGwvyTnc1kN1VPYYoyI=; b=L32wECerl7c6Xsx4/gD7xfyQ3LnWSwDmm9lVc4y5NnTflHzmxqAFg5Oc4RnBpPAGJi oSO6FGQnjv927vwawHxkkAe9ED9/GGi6liPknc/+Wd02qVENpa+OJAc1Tcd65G8rUDvG OuBn1sOf2wNpvFUnPmWlzJY3S6E7kDEBBNrIOsGUO+vrBPPi81s7N2juKtFXCO62KzG/ Y/+pAbsCBNfOjiNXcnZ3Z06LUOqwxMxopn0kXM+cgNbmDTRJeU9L5ZQqajKiNpnuRTCP CrCFbQFP5WajklD1QCDNAt+KCR7IaRjMY4+K9KG2izs66iPN59+XMv65OT+9a/s+L83T 4Shg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=klfrzuCLyyxZegIauwCuagWJDGwvyTnc1kN1VPYYoyI=; b=jKejeNKmL27LwiyLB/AYfgd/WlPr6xNQY6j8IFGMqVoZ9PiGoyjBmai9DOz4obsS7J QNLlpPWfkgmiWJWhu5rIH5BPDHPsq6SiFpcn5hw4hCAZ3B21FsixmBkdIknOwyOC/eVy fDmr4YhYDur2K1bOxSZJRk+eZjTzAUDJn/N93cpO1KjmOlXaGoDjxwbF52AWk5nKeZ0I ++OdjjOI7bR4B7P5JgdDjsJ6EJgcligP0IAafzygF6xnoyqepCg0+fm4I3l/Obk3Rnlk P19gHFLrrKQKqoQGgUAW63jEXapY8dKhh85Y53Rc1Ed/XyfZMSN9TnWRLWwLNdAsuEP2 08fA== X-Gm-Message-State: AMCzsaVOll4vSFIVpD0KSDKI37J/OFvU69I94CIRMviNjtHDnR3Lv5CP lmSTLNSjsh7jQoQY8FMBTwt+XJNCuXWzTwZA3Hg= X-Google-Smtp-Source: ABhQp+Rse0PgNd2NhZS+j0Q3LF3HRP7BtMgKvcNCq1qv/BTP+lLmU5x1qltGKwexyjyy77AD8AWgDstnpazKhykCz+I= X-Received: by 10.237.33.186 with SMTP id l55mr1914747qtc.71.1509444314202; Tue, 31 Oct 2017 03:05:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.237.52.198 with HTTP; Tue, 31 Oct 2017 03:05:13 -0700 (PDT) In-Reply-To: <20171031095920.5jn7ib63ofdm7wwf@mguzik> References: <1509243567.56824.103.camel@freebsd.org> <3a71dd31-99cb-c891-9d52-a7f2e7010011@FreeBSD.org> <1509293552.21609.5.camel@freebsd.org> <1509294247.21609.12.camel@freebsd.org> <7b59ff3d-3458-0bca-e6b4-13454b13efb0@FreeBSD.org> <1509296624.21609.24.camel@freebsd.org> <0fad1391-726d-8215-075d-9411abdf6edb@selasky.org> <1509379503.21609.103.camel@freebsd.org> <54876119-9262-c15b-d3fe-1220cf7f17f6@FreeBSD.org> <20171031095920.5jn7ib63ofdm7wwf@mguzik> From: Mateusz Guzik Date: Tue, 31 Oct 2017 11:05:13 +0100 Message-ID: Subject: Re: Allow faster eventhandler dispatching by keeping pointers to handler lists. To: Matt Joras Cc: Ian Lepore , Hans Petter Selasky , "freebsd-arch@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Oct 2017 10:05:15 -0000 On Tue, Oct 31, 2017 at 10:59 AM, Mateusz Guzik wrote: > On Mon, Oct 30, 2017 at 09:38:49AM -0700, Matt Joras wrote: > > On 10/30/2017 09:05, Ian Lepore wrote: > > > On Mon, 2017-10-30 at 09:17 +0100, Hans Petter Selasky wrote: > > >> On 10/29/17 18:03, Ian Lepore wrote: > > >>> Oh. Right. Hmmm, I think malloc() is required to support the case > > >>> where a handler registers before the static list init is invoked, > and I > > >>> do think that's a useful feature to preserve. That means the lists > > >>> aren't really static, though, which then makes STATIC a bit out of > > >>> place in the new function/macro names. > > >> Why not use RCU here, and then update sys/queue.h to be RCU safe? > > >> > > >> --HPS > > > I'm not sure how that suggestion relates to that paragraph, but the > > > code already does use a form of RCU to handle deletion of registered > > > handlers from individual event lists. It could probably benefit from a > > > rewrite to more fully embrace RCU and use it for insertions as well, > > > but that's beyond the scope of these changes, which are trying to > > > eliminate the worst uses of a global lock at a higher level than > > > maintaining the list for a single event. > > > > > > -- Ian > > > > Agreed. There are all sorts of clever things we can do with a more > > comprehensive rewrite. It would be a nice place to try out RCU-ish > > methods of handling object removal without locks. > > The current EH code delightet us long enough. It has to be dropped. > > delighted even > I don't think RCU (which we don't have anyway, apart from a rather poor > substitute) is suitable for use here. > > First off, vast majority of all lists are never going away and most > callbacks are not going away either. > > The gist is that in the worst case we can use rmlocks, which boil down > to per-cpu locks. Employing rcu would require refing and unrefing which > slows things down due to cacheline bounces. > > So here is is roughly how I see it: > 1. always-existing list + never-removed callbacks > > Note that here callbacks can be *added*, but never removed. If > implemented as a list it is really trivial with what hps referred to as > "rcu safe" macros - you set everything up and update the ->next pointer > last. Then the list is always safe to traverse with the worst case being > you finished before the addition of the next element completed. But you > never see half-constructed anything. > > 2. always-existing list + removable callbacks > > They can be stored *separately* to never-removed callbacks. Here is > where rmlocks come to play. An important detail is that the current eh > mechanism allows deregeistration to be performed while *within* a > callback. I think that's a rather weird-ass feature which needs to be > dropped from the replacement. I don't see any added value and I see > *loss* in that things have to get refed/unrefed and stuff relocked. > > 3. lists which can disappear > > Once more rmlocks. > I can't stress enough that writing to shared areas is a major performance problem on multicore systems. Code which does it for no good reason is an excellent candidate for a revamp. I don't see how said writing can be avoided while providing the current EH guarantee that things can be deregistered from the callback. One more reason to create a *new* mechanism from scratch and let the old code die off. -- Mateusz Guzik