From owner-freebsd-arch@FreeBSD.ORG Mon Sep 24 21:23:22 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F1CA16A417; Mon, 24 Sep 2007 21:23:22 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail6.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id D749613C447; Mon, 24 Sep 2007 21:23:21 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8p) with ESMTP id 211191859-1834499 for multiple; Mon, 24 Sep 2007 17:21:18 -0400 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l8OLMZef001128; Mon, 24 Sep 2007 17:22:35 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: Jeff Roberson Date: Mon, 24 Sep 2007 17:22:28 -0400 User-Agent: KMail/1.9.6 References: <3bbf2fe10709221932i386f65b9h6f47ab4bee08c528@mail.gmail.com> <200709241152.41660.jhb@freebsd.org> <20070924135554.F547@10.0.0.1> In-Reply-To: <20070924135554.F547@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200709241722.28670.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Mon, 24 Sep 2007 17:22:35 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/4381/Mon Sep 24 13:20:51 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Attilio Rao , freebsd-smp@freebsd.org, freebsd-arch@freebsd.org Subject: Re: rwlocks: poor performance with adaptive spinning X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2007 21:23:22 -0000 On Monday 24 September 2007 04:57:06 pm Jeff Roberson wrote: > On Mon, 24 Sep 2007, John Baldwin wrote: > > > On Saturday 22 September 2007 10:32:06 pm Attilio Rao wrote: > >> Recently several people have reported problems of starvation with rwlocks. > >> In particular, users which tried to use rwlock on big SMP environment > >> (16+ CPUs) found them rather subjected to poor performances and to > >> starvation of waiters. > >> > >> Inspecting the code, something strange about adaptive spinning popped > >> up: basically, for rwlocks, adaptive spinning stubs seem to be > >> customed too down in the decisioning-loop. > >> The desposition of the stub will let the thread that would adaptively > >> spin, to set the respecitve (both read or write) waiters flag on, > >> which means that the owner of the lock will go down in the hard path > >> of locking functions and will performe a full wakeup even if the > >> waiters queues can result empty. This is a big penalty for adaptive > >> spinning which can make it completely useless. > >> In addiction to this, adaptive spinning only runs in the turnstile > >> spinlock path which is not ideal. > >> This patch ports the approach alredy used for adaptive spinning in sx > >> locks to rwlocks: > >> http://users.gufi.org/~rookie/works/patches/kern_rwlock.diff > >> > >> In sx it is unlikely to see big benefits because they are held for too > >> long times, but for rwlocks situation is rather different. > >> I would like to see if people can do benchmarks with this patch (maybe > >> in private environments?) as I'm not able to do them in short times. > >> > >> Adaptive spinning in rwlocks can be improved further with other tricks > >> (like adding a backoff counter, for example, or trying to spin with > >> the lock held in read mode too), but we first should be sure to start > >> with a solid base. > > > > I did this for mutexes and rwlocks over a year ago and Kris found it was > > slower in benchmarks. www.freebsd.org/~jhb/patches/lock_adapt.patch is the > > last thing I sent kris@ to test (it only has the mutex changes). This might > > be more optimal post-thread_lock since thread_lock seems to have heavily > > pessimized adaptive spinning because it now enqueues the thread and then > > dequeues it again before doing the adaptive spin. I liked the approach > > orginially because it simplifies the code a lot. A separate issue is that > > writers don't spin at all if a reader holds the lock, and I think one thing > > to test for that would be an adaptive spin with a static timeout. > > We don't enqueue the thread until the same place. We just acquire an > extra spinlock. The thread is not enqueued until turnstile_wait() as > before. Oh. That's what I get for assuming what trywait() and cancel() did based on their names. It is still more overhead than before though, so simplifying adaptive spinning might still be a win now as opposed to before. -- John Baldwin