From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 05:05:23 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E9F1106566C for ; Mon, 24 Mar 2008 05:05:23 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6E7798FC26 for ; Mon, 24 Mar 2008 05:05:23 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2O55Kgs022202 for ; Mon, 24 Mar 2008 01:05:21 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sun, 23 Mar 2008 19:06:40 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org Message-ID: <20080323185402.C910@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: buffer queue lock contention X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 05:05:23 -0000 http://people.freebsd.org/~jeff/bqlock.diff This patch makes a seperate lock for each queue of the buffer cache. It is intended to reduce contention on the single bqlock. In most cases this is quite trivial however getnewbuffer() warrants some discussion. In getnewbuffer() we walk several queues in order looking for a buffer to recycle or use directly. With a single lock you're guaranteed that as you're looking at one queue a new buffer won't come available on an earlier queue. You're also guaranteed that you can sleep atomically while waiting for new buffers. In my patch the queues are independently synchronized and so these guarantees are no longer present. A thread could scan each list sucessively and not find any suitable buffers and then go to sleep even after a new buffer had been posted to one of the other queues. I don't believe this is a significant problem because the very next brelse is likely to wakeup the getnewbuf sleeper. With a buffer cache under suitable pressure the wakeups are likely to happen quite frequently. The other part of this diff is to avoid acquiring the global 'needs buffer' and 'running buf request' locks when the variables they protect are not set. This would make it possible for one thread to go to sleep simultaneously with another thread releasing a buffer that would wake it up and this wakeup would be missed. However, as in the bqlock case, another brelse is not far behind to unstick this process. Presently we call bufcountwakeup() and bufspacewakeup() on almost every call to brelse() and thus acquire these global locks several times. When we've actually done io we acquire a third global lock with runningbufwakeup(). All of these global locks effectively add latency and cost to each io. Thanks, Jeff