From owner-freebsd-current@FreeBSD.ORG Fri Jul 8 19:41:23 2005 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B063C16A41C for ; Fri, 8 Jul 2005 19:41:23 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 671D343D53 for ; Fri, 8 Jul 2005 19:41:23 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 93EA646B17; Fri, 8 Jul 2005 15:41:22 -0400 (EDT) Date: Fri, 8 Jul 2005 20:41:22 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Luigi Rizzo In-Reply-To: <20050708110742.A6284@xorpc.icir.org> Message-ID: <20050708203537.H34251@fledge.watson.org> References: <20050705053114.A96381@xorpc.icir.org> <35386.1120575587@phk.freebsd.dk> <20050705103353.A8185@xorpc.icir.org> <20050708110742.A6284@xorpc.icir.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: s223560@studenti.ing.unipi.it, current@freebsd.org Subject: Re: location of bioq lock X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2005 19:41:23 -0000 On Fri, 8 Jul 2005, Luigi Rizzo wrote: > 1) put the lock in the struct bio_queue_head. > This is the same thing done in struct g_bioq defined in > sys/geom/geom.h . Quite clean, except that perhaps some > users of bio_queue_head still run under Giant (e.g. cam/scsi ?) > and so it is not possible to 'bring in' the lock. > > 2) change bioq_init() so that it takes also a pointer to the mtx > that protects the queue. > This is probably less clean, but perhaps a bit more flexible because > the queue and its lock are decoupled. Also it permits to deal > with the 'Giant' case easily. > > Other ideas ? In the network stack work, we started out with locks tightly coupled with the queues they protected as part of an early design decision to embed mutexes in ifqueue's, one of the widely used queueing structures. We're actually exploring backing off that decision now such that components use queues as a "library", and use their own synchronization to protect the queue. This allows, for example, lock coalescing across multiple queues, or combining of queue mutexes with larger component locks. It also allows queues to be agnostic of the lock type that is used to protect them, so a consumer could use an ex/rwlock or the like, or for that matter a spin lock. It also allows lock-free access to the queue where that is appropriate. FYI, one interesting point regarding lock order: in the ifqueue lock model, ifqueue locks were leaf mutexes. However, if the scope of locks protection queues expands, the replacement locks may well not be leaf mutexes. This is relevant in "hand-off" scenarios, where before contention on a queue mutex was very unlikely during a "grab, insert, drop" scenario, contention chances are increased as the lock might also cover other significantly time-consuming things, such as data copies, hardware I/O interactions, and so on. This might, or might not, outweight the overhead of the additional locking, and is worth keeping in mind. So based on that experience, my suggestion is to make locking a property of the consumer of the API, not the provider, and to create macros or queue wrapper functions in the consumer that *are* aware of the locking semantics, for the purposes of code simplification, assertion placement, and so on. However, the network stack is fairly different from the storage I/O stack, so the lessons (while interesting) might well not hold there. Robert N M Watson