From owner-freebsd-current@FreeBSD.ORG Wed Jun 21 18:46:34 2006 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 41F1416A479 for ; Wed, 21 Jun 2006 18:46:34 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED77343D46 for ; Wed, 21 Jun 2006 18:46:33 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 8705246B3B; Wed, 21 Jun 2006 14:46:33 -0400 (EDT) Date: Wed, 21 Jun 2006 19:46:33 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John-Mark Gurney In-Reply-To: <20060621183543.GC82074@funkthat.com> Message-ID: <20060621194412.N8526@fledge.watson.org> References: <20060612054115.GA42379@xor.obsecurity.org> <20060621183543.GC82074@funkthat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: current@FreeBSD.org, Kris Kennaway Subject: Re: FILEDESC_LOCK() implementation X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2006 18:46:34 -0000 On Wed, 21 Jun 2006, John-Mark Gurney wrote: > Kris Kennaway wrote this message on Mon, Jun 12, 2006 at 01:41 -0400: >> I fixed mutex profiling to a) not be as wrong and b) not suck so very much, >> and here is a revised profiling trace from mysql supersmack on a 12 cpu >> E4500, sorted by ratio of cnt_lock/count; filedesc lock contention (via >> FILEDESC_[UN]LOCK()) is the major mutex contention problem. > > Should we also look at breaking down filedesc lock to have multiple locks > over the range? I am thinking of writing a program that will have 32 > threads (sun4v) and all threads will be doing heavy i/o, and will be even > more heavily contested on FILEDESC than the supersmack benchmark would be... > > Though this doesn't solve the problem of all 32 threads trying to do i/o on > a fd in the same block though... src/tools/tools/netrate/{http,httpd}, running in threaded mode (-t). http is a client, and accesses lots of independent fds from different threads, contending the filedesc lock but not a single fd lock, whereas httpd will do both, due to accepting connections. I would optimize very carefully here, the trade-offs are tricky, and we may find that by making locking more complex, we cause cache problems, increase lock hold periods, etc, even if we decrease contention. I've wondered a bit about a model where we loan fd's to threads to optimize repeated access to the same fd by the same thread, but this mostly makes sense in the context of a 1:1 model rather than an m:n model. Robert N M Watson Computer Laboratory University of Cambridge