From owner-freebsd-current@FreeBSD.ORG Wed Jun 21 21:51:47 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1660316A479; Wed, 21 Jun 2006 21:51:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5AD2843D72; Wed, 21 Jun 2006 21:51:46 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k5LLpexV042800; Wed, 21 Jun 2006 17:51:41 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-current@freebsd.org Date: Wed, 21 Jun 2006 17:45:41 -0400 User-Agent: KMail/1.9.1 References: <20060612054115.GA42379@xor.obsecurity.org> <20060621201927.GJ28128@groat.ugcs.caltech.edu> <20060621214346.G8526@fledge.watson.org> In-Reply-To: <20060621214346.G8526@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200606211745.42525.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Wed, 21 Jun 2006 17:51:42 -0400 (EDT) X-Virus-Scanned: ClamAV 0.87.1/1559/Wed Jun 21 10:23:13 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: Paul Allen , Robert Watson , current@freebsd.org Subject: Re: FILEDESC_LOCK() implementation X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2006 21:51:47 -0000 On Wednesday 21 June 2006 16:46, Robert Watson wrote: > On Wed, 21 Jun 2006, Paul Allen wrote: > > > From Robert Watson , Wed, Jun 21, 2006 at 07:46:33PM +0100: > >> I would optimize very carefully here, the trade-offs are tricky, and we > >> may find that by making locking more complex, we cause cache problems, > >> increase lock hold periods, etc, even if we decrease contention. I've > >> wondered a bit about a model where we loan fd's to threads to optimize > >> repeated access to the same fd by the same thread, but this mostly makes > >> sense in the context of a 1:1 model rather than an m:n model. > > I apologize for not understanding all of the uses of the FILEDESC lock > > but, isn't the more obvious partitioning per-cpu: each cpu may allocate > > from a range of fd, which cpu cache used depends on where the thread > > happens to be running. When closing a fd, it is returned to the local > > (possibly different cpu cache). A watermark is used to generate an IPI > > message to rebalance the caches as needed. > > The issue is actually a bit different than that. We in effect already do > the above using UMA. > > The problem is this: when you have threads in the same process, file > descriptor lookup is performed against a common file descriptor array. That > array is protected by a lock, the filedesc lock. When lots of threads > simultaneously perform file descriptor operations, they contend on the file > descriptor array lock. So if you have 30 threads all doing I/O, they are > constantly looking up file descriptors and bumping into each other. This is > particularly noticeable for network workloads, where many operations are > very fast, and so they occur in significant quantity. The M:N threading > library actually handles this quite well by bounding the number of threads > trying to acquire the lock to the number of processors, but with libthr you > get pretty bad performance. This contention problem also affects MySQL, > etc. > > You can imagine a number of ways to work on this, but it's a tricky problem > that has to be looked at carefully. Are the lookup operations using a shared lock so that only things like open and close would actually contend? -- John Baldwin