From owner-freebsd-arch@FreeBSD.ORG Fri Aug 3 02:03:56 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17DD716A41B; Fri, 3 Aug 2007 02:03:56 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id BC0A313C480; Fri, 3 Aug 2007 02:03:55 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l7323rr4042573 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Thu, 2 Aug 2007 22:03:54 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Thu, 2 Aug 2007 19:06:28 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Alfred Perlstein In-Reply-To: <20070803014445.GS92956@elvis.mu.org> Message-ID: <20070802190033.J561@10.0.0.1> References: <20070702230728.E552@10.0.0.1> <20070703181242.T552@10.0.0.1> <20070704105525.GU45894@elvis.mu.org> <20070704114005.X552@10.0.0.1> <20070729180722.GB85196@rot26.obsecurity.org> <20070802174819.S561@10.0.0.1> <20070803014445.GS92956@elvis.mu.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, Kris Kennaway Subject: Re: Fine grain select locking. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2007 02:03:56 -0000 On Thu, 2 Aug 2007, Alfred Perlstein wrote: > * Jeff Roberson [070802 17:52] wrote: >> >> I believe filedescriptor locking is the place where we are most lacking. >> The new sx helped tremendously. However, this is still going to be a >> scalability limiter. I have looked into both linux and solaris's solution >> to this problem. Briefly, linux uses RCU to protect the list, which is >> close to ideal as this is certainly a read heavy workload. Solaris on the >> other hand uses the actual file lock to protect the descriptor slot. So >> they fetch the file pointer, lock it, and then check to see if they lost a >> race with the slot being reassigned while they were acquiring the lock. >> This approach is perhaps better than rcu in many cases except when the >> descriptor set is expanded. Then they have to lock every file in the set. > > Certainly this is an extreme edge case... ? Well that may be true, yes. However, there are other problems with this scheme. For example, flags settings could be done entirely with cmpset, without using a lock at all. In most cases we're just setting a bit which can be done with atomic_set. When we're doing multiple operations we could compute the value and attempt to est it in a loop. So we can totally eliminate locking the descriptor here. We also could use atomic ops to protect the file descriptor reference count. This would eliminate another use of the FILE_LOCK(). I'm not sure if it's possible to merge this with an approach that uses the FILE_LOCK() to protect the descriptor table. Although I've not thought it all the way through. If the ref count and flags were done with atomics the main consumer of FILE_LOCK would actually be the unix domain socket garbage collection code. How's that for old unix baggage. Do many programs actually pass around descriptors these days? inetd? others? It might be worth it to lock this seperately from the file lock. Anyway, these things need to be explored for 8.0. With more cores and more multi-threaded applications file desc locking is one of the first points of contention as evidenced by the lock profiles and the sophistication of the solutions in other kernels. Jeff > > I could see it happening if we started with low limits, but perhaps > by keeping counters/stats we could tell people how to tune their > systems, or even autotune them. > > -Alfred >