From owner-freebsd-arch@FreeBSD.ORG Thu Aug 16 06:58:27 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33F3216A421 for ; Thu, 16 Aug 2007 06:58:27 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id E14C513C428 for ; Thu, 16 Aug 2007 06:58:26 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.103] (c-67-160-44-208.hsd1.wa.comcast.net [67.160.44.208]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l7G6wNTV035703 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Thu, 16 Aug 2007 02:58:24 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Thu, 16 Aug 2007 00:01:18 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: arch@freebsd.org Message-ID: <20070815233852.X568@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: file locking. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2007 06:58:27 -0000 I have been looking at file locking for 8.0 and have come up with a way to completely eliminate the file lock, reduce the size of struct file by ~4 pointers (22%), remove the global list of files and associated lock, and restrict the scope of unp_gc while removing several race conditions. The whole thing hinges on reducing the complexity and scope of unp_gc to remove several fields from struct file. The remaining parts can be protected by atomics or are already protected by other locks. f_count and f_type are now completely updated using atomics. The ref counting with atomics results in significantly fewer atomics and cheaper fhold/fdrop. Protecting f_type was only complicated in cases where there were compound operations done on it, which are now implemented with atomic_cmpset_int loops. The unix domain socket garbage collection was changed to scan the list of unp sockets rather than the list of all files. This code is only responsible for finding dead cycles of unp sockets which reference each other. Evaluating other descriptors is not necessary. This allowed me to move f_gcflag and f_msgcount into unpcb. This also removed a use of the global filelist allowing me to remove two pointers from struct file. The only negative part of the new algorithm is that a back-pointer to the referencing struct file must be stored in any unix domain socket that is referenced via another in a rights message. It is a slight layering violation but there is only ever 1 file for each unix domain socket so it is correct. This is required because the garbage collection algorithm needs to know about the external references via the file. The patch is available at: http://people.freebsd.org/~jeff/fd.diff pho and kris have both tested this and found it to be stable. Kris has done some performance measurement and found it to be a win on microbenchmarks. I can't imagine a case that would actually be slower, except perhaps endless loops of sysctl kern.file. This also removes several sources of contention for multithreaded applications in particular, but also a global lock on allocating/freeing files. This also resolves several cases where f_flag was not protected when it should be, as well as removing race conditions in the garbage collector code due to dropping locks, and fixing unprotected variables in the garbage collection code. I intend to commit this soon after the 7.0 branch is made. I also have the final revision of my improved select locking ready. Thanks, Jeff