From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 3 11:11:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86C841065680; Tue, 3 Jun 2008 11:11:11 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 572158FC1E; Tue, 3 Jun 2008 11:11:11 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 7DD1646C0D; Tue, 3 Jun 2008 07:11:10 -0400 (EDT) Date: Tue, 3 Jun 2008 12:11:10 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <484524CF.2090709@FreeBSD.org> Message-ID: <20080603120818.K426@fledge.watson.org> References: <200805281446.m4SEkojn099133@lurza.secnetix.de> <64200F15-4444-44FE-B904-673543441F35@FreeBSD.org> <4844751C.80704@FreeBSD.org> <20080603111935.K426@fledge.watson.org> <484524CF.2090709@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: ssouhlal@freebsd.org, Garance A Drosihn , freebsd-hackers@freebsd.org Subject: Re: Impact of having a large number of open file descriptors X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 11:11:11 -0000 On Tue, 3 Jun 2008, Kris Kennaway wrote: >> I think there's also considerable overlap with other kernel event systems, >> such as audit, and we might benefit from thinking seriously about enhancing >> those event systems rather than introducing a new one. The design of >> fsevents is pretty much entirely dictated by the needs of Spotlight and >> later Time Machine. In particular, it's not clear to me that the >> persistency requirements, which are a large part of the fsevents design, >> are important to us... or are they? > > Yes, I keep forgetting about audit for some reason :) It might be that this > is already good enough for my use case, although having to maintain a path > -> inode mapping for millions of files will be potentially onerous (same for > kevent anyway though). Persistency across reboots for unread events would > be nice but probably not essential (or worth the trouble). One interesting design choice in fsevents is that the event record is submitted on file close, rather than on file open. This doesn't properly handle writes due to memory mappings of the file, but does mean that you don't get the "Odd KDE PDF effect", in which KDE PDF viewer detects the open/write but not the close, so opens and refreshes the PDF before the file is completely rewritten, leading to a partial rendering. Audit captures open(2) and close(2), but only open(2) has a path on FreeBSD. This could be changed, however, to use a generated path on close(2), if available. Right now, there are quite a few file systems, especially synthetic ones such as procfs and devfs, which don't use the name cache, meaning that the path lookup doesn't work for them. Also, our name cache invalidation is pretty strict (to the point of being a performance problem in the NFS client), so path generation can be quite unreliable if it involves active directories. A mini-project in which someone improves the reliability of path generation is probably a prerequisite for any real work in this area. Robert N M Watson Computer Laboratory University of Cambridge