From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 3 11:55:25 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 838DC106566B for ; Tue, 3 Jun 2008 11:55:25 +0000 (UTC) (envelope-from keramida@ceid.upatras.gr) Received: from igloo.linux.gr (igloo.linux.gr [62.1.205.36]) by mx1.freebsd.org (Postfix) with ESMTP id 9D6AB8FC1C for ; Tue, 3 Jun 2008 11:55:24 +0000 (UTC) (envelope-from keramida@ceid.upatras.gr) Received: from kobe.laptop (adsl110-203.kln.forthnet.gr [77.49.229.203]) (authenticated bits=128) by igloo.linux.gr (8.14.3/8.14.3/Debian-4) with ESMTP id m539kjGS000723 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 3 Jun 2008 12:46:53 +0300 Received: from kobe.laptop (kobe.laptop [127.0.0.1]) by kobe.laptop (8.14.2/8.14.2) with ESMTP id m539kioR007858; Tue, 3 Jun 2008 12:46:45 +0300 (EEST) (envelope-from keramida@ceid.upatras.gr) Received: (from keramida@localhost) by kobe.laptop (8.14.2/8.14.2/Submit) id m539khV3007857; Tue, 3 Jun 2008 12:46:43 +0300 (EEST) (envelope-from keramida@ceid.upatras.gr) From: Giorgos Keramidas To: Garance A Drosihn References: <200805281446.m4SEkojn099133@lurza.secnetix.de> <64200F15-4444-44FE-B904-673543441F35@FreeBSD.org> <4844751C.80704@FreeBSD.org> Date: Tue, 03 Jun 2008 12:46:43 +0300 In-Reply-To: (Garance A. Drosihn's message of "Mon\, 2 Jun 2008 21\:07\:15 -0400") Message-ID: <87prqykfzw.fsf@kobe.laptop> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-MailScanner-ID: m539kjGS000723 X-Hellug-MailScanner: Found to be clean X-Hellug-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-3.758, required 5, autolearn=not spam, ALL_TRUSTED -1.80, AWL 0.64, BAYES_00 -2.60) X-Hellug-MailScanner-From: keramida@ceid.upatras.gr X-Spam-Status: No Cc: freebsd-hackers@freebsd.org, ssouhlal@freebsd.org Subject: Re: Impact of having a large number of open file descriptors X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 11:55:25 -0000 On Mon, 2 Jun 2008 21:07:15 -0400, Garance A Drosihn wrote: > At 12:33 AM +0200 6/3/08, Kris Kennaway wrote: >>Ivan Voras wrote: >>>Suleiman Souhlal wrote: >>> >>>> I have an old patch that makes kqueue monitor every file write on >>>> the system and return the inode number in the knote's data field: >>>> http://people.freebsd.org/~ssouhlal/testing/kqueue-anyvnode-20050503.diff >>>> . >>>> >>>>I'd think it shouldn't be too hard to make it per-mountpoint.. >> >> FWIW, I would love to use this. I have situations where I have huge >> numbers of files and need to cheaply detect changes so I can >> resynchronize them to remote machines. > > I remember a discussion of changes to MacOS10 in Leopard which made it > easier to implement features such as Spotlight and TimeMachine. The > description starts here, I think: > > http://arstechnica.com/reviews/os/mac-os-x-10-5.ars/7 > > the section on file-system events. > > The idea I thought was interesting was to save the metadata on a > directory basis, instead of saving it on the file. So, if file > /some/dir/fname was changed, then they'd record that *some* file under > /some/dir has changed. > > So when your userland process comes along later on, it still has to > scan all files in that directory to see which file(s) actually > changed. But that's a lot less work than scanning all files in the > filesystem, and it also means there is much less data that has to be > kept track of. > > I have no idea how easy it would be to implement something similar on > FreeBSD, but the strategy seemed like a pretty neat idea. It sounds like a useful compromise between the number of tracked entries and scanning the entire fs :)