From owner-freebsd-hackers Wed Sep 2 19:36:53 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id TAA26845 for freebsd-hackers-outgoing; Wed, 2 Sep 1998 19:36:53 -0700 (PDT) (envelope-from owner-freebsd-hackers@FreeBSD.ORG) Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id TAA26839 for ; Wed, 2 Sep 1998 19:36:51 -0700 (PDT) (envelope-from tlambert@usr07.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.8.8/8.8.8) id TAA05813; Wed, 2 Sep 1998 19:35:46 -0700 (MST) Received: from usr07.primenet.com(206.165.6.207) via SMTP by smtp03.primenet.com, id smtpd005741; Wed Sep 2 19:35:36 1998 Received: (from tlambert@localhost) by usr07.primenet.com (8.8.5/8.8.5) id TAA07404; Wed, 2 Sep 1998 19:35:32 -0700 (MST) From: Terry Lambert Message-Id: <199809030235.TAA07404@usr07.primenet.com> Subject: Re: Reading/writing /usr/ports is VERY slow To: mike@smith.net.au (Mike Smith) Date: Thu, 3 Sep 1998 02:35:32 +0000 (GMT) Cc: tlambert@primenet.com, cmascott@world.std.com, hackers@FreeBSD.ORG In-Reply-To: <199809021853.SAA01561@dingo.cdrom.com> from "Mike Smith" at Sep 2, 98 06:53:31 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > This is only useful if the layout has clustered related directories on > the disk. The current code does a good job of keeping them a long way > apart. > > > In fact, directory locality is loosely assureed. > > Almost the opposite is true, especially if you don't make a common > practice of deleting lots of directories. I think we are miscommunicating here. Directory *contents* are loosely assured locality. Directory *heirarchies* are loosely assured near-perfect random non-locality. Per the "find" command argument, this issue is "breadth-first" vs. "depth-first". > > I suspect that you are using a disk that either does not support > > tagged command queues, or supports them, but they are not enabled. > > Multiple device openings are unuseful in this case, as you previously > noted the lookup code doesn't (can't) attempt to read-ahead (it can't > know where /a/b is until it has read /a, etc.) If you are performing a > set of directory recursions, I/O (modulo the cache) will be performed > strictly sequentially. OK. Yes, this breaks down quickly. Either reimplement the directory code as a btree, or, better, "avoid writing code that results in diretory recursions". I think the ports problem in this respect is that the directory entry cache is too small for the traversal being done. Basically, "some idiot is using the FS directory hierarchy as if it were a database index". 8-). The correct thing to do here would probably be to create a "fast find" index, and have find use this (SunOS 4.1 has this; you create the index at 3 am from cron -- 3am because that's when you break to go to Naugles for egg-and-bean burritos 8-)). Alternately, a "portfinder" port would help... Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message