From owner-freebsd-fs@FreeBSD.ORG Sat Mar 26 09:35:54 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ABBCE16A4CE for ; Sat, 26 Mar 2005 09:35:54 +0000 (GMT) Received: from salmon.maths.tcd.ie (salmon.maths.tcd.ie [134.226.81.11]) by mx1.FreeBSD.org (Postfix) with SMTP id C72D943D54 for ; Sat, 26 Mar 2005 09:35:53 +0000 (GMT) (envelope-from dwmalone@maths.tcd.ie) Received: from walton.maths.tcd.ie by salmon.maths.tcd.ie with SMTP id ; 26 Mar 2005 09:35:52 +0000 (GMT) To: Scott Long In-reply-to: Your message of "Fri, 25 Mar 2005 21:54:21 MST." <4244EAFD.1030304@samsco.org> X-Request-Do: Date: Sat, 26 Mar 2005 09:35:52 +0000 From: David Malone Message-ID: <200503260935.aa92067@salmon.maths.tcd.ie> cc: freebsd-fs@freebsd.org Subject: Re: UFS Subdirectory limit. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Mar 2005 09:35:54 -0000 [I'm not sure if Scott's comments were in relation to David Schultz's suggestion or mine...] > I thought that we already discussed this in the past year. There are > significant compatibility concerns here. What happens if you use an > old fsck binary on a new filesystem? Since you haven't changed the > magic, it has no way of knowing that nlink needs to be handled > differently. In the scheme I suggested, a flag in the superblock is used to record when the new scheme is in use. I guess das's scheme would do something similar. In my case, when you run an old fsck, the link counts are just set to their traditional values by fsck (unless you have a directory with too many links, in which case fsck will do whatever it has always done in this situation). (You also don't need to turn such a feature on by default. Both schemes can also be read-only compatible with old systems too.) > Also, the more important > concern is that large directories simply don't scale in UFS. Lookups > are a linear operation, and while DIRHASH helps, it really doesn't scale > well to 150k entries. It seems to work passably well actually, not that I've benchmarked it carefully at this size. My junkmail maildir has 164953 entries at the moment, and is pretty much continiously appended to without creating any problems for the machine it lives on. Dirhash doesn't care if the entries are subdirectories or files. If the directory entries are largely static, the name cache should do all the work, and it is well capable of dealing with lots of files. We should definitely look at what sort of filesystem features we're likely to need in the future, but I just wanted to see if we can offer people a sloution that doesn't mean waiting for FreeBSD 6 or 7. (164955 files now ;-) David.