From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 11:35:37 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF36716A41C for ; Sun, 5 Jun 2005 11:35:37 +0000 (GMT) (envelope-from impasse@twokings.com) Received: from ppp83-237-226-201.pppoe.mtu-net.ru (ppp83-237-226-201.pppoe.mtu-net.ru [83.237.226.201]) by mx1.FreeBSD.org (Postfix) with SMTP id BE25E43D48 for ; Sun, 5 Jun 2005 11:35:35 +0000 (GMT) (envelope-from impasse@twokings.com) Received: from [194.110.31.40] (port=4290 helo=[rapture]) by ppp83-237-226-201.pppoe.mtu-net.ru with esmtp id 140381574clearings126868 for freebsd-fs@freebsd.org; Sun, 5 Jun 2005 15:35:33 +0400 Mime-Version: 1.0 (Apple Message framework v728) Content-Transfer-Encoding: 7bit Message-Id: <175176383.5618831193@ppp83-237-226-201.pppoe.mtu-net.ru> Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-fs@freebsd.org From: Odette Date: Sun, 5 Jun 2005 15:35:32 +0400 X-Mailer: Apple Mail (2.728) Subject: For professional documents use professional software... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 11:35:37 -0000 Don't be a fuddy-duddy...use the software everyone's using... http://tchb.tifqwat48lti8ub.aladfala9.com I have read your book and much like it. Fundamentalists are to Christianity what paint-by-numbers is to art. From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 15:36:20 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D329416A41C; Sun, 5 Jun 2005 15:36:20 +0000 (GMT) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id AFC2A43D54; Sun, 5 Jun 2005 15:36:20 +0000 (GMT) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.50 (FreeBSD)) id 1DexAi-000JKb-AZ; Sun, 05 Jun 2005 15:36:20 +0000 Received: from [127.0.0.1] (helo=roam.psg.com.psg.com) by roam.psg.com with esmtp (Exim 4.51 (FreeBSD)) id 1DexAc-000CxB-Ox; Sun, 05 Jun 2005 08:36:14 -0700 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17059.7150.269428.448187@roam.psg.com> Date: Sun, 5 Jun 2005 08:36:14 -0700 To: freebsd-fs@freebsd.org, FreeBSD Current Cc: Subject: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 15:36:21 -0000 to mirror someone's stuff, i need an fs which is happy with a jillion small files. they're linux geeks, so suggested reiserfs. but that appears (from -current's /sys/i386/conf/NOTES) to only have read- only support on freebsd. is there another path? randy From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 15:44:16 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6D54516A41F for ; Sun, 5 Jun 2005 15:44:16 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: from web41204.mail.yahoo.com (web41204.mail.yahoo.com [66.218.93.37]) by mx1.FreeBSD.org (Postfix) with SMTP id CB2DC43D58 for ; Sun, 5 Jun 2005 15:44:15 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 77740 invoked by uid 60001); 5 Jun 2005 15:44:15 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=VdghcDcV2OQbA3oVyPerTv5CxqHHwJvSSmXt6LeUbMGBa/qLKJTAQQFxNjyxp3Wlr4cdUpw0447HvKan+OMyV6OZfEyedWGp4DRmuwQvB7mmG5PxNiu7eHAdHd8zmY+NGwAQIOOSbjcWCjTmRl+BYZLnJ3wm56EkdM89kzryJvI= ; Message-ID: <20050605154415.77738.qmail@web41204.mail.yahoo.com> Received: from [83.129.184.237] by web41204.mail.yahoo.com via HTTP; Sun, 05 Jun 2005 08:44:14 PDT Date: Sun, 5 Jun 2005 08:44:14 -0700 (PDT) From: Arne "Wörner" To: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current In-Reply-To: <17059.7150.269428.448187@roam.psg.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 15:44:16 -0000 --- Randy Bush wrote: > to mirror someone's stuff, i need an fs which is happy with > a jillion small files. they're linux geeks, so suggested > reiserfs. but that appears (from -current's > /sys/i386/conf/NOTES) to only have read-only support on > freebsd. is there another path? > I say, do you think a database might be a solution (e. g. mysql)? Arne __________________________________ Do you Yahoo!? Yahoo! Mail - Find what you need with new enhanced search. http://info.mail.yahoo.com/mail_250 From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 15:47:15 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BDCA116A447; Sun, 5 Jun 2005 15:47:13 +0000 (GMT) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9531D43D58; Sun, 5 Jun 2005 15:47:13 +0000 (GMT) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.50 (FreeBSD)) id 1DexLF-000Jbb-9c; Sun, 05 Jun 2005 15:47:13 +0000 Received: from [127.0.0.1] (helo=roam.psg.com.psg.com) by roam.psg.com with esmtp (Exim 4.51 (FreeBSD)) id 1DexL9-000NLZ-Ey; Sun, 05 Jun 2005 08:47:07 -0700 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17059.7801.923721.861116@roam.psg.com> Date: Sun, 5 Jun 2005 08:47:05 -0700 To: Arne "Wörner" References: <17059.7150.269428.448187@roam.psg.com> <20050605154415.77738.qmail@web41204.mail.yahoo.com> Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 15:47:15 -0000 >> to mirror someone's stuff, i need an fs which is happy with >> a jillion small files. they're linux geeks, so suggested >> reiserfs. but that appears (from -current's >> /sys/i386/conf/NOTES) to only have read-only support on >> freebsd. is there another path? > I say, do you think a database might be a solution (e. g. mysql)? no. i am trying to do a site mirror, not tell the site how to organize their data. randy From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 18:03:19 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BA05816A41C; Sun, 5 Jun 2005 18:03:19 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from sp.dominia.org (efnet-math.org [69.60.109.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 625FE43D1F; Sun, 5 Jun 2005 18:03:19 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [192.168.1.12] (63-170-138-118.cst-sg.blacksburg.ntc-com.net [63.170.138.118]) (authenticated bits=0) by sp.dominia.org (8.13.1/8.13.1) with ESMTP id j55I3HxG015548 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Sun, 5 Jun 2005 14:03:18 -0400 Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Suleiman Souhlal Date: Sun, 5 Jun 2005 14:03:10 -0400 To: current@FreeBSD.org X-Mailer: Apple Mail (2.730) Cc: fs@FreeBSD.org Subject: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 18:03:19 -0000 Hi, IFS is basically FFS, minus the namespace. Files are referenced by their inode number, thus making file lookup very fast. This is very useful for things that keep the file names in their own database anyway, such as web or news caches. Files are created by opening the "newfile" file: fd = open("newfile", O_CREAT|O_RDWR, 0644); fstat(fd, &st); printf("new file is %d\n", (int)st.st_ino); Once a file has been created it is possible to accessing by just opening it the file named after the inode number. For example: fd = open("5", O_RDWR); This filesystem was imported back in the year 2000, by Adrian Chadd (adrian@) and removed in 2002, before the introduction of UFS2 in order not to slow down the development of UFS2. I have fixed it so that it is usable on -CURRENT. You can find the kernel bits at http://people.freebsd.org/~ssouhlal/testing/ ifs-20050605-2.diff and the userland bits at http:// people.freebsd.org/~ssouhlal/testing/ifs-20050605-userland-2.diff . (You need to copy src/sbin/fsck_ffs to src/sbin/fsck_ifs and src/sbin/ mount_ufs to src/sbin/mount_ifs, before applying the userland patch). You can use the regular newfs(8) to create such a filesystem. You might also want to specify the -n option, so that the .snap directory is not created at newfs, since directories are useless, in IFS. If you don't you'll end up with a useless inode, which is not really a big deal. Once the filesystem is created, you can mount it with mount_ifs. You can find additional comments in src/sys/ufs/ifs/README in the patch. Is anyone interested in seeing this committed? Bye, -- Suleiman Souhlal | ssouhlal@vt.edu The FreeBSD Project | ssouhlal@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 18:14:42 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 691BD16A41F; Sun, 5 Jun 2005 18:14:42 +0000 (GMT) (envelope-from matt@gsicomp.on.ca) Received: from skippyii.compar.com (skippyii.compar.com [216.208.38.130]) by mx1.FreeBSD.org (Postfix) with ESMTP id F279D43D4C; Sun, 5 Jun 2005 18:14:41 +0000 (GMT) (envelope-from matt@gsicomp.on.ca) Received: from hermes (CPE00062566c7bb-CM000039c69a66.cpe.net.cable.rogers.com [69.193.82.185]) by skippyii.compar.com (8.13.1/8.13.1) with ESMTP id j55IPTdw000564; Sun, 5 Jun 2005 14:25:30 -0400 (EDT) (envelope-from matt@gsicomp.on.ca) Message-ID: <003701c569fa$7e4e0530$1200a8c0@gsicomp.on.ca> From: "Matt Emmerton" To: "Randy Bush" , =?iso-8859-1?Q?Arne_W=F6rner?= References: <17059.7150.269428.448187@roam.psg.com><20050605154415.77738.qmail@web41204.mail.yahoo.com> <17059.7801.923721.861116@roam.psg.com> Date: Sun, 5 Jun 2005 14:15:02 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1437 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441 Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 18:14:42 -0000 > >> to mirror someone's stuff, i need an fs which is happy with > >> a jillion small files. they're linux geeks, so suggested > >> reiserfs. but that appears (from -current's > >> /sys/i386/conf/NOTES) to only have read-only support on > >> freebsd. is there another path? > > I say, do you think a database might be a solution (e. g. mysql)? > > no. i am trying to do a site mirror, not tell the site how to > organize their data. UFS[2] will be just fine -- but you will need to ensure that you have more inodes than the default and increase dirhash to keep performance at a reasonable level. -- Matt Emmerton From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 18:17:07 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 01E1616A41C for ; Sun, 5 Jun 2005 18:17:07 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from smtp4.server.rpi.edu (smtp4.server.rpi.edu [128.113.2.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA3B443D4C for ; Sun, 5 Jun 2005 18:17:04 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp4.server.rpi.edu (8.13.0/8.13.0) with ESMTP id j55IH2EO013840; Sun, 5 Jun 2005 14:17:03 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <17059.7150.269428.448187@roam.psg.com> References: <17059.7150.269428.448187@roam.psg.com> Date: Sun, 5 Jun 2005 14:17:01 -0400 To: Randy Bush , freebsd-fs@freebsd.org From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-CanItPRO-Stream: default X-RPI-SA-Score: undef - spam-scanning disabled X-Scanned-By: CanIt (www . canit . ca) on 128.113.2.4 Cc: Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 18:17:07 -0000 At 8:36 AM -0700 6/5/05, Randy Bush wrote: >to mirror someone's stuff, i need an fs which is happy with a jillion >small files. they're linux geeks, so suggested reiserfs. but that >appears (from -current's /sys/i386/conf/NOTES) to only have read- >only support on freebsd. is there another path? Do individual directories have jillions of files, or is a a large tree of files? Could you break it up into multiple partitions for your mirror of it, and 'newfs' each of those to hold plenty of extra inodes? -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 20:04:46 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6179516A41C; Sun, 5 Jun 2005 20:04:46 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from sp.dominia.org (efnet-math.org [69.60.109.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id CBEB643D58; Sun, 5 Jun 2005 20:04:45 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [192.168.1.12] (63-170-138-118.cst-sg.blacksburg.ntc-com.net [63.170.138.118]) (authenticated bits=0) by sp.dominia.org (8.13.1/8.13.1) with ESMTP id j55K4ibO016078 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Sun, 5 Jun 2005 16:04:44 -0400 In-Reply-To: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <67D7394E-A398-40D0-A2D6-9AFE569A4A3E@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Suleiman Souhlal Date: Sun, 5 Jun 2005 16:04:38 -0400 To: Suleiman Souhlal X-Mailer: Apple Mail (2.730) Cc: current@FreeBSD.org, fs@FreeBSD.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 20:04:46 -0000 On Jun 5, 2005, at 2:03 PM, Suleiman Souhlal wrote: > I have fixed it so that it is usable on -CURRENT. You can find the > kernel bits at http://people.freebsd.org/~ssouhlal/testing/ > ifs-20050605-2.diff and the userland bits at http:// > people.freebsd.org/~ssouhlal/testing/ifs-20050605-userland-2.diff . > (You need to copy src/sbin/fsck_ffs to src/sbin/fsck_ifs and src/ > sbin/mount_ufs to src/sbin/mount_ifs, before applying the userland > patch). I forgot to mention that you also need to add "options IFS" to your kernel config file, if you want to use it. -- Suleiman Souhlal | ssouhlal@vt.edu The FreeBSD Project | ssouhlal@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 20:59:34 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CCFC616A41C; Sun, 5 Jun 2005 20:59:34 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 70DEF43D1F; Sun, 5 Jun 2005 20:59:31 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [172.28.176.26] ([12.174.84.3]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j55L5b9W047516; Sun, 5 Jun 2005 15:05:41 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A36767.6060301@samsco.org> Date: Sun, 05 Jun 2005 14:58:15 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Randy Bush References: <17059.7150.269428.448187@roam.psg.com> In-Reply-To: <17059.7150.269428.448187@roam.psg.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 20:59:34 -0000 Randy Bush wrote: > to mirror someone's stuff, i need an fs which is happy with a jillion > small files. they're linux geeks, so suggested reiserfs. but that > appears (from -current's /sys/i386/conf/NOTES) to only have read- > only support on freebsd. is there another path? > > randy > See Suleiman Souhal's patch to re-introduce IFS. IFS is ideal when the naming of files is not important, but quick access to 'zillions' of files is important. The downside is that your apps need to be written to know how to use it (which isn't hard, but isn't portable either). Scott From owner-freebsd-fs@FreeBSD.ORG Sun Jun 5 21:16:41 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A3E316A41C for ; Sun, 5 Jun 2005 21:16:41 +0000 (GMT) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32FEB43D54 for ; Sun, 5 Jun 2005 21:16:41 +0000 (GMT) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.50 (FreeBSD)) id 1Df2U4-0001hl-Ms; Sun, 05 Jun 2005 21:16:40 +0000 Received: from [127.0.0.1] (helo=roam.psg.com.psg.com) by roam.psg.com with esmtp (Exim 4.51 (FreeBSD)) id 1Df2Ty-000JSG-VB; Sun, 05 Jun 2005 14:16:35 -0700 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17059.27570.462035.618336@roam.psg.com> Date: Sun, 5 Jun 2005 14:16:34 -0700 To: Scott Long References: <17059.7150.269428.448187@roam.psg.com> <42A36767.6060301@samsco.org> Cc: freebsd-fs@freebsd.org Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jun 2005 21:16:41 -0000 >> to mirror someone's stuff, i need an fs which is happy with a jillion >> small files. they're linux geeks, so suggested reiserfs. but that >> appears (from -current's /sys/i386/conf/NOTES) to only have read- >> only support on freebsd. is there another path? > See Suleiman Souhal's patch to re-introduce IFS. IFS is ideal when the > naming of files is not important, but quick access to 'zillions' of > files is important. The downside is that your apps need to be written > to know how to use it (which isn't hard, but isn't portable either). the app is apache randy From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 03:31:48 2005 Return-Path: X-Original-To: fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2819316A41C; Mon, 6 Jun 2005 03:31:48 +0000 (GMT) (envelope-from sobomax@web.portaone.com) Received: from www.portaone.com (web.portaone.com [195.70.151.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7CA5843D4C; Mon, 6 Jun 2005 03:31:46 +0000 (GMT) (envelope-from sobomax@web.portaone.com) Received: from www.portaone.com (localhost [127.0.0.1]) by www.portaone.com (8.12.11/8.12.11) with ESMTP id j563VjQC081205 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Jun 2005 05:31:45 +0200 (CEST) (envelope-from sobomax@web.portaone.com) Received: (from sobomax@localhost) by www.portaone.com (8.12.11/8.12.11/Submit) id j563Vj8F081204; Mon, 6 Jun 2005 05:31:45 +0200 (CEST) (envelope-from sobomax) Date: Mon, 6 Jun 2005 05:31:45 +0200 From: Maxim Sobolev To: Suleiman Souhlal Message-ID: <20050606033145.GA80739@www.portaone.com> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> User-Agent: Mutt/1.5.8i X-Virus-Scanned: ClamAV 0.83/915/Sun Jun 5 20:15:22 2005 on www.portaone.com X-Virus-Status: Clean Cc: current@FreeBSD.ORG, fs@FreeBSD.ORG Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 03:31:48 -0000 Are there any real users of this FS? If my memory serves, Adrian's plan was to add IFS support into the squid. I wonder if it had happened and whether or not it actually provides any real performance benefit. -Maxim On Sun, Jun 05, 2005 at 02:03:10PM -0400, Suleiman Souhlal wrote: > Hi, > > IFS is basically FFS, minus the namespace. Files are referenced by > their inode number, thus making file lookup very fast. This is very > useful for things that keep the file names in their own database > anyway, such as web or news caches. > > Files are created by opening the "newfile" file: > > fd = open("newfile", O_CREAT|O_RDWR, 0644); > fstat(fd, &st); > printf("new file is %d\n", (int)st.st_ino); > > Once a file has been created it is possible to accessing by just > opening it the file named after the inode number. For example: > > fd = open("5", O_RDWR); > > This filesystem was imported back in the year 2000, by Adrian Chadd > (adrian@) and removed in 2002, before the introduction of UFS2 in > order not to slow down the development of UFS2. > > I have fixed it so that it is usable on -CURRENT. You can find the > kernel bits at http://people.freebsd.org/~ssouhlal/testing/ > ifs-20050605-2.diff and the userland bits at http:// > people.freebsd.org/~ssouhlal/testing/ifs-20050605-userland-2.diff . > (You need to copy src/sbin/fsck_ffs to src/sbin/fsck_ifs and src/sbin/ > mount_ufs to src/sbin/mount_ifs, before applying the userland patch). > > You can use the regular newfs(8) to create such a filesystem. You > might also want to specify the -n option, so that the .snap directory > is not created at newfs, since directories are useless, in IFS. If > you don't you'll end up with a useless inode, which is not really a > big deal. > Once the filesystem is created, you can mount it with mount_ifs. > > You can find additional comments in src/sys/ufs/ifs/README in the patch. > > Is anyone interested in seeing this committed? > > Bye, > -- > Suleiman Souhlal | ssouhlal@vt.edu > The FreeBSD Project | ssouhlal@FreeBSD.org > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 04:04:00 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A1ED116A41C; Mon, 6 Jun 2005 04:04:00 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from sp.dominia.org (efnet-math.org [69.60.109.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 39DB843D48; Mon, 6 Jun 2005 04:04:00 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [192.168.1.12] (63-170-138-118.cst-sg.blacksburg.ntc-com.net [63.170.138.118]) (authenticated bits=0) by sp.dominia.org (8.13.1/8.13.1) with ESMTP id j5643txr017815 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Mon, 6 Jun 2005 00:03:56 -0400 In-Reply-To: <20050606033145.GA80739@www.portaone.com> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Suleiman Souhlal Date: Mon, 6 Jun 2005 00:03:36 -0400 To: Maxim Sobolev X-Mailer: Apple Mail (2.730) Cc: current@FreeBSD.org, fs@FreeBSD.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 04:04:00 -0000 Hi Maxim, On Jun 5, 2005, at 11:31 PM, Maxim Sobolev wrote: > Are there any real users of this FS? If my memory serves, > Adrian's plan was to add IFS support into the squid. I > wonder if it had happened and whether or not it actually > provides any real performance benefit. AFAIK Adrian had experimental patches to make squid use it, but they were never actually imported into squid. -- Suleiman Souhlal | ssouhlal@vt.edu The FreeBSD Project | ssouhlal@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 04:17:48 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 91D6416A41C; Mon, 6 Jun 2005 04:17:48 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from smtp3.server.rpi.edu (smtp3.server.rpi.edu [128.113.2.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4622443D1F; Mon, 6 Jun 2005 04:17:48 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp3.server.rpi.edu (8.13.0/8.13.0) with ESMTP id j564HjaV013680; Mon, 6 Jun 2005 00:17:46 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> Date: Mon, 6 Jun 2005 00:17:44 -0400 To: Suleiman Souhlal , current@freebsd.org From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-CanItPRO-Stream: default X-RPI-SA-Score: undef - spam-scanning disabled X-Scanned-By: CanIt (www . canit . ca) on 128.113.2.3 Cc: fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 04:17:48 -0000 At 2:03 PM -0400 6/5/05, Suleiman Souhlal wrote: > >This filesystem was imported back in the year 2000, by Adrian >Chadd (adrian@) and removed in 2002, before the introduction >of UFS2 in order not to slow down the development of UFS2. >You can find additional comments in src/sys/ufs/ifs/README in >the patch. > >Is anyone interested in seeing this committed? It sounds to me like this would be interesting to have, as long as it wouldn't be much work to keep it up-to-date now. I have a vague notion that this might be very useful for AFS file servers, for instance. (although that would require some more work on the OpenAFS-side of things...) I haven't had a chance to try this yet, but I might try some tests with it next weekend. I could try it on PPC and Sparc64, in addition to i386. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 04:55:01 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2CFDC16A41C; Mon, 6 Jun 2005 04:55:01 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7992243D48; Mon, 6 Jun 2005 04:54:57 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [10.0.2.2] ([12.174.84.3]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j5650s2w049463; Sun, 5 Jun 2005 23:00:59 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A3D6CF.2000504@samsco.org> Date: Sun, 05 Jun 2005 22:53:35 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Maxim Sobolev References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> In-Reply-To: <20050606033145.GA80739@www.portaone.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 04:55:01 -0000 Maxim Sobolev wrote: > Are there any real users of this FS? If my memory serves, > Adrian's plan was to add IFS support into the squid. I > wonder if it had happened and whether or not it actually > provides any real performance benefit. > > -Maxim > IFS is centered around the observation that most apps already keep a metamapping of 'filename ABC = data XYZ', where the filename is really nothing more than a unique identifier for the data. Expressive filenames and file hierachies are useful for humans, but matter little to computers. The name "squid/cache/a/b/abc" has as much meaning to a computer as the name "12345" (which might be the inode number of the file). So taking this into account, you can strip out all the overhead of recursive namei lookups, directory data reads, directory hashing, etc, and just use the inode number directly as the identifier for inodes. Looking up by inode is a O(1) operation consisting of merely using the number to index into a inode array in a particular cylinder group. It's a huge win for CPU overhead in the filesystem, especially when we start talking about increasing the size of m_links field and possibly going to 64-bit inode numbers. Squid is an ideal test vehicle for this, and I hope that the rumored patches for it can re-materialize. However, it's a circular argument to say that the presence of IFS should rely on IFS apps being present; I'm very glad that Suleiman is working on this, and I hope it reaps some benefit. Scott > On Sun, Jun 05, 2005 at 02:03:10PM -0400, Suleiman Souhlal wrote: > >>Hi, >> >>IFS is basically FFS, minus the namespace. Files are referenced by >>their inode number, thus making file lookup very fast. This is very >>useful for things that keep the file names in their own database >>anyway, such as web or news caches. >> >>Files are created by opening the "newfile" file: >> >> fd = open("newfile", O_CREAT|O_RDWR, 0644); >> fstat(fd, &st); >> printf("new file is %d\n", (int)st.st_ino); >> >>Once a file has been created it is possible to accessing by just >>opening it the file named after the inode number. For example: >> >> fd = open("5", O_RDWR); >> >>This filesystem was imported back in the year 2000, by Adrian Chadd >>(adrian@) and removed in 2002, before the introduction of UFS2 in >>order not to slow down the development of UFS2. >> >>I have fixed it so that it is usable on -CURRENT. You can find the >>kernel bits at http://people.freebsd.org/~ssouhlal/testing/ >>ifs-20050605-2.diff and the userland bits at http:// >>people.freebsd.org/~ssouhlal/testing/ifs-20050605-userland-2.diff . >>(You need to copy src/sbin/fsck_ffs to src/sbin/fsck_ifs and src/sbin/ >>mount_ufs to src/sbin/mount_ifs, before applying the userland patch). >> >>You can use the regular newfs(8) to create such a filesystem. You >>might also want to specify the -n option, so that the .snap directory >>is not created at newfs, since directories are useless, in IFS. If >>you don't you'll end up with a useless inode, which is not really a >>big deal. >>Once the filesystem is created, you can mount it with mount_ifs. >> >>You can find additional comments in src/sys/ufs/ifs/README in the patch. >> >>Is anyone interested in seeing this committed? >> >>Bye, >>-- >>Suleiman Souhlal | ssouhlal@vt.edu >>The FreeBSD Project | ssouhlal@FreeBSD.org >> >>_______________________________________________ >>freebsd-current@freebsd.org mailing list >>http://lists.freebsd.org/mailman/listinfo/freebsd-current >>To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> >> > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 05:06:24 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9F5216A41C; Mon, 6 Jun 2005 05:06:24 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from sp.dominia.org (efnet-math.org [69.60.109.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3D89D43D53; Mon, 6 Jun 2005 05:06:24 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [192.168.1.12] (63-170-138-118.cst-sg.blacksburg.ntc-com.net [63.170.138.118]) (authenticated bits=0) by sp.dominia.org (8.13.1/8.13.1) with ESMTP id j56562pU018062 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Mon, 6 Jun 2005 01:06:20 -0400 In-Reply-To: <42A3D6CF.2000504@samsco.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Suleiman Souhlal Date: Mon, 6 Jun 2005 01:05:51 -0400 To: Scott Long X-Mailer: Apple Mail (2.730) Cc: Maxim Sobolev , current@FreeBSD.org, fs@FreeBSD.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 05:06:24 -0000 Hi, On Jun 6, 2005, at 12:53 AM, Scott Long wrote: > Maxim Sobolev wrote: > >> Are there any real users of this FS? If my memory serves, >> Adrian's plan was to add IFS support into the squid. I >> wonder if it had happened and whether or not it actually >> provides any real performance benefit. >> -Maxim >> > > IFS is centered around the observation that most apps already keep a > metamapping of 'filename ABC = data XYZ', where the filename is really > nothing more than a unique identifier for the data. Expressive > filenames and file hierachies are useful for humans, but matter > little to computers. The name "squid/cache/a/b/abc" has as much > meaning to a computer as the name "12345" (which might be the inode > number of the file). So taking this into account, you can strip > out all the overhead of recursive namei lookups, directory data reads, > directory hashing, etc, and just use the inode number directly as the > identifier for inodes. Looking up by inode is a O(1) operation > consisting of merely using the number to index into a inode array in a > particular cylinder group. It's a huge win for CPU overhead in the > filesystem, especially when we start talking about increasing the size > of m_links field and possibly going to 64-bit inode numbers. Talking about going to 64-bit inode numbers, how would we deal with the change in stat(2)? -- Suleiman Souhlal | ssouhlal@vt.edu The FreeBSD Project | ssouhlal@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 05:47:43 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3CB1E16A425; Mon, 6 Jun 2005 05:47:43 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from smtp4.server.rpi.edu (smtp4.server.rpi.edu [128.113.2.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 966E943E13; Mon, 6 Jun 2005 05:47:00 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp4.server.rpi.edu (8.13.0/8.13.0) with ESMTP id j565kwBi028174; Mon, 6 Jun 2005 01:46:59 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> Date: Mon, 6 Jun 2005 01:46:57 -0400 To: Suleiman Souhlal , Scott Long From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-CanItPRO-Stream: default X-RPI-SA-Score: undef - spam-scanning disabled X-Scanned-By: CanIt (www . canit . ca) on 128.113.2.4 Cc: current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 05:47:43 -0000 At 1:05 AM -0400 6/6/05, Suleiman Souhlal wrote: > >On Jun 6, 2005, at 12:53 AM, Scott Long wrote: > >>It's a huge win for CPU overhead in the filesystem, especially >>when we start talking about increasing the size of m_links >>field and possibly going 64-bit inode numbers. > >Talking about going to 64-bit inode numbers, how would we deal >with the change in stat(2)? By making some sort of incompatible change to stat(2). This has been discussed from time-to-time. It's another change that I would have liked to have seen (at least for the stat routines) in 6.0, but right now I suspect it will not happen until 7.0. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 06:27:36 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A678F16A41C; Mon, 6 Jun 2005 06:27:36 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from sp.dominia.org (efnet-math.org [69.60.109.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4052043D1D; Mon, 6 Jun 2005 06:27:36 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [192.168.1.12] (63-170-138-118.cst-sg.blacksburg.ntc-com.net [63.170.138.118]) (authenticated bits=0) by sp.dominia.org (8.13.1/8.13.1) with ESMTP id j566RYFg018352 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Mon, 6 Jun 2005 02:27:35 -0400 In-Reply-To: <20050606033145.GA80739@www.portaone.com> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Suleiman Souhlal Date: Mon, 6 Jun 2005 02:27:20 -0400 To: Maxim Sobolev X-Mailer: Apple Mail (2.730) Cc: current@FreeBSD.org, fs@FreeBSD.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 06:27:36 -0000 Hi, On Jun 5, 2005, at 11:31 PM, Maxim Sobolev wrote: > Are there any real users of this FS? If my memory serves, > Adrian's plan was to add IFS support into the squid. I > wonder if it had happened and whether or not it actually > provides any real performance benefit. I just thought about another thing IFS would be very useful for: Helping to develop UFS journaling. Since the filesystem itself is very simple, and directories are not present, it should be pretty easy to journal, as the number of places that update the metadata is pretty low. This should be extremely useful when testing out the journal operations, once they have been implemented. -- Suleiman Souhlal | ssouhlal@vt.edu The FreeBSD Project | ssouhlal@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 10:51:01 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9251D16A41C; Mon, 6 Jun 2005 10:51:01 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6BD0A43D1F; Mon, 6 Jun 2005 10:51:01 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dsl093-001-248.det1.dsl.speakeasy.net [66.93.1.248]) by citi.umich.edu (Postfix) with ESMTP id AFFBA1BBF8; Mon, 6 Jun 2005 06:51:00 -0400 (EDT) To: current@freebsd.org, fs@freebsd.org From: Jim Rees In-Reply-To: Scott Long, Sun, 05 Jun 2005 22:53:35 MDT Date: Mon, 06 Jun 2005 06:51:00 -0400 Sender: rees@citi.umich.edu Message-Id: <20050606105100.AFFBA1BBF8@citi.umich.edu> Cc: Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 10:51:01 -0000 OpenAFS could benefit from something like IFS, although I'm not volunteering to do the work. The client has complicated code to keep its thousands of cache files in a hierarchy just to reduce name lookup time, and the server originally used a special "open by inode" system call that had been hacked in to the kernel. From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 12:14:39 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5ACBA16A41C; Mon, 6 Jun 2005 12:14:39 +0000 (GMT) (envelope-from bms@spc.org) Received: from arginine.spc.org (arginine.spc.org [83.167.185.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A8EE43D49; Mon, 6 Jun 2005 12:14:37 +0000 (GMT) (envelope-from bms@spc.org) Received: from localhost (localhost [127.0.0.1]) by arginine.spc.org (Postfix) with ESMTP id 24A1D651F7; Mon, 6 Jun 2005 13:12:37 +0100 (BST) Received: from arginine.spc.org ([127.0.0.1]) by localhost (arginine.spc.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 94191-04; Mon, 6 Jun 2005 13:12:36 +0100 (BST) Received: from empiric.dek.spc.org (host81-134-123-217.in-addr.btopenworld.com [81.134.123.217]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by arginine.spc.org (Postfix) with ESMTP id 5FC33651EE; Mon, 6 Jun 2005 13:12:36 +0100 (BST) Received: by empiric.dek.spc.org (Postfix, from userid 1001) id AC59E6266; Mon, 6 Jun 2005 13:14:33 +0100 (BST) Date: Mon, 6 Jun 2005 13:14:33 +0100 From: Bruce M Simpson To: Jim Rees Message-ID: <20050606121433.GE734@empiric.icir.org> References: <20050606105100.AFFBA1BBF8@citi.umich.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050606105100.AFFBA1BBF8@citi.umich.edu> Cc: current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 12:14:39 -0000 On Mon, Jun 06, 2005 at 06:51:00AM -0400, Jim Rees wrote: > OpenAFS could benefit from something like IFS, although I'm not volunteering > to do the work. The client has complicated code to keep its thousands of > cache files in a hierarchy just to reduce name lookup time, and the server > originally used a special "open by inode" system call that had been hacked > in to the kernel. And it looks like we as a project are eating our words now, which is no bad thing:- http://mail-index.netbsd.org/tech-kern/2000/11/28/0030.html :-) Regards, BMS From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 13:47:50 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 25A1216A41C; Mon, 6 Jun 2005 13:47:50 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 93C8D43D53; Mon, 6 Jun 2005 13:47:46 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [10.0.2.2] ([12.174.84.3]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j56Dreaf052274; Mon, 6 Jun 2005 07:53:47 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A453B5.3020006@samsco.org> Date: Mon, 06 Jun 2005 07:46:29 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Garance A Drosihn References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 13:47:50 -0000 Garance A Drosihn wrote: > At 1:05 AM -0400 6/6/05, Suleiman Souhlal wrote: > >> >> On Jun 6, 2005, at 12:53 AM, Scott Long wrote: >> >>> It's a huge win for CPU overhead in the filesystem, especially >>> when we start talking about increasing the size of m_links >>> field and possibly going 64-bit inode numbers. >> >> >> Talking about going to 64-bit inode numbers, how would we deal >> with the change in stat(2)? > > > By making some sort of incompatible change to stat(2). This has > been discussed from time-to-time. It's another change that I > would have liked to have seen (at least for the stat routines) > in 6.0, but right now I suspect it will not happen until 7.0. > We can't go making incremental incompatibilities to the filesystem without a good deal of planning. This is the type of thing that would go into a 'UFS3'. I have some long-term plans here, but I need to get the initial proof-of-concept journalling working before I start to seriously consider what else would be in UFS3. Scott From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 14:46:14 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 81FF916A41C; Mon, 6 Jun 2005 14:46:14 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1BDE843D4C; Mon, 6 Jun 2005 14:46:13 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost.des.no [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D8EC260F2; Mon, 6 Jun 2005 16:45:58 +0200 (CEST) Received: from xps.des.no (des.no [80.203.228.37]) by tim.des.no (Postfix) with ESMTP id C393460F1; Mon, 6 Jun 2005 16:45:58 +0200 (CEST) Received: by xps.des.no (Postfix, from userid 1001) id B95AF33C3B; Mon, 6 Jun 2005 16:45:58 +0200 (CEST) To: Scott Long References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) Date: Mon, 06 Jun 2005 16:45:58 +0200 In-Reply-To: <42A453B5.3020006@samsco.org> (Scott Long's message of "Mon, 06 Jun 2005 07:46:29 -0600") Message-ID: <86oeaj1r2x.fsf@xps.des.no> User-Agent: Gnus/5.110002 (No Gnus v0.2) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on tim.des.no X-Spam-Level: X-Spam-Status: No, score=-4.8 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 14:46:14 -0000 Scott Long writes: > We can't go making incremental incompatibilities to the filesystem > without a good deal of planning. This is the type of thing that > would go into a 'UFS3'. This is primarily an API issue, not a filesystem layout issue. We already have at least one filesystem with 64-bit inodes (msdosfs). DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 14:57:12 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A87A616A41C; Mon, 6 Jun 2005 14:57:12 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 40B8343D1D; Mon, 6 Jun 2005 14:57:09 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [10.0.2.2] ([12.174.84.3]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j56F31j4052641; Mon, 6 Jun 2005 09:03:06 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A463EF.5060401@samsco.org> Date: Mon, 06 Jun 2005 08:55:43 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> In-Reply-To: <86oeaj1r2x.fsf@xps.des.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 14:57:12 -0000 Dag-Erling Smørgrav wrote: > Scott Long writes: > >>We can't go making incremental incompatibilities to the filesystem >>without a good deal of planning. This is the type of thing that >>would go into a 'UFS3'. > > > This is primarily an API issue, not a filesystem layout issue. We > already have at least one filesystem with 64-bit inodes (msdosfs). > > DES What do you mean it's not a layout issue? We can't make incompatible layout changes whever we feel like it, or else transportability of filesystems is completely lost and everyone who wants to boot more than just the Last And Greatest on their system winds up with unnessary pain. Anyways, I'm not looking for someone to get a wild idea that we need UFS3 right now. There are a bunch of features that would be ideal for UFS3 at a later date when we've had time to sit down and think about how to do it right. Going off and adjusting di_nlink to 32 bits and dot_ino to 64 bits and declaring that to be 'UFS3' is not a good strategy. Scott From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 15:24:41 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D593516A41C; Mon, 6 Jun 2005 15:24:41 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DB3443D4C; Mon, 6 Jun 2005 15:24:41 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost.des.no [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id CC90B60F0; Mon, 6 Jun 2005 17:24:34 +0200 (CEST) Received: from xps.des.no (des.no [80.203.228.37]) by tim.des.no (Postfix) with ESMTP id B46C360EF; Mon, 6 Jun 2005 17:24:34 +0200 (CEST) Received: by xps.des.no (Postfix, from userid 1001) id A395633C3B; Mon, 6 Jun 2005 17:24:34 +0200 (CEST) To: Scott Long References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> <42A463EF.5060401@samsco.org> From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) Date: Mon, 06 Jun 2005 17:24:34 +0200 In-Reply-To: <42A463EF.5060401@samsco.org> (Scott Long's message of "Mon, 06 Jun 2005 08:55:43 -0600") Message-ID: <86fyvvqzil.fsf@xps.des.no> User-Agent: Gnus/5.110002 (No Gnus v0.2) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on tim.des.no X-Spam-Level: X-Spam-Status: No, score=-4.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 15:24:42 -0000 Scott Long writes: > Dag-Erling Sm=F8rgrav wrote: > > This is primarily an API issue, not a filesystem layout issue. We > > already have at least one filesystem with 64-bit inodes (msdosfs). > What do you mean it's not a layout issue? We can't make incompatible > layout changes whever we feel like it, or else transportability of > filesystems is completely lost and everyone who wants to boot more > than just the Last And Greatest on their system winds up with > unnessary pain. Changing the stat(2) API to support 64-bit inodes does not require us to simultaneously change the on-disk layout of every filesystem we support to use 64-bit inodes. However, if we want to fully support filesystems with 64-bit inodes (such as FAT32, which currently uses a convoluted hack to map the 64-bit offset of a directory entry into a 32-bit inode), we need to change the API. The ironic thing is that we already have a 64-bit stat(2)... for Linux ABI compatibility. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 15:51:05 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E8F0316A41C; Mon, 6 Jun 2005 15:51:05 +0000 (GMT) (envelope-from scottl@pooker.samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3529F43D1F; Mon, 6 Jun 2005 15:51:04 +0000 (GMT) (envelope-from scottl@pooker.samsco.org) Received: from pooker.samsco.org (localhost [127.0.0.1]) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j56Fv5Hu053010; Mon, 6 Jun 2005 09:57:05 -0600 (MDT) (envelope-from scottl@pooker.samsco.org) Received: from localhost (scottl@localhost) by pooker.samsco.org (8.13.3/8.13.1/Submit) with ESMTP id j56Fv4Ws053007; Mon, 6 Jun 2005 09:57:04 -0600 (MDT) (envelope-from scottl@pooker.samsco.org) Date: Mon, 6 Jun 2005 09:57:04 -0600 (MDT) From: Scott Long To: =?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?= In-Reply-To: <86fyvvqzil.fsf@xps.des.no> Message-ID: <20050606095117.Q52957@pooker.samsco.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> <42A463EF.5060401@samsco.org> <86fyvvqzil.fsf@xps.des.no> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-123071739-1118073379=:52957" X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 15:51:06 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-123071739-1118073379=:52957 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Mon, 6 Jun 2005, [iso-8859-1] Dag-Erling Sm=F8rgrav wrote: > Scott Long writes: >> Dag-Erling Sm=F8rgrav wrote: >>> This is primarily an API issue, not a filesystem layout issue. We >>> already have at least one filesystem with 64-bit inodes (msdosfs). >> What do you mean it's not a layout issue? We can't make incompatible >> layout changes whever we feel like it, or else transportability of >> filesystems is completely lost and everyone who wants to boot more >> than just the Last And Greatest on their system winds up with >> unnessary pain. > > Changing the stat(2) API to support 64-bit inodes does not require us > to simultaneously change the on-disk layout of every filesystem we > support to use 64-bit inodes. However, if we want to fully support > filesystems with 64-bit inodes (such as FAT32, which currently uses a > convoluted hack to map the 64-bit offset of a directory entry into a > 32-bit inode), we need to change the API. > > The ironic thing is that we already have a 64-bit stat(2)... for > Linux ABI compatibility. > Ah, I see your point. Well, it's not too late to address this for 6.0, and it might be a really good idea to think about it now. Is there=20 anything else that should be bumped along with it? Scott --0-123071739-1118073379=:52957-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 16:04:08 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6173216A41C; Mon, 6 Jun 2005 16:04:08 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA7C043D1F; Mon, 6 Jun 2005 16:04:07 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost.des.no [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id B021B60F0; Mon, 6 Jun 2005 18:03:59 +0200 (CEST) Received: from xps.des.no (des.no [80.203.228.37]) by tim.des.no (Postfix) with ESMTP id 975A660EF; Mon, 6 Jun 2005 18:03:59 +0200 (CEST) Received: by xps.des.no (Postfix, from userid 1001) id 84D4433C3B; Mon, 6 Jun 2005 18:03:59 +0200 (CEST) To: Scott Long References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> <42A463EF.5060401@samsco.org> <86fyvvqzil.fsf@xps.des.no> <20050606095117.Q52957@pooker.samsco.org> From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) Date: Mon, 06 Jun 2005 18:03:59 +0200 In-Reply-To: <20050606095117.Q52957@pooker.samsco.org> (Scott Long's message of "Mon, 6 Jun 2005 09:57:04 -0600 (MDT)") Message-ID: <863brvqxow.fsf@xps.des.no> User-Agent: Gnus/5.110002 (No Gnus v0.2) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on tim.des.no X-Spam-Level: X-Spam-Status: No, score=-4.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 16:04:08 -0000 Scott Long writes: > On Mon, 6 Jun 2005, [iso-8859-1] Dag-Erling Sm=F8rgrav wrote: > > Changing the stat(2) API to support 64-bit inodes does not require us > > to simultaneously change the on-disk layout of every filesystem we > > support to use 64-bit inodes. However, if we want to fully support > > filesystems with 64-bit inodes (such as FAT32, which currently uses a > > convoluted hack to map the 64-bit offset of a directory entry into a > > 32-bit inode), we need to change the API. > Ah, I see your point. Well, it's not too late to address this for 6.0, > and it might be a really good idea to think about it now. Is there > anything else that should be bumped along with it? Not that I know of. I believe the best way to do this is the way Linux did it: introduce new *stat64() syscalls and keep the old ones around. #define magic in will take care of making *stat64() look like *stat(). DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 16:12:30 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2A6CE16A41C; Mon, 6 Jun 2005 16:12:30 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9907543D48; Mon, 6 Jun 2005 16:12:27 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [10.4.249.251] ([206.13.39.129]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j56GIEKV053152; Mon, 6 Jun 2005 10:18:25 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A475D6.4020906@samsco.org> Date: Mon, 06 Jun 2005 10:12:06 -0600 From: Scott Long User-Agent: Mozilla Thunderbird 1.0.2 (Macintosh/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> <42A463EF.5060401@samsco.org> <86fyvvqzil.fsf@xps.des.no> <20050606095117.Q52957@pooker.samsco.org> <863brvqxow.fsf@xps.des.no> In-Reply-To: <863brvqxow.fsf@xps.des.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.0 required=3.8 tests=none autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org, Scott Long Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 16:12:30 -0000 Dag-Erling Smørgrav wrote: > Scott Long writes: > >>On Mon, 6 Jun 2005, [iso-8859-1] Dag-Erling Smørgrav wrote: >> >>>Changing the stat(2) API to support 64-bit inodes does not require us >>>to simultaneously change the on-disk layout of every filesystem we >>>support to use 64-bit inodes. However, if we want to fully support >>>filesystems with 64-bit inodes (such as FAT32, which currently uses a >>>convoluted hack to map the 64-bit offset of a directory entry into a >>>32-bit inode), we need to change the API. >> >>Ah, I see your point. Well, it's not too late to address this for 6.0, >>and it might be a really good idea to think about it now. Is there >>anything else that should be bumped along with it? > > > Not that I know of. > > I believe the best way to do this is the way Linux did it: introduce > new *stat64() syscalls and keep the old ones around. #define magic in > will take care of making *stat64() look like *stat(). > > DES So one of the advantages that we have with the 5.x -> 6.0 migration right now is that it's still possible to run a 5.x userland with a 6.x kernel without much problem. Changing fundamental syscalls and structures would defeat this and make life much harder for people that want to sell 6.0 as a painless migration. On the surface I like your idea of stat64 (regardless of politics of having 64-bit specific in the API names), but I'd like to think on it a bit. In the mean time I'm off to listen to Steve profess his love to Intel ;-) Scott From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 16:40:31 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 97E8F16A41C; Mon, 6 Jun 2005 16:40:31 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4A80B43D48; Mon, 6 Jun 2005 16:40:31 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.13.0/8.13.0) with ESMTP id j56GeUQJ024411; Mon, 6 Jun 2005 09:40:30 -0700 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.13.0/8.13.0/Submit) id j56GeU5f024409; Mon, 6 Jun 2005 09:40:30 -0700 Date: Mon, 6 Jun 2005 09:40:30 -0700 From: Brooks Davis To: Suleiman Souhlal Message-ID: <20050606164030.GB22790@odin.ac.hmc.edu> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xXmbgvnjoT4axfJE" Content-Disposition: inline In-Reply-To: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new X-Spam-Status: No, hits=0.0 required=8.0 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on odin.ac.hmc.edu Cc: current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 16:40:31 -0000 --xXmbgvnjoT4axfJE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 05, 2005 at 02:03:10PM -0400, Suleiman Souhlal wrote: > Hi, >=20 > IFS is basically FFS, minus the namespace. Files are referenced by =20 > their inode number, thus making file lookup very fast. This is very =20 > useful for things that keep the file names in their own database =20 > anyway, such as web or news caches. >=20 > Files are created by opening the "newfile" file: >=20 > fd =3D open("newfile", O_CREAT|O_RDWR, 0644); > fstat(fd, &st); > printf("new file is %d\n", (int)st.st_ino); >=20 > Once a file has been created it is possible to accessing by just =20 > opening it the file named after the inode number. For example: >=20 > fd =3D open("5", O_RDWR); >=20 > This filesystem was imported back in the year 2000, by Adrian Chadd =20 > (adrian@) and removed in 2002, before the introduction of UFS2 in =20 > order not to slow down the development of UFS2. >=20 > I have fixed it so that it is usable on -CURRENT. You can find the =20 > kernel bits at http://people.freebsd.org/~ssouhlal/testing/=20 > ifs-20050605-2.diff and the userland bits at http://=20 > people.freebsd.org/~ssouhlal/testing/ifs-20050605-userland-2.diff . > (You need to copy src/sbin/fsck_ffs to src/sbin/fsck_ifs and src/sbin/=20 > mount_ufs to src/sbin/mount_ifs, before applying the userland patch). >=20 > You can use the regular newfs(8) to create such a filesystem. You =20 > might also want to specify the -n option, so that the .snap directory =20 > is not created at newfs, since directories are useless, in IFS. If =20 > you don't you'll end up with a useless inode, which is not really a =20 > big deal. > Once the filesystem is created, you can mount it with mount_ifs. >=20 > You can find additional comments in src/sys/ufs/ifs/README in the patch. >=20 > Is anyone interested in seeing this committed? I would very much like to see this committed. The vast majority of high performance computing data management systems could strongly benefit from a system like IFS. If it's in the tree and usable in a shipping release, that gives us a powerful edge in convincing people to use FreeBSD as a backed for such a system. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --xXmbgvnjoT4axfJE Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD4DBQFCpHxzXY6L6fI4GtQRAvWPAJ4/VfOFmgzhvpFLW+i4PVFka3V/OgCYt2VT ZhlnhX9bXIpCWOnUEMnHqA== =MngO -----END PGP SIGNATURE----- --xXmbgvnjoT4axfJE-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 17:51:57 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8C73216A41C; Mon, 6 Jun 2005 17:51:57 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from smtp3.server.rpi.edu (smtp3.server.rpi.edu [128.113.2.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1AAD943D60; Mon, 6 Jun 2005 17:51:56 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp3.server.rpi.edu (8.13.0/8.13.0) with ESMTP id j56HponH005719; Mon, 6 Jun 2005 13:51:54 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <42A453B5.3020006@samsco.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> Date: Mon, 6 Jun 2005 13:51:49 -0400 To: Scott Long From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-CanItPRO-Stream: default X-RPI-SA-Score: undef - spam-scanning disabled X-Scanned-By: CanIt (www . canit . ca) on 128.113.2.3 Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 17:51:57 -0000 At 7:46 AM -0600 6/6/05, Scott Long wrote: >Garance A Drosihn wrote: >>At 1:05 AM -0400 6/6/05, Suleiman Souhlal wrote: >>> >>>Talking about going to 64-bit inode numbers, how would we deal >>>with the change in stat(2)? >> >> >>By making some sort of incompatible change to stat(2). This has >>been discussed from time-to-time. It's another change that I >>would have liked to have seen (at least for the stat routines) >>in 6.0, but right now I suspect it will not happen until 7.0. > >We can't go making incremental incompatibilities to the filesystem >without a good deal of planning. This is the type of thing that >would go into a 'UFS3'. I am not talking about a new file system (not for 6.0, certainly!). I am just talking about a new layout for 'struct stat'. Yes, it is an incompatible change, but it would not be changing any filesystems. In fact, all I wanted for 6.0 was a new 'struct stat' where all the fields are the same size as they are right now, but many of them have extra space reserved so that some later commits could change from 32-bit values to 64-bit values. I realize that even what I wanted to do is too much of a jump at this point for 6.0-release. I'm simply saying that if we drop back to a year ago, at that time I was hoping a new 'struct stat' would be ready for 6.0. This was explicitly discussed at the dev-summit at the 2004 Usenix in Boston, for instance. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 19:55:13 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5121916A41C; Mon, 6 Jun 2005 19:55:13 +0000 (GMT) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B7E843D49; Mon, 6 Jun 2005 19:55:12 +0000 (GMT) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.1/8.13.1) with ESMTP id j56JsvcS019388; Mon, 6 Jun 2005 12:55:06 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <200506061955.j56JsvcS019388@gw.catspoiler.org> Date: Mon, 6 Jun 2005 12:54:57 -0700 (PDT) From: Don Lewis To: scottl@samsco.org In-Reply-To: <42A453B5.3020006@samsco.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: ssouhlal@FreeBSD.org, current@FreeBSD.org, fs@FreeBSD.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 19:55:13 -0000 On 6 Jun, Scott Long wrote: > Garance A Drosihn wrote: >> At 1:05 AM -0400 6/6/05, Suleiman Souhlal wrote: >> >>> >>> On Jun 6, 2005, at 12:53 AM, Scott Long wrote: >>> >>>> It's a huge win for CPU overhead in the filesystem, especially >>>> when we start talking about increasing the size of m_links >>>> field and possibly going 64-bit inode numbers. >>> >>> >>> Talking about going to 64-bit inode numbers, how would we deal >>> with the change in stat(2)? >> >> >> By making some sort of incompatible change to stat(2). This has >> been discussed from time-to-time. It's another change that I >> would have liked to have seen (at least for the stat routines) >> in 6.0, but right now I suspect it will not happen until 7.0. >> > > We can't go making incremental incompatibilities to the filesystem > without a good deal of planning. This is the type of thing that > would go into a 'UFS3'. I have some long-term plans here, but I > need to get the initial proof-of-concept journalling working before > I start to seriously consider what else would be in UFS3. cough ... larger cylinder groups ... cough From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 20:23:31 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D1EB816A41C; Mon, 6 Jun 2005 20:23:31 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from smtp4.server.rpi.edu (smtp4.server.rpi.edu [128.113.2.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 760CD43D4C; Mon, 6 Jun 2005 20:23:31 +0000 (GMT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp4.server.rpi.edu (8.13.0/8.13.0) with ESMTP id j56KNQ3p019855; Mon, 6 Jun 2005 16:23:27 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <20050606095117.Q52957@pooker.samsco.org> References: <82ACAD58-B179-44E2-852F-60F25C0BBBC1@FreeBSD.org> <20050606033145.GA80739@www.portaone.com> <42A3D6CF.2000504@samsco.org> <0A6C1F19-A734-4EC8-BE97-2D000D189968@FreeBSD.org> <42A453B5.3020006@samsco.org> <86oeaj1r2x.fsf@xps.des.no> <42A463EF.5060401@samsco.org> <86fyvvqzil.fsf@xps.des.no> <20050606095117.Q52957@pooker.samsco.org> Date: Mon, 6 Jun 2005 16:23:25 -0400 To: Scott Long , Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= From: Garance A Drosihn Content-Type: text/plain; charset="iso-8859-1" ; format="flowed" Content-Transfer-Encoding: quoted-printable X-CanItPRO-Stream: default X-RPI-SA-Score: undef - spam-scanning disabled X-Scanned-By: CanIt (www . canit . ca) on 128.113.2.4 Cc: Suleiman Souhlal , current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 20:23:32 -0000 At 9:57 AM -0600 6/6/05, Scott Long wrote: >On Mon, 6 Jun 2005, [iso-8859-1] Dag-Erling Sm=F8rgrav wrote: >> >>Changing the stat(2) API to support 64-bit inodes does not require us >>to simultaneously change the on-disk layout of every filesystem we >>support to use 64-bit inodes. However, if we want to fully support >>filesystems with 64-bit inodes (such as FAT32, which currently uses a >>convoluted hack to map the 64-bit offset of a directory entry into a >>32-bit inode), we need to change the API. >> >>The ironic thing is that we already have a 64-bit stat(2)... for >>Linux ABI compatibility. > >Ah, I see your point. Well, it's not too late to address this for >6.0, and it might be a really good idea to think about it now. Is >there anything else that should be bumped along with it? Ooo. Well, if we *could* lay the groundwork for a newer stat() routine in time for 6.0, then I'd really like to see that. The first time I read this message I thought you were saying "it IS too late...", which would have been understandable. There are a number of other fields in 'struct stat' that people have wanted to see expanded. But I'm not sure if that's what you meant by your question of "anything else to get bumped?" -- Garance Alistair Drosehn =3D gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 23:01:37 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3EC5E16A41C; Mon, 6 Jun 2005 23:01:37 +0000 (GMT) (envelope-from julian@elischer.org) Received: from bigwoop.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 105ED43D49; Mon, 6 Jun 2005 23:01:37 +0000 (GMT) (envelope-from julian@elischer.org) Received: from [208.206.78.97] (julian.vicor-nb.com [208.206.78.97]) by bigwoop.vicor-nb.com (Postfix) with ESMTP id 6C1187A403; Mon, 6 Jun 2005 16:01:36 -0700 (PDT) Message-ID: <42A4D5D0.9040500@elischer.org> Date: Mon, 06 Jun 2005 16:01:36 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.7) Gecko/20050423 X-Accept-Language: en, hu MIME-Version: 1.0 To: Randy Bush References: <17059.7150.269428.448187@roam.psg.com> In-Reply-To: <17059.7150.269428.448187@roam.psg.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 23:01:37 -0000 what. all in one directory? I've only had up to 500,000 files in one directory on FreeBSD. Randy Bush wrote: >to mirror someone's stuff, i need an fs which is happy with a jillion >small files. they're linux geeks, so suggested reiserfs. but that >appears (from -current's /sys/i386/conf/NOTES) to only have read- >only support on freebsd. is there another path? > >randy > >_______________________________________________ >freebsd-fs@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-fs >To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 23:03:30 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 11BB216A41C; Mon, 6 Jun 2005 23:03:30 +0000 (GMT) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id DEAD243D1D; Mon, 6 Jun 2005 23:03:29 +0000 (GMT) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.50 (FreeBSD)) id 1DfQcz-000FX1-Ft; Mon, 06 Jun 2005 23:03:29 +0000 Received: from [127.0.0.1] (helo=roam.psg.com.psg.com) by roam.psg.com with esmtp (Exim 4.51 (FreeBSD)) id 1DfQcs-0001Ct-7T; Mon, 06 Jun 2005 16:03:22 -0700 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17060.54841.623276.347321@roam.psg.com> Date: Mon, 6 Jun 2005 16:03:21 -0700 To: Julian Elischer References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 23:03:30 -0000 > what. all in one directory? >> to mirror someone's stuff, i need an fs which is happy with a jillion >> small files. they're linux geeks, so suggested reiserfs. but that >> appears (from -current's /sys/i386/conf/NOTES) to only have read- >> only support on freebsd. is there another path? turns out the person requesting the mirror did not get it. linuxoid. its a whole dir tree with at most 10k files per dir. it's like a news spool. end of thread. sorry to have bothered folk. randy From owner-freebsd-fs@FreeBSD.ORG Mon Jun 6 23:35:25 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D446B16A41C; Mon, 6 Jun 2005 23:35:25 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81BA643D48; Mon, 6 Jun 2005 23:35:25 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 206ED46B2D; Mon, 6 Jun 2005 19:35:25 -0400 (EDT) Date: Tue, 7 Jun 2005 00:36:12 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Jim Rees In-Reply-To: <20050606105100.AFFBA1BBF8@citi.umich.edu> Message-ID: <20050607003532.F42703@fledge.watson.org> References: <20050606105100.AFFBA1BBF8@citi.umich.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: current@freebsd.org, fs@freebsd.org Subject: Re: [PATCH] IFS: Inode FileSystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jun 2005 23:35:26 -0000 On Mon, 6 Jun 2005, Jim Rees wrote: > OpenAFS could benefit from something like IFS, although I'm not > volunteering to do the work. The client has complicated code to keep > its thousands of cache files in a hierarchy just to reduce name lookup > time, and the server originally used a special "open by inode" system > call that had been hacked in to the kernel. In Arla, they use the file handle routines, which accomplish some of the same objectives -- insert for a new cache file is still expensive, but you can lookup using the file handle and avoid walking the name space. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Tue Jun 7 12:31:10 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6842E16A43C; Tue, 7 Jun 2005 12:31:10 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 16CAC43D5D; Tue, 7 Jun 2005 12:31:09 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j57CUrfG008421; Tue, 7 Jun 2005 07:31:00 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42A59367.6060307@centtech.com> Date: Tue, 07 Jun 2005 07:30:31 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Julian Elischer References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> In-Reply-To: <42A4D5D0.9040500@elischer.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/918/Tue Jun 7 03:35:05 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2005 12:31:10 -0000 Julian Elischer wrote: > what. all in one directory? > > I've only had up to 500,000 files in one directory on FreeBSD. The only problems I've had with a directory with millions of files is things like ls -al with attempt to sort the list, but the list doesn't fit into memory. Access to the files is of course very snappy. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Tue Jun 7 16:56:09 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 90B3F16A41C; Tue, 7 Jun 2005 16:56:09 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4A7ED43D53; Tue, 7 Jun 2005 16:56:09 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 4C3AB46B2D; Tue, 7 Jun 2005 12:56:08 -0400 (EDT) Date: Tue, 7 Jun 2005 17:57:02 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Eric Anderson In-Reply-To: <42A59367.6060307@centtech.com> Message-ID: <20050607175242.D61131@fledge.watson.org> References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current , Julian Elischer Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2005 16:56:09 -0000 On Tue, 7 Jun 2005, Eric Anderson wrote: > Julian Elischer wrote: >> what. all in one directory? >> >> I've only had up to 500,000 files in one directory on FreeBSD. > > The only problems I've had with a directory with millions of files is > things like ls -al with attempt to sort the list, but the list doesn't > fit into memory. Access to the files is of course very snappy. Ditto. I regularly use directories with tens and hundreds of thousands of entries as a result of manipulating very large folders with the Cyrus server. I run into the following two classes of problems: - Some appliations behave poorly with large trees. ls(1) is the classic example -- sorting 150,000 strings is expensive, and should be avoided. It also requires holding al the strings in memory rather than continuing the iteration. fts ns bad about this, so many applications that use fts suffer from this. With the sort issue, -f makes a big difference. - Some operations become more expensive -- as directories grow, the cost of adding new entries gets more expensive. You'll notice this fairly substantailly if you untar a tar file with many entries in the same directory -- early on, cost of insert for a new item is very cheap, but it rapidly slows down from h thousands of inserts per second to hundreds or less. I notice this if I restore a large Cyrus directory from backup. - UFS_DIRHASH really helps with large directory performance by reducing the cost of lookup, but at the cost of memory. Make sure the box has lots of memory. All this said -- FreeBSD works really well for me with large file counts, I rarely hit the edge cases where there is a problem. Most problems are with applications, and when you are using more extreme file system layouts, you typically are using applications customized for that andso they do the right things. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 07:25:57 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 942AE16A41C; Wed, 8 Jun 2005 07:25:57 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F79B43D53; Wed, 8 Jun 2005 07:25:56 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id A560360F3; Wed, 8 Jun 2005 09:25:48 +0200 (CEST) Received: from xps.des.no (des.no [80.203.228.37]) by tim.des.no (Postfix) with ESMTP id 91F7B60F2; Wed, 8 Jun 2005 09:25:48 +0200 (CEST) Received: by xps.des.no (Postfix, from userid 1001) id 826EF33C3B; Wed, 8 Jun 2005 09:25:48 +0200 (CEST) To: Robert Watson References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) Date: Wed, 08 Jun 2005 09:25:48 +0200 In-Reply-To: <20050607175242.D61131@fledge.watson.org> (Robert Watson's message of "Tue, 7 Jun 2005 17:57:02 +0100 (BST)") Message-ID: <86ll5lmhs3.fsf@xps.des.no> User-Agent: Gnus/5.110002 (No Gnus v0.2) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on tim.des.no X-Spam-Level: X-Spam-Status: No, score=-5.1 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 Cc: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current , Eric Anderson , Julian Elischer Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 07:25:57 -0000 Robert Watson writes: > - Some appliations behave poorly with large trees. ls(1) is the classic > example -- sorting 150,000 strings is expensive, and should be avoided. That's because fts's sorting code is brain-dead. It starts by reading the entire directory into a linked list, then copies that list into an array which it passes to qsort(), and finally converts the array back into a linked list. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 07:46:29 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 318F616A41C; Wed, 8 Jun 2005 07:46:29 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from kane.otenet.gr (kane.otenet.gr [195.170.0.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 72E6043D1D; Wed, 8 Jun 2005 07:46:28 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from orion.daedalusnetworks.priv (aris.bedc.ondsl.gr [62.103.39.226]) by kane.otenet.gr (8.13.4/8.13.4/Debian-1) with SMTP id j587kGGO006416; Wed, 8 Jun 2005 10:46:16 +0300 Received: from orion.daedalusnetworks.priv (orion [127.0.0.1]) by orion.daedalusnetworks.priv (8.13.3/8.13.3) with ESMTP id j587kF1F001054; Wed, 8 Jun 2005 10:46:15 +0300 (EEST) (envelope-from keramida@freebsd.org) Received: (from keramida@localhost) by orion.daedalusnetworks.priv (8.13.3/8.13.3/Submit) id j587kEFC001053; Wed, 8 Jun 2005 10:46:14 +0300 (EEST) (envelope-from keramida@freebsd.org) Date: Wed, 8 Jun 2005 10:46:14 +0300 From: Giorgos Keramidas To: Dag-Erling Sm?rgrav Message-ID: <20050608074613.GA979@orion.daedalusnetworks.priv> References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86ll5lmhs3.fsf@xps.des.no> Cc: freebsd-fs@freebsd.org, Randy Bush , FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 07:46:29 -0000 On 2005-06-08 09:25, Dag-Erling Sm?rgrav wrote: >Robert Watson writes: >> - Some appliations behave poorly with large trees. ls(1) is the classic >> example -- sorting 150,000 strings is expensive, and should be avoided. > > That's because fts's sorting code is brain-dead. It starts by reading > the entire directory into a linked list, then copies that list into an > array which it passes to qsort(), and finally converts the array back > into a linked list. Is there a better way to sort a linked list (not necessarily a singly-linked list, like the one fts_link is used for). If it makes things easier on the sorting side, we could always convert fts_link to a real `LIST_ENTRY(FTSENT) fts_link', but I suspect that sorting would still involve at least some sort of array, unless we stop using qsort(). From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 07:50:50 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9196016A41F; Wed, 8 Jun 2005 07:50:50 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0800C43D5C; Wed, 8 Jun 2005 07:50:49 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id ECD0B60F3; Wed, 8 Jun 2005 09:50:44 +0200 (CEST) Received: from xps.des.no (des.no [80.203.228.37]) by tim.des.no (Postfix) with ESMTP id 5EA6560F2; Wed, 8 Jun 2005 09:50:44 +0200 (CEST) Received: by xps.des.no (Postfix, from userid 1001) id 38AA833C3B; Wed, 8 Jun 2005 09:50:44 +0200 (CEST) To: Giorgos Keramidas References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> <20050608074613.GA979@orion.daedalusnetworks.priv> From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) Date: Wed, 08 Jun 2005 09:50:44 +0200 In-Reply-To: <20050608074613.GA979@orion.daedalusnetworks.priv> (Giorgos Keramidas's message of "Wed, 8 Jun 2005 10:46:14 +0300") Message-ID: <86zmu1l223.fsf@xps.des.no> User-Agent: Gnus/5.110002 (No Gnus v0.2) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on tim.des.no X-Spam-Level: X-Spam-Status: No, score=-5.1 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 Cc: freebsd-fs@freebsd.org, Randy Bush , FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 07:50:50 -0000 Giorgos Keramidas writes: > Is there a better way to sort a linked list (not necessarily a > singly-linked list, like the one fts_link is used for). Don't build a linked list to begin with. The comparison function is known at the time the directory entries are read, so it should be a simple matter to read them into a red-black tree instead of a singly- linked list. I'm working on a patch. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 07:52:19 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C0EB316A41C; Wed, 8 Jun 2005 07:52:19 +0000 (GMT) (envelope-from cperciva@freebsd.org) Received: from pd4mo3so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7789943D48; Wed, 8 Jun 2005 07:52:19 +0000 (GMT) (envelope-from cperciva@freebsd.org) Received: from pd5mr5so.prod.shaw.ca (pd5mr5so-qfe3.prod.shaw.ca [10.0.141.181]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0IHR009CM9V6VT60@l-daemon>; Wed, 08 Jun 2005 01:52:18 -0600 (MDT) Received: from pn2ml4so.prod.shaw.ca ([10.0.121.148]) by pd5mr5so.prod.shaw.ca (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0IHR006519V6HN40@pd5mr5so.prod.shaw.ca>; Wed, 08 Jun 2005 01:52:18 -0600 (MDT) Received: from [192.168.0.60] (S0106006067227a4a.vc.shawcable.net [24.87.209.6]) by l-daemon (iPlanet Messaging Server 5.2 HotFix 1.18 (built Jul 28 2003)) with ESMTP id <0IHR001079V5MA@l-daemon>; Wed, 08 Jun 2005 01:52:18 -0600 (MDT) Date: Wed, 08 Jun 2005 00:52:17 -0700 From: Colin Percival In-reply-to: <20050608074613.GA979@orion.daedalusnetworks.priv> To: Giorgos Keramidas Message-id: <42A6A3B1.4090607@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Accept-Language: en-us, en X-Enigmail-Version: 0.91.0.0 References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> <20050608074613.GA979@orion.daedalusnetworks.priv> User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050406) Cc: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 07:52:19 -0000 Giorgos Keramidas wrote: > On 2005-06-08 09:25, Dag-Erling Sm?rgrav wrote: >>That's because fts's sorting code is brain-dead. It starts by reading >>the entire directory into a linked list, then copies that list into an >>array which it passes to qsort(), and finally converts the array back >>into a linked list. > > Is there a better way to sort a linked list How do you define "better"? You can merge-sort a singly-linked list quite easily, but converting it to an array and back would probably be faster. Colin Percival From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 08:02:46 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2A25716A41C; Wed, 8 Jun 2005 08:02:46 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from rosebud.otenet.gr (rosebud.otenet.gr [195.170.0.94]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DF2A43D53; Wed, 8 Jun 2005 08:02:44 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from orion.daedalusnetworks.priv (aris.bedc.ondsl.gr [62.103.39.226]) by rosebud.otenet.gr (8.13.4/8.13.4/Debian-1) with SMTP id j5882Z5r032677; Wed, 8 Jun 2005 11:02:36 +0300 Received: from orion.daedalusnetworks.priv (orion [127.0.0.1]) by orion.daedalusnetworks.priv (8.13.3/8.13.3) with ESMTP id j5882ZMW001281; Wed, 8 Jun 2005 11:02:35 +0300 (EEST) (envelope-from keramida@freebsd.org) Received: (from keramida@localhost) by orion.daedalusnetworks.priv (8.13.3/8.13.3/Submit) id j5882Ys7001280; Wed, 8 Jun 2005 11:02:34 +0300 (EEST) (envelope-from keramida@freebsd.org) Date: Wed, 8 Jun 2005 11:02:34 +0300 From: Giorgos Keramidas To: Colin Percival Message-ID: <20050608080234.GA1226@orion.daedalusnetworks.priv> References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> <20050608074613.GA979@orion.daedalusnetworks.priv> <42A6A3B1.4090607@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <42A6A3B1.4090607@freebsd.org> Cc: Randy Bush , freebsd-fs@freebsd.org, FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 08:02:46 -0000 On 2005-06-08 00:52, Colin Percival wrote: > Giorgos Keramidas wrote: > > On 2005-06-08 09:25, Dag-Erling Sm?rgrav wrote: > >>That's because fts's sorting code is brain-dead. It starts by reading > >>the entire directory into a linked list, then copies that list into an > >>array which it passes to qsort(), and finally converts the array back > >>into a linked list. > > > > Is there a better way to sort a linked list > > How do you define "better"? You can merge-sort a singly-linked list > quite easily, but converting it to an array and back would probably > be faster. Better, in this case, would be any of: a. faster b. faster and less demanding in memory The red-black tree des mentioned is certainly faster to traverse, but not necessarily less demanding in memory. The memory load when a red-black tree is used will be amortized to a range of "add FTSENT" operations, so it seems nice :-) From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 08:03:12 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6F9B516A41C; Wed, 8 Jun 2005 08:03:12 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from rosebud.otenet.gr (rosebud.otenet.gr [195.170.0.94]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8E1F943D53; Wed, 8 Jun 2005 08:03:11 +0000 (GMT) (envelope-from keramida@freebsd.org) Received: from orion.daedalusnetworks.priv (aris.bedc.ondsl.gr [62.103.39.226]) by rosebud.otenet.gr (8.13.4/8.13.4/Debian-1) with SMTP id j58834Dp000676; Wed, 8 Jun 2005 11:03:04 +0300 Received: from orion.daedalusnetworks.priv (orion [127.0.0.1]) by orion.daedalusnetworks.priv (8.13.3/8.13.3) with ESMTP id j58834Qj001300; Wed, 8 Jun 2005 11:03:04 +0300 (EEST) (envelope-from keramida@freebsd.org) Received: (from keramida@localhost) by orion.daedalusnetworks.priv (8.13.3/8.13.3/Submit) id j58834Zc001299; Wed, 8 Jun 2005 11:03:04 +0300 (EEST) (envelope-from keramida@freebsd.org) Date: Wed, 8 Jun 2005 11:03:04 +0300 From: Giorgos Keramidas To: Dag-Erling Sm?rgrav Message-ID: <20050608080304.GB1226@orion.daedalusnetworks.priv> References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> <20050608074613.GA979@orion.daedalusnetworks.priv> <86zmu1l223.fsf@xps.des.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86zmu1l223.fsf@xps.des.no> Cc: freebsd-fs@freebsd.org, Randy Bush , FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 08:03:12 -0000 On 2005-06-08 09:50, Dag-Erling Sm?rgrav wrote: > Giorgos Keramidas writes: > > Is there a better way to sort a linked list (not necessarily a > > singly-linked list, like the one fts_link is used for). > > Don't build a linked list to begin with. Hehe. Exactly. > The comparison function is known at the time the directory entries are > read, so it should be a simple matter to read them into a red-black > tree instead of a singly- linked list. I'm working on a patch. Thanks :) From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 08:13:28 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 587D716A421 for ; Wed, 8 Jun 2005 08:13:28 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: from web41206.mail.yahoo.com (web41206.mail.yahoo.com [66.218.93.39]) by mx1.FreeBSD.org (Postfix) with SMTP id 6471043D55 for ; Wed, 8 Jun 2005 08:13:27 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 3457 invoked by uid 60001); 8 Jun 2005 08:13:26 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=xkc42WyJ/hlPWd6MJYgpLoq6C06d7o9SI5Xx8rH0wSdvasjaQ+vQxyWPQH41KRUuW3ZVFkEY5++u3glEYqiwl7w1xCgElsuqfQmSwVygoBUzqgSGNTrZb9uaiPalmEmvw/O50zncRHVfj7NFh7g96a3nI+zf0Fv/iJPrBex52i4= ; Message-ID: <20050608081326.3455.qmail@web41206.mail.yahoo.com> Received: from [213.54.157.31] by web41206.mail.yahoo.com via HTTP; Wed, 08 Jun 2005 01:13:26 PDT Date: Wed, 8 Jun 2005 01:13:26 -0700 (PDT) From: Arne "Wörner" To: Giorgos Keramidas , Colin Percival In-Reply-To: <20050608080234.GA1226@orion.daedalusnetworks.priv> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, Randy Bush , FreeBSD Current , Robert Watson , Julian Elischer , Eric Anderson Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 08:13:28 -0000 --- Giorgos Keramidas wrote: > Better, in this case, would be any of: > > a. faster > b. faster and less demanding in memory > > The red-black tree des mentioned is certainly faster to > traverse, but not necessarily less demanding in memory. > The memory load when a red-black tree is used will be > amortized to a range of "add FTSENT" operations, so > it seems nice :-) > I just remembered or found this algorithm and data structure: 1. a dynamic array of char* L (resized by realloc or so) 2. a dynamic array of char D (resized by realloc or so) 3. the number of char* entries in L C 4. the first unused entry of D L's entries would point into the data in D D contains the data that should be sorted If ls(1) wants to insert a new directory entry, it adds the data to the tail of D, and it adds the pointer to the former tail of D at the right position in L. But I do not know if the shift operations in L are not so good... But the memory usage looks quite good to me... I say, the sort of the directory entries should have a time complexity of O(N*log(N))? Do you think, I should try to implement it? -Arne __________________________________ Discover Yahoo! Get on-the-go sports scores, stock quotes, news and more. Check it out! http://discover.yahoo.com/mobile.html From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 09:29:06 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2325616A41F for ; Wed, 8 Jun 2005 09:29:06 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: from web41214.mail.yahoo.com (web41214.mail.yahoo.com [66.218.93.47]) by mx1.FreeBSD.org (Postfix) with SMTP id D32F043D5C for ; Wed, 8 Jun 2005 09:29:05 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 20124 invoked by uid 60001); 8 Jun 2005 09:29:05 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=aIXiTKNAjhOQsxSXrYLUo3JL8Jst3EqWL7UuWV8JnkKOxPimZ64NTyJKX0IkeN99BQ9jhfBJ78hhkA/vpWccVNj739Pe24XOauT6XpKP+Nf12ior6F7axgEKqZrtBwOqyC0FnKVonxogYlC86+oZ6Ho3l9Yk1rr3zHM1Zk2iodo= ; Message-ID: <20050608092905.20122.qmail@web41214.mail.yahoo.com> Received: from [213.54.157.31] by web41214.mail.yahoo.com via HTTP; Wed, 08 Jun 2005 02:29:05 PDT Date: Wed, 8 Jun 2005 02:29:05 -0700 (PDT) From: Arne "Wörner" To: freebsd-fs@freebsd.org In-Reply-To: <20050608081326.3455.qmail@web41206.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 09:29:06 -0000 --- Arne Wörner wrote: > --- Giorgos Keramidas wrote: > But I do not know if the shift operations in L are not so > good... > > But the memory usage looks quite good to me... > > I say, the sort of the directory entries should have a time > complexity of O(N*log(N))? > > Do you think, I should try to implement it? > Oooops... I just compared my algorithm to ls(1). And ls(1) is about 12 times faster than my algorithm with a 120000 files directory. First putting the entries into my data structure (but without sorting) and then using qsort is a little bit faster than ls(1). -Arne :-) __________________________________ Discover Yahoo! Find restaurants, movies, travel and more fun for the weekend. Check it out! http://discover.yahoo.com/weekend.html From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 14:11:34 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E894116A41C for ; Wed, 8 Jun 2005 14:11:33 +0000 (GMT) (envelope-from ut@bhi-hamburg.de) Received: from mail.terralink.de (mail.tlink.de [217.9.16.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CD9543D1D for ; Wed, 8 Jun 2005 14:11:32 +0000 (GMT) (envelope-from ut@bhi-hamburg.de) Received: from schweinkram.pleach-hamburg.de (p548EB77D.dip.t-dialin.net [84.142.183.125]) by mail.terralink.de (Postfix) with ESMTP id 4BAC8B8A1 for ; Wed, 8 Jun 2005 16:11:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by schweinkram.pleach-hamburg.de (Postfix) with ESMTP id AF77F6A89EB; Wed, 8 Jun 2005 16:03:54 +0200 (CEST) Received: from by localhost (amavisd-new, port ) id nED73pk2; Wed, 8 Jun 2005 16:03:53 +0200 (CEST) Received: by schweinkram.pleach-hamburg.de (Postfix, from userid 65534) id 1D0446A89C5; Wed, 8 Jun 2005 16:03:53 +0200 (CEST) Received: from titusn (titusn.pleach-hamburg.de [192.168.1.19]) by schweinkram.pleach-hamburg.de (Postfix) with SMTP id AB4656A5B58; Wed, 8 Jun 2005 16:03:49 +0200 (CEST) From: "Titus von Boxberg" To: Date: Wed, 8 Jun 2005 16:06:52 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 Importance: Normal X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on schweinkram.pleach-hamburg.de X-Spam-Status: No, hits=1.6 required=5.0 tests=BAYES_60 autolearn=no version=2.60 X-Spam-Level: * X-Virus-Scanned: by amavisd-new at pleach-hamburg.de Subject: Again: Inconsistent snapshot contents X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 14:11:34 -0000 Hi, I sent a mail about inconsistent snapshot contents a week ago but did not get a response yet. Please notify me at least if this is the wrong mailing list for these questions. This is the abstract: On two servers (5.2.1-RELEASE and 5.3-RELEASE) we use the following backup script: #!/bin sh # .... mksnap_ffs / /.snap/backupsnap mdconfig -a -t vnode -f /.snap/backupsnap -u 5 mount -o ro /dev/md5 /mnt cd /mnt cp fileinquestion /tmp/x1 # see below tar cf /dev/sa0 .... cp fileinquestion /tmp/x2 # see below tar df /dev/sa0 if [ $? -eq 1 ] ; then # see below tar -C /tmp -xf /dev/sa0 fileinquestion mv /tmp/fileinquestion /tmp/x3 fi # .... ASSUMPTIONS: I cannot see anything wrong with the script or the underlying assumptions (snapshots are COMPLETE consistent views of the state of the file system at a certain instant). I think it's highly unlikely that the tar command(s) are able to alter the read only mounted snapshot. STATISTICS: The verify tar almost always complains that at least one (and always the same) file(s) is/are differing. I tracked the file contents of one file that the verify tar usually complains about and could see that its contents ON THE SNAPSHOT (but NOT its size) frequently differ over time. The contents at least of this file in question are modified every minute or so OUTSIDE (of course) the snapshot. It does not have anything to do with the tape. The problem also occurs when tarring onto or from a disk file on a holding disk. I'm also quite sure that this problem is NOT reproducible if the backup process (tar) is not run. It is not sufficient to create the snapshot and compare over time ***without also starting the backup command***. The hardware configs of both servers are quite different (AthlonXP2800/512/ATA60GB vs. Celeron600/256/ATARAID 80GB). CONCLUSION 1 --> I'm quite sure now that the contents of the snapshot may be inconsistent over time. CONCLUSION 2 --> There is some bug in the kernel maybe related to a side effect that occurs somehow when the tar commands are run AND snapshotted files change (outside the snapshot) in a certain manner (by content but apparently NOT in size). QUESTIONS: Is something wrong with the ASSUMPTIONS above? Have I overlooked some bugfix between 5.3-RELEASE and 5-STABLE? Is this the wrong mailing list? Should I post this question / problem to some other list? Which list? My FreeBSD kernel specific skills are currently restricted to writing device drivers, but I'm willing and able to help by coding and installing or using any hook or debugging routine into the ufs kernel code if someone could point me to the right place. This would be the optimum answer on this mail. Thanks in advance. Titus From owner-freebsd-fs@FreeBSD.ORG Wed Jun 8 16:06:45 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D32216A41C; Wed, 8 Jun 2005 16:06:45 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.70]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3CBC943D48; Wed, 8 Jun 2005 16:06:44 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from mac.com (smtpin07-en2 [10.13.10.152]) by smtpout.mac.com (Xserve/8.12.11/smtpout13/MantshX 4.0) with ESMTP id j58G6bIT014365; Wed, 8 Jun 2005 09:06:38 -0700 (PDT) Received: from [192.168.1.6] (pool-68-161-69-6.ny325.east.verizon.net [68.161.69.6]) (authenticated bits=0) by mac.com (Xserve/smtpin07/MantshX 4.0) with ESMTP id j58G6YKu028817; Wed, 8 Jun 2005 09:06:35 -0700 (PDT) In-Reply-To: <42A6A3B1.4090607@freebsd.org> References: <17059.7150.269428.448187@roam.psg.com> <42A4D5D0.9040500@elischer.org> <42A59367.6060307@centtech.com> <20050607175242.D61131@fledge.watson.org> <86ll5lmhs3.fsf@xps.des.no> <20050608074613.GA979@orion.daedalusnetworks.priv> <42A6A3B1.4090607@freebsd.org> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Charles Swiger Date: Wed, 8 Jun 2005 12:06:32 -0400 To: Colin Percival X-Mailer: Apple Mail (2.730) Cc: freebsd-fs@freebsd.org, Randy Bush , FreeBSD Current , Robert Watson , Giorgos Keramidas , Eric Anderson , Julian Elischer Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 16:06:45 -0000 On Jun 8, 2005, at 3:52 AM, Colin Percival wrote: > Giorgos Keramidas wrote: >> On 2005-06-08 09:25, Dag-Erling Sm?rgrav wrote: >>> That's because fts's sorting code is brain-dead. It starts by >>> reading >>> the entire directory into a linked list, then copies that list >>> into an >>> array which it passes to qsort(), and finally converts the array >>> back >>> into a linked list. >> >> Is there a better way to sort a linked list > > How do you define "better"? Most people think "better" means "it runs faster". :-) > You can merge-sort a singly-linked list quite easily, but converting > it to an array and back would probably be faster. Ugh. Converting a list to an array and back simply to sort requires roughly twice the working memory, and the work you're doing to accomplish this conversion is time spent not actually solving anything useful. It's best to pick one data structure based on what the task requires and what the access patterns are. If you know that the number of elements is fixed and will never change, arrays are fine, but if the # of elements changes dynamicly, something like a list or red-black tree is a better bet. A list is more suitable for linear traversal, whereas a RB-tree handles random access better. qsort() is wonderful, but as the size of individual data elements grows, the overhead of copying them around in the array becomes large enough that you are better off rearranging list pointers rather than using an array representation. A long time ago, the break-even point for using lists rather than arrays was somewhere around sizeof(el) > 64, but it's been a while.... -- -Chuck From owner-freebsd-fs@FreeBSD.ORG Thu Jun 9 19:11:19 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8879B16A41C; Thu, 9 Jun 2005 19:11:19 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from gate.bitblocks.com (bitblocks.com [209.204.185.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33D6843D49; Thu, 9 Jun 2005 19:11:18 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost [127.0.0.1]) by gate.bitblocks.com (8.13.3/8.13.1) with ESMTP id j59JBGQn096385; Thu, 9 Jun 2005 12:11:16 -0700 (PDT) (envelope-from bakul@bitblocks.com) Message-Id: <200506091911.j59JBGQn096385@gate.bitblocks.com> To: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=) In-reply-to: Your message of "Wed, 08 Jun 2005 09:50:44 +0200." <86zmu1l223.fsf@xps.des.no> Date: Thu, 09 Jun 2005 12:11:15 -0700 From: Bakul Shah Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jun 2005 19:11:19 -0000 > Giorgos Keramidas writes: > > Is there a better way to sort a linked list (not necessarily a > > singly-linked list, like the one fts_link is used for). > > Don't build a linked list to begin with. The comparison function is > known at the time the directory entries are read, so it should be a > simple matter to read them into a red-black tree instead of a singly- > linked list. I'm working on a patch. Unless the tree needs to grow dynamically, a red-black tree won't buy you anything. It is faster to use mergesort on an array than a red black tree implementation. They have the same O n log n complexity but per item overhead is higher in r-b perhaps due to more mallocs. Even a skewheap will be faster than r-b since you don't need to search, but just return items in sort order. Start with a small array. When it gets full double it (or grow by half if you want to waste less memory). In my tests, with input file constructed from multiple copies of /usr/share/dict/words rather than a directory, the mergesort+array was 3 times faster than red-black for upto at least 250K entries. At about 10M entries and 100MB of data, mergesort+array was still about twice as fast and used less memory. Using mmap the performance doubled and memory use went down! Benchmark program (but not with the mmap code) attached:-) % wc z 10000000 10000000 105733020 z % time a.out redblack < z 15.445u 1.314s 0:16.76 99.9% 10+320297k 0+0io 0pf+0w % time a.out array < z 9.524u 0.691s 0:10.21 100.0% 10+240289k 0+0io 0pf+0w % time a.out -m array < z 4.632u 0.653s 0:05.28 100.0% 10+151192k 0+0io 0pf+0w -- bakul [source compacted to fit on one page] #include #include #include int debug; #define strndup(s,n) strncpy(malloc(n+1), s, n) int compare(const void*a, const void* b) { return strcmp(*(char**)a, *(char**)b); } do_ar() { char* line; size_t len; int c = 0, i; int s = 16; char** a = (char**)malloc(s*sizeof(char*)); if (debug) fprintf(stderr, "Using array+mergesort\n"); while (line = fgetln(stdin, &len)) { if (c >= s) a = (char**)realloc(a, (s *= 2)*sizeof(char*)); a[c++] = strndup(line, len-1); } mergesort(a, c, sizeof(char*), compare); if (debug) for (i = 0; i < c; i++) printf("%s\n", a[i]); } typedef struct node { char *name; RB_ENTRY(node) links; } Node; int cmp(Node* a, Node* b) { return strcmp(a->name, b->name); } RB_HEAD(rb_tree, node); RB_GENERATE(rb_tree, node, links, cmp); do_rb() { char* line; size_t len; struct rb_tree rb; Node* n; if (debug) fprintf(stderr, "Using red-back tree\n"); RB_INIT(&rb); while (line = fgetln(stdin, &len)) { struct node* node = malloc(sizeof *node); node->name = strndup(line, len-1); RB_INSERT(rb_tree, &rb, node); } if (debug) RB_FOREACH(n, rb_tree, &rb) printf("%s\n", n->name); } int main(int c, char** v) { if (c > 1 && strcmp(v[1], "-d") == 0) { debug = 1; c--; v++; } if (c > 1 && v[1][0] == 'r') do_rb(); else do_ar(); } From owner-freebsd-fs@FreeBSD.ORG Thu Jun 9 19:15:57 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7397B16A41C; Thu, 9 Jun 2005 19:15:57 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20E4E43D1F; Thu, 9 Jun 2005 19:15:56 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 733C11EC338; Thu, 9 Jun 2005 21:15:55 +0200 (CEST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.3) with ESMTP id j59JFap8009020; Thu, 9 Jun 2005 21:15:37 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Bakul Shah From: "Poul-Henning Kamp" In-Reply-To: Your message of "Thu, 09 Jun 2005 12:11:15 PDT." <200506091911.j59JBGQn096385@gate.bitblocks.com> Date: Thu, 09 Jun 2005 21:15:36 +0200 Message-ID: <9019.1118344536@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: freebsd-fs@freebsd.org, FreeBSD Current Subject: Re: you are in an fs with millions of small files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jun 2005 19:15:57 -0000 In message <200506091911.j59JBGQn096385@gate.bitblocks.com>, Bakul Shah writes: >Start with a small array. When it gets full double it (or >grow by half if you want to waste less memory). If I might make an observation... You do not waste malloc'ed memory until you access it. You waste some page table entries etc in the kernel, but the actual pages do not become part of the memory-pressure mob-rule until you touch it the first time. This is a very important point which people still (after 20 years of virtual memory systems and 10 years of phkmalloc) still do not seem to have fully understood. I would advice a simple doubling and a terminal realloc to cut down to actual size. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-fs@FreeBSD.ORG Fri Jun 10 02:14:54 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0121216A41C for ; Fri, 10 Jun 2005 02:14:54 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 91E4A43D1D for ; Fri, 10 Jun 2005 02:14:50 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.42.23] (andersonbox3.centtech.com [192.168.42.23]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j5A2Enag022105 for ; Thu, 9 Jun 2005 21:14:50 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42A8F780.6000408@centtech.com> Date: Thu, 09 Jun 2005 21:14:24 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Interesting book on filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2005 02:14:54 -0000 I'm sure it's been mentioned before, but just in case it hasn't: http://www.nobius.org/~dbg/ Has a pdf of a "Practical File System Design..." book and a "File System Construction Kit". I'm going to download and play with the kit - not sure if will work without too much trouble on FreeBSD. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Jun 10 02:59:52 2005 Return-Path: X-Original-To: fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C783216A41C; Fri, 10 Jun 2005 02:59:52 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.232]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8334A43D49; Fri, 10 Jun 2005 02:59:52 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id B109E51262; Thu, 9 Jun 2005 22:59:51 -0400 (EDT) Date: Thu, 9 Jun 2005 22:59:51 -0400 From: Kris Kennaway To: David Adam Message-ID: <20050610025951.GA42327@xor.obsecurity.org> References: <20050610042211.5d214150.jylefort@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2fHTh5uZTiUOsy+g" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: Jean-Yves Lefort , freebsd-stable@FreeBSD.org, fs@FreeBSD.org Subject: Re: UFS2 partition with negative used space X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2005 02:59:52 -0000 --2fHTh5uZTiUOsy+g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 10, 2005 at 10:32:04AM +0800, David Adam wrote: > (The mail to this node is rather slow, so I'm sure someone else will have > replied by now.) >=20 > On Fri, 10 Jun 2005, Jean-Yves Lefort wrote: > > Filesystem Size Used Avail Capacity M= ounted on > > /dev/ad0s1e 989M -46M 956M -5% /= var/tmp > > > > Any hints? >=20 > Yep: delete some files on /var/tmp. :-) >=20 > If you're asking 'how can I have negative disk space?', you might want to > read newfs(8) and tunefs(8), particularly the sections dealing with the > -m flag. Look closer :-) In comment to the original question: when unmounting filesystems on 5.x and 6.x which have had a lot of activity I commonly see status messages about negative number of files/blocks being used: ffs_vfsops.c: printf("%s: unmount pending error: blocks %jd files= %d\n", Kris --2fHTh5uZTiUOsy+g Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCqQInWry0BWjoQKURAkBHAJwOOtXzvAgfYKK3t5VrE3re7eey+gCfVUq/ 6btxRfmEI09gA5X2SrYZ6UI= =gvt/ -----END PGP SIGNATURE----- --2fHTh5uZTiUOsy+g-- From owner-freebsd-fs@FreeBSD.ORG Fri Jun 10 16:07:56 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6EF7616A41F for ; Fri, 10 Jun 2005 16:07:56 +0000 (GMT) (envelope-from afields@afields.ca) Received: from afields.ca (afields.ca [216.194.67.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1756C43D49 for ; Fri, 10 Jun 2005 16:07:56 +0000 (GMT) (envelope-from afields@afields.ca) Received: from afields.ca (localhost.afields.ca [127.0.0.1]) by afields.ca (8.12.11/8.12.11) with ESMTP id j5AG7rAs076357; Fri, 10 Jun 2005 12:07:53 -0400 (EDT) (envelope-from afields@afields.ca) Received: (from afields@localhost) by afields.ca (8.12.11/8.12.11/Submit) id j5AG7rdj076356; Fri, 10 Jun 2005 12:07:53 -0400 (EDT) (envelope-from afields) Date: Fri, 10 Jun 2005 12:07:52 -0400 From: Allan Fields To: Eric Anderson Message-ID: <20050610160752.GA56346@afields.ca> References: <42A8F780.6000408@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <42A8F780.6000408@centtech.com> User-Agent: Mutt/1.4i Cc: freebsd-fs@freebsd.org Subject: Re: Interesting book on filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2005 16:07:56 -0000 On Thu, Jun 09, 2005 at 09:14:24PM -0500, Eric Anderson wrote: > I'm sure it's been mentioned before, but just in case it hasn't: > > http://www.nobius.org/~dbg/ > > Has a pdf of a "Practical File System Design..." book and a "File System > Construction Kit". I'm going to download and play with the kit - not > sure if will work without too much trouble on FreeBSD. > > Eric I downloaded and read this one as well. For a while I was looking in the stores, but it went out of fashion. Thanks to the author for putting it online. The idea behind BFS's extended attributes in the inode is different than the UFS 2 implementation, are you interested in seeing any further enhancements to file systems under BSD?. One idea floated was adding journalling and/or transactions to UFS. Then there is also the issue of cache coherency. Another, is to start adding semantic enhancements to the file system. In talking with Marshall Kirk McKusick, the prefered approach from a BSD perspective might be incremental addition of features in a way that will render UFS workable and w/o the likelihood of data loss/corruption much like the dirhash implementation. -- Allan Fields From owner-freebsd-fs@FreeBSD.ORG Fri Jun 10 16:21:00 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F155816A41C for ; Fri, 10 Jun 2005 16:21:00 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id A0F1F43D1F for ; Fri, 10 Jun 2005 16:21:00 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j5AGKw9K028954; Fri, 10 Jun 2005 11:20:58 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42A9BDD0.7010702@centtech.com> Date: Fri, 10 Jun 2005 11:20:32 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Allan Fields References: <42A8F780.6000408@centtech.com> <20050610160752.GA56346@afields.ca> In-Reply-To: <20050610160752.GA56346@afields.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Interesting book on filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2005 16:21:01 -0000 Allan Fields wrote: > On Thu, Jun 09, 2005 at 09:14:24PM -0500, Eric Anderson wrote: > >>I'm sure it's been mentioned before, but just in case it hasn't: >> >>http://www.nobius.org/~dbg/ >> >>Has a pdf of a "Practical File System Design..." book and a "File System >> Construction Kit". I'm going to download and play with the kit - not >>sure if will work without too much trouble on FreeBSD. >> >>Eric > > > I downloaded and read this one as well. For a while I was looking > in the stores, but it went out of fashion. Thanks to the author > for putting it online. > > The idea behind BFS's extended attributes in the inode is different > than the UFS 2 implementation, are you interested in seeing any > further enhancements to file systems under BSD?. I mentioned on the -current list that I'd *love* to have a clustered filesystem under FreeBSD. I think it's the future of filesystems, much as journaling is becoming prevalent today. > One idea floated was adding journalling and/or transactions to UFS. > Then there is also the issue of cache coherency. Scott Long is currently working on adding journaling to UFS. I think he's making good progress. > Another, is to start adding semantic enhancements to the file > system. In talking with Marshall Kirk McKusick, the prefered > approach from a BSD perspective might be incremental addition > of features in a way that will render UFS workable and w/o > the likelihood of data loss/corruption much like the dirhash > implementation. Seems like filesystem innovation is not so popular these days, but I think with a little brainstorming some creative individuals could come up with some very powerful and unique solutions to problems we are just used to dealing with.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------