From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 05:15:40 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7111216A417 for ; Mon, 15 Oct 2007 05:15:40 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n4.bullet.ukl.yahoo.com (n4.bullet.ukl.yahoo.com [217.146.182.181]) by mx1.freebsd.org (Postfix) with SMTP id DA68F13C45A for ; Mon, 15 Oct 2007 05:15:39 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [217.12.4.215] by n4.bullet.ukl.yahoo.com with NNFMP; 15 Oct 2007 05:01:52 -0000 Received: from [216.252.122.216] by t2.bullet.ukl.yahoo.com with NNFMP; 15 Oct 2007 05:01:52 -0000 Received: from [69.147.84.116] by t1.bullet.sp1.yahoo.com with NNFMP; 15 Oct 2007 05:01:51 -0000 Received: from [127.0.0.1] by omp208.mail.sp1.yahoo.com with NNFMP; 15 Oct 2007 05:01:51 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 855819.57214.bm@omp208.mail.sp1.yahoo.com Received: (qmail 64391 invoked by uid 60001); 15 Oct 2007 05:01:51 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=JczDcgOcH19NxY1shzp9ziDlAtzEBpecjiTG6Pg9OHRqcLO+QuMEV6pSnoVKctSi9zXFVtxgclxRWmY0+7Iyjj4sSbWfCr0lvLGzLbADpn4RmW/kjJ9AH6gNJT+nxu2hew+W1TB+gmo4dzzhTVmK6EnHp1uOJA/+mZmGUHpBq04=; X-YMail-OSG: CCs.x.MVM1lPNS7.d6XtaAcFgIld958Lr1ejkSj0PPmMnty5JJB4pnrrcPoc8lB8AImSosyDhlgFY1J6RxmJGnicGx6fhxVFexdY Received: from [71.136.233.92] by web45603.mail.sp1.yahoo.com via HTTP; Sun, 14 Oct 2007 22:01:51 PDT Date: Sun, 14 Oct 2007 22:01:51 -0700 (PDT) From: Juri Mianovich To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <759671.62342.qm@web45603.mail.sp1.yahoo.com> Subject: comparing two filesystems with different newfs values ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 05:15:40 -0000 system A has these three partitions that I created in sysinstall, using the default sysinstall newfs values: /dev/aacd0s1e 465759710 434158260 3655868 99% /mount1 /dev/aacd1s1d 1891513834 1746678920 31344084 98% /mount2 /dev/aacd2s1d 2030137706 1801943816 228193890 89% /mount3 So, doing the math, the total space used of the three filesystems is: 3982780996 (roughly 4 TB) I just created a new filesystem on system B, where I used newfs on raw disk to create a SINGLE large >2TB partition. The newfs command I used was: newfs -i 32768 -U /dev/aacd1 I then used rsync to transfer ALL of the data from the old system to the new system. Now that I am done, and I have re-run rsync several times to be sure that all of the data is in place on the new system, the space used on the new system is: 3552249780 That's a difference of almost .5 TB ... and furthermore, I would think having less dense inodes would actually _increase_ the effective space that all those files take up, not _decrease_ it ... So is this expected ? Does it have something to do with moving the data from three partitions to one ? The bottom line is, I want to be SURE that all of the data is transferred before I scrap system A ... rsync is telling me all the data is there, because when I re-run it, nothing new gets transferred ... but I really want to "prove" it by looking at the total file size ... and I can't seem to do that currently. Comments or suggestions ? ____________________________________________________________________________________ Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. http://farechase.yahoo.com/ From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 06:41:34 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2657E16A417 for ; Mon, 15 Oct 2007 06:41:34 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 77F6C13C448 for ; Mon, 15 Oct 2007 06:41:33 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 44355 invoked by uid 2001); 15 Oct 2007 06:41:31 -0000 Date: Mon, 15 Oct 2007 01:41:31 -0500 From: "Rick C. Petty" To: Juri Mianovich Message-ID: <20071015064131.GA44208@keira.kiwi-computer.com> References: <759671.62342.qm@web45603.mail.sp1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <759671.62342.qm@web45603.mail.sp1.yahoo.com> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org Subject: Re: comparing two filesystems with different newfs values ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 06:41:34 -0000 On Sun, Oct 14, 2007 at 10:01:51PM -0700, Juri Mianovich wrote: > > So, doing the math, the total space used of the three > filesystems is: 3982780996 > > (roughly 4 TB) > > I just created a new filesystem on system B, where I > used newfs on raw disk to create a SINGLE large >2TB > partition. The newfs command I used was: > > newfs -i 32768 -U /dev/aacd1 > > I then used rsync to transfer ALL of the data from the Which rsync options did you use? > old system to the new system. Now that I am done, and > I have re-run rsync several times to be sure that all > of the data is in place on the new system, the space How many times is "several"? And why do you need to run it several times? > used on the new system is: > > 3552249780 > > That's a difference of almost .5 TB ... and > furthermore, I would think having less dense inodes > would actually _increase_ the effective space that all > those files take up, not _decrease_ it ... Why would you think that? By "less dense" I'm assuming you mean lowering the inode density value (via the -i newfs option). But lowering it from what? If you don't specify the option, you will get more inodes. Specifying the option generally lowers the total number of inodes allocated. Using a larger number will decrease the total number of inodes. In your case, you actually increased the inode density (the default is about 8192). Therefore, you've reduced the number of inodes available by about one quarter. Also the number of inodes has nothing to do with how much space should be taken up by a given set of files-- just how much space is available on the device for storing data. > So is this expected ? No idea. You haven't provided enough information. I'll assume your "used space" number comes from df(1). Do you have any sparse files? I typically use "rsync -avHSP" which copies sparse files without allocating blocks unnecessarily. Another thing, if you have a lot of directories and those directories were large at one time but are now less full, copying the files will actually allocate fewer blocks for those directories. I've often seen a minor discrepency due to that. > Does it have something to do > with moving the data from three partitions to one ? Only if you copied files ontop of each other, but that's not likely the case if you've repeated the rsync and it didn't transfer files. > The bottom line is, I want to be SURE that all of the > data is transferred before I scrap system A ... rsync > is telling me all the data is there, because when I > re-run it, nothing new gets transferred ... but I There you go. That's your proof. Assuming you use the same rsync command that I do. -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 08:42:17 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5448616A419; Mon, 15 Oct 2007 08:42:17 +0000 (UTC) (envelope-from henrik@brixandersen.dk) Received: from solow.pil.dk (relay.pil.dk [195.41.47.164]) by mx1.freebsd.org (Postfix) with ESMTP id 12C9C13C46B; Mon, 15 Oct 2007 08:42:17 +0000 (UTC) (envelope-from henrik@brixandersen.dk) Received: from tirith.brixandersen.dk (osgiliath.brixandersen.dk [87.53.223.189]) by solow.pil.dk (Postfix) with ESMTP id 7662B1CC0F9; Mon, 15 Oct 2007 10:26:30 +0200 (CEST) Received: by tirith.brixandersen.dk (Postfix, from userid 1001) id AA35517036; Mon, 15 Oct 2007 10:26:29 +0200 (CEST) Date: Mon, 15 Oct 2007 10:26:29 +0200 From: Henrik Brix Andersen To: Claus Guttesen Message-ID: <20071015082629.GB1290@tirith.brixandersen.dk> Mail-Followup-To: Claus Guttesen , Pawel Jakub Dawidek , freebsd-fs@freebsd.org, freebsd-current@freebsd.org References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="d6Gm4EdcadzBjdND" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.brixandersen.dk/files/HenrikBrixAndersen.asc User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, Pawel Jakub Dawidek Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 08:42:17 -0000 --d6Gm4EdcadzBjdND Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 08, 2007 at 07:18:10PM +0200, Claus Guttesen wrote: > > I was able to reproduce the panic by rsyncing big files and trying > > bonnie++ test suggested in this thread. > > > > Can you guys retry with this patch: > > > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > > looking for a solution that can be used for 7.0 before we find a better > > fix. >=20 > Congrats Pawel! You made my server survive my rsync of 90 GB. :-) >=20 > This is on same src as the one that required a reboot except for your > patch. So this fix does 'alleviate kmem_map too small' in my case. While we have come across the 'kmem_map too small' panics in the past, these are now solved - but our rsync processes still hang at random, just spinning the CPU. This is when running multiple rsyncs to the same zpool: http://www.brixandersen.dk/tmp/zfs-rsync-debug.1.txt This is reproduceable (but only after running an intensive set of rsync processes for ~10-12 hours or so) on our i386 RELENG_7 box with the above mentioned patch, 4GB RAM, KVA_PAGES=3D512 and the following settings in loader.conf: vfs.zfs.prefetch_disable=3D"1" vfs.zfs.zil_disable=3D"1" vfs.zfs.arc_max=3D"209715200" vm.kmem_size=3D"1258291200" vm.kmem_size_max=3D"1258291200" Regards, Brix --=20 Henrik Brix Andersen --d6Gm4EdcadzBjdND Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) Comment: GnuPG signed iD8DBQFHEyQ0v+Q4flTiePgRAidfAJ9wrMlBNTAZTdFHju8ycTDLO3j9oQCgojS1 +A03Kj/qCExUv6TZ8AM0xsA= =KyZg -----END PGP SIGNATURE----- --d6Gm4EdcadzBjdND-- From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 09:47:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 593C516A41B; Mon, 15 Oct 2007 09:47:43 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id EB1AA13C44B; Mon, 15 Oct 2007 09:47:42 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 1D63845E93; Mon, 15 Oct 2007 11:47:41 +0200 (CEST) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id CE09045683; Mon, 15 Oct 2007 11:47:28 +0200 (CEST) Date: Mon, 15 Oct 2007 11:47:16 +0200 From: Pawel Jakub Dawidek To: Claus Guttesen , freebsd-fs@freebsd.org, freebsd-current@freebsd.org Message-ID: <20071015094716.GG10170@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071015082629.GB1290@tirith.brixandersen.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kadn00tgSopKmJ1H" Content-Disposition: inline In-Reply-To: <20071015082629.GB1290@tirith.brixandersen.dk> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 09:47:43 -0000 --kadn00tgSopKmJ1H Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 15, 2007 at 10:26:29AM +0200, Henrik Brix Andersen wrote: > On Mon, Oct 08, 2007 at 07:18:10PM +0200, Claus Guttesen wrote: > > > I was able to reproduce the panic by rsyncing big files and trying > > > bonnie++ test suggested in this thread. > > > > > > Can you guys retry with this patch: > > > > > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > > > > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > > > looking for a solution that can be used for 7.0 before we find a bett= er > > > fix. > >=20 > > Congrats Pawel! You made my server survive my rsync of 90 GB. :-) > >=20 > > This is on same src as the one that required a reboot except for your > > patch. So this fix does 'alleviate kmem_map too small' in my case. >=20 > While we have come across the 'kmem_map too small' panics in the past, > these are now solved - but our rsync processes still hang at random, > just spinning the CPU. This is when running multiple rsyncs to the > same zpool: >=20 > http://www.brixandersen.dk/tmp/zfs-rsync-debug.1.txt This looks like VFS deadlock between process 42575 and 42585. You debugging is really nice, but unfortunately there is no backtrace of process 42575, so we don't know where and why it's stuck... Can you next time take 'show lockedvnods', 'show alllocks' (as you did this time), but also 'alltrace'? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --kadn00tgSopKmJ1H Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHEzckForvXbEpPzQRAv5hAJ4jSUxA01dxxvRcFHaAbalKQzGJ9wCgtaBV 2r3jy+/H/uoQlWWzbnTlrwA= =ywo+ -----END PGP SIGNATURE----- --kadn00tgSopKmJ1H-- From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 09:54:06 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B179616A420; Mon, 15 Oct 2007 09:54:06 +0000 (UTC) (envelope-from henrik@brixandersen.dk) Received: from solow.pil.dk (relay.pil.dk [195.41.47.164]) by mx1.freebsd.org (Postfix) with ESMTP id 7342813C4A6; Mon, 15 Oct 2007 09:54:06 +0000 (UTC) (envelope-from henrik@brixandersen.dk) Received: from tirith.brixandersen.dk (osgiliath.brixandersen.dk [87.53.223.189]) by solow.pil.dk (Postfix) with ESMTP id C32661CC0E3; Mon, 15 Oct 2007 11:54:05 +0200 (CEST) Received: by tirith.brixandersen.dk (Postfix, from userid 1001) id 000FB17036; Mon, 15 Oct 2007 11:54:04 +0200 (CEST) Date: Mon, 15 Oct 2007 11:54:04 +0200 From: Henrik Brix Andersen To: Pawel Jakub Dawidek Message-ID: <20071015095404.GF1290@tirith.brixandersen.dk> Mail-Followup-To: Pawel Jakub Dawidek , Claus Guttesen , freebsd-fs@freebsd.org, freebsd-current@freebsd.org References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071015082629.GB1290@tirith.brixandersen.dk> <20071015094716.GG10170@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="raC6veAxrt5nqIoY" Content-Disposition: inline In-Reply-To: <20071015094716.GG10170@garage.freebsd.pl> X-PGP-Key: http://www.brixandersen.dk/files/HenrikBrixAndersen.asc User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, Claus Guttesen Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 09:54:06 -0000 --raC6veAxrt5nqIoY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 15, 2007 at 11:47:16AM +0200, Pawel Jakub Dawidek wrote: > This looks like VFS deadlock between process 42575 and 42585. You > debugging is really nice, but unfortunately there is no backtrace of > process 42575, so we don't know where and why it's stuck... >=20 > Can you next time take 'show lockedvnods', 'show alllocks' (as you did > this time), but also 'alltrace'? Will do. Thank you for the swift reply :) Regards, Brix --=20 Henrik Brix Andersen --raC6veAxrt5nqIoY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) Comment: GnuPG signed iD8DBQFHEzi8v+Q4flTiePgRAnwUAKCYeHd91oHOyoazwWJZmQNCsZm4LACfZ3cC 50O7Zw1VilRrhfXHq0NmUFg= =JhxJ -----END PGP SIGNATURE----- --raC6veAxrt5nqIoY-- From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 11:06:15 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3701716A421 for ; Mon, 15 Oct 2007 11:06:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F2F5813C442 for ; Mon, 15 Oct 2007 11:06:14 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l9FB6ESM080427 for ; Mon, 15 Oct 2007 11:06:14 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l9FB6EqF080425 for freebsd-fs@FreeBSD.org; Mon, 15 Oct 2007 11:06:14 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Oct 2007 11:06:14 GMT Message-Id: <200710151106.l9FB6EqF080425@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 11:06:15 -0000 From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 17:47:07 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E99416A418 for ; Mon, 15 Oct 2007 17:47:07 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0749C13C4A6 for ; Mon, 15 Oct 2007 17:47:07 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l9FHl6cv014938 for ; Mon, 15 Oct 2007 17:47:06 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l9FHl6Xx014934 for freebsd-fs@FreeBSD.org; Mon, 15 Oct 2007 17:47:06 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Oct 2007 17:47:06 GMT Message-Id: <200710151747.l9FHl6Xx014934@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 17:47:07 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/114856 fs [ntfs] [patch] Bug in NTFS allows bogus file modes. o kern/116170 fs Kernel panic when mounting /tmp 4 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/114847 fs [ntfs] [patch] dirmask support for NTFS ala MSDOSFS 1 problem total. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 15:13:06 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16B5D16A418 for ; Tue, 16 Oct 2007 15:13:06 +0000 (UTC) (envelope-from jason@wilma.widomaker.com) Received: from wilma.widomaker.com (ip204-238-183-243.east.widomaker.com [204.238.183.243]) by mx1.freebsd.org (Postfix) with ESMTP id 6A04613C480 for ; Tue, 16 Oct 2007 15:13:05 +0000 (UTC) (envelope-from jason@wilma.widomaker.com) Received: from wilma.widomaker.com (localhost [127.0.0.1]) by wilma.widomaker.com (8.13.8/8.13.6) with ESMTP id l9GErLPq005686; Tue, 16 Oct 2007 10:53:21 -0400 (EDT) (envelope-from jason@wilma.widomaker.com) Received: (from jason@localhost) by wilma.widomaker.com (8.13.8/8.13.6/Submit) id l9GErKcN005685; Tue, 16 Oct 2007 10:53:20 -0400 (EDT) (envelope-from jason) Date: Tue, 16 Oct 2007 10:53:19 -0400 From: Jason Harris To: Juri Mianovich Message-ID: <20071016145319.GA4778@wilma.widomaker.com> References: <759671.62342.qm@web45603.mail.sp1.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8" Content-Disposition: inline In-Reply-To: <759671.62342.qm@web45603.mail.sp1.yahoo.com> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org, Jason Harris Subject: Re: comparing two filesystems with different newfs values ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 15:13:06 -0000 --MGYHOYXEY6WxJCY8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 14, 2007 at 10:01:51PM -0700, Juri Mianovich wrote: > The bottom line is, I want to be SURE that all of the > data is transferred before I scrap system A ... rsync > is telling me all the data is there, because when I > re-run it, nothing new gets transferred ... but I > really want to "prove" it by looking at the total file > size ... and I can't seem to do that currently. If you want files and their sizes only, try: %find . -type f -ls | awk '{print $7 "\t" $11}' | tee `hostname` then "sort +1" them and diff or "sort -m +1 ... | uniq -u" to find the unique entries. If you have the time to hash all that data, try: %find . -type f | xargs /sbin/md5 (or sha1/rmd160/sha256) | tee ... to make sure the copied files hash identically. "rsync -c" will use md4 hash equality instead of size and modtime equality to pronounce files "indentical." But, if you're going to hash the files, you might as well use sha1 or rmd160 at minimum. --=20 Jason Harris | NIC: JH329, PGP: This _is_ PGP-signed, isn't it? jharris@widomaker.com _|_ web: http://keyserver.kjsl.com/~jharris/ Got photons? (TM), (C) 2004 --MGYHOYXEY6WxJCY8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iJwEARECAF0FAkcU0F9WGGh0dHA6Ly9rZXlzZXJ2ZXIua2pzbC5jb206MTEzNzEv cGtzL2xvb2t1cD9vcD1nZXQmc2VhcmNoPTB4RDM5REEwRTMmd2VoYXZleW91bm93 PXRydWUACgkQSypIl9OdoONMOACYjLsyezcUu4bNW4r/jQIB4k0bnACgoyqMgzMd o1cOTvMHVHVwNDU8Mhw= =kK6q -----END PGP SIGNATURE----- --MGYHOYXEY6WxJCY8-- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 16:12:38 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C488B16A41A for ; Tue, 16 Oct 2007 16:12:38 +0000 (UTC) (envelope-from kupfer@ldc.upenn.edu) Received: from lorax.ldc.upenn.edu (lorax.ldc.upenn.edu [158.130.16.184]) by mx1.freebsd.org (Postfix) with ESMTP id AB8BF13C465 for ; Tue, 16 Oct 2007 16:12:37 +0000 (UTC) (envelope-from kupfer@ldc.upenn.edu) Received: by lorax.ldc.upenn.edu (Postfix, from userid 33361) id CE0C5B24B2; Tue, 16 Oct 2007 11:49:13 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lorax.ldc.upenn.edu (Postfix) with ESMTP id C4957B2492 for ; Tue, 16 Oct 2007 11:49:13 -0400 (EDT) Date: Tue, 16 Oct 2007 11:49:13 -0400 (EDT) From: Paul Kupfer To: freebsd-fs@freebsd.org Message-ID: <20071016114035.O64828@lorax.ldc.upenn.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 16:12:38 -0000 I am seeing a very odd problem on a FreeBSD 6.1-PRERELEASE #0 mail server we have. We originally had a separate scsi device mounted as /var/mail (note this was underneath the mountpoint for /var). We reached a point where space on /var was running out, and running a df on that volume would show that there was only 160M out of 5.8G available (and 5.2G used). So I ran "du -hc /var" to see where I could remove some old files. du output shows the total of /var to be 311M, which is vastly different than what du tells me. I thought that this was perhaps a result of the way we had things mounted, and so I unmount the /var/mail device and remounted it as /mail, then put a symlink from /mail to /var/mail, still no dice, same results. I have not rebooted the machine yet (was hoping I wouldn't have to since it is a mail server). Is it possible that the inode tables are "out of whack" since changing these mountpoints, perhaps an fsck? Anyone ever seen anything like this before? Any help would be greatly appreciated. -Paul- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 16:30:08 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B4AB16A418 for ; Tue, 16 Oct 2007 16:30:08 +0000 (UTC) (envelope-from bv@bilver.wjv.com) Received: from wjv.com (fl-65-40-24-38.sta.embarqhsd.net [65.40.24.38]) by mx1.freebsd.org (Postfix) with ESMTP id B5FB513C459 for ; Tue, 16 Oct 2007 16:30:07 +0000 (UTC) (envelope-from bv@bilver.wjv.com) Received: from bilver.wjv.com (localhost.wjv.com [127.0.0.1]) by wjv.com (8.14.1/8.13.1) with ESMTP id l9GGTrpB079356; Tue, 16 Oct 2007 12:29:53 -0400 (EDT) (envelope-from bv@bilver.wjv.com) Received: (from bv@localhost) by bilver.wjv.com (8.14.1/8.13.1/Submit) id l9GGTmDP079355; Tue, 16 Oct 2007 12:29:48 -0400 (EDT) (envelope-from bv) Date: Tue, 16 Oct 2007 12:29:48 -0400 From: Bill Vermillion To: Paul Kupfer Message-ID: <20071016162948.GA79131@wjv.com> References: <20071016114035.O64828@lorax.ldc.upenn.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071016114035.O64828@lorax.ldc.upenn.edu> User-Agent: Mutt/1.4.2.2i Organization: W.J.Vermillion / Orlando - Winter Park ReplyTo: bv@wjv.com Cc: freebsd-fs@freebsd.org Subject: Re: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bv@wjv.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 16:30:08 -0000 On Tue, Oct 16, 2007 at 11:49 Paul Kupfer saw "Error reading FAT table? Try SKINNY table?" And promptly said: > I am seeing a very odd problem on a FreeBSD 6.1-PRERELEASE #0 > mail server we have. We originally had a separate scsi device > mounted as /var/mail (note this was underneath the mountpoint > for /var). We reached a point where space on /var was running > out, and running a df on that volume would show that there was > only 160M out of 5.8G available (and 5.2G used). > So I ran "du -hc /var" to see where I could remove some old > files. du output shows the total of /var to be 311M, which is > vastly different than what du tells me. I thought that this was > perhaps a result of the way we had things mounted, and so I > unmount the /var/mail device and remounted it as /mail, then put > a symlink from /mail to /var/mail, still no dice, same results. > I have not rebooted the machine yet (was hoping I wouldn't have > to since it is a mail server). Is it possible that the inode > tables are "out of whack" since changing these mountpoints, > perhaps an fsck? Anyone ever seen anything like this before? > Any help would be greatly appreciated. I've seen it many times in my XX years of Unix work. du - shows the amount of blocks used by the system df - shows the amunt of space it thinks is being used. The latter will compute the length of sparse files so df will show less free space than du. Sparse file will have blocks of no data in them. These are typically created by database programs. And if you removed files that were open by some application, you won't be able to see those files but the application will have them open and still be adding to the lenght of the file. When removing files like these - shutdown the ap - remove the file and restart the ap. Depending upon the application - if it's something being run out of syslog, you may have to 'touch' the file, as syslog will not create a missing file unless you give it the -C option - only available in releases after the 4.x series. Bill -- Bill Vermillion - bv @ wjv . com From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 21:25:29 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2583B16A473 for ; Tue, 16 Oct 2007 21:25:29 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id A9AA513C478 for ; Tue, 16 Oct 2007 21:25:28 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id B6CC545F3F; Tue, 16 Oct 2007 23:25:26 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id C365945E8F; Tue, 16 Oct 2007 23:25:20 +0200 (CEST) Date: Tue, 16 Oct 2007 23:25:02 +0200 From: Pawel Jakub Dawidek To: Bill Vermillion Message-ID: <20071016212502.GJ16627@garage.freebsd.pl> References: <20071016114035.O64828@lorax.ldc.upenn.edu> <20071016162948.GA79131@wjv.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="++alDQ2ROsODg1x+" Content-Disposition: inline In-Reply-To: <20071016162948.GA79131@wjv.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, Paul Kupfer Subject: Re: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 21:25:29 -0000 --++alDQ2ROsODg1x+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 16, 2007 at 12:29:48PM -0400, Bill Vermillion wrote: > du - shows the amount of blocks used by the system > df - shows the amunt of space it thinks is being used. >=20 > The latter will compute the length of sparse files so df > will show less free space than du. Sparse file will have=20 > blocks of no data in them. These are typically created > by database programs. Please, verify before giving an answer. Neither du(1) nor df(1) include sparse blocks in their calculations: # ls -lh /mnt/tmp/foo -rw-r----- 1 root wheel 10G 16 pa# 23:19 /mnt/tmp/foo # df -h /mnt/tmp/foo Filesystem Size Used Avail Capacity Mounted on /dev/md0 9.4M 52K 8.6M 1% /mnt/tmp --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --++alDQ2ROsODg1x+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHFSwuForvXbEpPzQRAtAEAJ9GY8F+gTUadyRUMlw51TP9ukVSvACfVI/w CoexGKv6VrgRhb6mPuyi64w= =k+VL -----END PGP SIGNATURE----- --++alDQ2ROsODg1x+-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 11:49:31 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90F5116A421 for ; Wed, 17 Oct 2007 11:49:31 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4C7FA13C4B2 for ; Wed, 17 Oct 2007 11:49:31 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 8D2C32093; Wed, 17 Oct 2007 13:49:21 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 6F470208D; Wed, 17 Oct 2007 13:49:21 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 559B584486; Wed, 17 Oct 2007 13:49:21 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: bv@wjv.com References: <20071016114035.O64828@lorax.ldc.upenn.edu> <20071016162948.GA79131@wjv.com> Date: Wed, 17 Oct 2007 13:49:21 +0200 In-Reply-To: <20071016162948.GA79131@wjv.com> (Bill Vermillion's message of "Tue\, 16 Oct 2007 12\:29\:48 -0400") Message-ID: <86bqayezb2.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Paul Kupfer Subject: Re: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:49:31 -0000 Bill Vermillion writes: > I've seen it many times in my XX years of Unix work. > > du - shows the amount of blocks used by the system > df - shows the amunt of space it thinks is being used. > > The latter will compute the length of sparse files so df will show > less free space than du. Sparse file will have blocks of no data in > them. These are typically created by database programs. Utterly wrong. df will show the amount of space used in the file system, including space used by files that have no directory entry (e.g. temporary files which the application unlinks immediately after opening) du will show the amount of space used by the files it is able to find by traversing the directories specified on the command line. Both will DTRT for sparse files. Note that when running 'du *' in the root of a file system, the shell will most likely not include directories and files whose names begin with a dot in its expansion of '*'. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 11:56:20 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2BE7816A41B for ; Wed, 17 Oct 2007 11:56:20 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id E6F9A13C468 for ; Wed, 17 Oct 2007 11:56:19 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l9HBuIqS072294; Wed, 17 Oct 2007 06:56:19 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <4715F859.9060102@freebsd.org> Date: Wed, 17 Oct 2007 06:56:09 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Paul Kupfer References: <20071016114035.O64828@lorax.ldc.upenn.edu> In-Reply-To: <20071016114035.O64828@lorax.ldc.upenn.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-fs@freebsd.org Subject: Re: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:56:20 -0000 Paul Kupfer wrote: > > I am seeing a very odd problem on a FreeBSD 6.1-PRERELEASE #0 mail > server we have. We originally had a separate scsi device mounted as > /var/mail (note this was underneath the mountpoint for /var). We > reached a point where space on /var was running out, and running a df on > that volume would show that there was only 160M out of 5.8G available > (and 5.2G used). > > So I ran "du -hc /var" to see where I could remove some old files. du > output shows the total of /var to be 311M, which is vastly different > than what du tells me. I thought that this was perhaps a result of the > way we had things mounted, and so I unmount the /var/mail device and > remounted it as /mail, then put a symlink from /mail to /var/mail, still > no dice, same results. > > I have not rebooted the machine yet (was hoping I wouldn't have to > since it is a mail server). Is it possible that the inode tables are > "out of whack" since changing these mountpoints, perhaps an fsck? > Anyone ever seen anything like this before? > > Any help would be greatly appreciated. My guess is that you run apache, and the log file got huge, rotated out, but apache still has it open. You could install lsof possibly and see what files it has open, and it will show you their size also. Restarting apache should also clear it. Eric From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 14:44:17 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A572816A417 for ; Wed, 17 Oct 2007 14:44:17 +0000 (UTC) (envelope-from kupfer@ldc.upenn.edu) Received: from lorax.ldc.upenn.edu (lorax.ldc.upenn.edu [158.130.16.184]) by mx1.freebsd.org (Postfix) with ESMTP id 727EA13C45D for ; Wed, 17 Oct 2007 14:44:17 +0000 (UTC) (envelope-from kupfer@ldc.upenn.edu) Received: by lorax.ldc.upenn.edu (Postfix, from userid 33361) id 6359DB2482; Wed, 17 Oct 2007 10:44:19 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lorax.ldc.upenn.edu (Postfix) with ESMTP id 5A0F3B2480; Wed, 17 Oct 2007 10:44:19 -0400 (EDT) Date: Wed, 17 Oct 2007 10:44:19 -0400 (EDT) From: Paul Kupfer To: Eric Anderson In-Reply-To: <4715F859.9060102@freebsd.org> Message-ID: <20071017104120.S47584@lorax.ldc.upenn.edu> References: <20071016114035.O64828@lorax.ldc.upenn.edu> <4715F859.9060102@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: du and df discrepancy X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 14:44:17 -0000 On Wed, 17 Oct 2007, Eric Anderson wrote: > Paul Kupfer wrote: >> >> I am seeing a very odd problem on a FreeBSD 6.1-PRERELEASE #0 mail server >> we have. We originally had a separate scsi device mounted as /var/mail >> (note this was underneath the mountpoint for /var). We reached a point >> where space on /var was running out, and running a df on that volume would >> show that there was only 160M out of 5.8G available (and 5.2G used). >> >> So I ran "du -hc /var" to see where I could remove some old files. du >> output shows the total of /var to be 311M, which is vastly different than >> what du tells me. I thought that this was perhaps a result of the way we >> had things mounted, and so I unmount the /var/mail device and remounted it >> as /mail, then put a symlink from /mail to /var/mail, still no dice, same >> results. >> >> I have not rebooted the machine yet (was hoping I wouldn't have to since >> it is a mail server). Is it possible that the inode tables are "out of >> whack" since changing these mountpoints, perhaps an fsck? Anyone ever seen >> anything like this before? >> >> Any help would be greatly appreciated. > > > My guess is that you run apache, and the log file got huge, rotated out, but > apache still has it open. You could install lsof possibly and see what files > it has open, and it will show you their size also. Restarting apache should > also clear it. This was right on. I restarted apache and voila! Thank you all for your help. It appears that the inodes for these files were still locked by apache, see the description above. -Paul- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 22:27:32 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3A3116A418 for ; Wed, 17 Oct 2007 22:27:31 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from signal.itea.ntnu.no (signal.itea.ntnu.no [129.241.190.231]) by mx1.freebsd.org (Postfix) with ESMTP id B718113C459 for ; Wed, 17 Oct 2007 22:27:30 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from localhost (localhost [127.0.0.1]) by signal.itea.ntnu.no (Postfix) with ESMTP id 955E534486 for ; Thu, 18 Oct 2007 00:09:40 +0200 (CEST) Received: from caracal.stud.ntnu.no (caracal.stud.ntnu.no [129.241.56.185]) by signal.itea.ntnu.no (Postfix) with ESMTP for ; Thu, 18 Oct 2007 00:09:40 +0200 (CEST) Received: by caracal.stud.ntnu.no (Postfix, from userid 2312) id 3C4EE6240FD; Thu, 18 Oct 2007 00:09:48 +0200 (CEST) Date: Thu, 18 Oct 2007 00:09:48 +0200 From: Ulf Lilleengen To: freebsd-fs@freebsd.org Message-ID: <20071017220948.GA4279@stud.ntnu.no> References: <20070816100526.GA31897@stud.ntnu.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070816100526.GA31897@stud.ntnu.no> User-Agent: Mutt/1.5.9i X-Content-Scanned: with sophos and spamassassin at mailgw.ntnu.no. Subject: Re: [PATCH] Make fdescfs MPSAFE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 22:27:32 -0000 On tor, aug 16, 2007 at 12:05:26 +0200, Ulf Lilleengen wrote: > Hi, > > To be able to better understand VFS and locking in general, I started making > fdescfs MPSAFE. I'm not experienced with any of these things, so there might be > some errors, although I've looked through much VFS code and code for other FS > like nullfs. I've tested it by running two pthreads on the same fd, and that seamt > to work, but there might be other cases where it will fail. > > Patch is attached. > Attached is a new patch that uses rwlocks instead of mutexes (since reading the hash_table is frequently done). Also, it adds checking so that there is no duplicates in the hash table before inserting the new fdescnode. And add a mising hashfree(). I also checked to see if there was any issues regarding Jeffs new patch to use atomic operations on the file structure, but there wasn't any obvious places where this affects fdescfs. Patch here: http://folk.ntnu.no/lulf/patches/freebsd/fdescfs/fdescfs_lock.diff -- Ulf Lilleengen From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 06:11:03 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC9DE16A418; Thu, 18 Oct 2007 06:11:03 +0000 (UTC) (envelope-from darrenr@freebsd.org) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id BAD3713C48A; Thu, 18 Oct 2007 06:11:03 +0000 (UTC) (envelope-from darrenr@freebsd.org) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id 8EF3B2FDCF; Thu, 18 Oct 2007 02:11:02 -0400 (EDT) Received: from heartbeat1.messagingengine.com ([10.202.2.160]) by compute1.internal (MEProxy); Thu, 18 Oct 2007 02:11:02 -0400 X-Sasl-enc: cJ5nfcCaoU07XwcK2SOe+82l+0ohlXPJWswC5yjXdEXo 1192687862 Received: from [192.168.1.235] (64-142-85-108.dsl.dynamic.sonic.net [64.142.85.108]) by mail.messagingengine.com (Postfix) with ESMTP id ECD6716B; Thu, 18 Oct 2007 02:11:01 -0400 (EDT) Message-ID: <4716F8FD.1080901@freebsd.org> Date: Wed, 17 Oct 2007 23:11:09 -0700 From: Darren Reed User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> In-Reply-To: <20071008121523.GM2327@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 06:11:04 -0000 So, I just had another of these panics: panic: kmem_alloc(131072): kmem_map too small: 400011264 total allocated what I did to provoke it: zfs rollback room/freebsd@presnap From loader.conf: vm.kmem_size=536870912 vm.kmem_size_max=536870912 vfs.zfs.arc_max=409715200 vfs.zfs.prefetch_disable="1" Darren From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 07:33:41 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2803C16A41B for ; Thu, 18 Oct 2007 07:33:41 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id 6EBDC13C4B3; Thu, 18 Oct 2007 07:33:40 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47170C54.6050502@FreeBSD.org> Date: Thu, 18 Oct 2007 09:33:40 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Ulf Lilleengen References: <20070816100526.GA31897@stud.ntnu.no> <20071017220948.GA4279@stud.ntnu.no> In-Reply-To: <20071017220948.GA4279@stud.ntnu.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: [PATCH] Make fdescfs MPSAFE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 07:33:41 -0000 Ulf Lilleengen wrote: > On tor, aug 16, 2007 at 12:05:26 +0200, Ulf Lilleengen wrote: >> Hi, >> >> To be able to better understand VFS and locking in general, I started making >> fdescfs MPSAFE. I'm not experienced with any of these things, so there might be >> some errors, although I've looked through much VFS code and code for other FS >> like nullfs. I've tested it by running two pthreads on the same fd, and that seamt >> to work, but there might be other cases where it will fail. >> >> Patch is attached. >> > > Attached is a new patch that uses rwlocks instead of mutexes (since reading > the hash_table is frequently done). Also, it adds checking so that there is no > duplicates in the hash table before inserting the new fdescnode. And add a > mising hashfree(). > > I also checked to see if there was any issues regarding Jeffs new patch to > use atomic operations on the file structure, but there wasn't any obvious > places where this affects fdescfs. > > Patch here: > http://folk.ntnu.no/lulf/patches/freebsd/fdescfs/fdescfs_lock.diff This might be OK but you should be aware that rwlocks can be slower than mutexes when there is a suitably mixed read/write workload. We don't do the same adaptive spinning for wlocks as for mutexes when they are held by shared holders (since we don't track who they are so can't track whether they're running), and it is possible for readers to starve writers. If possible some benchmarks trying to find the worst case behaviour would be useful. Kris From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 08:34:42 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0CAF16A420 for ; Thu, 18 Oct 2007 08:34:42 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from merke.itea.ntnu.no (merke.itea.ntnu.no [129.241.7.61]) by mx1.freebsd.org (Postfix) with ESMTP id 5217C13C47E for ; Thu, 18 Oct 2007 08:34:42 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from localhost (localhost [127.0.0.1]) by merke.itea.ntnu.no (Postfix) with ESMTP id 6F75B13C4C4; Thu, 18 Oct 2007 10:34:40 +0200 (CEST) Received: from caracal.stud.ntnu.no (caracal.stud.ntnu.no [129.241.56.185]) by merke.itea.ntnu.no (Postfix) with ESMTP; Thu, 18 Oct 2007 10:34:40 +0200 (CEST) Received: by caracal.stud.ntnu.no (Postfix, from userid 2312) id 411AF6240F9; Thu, 18 Oct 2007 10:34:48 +0200 (CEST) Date: Thu, 18 Oct 2007 10:34:48 +0200 From: Ulf Lilleengen To: Kris Kennaway Message-ID: <20071018083448.GA1079@stud.ntnu.no> References: <20070816100526.GA31897@stud.ntnu.no> <20071017220948.GA4279@stud.ntnu.no> <47170C54.6050502@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47170C54.6050502@FreeBSD.org> User-Agent: Mutt/1.5.9i X-Content-Scanned: with sophos and spamassassin at mailgw.ntnu.no. Cc: freebsd-fs@freebsd.org Subject: Re: [PATCH] Make fdescfs MPSAFE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 08:34:42 -0000 On tor, okt 18, 2007 at 09:33:40 +0200, Kris Kennaway wrote: > Ulf Lilleengen wrote: > >On tor, aug 16, 2007 at 12:05:26 +0200, Ulf Lilleengen wrote: > >>Hi, > >> > >>To be able to better understand VFS and locking in general, I started > >>making > >>fdescfs MPSAFE. I'm not experienced with any of these things, so there > >>might be > >>some errors, although I've looked through much VFS code and code for > >>other FS > >>like nullfs. I've tested it by running two pthreads on the same fd, and > >>that seamt > >>to work, but there might be other cases where it will fail. > >> > >>Patch is attached. > >> > > > >Attached is a new patch that uses rwlocks instead of mutexes (since reading > >the hash_table is frequently done). Also, it adds checking so that there > >is no > >duplicates in the hash table before inserting the new fdescnode. And add a > >mising hashfree(). > > > >I also checked to see if there was any issues regarding Jeffs new patch to > >use atomic operations on the file structure, but there wasn't any obvious > >places where this affects fdescfs. > > > >Patch here: > >http://folk.ntnu.no/lulf/patches/freebsd/fdescfs/fdescfs_lock.diff > > This might be OK but you should be aware that rwlocks can be slower than > mutexes when there is a suitably mixed read/write workload. We don't do > the same adaptive spinning for wlocks as for mutexes when they are held > by shared holders (since we don't track who they are so can't track > whether they're running), and it is possible for readers to starve writers. > > If possible some benchmarks trying to find the worst case behaviour > would be useful. Good point. I guess having a benchmark where one would open #hashentries files, and then have #hashentries threads reading (accessing the files) and one thread writing (closing perhaps) could produce the starvation? I got the impression from locking(9) that a mutex was essentially a wrwlock, but if that's not exactly true... then I don't think this part is very heavy contested anyway, so perhaps changing it back to a mutex saves me for a lot of trouble. (nullfs does this with mutexes btw). -- Ulf Lilleengen From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 08:47:05 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99F2916A419 for ; Thu, 18 Oct 2007 08:47:05 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id E567213C48D; Thu, 18 Oct 2007 08:47:04 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47171D89.7030906@FreeBSD.org> Date: Thu, 18 Oct 2007 10:47:05 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Ulf Lilleengen References: <20070816100526.GA31897@stud.ntnu.no> <20071017220948.GA4279@stud.ntnu.no> <47170C54.6050502@FreeBSD.org> <20071018083448.GA1079@stud.ntnu.no> In-Reply-To: <20071018083448.GA1079@stud.ntnu.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: [PATCH] Make fdescfs MPSAFE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 08:47:05 -0000 Ulf Lilleengen wrote: > On tor, okt 18, 2007 at 09:33:40 +0200, Kris Kennaway wrote: >> Ulf Lilleengen wrote: >>> On tor, aug 16, 2007 at 12:05:26 +0200, Ulf Lilleengen wrote: >>>> Hi, >>>> >>>> To be able to better understand VFS and locking in general, I started >>>> making >>>> fdescfs MPSAFE. I'm not experienced with any of these things, so there >>>> might be >>>> some errors, although I've looked through much VFS code and code for >>>> other FS >>>> like nullfs. I've tested it by running two pthreads on the same fd, and >>>> that seamt >>>> to work, but there might be other cases where it will fail. >>>> >>>> Patch is attached. >>>> >>> Attached is a new patch that uses rwlocks instead of mutexes (since reading >>> the hash_table is frequently done). Also, it adds checking so that there >>> is no >>> duplicates in the hash table before inserting the new fdescnode. And add a >>> mising hashfree(). >>> >>> I also checked to see if there was any issues regarding Jeffs new patch to >>> use atomic operations on the file structure, but there wasn't any obvious >>> places where this affects fdescfs. >>> >>> Patch here: >>> http://folk.ntnu.no/lulf/patches/freebsd/fdescfs/fdescfs_lock.diff >> This might be OK but you should be aware that rwlocks can be slower than >> mutexes when there is a suitably mixed read/write workload. We don't do >> the same adaptive spinning for wlocks as for mutexes when they are held >> by shared holders (since we don't track who they are so can't track >> whether they're running), and it is possible for readers to starve writers. >> >> If possible some benchmarks trying to find the worst case behaviour >> would be useful. > Good point. I guess having a benchmark where one would open #hashentries > files, and then have #hashentries threads reading (accessing the files) and > one thread writing (closing perhaps) could produce the starvation? Perhaps, give it a try :) > I got the impression from locking(9) that a mutex was essentially a wrwlock, > but if that's not exactly true... Conceptually yes, but as mentioned wlocks do not have some of the optimizations that are important for mutexes. > then I don't think this part is very heavy > contested anyway, so perhaps changing it back to a mutex saves me for a lot > of trouble. (nullfs does this with mutexes btw). Well, I wouldn't suggest giving up without trying. It is always important to benchmark SMP changes no matter what you are doing (e.g. try to find the worst case scenario and make sure it's not worse), because sometimes there are effects you didnt think of. Kris