From owner-freebsd-fs@FreeBSD.ORG Sun Nov 25 01:32:22 2007 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74CED16A417 for ; Sun, 25 Nov 2007 01:32:22 +0000 (UTC) (envelope-from vova@sw.ru) Received: from relay.sw.ru (mailhub.sw.ru [195.214.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id EC78713C447 for ; Sun, 25 Nov 2007 01:32:21 +0000 (UTC) (envelope-from vova@sw.ru) Received: from vbook.fbsd.ru (swsoft-msk-nat.sw.ru [195.214.232.10]) by relay.sw.ru (8.13.4/8.13.4) with ESMTP id lAP0u3ac031558 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 25 Nov 2007 03:56:05 +0300 (MSK) Received: from vova by vbook.fbsd.ru with local (Exim 4.68 (FreeBSD)) (envelope-from ) id 1Iw5Ji-0001Ta-BM for fs@freebsd.org; Sun, 25 Nov 2007 03:25:46 +0300 From: Vladimir Grebenschikov To: fs@freebsd.org Content-Type: text/plain Content-Transfer-Encoding: 7bit Organization: SWsoft Date: Sun, 25 Nov 2007 03:25:45 +0300 Message-Id: <1195950345.4014.15.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.12.1 FreeBSD GNOME Team Port Sender: Vladimir Grebenschikov Cc: Subject: fsck -y failed to cure UFS2 volume with lost '.' entry in one direcotry X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vova@fbsd.ru List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2007 01:32:22 -0000 Hi fs I have a trouble with incurable /usr partition. Usual output of background fsck after unclean shutdown is: Nov 24 20:50:58 vbook fsck: /dev/ad0s2d: MISSING '.' I=7113132 OWNER=root MODE=40755 Nov 24 20:50:58 vbook fsck: /dev/ad0s2d: SIZE=512 MTIME=Oct 16 02:11 2007 Nov 24 20:50:58 vbook fsck: /dev/ad0s2d: DIR=/compat/linux/usr/lib/locale/af_ZA Nov 24 20:50:58 vbook fsck: Nov 24 20:50:58 vbook fsck: /dev/ad0s2d: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. After that everything works fine until reboot. On reboot it starts foreground fsck and fails. Then, after manual fsck /usr it complains about error, but marks FS as clean. Then, after boot (usual boot without problems) - directory still have no '.' entry: # ls -al /compat/linux/usr/lib/locale/af_ZA total 16 drwxr-xr-x 3 root wheel 6656 Oct 16 02:18 .. # ls -ldi /compat/linux/usr/lib/locale/af_ZA/ 7113132 drwxr-xr-x 1 root wheel 512 Nov 25 02:14 /compat/linux/usr/lib/locale/af_ZA/ # and that directory can't be removed: # rmdir /compat/linux/usr/lib/locale/af_ZA rmdir: /compat/linux/usr/lib/locale/af_ZA: Invalid argument # # fsdb /dev/ad0s2d ** /dev/ad0s2d (NO WRITE) Editing file system `/dev/ad0s2d' Last Mounted on /usr current inode: directory I=2 MODE=40755 SIZE=512 BTIME=May 7 08:50:14 2006 [0 nsec] MTIME=Nov 24 20:41:39 2007 [0 nsec] CTIME=Nov 24 20:41:41 2007 [0 nsec] ATIME=Nov 25 03:01:02 2007 [0 nsec] OWNER=root GRP=wheel LINKCNT=24 FLAGS=0 BLKCNT=4 GEN=13bb592b fsdb (inum: 2)> inode 7113132 current inode: directory I=7113132 MODE=40755 SIZE=512 BTIME=Jun 26 22:58:21 2007 [0 nsec] MTIME=Nov 25 02:14:10 2007 [0 nsec] CTIME=Nov 25 02:14:10 2007 [0 nsec] ATIME=Nov 25 03:14:27 2007 [0 nsec] OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=4 GEN=98380ba fsdb (inum: 7113132)> ls slot 0 ino 0 reclen 20: regular, `LC_MESSAGES' slot 1 ino 6853672 reclen 492: directory, `..' fsdb (inum: 7113132)> Looks like - a bug in fsck. Any hints, how I can cure such partition ? Looks like I can do 'ln 7113132 .' in fsdb, or just 'clear 7113132' But I am not sure that it is right way to move. Also I think it is better to report bug in fsck. # uname -a FreeBSD vbook.fbsd.ru 7.0-BETA2 FreeBSD 7.0-BETA2 #39: Fri Nov 9 10:17:40 MSK 2007 root@vbook.fbsd.ru:/usr/obj/usr/src/sys/VBOOK i386 # dumpfs /dev/ad0s2d | head -n 20 magic 19540119 (UFS2) time Sun Nov 25 03:20:09 2007 superblock location 65536 id [ 464080f5 d57b82de ] ncg 388 size 36437305 blocks 35288781 bsize 16384 shift 14 mask 0xffffc000 fsize 2048 shift 11 mask 0xfffff800 frag 8 shift 3 fsbtodb 2 minfree 8% optim time symlinklen 120 maxbsize 16384 maxbpg 2048 maxcontig 8 contigsumsize 8 nbfree 473556 ndir 225259 nifree 7740525 nffree 216969 bpg 11761 fpg 94088 ipg 23552 unrefs 0 nindir 2048 inopb 64 maxfilesize 140806241583103 sbsize 2048 cgsize 16384 csaddr 3000 cssize 8192 sblkno 40 cblkno 48 iblkno 56 dblkno 3000 cgrotor 340 fmod 0 ronly 0 clean 0 avgfpdir 64 avgfilesize 16384 flags soft-updates fsmnt /usr volname swuid 0 -- Vladimir B. Grebenschikov vova@fbsd.ru From owner-freebsd-fs@FreeBSD.ORG Sun Nov 25 21:52:25 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F95916A41B; Sun, 25 Nov 2007 21:52:25 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EC11413C45A; Sun, 25 Nov 2007 21:52:24 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lAPLqOS7088479; Sun, 25 Nov 2007 21:52:24 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lAPLqOuB088475; Sun, 25 Nov 2007 21:52:24 GMT (envelope-from remko) Date: Sun, 25 Nov 2007 21:52:24 GMT Message-Id: <200711252152.lAPLqOuB088475@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: bin/118249: mv(1): moving a directory changes its mtime X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2007 21:52:25 -0000 Synopsis: mv(1): moving a directory changes its mtime Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Sun Nov 25 21:52:09 UTC 2007 Responsible-Changed-Why: reassign to filesystem guru's ;) http://www.freebsd.org/cgi/query-pr.cgi?pr=118249 From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 07:00:06 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F79716A41A for ; Mon, 26 Nov 2007 07:00:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1EE9F13C43E for ; Mon, 26 Nov 2007 07:00:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lAQ705M6012440 for ; Mon, 26 Nov 2007 07:00:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lAQ705ue012439; Mon, 26 Nov 2007 07:00:05 GMT (envelope-from gnats) Date: Mon, 26 Nov 2007 07:00:05 GMT Message-Id: <200711260700.lAQ705ue012439@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Bruce Evans Cc: Subject: Re: misc/118249: moving a directory changes its mtime X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Bruce Evans List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2007 07:00:06 -0000 The following reply was made to PR bin/118249; it has been noted by GNATS. From: Bruce Evans To: Joe Peterson Cc: freebsd-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org Subject: Re: misc/118249: moving a directory changes its mtime Date: Mon, 26 Nov 2007 17:54:12 +1100 (EST) On Sun, 25 Nov 2007, Joe Peterson wrote: >> Description: > Moving a directory to a different directory changes its "mtime". This behavior seems odd compared with other "Unix" systems (tried on Mac OS X and Linux). Also, moving a file to a different directory does *not* change its mtime, making this behavior inconsistent. Also, it is not typically desirable to touch mtime simply by being moved, which loses track of the last time the dir's contents were actually changed. Please use line much shorter than 417 characters. >> How-To-Repeat: > mkdir a b > (check timestamps using stat or "ls -ld" and "ls -lcd") > mv b a > (check timestamps again) > > Both "a" and "b" will now have new mtime and ctime). It is expected that "a" will have a new mtime and ctime, but only the ctime on "b" should have changed. b's contents did change -- its ".." entry moved. However, POSIX only requires marking for update the ctime and mtime of the parent directory for each file (only upon successful completion). I've been running regression tests on timestamps for rename() for more than 15 years and am surprised that they don't notice this bug. The (mis)implementation of marking for update the ctime and mtime of the moved directory seems to be just to call some function (probably ufs_direnter()) which does the marking. ufs_rename() only sets IN_CHANGE and IN_RENAME directly. ufs_rename() has a related bug on unsuccessful completion. It unnecessarily marks IN_CHANGE near its beginning, long before successful completion, so rename() usually clobbers ctimes on failure. This is easy to fix by setting the correct flag for marking inode modifications (IN_MODIFIED). (IN_CHANGE used to mean inode-modified, but is now just the ctime update mark, apart from this and some similar bugs.) With this fix, there are no direct settings of IN_CHANGE left in ufs_rename(). According to my notes, ufs_direnter() and ufs)dirremove are resposible for marking all the necessary updates, and they have the same bug of doing this before successful completion. My regression tests haven't reported any failures from them but I think failures can occur for disk-full and I/O errors and the former is easy to test. mv across file systems clobbers directory times and much more (links...). Bruce From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 11:06:58 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73AC716A4AC for ; Mon, 26 Nov 2007 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5DEA213C4E3 for ; Mon, 26 Nov 2007 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lAQB6wvR025419 for ; Mon, 26 Nov 2007 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lAQB6vBd025415 for freebsd-fs@FreeBSD.org; Mon, 26 Nov 2007 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Nov 2007 11:06:57 GMT Message-Id: <200711261106.lAQB6vBd025415@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2007 11:06:58 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/114856 fs [ntfs] [patch] Bug in NTFS allows bogus file modes. o kern/116170 fs Kernel panic when mounting /tmp 4 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/114847 fs [ntfs] [patch] dirmask support for NTFS ala MSDOSFS o bin/118249 fs mv(1): moving a directory changes its mtime 2 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 14:17:24 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 532B216A498 for ; Mon, 26 Nov 2007 14:17:24 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id A3A0413C45A for ; Mon, 26 Nov 2007 14:17:23 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from localhost (marvin-mail [192.168.0.2]) by marvin.harmless.hu (Postfix) with ESMTP id D42357C0CE8 for ; Mon, 26 Nov 2007 14:56:51 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) (Debian) at harmless.hu Received: from marvin.harmless.hu ([192.168.0.2]) by localhost (marvin.harmless.hu [192.168.0.2]) (amavisd-new, port 10024) with ESMTP id c1FthZV4w2al for ; Mon, 26 Nov 2007 14:56:51 +0100 (CET) Received: from marvin.harmless.hu (localhost [127.0.0.1]) by marvin.harmless.hu (Postfix) with ESMTP id 28F907C0CE6 for ; Mon, 26 Nov 2007 14:56:50 +0100 (CET) Date: Mon, 26 Nov 2007 14:56:50 +0100 From: Gergely CZUCZY To: freebsd-fs@freebsd.org Message-ID: <20071126135650.GA97352@harmless.hu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=x-unknown; protocol="application/pgp-signature"; boundary="YiEDa0DAkWCtVeE4" Content-Disposition: inline User-Agent: mutt-ng/devel-r804 (FreeBSD) Subject: mysql on ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2007 14:17:24 -0000 --YiEDa0DAkWCtVeE4 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I'm trying to test a MySQL on ZFS setup, and I'm having some troubles. At first I've copied all the mysql data files from the UFS filesystem to the ZFS one, started mysqld and launched sysbench at it. On the UFS mysql was using around 100-180% CPU and did the benchmark tasks pretty quick (in 60-120 seconds), and now it just stands there for at least 10 minutes. In top I see mysql in zfs state: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 6244 mysql 3 45 0 2071M 202M zfs:(& 1 0:47 0.98% mysqld In gstat I see the ZFS is reading all both disks in the zmirror like a madman, but mysql pretty does nothing it seems. mysqladmin says it has around 23qps while on UFS it used to report 6-800qps. vmstats shows if the system weren't on load entirely: r b w avm fre flt re pi po fr sr ad2 da0 in sy cs us= sy id 0 0 0 2242452 3224164 180 0 0 0 2862 0 0 0 407 1097 3299 = 0 5 95 FreeBSD sees 6G memory in this box (i've got 14G in it, I haven't had the time to check yet why it doesn't see the rest. it used to see all the memory). FreeBSD sqltest.in.publishing.hu 7.0-BETA2 FreeBSD 7.0-BETA2 #3: Mon Nov 5= 10:49:48 CET 2007 toor@sqltest.in.publishing.hu:/usr/obj/usr/src/sys/S= QLTEST amd64 mysql-server-5.0.45_1 Multithreaded SQL database (server) pool: zm state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zm ONLINE 0 0 0 mirror ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 errors: No known data errors Filesystem Size Used Avail Capacity Mounted on zm 40G 128K 40G 0% /zm zm/qemu 40G 128K 40G 0% /zm/qemu zm/mysql 42G 2.0G 40G 5% /zm/mysql /zm/mysql 42G 2.0G 40G 5% /var/db/mysql (yes, i'm using it through a nullmount) In mysql I'm using myisam (no chance for innodb). I think this behaviour is quite abnormal. Did anybody met this issue, is this known? Where should I start digging around to solve this (tried google, not much help)? Where can be the bottleneck that gives this whole thing? What have I done wrong during the setup? I need a bit of help to make this scenario work... Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --YiEDa0DAkWCtVeE4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) owGNVk2LI8cZXntxDgXG+OoQ8oKxdwaPvnqkGUlmWI8141llV7PrkYZNbLBdUpek 8nRXaaqqpe25G3LIIeRqQ8gPCCSQQ8hPcCAfkLtJyCG3kH+Q5+3umZ0EjLeEuruq 3q963q/6+at377z8+p9+87uP3/nZL7586dfix9PtNAvBLGqpdGttaq1ms1Xr7e92 otpubdZq9eY9td9td5vdzsmP/vKrgTVBmVCb5CvVp6CehcYqkdq8S7OldF6FgyzM a11xTXek/cp6HbQ1fdIm0Ubd7E2cNH6uXO3YzGyszaJPl5kNKq6tnDZBThMlxAOV JHZHiOG9lILLQUXBQrEPJGmUjz98RNbQRx+MCcqz1Q5JExMTL+Waib1NlQjOZpDm 63QYaK4dmIf31opmdqVVTDJJKCwVpbm/TCiWQYII5DR3NuUdcQ75xVLug0oLC0DP Wq1RO+SDdLC7FBAXJiQyM7Ml1sAyVfgUMpAOdSEem4KZRZYKN9JT5tlYCTvBCx/U Wt3mWzR4cl4Ii3Vc8BSC4KgLEaS/8LRyKoQcqOnZBW1pQ3tgjJqAYmZN7LdLNIzd QDN9nuHYsBQbLMwpMbeOYFWiJHZaTUq1yYAsbBzCRruiISRdwwLpV3PPAoLqC6In wyM6Hx+fnR6OjokmD87oydmQTocDzGg8/IhfdHY8pvHkcILJANPJkGnpKZ9r8Hg0 Ojw9ErQXtduVjnLsErU7/NEkipr7rRHxOxqx/v7W29TirX57H896r/tWhXph9ILN q8y+9pD25JSMC3zh6KkNSwDK8OnSE1epdg5QJPpCCUmpjFNpdmiahcqsCubYIiQM 2FkUAIWSFCFV0IAH0rzMITWIJTxa+TLavVx52iwRPByo7HWwZh6RgShyamVdoL1a t9kEHZBfp3wE4Ly0G8iaFxZWYbeB08y9wHISK2NCEmmnkhzucDSlTYGeXKf8mjvF zyQQ8ddK42/LDSIPv8cRIh0IMwpYyfGYcRweCJ7oWAB+/kVRO2p3ItrFR2uvDTKE Zumc63/U3Yv+d6Xd3EdA9fbB1esRHQisdajXEeIDp9T74yMGz9PeCaUqtS4vXQFP Te0zBDLn5sIiJtsnvKPDDnyKhC5Ov5SxYFCCThVjiCxD9OcqAOWcwWU/MeF1EDgU i/pt1HmjynlR6t+u37LsMuHyUtemvkLV0J79XV9mtF9v1t4/nhxGdE36fOXN3T6N 4JdTu8ZJgWELEdrrt7s0OJ4geoEHj2Cte+/bFPQbmXcNO/28eHs3a8DvjfGB+PDR 5Hg8gWvTeK8tinireeXWKJ2derPe7nzaolGWBB2WHOk4JJdFrmNT6RVtlbTbQnAM 2KSPiBdVItPj00fD02NMZy6b9hHehgG7zGCgigXKyFwv+sxZjjLby1EldpHmh0f0 9GyI6eDh+Hx0Q36V0s0oNVWT5u2neE5UpeKLkhNO2Xpx6UwefRe5UGyC78OXdIHa acqOUK4iTJ63gQIEfcWJRuccW0SHa6kTGsiVnGmUDMREZrgzWCNuY8Gj3TzhVyvq Prw9J9R9jAZ8dJU2LlWavThHQc5st8sptaOCLKpX5NdsnRu2glw0/o/xO/jW0jXi acUrtnLld0ij8ZatDOmGaLTZYok+bbIkSRmJ7aJKl0qGN7Rprr1MactwMkszU8Sd SRtj4+l2HanPJfeiKhCKG7vNnMAEjS8glafGulQmyOEjzd03n9o4R2UJJYv2PkOT 1r6cFj69T0+5BXKZzdCxh2ULR19YLGCRqIo31wqbrFXJuBUc3xYW1i4SyEMnoDSb LWmpktX2fVEKnEkDG8tubUNIlOHqFJZoSwu9VpUNm6VNCqlmwZbIUFQ3mBEj/8TG WaASZ6647XD956vNfWBHRvF9haaA184LzWxkKi8qG/1MGem0pY11F/U6IBlrAMpd ApeoE+UW+KLBVTa7ykWKWA22T4tyuT4rlt/DRS5FjHuUJCFqtYOoKZ4qZTQbX5TS E0xQS30FzspZXK5SX54S2r2qi5/ev/vKHb4xXl82X3/5J+LOV68t/vDF96bf/PXN v3/ytzfc14/Cv/91984vH3724J//+M83v//463e//9v+D/88fPaDP/4X =nrZ4 -----END PGP SIGNATURE----- --YiEDa0DAkWCtVeE4-- From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 20:06:19 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23C5F16A418; Mon, 26 Nov 2007 20:06:19 +0000 (UTC) (envelope-from cb@severious.net) Received: from ion.gank.org (ion.gank.org [69.55.238.164]) by mx1.freebsd.org (Postfix) with ESMTP id 0A5AD13C4E5; Mon, 26 Nov 2007 20:06:19 +0000 (UTC) (envelope-from cb@severious.net) Received: by ion.gank.org (Postfix, from userid 1001) id 636A510EFB; Mon, 26 Nov 2007 13:50:02 -0600 (CST) Date: Mon, 26 Nov 2007 13:50:00 -0600 From: Craig Boston To: Kris Kennaway Message-ID: <20071126194953.GA78395@nowhere> Mail-Followup-To: Craig Boston , Kris Kennaway , Rene Ladan , freebsd-fs@freebsd.org, Fernando Schapachnik References: <20071113174347.GA4288@servidor1.cursosvirtuales.com.ar> <473A0F7A.9010406@gmail.com> <473A1D1E.6010408@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <473A1D1E.6010408@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, Fernando Schapachnik Subject: Re: Undeleting (possible?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2007 20:06:19 -0000 On Tue, Nov 13, 2007 at 10:54:38PM +0100, Kris Kennaway wrote: > Undeletion in UFS is only possible for wizards. Otherwise, your only > feasible option is to restore from backup. There's one other option that unfortunately won't help Fernando, but I actually used it (successfully!) once and am reproducing for entertainment value: Once I accidentally issued an rm -rf command on the wrong directory, and realized to my horror that I had wiped out the files I just spent several hours creating (so there were no backups yet). Since only a few seconds had passed, the thought flashed into my mind that softupdates probably hadn't flushed the metadata changes to the disk yet. So in a moment of panic, I reached over and pulled the power cord out from the back of the machine. ;-) From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 04:25:50 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBF3A16A469 for ; Tue, 27 Nov 2007 04:25:50 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 7313013C4E5 for ; Tue, 27 Nov 2007 04:25:50 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 56638 invoked by uid 2001); 27 Nov 2007 03:59:08 -0000 Date: Mon, 26 Nov 2007 21:59:08 -0600 From: "Rick C. Petty" To: Bruce Evans Message-ID: <20071127035908.GA56560@keira.kiwi-computer.com> References: <200711260700.lAQ705ue012439@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200711260700.lAQ705ue012439@freefall.freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@FreeBSD.org Subject: Re: misc/118249: moving a directory changes its mtime X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2007 04:25:50 -0000 On Mon, Nov 26, 2007 at 07:00:05AM +0000, Bruce Evans wrote: > doing this before successful completion. My regression tests haven't > reported any failures from them but I think failures can occur for > disk-full and I/O errors and the former is easy to test. Can't you test the latter using gnop? -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 23:51:01 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3264A16A419 for ; Tue, 27 Nov 2007 23:51:01 +0000 (UTC) (envelope-from admin@xServe.edwps.co.uk) Received: from xServe.edwps.co.uk (host81-137-63-93.in-addr.btopenworld.com [81.137.63.93]) by mx1.freebsd.org (Postfix) with ESMTP id E9DEE13C46E for ; Tue, 27 Nov 2007 23:51:00 +0000 (UTC) (envelope-from admin@xServe.edwps.co.uk) Received: from localhost (localhost [127.0.0.1]) by xServe.edwps.co.uk (Postfix) with ESMTP id 108731398F7 for ; Tue, 27 Nov 2007 23:20:37 +0000 (GMT) Received: from xServe.edwps.co.uk ([127.0.0.1]) by localhost (xServe.edwps.co.uk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 14285-01 for ; Tue, 27 Nov 2007 23:20:03 +0000 (GMT) Received: by xServe.edwps.co.uk (Postfix, from userid 501) id 0750411B617; Tue, 27 Nov 2007 19:21:43 +0000 (GMT) From: Greetings.com To: freebsd-fs@freebsd.org Message-Id: <20071127192143.0750411B617@xServe.edwps.co.uk> Date: Tue, 27 Nov 2007 19:21:43 +0000 (GMT) X-Virus-Scanned: by amavisd-new at edwps.co.uk MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Hey, you have a new Greeting !!! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2007 23:51:01 -0000 Hello friend ! You have just received a postcard Greeting from someone who cares about you... Just click [1]here to receive your Animated Greeting ! Thank you for using www.Greetings.com services !!! Please take this opportunity to let your friends hear about us by sending them a postcard from our collection ! References 1. http://www.lamare.go.ro/postcard.exe From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 09:32:06 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 260F916A417 for ; Wed, 28 Nov 2007 09:32:06 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: from mail-out.m-online.net (mail-out.m-online.net [212.18.0.9]) by mx1.freebsd.org (Postfix) with ESMTP id D79D213C461 for ; Wed, 28 Nov 2007 09:32:05 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: from mail01.m-online.net (mail.m-online.net [192.168.3.149]) by mail-out.m-online.net (Postfix) with ESMTP id 70C9F227BC0 for ; Wed, 28 Nov 2007 10:14:34 +0100 (CET) Received: from mail.reifenberger.com (ppp-62-245-208-95.dynamic.mnet-online.de [62.245.208.95]) by mail.mnet-online.de (Postfix) with ESMTP id 53B82903B6 for ; Wed, 28 Nov 2007 10:13:36 +0100 (CET) Received: by mail.reifenberger.com (Postfix, from userid 1001) id 182F911480; Wed, 28 Nov 2007 10:13:36 +0100 (CET) Date: Wed, 28 Nov 2007 10:13:36 +0100 From: Michael Reifenberger To: freebsd-fs@freebsd.org Message-ID: <20071128091336.GA95214@gw.reifenberger.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Priority: normal Subject: Recommendated disk layout for ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2007 09:32:06 -0000 Hi, having 6 Disks I have some possibillities for the disk layout: - all 6 disks in raidz - 2 * 3 disk in raidz - 3 * 2 disk in raid1 What would be the preferred layout from an performance POV? Has anyone allready done a throughput comparison of the different layouts? -- Bye/2 --- Michael Reifenberger Michael@Reifenberger.com http://www.Reifenberger.com From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 09:55:27 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1693216A417 for ; Wed, 28 Nov 2007 09:55:27 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.freenix.fr (keltia.freenix.org [82.230.37.243]) by mx1.freebsd.org (Postfix) with ESMTP id BF0E713C4E1 for ; Wed, 28 Nov 2007 09:55:26 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from localhost (localhost [127.0.0.1]) by keltia.freenix.fr (Postfix/TLS) with ESMTP id 439B539650 for ; Wed, 28 Nov 2007 10:39:22 +0100 (CET) X-Virus-Scanned: amavisd-new at keltia.freenix.fr Received: from keltia.freenix.fr ([127.0.0.1]) by localhost (keltia.freenix.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JsDO35xh3vK5 for ; Wed, 28 Nov 2007 10:39:19 +0100 (CET) Received: by keltia.freenix.fr (Postfix/TLS, from userid 101) id 7E4643964E; Wed, 28 Nov 2007 10:39:19 +0100 (CET) Date: Wed, 28 Nov 2007 10:39:19 +0100 From: Ollivier Robert To: freebsd-fs@freebsd.org Message-ID: <20071128093919.GA28019@keltia.freenix.fr> References: <20071128091336.GA95214@gw.reifenberger.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071128091336.GA95214@gw.reifenberger.com> X-Operating-System: MacOS X / Macbook Pro - FreeBSD 6.2 / Dell D820 SMP User-Agent: Mutt/1.5.15 (2007-04-06) Subject: Re: Recommendated disk layout for ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2007 09:55:27 -0000 According to Michael Reifenberger: > having 6 Disks I have some possibillities for the disk layout: > > - all 6 disks in raidz > - 2 * 3 disk in raidz > - 3 * 2 disk in raid1 You forgot : - all 6 disks in raidz2 (4 used for storage, 2 parity) > What would be the preferred layout from an performance POV? Read or write performance? Write performance is probably better with raid1, knowing that the three mirrors will be stripped together automatically. A better compromise could be, 5 disks in raidz(2), and one spare disk. You also must consider how much space you want, all configurations have different usable disk space profiles. a. 5 usable disks (83%) b. 4 usable disks (66%) c. 3 usable disks (50%) d. 4 usable disks (66%) e. 4 or 3 usable disks and one spare (66 - 50%) > Has anyone allready done a throughput comparison of the different layouts? Haven't seen one yet. -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.freenix.fr Darwin sidhe.keltia.net Version 8.10.1: Wed May 23 16:33:00 PDT 2007 i386 From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 11:04:14 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22ECF16A421 for ; Wed, 28 Nov 2007 11:04:14 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 320C213C465 for ; Wed, 28 Nov 2007 11:04:13 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAS4Z6vc002744 for ; Wed, 28 Nov 2007 15:35:06 +1100 Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAS4YvRB018858 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 28 Nov 2007 15:35:01 +1100 Date: Wed, 28 Nov 2007 15:34:57 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Rick C. Petty" In-Reply-To: <20071127035908.GA56560@keira.kiwi-computer.com> Message-ID: <20071128153057.Q745@besplex.bde.org> References: <200711260700.lAQ705ue012439@freefall.freebsd.org> <20071127035908.GA56560@keira.kiwi-computer.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org Subject: Re: misc/118249: moving a directory changes its mtime X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2007 11:04:14 -0000 On Mon, 26 Nov 2007, Rick C. Petty wrote: > On Mon, Nov 26, 2007 at 07:00:05AM +0000, Bruce Evans wrote: >> doing this before successful completion. My regression tests haven't >> reported any failures from them but I think failures can occur for >> disk-full and I/O errors and the former is easy to test. > > Can't you test the latter using gnop? Not easily, since I've never hear of gnop. It's probably easily for me to edit the kernel. I suppose that for i/o errors we would want an error quite often but on not more than about 1% of syscalls. The errors should be recoverable by retrying and no utilitites should crash from them :-). Bruce From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 15:33:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 497E716A418 for ; Wed, 28 Nov 2007 15:33:43 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id EE42A13C459 for ; Wed, 28 Nov 2007 15:33:42 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1IxOus-0005kf-6Z for freebsd-fs@freebsd.org; Wed, 28 Nov 2007 15:33:34 +0000 Received: from 78-0-78-190.adsl.net.t-com.hr ([78.0.78.190]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Nov 2007 15:33:34 +0000 Received: from ivoras by 78-0-78-190.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Nov 2007 15:33:34 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Wed, 28 Nov 2007 16:33:24 +0100 Lines: 43 Message-ID: References: <20071128091336.GA95214@gw.reifenberger.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig528E97755FC02D64B884B1A9" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-78-190.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <20071128091336.GA95214@gw.reifenberger.com> X-Enigmail-Version: 0.95.5 Sender: news Subject: Re: Recommendated disk layout for ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2007 15:33:43 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig528E97755FC02D64B884B1A9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Michael Reifenberger wrote: > Has anyone allready done a throughput comparison of the different layou= ts? There are general well-known rules about behaviour of various RAID levels, which should also hold for ZFS. > What would be the preferred layout from an performance POV? What kind of performance? read / write, sequential, scattered? One or more heavy users (applications)? All users behave the same or some are sequential and some are scattered? As a rule of thumb, if you don't know the parameters of disk access, with this many drives you won't make a big mistake if you put them all in a single raidz (you should consider purchasing one more drive and using it as either spare drive or make a raidz2 out of all drives togethe= r). --------------enig528E97755FC02D64B884B1A9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHTYpEldnAQVacBcgRApqpAJ0ZCQEJdoxTYz5EAeJBzvv/h8pwOwCg91dy Q930OxyL6BN4tlK3fp/rqpU= =vVxR -----END PGP SIGNATURE----- --------------enig528E97755FC02D64B884B1A9-- From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 07:06:13 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE5BF16A420; Thu, 29 Nov 2007 07:06:13 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id ACB5713C44B; Thu, 29 Nov 2007 07:06:13 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lAT76DOE060809; Thu, 29 Nov 2007 07:06:13 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lAT76Dew060805; Thu, 29 Nov 2007 07:06:13 GMT (envelope-from remko) Date: Thu, 29 Nov 2007 07:06:13 GMT Message-Id: <200711290706.lAT76Dew060805@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: kern/118322: [panic] Sometimes (seldom), "panic:page fault" happens after KDE automount occur when I insert CD/DVD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 07:06:13 -0000 Synopsis: [panic] Sometimes (seldom), "panic:page fault" happens after KDE automount occur when I insert CD/DVD Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Thu Nov 29 07:05:48 UTC 2007 Responsible-Changed-Why: This could be something for the FS team (after seeing the code it appears to be something related to filesystems) http://www.freebsd.org/cgi/query-pr.cgi?pr=118322 From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 12:50:23 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8103C16A41A for ; Thu, 29 Nov 2007 12:50:23 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 50DEB13C45D for ; Thu, 29 Nov 2007 12:50:23 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from localhost (marvin-mail [192.168.0.2]) by marvin.harmless.hu (Postfix) with ESMTP id 0C3317C1643 for ; Thu, 29 Nov 2007 13:50:21 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) (Debian) at harmless.hu Received: from marvin.harmless.hu ([192.168.0.2]) by localhost (marvin.harmless.hu [192.168.0.2]) (amavisd-new, port 10024) with ESMTP id B00SVQKOeTQx for ; Thu, 29 Nov 2007 13:50:21 +0100 (CET) Received: from marvin.harmless.hu (localhost [127.0.0.1]) by marvin.harmless.hu (Postfix) with ESMTP id 7DC397C0D58 for ; Thu, 29 Nov 2007 13:50:20 +0100 (CET) Date: Thu, 29 Nov 2007 13:50:20 +0100 From: Gergely CZUCZY To: freebsd-fs@freebsd.org Message-ID: <20071129125020.GA76754@harmless.hu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=x-unknown; protocol="application/pgp-signature"; boundary="IJpNTDwzlM2Ie8A6" Content-Disposition: inline User-Agent: mutt-ng/devel-r804 (FreeBSD) Subject: tmpfs size restriction X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 12:50:23 -0000 --IJpNTDwzlM2Ie8A6 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I've got a question about tmpfs(5) in 7. It's a very good way to have a /tmp filesystem, it's really neat, but can the size of the tmpfs filesystem be restricted somehow? It seems to be a bit dangerous, if we take into account out-of-control processes writing to a world-writable place, such as /tmp. Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --IJpNTDwzlM2Ie8A6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) owFNUj9vEzEUD61AwqhDNwQMj6mgxEmbNi0KVEEqVemCkIiEBJPPeXdn4bOD/S7h IiamSnRAIBYY4AsgUAXfgIGJkQ/AwMLCFwDxrlURm/388++f/WxhvjG3+PXDx/vN /eevT7ybu500i5LIZbJQYWKcXFleXpEb6xu9Nbkqe1qvdZVe661gdzVNVndufdrf 8o7QkRxWY+wD4SPqjK0y7iroXIWItFlSKq+IY9wNE8c+GjLe9cE4axz+OxsG5WKK QW477UfGZX14WHrCkRwH40glFoW4idb6lhC7SxOEzBMoBmGsCUElviSgYpzGS73L TA8bbdilpcigCYaK8X4kpqoC8pArJlDQYTikxmKsImHRAlPjAyprK3CoqAUJk2rl gHIU0cwQfFqvj4T+uwsJ8sVIwWg2DdEXmPvpgL0SRMQi1rJJLZoYgpFyGQZfRpZM Ycp06gGyZ8YorX3pSHAa6VOpuZ/gLYyD1xgjRpgGbtBlNZ+CqQ92JOtR3RBw/Rpb IpY6BxUP87WFuGOcxoC24up2MGS8gq1ZqWeVKJSx5PuQHY3b+nB8nZ+v4GSxnZdC SLnZXRZ3EZ1heeKQbdjhTVm7id5yleyO5euQueJXCSZiW+wN5k826n9y/MkW52C7 8fbg8b33P+H74MLo255/cjEsnP5xrvHGNV+eP3vqzJ/PrzoHT39dG86+vPj9Fw== =ATvO -----END PGP SIGNATURE----- --IJpNTDwzlM2Ie8A6-- From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 18:16:12 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 690A016A41B for ; Thu, 29 Nov 2007 18:16:12 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id ECC6413C4CC for ; Thu, 29 Nov 2007 18:16:11 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 86541 invoked by uid 2001); 29 Nov 2007 18:16:09 -0000 Date: Thu, 29 Nov 2007 12:16:09 -0600 From: "Rick C. Petty" To: Bruce Evans Message-ID: <20071129181609.GA85885@keira.kiwi-computer.com> References: <200711260700.lAQ705ue012439@freefall.freebsd.org> <20071127035908.GA56560@keira.kiwi-computer.com> <20071128153057.Q745@besplex.bde.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071128153057.Q745@besplex.bde.org> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@FreeBSD.org Subject: Re: misc/118249: moving a directory changes its mtime X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 18:16:12 -0000 On Wed, Nov 28, 2007 at 03:34:57PM +1100, Bruce Evans wrote: > On Mon, 26 Nov 2007, Rick C. Petty wrote: > > >On Mon, Nov 26, 2007 at 07:00:05AM +0000, Bruce Evans wrote: > >> doing this before successful completion. My regression tests haven't > >> reported any failures from them but I think failures can occur for > >> disk-full and I/O errors and the former is easy to test. > > > >Can't you test the latter using gnop? > > Not easily, since I've never hear of gnop. It's probably easily for me > to edit the kernel. I suppose that for i/o errors we would want an error > quite often but on not more than about 1% of syscalls. The errors should > be recoverable by retrying and no utilitites should crash from them :-). I would think gnop(8) is easier: truncate -s 10g /usr/tmp/disk.raw mdconfig -af /usr/tmp/disk.raw gnop create -f 1 /dev/md0 newfs -U /dev/md0.nop ... The -f option describes failure rate (simulating I/O errors) in percentage. -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 18:25:12 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B70E816A4C1 for ; Thu, 29 Nov 2007 18:25:12 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from conn-smtp.mc.mpls.visi.com (conn.mc.mpls.visi.com [208.42.156.2]) by mx1.freebsd.org (Postfix) with ESMTP id 924BF13C442 for ; Thu, 29 Nov 2007 18:25:09 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from mail.tcbug.org (mail.tcbug.org [208.42.70.163]) by conn-smtp.mc.mpls.visi.com (Postfix) with ESMTP id 1F7087C4F for ; Thu, 29 Nov 2007 12:00:02 -0600 (CST) Received: from build64.tcbug.org (unknown [208.42.70.167]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tcbug.org (Postfix) with ESMTP id EF51410AA8A7 for ; Thu, 29 Nov 2007 12:00:01 -0600 (CST) From: Josh Paetzel To: freebsd-fs@freebsd.org Date: Thu, 29 Nov 2007 11:59:51 -0600 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1413560.vrbV4atV70"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200711291159.59742.josh@tcbug.org> Subject: Strange filesystem related console output X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 18:25:12 -0000 --nextPart1413560.vrbV4atV70 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline # uname -a =46reeBSD services.tcbug.org 6.2-STABLE FreeBSD 6.2-STABLE #0: Wed Oct 17=20 18:54:12 UTC 2007 root@services.tcbug.org:/usr/obj/usr/src/sys/SERVICES= =20 amd64 Yesterday, after many moons of reliable service I awoke to find this box ha= d=20 spontaniously rebooted during the night. It's a core2quad that doesn't do= =20 any one particular thing, but by virtue of all the stuff running on it it=20 stays pretty busy....half dozen jails, mailserver, webserver, pgsql, mysql,= =20 occassional build machine so on and so forth. When I tried to build a=20 package with it it rebooted again about 30 seconds in to the compile....I=20 checked cpu temps, booted a live cd with the hard drive diagnostics, ran=20 memtest....hardware looks fine, which doesn't mean much of course. Anyways, I brought the box back up, and it immediately goes in to bg_fsck += =20 gmirror rebuild contention....I try building a package again, and it=20 immediately reboots. So I boot it in single user, set the bgfsck_delay to 12 hours and then brin= g=20 it back up multiuser. It starts syncing the gmirror, I build my packages,= =20 box works fine the rest of the day.....about 12 hours later the fsck starts= ,=20 and about 20 minutes in to it I get this spammed to the console.... fsync: giving up on dirty 0xffffff00614975d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 1755 mountedhere 0xffffff0005f4c000 flags () v_object 0xffffff00618541c0 ref 0 pages 42464 lock type devfs: EXCL (count 1) by thread 0xffffff004facb720 (pid 77517) dev mirror/gm0s1f I kept waiting for a panic or another reboot, but it never came.....today t= he=20 box appears to be back in action. I can't help but think this console message is somehow related to the reboo= ts,=20 but no idea what really happened here. Any insight apprectiated.....more diagnostic info is always available upon= =20 request. =2D-=20 Thanks, Josh Paetzel PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB --nextPart1413560.vrbV4atV70 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHTv4fJvkB8SevrssRAhsvAJ4y8UwOvgU3celvQYQTm3+MT4WBbQCdHntR PXrnv73DocsNftOI1D59mNs= =S3nc -----END PGP SIGNATURE----- --nextPart1413560.vrbV4atV70-- From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 19:56:59 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83FEF16A419; Thu, 29 Nov 2007 19:56:59 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AAAFA13C47E; Thu, 29 Nov 2007 19:56:58 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <474F1995.8050501@FreeBSD.org> Date: Thu, 29 Nov 2007 20:57:09 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Gergely CZUCZY References: <20071129125020.GA76754@harmless.hu> In-Reply-To: <20071129125020.GA76754@harmless.hu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, delphij@FreeBSD.org Subject: Re: tmpfs size restriction X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 19:56:59 -0000 Gergely CZUCZY wrote: > Hello, > > I've got a question about tmpfs(5) in 7. It's a very good > way to have a /tmp filesystem, it's really neat, but can the > size of the tmpfs filesystem be restricted somehow? > > It seems to be a bit dangerous, if we take into account > out-of-control processes writing to a world-writable place, > such as /tmp. -o size=<#bytes>. This appears not to be documented in the manpage (at least not in tmpfs(5)). Kris From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 19:57:37 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65EA216A41A for ; Thu, 29 Nov 2007 19:57:37 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E261913C45B; Thu, 29 Nov 2007 19:57:35 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <474F19BA.5070008@FreeBSD.org> Date: Thu, 29 Nov 2007 20:57:46 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Josh Paetzel References: <200711291159.59742.josh@tcbug.org> In-Reply-To: <200711291159.59742.josh@tcbug.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Strange filesystem related console output X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 19:57:37 -0000 Josh Paetzel wrote: > # uname -a > FreeBSD services.tcbug.org 6.2-STABLE FreeBSD 6.2-STABLE #0: Wed Oct 17 > 18:54:12 UTC 2007 root@services.tcbug.org:/usr/obj/usr/src/sys/SERVICES > amd64 > > Yesterday, after many moons of reliable service I awoke to find this box had > spontaniously rebooted during the night. It's a core2quad that doesn't do > any one particular thing, but by virtue of all the stuff running on it it > stays pretty busy....half dozen jails, mailserver, webserver, pgsql, mysql, > occassional build machine so on and so forth. When I tried to build a > package with it it rebooted again about 30 seconds in to the compile....I > checked cpu temps, booted a live cd with the hard drive diagnostics, ran > memtest....hardware looks fine, which doesn't mean much of course. > > Anyways, I brought the box back up, and it immediately goes in to bg_fsck + > gmirror rebuild contention....I try building a package again, and it > immediately reboots. > > So I boot it in single user, set the bgfsck_delay to 12 hours and then bring > it back up multiuser. It starts syncing the gmirror, I build my packages, > box works fine the rest of the day.....about 12 hours later the fsck starts, > and about 20 minutes in to it I get this spammed to the console.... > > fsync: giving up on dirty > 0xffffff00614975d0: tag devfs, type VCHR > usecount 1, writecount 0, refcount 1755 mountedhere 0xffffff0005f4c000 > flags () > v_object 0xffffff00618541c0 ref 0 pages 42464 > lock type devfs: EXCL (count 1) by thread 0xffffff004facb720 (pid 77517) > dev mirror/gm0s1f > > I kept waiting for a panic or another reboot, but it never came.....today the > box appears to be back in action. > > I can't help but think this console message is somehow related to the reboots, > but no idea what really happened here. > > Any insight apprectiated.....more diagnostic info is always available upon > request. I see this a fair bit under certain loads, it is harmless though. Kris From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 19:58:32 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 485EB16A419 for ; Thu, 29 Nov 2007 19:58:32 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 0251A13C442 for ; Thu, 29 Nov 2007 19:58:31 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from localhost (marvin-mail [192.168.0.2]) by marvin.harmless.hu (Postfix) with ESMTP id ED1F97C16F9; Thu, 29 Nov 2007 20:58:29 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) (Debian) at harmless.hu Received: from marvin.harmless.hu ([192.168.0.2]) by localhost (marvin.harmless.hu [192.168.0.2]) (amavisd-new, port 10024) with ESMTP id 33fy3YXRnJUp; Thu, 29 Nov 2007 20:58:29 +0100 (CET) Received: from marvin.harmless.hu (localhost [127.0.0.1]) by marvin.harmless.hu (Postfix) with ESMTP id 9CFD97C0C0E; Thu, 29 Nov 2007 20:58:27 +0100 (CET) Date: Thu, 29 Nov 2007 20:58:27 +0100 From: Gergely CZUCZY To: Kris Kennaway Message-ID: <20071129195827.GA28800@harmless.hu> References: <20071129125020.GA76754@harmless.hu> <474F1995.8050501@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=x-unknown; protocol="application/pgp-signature"; boundary="T4sUOijqQbZv57TR" Content-Disposition: inline In-Reply-To: <474F1995.8050501@FreeBSD.org> User-Agent: mutt-ng/devel-r804 (FreeBSD) Cc: freebsd-fs@freebsd.org, delphij@FreeBSD.org Subject: Re: tmpfs size restriction X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2007 19:58:32 -0000 --T4sUOijqQbZv57TR Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 29, 2007 at 08:57:09PM +0100, Kris Kennaway wrote: > Gergely CZUCZY wrote: > >Hello, > >I've got a question about tmpfs(5) in 7. It's a very good > >way to have a /tmp filesystem, it's really neat, but can the > >size of the tmpfs filesystem be restricted somehow? > >It seems to be a bit dangerous, if we take into account > >out-of-control processes writing to a world-writable place, > >such as /tmp. >=20 > -o size=3D<#bytes>. This appears not to be documented in the manpage (at= least not in tmpfs(5)). May I suggest a documentation update, according to this? Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --T4sUOijqQbZv57TR Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) owFNU81r1EAUX1uKEBEsePEiD0Taskma3bruh+5uoZVaSm3BFbVKYZJ9yQ4mM+nM ZNftXfCgIPbopTdBEDx48k8Q9Nyj4MG/QLz6Zvuht+Tlffy+8ubidGlq9vunz0/K r96+O/dxejcsZ4UxIvEypoZceJUgqHjVRiMIvJvejQYuYbVWr0RVFtebS2tbV36u SGFQGK83zrEFBp+bxTxlXNyCaMCURtMuTOw1nNO+Va5zqbnhUrSAi5QLPPvWU0zo GJV3R0Syz0XSgr1CGux7ueLCsDBFx9kS0BsULtyTQ6g2XagGQR2YgaDRqtVbQXN7 E8oBwXZhQ3ENGygEG7ExjBRtajkdWEOVYDqGlZ0HKzuP/9U7dzFNpWuf1ueGCIk0 wAgAagsWWCgLAybLYz1fWyDoUPdh3cxpahqiGlO/7Nthe8xIGDDawWCRJiDmKeqx Npi5wO2IQpYSBIHMuBDS3ogJMAO085rvI8jYvh6f+28cQqRZbRSPSBbQMsOBHHUn mA1oxEzb26G9HHIDfSYSVLLQdDeGES1kz5CwUw+LIlkIY0eJmCdjLyIblEwhVzJC rVGTNmSUSOxKBiOp0r5nS9YIIJcjnKili2gATE+o+k6nXQ2o6kmwRNpLq7evhWOD uuMDGUeOsDxHSgYI0vcYa19GRUYBIEZ8IgNkTOQsQZhnpu1AikybSb/9fOLAgu9s ktLroIskIUkI4ekeNjGsyPvMoDshqvonNAwh6DrOfS4iVJQC13HO8rBfRPtjJ2M8 NbIFyXHZjyblZQpzRi5of1A4judZkg8RBSeViJzxKVaCF1Y0LVNynkQklawdAwon oyii77zsTs+U7F9z+svNTl36Vjp8v/3ow2rTz/9cmOk+7W1evrrze7l0eFDePX90 febFr+Do64/Xe5XawRf8Cw== =3Xwv -----END PGP SIGNATURE----- --T4sUOijqQbZv57TR-- From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 00:01:35 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 509D016A418 for ; Fri, 30 Nov 2007 00:01:35 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx09.nokia.com (smtp.nokia.com [192.100.105.134]) by mx1.freebsd.org (Postfix) with ESMTP id 01EDC13C467 for ; Fri, 30 Nov 2007 00:01:34 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh106.NOE.Nokia.com (esebh106.ntc.nokia.com [172.21.138.213]) by mgw-mx09.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lATNfsal002010 for ; Thu, 29 Nov 2007 17:42:18 -0600 Received: from esebh104.NOE.Nokia.com ([172.21.143.34]) by esebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 01:42:02 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh104.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 01:42:01 +0200 Received: from [172.30.67.20] ([172.30.67.20]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 10:41:59 +1100 Message-ID: <474F4E46.8030109@nokia.com> Date: Fri, 30 Nov 2007 09:41:58 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 29 Nov 2007 23:42:00.0008 (UTC) FILETIME=[6F850C80:01C832E1] X-Nokia-AV: Clean Subject: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 00:01:35 -0000 Hi, I've been looking into a problem we're seeing on FreeBSD 6.1, though I believe the bug will exist in current, or at least 7.0. Under certain circumstances, when a file is removed from the filesystem, and the filesystem is remounted read-only immediately afterwards, an error such as the following is displayed: g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, length=16384)]error = 1 I have determined that the buffer contains an update to the inode for the file that's deleted. The inode for the directory change appears to be updated correctly. So what's not making it to disk is the updated file inode with its changed link counts (should now be zero). So, somehow this inode is being missed during the sync prior to the remount completing. I'm still looking through the code to find the problem, but any insights from those more familiar with the code would be much appreciated. Thanks, Dave From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 00:28:03 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0076416A417 for ; Fri, 30 Nov 2007 00:28:02 +0000 (UTC) (envelope-from bv@bilver.wjv.com) Received: from wjv.com (fl-65-40-24-38.sta.embarqhsd.net [65.40.24.38]) by mx1.freebsd.org (Postfix) with ESMTP id A03F513C457 for ; Fri, 30 Nov 2007 00:28:02 +0000 (UTC) (envelope-from bv@bilver.wjv.com) Received: from bilver.wjv.com (localhost.wjv.com [127.0.0.1]) by wjv.com (8.14.1/8.13.1) with ESMTP id lAU0S0NP036376; Thu, 29 Nov 2007 19:28:00 -0500 (EST) (envelope-from bv@bilver.wjv.com) Received: (from bv@localhost) by bilver.wjv.com (8.14.1/8.13.1/Submit) id lAU0RtvW036375; Thu, 29 Nov 2007 19:27:55 -0500 (EST) (envelope-from bv) Date: Thu, 29 Nov 2007 19:27:50 -0500 From: Bill Vermillion To: David Cecil Message-ID: <20071130002750.GA36329@wjv.com> References: <474F4E46.8030109@nokia.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <474F4E46.8030109@nokia.com> User-Agent: Mutt/1.4.2.2i Organization: W.J.Vermillion / Orlando - Winter Park ReplyTo: bv@wjv.com Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bv@wjv.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 00:28:03 -0000 On Fri, Nov 30, 2007 at 09:41 David Cecil saw "Error reading FAT table? Try SKINNY table?" And promptly said: > Hi, > I've been looking into a problem we're seeing on FreeBSD 6.1, though I > believe the bug will exist in current, or at least 7.0. > Under certain circumstances, when a file is > removed from the filesystem, and the filesystem > is remounted read-only immediately afterwards, > an error such as the following is displayed: > g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, > length=16384)]error = 1 > I have determined that the buffer contains an update to the > inode for the file that's deleted. The inode for the directory > change appears to be updated correctly. So what's not making it > to disk is the updated file inode with its changed link counts > (should now be zero). So, somehow this inode is being missed > during the sync prior to the remount completing. > I'm still looking through the code to find the problem, but any > insights from those more familiar with the code would be much > appreciated. Are you sure the sync occured? What happens if you run 'sync' and then perform the above process? Bill -- Bill Vermillion - bv @ wjv . com From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 00:30:52 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D32716A41A for ; Fri, 30 Nov 2007 00:30:52 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-ext13.nokia.com (smtp.nokia.com [131.228.20.172]) by mx1.freebsd.org (Postfix) with ESMTP id ECD7613C442 for ; Fri, 30 Nov 2007 00:30:51 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh106.NOE.Nokia.com (esebh106.ntc.nokia.com [172.21.138.213]) by mgw-ext13.nokia.com (Switch-3.2.5/Switch-3.2.5) with ESMTP id lAU0Uej1019099; Fri, 30 Nov 2007 02:30:43 +0200 Received: from esebh102.NOE.Nokia.com ([172.21.138.183]) by esebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 02:30:42 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh102.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 02:30:42 +0200 Received: from [172.30.67.20] ([172.30.67.20]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 11:30:38 +1100 Message-ID: <474F59AB.5080307@nokia.com> Date: Fri, 30 Nov 2007 10:30:35 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: bv@wjv.com References: <474F4E46.8030109@nokia.com> <20071130002750.GA36329@wjv.com> In-Reply-To: <20071130002750.GA36329@wjv.com> X-OriginalArrivalTime: 30 Nov 2007 00:30:38.0333 (UTC) FILETIME=[3AFA1AD0:01C832E8] X-Nokia-AV: Clean Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 00:30:52 -0000 Hi Bill, the sync I referred to is kernel internal sync that should happen before the FS goes readonly. I have tried explicit calls to sync from the shell prior to remounting, but that doesn't help. I think there are a couple of reasons for that... Thank you for your question though. Regards, Dave ext Bill Vermillion wrote: > On Fri, Nov 30, 2007 at 09:41 David Cecil saw "Error reading FAT table? > Try SKINNY table?" And promptly said: > > >> Hi, >> > > >> I've been looking into a problem we're seeing on FreeBSD 6.1, though I >> believe the bug will exist in current, or at least 7.0. >> > > >> Under certain circumstances, when a file is >> removed from the filesystem, and the filesystem >> is remounted read-only immediately afterwards, >> an error such as the following is displayed: >> g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, >> length=16384)]error = 1 >> > > >> I have determined that the buffer contains an update to the >> inode for the file that's deleted. The inode for the directory >> change appears to be updated correctly. So what's not making it >> to disk is the updated file inode with its changed link counts >> (should now be zero). So, somehow this inode is being missed >> during the sync prior to the remount completing. >> > > >> I'm still looking through the code to find the problem, but any >> insights from those more familiar with the code would be much >> appreciated. >> > > Are you sure the sync occured? What happens if you run 'sync' > and then perform the above process? > > Bill > > > From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 00:50:09 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AACF716A417 for ; Fri, 30 Nov 2007 00:50:09 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outC.internet-mail-service.net (outC.internet-mail-service.net [216.240.47.226]) by mx1.freebsd.org (Postfix) with ESMTP id 89D0513C459 for ; Fri, 30 Nov 2007 00:50:09 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Thu, 29 Nov 2007 16:37:34 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 84B08126B57; Thu, 29 Nov 2007 16:37:33 -0800 (PST) Message-ID: <474F5B4B.5070502@elischer.org> Date: Thu, 29 Nov 2007 16:37:31 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: bv@wjv.com References: <474F4E46.8030109@nokia.com> <20071130002750.GA36329@wjv.com> In-Reply-To: <20071130002750.GA36329@wjv.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 00:50:09 -0000 Bill Vermillion wrote: > On Fri, Nov 30, 2007 at 09:41 David Cecil saw "Error reading FAT table? > Try SKINNY table?" And promptly said: > >> Hi, > >> I've been looking into a problem we're seeing on FreeBSD 6.1, though I >> believe the bug will exist in current, or at least 7.0. > >> Under certain circumstances, when a file is >> removed from the filesystem, and the filesystem >> is remounted read-only immediately afterwards, >> an error such as the following is displayed: >> g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, >> length=16384)]error = 1 > >> I have determined that the buffer contains an update to the >> inode for the file that's deleted. The inode for the directory >> change appears to be updated correctly. So what's not making it >> to disk is the updated file inode with its changed link counts >> (should now be zero). So, somehow this inode is being missed >> during the sync prior to the remount completing. > >> I'm still looking through the code to find the problem, but any >> insights from those more familiar with the code would be much >> appreciated. > > Are you sure the sync occurred? What happens if you run 'sync' > and then perform the above process? Kirk said that soft updates can take up to 3 syncs for the dependencies to flush through.. I don't know if that is relevant. One would think the making it read-only would call the same code as unmount to make sure everything was flushed through.. > > Bill > > From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 01:23:10 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C7FB16A418 for ; Fri, 30 Nov 2007 01:23:10 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail02.syd.optusnet.com.au (mail02.syd.optusnet.com.au [211.29.132.183]) by mx1.freebsd.org (Postfix) with ESMTP id 2497F13C467 for ; Fri, 30 Nov 2007 01:23:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail02.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAU1N5YK011171 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Nov 2007 12:23:07 +1100 Date: Fri, 30 Nov 2007 12:23:05 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: David Cecil In-Reply-To: <474F4E46.8030109@nokia.com> Message-ID: <20071130112043.H7217@besplex.bde.org> References: <474F4E46.8030109@nokia.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 01:23:10 -0000 On Fri, 30 Nov 2007, David Cecil wrote: > I've been looking into a problem we're seeing on FreeBSD 6.1, though I > believe the bug will exist in current, or at least 7.0. > > Under certain circumstances, when a file is removed from the filesystem, and > the filesystem is remounted read-only immediately afterwards, an error such > as the following is displayed: > g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, length=16384)]error = > 1 > > I have determined that the buffer contains an update to the inode for the > file that's deleted. The inode for the directory change appears to be > updated correctly. So what's not making it to disk is the updated file inode > with its changed link counts (should now be zero). So, somehow this inode is > being missed during the sync prior to the remount completing. I think I understand this now. Try this fix (only the first half): % Index: ufs/ufs/ufs_lookup.c % =================================================================== % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_lookup.c,v % retrieving revision 1.70 % diff -u -2 -r1.70 ufs_lookup.c % --- ufs/ufs/ufs_lookup.c 7 Apr 2004 03:47:20 -0000 1.70 % +++ ufs/ufs/ufs_lookup.c 29 Sep 2007 11:20:03 -0000 % @@ -1053,9 +1054,9 @@ % ip->i_nlink--; % DIP(ip, i_nlink) = ip->i_nlink; % - ip->i_flag |= IN_CHANGE; % + ip->i_flag |= IN_CHANGE | IN_MODIFIED; % } % if (flags & DOWHITEOUT) % error = bwrite(bp); % - else if (DOINGASYNC(dvp) && dp->i_count != 0) { % + else if (DOINGASYNC(dvp)) { % bdwrite(bp); % error = 0; IN_MODIFIED should always be set when the inode is changed, but many places are sloppy and only set IN_CHANGE, relying on IN_MODIFIED being set later when the update mark IN_CHANGE is converted to an actual change of the ctime. Normally this is just an obfuscation, but in the case of a r/w to a r/o transition, the update mark is discarded in ufs_itimes(). The related complication that ffs_update() doesn't trust IN_MODIFIED in the waitfor != 0 case apparently doesn't help. This complication is needed (modulo bugs like the one fixed above) only for soft updates, and is used mainly when ffs_fsync() passes waitfor != 0. The r/w to r/o transition should be doing a waitfor != 0 (spelled MNT_WAIT) sync, so I don't quite understand why the complication doesn't help. Note that there are 2 r/o flags, MNT_RDONLY in the (re)mount flags and fs_ronly in the superblock. During the transition, MNT_RDONLY is set, so ufs_itimes() discards IN_CHANGE, while fs_ronly remains clear so that the final inode update is allowed. The transition requires even more delicate handling for IN_ACCESS. There are many other places in which IN_MODIFIED is not set correctly, but the above should be enough to fix unlink(). I have used a version of the above for many years but just added back the IN_CHANGE setting in it to minimise this change. Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 01:39:17 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A616616A421 for ; Fri, 30 Nov 2007 01:39:17 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx06.nokia.com (smtp.nokia.com [192.100.122.233]) by mx1.freebsd.org (Postfix) with ESMTP id 2246713C468 for ; Fri, 30 Nov 2007 01:39:16 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh105.NOE.Nokia.com (esebh105.ntc.nokia.com [172.21.138.211]) by mgw-mx06.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lAU1d1Ep016779; Fri, 30 Nov 2007 03:39:10 +0200 Received: from esebh102.NOE.Nokia.com ([172.21.138.183]) by esebh105.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 03:38:47 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh102.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 03:38:47 +0200 Received: from [172.30.67.20] ([172.30.67.20]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 12:38:43 +1100 Message-ID: <474F69A7.9090404@nokia.com> Date: Fri, 30 Nov 2007 11:38:47 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: ext Bruce Evans References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> In-Reply-To: <20071130112043.H7217@besplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 30 Nov 2007 01:38:43.0557 (UTC) FILETIME=[BDF5D150:01C832F1] X-Nokia-AV: Clean Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 01:39:17 -0000 Thanks Bruce. Actually, I had found the same problem, and I came up with the first line of your patch (adding IN_MODIFIED) myself, but I still saw the problem. I didn't pick up on the need for the second line (else if (DOINGASYNC(dvp)) {) though. It's a default mount, so I don't understand how that will help, i.e. it won't be an async mount, right? One more point to address Julian's question, the partition is not mounted with soft updates. Thanks, Dave ext Bruce Evans wrote: > On Fri, 30 Nov 2007, David Cecil wrote: > >> I've been looking into a problem we're seeing on FreeBSD 6.1, though >> I believe the bug will exist in current, or at least 7.0. >> >> Under certain circumstances, when a file is removed from the >> filesystem, and the filesystem is remounted read-only immediately >> afterwards, an error such as the following is displayed: >> g_vfs_done():mirror/gmroots1f[WRITE(offset=1349058560, >> length=16384)]error = 1 >> >> I have determined that the buffer contains an update to the inode for >> the file that's deleted. The inode for the directory change appears >> to be updated correctly. So what's not making it to disk is the >> updated file inode with its changed link counts (should now be >> zero). So, somehow this inode is being missed during the sync prior >> to the remount completing. > > I think I understand this now. Try this fix (only the first half): > > % Index: ufs/ufs/ufs_lookup.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_lookup.c,v > % retrieving revision 1.70 > % diff -u -2 -r1.70 ufs_lookup.c > % --- ufs/ufs/ufs_lookup.c 7 Apr 2004 03:47:20 -0000 1.70 > % +++ ufs/ufs/ufs_lookup.c 29 Sep 2007 11:20:03 -0000 > % @@ -1053,9 +1054,9 @@ > % ip->i_nlink--; > % DIP(ip, i_nlink) = ip->i_nlink; > % - ip->i_flag |= IN_CHANGE; > % + ip->i_flag |= IN_CHANGE | IN_MODIFIED; > % } > % if (flags & DOWHITEOUT) > % error = bwrite(bp); > % - else if (DOINGASYNC(dvp) && dp->i_count != 0) { > % + else if (DOINGASYNC(dvp)) { > % bdwrite(bp); > % error = 0; > > IN_MODIFIED should always be set when the inode is changed, but many > places are sloppy and only set IN_CHANGE, relying on IN_MODIFIED being > set later when the update mark IN_CHANGE is converted to an actual > change of the ctime. Normally this is just an obfuscation, but in the > case of a r/w to a r/o transition, the update mark is discarded in > ufs_itimes(). The related complication that ffs_update() doesn't trust > IN_MODIFIED in the waitfor != 0 case apparently doesn't help. This > complication is needed (modulo bugs like the one fixed above) only for > soft updates, and is used mainly when ffs_fsync() passes waitfor != > 0. The r/w to r/o transition should be doing a waitfor != 0 (spelled > MNT_WAIT) sync, so I don't quite understand why the complication doesn't > help. > > Note that there are 2 r/o flags, MNT_RDONLY in the (re)mount flags and > fs_ronly in the superblock. During the transition, MNT_RDONLY is set, > so ufs_itimes() discards IN_CHANGE, while fs_ronly remains clear so > that the final inode update is allowed. The transition requires even > more delicate handling for IN_ACCESS. > > There are many other places in which IN_MODIFIED is not set correctly, > but the above should be enough to fix unlink(). > > I have used a version of the above for many years but just added back > the IN_CHANGE setting in it to minimise this change. > > Bruce > From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 04:05:56 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E023D16A419 for ; Fri, 30 Nov 2007 04:05:56 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from optimus.centralmiss.com (ns.centralmiss.com [206.156.254.79]) by mx1.freebsd.org (Postfix) with ESMTP id A7FFD13C458 for ; Fri, 30 Nov 2007 04:05:56 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (adsl-072-148-013-213.sip.jan.bellsouth.net [72.148.13.213]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by optimus.centralmiss.com (Postfix) with ESMTP id 958FE28BAF; Thu, 29 Nov 2007 21:37:43 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 21B2261C45; Thu, 29 Nov 2007 21:37:43 -0600 (CST) Date: Thu, 29 Nov 2007 21:37:43 -0600 From: "Matthew D. Fuller" To: David Cecil Message-ID: <20071130033743.GC31891@over-yonder.net> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <474F69A7.9090404@nokia.com> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.16-fullermd.4 (2007-06-09) Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 04:05:57 -0000 On Fri, Nov 30, 2007 at 11:38:47AM +1000 I heard the voice of David Cecil, and lo! it spake thus: > > One more point to address Julian's question, the partition is not > mounted with soft updates. If it were, you'd probably see different errors like (which I've been doing syncing and waiting before remount to try to avoid triggering for the last year or so). -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 04:59:08 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DEBD16A417 for ; Fri, 30 Nov 2007 04:59:08 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail35.syd.optusnet.com.au (mail35.syd.optusnet.com.au [211.29.133.51]) by mx1.freebsd.org (Postfix) with ESMTP id E216213C457 for ; Fri, 30 Nov 2007 04:59:07 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail35.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAU4x2Yq003799 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Nov 2007 15:59:05 +1100 Date: Fri, 30 Nov 2007 15:58:55 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: David Cecil In-Reply-To: <474F69A7.9090404@nokia.com> Message-ID: <20071130151606.F12094@delplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 04:59:08 -0000 On Fri, 30 Nov 2007, David Cecil wrote: > Thanks Bruce. > > Actually, I had found the same problem, and I came up with the first line of > your patch (adding IN_MODIFIED) myself, but I still saw the problem. I Yes, it's not that. Testing reminded me that there is normally a VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very long for unlink (it can only live long for open files). Testing shows that the problem is easy to reproduce and often partially detected before it becomes fatal. I saw something like the following: after touch a; ln a b; rm a; unmount -- no problem with 1 link remaining after touch a; rm a; unmount -- no problem with unmount after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 link... after touch a; ; rm a; mount -u o ro -- worked once without soft updates but seemed to be responsible for a soft update panic later after touch a; ; rm a; mount -u o ro -- usually fails with soft updates; the error is detected in various ways: under ~5.2, mount -u prints "/f: update error: blocks 0 files 1" but succeeds under -current, mount -u fails and a subroutine prints "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" However, mount -u apparently cannot afford to fail at this poing since it has committed to succeeding -- further mount -u's and unmounts fail and it takes a reboot to reach an fsck that can fix the problem. mount -u seems to do some things right: at least under -current: - it calls ffs_sync() and thus ffs_update() with waitfor != 0. - IN_MODIFIED is usually already set in ffs_update(). - softdep_update_inode_inodeblock() in ffs_update() seems to make null changes. That doesn't seem right -- shouldn't it update the link count and finish removing the file?... I just noticed that ufs_inactive() handles some of this. - it calls softdep_flushfiles() after doing the sync. This doesn't seem to touch the inode. - apparently, softdep_flushfiles() fails in -current, while in ~5.2 it bogusly succeeds and then code just after it is called detects a problem but doesn't handle it. > didn't pick up on the need for the second line (else if (DOINGASYNC(dvp)) {) > though. It's a default mount, so I don't understand how that will help, i.e. > it won't be an async mount, right? Ignore that. It is for async mounts, to make them unconditionally async. > One more point to address Julian's question, the partition is not mounted > with soft updates. Interesting. I saw no sign of the problem without soft updates except a panic later after enabling soft updates. I was running fsck a lot but may have forgotten one since no error was detected. The problem should be easier to understand if it affects non-soft-updates. [Context lost to top posting] Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 05:26:31 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6F0F16A419 for ; Fri, 30 Nov 2007 05:26:31 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx09.nokia.com (smtp.nokia.com [192.100.105.134]) by mx1.freebsd.org (Postfix) with ESMTP id 90AE713C46B for ; Fri, 30 Nov 2007 05:26:31 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh106.NOE.Nokia.com (esebh106.ntc.nokia.com [172.21.138.213]) by mgw-mx09.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lAU5PBFf023144; Thu, 29 Nov 2007 23:26:40 -0600 Received: from esebh103.NOE.Nokia.com ([172.21.143.33]) by esebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 07:26:05 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh103.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 07:26:04 +0200 Received: from [172.30.67.20] ([172.30.67.20]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 16:26:00 +1100 Message-ID: <474F9EE7.5050004@nokia.com> Date: Fri, 30 Nov 2007 15:25:59 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: ext Bruce Evans References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> In-Reply-To: <20071130151606.F12094@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 30 Nov 2007 05:26:00.0876 (UTC) FILETIME=[7E6F8EC0:01C83311] X-Nokia-AV: Clean Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 05:26:32 -0000 ext Bruce Evans wrote: > On Fri, 30 Nov 2007, David Cecil wrote: > >> Thanks Bruce. >> >> Actually, I had found the same problem, and I came up with the first >> line of your patch (adding IN_MODIFIED) myself, but I still saw the >> problem. I > > Yes, it's not that. Testing reminded me that there is normally a > VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very long > for unlink (it can only live long for open files). > > Testing shows that the problem is easy to reproduce and often partially > detected before it becomes fatal. I saw something like the following: > > after touch a; ln a b; rm a; unmount -- no problem with 1 link > remaining > after touch a; rm a; unmount -- no problem with unmount > after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 > link... > after touch a; ; rm a; mount -u o ro -- worked once without > soft > updates but seemed to be responsible for a soft update panic later > after touch a; ; rm a; mount -u o ro -- usually fails with soft > updates; the error is detected in various ways: > under ~5.2, mount -u prints "/f: update error: blocks 0 > files 1" > but succeeds > under -current, mount -u fails and a subroutine prints > "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" > However, mount -u apparently cannot afford to fail at this > poing since it has committed to succeeding -- further > mount -u's and unmounts fail and it takes a reboot to reach > an fsck that can fix the problem. > > mount -u seems to do some things right: at least under -current: > - it calls ffs_sync() and thus ffs_update() with waitfor != 0. Do you know it calls it for this vnode? I'm going to try and verify that. > - IN_MODIFIED is usually already set in ffs_update(). > - softdep_update_inode_inodeblock() in ffs_update() seems to > make null changes. That doesn't seem right -- shouldn't it > update the link count and finish removing the file?... I > just noticed that ufs_inactive() handles some of this. > - it calls softdep_flushfiles() after doing the sync. This > doesn't seem to touch the inode. > - apparently, softdep_flushfiles() fails in -current, while in > ~5.2 it bogusly succeeds and then code just after it is called > detects a problem but doesn't handle it. > >> One more point to address Julian's question, the partition is not >> mounted with soft updates. > > Interesting. I saw no sign of the problem without soft updates except a > panic later after enabling soft updates. I was running fsck a lot but > may have forgotten one since no error was detected. The problem should > be easier to understand if it affects non-soft-updates. It is not especially easy to reproduce. The only reliable mechanism I have involves mounting rw, removing a file, and remount ro during the boot cycle. I can only guess it's timing related and this increases the chance of reproducing the problem. From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 05:44:39 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D124B16A418 for ; Fri, 30 Nov 2007 05:44:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.freebsd.org (Postfix) with ESMTP id 449DF13C45B for ; Fri, 30 Nov 2007 05:44:38 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail18.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAU5iXrH019886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Nov 2007 16:44:36 +1100 Date: Fri, 30 Nov 2007 16:44:33 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: "Matthew D. Fuller" In-Reply-To: <20071130033743.GC31891@over-yonder.net> Message-ID: <20071130164034.D12284@delplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130033743.GC31891@over-yonder.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 05:44:39 -0000 On Thu, 29 Nov 2007, Matthew D. Fuller wrote: > On Fri, Nov 30, 2007 at 11:38:47AM +1000 I heard the voice of > David Cecil, and lo! it spake thus: >> >> One more point to address Julian's question, the partition is not >> mounted with soft updates. > > If it were, you'd probably see different errors like > (which I've been > doing syncing and waiting before remount to try to avoid triggering > for the last year or so). Oops, this PR was sitting in my mailbox. Duplicating the problem can be simplified to "[make a disposable file system with soft updates and mount it rw on /mnt]; touch /mnt/foo; mount -u -o ro /foo". I haven't been able to duplicate a problem without soft updates. Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 06:03:14 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6015F16A419 for ; Fri, 30 Nov 2007 06:03:14 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail10.syd.optusnet.com.au (mail10.syd.optusnet.com.au [211.29.132.191]) by mx1.freebsd.org (Postfix) with ESMTP id C7DE113C43E for ; Fri, 30 Nov 2007 06:03:13 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAU639vd026626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Nov 2007 17:03:10 +1100 Date: Fri, 30 Nov 2007 17:00:21 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans In-Reply-To: <20071130164034.D12284@delplex.bde.org> Message-ID: <20071130165529.V954@besplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130033743.GC31891@over-yonder.net> <20071130164034.D12284@delplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 06:03:14 -0000 On Fri, 30 Nov 2007, Bruce Evans wrote: > Duplicating the problem can be simplified to > "[make a disposable file system with soft updates and mount it rw on /mnt]; > touch /mnt/foo; mount -u -o ro /foo". Oops, this is missing a rm, and doesn't work with it. What actually "works" for me (with /f instead of /mnt) is: # cp /etc/passwd /f/a; sync; sleep 1; sync; sleep 1; sync; sleep 1; fsck -n /f; rm /f/a; mount -u -o fstab,ro /f; fsck -n /f This has produced the "softdep_waitidle: Failed to flush worklist for ..." message consistently 5-10 times so far. It takes a reboot per test. Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 06:03:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE76D16A46E for ; Fri, 30 Nov 2007 06:03:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 4230E13C458 for ; Fri, 30 Nov 2007 06:03:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1IxyQe-00020z-8i for freebsd-fs@freebsd.org; Fri, 30 Nov 2007 07:28:44 +0200 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id lAU5SfLU057523; Fri, 30 Nov 2007 07:28:41 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id lAU5Sfug057522; Fri, 30 Nov 2007 07:28:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Nov 2007 07:28:41 +0200 From: Kostik Belousov To: Bruce Evans Message-ID: <20071130052840.GH83121@deviant.kiev.zoral.com.ua> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VbfcI4OLZ4XW0yH2" Content-Disposition: inline In-Reply-To: <20071130151606.F12094@delplex.bde.org> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: 81fdb9d644701462f7849401ec726e21 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1835 [Nov 29 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 06:03:43 -0000 --VbfcI4OLZ4XW0yH2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 30, 2007 at 03:58:55PM +1100, Bruce Evans wrote: > On Fri, 30 Nov 2007, David Cecil wrote: >=20 > >Thanks Bruce. > > > >Actually, I had found the same problem, and I came up with the first lin= e=20 > >of your patch (adding IN_MODIFIED) myself, but I still saw the problem. = I >=20 > Yes, it's not that. Testing reminded me that there is normally a > VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very long > for unlink (it can only live long for open files). >=20 > Testing shows that the problem is easy to reproduce and often partially > detected before it becomes fatal. I saw something like the following: >=20 > after touch a; ln a b; rm a; unmount -- no problem with 1 link remain= ing > after touch a; rm a; unmount -- no problem with unmount > after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 link.= .. > after touch a; ; rm a; mount -u o ro -- worked once without soft > updates but seemed to be responsible for a soft update panic later > after touch a; ; rm a; mount -u o ro -- usually fails with soft > updates; the error is detected in various ways: > under ~5.2, mount -u prints "/f: update error: blocks 0 files 1" > but succeeds > under -current, mount -u fails and a subroutine prints > "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" > However, mount -u apparently cannot afford to fail at this > poing since it has committed to succeeding -- further > mount -u's and unmounts fail and it takes a reboot to reach > an fsck that can fix the problem. >=20 > mount -u seems to do some things right: at least under -current: > - it calls ffs_sync() and thus ffs_update() with waitfor !=3D 0. > - IN_MODIFIED is usually already set in ffs_update(). > - softdep_update_inode_inodeblock() in ffs_update() seems to > make null changes. That doesn't seem right -- shouldn't it > update the link count and finish removing the file?... I > just noticed that ufs_inactive() handles some of this. > - it calls softdep_flushfiles() after doing the sync. This > doesn't seem to touch the inode. > - apparently, softdep_flushfiles() fails in -current, while in > ~5.2 it bogusly succeeds and then code just after it is called > detects a problem but doesn't handle it. >=20 > >didn't pick up on the need for the second line (else if (DOINGASYNC(dvp)= )=20 > >{) though. It's a default mount, so I don't understand how that will=20 > >help, i.e. it won't be an async mount, right? >=20 > Ignore that. It is for async mounts, to make them unconditionally async. >=20 > >One more point to address Julian's question, the partition is not mounte= d=20 > >with soft updates. >=20 > Interesting. I saw no sign of the problem without soft updates except a > panic later after enabling soft updates. I was running fsck a lot but > may have forgotten one since no error was detected. The problem should > be easier to understand if it affects non-soft-updates. >=20 > [Context lost to top posting] >=20 As a speculation, it might be that ufs_inactive() should conditionalize on fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? --VbfcI4OLZ4XW0yH2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHT5+IC3+MBN1Mb4gRAswwAKC91VxP7p5kWAd55ar832Zqe+YULQCfc/5T Y3Pyj1zTUnBdjJrFx4U6gZM= =YcGm -----END PGP SIGNATURE----- --VbfcI4OLZ4XW0yH2-- From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 07:14:48 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42FD816A542 for ; Fri, 30 Nov 2007 07:14:48 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx09.nokia.com (smtp.nokia.com [192.100.105.134]) by mx1.freebsd.org (Postfix) with ESMTP id F25C313C458 for ; Fri, 30 Nov 2007 07:14:47 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh108.NOE.Nokia.com (esebh108.ntc.nokia.com [172.21.143.145]) by mgw-mx09.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lAU7DrQe017046; Fri, 30 Nov 2007 01:14:56 -0600 Received: from esebh104.NOE.Nokia.com ([172.21.143.34]) by esebh108.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 09:14:03 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh104.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 09:14:03 +0200 Received: from [172.30.67.20] ([172.30.67.20]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 30 Nov 2007 18:13:59 +1100 Message-ID: <474FB836.5060905@nokia.com> Date: Fri, 30 Nov 2007 17:13:58 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: ext Bruce Evans References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> In-Reply-To: <20071130151606.F12094@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 30 Nov 2007 07:13:59.0360 (UTC) FILETIME=[93E9C400:01C83320] X-Nokia-AV: Clean Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 07:14:48 -0000 I've determined the following for the scenario I have. These steps are executed during the boot cycle, and I reproduce the problem about 1 in 5-10 times: 1. mount -u -w / 2. rm -f /etc/myfile 3. mount -u -o ro / 1. finished Remounted R/W 2. started ufs_remove 786 ffs_truncate 268 ffs_update 87 ffs_update 92 ffs_update 99 ffs_update 140 ffs_update 87 ffs_update 92 ffs_update 99 ffs_update 140 2. finished: Removed file 3. Finished Remounted R/O Note that line 140 in ffs_update is the call to bdwrite, not bwrite. Investigations ongoing... Dave ext Bruce Evans wrote: > On Fri, 30 Nov 2007, David Cecil wrote: > >> Thanks Bruce. >> >> Actually, I had found the same problem, and I came up with the first >> line of your patch (adding IN_MODIFIED) myself, but I still saw the >> problem. I > > Yes, it's not that. Testing reminded me that there is normally a > VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very long > for unlink (it can only live long for open files). > > Testing shows that the problem is easy to reproduce and often partially > detected before it becomes fatal. I saw something like the following: > > after touch a; ln a b; rm a; unmount -- no problem with 1 link > remaining > after touch a; rm a; unmount -- no problem with unmount > after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 > link... > after touch a; ; rm a; mount -u o ro -- worked once without > soft > updates but seemed to be responsible for a soft update panic later > after touch a; ; rm a; mount -u o ro -- usually fails with soft > updates; the error is detected in various ways: > under ~5.2, mount -u prints "/f: update error: blocks 0 > files 1" > but succeeds > under -current, mount -u fails and a subroutine prints > "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" > However, mount -u apparently cannot afford to fail at this > poing since it has committed to succeeding -- further > mount -u's and unmounts fail and it takes a reboot to reach > an fsck that can fix the problem. > > mount -u seems to do some things right: at least under -current: > - it calls ffs_sync() and thus ffs_update() with waitfor != 0. > - IN_MODIFIED is usually already set in ffs_update(). > - softdep_update_inode_inodeblock() in ffs_update() seems to > make null changes. That doesn't seem right -- shouldn't it > update the link count and finish removing the file?... I > just noticed that ufs_inactive() handles some of this. > - it calls softdep_flushfiles() after doing the sync. This > doesn't seem to touch the inode. > - apparently, softdep_flushfiles() fails in -current, while in > ~5.2 it bogusly succeeds and then code just after it is called > detects a problem but doesn't handle it. > >> didn't pick up on the need for the second line (else if >> (DOINGASYNC(dvp)) {) though. It's a default mount, so I don't >> understand how that will help, i.e. it won't be an async mount, right? > > Ignore that. It is for async mounts, to make them unconditionally async. > >> One more point to address Julian's question, the partition is not >> mounted with soft updates. > > Interesting. I saw no sign of the problem without soft updates except a > panic later after enabling soft updates. I was running fsck a lot but > may have forgotten one since no error was detected. The problem should > be easier to understand if it affects non-soft-updates. > > [Context lost to top posting] > > Bruce > From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 16:34:09 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26E9916A4A1 for ; Fri, 30 Nov 2007 16:34:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.freebsd.org (Postfix) with ESMTP id BD22713C46B for ; Fri, 30 Nov 2007 16:34:07 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail03.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lAUGY3Np005117 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Dec 2007 03:34:05 +1100 Date: Sat, 1 Dec 2007 03:34:03 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Kostik Belousov In-Reply-To: <20071130052840.GH83121@deviant.kiev.zoral.com.ua> Message-ID: <20071201030305.G1170@besplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 16:34:09 -0000 On Fri, 30 Nov 2007, Kostik Belousov wrote: > On Fri, Nov 30, 2007 at 03:58:55PM +1100, Bruce Evans wrote: >> On Fri, 30 Nov 2007, David Cecil wrote: >... >>> One more point to address Julian's question, the partition is not mounted >>> with soft updates. >> >> Interesting. I saw no sign of the problem without soft updates except a >> panic later after enabling soft updates. I was running fsck a lot but >> may have forgotten one since no error was detected. The problem should >> be easier to understand if it affects non-soft-updates. > > As a speculation, it might be that ufs_inactive() should conditionalize on > fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the > IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? Something like that seems to be right. The folowing hack in ufs_inactive() seems to fix the problem with sift updates, as does unsetting MNT_RDONLY for the whole VOP_SYNC() in ffs_mount(). % Index: ufs_inode.c % =================================================================== % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_inode.c,v % retrieving revision 1.53 % diff -u -2 -r1.53 ufs_inode.c % --- ufs_inode.c 7 Apr 2004 03:47:20 -0000 1.53 % +++ ufs_inode.c 30 Nov 2007 12:58:39 -0000 % @@ -59,4 +59,6 @@ % #endif % % +#include % + % /* % * Last reference to an inode. If necessary, write or delete it. % @@ -118,6 +120,15 @@ % ip->i_flag &= ~IN_ACCESS; % } else { % + int wasro = 0; % + % (void) vn_write_suspend_wait(vp, NULL, V_WAIT); % + if (vp->v_mount->mnt_flag & MNT_RDONLY && % + ip->i_fs->fs_ronly == 0) { % + vp->v_mount->mnt_flag &= ~MNT_RDONLY; % + wasro = 1; % + } % UFS_UPDATE(vp, 0); % + if (wasro) % + vp->v_mount->mnt_flag |= MNT_RDONLY; % } % } I didn't bother with correct locking here (only tested under UP). With this change, the VOP_SYNC() in ffs_mount() for MNT_UPDATE seems to flush everything in simple cases (with no open files), just like the VOP_SYNC() in unmount() flushes everything before ffs_unmount() is reached. OTOH, without a forced flush, soft updates takes a long time to flush things -- more like 3 syncer periods than 1 for non-waitfor syncs. With soft updates, the above is called from deep in VOP_SYNC(). It's strange that the above non-waitfor UFS_UPDATE() is used inside of waitfor syncs. It apparently works because the waitfor syncs wait for it later, but only if it is non-null. BTW, *_mount() has lots of bogusness related to string options. In particular, ffs_mount() for update from r/w to r/o checks the "ro" string option and sets MNT_RDONLY later, but MNT_RDONLY is already set and other things depend on it being set early. Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 17:44:33 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65C8416A421 for ; Fri, 30 Nov 2007 17:44:33 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from optimus.centralmiss.com (ns.centralmiss.com [206.156.254.79]) by mx1.freebsd.org (Postfix) with ESMTP id 3FE0713C4E1 for ; Fri, 30 Nov 2007 17:44:33 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (adsl-072-148-013-213.sip.jan.bellsouth.net [72.148.13.213]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by optimus.centralmiss.com (Postfix) with ESMTP id 828E2284A1; Fri, 30 Nov 2007 11:44:32 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 0A0AD61C44; Fri, 30 Nov 2007 11:44:32 -0600 (CST) Date: Fri, 30 Nov 2007 11:44:31 -0600 From: "Matthew D. Fuller" To: Bruce Evans Message-ID: <20071130174431.GE31891@over-yonder.net> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130033743.GC31891@over-yonder.net> <20071130164034.D12284@delplex.bde.org> <20071130165529.V954@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071130165529.V954@besplex.bde.org> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.16-fullermd.4 (2007-06-09) Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2007 17:44:33 -0000 On Fri, Nov 30, 2007 at 05:00:21PM +1100 I heard the voice of Bruce Evans, and lo! it spake thus: > > Oops, this is missing a rm, and doesn't work with it. Yes, I haven't been able to reliably reproduce it without rm'ing stuff; mtree just gave me a nice easy way to make a bunch of stuff for testing. I mostly saw it with /usr/ports, which I keep mounted r/o except when building (which means the last thing before remounting is generally rm'ing a big tree). It has happened with / too, where I'd remount to edit something in /etc, but vim may do a rename() of a temp file instead of editing in place, so maybe it's still removing an inode. Last year, it used to not cause the softdep_waitidle messages and prevent the fs from being remounted. Instead, it would give an error like: hostname kernel: /: update error: blocks 28 files 2 and WOULD remount it, and even set the clean flag, but would still leave turds lying around that would need a manual fsck to clean up (fsck -p obviously would completely skip it, since it was marked clean). It was early this year that it moved from that annoying to the "locked fs" crippling variant. (n.b.: I don't have any real evidence that it's a mutation of the same problem, rather than two different ones, aside from the trigger condition apparently being the same, and the newer completely replacing the older.) > It takes a reboot per test. Yah :( I've been retrying it every few months post-upgrade, when I could just reboot the mess away. And otherwise just being very careful to sync a bunch of times, and wait a few minutes before `mount -u -oro`ing stuff. Rather a pain. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 00:42:55 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 556DB16A419 for ; Sat, 1 Dec 2007 00:42:55 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx09.nokia.com (smtp.nokia.com [192.100.105.134]) by mx1.freebsd.org (Postfix) with ESMTP id 2236C13C45A for ; Sat, 1 Dec 2007 00:42:54 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh106.NOE.Nokia.com (esebh106.ntc.nokia.com [172.21.138.213]) by mgw-mx09.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lB10gZxI010898; Fri, 30 Nov 2007 18:43:04 -0600 Received: from esebh104.NOE.Nokia.com ([172.21.143.34]) by esebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 02:42:51 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh104.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 02:42:51 +0200 Received: from [172.30.10.196] ([172.30.10.196]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 11:42:46 +1100 Message-ID: <4750AE03.9010603@nokia.com> Date: Sat, 01 Dec 2007 10:42:43 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: ext Bruce Evans References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <20071201030305.G1170@besplex.bde.org> In-Reply-To: <20071201030305.G1170@besplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 01 Dec 2007 00:42:46.0672 (UTC) FILETIME=[17839100:01C833B3] X-Nokia-AV: Clean Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 00:42:55 -0000 Hi Bruce, I tried the change and it doesn't fix the non-sofdep problem. Looking at that code got me thinking though. I thought that a workable solution for us short term might be to call UFS_UPDATE with waitfor == 1 if the filesystem is mounted synchronous. So, the scenario is: mount -u -w -o sync / remove file mount -u -o ro / I observed that the second call to ffs_update with the offending vnode caused bwrite to be called, though the first call was still bdwrite. Anyhow, the problem still occurred. I tried clearing the B_ASYNC flag in ffs_update if bwrite is called, but still the problem occurs. Obviously I'm missing something. Dave ext Bruce Evans wrote: > On Fri, 30 Nov 2007, Kostik Belousov wrote: > >> On Fri, Nov 30, 2007 at 03:58:55PM +1100, Bruce Evans wrote: >>> On Fri, 30 Nov 2007, David Cecil wrote: >> ... >>>> One more point to address Julian's question, the partition is not >>>> mounted >>>> with soft updates. >>> >>> Interesting. I saw no sign of the problem without soft updates >>> except a >>> panic later after enabling soft updates. I was running fsck a lot but >>> may have forgotten one since no error was detected. The problem should >>> be easier to understand if it affects non-soft-updates. >> >> As a speculation, it might be that ufs_inactive() should >> conditionalize on >> fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the >> IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? > > Something like that seems to be right. The folowing hack in > ufs_inactive() > seems to fix the problem with sift updates, as does unsetting MNT_RDONLY > for the whole VOP_SYNC() in ffs_mount(). > > % Index: ufs_inode.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_inode.c,v > % retrieving revision 1.53 > % diff -u -2 -r1.53 ufs_inode.c > % --- ufs_inode.c 7 Apr 2004 03:47:20 -0000 1.53 > % +++ ufs_inode.c 30 Nov 2007 12:58:39 -0000 > % @@ -59,4 +59,6 @@ > % #endif > % % +#include > % + > % /* > % * Last reference to an inode. If necessary, write or delete it. > % @@ -118,6 +120,15 @@ > % ip->i_flag &= ~IN_ACCESS; > % } else { > % + int wasro = 0; > % + > % (void) vn_write_suspend_wait(vp, NULL, V_WAIT); > % + if (vp->v_mount->mnt_flag & MNT_RDONLY && > % + ip->i_fs->fs_ronly == 0) { > % + vp->v_mount->mnt_flag &= ~MNT_RDONLY; > % + wasro = 1; > % + } > % UFS_UPDATE(vp, 0); > % + if (wasro) > % + vp->v_mount->mnt_flag |= MNT_RDONLY; > % } > % } > > I didn't bother with correct locking here (only tested under UP). > > With this change, the VOP_SYNC() in ffs_mount() for MNT_UPDATE seems > to flush everything in simple cases (with no open files), just like > the VOP_SYNC() in unmount() flushes everything before ffs_unmount() is > reached. OTOH, without a forced flush, soft updates takes a long > time to flush things -- more like 3 syncer periods than 1 for > non-waitfor syncs. With soft updates, the above is called from deep > in VOP_SYNC(). It's strange that the above non-waitfor UFS_UPDATE() > is used inside of waitfor syncs. It apparently works because the > waitfor syncs wait for it later, but only if it is non-null. > > BTW, *_mount() has lots of bogusness related to string options. In > particular, ffs_mount() for update from r/w to r/o checks the "ro" > string option and sets MNT_RDONLY later, but MNT_RDONLY is already set > and other things depend on it being set early. > > Bruce > From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 02:01:35 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67C8216A417 for ; Sat, 1 Dec 2007 02:01:35 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail13.syd.optusnet.com.au (mail13.syd.optusnet.com.au [211.29.132.194]) by mx1.freebsd.org (Postfix) with ESMTP id 026F813C478 for ; Sat, 1 Dec 2007 02:01:34 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail13.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lB121UCi025237 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Dec 2007 13:01:32 +1100 Date: Sat, 1 Dec 2007 13:01:30 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: "Matthew D. Fuller" In-Reply-To: <20071130174431.GE31891@over-yonder.net> Message-ID: <20071201125029.O15170@delplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130033743.GC31891@over-yonder.net> <20071130164034.D12284@delplex.bde.org> <20071130165529.V954@besplex.bde.org> <20071130174431.GE31891@over-yonder.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 02:01:35 -0000 On Fri, 30 Nov 2007, Matthew D. Fuller wrote: > On Fri, Nov 30, 2007 at 05:00:21PM +1100 I heard the voice of > Bruce Evans, and lo! it spake thus: >> >> Oops, this is missing a rm, and doesn't work with it. > > Last year, it used to not cause the softdep_waitidle messages and > prevent the fs from being remounted. Instead, it would give an error > like: > > hostname kernel: /: update error: blocks 28 files 2 softdep_waitidle() is new. It now detects the problem earlier and handles it more robustly by not allowing the mount -u. Well, maybe this is less robust since it also doesn't allow unmount. > and WOULD remount it, and even set the clean flag, but would still > leave turds lying around that would need a manual fsck to clean up > (fsck -p obviously would completely skip it, since it was marked > clean). It was early this year that it moved from that annoying to It also shows bugs in fsck: - even with the file system not marked clean (with later versions), fsck -p doesn't notice the problem. - fsck notices the problem, but takes 2 or 3 passes to fix it, and doesn't notice that it needs several passes. > the "locked fs" crippling variant. (n.b.: I don't have any real > evidence that it's a mutation of the same problem, rather than two > different ones, aside from the trigger condition apparently being the > same, and the newer completely replacing the older.) I think it is the same. softdep_waitidle() just waits a bit to flush the dependencies after starting the flushing, but the bug gives an unflushable dependency so the wait always times out. >> It takes a reboot per test. And has remarkable timing dependencies. Once I got into a state in which the bug didn't appear when exercised in a loop with the same delays that seemed to cause it fairly deterministically other times. Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 02:23:38 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1750316A418 for ; Sat, 1 Dec 2007 02:23:38 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail13.syd.optusnet.com.au (mail13.syd.optusnet.com.au [211.29.132.194]) by mx1.freebsd.org (Postfix) with ESMTP id A5F1813C442 for ; Sat, 1 Dec 2007 02:23:37 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail13.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lB12NZUG020780 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Dec 2007 13:23:35 +1100 Date: Sat, 1 Dec 2007 13:23:34 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: David Cecil In-Reply-To: <4750AE03.9010603@nokia.com> Message-ID: <20071201130225.E15170@delplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <20071201030305.G1170@besplex.bde.org> <4750AE03.9010603@nokia.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 02:23:38 -0000 On Sat, 1 Dec 2007, David Cecil wrote: > I tried the change and it doesn't fix the non-sofdep problem. Urk. Apparently, the extra ffs_update() activity only helps as a side effect. > Looking at that code got me thinking though. I thought that a workable > solution for us short term might be to call UFS_UPDATE with waitfor == 1 if > the filesystem is mounted synchronous. It is strange that ithat and some other things are fully async even if the fs is mounted (not-quite-)fully sync (or the file is open with O_SYNC). I;ve thought a bit about this and decided that it is mostly a good optimization -- some things can be reconstructed by fsck and it is a good optimization to not write then synchronously. Note that waitfor == 1 in UFS_UPDATE() forces an i/o unconditionally. In some kernels I use a patch (posted to this list a month ago) to avoid i/o if the inode hasn't changed and the buffer containing the inode isn't dirty. This doesn't seem to affect the bug, but it might make the bug larger for soft updates, since soft updates might be depending on the i/o to trigger a callback. > So, the scenario is: > mount -u -w -o sync / > remove file > mount -u -o ro / > > I observed that the second call to ffs_update with the offending vnode caused > bwrite to be called, though the first call was still bdwrite. Anyhow, the > problem still occurred. I tried clearing the B_ASYNC flag in ffs_update if > bwrite is called, but still the problem occurs. Obviously I'm missing > something. When is the second update? Soft updates seems to trigger delayed updates as part of syncing, but I don't see how there could be more than one non-null ffs_update that does a bwrite for the non-soft updates sync mounted case, and for the soft updates sync mounted case the sync shouldn't be delayed. Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 04:28:05 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A1EF16A417 for ; Sat, 1 Dec 2007 04:28:05 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-mx06.nokia.com (smtp.nokia.com [192.100.122.233]) by mx1.freebsd.org (Postfix) with ESMTP id 12A8913C45B for ; Sat, 1 Dec 2007 04:28:04 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh105.NOE.Nokia.com (esebh105.ntc.nokia.com [172.21.138.211]) by mgw-mx06.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id lB14RkHZ007894; Sat, 1 Dec 2007 06:28:02 +0200 Received: from esebh103.NOE.Nokia.com ([172.21.143.33]) by esebh105.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 06:27:55 +0200 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by esebh103.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 06:27:54 +0200 Received: from [172.30.10.196] ([172.30.10.196]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 1 Dec 2007 15:27:49 +1100 Message-ID: <4750E2C4.9030003@nokia.com> Date: Sat, 01 Dec 2007 14:27:48 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: ext Bruce Evans References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <20071201030305.G1170@besplex.bde.org> <4750AE03.9010603@nokia.com> <20071201130225.E15170@delplex.bde.org> In-Reply-To: <20071201130225.E15170@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 01 Dec 2007 04:27:50.0199 (UTC) FILETIME=[883E4470:01C833D2] X-Nokia-AV: Clean Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 04:28:05 -0000 ext Bruce Evans wrote: > On Sat, 1 Dec 2007, David Cecil wrote: > >> I tried the change and it doesn't fix the non-sofdep problem. > > Urk. Apparently, the extra ffs_update() activity only helps as a side > effect. > >> Looking at that code got me thinking though. I thought that a >> workable solution for us short term might be to call UFS_UPDATE with >> waitfor == 1 if the filesystem is mounted synchronous. > > It is strange that ithat and some other things are fully async even if > the > fs is mounted (not-quite-)fully sync (or the file is open with O_SYNC). Yes, I've noticed that too. > I;ve thought a bit about this and decided that it is mostly a good > optimization -- some things can be reconstructed by fsck and it is a good > optimization to not write then synchronously. > > Note that waitfor == 1 in UFS_UPDATE() forces an i/o unconditionally. Yes, that's why in the short term I thought I'd tie it to MNT_SYNCHRONOUS. > In > some kernels I use a patch (posted to this list a month ago) to avoid i/o > if the inode hasn't changed and the buffer containing the inode isn't > dirty. This doesn't seem to affect the bug, but it might make the bug > larger for soft updates, since soft updates might be depending on the i/o > to trigger a callback. > >> So, the scenario is: >> mount -u -w -o sync / >> remove file >> mount -u -o ro / >> >> I observed that the second call to ffs_update with the offending >> vnode caused bwrite to be called, though the first call was still >> bdwrite. Anyhow, the problem still occurred. I tried clearing the >> B_ASYNC flag in ffs_update if bwrite is called, but still the problem >> occurs. Obviously I'm missing something. > > When is the second update? The first is from ffs_truncate. I don't know where the second call comes from, yet. > Soft updates seems to trigger delayed updates > as part of syncing, but I don't see how there could be more than one > non-null ffs_update that does a bwrite for the non-soft updates sync > mounted > case, and for the soft updates sync mounted case the sync shouldn't be > delayed. Unfortunately, for this weekend at least, I probably won't be able to investigate this too much further. Project deadlines mean I'm going to have to work around this another way, unfortunately. I'll try and get back to it though. Regards, Dave From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 07:27:11 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1832D16A418 for ; Sat, 1 Dec 2007 07:27:11 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (adsl-75-1-14-242.dsl.scrm01.sbcglobal.net [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id CFDF413C44B for ; Sat, 1 Dec 2007 07:27:10 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id lB17Pefl011945; Fri, 30 Nov 2007 23:25:44 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200712010725.lB17Pefl011945@gw.catspoiler.org> Date: Fri, 30 Nov 2007 23:25:40 -0800 (PST) From: Don Lewis To: brde@optusnet.com.au In-Reply-To: <20071201030305.G1170@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 07:27:11 -0000 On 1 Dec, Bruce Evans wrote: > Something like that seems to be right. The folowing hack in ufs_inactive() > seems to fix the problem with sift updates, as does unsetting MNT_RDONLY > for the whole VOP_SYNC() in ffs_mount(). > > % Index: ufs_inode.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_inode.c,v > % retrieving revision 1.53 > % diff -u -2 -r1.53 ufs_inode.c > % --- ufs_inode.c 7 Apr 2004 03:47:20 -0000 1.53 > % +++ ufs_inode.c 30 Nov 2007 12:58:39 -0000 > % @@ -59,4 +59,6 @@ > % #endif > % > % +#include > % + > % /* > % * Last reference to an inode. If necessary, write or delete it. > % @@ -118,6 +120,15 @@ > % ip->i_flag &= ~IN_ACCESS; > % } else { > % + int wasro = 0; > % + > % (void) vn_write_suspend_wait(vp, NULL, V_WAIT); > % + if (vp->v_mount->mnt_flag & MNT_RDONLY && > % + ip->i_fs->fs_ronly == 0) { > % + vp->v_mount->mnt_flag &= ~MNT_RDONLY; > % + wasro = 1; > % + } > % UFS_UPDATE(vp, 0); > % + if (wasro) > % + vp->v_mount->mnt_flag |= MNT_RDONLY; > % } > % } I'm somewhat suprised that this makes a difference. I didn't see anything obvious under UFS_UPDATE() that looks at MNT_RDONLY. > I didn't bother with correct locking here (only tested under UP). I suspect that clearing MNT_RDONLY like this could allow more write operations to be started from userland. > With this change, the VOP_SYNC() in ffs_mount() for MNT_UPDATE seems > to flush everything in simple cases (with no open files), just like > the VOP_SYNC() in unmount() flushes everything before ffs_unmount() is > reached. OTOH, without a forced flush, soft updates takes a long > time to flush things -- more like 3 syncer periods than 1 for > non-waitfor syncs. With soft updates, the above is called from deep > in VOP_SYNC(). It's strange that the above non-waitfor UFS_UPDATE() > is used inside of waitfor syncs. It apparently works because the > waitfor syncs wait for it later, but only if it is non-null. I think the intent is that this call to UFS_UPDATE() is to trigger the writes of any dirty data blocks, which must be written before the inode can be written. The write of the inode itself will probably be triggered by the next iteration of the syncer, by vflush() (in the unmount or remount case), or possibly by a soft updates callback. Also, I think that waiting to flush the inode might allow multiple inodes in the same block to be written at once. From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 07:27:11 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B14416A41B for ; Sat, 1 Dec 2007 07:27:11 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (adsl-75-1-14-242.dsl.scrm01.sbcglobal.net [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 2CDB413C455 for ; Sat, 1 Dec 2007 07:27:11 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id lB17FlZw011929; Fri, 30 Nov 2007 23:15:51 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200712010715.lB17FlZw011929@gw.catspoiler.org> Date: Fri, 30 Nov 2007 23:15:47 -0800 (PST) From: Don Lewis To: kostikbel@gmail.com In-Reply-To: <20071130052840.GH83121@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 07:27:11 -0000 On 30 Nov, Kostik Belousov wrote: > On Fri, Nov 30, 2007 at 03:58:55PM +1100, Bruce Evans wrote: >> On Fri, 30 Nov 2007, David Cecil wrote: >> >> >Thanks Bruce. >> > >> >Actually, I had found the same problem, and I came up with the first line >> >of your patch (adding IN_MODIFIED) myself, but I still saw the problem. I >> >> Yes, it's not that. Testing reminded me that there is normally a >> VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very long >> for unlink (it can only live long for open files). >> >> Testing shows that the problem is easy to reproduce and often partially >> detected before it becomes fatal. I saw something like the following: >> >> after touch a; ln a b; rm a; unmount -- no problem with 1 link remaining >> after touch a; rm a; unmount -- no problem with unmount >> after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 link... >> after touch a; ; rm a; mount -u o ro -- worked once without soft >> updates but seemed to be responsible for a soft update panic later >> after touch a; ; rm a; mount -u o ro -- usually fails with soft >> updates; the error is detected in various ways: >> under ~5.2, mount -u prints "/f: update error: blocks 0 files 1" >> but succeeds >> under -current, mount -u fails and a subroutine prints >> "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" >> However, mount -u apparently cannot afford to fail at this >> poing since it has committed to succeeding -- further >> mount -u's and unmounts fail and it takes a reboot to reach >> an fsck that can fix the problem. >> >> mount -u seems to do some things right: at least under -current: >> - it calls ffs_sync() and thus ffs_update() with waitfor != 0. >> - IN_MODIFIED is usually already set in ffs_update(). >> - softdep_update_inode_inodeblock() in ffs_update() seems to >> make null changes. That doesn't seem right -- shouldn't it >> update the link count and finish removing the file?... I >> just noticed that ufs_inactive() handles some of this. >> - it calls softdep_flushfiles() after doing the sync. This >> doesn't seem to touch the inode. >> - apparently, softdep_flushfiles() fails in -current, while in >> ~5.2 it bogusly succeeds and then code just after it is called >> detects a problem but doesn't handle it. >> >> >didn't pick up on the need for the second line (else if (DOINGASYNC(dvp)) >> >{) though. It's a default mount, so I don't understand how that will >> >help, i.e. it won't be an async mount, right? >> >> Ignore that. It is for async mounts, to make them unconditionally async. >> >> >One more point to address Julian's question, the partition is not mounted >> >with soft updates. >> >> Interesting. I saw no sign of the problem without soft updates except a >> panic later after enabling soft updates. I was running fsck a lot but >> may have forgotten one since no error was detected. The problem should >> be easier to understand if it affects non-soft-updates. >> >> [Context lost to top posting] >> > > As a speculation, it might be that ufs_inactive() should conditionalize on > fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the > IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? That sounds reasonable to me. I see that ffs_update(), which is called by ufs_inactive(), looks at fs_ronly. The other difference that I see between the remount to read-only case, which is broken, and the unmount case, which is presumably working, is that the remount case calls ffs_flushfiles with the WRITECLOSE flag, which makes me a little suspicious of the WRITECLOSE code in vflush(). From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 08:07:19 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9361716A417 for ; Sat, 1 Dec 2007 08:07:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 3620113C45B for ; Sat, 1 Dec 2007 08:07:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1IyNNZ-00031y-9z; Sat, 01 Dec 2007 10:07:17 +0200 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id lB187A75031536; Sat, 1 Dec 2007 10:07:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id lB1879VC031535; Sat, 1 Dec 2007 10:07:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 1 Dec 2007 10:07:09 +0200 From: Kostik Belousov To: Don Lewis Message-ID: <20071201080709.GO83121@deviant.kiev.zoral.com.ua> References: <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <200712010715.lB17FlZw011929@gw.catspoiler.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f4HxWLVbzokH9yio" Content-Disposition: inline In-Reply-To: <200712010715.lB17FlZw011929@gw.catspoiler.org> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: cba338bb38be4578e9980d7a4db64caa X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1836 [Nov 30 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 08:07:19 -0000 --f4HxWLVbzokH9yio Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 30, 2007 at 11:15:47PM -0800, Don Lewis wrote: > On 30 Nov, Kostik Belousov wrote: > > On Fri, Nov 30, 2007 at 03:58:55PM +1100, Bruce Evans wrote: > >> On Fri, 30 Nov 2007, David Cecil wrote: > >>=20 > >> >Thanks Bruce. > >> > > >> >Actually, I had found the same problem, and I came up with the first = line=20 > >> >of your patch (adding IN_MODIFIED) myself, but I still saw the proble= m. I > >>=20 > >> Yes, it's not that. Testing reminded me that there is normally a > >> VOP_INACTIVE() after unlink so the IN_CHANGE mark doesn't live very lo= ng > >> for unlink (it can only live long for open files). > >>=20 > >> Testing shows that the problem is easy to reproduce and often partially > >> detected before it becomes fatal. I saw something like the following: > >>=20 > >> after touch a; ln a b; rm a; unmount -- no problem with 1 link rem= aining > >> after touch a; rm a; unmount -- no problem with unmount > >> after touch a; ln a b; rm a; mount -u o ro -- no problem with 1 li= nk... > >> after touch a; ; rm a; mount -u o ro -- worked once without = soft > >> updates but seemed to be responsible for a soft update panic la= ter > >> after touch a; ; rm a; mount -u o ro -- usually fails with s= oft > >> updates; the error is detected in various ways: > >> under ~5.2, mount -u prints "/f: update error: blocks 0 file= s 1" > >> but succeeds > >> under -current, mount -u fails and a subroutine prints > >> "softdep_waitidle: Failed to flush worklist for 0xc3e1a29c" > >> However, mount -u apparently cannot afford to fail at this > >> poing since it has committed to succeeding -- further > >> mount -u's and unmounts fail and it takes a reboot to reach > >> an fsck that can fix the problem. > >>=20 > >> mount -u seems to do some things right: at least under -current: > >> - it calls ffs_sync() and thus ffs_update() with waitfor !=3D 0. > >> - IN_MODIFIED is usually already set in ffs_update(). > >> - softdep_update_inode_inodeblock() in ffs_update() seems to > >> make null changes. That doesn't seem right -- shouldn't it > >> update the link count and finish removing the file?... I > >> just noticed that ufs_inactive() handles some of this. > >> - it calls softdep_flushfiles() after doing the sync. This > >> doesn't seem to touch the inode. > >> - apparently, softdep_flushfiles() fails in -current, while in > >> ~5.2 it bogusly succeeds and then code just after it is called > >> detects a problem but doesn't handle it. > >>=20 > >> >didn't pick up on the need for the second line (else if (DOINGASYNC(d= vp))=20 > >> >{) though. It's a default mount, so I don't understand how that will= =20 > >> >help, i.e. it won't be an async mount, right? > >>=20 > >> Ignore that. It is for async mounts, to make them unconditionally asy= nc. > >>=20 > >> >One more point to address Julian's question, the partition is not mou= nted=20 > >> >with soft updates. > >>=20 > >> Interesting. I saw no sign of the problem without soft updates except= a > >> panic later after enabling soft updates. I was running fsck a lot but > >> may have forgotten one since no error was detected. The problem should > >> be easier to understand if it affects non-soft-updates. > >>=20 > >> [Context lost to top posting] > >>=20 > >=20 > > As a speculation, it might be that ufs_inactive() should conditionalize= on > > fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the > > IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? >=20 > That sounds reasonable to me. I see that ffs_update(), which is called > by ufs_inactive(), looks at fs_ronly. The patch (compile-tested only) is below. diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c index cbccc62..687011d 100644 --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -79,6 +79,7 @@ static void ffs_oldfscompat_read(struct fs *, struct ufsm= ount *, ufs2_daddr_t); static void ffs_oldfscompat_write(struct fs *, struct ufsmount *); static void ffs_ifree(struct ufsmount *ump, struct inode *ip); +static int ffs_isronly(struct ufsmount *ump); static vfs_init_t ffs_init; static vfs_uninit_t ffs_uninit; static vfs_extattrctl_t ffs_extattrctl; @@ -748,6 +749,7 @@ ffs_mountfs(devvp, mp, td) ump->um_valloc =3D ffs_valloc; ump->um_vfree =3D ffs_vfree; ump->um_ifree =3D ffs_ifree; + ump->um_isronly =3D ffs_isronly; mtx_init(UFS_MTX(ump), "FFS", "FFS Lock", MTX_DEF); bcopy(bp->b_data, ump->um_fs, (u_int)fs->fs_sbsize); if (fs->fs_sbsize < SBLOCKSIZE) @@ -1862,3 +1864,12 @@ ffs_geom_strategy(struct bufobj *bo, struct buf *bp) } g_vfs_strategy(bo, bp); } + +static int +ffs_isronly(struct ufsmount *ump) +{ + struct fs *fs =3D ump->um_fs; + + return (fs->fs_ronly); +} + diff --git a/sys/ufs/ufs/ufs_inode.c b/sys/ufs/ufs/ufs_inode.c index 448f436..a9abbeb 100644 --- a/sys/ufs/ufs/ufs_inode.c +++ b/sys/ufs/ufs/ufs_inode.c @@ -90,8 +90,7 @@ ufs_inactive(ap) ufs_gjournal_close(vp); #endif if ((ip->i_effnlink =3D=3D 0 && DOINGSOFTDEP(vp)) || - (ip->i_nlink <=3D 0 && - (vp->v_mount->mnt_flag & MNT_RDONLY) =3D=3D 0)) { + (ip->i_nlink <=3D 0 && !UFS_ISRONLY(ip->i_ump))) { loop: if (vn_start_secondary_write(vp, &mp, V_NOWAIT) !=3D 0) { /* Cannot delete file while file system is suspended */ @@ -121,7 +120,7 @@ ufs_inactive(ap) } if (ip->i_effnlink =3D=3D 0 && DOINGSOFTDEP(vp)) softdep_releasefile(ip); - if (ip->i_nlink <=3D 0 && (vp->v_mount->mnt_flag & MNT_RDONLY) =3D=3D 0) { + if (ip->i_nlink <=3D 0 && !UFS_ISRONLY(ip->i_ump)) { #ifdef QUOTA if (!getinoquota(ip)) (void)chkiq(ip, -1, NOCRED, FORCE); diff --git a/sys/ufs/ufs/ufsmount.h b/sys/ufs/ufs/ufsmount.h index 80fe3fa..57e7dd0 100644 --- a/sys/ufs/ufs/ufsmount.h +++ b/sys/ufs/ufs/ufsmount.h @@ -93,6 +93,7 @@ struct ufsmount { int (*um_valloc)(struct vnode *, int, struct ucred *, struct vnode **); int (*um_vfree)(struct vnode *, ino_t, int); void (*um_ifree)(struct ufsmount *, struct inode *); + int (*um_isronly)(struct ufsmount *); }; =20 #define UFS_BALLOC(aa, bb, cc, dd, ee, ff) VFSTOUFS((aa)->v_mount)->um_bal= loc(aa, bb, cc, dd, ee, ff) @@ -102,6 +103,7 @@ struct ufsmount { #define UFS_VALLOC(aa, bb, cc, dd) VFSTOUFS((aa)->v_mount)->um_valloc(aa, = bb, cc, dd) #define UFS_VFREE(aa, bb, cc) VFSTOUFS((aa)->v_mount)->um_vfree(aa, bb, cc) #define UFS_IFREE(aa, bb) ((aa)->um_ifree(aa, bb)) +#define UFS_ISRONLY(aa) ((aa)->um_isronly(aa)) =20 #define UFS_LOCK(aa) mtx_lock(&(aa)->um_lock) #define UFS_UNLOCK(aa) mtx_unlock(&(aa)->um_lock) --f4HxWLVbzokH9yio Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHURYtC3+MBN1Mb4gRAuz7AJ402T07oC29X4x7cl3OPoh/EylrqQCcDvoK 82kvkH2TYZEMHmlkfIoZ8Gc= =r8eH -----END PGP SIGNATURE----- --f4HxWLVbzokH9yio-- From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 11:32:25 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD5AF16A41B; Sat, 1 Dec 2007 11:32:25 +0000 (UTC) (envelope-from johan@headweb.com) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id 9422E13C4F2; Sat, 1 Dec 2007 11:32:25 +0000 (UTC) (envelope-from johan@headweb.com) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id 49B67D472CE; Sat, 1 Dec 2007 12:17:22 +0100 (CET) X-Virus-Scanned: amavisd-new at stromnet.se Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id ON9RJZ3IwFUo; Sat, 1 Dec 2007 12:17:18 +0100 (CET) Received: from [172.28.1.102] (90-224-172-102-no129.tbcn.telia.com [90.224.172.102]) by core.stromnet.se (Postfix) with ESMTP id 9BFEAD472CD; Sat, 1 Dec 2007 12:17:18 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v752.3) Content-Transfer-Encoding: quoted-printable Message-Id: <66A69F9D-E4C1-4647-AEE7-E6F18010A1A3@headweb.com> Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org, freebsd-current@freebsd.org From: =?ISO-8859-1?Q?Johan_Str=F6m?= Date: Sat, 1 Dec 2007 12:16:45 +0100 X-Mailer: Apple Mail (2.752.3) Cc: Subject: scrambled (gmirror) dmesg output X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 11:32:25 -0000 Hello Im playing with a new box running RELENG_7.0 from yesterday. I got =20 two discs with gmirror on ad[6|14]s1a and zfs-mirror on s1d. When o =20 do atacontrol detach ata7 (detach ad14), i get this in dmesG: (first time) subdisk14: detached ad14: detached GEOM_MIRROR: Device gm1b: provider ad14s1bG dEiOsMc_oMnInReRcOtRe:d .De vice gm1: provider ad14s1a disconnected. (second time, detaching again after reattach) subdisk14: detached ad14: detached GEOMG_EMOIMR_RMOIRR:R ORD:e viDceev icgem 1bg:m 1p:r opvriodveird era =20= d1a4ds114bs 1dai sdciosncnoencnteecdt.ed. huh? :) Some print raceing or something? Btw, Im doing ZFS'ed root as on wiki, but i added gmirror to the root =20= partition to (and steps to install from one disc to the other, then =20 boot over and add the original disc to mirrors).. I've documented the steps (or at least the commands and some simple =20 comments), would anyone be interested in having it, on the wiki or =20 otherwise? -- Johan Str=F6m Stromnet johan@stromnet.se http://www.stromnet.se/ From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 11:34:50 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4556E16A474 for ; Sat, 1 Dec 2007 11:34:50 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by mx1.freebsd.org (Postfix) with ESMTP id E2BD113C46E for ; Sat, 1 Dec 2007 11:34:49 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lB1BYjFJ030796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Dec 2007 22:34:48 +1100 Date: Sat, 1 Dec 2007 22:34:45 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Kostik Belousov In-Reply-To: <20071201080709.GO83121@deviant.kiev.zoral.com.ua> Message-ID: <20071201215706.B12006@besplex.bde.org> References: <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <200712010715.lB17FlZw011929@gw.catspoiler.org> <20071201080709.GO83121@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, Don Lewis Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 11:34:50 -0000 On Sat, 1 Dec 2007, Kostik Belousov wrote: > On Fri, Nov 30, 2007 at 11:15:47PM -0800, Don Lewis wrote: >> On 30 Nov, Kostik Belousov wrote: >>> As a speculation, it might be that ufs_inactive() should conditionalize on >>> fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the >>> IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? >> >> That sounds reasonable to me. I see that ffs_update(), which is called >> by ufs_inactive(), looks at fs_ronly. > The patch (compile-tested only) is below. > > diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c > index cbccc62..687011d 100644 > --- a/sys/ufs/ffs/ffs_vfsops.c > +++ b/sys/ufs/ffs/ffs_vfsops.c > @@ -79,6 +79,7 @@ static void ffs_oldfscompat_read(struct fs *, struct ufsmount *, > ufs2_daddr_t); > static void ffs_oldfscompat_write(struct fs *, struct ufsmount *); > static void ffs_ifree(struct ufsmount *ump, struct inode *ip); > +static int ffs_isronly(struct ufsmount *ump); > static vfs_init_t ffs_init; > static vfs_uninit_t ffs_uninit; > static vfs_extattrctl_t ffs_extattrctl; > @@ -748,6 +749,7 @@ ffs_mountfs(devvp, mp, td) > ump->um_valloc = ffs_valloc; > ump->um_vfree = ffs_vfree; > ump->um_ifree = ffs_ifree; > + ump->um_isronly = ffs_isronly; > mtx_init(UFS_MTX(ump), "FFS", "FFS Lock", MTX_DEF); > bcopy(bp->b_data, ump->um_fs, (u_int)fs->fs_sbsize); > if (fs->fs_sbsize < SBLOCKSIZE) I don't like these indirections. The layering provided by them is pseudo. > @@ -1862,3 +1864,12 @@ ffs_geom_strategy(struct bufobj *bo, struct buf *bp) > } > g_vfs_strategy(bo, bp); > } > + > +static int > +ffs_isronly(struct ufsmount *ump) > +{ > + struct fs *fs = ump->um_fs; > + > + return (fs->fs_ronly); > +} > + Could be ump->um_fs->fs_ronly. > diff --git a/sys/ufs/ufs/ufs_inode.c b/sys/ufs/ufs/ufs_inode.c > index 448f436..a9abbeb 100644 > --- a/sys/ufs/ufs/ufs_inode.c > +++ b/sys/ufs/ufs/ufs_inode.c > @@ -90,8 +90,7 @@ ufs_inactive(ap) > ufs_gjournal_close(vp); > #endif > if ((ip->i_effnlink == 0 && DOINGSOFTDEP(vp)) || > - (ip->i_nlink <= 0 && > - (vp->v_mount->mnt_flag & MNT_RDONLY) == 0)) { > + (ip->i_nlink <= 0 && !UFS_ISRONLY(ip->i_ump))) { > loop: > if (vn_start_secondary_write(vp, &mp, V_NOWAIT) != 0) { > /* Cannot delete file while file system is suspended */ This might help for non-soft updates, but with soft updates I also see the problem in ~5.2 which doesn't have any r/o check here (5.2 doesn't have this loop at all). I haven't seen the problem in ~5.2 without soft updates. > @@ -121,7 +120,7 @@ ufs_inactive(ap) > } > if (ip->i_effnlink == 0 && DOINGSOFTDEP(vp)) > softdep_releasefile(ip); > - if (ip->i_nlink <= 0 && (vp->v_mount->mnt_flag & MNT_RDONLY) == 0) { > + if (ip->i_nlink <= 0 && !UFS_ISRONLY(ip->i_ump)) { > #ifdef QUOTA > if (!getinoquota(ip)) > (void)chkiq(ip, -1, NOCRED, FORCE); 5.2 only has this i_nlink check, before a v_write_suspend_wait() call, and without any r/o check. With the MNT_RDONLY check here, the truncation isn't done, which is clearly wrong. However, I think this isn't the main problem here. The truncation always does get done in simple test cases like "rm /f/a; mount -u -o ro /f" which show the problem. Then there is no race between the rm and the mount -- ufs_inactive() completes normally in the rm process before the mount process sets MNT_RDONLY. I think soft updates (or just sync?) calls ufs_inactive again later when MNT_RDONLY is set, but it should find nothing more to do then. Anyway, doesn't the vn_start_write() in ffs_mount() wait for all writers like the rm process to complete? Mount sets MNT_RDONLY before that, so there may be a problem before that. There is only a single vn_start_write() call in vfs_mount.c. It is interesting that this wraps ffs_unmount() which seems to work. Mount-update may need to wait for writers to go away before it does things even more than unmount, since it changes active mount flags like MNT_RDONLY. Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 11:57:24 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F140916A417 for ; Sat, 1 Dec 2007 11:57:24 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id EC38913C442 for ; Sat, 1 Dec 2007 11:57:24 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 881FC1CC079; Sat, 1 Dec 2007 03:37:50 -0800 (PST) Date: Sat, 1 Dec 2007 03:37:50 -0800 From: Jeremy Chadwick To: Johan =?iso-8859-1?Q?Str=F6m?= Message-ID: <20071201113750.GA81186@eos.sc1.parodius.com> References: <66A69F9D-E4C1-4647-AEE7-E6F18010A1A3@headweb.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <66A69F9D-E4C1-4647-AEE7-E6F18010A1A3@headweb.com> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: scrambled (gmirror) dmesg output X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 11:57:25 -0000 On Sat, Dec 01, 2007 at 12:16:45PM +0100, Johan Ström wrote: > Hello > Im playing with a new box running RELENG_7.0 from yesterday. I got two > discs with gmirror on ad[6|14]s1a and zfs-mirror on s1d. When o do > atacontrol detach ata7 (detach ad14), i get this in dmesG: > > (first time) > subdisk14: detached > ad14: detached > GEOM_MIRROR: Device gm1b: provider ad14s1bG dEiOsMc_oMnInReRcOtRe:d .De > vice gm1: provider ad14s1a disconnected. > > (second time, detaching again after reattach) > subdisk14: detached > ad14: detached > GEOMG_EMOIMR_RMOIRR:R ORD:e viDceev icgem 1bg:m 1p:r opvriodveird era > d1a4ds114bs 1dai sdciosncnoencnteecdt.ed. > > huh? :) Some print raceing or something? The problem isn't specific to GEOM or ZFS. It's a known issue with two kernel printf()s being called simultaneously. There are older threads discussing the issue. I can dig up URLs if you want to read them, but I don't have them available quickly... -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 16:07:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3146F16A418 for ; Sat, 1 Dec 2007 16:07:43 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.freebsd.org (Postfix) with ESMTP id AE3F813C45A for ; Sat, 1 Dec 2007 16:07:42 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail18.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lB1G7FvG008283 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 2 Dec 2007 03:07:24 +1100 Date: Sun, 2 Dec 2007 03:07:10 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans In-Reply-To: <20071201030305.G1170@besplex.bde.org> Message-ID: <20071202020213.I2849@besplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <20071201030305.G1170@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 16:07:43 -0000 On Sat, 1 Dec 2007, Bruce Evans wrote: > On Fri, 30 Nov 2007, Kostik Belousov wrote: >> As a speculation, it might be that ufs_inactive() should conditionalize on >> fs_ronly instead of MNT_RDONLY. Then, VOP_INACTIVE() would set up the >> IN_CHANGE|IN_UPDATE and finally call the ffs_update() ? > > Something like that seems to be right. The folowing hack in ufs_inactive() > seems to fix the problem with sift updates, as does unsetting MNT_RDONLY > for the whole VOP_SYNC() in ffs_mount(). > > % Index: ufs_inode.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_inode.c,v > % retrieving revision 1.53 > % diff -u -2 -r1.53 ufs_inode.c > % --- ufs_inode.c 7 Apr 2004 03:47:20 -0000 1.53 > % +++ ufs_inode.c 30 Nov 2007 12:58:39 -0000 > % @@ -59,4 +59,6 @@ > % #endif > % % +#include > % + > % /* > % * Last reference to an inode. If necessary, write or delete it. > % @@ -118,6 +120,15 @@ > % ip->i_flag &= ~IN_ACCESS; > % } else { > % + int wasro = 0; > % + > % (void) vn_write_suspend_wait(vp, NULL, V_WAIT); > % + if (vp->v_mount->mnt_flag & MNT_RDONLY && > % + ip->i_fs->fs_ronly == 0) { > % + vp->v_mount->mnt_flag &= ~MNT_RDONLY; > % + wasro = 1; > % + } > % UFS_UPDATE(vp, 0); > % + if (wasro) > % + vp->v_mount->mnt_flag |= MNT_RDONLY; > % } > % } > > I didn't bother with correct locking here (only tested under UP). > > With this change, the VOP_SYNC() in ffs_mount() for MNT_UPDATE seems > to flush everything in simple cases (with no open files), just like > the VOP_SYNC() in unmount() flushes everything before ffs_unmount() is > reached. OTOH, without a forced flush, soft updates takes a long > time to flush things -- more like 3 syncer periods than 1 for > non-waitfor syncs. With soft updates, the above is called from deep > in VOP_SYNC(). It's strange that the above non-waitfor UFS_UPDATE() > is used inside of waitfor syncs. It apparently works because the > waitfor syncs wait for it later, but only if it is non-null. Here is a non-hackish patch which explains why ignoring MNT_RDONLY in the above or in ffs_mount() helps. It just fixes the confusion between IN_MODIFIED and IN_CHANGE in critical places. % Index: ffs_softdep.c % =================================================================== % RCS file: /home/ncvs/src/sys/ufs/ffs/ffs_softdep.c,v % retrieving revision 1.214 % diff -u -2 -r1.214 ffs_softdep.c % --- ffs_softdep.c 9 Nov 2007 11:04:36 -0000 1.214 % +++ ffs_softdep.c 1 Dec 2007 14:25:49 -0000 % @@ -2776,5 +2776,5 @@ % DIP_SET(ip, i_blocks, DIP(ip, i_blocks) + \ % freeblks->fb_chkcnt - blocksreleased); % - ip->i_flag |= IN_CHANGE; % + ip->i_flag |= IN_MODIFIED; % vput(vp); % } % @@ -3575,5 +3575,5 @@ % ip->i_nlink--; % DIP_SET(ip, i_nlink, ip->i_nlink); % - ip->i_flag |= IN_CHANGE; % + ip->i_flag |= IN_MODIFIED; % if (ip->i_nlink < ip->i_effnlink) % panic("handle_workitem_remove: bad file delta"); % @@ -3594,5 +3594,5 @@ % ip->i_nlink -= 2; % DIP_SET(ip, i_nlink, ip->i_nlink); % - ip->i_flag |= IN_CHANGE; % + ip->i_flag |= IN_MODIFIED; % if (ip->i_nlink < ip->i_effnlink) % panic("handle_workitem_remove: bad dir delta"); % @@ -3635,5 +3635,5 @@ % WORKLIST_INSERT(&inodedep->id_inowait, &dirrem->dm_list); % FREE_LOCK(&lk); % - ip->i_flag |= IN_CHANGE; % + ip->i_flag |= IN_MODIFIED; % ffs_update(vp, 0); % vput(vp); Without this change, soft updates depends on IN_CHANGE being converted to IN_MODIFIED by ufs_itimes(), but this conversion doesn't happen when MNT_RDONLY is set. With soft updates, changes are often delayed until sync time, and when the sync is for mount-update it is done after setting MNT_RDONLY so the above doesn't work. Soft updates sometimes forces an inode update using UFS_UPDATE(..., 1) but it doesn't do this in all cases even for MNT_UPDATE syncs. The update is first done asynchronously (except for the bug) by setting IN_* in the above so that ffs_update() is (supposed to be) non-null when it is called by ufs_inactive() or directly in 1 case in the above. Then for MNT_UPDATE cases, something should wait later. This seems to be broken for some cases but not for mount-update or unmount. Setting IN_CHANGE in the above also breaks ctimes slightly. I think the bug is visible from userland in some cases. Consider the operarations rename; stat; sync; sync. The mv sets update marks; the stat (actually ufs_inactive() at the end of the rename if the file is not open) converts the marks to timestamps that should be final; then the sync runs the above code which sets the ctime mark again, and one of the syncs (should be the first one, but that seems to be broken) converts the ctime mark to a timestamp and thus breaks the timestamp already reported to userland. The problem area can be investigated without creating corrupt file systems by using fsync(2): on a new file system /f: touch a while :; do mv a b fsync b # optionally sleep a short time 0.1-0.5 seconds fsck -n /f mv b a unmount /f; mount /f # Don't trust sync done I watched the mv and the fsync using ddb and the fsck. With soft updates, (even with sync mounts) the fsync doesn't actually complete the i/o. Ddb shows that the work list is not completed when fsync returns, and the fsck shows that the file system is inconsistent when the fsck returns. After sleeping just a fraction of a second, the fsck shows a consistent file system. 0.1 seconds usually isn't enough, but 0.5 seconds usually is. I think it is softdep_waitidle() that is waiting long enough for the unmount and remount cases to work. Without soft updates (even with async mounts), the fsync seems to work correctly -- no sleep is needed for the fsck to usually show a consistent fs -- though in the async case this just because no i/o has been done in the usual case, and in the sync case the fs is consistent without the fsync. Without the above patch, after "mv a b", "fsync b" is insufficient to make "mount -u -o ro /f" not corrupt the fs (because the mv generates 3 or 4 entries in the worklist, and the fsync generates 0 or 1 more and always ends up with 1 or 2 not handled, and then the sync for mount-update usually doesn't make any more progress). I didn't check if a short delay after the fsync is sufficient, but expect that it is. With the above patch, a loop doing "mv a b; mount -u -o ro /f; fsck -n /f; mount -u -o rw /f; mv b a" shows no problems. However, the original problem was without soft updates, so there must be more bugs here. > BTW, *_mount() has lots of bogusness related to string options. In > particular, ffs_mount() for update from r/w to r/o checks the "ro" > string option and sets MNT_RDONLY later, but MNT_RDONLY is already set > and other things depend on it being set early. Setting MNT_RDONLY too early can probably cause critical updates to be lost in the non-soft updates case due to the IN_CHANGE/IN_MODIFIED confusion, but only in cases where mount races with other operations. While testing soft updates without any patches, I got a panic (dup alloc?) immediately when I raced the mv+remount loop (or was it just mv+fsync?) with another loop doing just remount. Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 18:33:44 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB0F716A417 for ; Sat, 1 Dec 2007 18:33:44 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail34.syd.optusnet.com.au (mail34.syd.optusnet.com.au [211.29.133.218]) by mx1.freebsd.org (Postfix) with ESMTP id 5372E13C44B for ; Sat, 1 Dec 2007 18:33:43 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail34.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lB1IXdNs000991 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 2 Dec 2007 05:33:41 +1100 Date: Sun, 2 Dec 2007 05:33:39 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans In-Reply-To: <20071202020213.I2849@besplex.bde.org> Message-ID: <20071202043649.I1094@besplex.bde.org> References: <474F4E46.8030109@nokia.com> <20071130112043.H7217@besplex.bde.org> <474F69A7.9090404@nokia.com> <20071130151606.F12094@delplex.bde.org> <20071130052840.GH83121@deviant.kiev.zoral.com.ua> <20071201030305.G1170@besplex.bde.org> <20071202020213.I2849@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 18:33:44 -0000 On Sun, 2 Dec 2007, Bruce Evans wrote: > Here is a non-hackish patch which explains why ignoring MNT_RDONLY in > the above or in ffs_mount() helps. It just fixes the confusion between > IN_MODIFIED and IN_CHANGE in critical places. > > % Index: ffs_softdep.c > ... > > Without this change, soft updates depends on IN_CHANGE being converted > to IN_MODIFIED by ufs_itimes(), but this conversion doesn't happen > when MNT_RDONLY is set. With soft updates, changes are often delayed > until sync time, and when the sync is for mount-update it is done after > setting MNT_RDONLY so the above doesn't work. > ... > Without the above patch, after "mv a b", "fsync b" is insufficient to > make "mount -u -o ro /f" not corrupt the fs (because the mv generates > 3 or 4 entries in the worklist, and the fsync generates 0 or 1 more > and always ends up with 1 or 2 not handled, and then the sync for > mount-update usually doesn't make any more progress). I didn't check > if a short delay after the fsync is sufficient, but expect that it is. > With the above patch, a loop doing "mv a b; mount -u -o ro /f; fsck > -n /f; mount -u -o rw /f; mv b a" shows no problems. However, the > original problem was without soft updates, so there must be more bugs > here. The above seems to fix "mv a b", but for "rm a" it only restores the previous symptoms of the bug -- the "failed to flush worklist" error is gone but the "update error" is back. The following simiilar patch seems to fix "rm a" in 5.2: % Index: ufs_inode.c % =================================================================== % RCS file: /home/ncvs/src/sys/ufs/ufs/ufs_inode.c,v % retrieving revision 1.53 % diff -u -2 -r1.53 ufs_inode.c % --- ufs_inode.c 7 Apr 2004 03:47:20 -0000 1.53 % +++ ufs_inode.c 1 Dec 2007 17:26:30 -0000 % @@ -108,5 +111,5 @@ % ip->i_mode = 0; % DIP(ip, i_mode) = 0; % - ip->i_flag |= IN_CHANGE | IN_UPDATE; % + ip->i_flag |= IN_MODIFIED; % if (DOINGSOFTDEP(vp)) % softdep_change_linkcnt(ip); This is in ufs_inactive(). Soft updates calls back here from the sync. softdep_change_linkcount() just creates a dependency and depends on its callers to set IN_* flags which will cause i/o to flush the dependency. Here the caller sets flags that had no effect in the MNT_RDONLY case, the same as in the previous patch. I removed the IN_CHANGE and IN_UPDATE flags completely since they just clobber the previous times (set by a non-delayed ufs_inactive but probably already clobbered by a delayed truncation). (We are clearing out parts of the inode, so the times don't really matter, but unless something else clears the times it is better to leave the non-delayed times which record when the update marks for the userland remove were converted to timestamps.) Just before the above, the file is truncated. ffs_truncate() has the usual confusion between IN_CHANGE and IN_MODIFIED. It sets (IN_CHANGE | IN_UPDATE) in 7 different places, but never sets IN_MODIFIED. In my test case of "rm a; mount -u -o ro /f", this isn't a problem, because IN_MODIFIED is usually already set set ffs_truncate() doesn't forget to start i/o. ffs_truncate() calls ffs_update() with waitfor == 1 in many cases, but in my test case it always calls ffs_update() with waitfor == 0. Any call to ffs_update() in it cancels any previous setting of IN_MODIFIED when modifications are moved to the buffer. Thus when we forget to set the IN_MODIFIED in the above code, in the MNT_RDONLY case we always have inconsistencies: - non-soft updates case: we have at least a cleared i_rdev and i_mode with no means to write them. Is this enough to cause the original problem? - soft updates case: we also have a dependency with no means to flush it. I think this change doesn't help for -current yet, since this code is unreachable when MNT_RDONLY is set. The truncation code is also unreachable. kib's patch makes both reachable again. I observe/predict/explain the following error behaviours for soft updates (still some guesses): -current the first patch: "/f: update error: blocks 32 files 1" (because the truncation and other things aren't done, but worklist items aren't created for them so the problem isn't detected in softdep_waitidle). -5.2 with first patch: "/f: update error: blocks 0 files 1" (because the truncation works accidentally, and softdep_waitidle() doesn't exist to detect the unflushable worklist item created by the reachable softdep_change_linkcnt() call). Panic later (due to broken worklist after a buggy remount is permitted?). -current with first patch and kib's patch: "softdep_waitidle: ..." (because the bug fixed in the second patch is exposed again and softdep_waitidle() detects it). Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 22:07:59 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC3F316A46E for ; Sat, 1 Dec 2007 22:07:59 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (adsl-75-1-14-242.dsl.scrm01.sbcglobal.net [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 806AD13C46A for ; Sat, 1 Dec 2007 22:07:59 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id lB1M7oNg015468; Sat, 1 Dec 2007 14:07:54 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200712012207.lB1M7oNg015468@gw.catspoiler.org> Date: Sat, 1 Dec 2007 14:07:50 -0800 (PST) From: Don Lewis To: brde@optusnet.com.au In-Reply-To: <20071201215706.B12006@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 22:07:59 -0000 On 1 Dec, Bruce Evans wrote: > On Sat, 1 Dec 2007, Kostik Belousov wrote: >> +static int >> +ffs_isronly(struct ufsmount *ump) >> +{ >> + struct fs *fs = ump->um_fs; >> + >> + return (fs->fs_ronly); >> +} >> + > > Could be ump->um_fs->fs_ronly. That's the change that I would have made. A #include for would have to be added, which some might argue would be a layering violation. I'd prefer to avoid the extra indirection. > Anyway, doesn't the vn_start_write() in ffs_mount() wait for all writers > like the rm process to complete? vn_start_write() is used to block writes during certain stages of snapshot creation. From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 22:14:54 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F78C16A417 for ; Sat, 1 Dec 2007 22:14:54 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (adsl-75-1-14-242.dsl.scrm01.sbcglobal.net [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 35AF613C4CE for ; Sat, 1 Dec 2007 22:14:54 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id lB1MEl2Q015881; Sat, 1 Dec 2007 14:14:50 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200712012214.lB1MEl2Q015881@gw.catspoiler.org> Date: Sat, 1 Dec 2007 14:14:47 -0800 (PST) From: Don Lewis To: brde@optusnet.com.au In-Reply-To: <20071202020213.I2849@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org Subject: Re: File remove problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 22:14:54 -0000 On 2 Dec, Bruce Evans wrote: > Here is a non-hackish patch which explains why ignoring MNT_RDONLY in > the above or in ffs_mount() helps. It just fixes the confusion between > IN_MODIFIED and IN_CHANGE in critical places. > > % Index: ffs_softdep.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/ufs/ffs/ffs_softdep.c,v > % retrieving revision 1.214 > % diff -u -2 -r1.214 ffs_softdep.c > % --- ffs_softdep.c 9 Nov 2007 11:04:36 -0000 1.214 > % +++ ffs_softdep.c 1 Dec 2007 14:25:49 -0000 > % @@ -2776,5 +2776,5 @@ > % DIP_SET(ip, i_blocks, DIP(ip, i_blocks) + \ > % freeblks->fb_chkcnt - blocksreleased); > % - ip->i_flag |= IN_CHANGE; > % + ip->i_flag |= IN_MODIFIED; > % vput(vp); > % } > % @@ -3575,5 +3575,5 @@ > % ip->i_nlink--; > % DIP_SET(ip, i_nlink, ip->i_nlink); > % - ip->i_flag |= IN_CHANGE; > % + ip->i_flag |= IN_MODIFIED; > % if (ip->i_nlink < ip->i_effnlink) > % panic("handle_workitem_remove: bad file delta"); > % @@ -3594,5 +3594,5 @@ > % ip->i_nlink -= 2; > % DIP_SET(ip, i_nlink, ip->i_nlink); > % - ip->i_flag |= IN_CHANGE; > % + ip->i_flag |= IN_MODIFIED; > % if (ip->i_nlink < ip->i_effnlink) > % panic("handle_workitem_remove: bad dir delta"); > % @@ -3635,5 +3635,5 @@ > % WORKLIST_INSERT(&inodedep->id_inowait, &dirrem->dm_list); > % FREE_LOCK(&lk); > % - ip->i_flag |= IN_CHANGE; > % + ip->i_flag |= IN_MODIFIED; > % ffs_update(vp, 0); > % vput(vp); > > Without this change, soft updates depends on IN_CHANGE being converted > to IN_MODIFIED by ufs_itimes(), but this conversion doesn't happen > when MNT_RDONLY is set. With soft updates, changes are often delayed > until sync time, and when the sync is for mount-update it is done after > setting MNT_RDONLY so the above doesn't work. ufs_itimes() should probably also be looking at fs_ronly instead of MNT_RDONLY, *but* all the paths leading from userland to ufs_itimes() would need to be checked to verify that they check MNT_RDONLY to prevent new file system write operations from happening while the remount is in progress.