From owner-freebsd-fs@FreeBSD.ORG Mon Jul 5 21:56:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE487106566B for ; Mon, 5 Jul 2010 21:56:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 576BB8FC0A for ; Mon, 5 Jul 2010 21:56:45 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o65LuDhV081499 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2010 00:56:13 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o65LuDxv094852; Tue, 6 Jul 2010 00:56:13 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o65LuCiA094851; Tue, 6 Jul 2010 00:56:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 6 Jul 2010 00:56:12 +0300 From: Kostik Belousov To: Charles Sprickman Message-ID: <20100705215612.GY13238@deviant.kiev.zoral.com.ua> References: <20100705213502.GX13238@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VkR3OQEnNfOLYsi4" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.1 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS, NUMERIC_HTTP_ADDR, URI_HEX autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: 7.2 - ufs2 corruption X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2010 21:56:48 -0000 --VkR3OQEnNfOLYsi4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 05, 2010 at 05:37:29PM -0400, Charles Sprickman wrote: > On Tue, 6 Jul 2010, Kostik Belousov wrote: >=20 > >On Mon, Jul 05, 2010 at 05:23:03PM -0400, Charles Sprickman wrote: > >>Howdy, > >> > >>I've posted previously about this, but I'm going to give it one more sh= ot > >>before I start reformatting and/or upgrading things. > >> > >>I have a largish filesystem (1.3TB) that holds a few jails, the main one > >>being a mail server. Running 7.2/amd64 on a Dell 2970 with the mfi > >>raid card, 6GB RAM, UFS2 (SU was enabled, I disabled it for testing to > >>no effect) > >> > >>The symptoms are as follows: > >> > >>Various applications will log messages about "bad file descriptors" (im= ap, > >>rsync backup script, quota counter): > >> > >>du: > >>./cur/1271801961.M21831P98582V0000005BI08E85975_0.foo.net,S=3D2824:2,S: > >>Bad file descriptor > >> > >>The kernel also starts logging messages like this to the console: > >> > >>g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384= )]error > >>=3D 5 > >>g_vfs_done():mfid0s1e[READ(offset=3D-7347040593908226048,=20 > >>length=3D16384)]error > >>=3D 5 > >>g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384= )]error > >>=3D 5 > >>g_vfs_done():mfid0s1e[READ(offset=3D-7347040593908226048,=20 > >>length=3D16384)]error > >>=3D 5 > >>g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384= )]error > >>=3D 5 > >> > >>Note that the offsets look a bit... suspicious, especially those negati= ve > >>ones. > >> > >>Usually within a day or two of those "g_vfs_done()" messages showing up > >>the box will panic shortly after the daily run. Things are hosed up > >>enough that it is unable to save a dump. The panic always looks like > >>this: > >> > >>panic: ufs_dirbad: /spool: bad dir ino 151699770 at offset 163920: mang= led > >>entry > >>cpuid =3D 0 > >>Uptime: 70d22h56m48s > >>Physical memory: 6130 MB > >>Dumping 811 MB: 796 780 764 748 732 716 700 684 668 652 636 620 604 588 > >>572 556 540 524 508 492 476 460 444 428 412 396 380 364 348 332 316 300 > >>284 > >>** DUMP FAILED (ERROR 16) ** > >> > >>panic: ufs_dirbad: /spool: bad dir ino 150073505 at offset 150: mangled > >>entry > >>cpuid =3D 2 > >>Uptime: 13d22h30m21s > >>Physical memory: 6130 MB > >>Dumping 816 MB: 801 785 769 753 737 721 705 689 > >>** DUMP FAILED (ERROR 16) ** > >>Automatic reboot in 15 seconds - press a key on the console to abort > >>Rebooting... > >> > >>The fs, specifically "/spool" (which is where the errors always > >>originate), will be pretty trashed and require a manual fsck. The first > >>pass finds/fixes errors, but does not mark the fs clean. It can take > >>anywhere from 2-4 passes to get a clean fs. > >> > >>The box then runs fine for a few weeks or a few months until the > >>"g_vfs_done" errors start popping up, then it's a repeat. > >> > >>Are there any *known* issues with either the fs or possibly the mfi dri= ver > >>in 7.2? > >> > >>My plan was to do something like this: > >> > >>-shut down services and copy all of /spool off to the backups server > >>-newfs /spool > >>-copy everything back > >> > >>Then if it continues, repeat the above with a 7.3 upgrade before running > >>newfs. > >> > >>If it still continues, then just go nuts and see what 8.0 or 8.1 does. > >>But I'd really like to avoid that. > >> > >>Any tips? > > > >Show "df -i" output for the the affected filesystem. >=20 > Here you go: >=20 > [spork@bigmail ~]$ df -i /spool > Filesystem 1K-blocks Used Avail Capacity iused ifree=20 > %iused Mounted on > /dev/mfid0s1g 1359086872 70105344 1180254580 6% 4691134 171006784=20 > 3% /spool I really expected to see the count of inodes on the fs to be bigger then 2G. It is not, but it is greater then 1G. Just to make sure: you do not get any messages from mfi(4) about disk errors ? You could try to format the partition with less inodes, see -i switch for the newfs. Make it less then 1G, and try your load again. The bug with handling volume with >2G inodes was fixed on RELENG_7 after 7.3 was released. Your simptoms are very similar to what happen when the bug is hit. --VkR3OQEnNfOLYsi4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwyVPwACgkQC3+MBN1Mb4ikUQCfVAxXmTgOafR2Sd7zHnxOVv5q sooAoLaDkKGEXq25REpgWLOacFoZ0zMn =EKli -----END PGP SIGNATURE----- --VkR3OQEnNfOLYsi4--