Date: Wed, 3 May 2006 14:44:08 -0400 From: Kris Kennaway <kris@obsecurity.org> To: Peter Holm <peter@holm.cc> Cc: freebsd-fs@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: Stress testing the UFS2 filesystem Message-ID: <20060503184408.GA31172@xor.obsecurity.org> In-Reply-To: <18034.193.3.141.124.1146642890.squirrel@webmail7.pair.com> References: <20060502193900.GA94069@peter.osted.lan> <1541458526.20060503003229@merdin.com> <20060502221306.GD95348@xor.obsecurity.org> <44584421.3000807@cs.tu-berlin.de> <20060503072013.GA2926@xor.obsecurity.org> <18034.193.3.141.124.1146642890.squirrel@webmail7.pair.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--gBBFr7Ir9EOA20Yy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 03, 2006 at 09:54:50AM +0200, Peter Holm wrote: >=20 > > On Wed, May 03, 2006 at 07:48:17AM +0200, Bj?rn K?nig wrote: > >> Kris Kennaway schrieb: > >> >On Wed, May 03, 2006 at 12:32:29AM +0400, Pavel Merdine wrote: > >> >>Of course I think we could do patches to overcome corrupting panics, > >> >>but the core FreeBSD team would not accept this, as they are happy > >> >>with panics and corruptions they make to other filesystems. > >> > > >> >Of course not, don't make silly accusations :-) > >> > > >> >The problem is much more difficult to solve than "making the panic an > >> >error return". > >> > >> I'm interested in more information about this issue. Do you have a > >> reference to an old discussion about this topic or do you like to > >> explain it a little bit further for me (and probably others)? > > > > See the URL that Peter provided in his original post. > > > > The issue that he is testing is how well the filesystem behaves when > > you arbitrarily damage it and then run fsck (ideally, fsck should > > detect all of the damage and repair it). He seems to have found cases > > where fsck does not detect and repair the damage, leading to panics at > > runtime. > > >=20 > Actually the filesystems mounts without any problems if fsck is run first. >=20 > The objective of this exercise was to show that background fsck may lead > to panics. This was a problem I saw a lot a year ago when I did some > testing of patches and in the cause of a working day saw two or three > panics. With background fsck I would from time to time get a secondary > panic, which typically zapped the original crash dump. Oh, I misunderstood then :( Yes, this is pretty much to be expected: bg fsck depends on the (fairly strong) assumption that the only kinds of filesystem damage that are present at startup on filesystems with soft updates enabled are a) survivable (i.e. will not cause runtime problems before they are repaired), and b) may be repaired online. Modulo modern disk hardware violating these assumptions anyway, bg fsck more or less works as long as you only have "power failure" shutdowns. When your kernel panics instead (especially if it's a filesystem-related panic), all bets are off. With its dying breath, your kernel may decide to scribble all over your filesystem, causing arbitrary damage to it. Many of these types of damage are not "survivable", as you have demonstrated -- in fact the very existence of fsck is proof that the kernel is not designed to handle arbitrary damage at runtime. So the moral is that if your kernel is panicking a lot, turn off bg fsck or you'll probably hit other filesystem panics at runtime. I don't think you can completely prevent this problem, but one thing that may help would be for the kernel to attempt to write a marker to the dump device when it panics, and if this marker is present at boot time a fg fsck is forced. Of course the kernel will not always be able to do this, but it should work most of the time (since crashdumps usually work for most panics). Kris --gBBFr7Ir9EOA20Yy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEWPn4Wry0BWjoQKURAiiVAKCBDzQFis6Ett3TWstAMB2K63wu0QCg+VTm 9FQcMU9KZgpdm5yd+IfwJBM= =yycR -----END PGP SIGNATURE----- --gBBFr7Ir9EOA20Yy--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060503184408.GA31172>