Date: Sat, 21 Sep 2013 14:48:11 +0400 From: Lev Serebryakov <lev@FreeBSD.org> To: freebsd-fs <freebsd-fs@freebsd.org> Cc: Kirk McKusick <mckusick@mckusick.com> Subject: Strange UFS write problem & SU+J "unexpected inconsistences" on 9.1-STABLE r253105 after it on OTHER filesystems. Message-ID: <724152380.20130921144811@serebryakov.spb.ru>
next in thread | raw e-mail | index | archive | help
Hello, freebsd-fs. My server paniced tonight with UFS problem: ufs/root[WRITE(offset=385499136, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385564672, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385712128, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385826816, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=770703360, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=770719744, length=16384)]error = 22 g_vfs_done():ufs/var[WRITE(offset=268539904, length=2048)]error = 22 /var: got error 22 while accessing filesystem panic: softdep_deallocate_dependencies: unrecovered I/O error cpuid = 0 KDB: stack backtrace: #0 0xffffffff8047a836 at kdb_backtrace+0x66 #1 0xffffffff8044382e at panic+0x1ce #2 0xffffffff8059c040 at clear_remove+0 #3 0xffffffff804bf835 at brelse+0x75 #4 0xffffffff804c2258 at bufdone+0x68 #5 0xffffffff804bcb0e at biodone+0xae #6 0xffffffff803e289c at g_io_schedule_up+0xac #7 0xffffffff803e2ffc at g_up_procbody+0x5c #8 0xffffffff804144ef at fork_exit+0x11f #9 0xffffffff805f53de at fork_trampoline+0xe and "fsck_ffs" refused to fix two OTHER (/usr and /tmp) SU+J-enabled FFSes with same messages: Journal file sequence mismatch XXX != YYY UNEXPECTED SU+J INCONSISTENCY INTERNAL ERROR: GOT TO reply() UNIXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY and exited with signal 11. So, here are two questions: (1) What does "error 22" mean? Disk doesn't show ANY errors in S.M.A.R.T. (and all internal tests are Ok). Also, here are NO ANY driver (AHCI) errors in post-mortem dump. It doesn't look like hardware problem. (2) How to avoid fsck refuses in such situations? Why OTHER (not ones with write errors) FSes get errors? It looks like one another problem with SU+J. Please note, these FSes reside directly on SATA drive, without any software or hardware RAIDs. I have dumped both FSes with "dumpfs" and "dumpfs -f" before manual check and have block-dumped /tmp (as it is small enough). You could find them at http://lev.serebryakov.spb.ru/FreeBSD/suj-crash/ -- // Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?724152380.20130921144811>