From owner-freebsd-fs@FreeBSD.ORG Sat Sep 21 10:48:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7AFCB434 for ; Sat, 21 Sep 2013 10:48:17 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 406F327D6 for ; Sat, 21 Sep 2013 10:48:17 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:5b:71c5:b3bc:c366]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 906854AC58; Sat, 21 Sep 2013 14:48:15 +0400 (MSK) Date: Sat, 21 Sep 2013 14:48:11 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <724152380.20130921144811@serebryakov.spb.ru> To: freebsd-fs Subject: Strange UFS write problem & SU+J "unexpected inconsistences" on 9.1-STABLE r253105 after it on OTHER filesystems. MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Kirk McKusick X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Sep 2013 10:48:17 -0000 Hello, freebsd-fs. My server paniced tonight with UFS problem: ufs/root[WRITE(offset=385499136, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385564672, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385712128, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=385826816, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=770703360, length=16384)]error = 22 g_vfs_done():ufs/root[WRITE(offset=770719744, length=16384)]error = 22 g_vfs_done():ufs/var[WRITE(offset=268539904, length=2048)]error = 22 /var: got error 22 while accessing filesystem panic: softdep_deallocate_dependencies: unrecovered I/O error cpuid = 0 KDB: stack backtrace: #0 0xffffffff8047a836 at kdb_backtrace+0x66 #1 0xffffffff8044382e at panic+0x1ce #2 0xffffffff8059c040 at clear_remove+0 #3 0xffffffff804bf835 at brelse+0x75 #4 0xffffffff804c2258 at bufdone+0x68 #5 0xffffffff804bcb0e at biodone+0xae #6 0xffffffff803e289c at g_io_schedule_up+0xac #7 0xffffffff803e2ffc at g_up_procbody+0x5c #8 0xffffffff804144ef at fork_exit+0x11f #9 0xffffffff805f53de at fork_trampoline+0xe and "fsck_ffs" refused to fix two OTHER (/usr and /tmp) SU+J-enabled FFSes with same messages: Journal file sequence mismatch XXX != YYY UNEXPECTED SU+J INCONSISTENCY INTERNAL ERROR: GOT TO reply() UNIXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY and exited with signal 11. So, here are two questions: (1) What does "error 22" mean? Disk doesn't show ANY errors in S.M.A.R.T. (and all internal tests are Ok). Also, here are NO ANY driver (AHCI) errors in post-mortem dump. It doesn't look like hardware problem. (2) How to avoid fsck refuses in such situations? Why OTHER (not ones with write errors) FSes get errors? It looks like one another problem with SU+J. Please note, these FSes reside directly on SATA drive, without any software or hardware RAIDs. I have dumped both FSes with "dumpfs" and "dumpfs -f" before manual check and have block-dumped /tmp (as it is small enough). You could find them at http://lev.serebryakov.spb.ru/FreeBSD/suj-crash/ -- // Black Lion AKA Lev Serebryakov