From owner-freebsd-fs@FreeBSD.ORG Mon Jul 20 13:41:16 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1EF4D106564A for ; Mon, 20 Jul 2009 13:41:16 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id B30078FC18 for ; Mon, 20 Jul 2009 13:41:15 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: (qmail 35727 invoked from network); 20 Jul 2009 13:14:33 -0000 Received: from unknown (HELO ?10.0.0.135?) (spawk@128.238.64.31) by acm.poly.edu with AES256-SHA encrypted SMTP; 20 Jul 2009 13:14:33 -0000 Message-ID: <4A646DA8.2050201@acm.poly.edu> Date: Mon, 20 Jul 2009 09:14:16 -0400 From: Boris Kochergin User-Agent: Thunderbird 2.0.0.19 (X11/20090108) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: geom_mirror/UFS weirdness with 7.2-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2009 13:41:16 -0000 Ahoy. I noticed some very odd things in my file server's kernel buffer this morning (there were actually a ton of these--this is a snippet): Jul 20 05:54:10 exodus smartd[763]: Device: /dev/ad1, FAILED SMART self-check. BACK UP DATA NOW! Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=-4569735194538825728, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 8806809555123731765, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=-7379534036451358720, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 5076457091570183473, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=8010486245121781760, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 1859394416713520222, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8993345361309361786, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 5442079378433802039, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block -386473693423142096, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block -6538821972600747, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 7280706165078778427, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 2788813269397304649, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8324768912580846323, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8358350883763232034, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block The filesystem lives atop a two-disk geom_mirror: # gmirror status Name Status Components mirror/boot COMPLETE ad0 ad1 ...and the output of "df" on it also shows something odd: # df / Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mirror/boots1 37846636 -4058799239201906816 4058799239236725722 -11656883301279% / The system is a: # uname -a FreeBSD exodus.poly.edu 7.2-STABLE FreeBSD 7.2-STABLE #3: Sat Jul 11 16:22:02 EDT 2009 root@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Regarding smartd yelling at me about /dev/ad1, it's been doing that for long while before this. There is one sector on the drive that cannot be read, but the disk has otherwise been fine for months. My experience with geom_mirror has been that it disconnects members from an array if they experience I/O errors, so this seems to be something different. Any clues? -Boris