From owner-freebsd-bugs@FreeBSD.ORG Thu Mar 18 12:00:38 2004 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3598116A4CE for ; Thu, 18 Mar 2004 12:00:38 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18AF543D4C for ; Thu, 18 Mar 2004 12:00:34 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i2IK0Xbv004023 for ; Thu, 18 Mar 2004 12:00:33 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i2IK0X45004020; Thu, 18 Mar 2004 12:00:33 -0800 (PST) (envelope-from gnats) Resent-Date: Thu, 18 Mar 2004 12:00:33 -0800 (PST) Resent-Message-Id: <200403182000.i2IK0X45004020@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Dmitry Kazarov Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D896F16A4CE for ; Thu, 18 Mar 2004 11:51:12 -0800 (PST) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id CE2CE43D1F for ; Thu, 18 Mar 2004 11:51:12 -0800 (PST) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.12.10/8.12.10) with ESMTP id i2IJpC72075263 for ; Thu, 18 Mar 2004 11:51:12 -0800 (PST) (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.12.10/8.12.10/Submit) id i2IJpChd075261; Thu, 18 Mar 2004 11:51:12 -0800 (PST) (envelope-from nobody) Message-Id: <200403181951.i2IJpChd075261@www.freebsd.org> Date: Thu, 18 Mar 2004 11:51:12 -0800 (PST) From: Dmitry Kazarov To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Subject: kern/64433: vinum: mirrored volume stopped responding when one of it's subdisk failed to write X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Mar 2004 20:00:38 -0000 >Number: 64433 >Category: kern >Synopsis: vinum: mirrored volume stopped responding when one of it's subdisk failed to write >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Mar 18 12:00:33 PST 2004 >Closed-Date: >Last-Modified: >Originator: Dmitry Kazarov >Release: FreeBSD 4.9 STABLE >Organization: MIDIKON >Environment: FreeBSD serv.willcom.ru 4.9-STABLE FreeBSD 4.9-STABLE #1: Wed Mar 17 10:06:14 MSK 2004 root@serv.willcom.ru:/var/obj/usr/src/sys/SERV i386 >Description: Disk /dev/da1 (Ava) is known to have some surface defects. Dist /dev/da0 (Rex) has no defects. At the time of daily periodic scripts (3:01 am) alot of write ops failed on plex located on Ava (da1) and whole (!) volume became unavailable and server stopped responding because it's a /var file system. Server was revived by hardware reset (at 10:07 am). Kernel was alive (consoles switched, network worked), but Ctrl-Alt-Del did not worked. # vinum printconfig drive Rex device /dev/da0s1h drive Ava device /dev/da1s1h volume root volume usr volume var volume home plex name root.Rex org concat vol root plex name usr.Rex org concat vol usr plex name var.Rex org concat vol var plex name home.Rex org concat vol home plex name root.Ava org concat vol root plex name usr.Ava org concat vol usr plex name var.Ava org concat vol var plex name home.Ava org concat vol home sd name var.Ava.s1 drive Ava plex var.Ava len 4192256s driveoffset 1024s plexoffset 14680064s sd name root.Rex.s0 drive Rex plex root.Rex len 524288s driveoffset 4193280s plexoffset 0s sd name usr.Rex.s0 drive Rex plex usr.Rex len 8388608s driveoffset 4717568s plexoffset 0s sd name var.Rex.s0 drive Rex plex var.Rex len 14680064s driveoffset 13106176s plexoffset 0s sd name home.Rex.s0 drive Rex plex home.Rex len 8150078s driveoffset 27786240s plexoffset 0s sd name var.Rex.s1 drive Rex plex var.Rex len 4192256s driveoffset 1024s plexoffset 14680064s sd name root.Ava.s0 drive Ava plex root.Ava len 524288s driveoffset 4193280s plexoffset 0s sd name usr.Ava.s0 drive Ava plex usr.Ava len 8388608s driveoffset 4717568s plexoffset 0s sd name var.Ava.s0 drive Ava plex var.Ava len 14680064s driveoffset 13106176s plexoffset 0s sd name home.Ava.s0 drive Ava plex home.Ava len 8150078s driveoffset 27786240s plexoffset 0s >From /var/log/messages Mar 18 03:02:34 serv /kernel: vinum: var.Ava.s0 is stale by force Mar 18 03:02:34 serv /kernel: vinum: var.Ava is corrupt Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 20775744 for 4096 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 7669568 for 4096 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 21861376 for 4096 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 8755200 for 4096 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13106192 for 2048 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 16 for 2048 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13111808 for 2048 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 5632 for 2048 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13476452 for 4096 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 370276 for 4096 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14575680 for 16384 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1469504 for 16384 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14576384 for 16384 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1470208 for 16384 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14575776 for 16384 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1469600 for 16384 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 21860864 for 2048 bytes Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 8754688 for 2048 bytes Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13476364 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 370188 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14572704 for 8192 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1466528 for 8192 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573568 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467392 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573888 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467712 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14571072 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1464896 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14570208 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1464032 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577408 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1471232 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577312 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1471136 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577056 for 32768 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1470880 for 32768 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573696 for 32768 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467520 for 32768 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573856 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467680 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14572864 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1466688 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14575360 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1469184 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573440 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467264 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14576320 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1470144 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 22660312 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 9554136 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24048832 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10942656 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24048604 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10942428 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20786780 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7680604 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20776960 for 4096 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7670784 for 4096 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24050620 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10944444 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24051488 for 8192 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10945312 for 8192 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20779584 for 6144 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7673408 for 6144 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21860968 for 12288 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8754792 for 12288 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21884828 for 2048 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8778652 for 2048 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21881920 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8775744 for 16384 bytes Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 22611296 for 16384 bytes Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 9505120 for 16384 bytes Mar 18 03:02:38 serv /kernel: fatal :var.Ava.s0 write error, block 20775648 for 16384 bytes Mar 18 03:02:38 serv /kernel: var.Ava.s0: user buffer block 7669472 for 16384 bytes Mar 18 10:07:31 serv /kernel: vinum: root.Ava.s0 is stale by force Mar 18 10:07:31 serv /kernel: vinum: root.Ava is faulty Mar 18 10:07:31 serv /kernel: fatal :root.Ava.s0 write error, block 4578400 for 16384 bytes Mar 18 10:07:31 serv /kernel: root.Ava.s0: user buffer block 385120 for 16384 bytes Mar 18 10:07:31 serv /kernel: vinum: usr.Ava.s0 is crashed by force Mar 18 10:07:31 serv /kernel: vinum: usr.Ava is faulty Mar 18 10:07:31 serv /kernel: fatal:usr.Ava.s0 read error, block 11783340 for 2048 bytes Mar 18 10:07:31 serv /kernel: usr.Ava.s0: user buffer block 7065772 for 2048 bytes Mar 18 10:07:31 serv /kernel: Copyright (c) 1992-2003 The FreeBSD Project. >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: