From owner-freebsd-questions@FreeBSD.ORG Sun Aug 29 13:19:05 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0FE2016A4CE; Sun, 29 Aug 2004 13:19:05 +0000 (GMT) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id D951E43D39; Sun, 29 Aug 2004 13:19:04 +0000 (GMT) (envelope-from pgiessel@mac.com) Received: from mac.com (smtpin07-en2 [10.13.10.152]) by smtpout.mac.com (Xserve/MantshX 2.0) with ESMTP id i7TDJ4Jd020365; Sun, 29 Aug 2004 06:19:04 -0700 (PDT) Received: from [63.125.5.200] ([63.125.5.200]) (authenticated bits=0) by mac.com (Xserve/smtpin07/MantshX 4.0) with ESMTP id i7TDJ2We018648; Sun, 29 Aug 2004 06:19:04 -0700 (PDT) Date: Sun, 29 Aug 2004 06:19:54 -0700 From: "Peter A. Giessel" To: "Greg 'groggy' Lehey" , freebsd-questions@freebsd.org X-Priority: 3 In-Reply-To: <20040823004213.GX92256@wantadilla.lemis.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-Mailer: Mailsmith 2.1.2 (Blindsider) Subject: Re: vinum, fsck generates errors when Plex 1 is up X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Aug 2004 13:19:05 -0000 > * What problems are you having? fsck reports dozens and dozens of errors and the filesystem as corrupt whenever I start Plex 1. > * Which version of FreeBSD are you running? 5.2.1-RELEASE > * Have you made any changes to the system sources, including Vinum? No, I recompiled the kernel a couple of times, most recently to remove USB so that the following stops (and after removing USB it did stop): Aug 18 09:07:32 raid51 kernel: usb0: 1 scheduling overruns Aug 18 09:07:55 raid51 last message repeated 465 times > * Supply the output of the vinum list command. If you can't start > Vinum, supply the on-disk configuration, as described below. If you > can't start Vinum, then (and only then) send a copy of the > configuration file. It is currently as follows, after # vinum stop array.p1 12 drives: D two State: up /dev/ad18s1h A: 0/190732 MB (0%) D one State: up /dev/ad16s1h A: 0/190732 MB (0%) D four State: up /dev/ad15s1h A: 47/190779 MB (0%= ) D five State: up /dev/ad14s1h A: 47/190843 MB (0%= ) D three State: up /dev/ad12s1h A: 47/190843 MB (0%= ) D twelve State: up /dev/ad11s1h A: 47/190779 MB (0%= ) D ten State: up /dev/ad10s1h A: 47/190779 MB (0%= ) D eleven State: up /dev/ad8s1h A: 47/190779 MB (0%= ) D eight State: up /dev/ad7s1h A: 47/190779 MB (0%= ) D nine State: up /dev/ad6s1h A: 47/190843 MB (0%= ) D seven State: up /dev/ad5s1h A: 0/190732 MB (0%) D six State: up /dev/ad4s1h A: 0/190732 MB (0%) 1 volumes: V array State: up Plexes: 2 Size: 931 GB 2 plexes: P array.p0 R5 State: up Subdisks: 6 Size: 931 GB P array.p1 R5 State: faulty Subdisks: 6 Size: 931 GB 12 subdisks: S array.p0.s0 State: up D: one Size: 186 GB S array.p0.s1 State: up D: two Size: 186 GB S array.p0.s2 State: up D: three Size: 186 GB S array.p0.s3 State: up D: four Size: 186 GB S array.p0.s4 State: up D: five Size: 186 GB S array.p0.s5 State: up D: eleven Size: 186 GB S array.p1.s0 State: obsolete D: six Size: 186 GB S array.p1.s1 State: obsolete D: seven Size: 186 GB S array.p1.s2 State: down D: eight Size: 186 GB S array.p1.s3 State: down D: nine Size: 186 GB S array.p1.s4 State: down D: ten Size: 186 GB S array.p1.s5 State: down D: twelve Size: 186 GB The following was the "vinum l" from after "start array.p1" finished: 12 drives: D two State: up /dev/ad18s1h A: 0/190732 MB (0%) D one State: up /dev/ad16s1h A: 0/190732 MB (0%) D four State: up /dev/ad15s1h A: 47/190779 MB (0%= ) D five State: up /dev/ad14s1h A: 47/190843 MB (0%= ) D three State: up /dev/ad12s1h A: 47/190843 MB (0%= ) D twelve State: up /dev/ad11s1h A: 47/190779 MB (0%= ) D ten State: up /dev/ad10s1h A: 47/190779 MB (0%= ) D eleven State: up /dev/ad8s1h A: 47/190779 MB (0%= ) D eight State: up /dev/ad7s1h A: 47/190779 MB (0%= ) D nine State: up /dev/ad6s1h A: 47/190843 MB (0%= ) D seven State: up /dev/ad5s1h A: 0/190732 MB (0%) D six State: up /dev/ad4s1h A: 0/190732 MB (0%) 1 volumes: V array State: up Plexes: 2 Size: 931 GB 2 plexes: P array.p0 R5 State: up Subdisks: 6 Size: 931 GB P array.p1 R5 State: up Subdisks: 6 Size: 931 GB 12 subdisks: S array.p0.s0 State: up D: one Size: 186 GB S array.p0.s1 State: up D: two Size: 186 GB S array.p0.s2 State: up D: three Size: 186 GB S array.p0.s3 State: up D: four Size: 186 GB S array.p0.s4 State: up D: five Size: 186 GB S array.p0.s5 State: up D: eleven Size: 186 GB S array.p1.s0 State: up D: six Size: 186 GB S array.p1.s1 State: up D: seven Size: 186 GB S array.p1.s2 State: up D: eight Size: 186 GB S array.p1.s3 State: up D: nine Size: 186 GB S array.p1.s4 State: up D: ten Size: 186 GB S array.p1.s5 State: up D: twelve Size: 186 GB > * Supply an extract of the Vinum history file. Unless you have > explicitly renamed it, it will be /var/log/vinum_history. This file > can get very big; please limit it to the time around when you have > the problems. Each line contains a timestamp at the beginning, so you > will have no difficulty in establishing which data is of relevance. Just to be sure that it wasn't flakey drives, I replaced the two drives that were on the same IDE channel: 25 Aug 2004 13:53:15.075384 *** vinum started *** 25 Aug 2004 13:53:17.055387 l=20 25 Aug 2004 13:53:42.492432 create configfile=20 drive ten device /dev/ad10s1h drive twelve device /dev/ad11s1h 25 Aug 2004 13:54:03.764526 quit=20 27 Aug 2004 12:46:11.938029 *** vinum started *** 27 Aug 2004 12:46:13.722421 l=20 27 Aug 2004 12:46:23.610499 start array.p1=20 27 Aug 2004 12:46:35.272605 l=20 27 Aug 2004 12:46:23.610499 start array.p1=20 27 Aug 2004 12:46:23.610499 start array.p1=20 27 Aug 2004 12:46:23.610499 start array.p1=20 27 Aug 2004 12:46:23.610499 start array.p1=20 28 Aug 2004 04:51:14.991231 l=20 28 Aug 2004 04:54:17.339258 l=20 28 Aug 2004 07:59:00.619752 l=20 28 Aug 2004 08:03:40.666383 l=20 28 Aug 2004 08:04:48.512668 l=20 27 Aug 2004 12:46:23.610499 start array.p1=20 27 Aug 2004 12:46:23.610499 start array.p1=20 28 Aug 2004 08:33:03.552759 l=20 28 Aug 2004 08:33:11.259411 quit=20 28 Aug 2004 08:33:48.425298 *** vinum started *** 28 Aug 2004 08:33:48.427352 l=20 28 Aug 2004 08:59:04.430760 *** vinum started *** 28 Aug 2004 08:59:04.432800 l=20 28 Aug 2004 08:59:20.222726 *** vinum started *** 28 Aug 2004 08:59:20.224762 stop array.p1=20 > * Supply an extract of the file /var/log/messages. Restrict the > extract to the same time frame as the history file. Again, each line > contains a timestamp at the beginning, so you will have no difficulty > in establishing which data is of relevance. Aug 25 21:19:58 raid51 login: ROOT LOGIN (root) ON ttyv0 Aug 25 21:20:04 raid51 reboot: rebooted by root Aug 25 21:20:04 raid51 syslogd: exiting on signal 15 Aug 25 21:55:08 raid51 syslogd: kernel boot file is /boot/kernel/kernel Aug 25 21:55:08 raid51 kernel: Copyright (c) 1992-2004 The FreeBSD Project. Aug 25 21:55:08 raid51 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, = 1989, 1991, 1992, 1993, 1994 Aug 25 21:55:08 raid51 kernel: The Regents of the University of California.= All rights reserved. Aug 25 21:55:08 raid51 kernel: FreeBSD 5.2.1-RELEASE #0: Thu Aug 19 09:33:0= 0 AKDT 2004 The Vinum timestamps were in single user mode (GMT), the other timestamps are in AKDT (GMT+8), so that log extract covers the first vinum period above. Aug 27 19:14:13 raid51 login: ROOT LOGIN (root) ON ttyv0 Aug 27 19:14:16 raid51 reboot: rebooted by root Aug 27 19:14:16 raid51 syslogd: exiting on signal 15 Aug 28 18:09:09 raid51 syslogd: kernel boot file is /boot/kernel/kernel Aug 28 18:09:09 raid51 kernel: Copyright (c) 1992-2004 The FreeBSD Project. Aug 28 18:09:09 raid51 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, = 1989, 1991, 1992, 1993, 1994 Aug 28 18:09:09 raid51 kernel: The Regents of the University of California.= All rights reserved. Aug 28 18:09:09 raid51 kernel: FreeBSD 5.2.1-RELEASE #0: Thu Aug 19 09:33:0= 0 AKDT 2004 Aug 28 18:09:09 raid51 kernel: root@raid51.dot.state.ak.us:/usr/obj/usr/src= /sys/NOUSB Aug 28 18:09:09 raid51 kernel: Preloaded elf kernel "/boot/kernel/kernel" a= t 0xc0ac5000. Aug 28 18:09:09 raid51 kernel: Preloaded elf module "/boot/kernel/vinum.ko"= at 0xc0ac5294. Aug 28 18:09:09 raid51 kernel: Preloaded elf module "/boot/kernel/acpi.ko" = at 0xc0ac5340. Aug 28 18:09:09 raid51 kernel: ACPI APIC Table: Aug 28 18:09:09 raid51 kernel: Timecounter "i8254" frequency 1193182 Hz qua= lity 0 Aug 28 18:09:09 raid51 kernel: CPU: AMD Athlon(tm) MP 2400+ (2000.08-MHz 68= 6-class CPU) Aug 28 18:09:09 raid51 kernel: Origin =3D "AuthenticAMD" Id =3D 0x681 Ste= pping =3D 1 Aug 28 18:09:09 raid51 kernel: Features=3D0x383fbff Aug 28 18:09:09 raid51 kernel: AMD Features=3D0xc0480000 Aug 28 18:09:09 raid51 kernel: real memory =3D 1073217536 (1023 MB) Aug 28 18:09:09 raid51 kernel: avail memory =3D 1033003008 (985 MB) Aug 28 18:09:09 raid51 kernel: ioapic0 irqs 0-23 on motherboa= rd Aug 28 18:09:09 raid51 kernel: Pentium Pro MTRR support enabled Aug 28 18:09:09 raid51 kernel: npx0: [FAST] Aug 28 18:09:09 raid51 kernel: npx0: on motherboard Aug 28 18:09:09 raid51 kernel: npx0: INT 16 interface Aug 28 18:09:09 raid51 kernel: acpi0: on motherboard Aug 28 18:09:09 raid51 kernel: pcibios: BIOS version 2.10 Aug 28 18:09:09 raid51 kernel: Using $PIR table, 14 entries at 0xc00fdee0 Aug 28 18:09:09 raid51 kernel: acpi0: Power Button (fixed) Aug 28 18:09:09 raid51 kernel: acpi0: Sleep Button (fixed) Aug 28 18:09:09 raid51 kernel: Timecounter "ACPI-safe" frequency 3579545 Hz= quality 1000 Aug 28 18:09:09 raid51 kernel: acpi_timer0: <24-bit timer at 3.579545MHz> p= ort 0x8008-0x800b on acpi0 Aug 28 18:09:09 raid51 kernel: acpi_cpu0: on acpi0 Aug 28 18:09:09 raid51 kernel: acpi_cpu1: on acpi0 Aug 28 18:09:09 raid51 kernel: device_probe_and_attach: acpi_cpu1 attach re= turned 6 Aug 28 18:09:09 raid51 kernel: acpi_button0: on acpi0 Aug 28 18:09:09 raid51 kernel: pcib0: port 0x8080-0x= 80ff,0x8000-0x807f,0xcf8-0xcff iomem 0xd8000-0xdbfff on acpi0 Aug 28 18:09:09 raid51 kernel: pci0: on pcib0 Aug 28 18:09:09 raid51 kernel: agp0: port 0x10= 60-0x1063 mem 0xe8500000-0xe8500fff,0xec000000-0xefffffff at device 0.0 on = pci0 Aug 28 18:09:09 raid51 kernel: pcib1: at device 1.0 o= n pci0 Aug 28 18:09:09 raid51 kernel: pci1: on pcib1 Aug 28 18:09:09 raid51 kernel: pci1: at device 5.0 (no drive= r attached) Aug 28 18:09:09 raid51 kernel: isab0: at device 7.0 on pci= 0 Aug 28 18:09:09 raid51 kernel: isa0: on isab0 Aug 28 18:09:09 raid51 kernel: atapci0: port 0= xf000-0xf00f at device 7.1 on pci0 Aug 28 18:09:09 raid51 kernel: ata0: at 0x1f0 irq 14 on atapci0 Aug 28 18:09:09 raid51 kernel: ata0: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata1: at 0x170 irq 15 on atapci0 Aug 28 18:09:09 raid51 kernel: ata1: [MPSAFE] Aug 28 18:09:09 raid51 kernel: pci0: at device 7.3 (n= o driver attached) Aug 28 18:09:09 raid51 kernel: atapci1: port = 0x1050-0x105f,0x1064-0x1067,0x1068-0x106f,0x1070-0x1073,0x1078-0x107f mem 0= xe8020000-0xe80200ff irq 21 at device 9.0 on pci0 Aug 28 18:09:09 raid51 kernel: atapci1: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata2: at 0xe8020000 on atapci1 Aug 28 18:09:09 raid51 kernel: ata2: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata3: at 0xe8020000 on atapci1 Aug 28 18:09:09 raid51 kernel: ata3: [MPSAFE] Aug 28 18:09:09 raid51 kernel: em0: port 0x1000-0x103f mem 0xe8000000-0xe801ffff irq 23 at de= vice 11.0 on pci0 Aug 28 18:09:09 raid51 kernel: em0: Speed:N/A Duplex:N/A Aug 28 18:09:09 raid51 kernel: pcib2: at device 16.0 = on pci0 Aug 28 18:09:09 raid51 kernel: pci2: on pcib2 Aug 28 18:09:09 raid51 kernel: pci2: at device 0.0 (no dr= iver attached) Aug 28 18:09:09 raid51 kernel: atapci2: port = 0x3040-0x304f,0x3070-0x3073,0x3078-0x307f,0x3074-0x3077,0x3080-0x3087 mem 0= xe8222000-0xe82220ff irq 16 at device 4.0 on pci2 Aug 28 18:09:09 raid51 kernel: atapci2: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata4: at 0xe8222000 on atapci2 Aug 28 18:09:09 raid51 kernel: ata4: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata5: at 0xe8222000 on atapci2 Aug 28 18:09:09 raid51 kernel: ata5: [MPSAFE] Aug 28 18:09:09 raid51 kernel: atapci3: port = 0x3050-0x305f,0x3088-0x308b,0x3090-0x3097,0x308c-0x308f,0x3098-0x309f mem 0= xe8222400-0xe82224ff irq 17 at device 5.0 on pci2 Aug 28 18:09:09 raid51 kernel: atapci3: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata6: at 0xe8222400 on atapci3 Aug 28 18:09:09 raid51 kernel: ata6: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata7: at 0xe8222400 on atapci3 Aug 28 18:09:09 raid51 kernel: ata7: [MPSAFE] Aug 28 18:09:09 raid51 kernel: atapci4: port = 0x3060-0x306f,0x30a0-0x30a3,0x30a8-0x30af,0x30a4-0x30a7,0x30b0-0x30b7 mem 0= xe8222800-0xe82228ff irq 18 at device 6.0 on pci2 Aug 28 18:09:09 raid51 kernel: atapci4: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata8: at 0xe8222800 on atapci4 Aug 28 18:09:09 raid51 kernel: ata8: [MPSAFE] Aug 28 18:09:09 raid51 kernel: ata9: at 0xe8222800 on atapci4 Aug 28 18:09:09 raid51 kernel: ata9: [MPSAFE] Aug 28 18:09:09 raid51 kernel: fxp0: port 0x= 3000-0x303f mem 0xe8200000-0xe821ffff,0xe8221000-0xe8221fff irq 18 at devic= e 8.0 on pci2 Aug 28 18:09:09 raid51 kernel: fxp0: Ethernet address 00:e0:81:25:02:ab Aug 28 18:09:09 raid51 kernel: miibus0: on fxp0 Aug 28 18:09:09 raid51 kernel: inphy0: on m= iibus0 Aug 28 18:09:09 raid51 kernel: inphy0: 10baseT, 10baseT-FDX, 100baseTX, 10= 0baseTX-FDX, auto Aug 28 18:09:09 raid51 kernel: atkbdc0: port = 0x64,0x60 irq 1 on acpi0 Aug 28 18:09:09 raid51 kernel: atkbd0: flags 0x1 irq 1 on atk= bdc0 Aug 28 18:09:09 raid51 kernel: kbd0 at atkbd0 Aug 28 18:09:09 raid51 kernel: sio0 port 0x3f8-0x3ff irq 4 on acpi0 Aug 28 18:09:09 raid51 kernel: sio0: type 16550A Aug 28 18:09:09 raid51 kernel: sio1 port 0x2f8-0x2ff irq 3 on acpi0 Aug 28 18:09:09 raid51 kernel: sio1: type 16550A Aug 28 18:09:09 raid51 kernel: ppc0 port 0x778-0x77f,0x378-0x37f irq 7 drq = 3 on acpi0 Aug 28 18:09:09 raid51 kernel: ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) = in COMPATIBLE mode Aug 28 18:09:09 raid51 kernel: ppc0: FIFO with 16/16/9 bytes threshold Aug 28 18:09:09 raid51 kernel: ppbus0: on ppc0 Aug 28 18:09:09 raid51 kernel: plip0: on ppbus0 Aug 28 18:09:09 raid51 kernel: lpt0: on ppbus0 Aug 28 18:09:09 raid51 kernel: lpt0: Interrupt-driven port Aug 28 18:09:09 raid51 kernel: ppi0: on ppbus0 Aug 28 18:09:09 raid51 kernel: fdc0: port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0 Aug 28 18:09:09 raid51 kernel: acpi_cpu1: on acpi0 Aug 28 18:09:09 raid51 kernel: device_probe_and_attach: acpi_cpu1 attach re= turned 6 Aug 28 18:09:09 raid51 kernel: orm0: