From owner-freebsd-stable@FreeBSD.ORG Thu Dec 30 00:22:53 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D099816A4CE for ; Thu, 30 Dec 2004 00:22:53 +0000 (GMT) Received: from relay01.roc.ny.frontiernet.net (relay01.roc.ny.frontiernet.net [66.133.131.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3985A43D46 for ; Thu, 30 Dec 2004 00:22:53 +0000 (GMT) (envelope-from drew@mykitchentable.net) Received: from filter01.roc.ny.frontiernet.net (filter01.roc.ny.frontiernet.net [66.133.131.176]) by relay01.roc.ny.frontiernet.net (Postfix) with ESMTP id 6920C10003 for ; Thu, 30 Dec 2004 00:22:52 +0000 (UTC) Received: from relay01.roc.ny.frontiernet.net ([66.133.131.34]) [66.133.131.176]) (amavisd-new, port 10024) with LMTP id 06494-02-75 for ; Thu, 30 Dec 2004 00:22:52 +0000 (UTC) Received: from blacklamb.mykitchentable.net (67-137-45-225.dsl2.elk.ca.frontiernet.net [67.137.45.225]) by relay01.roc.ny.frontiernet.net (Postfix) with ESMTP id BBA38FFEB for ; Thu, 30 Dec 2004 00:22:49 +0000 (UTC) Received: from [165.107.42.204] (unknown [165.107.42.204]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by blacklamb.mykitchentable.net (Postfix) with ESMTP id 386173BF3C0 for ; Wed, 29 Dec 2004 16:22:47 -0800 (PST) Message-ID: <41D34A5B.6080504@mykitchentable.net> Date: Wed, 29 Dec 2004 16:22:51 -0800 From: Drew Tomlinson User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new-20040701 (2.0) at filter01.roc.ny.frontiernet.net Subject: Multiple Vinum Problems Since Upgrade From 4.9 To 4.10 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Dec 2004 00:22:54 -0000 Since an upgrade from 4.9 to 4.10, I've had problems with vinum. The basic problem is that upon reboot, two of my vinum drives show up as "referenced" and thus create the associated chaos. I've tried many things and fiddled around quite a bit so I can't say exactly what I've done. I can include all of the entries in the history file since Oct. 31 if that's a help but it would be a long list. So prior to digging that deep, I will describe where I stand currently and where I want to finish. Currently, I have one vinum volume that I use for /usr. Fortunately this volume is up and running or I would really be in a mess. Here's the 'vinum list' output in this state: blacklamb# vinum vinum -> list 2 drives: D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) 1 volumes: V usr State: up Plexes: 1 Size: 16 GB 1 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB 2 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB I want to add another volume and mount it on /ftp. After creating the volume, vinum sees it and it appears OK as indicated in this output: vinum -> list 5 drives: D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) D ftp1 State: up Device /dev/ad0s1h Avail: 0/76319 MB (0%) D ftp2 State: up Device /dev/ad1s1h Avail: 0/76319 MB (0%) D ftp3 State: up Device /dev/da3s1h Avail: 0/114473 MB (0%) 2 volumes: V usr State: up Plexes: 1 Size: 16 GB V ftp State: up Plexes: 1 Size: 260 GB 2 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB P ftp.p0 C State: up Subdisks: 3 Size: 260 GB 5 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB S ftp.p0.s0 State: up PO: 0 B Size: 74 GB S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB S ftp.p0.s2 State: up PO: 149 GB Size: 111 GB Next I 'quit' the vinum command line and fsck my newly created volume. It finishes successfully and I mount it: blacklamb# df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/da0s1a 302350 102088 176074 37% / /dev/vinum/usr 16639674 5433762 9874739 35% /usr procfs 4 4 0 100% /proc /dev/vinum/ftp 265119539 119729707 132133856 48% /ftp I recheck with the 'vinum list' command and the output is the same as above. Now I cross my fingers and reboot to see if the volume comes up properly on startup. However ftp3 (/dev/da3s1h) comes up 'referenced' and causes problems. Mounting root from ufs:/dev/da0s1a dumpon: crash dumps to /dev/da1s1b (13, 131081) vinum: loaded vinum: reading configuration from /dev/ad1s1h vinum: ftp.p0.s2 is crashed vinum: ftp.p0 is corrupt vinum: updating configuration from /dev/ad0s1h vinum: updating configuration from /dev/da1s1h vinum: updating configuration from /dev/da0s1h vinum: /dev is mounted read-only, not rebuilding /dev/vinum Warning: defective objects D ftp3 State: referenced Device Avail: 0/0 MB P ftp.p0 C State: corrupt Subdisks: 3 Size: 260 GB S ftp.p0.s2 State: crashed PO: 149 GB Size: 111 GB swapon: adding /dev/da1s1b as swap device Automatic boot in progress... /dev/da0s1a: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1a: clean, 100130 free (938 frags, 12399 blocks, 0.6% fragmentation) /dev/vinum/ftp: CANNOT READ: BLK 546979872 /dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. /dev/vinum/ftp: CANNOT WRITE: BLK 2800 /dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. /dev/vinum/usr: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/vinum/usr: clean, 11205903 free (152463 frags, 1381680 blocks, 0.9% fragmentation) THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY: /dev/vinum/ftp (/ftp) File system preen failed, trying fsck -y . . . ** /dev/da0s1a ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2053 files, 51045 used, 100130 free (938 frags, 12399 blocks, 0.6% fragmentation) ** /dev/vinum/usr ** Last Mounted on /usr ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 401072 files, 5433771 used, 11205903 free (152463 frags, 1381680 blocks, 0.9% fragmentation) ** /dev/vinum/ftp CANNOT READ: BLK 546979872 UNEXPECTED SOFT UPDATE INCONSISTENCY CONTINUE? yes THE FOLLOWING DISK SECTORS COULD NOT BE READ: 546979872, 546979873, 546979874, 546979875, /dev/vinum/ftp: CANNOT FIGURE OUT FILE SYSTEM PARTITION UNEXPECTED SOFT UPDATE INCONSISTENCY WARNING: R/W mount of /ftp denied. Filesystem is not clean - run fsck mount: /dev/vinum/ftp: Operation not permitted Mounting /etc/fstab filesystems failed, startup aborted Enter full pathname of shell or RETURN for /bin/sh: I enter the shell and run 'vinum list': vinum -> list 4 drives: D ftp1 State: up Device /dev/ad0s1h Avail: 0/76319 MB (0%) D ftp2 State: up Device /dev/ad1s1h Avail: 0/76319 MB (0%) D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) D ftp3 State: referenced Device Avail: 0/0 MB 2 volumes: V usr State: up Plexes: 1 Size: 16 GB V ftp State: up Plexes: 1 Size: 260 GB 2 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB P ftp.p0 C State: corrupt Subdisks: 3 Size: 260 GB 5 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB S ftp.p0.s0 State: up PO: 0 B Size: 74 GB S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB S ftp.p0.s2 State: crashed PO: 149 GB Size: 111 GB Now I delete all references to the ftp volume by running these commands: vinum -> rm -f ftp.p0.s2 vinum: removing ftp.p0.s2 Correcting length of ftp.p0: was 547043829, is 312602446 vinum: ftp.p0 is up vinum -> rm -f ftp.p0.s1 vinum: removing ftp.p0.s1 Correcting length of ftp.p0: was 312602446, is 156301223 vinum -> rm -f ftp.p0.s0 vinum: removing ftp.p0.s0 vinum -> rm -f ftp.p0 vinum: removing ftp.p0 vinum: ftp is down vinum -> rm -f ftp vinum: removing ftp vinum -> rm -f ftp3 vinum -> rm -f ftp2 vinum -> rm -f ftp1 Yet the 'reference to ftp3 remains. vinum -> list 1 drives: D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) D ftp3 State: referenced Device Avail: 0/0 MB 1 volumes: V usr State: up Plexes: 1 Size: 16 GB 1 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB 2 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB Why does the list output show "1 drives:" when there are three listed and two working? Anyway, I've been here before and I quit vinum. Then I issue these commands: # df Filesystem 512-blocks Used Avail Capacity Mounted on /dev/da0s1a 604700 204180 352144 37% / /dev/vinum/usr 33279348 10867544 19749458 35% /usr procfs 8 8 0 100% /proc # umount /usr # vinum stop vinum: unloaded vinum unloaded # vinum start vinum: loaded vinum: reading configuration from /dev/da1s1h vinum: updating configuration from /dev/da0s1h Then I issue a 'vinum list' and all seems well. # vinum vinum -> list 2 drives: D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) 1 volumes: V usr State: up Plexes: 1 Size: 16 GB 1 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB 2 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB vinum -> Since everything appears correct, this seems like a good place to do 'vinum saveconfig'. Next I edit /etc/fstab so only /ftp is not mounted and reboot. The system reboots fine and the 'vinum list' is as it is immediately above. Running in full production mode, I create the ftp volume again: vinum -> create -f vinum_ftp.conf Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp1 is up Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp2 is up Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp3 is up 5 drives: D disk1 State: up Device /dev/da0s1h Avail: 0/8383 MB (0%) D disk2 State: up Device /dev/da1s1h Avail: 0/8383 MB (0%) D ftp1 State: up Device /dev/ad0s1h Avail: 0/76319 MB (0%) D ftp2 State: up Device /dev/ad1s1h Avail: 0/76319 MB (0%) D ftp3 State: up Device /dev/da3s1h Avail: 0/114473 MB (0%) 2 volumes: V usr State: up Plexes: 1 Size: 16 GB V ftp State: up Plexes: 1 Size: 260 GB 2 plexes: P usr.p0 S State: up Subdisks: 2 Size: 16 GB P ftp.p0 C State: up Subdisks: 3 Size: 260 GB 5 subdisks: S usr.p0.s0 State: up PO: 0 B Size: 8383 MB S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB S ftp.p0.s0 State: up PO: 0 B Size: 74 GB S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB S ftp.p0.s2 State: up PO: 149 GB Size: 111 GB Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s0 is up Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s1 is up Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s2 is up Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0 is up Dec 24 16:37:45 blacklamb /kernel: vinum: ftp is up After running fsck, I can mount and access the volume. The system continues to run fine until the next reboot where the same problems start all over again. I also have the same problem with another volume I wish to run called "backup". How can I fix my vinum volumes so they survive reboots? Thanks for your help and Happy Holidays! Drew