Date: Wed, 18 Apr 2012 16:01:25 GMT From: Peter Maloney <peter.maloney@brockmann-consult.de> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/167065: [zfs] boot fails when a spare is the boot disk Message-ID: <201204181601.q3IG1PSZ078904@red.freebsd.org> Resent-Message-ID: <201204181610.q3IGA53G092321@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 167065
>Category: misc
>Synopsis: [zfs] boot fails when a spare is the boot disk
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Apr 18 16:10:05 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator: Peter Maloney
>Release: 8-STABLE Feb 4 2012
>Organization:
Brockmann Consult
>Environment:
FreeBSD bcnas1bak.bc.local 8.2-STABLE-20120204 FreeBSD 8.2-STABLE-20120204 #1: Fri Feb 10 15:35:01 CET 2012 root@bcnas1bak.bc.local:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
The story:
A disk faulted. It was the first disk in the chassis (so it is the one the controller selects to boot).
I added a spare (as the 2nd disk in the chassis), and replaced the faulted disk with the spare.
Then I got the system to hang for unrelated reasons, removed the bad disk, and rebooted.
(unrelated reasons: I added some zpool v16 zfs v4 disks I created on FreeBSD, tried mouting in Linux with zfs-fuse, which told me a device was missing, and then put them back in the FreeBSD and imported, which also said a device was missing, and then "zpool import -m -F -N" or something similar made it hang. parted on Linux said the disks were ext3, which may be related to the problem.)
And on boot, to my surprise, it complained saying I can only boot from mirrors, raidz, etc. (a list which did not include spares). The second disk was defective in a different way, so I didn't try putting it first in the chassis, but that is a workaround, not a bug fix.
>How-To-Repeat:
Create a virtual machine with disks in this order:
boot mirror disk 1
spare disk
boot mirror disk 2
Boot it up. (using gpt slices, matching what I set up before). Replace the boot mirror disk 1 with the spare with:
# gpart create -s gpt da1
# gpart add -b 34 -s 128 -t freebsd-boot da1
# gpart add -b 129024 -s 32768 -t freebsd-swap da1
# gpart add -b 258048 -s 8386560 -t freebsd-zfs -l sp_root0 da1
# zpool add zroot spare gpt/sp_root0
# zpool replace zroot gpt/root0 gpt/sp_root0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
Wait for resilver.
$ zpool status zroot
-------------------------------------------------
pool: zroot
state: ONLINE
scan: resilvered 1.33G in 0h5m with 0 errors on Wed Apr 18 19:42:19 2012
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
spare-0 ONLINE 0 0 0
gpt/root0 ONLINE 0 0 0
gpt/sp_root0 ONLINE 0 0 0
gpt/root1 ONLINE 0 0 0
spares
665168513187118073 INUSE was /dev/gpt/sp_root0
errors: No known data errors
-------------------------------------------------
# shutdown -p now
Remove disk 1 and 3 (the old boot disks).
Start up again.
During boot, you get this output:
-------------------------------------------------
ZFS: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0
FreeBSD/x86 boot
Default: zroot:/boot/kernel/kernel
boot:
ZFS: unexpected object set type 0
FreeBSD/x86 boot
Default: zroot:/boot/kernel/kernel
boot:
-------------------------------------------------
>Fix:
Workaround was to boot the DVD, load the zfs modules, and then "zpool detach" the bad disk (converting the spare to a normal mirror), and detach the other root disk, which I wanted to remove anyway (to upgrade firmware).
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201204181601.q3IG1PSZ078904>