Date: Mon, 26 Oct 2009 00:45:44 +0100 From: "Merijn Verstraaten" <merijn@inconsistent.nl> To: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable" Message-ID: <op.u2dqyh1o4534sa@localhost>
next in thread | raw e-mail | index | archive | help
> Ok, can you retest with -CURRENT? I just committed some fixes on
> Friday. I'm having real difficulty in reproducing these issues. Most
> of the problems that I've run into so far had to do with the system not
> knowing about all of the vdevs when it wanted to read something. In
> your case, it looks like you are making it to boot3 and it appears to be
> seeing all 4 of your disks. Right now, I've been trying to track down
> an issue wher the MOS can't be read, which basically means that we have
> screwed up the root block pointer somehow. I haven't been able to
> reproduce that issue in qemu, I have been able to reproduce it with
> VirtualBox, but it is really time consuming trying to work in vbox since
> I have to reconvert all of the disk images every time I make a change.
> I'm actually a bit concerned that it hinges on how many drives are
> visible to the bios at various points in time.
> robert.
I noticed this thread in the archives and I've been having the same
problems as Radek (see belw) with root on ZFS RAID-Z not booting the
machine. When I install world from the 8.0-RC1 USB image I can boot from
RAID-Z with no problems. But when I csup'ed to RELENG_8 yesterday
(Saturday, so this should be after you committed the new changes) and
build+installed world booting from RAID-Z broke with the exact same
problems as Radek describes below. I just tried again to be absolutely
sure (It just occured to me I perhaps should have done "gpart show",
"zpool status" and "zpool get all z" before breaking the system, but oh
well...).
I have 4 SATA disks ad4, ad6, ad8 and ad10 with three partitions each,
adXp1 is a freebsd-boot partition like Radek has, adXp2 is a 1GB
freebsd-swap partition and adXp3 has the rest of the disk as a freebsd-zfs
partition. These 4 p3 partitions are placed in a single RAID-Z vdev. zpool
status looks pretty much like that of Radek too. I have my root partition
in tank/root and mount things like tank/usr and tank/var inside tank/root.
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
prompt(s):
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
boot:
If I wait a while this prompt repeats itself. I've beent rying to test
inside VirtualBox to see if I could report more detail on what's going
wrong but I ran into the same problem as you with the "MOS can't be read"
error.
This machine and zpool don't (at this time) contain any valuable data, so
if you need someone to test anything for you I'd gladly volunteer my box
if it helps resolve things faster.
Kind regards,
Merijn Verstraaten
On Sat, 2009-10-24 at 19:44 +0200, Radek Valášek wrote:
> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
> > >> Robert Noland napsal(a):
> >> >>> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> >>> >>> >>>> Hi,
> >>>>
> >>>> I want to ask if there is something new in adding support to>>>>
> gptzfsboot/zfsboot for reading gang-blocks?
> >>>> >>>> >
> > I think that the gang block patch will work, though still haven't
> gotten
> > it tested. However, I'm fairly confident that the issue is not gang
> > block related. Right now, I have setup a disk like this:
> >
> > => 34 1953525101 ada1 GPT (932G)
> > 34 128 1 freebsd-boot (64K)
> > 162 8388608 2 freebsd-swap (4.0G)
> > 8388770 648019968 3 freebsd-zfs (309G)
> > 656408738 648019968 4 freebsd-zfs (309G)
> > 1304428706 648019968 5 freebsd-zfs (309G)
> > 1952448674 1076461 - free - (526M)
> >
> > Note that this is not a raidz pool right now. It is just 3 toplevel
> > partitions setup as a single pool. I finally have this configuration
> > working reliably. At least in this case, the issue is due to all of
> the
> > partitions not being probed during early boot and so not being added to
> > the list of vdevs for the pool. When zio_read finds a dva that points
> > to a device it doesn't know about, it gives up and whines.
> >
> > Can you detail for me how you have everything configured, so that I can
> > try to replicate it. gpart show, zpool status and zpool get all <pool>
> > would be good. I'm not sure that I have enough spare disks lying
> around
> > to do this properly, but maybe I can use virtual disks or something.
> >
> > robert.
> >
> > Sorry for not responding so long. Here are details you want from me:
> # gpart show
> => 34 1953525101 ad6 GPT (932G)
> 34 128 1 freebsd-boot (64K)
> 162 1953524973 2 freebsd-zfs (932G)
> => 34 1953525101 ad8 GPT (932G)
> 34 128 1 freebsd-boot (64K)
> 162 1953524973 2 freebsd-zfs (932G)
> => 34 1953525101 ad10 GPT (932G)
> 34 128 1 freebsd-boot (64K)
> 162 1953524973 2 freebsd-zfs (932G)
> => 34 1953525101 ad12 GPT (932G)
> 34 128 1 freebsd-boot (64K)
> 162 1953524973 2 freebsd-zfs (932G)
> # zpool status
> pool: z
> state: ONLINE
> scrub: none requested
> config:
> NAME STATE READ WRITE CKSUM
> z ONLINE 0 0 0
> raidz1 ONLINE 0 0 0
> ad6p2 ONLINE 0 0 0
> ad8p2 ONLINE 0 0 0
> ad10p2 ONLINE 0 0 0
> ad12p2 ONLINE 0 0 0
> errors: No known data errors
> # zpool get all z
> NAME PROPERTY VALUE SOURCE
> z size 3.62T -
> z used 4.62G -
> z available 3.62T -
> z capacity 0% -
> z altroot - default
> z health ONLINE -
> z guid 17857007133862981114 -
> z version 13 default
> z bootfs z/system local
> z delegation on default
> z autoreplace off default
> z cachefile - default
> z failmode wait default
> z listsnapshots off default
> I've tested your patches but it seems that you're right and it's notgang
> related issue. I was able to discover these things on a fullyfunctional
> zfs pool (system compiled with your patches):
> 1, If I overwrite the file /boot/loader.conf (with copy of itself, or
> when upgrading kernel/world), next reboot comes with these messages:
> BTX loader 1.00 BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> Then I'm still able to boot the system, but I must set the bootvariables
> included in loader.conf by hand
> 2, Next I overwrite the file /boot/loader (with copy of itself, or when
> upgrading kernel/world) and reboot comes with these messages:
> BTX loader 1.00 BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> Unable to load a kernel!
> After that I'm no longer able to boot the system from zfs pool.
> Hope you have some ideas...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?op.u2dqyh1o4534sa>