Date: Tue, 26 Jun 2012 21:40:12 GMT From: George Hartzell <hartzell@alerce.com> To: freebsd-fs@FreeBSD.org Subject: Re: kern/166566: [zfs] zfs split renders 2 disk (MBR based) mirror unbootable Message-ID: <201206262140.q5QLeCY2025612@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/166566; it has been noted by GNATS.
From: George Hartzell <hartzell@alerce.com>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: hartzell@alerce.com,
bug-followup@FreeBSD.org
Subject: Re: kern/166566: [zfs] zfs split renders 2 disk (MBR based) mirror
unbootable
Date: Tue, 26 Jun 2012 14:38:17 -0700
Andriy Gapon writes:
>
> [restoring bug-followup]
>
> on 25/06/2012 19:48 George Hartzell said the following:
> > Here are two images (more than one screen full) of the lsdev -v output
> > from the loader that *actually loads the system* (when it's
> > working...).
>
> I have a theory of what's going on.
>
> I believe that after zpool split the following items get updated with new
> information:
> - vdev label on the disk that remains in the main pool (ada3 + zroot)
> - vdev label on the disk that goes to the new pool (ada1 + zsplitroot)
> - zpool.cache file in the main/active/remaining pool (zroot)
>
> The following item still has outdated information:
> - zpool.cache file in the new pool (zsplitroot)
>
> This happens because the new pool gets the contents of the original pool at
> split start time (before any new ids are generated). The file can not be
> updated automatically because the new pool remains "un-imported" (exported)
> after the split. If it is desired that the zsplitroot's zpool.cache is updated
> it has to be done manually - by importing the pool, etc.
>
> I believe that what you see is a result of you always booting in such a way that
> the zfs boot code and zfsloader find zsplitroot pool before zroot pool. This is
> confirmed by the screenshot which shows that zsplitroot is listed before zroot.
> Because of that the stale zpool.cache file is used and as a result the ZFS code
> in kernel can not find disks/pools based on the stale IDs.
>
> I think that you have to change the boot order using BIOS, so that you boot from
> ada3 disk. You should verify at the loader prompt that that is indeed the case
> and zroot is found first and is used as a boot pool.
>
> If your BIOS either doesn't allow to change boot order, or lies about it or
> doesn't change bios disk numbering such that a boot disk is the first drive
> (disk0 / "BIOS drive C"), then I recommend thatyou set 'currdev' loader variable
> to point to zroot pool. Depending on your zfsloader version it should be done
> in one of the following ways:
> set currdev=zfs:zroot:
> set currdev=zfs1
> You can examine default value of the variable (with 'show' command) to see which
> scheme should be used.
>
> Please test this.
We're very close.
First thing, I discovered that I was wrong about its being able to
boot from either disk. I don't think I ever misspoke to you (don't
see it in the bug report, can't find it in our personal emails) but I
certainly had it in my head that I had tried booting from both disks
and neither worked. It turns out that one will boot but the other
will not.
Some background (for the bug trail). This is a mac pro with four
internal SATA disks. When you power on with the option key held down
you're presented with a graphic view of things that you can boot from.
In my configuration I have two disks labeled "dinky 1" and "dinky 2"
(Mac disks in a Mac OS X software raid) and two labeled "Windows" (the
boot stuff considers anything with an MBR to be Windows, sigh...).
Macs play fast and loose with device numbering/naming and I have no
way to tell the Windows devices apart. For the rest of this
discussion I'll just refer to the four drives as A, B, C, and D (dinky
1, dinky 2, Windows, Windows) in left to right order.
Out of habit, I tend to boot into BSD on drive C (two hits of the
right arrow key).
While I was playing with setting currdev in the loader I realized that
I could boot from disk D but not from disk C (no matter how I set
currdev).
It turns out that when I boot from drive C and do an lsdev -v at the
loader prompt I get
...
zsplitroot ada1s1a
zroot ada3s1a
but/and when I boot from drive D and do an lsdev -v at the loader
prompt I get
...
zroot ada3s1a
zsplitroot ada1s1a
Notice that the order is different (confirming your observation).
When I boot off of C using vfs.zfs.debug=1 I get messages about
mismatched GUID's and failure to open the device it's looking for.
When I boot off of D things are fine. This is consistent with your
idea that there is incorrect information in the zpool.cache file on
the filesystem in the zsplitroot pool.
currdev does not seem to have any effect, it looks like something else
is being used to find the initial zfs pool(s).
I'm not sure what there is to do to make the situation better. It's
(probably) not the usual use case that zpool splilt is being used to
split the pool that contains the filesystem that contains the
zpool.cache file, so it would be an awfully special case to do
something special to handle it.
It seems like "the right thing" to do would be for the user (me) to do
the zpool split with the -R option then copy the correct zpool.cache
file into the split-off pool's root filesystem. I'll repair my
currently broken mirror and give that a try.
Thanks for all the help!
g.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201206262140.q5QLeCY2025612>