Date: Wed, 26 Mar 2014 11:49:12 -0400 From: Kurt Lidl <lidl@pi-coral.com> To: freebsd-virtualization@freebsd.org Subject: booting bhyve off zfs - a cautionary tale Message-ID: <5332F6F8.1030400@pi-coral.com>
next in thread | raw e-mail | index | archive | help
Greetings all -
I'm writing to report a solution to a puzzling problem that I've had for
a while and just
worked out a solution over the last two days.
I've been working with the boot-bhyve-from-zfs patch that has been
floating around for a while, and had good success with it initially.
My configuration is such: the machine that run the hypervisor only has
ZFS filesystems. As such, I just create new zvols on that machine, and
map those into bhyve as the various vtbdX devices.
I have a script that will create the zvols, then carefully partitions
the zvol,
create a zpool on the zvol, create a few other zfs filesystems on the zpool,
and unpack the distribution .txz files into the resulting zpool.
This worked well for a while and I could create new virtual machines without
issue. And then, after an upgrade of the hosting machine to a more recent
stable/10 image, all the new images I created would fail to start properly.
The userboot.so support would locate the kernel, and load it, but the
mounting of the root partition would fail:
--- snip, snip ---
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
Timecounter "TSC-low" frequency 1149929960 Hz quality 800
Trying to mount root from zfs:nine5/ROOT/default []...
Mounting from zfs:nine5/ROOT/default failed with error 45.
Loader variables:
vfs.root.mountfrom=zfs:nine5/ROOT/default
--- snip, snip ---
After far too much time groveling through the boot code, I finally had the
realization of what was going on. Since I was creating the new zpool for
the client machine under stable/10, and I couldn't boot it under stable/9,
there must have been some incompatibility introduced.
Anyway, to make a long debugging story shorter, the newest zpool-features
(enabled by default) in stable/10 prevent the stable/9 kernel from
successfully
being about to mountroot() the zpool that created under stable/10.
It turns out there is a flag to suppress enabling all the zpool-features
when
a new zpool is created. So I switched my script to generate the zpool,
specifying "zpool create -d ...", and then enabling the features that
are OK for stable/9:
zpool set feature@async_destroy=enabled ${zroot}
zpool set feature@empty_bpobj=enabled ${zroot}
zpool set feature@lz4_compress=enabled ${zroot}
After these small changes with regards to the creation of the zpool,
I can now successfully boot the resulting zpool-on-a-zvol without issue.
Hopefully this information will save someone a bunch of time debugging
this problem in the future.
-Kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5332F6F8.1030400>