Date: Wed, 07 Feb 2018 01:05:58 -0600 From: Scott Bennett <bennett@sdf.org> To: dch@skunkwerks.at Cc: freebsd-questions@freebsd.org Subject: Re: ZFS error message questions Message-ID: <201802070705.w1775ww7029499@sdf.org> In-Reply-To: <1517435952.2407347.1255099432.2DE20786@webmail.messagingengine.com> References: <201801100811.w0A8B62K027111@sdf.org> <1517435952.2407347.1255099432.2DE20786@webmail.messagingengine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Dave Cottlehuber <dch@skunkwerks.at> wrote:
Please pardon my delay in replying. I wanted to digest an out-of-date
thread referenced in the zfs(8) man page, your message, and your references
at the end, as well as giving a few things a try before replying.
> On Wed, 10 Jan 2018, at 09:11, Scott Bennett wrote:
>
> Hopefully somebody replied off-line to get you out of your dilemma, Scott.
No, no one did. I was startled to receive your belated note, for which
I thank you very, very much.
> If not, at least I hope this is useful to your understanding.
It has helped, yes.
>
> > After running 11.1-STABLE with root on ZFS for a week and a half or so,
> > I apparently screwed something up such that the system became unusable while
> > trying to enter multi-user mode. It looked like the easiest way to correct
> > the mess would be to wipe it out and re-install.
>
> I'm a huge fan of using Boot Environments via sysutils/beadm-devel for this.
> Basically your / filesystem lives in zroot/ROOT/a_meaningful_name, except
> you can:
>
> - have multiple copies / versions
> - select one of those at start time
> - keep "anything useful" on a dataset that lives outside (zroot/usr/home for example)
> so that it stays between environments
> - I can build and install from source into a new boot environment
> - or try out the next release safely inside there
> - have super powers for sysadmins
ZFS is a huge chunk of software, and it is very incompletely documented.
As a consequence, it took quite a lot to convince me to commit my archives to
it a few years ago. Booting from a ZFS pool still seems very uncertain to me
because of the lack of documentation for something that would contain the
operating system itself. However, because of what happened in October 2016
with 10.3-STABLE, boot environments were the temptation that pushed me over
the edge for the advantages you list above. I suspect my understanding of
boot environments is still not good enough, but, in part due to your reply,
I am now on 11.1-STABLE (r328491) with several boot environments created and
installed. As I type, my system is running that version with a kernel that
is GENERIC but for the use of the 4BSD scheduler. (Does anyone actually use
the piece-of-crap ULE scheduler by choice? Why is it the default scheduler?)
My next move will be to update my customized kernel config to 11.1.
After that, I have something potentially trickier to do. 11.1-STABLE was
installed from a DVD originally. Unfortunately, bsdinstall doesn't appear to
be much of an improvement over sysinstall. It does allow a ZFS-based
installation, but for a ZFS installation, it rejects all user-supplied disk
partitioning and forces a three-partition (pmbr, swap, root pool) installation.
I have two HDDs that were mostly mirrored (with gmirror(8)) as my boot drives,
but the default partitioning for a ZFS installation by bsdinstall is not what
I want. I currently still have 10.3-STABLE installed on ada0 and 11.1-STABLE
installed on ada1. I now want to partition ada0 to a more satisfying layout,
initialize whatever needs to be initialized on it for a ZFS-booted
installation, and then copy from the current pool (on ada1) to that pool (on
ada0), destroy the partition table on ada1, copy the table from ada0 to ada1,
copy/install the pmbr stuff and anything else needed, then zpool attach the
ada1 system partition to the ada0 system pool, and reboot into a properly
partitioned, mirrored system. The problem is knowing all the details that
have to be satisfied in order to be able to boot afterward from either drive.
>
> > However, I wanted to save the home directory tree because there had been
> > important changes in it since it had been imported from the 10.3-STABLE system
> > with root, /usr/home, and so on on UFS2, so I ran
> >
> > zfs send -ReL -I @15dec2017.freshlyinstalled-11.1S-r236620 zrootx/usr/
> > home@28dec2017 > /mnt/bootdisks/usrhome.28dec2017.zfs-send
> >
> > which is supposed, as I understand it, to have saved a full replication stream,
> > including four snapshots. Now I wish to receive this into the newly
> > re-installed system, but here is what happens.
> >
> > Script started on Sun Jan 7 00:54:39 2018
> > # zfs recv -dv zrootx </mnt/bootdisks/usrhome.28dec2017.zfs-send
> > receiving incremental stream of zrootx/usr/home@16dec2017.post-import-
> > from-10.3S into zrootx/usr/home@16dec2017.post-import-from-10.3S
> > cannot receive incremental stream: most recent snapshot of zrootx/usr/
> > home does not
> > match incremental source
> > # exit
>
> Almost - you're not starting from the beginning of the zfs dataset, but a snapshot
> partway through, 15dec2017.freshlyinstalled-11.1S-r236620.
>
> As an example if we squint hard and pretend that the dataset is just a simple
> array, e.g. dataset [0,1,2,3,4] is the entire dataset and all snapshots, then
> you've sent [2,3,4] along, but you still need [0,1] to be able to have all the data,
> and also to be able to create a valid dataset you'll need the first few chunks - anything
> that hasn't changed in [2,3,4] is only going to exist in the early blocks in [0,1].
>
After at least fifty readings of that part of the zfs(8) man page, I had
never understood that. After getting no response within a few days to my
inquiry, I eventually figured out that there was another way to interpret the
man page that yielded that understanding. :-( It could have been better
written. (N.B. I began doing operating system work about 49 years ago.)
> If you don't need the earlier snapshots, its sufficient to `zfs send -Lev zrootx/usr/home@28dec2017 > zrootx_usr_home_28dec2017.zfs`. Just remember that's not recursive so you'll
But if I want the snapshots, will adding "R" to the options take care
of it?
> need to do that for each dataset separately. This is likely to be a lot smaller than the
> full set with all intermediary snapshots.
Unfortunately, that is all now blown away, and the computations have been
redone or obviated by passage of time.
>
> > [much text deleted -- SB]
> >
>
> Some reference for future BE (boot environment fun) are here:
>
> http://www.infracaninophile.co.uk/articles/zfs-update-management/
> http://www.callfortesting.org/bhyve-boot-environments/
> https://dan.langille.org/2017/09/30/upgrading-from-freebsd-10-3-to-11-1-via-freebsd-update-and-beadm/
>
I have now had time to read all of the above and digest them, which has,
I hope, led to a clearer understanding of what goes on with boot environments
and how to use them to do what I want to do. Thank you for including them in
your reply.
Now, if only someone were to finish documenting ZFS...
Thanks very much again for your reply. It has been very helpful.
Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet: bennett at sdf.org *xor* bennett at freeshell.org *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good *
* objection to the introduction of that bane of all free governments *
* -- a standing army." *
* -- Gov. John Hancock, New York Journal, 28 January 1790 *
**********************************************************************
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201802070705.w1775ww7029499>