From owner-freebsd-questions@FreeBSD.ORG  Sat Sep 25 18:18:48 2010
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B06831065698
	for <freebsd-questions@freebsd.org>;
	Sat, 25 Sep 2010 18:18:48 +0000 (UTC)
	(envelope-from bonomi@mail.r-bonomi.com)
Received: from mail.r-bonomi.com (ns2.r-bonomi.com [204.87.227.129])
	by mx1.freebsd.org (Postfix) with ESMTP id 7EC448FC1D
	for <freebsd-questions@freebsd.org>;
	Sat, 25 Sep 2010 18:18:48 +0000 (UTC)
Received: (from bonomi@localhost)
	by mail.r-bonomi.com (8.14.3/rdb1) id o8PIGnnG027025;
	Sat, 25 Sep 2010 13:16:49 -0500 (CDT)
Date: Sat, 25 Sep 2010 13:16:49 -0500 (CDT)
From: Robert Bonomi <bonomi@mail.r-bonomi.com>
Message-ID: <201009251816.o8PIGnnG027025@mail.r-bonomi.com>
To: freebsd-questions@freebsd.org, tg@gmplib.org
Cc: 
Subject: Re: Mount order for ZFS, jails, and nullfs
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 25 Sep 2010 18:18:48 -0000

> From owner-freebsd-questions@freebsd.org  Fri Sep 24 11:02:59 2010
> To: freebsd-questions@freebsd.org
> From: Torbjorn Granlund <tg@gmplib.org>
> Date: Fri, 24 Sep 2010 18:03:06 +0200
> Subject: Mount order for ZFS, jails, and nullfs
>
> In jails, I'd like a local ZFS /, a read-only nullfs-mounted /usr, and a
> local /usr/local.  (I'd also have read-only nullfs-mounted /bin, /lib,
> /libexec, but let's forget about that for now.)
>
> This way, I can upgrade the master /usr once, in one place, and have all
> jails inherit it.  And my dear jail inmates can install anything in
> /usr/local (such as their favourite packages/ports).
>
> I.e., things should look like this:
>
> /myjail/                zfs
> /myjail/usr             nullfs ro
> /myjail/usr/local       zfs
>
> There is no problem to make this happen by issuing a handful of commands
> manually after boot, but I cannot seem to get it to work automatically,
> with existing boot mechanism.  The problem is that the mount of /usr
> will be attempted before ZFS has mounted /myjail, the jail's root.
>
> ZFS maintains its own mount table.  It is possible to disable the
> automated mounting in ZFS by specifying the pseudo mountpoint "legacy",
> and then--according to the FreeBSD manual--mount it with mount(8).
> Unfortunately, FreeBSD's mount seems unable to perform mounts of ZFS
> volumes.
>
> How can I achieve the result I want after a reboot?

if the traditional way doesn't work for you, the solution is "don't use
the traditional way'.  <grin?

Seriously.

For the stuff that has to  mount _after_ zfs is running (i.e. '/myjail/usr'),
mark them as 'noauto' in the fstab.  

Do the 'equivalent' for any zfs fs that mounts _below_ those points (i.e.,
'/myjail/user/local').

That way the normal stages in the boot process will _not- mount them 
prematurely.

THEN, add a boot script that "requires zfs' and wherever the actual zfs
mounts are done. In -that- script, insert 'manual' commands, _in_the_order_
you need things to happen in.

It is advisable to drop notes in /etc/fstab and the zfs equivalent, that
these filesystems are automatically mounted by the {mumble} script.  Such
notes save _lots_ of hair-tearing in the future, when you have to trouble-
shoot something you "don't remember" the details of doing.