From owner-freebsd-fs@FreeBSD.ORG Sat Sep 17 15:59:10 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F37A1065670; Sat, 17 Sep 2011 15:59:10 +0000 (UTC) (envelope-from jeremie@le-hen.org) Received: from smtp5-g21.free.fr (unknown [IPv6:2a01:e0c:1:1599::14]) by mx1.freebsd.org (Postfix) with ESMTP id 5BFC58FC19; Sat, 17 Sep 2011 15:59:07 +0000 (UTC) Received: from endor.tataz.chchile.org (unknown [82.233.239.98]) by smtp5-g21.free.fr (Postfix) with ESMTP id 1819CD480D5; Sat, 17 Sep 2011 17:59:01 +0200 (CEST) Received: from felucia.tataz.chchile.org (felucia.tataz.chchile.org [192.168.1.9]) by endor.tataz.chchile.org (Postfix) with ESMTP id 3DDAA34081; Sat, 17 Sep 2011 15:59:00 +0000 (UTC) Received: by felucia.tataz.chchile.org (Postfix, from userid 1000) id 2D0D0A1349; Sat, 17 Sep 2011 15:59:00 +0000 (UTC) Date: Sat, 17 Sep 2011 17:59:00 +0200 From: Jeremie Le Hen To: Martin Matuska Message-ID: <20110917155859.GA8243@felucia.tataz.chchile.org> References: <20110905195458.GA7863@felucia.tataz.chchile.org> <4E65393F.9070401@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E65393F.9070401@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, Jeremie Le Hen Subject: Re: Difficulties to use ZFS root: ROOT MOUNT ERROR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2011 15:59:10 -0000 Hi, On Mon, Sep 05, 2011 at 11:03:59PM +0200, Martin Matuska wrote: > On 5. 9. 2011 21:54, Jeremie Le Hen wrote: > > Hi list, > > > > I've followed the instructions documented here: > > http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition > > > > The kernel starts correctly so this rules out any problem regarding > > boot0, zfsboot and ZFS loader. > > > > But when the kernel tries to mount the root filesystem, it fails with > > the following output: > > > > % Trying to mount root from zfs:zroot > > % ROOT MOUNT ERROR: > > % If you have invalid mount options, reboot, and first try the following from > > % the loader prompt: > > % > > % set vfs.root.mountfrom.options=rw > > % > > % and then remove the invalid mount options from /etc/fstab. > > % > > % Loader variables: > > % vfs.root.mountfrom=zfs:zroot > > % vfs.root.mountfrom.options=rw > > > > >From a netboot'd FreeBSD: > > > > # zfs import zroot > > % # zpool get bootfs zroot > > % NAME PROPERTY VALUE SOURCE > > % zroot bootfs zroot local > > % > > % # zfs list -o name,canmount,mountpoint > > % NAME CANMOUNT MOUNTPOINT > > % zroot on legacy > > % zroot/tmp on /tmp > > % zroot/usr on /usr > > % zroot/usr/home on /usr/home > > % zroot/usr/ports on /usr/ports > > % zroot/usr/ports/distfiles on /usr/ports/distfiles > > % zroot/usr/ports/packages on /usr/ports/packages > > % zroot/usr/src on /usr/src > > % zroot/usr/src8 on /usr/src8 > > % zroot/var on /var > > % zroot/var/crash on /var/crash > > % zroot/var/db on /var/db > > % zroot/var/db/pkg on /var/db/pkg > > % zroot/var/empty on /var/empty > > % zroot/var/log on /var/log > > % zroot/var/mail on /var/mail > > % zroot/var/run on /var/run > > % zroot/var/tmp on /var/tmp > > % # zfs export zroot > > > > /boot/zfs/zpool.cache exists in the zroot filesystem: > > > > % # zpool import -R /mnt zroot > > % # zfs set mountpoint=/ zroot > > % # ls -l /mnt/boot/zfs/zpool.cache > > % -rw-r--r-- 1 root wheel 924 Sep 5 07:31 /mnt/boot/zfs/zpool.cache > > % # grep zfs /mnt/boot/loader.conf /mnt/etc/rc.conf > > % /mnt/boot/loader.conf:zfs_load="YES" > > % /mnt/boot/loader.conf:vfs.root.mountfrom="zfs:zroot" > > % /mnt/etc/rc.conf:zfs_enable="YES" > > > > > > Any idea why this error occurs? > > > > Thanks > It might be a problem in the zpool.cache. > If you read the zpool(8) manpage properly, you will find this: > > -R root > Equivalent to "-o cachefile=none,altroot=root" > > If you mount a pool with an alternate root and want to update the > cachefile, you have to explicitly state the cachefile. > (e.g. zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot) > Second, you should not have an exported pool for booting (from viewpoint > of target system's zfs.cache). > Third, you don't need a legacy mount for zroot. You can leave it to "/" > but you don't have to. > I personally prefer having everyting one level deeper (e.g. pool/root, > pool/root/var, etc.). > > Therefore I suggest: > zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot > zfs set mountpoint=/ zroot > cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache > shutdown -r now > (as you can see I have not exported the pool) I've just had the opportunity to migrate another server to ZFS. This time I followed your advice by "having everything one level deeper". That is zroot/root is "/". I've modified "zfs:mountfrom" in loader.conf(5) accordingly. obiwan:~# zpool get all zroot NAME PROPERTY VALUE SOURCE zroot size 147G - zroot capacity 2% - zroot altroot /mnt local zroot health ONLINE - zroot guid 12889954819379028468 default zroot version 28 default zroot bootfs zroot/root local zroot delegation on default zroot autoreplace off default zroot cachefile none local zroot failmode wait default zroot listsnapshots off default zroot autoexpand off default zroot dedupditto 0 default zroot dedupratio 1.00x - zroot free 143G - zroot allocated 3.69G - zroot readonly off - obiwan:~# zfs list -o name,mounted,canmount,mountpoint -r zroot | grep -v /jails NAME MOUNTED CANMOUNT MOUNTPOINT zroot no on none zroot/root yes on /mnt zroot/root/root yes on /mnt/root zroot/root/tmp yes on /mnt/tmp zroot/root/usr yes on /mnt/usr zroot/root/usr/local yes on /mnt/usr/local zroot/root/usr/obj yes on /mnt/usr/obj zroot/root/usr/pkgsrc yes on /mnt/usr/pkgsrc zroot/root/usr/pkgsrc/distfiles yes on /mnt/usr/pkgsrc/distfiles zroot/root/usr/ports yes on /mnt/usr/ports zroot/root/usr/ports/distfiles yes on /mnt/usr/ports/distfiles zroot/root/usr/ports/packages yes on /mnt/usr/ports/packages zroot/root/usr/src yes on /mnt/usr/src zroot/root/var yes on /mnt/var zroot/root/var/crash yes on /mnt/var/crash zroot/root/var/db yes on /mnt/var/db zroot/root/var/db/pkg yes on /mnt/var/db/pkg zroot/root/var/empty yes on /mnt/var/empty zroot/root/var/log yes on /mnt/var/log zroot/root/var/mail yes on /mnt/var/mail zroot/root/var/run yes on /mnt/var/run zroot/root/var/tmp yes on /mnt/var/tmp The kernel boots fine, it finds the root filesystem, but fails miserably when running rc.d scripts because deeper datasets are not mounted (/var, /usr, ...). FWIW, I escaped to DDB and typed "show mount". Besides /dev, / was indeed mounted from zoot/root and /tmp was /dev/md0 for an unknown reason. I've been fiddling this this for 3 hours this afternoon without luck. Does anyone have an idea on this please? Thanks. Regards, -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne