From owner-freebsd-questions@freebsd.org Wed Apr 24 07:35:19 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87AB2158FD27 for ; Wed, 24 Apr 2019 07:35:19 +0000 (UTC) (envelope-from trond.endrestol@ximalas.info) Received: from enterprise.ximalas.info (enterprise.ximalas.info [IPv6:2001:700:1100:1::8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "ximalas.info", Issuer "Hostmaster ximalas.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id ABE8180530 for ; Wed, 24 Apr 2019 07:35:18 +0000 (UTC) (envelope-from trond.endrestol@ximalas.info) Received: from enterprise.ximalas.info (Ximalas@localhost [127.0.0.1]) by enterprise.ximalas.info (8.15.2/8.15.2) with ESMTPS id x3O7ZART008082 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 24 Apr 2019 09:35:10 +0200 (CEST) (envelope-from trond.endrestol@ximalas.info) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ximalas.info; s=default; t=1556091310; bh=btEqu1XC3KGc/kcR+4zX0b1p2iIZzZmH8lznAS4SHv4=; h=Date:From:To:Subject:In-Reply-To:References; b=YqhxPcaZrgdT0cms9ia49HPTOrhXzj3ktzeJ1KOBO/yh5TXlfoucqcNvS1fL5ZeIx KWmZvrZJobxd0EUJXaEhdDoPwLmPlfNaAuT3dBNclpO15ZO4FWj48lZqjcume0BKcu s62glJ1FWvgo8QE5bkH7YKu0Zu96a1gWj+jXRN+MYPSisemP2vfhCcpVeZHxQ0QCO4 SZAMEEf09/mCDHDnQgAPfHleOHTc0quJQw57b0kF77NH8gYdFTithximX7EEoe3neL ldK8HnAmK0RrN7flfucbw5R+AhTPBcQmLSeltt9cYaECfRkCy7FD8uM+kpBtlmzKS/ 5mlMii/8Vhn5Q== Received: from localhost (trond@localhost) by enterprise.ximalas.info (8.15.2/8.15.2/Submit) with ESMTP id x3O7Z9Yw008079 for ; Wed, 24 Apr 2019 09:35:09 +0200 (CEST) (envelope-from trond.endrestol@ximalas.info) X-Authentication-Warning: enterprise.ximalas.info: trond owned process doing -bs Date: Wed, 24 Apr 2019 09:35:09 +0200 (CEST) From: =?UTF-8?Q?Trond_Endrest=C3=B8l?= Sender: Trond.Endrestol@ximalas.info To: freebsd-questions@freebsd.org Subject: Re: ZFS is auto-mounting in wrong order In-Reply-To: <20190418233319.GA28238@g5.umpquanet.com> Message-ID: References: <20190418233319.GA28238@g5.umpquanet.com> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) OpenPGP: url=http://ximalas.info/about/tronds-openpgp-public-key MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-1.2 required=5.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on enterprise.ximalas.info X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Apr 2019 07:35:19 -0000 On Thu, 18 Apr 2019 16:33-0700, Jim Long wrote: > I have a system that boots from ZFS. I recently upgraded it from > FreeBSD 11.2-STABLE to 12.0-STABLE: > > FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC amd64 > > I'm seeing certain directories as empty, when they should have lots of > content. The problem can be worked around by unmounting key > filesystems, and re-mounting them in the proper order. We are both experiencing some unforeseen side-effects of r345578, https://svnweb.freebsd.org/base?view=revision&revision=345578. I first observed this odd behaviour when upgrading from r345628 to r346220. I assumed some of the ZFS changes between these revision were to blame, i.e. r346126, r346128, and/or r346130, but alas, r346125 exhibits the same symptoms as r346220. I mistook this issue for being deterministic, but it isn't. I have rebooted my system now running r346269 a couple of times today, and sometimes the parallel mounting succeeds and sometimes it doesn't. In my particular case it's better to mount all filesystems belonging to my root pool in sequential order before mounting in parallel the remaining filesystems from the other pools. Afterall, I only have about 135 filesystems to mount. I will open a PR suggesting parallel mounting be made optional for us who don't have thousands of filesystems to be mounted at boot time. Maybe doing the root pool before other pools is the way forward. See the discussion I initiated on -stable@, https://lists.freebsd.org/pipermail/freebsd-stable/2019-April/090898.html I did make one change today. /var/db from the root pool is no longer a separate filesystem. Its contents were moved aside to a directory on the root filesystem, /var/db1, the canmount property was set to off, and the new directory assumed the correct name of /var/db. For now, I boot my affected system in single-user mode and run my remount shell script saved as /remount-filesystems.sh before exiting to multiuser mode: #!/bin/sh # To be run while in singleuser mode, # preferably (re)booted directly to singleuser mode. PATH="/bin:/sbin:/usr/bin:/usr/sbin:/rescue" export PATH killall devd killall local-unbound killall moused killall wpa_supplicant umount /usr/compat/linux/dev/fd umount /usr/compat/linux/dev umount /usr/compat/linux/proc umount /usr/compat/linux/sys zfs unmount -a mount -uw / for fs in `zfs list -Hro canmount,name enterprise_zroot | egrep -v '(^off)|(enterprise_zroot$)|(enterprise_zroot/ROOT)|(enterprise_zroot/do-not-destroy)' | awk '{print $2}'`; do zfs mount ${fs} done for fs in `zfs list -Hro canmount,name enterprise_zdata | egrep -v '(^off)|(enterprise_zdata$)|(enterprise_zdata/do-not-destroy)' | awk '{print $2}'`; do zfs mount ${fs} done mount -al echo "You may now attempt to exit to multiuser mode ..." # EOF -- Trond.