From owner-freebsd-questions@freebsd.org Thu Apr 18 23:37:27 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09B78157F5DD for ; Thu, 18 Apr 2019 23:37:27 +0000 (UTC) (envelope-from freebsd-questions@umpquanet.com) Received: from g5.umpquanet.com (ns.umpquanet.com [209.216.177.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3EC3A85789 for ; Thu, 18 Apr 2019 23:37:19 +0000 (UTC) (envelope-from freebsd-questions@umpquanet.com) Received: from g5.umpquanet.com (localhost [127.0.0.1]) by g5.umpquanet.com (8.15.2/8.15.2) with ESMTP id x3INXJQQ036600 for ; Thu, 18 Apr 2019 16:33:19 -0700 (PDT) (envelope-from freebsd-questions@umpquanet.com) Received: (from james@localhost) by g5.umpquanet.com (8.15.2/8.15.2/Submit) id x3INXJeW036599 for freebsd-questions@freebsd.org; Thu, 18 Apr 2019 16:33:19 -0700 (PDT) (envelope-from freebsd-questions@umpquanet.com) X-Authentication-Warning: g5.umpquanet.com: james set sender to freebsd-questions@umpquanet.com using -f Date: Thu, 18 Apr 2019 16:33:19 -0700 From: Jim Long To: freebsd-questions@freebsd.org Subject: ZFS is auto-mounting in wrong order Message-ID: <20190418233319.GA28238@g5.umpquanet.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.11.4 (2019-03-13) X-Rspamd-Queue-Id: 3EC3A85789 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd-questions@umpquanet.com designates 209.216.177.146 as permitted sender) smtp.mailfrom=freebsd-questions@umpquanet.com X-Spamd-Result: default: False [-2.21 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.80)[-0.802,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; HAS_XAW(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.97)[-0.966,0]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; MX_GOOD(-0.01)[g5.umpquanet.com]; NEURAL_HAM_SHORT(-0.12)[-0.117,0]; DMARC_NA(0.00)[umpquanet.com]; IP_SCORE(-0.01)[country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11233, ipnet:209.216.160.0/19, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 23:37:27 -0000 I have a system that boots from ZFS. I recently upgraded it from FreeBSD 11.2-STABLE to 12.0-STABLE: FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC amd64 I'm seeing certain directories as empty, when they should have lots of content. The problem can be worked around by unmounting key filesystems, and re-mounting them in the proper order. Troubleshooting leads me to believe that the ZFS system is not mounting them in the proper order, but I don't know of any way to either log the order that they are mounted in at boot time, nor to inspect on a hypothetical "dry-run" basis what ZFS thinks the correct mount order is. I did not have this problem prior to the 11 -> 12 upgrade, if that matters. Here are my auto-mountable filesystems: # zfs get -rt filesystem canmount | grep -vw off NAME PROPERTY VALUE SOURCE electron canmount on default electron/data canmount on local electron/data/backup canmount on local electron/data/backup/jimsdesk canmount on default electron/system canmount on default electron/system/ROOT canmount on default electron/system/ROOT/default canmount on local electron/system/home canmount on default electron/system/tmp canmount on default electron/system/usr canmount on default electron/system/usr/local canmount on default electron/system/var canmount on default And their mountpoints: # zfs get -Hrt filesystem canmount | grep -vw off | while read FS b; do echo $FS; done | xargs zfs list NAME USED AVAIL REFER MOUNTPOINT electron 80.0T 403G 201K none electron/data 46.4T 34.0T 238K /data electron/data/backup 6.67T 34.0T 4.81T /data/backup electron/data/backup/jimsdesk 267G 34.0T 267G /data/backup/jimsdesk electron/system 13.4G 403G 201K none electron/system/ROOT 560M 403G 201K none electron/system/ROOT/default 373M 403G 279M / electron/system/home 292K 403G 292K /home electron/system/tmp 15.5M 403G 13.8M /tmp electron/system/usr 12.3G 403G 10.6G /usr electron/system/usr/local 1007M 403G 569M /usr/local electron/system/var 555M 403G 516M /var Let's look at electron/data and its children, although I suspect the same problem is happening with electron/system/usr and its child electron/system/usr/local. None of the electron/data tree is currently mounted: # zfs mount electron/system/ROOT/default / electron/system/tmp /tmp electron/system/var /var electron/system/home /home electron/system/usr /usr electron/system/usr/local /usr/local Since they all mount under /data, the /data directory is currently empty: # find /data -ls 1587 1 drwxr-xr-x 2 root wheel 2 Apr 18 15:33 /data Now let's mount the remaining auto-mount filesystems: # zfs mount -a; zfs mount electron/system/ROOT/default / electron/system/tmp /tmp electron/system/var /var electron/system/home /home electron/system/usr /usr electron/system/usr/local /usr/local electron/data /data electron/data/backup /data/backup electron/data/backup/jimsdesk /data/backup/jimsdesk Inspect /data and /data/backup: # ls /data backup/ slide/ xqueue/ spare/ vince/ mendel/ rain/ # ls /data/backup jimsdesk/ There should be a lot more stuff in /data/backup than that. It's only showing the mountpoint for electron/data/backup/jimsdesk. Here's the work-around to get it to show up: # zfs umount electron/data/backup/jimsdesk # zfs umount electron/data/backup # zfs mount electron/data/backup # find /data/backup -maxdepth 1 /data/backup /data/backup/.zfs /data/backup/rayleigh.CentOS-7 /data/backup/horton /data/backup/lannister /data/backup/rayleigh.archived /data/backup/Thunder /data/backup/luna /data/backup/bioinformatics /data/backup/shiny /data/backup/salmon /data/backup/rayleigh.CentOS-6 /data/backup/stark /data/backup/hoh2 /data/backup/jimsdesk /data/backup/love /data/backup/rayleigh So clearly the filesystem contents are still there, they were just invisible before. Since /data is a mountpoint for electron/data, we would expect it to be empty, and indeed we verified above that it WAS empty, prior to doing the 'zfs mount -a'. But instead I find: # zfs umount electron/data/backup # zfs umount electron/data # find /data /data /data/backup # df -m /data/backup Filesystem 1M-blocks Used Avail Capacity Mounted on electron/system/ROOT/default 413020 278 412742 0% / /data/backup should not exist in electron/system/ROOT/default. So I rmdir'ed it and rebooted. After the reboot, and after unmounting electron/data/backup/jimsdesk, electron/data/backup and electron/data it again was present. This seems like strong evidence that /data/backup is getting mounted before /data gets mounted, and ZFS is auto-creating the mountpoint directory. How can I definitively SEE the order in which ZFS will mount filesystems, either in a "dry-run" mode, or have it logged in real time during the boot process? Even more to the point, how can I ensure that /data gets mounted before /data/backup gets mounted before /data/backup/jimsdesk, and likewise /usr gets mounted before /usr/local? Thank you! Jim