From owner-freebsd-questions@freebsd.org Fri Apr 19 19:39:36 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED2951576AAA for ; Fri, 19 Apr 2019 19:39:35 +0000 (UTC) (envelope-from freebsd-questions@umpquanet.com) Received: from g5.umpquanet.com (ns.umpquanet.com [209.216.177.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CAE678E983 for ; Fri, 19 Apr 2019 19:39:34 +0000 (UTC) (envelope-from freebsd-questions@umpquanet.com) Received: from g5.umpquanet.com (localhost [127.0.0.1]) by g5.umpquanet.com (8.15.2/8.15.2) with ESMTP id x3JJcuDx013591; Fri, 19 Apr 2019 12:38:56 -0700 (PDT) (envelope-from freebsd-questions@umpquanet.com) Received: (from james@localhost) by g5.umpquanet.com (8.15.2/8.15.2/Submit) id x3JJctfm013590; Fri, 19 Apr 2019 12:38:55 -0700 (PDT) (envelope-from freebsd-questions@umpquanet.com) X-Authentication-Warning: g5.umpquanet.com: james set sender to freebsd-questions@umpquanet.com using -f Date: Fri, 19 Apr 2019 12:38:55 -0700 From: Jim Long To: Paul Mather Cc: freebsd-questions@freebsd.org Subject: Re: ZFS is auto-mounting in wrong order Message-ID: <20190419193855.GA10389@g5.umpquanet.com> References: <145D9F3E-A3F8-4153-A7CB-950F1A62F958@gromit.dlib.vt.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <145D9F3E-A3F8-4153-A7CB-950F1A62F958@gromit.dlib.vt.edu> User-Agent: Mutt/1.11.4 (2019-03-13) X-Rspamd-Queue-Id: CAE678E983 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd-questions@umpquanet.com designates 209.216.177.146 as permitted sender) smtp.mailfrom=freebsd-questions@umpquanet.com X-Spamd-Result: default: False [-2.76 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.92)[-0.915,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; NEURAL_HAM_LONG(-0.99)[-0.989,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; DMARC_NA(0.00)[umpquanet.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: g5.umpquanet.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.54)[-0.536,0]; IP_SCORE(-0.01)[country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11233, ipnet:209.216.160.0/19, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 19:39:36 -0000 Thank you for taking the time to respond, Paul. On Fri, Apr 19, 2019 at 10:31:26AM -0400, Paul Mather wrote: > > I notice you have an "electron/system/ROOT" ZFS file system, so maybe you > are using boot environments, too? Do you have any ZFS file systems marked > "noauto" for the "canmount" property? It doesn't seem so, but I thought I > would double-check. No, I'm not using any 'noauto'. My original post shows all filesystems that are not 'canmount=off'. Also, my root FS electron/system/ROOT/default has no child filesystems, which seems to be what /etc/rc.d/zfsbe handles. > > How can I definitively SEE the order in which ZFS will mount > > filesystems, either in a "dry-run" mode, or have it logged in real time > > during the boot process? > > The "service -e" command will show you the order in which services will be > started. In my 12-STABLE system, ZFS related startup scripts seem to run > in this order: > > % service -e | grep zfs > /etc/rc.d/zfsbe > /etc/rc.d/zfs > /etc/rc.d/zfsd I don't use zfsd, and the core of what /etc/rc.d/zfs does is 'zfs mount -va', so I still have no visibility into how 'zfs mount' is ordering the filesystem mounts. But it is definitely 'zfs mount -va' that is doing the wrong thing. I've fashioned this test script to highlight the observable differences between a correct, manual mounting, and the incorrect mount order used by zfs mount -a: --snip-- #!/usr/bin/env bash # correct mount ordering: MANUAL=" zfs mount electron/data zfs mount electron/data/backup zfs mount electron/data/backup/jimsdesk " # automatic mount ordering: AUTO=" zfs mount -va " case $1 in manual) MOUNT="$MANUAL" ;; auto) MOUNT="$AUTO" ;; *) exit esac cat << EOF | sh -v zfs mount | grep '/data' mkdir /data find /data $MOUNT zfs mount | grep '/data' find /data | head -3000000 | wc -l zfs umount /data/backup/jimsdesk zfs umount /data/backup zfs umount /data rmdir /data find /data EOF --snip-- Here's a run of that script using the manual mounting order. The mountpoint /data is created fresh at the start. When the filesystems are correctly mounted, there are over 3 million files under /data. After unmounting, /data is empty, and can be rmdir'ed. --snip-- electron : 12:26:00 /root# ./zfs-mount-test.sh manual zfs mount | grep '/data' mkdir /data find /data /data zfs mount electron/data zfs mount electron/data/backup zfs mount electron/data/backup/jimsdesk zfs mount | grep '/data' electron/data /data electron/data/backup /data/backup electron/data/backup/jimsdesk /data/backup/jimsdesk find /data | head -3000000 | wc -l 3000000 zfs umount /data/backup/jimsdesk zfs umount /data/backup zfs umount /data rmdir /data find /data find: /data: No such file or directory --snip-- Here's a run of that script using the automatic mounting order. When the filesystems are automatically mounted, fewer than 1.5 million files are present under /data. After unmounting everything, /data is not clean, and rmdir returns an error. --snip-- electron : 12:26:17 /root# ./zfs-mount-test.sh auto zfs mount | grep '/data' mkdir /data find /data /data zfs mount -va zfs mount | grep '/data' electron/data /data electron/data/backup /data/backup electron/data/backup/jimsdesk /data/backup/jimsdesk find /data | head -3000000 | wc -l 1481445 zfs umount /data/backup/jimsdesk zfs umount /data/backup cannot unmount '/data/backup': not a mountpoint zfs umount /data rmdir /data rmdir: /data: Directory not empty find /data /data /data/backup --snip-- Is this a known issue? Should I submit a PR? Thank you! Jim