From owner-freebsd-questions@freebsd.org Fri Apr 19 14:31:34 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC6E7156E1E9 for ; Fri, 19 Apr 2019 14:31:34 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.49.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A19BC81764 for ; Fri, 19 Apr 2019 14:31:33 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from mather.gromit23.net (c-98-244-101-97.hsd1.va.comcast.net [98.244.101.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 209AE497; Fri, 19 Apr 2019 10:31:27 -0400 (EDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: ZFS is auto-mounting in wrong order From: Paul Mather In-Reply-To: Date: Fri, 19 Apr 2019 10:31:26 -0400 Cc: Jim Long Content-Transfer-Encoding: 7bit Message-Id: <145D9F3E-A3F8-4153-A7CB-950F1A62F958@gromit.dlib.vt.edu> References: To: freebsd-questions@freebsd.org X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: A19BC81764 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=vt.edu (policy=none) X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; IP_SCORE(-1.51)[ip: (-3.92), ipnet: 128.173.0.0/16(-1.96), asn: 1312(-1.59), country: US(-0.06)]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[chumby.dlib.vt.edu,gromit.dlib.vt.edu]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; R_SPF_NA(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[97.101.244.98.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 14:31:34 -0000 Jim Long wrote: > I have a system that boots from ZFS. I recently upgraded it from > FreeBSD 11.2-STABLE to 12.0-STABLE: > > FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC amd64 > > I'm seeing certain directories as empty, when they should have lots of > content. The problem can be worked around by unmounting key > filesystems, and re-mounting them in the proper order. > > Troubleshooting leads me to believe that the ZFS system is not mounting > them in the proper order, but I don't know of any way to either log the > order that they are mounted in at boot time, nor to inspect on a > hypothetical "dry-run" basis what ZFS thinks the correct mount order is. > > I did not have this problem prior to the 11 -> 12 upgrade, if that > matters. I encountered this problem in upgrading from 11-STABLE -> 12-STABLE. Under 11-STABLE and prior, when upgrading in single-user mode I would use "zfs mount -a" to mount my ZFS file systems. Under 12-STABLE, this resulted in many "empty directories" as you mention. It caused the subsequent "make installworld" to fail. In my case, I discovered it was the way my boot environment had been set up that was causing this. The file systems under my boot environment "root" (pool/ROOT) were all marked as "noauto". Under 11-STABLE, "zfs mount -a" would successfully mount these file systems (even though the man page said it shouldn't). Under 12-STABLE, "zfs mount -a" acts in accordance with the man page and does not mount these file systems in response to "zfs mount -a". My workaround was now to mount ZFS file systems as the system does: first execute /etc/rc.d/zfsbe and then /etc/rc.d/zfs, i.e., "/etc/rc.d/zfsbe start ; /etc/rc.d/zfs start". That correctly mounts the active boot environment and then the rest of the ZFS file systems on top of it. I notice you have an "electron/system/ROOT" ZFS file system, so maybe you are using boot environments, too? Do you have any ZFS file systems marked "noauto" for the "canmount" property? It doesn't seem so, but I thought I would double-check. > This seems like strong evidence that /data/backup is getting mounted > before /data gets mounted, and ZFS is auto-creating the mountpoint > directory. > > How can I definitively SEE the order in which ZFS will mount > filesystems, either in a "dry-run" mode, or have it logged in real time > during the boot process? The "service -e" command will show you the order in which services will be started. In my 12-STABLE system, ZFS related startup scripts seem to run in this order: % service -e | grep zfs /etc/rc.d/zfsbe /etc/rc.d/zfs /etc/rc.d/zfsd The zfsbe service handles mounting of the active boot environment. The zfs service mounts the rest of the ZFS filesystems (the non-"noauto" ones, that is). (The zfsd service runs the ZFS fault management daemon, driven by devd events.) You could run these scripts manually via "/bin/sh -x" to trace what they are doing. I hope this helps. Cheers, Paul.