From owner-freebsd-fs@freebsd.org Wed Jul 17 03:37:44 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA1CCA2179 for ; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 87FCA8B009 for ; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 87958A2178; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 875C7A2177 for ; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 68B7E8B008 for ; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1F745187CF for ; Wed, 17 Jul 2019 03:37:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x6H3bhll046865 for ; Wed, 17 Jul 2019 03:37:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x6H3bhGq046856 for fs@FreeBSD.org; Wed, 17 Jul 2019 03:37:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 237517] ZFS parallel mounting sometimes misses mounting intermediate filesystems Date: Wed, 17 Jul 2019 03:37:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fullermd@over-yonder.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 68B7E8B008 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jul 2019 03:37:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D237517 --- Comment #12 from fullermd@over-yonder.net --- Now, bad news: I upgraded another system from a late-Dec stable/12 to last night's stable/12, and it also came up broken. Whee. But, good news: I've managed to track down how to reproduce it! It has to = do with having multiple ZFS FS trees mounted on the same point. This apparent= ly happened on several of my systems because I have a lot of pool/usr and pool/usr/local (mounted in the places you'd expect), but also pool/bsd/src = and pool/bsd/obj mounted on /usr/{src,obj}. So I grabbed the FreeBSD-12.0-STABLE-amd64-20190711-r349903.raw.xz snapshot= and booted it up in bhyve. I created a separate ~100 meg scratch zvol and also attached it in, as the backing disk for a test zpool (two -d's to /usr/share/examples/vmrun.sh) creatively named 'ztst'. And then worked up the repro script I'm attaching. Something like half the time, it "succeeds" in failing. It seems that the way in which the zfs create's write stuff out is what causes the variability. When it shows up = the failure in a run, repeated `service zfs restart`'s will (almost?) always continue to show the failure, but in runs where everything shows up fine the first time, they'll continue to show up fine. With any luck, this should let somebody familiar with the code quickly pinp= oint the cause of the issues. I assume you can just run the script 'till it fai= ls, then start instrumenting the crap out of zfs_foreach_mountpoint(). (obviously, you don't want to try running this on a ZFS root system, or Bad Things will probably happen the first time it runs "zfs unmount -a"!) An example "broken" run: # /try2_zfs.sh Created: /ztst/t2fs/afile /ztst/t2fs/somedir/afile /ztst/t2fs/somedir/anotherdir/afile /ztst/t2fs/somedir/subdir/afile Restarted: /ztst/t2fs/afile /ztst/t2fs/somedir/afile Restarted serially: /ztst/t2fs/afile /ztst/t2fs/somedir/afile /ztst/t2fs/somedir/anotherdir/afile /ztst/t2fs/somedir/subdir/afile --=20 You are receiving this mail because: You are the assignee for the bug.=