From owner-freebsd-stable@freebsd.org Mon Aug 14 12:07:46 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE1A8DC2836 for ; Mon, 14 Aug 2017 12:07:46 +0000 (UTC) (envelope-from tsuroerusu@gmail.com) Received: from mail-vk0-x232.google.com (mail-vk0-x232.google.com [IPv6:2607:f8b0:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 987E97E2FF for ; Mon, 14 Aug 2017 12:07:46 +0000 (UTC) (envelope-from tsuroerusu@gmail.com) Received: by mail-vk0-x232.google.com with SMTP id j189so30702077vka.0 for ; Mon, 14 Aug 2017 05:07:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=DWKYB4RTBKuL+8bRkalrsduMiCE7ysLiY46MzgJVgvU=; b=FAQdy6ZbcRu9FEQNCrknBPvXbDby4GGsSaMItZenrH8JrH3A9Zx/T25SM6rCn0gPYs xtPnoeIyPLcmq4BijWmC+CpO9MNcFSV3l46v8eqSdWKK4g6w2biZnn6kKn/N2RMohEoJ f3Yz6Ga5ghx2Vmegk7gk9dpVekpCLJZ7vUBgl316VZntiTqhda1iosV2in3B53tx8Y3p ntcv2kGzvVRKGKNmwI0Ayky+mY7CkiwS3fe/+rOkFDOtMoU0SOgwmMy1OebFdgAWRB0T I9OqU2SuM+LJbnLUxx4mavit4410cwpmprKvEkrPUGvlW5GEoU8yae9R97aLHWC7uqeK 4MWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=DWKYB4RTBKuL+8bRkalrsduMiCE7ysLiY46MzgJVgvU=; b=nNf85yCWsoI5psY0rQIv5rI2geGqldyO82fdIF1PDtzQn3r+JvU6bK1D6XSDxLmP3z 7Msre2YU3GFwAwu28KHr1ciVtj6TnPPFBSipBT87Qs4D72M1JfzVqresF0rbi7MZcwy+ ndjVD5cc/vljNDd21MYPF+5nrJCA87biCcDe2DQ1Nf8Ass1RQog3X9OqRbqQsnVAYGRS s6GLQ/3UNGGrIJgb6MJcK1QLeqCDZSiOiAX/UMKoETFxXqrm00U86olO9wePhYS/CCYU vsn1OybeANOGRYJ39QIQk3AVGsnUojhzTE00z9moHtgS6r1jppvFQuaFzoF1Yg3k+FRp V+xg== X-Gm-Message-State: AHYfb5iUSYWGu3XLJpM7svmsolxo4T2E//Ffhc80/+wJnxAATSTDp8py N4TIE39Pq8+IJLmOMnmdOzpYO3cZtigs X-Received: by 10.31.213.70 with SMTP id m67mr6714470vkg.144.1502712465428; Mon, 14 Aug 2017 05:07:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.157.79 with HTTP; Mon, 14 Aug 2017 05:07:44 -0700 (PDT) In-Reply-To: References: From: tsuroerusu@gmail.com Date: Mon, 14 Aug 2017 14:07:44 +0200 Message-ID: Subject: Re: nullfs making contents of ZFS datasets invisible even after unmounting To: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Aug 2017 12:07:47 -0000 With a little bit of help, I managed to figure out what the problem was. It turned out that I had run, head first, into the wall of nullfs not operating across file systems. Thus inside the jail, the data was being saved to a regular directory on my "storage/cloud"-dataset (Mountpoint: /storage/cloud), whereas outside the jail the child file system "storage/cloud/bc", having been left empty, was mounted over the folder "bc" on "storage/cloud", where the files actually resided, thus "hiding" them from view outside the jail. A simple "zfs unmount storage/cloud/bc" revealed them. Anybody reading this in the future: If you are using ZFS and want to put a file system and all of its children into a jail then you cannot use nullfs. Instead you need to either change the actual mount point using "zfs set mountpoint=/jails/jail1/storage storage/path/to/filesystem" or to attach the entire dataset to the jail, and allow it to be managed from within the jail using "zfs jail". 2017-08-14 10:34 GMT+02:00 : > I am running a FreeBSD 11.1 system with ZFS and jails and I mount a > file system on my storage pool (/storage/cloud) into the jail > (/jails/cloud/storage) via nullfs, and that works fine for what the > jail does. However I just noticed that outside the jail, I can only > see the mount points of the file system children, but not the > contents. > > For example if I do: ls -la /storage/cloud/* (As root) then nothing > shows up, but they are present inside the jail. Even if I stop the > jail, they still are not present outside the jail. > > I just tried disabling jails in /etc/rc.conf and rebooting the system > and EVEN THEN the files do not reappear outside the jail in their > original location, yet zfs list reports the space being used. But if I > then (without rebooting) enable the jail, the files show up in the > nullfs-mounted location inside the jail! > > As an experiment, I tried creating /mnt/test and /mnt/test2 and > creates a folder and a few files in test, and then nullfs-mounted it > onto test2, and I could then see the files in both locations. > > Does anybody know what might be causing this? The fact that my data > stays hidden after a reboot is rather worrisome.