Date: Fri, 06 Sep 2019 15:18:24 +0200 From: "Florian Schulze" <mail@florian-schulze.net> To: "Konstantin Belousov" <kostikbel@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: held file reference issue with ZFS and nullfs Message-ID: <5B1BEA13-F8CB-4E3C-9236-629892AF48F5@florian-schulze.net> In-Reply-To: <20190906124336.GE2559@kib.kiev.ua> References: <45B080A9-DE0F-4633-91F8-71438408D4B8@florian-schulze.net> <20190906124336.GE2559@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi! On 6 Sep 2019, at 14:43, Konstantin Belousov wrote: Thanks for the reply ... > On Fri, Sep 06, 2019 at 12:44:46PM +0200, Florian Schulze wrote: >> Hi! >> >> Since FreeBSD 12 (updated from 10.3, I skipped 11.x completely, the = >> box >> started around 9.3) I have the issue that ZFS is not freeing up space >> for some deleted files. The filesystems where this happens are = >> mounted >> into multiple jails via nullfs. Only one jail has write access, the >> others are read only. When files are deleted the space for them is = >> not >> freed. I can still see their objects via zdb. When I unmount one of = >> the >> read only nullfs mounts the space is freed and the objects released. >> >> I already used lsof, procstat and fstat to see if any process still = >> has >> a reference to the file, but that is not the case. But it seems to >> matter which nullfs mount is unmounted, it is always one of the read >> only ones. The processes which access the read only mounts are >> completely different, it only seems to matter that the files are = >> opened >> at all. Killing the processes doesn't help, only unmounting the = >> nullfs. > There were some bugs in past where nullfs referenced a lower vnode but > did not dereferenced it. Could you be more concrete about "past"? In the past of the 12.0 = version, or during 11.x? >> I have one zfs filesystem where this happens quite often, one were it >> happens sometimes and a few others which have a similar setup and = >> where >> I never noticed it (though the average file size on them is smaller). >> >> I asked in #freebsd about this and koobs said I should write to this >> list and CC kib@freebsd.org and mgj@freebsd.org >> He also did a quick look at the nullfs changes between 10.3 and 12.0 = >> and >> spotted the following change, which he said I should mention as well: >> https://github.com/freebsd/freebsd/commit/82f9c275c43da09f404546cceeff= 187a90ecc573#diff-81e7d6520611101890dd6425324dd8f8 >> >> Is there a known bug there? Could the stale NFS handle cause the = >> leak? >> Where is that NFS handle coming from? > > So what is the exact version of your system ? If 12.0, upgrade kernel > to latest stable/12 and see if it helps with the leak. I'm on 12.0-RELEASE-p3. I will probably be able to try 12.0-RELEASE-p10 = during the weekend. Not sure I will find the time to try and compile my = own kernel, or is there a way to do it using freebsd-update by using a = snapshot? If not, I'd probably rather wait for 12.1. Regards, Florian Schulze
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5B1BEA13-F8CB-4E3C-9236-629892AF48F5>