From owner-freebsd-fs@freebsd.org Fri Sep 6 13:18:31 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C5F93F3C21 for ; Fri, 6 Sep 2019 13:18:31 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46Pymj567Sz4MWX for ; Fri, 6 Sep 2019 13:18:28 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 461AB21EEB; Fri, 6 Sep 2019 09:18:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 06 Sep 2019 09:18:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= florian-schulze.net; h=from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-type :content-transfer-encoding; s=fm1; bh=We54kscZEd2ao2LVxNpD1mKz1R rs1jFah/dvZH9mnuM=; b=5SBK15+T8tSmhMOLhn2NSL26nGH6D0BNz4OrHXgj3c IuJPhpx3HRzIXw7pEw/42HO2hqZ7bc28M1FTizD1W6WhuwaPMQ1VHgh8dqBv3Sqw bkOLQbn2OuzUUGFkDFv7Bjns2U/eT0DgM7Uas9oes+jfZ97CODCVYQTpv4bnqbQh N+tmsAPXvNeqQtEHL8XZnkBMf5Bm2HYgvJCM/3fPNLN6HlxFzkTrTVEO8sfFI0GR BiOgqX1AbxShDnzj2/mtpHlY6EPPuxC4EvBX/0dOyk9sPy5kpea+xZhpwbHJKBEj kv9CAd5DhaP4wkHlTIJsXkzBeLrXOq6TUgLhlB2ENVVg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=We54kscZEd2ao2LVxNpD1mKz1Rrs1jFah/dvZH9mn uM=; b=FjjZtM64bhqOthIvuiozGqxGXe0xL9c1cdpKejD8mcBRTgTN9MOk/EJoY QXMD9ZS/Yzdm0xhtb/cU36dYJwu11wonW9Fll8tMWK5BIe2RW9sZEvs1Og7OTXVa 6KDSDgDT12GILfGduFf9CYcNdhqlb9Ewbgv3m1SCGhuBbOzJR/BUn9DGgAkXvxXc Bk6QNLv0UMGuhe3Q5BiY94IF4tZlUF0x5eQQYCG9zEotzDVH5D+Bw65lfjIkt3PT otIdn6B0sPDpAgtrpHiCBV1YNsJ34yND5iX2WTl31AUW8s4S8ox1abErQaoEV1Jn 5oOsPnIwna80s+vV/1L3XzLz2tJPQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrudejledghedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffokfgjfhggtgfgsehtqh hmtdertddtnecuhfhrohhmpedfhfhlohhrihgrnhcuufgthhhulhiivgdfuceomhgrihhl sehflhhorhhirghnqdhstghhuhhliigvrdhnvghtqeenucffohhmrghinhepghhithhhuh gsrdgtohhmnecukfhppeejledrvdeghedrheekrddukeejnecurfgrrhgrmhepmhgrihhl fhhrohhmpehmrghilhesfhhlohhrihgrnhdqshgthhhulhiivgdrnhgvthenucevlhhush htvghrufhiiigvpedt X-ME-Proxy: Received: from [192.168.1.182] (p4ff53abb.dip0.t-ipconnect.de [79.245.58.187]) by mail.messagingengine.com (Postfix) with ESMTPA id 539D7D60057; Fri, 6 Sep 2019 09:18:27 -0400 (EDT) From: "Florian Schulze" To: "Konstantin Belousov" Cc: freebsd-fs@freebsd.org Subject: Re: held file reference issue with ZFS and nullfs Date: Fri, 06 Sep 2019 15:18:24 +0200 X-Mailer: MailMate (1.12.5r5635) Message-ID: <5B1BEA13-F8CB-4E3C-9236-629892AF48F5@florian-schulze.net> In-Reply-To: <20190906124336.GE2559@kib.kiev.ua> References: <45B080A9-DE0F-4633-91F8-71438408D4B8@florian-schulze.net> <20190906124336.GE2559@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable Autocrypt: addr=mail@florian-schulze.net; keydata=mQENBFliSUoBCADLKPuy1qhTI+A3XRleyZ7WlXhRfYiXBCSMiIqUb+nx88xKLVAclRJqMnqlJHzAJcb4JW9i3riRU+V1rVG9BDaBCCih5Si0BQO5W84SpaadTdcE3rUO8Ihdxc6dJN9DiAtp26WyO1fXSA2NkwpakIerxlF91knkrnSa/VgLrlB+Ahtcisbsn9XYSnv0wr/njQJOfJMsmJfWAKXWWaWDWb0Z4/UITsBe259Ppkf6bTYS3j7oCPqygPzdyc14ILINb0N3ksT3tKLiS4b2gh7RAO4uf/C8TvlZSYvoX/sel5xFFxcgmqG+V0d4jjhdvY5zFHU+/Egr5y3HPgSQjevEHsNxABEBAAG0KkZsb3JpYW4gU2NodWx6ZSA8bWFpbEBmbG9yaWFuLXNjaHVsemUubmV0PokBOQQTAQgAIwUCWWJJSgIbAwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEC0N2jSLcB7qWo4H/jEgS6Id1TkRtlh+vZanpswS83sH8pOfc20qHbPEM2a1pk9aEgvvnckaeLiGVYaHKZccxRaE2E6zGKW26sI5WPcMehCZkiYf4zxfoppWsWDIGAsuptByHT+etkQVEANwNdDM5hIn4Hq0e6gsH1zu68SBhH4FtW4+ukCK3WrHZ0QJBoECU1otWxhP5BqOO8h9OCY2vESxHC2nH+y3vsqVL43JQWmBUv7uaaX4OusdWUZCW8jlYnWd//cmt3RKHh4PCKY7JK0itzoUi+DnnHBN7jxz2FFjX3+BinUeZX3bpXQhw3Xsv1ZXW9aeyGLgzYOuLdBSXmXJcetyJ0heCrOwZUS5AQ0EWWJJSgEIANlJdoChOEMMB7vh3YB0HjrBpK2+iyWC1Y1VCWOpKg/PiYq5GmZDMh1SGq+s/3ZVG5kAYcSKqdHSBqC8sRjY/gU/hysR3w9IU0J5/j1jTFZsw09vTfloEVRAJaAEyOAKvgUw+ fwiGhJsJPS9W8dqieVovqft5DpzNIBztPJxfiDrlHvKbxrwyCEW9+EAHMVJf7F6wqp0rAnlxZQujasL7Urx1Kl0RKT9HT8QHyzTJFpbpuO0hApiWCWD5GCy3AIc0KK3jJIz3ODCVFSGhbLiV0j9FB4j3wmM5JSvsyMhLZAkaFy/kqmlaVeroLWMBbmKUfEA968b37O+kILc3iXYQmMAEQEAAYkBHwQYAQgACQUCWWJJSgIbDAAKCRAtDdo0i3Ae6oyuB/4rh42bgTYncPkMv0l9jsKTB61wwe2Imxfxd52Gk3Ev5slaSFYl2r2FwhTFCX5Ay5XXqoueIG2AlZY2do7jnuLW66dye4GAI67PgKkZfzWJqD+q3bOimnmdlyKM1x2AxGnwPBEdjc4sShFm/uPOPPGJu4f8I9PKjGI62RJUCGHN9KxYJclomhNYe6n1xDsF0e3lDEtWn+KL1WesHO/NY2Mv//qGs9dqIu7rdAXrXiBQaSUz8ol0o4Z924qEWX+23tlWn3XLZPbFPJAB8sPHKWapt3nR/zaPJqfqkqwKWnWFdiKaKIB9GhlXmBlq+0alpjNjbicGlOF1YGF4agbFyoCb X-Rspamd-Queue-Id: 46Pymj567Sz4MWX X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=florian-schulze.net header.s=fm1 header.b=5SBK15+T; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=FjjZtM64; dmarc=none; spf=pass (mx1.freebsd.org: domain of mail@florian-schulze.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=mail@florian-schulze.net X-Spamd-Result: default: False [-4.57 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[florian-schulze.net:s=fm1,messagingengine.com:s=fm3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[florian-schulze.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[florian-schulze.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.990,0]; IP_SCORE(-3.48)[ip: (-9.81), ipnet: 66.111.4.0/24(-4.84), asn: 11403(-2.68), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1]; RECEIVED_SPAMHAUS_PBL(0.00)[187.58.245.79.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Sep 2019 13:18:31 -0000 Hi! On 6 Sep 2019, at 14:43, Konstantin Belousov wrote: Thanks for the reply ... > On Fri, Sep 06, 2019 at 12:44:46PM +0200, Florian Schulze wrote: >> Hi! >> >> Since FreeBSD 12 (updated from 10.3, I skipped 11.x completely, the = >> box >> started around 9.3) I have the issue that ZFS is not freeing up space >> for some deleted files. The filesystems where this happens are = >> mounted >> into multiple jails via nullfs. Only one jail has write access, the >> others are read only. When files are deleted the space for them is = >> not >> freed. I can still see their objects via zdb. When I unmount one of = >> the >> read only nullfs mounts the space is freed and the objects released. >> >> I already used lsof, procstat and fstat to see if any process still = >> has >> a reference to the file, but that is not the case. But it seems to >> matter which nullfs mount is unmounted, it is always one of the read >> only ones. The processes which access the read only mounts are >> completely different, it only seems to matter that the files are = >> opened >> at all. Killing the processes doesn't help, only unmounting the = >> nullfs. > There were some bugs in past where nullfs referenced a lower vnode but > did not dereferenced it. Could you be more concrete about "past"? In the past of the 12.0 = version, or during 11.x? >> I have one zfs filesystem where this happens quite often, one were it >> happens sometimes and a few others which have a similar setup and = >> where >> I never noticed it (though the average file size on them is smaller). >> >> I asked in #freebsd about this and koobs said I should write to this >> list and CC kib@freebsd.org and mgj@freebsd.org >> He also did a quick look at the nullfs changes between 10.3 and 12.0 = >> and >> spotted the following change, which he said I should mention as well: >> https://github.com/freebsd/freebsd/commit/82f9c275c43da09f404546cceeff= 187a90ecc573#diff-81e7d6520611101890dd6425324dd8f8 >> >> Is there a known bug there? Could the stale NFS handle cause the = >> leak? >> Where is that NFS handle coming from? > > So what is the exact version of your system ? If 12.0, upgrade kernel > to latest stable/12 and see if it helps with the leak. I'm on 12.0-RELEASE-p3. I will probably be able to try 12.0-RELEASE-p10 = during the weekend. Not sure I will find the time to try and compile my = own kernel, or is there a way to do it using freebsd-update by using a = snapshot? If not, I'd probably rather wait for 12.1. Regards, Florian Schulze