From owner-freebsd-fs@freebsd.org Fri Sep 6 10:44:57 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6859EFE1D for ; Fri, 6 Sep 2019 10:44:57 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46PvMS1zF7z4BHX; Fri, 6 Sep 2019 10:44:51 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 64A5222223; Fri, 6 Sep 2019 06:44:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 06 Sep 2019 06:44:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= florian-schulze.net; h=from:to:cc:subject:date:message-id :mime-version:content-type:content-transfer-encoding; s=fm1; bh= HNGgstATYar7EFWr3VYoJ4c789KV29q73rqXi3ozWGQ=; b=0/VsvBn1caH4gF7I Um7WRel21kPS7qvaDr2e0V7tSRxs2mSpRwFsaA5SYi8XwyF+mVxxVOf1gVyT22HS TrA5OPn6fhUKD/Ys8ggCdM88nbt15Is33woIbhykUAt2KV7mF/ceufjd9cpxnmz4 7JegGLpPEU6/bv1rueZ1Gy4AC2ya6BXTw+bQCO3iF2SzL1F9Eh0cIgnMhd7+uonV vdeWOVH71D0gEGolHEyCFDz1LOS0LfUChLu9co7C/B2anDKJciuQ8JTYJ7TML/zJ Rz8JIhanDXBA//LkAEVubRuDYyLX0quWoGQYSgHfkTqeyEckyUAt01yuVWbQT4Tu VuHOug== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=HNGgst ATYar7EFWr3VYoJ4c789KV29q73rqXi3ozWGQ=; b=vfBFGKXY2tQi6tLXOCI/zj yjVQEV7HvEE4m9UaRuF9Egt8p6hvLLkeyjPwuWTk0sv7WyaAln5gPNbuj9xzEqTj xv/OSdwTppBVFvctw5qLto0K/nHpVh+XRBDRicCx/EF7Lg1xIxfujfa9tbo0RU/K cPVBhAxPyXqJEMtrCd1xWgfHnxHHkfYC7aTFzUTJX9lwftILGyG0ZN89j+UkRUmX mp4c0BPlroz2DbD1NF+gs7FugD6TwUMsMZ76PoTDWPBIj2KVbMUY1K0SYyl7/gNQ YeFVjWsbE+EYiofHGOpnz1r/rCVYnIGVla5FUR3CDL3zHzxuSzonpQQMY6awGTmA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrudejledgudelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffokfggtgfgsehtqhhmtd ertddtnecuhfhrohhmpedfhfhlohhrihgrnhcuufgthhhulhiivgdfuceomhgrihhlsehf lhhorhhirghnqdhstghhuhhliigvrdhnvghtqeenucffohhmrghinhepghhithhhuhgsrd gtohhmnecukfhppeejledrvdeghedrheekrddukeejnecurfgrrhgrmhepmhgrihhlfhhr ohhmpehmrghilhesfhhlohhrihgrnhdqshgthhhulhiivgdrnhgvthenucevlhhushhtvg hrufhiiigvpedt X-ME-Proxy: Received: from [192.168.1.182] (p4ff53abb.dip0.t-ipconnect.de [79.245.58.187]) by mail.messagingengine.com (Postfix) with ESMTPA id 230038005C; Fri, 6 Sep 2019 06:44:50 -0400 (EDT) From: "Florian Schulze" To: freebsd-fs@freebsd.org Cc: kib@freebsd.org, mgj@freebsd.org Subject: held file reference issue with ZFS and nullfs Date: Fri, 06 Sep 2019 12:44:46 +0200 X-Mailer: MailMate (1.12.5r5635) Message-ID: <45B080A9-DE0F-4633-91F8-71438408D4B8@florian-schulze.net> MIME-Version: 1.0 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable Autocrypt: addr=mail@florian-schulze.net; keydata=mQENBFliSUoBCADLKPuy1qhTI+A3XRleyZ7WlXhRfYiXBCSMiIqUb+nx88xKLVAclRJqMnqlJHzAJcb4JW9i3riRU+V1rVG9BDaBCCih5Si0BQO5W84SpaadTdcE3rUO8Ihdxc6dJN9DiAtp26WyO1fXSA2NkwpakIerxlF91knkrnSa/VgLrlB+Ahtcisbsn9XYSnv0wr/njQJOfJMsmJfWAKXWWaWDWb0Z4/UITsBe259Ppkf6bTYS3j7oCPqygPzdyc14ILINb0N3ksT3tKLiS4b2gh7RAO4uf/C8TvlZSYvoX/sel5xFFxcgmqG+V0d4jjhdvY5zFHU+/Egr5y3HPgSQjevEHsNxABEBAAG0KkZsb3JpYW4gU2NodWx6ZSA8bWFpbEBmbG9yaWFuLXNjaHVsemUubmV0PokBOQQTAQgAIwUCWWJJSgIbAwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEC0N2jSLcB7qWo4H/jEgS6Id1TkRtlh+vZanpswS83sH8pOfc20qHbPEM2a1pk9aEgvvnckaeLiGVYaHKZccxRaE2E6zGKW26sI5WPcMehCZkiYf4zxfoppWsWDIGAsuptByHT+etkQVEANwNdDM5hIn4Hq0e6gsH1zu68SBhH4FtW4+ukCK3WrHZ0QJBoECU1otWxhP5BqOO8h9OCY2vESxHC2nH+y3vsqVL43JQWmBUv7uaaX4OusdWUZCW8jlYnWd//cmt3RKHh4PCKY7JK0itzoUi+DnnHBN7jxz2FFjX3+BinUeZX3bpXQhw3Xsv1ZXW9aeyGLgzYOuLdBSXmXJcetyJ0heCrOwZUS5AQ0EWWJJSgEIANlJdoChOEMMB7vh3YB0HjrBpK2+iyWC1Y1VCWOpKg/PiYq5GmZDMh1SGq+s/3ZVG5kAYcSKqdHSBqC8sRjY/gU/hysR3w9IU0J5/j1jTFZsw09vTfloEVRAJaAEyOAKvgUw+ fwiGhJsJPS9W8dqieVovqft5DpzNIBztPJxfiDrlHvKbxrwyCEW9+EAHMVJf7F6wqp0rAnlxZQujasL7Urx1Kl0RKT9HT8QHyzTJFpbpuO0hApiWCWD5GCy3AIc0KK3jJIz3ODCVFSGhbLiV0j9FB4j3wmM5JSvsyMhLZAkaFy/kqmlaVeroLWMBbmKUfEA968b37O+kILc3iXYQmMAEQEAAYkBHwQYAQgACQUCWWJJSgIbDAAKCRAtDdo0i3Ae6oyuB/4rh42bgTYncPkMv0l9jsKTB61wwe2Imxfxd52Gk3Ev5slaSFYl2r2FwhTFCX5Ay5XXqoueIG2AlZY2do7jnuLW66dye4GAI67PgKkZfzWJqD+q3bOimnmdlyKM1x2AxGnwPBEdjc4sShFm/uPOPPGJu4f8I9PKjGI62RJUCGHN9KxYJclomhNYe6n1xDsF0e3lDEtWn+KL1WesHO/NY2Mv//qGs9dqIu7rdAXrXiBQaSUz8ol0o4Z924qEWX+23tlWn3XLZPbFPJAB8sPHKWapt3nR/zaPJqfqkqwKWnWFdiKaKIB9GhlXmBlq+0alpjNjbicGlOF1YGF4agbFyoCb X-Rspamd-Queue-Id: 46PvMS1zF7z4BHX X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=florian-schulze.net header.s=fm1 header.b=0/VsvBn1; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=vfBFGKXY; dmarc=none; spf=pass (mx1.freebsd.org: domain of mail@florian-schulze.net designates 66.111.4.26 as permitted sender) smtp.mailfrom=mail@florian-schulze.net X-Spamd-Result: default: False [-4.53 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[florian-schulze.net:s=fm1,messagingengine.com:s=fm3]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_MISSING_CHARSET(2.50)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[florian-schulze.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[florian-schulze.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.979,0]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[26.4.111.66.list.dnswl.org : 127.0.5.1]; RECEIVED_SPAMHAUS_PBL(0.00)[187.58.245.79.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.45)[ip: (-9.69), ipnet: 66.111.4.0/24(-4.84), asn: 11403(-2.68), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Sep 2019 10:44:57 -0000 Hi! Since FreeBSD 12 (updated from 10.3, I skipped 11.x completely, the box = started around 9.3) I have the issue that ZFS is not freeing up space = for some deleted files. The filesystems where this happens are mounted = into multiple jails via nullfs. Only one jail has write access, the = others are read only. When files are deleted the space for them is not = freed. I can still see their objects via zdb. When I unmount one of the = read only nullfs mounts the space is freed and the objects released. I already used lsof, procstat and fstat to see if any process still has = a reference to the file, but that is not the case. But it seems to = matter which nullfs mount is unmounted, it is always one of the read = only ones. The processes which access the read only mounts are = completely different, it only seems to matter that the files are opened = at all. Killing the processes doesn't help, only unmounting the nullfs. Today I noticed an odd message when I used zfs diff: "Unable to = determine path or stats for object 6 in = =2E..@zfs-diff-15651-00000001d86eb8cb: Stale NFS file handle". I don't = have NFS enabled anywhere (just checked the properties) and it never was = enabled! The zdb output for object 6: Dataset ... [ZPL], ID 18405, cr_txg 36040566, 10.2G, 43 objects Object lvl iblk dblk dsize dnsize lsize %full type 6 1 16K 16K 16K 512 32K 100.00 SA attr = layouts ZFS_DBGMSG(zdb): spa_open_common: opening ... spa_load(tank2, config trusted): LOADING disk vdev '/dev/diskid/DISK-WD-...': best uberblock found for spa tank2. = txg 40503599 spa_load(tank2, config untrusted): using uberblock with txg=3D40503599 spa_load(tank2, config trusted): spa_load_verify found 0 metadata errors = and 2 data errors spa_load(tank2, config trusted): LOADED In the zfs diff was also a line "- ...(on_delete_queue)". I have one zfs filesystem where this happens quite often, one were it = happens sometimes and a few others which have a similar setup and where = I never noticed it (though the average file size on them is smaller). I asked in #freebsd about this and koobs said I should write to this = list and CC kib@freebsd.org and mgj@freebsd.org He also did a quick look at the nullfs changes between 10.3 and 12.0 and = spotted the following change, which he said I should mention as well: https://github.com/freebsd/freebsd/commit/82f9c275c43da09f404546cceeff187= a90ecc573#diff-81e7d6520611101890dd6425324dd8f8 Is there a known bug there? Could the stale NFS handle cause the leak? = Where is that NFS handle coming from? Regards, Florian Schulze