From nobody Wed Nov 12 13:21:46 2025 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d63vL1grrz6H010 for ; Wed, 12 Nov 2025 13:21:18 +0000 (UTC) (envelope-from anthony.pankov@yahoo.com) Received: from sonic305-20.consmr.mail.ir2.yahoo.com (sonic305-20.consmr.mail.ir2.yahoo.com [77.238.177.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4d63vJ5cWSz3m1S for ; Wed, 12 Nov 2025 13:21:16 +0000 (UTC) (envelope-from anthony.pankov@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=MEyzW39y; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of anthony.pankov@yahoo.com designates 77.238.177.82 as permitted sender) smtp.mailfrom=anthony.pankov@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1762953673; bh=JLwSxOQV577wYk6A9F50v8mkPmgq0N+sERuXBYDAHG8=; h=Date:From:To:Subject:References:From:Subject:Reply-To; b=MEyzW39yduifBpV3rwWyLvpV/rwMd91MzkRERFYTY6y1UrBINtg/4CsMfmFzoFrzFz48KKMwXDN0ALIj3zOuv8lwjAcgZZF1iXn3aLpXIPyD1zPNi85A8bmXUS6q2Go2iX4LCuWAQh14x3yQe6zv/GehVHYPA7rNE4X2cb4m1sKVhHqA6Md52foFu10u6vqxEzQZyHA1Z+aqZm10i+IGJvB6SGSv9+9x8Lt8MbArYHs8LUkLBrmgnm2204w81+8L3YdBBWhD9Z4GPSX8j6ls/9oGG2ZKs/rFQvsM55wTtykVfpbgSJdVbsWczCc+tZZtxRWAc2ij5X7qpKUvHoRwMQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1762953673; bh=JNAOMkFvRna77/sio/ZPaO/OLY1n7DqUbcyLEn4YyqJ=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=cKuCoQialOYJyY/hnt7rHxUd9bRntJVyVL4qdo2SEHLa/UU8HduG6gbQaI9eVo+iKeFXKHfsWF7l9pg1FtyAsQ8xib0rpZhEXC9DDCFEgsLTTPwn7lUILdKv82/8/6Hgpb8Ko//+GTgvbjT57ZV9Xx/FqI4x+O8gZ/XsPhMhIRAXCyZyrskfBlT/6Qm0eDNZgwKhvp0ZAxQQEXAlKjU4JYxQk73reQZemKo1iy52A2IyXPSiTKOAs+rHmmZbxbZ4xz7jRwgFpcaJ4NUp95kqd1VZOCrXN6bijdWNe+glRUkV7Jt0W4S2BLC03U3qLnffhxc6WNDRfRokWGw1qRv/ew== X-YMail-OSG: pku_2tMVM1lOEkh8Y.2bWfz_BVbAlG50QH84DG4OvL4OpMccX7PXfJHX7uhy7.r hFmCqLFqqS2AfPZVpH_iFhi5v_komm.iYyHPP2xklp9zXSuyMb2W.sCuGE.4k24vnwpfRsBSfCNT BR7DGoRIhtXG6H052WPR4C6rqqFWdQF0JsnbmI14QijCOP3mkdrkj30R4sr133yDZsfcpcqt_6j0 3qcQOfXH7SpwJFm_fzWPpW1w95uqbfg6QLhNr7Bf5TfC_Zd3BMRcc8gf91mmxzETvKG2WqlEtiS. PCK7S6IwV0fv06Usl6OhmonKnCjD989f.K0NjP1FikQ0N16SYSQV8__mTU24o9rcaKtE4jDq6Sk7 Z.uCLOTq33DMzfhhsPgTyVMx24AMPeTy3Y.XMUvwdZGPqS6wO4vUKeSywyi7G9lSbHKRFyAzq32p QO9eqC50XsY3CIboHTkahD1W7RqDtc5kdzidN51fu3YOB7sj.5j0EYYMISvXIdC3_P3pKpbX7OTm IefJ2lTq_v9CvO5BoA8VbZYwmHJ5E0L3s.AUfinqy1URnbtWgYL09yKjFrPxmhrjtBfpqIMq9R9O ET1gCxXw7boe9z3I_BazL6cwcl9JFF2Gy6mMLJQd9hIG3QUjOHzlHg9NiW304XH.uPYh47zUjA1S 4NhNn2q9zGEhiUXyG.jtNim4aG6GNv7baFcRURRHxFtv5zCKQXR4lAyf2iUkgS8QcUFacn8LbWAO kbSeL7vhu1CWmxm1XkCbl.Cx4whGYmTHNaKf__ZOVPQoeaccoDhC2Uo8RwXa3I_I14kbE9XEwSpQ REKI8jULu.bLHqjU9JU74zo5rtGrEvNqdTCAHw7sw.EdCiwJhET0qaC0AMGhr81zgNvbIFoiC8gL xPuy7xv7uf3hxYvOmttDjj2zSPmI2MQRy4Poj0zBGNx3xmbmOphFJENU8aMxBzqobVCyPPXcBzc3 PjKayujKqUgh_ksEjdasl6q.KdTv60YTwlgc5Sb1Sl1v.FUocKnaY9PNSs9MG.a6kRPj_rYn861A IzwSq_.jZNR4SUi7uE59whupLylIwx.5Mc9bMQBBwK2L8qaIbRSFXYb1Gce3hFtrrpfD7Z46NUCt _tU5RsCL6d_7ye4fl6ttdDXoLxhXBb5LDgB.L.2DJZg4G0SrIfgiukZ5S_3YxjY4myKxcYYmqpIR .z1p0vAUTAsASzTjZYbJ91Nyu8ja_vKpbhCjIfHWwjfQvzeaRjTvGZrAe0yuPFisnR7Q3G4rq7Ll ltVahdgfO3aGNBYoJIqxFhEmvpd0sc1bOpovVwyjxxbiYkM8lzJpe6HtFvk7eG.v5llMdPadeiKS MwaDmnrWXWzKzz7keDuThLc9_g7Am1tbpC2NmgKnHs5cB2Q_9Z4h3wbnaFyN2d.WmzzTgBM7ISNS jBX78t9Kim4Extw7Zfr_ODaO7gtcHEqzBruAux_qIGYinHjkz8hufWE45.wPY9Qlrv5mvZ5kqDDC mG3Fzk.H6I7QnWe_oVKrGylstMYnEorgDf0rGUsJJ.AL29C1L1OpSp6Swro5tg64z7kI_daMDFxR G1WRTiSInovhudFXfapDT3pkBVNwxbK5ym5Bd0NUU2doOEJotcJCuwN1j3YblZX9lru45TXbop6O rkS1lUV.ct31yUzV0KKFr39ufFscm2CG9RgHqrTNaHALec9vs56lKr7WysUQ7Rb7ZsLRp4sqE4HU bgjjmVhkuxq1tFQFLLLGjmPdCiLvW2j2bnHJI89XCqbPTyFGTKQw1dvIHQ59Zcg1hJjdbpM_Cglq e5GbBI36YddTgBcJCdgl_0TI9TzqGrFlyu92ZxI0C6nJE8eqoI9UScEb257vk1zjxc7IbfVODiAQ To28oAZh2tiI9WBuQprzBkn8ZT2_hgdwIKuWa3divtmWfV7yrim1HnsCXcrb.YwmUDiGMce3YCXo G5xs6_t0p47JSBS04xRYA_N1MadYLCMaA9ROhsXV_8dMtAqjFJD7QvZ5KzQhw3yFfVIxUntwi8U7 6B6u.sNQ27ROEGkp_YgYrdhc0e9u.3A3zEB2BIg9.z52lBiUBNPHUVl3TnN_8GfuvlqUO0SRUcJI HGUH3H3wkvbkQL2HfQ8ysnam6Lj3DKPdJiP5y7kpm_KEVGsfk2rAiMhCOkrCboyJD4ffu_VVnrwc DSLCo1_qHMe0zirRaaA8CbZ8fsDiE0i_C4cLduJviJXmAlowjVfqzr5cml9Q5TqfEOdMVlg5x6Yq 0bw-- X-Sonic-MF: X-Sonic-ID: 08839bc1-4aa3-473e-89e3-c2aa75bf6fdb Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.ir2.yahoo.com with HTTP; Wed, 12 Nov 2025 13:21:13 +0000 Received: by hermes--production-ir2-5fcfdd8d7f-zx548 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8b0becda5bcefbe06930eff224b49dd4; Wed, 12 Nov 2025 13:21:11 +0000 (UTC) Date: Wed, 12 Nov 2025 16:21:46 +0300 From: Anthony Pankov Message-ID: <1223233385.20251112162146@yahoo.com> To: fs@FreeBSD.org Subject: ZFS left sticked to nonexisting jail List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit References: <1223233385.20251112162146.ref@yahoo.com> X-Mailer: WebService/1.1.24652 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.37 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_LONG(-0.99)[-0.989]; NEURAL_SPAM_SHORT(0.72)[0.720]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; RWL_MAILSPIKE_GOOD(-0.10)[77.238.177.82:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:34010, ipnet:77.238.176.0/22, country:GB]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[fs@FreeBSD.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[77.238.177.82:from] X-Rspamd-Queue-Id: 4d63vJ5cWSz3m1S Dear FS, I've used jail config with such a string: X{ ... exec.release+="/sbin/zfs unjail ${name} ptank/poudriere"; ... } It seems wrong because "jail -r X" said something like "zfs unjail: bad jail name/id" (I will use "exec.prestop" in the future). After this zfs left mounted: # ls /aj/builderlc/body/usr/local/poudriere/distfiles/ yasm-1.3.0.tar.gz zfs.in zstd-1.5.6.tar.zst zstd-1.5.7.tar.zst ... Trying to manually unmount jail: #zfs umount -r /aj/builderlc/body/usr/local/poudriere/ # report no error but mount still here. Trying to unjail it again is impossible because the jail is not exist and I can't provide any meaningful name/jid argument for "zfs unjail". I don't know is this a bug or what? -- Best regards, Anthony Pankov mailto:anthony.pankov@yahoo.com From nobody Wed Nov 12 19:25:17 2025 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6CzP0hj7z6GTL1 for ; Wed, 12 Nov 2025 19:25:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6CzN73Dyz3t67 for ; Wed, 12 Nov 2025 19:25:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762975521; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:autocrypt:autocrypt; bh=oCbpTFwiQFd6GBwn6TQtNcldV/apyzeyxDHhfOe8Qlg=; b=vsPcQOQpaKHgtHxjg6dR6QaI3iloR98kZ7kscJX0E6caeVM/l2WbiT66SsmFjeOD4apZSF uCknZsXVipuar4CNnbPffe9K+oXUaIJV/ASwhcTP595+AVKiHT7EZQ7MlzAbUgvFUYlQ4I tq2s7XrbPDQe/LRvmNSBe0vtroq8oxXRLGAN3l7s+QGwstHmdTYggSq9JNPDAPTxAonf9B +OhOnUbxUQy6WynrHCKf90GZYCuGg1mRIHfqmkMTCh2LUzYzRzsf11V+3vAYvwk0r6bnNz 0HqBGV0YCmvMe8omCB5RdP0kzR/wK8ce2igjPC5t3PbCWjJwwQYQE2dTllB1gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762975521; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:autocrypt:autocrypt; bh=oCbpTFwiQFd6GBwn6TQtNcldV/apyzeyxDHhfOe8Qlg=; b=LyjonjFvCdJBRnLl91SdJUzTswYUlly3R8BUUEZVpJIbHd3iW2SfaHI851IB2vk16bMX4K ZDtwvkkK30MBjn6bvoj3A9etxk9H38dWb0YS2rFeoe+vVf1sarhA00aE5keok4crO60ofE e4PFPb2k8qySwzNTiJ5c/N9qYTS5GykqaLHsiHSNQGFqQAWoPNI/n7bnR0/sSEZ9EtwJC9 JPkzrtDEdfgN1pdFea27M4qj09CRZfcZjYI+7N/Zx3hfxes68MmSuX6D45TZ6tcZUJK5hn OqgkwdTFqdUTcUjAgJXbci6GwWlPSZBixi0pB4FYcnXA9ZwGxzLIMSkXo+gFKQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1762975521; a=rsa-sha256; cv=none; b=om8aBzrITxFYJ9pQqKQtMcHdnkzsnfxLxl4Tq4bsK7yFyDYxNMmB9s0Wcai8D+1SbzyWdB 6gpYbNTAqLzYkh6P/kzt3CuWjbTmgqwfIm3IFk4AYCS9wuCSWpD/oifHb6tSyWVWfweu1+ DXc9LXRtTwArAd38y/ktDwNXA0Uru7gJOVCyYYR/n7Y2xoUl/qbhla/YPQOMGbJmIGxQPh JMT0D4f0B6idIYnopLAxR9SwxmQB0CtHYFTKJSot7rhjRG+H9+yV1ZAAVYTzp3CCv16ToI Knif55Nczd7Wqk3pT+/tCfMzbGCvGIia79IKlozvwT6ViLQurstU9N8FPoFdLg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [192.168.0.88] (unknown [93.188.39.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4d6CzN4TNnzpBT for ; Wed, 12 Nov 2025 19:25:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: Date: Wed, 12 Nov 2025 21:25:17 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: fs@FreeBSD.org From: Andriy Gapon Subject: ZFS next-boot configuration ignored by loader.efi Autocrypt: addr=avg@FreeBSD.org; keydata= xsDNBGcKrHEBDADRvwQOK0b/yo4ys5cs6bOQMhEh4xtfbaZ/CU00cpPgUip3sOZCdrtMWlRC g25z97prxE9pKueZi+HXDhIPpa9xl14ghqF4oYScuJ1i18HyiOH2y5Q3Vv/TtFiSzicd3EAu QgS3jVidpgDSPDdj2Yz3UxYpZ+PuFl6nOnvCvqOFcjUlzKCyPaiN2b86l1Nscmhnc+zQ/faB erUOEFEDQbWMA5YfXi8HrbeR16hfRfGt7E0aMDlIj9FIPIq71UWMN9CimPgs4+rbNr1MAlLa z4GxSDhVYZEY5rqtCzr+PLXboRQWnaUwXl0/biw9enf17NHdYv1SNAFTX2eC4dZ3qBVI74dS PgNprm+PMfz+6Hhs/dAv+Nan5nVhg3EFIjYTiy0MnjMSq8uI0v0ykpAGAcJJ5xl6d23aLxgN 6f0z6pJRCO0hGPgU7UzvFD0MxJxmbzqdT1R51KDan1oD41b+tjl2LMBuCDCoB0U44Pu0zLdp xMfFTxCXtwIYKIUxwd28jwMAEQEAAc0eQW5kcml5IEdhcG9uIDxhdmdARnJlZUJTRC5vcmc+ wsENBBMBCAA3FiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHEFCQeEzgACGwMECwkIBwUV CAkKCwUWAgMBAAAKCRDMM63k0uPru5tSDACFK15LLbq89RSQ6QMnjiIm1t/wYJyumb519MHu Dhzxx1lbr8oghf0RHtF6kYRLQPaW2VdToi74pRobd3CN4bhZKDLSL6WfTn17RfavDjL6Njwp KBo30CkOeYKWq1mDmo0xEoQj8cc7ybEZnus+YScZOpj8Ti4EFwhRt6SHer7YDb161IHKL8m4 MsCxpFSGEjbKj8Iul3Ri/fTOO8w14ivcuEEQIvJt4/+4YV5Az8G23wKzL/3aJ7SOT3oYGmR9 atBTmVO3DlODjM+rZLegd8SfLSPTcBTHspWE5duemIzZbEX3BP77r3Qx4Fo5Tkit3bG1XVar yPQato+sFGFEGifdE9USBQoAoOaaeZevwAWjDU0TIuCT0CUe0sKtQuNP4LRq0n9EEHOXBu9a CfdMhFUSkAZnuE7miSVwgPvoVNJ1stA37EXLN/sVsWik7wslTQ5vF81VpdGFiwoQPOe2XEKh ogcwGSnXbwv1gD4x+Gz/7Y+kFyr1NY+4/nSaeXVcS2fOwM0EZwqscgEMAMQTe6ypAmQe/TFO HqKD2hfFKdksTptKi6uEh8xIwct8G/0FBldDWXo9eu8CGr/ZrDg0/bAwJxbaLRQCMH19Gq2Y hLvZ1QK5GQJVzZKcqfxbF2LiDUTs6WkdOBIhGpdDy7p1xFrvqCGCtNFYHuGYm067EozibBSF BWAPstKu2FQuVHZNMOfs7p3OIz3Yfqu9woXDeg3/8G2qVQJINe+8EwXKlhgh4CyDbq7nAZoA kIu1SE9z9u3WI5mcNy/0dFmVUsFxBqRC3ewbvzie8tKyZ9yFOlaZPT0Y4nRBXQTI3mLZ8zQ8 mtrWK5OOmrJ02kdeO9RBXe+OMaUUWMf92ZIoBFb4HP6N+B+4N1y1OwULousfl7JRoYxA4MRL ls7E2sSoJvrEBTJB3Pc34xu8rsJ1A5V3NgN6djX8yEZYpTRkcmrBeWy/ofDqZPVqneAx0LRm eldDS9msXDW4KXODyPZ+9unvmHAcoH0xaBYaSH44CDZDQDg4LNcmbOvuu1TEXBJhjQARAQAB wsD8BBgBCAAmFiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHMFCQeEzgACGwwACgkQzDOt 5NLj67sUCAv5AXqgWnYN9EblapMbZjkiqL8pZQ0GNqh+Pg9FwbyULxjtRTO6rD4D0IxizByb ef+neeUNyYlagt5nfKMysEr0SU/gHKCi8vyTF/63ukMrGUNGmJJxrndl5ZYKC6j6eX7twrZF L1Uvlmn6FnQ22red5kHO93fDjG4zaDIZvHfwj7kzjZ4tpC7Byinf88s14mdZeScc0PnU2hj4 UGYju/wg2FF4YxaZYhcmdTiRYY0Wx85XSMZv19pnn78sadEuRvfRd4JTmw++j1xGXeqQGWzz /CTG5/Ex9GAkQ02hZbmi236byDXoet4G8TEyOph9QFVkV9bNd0jQZaFZPGEj4PSPUYGAF7s5 xJaNGgctC3aZ7WjEv1FBoo44XCU4xcjJ1wZQUrHxRhx6TW0Jtcl0U9qfKFW30TSPo6RyiXuj X4ltWKAtjoXB8nUmEJckaz7IRu2b4pXDeazZuz5JBygUs10yJjDxh2vFQZo0KaBAPx9MZlPn gpPTjT15L8xGftEjQXF6 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit I am in a bit of a problem reporting mood today, so here's another report. I was upgrading a system like I usually do, using ZFS boot environments. First, I activated a new BE with bectl activate -t. The I rebooted and to my surprise the boot was to the bootfs BE, not to the temporary activated one. Also, bectl list showed that the new BE still had the temporary activation. It was as if loader.efi completely ignored presence of the next-boot configuration. The configuration was neither honored nor cleared. I examined the serial console log for any potential clues but couldn't find anything unusual except for one message: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool tank But tank is *not* the boot/root pool, it's purely a data pool. Still, I wonder if that error could have unintended consequences for next-boot. Here is a bigger log snippet, just in case: Consoles: EFI console ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool tank Reading loader env vars from /efi/freebsd/loader.env Setting currdev to disk0p1: FreeBSD/amd64 EFI loader, Revision 3.0 Command line arguments: loader.efi Image base: 0x7f0e1000 EFI version: 2.40 EFI Firmware: American Megatrends (rev 5.11) Console: efi (0x20000000) Load Path: \EFI\FREEBSD\LOADER.EFI Load Device: PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SAS(0x4433221104000000,0x0,0x1,SATA,External,Direct,0,0x0)/HD(1,GPT,1783F6B9-13FF-11EE-89E9-002590DCD8F9,0x28,0x82000) BootCurrent: 0000 BootOrder: 0000[*] 0001 0005 0004 0003 0002 BootInfo Path: HD(1,GPT,1783F6B9-13FF-11EE-89E9-002590DCD8F9,0x28,0x82000)/\EFI\FREEBSD\LOADER.EFI Ignoring Boot0000: Only one DP found Trying ESP: PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SAS(0x4433221104000000,0x0,0x1,SATA,External,Direct,0,0x0)/HD(1,GPT,1783F6B9-13FF-11EE-89E9-002590DCD8F9,0x28,0x82000) Setting currdev to disk0p1: Trying: PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SAS(0x4433221104000000,0x0,0x1,SATA,External,Direct,0,0x0)/HD(2,GPT,2418C833-13FF-11EE-89E9-002590DCD8F9,0x82028,0x800) Setting currdev to disk0p2: Trying: PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SAS(0x4433221104000000,0x0,0x1,SATA,External,Direct,0,0x0)/HD(3,GPT,31227754-13FF-11EE-89E9-002590DCD8F9,0x82828,0x1000000) Setting currdev to disk0p3: Trying: PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/SAS(0x4433221104000000,0x0,0x1,SATA,External,Direct,0,0x0)/HD(4,GPT,357485B5-13FF-11EE-89E9-002590DCD8F9,0x1082828,0x1C143120) Setting currdev to zfs:zroot/ROOT/default: Loading /boot/defaults/loader.conf Loading /boot/defaults/loader.conf Loading /boot/device.hints Loading /boot/loader.conf Loading /boot/loader.conf.local -- Andriy Gapon From nobody Thu Nov 13 10:41:49 2025 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6cLN6DNlz6GfDw for ; Thu, 13 Nov 2025 10:43:08 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6cLN5j5gz3cww for ; Thu, 13 Nov 2025 10:43:08 +0000 (UTC) (envelope-from ronald@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763030588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NbDocZUgEWAmv908jVCtUNQ0w8qaehus0EX7yIBDPWo=; b=cPtzBXRkA6qz+A0kuvVFzfm4gM3pnkbaNEfXpSgijWV/SCeLqTc3gv72+cDshBaR0G+3c9 0N9iENOxrFzLPIADfyuF/CjejkwjTBAuoeT0MoDI1gw2YBe4v/lB47/yt+xWCYub50i47a vON5wDkepbnNZ/cdCEHV2B8eGFGgvf+Y2tMrpXPH24CmA4GnYQIdNH+pCziGXBRU6L+DVv 2mG6jc164Ftxo9EPNobvHWr1vDdEY/Vb+yjjhiyRZxeWOKQo4AFJxTbm9HbWonFr47kSbF +GuaQUSFC24bYUYIFnthgOZBgT5LLR7VIT+i+8rc7WgXj3CYvLF90OFgEuTxqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763030588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NbDocZUgEWAmv908jVCtUNQ0w8qaehus0EX7yIBDPWo=; b=uDHD6hTQlbm7T6WXz6+LEd6+zmh8hybk0Q13DCchTUCiq+VtOM05MpjpEodJOBgf1KClxz J+cv47aEs0tPoY95Yw0K1XKidnr9SyUyLtrlsBqowuLW/LSzZbGk/YmESAEiFk8CDmgANc aO963NvKVPp5eiRvt1pi/ovuSlViIHJmGU/8XmbaqyU8yCCkVtn6MAsrXAL5nz3FFocC4C 8nz9i1qO9mleyhENmRAXqWkh/H7P+heAQabN/7beVCmuorTve12RMvaDjkvUwAAR1Lcwte fy9km0FroUmXIAWYf2EYMhI6C0zVpZQ6fxoyeXbyaqey3lJbbR2R3telXtnqew== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763030588; a=rsa-sha256; cv=none; b=qRglUarUrOGdRsGb84mU/kPr4OM/7403mxMDmHrVmFkvl8tMviHiyWQYTDk5AOhn+yUxgX JdlJNFXKHOoGEcSy906ha+BTTDU3ko/7S9Btitis17dptigfq9ZCykYnKjpDK1njo5Czy+ 8mbE3VhWW6PCpx8EyAVCYBBrnOPWy9bSmAhV1H9rv+JprQtNRWdbV6ULtEl5U/d28MopNX qkL+LNX9aL8sh3Cf+RSRtVlwbe6fw3plDBTWyxa1DJMjHpg+6osvAW333ZOGskbwo/26zt AjMpk0axqP8bpCoBuG/+H34RWN5Xo4XF/7e+Zqx6G1GDe3yysvIX41lgI2MKOA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPV6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242] (2001-1c00-270f-14b0-fd1b-708f-f8d8-f242.cable.dynamic.v6.ziggo.nl [IPv6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: ronald/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4d6cLN3YH6z17RX for ; Thu, 13 Nov 2025 10:43:08 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Message-ID: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> Date: Thu, 13 Nov 2025 11:41:49 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: fs@FreeBSD.org Content-Language: en-US From: Ronald Klop Subject: nfs hang Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi, I have setup nfsd in a jail. It exports zfs fs. The kernel is 16-CURRENT/aarch64. Jails are 14.3-RELEASE. $ cat /data/jails/pkg/_root/etc/exports V4: / -sec=sys /usr/local/poudriere/data/logs/bulk -sec=sys -maproot=root /usr/local/poudriere/data/packages -sec=sys -maproot=root /usr/ports -sec=sys The clients run poudriere in jails. Now and than I get hanging processes and unresponsive nfs server messages. All NFS threads are in this state: [root@rpi4 ~]# procstat -kk 5973 PID TID COMM TDNAME KSTACK 5973 100541 nfsd nfsd: master mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 5973 100784 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 5973 100785 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 5973 100786 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 ... and a couple more similar lines ... In rc.conf: nfs_server_enable=YES mountd_enable=YES nfsv4_server_only=YES nfs_server_flags="-t" The filesystems are a zfs legacy mount in the jail: # grep zfs /data/jails/pkg/fstab zrpi4/data/poudriere-logs-bulk /data/jails/pkg/_root/usr/local/poudriere/data/logs/bulk zfs rw 0 0 zrpi4/data/poudriere-packages /data/jails/pkg/_root/usr/local/poudriere/data/packages zfs rw 0 0 zdata4/ports /data/jails/pkg/_root/usr/ports zfs rw 0 0 Interestingly I also have a bash process hanging which should not access NFS at the moment: # procstat -kk 83175 PID TID COMM TDNAME KSTACK 83175 111203 bash - mi_switch+0x100 sleeplk+0xf8 lockmgr_slock_hard+0x29c _vn_lock+0x50 vget_finish+0x28 cache_fplookup_final_child+0x54 cache_fplookup+0x538 namei+0xd8 kern_statat+0xd4 sys_fstatat+0x2c do_el0_sync+0x6b4 handle_el0_sync+0x4c Any thoughts? Regards, Ronald. From nobody Thu Nov 13 10:45:36 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6cNx2T8Dz6GfWw for ; Thu, 13 Nov 2025 10:45:21 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6cNx10lVz3dqv for ; Thu, 13 Nov 2025 10:45:21 +0000 (UTC) (envelope-from ronald@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763030721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wucjTU0HEdlxzSTpx741wiONxyOiOL/S1fWhSuv3bGc=; b=GyofJmgwW1WEIgCLOwY1ZJYiIsGVsEzeWtFPMaCCSN/jmi4b2qpsMkvO6dJOF8cdSi7+5n 0RxWZuZ5fjhJv5n6nGg4X9kTbbgWUCKVhuFD+OlXVddimAt/mHQvmp7RuWNO+o3nokKXV8 lZlFcEoYrSlvwBF5/vVnf1/Byjn8hWXtfDuMZ1exp9o4uTfLl339uR/lVOakBhDkAdN0d4 JwHFpGpRsUYB2T80WW9+cp998hcvkLz0AiBx1IoSkELHYbqH/vzkTeUCpcsubk0KX96qYV nFsfOxT+pj+d2+Nmg9Zjy7rIvwN+p+3Ds+3AVeML2YPGwOOmBAShPg+Ylji4Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763030721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wucjTU0HEdlxzSTpx741wiONxyOiOL/S1fWhSuv3bGc=; b=a0XLQlZvMwirAE5NXEiWnbayVcV37nGce8btB22HJViIkEgI/wv5YjPpzXFhAh+5CGMsQp GtQYv2emldL6hkXwwj4qEpT8LNnzJZ5NRivD8kZP+Mi2OdRwysyOuY2mKp6lRn4OEVn8EV BjyC6qDXJpaFzsdBy3yAim07gNrj1utO7jAu38Z2PML0DAYeJTbja6sYBmK+daqEDS3niM uND9JZ8qcfezCZmUPGNE54L5oLj73L3VkwrKYNOag6F0kD2P5jS+U95Urp3/fF95ttmfuL sv+GXZ4OLvDkYPhi5ItWHlRXPd3KDqlbqcvGK0lS0wkRNdipXwGC1WAfk3lQOg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763030721; a=rsa-sha256; cv=none; b=itEH94v/qoqHrUGburXXB/c9QEmB22Gz8bdo1aoAq/Oh4j1ZSDQb8PW8B9Z4SLJeELXx6D VaNvoMUGwUsav5o/YwbBWUl9NQ4JBw88c6QPcWE/7Sm6XVSB9iCHzkS6DfX0A6/okyezUN 9Ah1JkgceaJx90aYRGbqdEE1aQPXmgKxuAN4+TqVwnN4ty8eOW5qkOuCOoX6HgnWoT1BTs eBTeAnSaSlnyiFd05pcLc3Cg7ltebT1gbu64VkHFqerDH0ulPLNIETE7rLpp/zQf6k7Eje J9YHwI8W58w3es2vVKPmagz59+sfoaW2DkzoPPeQUcpwLN6BanwavN8ctmZU3g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPV6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242] (2001-1c00-270f-14b0-fd1b-708f-f8d8-f242.cable.dynamic.v6.ziggo.nl [IPv6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: ronald/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4d6cNw6K5mz18M5 for ; Thu, 13 Nov 2025 10:45:20 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Message-ID: <4bb1700c-b71a-43ee-bf67-63172602205c@FreeBSD.org> Date: Thu, 13 Nov 2025 11:45:36 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: nfs hang To: freebsd-fs@freebsd.org References: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> Content-Language: en-US From: Ronald Klop In-Reply-To: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Op 13-11-2025 om 11:41 schreef Ronald Klop: > Hi, > > I have setup nfsd in a jail. It exports zfs fs. The kernel is 16-CURRENT/aarch64. Jails are 14.3-RELEASE. > $ cat /data/jails/pkg/_root/etc/exports > V4:    /    -sec=sys > > /usr/local/poudriere/data/logs/bulk    -sec=sys -maproot=root > /usr/local/poudriere/data/packages    -sec=sys -maproot=root > > /usr/ports    -sec=sys > > > The clients run poudriere in jails. > > Now and than I get hanging processes and unresponsive nfs server messages. > > All NFS threads are in this state: > [root@rpi4 ~]# procstat -kk 5973 >   PID    TID COMM                TDNAME              KSTACK >  5973 100541 nfsd                nfsd: master        mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >  5973 100784 nfsd                nfsd: service       mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >  5973 100785 nfsd                nfsd: service       mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >  5973 100786 nfsd                nfsd: service       mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > ... and a couple more similar lines ... > > In rc.conf: > nfs_server_enable=YES > mountd_enable=YES > nfsv4_server_only=YES > nfs_server_flags="-t" > > The filesystems are a zfs legacy mount in the jail: > # grep zfs /data/jails/pkg/fstab > zrpi4/data/poudriere-logs-bulk    /data/jails/pkg/_root/usr/local/poudriere/data/logs/bulk    zfs    rw    0    0 > zrpi4/data/poudriere-packages    /data/jails/pkg/_root/usr/local/poudriere/data/packages    zfs    rw    0    0 > zdata4/ports    /data/jails/pkg/_root/usr/ports    zfs    rw    0    0 > > > Interestingly I also have a bash process hanging which should not access NFS at the moment: > # procstat -kk 83175 >   PID    TID COMM                TDNAME              KSTACK > 83175 111203 bash                -                   mi_switch+0x100 sleeplk+0xf8 lockmgr_slock_hard+0x29c _vn_lock+0x50 vget_finish+0x28 cache_fplookup_final_child+0x54 cache_fplookup+0x538 namei+0xd8 kern_statat+0xd4 sys_fstatat+0x2c do_el0_sync+0x6b4 handle_el0_sync+0x4c > > Any thoughts? > > Regards, > Ronald. > > Just noticed this on the console: newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) newnfs: Logged 10 times about fileid corruption; going quiet to avoid spamming logs excessively. (Limit is: 10). From nobody Thu Nov 13 11:09:45 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6cxz3yXKz6GhBM for ; Thu, 13 Nov 2025 11:10:31 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout-a6-smtp.messagingengine.com (fout-a6-smtp.messagingengine.com [103.168.172.149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6cxy54Xcz3k3F for ; Thu, 13 Nov 2025 11:10:30 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=oO2HgMiV; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=MTR4M6GX; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.149 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from phl-compute-11.internal (phl-compute-11.internal [10.202.2.51]) by mailfout.phl.internal (Postfix) with ESMTP id 6C4EAEC01C6 for ; Thu, 13 Nov 2025 06:10:26 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-11.internal (MEProxy); Thu, 13 Nov 2025 06:10:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1763032226; x=1763118626; bh=X+RPPFD0K1 iMgDZLRQERbXYtCoO0aflbnCkSXegENPk=; b=oO2HgMiV79gW2cvllbw+BfFk+T bJIPeHDrU8gFkWwZYTOVurCGi1nuGYAZsLPMDx11Or+Yu84CRqtDGmRvJAdQL0Hn WVjtLAxIzRPbo8eM7b5t048DVSQXKbS6fD4QbIOt5UbyioHtuU4Y1msJm5G0Jdjy Ls45JLUQnfmdL1nuqySidROgClEGFQZsLNKJj9zE4QTWZA6htvGJUAFcTz456NZx 72repBoUhIj83buPleccfuQ0cKFv4p1LxQm+xavjQ5I9rzFiImVUL7vAKVeseZby HzwQ2BMSeHWi03G+lkHKc3z3hItMdsNTRrpAl9W7Ock60NG4O0Ha8QuT5Abg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1763032226; x=1763118626; bh=X+RPPFD0K1iMgDZLRQERbXYtCoO0aflbnCk SXegENPk=; b=MTR4M6GXofJCAq5wq9+kPtgH2v522vo9utr/b8o23qYVl1i6NA5 M/6/3Ua6Eu1/oSy4rXsi/hYrufvK1hnjO2zibE+dcyzSpqdOmfdAXoJtAzzVq25e cB5fkFx5nPjsuIbLAMJA0zbLZNPQgkIBDwEugHeCoM9LZ+wNlw+FLrPnOB0I4AeU eJOW/7wpXl31syRoXjmV4wt2HevtR4E79t1gZk5UYfb9SOM0sg81Q5WqWK9AuYk1 X27ghFktSAo2x5k/9vbRLjM0Jx6N6zFLg/rHjuw0BGHNFK4HvQBKuuJLniq+lmCT MN/MoTOl269k+4bPYR5Tdi7hZXGWBvC3lXA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggddvtdeijeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtredttd dtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgrthht vghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddtheehgf eunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhho ihgusehfqdhmrdhfmhdpnhgspghrtghpthhtohepuddpmhhouggvpehsmhhtphhouhhtpd hrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 13 Nov 2025 06:10:25 -0500 (EST) Date: Thu, 13 Nov 2025 11:09:45 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: nfs hang Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.85)[-0.853]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.149:from]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ASN(0.00)[asn:151847, ipnet:103.168.172.0/24, country:AU]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4d6cxy54Xcz3k3F On Thu, Nov 13, 2025 at 11:41:49AM +0100, Ronald Klop wrote: > >In rc.conf: >nfs_server_enable=YES >mountd_enable=YES >nfsv4_server_only=YES >nfs_server_flags="-t" I have nfs_server_flags="-u -t --minthreads=12 --maxthreads=120" What do you have these set to? on the server. And also rpc_statd_enable="YES" rpc_lockd_enable="YES" on both client and server. The server has HT turned OFF, 4 cpus, 64GB ram. What's your RAM/cpu? Does your server do much other than nfsd? Mine doesn't. -- From nobody Thu Nov 13 13:06:58 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6gXg1rMnz6GqpK for ; Thu, 13 Nov 2025 13:07:15 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6gXf73cZz43Xl for ; Thu, 13 Nov 2025 13:07:14 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-6419aaced59so1175620a12.0 for ; Thu, 13 Nov 2025 05:07:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763039232; x=1763644032; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KWXGkRNjdBQoVXxMQAgZ54byaDy99/dzQs42I8PcDPs=; b=g0cOQy1LwC4qpyDD+C52mKbrzKxpnvh+bDOSXZbVGJMaiPhBZ8u3iUrwN5oNAM/XfO 6IcVHWgNleGGu/cJC2kNoF3eqsPbWNYbaPXcTkoM4lIo/Gb4k7D7JpPGdioOmiZD/Rbk 525uXMTWvYskRyqK1f1UZvOYF53buEyVnBbQYToddCFOpgzNYlyz0qzRFaPgVhwQARPO BBhAbcAY37BXK9s2BhqAbTTThOVS0ln5I+EKsp6pDKSp8S5+IvGL8Gye6z/ndKv1J/NG tb7FmxaBXI512TjaydEPaikjQcy8Z46ZdntNt53fG7TBhb2jDv8cmB3Y6WxdbBp0pC9F ccKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763039232; x=1763644032; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=KWXGkRNjdBQoVXxMQAgZ54byaDy99/dzQs42I8PcDPs=; b=dfk/hmoXFwXKLU3eePw4yjkas6Qp09Wom1p8EvuT226I7EwirdpPdOLHF4Mwb8QJWV O3UzQMR2uDjymjmfCoYlJI7fWXB3MXnaXMAZy+fcy6yd8mRhSuRjQxxnFdxv7O60x+Or /h9Tm4ID2ZlZTet2kWKNMq52SSFYda8aeJEXreEN3chGYktWYDBky/FMQ0734Eh7Ap9X szK4HY+DSHpQvFyDZScWJgDvaUhepYOcxziJmTRheToZNg6ozvCCyuy4AqeBIV2uyC1A JElhWaRHZlbOi77AZCqfBXptIjxzS/0W/4IiaUxrv0PUu3S9Er9ydONOCMJqLGvC5BXQ VZ+w== X-Gm-Message-State: AOJu0YwF7EdViCRBaAXT3vPcXUmoFSoaufum9rrZPQvguWhe2lsgW23A fXn3oyg8o+owRSbDq5s3KWckp8bKWOS6KqEfXOUw8O1k2Ui/KiHusaD9ueEFtN+8v5IPSwhUPHA 8lkhWLmRaeDMTBq+ZYh9OMlcH9UtQBw== X-Gm-Gg: ASbGncv0QCNDz5ZoeVi05O8AcpSSr3RJqFhVwECAiYXY2SVb2RDsv4NrDHBdz2EzB8X mludkeIHiN9AcGBwVagU20ecnAwq/E29bVE/Phxu3w04TnmWN+DXESC3LkS+jXaCkFfjUVvSs5j BmsWma6Up9uNE+PrsrCSsg+wafyMQOG6IVgXyl1zpxk/h2Rk5UiYGn2+mQBJ1mGYfP+ebgOi+D0 kNTYhcAfZKH+aj7zYCFF3n3gT9Fz3jti8b7NgBlwmRNC6BviKPcW4PAEPE= X-Google-Smtp-Source: AGHT+IElehF8sHx+yMwxKB+9etYNzFAEu6+FJoV2iwTtTMhMXJk6LzuDgmoc0rBNmFGdASlWA7ke/kB9piz9SvwXn6c= X-Received: by 2002:a05:6402:13c1:b0:640:528c:7d49 with SMTP id 4fb4d7f45d1cf-6431a57e4d4mr6197718a12.38.1763039231678; Thu, 13 Nov 2025 05:07:11 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> <4bb1700c-b71a-43ee-bf67-63172602205c@FreeBSD.org> In-Reply-To: <4bb1700c-b71a-43ee-bf67-63172602205c@FreeBSD.org> From: Rick Macklem Date: Thu, 13 Nov 2025 05:06:58 -0800 X-Gm-Features: AWmQ_bmfmcPS1NRGoaGpnQBwfzbEOR7erP76-_HMRTTkKGpkCmMmWHtpY5Mc_uY Message-ID: Subject: Re: nfs hang To: Ronald Klop Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4d6gXf73cZz43Xl On Thu, Nov 13, 2025 at 2:45=E2=80=AFAM Ronald Klop wr= ote: > > Op 13-11-2025 om 11:41 schreef Ronald Klop: > > Hi, > > > > I have setup nfsd in a jail. It exports zfs fs. The kernel is 16-CURREN= T/aarch64. Jails are 14.3-RELEASE. > > $ cat /data/jails/pkg/_root/etc/exports > > V4: / -sec=3Dsys > > > > /usr/local/poudriere/data/logs/bulk -sec=3Dsys -maproot=3Droot > > /usr/local/poudriere/data/packages -sec=3Dsys -maproot=3Droot > > > > /usr/ports -sec=3Dsys > > > > > > The clients run poudriere in jails. > > > > Now and than I get hanging processes and unresponsive nfs server messag= es. > > > > All NFS threads are in this state: > > [root@rpi4 ~]# procstat -kk 5973 > > PID TID COMM TDNAME KSTACK > > 5973 100541 nfsd nfsd: master mi_switch+0x100 s= leepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_ca= ll+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xf= c nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close= +0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4= null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > > 5973 100784 nfsd nfsd: service mi_switch+0x100 s= leepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_ca= ll+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xf= c nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close= +0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4= null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > > 5973 100785 nfsd nfsd: service mi_switch+0x100 s= leepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_ca= ll+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xf= c nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close= +0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4= null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > > 5973 100786 nfsd nfsd: service mi_switch+0x100 s= leepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_ca= ll+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xf= c nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close= +0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4= null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > > ... and a couple more similar lines ... > > > > In rc.conf: > > nfs_server_enable=3DYES > > mountd_enable=3DYES > > nfsv4_server_only=3DYES > > nfs_server_flags=3D"-t" > > > > The filesystems are a zfs legacy mount in the jail: > > # grep zfs /data/jails/pkg/fstab > > zrpi4/data/poudriere-logs-bulk /data/jails/pkg/_root/usr/local/poudr= iere/data/logs/bulk zfs rw 0 0 > > zrpi4/data/poudriere-packages /data/jails/pkg/_root/usr/local/poudri= ere/data/packages zfs rw 0 0 > > zdata4/ports /data/jails/pkg/_root/usr/ports zfs rw 0 0 > > > > > > Interestingly I also have a bash process hanging which should not acces= s NFS at the moment: > > # procstat -kk 83175 > > PID TID COMM TDNAME KSTACK > > 83175 111203 bash - mi_switch+0x100 sl= eeplk+0xf8 lockmgr_slock_hard+0x29c _vn_lock+0x50 vget_finish+0x28 cache_fp= lookup_final_child+0x54 cache_fplookup+0x538 namei+0xd8 kern_statat+0xd4 sy= s_fstatat+0x2c do_el0_sync+0x6b4 handle_el0_sync+0x4c > > > > Any thoughts? > > > > Regards, > > Ronald. > > > > > > > Just noticed this on the console: > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5= 166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:2= 89259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d= 39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5= 166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:2= 89259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d= 39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5= 166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:2= 89259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5= 166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:2= 89259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) > newnfs: Logged 10 times about fileid corruption; going quiet to avoid spa= mming logs excessively. (Limit is: 10). Do you have more than one client mounting the file system? If you do, make sure they all have different /etc/hostid's. (Cloning a system disk without deleting /etc/hostid can result in multiple clients with the same /etc/hostid. That mean they are "the same client" to the NFSv4 server and that can cause the above.) If this is not the problem, I don't know why you'd see the above but I suspect the above explains the hang. rick > > From nobody Thu Nov 13 17:17:48 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6n7V6tY3z6H99p for ; Thu, 13 Nov 2025 17:19:18 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6n7V6Sp1z3m9L; Thu, 13 Nov 2025 17:19:18 +0000 (UTC) (envelope-from ronald@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763054358; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7yPAMp+UxinZIMW2r3W/EfRoEmKgmebxRTM5FfyL5Dc=; b=fy2ZHp9VuBmgjBVgpI2+0D7dmIiU5pIdJe2nB+UISwgCcczDXIHIK1VYm+qTlqMxKnXobG p2Es/lRPvExiQRLpaUNILeYQah1g/H5yZGYPpVZ2kytXMPlS3xeVRukmaXDb4IiMF6G8tE wekO9hSno4CpQLvAq4ib2wjRdZlBqshXvcE3n2JL49vGiCjQaq+tZppwBRNzL5GxLIwj3m KkAmWV/48hizeGEIK1azmyDqQVdqLQBndJ21SPjwS2A0Ql8cqDASlngTt1r6LWBph8VQix YNvSFCKwn1WOkOKjT8rQ+LqWLMt8mW/7BeY5WDlBRIFKvehJ+IzyZcxBOk94aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763054358; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7yPAMp+UxinZIMW2r3W/EfRoEmKgmebxRTM5FfyL5Dc=; b=KkmqMLx3w7Ke0Hd+4VN+UiOSX/UzHlMagDRwWVxRCIqebOKi30WgD+zORB4bLC5Y4XaH1S Q4zxMWylcJF3RUwqMJn80j3r8UiBH4soHqWl/HloEIp7PByqogDE9eSj0OmycLClIS03Rj 7f6GRCNfLpJN3oLmzxXLWBKyxZcf1kuK6gmgjenZrzt3aMcIj2SEXLixzd6tF4BMEAJFYJ 6EC+2xtSLZxN4HCGMlbIpShIXfvzV+gMwmLw4NTQ/6kQt2CeUKZ3KrD2FFOgcfhGx0zQ/q 4DT7fR17s78zORlI4l7LTgCoreE6zhuBup/48+Zy4lwEy5OeQgpPiTYSM0bkLA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763054358; a=rsa-sha256; cv=none; b=OQkrZxt0+ub9c7Ed6IP4F4/JJnjzan2brtwgP56ENoL57l7UccObpkoqAYKcYUeG/aIgBs p/PrVkTWGfpgTgs7l5UbXtAcCZ9Yfj3BTszCNeoUnvOLRtla/m+3lycBpmGjdT4N8mz9ls nyWO86pmOpH+jyUjt2867QW2lcqfFv/QQRoMX9jZcOpV8qFcDg6GJoiZpap3Ett1lEQJds o6cHECvQXbKyWlcFs2yDK9/S5uWDcstilA9qeprkCw8KFVz8FjcgWvTDL9OxnoM6NLJ0HG 0bR7/P7RsqCSaM5IXHSk7K40KnX6Lx2I5izaSz3kd0aj/0Y5Yh2ple/1tWhRvQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPV6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242] (2001-1c00-270f-14b0-fd1b-708f-f8d8-f242.cable.dynamic.v6.ziggo.nl [IPv6:2001:1c00:270f:14b0:fd1b:708f:f8d8:f242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: ronald/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4d6n7V36YSz3Yx; Thu, 13 Nov 2025 17:19:18 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Content-Type: multipart/mixed; boundary="------------a8mRsUOJPL26AFk0M1Fz4iAT" Message-ID: <1d48ae49-66b3-467d-8ff1-6b277d68246c@FreeBSD.org> Date: Thu, 13 Nov 2025 18:17:48 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: nfs hang To: Rick Macklem Cc: freebsd-fs@freebsd.org References: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> <4bb1700c-b71a-43ee-bf67-63172602205c@FreeBSD.org> Content-Language: en-US From: Ronald Klop In-Reply-To: This is a multi-part message in MIME format. --------------a8mRsUOJPL26AFk0M1Fz4iAT Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Op 13-11-2025 om 14:06 schreef Rick Macklem: > On Thu, Nov 13, 2025 at 2:45 AM Ronald Klop wrote: >> >> Op 13-11-2025 om 11:41 schreef Ronald Klop: >>> Hi, >>> >>> I have setup nfsd in a jail. It exports zfs fs. The kernel is 16-CURRENT/aarch64. Jails are 14.3-RELEASE. >>> $ cat /data/jails/pkg/_root/etc/exports >>> V4: / -sec=sys >>> >>> /usr/local/poudriere/data/logs/bulk -sec=sys -maproot=root >>> /usr/local/poudriere/data/packages -sec=sys -maproot=root >>> >>> /usr/ports -sec=sys >>> >>> >>> The clients run poudriere in jails. >>> >>> Now and than I get hanging processes and unresponsive nfs server messages. >>> >>> All NFS threads are in this state: >>> [root@rpi4 ~]# procstat -kk 5973 >>> PID TID COMM TDNAME KSTACK >>> 5973 100541 nfsd nfsd: master mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >>> 5973 100784 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >>> 5973 100785 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >>> 5973 100786 nfsd nfsd: service mi_switch+0x100 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc_call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_close+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 >>> ... and a couple more similar lines ... >>> >>> In rc.conf: >>> nfs_server_enable=YES >>> mountd_enable=YES >>> nfsv4_server_only=YES >>> nfs_server_flags="-t" >>> >>> The filesystems are a zfs legacy mount in the jail: >>> # grep zfs /data/jails/pkg/fstab >>> zrpi4/data/poudriere-logs-bulk /data/jails/pkg/_root/usr/local/poudriere/data/logs/bulk zfs rw 0 0 >>> zrpi4/data/poudriere-packages /data/jails/pkg/_root/usr/local/poudriere/data/packages zfs rw 0 0 >>> zdata4/ports /data/jails/pkg/_root/usr/ports zfs rw 0 0 >>> >>> >>> Interestingly I also have a bash process hanging which should not access NFS at the moment: >>> # procstat -kk 83175 >>> PID TID COMM TDNAME KSTACK >>> 83175 111203 bash - mi_switch+0x100 sleeplk+0xf8 lockmgr_slock_hard+0x29c _vn_lock+0x50 vget_finish+0x28 cache_fplookup_final_child+0x54 cache_fplookup+0x538 namei+0xd8 kern_statat+0xd4 sys_fstatat+0x2c do_el0_sync+0x6b4 handle_el0_sync+0x4c >>> >>> Any thoughts? >>> >>> Regards, >>> Ronald. >>> >>> >> >> >> Just noticed this on the console: >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc87:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) >> newnfs: Logged 10 times about fileid corruption; going quiet to avoid spamming logs excessively. (Limit is: 10). > Do you have more than one client mounting the file system? > If you do, make sure they all have different /etc/hostid's. > (Cloning a system disk without deleting /etc/hostid can > result in multiple clients with the same /etc/hostid. That > mean they are "the same client" to the NFSv4 server > and that can cause the above.) > > If this is not the problem, I don't know why you'd see the > above but I suspect the above explains the hang. > > rick > Two clients. Both have different /etc/hostid. I noticed that the procstat stacks start with "null_reclaim". And poudriere null-mounts the nfs mounts in the poudriere-jails. Could nfs+nullfs give some trouble? Or maybe it is just nullfs that hangs everything and the nfs stuff is just a result of it. At the same moment I had git hanging on a non-NFS mount. See attachment for the procstat which also includes nfs-calls. Regards, Ronald. --------------a8mRsUOJPL26AFk0M1Fz4iAT Content-Type: text/plain; charset=UTF-8; name="git.hang.txt" Content-Disposition: attachment; filename="git.hang.txt" Content-Transfer-Encoding: base64 G1tPIHByb2NzdGF0IC1rayA4MTU2MAogIFBJRCAgICBUSUQgQ09NTSAgICAgICAgICAgICAg ICBURE5BTUUgICAgICAgICAgICAgIEtTVEFDSyAgICAgICAgICAgICAgICAgICAgICAgCjgx NTYwIDEwNDUwNCBnaXQgICAgICAgICAgICAgICAgIC0gICAgICAgICAgICAgICAgICAgbWlf c3dpdGNoKzB4MTAwIHNsZWVwcV9jYXRjaF9zaWduYWxzKzB4M2U0IHNsZWVwcV93YWl0X3Np ZysweGMgX3NsZWVwKzB4MWMwIHVtdHhxX3NsZWVwKzB4MmIwIGRvX3dhaXQrMHgyZWMgX191 bXR4X29wX3dhaXQrMHg2OCBzeXNfX3VtdHhfb3ArMHg4YyBkb19lbDBfc3luYysweDZiNCBo YW5kbGVfZWwwX3N5bmMrMHg0YyAKODE1NjAgMTM1ODE3IGdpdCAgICAgICAgICAgICAgICAg LSAgICAgICAgICAgICAgICAgICBtaV9zd2l0Y2grMHgxMDAgc2xlZXBsaysweGY4IGxvY2tt Z3JfeGxvY2tfaGFyZCsweDI5YyBudWxsX2xvY2srMHhlOCBfdm5fbG9jaysweDUwIHZmc19s b29rdXArMHgxMjAgbmFtZWkrMHgyMmMga2Vybl9zdGF0YXQrMHhkNCBzeXNfZnN0YXRhdCsw eDJjIGRvX2VsMF9zeW5jKzB4NmI0IGhhbmRsZV9lbDBfc3luYysweDRjIAo4MTU2MCAxMzU4 MTggZ2l0ICAgICAgICAgICAgICAgICAtICAgICAgICAgICAgICAgICAgIG1pX3N3aXRjaCsw eDEwMCBzbGVlcHFfY2F0Y2hfc2lnbmFscysweDNlNCBzbGVlcHFfdGltZWR3YWl0X3NpZysw eDE4IF9zbGVlcCsweDFhMCBjbG50X3ZjX2NhbGwrMHg4MTQgY2xudF9yZWNvbm5lY3RfY2Fs bCsweDk2MCBuZXduZnNfcmVxdWVzdCsweGFjYyBuZnNycGNfY2xvc2VycGMrMHhmYyBuZnNj bF90cnljbG9zZSsweDU4IG5mc3JwY19kb2Nsb3NlKzB4Mjk0IG5mc2NsX2RvY2xvc2UrMHgz OTAgbmZzcnBjX2Nsb3NlKzB4MjggbmNsX2luYWN0aXZlKzB4MTRjIHZvcF9zaWdkZWZlcisw eDM0IHZpbmFjdGl2ZWYrMHhiOCB2cHV0X2ZpbmFsKzB4MWY0IG51bGxfcmVjbGFpbSsweDFh MCBWT1BfUkVDTEFJTV9BUFYrMHgyMCAKODE1NjAgMTM1ODE5IGdpdCAgICAgICAgICAgICAg ICAgLSAgICAgICAgICAgICAgICAgICBtaV9zd2l0Y2grMHgxMDAgc2xlZXBxX2NhdGNoX3Np Z25hbHMrMHgzZTQgc2xlZXBxX3RpbWVkd2FpdF9zaWcrMHgxOCBfc2xlZXArMHgxYTAgY2xu dF92Y19jYWxsKzB4ODE0IGNsbnRfcmVjb25uZWN0X2NhbGwrMHg5NjAgbmV3bmZzX3JlcXVl c3QrMHhhY2MgbmZzcnBjX2Nsb3NlcnBjKzB4ZmMgbmZzY2xfdHJ5Y2xvc2UrMHg1OCBuZnNy cGNfZG9jbG9zZSsweDI5NCBuZnNjbF9kb2Nsb3NlKzB4MzkwIG5mc3JwY19jbG9zZSsweDI4 IG5jbF9pbmFjdGl2ZSsweDE0YyB2b3Bfc2lnZGVmZXIrMHgzNCB2aW5hY3RpdmVmKzB4Yjgg dnB1dF9maW5hbCsweDFmNCBudWxsX3JlY2xhaW0rMHgxYTAgVk9QX1JFQ0xBSU1fQVBWKzB4 MjAgCjgxNTYwIDEzNTgyMCBnaXQgICAgICAgICAgICAgICAgIC0gICAgICAgICAgICAgICAg ICAgbWlfc3dpdGNoKzB4MTAwIHNsZWVwbGsrMHhmOCBsb2NrbWdyX3hsb2NrX2hhcmQrMHgy OWMgbnVsbF9sb2NrKzB4ZTggX3ZuX2xvY2srMHg1MCB2ZnNfbG9va3VwKzB4MTIwIG5hbWVp KzB4MjJjIGtlcm5fc3RhdGF0KzB4ZDQgc3lzX2ZzdGF0YXQrMHgyYyBkb19lbDBfc3luYysw eDZiNCBoYW5kbGVfZWwwX3N5bmMrMHg0YyAKODE1NjAgMTM1ODIxIGdpdCAgICAgICAgICAg ICAgICAgLSAgICAgICAgICAgICAgICAgICBtaV9zd2l0Y2grMHgxMDAgc2xlZXBsaysweGY4 IGxvY2ttZ3JfeGxvY2tfaGFyZCsweDI5YyBudWxsX2xvY2srMHhlOCBfdm5fbG9jaysweDUw IHZmc19sb29rdXArMHgxMjAgbmFtZWkrMHgyMmMga2Vybl9zdGF0YXQrMHhkNCBzeXNfZnN0 YXRhdCsweDJjIGRvX2VsMF9zeW5jKzB4NmI0IGhhbmRsZV9lbDBfc3luYysweDRjIAo4MTU2 MCAxMzU4MjIgZ2l0ICAgICAgICAgICAgICAgICAtICAgICAgICAgICAgICAgICAgIG1pX3N3 aXRjaCsweDEwMCBzbGVlcGxrKzB4ZjggbG9ja21ncl94bG9ja19oYXJkKzB4MjljIG51bGxf bG9jaysweGU4IF92bl9sb2NrKzB4NTAgdmZzX2xvb2t1cCsweDEyMCBuYW1laSsweDIyYyBr ZXJuX3N0YXRhdCsweGQ0IHN5c19mc3RhdGF0KzB4MmMgZG9fZWwwX3N5bmMrMHg2YjQgaGFu ZGxlX2VsMF9zeW5jKzB4NGMgCjgxNTYwIDEzNTgyMyBnaXQgICAgICAgICAgICAgICAgIC0g ICAgICAgICAgICAgICAgICAgbWlfc3dpdGNoKzB4MTAwIHNsZWVwbGsrMHhmOCBsb2NrbWdy X3hsb2NrX2hhcmQrMHgyOWMgbnVsbF9sb2NrKzB4ZTggX3ZuX2xvY2srMHg1MCB2ZnNfbG9v a3VwKzB4MTIwIG5hbWVpKzB4MjJjIGtlcm5fc3RhdGF0KzB4ZDQgc3lzX2ZzdGF0YXQrMHgy YyBkb19lbDBfc3luYysweDZiNCBoYW5kbGVfZWwwX3N5bmMrMHg0YyAKODE1NjAgMTM1ODI0 IGdpdCAgICAgICAgICAgICAgICAgLSAgICAgICAgICAgICAgICAgICBtaV9zd2l0Y2grMHgx MDAgc2xlZXBsaysweGY4IGxvY2ttZ3JfeGxvY2tfaGFyZCsweDI5YyBudWxsX2xvY2srMHhl OCBfdm5fbG9jaysweDUwIHZmc19sb29rdXArMHgxMjAgbmFtZWkrMHgyMmMga2Vybl9zdGF0 YXQrMHhkNCBzeXNfZnN0YXRhdCsweDJjIGRvX2VsMF9zeW5jKzB4NmI0IGhhbmRsZV9lbDBf c3luYysweDRjIAo4MTU2MCAxMzU4MjUgZ2l0ICAgICAgICAgICAgICAgICAtICAgICAgICAg ICAgICAgICAgIG1pX3N3aXRjaCsweDEwMCBzbGVlcHFfY2F0Y2hfc2lnbmFscysweDNlNCBz bGVlcHFfdGltZWR3YWl0X3NpZysweDE4IF9zbGVlcCsweDFhMCBjbG50X3ZjX2NhbGwrMHg4 MTQgY2xudF9yZWNvbm5lY3RfY2FsbCsweDk2MCBuZXduZnNfcmVxdWVzdCsweGFjYyBuZnNy cGNfY2xvc2VycGMrMHhmYyBuZnNjbF90cnljbG9zZSsweDU4IG5mc3JwY19kb2Nsb3NlKzB4 Mjk0IG5mc2NsX2RvY2xvc2UrMHgzOTAgbmZzcnBjX2Nsb3NlKzB4MjggbmNsX2luYWN0aXZl KzB4MTRjIHZvcF9zaWdkZWZlcisweDM0IHZpbmFjdGl2ZWYrMHhiOCB2cHV0X2ZpbmFsKzB4 MWY0IG51bGxfcmVjbGFpbSsweDFhMCBWT1BfUkVDTEFJTV9BUFYrMHgyMCAKODE1NjAgMTM1 ODI2IGdpdCAgICAgICAgICAgICAgICAgLSAgICAgICAgICAgICAgICAgICBtaV9zd2l0Y2gr MHgxMDAgc2xlZXBxX2NhdGNoX3NpZ25hbHMrMHgzZTQgc2xlZXBxX3RpbWVkd2FpdF9zaWcr MHgxOCBfc2xlZXArMHgxYTAgY2xudF92Y19jYWxsKzB4ODE0IGNsbnRfcmVjb25uZWN0X2Nh bGwrMHg5NjAgbmV3bmZzX3JlcXVlc3QrMHhhY2MgbmZzcnBjX2Nsb3NlcnBjKzB4ZmMgbmZz Y2xfdHJ5Y2xvc2UrMHg1OCBuZnNycGNfZG9jbG9zZSsweDI5NCBuZnNjbF9kb2Nsb3NlKzB4 MzkwIG5mc3JwY19jbG9zZSsweDI4IG5jbF9pbmFjdGl2ZSsweDE0YyB2b3Bfc2lnZGVmZXIr MHgzNCB2aW5hY3RpdmVmKzB4YjggdnB1dF9maW5hbCsweDFmNCBudWxsX3JlY2xhaW0rMHgx YTAgVk9QX1JFQ0xBSU1fQVBWKzB4MjAgCjgxNTYwIDEzNTgyNyBnaXQgICAgICAgICAgICAg ICAgIC0gICAgICAgICAgICAgICAgICAgbWlfc3dpdGNoKzB4MTAwIHNsZWVwcV9jYXRjaF9z aWduYWxzKzB4M2U0IHNsZWVwcV90aW1lZHdhaXRfc2lnKzB4MTggX3NsZWVwKzB4MWEwIGNs bnRfdmNfY2FsbCsweDgxNCBjbG50X3JlY29ubmVjdF9jYWxsKzB4OTYwIG5ld25mc19yZXF1 ZXN0KzB4YWNjIG5mc3JwY19jbG9zZXJwYysweGZjIG5mc2NsX3RyeWNsb3NlKzB4NTggbmZz cnBjX2RvY2xvc2UrMHgyOTQgbmZzY2xfZG9jbG9zZSsweDM5MCBuZnNycGNfY2xvc2UrMHgy OCBuY2xfaW5hY3RpdmUrMHgxNGMgdm9wX3NpZ2RlZmVyKzB4MzQgdmluYWN0aXZlZisweGI4 IHZwdXRfZmluYWwrMHgxZjQgbnVsbF9yZWNsYWltKzB4MWEwIFZPUF9SRUNMQUlNX0FQVisw eDIwIAo4MTU2MCAxMzU4MjggZ2l0ICAgICAgICAgICAgICAgICAtICAgICAgICAgICAgICAg ICAgIG1pX3N3aXRjaCsweDEwMCBzbGVlcGxrKzB4ZjggbG9ja21ncl94bG9ja19oYXJkKzB4 MjljIG51bGxfbG9jaysweGU4IF92bl9sb2NrKzB4NTAgdmZzX2xvb2t1cCsweDEyMCBuYW1l aSsweDIyYyBrZXJuX3N0YXRhdCsweGQ0IHN5c19mc3RhdGF0KzB4MmMgZG9fZWwwX3N5bmMr MHg2YjQgaGFuZGxlX2VsMF9zeW5jKzB4NGMgCjgxNTYwIDEzNTgyOSBnaXQgICAgICAgICAg ICAgICAgIC0gICAgICAgICAgICAgICAgICAgbWlfc3dpdGNoKzB4MTAwIHNsZWVwbGsrMHhm OCBsb2NrbWdyX3hsb2NrX2hhcmQrMHgyOWMgbnVsbF9sb2NrKzB4ZTggX3ZuX2xvY2srMHg1 MCB2ZnNfbG9va3VwKzB4MTIwIG5hbWVpKzB4MjJjIGtlcm5fc3RhdGF0KzB4ZDQgc3lzX2Zz dGF0YXQrMHgyYyBkb19lbDBfc3luYysweDZiNCBoYW5kbGVfZWwwX3N5bmMrMHg0YyAKODE1 NjAgMTM1ODMwIGdpdCAgICAgICAgICAgICAgICAgLSAgICAgICAgICAgICAgICAgICBtaV9z d2l0Y2grMHgxMDAgc2xlZXBsaysweGY4IGxvY2ttZ3JfeGxvY2tfaGFyZCsweDI5YyBudWxs X2xvY2srMHhlOCBfdm5fbG9jaysweDUwIHZmc19sb29rdXArMHgxMjAgbmFtZWkrMHgyMmMg a2Vybl9zdGF0YXQrMHhkNCBzeXNfZnN0YXRhdCsweDJjIGRvX2VsMF9zeW5jKzB4NmI0IGhh bmRsZV9lbDBfc3luYysweDRjIAo4MTU2MCAxMzU4MzEgZ2l0ICAgICAgICAgICAgICAgICAt ICAgICAgICAgICAgICAgICAgIG1pX3N3aXRjaCsweDEwMCBzbGVlcGxrKzB4ZjggbG9ja21n cl94bG9ja19oYXJkKzB4MjljIG51bGxfbG9jaysweGU4IF92bl9sb2NrKzB4NTAgdmZzX2xv b2t1cCsweDEyMCBuYW1laSsweDIyYyBrZXJuX3N0YXRhdCsweGQ0IHN5c19mc3RhdGF0KzB4 MmMgZG9fZWwwX3N5bmMrMHg2YjQgaGFuZGxlX2VsMF9zeW5jKzB4NGMgCjgxNTYwIDEzNTgz MiBnaXQgICAgICAgICAgICAgICAgIC0gICAgICAgICAgICAgICAgICAgbWlfc3dpdGNoKzB4 MTAwIHNsZWVwbGsrMHhmOCBsb2NrbWdyX3hsb2NrX2hhcmQrMHgyOWMgbnVsbF9sb2NrKzB4 ZTggX3ZuX2xvY2srMHg1MCB2ZnNfbG9va3VwKzB4MTIwIG5hbWVpKzB4MjJjIGtlcm5fc3Rh dGF0KzB4ZDQgc3lzX2ZzdGF0YXQrMHgyYyBkb19lbDBfc3luYysweDZiNCBoYW5kbGVfZWww X3N5bmMrMHg0YyAKODE1NjAgMTM1ODMzIGdpdCAgICAgICAgICAgICAgICAgLSAgICAgICAg ICAgICAgICAgICBtaV9zd2l0Y2grMHgxMDAgc2xlZXBxX2NhdGNoX3NpZ25hbHMrMHgzZTQg c2xlZXBxX3RpbWVkd2FpdF9zaWcrMHgxOCBfc2xlZXArMHgxYTAgY2xudF92Y19jYWxsKzB4 ODE0IGNsbnRfcmVjb25uZWN0X2NhbGwrMHg5NjAgbmV3bmZzX3JlcXVlc3QrMHhhY2MgbmZz cnBjX2Nsb3NlcnBjKzB4ZmMgbmZzY2xfdHJ5Y2xvc2UrMHg1OCBuZnNycGNfZG9jbG9zZSsw eDI5NCBuZnNjbF9kb2Nsb3NlKzB4MzkwIG5mc3JwY19jbG9zZSsweDI4IG5jbF9pbmFjdGl2 ZSsweDE0YyB2b3Bfc2lnZGVmZXIrMHgzNCB2aW5hY3RpdmVmKzB4YjggdnB1dF9maW5hbCsw eDFmNCBudWxsX3JlY2xhaW0rMHgxYTAgVk9QX1JFQ0xBSU1fQVBWKzB4MjAgCjgxNTYwIDEz NTgzNCBnaXQgICAgICAgICAgICAgICAgIC0gICAgICAgICAgICAgICAgICAgbWlfc3dpdGNo KzB4MTAwIHNsZWVwbGsrMHhmOCBsb2NrbWdyX3hsb2NrX2hhcmQrMHgyOWMgbnVsbF9sb2Nr KzB4ZTggX3ZuX2xvY2srMHg1MCB2ZnNfbG9va3VwKzB4MTIwIG5hbWVpKzB4MjJjIGtlcm5f c3RhdGF0KzB4ZDQgc3lzX2ZzdGF0YXQrMHgyYyBkb19lbDBfc3luYysweDZiNCBoYW5kbGVf ZWwwX3N5bmMrMHg0YyAKODE1NjAgMTM1ODM1IGdpdCAgICAgICAgICAgICAgICAgLSAgICAg ICAgICAgICAgICAgICBtaV9zd2l0Y2grMHgxMDAgc2xlZXBsaysweGY4IGxvY2ttZ3JfeGxv Y2tfaGFyZCsweDI5YyBudWxsX2xvY2srMHhlOCBfdm5fbG9jaysweDUwIHZmc19sb29rdXAr MHgxMjAgbmFtZWkrMHgyMmMga2Vybl9zdGF0YXQrMHhkNCBzeXNfZnN0YXRhdCsweDJjIGRv X2VsMF9zeW5jKzB4NmI0IGhhbmRsZV9lbDBfc3luYysweDRjIAo4MTU2MCAxMzU4MzYgZ2l0 ICAgICAgICAgICAgICAgICAtICAgICAgICAgICAgICAgICAgIG1pX3N3aXRjaCsweDEwMCBz bGVlcHFfY2F0Y2hfc2lnbmFscysweDNlNCBzbGVlcHFfdGltZWR3YWl0X3NpZysweDE4IF9z bGVlcCsweDFhMCBjbG50X3ZjX2NhbGwrMHg4MTQgY2xudF9yZWNvbm5lY3RfY2FsbCsweDk2 MCBuZXduZnNfcmVxdWVzdCsweGFjYyBuZnNycGNfY2xvc2VycGMrMHhmYyBuZnNjbF90cnlj bG9zZSsweDU4IG5mc3JwY19kb2Nsb3NlKzB4Mjk0IG5mc2NsX2RvY2xvc2UrMHgzOTAgbmZz cnBjX2Nsb3NlKzB4MjggbmNsX2luYWN0aXZlKzB4MTRjIHZvcF9zaWdkZWZlcisweDM0IHZp bmFjdGl2ZWYrMHhiOCB2cHV0X2ZpbmFsKzB4MWY0IG51bGxfcmVjbGFpbSsweDFhMCBWT1Bf UkVDTEFJTV9BUFYrMHgyMCAKCg== --------------a8mRsUOJPL26AFk0M1Fz4iAT-- From nobody Thu Nov 13 20:04:26 2025 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6rp31GQgz6HLwT for ; Thu, 13 Nov 2025 20:04:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6rp30jSVz44fF for ; Thu, 13 Nov 2025 20:04:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763064267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JwgCkTxVM9aY5pcCrxLRZ4ZUHBp396TOOhp8CM2EtzQ=; b=sANmp4kvoc0NI/Z3L1EXombFkJRuDEyCizRSlY4WkrQcEJh7LH+6ltQo6I+NCzMYo16vVh zjHQgdpRc4mTAJb4bETUmuUz42i5bdbEJdvC+Tsn9h8Cy6oB6P1r74Fn79vKSq5Crdj1Wc jsBo8WDmzDQZR9VNYQoCZ68t7R7J7ZAthxyx/vm+I52Q6mS7m7+5FEXsBkWW4SbFR/Bc6/ Cdo7MB0qxHuRqRw/qeWLMg81tSLI0SiFPxHHg5KZRZOg8OrUYvw42O4kBFnQ4OfsA6VHJ6 ay1suHAnv/nQYXyqbfgAhiKzF+mgsActN7e61hYxntvs4LNl0tQJVDkFOABjJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763064267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JwgCkTxVM9aY5pcCrxLRZ4ZUHBp396TOOhp8CM2EtzQ=; b=q+U6ExUM/BuqnxTAfXox+fCij9Ngw28IaN9wygsDN1bdcblBn5Wa6uxF2mslNb18f/JEBB d8SCeDTJJvBXPq/RMuIct5o+Ux92G2C5iZCYZqjaKeeNkc0AXfRkmdKs6D0KvDhdSLOPK0 t7iWMsQSI1D2ZojbGW9K7yDUNfzkCqzomCPcHbc+OOXiRbhtkypKDsw0rRcfkaqB2ZEorJ fGIITMy2kEMGwiGEd40pAht/tTcEnKUIakDd1NY/2aUh/HRarQeN99NUjVt6Ru5pqFV8EM CHelH/MyOOug1ohfnQAe++roqkN1gDjd75QmmXlVBUQQNYw+HJAxYxpii2TRDw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763064267; a=rsa-sha256; cv=none; b=xhLinFU0Zwtq/9ocF3MB5PdH9vMkk+/oDqKHWqbhllYWPgIUly9vyHjG0cOGf4eMBs4KKD sAAW5ZpIGwSH6vT4VSSodJVPi59He/a4bycmukeiWbJ1XPyB41QU9LifeOBntiZWekfo7i 9G8ZbF6TNbshAXJis5F1stOuLt4VrtWbetWuRQ8foFrzl30DapvaMoDv2tYwUlqQ92sVdt k1ImK2/EsYukxlnqTrFHn2cYPwf6Jh/4LDaSsnw4KYH25iir1alFE26enpf+Ncig+BoEbY +g9wkvZ6inmJ+DyELU2HFvkdFHrQdemmXpebYWBasm2ByE8/U1sBRX49avew8g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4d6rp30F4hzc1T for ; Thu, 13 Nov 2025 20:04:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 5ADK4Qqk091647 for ; Thu, 13 Nov 2025 20:04:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 5ADK4Q4e091646 for fs@FreeBSD.org; Thu, 13 Nov 2025 20:04:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 291002] ZFS error created on otherwise running and sane system. Date: Thu, 13 Nov 2025 20:04:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D291002 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |regression Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Nov 13 21:07:05 2025 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6tBL17Lgz6GCfy for ; Thu, 13 Nov 2025 21:07:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6tBL0VRhz4HT7 for ; Thu, 13 Nov 2025 21:07:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763068026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=okiwQrXVl+luU8e1k2z/a3z8LwlpdoDfujrLr4j3Mlc=; b=pro/4pqA3tXSXAHZ6G4YpQs1asVLdhq+L2zFATynipSArDOrHZUkd2yI+8RLo494Iu92mW jCGxf0d7Y0RJYOICr1+g7b3NmbZReZNln+jSBVajCK8xyCcZV9MFa/eoFp3EAvLDf2PYZB vMDiEHq6C2boWL/HcLe9Y4PQoaeV9w37bBVk0rxlCZRyJXookkwEQ4Wlkq7wBaIMSnnhwd mkv6vwf27cwg4pZQvPhY3+gg8i8oALd3zqjvNZZYMC+xTs490WxQux6gRveOKzB8CAQtlP p+g2wpr3N6C59Hfkgk+uLXzDZnIw/8JhjJK+iWQIOYyLPGqjQ1e8c+Ow3EgjvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763068026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=okiwQrXVl+luU8e1k2z/a3z8LwlpdoDfujrLr4j3Mlc=; b=fJJoWaGFteKUqoWJYSS/RhQL6Hh6CeLcFnQh7wwQxZu6iKySzEAtAuUsPZr9zFIme9jNYr 37JVH/2T6ED9xrorRt9nX50vvtQ+Rv6renc/jaCVKoBJukdH+Rnbv8KsdA1eG3+qhcqsj8 3cQzBEnMh+PrjrMRdQvT3rpyXc/NS6Fr/FG7K+/hppAtgCXxJbsPlXnmX8d26ZJgy6Sc6E e9XGhbpc4L558mHE25RirPvmlVqAZmeosB5rBEePZxWJXn19JHaskbJ1MJ5jYcpIoD1JlW fqvaqkpLZfqdmTNNMAZR0uQiFT5IS21AOd+3w5vtoGkW55o5t58jGDCTJgHPgw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763068026; a=rsa-sha256; cv=none; b=O5GSKp3KPxI7h487HTTNI2ArYFboxPZ40FbHTXvBa88p5xmGDz1Yw+aSOGDXYpr1ZGUjMi WA8BQZTCJ66arDlOeWsUx92Tk56qvY/kOTLC2Wz+CtWLKV/DB6Dp6/nEuPZl2Qh410E2vQ HBafsnwZTZo3PBeJphISxfBpVkyC09hWUC8tsBcVOxqto05NcAVDy0joZml9j1m/0Wf/sQ qpzc+RMto2s+kKS/DV4ezp1S/RdSCR+AUVn8rAe5Zyc0rKXIPCJmY1yHZuiPK6+rgOEXvQ S1DuwhgTk7Q9jfdJm8ZSrrr5ownY3+sKUOzmVp78VHyPLn9AH10i1BgFyY2klQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4d6tBK6xcWzdGv for ; Thu, 13 Nov 2025 21:07:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 5ADL751F086026 for ; Thu, 13 Nov 2025 21:07:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 5ADL75Mn086025 for fs@FreeBSD.org; Thu, 13 Nov 2025 21:07:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 291002] ZFS error created on otherwise running and sane system. Date: Thu, 13 Nov 2025 21:07:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dave@daveg.ca X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D291002 --- Comment #2 from David Gilbert --- Spinning my tires a bit on this. I realized it's 4 replications, actually.= =20 The first time this happened, ~/.local was just a directory in ~ (dgilbert)= .=20 Then I made ~/.local it's own filesystem ... because a) zfs, and b) the wri= tes from boloo are insane --- generating gigabytes of filesystem churn. For a = 50G file and 8 snapshots 15 minutes apart, it was consuming over 300G. Right now (with the error sitting there)... baloo is still churning. I hav= en't killed it yet. I gather there's some compression. [3:41:341]root@hit:/home/dgilbert> zfs list -rt all zhit/home/dgilbert/.loc= al NAME USED A= VAIL REFER MOUNTPOINT zhit/home/dgilbert/.local 19.4G 3= .16T 19.4G /home/dgilbert/.local zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-13h30 39.5K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-13h45 39.5K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-14h15 72K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-14h30 77K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-14h45 110K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-15h15 35.5K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-15h30 35.5K = - 19.4G - zhit/home/dgilbert/.local@zfs-auto-snap_frequent-2025-11-13-15h45 35.5K = - 19.4G - [3:42:342]root@hit:/home/dgilbert> ll -h .local/baloo ls: .local/baloo: No such file or directory [3:43:343]root@hit:/home/dgilbert> ll -h .local/share/baloo/index -rw-r--r-- 1 dgilbert wheel 28G Nov 13 07:16 .local/share/baloo/index [3:44:344]root@hit:/home/dgilbert> du -h .local/share/baloo/index 18G .local/share/baloo/index ... so that's not bad... but I don't know what's happening to the actual in= dex. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Nov 13 23:06:46 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6wrr4dlzz6GPvr for ; Thu, 13 Nov 2025 23:07:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6wrr2g5Gz3Md5 for ; Thu, 13 Nov 2025 23:07:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-6419aaced59so2131917a12.0 for ; Thu, 13 Nov 2025 15:07:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763075221; x=1763680021; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=jfeuXdKhqJ8buYdvYuOJsFRtQLK/NeJVyR7+7w0i9J0=; b=KPWnc2ko9d7Whcf3vYY1NQln6O6hW1jaM7t6GW9uRqjU/OecyZN6uUYBRGxs7BcnU8 7nLyI/RbSjfYlwwC9SYj0IrdGRht2McjPb50p2ivWKkS/o85pZ25+2smHqfkfkrm/vlz UxaYnR//Rqj9xYp/5j0qZA2+8lCMW0UoROOAlfCWfNqvJ1VpPAjZDKy3ITBE2hqJdiPd Zey9GfYQ11HBW6OYqe6EGSH2/tzrsE6r/7vsgpCSmRoB5AXVN5swvW33VTqgbaYrmRIZ Xj4k0PqJsCVz2Ov90w+yn89icYfJvpoeUVJj3Y6xgl9hQ1+GHpBd9P6BCODGsezgn0wZ XMNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763075221; x=1763680021; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=jfeuXdKhqJ8buYdvYuOJsFRtQLK/NeJVyR7+7w0i9J0=; b=LIJNkG+VNFQXrmkPcuXez8GssbgblUjfhPyZQN4IT8n+3Ni6+fN57sUtBpnnKlkDlJ FgGNIkpySP+sCquNVoi6MAzorxUtavmd/ny1mik6zqivW8oAgFrsaVBLV/+90eeN8SAO w0iABnvL4kzaS6HgP8BO5wbPA/giDz6DbTXTuIBxU+ohN1dGQhyBvfSsi5/GlQ88YVKA 5D0upfTPfdpNGfM74CP14hwRLaGB3NM4dblAUEtYdMLRBeJjnwKLfbQNElaxrbjvGeiO XOO+Hg2TCqMxyTzJ+GHB5A3+Y38pfe0xmj/khHwNjp1Tr7PARlz+9dN0TkzCUp4AngGY I1mA== X-Gm-Message-State: AOJu0Yyl+kYLqC3eDxL9UdWm23q3+vQNWRT9bxSQBcOKOlSgp8FkZQDX dbD/q9gv5RAAoPQq892z87DFJqbZ1wXlnuD9ZBaGSVu/XSTEYITRFdtt6F2Nsa+cfAJf60JxWnT Ccm6qPICjFOE6FK/VZFFqXwLM7lXzonOhxnk= X-Gm-Gg: ASbGncs5wW0eeeiRnOqOt94p1gktLAcdLNYa1tlELsqBTXr5ISP+YagQ/TpuvqfZraT COTvMM3fDeKGdFfUJ2ebusPFIgcVf0IYYrX7PMo6MFsVH4ur3D+lsQG3CZnu0IxRJWpkXvAwT6a jrXKOV7ao1dCVQFLGrrUD0mNWCCWTSysitHgSN6NPiRBYUUlwysjIpsKG+AmCmMqSYe4NdL1yIV lFa2jRO2ubr4wUf4GB9tZoIoBh22pcsj3gd8VvHHRL4CD2eKbob1xjXzaCVyOdoEJ5iPQ== X-Google-Smtp-Source: AGHT+IHoPyYmP+Cu/92DIhNIirTXQbUO75xWYj+wmG1rOwo1D8Tc2bDHfvW0K9s+CUgtPgTe89aUsqxBLSnnJAiBi54= X-Received: by 2002:a05:6402:430b:b0:640:e943:fbbf with SMTP id 4fb4d7f45d1cf-64350e2188emr928179a12.11.1763075220850; Thu, 13 Nov 2025 15:07:00 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <57797995-09bf-48ca-a232-8b91bc93adaf@FreeBSD.org> <4bb1700c-b71a-43ee-bf67-63172602205c@FreeBSD.org> <1d48ae49-66b3-467d-8ff1-6b277d68246c@FreeBSD.org> In-Reply-To: <1d48ae49-66b3-467d-8ff1-6b277d68246c@FreeBSD.org> From: Rick Macklem Date: Thu, 13 Nov 2025 15:06:46 -0800 X-Gm-Features: AWmQ_bnKD67bY7CHmoWNFktrvPaE7c5xsSTOQrrST3pOZrkKv_UL2buxE40f97c Message-ID: Subject: Re: nfs hang To: Ronald Klop Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4d6wrr2g5Gz3Md5 On Thu, Nov 13, 2025 at 9:19=E2=80=AFAM Ronald Klop wr= ote: > > Op 13-11-2025 om 14:06 schreef Rick Macklem: > > On Thu, Nov 13, 2025 at 2:45=E2=80=AFAM Ronald Klop wrote: > >> > >> Op 13-11-2025 om 11:41 schreef Ronald Klop: > >>> Hi, > >>> > >>> I have setup nfsd in a jail. It exports zfs fs. The kernel is 16-CURR= ENT/aarch64. Jails are 14.3-RELEASE. > >>> $ cat /data/jails/pkg/_root/etc/exports > >>> V4: / -sec=3Dsys > >>> > >>> /usr/local/poudriere/data/logs/bulk -sec=3Dsys -maproot=3Droot > >>> /usr/local/poudriere/data/packages -sec=3Dsys -maproot=3Droot > >>> > >>> /usr/ports -sec=3Dsys > >>> > >>> > >>> The clients run poudriere in jails. > >>> > >>> Now and than I get hanging processes and unresponsive nfs server mess= ages. > >>> > >>> All NFS threads are in this state: > >>> [root@rpi4 ~]# procstat -kk 5973 > >>> PID TID COMM TDNAME KSTACK > >>> 5973 100541 nfsd nfsd: master mi_switch+0x10= 0 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc= _call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+= 0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_cl= ose+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x= 1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > >>> 5973 100784 nfsd nfsd: service mi_switch+0x10= 0 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc= _call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+= 0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_cl= ose+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x= 1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > >>> 5973 100785 nfsd nfsd: service mi_switch+0x10= 0 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc= _call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+= 0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_cl= ose+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x= 1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > >>> 5973 100786 nfsd nfsd: service mi_switch+0x10= 0 sleepq_catch_signals+0x3e4 sleepq_timedwait_sig+0x18 _sleep+0x1a0 clnt_vc= _call+0x814 clnt_reconnect_call+0x960 newnfs_request+0xacc nfsrpc_closerpc+= 0xfc nfscl_tryclose+0x58 nfsrpc_doclose+0x294 nfscl_doclose+0x390 nfsrpc_cl= ose+0x28 ncl_inactive+0x14c vop_sigdefer+0x34 vinactivef+0xb8 vput_final+0x= 1f4 null_reclaim+0x1a0 VOP_RECLAIM_APV+0x20 > >>> ... and a couple more similar lines ... > >>> > >>> In rc.conf: > >>> nfs_server_enable=3DYES > >>> mountd_enable=3DYES > >>> nfsv4_server_only=3DYES > >>> nfs_server_flags=3D"-t" > >>> > >>> The filesystems are a zfs legacy mount in the jail: > >>> # grep zfs /data/jails/pkg/fstab > >>> zrpi4/data/poudriere-logs-bulk /data/jails/pkg/_root/usr/local/pou= driere/data/logs/bulk zfs rw 0 0 > >>> zrpi4/data/poudriere-packages /data/jails/pkg/_root/usr/local/poud= riere/data/packages zfs rw 0 0 > >>> zdata4/ports /data/jails/pkg/_root/usr/ports zfs rw 0 = 0 > >>> > >>> > >>> Interestingly I also have a bash process hanging which should not acc= ess NFS at the moment: > >>> # procstat -kk 83175 > >>> PID TID COMM TDNAME KSTACK > >>> 83175 111203 bash - mi_switch+0x100 = sleeplk+0xf8 lockmgr_slock_hard+0x29c _vn_lock+0x50 vget_finish+0x28 cache_= fplookup_final_child+0x54 cache_fplookup+0x538 namei+0xd8 kern_statat+0xd4 = sys_fstatat+0x2c do_el0_sync+0x6b4 handle_el0_sync+0x4c > >>> > >>> Any thoughts? > >>> > >>> Regards, > >>> Ronald. > >>> > >>> > >> > >> > >> Just noticed this on the console: > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283= e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc8= 7:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f= 2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283= e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc8= 7:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 263bc0f= 2:d39c94de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283= e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc8= 7:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid b7c1283= e:5166a2de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: server 'pkg.thuis.klop.ws' error: fileid changed. fsid 3750dc8= 7:289259de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE= ) > >> newnfs: Logged 10 times about fileid corruption; going quiet to avoid = spamming logs excessively. (Limit is: 10). > > Do you have more than one client mounting the file system? > > If you do, make sure they all have different /etc/hostid's. > > (Cloning a system disk without deleting /etc/hostid can > > result in multiple clients with the same /etc/hostid. That > > mean they are "the same client" to the NFSv4 server > > and that can cause the above.) > > > > If this is not the problem, I don't know why you'd see the > > above but I suspect the above explains the hang. > > > > rick > > > > > Two clients. Both have different /etc/hostid. > > I noticed that the procstat stacks start with "null_reclaim". And poudrie= re null-mounts the nfs mounts in the poudriere-jails. > > Could nfs+nullfs give some trouble? Yes, although I cannot say why. A null_reclaim would mean that the nullfs vnode is being reclaimed. However, if the underlying NFS vnode is still Open by some file descriptor, it cannot be NFSv4 Closed at that point. (An NFSv4 Close cannot happen until all opens by all filedescriptors for the file have been closed, because the NFS client below the VOP_XXX() layer cannot know which NFSv4 Open (done by different open_owners, which refer to process on client) are being closed. You might try the "oneopenown" NFSv4 mount option. That makes the NFSv4.1/4.2 client use a single NFSv4 Open for all opens of the same file (it cannot be done for NFSv4.0, but you probably are using NFSv4.1/4.2 mounts). rick > Or maybe it is just nullfs that hangs everything and the nfs stuff is jus= t a result of it. > > At the same moment I had git hanging on a non-NFS mount. See attachment f= or the procstat which also includes nfs-calls. > > Regards, > Ronald. From nobody Fri Nov 14 00:51:20 2025 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6z9S3Kg9z6GYNt for ; Fri, 14 Nov 2025 00:51:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-25.consmr.mail.gq1.yahoo.com (sonic312-25.consmr.mail.gq1.yahoo.com [98.137.69.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6z9Q4G1rz3g2F for ; Fri, 14 Nov 2025 00:51:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="sJkR/OtT"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1763081496; bh=TGq96YxqzdWD8ZLm02zZSkNFtRoPWMC5K+nXQYDqIqI=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=sJkR/OtT0hAdTp+CdA77ODTOv8A+girSr+yQK7aFbJ4x91mzl/5kvPCvDtnwGtc8G2YY+4hapXoeEwn6/cFPhnNWGu7+W8wRaDnw8b7RZXmNiyAnpaPXfKgHj8bpowLJbE5Dr9DHJZZF8OBmMY6vKIMKuHiSe5TsF4fERJ60/3tRuqFgUtz2zFS5Kfyq92Hy4ghW9bT3E+ekiIW0QI4wCnipi0/Ouddbh8xRfuzbMRB6WprgEMkyF4s4KxMnLSHXJcdHoO+2R4IMoJwPR0vInVutSZNLMp4EoVgMWxGGKrFrwdkHlrT6jwpiKMLyZE0YT6zNfXGV87v4nMrRGZnfXw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1763081496; bh=1jzO2CtzeiF5wGjYbiyQ3yK9s8D0Ge/Ov5qvkoHRzKc=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=hNtyKZQVhPcQhNFvro40IGHdR0zJ5LWKGaH7U9znzcRCGPeWebieMm/2v/iQMKHXohK+BKosdhq8iOwVM5yVFfDRXsjQKmoW02iek0Ad/OYia90hMPeEp83KzS7SEf7Xf/2qCUosIQP7Yfd52VJ632culQeoWv4jD75xyelxApzd49gaxHSCcOQLrP19fUJFZaJPpoaDhutjurS5CtEvqkfIlOylgLrFR8bZlZvCmEHRp+KFjA4W3CQMENpVtIHiv4Npk5l7Hw3ZhhltdgAOJxDgmk7pNVad0dPYCAmu55t2MC6tGnVbUp5fih9MCSURRpwujzj2C8VocbxbS5+iTA== X-YMail-OSG: pK5un8gVM1mf3OlaDi19c0RwAL_U7nvYYFXbY7Yo8tg2uf0kX3_sNkw5RbmHp5n fgA0VwsBkZJH2gSDFebITPUSE1vs2r4OCRc1j5QvljTGV6GKXzB2meA861IF3Lgi.unJE9ShJ6IK crKYdUtmC1NfJSXUxxvqdAEmPKtHM2yOYE3uXXCcBsZshnlnlTUXf8FL0k.RPPYoAFe1pEOstoI1 L9Wah7WvkttBQ4sgegfye6Py4QzV3m.QYLcS6_sPEowzVzBxjTXE8vLwx9iiC4u.x6NaN.BI.MiM y6dCoMaPC7Gv5fQRfKU1RR8Nv13VKsKt3e4NxaQWHOhDQxqYnRp3vQM_aa8CGepjbCz45rwh_j8k 1qAhkAi4beWutNwa8Rth49CYXkeTKeesw_P3qtsiAGgx75MRkGyQNmqSZ0moYVWJ8NYCLlgQ1SRh MX3LE.UkXOXrUasadOjZk0JoArnZnTKAF9c3ckweISHUiLPZldxjRPoCGO2GwQtaFzmcENw0lgVb Afn8Hw5ygXFehbNy6zjjXBMFO_2N8vcVuhTD0uk6jAGCudxdhurERolSJf2rl9r9A3M.wFmaU8cx 8JJWY41FzghG.Xqx.1dwHJyWqYJsEjYTAyrTK1DRKSRtdmhVOSvOb6T1iRi4C0z3TpZh4HIBhn5n _ox2PXrZlV9AWHjP9DpR_mXFw8mfCCEdFREvRZu0pF.YNrjJZ620ntzFE5E4eIhTvrEZNXXBzods kyYVfqRNcnUDAHCzmQgep40tc5mpwQ6owsxQApcjD57qKrC5ZD8tRuWFC8qgrUxDodc5jvlE9dt5 doVZmyprdRcng23G_BmZbPehYNr5QsQkUfEtZv2rwt2VjpykErG.Us09AQQB_3o2UhHLAI1vusao KNRl.JUZZXsz.lByouMAvjmPY7G1xGlkeKbg3EpELMBGnVYr0YX.LQ.S6Zk9duBFSkZ7rfI9hrcO EB8FGhYPkOsGUqce66I_garyvDjuhvwFK9C7lFGgjLRT2UNXhMSYcxTUBsf3Uhx.7AYhZrhkhGHN fwnQNZjjp7VIExME80FiWx3tRSN0NGUUlPINnCdbHTt37AUV_SQuTae7Ryq7YJMa3ql34r7IARS8 ADFP2jK3oda_0hDJNfGNC5UiEfqOSp3xJ3tXqwc_MYAm21Ya4kFzsIXgPzYhXV9PsvKB3hQ_F5Xd .TB.b5hoGi_amoryT.eMG27nAhqn5spufZWH2A9VDL7Sb7PyDZvkEHsHnDnN0X30PoQ0nZREPjV7 7kx_d4GCxUbr77ksCgxEkn5snIfl9I1_lQvSEBadKsLgcMw7uuNbhO.S5WhvJm_LqV1fXjWLwaxd r0sqO2q_tQwh73eByE0UbVQzhSwLO0wia8RDD39YE4GnAYYfNgw9C0A3FTVrEypTyL6TCxyheJjf uwNqpyDa191XX3U2zmwkYuTURnMcKt1RxhE4aKDksi08zDdShyqHtbJqcjERXydNSQ_0eiKZrfZR WV4RR15vpamxdyXtZT47xic5AdWo0AtFq4RmSMiH8UzhypUcz8aWpqMJaPAOZWWethut9bSDfSNe wjmaWLKt5wvFnI7DW4n0lbPG4_Xsi1LKk2gd4CeIHzh_XJ50.SUy5IXnHRhCYtIiSQkuI0zbVQxJ MetbzN9_vPghDOd4Q4D3sn6yEc1Kn3wAk3ZuJP2LU7sUTtN378XlBOgmrwF3lL.hQ1TeSg_qMv0E 1JbC1vaw27LZTWG2crHaB0F2frczjUBn.QcB5ikC0zoLkCl1Jjs2ik_naBBg4SNIfIvUikbYN_wU oSL7J4EDHPmZEjhufNdRpQNJsLuq2bJgq8xsehrP_Ty_g34e4Ce7cIYcnxkh0T_wWHnPXl_ajlmp 85fM.Qjp5n3Md.r31GvS_9nsgm3XR7kWWYQ0sVRaiXFcrqkyh.nEclTBDv01MW02UCRcJIav_Cit Ll0zF6XeVYNyvlqJMn8i246P9CTmUXGMpLOi8ZVroPfY6NoaA8pXHtq7KzqoUv7wgt.NxdTi4Ldg Pd7gORY.1bI9lQAAtPRu7KOV6Eut_6he.MF_uSWUIZXCQmYUX5yNEZBoDUxcM49goD33iwahlT8A C8Q6Z4co0dcl2OaLRX75AkBqfcbhCznnEahpTz1aLuwCUZ8vRkdE0YKwvqx13z6k0PGahaVFkh3a kFAu1iq6L06c6ksaRH8S9hvJzPMqOCTuxANvNz0naMJyrAk7c1DYkipv821j1Vce8SYNbqgg1XC2 TxWMrpuJKSFqGCiOS6S0vHsAN2hoHxz.S_SK1kkvSD2W92C9ypk3aShS5_edn5vpPZD3IxN1plOp SXY5W X-Sonic-MF: X-Sonic-ID: 20f60771-caf4-43e7-97f2-12cac528f93f Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Fri, 14 Nov 2025 00:51:36 +0000 Received: by hermes--production-gq1-76c986f798-2d5kj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2f365ac67aafc99443667eb3dfdae032; Fri, 14 Nov 2025 00:51:31 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: nfs hang Message-Id: Date: Thu, 13 Nov 2025 16:51:20 -0800 To: Ronald Klop , rick.macklem@gmail.com, freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3826.700.81) References: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[freebsd.org,gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.206:from]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.206:from]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-Rspamd-Queue-Id: 4d6z9Q4G1rz3g2F Ronald Klop wrote on Date: Thu, 13 Nov 2025 17:17:48 UTC : > Op 13-11-2025 om 14:06 schreef Rick Macklem: > > On Thu, Nov 13, 2025 at 2:45=E2=80=AFAM Ronald Klop = wrote: > >> > >> Op 13-11-2025 om 11:41 schreef Ronald Klop: > >>> . . . > >>> > >>> > >> > >> > >> . . . > > Do you have more than one client mounting the file system? > > If you do, make sure they all have different /etc/hostid's. > > (Cloning a system disk without deleting /etc/hostid can > > result in multiple clients with the same /etc/hostid. That > > mean they are "the same client" to the NFSv4 server > > and that can cause the above.) > >=20 > > If this is not the problem, I don't know why you'd see the > > above but I suspect the above explains the hang. > >=20 > > rick > >=20 >=20 >=20 > Two clients. Both have different /etc/hostid. >=20 > I noticed that the procstat stacks start with "null_reclaim". And = poudriere null-mounts the nfs mounts in the poudriere-jails. Do the poudriere jails on each host use the host's /etc/hostid (by = content)? Any worries about needing poudriere jail /etc/hostid content uniqueness? > Could nfs+nullfs give some trouble? Or maybe it is just nullfs that = hangs everything and the nfs stuff is just a result of it. >=20 > At the same moment I had git hanging on a non-NFS mount. See = attachment for the procstat which also includes nfs-calls. =3D=3D=3D Mark Millard marklmi at yahoo.com