From nobody Sun Oct 29 11:41:12 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJDxm6X9sz4yrhZ for ; Sun, 29 Oct 2023 11:41:16 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJDxm1PMJz3LYN for ; Sun, 29 Oct 2023 11:41:16 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=kS7DhZ2A; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=tcXr1455; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 859085C00B1 for ; Sun, 29 Oct 2023 07:41:15 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Sun, 29 Oct 2023 07:41:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to; s=fm3; t=1698579675; x=1698666075; bh=GCRZz3RZulNbmJxV2bfeOhGNd Hnqat2jhBLLWoutl68=; b=kS7DhZ2AqEmBd5gJWGcb0+2oS7s4SVlyUgW4u6Iv5 ONLofs+kKz8IDRmdIf0/OcbAmfL3d6dnWrwQ8mbk5K9IGTPKPTPjVYwBbyxYP37t xMi0nb4TlxhOJTAFoOaPASUoQlAc9/oXEY1N3SpBnwqfrRGh3hncW0S1u1R6CCeR qVXVUnkbETyqI12UfOG75aQhr4MIb/OLNT1DYcjtzoB5ZDFB8G/57KMuPRQPnaxq f/rS6Ue0+EOBaAUBqH1aEANzaCZ6tzdlgOMcTe8WUF+psK94Bml+yKAqvLoRFdCt +0K+O9WbyFx9uiE5VSLLGhAwzE8UAW2pATIlMTowKeLhQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1698579675; x=1698666075; bh=GCRZz3RZulNbmJxV2bfeOhGNdHnqat2jhBL LWoutl68=; b=tcXr1455V1W5g/WUvm4PpE6eymfu2XQEcLvzTxIJdQhtW3lTx3E DQVW/TaD4W/1GM3moEXwfRxOPMB5lElcndWWjOxgADnALkcILxdLE4IgPgL41fhC f6yJjoH3yVUVKjfMmCN2j2WEA0TveumDseGb9nZUTQmb4C2rt3M/0tNkTUx8EcQW TcVenbaGMxgvOC+tlH/1wkSGTxYOXTbw9It6zqn347en2f6+K4J5EsCYOQQkdVZk 9zowRtfdlIxtPzCCDCOFeBYWQo4UR6HE2fukvXOJ4YvP7ifnpq9aIe6mTNX51Dbq 6Dqdr+FuinoT+cAB7SzHzbCZYPljd0Rm0NQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrleekgddvlecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehttdertddttd dvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrghtthgv rhhnpeevudffiedvffffgffhgeefjeefffdtieetheetkeefhfdvfefgtedtueehgeffue enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhi ugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 29 Oct 2023 07:41:14 -0400 (EDT) Date: Sun, 29 Oct 2023 11:41:12 +0000 From: void To: freebsd-fs@freebsd.org Subject: optimising nfs and nfsd Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-4.61 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.91)[-0.911]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.25:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SJDxm1PMJz3LYN X-Spamd-Bar: ---- Hello list, The nfs instructions in the handbook are rather terse. I know there's been lots of new development with nfs. The zfs property of sharenfs, version4, KTLS etc. The "readahead" property for clients. Would anyone here please point me to up-to-date resources? My context is nfs server exporting via sharenfs with freebsd14, -current and debian-based linux clients on a gigabit LAN. Some workloads are very interactive. Some are not, such as backups. It works, but it maybe can work better. Thanks! -- From nobody Sun Oct 29 20:28:03 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJSds4bKFz4y9Jf for ; Sun, 29 Oct 2023 20:28:17 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJSdr38S1z4Tfl for ; Sun, 29 Oct 2023 20:28:16 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hAeqnCQK; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::332 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x332.google.com with SMTP id 46e09a7af769-6cd1918afb2so2311454a34.0 for ; Sun, 29 Oct 2023 13:28:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698611295; x=1699216095; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=oCffSGV3vofi2RNdfxThXC4yTBUurlw1ELczXkgaKhQ=; b=hAeqnCQKJv8ECMZdQ4HkUeaeu2edbazntX5vBrZgQvFENjLLEI4UMf9bv68/9foWLw wHD9v7fLW445wwYz4Doh8/BqapIgw5Z2SvERs7oPq+18FQ9DsDMPM8NECAqpS/CretjM t5ZJ4RbcvUkwUvYePqmy2uOwBUTCsLapemFI6zdF644lPdCfeNfNQsS0/OeoosZwxrMV gP7XcG4JtnUok1tmO9wjq4NukidMCPE52bG0vYi9UIE9/ZVMgA0P8d8nKQSIOqTTy1tj UAaJBW6C/3KRYfzImwXXfT+ct8oO5P/nWrjigxXwPbCcrPZq7LIwrwaDv7Dw0FFGgfnr PChg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698611295; x=1699216095; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oCffSGV3vofi2RNdfxThXC4yTBUurlw1ELczXkgaKhQ=; b=i8zPeLql9WJvCB0+b4Hsikr3qefFynQHl4xnhMY/EGGZCgIs4UbAo9YiITLihu27Ls 9YdKRoXHhtAslFw7qXUxtvno00Eusi9AKEMopYKb+6+DICkxVNWBOzx9MvqIg9bbnB+J l8DkVBibKURl1NzI0UWFAKy6ft/KJ2hcGW2H63VI+3bE2pRWiCfogpl8PKLvyhWfBegC Wmo/tQYrIqzbHxQfa0oWbb7jCoSl1YGhbLdCRrWM69BkRpoEQZLZNBDuM0E48kl+3QUI vJg+k3qiP9TJYp56fKoavVdqWx1UKbOEx3OFh4XbEFjBttAG9POHcLXAxL5jHWYGtutF 7lsg== X-Gm-Message-State: AOJu0YxFWd6/EjR0GxJ3DESa36j8zasO49sn+o03szzh9JsILZkjMQ9h w7i+CKFPmWULsYksg0Zn8EZi8AlA2/50rwBuxhc2/idGlKVS X-Google-Smtp-Source: AGHT+IFPJn2c8LhsaumPk+/FquOyklpWTpisWLbvVWdTCI/HiSFQS119yUwdIP0+xqg+iKJJnBMTtxSHN8MCzEJYOOM= X-Received: by 2002:a05:6830:1c7:b0:6b9:8ea6:fbc2 with SMTP id r7-20020a05683001c700b006b98ea6fbc2mr9961966ota.27.1698611295018; Sun, 29 Oct 2023 13:28:15 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Sun, 29 Oct 2023 13:28:03 -0700 Message-ID: Subject: Re: optimising nfs and nfsd To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.966]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::332:server fail]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::332:from]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SJSdr38S1z4Tfl X-Spamd-Bar: --- On Sun, Oct 29, 2023 at 4:41=E2=80=AFAM void wrote: > > Hello list, > > The nfs instructions in the handbook are rather terse. > > I know there's been lots of new development with nfs. > The zfs property of sharenfs, version4, KTLS etc. > The "readahead" property for clients. > > Would anyone here please point me to up-to-date > resources? My context is nfs server exporting > via sharenfs with freebsd14, -current and debian-based > linux clients on a gigabit LAN. I have a few primitive docs, but they do not cover what you are interested in (all found at https://people.freebsd.org/~rmacklem): nfs-krb5-setup.txt nfs-over-tls-setup.txt nfsd-vnet-prison-setup.txt pnfs-planb-setup.txt However, here are a few comments that might be useful... - If you do "nfsstat -m" on the client(s), you will find out what they are using. If the mounts are NFSv4.0, consider switching to NFSv4.1/4.2. (I consider NFSv4.0 a deprecated protocol. NFSv4.1/4.2 does assorted things better, including something called "sessions" which replaces use of the DRC.) If the mounts are NFSv3 and work ok for you, that's fine. NFSv3 will actually perform better than NFSv4, but will lack things like good byte range locking support. - If you do something like: dd if=3D/nfsmount/bigfile of=3D/dev/null bs=3D1M and get wire speed (100+ Mbytes/sec for 1Gbps), then there probably is not much more you can do performance wise. Mount options like "readahead/rsize/wsize/nconnect" can improve performance if the mount is not already running at close to wirespeed. (They are all in the "try it and see if it helps/your mileage may vary" category.) The Linux client folk do try and make defaults work well. Interrupt moderation... - Most NICs do not generate an interrupt for every packet sent/received to avoid an interrupt flood. Unfortunately, this can delay RPC message handling and have a negative impact on NFS performance, since it primarily depends on RPC RTT and not bandwidth. Most NIC drivers do have tunable(s) for this. Again, your mileage may vary.. If you are using NFSv3 or NFSv4.0 mounts, performance can be improved by tuning or disabling the duplicate request cache (DRC). The DRC is an oddball, in that it improves correctness (avoiding non-idempotent RPCs being performed multiple times), but slows performance. For a good LAN, TCP may not need this. (For TCP mounts, RPCs are only retried after the RPC layer gives up on a TCP connection and creates a new one. With a good LAN, this should be a rare occurrence.) # sysctl vfs.nfsd.cachetcp=3D0 is the extreme tuning case that turns the DRC off to TCP. (Again, this is irrelevant for NFSv4.1/4.2 mounts, since they do not use the DRC.) NFS-over-TCP (called RPC-over-TCP by the Linux folk) is discussed in one of the primitive docs mentioned above. It uses the KTLS to run all the NFS traffic within a TLS1.3 session (not the same as an NFSv4.1/4.2 session). This has obvious security advantages, but can result in about 1/3rd of a CPU core being used/per NFS connection for encryption/decryption when the NFS mount is busy. I am not sure quite where the LInux client patches are at this point. I know they have been testing them, but I suspect you need a very recent Linux kernel to get the support, so I doubt they are in most distros yet? In summary, if you are getting near wire speed and you are comfortable with your security situation, then there isn't much else to do. rick > > Some workloads are very interactive. Some are not, > such as backups. It works, but it maybe can work > better. > > Thanks! > -- > From nobody Sun Oct 29 20:30:21 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJShV5Bbtz4y9G1 for ; Sun, 29 Oct 2023 20:30:34 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJShV0DPQz4V64 for ; Sun, 29 Oct 2023 20:30:34 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=OUPuliTm; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-28028f92709so770454a91.0 for ; Sun, 29 Oct 2023 13:30:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698611432; x=1699216232; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Sx5iOtvxBlK7/5juWu/6RUT6jJG4sDxErxES0aPTtKg=; b=OUPuliTmMHIq2fRfgpylXybsOcfQHycCbQPcPLazjK43CY7p6BBSH7n9CVz6dvKm/Q drqB73JGROZY5ZvP7MFzB7kxXZgO/kUMiEwZn+ZXB4yJY57hl3qd93I+LTOeW5u2uANs mdxGj9lMzgGrzT8Hu6R8eRxkpcIBJLA0Z+BaxM4o4TKvf+V/L+Egj3gTlbs4Lrj9OLeX /4gwb7NcsvE2ofIIbVOifxFhkVIVeusy6SkxR20vorylc3LQPgiqWShvbxw/pkYrKrb3 yJVGmNO4dZA1c0KgkLDCx6iEZd2cAgJUygPm+HthF22FxsqwVI2ZTrPJg5RUarcZg3VF o8sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698611432; x=1699216232; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sx5iOtvxBlK7/5juWu/6RUT6jJG4sDxErxES0aPTtKg=; b=xBtld1ah62f0ptENUItk2yx7rjeG8WV2D/rpxMUiAbz8OW7lqtQld40NQERVxVb4ky 5zEXJBjg7vi3x1HJBNoJ3e3f6Q6NojZmSfI8+sANWQNLhNIPGTCV/aSA9G0mS0Vc2zO2 fQLWvibHPaz1rmBHVXeb+mFK7Hp9nbvJzn8urFsCgJuAXABuVGO0WXEnCGEb2JEpI+Q6 ETf+ylaIC1IhJZKI2IOtNrcv5GAF/6G0dW2QOQWkUA0ySDKC8tLpVb72RVVSOUuZPZar i9FGNvF7JbFmtjQzn6e/sIkvdxlhE6E/wWRKtBeePHziVbbB4PKftb2QZJlo7AXsBcyx T3Bw== X-Gm-Message-State: AOJu0Ywyo88jnehPK5L5nB2hiDkvw8/QIwDd48+Sm0XJB0DAfGJiDOmM 1+rNU+Xl+B0kQKX7RzfnMj/ENhlIZCzxvU3e6RrikEN+GA== X-Google-Smtp-Source: AGHT+IEYDY+ueF8ivRbxkvPdLkSslIN0vETpiFUuicZ/AdTkApnrKxqa0CNMMLpm/Kl6yVsZt5Wy+j/cf9ur8UoaYgE= X-Received: by 2002:a17:90a:9708:b0:268:808:8e82 with SMTP id x8-20020a17090a970800b0026808088e82mr9940504pjo.1.1698611432268; Sun, 29 Oct 2023 13:30:32 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Sun, 29 Oct 2023 13:30:21 -0700 Message-ID: Subject: Re: optimising nfs and nfsd To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.966]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::102a:server fail]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102a:from]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SJShV0DPQz4V64 X-Spamd-Bar: --- Oh, and since I only get to test in my little at home nowhere near server grade environment, I'd appreciate comments from others related to NFS performance, too. rick On Sun, Oct 29, 2023 at 1:28=E2=80=AFPM Rick Macklem wrote: > > On Sun, Oct 29, 2023 at 4:41=E2=80=AFAM void wrote: > > > > Hello list, > > > > The nfs instructions in the handbook are rather terse. > > > > I know there's been lots of new development with nfs. > > The zfs property of sharenfs, version4, KTLS etc. > > The "readahead" property for clients. > > > > Would anyone here please point me to up-to-date > > resources? My context is nfs server exporting > > via sharenfs with freebsd14, -current and debian-based > > linux clients on a gigabit LAN. > I have a few primitive docs, but they do not cover what you > are interested in (all found at https://people.freebsd.org/~rmacklem): > nfs-krb5-setup.txt > nfs-over-tls-setup.txt > nfsd-vnet-prison-setup.txt > pnfs-planb-setup.txt > > However, here are a few comments that might be useful... > - If you do "nfsstat -m" on the client(s), you will find out what > they are using. If the mounts are NFSv4.0, consider switching > to NFSv4.1/4.2. (I consider NFSv4.0 a deprecated protocol. > NFSv4.1/4.2 does assorted things better, including something > called "sessions" which replaces use of the DRC.) > If the mounts are NFSv3 and work ok for you, that's fine. NFSv3 > will actually perform better than NFSv4, but will lack things like > good byte range locking support. > - If you do something like: > dd if=3D/nfsmount/bigfile of=3D/dev/null bs=3D1M > and get wire speed (100+ Mbytes/sec for 1Gbps), then > there probably is not much more you can do performance wise. > Mount options like "readahead/rsize/wsize/nconnect" can improve > performance if the mount is not already running at close to wirespeed. > (They are all in the "try it and see if it helps/your mileage may vary" > category.) > The Linux client folk do try and make defaults work well. > > Interrupt moderation... > - Most NICs do not generate an interrupt for every packet sent/received > to avoid an interrupt flood. Unfortunately, this can delay RPC message > handling and have a negative impact on NFS performance, since it > primarily depends on RPC RTT and not bandwidth. Most NIC drivers > do have tunable(s) for this. Again, your mileage may vary.. > > If you are using NFSv3 or NFSv4.0 mounts, performance can be > improved by tuning or disabling the duplicate request cache (DRC). > The DRC is an oddball, in that it improves correctness (avoiding > non-idempotent RPCs being performed multiple times), but slows > performance. For a good LAN, TCP may not need this. (For TCP > mounts, RPCs are only retried after the RPC layer gives up on a > TCP connection and creates a new one. With a good LAN, this > should be a rare occurrence.) > # sysctl vfs.nfsd.cachetcp=3D0 > is the extreme tuning case that turns the DRC off to TCP. > (Again, this is irrelevant for NFSv4.1/4.2 mounts, since they do > not use the DRC.) > > NFS-over-TCP (called RPC-over-TCP by the Linux folk) is > discussed in one of the primitive docs mentioned above. > It uses the KTLS to run all the NFS traffic within a TLS1.3 > session (not the same as an NFSv4.1/4.2 session). > This has obvious security advantages, but can result in > about 1/3rd of a CPU core being used/per NFS connection > for encryption/decryption when the NFS mount is busy. > I am not sure quite where the LInux client patches are > at this point. I know they have been testing them, but I > suspect you need a very recent Linux kernel to get the > support, so I doubt they are in most distros yet? > > In summary, if you are getting near wire speed and you > are comfortable with your security situation, then there > isn't much else to do. > > rick > > > > > > > Some workloads are very interactive. Some are not, > > such as backups. It works, but it maybe can work > > better. > > > > Thanks! > > -- > > From nobody Sun Oct 29 21:00:26 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJTLy4WWsz4yC57 for ; Sun, 29 Oct 2023 21:00:26 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJTLy2BZ3z4Yrf for ; Sun, 29 Oct 2023 21:00:26 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698613226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=aXbYsAAETFoqkvUcGAb5jD70x/ntZNR/1ucC+rLv9iE=; b=mhVxyflDWJk2G8kaXAHLeKt7ZhjaKX7bNLpcMWOY0JlgWuxDNvhv/Jqmkf5Z1vx55TaVCM 0ZyDu9ZkjtdmLSYN7srmxPf+I/1fM9o00okjppFZ6aB/K3EQb7H7/gYyHPFkJIHNFePSYb IU2wY6zhtYUeGcEyJFmfcvKQHD/OnONPNYR2G9yDR4/AIIFLupKZ0vKfabLFc6ByNytvpW E9U2CSYVVRwwsXPcG58ksbJKrLsXR33NsUrkevRlH+K+1CtB5eqfW5Xmameksk1t7MatVb Uz5i90eKAyYKPgLqd3jRlGxeXOzXjd4HhWQfRDpgQ+pV5LTicAq/l4owaL5NOA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698613226; a=rsa-sha256; cv=none; b=OyXcYFAH36fV+/jo6pwYBer7uIOsJVpf7pD7YPVtN2Gami0ykd6ZKkM4Iv2CY4rWg6gNvK EeqR4mcsNA804lD9LYZjZalwoOgHIO3aEGBs2p2Y7RVWDSX7aGarnx9UhApizm2ldaqA0u J6wRyYCiQyV0fBqSFw+iGxN6cM4P153D7yIVYnXMGw4wgawqo7c2/9RDfQdxf0HHTCrrCi ERNqOho3dxLnLxIEqSDkzdoeaRIIf63C9h8kghLiPfCShUUppXiz7B8q0/Z9Mk5vc10iBF pLR7fLCjZ1PliXPJfj3lmm7sXlvXaTlLu/yRPD6a3CyBxxpsJKQf9IkzOQ8KJQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SJTLy1JDxztJQ for ; Sun, 29 Oct 2023 21:00:26 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 39TL0Qu8097739 for ; Sun, 29 Oct 2023 21:00:26 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 39TL0Q1R097738 for fs@FreeBSD.org; Sun, 29 Oct 2023 21:00:26 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202310292100.39TL0Q1R097738@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 29 Oct 2023 21:00:26 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16986132261.1DfFF.95754" Content-Transfer-Encoding: 7bit --16986132261.1DfFF.95754 Date: Sun, 29 Oct 2023 21:00:26 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 244692 | gjournal: Does not support TRIM Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 5 problems total for which you should take action. --16986132261.1DfFF.95754 Date: Sun, 29 Oct 2023 21:00:26 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w
Open        |    244692 | gjournal: Does not support TRIM
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

5 problems total for which you should take action.
--16986132261.1DfFF.95754-- From nobody Mon Oct 30 08:41:05 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJmvT4g5Vz4yrV7 for ; Mon, 30 Oct 2023 08:41:09 +0000 (UTC) (envelope-from yushang@outlook.com) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2035.outbound.protection.outlook.com [40.92.40.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJmvS2fKBz4HTS for ; Mon, 30 Oct 2023 08:41:08 +0000 (UTC) (envelope-from yushang@outlook.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=outlook.com header.s=selector1 header.b="KfbojC/R"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of yushang@outlook.com designates 40.92.40.35 as permitted sender) smtp.mailfrom=yushang@outlook.com; dmarc=pass (policy=none) header.from=outlook.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mVy6Wkg4DK7dOsP95cYrSyct8ZWuXJcwoAs8HAdtOnAd8G2lUAV/wFfwad0GWFPz8JMFocj6g+PhnhR5kyRUtxHaeKwq0P7NoCrR+BMWWy5vSoCB4TxsXK2gNv0qa61v1+RE6g4l9OUrZSvLsmccAYCKZR9PXX2dciuXN3657rzzXw5wYmd8e8rvQcA5SPCa496G1sTLrAGHImzQlzydTNbox+EafTbD+d6apvSmx9sWfjWbl4BKgp0tMHgIOh/DDH0q0Qp7mAEB1sqnnYJGPaVVZ7cZDqpLGkDBxu4baEsr9ITADA6Vigw64RreQfw+KfHu9/HVOu7luprUqO96FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fF2SL20qwNEG4B6lXq4IdgFZqRjXEK7VgMcUXTYXLxU=; b=PyE4BveD0OzHwbH4Rzfusfys4DM4cO3miNNrvCGGR+85daQ6eNPptUal9BCeJF5TI9e2PbikYk2rMhUUUlO6OfYKh4yCd3nSPuHgnhAio60yB0svkA0HUKvPs4yORddNcQt2quFn/0faG41EopUqumhdFBgnwG7m5E4fZc8sLt6PlFhZG7D/c4+eUTRKLrS8W9RcV6b0YyVyOKbIslDkrxfmCYtdLm51kq/gozETpN9luA0V5BNMkh6SdKtGF1b+A2f8pN5MY7pd7GKetfQcNNzQCVM53J8oAQdLSg3u2ERlw6U9xTL4RNTsFSnAit+mh25zv7hW9FfVNAiPI2maDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fF2SL20qwNEG4B6lXq4IdgFZqRjXEK7VgMcUXTYXLxU=; b=KfbojC/RNXqOt6ANIHSRvBNnkNMb31bfo+zHmzcoQnnTBPwu0fbEHaLsq5Q7Ffdnn2Dv4jgfp6Lus3maqJuWiZ46Lf4QhkciHltKMOf3gNDfqk/CUoeTS7MUEcvRlaD+XD/Zlz56ZRhCMpqSABnagCjs0b1Yxuy4NsdZnH6v0NM5g+75LloBdfnTZc4uCPPB0zXYxvwu2CYA71GEGEOknVsHHm+0pwsQWVbOWRH8oV0lY3WPzzPnfRVw+buIdhu0c52wL5pbMvOu95HPuKH4JmAjoRePmyf45R4AKqcgcDdIouLPMDHLjyzIoM6VN+WMXF+ZFVmjv8FuFhRgqSZCvQ== Received: from SN4PR17MB5862.namprd17.prod.outlook.com (2603:10b6:806:216::20) by BL3PR17MB6138.namprd17.prod.outlook.com (2603:10b6:208:3bd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.14; Mon, 30 Oct 2023 08:41:06 +0000 Received: from SN4PR17MB5862.namprd17.prod.outlook.com ([fe80::7cd3:46fd:62c8:4f15]) by SN4PR17MB5862.namprd17.prod.outlook.com ([fe80::7cd3:46fd:62c8:4f15%6]) with mapi id 15.20.6933.011; Mon, 30 Oct 2023 08:41:06 +0000 From: YU SHANG To: "freebsd-fs@freebsd.org" Subject: about the vnode free list Thread-Topic: about the vnode free list Thread-Index: AQHaCwtAPjPw+C3Seku3u3HbKqdTig== Date: Mon, 30 Oct 2023 08:41:05 +0000 Message-ID: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [JnMpFJKwteRQDJI0GyyKf4s9L/uo/J0L] x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN4PR17MB5862:EE_|BL3PR17MB6138:EE_ x-ms-office365-filtering-correlation-id: 67cb5197-1f09-4bc3-cf9c-08dbd923f4f8 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UzmNMhRjkCcChbaz1R2+Xz7so/l2rADS07pLMy1Btde8mkwzWZDx8snzajCYRyvhcYzGCO2Il7bqvxvje1OICTAPvbTFS5MV3kH75EM+RmK+36FujYWmopwBNq10yjly0X/inxCgFAsHqzwWqZnWx/X+8L2N2x1OIJnYmKQ+hbjNjHEBJABoZ8kZp1Vbp9Y3wiHX0nLzJaIyZG4he6ex/NUZv5aO2BeUNUjdPYSOsOwKbZFYfHy7aMsuP7dOdRrFTndTs44QFZ3l38BhFyWLoyF+3B8Tf8de6HMFmr8BL2IHIKw+d7QJrtvSTnldqUT48kLPzro7P6tPtIjgZFaS80W3RVQJCy7F4CwFPcOgUVuw5x1NAbZt2keWaOAW9MVq2Fdd2FNoKTMZoT5O7qnnK8ibHNgFgs89PviNgjoKY7ktByWEddjBT3rNZtVq2N+75AOsdWeXJ3mAwJQ2ob6qjO1NTwEdzuPWWn1eek5IcrCCOYsT89sxN/t7As8fDFBJTWQ/1GHulS+N5lSQK2syodMkv1S2b5kf3EfsmiBg4SjK2cIP2ahrTZktIocvswbb x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?saD7Uc2te+FndPeKIjYjOJbJWQikmLnZhfULPJ02oj8mBU0uHKjhDhfnk5Jt?= =?us-ascii?Q?f4KMPFAmJRs29r2e1LjyEkEN3Oj11wwqv1UGhKm8YC2aTD/Wal+qVIExyix3?= =?us-ascii?Q?538bFdfzK/nGtwQjjFNEhbtt5hY6req0novY4RT3dEC9/G9KFjvsva6nbSPT?= =?us-ascii?Q?3qKWr6wxaHO7phjGn2GSqI7BZeavtHNWjE+Y+CTe9q1VfPZahHoxIlFjx5Jt?= =?us-ascii?Q?6OaxTitUaFGxWk6reg0CC4I1UQhfWHCIm+atLhAOFyh1XiQQc8TbMuXl40tL?= =?us-ascii?Q?L4wXlOjBG3OZ4xxK6GtPlmig76nMsF2sjBdmLr8ArAWL/SBUIcxTcwAFS+8h?= =?us-ascii?Q?dqREdl4rVMXjgpN5LgSKyzMiwriqw2DvQI5PoRsa8DrpKFMm+8KRsQCec0nD?= =?us-ascii?Q?65Pxn8aZmlcIi3G0z8uO7O2FhzJvwGNVAV6kjrbGxxhSErZymPMY4SEXXkHX?= =?us-ascii?Q?1hPbiTRmWFiLYywaNqjCVOeb9vmHjVYmGgCMiG8BwNSAg97LOMe/hHy9wlK6?= =?us-ascii?Q?6IvSZ9y+DRcA8gdC+tDl82lnGNtXRRx0IBb/VDnBJXeqvfxytN91h2MHpT4O?= =?us-ascii?Q?SxEQfNKkuS1iBQYoShUQGF5BVTDTgq0Ta4/xBDhdDfgWItOB8oSLtvGjPPp4?= =?us-ascii?Q?c3/c4DbAtS7rNweEF5RG7zjihZPiuJWNmg8nt3WgtcUFO0R9nKwNNOvb5LeJ?= =?us-ascii?Q?N8VBaxY8oCCTmjI1r4doYPaK1XYnerblvskYGUhfFeROGc4tTxWoTbkND2Se?= =?us-ascii?Q?AWMXT5RggsO7qj/BDMYS3KIejQUTe//a/uoa9fnmr5EODjwRBjOhviWqlD1P?= =?us-ascii?Q?s933GYoeB12oepRHd94k4swbIpEXCswZF7yF319xF7C5xkuJLSJ5QsMAWoTK?= =?us-ascii?Q?SRpN6B6ykoxi5xRI39s5GlPYoLvljaQlEcwbx2LEhbkoCnW2O5u65R42VLNl?= =?us-ascii?Q?4DM3I/RET74aFpiHTWXdY9EPXd7rOhOHUXw7dscqJBF7zw3H00uNOSIpeJka?= =?us-ascii?Q?h3m+kjxj46ZKqZhaTGzI0hfdDCW6UJUJ/hcXHd+RvW2lqRo/ggAJK/qbbLKb?= =?us-ascii?Q?QkMJwfS81UmsHSxeIO2f0zqUADZaFI3zcaxPjWW9ycDECNP1dY1pMViiMcwp?= =?us-ascii?Q?KPYxr0GqwtM2S+/zO/URY31YWHYUlD3bFKfM8dgMNpTCm3W6T6uNirHNd5Ou?= =?us-ascii?Q?dcTWd4K0H46sLs9C9C/wdjaDOmKgR5bJF652h2w4VQKtpjeIalsQA6A//80?= =?us-ascii?Q?=3D?= Content-Type: multipart/alternative; boundary="_000_SN4PR17MB586231A4736EE906CF0B93E4A3A1ASN4PR17MB5862namp_" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR17MB5862.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 67cb5197-1f09-4bc3-cf9c-08dbd923f4f8 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Oct 2023 08:41:05.9478 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR17MB6138 X-Spamd-Result: default: False [-4.96 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.96)[-0.958]; DMARC_POLICY_ALLOW(-0.50)[outlook.com,none]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; R_DKIM_ALLOW(-0.20)[outlook.com:s=selector1]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; FREEMAIL_ENVFROM(0.00)[outlook.com]; RCVD_IN_DNSWL_NONE(0.00)[40.92.40.35:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[outlook.com:dkim]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[outlook.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_FROM(0.00)[outlook.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.40.35:from] X-Rspamd-Queue-Id: 4SJmvS2fKBz4HTS X-Spamd-Bar: ---- --_000_SN4PR17MB586231A4736EE906CF0B93E4A3A1ASN4PR17MB5862namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi guys, I'm reading the man page of getnewvnode which says that the vnode is either= freshly allocated or taken from the free list. What I can't understand is = where is the free list,is it the vnode_list? Many thanks. --_000_SN4PR17MB586231A4736EE906CF0B93E4A3A1ASN4PR17MB5862namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi guys,
I'm reading the man page of getnewvnode which says that th= e vnode is either freshly allocated or taken from the free list. What I can= 't understand is where is the free list,is it the vnode_list? Many thanks.<= /div>


--_000_SN4PR17MB586231A4736EE906CF0B93E4A3A1ASN4PR17MB5862namp_-- From nobody Mon Oct 30 13:47:26 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SJvk12txGz4xxQC for ; Mon, 30 Oct 2023 13:48:25 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SJvk01Nm8z3CW4 for ; Mon, 30 Oct 2023 13:48:24 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=AjKTfDy8; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=lAjLhCM2; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.27 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id BC9755C01A5 for ; Mon, 30 Oct 2023 09:48:23 -0400 (EDT) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Mon, 30 Oct 2023 09:48:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698673703; x=1698760103; bh=t/ C3TpCCd+XlXcI673X4/NK3AWuZ2u9jDKkOcDvehbA=; b=AjKTfDy8EY7S46WzzV ulXoNPUZUXt51kn5h3b4IbohO8ZSqhFLFq5XX1mOrlKSUWtr0Zkxw02GYGHErR12 WNcnzDFxyWXr/a3fWin/kXROhSfHPTYyb0zzMdHvjJQLgkScPGGf9+3E4lU6nUc/ z/l7ZzkRm2LzTo5ygTdugKe1mxlY7yay9T0eWNR/p6jVlSSTxTDafgGqtnZxjJsr /A3BauxOcSbcnZLtLP9dSuwko7ES3YF70vG0IYox0oXf23YVfkA2BzMCkfqrzxnq SjQxIzpthw9y4RFi3ISGvwQ4U/i0aY+RjnA+MYmOGbvjiz5mznz/rc4eCxZf74JT 81GA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698673703; x=1698760103; bh=t/C3TpCCd+XlX cI673X4/NK3AWuZ2u9jDKkOcDvehbA=; b=lAjLhCM2xz0vVE+Idt7mfcOU87gl+ yah8Y2Ju9gdBnVAcMYIaFnIAClHTY2cAm5pAdA/kef+E5RMv3m4r872HBMuf0sAT gRfVSpQcuNpoRpQ2yBrGioV3/bvnYv/1O0+F4bbP84jiYcA944wQM3ZeOT6vSEBm xqRDl1YynGuli0w8BQj8TzkdoNpSDIMu4qK3/T7WLg3dWF2hhol22xHk+Z8aUiNq 2PDfcSY8yTCTgTwKrSY2Nb9Sv2Dj9vAQSQ6t8Kc/z1DymcMrkwopGzoWzJTCi2TJ dsv9F6hIahOibPw8/9rA6iwPQgf+8mZMRp8MYV9c9s1NcFk3feoVrNwQA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddttddgheegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeitedvueehtdehtddvhfeuhfevhedvieelvdeiffehveelheegfedule ejudekvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm pehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 825942A20085; Mon, 30 Oct 2023 09:48:23 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1048-g9229b632c5-fm-20231019.001-g9229b632 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> In-Reply-To: References: Date: Mon, 30 Oct 2023 13:47:26 +0000 From: void To: freebsd-fs Subject: Re: optimising nfs and nfsd Content-Type: text/plain X-Spamd-Result: default: False [-5.03 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; NEURAL_HAM_MEDIUM(-0.95)[-0.948]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.27:from]; XM_UA_NO_VERSION(0.01)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4SJvk01Nm8z3CW4 X-Spamd-Bar: ----- Hi Rick, thanks for the info On Sun, 29 Oct 2023, at 20:28, Rick Macklem wrote: > In summary, if you are getting near wire speed and you > are comfortable with your security situation, then there > isn't much else to do. It seems to depend on the nature of the workload. Sometimes wire speed, sometimes half that. And then: 1. some clients - many reads of small files, hardly any writes 2. others - many reads, loads of writes 3. same as {1,2} above, huge files 4. how many clients access at once 5. how many clients of [1] and [2] types access at the same time looking for an all-in-one synthetic tester if there's such a thing. Large single client transfers client to server are wire speed. Not tested much else, (not sure how), except with dd but that's not really a real-world workload. I'll try the things you suggested. what I can report now, on the server, so before nfs is considered: dd if=/dev/urandom of=test-128k.bin bs=128k count=64000 status=progress 8346009600 bytes (8346 MB, 7959 MiB) transferred 59.001s, 141 MB/s dd if=test-128k.bin of=/dev/null bs=128k status=progress 6550061056 bytes (6550 MB, 6247 MiB) transferred 3.007s, 2178 MB/s dd if=/dev/urandom of=test-4k.bin bs=4k count=2048000 status=progress 8301215744 bytes (8301 MB, 7917 MiB) transferred 78.063s, 106 MB/s dd if=test-4k.bin of=/dev/null bs=4k status=progress 7725998080 bytes (7726 MB, 7368 MiB) transferred 10.002s, 772 MB/s dd if=/dev/urandom of=test-512b.bin bs=512 count=16384000 status=progress 8382560256 bytes (8383 MB, 7994 MiB) transferred 208.019s, 40 MB/s dd if=test-512b.bin of=/dev/null bs=512 status=progress 8304610304 bytes (8305 MB, 7920 MiB) transferred 63.062s, 132 MB/s From nobody Tue Oct 31 00:31:50 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKB0h5lVvz4ycQP for ; Tue, 31 Oct 2023 00:32:04 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKB0h3mpkz4Yvh for ; Tue, 31 Oct 2023 00:32:04 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5aaebfac4b0so3253957a12.2 for ; Mon, 30 Oct 2023 17:32:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698712321; x=1699317121; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=R9Zl0bNeLKTdJ1Ioh34rrkQfQ4hsZxFC7Qr7hsZE+qI=; b=Coi8se2Sixg5u+On+8eRTZq7nzQ0t9SCi9rtinXaqoqVyeavWOlxsVOIDCP2In3CBc i/kDnidMlkphSJmOkyNK2oOQtCSabNicTyKqiP6oKy+JA2Ezl+wR5c7uM9vU915bwfat XPnYH4Mm1tc1T4RYLAHVDwlWs3AwnpVG/Dez/amMyq2onaCtwavS6sLSZPCtdTR1r+41 z4IOGk42jVCPC6qibOrrXsE+nVgu3o7mwvO+D1ymz82rxCMko0e+lctT4g3s/mxsjGSp XQs/6BcUys6P14P1JbNpW/GlI4Ywr45juMFXuJTCvLsUtKidMP/8cgtdqELT1xOw5z5g hfSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698712321; x=1699317121; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R9Zl0bNeLKTdJ1Ioh34rrkQfQ4hsZxFC7Qr7hsZE+qI=; b=ZXEUFmo23DEonUq2Z/wpjLnDPr1nF9q4mB+nqvU3dDP0gG5g9zPAn6yjuP8trLOac8 VBrHOH7dUZV6LlRlcZyBb27N1eoanKXdyD8DtSYJXsvkEIZBPkIauX6ZwHF59oBLSF4I WsYeRhfv/mzqAa8oQnsNm/KZDCsobP+N2ATuW8y3D1PkLmhlo3Fz4Fxx41MYYZ1IOXSj 2AZ+6rq7hCA+1OG7mDDx22xZopqG3ZiIOzIkxHDrf9g+qLLZjUWTENnbkp4GSRMU5sIv 652oQ1RD88QPZQD8O6ujwOBRI5xYYhRMdhqj2U7r/2CLLfjBrySacsvkUCILN3yLA9Bx wDoQ== X-Gm-Message-State: AOJu0Yxrc1tzRLxRrQ7rhGc2mQMi744K192MxfhlSndjFnBE9tWV2AHy nPHUvgNuL7EhjQVDnh8S8/pUfcrPN+gDvcQLatYWlMsZJg== X-Google-Smtp-Source: AGHT+IE4zuhoW83G3tPOkmpAx1tfTqat+JLE9i6UFeHgPeYgM6dsT7uI1FFjfrJ0MvI4fttqF/6jZjjvezUiL4R6p6E= X-Received: by 2002:a05:6a21:7905:b0:179:f7cc:c7e6 with SMTP id bg5-20020a056a21790500b00179f7ccc7e6mr9549351pzc.38.1698712321428; Mon, 30 Oct 2023 17:32:01 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> In-Reply-To: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> From: Rick Macklem Date: Mon, 30 Oct 2023 17:31:50 -0700 Message-ID: Subject: Re: optimising nfs and nfsd To: void Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4SKB0h3mpkz4Yvh On Mon, Oct 30, 2023 at 6:48=E2=80=AFAM void wrote: > > Hi Rick, thanks for the info > > On Sun, 29 Oct 2023, at 20:28, Rick Macklem wrote: > > > In summary, if you are getting near wire speed and you > > are comfortable with your security situation, then there > > isn't much else to do. > > It seems to depend on the nature of the workload. Sometimes > wire speed, sometimes half that. And then: > > 1. some clients - many reads of small files, hardly any writes > 2. others - many reads, loads of writes > 3. same as {1,2} above, huge files > 4. how many clients access at once > 5. how many clients of [1] and [2] types access at the same time Well, here's a couple more things to look at: - Number of nfsd threads. I prefer to set the min/max to the same value (which is what the "-n" option on nfsd does). Then, after the server has been running for a while in production load, I do: # ps axHl | fgrep nfsd and I look to see how many of the threads have a TIME of 0:00.00. (These are extra tthreads that are not needed.) If there is a moderate number of these, I consider it aok. If there are none of these, more could improve NFS performance. If there are lots of these, the number can be decreased, but they don't result in much overhead, so I err on the large # side. - If you have min set to less than max, the above trick doesn't work, but I'd say that if the command shows the max# of threads, it could be increased. This number can be configured via options on the nfsd command line. If you aren't running nfsd in a jail, you can also fiddle with them via the sysctls: vfs.nfsd.minthreads vfs.nfsd.maxthreads The caveat is that, if the NFS server is also doing other things, increasing the number of nfsd threads can result in nfsd "hogging" the system. --> You might be forced to reduce the number of threads to avoid this. I prefer to set min/max to the same value for a couple of reasons... - The above trick for determining if I have enough threads works. - NFS traffic is very bursty. I want the threads to be sitting there ready to handle a burst of RPC requests, instead of the server code spinning up threads after it sees the burst of requests. - Extra threads are not much overhead. An entry in the proc table plus a few Kbytes for a kernel stack. (Others will disagree with this, I suspect;-) NFSv4 server hash table sizes: Run "nfsstat -E -s" on the server after it has been up under production load for a while. Look at the section near the end called "Server:". The number under "Clients" should be roughly the number of client systems that have NFSv4 mounts against the server. The two tunables: vfs.nfsd.clienthashsize vfs.nfsd.sessionhashsize should be something like 10% of the number of Clients. Then add the numbers under "Opens", "Locks" and "Delegs": The two tunables: vfs.nfsd.fhhashsize vfs.nfsd.statehashsize should be something like 5-10% of that total. If the sizes are a lot less that the above, the nfsd will spend more CPU rattling down rather long lists of entries, searching for a match. The above four tunables must be set in /boot/loader.conf and the NFS server system rebooted for the change to take effect. Now, this one is in the "buyer beware" category... NFS clients can do writes one of two ways (there are actually others but they aren't worth discussing): A - Write/unstable, Write/unstable,...,Commit B - Write/file_sync, Write/file_sync,... After the Commit for (A) and after every Write for (B), the server is required to have all data/metadata changes committed to stable storage, so that a crash immediately after replying to the RPC will not result in data loss/corruption. The problem is that this can result in slow write performance for an NFS server. If you understand that data loss/corruption can occur after a server crash/reboot and can live with that, an NFS server can be configured to "cheat" and not commit the data/metadata to stable storage right away, improving performance. I'm no ZFS guy, but I think "sync=3Ddisabled" does this for ZFS. You can also set: vfs.nfsd.async=3D1 to make the NFS server reply that data has been File_sync'd so that the client never needs to do a Commit even when it specified Unstab= le. *** Do this at your peril. Back when I worked for a living, I did this on a NFS server that stored undergrad student home dirs. The server was slow but solid and undergrads could have survived some corruption if the server did crash/reboot (I don't recall that it ever did crash). Again, I'm not an NFS guy, but I think that setting up a ZIL on a dedicated fast storage device (or a mirrored pair of them) is the better/correct way = to deal with this. NIC performance: - Most NFS requests/replies are small (100-200byte) messages that end up in their own net packet. This implies that a 1Gbps NIC might handle 1000+ messages in each directi= on per second, concurrently. --> I strongly suspect that not all 1Gbps NICs/drivers can handle 1000+ s= ends and 1000+ receives in a seconds. If it cannot, that will impact NFS performance. A simple test that will load a NFS server for this is a "ls -lR" of a large subtree of small directories on the NFS mount. --> The fix is probably using a different NIC/driver. > > looking for an all-in-one synthetic tester if there's such a thing. None that I am aware of: SPEC had (does SPEC still exist?) a NFS server load benchmark, but it was not a freebie, so I have no access to it. (If I recall correctly, you/your company had to become a SPEC member, agree to the terms under which testing and publication of results could be done, etc and so forth.) rick > > Large single client transfers client to server are wire speed. > Not tested much else, (not sure how), except with dd but that's > not really a real-world workload. I'll try the things you suggested. > > what I can report now, on the server, so before nfs is considered: > > dd if=3D/dev/urandom of=3Dtest-128k.bin bs=3D128k count=3D64000 status=3D= progress > 8346009600 bytes (8346 MB, 7959 MiB) transferred 59.001s, 141 MB/s > > dd if=3Dtest-128k.bin of=3D/dev/null bs=3D128k status=3Dprogress > 6550061056 bytes (6550 MB, 6247 MiB) transferred 3.007s, 2178 MB/s > > dd if=3D/dev/urandom of=3Dtest-4k.bin bs=3D4k count=3D2048000 status=3Dpr= ogress > 8301215744 bytes (8301 MB, 7917 MiB) transferred 78.063s, 106 MB/s > > dd if=3Dtest-4k.bin of=3D/dev/null bs=3D4k status=3Dprogress > 7725998080 bytes (7726 MB, 7368 MiB) transferred 10.002s, 772 MB/s > > dd if=3D/dev/urandom of=3Dtest-512b.bin bs=3D512 count=3D16384000 status= =3Dprogress > 8382560256 bytes (8383 MB, 7994 MiB) transferred 208.019s, 40 MB/s > > dd if=3Dtest-512b.bin of=3D/dev/null bs=3D512 status=3Dprogress > 8304610304 bytes (8305 MB, 7920 MiB) transferred 63.062s, 132 MB/s > From nobody Tue Oct 31 00:53:26 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKBTT1NgZz4ydhh for ; Tue, 31 Oct 2023 00:53:33 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKBTS6rB8z4fVh for ; Tue, 31 Oct 2023 00:53:32 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.17.1/8.17.1) with ESMTP id 39V0rQ5N007060; Tue, 31 Oct 2023 02:53:29 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 39V0rQ5N007060 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 39V0rQPC007059; Tue, 31 Oct 2023 02:53:26 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Tue, 31 Oct 2023 02:53:26 +0200 From: Konstantin Belousov To: YU SHANG Cc: "freebsd-fs@freebsd.org" Subject: Re: about the vnode free list Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4SKBTS6rB8z4fVh On Mon, Oct 30, 2023 at 08:41:05AM +0000, YU SHANG wrote: > Hi guys, > I'm reading the man page of getnewvnode which says that the vnode is either freshly allocated or taken from the free list. What I can't understand is where is the free list,is it the vnode_list? Many thanks. > There is indeed no free list. For the long time, really unused vnode is freed back to UMA. The global vnode list contains vnodes that have the identity, this way the filesystem' data cache is implemented. When system needs a spare vnode and the total vnode limit is reached, vnlru frees some cached vnode that seems to be unused. Read the code starting from vn_alloc(), there were recently some rototiling of the code. From nobody Tue Oct 31 10:16:01 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKQyl3SlGz4yBQL for ; Tue, 31 Oct 2023 10:16:15 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKQyk3QXQz4QXd for ; Tue, 31 Oct 2023 10:16:14 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="R/VDR5cK"; spf=pass (mx1.freebsd.org: domain of alexleidingerde@gmail.com designates 2a00:1450:4864:20::536 as permitted sender) smtp.mailfrom=alexleidingerde@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-5435336ab0bso2296357a12.1 for ; Tue, 31 Oct 2023 03:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698747372; x=1699352172; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=6r/P8/66g9tOFKPyW2qA8ORmY+m4l+XveS+CjgVTN6U=; b=R/VDR5cKip8yCKVZhZe/lHEGTc38W1wJunGsiss2IIAtCTDr9Ay0Gk8pbVHciMrhbr PvfDEB0zeB+hccAts6iTwv9uvUneKF9MnW9Z7Iw3RrQK5RfM8UjKZuOdYkSo74dnn2cu 6kwbe0XgoAs8dKb5Csj2lH/hsa+JzEtHjjKA6IEEVMC+WYqCaEwoK9Fa3oqHj7K/msia i/XebLSXyMLsT5haPXUUe8nPOLKD9tD0b6eQrebEboYpAukOLYJlDyuntSPQfIlEu9Wx maRsITNwmrHtvndBVfsXB1WHsqG3rz/nIONWnwhsjGj0n2Yb2Cj46wYY9hw0MxLOa60r iD+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698747372; x=1699352172; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6r/P8/66g9tOFKPyW2qA8ORmY+m4l+XveS+CjgVTN6U=; b=VfJXea5rREX+Qbv2C2ld/oGkVrEWbVOiMe3UuQc4Zox6SBtBofRB/UQYiPRBOGBGgG XDItKomtc4A4Tj5Igj9f9RN8+niHX4tnePi3JTRtQtAQsO+OcWha9RcXnTqehm9QgAoU AVV4t9yI13ZNw2B+ZKp5cSSzHvmcbyy1wNdcW1WnSWQ5ejufv8b1dBTjBJa9Rc7NlOjY fjSe7EjrJndOPinTE+rlTuQJ8aR1vdQFfM7u7Xd753uuqc+GMtTvLwnjitowSuPVklT+ hxWFZclJGYtH3j96u/nJi0E7rd0U9IYjc3bWWQATkcrNSNn3rlNZgoneYKNfaMUv2eHd u78Q== X-Gm-Message-State: AOJu0YzCBBNFIkytDQHJQTbsbG+Rj4zXjlrFXBXXwQ0SIMOjEkBuT7pP TZTZnMss4pWzkuvMM75zd0tecX0wBBaksc6Vylw+oI6JI6MkppoJXWA= X-Google-Smtp-Source: AGHT+IGTnAtwOeXpSvKfhVNvAOW4i1jBY94CMyZlPru0hjzAMk45F5SS0xw5IcKp3Crlx4z2rFiyxidMO4yXEaXV38U= X-Received: by 2002:aa7:c2cb:0:b0:53d:fe98:fd48 with SMTP id m11-20020aa7c2cb000000b0053dfe98fd48mr9753704edp.3.1698747372204; Tue, 31 Oct 2023 03:16:12 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 From: Alexander Leidinger Date: Tue, 31 Oct 2023 11:16:01 +0100 Message-ID: Subject: ZFS txg rollback: expected timeframe? To: freebsd-fs@freebsd.org Content-Type: multipart/alternative; boundary="000000000000af01d806090071d2" X-Spamd-Result: default: False [-1.11 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-0.95)[-0.954]; NEURAL_HAM_MEDIUM(-0.90)[-0.903]; NEURAL_SPAM_SHORT(0.75)[0.751]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_ONE(0.00)[1]; BLOCKLISTDE_FAIL(0.00)[2a00:1450:4864:20::536:server fail]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::536:from]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SKQyk3QXQz4QXd X-Spamd-Bar: - --000000000000af01d806090071d2 Content-Type: text/plain; charset="UTF-8" Hi, yes, the answer to $Subject is hard. I know. Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 setup) in a way that a normal import of the pool panics the machine with "VERIFY3(l->blk_birth == r->blk_birth) failed (101867360 == 101867222)". There are backups, but a zpoool import with "-N -F -T xxx" should work too and remove the need to restore from full backup (via USB) + incremental (from S3/tarsnap). During the crash there was a poudriere run of maybe 15 ports active (including qt6-), ccache is in use for this. The rest (in amounts of data) is just little stuff. What is the expectation of the runtime on 5k rpm spinning rust (WD red)? So far all the disks are at 100% (gstat) since about 26h. On a related note: Is there a reason why "-T" is not documented? After a while I get "Panic String: deadlres_td_sleep_q: possible deadlock detected for 0xfffffe023ae831e0 (l2arc_feed_thread), blocked for 1802817 ticks" during such an import and I had to set debug.deadlkres.blktime_threshold: 1215752191 debug.deadlkres.slptime_threshold: 1215752191 Setting vfs.zfs.deadman.enabled=0 didn't help (it's still set). Is there something more wrong with my pool than expected, or is this some kind of a bug that such an import is triggering this panic? The SSDs with l2arc and ZIL don't show up at all in the gstat, and I don't expect them to show up on an import with rollback to a previous txg, as such I was surprised to see such a panic. Bye, Alexander. --000000000000af01d806090071d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

yes, the answer to $Subj= ect is hard. I know.

Issue: a overheating CPU may = have corrupted a zpool (4 * 4TB in raidz2 setup) in a way that a normal imp= ort of the pool panics the machine with "VERIFY3(l->blk_birth =3D= =3D r->blk_birth) failed (101867360 =3D=3D 101867222)".
<= br>
There are backups, but a zpoool import with "-N -F -T xx= x" should work too and remove the need to restore from full backup (vi= a=C2=A0 USB) + incremental (from S3/tarsnap).

Duri= ng the crash there was a poudriere run of maybe 15 ports active (including = qt6-<web-something>), ccache is in use for this. The rest (in amounts= of data) is just little stuff.

What is the expect= ation of the runtime on 5k rpm spinning rust (WD red)? So far all the disks= are at 100% (gstat) since about 26h.

On a related= note:
Is there a reason why "-T" is not documented?
After a while I get "Panic String: deadlres_td_sleep_q: possib= le deadlock detected for 0xfffffe023ae831e0 (l2arc_feed_thread), blocked fo= r 1802817 ticks" during such an import and I had to set
debug.deadlkres.blktime_threshold: 1215752191
deb= ug.deadlkres.slptime_threshold: 1215752191
Setting vfs.zfs.de= adman.enabled=3D0 didn't help (it's still set).
Is there = something more wrong with my pool than expected, or is this some kind of a = bug that such an import is triggering this panic?
The SSDs with l= 2arc and ZIL don't show up at all in the gstat, and I don't expect = them to show up on an import with rollback to a previous txg, as such I was= surprised to see such a panic.

Bye,
Alexander.
--000000000000af01d806090071d2-- From nobody Tue Oct 31 12:10:07 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKTVQ5QFvz4yJ5f for ; Tue, 31 Oct 2023 12:10:22 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKTVQ1CjMz4YMd for ; Tue, 31 Oct 2023 12:10:22 +0000 (UTC) (envelope-from rincebrain@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-40850b244beso43225985e9.2 for ; Tue, 31 Oct 2023 05:10:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698754219; x=1699359019; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jKjABxk1a5Ld7Acm0UzbHdcbmdwNdnMPDXUVpZ50RHo=; b=MTUKRdc+Bvd4/lXOJWH6OXTPqKGrniFeabjfr77kqonzaKwLTMY+AkS3YOiZmeU4kg Ou9XcQMJh3r9BiEyhetFdhus9wf62Iwr+Ldchm8krGMgvdebJoxv5ZlI3x5B9WIuO1H6 ZZKWtpaWcRt+y3DXOyYNeM2lndvw2BNkZYXI4MfkuSQB9hLpjbBANdZgzeng+tvUt3f/ 8x3lrTLBg692oDet7l7O3vXqtLVyOxHggkrjk4qo7qzBg8Y+1bIDTHwDrxNtCFgdg9pw 3I5K3LQ+OX8Fit5HljKjqeoGXXy6k2u9bMlGHLQmBqnn7atgSvIu/L6a8mjjAR3/+jBX 7l6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698754219; x=1699359019; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jKjABxk1a5Ld7Acm0UzbHdcbmdwNdnMPDXUVpZ50RHo=; b=gjnVrN4gKElgvr11Gwho8PSYTXy4nqS52DjxKGl8Lf0msC4cUdEtJhChtEtx1H9EZy jGJASqAY02RA5wGn+79Wu1UnvGUnZEEBrDCkCrmyCGnvtJ+vf6n+vA+QK/4ixsxjR0Kf 2pIZIu8t+SDlcOkoEfj1S3NTBqZOvMokwuDlmSuwhEi7UtTDXCjsMuLTlgTU4hkYoo2x nJgRGy9rRjfip0oTF3EjItIaU89Cq4GJU5gqIZMPjLb2i+n80t3HVxBcxOhoH63Rv55w w85FTpmkbytqMs0pUBO4Hjqwu/tQBHYyJ8ZYs4pDkAu7yq39uOvZ3M52BlM3uC0gFV86 9r2w== X-Gm-Message-State: AOJu0YyiEf+cdypCYGWw+N9kg+U9PinUMqpddpQgT3f+ON1cTWP9EFuY TyWsoIp+jmMW+z7YsPHHM/DcgISW+D2cbkODlJM= X-Google-Smtp-Source: AGHT+IHLehR0CQKZ42Kcd0m/+L7HRqBjvvFHX48n2ESXs3sS7D0xaFXPBvi+oD49wyyyeHZVwyLVW4YHbRU6Q3pO9xk= X-Received: by 2002:a05:600c:1988:b0:409:19a0:d26f with SMTP id t8-20020a05600c198800b0040919a0d26fmr9913082wmq.23.1698754219119; Tue, 31 Oct 2023 05:10:19 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rich Date: Tue, 31 Oct 2023 08:10:07 -0400 Message-ID: Subject: Re: ZFS txg rollback: expected timeframe? To: Alexander Leidinger Cc: freebsd-fs@freebsd.org Content-Type: multipart/alternative; boundary="000000000000ca9a340609020944" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SKTVQ1CjMz4YMd --000000000000ca9a340609020944 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable -T is documented. It's going to take a while by default because it tries to walk every block in the pool and verify it passes checksum before the import succeeds. There's tunables for changing that if you don't mind finding they're sad later - spa_load_verify_metadata and spa_load_verify_data. On Tue, Oct 31, 2023 at 7:36=E2=80=AFAM Alexander Leidinger < alexleidingerde@gmail.com> wrote: > Hi, > > yes, the answer to $Subject is hard. I know. > > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 > setup) in a way that a normal import of the pool panics the machine with > "VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867360 =3D=3D 10186= 7222)". > > There are backups, but a zpoool import with "-N -F -T xxx" should work to= o > and remove the need to restore from full backup (via USB) + incremental > (from S3/tarsnap). > > During the crash there was a poudriere run of maybe 15 ports active > (including qt6-), ccache is in use for this. The rest (in > amounts of data) is just little stuff. > > What is the expectation of the runtime on 5k rpm spinning rust (WD red)? > So far all the disks are at 100% (gstat) since about 26h. > > On a related note: > Is there a reason why "-T" is not documented? > After a while I get "Panic String: deadlres_td_sleep_q: possible deadlock > detected for 0xfffffe023ae831e0 (l2arc_feed_thread), blocked for 1802817 > ticks" during such an import and I had to set > debug.deadlkres.blktime_threshold: 1215752191 > debug.deadlkres.slptime_threshold: 1215752191 > Setting vfs.zfs.deadman.enabled=3D0 didn't help (it's still set). > Is there something more wrong with my pool than expected, or is this some > kind of a bug that such an import is triggering this panic? > The SSDs with l2arc and ZIL don't show up at all in the gstat, and I don'= t > expect them to show up on an import with rollback to a previous txg, as > such I was surprised to see such a panic. > > Bye, > Alexander. > --000000000000ca9a340609020944 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
-T is documented.

It's going to tak= e a while by default because it tries to walk every block in the pool and v= erify it passes checksum before the import succeeds. There's tunables f= or changing that if you don't mind finding they're sad later -=C2= =A0spa_load_verify_metadata and=C2=A0spa_load_verify_data.

<= div class=3D"gmail_quote">
On Tue, Oct= 31, 2023 at 7:36=E2=80=AFAM Alexander Leidinger <alexleidingerde@gmail.com> wrote:
Hi,

yes, the answer to $Subject is hard. I know.
<= div>
Issue: a overheating CPU may have corrupted a zpool (4 *= 4TB in raidz2 setup) in a way that a normal import of the pool panics the = machine with "VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (= 101867360 =3D=3D 101867222)".

There are backu= ps, but a zpoool import with "-N -F -T xxx" should work too and r= emove the need to restore from full backup (via=C2=A0 USB) + incremental (f= rom S3/tarsnap).

During the crash there was a poud= riere run of maybe 15 ports active (including qt6-<web-something>), c= cache is in use for this. The rest (in amounts of data) is just little stuf= f.

What is the expectation of the runtime on 5k rp= m spinning rust (WD red)? So far all the disks are at 100% (gstat) since ab= out 26h.

On a related note:
Is there a r= eason why "-T" is not documented?
After a while I get &= quot;Panic String: deadlres_td_sleep_q: possible deadlock detected for 0xff= fffe023ae831e0 (l2arc_feed_thread), blocked for 1802817 ticks" during = such an import and I had to set
debug.= deadlkres.blktime_threshold: 1215752191
debug.deadlkres.slptime_threshol= d: 1215752191
Setting vfs.zfs.deadman.enabled=3D0 didn't = help (it's still set).
Is there something more wrong with my = pool than expected, or is this some kind of a bug that such an import is tr= iggering this panic?
The SSDs with l2arc and ZIL don't show u= p at all in the gstat, and I don't expect them to show up on an import = with rollback to a previous txg, as such I was surprised to see such a pani= c.

Bye,
Alexander.
--000000000000ca9a340609020944-- From nobody Tue Oct 31 12:15:16 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKTcJ2ngtz4yJs3 for ; Tue, 31 Oct 2023 12:15:28 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-3.mit.edu (outgoing-exchange-3.mit.edu [18.9.28.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKTcJ0VBKz4b2X for ; Tue, 31 Oct 2023 12:15:28 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none Received: from oc11exedge2.exchange.mit.edu (OC11EXEDGE2.EXCHANGE.MIT.EDU [18.9.3.18]) by outgoing-exchange-3.mit.edu (8.14.7/8.12.4) with ESMTP id 39VCFOFO018173; Tue, 31 Oct 2023 08:15:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1698754527; bh=h9etjjCdAd3+DaiA36izs9gS4oQqMQmAFogV1XuzD8I=; h=From:Subject:Date:Message-ID:Content-Type:MIME-Version; b=KHshPT/HU3JeFmnvXDqzH0ZLDjM6aI1NhRnD6yGUKg1Sf82cfMaXCvyRsjcU9pwGn YF9+5m1kegx3Dz05CK+3MGAaiT7xoZsnrNvp/ZZ17rw3G2l6kE5oBp4/4YEP+AxTev jRm2gmdo41RCI14vOueLn1BSlxZpHr1m5bsTVCwUbxpsZZOSvOtJ+3kioD7goZzVk5 czgzbOCm078Va32sFccUPX2wlixQ6X5EMPbBwPIZV85l/dFz3Q/dI+ooclmOC2VFnP jKysOOZOLCFDl2hCgEahD3peMLQWE3dMaAj3iF5E2G1Q+cQlrv/V0wPdnl++1gpist mStESXO2oi6Uw== Received: from w92expo25.exchange.mit.edu (18.7.74.31) by oc11exedge2.exchange.mit.edu (18.9.3.18) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 08:14:52 -0400 Received: from oc11exhyb1.exchange.mit.edu (18.9.1.60) by w92expo25.exchange.mit.edu (18.7.74.31) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Tue, 31 Oct 2023 08:15:24 -0400 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (104.47.73.40) by oc11exhyb1.exchange.mit.edu (18.9.1.60) with Microsoft SMTP Server (TLS) id 15.0.1497.48 via Frontend Transport; Tue, 31 Oct 2023 08:15:23 -0400 Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by BN0PR01MB7167.prod.exchangelabs.com (2603:10b6:408:156::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Tue, 31 Oct 2023 12:15:17 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1%4]) with mapi id 15.20.6907.032; Tue, 31 Oct 2023 12:15:16 +0000 From: "John F Carr" To: Alexander Leidinger CC: "freebsd-fs@freebsd.org" Subject: Re: ZFS txg rollback: expected timeframe? Thread-Topic: ZFS txg rollback: expected timeframe? Thread-Index: AQHaC+NTekXE6fESM0qbEp/e00mxprBjz/0A Date: Tue, 31 Oct 2023 12:15:16 +0000 Message-ID: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|BN0PR01MB7167:EE_ x-ms-office365-filtering-correlation-id: fecdecaa-44be-418b-c602-08dbda0b0b1b x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: bTYrR+ljlgTdDFmcUDBm0mXpjC47RLYF1AxQpdLu6idOfHKJ7VJ1YxSrQ8bVz9okuxrsswZ2XbvWYh6VKn+XoHzrWqVnTUJWmwxTNjU//OXI4NcvWmxnSOTSQVuendAeQEZC6NDaP2+43WdSZVLp/mC8xPfz7lqL6O7Mw2VSZ5jFKi5SGBxzja2X6dO84vx6NOkH9sBZorzmkstalaBQyYHd34ofZ0MRJ3jQBj+ptye2WXtb5VrPoOKXz4MO64J/Gywj4FwJHCzECUh46VYAyGiyOaMQPg8WAjo21axWYpQRujR0sRT18eKp0/E1rYauaC6vPZb4mflGnmvm0bCaFDAV0lyhjPW9IEDH+kr+s813EwvkDqriI/dof/uKgXvolK1WJLc661iThBkE3oLE88YDcv2/vcjkFooXllN8AwN8wEaPrHRG4jw2jZ5zKPDD0undNDnAHVUfoJL29hSzabKO6LAoKbfOVDNZIQDbAoKaQCHhPLNEpw7gcx+7vrWHj1JA1mNw4OINddHblRA42zE/6BMSwlVonZp6c7CTBV5h1BNj8IfCrL0LXUeDczw4WcNBqMq84BhQSk3E3KVrjRAAluF+wcQ0KK7SerULfTE= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(136003)(376002)(366004)(396003)(346002)(230922051799003)(1800799009)(451199024)(64100799003)(186009)(75432002)(478600001)(6916009)(786003)(91956017)(86362001)(6506007)(316002)(66946007)(66476007)(64756008)(66556008)(66446008)(38100700002)(76116006)(38070700009)(33656002)(2616005)(6512007)(966005)(4326008)(99936003)(2906002)(122000001)(36756003)(71200400001)(8936002)(8676002)(83380400001)(4744005)(5660300002)(41300700001)(6486002)(53546011);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?7VuIMKp4Ha8d8DkRkMS+AfRZoRxdv9YLVZ21rvwnohSkgU4t5TYyktqWGdCR?= =?us-ascii?Q?6qa5lA5S6B5K2s1Ma+0LgLOSne6GO1sL5tM3/D6Wm5/7H55o7opmmmPdizHN?= =?us-ascii?Q?ZISxKfWifPpbYFZuBnygsG/Z3YduxRP3OHpA9HsFqAAkxWDeysty7HG9I1z1?= =?us-ascii?Q?92CElfpdbBE0aLYF42cVSTsXCBhqYkwTJThfWzT9bN9vjX1jHu/I5LbKMe/E?= =?us-ascii?Q?l6dqZRcKkA/pK4r7TOxHyu6ngFChNPDpM6kYboiFUV/xPHHq02LD3SgjM6kT?= =?us-ascii?Q?pb2TwcOeAFD6TQVycDnDhL1CLQOrkq4RQCXGQzv8pvm5US8qr6/+j9z/6ctp?= =?us-ascii?Q?zQNvKKnSfyjQ17qrnPqPhO1IEIMmhPotWb2YeEphWJrc0nF9phe3eBQ6nzVG?= =?us-ascii?Q?2ZvYRbg1sTJQF8FliJ8/39AMGXaHUWEpvaFSZRRrHdjzTFkTX3ClMx6DsWzG?= =?us-ascii?Q?u8X3ruenU/mR9XR+KefBe/CGGleo84gu0+ILAUc/3OHknh7KSQjYNTVnfuKy?= =?us-ascii?Q?+bEj5pPE+vLbLKo5EElrCWuwDYUzsfN6PDhWtDliYRQD358GtWNl7BhGOwFo?= =?us-ascii?Q?01s4+LWmJJPwyAQw7Y5vcYweCkzlztUCWVfCChEy0FcmnX3EExA79DU3nW0s?= =?us-ascii?Q?ao1wEl+b0JW7v3iR2lGX0aoFfUaHLp+S/Hc3p+FkcDmHNhSysTynrKc/i50h?= =?us-ascii?Q?+UGWmP1m1gJGJtvJV/i9hQlY6LptE258KYH5uhguyC/p7Jx0FHEYONNz1eev?= =?us-ascii?Q?UjCbViZ6Sr/3lkdK55bs75z8pBvCZwAc0lxFfG/7P40kyKhGvU8BRdOAR0hk?= =?us-ascii?Q?5lS4zdu+kJisE5VaTcb6ItTDbGQnbgqf8da5Q2BBBEXHloTxu1UUS/eh4Ztl?= =?us-ascii?Q?NZXC5I3OQ/wRx0Y4r2GufSLylLiE+IKqK8e3TG109ysQCpLcoTpGeWrVEVn5?= =?us-ascii?Q?abFpFmBkEBxfTgUBcOyDCO7Akj43iOsHx0ANXG29htRxbj5y4FTMbFQd41cr?= =?us-ascii?Q?MwQI99qmIyH9WBSHxooaPloaXXDzTaN3cAc17JEa8+CNO3wuX1btmUALhmxT?= =?us-ascii?Q?5j8hmJZuo/xZmBv5LC3id9JNgCyvbidugEwak10yjOnfiscHaX51Eh7fMTVb?= =?us-ascii?Q?fs85trFwKCN9jKOd40ffKpv5kLDSQ+v+DUWbxXR+RHYxEAhidk6a+8RNLsxm?= =?us-ascii?Q?kpbJHJdy1itIQR9hmWrgNs0Iz8KjkTgVNJtrqCaDizc0kGgc1In4Tnc4ul/D?= =?us-ascii?Q?nKtLVnk70mXrKMons8ZxuM6txhxn6y6KzH00UOCPK9H3mOdg1CzC5ivsnNDA?= =?us-ascii?Q?ArBH44Q56B70l87ShJyE16SpNfxcS+dKTQiIYH/+hd7AaiWACUKPnMdFaBbz?= =?us-ascii?Q?ILIxjqLsaEI75EHyLdf9TjiAOBm1UXEBOk5Xnc+c6xEzUUoM/t0hmiT/eHl9?= =?us-ascii?Q?LaaLp+pTLDmbASHRDPr5Y5kH1m53Njioml8X9kAT/8O6jOk4ceGVvub4ElIS?= =?us-ascii?Q?kdA7YzOlms5NT5QM5T/D1gOADuiw40cgRXQGraZp3Addei+7UtAN1zaklfa4?= =?us-ascii?Q?0mG0erI7SJPcLyePuNzKnl7xP+hB+l05H0Ys8O/uoi0vFWTWku0ZDae/D4cF?= =?us-ascii?Q?qOlWZpaT4sEwspZuuFFpXgjU/y0l+2Q5Adq8LfllxObS?= Content-Type: multipart/mixed; boundary="_002_18B0B6B6923742D09FB2D55CE72E1CCAmitedu_" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: fecdecaa-44be-418b-c602-08dbda0b0b1b X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2023 12:15:16.8338 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7n/JX0LnIzatbwfH1e0jQ1JZ09X42r6cEDrvr89wt1JwxluH6cNeNmHWoQNf/Ejt X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR01MB7167 X-OriginatorOrg: mit.edu X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US] X-Rspamd-Queue-Id: 4SKTcJ0VBKz4b2X --_002_18B0B6B6923742D09FB2D55CE72E1CCAmitedu_ Content-Type: text/plain; charset="us-ascii" Content-ID: <28AF03ED5B34C14CB1D9ECBDBE02833C@prod.exchangelabs.com> Content-Transfer-Encoding: quoted-printable > On Oct 31, 2023, at 06:16, Alexander Leidinger wrote: >=20 > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 se= tup) in a way that a normal import of the pool panics the machine with "VER= IFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867360 =3D=3D 101867222)"= . >=20 I disabled that assertion because it gives a false alarm with some combinai= on of deduplication, cloning, and snapshotting on one of my systems. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 https://github.com/openzfs/zfs/issues/11480 --_002_18B0B6B6923742D09FB2D55CE72E1CCAmitedu_ Content-Type: application/octet-stream; name="z.diff" Content-Description: z.diff Content-Disposition: attachment; filename="z.diff"; size=891; creation-date="Tue, 31 Oct 2023 12:15:16 GMT"; modification-date="Tue, 31 Oct 2023 12:15:16 GMT" Content-ID: <0E283381E9EA0543907D8E0E9B071DFB@prod.exchangelabs.com> Content-Transfer-Encoding: base64 Y29tbWl0IDNhOTlkNjE0NzY2ZGU4NWQxMzc4OWFiYjJlMTkxNzVjNDY2NmQyNjANCkF1dGhvcjog Sm9obiBGLiBDYXJyIDxqZmNAbWl0LmVkdT4NCkRhdGU6ICAgRnJpIFNlcCA4IDE0OjEzOjAwIDIw MjMgLTA0MDANCg0KICAgIERpc2FibGUgaW5jb3JyZWN0IFpGUyBhc3NlcnRpb24NCg0KZGlmZiAt LWdpdCBhL3N5cy9jb250cmliL29wZW56ZnMvbW9kdWxlL3pmcy9kc2xfZGVhZGxpc3QuYyBiL3N5 cy9jb250cmliL29wZW56ZnMvbW9kdWxlL3pmcy9kc2xfZGVhZGxpc3QuYw0KaW5kZXggOTgyN2Vi MTQ3MjhkLi43ZmVhMTJhYjUyYzUgMTAwNjQ0DQotLS0gYS9zeXMvY29udHJpYi9vcGVuemZzL21v ZHVsZS96ZnMvZHNsX2RlYWRsaXN0LmMNCisrKyBiL3N5cy9jb250cmliL29wZW56ZnMvbW9kdWxl L3pmcy9kc2xfZGVhZGxpc3QuYw0KQEAgLTk5OSw4ICs5OTksMTAgQEAgbGl2ZWxpc3RfY29tcGFy ZShjb25zdCB2b2lkICpsYXJnLCBjb25zdCB2b2lkICpyYXJnKQ0KIAkvKiBpZiB2ZGV2cyBhcmUg ZXF1YWwsIHNvcnQgYnkgb2Zmc2V0cy4gKi8NCiAJdWludDY0X3QgbF9kdmEwX29mZnNldCA9IERW QV9HRVRfT0ZGU0VUKCZsLT5ibGtfZHZhWzBdKTsNCiAJdWludDY0X3Qgcl9kdmEwX29mZnNldCA9 IERWQV9HRVRfT0ZGU0VUKCZyLT5ibGtfZHZhWzBdKTsNCisjaWYgMCAvKiBzZWUgYnVnIDI2MTUz OCBhbmQgaHR0cHM6Ly9naXRodWIuY29tL29wZW56ZnMvemZzL2lzc3Vlcy8xMTQ4MCAqLw0KIAlp ZiAobF9kdmEwX29mZnNldCA9PSByX2R2YTBfb2Zmc2V0KQ0KIAkJQVNTRVJUM1UobC0+YmxrX2Jp cnRoLCA9PSwgci0+YmxrX2JpcnRoKTsNCisjZW5kaWYNCiAJcmV0dXJuIChUUkVFX0NNUChsX2R2 YTBfb2Zmc2V0LCByX2R2YTBfb2Zmc2V0KSk7DQogfQ0KIA0K --_002_18B0B6B6923742D09FB2D55CE72E1CCAmitedu_-- From nobody Tue Oct 31 13:57:34 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKWtn0MLZz4yPD4 for ; Tue, 31 Oct 2023 13:58:09 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-1.mit.edu (outgoing-exchange-1.mit.edu [18.9.28.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKWtl6kMtz3GBP for ; Tue, 31 Oct 2023 13:58:07 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mit.edu header.s=outgoing header.b=UmpK9uKx; spf=pass (mx1.freebsd.org: domain of jfc@mit.edu designates 18.9.28.15 as permitted sender) smtp.mailfrom=jfc@mit.edu; dmarc=pass (policy=none) header.from=mit.edu Received: from w92exedge3.exchange.mit.edu (W92EXEDGE3.EXCHANGE.MIT.EDU [18.7.73.15]) by outgoing-exchange-1.mit.edu (8.14.7/8.12.4) with ESMTP id 39VDvXcA001542 for ; Tue, 31 Oct 2023 09:58:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1698760685; bh=mesJnXWKcdLpuZ4254Q9DN4Jh3bo+ssnb90wyO27oWo=; h=From:Subject:Date:Message-ID:Content-Type:MIME-Version; b=UmpK9uKxmJ1MZT0ZAahgvqjurn5EAbUgDDso71vkaBiG3NrLvdZ67/HT8ayUbqUpJ WwzqFh7Y9OhETcJr+Xf4+q46linRD41gOAPV/03psNWSOOg/4ZKwfu3mAWOP3EsFSW okiS5cJnejjWqONHdrlf1xa6aGl8S4VXewwuVR0z+QB9qBQWgaHRoCMcIFNxWfVDUs SlMGKorRgRHXRAkdy5y1sqvl6gDnjYkGHzaJSr62xVl1uBXXQkosVAGUKv4TLQSHx4 zPZGxBo7ogOvWPMCSEXXycGA6moNHT1cJXKX8tLhDsLyFhzd5cFEtnC2R4Ug/RXD6f c34/F1zV+nSzA== Received: from OC11EXPO27.exchange.mit.edu (18.9.4.98) by w92exedge3.exchange.mit.edu (18.7.73.15) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 09:57:34 -0400 Received: from oc11exhyb6.exchange.mit.edu (18.9.1.111) by OC11EXPO27.exchange.mit.edu (18.9.4.98) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Tue, 31 Oct 2023 09:57:37 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.168) by oc11exhyb6.exchange.mit.edu (18.9.1.111) with Microsoft SMTP Server (TLS) id 15.0.1497.48 via Frontend Transport; Tue, 31 Oct 2023 09:57:37 -0400 Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by CYYPR01MB8291.prod.exchangelabs.com (2603:10b6:930:c4::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Tue, 31 Oct 2023 13:57:35 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1%4]) with mapi id 15.20.6907.032; Tue, 31 Oct 2023 13:57:34 +0000 From: "John F Carr" To: Freebsd fs Subject: vnode_init took over 6 minutes Thread-Topic: vnode_init took over 6 minutes Thread-Index: AQHaDAIyORVK7QVkx0+UPxykgQSM3g== Date: Tue, 31 Oct 2023 13:57:34 +0000 Message-ID: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|CYYPR01MB8291:EE_ x-ms-office365-filtering-correlation-id: 12fb0a59-5da3-41f8-963b-08dbda19556c x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Q5+1gM9C1FActITEnJHvjZ++UBMXr7Rb5Jcp7PpH7hfUj8rfTqYLlUC9IWOPGFmGxb1M+Wm3IIjkTdkMOv5//odDxQ2aGR65+ixYw4USYaU3e7ccotEnrONmyl8Hda3RbDzbvcC+WoSoDVia5Gp162F3vJN3912Ib0L8lWdbG9HFBP+zfj2iDtE8NVd9Nw4xyZWAqK1EXTO4W1d8ovy8FvDtoy8fBoTOZ4ILPME3l8ImR5vAHUE+BFLrhhYk9PfPEAtFV1KNLSWXi01xL7X0N0jWvG4i+JOFdnXX4GPrbjd1ipOG9gJ4iQbZ5t2RCNJr7oIsZgJV2z7VAVn81u03X37XhbxZykFfHbLcYXjmnZq/SQRJURLg+c0dTgsH5Ix2Betu4rKLci7Vh8nMB2ZjSIfSfsYVAyiVlgYbpBwYLH1w2yicYsZNj616BNhzXW1jKcEO8CtKnqEO4vEYsmM6HO5syexgmuwLi7IjKioh0t9Ai6NKq4+AIhaGWjA2FensjI80xy2E7ZiHrfrBIiVPi7wTYwQfMb5D1GxTU7wv79I/Reg+1/MzaLIkqZ0jRsye1RbW5wTWu3nPjrNQ1mE8x5iRfaxO7gjrn6CTJrWXbd6jQgJcWCrAtlX7AUrzPcgg x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(376002)(346002)(39860400002)(396003)(366004)(230922051799003)(451199024)(186009)(1800799009)(64100799003)(75432002)(71200400001)(2906002)(478600001)(6486002)(2616005)(38070700009)(122000001)(38100700002)(33656002)(36756003)(83380400001)(86362001)(6512007)(6506007)(8936002)(8676002)(5660300002)(41300700001)(91956017)(6916009)(786003)(316002)(66946007)(76116006)(66556008)(66476007)(66446008)(64756008);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?u2EaNNvYLq9RyiDFIWVMX40IjHMrTGQTXOowl/1KVLVMIskHb7ylErO/cvyX?= =?us-ascii?Q?lj3XG18fNpkbf/NPSwJ0KAXEd9oXG8uyVl8cSQNaazkIwYtNwxk23U7OIXMX?= =?us-ascii?Q?NO0Xi6xcLgGBPI2wJwhYn/1ZVCe+dpqXI/IHOwyEENRx7dRKH44CUMzae2aH?= =?us-ascii?Q?p6q9H3i+6stbClvpELXXg7F0fyXykF7vEiSUlLJqyupUjGhjRNEAgxhsfHTx?= =?us-ascii?Q?4uP7qnjWW4cbpPrN8+3od12yQaAk0+XvPAcnOjx8DeUT+gOCjzX2hBJCUj09?= =?us-ascii?Q?+4G68w27sshSNv2ep1//PesJ2UcxDZmDC/IuLlTMHTz0OV4YyHNqO8mS/2lC?= =?us-ascii?Q?jYmy2MmrVYADa+jQ2usEP81l5Fy1wJSybIAPudz11vFqs+WJi6g5dGW/mZDm?= =?us-ascii?Q?9hQbOYDJtKPt7q3NcLfwYSSH8au8UUUOU1JZHj50DeLk1ojYuR6/lxqjh0bk?= =?us-ascii?Q?2YYeBslFEVjsevg0OnuR0w4gxFbKeaDuzRxQ21ZUft1GXPVQy3CWHwjzyhDd?= =?us-ascii?Q?YCoF6hgkVOOFNIRSsEIs7gz8okampWPXLORXRqNe644x3sl0X4fLd7ngu6b0?= =?us-ascii?Q?hRIrQTq84hLdedOm60u5noeqcpGA5Kp6eGkLjBWc0WDWzlhuDccNYV313bDI?= =?us-ascii?Q?lWJtQTEWS+d1IUSpcdAUaR08gxV+r9mYnDMcxxyht+/yHFfh96rkVWjdPPoj?= =?us-ascii?Q?Wvdi+90DJGPkJYLiy3X/Ecju02IDndEDKgmjsbWJXv/NI4CHbdZFtzZsnJFX?= =?us-ascii?Q?bCS8WrlWtRx66KEDf2+hSACgr9/q4X1D8cRj40CAZmrbi/qdMOdxUiNVTjzv?= =?us-ascii?Q?w/fipwsmRJQbjdRUx/kLo/CAne6Fg/CL3j1Adf+OdHEaa/Ty0H7H1lgyrCVD?= =?us-ascii?Q?BbrVXF4TfGzODXi3jJiDm4YPdmrSOjPt8R26nWoQZYNxBzkiGh0PdHfZxRQW?= =?us-ascii?Q?lJiC67CaUvuRFlqD4zs2M9u3aOvU7qbl3EV7mJmW9tXaM3r0TwVi/NeHLjN+?= =?us-ascii?Q?VToYsF8Q9u5PB4x1J1V5swRHby8eYGYN0IHoCRGzp6y+Q9QJJAmquftnKmqY?= =?us-ascii?Q?hvP3OvGzhW/OoD2kd3WBcheOt4u9ZUZdOhVVTMMX5Cs/VQWnYNUUjltKe1M1?= =?us-ascii?Q?blCxwkAaQoX5017gtS3iSh9Npko5oq65e8JY0SpAlTZOF7dIchPVB337V2xS?= =?us-ascii?Q?uoCBP811s6/CNbh7GE7SxIsezwwPnBWkkiLdsCJiSU7roeJKpjsABmNhkeZ5?= =?us-ascii?Q?5miIZsjS/nDYM6B4GZ+yMlmkRzF8oEqLkDf7yS5d+gHQ6MA1K8VfpnBZiAzz?= =?us-ascii?Q?ZTNV5J+ivJjYbgoXEDfctc/8OYxHcOp3e5Jjs6avcSGk0c3kAGcbM7/wr3Wx?= =?us-ascii?Q?tk5FuhYiafVn+5YFiu+8IU4HChRROqf5kMKfvlee6Lb82tUPaF4jWmS05nD9?= =?us-ascii?Q?pXfIoNR+P/YQmYW5za0SIC3iPRAp2qzVlAhlxEL3S8b1ie3DskKv4EjG0U90?= =?us-ascii?Q?MH/n4xNDTI0Gfsd8zjsOmhAYNfDuixyvDntPkgrHfRdDgMI4cdtD4SkQ2hmy?= =?us-ascii?Q?59qdsTLfja0ZPZfyv+Ffk+kQ6RqkAh6azn4oFgNvEKjGY0cp7HcKfyVsHzVZ?= =?us-ascii?Q?0vvq4zENwN8C4q5AraEF+chYnj3pofDlJNZSjj5yQteN?= Content-Type: text/plain; charset="us-ascii" Content-ID: <82B9792814CFB041B0C6A69E3E0110C4@prod.exchangelabs.com> Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 12fb0a59-5da3-41f8-963b-08dbda19556c X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2023 13:57:34.5020 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zgPgwk7/3S8JqUcpQ+pmdIm/+2+MXUIb0cTFLwWRrNez1e4xtM8KZgKbVYpLjpxu X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR01MB8291 X-OriginatorOrg: mit.edu X-Spamd-Result: default: False [-5.80 / 15.00]; DWL_DNSWL_LOW(-1.00)[mit.edu:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DMARC_POLICY_ALLOW(-0.50)[mit.edu,none]; R_SPF_ALLOW(-0.20)[+ip4:18.9.28.0/24]; RCVD_IN_DNSWL_MED(-0.20)[18.9.28.15:from]; R_DKIM_ALLOW(-0.20)[mit.edu:s=outgoing]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[18.7.73.15:received]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[mit.edu:+]; FROM_EQ_ENVFROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[18.9.1.111:received,104.47.56.168:received]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SKWtl6kMtz3GBP X-Spamd-Bar: ----- During a poudriere build control-T showed an awk process was stuck for over 6 minutes trying to acquire a lock in vnode_init. load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 keg_alloc_slab+= 0x277 zone_import+0x143 cache_alloc+0x3ed cache_alloc_retry+0x23 getnewvnod= e_reserve+0x20 zfs_zget+0x1f zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_l= ookup+0x3e0 zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 cache_fploo= kup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea vn_open_cred+0x48d=20 The stack trace was the same for several minutes. System CPU time was incr= easing. Address vnode_init+0xc3 corresponds to the mtx_lock call here. vp->v_holdcnt =3D VHOLD_NO_SMR; vp->v_type =3D VNON; mtx_lock(&vnode_list_mtx); TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); mtx_unlock(&vnode_list_mtx); return (0); Address __mtx_lock_sleep+0xf8 is the instruction after a call to lock_delay= . ps says the command line was /usr/bin/awk -f /usr/bin/awk old-default/2023-10-31_08h21m03s/.poudriere.bu= ilders old-default/2023-10-31_08h21m03s/.poudriere.buildname ... with the full list of input files exceeding the ~2KB command line length li= mit of ps. "/usr/bin/awk" is probably not the real second argument. It would cause an immediate syntax error. The hang resolved within a few more minutes and poudriere is continuing hap= pily.=20 I have never seen such behavior before. Code in vfs_subr.c tries not to ho= ld the vnode_list_mtx lock for too long. Any thoughts? FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread Zen 2= , zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB RAM. From nobody Tue Oct 31 14:07:15 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKX5Y21mYz4yQX8 for ; Tue, 31 Oct 2023 14:07:29 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKX5X4LnFz3MZb for ; Tue, 31 Oct 2023 14:07:28 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-53ed4688b9fso8881781a12.0 for ; Tue, 31 Oct 2023 07:07:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698761247; x=1699366047; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=cC7V4pM7csDEM69YoXiaQUEaguJdFhvTjBMbDQWVjVE=; b=YThVhNhzRykoxeGpEgGDV3lfdzJ55uDWtqFLJ/x9+bZcno2Ky9mXIqV8Vre+qkBzbb Z/NhEqGHA0QAZp9hc8sXeBy1Rvk1OdEihimdAPK8aNJMKrtsxaruJf2FshZ9zMhxcJCu eG+YtuXsOv2rjWWSKZBbx+VwqpxtNX6j3AskEx+g4KyY/DGo3e3sQt/tsxKM+KIUN4rf LorvM0Hwl4wZ7q7Ty1ABgU5XOIzhoyiukUSf09FyN2JjGxRkYdnPMVQLCeeeQPeMbcGp 0E88nGhbvu388LFE6lyyz7f9xLkAFWAOk1p5eE7LT03IAhbEHPvaVJLFpPxOhJ9++mgH uVFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698761247; x=1699366047; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cC7V4pM7csDEM69YoXiaQUEaguJdFhvTjBMbDQWVjVE=; b=TyrU9ofHgz23E0GU7Z2HeLXYu+Vc9TWvUOw6yQxxiktGwUxLy1qPnJl1Vuv0rYaQbE p9LirRnJLcHkdQqFg718Aj375OI4MyZFfMkk2mNA7bvHvjV3uHZbIAJXYW5akbwQ52HY faE+NnJMLgPP58sXHNe+o/NgWOg4PN43RlRuMUKUIbvLV6pZphchu7+gJu+ngBsZ6A2S hi0quscc9KO9wbXNqw/5pm/Ju1N0RPFVzx/1ryVkmdnOdGTAb8EunrFXt34lc5KRwN1C iyB7yK1kIofF+Gb91Qv+S0kruUI+/ksArpEaLciwaEZF9C5ziLMSdQH72BL2Af7KI/P5 1r0g== X-Gm-Message-State: AOJu0YxKH8jTShDJKgxZejbmtXGs846ILMbuBa4setiRAjz8PU7HFz0D t3tInjnbLPAzrsN6kTs45CrPWC93RCzDXcArJH8158ZtWfiQpXS1 X-Google-Smtp-Source: AGHT+IHf2v2qi9nRU9/X3rSLJFFKyx2HZbMOHQeGGzRFuHJWfEPY4hsd895HdgnNqitN04OnVdPnLt1L53SnyFnUr0o= X-Received: by 2002:aa7:d0ca:0:b0:530:74ed:fc8a with SMTP id u10-20020aa7d0ca000000b0053074edfc8amr10475436edo.41.1698761246724; Tue, 31 Oct 2023 07:07:26 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alexander Leidinger Date: Tue, 31 Oct 2023 15:07:15 +0100 Message-ID: Subject: Re: ZFS txg rollback: expected timeframe? To: Rich Cc: freebsd-fs@freebsd.org Content-Type: multipart/alternative; boundary="000000000000ab56d8060903ac4c" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SKX5X4LnFz3MZb --000000000000ab56d8060903ac4c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 31, 2023 at 1:10=E2=80=AFPM Rich wrote: > -T is documented. > Ok, let me rephrase: "s/documented/under documented/". For example: how to find the valid TXG number. > It's going to take a while by default because it tries to walk every bloc= k > in the pool and verify it passes checksum before the import succeeds. > There's tunables for changing that if you don't mind finding they're sad > later - spa_load_verify_metadata and spa_load_verify_data. > Interesting... I will think about that. Thanks, Alexander. --000000000000ab56d8060903ac4c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Oct 31, 2023 at 1:10=E2=80=AFPM R= ich <rincebrain@gmail.com>= ; wrote:
-T is documented.

Ok, let me rephrase: "s/documented/under documented/= ". For example: how to find the valid TXG number.
=C2=A0
I= t's going to take a while by default because it tries to walk every blo= ck in the pool and verify it passes checksum before the import succeeds. Th= ere's tunables for changing that if you don't mind finding they'= ;re sad later -=C2=A0spa_load_verify_metadata and=C2=A0spa_load_verify_data= .

Interesting... I will think a= bout that.

Thanks,
Alexander.
=
--000000000000ab56d8060903ac4c-- From nobody Tue Oct 31 14:16:00 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKXHQ3D2Cz4yQtj for ; Tue, 31 Oct 2023 14:16:02 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKXHQ0rGGz3Ngw for ; Tue, 31 Oct 2023 14:16:02 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3add37de892so2874898b6e.1 for ; Tue, 31 Oct 2023 07:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698761761; x=1699366561; darn=freebsd.org; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0ZfA8SyXKR32Bbnq/28ddJC/pC65dUPe0rsyyVHuIC4=; b=fgIVyVF8AEcVNMIL42AXTz4sxPxsNUxRHfKe7weFU7+NqYAqTNp1raDzP9Oz55BxBQ qJA0sJVBwztJ6EF2APnYV9gbOKQqqRyp0Qvok6LYipdqy6RZYx7sn+im+HvbhyFJWrAw HNEmVPJrAEcvJuxJTCtbOh3k1XXqJx71ntci5ptcdpRncxxNzfKEi4/ONFFRyruFK8bm IzYTgfefxnILiPplZ8fTgNq2HStIxUUpD9JDecojIPri8SDuo7k4DDqQOorZ75aiMGfc wdy2udrbF1fE1+avZH7/UPRpwEpLQ/QP2IBsOD3i++9MhMgVmwGBNszl0F8DcjKP4o0U hZIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698761761; x=1699366561; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0ZfA8SyXKR32Bbnq/28ddJC/pC65dUPe0rsyyVHuIC4=; b=rUZsC5K7EpB8Cs4v6FjpbMFD3mjPzIXJcrB1xjuGI+CajMYkR1asphpFVHMfU6UprO FpUDHXhkuGSihkyo4ibrTZJAxpfKIhOUoIkTRUqrm2JmTCxVUY4GLFahwPv8qpyqDlmH IUS6dqgeS9BKqsVNg365mJfCFb0PXZSsQUxYcQcoiQU7rT9eUs58BEPT+Na1bo5QicQH HBwOtrcihWKNdJHaFctuJzU4pwGmpd37igqgPAwJzw9gaMoFOSYkpQVIQHryIXzRu51v +Han4qbM4NR4XdxsGldVQ4QJfn4hfbIJ3APnPgooG8YZuYYgfjS3t3BsjOWM2zY71ACC gpGA== X-Gm-Message-State: AOJu0YwU/JAckt6NEFRkeLjJ8G3Dxz/tFQtpSDbvAILnFFUDqUxN6uge 8rL3skb1aXe826WSGdDPGEZE/XYDEmjx6c50scFC1g42 X-Google-Smtp-Source: AGHT+IHHXrvKRRa5cPzdNMJmfSQBpQmktEpIm7GRPKMCuOjE3KPpwt1vdsAWYiLBkX1ufod1t46ivXxPy1b1ca+LvAg= X-Received: by 2002:aca:1816:0:b0:3a6:febf:fb with SMTP id h22-20020aca1816000000b003a6febf00fbmr10615213oih.22.1698761760670; Tue, 31 Oct 2023 07:16:00 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:d1:0:b0:4f0:1250:dd51 with HTTP; Tue, 31 Oct 2023 07:16:00 -0700 (PDT) In-Reply-To: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> From: Mateusz Guzik Date: Tue, 31 Oct 2023 15:16:00 +0100 Message-ID: Subject: Re: vnode_init took over 6 minutes To: John F Carr Cc: Freebsd fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4SKXHQ0rGGz3Ngw On 10/31/23, John F Carr wrote: > During a poudriere build control-T showed an awk process was stuck > for over 6 minutes trying to acquire a lock in vnode_init. > > load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k > __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 > keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed > cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f > zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 > zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 > cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea > vn_open_cred+0x48d > > The stack trace was the same for several minutes. System CPU time was > increasing. > > Address vnode_init+0xc3 corresponds to the mtx_lock call here. > > vp->v_holdcnt = VHOLD_NO_SMR; > vp->v_type = VNON; > mtx_lock(&vnode_list_mtx); > TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); > mtx_unlock(&vnode_list_mtx); > return (0); > > Address __mtx_lock_sleep+0xf8 is the instruction after a call to > lock_delay. > > ps says the command line was > > /usr/bin/awk -f /usr/bin/awk > old-default/2023-10-31_08h21m03s/.poudriere.builders > old-default/2023-10-31_08h21m03s/.poudriere.buildname ... > > with the full list of input files exceeding the ~2KB command line length > limit of ps. > "/usr/bin/awk" is probably not the real second argument. It would cause an > immediate syntax error. > > The hang resolved within a few more minutes and poudriere is continuing > happily. > > I have never seen such behavior before. Code in vfs_subr.c tries not to > hold the > vnode_list_mtx lock for too long. > > Any thoughts? > > > FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread Zen > 2, > zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB RAM. > what does "sysctl vfs.vnode" say -- Mateusz Guzik From nobody Tue Oct 31 14:27:33 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKXY10DNpz4yRtV for ; Tue, 31 Oct 2023 14:27:49 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKXXz3rNXz3Pqn; Tue, 31 Oct 2023 14:27:47 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-507d7b73b74so7856682e87.3; Tue, 31 Oct 2023 07:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698762466; x=1699367266; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=8emz0avbvlo2JIa8O4kBQh2g6leIayntVXVhaiV+tjw=; b=bqkFS7IfbSUFkNEZrW/BI48WLAvf+XN2k0lzW2KHQ6a5eHFPGxXgj+2AICKMpC+gwB okae2rFoS4Hn7KAtZtHm6jHCWXBHSWyuUkJ57TBOTiMGjJ8Pt10tR97Hxx+KO2gE3ttm 7OQQpZ8Rot/JpI/IlEi2hsR5gTt4jONJbco5wLSmxEKe8Ou/NBPp7Vg0KhJnLMidP3R8 u8UueSG/PlUd6rIIwcrleOUYauudBAWqw8Y2mv/Xtv1siuUJZpFnock/l5mDq9PWUJMG g+asMRdMVkbhDt4cyoQkX+4MUdr2bBxTraUjLnQbk39DgYD8t6y6G4qL9UW1aIIpldZx FjmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698762466; x=1699367266; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8emz0avbvlo2JIa8O4kBQh2g6leIayntVXVhaiV+tjw=; b=tUsrgoqpiXCEnEsZ+bmf3Zt4wrY/J/fIugxyZ2rqAlGnbzOJmqU3n7ooVC3TgTSV69 a2dMpdQCyWD6kI1KFTOSEKIkSLprlYU+vKvqpC9YqqYt1xhf2NRFP3Y430MqXjJ9BRMg VnpPm3amPpXLZCPyc7nFiZrjiu0XTFuqhBMeluiWrvg0XAMxcXFIkiKla4s7i4z4pO8M SQCUtv8hk68JjkYXGY5NWSRLgG0juABQ7T0p1QBhMm1ZjTQpDnJ+Qbp6AeeABNuDaz4c cgyJWmwPKUn0u3/HMhPZnROuHNlo0A/5rlUtbtxPjqnLdR+tuIUcOcP9eSyJBMwwzNE2 ttXQ== X-Gm-Message-State: AOJu0YwsCzIswYbnLKCNQURbHjh37O91IRLj4iNyjCoyi2ej9TtWxFVN KAVhtowoPkywxo8Xc3iWI9L0R3A+Ibkdhki/LLYhp3ZXEX6Puu5U X-Google-Smtp-Source: AGHT+IFoMuzoJ3v9Fccab4YPq7IyGx918cFDunGtKXTDO45aD2MRLl5M7Orklm+a+LqSJedIYyqwvtrB9CnxdtpXOIE= X-Received: by 2002:ac2:44ae:0:b0:503:38f2:6e1 with SMTP id c14-20020ac244ae000000b0050338f206e1mr8219613lfm.5.1698762465190; Tue, 31 Oct 2023 07:27:45 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> In-Reply-To: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> From: Alexander Leidinger Date: Tue, 31 Oct 2023 15:27:33 +0100 Message-ID: Subject: Re: ZFS txg rollback: expected timeframe? To: John F Carr Cc: "freebsd-fs@freebsd.org" , mm@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004ba942060903f531" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SKXXz3rNXz3Pqn --0000000000004ba942060903f531 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 31, 2023 at 1:15=E2=80=AFPM John F Carr wrote: > > > > On Oct 31, 2023, at 06:16, Alexander Leidinger < > alexleidingerde@gmail.com> wrote: > > > > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 > setup) in a way that a normal import of the pool panics the machine with > "VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867360 =3D=3D 10186= 7222)". > > > > I disabled that assertion because it gives a false alarm with some > combinaion > of deduplication, cloning, and snapshotting on one of my systems. > > See > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261538 > https://github.com/openzfs/zfs/issues/11480 I don't have deduplication on this pool. There are clones, and snapshots, and there could be recent ones if poudriere does some. Is it still a false alarm in this case? If yes, you say a kernel with this patch applied should let me import the pool without rollback? The github issue is from 2022, I have my doubts that this is the same issue we see. I rather expect some issues around the copy_file_range(2) related code for ZFS which was re-enabled 20 days ago (maybe it is valid to remove this assert, or maybe the block cloning part needs some tweak). CC Martin for the block cloning part. Bye, Alexander. --0000000000004ba942060903f531 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Oct 31, 2023 at 1:15=E2=80=AFPM J= ohn F Carr <jfc@mit.edu> wrote:


> On Oct 31, 2023, at 06:16, Alexander Leidinger <alexleidingerde@gmail.com&g= t; wrote:
>
> Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2= setup) in a way that a normal import of the pool panics the machine with &= quot;VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867360 =3D= =3D 101867222)".
>

I disabled that assertion because it gives a false alarm with some combinai= on
of deduplication, cloning, and snapshotting on one of my systems.

See

=C2=A0https://bugs.freebsd.org/bugzilla/sh= ow_bug.cgi?id=3D261538
=C2=A0https://github.com/openzfs/zfs/issues/11480

I don't have deduplication on this pool. T= here are clones, and snapshots,=C2=A0and there could be recent ones if poud= riere does some. Is it still a false alarm in this case? If yes, you say a = kernel with this patch applied should let me import the pool without rollba= ck?

The github issue is from 2022, I have my doubt= s that this is the same issue we see. I rather expect some issues around th= e copy_file_range(2) related code for ZFS which was re-enabled 20 days ago = (maybe it is valid to remove this assert, or maybe the block cloning part n= eeds some tweak). CC Martin for the block cloning part.

Bye,
Alexander.
--0000000000004ba942060903f531-- From nobody Tue Oct 31 14:38:04 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKXmv2cwdz4yS4D for ; Tue, 31 Oct 2023 14:38:07 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc2a.google.com (mail-oo1-xc2a.google.com [IPv6:2607:f8b0:4864:20::c2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKXmt4tdhz3QXV for ; Tue, 31 Oct 2023 14:38:06 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=nEULfTpH; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::c2a as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-5872c6529e1so393144eaf.0 for ; Tue, 31 Oct 2023 07:38:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698763085; x=1699367885; darn=freebsd.org; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=yuIgZ8eGI9LCVHt6uMDHDxnUY2ZjrbScT/lrp36/mSc=; b=nEULfTpHFVNjepePtkQvE7I8k7VnhdhAowYKYJJHwiRpOSqbDVr9orINAPVhRucbD7 OJQWIluTFrka9ugrpSB9qwcVIBmac4R5vY1TosfahGRF4KipPc5O6sIRBICRz0+GZZdF H1LHD471qSR9HacWH+LwJWgj6lSsbDvVhNINps6sY3dFz4c4MkSp5ChA77cn0rHFO599 ndphPqSYR5JQCPdlb89E12LZt41Qh8hg5iHV93WuXRw6hUdpm3rLhIJptZMcb3FDZK2N 3ziZqLagonDIcTWY2fyyBhTT/0IP/qkUhijlBqJK3WOul+PSagXsI3JKOx8XvCQ/p/sn xQGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698763085; x=1699367885; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yuIgZ8eGI9LCVHt6uMDHDxnUY2ZjrbScT/lrp36/mSc=; b=NZSaNJT4RNthjQhtCI9ND/pSPkQxRFf3BtpQFi0OU/f1LYahV4ozoYUKZmaMRopH7y baKhAAnXL2U5rw4Z7CrgP2Yd/txeTfaYVbn/neyIHddy3FXQhFZQhqqhHC3I33jQQ6C2 eCu6DDVYEXDgyuSWJsfI31r30RFMgOCoY6FIvZZWaS3v91Ge+at7XELVYoquVPmgQsyc yI9cRutLEyLGcuBiHBM5d9f9+a25XR4Cfnj32WrKp/qPCKkXaeL/LR2gb2Xr9xuIfh+Z vXIVF6fx1xwrufGce3/BsiHSyNcHtqezW8Y2c/+LJvgI4+6/FsatYDC0ckiK/Jt+fNC5 0Wbw== X-Gm-Message-State: AOJu0Ywpn1rkPBRAZLdHRqy7tSemnErIPf3sDp7gb10HxWEdBKb4Aehy ImkY6XNxYlUV4W2Qo1klRhWzQxYsB9UAy60eb444uMk5 X-Google-Smtp-Source: AGHT+IFzBhqP61j490rLtUUjCe4xZRx+XzP2d8nTOMPH0WcB5cF7EtmCEmFM0QuaMxPNbrOj/aE6fLqOaOqMdpBfr4I= X-Received: by 2002:a4a:d48b:0:b0:581:e850:6e8 with SMTP id o11-20020a4ad48b000000b00581e85006e8mr12618217oos.1.1698763085081; Tue, 31 Oct 2023 07:38:05 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:d1:0:b0:4f0:1250:dd51 with HTTP; Tue, 31 Oct 2023 07:38:04 -0700 (PDT) In-Reply-To: References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> From: Mateusz Guzik Date: Tue, 31 Oct 2023 15:38:04 +0100 Message-ID: Subject: Re: vnode_init took over 6 minutes To: John F Carr Cc: Freebsd fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.89 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.97)[-0.974]; NEURAL_HAM_SHORT(-0.92)[-0.918]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c2a:from]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4SKXmt4tdhz3QXV X-Spamd-Bar: --- On 10/31/23, Mateusz Guzik wrote: > On 10/31/23, John F Carr wrote: >> During a poudriere build control-T showed an awk process was stuck >> for over 6 minutes trying to acquire a lock in vnode_init. >> >> load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k >> __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 >> keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed >> cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f >> zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 >> zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 >> cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea >> vn_open_cred+0x48d >> >> The stack trace was the same for several minutes. System CPU time was >> increasing. >> >> Address vnode_init+0xc3 corresponds to the mtx_lock call here. >> >> vp->v_holdcnt = VHOLD_NO_SMR; >> vp->v_type = VNON; >> mtx_lock(&vnode_list_mtx); >> TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); >> mtx_unlock(&vnode_list_mtx); >> return (0); >> >> Address __mtx_lock_sleep+0xf8 is the instruction after a call to >> lock_delay. >> >> ps says the command line was >> >> /usr/bin/awk -f /usr/bin/awk >> old-default/2023-10-31_08h21m03s/.poudriere.builders >> old-default/2023-10-31_08h21m03s/.poudriere.buildname ... >> >> with the full list of input files exceeding the ~2KB command line length >> limit of ps. >> "/usr/bin/awk" is probably not the real second argument. It would cause >> an >> immediate syntax error. >> >> The hang resolved within a few more minutes and poudriere is continuing >> happily. >> >> I have never seen such behavior before. Code in vfs_subr.c tries not to >> hold the >> vnode_list_mtx lock for too long. >> >> Any thoughts? >> >> >> FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread Zen >> 2, >> zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB RAM. >> > > what does "sysctl vfs.vnode" say that and "sysctl vm.uma.VNODE" -- Mateusz Guzik From nobody Tue Oct 31 15:25:27 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKYrm3BYpz4yVyP for ; Tue, 31 Oct 2023 15:26:32 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-5.mit.edu (outgoing-exchange-5.mit.edu [18.9.28.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKYrm1MrZz3W2p for ; Tue, 31 Oct 2023 15:26:32 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none Received: from w92exedge4.exchange.mit.edu (W92EXEDGE4.EXCHANGE.MIT.EDU [18.7.73.16]) by outgoing-exchange-5.mit.edu (8.14.7/8.12.4) with ESMTP id 39VFQS2k012067; Tue, 31 Oct 2023 11:26:29 -0400 Received: from w92exhyb3.exchange.mit.edu (18.7.71.73) by w92exedge4.exchange.mit.edu (18.7.73.16) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 11:24:59 -0400 Received: from oc11exhyb6.exchange.mit.edu (18.9.1.111) by w92exhyb3.exchange.mit.edu (18.7.71.73) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 11:25:29 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.168) by oc11exhyb6.exchange.mit.edu (18.9.1.111) with Microsoft SMTP Server (TLS) id 15.0.1497.48 via Frontend Transport; Tue, 31 Oct 2023 11:25:29 -0400 Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by BN0PR01MB6831.prod.exchangelabs.com (2603:10b6:408:14b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.31; Tue, 31 Oct 2023 15:25:28 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1%4]) with mapi id 15.20.6907.032; Tue, 31 Oct 2023 15:25:27 +0000 From: "John F Carr" To: Mateusz Guzik CC: Freebsd fs Subject: Re: vnode_init took over 6 minutes Thread-Topic: vnode_init took over 6 minutes Thread-Index: AQHaDAIzJL6ZDeRtpkqkNz6JuK6CTLBj8YcAgAATW4A= Date: Tue, 31 Oct 2023 15:25:27 +0000 Message-ID: <8F17FEC3-C3CB-410D-ACEC-ADD12B3A747A@mit.edu> References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|BN0PR01MB6831:EE_ x-ms-office365-filtering-correlation-id: 5043ea77-ddb7-42b7-69ee-08dbda259c67 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: P4evgtrnDYXO4+K0ETSvIozTZssnvOyOdXdqWcCuMJsGgJtpHMHr6aXraGrojsWU0OqohbJ0vBAL7wbKLjw3YdJIhUqmKKIdWRACnCWrFseGeWHp6gGSBdV1CcqTMo8AoGX6lLr2SzKNF2HTZLXPzmCJ7SL3QCtuQAhwXSsGk0bTvD5406Ad7FyrlM9zQQRGLyQ99RlhQicY3fLA90qKCWb4/cK3rplnxdFUxtM6Xeh9MAQ7Jd/qxlUhSymnHSmrd+yqv0GSg8CD5eULbOF7Ngo+WFFYAd2eo/VFMlG9kpJNxSOGWK7sRyHmvMU3sBMNsl3rjjvybu0ObcEflGXzt12N8WJrm4PT0dWkx1KDxxUOmzm1SCWJs6+IS8GOwsWICTyqjJcqNTeqfyjGd0yJv7PkuSkLr5w6+N/VTWbT+Z1fiAEM9u7gxNAnGM1sW0LpeeAfWSqfS5noHTZ1NeXC24f7J12EMQ/cBaRj2FdsYQ6y/oSMFs3su/umJ8ei+15oUrYuQRrkUeZPh5iEWyWteOcSBsmPk0QfB7wRaY7zHNeUxzRoHQ3tfqbHif7xKdjPqgUO6JmsuEPaRbGNXuykZVzyHmEDFApmQlR0r+k9savLV7ZxaYhPuwsWi34gzt/I x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(39860400002)(366004)(346002)(136003)(396003)(230922051799003)(451199024)(1800799009)(64100799003)(186009)(38070700009)(38100700002)(2906002)(36756003)(4326008)(33656002)(66946007)(8676002)(8936002)(66556008)(75432002)(5660300002)(122000001)(76116006)(478600001)(71200400001)(6916009)(316002)(66446008)(64756008)(83380400001)(6506007)(91956017)(66476007)(786003)(86362001)(2616005)(53546011)(41300700001)(6486002)(6512007);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?qNrGwNwEUIgCkIDNMdb/eRaBnnyr+/+k64N4ALQRk6VWEBeMyojI1Ihtr1RW?= =?us-ascii?Q?sxyGu+Wlq/fy8Hj1Aar5RBEM9h9o3Mo0c/GcE5C0VKIfcnhPz/fzdkRcT9PQ?= =?us-ascii?Q?a/k+sKpp0ly0Av8w+miQEQbGBpGhFV/pRQu5JW7Bn9VpLeXko82YpI2KjPCG?= =?us-ascii?Q?MCyW9je78ef/2NwQ5OecplaFpwMqMoiFRGcFJJP9Z18XwB2XfEDXDnudXnKm?= =?us-ascii?Q?lldcYTXijTEktMykHtY4mGmMDXQu7T84P1c8a0s21yxnx4DuwE6LgcenisKB?= =?us-ascii?Q?+fwLrrBPF5BddOEaXduTI5URhfziEuDqtJLvQCA/2r7ti5XQqqijEpJFImh+?= =?us-ascii?Q?tQi9+WuX6uZfPReZoG03OEOZeJwoh2M15UjHgFv9l43u+8dxsfzruTiUpIFl?= =?us-ascii?Q?x6yTy+j/snxzYAZELTF9Xg7DxH9/NNC0+r3exlwYsKjCi40LqxooHGPOfcOm?= =?us-ascii?Q?TJUWyMYmb1NJMlOYBlpSa0TqQNbppq+ZBp69GZ+cgd0iv2kXckqg9ScMeoZc?= =?us-ascii?Q?9c8upuScAZSjpno7tCoMgoDbUWRJ0Jndxf8rZn5JtLtfg3l7+KgkSyO71Mvx?= =?us-ascii?Q?Df4i22WojVCVBScoYuKDdBPaYM9bHIff8Wf+8hMnporRQeR9KiAnp9PUgXpy?= =?us-ascii?Q?nNZQ/0HB2PkoMKaKW1AX+XH4CUz/HNy8A7yxnDSw7ISSzRNxVVJOYpyJpbSb?= =?us-ascii?Q?5Zpy80ELB0suUeWv79D2RhLqRF0moipAgpdoGeJu6bvKvrtaAkr7HNuIOZRr?= =?us-ascii?Q?AKVPpPT87W6blSGK1Im2isi0LKj2LEWQ69iXTQ2MntZLp3aywuF/jJuRx1hZ?= =?us-ascii?Q?Ph/2bzE8bVHmRNMenV7/BYrkI07rS1qOd9muPQljUT6jsCzo+Rm2fRv9hguA?= =?us-ascii?Q?+2hHvtLD9XL71RucAdwBXxTcPEGQ7u58JJWO/+/jx6yx4b/c+6sXavYJnWii?= =?us-ascii?Q?CqypkTW3pSizODxTTY/JyP6stvx1JjKORSSQ2MSaPLLHOQtO9v5eY2bhjSqc?= =?us-ascii?Q?NUynaXnJRP7WqTw3yWv4ql2Q5u4Y+kbcNoC7EitT8pw1e5bPMwR3CvIWdrXh?= =?us-ascii?Q?FuM4IcBYevLRbhsMMqvqH0SvabilGPGxwSUN+N0H6DUf7AwJsP1DtZN7iYa2?= =?us-ascii?Q?7HXlf5ShSi9/1yCg6KLNVWez2H2A3hezC3xdsN/iSED1cWu/1Ty+lzQhRA3G?= =?us-ascii?Q?RlGa0v9YGi0k1/0aQdGgTfXtyL5ej9pvjrtAZVJYox5RamaD0++c0h2cRHXg?= =?us-ascii?Q?I3ApS6YZ//uvtxGNkOGJMbpSzJ1luA6jvG7wa6QyxNtmHahngi8Uh1JPkO1L?= =?us-ascii?Q?o0P61M7HbNP+VUaofFmwEvvFV4yFx4YgcKtszO3mAaSFIWTDq7JEnpgu52am?= =?us-ascii?Q?PJQVe/qQ2qVt/lzWdB2siZr5Ep3WIimFdF0DkHETbgK2kPTDo2qfdsfu2A+j?= =?us-ascii?Q?XxZ0iyBFP+HeBp6WHHt2eI1nyApgRtLzDTJDij/Iglb0UvNdNGpLq/Vm4PPe?= =?us-ascii?Q?Fy/gW1DyKd+ZN/blycgq/5R1N1LDGSWXmjvOjjD2tdFtBMVh3spkM5ptR05Q?= =?us-ascii?Q?v1M4iDv2H9W0JqKKFEzv0JszdfstcAz/T1gMEXsPbBVKV4zZDPGP82rXNfQn?= =?us-ascii?Q?h6HM6bghADgPVnvL2UEgzA/KOegr/BbH35TN3ATFiJhv?= Content-Type: text/plain; charset="us-ascii" Content-ID: <843CC50C8A236746B1C81FD30FA2A044@prod.exchangelabs.com> Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5043ea77-ddb7-42b7-69ee-08dbda259c67 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2023 15:25:27.5539 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EVmHVbnKhtHaZ7tXfi0WiO33qLEujsRDGjPI9g3z34ItYj7brkEaYvtyRtXVWcCj X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR01MB6831 X-OriginatorOrg: mit.edu X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US] X-Rspamd-Queue-Id: 4SKYrm1MrZz3W2p > On Oct 31, 2023, at 10:16, Mateusz Guzik wrote: >=20 > On 10/31/23, John F Carr wrote: >> During a poudriere build control-T showed an awk process was stuck >> for over 6 minutes trying to acquire a lock in vnode_init. >>=20 >> load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k >> __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 >> keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed >> cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f >> zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 >> zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 >> cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea >> vn_open_cred+0x48d >>=20 >> The stack trace was the same for several minutes. System CPU time was >> increasing. >>=20 >> Address vnode_init+0xc3 corresponds to the mtx_lock call here. >>=20 >> vp->v_holdcnt =3D VHOLD_NO_SMR; >> vp->v_type =3D VNON; >> mtx_lock(&vnode_list_mtx); >> TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); >> mtx_unlock(&vnode_list_mtx); >> return (0); >>=20 >> Address __mtx_lock_sleep+0xf8 is the instruction after a call to >> lock_delay. >>=20 >> ps says the command line was >>=20 >> /usr/bin/awk -f /usr/bin/awk >> old-default/2023-10-31_08h21m03s/.poudriere.builders >> old-default/2023-10-31_08h21m03s/.poudriere.buildname ... >>=20 >> with the full list of input files exceeding the ~2KB command line length >> limit of ps. >> "/usr/bin/awk" is probably not the real second argument. It would cause= an >> immediate syntax error. >>=20 >> The hang resolved within a few more minutes and poudriere is continuing >> happily. >>=20 >> I have never seen such behavior before. Code in vfs_subr.c tries not to >> hold the >> vnode_list_mtx lock for too long. >>=20 >> Any thoughts? >>=20 >>=20 >> FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread Ze= n >> 2, >> zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB RAM= . >>=20 >=20 > what does "sysctl vfs.vnode" say >=20 >=20 > --=20 > Mateusz Guzik With the builds now complete, vfs.vnode.vnlru.uma_reclaim_calls: 0 vfs.vnode.vnlru.kicks: 0 vfs.vnode.vnlru.max_free_per_call: 10000 vfs.vnode.vnlru.failed_runs: 0 vfs.vnode.vnlru.direct_recycles_free: 152871194 vfs.vnode.vnlru.recycles_free: 77190881 vfs.vnode.vnlru.recycles: 0 vfs.vnode.stats.alloc_sleeps: 0 vfs.vnode.stats.free: 1269702 vfs.vnode.stats.skipped_requeues: 9375158 vfs.vnode.stats.created: 235594976 vfs.vnode.stats.count: 1407640 vfs.vnode.param.wantfree: 681059 vfs.vnode.param.limit: 2724237 Uptime is 11 days, mostly with low I/O rates. From nobody Tue Oct 31 15:42:10 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKZBr3NqBz4yX2H for ; Tue, 31 Oct 2023 15:42:12 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKZBr0clyz3XCB for ; Tue, 31 Oct 2023 15:42:12 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-5872c6529e1so437128eaf.0 for ; Tue, 31 Oct 2023 08:42:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698766931; x=1699371731; darn=freebsd.org; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=XlsqFRRu8jGVXKMFLZE2HUhuAJpkE/TTAurOJG4znNM=; b=YaNGlKLfh8U6XGftIxAWL2UFvT2t66CbY35EHtqIFcwczk3WDdUGuW9VzAvb5xzlBr R/6F6U2R0Rt4dzfl8/9bb57v3qTqBJBQ62SiRIpy5h6vIAYluxxd9twrrFQSz6bZuaTq IulzRzQ+jo6K4HxHSH141ZRSxAkHTsdEW1AIyXWb5F2nXRAEODW4ZjuEBixiWFd5Fq9r NsVhIjatjVOq1ukKoObCyVFcNcrk8kjO0n2XD2chpMGpc+0jPlj1lhVxvHxCLyh00EC+ VNIeuzN/MjzhN4PbncHREUWEBXISeLCw3K0KuJfYdNaa90FLGiSC5mUQldSUuVziT/hG i/gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698766931; x=1699371731; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XlsqFRRu8jGVXKMFLZE2HUhuAJpkE/TTAurOJG4znNM=; b=eRVFpcb3VKmGBLTQLxqrg4VozakrmdwV4zOQnNYdEsffXLRmuobwVj0e5r2uu9uPaq sDgjGrcEWGT+RLxkoAzL+9AiMxi24/1LdCNlJGi9SxNbXSpDOe+bDj8AlAp6ToDzh3zn EXpzDxYTfT2EkC3MKWmwFaHT/IFV8xRC/E4QcQM2UnwTSVzv5lgQl5EFASZM25FBPH7B Vgp+dtejx7w/3SMltA9EYwhzLBX1MArWmT4+cmDQnCaGLGZQ7bQ2vZgut1MW1h9HMNIz qV970A1OKIW9GYY7dgB8E2eQAeAwDqyv4lSl5LxnhMjbs+byHS5WkEbiWHyLeG8MqB4v 9krA== X-Gm-Message-State: AOJu0YxWhFFU+npUWnF4pkkKAaSSVzyopslZ4ssxBBHK1zBiTrtakXrK 7dey1j/Ds01O7nq8hzT/ZGq78idCJqZPbmyZe1c= X-Google-Smtp-Source: AGHT+IEQI7D50K3CJ/E9uxZfBUrWSZSH+/918txpCj+t9raEMZu3sT/zgNng6AxcCuXX6TL3oZV4jGfFA3hX1BWkjPo= X-Received: by 2002:a05:6820:3d3:b0:586:a887:4633 with SMTP id s19-20020a05682003d300b00586a8874633mr10967917ooj.5.1698766930797; Tue, 31 Oct 2023 08:42:10 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:d1:0:b0:4f0:1250:dd51 with HTTP; Tue, 31 Oct 2023 08:42:10 -0700 (PDT) In-Reply-To: <8F17FEC3-C3CB-410D-ACEC-ADD12B3A747A@mit.edu> References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> <8F17FEC3-C3CB-410D-ACEC-ADD12B3A747A@mit.edu> From: Mateusz Guzik Date: Tue, 31 Oct 2023 16:42:10 +0100 Message-ID: Subject: Re: vnode_init took over 6 minutes To: John F Carr Cc: Freebsd fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4SKZBr0clyz3XCB On 10/31/23, John F Carr wrote: > > >> On Oct 31, 2023, at 10:16, Mateusz Guzik wrote: >> >> On 10/31/23, John F Carr wrote: >>> During a poudriere build control-T showed an awk process was stuck >>> for over 6 minutes trying to acquire a lock in vnode_init. >>> >>> load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k >>> __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 >>> keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed >>> cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f >>> zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 >>> zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 >>> cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea >>> vn_open_cred+0x48d >>> >>> The stack trace was the same for several minutes. System CPU time was >>> increasing. >>> >>> Address vnode_init+0xc3 corresponds to the mtx_lock call here. >>> >>> vp->v_holdcnt = VHOLD_NO_SMR; >>> vp->v_type = VNON; >>> mtx_lock(&vnode_list_mtx); >>> TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); >>> mtx_unlock(&vnode_list_mtx); >>> return (0); >>> >>> Address __mtx_lock_sleep+0xf8 is the instruction after a call to >>> lock_delay. >>> >>> ps says the command line was >>> >>> /usr/bin/awk -f /usr/bin/awk >>> old-default/2023-10-31_08h21m03s/.poudriere.builders >>> old-default/2023-10-31_08h21m03s/.poudriere.buildname ... >>> >>> with the full list of input files exceeding the ~2KB command line length >>> limit of ps. >>> "/usr/bin/awk" is probably not the real second argument. It would cause >>> an >>> immediate syntax error. >>> >>> The hang resolved within a few more minutes and poudriere is continuing >>> happily. >>> >>> I have never seen such behavior before. Code in vfs_subr.c tries not to >>> hold the >>> vnode_list_mtx lock for too long. >>> >>> Any thoughts? >>> >>> >>> FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread >>> Zen >>> 2, >>> zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB >>> RAM. >>> >> >> what does "sysctl vfs.vnode" say >> >> >> -- >> Mateusz Guzik > > With the builds now complete, > > vfs.vnode.vnlru.uma_reclaim_calls: 0 > vfs.vnode.vnlru.kicks: 0 > vfs.vnode.vnlru.max_free_per_call: 10000 > vfs.vnode.vnlru.failed_runs: 0 > vfs.vnode.vnlru.direct_recycles_free: 152871194 > vfs.vnode.vnlru.recycles_free: 77190881 > vfs.vnode.vnlru.recycles: 0 > vfs.vnode.stats.alloc_sleeps: 0 > vfs.vnode.stats.free: 1269702 > vfs.vnode.stats.skipped_requeues: 9375158 > vfs.vnode.stats.created: 235594976 > vfs.vnode.stats.count: 1407640 > vfs.vnode.param.wantfree: 681059 > vfs.vnode.param.limit: 2724237 > > Uptime is 11 days, mostly with low I/O rates. > > please add the uma stats: "sysctl vm.uma.VNODE" -- Mateusz Guzik From nobody Tue Oct 31 15:57:59 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKZYm5NBqz4yXrB for ; Tue, 31 Oct 2023 15:58:36 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-7.mit.edu (outgoing-exchange-7.mit.edu [18.9.28.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKZYm3KxRz3YSM for ; Tue, 31 Oct 2023 15:58:36 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none Received: from oc11exedge2.exchange.mit.edu (OC11EXEDGE2.EXCHANGE.MIT.EDU [18.9.3.18]) by outgoing-exchange-7.mit.edu (8.14.7/8.12.4) with ESMTP id 39VFvdha019072; Tue, 31 Oct 2023 11:58:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1698767915; bh=9Eqk2NsFNax1ILb2xwmeHE9yJ4QjLyie1zeDpx28XN4=; h=From:Subject:Date:Message-ID:Content-Type:MIME-Version; b=ER4qd76x/n3V1Cqy0wBMM6MeKAsAb4SBD+f6K5gpKGlPW/x0vtN+HcXcHEfeNTkcU 3vWcyZZuyEbCmFxT1h3y/N0uch/SNDA67bXXLHOk8K3z44ZwtBoARKMrWvAzIxKNtA yGTpBzaWdDRMwEyPb5GCMiibEvyvmp+61c1Rd0SG/snal+QXM0JV+NRWGXJEoJWjrx YPyKgvRwatnGRl2wPoijExYmgZZ3cqebzrhCVRjY0dny5iN2zMIBLF9cLdAhwS8Ksn STpuUIWfwsffWH0LFsmTxeNEr+8JNYEF4dWQk8en7wenbtf5OHNCapniGKfFE9F5nU qXpQXxXSssMJQ== Received: from oc11exhyb4.exchange.mit.edu (18.9.1.100) by oc11exedge2.exchange.mit.edu (18.9.3.18) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 11:57:29 -0400 Received: from oc11exhyb3.exchange.mit.edu (18.9.1.99) by oc11exhyb4.exchange.mit.edu (18.9.1.100) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 31 Oct 2023 11:58:01 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.41) by oc11exhyb3.exchange.mit.edu (18.9.1.99) with Microsoft SMTP Server (TLS) id 15.0.1497.48 via Frontend Transport; Tue, 31 Oct 2023 11:58:01 -0400 Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by IA0PR01MB8615.prod.exchangelabs.com (2603:10b6:208:48a::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Tue, 31 Oct 2023 15:57:59 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::2e94:6383:c13:23b1%4]) with mapi id 15.20.6907.032; Tue, 31 Oct 2023 15:57:59 +0000 From: "John F Carr" To: Mateusz Guzik CC: Freebsd fs Subject: Re: vnode_init took over 6 minutes Thread-Topic: vnode_init took over 6 minutes Thread-Index: AQHaDAIzJL6ZDeRtpkqkNz6JuK6CTLBj8YcAgAATW4CAAAS4AIAABF4A Date: Tue, 31 Oct 2023 15:57:59 +0000 Message-ID: <29268BEF-8620-4CAE-B69D-2D2C2EF59868@mit.edu> References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> <8F17FEC3-C3CB-410D-ACEC-ADD12B3A747A@mit.edu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|IA0PR01MB8615:EE_ x-ms-office365-filtering-correlation-id: 839474c4-f6de-4512-e978-08dbda2a2796 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: l4aKj9eD+HT/iIBtP8yg+Va32NGMjfvQtm63hwEU/9kZcCBpfjl4hQmS8wGvIvY5XAXr/g00wDmMrEzNfkYbeTZw0rh0fnVHYmAk54+QIOgLLO8759ZYveS98eDI61DZdf18xf5653mpm0i+JIOMGnhUV+Ox0fZjmVeIaCHqW5wCb/vp5a5Zwk8ZF2BMAHycONyv4/e43FwEWY7CSjUhdicBBahS3cQWjOw6ciXgkcuat/Q4b2hDl7AGPirKr5IO/FgoL/h0ljjZrUiQKlMeBuFATU1/oH0qAL9pYWO4iixp80iE/g8oaDIGMWOTqdfCmTE0U9DlGLqn/6ZEL5//DPaKHvGxcbDbtedK90EMArd+oh7/Sy0e1sdNW6xuX9jSKi1BgMxKTWhGZYXEuuauNT1eJZjqMA3QIheP576w7yFgza4GJ7TLkseZNyWMLYbCeIR6N6Kqjn0hJ19zOI1aXJmiJLaJF4mXPV1rMYumR4aOd2M2niH1yc9yPVMFCsL/vRl5B2KW1B3sBfTrmTocqj2J8AWSjyNjG7kEC2cJt63uKvO7N5XGXgeYENXYDKY1tsnCmFqdnQekvo24ij8P6LJqMIC1W70quXQerRwnnBIMFFbKazvvVXcWTEXy1Mi9 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(396003)(39860400002)(376002)(366004)(136003)(230922051799003)(1800799009)(451199024)(186009)(64100799003)(6916009)(66446008)(2906002)(38100700002)(41300700001)(71200400001)(316002)(66946007)(786003)(66476007)(53546011)(64756008)(76116006)(66556008)(478600001)(91956017)(122000001)(6506007)(6512007)(6486002)(83380400001)(86362001)(5660300002)(8676002)(36756003)(33656002)(4326008)(2616005)(8936002)(75432002)(38070700009);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FDog2AUQAIopDD3VjtOmwlKD6OSAPifwSdNwcED6vt4pes6jcR8pIwzf2ga8?= =?us-ascii?Q?fZoeAwFWsTdi8gTnYpqvJ2WhGaPX8hDblXoeZluNRePt/O9QP9vf5BzkbPty?= =?us-ascii?Q?jhk6QvSOX2rTmOC5jQryYbVYgf9v6Mw/gmvWDrQQV3vW1hWj9DYNJSJF9K16?= =?us-ascii?Q?p5dXBfkm2UWZlxmQFKHZsr+bQkdh8ilfAB3hICYxkunsuP3uXhoH1COLCv96?= =?us-ascii?Q?snAhsKiOyO7CPWBqZCGluuZxy8zbwS4fvl+pRFPcWWb6fFgmEiyqBy11NsVO?= =?us-ascii?Q?OCM03JzKUCFl4Cz5zFNRNElEYxsIh4ivAnZutmASk7UQKilqldNtoATSGTYd?= =?us-ascii?Q?THYVsfKpC1W0ea4eU8F5jZeGfxFgrUk19wbliwT6tM7iHgqFP3H8sfYLhgI9?= =?us-ascii?Q?un3KWAM+2j5/a6M20Sw9FkMvxql4w2vbbEompshbOEAgTTtgwF/y726J3/Sw?= =?us-ascii?Q?9wfueWuenbTDtygAqPUzxWvysleF7ICBIs1N0QpwUzrslbEmvtDnz0KskiZp?= =?us-ascii?Q?7PJ0UNdLBzT69K4lXvRWSam9psdX3mBYPW7Cdes+Fk243QWTH/SQnv8mItUv?= =?us-ascii?Q?cyjoXxfDHWfPFtbrUKuMvOdQ5zphhUla8TsMISOcNWVR4NegnMhZ7EeA9L06?= =?us-ascii?Q?4Tkgd/twJafr8K2bUlZJn8bKU7o3DaTVjMQvE4UmtYYaeNNICyCYomUhKAY+?= =?us-ascii?Q?Q0LfXqUhtUmAeZC77jhjv8kt2/iH2Q1WS6h6Y3A2Iwe/BjwW5APhXnp0DEpn?= =?us-ascii?Q?SGJIyemm6oSPIFMTTZdhmRhSk2ltMpAfuA2tcNhgF7Kp9f+WnHRGTT9Pgrk4?= =?us-ascii?Q?YVLj7RkMiSX+gyEZT5KKjMzOuHKjn72Zk1WmZYzuDGuCxkw5/Hkj4tS6587q?= =?us-ascii?Q?srV9RPKgA/k1B3DGQNLwChMKGyAiATWUTXr73LYZ9ZQww2ow7YOl2wFquSfJ?= =?us-ascii?Q?higx3dLZfybzIzUZAp4cBfWd8dRT6u525ESIdwP71vWusXb6Vx87CLlS3R3X?= =?us-ascii?Q?9krKSWQb9BupfhKRPYlVVfLlY0CEZmD4FFoqMu29t6zgq4YGwzq83XPqNgB3?= =?us-ascii?Q?SOfjVxqwxivCU5VkNyHdqQ1SMfN0f+gV5LHWmkLZ0kVPGqpW4K0PYrepJvY/?= =?us-ascii?Q?Dqb11DlCphrd6ZclIHvY9gK2LfzFQk9mg81x3RtD1g6iH5CrCypTssl9APM4?= =?us-ascii?Q?yc7WBluFqNbugZWT4V0Rn0um0MxWq9rwD01fEjgzURSHo7VF2d+/LKSpgv33?= =?us-ascii?Q?j8wWqifJ6mEmNyL9wDz1+oDVjpZZgRqV0bEq591CY8u14jqnuPq6q7a1KV02?= =?us-ascii?Q?RkEerAROy5yUO/geCzAhjKpPjxPKh5cBdVhnvF3jgHafXeU//GmeHYxa6Qvl?= =?us-ascii?Q?LIE3yfFhgpxzBUOhBQoqk8LXD2xKZQSRgqA8NJ1paHIzvR9lfQXuk/tGszpA?= =?us-ascii?Q?eeyp5Ffz9NHRVKqHH7XPvs+ZZh4p1TFWLssIaVk9gJ6mteOV+LYo7dBTRGmf?= =?us-ascii?Q?dkKivWPSBHKDqmwyEY1GNexOcviA2xAVSaXLKkzaaooQ+yUoYBOOnQ+KpSlh?= =?us-ascii?Q?Dmd2p+zuzUw/szp0QnmU/mGWGkPcrgniWQqn/gHfgThgpM0fksDbG5S2krwx?= =?us-ascii?Q?r1emHGBUeqZXm0yLi6cp/A0caXRN/t7e1GIgijl0f4SK?= Content-Type: text/plain; charset="us-ascii" Content-ID: <3467FD3A0A53BD449E6E51D5D874DE0A@prod.exchangelabs.com> Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 839474c4-f6de-4512-e978-08dbda2a2796 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2023 15:57:59.0567 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: IMbUdageHO8m75HBcN2r7F2Uo4p6OGW3bL5+WyCYPm6xi1sw0jGUMda9Uv4816xJ X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR01MB8615 X-OriginatorOrg: mit.edu X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US] X-Rspamd-Queue-Id: 4SKZYm3KxRz3YSM > On Oct 31, 2023, at 11:42, Mateusz Guzik wrote: >=20 > On 10/31/23, John F Carr wrote: >>=20 >>=20 >>> On Oct 31, 2023, at 10:16, Mateusz Guzik wrote: >>>=20 >>> On 10/31/23, John F Carr wrote: >>>> During a poudriere build control-T showed an awk process was stuck >>>> for over 6 minutes trying to acquire a lock in vnode_init. >>>>=20 >>>> load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k >>>> __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 >>>> keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed >>>> cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f >>>> zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 >>>> zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 >>>> cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea >>>> vn_open_cred+0x48d >>>>=20 >>>> The stack trace was the same for several minutes. System CPU time was >>>> increasing. >>>>=20 >>>> Address vnode_init+0xc3 corresponds to the mtx_lock call here. >>>>=20 >>>> vp->v_holdcnt =3D VHOLD_NO_SMR; >>>> vp->v_type =3D VNON; >>>> mtx_lock(&vnode_list_mtx); >>>> TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); >>>> mtx_unlock(&vnode_list_mtx); >>>> return (0); >>>>=20 >>>> Address __mtx_lock_sleep+0xf8 is the instruction after a call to >>>> lock_delay. >>>>=20 >>>> ps says the command line was >>>>=20 >>>> /usr/bin/awk -f /usr/bin/awk >>>> old-default/2023-10-31_08h21m03s/.poudriere.builders >>>> old-default/2023-10-31_08h21m03s/.poudriere.buildname ... >>>>=20 >>>> with the full list of input files exceeding the ~2KB command line leng= th >>>> limit of ps. >>>> "/usr/bin/awk" is probably not the real second argument. It would cau= se >>>> an >>>> immediate syntax error. >>>>=20 >>>> The hang resolved within a few more minutes and poudriere is continuin= g >>>> happily. >>>>=20 >>>> I have never seen such behavior before. Code in vfs_subr.c tries not = to >>>> hold the >>>> vnode_list_mtx lock for too long. >>>>=20 >>>> Any thoughts? >>>>=20 >>>>=20 >>>> FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread >>>> Zen >>>> 2, >>>> zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB >>>> RAM. >>>>=20 >>>=20 >>> what does "sysctl vfs.vnode" say >>>=20 >>>=20 >>> -- >>> Mateusz Guzik >>=20 >> With the builds now complete, >>=20 >> vfs.vnode.vnlru.uma_reclaim_calls: 0 >> vfs.vnode.vnlru.kicks: 0 >> vfs.vnode.vnlru.max_free_per_call: 10000 >> vfs.vnode.vnlru.failed_runs: 0 >> vfs.vnode.vnlru.direct_recycles_free: 152871194 >> vfs.vnode.vnlru.recycles_free: 77190881 >> vfs.vnode.vnlru.recycles: 0 >> vfs.vnode.stats.alloc_sleeps: 0 >> vfs.vnode.stats.free: 1269702 >> vfs.vnode.stats.skipped_requeues: 9375158 >> vfs.vnode.stats.created: 235594976 >> vfs.vnode.stats.count: 1407640 >> vfs.vnode.param.wantfree: 681059 >> vfs.vnode.param.limit: 2724237 >>=20 >> Uptime is 11 days, mostly with low I/O rates. >>=20 >>=20 >=20 > please add the uma stats: "sysctl vm.uma.VNODE" >=20 > --=20 > Mateusz Guzik vm.uma.VNODE.stats.xdomain: 0 vm.uma.VNODE.stats.fails: 0 vm.uma.VNODE.stats.frees: 245129082 vm.uma.VNODE.stats.allocs: 246536724 vm.uma.VNODE.stats.current: 1407642 vm.uma.VNODE.domain.0.timin: 215 vm.uma.VNODE.domain.0.limin: 4613 vm.uma.VNODE.domain.0.wss: 10080 vm.uma.VNODE.domain.0.bimin: 391986 vm.uma.VNODE.domain.0.imin: 391986 vm.uma.VNODE.domain.0.imax: 391986 vm.uma.VNODE.domain.0.nitems: 391986 vm.uma.VNODE.limit.bucket_max: 18446744073709551615 vm.uma.VNODE.limit.sleeps: 0 vm.uma.VNODE.limit.sleepers: 0 vm.uma.VNODE.limit.max_items: 0 vm.uma.VNODE.limit.items: 0 vm.uma.VNODE.keg.domain.0.free_slabs: 0 vm.uma.VNODE.keg.domain.0.free_items: 324187 vm.uma.VNODE.keg.domain.0.pages: 266134 vm.uma.VNODE.keg.efficiency: 95 vm.uma.VNODE.keg.reserve: 0 vm.uma.VNODE.keg.align: 7 vm.uma.VNODE.keg.ipers: 8 vm.uma.VNODE.keg.ppera: 1 vm.uma.VNODE.keg.rsize: 488 vm.uma.VNODE.keg.name: VNODE vm.uma.VNODE.bucket_size_max: 254 vm.uma.VNODE.bucket_size: 88 vm.uma.VNODE.flags: 0xd0000 vm.uma.VNODE.size: 488 From nobody Tue Oct 31 16:06:08 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKZkV0fTnz4yYGx for ; Tue, 31 Oct 2023 16:06:10 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc34.google.com (mail-oo1-xc34.google.com [IPv6:2607:f8b0:4864:20::c34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKZkV09ctz3Yth for ; Tue, 31 Oct 2023 16:06:10 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc34.google.com with SMTP id 006d021491bc7-586b512ba0aso2731967eaf.3 for ; Tue, 31 Oct 2023 09:06:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698768369; x=1699373169; darn=freebsd.org; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=KfyJ0mRuQVJMGaLMLo1R4TQse8O9aHhaKOb1H7jmNv0=; b=YETFURdqYLGwBcbGWMgYvpbDvyuRbGMVaEuE0TKZ/p0aHSwMHK3LT8pIw2WwxosOoT ql/l5XGOoWXfbf3iVpdbW1s0IVNA6YxtwK6ptGXH6H28txQqcH5MO7WphvHcHVLAxoO7 gXHTuPoMJNTPvtIue9R3X0vLtCaP/+FsEx1SuCZEIksCKVIF0Y9tLEw5Ab5f23k4DQNP fyhGA5UUMOwXZe4QeWzYpurjSFlW9rUt6cONNhRgTkIU3F37lNLlLO7dWlgSl9kV+YbZ XXS0jgY4NddB6X/bYU0EEAmc/WkRj+8OWHGTevkUKMQsX77ZtDBSzxS6nCTSkLdxAQoN /bdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698768369; x=1699373169; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KfyJ0mRuQVJMGaLMLo1R4TQse8O9aHhaKOb1H7jmNv0=; b=rHsLg3nKLsW/HOQo3DQh7YBlxSkinCSYUv+wNqA7Ho0rTxGkmSaqXXIL4S7myRtfbh ozySWM0WpglbjCcB8PU+LAql3ys7S5+Ndedj0grLa9NpJo7CSqamcaZi0+AmzubvPuyj Css198tvU9+a8Abg4fZsqnTH2F5hDqUXkn1fcxjc0VG3ZjzstgdVovq9sRWShTvEtPer DCqii3u8a8OeWw/xxGOBJMkVu+nTcKj9iGHLsdvK/uN5+RLd4l7RTgj1UwjJWo0B4H/p C+WO3aG1eNr4D5Zfp3bfa35e9oJIvQLgs4lO/ZfXE7CTx3UzbdUn1BMRvUPWPn9M6eCu A4Bw== X-Gm-Message-State: AOJu0YzJRRJyXmPcYQoQuqRLXXAP2+5sbBdpyuUAMCdkzD0rwZdb6Pu2 pa+8c5dxMj0Tpslz1joFq8MG8Soth7IsEGn2ScTKBctc X-Google-Smtp-Source: AGHT+IH3DA09epKut5+THAPC0oh007GZ4uCBJfmI03bO05YDUbRW2tcuyeR6Pzk4oURb4cu6205jd9w0Xhewha4n4mY= X-Received: by 2002:a4a:c885:0:b0:587:2b9b:985a with SMTP id t5-20020a4ac885000000b005872b9b985amr2228910ooq.9.1698768368626; Tue, 31 Oct 2023 09:06:08 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:d1:0:b0:4f0:1250:dd51 with HTTP; Tue, 31 Oct 2023 09:06:08 -0700 (PDT) In-Reply-To: <29268BEF-8620-4CAE-B69D-2D2C2EF59868@mit.edu> References: <67FBC292-E83D-4171-BE79-D28D5578F031@mit.edu> <8F17FEC3-C3CB-410D-ACEC-ADD12B3A747A@mit.edu> <29268BEF-8620-4CAE-B69D-2D2C2EF59868@mit.edu> From: Mateusz Guzik Date: Tue, 31 Oct 2023 17:06:08 +0100 Message-ID: Subject: Re: vnode_init took over 6 minutes To: John F Carr Cc: Freebsd fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4SKZkV09ctz3Yth On 10/31/23, John F Carr wrote: > > >> On Oct 31, 2023, at 11:42, Mateusz Guzik wrote: >> >> On 10/31/23, John F Carr wrote: >>> >>> >>>> On Oct 31, 2023, at 10:16, Mateusz Guzik wrote: >>>> >>>> On 10/31/23, John F Carr wrote: >>>>> During a poudriere build control-T showed an awk process was stuck >>>>> for over 6 minutes trying to acquire a lock in vnode_init. >>>>> >>>>> load: 8.58 cmd: awk 51238 [running] 392.88r 0.00u 392.88s 100% 2604k >>>>> __mtx_lock_sleep+0xf8 __mtx_lock_flags+0xa4 vnode_init+0xc3 >>>>> keg_alloc_slab+0x277 zone_import+0x143 cache_alloc+0x3ed >>>>> cache_alloc_retry+0x23 getnewvnode_reserve+0x20 zfs_zget+0x1f >>>>> zfs_dirent_lookup+0x16d zfs_dirlook+0x7f zfs_lookup+0x3e0 >>>>> zfs_freebsd_cachedlookup+0x74 vfs_cache_lookup+0xa7 >>>>> cache_fplookup_noentry+0x241 cache_fplookup+0x575 namei+0x1ea >>>>> vn_open_cred+0x48d >>>>> >>>>> The stack trace was the same for several minutes. System CPU time was >>>>> increasing. >>>>> >>>>> Address vnode_init+0xc3 corresponds to the mtx_lock call here. >>>>> >>>>> vp->v_holdcnt = VHOLD_NO_SMR; >>>>> vp->v_type = VNON; >>>>> mtx_lock(&vnode_list_mtx); >>>>> TAILQ_INSERT_BEFORE(vnode_list_free_marker, vp, v_vnodelist); >>>>> mtx_unlock(&vnode_list_mtx); >>>>> return (0); >>>>> >>>>> Address __mtx_lock_sleep+0xf8 is the instruction after a call to >>>>> lock_delay. >>>>> >>>>> ps says the command line was >>>>> >>>>> /usr/bin/awk -f /usr/bin/awk >>>>> old-default/2023-10-31_08h21m03s/.poudriere.builders >>>>> old-default/2023-10-31_08h21m03s/.poudriere.buildname ... >>>>> >>>>> with the full list of input files exceeding the ~2KB command line >>>>> length >>>>> limit of ps. >>>>> "/usr/bin/awk" is probably not the real second argument. It would >>>>> cause >>>>> an >>>>> immediate syntax error. >>>>> >>>>> The hang resolved within a few more minutes and poudriere is >>>>> continuing >>>>> happily. >>>>> >>>>> I have never seen such behavior before. Code in vfs_subr.c tries not >>>>> to >>>>> hold the >>>>> vnode_list_mtx lock for too long. >>>>> >>>>> Any thoughts? >>>>> >>>>> >>>>> FreeBSD 13.2-STABLE up through commit b180f0040f95, 24 core 48 thread >>>>> Zen >>>>> 2, >>>>> zfs pool on SSD connected via 12 Gbps SAS with cache on NVME, 160 GB >>>>> RAM. >>>>> >>>> >>>> what does "sysctl vfs.vnode" say >>>> >>>> >>>> -- >>>> Mateusz Guzik >>> >>> With the builds now complete, >>> >>> vfs.vnode.vnlru.uma_reclaim_calls: 0 >>> vfs.vnode.vnlru.kicks: 0 >>> vfs.vnode.vnlru.max_free_per_call: 10000 >>> vfs.vnode.vnlru.failed_runs: 0 >>> vfs.vnode.vnlru.direct_recycles_free: 152871194 >>> vfs.vnode.vnlru.recycles_free: 77190881 >>> vfs.vnode.vnlru.recycles: 0 >>> vfs.vnode.stats.alloc_sleeps: 0 >>> vfs.vnode.stats.free: 1269702 >>> vfs.vnode.stats.skipped_requeues: 9375158 >>> vfs.vnode.stats.created: 235594976 >>> vfs.vnode.stats.count: 1407640 >>> vfs.vnode.param.wantfree: 681059 >>> vfs.vnode.param.limit: 2724237 >>> >>> Uptime is 11 days, mostly with low I/O rates. >>> >>> >> >> please add the uma stats: "sysctl vm.uma.VNODE" >> >> -- >> Mateusz Guzik > > vm.uma.VNODE.stats.xdomain: 0 > vm.uma.VNODE.stats.fails: 0 > vm.uma.VNODE.stats.frees: 245129082 > vm.uma.VNODE.stats.allocs: 246536724 > vm.uma.VNODE.stats.current: 1407642 > vm.uma.VNODE.domain.0.timin: 215 > vm.uma.VNODE.domain.0.limin: 4613 > vm.uma.VNODE.domain.0.wss: 10080 > vm.uma.VNODE.domain.0.bimin: 391986 > vm.uma.VNODE.domain.0.imin: 391986 > vm.uma.VNODE.domain.0.imax: 391986 > vm.uma.VNODE.domain.0.nitems: 391986 > vm.uma.VNODE.limit.bucket_max: 18446744073709551615 > vm.uma.VNODE.limit.sleeps: 0 > vm.uma.VNODE.limit.sleepers: 0 > vm.uma.VNODE.limit.max_items: 0 > vm.uma.VNODE.limit.items: 0 > vm.uma.VNODE.keg.domain.0.free_slabs: 0 > vm.uma.VNODE.keg.domain.0.free_items: 324187 > vm.uma.VNODE.keg.domain.0.pages: 266134 > vm.uma.VNODE.keg.efficiency: 95 > vm.uma.VNODE.keg.reserve: 0 > vm.uma.VNODE.keg.align: 7 > vm.uma.VNODE.keg.ipers: 8 > vm.uma.VNODE.keg.ppera: 1 > vm.uma.VNODE.keg.rsize: 488 > vm.uma.VNODE.keg.name: VNODE > vm.uma.VNODE.bucket_size_max: 254 > vm.uma.VNODE.bucket_size: 88 > vm.uma.VNODE.flags: 0xd0000 > vm.uma.VNODE.size: 488 > > > Ok, this is mostly what I was looking for: > vm.uma.VNODE.bucket_size: 88 That and vnlru being active. That codepath which appeared stalled may very well have been making progress. There is an API deficiency in UMA where it calls the constructor/destructor for each object separately. Meaning in this case you got 88 back-to-back lock acquires. Thrown in other CPUs into the mix which try to do the same thing (or relinquish vnode memory) vs vnlru you get the cost of waiting on it multiplied big time. Now vnlru scans happen more frequently than they need to and may hold up the lock for more than justified, but key point is that this problem is artificially exacerbated * 88 for each CPU participating. I'm going to hack something up about it. -- Mateusz Guzik From nobody Tue Oct 31 17:09:17 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKc7c60Zvz4ydC1 for ; Tue, 31 Oct 2023 17:09:32 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKc7b367cz3flf for ; Tue, 31 Oct 2023 17:09:31 +0000 (UTC) (envelope-from alexleidingerde@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=XhgrqWno; spf=pass (mx1.freebsd.org: domain of alexleidingerde@gmail.com designates 2a00:1450:4864:20::530 as permitted sender) smtp.mailfrom=alexleidingerde@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x530.google.com with SMTP id 4fb4d7f45d1cf-53b32dca0bfso68815a12.0 for ; Tue, 31 Oct 2023 10:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698772168; x=1699376968; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=UWPwT47eSubSmEFa4OWRIo/YDelZv/5Y2lZNACB5bjU=; b=XhgrqWno71JJwnrAzuwL1RfjvEMx3cIp7Cc9cVPZejohyXjYDMJMvse+iKiP+aR8SX UJpdz/7xJPS6ZQSnkSM9gWFniUJmKjD7gSBHlDCabMNg04xqlgYk1W752rpIyiBuJxOI yuBSgs8m1TOFeYDdcMR3vn4XlZT3GFv8aNvKw5CvYB3gPlwB8Khx4g28N74SuWPS2NUk tYo0xx1Xygfkz8y6p9dK8kpLLzDNIMyWjI0VcPrRwp9PgBvXy58J6usLxMsW3BirEFph Gr3Up7guprx0rHSWxmqP1R3YDBURvFV+LhsP2LGfrzlwhO5j+Z7nmEphpQLtC/6jkMgh 4JEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698772168; x=1699376968; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UWPwT47eSubSmEFa4OWRIo/YDelZv/5Y2lZNACB5bjU=; b=iyDDAKiDqkGiYL8RVxqMn9PaskikgUmxVi+5Qygl1hS53HTb5aTq0yjbmOQxuZSErA aduL67uAc19mORc12XQrx32OZMeQvWIaKb3GpkpPONx47veDKyU+4Xkx5Alg6hW+odOP LQxLSG1Gsp8AZIcg+rb43qMTXShYUPpgj9ztNZIjgsZt0m8zRN4Y4u4us2qxGNqQbZOf +9j7y80vMWolQGgEdpxEMugssVxIkmYgB2MAQHFr6QzKwo/m+XBY3XYb3F3vDCvlTNXf JWhYAAww8ehSriSxG5lsxlP5F6mzhafg5/yquFR8vsQT8rYlZ+BQp3Jvg0oUvMGso7NU 9B8A== X-Gm-Message-State: AOJu0YxKTVy6gHHed0FOEIoDgOtBg7OFiomUeRyoFDNzsxkyQLH84wwV jwvvz7LM6e9eNi/39zkdYo8z9rieINRoFagvGsKA2k3k0ceT9Af0 X-Google-Smtp-Source: AGHT+IFINpSa3H4LBlmmDasIl6uwMtqWCZlfcnFhzrejGJ7+HGiC5npW5ZrhJVeQ6lihDb/WxUupF/50GZIi+Y+TrcM= X-Received: by 2002:a50:c31e:0:b0:543:5aa2:b287 with SMTP id a30-20020a50c31e000000b005435aa2b287mr2960769edb.20.1698772168476; Tue, 31 Oct 2023 10:09:28 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alexander Leidinger Date: Tue, 31 Oct 2023 18:09:17 +0100 Message-ID: Subject: Re: ZFS txg rollback: expected timeframe? To: freebsd-fs@freebsd.org Content-Type: multipart/alternative; boundary="000000000000a80f3b06090637bf" X-Spamd-Result: default: False [-2.90 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.90)[-0.904]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::530:from]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SKc7b367cz3flf X-Spamd-Bar: -- --000000000000a80f3b06090637bf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 31, 2023 at 11:16=E2=80=AFAM Alexander Leidinger < alexleidingerde@gmail.com> wrote: > > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 > setup) in a way that a normal import of the pool panics the machine with > "VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867360 =3D=3D 10186= 7222)". > ... > After a while I get "Panic String: deadlres_td_sleep_q: possible deadlock > detected for 0xfffffe023ae831e0 (l2arc_feed_thread), blocked for 1802817 > ticks" during such an import and I had to set > 3rd panic string during "import ... -T ...": VERIFY3(tx->tx_threads =3D=3D= 2) failed (0 =3D=3D 2) Bye, Alexander. --000000000000a80f3b06090637bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, Oct 31, 2023 at 11:16=E2=80=AFAM Alexander Leidinger <= ;alexleidingerde@gmail.com= > wrote:

Issue: a overheating CPU may have corrupted a zpool (= 4 * 4TB in raidz2 setup) in a way that a normal import of the pool panics t= he machine with "VERIFY3(l->blk_birth =3D=3D r->blk_birth) faile= d (101867360 =3D=3D 101867222)".
...
After a while I get "Panic String: deadlres_td_sleep_q: = possible deadlock detected for 0xfffffe023ae831e0 (l2arc_feed_thread), bloc= ked for 1802817 ticks" during such an import and I had to set
=C2=A0
3rd panic string during "import = ... -T ...":=C2=A0 VERIFY3(tx->tx_threads =3D=3D 2) failed (0 =3D= =3D 2)

Bye,
Alexander.
--000000000000a80f3b06090637bf-- From nobody Tue Oct 31 17:18:19 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKcKs6szjz4ydQQ for ; Tue, 31 Oct 2023 17:18:25 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKcKr1Fr1z4Cwg for ; Tue, 31 Oct 2023 17:18:24 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=XSvvxEkq; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=PRAPmetA; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 9243B3200A93 for ; Tue, 31 Oct 2023 13:18:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 31 Oct 2023 13:18:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698772702; x=1698859102; bh=y7 eqnJYtOsotvBMvVFcAmDRmn3aNkTJqoyjnMNVBy+4=; b=XSvvxEkqXGnCFOJhQa cZX+atcNDMHkBbAlpoYVp02TzjJzHvE8Jw/MyVi9WdtFLNL/YNRSvgZy4caNdOQU 6VBlghbe/1g2yJK3wYaJ+QI3vVnGnx5CuCfFAOOxHS5fvYlgel3HIIso542CQOp8 /muVLrWPGey1WWDYJ7StoeK/tOUhuBJkI5CUyWg5VJF2L+1H3Gh9QFOzScdxygcT DfZPlAnPbA79SAgij3Mfezz8PSUILstFrjEe82ohBo2kJqWNQPaOJkWZfDY5izJZ cd8o0K4eprAyQiYKeQX148KwOYSYKR0WcwPlf4gEWin97zGcpCbB8bz8Qgjdi7eB wzqg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698772702; x=1698859102; bh=y7eqnJYtOsotv BMvVFcAmDRmn3aNkTJqoyjnMNVBy+4=; b=PRAPmetA+8dEcd84GECh083iYo0iR 2+pI1wa0ikl/J5wgaLrCgyd+IHUxtgy3k4+Ojs9piAKFGfXKLHpmbyOo7V9F1vta fP9og/SNoLY3Mimngx2qkhL09VQjZyntN/l3i3ehCyuZXly3kY6MUqPxGkdR2/+6 quNahrnsU6NBN+ufXrhHZ3RxIzwAuUVTstQw5VIHXWrW+QQqcP6eeabmqVMRyATW CUV3ipcA7/VdRI09TDRQcmKFKf3ZzziG8VjghJFS6LvlZxWWxJvK8r8f6qfTTlkW PW1kmSPNghBk2cdRwctq1A1EM9k2Wje6rIFVySA2CrNWPMzqkj4oVOwDw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtvddgleejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 31 Oct 2023 13:18:21 -0400 (EDT) Date: Tue, 31 Oct 2023 17:18:19 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: optimising nfs and nfsd Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-4.80 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(-0.20)[64.147.123.25:from]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.25]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.25:from]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_FROM(0.00)[f-m.fm]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SKcKr1Fr1z4Cwg X-Spamd-Bar: ---- On Mon, Oct 30, 2023 at 05:31:50PM -0700, Rick Macklem wrote: >Well, here's a couple more things to look at: >- Number of nfsd threads. I prefer to set the min/max to the same > value (which is what the "-n" option on nfsd does). Then, after > the server has been running for a while in production load, I do: > # ps axHl | fgrep nfsd > and I look to see how many of the threads have a TIME of > 0:00.00. (These are extra tthreads that are not needed.) > If there is a moderate number of these, I consider it aok. > If there are none of these, more could improve NFS performance. > If there are lots of these, the number can be decreased, but they > don't result in much overhead, so I err on the large # side. > - If you have min set to less than max, the above trick doesn't > work, but I'd say that if the command shows the max# of threads, > it could be increased. >This number can be configured via options on the nfsd command line. >If you aren't running nfsd in a jail, you can also fiddle with them via >the sysctls: >vfs.nfsd.minthreads >vfs.nfsd.maxthreads root@storage:/root# ps axHl | ug -e server | ug -e "0:00.00" | ug -e rpcsvc | wc -l 26 How many is a moderate number? Right now there's three clients connected using version 3, one of the clients is doing a lot of (small) i/o root@storage:/root# ug nfs /etc/rc.conf* /etc/rc.conf 19: nfs_server_enable="YES" I've not specified -n anywhere, so all defaults, nothing in /etc/exports. All nfsd capability is via the sharenfs zfs property >The caveat is that, if the NFS server is also doing other things, >increasing the number of nfsd threads can result in nfsd "hogging" >the system. NFSD is the machine's only role, so am concerned only with I guess high-availability/responsiveness, and tuning, if it needs it so it functions best with the hardware. The cpu on the nfs server is a Xeon E5-2407 @2.2GHz. HT is disabled, so 4 cpus available, and there's 64GB RAM. I can't do much about the CPU apart from enabling HT which might not be an issue as there's no routed ipv4. I can increase the RAM if required though. This top(1) output is fairly typical: last pid: 55448; load averages: 0.04, 0.12, 0.14 up 12+10:20:03 14:55:45 54 processes: 1 running, 53 sleeping CPU: 0.1% user, 0.0% nice, 4.0% system, 0.1% interrupt, 95.8% idle Mem: 96K Active, 879M Inact, 182M Laundry, 59G Wired, 1965M Free ARC: 54G Total, 21G MFU, 31G MRU, 664M Header, 1362M Other 50G Compressed, 78G Uncompressed, 1.56:1 Ratio Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 82935 root 32 -60 0 12M 2276K rpcsvc 0 113:03 4.22% nfsd I could probably set vfs.zfs.arc.max. Right now it's at the default 0 = no maximum. >NFSv4 server hash table sizes: >Run "nfsstat -E -s" on the server after it has been up under production >load for a while. it seems v4 isn't running or able to run. The client gets an error if I try to specify vers=4 in the sharenfs property on the server: zfs set sharenfs="vers=4 maproot=root -alldirs -network 192.168.1.0 -mask 255.255.255.0" zroot/vms-volumes then "service mountd restart && service nfsd restart" then, on the client # mount /vms-volumes [tcp] 192.168.1.102:/zroot/vms-volumes: Permission denied removing the vers=4 then restart rpcbind & nfsd on the server, it works fine, but version 3. >Look at the section near the end called "Server:". >The number under "Clients" should be roughly the number of client >systems that have NFSv4 mounts against the server. yep that shows 0 as expected as no v4. Here's the stats with just -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 85998 942 176996 0 8661219 33713618 0 898 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 9 0 0 15 43 62 55 23111 Mknod Fsstat Fsinfo PathConf Commit 0 191 39 42 111088 Server Write WriteOps WriteRPC Opsaved 33713618 33713618 0 Server Cache Inprog Non-Idem Misses 0 0 42775262 thanks again for your help and advice, I'm taking notes :D The bit following >The two tunables: >vfs.nfsd.clienthashsize >vfs.nfsd.sessionhashsize I'll reply to in a followup email -- From nobody Tue Oct 31 20:34:46 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKhhh0JFYz4yrWh for ; Tue, 31 Oct 2023 20:35:00 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKhhg1ffKz4SmR for ; Tue, 31 Oct 2023 20:34:59 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=EUbzjv49; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1034 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-2800f7c8125so156699a91.1 for ; Tue, 31 Oct 2023 13:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698784497; x=1699389297; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=L13QB782fbIW86zXVH4oxIhAuGHcXG82TH+cU3i43Ec=; b=EUbzjv49JDkUSV/macCeuHgQkAieL1wNVf1RPbOR03MLrWwZccdl5lsqg8Qv50ZM5o 2ipD0gbPzUfgyKdpVkf23+HXyOlOmhAbPtIbV7+eDpMdccVyJPRtlNkVPoUeAW9IShPG CQVnQQnPdH9mfRNtW2TKyCQd2D0+Ra8B+xA7PdGxYtElkiiSfR2922X5JwmrGnVayxUg 50yB9bYbFh8aVXwd+unTU47ZeBZYhZuj+jGJmrjrfmdizgQOwd7To4EAqziRqreYFZQ8 ughssOBYPUBPiErIwR6wvwaWsU2B+yGewj/XMZoV0PsAvEj41dAa5WqZd7j6Hglt0N+n fWBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698784497; x=1699389297; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L13QB782fbIW86zXVH4oxIhAuGHcXG82TH+cU3i43Ec=; b=ekOLnTpVQspd+B2l5raImi+dMxuGonlGx9w2tr5pT7CRXv9WWneQPMwquKMzhurAg+ svwuldo0oj6JX401PosNJ32Cou5PMQC57B/UH9Yee2nlBHpej+hryvLGWiY0URdTXz7k lp1kip98wsyxTMqdrEkC5DHas5Z9g4UN4kLDn/BvK3hDt5wdGKmUMAww6ooKfyat6TxD pjMt8jfAmoFdOlVhTwFL+CYT85rvwo2bmRIl1qdhg0jisH4BfMUQRkQ1Ma55wCFTKZnI 7WAyFqeMCtuNdAZ7LMIjsJzKL5IT8RFMWXymxAWx8d9CYwVv8tQ9/iWGXE3XKZqPwdMA eezg== X-Gm-Message-State: AOJu0YwO3C3hAGKu+PTf2/6w4Ca3z0MsW+5f0csg/kTxeqQbxmSksVZO IyBFMskHXAKS0mA2SAvBBmWTQef5Ad6/GEEerdkxlwqJcw== X-Google-Smtp-Source: AGHT+IFhjfL3NnLZuUaPLS8RehTriRSN78oKa/oruitXW1dbUhpvJbJcjJdZF77Z7W/e2TZpq9cGCs5+6CvE+tvRxdU= X-Received: by 2002:a17:90b:4d8e:b0:280:72b:397d with SMTP id oj14-20020a17090b4d8e00b00280072b397dmr850559pjb.20.1698784497601; Tue, 31 Oct 2023 13:34:57 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> In-Reply-To: From: Rick Macklem Date: Tue, 31 Oct 2023 13:34:46 -0700 Message-ID: Subject: Re: optimising nfs and nfsd To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.990]; NEURAL_HAM_LONG(-0.98)[-0.981]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1034:from]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SKhhg1ffKz4SmR X-Spamd-Bar: --- On Tue, Oct 31, 2023 at 10:18=E2=80=AFAM void wrote: > > On Mon, Oct 30, 2023 at 05:31:50PM -0700, Rick Macklem wrote: > > >Well, here's a couple more things to look at: > >- Number of nfsd threads. I prefer to set the min/max to the same > > value (which is what the "-n" option on nfsd does). Then, after > > the server has been running for a while in production load, I do: > > # ps axHl | fgrep nfsd > > and I look to see how many of the threads have a TIME of > > 0:00.00. (These are extra tthreads that are not needed.) > > If there is a moderate number of these, I consider it aok. > > If there are none of these, more could improve NFS performance. > > If there are lots of these, the number can be decreased, but they > > don't result in much overhead, so I err on the large # side. > > - If you have min set to less than max, the above trick doesn't > > work, but I'd say that if the command shows the max# of threads, > > it could be increased. > >This number can be configured via options on the nfsd command line. > >If you aren't running nfsd in a jail, you can also fiddle with them via > >the sysctls: > >vfs.nfsd.minthreads > >vfs.nfsd.maxthreads > > root@storage:/root# ps axHl | ug -e server | ug -e "0:00.00" | ug -e rpcs= vc | wc -l > 26 > > How many is a moderate number? Right now there's three clients connected > using version 3, one of the clients is doing a lot of (small) i/o 26 that are doing nothing should be more than enough. (I probably would consider 8 a moderate number.) > > root@storage:/root# ug nfs /etc/rc.conf* > /etc/rc.conf > 19: nfs_server_enable=3D"YES" > > I've not specified -n anywhere, so all defaults, nothing in /etc/exports. > All nfsd capability is via the sharenfs zfs property I think the default is 8/core. Anyhow, you seem to have enough. > > >The caveat is that, if the NFS server is also doing other things, > >increasing the number of nfsd threads can result in nfsd "hogging" > >the system. > > NFSD is the machine's only role, so am concerned only with I guess > high-availability/responsiveness, and tuning, if it needs it so it > functions best with the hardware. > > The cpu on the nfs server is a Xeon E5-2407 @2.2GHz. HT is disabled, > so 4 cpus available, and there's 64GB RAM. I can't do much about the CPU > apart from enabling HT which might not be an issue as there's no > routed ipv4. I can increase the RAM if required though. > > This top(1) output is fairly typical: > > last pid: 55448; load averages: 0.04, 0.12, 0.14 up 12+10:20:03 14:5= 5:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.1% user, 0.0% nice, 4.0% system, 0.1% interrupt, 95.8% idle > Mem: 96K Active, 879M Inact, 182M Laundry, 59G Wired, 1965M Free > ARC: 54G Total, 21G MFU, 31G MRU, 664M Header, 1362M Other > 50G Compressed, 78G Uncompressed, 1.56:1 Ratio > Swap: 8192M Total, 8192M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU C= OMMAND > 82935 root 32 -60 0 12M 2276K rpcsvc 0 113:03 4.22%= nfsd > > I could probably set vfs.zfs.arc.max. Right now it's at the default 0 =3D= no maximum. > > >NFSv4 server hash table sizes: > >Run "nfsstat -E -s" on the server after it has been up under production > >load for a while. > > it seems v4 isn't running or able to run. The client gets an error > if I try to specify vers=3D4 in the sharenfs property on the server: You need: nfsv4_server_enable=3D"YES" in /etc/rc.conf. > > zfs set sharenfs=3D"vers=3D4 maproot=3Droot -alldirs -network 192.168.1.0 > -mask 255.255.255.0" zroot/vms-volumes > > then "service mountd restart && service nfsd restart" > > then, on the client > # mount /vms-volumes > [tcp] 192.168.1.102:/zroot/vms-volumes: Permission denied > > removing the vers=3D4 then restart rpcbind & nfsd on the server, it works= fine, > but version 3. You won't get better performance from NFSv4, so if NFSv3 is working for you= , I don't see a rason to change. > > >Look at the section near the end called "Server:". > >The number under "Clients" should be roughly the number of client > >systems that have NFSv4 mounts against the server. > > yep that shows 0 as expected as no v4. Here's the stats with just -s > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 85998 942 176996 0 8661219 33713618 0 898 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 9 0 0 15 43 62 55 23111 > Mknod Fsstat Fsinfo PathConf Commit > 0 191 39 42 111088 > > Server Write > WriteOps WriteRPC Opsaved > 33713618 33713618 0 > Server Cache > Inprog Non-Idem Misses > 0 0 42775262 > > thanks again for your help and advice, I'm taking notes :D > > The bit following > >The two tunables: > >vfs.nfsd.clienthashsize > >vfs.nfsd.sessionhashsize > > I'll reply to in a followup email None of these matter for NFSv3. rick > > -- > From nobody Thu Nov 2 22:04:40 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SLybD41Ldz4yTRy for ; Thu, 2 Nov 2023 22:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SLybD2Y7fz4Yt4 for ; Thu, 2 Nov 2023 22:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698962680; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BPqi4obOFAQOdPRt6+rKaKgCXnvQkxcAew6xgRiZkwE=; b=JsbbXdqHYzspo6MgOr/5xRhAxFOZJXZuEmNhny/ef/N3s04eGXMIaSP7gKf1Fs7vNG7eE4 Y1DKb+FspAdLRsF7+nvTs8Lz5qv4bmi9LbStOep3tzPV+DJb6uFkPBbMvPTme8VuuIU/XT ztriuLxwL5BtRBRL9vHbAEeO+w/shofDNJP5gS9o30ocXcU9IrIH/D3b1SK53J9WCiiCaB ra8EWchE2D6DOQYy73iLc1x4LXbtd2KF0YfS4plTuh1zfADjslF2ycdUIWbwewiR+na2+s +yAPi0msppWLS3Zc/e96NNrfhr+VXqrH4p8rrzoqsLPlzwl/m59cyRQIrW1AHw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698962680; a=rsa-sha256; cv=none; b=o+gWliL0tE7nlfcaJJF4dkAAxPL96TY4VcBCJXX9Gy4SFZQBL9jkBu+vqVNM3KYFCj5hs8 IHHRh/fR8cFnwUyArjaDY8JxdkfRz2KHgudgZTpyLS2Uzgx9I0P7a1msR6nTpeklQzXFk8 m7XtGOGv4FIxWayabPKv/6qx6Pj55Uo8ot5jixN2eu3/pFIKgyIZkgNr4sZKbfRmeeId7Y L3aL6eSJSJ5qY9CxMvKIuEV6Ldd7L+nLPCF1sndcXUBSxbYypaoL8WeEYx3d6Wvk7ZBdZW A/QWQmQJhr7qhYVbDFbWk+Mr89YLFVcRe/c2ffuey+9ReP512SNG4YN5nJnNfQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SLybD1cmCzmN4 for ; Thu, 2 Nov 2023 22:04:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3A2M4e0Z022600 for ; Thu, 2 Nov 2023 22:04:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3A2M4eCc022599 for fs@FreeBSD.org; Thu, 2 Nov 2023 22:04:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274346] kernel panic/page fault in nfs_commonkrpc.c::newnfs_request(), due to duplicate hostid's Date: Thu, 02 Nov 2023 22:04:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: mfc-stable13? mfc-stable14? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274346 --- Comment #6 from commit-hook@FreeBSD.org --- A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D098273e649c647d5472d518c5023477ad= 15b7c3f commit 098273e649c647d5472d518c5023477ad15b7c3f Author: Rick Macklem AuthorDate: 2023-10-18 02:40:23 +0000 Commit: Rick Macklem CommitDate: 2023-11-02 22:02:22 +0000 nfsd: Fix a server crash PR#274346 reports a crash which appears to be caused by a NULL default session being destroyed. This patch should avoid the crash. PR: 274346 (cherry picked from commit db7257ef972ed75e33929d39fd791d3699b53c63) sys/fs/nfs/nfs_commonkrpc.c | 9 +++++++++ sys/fs/nfs/nfs_commonsubs.c | 6 ++++-- 2 files changed, 13 insertions(+), 2 deletions(-) --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Thu Nov 2 23:36:57 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SM0dj3xQjz4yqQY for ; Thu, 2 Nov 2023 23:36:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SM0dj2vrtz3J0t for ; Thu, 2 Nov 2023 23:36:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698968217; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GkgwK7ydeYEyLO1uwdy5Ko8xc0vkLh3enqFaCCQhyxM=; b=BUfHwSwwgS//5Tf4PrompC53QMm/75LRfsiBQhtu+RjHF5H/f6lnTWx0CAsN9HlJ8imMKv igugJSAfqlCNQQ8A8NgCtW07zHrL0LOi8mx6jphzhhZMYwCO8W1mv/9eYUwyeeuQXDKmty 1KHdQNAn/VUMg89ODKxWAWsOE6tppBq7mRGZAsp1vfuum8q84IbVKcoqSgpFzbgf43YMF7 hpqhMFMmxG2GsP9zyDB3JrpXq6F7ezqbVK/X9M1Qp1Gvg0gL6MKnLXOR1rCtNNMP/0wTGC qUngW40NVo6jI3H98eefO955KhbnearTPBbKOkvaJ9SzwvjQOR8+03Ia+ZojRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698968217; a=rsa-sha256; cv=none; b=UZPt5SnsHZJs9H+dFiwG227Vp7DsWAdWstpsTwiXsthJJe6aNj3EI2jMqe2d22ApqIFOJd i0d0PxOck2UsqiDCy7sebsXK3lWqklf8xz+W1cXe9g8GYG/5sfRxuZZtPC3rdXGGVSC/mk vmmxxJEsFN8LfEDmWTqsKBvWkAmE7On56o1owHnS3i6pidTLzEvNQMtgeJ+cjLnpKqbIlv eXA06+ErhTKyZlhdlkHNSGjcT48GO1P7lroqObpNSHQ7UkH6tpAVMaVs59eFXCRMBkxEpx 6JA2sxRVpe6Ov5+5IjUXzeIsD37sDKFKPUkNpGWW53GwROJkp2XMqxNm1k7KRw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SM0dj1vcYzp6B for ; Thu, 2 Nov 2023 23:36:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3A2Navec052337 for ; Thu, 2 Nov 2023 23:36:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3A2NavRD052335 for fs@FreeBSD.org; Thu, 2 Nov 2023 23:36:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274346] kernel panic/page fault in nfs_commonkrpc.c::newnfs_request(), due to duplicate hostid's Date: Thu, 02 Nov 2023 23:36:57 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: mfc-stable13? mfc-stable14? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274346 --- Comment #7 from commit-hook@FreeBSD.org --- A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D18d51c3c305f233a75fc64f8e5711306d= d05a8fc commit 18d51c3c305f233a75fc64f8e5711306dd05a8fc Author: Rick Macklem AuthorDate: 2023-10-18 02:40:23 +0000 Commit: Rick Macklem CommitDate: 2023-11-02 23:35:25 +0000 nfsd: Fix a server crash PR#274346 reports a crash which appears to be caused by a NULL default session being destroyed. This patch should avoid the crash. PR: 274346 (cherry picked from commit db7257ef972ed75e33929d39fd791d3699b53c63) sys/fs/nfs/nfs_commonkrpc.c | 9 +++++++++ sys/fs/nfs/nfs_commonsubs.c | 6 ++++-- 2 files changed, 13 insertions(+), 2 deletions(-) --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Thu Nov 2 23:51:38 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SM0yf6MbBz4yt5W for ; Thu, 2 Nov 2023 23:51:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SM0yf5MnGz3LJ9 for ; Thu, 2 Nov 2023 23:51:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698969098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q/B0kIwLO6DUF+wN+JnmkZSKnsG4OjqvvRMWIy/u2rs=; b=gUhgWf4XDneJAyaRp7crmwdodnSwwEMLJmxB0YW9JMl9mMIm7u32dDSxdB/WVgUL25yNXb NvuEP+QaKMeCnXvWgrHayZ6I0/MbqEGt5iZgyx6e1CWP0e9u9z9tk9I6hN5b8KsDRrN7Gc MhDvEBU6N2y9vzutoolt6ZYgNgpgR8R6W/C0MGPuDH5EvA/3ezUxR8KMm5aoizAxp/gv15 pba+BbToOd17+vEAiPoiLL99IEn7KzAVBdUKFG2gAiTXTnHCzY5FQBDIShPwEiJWxSEdVn r8zYhRQrv4gIpn4Sjui8zMKQwJTtXSYBdMJXn9XlkmKLe0b69XLqv++wNTnteg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698969098; a=rsa-sha256; cv=none; b=Cfjv3aZZQGdjvC6SfYGzm6DSIxvFSitBKDJBWQs6Y69MNr1AgZc83EJ+LjFqBIYoOApenh IKNCciThwI5iIe7Og8TInVw2VaKaNwVNudWaRTV/ImXnTgyizPm4yWYoP9hhmE+2dFIs0w 7/QwjEp3ev8ihG2q0Go2wjCXH71hDlJTCyyaSPKVXncPn06nimGhu1Ch0B1y6LQsWl/dZq vZixpWGhaYZqKyGBfYCQtEPxQ7IcECmpdbcKHcnypx6rcRAgNx3I35qz2Lc0wEnhU0IEeo Smrokya0x2thDgsNq7ZAXUNzQ32lvalIKwxDo20l/fQhToWPX0neZEygjojmRQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SM0yf4S3dzqLh for ; Thu, 2 Nov 2023 23:51:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3A2NpcFt075017 for ; Thu, 2 Nov 2023 23:51:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3A2NpcLT075015 for fs@FreeBSD.org; Thu, 2 Nov 2023 23:51:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274346] kernel panic/page fault in nfs_commonkrpc.c::newnfs_request(), due to duplicate hostid's Date: Thu, 02 Nov 2023 23:51:38 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: mfc-stable13+ mfc-stable14+ X-Bugzilla-Changed-Fields: flagtypes.name resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274346 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|mfc-stable13?, |mfc-stable13+, |mfc-stable14? |mfc-stable14+ Resolution|--- |FIXED Status|In Progress |Closed --- Comment #8 from Rick Macklem --- The patch has been MFC'd. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Fri Nov 3 00:37:12 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SM1zD3Nssz502Xq for ; Fri, 3 Nov 2023 00:37:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SM1zD12wXz3SVp for ; Fri, 3 Nov 2023 00:37:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698971832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OGrLBCHu7NX1z3sqotF7k7gFihCjMR3DH0Ua/9Uy2x0=; b=sshy8fgyXLyWLO9rWQY8oj/TCrIn4x9YVvmItDafrk3YYtMqBom//cf7sDi6zwje9lRF5M HjRsat0WW4XEzkpp+XhGUTl3ig9+12hPHeGR6D+ILcY2IOBu5s+nN/Jviro+bbkVn3evw6 xkMoShW/PtnEB8X8GfQuFcx/iEm5JlPLB5CGZmkJ02pk7Q2HcmtNljk5IJOXEulfwGUYKU pdlifjZTC/iydehdUpthO+Do5pV0assWMVuUbmDEYoU+gOiv7redQsVvXKW5O7w7/ROJBz hDhDYQ+vDqqHetq8HydY+kLX9Z9iJJK/ycjClnJ3g2B7DUFpGT8hTZX9zD0Q/g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698971832; a=rsa-sha256; cv=none; b=gp9c0qoHUwF8rgWEjexMp4cYlIpwFNQPeOgV9X/vARLmZOeNuimGHgyElsfPxZUoeYDbcS nmBrAQ3PBShMQ3w7Gaiiygieq6xIQXyOWsQ9suIQgNJwaZquoCEV2Kh4KT8GRoW7Kvd0uX tudUOMJKSM1ORltI/Z3Er1sgKbkqAcGFlTtAFZl/kqSiNK1NZkpCyvofv7c6L9SVMJllzq I85oUC+RjmiLshk98TcTPvFST8XqD9C7UG7xHaOJi7TOPZVoTHioamBU4+y1YqH9DuHp81 Ca7zZSliwMqpfKsZ/lqai1K6R8Nh4CwlNAcvpUWy/zn6AtN+UrXZ1RYjGBjDCw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SM1zD06smzrZk for ; Fri, 3 Nov 2023 00:37:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3A30bBWR050118 for ; Fri, 3 Nov 2023 00:37:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3A30bBFG050117 for fs@FreeBSD.org; Fri, 3 Nov 2023 00:37:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274878] ZFS: sysctl "kstat.zfs.POOLNAME.dataset.objset-0xXXXX.dataset_name" not updated when renaming zvols Date: Fri, 03 Nov 2023 00:37:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274878 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Nov 3 15:19:11 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMPXv40lgz4yTtr for ; Fri, 3 Nov 2023 15:19:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMPXv0s2nz3Zvd for ; Fri, 3 Nov 2023 15:19:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699024751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NhbR9b96J4hOEd5xZLRCboL/17q35cKObWEvpkHOzAU=; b=qWrGSrqO/HD34xRng8ZICVlkojZEyLH3cQHfPgNFgMwEl+kMxsP5fX1et8WTB9wgzFgQAp 0yBoa8GhJM5e0ryFLFOk8MIc6fc7oCFA6fosRLbgMmf4PB4mJwoueBg9jOxCFXKb4iQjbl 7iN4A8sOB0oovf+G/8zn/aYM0qFGicUSi+7D2KA7pku9Xk+GXm7fNdHYgT3Cs/IpWtmBvt O8+7ZC6Fvf+idqu70yqxDlxmETMZVb/V47KjZl8cR9aeCuujUwlWUD5synhzTXoM/tTebN U2rtP24I3j4q1Iqa8zVk2xASHb60wG5Zq+DQNcfBFyQPTrcAdvmYm1HKnZ5aDg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1699024751; a=rsa-sha256; cv=none; b=eTpXLjDvQ0gxVFuQaSLITwcOPtihxmHtypZipHb74oI5ChuerFwvGxeBjk3YJF0z5lH8V7 p+5IeCc00Fak/DcbqKCJD6/b5ImMXmyE0+2gUSX1ywn7Wy9P35DE1ws9cPDsEg2Gn6CtNf 3XHGbjZoPaDENwv54o0iSQLUS20TWk0rQZtBKEqZkJFv9NVx1g50zg+tVD5qnaBAP5JfX6 CSnviDK0gpw5gqWjWgu4d34kwviPYdf+EZZUhbHiOIjEtAfwCyKuWC5ibBg6vjaSF27MYk LRMR9Kv8hxgxjed8ZcWbMtsR/vdfXEecef7KfzYnwWbiOQDuX9mNp/2hodkpvw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SMPXt6wzjz34x for ; Fri, 3 Nov 2023 15:19:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3A3FJAGf093031 for ; Fri, 3 Nov 2023 15:19:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3A3FJA7x093028 for fs@FreeBSD.org; Fri, 3 Nov 2023 15:19:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274878] ZFS: sysctl "kstat.zfs.POOLNAME.dataset.objset-0xXXXX.dataset_name" not updated when renaming zvols Date: Fri, 03 Nov 2023 15:19:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: asomers@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274878 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |asomers@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 4 12:00:49 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMx5t6V9Bz509R6 for ; Sat, 4 Nov 2023 12:01:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-23.consmr.mail.gq1.yahoo.com (sonic312-23.consmr.mail.gq1.yahoo.com [98.137.69.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMx5r6vNhz3Ynh for ; Sat, 4 Nov 2023 12:01:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=bP3GT4hx; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1699099262; bh=nusVZbEIP7mIpV+w/Dlh/JQE63k1F/z3QuQrgCOx1eY=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=bP3GT4hxrSKfds8EwGn+Os4SBGadlpBCEitZeVE7MX6TuKc231WlEZmKB3sW5EZqO5aYnGrase2IyuEibap4o7HacnT1U1aIegLe4jtmdXb9lTHbcmUlsPrhs/6FzAXKcBBdtxEG5un2iXvGHZDlGABz0IbwvBibMgizgdlW6qUmFknIOIkSq7uPuQvBTrbBWC3wt+LH1XqqcKAdPQyIiVgFoopnB+090qfGLTzN4Jn1BL0LM6td3mod63NoZCgczajazQH0NoDEaCtB2uCkXSjhWZeymRFdjbcQpJA5qszHZTUCwYg9NzEiq9h6EcrvVqk5hgxLdY1fZBkHsZ16dA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1699099262; bh=fomoHiZcD/o6UKTDdkR8wPfRoRKKRSsr5koRRrfgMq2=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=S/v5RDJvz0Z1onKxCRUvqE35gpleGBcJp6x1c4WfX1YW8zS+GWO/yMBtM+lxoo7XK6Uv90rvvWcNUel9ZgHCMaaBrM8ZT6nL85iZR/3xX1Z6b1jhO1V9uEI7+S7tQNqRE0hI8dLql+qTPKTlknTPDgFP5SJtlKyzoWsrMKKmLCKZzVRnHHa+Xd/oziZPy5P2Vd0rFjRX7v5Ov6auTFyF6kPUszfj/20hHxKytDGEgNiRjToccawsrGwfb7NZOxpvvU0kTWfQipdHo6iSybySYBp+zF5zJRMOPYXpJ/OF9LYiYDo9/O7vknL8hAmkq6Z9zKJfFvdgmsFJHI8tvRpUzg== X-YMail-OSG: Yv6G.HsVM1kAS2wztmMU6Z9dwmtBXRp7y_9bwePkYGVKiPJgpdp3KDszIN0Rl0C bXqUDs4s5FBgEtWo4zGUJJ_p2QU9dSFYxFmISEdj4bAPY_tSnDOFLNUUtNPdP_5BcWa_C8N3NPBI 3BG4gVrTOoWv2_gNV2zkr05dAo1ZvkDg.V5pNiwjHGJv7JlKiN.qlfCvUPdUBipJQLwBVRSGLXW1 XpEw_V_F6FjRP8VvHyXP3YjoVBYH6aeFKemKxY15zGRURPjI2VnAdg4czwzQmGUthpB10527QdZi id6Cm_LLzccWiJPxmaBh7yxkOqn06tiQ.mp2D3xxrrC9GbqINW009aOz9N0M8dKSpiE.GE.QtdKa tPV7dy.jPfwkyEGFlCk3tjeyoDf1fPY2ziTZz_Vzcoe8wGGVHZBllhDkwQgofKackyHGTpzmfU.. qgnO8IL1o0jBk9eo6AVHKr4Vlv3iktWWtIh2Jl0oDTmDwVh81s.A169vYKa6wP7ndzFwH66RYgQX aUitlEBkBMYGyCU2Nb4vFPhaeceB0ABOOvJeBqRaPAlvbdVoqVgyrwDmAecmFwk9460EZDuJ6bAN 6OhsafGJ_qmGjPr43B4woLzUPvUB16t9_yNl84uY0wLMokuSaA4K9aBAy0vZcwZkgQ7F.RXWOzwG 4jARFew__dZo6k.qSkosXcuAHoXusOkxuy5Cv9YOdwlzMCyWgGDjiImxC72PsLHCUr1dXTahUCHs DAOl85Qm9Bg45exectTpLsWT8g7A1KUGNfm5VszQRDjLbi_nf8GovKnyGOvirODJDt67ZU9g9dvd bgtfDQwR0FadoWFjKqS.gAWnClsXmNJKN8AwXKOuivRZfnzKUE5LOLvbDQtrO0mIICqx_SYbvyzp EErkUfEx577CO5L7ToNuUKcDwmOOMoiRKRN.NSYhk4JeBglC73s.tuUVa5kC6euuEcsZBIbVY6AQ ZOghWM.VK5NPDn1n6d_WdmarsOAtAHzLcjKZY8My8vvd09Rq5ahZt6T_L.Umj3kyCkR9Tj5TP43W hqjhDt_02.RvqW.PiiOofq1Vp2kaChdQa4iL.sl75I.FHvf6sobZUcPYo2c3yTzrgWXH1dkDNBFn 7LFOsNpDOf0bO4ur1NQ9grCeBaepJmE9CecGJn00hb8kKu9CuKt1w.sC1cDlIUYnsSvHk2vxfsBr pgXz2v6RbQV.ryiI4q5ta_p8FFbQfGw1Fy84hjwt_2pWe3UEmBx0Y0rm5jEDNfbZHLBEatvaphso cqkC8KwO9kLFrkNwmEkZBq02lJouDGAd83BV2PomFydSVqEOFxfKpwuOiKG7R_M_7ScWNcDEjQQw hBqrUn5hLDgHDBPg44EAHfvbfwVtrkjjdcf8i0KEBqpojKjxWVJeCnN1u_eaSp1JSVkzEiZjeE5u 4KL6TJeNAx5kc9VO4_2nXXysUBuRiUosANJcoHCciUEv7iXQOCYj6o2Csjq.XN0kvF1Iq6fObRQR j_Rmc4MPDv0I3pUIZbOa6GUgMv1kCLQ16eTX7QICGbayRUgDkis7PpJqPJtAq6ajdmVk36xF_fFr teDKBi5kFq7NBHztQROCbotXJ3iO7L6lwtk0usxEOvCyjIQJEOomcfvJr_mQ0vpE4x4InO5LF8bO ccrewsqEItrhN5p5rxAsVDhtiM.APYZrAZHb78azEJckp8FiUlNdqmwl_xG4SAfvP5xhsAtcrwZ9 ufsBIBl9aRrbSrMyKBwVbzko5MFcWcrcT.d12ik90ETHEg9KaOQCuav2yirk5IfEdq9DtW57sp6c tA7QEPOsyAwYsCULiPUMn2lNgDmbThmuFeV6CqpGk92F8HNiHoZ.Xn08MXg8gAjfwb.D2iwzl65E sVB.2UjWdLvopGDiVR9y4SZswhBCZS7fMTmdYyMHCSd23cPR7TuP8WcC9Hk9Oq5w1bl_savOf8DC tiQO3PIvt007xRMXUbpMpKGGQXZkg7JF8176rjyyu8i06XNjL3qal0uIGE1NeLBbFlVa8XwccqZg Sk.8R8X732KuMBpcrRxFMBW6lD043lr7fSwgB.25aso4W9QpAITQiAvTwfizDxHRT2ZmVgy53ied otGRRacv2tascX03gBT9z_FpIaeBgi.xI0fqjqMGw4LVuJ2hXRQEd1OV8ucrC40E.Qtsk9CsHS4c BjnL5l0LFeT1VBBsLpL98BChrrj92CQdns4Q4qmbUmFThDoXzT3blXq8syBa0Eq7x3U_wubVuma5 UpaJfbJg_zuFPCrqfFTkZVfHnRFmiS0anxvf5qoZtlW9JqHwbm1rX593WhHanbvjTyQP3tZUFWX0 DE6l3aw-- X-Sonic-MF: X-Sonic-ID: 1455f27e-2baa-43e6-828b-89dbe6fc637b Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Sat, 4 Nov 2023 12:01:02 +0000 Received: by hermes--production-ne1-56df75844-pr5zc (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a3da3ff4ff01476dd0fb07b1c2f3b19b; Sat, 04 Nov 2023 12:01:01 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: poudriere bulk -a fails on UFS: "Too many links" under logs/bulk/latest-per-pkg/ and then "Failed: starting" Message-Id: Date: Sat, 4 Nov 2023 05:00:49 -0700 Cc: Kirk McKusick , "bapt@freebsd.org" To: Current FreeBSD , freebsd-fs@freebsd.org, FreeBSD Mailing List X-Mailer: Apple Mail (2.3774.200.91.1.1) References: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.204:from]; BLOCKLISTDE_FAIL(0.00)[98.137.69.204:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.204:from]; FREEMAIL_FROM(0.00)[yahoo.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SMx5r6vNhz3Ynh X-Spamd-Bar: --- Looks like there are too many ports for "poudiere bulk -a" to work on = UFS: more than 32767 directory files to go in the likes of = logs/bulk/latest-per-pkg/ as an example. Context: # uname -apKU you have mail FreeBSD amd64-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT #126 = main-n266130-d521abdff236-dirty: Tue Oct 24 18:17:40 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1500002 1500002 =3D> 40 2930277088 nda1 GPT (1.4T) 40 2008 - free - (1.0M) 2048 532480 1 efi (260M) 534528 1562624 - free - (763M) 2097152 2919235584 2 freebsd-ufs (1.4T) 2921332736 8944392 - free - (4.3G) # ~/fbsd-based-on-what-commit.sh -C /usr/ports 6ec8e3450b29 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/sdts++: = Mark DEPRECATED Author: Muhammad Moinur Rahman Commit: Muhammad Moinur Rahman CommitDate: 2023-10-21 19:01:38 +0000 branch: main merge-base: 6ec8e3450b29462a590d09fb0b07ed214d456bd5 merge-base: CommitDate: 2023-10-21 19:01:38 +0000 n637598 (--first-parent --count for merge-base) # ls -a /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ | wc -l 32767 =46rom the bulk -a output: . . . [30:24:09] [32] [00:00:42] Finished devel/p5-DateTime-Calendar-Hebrew | = p5-DateTime-Calendar-Hebrew-0.05_1: Success [30:24:10] [32] [00:00:00] Building dns/knot3 | knot3-3.3.1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/knot3: Too = many links [30:24:13] [23] [00:00:22] Finished sysutils/ttyload | ttyload-0.5.3_1: = Success [30:24:14] [05] [00:01:44] Finished deskutils/mytetra | mytetra-1.43.27: = Success [30:24:15] [23] [00:00:01] Building audio/komposter | = komposter-g20201211_1 [30:24:16] [05] [00:00:00] Building devel/pear-SebastianBergmann_PHPCPD = | php81-pear-SebastianBergmann_PHPCPD-2.0.0 [30:24:16] [08] [00:18:43] Finished mail/cyrus-imapd36@http | = cyrus-imapd36-http-3.6.3: Success mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/komposter: Too = many links [30:24:17] [08] [00:00:00] Building sysutils/p5-Samba-SIDhelper | = p5-Samba-SIDhelper-0.0.0_3 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/php81-pear-SebastianBer= gmann_PHPCPD: Too many links [30:24:18] [19] [00:00:38] Finished devel/p5-Test-WWW-Mechanize-CGIApp | = p5-Test-WWW-Mechanize-CGIApp-0.05_1: Success [30:24:18] [12] [00:00:24] Finished net-mgmt/p5-Mon | p5-Mon-0.11_1: = Success [30:24:19] [19] [00:00:00] Building mail/squirrelmail-plugins@php82 | = squirrelmail-plugins-php82-1.0_2 [30:24:19] [12] [00:00:00] Building security/bsmtrace | bsmtrace-1.4_1 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/p5-Samba-SIDhelper: = Too many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/squirrelmail-plugins-ph= p82: Too many links mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/bsmtrace: Too = many links [30:24:24] [17] [00:00:29] Finished devel/p5-Class-Delegation | = p5-Class-Delegation-1.9.0: Success [30:24:24] [17] [00:00:00] Building devel/py-memory-profiler@py39 | = py39-memory-profiler-0.61.0 [30:24:25] [13] [00:00:21] Finished converters/nomyso | nomyso-4.3: = Success [30:24:25] [13] [00:00:00] Building devel/rubygem-aws-sdk | = rubygem-aws-sdk-3.1.0 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/py39-memory-profiler: = Too many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/rubygem-aws-sdk: Too = many links [30:24:29] [16] [00:15:52] Finished games/sauerbraten | = sauerbraten-20201221_2: Success [30:24:30] [16] [00:00:00] Building lang/ccl | ccl-1.12_2 [30:24:30] [20] [00:09:58] Finished security/sssd@default | = sssd-1.16.5_10: Success mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ccl: Too many = links [30:24:33] [20] [00:00:00] Building games/xpired | xpired-1.22_24 [30:24:33] [18] [00:01:12] Finished irc/miau | miau-0.6.6_3: Success [30:24:34] [18] [00:00:00] Building textproc/rubygem-nokogumbo | = rubygem-nokogumbo-2.0.5_2 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/xpired: Too = many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/rubygem-nokogumbo: = Too many links [30:24:39] [22] [00:01:12] Finished net/zsync | zsync-0.6.2_2: Success [30:24:39] [22] [00:00:00] Building devel/distcc | distcc-3.4 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/distcc: Too = many links [30:24:49] [01] [00:05:11] Finished audio/myxer | myxer-1.2.1_26: = Success [30:24:50] [01] [00:00:00] Building security/theonionbox | = theonionbox-4.3.1_2 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/theonionbox: = Too many links [30:24:56] [21] [00:01:41] Finished dns/knock | knockpy-5.4.0_2: Success [30:24:57] [29] [00:01:56] Finished games/meandmyshadow | = meandmyshadow-0.5a_2: Success [30:24:58] [21] [00:00:00] Building biology/phyml | = phyml-3.3.20220408_1,1 [30:24:58] [29] [00:00:00] Building security/globalprotect-openconnect | = globalprotect-openconnect-1.4.9_1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/phyml: Too = many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/globalprotect-openconne= ct: Too many links [30:25:00] [06] [00:01:21] Finished www/mod_auth_openidc | = ap24-mod_auth_openidc-2.4.14.2: Success [30:25:00] [06] [00:00:00] Building net-mgmt/torrus | torrus-2.09_2 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/torrus: Too = many links [30:25:03] [24] [00:02:29] Finished www/dooble@qt5 | = dooble-qt5-2023.08.30: Success [30:25:03] [24] [00:00:00] Building games/openarena-oax | = openarena-oax-B51_1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/openarena-oax: = Too many links [30:25:19] [28] [00:02:11] Finished www/npm-node20 | npm-node20-10.2.0: = Success [30:25:20] [28] [00:00:00] Building editors/hexcurse | hexcurse-1.60.0 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/hexcurse: Too = many links [30:25:27] [25] [00:16:04] Finished mail/cyrus-imapd34@http | = cyrus-imapd34-http-3.4.6: Success [30:25:27] [25] [00:00:00] Building www/rustypaste-cli | = rustypaste-cli-0.8.0 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/rustypaste-cli: Too = many links [30:25:39] [10] [00:01:34] Finished misc/lightgbm | lightgbm-4.1.0: = Success [30:25:40] [10] [00:00:00] Building audio/mkcue | mkcue-1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/mkcue: Too = many links [30:26:10] [01] [30:26:10] Finished security/theonionbox | = theonionbox-4.3.1_2: Failed: starting [30:26:11] [05] [30:26:11] Finished devel/pear-SebastianBergmann_PHPCPD = | php81-pear-SebastianBergmann_PHPCPD-2.0.0: Failed: starting [30:26:11] [01] [00:00:00] Building editors/openoffice-4 | = apache-openoffice-4.1.14_3 [30:26:12] [06] [30:26:12] Finished net-mgmt/torrus | torrus-2.09_2: = Failed: starting [30:26:12] [05] [00:00:00] Building databases/pgreplay | = pgreplay-1.3.0_2 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/apache-openoffice: = Too many links [30:26:13] [08] [30:26:13] Finished sysutils/p5-Samba-SIDhelper | = p5-Samba-SIDhelper-0.0.0_3: Failed: starting [30:26:13] [06] [00:00:00] Building cad/qflow | qflow-1.4.100_1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/pgreplay: Too = many links [30:26:14] [10] [30:26:14] Finished audio/mkcue | mkcue-1: Failed: = starting [30:26:14] [08] [00:00:00] Building sysutils/mcweject | mcweject-1.1 [30:26:14] [10] [00:00:00] Building devel/pear-Net_Gearman@php83 | = php83-pear-Net_Gearman-0.2.3_3 [30:26:14] [12] [30:26:14] Finished security/bsmtrace | bsmtrace-1.4_1: = Failed: starting mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/qflow: Too = many links mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/mcweject: Too = many links [30:26:15] [13] [30:26:15] Finished devel/rubygem-aws-sdk | = rubygem-aws-sdk-3.1.0: Failed: starting [30:26:15] [12] [00:00:00] Building net/cloud-init-devel@py39 | = py39-cloud-init-devel-23.3.1 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/php83-pear-Net_Gearman:= Too many links [30:26:16] [11] [00:02:11] Finished japanese/font-plemoljp | = ja-font-plemoljp-1.6.0: Success [30:26:16] [16] [30:26:16] Finished lang/ccl | ccl-1.12_2: Failed: = starting [30:26:16] [13] [00:00:00] Building devel/py-pyupgrade@py39 | = py39-pyupgrade-3.15.0 mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/py39-cloud-init-devel: = Too many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/py39-pyupgrade: Too = many links [30:26:18] [16] [00:00:00] Building lang/frawk | frawk-0.4.7 [30:26:18] [17] [30:26:18] Finished devel/py-memory-profiler@py39 | = py39-memory-profiler-0.61.0: Failed: starting [30:26:19] [18] [30:26:19] Finished textproc/rubygem-nokogumbo | = rubygem-nokogumbo-2.0.5_2: Failed: starting [30:26:19] [17] [00:00:00] Building net/rubygem-google-cloud-spanner | = rubygem-google-cloud-spanner-2.18.1 mkdir: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/frawk: Too = many links mkdir: = /usr/local/poudriere/data/logs/bulk/latest-per-pkg/rubygem-google-cloud-sp= anner: Too many links [30:26:20] [18] [00:00:00] Building www/p5-HTML-Widgets-SelectLayers | = p5-HTML-Widgets-SelectLayers-0.07_2 [30:26:20] [19] [30:26:20] Finished mail/squirrelmail-plugins@php82 | = squirrelmail-plugins-php82-1.0_2: Failed: starting . . . # poudriere status -b [main-amd64-bulk_a-default] [2023-11-02_21h44m56s] [parallel_build:] = Queued: 34683 Built: 32285 Failed: 1038 Skipped: 366 Ignored: 320 = Fetched: 0 Tobuild: 674 Time: 30:50:53 ID TOTAL ORIGIN PKGNAME = PHASE PHASE TMPFS CPU% MEM% [29] 00:00:30 misc/swissfileknife | = swissfileknife-1.9.9.0 starting 00:00:30 1.28 GiB = [15] 00:52:19 cad/kicad-library-packages3d | = kicad-library-packages3d-7.0.2_2 package 00:48:06 16.55 = GiB 100% 0.1% [30] 00:00:30 sysutils/mac_rtprio | = mac_rtprio-kmod-g20170417 starting 00:00:30 1.28 GiB = [01] 00:00:00 net/nakenchat | = nakenchat-3.00.b1 starting 00:00:00 1.43 GiB = [16] 00:00:31 ports-mgmt/pkgs_which | = pkgs_which-0.4.1 starting 00:00:31 1.28 GiB = [31] 00:00:30 x11-themes/Kvantum | = Kvantum-qt5-1.0.10 starting 00:00:30 1.28 GiB = [02] 00:00:00 graphics/pear-Image_Barcode@php83 | = php83-pear-Image_Barcode-1.1.3 starting 00:00:00 1.43 GiB = [17] 00:00:31 games/egoboo | = egoboo-2.8.1_1,1 starting 00:00:31 1.28 GiB = [32] 00:00:30 textproc/fa-aspell | = fa-aspell-0.11.0_1,2 starting 00:00:30 1.28 GiB = [03] 00:00:00 comms/sigdigger | = sigdigger-0.3.0.1_1 starting 00:00:00 1.39 GiB = [18] 00:00:31 devel/py-pdoc@py39 | = py39-pdoc-14.1.0 starting 00:00:31 1.28 GiB = [04] 00:00:00 graphics/pecl-qrencode@php80 | = php80-pecl-qrencode-0.11 starting 00:00:00 1.31 GiB = [19] 00:00:31 devel/websvn@php81 | = websvn-php81-2.8.2_1 starting 00:00:31 1.28 GiB = [05] 00:00:00 mail/squirrelmail-quota_usage-plugin@php82 | = squirrelmail-quota_usage-plugin-php82-1.3.1_3 starting 00:00:00 1.28 GiB = [20] 00:00:31 textproc/p5-Text-DHCPLeases | = p5-Text-DHCPLeases-1.0_1 starting 00:00:31 1.28 GiB = [06] 00:00:00 textproc/html2fo | = html2fo-0.4.2 starting 00:00:00 1.29 GiB = [21] 00:00:31 graphics/partio | = partio-1.14.6_2 starting 00:00:31 1.28 GiB = [07] 00:00:00 math/py-pdal@py39 | = py39-pdal-3.0.2_2 starting 00:00:00 1.28 GiB = [22] 00:00:31 cad/cura | = Cura-4.13.1_4,2 starting 00:00:31 1.28 GiB = [08] 00:00:00 mail/mboxstats | = mboxstats-3.1 starting 00:00:00 1.28 GiB = [23] 00:00:31 textproc/p5-Text-HikiDoc | = p5-Text-HikiDoc-1.023 starting 00:00:31 1.28 GiB = [09] = crashed = [24] 00:00:31 lang/zephir@php81 | = php81-zephir-0.17.0 starting 00:00:31 1.28 GiB = [10] 00:00:32 databases/php-xapian@php82 | = php82-xapian-1.4.23 starting 00:00:32 1.28 GiB = [25] 00:00:31 security/stegify | = stegify-1.2.2_14 starting 00:00:31 1.28 GiB = [11] 00:00:32 science/py-qspin@py39 | = py39-qspin-2.3.3 starting 00:00:32 1.28 GiB = [26] 02:08:19 emulators/libretro-mame | = libretro-mame-20220124_1 build 02:06:29 3.67 GiB = 0% 0.1% [12] 00:00:32 devel/pear-PEAR_Info@php83 | = php83-pear-PEAR_Info-1.9.2_4 starting 00:00:32 1.28 GiB = [27] 00:00:30 www/moodle42@php82 | = moodle42-php82-4.2.3 starting 00:00:30 1.28 GiB = [13] 00:00:32 sysutils/luckybackup | = luckybackup-0.5.0_2 starting 00:00:32 1.28 GiB = [28] 00:00:30 news/mmail | = mmail-0.52 starting 00:00:30 1.28 GiB = [14] 00:00:32 math/giacxcas | = giacxcas-1.9.0.55 starting 00:00:32 1.28 GiB = =3D>> Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-11-02_2= 1h44m56s =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Nov 4 17:35:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SN4Xz2K4Dz4yym2 for ; Sat, 4 Nov 2023 17:36:35 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SN4Xx3TN4z3SfY; Sat, 4 Nov 2023 17:36:33 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=GXCejtz7; spf=pass (mx1.freebsd.org: domain of Alexander@Leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@Leidinger.net; dmarc=pass (policy=quarantine) header.from=leidinger.net List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1699119379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=tMR/lAdZI0U5GT68UfQoWUXEsexJn6VZgSY96+oJioU=; b=GXCejtz7oP3rJw0cNAkdbgKMNXTtOLthEsI35Dr3YCYLcr7gG+DDDaB7uV/ksSVF1/dVcy o+/uiLnGj8XtYFD5A8RkW8INez3gLA+NKdqGBFCbRAm3BF1tjjeHDkKX9sfhCM+tjv8tT2 GtTaDXPVANGNAObRyHIZpcu430mw1fAjQrnmOnrZJ2ZzQRrGE1ggQ6nMpK4l6TcKjl8ok3 IiI5z8PtiAwmTR93Sr/+bbm86+itm+SnT+K+t5yse9d5bwPoC6HEA5zrbD9rZ5T23HbVRO uD7vWOkmQpSd9XQxD+KD3QP7uu25nbraExF0FpGty0ZM6FGfht+BXe4O/cBypQ== Date: Sat, 04 Nov 2023 18:35:32 +0100 From: Alexander Leidinger To: mm@freebsd.org Cc: John F Carr , freebsd-fs@freebsd.org Subject: Re: ZFS txg rollback: expected timeframe? In-Reply-To: References: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> Message-ID: X-Sender: Alexander@Leidinger.net Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_4b2544ccaa684e053d024cb13f060892"; micalg=pgp-sha256 X-Spamd-Result: default: False [-5.05 / 15.00]; SIGNED_PGP(-2.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.954]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; TO_DN_SOME(0.00)[]; HAS_ORG_HEADER(0.00)[]; HAS_ATTACHMENT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SN4Xx3TN4z3SfY X-Spamd-Bar: ----- This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_4b2544ccaa684e053d024cb13f060892 Content-Type: multipart/alternative; boundary="=_aa60ead52848b6faf5ce93df29d0a9f0" --=_aa60ead52848b6faf5ce93df29d0a9f0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2023-10-31 15:27, schrieb Alexander Leidinger: > On Tue, Oct 31, 2023 at 1:15 PM John F Carr wrote: > >>> On Oct 31, 2023, at 06:16, Alexander Leidinger >>> wrote: >>> >>> Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in >>> raidz2 setup) in a way that a normal import of the pool panics the >>> machine with "VERIFY3(l->blk_birth == r->blk_birth) failed (101867360 >>> == 101867222)". >>> >> >> I disabled that assertion because it gives a false alarm with some >> combinaion >> of deduplication, cloning, and snapshotting on one of my systems. >> >> See >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538 >> https://github.com/openzfs/zfs/issues/11480 > > I don't have deduplication on this pool. There are clones, and > snapshots, and there could be recent ones if poudriere does some. Is it > still a false alarm in this case? If yes, you say a kernel with this > patch applied should let me import the pool without rollback? > > The github issue is from 2022, I have my doubts that this is the same > issue we see. I rather expect some issues around the copy_file_range(2) > related code for ZFS which was re-enabled 20 days ago (maybe it is > valid to remove this assert, or maybe the block cloning part needs some > tweak). CC Martin for the block cloning part. So in the end I was at least able to import the pool read-only with the patch to disable this VERIFY3 panic. After a final incremental backup I re-created the pool (with vfs.zfs.bclone_enabled=0) and restored all datasets. Now some checks (this VERIFY3 part is enabled again) and then a full backup. There is still the question what caused it. With the above report from John about some issues when dedup is enabled (which wasn't on this pool until the default of bclone_enabled was changed to 1, which is some kind of dedup internal to ZFS as far as I was understanding the description of block cloning in the openzfs ticket of block cloning) I have some reservations about enabling it again. Martin, maybe it's a good idea to disable block cloning again, until someone with the corresponding OpenZFS knowledge has investigated this... Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_aa60ead52848b6faf5ce93df29d0a9f0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

Am 2023-10-31 15:27, schrieb Alexander Leidinger:

On Tue, Oct 31, 2023 at 1:15=E2=80=AFPM John F Carr <jfc@mit.edu> wrote:<= /div>


> On Oct 31= , 2023, at 06:16, Alexander Leidinger <alexleidingerde@gmail.com> wrote:
>
> Issue: a overheating CPU may have corrupted a zpool (4 * = 4TB in raidz2 setup) in a way that a normal import of the pool panics the m= achine with "VERIFY3(l->blk_birth =3D=3D r->blk_birth) failed (101867= 360 =3D=3D 101867222)".
>

I disabled that assertion bec= ause it gives a false alarm with some combinaion
of deduplication, clo= ning, and snapshotting on one of my systems.

See

&nbs= p;https://bugs.freebsd.org/bugzil= la/show_bug.cgi?id=3D261538
 https= ://github.com/openzfs/zfs/issues/11480
 
I don't have deduplication on this pool. There are clones, and snapsho= ts, and there could be recent ones if poudriere does some. Is it still= a false alarm in this case? If yes, you say a kernel with this patch appli= ed should let me import the pool without rollback?
 
The github issue is from 2022, I have my doubts that this is the same = issue we see. I rather expect some issues around the copy_file_range(2) rel= ated code for ZFS which was re-enabled 20 days ago (maybe it is valid to re= move this assert, or maybe the block cloning part needs some tweak). CC Mar= tin for the block cloning part.

So in the end I was at least able to import the pool read-only with the = patch to disable this VERIFY3 panic. After a final incremental backup I re-= created the pool (with vfs.zfs.bclone_enabled=3D0) and restored all dataset= s. Now some checks (this VERIFY3 part is enabled again) and then a full bac= kup.

There is still the question what caused it. With the above report from J= ohn about some issues when dedup is enabled (which wasn't on this pool unti= l the default of bclone_enabled was changed to 1, which is some kind of ded= up internal to ZFS as far as I was understanding the description of block c= loning in the openzfs ticket of block cloning) I have some reservations abo= ut enabling it again.

Martin, maybe it's a good idea to disable block cloning again, until som= eone with the corresponding OpenZFS knowledge has investigated this...

Bye,
Alexander.

--
= http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http:/= /www.FreeBSD.org    n= etchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
--=_aa60ead52848b6faf5ce93df29d0a9f0-- --=_4b2544ccaa684e053d024cb13f060892 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmVGgPEACgkQEg2wmwP4 2IYAAA//TZInMtjfWk2931KSHTlTJZVSHxwlE0y9mV4kr0ood/JRp5R5hfB75f7k qzISrULuPsnf4hTHATlgoM1O3fCzmNnc8WqOwZh0fIY7yTvtPbnEXR1W1yC6bC2u CgVXN3eVdTPG+gwI/AxvYoKOj9ykV/ArbSA5oj4Tt6bIsInpCSKwG9wmUdWSU/v4 tRPLkA0vRM26PyAXj7X5LwHGZVZeR7xD1iOvdejMAsYiXgV7/T2X8PQPA7N4T1qW ZQIeFWB7XwRbg//egh5HsI0brdPYnxOcT/J+ebUV//O+/hEGKnPdVLMiqbVz7ZLz wZTNf3eu0h8WMI0Gq+Sg3xwxEAELivUGBpl5TKeaPwidxyqgJ55u9s0GZCKUALh2 S5itoP/SImPZKf9vu5JG3ma+U1Ui90+CU0MGmifJ89o1kZR+kZrRkjQQJHdmuCqa /3rwuaDAtpKaX7qRxHvLR784s28LlzHRqxllV1dcck8IsQH/tVgQXq4riiQlrCLQ dE8+TyDD+sNHfYPHJNoLHaHvudJ6NaXCkjvjTnAHt1OEuoCc6qKFVDTIdOiIZ4Ov 3l8BGECnaa1Zry+O0ZDQH+985xPM8MQQ7Rfe1gJ0b+KRFNgUUQoSwbgBZn7HL+ep YpXwiG9WmmsEemi3UkAdnT3C6iAXlY9RVY1qZoWKRVRviLuhGck= =Xm+x -----END PGP SIGNATURE----- --=_4b2544ccaa684e053d024cb13f060892-- From nobody Sat Nov 4 21:34:48 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SN9r71ZnNz4ymwQ for ; Sat, 4 Nov 2023 21:35:03 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from www541.your-server.de (www541.your-server.de [213.133.107.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SN9r66Xkbz4Yh3 for ; Sat, 4 Nov 2023 21:35:02 +0000 (UTC) (envelope-from mm@FreeBSD.org) Authentication-Results: mx1.freebsd.org; none Received: from sslproxy03.your-server.de ([88.198.220.132]) by www541.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qzOI7-0001tS-Lk; Sat, 04 Nov 2023 22:34:51 +0100 Received: from [188.167.171.2] (helo=[10.0.9.225]) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qzOI6-000Won-Kf; Sat, 04 Nov 2023 22:34:50 +0100 Content-Type: multipart/alternative; boundary="------------6M70tl1GHxx17km1YMm0xKtr" Message-ID: <2f93493b-a9cc-4c71-848a-efc55751a33e@FreeBSD.org> Date: Sat, 4 Nov 2023 22:34:48 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS txg rollback: expected timeframe? Content-Language: en-US To: Alexander Leidinger Cc: John F Carr , freebsd-fs@freebsd.org References: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> From: Martin Matuska In-Reply-To: X-Authenticated-Sender: martin@matuska.de X-Virus-Scanned: Clear (ClamAV 0.103.10/27082/Sat Nov 4 08:38:24 2023) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.133.96.0/19, country:DE] X-Rspamd-Queue-Id: 4SN9r66Xkbz4Yh3 This is a multi-part message in MIME format. --------------6M70tl1GHxx17km1YMm0xKtr Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi Alexander, I am already running block cloning with stable/14 on many production servers and also on my Ubuntu 23.10 notebook. The referenced issue appeared way before block cloning was even introduced so that is unrelated to me. Block cloning is not the same as deduplication. When copying a file with copy_file_range(2) between two datasets on the same pool, blocks of data are only referenced instead of doing a real copy. That is all magic that is being done. Did you check "zpool get bcloneused,bclonesaved poolname" before re-creating the pool? That would tell you if any blocks were cloned using block cloning at all. Cheers, Martin On 04/11/2023 18:35, Alexander Leidinger wrote: > > Am 2023-10-31 15:27, schrieb Alexander Leidinger: > >> On Tue, Oct 31, 2023 at 1:15 PM John F Carr wrote: >> >> >> >> > On Oct 31, 2023, at 06:16, Alexander Leidinger >> wrote: >> > >> > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in >> raidz2 setup) in a way that a normal import of the pool panics >> the machine with "VERIFY3(l->blk_birth == r->blk_birth) failed >> (101867360 == 101867222)". >> > >> >> I disabled that assertion because it gives a false alarm with >> some combinaion >> of deduplication, cloning, and snapshotting on one of my systems. >> >> See >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538 >> https://github.com/openzfs/zfs/issues/11480 >> >> I don't have deduplication on this pool. There are clones, and >> snapshots, and there could be recent ones if poudriere does some. Is >> it still a false alarm in this case? If yes, you say a kernel with >> this patch applied should let me import the pool without rollback? >> The github issue is from 2022, I have my doubts that this is the same >> issue we see. I rather expect some issues around the >> copy_file_range(2) related code for ZFS which was re-enabled 20 days >> ago (maybe it is valid to remove this assert, or maybe the block >> cloning part needs some tweak). CC Martin for the block cloning part. > > So in the end I was at least able to import the pool read-only with > the patch to disable this VERIFY3 panic. After a final incremental > backup I re-created the pool (with vfs.zfs.bclone_enabled=0) and > restored all datasets. Now some checks (this VERIFY3 part is enabled > again) and then a full backup. > > There is still the question what caused it. With the above report from > John about some issues when dedup is enabled (which wasn't on this > pool until the default of bclone_enabled was changed to 1, which is > some kind of dedup internal to ZFS as far as I was understanding the > description of block cloning in the openzfs ticket of block cloning) I > have some reservations about enabling it again. > > Martin, maybe it's a good idea to disable block cloning again, until > someone with the corresponding OpenZFS knowledge has investigated this... > > Bye, > Alexander. > > -- > http://www.Leidinger.net > Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org >  : PGP 0x8F31830F9F2772BF --------------6M70tl1GHxx17km1YMm0xKtr Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi Alexander,

I am already running block cloning with stable/14 on many production servers and also on my Ubuntu 23.10 notebook. The referenced issue appeared way before block cloning was even introduced so that is unrelated to me. Block cloning is not the same as deduplication. When copying a file with copy_file_range(2) between two datasets on the same pool, blocks of data are only referenced instead of doing a real copy. That is all magic that is being done. Did you check "zpool get bcloneused,bclonesaved poolname" before re-creating the pool? That would tell you if any blocks were cloned using block cloning at all.

Cheers,
Martin

On 04/11/2023 18:35, Alexander Leidinger wrote:

Am 2023-10-31 15:27, schrieb Alexander Leidinger:

On Tue, Oct 31, 2023 at 1:15 PM John F Carr <jfc@mit.edu> wrote:


> On Oct 31, 2023, at 06:16, Alexander Leidinger <alexleidingerde@gmail.com> wrote:
>
> Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in raidz2 setup) in a way that a normal import of the pool panics the machine with "VERIFY3(l->blk_birth == r->blk_birth) failed (101867360 == 101867222)".
>

I disabled that assertion because it gives a false alarm with some combinaion
of deduplication, cloning, and snapshotting on one of my systems.

See

 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538
 https://github.com/openzfs/zfs/issues/11480
 
I don't have deduplication on this pool. There are clones, and snapshots, and there could be recent ones if poudriere does some. Is it still a false alarm in this case? If yes, you say a kernel with this patch applied should let me import the pool without rollback?
 
The github issue is from 2022, I have my doubts that this is the same issue we see. I rather expect some issues around the copy_file_range(2) related code for ZFS which was re-enabled 20 days ago (maybe it is valid to remove this assert, or maybe the block cloning part needs some tweak). CC Martin for the block cloning part.

So in the end I was at least able to import the pool read-only with the patch to disable this VERIFY3 panic. After a final incremental backup I re-created the pool (with vfs.zfs.bclone_enabled=0) and restored all datasets. Now some checks (this VERIFY3 part is enabled again) and then a full backup.

There is still the question what caused it. With the above report from John about some issues when dedup is enabled (which wasn't on this pool until the default of bclone_enabled was changed to 1, which is some kind of dedup internal to ZFS as far as I was understanding the description of block cloning in the openzfs ticket of block cloning) I have some reservations about enabling it again.

Martin, maybe it's a good idea to disable block cloning again, until someone with the corresponding OpenZFS knowledge has investigated this...

Bye,
Alexander.

--
http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
--------------6M70tl1GHxx17km1YMm0xKtr-- From nobody Sat Nov 4 23:54:10 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SNDxB1pChz50Gqg for ; Sat, 4 Nov 2023 23:54:38 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SNDx95nDCz3MtJ; Sat, 4 Nov 2023 23:54:37 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1699142068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Ba5IIATon4kozVwPGc60TEoOAw5mq3OinwHIP4/T2OM=; b=Iz2ZoNxSNXKiJzvfz6opXcdQcW5CpkimkyJa42GAzkhSi+JyjvhbONnoVX7/RTOF3nckNY +ntHjGU9NlkcOBvcoFe/HfFm310QkG81O8WDWVQIfoubniA37Aq9zoiwx8HPLE/UbxWY0M zrY/b1Ckq4Z1vDVwBlKmaw+LECGD8Vn+E5iBpecNP57iS9QluJFzq/F1ODhljSZIbeb6DY oIbrz9iR/PrUv1uxD1cWffXnm0LvCjR90rRNlc5UJB6wwuwkYcNVju/jQhoOKn1UIN9zFv scDIa/eFMfj/8S6Dx2Els6ew5ts94Sa8NFGK0eiAaw64Z3PRzvu+OGt6Pjg2ig== Date: Sun, 05 Nov 2023 00:54:10 +0100 From: Alexander Leidinger To: Martin Matuska Cc: John F Carr , freebsd-fs@freebsd.org Subject: Re: ZFS txg rollback: expected timeframe? In-Reply-To: <2f93493b-a9cc-4c71-848a-efc55751a33e@FreeBSD.org> References: <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> <2f93493b-a9cc-4c71-848a-efc55751a33e@FreeBSD.org> Message-ID: <855504b600b8c6b89b9f13a371954e95@Leidinger.net> X-Sender: Alexander@Leidinger.net Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_ea40b4d63b8a10e0f54b1bacf0ad3196"; micalg=pgp-sha256 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE] X-Rspamd-Queue-Id: 4SNDx95nDCz3MtJ This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_ea40b4d63b8a10e0f54b1bacf0ad3196 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Am 2023-11-04 22:34, schrieb Martin Matuska: > Hi Alexander, > > I am already running block cloning with stable/14 on many production > servers and also on my Ubuntu 23.10 notebook. The referenced issue > appeared way before block cloning was even introduced so that is > unrelated to me. Block cloning is not the same as deduplication. When > copying a file with copy_file_range(2) between two datasets on the same > pool, blocks of data are only referenced instead of doing a real copy. > That is all magic that is being done. Did you check "zpool get > bcloneused,bclonesaved poolname" before re-creating the pool? That > would tell you if any blocks were cloned using block cloning at all. I do not think the referenced issue is what I was seeing as such (as it speaks about the dedup being in use). I don't have any dedup in this pool. I'm aware that block cloning is not the same as dedup, but I think I remember something about having parts of the same tech (different files may reference the same blocks) being used for block cloning with some parts more lightweight than dedup on top. The question here for me is, if the panic message is something which is a valid case for block cloning which can happen and shouldn't create a panic, or not. No, I didn't check those values. But I have the locally modified properties in a file, and the feature block_cloning was active, and not only enabled. So there should have be some cloned blocks. Right now in the re-created pool it is only enabled (as expected). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_ea40b4d63b8a10e0f54b1bacf0ad3196 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmVG2bAACgkQEg2wmwP4 2IZoRA//fi+oYyV7XBVndgTghBxHYNP4wg4wzu3qlVyy4kH1QTVeLLxOwe/EswGv SyzfXZKWC+t8up5KhPW2I8N3U6f2Z3CKDkIw5/8eKJb6rLNONJEfEytNecPpRGIy MMMqL1Ve58xVEmFuvhouC8mAR4ScRoSvQCnOXtZXR1angdKmMwuAEeCuXzZqHNyR UJGdlaeNteUWiNOFkdWIiklEFNng6k/Od92EeiI+UmVZ7ciJT0++8iAlf3TIckBj ZBLFJSw0sy5wbYYCVDTYgW8v0WKLmAw4rMRAZ41o9BnQCgz86BdYi20548Tb/rmN PzJPKuJu3vXSCuRM4gzgNGBpW1NoiE5q6kOyUPO4HOeqTT9b7Uj5FmGt+Tpl1mTf gZFFHwFpm3pai+yonqXY8WLSS/LCXPaRpG6HN/Xik7WDUMPP6eB/NL1VRTVX5zai +z2dDmzLa1tuDCYhvp4wpNQZ3XaFDyv+GyguNFmSAhlYwJf2+Yrc/5lT159ZcCHj DNlMg5sl6DZqaZPQp7wwtwiL9sKyjv4x8AM1RErEn4SkAdaY5xS2zvT9s0bVBFJI M/wsN83i+3XlJGbvzfxTRU/eOjB2LNoe3g3FExeYYNOzjPJ9LbDoD/j/q5gzpndM vRT/aWirO0wL03+6VhNbRNFMYLDUhEgXm0yf/hM9e66vJyut6Ww= =FOEX -----END PGP SIGNATURE----- --=_ea40b4d63b8a10e0f54b1bacf0ad3196--