From nobody Mon Oct 2 23:08:35 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzxTX0hthz4wbpx for ; Mon, 2 Oct 2023 23:08:48 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oi1-x22b.google.com (mail-oi1-x22b.google.com [IPv6:2607:f8b0:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzxTW2fNXz3bk2 for ; Mon, 2 Oct 2023 23:08:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=X02AUZSd; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::22b as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x22b.google.com with SMTP id 5614622812f47-3ae55c2e12bso221816b6e.0 for ; Mon, 02 Oct 2023 16:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696288125; x=1696892925; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5XfvAE7ksFCBaPOsZqL9T6ZwsuztRJgQXBwMBl5XHyw=; b=X02AUZSd3Y91W58Fp8kBikbvTodCvQnI8zOCX5pGXxQarusM2t4VaNrm+Lrxnfz/Z8 /eLAzutBNcW4XecBdCL/xe43ji0sNJtnaL3CWJcaqBr9TKkAaQBpMgqpn807cribDoY5 KN6nNBIX4dZ1oTp7R/0bwx+f2JoBtZckV1cfmI8gnnJ3P0nGjwbyWzlQ+HMfHdfV39pb ixPyQfTp66lMr1mbt4N/odAeHNH7kWvF0MqqZikeX1tJgWD6MtjsB19x3UZCmoE/8XgJ Du2eMc/IxpqpUEmOFja6tqFSJaL9eVXI9xabDW8KNj2zZIzkAhiB1JIpAEtb9mQtB2Dd 5f8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696288125; x=1696892925; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5XfvAE7ksFCBaPOsZqL9T6ZwsuztRJgQXBwMBl5XHyw=; b=k6hjW9T8zIqwbPhpQlJK4vscKcsCnvW/IO4Ay/vsN9rE5493+GOH+vCZU0IMlRbP8z PvBFPkDO/n5NEtjby8Yhhfn9uKXHArPgP2xtKq+HFaqcol7hj7Qh9bxb2p60fg2B4IrB Ul8G/2QxDgi4rsC4OPBB5FWCn3StvEXpvUjbm2JU22WQiPy7e7ZDfl18ZqbsgGuxO5Bi jrLk/Q5hHmsFzmT5LYIz0ARxwYvvNTkJbCVTuBPKesArKOpVDCN2OvwQVHcBc8Okeb64 j9cM9xCS3p0x2VLlkjydXeAIrb5nyOldGo0jzFF8qLEk1xnTzj9rtPHdkO/WWuh1+1ro HHTA== X-Gm-Message-State: AOJu0YzDUkpoLs+iRxn4rRvLMF+W99JzbSwMQJC7llyCWQ1RDGExIj1G mYau2dcHlPl8ux2Jg0uX3fd3/P3h9cK/wJgdYkq6YYI= X-Google-Smtp-Source: AGHT+IFrgZmxmQpP9TtT0/EH6DRqdwuGLKX4/Z+BjIohieqf0l/RSPlKhWBCUvRH0z42rbjEaPG7IrudPmyuKxoGG3g= X-Received: by 2002:a05:6808:1448:b0:3a9:11cb:d5a6 with SMTP id x8-20020a056808144800b003a911cbd5a6mr17026280oiv.32.1696288125039; Mon, 02 Oct 2023 16:08:45 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Mon, 2 Oct 2023 16:08:35 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: multipart/mixed; boundary="00000000000021f1f20606c3dbed" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.19 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.91)[-0.906]; NEURAL_SPAM_SHORT(0.71)[0.711]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_RCPT(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22b:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; HAS_ATTACHMENT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RzxTW2fNXz3bk2 --00000000000021f1f20606c3dbed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Oct 1, 2023 at 8:01=E2=80=AFPM J David wr= ote: > > On Sat, Sep 30, 2023 at 6:06=E2=80=AFPM Rick Macklem wrote: > > --> I'd suggest you try and disable delegations. I do not know how to > > do this on > > the Linux server, but not running the nfscbd(8) daemon should stop= them > > from being issued. > > The nfscbd daemon is not running on any of the clients. > > > If the Linux server still > > issues delegations > > How would I determine that? nfsstat -E -c and then look at the number under "Delegs". It is a current count of delegations, so if it remains 0 over time, no delegations are being issued. > > > Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. > > I have never seen (and do not recall anyone else reporting) this error > > return. > > It was days earlier than this issue and *might* be related to someone > rebooting an NFS server the client is connected to at that time. Hard > to tell for sure without timestamps. Yes. This would occur after a NFS server reboot or some other occurrence that causes the client to do a TCP reconnect. I have attached a small patch which should make the NFS client handle this error correctly. rick > > > I'd suggest you first check network connetcivity. Both NFS client and N= FS > > server should be able to ping each other. > > The network connectivity is OK; network interface error counts are > monitored with munin on all systems, and don't appear to be an issue. > > On that specific system's dedicated interface for NFS: > > Name Mtu Network Address Ipkts Ierrs Idrop > Opkts Oerrs Coll > net1 1500 (redacted) 18754677570 0 0 > 18598354347 0 0 > > Also, in both cases (which occurred with different client & server > pairs), there were three different directories from the server mounted > on the client. > > One was affected. The other two worked fine. Listing files and > directories. Locking. Calculating hashes of large files. > Creating/writing files. Etc. > > E.g. /a /b and /c were mounted from nfs.server and /a and /b were fine > while /c was hung. (And, as I mentioned before but seems worth > repeating here for completeness, other FreeBSD clients had /c mounted > with no problems while it was hung on this machine.) > > > > If that is the case, then I'd suggest you capture packets. On the FreeB= SD > > end: > > # tcpdump -s 0 -w out.pcap host > > Let this run for a while and then pull out.pcap into wireshark and see = what > > traffic is going between the NFS client and server. > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > If/when the issue happens again, I will attempt to do this and report bac= k. > > Thanks! --00000000000021f1f20606c3dbed Content-Type: application/octet-stream; name="replyuncached.patch" Content-Disposition: attachment; filename="replyuncached.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_ln9i58ge0 LS0tIHN5cy9mcy9uZnMvbmZzX2NvbW1vbmtycGMuYy5yZXBseXVuY2FjaGVkCTIwMjMtMTAtMDEg MTY6NDc6MDQuMDI2ODcwMDAwIC0wNzAwCisrKyBzeXMvZnMvbmZzL25mc19jb21tb25rcnBjLmMJ MjAyMy0xMC0wMSAxNjo1MDoxOC41Nzc1MTQwMDAgLTA3MDAKQEAgLTEyOTQsNyArMTI5NCw4IEBA IHRyeWFnYWluOgogCQkJICAgICBuZC0+bmRfcHJvY251bSAhPSBORlNQUk9DX0xPQ0tVKSkpIHx8 CiAJCQkgICAgKG5kLT5uZF9yZXBzdGF0ID09IE5GU0VSUl9ERUxBWSAmJgogCQkJICAgICAobmQt Pm5kX2ZsYWcgJiBORF9ORlNWNCkgPT0gMCkgfHwKLQkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBO RlNFUlJfUkVTT1VSQ0UpIHsKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVTT1VS Q0UgfHwKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVUUllVTkNBQ0hFRFJFUCkg ewogCQkJCS8qIENsaXAgYXQgTkZTX1RSWUxBVEVSREVMLiAqLwogCQkJCWlmICh0aW1lc3BlY2Nt cCgmdHJ5bGF0ZXJfZGVsYXksCiAJCQkJICAgICZuZnNfdHJ5bGF0ZXJfbWF4LCA+KSkK --00000000000021f1f20606c3dbed--