Date: Mon, 2 Oct 2023 16:08:35 -0700 From: Rick Macklem <rick.macklem@gmail.com> To: J David <j.david.lists@gmail.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: FreeBSD 13.2 NFS client mount hangs Message-ID: <CAM5tNy52x2s=9Os%2BPAa=-iz7F_o_4_9XxJbRAR28V1v9A4nN6A@mail.gmail.com> In-Reply-To: <CABXB=RSUJ3mpYF5puAm0hSxeavozxyf7Ruab8mPrtBOu6bxM-w@mail.gmail.com> References: <CABXB=RRSHMhZQFL28eHKjhAYmU87qjpQ=B1=8VRSZoXat9=r5A@mail.gmail.com> <CAM5tNy4sqc18UCZF0vgL%2BXP6vF0wgt_3Yi07yY4wqeuzs6haMA@mail.gmail.com> <CABXB=RSUJ3mpYF5puAm0hSxeavozxyf7Ruab8mPrtBOu6bxM-w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000021f1f20606c3dbed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Oct 1, 2023 at 8:01=E2=80=AFPM J David <j.david.lists@gmail.com> wr= ote: > > On Sat, Sep 30, 2023 at 6:06=E2=80=AFPM Rick Macklem <rick.macklem@gmail.= com> wrote: > > --> I'd suggest you try and disable delegations. I do not know how to > > do this on > > the Linux server, but not running the nfscbd(8) daemon should stop= them > > from being issued. > > The nfscbd daemon is not running on any of the clients. > > > If the Linux server still > > issues delegations > > How would I determine that? nfsstat -E -c and then look at the number under "Delegs". It is a current count of delegations, so if it remains 0 over time, no delegations are being issued. > > > Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. > > I have never seen (and do not recall anyone else reporting) this error > > return. > > It was days earlier than this issue and *might* be related to someone > rebooting an NFS server the client is connected to at that time. Hard > to tell for sure without timestamps. Yes. This would occur after a NFS server reboot or some other occurrence that causes the client to do a TCP reconnect. I have attached a small patch which should make the NFS client handle this error correctly. rick > > > I'd suggest you first check network connetcivity. Both NFS client and N= FS > > server should be able to ping each other. > > The network connectivity is OK; network interface error counts are > monitored with munin on all systems, and don't appear to be an issue. > > On that specific system's dedicated interface for NFS: > > Name Mtu Network Address Ipkts Ierrs Idrop > Opkts Oerrs Coll > net1 1500 <Link#2> (redacted) 18754677570 0 0 > 18598354347 0 0 > > Also, in both cases (which occurred with different client & server > pairs), there were three different directories from the server mounted > on the client. > > One was affected. The other two worked fine. Listing files and > directories. Locking. Calculating hashes of large files. > Creating/writing files. Etc. > > E.g. /a /b and /c were mounted from nfs.server and /a and /b were fine > while /c was hung. (And, as I mentioned before but seems worth > repeating here for completeness, other FreeBSD clients had /c mounted > with no problems while it was hung on this machine.) > > > > If that is the case, then I'd suggest you capture packets. On the FreeB= SD > > end: > > # tcpdump -s 0 -w out.pcap host <nfs-server-name> > > Let this run for a while and then pull out.pcap into wireshark and see = what > > traffic is going between the NFS client and server. > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > If/when the issue happens again, I will attempt to do this and report bac= k. > > Thanks! --00000000000021f1f20606c3dbed Content-Type: application/octet-stream; name="replyuncached.patch" Content-Disposition: attachment; filename="replyuncached.patch" Content-Transfer-Encoding: base64 Content-ID: <f_ln9i58ge0> X-Attachment-Id: f_ln9i58ge0 LS0tIHN5cy9mcy9uZnMvbmZzX2NvbW1vbmtycGMuYy5yZXBseXVuY2FjaGVkCTIwMjMtMTAtMDEg MTY6NDc6MDQuMDI2ODcwMDAwIC0wNzAwCisrKyBzeXMvZnMvbmZzL25mc19jb21tb25rcnBjLmMJ MjAyMy0xMC0wMSAxNjo1MDoxOC41Nzc1MTQwMDAgLTA3MDAKQEAgLTEyOTQsNyArMTI5NCw4IEBA IHRyeWFnYWluOgogCQkJICAgICBuZC0+bmRfcHJvY251bSAhPSBORlNQUk9DX0xPQ0tVKSkpIHx8 CiAJCQkgICAgKG5kLT5uZF9yZXBzdGF0ID09IE5GU0VSUl9ERUxBWSAmJgogCQkJICAgICAobmQt Pm5kX2ZsYWcgJiBORF9ORlNWNCkgPT0gMCkgfHwKLQkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBO RlNFUlJfUkVTT1VSQ0UpIHsKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVTT1VS Q0UgfHwKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVUUllVTkNBQ0hFRFJFUCkg ewogCQkJCS8qIENsaXAgYXQgTkZTX1RSWUxBVEVSREVMLiAqLwogCQkJCWlmICh0aW1lc3BlY2Nt cCgmdHJ5bGF0ZXJfZGVsYXksCiAJCQkJICAgICZuZnNfdHJ5bGF0ZXJfbWF4LCA+KSkK --00000000000021f1f20606c3dbed--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy52x2s=9Os%2BPAa=-iz7F_o_4_9XxJbRAR28V1v9A4nN6A>