From nobody Sat Oct 7 00:12:10 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2Qj40tPJz4w77r for ; Sat, 7 Oct 2023 00:12:24 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2Qj33M3Jz4YLZ for ; Sat, 7 Oct 2023 00:12:23 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="U/mRQ7ML"; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::102f as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-278eaffd81dso1966537a91.0 for ; Fri, 06 Oct 2023 17:12:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696637542; x=1697242342; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=USrbJOBF1BmWR5xp08bZiAs/J+8kUe2LHIy6uZYQoaA=; b=U/mRQ7ML6ihnWXXT9S3m++eaOjUag2knFESYSlIJ9JOZ8k4Nczp5jg8ouJ9RGf67KY hQ0yGP9BBX63Q1Rytc4ZmonAqNl+9awghuy5Yt4Kr3pm/4lO2hROSSMIktQwG4YiTLm/ XCBQpDfldKlgBq22AwyGLEDnv6gmX9WI6+wJ2C7edmnFmk2B5G4elG24c7hAo64g7+zN a+7gvAG75A8kmaeU+ECU8deiqdHuZn5yKoV/UPL+2yvE4Vl3odPVmTZLdr5lQLquxkzH yDGRRi02rnoWnCAu1+O6MhXDM5/0bXpHKEiKEedCEKqsMm+7vX7UCikir8jZfNEQl+/x nwzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696637542; x=1697242342; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=USrbJOBF1BmWR5xp08bZiAs/J+8kUe2LHIy6uZYQoaA=; b=SugyGLlv/+0nqzJhCQvml7DI/NOTg3BHnqAuVUOGKJ/t3Nyyls0QGMVX/30CHhIbYv YTetVOxtPoud6CVDjqlC0HM0qI/c1HEFpOw8SUcUmOZxreGzc61IrrAgxKF6mReFiO7/ Il7mau1BsAq+QfSCa+fmdDNGVNnD3WM1GltGyms4gaoVl9JWX2iBmKIWe0droxsFUBbi uLBbXRx9NN1rSnowpfOzJRHhKk015FRFmEk+LLAXAS1YtkreEGjQGruSDoInIPZoHThR Z1LZDUV0Vk1Xsm7ufxYCK7JMSCi9ze7aT58y0gI8DduCEfqROFOZn1DWvol3ySSXOM0R 9Jxg== X-Gm-Message-State: AOJu0YwmFpMlyGhRHkwWaXjwOeA01DIe+9gCF4nXl5E2nu1Xk8Rd0Phq utYUubd9ofJCYr7q/tf2LEfkg0Gra1/zXkGEyg== X-Google-Smtp-Source: AGHT+IEUB8y7FlHGgiClQHDg+BF+epMpAgujM4WSfT70NbVVtdlBESpI3lHUW+5X47FNYlnJE0ASnifdqfGqZqrBl44= X-Received: by 2002:a17:90b:4f8c:b0:268:b66b:d9f6 with SMTP id qe12-20020a17090b4f8c00b00268b66bd9f6mr8663581pjb.18.1696637542107; Fri, 06 Oct 2023 17:12:22 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Fri, 6 Oct 2023 17:12:10 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102f:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2Qj33M3Jz4YLZ On Fri, Oct 6, 2023 at 10:48=E2=80=AFAM J David w= rote: > > On Mon, Oct 2, 2023 at 7:08=E2=80=AFPM Rick Macklem wrote: > > > The nfscbd daemon is not running on any of the clients. > > > > > > > If the Linux server still > > > > issues delegations > > > > > > How would I determine that? > > nfsstat -E -c > > and then look at the number under "Delegs". It is a current count of > > delegations, so if it remains 0 over time, no delegations are being iss= ued. > > But if this is done from a client that is not running nfscbd, isn't it > pretty well guaranteed to be zero? Yes. If the NFS server is functioning correctly and there is no nfscbd running, it should remain at 0. > > Checking all the clients I can find, "Deleg" is zero on all of them. > On about half, "DelegRet" is nonzero but small (1-100), but I don't > know what that is or if it's related. It implies that the client is doing a recovery after a NFS server reboot or something like that. Since no delegations are being issued, the DelegRet operations would/should be related to a recovery after a NFS server reboot. > > > I have attached a small patch which should make the NFS client handle > > this error correctly. > > I will look for a way to try this patch, but the clients in this case > are all managed with freebsd-update and don't have enough disk space > to build a kernel locally, so it may be tricky. > > > > > # tcpdump -s 0 -w out.pcap host > > > > Let this run for a while and then pull out.pcap into wireshark and = see what > > > > traffic is going between the NFS client and server. > > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > > > > > If/when the issue happens again, I will attempt to do this and report= back. > > I am also working on getting access to Wireshark. > > In the interim, it did happen again, so the best I can do is put a > little bit of tcpdump output here: https://pastebin.com/UDrphwr5 . This appears to be text. I need the actual pcap file captured by tcpdump. > > I can't vouch for "correct" but it does mostly seem to decode the NFS pac= kets. > > It seems to loop the same couple of actions with long delays (15 > seconds) between retries: > > This sequence: > +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr" > in packet body) > +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in > packet body) > +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids > 1/53 sz 0 ("Renew" in packet body) > +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request > couldn't be completed in time ("Getattr" in packet body) If the server is not replying to a Getattr, then it is broken. That will certainly hang a NFS client. > > Repeats after 15 seconds: > +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in > packet body) > ... etc > > The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid > (transaction/request ID?) increments each time. After 15sec, the client gives up on waiting for the reply from the server, creates a new TCP connection and tries the RPC again. Then it looks like the server does not reply again. If you can give me the pcap file, I will take a look at it, but this suggests a broken NFS server. rick > > Maybe that will provide a lucky flash of insight in the interim. > > Thanks!