From nobody Sat Oct 7 00:31:56 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2R7s1QLnz4w9xp for ; Sat, 7 Oct 2023 00:32:09 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2R7r2WfRz4ZWg for ; Sat, 7 Oct 2023 00:32:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=VgHBR5mX; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1029 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-279294d94acso3019676a91.0 for ; Fri, 06 Oct 2023 17:32:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696638727; x=1697243527; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=3UD5HrjRvFPipx8M713ltCWd4/lEv5odr8l686NX73Q=; b=VgHBR5mXniOOtqtr0oVPCuFYFQ+wK7CNOoRxTW9iiFJmCrLB94XIXi1ohxDd3dLTxr nK3YoRR5F0N9rAx/Z25aaE56tnGnjXFoodRzyhKHacs2X3JHDRwnRRx9DRlLPduO3l51 DNd9QGPa9YaNN92dVUAvpQZd5XIn/yKmsbJ4It46bj0zCj2f1OiTxIGna5oRvyHSv3jm u+r3yOP5a5Di8my7qXKdQXXscgxHVo3PN2bzcxhdGPXH1xV/V+8SZhMZtnVFJh3/I1jK XjtZfKjWeMQEkqbIeEYrju5s8EesuU9oJx5j2S8Nt8o86RsOFQeBL2D80+Qj8Fi7xpgX YSyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696638727; x=1697243527; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3UD5HrjRvFPipx8M713ltCWd4/lEv5odr8l686NX73Q=; b=cd6V0rww2wLDPp5N9qADLQZPD4Zf6KSLau3SVfePiTDGClThKlPlcRM0N/RabxNaB5 HYNPurSBFqkOmtpCevJ1PBr24B9gQTh408FWKebfTJ0dvmii5XaZ1j+Db9wn3Fw76RrL sL9vGKqOExMzzJnVt4FeCFAQqzYQcrdZ364Wz/xT8mvVLLIvTJW26y7lWxkhBMPZ+itl 3NItagnPUpOQb272DXV31kgR2ziVtpvJkLmGc4AVA3QUnTRVc/44G6K6G5easoA9ePv5 jR/5yVmJQS/XA1JD6B2yB4sPRsD1QVnt3IlpTpOHyGdSTaBc/k4YeklCw59E2y1chx/M D12g== X-Gm-Message-State: AOJu0Yxlj4hqIbl6fumCJFHU5P479yIDSWW8CQEx4FnHQl8BNmfx5NBF on4FQeuQPbr/iNZYjb+BvrtO25zHC8Ovg5w/gFAuqEI= X-Google-Smtp-Source: AGHT+IF7GPsE4RxQN8ctkwn3jyJmIeey2NLj03NixB+X7bt5dqL4T/48oBxdK0ORLi+jEeB3Vq9xeyrP3A13QRFREd8= X-Received: by 2002:a17:90b:3907:b0:276:5512:13ab with SMTP id ob7-20020a17090b390700b00276551213abmr9021930pjb.10.1696638727087; Fri, 06 Oct 2023 17:32:07 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Fri, 6 Oct 2023 17:31:56 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1029:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2R7r2WfRz4ZWg On Fri, Oct 6, 2023 at 10:48=E2=80=AFAM J David w= rote: > > On Mon, Oct 2, 2023 at 7:08=E2=80=AFPM Rick Macklem wrote: > > > The nfscbd daemon is not running on any of the clients. > > > > > > > If the Linux server still > > > > issues delegations > > > > > > How would I determine that? > > nfsstat -E -c > > and then look at the number under "Delegs". It is a current count of > > delegations, so if it remains 0 over time, no delegations are being iss= ued. > > But if this is done from a client that is not running nfscbd, isn't it > pretty well guaranteed to be zero? > > Checking all the clients I can find, "Deleg" is zero on all of them. > On about half, "DelegRet" is nonzero but small (1-100), but I don't > know what that is or if it's related. > > > I have attached a small patch which should make the NFS client handle > > this error correctly. > > I will look for a way to try this patch, but the clients in this case > are all managed with freebsd-update and don't have enough disk space > to build a kernel locally, so it may be tricky. > > > > > # tcpdump -s 0 -w out.pcap host > > > > Let this run for a while and then pull out.pcap into wireshark and = see what > > > > traffic is going between the NFS client and server. > > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > > > > > If/when the issue happens again, I will attempt to do this and report= back. > > I am also working on getting access to Wireshark. > > In the interim, it did happen again, so the best I can do is put a > little bit of tcpdump output here: https://pastebin.com/UDrphwr5 . Maybe someone who is familiar with tcpdump output can look at this. (I always use wireshark.) It looks to me like the TCP checksum is failing on the client->server reque= st, but maybe I am not reading it correctly. If the checksum is incorrect, then there is something badly broken in your network fabric. rick > > I can't vouch for "correct" but it does mostly seem to decode the NFS pac= kets. > > It seems to loop the same couple of actions with long delays (15 > seconds) between retries: > > This sequence: > +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr" > in packet body) > +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in > packet body) > +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids > 1/53 sz 0 ("Renew" in packet body) > +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request > couldn't be completed in time ("Getattr" in packet body) > > Repeats after 15 seconds: > +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in > packet body) > ... etc > > The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid > (transaction/request ID?) increments each time. > > Maybe that will provide a lucky flash of insight in the interim. > > Thanks!