From nobody Mon Oct 2 03:01:31 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzQhp0Vf7z4w9bW for ; Mon, 2 Oct 2023 03:01:46 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzQhm6dzzz3JFn for ; Mon, 2 Oct 2023 03:01:44 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hn7efpTQ; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::32b as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32b.google.com with SMTP id 46e09a7af769-6c615df24c0so4909351a34.1 for ; Sun, 01 Oct 2023 20:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696215703; x=1696820503; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=g7wzO7WbjwY2U3Txw3RTsN1TprQF4vORrLfrmN2i4yw=; b=hn7efpTQFonpIEKKlz+6ZLIocRGhVp/FkFxUL3ZuzdYYxpIEHfPyk4EIUvshW+o+CL tDRMkoU2N3rabQlnciZSvrwcr4tI7ym2qBKi0QF/+IER7rSwbSzCtNQEokpzqleUnBUF GzhjCyyHbxxIiawrFKDcht7XOJuR7n7crkz5e/k6QGHMC43jzDKmiUwkug2zYpxFeipV dqWZqwRRnVhW3qMXeaJ5+fbJ4YyT76wDIjuBr0PUmfstEUbAUlRAaRCTSN6QD/s8ZvDz RkSmiADJNSM82fgGk6V1suq+ZdQqR9wuy9ToJ4zo6gj3+h8AQ1kXBRzVR78QCJpn00kI Zhmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696215703; x=1696820503; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g7wzO7WbjwY2U3Txw3RTsN1TprQF4vORrLfrmN2i4yw=; b=hvapc5c+vbIwoDERTcy37Af30xWuZQ8QqL8XwJZBau0uHNbYoImmeomNvmgN6Facj2 w/50kdNIWQ9e/S3rpyx2OOeWNVICnDlwhBy35F35tYziUDiJDPUYdvPGByY7Eaiez0dF AR4UbZKrRc6pwQlO+pgW2UD9Lf3oHSXKBTsKMRkTwTaar/OyyE+pkTWWB/QxXx+Cytxt TkxM8TlTBBPXgzaNge0yaZabYBzV7zaiBKxvIzUf+p3jdR7pjnQp9ellK/bZuSpC58/c scWLy3tTZUM4CALNBTJFcO9Hzs4CPv+TlHbNUH+Dwey9MfnKG0O2aAIjchK9muKMD7qV cW7Q== X-Gm-Message-State: AOJu0YzSrQJz8gp29K4ADYFV/kKYLvc9vfKvSmLop2LI0qzwcRcFVXIJ 1Gh9LRz3Vj9yvmKWfSbcOm+lWC4qo/rxuojq4qE9FA2Z X-Google-Smtp-Source: AGHT+IH7Lz2bsgXRlXVmEy1N1FWU+B2jkCJOUaji3ss9W4t8nenXrc8mQ+UBkedmH21ynkCII1PWna4aO4qUduAH/vM= X-Received: by 2002:a05:6358:9189:b0:134:f326:e819 with SMTP id j9-20020a056358918900b00134f326e819mr12296377rwa.29.1696215702980; Sun, 01 Oct 2023 20:01:42 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sun, 1 Oct 2023 23:01:31 -0400 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32b:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RzQhm6dzzz3JFn On Sat, Sep 30, 2023 at 6:06=E2=80=AFPM Rick Macklem wrote: > --> I'd suggest you try and disable delegations. I do not know how to > do this on > the Linux server, but not running the nfscbd(8) daemon should stop t= hem > from being issued. The nfscbd daemon is not running on any of the clients. > If the Linux server still > issues delegations How would I determine that? > Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. > I have never seen (and do not recall anyone else reporting) this error > return. It was days earlier than this issue and *might* be related to someone rebooting an NFS server the client is connected to at that time. Hard to tell for sure without timestamps. > I'd suggest you first check network connetcivity. Both NFS client and NFS > server should be able to ping each other. The network connectivity is OK; network interface error counts are monitored with munin on all systems, and don't appear to be an issue. On that specific system's dedicated interface for NFS: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll net1 1500 (redacted) 18754677570 0 0 18598354347 0 0 Also, in both cases (which occurred with different client & server pairs), there were three different directories from the server mounted on the client. One was affected. The other two worked fine. Listing files and directories. Locking. Calculating hashes of large files. Creating/writing files. Etc. E.g. /a /b and /c were mounted from nfs.server and /a and /b were fine while /c was hung. (And, as I mentioned before but seems worth repeating here for completeness, other FreeBSD clients had /c mounted with no problems while it was hung on this machine.) > If that is the case, then I'd suggest you capture packets. On the FreeBSD > end: > # tcpdump -s 0 -w out.pcap host > Let this run for a while and then pull out.pcap into wireshark and see wh= at > traffic is going between the NFS client and server. > (Unlike tcpdump, wireshark does know how to decode NFS properly.) If/when the issue happens again, I will attempt to do this and report back. Thanks!