From nobody Wed May 29 16:26:44 2024
X-Original-To: fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VqFBr3SL5z5Mnmq
for <fs@mlmmj.nyi.freebsd.org>; Wed, 29 May 2024 16:26:44 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VqFBr0Fs1z4gvD
for <fs@FreeBSD.org>; Wed, 29 May 2024 16:26:44 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717000004; a=rsa-sha256; cv=none;
b=KoX4A/F7OfUEUF5C2UN1YIldWH+VCowMmZfp5rUorXolQZSn5f+efAbc6LO1jjbq3h6dfO
X1Nx+0ls6QEXO+2Bkg5QNmPdAH3ixp5FVp7kWtRZe/I9bNVS9fTRL7HKhUw82H7iZf75T7
xadtn/TdJsLetOfT0eq6W/gC34XOO+ZM+gkGDevyQ0oMSmgcHX4g5IoVrxeL5gvSC+Eqaa
0PJ2ENRsHPSqK2KkKAIklnnDJrHWVAlOefcm5CtXH9VlR8GB1Ok8/RpWc9M+fT+vWwM8Bm
S+YfDzPA+TPaoCTp+z3c/qAc55w9T+Hyigp0IuxaGvCTXPXfBjp8cdS7S5j33A==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1717000004;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=rv5utHElhbI9uCbrHYxNCewQ+lv4MDDGJ7LYCB1Qo2w=;
b=uXKwo4xEO6iYbnnmO0VlyNiGmU6922R0wB3vSJzZ7bZk4Ar1EBh/fypOZYK5Qsq48g84z2
rWxNbP7yPh8zQP78byeM9XYqOvXqLjos8X/lu3xB401ta7e6I1MoorJc7bLAWFrrOsi8nq
S226FMNSfJlWw7J9CLedSdHCHb87JbhuCrXHa7bimraPdAkoENdGRHIi6cBLR1Iq6Dlk7U
ITxRo8i2hKWznYmR3Su+H3Jxs/Ec0ghxinpGs29SxQcrl2CafG+y1YH3PWFyN75y2YfK4v
Y0Os9MWHbT3kgrJuWPyLXWonURR3NSjcy84Tto7YrHtinLT+GNwwMoFbeXMOBw==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VqFBq6sF4z13R9
for <fs@FreeBSD.org>; Wed, 29 May 2024 16:26:43 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 44TGQhsS006038
for <fs@FreeBSD.org>; Wed, 29 May 2024 16:26:43 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 44TGQhr6006037
for fs@FreeBSD.org; Wed, 29 May 2024 16:26:43 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed
Date: Wed, 29 May 2024 16:26:44 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 15.0-CURRENT
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: jfc@mit.edu
X-Bugzilla-Status: In Progress
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-276341-3630-21pc2v80rr@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-276341-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-276341-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341
--- Comment #26 from John F. Carr <jfc@mit.edu> ---
The fix has been committed upstream and the corresponding bug closed.
I think this is worth an out of band cherry-pick rather than waiting for a
routine merge. It's a one-liner.
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From nobody Fri May 31 13:32:33 2024
X-Original-To: fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrPDy3D9lz5MvVK
for <fs@mlmmj.nyi.freebsd.org>; Fri, 31 May 2024 13:32:34 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VrPDy2BpSz4sms
for <fs@FreeBSD.org>; Fri, 31 May 2024 13:32:34 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717162354; a=rsa-sha256; cv=none;
b=Fm3APLVIEAbQB9n5xCFRq9A1GFffIIqOzIAjml2GjicIAS/yBOX7GvU+AYwihGKyubWC5u
FI1NZUhwlMGcFrYCQJI+3MDuCcU02BBXOkycaRTblM3Ph48yJ55qivDVUBgBwHjOHhi5Tj
BXsU/9cCHc1/2ujgll2tmTYscNT5BFkgfkT6QOdfu2W7JRT83hslZMVlzGgvfm0k2A6gda
5ODymOR3XTsjwWKFV7zLtzZkSbbwdd6eePKriKZaK4qCGszR46IRvdrp0iK0Q5Do07huSS
Qp40qwG60Aadhg/29Pt6KaduuPEJNfxtfAexwRu/oeP26R289H3IH+gyeKNNFw==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1717162354;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=d0VvNbiqy4nZrGdAd+sTN0tCbJNTllMTwH+F/GjPvak=;
b=ZzHAqxrRcryU4ACgPfuF1J5yTDZ8f+gyvfetm6nVoWAETuKx/647hutBGbUqiPeRQ+G6CU
1avkD+yzqitL5hU/DcHVynARlfg0ID0VaXuK0sqDFnQs4JM3qYMzy56a2Dg0sNx/rvga04
EI2+PcFsBk4i9rWy9L2WrnF8gjnlOrZ74+ge/nfd81c+t9kWLvUST1hcKFv0zp68MsLjME
Sw0iRCdlaPti3/5LaJwWKo8ruLypYsVYyADuj8Hceh2bTRos3HS56CM3j2GIgctL9EcB4A
6RwpuNveayQVtP3Ftk37+q4NvAZVH3f/zD86AQqPXgKT/ZvTE73t+yx/+xHR1A==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VrPDy1p3VzQ9s
for <fs@FreeBSD.org>; Fri, 31 May 2024 13:32:34 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 44VDWYIi022768
for <fs@FreeBSD.org>; Fri, 31 May 2024 13:32:34 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 44VDWY4o022767
for fs@FreeBSD.org; Fri, 31 May 2024 13:32:34 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 276341] zfs panic: VERIFY3(rc->rc_count == number) failed
Date: Fri, 31 May 2024 13:32:33 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 15.0-CURRENT
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: mav@FreeBSD.org
X-Bugzilla-Status: In Progress
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-276341-3630-or7VkP50ad@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-276341-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-276341-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276341
--- Comment #27 from Alexander Motin <mav@FreeBSD.org> ---
Martin just done the routine merge, so the patch is in master.
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From nobody Fri May 31 15:17:21 2024
X-Original-To: fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrRYs6x1hz5N3Y7
for <fs@mlmmj.nyi.freebsd.org>; Fri, 31 May 2024 15:17:21 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VrRYs3fBJz53Th
for <fs@FreeBSD.org>; Fri, 31 May 2024 15:17:21 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717168641; a=rsa-sha256; cv=none;
b=C4+zQBevMhWBdrYCI6lA5HIaLa5/Ek2ZYR1psboef8ubIjHu3f74mwiVHBMhI+QPyBOE3b
ubMvFv4K6l91fFJAdWQZV9ihTEbJEFvyIvHSGxyQ9zVgqQrhkq9FnNagnXES5whfj/OKLl
0Y4fskiGWQ3QQ0hlYNFMFwPRH4/4ZfboUr2NjhpOs9V/ub+cp3rAUqAyv+h4kGa7YQRI8a
l5w0/0MMSwSNh8qR+UTl9+LzUUDw2hP071LMOHsjxhIiKhuYF+plRZ7h5b3ee8QpRig158
HJ/E4Bh16JPbvIuBinTXDrhb2f/bCpTubyg1DObidVBczpRmDoxwFObxj4p4dQ==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1717168641;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=VMHi7Tu84Xjp1XYaGChBMDzYqTtC7YopQanXlgRoEbQ=;
b=BGwlx7e4eSUMBEYIlKqLJGkT+8iQLJBU+Riw60ZbM0p0HDjoiqWqHqIiOjjvscytY3+ciH
IBx+Flm0rHLvjGVCUeSA8wcZRzuCnYzoAoz7cdPuQt9gBB3jUSO10Hwj09fLzHoJ8dCelz
2MENV4fhHDgmuaSM0FY+RsbmkB6ygUueoG73WyQHSmg01G5ntU2EIKdKDxW1u93vv4d+oF
+0JKRRlUesfP/VApkaxwqzkx1WhDa8bE8VOTfg59pwdlzt0m9rkAETvLTlkzV6yK2wN396
4++MArH1E5EVlNciqu8JJkFlc2JnVOxVN/EnGROO8A2J2EYOApuK4GeS/mTpaQ==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VrRYs38FBzTNP
for <fs@FreeBSD.org>; Fri, 31 May 2024 15:17:21 +0000 (UTC)
(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 44VFHL2q095155
for <fs@FreeBSD.org>; Fri, 31 May 2024 15:17:21 GMT
(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 44VFHLk6095154
for fs@FreeBSD.org; Fri, 31 May 2024 15:17:21 GMT
(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: fs@FreeBSD.org
Subject: [Bug 278958] zfs panic: page fault in sync_dnodes_task
Date: Fri, 31 May 2024 15:17:21 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 14.0-RELEASE
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: nunziotocci2000@gmail.com
X-Bugzilla-Status: New
X-Bugzilla-Resolution:
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: fs@FreeBSD.org
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-278958-3630-8GB4cNgpno@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-278958-3630@https.bugs.freebsd.org/bugzilla/>
References: <bug-278958-3630@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D278958
--- Comment #4 from nunziotocci2000@gmail.com ---
We have not gotten arround to replacing the drive yet. Meanwhile we have a =
new
panic:
Fatal trap 12: page fault while in kernel mode
cpuid =3D 28; apic id =3D 1c
fault virtual address =3D 0x0
fault code =3D supervisor read data, page not present
instruction pointer =3D 0x20:0xffffffff80eb9c53
stack pointer =3D 0x28:0xfffffe02b60c7b90
frame pointer =3D 0x28:0xfffffe02b60c7bb0
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 47378 (sshd)
rdi: ffffffff81a135a8 rsi: ffffffff81a135a8 rdx: 0000000000000000
rcx: fffff80a9be5dc60 r8: 0000000000000000 r9: fffff80a9be5dc60
rax: 0000000000000000 rbx: fffffe011d24f000 rbp: fffffe02b60c7bb0
r10: ffffffff81a135a8 r11: 0000000000000000 r12: 0000000000000000
r13: fffff8075191c2a0 r14: fffff807e1ed8240 r15: 0000000000000000
trap number =3D 12
panic: page fault
cpuid =3D 28
time =3D 1717134122
KDB: stack backtrace:
#0 0xffffffff80b9009d at kdb_backtrace+0x5d
#1 0xffffffff80b431a2 at vpanic+0x132
#2 0xffffffff80b43063 at panic+0x43
#3 0xffffffff8100c85c at trap_fatal+0x40c
#4 0xffffffff8100c8af at trap_pfault+0x4f
#5 0xffffffff80fe3ac8 at calltrap+0x8
#6 0xffffffff80ebd5ab at vm_map_entry_delete+0x1b
#7 0xffffffff80eb8d0b at vm_map_delete+0x7b
#8 0xffffffff80ebd84c at vm_map_remove+0x9c
#9 0xffffffff80bb5f79 at pipeclose+0x2f9
#10 0xffffffff80bb5970 at pipe_close+0x60
#11 0xffffffff80ae2221 at _fdrop+0x11
#12 0xffffffff80ae543a at closef+0x24a
#13 0xffffffff80ae9194 at closefp_impl+0x64
#14 0xffffffff8100d119 at amd64_syscall+0x109
#15 0xffffffff80fe43db at fast_syscall_common+0xf8
Interestingly enough, the only process listed in the `ps` output of the
core.txt in a `pipewr` state was the `zfs send` of a backup.
Also, myself and my father have been working on our backups scripts and we'=
ve
had other issues with zfs described here:
https://forums.freebsd.org/threads/zfs-dataset-size-is-out-of-control-zfs-s=
end-recv-hangs-on-random-datasets.93549/
--=20
You are receiving this mail because:
You are the assignee for the bug.=
From nobody Fri May 31 19:39:01 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrYN60nv2z5LFcX
for <freebsd-fs@mlmmj.nyi.freebsd.org>; Fri, 31 May 2024 19:39:18 +0000 (UTC)
(envelope-from jdavidlists@gmail.com)
Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VrYN46XF5z4Mq0
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 19:39:16 +0000 (UTC)
(envelope-from jdavidlists@gmail.com)
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=gmail.com header.s=20230601 header.b=W3dBpj8v;
dmarc=pass (policy=none) header.from=gmail.com;
spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::52e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com
Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-6bfd4b88608so1858972a12.1
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 12:39:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1717184355; x=1717789155; darn=freebsd.org;
h=content-transfer-encoding:to:subject:message-id:date:from
:mime-version:from:to:cc:subject:date:message-id:reply-to;
bh=W4QHMAJ1ILfade4FUBkFt7iM1hDVYntxcHUrWnj+2m4=;
b=W3dBpj8viOVe22J0qLweQZQ3hxQm8HyMZBkljIzRS4F77BK5PSPZK1hyRsHrtFfPpW
/9ynJrmx7HzndjraS+zqBGE1ttIniuxoMP+1P8zdMPm2Olj+fg/YNsTuHwmBgkVYbV3B
frtHP++qpd7mQYodUT3O1HxKBStJaIctYjp8dtYJteS74SymyCQcqyYB2c8NffMnqYYC
Yg/geze1yCSL4OxJWAJII/dCAU67WIeNmRNIlVpLRJhefYh0H1noQNYjZgyBkqmkXpFY
fArCLXzxB7mb5djlaluqYCxut2O3hCgVaxIQ2eKcrgfPVfbIdtBOQFGbigaUNqZgtW7G
1qGQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1717184355; x=1717789155;
h=content-transfer-encoding:to:subject:message-id:date:from
:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=W4QHMAJ1ILfade4FUBkFt7iM1hDVYntxcHUrWnj+2m4=;
b=F9R+vdGqraAJWmcb000yReFo/EvhRvcy3dGkJ/+VdQYcd9qTLF5ANVQQ8mREtmeN44
EhWewG+4NKAW5R75F8wGPaqWUX0MvZ73eOUm1RLvxydRT0fmE5MT0FmCKcVkVJJENbj2
xYpvcoS4JjahXsUj1/qgPFczS2jV/VDMaZSdztSYNlck+ZJTkC2RFa7kmPiN6QRCIayt
Oix2X1T6HhXjYWGLB2PiZjpfhXjkTlKxsBEoKVDERHWKHU8BSFCgX8MRrnaUp8i3MmgJ
RUWn26QAM5WJW+1fiOkXqvLBVz5XmZd2Ya55MpQfzcVYuRJe/rGPg4s3205T5a8Cemp+
30Wg==
X-Gm-Message-State: AOJu0YxoDELxYveCTJsb9d2AAEYZ1Ec4lWZe5GpRWYWk0CdRobEwsk6q
VpG821JmKu2e9PI9zYpZI+HDng68q9H2Bq+jn3slK4mSvQdh+0O65rNi1qfXCtQjBFY9eoP2Ny/
lcvseHGGeZ1Knm5udDUTFVi3rGkA+Q2AE
X-Google-Smtp-Source: AGHT+IEN8dQPEvFWFaj5qSKewIU+4IJvbdRxFKX/rN14n2VEgmHyHi6w5hj8iGumxIYvJbFtP4eXKMNSKiC4IMZy1Kc=
X-Received: by 2002:a17:90a:ac03:b0:2b9:e3:3561 with SMTP id
98e67ed59e1d1-2c1dc5d8b76mr2555527a91.43.1717184354591; Fri, 31 May 2024
12:39:14 -0700 (PDT)
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
From: J David <j.david.lists@gmail.com>
Date: Fri, 31 May 2024 15:39:01 -0400
Message-ID: <CABXB=RTH6s25EMQzDbg8WPr1hKtjKOJGVh3no0TFcNC=ufwXCw@mail.gmail.com>
Subject: NFSv4 hangs on 13.3
To: FreeBSD FS <freebsd-fs@freebsd.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.99 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
NEURAL_HAM_LONG(-1.00)[-0.999];
NEURAL_HAM_SHORT(-1.00)[-0.995];
DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36];
MIME_GOOD(-0.10)[text/plain];
RCVD_TLS_LAST(0.00)[];
TO_MATCH_ENVRCPT_ALL(0.00)[];
ARC_NA(0.00)[];
FROM_HAS_DN(0.00)[];
MIME_TRACE(0.00)[0:+];
RCPT_COUNT_ONE(0.00)[1];
FREEMAIL_ENVFROM(0.00)[gmail.com];
FREEMAIL_FROM(0.00)[gmail.com];
TO_DN_ALL(0.00)[];
DKIM_TRACE(0.00)[gmail.com:+];
DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
MID_RHS_MATCH_FROMTLD(0.00)[];
PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
FROM_EQ_ENVFROM(0.00)[];
TAGGED_FROM(0.00)[];
MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
RCVD_COUNT_ONE(0.00)[1];
ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
MISSING_XM_UA(0.00)[];
RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52e:from]
X-Rspamd-Queue-Id: 4VrYN46XF5z4Mq0
In an attempt to narrow down the problems we have with NFS v4.2, we've
set up a FreeBSD 13.3 server that mirrors some read-only data from our
Linux NFS servers.
We are still observing occasional hangs on the FreeBSD 13.3 client machines=
.
When these hangs occur, anything that touches the mountpoint
(including "umount -N") hangs indefinitely. There is nothing in dmesg.
The only even slightly informative data I could gather is an "ls
/mount/point" kstack:
$ sudo procstat -kk 94676
PID TID COMM TDNAME KSTACK
94676 976876 ls - mi_switch+0xbf
sleeplk+0xea lockmgr_slock_hard+0x3a5 nfs_lock+0x29 vop_sigdefer+0x2a
_vn_lock+0x47 vfs_cache_root+0x9d vfs_root_sigdefer+0x35 lookup+0x88a
namei+0x24a kern_statat+0xf8 sys_fstatat+0x27 amd64_syscall+0x110
fast_syscall_common+0xf8
During this time, the NFS server continues serving other clients, and
the affected client can access other NFS servers without hanging.
The "nfsstat -m" for this mountpoint looks like this:
nfsv4,minorversion=3D2,oneopenown,tcp,resvport,nconnect=3D1,hard,cto,nolock=
d,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=
=3D60,negnametimeo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536,rea=
dahead=3D1,wcommitsize=3D16777216,timeout=3D120,retrans=3D2147483647
with these mount flags in /etc/fstab:
ro,nfsv4,minorversion=3D2,tcp,nosuid,noatime,nolockd,oneopenown
Is there anything more we can do to help identify this issue?
Thanks for any advice!
From nobody Fri May 31 20:11:20 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrZ5J71Gvz5LJGW
for <freebsd-fs@mlmmj.nyi.freebsd.org>; Fri, 31 May 2024 20:11:32 +0000 (UTC)
(envelope-from rick.macklem@gmail.com)
Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VrZ5J2QLqz4Rwf
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 20:11:32 +0000 (UTC)
(envelope-from rick.macklem@gmail.com)
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=gmail.com header.s=20230601 header.b=UdCO7XoF;
dmarc=pass (policy=none) header.from=gmail.com;
spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::531 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com
Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-6c702226b0aso203722a12.1
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 13:11:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1717186291; x=1717791091; darn=freebsd.org;
h=content-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:from:to:cc:subject:date
:message-id:reply-to;
bh=rVJsNG2y9MiYAXqao7yD74ScjdqXl7f0RY1IaJ57rNQ=;
b=UdCO7XoFI6ZBzgSVytjw0pX5OsiwoIs0jyPCC0m3FeMqDa8J9eFLNNtqgdyQKhlUpg
q9UdaN+1ZQcHbt0rMZWEu+U3ADcKPDTXEA6QjvRq4aIeCCrF/UqyKeKPZuYmGBt0NvzC
UvA7bSS6svQ8Bi+x8nehVi0jfRDKKkhQ6Ob/c98g6Z7q3x/GCJ0jynKdP8UmGVAd5Iba
YtFt6M8WkGzqFsTeWiJU33CfI1P7Qw/7MugvfsjBtEH8NBRSK971hLEEUo3uSDF0aL08
gJgyEQCR0ko7Hxeu8vf4X2KSzS516IXy39eVsvF94r63Khx+6rIAbxUXjT1T1WrnXueB
bpTw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1717186291; x=1717791091;
h=content-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=rVJsNG2y9MiYAXqao7yD74ScjdqXl7f0RY1IaJ57rNQ=;
b=efbodtzuLTIwdWQZx78YFeuFduhv650NuLjPTy1sraUgiqA/23Rg3/XSIECi97ef07
R4qpFbcX7RiVLkWihFSEYWobguEur8X7tiT8y/meJYw5LmW+ofCkhNiNcAgZL8bghFc+
LM3uNa5o09iTacjYJQW8tTKg6nKR7bS5sPqX0XCfEV/I8cvk+2AnZiuB+MAkVZutDs4a
kg/eK8yGPk4N5uApYrl36J0pl0jiNA/7kAo3dnTFSFkdYREfngoG+SlHxypfTQI/weMt
rT/sQ9aFzojs8+QRvaOpu+ltbAhDQIuyCQBr76frUSw93r8WwBDNrnPtIcaoNHhLRdIi
GnlQ==
X-Gm-Message-State: AOJu0YxZ9iWoLEXJchRzOM1K/StcsTAE/OC7s8zKNq2sw7sEZKE9rFL8
ct+gzIiZscNvN2G8LlPFlq3g4vnJiJ/8oUHXLqqeuh1ARP5unHtKrz3IgsA9nNZq2FzqSUrnsiq
iT+ceX0PWDfyC6zyKwfZkxm+UFEL0
X-Google-Smtp-Source: AGHT+IH4G+3Bu+INPC54DveA5bmcnrJ9/bsFRZJJLyAbx7BAMf8/Yck6SX3/PppnREkIMF+Tn2TMEgH+hU0bXyzqYgc=
X-Received: by 2002:a17:90a:c7cf:b0:2b0:e497:56f2 with SMTP id
98e67ed59e1d1-2c1c43ee4c1mr4354323a91.10.1717186290769; Fri, 31 May 2024
13:11:30 -0700 (PDT)
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <CABXB=RTH6s25EMQzDbg8WPr1hKtjKOJGVh3no0TFcNC=ufwXCw@mail.gmail.com>
In-Reply-To: <CABXB=RTH6s25EMQzDbg8WPr1hKtjKOJGVh3no0TFcNC=ufwXCw@mail.gmail.com>
From: Rick Macklem <rick.macklem@gmail.com>
Date: Fri, 31 May 2024 13:11:20 -0700
Message-ID: <CAM5tNy7YSTDwVm_oMcu=1PqRyXtTmvzTF058nd1WCXyCHwrtTQ@mail.gmail.com>
Subject: Re: NFSv4 hangs on 13.3
To: J David <j.david.lists@gmail.com>
Cc: FreeBSD FS <freebsd-fs@freebsd.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.72 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
NEURAL_HAM_LONG(-1.00)[-0.999];
NEURAL_HAM_SHORT(-0.72)[-0.722];
DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36];
R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
MIME_GOOD(-0.10)[text/plain];
RCVD_TLS_LAST(0.00)[];
FREEMAIL_ENVFROM(0.00)[gmail.com];
TO_DN_ALL(0.00)[];
FREEMAIL_FROM(0.00)[gmail.com];
MIME_TRACE(0.00)[0:+];
TAGGED_FROM(0.00)[];
RCPT_COUNT_TWO(0.00)[2];
ARC_NA(0.00)[];
FREEMAIL_TO(0.00)[gmail.com];
FROM_HAS_DN(0.00)[];
MISSING_XM_UA(0.00)[];
DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
TO_MATCH_ENVRCPT_SOME(0.00)[];
FROM_EQ_ENVFROM(0.00)[];
DKIM_TRACE(0.00)[gmail.com:+];
MID_RHS_MATCH_FROMTLD(0.00)[];
TAGGED_RCPT(0.00)[];
MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
RCVD_COUNT_ONE(0.00)[1];
ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::531:from]
X-Rspamd-Queue-Id: 4VrZ5J2QLqz4Rwf
On Fri, May 31, 2024 at 12:39=E2=80=AFPM J David <j.david.lists@gmail.com> =
wrote:
>
> In an attempt to narrow down the problems we have with NFS v4.2, we've
> set up a FreeBSD 13.3 server that mirrors some read-only data from our
> Linux NFS servers.
>
> We are still observing occasional hangs on the FreeBSD 13.3 client machin=
es.
>
> When these hangs occur, anything that touches the mountpoint
> (including "umount -N") hangs indefinitely. There is nothing in dmesg.
>
> The only even slightly informative data I could gather is an "ls
> /mount/point" kstack:
>
> $ sudo procstat -kk 94676
> PID TID COMM TDNAME KSTACK
> 94676 976876 ls - mi_switch+0xbf
> sleeplk+0xea lockmgr_slock_hard+0x3a5 nfs_lock+0x29 vop_sigdefer+0x2a
> _vn_lock+0x47 vfs_cache_root+0x9d vfs_root_sigdefer+0x35 lookup+0x88a
> namei+0x24a kern_statat+0xf8 sys_fstatat+0x27 amd64_syscall+0x110
> fast_syscall_common+0xf8
>
> During this time, the NFS server continues serving other clients, and
> the affected client can access other NFS servers without hanging.
>
> The "nfsstat -m" for this mountpoint looks like this:
>
> nfsv4,minorversion=3D2,oneopenown,tcp,resvport,nconnect=3D1,hard,cto,nolo=
ckd,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametim=
eo=3D60,negnametimeo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536,r=
eadahead=3D1,wcommitsize=3D16777216,timeout=3D120,retrans=3D2147483647
>
> with these mount flags in /etc/fstab:
>
> ro,nfsv4,minorversion=3D2,tcp,nosuid,noatime,nolockd,oneopenown
>
> Is there anything more we can do to help identify this issue?
Collect the output of:
# ps axHl
and
# procstat -kk -a
What you show above is just a thread waiting for a lock on an NFS vnode.
What we need to figure out is what thread is holding the lock on that vnode
while waiting for something else.
rick
>
> Thanks for any advice!
>
From nobody Fri May 31 21:10:34 2024
X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrbPg608Mz5LNc8
for <freebsd-fs@mlmmj.nyi.freebsd.org>; Fri, 31 May 2024 21:10:47 +0000 (UTC)
(envelope-from rick.macklem@gmail.com)
Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "smtp.gmail.com", Issuer "WR4" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4VrbPg1QxYz4YZ1
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 21:10:47 +0000 (UTC)
(envelope-from rick.macklem@gmail.com)
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=gmail.com header.s=20230601 header.b="M/VSuNcw";
dmarc=pass (policy=none) header.from=gmail.com;
spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1031 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com
Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-2bfffa3c748so1948507a91.3
for <freebsd-fs@freebsd.org>; Fri, 31 May 2024 14:10:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1717189845; x=1717794645; darn=freebsd.org;
h=content-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:from:to:cc:subject:date
:message-id:reply-to;
bh=jkbO8XinZCFgu/g6mociXTje8yr1zevRMJClucVvgZc=;
b=M/VSuNcwfmOjTLWUFh7XMLu/aDeW8OBZ98gK/dvqu+7dObAFRA+hp5nX8pLg6tNGYt
6kXnHYmxF1+TlqSpqdP4r69Pz6gtStF45Uz8RBUy+pgFz/XLD6TCKNM7it2rRMwqrpYf
IWvK4rNP6ZsSWxLu3wa430qi4Dgpi4usfvPRArdb5ZCMeMg2L6Sk40egBWwfFsDvdD45
C2oxO/CrShYw5amXwDh8BiiUiA8QPplpVxvjsnTivRS7Lc9qTebzeNJomizwTXOYvAuM
23UUmkEyGDcI0BWTH0pNH77o+4/qdjOYkL+8RhUUsqZm9QVM2+cGB8lGezaue666HTWL
LULA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1717189845; x=1717794645;
h=content-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=jkbO8XinZCFgu/g6mociXTje8yr1zevRMJClucVvgZc=;
b=gM8bUdWPp4AuLB4oewSb06sA0Yatdt/S3en1xrPeR5/ouleDDpc5w2WV/poomK8/8w
mtuNB5e+OCPjm26YEtFF+aWd7e81STZo1TJ2fI41lm/weFX9502PdViLOupxvWo7WCuP
6fD1Vt82L4Mi7QpOj/t3TdV6DtqK0/Ry2poCQkAed2UHM0KXRhhM8fkLTH5b0YLfETJe
P3YtpNkN3e8fM2vgMfMzcWLgz++OSUdNOM2AHmRJKsrUqiLAdTtm8Jm/1e4zSiQhwpF9
rVF/Ru4DoiZJa5qmMgGJODbVLcqObQ9JpTpi9Obv+C/fj2HNPDWOdvlGB3PtBzlEQEGt
BRcg==
X-Gm-Message-State: AOJu0Yz0HSGCCmKfgXlt/ipzsCPy6e9sd7nyy+UkIqfxC0GPc+IM9l1j
slHiTXdw4FMJre6dgtK0LuWP8iUxPwgCUFWblzsmlc1zX+FGWyyvQP8KEldAYXkT/OLrMsFyZg8
yqvUNuV82pmgSNdZchAOioRCl9g==
X-Google-Smtp-Source: AGHT+IEMZ1075G0aBFqUCwADObXHaP2bshaTOJNXncgfu4AQnzqFFDAN4wTyXGIJGKJd6n7NwfWgPer4d9lZzuXK278=
X-Received: by 2002:a17:90a:d593:b0:2c1:bb0c:9af4 with SMTP id
98e67ed59e1d1-2c1dc4b479cmr3355620a91.0.1717189845033; Fri, 31 May 2024
14:10:45 -0700 (PDT)
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
References: <CABXB=RTH6s25EMQzDbg8WPr1hKtjKOJGVh3no0TFcNC=ufwXCw@mail.gmail.com>
<CAM5tNy7YSTDwVm_oMcu=1PqRyXtTmvzTF058nd1WCXyCHwrtTQ@mail.gmail.com>
In-Reply-To: <CAM5tNy7YSTDwVm_oMcu=1PqRyXtTmvzTF058nd1WCXyCHwrtTQ@mail.gmail.com>
From: Rick Macklem <rick.macklem@gmail.com>
Date: Fri, 31 May 2024 14:10:34 -0700
Message-ID: <CAM5tNy4x3KQAX9D9EafjFQbdDqRUPus6mzO7mrc2ZSM_Ks=kOA@mail.gmail.com>
Subject: Re: NFSv4 hangs on 13.3
To: J David <j.david.lists@gmail.com>
Cc: FreeBSD FS <freebsd-fs@freebsd.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.99 / 15.00];
NEURAL_HAM_MEDIUM(-1.00)[-1.000];
NEURAL_HAM_LONG(-1.00)[-0.999];
NEURAL_HAM_SHORT(-0.99)[-0.994];
DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c];
R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
MIME_GOOD(-0.10)[text/plain];
TO_DN_ALL(0.00)[];
FREEMAIL_ENVFROM(0.00)[gmail.com];
RCVD_TLS_LAST(0.00)[];
FREEMAIL_FROM(0.00)[gmail.com];
MIME_TRACE(0.00)[0:+];
TAGGED_FROM(0.00)[];
RCPT_COUNT_TWO(0.00)[2];
ARC_NA(0.00)[];
FREEMAIL_TO(0.00)[gmail.com];
FROM_HAS_DN(0.00)[];
MISSING_XM_UA(0.00)[];
DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org];
TO_MATCH_ENVRCPT_SOME(0.00)[];
FROM_EQ_ENVFROM(0.00)[];
DKIM_TRACE(0.00)[gmail.com:+];
MID_RHS_MATCH_FROMTLD(0.00)[];
TAGGED_RCPT(0.00)[];
MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org];
RCVD_COUNT_ONE(0.00)[1];
ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1031:from]
X-Rspamd-Queue-Id: 4VrbPg1QxYz4YZ1
On Fri, May 31, 2024 at 1:11=E2=80=AFPM Rick Macklem <rick.macklem@gmail.co=
m> wrote:
>
> On Fri, May 31, 2024 at 12:39=E2=80=AFPM J David <j.david.lists@gmail.com=
> wrote:
> >
> > In an attempt to narrow down the problems we have with NFS v4.2, we've
> > set up a FreeBSD 13.3 server that mirrors some read-only data from our
> > Linux NFS servers.
> >
> > We are still observing occasional hangs on the FreeBSD 13.3 client mach=
ines.
> >
> > When these hangs occur, anything that touches the mountpoint
> > (including "umount -N") hangs indefinitely. There is nothing in dmesg.
> >
> > The only even slightly informative data I could gather is an "ls
> > /mount/point" kstack:
> >
> > $ sudo procstat -kk 94676
> > PID TID COMM TDNAME KSTACK
> > 94676 976876 ls - mi_switch+0xbf
> > sleeplk+0xea lockmgr_slock_hard+0x3a5 nfs_lock+0x29 vop_sigdefer+0x2a
> > _vn_lock+0x47 vfs_cache_root+0x9d vfs_root_sigdefer+0x35 lookup+0x88a
> > namei+0x24a kern_statat+0xf8 sys_fstatat+0x27 amd64_syscall+0x110
> > fast_syscall_common+0xf8
> >
> > During this time, the NFS server continues serving other clients, and
> > the affected client can access other NFS servers without hanging.
> >
> > The "nfsstat -m" for this mountpoint looks like this:
> >
> > nfsv4,minorversion=3D2,oneopenown,tcp,resvport,nconnect=3D1,hard,cto,no=
lockd,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,namet=
imeo=3D60,negnametimeo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536=
,readahead=3D1,wcommitsize=3D16777216,timeout=3D120,retrans=3D2147483647
> >
> > with these mount flags in /etc/fstab:
> >
> > ro,nfsv4,minorversion=3D2,tcp,nosuid,noatime,nolockd,oneopenown
> >
> > Is there anything more we can do to help identify this issue?
> Collect the output of:
> # ps axHl
> and
> # procstat -kk -a
Oh, and
# netstat -a
to see what the TCP connections are up to.
rick
>
> What you show above is just a thread waiting for a lock on an NFS vnode.
> What we need to figure out is what thread is holding the lock on that vno=
de
> while waiting for something else.
>
> rick
>
> >
> > Thanks for any advice!
> >
From nobody Sun Jun 2 21:00:25 2024
X-Original-To: fs@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vsq4p34brz5JxDr
for <fs@mlmmj.nyi.freebsd.org>; Sun, 02 Jun 2024 21:00:26 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4Vsq4p0LjDz4mck
for <fs@FreeBSD.org>; Sun, 2 Jun 2024 21:00:26 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717362026; a=rsa-sha256; cv=none;
b=d9Pg1lI/+VsnlExqjlJ6J0ppUg+i0X7u9AZeqy5qC9StTjLzyhjFx/2/F0ZaNOb4Dglq2F
QGlGckYi8smuUxRA9RPq45qZ9+iSYzS+0u7XvJVj+h5zX9fC406eSoWTlhqm9x5SU8Qaps
ivzSqPcW8IwDf+0otuzUi4to8Fn2hJH9ZO2wLkN78k0SoY0hpOV3k1leOjQldD9Nm9NMq0
t6tswWJogzE+CCXXuF+sC3J17yDsttJqFRCELU3eTWnq4fsJQ/3wTgftkd6nJGb/Wk/0Zl
OwnVYadqJz5v5aMgorO/n5BwexsWyaKplAmT9M9t/zlFdjz5hfsROcSyranBpw==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1717362026;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=ym4kJcjZo2MvKfzncwkcXzDUrRPfwn+hbSnYS2gJRoY=;
b=UfadVIddX82jubQR5XeYxaqtbDKboZZVol2HzN0SiSUPhAMOj2FHqyzVV5OiiMUUd2xO7M
Wu+2ZL6rtnwS6eZrxYukbO4OIhqdQe3uA605TrY0Wwk98lLKDe8ku+qpJ5Y1irJxVihehW
A6ciOrgFezVA9HSbGXaeaEoUhrBQSNIIr9jVsQKkkYm7TSJxUyc/uvpEvoKLYtg7R7IcfT
s8bbVEwWHJirMHyscwiirYt5zWsYwRj60dJsbnZ1t4N+q1mBxSBlNnMTbyhVr5rt5U4amI
/9OWardLBDmJmx2XH0RHdMeWUfhls1j7JZh2LkEyKfGRGY62rN3Jpq8zFzDbEg==
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Vsq4n6nmQz15sB
for <fs@FreeBSD.org>; Sun, 2 Jun 2024 21:00:25 +0000 (UTC)
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 452L0P2P090751
for <fs@FreeBSD.org>; Sun, 2 Jun 2024 21:00:25 GMT
(envelope-from bugzilla-noreply@FreeBSD.org)
Received: (from bugzilla@localhost)
by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 452L0Pdp090750
for fs@FreeBSD.org; Sun, 2 Jun 2024 21:00:25 GMT
(envelope-from bugzilla-noreply@FreeBSD.org)
Message-Id: <202406022100.452L0Pdp090750@kenobi.freebsd.org>
X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f
From: bugzilla-noreply@FreeBSD.org
To: fs@FreeBSD.org
Subject: Problem reports for fs@FreeBSD.org that need special attention
Date: Sun, 2 Jun 2024 21:00:25 +0000
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-fs
List-Help: <mailto:freebsd-fs+help@freebsd.org>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org>
Sender: owner-freebsd-fs@FreeBSD.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="17173620254.e76A.86741"
Content-Transfer-Encoding: 7bit
--17173620254.e76A.86741
Date: Sun, 2 Jun 2024 21:00:25 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
To view an individual PR, use:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id).
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.
Status | Bug Id | Description
------------+-----------+---------------------------------------------------
Open | 231794 | zfs: Panic due to ARC related KVA memory exhausti
Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open | 266409 | 13.1-RELEASE amd64 crashes in: sa_handle_destroy
Open | 268162 | zfskeys_enable: each successful load of a key is
Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open | 271384 | zfs_load is not suitably documented
Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =
7 problems total for which you should take action.
--17173620254.e76A.86741
Date: Sun, 2 Jun 2024 21:00:25 +0000
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"
<pre style="font-family: monospace;">
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.
Status | Bug Id | Description
------------+-----------+---------------------------------------------------
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231794">231794</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231794">zfs: Panic due to ARC related KVA memory exhausti</a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251035">251035</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251035">ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W</a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266409">266409</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266409">13.1-RELEASE amd64 crashes in: sa_handle_destroy </a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268162">268162</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268162">zfskeys_enable: each successful load of a key is </a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269503">269503</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269503">docs.freebsd.org: default vfs.zfs.arc.meta_limit</a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271384">271384</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271384">zfs_load is not suitably documented</a>
Open | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226130">226130</a> | <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226130">ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =</a>
7 problems total for which you should take action.
</pre>
--17173620254.e76A.86741--