From nobody Mon Sep 23 21:14:42 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XCG3P7432z5WWhf for ; Mon, 23 Sep 2024 21:14:57 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XCG3N4yS0z4jpk for ; Mon, 23 Sep 2024 21:14:56 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZllJaAtF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::436 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-718d704704aso4191970b3a.3 for ; Mon, 23 Sep 2024 14:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727126095; x=1727730895; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0qhpXe4GeeBFFbJiZbv4xglWQOLsclZT3KNqIRpfeks=; b=ZllJaAtFLi15A79VqCBjDfHdSME3dsY6PoBmT1GnwgQ/LLzdnMfPFxyhzG7PeFKzEV 2yUXiGlfFuOyTonNcX5y7vGMD6F7d5a7UnS/4L19YRAmvM3im+YAaRdIyEN1wozHMsVF UzL0QyPol+2lZYqqj4FXTaL/9ShN8bRnN+FHQx3xlXybKWO4GBmzfsqtpG0QIxwv5u/4 CA7txXwd2VP/Vhwpa1hrebryE4YP29nmLkxsXVGShPa7ChaFXoVm1rcuC2LZKdPvLBE/ YSKUkL2vSGN5xexI1raQZsyf1jaJ0oFaaL3FACp6ACl/mHYDCO9XPyVu44DOf3EA03MH +PcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727126095; x=1727730895; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0qhpXe4GeeBFFbJiZbv4xglWQOLsclZT3KNqIRpfeks=; b=N0A17KjtckQq2BxzI1xmKNmD/ggUbpLmoLmsmw9pkG/PbhQkwfZ+NLTFMcoiLC0nli yy5BaAhhfhckUR0PzX2Mhs/PxtA+HBofSYDYssi5eosOFJhFQE/p/doeTH0FRmOkx0aR SadS7dc6wqAdODsf2kMSh58dx+ylEQPYsfzaGbwhqfXBRP80E9ZczkK6+sn7QzEIWs6H emKdOPz9NSafxG1RDuempt7G97E4+fW0X9Efh3WO32MLvkvyDS9LWaaN8Ri7M3jvfl/3 Jgr8veoqS+QRg/uhAWyNlr0ETVlRGMs/xtwBbQGTJwcrZeGb1GAfxYa+cy03wZJJPPL6 2Wng== X-Gm-Message-State: AOJu0Yx4kd3I0WWOAHPx2jqnEHoMlxZy25125wYNIUD1xRvb25xdPt7x IECi5Q3CAGz4Fif8vEOL7Ca+xSjuxrSHuEzUyTavc88nlF4r8aTlsQ6PLsBMi/gIQ6aCBjSvStS tKKEHTMZuH52vd7uOwFh0QI/fTYo= X-Google-Smtp-Source: AGHT+IHb/ZJhWl9KrgHToVJYf9HMTABj0e838Prl+fgw+Az//mLFMVQQmfFilp5tw9F/UxBB7Shav8+MxnbxJg1ODwA= X-Received: by 2002:a05:6a00:ac1:b0:714:15ff:a2a4 with SMTP id d2e1a72fcca58-7199c96c440mr18137769b3a.13.1727126094521; Mon, 23 Sep 2024 14:14:54 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Mon, 23 Sep 2024 17:14:42 -0400 Message-ID: Subject: Re: panic: nfsv4root ref cnt cpuid = 1 To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.80 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; NEURAL_HAM_MEDIUM(-0.82)[-0.817]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::436:from] X-Rspamd-Queue-Id: 4XCG3N4yS0z4jpk X-Spamd-Bar: --- We've had some other kernel panics that may be related to this. At the very least, the call stack is the same up to nfsrpc_lookup+0x87f. Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff809da260 stack pointer = 0x28:0xfffffe0111f18438 frame pointer = 0x28:0xfffffe0111f18470 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 14676 (sh) rdi: 0000000000000028 rsi: fffff80071cee200 rdx: 0000000000000000 rcx: 0000000000000000 r8: 0000000000000032 r9: fffffe0111f19000 rax: 0000000000000000 rbx: fffff80161e4b000 rbp: fffffe0111f18470 r10: 00000000000001f4 r11: fffff8024e1e5760 r12: 0000000000000000 r13: 0000000000000000 r14: fffff80071cee200 r15: 0000000000000000 trap number = 12 panic: page fault cpuid = 0 time = 1727064012 KDB: stack backtrace: #0 0xffffffff80b7fefd at kdb_backtrace+0x5d #1 0xffffffff80b32bd1 at vpanic+0x131 #2 0xffffffff80b32a93 at panic+0x43 #3 0xffffffff8100091b at trap_fatal+0x40b #4 0xffffffff81000966 at trap_pfault+0x46 #5 0xffffffff80fd6d48 at calltrap+0x8 #6 0xffffffff809f9eef at nfsrpc_lookup+0x87f #7 0xffffffff80a0e2fd at nfs_lookup+0x43d #8 0xffffffff80c0341a at vop_sigdefer+0x2a #9 0xffffffff8302c3a7 at null_lookup+0xc7 #10 0xffffffff80c08745 at vfs_lookup+0x425 #11 0xffffffff80c079b8 at namei+0x238 #12 0xffffffff80c2d2da at vn_open_cred+0x53a #13 0xffffffff80c239a8 at openatfp+0x268 #14 0xffffffff80c236b8 at sys_open+0x28 #15 0xffffffff810011c0 at amd64_syscall+0x100 #16 0xffffffff80fd765b at fast_syscall_common+0xf8 I don't know if this gives any more insight or confirmation to your theory about the problem, but it seems worth sharing. We got three of these panics (and two of the first kind) on different machines in the past 24 hours, so I'll definitely at least be experimenting with bulk NFS mounts instead of one NFS + nullfs bulk mounts in addition to trying the patch. Thanks!