From nobody Fri Feb 9 22:41:53 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TWpkr5pq0z5BTDq for ; Fri, 9 Feb 2024 22:42:12 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TWpkr3zfkz4Lhy for ; Fri, 9 Feb 2024 22:42:12 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-53fbf2c42bfso1008744a12.3 for ; Fri, 09 Feb 2024 14:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707518531; x=1708123331; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Vc5hlh0C/iI+/RibW/yHYG1ocRxfyghTuBq2n+Xog2M=; b=QCx02570j+03wrtBhkb1PtWQvHk5y9/a8aXTmltyCFqtijmgOSABN0ujVCf6Hvm0zy awsm9LpPBKb2A7MEJhfzgHKn2+UGCj1MvC8cEbnERakh5gB1cvFbP9C/rsU7XAuYjVQl EoPorRiuFgjn8QwducorNkEx5l9M4wvJxNRZL2mZ3jUykFDgTal4WpwoEfn50mVwkZN7 lhbdYXFq4NJElm+98zBoxhRix6hclE58DMnijESMFU6LAPDcJ8ScBi961lEOPsLViixX AdyK2HTHEIkIclkOF+0oQcxQ9UwSKfIZv6z91EAm0Z8uH52OB7psuRLjVwitYZHAYURM 861w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707518531; x=1708123331; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vc5hlh0C/iI+/RibW/yHYG1ocRxfyghTuBq2n+Xog2M=; b=co5EKllnPvnzTMm3cLy+DyBkvFhWb1+HSvwX1TmaMoJvikYAwDc5E+KkNf9fgtrRp7 +lA39ZLowC5n+MdlGlhA/ZaUYtVfH5UBfy6E5ZShL8OcK2Yy2FKBKWSzVyqCyMzDaAoJ siolPKNWM4QS50fOfaVTPen920pF0/pWwY6ITQ17+0m7Q0p+utlat4xjDgg7fcYwn8Z6 66HHxiims0iuRdW+E0bz0CwB5HcuHYF7fvf4tkCVByIn4Nvt7BzYCGrcQODPyWhEnCZx 1OKaaU6QLrEDZen0bTcN1ODeg96B94O/ULIkVvacs+9Q4av/+RUKFHw6mDzI508GT12t IPSQ== X-Gm-Message-State: AOJu0YzOvu2X9TIYGpIxzCXVe6XA5CgEq2ycNNlbnRYDPjTy5OH4nzbl nzRnhhS32LQQGeK9cUeegbZCXncXjsvXITc+hkZUHdxnPWKEJLlGyIkwCXGsqHAohcr5j4pe9A/ QiD+M7LodxNc2wV7q/6pRDDvMcypRtwE= X-Google-Smtp-Source: AGHT+IEY3KyvRWqWcv21TRXCR9gmmuTk8L1Mu0q+qD4v0XG9C4Zk9h1xPN/r0I5ZXy7oMHyn4GUQX9P+nAwfkG1NyHI= X-Received: by 2002:a05:6a21:9188:b0:19e:8b31:f1ae with SMTP id tp8-20020a056a21918800b0019e8b31f1aemr662751pzb.29.1707518530865; Fri, 09 Feb 2024 14:42:10 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Fri, 9 Feb 2024 14:41:53 -0800 Message-ID: Subject: Re: FreeBSD panics possibly caused by nfs clients To: "Matthew L. Dailey" Cc: "freebsd-current@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4TWpkr3zfkz4Lhy X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Fri, Feb 9, 2024 at 10:23=E2=80=AFAM Matthew L. Dailey wrote: > > I had my first kernel panic with a KASAN kernel after only 01:27. This > first panic was a "double fault," which isn't anything we've seen > previously - usually we've seen trap 9 or trap 12, but sometimes others. > Based on the backtrace, it definitely looks like KASAN caught something, > but I don't have the expertise to know if this points to anything > specific. From the backtrace, it looks like this might have originated > in ipfw code. I do not know anything about the KASAN kernel, but I do see it increases the kernel stack to 6 pages and when I see a double fault, it is usually a kernel stack being blown. You might check what it is: # sysctl -a | fgrep kstack_pages If it is already 6, you could try bumping it to 8. Just add this line to /boot/loader.conf kern.kstack_pages=3D8 and reboot. rick > > Please let me know what other info I can provide or what I can do to dig > deeper. > > Thanks!! > > Panic message: > [5674] Fatal double fault > [5674] rip 0xffffffff812f6e32 rsp 0xfffffe014677afe0 rbp 0xfffffe014677b4= 30 > [5674] rax 0x1fffffc028cef620 rdx 0xf2f2f2f8f2f2f2f2 rbx 0x1 > [5674] rcx 0xdffff7c000000000 rsi 0xfffffe004086a4a0 rdi 0xf8f8f8f8f2f2f2= f8 > [5674] r8 0xf8f8f8f8f8f8f8f8 r9 0x162a r10 0x835003002d3a64e1 > [5674] r11 0 r12 0xfffff78028cef620 r13 0xfffffe004086a440 > [5674] r14 0xfffffe01488c0560 r15 0x26f40 rflags 0x10006 > [5674] cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b > [5674] fsbase 0x95d1d81a130 gsbase 0xffffffff84a14000 kgsbase 0 > [5674] cpuid =3D 4; apic id =3D 08 > [5674] panic: double fault > [5674] cpuid =3D 4 > [5674] time =3D 1707498420 > [5674] KDB: stack backtrace: > [5674] Uptime: 1h34m34s > > Backtrace: > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 > #1 doadump (textdump=3D) at > /usr/src/sys/kern/kern_shutdown.c:405 > #2 0xffffffff8128b7dc in kern_reboot (howto=3Dhowto@entry=3D260) > at /usr/src/sys/kern/kern_shutdown.c:526 > #3 0xffffffff8128c000 in vpanic ( > fmt=3Dfmt@entry=3D0xffffffff82589a00 "double fault", > ap=3Dap@entry=3D0xfffffe0040866de0) at > /usr/src/sys/kern/kern_shutdown.c:970 > #4 0xffffffff8128bd75 in panic (fmt=3D0xffffffff82589a00 "double > fault") > at /usr/src/sys/kern/kern_shutdown.c:894 > #5 0xffffffff81c4b335 in dblfault_handler (frame=3D) > at /usr/src/sys/amd64/amd64/trap.c:1012 > #6 > #7 0xffffffff812f6e32 in sched_clock (td=3Dtd@entry=3D0xfffffe01488c0560= , > cnt=3Dcnt@entry=3D1) at /usr/src/sys/kern/sched_ule.c:2601 > #8 0xffffffff8119e2a7 in statclock (cnt=3Dcnt@entry=3D1, > usermode=3Dusermode@entry=3D0) at /usr/src/sys/kern/kern_clock.c:760 > #9 0xffffffff8119fb67 in handleevents (now=3Dnow@entry=3D24371855699832, > fake=3Dfake@entry=3D0) at /usr/src/sys/kern/kern_clocksource.c:195 > #10 0xffffffff811a10cc in timercb (et=3D, arg=3D) > at /usr/src/sys/kern/kern_clocksource.c:353 > #11 0xffffffff81dcd280 in lapic_handle_timer (frame=3D0xfffffe014677b750) > at /usr/src/sys/x86/x86/local_apic.c:1343 > #12 > #13 __asan_load8_noabort (addr=3D18446741880219689232) > at /usr/src/sys/kern/subr_asan.c:1113 > #14 0xffffffff851488b8 in ?? () from /boot/thayer/ipfw.ko > #15 0xfffffe0100000000 in ?? () > #16 0xffffffff8134dcd5 in pcpu_find (cpuid=3D1238425856) > at /usr/src/sys/kern/subr_pcpu.c:286 > #17 0xffffffff85151f6f in ?? () from /boot/thayer/ipfw.ko > #18 0x0000000000000000 in ?? ()