Date: Mon, 19 Feb 2024 14:50:19 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: "Matthew L. Dailey" <Matthew.L.Dailey@dartmouth.edu> Cc: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: FreeBSD panics possibly caused by nfs clients Message-ID: <CAM5tNy5CbGNOWcpp43uTY9=iha2aAYdLcCNWJLsG0=Fvg=dZGA@mail.gmail.com> In-Reply-To: <04d1f2e1-021e-42fb-9732-94fa98fd05fc@dartmouth.edu> References: <c5d44484-8660-4b8b-a379-79423cb208f6@dartmouth.edu> <ZcZNDtN1nNJmo8cS@nuc> <c9eca81a-9eff-4b17-9928-bee2c79cef8f@dartmouth.edu> <b3243928-4d66-4c5e-9745-254d57f1cc5e@dartmouth.edu> <ZcaWkUwMlBCZCUhg@nuc> <3ea6d241-b9cc-4294-aef8-ae1c6d9d8161@dartmouth.edu> <ZcanttlCzNFvMM7S@nuc> <53139ffd-3e42-4aaf-a523-b8f4dc8b29a9@dartmouth.edu> <04d1f2e1-021e-42fb-9732-94fa98fd05fc@dartmouth.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 19, 2024 at 7:44=E2=80=AFAM Matthew L. Dailey <Matthew.L.Dailey@dartmouth.edu> wrote: > > Hi all, > > So I finally induced a panic on a "pure" ufs system - root and exported > filesystem were both ufs. So, I think this definitively rules out zfs as > a source of the issue. > > This panic was on 14.0p5 without debugging options, so the core may not > be helpful. The panic and backtrace are below in case they're > interesting to anyone. > > Next, I'm going to try a CURRENT kernel without debugging options > enabled just to see if I can finally induce a panic here. My other two > VMs running CURRENT with full debugging are still clanking along. You might want to set kern.kstack_pages=3D6 in /boot/loader.conf in these setups. I would normally expect double faults when a kernel stack is blown, but maybe there is a reason that you do now see that for a blown kernel stack. (The impact of increasing stack pages from 4->6 should be minimal.) rick > > -Matt > > [218716] Fatal trap 12: page fault while in kernel mode > [218716] cpuid =3D 4; apic id =3D 08 > [218716] fault virtual address =3D 0x100000017 > [218716] fault code =3D supervisor read data, page not presen= t > [218716] instruction pointer =3D 0x20:0xffffffff80e9165d > [218716] stack pointer =3D 0x28:0xfffffe010b5aa3b0 > [218716] frame pointer =3D 0x28:0xfffffe010b5aa400 > [218716] code segment =3D base 0x0, limit 0xfffff, type 0x1b > [218716] =3D DPL 0, pres 1, long 1, def32 0, gran = 1 > [218716] processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > [218716] current process =3D 49575 (nfsd: service) > [218716] rdi: 00000000ffffffff rsi: fffff800038ec900 rdx: fffffe00d932600= 0 > [218716] rcx: 0000000000030eb0 r8: 0000000000000000 r9: fffffe010b5aa41= 0 > [218716] rax: 00000000008f0eb0 rbx: fffff8038ac4cd00 rbp: fffffe010b5aa40= 0 > [218716] r10: 0000000000000000 r11: 0000000000000000 r12: 000000000000000= 0 > [218716] r13: fffff80003647c00 r14: fffff802f9dced00 r15: fffff800038ec90= 0 > [218716] trap number =3D 12 > [218716] panic: page fault > [218716] cpuid =3D 4 > [218716] time =3D 1708319487 > [218716] KDB: stack backtrace: > [218716] #0 0xffffffff80b9309d at kdb_backtrace+0x5d > [218716] #1 0xffffffff80b461a2 at vpanic+0x132 > [218716] #2 0xffffffff80b46063 at panic+0x43 > [218716] #3 0xffffffff8101d85c at trap_fatal+0x40c > [218716] #4 0xffffffff8101d8af at trap_pfault+0x4f > [218716] #5 0xffffffff80ff3fe8 at calltrap+0x8 > [218716] #6 0xffffffff80e8716e at newdirrem+0x8be > [218716] #7 0xffffffff80e866fa at softdep_setup_remove+0x1a > [218716] #8 0xffffffff80ea71af at ufs_dirremove+0x21f > [218716] #9 0xffffffff80ead4f4 at ufs_remove+0xb4 > [218716] #10 0xffffffff810f1428 at VOP_REMOVE_APV+0x28 > [218716] #11 0xffffffff80a60db4 at nfsvno_removesub+0xc4 > [218716] #12 0xffffffff80a52699 at nfsrvd_remove+0x1b9 > [218716] #13 0xffffffff80a374d4 at nfsrvd_dorpc+0x1854 > [218716] #14 0xffffffff80a4e76f at nfssvc_program+0x82f > [218716] #15 0xffffffff80e34080 at svc_run_internal+0xb50 > [218716] #16 0xffffffff80e3475b at svc_thread_start+0xb > [218716] #17 0xffffffff80b00b7f at fork_exit+0x7f > [218716] Uptime: 2d12h45m16s > [218716] Dumping 985 out of 16350 > MB:..2%..12%..22%..31%..41%..51%..61%..72%..82%..91% > > > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 > #1 doadump (textdump=3D<optimized out>) at > /usr/src/sys/kern/kern_shutdown.c:405 > #2 0xffffffff80b45d37 in kern_reboot (howto=3D260) > at /usr/src/sys/kern/kern_shutdown.c:526 > #3 0xffffffff80b4620f in vpanic (fmt=3D0xffffffff81147c9c "%s", > ap=3Dap@entry=3D0xfffffe010b5aa200) at > /usr/src/sys/kern/kern_shutdown.c:970 > #4 0xffffffff80b46063 in panic (fmt=3D<unavailable>) > at /usr/src/sys/kern/kern_shutdown.c:894 > #5 0xffffffff8101d85c in trap_fatal (frame=3D0xfffffe010b5aa2f0, > eva=3D4294967319) > at /usr/src/sys/amd64/amd64/trap.c:952 > #6 0xffffffff8101d8af in trap_pfault (frame=3D0xfffffe010b5aa2f0, > usermode=3Dfalse, signo=3D<optimized out>, ucode=3D<optimized out>) > at /usr/src/sys/amd64/amd64/trap.c:760 > #7 <signal handler called> > #8 cancel_diradd (dap=3D0xfffff8038ac4cd00, > dirrem=3Ddirrem@entry=3D0xfffff800038ec900, > jremref=3Djremref@entry=3D0xfffff802f9dced00, > dotremref=3Ddotremref@entry=3D0x0, > dotdotremref=3Ddotdotremref@entry=3D0x0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9028 > #9 0xffffffff80e8716e in newdirrem (bp=3D<optimized out>, > dp=3Ddp@entry=3D0xfffff800037fea80, ip=3Dip@entry=3D0xfffff8006b3b93= 00, > isrmdir=3Disrmdir@entry=3D0, > prevdirremp=3Dprevdirremp@entry=3D0xfffffe010b5aa4b0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9480 > #10 0xffffffff80e866fa in softdep_setup_remove (bp=3D0xffffffff, > dp=3D0xfffff800038ec900, dp@entry=3D0xfffff800037fea80, > ip=3D0xfffffe00d9326000, > ip@entry=3D0xfffff8006b3b9300, isrmdir=3D200368, isrmdir@entry=3D0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9176 > #11 0xffffffff80ea71af in ufs_dirremove (dvp=3Ddvp@entry=3D0xfffff801f764= be00, > ip=3Dip@entry=3D0xfffff8006b3b9300, flags=3D<optimized out>, > isrmdir=3Disrmdir@entry=3D0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:11= 98 > #12 0xffffffff80ead4f4 in ufs_remove (ap=3D0xfffffe010b5aa5d8) > at /usr/src/sys/ufs/ufs/ufs_vnops.c:1054 > #13 0xffffffff810f1428 in VOP_REMOVE_APV ( > vop=3D0xffffffff8172f2d0 <ffs_vnodeops2>, a=3Da@entry=3D0xfffffe010b= 5aa5d8) > at vnode_if.c:1534 > #14 0xffffffff80a60db4 in VOP_REMOVE (dvp=3D0x8f0eb0, vp=3D0xfffff800539b= 7380, > cnp=3D0x30eb0) at ./vnode_if.h:789 > #15 nfsvno_removesub (ndp=3D0xfffffe010b5aa858, is_v4=3D<optimized out>, > cred=3D<optimized out>, p=3Dp@entry=3D0xfffffe010ae803a0, > exp=3Dexp@entry=3D0xfffffe010b5aaa88) > at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:1466 > #16 0xffffffff80a52699 in nfsrvd_remove (nd=3D0xfffffe010b5aabf8, > isdgram=3D<optimized out>, dp=3D<optimized out>, exp=3D0xfffffe010b5= aaa88) > at /usr/src/sys/fs/nfsserver/nfs_nfsdserv.c:1583 > #17 0xffffffff80a374d4 in nfsrvd_compound (nd=3D0xfffffe010b5aabf8, isdgr= am=3D0, > tag=3D<optimized out>, taglen=3D0, minorvers=3D<optimized out>) > at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:1322 > #18 nfsrvd_dorpc (nd=3Dnd@entry=3D0xfffffe010b5aabf8, isdgram=3Disdgram@e= ntry=3D0, > tag=3D<optimized out>, taglen=3D0, minorvers=3D<optimized out>) > at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:633 > #19 0xffffffff80a4e76f in nfs_proc (nd=3D0xfffffe010b5aabf8, > xid=3D<optimized out>, xprt=3D0xfffff800037b4800, rpp=3D<optimized o= ut>) > at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:464 > #20 nfssvc_program (rqst=3D0xfffff80128ac9000, xprt=3D0xfffff800037b4800) > at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:348 > #21 0xffffffff80e34080 in svc_executereq (rqstp=3D0xfffff80128ac9000) > at /usr/src/sys/rpc/svc.c:1036 > #22 svc_run_internal (grp=3Dgrp@entry=3D0xfffff800033c2100, > ismaster=3Dismaster@entry=3D0) at /usr/src/sys/rpc/svc.c:1312 > #23 0xffffffff80e3475b in svc_thread_start (arg=3D0xffffffff, > arg@entry=3D0xfffff800033c2100) at /usr/src/sys/rpc/svc.c:1340 > #24 0xffffffff80b00b7f in fork_exit ( > callout=3D0xffffffff80e34750 <svc_thread_start>, arg=3D0xfffff800033= c2100, > frame=3D0xfffffe010b5aaf40) at /usr/src/sys/kern/kern_fork.c:1160 > #25 <signal handler called> > #26 0x000033af7af143ea in ?? ()
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy5CbGNOWcpp43uTY9=iha2aAYdLcCNWJLsG0=Fvg=dZGA>