Date: Wed, 5 May 2021 11:08:49 +0200 From: Michael Schmiedgen <schmiedgen@gmx.net> To: Mark Johnston <markj@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: page fault while in kernel mode - after upgrade from 12.2 to 13.0 Message-ID: <8016b487-4256-73d1-0e61-7e245956f718@gmx.net> In-Reply-To: <YJGaUnWCPVXRC4NC@nuc> References: <d7c3bfbd-2e54-c0f4-ec23-5dab08287ea3@gmx.net> <YJBS8YMZFkMtWPEu@nuc> <d37716a3-927d-b200-c805-b31d7b36383d@gmx.net> <YJGaUnWCPVXRC4NC@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04.05.2021 21:02, Mark Johnston wrote:
> On Tue, May 04, 2021 at 08:38:39PM +0200, Michael Schmiedgen wrote:
>> Hi Mark,
>>
>> sorry for the delay, I only can test after work. I triggered another 2 =
panics, this time
>> with a different result (see below). Can I provide some more informatio=
n?
>
> This looks like fairly random kernel memory corruption. Are you able to
> build an INVARIANTS kernel and test that? Assuming you're using 13.0,
> you'd grab the 13.0 sources, add "options INVARIANT_SUPPORT" and
> "options INVARIANTS" to the GENERIC kernel configuration in
> sys/amd64/conf, and do a "make buildkernel installkernel".
I will try INVARIANTS after work, but in the meantime I got 2 more panics
from tonight.
=2D-- #1
Fatal trap 12: page fault while in kernel mode
cpuid =3D 7; apic id =3D 07
fault virtual address =3D 0x8
fault code =3D supervisor read data, page not present
instruction pointer =3D 0x20:0xffffffff80ca599c
stack pointer =3D 0x28:0xfffffe0115bc46c0
frame pointer =3D 0x28:0xfffffe0115bc4700
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 12 (swi1: netisr 0)
trap number =3D 12
panic: page fault
cpuid =3D 7
time =3D 1620172732
KDB: stack backtrace:
#0 0xffffffff80c57345 at kdb_backtrace+0x65
#1 0xffffffff80c09d21 at vpanic+0x181
#2 0xffffffff80c09b93 at panic+0x43
#3 0xffffffff8108b187 at trap_fatal+0x387
#4 0xffffffff8108b1df at trap_pfault+0x4f
#5 0xffffffff8108a83d at trap+0x27d
#6 0xffffffff810617a8 at calltrap+0x8
#7 0xffffffff80dbf0ae at tcp_do_segment+0x10ce
#8 0xffffffff80dbd21e at tcp_input+0xabe
#9 0xffffffff80dafc15 at ip_input+0x125
#10 0xffffffff80d3fa7b at swi_net+0x12b
#11 0xffffffff80bcae5d at ithread_loop+0x24d
#12 0xffffffff80bc7c5e at fork_exit+0x7e
#13 0xffffffff8106282e at fork_trampoline+0xe
Uptime: 5h36m39s
Dumping 7281 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.=
.91%
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str=
uct pcpu,
(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1 doadump (textdump=3D<optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xffffffff80c09916 in kern_reboot (howto=3D260)
at /usr/src/sys/kern/kern_shutdown.c:486
#3 0xffffffff80c09d90 in vpanic (fmt=3D<optimized out>, ap=3D<optimized o=
ut>)
at /usr/src/sys/kern/kern_shutdown.c:919
#4 0xffffffff80c09b93 in panic (fmt=3D<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:843
#5 0xffffffff8108b187 in trap_fatal (frame=3D0xfffffe0115bc4600, eva=3D8)
at /usr/src/sys/amd64/amd64/trap.c:915
#6 0xffffffff8108b1df in trap_pfault (frame=3Dframe@entry=3D0xfffffe0115b=
c4600,
usermode=3Dfalse, signo=3D<optimized out>, signo@entry=3D0x0,
ucode=3D<optimized out>, ucode@entry=3D0x0)
at /usr/src/sys/amd64/amd64/trap.c:732
#7 0xffffffff8108a83d in trap (frame=3D0xfffffe0115bc4600)
at /usr/src/sys/amd64/amd64/trap.c:398
#8 <signal handler called>
#9 sbcut_internal (sb=3D0xfffff8043bc00610, len=3D57, len@entry=3D304)
at /usr/src/sys/kern/uipc_sockbuf.c:1491
#10 0xffffffff80ca5b8a in sbcut_locked (sb=3D0xfffff8043bc00610,
len=3D-1796951296, len@entry=3D304) at /usr/src/sys/kern/uipc_sockbuf=
.c:1591
#11 0xffffffff80dbf0ae in tcp_do_segment (m=3D0xfffff8024b9a6900,
th=3D<optimized out>, so=3D<optimized out>, tp=3D<optimized out>,
drop_hdrlen=3D52, tlen=3D<optimized out>, iptos=3D0 '\000')
at /usr/src/sys/netinet/tcp_input.c:2918
#12 0xffffffff80dbd21e in tcp_input (mp=3D<optimized out>,
offp=3D<optimized out>, proto=3D<optimized out>)
at /usr/src/sys/netinet/tcp_input.c:1382
#13 0xffffffff80dafc15 in ip_input (m=3D0x0)
at /usr/src/sys/netinet/ip_input.c:829
#14 0xffffffff80d3fa7b in netisr_process_workstream_proto (
nwsp=3D<optimized out>, proto=3D1) at /usr/src/sys/net/netisr.c:919
#15 swi_net (arg=3D<optimized out>) at /usr/src/sys/net/netisr.c:966
#16 0xffffffff80bcae5d in intr_event_execute_handlers (p=3D<optimized out>=
,
ie=3D0xfffff80003b88c00) at /usr/src/sys/kern/kern_intr.c:1168
#17 ithread_execute_handlers (p=3D<optimized out>, ie=3D0xfffff80003b88c00=
)
at /usr/src/sys/kern/kern_intr.c:1181
#18 ithread_loop (arg=3Darg@entry=3D0xfffff80003b95d20)
at /usr/src/sys/kern/kern_intr.c:1269
#19 0xffffffff80bc7c5e in fork_exit (
callout=3D0xffffffff80bcac10 <ithread_loop>, arg=3D0xfffff80003b95d20=
,
frame=3D0xfffffe0115bc4b00) at /usr/src/sys/kern/kern_fork.c:1069
=2D-- #2
Unread portion of the kernel message buffer:
panic: sbappendaddr_locked
cpuid =3D 2
time =3D 1620181490
KDB: stack backtrace:
#0 0xffffffff80c57345 at kdb_backtrace+0x65
#1 0xffffffff80c09d21 at vpanic+0x181
#2 0xffffffff80c09b93 at panic+0x43
#3 0xffffffff80ca51e0 at sbappendaddr_locked_internal+0
#4 0xffffffff82c4efd0 at divert_packet+0x1a0
#5 0xffffffff82c2bc81 at ipfw_check_packet+0x2c1
#6 0xffffffff80d41f87 at pfil_run_hooks+0x97
#7 0xffffffff80dafeb5 at ip_input+0x3c5
#8 0xffffffff80d3f2da at netisr_dispatch_src+0xca
#9 0xffffffff80d23a68 at ether_demux+0x148
#10 0xffffffff80d24dec at ether_nh_input+0x34c
#11 0xffffffff80d3f2da at netisr_dispatch_src+0xca
#12 0xffffffff80d23eb9 at ether_input+0x69
#13 0xffffffff80d2074a at if_input+0xa
#14 0xffffffff8060a98e at bge_rxeof+0x49e
#15 0xffffffff80607f27 at bge_intr_task+0x1a7
#16 0xffffffff80c6afe1 at taskqueue_run_locked+0x181
#17 0xffffffff80c6c2fc at taskqueue_thread_loop+0xac
Uptime: 2h21m11s
Dumping 8148 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.=
.91%
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str=
uct pcpu,
(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1 doadump (textdump=3D<optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xffffffff80c09916 in kern_reboot (howto=3D260)
at /usr/src/sys/kern/kern_shutdown.c:486
#3 0xffffffff80c09d90 in vpanic (fmt=3D<optimized out>, ap=3D<optimized o=
ut>)
at /usr/src/sys/kern/kern_shutdown.c:919
#4 0xffffffff80c09b93 in panic (fmt=3D<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:843
#5 0xffffffff80ca51e0 in sbappendaddr_locked (sb=3D0xfffff8002829a8a8,
asa=3D0xfffffe0115ebc5a0, m0=3D0xfffff804a977b700, control=3D0x0)
at /usr/src/sys/kern/uipc_sockbuf.c:1198
#6 0xffffffff82c4efd0 in divert_packet (m=3D0xfffff804a977b700,
incoming=3D<optimized out>) at /usr/src/sys/netinet/ip_divert.c:285
#7 0xffffffff82c2bc81 in ipfw_divert (m0=3D0xfffffe0115ebc760,
args=3D0xfffffe0115ebc610, tee=3D<optimized out>)
at /usr/src/sys/netpfil/ipfw/ip_fw_pfil.c:525
#8 ipfw_check_packet (m0=3D0xfffffe0115ebc760, ifp=3D0xfffff8000506f000,
flags=3D65536, ruleset=3D<optimized out>, inp=3D0x0)
at /usr/src/sys/netpfil/ipfw/ip_fw_pfil.c:283
#9 0xffffffff80d41f87 in pfil_run_hooks (head=3D<optimized out>, p=3D...,
ifp=3Difp@entry=3D0xfffff8000506f000, flags=3Dflags@entry=3D65536,
inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c:187
#10 0xffffffff80dafeb5 in ip_input (m=3D0x0)
at /usr/src/sys/netinet/ip_input.c:610
#11 0xffffffff80d3f2da in netisr_dispatch_src (proto=3D1,
source=3D<optimized out>, source@entry=3D0, m=3D<unavailable>)
at /usr/src/sys/net/netisr.c:1143
#12 0xffffffff80d3f5cf in netisr_dispatch (proto=3D<unavailable>,
m=3D<unavailable>) at /usr/src/sys/net/netisr.c:1234
#13 0xffffffff80d23a68 in ether_demux (ifp=3Difp@entry=3D0xfffff8000506f00=
0,
m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:923
#14 0xffffffff80d24dec in ether_input_internal (ifp=3D0xfffff8000506f000,
m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:709
#15 ether_nh_input (m=3D<optimized out>) at /usr/src/sys/net/if_ethersubr.=
c:739
#16 0xffffffff80d3f2da in netisr_dispatch_src (proto=3Dproto@entry=3D5,
source=3D<optimized out>, source@entry=3D0, m=3D<unavailable>,
m@entry=3D0xfffff804a977b700) at /usr/src/sys/net/netisr.c:1143
#17 0xffffffff80d3f5cf in netisr_dispatch (proto=3D<unavailable>,
proto@entry=3D5, m=3D<unavailable>, m@entry=3D0xfffff804a977b700)
at /usr/src/sys/net/netisr.c:1234
#18 0xffffffff80d23eb9 in ether_input (ifp=3D<optimized out>,
ifp@entry=3D<error reading variable: value is not available>,
m=3D0xfffff804a977b700,
m@entry=3D<error reading variable: value is not available>)
at /usr/src/sys/net/if_ethersubr.c:830
#19 0xffffffff80d2074a in if_input (ifp=3D<unavailable>,
ifp@entry=3D0xfffff8000506f000, sendmp=3D<unavailable>,
sendmp@entry=3D0xfffff804a977b700) at /usr/src/sys/net/if.c:4391
#20 0xffffffff8060a98e in bge_rxeof (sc=3Dsc@entry=3D0xfffffe0115cd4000,
rx_prod=3Drx_prod@entry=3D448, holdlck=3Dholdlck@entry=3D0)
at /usr/src/sys/dev/bge/if_bge.c:4412
#21 0xffffffff80607f27 in bge_intr_task (arg=3D0xfffffe0115cd4000,
pending=3D<optimized out>) at /usr/src/sys/dev/bge/if_bge.c:4642
#22 0xffffffff80c6afe1 in taskqueue_run_locked (
queue=3Dqueue@entry=3D0xfffff80005051d00)
at /usr/src/sys/kern/subr_taskqueue.c:476
#23 0xffffffff80c6c2fc in taskqueue_thread_loop (arg=3D<optimized out>,
arg@entry=3D0xfffffe0115cdb568) at /usr/src/sys/kern/subr_taskqueue.c=
:793
#24 0xffffffff80bc7c5e in fork_exit (
callout=3D0xffffffff80c6c250 <taskqueue_thread_loop>,
arg=3D0xfffffe0115cdb568, frame=3D0xfffffe0115ebcb00)
at /usr/src/sys/kern/kern_fork.c:1069
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8016b487-4256-73d1-0e61-7e245956f718>