Date: Tue, 25 Apr 2017 21:03:48 -0400 From: Tom Uffner <tom@uffner.com> To: freebsd-current <freebsd-current@freebsd.org> Subject: panics in network stack in 12-current Message-ID: <bccb9c51-3dc2-ea38-15ff-0dbef83b25c1@uffner.com>
next in thread | raw e-mail | index | archive | help
Since updating my -current box to 12 several months ago, I have been trying to
pin down several elusive and probably related panics.
they always manifest a a trap out of rw_wlock_hard()
i am fairly certain that r302409 was stable, revs up through r306792 may be
stable, or perhaps I just didn't wait long enough for my system to panic. I
don't know of anything that I can reproducably poke at to trigger this.
r306807 is definitely bad, as is everything up through r309124. I haven't seen
anything on the mailing lists or in the SVN logs that looks like it is related
to my problem.
my hardware is an Asus M4A77TD MB, AMD Phenom 2 X6 1100T CPU (for some of
this time I had an Athlon 2 X2, but upgraded recently), and RealTek 8168
PCIe Gigabit NIC.
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #33 r306807M:
Tue Apr 18 17:09:55 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
in revs between 306807-307125, the panics have been in flowcleaner, in more
recent ones, they happen in arbitrary userspace processes that make heavy use
of the network.
I know I should try the latest rev to see if it went away. aside from that,
any thoughts on how I should proceed?
Mon Apr 17 02:52:10 EDT 2017
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #32 r306821M:
Fri Apr 7 02:11:44 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
panic: page fault
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x3b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8057820d
stack pointer = 0x28:0xfffffe046a422650
frame pointer = 0x28:0xfffffe046a422690
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 697 (ntpd)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe046a4222b0
vpanic() at vpanic+0x186/frame 0xfffffe046a422330
panic() at panic+0x43/frame 0xfffffe046a422390
trap_fatal() at trap_fatal+0x331/frame 0xfffffe046a4223f0
trap_pfault() at trap_pfault+0x14f/frame 0xfffffe046a422430
trap() at trap+0x21e/frame 0xfffffe046a422580
calltrap() at calltrap+0x8/frame 0xfffffe046a422580
--- trap 0xc, rip = 0xffffffff8057820d, rsp = 0xfffffe046a422650, rbp =
0xfffffe046a422690 ---
__rw_wlock_hard() at __rw_wlock_hard+0xad/frame 0xfffffe046a422690
ip_output() at ip_output+0x483/frame 0xfffffe046a4227c0
udp_send() at udp_send+0xb8f/frame 0xfffffe046a422890
sosend_dgram() at sosend_dgram+0x431/frame 0xfffffe046a422910
kern_sendit() at kern_sendit+0x178/frame 0xfffffe046a4229c0
sendit() at sendit+0x179/frame 0xfffffe046a422a10
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe046a422a60
amd64_syscall() at amd64_syscall+0x391/frame 0xfffffe046a422bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe046a422bf0
--- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8013c9cba, rsp =
0x7fffdfffc7e8, rbp = 0x7fffdfffc830 ---
Mon Apr 17 03:19:00 EDT 2017
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #32 r306821M:
Fri Apr 7 02:11:44 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
panic: page fault
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x3b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8057820d
stack pointer = 0x28:0xfffffe0469a0eab0
frame pointer = 0x28:0xfffffe0469a0eaf0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 21 (flowcleaner)
trap number = 12
Timeout initializing vt_vga
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0469a0e710
vpanic() at vpanic+0x186/frame 0xfffffe0469a0e790
panic() at panic+0x43/frame 0xfffffe0469a0e7f0
trap_fatal() at trap_fatal+0x331/frame 0xfffffe0469a0e850
trap_pfault() at trap_pfault+0x14f/frame 0xfffffe0469a0e890
trap() at trap+0x21e/frame 0xfffffe0469a0e9e0
calltrap() at calltrap+0x8/frame 0xfffffe0469a0e9e0
--- trap 0xc, rip = 0xffffffff8057820d, rsp = 0xfffffe0469a0eab0, rbp =
0xfffffe0469a0eaf0 ---
__rw_wlock_hard() at __rw_wlock_hard+0xad/frame 0xfffffe0469a0eaf0
flowtable_clean_vnet() at flowtable_clean_vnet+0x496/frame 0xfffffe0469a0eb80
flowtable_cleaner() at flowtable_cleaner+0x90/frame 0xfffffe0469a0ebb0
fork_exit() at fork_exit+0x75/frame 0xfffffe0469a0ebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0469a0ebf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Mon Apr 17 02:25:20 EDT 2017
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #32 r306821M:
Fri Apr 7 02:11:44 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
panic: page fault
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x3b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8057820d
stack pointer = 0x28:0xfffffe046a4d1650
frame pointer = 0x28:0xfffffe046a4d1690
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 974 (python2.7)
trap number = 12
Timeout initializing vt_vga
panic: page fault
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe046a4d12b0
vpanic() at vpanic+0x186/frame 0xfffffe046a4d1330
panic() at panic+0x43/frame 0xfffffe046a4d1390
trap_fatal() at trap_fatal+0x331/frame 0xfffffe046a4d13f0
trap_pfault() at trap_pfault+0x14f/frame 0xfffffe046a4d1430
trap() at trap+0x21e/frame 0xfffffe046a4d1580
calltrap() at calltrap+0x8/frame 0xfffffe046a4d1580
--- trap 0xc, rip = 0xffffffff8057820d, rsp = 0xfffffe046a4d1650, rbp =
0xfffffe046a4d1690 ---
__rw_wlock_hard() at __rw_wlock_hard+0xad/frame 0xfffffe046a4d1690
ip_output() at ip_output+0x483/frame 0xfffffe046a4d17c0
udp_send() at udp_send+0xb8f/frame 0xfffffe046a4d1890
sosend_dgram() at sosend_dgram+0x431/frame 0xfffffe046a4d1910
kern_sendit() at kern_sendit+0x178/frame 0xfffffe046a4d19c0
sendit() at sendit+0x179/frame 0xfffffe046a4d1a10
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe046a4d1a60
amd64_syscall() at amd64_syscall+0x391/frame 0xfffffe046a4d1bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe046a4d1bf0
--- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x801397cba, rsp =
0x7fffdf5f7618, rbp = 0x7fffdf5f7660 ---
Mon Apr 17 02:50:19 EDT 2017
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #32 r306821M:
Fri Apr 7 02:11:44 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
panic: page fault
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x3b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8057820d
stack pointer = 0x28:0xfffffe03e63fb560
frame pointer = 0x28:0xfffffe03e63fb5a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 11 (swi4: clock (0))
trap number = 12
Timeout initializing vt_vga
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe03e63fb1c0
vpanic() at vpanic+0x186/frame 0xfffffe03e63fb240
panic() at panic+0x43/frame 0xfffffe03e63fb2a0
trap_fatal() at trap_fatal+0x331/frame 0xfffffe03e63fb300
trap_pfault() at trap_pfault+0x14f/frame 0xfffffe03e63fb340
trap() at trap+0x21e/frame 0xfffffe03e63fb490
calltrap() at calltrap+0x8/frame 0xfffffe03e63fb490
--- trap 0xc, rip = 0xffffffff8057820d, rsp = 0xfffffe03e63fb560, rbp =
0xfffffe03e63fb5a0 ---
__rw_wlock_hard() at __rw_wlock_hard+0xad/frame 0xfffffe03e63fb5a0
in_lltable_lookup() at in_lltable_lookup+0xa8/frame 0xfffffe03e63fb5c0
llentry_alloc() at llentry_alloc+0x3c/frame 0xfffffe03e63fb610
flowtable_lookup_common() at flowtable_lookup_common+0x2d1/frame
0xfffffe03e63fb6f0
flowtable_lookup() at flowtable_lookup+0x9c/frame 0xfffffe03e63fb740
ip_output() at ip_output+0xaa/frame 0xfffffe03e63fb870
tcp_output() at tcp_output+0x175e/frame 0xfffffe03e63fba00
tcp_timer_rexmt() at tcp_timer_rexmt+0x442/frame 0xfffffe03e63fba30
softclock_call_cc() at softclock_call_cc+0x14a/frame 0xfffffe03e63fbaf0
softclock() at softclock+0x6d/frame 0xfffffe03e63fbb20
intr_event_execute_handlers() at intr_event_execute_handlers+0xbc/frame
0xfffffe03e63fbb60
ithread_loop() at ithread_loop+0xa6/frame 0xfffffe03e63fbbb0
fork_exit() at fork_exit+0x75/frame 0xfffffe03e63fbbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe03e63fbbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Sat Apr 22 18:06:14 EDT 2017
FreeBSD discordia.uffner.com 12.0-CURRENT FreeBSD 12.0-CURRENT #33 r306807M:
Tue Apr 18 17:09:55 EDT 2017
tom@discordia.uffner.com:/usr/obj/usr/src/sys/DISCORDIA amd64
panic: page fault
GNU gdb (GDB) 7.11.1 [GDB v7.11.1 for FreeBSD]
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from
/usr/lib/debug//boot/kernel/kernel.debug...done.
done.
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x3b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8057820d
stack pointer = 0x28:0xfffffe046a7804b0
frame pointer = 0x28:0xfffffe046a7804f0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 4643 (DNS Resolver #749)
trap number = 12
Timeout initializing vt_vga
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe046a780110
vpanic() at vpanic+0x186/frame 0xfffffe046a780190
panic() at panic+0x43/frame 0xfffffe046a7801f0
trap_fatal() at trap_fatal+0x331/frame 0xfffffe046a780250
trap_pfault() at trap_pfault+0x14f/frame 0xfffffe046a780290
trap() at trap+0x21e/frame 0xfffffe046a7803e0
calltrap() at calltrap+0x8/frame 0xfffffe046a7803e0
--- trap 0xc, rip = 0xffffffff8057820d, rsp = 0xfffffe046a7804b0, rbp =
0xfffffe046a7804f0 ---
__rw_wlock_hard() at __rw_wlock_hard+0xad/frame 0xfffffe046a7804f0
in_lltable_lookup() at in_lltable_lookup+0xa8/frame 0xfffffe046a780510
llentry_alloc() at llentry_alloc+0x3c/frame 0xfffffe046a780560
flowtable_lookup_common() at flowtable_lookup_common+0x2d1/frame
0xfffffe046a780640
flowtable_lookup() at flowtable_lookup+0x9c/frame 0xfffffe046a780690
ip_output() at ip_output+0xaa/frame 0xfffffe046a7807c0
udp_send() at udp_send+0xb8f/frame 0xfffffe046a780890
sosend_dgram() at sosend_dgram+0x431/frame 0xfffffe046a780910
kern_sendit() at kern_sendit+0x178/frame 0xfffffe046a7809c0
sendit() at sendit+0x179/frame 0xfffffe046a780a10
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe046a780a60
amd64_syscall() at amd64_syscall+0x391/frame 0xfffffe046a780bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe046a780bf0
--- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8020d5cba, rsp =
0x7fffd9364728, rbp = 0x7fffd9364770 ---
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bccb9c51-3dc2-ea38-15ff-0dbef83b25c1>