Date: Thu, 22 Sep 2016 20:48:29 +0800 From: Ben Woods <woodsb02@gmail.com> To: FreeBSD Current <freebsd-current@freebsd.org> Subject: Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837) Message-ID: <CAOc73CDZnPy1JJ8DBJyANsYqPkV35__ojfE%2BFbfbT3hTZw7MgA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi everyone,
I am currently experiencing semi-regular kernel crashes on my FreeBSD
12-current machine. I am new to kernel debugging, and hoping someone can
have a look at the debugging output below to point me in the direction of
what the problem might be.
My machine is a FreeNAS-mini from iXsystems which I have formatted and
installed stock FreeBSD onto. My kernel is the default generic-nodebug with
the VIMAGE options added.
$ uname -a
FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat
Sep 3 12:29:01 AWST 2016
woodsb02@freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE
amd64
$ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE
# SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file
# for FreeBSD/amd64
include GENERIC-NODEBUG
ident GENERIC-NODEBUG-VIMAGE
#nooptions SCTP # Stream Control Transmission Protocol
options VIMAGE # VNET/Vimage support
Output from kernel crash dump debug with kgdb below:
/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug
/var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80b4d91c
stack pointer = 0x28:0xfffffe046813a440
frame pointer = 0x28:0xfffffe046813a470
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 33487 (sh)
Uptime: 15m16s
Dumping 1664 out of 16338
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...
done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
#0 doadump (textdump=1) at pcpu.h:221
221 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) list *0xffffffff80b4d91c
0xffffffff80b4d91c is in turnstile_broadcast
(/usr/src/sys/kern/subr_turnstile.c:837).
832
833 /*
834 * Transfer the blocked list to the pending list.
835 */
836 mtx_lock_spin(&td_contested_lock);
837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
838 mtx_unlock_spin(&td_contested_lock);
839
840 /*
841 * Give a turnstile to each thread. The last thread gets
Current language: auto; currently minimal
(kgdb) backtrace
#0 doadump (textdump=1) at pcpu.h:221
#1 0xffffffff80aea40e in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80aea9db in vpanic (fmt=<value optimized out>, ap=<value
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80aea813 in panic (fmt=0x0) at
/usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff8039e197 in db_panic (addr=<value optimized out>,
have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486
#5 0xffffffff8039d689 in db_command (cmd_table=<value optimized out>) at
/usr/src/sys/ddb/db_command.c:453
#6 0xffffffff8039d3e4 in db_command_loop () at
/usr/src/sys/ddb/db_command.c:506
#7 0xffffffff803a053b in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80b36b33 in kdb_trap (type=<value optimized out>, code=<value
optimized out>, tf=<value optimized out>) at
/usr/src/sys/kern/subr_kdb.c:654
#9 0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at
/usr/src/sys/amd64/amd64/trap.c:836
#10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390,
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960,
tid=<value optimized out>, file=<value optimized out>, line=<value
optimized out>)
at /usr/src/sys/kern/kern_rwlock.c:1027
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>,
args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>,
traced=0) at subr_syscall.c:135
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
#23 0x0000000800b468ea in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 11
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
442 (void) trap_pfault(frame, FALSE);
(kgdb) list
437
438 KASSERT(cold || td->td_ucred != NULL,
439 ("kernel trap doesn't have ucred"));
440 switch (type) {
441 case T_PAGEFLT: /* page fault */
442 (void) trap_pfault(frame, FALSE);
443 goto out;
444
445 case T_DNA:
446 if (PCB_USER_FPU(td->td_pcb))
(kgdb) print td
$1 = (struct thread *) 0xfffff8032893aa00
(kgdb) print td->td_ucred
$2 = (struct ucred *) 0xfffff8004005ec00
(kgdb) print type
$3 = 12
(kgdb) up
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
236 call trap_check
Current language: auto; currently asm
(kgdb) up
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
Current language: auto; currently minimal
(kgdb) up
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960,
tid=<value optimized out>, file=<value optimized out>, line=<value
optimized out>)
at /usr/src/sys/kern/kern_rwlock.c:1027
1027 turnstile_broadcast(ts, queue);
(kgdb) up
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
2960 VM_OBJECT_WUNLOCK(object);
(kgdb) up
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
3077 result = vm_map_delete(map, start, end);
(kgdb) up
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
1096 vm_map_remove(map, vm_map_min(map),
vm_map_max(map));
(kgdb) up
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
896 error = exec_new_vmspace(imgp, sv);
(kgdb) up
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>,
args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
603 error = (*execsw[i]->ex_imgact)(imgp);
(kgdb) up
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
219 error = kern_execve(td, &args, NULL);
(kgdb) up
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>,
traced=0) at subr_syscall.c:135
135 error = (sa->callp->sy_call)(td, sa->args);
(kgdb) up
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
396 call amd64_syscall
Current language: auto; currently asm
(kgdb) up
#23 0x0000000800b468ea in ?? ()
(kgdb) up
Initial frame selected; you cannot go up.
(kgdb) quit
Regards,
Ben
--
From: Benjamin Woods
woodsb02@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOc73CDZnPy1JJ8DBJyANsYqPkV35__ojfE%2BFbfbT3hTZw7MgA>