From owner-freebsd-current@freebsd.org Thu Sep 22 12:48:31 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B133BE5C32 for ; Thu, 22 Sep 2016 12:48:31 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 34D9D22D for ; Thu, 22 Sep 2016 12:48:31 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: by mail-io0-x22f.google.com with SMTP id m186so84026168ioa.2 for ; Thu, 22 Sep 2016 05:48:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=3Vyk/JO/tJhZ+hPBU8vImM0WP9K8RHbzCFJRtxN8tOY=; b=kPkqUZVyxOmKc6RHbKXmk5/sKQLIZ5N+NWSWzrU2se/92/9R3SPa64nDx7Fuzy0sB1 wzUUjhcsZEcqBNHi3s2k+g+6KjLE3OWDAYuUOCjlz8qM8OwjaJ0Dvo3rqrE9369VlVEt 1JrAto82EFacX9UsuAxs3E8GGguQH6TGlx05YgvKBqcd9RL1kpjnTZMK7z+c8Cd2p2YV bR5ju2NFGmjnZJlalz/g64/RbpnzHijSj90HXZVP9u0hRMHXySzDQuLgH9tbgVORk3Wm txhcK6Q7hODjvyk93nJCo1fNMuPfVLcB12/cAdwrbFgzhJ/asJ9ULjbyZZ7VnpF8N62o +mIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=3Vyk/JO/tJhZ+hPBU8vImM0WP9K8RHbzCFJRtxN8tOY=; b=Klrc1tz76gXYM01V3ucSe5HNzYuAX40EPyuEOX8TpKqgWSLbuOEqdxZts5oFVCwjPC EZGxX7r9Kz8aTFNpjOqqX4W7rgFuMSsfsMOcEA8Xiz6k7ghQ6/46JUYiBOLw52Ks8jI+ nLXjBdqMRKZ3ctpardLBEqQlnTO8wba6Q6j+rxAhnadzpsQxdPd5+m/jGn80PCiQKCmW AQ9Lt7/ZJIj1H3whwU3IWT6+sXKFcqxshVuibUHI23vOX31hBuwUd3Ri4fNEBOEH8y3B JerNnwNonpCGmxBv3e+NqBCyakkt1oCTgMWRy7Kzdofy44cqi3D8AdeMYPbNvs/Vgc8p bhZQ== X-Gm-Message-State: AE9vXwMV6BID+4zoCZqiYts471y55Gpv7v67B4M2beckaai04ghq/lulfhb9WB3J9bUiwvZ3WoXBPv1pje+QTQ== X-Received: by 10.107.9.29 with SMTP id j29mr2678956ioi.153.1474548510084; Thu, 22 Sep 2016 05:48:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.26.3 with HTTP; Thu, 22 Sep 2016 05:48:29 -0700 (PDT) From: Ben Woods Date: Thu, 22 Sep 2016 20:48:29 +0800 Message-ID: Subject: Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837) To: FreeBSD Current Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2016 12:48:31 -0000 Hi everyone, I am currently experiencing semi-regular kernel crashes on my FreeBSD 12-current machine. I am new to kernel debugging, and hoping someone can have a look at the debugging output below to point me in the direction of what the problem might be. My machine is a FreeNAS-mini from iXsystems which I have formatted and installed stock FreeBSD onto. My kernel is the default generic-nodebug with the VIMAGE options added. $ uname -a FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat Sep 3 12:29:01 AWST 2016 woodsb02@freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE amd64 $ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE # SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file # for FreeBSD/amd64 include GENERIC-NODEBUG ident GENERIC-NODEBUG-VIMAGE #nooptions SCTP # Stream Control Transmission Protocol options VIMAGE # VNET/Vimage support Output from kernel crash dump debug with kgdb below: /usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x30 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80b4d91c stack pointer = 0x28:0xfffffe046813a440 frame pointer = 0x28:0xfffffe046813a470 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 33487 (sh) Uptime: 15m16s Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug... done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko #0 doadump (textdump=1) at pcpu.h:221 221 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) list *0xffffffff80b4d91c 0xffffffff80b4d91c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837). 832 833 /* 834 * Transfer the blocked list to the pending list. 835 */ 836 mtx_lock_spin(&td_contested_lock); 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 838 mtx_unlock_spin(&td_contested_lock); 839 840 /* 841 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) backtrace #0 doadump (textdump=1) at pcpu.h:221 #1 0xffffffff80aea40e in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff80aea9db in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80aea813 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690 #4 0xffffffff8039e197 in db_panic (addr=, have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486 #5 0xffffffff8039d689 in db_command (cmd_table=) at /usr/src/sys/ddb/db_command.c:453 #6 0xffffffff8039d3e4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506 #7 0xffffffff803a053b in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:251 #8 0xffffffff80b36b33 in kdb_trap (type=, code=, tf=) at /usr/src/sys/kern/subr_kdb.c:654 #9 0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at /usr/src/sys/amd64/amd64/trap.c:836 #10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691 #11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=, file=, line=) at /usr/src/sys/kern/kern_rwlock.c:1027 #15 0xffffffff80e525dc in vm_map_delete (map=, start=, end=) at /usr/src/sys/vm/vm_map.c:2960 #16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077 #17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=) at /usr/src/sys/kern/imgact_elf.c:896 #19 0xffffffff80a9670d in kern_execve (td=, args=, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603 #20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 #21 0xffffffff80fddde8 in amd64_syscall (td=, traced=0) at subr_syscall.c:135 #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #23 0x0000000800b468ea in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up 11 #11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 442 (void) trap_pfault(frame, FALSE); (kgdb) list 437 438 KASSERT(cold || td->td_ucred != NULL, 439 ("kernel trap doesn't have ucred")); 440 switch (type) { 441 case T_PAGEFLT: /* page fault */ 442 (void) trap_pfault(frame, FALSE); 443 goto out; 444 445 case T_DNA: 446 if (PCB_USER_FPU(td->td_pcb)) (kgdb) print td $1 = (struct thread *) 0xfffff8032893aa00 (kgdb) print td->td_ucred $2 = (struct ucred *) 0xfffff8004005ec00 (kgdb) print type $3 = 12 (kgdb) up #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 236 call trap_check Current language: auto; currently asm (kgdb) up #13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); Current language: auto; currently minimal (kgdb) up #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=, file=, line=) at /usr/src/sys/kern/kern_rwlock.c:1027 1027 turnstile_broadcast(ts, queue); (kgdb) up #15 0xffffffff80e525dc in vm_map_delete (map=, start=, end=) at /usr/src/sys/vm/vm_map.c:2960 2960 VM_OBJECT_WUNLOCK(object); (kgdb) up #16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077 3077 result = vm_map_delete(map, start, end); (kgdb) up #17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 1096 vm_map_remove(map, vm_map_min(map), vm_map_max(map)); (kgdb) up #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=) at /usr/src/sys/kern/imgact_elf.c:896 896 error = exec_new_vmspace(imgp, sv); (kgdb) up #19 0xffffffff80a9670d in kern_execve (td=, args=, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603 603 error = (*execsw[i]->ex_imgact)(imgp); (kgdb) up #20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 219 error = kern_execve(td, &args, NULL); (kgdb) up #21 0xffffffff80fddde8 in amd64_syscall (td=, traced=0) at subr_syscall.c:135 135 error = (sa->callp->sy_call)(td, sa->args); (kgdb) up #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 396 call amd64_syscall Current language: auto; currently asm (kgdb) up #23 0x0000000800b468ea in ?? () (kgdb) up Initial frame selected; you cannot go up. (kgdb) quit Regards, Ben -- From: Benjamin Woods woodsb02@gmail.com