From owner-freebsd-bugs@FreeBSD.ORG Tue Mar 29 09:00:23 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C89F106566C for ; Tue, 29 Mar 2011 09:00:23 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9D7F18FC14 for ; Tue, 29 Mar 2011 09:00:22 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p2T90MMB051427 for ; Tue, 29 Mar 2011 09:00:22 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p2T90MDA051426; Tue, 29 Mar 2011 09:00:22 GMT (envelope-from gnats) Resent-Date: Tue, 29 Mar 2011 09:00:22 GMT Resent-Message-Id: <201103290900.p2T90MDA051426@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Sergey Kandaurov Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1313106564A for ; Tue, 29 Mar 2011 08:54:28 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id CFC498FC15 for ; Tue, 29 Mar 2011 08:54:28 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p2T8sSim077242 for ; Tue, 29 Mar 2011 08:54:28 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p2T8sSBZ077241; Tue, 29 Mar 2011 08:54:28 GMT (envelope-from nobody) Message-Id: <201103290854.p2T8sSBZ077241@red.freebsd.org> Date: Tue, 29 Mar 2011 08:54:28 GMT From: Sergey Kandaurov To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/156026: arpintr()->in_lltable_lookup() 8.1 bce(4) crash X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2011 09:00:23 -0000 >Number: 156026 >Category: kern >Synopsis: arpintr()->in_lltable_lookup() 8.1 bce(4) crash >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Mar 29 09:00:22 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Sergey Kandaurov >Release: 8.1-RELEASE-p2 >Organization: >Environment: FreeBSD 8.1-RELEASE-p2 amd64 >Description: It's GENERIC w/ : options KDB options DDB options BREAK_TO_DEBUGGER options IPFIREWALL options IPFIREWALL_FORWARD options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=500 options IPFIREWALL_DEFAULT_TO_ACCEPT options LIBALIAS options IPFIREWALL_NAT options QUOTA device carp There are 3 crashes seen between February and March on different boxes but identical h/w. Backtrace of crashed proc is the same. db> bt Tracing pid 12 tid 100039 td 0xffffff00029a23e0 propagate_priority() at propagate_priority+0x72 turnstile_wait() at turnstile_wait+0x1aa _rw_wlock_hard() at _rw_wlock_hard+0xfa in_lltable_lookup() at in_lltable_lookup+0x12b arpintr() at arpintr+0x9d6 netisr_dispatch_src() at netisr_dispatch_src+0x7e ether_demux() at ether_demux+0x14d ether_input() at ether_input+0x17b ether_demux() at ether_demux+0x6f ether_input() at ether_input+0x17b bce_intr() at bce_intr+0x3b0 intr_event_execute_handlers() at intr_event_execute_handlers+0xfd ithread_loop() at ithread_loop+0x8e fork_exit() at fork_exit+0x118 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 --- Other ddb data available per each crash (attached in shar). >How-To-Repeat: Happens by accident on a moderately loaded box. Usual numbers: 2799 processes:12 running, 2750 sleeping, 37 zombie packets errs idrops bytes packets errs bytes colls 1723 0 0 198578 2166 0 2415553 0 interrupt total rate irq4: uart0 309128 6 irq15: ata1 35 0 irq17: aac0 13404675 293 cpu0: timer 90914103 1993 irq256: bce0 2914454 63 irq257: bce1 109935240 2410 cpu1: timer 90974578 1994 cpu3: timer 91076473 1997 cpu2: timer 91059038 1996 cpu4: timer 90930409 1993 cpu5: timer 91033366 1996 cpu6: timer 91085723 1997 cpu7: timer 91111107 1997 Total 854748329 18743 >Fix: Patch attached with submission follows: # This is a shell archive. Save it in a file, remove anything before # this line, and then unpack it by entering "sh file". Note, it may # create directories; files and directories will be owned by you and # have default permissions. # # This archive contains: # # crash1.txt # crash2.txt # crash3.txt # echo x - crash1.txt sed 's/^X//' >crash1.txt << '24f218c6efe2375d7c9a281e6cc61910' Xdb> x/s *panicstr X0: *** error reading from address 0 *** Xdb> bt XTracing pid 12 tid 100039 td 0xffffff00029a23e0 Xpropagate_priority() at propagate_priority+0x72 Xturnstile_wait() at turnstile_wait+0x1aa X_rw_wlock_hard() at _rw_wlock_hard+0xfa Xin_lltable_lookup() at in_lltable_lookup+0x12b Xarpintr() at arpintr+0x9d6 Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e Xether_demux() at ether_demux+0x14d Xether_input() at ether_input+0x17b Xether_demux() at ether_demux+0x6f Xether_input() at ether_input+0x17b Xbce_intr() at bce_intr+0x3b0 Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd Xithread_loop() at ithread_loop+0x8e Xfork_exit() at fork_exit+0x118 Xfork_trampoline() at fork_trampoline+0xe X--- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 --- Xdb> show proc 12 XProcess 12 (intr) at 0xffffff00026e7000: X state: NORMAL X uid: 0 gids: 0 X parent: pid 0 at 0xffffffff80c7e3e0 X ABI: null X threads: 24 X100065 I [irq1: atkbd0] X100064 I [swi0: uart] X100063 I [irq15: ata1] X100062 I [irq14: ata0] X100045 I [irq22: uhci1 uhci3] X100040 I [irq23: uhci0 uhci2+] X100039 L *lle 0xffffff00094123c0 [irq257: bce1] X100038 I [irq256: bce0] X100036 I [irq17: aac0] X100035 I [irq9: acpi0] X100034 I [swi5: +] X100032 I [swi2: cambio] X100028 I [swi6: task queue] X100027 I [swi6: Giant taskq] X100020 I [swi3: vm] X100019 I [swi4: clock] X100018 I [swi4: clock] X100017 Run CPU 3 [swi4: clock] X100016 I [swi4: clock] X100015 Run CPU 4 [swi4: clock] X100014 I [swi4: clock] X100013 I [swi4: clock] X100012 I [swi4: clock] X100011 I [swi1: netisr 0] Xdb> ps X pid ppid pgrp uid state wmesg wchan cmd X32064 13554 13554 31888 RL CPU 5 httpd X32063 15369 15369 33268 S sbwait 0xffffff0054d33234 httpd X32062 15369 15369 33268 S select 0xffffff0245521bc0 httpd X32061 15369 15369 33268 S select 0xffffff00286cc0c0 httpd X32060 15369 15369 33268 S select 0xffffff01e78db640 httpd X32059 14231 14231 43024 S select 0xffffff0501ab61c0 httpd X32058 45079 45079 33717 S sbwait 0xffffff0397c31be4 httpd X32057 45079 45079 33717 S select 0xffffff035d057bc0 httpd X32056 32055 32055 22 S select 0xffffff0102867640 sshd X32055 1881 32055 0 Ss sbwait 0xffffff06011a3694 sshd X32054 13522 13522 33665 S accept 0xffffff0028c7430e httpd X32053 39869 39869 28053 S accept 0xffffff0397c20dae httpd X32052 15280 15280 32867 S select 0xffffff002c0dbbc0 httpd X32051 4253 4253 39534 S select 0xffffff02df7c57c0 httpd X32050 11560 11560 11477 S select 0xffffff0501aca3c0 httpd X32049 11560 11560 11477 S select 0xffffff00751ec8c0 httpd X32048 15369 15369 33268 S select 0xffffff0037190c40 httpd X32047 15369 15369 33268 S select 0xffffff0037ea24c0 httpd X32046 45079 45079 33717 S select 0xffffff01e73a17c0 httpd Xdb> show msgbuf Xmsgbufp = 0xffffffff81020fe0 Xmagic = 63062, size = 65504, r= 820871, w = 821375, ptr = 0xffffffff81011000, cksum= 5524040 Xkernel trap 12 with interrupts disabled X X XFatal trap 12: page fault while in kernel mode Xcpuid = 1; apic id = 01 Xfault virtual address = 0xc Xfault code = supervisor read data, page not present Xinstruction pointer = 0x20:0xffffffff805f30d2 Xstack pointer = 0x28:0xffffff82b155d830 Xframe pointer = 0x28:0xffffff82b155d860 Xcode segment = base 0x0, limit 0xfffff, type 0x1b X = DPL 0, pres 1, long 1, def32 0, gran 1 Xprocessor eflags = resume, IOPL = 0 Xcurrent process = 12 (irq257: bce1) X Xdb> show allpcpu XCurrent CPU: 1 X Xcpuid = 0 Xdynamic pcpu = 0x2a3f80 Xcurthread = 0xffffff04937443e0: pid 31741 "gtar" Xcurpcb = 0xffffff82b9fead40 Xfpcurthread = 0xffffff04937443e0: pid 31741 "gtar" Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0" Xcurpmap = 0 Xtssp = 0xffffffff80cf0080 Xcommontssp = 0xffffffff80cf0080 Xrsp0 = 0xffffff82b9fead40 Xgs32p = 0xffffffff80ceeeb8 Xldt = 0xffffffff80ceeef8 Xtss = 0xffffffff80ceeee8 X Xcpuid = 1 Xdynamic pcpu = 0xffffff807f418f80 Xcurthread = 0xffffff00029a23e0: pid 12 "irq257: bce1" Xcurpcb = 0xffffff82b155dd40 Xfpcurthread = none Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1" Xcurpmap = 0 Xtssp = 0xffffffff80cf00e8 Xcommontssp = 0xffffffff80cf00e8 Xrsp0 = 0xffffff82b155dd40 Xgs32p = 0xffffffff80ceef20 Xldt = 0xffffffff80ceef60 Xtss = 0xffffffff80ceef50 X Xcpuid = 2 Xdynamic pcpu = 0xffffff807f41ff80 Xcurthread = 0xffffff02df2d57c0: pid 73366 "sendbackup" Xcurpcb = 0xffffff82b7bfad40 Xfpcurthread = none Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2" Xcurpmap = 0 Xtssp = 0xffffffff80cf0150 Xcommontssp = 0xffffffff80cf0150 Xrsp0 = 0xffffff82b7bfad40 Xgs32p = 0xffffffff80ceef88 Xldt = 0xffffffff80ceefc8 Xtss = 0xffffffff80ceefb8 X Xcpuid = 3 Xdynamic pcpu = 0xffffff807f426f80 Xcurthread = 0xffffff00026fb000: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000e2d40 Xfpcurthread = none Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3" Xcurpmap = 0 Xtssp = 0xffffffff80cf01b8 Xcommontssp = 0xffffffff80cf01b8 Xrsp0 = 0xffffff80000e2d40 Xgs32p = 0xffffffff80ceeff0 Xldt = 0xffffffff80cef030 Xtss = 0xffffffff80cef020 X Xcpuid = 4 Xdynamic pcpu = 0xffffff807f42df80 Xcurthread = 0xffffff00026fb7c0: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000d8d40 Xfpcurthread = none Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4" Xcurpmap = 0 Xtssp = 0xffffffff80cf0220 Xcommontssp = 0xffffffff80cf0220 Xrsp0 = 0xffffff80000d8d40 Xgs32p = 0xffffffff80cef058 Xldt = 0xffffffff80cef098 Xtss = 0xffffffff80cef088 X Xcpuid = 5 Xdynamic pcpu = 0xffffff807f434f80 Xcurthread = 0xffffff008f1b67c0: pid 32064 "httpd" Xcurpcb = 0xffffff82b6ca0d40 Xfpcurthread = none Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5" Xcurpmap = 0 Xtssp = 0xffffffff80cf0288 Xcommontssp = 0xffffffff80cf0288 Xrsp0 = 0xffffff82b6ca0d40 Xgs32p = 0xffffffff80cef0c0 Xldt = 0xffffffff80cef100 Xtss = 0xffffffff80cef0f0 X Xcpuid = 6 Xdynamic pcpu = 0xffffff807f43bf80 Xcurthread = 0xffffff009cc0c000: pid 3649 "gtar" Xcurpcb = 0xffffff82b8b27d40 Xfpcurthread = 0xffffff009cc0c000: pid 3649 "gtar" Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6" Xcurpmap = 0 Xtssp = 0xffffffff80cf02f0 Xcommontssp = 0xffffffff80cf02f0 Xrsp0 = 0xffffff82b8b27d40 Xgs32p = 0xffffffff80cef128 Xldt = 0xffffffff80cef168 Xtss = 0xffffffff80cef158 X Xcpuid = 7 Xdynamic pcpu = 0xffffff807f442f80 Xcurthread = 0xffffff00026e97c0: pid 11 "idle: cpu7" Xcurpcb = 0xffffff800009cd40 Xfpcurthread = none Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7" Xcurpmap = 0 Xtssp = 0xffffffff80cf0358 Xcommontssp = 0xffffffff80cf0358 Xrsp0 = 0xffffff800009cd40 Xgs32p = 0xffffffff80cef190 Xldt = 0xffffffff80cef1d0 Xtss = 0xffffffff80cef1c0 X Xdb> show lockedvnods XLocked vnodes X X0xffffff01ced5f3b0: tag ufs, type VREG X usecount 1, writecount 0, refcount 39883 mountedhere 0 X flags () X v_object 0xffffff008f8ff948 ref 0 pages 159524 X lock type ufs: SHARED (count 1) X ino 108758910, on dev aacd0s1g X X0xffffff0397e37ce8: tag ufs, type VDIR X usecount 1, writecount 0, refcount 2 mountedhere 0 X flags () X v_object 0xffffff0397e3de58 ref 0 pages 0 X lock type ufs: SHARED (count 1) X ino 172089992, on dev aacd0s1g X X0xffffff04647aa588: tag ufs, type VREG X usecount 4, writecount 0, refcount 5 mountedhere 0 X flags () X v_object 0xffffff04d13a55e8 ref 2 pages 6 X lock type ufs: SHARED (count 1) X ino 44308534, on dev aacd0s1g Xdb> bt 31741 XTracing pid 31741 tid 104937 td 0xffffff04937443e0 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffffff80cf7e40, rbp = 0xffffff82b9fea550 --- XDELAY() at DELAY+0x64 X_thread_lock_flags() at _thread_lock_flags+0xb0 Xintr_event_schedule_thread() at intr_event_schedule_thread+0x72 Xcallout_tick() at callout_tick+0xf8 Xhardclock_cpu() at hardclock_cpu+0xd2 Xhardclock() at hardclock+0x23 Xlapic_handle_timer() at lapic_handle_timer+0xd2 XXtimerint() at Xtimerint+0x8c X--- interrupt, rip = 0xffffffff806472f3, rsp = 0xffffff82b9fea770, rbp = 0xffffff82b9fea7d0 --- Xvputx() at vputx+0x183 Xlookup() at lookup+0x280 Xnamei() at namei+0x52b Xkern_statat_vnhook() at kern_statat_vnhook+0x8f Xkern_statat() at kern_statat+0x15 Xlstat() at lstat+0x2a Xsyscall() at syscall+0x1e7 XXfast_syscall() at Xfast_syscall+0xe1 X--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800a7c10c, rsp = 0x7fffffffe6c8, rbp = 0x801295cf9 --- Xdb> bt 11 XTracing pid 11 tid 100010 td 0xffffff00026ea7c0 Xsched_switch() at sched_switch+0xea Xmi_switch() at mi_switch+0x16f Xsched_preempt() at sched_preempt+0xb5 Xipi_bitmap_handler() at ipi_bitmap_handler+0x70 XXipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b X--- interrupt, rip = 0xffffffff805a4209, rsp = 0xffffff80000bfb60, rbp = 0xffffff80000bfba0 --- X_thread_lock_flags() at _thread_lock_flags+0x69 Xsched_idletd() at sched_idletd+0xde Xfork_exit() at fork_exit+0x118 Xfork_trampoline() at fork_trampoline+0xe X--- trap 0, rip = 0, rsp = 0xffffff80000bfd30, rbp = 0 --- Xdb> bt 12 XTracing pid 12 tid 100065 td 0xffffff0009001ba0 Xfork_trampoline() at fork_trampoline Xdb> bt 73366 XTracing pid 73366 tid 103097 td 0xffffff02df2d57c0 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffff8000059fe0, rbp = 0xffffff82b7bfa930 --- XDELAY() at DELAY+0x64 X_mtx_lock_spin() at _mtx_lock_spin+0x6a Xsched_add() at sched_add+0x117 Xsetrunnable() at setrunnable+0x78 Xsleepq_broadcast() at sleepq_broadcast+0x7c Xwakeup() at wakeup+0x27 Xpipe_read() at pipe_read+0x3ff Xdofileread() at dofileread+0xa1 Xkern_readv() at kern_readv+0x60 Xread() at read+0x55 Xsyscall() at syscall+0x1e7 XXfast_syscall() at Xfast_syscall+0xe1 X--- syscall (3, FreeBSD ELF64, read), rip = 0x8019033ec, rsp = 0x7fffffff6058, rbp = 0x400 --- Xdb> bt 32064 XTracing pid 32064 tid 102311 td 0xffffff008f1b67c0 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffff800006efe0, rbp = 0xffffff82b6ca01f0 --- XDELAY() at DELAY+0x64 X_mtx_lock_spin() at _mtx_lock_spin+0x6a Xsmp_tlb_shootdown() at smp_tlb_shootdown+0xfa Xpmap_invalidate_range() at pmap_invalidate_range+0xae Xvnode_pager_generic_getpages() at vnode_pager_generic_getpages+0xa69 Xvnode_pager_getpages() at vnode_pager_getpages+0x91 Xvm_fault() at vm_fault+0x644 Xtrap_pfault() at trap_pfault+0x132 Xtrap() at trap+0x35e Xcalltrap() at calltrap+0x8 X--- trap 0xc, rip = 0xffffffff808bfe0d, rsp = 0xffffff82b6ca08f0, rbp = 0xffffff82b6ca0970 --- Xcopyin() at copyin+0x3d Xm_uiotombuf() at m_uiotombuf+0xcb Xsosend_generic() at sosend_generic+0x328 Xsoo_write() at soo_write+0x5e Xdofilewrite() at dofilewrite+0x85 Xkern_writev() at kern_writev+0x60 Xfreebsd32_writev() at freebsd32_writev+0x41 Xia32_syscall() at ia32_syscall+0x1eb XXint0x80_syscall() at Xint0x80_syscall+0x95 X--- syscall (121, FreeBSD ELF32, freebsd32_writev), rip = 0x281a092f, rsp = 0xffffd83c, rbp = 0xffffd868 --- Xdb> bt 3649 XTracing pid 3649 tid 103874 td 0xffffff009cc0c000 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808b5348, rsp = 0xffffff8000075fe0, rbp = 0xffffff82b8b27650 --- Xsmp_tlb_shootdown() at smp_tlb_shootdown+0x98 Xpmap_invalidate_range() at pmap_invalidate_range+0xae Xvfs_vmio_release() at vfs_vmio_release+0x143 Xgetnewbuf() at getnewbuf+0x9be Xgetblk() at getblk+0x368 Xcluster_rbuild() at cluster_rbuild+0x761 Xcluster_read() at cluster_read+0x502 Xffs_read() at ffs_read+0x3bc Xvn_read() at vn_read+0x2d9 Xdofileread() at dofileread+0xa1 Xkern_readv() at kern_readv+0x60 Xread() at read+0x55 Xsyscall() at syscall+0x1e7 XXfast_syscall() at Xfast_syscall+0xe1 X--- syscall (3, FreeBSD ELF64, read), rip = 0x800a8b3ec, rsp = 0x7fffffffe6c8, rbp = 0x131d3ea4 --- Xdb> X 24f218c6efe2375d7c9a281e6cc61910 echo x - crash2.txt sed 's/^X//' >crash2.txt << '67367b08c0869beabc309c4d0c66c9a9' Xdb> bt XTracing pid 12 tid 100039 td 0xffffff00029a23e0 Xpropagate_priority() at propagate_priority+0x72 Xturnstile_wait() at turnstile_wait+0x1aa X_rw_wlock_hard() at _rw_wlock_hard+0xfa Xin_lltable_lookup() at in_lltable_lookup+0x12b Xarpintr() at arpintr+0x9d6 Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e Xether_demux() at ether_demux+0x14d Xether_input() at ether_input+0x17b Xether_demux() at ether_demux+0x6f Xether_input() at ether_input+0x17b Xbce_intr() at bce_intr+0x3b0 Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd Xithread_loop() at ithread_loop+0x8e Xfork_exit() at fork_exit+0x118 Xfork_trampoline() at fork_trampoline+0xe X--- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 --- Xdb> show proc 12 XProcess 12 (intr) at 0xffffff00026e7000: X state: NORMAL X uid: 0 gids: 0 X parent: pid 0 at 0xffffffff80c7e3e0 X ABI: null X threads: 24 X100065 I [irq1: atkbd0] X100064 I [swi0: uart uart] X100063 I [irq15: ata1] X100062 I [irq14: ata0] X100045 I [irq22: uhci1 uhci3] X100040 I [irq23: uhci0 uhci2+] X100039 L *lle 0xffffff032af863c0 [irq257: bce1] X100038 I [irq256: bce0] X100036 I [irq17: aac0] X100035 I [irq9: acpi0] X100034 I [swi5: +] X100032 I [swi2: cambio] X100028 I [swi6: task queue] X100027 I [swi6: Giant taskq] X100020 I [swi3: vm] X100019 Run CPU 1 [swi4: clock] X100018 Run CPU 5 [swi4: clock] X100017 Run CPU 6 [swi4: clock] X100016 I [swi4: clock] X100015 Run CPU 3 [swi4: clock] X100014 I [swi4: clock] X100013 CanRun [swi4: clock] X100012 I [swi4: clock] X100011 I [swi1: netisr 0] Xdb> ps X pid ppid pgrp uid state wmesg wchan cmd X53746 53743 51388 0 R CPU 4 head X53745 53743 51388 0 S piperd 0xffffff050c5ea5b0 sort X53744 53743 51388 0 R find X53743 51397 51388 0 S wait 0xffffff04bfd5b460 sh X53626 49623 49623 37465 S accept 0xffffff022ab955b6 httpd X53625 49623 49623 37465 S accept 0xffffff022ab955b6 httpd X53575 65122 65122 5597 S accept 0xffffff04e8ef030e httpd X53560 68225 68225 44526 S accept 0xffffff0156956066 httpd X53552 98656 98656 45454 S accept 0xffffff02e9975066 httpd X53550 9163 9163 45695 S accept 0xffffff041026cb06 httpd X53549 9163 9163 45695 S accept 0xffffff041026cb06 httpd X53543 62632 62632 42270 S sbwait 0xffffff022a8d44dc httpd X53541 62632 62632 42270 S select 0xffffff01a2fb6440 httpd X53530 67266 67266 36958 S sbwait 0xffffff04c29243ec httpd X53509 22498 65331 10395 R perl5.8.8 X53410 49623 49623 37465 S select 0xffffff015c6afbc0 httpd X53357 65545 65545 10900 S select 0xffffff02f289ddc0 httpd X53335 66113 66113 16234 S lockf 0xffffff03d23c1100 httpd X53334 65251 65251 8335 S accept 0xffffff060b393b06 httpd X53309 9163 9163 45695 S select 0xffffff01a2cb1240 httpd X53306 62632 62632 42270 S select 0xffffff01a2628a40 httpd X53305 86977 86977 45532 S select 0xffffff010971bb40 httpd X53201 65429 65429 9400 S accept 0xffffff05aeea5dae httpd X53200 65429 65429 9400 S accept 0xffffff05aeea5dae httpd X53199 65429 65429 9400 S accept 0xffffff05aeea5dae httpd X53110 66113 66113 16234 S select 0xffffff045cabcac0 httpd X53104 65604 65604 13307 S select 0xffffff01a295b540 httpd X53103 88924 88924 11904 S accept 0xffffff0009a3d85e httpd X53101 65141 65141 10330 S accept 0xffffff01dce31dae httpd X53097 67266 67266 36958 S select 0xffffff0135c45e40 httpd X53096 68233 68233 47018 S accept 0xffffff012194e30e httpd X53095 67266 67266 36958 S select 0xffffff01a295e440 httpd X53094 68233 68233 47018 S accept 0xffffff012194e30e httpd X53041 65429 65429 9400 S sbwait 0xffffff0153601e8c httpd X53040 65429 65429 9400 S sbwait 0xffffff02db91d144 httpd X53037 66752 66752 25536 S select 0xffffff01a2bd7640 httpd X53036 66752 66752 25536 S lockf 0xffffff0153cfa300 httpd X53034 66752 66752 25536 S lockf 0xffffff01a29efa80 httpd X53033 66752 66752 25536 S lockf 0xffffff0135cf0d80 httpd X53032 66752 66752 25536 S lockf 0xffffff017f231200 httpd X53030 66752 66752 25536 S lockf 0xffffff028d2f6380 httpd Xdb> show msgbuf Xmsgbufp = 0xffffffff81020fe0 Xmagic = 63062, size = 65504, r= 554532, w = 555036, ptr = 0xffffffff81011000, cksum= 5605281 Xkernel trap 12 with interrupts disabled X X XFatal trap 12: page fault while in kernel mode Xcpuid = 2; apic id = 02 Xfault virtual address = 0xc Xfault code = supervisor read data, page not present Xinstruction pointer = 0x20:0xffffffff805f30c2 Xstack pointer = 0x28:0xffffff82b155d830 Xframe pointer = 0x28:0xffffff82b155d860 Xcode segment = base 0x0, limit 0xfffff, type 0x1b X = DPL 0, pres 1, long 1, def32 0, gran 1 Xprocessor eflags = resume, IOPL = 0 Xcurrent process = 12 (irq257: bce1) X Xdb> show allpcpu XCurrent CPU: 2 X Xcpuid = 0 Xdynamic pcpu = 0x2a3f80 Xcurthread = 0xffffff0061aca000: pid 52989 "httpd" Xcurpcb = 0xffffff82b7e2fd40 Xfpcurthread = none Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0" Xcurpmap = 0 Xtssp = 0xffffffff80cf0080 Xcommontssp = 0xffffffff80cf0080 Xrsp0 = 0xffffff82b7e2fd40 Xgs32p = 0xffffffff80ceeeb8 Xldt = 0xffffffff80ceeef8 Xtss = 0xffffffff80ceeee8 X Xcpuid = 1 Xdynamic pcpu = 0xffffff807f418f80 Xcurthread = 0xffffff00027003e0: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000ecd40 Xfpcurthread = none Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1" Xcurpmap = 0 Xtssp = 0xffffffff80cf00e8 Xcommontssp = 0xffffffff80cf00e8 Xrsp0 = 0xffffff80000ecd40 Xgs32p = 0xffffffff80ceef20 Xldt = 0xffffffff80ceef60 Xtss = 0xffffffff80ceef50 X Xcpuid = 2 Xdynamic pcpu = 0xffffff807f41ff80 Xcurthread = 0xffffff00029a23e0: pid 12 "irq257: bce1" Xcurpcb = 0xffffff82b155dd40 Xfpcurthread = none Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2" Xcurpmap = 0 Xtssp = 0xffffffff80cf0150 Xcommontssp = 0xffffffff80cf0150 Xrsp0 = 0xffffff82b155dd40 Xgs32p = 0xffffffff80ceef88 Xldt = 0xffffffff80ceefc8 Xtss = 0xffffffff80ceefb8 X Xcpuid = 3 Xdynamic pcpu = 0xffffff807f426f80 Xcurthread = 0xffffff00026fb7c0: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000d8d40 Xfpcurthread = none Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3" Xcurpmap = 0 Xtssp = 0xffffffff80cf01b8 Xcommontssp = 0xffffffff80cf01b8 Xrsp0 = 0xffffff80000d8d40 Xgs32p = 0xffffffff80ceeff0 Xldt = 0xffffffff80cef030 Xtss = 0xffffffff80cef020 X Xcpuid = 4 Xdynamic pcpu = 0xffffff807f42df80 Xcurthread = 0xffffff014bbbbba0: pid 53746 "head" Xcurpcb = 0xffffff82b92c0d40 Xfpcurthread = none Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4" Xcurpmap = 0 Xtssp = 0xffffffff80cf0220 Xcommontssp = 0xffffffff80cf0220 Xrsp0 = 0xffffff82b92c0d40 Xgs32p = 0xffffffff80cef058 Xldt = 0xffffffff80cef098 Xtss = 0xffffffff80cef088 X Xcpuid = 5 Xdynamic pcpu = 0xffffff807f434f80 Xcurthread = 0xffffff00026f6ba0: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000e7d40 Xfpcurthread = none Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5" Xcurpmap = 0 Xtssp = 0xffffffff80cf0288 Xcommontssp = 0xffffffff80cf0288 Xrsp0 = 0xffffff80000e7d40 Xgs32p = 0xffffffff80cef0c0 Xldt = 0xffffffff80cef100 Xtss = 0xffffffff80cef0f0 X Xcpuid = 6 Xdynamic pcpu = 0xffffff807f43bf80 Xcurthread = 0xffffff00026fb000: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000e2d40 Xfpcurthread = none Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6" Xcurpmap = 0 Xtssp = 0xffffffff80cf02f0 Xcommontssp = 0xffffffff80cf02f0 Xrsp0 = 0xffffff80000e2d40 Xgs32p = 0xffffffff80cef128 Xldt = 0xffffffff80cef168 Xtss = 0xffffffff80cef158 X Xcpuid = 7 Xdynamic pcpu = 0xffffff807f442f80 Xcurthread = 0xffffff008976eba0: pid 45168 "httpd" Xcurpcb = 0xffffff82b71b4d40 Xfpcurthread = 0xffffff008976eba0: pid 45168 "httpd" Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7" Xcurpmap = 0 Xtssp = 0xffffffff80cf0358 Xcommontssp = 0xffffffff80cf0358 Xrsp0 = 0xffffff82b71b4d40 Xgs32p = 0xffffffff80cef190 Xldt = 0xffffffff80cef1d0 Xtss = 0xffffffff80cef1c0 X Xdb> show lockedvnods XLocked vnodes X X0xffffff0109771938: tag ufs, type VDIR X usecount 3, writecount 0, refcount 11 mountedhere 0 X flags () X v_object 0xffffff01096ad948 ref 0 pages 23 X lock type ufs: SHARED (count 1) X ino 4752, on dev aacd0s1f Xdb> bt 52989 XTracing pid 52989 tid 103210 td 0xffffff0061aca000 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808c45c2, rsp = 0xffffffff80cf7e40, rbp = 0xffffff82b7e2f8d0 --- XDELAY() at DELAY+0x62 X_thread_lock_flags() at _thread_lock_flags+0xb0 Xsleepq_add() at sleepq_add+0x8c X_sleep() at _sleep+0x159 Xsoreceive_generic() at soreceive_generic+0xeba Xdofileread() at dofileread+0xa1 Xkern_readv() at kern_readv+0x60 Xread() at read+0x55 Xia32_syscall() at ia32_syscall+0x1eb XXint0x80_syscall() at Xint0x80_syscall+0x95 X--- syscall (3, FreeBSD ELF32, read), rip = 0x281b834f, rsp = 0xffffb89c, rbp = 0xffffb8b8 --- Xdb> bt 53746 XTracing pid 53746 tid 104263 td 0xffffff014bbbbba0 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff808c45c4, rsp = 0xffffff8000067fe0, rbp = 0xffffff82b92c05e0 --- XDELAY() at DELAY+0x64 X_thread_lock_flags() at _thread_lock_flags+0xb0 Xintr_event_schedule_thread() at intr_event_schedule_thread+0x72 Xcallout_tick() at callout_tick+0xf8 Xhardclock_cpu() at hardclock_cpu+0xd2 Xlapic_handle_timer() at lapic_handle_timer+0xff XXtimerint() at Xtimerint+0x8c X--- interrupt, rip = 0xffffffff808be298, rsp = 0xffffff82b92c07e0, rbp = 0xffffff82b92c0840 --- Xpmap_remove() at pmap_remove+0x2a8 Xvm_map_delete() at vm_map_delete+0xf4 Xvm_map_fixed() at vm_map_fixed+0x78 Xvm_mmap() at vm_mmap+0x51d Xmmap() at mmap+0x219 Xsyscall() at syscall+0x1e7 XXfast_syscall() at Xfast_syscall+0xe1 X--- syscall (477, FreeBSD ELF64, mmap), rip = 0x80050c86c, rsp = 0x7fffffffe278, rbp = 0x7fffffffe370 --- Xdb> bt 45168 XTracing pid 45168 tid 102571 td 0xffffff008976eba0 Xcpustop_handler() at cpustop_handler+0x40 Xipi_nmi_handler() at ipi_nmi_handler+0x30 Xtrap() at trap+0x175 Xnmi_calltrap() at nmi_calltrap+0x8 X--- trap 0x13, rip = 0xffffffff805a3d8d, rsp = 0xffffff800007cfe0, rbp = 0xffffff82b71b4860 --- X_mtx_lock_sleep() at _mtx_lock_sleep+0xdd Xpmap_enter() at pmap_enter+0x2f7 Xvm_fault() at vm_fault+0x14dc Xtrap_pfault() at trap_pfault+0x132 Xtrap() at trap+0x4dc Xcalltrap() at calltrap+0x8 X--- trap 0xc, rip = 0x281b7b96, rsp = 0xfffe2424, rbp = 0xfffe24e8 --- Xdb> bt 11 XTracing pid 11 tid 100010 td 0xffffff00026ea7c0 Xsched_switch() at sched_switch+0xea Xmi_switch() at mi_switch+0x16f Xsched_preempt() at sched_preempt+0xb5 Xipi_bitmap_handler() at ipi_bitmap_handler+0x70 XXipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b X--- interrupt, rip = 0xffffffff805a41f4, rsp = 0xffffff80000bfb60, rbp = 0xffffff80000bfba0 --- X_thread_lock_flags() at _thread_lock_flags+0x64 Xsched_idletd() at sched_idletd+0xde Xfork_exit() at fork_exit+0x118 Xfork_trampoline() at fork_trampoline+0xe X--- trap 0, rip = 0, rsp = 0xffffff80000bfd30, rbp = 0 --- Xdb> bt 12 XTracing pid 12 tid 100065 td 0xffffff0009001ba0 Xfork_trampoline() at fork_trampoline X 67367b08c0869beabc309c4d0c66c9a9 echo x - crash3.txt sed 's/^X//' >crash3.txt << 'd8e1b7c43219a397cc6f0ec53abfcb51' Xdb> bt XTracing pid 12 tid 100038 td 0xffffff00029a17c0 Xpropagate_priority() at propagate_priority+0x72 Xturnstile_wait() at turnstile_wait+0x1aa X_rw_wlock_hard() at _rw_wlock_hard+0xfa Xin_lltable_lookup() at in_lltable_lookup+0x12b Xarpintr() at arpintr+0x9d6 Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e Xether_demux() at ether_demux+0x14d Xether_input() at ether_input+0x17b Xbce_intr() at bce_intr+0x3b0 Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd Xithread_loop() at ithread_loop+0x8e Xfork_exit() at fork_exit+0x118 Xfork_trampoline() at fork_trampoline+0xe X--- trap 0, rip = 0, rsp = 0xffffff82b1554d30, rbp = 0 --- Xdb> show proc 12 XProcess 12 (intr) at 0xffffff00026e7000: X state: NORMAL X uid: 0 gids: 0 X parent: pid 0 at 0xffffffff80c7e3e0 X ABI: null X threads: 24 X100065 I [irq1: atkbd0] X100064 I [swi0: uart] X100063 I [irq15: ata1] X100062 I [irq14: ata0] X100045 I [irq22: uhci1 uhci3] X100040 I [irq23: uhci0 uhci2+] X100039 I [irq257: bce1] X100038 L *lle 0xffffff05ad99b9c0 [irq256: bce0] X100036 Run CPU 4 [irq17: aac0] X100035 I [irq9: acpi0] X100034 I [swi5: +] X100032 I [swi2: cambio] X100028 I [swi6: task queue] X100027 I [swi6: Giant taskq] X100020 I [swi3: vm] X100019 CanRun [swi4: clock] X100018 I [swi4: clock] X100017 CanRun [swi4: clock] X100016 I [swi4: clock] X100015 I [swi4: clock] X100014 Run CPU 2 [swi4: clock] X100013 Run CPU 3 [swi4: clock] X100012 I [swi4: clock] X100011 I [swi1: netisr 0] Xdb> ps X pid ppid pgrp uid state wmesg wchan cmd X22640 13700 13700 27830 S select 0xffffff018dc0d640 httpd X22639 13700 13700 27830 S select 0xffffff00a2555440 httpd X22638 13700 13700 27830 S select 0xffffff0208c336c0 httpd X22637 13700 13700 27830 S select 0xffffff038073c7c0 httpd X22636 13700 13700 27830 S select 0xffffff0142b5c340 httpd X22635 13700 13700 27830 S select 0xffffff038082c6c0 httpd X22634 13700 13700 27830 S select 0xffffff0137773040 httpd X22633 13700 13700 27830 S select 0xffffff04925374c0 httpd X22632 10373 10373 11233 S accept 0xffffff00451bdb06 httpd X22631 10433 10433 12845 S sbwait 0xffffff01b40fff7c httpd X22630 10120 10120 10980 R httpd X22629 13700 13700 27830 S select 0xffffff0009492640 httpd X22628 13700 13700 27830 S select 0xffffff04924a9840 httpd X22627 13700 13700 27830 S select 0xffffff0492737b40 httpd X22626 13700 13700 27830 S select 0xffffff00a2823340 httpd X22624 13700 13700 27830 S select 0xffffff0492c13cc0 httpd X22623 13700 13700 27830 S select 0xffffff0124394140 httpd X22621 12104 12104 45628 S accept 0xffffff00b22c55b6 httpd X22620 10099 10099 11299 S accept 0xffffff0009afcb06 httpd Xdb> show msgbuf Xmsgbufp = 0xffffffff81020fe0 Xmagic = 63062, size = 65504, r= 972658, w = 973162, ptr = 0xffffffff81011000, cksum= 5556648 Xkernel trap 12 with interrupts disabled X X XFatal trap 12: page fault while in kernel mode Xcpuid = 0; apic id = 00 Xfault virtual address = 0xc Xfault code = supervisor read data, page not present Xinstruction pointer = 0x20:0xffffffff805f30d2 Xstack pointer = 0x28:0xffffff82b1554890 Xframe pointer = 0x28:0xffffff82b15548c0 Xcode segment = base 0x0, limit 0xfffff, type 0x1b X = DPL 0, pres 1, long 1, def32 0, gran 1 Xprocessor eflags = resume, IOPL = 0 Xcurrent process = 12 (irq256: bce0) X Xdb> show allpcpu XCurrent CPU: 0 X Xcpuid = 0 Xdynamic pcpu = 0x2a3f80 Xcurthread = 0xffffff00029a17c0: pid 12 "irq256: bce0" Xcurpcb = 0xffffff82b1554d40 Xfpcurthread = none Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0" Xcurpmap = 0 Xtssp = 0xffffffff80cf0080 Xcommontssp = 0xffffffff80cf0080 Xrsp0 = 0xffffff82b1554d40 Xgs32p = 0xffffffff80ceeeb8 Xldt = 0xffffffff80ceeef8 Xtss = 0xffffffff80ceeee8 X Xcpuid = 1 Xdynamic pcpu = 0xffffff807f418f80 Xcurthread = 0xffffff0380f1d000: pid 11389 "httpd" Xcurpcb = 0xffffff82b81f9d40 Xfpcurthread = none Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1" Xcurpmap = 0 Xtssp = 0xffffffff80cf00e8 Xcommontssp = 0xffffffff80cf00e8 Xrsp0 = 0xffffff82b81f9d40 Xgs32p = 0xffffffff80ceef20 Xldt = 0xffffffff80ceef60 Xtss = 0xffffffff80ceef50 X Xcpuid = 2 Xdynamic pcpu = 0xffffff807f41ff80 Xcurthread = 0xffffff00026fbba0: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000d3d40 Xfpcurthread = none Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2" Xcurpmap = 0 Xtssp = 0xffffffff80cf0150 Xcommontssp = 0xffffffff80cf0150 Xrsp0 = 0xffffff80000d3d40 Xgs32p = 0xffffffff80ceef88 Xldt = 0xffffffff80ceefc8 Xtss = 0xffffffff80ceefb8 X Xcpuid = 3 Xdynamic pcpu = 0xffffff807f426f80 Xcurthread = 0xffffff00026fc000: pid 12 "swi4: clock" Xcurpcb = 0xffffff80000ced40 Xfpcurthread = none Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3" Xcurpmap = 0 Xtssp = 0xffffffff80cf01b8 Xcommontssp = 0xffffffff80cf01b8 Xrsp0 = 0xffffff80000ced40 Xgs32p = 0xffffffff80ceeff0 Xldt = 0xffffffff80cef030 Xtss = 0xffffffff80cef020 X Xcpuid = 4 Xdynamic pcpu = 0xffffff807f42df80 Xcurthread = 0xffffff00029a3000: pid 12 "irq17: aac0" Xcurpcb = 0xffffff82b1546d40 Xfpcurthread = none Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4" Xcurpmap = 0 Xtssp = 0xffffffff80cf0220 Xcommontssp = 0xffffffff80cf0220 Xrsp0 = 0xffffff82b1546d40 Xgs32p = 0xffffffff80cef058 Xldt = 0xffffffff80cef098 Xtss = 0xffffffff80cef088 X Xcpuid = 5 Xdynamic pcpu = 0xffffff807f434f80 Xcurthread = 0xffffff0020b46000: pid 21095 "httpd" Xcurpcb = 0xffffff82b5193d40 Xfpcurthread = 0xffffff0020b46000: pid 21095 "httpd" Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5" Xcurpmap = 0 Xtssp = 0xffffffff80cf0288 Xcommontssp = 0xffffffff80cf0288 Xrsp0 = 0xffffff82b5193d40 Xgs32p = 0xffffffff80cef0c0 Xldt = 0xffffffff80cef100 Xtss = 0xffffffff80cef0f0 X Xcpuid = 6 Xdynamic pcpu = 0xffffff807f43bf80 Xcurthread = 0xffffff00026e93e0: pid 11 "idle: cpu6" Xcurpcb = 0xffffff80000a1d40 Xfpcurthread = none Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6" Xcurpmap = 0 Xtssp = 0xffffffff80cf02f0 Xcommontssp = 0xffffffff80cf02f0 Xrsp0 = 0xffffff80000a1d40 Xgs32p = 0xffffffff80cef128 Xldt = 0xffffffff80cef168 Xtss = 0xffffffff80cef158 Xcpuid = 7 Xdynamic pcpu = 0xffffff807f442f80 Xcurthread = 0xffffff00026e97c0: pid 11 "idle: cpu7" Xcurpcb = 0xffffff800009cd40 Xfpcurthread = none Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7" Xcurpmap = 0 Xtssp = 0xffffffff80cf0358 Xcommontssp = 0xffffffff80cf0358 Xrsp0 = 0xffffff800009cd40 Xgs32p = 0xffffffff80cef190 Xldt = 0xffffffff80cef1d0 Xtss = 0xffffffff80cef1c0 X Xdb> show lockedvnods XLocked vnodes X X0xffffff0515307760: tag ufs, type VREG X usecount 1, writecount 0, refcount 3 mountedhere 0 X flags () X v_object 0xffffff03effa7870 ref 0 pages 1 X lock type ufs: SHARED (count 1) X ino 115366410, on dev aacd0s1g X d8e1b7c43219a397cc6f0ec53abfcb51 exit >Release-Note: >Audit-Trail: >Unformatted: