From owner-freebsd-bugs@FreeBSD.ORG Fri Jan 16 17:30:01 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D59AA106566B for ; Fri, 16 Jan 2009 17:30:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AEE5E8FC19 for ; Fri, 16 Jan 2009 17:30:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n0GHU1bk009403 for ; Fri, 16 Jan 2009 17:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n0GHU1tV009397; Fri, 16 Jan 2009 17:30:01 GMT (envelope-from gnats) Resent-Date: Fri, 16 Jan 2009 17:30:01 GMT Resent-Message-Id: <200901161730.n0GHU1tV009397@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5592106566B for ; Fri, 16 Jan 2009 17:25:15 +0000 (UTC) (envelope-from root@zavijava4.math.ualberta.ca) Received: from zavijava4.math.ualberta.ca (zavijava4.math.ualberta.ca [129.128.206.34]) by mx1.freebsd.org (Postfix) with ESMTP id 670818FC14 for ; Fri, 16 Jan 2009 17:25:15 +0000 (UTC) (envelope-from root@zavijava4.math.ualberta.ca) Received: from zavijava4.math.ualberta.ca (localhost.math.ualberta.ca [127.0.0.1]) by zavijava4.math.ualberta.ca (8.14.3/8.14.3) with ESMTP id n0GGqE3A008424 for ; Fri, 16 Jan 2009 09:52:14 -0700 (MST) (envelope-from root@zavijava4.math.ualberta.ca) Received: (from root@localhost) by zavijava4.math.ualberta.ca (8.14.3/8.14.3/Submit) id n0GGqEWo008423; Fri, 16 Jan 2009 09:52:14 -0700 (MST) (envelope-from root) Message-Id: <200901161652.n0GGqEWo008423@zavijava4.math.ualberta.ca> Date: Fri, 16 Jan 2009 09:52:14 -0700 (MST) From: To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/130628: NFS / rpc.lockd deadlock on 7.1-R X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bvowk@math.ualberta.ca List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2009 17:30:02 -0000 >Number: 130628 >Category: kern >Synopsis: NFS / rpc.lockd deadlock on 7.1-R >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Jan 16 17:30:01 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Barkley Vowk >Release: FreeBSD 7.1-RELEASE-p1 amd64 >Organization: University of Alberta >Environment: System: FreeBSD zavijava4.math.ualberta.ca 7.1-RELEASE-p1 FreeBSD 7.1-RELEASE-p1 #10: Wed Jan 14 12:13:19 MST 2009 root@zavijava4.math.ualberta.ca:/usr/obj/usr/src/sys/GENERIC amd64 >Description: 7.1-R deadlocks when acting as an NFS server for a debian box running 2.6.18-6-amd64. The problem can be reproduced in a few minutes with dbench running on the linux machine into an nfs mounted directory from the 7.1 box. >How-To-Repeat: mount zj4:/home /mnt ; cd /mnt ; dbench -D . 5 With a generic kernel, it lasts about 5 seconds. With all the debugging options it can last a few minutes. I've included two dumps as requested from the debugging deadlocks section of the handbook. KDB: enter: Break sequence on console [thread pid 36 tid 100035 ] Stopped at kdb_enter_why+0x3d: movq $0,0x608808(%rip) db> ps pid ppid pgrp uid state wmesg wchan cmd 898 894 898 0 S+ select 0xffffffff80bc59e0 top 894 891 894 0 Ss+ pause 0xffffff000370e5c8 csh 891 818 891 0 Ss select 0xffffffff80bc59e0 sshd 890 874 48 0 S nanslp 0xffffffff80ade5e8 fsck_ufs 885 1 885 0 Ss+ ttyin 0xffffff0002f07c10 getty 884 1 884 0 Ss+ ttyin 0xffffff0002f29810 getty 883 1 883 0 Ss+ ttyin 0xffffff0002f29c10 getty 882 1 882 0 Ss+ ttyin 0xffffff0002f2a010 getty 881 1 881 0 Ss+ ttyin 0xffffff0002f2b410 getty 880 1 880 0 Ss+ ttyin 0xffffff0002f1d810 getty 879 1 879 0 Ss+ ttyin 0xffffff0002f2c010 getty 878 1 878 0 Ss+ ttyin 0xffffff0002f2bc10 getty 877 1 877 0 Ss+ ttyin 0xffffff0002f1c810 getty 874 872 48 0 S+ wait 0xffffff0003077000 fsck 873 1 48 0 S+ piperd 0xffffff000340fc60 logger 872 1 48 0 S+ wait 0xffffff00030f29b0 sh 834 1 834 0 Ss nanslp 0xffffffff80ade5e8 cron 828 1 828 25 Ss pause 0xffffff00030abaa0 sendmail 824 1 824 0 Ss select 0xffffffff80bc59e0 sendmail 818 1 818 0 Ss select 0xffffffff80bc59e0 sshd 803 802 802 123 S select 0xffffffff80bc59e0 ntpd 802 1 802 0 Ss select 0xffffffff80bc59e0 ntpd 738 1 738 0 Rs CPU 3 rpc.lockd 725 724 724 0 S kqread 0xffffff0003c5a000 rpc.statd 724 1 724 0 Ss select 0xffffffff80bc59e0 rpc.statd 711 703 703 0 S - 0xffffff000328d600 nfsd 710 703 703 0 S - 0xffffff0002fbc400 nfsd 709 703 703 0 S - 0xffffff000328c600 nfsd 708 703 703 0 S - 0xffffff0003149000 nfsd 707 703 703 0 S - 0xffffff0003114600 nfsd 706 703 703 0 S - 0xffffff0003293600 nfsd 705 703 703 0 S - 0xffffff0003112200 nfsd 704 703 703 0 S - 0xffffff0003293200 nfsd 703 1 703 0 Ss select 0xffffffff80bc59e0 nfsd 701 1 701 0 Ss select 0xffffffff80bc59e0 mountd 680 1 680 0 Ss select 0xffffffff80bc59e0 rpcbind 605 1 605 0 Ss select 0xffffffff80bc59e0 syslogd 550 1 550 0 Ss select 0xffffffff80bc59e0 devd 47 0 0 0 SL sdflush 0xffffffff80bd7418 [softdepflush] 46 0 0 0 SL syncer 0xffffffff80ade1c0 [syncer] 45 0 0 0 SL vlruwt 0xffffff0002f734d8 [vnlru] 44 0 0 0 SL psleep 0xffffffff80bc63dc [bufdaemon] 43 0 0 0 SL pgzero 0xffffffff80bd9264 [pagezero] 42 0 0 0 SL psleep 0xffffffff80bd8548 [vmdaemon] 41 0 0 0 SL psleep 0xffffffff80bd850c [pagedaemon] 40 0 0 0 SL waiting_ 0xffffffff80bca268 [sctp_iterator] 39 0 0 0 WL [irq1: atkbd0] 38 0 0 0 WL [swi0: sio] 37 0 0 0 WL [irq257: bge1] 36 0 0 0 RL CPU 2 [irq256: bge0] 35 0 0 0 WL [irq24: arcmsr0] 34 0 0 0 WL [irq16: fxp0] 33 0 0 0 SL usbevt 0xffffff00012d1420 [usb2] 32 0 0 0 SL usbevt 0xffffffff8103a420 [usb1] 31 0 0 0 SL usbtsk 0xffffffff80ad76e8 [usbtask-dr] 30 0 0 0 SL usbtsk 0xffffffff80ad76c0 [usbtask-hc] 29 0 0 0 SL usbevt 0xffffffff81038420 [usb0] 28 0 0 0 WL [irq10: ohci0 ohci+] 27 0 0 0 WL [irq15: ata1] 26 0 0 0 WL [irq14: ata0] 25 0 0 0 WL [irq11: atapci0+] 24 0 0 0 WL [irq9: acpi0] 23 0 0 0 SL - 0xffffff000116fd00 [thread taskq] 22 0 0 0 WL [swi5: +] 9 0 0 0 SL - 0xffffff0001251200 [kqueue taskq] 8 0 0 0 SL - 0xffffff0001251300 [acpi_task_2] 7 0 0 0 SL - 0xffffff0001251300 [acpi_task_1] 6 0 0 0 SL - 0xffffff0001251300 [acpi_task_0] 21 0 0 0 WL [swi2: cambio] 5 0 0 0 SL ccb_scan 0xffffffff80aa5260 [xpt_thrd] 20 0 0 0 WL [swi6: task queue] 19 0 0 0 WL [swi6: Giant taskq] 18 0 0 0 SL - 0xffffffff80ade1c8 [yarrow] 4 0 0 0 SL - 0xffffffff80ad8268 [g_down] 3 0 0 0 SL - 0xffffffff80ad8260 [g_up] 2 0 0 0 SL - 0xffffffff80ad8250 [g_event] 17 0 0 0 WL [swi3: vm] 16 0 0 0 RL CPU 1 [swi4: clock sio] 15 0 0 0 WL [swi1: net] 14 0 0 0 RL CPU 0 [idle: cpu0] 13 0 0 0 RL [idle: cpu1] 12 0 0 0 RL [idle: cpu2] 11 0 0 0 RL [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00010fd9b0 [init] 10 0 0 0 SL audit_wo 0xffffffff80bd67d0 [audit] 0 0 0 0 SLs sched 0xffffffff80ad8360 [swapper] db> db> show pcpu cpuid = 2 curthread = 0xffffff0002e74000: pid 36 "irq256: bge0" curpcb = 0xffffffffae62bd40 fpcurthread = none idlethread = 0xffffff0001100370: pid 12 "idle: cpu2" spin locks held: exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 db> show allpcpu Current CPU: 2 cpuid = 0 curthread = 0xffffff000110c6e0: pid 14 "idle: cpu0" curpcb = 0xffffffffac50bd40 fpcurthread = none idlethread = 0xffffff000110c6e0: pid 14 "idle: cpu0" spin locks held: cpuid = 1 curthread = 0xffffff000110c000: pid 16 "swi4: clock sio" curpcb = 0xffffffffac518d40 fpcurthread = none idlethread = 0xffffff0001100000: pid 13 "idle: cpu1" spin locks held: cpuid = 2 curthread = 0xffffff0002e74000: pid 36 "irq256: bge0" curpcb = 0xffffffffae62bd40 fpcurthread = none idlethread = 0xffffff0001100370: pid 12 "idle: cpu2" spin locks held: exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 cpuid = 3 curthread = 0xffffff00032baa50: pid 738 "rpc.lockd" curpcb = 0xffffffffb0aa3d40 fpcurthread = none idlethread = 0xffffff00011006e0: pid 11 "idle: cpu3" spin locks held: db> db> show locks exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480 exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 db> show alllocks Process 738 (rpc.lockd) thread 0xffffff00032baa50 (100097) exclusive sx so_rcv_sx r = 0 (0xffffff00034f5850) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150 exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff00031d1000) locked @ /usr/src/sys/rpc/svc_vc.c:614 Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035) exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480 exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007) shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155 db> db> show lockedvnods Locked vnodes db> db> alltrace Tracing command top pid 898 tid 100120 td 0xffffff0003a76370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e kern_select() at kern_select+0x7f0 select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800a8eadc, rsp = 0x7fffffffe9c8, rbp = 0x11 --- Tracing command csh pid 894 tid 100116 td 0xffffff0003105a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80093419c, rsp = 0x7fffffffe608, rbp = 0x800c3f300 --- Tracing command sshd pid 891 tid 100115 td 0xffffff0003103000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe4b8, rbp = 0x7fffffffe540 --- Tracing command fsck_ufs pid 890 tid 100067 td 0xffffff000304d6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80080d8ac, rsp = 0x7fffffffdbf8, rbp = 0x1 --- Tracing command getty pid 885 tid 100114 td 0xffffff0003103370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 884 tid 100113 td 0xffffff00031036e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 883 tid 100112 td 0xffffff0003103a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 882 tid 100111 td 0xffffff00030f6000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 881 tid 100104 td 0xffffff00032b9000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 880 tid 100066 td 0xffffff0003055000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 879 tid 100053 td 0xffffff0003055a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 878 tid 100047 td 0xffffff000304da50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 877 tid 100078 td 0xffffff00030796e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command fsck pid 874 tid 100073 td 0xffffff0003054a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80069ce8c, rsp = 0x7fffffffe378, rbp = 0x3 --- Tracing command logger pid 873 tid 100051 td 0xffffff0002fa4a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x2bc dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800717b5c, rsp = 0x7fffffffe908, rbp = 0x7fffffffee18 --- Tracing command sh pid 872 tid 100083 td 0xffffff00031046e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80091be8c, rsp = 0x7fffffffcf98, rbp = 0x30 --- Tracing command cron pid 834 tid 100072 td 0xffffff00030f6370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c --- Tracing command sendmail pid 828 tid 100056 td 0xffffff0002e74a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 --- Tracing command sendmail pid 824 tid 100065 td 0xffffff00030f66e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e kern_select() at kern_select+0x7f0 select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 --- Tracing command sshd pid 818 tid 100105 td 0xffffff0003104000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 --- Tracing command ntpd pid 803 tid 100055 td 0xffffff0002fa4000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e poll() at poll+0x1bf syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 --- Tracing command ntpd pid 802 tid 100099 td 0xffffff00032ba370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f poll() at poll+0x567 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x323 --- Tracing command rpc.lockd pid 738 tid 100097 td 0xffffff00032baa50 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffb0aa3370 --- _rw_wlock_hard() at _rw_wlock_hard+0x167 _rw_wlock() at _rw_wlock+0xc0 tcp_usr_rcvd() at tcp_usr_rcvd+0x3e soreceive_generic() at soreceive_generic+0x498 svc_vc_recv() at svc_vc_recv+0x169 svc_run() at svc_run+0x146 nlm_syscall() at nlm_syscall+0x79c syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 --- Tracing command rpc.statd pid 725 tid 100101 td 0xffffff00032b9a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_kevent() at kern_kevent+0x324 kevent() at kevent+0x90 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 --- Tracing command rpc.statd pid 724 tid 100049 td 0xffffff0003056000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e kern_select() at kern_select+0x7f0 select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 --- Tracing command nfsd pid 711 tid 100093 td 0xffffff00032bca50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 --- Tracing command nfsd pid 710 tid 100092 td 0xffffff00032bd000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 --- Tracing command nfsd pid 709 tid 100050 td 0xffffff0002fa5000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 --- Tracing command nfsd pid 708 tid 100052 td 0xffffff0002fa46e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 --- Tracing command nfsd pid 707 tid 100060 td 0xffffff0002e74370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 --- Tracing command nfsd pid 706 tid 100103 td 0xffffff00032b9370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 --- Tracing command nfsd pid 705 tid 100074 td 0xffffff00030546e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 --- Tracing command nfsd pid 704 tid 100061 td 0xffffff00030f76e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 --- Tracing command nfsd pid 703 tid 100068 td 0xffffff000304d370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 --- Tracing command mountd pid 701 tid 100090 td 0xffffff0003079000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 --- Tracing command rpcbind pid 680 tid 100091 td 0xffffff00032bd370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e poll() at poll+0x1bf syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 --- Tracing command syslogd pid 605 tid 100095 td 0xffffff00032bc370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 --- Tracing command devd pid 550 tid 100064 td 0xffffff00030f6a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 --- Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e softdep_flush() at softdep_flush+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 --- Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sched_sync() at sched_sync+0x61a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 --- Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vnlru_proc() at vnlru_proc+0x696 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 --- Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e buf_daemon() at buf_daemon+0x208 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 --- Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 --- Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 --- Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pageout() at vm_pageout+0xaed fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 --- Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sctp_iterator_thread() at sctp_iterator_thread+0x54 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 --- Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 --- Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 --- Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50 fork_trampoline() at fork_trampoline Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000 kdb_enter_why() at kdb_enter_why+0x3d siointr1() at siointr1+0x238 siointr() at siointr+0x2e intr_execute_handlers() at intr_execute_handlers+0x92 Xapic_isr1() at Xapic_isr1+0x7f --- interrupt, rip = 0xffffffff804b03ed, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x12d _mtx_lock_flags() at _mtx_lock_flags+0x111 svc_vc_soupcall() at svc_vc_soupcall+0x22 sowakeup() at sowakeup+0xf8 tcp_do_segment() at tcp_do_segment+0x1e08 tcp_input() at tcp_input+0xa00 ip_input() at ip_input+0xc0 ether_demux() at ether_demux+0x1ed ether_input() at ether_input+0x19d bge_intr() at bge_intr+0x49f ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 --- Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 --- Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50 fork_trampoline() at fork_trampoline Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 --- Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 --- Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 --- Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 --- Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 --- Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 --- Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 --- Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 --- Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 --- Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 --- Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 --- Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 --- Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 --- Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50 fork_trampoline() at fork_trampoline Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 --- Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 --- Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 --- Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 --- Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 --- _rw_wlock_hard() at _rw_wlock_hard+0x167 _rw_wlock() at _rw_wlock+0xc0 tcp_timer_delack() at tcp_timer_delack+0x58 softclock() at softclock+0x299 ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 --- Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 --- Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffff80be4070, rbp = 0xffffffffac50bba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 --- Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 --- Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 critical_exit() at critical_exit+0x91 intr_execute_handlers() at intr_execute_handlers+0xe9 Xapic_isr1() at Xapic_isr1+0x7f --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 --- Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac4fcb90, rbp = 0xffffffffac4fcba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 --- Tracing command init pid 1 tid 100001 td 0xffffff0001100a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 --- Tracing command audit pid 10 tid 100000 td 0xffffff0001101000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x15e audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 --- Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e scheduler() at scheduler+0x2cd mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> db>reboot cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 Stopped at kdb_enter_why+0x3d: movq $0,0x608808(%rip) db> ps pid ppid pgrp uid state wmesg wchan cmd 895 1 895 0 Ss+ ttyin 0xffffff0002f07c10 getty 894 1 894 0 Ss+ ttyin 0xffffff0002f29810 getty 893 1 893 0 Ss+ ttyin 0xffffff0002f29c10 getty 892 1 892 0 Ss+ ttyin 0xffffff0002f2a010 getty 891 1 891 0 Ss+ ttyin 0xffffff0002f2b410 getty 890 1 890 0 Ss+ ttyin 0xffffff0002f1d810 getty 889 1 889 0 Ss+ ttyin 0xffffff0002f2c010 getty 888 1 888 0 Ss+ ttyin 0xffffff0002f2bc10 getty 887 1 887 0 Ss+ ttyin 0xffffff0002f1c810 getty 844 1 844 0 Ss nanslp 0xffffffff80ade5e8 cron 838 1 838 25 Ss pause 0xffffff0002f725c8 sendmail 832 1 832 0 Ss select 0xffffffff80bc59e0 sendmail 826 1 826 0 Ss select 0xffffffff80bc59e0 sshd 811 810 810 123 S select 0xffffffff80bc59e0 ntpd 810 1 810 0 Ss select 0xffffffff80bc59e0 ntpd 746 1 746 0 Rs CPU 0 rpc.lockd 733 732 732 0 S kqread 0xffffff0003570400 rpc.statd 732 1 732 0 Ss select 0xffffffff80bc59e0 rpc.statd 720 711 711 0 S - 0xffffff000312a000 nfsd 718 711 711 0 S - 0xffffff000339d600 nfsd 717 711 711 0 S - 0xffffff0003301000 nfsd 716 711 711 0 S - 0xffffff0002fbc800 nfsd 715 711 711 0 S - 0xffffff00032ff400 nfsd 714 711 711 0 S - 0xffffff00030e2200 nfsd 713 711 711 0 S - 0xffffff00032ffc00 nfsd 712 711 711 0 S - 0xffffff00030d2c00 nfsd 711 1 711 0 Ss select 0xffffffff80bc59e0 nfsd 709 1 709 0 Ss select 0xffffffff80bc59e0 mountd 688 1 688 0 Ss select 0xffffffff80bc59e0 rpcbind 613 1 613 0 Ss select 0xffffffff80bc59e0 syslogd 558 1 558 0 Ss select 0xffffffff80bc59e0 devd 47 0 0 0 SL sdflush 0xffffffff80bd7418 [softdepflush] 46 0 0 0 SL syncer 0xffffffff80ade1c0 [syncer] 45 0 0 0 SL vlruwt 0xffffff0002f734d8 [vnlru] 44 0 0 0 SL psleep 0xffffffff80bc63dc [bufdaemon] 43 0 0 0 SL pgzero 0xffffffff80bd9264 [pagezero] 42 0 0 0 SL psleep 0xffffffff80bd8548 [vmdaemon] 41 0 0 0 SL psleep 0xffffffff80bd850c [pagedaemon] 40 0 0 0 SL waiting_ 0xffffffff80bca268 [sctp_iterator] 39 0 0 0 WL [irq1: atkbd0] 38 0 0 0 WL [swi0: sio] 37 0 0 0 WL [irq257: bge1] 36 0 0 0 RL CPU 2 [irq256: bge0] 35 0 0 0 WL [irq24: arcmsr0] 34 0 0 0 WL [irq16: fxp0] 33 0 0 0 SL usbevt 0xffffff00012d1420 [usb2] 32 0 0 0 SL usbevt 0xffffffff8103a420 [usb1] 31 0 0 0 SL usbtsk 0xffffffff80ad76e8 [usbtask-dr] 30 0 0 0 SL usbtsk 0xffffffff80ad76c0 [usbtask-hc] 29 0 0 0 SL usbevt 0xffffffff81038420 [usb0] 28 0 0 0 WL [irq10: ohci0 ohci+] 27 0 0 0 WL [irq15: ata1] 26 0 0 0 WL [irq14: ata0] 25 0 0 0 WL [irq11: atapci0+] 24 0 0 0 WL [irq9: acpi0] 23 0 0 0 SL - 0xffffff000116fd00 [thread taskq] 22 0 0 0 WL [swi5: +] 9 0 0 0 SL - 0xffffff0001251200 [kqueue taskq] 8 0 0 0 SL - 0xffffff0001251300 [acpi_task_2] 7 0 0 0 SL - 0xffffff0001251300 [acpi_task_1] 6 0 0 0 SL - 0xffffff0001251300 [acpi_task_0] 21 0 0 0 WL [swi2: cambio] 5 0 0 0 SL ccb_scan 0xffffffff80aa5260 [xpt_thrd] 20 0 0 0 WL [swi6: task queue] 19 0 0 0 WL [swi6: Giant taskq] 18 0 0 0 SL - 0xffffffff80ade1c8 [yarrow] 4 0 0 0 SL - 0xffffffff80ad8268 [g_down] 3 0 0 0 SL - 0xffffffff80ad8260 [g_up] 2 0 0 0 SL - 0xffffffff80ad8250 [g_event] 17 0 0 0 WL [swi3: vm] 16 0 0 0 RL CPU 1 [swi4: clock sio] 15 0 0 0 WL [swi1: net] 14 0 0 0 RL [idle: cpu0] 13 0 0 0 RL [idle: cpu1] 12 0 0 0 RL [idle: cpu2] 11 0 0 0 RL CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00010fd9b0 [init] 10 0 0 0 SL audit_wo 0xffffffff80bd67d0 [audit] 0 0 0 0 SLs sched 0xffffffff80ad8360 [swapper] db> show pcpu cpuid = 2 curthread = 0xffffff0002e74000: pid 36 "irq256: bge0" curpcb = 0xffffffffae62bd40 fpcurthread = none idlethread = 0xffffff0001100370: pid 12 "idle: cpu2" spin locks held: exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 db> show allpcpu Current CPU: 2 cpuid = 0 curthread = 0xffffff0003300000: pid 746 "rpc.lockd" curpcb = 0xffffffffb0ac1d40 fpcurthread = none idlethread = 0xffffff000110c6e0: pid 14 "idle: cpu0" spin locks held: cpuid = 1 curthread = 0xffffff000110c000: pid 16 "swi4: clock sio" curpcb = 0xffffffffac518d40 fpcurthread = none idlethread = 0xffffff0001100000: pid 13 "idle: cpu1" spin locks held: cpuid = 2 curthread = 0xffffff0002e74000: pid 36 "irq256: bge0" curpcb = 0xffffffffae62bd40 fpcurthread = none idlethread = 0xffffff0001100370: pid 12 "idle: cpu2" spin locks held: exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 cpuid = 3 curthread = 0xffffff00011006e0: pid 11 "idle: cpu3" curpcb = 0xffffffffac4fcd40 fpcurthread = none idlethread = 0xffffff00011006e0: pid 11 "idle: cpu3" spin locks held: db> show locks exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480 exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 db> show alllocks Process 746 (rpc.lockd) thread 0xffffff0003300000 (100103) exclusive sx so_rcv_sx r = 0 (0xffffff000337c130) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150 exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff0003bc6400) locked @ /usr/src/sys/rpc/svc_vc.c:614 Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035) exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480 exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391 Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007) shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155 db> show lockedvnods Locked vnodes db> alltrace Tracing command getty pid 895 tid 100139 td 0xffffff00034ce6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 894 tid 100138 td 0xffffff00034cea50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 893 tid 100136 td 0xffffff0003536370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 892 tid 100135 td 0xffffff00031ba6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 891 tid 100132 td 0xffffff000342f370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 890 tid 100057 td 0xffffff0003020a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 889 tid 100130 td 0xffffff000342fa50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 888 tid 100120 td 0xffffff000343e370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command getty pid 887 tid 100127 td 0xffffff000344b6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 giant_read() at giant_read+0x65 devfs_read_f() at devfs_read_f+0x81 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 --- Tracing command cron pid 844 tid 100093 td 0xffffff00031e36e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c --- Tracing command sendmail pid 838 tid 100049 td 0xffffff0002fa5370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 --- Tracing command sendmail pid 832 tid 100131 td 0xffffff000342f6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e kern_select() at kern_select+0x7f0 select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 --- Tracing command sshd pid 826 tid 100053 td 0xffffff00030236e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 --- Tracing command ntpd pid 811 tid 100102 td 0xffffff0003300370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e poll() at poll+0x1bf syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 --- Tracing command ntpd pid 810 tid 100126 td 0xffffff000344ba50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f poll() at poll+0x567 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x32b --- Tracing command rpc.lockd pid 746 tid 100103 td 0xffffff0003300000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffff80be4070, rbp = 0xffffffffb0ac1370 --- _rw_wlock_hard() at _rw_wlock_hard+0x167 _rw_wlock() at _rw_wlock+0xc0 tcp_usr_rcvd() at tcp_usr_rcvd+0x3e soreceive_generic() at soreceive_generic+0x498 svc_vc_recv() at svc_vc_recv+0x169 svc_run() at svc_run+0x146 nlm_syscall() at nlm_syscall+0x79c syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 --- Tracing command rpc.statd pid 733 tid 100122 td 0xffffff000343da50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_kevent() at kern_kevent+0x324 kevent() at kevent+0x90 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 --- Tracing command rpc.statd pid 732 tid 100100 td 0xffffff00030c7a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e kern_select() at kern_select+0x7f0 select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 --- Tracing command nfsd pid 720 tid 100115 td 0xffffff00030be6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 --- Tracing command nfsd pid 718 tid 100114 td 0xffffff00030bea50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 --- Tracing command nfsd pid 717 tid 100113 td 0xffffff00030c7000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 --- Tracing command nfsd pid 716 tid 100112 td 0xffffff00030c7370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 --- Tracing command nfsd pid 715 tid 100111 td 0xffffff00030c76e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 --- Tracing command nfsd pid 714 tid 100058 td 0xffffff00030206e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 --- Tracing command nfsd pid 713 tid 100099 td 0xffffff00031e3000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 --- Tracing command nfsd pid 712 tid 100054 td 0xffffff0002fa5000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 nfssvc() at nfssvc+0x2de syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 --- Tracing command nfsd pid 711 tid 100048 td 0xffffff00030246e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 --- Tracing command mountd pid 709 tid 100083 td 0xffffff0002e74a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 --- Tracing command rpcbind pid 688 tid 100098 td 0xffffff00031e3370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e poll() at poll+0x1bf syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 --- Tracing command syslogd pid 613 tid 100087 td 0xffffff00031ba370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 --- Tracing command devd pid 558 tid 100082 td 0xffffff0002fa4000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x81c select() at select+0x56 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 --- Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e softdep_flush() at softdep_flush+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 --- Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sched_sync() at sched_sync+0x61a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 --- Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vnlru_proc() at vnlru_proc+0x696 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 --- Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e buf_daemon() at buf_daemon+0x208 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 --- Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 --- Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 --- Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pageout() at vm_pageout+0xaed fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 --- Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sctp_iterator_thread() at sctp_iterator_thread+0x54 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 --- Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 --- Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 --- Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50 fork_trampoline() at fork_trampoline Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000 kdb_enter_why() at kdb_enter_why+0x3d siointr1() at siointr1+0x238 siointr() at siointr+0x2e intr_execute_handlers() at intr_execute_handlers+0x92 Xapic_isr1() at Xapic_isr1+0x7f --- interrupt, rip = 0xffffffff804b03e9, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x129 _mtx_lock_flags() at _mtx_lock_flags+0x111 svc_vc_soupcall() at svc_vc_soupcall+0x22 sowakeup() at sowakeup+0xf8 tcp_do_segment() at tcp_do_segment+0x1e08 tcp_input() at tcp_input+0xa00 ip_input() at ip_input+0xc0 ether_demux() at ether_demux+0x1ed ether_input() at ether_input+0x19d bge_intr() at bge_intr+0x49f ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 --- Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 --- Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50 fork_trampoline() at fork_trampoline Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 --- Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 --- Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 --- Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 --- Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 --- Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 --- Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 --- Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 --- Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 --- Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 --- Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 --- Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x7e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 --- Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 --- Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50 fork_trampoline() at fork_trampoline Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 --- Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 --- Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 --- Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 --- Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804bbe9b, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 --- _rw_wlock_hard() at _rw_wlock_hard+0x16b _rw_wlock() at _rw_wlock+0xc0 tcp_timer_delack() at tcp_timer_delack+0x58 softclock() at softclock+0x299 ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 --- Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x3c5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 --- Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac50bb90, rbp = 0xffffffffac50bba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 --- Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 --- Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 --- Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffac4fcba0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x57 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 --- Tracing command init pid 1 tid 100001 td 0xffffff0001100a50 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1f9 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 --- Tracing command audit pid 10 tid 100000 td 0xffffff0001101000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x15e audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 --- Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xe4 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e scheduler() at scheduler+0x2cd mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> >Fix: >Release-Note: >Audit-Trail: >Unformatted: