From owner-freebsd-bugs@FreeBSD.ORG Thu Oct 4 19:20:02 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D93716A421 for ; Thu, 4 Oct 2007 19:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 731BF13C461 for ; Thu, 4 Oct 2007 19:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l94JK2LX075808 for ; Thu, 4 Oct 2007 19:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l94JK2OJ075807; Thu, 4 Oct 2007 19:20:02 GMT (envelope-from gnats) Resent-Date: Thu, 4 Oct 2007 19:20:02 GMT Resent-Message-Id: <200710041920.l94JK2OJ075807@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Joao Pedras Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6687D16A46C for ; Thu, 4 Oct 2007 19:17:23 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 57AD113C4B7 for ; Thu, 4 Oct 2007 19:17:23 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l94JHNET087523 for ; Thu, 4 Oct 2007 19:17:23 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.1/8.14.1/Submit) id l94JHNd6087514; Thu, 4 Oct 2007 19:17:23 GMT (envelope-from nobody) Message-Id: <200710041917.l94JHNd6087514@www.freebsd.org> Date: Thu, 4 Oct 2007 19:17:23 GMT From: Joao Pedras To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: misc/116932: System locking up while carrying out remote ssh session on -current X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2007 19:20:02 -0000 >Number: 116932 >Category: misc >Synopsis: System locking up while carrying out remote ssh session on -current >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Oct 04 19:20:01 GMT 2007 >Closed-Date: >Last-Modified: >Originator: Joao Pedras >Release: 7-CURRENT >Organization: >Environment: NOTE: Tried with -CURRENT from October 4th and problem still persists. Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT #2: Fri Sep 28 11:45:50 PDT 2007 root@daemons2.somedomain.com:/usr/obj/usr/src/sys/GENERIC WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Dual-Core AMD Opteron(tm) Processor 2210 (1800.09-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x40f12 Stepping = 2 Features=0x178bfbff Features2=0x2001 AMD Features=0xea500800 AMD Features2=0x1f Cores per package: 2 usable memory = 4281348096 (4083 MB) avail memory = 4123906048 (3932 MB) ACPI APIC Table: <082907 APIC1947> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: <082907 XSDT1947> on motherboard acpi0: [ITHREAD] acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 25000000 Hz quality 900 acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1400000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0 cpu0: on acpi0 powernow0: on cpu0 cpu1: on acpi0 powernow1: on cpu1 cpu2: on acpi0 powernow2: on cpu2 cpu3: on acpi0 powernow3: on cpu3 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: port 0x2f00-0x2f7f at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xfdefb000-0xfdefbfff irq 21 at device 2.0 on pci0 ohci0: [GIANT-LOCKED] ohci0: [ITHREAD] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 10 ports with 10 removable, self powered ehci0: mem 0xfdefac00-0xfdefacff irq 22 at device 2.1 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb1: EHCI version 1.0 usb1: companion controller, 10 ports each: usb0 usb1: on ehci0 usb1: USB revision 2.0 uhub1: on usb1 uhub1: 10 ports with 10 removable, self powered atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 4.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xc480-0xc487,0xc400-0xc403,0xc080-0xc087,0xc000-0xc003,0xbc00-0xbc0f mem 0xfdef9000-0xfdef9fff irq 23 at device 5.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0xb880-0xb887,0xb800-0xb803,0xb480-0xb487,0xb400-0xb403,0xb080-0xb08f mem 0xfdef8000-0xfdef8fff irq 20 at device 5.1 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] atapci3: port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f mem 0xfdef7000-0xfdef7fff irq 21 at device 5.2 on pci0 atapci3: [ITHREAD] ata6: on atapci3 ata6: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] pcib1: at device 6.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd0ff mem 0xf0000000-0xf7ffffff,0xfdff0000-0xfdffffff irq 16 at device 7.0 on pci1 nfe0: port 0xa400-0xa407 mem 0xfdef6000-0xfdef6fff,0xfdefa800-0xfdefa8ff,0xfdefa400-0xfdefa40f irq 22 at device 8.0 on pci0 miibus0: on nfe0 e1000phy0: PHY 16 on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto nfe0: Ethernet address: 00:e0:81:48:f9:64 nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe1: port 0xa080-0xa087 mem 0xfdef5000-0xfdef5fff,0xfdefa000-0xfdefa0ff,0xfdef4c00-0xfdef4c0f irq 23 at device 9.0 on pci0 miibus1: on nfe1 e1000phy1: PHY 17 on miibus1 e1000phy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto nfe1: Ethernet address: 00:e0:81:48:f9:65 nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] pcib2: at device 10.0 on pci0 pci2: on pcib2 mpt0: port 0xe000-0xe0ff mem 0xfebfc000-0xfebfffff,0xfebe0000-0xfebeffff irq 17 at device 0.0 on pci2 mpt0: [ITHREAD] mpt0: MPI Version=1.5.13.0 mpt0: mpt_cam_event: 0x16 mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required). mpt0: mpt_cam_event: 0x16 mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required). pcib3: at device 13.0 on pci0 pci3: on pcib3 pcib4: at device 0.0 on pci3 pci4: on pcib4 pcib5: at device 0.1 on pci3 pci5: on pcib5 pcib6: at device 14.0 on pci0 pci6: on pcib6 pcib7: at device 15.0 on pci0 pci7: on pcib7 acpi_button0: on acpi0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 drq 0 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ipmi0: on isa0 ipmi0: KCS mode found at io 0xca8 alignment 0x4 on isa orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xca7ff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: DVDROM at ata0-master UDMA100 ad4: 238475MB at ata2-master SATA300 ipmi0: IPMI device rev. 0, firmware rev. 0.2, version 2.0 ipmi0: Number of channels 4 ipmi0: Attached watchdog SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! WARNING: WITNESS option enabled, expect reduced performance. GEOM_LABEL: Label for provider acd0 is iso9660/OpenBSD4.1 CD 3 www.OpenBSD.org. Trying to mount root from ufs:/dev/ad4s1a WARNING: /var was not properly dismounted nfe0: watchdog timeout nfe0: watchdog timeout nfe0: link state changed to DOWN nfe0: link state changed to UP >Description: My apologies for the simplistic way of reproducing the issue but it is indeed the only way I know how to cause it. Please feel free to suggest/request other scenarios so that this issue can be resolved. While watching a 'make buildworld' over an ssh session, established over an ipsec link between two remote locations, the system will lock up. The same task with a session established over the local network does not cause the lock up. I don't discard the possibility that the system is simply too slow but there are no replies to pings, console interaction or anything else. It is possible though to break into the debugger with CTRL+ALT+ESC. The system where this occurs is on a RFC 1918 network and is accessible over the vpn link. The system is *not* the router, just a plain client. Software: The OS is plain -CURRENT, with no ports added, one of the nfe interfaces obtains config through DHCP and the other is not being used. I basically installed the OS to test the hardware from a snapshot CD, sup'ed it and started noticing this issue from the start. I have tried with GENERIC 4BSD and also with ULE. Before the lockup I see a few 'kernel: rtfree: 0xPPPPPPP has 1 refs' messages, I also have tried without IPv6. Judging by some recent post on freebsd-current these warnings seem to be associated with IPv6. The problem occurs with all of the above variations (4BSD, ULE, no IPv6). The rest of kernel config is GENERIC. Hardware: This problem happens consistently on a couple CURRENTs running on similar hardware (Tyan mobo S2932). One of them boots off the one SATA hd is has, which is connected to the onboard controller. The other one boots off a RAID1 array from the onboard LSI SAS controller. This second system also has a RAID5 array in a additional LSI 300-8XLP. Both machines have the additional IPMI card ( M3291) installed. The problem still occurs *without* the ipmi cards. Tried also with an old faithful fxp() card and no luck. from debugger after the lock up: [root@daemons2 ~]# [root@daemons2 ~]# [root@daemons2 ~]# rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs rtfree: 0xffffff00044771e0 has 1 refs KDB: enter: manual escape to debugger <<<< LOCKED UP BEFORE THIS; THIS CTRL+ALT+ESC [thread pid 19 tid 100010 ] Stopped at kdb_enter+0x31: leave db> ps pid ppid pgrp uid state wmesg wchan cmd 21498 21496 778 0 S+ piperd 0xffffff000c3c82e8 sed 21497 21496 778 0 S+ vmo_de 0xffffffff80a1de48 sh 21496 21492 778 0 S+ wait 0xffffff0008f97000 sh 21492 21139 778 0 S+ piperd 0xffffff00044ca2e8 make 21139 21132 778 0 S+ wait 0xffffff0004375000 sh 21132 21131 778 0 S+ select 0xffffffff80aa6db0 make 21131 4753 778 0 S+ wait 0xffffff0004098460 sh 4753 4751 778 0 S+ pipewr 0xffffff0004369ba0 make 4751 4660 778 0 S+ wait 0xffffff0004518000 sh 4660 778 778 0 S+ ttywri 0xffffff0004058450 make 778 775 778 0 S+ wait 0xffffff00041e1460 sh 775 772 775 0 Ss+ wait 0xffffff0004432000 bash 772 705 772 0 Ls *rtentry 0xffffff0003be5640 sshd 771 765 771 0 S+ ttyin 0xffffff0003bc6c10 bash 766 1 766 0 Ss+ ttyin 0xffffff0003bce810 getty 765 1 765 0 Ss+ wait 0xffffff00042258c0 login 764 1 764 0 Ss+ ttyin 0xffffff0003bf3c10 getty 763 1 763 0 Ss+ ttyin 0xffffff0003bf4010 getty 721 1 721 0 Ss nanslp 0xffffffff80a1e1c8 cron --More-- 715 1 715 25 Ss pause 0xffffff00044310c0 sendmail 711 1 711 0 Ss select 0xffffffff80aa6db0 sendmail 705 1 705 0 Ss select 0xffffffff80aa6db0 sshd 575 1 575 0 Ss select 0xffffffff80aa6db0 syslogd 529 1 529 0 Ss select 0xffffffff80aa6db0 devd 324 1 324 65 Ss select 0xffffffff80aa6db0 dhclient 296 1 52 0 S+ select 0xffffffff80aa6db0 dhclient 162 1 162 0 Ss pause 0xffffff00041b70c0 adjkerntz 51 0 0 0 SL - 0xffffffff80a1de48 [schedcpu] 50 0 0 0 SL sdflush 0xffffffff80ab6dc8 [softdepflush] 49 0 0 0 SL vlruwt 0xffffff0003be1000 [vnlru] 48 0 0 0 SL syncer 0xffffffff80a1de40 [syncer] 47 0 0 0 SL psleep 0xffffffff80aa763c [bufdaemon] 46 0 0 0 SL pgzero 0xffffffff80ab8844 [pagezero] 45 0 0 0 SL psleep 0xffffffff80ab7b88 [vmdaemon] 44 0 0 0 SL psleep 0xffffffff80ab7b4c [pagedaemon] 43 0 0 0 SL ipmireq 0xffffff0003be8b48 [ipmi0: kcs] 42 0 0 0 SL waiting_ 0xffffffff80aaad48 [sctp_iterator] 41 0 0 0 WL [irq1: atkbd0] 40 0 0 0 SL - 0xffffff0003bde448 [fdc0] --More-- 39 0 0 0 WL [swi0: sio] 38 0 0 0 SL idle 0xffffffff80e50000 [mpt_recovery0] 37 0 0 0 WL [irq17: mpt0] 36 0 0 0 SL - 0xffffff0003ba6680 [nfe1 taskq] 35 0 0 0 SL - 0xffffff0003a00b80 [nfe0 taskq] 34 0 0 0 WL [irq20: atapci2] 33 0 0 0 WL [irq23: atapci1] 32 0 0 0 WL [irq15: ata1] 31 0 0 0 WL [irq14: ata0] 30 0 0 0 SL usbevt 0xffffff0001244420 [usb1] 29 0 0 0 WL [irq22: ehci0] 28 0 0 0 SL usbtsk 0xffffffff80a19808 [usbtask-dr] 27 0 0 0 SL usbtsk 0xffffffff80a197e0 [usbtask-hc] 26 0 0 0 SL usbevt 0xffffffff80e44420 [usb0] 25 0 0 0 WL [irq21: ohci0+] 24 0 0 0 WL [irq9: acpi0] 23 0 0 0 SL - 0xffffff0001203880 [thread taskq] 22 0 0 0 LL *rtentry 0xffffff000104e640 [swi5: +] 21 0 0 0 WL [swi2: cambio] 9 0 0 0 SL ccb_scan 0xffffffff809e8540 [xpt_thrd] --More-- 8 0 0 0 SL - 0xffffff0001203c00 [acpi_task_2] 7 0 0 0 SL - 0xffffff0001203c00 [acpi_task_1] 6 0 0 0 SL - 0xffffff0001203c00 [acpi_task_0] 5 0 0 0 SL - 0xffffff0001235200 [kqueue taskq] 20 0 0 0 WL [swi6: task queue] 19 0 0 0 RL CPU 0 [swi6: Giant taskq] 18 0 0 0 SL - 0xffffffff80a1de48 [yarrow] 4 0 0 0 SL - 0xffffffff80a1a7b8 [g_down] 3 0 0 0 SL - 0xffffffff80a1a7b0 [g_up] 2 0 0 0 SL - 0xffffffff80a1a7a0 [g_event] 17 0 0 0 WL [swi3: vm] 16 0 0 0 LL *inp 0xffffff00085726e0 [swi4: clock sio] 15 0 0 0 WL [swi1: net] 14 0 0 0 RL [idle: cpu0] 13 0 0 0 RL CPU 1 [idle: cpu1] 12 0 0 0 RL CPU 2 [idle: cpu2] 11 0 0 0 RL CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00010e48c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80ab62a0 [audit] 0 0 0 0 WLs [swapper] --More-- b> trace Tracing pid 19 tid 100010 td 0xffffff00010e8340 kdb_enter() at kdb_enter+0x31 scgetc() at scgetc+0x461 sckbdevent() at sckbdevent+0xa4 kbdmux_intr() at kbdmux_intr+0x43 kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20 taskqueue_run() at taskqueue_run+0x94 ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac266d30, rbp = 0 --- db> alltrace Tracing command sed pid 21498 tid 100148 td 0xffffff000423e9c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 pipe_read() at pipe_read+0x2bc dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x80071743c, rsp = 0x7fffffffe808, rbp = 0x7fffffffea40 --- Tracing command sh pid 21497 tid 100064 td 0xffffff000417a000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 --More-- leepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 vm_object_deallocate() at vm_object_deallocate+0x4d1 vm_map_delete() at vm_map_delete+0x19d vm_map_remove() at vm_map_remove+0x59 exec_new_vmspace() at exec_new_vmspace+0x20b exec_elf64_imgact() at exec_elf64_imgact+0x1bd kern_execve() at kern_execve+0x31a execve() at execve+0x3d syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (59, FreeBSD ELF64, execve), rip = 0x80092aa6c, rsp = 0x7fffffffe5a8, rbp = 0x800b09588 --- Tracing command sh pid 21496 tid 100180 td 0xffffff0008fbc340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 --More-- ern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe7c8, rbp = 0x30a --- Tracing command make pid 21492 tid 100181 td 0xffffff0008fbc000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 pipe_read() at pipe_read+0x2bc dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x43bd9c, rsp = 0x7fffffffd378, rbp = 0x7fffffffd390 --- --More-- Tracing command sh pid 21139 tid 100120 td 0xffffff0004374000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe5b8, rbp = 0x30a --- Tracing command make pid 21132 tid 100170 td 0xffffff0008ee39c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 --More-- cv_timedwait_sig() at _cv_timedwait_sig+0x16a kern_select() at kern_select+0x7df select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x43bd1c, rsp = 0x7fffffffdc68, rbp = 0x80066a580 --- Tracing command sh pid 21131 tid 100052 td 0xffffff0003be4000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe808, rbp = 0x30a --- --More-- racing command make pid 4753 tid 100066 td 0xffffff0004179680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 pipe_write() at pipe_write+0x476 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x4c write() at write+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (4, FreeBSD ELF64, write), rip = 0x43bd7c, rsp = 0x7fffffffdc88, rbp = 0x209 --- Tracing command sh pid 4751 tid 100129 td 0xffffff00044339c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab --More-- leepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe7f8, rbp = 0x30a --- Tracing command make pid 4660 tid 100127 td 0xffffff0004435340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x20a ttysleep() at ttysleep+0x25 ttwrite() at ttwrite+0x2fd giant_write() at giant_write+0x51 devfs_write_f() at devfs_write_f+0x9c dofilewrite() at dofilewrite+0x85 --More-- ern_writev() at kern_writev+0x4c write() at write+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (4, FreeBSD ELF64, write), rip = 0x43bd7c, rsp = 0x7fffffffd428, rbp = 0x60 --- Tracing command sh pid 778 tid 100094 td 0xffffff0004212680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe8d8, rbp = 0x30a --- Tracing command bash pid 775 tid 100117 td 0xffffff00041dc9c0 --More-- ched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b7aaac, rsp = 0x7fffffffe9c8, rbp = 0x800e6c140 --- Tracing command sshd pid 772 tid 100054 td 0xffffff00033c7680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 turnstile_wait() at turnstile_wait+0x24d _mtx_lock_sleep() at _mtx_lock_sleep+0xc2 _mtx_lock_flags() at _mtx_lock_flags+0xe1 rt_check() at rt_check+0x10e arpresolve() at arpresolve+0x64 --More-- ther_output() at ether_output+0x308 ip_output() at ip_output+0xabc tcp_output() at tcp_output+0xad6 tcp_usr_send() at tcp_usr_send+0x2e9 sosend_generic() at sosend_generic+0x347 soo_write() at soo_write+0x38 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x4c write() at write+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (4, FreeBSD ELF64, write), rip = 0x8016f541c, rsp = 0x7fffffffe4f8, rbp = 0x5337c0 --- Tracing command bash pid 771 tid 100062 td 0xffffff000417a680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 --More-- tysleep() at ttysleep+0x25 ttread() at ttread+0x306 giant_read() at giant_read+0x51 devfs_read_f() at devfs_read_f+0x72 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800bf243c, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b7 --- Tracing command getty pid 766 tid 100065 td 0xffffff00041799c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 ttysleep() at ttysleep+0x25 ttread() at ttread+0x306 --More-- iant_read() at giant_read+0x51 devfs_read_f() at devfs_read_f+0x72 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 --- Tracing command login pid 765 tid 100107 td 0xffffff0004295340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --More-- -- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009c8aac, rsp = 0x7fffffffed08, rbp = 0 --- Tracing command getty pid 764 tid 100095 td 0xffffff0004212340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 ttysleep() at ttysleep+0x25 ttread() at ttread+0x306 giant_read() at giant_read+0x51 devfs_read_f() at devfs_read_f+0x72 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 --- --More-- racing command getty pid 763 tid 100118 td 0xffffff00041dc680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 ttysleep() at ttysleep+0x25 ttread() at ttread+0x306 giant_read() at giant_read+0x51 devfs_read_f() at devfs_read_f+0x72 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x4c read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 --- Tracing command cron pid 721 tid 100115 td 0xffffff0004262340 sched_switch() at sched_switch+0x176 --More-- i_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x20a kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80091303c, rsp = 0x7fffffffec28, rbp = 0x3c --- Tracing command sendmail pid 715 tid 100114 td 0xffffff0004262680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 --More-- yscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d0f97c, rsp = 0x7fffffffda48, rbp = 0x1 --- Tracing command sendmail pid 711 tid 100055 td 0xffffff00033c7340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16a kern_select() at kern_select+0x7df select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800d9c3bc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7f0 --- Tracing command sshd pid 705 tid 100100 td 0xffffff00041e2000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 --More-- leepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x80b select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x8016f53bc, rsp = 0x7fffffffe618, rbp = 0x2 --- Tracing command syslogd pid 575 tid 100060 td 0xffffff000417b000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x80b select() at select+0x56 syscall() at syscall+0x1ce --More-- fast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x8008263bc, rsp = 0x7fffffffdde8, rbp = 0x8 --- Tracing command devd pid 529 tid 100076 td 0xffffff00041b5680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f kern_select() at kern_select+0x80b select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x4388dc, rsp = 0x7fffffffe928, rbp = 0x7fffffffed50 --- Tracing command dhclient pid 324 tid 100061 td 0xffffff000417a9c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 --More-- leepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16a poll() at poll+0x1e3 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d120c, rsp = 0x7fffffffed98, rbp = 0x144b118 --- Tracing command dhclient pid 296 tid 100069 td 0xffffff00041759c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f poll() at poll+0x567 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d120c, rsp = 0x7fffffffed98, rbp = 0x5 --- --More-- racing command adjkerntz pid 162 tid 100075 td 0xffffff00041b59c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80068497c, rsp = 0x7fffffffed08, rbp = 0x7fffffffed80 --- Tracing command schedcpu pid 51 tid 100058 td 0xffffff00040a0340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 schedcpu_thread() at schedcpu_thread+0x21e --More-- ork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef6ed30, rbp = 0 --- Tracing command softdepflush pid 50 tid 100059 td 0xffffff00040a0000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 softdep_flush() at softdep_flush+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef69d30, rbp = 0 --- Tracing command vnlru pid 49 tid 100039 td 0xffffff00033c6000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c --More-- sleep() at _sleep+0x314 vnlru_proc() at vnlru_proc+0x696 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae340d30, rbp = 0 --- Tracing command syncer pid 48 tid 100040 td 0xffffff00033c29c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 sched_sync() at sched_sync+0x53d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae33bd30, rbp = 0 --- Tracing command bufdaemon pid 47 tid 100041 td 0xffffff00033c2680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 --More-- leepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 buf_daemon() at buf_daemon+0x208 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae336d30, rbp = 0 --- Tracing command pagezero pid 46 tid 100042 td 0xffffff00033c2340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae331d30, rbp = 0 --- Tracing command vmdaemon pid 45 tid 100043 td 0xffffff00033c2000 --More-- ched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae32cd30, rbp = 0 --- Tracing command pagedaemon pid 44 tid 100044 td 0xffffff00012529c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 vm_pageout() at vm_pageout+0xacd fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2fdd30, rbp = 0 --- --More-- Tracing command ipmi0: kcs pid 43 tid 100045 td 0xffffff0003be69c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x14e ipmi_dequeue_request() at ipmi_dequeue_request+0x67 kcs_loop() at kcs_loop+0x3d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef19d30, rbp = 0 --- Tracing command sctp_iterator pid 42 tid 100046 td 0xffffff0003be6680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 sctp_iterator_thread() at sctp_iterator_thread+0x47 --More-- ork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef3cd30, rbp = 0 --- Tracing command irq1: atkbd0 pid 41 tid 100047 td 0xffffff0003be6340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef0fd30, rbp = 0 --- Tracing command fdc0 pid 40 tid 100048 td 0xffffff0003be6000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 fdc_thread() at fdc_thread+0x91b fork_exit() at fork_exit+0x12a --More-- ork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffaef0ad30, rbp = 0 --- Tracing command swi0: sio pid 39 tid 100030 td 0xffffff0001251680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2e4d30, rbp = 0 --- Tracing command mpt_recovery0 pid 38 tid 100031 td 0xffffff0001251340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 mpt_recovery_thread() at mpt_recovery_thread+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --More-- -- trap 0, rip = 0, rsp = 0xffffffffac2dfd30, rbp = 0 --- Tracing command irq17: mpt0 pid 37 tid 100032 td 0xffffff0001251000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2dad30, rbp = 0 --- Tracing command nfe1 taskq pid 36 tid 100033 td 0xffffff00012399c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 msleep_spin() at msleep_spin+0x1fb taskqueue_thread_loop() at taskqueue_thread_loop+0x45 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2d5d30, rbp = 0 --- --More-- Tracing command nfe0 taskq pid 35 tid 100034 td 0xffffff0001239680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 msleep_spin() at msleep_spin+0x1fb taskqueue_thread_loop() at taskqueue_thread_loop+0x45 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2d0d30, rbp = 0 --- Tracing command irq20: atapci2 pid 34 tid 100035 td 0xffffff00033c7000 fork_trampoline() at fork_trampoline Tracing command irq23: atapci1 pid 33 tid 100036 td 0xffffff00033c69c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a --More-- ork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae34fd30, rbp = 0 --- Tracing command irq15: ata1 pid 32 tid 100037 td 0xffffff00033c6680 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 31 tid 100038 td 0xffffff00033c6340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae345d30, rbp = 0 --- Tracing command usb1 pid 30 tid 100022 td 0xffffff0001238340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 --More-- sb_event_thread() at usb_event_thread+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2b7d30, rbp = 0 --- Tracing command irq22: ehci0 pid 29 tid 100023 td 0xffffff0001238000 fork_trampoline() at fork_trampoline Tracing command usbtask-dr pid 28 tid 100024 td 0xffffff00011119c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2add30, rbp = 0 --- Tracing command usbtask-hc pid 27 tid 100025 td 0xffffff0001111680 --More-- ched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2a8d30, rbp = 0 --- Tracing command usb0 pid 26 tid 100026 td 0xffffff0001252680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 usb_event_thread() at usb_event_thread+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2f8d30, rbp = 0 --- --More-- Tracing command irq21: ohci0+ pid 25 tid 100027 td 0xffffff0001252340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2f3d30, rbp = 0 --- Tracing command irq9: acpi0 pid 24 tid 100028 td 0xffffff0001252000 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 23 tid 100029 td 0xffffff00012519c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a --More-- ork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2e9d30, rbp = 0 --- Tracing command swi5: + pid 22 tid 100015 td 0xffffff0001110340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 turnstile_wait() at turnstile_wait+0x24d _mtx_lock_sleep() at _mtx_lock_sleep+0xc2 _mtx_lock_flags() at _mtx_lock_flags+0xe1 rt_setgate() at rt_setgate+0x26e rtredirect() at rtredirect+0x250 icmp_input() at icmp_input+0x537 ip_input() at ip_input+0xc0 ether_demux() at ether_demux+0x1d9 ether_input() at ether_input+0x1d6 nfe_int_task() at nfe_int_task+0x949 taskqueue_run() at taskqueue_run+0x94 ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --More-- -- trap 0, rip = 0, rsp = 0xffffffffac28fd30, rbp = 0 --- Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff0001110000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac28ad30, rbp = 0 --- Tracing command xpt_thrd pid 9 tid 100017 td 0xffffff00010eb9c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac284d30, rbp = 0 --- --More-- Tracing command acpi_task_2 pid 8 tid 100018 td 0xffffff0001239340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2cbd30, rbp = 0 --- Tracing command acpi_task_1 pid 7 tid 100019 td 0xffffff0001239000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a --More-- ork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2c6d30, rbp = 0 --- Tracing command acpi_task_0 pid 6 tid 100020 td 0xffffff00012389c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2c1d30, rbp = 0 --- Tracing command kqueue taskq pid 5 tid 100021 td 0xffffff0001238680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x327 --More-- askqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2bcd30, rbp = 0 --- Tracing command swi6: task queue pid 20 tid 100009 td 0xffffff00010e8680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac26bd30, rbp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100010 td 0xffffff00010e8340 kdb_enter() at kdb_enter+0x31 scgetc() at scgetc+0x461 sckbdevent() at sckbdevent+0xa4 kbdmux_intr() at kbdmux_intr+0x43 kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20 taskqueue_run() at taskqueue_run+0x94 --More-- thread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac266d30, rbp = 0 --- Tracing command yarrow pid 18 tid 100011 td 0xffffff0001111340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 random_kthread() at random_kthread+0x191 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac2a3d30, rbp = 0 --- Tracing command g_down pid 4 tid 100012 td 0xffffff0001111000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 --More-- leepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 g_io_schedule_down() at g_io_schedule_down+0x20f g_down_procbody() at g_down_procbody+0x78 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac29ed30, rbp = 0 --- Tracing command g_up pid 3 tid 100013 td 0xffffff00011109c0 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x78 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac299d30, rbp = 0 --- --More-- racing command g_event pid 2 tid 100014 td 0xffffff0001110680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x314 g_event_procbody() at g_event_procbody+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac294d30, rbp = 0 --- Tracing command swi3: vm pid 17 tid 100004 td 0xffffff00010e5000 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 16 tid 100005 td 0xffffff00010eb680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 turnstile_wait() at turnstile_wait+0x24d _mtx_lock_sleep() at _mtx_lock_sleep+0xc2 _mtx_lock_flags() at _mtx_lock_flags+0xe1 --More-- cp_timer_rexmt() at tcp_timer_rexmt+0x5c softclock() at softclock+0x28d ithread_loop() at ithread_loop+0xe0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac27fd30, rbp = 0 --- Tracing command swi1: net pid 15 tid 100006 td 0xffffff00010eb340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 ithread_loop() at ithread_loop+0x39a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac27ad30, rbp = 0 --- Tracing command idle: cpu0 pid 14 tid 100007 td 0xffffff00010eb000 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 maybe_preempt() at maybe_preempt+0xf3 sched_add() at sched_add+0x1ce --More-- ntr_event_schedule_thread() at intr_event_schedule_thread+0xb5 intr_execute_handlers() at intr_execute_handlers+0xe1 Xapic_isr2() at Xapic_isr2+0x7f --- interrupt, rip = 0xffffffff806cab26, rsp = 0xffffffffac275bc0, rbp = 0xffffffffac275bd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x35 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac275d30, rbp = 0 --- Tracing command idle: cpu1 pid 13 tid 100008 td 0xffffff00010e89c0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab790ff0, rbp = 0xffffffffac270bd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x35 --More-- ork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac270d30, rbp = 0 --- Tracing command idle: cpu2 pid 12 tid 100000 td 0xffffff00010e8000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab795ff0, rbp = 0xffffffffac261bd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x35 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac261d30, rbp = 0 --- Tracing command idle: cpu3 pid 11 tid 100001 td 0xffffff00010e59c0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 --More-- rap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab79aff0, rbp = 0xffffffffac25cbd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a4 sched_idletd() at sched_idletd+0x35 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac25cd30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00010e5680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x2ab sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2e3 kern_wait() at kern_wait+0x3cb wait4() at wait4+0x35 syscall() at syscall+0x1ce --More-- fast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf5c, rsp = 0x7fffffffe8e8, rbp = 0x7fffffffed70 --- Tracing command audit pid 10 tid 100003 td 0xffffff00010e5340 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x14e audit_worker() at audit_worker+0x86 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac252d30, rbp = 0 --- Tracing command swapper pid 0 tid 0 td 0xffffffff80a1ad20 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 scheduler() at scheduler+0x3c1 mi_startup() at mi_startup+0x59 btext() at btext+0x2c --More-- b>      panic panic: from debugger cpuid = 0 Uptime: 8m24s Physical memory: 4083 MB Dumping 303 MB: 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Dump complete Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. ~ >How-To-Repeat: I am not sure if this issue can be easily reproduced on a different setting. >Fix: None. >Release-Note: >Audit-Trail: >Unformatted: