From owner-freebsd-bugs@FreeBSD.ORG Tue Oct 15 21:40:02 2013 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9AF716C0 for ; Tue, 15 Oct 2013 21:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 707032A79 for ; Tue, 15 Oct 2013 21:40:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9FLe2IP059709 for ; Tue, 15 Oct 2013 21:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9FLe2LZ059707; Tue, 15 Oct 2013 21:40:02 GMT (envelope-from gnats) Resent-Date: Tue, 15 Oct 2013 21:40:02 GMT Resent-Message-Id: <201310152140.r9FLe2LZ059707@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Anton Shterenlikht Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 21BBA651 for ; Tue, 15 Oct 2013 21:34:20 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog121.obsmtp.com (eu1sys200aog121.obsmtp.com [207.126.144.151]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 078CC2A42 for ; Tue, 15 Oct 2013 21:34:18 +0000 (UTC) Received: from mail-ee0-f54.google.com ([74.125.83.54]) (using TLSv1) by eu1sys200aob121.postini.com ([207.126.147.11]) with SMTP ID DSNKUl202YsBv4UPZVG7+c53AXo2wQHcTqas@postini.com; Tue, 15 Oct 2013 21:34:19 UTC Received: by mail-ee0-f54.google.com with SMTP id e53so4298204eek.13 for ; Tue, 15 Oct 2013 14:34:17 -0700 (PDT) Received: from mech-aslap33.men.bris.ac.uk (cpc2-aztw9-0-0-cust169.18-1.cable.virginmedia.com. [77.97.103.170]) by mx.google.com with ESMTPSA id bn13sm171304257eeb.11.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Oct 2013 14:34:16 -0700 (PDT) Received: from mech-aslap33.men.bris.ac.uk (localhost [127.0.0.1]) by mech-aslap33.men.bris.ac.uk (8.14.7/8.14.7) with ESMTP id r9FLYEXE010288 for ; Tue, 15 Oct 2013 22:34:14 +0100 (BST) (envelope-from mexas@mech-aslap33.men.bris.ac.uk) Received: (from mexas@localhost) by mech-aslap33.men.bris.ac.uk (8.14.7/8.14.7/Submit) id r9FLYE06010287; Tue, 15 Oct 2013 22:34:14 +0100 (BST) (envelope-from mexas) Message-Id: <201310152134.r9FLYE06010287@mech-aslap33.men.bris.ac.uk> Date: Tue, 15 Oct 2013 22:34:14 +0100 (BST) From: Anton Shterenlikht Sender: Anton Shterenlikht To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.114 Subject: kern/183007: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Anton Shterenlikht List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 21:40:02 -0000 >Number: 183007 >Category: kern >Synopsis: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 15 21:40:01 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Anton Shterenlikht >Release: FreeBSD 10.0-CURRENT amd64 >Organization: University of Bristol >Environment: System: FreeBSD mech-aslap33.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r255093: Sat Aug 31 13:49:24 BST 2013 root@mech-aslap33.men.bris.ac.uk:/usr/obj/usr/src/sys/MINKY amd64 >Description: below are the outputs of ddb commands: show msgbuf show proc run lockinfo show pcpu bt show threads ps alltrace show pginfo db> show msgbuf msgbufp = 0xe0000040fffdffb8 magic = 63062, size = 98232, r= 11031, w = 11766, ptr = 0xe0000040fffc8000, cksum= 898967 panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks cpuid = 1 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc00000060b900) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ee1b48, 0x9ffc000000592a40, 0x40c, 0x9ffc0000010b48a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000dc54d8, 0xa00000009de3b518) at vpanic+0x260 panic(0x9ffc000000dc54d8, 0x9ffc000000dc5768, 0xe0000000128ecd80, 0xdbbdc) at panic+0x80 deadlkres(0xdbbdc, 0xe0000000128ecd80, 0x9ffc000000dc8140, 0x9ffc000000dc5468) at deadlkres+0x420 fork_exit(0x9ffc000000e23230, 0x0, 0xa00000009de3b550) at fork_exit+0x120 enter_userland() at enter_userland KDB: enter: panic GDB: debug ports: uart GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #7 r255488: Tue Oct 15 10:09:07 BST 2013 root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Madison II (1600 MHz Itanium 2) Origin = "GenuineIntel" Revision = 2 Features = 0x1 real memory = 10737418240 (10240 MB) avail memory = 10484695040 (9998 MB) FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP) cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0 random: initialized Event timer "ITC" frequency 1600000000 Hz quality 1000 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603) ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603) acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 cpu1: on acpi0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0 acpi_tz0: on acpi0 pcib0: on acpi0 pcib0: could not evaluate _ADR - AE_NOT_FOUND pci0: on pcib0 ohci0: mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0 usbus0 on ohci0 ohci1: mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0 usbus1 on ohci1 ehci0: mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0 usbus2: EHCI version 0.95 usbus2 on ehci0 pci0: at device 2.0 (no driver attached) pcib1: on acpi0 pcib1: could not evaluate _ADR - AE_NOT_FOUND pci32: on pcib1 mpt0: port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32 mpt0: MPI Version=1.2.12.0 mpt1: port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32 mpt1: MPI Version=1.2.12.0 em0: port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32 em0: Ethernet address: 00:13:21:5b:05:1c em1: port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32 em1: Ethernet address: 00:13:21:5b:05:1d pcib2: on acpi0 pcib2: could not evaluate _ADR - AE_NOT_FOUND pci64: on pcib2 pcib3: on acpi0 pcib3: could not evaluate _ADR - AE_NOT_FOUND pci96: on pcib3 pcib4: on acpi0 pcib4: could not evaluate _ADR - AE_NOT_FOUND pci128: on pcib4 pcib5: on acpi0 pcib5: could not evaluate _ADR - AE_NOT_FOUND pci192: on pcib5 isp0: port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192 pcib6: on acpi0 pcib6: could not evaluate _ADR - AE_NOT_FOUND pci224: on pcib6 uart0: mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224 puc0: mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224 uart1: at port 1 on puc0 uart1: console (9600,n,8,1) uart2: at port 3 on puc0 vgapci0: port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224 uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0 uart4: debug port (9600,n,8,1) ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 IP Filter: v5.1.2 initialized. Default = block all, Logging = enabled usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 ugen2.1: at usbus2 uhub0: on usbus2 ugen1.1: at usbus1 uhub1: on usbus1 ugen0.1: at usbus0 uhub2: on usbus0 uhub1: 2 ports with 2 removable, self powered uhub2: 3 ports with 3 removable, self powered uhub0: 5 ports with 5 removable, self powered Expensive timeout(9) function: 0x9ffc000000e1e1e0(0x9ffc000000ee3c00) 0.020669666 s Expensive timeout(9) function: 0x9ffc000000e222e0(0xa0000000009ed320) 0.282418016 s da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 320.000MB/s transfersWARNING: WITNESS option enabled, expect reduced performance. (160.000MHz, offset 63WARNING: DIAGNOSTIC option enabled, expect reduced performance. , 16bit) da0: Command Queueing enabled da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit) da1: Command Queueing enabled da1: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C) da2 at mpt1 bus 0 scbus1 target 2 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit) da2: Command Queueing enabled da2: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C) Trying to mount root from ufs:/dev/da0p2 [rw]... WARNING: / was not properly dismounted WITNESS: unable to allocate a new witness object <118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c. <118>Setting hostid: 0x8968724b. <118>dumpon: /dev/da4p1: No such file or directory <118>/etc/rc: WARNING: unable to specify /dev/da4p1 as a dump device <118>Starting ddb. <118>Entropy harvesting: interrupts ethernet point_to_point kickstart. <118>swapon: /dev/da4p1: No such file or directory <118>Starting file system checks: da3 at isp0 bus 0 scbus2 target 0 lun 1 da3: Fixed Direct Access SCSI-4 device da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da3: Command Queueing enabled da3: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da4 at isp0 bus 0 scbus2 target 0 lun 2 da4: Fixed Direct Access SCSI-4 device da4: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da4: Command Queueing enabled da4: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da5 at isp0 bus 0 scbus2 target 0 lun 3 da5: Fixed Direct Access SCSI-4 device da5: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da5: Command Queueing enabled da5: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C) da6 at isp0 bus 0 scbus2 target 0 lun 5 da6: Fixed Direct Access SCSI-4 device da6: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000 da6: Command Queueing enabled da6: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C) <118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0p2: 582356 files, 3927896 used, 30688528 free (183320 frags, 3813151 blocks, 0.5% fragmentation) <118>/dev/da3p1: INCORRECT BLOCK COUNT I=642145 (8 should be 0) (CORRECTED) <118>/dev/da3p1: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da3p1: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da3p1: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da3p1: 1967 files, 7223674 used, 9999687 free (263 frags, 1249928 blocks, 0.0% fragmentation) <118>/dev/da5p1: 436860 files, 1247548 used, 15975813 free (36405 frags, 1992426 blocks, 0.2% fragmentation) <118>/dev/da6p1: 5481 files, 1323234 used, 68104269 free (5773 frags, 8512312 blocks, 0.0% fragmentation) <118>Mounting local file systems:. <118>Writing entropy file:. <118>Setting hostname: mech-as28.men.bris.ac.uk. <118>Enabling ipfilter. <118>Starting Network: lo0 em0 em1. <118>lo0: flags=8049 metric 0 mtu 16384 <118> options=600003 <118> inet6 ::1 prefixlen 128 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 <118> inet 127.0.0.1 netmask 0xff000000 <118> nd6 options=21 <118>em0: flags=8843 metric 0 mtu 1500 <118> options=209b <118> ether 00:13:21:5b:05:1c <118> inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255 <118> inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1 <118> nd6 options=29 <118> media: Ethernet autoselect <118> status: no carrier <118>em1: flags=8843 metric 0 mtu 1500 <118> options=209b <118> ether 00:13:21:5b:05:1d <118> inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255 <118> inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2 <118> nd6 options=29 <118> media: Ethernet autoselect (100baseTX ) <118> status: active <118>filter sync'd <118>Starting devd. <118>add net default: gateway 137.222.187.250 <118>add net fe80::: gateway ::1 <118>add net ff02::: gateway ::1 <118>add net ::ffff:0.0.0.0: gateway ::1 <118>add net ::0.0.0.0: gateway ::1 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>Starting ipmon. <118>Creating and/or trimming log files. <118>Starting syslogd. <118>No core dumps found. <118>Clearing /tmp (X related). <118>Recovering vi editor sessions:. <118>Updating motd:. <118>Mounting late file systems:. <118>Starting ntpd. <118>Starting bsdstats. <118>Posting monthly OS statistics to rpt.bsdstats.org <118>Performing sanity check on sshd configuration. <118>Starting sshd. <118>Starting cron. <118>Local package initialization:cd: /var/portbuild/: No such file or directory <118>. <118>Starting inetd. <118>Starting background file system checks in 60 seconds. <118> <118>Tue Oct 15 16:15:03 BST 2013 <118>Oct 15 16:15:43 mech-as28 su: mexas to root on /dev/pts/0 <118>Oct 15 16:26:25 mech-as28 su: mexas to root on /dev/pts/1 <118>Oct 15 20:42:07 mech-as28 su: mexas to root on /dev/pts/2 db> db> show proc Process 0 (kernel) at 0x9ffc000000efeac8: state: NORMAL uid: 0 gids: 0 ABI: null threads: 13 100053 Run CPU 1 [deadlkres] 100050 D - 0xe000000011795300 [system_taskq_1] 100049 D - 0xe000000011795300 [system_taskq_0] 100045 D - 0xe0000000111a2500 [em1 taskq] 100044 D - 0xe0000000111a2600 [em0 taskq] 100023 D - 0xe0000000111a5300 [ffs_trim taskq] 100020 RunQ [thread taskq] 100017 D - 0xe0000000111a5900 [acpi_task_2] 100016 D - 0xe0000000111a5900 [acpi_task_1] 100015 D - 0xe0000000111a5900 [acpi_task_0] 100014 D - 0xe0000000111a5a00 [kqueue taskq] 100009 D - 0xe0000000111a5d00 [firmware taskq] 100000 D swapin 0x9ffc000000efeac8 [swapper] db> db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000120e1ad0: tag devfs, type VDIR usecount 3, writecount 0, refcount 4 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) lock type devfs: EXCL by thread 0xe000000015800900 (pid 20716, sh, tid 100129) with exclusive waiters pending 0xe0000000120e3860: tag devfs, type VCHR usecount 29, writecount 16, refcount 30 mountedhere 0xe0000000111a7800 flags (VV_ETERNALDEV|VI_ACTIVE) lock type devfs: EXCL by thread 0xe0000000157a4d80 (pid 69993, sh, tid 100125) with exclusive waiters pending dev null db> db> show pcpu cpuid = 1 dynamic pcpu = 0x400009f8d2680 curthread = 0xe00000001178cd80: pid 0 "deadlkres" curpcb = 0 fpcurthread = none idlethread = 0xe000000011182480: tid 100004 "idle: cpu1" MD: vhpt = 0xe000000011200000 MD: lid = 0x1000000 MD: clock = 0x1caaca1b73b6 MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0xe000000011158a30 MD: pmap = 0x9ffc000000eff500 spin locks held: db> db> bt Tracing pid 0 tid 100053 td 0xe00000001178cd80 db> db> show threads 100138 (0xe0000000157e5b00) (stack 0xa0000000002ca000) *** 100139 (0xe0000000157e5680) (stack 0xa0000000002d2000) *** 100147 (0xe0000001d2c36000) (stack 0xa000000000302000) *** 100093 (0xe00000001258b680) (stack 0xa00000009df5e000) *** 100120 (0xe00000001574bb00) (stack 0xa00000000023a000) *** 100078 (0xe00000001258a000) (stack 0xa00000009defe000) *** 100112 (0xe00000001574a000) (stack 0xa00000009dff6000) *** 100104 (0xe0000000128ecd80) (stack 0xa00000009dfb6000) *** 100151 (0xe0000001d2c37b00) (stack 0xa000000000322000) *** 100116 (0xe000000015770000) (stack 0xa00000000021a000) *** 100119 (0xe000000015771680) (stack 0xa000000000232000) *** 100107 (0xe000000012919b00) (stack 0xa00000009dfce000) *** 100117 (0xe0000000157a4000) (stack 0xa000000000222000) *** 100126 (0xe0000000157a4900) (stack 0xa00000000026a000) *** 100129 (0xe000000015800900) (stack 0xa000000000282000) *** 100100 (0xe00000001279bb00) (stack 0xa00000009df96000) *** 100125 (0xe0000000157a4d80) (stack 0xa000000000262000) *** 100131 (0xe0000000157e4480) (stack 0xa000000000292000) *** 100118 (0xe000000015771b00) (stack 0xa00000000022a000) *** 100103 (0xe0000000128ed200) (stack 0xa00000009dfae000) *** 100056 (0xe000000012168d80) (stack 0xa00000009de4e000) *** 100070 (0xe0000000124f8900) (stack 0xa00000009debe000) *** 100073 (0xe00000001258a900) (stack 0xa00000009ded6000) *** 100098 (0xe0000000128ec480) (stack 0xa00000009df86000) *** 100096 (0xe000000012918480) (stack 0xa00000009df76000) *** 100099 (0xe0000000128ec000) (stack 0xa00000009df8e000) *** 100091 (0xe00000001258b200) (stack 0xa00000009df4e000) *** 100065 (0xe000000012168480) (stack 0xa00000009de96000) *** 100095 (0xe000000012918900) (stack 0xa00000009df6e000) *** 100081 (0xe00000001279a000) (stack 0xa00000009df16000) *** 100088 (0xe00000001279ad80) (stack 0xa00000009df36000) *** 100068 (0xe0000000124f9200) (stack 0xa00000009deae000) *** 100079 (0xe0000000124f9b00) (stack 0xa00000009df06000) *** 100076 (0xe0000000123fa900) (stack 0xa00000009deee000) *** 100074 (0xe0000000123fb200) (stack 0xa00000009dede000) *** 100066 (0xe000000012168000) (stack 0xa00000009de9e000) *** 100064 (0xe0000000123fb680) (stack 0xa00000009de8e000) *** 100063 (0xe0000000123fbb00) (stack 0xa00000009de86000) *** 100062 (0xe000000012169200) (stack 0xa00000009de7e000) *** 100061 (0xe000000012169680) (stack 0xa00000009de76000) *** 100060 (0xe000000012169b00) (stack 0xa00000009de6e000) *** 100059 (0xe0000000123fa000) (stack 0xa00000009de66000) *** 100058 (0xe0000000123fa480) (stack 0xa00000009de5e000) *** 100057 (0xe000000012168900) (stack 0xa00000009de56000) *** 100054 (0xe00000001178c900) (stack 0xa00000009de3e000) *** 100052 (0xe00000001178d200) (stack 0xa00000009de2c000) *** 100051 (0xe00000001178d680) (stack 0xa00000009de24000) *** 100047 (0xe00000001174db00) (stack 0xa00000009de04000) *** 100043 (0xe00000001174c000) (stack 0xa00000009ddc6000) *** 100041 (0xe00000001174c900) (stack 0xa00000009dd68000) *** 100039 (0xe000000011744480) (stack 0xa00000009dd0a000) *** 100038 (0xe000000011744900) (stack 0xa00000009dd02000) *** 100037 (0xe000000011744d80) (stack 0xa00000009dcfa000) *** 100036 (0xe000000011745200) (stack 0xa00000009dcf2000) *** 100034 (0xe000000011480d80) (stack 0xa0000000e2fe6000) *** 100033 (0xe000000011481200) (stack 0xa0000000e2fde000) *** 100032 (0xe000000011481680) (stack 0xa0000000e2fd6000) *** 100031 (0xe000000011481b00) (stack 0xa0000000e2fce000) *** 100029 (0xe000000011425680) (stack 0xa0000000e2f36000) *** 100028 (0xe000000011425b00) (stack 0xa0000000e2f2e000) *** 100027 (0xe000000011480000) (stack 0xa0000000e2f26000) *** 100026 (0xe000000011480480) (stack 0xa0000000e2f1e000) *** 100024 (0xe000000011424000) (stack 0xa0000000e2e8a000) *** 100013 (0xe0000000111a1680) (stack 0xa0000000a07f6000) *** 100012 (0xe0000000111a1b00) (stack 0xa0000000a07ee000) *** 100011 (0xe0000000111b0000) (stack 0xa0000000a07e6000) *** 100010 (0xe000000011183680) (stack 0xa0000000a07de000) *** 100048 (0xe00000001174d680) (stack 0xa00000009de0c000) *** 100046 (0xe00000001178c000) (stack 0xa00000009ddde000) *** 100042 (0xe00000001174c480) (stack 0xa00000009dd70000) *** 100040 (0xe000000011744000) (stack 0xa00000009dd12000) *** 100035 (0xe000000011480900) (stack 0xa00000009dcea000) *** 100030 (0xe000000011425200) (stack 0xa0000000e2fc6000) *** 100025 (0xe0000000111b1b00) (stack 0xa0000000e2f16000) *** 100022 (0xe000000011424900) (stack 0xa0000000e2e7a000) *** 100021 (0xe000000011424d80) (stack 0xa0000000e2e72000) *** 100019 (0xe0000000111b0900) (stack 0xa0000000e2e62000) *** 100018 (0xe0000000111b0d80) (stack 0xa0000000e2e5a000) *** 100008 (0xe0000000111a0000) (stack 0xa0000000a07ce000) *** 100007 (0xe0000000111a0480) (stack 0xa0000000a07c6000) *** 100006 (0xe0000000111a0900) (stack 0xa0000000a07be000) *** 100005 (0xe000000011182000) (stack 0xa0000000a07b6000) *** 100004 (0xe000000011182480) (stack 0xa0000000a07ae000) *** 100003 (0xe000000011182900) (stack 0xa0000000a07a6000) *** 100002 (0xe000000011182d80) (stack 0xa0000000a079e000) *** 100001 (0xe000000011183200) (stack 0xa0000000a0796000) *** 100053 (0xe00000001178cd80) (stack 0xa00000009de34000) *** 100050 (0xe00000001174cd80) (stack 0xa00000009de1c000) *** 100049 (0xe00000001174d200) (stack 0xa00000009de14000) *** 100045 (0xe000000011745680) (stack 0xa00000009ddd6000) *** 100044 (0xe000000011745b00) (stack 0xa00000009ddce000) *** 100023 (0xe000000011424480) (stack 0xa0000000e2e82000) *** 100020 (0xe0000000111b0480) (stack 0xa0000000e2e6a000) *** 100017 (0xe0000000111b1200) (stack 0xa0000000e2e52000) *** 100016 (0xe0000000111b1680) (stack 0xa0000000e2e4a000) *** 100015 (0xe0000000111a0d80) (stack 0xa0000000e2e42000) *** 100014 (0xe0000000111a1200) (stack 0xa0000000e2e3a000) *** 100009 (0xe000000011183b00) (stack 0xa0000000a07d6000) *** 100000 (0x9ffc000000efef70) (stack 0xe0000040fffc0000) *** db> db> ps pid ppid pgrp uid state wmesg wchan cmd 37126 1147 1147 0 D+ allproc 0x9ffc000000f0a350 sh 37125 42826 1147 0 DE+ allproc 0x9ffc000000f0a350 sed 37124 42826 1147 0 DE+ allproc 0x9ffc000000f0a350 awk 37123 42826 1147 0 DE+ proctree 0x9ffc000000f0a370 awk 37115 37110 1147 0 S+J piperd 0xe0000000129509a0 as 37111 37110 1147 0 LL+J *pmap pv 0xe00000001216b8c0 cc1plus 37110 28849 1147 0 S+J wait 0xe000000015790940 g++ 36890 36889 1147 0 LL+J *pmap pv 0xe00000001216b8c0 ld 36889 11278 1147 0 SW+J wait 0xe0000000157664a0 c++ 36591 36590 1147 0 LL+J *pmap pv 0xe00000001216b8c0 cc1plus 36590 14781 1147 0 SW+J wait 0xe00000001290d280 xg++ 28849 61132 1147 0 S+J wait 0xe000000015767720 gmake 18342 18341 18342 1001 Ds+ allproc 0x9ffc000000f0a350 tcsh 18341 18317 18317 1001 S select 0xe000000010c286c0 sshd 18317 924 18317 0 Ss select 0xe00000001273f640 sshd 14781 2824 1147 0 SW+J wait 0xe00000001572e000 gmake 11278 59577 1147 0 SW+J wait 0xe000000012762de0 gmake 61132 61124 1147 0 SW+J wait 0xe0000000157ec4a0 gmake 61124 61123 1147 0 SW+J wait 0xe000000015791720 gmake 61123 60934 1147 0 SW+J wait 0xe0000001d4970000 sh 60934 60932 1147 0 SW+J wait 0xe000000011797280 make 60932 20716 1147 0 SW+ wait 0xe0000001d300cde0 sh 59577 59537 1147 0 SW+J wait 0xe0000000124a6940 gmake 59537 59516 1147 0 SW+J wait 0xe000000015791280 gmake 59516 59515 1147 0 SW+J wait 0xe000000015a1b720 gmake 59515 59505 1147 0 SW+J wait 0xe00000001572f280 sh 59505 59479 1147 0 SW+J wait 0xe000000015a1b280 make 59479 59477 1147 0 SW+J wait 0xe0000000157b9280 make 59477 69993 1147 0 SW+ wait 0xe00000001572e4a0 sh 2824 77085 1147 0 SW+J wait 0xe000000015766940 gmake 28012 20716 1147 0 S+ piperd 0xe00000001586a738 tee 20716 1147 1147 0 D+ devfs 0xe0000000120e38c8 sh 77085 77056 1147 0 SW+J wait 0xe0000000157ecde0 gmake 77056 77055 1147 0 SW+J wait 0xe000000012762000 gmake 77055 77043 1147 0 SW+J wait 0xe0000000127ae940 sh 77043 77023 1147 0 SW+J wait 0xe0000000157b8000 make 77023 77021 1147 0 SW+J wait 0xe0000000157904a0 make 77021 62311 1147 0 SW+ wait 0xe0000000123f4940 sh 70398 69993 1147 0 S+ piperd 0xe000000012863ce0 tee 69993 1147 1147 0 D+ proctree 0x9ffc000000f0a370 sh 67686 62311 1147 0 S+ piperd 0xe0000001d4ed1340 tee 62311 1147 1147 0 D+ devfs 0xe0000000120e1b38 sh 42826 1147 1147 0 S+ wait 0xe00000001290c940 sh 1147 1030 1147 0 S+ piperd 0xe00000001586ae70 sh 1122 1121 1122 0 S+ ttyin 0xe00000001254e4a8 sh 1121 1047 1121 1001 SW+ wait 0xe00000001289c4a0 su 1047 1046 1047 1001 SWs+ pause 0xe0000000127ae548 tcsh 1046 1042 1042 1001 S select 0xe00000001273fd40 sshd 1042 924 1042 0 Ss select 0xe000000010c287c0 sshd 1030 1029 1030 0 SW+ wait 0xe00000001289d720 sh 1029 1025 1029 1001 SW+ wait 0xe00000001289c940 su 1025 1024 1025 1001 SWs+ pause 0xe00000001290c0a8 tcsh 1024 1022 1022 1001 S select 0xe000000010c289c0 sshd 1022 924 1022 0 Ss select 0xe00000001273f7c0 sshd 997 1 997 0 Ss+ ttyin 0xe0000000114288a8 getty 996 1 1 0 S ttydcd 0xe000000011428ce8 getty 974 1 974 0 Ss select 0xe000000010c288c0 inetd 935 1 935 0 LLs *pmap pv 0xe00000001216b8c0 cron 930 1 930 25 SWs pause 0xe0000000124a7328 sendmail 927 1 927 0 Rs sendmail 924 1 924 0 Ss select 0xe000000010c28e40 sshd 864 1 864 0 LLs *pmap pv 0xe00000001216b8c0 ntpd 784 1 784 0 Rs syslogd 775 1 775 0 Ss select 0xe000000010c29040 ipmon 653 1 653 0 Ss select 0xe00000001273fe40 devd 138 1 138 0 SWs pause 0xe0000000111d77c8 adjkerntz 23 0 0 0 DL sdflush 0x9ffc000000ee2d30 [softdepflush] 22 0 0 0 RL [vnlru] 21 0 0 0 RL [syncer] 20 0 0 0 RL [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ee30d4 [pagezero] 18 0 0 0 RL CPU 0 [vmdaemon] 17 0 0 0 LL *pmap pv 0xe00000001216b8c0 [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ee33a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc000001104130 [sctp_iterator] 5 0 0 0 RL (threaded) [zfskern] 100052 RunQ [l2arc_feed_thread] 100051 RunQ [arc_reclaim_thread] 4 0 0 0 DL ispf 0xe000000011768000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ee0a50 [acpi_thermal] 14 0 0 0 RL [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100012 D - 0x9ffc000000ee1270 [g_down] 100011 D - 0x9ffc000000ee1268 [g_up] 100010 D - 0x9ffc000000ee1258 [g_event] 12 0 0 0 WL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100018 I [swi2: cambio] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011180de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000011087c8 [audit] 0 0 0 0 RLs (threaded) [kernel] 100053 Run CPU 1 [deadlkres] 100050 D - 0xe000000011795300 [system_taskq_1] 100049 D - 0xe000000011795300 [system_taskq_0] 100045 D - 0xe0000000111a2500 [em1 taskq] 100044 D - 0xe0000000111a2600 [em0 taskq] 100023 D - 0xe0000000111a5300 [ffs_trim taskq] 100020 RunQ [thread taskq] 100017 D - 0xe0000000111a5900 [acpi_task_2] 100016 D - 0xe0000000111a5900 [acpi_task_1] 100015 D - 0xe0000000111a5900 [acpi_task_0] 100014 D - 0xe0000000111a5a00 [kqueue taskq] 100009 D - 0xe0000000111a5d00 [firmware taskq] 100000 D swapin 0x9ffc000000efeac8 [swapper] db> db> alltrace Tracing command sh pid 37126 tid 100138 td 0xe0000000157e5b00 Tracing command sed pid 37125 tid 100139 td 0xe0000000157e5680 Tracing command awk pid 37124 tid 100147 td 0xe0000001d2c36000 Tracing command awk pid 37123 tid 100093 td 0xe00000001258b680 Tracing command as pid 37115 tid 100120 td 0xe00000001574bb00 Tracing command cc1plus pid 37111 tid 100078 td 0xe00000001258a000 Tracing command g++ pid 37110 tid 100112 td 0xe00000001574a000 Tracing command ld pid 36890 tid 100104 td 0xe0000000128ecd80 Tracing command c++ pid 36889 tid 100115 td 0xe000000012919200 Tracing command cc1plus pid 36591 tid 100151 td 0xe0000001d2c37b00 Tracing command xg++ pid 36590 tid 100101 td 0xe0000000128edb00 Tracing command gmake pid 28849 tid 100116 td 0xe000000015770000 Tracing command tcsh pid 18342 tid 100119 td 0xe000000015771680 Tracing command sshd pid 18341 tid 100107 td 0xe000000012919b00 Tracing command sshd pid 18317 tid 100117 td 0xe0000000157a4000 Tracing command gmake pid 14781 tid 100111 td 0xe00000001574a480 Tracing command gmake pid 11278 tid 100077 td 0xe00000001258a480 Tracing command gmake pid 61132 tid 100134 td 0xe000000015801680 Tracing command gmake pid 61124 tid 100122 td 0xe000000015770d80 Tracing command sh pid 61123 tid 100149 td 0xe0000001d492c480 Tracing command make pid 60934 tid 100067 td 0xe00000001178db00 Tracing command sh pid 60932 tid 100141 td 0xe000000015801200 Tracing command gmake pid 59577 tid 100071 td 0xe0000000124f8480 Tracing command gmake pid 59537 tid 100123 td 0xe0000000157a5680 Tracing command gmake pid 59516 tid 100145 td 0xe0000001d2c36480 Tracing command sh pid 59515 tid 100105 td 0xe0000000128ec900 Tracing command make pid 59505 tid 100135 td 0xe000000015800480 Tracing command make pid 59479 tid 100127 td 0xe0000000157a4480 Tracing command sh pid 59477 tid 100110 td 0xe00000001574a900 Tracing command gmake pid 2824 tid 100109 td 0xe000000012919680 Tracing command tee pid 28012 tid 100126 td 0xe0000000157a4900 Tracing command sh pid 20716 tid 100129 td 0xe000000015800900 Tracing command gmake pid 77085 tid 100132 td 0xe0000000157e4000 Tracing command gmake pid 77056 tid 100080 td 0xe0000000124f9680 Tracing command sh pid 77055 tid 100085 td 0xe00000001258ad80 Tracing command make pid 77043 tid 100121 td 0xe000000015771200 Tracing command make pid 77023 tid 100113 td 0xe000000015770900 Tracing command sh pid 77021 tid 100075 td 0xe0000000123fad80 Tracing command tee pid 70398 tid 100100 td 0xe00000001279bb00 Tracing command sh pid 69993 tid 100125 td 0xe0000000157a4d80 Tracing command tee pid 67686 tid 100131 td 0xe0000000157e4480 Tracing command sh pid 62311 tid 100118 td 0xe000000015771b00 Tracing command sh pid 42826 tid 100103 td 0xe0000000128ed200 Tracing command sh pid 1147 tid 100056 td 0xe000000012168d80 Tracing command sh pid 1122 tid 100070 td 0xe0000000124f8900 Tracing command su pid 1121 tid 100090 td 0xe00000001258bb00 Tracing command tcsh pid 1047 tid 100086 td 0xe00000001279b680 Tracing command sshd pid 1046 tid 100073 td 0xe00000001258a900 Tracing command sshd pid 1042 tid 100098 td 0xe0000000128ec480 Tracing command sh pid 1030 tid 100094 td 0xe000000012918d80 Tracing command su pid 1029 tid 100097 td 0xe000000012918000 Tracing command tcsh pid 1025 tid 100092 td 0xe00000001279a480 Tracing command sshd pid 1024 tid 100096 td 0xe000000012918480 Tracing command sshd pid 1022 tid 100099 td 0xe0000000128ec000 Tracing command getty pid 997 tid 100091 td 0xe00000001258b200 Tracing command getty pid 996 tid 100065 td 0xe000000012168480 Tracing command inetd pid 974 tid 100095 td 0xe000000012918900 Tracing command cron pid 935 tid 100081 td 0xe00000001279a000 Tracing command sendmail pid 930 tid 100069 td 0xe0000000124f8d80 Tracing command sendmail pid 927 tid 100088 td 0xe00000001279ad80 Tracing command sshd pid 924 tid 100068 td 0xe0000000124f9200 Tracing command ntpd pid 864 tid 100079 td 0xe0000000124f9b00 Tracing command syslogd pid 784 tid 100076 td 0xe0000000123fa900 Tracing command ipmon pid 775 tid 100074 td 0xe0000000123fb200 Tracing command devd pid 653 tid 100066 td 0xe000000012168000 Tracing command adjkerntz pid 138 tid 100055 td 0xe00000001178c480 Tracing command softdepflush pid 23 tid 100064 td 0xe0000000123fb680 Tracing command vnlru pid 22 tid 100063 td 0xe0000000123fbb00 Tracing command syncer pid 21 tid 100062 td 0xe000000012169200 Tracing command bufdaemon pid 20 tid 100061 td 0xe000000012169680 Tracing command pagezero pid 19 tid 100060 td 0xe000000012169b00 Tracing command vmdaemon pid 18 tid 100059 td 0xe0000000123fa000 Tracing command pagedaemon pid 17 tid 100058 td 0xe0000000123fa480 Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012168900 Tracing command sctp_iterator pid 6 tid 100054 td 0xe00000001178c900 Tracing command zfskern pid 5 tid 100052 td 0xe00000001178d200 Tracing command zfskern pid 5 tid 100051 td 0xe00000001178d680 Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe00000001174db00 Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe00000001174c000 Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe00000001174c900 Tracing command usb pid 16 tid 100039 td 0xe000000011744480 Tracing command usb pid 16 tid 100038 td 0xe000000011744900 Tracing command usb pid 16 tid 100037 td 0xe000000011744d80 Tracing command usb pid 16 tid 100036 td 0xe000000011745200 Tracing command usb pid 16 tid 100034 td 0xe000000011480d80 Tracing command usb pid 16 tid 100033 td 0xe000000011481200 Tracing command usb pid 16 tid 100032 td 0xe000000011481680 Tracing command usb pid 16 tid 100031 td 0xe000000011481b00 Tracing command usb pid 16 tid 100029 td 0xe000000011425680 Tracing command usb pid 16 tid 100028 td 0xe000000011425b00 Tracing command usb pid 16 tid 100027 td 0xe000000011480000 Tracing command usb pid 16 tid 100026 td 0xe000000011480480 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011424000 Tracing command rand_harvestq pid 14 tid 100013 td 0xe0000000111a1680 Tracing command geom pid 13 tid 100012 td 0xe0000000111a1b00 Tracing command geom pid 13 tid 100011 td 0xe0000000111b0000 Tracing command geom pid 13 tid 100010 td 0xe000000011183680 Tracing command intr pid 12 tid 100048 td 0xe00000001174d680 Tracing command intr pid 12 tid 100046 td 0xe00000001178c000 Tracing command intr pid 12 tid 100042 td 0xe00000001174c480 Tracing command intr pid 12 tid 100040 td 0xe000000011744000 Tracing command intr pid 12 tid 100035 td 0xe000000011480900 Tracing command intr pid 12 tid 100030 td 0xe000000011425200 Tracing command intr pid 12 tid 100025 td 0xe0000000111b1b00 Tracing command intr pid 12 tid 100022 td 0xe000000011424900 Tracing command intr pid 12 tid 100021 td 0xe000000011424d80 Tracing command intr pid 12 tid 100019 td 0xe0000000111b0900 Tracing command intr pid 12 tid 100018 td 0xe0000000111b0d80 Tracing command intr pid 12 tid 100008 td 0xe0000000111a0000 Tracing command intr pid 12 tid 100007 td 0xe0000000111a0480 Tracing command intr pid 12 tid 100006 td 0xe0000000111a0900 Tracing command intr pid 12 tid 100005 td 0xe000000011182000 Tracing command idle pid 11 tid 100004 td 0xe000000011182480 Tracing command idle pid 11 tid 100003 td 0xe000000011182900 Tracing command init pid 1 tid 100002 td 0xe000000011182d80 Tracing command audit pid 10 tid 100001 td 0xe000000011183200 Tracing command kernel pid 0 tid 100053 td 0xe00000001178cd80 Tracing command kernel pid 0 tid 100050 td 0xe00000001174cd80 Tracing command kernel pid 0 tid 100049 td 0xe00000001174d200 Tracing command kernel pid 0 tid 100045 td 0xe000000011745680 Tracing command kernel pid 0 tid 100044 td 0xe000000011745b00 Tracing command kernel pid 0 tid 100023 td 0xe000000011424480 Tracing command kernel pid 0 tid 100020 td 0xe0000000111b0480 Tracing command kernel pid 0 tid 100017 td 0xe0000000111b1200 Tracing command kernel pid 0 tid 100016 td 0xe0000000111b1680 Tracing command kernel pid 0 tid 100015 td 0xe0000000111a0d80 Tracing command kernel pid 0 tid 100014 td 0xe0000000111a1200 Tracing command kernel pid 0 tid 100009 td 0xe000000011183b00 Tracing command kernel pid 0 tid 100000 td 0x9ffc000000efef70 db> db> show pginfo 0xe0000000128ecd80 page 0xe0000000128ecd80 obj 0 pidx 0x9ffc000000f27760 phys 0x0 q 0 hold 0 wire 10074176 af 0x0 of 0x0 f 0xa000 act 24 busy 1273e880 valid 0x1240 dirty 0x0 db> db> textdump dump db> call doadump = 0x6 db> >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: