Date: Tue, 25 Mar 2014 12:30:01 GMT From: Anton Shterenlikht <mexas@bris.ac.uk> To: freebsd-ia64@FreeBSD.org Subject: ia64/183227: panic: uma_zfree: Freeing to non free bucket index. Message-ID: <201403251230.s2PCU1d6016335@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR ia64/183227; it has been noted by GNATS.
From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:
Subject: ia64/183227: panic: uma_zfree: Freeing to non free bucket index.
Date: Tue, 25 Mar 2014 05:15:42 -0700 (PDT)
This seems to be the same panic, but now in r262690.
# cat /var/crash/info.last
Dump header from device /dev/da0p3
Architecture: ia64
Architecture Version: 1
Dump Length: 73728B (0 MB)
Blocksize: 512
Dumptime: Tue Mar 25 03:39:36 2014
Hostname: mech-as221.men.bris.ac.uk
Magic: FreeBSD Text Dump
Version String: FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
Panic String: uma_zfree: Freeing to non free bucket index.
Dump Parity: 730754683
Bounds: 3
Dump Status: good
#
# cat /var/crash/panic.txt
uma_zfree: Freeing to non free bucket index.
# cat /var/crash/version.txt
FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
#
*************************************
msgbuf.txt
*************************************
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
CPU: Madison (1500 MHz Itanium 2)
Origin = "GenuineIntel" Revision = 5
Features = 0x1<LB>
real memory = 6442450944 (6144 MB)
avail memory = 6291316736 (5999 MB)
FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
random: <Software, Yarrow> initialized
Event timer "ITC" frequency 1500000000 Hz quality 1000
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
acpi_tz0: <Thermal Zone> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
ohci0: <NEC uPD 9210 USB controller> mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0
usbus0 on ohci0
ohci1: <NEC uPD 9210 USB controller> mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0
usbus1 on ohci1
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0
usbus2: EHCI version 0.95
usbus2 on ehci0
pci0: <mass storage, ATA> at device 2.0 (no driver attached)
pci0: <network, ethernet> at device 3.0 (no driver attached)
pcib1: <ACPI Host-PCI bridge> on acpi0
pcib1: could not evaluate _ADR - AE_NOT_FOUND
pci32: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32
mpt0: MPI Version=1.2.12.0
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32
mpt1: MPI Version=1.2.12.0
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x000105> mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32
bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit
miibus0: <MII bus> on bge0
brgphy0: <BCM5701 1000BASE-T media interface> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:11:0a:31:46:06
pcib2: <ACPI Host-PCI bridge> on acpi0
pcib2: could not evaluate _ADR - AE_NOT_FOUND
pci64: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pcib3: could not evaluate _ADR - AE_NOT_FOUND
pci96: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pcib4: could not evaluate _ADR - AE_NOT_FOUND
pci128: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pcib5: could not evaluate _ADR - AE_NOT_FOUND
pci192: <ACPI PCI bus> on pcib5
pcib6: <ACPI Host-PCI bridge> on acpi0
pcib6: could not evaluate _ADR - AE_NOT_FOUND
pci224: <ACPI PCI bus> on pcib6
uart0: <HP Auxiliary Diva Serial Port> mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224
puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224
uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
uart1: console (9600,n,8,1)
uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224
vgapci0: Boot video device
uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
uart4: debug port (9600,n,8,1)
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <NEC> at usbus0
ugen1.1: <NEC> at usbus1
uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <NEC> at usbus2
uhub2: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub1: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
random: unblocking device.
uhub2: 5 ports with 5 removable, self powered
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <HP 73.4G MAX3073NC HPC1> Fixed Direct Access SCSI-3 device
da0: Serial Number X1015960 0613
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST373454LC D404> Fixed Direct Access SCSI-3 device
da1: Serial Number 3KP345CH
da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
da1: Command Queueing enabled
da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
da2 at mpt1 bus 0 scbus1 target 2 lun 0
da2: <SEAGATE ST373454LC D402> Fixed Direct Access SCSI-3 device
da2: Serial Number 3KP0WEXZ
da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
da2: Command Queueing enabled
da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0p2 [rw]...
WARNING: / was not properly dismounted
<118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a.
<118>Setting hostid: 0x688171f3.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point swi.
<118>Starting file system checks:
<118>/dev/da0p2: UNREF FILE I=6288401 OWNER=root MODE=10644
<118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014 (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288429 OWNER=root MODE=10644
<118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014 (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288431 OWNER=root MODE=10644
<118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014 (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288432 OWNER=root MODE=10644
<118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014 (CLEARED)
<118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da0p2: SUMMARY INFORMATION BAD (SALVAGED)
<118>/dev/da0p2: BLK(S) MISSING IN BIT MAPS (SALVAGED)
<118>/dev/da0p2: 761703 files, 11337625 used, 19128012 free (156372 frags, 2371455 blocks, 0.5% fragmentation)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-as221.men.bris.ac.uk.
<118>Starting Network: lo0 bge0.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118> inet6 ::1 prefixlen 128
<118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
<118> inet 127.0.0.1 netmask 0xff000000
<118> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118> options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
<118> ether 00:11:0a:31:46:06
<118> inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255
<118> inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x1
<118> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118> media: Ethernet autoselect (none)
<118> status: no carrier
<118>Starting devd.
<118>Starting pflogd:
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc47 /usr/local/lib/gcc48 /usr/local/lib/graphviz
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>savecore: reboot
<118>Mar 21 14:10:17 mech-as221 savecore: reboot
<118>savecore: writing core to /var/crash/textdump.tar.2
<118>/var/crash/vmcore.2 not found
<118>Starting casperd.
<118>Clearing /tmp (X related).
<118>Recovering vi editor sessions:.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Performing sanity check on nginx configuration:
<118>nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
<118>nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
<118>Starting nginx.
<118>Starting bsdstats.
<118>Posting monthly OS statistics to rpt.bsdstats.org
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting sendmail_submit.
<118>Starting sendmail_msp_queue.
<118>Starting cron.
<118>Local package initialization:cd: /var/portbuild/: No such file or directory
<118>.
<118>Starting inetd.
<118>Starting background file system checks in 60 seconds.
<118>
<118>Fri Mar 21 14:10:22 GMT 2014
<118>Mar 21 14:11:03 mech-as221 login: ROOT LOGIN (root) ON ttyu1
lock order reversal:
1st 0xa00000005f6bf058 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
2nd 0xe000000015952400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
_witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b31b08) at _witness_debugger+0x60
witness_checkorder(0xe000000015952400, 0x9ffc000000b03808, 0x9ffc000000b31b08, 0x11c, 0x0) at witness_checkorder+0x15c0
_sx_xlock(0xe000000015952400, 0x0, 0x9ffc000000b31b08, 0x11c) at _sx_xlock+0x120
ufsdirhash_acquire(0xe0000000162cdb90) at ufsdirhash_acquire+0x70
ufsdirhash_newblk(0xe0000000162cdb90, 0xa200, 0x0, 0x9ffc0000008d59a0, 0xda1, 0xa000000090471030) at ufsdirhash_newblk+0x20
ufs_direnter(0xe0000000162f29c0, 0x0, 0xa000000090471068, 0xa000000090471310, 0x0, 0x1) at ufs_direnter+0x530
ufs_rename(0xa000000090471240, 0xa000000090471068, 0x0) at ufs_rename+0x1590
VOP_RENAME_APV(0x9ffc000000bc2740, 0xa000000090471368, 0x0, 0x0, 0x9ffc0000005c1f10, 0x797, 0x9ffc000000dd1720) at VOP_RENAME_APV+0x230
kern_renameat(0xa000000090471310, 0xa000000090471368, 0x0, 0xffffffffffffff9c, 0x12210c238, 0x0) at kern_renameat+0x630
kern_rename(0xe000000012613b00, 0x12210c1d8, 0x12210c238, 0x0) at kern_rename+0x40
sys_rename(0xe000000012613b00, 0xa0000000904714e8, 0x9ffc000000986c80, 0x48d) at sys_rename+0x30
syscall(0xe000000012ade940, 0x12210c238, 0x12210c108, 0xe000000012613b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
lock order reversal:
1st 0xe0000000129d47b8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851
2nd 0xe000000016c63658 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2101
KDB: stack backtrace:
db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
_witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60
witness_checkorder(0xe000000016c63658, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe000000016c63720, 0x9ffc000000ae1498) at witness_checkorder+0x15c0
__lockmgr_args(0xe000000016c63658, 0x80500, 0xe000000016c63720, 0x9ffc000000ae1498, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0
vop_stdlock(0xa0000000905790e0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80
VOP_LOCK1_APV(0x9ffc000000bad388, 0xa0000000905790c0, 0xe000000012ef7680) at VOP_LOCK1_APV+0x230
_vn_lock(0xe000000016c635f0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130
vget(0xe000000016c635f0, 0x80500, 0xe000000012ef7680, 0x9ffc000000242ca0) at vget+0x100
devfs_allocv(0xe000000103d57300, 0xe000000012a683c8, 0x80500, 0xa000000090579110) at devfs_allocv+0x1b0
devfs_root(0xe000000012a683c8, 0x80000, 0xa000000090579160, 0xe00000001b080190, 0x9ffc00000059dea0, 0x17b7, 0x8) at devfs_root+0x60
vfs_donmount(0xe000000012ef7680, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50
sys_nmount(0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0
syscall(0xe000000012ef1280, 0x8, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
lock order reversal:
1st 0xe000000016c47658 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851
2nd 0xe000000016c562d8 ufs (ufs) @ /usr/src/sys/fs/nullfs/null_vnops.c:620
KDB: stack backtrace:
db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
_witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000ae6b28) at _witness_debugger+0x60
witness_checkorder(0xe000000016c562d8, 0x9ffc000000b03808, 0x9ffc000000ae6b28, 0x26c, 0xe000000016c563a0, 0x9ffc000000aecf70) at witness_checkorder+0x15c0
__lockmgr_args(0xe000000016c562d8, 0x80500, 0xe000000016c563a0, 0x9ffc000000aecf70, 0x60, 0x33, 0x9ffc000000ae6b28, 0x26c) at __lockmgr_args+0x10a0
ffs_lock(0xa0000000904d10e0, 0xe000000016c562d8, 0x80500) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000bc1e00, 0xa0000000904d10c0, 0x9ffc00000034ebd0, 0x0) at VOP_LOCK1_APV+0x230
null_lock(0xa0000000904d10f0, 0x9ffc000000ae6b28, 0xe000000016c65860) at null_lock+0x1a0
VOP_LOCK1_APV(0x9ffc000000b8aa18, 0xa0000000904d10f0, 0x2) at VOP_LOCK1_APV+0x230
_vn_lock(0xe000000016c65860, 0x80400, 0x9ffc000000ae6a38, 0x118) at _vn_lock+0x130
nullfs_root(0xe000000012a66a28, 0x80000, 0xa0000000904d1160, 0xe000000016c65860, 0x9ffc00000059dea0, 0x17b7, 0x8) at nullfs_root+0x90
vfs_donmount(0xe000000012a52d80, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50
sys_nmount(0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0
syscall(0xe000000012e17280, 0x8, 0x0, 0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
<118>Mar 24 08:49:56 mech-as221 su: BAD SU mexas to root on /dev/pts/0
<118>Mar 24 08:50:01 mech-as221 su: mexas to root on /dev/pts/0
<118>Mar 24 11:14:02 mech-as221 su: mexas to root on /dev/pts/0
<118>Mar 24 13:19:28 mech-as221 su: mexas to root on /dev/pts/0
lock order reversal:
1st 0xe0000000170162d8 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1720
2nd 0xe0000040876f6548 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2101
KDB: stack backtrace:
db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
_witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60
witness_checkorder(0xe0000040876f6548, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe0000040876f6610, 0xa000000001362370) at witness_checkorder+0x15c0
__lockmgr_args(0xe0000040876f6548, 0x80500, 0xe0000040876f6610, 0xa000000001362370, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0
vop_stdlock(0xa00000008ff0f4d0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80
VOP_LOCK1_APV(0x9ffc000000bad388, 0xa00000008ff0f4b0, 0x9ffc000000f0d470) at VOP_LOCK1_APV+0x230
_vn_lock(0xe0000040876f64e0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130
vget(0xe0000040876f64e0, 0x80500, 0xe0000000122b8900, 0x0) at vget+0x100
vfs_msync(0xe0000001076c52e8, 0x4, 0xe0000040876f64e0, 0x9ffc0000005b4dd0) at vfs_msync+0x120
sync_fsync(0x9ffc000000b0e198, 0x9ffc000000f65a08, 0x240000) at sync_fsync+0x210
VOP_FSYNC_APV(0x9ffc000000badcc8, 0xa00000008ff0f528, 0x0, 0x9ffc000000b42028, 0x9ffc0000005b6390) at VOP_FSYNC_APV+0x270
sched_sync(0xe000000017016278, 0xe0000000170163d8, 0xe000000017016270, 0xe0000000121cfd00) at sched_sync+0x650
fork_exit(0x9ffc000000b4bcb0, 0x0, 0xa00000008ff0f550) at fork_exit+0x120
enter_userland() at enter_userland
panic: uma_zfree: Freeing to non free bucket index.
cpuid = 1
KDB: stack backtrace:
db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc000000450850, 0x40c, 0x9ffc000000dd1720) at kdb_backtrace+0xc0
vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x9ffc000000c2dcd0, 0x9ffc000000bfa490) at vpanic+0x260
kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120
uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0
free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150
zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30
zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0
zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170
zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580
zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0
zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290
zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50
zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0
taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50
taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0
taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130
fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120
enter_userland() at enter_userland
KDB: enter: panic
******************************
ddb.txt
******************************
db:0:kdb.enter.panic> run lockinfo
db:1:lockinfo> show locks
db:1:locks> show alllocks
Process 80735 (cc1plus) thread 0xe000000012a50d80 (100357)
shared lockmgr zfs (zfs) r = 0 (0xe0000000155c1178) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 76331 (find) thread 0xe000000012e5fb00 (100376)
shared lockmgr zfs (zfs) r = 0 (0xe000000015ec82d8) locked @ /usr/src/sys/kern/vfs_lookup.c:518
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.panic> show pcpu
cpuid = 1
dynamic pcpu = 0x400005fc29800
curthread = 0xe0000000121dbb00: pid 0 "zio_read_intr"
curpcb = 0
fpcurthread = none
idlethread = 0xe000000010b95680: tid 100004 "idle: cpu1"
MD: vhpt = 0xe00000003f100000
MD: lid = 0x1000000
MD: clock = 0x1a4179a98ef4b
MD: clock_mode = 2
MD: clock_load = 0
MD: stats = 0xe000000010b6ea30
MD: pmap = 0x9ffc000000c22dc0
spin locks held:
db:0:kdb.enter.panic> bt
Tracing pid 0 tid 100213 td 0xe0000000121dbb00
kdb_enter(0x9ffc000000af8ed8, 0x9ffc000000af8ed8, 0x9ffc0000004508a0, 0x40c) at kdb_enter+0x92
vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x100, 0x9ffc000000bfa490) at vpanic+0x2b0
kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120
uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0
free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150
zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30
zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0
zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170
zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580
zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0
zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290
zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50
zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0
taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50
taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0
taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130
fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120
enter_userland() at enter_userland
db:0:kdb.enter.panic> ps
pid ppid pgrp uid state wmesg wchan cmd
80756 80740 588 0 RJ CPU 0 c++
80740 15731 588 0 SJ wait 0xe000000104544de0 sh
80736 80734 588 0 SJ piperd 0xe00000001235d810 as
80735 80734 588 0 DJ zio->io_ 0xe0000040bebaf518 cc1plus
80734 47211 588 0 SJ wait 0xe0000040810f1280 c++
80733 49525 588 0 S nanslp 0x9ffc000000c2fa39 sleep
76333 76330 64193 0 S piperd 0xe00000001235ce70 wc
76332 76330 64193 0 S piperd 0xe000000018bed5a8 tee
76331 76330 64193 0 D zio->io_ 0xe0000000124db180 find
76330 76326 64193 0 S wait 0xe000000012ef1280 sh
76326 64478 64193 0 S piperd 0xe0000040810075a8 sh
64479 64471 64193 0 S piperd 0xe000000018bed340 mail
64478 64471 64193 0 S wait 0xe0000040811c7280 sh
64471 64470 64193 0 SW wait 0xe0000001045384a0 sh
64470 64468 64193 0 SW wait 0xe0000000122b7280 lockf
64468 64467 64193 0 SW wait 0xe000000012add720 sh
64467 64203 64193 0 SW wait 0xe000000178f0c940 sh
64204 64196 64193 0 S piperd 0xe00000001235d0d8 mail
64203 64196 64193 0 SW wait 0xe00000010452ede0 sh
64196 64195 64193 0 SW wait 0xe000000018bdd720 sh
64195 64193 64193 0 SW wait 0xe000000012e17720 lockf
64193 64191 64193 0 SWs wait 0xe000000012ef0de0 sh
64191 786 786 0 S piperd 0xe000000012e06738 cron
47211 92677 588 0 SJ wait 0xe0000001045444a0 make
92677 92676 588 0 SWJ wait 0xe00000010452f720 make
92676 92652 588 0 SWJ wait 0xe000000012ef44a0 sh
92652 92650 588 0 SWJ wait 0xe00000010452e940 make
92650 59384 588 0 SW wait 0xe000000018bdc940 sh
59715 59384 588 0 S piperd 0xe000000012e070d8 tee
59384 3381 588 0 S select 0xe0000000123ec4c0 sh
15731 87774 588 0 SJ wait 0xe000004080a1ade0 gmake
87774 87772 588 0 SWJ wait 0xe0000000122b0000 gmake
87772 87688 588 0 SWJ wait 0xe0000001045a5720 sh
87688 87686 588 0 SWJ wait 0xe000004080a1b280 make
87686 63244 588 0 SW wait 0xe0000001044e4940 sh
63399 63244 588 0 S piperd 0xe000000018becc08 tee
63244 3381 588 0 S select 0xe0000000123eb5c0 sh
49525 3381 588 0 S wait 0xe000000012ade940 sh
3381 588 588 0 S select 0xe0000000123eb940 sh
593 587 593 25 Ss piperd 0xe00000010459ac08 sendmail
588 587 588 0 SWs wait 0xe0000000122b6de0 sh
587 786 786 0 S piperd 0xe00000010459a268 cron
875 1 875 0 Ss+ ttyin 0xe000000011c30ca8 getty
844 1 1 0 S ttydcd 0xe000000011c310e8 getty
821 1 821 0 Ss select 0xe0000000122505c0 inetd
786 1 786 0 Ss nanslp 0x9ffc000000c2fa38 cron
782 1 782 25 Ss pause 0xe000000012adc9e8 sendmail
779 1 779 0 Ss select 0xe0000000123ec140 sendmail
776 1 776 0 Ss select 0xe0000000123ec3c0 sshd
723 721 721 80 S kqread 0xe000000010bbb300 nginx
721 1 721 0 SWs pause 0xe000000010b93328 nginx
702 1 702 0 Ss select 0xe000000012250fc0 ntpd
642 1 642 0 Ss select 0xe000000012250dc0 casperd
641 1 22 0 S+ select 0xe000000012250c40 casperd
612 1 612 0 Ss select 0xe000000012250d40 syslogd
486 1 486 0 Ss select 0xe000000012250e40 devd
135 1 135 0 SWs pause 0xe000000012adce88 adjkerntz
40 0 0 0 DL (threaded) [zfskern]
100341 D tx->tx_s 0xe0000000125d8208 [txg_thread_enter]
100340 D tx->tx_q 0xe0000000125d8228 [txg_thread_enter]
100334 D spa->spa 0xe0000000123daac0 [trim tank]
100067 D l2arc_fe 0xa0000000013d5540 [l2arc_feed_thread]
100066 D arc_recl 0xa0000000013c5720 [arc_reclaim_thread]
21 0 0 0 DL sdflush 0x9ffc000000bfb890 [softdepflush]
20 0 0 0 DL vlruwt 0xe0000000122b0de0 [vnlru]
19 0 0 0 DL syncer 0x9ffc000000f65b10 [syncer]
18 0 0 0 DL psleep 0x9ffc000000bfac70 [bufdaemon]
17 0 0 0 DL pgzero 0x9ffc000000bfbbfc [pagezero]
9 0 0 0 DL psleep 0x9ffc000000bfbb64 [vmdaemon]
8 0 0 0 DL psleep 0x9ffc000000bfbb94 [pagedaemon]
5 0 0 0 DL waiting_ 0x9ffc000000f681f0 [sctp_iterator]
4 0 0 0 DL idle 0xa000000000a22000 [mpt_recovery1]
3 0 0 0 DL idle 0xa000000000a0a000 [mpt_recovery0]
16 0 0 0 DL (threaded) [usb]
100039 D - 0xa000000000a06e18 [usbus2]
100038 D - 0xa000000000a06dc0 [usbus2]
100037 D - 0xa000000000a06d68 [usbus2]
100036 D - 0xa000000000a06d10 [usbus2]
100034 D - 0xa0000000009fd460 [usbus1]
100033 D - 0xa0000000009fd408 [usbus1]
100032 D - 0xa0000000009fd3b0 [usbus1]
100031 D - 0xa0000000009fd358 [usbus1]
100029 D - 0xa0000000009f9460 [usbus0]
100028 D - 0xa0000000009f9408 [usbus0]
100027 D - 0xa0000000009f93b0 [usbus0]
100026 D - 0xa0000000009f9358 [usbus0]
15 0 0 0 DL tzpoll 0x9ffc000000bf9670 [acpi_thermal]
2 0 0 0 DL (threaded) [cam]
100051 D ccb_scan 0x9ffc000000bfbf38 [scanner]
100015 D - 0x9ffc000000bfc080 [doneq0]
14 0 0 0 DL - 0x9ffc000000bf98e8 [rand_harvestq]
13 0 0 0 DL (threaded) [geom]
100011 D - 0x9ffc000000bf9e28 [g_down]
100010 D - 0x9ffc000000bf9e20 [g_up]
100009 D - 0x9ffc000000bf9e10 [g_event]
12 0 0 0 WL (threaded) [intr]
100045 I [swi0: uart uart+++]
100044 I [irq29: bge0]
100042 I [irq28: mpt1]
100040 I [irq27: mpt0]
100035 I [irq18: ehci0]
100030 I [irq17: ohci1]
100025 I [irq16: ohci0]
100023 I [swi6: Giant taskq]
100021 I [swi5: fast taskq]
100014 I [swi6: task queue]
100008 I [swi1: netisr 0]
100007 I [swi3: vm]
100006 I [swi4: clock (1)]
100005 I [swi4: clock (0)]
11 0 0 0 RL (threaded) [idle]
100004 CanRun [idle: cpu1]
100003 CanRun [idle: cpu0]
1 0 1 0 SLs wait 0xe000000010b924a0 [init]
10 0 0 0 DL audit_wo 0x9ffc000000f6c888 [audit]
0 0 0 0 RLs (threaded) [kernel]
100073 D - 0xe00000001225b600 [zil_clean]
100075 D - 0xe00000001225a600 [zil_clean]
100081 D - 0xe00000405fb09100 [zil_clean]
100092 D - 0xe00000010471b300 [zil_clean]
100467 D - 0xe00000010471b900 [zil_clean]
100356 D - 0xe00000001225bb00 [zil_clean]
100355 D - 0xe00000001225bc00 [zil_clean]
100354 D - 0xe000000010bbb000 [zil_clean]
100353 D - 0xe000000010bbaf00 [zil_clean]
100352 D - 0xe00000001225bd00 [zil_clean]
100351 D - 0xe000000010bbae00 [zil_clean]
100345 D - 0xe000000010bbac00 [zil_clean]
100339 D - 0xe00000001225c300 [metaslab_group_task]
100338 D - 0xe00000001225c400 [metaslab_group_task]
100337 D - 0xe00000001225c500 [zfs_vn_rele_taskq]
100336 D - 0xe00000001225c600 [metaslab_group_task]
100335 D - 0xe00000001225c700 [metaslab_group_task]
100333 D - 0xe00000001225c800 [zio_ioctl_intr]
100332 D - 0xe00000001225c900 [zio_ioctl_issue]
100331 D - 0xe00000001225ca00 [zio_claim_intr]
100330 D - 0xe00000001225cb00 [zio_claim_issue]
100329 D - 0xe00000001225cc00 [zio_free_intr]
100328 D - 0xe00000001225cd00 [zio_free_issue_7_11]
100327 D - 0xe00000001225cd00 [zio_free_issue_7_10]
100326 D - 0xe00000001225cd00 [zio_free_issue_7_9]
100325 D - 0xe00000001225cd00 [zio_free_issue_7_8]
100324 D - 0xe00000001225cd00 [zio_free_issue_7_7]
100323 D - 0xe00000001225cd00 [zio_free_issue_7_6]
100322 D - 0xe00000001225cd00 [zio_free_issue_7_5]
100321 D - 0xe00000001225cd00 [zio_free_issue_7_4]
100320 D - 0xe00000001225cd00 [zio_free_issue_7_3]
100319 D - 0xe00000001225cd00 [zio_free_issue_7_2]
100318 D - 0xe00000001225cd00 [zio_free_issue_7_1]
100317 D - 0xe00000001225cd00 [zio_free_issue_7_0]
100316 D - 0xe00000001225ce00 [zio_free_issue_6_11]
100315 D - 0xe00000001225ce00 [zio_free_issue_6_10]
100314 D - 0xe00000001225ce00 [zio_free_issue_6_9]
100313 D - 0xe00000001225ce00 [zio_free_issue_6_8]
100312 D - 0xe00000001225ce00 [zio_free_issue_6_7]
100311 D - 0xe00000001225ce00 [zio_free_issue_6_6]
100310 D - 0xe00000001225ce00 [zio_free_issue_6_5]
100309 D - 0xe00000001225ce00 [zio_free_issue_6_4]
100308 D - 0xe00000001225ce00 [zio_free_issue_6_3]
100307 D - 0xe00000001225ce00 [zio_free_issue_6_2]
100306 D - 0xe00000001225ce00 [zio_free_issue_6_1]
100305 D - 0xe00000001225ce00 [zio_free_issue_6_0]
100304 D - 0xe00000001225cf00 [zio_free_issue_5_11]
100303 D - 0xe00000001225cf00 [zio_free_issue_5_10]
100302 D - 0xe00000001225cf00 [zio_free_issue_5_9]
100301 D - 0xe00000001225cf00 [zio_free_issue_5_8]
100300 D - 0xe00000001225cf00 [zio_free_issue_5_7]
100299 D - 0xe00000001225cf00 [zio_free_issue_5_6]
100298 D - 0xe00000001225cf00 [zio_free_issue_5_5]
100297 D - 0xe00000001225cf00 [zio_free_issue_5_4]
100296 D - 0xe00000001225cf00 [zio_free_issue_5_3]
100295 D - 0xe00000001225cf00 [zio_free_issue_5_2]
100294 D - 0xe00000001225cf00 [zio_free_issue_5_1]
100293 D - 0xe00000001225cf00 [zio_free_issue_5_0]
100292 D - 0xe00000001225d000 [zio_free_issue_4_11]
100291 D - 0xe00000001225d000 [zio_free_issue_4_10]
100290 D - 0xe00000001225d000 [zio_free_issue_4_9]
100289 D - 0xe00000001225d000 [zio_free_issue_4_8]
100288 D - 0xe00000001225d000 [zio_free_issue_4_7]
100287 D - 0xe00000001225d000 [zio_free_issue_4_6]
100286 D - 0xe00000001225d000 [zio_free_issue_4_5]
100285 D - 0xe00000001225d000 [zio_free_issue_4_4]
100284 D - 0xe00000001225d000 [zio_free_issue_4_3]
100283 D - 0xe00000001225d000 [zio_free_issue_4_2]
100282 D - 0xe00000001225d000 [zio_free_issue_4_1]
100281 D - 0xe00000001225d000 [zio_free_issue_4_0]
100280 D - 0xe00000001225d100 [zio_free_issue_3_11]
100279 D - 0xe00000001225d100 [zio_free_issue_3_10]
100278 D - 0xe00000001225d100 [zio_free_issue_3_9]
100277 D - 0xe00000001225d100 [zio_free_issue_3_8]
100276 D - 0xe00000001225d100 [zio_free_issue_3_7]
100275 D - 0xe00000001225d100 [zio_free_issue_3_6]
100274 D - 0xe00000001225d100 [zio_free_issue_3_5]
100273 D - 0xe00000001225d100 [zio_free_issue_3_4]
100272 D - 0xe00000001225d100 [zio_free_issue_3_3]
100271 D - 0xe00000001225d100 [zio_free_issue_3_2]
100270 D - 0xe00000001225d100 [zio_free_issue_3_1]
100269 D - 0xe00000001225d100 [zio_free_issue_3_0]
100268 D - 0xe00000001225d200 [zio_free_issue_2_11]
100267 D - 0xe00000001225d200 [zio_free_issue_2_10]
100266 D - 0xe00000001225d200 [zio_free_issue_2_9]
100265 D - 0xe00000001225d200 [zio_free_issue_2_8]
100264 D - 0xe00000001225d200 [zio_free_issue_2_7]
100263 D - 0xe00000001225d200 [zio_free_issue_2_6]
100262 D - 0xe00000001225d200 [zio_free_issue_2_5]
100261 D - 0xe00000001225d200 [zio_free_issue_2_4]
100260 D - 0xe00000001225d200 [zio_free_issue_2_3]
100259 D - 0xe00000001225d200 [zio_free_issue_2_2]
100258 D - 0xe00000001225d200 [zio_free_issue_2_1]
100257 D - 0xe00000001225d200 [zio_free_issue_2_0]
100256 D - 0xe00000001225d300 [zio_free_issue_1_11]
100255 D - 0xe00000001225d300 [zio_free_issue_1_10]
100254 D - 0xe00000001225d300 [zio_free_issue_1_9]
100253 D - 0xe00000001225d300 [zio_free_issue_1_8]
100252 D - 0xe00000001225d300 [zio_free_issue_1_7]
100251 D - 0xe00000001225d300 [zio_free_issue_1_6]
100250 D - 0xe00000001225d300 [zio_free_issue_1_5]
100249 D - 0xe00000001225d300 [zio_free_issue_1_4]
100248 D - 0xe00000001225d300 [zio_free_issue_1_3]
100247 D - 0xe00000001225d300 [zio_free_issue_1_2]
100246 D - 0xe00000001225d300 [zio_free_issue_1_1]
100245 D - 0xe00000001225d300 [zio_free_issue_1_0]
100244 D - 0xe000000010bbab00 [zio_free_issue_0_11]
100243 D - 0xe000000010bbab00 [zio_free_issue_0_10]
100242 D - 0xe000000010bbab00 [zio_free_issue_0_9]
100241 D - 0xe000000010bbab00 [zio_free_issue_0_8]
100240 D - 0xe000000010bbab00 [zio_free_issue_0_7]
100239 D - 0xe000000010bbab00 [zio_free_issue_0_6]
100238 D - 0xe000000010bbab00 [zio_free_issue_0_5]
100237 D - 0xe000000010bbab00 [zio_free_issue_0_4]
100236 D - 0xe000000010bbab00 [zio_free_issue_0_3]
100235 D - 0xe000000010bbab00 [zio_free_issue_0_2]
100234 D - 0xe000000010bbab00 [zio_free_issue_0_1]
100233 D - 0xe000000010bbab00 [zio_free_issue_0_0]
100232 D - 0xe000000010bbaa00 [zio_write_intr_high]
100231 D - 0xe000000010bbaa00 [zio_write_intr_high]
100230 D - 0xe000000010bbaa00 [zio_write_intr_high]
100229 D - 0xe000000010bbaa00 [zio_write_intr_high]
100228 D - 0xe000000010bbaa00 [zio_write_intr_high]
100227 D - 0xe000000010bba900 [zio_write_intr_7]
100226 D - 0xe000000010bba900 [zio_write_intr_6]
100225 D - 0xe000000010bba900 [zio_write_intr_5]
100224 D - 0xe000000010bba900 [zio_write_intr_4]
100223 D - 0xe000000010bba900 [zio_write_intr_3]
100222 D - 0xe000000010bba900 [zio_write_intr_2]
100221 D - 0xe000000010bba900 [zio_write_intr_1]
100220 D - 0xe000000010bba900 [zio_write_intr_0]
100219 D - 0xe000000010bba800 [zio_write_issue_hig]
100218 D - 0xe000000010bba800 [zio_write_issue_hig]
100217 D - 0xe000000010bba800 [zio_write_issue_hig]
100216 D - 0xe000000010bba800 [zio_write_issue_hig]
100215 D - 0xe000000010bba800 [zio_write_issue_hig]
100214 D - 0xe000000010bba700 [zio_write_issue]
100213 Run CPU 1 [zio_read_intr]
100212 D - 0xe000000010bba500 [zio_read_issue_7]
100211 D - 0xe000000010bba500 [zio_read_issue_6]
100210 D - 0xe000000010bba500 [zio_read_issue_5]
100209 D - 0xe000000010bba500 [zio_read_issue_4]
100208 D - 0xe000000010bba500 [zio_read_issue_3]
100207 D - 0xe000000010bba500 [zio_read_issue_2]
100206 D - 0xe000000010bba500 [zio_read_issue_1]
100205 D - 0xe000000010bba500 [zio_read_issue_0]
100204 D - 0xe000000010bba400 [zio_null_intr]
100203 D - 0xe000000010bba300 [zio_null_issue]
100201 D - 0xe000000010bb9d00 [metaslab_group_task]
100200 D - 0xe000000010bb9e00 [metaslab_group_task]
100065 D - 0xe00000001225db00 [system_taskq_1]
100064 D - 0xe00000001225db00 [system_taskq_0]
100050 D - 0xe000000010bbbb00 [CAM taskq]
100046 D - 0x9ffc000000c2e7d1 [deadlkres]
100022 D - 0xe000000010bbb500 [thread taskq]
100020 D - 0xe000000010bbb700 [kqueue taskq]
100019 D - 0xe000000010bbb800 [ffs_trim taskq]
100018 D - 0xe000000010bbb900 [acpi_task_2]
100017 D - 0xe000000010bbb900 [acpi_task_1]
100016 D - 0xe000000010bbb900 [acpi_task_0]
100012 D - 0xe000000010bbbd00 [firmware taskq]
100000 D swapin 0x9ffc000000c22388 [swapper]
db:0:kdb.enter.panic> alltrace
Tracing command c++ pid 80756 tid 101342 td 0xe000000107665b00
ia64_ih_stop(0x9ffc00000096cb40, 0x40b) at ia64_ih_stop+0x80
ia64_handle_intr(0xa000000090749400) at ia64_handle_intr+0x110
ivt_External_Interrupt() at ivt_External_Interrupt+0x30
Tracing command sh pid 80740 tid 100405 td 0xe000000012a52000
cpu_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c814a8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012a52000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000104544de0, 0x6c, 0xe000000012a52378, 0xe000000012a52000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000104544de0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000104544de0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000104544de0, 0xe000000104544ee0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012a52000, 0x7, 0x0, 0xa0000000905a9390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012a52000, 0xffffffff, 0xa0000000905a9390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012a52000, 0xa0000000905a94e8, 0xa0000000905a9390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000104544de0, 0x7fffffffffffb2f0, 0x0, 0xe000000012a52000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command as pid 80736 tid 100390 td 0xe000000018be8900
cpu_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c818e0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000018be8900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe00000001235d810, 0x5c, 0xe000000018be8c78, 0xe000000018be8900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe00000001235d810, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe00000001235d810, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe00000001235d810, 0xe00000001235da50, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012360f24, 0xa000000090559360, 0xa000000090559378, 0x0, 0xe00000001235d918) at pipe_read+0x740
dofileread(0xe000000018be8900, 0x0, 0xe000000012360f00, 0xa000000090559360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000018be8900, 0x0, 0xa000000090559360) at kern_readv+0xa0
sys_read(0xe000000018be8900, 0xa0000000905594e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000018bdc000, 0x1208de000, 0x2000, 0xe000000018be8900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command cc1plus pid 80735 tid 100357 td 0xe000000012a50d80
cpu_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c81ea8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012a50d80, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040bebaf518, 0x0, 0xe000000012a510f8, 0xe000000012a50d80) at sleepq_switch+0x2e0
sleepq_wait(0xe0000040bebaf518, 0x0, 0x9ffc000000b008c0, 0xe000000012a50d80, 0x9ffc0000003b9520) at sleepq_wait+0xb0
_cv_wait(0xe0000040bebaf518, 0xe0000040bebaf4f8, 0xe000000012a50d80, 0x0) at _cv_wait+0x4d0
zio_wait(0xe0000040bebaf1f8, 0xe0000040bebaf4e8, 0xe0000040bebaf4f8, 0xe0000040bebaf518, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150
dbuf_read(0xe00000408e9383f0, 0xe0000040bebaf1f8, 0xa, 0xa000000090450ea8) at dbuf_read+0xd70
dmu_buf_hold(0xe00000408e9383f0, 0xa, 0x0, 0x0, 0xa000000090450ee0, 0x1) at dmu_buf_hold+0x1d0
zap_lockdir(0xe000000012aab400, 0x5f652, 0x0, 0x1, 0x1, 0x0, 0xa000000090450f20) at zap_lockdir+0x50
zap_lookup_norm(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0x0, 0x0) at zap_lookup_norm+0x60
zap_lookup(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0xa0000000012b3eb0, 0xb1e) at zap_lookup+0x60
zfs_dirent_lock(0xa000000090450f80, 0xe00000001ce0aa10, 0xa000000090450fc0, 0xa000000090450f88, 0x6, 0x0, 0x0, 0xe00000001b934f00) at zfs_dirent_lock+0xbb0
zfs_dirlook(0xe00000001ce0aa10, 0xa000000090450fc0, 0xa0000000904512e8, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
zfs_lookup(0xe0000000155c1110, 0xa000000090450fc0, 0xa0000000904512e8, 0xa000000090451310, 0x0, 0xe000004082290200, 0xe000000012a50d80, 0x0) at zfs_lookup+0x7a0
zfs_freebsd_lookup(0xa0000000904510c0, 0xa000000090451310, 0xa000000090451348) at zfs_freebsd_lookup+0x100
VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510c0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270
vfs_cache_lookup(0xe0000000155c1110, 0x0, 0xa000000090451310) at vfs_cache_lookup+0x1f0
VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510f8, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270
lookup(0xa000000090451288, 0x200000, 0xe0000000155c1110, 0x0, 0xa000000090451318) at lookup+0xe30
namei(0xa000000090451288, 0xe0000000123c2000, 0x9ffc0000008fe220, 0x0, 0xa0000000904512f8) at namei+0xb40
vn_open_cred(0xa000000090451288, 0xa000000090451390, 0x1a4, 0x0, 0xe000004082290200, 0xe00000001233ecd0) at vn_open_cred+0x500
vn_open(0xa000000090451288, 0xa000000090451390, 0x1a4, 0xe00000001233ecd0, 0x9ffc0000005c3ee0, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012a50d80, 0xffffffffffffff9c, 0x1214529e0, 0x0, 0x0, 0x1b6) at kern_openat+0x490
kern_open(0xe000000012a50d80, 0x1214529e0, 0x0, 0x8000, 0x1b6) at kern_open+0x40
sys_open(0xe000000012a50d80, 0xa0000000904514e8, 0x9ffc000000986c80, 0x48d) at sys_open+0x40
syscall(0xe0000000123c2000, 0x8000, 0x1b6, 0xe000000012a50d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command c++ pid 80734 tid 100487 td 0xe000004081208480
cpu_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000004081208480, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040810f1280, 0x6c, 0xe0000040812087f8, 0xe000004081208480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040810f1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040810f1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040810f1280, 0xe0000040810f1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000004081208480, 0x0, 0x13b5f, 0xa000000090711390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000004081208480, 0x13b5f, 0xa000000090711390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000004081208480, 0xa0000000907114e8, 0xa000000090711390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040810f1280, 0x1208060d8, 0x0, 0xe000004081208480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command sleep pid 80733 tid 100414 td 0xe000000104522900
cpu_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c81818, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000104522900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0x9ffc000000c2fa39, 0x6c, 0xe000000104522c78, 0xe000000104522900) at sleepq_switch+0x2e0
sleepq_catch_signals(0x9ffc000000c2fa39, 0x6c, 0x9ffc000000b008c0, 0xe000000104522c90) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0x9ffc000000c2fa39, 0x6c, 0x20000000, 0x9ffc000000462c50, 0x998) at sleepq_timedwait_sig+0x30
_sleep(0x9ffc000000c2fa39, 0x0, 0x6c, 0x9ffc000000afaba8, 0x1, 0x20000000, 0x200) at _sleep+0x750
kern_nanosleep(0xe000000104522900, 0x200000000, 0xa0000000905f1380) at kern_nanosleep+0x2a0
sys_nanosleep(0xe000000104522900, 0xa0000000905f14e8, 0x0, 0x9ffc000000986c80) at sys_nanosleep+0xb0
syscall(0xe00000010452f280, 0x7fffffffffffeb40, 0x7fffffffffffeb10, 0xe000000104522900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command wc pid 76333 tid 100374 td 0xe000000012a52900
cpu_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c81750, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012a52900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe00000001235ce70, 0x5c, 0xe000000012a52c78, 0xe000000012a52900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe00000001235ce70, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe00000001235ce70, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe00000001235ce70, 0xe00000001235d0b0, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe00000001233ff14, 0xa0000000904d9360, 0xa0000000904d9378, 0x0, 0xe00000001235cf78) at pipe_read+0x740
dofileread(0xe000000012a52900, 0x0, 0xe00000001233fef0, 0xa0000000904d9360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012a52900, 0x0, 0xa0000000904d9360) at kern_readv+0xa0
sys_read(0xe000000012a52900, 0xa0000000904d94e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000012e16de0, 0x7ffffffffffeece0, 0x10000, 0xe000000012a52900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command tee pid 76332 tid 100451 td 0xe000004081060000
cpu_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c80d28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000004081060000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000018bed5a8, 0x5c, 0xe000004081060378, 0xe000004081060000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000018bed5a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000018bed5a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000018bed5a8, 0xe000000018bed7e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000018c3aac4, 0xa000000090701360, 0xa000000090701378, 0x0, 0xe000000018bed6b0) at pipe_read+0x740
dofileread(0xe000004081060000, 0x0, 0xe000000018c3aaa0, 0xa000000090701360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000004081060000, 0x0, 0xa000000090701360) at kern_readv+0xa0
sys_read(0xe000004081060000, 0xa0000000907014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe0000040811c6000, 0x120804000, 0x2000, 0xe000004081060000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command find pid 76331 tid 100376 td 0xe000000012e5fb00
cpu_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c80148, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012e5fb00, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000000124db180, 0x0, 0xe000000012e5fe78, 0xe000000012e5fb00) at sleepq_switch+0x2e0
sleepq_wait(0xe0000000124db180, 0x0, 0x9ffc000000b008c0, 0xe000000012e5fb00, 0x9ffc0000003b9520) at sleepq_wait+0xb0
_cv_wait(0xe0000000124db180, 0xe0000000124db160, 0xe000000012e5fb00, 0x0) at _cv_wait+0x4d0
zio_wait(0xe0000000124dae60, 0xe0000000124db150, 0xe0000000124db160, 0xe0000000124db180, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150
dbuf_read(0xe0000001078e07e0, 0xe0000000124dae60, 0x22, 0xa0000000904e8ed8, 0xe0000001078e0800, 0xe0000001078e0830) at dbuf_read+0xd70
dnode_hold_impl(0xe00000001290cc00, 0x4129e, 0x1, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold_impl+0x210
dnode_hold(0xe00000001290cc00, 0x4129e, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold+0x40
dmu_bonus_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10) at dmu_bonus_hold+0x40
sa_buf_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10, 0xa000000001276000, 0xd20) at sa_buf_hold+0x40
zfs_zget(0xe000000012a5a000, 0x4129e, 0xa0000000904e8fa8, 0x8, 0x1, 0xe000000012a5a6a8, 0xa0000000012b3eb0, 0xa00000000137fcd8) at zfs_zget+0x120
zfs_dirent_lock(0xa0000000904e8fa0, 0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e8fa8, 0x6, 0x0, 0x0, 0xe00000001b8d7000) at zfs_dirent_lock+0xc10
zfs_dirlook(0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e9220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
zfs_lookup(0xe000000015ec8270, 0xa0000000904e8fe0, 0xa0000000904e9220, 0xa0000000904e9248, 0x0, 0xe000004082290100, 0xe000000012e5fb00, 0x0) at zfs_lookup+0x7a0
zfs_freebsd_lookup(0xa0000000904e90e0, 0xa0000000904e9248, 0xa0000000904e9280) at zfs_freebsd_lookup+0x100
VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e90e0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270
vfs_cache_lookup(0xe000000015ec8270, 0x0, 0xa0000000904e9248) at vfs_cache_lookup+0x1f0
VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e9118, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270
lookup(0xa0000000904e91c0, 0x200000, 0xe000000015ec8270, 0x0, 0xa0000000904e9250) at lookup+0xe30
namei(0xa0000000904e91c0, 0xa0000000904e91f8, 0x4000104, 0x0, 0xa0000000904e9230, 0xa0000000904e9278, 0xa0000000904e9270, 0xa0000000904e9220) at namei+0xb40
kern_statat_vnhook(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320, 0x0) at kern_statat_vnhook+0xf0
kern_statat(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320) at kern_statat+0x40
sys_fstatat(0xe000000012e5fb00, 0xa0000000904e94e8, 0x9ffc000000986c80, 0x48d) at sys_fstatat+0x50
syscall(0xe000000012e164a0, 0x120873ac8, 0x120873a50, 0xe000000012e5fb00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command sh pid 76330 tid 100394 td 0xe000000012ef7680
cpu_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012ef7680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012ef1280, 0x6c, 0xe000000012ef79f8, 0xe000000012ef7680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012ef1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012ef1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012ef1280, 0xe000000012ef1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012ef7680, 0x7, 0x0, 0xa000000090579390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012ef7680, 0xffffffff, 0xa000000090579390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012ef7680, 0xa0000000905794e8, 0xa000000090579390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012ef1280, 0x7fffffffffffe4d0, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command sh pid 76326 tid 100386 td 0xe000000012e5e000
cpu_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c81c28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012e5e000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040810075a8, 0x5c, 0xe000000012e5e378, 0xe000000012e5e000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040810075a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040810075a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040810075a8, 0xe0000040810077e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000018c3a2f4, 0xa000000090539360, 0xa000000090539378, 0x0, 0xe0000040810076b0) at pipe_read+0x740
dofileread(0xe000000012e5e000, 0x3, 0xe000000018c3a2d0, 0xa000000090539360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012e5e000, 0x3, 0xa000000090539360) at kern_readv+0xa0
sys_read(0xe000000012e5e000, 0xa0000000905394e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000018bdd280, 0x7fffffffffffe800, 0x80, 0xe000000012e5e000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command mail pid 64479 tid 100398 td 0xe000000012ef6900
cpu_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c81098, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012ef6900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000018bed340, 0x5c, 0xe000000012ef6c78, 0xe000000012ef6900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000018bed340, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000018bed340, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000018bed340, 0xe000000018bed580, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000004069110d44, 0xa000000090401360, 0xa000000090401378, 0x0, 0xe000000018bed448) at pipe_read+0x740
dofileread(0xe000000012ef6900, 0x0, 0xe000004069110d20, 0xa000000090401360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012ef6900, 0x0, 0xa000000090401360) at kern_readv+0xa0
sys_read(0xe000000012ef6900, 0xa0000000904014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000012ef04a0, 0x12080e000, 0x2000, 0xe000000012ef6900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command sh pid 64478 tid 100447 td 0xe0000001045b5680
cpu_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c81f70, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000001045b5680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040811c7280, 0x6c, 0xe0000001045b59f8, 0xe0000001045b5680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040811c7280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040811c7280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040811c7280, 0xe0000040811c7380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000001045b5680, 0x7, 0x0, 0xa0000000906e1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000001045b5680, 0xffffffff, 0xa0000000906e1390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000001045b5680, 0xa0000000906e14e8, 0xa0000000906e1390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040811c7280, 0x7fffffffffffe6e0, 0x0, 0xe0000001045b5680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
Tracing command sh pid 64471 tid 100401 td 0xe000000104523680
Tracing command lockf pid 64470 tid 100359 td 0xe000000012a50480
Tracing command sh pid 64468 tid 100364 td 0xe000000012a26900
Tracing command sh pid 64467 tid 101341 td 0xe000000107665200
Tracing command mail pid 64204 tid 100446 td 0xe0000001045b5b00
Tracing command sh pid 64203 tid 100421 td 0xe000000104522480
Tracing command sh pid 64196 tid 100385 td 0xe000000012e5e480
Tracing command lockf pid 64195 tid 100372 td 0xe000000012a53200
Tracing command sh pid 64193 tid 100395 td 0xe000000012ef7200
Tracing command cron pid 64191 tid 100488 td 0xe000004081208000
Tracing command make pid 47211 tid 100407 td 0xe000000012ee9680
Tracing command make pid 92677 tid 100412 td 0xe000000104522d80
Tracing command sh pid 92676 tid 100382 td 0xe000000012ee8480
Tracing command make pid 92652 tid 100422 td 0xe000000104522000
Tracing command sh pid 92650 tid 100388 td 0xe000000012a27680
Tracing command tee pid 59715 tid 100419 td 0xe00000010458e480
Tracing command sh pid 59384 tid 100363 td 0xe000000012a26d80
Tracing command gmake pid 15731 tid 100434 td 0xe000000104591200
Tracing command gmake pid 87774 tid 100371 td 0xe000000012a53680
Tracing command sh pid 87772 tid 100438 td 0xe000004080a2fb00
Tracing command make pid 87688 tid 100433 td 0xe000000104591680
Tracing command sh pid 87686 tid 100420 td 0xe00000010458e000
Tracing command tee pid 63399 tid 100387 td 0xe000000012a27b00
Tracing command sh pid 63244 tid 100365 td 0xe000000012a26480
Tracing command sh pid 49525 tid 100361 td 0xe000000012613b00
Tracing command sh pid 3381 tid 100383 td 0xe000000012ee8000
Tracing command sendmail pid 593 tid 100368 td 0xe000000012a99680
Tracing command sh pid 588 tid 100360 td 0xe000000012a50000
Tracing command cron pid 587 tid 100445 td 0xe000000107664000
Tracing command getty pid 875 tid 100377 td 0xe000000012e5f680
Tracing command getty pid 844 tid 100059 td 0xe000000011123680
Tracing command inetd pid 821 tid 100370 td 0xe000000012a53b00
Tracing command cron pid 786 tid 100344 td 0xe00000001243ed80
Tracing command sendmail pid 782 tid 100367 td 0xe000000012a99b00
Tracing command sendmail pid 779 tid 100049 td 0xe0000000121da480
Tracing command sshd pid 776 tid 100048 td 0xe0000000121da900
Tracing command nginx pid 723 tid 100060 td 0xe000000011123200
Tracing command nginx pid 721 tid 100061 td 0xe000000011122d80
Tracing command ntpd pid 702 tid 100362 td 0xe000000012a27200
Tracing command casperd pid 642 tid 100343 td 0xe00000001243f200
Tracing command casperd pid 641 tid 100358 td 0xe000000012a50900
Tracing command syslogd pid 612 tid 100369 td 0xe000000012a99200
Tracing command devd pid 486 tid 100062 td 0xe0000000123be480
Tracing command adjkerntz pid 135 tid 100366 td 0xe000000012a26000
Tracing command zfskern pid 40 tid 100341 td 0xe000000012612480
Tracing command zfskern pid 40 tid 100340 td 0xe000000012612900
Tracing command zfskern pid 40 tid 100334 td 0xe000000012438d80
Tracing command zfskern pid 40 tid 100067 td 0xe0000000123b8d80
Tracing command zfskern pid 40 tid 100066 td 0xe0000000123b9200
Tracing command softdepflush pid 21 tid 100058 td 0xe0000000122b8000
Tracing command vnlru pid 20 tid 100057 td 0xe0000000122b8480
Tracing command syncer pid 19 tid 100056 td 0xe0000000122b8900
Tracing command bufdaemon pid 18 tid 100055 td 0xe0000000122b8d80
Tracing command pagezero pid 17 tid 100054 td 0xe0000000122b9200
Tracing command vmdaemon pid 9 tid 100053 td 0xe0000000122b9680
Tracing command pagedaemon pid 8 tid 100052 td 0xe0000000122b9b00
Tracing command sctp_iterator pid 5 tid 100047 td 0xe0000000121dad80
Tracing command mpt_recovery1 pid 4 tid 100043 td 0xe000000010d06d80
Tracing command mpt_recovery0 pid 3 tid 100041 td 0xe000000010d07680
Tracing command usb pid 16 tid 100039 td 0xe000000011122000
Tracing command usb pid 16 tid 100038 td 0xe000000011122480
Tracing command usb pid 16 tid 100037 td 0xe000000011122900
Tracing command usb pid 16 tid 100036 td 0xe000000010c33b00
Tracing command usb pid 16 tid 100034 td 0xe000000010d04480
Tracing command usb pid 16 tid 100033 td 0xe000000010d04900
Tracing command usb pid 16 tid 100032 td 0xe000000010d04d80
Tracing command usb pid 16 tid 100031 td 0xe000000010d05200
Tracing command usb pid 16 tid 100029 td 0xe000000010d05b00
Tracing command usb pid 16 tid 100028 td 0xe000000010d06000
Tracing command usb pid 16 tid 100027 td 0xe000000010bb1200
Tracing command usb pid 16 tid 100026 td 0xe000000010bb1680
Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000010c32000
Tracing command cam pid 2 tid 100051 td 0xe000000011123b00
Tracing command cam pid 2 tid 100015 td 0xe000000010b97680
Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000010bb0000
Tracing command geom pid 13 tid 100011 td 0xe000000010bb0900
Tracing command geom pid 13 tid 100010 td 0xe000000010bb0d80
Tracing command geom pid 13 tid 100009 td 0xe000000010b94000
Tracing command intr pid 12 tid 100045 td 0xe000000010d06480
Tracing command intr pid 12 tid 100044 td 0xe000000010d06900
Tracing command intr pid 12 tid 100042 td 0xe000000010d07200
Tracing command intr pid 12 tid 100040 td 0xe000000010d07b00
Tracing command intr pid 12 tid 100035 td 0xe000000010d04000
Tracing command intr pid 12 tid 100030 td 0xe000000010d05680
Tracing command intr pid 12 tid 100025 td 0xe000000010bb1b00
Tracing command intr pid 12 tid 100023 td 0xe000000010c3248
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201403251230.s2PCU1d6016335>