From owner-freebsd-ia64@FreeBSD.ORG Tue Mar 25 12:30:02 2014 Return-Path: Delivered-To: freebsd-ia64@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 11042643 for ; Tue, 25 Mar 2014 12:30:02 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F101739C for ; Tue, 25 Mar 2014 12:30:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2PCU19H016336 for ; Tue, 25 Mar 2014 12:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2PCU1d6016335; Tue, 25 Mar 2014 12:30:01 GMT (envelope-from gnats) Date: Tue, 25 Mar 2014 12:30:01 GMT Message-Id: <201403251230.s2PCU1d6016335@freefall.freebsd.org> To: freebsd-ia64@FreeBSD.org Cc: From: Anton Shterenlikht Subject: ia64/183227: panic: uma_zfree: Freeing to non free bucket index. X-BeenThere: freebsd-ia64@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: Anton Shterenlikht List-Id: Porting FreeBSD to the IA-64 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 12:30:02 -0000 The following reply was made to PR ia64/183227; it has been noted by GNATS. From: Anton Shterenlikht To: bug-followup@freebsd.org Cc: Subject: ia64/183227: panic: uma_zfree: Freeing to non free bucket index. Date: Tue, 25 Mar 2014 05:15:42 -0700 (PDT) This seems to be the same panic, but now in r262690. # cat /var/crash/info.last Dump header from device /dev/da0p3 Architecture: ia64 Architecture Version: 1 Dump Length: 73728B (0 MB) Blocksize: 512 Dumptime: Tue Mar 25 03:39:36 2014 Hostname: mech-as221.men.bris.ac.uk Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI Panic String: uma_zfree: Freeing to non free bucket index. Dump Parity: 730754683 Bounds: 3 Dump Status: good # # cat /var/crash/panic.txt uma_zfree: Freeing to non free bucket index. # cat /var/crash/version.txt FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI # ************************************* msgbuf.txt ************************************* GDB: debug ports: uart GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014 root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. CPU: Madison (1500 MHz Itanium 2) Origin = "GenuineIntel" Revision = 5 Features = 0x1 real memory = 6442450944 (6144 MB) avail memory = 6291316736 (5999 MB) FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP) cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0 random: initialized Event timer "ITC" frequency 1500000000 Hz quality 1000 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603) ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603) acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) cpu0: on acpi0 cpu1: on acpi0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0 acpi_tz0: on acpi0 pcib0: on acpi0 pcib0: could not evaluate _ADR - AE_NOT_FOUND pci0: on pcib0 ohci0: mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0 usbus0 on ohci0 ohci1: mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0 usbus1 on ohci1 ehci0: mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0 usbus2: EHCI version 0.95 usbus2 on ehci0 pci0: at device 2.0 (no driver attached) pci0: at device 3.0 (no driver attached) pcib1: on acpi0 pcib1: could not evaluate _ADR - AE_NOT_FOUND pci32: on pcib1 mpt0: port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32 mpt0: MPI Version=1.2.12.0 mpt1: port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32 mpt1: MPI Version=1.2.12.0 bge0: mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32 bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:11:0a:31:46:06 pcib2: on acpi0 pcib2: could not evaluate _ADR - AE_NOT_FOUND pci64: on pcib2 pcib3: on acpi0 pcib3: could not evaluate _ADR - AE_NOT_FOUND pci96: on pcib3 pcib4: on acpi0 pcib4: could not evaluate _ADR - AE_NOT_FOUND pci128: on pcib4 pcib5: on acpi0 pcib5: could not evaluate _ADR - AE_NOT_FOUND pci192: on pcib5 pcib6: on acpi0 pcib6: could not evaluate _ADR - AE_NOT_FOUND pci224: on pcib6 uart0: mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224 puc0: mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224 uart1: at port 1 on puc0 uart1: console (9600,n,8,1) uart2: at port 3 on puc0 vgapci0: port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224 vgapci0: Boot video device uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0 uart4: debug port (9600,n,8,1) Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 ugen1.1: at usbus1 uhub0: on usbus0 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 uhub1: 2 ports with 2 removable, self powered uhub0: 3 ports with 3 removable, self powered random: unblocking device. uhub2: 5 ports with 5 removable, self powered da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: Serial Number X1015960 0613 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: Serial Number 3KP345CH da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit) da1: Command Queueing enabled da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C) da2 at mpt1 bus 0 scbus1 target 2 lun 0 da2: Fixed Direct Access SCSI-3 device da2: Serial Number 3KP0WEXZ da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit) da2: Command Queueing enabled da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0p2 [rw]... WARNING: / was not properly dismounted <118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a. <118>Setting hostid: 0x688171f3. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) <118>Starting ddb. <118>Entropy harvesting: interrupts ethernet point_to_point swi. <118>Starting file system checks: <118>/dev/da0p2: UNREF FILE I=6288401 OWNER=root MODE=10644 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288429 OWNER=root MODE=10644 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288431 OWNER=root MODE=10644 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014 (CLEARED) <118>/dev/da0p2: UNREF FILE I=6288432 OWNER=root MODE=10644 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014 (CLEARED) <118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0p2: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0p2: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0p2: 761703 files, 11337625 used, 19128012 free (156372 frags, 2371455 blocks, 0.5% fragmentation) <118>Mounting local file systems:. <118>Writing entropy file:. <118>Setting hostname: mech-as221.men.bris.ac.uk. <118>Starting Network: lo0 bge0. <118>lo0: flags=8049 metric 0 mtu 16384 <118> options=600003 <118> inet6 ::1 prefixlen 128 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 <118> inet 127.0.0.1 netmask 0xff000000 <118> nd6 options=21 <118>bge0: flags=8843 metric 0 mtu 1500 <118> options=8009b <118> ether 00:11:0a:31:46:06 <118> inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255 <118> inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x1 <118> nd6 options=29 <118> media: Ethernet autoselect (none) <118> status: no carrier <118>Starting devd. <118>Starting pflogd: <118>add net default: gateway 137.222.187.250 <118>add net fe80::: gateway ::1 <118>add net ff02::: gateway ::1 <118>add net ::ffff:0.0.0.0: gateway ::1 <118>add net ::0.0.0.0: gateway ::1 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc47 /usr/local/lib/gcc48 /usr/local/lib/graphviz <118>Creating and/or trimming log files. <118>Starting syslogd. <118>savecore: reboot <118>Mar 21 14:10:17 mech-as221 savecore: reboot <118>savecore: writing core to /var/crash/textdump.tar.2 <118>/var/crash/vmcore.2 not found <118>Starting casperd. <118>Clearing /tmp (X related). <118>Recovering vi editor sessions:. <118>Updating motd:. <118>Mounting late file systems:. <118>Starting ntpd. <118>Performing sanity check on nginx configuration: <118>nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok <118>nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful <118>Starting nginx. <118>Starting bsdstats. <118>Posting monthly OS statistics to rpt.bsdstats.org <118>Performing sanity check on sshd configuration. <118>Starting sshd. <118>Starting sendmail_submit. <118>Starting sendmail_msp_queue. <118>Starting cron. <118>Local package initialization:cd: /var/portbuild/: No such file or directory <118>. <118>Starting inetd. <118>Starting background file system checks in 60 seconds. <118> <118>Fri Mar 21 14:10:22 GMT 2014 <118>Mar 21 14:11:03 mech-as221 login: ROOT LOGIN (root) ON ttyu1 lock order reversal: 1st 0xa00000005f6bf058 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081 2nd 0xe000000015952400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b31b08) at _witness_debugger+0x60 witness_checkorder(0xe000000015952400, 0x9ffc000000b03808, 0x9ffc000000b31b08, 0x11c, 0x0) at witness_checkorder+0x15c0 _sx_xlock(0xe000000015952400, 0x0, 0x9ffc000000b31b08, 0x11c) at _sx_xlock+0x120 ufsdirhash_acquire(0xe0000000162cdb90) at ufsdirhash_acquire+0x70 ufsdirhash_newblk(0xe0000000162cdb90, 0xa200, 0x0, 0x9ffc0000008d59a0, 0xda1, 0xa000000090471030) at ufsdirhash_newblk+0x20 ufs_direnter(0xe0000000162f29c0, 0x0, 0xa000000090471068, 0xa000000090471310, 0x0, 0x1) at ufs_direnter+0x530 ufs_rename(0xa000000090471240, 0xa000000090471068, 0x0) at ufs_rename+0x1590 VOP_RENAME_APV(0x9ffc000000bc2740, 0xa000000090471368, 0x0, 0x0, 0x9ffc0000005c1f10, 0x797, 0x9ffc000000dd1720) at VOP_RENAME_APV+0x230 kern_renameat(0xa000000090471310, 0xa000000090471368, 0x0, 0xffffffffffffff9c, 0x12210c238, 0x0) at kern_renameat+0x630 kern_rename(0xe000000012613b00, 0x12210c1d8, 0x12210c238, 0x0) at kern_rename+0x40 sys_rename(0xe000000012613b00, 0xa0000000904714e8, 0x9ffc000000986c80, 0x48d) at sys_rename+0x30 syscall(0xe000000012ade940, 0x12210c238, 0x12210c108, 0xe000000012613b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return lock order reversal: 1st 0xe0000000129d47b8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851 2nd 0xe000000016c63658 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60 witness_checkorder(0xe000000016c63658, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe000000016c63720, 0x9ffc000000ae1498) at witness_checkorder+0x15c0 __lockmgr_args(0xe000000016c63658, 0x80500, 0xe000000016c63720, 0x9ffc000000ae1498, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0 vop_stdlock(0xa0000000905790e0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80 VOP_LOCK1_APV(0x9ffc000000bad388, 0xa0000000905790c0, 0xe000000012ef7680) at VOP_LOCK1_APV+0x230 _vn_lock(0xe000000016c635f0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130 vget(0xe000000016c635f0, 0x80500, 0xe000000012ef7680, 0x9ffc000000242ca0) at vget+0x100 devfs_allocv(0xe000000103d57300, 0xe000000012a683c8, 0x80500, 0xa000000090579110) at devfs_allocv+0x1b0 devfs_root(0xe000000012a683c8, 0x80000, 0xa000000090579160, 0xe00000001b080190, 0x9ffc00000059dea0, 0x17b7, 0x8) at devfs_root+0x60 vfs_donmount(0xe000000012ef7680, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50 sys_nmount(0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0 syscall(0xe000000012ef1280, 0x8, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return lock order reversal: 1st 0xe000000016c47658 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851 2nd 0xe000000016c562d8 ufs (ufs) @ /usr/src/sys/fs/nullfs/null_vnops.c:620 KDB: stack backtrace: db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000ae6b28) at _witness_debugger+0x60 witness_checkorder(0xe000000016c562d8, 0x9ffc000000b03808, 0x9ffc000000ae6b28, 0x26c, 0xe000000016c563a0, 0x9ffc000000aecf70) at witness_checkorder+0x15c0 __lockmgr_args(0xe000000016c562d8, 0x80500, 0xe000000016c563a0, 0x9ffc000000aecf70, 0x60, 0x33, 0x9ffc000000ae6b28, 0x26c) at __lockmgr_args+0x10a0 ffs_lock(0xa0000000904d10e0, 0xe000000016c562d8, 0x80500) at ffs_lock+0x160 VOP_LOCK1_APV(0x9ffc000000bc1e00, 0xa0000000904d10c0, 0x9ffc00000034ebd0, 0x0) at VOP_LOCK1_APV+0x230 null_lock(0xa0000000904d10f0, 0x9ffc000000ae6b28, 0xe000000016c65860) at null_lock+0x1a0 VOP_LOCK1_APV(0x9ffc000000b8aa18, 0xa0000000904d10f0, 0x2) at VOP_LOCK1_APV+0x230 _vn_lock(0xe000000016c65860, 0x80400, 0x9ffc000000ae6a38, 0x118) at _vn_lock+0x130 nullfs_root(0xe000000012a66a28, 0x80000, 0xa0000000904d1160, 0xe000000016c65860, 0x9ffc00000059dea0, 0x17b7, 0x8) at nullfs_root+0x90 vfs_donmount(0xe000000012a52d80, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50 sys_nmount(0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0 syscall(0xe000000012e17280, 0x8, 0x0, 0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return <118>Mar 24 08:49:56 mech-as221 su: BAD SU mexas to root on /dev/pts/0 <118>Mar 24 08:50:01 mech-as221 su: mexas to root on /dev/pts/0 <118>Mar 24 11:14:02 mech-as221 su: mexas to root on /dev/pts/0 <118>Mar 24 13:19:28 mech-as221 su: mexas to root on /dev/pts/0 lock order reversal: 1st 0xe0000000170162d8 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1720 2nd 0xe0000040876f6548 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60 witness_checkorder(0xe0000040876f6548, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe0000040876f6610, 0xa000000001362370) at witness_checkorder+0x15c0 __lockmgr_args(0xe0000040876f6548, 0x80500, 0xe0000040876f6610, 0xa000000001362370, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0 vop_stdlock(0xa00000008ff0f4d0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80 VOP_LOCK1_APV(0x9ffc000000bad388, 0xa00000008ff0f4b0, 0x9ffc000000f0d470) at VOP_LOCK1_APV+0x230 _vn_lock(0xe0000040876f64e0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130 vget(0xe0000040876f64e0, 0x80500, 0xe0000000122b8900, 0x0) at vget+0x100 vfs_msync(0xe0000001076c52e8, 0x4, 0xe0000040876f64e0, 0x9ffc0000005b4dd0) at vfs_msync+0x120 sync_fsync(0x9ffc000000b0e198, 0x9ffc000000f65a08, 0x240000) at sync_fsync+0x210 VOP_FSYNC_APV(0x9ffc000000badcc8, 0xa00000008ff0f528, 0x0, 0x9ffc000000b42028, 0x9ffc0000005b6390) at VOP_FSYNC_APV+0x270 sched_sync(0xe000000017016278, 0xe0000000170163d8, 0xe000000017016270, 0xe0000000121cfd00) at sched_sync+0x650 fork_exit(0x9ffc000000b4bcb0, 0x0, 0xa00000008ff0f550) at fork_exit+0x120 enter_userland() at enter_userland panic: uma_zfree: Freeing to non free bucket index. cpuid = 1 KDB: stack backtrace: db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc000000450850, 0x40c, 0x9ffc000000dd1720) at kdb_backtrace+0xc0 vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x9ffc000000c2dcd0, 0x9ffc000000bfa490) at vpanic+0x260 kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120 uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0 free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150 zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30 zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0 zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170 zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580 zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0 zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290 zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50 zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0 taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50 taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0 taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130 fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120 enter_userland() at enter_userland KDB: enter: panic ****************************** ddb.txt ****************************** db:0:kdb.enter.panic> run lockinfo db:1:lockinfo> show locks db:1:locks> show alllocks Process 80735 (cc1plus) thread 0xe000000012a50d80 (100357) shared lockmgr zfs (zfs) r = 0 (0xe0000000155c1178) locked @ /usr/src/sys/kern/vfs_subr.c:2101 Process 76331 (find) thread 0xe000000012e5fb00 (100376) shared lockmgr zfs (zfs) r = 0 (0xe000000015ec82d8) locked @ /usr/src/sys/kern/vfs_lookup.c:518 db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.panic> show pcpu cpuid = 1 dynamic pcpu = 0x400005fc29800 curthread = 0xe0000000121dbb00: pid 0 "zio_read_intr" curpcb = 0 fpcurthread = none idlethread = 0xe000000010b95680: tid 100004 "idle: cpu1" MD: vhpt = 0xe00000003f100000 MD: lid = 0x1000000 MD: clock = 0x1a4179a98ef4b MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0xe000000010b6ea30 MD: pmap = 0x9ffc000000c22dc0 spin locks held: db:0:kdb.enter.panic> bt Tracing pid 0 tid 100213 td 0xe0000000121dbb00 kdb_enter(0x9ffc000000af8ed8, 0x9ffc000000af8ed8, 0x9ffc0000004508a0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x100, 0x9ffc000000bfa490) at vpanic+0x2b0 kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120 uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0 free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150 zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30 zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0 zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170 zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580 zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0 zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290 zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50 zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0 taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50 taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0 taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130 fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120 enter_userland() at enter_userland db:0:kdb.enter.panic> ps pid ppid pgrp uid state wmesg wchan cmd 80756 80740 588 0 RJ CPU 0 c++ 80740 15731 588 0 SJ wait 0xe000000104544de0 sh 80736 80734 588 0 SJ piperd 0xe00000001235d810 as 80735 80734 588 0 DJ zio->io_ 0xe0000040bebaf518 cc1plus 80734 47211 588 0 SJ wait 0xe0000040810f1280 c++ 80733 49525 588 0 S nanslp 0x9ffc000000c2fa39 sleep 76333 76330 64193 0 S piperd 0xe00000001235ce70 wc 76332 76330 64193 0 S piperd 0xe000000018bed5a8 tee 76331 76330 64193 0 D zio->io_ 0xe0000000124db180 find 76330 76326 64193 0 S wait 0xe000000012ef1280 sh 76326 64478 64193 0 S piperd 0xe0000040810075a8 sh 64479 64471 64193 0 S piperd 0xe000000018bed340 mail 64478 64471 64193 0 S wait 0xe0000040811c7280 sh 64471 64470 64193 0 SW wait 0xe0000001045384a0 sh 64470 64468 64193 0 SW wait 0xe0000000122b7280 lockf 64468 64467 64193 0 SW wait 0xe000000012add720 sh 64467 64203 64193 0 SW wait 0xe000000178f0c940 sh 64204 64196 64193 0 S piperd 0xe00000001235d0d8 mail 64203 64196 64193 0 SW wait 0xe00000010452ede0 sh 64196 64195 64193 0 SW wait 0xe000000018bdd720 sh 64195 64193 64193 0 SW wait 0xe000000012e17720 lockf 64193 64191 64193 0 SWs wait 0xe000000012ef0de0 sh 64191 786 786 0 S piperd 0xe000000012e06738 cron 47211 92677 588 0 SJ wait 0xe0000001045444a0 make 92677 92676 588 0 SWJ wait 0xe00000010452f720 make 92676 92652 588 0 SWJ wait 0xe000000012ef44a0 sh 92652 92650 588 0 SWJ wait 0xe00000010452e940 make 92650 59384 588 0 SW wait 0xe000000018bdc940 sh 59715 59384 588 0 S piperd 0xe000000012e070d8 tee 59384 3381 588 0 S select 0xe0000000123ec4c0 sh 15731 87774 588 0 SJ wait 0xe000004080a1ade0 gmake 87774 87772 588 0 SWJ wait 0xe0000000122b0000 gmake 87772 87688 588 0 SWJ wait 0xe0000001045a5720 sh 87688 87686 588 0 SWJ wait 0xe000004080a1b280 make 87686 63244 588 0 SW wait 0xe0000001044e4940 sh 63399 63244 588 0 S piperd 0xe000000018becc08 tee 63244 3381 588 0 S select 0xe0000000123eb5c0 sh 49525 3381 588 0 S wait 0xe000000012ade940 sh 3381 588 588 0 S select 0xe0000000123eb940 sh 593 587 593 25 Ss piperd 0xe00000010459ac08 sendmail 588 587 588 0 SWs wait 0xe0000000122b6de0 sh 587 786 786 0 S piperd 0xe00000010459a268 cron 875 1 875 0 Ss+ ttyin 0xe000000011c30ca8 getty 844 1 1 0 S ttydcd 0xe000000011c310e8 getty 821 1 821 0 Ss select 0xe0000000122505c0 inetd 786 1 786 0 Ss nanslp 0x9ffc000000c2fa38 cron 782 1 782 25 Ss pause 0xe000000012adc9e8 sendmail 779 1 779 0 Ss select 0xe0000000123ec140 sendmail 776 1 776 0 Ss select 0xe0000000123ec3c0 sshd 723 721 721 80 S kqread 0xe000000010bbb300 nginx 721 1 721 0 SWs pause 0xe000000010b93328 nginx 702 1 702 0 Ss select 0xe000000012250fc0 ntpd 642 1 642 0 Ss select 0xe000000012250dc0 casperd 641 1 22 0 S+ select 0xe000000012250c40 casperd 612 1 612 0 Ss select 0xe000000012250d40 syslogd 486 1 486 0 Ss select 0xe000000012250e40 devd 135 1 135 0 SWs pause 0xe000000012adce88 adjkerntz 40 0 0 0 DL (threaded) [zfskern] 100341 D tx->tx_s 0xe0000000125d8208 [txg_thread_enter] 100340 D tx->tx_q 0xe0000000125d8228 [txg_thread_enter] 100334 D spa->spa 0xe0000000123daac0 [trim tank] 100067 D l2arc_fe 0xa0000000013d5540 [l2arc_feed_thread] 100066 D arc_recl 0xa0000000013c5720 [arc_reclaim_thread] 21 0 0 0 DL sdflush 0x9ffc000000bfb890 [softdepflush] 20 0 0 0 DL vlruwt 0xe0000000122b0de0 [vnlru] 19 0 0 0 DL syncer 0x9ffc000000f65b10 [syncer] 18 0 0 0 DL psleep 0x9ffc000000bfac70 [bufdaemon] 17 0 0 0 DL pgzero 0x9ffc000000bfbbfc [pagezero] 9 0 0 0 DL psleep 0x9ffc000000bfbb64 [vmdaemon] 8 0 0 0 DL psleep 0x9ffc000000bfbb94 [pagedaemon] 5 0 0 0 DL waiting_ 0x9ffc000000f681f0 [sctp_iterator] 4 0 0 0 DL idle 0xa000000000a22000 [mpt_recovery1] 3 0 0 0 DL idle 0xa000000000a0a000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa000000000a06e18 [usbus2] 100038 D - 0xa000000000a06dc0 [usbus2] 100037 D - 0xa000000000a06d68 [usbus2] 100036 D - 0xa000000000a06d10 [usbus2] 100034 D - 0xa0000000009fd460 [usbus1] 100033 D - 0xa0000000009fd408 [usbus1] 100032 D - 0xa0000000009fd3b0 [usbus1] 100031 D - 0xa0000000009fd358 [usbus1] 100029 D - 0xa0000000009f9460 [usbus0] 100028 D - 0xa0000000009f9408 [usbus0] 100027 D - 0xa0000000009f93b0 [usbus0] 100026 D - 0xa0000000009f9358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000bf9670 [acpi_thermal] 2 0 0 0 DL (threaded) [cam] 100051 D ccb_scan 0x9ffc000000bfbf38 [scanner] 100015 D - 0x9ffc000000bfc080 [doneq0] 14 0 0 0 DL - 0x9ffc000000bf98e8 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100011 D - 0x9ffc000000bf9e28 [g_down] 100010 D - 0x9ffc000000bf9e20 [g_up] 100009 D - 0x9ffc000000bf9e10 [g_event] 12 0 0 0 WL (threaded) [intr] 100045 I [swi0: uart uart+++] 100044 I [irq29: bge0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100023 I [swi6: Giant taskq] 100021 I [swi5: fast taskq] 100014 I [swi6: task queue] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 100006 I [swi4: clock (1)] 100005 I [swi4: clock (0)] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000010b924a0 [init] 10 0 0 0 DL audit_wo 0x9ffc000000f6c888 [audit] 0 0 0 0 RLs (threaded) [kernel] 100073 D - 0xe00000001225b600 [zil_clean] 100075 D - 0xe00000001225a600 [zil_clean] 100081 D - 0xe00000405fb09100 [zil_clean] 100092 D - 0xe00000010471b300 [zil_clean] 100467 D - 0xe00000010471b900 [zil_clean] 100356 D - 0xe00000001225bb00 [zil_clean] 100355 D - 0xe00000001225bc00 [zil_clean] 100354 D - 0xe000000010bbb000 [zil_clean] 100353 D - 0xe000000010bbaf00 [zil_clean] 100352 D - 0xe00000001225bd00 [zil_clean] 100351 D - 0xe000000010bbae00 [zil_clean] 100345 D - 0xe000000010bbac00 [zil_clean] 100339 D - 0xe00000001225c300 [metaslab_group_task] 100338 D - 0xe00000001225c400 [metaslab_group_task] 100337 D - 0xe00000001225c500 [zfs_vn_rele_taskq] 100336 D - 0xe00000001225c600 [metaslab_group_task] 100335 D - 0xe00000001225c700 [metaslab_group_task] 100333 D - 0xe00000001225c800 [zio_ioctl_intr] 100332 D - 0xe00000001225c900 [zio_ioctl_issue] 100331 D - 0xe00000001225ca00 [zio_claim_intr] 100330 D - 0xe00000001225cb00 [zio_claim_issue] 100329 D - 0xe00000001225cc00 [zio_free_intr] 100328 D - 0xe00000001225cd00 [zio_free_issue_7_11] 100327 D - 0xe00000001225cd00 [zio_free_issue_7_10] 100326 D - 0xe00000001225cd00 [zio_free_issue_7_9] 100325 D - 0xe00000001225cd00 [zio_free_issue_7_8] 100324 D - 0xe00000001225cd00 [zio_free_issue_7_7] 100323 D - 0xe00000001225cd00 [zio_free_issue_7_6] 100322 D - 0xe00000001225cd00 [zio_free_issue_7_5] 100321 D - 0xe00000001225cd00 [zio_free_issue_7_4] 100320 D - 0xe00000001225cd00 [zio_free_issue_7_3] 100319 D - 0xe00000001225cd00 [zio_free_issue_7_2] 100318 D - 0xe00000001225cd00 [zio_free_issue_7_1] 100317 D - 0xe00000001225cd00 [zio_free_issue_7_0] 100316 D - 0xe00000001225ce00 [zio_free_issue_6_11] 100315 D - 0xe00000001225ce00 [zio_free_issue_6_10] 100314 D - 0xe00000001225ce00 [zio_free_issue_6_9] 100313 D - 0xe00000001225ce00 [zio_free_issue_6_8] 100312 D - 0xe00000001225ce00 [zio_free_issue_6_7] 100311 D - 0xe00000001225ce00 [zio_free_issue_6_6] 100310 D - 0xe00000001225ce00 [zio_free_issue_6_5] 100309 D - 0xe00000001225ce00 [zio_free_issue_6_4] 100308 D - 0xe00000001225ce00 [zio_free_issue_6_3] 100307 D - 0xe00000001225ce00 [zio_free_issue_6_2] 100306 D - 0xe00000001225ce00 [zio_free_issue_6_1] 100305 D - 0xe00000001225ce00 [zio_free_issue_6_0] 100304 D - 0xe00000001225cf00 [zio_free_issue_5_11] 100303 D - 0xe00000001225cf00 [zio_free_issue_5_10] 100302 D - 0xe00000001225cf00 [zio_free_issue_5_9] 100301 D - 0xe00000001225cf00 [zio_free_issue_5_8] 100300 D - 0xe00000001225cf00 [zio_free_issue_5_7] 100299 D - 0xe00000001225cf00 [zio_free_issue_5_6] 100298 D - 0xe00000001225cf00 [zio_free_issue_5_5] 100297 D - 0xe00000001225cf00 [zio_free_issue_5_4] 100296 D - 0xe00000001225cf00 [zio_free_issue_5_3] 100295 D - 0xe00000001225cf00 [zio_free_issue_5_2] 100294 D - 0xe00000001225cf00 [zio_free_issue_5_1] 100293 D - 0xe00000001225cf00 [zio_free_issue_5_0] 100292 D - 0xe00000001225d000 [zio_free_issue_4_11] 100291 D - 0xe00000001225d000 [zio_free_issue_4_10] 100290 D - 0xe00000001225d000 [zio_free_issue_4_9] 100289 D - 0xe00000001225d000 [zio_free_issue_4_8] 100288 D - 0xe00000001225d000 [zio_free_issue_4_7] 100287 D - 0xe00000001225d000 [zio_free_issue_4_6] 100286 D - 0xe00000001225d000 [zio_free_issue_4_5] 100285 D - 0xe00000001225d000 [zio_free_issue_4_4] 100284 D - 0xe00000001225d000 [zio_free_issue_4_3] 100283 D - 0xe00000001225d000 [zio_free_issue_4_2] 100282 D - 0xe00000001225d000 [zio_free_issue_4_1] 100281 D - 0xe00000001225d000 [zio_free_issue_4_0] 100280 D - 0xe00000001225d100 [zio_free_issue_3_11] 100279 D - 0xe00000001225d100 [zio_free_issue_3_10] 100278 D - 0xe00000001225d100 [zio_free_issue_3_9] 100277 D - 0xe00000001225d100 [zio_free_issue_3_8] 100276 D - 0xe00000001225d100 [zio_free_issue_3_7] 100275 D - 0xe00000001225d100 [zio_free_issue_3_6] 100274 D - 0xe00000001225d100 [zio_free_issue_3_5] 100273 D - 0xe00000001225d100 [zio_free_issue_3_4] 100272 D - 0xe00000001225d100 [zio_free_issue_3_3] 100271 D - 0xe00000001225d100 [zio_free_issue_3_2] 100270 D - 0xe00000001225d100 [zio_free_issue_3_1] 100269 D - 0xe00000001225d100 [zio_free_issue_3_0] 100268 D - 0xe00000001225d200 [zio_free_issue_2_11] 100267 D - 0xe00000001225d200 [zio_free_issue_2_10] 100266 D - 0xe00000001225d200 [zio_free_issue_2_9] 100265 D - 0xe00000001225d200 [zio_free_issue_2_8] 100264 D - 0xe00000001225d200 [zio_free_issue_2_7] 100263 D - 0xe00000001225d200 [zio_free_issue_2_6] 100262 D - 0xe00000001225d200 [zio_free_issue_2_5] 100261 D - 0xe00000001225d200 [zio_free_issue_2_4] 100260 D - 0xe00000001225d200 [zio_free_issue_2_3] 100259 D - 0xe00000001225d200 [zio_free_issue_2_2] 100258 D - 0xe00000001225d200 [zio_free_issue_2_1] 100257 D - 0xe00000001225d200 [zio_free_issue_2_0] 100256 D - 0xe00000001225d300 [zio_free_issue_1_11] 100255 D - 0xe00000001225d300 [zio_free_issue_1_10] 100254 D - 0xe00000001225d300 [zio_free_issue_1_9] 100253 D - 0xe00000001225d300 [zio_free_issue_1_8] 100252 D - 0xe00000001225d300 [zio_free_issue_1_7] 100251 D - 0xe00000001225d300 [zio_free_issue_1_6] 100250 D - 0xe00000001225d300 [zio_free_issue_1_5] 100249 D - 0xe00000001225d300 [zio_free_issue_1_4] 100248 D - 0xe00000001225d300 [zio_free_issue_1_3] 100247 D - 0xe00000001225d300 [zio_free_issue_1_2] 100246 D - 0xe00000001225d300 [zio_free_issue_1_1] 100245 D - 0xe00000001225d300 [zio_free_issue_1_0] 100244 D - 0xe000000010bbab00 [zio_free_issue_0_11] 100243 D - 0xe000000010bbab00 [zio_free_issue_0_10] 100242 D - 0xe000000010bbab00 [zio_free_issue_0_9] 100241 D - 0xe000000010bbab00 [zio_free_issue_0_8] 100240 D - 0xe000000010bbab00 [zio_free_issue_0_7] 100239 D - 0xe000000010bbab00 [zio_free_issue_0_6] 100238 D - 0xe000000010bbab00 [zio_free_issue_0_5] 100237 D - 0xe000000010bbab00 [zio_free_issue_0_4] 100236 D - 0xe000000010bbab00 [zio_free_issue_0_3] 100235 D - 0xe000000010bbab00 [zio_free_issue_0_2] 100234 D - 0xe000000010bbab00 [zio_free_issue_0_1] 100233 D - 0xe000000010bbab00 [zio_free_issue_0_0] 100232 D - 0xe000000010bbaa00 [zio_write_intr_high] 100231 D - 0xe000000010bbaa00 [zio_write_intr_high] 100230 D - 0xe000000010bbaa00 [zio_write_intr_high] 100229 D - 0xe000000010bbaa00 [zio_write_intr_high] 100228 D - 0xe000000010bbaa00 [zio_write_intr_high] 100227 D - 0xe000000010bba900 [zio_write_intr_7] 100226 D - 0xe000000010bba900 [zio_write_intr_6] 100225 D - 0xe000000010bba900 [zio_write_intr_5] 100224 D - 0xe000000010bba900 [zio_write_intr_4] 100223 D - 0xe000000010bba900 [zio_write_intr_3] 100222 D - 0xe000000010bba900 [zio_write_intr_2] 100221 D - 0xe000000010bba900 [zio_write_intr_1] 100220 D - 0xe000000010bba900 [zio_write_intr_0] 100219 D - 0xe000000010bba800 [zio_write_issue_hig] 100218 D - 0xe000000010bba800 [zio_write_issue_hig] 100217 D - 0xe000000010bba800 [zio_write_issue_hig] 100216 D - 0xe000000010bba800 [zio_write_issue_hig] 100215 D - 0xe000000010bba800 [zio_write_issue_hig] 100214 D - 0xe000000010bba700 [zio_write_issue] 100213 Run CPU 1 [zio_read_intr] 100212 D - 0xe000000010bba500 [zio_read_issue_7] 100211 D - 0xe000000010bba500 [zio_read_issue_6] 100210 D - 0xe000000010bba500 [zio_read_issue_5] 100209 D - 0xe000000010bba500 [zio_read_issue_4] 100208 D - 0xe000000010bba500 [zio_read_issue_3] 100207 D - 0xe000000010bba500 [zio_read_issue_2] 100206 D - 0xe000000010bba500 [zio_read_issue_1] 100205 D - 0xe000000010bba500 [zio_read_issue_0] 100204 D - 0xe000000010bba400 [zio_null_intr] 100203 D - 0xe000000010bba300 [zio_null_issue] 100201 D - 0xe000000010bb9d00 [metaslab_group_task] 100200 D - 0xe000000010bb9e00 [metaslab_group_task] 100065 D - 0xe00000001225db00 [system_taskq_1] 100064 D - 0xe00000001225db00 [system_taskq_0] 100050 D - 0xe000000010bbbb00 [CAM taskq] 100046 D - 0x9ffc000000c2e7d1 [deadlkres] 100022 D - 0xe000000010bbb500 [thread taskq] 100020 D - 0xe000000010bbb700 [kqueue taskq] 100019 D - 0xe000000010bbb800 [ffs_trim taskq] 100018 D - 0xe000000010bbb900 [acpi_task_2] 100017 D - 0xe000000010bbb900 [acpi_task_1] 100016 D - 0xe000000010bbb900 [acpi_task_0] 100012 D - 0xe000000010bbbd00 [firmware taskq] 100000 D swapin 0x9ffc000000c22388 [swapper] db:0:kdb.enter.panic> alltrace Tracing command c++ pid 80756 tid 101342 td 0xe000000107665b00 ia64_ih_stop(0x9ffc00000096cb40, 0x40b) at ia64_ih_stop+0x80 ia64_handle_intr(0xa000000090749400) at ia64_handle_intr+0x110 ivt_External_Interrupt() at ivt_External_Interrupt+0x30 Tracing command sh pid 80740 tid 100405 td 0xe000000012a52000 cpu_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c814a8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012a52000, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe000000104544de0, 0x6c, 0xe000000012a52378, 0xe000000012a52000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000104544de0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000104544de0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe000000104544de0, 0xe000000104544ee0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000000012a52000, 0x7, 0x0, 0xa0000000905a9390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000000012a52000, 0xffffffff, 0xa0000000905a9390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe000000012a52000, 0xa0000000905a94e8, 0xa0000000905a9390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe000000104544de0, 0x7fffffffffffb2f0, 0x0, 0xe000000012a52000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command as pid 80736 tid 100390 td 0xe000000018be8900 cpu_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c818e0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000018be8900, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe00000001235d810, 0x5c, 0xe000000018be8c78, 0xe000000018be8900) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001235d810, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001235d810, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe00000001235d810, 0xe00000001235da50, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe000000012360f24, 0xa000000090559360, 0xa000000090559378, 0x0, 0xe00000001235d918) at pipe_read+0x740 dofileread(0xe000000018be8900, 0x0, 0xe000000012360f00, 0xa000000090559360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000018be8900, 0x0, 0xa000000090559360) at kern_readv+0xa0 sys_read(0xe000000018be8900, 0xa0000000905594e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe000000018bdc000, 0x1208de000, 0x2000, 0xe000000018be8900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command cc1plus pid 80735 tid 100357 td 0xe000000012a50d80 cpu_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c81ea8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012a50d80, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe0000040bebaf518, 0x0, 0xe000000012a510f8, 0xe000000012a50d80) at sleepq_switch+0x2e0 sleepq_wait(0xe0000040bebaf518, 0x0, 0x9ffc000000b008c0, 0xe000000012a50d80, 0x9ffc0000003b9520) at sleepq_wait+0xb0 _cv_wait(0xe0000040bebaf518, 0xe0000040bebaf4f8, 0xe000000012a50d80, 0x0) at _cv_wait+0x4d0 zio_wait(0xe0000040bebaf1f8, 0xe0000040bebaf4e8, 0xe0000040bebaf4f8, 0xe0000040bebaf518, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150 dbuf_read(0xe00000408e9383f0, 0xe0000040bebaf1f8, 0xa, 0xa000000090450ea8) at dbuf_read+0xd70 dmu_buf_hold(0xe00000408e9383f0, 0xa, 0x0, 0x0, 0xa000000090450ee0, 0x1) at dmu_buf_hold+0x1d0 zap_lockdir(0xe000000012aab400, 0x5f652, 0x0, 0x1, 0x1, 0x0, 0xa000000090450f20) at zap_lockdir+0x50 zap_lookup_norm(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0x0, 0x0) at zap_lookup_norm+0x60 zap_lookup(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0xa0000000012b3eb0, 0xb1e) at zap_lookup+0x60 zfs_dirent_lock(0xa000000090450f80, 0xe00000001ce0aa10, 0xa000000090450fc0, 0xa000000090450f88, 0x6, 0x0, 0x0, 0xe00000001b934f00) at zfs_dirent_lock+0xbb0 zfs_dirlook(0xe00000001ce0aa10, 0xa000000090450fc0, 0xa0000000904512e8, 0x0, 0x0, 0x0) at zfs_dirlook+0x240 zfs_lookup(0xe0000000155c1110, 0xa000000090450fc0, 0xa0000000904512e8, 0xa000000090451310, 0x0, 0xe000004082290200, 0xe000000012a50d80, 0x0) at zfs_lookup+0x7a0 zfs_freebsd_lookup(0xa0000000904510c0, 0xa000000090451310, 0xa000000090451348) at zfs_freebsd_lookup+0x100 VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510c0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270 vfs_cache_lookup(0xe0000000155c1110, 0x0, 0xa000000090451310) at vfs_cache_lookup+0x1f0 VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510f8, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270 lookup(0xa000000090451288, 0x200000, 0xe0000000155c1110, 0x0, 0xa000000090451318) at lookup+0xe30 namei(0xa000000090451288, 0xe0000000123c2000, 0x9ffc0000008fe220, 0x0, 0xa0000000904512f8) at namei+0xb40 vn_open_cred(0xa000000090451288, 0xa000000090451390, 0x1a4, 0x0, 0xe000004082290200, 0xe00000001233ecd0) at vn_open_cred+0x500 vn_open(0xa000000090451288, 0xa000000090451390, 0x1a4, 0xe00000001233ecd0, 0x9ffc0000005c3ee0, 0x695, 0x0) at vn_open+0x50 kern_openat(0xe000000012a50d80, 0xffffffffffffff9c, 0x1214529e0, 0x0, 0x0, 0x1b6) at kern_openat+0x490 kern_open(0xe000000012a50d80, 0x1214529e0, 0x0, 0x8000, 0x1b6) at kern_open+0x40 sys_open(0xe000000012a50d80, 0xa0000000904514e8, 0x9ffc000000986c80, 0x48d) at sys_open+0x40 syscall(0xe0000000123c2000, 0x8000, 0x1b6, 0xe000000012a50d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command c++ pid 80734 tid 100487 td 0xe000004081208480 cpu_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000004081208480, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe0000040810f1280, 0x6c, 0xe0000040812087f8, 0xe000004081208480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000040810f1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000040810f1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe0000040810f1280, 0xe0000040810f1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000004081208480, 0x0, 0x13b5f, 0xa000000090711390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000004081208480, 0x13b5f, 0xa000000090711390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe000004081208480, 0xa0000000907114e8, 0xa000000090711390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe0000040810f1280, 0x1208060d8, 0x0, 0xe000004081208480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sleep pid 80733 tid 100414 td 0xe000000104522900 cpu_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c81818, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000104522900, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0x9ffc000000c2fa39, 0x6c, 0xe000000104522c78, 0xe000000104522900) at sleepq_switch+0x2e0 sleepq_catch_signals(0x9ffc000000c2fa39, 0x6c, 0x9ffc000000b008c0, 0xe000000104522c90) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0x9ffc000000c2fa39, 0x6c, 0x20000000, 0x9ffc000000462c50, 0x998) at sleepq_timedwait_sig+0x30 _sleep(0x9ffc000000c2fa39, 0x0, 0x6c, 0x9ffc000000afaba8, 0x1, 0x20000000, 0x200) at _sleep+0x750 kern_nanosleep(0xe000000104522900, 0x200000000, 0xa0000000905f1380) at kern_nanosleep+0x2a0 sys_nanosleep(0xe000000104522900, 0xa0000000905f14e8, 0x0, 0x9ffc000000986c80) at sys_nanosleep+0xb0 syscall(0xe00000010452f280, 0x7fffffffffffeb40, 0x7fffffffffffeb10, 0xe000000104522900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command wc pid 76333 tid 100374 td 0xe000000012a52900 cpu_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c81750, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012a52900, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe00000001235ce70, 0x5c, 0xe000000012a52c78, 0xe000000012a52900) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe00000001235ce70, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe00000001235ce70, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe00000001235ce70, 0xe00000001235d0b0, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe00000001233ff14, 0xa0000000904d9360, 0xa0000000904d9378, 0x0, 0xe00000001235cf78) at pipe_read+0x740 dofileread(0xe000000012a52900, 0x0, 0xe00000001233fef0, 0xa0000000904d9360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012a52900, 0x0, 0xa0000000904d9360) at kern_readv+0xa0 sys_read(0xe000000012a52900, 0xa0000000904d94e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe000000012e16de0, 0x7ffffffffffeece0, 0x10000, 0xe000000012a52900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command tee pid 76332 tid 100451 td 0xe000004081060000 cpu_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c80d28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000004081060000, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe000000018bed5a8, 0x5c, 0xe000004081060378, 0xe000004081060000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000018bed5a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000018bed5a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe000000018bed5a8, 0xe000000018bed7e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe000000018c3aac4, 0xa000000090701360, 0xa000000090701378, 0x0, 0xe000000018bed6b0) at pipe_read+0x740 dofileread(0xe000004081060000, 0x0, 0xe000000018c3aaa0, 0xa000000090701360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000004081060000, 0x0, 0xa000000090701360) at kern_readv+0xa0 sys_read(0xe000004081060000, 0xa0000000907014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe0000040811c6000, 0x120804000, 0x2000, 0xe000004081060000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command find pid 76331 tid 100376 td 0xe000000012e5fb00 cpu_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c80148, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012e5fb00, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe0000000124db180, 0x0, 0xe000000012e5fe78, 0xe000000012e5fb00) at sleepq_switch+0x2e0 sleepq_wait(0xe0000000124db180, 0x0, 0x9ffc000000b008c0, 0xe000000012e5fb00, 0x9ffc0000003b9520) at sleepq_wait+0xb0 _cv_wait(0xe0000000124db180, 0xe0000000124db160, 0xe000000012e5fb00, 0x0) at _cv_wait+0x4d0 zio_wait(0xe0000000124dae60, 0xe0000000124db150, 0xe0000000124db160, 0xe0000000124db180, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150 dbuf_read(0xe0000001078e07e0, 0xe0000000124dae60, 0x22, 0xa0000000904e8ed8, 0xe0000001078e0800, 0xe0000001078e0830) at dbuf_read+0xd70 dnode_hold_impl(0xe00000001290cc00, 0x4129e, 0x1, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold_impl+0x210 dnode_hold(0xe00000001290cc00, 0x4129e, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold+0x40 dmu_bonus_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10) at dmu_bonus_hold+0x40 sa_buf_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10, 0xa000000001276000, 0xd20) at sa_buf_hold+0x40 zfs_zget(0xe000000012a5a000, 0x4129e, 0xa0000000904e8fa8, 0x8, 0x1, 0xe000000012a5a6a8, 0xa0000000012b3eb0, 0xa00000000137fcd8) at zfs_zget+0x120 zfs_dirent_lock(0xa0000000904e8fa0, 0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e8fa8, 0x6, 0x0, 0x0, 0xe00000001b8d7000) at zfs_dirent_lock+0xc10 zfs_dirlook(0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e9220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240 zfs_lookup(0xe000000015ec8270, 0xa0000000904e8fe0, 0xa0000000904e9220, 0xa0000000904e9248, 0x0, 0xe000004082290100, 0xe000000012e5fb00, 0x0) at zfs_lookup+0x7a0 zfs_freebsd_lookup(0xa0000000904e90e0, 0xa0000000904e9248, 0xa0000000904e9280) at zfs_freebsd_lookup+0x100 VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e90e0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270 vfs_cache_lookup(0xe000000015ec8270, 0x0, 0xa0000000904e9248) at vfs_cache_lookup+0x1f0 VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e9118, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270 lookup(0xa0000000904e91c0, 0x200000, 0xe000000015ec8270, 0x0, 0xa0000000904e9250) at lookup+0xe30 namei(0xa0000000904e91c0, 0xa0000000904e91f8, 0x4000104, 0x0, 0xa0000000904e9230, 0xa0000000904e9278, 0xa0000000904e9270, 0xa0000000904e9220) at namei+0xb40 kern_statat_vnhook(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320, 0x0) at kern_statat_vnhook+0xf0 kern_statat(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320) at kern_statat+0x40 sys_fstatat(0xe000000012e5fb00, 0xa0000000904e94e8, 0x9ffc000000986c80, 0x48d) at sys_fstatat+0x50 syscall(0xe000000012e164a0, 0x120873ac8, 0x120873a50, 0xe000000012e5fb00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 76330 tid 100394 td 0xe000000012ef7680 cpu_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012ef7680, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe000000012ef1280, 0x6c, 0xe000000012ef79f8, 0xe000000012ef7680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000012ef1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000012ef1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe000000012ef1280, 0xe000000012ef1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe000000012ef7680, 0x7, 0x0, 0xa000000090579390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe000000012ef7680, 0xffffffff, 0xa000000090579390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe000000012ef7680, 0xa0000000905794e8, 0xa000000090579390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe000000012ef1280, 0x7fffffffffffe4d0, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 76326 tid 100386 td 0xe000000012e5e000 cpu_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c81c28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012e5e000, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe0000040810075a8, 0x5c, 0xe000000012e5e378, 0xe000000012e5e000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000040810075a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000040810075a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe0000040810075a8, 0xe0000040810077e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe000000018c3a2f4, 0xa000000090539360, 0xa000000090539378, 0x0, 0xe0000040810076b0) at pipe_read+0x740 dofileread(0xe000000012e5e000, 0x3, 0xe000000018c3a2d0, 0xa000000090539360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012e5e000, 0x3, 0xa000000090539360) at kern_readv+0xa0 sys_read(0xe000000012e5e000, 0xa0000000905394e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe000000018bdd280, 0x7fffffffffffe800, 0x80, 0xe000000012e5e000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command mail pid 64479 tid 100398 td 0xe000000012ef6900 cpu_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c81098, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012ef6900, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe000000018bed340, 0x5c, 0xe000000012ef6c78, 0xe000000012ef6900) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000018bed340, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000018bed340, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe000000018bed340, 0xe000000018bed580, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0 pipe_read(0xe000004069110d44, 0xa000000090401360, 0xa000000090401378, 0x0, 0xe000000018bed448) at pipe_read+0x740 dofileread(0xe000000012ef6900, 0x0, 0xe000004069110d20, 0xa000000090401360, 0x0, 0x0) at dofileread+0x150 kern_readv(0xe000000012ef6900, 0x0, 0xa000000090401360) at kern_readv+0xa0 sys_read(0xe000000012ef6900, 0xa0000000904014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100 syscall(0xe000000012ef04a0, 0x12080e000, 0x2000, 0xe000000012ef6900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 64478 tid 100447 td 0xe0000001045b5680 cpu_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c81f70, 0x9ffc0000004aa1a0) at cpu_switch+0xd0 sched_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000001045b5680, 0x9ffc0000004e3290) at mi_switch+0x3f0 sleepq_switch(0xe0000040811c7280, 0x6c, 0xe0000001045b59f8, 0xe0000001045b5680) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000040811c7280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe0000040811c7280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30 _sleep(0xe0000040811c7280, 0xe0000040811c7380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0 kern_wait6(0xe0000001045b5680, 0x7, 0x0, 0xa0000000906e1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0 kern_wait(0xe0000001045b5680, 0xffffffff, 0xa0000000906e1390, 0x0, 0x0) at kern_wait+0xe0 sys_wait4(0xe0000001045b5680, 0xa0000000906e14e8, 0xa0000000906e1390, 0x9ffc000000986c80) at sys_wait4+0x70 syscall(0xe0000040811c7280, 0x7fffffffffffe6e0, 0x0, 0xe0000001045b5680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command sh pid 64471 tid 100401 td 0xe000000104523680 Tracing command lockf pid 64470 tid 100359 td 0xe000000012a50480 Tracing command sh pid 64468 tid 100364 td 0xe000000012a26900 Tracing command sh pid 64467 tid 101341 td 0xe000000107665200 Tracing command mail pid 64204 tid 100446 td 0xe0000001045b5b00 Tracing command sh pid 64203 tid 100421 td 0xe000000104522480 Tracing command sh pid 64196 tid 100385 td 0xe000000012e5e480 Tracing command lockf pid 64195 tid 100372 td 0xe000000012a53200 Tracing command sh pid 64193 tid 100395 td 0xe000000012ef7200 Tracing command cron pid 64191 tid 100488 td 0xe000004081208000 Tracing command make pid 47211 tid 100407 td 0xe000000012ee9680 Tracing command make pid 92677 tid 100412 td 0xe000000104522d80 Tracing command sh pid 92676 tid 100382 td 0xe000000012ee8480 Tracing command make pid 92652 tid 100422 td 0xe000000104522000 Tracing command sh pid 92650 tid 100388 td 0xe000000012a27680 Tracing command tee pid 59715 tid 100419 td 0xe00000010458e480 Tracing command sh pid 59384 tid 100363 td 0xe000000012a26d80 Tracing command gmake pid 15731 tid 100434 td 0xe000000104591200 Tracing command gmake pid 87774 tid 100371 td 0xe000000012a53680 Tracing command sh pid 87772 tid 100438 td 0xe000004080a2fb00 Tracing command make pid 87688 tid 100433 td 0xe000000104591680 Tracing command sh pid 87686 tid 100420 td 0xe00000010458e000 Tracing command tee pid 63399 tid 100387 td 0xe000000012a27b00 Tracing command sh pid 63244 tid 100365 td 0xe000000012a26480 Tracing command sh pid 49525 tid 100361 td 0xe000000012613b00 Tracing command sh pid 3381 tid 100383 td 0xe000000012ee8000 Tracing command sendmail pid 593 tid 100368 td 0xe000000012a99680 Tracing command sh pid 588 tid 100360 td 0xe000000012a50000 Tracing command cron pid 587 tid 100445 td 0xe000000107664000 Tracing command getty pid 875 tid 100377 td 0xe000000012e5f680 Tracing command getty pid 844 tid 100059 td 0xe000000011123680 Tracing command inetd pid 821 tid 100370 td 0xe000000012a53b00 Tracing command cron pid 786 tid 100344 td 0xe00000001243ed80 Tracing command sendmail pid 782 tid 100367 td 0xe000000012a99b00 Tracing command sendmail pid 779 tid 100049 td 0xe0000000121da480 Tracing command sshd pid 776 tid 100048 td 0xe0000000121da900 Tracing command nginx pid 723 tid 100060 td 0xe000000011123200 Tracing command nginx pid 721 tid 100061 td 0xe000000011122d80 Tracing command ntpd pid 702 tid 100362 td 0xe000000012a27200 Tracing command casperd pid 642 tid 100343 td 0xe00000001243f200 Tracing command casperd pid 641 tid 100358 td 0xe000000012a50900 Tracing command syslogd pid 612 tid 100369 td 0xe000000012a99200 Tracing command devd pid 486 tid 100062 td 0xe0000000123be480 Tracing command adjkerntz pid 135 tid 100366 td 0xe000000012a26000 Tracing command zfskern pid 40 tid 100341 td 0xe000000012612480 Tracing command zfskern pid 40 tid 100340 td 0xe000000012612900 Tracing command zfskern pid 40 tid 100334 td 0xe000000012438d80 Tracing command zfskern pid 40 tid 100067 td 0xe0000000123b8d80 Tracing command zfskern pid 40 tid 100066 td 0xe0000000123b9200 Tracing command softdepflush pid 21 tid 100058 td 0xe0000000122b8000 Tracing command vnlru pid 20 tid 100057 td 0xe0000000122b8480 Tracing command syncer pid 19 tid 100056 td 0xe0000000122b8900 Tracing command bufdaemon pid 18 tid 100055 td 0xe0000000122b8d80 Tracing command pagezero pid 17 tid 100054 td 0xe0000000122b9200 Tracing command vmdaemon pid 9 tid 100053 td 0xe0000000122b9680 Tracing command pagedaemon pid 8 tid 100052 td 0xe0000000122b9b00 Tracing command sctp_iterator pid 5 tid 100047 td 0xe0000000121dad80 Tracing command mpt_recovery1 pid 4 tid 100043 td 0xe000000010d06d80 Tracing command mpt_recovery0 pid 3 tid 100041 td 0xe000000010d07680 Tracing command usb pid 16 tid 100039 td 0xe000000011122000 Tracing command usb pid 16 tid 100038 td 0xe000000011122480 Tracing command usb pid 16 tid 100037 td 0xe000000011122900 Tracing command usb pid 16 tid 100036 td 0xe000000010c33b00 Tracing command usb pid 16 tid 100034 td 0xe000000010d04480 Tracing command usb pid 16 tid 100033 td 0xe000000010d04900 Tracing command usb pid 16 tid 100032 td 0xe000000010d04d80 Tracing command usb pid 16 tid 100031 td 0xe000000010d05200 Tracing command usb pid 16 tid 100029 td 0xe000000010d05b00 Tracing command usb pid 16 tid 100028 td 0xe000000010d06000 Tracing command usb pid 16 tid 100027 td 0xe000000010bb1200 Tracing command usb pid 16 tid 100026 td 0xe000000010bb1680 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000010c32000 Tracing command cam pid 2 tid 100051 td 0xe000000011123b00 Tracing command cam pid 2 tid 100015 td 0xe000000010b97680 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000010bb0000 Tracing command geom pid 13 tid 100011 td 0xe000000010bb0900 Tracing command geom pid 13 tid 100010 td 0xe000000010bb0d80 Tracing command geom pid 13 tid 100009 td 0xe000000010b94000 Tracing command intr pid 12 tid 100045 td 0xe000000010d06480 Tracing command intr pid 12 tid 100044 td 0xe000000010d06900 Tracing command intr pid 12 tid 100042 td 0xe000000010d07200 Tracing command intr pid 12 tid 100040 td 0xe000000010d07b00 Tracing command intr pid 12 tid 100035 td 0xe000000010d04000 Tracing command intr pid 12 tid 100030 td 0xe000000010d05680 Tracing command intr pid 12 tid 100025 td 0xe000000010bb1b00 Tracing command intr pid 12 tid 100023 td 0xe000000010c3248