Date: Fri, 1 Mar 2019 10:00:42 -0500 From: Nick Rogers <ncrogers@gmail.com> To: FreeBSD STABLE <freebsd-stable@freebsd.org> Subject: 12.0-RELEASE zfs/vnode deadlock issue Message-ID: <CAKOb=YaLSPPRrLWGPbcDt0tKqS2h3mT2dLo92aK3r-8O%2B3SHTw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Recently a number of my production 12.0 systems have experienced what I can only gather is a ZFS deadlock related to vnodes. It seems similar to the relatively recent FreeBSD-EN-18:18.zfs (ZFS vnode reclaim deadlock) problem. Previously the same systems were running 11.1-RELEASE without problem. Threads are always stuck with the stack around vn_lock->zfs_root->lookup->namei. When the system is in this state, a simple `ls /` or `ls /tmp` always hangs, but other datasets seem unaffected. I have a fairly straightforward ZFS root setup on a single pool with one SSD. The workload is a ruby/rails/nginx/postgresql backed web application combined with some data warehousing and other periodic tasks. Sometimes I can remote SSH in, other times that fails because the user shell fails to load, and I can run commands via `ssh ... command`. Sometimes the system is not accessible remotely at all, or it eventually becomes inaccessible if left long enough in this state. I always have to physically reboot the device because the shutdown procedure fails. The network stack (e.g. ping) seems to work completely fine whilst this is going on, until you try to interact with or spawn a process that hits the deadlock. Like previous similar ZFS deadlock issues, increasing kern.vnodes seems to make the system last longer by up to a few weeks, but is still a bandaid. However, I have yet to witness vnodes usage actually getting close to the maximum. I haven't had any luck reproducing this reliably, but eventually it happens after a few days or a few weeks... I managed to connect to a system in this state and grab a procstat and get (hopefully) something useful out of kgdb. I will note that although I was able to install debug symbols, I couldn't manage to get the source files onto it for kgdb purposes before I lost SSH access. I am hoping someone can help me figure out if this is a legitimate bug, or something already fixed in 12-STABLE. I wish I could reproduce it reliably to try against STABLE, but there doesn't appear to be any related ZFS fixes not in RELEASE. Thanks. Below is an abbreviated procstat and what I was able to get out of kgdb for an affected thread. Note that the thread backtrace is from a simple `ls` command. The procstat dump below is truncated because my last attempt to send this was rejected by this list for being too long - so a number of sh/cron processes and some zfs threads in a hung state were removed. ld# kgdb GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. done. sched_switch (td=0xfffff8002452a000, newtd=0xfffff80003625580, flags=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2112 2112 /usr/src/sys/kern/sched_ule.c: No such file or directory. (kgdb) tid 102023 (kgdb) bt #0 sched_switch (td=0xfffff801a83dc580, newtd=0xfffff80003550580, flags=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2112 #1 0xffffffff80d0e0a1 in mi_switch (flags=<optimized out>, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:439 #2 0xffffffff80d5c80c in sleepq_wait (wchan=<unavailable>, pri=<unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:692 #3 0xffffffff80cd9105 in sleeplk (lk=0xfffff800247307e8, flags=<optimized out>, ilk=<optimized out>, wmesg=<optimized out>, pri=<optimized out>, timo=51, queue=1) at /usr/src/sys/kern/kern_lock.c:300 #4 0xffffffff80cd7f85 in lockmgr_slock_hard (lk=<optimized out>, flags=<optimized out>, ilk=<optimized out>, file=<optimized out>, line=0, lwa=<optimized out>) at /usr/src/sys/kern/kern_lock.c:646 #5 0xffffffff813acc5e in VOP_LOCK1_APV (vop=<optimized out>, a=0xfffffe00f89dd450) at vnode_if.c:2087 #6 0xffffffff80de2820 in VOP_LOCK1 (vp=0xfffff80024730780, flags=2105344, file=0xffffffff814d4f74 "/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c", line=2074) at ./vnode_if.h:859 #7 _vn_lock (vp=0xfffff80024730780, flags=2105344, file=0xffffffff814d4f74 "/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c", line=2074) at /usr/src/sys/kern/vfs_vnops.c:1533 #8 0xffffffff8049f68d in zfs_root (vfsp=<optimized out>, flags=2105344, vpp=0xfffffe00f89dd558) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:2074 #9 0xffffffff80dc5d43 in lookup (ndp=0xfffffe00f89dd780) at /usr/src/sys/kern/vfs_lookup.c:961 #10 0xffffffff80dc4f9b in namei (ndp=0xfffffe00f89dd780) at /usr/src/sys/kern/vfs_lookup.c:444 #11 0xffffffff80ddc637 in kern_statat (td=0xfffff801a83dc580, flag=<optimized out>, fd=<optimized out>, path=0x800729c78 <error: Cannot access memory at address 0x800729c78>, pathseg=UIO_USERSPACE, sbp=0xfffffe00f89dd898, hook=0x0) at /usr/src/sys/kern/vfs_syscalls.c:2283 #12 0xffffffff80ddcdcf in sys_fstatat (td=<unavailable>, uap=0xfffff801a83dc940) at /usr/src/sys/kern/vfs_syscalls.c:2260 #13 0xffffffff81222449 in syscallenter (td=<optimized out>) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #14 amd64_syscall (td=0xfffff801a83dc580, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1076 #15 <signal handler called> #16 0x000000080047941a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe178 (kgdb) fr 7 #7 _vn_lock (vp=0xfffff80024730780, flags=2105344, file=0xffffffff814d4f74 "/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c", line=2074) at /usr/src/sys/kern/vfs_vnops.c:1533 1533 /usr/src/sys/kern/vfs_vnops.c: No such file or directory. (kgdb) print *vp $1 = {v_tag = 0xffffffff8144af45 "zfs", v_op = 0xffffffff81c64fd0 <zfs_vnodeops>, v_data = 0xfffff800247e8110, v_mount = 0xfffff800247e1000, v_nmntvnodes = {tqe_next = 0xfffff800247305a0, tqe_prev = 0xfffff800247e1060}, { v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0xfffff8016f1ff5b0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffff800247307d0}, v_cache_dd = 0x0, v_lock = {lock_object = { lo_name = 0xffffffff8144af45 "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0x0}, lk_lock = 18446735294395164034, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = { lo_name = 0xffffffff814e4508 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, v_vnlock = 0xfffff800247307e8, v_actfreelist = {tqe_next = 0x0, tqe_prev = 0xfffff80024730660}, v_bufobj = {bo_lock = {lock_object = { lo_name = 0xffffffff814a9e5f "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops = 0xffffffff81d38600 <buf_ops_bio>, bo_object = 0xfffff80006f5d100, bo_synclist = { le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff80024730780, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80024730898}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = { tqh_first = 0x0, tqh_last = 0xfffff800247308b8}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_domain = 1, bo_bsize = 131072}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = { rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80024730908}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 241, v_usecount = 240, v_iflag = 512, v_vflag = 1, v_mflag = 0, v_writecount = 0, v_hash = 4, v_type = VDIR} (kgdb) ld# procstat -kka PID TID COMM TDNAME KSTACK 0 100000 kernel swapper mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 swapper+0x6b btext+0x2c 0 100012 kernel thread taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100014 kernel kqueue_ctx taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100015 kernel config_0 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100017 kernel aiod_kick taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100018 kernel if_io_tqg_0 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100019 kernel if_io_tqg_1 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100020 kernel if_io_tqg_2 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100021 kernel if_io_tqg_3 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100022 kernel if_io_tqg_4 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100023 kernel if_io_tqg_5 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100024 kernel if_io_tqg_6 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100025 kernel if_io_tqg_7 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100026 kernel softirq_0 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100027 kernel softirq_1 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100028 kernel softirq_2 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100029 kernel softirq_3 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100030 kernel softirq_4 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100031 kernel softirq_5 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100032 kernel softirq_6 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100033 kernel softirq_7 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100034 kernel if_config_tqg_0 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100056 kernel firmware taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100057 kernel crypto_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100058 kernel crypto_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100059 kernel crypto_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100060 kernel crypto_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100061 kernel crypto_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100062 kernel crypto_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100063 kernel crypto_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100064 kernel crypto_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100096 kernel system_taskq_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100097 kernel system_taskq_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100098 kernel system_taskq_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100099 kernel system_taskq_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100100 kernel system_taskq_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100101 kernel system_taskq_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100102 kernel system_taskq_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100103 kernel system_taskq_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100104 kernel mca taskq mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100109 kernel arc_prune_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100110 kernel arc_prune_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100111 kernel arc_prune_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100112 kernel arc_prune_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100113 kernel arc_prune_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100114 kernel arc_prune_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100115 kernel arc_prune_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100116 kernel arc_prune_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100119 kernel dbu_evict mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100121 kernel z_vdev_file_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100122 kernel z_vdev_file_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100123 kernel z_vdev_file_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100124 kernel z_vdev_file_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100125 kernel z_vdev_file_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100126 kernel z_vdev_file_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100127 kernel z_vdev_file_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100128 kernel z_vdev_file_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100129 kernel z_vdev_file_8 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100130 kernel z_vdev_file_9 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100131 kernel z_vdev_file_10 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100132 kernel z_vdev_file_11 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100133 kernel z_vdev_file_12 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100134 kernel z_vdev_file_13 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100135 kernel z_vdev_file_14 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100136 kernel z_vdev_file_15 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100138 kernel zfsvfs mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100143 kernel acpi_task_0 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100144 kernel acpi_task_1 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100145 kernel acpi_task_2 mi_switch+0xe1 sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3 fork_exit+0x83 fork_trampoline+0xe 0 100146 kernel CAM taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100167 kernel zio_null_issue mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100168 kernel zio_null_intr mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100169 kernel zio_read_issue_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100170 kernel zio_read_issue_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100171 kernel zio_read_issue_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100172 kernel zio_read_issue_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100173 kernel zio_read_issue_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100174 kernel zio_read_issue_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100175 kernel zio_read_issue_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100176 kernel zio_read_issue_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100177 kernel zio_read_intr_0_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100178 kernel zio_read_intr_0_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100179 kernel zio_read_intr_0_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100180 kernel zio_read_intr_0_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100181 kernel zio_read_intr_0_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100182 kernel zio_read_intr_0_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100183 kernel zio_read_intr_0_6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100184 kernel zio_read_intr_0_7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100185 kernel zio_read_intr_0_8 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100186 kernel zio_read_intr_0_9 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100187 kernel zio_read_intr_0_10 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100188 kernel zio_read_intr_0_11 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100189 kernel zio_read_intr_1_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100393 kernel zio_free_intr mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100394 kernel zio_claim_issue mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100395 kernel zio_claim_intr mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100396 kernel zio_ioctl_issue mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100397 kernel zio_ioctl_intr mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100403 kernel dp_sync_taskq_0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100404 kernel dp_sync_taskq_1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100405 kernel dp_sync_taskq_2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100406 kernel dp_sync_taskq_3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100407 kernel dp_sync_taskq_4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100408 kernel dp_sync_taskq_5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100409 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100410 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100411 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100412 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100413 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100414 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100415 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100416 kernel dp_zil_clean_taskq_ mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100417 kernel zfs_vn_rele_taskq mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100418 kernel metaslab_group_task mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100419 kernel metaslab_group_task mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100420 kernel metaslab_group_task mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 0 100421 kernel metaslab_group_task mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83 fork_trampoline+0xe 1 100002 init - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369 fast_syscall_common+0x101 2 100065 crypto - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_proc+0x2e5 fork_exit+0x83 fork_trampoline+0xe 3 100066 crypto returns 0 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 4 100067 crypto returns 1 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 5 100068 crypto returns 2 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 6 100069 crypto returns 3 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 7 100070 crypto returns 4 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 8 100071 crypto returns 5 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 9 100072 crypto returns 6 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 10 100001 audit - mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 audit_worker+0x73 fork_exit+0x83 fork_trampoline+0xe 11 100003 idle idle: cpu0 acpi_cpu_idle+0x2e7 cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100004 idle idle: cpu1 mi_switch+0xe1 critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100005 idle idle: cpu2 acpi_cpu_idle+0x2e7 cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100006 idle idle: cpu3 mi_switch+0xe1 critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100007 idle idle: cpu4 acpi_cpu_idle+0x2e7 cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100008 idle idle: cpu5 mi_switch+0xe1 critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100009 idle idle: cpu6 acpi_cpu_idle+0x2e7 cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 11 100010 idle idle: cpu7 acpi_cpu_idle+0x2e7 cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83 fork_trampoline+0xe 12 100011 intr swi6: Giant taskq mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100013 intr swi5: fast taskq 12 100016 intr swi6: task queue mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100035 intr swi4: clock (0) mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100036 intr swi4: clock (1) 12 100037 intr swi4: clock (2) 12 100038 intr swi4: clock (3) 12 100039 intr swi4: clock (4) 12 100040 intr swi4: clock (5) 12 100041 intr swi4: clock (6) 12 100042 intr swi4: clock (7) 12 100043 intr swi1: netisr 0 mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100044 intr swi3: vm 12 100077 intr irq274: xhci0 mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100083 intr irq18: ehci0 ehci1 mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100094 intr irq305: ahci0 mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100095 intr swi0: uart uart mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 12 100141 intr swi1: pf send mi_switch+0xe1 ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe 13 100045 ng_queue ng_queue0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100046 ng_queue ng_queue1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100047 ng_queue ng_queue2 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100048 ng_queue ng_queue3 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100049 ng_queue ng_queue4 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100050 ng_queue ng_queue5 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100051 ng_queue ng_queue6 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 13 100052 ng_queue ng_queue7 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83 fork_trampoline+0xe 14 100053 geom g_event mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 g_run_events+0x4f fork_exit+0x83 fork_trampoline+0xe 14 100054 geom g_up mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 g_io_schedule_up+0xcc g_up_procbody+0x6d fork_exit+0x83 fork_trampoline+0xe 14 100055 geom g_down mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 g_io_schedule_down+0xcc g_down_procbody+0x6d fork_exit+0x83 fork_trampoline+0xe 15 100073 crypto returns 7 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83 fork_trampoline+0xe 16 100074 sequencer 00 - mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 seq_eventthread+0xdc fork_exit+0x83 fork_trampoline+0xe 17 100075 cam doneq0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83 fork_trampoline+0xe 17 100076 cam doneq1 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83 fork_trampoline+0xe 17 100147 cam scanner mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 xpt_scanner_thread+0x99 fork_exit+0x83 fork_trampoline+0xe 18 100078 usb usbus0 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100079 usb usbus0 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100080 usb usbus0 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100081 usb usbus0 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100082 usb usbus0 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100084 usb usbus1 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100085 usb usbus1 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100086 usb usbus1 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100087 usb usbus1 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100088 usb usbus1 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100089 usb usbus2 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100090 usb usbus2 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100091 usb usbus2 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100092 usb usbus2 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 18 100093 usb usbus2 mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83 fork_trampoline+0xe 19 100105 soaiod1 - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83 fork_trampoline+0xe 20 100106 soaiod2 - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83 fork_trampoline+0xe 21 100107 soaiod3 - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83 fork_trampoline+0xe 22 100108 soaiod4 - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83 fork_trampoline+0xe 23 100117 zfskern arc_reclaim_thread mi_switch+0xe1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a arc_reclaim_thread+0x146 fork_exit+0x83 fork_trampoline+0xe 23 100118 zfskern arc_dnlc_evicts_thr mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 arc_dnlc_evicts_thread+0x16f fork_exit+0x83 fork_trampoline+0xe 23 100120 zfskern dbuf_evict_thread mi_switch+0xe1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a dbuf_evict_thread+0x1c8 fork_exit+0x83 fork_trampoline+0xe 23 100137 zfskern l2arc_feed_thread mi_switch+0xe1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a l2arc_feed_thread+0x219 fork_exit+0x83 fork_trampoline+0xe 23 100398 zfskern trim zroot mi_switch+0xe1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a trim_thread+0x11f fork_exit+0x83 fork_trampoline+0xe 23 100434 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 txg_quiesce+0x21b txg_quiesce_thread+0x11b fork_exit+0x83 fork_trampoline+0xe 23 100435 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 txg_sync_thread+0x13b fork_exit+0x83 fork_trampoline+0xe 23 100436 zfskern solthread 0xfffffff mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83 fork_trampoline+0xe 23 100437 zfskern solthread 0xfffffff mi_switch+0xe1 sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83 fork_trampoline+0xe 24 100139 sctp_iterator - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 sctp_iterator_thread+0x59 fork_exit+0x83 fork_trampoline+0xe 25 100140 pf purge - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 pf_purge_thread+0xb9 fork_exit+0x83 fork_trampoline+0xe 26 100142 rand_harvestq - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 random_kthread+0x286 fork_exit+0x83 fork_trampoline+0xe 27 100148 enc_daemon0 - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 enc_daemon+0x349 fork_exit+0x83 fork_trampoline+0xe 28 100149 pagedaemon dom0 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 vm_pageout_worker+0x28e vm_pageout+0x15a fork_exit+0x83 fork_trampoline+0xe 28 100151 pagedaemon laundry: dom0 mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 vm_pageout_laundry_worker+0xfdd fork_exit+0x83 fork_trampoline+0xe 28 100153 pagedaemon uma mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 uma_reclaim_worker+0xca fork_exit+0x83 fork_trampoline+0xe 29 100150 vmdaemon - mi_switch+0xe1 sleepq_wait+0x2c _sleep+0x237 vm_daemon+0x94 fork_exit+0x83 fork_trampoline+0xe 30 100152 bufdaemon - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 buf_daemon+0xe8 fork_exit+0x83 fork_trampoline+0xe 30 100156 bufdaemon bufspacedaemon-0 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100157 bufdaemon bufspacedaemon-1 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100158 bufdaemon bufspacedaemon-2 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100159 bufdaemon bufspacedaemon-3 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100160 bufdaemon bufspacedaemon-4 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100161 bufdaemon bufspacedaemon-5 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 30 100162 bufdaemon bufspacedaemon-6 mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83 fork_trampoline+0xe 31 100154 vnlru - mi_switch+0xe1 sleepq_timedwait+0x2f _sleep+0x217 vnlru_proc+0x95 fork_exit+0x83 fork_trampoline+0xe 32 100155 syncer - mi_switch+0xe1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a sched_sync+0xc4 fork_exit+0x83 fork_trampoline+0xe 2925 100473 devd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 3014 100487 syslogd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 3121 100443 automountd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 autofs_ioctl+0xbd devfs_ioctl+0xad VOP_IOCTL_APV+0x7e vn_ioctl+0x1a4 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15e amd64_syscall+0x369 fast_syscall_common+0x101 3127 100461 autounmountd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 3246 100488 sshd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 7053 100664 smbd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 7062 100616 smbd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 7063 100670 smbd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 7064 100455 smbd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 7065 100519 snmptrapd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 10568 100483 nginx - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369 fast_syscall_common+0x101 10569 100803 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10570 100448 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10571 100527 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10572 100552 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10573 100617 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10574 101643 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10575 100610 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 10576 100653 nginx - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174 soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369 fast_syscall_common+0x101 20584 100460 openvpn - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21660 100615 postgres - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_utimesat+0x110 amd64_syscall+0x369 fast_syscall_common+0x101 21662 100612 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21663 100626 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21664 100450 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21665 100638 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21666 100635 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 21667 100700 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 22183 100518 getty - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 22184 100509 getty - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 22185 101647 getty - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 22435 100440 getty - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 24825 101365 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24851 100599 perl - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369 fast_syscall_common+0x101 24855 101640 ntpd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24865 100459 radiusd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 24865 101450 radiusd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_sem2_wait+0x68a __umtx_op_sem2_wait+0x4b amd64_syscall+0x369 fast_syscall_common+0x101 24869 100545 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24869 101382 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100813 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100960 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100961 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100962 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100968 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100973 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100979 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100981 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24870 100984 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24871 100541 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24871 100665 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24871 100986 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24871 100987 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24871 100989 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24871 101001 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24871 101037 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24871 101047 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24871 101051 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24873 100637 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24873 100828 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101055 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101064 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101077 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101098 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101102 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101118 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24873 101122 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24874 100516 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101036 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101129 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101143 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101144 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101147 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101194 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101259 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24874 101267 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24875 100539 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101071 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101284 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101289 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101298 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101334 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101347 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101350 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24875 101369 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24876 100642 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101384 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101411 ruby25 cluster.rb:253 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101423 ruby25 cluster.rb:287 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101429 ruby25 puma 001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101431 ruby25 reactor.rb:249 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101441 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101444 ruby25 thread_pool.rb* mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 24876 101447 ruby25 server.rb:358 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 24885 100816 snort - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 bpfread+0x318 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 24885 101457 snort - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369 fast_syscall_common+0x101 25043 100810 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 25043 101658 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 25044 100682 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 25422 100603 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 25433 100691 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 25442 100504 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 26015 100805 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 27660 100636 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 29381 100553 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 34928 100480 dhcpd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 36704 101146 perl - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9 kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101 36751 101709 cron - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446 dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 39368 101861 sendmail - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369 fast_syscall_common+0x101 39370 101856 mail.local - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77 sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101 40693 101766 perl - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77 sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101 40696 101911 sshd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 40698 101972 sshd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 40699 101310 bash - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369 fast_syscall_common+0x101 40711 101974 cron - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446 dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 40712 101395 sh - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77 sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101 40715 101969 su - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369 fast_syscall_common+0x101 40716 101976 csh - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369 fast_syscall_common+0x101 40722 101977 procstat - sysctl_kern_proc_kstack+0x282 sysctl_root_handler_locked+0x8b sysctl_root+0x257 userland_sysctl+0x17a sys___sysctl+0x5f amd64_syscall+0x369 fast_syscall_common+0x101 81907 100546 racoon - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 83204 100656 perl - mi_switch+0xe1 sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e _vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77 sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101 83208 100501 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 83209 100525 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 83210 100628 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 83424 100485 ladvd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 83425 100554 ladvd - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 83428 101357 perl - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369 fast_syscall_common+0x101 83444 100542 snmpd - <running> 83545 100486 sendmail - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101 83548 100814 sendmail - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100427 named isc-worker0000 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100431 named isc-worker0001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100432 named isc-worker0002 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100556 named - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369 fast_syscall_common+0x101 83554 100774 named isc-worker0003 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100819 named isc-worker0004 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100820 named isc-worker0005 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100849 named isc-worker0006 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 100990 named isc-worker0007 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 101481 named isc-timer mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83554 101534 named isc-socket mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 83556 100608 named - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369 fast_syscall_common+0x101 83556 101540 named isc-worker0000 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101541 named isc-worker0001 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101543 named isc-worker0002 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101545 named isc-worker0003 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101546 named isc-worker0004 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101549 named isc-worker0005 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101560 named isc-worker0006 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101583 named isc-worker0007 mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101596 named isc-timer mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83556 101600 named isc-socket mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71 sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101 83559 100514 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83559 101604 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 83562 100463 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83562 101486 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 83565 100821 ruby25 - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x369 fast_syscall_common+0x101 83565 101606 ruby25 ruby-timer-thr mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 84111 100668 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 84112 100613 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 84113 101274 postgres - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369 fast_syscall_common+0x101 84416 100550 cron - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205 kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369 fast_syscall_common+0x101
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKOb=YaLSPPRrLWGPbcDt0tKqS2h3mT2dLo92aK3r-8O%2B3SHTw>