Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 13 Dec 2013 05:14:32 +0000
From:      Ryan Baldwin <ryan.baldwin@nexusalpha.com>
To:        freebsd-fs@freebsd.org
Subject:   ZFS related hang with FreeBSD 9.2
Message-ID:  <52AA97B8.8060408@nexusalpha.com>

next in thread | raw e-mail | index | archive | help
Hi,

We have a server based on FreeBSD 9.2 which hangs at times on a daily 
basis. The longest uptime we have achieved is 5 days conversely it has 
stopped daily several days in a row.

When this occurs it appears there are two proceses stuck in 'tx->tx' 
state. In the top output shown these are snapshot-manager processes 
which create and destroy snapshots generally and sometime rollback 
filesystems to snapshots. When the lockup occurs other processes which 
try to access the file system can seem to end up stuck in state 
'rrl->r'. The reboot command that was issued to try and reboot the 
server has ended up stuck in this state as can be seen.

The server is not under particularly heavy load.

It has remained in this state for hours. The 'deadman handler'? does not 
appear to restart the system. Once this has occurred there is no further 
disk activity.

We did not experience this problem at all previously using 9.1 although 
we had less snapshot-manager processes before. We have built this server 
against 9.1 again now but it has only been going one day so far.

We can try and reproduce this problem again on 9.2 if by doing so we can 
gather any additional information that could help resolve this problem. 
Please let me know what other information would be helpful.

The hardware is a Dell R420 with Perc H310 raid controller in JBOD mode 
with the pool mirrored on two SAS disks.

Thanks

top and procstat output follow:


last pid: 76225;  load averages:  0.00,  0.00,  0.00       up 
0+20:04:22  09:56:24
46 processes:  1 running, 44 sleeping, 1 stopped
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free
ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other
Swap: 8192M Total, 8192M Free

   PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU 
COMMAND
  1170 root          1  20    0 16244K  5460K kqread 11   1:28  0.00% 
CustomInit
  1531 root          1  20    0 39648K  6368K tx->tx  7   1:17 0.00% 
snapshot-manager
  1135 root          1  20    0 16244K  5100K kqread  1   1:10  0.00% 
CustomInit
  2097 root          2  20    0  2581M   472M STOP    4   0:46  0.00% java
  1183 root          1  20    0 16244K  5504K kqread  0   0:22  0.00% 
CustomInit
  1145 root          1  20    0 16244K  5464K kqread  9   0:18  0.00% 
CustomInit
  1444 root          1  20    0 39648K  6352K tx->tx  5   0:13 0.00% 
snapshot-manager
  1628 root          1  20    0 39648K  6348K rrl->r 10   0:12 0.00% 
snapshot-manager
  1168 root          1  20    0 16244K  5388K kqread  7   0:07  0.00% 
CustomInit
  1535 root          1  20    0 22388K 14296K select  4   0:05  0.00% 
openvpn
  1163 root          1  20    0 16244K  5388K kqread  2   0:04  0.00% 
CustomInit
  1511 root          1  20    0 18292K 10192K select  0   0:04  0.00% 
openvpn
  1156 root          1  20    0 16244K  5392K kqread 10   0:04  0.00% 
CustomInit
  1174 root          1  20    0 16244K  5388K kqread  1   0:04  0.00% 
CustomInit
  1161 root          1  20    0 16244K  5392K kqread  4   0:03  0.00% 
CustomInit
   913 root          1  20    0 12076K  1820K select  7   0:03  0.00% 
syslogd
  2102 root          1  20    0   109M 13616K zfsvfs  3   0:03  0.00% 
data-processor
  1445 root          1  20    0 22388K 14296K select  5   0:02  0.00% 
openvpn
  1014 root          1  20    0 22256K  3360K select  4   0:02  0.00% ntpd
  1494 root          1  20    0 18292K 10192K select  2   0:01  0.00% 
openvpn
  1180 root          1  20    0 18292K  4780K select  0   0:01  0.00% 
openvpn
  1505 root          1  20    0 18292K  4712K select  2   0:01  0.00% 
openvpn
  1495 root          1  20    0 18292K  4712K select  5   0:00  0.00% 
openvpn
  1479 root          1  20    0 18292K  4708K select  1   0:00  0.00% 
openvpn
  1567 root          1  20    0 18292K  4712K select  3   0:00  0.00% 
openvpn
  1545 root          1  20    0 18292K  4712K select  1   0:00  0.00% 
openvpn
  1486 root          1  20    0 18292K  4708K select  0   0:00  0.00% 
openvpn
  1443 root          1  20    0 18292K  4436K select  9   0:00  0.00% 
openvpn
  1447 root          1  20    0 18292K  4448K select  4   0:00  0.00% 
openvpn
  1496 root          1  20    0 18292K  4708K select  3   0:00  0.00% 
openvpn
  1478 root          1  20    0 18292K  4704K select  5   0:00  0.00% 
openvpn
  1488 root          1  20    0 18292K 10192K select  5   0:00  0.00% 
openvpn
  1032 root          1  20    0 14176K  1860K nanslp  1   0:00  0.00% cron
76069 root          1  20    0  9948K  1680K rrl->r  7   0:00 0.00% reboot
   782 root          1  20    0 10376K  4416K select  5   0:00  0.00% devd
76196 root          1  20    0 51536K  5828K select  0   0:00  0.00% sshd
76203 root          1  20    0 51536K  5828K select  2   0:00  0.00% sshd
76205 root          1  20    0 17564K  3252K ttyin  11   0:00  0.00% csh
76198 root          1  20    0 17564K  3252K pause   8   0:00  0.00% csh
75718 root          1  20    0 35556K  3956K rrl->r 11   0:00 0.00% 
snapshot-manager-counts
75715 root          1  20    0 35556K  3956K rrl->r 10   0:00 0.00% 
snapshot-manager-counts
  1181 root          1  52    0 12084K  1676K ttyin   1   0:00  0.00% getty


0 100000 kernel           swapper          mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77 
btext+0x2c
     0 100032 kernel           firmware taskq   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100037 kernel           ffs_trim taskq   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100038 kernel           acpi_task_0      mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100039 kernel           acpi_task_1      mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100040 kernel           acpi_task_2      mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100041 kernel           kqueue taskq     mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100044 kernel           thread taskq     mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100052 kernel           bge0 taskq       mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100053 kernel           bge1 taskq       mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100062 kernel           mca taskq        mi_switch+0x186 
sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 
fork_exit+0x11f fork_trampoline+0xe
     0 100063 kernel           system_taskq_0   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100064 kernel           system_taskq_1   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100065 kernel           system_taskq_2   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100066 kernel           system_taskq_3   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100067 kernel           system_taskq_4   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100068 kernel           system_taskq_5   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100069 kernel           system_taskq_6   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100070 kernel           system_taskq_7   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100071 kernel           system_taskq_8   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100072 kernel           system_taskq_9   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100073 kernel           system_taskq_10  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100074 kernel           system_taskq_11  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100247 kernel           zio_null_issue   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100248 kernel           zio_null_intr    mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100249 kernel           zio_read_issue_0 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100250 kernel           zio_read_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100251 kernel           zio_read_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100252 kernel           zio_read_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100253 kernel           zio_read_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100254 kernel           zio_read_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100255 kernel           zio_read_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100256 kernel           zio_read_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100257 kernel           zio_read_intr_0  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100258 kernel           zio_read_intr_1  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100259 kernel           zio_read_intr_2  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100260 kernel           zio_read_intr_3  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100261 kernel           zio_read_intr_4  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100262 kernel           zio_read_intr_5  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100263 kernel           zio_read_intr_6  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100264 kernel           zio_read_intr_7  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100265 kernel           zio_read_intr_8  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100266 kernel           zio_read_intr_9  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100267 kernel           zio_read_intr_10 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100268 kernel           zio_read_intr_11 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100269 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100270 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100271 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100272 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100273 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100274 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100275 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100276 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100277 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100278 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100279 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100280 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100281 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100282 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100283 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100284 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100285 kernel           zio_write_issue_ mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100286 kernel           zio_write_intr_0 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100287 kernel           zio_write_intr_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100288 kernel           zio_write_intr_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100289 kernel           zio_write_intr_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100290 kernel           zio_write_intr_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100291 kernel           zio_write_intr_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100292 kernel           zio_write_intr_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100293 kernel           zio_write_intr_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100294 kernel           zio_write_intr_h mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100295 kernel           zio_write_intr_h mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100296 kernel           zio_write_intr_h mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100297 kernel           zio_write_intr_h mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100298 kernel           zio_write_intr_h mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100299 kernel           zio_free_issue_0 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100300 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100301 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100302 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100303 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100304 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100305 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100306 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100307 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100308 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100309 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100310 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100311 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100312 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100313 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100314 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100315 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100316 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100317 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100318 kernel           zio_free_issue_1 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100319 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100320 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100321 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100322 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100323 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100324 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100325 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100326 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100327 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100328 kernel           zio_free_issue_2 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100329 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100330 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100331 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100332 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100333 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100334 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100335 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100336 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100337 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100338 kernel           zio_free_issue_3 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100339 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100340 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100341 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100342 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100343 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100344 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100345 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100346 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100347 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100348 kernel           zio_free_issue_4 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100349 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100350 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100351 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100352 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100353 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100354 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100355 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100356 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100357 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100358 kernel           zio_free_issue_5 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100359 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100360 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100361 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100362 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100363 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100364 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100365 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100366 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100367 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100368 kernel           zio_free_issue_6 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100369 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100370 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100371 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100372 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100373 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100374 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100375 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100376 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100377 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100378 kernel           zio_free_issue_7 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100379 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100380 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100381 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100382 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100383 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100384 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100385 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100386 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100387 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100388 kernel           zio_free_issue_8 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100389 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100390 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100391 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100392 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100393 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100394 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100395 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100396 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100397 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100398 kernel           zio_free_issue_9 mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100399 kernel           zio_free_intr    mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100400 kernel           zio_claim_issue  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100401 kernel           zio_claim_intr   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100402 kernel           zio_ioctl_issue  mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100403 kernel           zio_ioctl_intr   mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100405 kernel           zfs_vn_rele_task mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100418 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100441 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100477 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100478 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100479 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100480 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100481 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100482 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100484 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100486 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100488 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100489 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100490 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100491 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100492 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100493 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100494 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100495 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100496 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100497 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100498 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100499 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100636 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     0 100638 kernel           zil_clean        mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f 
fork_trampoline+0xe
     1 100002 init             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540 
Xfast_syscall+0xf7
     2 100033 crypto           -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f 
fork_trampoline+0xe
     3 100034 crypto returns   -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 crypto_ret_proc+0x192 fork_exit+0x11f 
fork_trampoline+0xe
     4 100061 ctl_thrd         -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f 
fork_trampoline+0xe
     5 100075 zfskern          arc_reclaim_thre mi_switch+0x186 
sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d 
fork_exit+0x11f fork_trampoline+0xe
     5 100076 zfskern          l2arc_feed_threa mi_switch+0x186 
sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2 
fork_exit+0x11f fork_trampoline+0xe
     5 100404 zfskern          trim nsgroot     mi_switch+0x186 
sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68 
fork_exit+0x11f fork_trampoline+0xe
     5 100406 zfskern          txg_thread_enter mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79 
txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe
     5 100407 zfskern          txg_thread_enter mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35 
dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba 
txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
     5 100408 zfskern          zvol nsgroot/swa mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc fork_exit+0x11f 
fork_trampoline+0xe
     6 100077 sctp_iterator    -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f 
fork_trampoline+0xe
     7 100078 xpt_thrd         -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f 
fork_trampoline+0xe
     8 100079 ipmi0: kcs       -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d 
fork_exit+0x11f fork_trampoline+0xe
     9 100080 enc_daemon0      -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f 
fork_trampoline+0xe
    10 100001 audit            -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f 
fork_trampoline+0xe
    11 100003 idle             idle: cpu0 <running>
    11 100004 idle             idle: cpu1 <running>
    11 100005 idle             idle: cpu2 <running>
    11 100006 idle             idle: cpu3 <running>
    11 100007 idle             idle: cpu4 <running>
    11 100008 idle             idle: cpu5 <running>
    11 100009 idle             idle: cpu6 <running>
    11 100010 idle             idle: cpu7 <running>
    11 100011 idle             idle: cpu8 <running>
    11 100012 idle             idle: cpu9 <running>
    11 100013 idle             idle: cpu10      mi_switch+0x186 
critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f fork_trampoline+0xe
    11 100014 idle             idle: cpu11 <running>
    12 100015 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100016 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100017 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100018 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100019 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100020 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100021 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100022 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100023 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100024 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100025 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100026 intr             swi4: clock      mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100027 intr             swi3: vm
    12 100028 intr             swi1: netisr 0   mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100036 intr             swi6: task queue mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100042 intr             swi2: cambio     mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100043 intr             swi5: fast taskq
    12 100045 intr             swi6: Giant task mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100046 intr             irq264: mfi0     mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100047 intr             irq23: ehci0     mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100054 intr             irq22: ehci1     mi_switch+0x186 
ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
    12 100059 intr             irq267: ahci0
    12 100060 intr             swi0: uart uart
    13 100029 geom             g_event          mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440 fork_exit+0x11f 
fork_trampoline+0xe
    13 100030 geom             g_up             mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 g_io_schedule_up+0xe6 g_up_procbody+0x5c 
fork_exit+0x11f fork_trampoline+0xe
    13 100031 geom             g_down           mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 g_io_schedule_down+0x25f 
g_down_procbody+0x5c fork_exit+0x11f fork_trampoline+0xe
    14 100035 yarrow           -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea fork_exit+0x11f 
fork_trampoline+0xe
    15 100048 usb              usbus0           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100049 usb              usbus0           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100050 usb              usbus0           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100051 usb              usbus0           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100055 usb              usbus1           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100056 usb              usbus1           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100057 usb              usbus1           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    15 100058 usb              usbus1           mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f 
fork_trampoline+0xe
    16 100081 pagedaemon       -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34 fork_exit+0x11f 
fork_trampoline+0xe
    17 100082 vmdaemon         -                mi_switch+0x186 
sleepq_wait+0x42 _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f 
fork_trampoline+0xe
    18 100083 pagezero         -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83 fork_exit+0x11f 
fork_trampoline+0xe
    19 100084 bufdaemon        -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1 fork_exit+0x11f 
fork_trampoline+0xe
    20 100085 syncer           -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e 
sync_fsync+0x1a2 VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5 
fork_exit+0x11f fork_trampoline+0xe
    21 100086 vnlru            -                mi_switch+0x186 
sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56 
zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 
vnlru_free+0x362 vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe
    22 100087 softdepflush     -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375 fork_exit+0x11f 
fork_trampoline+0xe
   782 100432 devd             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef sys_select+0x5d 
amd64_syscall+0x540 Xfast_syscall+0xf7
   790 100545 pfpurge          -                mi_switch+0x186 
sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31 fork_exit+0x11f 
fork_trampoline+0xe
   913 100449 syslogd          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 
Xfast_syscall+0xf7
  1014 100581 ntpd             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 
Xfast_syscall+0xf7
  1028 100532 sshd             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 
Xfast_syscall+0xf7
  1032 100473 cron             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca 
kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540 
Xfast_syscall+0xf7
  1135 100558 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1145 100509 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1156 100508 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1161 100507 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1163 100521 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1168 100574 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1170 100561 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1174 100560 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1180 100420 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1181 100463 getty            -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 
dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 
Xfast_syscall+0xf7
  1182 100430 getty            -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 
dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 
Xfast_syscall+0xf7
  1183 100421 CustomInit          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
  1443 100599 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1444 100527 snapshot-manager     -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139 
dsl_destroy_snapshots_nvl+0x71 dsl_destroy_snapshot+0x4a 
zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b 
kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
  1445 100514 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1447 100557 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1478 100555 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1479 100551 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1486 100576 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1488 100563 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1494 100575 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1495 100442 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1496 100556 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1505 100431 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1511 100475 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1531 100429 snapshot-manager     -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139 
zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b 
kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
  1535 100434 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1545 100459 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1567 100539 openvpn          -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 
_cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 
amd64_syscall+0x540 Xfast_syscall+0xf7
  1628 100454 snapshot-manager     -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 
dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d 
devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 
Xfast_syscall+0xf7
  2097 100710 java             -                mi_switch+0x186 
thread_suspend_switch+0xcd thread_single+0x1b2 exit1+0x72 
sys_sys_exit+0xe amd64_syscall+0x540 Xfast_syscall+0xf7
  2097 100792 java             -                mi_switch+0x186 
sleepq_wait+0x42 __lockmgr_args+0x5cb vop_stdlock+0x39 
VOP_LOCK1_APV+0x70 _vn_lock+0x47 zfsctl_freebsd_root_lookup+0xd5 
VOP_LOOKUP_APV+0x62 lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 
kern_statat+0x15 sys_stat+0x2a amd64_syscall+0x540 Xfast_syscall+0xf7
  2102 100635 data-processor dm_siri_to_chiro mi_switch+0x186 
sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56 
zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 
vnlru_free+0x362 getnewvnode+0x27d gfs_file_create+0x4b 
gfs_dir_create+0x16 zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 
lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 
sys_stat+0x2a
75715 100424 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 
_cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 
dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d 
devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 
Xfast_syscall+0xf7
75718 100861 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 
_cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 
dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d 
devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 
Xfast_syscall+0xf7
75719 100578 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 
_cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 
dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d 
devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 
Xfast_syscall+0xf7
76069 101690 reboot           -                mi_switch+0x186 
sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e 
sys_sync+0x1e8 amd64_syscall+0x540 Xfast_syscall+0xf7
76196 100847 sshd             -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 
Xfast_syscall+0xf7
76198 101378 csh              -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540 
Xfast_syscall+0xf7
76200 100892 top              -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a 
tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 
dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 
Xfast_syscall+0xf7
76203 100447 sshd             - <running>
76205 101410 csh              -                mi_switch+0x186 
sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 
kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540 
Xfast_syscall+0xf7
76209 101030 procstat         -                <running>





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52AA97B8.8060408>