Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 14 Dec 2013 07:13:12 +0000
From:      Ryan Baldwin <ryan.baldwin@nexusalpha.com>
To:        Rod Taylor <rod.taylor@gmail.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: ZFS related hang with FreeBSD 9.2
Message-ID:  <52AC0508.5060704@nexusalpha.com>
In-Reply-To: <CAKddOFATpE0U8Z0AYhsBPwfym3XJWeh95Q%2BS3Ug_YZJmkhUKCQ@mail.gmail.com>
References:  <52AA97B8.8060408@nexusalpha.com> <CAKddOFATpE0U8Z0AYhsBPwfym3XJWeh95Q%2BS3Ug_YZJmkhUKCQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Yes we are using snapshots. The snapshot-manager processes are 
continually creating and destroying snapshots at a rate of 10,000's a 
day. We have had such a system running on FreeBSD 9.1 with just one 
snapshot-manager process several months on end and we are not aware of 
any problems with this system.

I run another type of system currently on FreeBSD 9.1 as a storage 
server and it serves terrabytes a day and uses snapshots and clones 
extensively and I have not had any problems with this system.

This is the first real problem I have come across. So far the 9.1 based 
version of the problem server is still going although it is still early 
days.


On 12/13/13 23:21, Rod Taylor wrote:
> Are you using snapshots?
>
> I've found ZFS Snapshots on 9.0, 9.1, and 9.2 regularly crash the 
> system. Delete the snapshots and don't create any new ones and 
> suddenly it's stable for months.
>
>
>
> On Fri, Dec 13, 2013 at 12:14 AM, Ryan Baldwin 
> <ryan.baldwin@nexusalpha.com <mailto:ryan.baldwin@nexusalpha.com>> wrote:
>
>     Hi,
>
>     We have a server based on FreeBSD 9.2 which hangs at times on a
>     daily basis. The longest uptime we have achieved is 5 days
>     conversely it has stopped daily several days in a row.
>
>     When this occurs it appears there are two proceses stuck in
>     'tx->tx' state. In the top output shown these are snapshot-manager
>     processes which create and destroy snapshots generally and
>     sometime rollback filesystems to snapshots. When the lockup occurs
>     other processes which try to access the file system can seem to
>     end up stuck in state 'rrl->r'. The reboot command that was issued
>     to try and reboot the server has ended up stuck in this state as
>     can be seen.
>
>     The server is not under particularly heavy load.
>
>     It has remained in this state for hours. The 'deadman handler'?
>     does not appear to restart the system. Once this has occurred
>     there is no further disk activity.
>
>     We did not experience this problem at all previously using 9.1
>     although we had less snapshot-manager processes before. We have
>     built this server against 9.1 again now but it has only been going
>     one day so far.
>
>     We can try and reproduce this problem again on 9.2 if by doing so
>     we can gather any additional information that could help resolve
>     this problem. Please let me know what other information would be
>     helpful.
>
>     The hardware is a Dell R420 with Perc H310 raid controller in JBOD
>     mode with the pool mirrored on two SAS disks.
>
>     Thanks
>
>     top and procstat output follow:
>
>
>     last pid: 76225;  load averages:  0.00,  0.00,  0.00 up 0+20:04:22
>      09:56:24
>     46 processes:  1 running, 44 sleeping, 1 stopped
>     CPU:     % user,     % nice,     % system,     % interrupt,     % idle
>     Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free
>     ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other
>     Swap: 8192M Total, 8192M Free
>
>       PID USERNAME    THR PRI NICE   SIZE    RES STATE   C TIME   WCPU
>     COMMAND
>      1170 root          1  20    0 16244K  5460K kqread 11 1:28  0.00%
>     CustomInit
>      1531 root          1  20    0 39648K  6368K tx->tx  7 1:17 0.00%
>     snapshot-manager
>      1135 root          1  20    0 16244K  5100K kqread  1 1:10  0.00%
>     CustomInit
>      2097 root          2  20    0  2581M   472M STOP    4 0:46  0.00%
>     java
>      1183 root          1  20    0 16244K  5504K kqread  0 0:22  0.00%
>     CustomInit
>      1145 root          1  20    0 16244K  5464K kqread  9 0:18  0.00%
>     CustomInit
>      1444 root          1  20    0 39648K  6352K tx->tx  5 0:13 0.00%
>     snapshot-manager
>      1628 root          1  20    0 39648K  6348K rrl->r 10 0:12 0.00%
>     snapshot-manager
>      1168 root          1  20    0 16244K  5388K kqread  7 0:07  0.00%
>     CustomInit
>      1535 root          1  20    0 22388K 14296K select  4 0:05  0.00%
>     openvpn
>      1163 root          1  20    0 16244K  5388K kqread  2 0:04  0.00%
>     CustomInit
>      1511 root          1  20    0 18292K 10192K select  0 0:04  0.00%
>     openvpn
>      1156 root          1  20    0 16244K  5392K kqread 10 0:04  0.00%
>     CustomInit
>      1174 root          1  20    0 16244K  5388K kqread  1 0:04  0.00%
>     CustomInit
>      1161 root          1  20    0 16244K  5392K kqread  4 0:03  0.00%
>     CustomInit
>       913 root          1  20    0 12076K  1820K select  7 0:03  0.00%
>     syslogd
>      2102 root          1  20    0   109M 13616K zfsvfs  3 0:03  0.00%
>     data-processor
>      1445 root          1  20    0 22388K 14296K select  5 0:02  0.00%
>     openvpn
>      1014 root          1  20    0 22256K  3360K select  4 0:02  0.00%
>     ntpd
>      1494 root          1  20    0 18292K 10192K select  2 0:01  0.00%
>     openvpn
>      1180 root          1  20    0 18292K  4780K select  0 0:01  0.00%
>     openvpn
>      1505 root          1  20    0 18292K  4712K select  2 0:01  0.00%
>     openvpn
>      1495 root          1  20    0 18292K  4712K select  5 0:00  0.00%
>     openvpn
>      1479 root          1  20    0 18292K  4708K select  1 0:00  0.00%
>     openvpn
>      1567 root          1  20    0 18292K  4712K select  3 0:00  0.00%
>     openvpn
>      1545 root          1  20    0 18292K  4712K select  1 0:00  0.00%
>     openvpn
>      1486 root          1  20    0 18292K  4708K select  0 0:00  0.00%
>     openvpn
>      1443 root          1  20    0 18292K  4436K select  9 0:00  0.00%
>     openvpn
>      1447 root          1  20    0 18292K  4448K select  4 0:00  0.00%
>     openvpn
>      1496 root          1  20    0 18292K  4708K select  3 0:00  0.00%
>     openvpn
>      1478 root          1  20    0 18292K  4704K select  5 0:00  0.00%
>     openvpn
>      1488 root          1  20    0 18292K 10192K select  5 0:00  0.00%
>     openvpn
>      1032 root          1  20    0 14176K  1860K nanslp  1 0:00  0.00%
>     cron
>     76069 root          1  20    0  9948K  1680K rrl->r  7 0:00 0.00%
>     reboot
>       782 root          1  20    0 10376K  4416K select  5 0:00  0.00%
>     devd
>     76196 root          1  20    0 51536K  5828K select  0 0:00  0.00%
>     sshd
>     76203 root          1  20    0 51536K  5828K select  2 0:00  0.00%
>     sshd
>     76205 root          1  20    0 17564K  3252K ttyin  11 0:00  0.00% csh
>     76198 root          1  20    0 17564K  3252K pause   8 0:00  0.00% csh
>     75718 root          1  20    0 35556K  3956K rrl->r 11 0:00 0.00%
>     snapshot-manager-counts
>     75715 root          1  20    0 35556K  3956K rrl->r 10 0:00 0.00%
>     snapshot-manager-counts
>      1181 root          1  52    0 12084K  1676K ttyin   1 0:00  0.00%
>     getty
>
>
>     0 100000 kernel           swapper          mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77
>     btext+0x2c
>         0 100032 kernel           firmware taskq mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100037 kernel           ffs_trim taskq mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100038 kernel           acpi_task_0  mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100039 kernel           acpi_task_1  mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100040 kernel           acpi_task_2  mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100041 kernel           kqueue taskq mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100044 kernel           thread taskq mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100052 kernel           bge0 taskq mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100053 kernel           bge1 taskq mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100062 kernel           mca taskq  mi_switch+0x186
>     sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100063 kernel           system_taskq_0 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100064 kernel           system_taskq_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100065 kernel           system_taskq_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100066 kernel           system_taskq_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100067 kernel           system_taskq_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100068 kernel           system_taskq_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100069 kernel           system_taskq_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100070 kernel           system_taskq_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100071 kernel           system_taskq_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100072 kernel           system_taskq_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100073 kernel           system_taskq_10  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100074 kernel           system_taskq_11  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100247 kernel           zio_null_issue mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100248 kernel           zio_null_intr  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100249 kernel           zio_read_issue_0 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100250 kernel           zio_read_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100251 kernel           zio_read_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100252 kernel           zio_read_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100253 kernel           zio_read_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100254 kernel           zio_read_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100255 kernel           zio_read_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100256 kernel           zio_read_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100257 kernel           zio_read_intr_0  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100258 kernel           zio_read_intr_1  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100259 kernel           zio_read_intr_2  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100260 kernel           zio_read_intr_3  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100261 kernel           zio_read_intr_4  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100262 kernel           zio_read_intr_5  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100263 kernel           zio_read_intr_6  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100264 kernel           zio_read_intr_7  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100265 kernel           zio_read_intr_8  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100266 kernel           zio_read_intr_9  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100267 kernel           zio_read_intr_10 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100268 kernel           zio_read_intr_11 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100269 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100270 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100271 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100272 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100273 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100274 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100275 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100276 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100277 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100278 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100279 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100280 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100281 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100282 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100283 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100284 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100285 kernel           zio_write_issue_ mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100286 kernel           zio_write_intr_0 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100287 kernel           zio_write_intr_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100288 kernel           zio_write_intr_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100289 kernel           zio_write_intr_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100290 kernel           zio_write_intr_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100291 kernel           zio_write_intr_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100292 kernel           zio_write_intr_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100293 kernel           zio_write_intr_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100294 kernel           zio_write_intr_h mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100295 kernel           zio_write_intr_h mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100296 kernel           zio_write_intr_h mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100297 kernel           zio_write_intr_h mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100298 kernel           zio_write_intr_h mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100299 kernel           zio_free_issue_0 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100300 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100301 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100302 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100303 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100304 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100305 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100306 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100307 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100308 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100309 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100310 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100311 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100312 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100313 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100314 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100315 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100316 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100317 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100318 kernel           zio_free_issue_1 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100319 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100320 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100321 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100322 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100323 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100324 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100325 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100326 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100327 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100328 kernel           zio_free_issue_2 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100329 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100330 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100331 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100332 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100333 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100334 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100335 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100336 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100337 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100338 kernel           zio_free_issue_3 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100339 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100340 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100341 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100342 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100343 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100344 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100345 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100346 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100347 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100348 kernel           zio_free_issue_4 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100349 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100350 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100351 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100352 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100353 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100354 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100355 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100356 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100357 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100358 kernel           zio_free_issue_5 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100359 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100360 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100361 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100362 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100363 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100364 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100365 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100366 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100367 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100368 kernel           zio_free_issue_6 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100369 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100370 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100371 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100372 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100373 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100374 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100375 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100376 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100377 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100378 kernel           zio_free_issue_7 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100379 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100380 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100381 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100382 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100383 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100384 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100385 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100386 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100387 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100388 kernel           zio_free_issue_8 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100389 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100390 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100391 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100392 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100393 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100394 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100395 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100396 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100397 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100398 kernel           zio_free_issue_9 mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100399 kernel           zio_free_intr  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100400 kernel           zio_claim_issue  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100401 kernel           zio_claim_intr mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100402 kernel           zio_ioctl_issue  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100403 kernel           zio_ioctl_intr mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100405 kernel           zfs_vn_rele_task mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100418 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100441 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100477 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100478 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100479 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100480 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100481 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100482 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100484 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100486 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100488 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100489 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100490 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100491 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100492 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100493 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100494 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100495 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100496 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100497 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100498 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100499 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100636 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         0 100638 kernel           zil_clean  mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
>     fork_exit+0x11f fork_trampoline+0xe
>         1 100002 init             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>         2 100033 crypto           -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f fork_trampoline+0xe
>     3 100034 <tel:3%20100034> crypto returns   -              
>      mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
>     crypto_ret_proc+0x192 fork_exit+0x11f fork_trampoline+0xe
>         4 100061 ctl_thrd         -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f
>     fork_trampoline+0xe
>         5 100075 zfskern          arc_reclaim_thre mi_switch+0x186
>     sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d
>     fork_exit+0x11f fork_trampoline+0xe
>         5 100076 zfskern          l2arc_feed_threa mi_switch+0x186
>     sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2
>     fork_exit+0x11f fork_trampoline+0xe
>         5 100404 zfskern          trim nsgroot mi_switch+0x186
>     sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68
>     fork_exit+0x11f fork_trampoline+0xe
>         5 100406 zfskern          txg_thread_enter mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79
>     txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe
>         5 100407 zfskern          txg_thread_enter mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35
>     dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba
>     txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
>         5 100408 zfskern          zvol nsgroot/swa mi_switch+0x186
>     sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc
>     fork_exit+0x11f fork_trampoline+0xe
>         6 100077 sctp_iterator    -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f
>     fork_trampoline+0xe
>         7 100078 xpt_thrd         -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f
>     fork_trampoline+0xe
>         8 100079 ipmi0: kcs       -  mi_switch+0x186 sleepq_wait+0x42
>     _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d
>     fork_exit+0x11f fork_trampoline+0xe
>         9 100080 enc_daemon0      -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f fork_trampoline+0xe
>        10 100001 audit            -  mi_switch+0x186 sleepq_wait+0x42
>     _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f fork_trampoline+0xe
>        11 100003 idle             idle: cpu0 <running>
>        11 100004 idle             idle: cpu1 <running>
>        11 100005 idle             idle: cpu2 <running>
>        11 100006 idle             idle: cpu3 <running>
>        11 100007 idle             idle: cpu4 <running>
>        11 100008 idle             idle: cpu5 <running>
>        11 100009 idle             idle: cpu6 <running>
>        11 100010 idle             idle: cpu7 <running>
>        11 100011 idle             idle: cpu8 <running>
>        11 100012 idle             idle: cpu9 <running>
>        11 100013 idle             idle: cpu10  mi_switch+0x186
>     critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f
>     fork_trampoline+0xe
>        11 100014 idle             idle: cpu11 <running>
>        12 100015 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100016 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100017 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100018 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100019 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100020 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100021 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100022 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100023 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100024 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100025 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100026 intr             swi4: clock  mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100027 intr             swi3: vm
>        12 100028 intr             swi1: netisr 0 mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100036 intr             swi6: task queue mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100042 intr             swi2: cambio mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100043 intr             swi5: fast taskq
>        12 100045 intr             swi6: Giant task mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100046 intr             irq264: mfi0 mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100047 intr             irq23: ehci0 mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100054 intr             irq22: ehci1 mi_switch+0x186
>     ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>        12 100059 intr             irq267: ahci0
>        12 100060 intr             swi0: uart uart
>     13 100029 <tel:13%20100029> geom       g_event        
>      mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440
>     fork_exit+0x11f fork_trampoline+0xe
>     13 100030 <tel:13%20100030> geom       g_up            
>     mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
>     g_io_schedule_up+0xe6 g_up_procbody+0x5c fork_exit+0x11f
>     fork_trampoline+0xe
>     13 100031 <tel:13%20100031> geom       g_down          
>     mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
>     g_io_schedule_down+0x25f g_down_procbody+0x5c fork_exit+0x11f
>     fork_trampoline+0xe
>        14 100035 yarrow           -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea
>     fork_exit+0x11f fork_trampoline+0xe
>        15 100048 usb              usbus0 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100049 usb              usbus0 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100050 usb              usbus0 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100051 usb              usbus0 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100055 usb              usbus1 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100056 usb              usbus1 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100057 usb              usbus1 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        15 100058 usb              usbus1 mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>     fork_trampoline+0xe
>        16 100081 pagedaemon       -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34
>     fork_exit+0x11f fork_trampoline+0xe
>        17 100082 vmdaemon         -  mi_switch+0x186 sleepq_wait+0x42
>     _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f fork_trampoline+0xe
>        18 100083 pagezero         -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83
>     fork_exit+0x11f fork_trampoline+0xe
>        19 100084 bufdaemon        -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1
>     fork_exit+0x11f fork_trampoline+0xe
>        20 100085 syncer           -  mi_switch+0x186 sleepq_wait+0x42
>     _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sync_fsync+0x1a2
>     VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5
>     fork_exit+0x11f fork_trampoline+0xe
>        21 100086 vnlru            -  mi_switch+0x186 sleepq_wait+0x42
>     _sx_slock_hard+0x318 _sx_slock+0x56 zfs_freebsd_reclaim+0x4b
>     VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
>     vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe
>        22 100087 softdepflush     -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375
>     fork_exit+0x11f fork_trampoline+0xe
>       782 100432 devd             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef
>     sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7
>       790 100545 pfpurge          -  mi_switch+0x186
>     sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31
>     fork_exit+0x11f fork_trampoline+0xe
>       913 100449 syslogd          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1014 100581 ntpd             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1028 100532 sshd             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1032 100473 cron             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca
>     kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1135 100558 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1145 100509 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1156 100508 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1161 100507 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1163 100521 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1168 100574 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1170 100561 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1174 100560 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1180 100420 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1181 100463 getty            -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
>     devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1182 100430 getty            -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
>     devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1183 100421 CustomInit          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>      1443 100599 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1444 100527 snapshot-manager     -  mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85
>     dsl_sync_task+0x139 dsl_destroy_snapshots_nvl+0x71
>     dsl_destroy_snapshot+0x4a zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d
>     devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1445 100514 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1447 100557 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1478 100555 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1479 100551 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1486 100576 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1488 100563 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1494 100575 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1495 100442 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1496 100556 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1505 100431 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1511 100475 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1531 100429 snapshot-manager     -  mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85
>     dsl_sync_task+0x139 zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d
>     devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1535 100434 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1545 100459 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1567 100539 openvpn          -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>     _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>      1628 100454 snapshot-manager     -  mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
>     dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
>     zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>     sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>     2097 100710 <tel:2097%20100710> java             -              
>      mi_switch+0x186 thread_suspend_switch+0xcd thread_single+0x1b2
>     exit1+0x72 sys_sys_exit+0xe amd64_syscall+0x540 Xfast_syscall+0xf7
>     2097 100792 <tel:2097%20100792> java             -              
>      mi_switch+0x186 sleepq_wait+0x42 __lockmgr_args+0x5cb
>     vop_stdlock+0x39 VOP_LOCK1_APV+0x70 _vn_lock+0x47
>     zfsctl_freebsd_root_lookup+0xd5 VOP_LOOKUP_APV+0x62 lookup+0x437
>     namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>     2102 100635 <tel:2102%20100635> data-processor dm_siri_to_chiro
>     mi_switch+0x186 sleepq_wait+0x42 _sx_slock_hard+0x318
>     _sx_slock+0x56 zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68
>     vgonel+0x134 vnlru_free+0x362 getnewvnode+0x27d
>     gfs_file_create+0x4b gfs_dir_create+0x16
>     zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 lookup+0x437
>     namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
>     75715 100424 snapshot-manager-counts mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
>     dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
>     zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>     sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>     75718 100861 snapshot-manager-counts mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
>     dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
>     zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>     sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>     75719 100578 snapshot-manager-counts mi_switch+0x186
>     sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
>     dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
>     zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>     sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>     76069 101690 reboot           -  mi_switch+0x186 sleepq_wait+0x42
>     _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sys_sync+0x1e8
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>     76196 100847 sshd             -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>     76198 101378 csh              -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>     76200 100892 top              -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>     tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
>     devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
>     amd64_syscall+0x540 Xfast_syscall+0xf7
>     76203 100447 sshd             - <running>
>     76205 101410 csh              -  mi_switch+0x186
>     sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>     kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>     Xfast_syscall+0xf7
>     76209 101030 procstat         -  <running>
>
>
>     _______________________________________________
>     freebsd-fs@freebsd.org <mailto:freebsd-fs@freebsd.org> mailing list
>     http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>     To unsubscribe, send any mail to
>     "freebsd-fs-unsubscribe@freebsd.org
>     <mailto:freebsd-fs-unsubscribe@freebsd.org>"
>
>




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52AC0508.5060704>