Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 14 Dec 2013 00:55:27 -0000
From:      "Steven Hartland" <killing@multiplay.co.uk>
To:        "Rod Taylor" <rod.taylor@gmail.com>, "Ryan Baldwin" <ryan.baldwin@nexusalpha.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: ZFS related hang with FreeBSD 9.2
Message-ID:  <E01DB08EA47D4897B73F8CCD55FFB103@multiplay.co.uk>
References:  <52AA97B8.8060408@nexusalpha.com> <CAKddOFATpE0U8Z0AYhsBPwfym3XJWeh95Q%2BS3Ug_YZJmkhUKCQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Are you doing any snapshot sends as well as interacting with
snapshots such as listing files in them via the .zfs?

If so make sure you have the following patch applied as
that can cause a deadlock between these two operations
http://svnweb.freebsd.org/changeset/base/258595

    Regards
    Steve

----- Original Message ----- 
From: "Rod Taylor" <rod.taylor@gmail.com>
To: "Ryan Baldwin" <ryan.baldwin@nexusalpha.com>
Cc: <freebsd-fs@freebsd.org>
Sent: Friday, December 13, 2013 11:21 PM
Subject: Re: ZFS related hang with FreeBSD 9.2


> Are you using snapshots?
> 
> I've found ZFS Snapshots on 9.0, 9.1, and 9.2 regularly crash the system.
> Delete the snapshots and don't create any new ones and suddenly it's stable
> for months.
> 
> 
> 
> On Fri, Dec 13, 2013 at 12:14 AM, Ryan Baldwin
> <ryan.baldwin@nexusalpha.com>wrote:
> 
>> Hi,
>>
>> We have a server based on FreeBSD 9.2 which hangs at times on a daily
>> basis. The longest uptime we have achieved is 5 days conversely it has
>> stopped daily several days in a row.
>>
>> When this occurs it appears there are two proceses stuck in 'tx->tx'
>> state. In the top output shown these are snapshot-manager processes which
>> create and destroy snapshots generally and sometime rollback filesystems to
>> snapshots. When the lockup occurs other processes which try to access the
>> file system can seem to end up stuck in state 'rrl->r'. The reboot command
>> that was issued to try and reboot the server has ended up stuck in this
>> state as can be seen.
>>
>> The server is not under particularly heavy load.
>>
>> It has remained in this state for hours. The 'deadman handler'? does not
>> appear to restart the system. Once this has occurred there is no further
>> disk activity.
>>
>> We did not experience this problem at all previously using 9.1 although we
>> had less snapshot-manager processes before. We have built this server
>> against 9.1 again now but it has only been going one day so far.
>>
>> We can try and reproduce this problem again on 9.2 if by doing so we can
>> gather any additional information that could help resolve this problem.
>> Please let me know what other information would be helpful.
>>
>> The hardware is a Dell R420 with Perc H310 raid controller in JBOD mode
>> with the pool mirrored on two SAS disks.
>>
>> Thanks
>>
>> top and procstat output follow:
>>
>>
>> last pid: 76225;  load averages:  0.00,  0.00,  0.00       up 0+20:04:22
>>  09:56:24
>> 46 processes:  1 running, 44 sleeping, 1 stopped
>> CPU:     % user,     % nice,     % system,     % interrupt,     % idle
>> Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free
>> ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other
>> Swap: 8192M Total, 8192M Free
>>
>>   PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>> COMMAND
>>  1170 root          1  20    0 16244K  5460K kqread 11   1:28  0.00%
>> CustomInit
>>  1531 root          1  20    0 39648K  6368K tx->tx  7   1:17 0.00%
>> snapshot-manager
>>  1135 root          1  20    0 16244K  5100K kqread  1   1:10  0.00%
>> CustomInit
>>  2097 root          2  20    0  2581M   472M STOP    4   0:46  0.00% java
>>  1183 root          1  20    0 16244K  5504K kqread  0   0:22  0.00%
>> CustomInit
>>  1145 root          1  20    0 16244K  5464K kqread  9   0:18  0.00%
>> CustomInit
>>  1444 root          1  20    0 39648K  6352K tx->tx  5   0:13 0.00%
>> snapshot-manager
>>  1628 root          1  20    0 39648K  6348K rrl->r 10   0:12 0.00%
>> snapshot-manager
>>  1168 root          1  20    0 16244K  5388K kqread  7   0:07  0.00%
>> CustomInit
>>  1535 root          1  20    0 22388K 14296K select  4   0:05  0.00%
>> openvpn
>>  1163 root          1  20    0 16244K  5388K kqread  2   0:04  0.00%
>> CustomInit
>>  1511 root          1  20    0 18292K 10192K select  0   0:04  0.00%
>> openvpn
>>  1156 root          1  20    0 16244K  5392K kqread 10   0:04  0.00%
>> CustomInit
>>  1174 root          1  20    0 16244K  5388K kqread  1   0:04  0.00%
>> CustomInit
>>  1161 root          1  20    0 16244K  5392K kqread  4   0:03  0.00%
>> CustomInit
>>   913 root          1  20    0 12076K  1820K select  7   0:03  0.00%
>> syslogd
>>  2102 root          1  20    0   109M 13616K zfsvfs  3   0:03  0.00%
>> data-processor
>>  1445 root          1  20    0 22388K 14296K select  5   0:02  0.00%
>> openvpn
>>  1014 root          1  20    0 22256K  3360K select  4   0:02  0.00% ntpd
>>  1494 root          1  20    0 18292K 10192K select  2   0:01  0.00%
>> openvpn
>>  1180 root          1  20    0 18292K  4780K select  0   0:01  0.00%
>> openvpn
>>  1505 root          1  20    0 18292K  4712K select  2   0:01  0.00%
>> openvpn
>>  1495 root          1  20    0 18292K  4712K select  5   0:00  0.00%
>> openvpn
>>  1479 root          1  20    0 18292K  4708K select  1   0:00  0.00%
>> openvpn
>>  1567 root          1  20    0 18292K  4712K select  3   0:00  0.00%
>> openvpn
>>  1545 root          1  20    0 18292K  4712K select  1   0:00  0.00%
>> openvpn
>>  1486 root          1  20    0 18292K  4708K select  0   0:00  0.00%
>> openvpn
>>  1443 root          1  20    0 18292K  4436K select  9   0:00  0.00%
>> openvpn
>>  1447 root          1  20    0 18292K  4448K select  4   0:00  0.00%
>> openvpn
>>  1496 root          1  20    0 18292K  4708K select  3   0:00  0.00%
>> openvpn
>>  1478 root          1  20    0 18292K  4704K select  5   0:00  0.00%
>> openvpn
>>  1488 root          1  20    0 18292K 10192K select  5   0:00  0.00%
>> openvpn
>>  1032 root          1  20    0 14176K  1860K nanslp  1   0:00  0.00% cron
>> 76069 root          1  20    0  9948K  1680K rrl->r  7   0:00 0.00% reboot
>>   782 root          1  20    0 10376K  4416K select  5   0:00  0.00% devd
>> 76196 root          1  20    0 51536K  5828K select  0   0:00  0.00% sshd
>> 76203 root          1  20    0 51536K  5828K select  2   0:00  0.00% sshd
>> 76205 root          1  20    0 17564K  3252K ttyin  11   0:00  0.00% csh
>> 76198 root          1  20    0 17564K  3252K pause   8   0:00  0.00% csh
>> 75718 root          1  20    0 35556K  3956K rrl->r 11   0:00 0.00%
>> snapshot-manager-counts
>> 75715 root          1  20    0 35556K  3956K rrl->r 10   0:00 0.00%
>> snapshot-manager-counts
>>  1181 root          1  52    0 12084K  1676K ttyin   1   0:00  0.00% getty
>>
>>
>> 0 100000 kernel           swapper          mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77
>> btext+0x2c
>>     0 100032 kernel           firmware taskq   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100037 kernel           ffs_trim taskq   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100038 kernel           acpi_task_0      mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100039 kernel           acpi_task_1      mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100040 kernel           acpi_task_2      mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100041 kernel           kqueue taskq     mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100044 kernel           thread taskq     mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100052 kernel           bge0 taskq       mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100053 kernel           bge1 taskq       mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100062 kernel           mca taskq        mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>>     0 100063 kernel           system_taskq_0   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100064 kernel           system_taskq_1   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100065 kernel           system_taskq_2   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100066 kernel           system_taskq_3   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100067 kernel           system_taskq_4   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100068 kernel           system_taskq_5   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100069 kernel           system_taskq_6   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100070 kernel           system_taskq_7   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100071 kernel           system_taskq_8   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100072 kernel           system_taskq_9   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100073 kernel           system_taskq_10  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100074 kernel           system_taskq_11  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100247 kernel           zio_null_issue   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100248 kernel           zio_null_intr    mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100249 kernel           zio_read_issue_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100250 kernel           zio_read_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100251 kernel           zio_read_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100252 kernel           zio_read_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100253 kernel           zio_read_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100254 kernel           zio_read_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100255 kernel           zio_read_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100256 kernel           zio_read_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100257 kernel           zio_read_intr_0  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100258 kernel           zio_read_intr_1  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100259 kernel           zio_read_intr_2  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100260 kernel           zio_read_intr_3  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100261 kernel           zio_read_intr_4  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100262 kernel           zio_read_intr_5  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100263 kernel           zio_read_intr_6  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100264 kernel           zio_read_intr_7  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100265 kernel           zio_read_intr_8  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100266 kernel           zio_read_intr_9  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100267 kernel           zio_read_intr_10 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100268 kernel           zio_read_intr_11 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100269 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100270 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100271 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100272 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100273 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100274 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100275 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100276 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100277 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100278 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100279 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100280 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100281 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100282 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100283 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100284 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100285 kernel           zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100286 kernel           zio_write_intr_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100287 kernel           zio_write_intr_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100288 kernel           zio_write_intr_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100289 kernel           zio_write_intr_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100290 kernel           zio_write_intr_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100291 kernel           zio_write_intr_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100292 kernel           zio_write_intr_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100293 kernel           zio_write_intr_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100294 kernel           zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100295 kernel           zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100296 kernel           zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100297 kernel           zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100298 kernel           zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100299 kernel           zio_free_issue_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100300 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100301 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100302 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100303 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100304 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100305 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100306 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100307 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100308 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100309 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100310 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100311 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100312 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100313 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100314 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100315 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100316 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100317 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100318 kernel           zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100319 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100320 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100321 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100322 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100323 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100324 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100325 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100326 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100327 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100328 kernel           zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100329 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100330 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100331 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100332 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100333 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100334 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100335 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100336 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100337 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100338 kernel           zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100339 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100340 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100341 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100342 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100343 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100344 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100345 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100346 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100347 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100348 kernel           zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100349 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100350 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100351 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100352 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100353 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100354 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100355 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100356 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100357 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100358 kernel           zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100359 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100360 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100361 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100362 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100363 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100364 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100365 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100366 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100367 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100368 kernel           zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100369 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100370 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100371 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100372 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100373 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100374 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100375 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100376 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100377 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100378 kernel           zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100379 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100380 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100381 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100382 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100383 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100384 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100385 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100386 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100387 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100388 kernel           zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100389 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100390 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100391 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100392 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100393 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100394 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100395 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100396 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100397 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100398 kernel           zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100399 kernel           zio_free_intr    mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100400 kernel           zio_claim_issue  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100401 kernel           zio_claim_intr   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100402 kernel           zio_ioctl_issue  mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100403 kernel           zio_ioctl_intr   mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100405 kernel           zfs_vn_rele_task mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100418 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100441 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100477 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100478 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100479 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100480 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100481 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100482 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100484 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100486 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100488 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100489 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100490 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100491 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100492 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100493 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100494 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100495 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100496 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100497 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100498 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100499 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100636 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     0 100638 kernel           zil_clean        mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>>     1 100002 init             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>     2 100033 crypto           -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f
>> fork_trampoline+0xe
>>     3 100034 crypto returns   -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 crypto_ret_proc+0x192 fork_exit+0x11f
>> fork_trampoline+0xe
>>     4 100061 ctl_thrd         -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f
>> fork_trampoline+0xe
>>     5 100075 zfskern          arc_reclaim_thre mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d
>> fork_exit+0x11f fork_trampoline+0xe
>>     5 100076 zfskern          l2arc_feed_threa mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2
>> fork_exit+0x11f fork_trampoline+0xe
>>     5 100404 zfskern          trim nsgroot     mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68 fork_exit+0x11f
>> fork_trampoline+0xe
>>     5 100406 zfskern          txg_thread_enter mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79
>> txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe
>>     5 100407 zfskern          txg_thread_enter mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35
>> dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba
>> txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
>>     5 100408 zfskern          zvol nsgroot/swa mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc fork_exit+0x11f
>> fork_trampoline+0xe
>>     6 100077 sctp_iterator    -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f
>> fork_trampoline+0xe
>>     7 100078 xpt_thrd         -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f
>> fork_trampoline+0xe
>>     8 100079 ipmi0: kcs       -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d
>> fork_exit+0x11f fork_trampoline+0xe
>>     9 100080 enc_daemon0      -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f
>> fork_trampoline+0xe
>>    10 100001 audit            -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f
>> fork_trampoline+0xe
>>    11 100003 idle             idle: cpu0 <running>
>>    11 100004 idle             idle: cpu1 <running>
>>    11 100005 idle             idle: cpu2 <running>
>>    11 100006 idle             idle: cpu3 <running>
>>    11 100007 idle             idle: cpu4 <running>
>>    11 100008 idle             idle: cpu5 <running>
>>    11 100009 idle             idle: cpu6 <running>
>>    11 100010 idle             idle: cpu7 <running>
>>    11 100011 idle             idle: cpu8 <running>
>>    11 100012 idle             idle: cpu9 <running>
>>    11 100013 idle             idle: cpu10      mi_switch+0x186
>> critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f fork_trampoline+0xe
>>    11 100014 idle             idle: cpu11 <running>
>>    12 100015 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100016 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100017 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100018 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100019 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100020 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100021 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100022 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100023 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100024 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100025 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100026 intr             swi4: clock      mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100027 intr             swi3: vm
>>    12 100028 intr             swi1: netisr 0   mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100036 intr             swi6: task queue mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100042 intr             swi2: cambio     mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100043 intr             swi5: fast taskq
>>    12 100045 intr             swi6: Giant task mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100046 intr             irq264: mfi0     mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100047 intr             irq23: ehci0     mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100054 intr             irq22: ehci1     mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>>    12 100059 intr             irq267: ahci0
>>    12 100060 intr             swi0: uart uart
>>    13 100029 geom             g_event          mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440 fork_exit+0x11f
>> fork_trampoline+0xe
>>    13 100030 geom             g_up             mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_io_schedule_up+0xe6 g_up_procbody+0x5c
>> fork_exit+0x11f fork_trampoline+0xe
>>    13 100031 geom             g_down           mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_io_schedule_down+0x25f g_down_procbody+0x5c
>> fork_exit+0x11f fork_trampoline+0xe
>>    14 100035 yarrow           -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100048 usb              usbus0           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100049 usb              usbus0           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100050 usb              usbus0           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100051 usb              usbus0           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100055 usb              usbus1           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100056 usb              usbus1           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100057 usb              usbus1           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    15 100058 usb              usbus1           mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>>    16 100081 pagedaemon       -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34 fork_exit+0x11f
>> fork_trampoline+0xe
>>    17 100082 vmdaemon         -                mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f
>> fork_trampoline+0xe
>>    18 100083 pagezero         -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83 fork_exit+0x11f
>> fork_trampoline+0xe
>>    19 100084 bufdaemon        -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1 fork_exit+0x11f
>> fork_trampoline+0xe
>>    20 100085 syncer           -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e
>> sync_fsync+0x1a2 VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5
>> fork_exit+0x11f fork_trampoline+0xe
>>    21 100086 vnlru            -                mi_switch+0x186
>> sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56
>> zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
>> vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe
>>    22 100087 softdepflush     -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375 fork_exit+0x11f
>> fork_trampoline+0xe
>>   782 100432 devd             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef sys_select+0x5d
>> amd64_syscall+0x540 Xfast_syscall+0xf7
>>   790 100545 pfpurge          -                mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31 fork_exit+0x11f
>> fork_trampoline+0xe
>>   913 100449 syslogd          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1014 100581 ntpd             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1028 100532 sshd             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1032 100473 cron             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca
>> kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1135 100558 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1145 100509 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1156 100508 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1161 100507 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1163 100521 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1168 100574 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1170 100561 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1174 100560 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1180 100420 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1181 100463 getty            -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1182 100430 getty            -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1183 100421 CustomInit          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1443 100599 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1444 100527 snapshot-manager     -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139
>> dsl_destroy_snapshots_nvl+0x71 dsl_destroy_snapshot+0x4a
>> zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1445 100514 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1447 100557 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1478 100555 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1479 100551 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1486 100576 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1488 100563 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1494 100575 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1495 100442 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1496 100556 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1505 100431 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1511 100475 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1531 100429 snapshot-manager     -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139
>> zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>>  1535 100434 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1545 100459 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1567 100539 openvpn          -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  1628 100454 snapshot-manager     -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44
>> dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d
>> devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540
>> Xfast_syscall+0xf7
>>  2097 100710 java             -                mi_switch+0x186
>> thread_suspend_switch+0xcd thread_single+0x1b2 exit1+0x72 sys_sys_exit+0xe
>> amd64_syscall+0x540 Xfast_syscall+0xf7
>>  2097 100792 java             -                mi_switch+0x186
>> sleepq_wait+0x42 __lockmgr_args+0x5cb vop_stdlock+0x39 VOP_LOCK1_APV+0x70
>> _vn_lock+0x47 zfsctl_freebsd_root_lookup+0xd5 VOP_LOOKUP_APV+0x62
>> lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15
>> sys_stat+0x2a amd64_syscall+0x540 Xfast_syscall+0xf7
>>  2102 100635 data-processor dm_siri_to_chiro mi_switch+0x186
>> sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56
>> zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
>> getnewvnode+0x27d gfs_file_create+0x4b gfs_dir_create+0x16
>> zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 lookup+0x437 namei+0x4ac
>> kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
>> 75715 100424 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 75718 100861 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 75719 100578 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 76069 101690 reboot           -                mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e
>> sys_sync+0x1e8 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 76196 100847 sshd             -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76198 101378 csh              -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76200 100892 top              -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76203 100447 sshd             - <running>
>> 76205 101410 csh              -                mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76209 101030 procstat         -                <running>
>>
>>
>> _______________________________________________
>> freebsd-fs@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E01DB08EA47D4897B73F8CCD55FFB103>