Date: Fri, 13 Dec 2013 05:14:32 +0000 From: Ryan Baldwin <ryan.baldwin@nexusalpha.com> To: freebsd-fs@freebsd.org Subject: ZFS related hang with FreeBSD 9.2 Message-ID: <52AA97B8.8060408@nexusalpha.com>
next in thread | raw e-mail | index | archive | help
Hi, We have a server based on FreeBSD 9.2 which hangs at times on a daily basis. The longest uptime we have achieved is 5 days conversely it has stopped daily several days in a row. When this occurs it appears there are two proceses stuck in 'tx->tx' state. In the top output shown these are snapshot-manager processes which create and destroy snapshots generally and sometime rollback filesystems to snapshots. When the lockup occurs other processes which try to access the file system can seem to end up stuck in state 'rrl->r'. The reboot command that was issued to try and reboot the server has ended up stuck in this state as can be seen. The server is not under particularly heavy load. It has remained in this state for hours. The 'deadman handler'? does not appear to restart the system. Once this has occurred there is no further disk activity. We did not experience this problem at all previously using 9.1 although we had less snapshot-manager processes before. We have built this server against 9.1 again now but it has only been going one day so far. We can try and reproduce this problem again on 9.2 if by doing so we can gather any additional information that could help resolve this problem. Please let me know what other information would be helpful. The hardware is a Dell R420 with Perc H310 raid controller in JBOD mode with the pool mirrored on two SAS disks. Thanks top and procstat output follow: last pid: 76225; load averages: 0.00, 0.00, 0.00 up 0+20:04:22 09:56:24 46 processes: 1 running, 44 sleeping, 1 stopped CPU: % user, % nice, % system, % interrupt, % idle Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1170 root 1 20 0 16244K 5460K kqread 11 1:28 0.00% CustomInit 1531 root 1 20 0 39648K 6368K tx->tx 7 1:17 0.00% snapshot-manager 1135 root 1 20 0 16244K 5100K kqread 1 1:10 0.00% CustomInit 2097 root 2 20 0 2581M 472M STOP 4 0:46 0.00% java 1183 root 1 20 0 16244K 5504K kqread 0 0:22 0.00% CustomInit 1145 root 1 20 0 16244K 5464K kqread 9 0:18 0.00% CustomInit 1444 root 1 20 0 39648K 6352K tx->tx 5 0:13 0.00% snapshot-manager 1628 root 1 20 0 39648K 6348K rrl->r 10 0:12 0.00% snapshot-manager 1168 root 1 20 0 16244K 5388K kqread 7 0:07 0.00% CustomInit 1535 root 1 20 0 22388K 14296K select 4 0:05 0.00% openvpn 1163 root 1 20 0 16244K 5388K kqread 2 0:04 0.00% CustomInit 1511 root 1 20 0 18292K 10192K select 0 0:04 0.00% openvpn 1156 root 1 20 0 16244K 5392K kqread 10 0:04 0.00% CustomInit 1174 root 1 20 0 16244K 5388K kqread 1 0:04 0.00% CustomInit 1161 root 1 20 0 16244K 5392K kqread 4 0:03 0.00% CustomInit 913 root 1 20 0 12076K 1820K select 7 0:03 0.00% syslogd 2102 root 1 20 0 109M 13616K zfsvfs 3 0:03 0.00% data-processor 1445 root 1 20 0 22388K 14296K select 5 0:02 0.00% openvpn 1014 root 1 20 0 22256K 3360K select 4 0:02 0.00% ntpd 1494 root 1 20 0 18292K 10192K select 2 0:01 0.00% openvpn 1180 root 1 20 0 18292K 4780K select 0 0:01 0.00% openvpn 1505 root 1 20 0 18292K 4712K select 2 0:01 0.00% openvpn 1495 root 1 20 0 18292K 4712K select 5 0:00 0.00% openvpn 1479 root 1 20 0 18292K 4708K select 1 0:00 0.00% openvpn 1567 root 1 20 0 18292K 4712K select 3 0:00 0.00% openvpn 1545 root 1 20 0 18292K 4712K select 1 0:00 0.00% openvpn 1486 root 1 20 0 18292K 4708K select 0 0:00 0.00% openvpn 1443 root 1 20 0 18292K 4436K select 9 0:00 0.00% openvpn 1447 root 1 20 0 18292K 4448K select 4 0:00 0.00% openvpn 1496 root 1 20 0 18292K 4708K select 3 0:00 0.00% openvpn 1478 root 1 20 0 18292K 4704K select 5 0:00 0.00% openvpn 1488 root 1 20 0 18292K 10192K select 5 0:00 0.00% openvpn 1032 root 1 20 0 14176K 1860K nanslp 1 0:00 0.00% cron 76069 root 1 20 0 9948K 1680K rrl->r 7 0:00 0.00% reboot 782 root 1 20 0 10376K 4416K select 5 0:00 0.00% devd 76196 root 1 20 0 51536K 5828K select 0 0:00 0.00% sshd 76203 root 1 20 0 51536K 5828K select 2 0:00 0.00% sshd 76205 root 1 20 0 17564K 3252K ttyin 11 0:00 0.00% csh 76198 root 1 20 0 17564K 3252K pause 8 0:00 0.00% csh 75718 root 1 20 0 35556K 3956K rrl->r 11 0:00 0.00% snapshot-manager-counts 75715 root 1 20 0 35556K 3956K rrl->r 10 0:00 0.00% snapshot-manager-counts 1181 root 1 52 0 12084K 1676K ttyin 1 0:00 0.00% getty 0 100000 kernel swapper mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77 btext+0x2c 0 100032 kernel firmware taskq mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100037 kernel ffs_trim taskq mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100038 kernel acpi_task_0 mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100039 kernel acpi_task_1 mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100040 kernel acpi_task_2 mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100041 kernel kqueue taskq mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100044 kernel thread taskq mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100052 kernel bge0 taskq mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100053 kernel bge1 taskq mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100062 kernel mca taskq mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 0 100063 kernel system_taskq_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100064 kernel system_taskq_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100065 kernel system_taskq_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100066 kernel system_taskq_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100067 kernel system_taskq_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100068 kernel system_taskq_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100069 kernel system_taskq_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100070 kernel system_taskq_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100071 kernel system_taskq_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100072 kernel system_taskq_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100073 kernel system_taskq_10 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100074 kernel system_taskq_11 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100247 kernel zio_null_issue mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100248 kernel zio_null_intr mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100249 kernel zio_read_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100250 kernel zio_read_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100251 kernel zio_read_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100252 kernel zio_read_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100253 kernel zio_read_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100254 kernel zio_read_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100255 kernel zio_read_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100256 kernel zio_read_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100257 kernel zio_read_intr_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100258 kernel zio_read_intr_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100259 kernel zio_read_intr_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100260 kernel zio_read_intr_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100261 kernel zio_read_intr_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100262 kernel zio_read_intr_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100263 kernel zio_read_intr_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100264 kernel zio_read_intr_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100265 kernel zio_read_intr_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100266 kernel zio_read_intr_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100267 kernel zio_read_intr_10 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100268 kernel zio_read_intr_11 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100269 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100270 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100271 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100272 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100273 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100274 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100275 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100276 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100277 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100278 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100279 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100280 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100281 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100282 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100283 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100284 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100285 kernel zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100286 kernel zio_write_intr_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100287 kernel zio_write_intr_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100288 kernel zio_write_intr_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100289 kernel zio_write_intr_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100290 kernel zio_write_intr_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100291 kernel zio_write_intr_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100292 kernel zio_write_intr_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100293 kernel zio_write_intr_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100294 kernel zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100295 kernel zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100296 kernel zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100297 kernel zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100298 kernel zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100299 kernel zio_free_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100300 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100301 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100302 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100303 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100304 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100305 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100306 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100307 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100308 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100309 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100310 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100311 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100312 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100313 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100314 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100315 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100316 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100317 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100318 kernel zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100319 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100320 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100321 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100322 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100323 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100324 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100325 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100326 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100327 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100328 kernel zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100329 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100330 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100331 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100332 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100333 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100334 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100335 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100336 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100337 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100338 kernel zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100339 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100340 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100341 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100342 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100343 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100344 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100345 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100346 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100347 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100348 kernel zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100349 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100350 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100351 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100352 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100353 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100354 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100355 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100356 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100357 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100358 kernel zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100359 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100360 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100361 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100362 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100363 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100364 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100365 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100366 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100367 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100368 kernel zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100369 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100370 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100371 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100372 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100373 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100374 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100375 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100376 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100377 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100378 kernel zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100379 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100380 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100381 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100382 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100383 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100384 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100385 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100386 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100387 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100388 kernel zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100389 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100390 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100391 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100392 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100393 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100394 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100395 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100396 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100397 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100398 kernel zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100399 kernel zio_free_intr mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100400 kernel zio_claim_issue mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100401 kernel zio_claim_intr mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100402 kernel zio_ioctl_issue mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100403 kernel zio_ioctl_intr mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100405 kernel zfs_vn_rele_task mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100418 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100441 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100477 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100478 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100479 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100480 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100481 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100482 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100484 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100486 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100488 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100489 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100490 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100491 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100492 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100493 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100494 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100495 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100496 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100497 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100498 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100499 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100636 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 0 100638 kernel zil_clean mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 1 100002 init - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540 Xfast_syscall+0xf7 2 100033 crypto - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f fork_trampoline+0xe 3 100034 crypto returns - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 crypto_ret_proc+0x192 fork_exit+0x11f fork_trampoline+0xe 4 100061 ctl_thrd - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f fork_trampoline+0xe 5 100075 zfskern arc_reclaim_thre mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d fork_exit+0x11f fork_trampoline+0xe 5 100076 zfskern l2arc_feed_threa mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2 fork_exit+0x11f fork_trampoline+0xe 5 100404 zfskern trim nsgroot mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68 fork_exit+0x11f fork_trampoline+0xe 5 100406 zfskern txg_thread_enter mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79 txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe 5 100407 zfskern txg_thread_enter mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35 dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe 5 100408 zfskern zvol nsgroot/swa mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc fork_exit+0x11f fork_trampoline+0xe 6 100077 sctp_iterator - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f fork_trampoline+0xe 7 100078 xpt_thrd - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f fork_trampoline+0xe 8 100079 ipmi0: kcs - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d fork_exit+0x11f fork_trampoline+0xe 9 100080 enc_daemon0 - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f fork_trampoline+0xe 10 100001 audit - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f fork_trampoline+0xe 11 100003 idle idle: cpu0 <running> 11 100004 idle idle: cpu1 <running> 11 100005 idle idle: cpu2 <running> 11 100006 idle idle: cpu3 <running> 11 100007 idle idle: cpu4 <running> 11 100008 idle idle: cpu5 <running> 11 100009 idle idle: cpu6 <running> 11 100010 idle idle: cpu7 <running> 11 100011 idle idle: cpu8 <running> 11 100012 idle idle: cpu9 <running> 11 100013 idle idle: cpu10 mi_switch+0x186 critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f fork_trampoline+0xe 11 100014 idle idle: cpu11 <running> 12 100015 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100016 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100017 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100018 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100019 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100020 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100021 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100022 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100023 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100024 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100025 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100026 intr swi4: clock mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100027 intr swi3: vm 12 100028 intr swi1: netisr 0 mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100036 intr swi6: task queue mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100042 intr swi2: cambio mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100043 intr swi5: fast taskq 12 100045 intr swi6: Giant task mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100046 intr irq264: mfi0 mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100047 intr irq23: ehci0 mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100054 intr irq22: ehci1 mi_switch+0x186 ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe 12 100059 intr irq267: ahci0 12 100060 intr swi0: uart uart 13 100029 geom g_event mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440 fork_exit+0x11f fork_trampoline+0xe 13 100030 geom g_up mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 g_io_schedule_up+0xe6 g_up_procbody+0x5c fork_exit+0x11f fork_trampoline+0xe 13 100031 geom g_down mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 g_io_schedule_down+0x25f g_down_procbody+0x5c fork_exit+0x11f fork_trampoline+0xe 14 100035 yarrow - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea fork_exit+0x11f fork_trampoline+0xe 15 100048 usb usbus0 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100049 usb usbus0 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100050 usb usbus0 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100051 usb usbus0 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100055 usb usbus1 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100056 usb usbus1 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100057 usb usbus1 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 15 100058 usb usbus1 mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe 16 100081 pagedaemon - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34 fork_exit+0x11f fork_trampoline+0xe 17 100082 vmdaemon - mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f fork_trampoline+0xe 18 100083 pagezero - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83 fork_exit+0x11f fork_trampoline+0xe 19 100084 bufdaemon - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1 fork_exit+0x11f fork_trampoline+0xe 20 100085 syncer - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sync_fsync+0x1a2 VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5 fork_exit+0x11f fork_trampoline+0xe 21 100086 vnlru - mi_switch+0x186 sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56 zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362 vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe 22 100087 softdepflush - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375 fork_exit+0x11f fork_trampoline+0xe 782 100432 devd - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7 790 100545 pfpurge - mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31 fork_exit+0x11f fork_trampoline+0xe 913 100449 syslogd - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7 1014 100581 ntpd - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7 1028 100532 sshd - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7 1032 100473 cron - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540 Xfast_syscall+0xf7 1135 100558 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1145 100509 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1156 100508 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1161 100507 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1163 100521 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1168 100574 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1170 100561 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1174 100560 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1180 100420 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1181 100463 getty - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 Xfast_syscall+0xf7 1182 100430 getty - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 Xfast_syscall+0xf7 1183 100421 CustomInit - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7 1443 100599 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1444 100527 snapshot-manager - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139 dsl_destroy_snapshots_nvl+0x71 dsl_destroy_snapshot+0x4a zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 1445 100514 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1447 100557 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1478 100555 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1479 100551 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1486 100576 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1488 100563 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1494 100575 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1495 100442 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1496 100556 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1505 100431 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1511 100475 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1531 100429 snapshot-manager - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139 zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 1535 100434 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1545 100459 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1567 100539 openvpn - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540 Xfast_syscall+0xf7 1628 100454 snapshot-manager - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 2097 100710 java - mi_switch+0x186 thread_suspend_switch+0xcd thread_single+0x1b2 exit1+0x72 sys_sys_exit+0xe amd64_syscall+0x540 Xfast_syscall+0xf7 2097 100792 java - mi_switch+0x186 sleepq_wait+0x42 __lockmgr_args+0x5cb vop_stdlock+0x39 VOP_LOCK1_APV+0x70 _vn_lock+0x47 zfsctl_freebsd_root_lookup+0xd5 VOP_LOOKUP_APV+0x62 lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a amd64_syscall+0x540 Xfast_syscall+0xf7 2102 100635 data-processor dm_siri_to_chiro mi_switch+0x186 sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56 zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362 getnewvnode+0x27d gfs_file_create+0x4b gfs_dir_create+0x16 zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a 75715 100424 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 75718 100861 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 75719 100578 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7 76069 101690 reboot - mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sys_sync+0x1e8 amd64_syscall+0x540 Xfast_syscall+0xf7 76196 100847 sshd - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7 76198 101378 csh - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540 Xfast_syscall+0xf7 76200 100892 top - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540 Xfast_syscall+0xf7 76203 100447 sshd - <running> 76205 101410 csh - mi_switch+0x186 sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295 kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540 Xfast_syscall+0xf7 76209 101030 procstat - <running>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52AA97B8.8060408>