Date: Wed, 28 Oct 2009 17:09:39 +0100 From: Borja Marcos <borjam@sarenet.es> To: freebsd-fs@freebsd.org Subject: 8.0-RC2: ZFS deadlock with zfs receive Message-ID: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>
next in thread | raw e-mail | index | archive | help
Hello, I've been sending some alltraces to pjd about this easy to reproduce problem. I am using zfs send/zfs receive to replicate a dataset from one server to another. At 1 minute intervals, an incremental snapshot is sent to update the dataset copy. If there is reading activity on the dataset copy, a deadlock can happen rendering ZFS and all the FS subsystem unusable. I've tried with 8.0RC2 and it still happens. How to reproduce: For example, while doing a make buildworld, replicate the dataset containing /usr/obj to the second machine. On the second machine a script just runs an endless loop copying all the contents of /usr/obj (which, remember, are being copied from the first machine using zfs receive) to another location, thus generating quite a lot of read activity. The time to deadlock is random, but soon the deadlock appears. The problem can be triggered by a backup accessing the destination dataset or when the security periodic scripts run at night. I tried to mitigate the problem by cloning a given snapshot at the destination and doing the backup from it, but the problem persists. Any ideas? Any way to mitigate this problem? Borja. This is the alltrace after breaking into ddb: >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 1914 root 1 62 0 11932K 2564K zfsvfs 0 0:51 0.00% >> bsdtar >> 1093 borjam 1 44 0 8304K 2464K CPU1 1 0:32 0.00% top >> 1913 root 1 54 0 11932K 2600K rrl->r 0 0:19 0.00% >> bsdtar >> 1019 root 1 44 0 25108K 4812K select 0 0:05 0.00% >> sshd >> 2008 root 1 76 0 13600K 1904K tx->tx 0 0:04 0.00% zfs >> 1089 borjam 1 44 0 37040K 5216K select 1 0:04 0.00% >> sshd >> 995 root 1 76 0 8252K 2652K pause 0 0:02 0.00% csh >> 840 root 1 44 0 11044K 3828K select 1 0:02 0.00% debug.ddb.capture.data: db> alltrace Tracing command sysctl pid 1373 tid 100183 td 0xffffff00177e5ab0 kdb_enter() at kdb_enter+0x3d kdb_sysctl_enter() at kdb_sysctl_enter+0x89 sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073d9dc, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe37c --- Tracing command csh pid 1369 tid 100181 td 0xffffff00177e6390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe638, rbp = 0x800c59400 --- Tracing command su pid 1368 tid 100159 td 0xffffff001284aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp = 0x7fffffffe598, rbp = 0x559 --- Tracing command unlink pid 1363 tid 100162 td 0xffffff001284a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1361 tid 100100 td 0xffffff000270bab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x551 --- Tracing command cron pid 1360 tid 100122 td 0xffffff0002af0390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1355 tid 100127 td 0xffffff0002aea000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1353 tid 100171 td 0xffffff00177e8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x549 --- Tracing command cron pid 1352 tid 100155 td 0xffffff001284d000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1347 tid 100157 td 0xffffff001284c720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1345 tid 100108 td 0xffffff0002916ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x541 --- Tracing command cron pid 1344 tid 100101 td 0xffffff000270b720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1337 tid 100172 td 0xffffff00177e8720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1334 tid 100098 td 0xffffff000270e390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x536 --- Tracing command newsyslog pid 1333 tid 100065 td 0xffffff00026d6ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ufs_lookup_() at ufs_lookup_+0xb46 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x2eb namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 stat() at stat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (188, FreeBSD ELF64, stat), rip = 0x80073251c, rsp = 0x7fffffffc2b8, rbp = 0x8009075e0 --- Tracing command cron pid 1331 tid 100147 td 0xffffff0002cb9000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command cron pid 1330 tid 100178 td 0xffffff00177e7000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1329 tid 100177 td 0xffffff00177e7390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1326 tid 100176 td 0xffffff00177e7720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x52e --- Tracing command cron pid 1324 tid 100115 td 0xffffff0002915000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1319 tid 100059 td 0xffffff000256a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1317 tid 100058 td 0xffffff0002650000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x525 --- Tracing command cron pid 1316 tid 100055 td 0xffffff0002650ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1309 tid 100112 td 0xffffff0002915ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1307 tid 100167 td 0xffffff0002cb8000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x51b --- Tracing command cron pid 1306 tid 100166 td 0xffffff0002af0720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command unlink pid 1301 tid 100164 td 0xffffff00026e7ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 lookup() at lookup+0x5f0 namei() at namei+0x4a9 kern_unlinkat() at kern_unlinkat+0xab syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffec68, rbp = 0 --- Tracing command sh pid 1299 tid 100103 td 0xffffff000270a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp = 0x7fffffffe618, rbp = 0x513 --- Tracing command cron pid 1298 tid 100160 td 0xffffff001284a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command sh pid 1289 tid 100161 td 0xffffff001284a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ufs_lookup_() at ufs_lookup_+0xb46 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x2eb namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 stat() at stat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (188, FreeBSD ELF64, stat), rip = 0x8009ae51c, rsp = 0x7fffffffe798, rbp = 0x800c06b80 --- Tracing command cron pid 1288 tid 100063 td 0xffffff000264f720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp = 0x7fffffffe4b8, rbp = 0x800a9e700 --- Tracing command zfs pid 1285 tid 100143 td 0xffffff0002cbb000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_wait_synced() at txg_wait_synced+0x7c zfsvfs_teardown() at zfsvfs_teardown+0x1b2 zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffbbf8, rbp = 0x7fffffffc930 --- Tracing command csh pid 1283 tid 100163 td 0xffffff0009979ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe598, rbp = 0x800c1d300 --- Tracing command sshd pid 1281 tid 100124 td 0xffffff0002aeaab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command zpool pid 1256 tid 100109 td 0xffffff0002916720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8010ce92c, rsp = 0x7fffffffaac8, rbp = 0 --- Tracing command csh pid 1252 tid 100158 td 0xffffff001284c000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe638, rbp = 0x800c59400 --- Tracing command su pid 1248 tid 100156 td 0xffffff001284cab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp = 0x7fffffffe598, rbp = 0x4e4 --- Tracing command csh pid 1243 tid 100137 td 0xffffff0002917720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe558, rbp = 0x800c1f700 --- Tracing command sshd pid 1242 tid 100154 td 0xffffff001284d390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1234 tid 100060 td 0xffffff000264fab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command bsdtar pid 1203 tid 100153 td 0xffffff000270aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp = 0x7fffffffe7a8, rbp = 0x80101a240 --- Tracing command bsdtar pid 1202 tid 100057 td 0xffffff0002650390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vacl_get_acl() at vacl_get_acl+0x55 __acl_get_file() at __acl_get_file+0xcf syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c, rsp = 0x7fffffffe1d8, rbp = 0x2 --- Tracing command csh pid 1200 tid 100152 td 0xffffff000270a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command bsdtar pid 1199 tid 100151 td 0xffffff0002af0ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp = 0x7fffffffe7a8, rbp = 0x80101a240 --- Tracing command bsdtar pid 1198 tid 100105 td 0xffffff000264c000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a rrw_enter() at rrw_enter+0xb1 zfs_freebsd_read() at zfs_freebsd_read+0x55 VOP_READ_APV() at VOP_READ_APV+0xaf vn_read() at vn_read+0x254 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp = 0x7fffffffe898, rbp = 0x1a87 --- Tracing command csh pid 1196 tid 100056 td 0xffffff0002650720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command bsdtar pid 1195 tid 100067 td 0xffffff00026d6720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x2c zfs_znode_alloc() at zfs_znode_alloc+0x39 zfs_mknode() at zfs_mknode+0x205 zfs_freebsd_create() at zfs_freebsd_create+0x583 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800e3557c, rsp = 0x7fffffffe758, rbp = 0x1a4 --- Tracing command bsdtar pid 1194 tid 100150 td 0xffffff0002cb8390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vacl_get_acl() at vacl_get_acl+0x55 __acl_get_file() at __acl_get_file+0xcf syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c, rsp = 0x7fffffffe1d8, rbp = 0x2 --- Tracing command csh pid 1192 tid 100138 td 0xffffff0002cbc390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command bsdtar pid 1191 tid 100113 td 0xffffff0002915720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp = 0x7fffffffe7a8, rbp = 0x80101a240 --- Tracing command bsdtar pid 1190 tid 100118 td 0xffffff0002914390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vacl_get_acl() at vacl_get_acl+0x55 __acl_get_file() at __acl_get_file+0xcf syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c, rsp = 0x7fffffffe1d8, rbp = 0x2 --- Tracing command csh pid 1188 tid 100144 td 0xffffff0002cb9ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command bsdtar pid 1187 tid 100139 td 0xffffff0002cbc000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp = 0x7fffffffe7a8, rbp = 0x80101a240 --- Tracing command bsdtar pid 1186 tid 100123 td 0xffffff0002af0000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vacl_get_acl() at vacl_get_acl+0x55 __acl_get_file() at __acl_get_file+0xcf syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c, rsp = 0x7fffffffe1d8, rbp = 0x2 --- Tracing command csh pid 1184 tid 100146 td 0xffffff0002cb9390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command bsdtar pid 1183 tid 100126 td 0xffffff0002aea390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x7b2 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x2eb namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800e354fc, rsp = 0x7fffffffe778, rbp = 0x801023380 --- Tracing command bsdtar pid 1182 tid 100066 td 0xffffff000264f000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x79b vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vacl_get_acl() at vacl_get_acl+0x55 __acl_get_file() at __acl_get_file+0xcf syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c, rsp = 0x7fffffffe1d8, rbp = 0x2 --- Tracing command csh pid 1180 tid 100120 td 0xffffff0002714ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe3a8, rbp = 0x800c43200 --- Tracing command csh pid 1147 tid 100142 td 0xffffff0002cbb390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp = 0x7fffffffe658, rbp = 0x1 --- Tracing command sshd pid 1146 tid 100111 td 0xffffff0002916000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1133 tid 100116 td 0xffffff0002914ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command csh pid 959 tid 100149 td 0xffffff0002cb8720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp = 0x7fffffffe738, rbp = 0x1 --- Tracing command su pid 958 tid 100148 td 0xffffff0002cb8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp = 0x7fffffffe598, rbp = 0x3bf --- Tracing command csh pid 952 tid 100125 td 0xffffff0002aea720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe558, rbp = 0x800c1f500 --- Tracing command sshd pid 951 tid 100130 td 0xffffff0002ae8390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command csh pid 946 tid 100141 td 0xffffff0002cbb720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp = 0x7fffffffe558, rbp = 0x800c1f500 --- Tracing command sshd pid 945 tid 100140 td 0xffffff0002cbbab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 942 tid 100128 td 0xffffff0002ae8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command sshd pid 939 tid 100132 td 0xffffff0002ae7ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command getty pid 908 tid 100117 td 0xffffff0002914720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 907 tid 100129 td 0xffffff0002ae8720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 906 tid 100106 td 0xffffff0002917390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 905 tid 100121 td 0xffffff0002714720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 904 tid 100136 td 0xffffff0002917ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 903 tid 100135 td 0xffffff0002ae7000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 902 tid 100134 td 0xffffff0002ae7390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 901 tid 100133 td 0xffffff0002ae7720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command cron pid 853 tid 100119 td 0xffffff0002914000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093d92c, rsp = 0x7fffffffeb28, rbp = 0x3c --- Tracing command sendmail pid 847 tid 100061 td 0xffffff000256a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3975c, rsp = 0x7fffffffcd68, rbp = 0x1 --- Tracing command sendmail pid 841 tid 100107 td 0xffffff0002917000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd772c, rsp = 0x7fffffffc1e8, rbp = 0x7fffffffc280 --- Tracing command sshd pid 834 tid 100110 td 0xffffff0002916390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp = 0x7fffffffddd8, rbp = 0x2 --- Tracing command syslogd pid 610 tid 100064 td 0xffffff000264f390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80085272c, rsp = 0x7fffffffdcb8, rbp = 0x1 --- Tracing command devd pid 503 tid 100114 td 0xffffff0002915390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x440cfc, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 --- Tracing command flowcleaner pid 22 tid 100054 td 0xffffff000256aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_ debug.ddb.capture.bufsize: 49152 debug.ddb.capture.inprogress: 0 debug.ddb.capture.maxbufsize: 5242880 debug.ddb.capture.bufoff: 49152 debug.ddb.scripting.unscript: debug.ddb.scripting.scripts: debug.ddb.textdump.do_version: 1 debug.ddb.textdump.do_panic: 1 debug.ddb.textdump.do_msgbuf: 1 debug.ddb.textdump.do_ddb: 1 debug.ddb.textdump.pending: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?804B79F6-27CE-40D2-8AB8-6FC378F448FA>