Date: Fri, 17 Jun 2016 11:53:19 +0800 (AWST) From: David Adam <zanchey@ucc.gu.uwa.edu.au> To: freebsd-fs@freebsd.org Subject: Processes wedging on ZFS accesses Message-ID: <alpine.DEB.2.11.1606162047020.26473@motsugo.ucc.gu.uwa.edu.au>
next in thread | raw e-mail | index | archive | help
Hi all, We're still having trouble with our 10.3-RELEASE-p3 fileserver using a ZFS pool. After a certain amount of uptime (usually a week or so), a Samba process will get stuck in D-state: max 2075 0.0 0.2 339928 26616 - D 26May16 0:19.59 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf Running find(1) over the hierarchy that the smbd process has open will also wedge in a D-state. Our backups also seem to get stuck, presumably in the same spot. `procstat -k` on the stuck processes (smbd and our stuck python-based backup program) shows: PID TID COMM TDNAME KSTACK 2075 100587 smbd - mi_switch+0xe1 sleepq_wait+0x3a _sx_slock_hard+0x31b namei+0x1c5 vn_open_cred+0x24d zfs_getextattr+0x1f2 VOP_GETEXTATTR_APV+0xa7 extattr_get_vp+0x15d sys_extattr_get_file+0xf4 amd64_syscall+0x40f Xfast_syscall+0xfb 2075 100623 smbd - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xca0 vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 knlist_remove_kq+0x24 filt_vfsdetach+0x22 knote_fdclose+0xef closefp+0x42 amd64_syscall+0x40f Xfast_syscall+0xfb 21676 101572 python2.7 - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x91a vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 vget+0x73 cache_lookup+0x5d5 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x40f Xfast_syscall+0xfb 36144 101585 python2.7 - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x91a vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 vget+0x73 cache_lookup+0x5d5 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x40f Xfast_syscall+0xfb Memory doesn't appear to be a problem, and we have the ARC wired to 10 GB maximum: Mem: 80M Active, 2127M Inact, 13G Wired, 36K Cache, 1643M Buf, 308M Free ARC: 9921M Total, 7056M MFU, 2434M MRU, 977K Anon, 162M Header, 267M Other Swap: 20G Total, 20G Free I'm getting a DDB kernel built, but is there any other information that is useful? One of the problems is that Samba won't start a new process for the user whose process is already wedged, so eventually user logins stop working. Thanks David Adam zanchey@ucc.gu.uwa.edu.au
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.DEB.2.11.1606162047020.26473>