From owner-freebsd-fs@freebsd.org Fri Jun 17 04:44:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 344BCA7874E for ; Fri, 17 Jun 2016 04:44:34 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B72691E58; Fri, 17 Jun 2016 04:44:33 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id f126so73798547wma.1; Thu, 16 Jun 2016 21:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=SOHw9BIclvoQNdD+T7aAviLuUlp7vX+xb5ySTif+fT0=; b=BLPISSjDyo8JCFlcEYl0He/1gNONsEI2ZMEiFRFxtZeNBVSRxDF75q4Jy92B+uL20j 5CPAF0WpvY+JeJ/mVGFFSdD4LEMHwagK7kNeNO1U4nlgs9FH0QimNlvUIFSo28Mpnp2t mE00rVIeJ8nwcwYsVVN8vXAYYKFdxxV9DsmU8Ons9obeUBeMGYTokXkc1LvO6mpYOF/p ZB0Bz2zhmWeQjJXWL65uu1+qyjrnjPCcRpvSXPfhrNdUjOwzC1UHEKcigiOgXwlfqv9y Ao3TwGujR751bt1jd74HZsvTMhtrpwM/8xAPf4P6b3ZGRDMoHlc9wbTWu+XQRlDi7CMG W47Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=SOHw9BIclvoQNdD+T7aAviLuUlp7vX+xb5ySTif+fT0=; b=DmlPy3C7Ct7KdG+w3hF5fhyyQKL7l0jWKJRlVqL02tDegoPtvHAQvtQBNk3gJQE9Pn sThMYhkaYBj0clxeOZrvzRi/mXXnSYAjQrwcSDT+QtPP7Wb4Z73ZDUtO4zQDSZVPepS0 3SSJpGsZ23s2tQ+C93gCPdxNzNsYa3lmQecVOrQDquu1SPMWwsQsQt9mfgTzy3F5Mr5i qMIkRIQV1Zr0T4o8K24mwWeaY8hmFXkxiTI0FLfqy+kZtMkbsWz7+Bj1Wwt2EUVFTDE/ jPllBxzfmVMmnkOYOkYa9q30vuCrFzNZgDfA5KU1Xu/cKLZ52IQnXxPQIz4IanJ6nfsZ rNgA== X-Gm-Message-State: ALyK8tIEdEGGR2KxoNHsfkiG/yitk5Hm4fEACwaRWBguthgpEZ3qbt2THyEf1M6yWwEUqw== X-Received: by 10.194.82.36 with SMTP id f4mr134490wjy.104.1466138670963; Thu, 16 Jun 2016 21:44:30 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by smtp.gmail.com with ESMTPSA id o129sm1740715wmb.17.2016.06.16.21.44.29 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Thu, 16 Jun 2016 21:44:29 -0700 (PDT) Date: Fri, 17 Jun 2016 06:44:27 +0200 From: Mateusz Guzik To: David Adam Cc: freebsd-fs@freebsd.org, avg@freebsd.org Subject: Re: Processes wedging on ZFS accesses Message-ID: <20160617044427.GA6575@dft-labs.eu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 04:44:34 -0000 On Fri, Jun 17, 2016 at 11:53:19AM +0800, David Adam wrote: > Hi all, > > We're still having trouble with our 10.3-RELEASE-p3 fileserver using a ZFS > pool. > > After a certain amount of uptime (usually a week or so), a Samba process > will get stuck in D-state: > > max 2075 0.0 0.2 339928 26616 - D 26May16 0:19.59 > /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf > > Running find(1) over the hierarchy that the smbd process has open will > also wedge in a D-state. > > Our backups also seem to get stuck, presumably in the same spot. > > `procstat -k` on the stuck processes (smbd and our stuck python-based > backup program) shows: > PID TID COMM TDNAME KSTACK > 2075 100587 smbd - mi_switch+0xe1 > sleepq_wait+0x3a _sx_slock_hard+0x31b namei+0x1c5 vn_open_cred+0x24d > zfs_getextattr+0x1f2 VOP_GETEXTATTR_APV+0xa7 extattr_get_vp+0x15d > sys_extattr_get_file+0xf4 amd64_syscall+0x40f Xfast_syscall+0xfb > 2075 100623 smbd - mi_switch+0xe1 > sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xca0 vop_stdlock+0x3c > VOP_LOCK1_APV+0xab _vn_lock+0x43 knlist_remove_kq+0x24 filt_vfsdetach+0x22 > knote_fdclose+0xef closefp+0x42 amd64_syscall+0x40f Xfast_syscall+0xfb > 21676 101572 python2.7 - mi_switch+0xe1 These 2 threads likely deadlocked each other: the first one has the vnode locked and tries to take the filedesc lock for reading the second one has the filedesc taken for writing and tries to lock the vnode The filedesc lock is going to be split which will get rid of this particular instance of the problem. However, I think the real bug is the fact that zfs_getextattr calls vn_open_cred with the vnode locked, but I don't know if we can simply unlock it and not revalidate anything (likely yes). Cc'ing some zfs people for comments. -- Mateusz Guzik