From owner-freebsd-current@FreeBSD.ORG Sat Nov 22 19:20:26 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A52F0223 for ; Sat, 22 Nov 2014 19:20:26 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 848EE8FE for ; Sat, 22 Nov 2014 19:20:25 +0000 (UTC) Received: from [192.168.200.212] (unknown [50.136.155.142]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id BE7D2193964 for ; Sat, 22 Nov 2014 19:20:23 +0000 (UTC) Subject: zfs/vfs lockups, via poudriere From: Sean Bruno Reply-To: sbruno@freebsd.org To: freebsd-current@freebsd.org Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-FDS7Yuw1ZMAYwiX4klol" Date: Sat, 22 Nov 2014 11:20:21 -0800 Message-ID: <1416684021.7423.77.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2014 19:20:26 -0000 --=-FDS7Yuw1ZMAYwiX4klol Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable bdrewery reported a vfs/zfs condition where operations will stall out and block (rm, mv, file) during a poudriere build. I've hit this now and it seems to be alleviated by setting vfs.lookup_shared=3D0 I seem to be able to trivially reproduce this on my builders and want to know if anyone is looking to diagnose this further. original message: https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html On my builders I see: procstat -kka | grep zfs 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0= x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+= 0xe=20 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fo= rk_trampoline+0xe=20 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a for= k_trampoline+0xe=20 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_tramp= oline+0xe=20 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0= x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+= 0xe=20 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_= trampoline+0xe=20 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV= +0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 ke= rn_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb=20 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x4 --=-FDS7Yuw1ZMAYwiX4klol Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJUcOH1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5keYMH/0qv44STnzEhLnRJQvp1TJju oppyD9CIYHE8fBB27mmD1lvECFWo/M6ueZLD/QUbyxdNLkwa5hhrRm3SRN2VBNVh fDzeSpxY/TTCDC36+Qe+6v1/sQ7ZAt2WHqZ8uTbApmQ1Rt5ehg1fKgdVx+15+Wym faz5DUD3Rf+Zpc8WrOFVrYYK/fh+2qr1NcjD7MK314hPeYiInaAmIBsJ4biAwZaq AdDh0mgtDPLtg5oVCWQcACEh/xrHTli1hsftWmoMUtQp4gx3LZE7O65YCcJZGqSX P4vPoW6Hyn4Gqww0vXA6PXaQHQqlfzYrul3kTAwnVFYsfRRQsyxnjs29F6fq0nw= =GCm3 -----END PGP SIGNATURE----- --=-FDS7Yuw1ZMAYwiX4klol--