From owner-freebsd-current@FreeBSD.ORG Sat Nov 22 22:10:45 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9BD79290; Sat, 22 Nov 2014 22:10:45 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C4947ACD; Sat, 22 Nov 2014 22:10:44 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA26477; Sun, 23 Nov 2014 00:12:29 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1XsIt0-000AAq-L3; Sun, 23 Nov 2014 00:10:34 +0200 Message-ID: <547109A2.9010506@FreeBSD.org> Date: Sun, 23 Nov 2014 00:09:38 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: sbruno@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: zfs/vfs lockups, via poudriere References: <1416684021.7423.77.camel@bruno> In-Reply-To: <1416684021.7423.77.camel@bruno> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2014 22:10:45 -0000 On 22/11/2014 21:20, Sean Bruno wrote: > bdrewery reported a vfs/zfs condition where operations will stall out > and block (rm, mv, file) during a poudriere build. I've hit this now > and it seems to be alleviated by setting vfs.lookup_shared=0 > > I seem to be able to trivially reproduce this on my builders and want to > know if anyone is looking to diagnose this further. > > original message: > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html > > On my builders I see: > > procstat -kka | grep zfs > > 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe > 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe > 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a fork_trampoline+0xe > 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_trampoline+0xe > 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe > 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_trampoline+0xe > 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb > 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4 The last line looks incomplete. -- Andriy Gapon