From owner-freebsd-current@FreeBSD.ORG Sun Nov 23 16:57:34 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A6146BA7; Sun, 23 Nov 2014 16:57:34 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 835F4B41; Sun, 23 Nov 2014 16:57:34 +0000 (UTC) Received: from [192.168.200.212] (unknown [50.136.155.142]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id C8CA4193964; Sun, 23 Nov 2014 16:57:26 +0000 (UTC) Subject: Re: zfs/vfs lockups, via poudriere From: Sean Bruno Reply-To: sbruno@freebsd.org To: Andriy Gapon In-Reply-To: <547109A2.9010506@FreeBSD.org> References: <1416684021.7423.77.camel@bruno> <547109A2.9010506@FreeBSD.org> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-I3IemHNtWgO3W2Vy4BM6" Date: Sun, 23 Nov 2014 08:57:25 -0800 Message-ID: <1416761846.1186.0.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: freebsd-current@FreeBSD.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2014 16:57:34 -0000 --=-I3IemHNtWgO3W2Vy4BM6 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Sun, 2014-11-23 at 00:09 +0200, Andriy Gapon wrote: > On 22/11/2014 21:20, Sean Bruno wrote: > > bdrewery reported a vfs/zfs condition where operations will stall out > > and block (rm, mv, file) during a poudriere build. I've hit this now > > and it seems to be alleviated by setting vfs.lookup_shared=3D0 > >=20 > > I seem to be able to trivially reproduce this on my builders and want t= o > > know if anyone is looking to diagnose this further. > >=20 > > original message: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.ht= ml > >=20 > > On my builders I see: > >=20 > > procstat -kka | grep zfs > >=20 > > 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wa= it+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampol= ine+0xe=20 > > 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9= a fork_trampoline+0xe=20 > > 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a= fork_trampoline+0xe=20 > > 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_t= rampoline+0xe=20 > > 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wa= it+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampol= ine+0xe=20 > > 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a f= ork_trampoline+0xe=20 > > 31071 100995 rm - mi_switch+0xe1 sleepq_wa= it+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP= _APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x53= 4 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb=20 > > 31075 100693 mv - mi_switch+0xe1 sleepq_wa= it+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x4 >=20 > The last line looks incomplete. >=20 >=20 hrm ... cut-n-paste fail I guess. procstat -kka | grep zfs 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0= x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+= 0xe=20 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fo= rk_trampoline+0xe=20 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a for= k_trampoline+0xe=20 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_tramp= oline+0xe=20 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0= x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+= 0xe=20 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_= trampoline+0xe=20 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV= +0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 ke= rn_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb=20 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43 vputx+0x28a zfs_rename_unlock+0x3e zfs_freebsd_rename+0xe39 V= OP_RENAME_APV+0xab kern_renameat+0x4a6 amd64_syscall+0x3fb Xfast_syscall+0x= fb=20 --=-I3IemHNtWgO3W2Vy4BM6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJUchH1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kQcsIAMPtB+8NwRinUTLNLMDXkRgN ixcZiWRaadnCDgUd2tg3+y5soCkyoOVmKbx1ZR0LkW31YbHTPFiVS2M1QQF3ggN2 9qERmS3piCg2rDG4hdN29Tt3enZdRNtOEApPb2LyS19WrsYUQNB/OllDJTS8Ej3Q 4d/EAQ+lr1nZqD2z+cL9MMZ3FJU64VZzMelXB0HspdGtsws4lw/aXfujmKx2u+dK vwlw+K08sL2vuejLUOyRVDNoDwScwUCiyXEFvmJAHllJvff6g11rY02SA9l1qx38 3iqEY7Fb3ZQefGKq3j2ySfb3lEGJC6mp6NpzjhUfzslQVihaqvnjFByqRAZw80o= =VXrP -----END PGP SIGNATURE----- --=-I3IemHNtWgO3W2Vy4BM6--