Date: Sun, 23 Nov 2014 08:57:25 -0800 From: Sean Bruno <sbruno@ignoranthack.me> To: Andriy Gapon <avg@FreeBSD.org> Cc: freebsd-current@FreeBSD.org Subject: Re: zfs/vfs lockups, via poudriere Message-ID: <1416761846.1186.0.camel@bruno> In-Reply-To: <547109A2.9010506@FreeBSD.org> References: <1416684021.7423.77.camel@bruno> <547109A2.9010506@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-I3IemHNtWgO3W2Vy4BM6 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Sun, 2014-11-23 at 00:09 +0200, Andriy Gapon wrote: > On 22/11/2014 21:20, Sean Bruno wrote: > > bdrewery reported a vfs/zfs condition where operations will stall out > > and block (rm, mv, file) during a poudriere build. I've hit this now > > and it seems to be alleviated by setting vfs.lookup_shared=3D0 > >=20 > > I seem to be able to trivially reproduce this on my builders and want t= o > > know if anyone is looking to diagnose this further. > >=20 > > original message: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.ht= ml > >=20 > > On my builders I see: > >=20 > > procstat -kka | grep zfs > >=20 > > 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wa= it+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampol= ine+0xe=20 > > 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9= a fork_trampoline+0xe=20 > > 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a= fork_trampoline+0xe=20 > > 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_t= rampoline+0xe=20 > > 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wa= it+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampol= ine+0xe=20 > > 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_ti= medwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a f= ork_trampoline+0xe=20 > > 31071 100995 rm - mi_switch+0xe1 sleepq_wa= it+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP= _APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x53= 4 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb=20 > > 31075 100693 mv - mi_switch+0xe1 sleepq_wa= it+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0= xab _vn_lock+0x4 >=20 > The last line looks incomplete. >=20 >=20 hrm ... cut-n-paste fail I guess. procstat -kka | grep zfs 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0= x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+= 0xe=20 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fo= rk_trampoline+0xe=20 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a for= k_trampoline+0xe=20 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_tramp= oline+0xe=20 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0= x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+= 0xe=20 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedw= ait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_= trampoline+0xe=20 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV= +0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 ke= rn_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb=20 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0= x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab = _vn_lock+0x43 vputx+0x28a zfs_rename_unlock+0x3e zfs_freebsd_rename+0xe39 V= OP_RENAME_APV+0xab kern_renameat+0x4a6 amd64_syscall+0x3fb Xfast_syscall+0x= fb=20 --=-I3IemHNtWgO3W2Vy4BM6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJUchH1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kQcsIAMPtB+8NwRinUTLNLMDXkRgN ixcZiWRaadnCDgUd2tg3+y5soCkyoOVmKbx1ZR0LkW31YbHTPFiVS2M1QQF3ggN2 9qERmS3piCg2rDG4hdN29Tt3enZdRNtOEApPb2LyS19WrsYUQNB/OllDJTS8Ej3Q 4d/EAQ+lr1nZqD2z+cL9MMZ3FJU64VZzMelXB0HspdGtsws4lw/aXfujmKx2u+dK vwlw+K08sL2vuejLUOyRVDNoDwScwUCiyXEFvmJAHllJvff6g11rY02SA9l1qx38 3iqEY7Fb3ZQefGKq3j2ySfb3lEGJC6mp6NpzjhUfzslQVihaqvnjFByqRAZw80o= =VXrP -----END PGP SIGNATURE----- --=-I3IemHNtWgO3W2Vy4BM6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1416761846.1186.0.camel>