Date: Sun, 23 Nov 2014 08:57:25 -0800 From: Sean Bruno <sbruno@ignoranthack.me> To: Andriy Gapon <avg@FreeBSD.org> Cc: freebsd-current@FreeBSD.org Subject: Re: zfs/vfs lockups, via poudriere Message-ID: <1416761846.1186.0.camel@bruno> In-Reply-To: <547109A2.9010506@FreeBSD.org> References: <1416684021.7423.77.camel@bruno> <547109A2.9010506@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Sun, 2014-11-23 at 00:09 +0200, Andriy Gapon wrote: > On 22/11/2014 21:20, Sean Bruno wrote: > > bdrewery reported a vfs/zfs condition where operations will stall out > > and block (rm, mv, file) during a poudriere build. I've hit this now > > and it seems to be alleviated by setting vfs.lookup_shared=0 > > > > I seem to be able to trivially reproduce this on my builders and want to > > know if anyone is looking to diagnose this further. > > > > original message: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html > > > > On my builders I see: > > > > procstat -kka | grep zfs > > > > 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe > > 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe > > 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a fork_trampoline+0xe > > 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_trampoline+0xe > > 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe > > 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_trampoline+0xe > > 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb > > 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4 > > The last line looks incomplete. > > hrm ... cut-n-paste fail I guess. procstat -kka | grep zfs 0 100666 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe 3 100151 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe 3 100152 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a fork_trampoline+0xe 3 100657 zfskern trim zroot mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_trampoline+0xe 3 100675 zfskern txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe 3 100676 zfskern txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_trampoline+0xe 31071 100995 rm - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb 31075 100693 mv - mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 vputx+0x28a zfs_rename_unlock+0x3e zfs_freebsd_rename+0xe39 VOP_RENAME_APV+0xab kern_renameat+0x4a6 amd64_syscall+0x3fb Xfast_syscall+0xfb [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJUchH1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kQcsIAMPtB+8NwRinUTLNLMDXkRgN ixcZiWRaadnCDgUd2tg3+y5soCkyoOVmKbx1ZR0LkW31YbHTPFiVS2M1QQF3ggN2 9qERmS3piCg2rDG4hdN29Tt3enZdRNtOEApPb2LyS19WrsYUQNB/OllDJTS8Ej3Q 4d/EAQ+lr1nZqD2z+cL9MMZ3FJU64VZzMelXB0HspdGtsws4lw/aXfujmKx2u+dK vwlw+K08sL2vuejLUOyRVDNoDwScwUCiyXEFvmJAHllJvff6g11rY02SA9l1qx38 3iqEY7Fb3ZQefGKq3j2ySfb3lEGJC6mp6NpzjhUfzslQVihaqvnjFByqRAZw80o= =VXrP -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1416761846.1186.0.camel>