Date: Fri, 17 May 2013 11:37:23 +0200 From: dennis berger <db@bsdsystems.de> To: Steven Hartland <killing@multiplay.co.uk> Cc: Jeremy Chadwick <jdc@koitsu.org>, FreeBSD stable <freebsd-stable@freebsd.org> Subject: Re: still mbuf leak in 9.0 / 9.1? Message-ID: <1186B7CE-EC84-42F6-8904-EDD0C4A5FFBD@bsdsystems.de> In-Reply-To: <4F319A22-E611-4EE6-A970-98315B15C12F@nipsi.de> References: <FDFFFCCB-BDF8-4E27-AF9D-D14D7E0D426D@nipsi.de> <CAFOYbcmF5WybuyJ9DuotcJf_u1FxwBKOLtHvpnT-05cVG6ES=A@mail.gmail.com> <004BC6EA-D8E6-473E-851C-9CDA7578510A@nipsi.de> <20130515211436.GA42790@icarus.home.lan> <696B5622-A95D-4187-A027-07ECC9B5AD1F@nipsi.de> <F3B040438E014E958372DCD64566CED4@multiplay.co.uk> <4F319A22-E611-4EE6-A970-98315B15C12F@nipsi.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi List, I can confirm that it is the bug you mentioned steven. Here is how I found it. I recorded hourly zfskern and nfsd stats. like this. echo "PROCSTAT" >> $reportname pgrep -S "(zfskern|nfsd)" | xargs procstat -kk >> $reportname luckily it crashed this night and logged this. 1910 101508 nfsd nfsd: service mi_switch+0x186 = sleepq_wait+0x42 _sleep+0x376 arc_lowmem+0x77 kmem_malloc+0xc1 = uma_large_malloc+0x4a malloc+0xd9 arc_get_data_buf+0xb5 = arc_read_nolock+0x1ec arc_read+0x93 dbuf_prefetch+0x12c = dmu_zfetch_dofetch+0x10b dmu_zfetch+0xaf8 dbuf_read+0x4a7 = dmu_buf_hold_array_by_dnode+0x16b dmu_buf_hold_array+0x67 = dmu_read_uio+0x3f zfs_freebsd_read+0x3e3=20 Maybe it would be good to merge this fix into RELENG_9_1 and distribute = a fix via freebsd-update what do you think? best, -dennis Am 16.05.2013 um 11:42 schrieb dennis berger: > This is indeed a ZFS+NFS system and I can see that istgt and nfs are = stuck in some ZIO state. Maybe it's this.=20 > Thank's for pointing out.=20 >=20 > Is it this ZFS+NFS deadlock? >=20 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c=20 > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c=20 > @@ -3720,8 +3720,16 @@ arc_lowmem(void *arg __unused, int howto = __unused)=20 > mutex_enter(&arc_reclaim_thr_lock);=20 > needfree =3D 1;=20 > cv_signal(&arc_reclaim_thr_cv);=20 > - while (needfree)=20 > - msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);=20= > +=20 > + /*=20 > + * It is unsafe to block here in arbitrary threads, because we = can come=20 > + * here from ARC itself and may hold ARC locks and thus risk a = deadlock=20 > + * with ARC reclaim thread.=20 > + */=20 > + if (curproc =3D=3D pageproc) {=20 > + while (needfree)=20 > + msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);=20= > + }=20 > mutex_exit(&arc_reclaim_thr_lock);=20 > mutex_exit(&arc_lowmem_lock);=20 > } >=20 > I'll try to crash our testsystem. I'll assume that stressing NFS = backed with ZFS a lot might trigger this bug? >=20 > -dennis >=20 >=20 > Am 16.05.2013 um 00:03 schrieb Steven Hartland: >=20 >> ----- Original Message ----- From: "dennis berger" <db@nipsi.de> >>> FreeBSD 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 = 09:23:10 UTC 2012 >>>=20 >>>> 3. Regarding this: >>>>>> A clean shutdown isn't possible though. It hangs after vnode >>>>>> cleaning, normally you would see detaching of usb devices here, = or >>>>>> other devices maybe? >>>> Please don't conflate this with your above issue. This is almost >>>> certainly unrelated. Please start a new thread about that if = desired. >>>=20 >>> Maybe this is a misunderstanding normally this system will shutdown = cleanly, of course. >>> This hang only appears after the network problem above. >>=20 >> If this is a ZFS system, its a known issue which is fixed in current, >> stable-9, stable-8 and the upcoming 8.4 release. >>=20 >> If not and you have USB devices see if the following sysctl helps: >> hw.usb.no_shutdown_wait=3D1 >>=20 >> Regards >> Steve >>=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> This e.mail is private and confidential between Multiplay (UK) Ltd. = and the person or entity to whom it is addressed. In the event of = misdirection, the recipient is prohibited from using, copying, printing = or otherwise disseminating it or any information contained in it.=20 >> In the event of misdirection, illegible or incomplete transmission = please telephone +44 845 868 1337 >> or return the E.mail to postmaster@multiplay.co.uk. >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1186B7CE-EC84-42F6-8904-EDD0C4A5FFBD>