Date: Wed, 4 Dec 2019 16:20:03 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: FreeBSD FS <freebsd-fs@freebsd.org> Cc: Andriy Gapon <avg@freebsd.org> Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <F8C0E18B-98D1-4A03-B98D-A5D49F559675@lysator.liu.se> In-Reply-To: <1119144A-C6DA-4CBF-8616-108FB50D2F8D@lysator.liu.se> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> <20191203162219.GI43802@raichu> <247C6990-BC6E-4E3D-8CEF-5A861D8A25EC@lysator.liu.se> <4D44B25E-2F15-4536-A653-DA242051C8A9@lysator.liu.se> <e15d66e2-dde4-aaff-87fb-cacd15c1e719@FreeBSD.org> <BDE8C3F8-7D55-4C4F-8E40-0DF2FD732D5A@lysator.liu.se> <1119144A-C6DA-4CBF-8616-108FB50D2F8D@lysator.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
Yep, it’s pmap_remove() that occasionally takes ~1s instead of being very quick. A test I ran:
kmem_unback: pmap_remove(kernel_pmap, 18446741878385815552, 18446741878385827840) took 1 seconds
kmem_free: kmem_unback(kmem_object, 18446741878385815552, 12288) took 1 seconds
page_free: kmem_free(kmem_arena, 18446741878385815552, 12288) took 1 seconds
keg_free_slab: keg->uk_freef(mem) {page_free} took 1 seconds
keg_drain: while()-keg_free_slab-loop took 17 seconds [23136 loops, 17 slow calls]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 17 seconds
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 17 seconds
zone_free_item(zone=UMA Zones): zone->uz_dtor() took 17 seconds
uma_zdestroy(zio_buf_12288) took 17 seconds
kmem_cache_destroy: uma_zdestroy(0xfffff803465dd980) [zio_buf_12288] took 17 seconds
kmem_cache_destroy(zio_buf_cache[20]) took 17 seconds
Ie, 17 out of 23136 calls to keg_free_slab()->page_free()->kmem_free()->kmem_unback()->pmap_remove() was slow (took atleast 1s).
I created a bug report (242427) on bugs.freebsd.org <http://bugs.freebsd.org/>; for this now in case someone knows how to debug the PMAP stuff… :-)
- Peter
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F8C0E18B-98D1-4A03-B98D-A5D49F559675>