Date: Tue, 3 Dec 2019 11:22:19 -0500 From: Mark Johnston <markj@freebsd.org> To: Andriy Gapon <avg@freebsd.org> Cc: Peter Eriksson <pen@lysator.liu.se>, freebsd-fs@freebsd.org Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <20191203162219.GI43802@raichu> In-Reply-To: <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Dec 03, 2019 at 11:03:11AM +0200, Andriy Gapon wrote: > On 03/12/2019 00:54, Mark Johnston wrote: > > On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote: > >> Sigh. > >> > >> Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times). > >> > >> After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run): > >> > >> kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds > >> kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds > >> kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds > >> kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds > >> kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds > >> kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds > >> kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds > >> kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds > >> > >> This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour). > >> > >> > >> So now on to finding out why uma_destroy() is taking so long… :-). > > > > uma_destroy() frees all of the memory cached in the zone back to the > > page allocator. This operation takes time proportional to the number of > > cached items. I would expect most of the time to be spent in > > zone_reclaim(), called by zone_dtor(). > > But spending *minutes* there is really unexpected. > I have never seen anything like that. > I wonder if there is anything untypical about the system's hardware (like a very > big number of processors) or configuration. Indeed, looking at the vmstat -z output this doesn't really make sense. It would be useful to confirm that we are in fact spending most of the time reclaiming items.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20191203162219.GI43802>