Date: Thu, 05 Dec 2019 15:31:02 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 242427] pmap_remove() sometimes is very slow causing 10+ minutes long reboots Message-ID: <bug-242427-227-sL912drs4e@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-242427-227@https.bugs.freebsd.org/bugzilla/> References: <bug-242427-227@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242427 --- Comment #6 from Peter Eriksson <pen@lysator.liu.se> --- (In reply to Konstantin Belousov from comment #5) Yes, you are correct (and I was wrong). Sorry about that.=20 I've rewritten the timing tests now to use nanouptime() instead, and now th= ings look slightly different. Albeit still takes a long time... Here's a sample from a server that has been up for some hours: keg_free_slab: keg->uk_freef(mem) {page_free} took 2077 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2123 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2107 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2190 =C2=B5s keg_drain: while()-keg_free_slab-loop took 163 s (163765748 =C2=B5s) = [240297 loops, 4 slow, avg=3D681 =C2=B5s, min=3D246 =C2=B5s, max=3D7922 =C2=B5s] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 163 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 163 s zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 163 s uma_zdestroy(zio_buf_16384) took 163 s kmem_cache_destroy: uma_zdestroy(0xfffff80346601880) [zio_buf_16384] took = 164 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 164 seconds keg_free_slab only prints the keg->uk_freef(mem) (calls page_free which cal= ls kmem_free) call timings if it takes over 2000=C2=B5s. Most of them are not = shown, the majority seems to be around 1000-1500=C2=B5s. Anyway, I wonder if one couldn't just skip all this freeing of kernel memor= y at reboot/shutdown time. Perhaps it could conditionally be disabled via a sysc= tl setting?=20 In order to test this I added such a sysctl and with the calls to kmem_cache_destroy() in zio_fini() disabled this machine reboots in under 1 minute instead of atleast 10 minutes (for a newly booted server). (The LSI = SAS HBA shutdown and the ZFS unmounting takes the most of that time). (I suppose the current code is needed for the case where you have dynamical= ly loaded zfs and wishes to kldunload it (without rebooting) - or one will hav= e a massive memory leak, but since we're rebooting almost immediately after thi= s I don't really see why we have to spend 10-60 minutes freeing all this memory :-). zio.fini() is in /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zi= o.c The rest are in /usr/src/sys/vm/uma_core.c & vm_kern.c A slightly longer output (verbose-level 2 so skipping some timing measureme= nts in keg_free_slab and below that might inflate the timing numbers) for refer= ence from a freshly rebooted server: Freeing ZFS ZIO caches: keg_drain: while()-keg_free_slab-loop took 994 ms (993875 =C2=B5s) [1450 lo= ops, 0 slow, avg=3D685 =C2=B5s, min=3D383 =C2=B5s, max=3D1713 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 7319 =C2=B5s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 14 ms uma_zdestroy(zio_buf_8192) took 19 ms keg_drain: while()-keg_free_slab-loop took 15 s (15886587 =C2=B5s) [23217 l= oops, 0 slow, avg=3D684 =C2=B5s, min=3D279 =C2=B5s, max=3D1745 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 15 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 15 s uma_zdestroy(zio_buf_12288) took 15 s kmem_cache_destroy: uma_zdestroy(0xfffff803465f6980) [zio_buf_12288] took 16 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[20]) took 16 seconds keg_drain: while()-keg_free_slab-loop took 61 s (61698566 =C2=B5s) [86029 l= oops, 11 slow, avg=3D717 =C2=B5s, min=3D265 =C2=B5s, max=3D2252 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 61 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 61 s uma_zdestroy(zio_buf_16384) took 61 s kmem_cache_destroy: uma_zdestroy(0xfffff803465f6880) [zio_buf_16384] took 62 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 62 seconds keg_drain: while()-keg_free_slab-loop took 87 s (86986351 =C2=B5s) [123793 = loops, 24 slow, avg=3D702 =C2=B5s, min=3D297 =C2=B5s, max=3D2398 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 87 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 87 s uma_zdestroy(zio_buf_131072) took 87 s kmem_cache_destroy: uma_zdestroy(0xfffff8034c904640) [zio_buf_131072] took = 87 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[224]) took 87 seconds keg_drain: while()-keg_free_slab-loop took 344 ms (5340963 =C2=B5s) [7730 l= oops, 0 slow, avg=3D690 =C2=B5s, min=3D371 =C2=B5s, max=3D1739 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 357 ms zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 364 ms uma_zdestroy(zio_data_buf_131072) took 370 ms kmem_cache_destroy: uma_zdestroy(0xfffff8034c904600) [zio_data_buf_131072] = took 6 seconds zio_fini: kmem_cache_destroy(zio_data_buf_cache[224]) took 6 seconds --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-242427-227-sL912drs4e>