From owner-freebsd-bugs@freebsd.org Thu Dec 5 15:31:04 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B9F121CC20A for ; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47TKS84R7Zz4KFC for ; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 964321CC209; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 960621CC208 for ; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47TKS82rJLz4KF9 for ; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 462591AF2D for ; Thu, 5 Dec 2019 15:31:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id xB5FV4Zj093351 for ; Thu, 5 Dec 2019 15:31:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id xB5FV486093350 for bugs@FreeBSD.org; Thu, 5 Dec 2019 15:31:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 242427] pmap_remove() sometimes is very slow causing 10+ minutes long reboots Date: Thu, 05 Dec 2019 15:31:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pen@lysator.liu.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Dec 2019 15:31:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242427 --- Comment #6 from Peter Eriksson --- (In reply to Konstantin Belousov from comment #5) Yes, you are correct (and I was wrong). Sorry about that.=20 I've rewritten the timing tests now to use nanouptime() instead, and now th= ings look slightly different. Albeit still takes a long time... Here's a sample from a server that has been up for some hours: keg_free_slab: keg->uk_freef(mem) {page_free} took 2077 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2123 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2107 =C2=B5s keg_free_slab: keg->uk_freef(mem) {page_free} took 2190 =C2=B5s keg_drain: while()-keg_free_slab-loop took 163 s (163765748 =C2=B5s) = [240297 loops, 4 slow, avg=3D681 =C2=B5s, min=3D246 =C2=B5s, max=3D7922 =C2=B5s] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 163 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 163 s zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 163 s uma_zdestroy(zio_buf_16384) took 163 s kmem_cache_destroy: uma_zdestroy(0xfffff80346601880) [zio_buf_16384] took = 164 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 164 seconds keg_free_slab only prints the keg->uk_freef(mem) (calls page_free which cal= ls kmem_free) call timings if it takes over 2000=C2=B5s. Most of them are not = shown, the majority seems to be around 1000-1500=C2=B5s. Anyway, I wonder if one couldn't just skip all this freeing of kernel memor= y at reboot/shutdown time. Perhaps it could conditionally be disabled via a sysc= tl setting?=20 In order to test this I added such a sysctl and with the calls to kmem_cache_destroy() in zio_fini() disabled this machine reboots in under 1 minute instead of atleast 10 minutes (for a newly booted server). (The LSI = SAS HBA shutdown and the ZFS unmounting takes the most of that time). (I suppose the current code is needed for the case where you have dynamical= ly loaded zfs and wishes to kldunload it (without rebooting) - or one will hav= e a massive memory leak, but since we're rebooting almost immediately after thi= s I don't really see why we have to spend 10-60 minutes freeing all this memory :-). zio.fini() is in /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zi= o.c The rest are in /usr/src/sys/vm/uma_core.c & vm_kern.c A slightly longer output (verbose-level 2 so skipping some timing measureme= nts in keg_free_slab and below that might inflate the timing numbers) for refer= ence from a freshly rebooted server: Freeing ZFS ZIO caches: keg_drain: while()-keg_free_slab-loop took 994 ms (993875 =C2=B5s) [1450 lo= ops, 0 slow, avg=3D685 =C2=B5s, min=3D383 =C2=B5s, max=3D1713 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 7319 =C2=B5s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 14 ms uma_zdestroy(zio_buf_8192) took 19 ms keg_drain: while()-keg_free_slab-loop took 15 s (15886587 =C2=B5s) [23217 l= oops, 0 slow, avg=3D684 =C2=B5s, min=3D279 =C2=B5s, max=3D1745 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 15 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 15 s uma_zdestroy(zio_buf_12288) took 15 s kmem_cache_destroy: uma_zdestroy(0xfffff803465f6980) [zio_buf_12288] took 16 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[20]) took 16 seconds keg_drain: while()-keg_free_slab-loop took 61 s (61698566 =C2=B5s) [86029 l= oops, 11 slow, avg=3D717 =C2=B5s, min=3D265 =C2=B5s, max=3D2252 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 61 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 61 s uma_zdestroy(zio_buf_16384) took 61 s kmem_cache_destroy: uma_zdestroy(0xfffff803465f6880) [zio_buf_16384] took 62 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 62 seconds keg_drain: while()-keg_free_slab-loop took 87 s (86986351 =C2=B5s) [123793 = loops, 24 slow, avg=3D702 =C2=B5s, min=3D297 =C2=B5s, max=3D2398 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 87 s zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 87 s uma_zdestroy(zio_buf_131072) took 87 s kmem_cache_destroy: uma_zdestroy(0xfffff8034c904640) [zio_buf_131072] took = 87 seconds zio_fini: kmem_cache_destroy(zio_buf_cache[224]) took 87 seconds keg_drain: while()-keg_free_slab-loop took 344 ms (5340963 =C2=B5s) [7730 l= oops, 0 slow, avg=3D690 =C2=B5s, min=3D371 =C2=B5s, max=3D1739 =C2=B5s, 0 zero] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 357 ms zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 364 ms uma_zdestroy(zio_data_buf_131072) took 370 ms kmem_cache_destroy: uma_zdestroy(0xfffff8034c904600) [zio_data_buf_131072] = took 6 seconds zio_fini: kmem_cache_destroy(zio_data_buf_cache[224]) took 6 seconds --=20 You are receiving this mail because: You are the assignee for the bug.=