From owner-freebsd-fs@freebsd.org Wed Dec 4 15:20:10 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 03EA51C6FB8 for ; Wed, 4 Dec 2019 15:20:10 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47SjFz1r8Hz3xYm; Wed, 4 Dec 2019 15:20:07 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 5344F4000E; Wed, 4 Dec 2019 16:20:04 +0100 (CET) Received: from [IPv6:2001:6b0:17:f002:8d12:4cc9:4751:6d43] (unknown [IPv6:2001:6b0:17:f002:8d12:4cc9:4751:6d43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 1D96A4000D; Wed, 4 Dec 2019 16:20:04 +0100 (CET) From: Peter Eriksson Message-Id: Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3601.0.10\)) Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Date: Wed, 4 Dec 2019 16:20:03 +0100 In-Reply-To: <1119144A-C6DA-4CBF-8616-108FB50D2F8D@lysator.liu.se> Cc: Andriy Gapon To: FreeBSD FS References: <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> <20191203162219.GI43802@raichu> <247C6990-BC6E-4E3D-8CEF-5A861D8A25EC@lysator.liu.se> <4D44B25E-2F15-4536-A653-DA242051C8A9@lysator.liu.se> <1119144A-C6DA-4CBF-8616-108FB50D2F8D@lysator.liu.se> X-Mailer: Apple Mail (2.3601.0.10) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 47SjFz1r8Hz3xYm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.91 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; URI_COUNT_ODD(1.00)[3]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; IP_SCORE(-3.41)[ip: (-8.93), ipnet: 130.236.0.0/16(-4.51), asn: 2843(-3.61), country: SE(-0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Dec 2019 15:20:10 -0000 Yep, it=E2=80=99s pmap_remove() that occasionally takes ~1s instead of = being very quick. A test I ran: kmem_unback: pmap_remove(kernel_pmap, 18446741878385815552, = 18446741878385827840) took 1 seconds kmem_free: kmem_unback(kmem_object, 18446741878385815552, 12288) took 1 = seconds page_free: kmem_free(kmem_arena, 18446741878385815552, 12288) took 1 = seconds keg_free_slab: keg->uk_freef(mem) {page_free} took 1 seconds keg_drain: while()-keg_free_slab-loop took 17 seconds [23136 loops, 17 = slow calls] zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 17 seconds zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 17 seconds zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 17 seconds uma_zdestroy(zio_buf_12288) took 17 seconds kmem_cache_destroy: uma_zdestroy(0xfffff803465dd980) [zio_buf_12288] = took 17 seconds kmem_cache_destroy(zio_buf_cache[20]) took 17 seconds Ie, 17 out of 23136 calls to = keg_free_slab()->page_free()->kmem_free()->kmem_unback()->pmap_remove() = was slow (took atleast 1s). I created a bug report (242427) on bugs.freebsd.org = for this now in case someone knows how to = debug the PMAP stuff=E2=80=A6 :-) - Peter