From owner-freebsd-fs@freebsd.org Tue Dec 3 17:31:52 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 806E11B6E83 for ; Tue, 3 Dec 2019 17:31:52 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47S8DR2h7lz3Bv5; Tue, 3 Dec 2019 17:31:51 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 94B7440006; Tue, 3 Dec 2019 18:31:49 +0100 (CET) Received: from [IPv6:2001:6b0:17:f002:182f:9bd3:109e:7075] (unknown [IPv6:2001:6b0:17:f002:182f:9bd3:109e:7075]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 77FF540005; Tue, 3 Dec 2019 18:31:49 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3601.0.10\)) Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() From: Peter Eriksson In-Reply-To: <20191203162219.GI43802@raichu> Date: Tue, 3 Dec 2019 18:31:49 +0100 Cc: Andriy Gapon , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: <247C6990-BC6E-4E3D-8CEF-5A861D8A25EC@lysator.liu.se> References: <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> <20191203162219.GI43802@raichu> To: FreeBSD FS X-Mailer: Apple Mail (2.3601.0.10) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 47S8DR2h7lz3Bv5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-3.67 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; IP_SCORE(-1.17)[ip: (-5.00), ipnet: 2001:6b0::/32(-0.46), asn: 1653(-0.38), country: EU(-0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Dec 2019 17:31:52 -0000 zio_fini: kmem_cache_destroy(zio_buf_cache & zio_data_buf_cache): zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 16 seconds uma_zdestroy(zio_buf_12288) took 16 seconds kmem_cache_destroy: uma_zdestroy(0xfffff8034678e980) [zio_buf_12288] = took 16 seconds kmem_cache_destroy(zio_buf_cache[20]) took 16 seconds zone_free_item(zone=3DUMA Zones): zone->uz_dtor() took 60 seconds uma_zdestroy(zio_buf_16384) took 60 seconds kmem_cache_destroy: uma_zdestroy(0xfffff8034678e880) [zio_buf_16384] = took 60 seconds kmem_cache_destroy(zio_buf_cache[28]) took 60 seconds Looks like it is the call to zone->uz_dtor that is taking time (added = time check around the various parts of zone_free_item(). - Peter > Indeed, looking at the vmstat -z output this doesn't really make = sense. > It would be useful to confirm that we are in fact spending most of the > time reclaiming items.