From owner-freebsd-fs@freebsd.org Fri Sep 7 08:55:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 404A0FF4F9D for ; Fri, 7 Sep 2018 08:55:33 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5EF98104A for ; Fri, 7 Sep 2018 08:55:32 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fyCY1-0009cJ-C6 for freebsd-fs@freebsd.org; Fri, 07 Sep 2018 10:55:30 +0200 Date: Fri, 7 Sep 2018 10:55:14 +0200 From: Yamagi Burmeister To: freebsd-fs@freebsd.org Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180907105514.b2601aeaeb8e4f1898322525@yamagi.org> In-Reply-To: <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 08:55:33 -0000 --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I just want a give a short conclusion for this problem. On Mon, 27 Aug 2018 17:19:10 +0300 Lev Serebryakov wrote: > On 27.08.2018 16:47, Yamagi Burmeister wrote: >=20 > > With this program I was able to bisect the source and identify commit > > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culprit: > > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repeat > tests to be sure, that this code is a problem. After about 2 weeks with "zfs_abd_scatter_enabled" set to 0 (in form of r338365 merged back to 11.2-RELEASE and vfs.zfs.abd_scatter_enabled set to 0) I can confirm that switching of scattered allocations helps, my performance problems are gone. The system is running fine and zfs-stats looks good. To be clear: Switching off scattered allocations has a very positive effect in this and only this workload. On our other ~100 hosts - some of them MySQL installations with other workload patterns - the scattered allocations are a benefit or at least don't hurt. Regards, Yamagi --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluSPPMACgkQ6xRy5x1Q JRUhFA/7Bx8s5WBs9Kui1PEvUqoNNjZIY2M8i6K3wuvSgMoIQRt8RIS6f9RjTVAv Gw0XikJZ+mQV3aXNuvev309IJTOK4lp76QqQWxsgO1165wy9qsXF9V0AXEO9YCmq Am0rgF+GwHwqeqZt7M2jGRer3NGZZlax+bTsjmWkY20k2RPFR1KNU06NB/TEkHh4 KiwFWX4Dl/wTs7ev1oQuBqlDTtCX7XywlPRvxjbDG+UFyz/EQxPexbthm3WtBOiX tNXSaFyMgPFflUbWjm+KxEtlOtpKv7k1ukUHdU7Tvf5VKOMMO2iMVMgz5VTc9FAp v9+/WBo8/BcAFfcLS9IU0pHgidpoj1yzmFY2OAuDcMUjvjxJmAsO9wHztVjHmisW dcTENsOX6MCrmHoXi5vP7pvMoKqkjIe8l1vO8feONFy3Tz9JfYiTToR1yziGOynQ minPTjwGdRAhvmrEDyAwVjOyvX5lxPVHHWKvQPiqXpqeGG5Xj5BG7APi3uQHhsKq pjGHSlnfkMjQFQb0J4UDVS/V9UH7xYE40qwg42Xb5gCnsPNxxaFuQOaAgrgPeHS9 jjGHgXxa3fYUte/FxgUV8ba7Vk9AVBYWRsa2m+zXAJr9sizf8pGunPnMalN9aNTx cvGY31H7Ro6MUEbUEa2yobC0mTsnbKAytXb/U9yz+KM9ahcjhds= =p3+5 -----END PGP SIGNATURE----- --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT--