From owner-freebsd-stable@freebsd.org Tue Aug 14 14:43:29 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9C3D107A1A5 for ; Tue, 14 Aug 2018 14:43:28 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (mail.moehre.org [195.96.35.7]) by mx1.freebsd.org (Postfix) with ESMTP id 2A18572643 for ; Tue, 14 Aug 2018 14:43:27 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (unknown [195.96.35.7]) by mail.moehre.org (Postfix) with ESMTP id CD911376E8; Tue, 14 Aug 2018 11:57:03 +0200 (CEST) X-Spam-Flag: NO X-Spam-Score: -100.928 X-Spam-Level: X-Spam-Status: No, score=-100.928 tagged_above=-999 required=5 tests=[ALL_TRUSTED=-1, AWL=-0.005, TW_ZJ=0.077, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mail.moehre.org ([195.96.35.7]) by mail.moehre.org (mail.moehre.org [195.96.35.7]) (amavisd-new, port 10024) with ESMTP id 2BP8lqkmFGVe; Tue, 14 Aug 2018 11:57:03 +0200 (CEST) Received: from localhost (p5793BBD5.dip0.t-ipconnect.de [87.147.187.213]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: coco@executive-computing.de) by mail.moehre.org (Postfix) with ESMTPSA id 2D4A2376E1; Tue, 14 Aug 2018 11:57:03 +0200 (CEST) Date: Tue, 14 Aug 2018 11:57:01 +0200 From: Marco Steinbach To: "Kevin P. Neal" Cc: freebsd-stable@freebsd.org Subject: Re: Jails on ZFS yielding 100% load on gstat Message-ID: <20180814115701.00007c83@executive-computing.de> In-Reply-To: <20180813145601.GA1072@neutralgood.org> References: <20180812205047.00002767@executive-computing.de> <20180813145601.GA1072@neutralgood.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Aug 2018 14:43:29 -0000 On Mon, 13 Aug 2018 10:56:01 -0400 "Kevin P. Neal" wrote: > On Sun, Aug 12, 2018 at 08:50:47PM +0200, Marco Steinbach wrote: > > Hi there. > > > > % zpool list > > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH > > ALTROOT zroot 5.41T 670G 4.75T - 13% 12% 1.00x > > ONLINE - > > > > % uname -a > > FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r322984 [...] amd64 > > > > > > I'm running multiple jails on ZFS, using ezjail to manage them, > > including a websever and a mailserver. The mailserver is using a > > MySQL database, otherwise depending on dovecot and postfix. Very > > low volume, just a few polls / logins per minute. > > > > I am experiencing very high loads as per gstat: > > > > dT: 1.021s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > > 4 181 0 0 0.0 169 873 20.1 98.2| ada0 > > 2 111 0 0 0.0 100 540 7.3 90.6| ada1 > > 0 88 0 0 0.0 76 458 1.4 43.3| ada2 > > 0 0 0 0 0.0 0 0 0.0 0.0| > > ada0p1 3 150 0 0 0.0 150 603 20.2 95.1| > > ada0p2 1 31 0 0 0.0 20 270 19.2 117.0| > > ada0p3 0 0 0 0 0.0 0 0 0.0 0.0| > > gpt/gptboot0 0 0 0 0 0.0 0 0 0.0 > > 0.0| ada1p1 1 85 0 0 0.0 85 341 8.4 > > 68.9| ada1p2 1 25 0 0 0.0 15 200 0.9 > > 75.0| ada1p3 0 0 0 0 0.0 0 0 0.0 > > 0.0| ada2p1 0 62 0 0 0.0 62 251 1.6 > > 9.9| ada2p2 0 26 0 0 0.0 15 208 0.5 > > 42.0| ada2p3 0 0 0 0 0.0 0 0 0.0 > > 0.0| gpt/gptboot1 0 0 0 0 0.0 0 0 > > 0.0 0.0| gpt/gptboot2 > > Say, these don't look very evenly spread. > > What's ada0p3? I posted 'list', instead of 'zpool status' :/ It's the third drive in my raidz1. > > These loads lead to the system suffering from very much delayed > > responses to even the basic task of echoing characters entered on > > the console, consequently rendering the services offered unusable > > to the users because of the delays. > > How much memory does your machine have, and how much is being used by > ZFS? You can get that from the default top display if you want. > > You can also use 'zpool iostat ' to see how much traffic is > going to specifically ZFS. > I see a lot of imbalanced IO on my zpools, although I am using the exact same model and firmware level drives in my raidz1 setups. All my machines have at least 32GB, I've limited the maximum ARC size to 4GB, though, since last time I checked ZFS took quite a large piece of the cake, and didn't give it back, when not limited by setting vfs.zfs.arc_max in loader.conf :) I've since found the culprit, I think. Please see my other post in this thread. MfG CoCo