From owner-freebsd-stable@freebsd.org Tue Aug 14 14:43:28 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 886BE107A1A4 for ; Tue, 14 Aug 2018 14:43:28 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (mail.moehre.org [195.96.35.7]) by mx1.freebsd.org (Postfix) with ESMTP id 2A20272644 for ; Tue, 14 Aug 2018 14:43:27 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (unknown [195.96.35.7]) by mail.moehre.org (Postfix) with ESMTP id 13E5A377C7; Tue, 14 Aug 2018 12:28:07 +0200 (CEST) X-Spam-Flag: NO X-Spam-Score: -100.928 X-Spam-Level: X-Spam-Status: No, score=-100.928 tagged_above=-999 required=5 tests=[ALL_TRUSTED=-1, AWL=-0.005, TW_ZF=0.077, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mail.moehre.org ([195.96.35.7]) by mail.moehre.org (mail.moehre.org [195.96.35.7]) (amavisd-new, port 10024) with ESMTP id kCubOWjM_kNA; Tue, 14 Aug 2018 12:28:06 +0200 (CEST) Received: from localhost (p5793BBD5.dip0.t-ipconnect.de [87.147.187.213]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: coco@executive-computing.de) by mail.moehre.org (Postfix) with ESMTPSA id 58E8D377C1; Tue, 14 Aug 2018 12:28:06 +0200 (CEST) Date: Tue, 14 Aug 2018 12:28:04 +0200 From: Marco Steinbach To: Mike Tancsa Cc: freebsd-stable@freebsd.org Subject: Re: Jails on ZFS yielding 100% load on gstat Message-ID: <20180814122731.00003176@executive-computing.de> In-Reply-To: <44a44e1d-ab92-38ec-e990-8fec925fb345@sentex.net> References: <20180812205047.00002767@executive-computing.de> <44a44e1d-ab92-38ec-e990-8fec925fb345@sentex.net> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Aug 2018 14:43:28 -0000 On Mon, 13 Aug 2018 10:12:26 -0400 Mike Tancsa wrote: > On 8/12/2018 2:50 PM, Marco Steinbach wrote: > > > > These loads lead to the system suffering from very much delayed > > responses to even the basic task of echoing characters entered on > > the console, consequently rendering the services offered unusable > > to the users because of the delays. > > > Do you have a LOT of files and or metadata ? Have a look at the cache > stats to see if you are perhaps grinding away on big directory > lookups? Install sysutils/zfs-stats and post > zfs-stats -a > > Also, does > top -mio -I > shed any light as to whats taking up the disk io ? > > ---Mike I've had these kinds of sudden load spikes on my raidz1 setups in the past, which I can't remember seeing, before all machines were migrated from UFS to ZFS. Stats during these spikes, that is, if the machine in question would still respond to commands, did show 100% load on gstat, with ZFS having low throughput (typically less than a megabyte per second). Top display varied wildly, up to the point where each and any process accessing storage would put loads > 80% on IO. Which was kind of to be expected, since something was clogging the queue. In this particular case, as noted in another post, a customers php script ran into a defective MySQL table, and instead of simply erroring out kept hammering it. Leading to what I assume to be a myriad of small IO operations -- probably amplified by the blocksize clash between MySQL and ZFS, which otherwise never impacted performance in my use-cases m| I'll gather more detailed stats the next time I see ZFS storage bogging down. MfG CoCo