Date: Tue, 14 Aug 2018 12:28:04 +0200 From: Marco Steinbach <coco@executive-computing.de> To: Mike Tancsa <mike@sentex.net> Cc: freebsd-stable@freebsd.org Subject: Re: Jails on ZFS yielding 100% load on gstat Message-ID: <20180814122731.00003176@executive-computing.de> In-Reply-To: <44a44e1d-ab92-38ec-e990-8fec925fb345@sentex.net> References: <20180812205047.00002767@executive-computing.de> <44a44e1d-ab92-38ec-e990-8fec925fb345@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 13 Aug 2018 10:12:26 -0400 Mike Tancsa <mike@sentex.net> wrote: > On 8/12/2018 2:50 PM, Marco Steinbach wrote: > > > > These loads lead to the system suffering from very much delayed > > responses to even the basic task of echoing characters entered on > > the console, consequently rendering the services offered unusable > > to the users because of the delays. > > > Do you have a LOT of files and or metadata ? Have a look at the cache > stats to see if you are perhaps grinding away on big directory > lookups? Install sysutils/zfs-stats and post > zfs-stats -a > > Also, does > top -mio -I > shed any light as to whats taking up the disk io ? > > ---Mike I've had these kinds of sudden load spikes on my raidz1 setups in the past, which I can't remember seeing, before all machines were migrated from UFS to ZFS. Stats during these spikes, that is, if the machine in question would still respond to commands, did show 100% load on gstat, with ZFS having low throughput (typically less than a megabyte per second). Top display varied wildly, up to the point where each and any process accessing storage would put loads > 80% on IO. Which was kind of to be expected, since something was clogging the queue. In this particular case, as noted in another post, a customers php script ran into a defective MySQL table, and instead of simply erroring out kept hammering it. Leading to what I assume to be a myriad of small IO operations -- probably amplified by the blocksize clash between MySQL and ZFS, which otherwise never impacted performance in my use-cases m| I'll gather more detailed stats the next time I see ZFS storage bogging down. MfG CoCo
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180814122731.00003176>