From owner-freebsd-stable@freebsd.org Fri May 10 14:53:50 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73B7F15A4974 for ; Fri, 10 May 2019 14:53:50 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 73A1F73B48; Fri, 10 May 2019 14:53:47 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PRA00JLQNCR1140@hades.sorbs.net>; Fri, 10 May 2019 08:07:41 -0700 (PDT) Subject: Re: ZFS... To: Miroslav Lachman <000.fbsd@quip.cz>, Alan Somers Cc: Dimitry Andric , FreeBSD-STABLE Mailing List References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <7D18A234-E7BF-4855-BD51-4AE2253DB1E4@sorbs.net> <805ee7f1-83f6-c59e-8107-4851ca9fce6e@quip.cz> <5de7f3d3-b34c-0382-b7d4-b7e38339649b@quip.cz> <8e443083-1254-520b-014d-2f9a94008533@quip.cz> From: Michelle Sullivan Message-id: <85f44376-32eb-885e-1eb8-d3a2e204bc84@sorbs.net> Date: Sat, 11 May 2019 00:53:38 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 In-reply-to: <8e443083-1254-520b-014d-2f9a94008533@quip.cz> X-Rspamd-Queue-Id: 73A1F73B48 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-1.89 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.840,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; NEURAL_HAM_LONG(-0.99)[-0.993,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sorbs.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[battlestar.sorbs.net,anaconda.sorbs.net,ninja.sorbs.net,catapilla.sorbs.net,scorpion.sorbs.net,desperado.sorbs.net]; NEURAL_HAM_SHORT(-0.50)[-0.500,0]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; SUBJ_ALL_CAPS(0.45)[6]; IP_SCORE(-0.30)[ip: (-0.74), ipnet: 72.12.192.0/19(-0.39), asn: 11114(-0.30), country: US(-0.06)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; CTE_CASE(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 May 2019 14:53:50 -0000 Miroslav Lachman wrote: > Alan Somers wrote on 2019/05/09 14:50: > > [...] > >> On 11.3 and even much older releases, you can greatly speed up scrub >> and resilver by tweaking some sysctls. If you have spinning rust, >> raise vfs.zfs.top_maxinflight so they'll do fewer seeks. I used to >> set it to 8192 on machines with 32GB of RAM. Raising >> vfs.zfs.resilver_min_time_ms to 5000 helps a little, too. > > I have this in sysctl.conf > vfs.zfs.scrub_delay=0 > vfs.zfs.top_maxinflight=128 > vfs.zfs.resilver_min_time_ms=5000 > vfs.zfs.resilver_delay=0 > > I found it somewhere in the mailinglist discussing this issue in the > past. > > Isn't yours 8192 too much? The machine in question has 4x SATA drives > on very dump and slow controller and only 5GB of RAM. > > Even if I read this > vfs.zfs.top_maxinflight: Maximum I/Os per top-level vdev > I am still not sure what it really means and how I can "calculate" > optimal value. I calculated it by looking at the iops using gstat and then multiplied it by the spindles in use. That seemed to give the optimum. Much lower and the drives had idle time.... to much and it causes 'pauses' whilst the writes happen. "Tuning" for me had no fine tuning, it seems very sledgehammerish ... big changes are noticeable for better or worse.. small changes you cannot tell by eye, and I guess measuring known operations (such as a controlled environment scrub) might show differing results but I suspect with other things going on these negligible changes are likely to be useless. > > As Michelle pointed there is drawback when sysctls are optimized for > quick scrub, but this machines is only running nightly backup script > fetching data from other 20 machines so this scrip sets sysctl back to > sane defaults during backup not really.. optimize for scrub should only affect the system whilst the scrub and resilvers are actually happening,.. the rest of the time my systems were not affected (noticeably) my problem was a scrub would kick off and last a couple of weeks (1 week heavily preferrencing the scrub) causing the video streaming to become stuttery .. which watching a movie whilst this was happening was really not good... especially as it wasn't for just a few seconds/minutes/hours. > sysctl vfs.zfs.scrub_delay=4 > /dev/null > sysctl vfs.zfs.top_maxinflight=32 > /dev/null > sysctl vfs.zfs.resilver_min_time_ms=3000 > /dev/null > sysctl vfs.zfs.resilver_delay=2 > /dev/null > > At the and it reloads back optimized settings from sysctel.conf > > Miroslav Lachman > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -- Michelle Sullivan http://www.mhix.org/