From owner-freebsd-stable@freebsd.org Fri May 10 10:01:33 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2C6D159E186 for ; Fri, 10 May 2019 10:01:32 +0000 (UTC) (envelope-from SRS0=AA0G=TK=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60B2069DCB; Fri, 10 May 2019 10:01:32 +0000 (UTC) (envelope-from SRS0=AA0G=TK=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 0A5F028411; Fri, 10 May 2019 12:01:21 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 29FB928417; Fri, 10 May 2019 12:01:20 +0200 (CEST) Subject: Re: ZFS... To: Alan Somers Cc: FreeBSD-STABLE Mailing List , Dimitry Andric References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <7D18A234-E7BF-4855-BD51-4AE2253DB1E4@sorbs.net> <805ee7f1-83f6-c59e-8107-4851ca9fce6e@quip.cz> <5de7f3d3-b34c-0382-b7d4-b7e38339649b@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <8e443083-1254-520b-014d-2f9a94008533@quip.cz> Date: Fri, 10 May 2019 12:01:22 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 60B2069DCB X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [5.62 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(1.00)[0.996,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(1.00)[0.998,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(0.99)[ip: (0.67), ipnet: 94.124.104.0/21(0.34), asn: 42000(3.86), country: CZ(0.08)]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; NEURAL_SPAM_LONG(1.00)[1.000,0]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; SUBJ_ALL_CAPS(0.45)[6]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=AA0G=TK=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=AA0G=TK=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 May 2019 10:01:33 -0000 Alan Somers wrote on 2019/05/09 14:50: [...] > On 11.3 and even much older releases, you can greatly speed up scrub > and resilver by tweaking some sysctls. If you have spinning rust, > raise vfs.zfs.top_maxinflight so they'll do fewer seeks. I used to > set it to 8192 on machines with 32GB of RAM. Raising > vfs.zfs.resilver_min_time_ms to 5000 helps a little, too. I have this in sysctl.conf vfs.zfs.scrub_delay=0 vfs.zfs.top_maxinflight=128 vfs.zfs.resilver_min_time_ms=5000 vfs.zfs.resilver_delay=0 I found it somewhere in the mailinglist discussing this issue in the past. Isn't yours 8192 too much? The machine in question has 4x SATA drives on very dump and slow controller and only 5GB of RAM. Even if I read this vfs.zfs.top_maxinflight: Maximum I/Os per top-level vdev I am still not sure what it really means and how I can "calculate" optimal value. As Michelle pointed there is drawback when sysctls are optimized for quick scrub, but this machines is only running nightly backup script fetching data from other 20 machines so this scrip sets sysctl back to sane defaults during backup sysctl vfs.zfs.scrub_delay=4 > /dev/null sysctl vfs.zfs.top_maxinflight=32 > /dev/null sysctl vfs.zfs.resilver_min_time_ms=3000 > /dev/null sysctl vfs.zfs.resilver_delay=2 > /dev/null At the and it reloads back optimized settings from sysctel.conf Miroslav Lachman