From owner-freebsd-stable@freebsd.org Fri May 10 15:17:36 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE15915A5062 for ; Fri, 10 May 2019 15:17:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D41B674BAD; Fri, 10 May 2019 15:17:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f169.google.com with SMTP id 188so5404354ljf.9; Fri, 10 May 2019 08:17:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XzmcA0laxnxhPbyb/E6kPO2rbJTe9FxVMn8tUXBbi7U=; b=ZA5axeAiaM8tUKCZwp5AOxxDeqgQbFrRqsTe2v2Kx0+D8WNbFn6wfMZhiOjTCIVIGv aMFA6HnKh5BOq41r0LiXg2/hXNwN0HlgOaL/UtyB9yBB1dqnIdLfJ4F463u7Pz8CciWr M4cVVMA3OFvQZbrB6cysdpqKDthZ9qzsyBFU9kiu3e8ByW+LY0Nl+KtDoy2Hgu+r6iE2 xoD1LZ2Atv14k0Mo023Y6wJkccSQgm6Z6iIbhrjvrQN6k60arPfuaUz1DSAfbn97KzlM CtjzGluEl86agc52VQSEmP/mabscSHWFq1f2h0N7L+rh6oFP23hFQ8ERKKay0kG1fNL2 yjxQ== X-Gm-Message-State: APjAAAVNVftx/WV7mSbo5HH1CfvnRx1LQV4YHXp3mBGuHIPuh8082Vz7 Ao98NDzLglv/ptK93H9Jn3FS23aOGP5MotKZRyGJouNN X-Google-Smtp-Source: APXvYqxipBvn6AZusTZDdXYEYhW4E3FbITDpDxza9g93XA+/eiC4kF3A/nJUIF8a6OsX6lY9svD4BCNQTXUFUpUD8eA= X-Received: by 2002:a2e:655b:: with SMTP id z88mr2882067ljb.108.1557501454143; Fri, 10 May 2019 08:17:34 -0700 (PDT) MIME-Version: 1.0 References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <7D18A234-E7BF-4855-BD51-4AE2253DB1E4@sorbs.net> <805ee7f1-83f6-c59e-8107-4851ca9fce6e@quip.cz> <5de7f3d3-b34c-0382-b7d4-b7e38339649b@quip.cz> <8e443083-1254-520b-014d-2f9a94008533@quip.cz> In-Reply-To: <8e443083-1254-520b-014d-2f9a94008533@quip.cz> From: Alan Somers Date: Fri, 10 May 2019 09:17:22 -0600 Message-ID: Subject: Re: ZFS... To: Miroslav Lachman <000.fbsd@quip.cz> Cc: FreeBSD-STABLE Mailing List , Dimitry Andric Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: D41B674BAD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.52 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.70)[-0.698,0]; SUBJ_ALL_CAPS(0.45)[6]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.968,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[169.208.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-1.30)[ip: (-0.50), ipnet: 209.85.128.0/17(-3.66), asn: 15169(-2.27), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 May 2019 15:17:37 -0000 On Fri, May 10, 2019 at 4:01 AM Miroslav Lachman <000.fbsd@quip.cz> wrote: > > Alan Somers wrote on 2019/05/09 14:50: > > [...] > > > On 11.3 and even much older releases, you can greatly speed up scrub > > and resilver by tweaking some sysctls. If you have spinning rust, > > raise vfs.zfs.top_maxinflight so they'll do fewer seeks. I used to > > set it to 8192 on machines with 32GB of RAM. Raising > > vfs.zfs.resilver_min_time_ms to 5000 helps a little, too. > > I have this in sysctl.conf > vfs.zfs.scrub_delay=0 > vfs.zfs.top_maxinflight=128 > vfs.zfs.resilver_min_time_ms=5000 > vfs.zfs.resilver_delay=0 > > I found it somewhere in the mailinglist discussing this issue in the past. > > Isn't yours 8192 too much? The machine in question has 4x SATA drives on > very dump and slow controller and only 5GB of RAM. I chose 8192 for a machine with 32GB of RAM, dozens of disks, and where resilver speed was more important than responsiveness. RAM usage does increase dramatically as you raise that sysctl, but so does resilver speed. Even top_maxinflight=8192 didn't max out the resilver speed. Higher values produced still higher resilver speeds, but used too much RAM. The reason that resilver speed depends on top_maxinflight is because ZFS issues the I/Os in object order, not LBA order. But vdev_queue will reorder them into LBA order. Allowing more I/Os to be in flight gives vdev_queue more power to reorder things. -Alan > > Even if I read this > vfs.zfs.top_maxinflight: Maximum I/Os per top-level vdev > I am still not sure what it really means and how I can "calculate" > optimal value. > > As Michelle pointed there is drawback when sysctls are optimized for > quick scrub, but this machines is only running nightly backup script > fetching data from other 20 machines so this scrip sets sysctl back to > sane defaults during backup > sysctl vfs.zfs.scrub_delay=4 > /dev/null > sysctl vfs.zfs.top_maxinflight=32 > /dev/null > sysctl vfs.zfs.resilver_min_time_ms=3000 > /dev/null > sysctl vfs.zfs.resilver_delay=2 > /dev/null > > At the and it reloads back optimized settings from sysctel.conf > > Miroslav Lachman