Date: Fri, 10 May 2019 09:17:22 -0600 From: Alan Somers <asomers@freebsd.org> To: Miroslav Lachman <000.fbsd@quip.cz> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, Dimitry Andric <dim@freebsd.org> Subject: Re: ZFS... Message-ID: <CAOtMX2iW6j_0GgCVFEiYJ1QG659p5CU0Chv0_=RJ92r7ZSywOQ@mail.gmail.com> In-Reply-To: <8e443083-1254-520b-014d-2f9a94008533@quip.cz> References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <d0118f7e-7cfc-8bf1-308c-823bce088039@denninger.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <A535026E-F9F6-4BBA-8287-87EFD02CF207@sorbs.net> <a82bfabe-a8c3-fd9a-55ec-52530d4eafff@denninger.net> <a1b78a63-0ef1-af51-4e33-a9a97a257c8b@sorbs.net> <CAMPTd_A7RYJ12pFyY4TzbXct82kWfr1hcEkSpDg7bjP25xjJGA@mail.gmail.com> <d91cf5@sorbs.net> <7D18A234-E7BF-4855-BD51-4AE2253DB1E4@sorbs.net> <E68600B3-F856-4909-AB6E-BDFCD8AAAB43@punkt.de> <805ee7f1-83f6-c59e-8107-4851ca9fce6e@quip.cz> <E980141F-48D9-4870-8FE1-9A5610F12826@FreeBSD.org> <5de7f3d3-b34c-0382-b7d4-b7e38339649b@quip.cz> <CAOtMX2ip_G0afh_4AaYcQt4=zX4qkCTfs0qb66DpCgcALZXS5g@mail.gmail.com> <8e443083-1254-520b-014d-2f9a94008533@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, May 10, 2019 at 4:01 AM Miroslav Lachman <000.fbsd@quip.cz> wrote: > > Alan Somers wrote on 2019/05/09 14:50: > > [...] > > > On 11.3 and even much older releases, you can greatly speed up scrub > > and resilver by tweaking some sysctls. If you have spinning rust, > > raise vfs.zfs.top_maxinflight so they'll do fewer seeks. I used to > > set it to 8192 on machines with 32GB of RAM. Raising > > vfs.zfs.resilver_min_time_ms to 5000 helps a little, too. > > I have this in sysctl.conf > vfs.zfs.scrub_delay=0 > vfs.zfs.top_maxinflight=128 > vfs.zfs.resilver_min_time_ms=5000 > vfs.zfs.resilver_delay=0 > > I found it somewhere in the mailinglist discussing this issue in the past. > > Isn't yours 8192 too much? The machine in question has 4x SATA drives on > very dump and slow controller and only 5GB of RAM. I chose 8192 for a machine with 32GB of RAM, dozens of disks, and where resilver speed was more important than responsiveness. RAM usage does increase dramatically as you raise that sysctl, but so does resilver speed. Even top_maxinflight=8192 didn't max out the resilver speed. Higher values produced still higher resilver speeds, but used too much RAM. The reason that resilver speed depends on top_maxinflight is because ZFS issues the I/Os in object order, not LBA order. But vdev_queue will reorder them into LBA order. Allowing more I/Os to be in flight gives vdev_queue more power to reorder things. -Alan > > Even if I read this > vfs.zfs.top_maxinflight: Maximum I/Os per top-level vdev > I am still not sure what it really means and how I can "calculate" > optimal value. > > As Michelle pointed there is drawback when sysctls are optimized for > quick scrub, but this machines is only running nightly backup script > fetching data from other 20 machines so this scrip sets sysctl back to > sane defaults during backup > sysctl vfs.zfs.scrub_delay=4 > /dev/null > sysctl vfs.zfs.top_maxinflight=32 > /dev/null > sysctl vfs.zfs.resilver_min_time_ms=3000 > /dev/null > sysctl vfs.zfs.resilver_delay=2 > /dev/null > > At the and it reloads back optimized settings from sysctel.conf > > Miroslav Lachman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2iW6j_0GgCVFEiYJ1QG659p5CU0Chv0_=RJ92r7ZSywOQ>