From owner-freebsd-fs@freebsd.org Wed Apr 27 11:55:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43BEDB1EB18 for ; Wed, 27 Apr 2016 11:55:03 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BE0A51A83 for ; Wed, 27 Apr 2016 11:55:02 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 5988C28416; Wed, 27 Apr 2016 13:55:00 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id D3A8328412; Wed, 27 Apr 2016 13:54:58 +0200 (CEST) Message-ID: <5720A892.5070807@quip.cz> Date: Wed, 27 Apr 2016 13:54:58 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: "Eric A. Borisch" CC: "freebsd-fs@freebsd.org" Subject: Re: How to speed up slow zpool scrub? References: <571F62AD.6080005@quip.cz> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 11:55:03 -0000 Eric A. Borisch wrote on 04/27/2016 00:26: >> Is there any tuning to make scrub faster in this idle time? >> Or is it better to do it other way - slower scrub with even lower priority >> taking for about one week but not affecting time of normal operations? (is >> it dangerous to have scrub running this long or reboot machine during the >> scrub?) > > It's likely easier to make scrub slower (but rsync faster) when the > system is not idle, but hopefully no slower otherwise. Yes, I've been thinking about this approach yesterday evening. > I would start with increasing the vfs.zfs.scrub_delay (in ticks; > typically 1/1000 of a second) settings to reduce scrub's impact to > other IO this without impacting resilvers. (Expect longer total time > as it will scrub less during rsyncs.) You can also increase > vfs.zfs.scan_idle if scrub_delay isn't enough, but do note that > vfs.zfs.scan_idle impacts resilvers, too -- so make sure you don't > also increase vfs.zfs.resilver_delay, or you may get very slow > resilvers during IO. > > The logic for this is in dsl_scan.c [1] -- look there for the final > word on what the different parameters do; there are a number of > informative comments throughout the file if you are not into the code > itself. > > See also vfs.zfs.vdev.*_max_active and vfs.zfs.vdev.*_min_active, if > you are really want more knobs to try. Than you for this pointers, I will look at it and hope find some way to make things better for this machine. > It's might also be worth trying '/usr/local/bin/perl > /usr/share/dtrace/toolkit/hotkernel' to see what the kernel is up > to... Showing results with 0.1% or more. kernel`atomic_add_long 647 0.1% kernel`uma_zfree_arg 666 0.1% kernel`ata_intel_sata_sidpr_read 695 0.1% kernel`uma_zalloc_arg 754 0.1% kernel`free 895 0.1% kernel`ata_tf_write 1011 0.1% kernel`bzero 1453 0.1% kernel`bcopy 1551 0.1% kernel`acpi_timer_get_timecount 2550 0.2% kernel`_sx_xunlock 2894 0.2% kernel`spinlock_exit 3347 0.3% kernel`_sx_xlock 3379 0.3% kernel`sched_idletd 4053 0.3% zfs.ko`lz4_decompress 5293 0.4% kernel`cpu_idle 65358 5.4% kernel`acpi_cpu_c1 1096734 89.8% zpool scrub and rsync backup is running at this time pool: tank0 state: ONLINE scan: scrub in progress since Sun Apr 24 03:01:35 2016 8.44T scanned out of 10.6T at 29.7M/s, 21h8m to go 0 repaired, 79.64% done > - Eric > > [1] https://svnweb.freebsd.org/base/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c?view=markup > or > https://github.com/freebsd/freebsd/blob/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c Thank you for your very useful reply! Miroslav Lachman