From owner-freebsd-fs@freebsd.org Tue Jul 5 19:07:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82A0B20E58 for ; Tue, 5 Jul 2016 19:07:27 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (mail.sorbs.net [67.231.146.200]) by mx1.freebsd.org (Postfix) with ESMTP id DC3CB1A33 for ; Tue, 5 Jul 2016 19:07:27 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0O9U00AALW4UHF00@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 05 Jul 2016 12:14:56 -0700 (PDT) Subject: Re: resilver slowdown.... To: "freebsd-fs@freebsd.org" References: <577BA643.6040206@sorbs.net> From: Michelle Sullivan Message-id: <577C056C.7070108@sorbs.net> Date: Tue, 05 Jul 2016 21:07:24 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 In-reply-to: <577BA643.6040206@sorbs.net> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 19:07:28 -0000 Michelle Sullivan wrote: > After retuning for priority to the resilver, I see massive (very good) > performance.... then at some random time later (or so it seemed) the > performance would drop off very quickly always at least to half the > performance, sometimes as much as a quarter of the speed... > > > At first I though it was memory based, then observing system with 19G > (of 32G) free figured it must be based on the stuff on the disk got > some 16T of very small files, and another 16T of video files so > figured it was probably going from large files to small files etc... > Nope, after a reboot and resetting the tuning params to the same each > time after the reboot I'd be seeing an immediate and sustained > performance... Watching it more it seems this is directly related: > > [root@colossus ~]# sysctl -a | grep \\.arc_ > vfs.zfs.arc_meta_limit: 4294967296 > vfs.zfs.arc_meta_used: 5529107352 > vfs.zfs.arc_min: 8589934592 > vfs.zfs.arc_max: 17179869184 Yup think it is related to the meta_limit and meta_used... Resilvering at 203M/s (according to zpool status) [root@colossus ~]# sysctl -a | grep \\.arc_ vfs.zfs.arc_meta_limit: 4294967296 vfs.zfs.arc_meta_used: 2187549528 vfs.zfs.arc_min: 8589934592 vfs.zfs.arc_max: 17179869184 Noticed it had slowed again.. around 64M/s ... [root@colossus ~]# sysctl -a | grep \\.arc_ vfs.zfs.arc_meta_limit: 4294967296 vfs.zfs.arc_meta_used: 4295111952 vfs.zfs.arc_min: 8589934592 vfs.zfs.arc_max: 17179869184 Rebooted and back to >120M/s (can't get a reading atm - too soon after the reboot) -- Michelle Sullivan http://www.mhix.org/