From owner-freebsd-fs@freebsd.org Sun Jan 22 16:45:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 403E0CBD16B for ; Sun, 22 Jan 2017 16:45:19 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB391A63 for ; Sun, 22 Jan 2017 16:45:18 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id r144so119024382wme.1 for ; Sun, 22 Jan 2017 08:45:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=kztyFzMq1UMn03GFuvtg4NEmyJPUTMc5qZEsQHJPIfA=; b=mX5jWgrr/dCtOwTQiPHpqNKu1BwAi56esj7D8eY3KEWmA5HmQpa7nYFbtmI6SOFWeJ ngid4e5grHMfzEOHLFjbX09sv7Oys3ufJb2iyCQwcg5LrFcKnNahfJ1AdlmZO7Mh1Ymi 0XdtviFcwQut7QF7e2YnEAouJLJYIUyYjXR5IA6gmVV/0TTJi2vSiYts9I/d78IRa+H4 tM0aShXJUKAMaVMI0A2oj7JB1Hg21bcWC6kYnPMS7qUg/QUJJHCjGG5QkAS1H7kJ8LS6 xWmX5DmSEpJ3HthSCGWpheIrEZ86f/4VYL+suEoJYf9PVDSwtHWbYb7CKHk9X/zZashw VZJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=kztyFzMq1UMn03GFuvtg4NEmyJPUTMc5qZEsQHJPIfA=; b=WXoJjkvMtoj5jh34GKk4+bLHV4dNtxgwceIXiJMDYS0ZPyeYaCLFGCcgHVBsXM8Vks DD7DgVktxcXEUXCF0+PJkGQ8SSoJY+CcqaGC5byiJVxYhljjhxYGZBTeghvfY6+YR7T7 2Y6XI9tjMn/w0ayEbubYymrf6+fG+lrX/3ZiMsVf0N2FnnvH158bEBPRKHMN97ilfM3M 0bq6q//WFf8DsVU8lD7FZNQdaLT242BM8dg4modmcymp+hUxCA5U8I198nQrIalQbia5 ulrcnPyr2i4GV55tszciV1AO013RSaxOurWDC6cEMbiJe3nPwMRqDzFcmrPfDgYrRYnF 1baA== X-Gm-Message-State: AIkVDXJr93iu9NqO5LKOsJxQbUrBxitoYIN4ufEj8SC99GMTEatHlBv0Put6nKQLZOYKSQ== X-Received: by 10.223.161.130 with SMTP id u2mr24236280wru.127.1485103516640; Sun, 22 Jan 2017 08:45:16 -0800 (PST) Received: from [192.168.1.211] (ppp-93-104-174-120.dynamic.mnet-online.de. [93.104.174.120]) by smtp.googlemail.com with ESMTPSA id d29sm16706377wmi.19.2017.01.22.08.45.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jan 2017 08:45:16 -0800 (PST) Subject: Re: stuck on low scrub performance To: freebsd-fs@freebsd.org References: From: Tobias Oberstein Message-ID: <8b408926-768b-8df9-e38c-b617cce30b16@gmail.com> Date: Sun, 22 Jan 2017 17:45:15 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jan 2017 16:45:19 -0000 FWIW, the issue is gone .. it's scrubbing with 130MB/s now: [oberstet@bvr-file1 ~/scm/parcit/RA/adr/services]$ zpool status tank pool: tank state: ONLINE scan: scrub in progress since Tue Jan 17 11:13:29 2017 55.1T scanned out of 89.4T at 127M/s, 78h42m to go 96K repaired, 61.64% done Note: the pool consists of 4 disk sets (striped) with 6 disks each (RAID-Z2). Am 19.01.2017 um 08:56 schrieb Tobias Oberstein: > Hi, > > I've got a 146TB RAID-Z2 magnetic platter pool which I am scrubbing, but > performance is very poor: > > pool: tank > state: ONLINE > scan: scrub in progress since Tue Jan 17 11:13:29 2017 > 5.02T scanned out of 89.4T at 32.2M/s, (scan is slow, no > estimated time) > 0 repaired, 5.61% done > > I've already tried modifying various knobs, with no success (I changed > oids while the scrub is running, without restarting - does that take > effect?). > > I've tried to collect as much info as possible using this script > > https://gist.github.com/oberstet/c66baf777ea77ae843b64271df7420a3 > > with this output > > https://gist.github.com/oberstet/a8e12227486ebeb1e85f6f56b880ce7d > > --- > > My expectation from the general performance of this array would be > scrubbing with 500MB/s (10x more) - this is what we regularily see on > this array when it goes to the magnetic platters on sequential read loads. > > Any helps would be greatly appreciated, as at that speed, the scrubbing > takes weeks =( > > Cheers, > /Tobias