From owner-freebsd-fs@freebsd.org Fri Jan 4 16:28:44 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DED0D1430B99 for ; Fri, 4 Jan 2019 16:28:43 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 073166F99F for ; Fri, 4 Jan 2019 16:28:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x829.google.com with SMTP id u47so36303350qtj.6 for ; Fri, 04 Jan 2019 08:28:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EpIkyrPkR4bWZrXqtMAw1HJS39ZWXwBIk3S8EYGU5gc=; b=fFsZfk81lHt/Cy/BpOWPFp+ciKM1/GXa003jEbg3ohl02LqG1gO+Vcp217F870c2kf Yi00DJ0MDBGgC/7r/r2roteflub4cgHbgiHBogAlRCvv+gDNHY9aZ62bvioGpjcqcMk5 K6cu/E3eqVrmeQm8hRjf5j7v7l5koI0zAn0uH+GLiqnIEJ8IR2ZMDumZBiAl4KOn3sHB s5ppK3gyBT11OuP9b2MH1CVaUxI8F7CdKDqT+RjblmTiMAv9xPRl4IIXlPZjKqN/z4Np n403OCSDQZWm9WiHCfr/ybB7SzwTZJPzPbaKjB/EStRc1A2eI/T3X8pld06p/te5ZFFk NBAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EpIkyrPkR4bWZrXqtMAw1HJS39ZWXwBIk3S8EYGU5gc=; b=DI18rPD8H9vyjv2GxdQ1T0SsfC71ZM6rRUtqL4xfFoJ0Yu4n7j6hYc64uYh3sRkPle H1qnqMEiFt/CHU9wULyoXA38d7z4vXMjIutaHC08nJc+VKqJ/qt6TrNU6w+zVENmpRSX aSb5rcCUbxS4lflHe2/eF2b7B1zdttp/r87HbYAra4FxhCRoz3Xqlx0bUAeanY3Iy3mw Hn25NTcCBz421ufoP38s4toDgZB25loHMuH7Pcb1USFwmpuoYiqbsI9Fj17/pw5MrCM1 xZD+Z+nWoRWPcsMc73XS6zcqaYMXO6M7ETiMtBJxdc/uPLaiM7bf/z3RWlOhzk1Zd7Bh H3Fg== X-Gm-Message-State: AA+aEWYL77UU5umc5g5pQTBfoEkDca0CAoXXNiwwvUpS2GrBsAlDm9vk Kp5eqLcdMdaXL+onBvPK2Wm+a/SSxspmPPNVDV7/Sg== X-Google-Smtp-Source: AFSGD/XSHQyQgEviajp/TYVr5AJhDoJ3DvoPFBd6TT/jy+W5bjd4wCVGRuqridnURK8Xl+LtlwKJ0E7CgZKBdiiQVgY= X-Received: by 2002:ac8:3f2d:: with SMTP id c42mr50755766qtk.33.1546619322157; Fri, 04 Jan 2019 08:28:42 -0800 (PST) MIME-Version: 1.0 References: <8ECF7513-9DFB-46EF-86BA-DB717D713792@sarenet.es> In-Reply-To: From: Warner Losh Date: Fri, 4 Jan 2019 09:28:31 -0700 Message-ID: Subject: Re: Interesting: ZFS scrub prefetch hurting sequential scrub performance? To: Borja Marcos Cc: FreeBSD FS X-Rspamd-Queue-Id: 073166F99F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=fFsZfk81 X-Spamd-Result: default: False [-4.56 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.972,0]; IP_SCORE(-2.59)[ip: (-9.02), ipnet: 2607:f8b0::/32(-2.17), asn: 15169(-1.66), country: US(-0.08)]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[9.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2019 16:28:44 -0000 On Fri, Jan 4, 2019 at 3:53 AM Borja Marcos wrote: > > > > On 3 Jan 2019, at 11:34, Borja Marcos wrote: > > > > > > Hi, > > > > I have noticed that my scrubs have become painfully slow. I am wonderin= g > wether I=E2=80=99ve just hit some worst case or maybe > > there is some interaction between the ZFS sequential scrub and scrub > prefetch. I don=E2=80=99t recall seeing this behavior > > before the sequential scrub code was committed. > > > > Did I hit some worst case or should scrub prefetch be disabled with the > new sequential scrub code? > > I have done a test with the old scrub code (vfs.zfs.zfs_scan_legacy=3D1) = and > I see a very similar behavior, with the > scrub stalling again. > > Once more, disabling prefetch for the scrub (vfs.zfs.no_scrub_prefetch=3D= 1) > solves the issue. > > I suffered this problem on 11 at some point but I attributed it (wrongly!= ) > to hardware problems at the time. > > Not I=E2=80=99ve just found a talk about a new prefetch mechanism for the= scrub by > Tom Caputi. Could it be the problem? > https://www.youtube.com/watch?v=3Dupn9tYh917s It's always been a hard problem to schedule background activity without affecting foreground performance. For Hard Drives this isn't so terrible to do: keep the queue depths small so that when any new work arrives, the latency in switching between the two workloads is small. With SSDs, it gets harder, though in a read-only workload it degenerates to about the same. SSDs do their own read ahead, sometimes, and they have lots of background activity that can be triggered by reads (like if it could read the block, but the error rate from the NAND was over some threshold, the drive might decide to copy all the data out of that block because data with that error rate won't be readable with the correction codes in place long enough to meet the retention specs). And writes can also trigger this background behavior. So switching between the foreground and background tasks becomes even more sluggish. But I think in ZFS' case, it may just be a bit of a bug in backing off the scrub operation to allow better local host performance... Warner