From owner-freebsd-fs@freebsd.org Thu Apr 5 17:11:57 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1BB4F93CC5 for ; Thu, 5 Apr 2018 17:11:56 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: from mail-vk0-x235.google.com (mail-vk0-x235.google.com [IPv6:2607:f8b0:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 74EBC86ACC; Thu, 5 Apr 2018 17:11:56 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: by mail-vk0-x235.google.com with SMTP id n64so11745463vkf.12; Thu, 05 Apr 2018 10:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JPtADEI/S+Rcb45a4GqyOmkr5vDXJWTTCPtep/EWhms=; b=btlgqcEnpweoyr6hJaIIuq51wR9tNJAOkPJysiqHPJuHVZV0lrJwT5HQ0GxocqHoKo WoG+6NGNHFOYmJw7IpJLZ/PEvqi4dHkaquH+27p6jPy84AqJsUf8geD0LxD4q9IqI2A/ g7fuefcCWxOulA3gKex4aCGb7+ouSx9j+TFYcAf1ZMMR5sC5SW3ZVLPqW5gnBPoQMTfu 6eyhzN6oyZEYsJvp/ZLWHzVNpil0CviHsnrbiRa/xiy8a2YJJAHfF1rbGDgcKS1Z/d6e oQX+K6Neh/dt0Auzb8X3Kndr7Dpo0tdB9+25gLiA2Sw3nCvsUATVUHtZYtAYLRdrP+Iw jQpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JPtADEI/S+Rcb45a4GqyOmkr5vDXJWTTCPtep/EWhms=; b=bAuH+TIXtvMwKTJn2Gk09D1pW29TY+CiyLfxKqc1cSitSitHwIrh0j+4OgRKF5m6ng 5ijFMcvDXTT1Hi77G13MQPoA783p4v33DUPD0/oZ3ncf9IRA2nmXn2BSybDoQzsGH6VH +b3E2xEqagIpXlTNbp2HCpDhAPo9SddBZ/yGsrimr2wNZp+cHaqoLWi2wMbCdAvlUt+k Uo/A6/rMWa6Qp0d3nQQ09UikB6P3wTSiRE3zajUUhBafSKB/LQbQYkQ5ytCKu/Sz7+5F 1mmaSRviuLA6hCN+0aLWAvhglAoGDvnFqD4CwtqJI9z/ENy7otS2yboBlN/fpfSEi3iN dxDw== X-Gm-Message-State: ALQs6tAr/xMXRDF2X0ba4gPgAtg0w0dpiXicHpx5MInikmTS4aZm/ud8 RCnuL5N+vmf7KtmfpiNm7wbXJdUQukx8nfahBBo= X-Google-Smtp-Source: AIpwx497EVPxPUy5nGy2wjsk+OD0y7hrvYi0pcDJSw02+B9IHpPNb+OD+Tx0c/fTR1ygOQIGEiyIxbeHR2H1thmf0dg= X-Received: by 10.31.147.212 with SMTP id v203mr13972242vkd.39.1522948315778; Thu, 05 Apr 2018 10:11:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.219.148 with HTTP; Thu, 5 Apr 2018 10:11:25 -0700 (PDT) In-Reply-To: References: <14c857cc-463f-a56e-bcf6-c0702da6d3bc@FreeBSD.org> From: Stilez Stilezy Date: Thu, 5 Apr 2018 18:11:25 +0100 Message-ID: Subject: Re: ZFS dedup write pathway - possible inefficiency or..? To: Andriy Gapon Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2018 17:11:57 -0000 Thanks Andriy, That does suck, but it's how bugs are. I'm using 10G and dedup, so even with good hardware the test case is definitely piling on the incoming data and workload during writes, probably as fast as the OS can take it, or a little more. Without dedup it flies, though. Hard to disentangle what's dedup overhead and what's dedup write bug, though, or to get an idea which is responsible for how much of the problem. As I don't know Illumos/OpenZFS's track record with bugs, is this likely to get attention/resolved "at some time this year" or is it a "who knows, bugs take forever, could be still here in 5 years time"? Is it helpful if I nudge and offer a "causing a problem" note on their bug tracker? Perhaps it's worth it in case it gives extra data? What do you reckon? Last, if I have high data rates but want to minimise the dirty data issue, wcan you suggest broadly, how to customise the dirty data/caching sysctls/loaders, to try and mitigate the impact (get best possible handling without losing 99% of throughput or sky-high latency?). I understand from your reply that there's no recipes in debugging, but any suggestions at all which way to try for at least some mitigation, or which values might be worth experimenting with, to reduce the effect of the problem pathway? Stilez On 4 April 2018 at 18:04, Andriy Gapon wrote: > On 02/04/2018 21:30, Stilez Stilezy wrote: > > Thanks on that tip, Andriy, > > > > I don't have the knowledge to tell if that's the first of the 2 issues > I'm > > seeing. It could be. What exactly is that bug describing and what > behaviour > > would it create? > > Sub-optimal performance for ZFS write throughput (and latency) when dedup > is > enabled and you are trying to write as fast as you can. > > > Assuming it's the same - > > > > 1) Are there any known workarounds or sysctls that help to reduce the > issue? > > I don't know of any. > > > 2) Are there any easy diagnostic tests I can easily do, to confirm if > this is > > the same behaviour as that bug? Nobody else uses the test system, so I > can > > change any sysctls to sane or extreme values for testing, or create > large txg's > > and spread out reads and writes to see what's happening and what > coincides with > > what. > > We used DTrace to observe internal ZFS behavior. > I do not have any simple recipes. > > > On 2 April 2018 at 18:47, Andriy Gapon > > wrote: > > > > On 02/04/2018 16:36, Stilez Stilezy wrote: > > > The first issue is specific to the > > > dedup write pathway. I've tested locally to a point where it > seems it's > > > not due to inadequate hardware and it's very consistent and > specific, even > > > on idle conditions/minimal load. I'm wondering whether there's a > code > > > bottleneck specifically affecting just the dedup write pathway. > > > > I think that this might be https://www.illumos.org/issues/8353 > > > > > > -- > > Andriy Gapon > > > > > > > -- > Andriy Gapon >