From owner-freebsd-fs@freebsd.org Thu Apr 5 19:56:24 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BE43F9E603 for ; Thu, 5 Apr 2018 19:56:24 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F0BAC6E274 for ; Thu, 5 Apr 2018 19:56:23 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-wm0-f47.google.com with SMTP id i3so10418591wmf.3 for ; Thu, 05 Apr 2018 12:56:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=SKgaRhr52mMRfKws1pB6BE8CSGetUzgXsOAt7/ir/KU=; b=oa3oBFu7KG3w4vC9Oa0kP1ZvIo8aB71zQTL+GLGXkvzYRLmjoxNHUQJ45CO2rC7pZ4 XSAhxdSLJEkSMeQz8h1nlId1D2yc4U1lP74SkHfEVNSgwfRdVJKRCYk8FjxQrRbQGiBa ETIgG1IJ+x1iIvynJACaFGfUdQpAJBaOGgZhZ6LvREFnP2L/lY5J+S7xreUKG7aQvbUB Vd/lNDTNXsVfjtJK6lLIDsts8Ta7uxdrenUulF2eQFORJ+eYNrnKiq/1XJerGJzeDupt gLKCCu5hJLN08puExtt9FjcLTfKRcgdqIBZPvUa08xpyKGaf5EvUkTfoyI4vmUGJJie3 vODw== X-Gm-Message-State: ALQs6tAIPRhuOWuJ0OHlx/1KfZmvoYdHfgHDfBaKU+GUQ5NYcJsZfK0d qkbmg9BRnay7qlGKW+s11TK9jNWB X-Google-Smtp-Source: AIpwx4+kp2RWzmVvOqRQg1r8v2wPuYm1hzQGcWLhfARHojKZDRQmOxYowWp0hPeEhSONWjyzGEb94w== X-Received: by 10.46.56.6 with SMTP id f6mr15246400lja.4.1522958176533; Thu, 05 Apr 2018 12:56:16 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 93-v6sm1685563lfy.5.2018.04.05.12.56.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Apr 2018 12:56:15 -0700 (PDT) Subject: Re: ZFS dedup write pathway - possible inefficiency or..? To: Stilez Stilezy Cc: freebsd-fs@freebsd.org References: <14c857cc-463f-a56e-bcf6-c0702da6d3bc@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <5fbf6ac8-fcb4-0d2f-9f8d-aad129eda118@FreeBSD.org> Date: Thu, 5 Apr 2018 22:56:14 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2018 19:56:24 -0000 On 05/04/2018 20:11, Stilez Stilezy wrote: > > That does suck, but it's how bugs are.  I'm using 10G and dedup, so even with > good hardware the test case is definitely piling on the incoming data and > workload during writes, probably as fast as the OS can take it, or a little > more. Without dedup it flies, though. Hard to disentangle what's dedup overhead > and what's dedup write bug, though, or to get an idea which is responsible for > how much of the problem. > > As I don't know Illumos/OpenZFS's track record with bugs, is this likely to get > attention/resolved "at some time this year" or is it a "who knows, bugs take > forever, could be still here in 5 years time"? Is it helpful if I nudge and > offer a "causing a problem" note on their bug tracker? Perhaps it's worth it in > case it gives extra data? What do you reckon? I don't know. At this time it seems that there is not much interest in the issue. But maybe tomorrow someone will get excited and fix the problem in an hour. Or maybe that will happen in 5 years. > Last, if I have high data rates but want to minimise the dirty data issue, wcan > you suggest broadly, how to customise the dirty data/caching sysctls/loaders, to > try and mitigate the impact (get best possible handling without losing 99% of > throughput or sky-high latency?). Minimizing maximum dirty data threshold has its flip side that also affects performance. I am far from sure that tuning that can actually help. If you want to experiment, you can start with $ sysctl -d vfs.zfs | fgrep -i dirty > I understand from your reply that there's no recipes in debugging, but any > suggestions at all which way to try for at least some mitigation, or which > values might be worth experimenting with, to reduce the effect of the problem > pathway? I don't know of any mitigation. But I want to note that typically "I want to use dedup" and "I want to be able to write as fast as possible" arise for very different use cases and both are rarely needed at the same time (except for benchmarks, synthetic tests, etc). If one needs streaming writes then usually there is not much to dedup. -- Andriy Gapon