From nobody Mon Jul 5 15:49:20 2021 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 960E97C8120 for ; Mon, 5 Jul 2021 15:49:23 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2001:470:6a18:411::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GJVVV5cj3z4cSs for ; Mon, 5 Jul 2021 15:49:22 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from [2001:470:6cc4:1:cd6:5836:ddba:7b54] (helo=balta.drayhouse.twisted.org.uk) by constantine.ingresso.co.uk with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.94.2 (FreeBSD)) (envelope-from ) id 1m0QqX-000NrQ-28 for stable@freebsd.org; Mon, 05 Jul 2021 15:49:21 +0000 Subject: Re: ZFS + mysql appears to be killing my SSD's To: stable@freebsd.org References: <89c37c3e-22e8-006e-5826-33bd7db7739e@ingresso.co.uk> <2fd9b7e4-dc75-fedc-28d7-b98191167e6b@freebsd.org> <9c71d627-55b8-2464-6cc9-489e4ce98049@ingresso.co.uk> From: Pete French Message-ID: <4b5a0367-96c2-1d3f-bcb2-a76bd66bc76e@ingresso.co.uk> Date: Mon, 5 Jul 2021 16:49:20 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4GJVVV5cj3z4cSs X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=ingresso.co.uk; spf=pass (mx1.freebsd.org: domain of petefrench@ingresso.co.uk designates 2001:470:6a18:411::3 as permitted sender) smtp.mailfrom=petefrench@ingresso.co.uk X-Spamd-Result: default: False [-1.76 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:6a18:411::3:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:6a18:411::3]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-0.93)[-0.930]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2001:470:6a18:411::3:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; NEURAL_SPAM_SHORT(0.96)[0.962]; DMARC_POLICY_ALLOW(-0.50)[ingresso.co.uk,none]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[stable]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 05/07/2021 16:09, Karl Denninger wrote: > As noted elsewhere assuming ashift=12 the answer on write amplification > is no. > > Geli should be initialized with -s 4096; I'm assuming you did that? Yup, I have everything defaulting to 4k sectors now, and I was hoping there was no write amplification going on if I did that. > I have a 5-unit geli-encrypted root pool, all Intel 240gb SSDs. They do > not report remaining lifetime via smart but do report indications of > trouble.  Here's one example snippet from one of the drives in that pool: Thanks for this ... its inyeresting to have something to compare against. Mine are Kingston 240G drives, though some of the replacements will be Crucial (i.e. Micron) > Device Statistics (GP Log 0x04) > Page  Offset Size        Value Flags Description > 0x01  =====  =               =  ===  == General Statistics (rev 2) == > 0x01  0x008  4             100  ---  Lifetime Power-On Resets > 0x01  0x018  6    118722148102  ---  Logical Sectors Written > 0x01  0x020  6        89033895  ---  Number of Write Commands > 0x01  0x028  6     93271951909  ---  Logical Sectors Read > 0x01  0x030  6         6797990  ---  Number of Read Commands So this is interesting, because my vaues for the machine I was describing arlier, where a drive was replaced are ada0:0x01 0x018 6 3084505072 --- Logical Sectors Written ada0:0x01 0x020 6 1219372767 --- Number of Write Commands ada0:0x01 0x028 6 1168765724 --- Logical Sectors Read ada0:0x01 0x030 6 56588781 --- Number of Read Commands ada1:0x01 0x018 6 2014784574 --- Logical Sectors Written ada1:0x01 0x020 6 68557083 --- Number of Write Commands ada1:0x01 0x028 6 41230034 --- Logical Sectors Read ada1:0x01 0x030 6 2438618 --- Number of Read Commands The replacement has more written sectors than the one which wasnt replaced. yet these are in RAID-1, so how can that be? The numbers for sectors written are way lower than yours, but the write commands are much higher. For a pair of drives in RAID-1 where ada0 has been replaced and resilvered from ada1, those looks very wrong to me. > 6 years in-use, roughly, and no indication of anything going on in terms > of warnings about utilization or wear-out.  There is a MYSQL database on yes, your figures are what I expect to see, but the ones above are what I actually have. The disparity between the two halves of the RAID are bothering me, I have to say. Do you have any tuning applies to your ZFS filesystems ? I have reduced the recrodsize on the data directories to 32k to better match innodb and have turned off atime and sync. But those are things I would expect to reduce the write load, not increase it. -pete.