Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 5 Jul 2021 16:49:20 +0100
From:      Pete French <petefrench@ingresso.co.uk>
To:        stable@freebsd.org
Subject:   Re: ZFS + mysql appears to be killing my SSD's
Message-ID:  <4b5a0367-96c2-1d3f-bcb2-a76bd66bc76e@ingresso.co.uk>
In-Reply-To: <f15cfc5a-e3c0-f1a7-c123-d369db9bc199@denninger.net>
References:  <89c37c3e-22e8-006e-5826-33bd7db7739e@ingresso.co.uk> <2fd9b7e4-dc75-fedc-28d7-b98191167e6b@freebsd.org> <9c71d627-55b8-2464-6cc9-489e4ce98049@ingresso.co.uk> <f15cfc5a-e3c0-f1a7-c123-d369db9bc199@denninger.net>

next in thread | previous in thread | raw e-mail | index | archive | help


On 05/07/2021 16:09, Karl Denninger wrote:
> As noted elsewhere assuming ashift=12 the answer on write amplification 
> is no.
> 
> Geli should be initialized with -s 4096; I'm assuming you did that?

Yup, I have everything defaulting to 4k sectors now, and I was hoping 
there was no write amplification going on if I did that.

> I have a 5-unit geli-encrypted root pool, all Intel 240gb SSDs. They do 
> not report remaining lifetime via smart but do report indications of 
> trouble.  Here's one example snippet from one of the drives in that pool:

Thanks for this ... its inyeresting to have something to
compare against. Mine are Kingston 240G drives, though some
of the replacements will be Crucial (i.e. Micron)



> Device Statistics (GP Log 0x04)
> Page  Offset Size        Value Flags Description
> 0x01  =====  =               =  ===  == General Statistics (rev 2) ==
> 0x01  0x008  4             100  ---  Lifetime Power-On Resets
> 0x01  0x018  6    118722148102  ---  Logical Sectors Written
> 0x01  0x020  6        89033895  ---  Number of Write Commands
> 0x01  0x028  6     93271951909  ---  Logical Sectors Read
> 0x01  0x030  6         6797990  ---  Number of Read Commands

So this is interesting, because my vaues for the machine I was
describing arlier, where a drive was replaced are

ada0:0x01  0x018  6      3084505072  ---  Logical Sectors Written
ada0:0x01  0x020  6      1219372767  ---  Number of Write Commands
ada0:0x01  0x028  6      1168765724  ---  Logical Sectors Read
ada0:0x01  0x030  6        56588781  ---  Number of Read Commands

ada1:0x01  0x018  6      2014784574  ---  Logical Sectors Written
ada1:0x01  0x020  6        68557083  ---  Number of Write Commands
ada1:0x01  0x028  6        41230034  ---  Logical Sectors Read
ada1:0x01  0x030  6         2438618  ---  Number of Read Commands

The replacement has more written sectors than the one which wasnt 
replaced. yet these are in RAID-1, so how can that be? The
numbers for sectors written are way lower than yours, but the write
commands are much higher. For a pair of drives in RAID-1 where ada0 has
been replaced and resilvered from ada1, those looks very wrong to me.


> 6 years in-use, roughly, and no indication of anything going on in terms 
> of warnings about utilization or wear-out.  There is a MYSQL database on

yes, your figures are what I expect to see, but the ones above are what 
I actually have. The disparity between the two halves of the RAID are 
bothering me, I have to say.

Do you have any tuning applies to your ZFS filesystems ? I have reduced
the recrodsize on the data directories to 32k to better match innodb
and have turned off atime and sync. But those are things I would expect
to reduce the write load, not increase it.

-pete.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4b5a0367-96c2-1d3f-bcb2-a76bd66bc76e>