Date: Sun, 11 Dec 2022 08:08:55 -0700 From: Warner Losh <imp@bsdimp.com> To: Artem Kuchin <artemkuchin76@gmail.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Everchanging bytes at the end of mirror disks Message-ID: <CANCZdfqTS6S7fYrLo3asGZ0WEWapgO8Ej7gGo09XkZdPNL90kA@mail.gmail.com> In-Reply-To: <f8ecd698-bc03-43f1-9596-8b3ec1432a5e@gmail.com> References: <85c5a64c-915e-d790-e617-c94f3fb7cd9a@gmail.com> <CANCZdfrO4M49i_WdidJUujQoJH1pAooTfxFtVjrNLU77EseH7Q@mail.gmail.com> <f8ecd698-bc03-43f1-9596-8b3ec1432a5e@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000009df31c05ef8ec4c6
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Sun, Dec 11, 2022 at 1:45 AM Artem Kuchin <artemkuchin76@gmail.com>
wrote:
> 11.12.2022 11:22, Warner Losh =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
>
>
>
> On Sat, Dec 10, 2022, 11:52 PM Artem Kuchin <artemkuchin76@gmail.com>
> wrote:
>
>> Hello!
>>
>> I am writing a small utility for myseld and part of it is comparing
>> gmirror disks. After running some tests i realized that some bytes at
>> the very end of disks are constantly changing.
>>
>
> The last sector has metadata about the mirror and about the mirror
> element. It's this latter data that differs.
>
>
> Thank you for reply. Then there are several question
>
> 1) Last SECTOR is not always 512KB or is it? Do i need to get block size
> from diskinfo and subtract its size from disk size?
>
diskinfo(1) will tell you, it's returned with the DIOCGSECTORSIZE ioctl.
> 2) Why its content is changing so often? On every write? How often? The
> only place to look for description is the gmirror sources?
>
When a mirror breaks (that is, writes can happen to one side but not the
other), we need to know right away which side is the more current one. The
gmirror does this by modifying the metadata to record how many writes have
happened to each mirror member (one reason that write is so expensive).
> It does not look good to me, but maybe i am wrong? Also, does it mean no
go for gmirror on ssd?
No. It's fine. All SSDs in the past 15-20 years have wear leveling (and
nearly all for an additional 10 years before that). It's quite hard to wear
out a device by repeated writing to one sector. You effectively have to
write the same amount of data you would if you were writing to multiple
sectors. SSDs are rated in 'drive writes per day': how many times you can
write to all the sectors of a drive, every day, for the warranty period of
the device. This is between 0.3 and 5 typically (though exceptions exist).
Any extra writes will be several orders of magnitude below this threshold
for all but the most insane write patterns (eg write all the odd sectors,
randomly, then write all the even sectors randomly, repeatedly). And if you
are doing an insane amount of writing, you likely wouldn't be using
gmirror.... It at most doubles the traffic to the drive, but if you have a
64k block size to UFS, you'd typically see only a few percent increase. So
unless you are writing your data to the drives at rates approaching the
endurance limit of the drive, this extra write won't be an issue.[*]
Warner
[*] It would theoretically be helpful,though, if gmirror could add an extra
N sectors to match the underlying physical hardware page sizes, but the
experiments I've done I've not been able to see a speed increase....
--0000000000009df31c05ef8ec4c6
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Sun, Dec 11, 2022 at 1:45 AM Artem=
Kuchin <<a href=3D"mailto:artemkuchin76@gmail.com">artemkuchin76@gmail.=
com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x">
=20
=20
=20
<div>
<div>11.12.2022 11:22, Warner Losh =D0=BF=D0=B8=D1=88=D0=B5=D1=82:<br>
</div>
<blockquote type=3D"cite">
=20
<div dir=3D"auto">
<div><br>
<br>
<div class=3D"gmail_quote">
<div dir=3D"ltr" class=3D"gmail_attr">On Sat, Dec 10, 2022,
11:52 PM Artem Kuchin <<a href=3D"mailto:artemkuchin76@gma=
il.com" target=3D"_blank">artemkuchin76@gmail.com</a>>
wrote:<br>
</div>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello!<br>
<br>
I am writing a small utility for myseld and part of it is
comparing <br>
gmirror disks. After running some tests i realized that
some bytes at <br>
the very end of disks are constantly changing.<br>
</blockquote>
</div>
</div>
<div dir=3D"auto"><br>
</div>
<div dir=3D"auto">The last sector has metadata about the mirror
and about the mirror element.=C2=A0 It's this latter data tha=
t
differs.</div>
<div dir=3D"auto"><br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Thank you for reply. Then there are several question</p>
<p>1) Last SECTOR is not always 512KB or is it? Do i need to get
block size from diskinfo and subtract its size from disk size?</p></d=
iv></blockquote><div><br></div><div>diskinfo(1) will tell you, it's ret=
urned with the=C2=A0DIOCGSECTORSIZE ioctl.</div><div>=C2=A0</div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s=
olid rgb(204,204,204);padding-left:1ex"><div>
<p>2) Why its content=C2=A0 is changing so often? On every write? How o=
ften? The only place to look for description is the gmirror sources?</p></d=
iv></blockquote><div>When a mirror breaks (that is, writes can happen to on=
e side but not the other), we need to know right away which side is the mor=
e current one. The gmirror does this by modifying the metadata to record ho=
w many writes have happened to each mirror member (one reason that write is=
so expensive).</div><div><br></div><div>> It does not look good to me, =
but maybe i am wrong? Also, does it mean no go for gmirror on ssd?</div><fo=
nt color=3D"#888888"><br class=3D"gmail-Apple-interchange-newline"></font><=
div>No. It's fine. All SSDs in the past 15-20 years have wear leveling =
(and nearly all for an additional 10 years before that). It's quite har=
d to wear out a device by repeated writing to one sector. You effectively h=
ave to write the same amount of data you would if you were writing to multi=
ple sectors. SSDs are rated in 'drive writes per day': how many tim=
es you can write to all the sectors of a drive, every day, for the warranty=
period of the device. This is between 0.3 and 5 typically (though exceptio=
ns exist).=C2=A0 Any extra writes will be several orders of magnitude=C2=A0=
below this threshold for all but the most insane write patterns (eg write a=
ll the odd sectors, randomly, then write all the even sectors randomly, rep=
eatedly). And if you are doing an insane amount of writing, you likely woul=
dn't be using gmirror.... It at most doubles the traffic to the drive, =
but if you have a 64k block size to UFS, you'd typically see only a few=
percent increase. So unless you are writing your data to the drives at rat=
es approaching the endurance limit of the drive, this extra write won't=
be an issue.[*]</div><div><br></div><div>Warner</div><div><br></div><div>[=
*] It would theoretically be helpful,though, if gmirror could add an extra =
N sectors to match the underlying physical hardware page sizes, but the exp=
eriments I've done I've not been able to see a speed increase....</=
div></div></div>
--0000000000009df31c05ef8ec4c6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqTS6S7fYrLo3asGZ0WEWapgO8Ej7gGo09XkZdPNL90kA>