Date: Mon, 22 Apr 2024 15:13:08 -0400 From: mike tancsa <mike@sentex.net> To: =?UTF-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@FreeBSD.org> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: block size: 512B configured, 4096B native all of a sudden Message-ID: <01bd9c82-52dd-4983-be08-a9c1810cbaaa@sentex.net> In-Reply-To: <9dd52865-6928-41dc-910b-82988fe6b2af@sentex.net> References: <9a593ca1-4975-438a-afec-d8dd5199dbcf@sentex.net> <86bk618lyh.fsf@ltc.des.dev> <d3536b16-fbdb-4a9f-a85c-47b1cc7213db@sentex.net> <867cgp8lgu.fsf@ltc.des.dev> <57a1cc6b-2bd2-4292-9fde-f8799ffa291a@sentex.net> <8634rd8l2p.fsf@ltc.des.dev> <e089d0a9-f659-4ebe-8d6b-b09e7ecaa265@sentex.net> <86y19575y9.fsf@ltc.des.dev> <9dd52865-6928-41dc-910b-82988fe6b2af@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------TeEYKHRauDcwvPTkUFKj80vS
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
On 4/22/2024 12:05 PM, mike tancsa wrote:
> On 4/22/2024 11:49 AM, Dag-Erling Smørgrav wrote:
>> mike tancsa<mike@sentex.net> writes:
>>> I was afraid of that. So basically I have to copy the entire pool or
>>> live with the performance penalty :( 73TB is a lot to copy / move.
>> Oh I didn't realize those disks were part of the larger pool, I thought
>> they were a separate pool named “special”. You might be able to simply
>> detach and then re-attach both drives, afaik a draid can function
>> without its special vdev.
>
> Yeah, its an actual special vdev I added to speed up metadata and
> small file access. zpool-detach I think only works on mirrors and not
> raidz vdevs. From searching google, there does not seem to be a way
> to detach a special vdev from a raidz pool.
>
>
I think I confirmed my suspicion with a quick test pool
truncate -s 100G /quirk-test/junk.raw
truncate -s 100G /quirk-test/junk2.raw
mdconfig -f /quirk-test/junk.raw
mdconfig -f /quirk-test/junk2.raw
gpart create -s gpt md0
gpart create -s gpt md1
gpart add -s 20G -t freebsd-zfs /dev/md0
gpart add -s 20G -t freebsd-zfs /dev/md0
gpart add -s 20G -t freebsd-zfs /dev/md0
gpart add -s 20G -t freebsd-zfs /dev/md1
gpart add -s 20G -t freebsd-zfs /dev/md1
gpart add -s 20G -t freebsd-zfs /dev/md1
zpool create testpool raidz1 /dev/md0p1 /dev/md1p1 /dev/md0p2 /dev/md1p2
zpool add testpool special mirror /dev/md0p3 /dev/md1p3
zpool status
pool: testpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
md0p1 ONLINE 0 0 0
md1p1 ONLINE 0 0 0
md0p2 ONLINE 0 0 0
md1p2 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
md0p3 ONLINE 0 0 0
md1p3 ONLINE 0 0 0
# zpool remove testpool mirror-1
cannot remove mirror-1: invalid config; all top-level vdevs must have
the same sector size and not be raidz.
zpool detach just nukes the mirror I think for the special vdev
# zpool detach testpool /dev/md1p3
# zpool status
pool: quirk-test
state: ONLINE
scan: resilvered 128G in 00:08:52 with 0 errors on Mon Apr 8
14:37:29 2024
config:
NAME STATE READ WRITE CKSUM
quirk-test ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da4p1 ONLINE 0 0 0
da1p1 ONLINE 0 0 0
da2p1 ONLINE 0 0 0
da5p1 ONLINE 0 0 0
errors: No known data errors
pool: testpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
md0p1 ONLINE 0 0 0
md1p1 ONLINE 0 0 0
md0p2 ONLINE 0 0 0
md1p2 ONLINE 0 0 0
special
md0p3 ONLINE 0 0 0
errors: No known data errors
# zpool remove testpool /dev/md0p3
cannot remove /dev/md0p3: invalid config; all top-level vdevs must have
the same sector size and not be raidz.
# zpool detach testpool /dev/md0p3
cannot detach /dev/md0p3: only applicable to mirror and replacing vdevs
---Mike
--------------TeEYKHRauDcwvPTkUFKj80vS
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 4/22/2024 12:05 PM, mike tancsa
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:9dd52865-6928-41dc-910b-82988fe6b2af@sentex.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">On 4/22/2024 11:49 AM, Dag-Erling
Smørgrav wrote:<br>
</div>
<blockquote type="cite" cite="mid:86y19575y9.fsf@ltc.des.dev">
<pre class="moz-quote-pre" wrap="">mike tancsa <a
class="moz-txt-link-rfc2396E" href="mailto:mike@sentex.net"
moz-do-not-send="true"><mike@sentex.net></a> writes:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">I was afraid of that. So basically I have to copy the entire pool or
live with the performance penalty :( 73TB is a lot to copy / move.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Oh I didn't realize those disks were part of the larger pool, I thought
they were a separate pool named “special”. You might be able to simply
detach and then re-attach both drives, afaik a draid can function
without its special vdev.
</pre>
</blockquote>
<p>Yeah, its an actual special vdev I added to speed up metadata
and small file access. zpool-detach I think only works on
mirrors and not raidz vdevs. From searching google, there does
not seem to be a way to detach a special vdev from a raidz pool.</p>
<p><br>
</p>
</blockquote>
<p>I think I confirmed my suspicion with a quick test pool</p>
<p>truncate -s 100G /quirk-test/junk.raw<br>
truncate -s 100G /quirk-test/junk2.raw<br>
<br>
mdconfig -f /quirk-test/junk.raw<br>
mdconfig -f /quirk-test/junk2.raw<br>
gpart create -s gpt md0<br>
gpart create -s gpt md1<br>
gpart add -s 20G -t freebsd-zfs /dev/md0<br>
gpart add -s 20G -t freebsd-zfs /dev/md0<br>
gpart add -s 20G -t freebsd-zfs /dev/md0<br>
gpart add -s 20G -t freebsd-zfs /dev/md1<br>
gpart add -s 20G -t freebsd-zfs /dev/md1<br>
gpart add -s 20G -t freebsd-zfs /dev/md1<br>
zpool create testpool raidz1 /dev/md0p1 /dev/md1p1 /dev/md0p2
/dev/md1p2<br>
zpool add testpool special mirror /dev/md0p3 /dev/md1p3<br>
zpool status</p>
<p><br>
pool: testpool<br>
state: ONLINE<br>
config:<br>
<br>
NAME STATE READ WRITE CKSUM<br>
testpool ONLINE 0 0 0<br>
raidz1-0 ONLINE 0 0 0<br>
md0p1 ONLINE 0 0 0<br>
md1p1 ONLINE 0 0 0<br>
md0p2 ONLINE 0 0 0<br>
md1p2 ONLINE 0 0 0<br>
special<br>
mirror-1 ONLINE 0 0 0<br>
md0p3 ONLINE 0 0 0<br>
md1p3 ONLINE 0 0 0<br>
</p>
<p><br>
</p>
<p> # zpool remove testpool mirror-1<br>
cannot remove mirror-1: invalid config; all top-level vdevs must
have the same sector size and not be raidz.</p>
<p>zpool detach just nukes the mirror I think for the special vdev</p>
<p> # zpool detach testpool /dev/md1p3<br>
<br>
# zpool status<br>
pool: quirk-test<br>
state: ONLINE<br>
scan: resilvered 128G in 00:08:52 with 0 errors on Mon Apr 8
14:37:29 2024<br>
config:<br>
<br>
NAME STATE READ WRITE CKSUM<br>
quirk-test ONLINE 0 0 0<br>
raidz1-0 ONLINE 0 0 0<br>
da4p1 ONLINE 0 0 0<br>
da1p1 ONLINE 0 0 0<br>
da2p1 ONLINE 0 0 0<br>
da5p1 ONLINE 0 0 0<br>
<br>
errors: No known data errors<br>
<br>
pool: testpool<br>
state: ONLINE<br>
config:<br>
<br>
NAME STATE READ WRITE CKSUM<br>
testpool ONLINE 0 0 0<br>
raidz1-0 ONLINE 0 0 0<br>
md0p1 ONLINE 0 0 0<br>
md1p1 ONLINE 0 0 0<br>
md0p2 ONLINE 0 0 0<br>
md1p2 ONLINE 0 0 0<br>
special<br>
md0p3 ONLINE 0 0 0<br>
<br>
errors: No known data errors<br>
<br>
# zpool remove testpool /dev/md0p3<br>
cannot remove /dev/md0p3: invalid config; all top-level vdevs must
have the same sector size and not be raidz.<br>
<br>
# zpool detach testpool /dev/md0p3<br>
cannot detach /dev/md0p3: only applicable to mirror and replacing
vdevs<br>
</p>
<p><br>
</p>
<p> ---Mike<br>
</p>
<p></p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:9dd52865-6928-41dc-910b-82988fe6b2af@sentex.net">
<p> </p>
<span style="white-space: pre-wrap">
</span> </blockquote>
</body>
</html>
--------------TeEYKHRauDcwvPTkUFKj80vS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01bd9c82-52dd-4983-be08-a9c1810cbaaa>