Date: Fri, 6 Sep 2024 17:48:57 -0400 From: Chris Ross <cross+freebsd@distal.com> To: Wes Morgan <morganw@gmail.com> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: Unable to replace drive in raidz1 Message-ID: <E50559CA-CC3D-45AE-82D7-172270BF4851@distal.com> In-Reply-To: <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com> References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <AC67D073-D476-41F5-AC53-F671430BB493@distal.com> <CAOtMX2h52d0vtceuwcDk2dzkH-fZW32inhk-dfjLMJxetVXKYg@mail.gmail.com> <CB79EC2B-E793-4561-95E7-D1CEEEFC1D72@distal.com> <CAOtMX2i_zFYuOnEK_aVkpO_M8uJCvGYW%2BSzLn3OED4n5fKFoEA@mail.gmail.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <CAOtMX2jfcd43sBpHraWA=5e_Ka=hMD654m-5=boguPPbYXE4yw@mail.gmail.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> <CAOtMX2hRJvt9uhctKvXO4R2tUNq9zeCEx6NZmc7Vk7fH=HO8eA@mail.gmail.com> <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> <E6C615C1-E9D2-4F0D-8DC2-710BAAF10954@distal.com> <E85B00B1-7205-486D-800C-E6837780E819@gmail.com> <E93A9CA8-6705-4C26-9F33-B620A365F4BD@distal.com> <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Sep 6, 2024, at 17:22, Wes Morgan <morganw@gmail.com> wrote:
>=20
> The labels are helpful for fstab, but zfs doesn't need fstab. In the =
early days of zfs on freebsd the unpartitioned device was recommended; =
maybe that's not accurate any longer, but I still follow it for a pool =
that contains vdevs with multiple devices (raidz).=20
>=20
> If you use, e.g., da0 in a pool, you cannot later replace it with a =
labeled device of the same size; it won't have enough sectors.=20
The problem is shown here. da3 was in a pool. Then, when the system =
rebooted, da3 was the kernels name for a different device in a different =
pool. Had I known then how to interact with the guid (status -g), I =
likely would=E2=80=99ve been fine.
>> So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, =
then I shut the system down, removed that physical device replacing it =
with a larger disk, and rebooted. I suspect the =E2=80=9Coffline=E2=80=9D=
s after that are me experimenting when it was telling me it couldn=E2=80=99=
t start the replace action I was asking for.
>=20
> This is probably where you made your mistake. Rebooting shifted =
another device into da3. When you tried to offline it, you were probably =
either targeting a device in a different raidz or one that wasn't in the =
pool. The output of those original offline commands would have been =
informative. You could also check dmesg and map the serial numbers to =
device assignments to figure out what device moved to da3.
I offline=E2=80=99d =E2=80=9Cda3=E2=80=9D before I rebooted. After =
rebooting, I tried the obvious and correct (i thought) =E2=80=9Czpool =
replace da3 da10=E2=80=9D only to get the error I=E2=80=99ve been =
getting since. Again, had I known how to use the guid for the device =
that used to be da3 but now isn=E2=80=99t, that might=E2=80=99ve worked. =
I can=E2=80=99t know now.
Then, while trying to fix the problem, I likely made it worse by trying =
to interact with da3, which in the pools brain was a missing disk in =
raidz1-0, but the kernel also knew /dev/da3 to be a working disk (that =
happened to be one of the drives in raidz1-1). I feel that zfs did =
something wrong somewhere if it _ever_ tried to talk to /dev/da3 when I =
said =E2=80=9Cda3=E2=80=9D after I rebooted and it found that device to =
be part of raidz1-1, but.
> Sounds about right. In another message it seemed like the pool had =
started an autoreplace. So I assume you have zfsd enabled? That is what =
issues the replace command. Strange that it is not anywhere in the pool =
history. There should be syslog entries for any actions it took.
I don=E2=80=99t think so. That message about some =E2=80=9Calready in =
replacing/spare config=E2=80=9D came up before anything else. At which =
point, I=E2=80=99d never had a spare in this pool, and there was no =
replace shown in zpool status.
> In your case, it appears that you had two missing devices - the =
original "da3" that was physically removed, and the new da3 that you =
forced offline. You added da10 as a spare, when what you needed to do =
was a replace. Spare devices do not auto-replace without zfsd running =
and autoreplace set to on.
I did offline =E2=80=9Cda3=E2=80=9D a couple of times, again thinking I =
was working with what zpool showed as =E2=80=9Cda3=E2=80=9D. If it did =
anything with /dev/da3 there, then I think that may be a bug. Or, at =
least, something that should be made more clear. It _didn=E2=80=99t_ =
offline the diskid/DISK-K1GMBN9D from raidz1-1, which is what the kernel =
has at da3. So.
> This should all be reported in zpool status. In your original message, =
there is no sign of a replacement in progress or a spare device, =
assuming you didn't omit something. If the pool is only showing that a =
single device is missing, and that device is to be replaced by da10, =
zero out the first and last sectors (I think a zfs label is 128k?) to =
wipe out any labels and use the replace command, not spare, e.g. "zpool =
replace tank da3 da10", or use the missing guid as suggested elsewhere. =
This should work based on the information provided.
I=E2=80=99ve never seen a replacement going on, and I have had the new =
disk =E2=80=9Cda10=E2=80=9D as a spare a couple of times while testing. =
But it wasn=E2=80=99t left there after I determined that that also =
didn=E2=80=99t let me get it replaced into the raidz.
And, that attempt to replace is what I=E2=80=99ve tried many times, with =
multiple id=E2=80=99s. I have cleared the label on da10 multiple times. =
That replace doesn=E2=80=99t work, giving this error message in all =
cases.
- Chris
% glabel status
Name Status Components
diskid/DISK-BTWL503503TW480QGN N/A ada0
gpt/l2arc N/A ada0p1
gptid/9d00849e-0b82-11ec-a143-84b2612f2c38 N/A ada0p1
diskid/DISK-K1GMBN9D N/A da3
diskid/DISK-3WJDHJ2J N/A da6
diskid/DISK-3WK3G1KJ N/A da7
diskid/DISK-3WJ7ZMMJ N/A da8
diskid/DISK-K1GMEDMD N/A da4
diskid/DISK-K1GMAX1D N/A da5
ufs/drive12 N/A da9
diskid/DISK-ZGG0A2PA N/A da10
% zpool status tank
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent =
errors.
Sufficient replicas exist for the pool to continue functioning =
in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the =
device
repaired.
scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 =
09:08:34 2024
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
da3 FAULTED 0 0 0 external =
device fault
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
diskid/DISK-K1GMBN9D ONLINE 0 0 0
diskid/DISK-K1GMEDMD ONLINE 0 0 0
diskid/DISK-K1GMAX1D ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
diskid/DISK-3WJDHJ2J ONLINE 0 0 0
diskid/DISK-3WK3G1KJ ONLINE 0 0 0
diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0
errors: No known data errors
% sudo zpool replace tank da3 da10
Password:
cannot replace da3 with da10: already in replacing/spare config; wait =
for completion or use 'zpool detach'
% zpool status -g tank
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent =
errors.
Sufficient replicas exist for the pool to continue functioning =
in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the =
device
repaired.
scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 =
09:08:34 2024
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
16506780107187041124 DEGRADED 0 0 0
9127016430593660128 FAULTED 0 0 0 external =
device fault
4094297345166589692 ONLINE 0 0 0
17850258180603290288 ONLINE 0 0 0
5104119975785735782 ONLINE 0 0 0
6752552549817423876 ONLINE 0 0 0
9072227575611698625 ONLINE 0 0 0
13778609510621402511 ONLINE 0 0 0
11410204456339324959 ONLINE 0 0 0
1083322824660576293 ONLINE 0 0 0
12505496659970146740 ONLINE 0 0 0
11847701970749615606 ONLINE 0 0 0
errors: No known data errors
% sudo zpool replace tank 9127016430593660128 da10
cannot replace 9127016430593660128 with da10: already in replacing/spare =
config; wait for completion or use 'zpool detach'
% sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA
cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in =
replacing/spare config; wait for completion or use 'zpool detach'
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E50559CA-CC3D-45AE-82D7-172270BF4851>