From nobody Fri Sep 6 15:56:25 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gnx5BVTz5VNqG for ; Fri, 06 Sep 2024 15:56:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yb1-xb30.google.com (mail-yb1-xb30.google.com [IPv6:2607:f8b0:4864:20::b30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gnx39jsz4r08 for ; Fri, 6 Sep 2024 15:56:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb30.google.com with SMTP id 3f1490d57ef6-e1a9dc3efc1so2636579276.2 for ; Fri, 06 Sep 2024 08:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725638196; x=1726242996; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/yM3OBXmyU8OOArhxvmtC2osgjgkl+YZ/cOkdoIPeS0=; b=Qtik9/L0DWwlePrkAivUpRRV00n8PdJl3o8UZy4+KdFdim2tD3PEEJeoBQPIt1TE2n 5EaGMxRnAWsnLm+i2Q59susBG10qibNDp0IBV7T7BwlTa/wHNgSSIr6WyqyNa7T7GPLi oShTJBD3+OQjMk++22SFV6kfA1yI2u1RKtZmbR4ffaYYODeTFMu+MgNm+hM4RlZeHyfm nPXIq86CT7fi6VzEFfp+E2a0g0ouMKaXizkeHE3/YNA1JTSaVCk3ifEzOJoArLUJRHuB CmQUidJN9SxOmDjV4jUX7rHm335nwX/1wDkrYREpgIXDfGk67fZHeSbCPpSXk/gykmb3 ubyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725638196; x=1726242996; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/yM3OBXmyU8OOArhxvmtC2osgjgkl+YZ/cOkdoIPeS0=; b=c9tLcOm5Cnn1rbFPvw7QllWSTvQJVglPLXGohFL3HY87/D++30OW0ZeoYa33Ed+7us 9Sy9ksdngQIW3fw0CMpv/ArQXNHk01ZgbSr65/NvILULc5a+Qon2cnKCD/fXFx2VI9sa VqoZkKcbktNAyZpzf4xYjvxb9P1Jn6DLbAeHEthOgPgbjcVC53W8IYtMLtRBXpyqWoWD dj7pQrcNJgBBw8+HeMWbL7Vx4xZ27Rm1gfPD+dDwZ6WBIw79qLSA/wvv9iVZ/MKx84cp ASApIWy1HKHKKEH5WuJYfVXy1Heis5QUQm9m1Zl6Mn/rG76/EffSt4ctkhbJHNt1RETh 58Qw== X-Forwarded-Encrypted: i=1; AJvYcCVVzVAsB3BoLE/kRgulWBnGlHg5BTRZg12DqMG1q8DfBYqM9dU84kP9+SRcj2WhMXV+k9WNd7HzDb5p@freebsd.org X-Gm-Message-State: AOJu0YxkDH6Y2cyMoFpOc9ht96wWDpOWEiYG0nGs+CZTAVwPPj5EMgEm EEk96Zr802BW08K1AAfVSSZpWzkbjy5V824/nD8VxV4RPznb/EBLvcIWs9TWVIQGkVjYp5ht7xv 5j3i+1feqmrFRap4afcmLTZgjnc9NTmfo X-Google-Smtp-Source: AGHT+IEnIcSPubevpinMljBMEABEdK/DlDR6MrkT6zH+EeOBoO47lyPFlhdcB1BC81RgsmlyiTjSI6lT3Xy8Q58rHkM= X-Received: by 2002:a05:6902:1791:b0:e1a:aa90:34a with SMTP id 3f1490d57ef6-e1d3487f13dmr3574888276.19.1725638196528; Fri, 06 Sep 2024 08:56:36 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> In-Reply-To: <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> From: Freddie Cash Date: Fri, 6 Sep 2024 08:56:25 -0700 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: mike tancsa Cc: Chris Ross , FreeBSD Filesystems Content-Type: multipart/alternative; boundary="000000000000b70c9506217573fa" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[freebsd]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4X0gnx39jsz4r08 --000000000000b70c9506217573fa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Sep 6, 2024 at 8:44=E2=80=AFAM mike tancsa wrote: > On 9/6/2024 10:24 AM, Chris Ross wrote: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external device fa= ult > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 > > > I would triple check to see what the devices are that are part of the > pool. I wish there was a way to tell zfs to only display one or the > other. So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to > diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are. I have some > controllers that will re-order the disks on every reboot. glabel status > and camcontrol devlist should help verify > You can't tell ZFS specifically to use one form of GEOM ID vs another, but you can tell the whole system which GEOM IDs to not use. Add the following to /boot/loader.conf: kern.geom.label.disk_ident.enable=3D"0" # Disable the auto-generated Disk IDs for disks kern.geom.label.gptid.enable=3D"0" # Disable the auto-generated GPT UUIDs f= or disks kern.geom.label.ufsid.enable=3D"0" # Disable the auto-generated UFS UUIDs f= or filesystems The first line will remove the diskid/DISK-* entries and show the device nodes (daX). The other two lines remove GPT and UFS UUIDs as well. All my ZFS systems have those entries in loader.conf, as I prefer to use GPT partition labels in my pools (gpt/label-name) where I list which specific JBOD chassis and drive bay the HD is located. That way, it doesn't matter if the device nodes are renumbered, as the labels don't change. Makes it much easier to find the specific drive to be replaced, whether in my home server with 6 drives or my backups servers at work with multiple JBODs and 92 drives. --=20 Freddie Cash fjwcash@gmail.com --000000000000b70c9506217573fa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, Sep 6, 2024 at 8:44=E2=80=AFAM mi= ke tancsa <mike@sentex.net> wr= ote:
=20 =20 =20
On 9/6/2024 10:24 AM, Chris Ross wrote:
	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external device faul=
t
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

I would triple check to see what the devices are that are part of the pool.=C2=A0 I wish there was a way to tell zfs to only display on= e or the other.=C2=A0 So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.=C2=A0 I have some controllers that will re-ord= er the disks on every reboot.=C2=A0 glabel status and camcontrol devlist should help verify

You can't tell ZFS = specifically to use one form of GEOM ID vs another, but you can tell the wh= ole system which GEOM IDs to not use.=C2=A0 Add the following to /boot/load= er.conf:

kern.geom.label.disk_ident.enable=3D"0"= ; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Disable the auto-generated Disk IDs = =C2=A0for disks
kern.geom.label.gptid.enable=3D"0" # Disable = the auto-generated GPT UUIDs for disks
kern.geom.label.ufsid.enable=3D&q= uot;0" # Disable the auto-generated UFS UUIDs for filesystems

The first line will remove the diskid/DISK-* entries and = show the device nodes (daX).=C2=A0 The other two lines remove GPT and UFS U= UIDs as well.

All my ZFS systems have those entrie= s in loader.conf, as I prefer to use GPT partition labels in my pools (gpt/= label-name) where I list which specific JBOD chassis and drive bay the HD i= s located.=C2=A0 That way, it doesn't matter if the device nodes are re= numbered, as the labels don't change.=C2=A0 Makes it much easier to fin= d the specific drive to be replaced, whether in my home server with 6 drive= s or my backups servers at work with multiple JBODs and 92 drives.

--
Freddie Cash
fjwcash@gmail.com
--000000000000b70c9506217573fa--