From nobody Fri Sep 6 15:43:38 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gW86wPVz5VMMg for ; Fri, 06 Sep 2024 15:43:48 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gW82Qg0z4nbb for ; Fri, 6 Sep 2024 15:43:48 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.18.1/8.18.1) with ESMTPS id 486FhcuB037239 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Fri, 6 Sep 2024 11:43:38 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:d98b:f91d:4232:8049] ([IPv6:2607:f3e0:0:4:d98b:f91d:4232:8049]) by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 486FhaYr048683 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 6 Sep 2024 11:43:37 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------AOmL5V0Y0p2bqj1SOXU1V6D0" Message-ID: <69093f39-f673-416e-aac9-59a80b936ab7@sentex.net> Date: Fri, 6 Sep 2024 11:43:38 -0400 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Unable to replace drive in raidz1 To: Chris Ross , FreeBSD Filesystems References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Content-Language: en-US From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; TAGGED_RCPT(0.00)[freebsd]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[sentex.net]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4X0gW82Qg0z4nbb This is a multi-part message in MIME format. --------------AOmL5V0Y0p2bqj1SOXU1V6D0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/6/2024 10:24 AM, Chris Ross wrote: > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external device fault > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 I would triple check to see what the devices are that are part of the pool.  I wish there was a way to tell zfs to only display one or the other.  So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.  I have some controllers that will re-order the disks on every reboot.  glabel status and camcontrol devlist should help verify     ---Mike --------------AOmL5V0Y0p2bqj1SOXU1V6D0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 9/6/2024 10:24 AM, Chris Ross wrote:
	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external device fault
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

I would triple check to see what the devices are that are part of the pool.  I wish there was a way to tell zfs to only display one or the other.  So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are.  I have some controllers that will re-order the disks on every reboot.  glabel status and camcontrol devlist should help verify


    ---Mike

--------------AOmL5V0Y0p2bqj1SOXU1V6D0--