From nobody Thu Apr 18 13:37:43 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VKzNz4Bjdz5GsyG for ; Thu, 18 Apr 2024 13:37:55 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VKzNy59M2z4Kv5 for ; Thu, 18 Apr 2024 13:37:54 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 43IDbmBP006465 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL) for ; Thu, 18 Apr 2024 09:37:48 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:48c2:ffd8:e147:2c2e] ([IPv6:2607:f3e0:0:4:48c2:ffd8:e147:2c2e]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 43IDbhkM095043 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Thu, 18 Apr 2024 09:37:47 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <82761f2b-475f-43ab-b045-95c60c330db7@sentex.net> Date: Thu, 18 Apr 2024 09:37:43 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: zfs boot error on mirrored drive Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.25 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.86)[-0.860]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[mike]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[sentex.net]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4VKzNy59M2z4Kv5 I have a strange bootup issue on a test box that I just updated from 11, to 12 and now RELENG13. Its a simple mirror. But if I boot up with ada0, I get an error and cannot boot further. Where as if I boot up with ada1, I get some errors and then proceeds to boot normally. For one thing, I dont get where zbackup1 is coming from. The pool I boot from is zroot ?!? Where would that reference be stored ? I think ada0 was part of another pool long ago (zbackup1). Here is the error when booting from disk 0 BIOS drive C: is disk0 BIOS drive D: is disk1 zio_read error: 5 zio_read error: 5 ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zbackup1 ZFS: can't find pool by guid Can't find /boot/zfsloader Can't find /boot/loader Can't find /boot/kernel/kernel FreeBSD/x86 boot Default: /boot/kernel/kernel boot: Here are the messages when booting from disk 1 Loading /boot/d| disk1 zio_read error: 5 zio_read error: 5 ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zbackup1 BIOS 615kB/3363840kB available memory FreeBSD/x86 bootstrap loader, Revision 1.1 and it continues on from there and boots After I did the zpool upgrade zroot, I did gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1   pool: zroot  state: ONLINE   scan: resilvered 195G in 01:04:31 with 0 errors on Mon May 25 16:08:21 2020 config:         NAME        STATE     READ WRITE CKSUM         zroot       ONLINE       0     0     0           mirror-0  ONLINE       0     0     0             ada0p3  ONLINE       0     0     0             ada1p3  ONLINE       0     0     0 errors: No known data errors Whats odd is that zbackup1 is not the name of the pool.  Where is that coming from ?  The pool I boot from is zroot on this box Loading /boot/defaults/loader.conf Loading /boot/loader.conf.local Loading /boot/d| disk1 zio_read error: 5 zio_read error: 5 ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zbackup1 BIOS 615kB/3363840kB available memory 1(nanobsd2)# gpart list ada0 Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 3907029127 first: 40 entries: 128 scheme: GPT Providers: 1. Name: ada0p1    Mediasize: 524288 (512K)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 20480    Mode: r0w0e0    efimedia: HD(1,GPT,55d32563-9eba-11ea-87ad-0007430611a8,0x28,0x400)    rawuuid: 55d32563-9eba-11ea-87ad-0007430611a8    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f    label: (null)    length: 524288    offset: 20480    type: freebsd-boot    index: 1    end: 1063    start: 40 2. Name: ada0p2    Mediasize: 21474836480 (20G)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 1048576    Mode: r1w1e0    efimedia: HD(2,GPT,55d3938d-9eba-11ea-87ad-0007430611a8,0x800,0x2800000)    rawuuid: 55d3938d-9eba-11ea-87ad-0007430611a8    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b    label: (null)    length: 21474836480    offset: 1048576    type: freebsd-swap    index: 2    end: 41945087    start: 2048 3. Name: ada0p3    Mediasize: 1978922958848 (1.8T)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 21475885056    Mode: r1w1e1    efimedia: HD(3,GPT,563d7775-9eba-11ea-87ad-0007430611a8,0x2800800,0xe6608000)    rawuuid: 563d7775-9eba-11ea-87ad-0007430611a8    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b    label: (null)    length: 1978922958848    offset: 21475885056    type: freebsd-zfs    index: 3    end: 3907028991    start: 41945088 Consumers: 1. Name: ada0    Mediasize: 2000398934016 (1.8T)    Sectorsize: 512    Mode: r2w2e3 0(nanobsd2)# gpart list ada1 Geom name: ada1 modified: false state: OK fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada1p1    Mediasize: 524288 (512K)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 20480    Mode: r0w0e0    efimedia: HD(1,GPT,25a0d50e-4207-11e6-8ebd-003048d6ef12,0x28,0x400)    rawuuid: 25a0d50e-4207-11e6-8ebd-003048d6ef12    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f    label: gptboot1    length: 524288    offset: 20480    type: freebsd-boot    index: 1    end: 1063    start: 40 2. Name: ada1p2    Mediasize: 21474836480 (20G)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 1048576    Mode: r1w1e0    efimedia: HD(2,GPT,25b0e143-4207-11e6-8ebd-003048d6ef12,0x800,0x2800000)    rawuuid: 25b0e143-4207-11e6-8ebd-003048d6ef12    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b    label: swap1    length: 21474836480    offset: 1048576    type: freebsd-swap    index: 2    end: 41945087    start: 2048 3. Name: ada1p3    Mediasize: 1978922958848 (1.8T)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 21475885056    Mode: r1w1e1    efimedia: HD(3,GPT,25c584a8-4207-11e6-8ebd-003048d6ef12,0x2800800,0xe6608000)    rawuuid: 25c584a8-4207-11e6-8ebd-003048d6ef12    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b    label: zfs1    length: 1978922958848    offset: 21475885056    type: freebsd-zfs    index: 3    end: 3907028991    start: 41945088 Consumers: 1. Name: ada1    Mediasize: 2000398934016 (1.8T)    Sectorsize: 512    Mode: r2w2e3 It looks like if I do just a 0(nanobsd2)# zpool import    pool: zbackup1      id: 5819795054093255811   state: UNAVAIL status: The pool was last accessed by another system.  action: The pool cannot be imported due to damaged devices or data.    see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY  config:         zbackup1                  UNAVAIL  insufficient replicas           raidz2-2                UNAVAIL  insufficient replicas             da5                   UNAVAIL  cannot open             mfisyspd8             UNAVAIL  cannot open             mfisyspd7             UNAVAIL  cannot open             mfisyspd6             UNAVAIL  cannot open             ada0                  UNAVAIL  cannot open             17460231907795801757  UNAVAIL  cannot open 0(nanobsd2)# it thinks ada0 is/was part of an old pool.  How do I get rid of that reference ?     ---Mike