Date: Sun, 7 Jan 2024 10:50:19 -0500 From: Karl Denninger <karl@denninger.net> To: stable@freebsd.org Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. Message-ID: <ef7a7af8-32cc-4ee3-a0fc-958c82dbaf5e@denninger.net> In-Reply-To: <cc136316-f285-41bd-8d59-c5adce06e277@quip.cz> References: <f97d80ee-0b01-4d68-beb5-53e905f0404c@FreeBSD.org> <e74464be-09b6-43e2-9365-7b0271b2d6eb@FreeBSD.org> <cc136316-f285-41bd-8d59-c5adce06e277@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms090201060502050605020208
Content-Type: multipart/alternative;
boundary="------------ipUfD8unXJhYrrBeDY0TfUUu"
--------------ipUfD8unXJhYrrBeDY0TfUUu
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
On 1/7/2024 10:38, Miroslav Lachman wrote:
> On 07/01/2024 15:49, Lev Serebryakov wrote:
>> On 05.01.2024 18:28, Lev Serebryakov wrote:
>>
>>> After that my server fails to boot, gtpzfsboot from second disk
>>> (ada1) reports several "zio_read error: 5" and
>>>
>>> ZFS: i/o error - all block copies unavailable
>>> ZFS: can't read MOS of pool zroot
>>>
>>> after that.
>> I've re-created pool from scratch
>>
>> zpool create znewroot ada0p3 && zfs send zroot | zfs receive
>> znewroot && zpool destroy zroot && zpool attach znewroot ada0p3 ada1p3
>>
>> but gptzfsboot still can not boot from it with same diagnostics :-(
>
> How large are your disks in a question?
>
> I was bitten by this not a long time ago when migrating my 2TB pool by
> zfs send to larger disks (4TB), then I see the error:
>
> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS of pool zroot
>
> .....
>
> It can also be avoided if your machine supports EFI boot, but my HP
> Microserver Gen 8 does not support it.
>
> Kind regards
> Miroslav Lachman
Yes, gptzfsboot (non-EFI) uses BIOS calls to read the headers and
ultimately the kernel; if the pool you wish boot from is not contained
entirely within the first 2TB of the drive (which can certainly happen
on a disk larger if you are not segregating the boot pool separate from
the rest of the space on a drive larger than that) it can fail to load
because some of the data it has to read is beyond the 32 bit block
offset that can be used.
If you have (for example) 4Tb drives the way to avoid this is to have
the boot pool be of sufficient size to hold the operating system and
then a second pool (that can be of any size) which holds all your other
data, with the boot pool comprised of vdevs that have all blocks with in
the first 2Tb.
--
Karl Denninger
karl@denninger.net
/The Market Ticker/
/[S/MIME encrypted email preferred]/
--------------ipUfD8unXJhYrrBeDY0TfUUu
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 1/7/2024 10:38, Miroslav Lachman
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:cc136316-f285-41bd-8d59-c5adce06e277@quip.cz">On
07/01/2024 15:49, Lev Serebryakov wrote:
<br>
<blockquote type="cite">On 05.01.2024 18:28, Lev Serebryakov
wrote:
<br>
<br>
<blockquote type="cite"> After that my server fails to boot,
gtpzfsboot from second disk (ada1) reports several "zio_read
error: 5" and
<br>
<br>
ZFS: i/o error - all block copies unavailable
<br>
ZFS: can't read MOS of pool zroot
<br>
<br>
after that.
<br>
</blockquote>
I've re-created pool from scratch
<br>
<br>
zpool create znewroot ada0p3 && zfs send zroot | zfs
receive znewroot && zpool destroy zroot && zpool
attach znewroot ada0p3 ada1p3
<br>
<br>
but gptzfsboot still can not boot from it with same
diagnostics :-(
<br>
</blockquote>
<br>
How large are your disks in a question?
<br>
<br>
I was bitten by this not a long time ago when migrating my 2TB
pool by zfs send to larger disks (4TB), then I see the error:
<br>
<br>
ZFS: i/o error - all block copies unavailable
<br>
ZFS: can't read MOS of pool zroot
<br>
<br>
.....<br>
<br>
It can also be avoided if your machine supports EFI boot, but my
HP Microserver Gen 8 does not support it.
<br>
<br>
Kind regards
<br>
Miroslav Lachman
<br>
</blockquote>
<p>Yes, gptzfsboot (non-EFI) uses BIOS calls to read the headers and
ultimately the kernel; if the pool you wish boot from is not
contained entirely within the first 2TB of the drive (which can
certainly happen on a disk larger if you are not segregating the
boot pool separate from the rest of the space on a drive larger
than that) it can fail to load because some of the data it has to
read is beyond the 32 bit block offset that can be used.<br>
</p>
<p>If you have (for example) 4Tb drives the way to avoid this is to
have the boot pool be of sufficient size to hold the operating
system and then a second pool (that can be of any size) which
holds all your other data, with the boot pool comprised of vdevs
that have all blocks with in the first 2Tb.</p>
<div class="moz-signature">-- <br>
Karl Denninger<br>
<a href="mailto:karl@denninger.net" class="moz-txt-link-freetext">karl@denninger.net</a><br>
<i>The Market Ticker</i><br>
<font size="-2"><i>[S/MIME encrypted email preferred]</i></font></div>
</body>
</html>
--------------ipUfD8unXJhYrrBeDY0TfUUu--
--------------ms090201060502050605020208
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
DbowggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL
MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw
FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf
BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4
MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD
dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1
ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI
KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD
0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY
vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn
uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24
SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E
6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH
YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL
h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd
zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE
FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q
EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ
TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5
c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY
MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC
AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN
gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9
oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj
tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K
uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv
HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK
17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/
Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA
6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY
UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBxIwggT6oAMCAQICEgLG8yH4PQFdbd9x
Ugmpzl1jXzANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlk
YTEZMBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENB
MSUwIwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBMB4XDTIyMDYyOTE2MTYz
NloXDTI3MDYyODE2MTYzNlowOjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEX
MBUGA1UEAwwOS2FybCBEZW5uaW5nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC
AQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvW
ZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B
3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTgy+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYI
XgVVPgfZZrbJJb5HWOQpvvhILpPCD3xsYJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMi
WapsatKm8mxuOOGOEBhAoTVTwUHlMNTg6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMb
NQm1mWREQhw3axgGLSntjjnznJr5vsvXSYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZM
qa20JLAF1YagutDiMRURU23iWS7bA9tMcXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5
CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1l
y+5ZOZbxBAZZMod4y4b4FiRUhRI97r9lCxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY
2BlA7ExM8XShMd9bRPZrNTokPQPUCWCgCdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEE
MDAuMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV
HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBD
bGllbnQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNV
HSMEgcIwgb+AFF3AXsKnjdPND5+bxVECGKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQ
MA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5
c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lz
dGVtcyBMTEMgMjAxNyBDQYITAORIioIQzl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJs
QGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAKquc7cu0xc8FNtAQauZvocDzWQj
7HG9YvMbWnMi+ckhiA3rdW5NwWg0HBhBho1YlnqV+ntCVE2L8ezohHWm+KAdfXgpraL86Vsn
3ywNlZu/3COMpo2ALuHln8YQtH3Y8ebvzKMdlf2b5WB+7mOFIxXIr4AnNOLKCkq5ZhzC6JW6
Jvw3P0csiGa3UrfatYID5NvPgkaQvEgimEjG3psZqwQTL2Wxohvw783PrDt3wS0XeNhvQ61g
3QJFZKuv+bmGH3YBSPo1t6NUGAr+JozX5lDihB8JGkBt/NwdYec49a08uL0BbPaAJ7NjuIPG
7Y0Ak7PXZT37yx/Zla9PzLMJFgbelOkaatdzbblMZPDEVZ27l4lGMmV83Lm3YP17sdAyS/Wp
mav7WmJUkQ9iuIKzSpdc82i9Mfujl1vbBtwtkHNPPtKuulIFM4ZwrPKjlVdLqTSqD8m9yHEi
Y0PuAooq63OpJWF6hvMaiIPBWEAVIaDW9uG0MshLl9DnHnMyrJTfuC33Z9mOGMz7dRBjJd5Y
W02xAzYnUuEBOpj+LQv5R8XIFMHFXktqEKvQrXeM2RU+PcZqKOBkTktxBLn3NI5VfA15Jk0c
5V5XcOqo3p2hvrwvXrinrb2pEREnoqmfrkXT3zOq5Y6ryRH8u734lGEF0dILXzoV4PM7XFit
oTePoEjmMYIFBDCCBQACAQEwgZEwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGEx
GTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTEl
MCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQQISAsbzIfg9AV1t33FSCanO
XWNfMA0GCWCGSAFlAwQCAwUAoIICQzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG
SIb3DQEJBTEPFw0yNDAxMDcxNTUwMjBaME8GCSqGSIb3DQEJBDFCBEBth1Cogkg3TZXHJ87G
f6umxdRP5GE5V7ibvFWApuf+FLWTq6Y7B3bOubQtZosQDiGzmtPkHLAq+2QunvReKcZXMGwG
CSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAO
BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw
gaIGCSsGAQQBgjcQBDGBlDCBkTB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTEZ
MBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENBMSUw
IwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBAhICxvMh+D0BXW3fcVIJqc5d
Y18wgaQGCyqGSIb3DQEJEAILMYGUoIGRMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9y
aWRhMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMg
Q0ExJTAjBgNVBAMMHEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0ECEgLG8yH4PQFdbd9x
Ugmpzl1jXzANBgkqhkiG9w0BAQEFAASCAgCnFWFkHY35wYwPoA8P28KTfCtF1Vz0q0FGPDjn
sdKnKyRjbJsreBn+Xs6ruKCZDiLNCa3V0RFfOwbxgakYnG0BGDT5wNlqyBY6ubDyvZTVmCzk
vJV5+ZfIioUGDcoPyJ0ymAScVi0zSsmWLt1fEQVXPEjawV1OMNofVfv6WJnKJ4IZT8+lm+Jz
yp4hit0+nL++seVJALoA3PbaG83nhEw6odKusKl6Tg+3O3ef2cVGv3iFk5VmTVAp3f7ZL26/
SAX6PTDTP92J7aG6nO4aI7MVdUpdRDjj22cHiAfmPdT0UNCeaasG55PDH1ss+Ai3jRnSMG1h
5QRSFwojLe8F+Kyzxsf5eGGGc/RqAIz5eKnvQp/Rzlv3DHqlJr+RRapwHT9mCzOSvYxiOt/V
28z9FynpEo6sp2a3Q7WmSYHQQfKnLxKael9GWvqjxuk+DLdOZTGOIu/S9CC2IS33AJ5UCWuH
KXCb7uRQHNmS/P/2Ticc4Ru2MawvSx6gGHOQ5N71fMixvvje6WU3/ND5pJSFCXZ7h915/H5W
9dVFyGcd2d+Hm4W0l634HJbR8uJ3rIyt7HrC/BC54yf/tVOeFObUng44wioUT9Y5nb0xJyJR
tFYJB19HA9FXx8d665q8doEfhU+omTjd04UjJvZP1Sr9SrqpJ3l5zvhMx1O4jVXYk5P1wgAA
AAAAAA==
--------------ms090201060502050605020208--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ef7a7af8-32cc-4ee3-a0fc-958c82dbaf5e>