Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 2 Mar 2020 09:58:48 -0600
From:      Karl Denninger <karl@denninger.net>
To:        freebsd-questions@freebsd.org
Subject:   Re: ZFS i/o error on boot unable to start system
Message-ID:  <60d118f3-7ffe-86af-d6e8-d785e9764cbc@denninger.net>
In-Reply-To: <a0f70632-5188-15a3-52b4-a57087d23013@sentex.net>
References:  <eb8f8f32fcf5559774daf3a772a1ad2e.squirrel@webmail.harte-lyne.ca> <a0f70632-5188-15a3-52b4-a57087d23013@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
On 3/2/2020 09:31, mike tancsa wrote:
> On 2/28/2020 8:51 AM, James B. Byrne via freebsd-questions wrote:
>> I have reported this on the forums as well.
>>
>> FreeBSD-12.1p2
>> raidz2 on 4x8TB HDD (reds)
>> root on zfs
>>
>> We did a hot restart of this host this morning and received the following on
>> the console:
>>
>> ZFS: i/o error - all block copies unavailable
>> ZFS: failed to read pool zroot directory object
>> qptzfsboot: failed to mount default pool zroot
>>
>> What has happened?  How do I get this system back up and online?
> Could be a number of things. e.g. you might have only installed the boot
> block on one disk and thats no longer in the boot order.  
Well, no.  The error is coming from the loader (otherwise there's no
ability to try to read ZFS at all.)  "gptzfsboot" implies legacy (not
EFI) booting.
> Your BIOS
> might have changed from legacy to EFI only and all of a sudden you
> cannot boot the disks as you dont have the right boot loader. (This
> happened to me once). 

Now THAT is possible and in fact sort of likely.

Let's say you have 4 disks, all of which are part of the pool in
question.  When you built the system all the disks got the loader.  Now
you upgrade the system through time and during one or more of these
cycles the pool gets new flag(s) which get turned on.  The upgrade also
includes the loader, but SOME of the disks don't get it updated.

When the system boots up until either the EFI loader or gptzfsboot
executes only ONE copy is used -- the one the BIOS decides to load,
which is under control of the BIOS' idea of what the boot order is. 
Further, if the pool is encrypted then the loader (gptzfsboot or the EFI
loader) has to know how to prompt for the key and read that too, and
which disks/partitions to try to unlock before attempting to taste them
(e.g. which have the "boot" flag turned on.)

Now, for whatever reason, the BIOS decides to read the disks in a
different order and it loads and starts the OLD loader, which was not
updated, off the disk it selects.  That loader can't read ANY of the
disks that have had the new flags set active, or it can't read an
encrypted disk, etc -- and thus you get the error.

> As others have said, try booting off a usb stick and see if the pool is
> OK and then take a look at the parts that are involved in the boot
> process as outlined by trond.endrestol@ximalas.info
Yes.  It is likely the issue is that the loader was updated on some but
not all disks but BE CAREFUL because writing that, if done wrong (e.g.
to the wrong partition, etc), can destroy the data on the drive.
-- 
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

[-- Attachment #2 --]
0	*H
010
	`He0	*H

00H^Ōc!5
H0
	*H
010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CA0
170817164217Z
270815164217Z0{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CA0"0
	*H
0
h-5B>[;olӴ0~͎O9}9Ye*$g!ukvʶLzN`jL>MD'7U45CB+kY`bd~b*c3Ny-78ju]9HeuέsӬDؽmgwER?&UURj'}9nWD i`XcbGz\gG=u%\Oi13ߝ4
K44pYQr]Ie/r0+eEޝݖ0C15Mݚ@JSZ(zȏNTa(25DD5.l<g[[ZarQQ%Buȴ~~`IohRbʳڟu2MS8EdFUClCMaѳ!}ș+2k/bųE,n当ꖛ\(8WV8	d]b	yXw	܊:I39
00U]^§Q\ӎ0U#0T039N0b010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CA	@Ui0U00U0
	*H
:P U!>vJnio-#ן]WyujǑR̀Q
nƇ!GѦFg\yLxgw=OPycehf[}ܷ['4ڝ\[p6\o.B&JF"ZC{;*o*mcCcLY߾`
t*S!񫶭(`]DHP5A~/NPp6=mhk밣'doA$86hm5ӚS@jެEgl
)0JG`%k35PaC?σ
׳HEt}!P㏏%*BxbQwaKG$6h¦Mve;[o-Iی&
I,Tcߎ#t wPA@l0P+KXBպT	zGv;NcI3&JĬUPNa?/%W6G۟N000k#Xd\=0
	*H
0{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CA0
170817212120Z
220816212120Z0W10	UUS10UFlorida10U
Cuda Systems LLC10Ukarl@denninger.net0"0
	*H
0
T[I-ΆϏdn;Å@שy.us~_ZG%<MYd\gvfnsa1'6Egyjs"C [{~_KPn+<*pv#Q+H/7[-vqDV^U>f%GX)H.|l`M(Cr>е͇6#odc"YljҦln8@5SA0&ۖ"OGj?UDWZ5	dDB7k-)9Izs-JAv
J6L$Ն1SmY.Lqw*SH;EF'DĦH]MOgQQ|Mٙג2Z9y@y]}6ٽeY9Y2xˆ$T=eCǺǵbn֛{j|@LLt1[Dk5:$=	`	M00<+00.0,+0 http://ocsp.cudasystems.net:88880	U00	`HB0U0U%0++03	`HB
&$OpenSSL Generated Client Certificate0U%՞V=؁;bzQ0U#0]^§Q\ӎϡ010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CAH^Ōc!5
H0U0karl@denninger.net0
	*H
۠A0-j%--$%g2#ޡ1^>{K+uGEv1ş7Af&b&O;.;A5*U)ND2bF|\=]<sˋL!wrw٧>YMÄ3\mWR hSv!_zvl? 3_ xU%\^#O*Gk̍YI_&Fꊛ@&1n”} ͬ:{hTP3B.;bU8:Z=^Gw8!k-@xE@i,+'Iᐚ:fhztX7/(hY` O.1}a`%RW^akǂpCAufgDixUTЩ/7}%=jnVZvcF<M=
2^GKH5魉
_O4ެByʈySkw=5@h.0z>
W1000{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0
	`HeE0	*H
	1	*H
0	*H
	1
200302155849Z0O	*H
	1B@x-)YaZрk&h,PLR
Xg"ɠ>Ֆz+Rq1[`^0l	*H
	1_0]0	`He*0	`He0
*H
0*H
0
*H
@0+0
*H
(0	+7100{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0*H
	10{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0
	*H
EAjf~NGaIv$/&0Vzrqۤ7E@;J:_ѲOqHk\}]عj^E|ҍx3Ox0!Ȭ߲d+ur~Ձk*,[JqwRN(Fǻ{y	0mrQ,ޒ4fFC*wc|/↉)ϏJK
뙾dČ3n	={pleI)Z!)oH;Ĵ3 WKyjJ*nI`dwLC&^:N0\Kg\C1qM8%XGܞi{Q-xnY*1S^+RDU^"rfC	 c
;]8L*
dCߔJҢ&C0{/x
tFN&T[KuǯGNUMijP+g@R=SlE!2

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?60d118f3-7ffe-86af-d6e8-d785e9764cbc>