From owner-freebsd-questions@freebsd.org Sat Feb 29 10:46:47 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0DEF25B507 for ; Sat, 29 Feb 2020 10:46:47 +0000 (UTC) (envelope-from trond.endrestol@ximalas.info) Received: from enterprise.ximalas.info (enterprise.ximalas.info [IPv6:2001:700:1100:1::8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "ximalas.info", Issuer "Hostmaster ximalas.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48V34P4trgz410c for ; Sat, 29 Feb 2020 10:46:45 +0000 (UTC) (envelope-from trond.endrestol@ximalas.info) Received: from enterprise.ximalas.info (Ximalas@localhost [127.0.0.1]) by enterprise.ximalas.info (8.15.2/8.15.2) with ESMTPS id 01TAkTnN065355 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 29 Feb 2020 11:46:30 +0100 (CET) (envelope-from trond.endrestol@ximalas.info) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ximalas.info; s=default; t=1582973192; bh=MYeEE+v703VluvqTSPqdHkkwQpabo4o28Eg+OK9OObw=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=MhXeTYmOLGFcLnXAeMekbwEvprQ/IA3IDrVRrSqca81p3wVEQVxayVljKWUzxG0lX 9z/rQZ4XLG/dtvnBnqWRDyjJdW+1j+SpY24YkzRve1BBye3T7eVnX1fBWbsPh6tEhp nQRYU7ZKFeeuOdJ69h2o7S1rjNTnFHZwF9RYR4VfwbUjRT0l3FXFnvY3gIze181bX/ +8utpsB08ik34f7TwvWIPL02LBb6tj7TRy23AqzHqTEjZlUQEQEM3YkMC9On84nG8Z SoSSGZRRbGHqGQ46K3PcrtdGuL791dKwdBpDDqeqNWmSu17tF0UskawLdwiuyfct8n rzVyhwD2cP54Q== Received: from localhost (trond@localhost) by enterprise.ximalas.info (8.15.2/8.15.2/Submit) with ESMTP id 01TAkTPi065352; Sat, 29 Feb 2020 11:46:29 +0100 (CET) (envelope-from trond.endrestol@ximalas.info) X-Authentication-Warning: enterprise.ximalas.info: trond owned process doing -bs Date: Sat, 29 Feb 2020 11:46:29 +0100 (CET) From: =?UTF-8?Q?Trond_Endrest=C3=B8l?= Sender: Trond.Endrestol@ximalas.info To: byrnejb@harte-lyne.ca cc: freebsd-questions@freebsd.org Subject: Re: ZFS i/o error on boot unable to start system In-Reply-To: Message-ID: References: User-Agent: Alpine 2.22 (BSF 395 2020-01-19) OpenPGP: url=http://ximalas.info/about/tronds-openpgp-public-key MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-1.2 required=5.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on enterprise.ximalas.info X-Rspamd-Queue-Id: 48V34P4trgz410c X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ximalas.info header.s=default header.b=MhXeTYmO; dmarc=pass (policy=none) header.from=ximalas.info; spf=pass (mx1.freebsd.org: domain of trond.endrestol@ximalas.info designates 2001:700:1100:1::8 as permitted sender) smtp.mailfrom=trond.endrestol@ximalas.info X-Spamd-Result: default: False [-5.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ximalas.info:s=default]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ximalas.info:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[ximalas.info,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:224, ipnet:2001:700::/32, country:NO]; IP_SCORE(-2.07)[ip: (-8.18), ipnet: 2001:700::/32(-1.40), asn: 224(-0.76), country: NO(-0.01)] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Feb 2020 10:46:48 -0000 On Fri, 28 Feb 2020 08:51-0500, James B. Byrne via freebsd-questions wrote: > I have reported this on the forums as well. > > FreeBSD-12.1p2 > raidz2 on 4x8TB HDD (reds) > root on zfs > > We did a hot restart of this host this morning and received the following on > the console: > > ZFS: i/o error - all block copies unavailable > ZFS: failed to read pool zroot directory object > qptzfsboot: failed to mount default pool zroot > > FreeBSD/x86 boot > ZFS: i/o error - all block copies unavailable > ZFS: can't fild dataset 0 > Default: zroot/<0x0> > boot: > > What has happened? How do I get this system back up and online? Inspect the hardware. Make sure all cables are connected. Maybe you have a bad disk cable. Boot off the install media and ensure all drives are visible in the dmesg. Next, use gpart show -p to ensure all the GPTs are valid and all the partitions are properly listed. If all checks out, try to import the pool. You might need to forcefully import the pool as it's already marked as being in use. Do not attempt to rewind the pool to the last checkpoint. If the import is successful, start a scrub, and run zpool status -v multiple times to inspect the results as they come in. Last year, ada0 in one of my systems went so bad that the primary GPT got corrupted, and the boot partition and the swap partition near the beginning of the drive was unavailable. I booted off ada1, which was a mirror of ada0. Luckily, ZFS stores enough metadata to know where the ZFS partition resided on ada0, and thus, ZFS was able to continue as if nothing bad had happened, and this was confirmed by running a scrub. I opted to replace all fives drives in this system as they had been in continuous use for a little over seven years. > My first thought is that in modifying rc.conf to change some ip4 address > assignments that I may have done something else inadvertently which has caused > this. I cannot think of any other changes made since the system was last > restarted a noon yesterday. The mere act of editing and saving a file might have had an adverse impact on ZFS, but I suspect something else is at play. Let us know how things are progressing. -- Trond.