From owner-freebsd-stable@freebsd.org Thu Nov 5 19:30:07 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02DDAA26DF9 for ; Thu, 5 Nov 2015 19:30:07 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 66700115D for ; Thu, 5 Nov 2015 19:30:05 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (bsdrookie.norma.com [192.168.7.144]) by elf.hq.norma.perm.ru (8.14.9/8.14.9) with ESMTP id tA5JTxbm048727 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Fri, 6 Nov 2015 00:30:00 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: freebsd-stable@freebsd.org From: "Eugene M. Zheganin" Subject: unable to boot a healthy zfs pool: all block copies unavailable X-Enigmail-Draft-Status: N1110 Message-ID: <563BAE37.2090205@norma.perm.ru> Date: Fri, 6 Nov 2015 00:29:59 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.hq.norma.perm.ru [192.168.3.10]); Fri, 06 Nov 2015 00:30:00 +0500 (YEKT) X-Spam-Status: No hits=-102.9 bayes=0.0000 testhits ALL_TRUSTED=-1, AWL=0.007, BAYES_00=-1.9, USER_IN_WHITELIST=-100 autolearn=ham autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on elf.hq.norma.perm.ru X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 19:30:07 -0000 Hi. Today one of my zfs pool disks dies, I was unable to change it on the fly (video board was blocking it) so I powered off, changed disk (not in root pool) and all of a sudden I realized that i cannot boot: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zroot gptzfsboot: failed to mount default pool zroot It was first reboot since October, 16th, when I installed recent -STABLE and upgraded zpool. I was pretty confident that I've installed loaders, but I tried to reinstall them - no luck. Then I built today's STABLE and installed loaders from it - same issue. I've even tried to install less recent loaders from a server nearby - same issue. Two years ago I have encountered similar (if not identical) issue: https://lists.freebsd.org/pipermail/freebsd-stable/2013-December/076317.html The main difference was it was i386. Now I have an amd64 machine. I even updated it's BIOS, and I still cannot boot. Zpool is fine: I'm writing this message from this exact machine, however, I had to boot it from today -STABLE from an USB stick. I've read about the zfsboottest utility and tried it on my unbootable pool - after the bried info about it (traated as healthy) it said "OK". I guess no errors were encountered. So... what can I do to restore the ability to boot from my root pool ? Thanks. Eugene. P.S. Some info about the pool below. let me know if it's not enough - I'll post more. [root@bsdrookie:/]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 1,79T 690G 1,12T - 14% 37% 1.00x ONLINE - [root@bsdrookie:/]# zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 704G 1,05T 2,07G legacy zroot/crypted 516G 1,06T 502G - zroot/jails 2,16G 1,05T 2,03G /usr/local/public/jails zroot/tmp 223M 1,05T 223M /tmp zroot/usr 182G 1,05T 17,0G /usr zroot/usr/home 145G 1,05T 145G /usr/home zroot/usr/ports 17,0G 1,05T 2,79G /usr/ports zroot/usr/ports/distfiles 14,3G 1,05T 14,3G /usr/ports/distfiles zroot/usr/ports/packages 384K 1,05T 384K /usr/ports/packages zroot/usr/public 2,11G 1,05T 2,11G /usr/local/public zroot/usr/src 1,56G 1,05T 1,56G /usr/src zroot/var 1,19G 1,05T 83,7M /var zroot/var/crash 992M 1,05T 992M /var/crash zroot/var/db 113M 1,05T 39,7M /var/db zroot/var/db/pkg 73,1M 1,05T 73,1M /var/db/pkg zroot/var/empty 144K 1,05T 144K /var/empty zroot/var/log 2,53M 1,05T 2,53M /var/log zroot/var/mail 272K 1,05T 272K /var/mail zroot/var/run 520K 1,05T 520K /var/run zroot/var/tmp 22,7M 1,05T 22,7M /var/tmp