From owner-freebsd-current@freebsd.org Mon Mar 19 23:00:09 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36DD4F518A0 for ; Mon, 19 Mar 2018 23:00:09 +0000 (UTC) (envelope-from kiri@kx.openedu.org) Received: from kx.openedu.org (flets-sg1027.kamome.or.jp [202.216.24.27]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84E606AA7A for ; Mon, 19 Mar 2018 23:00:07 +0000 (UTC) (envelope-from kiri@kx.openedu.org) Received: from kx.openedu.org (kx.openedu.org [202.216.24.27]) by kx.openedu.org (8.14.5/8.14.5) with ESMTP id w2JN04fx007127; Tue, 20 Mar 2018 08:00:05 +0900 (JST) (envelope-from kiri@kx.openedu.org) Message-Id: <201803192300.w2JN04fx007127@kx.openedu.org> Date: Tue, 20 Mar 2018 08:00:04 +0900 From: KIRIYAMA Kazuhiko To: freebsd-current@freebsd.org Cc: kiri@kx.openedu.org Subject: ZFS i/o error in recent 12.0 User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.6 MULE XEmacs/21.4 (patch 22) (Instant Classic) (amd64--freebsd) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Mar 2018 23:00:09 -0000 Hi, I've been encountered suddenly death in ZFS full volume machine(r330434) about 10 days after installation[1]: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zroot gptzfsboot: failed to mount default pool zroot FreeBSD/x86 boot ZFS: i/o error - all block copies unavailable ZFS: can't find dataset u Default: zroot/<0x0>: boot: Partition is bellow: # gpart show /dev/mfid0 => 40 31247564720 mfid0 GPT (15T) 40 409600 1 efi (200M) 409640 1024 2 freebsd-boot (512K) 410664 984 - free - (492K) 411648 268435456 3 freebsd-swap (128G) 268847104 30978715648 4 freebsd-zfs (14T) 31247562752 2008 - free - (1.0M) # But nothing had beed happend in old current ZFS full volume machine(r327038M). According to [2] the reason is boot/zfs/zpool.cache inconsistent. I've tried to cope with this by repairing /boot [3] from rescue bootable USB as follows: # kldload zfs # zpool import pool: zroot id: 17762298124265859537 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: zroot ONLINE mfid0p4 ONLINE # zpool import -fR /mnt zroot # df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 14G 1.6G 11G 13% / devfs 1.0K 1.0K 0B 100% /dev zroot/.dake 14T 18M 14T 0% /mnt/.dake zroot/ds 14T 96K 14T 0% /mnt/ds zroot/ds/backup 14T 88K 14T 0% /mnt/ds/backup zroot/ds/backup/kazu.pis 14T 31G 14T 0% /mnt/ds/backup/kazu.pis zroot/ds/distfiles 14T 7.9M 14T 0% /mnt/ds/distfiles zroot/ds/obj 14T 10G 14T 0% /mnt/ds/obj zroot/ds/packages 14T 4.0M 14T 0% /mnt/ds/packages zroot/ds/ports 14T 1.3G 14T 0% /mnt/ds/ports zroot/ds/src 14T 2.6G 14T 0% /mnt/ds/src zroot/tmp 14T 88K 14T 0% /mnt/tmp zroot/usr/home 14T 136K 14T 0% /mnt/usr/home zroot/usr/local 14T 10M 14T 0% /mnt/usr/local zroot/var/audit 14T 88K 14T 0% /mnt/var/audit zroot/var/crash 14T 88K 14T 0% /mnt/var/crash zroot/var/log 14T 388K 14T 0% /mnt/var/log zroot/var/mail 14T 92K 14T 0% /mnt/var/mail zroot/var/ports 14T 11M 14T 0% /mnt/var/ports zroot/var/tmp 14T 6.0M 14T 0% /mnt/var/tmp zroot/vm 14T 2.8G 14T 0% /mnt/vm zroot/vm/tbedfc 14T 1.6G 14T 0% /mnt/vm/tbedfc zroot 14T 88K 14T 0% /mnt/zroot # zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 51.1G 13.9T 88K /mnt/zroot zroot/.dake 18.3M 13.9T 18.3M /mnt/.dake zroot/ROOT 1.71G 13.9T 88K none zroot/ROOT/default 1.71G 13.9T 1.71G /mnt/mnt zroot/ds 45.0G 13.9T 96K /mnt/ds zroot/ds/backup 30.8G 13.9T 88K /mnt/ds/backup zroot/ds/backup/kazu.pis 30.8G 13.9T 30.8G /mnt/ds/backup/kazu.pis zroot/ds/distfiles 7.88M 13.9T 7.88M /mnt/ds/distfiles zroot/ds/obj 10.4G 13.9T 10.4G /mnt/ds/obj zroot/ds/packages 4.02M 13.9T 4.02M /mnt/ds/packages zroot/ds/ports 1.26G 13.9T 1.26G /mnt/ds/ports zroot/ds/src 2.56G 13.9T 2.56G /mnt/ds/src zroot/tmp 88K 13.9T 88K /mnt/tmp zroot/usr 10.4M 13.9T 88K /mnt/usr zroot/usr/home 136K 13.9T 136K /mnt/usr/home zroot/usr/local 10.2M 13.9T 10.2M /mnt/usr/local zroot/var 17.4M 13.9T 88K /mnt/var zroot/var/audit 88K 13.9T 88K /mnt/var/audit zroot/var/crash 88K 13.9T 88K /mnt/var/crash zroot/var/log 388K 13.9T 388K /mnt/var/log zroot/var/mail 92K 13.9T 92K /mnt/var/mail zroot/var/ports 10.7M 13.9T 10.7M /mnt/var/ports zroot/var/tmp 5.98M 13.9T 5.98M /mnt/var/tmp zroot/vm 4.33G 13.9T 2.75G /mnt/vm zroot/vm/tbedfc 1.58G 13.9T 1.58G /mnt/vm/tbedfc # zfs mount zroot/ROOT/default # cd /mnt/mnt/ # mv boot boot.bak # cp -RPp boot.bak boot # gpart show /dev/mfid0 => 40 31247564720 mfid0 GPT (15T) 40 409600 1 efi (200M) 409640 1024 2 freebsd-boot (512K) 410664 984 - free - (492K) 411648 268435456 3 freebsd-swap (128G) 268847104 30978715648 4 freebsd-zfs (14T) 31247562752 2008 - free - (1.0M) # gpart bootcode -b /mnt/mnt/boot/pmbr -p /boot/gptzfsboot -i 2 mfid0 partcode written to mfid0p2 bootcode written to mfid0 # cd # zpool export zroot # But can not boot: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zroot gptzfsboot: failed to mount default pool zroot FreeBSD/x86 boot Any idea? Best regards [1] http://ds.truefc.org/~kiri/freebsd/current/zfs/messages [2] https://lists.freebsd.org/pipermail/freebsd-questions/2016-February/270505.html [3] https://forums.freebsd.org/threads/zfs-i-o-error-all-block-copies-unavailable-invalid-format.55227/#post-312830 --- KIRIYAMA Kazuhiko