Date: Mon, 19 Mar 2018 19:09:31 -0400 From: Allan Jude <allanjude@freebsd.org> To: KIRIYAMA Kazuhiko <kiri@kx.openedu.org>, freebsd-current@freebsd.org Subject: Re: ZFS i/o error in recent 12.0 Message-ID: <e6e855ae-dcf3-7cd0-8c78-cdc414ce4f01@freebsd.org> In-Reply-To: <201803192300.w2JN04fx007127@kx.openedu.org> References: <201803192300.w2JN04fx007127@kx.openedu.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --zenOVxIYLQXM9LPpOVLPmpwA7MS5LlRSN Content-Type: multipart/mixed; boundary="L9RDf21KmKJBz1ri2GYLixyraeFX3VorL"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: KIRIYAMA Kazuhiko <kiri@kx.openedu.org>, freebsd-current@freebsd.org Message-ID: <e6e855ae-dcf3-7cd0-8c78-cdc414ce4f01@freebsd.org> Subject: Re: ZFS i/o error in recent 12.0 References: <201803192300.w2JN04fx007127@kx.openedu.org> In-Reply-To: <201803192300.w2JN04fx007127@kx.openedu.org> --L9RDf21KmKJBz1ri2GYLixyraeFX3VorL Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2018-03-19 19:00, KIRIYAMA Kazuhiko wrote: > Hi, >=20 > I've been encountered suddenly death in ZFS full volume > machine(r330434) about 10 days after installation[1]: >=20 > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool zroot > gptzfsboot: failed to mount default pool zroot >=20 > FreeBSD/x86 boot > ZFS: i/o error - all block copies unavailable > ZFS: can't find dataset u > Default: zroot/<0x0>: > boot:=20 >=20 > Partition is bellow: >=20 > # gpart show /dev/mfid0 > =3D> 40 31247564720 mfid0 GPT (15T) > 40 409600 1 efi (200M) > 409640 1024 2 freebsd-boot (512K) > 410664 984 - free - (492K) > 411648 268435456 3 freebsd-swap (128G) > 268847104 30978715648 4 freebsd-zfs (14T) > 31247562752 2008 - free - (1.0M) >=20 > #=20 >=20 > But nothing had beed happend in old current ZFS full volume > machine(r327038M). According to [2] the reason is boot/zfs/zpool.cache > inconsistent. I've tried to cope with this by repairing > /boot [3] from rescue bootable USB as follows: >=20 > # kldload zfs > # zpool import=20 > pool: zroot > id: 17762298124265859537 > state: ONLINE > action: The pool can be imported using its name or numeric identifier.= > config: >=20 > zroot ONLINE > mfid0p4 ONLINE > # zpool import -fR /mnt zroot > # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/da0p2 14G 1.6G 11G 13% / > devfs 1.0K 1.0K 0B 100% /dev > zroot/.dake 14T 18M 14T 0% /mnt/.dake > zroot/ds 14T 96K 14T 0% /mnt/ds > zroot/ds/backup 14T 88K 14T 0% /mnt/ds/back= up > zroot/ds/backup/kazu.pis 14T 31G 14T 0% /mnt/ds/back= up/kazu.pis > zroot/ds/distfiles 14T 7.9M 14T 0% /mnt/ds/dist= files > zroot/ds/obj 14T 10G 14T 0% /mnt/ds/obj > zroot/ds/packages 14T 4.0M 14T 0% /mnt/ds/pack= ages > zroot/ds/ports 14T 1.3G 14T 0% /mnt/ds/port= s > zroot/ds/src 14T 2.6G 14T 0% /mnt/ds/src > zroot/tmp 14T 88K 14T 0% /mnt/tmp > zroot/usr/home 14T 136K 14T 0% /mnt/usr/hom= e > zroot/usr/local 14T 10M 14T 0% /mnt/usr/loc= al > zroot/var/audit 14T 88K 14T 0% /mnt/var/aud= it > zroot/var/crash 14T 88K 14T 0% /mnt/var/cra= sh > zroot/var/log 14T 388K 14T 0% /mnt/var/log= > zroot/var/mail 14T 92K 14T 0% /mnt/var/mai= l > zroot/var/ports 14T 11M 14T 0% /mnt/var/por= ts > zroot/var/tmp 14T 6.0M 14T 0% /mnt/var/tmp= > zroot/vm 14T 2.8G 14T 0% /mnt/vm > zroot/vm/tbedfc 14T 1.6G 14T 0% /mnt/vm/tbed= fc > zroot 14T 88K 14T 0% /mnt/zroot > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > zroot 51.1G 13.9T 88K /mnt/zroot > zroot/.dake 18.3M 13.9T 18.3M /mnt/.dake > zroot/ROOT 1.71G 13.9T 88K none > zroot/ROOT/default 1.71G 13.9T 1.71G /mnt/mnt > zroot/ds 45.0G 13.9T 96K /mnt/ds > zroot/ds/backup 30.8G 13.9T 88K /mnt/ds/backup > zroot/ds/backup/kazu.pis 30.8G 13.9T 30.8G /mnt/ds/backup/kazu.pis > zroot/ds/distfiles 7.88M 13.9T 7.88M /mnt/ds/distfiles > zroot/ds/obj 10.4G 13.9T 10.4G /mnt/ds/obj > zroot/ds/packages 4.02M 13.9T 4.02M /mnt/ds/packages > zroot/ds/ports 1.26G 13.9T 1.26G /mnt/ds/ports > zroot/ds/src 2.56G 13.9T 2.56G /mnt/ds/src > zroot/tmp 88K 13.9T 88K /mnt/tmp > zroot/usr 10.4M 13.9T 88K /mnt/usr > zroot/usr/home 136K 13.9T 136K /mnt/usr/home > zroot/usr/local 10.2M 13.9T 10.2M /mnt/usr/local > zroot/var 17.4M 13.9T 88K /mnt/var > zroot/var/audit 88K 13.9T 88K /mnt/var/audit > zroot/var/crash 88K 13.9T 88K /mnt/var/crash > zroot/var/log 388K 13.9T 388K /mnt/var/log > zroot/var/mail 92K 13.9T 92K /mnt/var/mail > zroot/var/ports 10.7M 13.9T 10.7M /mnt/var/ports > zroot/var/tmp 5.98M 13.9T 5.98M /mnt/var/tmp > zroot/vm 4.33G 13.9T 2.75G /mnt/vm > zroot/vm/tbedfc 1.58G 13.9T 1.58G /mnt/vm/tbedfc > # zfs mount zroot/ROOT/default > # cd /mnt/mnt/ > # mv boot boot.bak > # cp -RPp boot.bak boot > # gpart show /dev/mfid0 > =3D> 40 31247564720 mfid0 GPT (15T) > 40 409600 1 efi (200M) > 409640 1024 2 freebsd-boot (512K) > 410664 984 - free - (492K) > 411648 268435456 3 freebsd-swap (128G) > 268847104 30978715648 4 freebsd-zfs (14T) > 31247562752 2008 - free - (1.0M) >=20 > # gpart bootcode -b /mnt/mnt/boot/pmbr -p /boot/gptzfsboot -i 2 mfid0 > partcode written to mfid0p2 > bootcode written to mfid0 > # cd > # zpool export zroot > #=20 >=20 > But can not boot: >=20 > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool zroot > gptzfsboot: failed to mount default pool zroot >=20 > FreeBSD/x86 boot >=20 > Any idea? >=20 > Best regards >=20 > [1] http://ds.truefc.org/~kiri/freebsd/current/zfs/messages > [2] https://lists.freebsd.org/pipermail/freebsd-questions/2016-February= /270505.html > [3] https://forums.freebsd.org/threads/zfs-i-o-error-all-block-copies-u= navailable-invalid-format.55227/#post-312830 >=20 > --- > KIRIYAMA Kazuhiko > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" >=20 Since you were about to 'zpool import' on the USB stick, this suggests the problem is with the boot blocks, not ZFS. The early boot phase (gptzfsboot) does not read zpool.cache, since that only lives ON the pool, and the pool has not been imported yet. Maybe kevans@ or imp@ who have been working on the boot code have some insight. --=20 Allan Jude --L9RDf21KmKJBz1ri2GYLixyraeFX3VorL-- --zenOVxIYLQXM9LPpOVLPmpwA7MS5LlRSN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJasEMsAAoJEBmVNT4SmAt+dAAQAKgSdxlJ/Dg1bT79SmWUHDhJ D0Yfsqwd83n80cwEAzZF2ZbQGGSbrDWYYcbfgWJaaOnsh7Tbq5WopLQtigE0N8D+ uANF2ERkPM9/Lgy8acYUYrE5GnAxYQvxaPbxBIvoyxXDMpUlSobQAuhPoikvyCHb J7cww/gYaCbJDFi7GH8Hwrf1mdBVPeHmDjTDOEqhFo2aSFbg9So1krWcwtNTOLQ1 D8ISm54V4Lurw9UY2f84du9HHabvKEi/FoHeOU/J3dPJavzEAIS6+G7eZ2vhTdUC Y6VTXp2TWpYtQc6I3Zfj/pOY0dkWfLlfBLRNFVA8+YH1522saCmhXCyAJmtyD6fR IL2MV11TY/UT6aduAHQrqakrGbzst7WLt6V15vzPTRcvWVr5/f6vmE2SdfKfe6ht nr0DNS+kJuvAKBahxPj/N4tl2F+sZV429i2kkZQa44HcLUaywx7JiGRSjn2Gghcd zDiI6/ytFvY/pbofs3TaUbY+gmMlznESckrADundiQ9oTu+qzlDyKbEUzn6zOSyP RlXdjBZjSVpvTP+h8iiiokPpOtAz6RgTbvyrAYlDq0BROfu05M1LCmLmJ84w8HFt kohC5IrYfzCjMle0xCh1uZuXYi2spZEooKNTwaQ+b0sPu86dVn0cqIRjuJWKX0bn u5ykKcfS7tBJakXe8+Vv =OM4I -----END PGP SIGNATURE----- --zenOVxIYLQXM9LPpOVLPmpwA7MS5LlRSN--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e6e855ae-dcf3-7cd0-8c78-cdc414ce4f01>