From owner-freebsd-bugs@freebsd.org Mon Nov 2 19:01:02 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7357B45825A for ; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQ2Lk2XHtz3cJj for ; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 54EA6458451; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 54B2845836F for ; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQ2Lk1fFrz3c6d for ; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1BAE4DF3B for ; Mon, 2 Nov 2020 19:01:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A2J11JD004532 for ; Mon, 2 Nov 2020 19:01:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0A2J11wM004529 for bugs@FreeBSD.org; Mon, 2 Nov 2020 19:01:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 250816] AWS EC2 ZFS cannot import its own export! Date: Mon, 02 Nov 2020 19:01:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: raj@gusw.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2020 19:01:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250816 Bug ID: 250816 Summary: AWS EC2 ZFS cannot import its own export! Product: Base System Version: 12.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: raj@gusw.net On a fresh deployment of the most recent official FreeBSD-12.2 EC2 AMI on Amazon. No complicated configurations. Only one added line in rc.conf=20 zfs_enable=3D"YES" without which zfs wouldn't even work. The summary overview is this: 1. zpool create .... works and creates the pool shown with zpool list 2. zpool export ... without error 3. zpool import ... says that one or more devices are corrupt Here is a (ba)sh script, you can just run this yourself: You see in it repeated attempts changing the options and zfs device type, n= one of which makes any difference. Here is the log on another system where it all worked: # mkdir zfstc # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # mkdir zfstd # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress= =3Denabled -O dedup=3Don -O compression=3Dlz4 testpool raidz $(for i in zfstd/* ; do r= eadlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 186K 176M - - 1% 0% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 14400958070908437474 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: testpool ONLINE raidz1-0 ONLINE md10 ONLINE md11 ONLINE # # for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i ; done # rm zfstc/* # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create testpool raidz $(for i in zfstd/* ; do readlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 156K 176M - - 1% 0% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 7399105644867648490 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: testpool ONLINE raidz1-0 ONLINE md10 ONLINE md11 ONLINE # # for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i ; done # rm zfstc/* # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create testpool mirror $(for i in zfstd/* ; do readlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 80M 67.5K 79.9M - - 1% 0% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 18245765184438368558 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: testpool ONLINE mirror-0 ONLINE md10 ONLINE md11 ONLINE # # for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i ; done # rm -r zfstc zfstd Now here on the new system where it fails: [root@geli ~]# mkdir zfstc [root@geli ~]# truncate -s 100M zfstc/0 [root@geli ~]# truncate -s 100M zfstc/1 [root@geli ~]# mkdir zfstd [root@geli ~]# for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $= i) zfstd/$(basename $i) ; done [root@geli ~]# [root@geli ~]# zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress=3Denabled -O dedup=3Don -O compression=3Dlz4 testpool = raidz $(for i in zfstd/* ; do readlink $i ; done) [root@geli ~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 182K 176M - - 1% 0% 1.00x ONLI= NE=20 - [root@geli ~]# zpool export testpool [root@geli ~]# zpool import -d zfstd pool: testpool id: 3796165815934978103 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 7895035226656775877 UNAVAIL corrupted data 5600170865066624323 UNAVAIL corrupted data [root@geli ~]# [root@geli ~]# for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i= ; done [root@geli ~]# rm zfstc/* [root@geli ~]# truncate -s 100M zfstc/0 [root@geli ~]# truncate -s 100M zfstc/1 [root@geli ~]# for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $= i) zfstd/$(basename $i) ; done [root@geli ~]# [root@geli ~]# zpool create testpool raidz $(for i in zfstd/* ; do readlink= $i ; done) [root@geli ~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 146K 176M - - 1% 0% 1.00x ONLI= NE=20 - [root@geli ~]# zpool export testpool [root@geli ~]# zpool import -d zfstd pool: testpool id: 17325954959132513026 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 7580076550357571857 UNAVAIL corrupted data 9867268050600021997 UNAVAIL corrupted data [root@geli ~]# [root@geli ~]# [root@geli ~]# for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i= ; done [root@geli ~]# rm zfstc/* [root@geli ~]# truncate -s 100M zfstc/0 [root@geli ~]# truncate -s 100M zfstc/1 [root@geli ~]# for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $= i) zfstd/$(basename $i) ; done [root@geli ~]# [root@geli ~]# zpool create testpool mirror $(for i in zfstd/* ; do readlin= k $i ; done) [root@geli ~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 80M 73K 79.9M - - 3% 0% 1.00x ONLI= NE=20 - [root@geli ~]# zpool export testpool [root@geli ~]# zpool import -d zfstd pool: testpool id: 7703888355221758527 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas mirror-0 UNAVAIL insufficient replicas 23134336724506526 UNAVAIL corrupted data 16413307577104054419 UNAVAIL corrupted data [root@geli ~]# [root@geli ~]# for i in zfstd/* ; do mdconfig -d -u $(readlink $i) && rm $i= ; done [root@geli ~]# rm -r zfstc zfstd If you are wondering if there is anything wrong with the md vnode device, I= can assure you that there is not, since I produced the MD5 hash on the underlyi= ng chunk files and through the /dev/md?? device with the same result. If you are wondering whether it is the create or export that is faulty or t= he import, I have proof that it is the import that is faulty. Why? Because I discovered this problem when I moved such files from the other FreeBSD syst= em to the new one, and failed on the import like that. First thing was run md5 hash over the files to see if they were corrupted. But no. And same files w= ith same checksum could be imported again on the old system. --=20 You are receiving this mail because: You are the assignee for the bug.=