From owner-freebsd-bugs@freebsd.org Tue Nov 3 00:11:31 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8393B45F092 for ; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQ9Dz2ySJz4Drb for ; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 6567545EDBC; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 652AB45ED95 for ; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQ9Dz1yfLz4F0N for ; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2763E11B20 for ; Tue, 3 Nov 2020 00:11:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A30BVAL087432 for ; Tue, 3 Nov 2020 00:11:31 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0A30BVwf087431 for bugs@FreeBSD.org; Tue, 3 Nov 2020 00:11:31 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 250816] AWS EC2 ZFS cannot import its own export! Date: Tue, 03 Nov 2020 00:11:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: raj@gusw.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 00:11:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250816 --- Comment #2 from Gunther Schadow --- Testing on other FreeBSD EC2 AMIs User data to facilitate the test with less manual work: --------------------------------------------- #!/bin/sh echo >>/etc/rc.conf echo 'zfs_enable=3D"YES"' >>/etc/rc.,conf --------------------------------------------- Here is my test protocol login as ec2-user then cut and paste: --------------------------------------------- su sh uname -a mkdir zfstc truncate -s 100M zfstc/0 truncate -s 100M zfstc/1 mkdir zfstd for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(base= name $i) ; done zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress=3De= nabled -O dedup=3Don -O compression=3Dlz4 testpool raidz $(for i in zfstd/* ; do r= eadlink $i ; done) zpool list zpool export testpool zpool import -d zfstd shutdown -p now --------------------------------------------- Now the test result: --------------------------------------------- # uname -a FreeBSD freebsd 11.4-RELEASE-p3 FreeBSD 11.4-RELEASE-p3 #0: Tue Sep 1 08:2= 2:33 UTC 2020 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERI= C=20 amd64 # mkdir zfstc # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # mkdir zfstd # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress= =3Denabled -O dedup=3Don -O compression=3Dlz4 testpool raidz $(for i in zfstd/* ; do r= eadlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 186K 176M - - 1% 0% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 488462546239790676 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: testpool ONLINE raidz1-0 ONLINE md0 ONLINE md1 ONLINE # # shutdown -p now Shutdown NOW! --------------------------------------------- Next, is it an issue with FreeBSD-12 in particular? First I will reproduce = the problem on a brand new 12.2 again, this time following the exact same proto= col that succeeded with 11.4 --------------------------------------------- ec2-user@freebsd:~ $ su root@freebsd:/home/ec2-user # sh root@freebsd:/home/ec2-user # uname -a FreeBSD freebsd 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC amd64 root@freebsd:/home/ec2-user # mkdir zfstc root@freebsd:/home/ec2-user # truncate -s 100M zfstc/0 root@freebsd:/home/ec2-user # truncate -s 100M zfstc/1 root@freebsd:/home/ec2-user # mkdir zfstd root@freebsd:/home/ec2-user # for i in zfstc/* ; do ln -s /dev/$(mdconfig -= a -t vnode -f $i) zfstd/$(basename $i) ; done root@freebsd:/home/ec2-user # root@freebsd:/home/ec2-user # zpool create -o feature@embedded_data=3Denabl= ed -o feature@lz4_compress=3Denabled -O dedup=3Don -O compression=3Dlz4 testpool = raidz $(for i in zfstd/* ; do readlink $i ; done) root@freebsd:/home/ec2-user # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 167K 176M - - 1% 0% 1.00x ONLI= NE=20 - root@freebsd:/home/ec2-user # zpool export testpool root@freebsd:/home/ec2-user # zpool import -d zfstd pool: testpool id: 7726589044207947012 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 17541269597952794361 UNAVAIL corrupted data 14116473632156932352 UNAVAIL corrupted data root@freebsd:/home/ec2-user # root@freebsd:/home/ec2-user # shutdown -p now Shutdown NOW! --------------------------------------------- Let's do the 12.2/ARM, perhaps something was screwed up only on the 12.2 am= d64 build --------------------------------------------- ec2-user@freebsd:~ $ su root@freebsd:/home/ec2-user # sh root@freebsd:/home/ec2-user # uname -a FreeBSD freebsd 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC arm64 root@freebsd:/home/ec2-user # mkdir zfstc root@freebsd:/home/ec2-user # truncate -s 100M zfstc/0 root@freebsd:/home/ec2-user # truncate -s 100M zfstc/1 root@freebsd:/home/ec2-user # mkdir zfstd root@freebsd:/home/ec2-user # for i in zfstc/* ; do ln -s /dev/$(mdconfig -= a -t vnode -f $i) zfstd/$(basename $i) ; done root@freebsd:/home/ec2-user # root@freebsd:/home/ec2-user # zpool create -o feature@embedded_data=3Denabl= ed -o feature@lz4_compress=3Denabled -O dedup=3Don -O compression=3Dlz4 testpool = raidz $(for i in zfstd/* ; do readlink $i ; done) root@freebsd:/home/ec2-user # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 167K 176M - - 1% 0% 1.00x ONLI= NE=20 - root@freebsd:/home/ec2-user # zpool export testpool root@freebsd:/home/ec2-user # zpool import -d zfstd pool: testpool id: 4979253895326493489 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 3861186839400824362 UNAVAIL corrupted data 14304100812262636401 UNAVAIL corrupted data root@freebsd:/home/ec2-user # root@freebsd:/home/ec2-user # shutdown -p now Shutdown NOW! --------------------------------------------- So it's obviously 12.2 specifically. Unfortunately there isn't an official = 12.1 that I could try, but a community 12.1-RELEASE AMI --------------------------------------------- $ su root@freebsd:/usr/home/ec2-user # sh # uname -a FreeBSD freebsd 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC arm64 # mkdir zfstc # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # mkdir zfstd # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress= =3Denabled -O dedup=3Don -O compression=3Dlz4 testpool raidz $(for i in zfstd/* ; do r= eadlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 165K 176M - - 1% 0% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 13451961690108720630 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-5E config: testpool UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 7947115053877644123 UNAVAIL corrupted data 12584889417144834990 UNAVAIL corrupted data # # shutdown -p now Shutdown NOW! --------------------------------------------- Now this is getting interesting! Shall we try 12.0-RELEASE now? Yes! Here is one that says "12.0-RELEASE amd64 ZFS", in this case I won't even add the u= ser data to set zfs_enable=3DYES in rc.conf.=20 --------------------------------------------- $ su root@freebsd:/usr/home/ec2-user # sh # uname -a FreeBSD freebsd 12.0-RELEASE-p13 FreeBSD 12.0-RELEASE-p13 GENERIC amd64 # mkdir zfstc # truncate -s 100M zfstc/0 # truncate -s 100M zfstc/1 # mkdir zfstd # for i in zfstc/* ; do ln -s /dev/$(mdconfig -a -t vnode -f $i) zfstd/$(basename $i) ; done # # zpool create -o feature@embedded_data=3Denabled -o feature@lz4_compress= =3Denabled -O dedup=3Don -O compression=3Dlz4 testpool raidz $(for i in zfstd/* ; do r= eadlink $i ; done) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT testpool 176M 624K 175M - - 1% 0% 1.00x ONLI= NE=20 - zroot 9.50G 2.01G 7.49G - - 2% 21% 1.00x ONLI= NE=20 - # zpool export testpool # zpool import -d zfstd pool: testpool id: 15013624344781576480 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: testpool ONLINE raidz1-0 ONLINE md0 ONLINE md1 ONLINE # # shutdown -p now Shutdown NOW! --------------------------------------------- Aaaaand... it's a winner! Maybe I should look for the word ZFS in the title? What is the significance of whether or not ZFS is mentioned? --=20 You are receiving this mail because: You are the assignee for the bug.=