From owner-freebsd-bugs@freebsd.org  Sat Jan 23 13:54:48 2016
Return-Path: <owner-freebsd-bugs@freebsd.org>
Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CE23A8DB16
 for <freebsd-bugs@mailman.ysv.freebsd.org>;
 Sat, 23 Jan 2016 13:54:48 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
 [IPv6:2001:1900:2254:206a::16:76])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 7CBA71DC5
 for <freebsd-bugs@FreeBSD.org>; Sat, 23 Jan 2016 13:54:48 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from bugs.freebsd.org ([127.0.1.118])
 by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NDsmPd028841
 for <freebsd-bugs@FreeBSD.org>; Sat, 23 Jan 2016 13:54:48 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
From: bugzilla-noreply@freebsd.org
To: freebsd-bugs@FreeBSD.org
Subject: [Bug 206521] Can't decrypt disks on ZFS+Geli installation after
 order of devices changed
Date: Sat, 23 Jan 2016 13:54:48 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: bin
X-Bugzilla-Version: 10.2-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: florian.ermisch@alumni.tu-berlin.de
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform
 op_sys bug_status bug_severity priority component assigned_to reporter
Message-ID: <bug-206521-8@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-bugs@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Bug reports <freebsd-bugs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-bugs>,
 <mailto:freebsd-bugs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-bugs/>
List-Post: <mailto:freebsd-bugs@freebsd.org>
List-Help: <mailto:freebsd-bugs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-bugs>,
 <mailto:freebsd-bugs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Jan 2016 13:54:48 -0000

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206521

            Bug ID: 206521
           Summary: Can't decrypt disks on ZFS+Geli installation after
                    order of devices changed
           Product: Base System
           Version: 10.2-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: bin
          Assignee: freebsd-bugs@FreeBSD.org
          Reporter: florian.ermisch@alumni.tu-berlin.de

(Rendered Markdown here: https://gist.github.com/0xf10e/ddefc6fad77d6b51672=
f)

## Preface

I added a 256GB mSATA SSD to my Lenovo x220
of which I planned to use a nice chunk as L2ARC
for the SATA HDD. Even without a SATA disk=20
present the SSD showed up as HDD2 in the BIOS.

Made a fresh installation of FreeBSD 10.2 amd64
with the memstick image and chose ZFS+Geli.=20
Booted, worked, everything just fine. I added a=20
SATA HDD which the BIOS listed as HDD0 just as=20
I expected from the previous SATA-only setup.
When I now tried to boot from the SSD I was=20
prompted for the GELI key of a different device
than before, `ada1p3` instead of `ada0p3`.
But my passphrase wasn't accepted. Never.
Re-installed, added HDD, same result.=20
Reinstalled with HDD present, the removed the=20
HDD, same result (then prompted for `ada0p3`=20
instead of `ada1p3`, of course).

When I restored the configuration which
was present during the installation process
(either remove the HDD or add it back in)=20
everything worked fine again.

## Using bhyve to reproduce

Created a VM, added more disks:
```
floh@fuchi-cyber220:~:1305% iohyve info
Name           Size  RAM  CPU  OS       Loader
fbsd102        -     2G   1    default  bhyveload
fbsd102/disk0  2G    2G   1    default  bhyveload
fbsd102/disk1  2G    2G   1    default  bhyveload
fbsd102/disk2  4G    2G   1    default  bhyveload
```

Install FreeBSD 10.2 (the one I used for my laptop):

  ```
  floh@fuchi-cyber220:~:1305% sudo iohyve install fbsd102
FreeBSD-10.2-RELEASE-amd64-disc1.iso
  Installing fbsd102...
  floh@fuchi-cyber220:~:1307% sudo iohyve console fbsd102
  Starting console on fbsd102...
  ~~. to escape console [uses cu(1) for console]
  Connected
  ```

* Chose Auto (ZFS), added only ada2/disk2 to the rootpool.

```
=E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80ZFS Configuration=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=90
=E2=94=82 Configure Options:                                 =E2=94=82
=E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=90 =E2=94=82
=E2=94=82 =E2=94=82 >>> Install          Proceed with Installation =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 T Pool Type/Disks:   stripe: 1 disk            =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 - Rescan Devices     *                         =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 - Disk Info          *                         =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 N Pool Name          zroot                     =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 4 Force 4K Sectors?  YES                       =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 E Encrypt Disks?     YES                       =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 P Partition Scheme   GPT                       =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 S Swap Size          256m                      =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 M Mirror Swap?       NO                        =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=82 W Encrypt Swap?      YES                       =E2=94=
=82 =E2=94=82
=E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=98 =E2=94=82
=E2=94=9C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=A4
=E2=94=82             <Select>       <Cancel>                =E2=94=82
=E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=
=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=
=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=
=E2=94=80=E2=94=80=E2=94=80=E2=94=98
```

* proceed with installation.
* remove disk1 from the VM:
  ```
  floh@fuchi-cyber220:~:1313% sudo iohyve remove fbsd102 disk1
  Are you sure you want to remove disk1 from fbsd102 [Y/N]? y
  floh@fuchi-cyber220:~:1314% iohyve info
  Name           Size  RAM  CPU  OS       Loader
  fbsd102        -     2G   1    default  bhyveload
  fbsd102/disk0  2G    2G   1    default  bhyveload
  fbsd102/disk2  4G    2G   1    default  bhyveload
  ```

* shuffle disks around:
  ```
  floh@fuchi-cyber220:~:1340% sudo zfs rename zroot/iohyve/fbsd102/disk{0,1}
  floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{2,0}
  floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{1,2}
  ```
* now the 4GB one is `disk0`:
  ```
  floh@fuchi-cyber220:~:1343% iohyve info
  Name           Size  RAM  CPU  OS       Loader
  fbsd102        -     2G   1    default  bhyveload
  fbsd102/disk0  4G    2G   1    default  bhyveload
  fbsd102/disk2  2G    2G   1    default  bhyveload
  ```

* boot the VM and try to get the rootpool imported:
```
floh@fuchi-cyber220:~:1344% sudo iohyve start fbsd102
Starting fbsd102... (Takes 15 seconds for FreeBSD guests)
floh@fuchi-cyber220:~:1344%
floh@fuchi-cyber220:~:1345% sudo iohyve console fbsd102
Starting console on fbsd102...
~~. to escape console [uses cu(1) for console]
Connected
  /
  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DWelcome to FreeBSD=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+


/boot/kernel/kernel text=3D0xfc8de8 data=3D0x1283b0+0x207880
syms=3D[0x8+0x145350+0x8+0x15fe20]
/boot/kernel/zfs.ko size 0x2f9b00 at 0x199e000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x6048 at 0x1c98000
/boot/kernel/geom_eli.ko size 0x21568 at 0x1c9f000
loading required module 'crypto'
/boot/kernel/crypto.ko size 0x35318 at 0x1cc1000
/boot/encryption.key size=3D0x1000
/boot/kernel/aesni.ko size 0x5a30 at 0x1cf8000
/boot/zfs/zpool.cache size=3D0x8bc
Booting...
Copyright (c) 1992-2015 The FreeBSD Project.

[... lots kernel messages ...]

ada1: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada1: Serial Number BHYVE-D12E-A75C-27F5
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2048MB (4194304 512 byte sectors: 16H 63S/T 4161C)
ada1: Previously was known as ad6
random: unblocking device.
Timecounter "TSC-low" frequency 1345403547 Hz quality 1000
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 2.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 1.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. No tries left.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 2.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 1.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. No tries left.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 2.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 1.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. No tries left.
Trying to mount root from zfs:zroot/ROOT/default []...
Mounting from zfs:zroot/ROOT/default failed with error 2.

Loader variables:
  vfs.root.mountfrom=3Dzfs:zroot/ROOT/default

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/acd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>
panic: mountroot: unable to (re-)mount root.
cpuid =3D 0
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff809e7f5f at vfs_mountroot+0x1eaf
#4 0xffffffff808f03b3 at start_init+0x53
#5 0xffffffff8091244a at fork_exit+0x9a
#6 0xffffffff80d30d2e at fork_trampoline+0xe
Uptime: 4m16s
```

And, no, srsly, I don't need more than 9 tries to type "foobar".

Changing the order of disks back would fix the problem.
As there's no BIOS menu to choose the boot device in bhyve
having no bootcode in (the original) disk0 makes this a
tiny bit difficult with bhyve.

--=20
You are receiving this mail because:
You are the assignee for the bug.=