From owner-freebsd-stable@freebsd.org Fri Oct 20 15:19:07 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEB6DE386AE for ; Fri, 20 Oct 2017 15:19:07 +0000 (UTC) (envelope-from mw@dudes.ch) Received: from mail.dudes.ch (mail.dudes.ch [193.73.211.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.dudes.ch", Issuer "StartCom Class 3 OV Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 963317EF6C for ; Fri, 20 Oct 2017 15:19:07 +0000 (UTC) (envelope-from mw@dudes.ch) Received: from mwoffice.virtualtec.office (pippin.virtualtec.ch [93.189.66.120]) (authenticated bits=0) by mail.dudes.ch (8.15.2/8.15.2) with ESMTPSA id v9KFDjl0077741 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 20 Oct 2017 17:13:46 +0200 (CEST) (envelope-from mw@dudes.ch) X-Authentication-Warning: mail.dudes.ch: Host pippin.virtualtec.ch [93.189.66.120] claimed to be mwoffice.virtualtec.office Date: Fri, 20 Oct 2017 17:13:45 +0200 From: Markus Wild To: freebsd-stable@freebsd.org Subject: System fails to boot with zfs root on nvme mirror when not using full disks Message-ID: <20171020171345.073bd9b1@mwoffice.virtualtec.office> X-Mailer: Claws Mail 3.15.1 (GTK+ 2.24.31; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.78 on 193.73.211.25 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 15:19:08 -0000 Hello list, I have a particularly odd problem, where I can't figure out what's going on, or whether I'm possibly doing something stupid... Short summary: - Supermicro X10DRI-T motherboard, using UEFI boot, 128G RAM, 2 CPUs - 2 Intel DC P3700 NVME cards, to be used as mirrored zfs root and mirrored log devices for a data pool - FreeBSD-11.1 release installs fine off memstick, and the built system boots correctly, but this system uses the entire disks (so I need to shrink the system to make room for log partitions) - so I then rebooted into usb live system, and did a gpart backup of the nvme drives (see later). zfs snapshot -r, zfs send -R > backup, gpart delete last index and recreate with shorter size on both drives, recreate zroot pool with correct ashift, restore with zfs receive from backup, set bootfs, reboot - the rebooting system bootloader finds the zroot pool correctly, and proceeds to load the kernel. However, when it's supposed to mount the root filesystem, I get: Trying to mount root from zfs:zroot/ROOT/default []... Mounting from zfs:zroot/ROOT/default failed with error 6. - when I list the available boot devices, all partitions of the nvme disks are listed - I can import this pool without any issues with the usb live system, there are no errors on import. - I then redid the whole exercise, but restored the previously backed up partition tables (full size zfs partition), did exactly the same steps as described above to restore the previous zfs filesystem, rebooted, and the system started up normally. So, my impression is: the kernel is doing something odd when trying to import the zroot pool, if that pool isn't using the whole physical disk space. Is there some way to enable debugging for the "mount a root zfs pool" process of the kernel, so see where this is failing? Here's a more detailled history of my commands to reduce the pool size: {I'm on a data pool directory here, where I can store stuff. I've booted from usb stick into live system} zpool import -Nf zroot zfs snapshot -r zroot@backup zfs send -R zroot@backup > zroot.zfs zpool destroy zroot gpart delete -i 3 nvd0 gpart delete -i 3 nvd1 gpart add -a 4k -s 50G -t freebsd-zfs -l zfs0 nvd0 gpart add -a 4k -s 50G -t freebsd-zfs -l zfs1 nvd1 sysctl vfs.zfs.min_auto_ashift=12 zpool create -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/zfs0 /dev/gpt/zfs1 zfs receive -Fud zroot < zroot.zfs zpool set bootfs=zroot/ROOT/default zroot reboot Also, here's a diff between full and partial gpart backup: --- nvd0.gpart 2017-10-20 14:04:26.583846000 +0200 +++ nvd0.gpart.new 2017-10-20 15:39:20.184445000 +0200 @@ -1,4 +1,4 @@ GPT 152 1 efi 40 409600 efiboot0 2 freebsd-swap 411648 33554432 swap0 -3 freebsd-zfs 33966080 747456512 zfs0 +3 freebsd-zfs 33966080 104857600 zfs0 I'm really at a loss to see the reason why this is failing. I can provide detailled dmesg output of the booting system, and screenshots of the kvm-efi-console of the failing boot. Thanks for any help:) Cheers, Markus