From owner-freebsd-stable@freebsd.org Fri Oct 20 16:51:47 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE2D1E3A889 for ; Fri, 20 Oct 2017 16:51:47 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B8A38285B for ; Fri, 20 Oct 2017 16:51:47 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.89 (FreeBSD)) (envelope-from ) id 1e5aWO-0007pv-VY; Fri, 20 Oct 2017 17:51:44 +0100 Date: Fri, 20 Oct 2017 17:51:44 +0100 From: Gary Palmer To: Markus Wild Cc: freebsd-stable@freebsd.org Subject: Re: System fails to boot with zfs root on nvme mirror when not using full disks Message-ID: <20171020165144.GC96120@in-addr.com> References: <20171020171345.073bd9b1@mwoffice.virtualtec.office> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171020171345.073bd9b1@mwoffice.virtualtec.office> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 16:51:47 -0000 On Fri, Oct 20, 2017 at 05:13:45PM +0200, Markus Wild wrote: > Hello list, > > I have a particularly odd problem, where I can't figure out what's going on, or whether I'm possibly doing something > stupid... > > Short summary: > - Supermicro X10DRI-T motherboard, using UEFI boot, 128G RAM, 2 CPUs > - 2 Intel DC P3700 NVME cards, to be used as mirrored zfs root and mirrored log devices for a data pool > - FreeBSD-11.1 release installs fine off memstick, and the built system boots correctly, but this system uses the entire > disks (so I need to shrink the system to make room for log partitions) > - so I then rebooted into usb live system, and did a gpart backup of the nvme drives (see later). zfs snapshot -r, zfs > send -R > backup, gpart delete last index and recreate with shorter size on both drives, recreate zroot pool with > correct ashift, restore with zfs receive from backup, set bootfs, reboot > - the rebooting system bootloader finds the zroot pool correctly, and proceeds to load the kernel. However, when it's > supposed to mount the root filesystem, I get: > Trying to mount root from zfs:zroot/ROOT/default []... > Mounting from zfs:zroot/ROOT/default failed with error 6. > - when I list the available boot devices, all partitions of the nvme disks are listed My suspicion is that you didn't make a backup of the new /boot/zfs/zpool.cache >From the USB live booted system, when you recreated the zpool you likely changed parameters that live in the cache. You need to copy that to the new boot pool or ZFS can get angry Just a suspicion Regards, Gary