From owner-freebsd-questions@freebsd.org Tue Aug 7 14:48:34 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF8571060E7E for ; Tue, 7 Aug 2018 14:48:33 +0000 (UTC) (envelope-from bsd@lordcow.org) Received: from mail.lordcow.org (lordcow.org [197.155.6.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "devaux.za.net", Issuer "Cow Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4055D7288C for ; Tue, 7 Aug 2018 14:48:32 +0000 (UTC) (envelope-from bsd@lordcow.org) Received: from lordcow.org (localhost [127.0.0.1]) by mail.lordcow.org (8.15.2/8.15.2) with ESMTPS id w77EPffN065484 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 7 Aug 2018 16:25:41 +0200 (SAST) (envelope-from lordcow@lordcow.org) X-Authentication-Warning: lordcow.org: Host localhost [127.0.0.1] claimed to be lordcow.org Received: (from lordcow@localhost) by lordcow.org (8.15.2/8.15.2/Submit) id w77EPZhn064984 for freebsd-questions@freebsd.org; Tue, 7 Aug 2018 16:25:35 +0200 (SAST) (envelope-from lordcow) Date: Tue, 7 Aug 2018 16:25:35 +0200 From: Gareth de Vaux To: freebsd-questions@freebsd.org Subject: Replacing failed drives on a raidz2 with geli Message-ID: <20180807142535.GA20017@lordcow.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lordcow.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2018 14:48:34 -0000 Hi all, I've installed a Root on ZFS raidz2 with geli encryption on FreeBSD 11.2 from the installer. I'm new to geli so trying to simulate a disk failure to make sure I can recover. Once I've replaced 3 drives the zpool seems fine but fails to reboot: http://lordcow.org/boot.jpg , specifically: ZFS: i/o error - all block copies unavailable I'm also asked to enter the passphrase for the new disks separately on boot up prior to the 3rd one failing. I'm guessing the bootcode isn't getting copied correctly but not sure where I'm going wrong, this's what I do after failing and zero'ing a drive: gpart create -s gpt ada0 gpart add -b 40 -s 512K -t freebsd-boot -l gptboot0 ada0 gpart add -b 2048 -s 2G -t freebsd-swap -l swap0 ada0 gpart add -s 15623856128 -t freebsd-zfs -l zfs0 ada0 (to match the tables of the other disks) gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 geli init -e AES-XTS -l 256 -s 4096 -b /dev/ada0p3 (Assuming these are the right parameters given what the installer does? Not using a key, just a passphrase. Should master keys on the existing array be backed up? how?) geli attach /dev/ada0p3 zpool replace zroot ada0p3.eli This succesfully resilvers the drive, but again on the 3rd drive it fails to boot. Any help appreciated.