From owner-freebsd-questions@freebsd.org Tue May 21 02:18:53 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93CAC159E09A for ; Tue, 21 May 2019 02:18:53 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "xray.he.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D4E76F2AE for ; Tue, 21 May 2019 02:18:51 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from 99.100.19.101 ([99.100.19.101]) by holgerdanske.com with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256:TLSv1.2:Kx=ECDH:Au=RSA:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Mon, 20 May 2019 19:17:39 -0700 From: David Christensen To: freebsd-questions@freebsd.org Subject: zpool import -- state: UNAVAIL, status: One or more devices are missing Message-ID: Date: Mon, 20 May 2019 19:17:31 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 9D4E76F2AE X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-0.69 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.842,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-0.42)[ipnet: 184.104.0.0/15(1.10), asn: 6939(-3.13), country: US(-0.06)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; DMARC_NA(0.00)[holgerdanske.com]; NEURAL_SPAM_SHORT(0.68)[0.677,0]; MX_GOOD(-0.01)[holgerdanske.com]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 May 2019 02:18:53 -0000 freebsd-questions: I have a computer with FreeBSD: 2019-05-20 16:59:02 toor@dipsy ~ # freebsd-version ; uname -a 11.2-RELEASE-p9 FreeBSD dipsy 11.2-RELEASE-p9 FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 The computer had a 16 GB USB 3.0 flash drive as the system drive, with ZFS boot, GELI swap, and GELI ZFS root, all created by the FreeBSD 11.2 RELEASE amd64 installer "Auto (ZFS)". The computer has three 1.5 TB HDD's with GELI ZFS RAIDZ1: root@dipsy:~ # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tracy_p0 4.06T 1.34T 2.72T - - 0% 33% 1.00x ONLINE - I exported the pool per "Migrating ZFS Storage Pools": https://docs.oracle.com/cd/E19253-01/819-5461/gbchy/index.html root@dipsy:~ # zpool export tracy_p0 Then I copied the raw USB system drive image to an SSD. The machine would not boot -- it appeared that the boot loader could not find the root partition. I booted the FreeBSD 11.2 installer to a rescue shell, mounted the SSD boot partition, and looked around. I found that /boot/loader.conf had some variables containing the USB device node 'da0', so I changed them to the SSD device node 'ada0': 2019-05-20 17:00:19 toor@dipsy ~ # cat /boot/loader.conf | grep ada geli_ada0s1d_keyfile0_load="YES" geli_ada0s1d_keyfile0_type="ada0s1d:geli_keyfile0" geli_ada0s1d_keyfile0_name="/boot/encryption.key" Somehow, I discovered that I also needed to move aside /boot/ssd/zpool.cache. Now the machine can boot from the SSD, but the RAIDZ1 was missing. Looking in /etc/rc.conf, I saw some variables containing the HDD device nodes 'ada0', 'ada1', and 'ada2', so I changed them to the new device nodes 'ada1, 'ada2', and 'ada3': 2019-05-20 17:22:00 toor@dipsy ~ # cat /etc/rc.conf | grep ada geli_ada1s1_flags="-k /root/raidz-geli.key -p" geli_ada2s1_flags="-k /root/raidz-geli.key -p" geli_ada3s1_flags="-k /root/raidz-geli.key -p" geli_devices="ada1s1 ada2s1 ada3s1" After rebooting again, FreeBSD could now see the all the drives, partitions, and GELI encrypted disk devices: 2019-05-20 17:15:01 toor@dipsy ~ # camcontrol devlist | grep ada at scbus0 target 0 lun 0 (pass0,ada0) at scbus2 target 0 lun 0 (pass1,ada1) at scbus3 target 0 lun 0 (pass2,ada2) at scbus4 target 0 lun 0 (pass3,ada3) 2019-05-20 17:15:53 toor@dipsy ~ # geli status Name Status Components ada0s1d.eli ACTIVE ada0s1d ada1s1.eli ACTIVE ada1s1 ada2s1.eli ACTIVE ada2s1 ada3s1.eli ACTIVE ada3s1 2019-05-20 17:16:08 toor@dipsy ~ # ls -1 /dev/ada* /dev/ada0 /dev/ada0s1 /dev/ada0s1a /dev/ada0s1b /dev/ada0s1d /dev/ada0s1d.eli /dev/ada1 /dev/ada1s1 /dev/ada1s1.eli /dev/ada2 /dev/ada2s1 /dev/ada2s1.eli /dev/ada3 /dev/ada3s1 /dev/ada3s1.eli But when I try to import the pool: 2019-05-20 17:21:21 toor@dipsy ~ # zpool import pool: tracy_p0 id: 6957586590476290317 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-3C config: tracy_p0 UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 8533131415900712652 UNAVAIL cannot open 11863649002236717295 UNAVAIL cannot open 16410494706036382010 UNAVAIL cannot open Following the 'see' link, STFW, RTFM, etc., I am unable to determine how to attach the drives. Furthermore, if I boot the original USB system drive and try to import the RAIDZ1: 2019-05-20 18:48:43 toor@dipsy ~ # zpool import pool: tracy_p0 id: 6957586590476290317 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-3C config: tracy_p0 UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 8533131415900712652 UNAVAIL cannot open 11863649002236717295 UNAVAIL cannot open 16410494706036382010 UNAVAIL cannot open Any suggestions? David