From owner-freebsd-questions@freebsd.org Tue May 21 07:12:59 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41AD815A50E6 for ; Tue, 21 May 2019 07:12:59 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [IPv6:2001:470:0:19b::b869:801b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "xray.he.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59E0A80A6E for ; Tue, 21 May 2019 07:12:58 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from 99.100.19.101 ([99.100.19.101]) by holgerdanske.com with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256:TLSv1.2:Kx=ECDH:Au=RSA:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Tue, 21 May 2019 00:12:56 -0700 Subject: Re: zpool import -- state: UNAVAIL, status: One or more devices are missing To: freebsd-questions@freebsd.org References: From: David Christensen Message-ID: Date: Tue, 21 May 2019 00:12:47 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 May 2019 07:12:59 -0000 On 5/20/19 7:17 PM, David Christensen wrote: > I exported the pool per "Migrating ZFS Storage Pools": > > https://docs.oracle.com/cd/E19253-01/819-5461/gbchy/index.html > > root@dipsy:~ # zpool export tracy_p0 > But when I try to import the pool: > > 2019-05-20 17:21:21 toor@dipsy ~ > # zpool import >    pool: tracy_p0 >      id: 6957586590476290317 >   state: UNAVAIL >  status: One or more devices are missing from the system. >  action: The pool cannot be imported. Attach the missing >     devices and try again. >    see: http://illumos.org/msg/ZFS-8000-3C >  config: > >     tracy_p0                  UNAVAIL  insufficient replicas >       raidz1-0                UNAVAIL  insufficient replicas >         8533131415900712652   UNAVAIL  cannot open >         11863649002236717295  UNAVAIL  cannot open >         16410494706036382010  UNAVAIL  cannot open I found the solution: Reboot SSD. When the following command is issued (via SSH): 2019-05-20 23:46:22 toor@dipsy ~ # zpool import pool: tracy_p0 id: 6957586590476290317 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-3C config: tracy_p0 UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas 8533131415900712652 UNAVAIL cannot open 11863649002236717295 UNAVAIL cannot open 16410494706036382010 UNAVAIL cannot open The following appears on the console: GEOM_ELI: Device ada2s1.eli destroyed. GEOM_ELI: Detached ada2s1.eli on last close. GEOM_ELI: Device ada1s1.eli destroyed. GEOM_ELI: Detached ada211.eli on last close. GEOM_ELI: Device ada3s1.eli destroyed. GEOM_ELI: Detached ada3s1.eli on last close. Suspect that some boot loader gives ZFS one chance to connect to GELI drives and, if ZFS fails, GELI will nuke the *.eli providers whenever the next thing tries to access them ('zpool import') (?). Look at /etc/rc.conf again for clue as to how to attach GELI providers: 2019-05-20 23:46:27 toor@dipsy ~ # grep ada /etc/rc.conf geli_ada1s1_flags="-k /root/raidz-geli.key -p" geli_ada2s1_flags="-k /root/raidz-geli.key -p" geli_ada3s1_flags="-k /root/raidz-geli.key -p" geli_devices="ada1s1 ada2s1 ada3s1" Attach them manually: 2019-05-21 00:00:11 toor@dipsy ~ # geli attach -k /root/raidz-geli.key -p -v ada1s1 Attached to ada1s1. Done. 2019-05-21 00:01:25 toor@dipsy ~ # geli attach -k /root/raidz-geli.key -p -v ada2s1 Attached to ada2s1. Done. 2019-05-21 00:01:27 toor@dipsy ~ # geli attach -k /root/raidz-geli.key -p -v ada3s1 Attached to ada3s1. Done. 2019-05-21 00:02:28 toor@dipsy ~ # geli status Name Status Components ada0s1d.eli ACTIVE ada0s1d ada1s1.eli ACTIVE ada1s1 ada2s1.eli ACTIVE ada2s1 ada3s1.eli ACTIVE ada3s1 2019-05-21 00:02:31 toor@dipsy ~ # ls /dev/ada* /dev/ada0 /dev/ada0s1d.eli /dev/ada2s1 /dev/ada0s1 /dev/ada1 /dev/ada2s1.eli /dev/ada0s1a /dev/ada1s1 /dev/ada3 /dev/ada0s1b /dev/ada1s1.eli /dev/ada3s1 /dev/ada0s1d /dev/ada2 /dev/ada3s1.eli Import zpool: 2019-05-21 00:03:43 toor@dipsy ~ # zpool import pool: tracy_p0 id: 6957586590476290317 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tracy_p0 ONLINE raidz1-0 ONLINE ada3s1.eli ONLINE ada1s1.eli ONLINE ada2s1.eli ONLINE 2019-05-21 00:04:30 toor@dipsy ~ # zpool import tracy_p0 2019-05-21 00:04:51 toor@dipsy ~ # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tracy_p0 4.06T 1.34T 2.72T - - 0% 33% 1.00x ONLINE - Easy once you know how! ;-) David