From owner-netperf-users@freebsd.org Tue Dec 15 21:24:25 2020 Return-Path: Delivered-To: netperf-users@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7AA34C2C66 for ; Tue, 15 Dec 2020 21:24:25 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CwWVJ6l9qz4v0d; Tue, 15 Dec 2020 21:24:24 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:20e5:8bab:f6d6:8c81] ([IPv6:2607:f3e0:0:4:20e5:8bab:f6d6:8c81]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 0BFLOMpS090645 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 15 Dec 2020 16:24:22 -0500 (EST) (envelope-from mike@sentex.net) Subject: zoo back online (was Re: zoo hang) From: mike tancsa To: Mateusz Guzik Cc: "netperf-admin@FreeBSD.org" , netperf-users@freebsd.org, Paul Holes References: <5483e76e-4a2f-3153-c10b-7902839c1b68@sentex.net> <8c26a0d3-3bd0-7535-0abc-3d1e9e5ac7c4@sentex.net> <64923d33-4bf2-0fd5-1b17-d6bd73e9fd32@sentex.net> Message-ID: <13a9ab42-1df8-c054-0c83-5708ab9d9e2b@sentex.net> Date: Tue, 15 Dec 2020 16:24:23 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.1 MIME-Version: 1.0 In-Reply-To: <64923d33-4bf2-0fd5-1b17-d6bd73e9fd32@sentex.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4CwWVJ6l9qz4v0d X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f3e0:0:3::19:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sentex.net]; SPAMHAUS_ZRD(0.00)[2607:f3e0:0:3::19:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[netperf-users] X-BeenThere: netperf-users@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Announcements and discussions related to the netperf cluster. " List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2020 21:24:25 -0000 OK, thanks to Josh P's suggestion, deleting the v2 bookmarks from the pool allowed us to boot. Booted from a temp drive, imported the pool, root@zoo-temp:~ # zpool import -R /mnt -f zooroot root@zoo-temp:~ # zfs list -t bookmark | grep ^z | awk '{print "zfs destroy "$1}' zfs destroy zooroot#zrepl_CURSOR_G_77296a02a81c78cc_J_push_to_drive zfs destroy zooroot/ROOT#zrepl_CURSOR_G_e27691751ed1660b_J_push_to_drive zfs destroy zooroot/ROOT/default#zrepl_CURSOR_G_607fa8e4c7df13b5_J_push_to_drive zfs destroy zooroot/tmp#zrepl_CURSOR_G_25ae8e2b8723a008_J_push_to_drive zfs destroy zooroot/usr#zrepl_CURSOR_G_344a884262b3e387_J_push_to_drive zfs destroy zooroot/usr/home#zrepl_CURSOR_G_2e4087f8f219bd83_J_push_to_drive zfs destroy zooroot/usr/ports#zrepl_CURSOR_G_fb8384d458dd82b3_J_push_to_drive zfs destroy zooroot/usr/src#zrepl_CURSOR_G_b867573acd8a57f8_J_push_to_drive zfs destroy zooroot/var#zrepl_CURSOR_G_ea9efdf01fdf65b5_J_push_to_drive zfs destroy zooroot/var/audit#zrepl_CURSOR_G_e71132efb0fee45a_J_push_to_drive zfs destroy zooroot/var/crash#zrepl_CURSOR_G_191c17e9538113f4_J_push_to_drive zfs destroy zooroot/var/log#zrepl_CURSOR_G_f30668295109ad60_J_push_to_drive zfs destroy zooroot/var/mail#zrepl_CURSOR_G_7d1eac92237e2603_J_push_to_drive zfs destroy zooroot/var/tmp#zrepl_CURSOR_G_d593288357e0a319_J_push_to_drive root@zoo-temp:~ # zfs list -t bookmark | grep ^z | awk '{print "zfs destroy "$1}' | sh root@zoo-temp:~ # root@zoo-temp:~ # zpool export zooroot root@zoo-temp:~ # and rebooted and its up. Sadly, will need to come up with another backup system as sysutils/zrepl uses bookmarks :(     ---Mike On 12/15/2020 1:46 PM, mike tancsa wrote: > Looks like the loader does not support v2 bookmarks. I am going to get > Paul to put in another disk to boot from, mjg will login, either destroy > the bookmarks or hack a loader fix that will allow the box to boot with > this feature.  Will be an hour or so as we have a office meeting at 2pm > we both have to attend. > >     ---Mike > > On 12/15/2020 1:28 PM, mike tancsa wrote: >> I am guessing because I was using zrepl from the ports to do replication >> / backup to a secondary disk, the use of the bookmark_v2 feature is not >> supported on ZoL ? Any way to recover from this ? >> >> >> On 12/15/2020 1:10 PM, mike tancsa wrote: >>> OK, but the first problem to deal with :( >>> >>>                                                                               >>> >>> BIOS drive C: is >>> disk0                                                          >>> BIOS drive D: is >>> disk1                                                          >>> ZFS: unsupported feature: >>> com.datto:bookmark_v2                                 >>> ZFS: pool zooroot is not >>> supported                                              >>>                                                                                 >>> >>> Can't find >>> /boot/zfsloader                                                      >>>                                                                                 >>> >>> Can't find >>> /boot/loader                                                         >>>                                                                                 >>> >>> Can't find >>> /boot/kernel/kernel                                                  >>>                                                                                 >>> >>> FreeBSD/x86 >>> boot                                                                >>> Default: >>> /boot/kernel/kernel                                                    >>> boot:                                                                           >>> >>>                                                                                 >>> >>> Can't find >>> /boot/kernel/kernel                                                  >>>                                                                                 >>> >>> FreeBSD/x86 >>> boot                                                                >>> Default: >>> /boot/kernel/kernel                                                    >>> boot:                                 >>> >>> On 12/15/2020 1:02 PM, Mateusz Guzik wrote: >>>> We need to update to r368649 for a pmap fix regardless of the above. I >>>> can do the work and make the box ready for the next reboot. >>>> >>>> On 12/15/20, mike tancsa wrote: >>>>> The USB backup disk was throwing errors and I was trying to export the >>>>> backup pool and it looks like the box is hung now. I am going to power >>>>> cycle it >>>>> >>>>>     ---Mike >>>>> >>>>> >>>>> >>>>>