From owner-netperf-users@freebsd.org Fri Dec 18 14:14:37 2020 Return-Path: Delivered-To: netperf-users@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C46814B9C5F for ; Fri, 18 Dec 2020 14:14:37 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cy9q151tLz4jRx; Fri, 18 Dec 2020 14:14:37 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:702b:f028:0:2b15] ([IPv6:2607:f3e0:0:4:702b:f028:0:2b15]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 0BIEEUV3088085 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 18 Dec 2020 09:14:30 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: zoo back online (was Re: zoo hang) To: George Neville-Neil Cc: Mateusz Guzik , "netperf-admin@FreeBSD.org" , netperf-users@freebsd.org, Paul Holes References: <5483e76e-4a2f-3153-c10b-7902839c1b68@sentex.net> <8c26a0d3-3bd0-7535-0abc-3d1e9e5ac7c4@sentex.net> <64923d33-4bf2-0fd5-1b17-d6bd73e9fd32@sentex.net> <13a9ab42-1df8-c054-0c83-5708ab9d9e2b@sentex.net> From: mike tancsa Message-ID: <6cef40cd-de57-aa84-bc70-ceea71add397@sentex.net> Date: Fri, 18 Dec 2020 09:14:31 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Cy9q151tLz4jRx X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: netperf-users@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Announcements and discussions related to the netperf cluster. " List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Dec 2020 14:14:37 -0000 Hi George,     I think the boot loader is now fixed as those features are white listed.  Will start backups once again via zrepl.     ---Mike On 12/17/2020 1:58 PM, George Neville-Neil wrote: > Howdy, > > How do we want to handle the old tank stuff? > > Best, > George > > > On 15 Dec 2020, at 16:24, mike tancsa wrote: > >> OK, thanks to Josh P's suggestion, deleting the v2 bookmarks from the >> pool allowed us to boot. >> >> Booted from a temp drive, imported the pool, >> >> root@zoo-temp:~ # zpool import -R /mnt -f zooroot >> root@zoo-temp:~ # zfs list -t bookmark | grep ^z | awk '{print "zfs >> destroy "$1}' >> zfs destroy zooroot#zrepl_CURSOR_G_77296a02a81c78cc_J_push_to_drive >> zfs destroy zooroot/ROOT#zrepl_CURSOR_G_e27691751ed1660b_J_push_to_drive >> zfs destroy >> zooroot/ROOT/default#zrepl_CURSOR_G_607fa8e4c7df13b5_J_push_to_drive >> zfs destroy zooroot/tmp#zrepl_CURSOR_G_25ae8e2b8723a008_J_push_to_drive >> zfs destroy zooroot/usr#zrepl_CURSOR_G_344a884262b3e387_J_push_to_drive >> zfs destroy >> zooroot/usr/home#zrepl_CURSOR_G_2e4087f8f219bd83_J_push_to_drive >> zfs destroy >> zooroot/usr/ports#zrepl_CURSOR_G_fb8384d458dd82b3_J_push_to_drive >> zfs destroy >> zooroot/usr/src#zrepl_CURSOR_G_b867573acd8a57f8_J_push_to_drive >> zfs destroy zooroot/var#zrepl_CURSOR_G_ea9efdf01fdf65b5_J_push_to_drive >> zfs destroy >> zooroot/var/audit#zrepl_CURSOR_G_e71132efb0fee45a_J_push_to_drive >> zfs destroy >> zooroot/var/crash#zrepl_CURSOR_G_191c17e9538113f4_J_push_to_drive >> zfs destroy >> zooroot/var/log#zrepl_CURSOR_G_f30668295109ad60_J_push_to_drive >> zfs destroy >> zooroot/var/mail#zrepl_CURSOR_G_7d1eac92237e2603_J_push_to_drive >> zfs destroy >> zooroot/var/tmp#zrepl_CURSOR_G_d593288357e0a319_J_push_to_drive >> root@zoo-temp:~ # zfs list -t bookmark | grep ^z | awk '{print "zfs >> destroy "$1}' | sh >> root@zoo-temp:~ # >> root@zoo-temp:~ # zpool export zooroot >> root@zoo-temp:~ # >> >> and rebooted and its up. Sadly, will need to come up with another backup >> system as sysutils/zrepl uses bookmarks :( >> >>     ---Mike >> >> On 12/15/2020 1:46 PM, mike tancsa wrote: >>> Looks like the loader does not support v2 bookmarks. I am going to get >>> Paul to put in another disk to boot from, mjg will login, either >>> destroy >>> the bookmarks or hack a loader fix that will allow the box to boot with >>> this feature.  Will be an hour or so as we have a office meeting at 2pm >>> we both have to attend. >>> >>>     ---Mike >>> >>> On 12/15/2020 1:28 PM, mike tancsa wrote: >>>> I am guessing because I was using zrepl from the ports to do >>>> replication >>>> / backup to a secondary disk, the use of the bookmark_v2 feature is >>>> not >>>> supported on ZoL ? Any way to recover from this ? >>>> >>>> >>>> On 12/15/2020 1:10 PM, mike tancsa wrote: >>>>> OK, but the first problem to deal with :( >>>>> >>>>>                                                                               >>>>> >>>>> >>>>> BIOS drive C: is >>>>> disk0                                                          >>>>> BIOS drive D: is >>>>> disk1                                                          >>>>> ZFS: unsupported feature: >>>>> com.datto:bookmark_v2                                 >>>>> ZFS: pool zooroot is not >>>>> supported                                              >>>>>                                                                                 >>>>> >>>>> >>>>> Can't find >>>>> /boot/zfsloader                                                      >>>>>                                                                                 >>>>> >>>>> >>>>> Can't find >>>>> /boot/loader                                                         >>>>>                                                                                 >>>>> >>>>> >>>>> Can't find >>>>> /boot/kernel/kernel                                                  >>>>>                                                                                 >>>>> >>>>> >>>>> FreeBSD/x86 >>>>> boot                                                                >>>>> Default: >>>>> /boot/kernel/kernel                                                    >>>>> >>>>> boot:                                                                           >>>>> >>>>> >>>>>                                                                                 >>>>> >>>>> >>>>> Can't find >>>>> /boot/kernel/kernel                                                  >>>>>                                                                                 >>>>> >>>>> >>>>> FreeBSD/x86 >>>>> boot                                                                >>>>> Default: >>>>> /boot/kernel/kernel                                                    >>>>> >>>>> boot:                                 >>>>> >>>>> On 12/15/2020 1:02 PM, Mateusz Guzik wrote: >>>>>> We need to update to r368649 for a pmap fix regardless of the >>>>>> above. I >>>>>> can do the work and make the box ready for the next reboot. >>>>>> >>>>>> On 12/15/20, mike tancsa wrote: >>>>>>> The USB backup disk was throwing errors and I was trying to >>>>>>> export the >>>>>>> backup pool and it looks like the box is hung now. I am going to >>>>>>> power >>>>>>> cycle it >>>>>>> >>>>>>>     ---Mike >>>>>>> >>>>>>> >>>>>>> >>>>>>> >