From owner-netperf-users@freebsd.org Sun Nov 22 03:11:24 2020 Return-Path: Delivered-To: netperf-users@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0AD1147ADDD for ; Sun, 22 Nov 2020 03:11:24 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CdwKl0ntXz4TCX; Sun, 22 Nov 2020 03:11:23 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:b54f:3bf5:d5cc:71b0] ([IPv6:2607:f3e0:0:4:b54f:3bf5:d5cc:71b0]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 0AM3BMxp023125 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 21 Nov 2020 22:11:22 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: zoo reboot Friday Nov 20 14:00 UTC From: mike tancsa To: Mateusz Guzik , Allan Jude Cc: Philip Paeps , "Bjoern A. Zeeb" , netperf-admin@freebsd.org, netperf-users@freebsd.org, Josh Paetzel References: <1f8e49ff-e3da-8d24-57f1-11f17389aa84@sentex.net> <270b65c0-8085-fe2f-cf4f-7a2e4c17a2e8@sentex.net> <163d1815-fc4a-7987-30c5-0a21e8383c93@sentex.net> <5a46fa23-b09f-86c2-0cef-a9fbb248f2ec@freebsd.org> <949305ed-c248-1ee1-2c53-552f2c732dbc@sentex.net> Message-ID: <2115dda4-bd6a-fe90-ffd9-2d53c67f830f@sentex.net> Date: Sat, 21 Nov 2020 22:11:23 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <949305ed-c248-1ee1-2c53-552f2c732dbc@sentex.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4CdwKl0ntXz4TCX X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-1.99 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; SPAMHAUS_ZRD(0.00)[2607:f3e0:0:3::19:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f3e0:0:3::19:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[netperf-users] X-BeenThere: netperf-users@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Announcements and discussions related to the netperf cluster. " List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2020 03:11:24 -0000 Just a quick update for tonight. I have everything restored (I think) from the morning of the 20th.  I need to sort out how the home directories were for some users.  I am just building world/kernel with WITHOUT_LLVM_ASSERTIONS=yes WITH_MALLOC_PRODUCTION=yes KERNCONF=GENERIC-NODEBUG and then will continue tomorrow afternoon with a reboot as well as try and sort out everyone's home directory location.  But all the data seems to be there anyways     ---Mike On 11/21/2020 12:47 PM, mike tancsa wrote: > OK, the new zoo is booting off a pair of 500G SSDs we donated.  I am > restoring to the raidz array tank > >  pigz -d -c zroot-.0.gz | zfs recv -vF tank/old > > mdtancsa@zoo2:~ % zpool status >   pool: tank >  state: ONLINE > config: > >         NAME        STATE     READ WRITE CKSUM >         tank        ONLINE       0     0     0 >           raidz1-0  ONLINE       0     0     0 >             ada1p1  ONLINE       0     0     0 >             ada3p1  ONLINE       0     0     0 >             ada4p1  ONLINE       0     0     0 >             ada5p1  ONLINE       0     0     0 >             ada6p1  ONLINE       0     0     0 > > errors: No known data errors > >   pool: zooroot >  state: ONLINE > config: > >         NAME        STATE     READ WRITE CKSUM >         zooroot     ONLINE       0     0     0 >           mirror-0  ONLINE       0     0     0 >             ada7p4  ONLINE       0     0     0 >             ada8p4  ONLINE       0     0     0 > > errors: No known data errors > > it seems to be working so far > > root@zoo2:/home/mdtancsa # zfs list -t snapshot > NAME                USED  AVAIL     REFER  MOUNTPOINT > tank/old@HOURLY30     0B      -      141K  - > tank/old@HOURLY40     0B      -      141K  - > tank/old@HOURLY50     0B      -      141K  - > tank/old@HOURLY00     0B      -      141K  - > tank/old@HOURLY10     0B      -      141K  - > tank/old@HOURLY20     0B      -      141K  - > tank/old@prev-1       0B      -      141K  - > tank/old@1            0B      -      141K  - > tank/old@2            0B      -      141K  - > tank/old@3            0B      -      141K  - > tank/old@4            0B      -      141K  - > tank/old@5            0B      -      141K  - > tank/old@6            0B      -      141K  - > tank/old@0            0B      -      141K  - > root@zoo2:/home/mdtancsa # > > I imagine it will take a while > > After the "level 0" is done, > > pigz -d -c zroot-.1.gz | zfs recv -v tank/old > > Unfortunately, I set up these backup scripts many years ago before I had > a sense of zfs and saw it all through the lens of dump/restore :(  It > was one of those, "I should get to fixing the backup soon" :( > >     ---Mike > > On 11/21/2020 11:18 AM, mike tancsa wrote: >> Just going to reinstall now. I will boot from 2 new SSDs and then use 4 >> 4TB in RAIDZ >> >> On 11/21/2020 12:47 AM, Mateusz Guzik wrote: >>> root@zoo2:/home/mjg # zdb -l /dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 >>> ------------------------------------ >>> LABEL 0 >>> ------------------------------------ >>> version: 5000 >>> name: 'zroot' >>> state: 0 >>> txg: 40630433 >>> pool_guid: 11911329414887727775 >>> errata: 0 >>> hostid: 3594518197 >>> hostname: 'zoo2.sentex.ca' >>> top_guid: 7321270789669113643 >>> guid: 9170931574354766059 >>> vdev_children: 4 >>> vdev_tree: >>> type: 'mirror' >>> id: 3 >>> guid: 7321270789669113643 >>> metaslab_array: 26179 >>> metaslab_shift: 32 >>> ashift: 9 >>> asize: 482373533696 >>> is_log: 0 >>> create_txg: 40274122 >>> children[0]: >>> type: 'disk' >>> id: 0 >>> guid: 9170931574354766059 >>> path: '/dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0' >>> whole_disk: 1 >>> create_txg: 40274122 >>> children[1]: >>> type: 'disk' >>> id: 1 >>> guid: 4871900363652985181 >>> path: '/dev/mfid1p2' >>> whole_disk: 1 >>> create_txg: 40274122 >>> features_for_read: >>> com.delphix:hole_birth >>> com.delphix:embedded_data >>> labels = 0 1 2 3 >>> >>> >>> On 11/21/20, Allan Jude wrote: >>>> On 2020-11-20 21:56, Mateusz Guzik wrote: >>>>> root@zoo2:/home/mjg # zpool import >>>>> pool: zroot >>>>> id: 11911329414887727775 >>>>> state: FAULTED >>>>> status: The pool metadata is corrupted. >>>>> action: The pool cannot be imported due to damaged devices or data. >>>>> The pool may be active on another system, but can be imported using >>>>> the '-f' flag. >>>>> see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 >>>>> config: >>>>> >>>>> zroot FAULTED corrupted data >>>>> mirror-0 DEGRADED >>>>> replacing-0 DEGRADED >>>>> 1517819109053923011 OFFLINE >>>>> ada0p3 ONLINE >>>>> ada1 ONLINE >>>>> mirror-1 ONLINE >>>>> ada3p3 ONLINE >>>>> ada4p3 ONLINE >>>>> mirror-2 ONLINE >>>>> ada5p3 ONLINE >>>>> ada6p3 ONLINE >>>>> mirror-3 ONLINE >>>>> gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 ONLINE >>>>> gptid/d98a2545-1a9c-11eb-8d25-0cc47a1f2fa0 ONLINE >>>>> >>>>> >>>>> On 11/21/20, Allan Jude wrote: >>>>>> On 2020-11-20 18:05, mike tancsa wrote: >>>>>>> OK. Although looks like I will have to pull it in from backups now :( >>>>>>> >>>>>>> >>>>>>> root@zoo2:/home/mdtancsa # zpool import -f -R /mnt zroot >>>>>>> cannot import 'zroot': I/O error >>>>>>> Destroy and re-create the pool from >>>>>>> a backup source. >>>>>>> root@zoo2:/home/mdtancsa # >>>>>>> >>>>>>> all the disks are there :( Not sure why its not importing ? >>>>>>> >>>>>> Can you get the output of just: >>>>>> >>>>>> zpool import >>>>>> >>>>>> To try to see what the issue might be >>>>>> >>>>>> -- >>>>>> Allan Jude >>>>>> >>>> The special vdev appears to be being see as just a plain mirror vdev, >>>> that is odd. >>>> >>>> zdb -l /dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 >>>> >>>> >>>> -- >>>> Allan Jude >>>>