From owner-netperf-users@freebsd.org Sat Nov 21 17:47:31 2020 Return-Path: Delivered-To: netperf-users@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A293946C6A0 for ; Sat, 21 Nov 2020 17:47:31 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cdgq65dGwz3D2l; Sat, 21 Nov 2020 17:47:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:b54f:3bf5:d5cc:71b0] ([IPv6:2607:f3e0:0:4:b54f:3bf5:d5cc:71b0]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 0ALHlTY0022053 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 21 Nov 2020 12:47:30 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: zoo reboot Friday Nov 20 14:00 UTC From: mike tancsa To: Mateusz Guzik , Allan Jude Cc: Philip Paeps , "Bjoern A. Zeeb" , netperf-admin@freebsd.org, netperf-users@freebsd.org References: <1f8e49ff-e3da-8d24-57f1-11f17389aa84@sentex.net> <270b65c0-8085-fe2f-cf4f-7a2e4c17a2e8@sentex.net> <163d1815-fc4a-7987-30c5-0a21e8383c93@sentex.net> <5a46fa23-b09f-86c2-0cef-a9fbb248f2ec@freebsd.org> Message-ID: <949305ed-c248-1ee1-2c53-552f2c732dbc@sentex.net> Date: Sat, 21 Nov 2020 12:47:30 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Cdgq65dGwz3D2l X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; RCPT_COUNT_FIVE(0.00)[6]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f3e0:0:3::19:from]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.6:email,0.0.0.2:email,0.0.0.3:email,0.0.0.0:email,0.0.0.4:email,0.0.0.1:email,0.0.0.5:email]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sentex.net]; SPAMHAUS_ZRD(0.00)[2607:f3e0:0:3::19:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[netperf-users] X-BeenThere: netperf-users@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Announcements and discussions related to the netperf cluster. " List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 17:47:31 -0000 OK, the new zoo is booting off a pair of 500G SSDs we donated.  I am restoring to the raidz array tank  pigz -d -c zroot-.0.gz | zfs recv -vF tank/old mdtancsa@zoo2:~ % zpool status   pool: tank  state: ONLINE config:         NAME        STATE     READ WRITE CKSUM         tank        ONLINE       0     0     0           raidz1-0  ONLINE       0     0     0             ada1p1  ONLINE       0     0     0             ada3p1  ONLINE       0     0     0             ada4p1  ONLINE       0     0     0             ada5p1  ONLINE       0     0     0             ada6p1  ONLINE       0     0     0 errors: No known data errors   pool: zooroot  state: ONLINE config:         NAME        STATE     READ WRITE CKSUM         zooroot     ONLINE       0     0     0           mirror-0  ONLINE       0     0     0             ada7p4  ONLINE       0     0     0             ada8p4  ONLINE       0     0     0 errors: No known data errors it seems to be working so far root@zoo2:/home/mdtancsa # zfs list -t snapshot NAME                USED  AVAIL     REFER  MOUNTPOINT tank/old@HOURLY30     0B      -      141K  - tank/old@HOURLY40     0B      -      141K  - tank/old@HOURLY50     0B      -      141K  - tank/old@HOURLY00     0B      -      141K  - tank/old@HOURLY10     0B      -      141K  - tank/old@HOURLY20     0B      -      141K  - tank/old@prev-1       0B      -      141K  - tank/old@1            0B      -      141K  - tank/old@2            0B      -      141K  - tank/old@3            0B      -      141K  - tank/old@4            0B      -      141K  - tank/old@5            0B      -      141K  - tank/old@6            0B      -      141K  - tank/old@0            0B      -      141K  - root@zoo2:/home/mdtancsa # I imagine it will take a while After the "level 0" is done, pigz -d -c zroot-.1.gz | zfs recv -v tank/old Unfortunately, I set up these backup scripts many years ago before I had a sense of zfs and saw it all through the lens of dump/restore :(  It was one of those, "I should get to fixing the backup soon" :(     ---Mike On 11/21/2020 11:18 AM, mike tancsa wrote: > Just going to reinstall now. I will boot from 2 new SSDs and then use 4 > 4TB in RAIDZ > > On 11/21/2020 12:47 AM, Mateusz Guzik wrote: >> root@zoo2:/home/mjg # zdb -l /dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 >> ------------------------------------ >> LABEL 0 >> ------------------------------------ >> version: 5000 >> name: 'zroot' >> state: 0 >> txg: 40630433 >> pool_guid: 11911329414887727775 >> errata: 0 >> hostid: 3594518197 >> hostname: 'zoo2.sentex.ca' >> top_guid: 7321270789669113643 >> guid: 9170931574354766059 >> vdev_children: 4 >> vdev_tree: >> type: 'mirror' >> id: 3 >> guid: 7321270789669113643 >> metaslab_array: 26179 >> metaslab_shift: 32 >> ashift: 9 >> asize: 482373533696 >> is_log: 0 >> create_txg: 40274122 >> children[0]: >> type: 'disk' >> id: 0 >> guid: 9170931574354766059 >> path: '/dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0' >> whole_disk: 1 >> create_txg: 40274122 >> children[1]: >> type: 'disk' >> id: 1 >> guid: 4871900363652985181 >> path: '/dev/mfid1p2' >> whole_disk: 1 >> create_txg: 40274122 >> features_for_read: >> com.delphix:hole_birth >> com.delphix:embedded_data >> labels = 0 1 2 3 >> >> >> On 11/21/20, Allan Jude wrote: >>> On 2020-11-20 21:56, Mateusz Guzik wrote: >>>> root@zoo2:/home/mjg # zpool import >>>> pool: zroot >>>> id: 11911329414887727775 >>>> state: FAULTED >>>> status: The pool metadata is corrupted. >>>> action: The pool cannot be imported due to damaged devices or data. >>>> The pool may be active on another system, but can be imported using >>>> the '-f' flag. >>>> see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 >>>> config: >>>> >>>> zroot FAULTED corrupted data >>>> mirror-0 DEGRADED >>>> replacing-0 DEGRADED >>>> 1517819109053923011 OFFLINE >>>> ada0p3 ONLINE >>>> ada1 ONLINE >>>> mirror-1 ONLINE >>>> ada3p3 ONLINE >>>> ada4p3 ONLINE >>>> mirror-2 ONLINE >>>> ada5p3 ONLINE >>>> ada6p3 ONLINE >>>> mirror-3 ONLINE >>>> gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 ONLINE >>>> gptid/d98a2545-1a9c-11eb-8d25-0cc47a1f2fa0 ONLINE >>>> >>>> >>>> On 11/21/20, Allan Jude wrote: >>>>> On 2020-11-20 18:05, mike tancsa wrote: >>>>>> OK. Although looks like I will have to pull it in from backups now :( >>>>>> >>>>>> >>>>>> root@zoo2:/home/mdtancsa # zpool import -f -R /mnt zroot >>>>>> cannot import 'zroot': I/O error >>>>>> Destroy and re-create the pool from >>>>>> a backup source. >>>>>> root@zoo2:/home/mdtancsa # >>>>>> >>>>>> all the disks are there :( Not sure why its not importing ? >>>>>> >>>>> Can you get the output of just: >>>>> >>>>> zpool import >>>>> >>>>> To try to see what the issue might be >>>>> >>>>> -- >>>>> Allan Jude >>>>> >>> The special vdev appears to be being see as just a plain mirror vdev, >>> that is odd. >>> >>> zdb -l /dev/gptid/db15e826-1a9c-11eb-8d25-0cc47a1f2fa0 >>> >>> >>> -- >>> Allan Jude >>> >