Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 11 Oct 2009 17:27:18 +0100
From:      Alex Trull <alextzfs@googlemail.com>
To:        freebsd-fs@freebsd.org
Cc:        pjd@freebsd.org
Subject:   Re: zraid2 loses a single disk and becomes difficult to recover
Message-ID:  <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com>
In-Reply-To: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>
References:  <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Well after trying alot of things (zpool import with or without cache file in
place, etc), it randomly managed to mount the pool up, atleast, with errors
- :

zfs list output:
cannot iterate filesystems: I/O error
NAME                      USED  AVAIL  REFER  MOUNTPOINT
fatman                   1.40T  1.70T  51.2K  /fatman
fatman/backup             100G  99.5G  95.5G  /fatman/backup
fatman/jail               422G  1.70T  60.5K  /fatman/jail
fatman/jail/havnor        198G  51.7G   112G  /fatman/jail/havnor
fatman/jail/mail         19.4G  30.6G  13.0G  /fatman/jail/mail
fatman/jail/syndicate    16.6G   103G  10.5G  /fatman/jail/syndicate
fatman/jail/thirdforces   159G  41.4G  78.1G  /fatman/jail/thirdforces
fatman/jail/web          24.8G  25.2G  22.3G  /fatman/jail/web
fatman/stash              913G  1.70T   913G  /fatman/stash

(end of the dmesg)
JMR: vdev_uberblock_load_done ubbest ub_txg=46475461 ub_timestamp=1255231841
JMR: vdev_uberblock_load_done ub_txg=46481476 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481476 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46475459 ub_timestamp=1255231780
JMR: vdev_uberblock_load_done ubbest ub_txg=46475458 ub_timestamp=1255231750
JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481473 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481472 ub_timestamp=1255234263
Solaris: WARNING: can't open objset for fatman/jail/margaret
Solaris: WARNING: can't open objset for fatman/jail/margaret
Solaris: WARNING: ZFS replay transaction error 86, dataset
fatman/jail/havnor, seq 0x25442, txtype 9

Solaris: WARNING: ZFS replay transaction error 86, dataset fatman/jail/mail,
seq 0x1e200, txtype 9

Solaris: WARNING: ZFS replay transaction error 86, dataset
fatman/jail/thirdforces, seq 0x732e3, txtype 9

[root@potjie /fatman/jail/mail]# zpool status -v
  pool: fatman
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 0h4m, 0.83% done, 8h21m to go
config:

    NAME           STATE     READ WRITE CKSUM
    fatman         DEGRADED     0     0    34
      raidz2       DEGRADED     0     0   384
        replacing  DEGRADED     0     0     0
          da2/old  REMOVED      0    24     0
          da2      ONLINE       0     0     0  1.71G resilvered
        ad4        ONLINE       0     0     0  21.3M resilvered
        ad6        ONLINE       0     0     0  21.4M resilvered
        ad20       ONLINE       0     0     0  21.3M resilvered
        ad22       ONLINE       0     0     0  21.3M resilvered
        ad17       ONLINE       0     0     0  21.3M resilvered
        da3        ONLINE       0     0     0  21.3M resilvered
        ad10       ONLINE       0     0     1  21.4M resilvered
        ad16       ONLINE       0     0     0  21.2M resilvered
    cache
      ad18         ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        fatman/jail/margaret:<0x0>
        fatman/jail/syndicate:<0x0>
        fatman/jail/mail:<0x0>
        /fatman/jail/mail/tmp
        fatman/jail/havnor:<0x0>
        fatman/jail/thirdforces:<0x0>
        fatman/backup:<0x0>

jail/margaret & backup isn't showing up in zfs list
jail/syndicate is showing up but isn't viewable

It seems the latest content on the better-looking zfs filesystems are quite
recent.

Any thoughts about what is going on ?

I have snapshots for africa on these zfs filesystems - any suggestions on
trying to get them back ?

--
Alex

2009/10/11 Alex Trull <alextzfs@googlemail.com>

> Hi All,
>
> My zraid2 has broken this morning on releng_7 zfs13.
>
> System failed this morning and came back without pool - having lost a disk.
>
> This is how I found the system:
>
>   pool: fatman
>  state: FAULTED
> status: One or more devices could not be used because the label is missing
>     or invalid.  There are insufficient replicas for the pool to continue
>     functioning.
> action: Destroy and re-create the pool from a backup source.
>    see: http://www.sun.com/msg/ZFS-8000-5E
>  scrub: none requested
> config:
>
>     NAME        STATE     READ WRITE CKSUM
>     fatman      FAULTED      0     0     1  corrupted data
>       raidz2    DEGRADED     0     0     6
>         da2     FAULTED      0     0     0  corrupted data
>         ad4     ONLINE       0     0     0
>         ad6     ONLINE       0     0     0
>         ad20    ONLINE       0     0     0
>         ad22    ONLINE       0     0     0
>         ad17    ONLINE       0     0     0
>         da2     ONLINE       0     0     0
>         ad10    ONLINE       0     0     0
>         ad16    ONLINE       0     0     0
>
> Initialy it complained that da3 had gone to da2 (da2 had failed and was no
> longer seen)
>
> I replaced the original da2 and bumped what was originaly da3 back up to
> da3 using the controllers ordering.
>
> [root@potjie /dev]# zpool status
>   pool: fatman
>  state: FAULTED
> status: One or more devices could not be used because the label is missing
>     or invalid.  There are insufficient replicas for the pool to continue
>     functioning.
> action: Destroy and re-create the pool from a backup source.
>    see: http://www.sun.com/msg/ZFS-8000-5E
>  scrub: none requested
> config:
>
>     NAME        STATE     READ WRITE CKSUM
>     fatman      FAULTED      0     0     1  corrupted data
>       raidz2    ONLINE       0     0     6
>         da2     UNAVAIL      0     0     0  corrupted data
>         ad4     ONLINE       0     0     0
>         ad6     ONLINE       0     0     0
>         ad20    ONLINE       0     0     0
>         ad22    ONLINE       0     0     0
>         ad17    ONLINE       0     0     0
>         da3     ONLINE       0     0     0
>         ad10    ONLINE       0     0     0
>         ad16    ONLINE       0     0     0
>
> Issue looks very similar to this (JMR's issue) :
> http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html
>
> I've tried the methods there without much result.
>
> Using JMR's patches/debugs to see what is going on, this is what I got:
>
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
>
> But JMR's patch still doesn't let me import even with a decremented txg
>
> I then had a look around the drives using zdb and some dirty script:
>
> [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo
> "$1";zdb -l "$1" |grep txg"}' | sh
> /dev/ad10
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad16
>     txg=46408223 <- old TXGid ?
>     txg=46408223
>     txg=46408223
>     txg=46408223
> /dev/ad17
>     txg=46408223 <- old TXGid ?
>     txg=46408223
>     txg=46408223
>     txg=46408223
> /dev/ad18 (ssd)
> /dev/ad19 (spare drive, removed from pool some time ago)
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
> /dev/ad20
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad22
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad4
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad6
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/da2 < new drive replaced broken da2
> /dev/da3
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
>
> I did not see any checksums or other issues on ad16 and ad17 previously,
> and I do check regularly.
>
> Any thoughts on what to try next ?
>
> Regards,
>
> Alex
>
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4d98b5320910110927o62f8f588r9acdeb40a19587ea>