Date: Sat, 11 May 2013 07:16:45 -0400 From: Outback Dingo <outbackdingo@gmail.com> To: Ronald Klop <ronald-freebsd8@klop.yi.org> Cc: freebsd-fs@freebsd.org Subject: Re: Corrupted zpool import -f FAILS state FAULTED Message-ID: <CAKYr3zxatz7Cr-CSMbcA8yHDPdxJBXUsPRjpMc8zq3-s9da2dA@mail.gmail.com> In-Reply-To: <op.wwwvwppi8527sy@ronaldradial> References: <CAKYr3zz1gLZArACqdrzkr6APVMvom6y-80omghoo4nb1KMTrKA@mail.gmail.com> <518CFA05.6090706@gmail.com> <CAKYr3zzP%2BrVin5dAk38C=2HfGym_b3v6je_m-HUtL3eet2fxig@mail.gmail.com> <op.wwwvwppi8527sy@ronaldradial>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, May 11, 2013 at 4:44 AM, Ronald Klop <ronald-freebsd8@klop.yi.org>wrote:
> On Fri, 10 May 2013 16:07:36 +0200, Outback Dingo <outbackdingo@gmail.com>
> wrote:
>
> On Fri, May 10, 2013 at 9:45 AM, Volodymyr Kostyrko <c.kworr@gmail.com
>> >wrote:
>>
>> 09.05.2013 15:31, Outback Dingo:
>>>
>>> ok zfsgurus, FreeBSD 9.1-STABLE box zpool import -f reports pool status
>>>
>>>> Faulted, one of more devices contains corrupted data, however its
>>>> showing
>>>> the guid as faulted in the poll, and not the actual disk device
>>>> /dev/daX,
>>>> the pool is a single vdev 24 disk raidz3. Essentially the hardward
>>>> platform
>>>> is a dual node system, with 8 enclosures connected to 24 SAS drives via
>>>> 4
>>>> LSI cards. I am not currently using geom_multipath, but the box is zoned
>>>> so
>>>> that each node can see 50% of the drives,
>>>> in case of Failure, carp kicks in and migrates "zpool import -af" the
>>>> pools
>>>> onto the other node. it seems as though somehow the pool is now seeing
>>>> guid
>>>> and not devices, not sure if they have switched devices ids due to a
>>>> reboot.
>>>>
>>>>
>>> Am not a zfs guru, but I'll try to help.
>>>
>>> Any console log snippets are welcome. What does "showing the guid as
>>> faulted in the pool" looks like.
>>>
>>> What are the guids for all partitions? Do they interlap for different
>>> nodes?
>>>
>>> ZFS recognizes devices by tasting they vdev labels and not by their
>>> logical location and naming. It can safely report any vdev location - but
>>> it requires the same set vdevs to bring pool online.
>>>
>>
>>
>> zdb shows valid data on the drives, no drives have been removed from the
>> box
>> whats confusing is why its using guids and not devices daX is what puzzles
>> me
>> camcontrol devlist and dmesg clearly show the devices are there. The SAS
>> bus is shared
>> so both nodes with 2 LSI controllers each see all drives. We were
>> utilizing
>> a failover script
>> if nodeA dies, carp would kick the script to import the pool to nodeB,
>> both
>> nodes are in the
>> same chassis and see all the enclosures and all the drives
>>
>
> Are the machines configured the same? As in _exactly_ the same. Glabel
> modules, hint files, sysctls, etc.
>
>
yes, both nodes are identical, from sysctl.conf to loader.conf, ive also
noticed that playing around with enclosure zoning on the system i can now
see which strikes me as quite odd..... now im wondering if i have a
controller flaking out. right now according to the zoning, gmultipath
should see 24+ LUNS however it sees nothing.
zpool import -f
pool: backup
id: 8548776274175948174
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: http://illumos.org/msg/ZFS-8000-EY
config:
backup UNAVAIL insufficient replicas
raidz3-0 UNAVAIL insufficient replicas
da32 ONLINE
da30 ONLINE
da29 ONLINE
da3 ONLINE
da4 ONLINE
da5 ONLINE
da6 ONLINE
da7 ONLINE
da8 ONLINE
label/big4 ONLINE
18084052867377310822 UNAVAIL cannot open
2641768775090614171 UNAVAIL cannot open
8083525846528480855 UNAVAIL cannot open
8200855950201180014 UNAVAIL cannot open
da37 ONLINE
da11 ONLINE
4678398398699137944 UNAVAIL cannot open
18315550984013241979 UNAVAIL cannot open
da22 ONLINE
da23 ONLINE
label/backup ONLINE
da25 ONLINE
da26 ONLINE
da27 ONLINE
> Ronald.
>
> ______________________________**_________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-fs<http://lists.freebsd.org/mailman/listinfo/freebsd-fs>;
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org<freebsd-fs-unsubscribe@freebsd.org>
> "
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKYr3zxatz7Cr-CSMbcA8yHDPdxJBXUsPRjpMc8zq3-s9da2dA>