Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 30 Jan 2012 12:07:40 -0500
From:      Lee Dilkie <lee@dilkie.com>
To:        Miroslav Lachman <000.fbsd@quip.cz>
Cc:        freebsd-geom <freebsd-geom@freebsd.org>
Subject:   Re: gmirror question, drive missing
Message-ID:  <4F26CE5C.20003@dilkie.com>
In-Reply-To: <4F249997.1010502@quip.cz>
References:  <4F24785F.20607@dilkie.com> <4F247D69.6000105@dilkie.com> <4F249997.1010502@quip.cz>

next in thread | previous in thread | raw e-mail | index | archive | help


On 1/28/2012 7:57 PM, Miroslav Lachman wrote:
> Lee Dilkie wrote:
>> additional.
>>
>> like I said, the original setup had the drives swapped. ad10, now ad11,
>> was the source of the failure.
>>
>> from the log files..
>>
>> +ad10: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=1740673583
>> +ad10: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> 
>> error=10<NID_NOT_FOUND>  LBA=1740673583
>> +GEOM_MIRROR: Request failed (error=5).
>> ad10[READ(offset=891224874496, length=4096)]
>> +GEOM_MIRROR: Device gm0: provider ad10 disconnected.
>
> Your problem is exactly the above error - the disk is marked as BROKEN
> by gmirror and thus not connected (re-synchronized) to gm0 anymore.
>
> If you are really sure you want to re-add this broken disk into
> gmirror gm0, you must clear metadata on it, then remove info about
> this provider from gmirror configuration and then insert it in to
> gmirror again.
>
> Example (if ad11 is the broken disk):
> gmirror clear -v ad11
> gmirror forget -v gm0
> gmirror insert -v gm0 ad11
>
> Maybe you can use gmirror remove instead of clear and forget, I am not
> sure.
>
> PS: I recommend you tu check the disk with sysutils/smartmontools:
>
> smartctl -a /dev/ad11

Thanks Miroslav (sorry for the late reply, my home internet connection
went down on the weekend and hasn't recovered).

Yes, I did do a smartctl "long" test on the drive and it came back clean
so I'm not sure it was a drive failure.

It was a very odd failure actually. The system should have continued to
run with the one drive gone but it didn't It stayed up and I was able to
access it using ssh for a while, but then it became clear that the
filesystem had "gone". Running applications couldn't access the
filesystem and eventually even ssh refused connections. Didn't happen
all at once though, first indication was imap complaining that it
couldn't access user mailboxes...

when I got someone to go in and reboot, the server wouldn't come up, the
failed drive, ad10 at the time, had no boot loader available... don't
know what happened to that drive but it was corrupted somehow (still is
in the same state if someone has an idea of what I could look for?).

I swapped drive cables to the other drive from the mirror and it came
right up and has been running fine ever since (I did a manual fsck to
fix the unclean shutdown).


what I *think* happened was some sort of system h/w failure (maybe) that
made both drives not work properly. There's no indication in the logs
but the logs seem to have stopped after the one indicated so I think the
remaining drive was no longer writable. just guessing.

-lee



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F26CE5C.20003>