Date: Tue, 01 Mar 2005 00:27:05 +0100 From: Uwe Doering <gemini@geminix.org> To: Don Bowman <don@SANDVINE.com> Cc: freebsd-stable@freebsd.org Subject: Re: Adaptec 3210S, 4.9-STABLE, corruption when disk fails Message-ID: <4223A8C9.5060702@geminix.org> In-Reply-To: <2BCEB9A37A4D354AA276774EE13FB8C224D34D@mailserver.sandvine.com> References: <2BCEB9A37A4D354AA276774EE13FB8C224D34D@mailserver.sandvine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Don Bowman wrote:
> I have a machine running:
>
> $ uname -a
> FreeBSD machine.phaedrus.sandvine.com 4.9-STABLE FreeBSD 4.9-STABLE #0:
> Fri Mar 19 10:39:07 EST 2004
> user@machine.phaedrus.sandvine.com:/usr/src/sys/compile/LABDB i386
>
> It has an adaptec 3210S raid controller running a single raid-5, and
> runs postgresql 7.4.6 as its primary application.
>
> 3 times now I have had a drive fail, and have had corrupted files in the
> postgresql cluster @ the same time.
>
> The time is too closely correlated to be a coincidence. It passes fsck @
> the time that I got to it a couple of hours later, and the filesystem
> seems to be ok (with a failed drive, the raid in 'degrade' mode).
>
> It appears that the drive failure and the postgresql failure occur @
> exactly the same time (monitoring with nagios, within 1hr accuracy). It
> would appear that for some file(s) bad data was returned.
>
> Does anyone have any suggestions?
In my experience, in a situation like this RAID controllers can block
the system for up to a couple of minutes, trying to revive a failed disk
drive by sending it bus reset commands and the like, until they
eventually give up and drop into degraded mode. With sufficiently
patient applications this is no problem, but if a program runs into
internal timeouts during this period of time bad things can happen.
My point is that while the disk controller may trigger the problem the
instance that actually corrupts the database might be PostgreSQL itself.
Of course, I'm aware that it's going to be quite hard to tell for sure
who the culprit is.
Uwe
--
Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers
gemini@geminix.org | http://www.escapebox.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4223A8C9.5060702>