Date: Fri, 12 Jan 2007 14:28:36 -0600 From: CyberLeo Kitsana <cyberleo@cyberleo.net> To: "R. B. Riddick" <arne_woerner@yahoo.com> Cc: FreeBSD Geom <freebsd-geom@freebsd.org> Subject: Re: geom_raid5 livelock? Message-ID: <45A7EF74.9050204@cyberleo.net> In-Reply-To: <666664.62691.qm@web30309.mail.mud.yahoo.com> References: <666664.62691.qm@web30309.mail.mud.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
R. B. Riddick wrote:
> --- CyberLeo Kitsana <cyberleo@cyberleo.net> wrote:
>> I've been making use of the geom_raid5 class for FreeBSD 6.2-PRERELEASE.
>> I've noticed an odd behavior lately, with the latest sources provided
>> (as of 2006-01-10) in which, with 'safeop' enabled, the raid5 will stop
>> responding.
>>
> Ohoh...
>
>> All works beautifully, until several dozen gigabytes are transferred to
>> or from the filesystem with safeop enabled, at which point the
>> filesystem will grow quickly less responsive, and eventually cease
>> responding entirely (processes stuck in diskwait) CPU usage is at 0%,
>> and all four members of the raid5 are being read at around 160kB/sec
>> (16kB/t, 10tps) constantly. It does not naturally recover within 72
>> hours. The mirror is unaffected by this behavior.
>>
> Strange... :-)
> SAFEOP means, that a failed disk leads to an IO error for every request, and
> that every read request reads all corresponding disk areas (if possible) and
> checks parity. SAFEOP mode is surely useful, if u want to be sure, that neither
> ur disks nor ur operating system provide bogus data... SAFEOP mode causes a lot
> of disk activity (e. g. in case of sequential read, it reads n-1 times (where n
> is the disk count of the RAID5) the whole stripe (n blocks)...).
> This special form of a read request is used by the rebuild-procedure, so that
> it should work fine...
>
> What does "graid5 list" say in those times?
> Are there any special messages logged via syslog in those times?
>
>> When this occurs, the moment safeop is disabled on the raid5, all the
>> problems cease, the filesystem begins responding and the programs resume.
>>
> So the kernel does not panic or so...? :-)
>
>> Is this intentional, an artifact of the hardware or layout I'm using, or
>> could this be indicative of an obscure bug somewhere? Can I provide any
>> additional information which would assist in tracking this down?
>>
> I would guess: An obscure bug in graid5...
>
> You could try to put gcache between the disks and graid5...
> And the syslog messages (if there r any) would be very interesting (like
> messages about read error or disk failure or so)...
Hence the testing. I wish to ensure there are as few problems as
possible when this is put into production.
The kernel does not panic, and everything resumes just fine as soon as
safeop is disabled. There are no new messages in the kernel log, nor in
syslog, and all disks are operating properly, if slowly (around 40
kilobytes per second each, with dd bs=4096).
Also, gcache doesn't seem to exist in the base system. Is this easier to
build than gjournal? (patching for that caused a whole ton of kernel and
world problems I'd rather not revisit)
The attached list was taken during the most recent lockup.
----
[cyberleo@mikayla ~]$ graid5 list
Geom name: raid5
State: COMPLETE CALM (safeop)
Status: Total=4, Online=4
Type: AUTOMATIC
Pending: (wqp 0 // 0)
Stripesize: 32768
MemUse: 147456 (msl 7)
Newest: -1
ID: 3906282509
Providers:
1. Name: raid5/raid5
Mediasize: 1193822846976 (1.1T)
Sectorsize: 512
Mode: r1w1e1
Consumers:
1. Name: ad6s2
Mediasize: 397940981760 (371G)
Sectorsize: 512
Mode: r2w2e2
DiskNo: 3
Error: No
2. Name: ad4s2
Mediasize: 397940981760 (371G)
Sectorsize: 512
Mode: r2w2e2
DiskNo: 2
Error: No
3. Name: ad2s2
Mediasize: 397940981760 (371G)
Sectorsize: 512
Mode: r2w2e2
DiskNo: 1
Error: No
4. Name: ad0s2
Mediasize: 397940981760 (371G)
Sectorsize: 512
Mode: r2w2e2
DiskNo: 0
Error: No
----
--
Fuzzy love,
-CyberLeo
Technical Administrator
CyberLeo.Net Webhosting
http://www.CyberLeo.Net
<CyberLeo@CyberLeo.Net>
Furry Peace! - http://www.fur.com/peace/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45A7EF74.9050204>