From owner-freebsd-geom@FreeBSD.ORG Fri Jan 12 20:29:18 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AF72A16A407 for ; Fri, 12 Jan 2007 20:29:18 +0000 (UTC) (envelope-from cyberleo@cyberleo.net) Received: from pizzabox.cyberleo.net (alpha.cyberleo.net [198.145.45.10]) by mx1.freebsd.org (Postfix) with ESMTP id 9306313C455 for ; Fri, 12 Jan 2007 20:29:18 +0000 (UTC) (envelope-from cyberleo@cyberleo.net) Received: (qmail 36556 invoked from network); 12 Jan 2007 20:29:17 -0000 Received: from adsl-69-212-1-127.dsl.chcgil.ameritech.net (HELO ?172.16.44.14?) (cyberleo@cyberleo.net@69.212.1.127) by alpha.cyberleo.net with ESMTPA; 12 Jan 2007 20:29:17 -0000 Message-ID: <45A7EF74.9050204@cyberleo.net> Date: Fri, 12 Jan 2007 14:28:36 -0600 From: CyberLeo Kitsana User-Agent: Thunderbird 1.5 (X11/20051201) MIME-Version: 1.0 To: "R. B. Riddick" References: <666664.62691.qm@web30309.mail.mud.yahoo.com> In-Reply-To: <666664.62691.qm@web30309.mail.mud.yahoo.com> Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Geom Subject: Re: geom_raid5 livelock? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2007 20:29:18 -0000 R. B. Riddick wrote: > --- CyberLeo Kitsana wrote: >> I've been making use of the geom_raid5 class for FreeBSD 6.2-PRERELEASE. >> I've noticed an odd behavior lately, with the latest sources provided >> (as of 2006-01-10) in which, with 'safeop' enabled, the raid5 will stop >> responding. >> > Ohoh... > >> All works beautifully, until several dozen gigabytes are transferred to >> or from the filesystem with safeop enabled, at which point the >> filesystem will grow quickly less responsive, and eventually cease >> responding entirely (processes stuck in diskwait) CPU usage is at 0%, >> and all four members of the raid5 are being read at around 160kB/sec >> (16kB/t, 10tps) constantly. It does not naturally recover within 72 >> hours. The mirror is unaffected by this behavior. >> > Strange... :-) > SAFEOP means, that a failed disk leads to an IO error for every request, and > that every read request reads all corresponding disk areas (if possible) and > checks parity. SAFEOP mode is surely useful, if u want to be sure, that neither > ur disks nor ur operating system provide bogus data... SAFEOP mode causes a lot > of disk activity (e. g. in case of sequential read, it reads n-1 times (where n > is the disk count of the RAID5) the whole stripe (n blocks)...). > This special form of a read request is used by the rebuild-procedure, so that > it should work fine... > > What does "graid5 list" say in those times? > Are there any special messages logged via syslog in those times? > >> When this occurs, the moment safeop is disabled on the raid5, all the >> problems cease, the filesystem begins responding and the programs resume. >> > So the kernel does not panic or so...? :-) > >> Is this intentional, an artifact of the hardware or layout I'm using, or >> could this be indicative of an obscure bug somewhere? Can I provide any >> additional information which would assist in tracking this down? >> > I would guess: An obscure bug in graid5... > > You could try to put gcache between the disks and graid5... > And the syslog messages (if there r any) would be very interesting (like > messages about read error or disk failure or so)... Hence the testing. I wish to ensure there are as few problems as possible when this is put into production. The kernel does not panic, and everything resumes just fine as soon as safeop is disabled. There are no new messages in the kernel log, nor in syslog, and all disks are operating properly, if slowly (around 40 kilobytes per second each, with dd bs=4096). Also, gcache doesn't seem to exist in the base system. Is this easier to build than gjournal? (patching for that caused a whole ton of kernel and world problems I'd rather not revisit) The attached list was taken during the most recent lockup. ---- [cyberleo@mikayla ~]$ graid5 list Geom name: raid5 State: COMPLETE CALM (safeop) Status: Total=4, Online=4 Type: AUTOMATIC Pending: (wqp 0 // 0) Stripesize: 32768 MemUse: 147456 (msl 7) Newest: -1 ID: 3906282509 Providers: 1. Name: raid5/raid5 Mediasize: 1193822846976 (1.1T) Sectorsize: 512 Mode: r1w1e1 Consumers: 1. Name: ad6s2 Mediasize: 397940981760 (371G) Sectorsize: 512 Mode: r2w2e2 DiskNo: 3 Error: No 2. Name: ad4s2 Mediasize: 397940981760 (371G) Sectorsize: 512 Mode: r2w2e2 DiskNo: 2 Error: No 3. Name: ad2s2 Mediasize: 397940981760 (371G) Sectorsize: 512 Mode: r2w2e2 DiskNo: 1 Error: No 4. Name: ad0s2 Mediasize: 397940981760 (371G) Sectorsize: 512 Mode: r2w2e2 DiskNo: 0 Error: No ---- -- Fuzzy love, -CyberLeo Technical Administrator CyberLeo.Net Webhosting http://www.CyberLeo.Net Furry Peace! - http://www.fur.com/peace/