From owner-freebsd-stable@FreeBSD.ORG Wed Jan 16 18:26:25 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65D1716A418; Wed, 16 Jan 2008 18:26:25 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from spork.qfe3.net (spork.qfe3.net [212.13.207.101]) by mx1.freebsd.org (Postfix) with ESMTP id 2096C13C43E; Wed, 16 Jan 2008 18:26:24 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from [81.104.123.28] (helo=voi.aagh.net) by spork.qfe3.net with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1JFCxy-000Fm3-55; Wed, 16 Jan 2008 18:26:22 +0000 Received: from freaky by voi.aagh.net with local (Exim 4.68 (FreeBSD)) (envelope-from ) id 1JFCxx-00029n-Db; Wed, 16 Jan 2008 18:26:21 +0000 Date: Wed, 16 Jan 2008 18:26:21 +0000 From: Thomas Hurst To: Kris Kennaway Message-ID: <20080116182621.GA4260@voi.aagh.net> Mail-Followup-To: Kris Kennaway , John Baldwin , freebsd-stable@freebsd.org, Peter Jeremy References: <20071203054207.GA1153@aleph.niw.com.au> <4787AAF4.1020905@FreeBSD.org> <20080112033147.GX60060@server.vk2pj.dyndns.org> <200801120005.38972.jhb@freebsd.org> <20080115020316.GA47831@voi.aagh.net> <478C7B04.9080102@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <478C7B04.9080102@FreeBSD.org> Organization: Not much. User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Thomas Hurst Cc: Peter Jeremy , freebsd-stable@freebsd.org, John Baldwin Subject: Re: Swapping caused by very large (regular) file size X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2008 18:26:25 -0000 * Kris Kennaway (kris@FreeBSD.org) wrote: >> Excellent. I've been seeing this behavior for a long time, mostly on >> backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd >> seeing a system with 8G of memory, 60% of which is just cache, swap >> out half a dozen things for no apparant reason. And to think, people >> on FreeNode ##freebsd just insisted I had a misconfigured system ;) 7 does indeed seem to resolve this. Also, no sign of corruption on my Marvell 88SX6081, at least during serial read tests. I'll test with a level 0 dump tonight; my writes go via tee to the filesystem and sha1(1) so I should pick up any silent corruption. >> My other IO related issue with 6 is IO to independent amr arrays >> blocking each other; e.g. daily run's find over amrd0 will invariably >> cause reads from amrd1 and amrd2 to freeze for seconds at a time (pr >> 114438). I'll be very interested to see if 7 fixes that. > > Sounds like it might be due to Giant contention, which I think is > indeed fixed in 7. Can you try it? I've updated to test and it seems that no, it's not fixed. Doing a find /usr >/dev/null and a dd if=/dump/bla/bigfile of=/dev/null bs=64k results in normal behavior, followed by a burst of writes and an IO queue depth on the order of 1,000, which freezes reads on all devices on the card while it clears. I'm not sure why I've not noticed the writes before, but they do make for one obvious culprit: atime updates. Presumably by the time the IO's get to amr(4), they're queued per-card, not per-array, so the pile of extra work for one array starves the rest too. Again, a dd from the Marvell ata(4) controller is completely oblivious, and maintains a steady 55MB/s while amrd0's servicing 1,000 write requests; amrd1/2 freeze. noatime would be the obvious solution, but it'd be nice if either the driver could keep the work queues more isolated, or the IO scheduler could avoid letting them starve out everything else. Or the system could emmit a cute fluffy kitten, and a pony. -- Thomas 'Freaky' Hurst http://hur.st/