Date: Wed, 16 Jan 2008 18:26:21 +0000 From: Thomas Hurst <tom.hurst@clara.net> To: Kris Kennaway <kris@FreeBSD.org> Cc: Peter Jeremy <peterjeremy@optushome.com.au>, freebsd-stable@freebsd.org, John Baldwin <jhb@freebsd.org> Subject: Re: Swapping caused by very large (regular) file size Message-ID: <20080116182621.GA4260@voi.aagh.net> In-Reply-To: <478C7B04.9080102@FreeBSD.org> References: <20071203054207.GA1153@aleph.niw.com.au> <4787AAF4.1020905@FreeBSD.org> <20080112033147.GX60060@server.vk2pj.dyndns.org> <200801120005.38972.jhb@freebsd.org> <20080115020316.GA47831@voi.aagh.net> <478C7B04.9080102@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
* Kris Kennaway (kris@FreeBSD.org) wrote:
>> Excellent. I've been seeing this behavior for a long time, mostly on
>> backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd
>> seeing a system with 8G of memory, 60% of which is just cache, swap
>> out half a dozen things for no apparant reason. And to think, people
>> on FreeNode ##freebsd just insisted I had a misconfigured system ;)
7 does indeed seem to resolve this. Also, no sign of corruption on my
Marvell 88SX6081, at least during serial read tests. I'll test with
a level 0 dump tonight; my writes go via tee to the filesystem and
sha1(1) so I should pick up any silent corruption.
>> My other IO related issue with 6 is IO to independent amr arrays
>> blocking each other; e.g. daily run's find over amrd0 will invariably
>> cause reads from amrd1 and amrd2 to freeze for seconds at a time (pr
>> 114438). I'll be very interested to see if 7 fixes that.
>
> Sounds like it might be due to Giant contention, which I think is
> indeed fixed in 7. Can you try it?
I've updated to test and it seems that no, it's not fixed. Doing a
find /usr >/dev/null and a dd if=/dump/bla/bigfile of=/dev/null bs=64k
results in normal behavior, followed by a burst of writes and an IO
queue depth on the order of 1,000, which freezes reads on all devices on
the card while it clears.
I'm not sure why I've not noticed the writes before, but they do make
for one obvious culprit: atime updates. Presumably by the time the IO's
get to amr(4), they're queued per-card, not per-array, so the pile of
extra work for one array starves the rest too.
Again, a dd from the Marvell ata(4) controller is completely oblivious,
and maintains a steady 55MB/s while amrd0's servicing 1,000 write
requests; amrd1/2 freeze.
noatime would be the obvious solution, but it'd be nice if either the
driver could keep the work queues more isolated, or the IO scheduler
could avoid letting them starve out everything else. Or the system
could emmit a cute fluffy kitten, and a pony.
--
Thomas 'Freaky' Hurst
http://hur.st/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080116182621.GA4260>