Date: Tue, 5 Mar 2013 10:22:52 -0500 From: Gary Palmer <gpalmer@freebsd.org> To: Garrett Wollman <wollman@hergotha.csail.mit.edu> Cc: stable@freebsd.org, killing@multiplay.co.uk Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults? Message-ID: <20130305152252.GA52706@in-addr.com> In-Reply-To: <201303050540.r255ecEC083742@hergotha.csail.mit.edu> References: <513524B2.6020600@denninger.net> <1362449266.92708.8.camel@btw.pki2.com> <51355F64.4040409@denninger.net> <201303050540.r255ecEC083742@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Mar 05, 2013 at 12:40:38AM -0500, Garrett Wollman wrote: > In article <8C68812328E3483BA9786EF15591124D@multiplay.co.uk>, > killing@multiplay.co.uk writes: > > >Now interesting you should say that I've seen a stall recently on ZFS > >only box running on 6 x SSD RAIDZ2. > > > >The stall was caused by fairly large mysql import, with nothing else > >running. > > > >Then it happened I thought the machine had wedged, but minutes (not > >seconds) later, everything sprung into action again. > > I have certainly seen what you might describe as "stalls", caused, so > far as I can tell, by kernel memory starvation. I've seen it take as > much as a half an hour to recover from these (which is too long for my > users). Right now I have the ARC limited to 64 GB (on a 96 GB file > server) and that has made it more stable, but it's still not behaving > quite as I would like, and I'm looking to put more memory into the > system (to be used for non-ARC functions). Looking at my munin > graphs, I find that backups in particular put very heavy pressure on, > doubling the UMA allocations over steady-state, and this takes about > four or five hours to climb back down. See > <http://people.freebsd.org/~wollman/vmstat_z-day.png>; for an example. > > Some of the stalls are undoubtedly caused by internal fragmentation > rather than actual data in use. (Solaris used to have this issue, and > some hooks were added to allow some amount of garbage collection with > the cooperation of the filesystem.) Just as a note that there was a page I read in the past few months that pointed out that having a huge ARC may not always be in the best interests of the system. Some operation on the filesystem (I forget what, apologies) caused the system to churn through the ARC and discard most of it, while regular I/O was blocked Unfortunately I cannot remember where I found that page now and I don't appear to have bookmarked it >From what has been said in this thread I'm not convinced that people are hitting this issue, however I would like to raise it for consideration Regards, Gary
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130305152252.GA52706>