Date: Fri, 8 Jan 2010 15:05:06 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: freebsd-stable@freebsd.org Subject: Re: ZFS performance degradation over time Message-ID: <20100108230506.GA7501@icarus.home.lan> In-Reply-To: <a1e355586b0e697490159a4733f94988.squirrel@www.noacks.org> References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <hi2nsf$do5$1@ger.gmane.org> <7346c5c61001080831w375d158fu5b1996ee58cb0f8d@mail.gmail.com> <a1e355586b0e697490159a4733f94988.squirrel@www.noacks.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jan 08, 2010 at 04:23:03PM -0500, Jonathan Noack wrote: > On Fri, January 8, 2010 11:31, Garrett Moore wrote: > > No, I haven't isolated the cause to only be uptime related. In my original > > email I mentioned that "as suggested by someone in the thread, it's > > probably > > not directly related to system uptime, but instead related to usage - the > > more usage, the worse the performance." > > > > I've been starting my system with different combinations of applications > > running to see what access patterns cause the most slowdown. So far, I > > don't > > have enough data to give anything concrete. > > > > This weekend I'll try some tests such as the one you describe, and see > > what > > happens. I have a strong suspicion that rTorrent is to blame, since I > > haven't seen major slowdowns in the last few days with rTorrent not > > running. > > rTorrent preallocates the space needed for the file download (and I'm > > downloading large 4GB+ files using it), and then writes to them in an > > unpredictable pattern, so maybe ZFS doesn't like being touched this way? > > Probably unrelated, but this prefetch issue results in a slowdown: > http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007481.html With the introduction of 8.0-RC1, FreeBSD explicitly disables prefetch if your system has less than 4GB usable[1] memory. The code does print a warning/FYI message when this condition is met[2]. You can disable the warning message by setting prefetch_disable="1" in loader.conf (and this naturally means prefetch is disabled). You can also explicitly enable prefetch by setting the value to "1", and this trumps the how-much-usable-RAM check. The OP doesn't disclose if he's enabled prefetch, so I assume he hasn't, which means he shouldn't be susceptible to the bug described above. All that said -- I know what the OP is referring to, as I've seen it myself (on RELENG_7, and possibly early releases of 8.0). The only way to relieve the pain, AFAIK, is to reboot. I do see some MFC's done about 13 hours ago to RELENG_7 and RELENG_8 that talk about the ARC and "paging pressure", which to me means decreased performance when it occurs... or maybe it helps with the kmem exhaustion problem? The brief description in the commit is simply not enough to suffice; it's almost like we need a "FreeBSD ZFS Newsletter" that documents what all the changes are that get committed, what they fix, and what's being worked on/tested in HEAD (for potential MFC). It would also be good to get some concise documentation with regards to what all the kstat.zfs.misc.arcstats.* counters mean and represent; admins like myself would love to track/graph these for helping correlate performance or stability issues, but we don't even know what they represent. For example, just earlier this week I read a semi-recent message talking about how a large kstat.zfs.misc.arcstats.evict_skip counter indicates something bad, yet I have no idea what "evict skip" means in reference to the ZFS core/model itself. [1]: Because of #2 (see above/below), I had to analyse the code. My system was amd64 + 4GB RAM yet ZFS was telling me the system did not have 4GB of RAM thus had disabled prefetch. My analysis is near the bottom of my blog post: http://koitsu.wordpress.com/2009/10/12/testing-out-freebsd-8-0-rc1/ [2]: The grammar of the message printed has gone through numerous revisions (I forget the exact number, but *at least* 2), and the most recent one is still too ambiguous, resulting in reader confusion. In my above blog post, I propose what the kernel message should read (in my opinion), which is significantly less ambiguous and hopefully won't cause "but I do have 4GB!" confusion. [3]: src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c http://freshbsd.org/2010/01/08/09/59/13 -- RELENG_8 http://freshbsd.org/2010/01/08/11/06/13 -- RELENG_7 It sure would be useful if these source files had $FreeBSD$ ID tags in them, rather than having to grep /var/db/sup. I swore there was some command which did this, but I might be thinking of object files. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100108230506.GA7501>