Date: Sun, 11 Jul 2010 14:02:03 -0700 From: "Scott Sanbeg" <ssanbeg@gmail.com> To: <freebsd-stable@freebsd.org> Subject: RE: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). Message-ID: <BAFE9F365EE945E0AF9402EFF07EE67C@sunrise> In-Reply-To: <20100711204757.GA81084@icarus.home.lan> References: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> <20100711204757.GA81084@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
Using Jeremy's suggestion as follows: 1) ZFS using a lot of memory but not freeing it as you expect For (1), try this in /boot/loader.conf and reboot: vfs.zfs.zio.use_uma="0" ... works like a charm for me. Thank you. Scott -----Original Message----- From: owner-freebsd-stable@freebsd.org [mailto:owner-freebsd-stable@freebsd.org] On Behalf Of Jeremy Chadwick Sent: Sunday, July 11, 2010 1:48 PM To: Richard Lee Cc: freebsd-stable@freebsd.org Subject: Re: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). On Sun, Jul 11, 2010 at 11:25:12AM -0700, Richard Lee wrote: > This is on clean FreeBSD 8.1 RC2, amd64, with 4GB memory. > > The closest I found by Googling was this: > http://forums.freebsd.org/showthread.php?t=9935 > > And it talks about all kinds of little tweaks, but in the end, the > only thing that actually works is the stupid 1-line perl code that > forces the kernal to free the memory allocated to (non-zfs) disk > cache, which is the "Inact"ive memory in "top." > > I have a 4-disk raidz pool, but that's unlikely to matter. > > Try to copy large files from non-zfs disk to zfs disk. FreeBSD will > cache the data read from non-zfs disk in memory, and free memory will > go down. This is as expected, obviously. > > Once there's very little free memory, one would expect whatever is > more important to kick out the cached data (Inact) and make memory > available. > > But when almost all of the memory is taken by disk cache (of non-zfs > file system), ZFS disks start threshing like mad and the write > throughput goes down in 1-digit MB/second. > > I believe it should be extremely easy to duplicate. Just plug in a > big USB drive formatted in UFS (msdosfs will likely do the same), and > copy large files from that USB drive to zfs pool. > > Right after clean boot, gstat will show something like 20+MB/s > movement from USB device (da*), and occasional bursts of activity on > zpool devices at very high rate. Once free memory is exhausted, zpool > devices will change to constant low-speed activity, with disks > threshing about constantly. > > I tried enabling/disabling prefetch, messing with vnode counts, > zfs.vdev.min/max_pending, etc. The only thing that works is that > stupid perl 1-liner (perl -e '$x="x"x1500000000'), which returns the > activity to that seen right after a clean boot. It doesn't last very > long, though, as the disk cache again consumes all the memory. > > Copying files between zfs devices doesn't seem to affect anything. > > I understand zfs subsystem has its own memory/cache management. > Can a zfs expert please comment on this? > > And is there a way to force the kernel to not cache non-zfs disk data? I believe you may be describing two separate issues: 1) ZFS using a lot of memory but not freeing it as you expect 2) Lack of disk I/O scheduler For (1), try this in /boot/loader.conf and reboot: # Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA # on 2010/05/24. # http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html vfs.zfs.zio.use_uma="0" For (2), may try gsched_rr: http://svnweb.freebsd.org/viewvc/base/releng/8.1/sys/geom/sched/README?view= markup -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BAFE9F365EE945E0AF9402EFF07EE67C>