From owner-freebsd-stable@FreeBSD.ORG Sun Jul 11 20:48:00 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 214A41065674 for ; Sun, 11 Jul 2010 20:48:00 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id C4C6D8FC17 for ; Sun, 11 Jul 2010 20:47:59 +0000 (UTC) Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12]) by qmta02.westchester.pa.mail.comcast.net with comcast id gkVc1e0030Fqzac52knzdw; Sun, 11 Jul 2010 20:47:59 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta08.westchester.pa.mail.comcast.net with comcast id gkny1e0073LrwQ23Uknz7r; Sun, 11 Jul 2010 20:47:59 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 736629B425; Sun, 11 Jul 2010 13:47:57 -0700 (PDT) Date: Sun, 11 Jul 2010 13:47:57 -0700 From: Jeremy Chadwick To: Richard Lee Message-ID: <20100711204757.GA81084@icarus.home.lan> References: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2010 20:48:00 -0000 On Sun, Jul 11, 2010 at 11:25:12AM -0700, Richard Lee wrote: > This is on clean FreeBSD 8.1 RC2, amd64, with 4GB memory. > > The closest I found by Googling was this: > http://forums.freebsd.org/showthread.php?t=9935 > > And it talks about all kinds of little tweaks, but in the end, the > only thing that actually works is the stupid 1-line perl code that > forces the kernal to free the memory allocated to (non-zfs) disk > cache, which is the "Inact"ive memory in "top." > > I have a 4-disk raidz pool, but that's unlikely to matter. > > Try to copy large files from non-zfs disk to zfs disk. FreeBSD will > cache the data read from non-zfs disk in memory, and free memory will > go down. This is as expected, obviously. > > Once there's very little free memory, one would expect whatever is > more important to kick out the cached data (Inact) and make memory > available. > > But when almost all of the memory is taken by disk cache (of non-zfs > file system), ZFS disks start threshing like mad and the write > throughput goes down in 1-digit MB/second. > > I believe it should be extremely easy to duplicate. Just plug in a > big USB drive formatted in UFS (msdosfs will likely do the same), and > copy large files from that USB drive to zfs pool. > > Right after clean boot, gstat will show something like 20+MB/s > movement from USB device (da*), and occasional bursts of activity on > zpool devices at very high rate. Once free memory is exhausted, zpool > devices will change to constant low-speed activity, with disks > threshing about constantly. > > I tried enabling/disabling prefetch, messing with vnode counts, > zfs.vdev.min/max_pending, etc. The only thing that works is that > stupid perl 1-liner (perl -e '$x="x"x1500000000'), which returns the > activity to that seen right after a clean boot. It doesn't last very > long, though, as the disk cache again consumes all the memory. > > Copying files between zfs devices doesn't seem to affect anything. > > I understand zfs subsystem has its own memory/cache management. > Can a zfs expert please comment on this? > > And is there a way to force the kernel to not cache non-zfs disk data? I believe you may be describing two separate issues: 1) ZFS using a lot of memory but not freeing it as you expect 2) Lack of disk I/O scheduler For (1), try this in /boot/loader.conf and reboot: # Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA # on 2010/05/24. # http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html vfs.zfs.zio.use_uma="0" For (2), may try gsched_rr: http://svnweb.freebsd.org/viewvc/base/releng/8.1/sys/geom/sched/README?view=markup -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |