From owner-freebsd-stable@FreeBSD.ORG Sun Jul 11 21:02:12 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA050106564A for ; Sun, 11 Jul 2010 21:02:12 +0000 (UTC) (envelope-from ssanbeg@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 78AFB8FC19 for ; Sun, 11 Jul 2010 21:02:12 +0000 (UTC) Received: by pwj9 with SMTP id 9so1721439pwj.13 for ; Sun, 11 Jul 2010 14:02:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:references :in-reply-to:subject:date:message-id:mime-version:content-type :content-transfer-encoding:x-mailer:x-mimeole:thread-index; bh=nKg7bzq89P4P6z8SfaUF/il0uY8TgRgf6Al+64sAcQY=; b=fJzsfLNlSeX+mpc9Q8MWR3aYqDfZomz7cBcCHfqStQdTebA9acngEYtXZx9HqNTJlD HnVVruuVRYsruiV2/lt0HTy/2VFJuY2ZtFBybtUnI5dxOcOEybkqPt7utNSmwZP1tMoB rE20CQCtdNotuKPFwSYhWmBHLsVuPm/5m7oZg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-type:content-transfer-encoding:x-mailer:x-mimeole :thread-index; b=NJQsfiLXCmvrrYbEOsQ9SE4+XwBIxXVaAhkiZtgsUoBpZrImvpF7oBQUJZ7Izux7W7 oYtp7OpayRBITeUsLHKY44XEtE7u0gWom2miPnY7EEl51fkcPI0uPWIxIm1lazig2aqd iWA7UG/0r16hjVPOgKMZz2hqPZnVsBIuS0sA4= Received: by 10.114.171.3 with SMTP id t3mr5920069wae.196.1278882131750; Sun, 11 Jul 2010 14:02:11 -0700 (PDT) Received: from sunrise (pool-98-117-108-60.sttlwa.fios.verizon.net [98.117.108.60]) by mx.google.com with ESMTPS id d39sm56074930wam.16.2010.07.11.14.02.10 (version=SSLv3 cipher=RC4-MD5); Sun, 11 Jul 2010 14:02:10 -0700 (PDT) From: "Scott Sanbeg" To: References: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> <20100711204757.GA81084@icarus.home.lan> In-Reply-To: <20100711204757.GA81084@icarus.home.lan> Date: Sun, 11 Jul 2010 14:02:03 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6002.18197 Thread-Index: AcshOseOnXpcAhRvR7uDLVQkhOX1OgAATHag Subject: RE: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2010 21:02:12 -0000 Using Jeremy's suggestion as follows: 1) ZFS using a lot of memory but not freeing it as you expect For (1), try this in /boot/loader.conf and reboot: vfs.zfs.zio.use_uma="0" ... works like a charm for me. Thank you. Scott -----Original Message----- From: owner-freebsd-stable@freebsd.org [mailto:owner-freebsd-stable@freebsd.org] On Behalf Of Jeremy Chadwick Sent: Sunday, July 11, 2010 1:48 PM To: Richard Lee Cc: freebsd-stable@freebsd.org Subject: Re: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). On Sun, Jul 11, 2010 at 11:25:12AM -0700, Richard Lee wrote: > This is on clean FreeBSD 8.1 RC2, amd64, with 4GB memory. > > The closest I found by Googling was this: > http://forums.freebsd.org/showthread.php?t=9935 > > And it talks about all kinds of little tweaks, but in the end, the > only thing that actually works is the stupid 1-line perl code that > forces the kernal to free the memory allocated to (non-zfs) disk > cache, which is the "Inact"ive memory in "top." > > I have a 4-disk raidz pool, but that's unlikely to matter. > > Try to copy large files from non-zfs disk to zfs disk. FreeBSD will > cache the data read from non-zfs disk in memory, and free memory will > go down. This is as expected, obviously. > > Once there's very little free memory, one would expect whatever is > more important to kick out the cached data (Inact) and make memory > available. > > But when almost all of the memory is taken by disk cache (of non-zfs > file system), ZFS disks start threshing like mad and the write > throughput goes down in 1-digit MB/second. > > I believe it should be extremely easy to duplicate. Just plug in a > big USB drive formatted in UFS (msdosfs will likely do the same), and > copy large files from that USB drive to zfs pool. > > Right after clean boot, gstat will show something like 20+MB/s > movement from USB device (da*), and occasional bursts of activity on > zpool devices at very high rate. Once free memory is exhausted, zpool > devices will change to constant low-speed activity, with disks > threshing about constantly. > > I tried enabling/disabling prefetch, messing with vnode counts, > zfs.vdev.min/max_pending, etc. The only thing that works is that > stupid perl 1-liner (perl -e '$x="x"x1500000000'), which returns the > activity to that seen right after a clean boot. It doesn't last very > long, though, as the disk cache again consumes all the memory. > > Copying files between zfs devices doesn't seem to affect anything. > > I understand zfs subsystem has its own memory/cache management. > Can a zfs expert please comment on this? > > And is there a way to force the kernel to not cache non-zfs disk data? I believe you may be describing two separate issues: 1) ZFS using a lot of memory but not freeing it as you expect 2) Lack of disk I/O scheduler For (1), try this in /boot/loader.conf and reboot: # Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA # on 2010/05/24. # http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html vfs.zfs.zio.use_uma="0" For (2), may try gsched_rr: http://svnweb.freebsd.org/viewvc/base/releng/8.1/sys/geom/sched/README?view= markup -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"