From owner-freebsd-fs@FreeBSD.ORG Wed Sep 15 10:46:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAF8F1065694 for ; Wed, 15 Sep 2010 10:46:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta05.emeryville.ca.mail.comcast.net (qmta05.emeryville.ca.mail.comcast.net [76.96.30.48]) by mx1.freebsd.org (Postfix) with ESMTP id CC1208FC20 for ; Wed, 15 Sep 2010 10:46:37 +0000 (UTC) Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51]) by qmta05.emeryville.ca.mail.comcast.net with comcast id 6ymd1f00216AWCUA5ymdPt; Wed, 15 Sep 2010 10:46:37 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta06.emeryville.ca.mail.comcast.net with comcast id 6ymc1f0013LrwQ28SymctR; Wed, 15 Sep 2010 10:46:37 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id EA4D89B423; Wed, 15 Sep 2010 03:46:35 -0700 (PDT) Date: Wed, 15 Sep 2010 03:46:35 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20100915104635.GA59871@icarus.home.lan> References: <20100908084855.GF2465@deviant.kiev.zoral.com.ua> <4C874F00.3050605@freebsd.org> <4C8D087B.5040404@freebsd.org> <03537796FAB54E02959E2D64FC83004F@multiplay.co.uk> <4C8D280F.3040803@freebsd.org> <3FBF66BF11AA4CBBA6124CA435A4A31B@multiplay.co.uk> <4C8E4212.30000@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, jhell , Pawel Jakub Dawidek , Andriy Gapon Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2010 10:46:38 -0000 On Wed, Sep 15, 2010 at 11:32:18AM +0100, Steven Hartland wrote: > Ok the results are in, the conclusions I can see from the data, others may see > more, are:- > > === Common === > * arc size on boot is ~180M with a target size of 6.5G > > === sendfile on === > * arc size increases on demand but peaks the min value > * The difference between min and max arc is taken up by inactive pages > * vm page daemon wakeups sit at a constant level once the machine has > filled memory and never fully empties swap. > > === sendfile off === > * arc size increases on demand all the way up the the max value > * vm cache count stays at almost zero all the time > > === conclusion === > The interaction of zfs and sendfile is causing large amounts of memory > to end up in the inactive pool and only the use of a hard min arc limit is > ensuring that zfs forces the vm to release said memory so that it can be > used by zfs arc. > > The source data, xls's and exported graphs can be found here:- > http://www.multiplaygameservers.com/dropzone/zfs-sendfile-results.zip Looks like Andriy just committed something to HEAD/CURRENT which might address this: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/141305 Commit: Author: avg Date: Wed Sep 15 10:31:27 2010 New Revision: 212650 URL: http://svn.freebsd.org/changeset/base/212650 -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |