From owner-freebsd-fs@FreeBSD.ORG Sat Sep 4 14:53:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D828810656A7 for ; Sat, 4 Sep 2010 14:53:34 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7F86F8FC08 for ; Sat, 4 Sep 2010 14:53:34 +0000 (UTC) Received: by gyg4 with SMTP id 4so1386686gyg.13 for ; Sat, 04 Sep 2010 07:53:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type; bh=PuYLOpaKEPRyPKZQeLSEN+72LlVdCVS3mGfWvDPnSJM=; b=qZm3ayCPJ7Y+578/d7UBwHXx4BZqvMPDLQ15LwLabtLXTqi3wToTmRWUpBaOmyEA8R FN6NepnzcOAxuAMmJeseLPdE+pbL1Ym0nIescTQaxJgR2y9R3nBiczUjDQdSpVOYqNZz X47xDLk7/LypwhiHGZSUIW0RKC5B/N7cnM3UE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type; b=rcK8yx4TYBeYHp6WuHUeXl3tDqtstQ/sUVkvqRvGhlMwS7JKnLdmW5Pq9OfeutCzoy YnmxrYSyiYUMo8TQR0FY7S279NCpDgXXrSybMVj5eH/R123gJDdxrwgkQz5iKgIJigcs y8XCppzIgm3GvzBdHmRiBaDjSOQXOEOIgaRYA= Received: by 10.151.102.21 with SMTP id e21mr389636ybm.103.1283612008315; Sat, 04 Sep 2010 07:53:28 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id q25sm3719120ybk.6.2010.09.04.07.53.26 (version=SSLv3 cipher=RC4-MD5); Sat, 04 Sep 2010 07:53:27 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C825D65.3040004@DataIX.net> Date: Sat, 04 Sep 2010 10:53:25 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Artem Belevich References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: multipart/mixed; boundary="------------000703040900060801010300" Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Sep 2010 14:53:34 -0000 This is a multi-part message in MIME format. --------------000703040900060801010300 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/03/2010 21:09, Artem Belevich wrote: >> For reference top shows:- >> Mem: 42M Active, 3129M Inact, 565M Wired, 3188K Cache, 19M Buf, 203M Free > > Here's your problem -- inactive list got all your memory and starved ZFS ARC. > Easy workaround is to set vfs.zfs.arc_min to a value that would > guarantee that ARC does not give up too much memory. Let's say - 2GB. > Be warned that it would effectively make those 2GB unavailable to > applications. > > Long term, though, there were number of patches posted on > freebsd-current and freebsd-hackers recently that do improve that > particular issue with ZFS. This patch in particular may help you: > http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html > >> Swap: 4096M Total, 1180K Used, 4095M Free >> >> So what are we missing and how to we get zfs to perform like ufs >> and use all free ram as cache? > Attached is the needfree patch mentioned in the URL alongside a local system patch to adjust kern.maxusers to no more than 512 on systems that can support it. Modern systems would always raise to a value of 512 or more so I do not see a need to keep it limited to 384. This setting also effects the outcome of other values in the system. Not saying that these will directly help your case in anyway shape or form but if you would like to give them a shot then they are here for the taking. Apply with: cd /usr/src patch -p1 date: Fri Sep 03 12:50:37 2010 -0400 files: sys/kern/subr_param.c description: Increase maxusers to 512 by default if it equals 512 or higher. diff -r 627cf9b49e0f -r deccbf75a766 sys/kern/subr_param.c --- a/sys/kern/subr_param.c Fri Sep 03 12:42:00 2010 -0400 +++ b/sys/kern/subr_param.c Fri Sep 03 12:50:37 2010 -0400 @@ -257,8 +257,8 @@ maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE); if (maxusers < 32) maxusers = 32; - if (maxusers > 384) - maxusers = 384; + if (maxusers > 512) + maxusers = 512; } /* --------------000703040900060801010300 Content-Type: text/plain; name="needfree.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="needfree.patch" changeset: 188:dafc39874132 user: J. Hellenthal date: Sat Sep 04 07:34:28 2010 -0400 files: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c description: PFC: r211762 arc.c needfree patch ------------------------------------------------------------------------ r211762 | avg | 2010-08-24 13:48:22 -0400 (Tue, 24 Aug 2010) | 20 lines zfs arc_reclaim_thread: no need to call arc_reclaim_needed when resetting needfree needfree is checked at the very start of arc_reclaim_needed. This change makes code easier to follow and maintain in face of potential changed in arc_reclaim_needed. Also, put the whole sub-block under _KERNEL because needfree can be set only in kernel code. To do: rename needfree to something else to aovid confusion with OpenSolaris global variable of the same name which is used in the same code, but has different meaning (page deficit). Note: I have an impression that locking around accesses to this variable as well as mutual notifications between arc_reclaim_thread and arc_lowmem are not proper. MFC after: 1 week ------------------------------------------------------------------------ diff -r 34a823aff8f6 -r dafc39874132 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Sep 04 07:23:09 2010 -0400 +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Sat Sep 04 07:34:28 2010 -0400 @@ -2317,12 +2317,12 @@ if (arc_eviction_list != NULL) arc_do_user_evicts(); - if (arc_reclaim_needed()) { +#ifdef _KERNEL + if (needfree) { needfree = 0; -#ifdef _KERNEL wakeup(&needfree); + } #endif - } /* block until needed, or one second, whichever is shorter */ CALLB_CPR_SAFE_BEGIN(&cpr); --------------000703040900060801010300 Content-Type: application/octet-stream; name="maxusers.patch.sig" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="maxusers.patch.sig" iQEcBAABAgAGBQJMgl1kAAoJEJBXh4mJ2FR+ztMIAJL+5MwxxHsSRmrTjt3RCtWvay7uRhnC 9HpEPJkzPNRzIwFx4ckMIuH5tMZTXHlRf6Xt0dTIYf28ExQFQyefVarMuFeK1XfESISt5PEm wDLPITr9AnEACE7IsPEpRZV0kuiPyThDybF+2eh682rPeFkHhvcNCFEkO2jpljXYdTXGbyc+ Csw2dIaMd5fFCWtXwNSCah7DoA0gsnTKRgcFJgan0/GMDLGd1N4QWTlqX0buyOGxCSDhrHZK 6vJRoMmDpJz7RzXNigiayzN8y3PgU6P2JbbHGws8edmQqWpbXHP35wmT6P5/+LhwbyplNu0U S/1U8IHFtRxTmxSL0/wGFjo= --------------000703040900060801010300 Content-Type: application/octet-stream; name="needfree.patch.sig" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="needfree.patch.sig" iQEcBAABAgAGBQJMgl1kAAoJEJBXh4mJ2FR+5ggIAJ41d2boBSnDtIXfrrWQyzvIaTV45naA Es5n94Vcx5DBkbQo915SzhOPko0l26Lkr11+oUwRpi3Cwf5ik1PaVS0CNFjZ+eCWXRBay55V AlnthIGC9K3/EbRrnuPMlKpFWbVptAOBSIRnaZVo2LU13TkkBNqFlg6AkBijcT0SCFeRVCOm xPDP/qRvOj8OEBXsbSfqC4YQajBXRy5nl7mFmK1vPfcwURG7/LUhV/jpUaCbj5QW8S58S7Zq /LxQ45DgRu5P3SK3IldeC+KJFgfxYW52zFQ9bavEuzake18OHbEFs6Qj6YiAcR0cnzA0/w9p wM6awh9TPOZ6XYaFaMgs3Ds= --------------000703040900060801010300--