From owner-freebsd-current@FreeBSD.ORG Sat Apr 18 21:17:04 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77FA0106576E; Sat, 18 Apr 2009 21:17:04 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from mail.wanderview.com (mail.wanderview.com [66.92.166.102]) by mx1.freebsd.org (Postfix) with ESMTP id 001308FC13; Sat, 18 Apr 2009 21:17:03 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from harkness.in.wanderview.com (harkness.in.wanderview.com [10.76.10.150]) (authenticated bits=0) by mail.wanderview.com (8.14.3/8.14.3) with ESMTP id n3ILH0Dk003279 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 18 Apr 2009 21:17:01 GMT (envelope-from ben@wanderview.com) Message-Id: <6535218D-6292-4F84-A8BA-FFA9B2E47F80@wanderview.com> From: Ben Kelly To: Alexander Leidinger In-Reply-To: <20090418094821.00002e67@unknown> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Sat, 18 Apr 2009 17:17:00 -0400 References: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> <20090418094821.00002e67@unknown> X-Mailer: Apple Mail (2.930.3) X-Spam-Score: -1.44 () ALL_TRUSTED X-Scanned-By: MIMEDefang 2.64 on 10.76.20.1 Cc: current@freebsd.org, fs@freebsd.org Subject: Re: ZFS: unlimited arc cache growth? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Apr 2009 21:17:04 -0000 On Apr 18, 2009, at 3:48 AM, Alexander Leidinger wrote: > On Fri, 17 Apr 2009 10:04:15 -0400 Ben Kelly > wrote: > I haven't tried killing pkgdb and looking at the stats, but on the > idle > machine (reboot after the panic and 5h of no use by me... the machine > fetches my mails, has a webmail + mysql + imap interface and is a > fileserver) the size is double of my max value. Again there's no real > load at this time, just fetching my mails (most traffic from the > FreeBSD lists) and a little bit of SpamAssassin filtering of them. > When > I logged in this morning the machine was rebooted about 5h ago by a > panic and no FS traffic was going on (100% idle). From looking at the code, its not too surprising it settles out at 2x your zfs_arc_max tunable. It looks like under normal conditions the arc_reclaim_thread only tries to evict buffers when the arc_size plus any ghost buffers is twice the value of arc_c: if (needfree || (2 * arc_c < arc_size + arc_mru_ghost->arcs_size + arc_mfu_ghost- >arcs_size)) arc_adjust(); (The needfree flag is only set when the system lowmem event is fired.) The arc_reclaim_thread checks this once a second. Perhaps this limit should be a tunable. Also, it might make sense to have a separate limit check for the ghost buffers. I was able to reproduce similar arc_size growth on my machine by running my rsync backup. After instrumenting the code it appeared that buffers were not being evicted because they were "indirect" and had been in the cache less than a second. The "indirect" flag is set based on the on-disk level field. When you see the arcstats.evict_skip sysctl going up this is probably what is happening. The comments in the code say this check is only for prefetch data, but it also triggers for indirect. I'm hesitant to make it really only affect prefetch buffers. Perhaps we could make the timeout a tunable or dynamic based on how far the cache is over its target. After the rsync completed my machine slowly evicts buffers until its back down to about twice arc_c. There was one case, however, where I saw it stop at about four times arc_c. In that case it was failing to evict buffers due to a missed lock. Its not clear yet if it was a buffer lock or hash lock. When this happens you'll see the arcstats.mutex_missed sysctl go up. I'm going to see if I can track down why this is occuring under idle conditions. That seems suspicious to me. Hope that helps. I'll let you know if I find anything else. - Ben