From owner-freebsd-fs@FreeBSD.ORG Fri Apr 17 14:20:14 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CD9D1065674; Fri, 17 Apr 2009 14:20:14 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from mail.wanderview.com (mail.wanderview.com [66.92.166.102]) by mx1.freebsd.org (Postfix) with ESMTP id A6CCB8FC1C; Fri, 17 Apr 2009 14:20:13 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from harkness.in.wanderview.com (harkness.in.wanderview.com [10.76.10.150]) (authenticated bits=0) by mail.wanderview.com (8.14.3/8.14.3) with ESMTP id n3HE4FBX003074 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 17 Apr 2009 14:04:15 GMT (envelope-from ben@wanderview.com) From: Ben Kelly To: Alexander Leidinger In-Reply-To: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> X-Priority: 3 (Normal) References: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> Message-Id: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Fri, 17 Apr 2009 10:04:15 -0400 X-Mailer: Apple Mail (2.930.3) X-Spam-Score: -1.44 () ALL_TRUSTED X-Scanned-By: MIMEDefang 2.64 on 10.76.20.1 Cc: current@freebsd.org, fs@freebsd.org Subject: Re: ZFS: unlimited arc cache growth? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Apr 2009 14:20:15 -0000 On Apr 17, 2009, at 8:50 AM, Alexander Leidinger wrote: > to fs@, please CC me, as I'm not subscribed. > > I monitored (by hand) a while the sysctls > kstat.zfs.misc.arcstats.size and kstat.zfs.misc.arcstats.hdr_size. > Both grow way higher (at some point I've seen more than 500M) than > what I have configured in vfs.zfs.arc_max (40M). > > After a while FS operations (e.g. pkgdb -F with about 900 > packages... my specific workload is the fixup of gnome packages > after the removal of the obsolete libusb port) get very slow (in my > specific example I let the pkgdb run several times over night and it > still is not finished). > > The big problem with this is, that at some point in time the machine > reboots (panic, page fault, page not present, during a fork1). I > have the impression (beware, I have a watchdog configured, as I > don't know if a triggered WD would cause the same panic, the > following is just a guess) that I run out of memory of some kind (I > have 1G RAM, i386, max kmem size 700M). I restarted pkgdb several > times after a reboot, and it continues to process the libusb > removal, but hey, this is anoying. > > Does someone see something similar to what I describe (mainly the > growth of the arc cache way beyond what is configured)? Anyone with > some ideas what to try? Can you provide the rest of the arcstats from sysctl? Also, does your arc_reclaim_thread process get any cycles when this problem occurs? What happens if you kill the pkgdb -F manually before it completes? Does the arc cache size come back down or is it stuck at the abnormally high level? At first glance it looks like the tunable limits the value of the arc_c target value, but that appears to only be a soft limit. There is code in there to shrink an ARC that has exceeded its arc_c value. It looks like that code is supposed to run from the arc_reclaim_thread. - Ben