From owner-freebsd-current@FreeBSD.ORG Sat Apr 18 12:58:38 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE88A1065C57; Sat, 18 Apr 2009 12:58:38 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: from mail-fx0-f167.google.com (mail-fx0-f167.google.com [209.85.220.167]) by mx1.freebsd.org (Postfix) with ESMTP id E20738FC16; Sat, 18 Apr 2009 12:58:37 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: by fxm11 with SMTP id 11so1268416fxm.43 for ; Sat, 18 Apr 2009 05:58:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.55.142 with SMTP id u14mr3348482bkg.121.1240059516812; Sat, 18 Apr 2009 05:58:36 -0700 (PDT) In-Reply-To: <20090418094821.00002e67@unknown> References: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> <20090418094821.00002e67@unknown> Date: Sat, 18 Apr 2009 14:58:36 +0200 Message-ID: From: =?ISO-8859-1?Q?Marius_N=FCnnerich?= To: Alexander Leidinger Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: fs@freebsd.org, current@freebsd.org, Ben Kelly Subject: Re: ZFS: unlimited arc cache growth? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Apr 2009 12:58:49 -0000 On Sat, Apr 18, 2009 at 09:48, Alexander Leidinger wrote: > On Fri, 17 Apr 2009 10:04:15 -0400 Ben Kelly wrote: > > >> On Apr 17, 2009, at 8:50 AM, Alexander Leidinger wrote: >> > to fs@, please CC me, as I'm not subscribed. >> > >> > I monitored (by hand) a while the sysctls >> > kstat.zfs.misc.arcstats.size and kstat.zfs.misc.arcstats.hdr_size. >> > Both grow way higher (at some point I've seen more than 500M) than >> > what I have configured in vfs.zfs.arc_max (40M). >> > >> > After a while FS operations (e.g. pkgdb -F with about 900 >> > packages... my specific workload is the fixup of gnome packages >> > after the removal of the obsolete libusb port) get very slow (in >> > my specific example I let the pkgdb run several times over night >> > and it still is not finished). >> > >> > The big problem with this is, that at some point in time the >> > machine reboots (panic, page fault, page not present, during a >> > fork1). I have the impression (beware, I have a watchdog >> > configured, as I don't know if a triggered WD would cause the same >> > panic, the following is just a guess) that I run out of memory of >> > some kind (I have 1G RAM, i386, max kmem size 700M). I restarted >> > pkgdb several times after a reboot, and it continues to process the >> > libusb removal, but hey, this is anoying. >> > >> > Does someone see something similar to what I describe (mainly the >> > growth of the arc cache way beyond what is configured)? Anyone >> > with some ideas what to try? >> >> Can you provide the rest of the arcstats from sysctl? =A0Also, does >> your arc_reclaim_thread process get any cycles when this problem >> occurs? What happens if you kill the pkgdb -F manually before it >> completes? Does the arc cache size come back down or is it stuck at >> the abnormally high level? > > I haven't tried killing pkgdb and looking at the stats, but on the idle > machine (reboot after the panic and 5h of no use by me... the machine > fetches my mails, has a webmail + mysql + imap interface and is a > fileserver) the size is double of my max value. Again there's no real > load at this time, just fetching my mails (most traffic from the > FreeBSD lists) and a little bit of SpamAssassin filtering of them. When > I logged in this morning the machine was rebooted about 5h ago by a > panic and no FS traffic was going on (100% idle). > > Currently the arc_reclaim_thread has 0:12 of accumulated CPU time, > the wcpu is at 0%, but it is in the running state. The machine is > about 80% idle. > [snip] How about adding a few DTrace probes into arc_reclaim_thread and see what it does?