Date: Sat, 18 Apr 2009 14:58:36 +0200 From: =?ISO-8859-1?Q?Marius_N=FCnnerich?= <marius@nuenneri.ch> To: Alexander Leidinger <Alexander@leidinger.net> Cc: fs@freebsd.org, current@freebsd.org, Ben Kelly <ben@wanderview.com> Subject: Re: ZFS: unlimited arc cache growth? Message-ID: <b649e5e0904180558u4e94b9e6p254067a3a53db097@mail.gmail.com> In-Reply-To: <20090418094821.00002e67@unknown> References: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> <D2B1B82F-AFCF-4161-BB9E-316EC976E360@wanderview.com> <20090418094821.00002e67@unknown>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Apr 18, 2009 at 09:48, Alexander Leidinger <Alexander@leidinger.net> wrote: > On Fri, 17 Apr 2009 10:04:15 -0400 Ben Kelly <ben@wanderview.com> wrote: > > >> On Apr 17, 2009, at 8:50 AM, Alexander Leidinger wrote: >> > to fs@, please CC me, as I'm not subscribed. >> > >> > I monitored (by hand) a while the sysctls >> > kstat.zfs.misc.arcstats.size and kstat.zfs.misc.arcstats.hdr_size. >> > Both grow way higher (at some point I've seen more than 500M) than >> > what I have configured in vfs.zfs.arc_max (40M). >> > >> > After a while FS operations (e.g. pkgdb -F with about 900 >> > packages... my specific workload is the fixup of gnome packages >> > after the removal of the obsolete libusb port) get very slow (in >> > my specific example I let the pkgdb run several times over night >> > and it still is not finished). >> > >> > The big problem with this is, that at some point in time the >> > machine reboots (panic, page fault, page not present, during a >> > fork1). I have the impression (beware, I have a watchdog >> > configured, as I don't know if a triggered WD would cause the same >> > panic, the following is just a guess) that I run out of memory of >> > some kind (I have 1G RAM, i386, max kmem size 700M). I restarted >> > pkgdb several times after a reboot, and it continues to process the >> > libusb removal, but hey, this is anoying. >> > >> > Does someone see something similar to what I describe (mainly the >> > growth of the arc cache way beyond what is configured)? Anyone >> > with some ideas what to try? >> >> Can you provide the rest of the arcstats from sysctl? =A0Also, does >> your arc_reclaim_thread process get any cycles when this problem >> occurs? What happens if you kill the pkgdb -F manually before it >> completes? Does the arc cache size come back down or is it stuck at >> the abnormally high level? > > I haven't tried killing pkgdb and looking at the stats, but on the idle > machine (reboot after the panic and 5h of no use by me... the machine > fetches my mails, has a webmail + mysql + imap interface and is a > fileserver) the size is double of my max value. Again there's no real > load at this time, just fetching my mails (most traffic from the > FreeBSD lists) and a little bit of SpamAssassin filtering of them. When > I logged in this morning the machine was rebooted about 5h ago by a > panic and no FS traffic was going on (100% idle). > > Currently the arc_reclaim_thread has 0:12 of accumulated CPU time, > the wcpu is at 0%, but it is in the running state. The machine is > about 80% idle. > [snip] How about adding a few DTrace probes into arc_reclaim_thread and see what it does?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b649e5e0904180558u4e94b9e6p254067a3a53db097>