From owner-freebsd-stable@FreeBSD.ORG Thu Feb 9 01:45:14 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21667106566C for ; Thu, 9 Feb 2012 01:45:14 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id D582B8FC15 for ; Thu, 9 Feb 2012 01:45:13 +0000 (UTC) Received: (qmail 44020 invoked by uid 0); 9 Feb 2012 01:45:13 -0000 Received: from smtp.bway.net (216.220.96.25) by xena.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 9 Feb 2012 01:45:13 -0000 Received: (qmail 44001 invoked by uid 90); 9 Feb 2012 01:45:12 -0000 Received: from unknown (HELO ?10.3.2.40?) (spork@96.57.144.66) by smtp.bway.net with (AES128-SHA encrypted) SMTP; 9 Feb 2012 01:45:12 -0000 Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Charles Sprickman In-Reply-To: Date: Wed, 8 Feb 2012 20:45:11 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <4CCD642A-BE63-4AD8-BD80-1F55425CA537@bway.net> References: <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> <4F330F38.3010806@quip.cz> <20120209002835.GA29400@icarus.home.lan> To: Artem Belevich X-Mailer: Apple Mail (2.1084) Cc: freebsd-stable , mm@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, Andriy Gapon , "Eugene M. Zheganin" , Jeremy Chadwick Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 01:45:14 -0000 On Feb 8, 2012, at 7:43 PM, Artem Belevich wrote: > On Wed, Feb 8, 2012 at 4:28 PM, Jeremy Chadwick > wrote: >> On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote: > ... >>> ARC Size: >>> Current Size: 1769 MB (arcsize) >>> Target Size (Adaptive): 512 MB (c) >>> Min Size (Hard Limit): 512 MB (zfs_arc_min) >>> Max Size (Hard Limit): 3584 MB (zfs_arc_max) >>>=20 >>> The target size is going down to the min size and after few more >>> days, the system is so slow, that I must reboot the machine. Then it >>> is running fine for about 107 days and then it all repeat again. >>>=20 >>> You can see more on MRTG graphs >>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/ >>> You can see links to other useful informations on top of the page >>> (arc_summary, top, dmesg, fs usage, loader.conf) >>>=20 >>> There you can see nightly backups (higher CPU load started at >>> 01:13), otherwise the machine is idle. >>>=20 >>> It coresponds with ARC target size lowering in last 5 days >>> = http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcst= ats_size.html >>>=20 >>> And with ARC metadata cache overflowing the limit in last 5 days >>> = http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_m= eta.html >>>=20 >>> I don't know what's going on and I don't know if it is something >>> know / fixed in newer releases. We are running a few more ZFS >>> systems on 8.2 without this issue. But those systems are in >>> different roles. >>=20 >> This sounds like the... damn, what is it called... some kind of = internal >> "counter" or "ticks" thing within the ZFS code that was discovered to >> only begin happening after a certain period of time (which correlated = to >> some number of days, possibly 107). I'm sorry that I can't be more >> specific, but it's been discussed heavily on the lists in the past, = and >> fixes for all of that were committed to RELENG_8. I wish I could >> remember the name of the function or macro or variable name it = pertained >> to, something like LTHAW or TLOCK or something like that. I would = say >> "I don't know why I can't remember", but I do know why I can't = remember: >> because I gave up trying to track all of these problems. >>=20 >> Does someone else remember this issue? CC'ing Martin who might = remember >> for certain. >=20 > It's LBOLT. :-) >=20 > And there was more than one related integer overflow. One of them > manifested itself as L2ARC feeding thread hogging CPU time after about > a month of uptime. Another one caused issue with ARC reclaim after 107 > days. See more details in this thread: >=20 > http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html This would be an excellent piece of information to have on one of the = ZFS wiki pages. The 107 day issue exists post-8.2, correct? Anyone on this=20= cc: list have permissions to edit those pages? Thanks, Charles >=20 > --Artem > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"