From owner-freebsd-stable@FreeBSD.ORG Thu Feb 9 01:08:04 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E79541065676; Thu, 9 Feb 2012 01:08:04 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 78B0D8FC0A; Thu, 9 Feb 2012 01:08:04 +0000 (UTC) Received: by yenl12 with SMTP id l12so801568yen.13 for ; Wed, 08 Feb 2012 17:08:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=DfpgNDsVDEzbhl16rAXBHnvr67Oaq/0VXzHc1LCuBNA=; b=tW0bR35IoNRSwokosa/pKdXmDf1xBFomoZDaRcm7JKldQqU00UzwqcS9XCefjHJSn2 gY0YYOCuSlB3M7vs/noe6uBBXLIt/xoaf3kfuom16JbQIiqhgM6zqxAXgs7zCuh2vC0X mFEpH0LS3pF53l9FZOSRwBMsALpf+NT9VoPmY= MIME-Version: 1.0 Received: by 10.101.51.12 with SMTP id d12mr11929277ank.69.1328748229003; Wed, 08 Feb 2012 16:43:49 -0800 (PST) Sender: artemb@gmail.com Received: by 10.147.47.6 with HTTP; Wed, 8 Feb 2012 16:43:48 -0800 (PST) In-Reply-To: <20120209002835.GA29400@icarus.home.lan> References: <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> <4F330F38.3010806@quip.cz> <20120209002835.GA29400@icarus.home.lan> Date: Wed, 8 Feb 2012 16:43:48 -0800 X-Google-Sender-Auth: 1490Z5Xe4WXY6GhkC6HY0dvkPm0 Message-ID: From: Artem Belevich To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "Eugene M. Zheganin" , freebsd-stable , Miroslav Lachman <000.fbsd@quip.cz>, Andriy Gapon , mm@freebsd.org Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 01:08:05 -0000 On Wed, Feb 8, 2012 at 4:28 PM, Jeremy Chadwick wrote: > On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote: ... >> ARC Size: >> =A0 =A0 =A0 =A0 =A0Current Size: =A0 =A0 =A0 =A0 =A0 =A0 1769 MB (arcsiz= e) >> =A0 =A0 =A0 =A0 =A0Target Size (Adaptive): =A0 512 MB (c) >> =A0 =A0 =A0 =A0 =A0Min Size (Hard Limit): =A0 =A0512 MB (zfs_arc_min) >> =A0 =A0 =A0 =A0 =A0Max Size (Hard Limit): =A0 =A03584 MB (zfs_arc_max) >> >> The target size is going down to the min size and after few more >> days, the system is so slow, that I must reboot the machine. Then it >> is running fine for about 107 days and then it all repeat again. >> >> You can see more on MRTG graphs >> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/ >> You can see links to other useful informations on top of the page >> (arc_summary, top, dmesg, fs usage, loader.conf) >> >> There you can see nightly backups (higher CPU load started at >> 01:13), otherwise the machine is idle. >> >> It coresponds with ARC target size lowering in last 5 days >> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arc= stats_size.html >> >> And with ARC metadata cache overflowing the limit in last 5 days >> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs= _meta.html >> >> I don't know what's going on and I don't know if it is something >> know / fixed in newer releases. We are running a few more ZFS >> systems on 8.2 without this issue. But those systems are in >> different roles. > > This sounds like the... damn, what is it called... some kind of internal > "counter" or "ticks" thing within the ZFS code that was discovered to > only begin happening after a certain period of time (which correlated to > some number of days, possibly 107). =A0I'm sorry that I can't be more > specific, but it's been discussed heavily on the lists in the past, and > fixes for all of that were committed to RELENG_8. =A0I wish I could > remember the name of the function or macro or variable name it pertained > to, something like LTHAW or TLOCK or something like that. =A0I would say > "I don't know why I can't remember", but I do know why I can't remember: > because I gave up trying to track all of these problems. > > Does someone else remember this issue? =A0CC'ing Martin who might remembe= r > for certain. It's LBOLT. :-) And there was more than one related integer overflow. One of them manifested itself as L2ARC feeding thread hogging CPU time after about a month of uptime. Another one caused issue with ARC reclaim after 107 days. See more details in this thread: http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html --Artem