Date: Thu, 09 Feb 2012 02:20:35 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> To: Artem Belevich <art@freebsd.org> Cc: Andriy Gapon <avg@freebsd.org>, "Eugene M. Zheganin" <emz@norma.perm.ru>, freebsd-stable <freebsd-stable@freebsd.org>, mm@freebsd.org, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: zfs arc and amount of wired memory Message-ID: <4F331F63.60707@quip.cz> In-Reply-To: <CAFqOu6jdHXs02918q28RUo_KSyhW2OdHkLyerM4fZVstR1MCEA@mail.gmail.com> References: <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> <4F330F38.3010806@quip.cz> <20120209002835.GA29400@icarus.home.lan> <CAFqOu6jdHXs02918q28RUo_KSyhW2OdHkLyerM4fZVstR1MCEA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Artem Belevich wrote: > On Wed, Feb 8, 2012 at 4:28 PM, Jeremy Chadwick > <freebsd@jdc.parodius.com> wrote: >> On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote: > ... >>> ARC Size: >>> Current Size: 1769 MB (arcsize) >>> Target Size (Adaptive): 512 MB (c) >>> Min Size (Hard Limit): 512 MB (zfs_arc_min) >>> Max Size (Hard Limit): 3584 MB (zfs_arc_max) >>> >>> The target size is going down to the min size and after few more >>> days, the system is so slow, that I must reboot the machine. Then it >>> is running fine for about 107 days and then it all repeat again. >>> >>> You can see more on MRTG graphs >>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/ >>> You can see links to other useful informations on top of the page >>> (arc_summary, top, dmesg, fs usage, loader.conf) >>> >>> There you can see nightly backups (higher CPU load started at >>> 01:13), otherwise the machine is idle. >>> >>> It coresponds with ARC target size lowering in last 5 days >>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html >>> >>> And with ARC metadata cache overflowing the limit in last 5 days >>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html >>> >>> I don't know what's going on and I don't know if it is something >>> know / fixed in newer releases. We are running a few more ZFS >>> systems on 8.2 without this issue. But those systems are in >>> different roles. >> >> This sounds like the... damn, what is it called... some kind of internal >> "counter" or "ticks" thing within the ZFS code that was discovered to >> only begin happening after a certain period of time (which correlated to >> some number of days, possibly 107). I'm sorry that I can't be more >> specific, but it's been discussed heavily on the lists in the past, and >> fixes for all of that were committed to RELENG_8. Thank you for your quick response. I am glad that it is fixed in 8.x. So I will upgrade this last old machine in few weeks. :) >> I wish I could >> remember the name of the function or macro or variable name it pertained >> to, something like LTHAW or TLOCK or something like that. I would say >> "I don't know why I can't remember", but I do know why I can't remember: >> because I gave up trying to track all of these problems. >> >> Does someone else remember this issue? CC'ing Martin who might remember >> for certain. > > It's LBOLT. :-) > > And there was more than one related integer overflow. One of them > manifested itself as L2ARC feeding thread hogging CPU time after about > a month of uptime. Another one caused issue with ARC reclaim after 107 > days. See more details in this thread: > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html Yes, it is exactly this problem. Thank you for the link to this thread. I am subscribed to freebsd-fs@ and I am reading it almost daily, but I missed this one! Thanks to both of you! Miroslav Lachman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F331F63.60707>