Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 09 Feb 2012 01:11:36 +0100
From:      Miroslav Lachman <000.fbsd@quip.cz>
To:        Andriy Gapon <avg@FreeBSD.org>
Cc:        "Eugene M. Zheganin" <emz@norma.perm.ru>, freebsd-stable <freebsd-stable@FreeBSD.org>
Subject:   Re: zfs arc and amount of wired memory
Message-ID:  <4F330F38.3010806@quip.cz>
In-Reply-To: <4F32DB30.6020600@FreeBSD.org>
References:  <4F30E284.8080905@norma.perm.ru> <4F310115.3070507@FreeBSD.org>	<4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org>	<4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org>	<4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org>	<4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Andriy Gapon wrote:
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 08.02.2012 02:17, Andriy Gapon wrote:
>>> [output snipped]
>>>
>>> Thank you.  I don't see anything suspicious/unusual there.
>>> Just case, do you have ZFS dedup enabled by a chance?
>>>
>>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>>> clues as to what got all that memory wired.
>>>
>> Nope, I don't have deduplication feature enabled.
>
> OK.  So, did you have a chance to inspect vmstat -m and vmstat -z?
>
>> By the way, today, after eating another 100M of wired memory this server hanged
>> out with multiple non-stopping messages
>>
>> swap_pager: indefinite wait buffer
>>
>> Since it's swapping on zvol, it looks to me like it could be the mentioned in
>> another thread here ("Swap on zvol - recommendable?") resource starvation issue;
>> may be it happens faster when the ARC isn't limited.
>
> It could be very well possible that swap on zvol doesn't work well when the
> kernel itself is starved on memory.
>
>> So I want to ask - how to report it and what should I include in such pr ?
>
> I am leaving swap-on-zvol issue aside.  Your original problem doesn't seem to be
> ZFS-related.  I suspect that you might be running into some kernel memory leak.
>   If you manage to reproduce the high wired value again, then vmstat -m and
> vmstat -z may provide some useful information.
>
> In this vein, do you use any out-of-tree kernel modules?
> Also, can you try to monitor your system to see when wired count grows?

I am seeing something similar on one of our machine. This is old 7.3 
with ZFS v13, that's why I did not reported it.

The machine is used as storage for backups made by rsync. All is running 
fine for about 107 days. Then backups are slower and slower because of 
some strange memory situation.

Mem: 15M Active, 17M Inact, 3620M Wired, 420K Cache, 48M Buf, 1166M Free

ARC Size:
          Current Size:             1769 MB (arcsize)
          Target Size (Adaptive):   512 MB (c)
          Min Size (Hard Limit):    512 MB (zfs_arc_min)
          Max Size (Hard Limit):    3584 MB (zfs_arc_max)

The target size is going down to the min size and after few more days, 
the system is so slow, that I must reboot the machine. Then it is 
running fine for about 107 days and then it all repeat again.

You can see more on MRTG graphs
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
You can see links to other useful informations on top of the page 
(arc_summary, top, dmesg, fs usage, loader.conf)

There you can see nightly backups (higher CPU load started at 01:13), 
otherwise the machine is idle.

It coresponds with ARC target size lowering in last 5 days
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html

And with ARC metadata cache overflowing the limit in last 5 days
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html

I don't know what's going on and I don't know if it is something know / 
fixed in newer releases. We are running a few more ZFS systems on 8.2 
without this issue. But those systems are in different roles.

Miroslav Lachman



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F330F38.3010806>