From owner-freebsd-stable@FreeBSD.ORG Thu Feb 9 00:11:41 2012 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E17B510656D0 for ; Thu, 9 Feb 2012 00:11:40 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 638538FC15 for ; Thu, 9 Feb 2012 00:11:40 +0000 (UTC) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B125628432; Thu, 9 Feb 2012 01:11:38 +0100 (CET) Received: from [192.168.1.2] (ip-86-49-61-235.net.upcbroadband.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2CE6F2842E; Thu, 9 Feb 2012 01:11:37 +0100 (CET) Message-ID: <4F330F38.3010806@quip.cz> Date: Thu, 09 Feb 2012 01:11:36 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.19) Gecko/20110420 Lightning/1.0b1 SeaMonkey/2.0.14 MIME-Version: 1.0 To: Andriy Gapon References: <4F30E284.8080905@norma.perm.ru> <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> In-Reply-To: <4F32DB30.6020600@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "Eugene M. Zheganin" , freebsd-stable Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 00:11:41 -0000 Andriy Gapon wrote: > on 08/02/2012 12:31 Eugene M. Zheganin said the following: >> Hi. >> >> On 08.02.2012 02:17, Andriy Gapon wrote: >>> [output snipped] >>> >>> Thank you. I don't see anything suspicious/unusual there. >>> Just case, do you have ZFS dedup enabled by a chance? >>> >>> I think that examination of vmstat -m and vmstat -z outputs may provide some >>> clues as to what got all that memory wired. >>> >> Nope, I don't have deduplication feature enabled. > > OK. So, did you have a chance to inspect vmstat -m and vmstat -z? > >> By the way, today, after eating another 100M of wired memory this server hanged >> out with multiple non-stopping messages >> >> swap_pager: indefinite wait buffer >> >> Since it's swapping on zvol, it looks to me like it could be the mentioned in >> another thread here ("Swap on zvol - recommendable?") resource starvation issue; >> may be it happens faster when the ARC isn't limited. > > It could be very well possible that swap on zvol doesn't work well when the > kernel itself is starved on memory. > >> So I want to ask - how to report it and what should I include in such pr ? > > I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be > ZFS-related. I suspect that you might be running into some kernel memory leak. > If you manage to reproduce the high wired value again, then vmstat -m and > vmstat -z may provide some useful information. > > In this vein, do you use any out-of-tree kernel modules? > Also, can you try to monitor your system to see when wired count grows? I am seeing something similar on one of our machine. This is old 7.3 with ZFS v13, that's why I did not reported it. The machine is used as storage for backups made by rsync. All is running fine for about 107 days. Then backups are slower and slower because of some strange memory situation. Mem: 15M Active, 17M Inact, 3620M Wired, 420K Cache, 48M Buf, 1166M Free ARC Size: Current Size: 1769 MB (arcsize) Target Size (Adaptive): 512 MB (c) Min Size (Hard Limit): 512 MB (zfs_arc_min) Max Size (Hard Limit): 3584 MB (zfs_arc_max) The target size is going down to the min size and after few more days, the system is so slow, that I must reboot the machine. Then it is running fine for about 107 days and then it all repeat again. You can see more on MRTG graphs http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/ You can see links to other useful informations on top of the page (arc_summary, top, dmesg, fs usage, loader.conf) There you can see nightly backups (higher CPU load started at 01:13), otherwise the machine is idle. It coresponds with ARC target size lowering in last 5 days http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html And with ARC metadata cache overflowing the limit in last 5 days http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html I don't know what's going on and I don't know if it is something know / fixed in newer releases. We are running a few more ZFS systems on 8.2 without this issue. But those systems are in different roles. Miroslav Lachman