From owner-freebsd-current@FreeBSD.ORG Mon Jan 6 11:55:18 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 41DC26F6 for ; Mon, 6 Jan 2014 11:55:18 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3D918D4 for ; Mon, 6 Jan 2014 11:55:17 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA21489; Mon, 06 Jan 2014 13:55:08 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1W08lw-0001TI-Al; Mon, 06 Jan 2014 13:55:08 +0200 Message-ID: <52CA9963.1050507@FreeBSD.org> Date: Mon, 06 Jan 2014 13:54:11 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Vladimir Sharun Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> In-Reply-To: <1389005433.815055146.2dcjke36@frv45.ukr.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Current FreeBSD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 11:55:18 -0000 on 06/01/2014 13:14 Vladimir Sharun said the following: > Dear Andriy and FreeBSD community, > > I got the few minutes run for this dtrace hook; here's the output for 15 minutes run: > > http://pastebin.com/pKm9kLwa > > Does it explain something ? The following makes me suspect a problem with L2ARC compression code. zfs.ko`l2arc_feed_thread+0x7d9 kernel`fork_exit+0x9a kernel`0xffffffff8069ad6e 95131 I am not sure if the buffers are leaked somehow or if they are actually in use. It's one of the very few places where data buffers are allocated without charging ARC. In all other places it's quite easy to match allocations and deallocations. But in L2ARC it is not obvious that all buffers get freed or when that happens. >> on 04/01/2014 14:50 Vladimir Sharun said the following: >> [snip] >>> ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other >> [snip] >>> ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP >> [snip] >>> zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0 >> >> I noticed a particular discrepancy between reported ARC usage and sizes of UMA >> zones used by ZFS code: >> >> 488217 * 131072 = ~59GB right there. >> >> There are several possibilities for this discrepancy: >> - bad accounting or reporting of ARC stats >> - those 128K buffers being used in a special way and thus not accounted as ARC >> - some sort of resource leak >> >> You could try to use DTrace to gather the stacks of all code paths that lead to >> allocation of those buffers. Something like: >> >> fbt::zio_data_buf_alloc:entry >> /arg0 == 131072/ >> { >> @[stack()] = count(); >> } >> >> This could be a start for understanding the issue. >> -- Andriy Gapon