From owner-freebsd-current@FreeBSD.ORG Sun Jan 5 11:16:57 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 85C8EC08 for ; Sun, 5 Jan 2014 11:16:57 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id D497C1663 for ; Sun, 5 Jan 2014 11:16:56 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA08001; Sun, 05 Jan 2014 13:14:13 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Vzlen-000N9q-AN; Sun, 05 Jan 2014 13:14:13 +0200 Message-ID: <52C93E4D.1050100@FreeBSD.org> Date: Sun, 05 Jan 2014 13:13:17 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Vladimir Sharun , Current FreeBSD Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> In-Reply-To: <1388839805.123581691.q97ijp8l@frv45.ukr.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jan 2014 11:16:57 -0000 on 04/01/2014 14:50 Vladimir Sharun said the following: [snip] > ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other [snip] > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP [snip] > zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0 I noticed a particular discrepancy between reported ARC usage and sizes of UMA zones used by ZFS code: 488217 * 131072 = ~59GB right there. There are several possibilities for this discrepancy: - bad accounting or reporting of ARC stats - those 128K buffers being used in a special way and thus not accounted as ARC - some sort of resource leak You could try to use DTrace to gather the stacks of all code paths that lead to allocation of those buffers. Something like: fbt::zio_data_buf_alloc:entry /arg0 == 131072/ { @[stack()] = count(); } This could be a start for understanding the issue. -- Andriy Gapon