From owner-freebsd-current@FreeBSD.ORG Thu Jan 30 11:16:31 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F00F0D44 for ; Thu, 30 Jan 2014 11:16:31 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 4293210B9 for ; Thu, 30 Jan 2014 11:16:30 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA13050; Thu, 30 Jan 2014 13:16:12 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1W8pbQ-0007n0-4J; Thu, 30 Jan 2014 13:16:12 +0200 Message-ID: <52EA3459.3070300@FreeBSD.org> Date: Thu, 30 Jan 2014 13:15:37 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Vladimir Sharun , Current FreeBSD Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> <52CA9963.1050507@FreeBSD.org> <1389676958.516993176.oq4lbgg7@frv45.ukr.net> <52D59E36.9040405@FreeBSD.org> <20140115102837.GA98983@hell.ukr.net> <52D66DB6.7030807@FreeBSD.org> <1390900795.258244476.v35k1338@frv45.ukr.net> In-Reply-To: <1390900795.258244476.v35k1338@frv45.ukr.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jan 2014 11:16:32 -0000 on 28/01/2014 11:28 Vladimir Sharun said the following: > Dear Andriy and FreeBSD community, > > After applying this path one of the systems runs fine (disk subsystem load low to moderate > - 10-20% busy sustained), > > Then I saw this patch was merged to the HEAD and we apply it to the one of the systems > with moderate to high disk load: 30-60% busy (11.0-CURRENT #7 r261118: Fri Jan 24 17:25:08 EET 2014) > > Within 4 days we experiencing the same leak(?) as without patch: > > last pid: 53841; load averages: 4.47, 4.18, 3.78 up 3+16:37:09 11:24:39 > 543 processes: 6 running, 537 sleeping > CPU: 8.7% user, 0.0% nice, 14.6% system, 1.4% interrupt, 75.3% idle > Mem: 22G Active, 1045M Inact, 98G Wired, 1288M Cache, 3284M Buf, 2246M Free > ARC: 73G Total, 3763M MFU, 62G MRU, 56M Anon, 1887M Header, 4969M Other > Swap: > > The ARC is populated within 30mins under load to the max (90Gb) then start decreasing. > > The delta between Wiread and ARC total start growing from typical 10-12Gb without L2 enabled > to the 25Gb with L2 enabled and counting (4 hours ago was 22Gb delta). First, have you checked that vmstat -z output contains the same anomaly as for in your original report? If yes, the please try to reproduce the problem with the following debugging patch: http://people.freebsd.org/~avg/l2arc-b_tmp_cdata-diag.patch Please make sure to compile your kernel (and modules) with INVARIANTS. -- Andriy Gapon