From owner-freebsd-current@FreeBSD.ORG Tue Jan 14 05:27:59 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F451BBF for ; Tue, 14 Jan 2014 05:27:59 +0000 (UTC) Received: from frv198.fwdcdn.com (frv198.fwdcdn.com [212.42.77.198]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 290DC1FCF for ; Tue, 14 Jan 2014 05:27:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-Id:To:Subject:From:Date; bh=P+lESE1Srwe9VJMy6EqhtuqXhzdWq1oFRnxWOhJh9IY=; b=MEJuYZRdtxjj5YQX5i2I2SeCSACqgmskHqHTebgj9xu6tdgHFqQP5uAfiQjnpcEao9gNry1VrQxoAu5TKuKR99/+9v8oqodCZT19PoPLUaKBd2XBqYl1DwioQ+/oTFW21IeSsFntZ2u5rv/MGkVIiHc/Td6bAEDterrbm5XJ49M=; Received: from [10.10.10.45] (helo=frv45.ukr.net) by frv198.fwdcdn.com with smtp ID 1W2wXU-0003By-GA for freebsd-current@freebsd.org; Tue, 14 Jan 2014 07:27:48 +0200 Date: Tue, 14 Jan 2014 07:27:47 +0200 From: Vladimir Sharun Subject: Re[2]: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) To: Current FreeBSD X-Mailer: mail.ukr.net 5.0 Message-Id: <1389676958.516993176.oq4lbgg7@frv45.ukr.net> In-Reply-To: <52CA9963.1050507@FreeBSD.org> References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> <52CA9963.1050507@FreeBSD.org> MIME-Version: 1.0 Received: from atz@ukr.net by frv45.ukr.net; Tue, 14 Jan 2014 07:27:47 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 05:27:59 -0000 Dear Andriy and FreeBSD community, > I am not sure if the buffers are leaked somehow or if they are actually in use. > It's one of the very few places where data buffers are allocated without > charging ARC. In all other places it's quite easy to match allocations and > deallocations. But in L2ARC it is not obvious that all buffers get freed or > when that happens. After one week under load I think we figure out the cause: it's L2ARC. Here's the top's header for 7d17h of the runtime: last pid: 46409; load averages: 0.37, 0.62, 0.70 up 7+17:14:01 07:24:10 173 processes: 1 running, 171 sleeping, 1 zombie CPU: 2.0% user, 0.0% nice, 3.5% system, 0.4% interrupt, 94.2% idle Mem: 8714M Active, 14G Inact, 96G Wired, 1929M Cache, 3309M Buf, 3542M Free ARC: 85G Total, 2558M MFU, 77G MRU, 28M Anon, 1446M Header, 4802M Other ARC related tunables: vm.kmem_size="110G" vfs.zfs.arc_max="90G" vfs.zfs.arc_min="42G" For more than 7 days of hard runtime the picture clearly shows: Wired minus ARC = 11..12Gb, ARC grow and shrinks in 80-87Gb range and the system runs just fine. So what shall we do with L2ARC leakage ?