From owner-freebsd-current@FreeBSD.ORG Tue Nov 4 18:15:18 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 375584BE for ; Tue, 4 Nov 2014 18:15:18 +0000 (UTC) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF63D919 for ; Tue, 4 Nov 2014 18:15:17 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id z12so14040198wgg.23 for ; Tue, 04 Nov 2014 10:15:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=dPyJCQpAOnemY7Do9lrLSD5+xG9kSJmVoeWS3fTjpnM=; b=W81ODdQyM0tVLI0TwxjC168BZCHBpLaOqJHbYlTV/363CYf17dpiFYZEiG/OW26eD8 RTXMpRWWYO/Yy1Ho55WTRXwDLLPtbW/hwzQW0Eqvz5NcBzuXzz/3Km8zRJhctFxv4LgX lpx7etfaHEZI8F8+Aoqv1W7+3mQ2mcCp5zXGXWqNMbJRrAVXSD2jNskieqH+Me+NK4rx Ml6Kku38lrVM8vZLg5QsqsZRpm3Az8oduu/q+sgytCIYoAbXo7p/v83G4dutK+YuzekD h7MRMJ0znSte6tPxDH3dA/B/GlRAnYPY3sh2Zj6pyCR77EKLcHKw3DK73sgVDKFZ2ei0 c6IQ== X-Gm-Message-State: ALoCoQnObHtP2mn5AsaKWyZjHgtUteS6F7nIkLao7P10IZoTxSbF0UjlK2JVZ2LDvKLM+ob5exYB X-Received: by 10.194.81.70 with SMTP id y6mr12400585wjx.113.1415124910057; Tue, 04 Nov 2014 10:15:10 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id el4sm13099164wid.11.2014.11.04.10.15.08 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Nov 2014 10:15:08 -0800 (PST) Message-ID: <54591758.7000909@multiplay.co.uk> Date: Tue, 04 Nov 2014 18:13:44 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Allan Jude , freebsd-current@freebsd.org, gibbs@freebsd.org, George Kola Subject: Re: r273165. ZFS ARC: possible memory leak to Inact References: <1415098949.596412362.8vxee7kf@frv41.fwdcdn.com> <5458CCB6.7020602@multiplay.co.uk> <1415107358607-5962421.post@n5.nabble.com> <54590B55.3040206@freebsd.org> In-Reply-To: <54590B55.3040206@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 18:15:18 -0000 On 04/11/2014 17:22, Allan Jude wrote: > snip... > Justin Gibbs and I were helping George from Voxer look at the same issue > they are having. They had ~169GB in inact, and only ~60GB being used for > ARC. > > Are there any further debugging steps we can recommend to him to help > investigate this? The various scripts attached to the ZS ARC behavior problem and fix PR will help provide detail this. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 I've seen it here where there's been bursts of ZFS I/O specifically write bursts. What happens is that ZFS will consume large amounts of space in various UMA zones to accommodate these bursts. The VM only triggers UMA reclaim when it sees pressure, however if the main memory consumer is ZFS ARC its possible that the require pressure will not be applied because when allocating ARC ZFS takes into account free memory. The result is it will back off its memory requirements before the reclaim is triggered leaving all the space allocated but not used. I was playing around with a patch, on that bug report, which added clear down of UMA within ZFS ARC to avoid just this behavior, but its very much me playing for testing the theory only. From what I've seen UMA needs something like the coloring which can be used to trigger clear down over time to prevent UMA zones sitting their eating large amounts of memory like they currently do. Regards Steve