From owner-freebsd-current@FreeBSD.ORG Wed Nov 5 10:21:32 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 516A2AD9 for ; Wed, 5 Nov 2014 10:21:32 +0000 (UTC) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3211214A for ; Wed, 5 Nov 2014 10:21:31 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1XlxQp-0007cb-TN for freebsd-current@freebsd.org; Wed, 05 Nov 2014 02:03:15 -0800 Date: Wed, 5 Nov 2014 02:03:15 -0800 (PST) From: Dmitriy Makarov To: freebsd-current@freebsd.org Message-ID: <1415181795899-5962711.post@n5.nabble.com> In-Reply-To: <5459ECDE.6090102@multiplay.co.uk> References: <1415098949.596412362.8vxee7kf@frv41.fwdcdn.com> <5458CCB6.7020602@multiplay.co.uk> <1415107358607-5962421.post@n5.nabble.com> <54590B55.3040206@freebsd.org> <54591758.7000909@multiplay.co.uk> <20141105061540.GA14812@blazingdot.com> <5459ECDE.6090102@multiplay.co.uk> Subject: Re: r273165. ZFS ARC: possible memory leak to Inact MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 10:21:32 -0000 Steven Hartland wrote > On 05/11/2014 06:15, Marcus Reid wrote: >> On Tue, Nov 04, 2014 at 06:13:44PM +0000, Steven Hartland wrote: >>> On 04/11/2014 17:22, Allan Jude wrote: >>>> snip... >>>> Justin Gibbs and I were helping George from Voxer look at the same >>>> issue >>>> they are having. They had ~169GB in inact, and only ~60GB being used >>>> for >>>> ARC. >>>> >>>> Are there any further debugging steps we can recommend to him to help >>>> investigate this? >>> The various scripts attached to the ZS ARC behavior problem and fix PR >>> will help provide detail this. >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 >>> >>> I've seen it here where there's been bursts of ZFS I/O specifically >>> write bursts. >>> >>> What happens is that ZFS will consume large amounts of space in various >>> UMA zones to accommodate these bursts. >> If you push the vmstat -z that he provided through the arc summary >> script, you'll see that this is not what is happening. His uma stats >> match up with his arc, and do not account for his inactive memory. >> >> uma script summary: >> >> Totals >> oused: 5.860GB, ofree: 1.547GB, ototal: 7.407GB >> zused: 56.166GB, zfree: 3.918GB, ztotal: 60.084GB >> used: 62.026GB, free: 5.465GB, total: 67.491GB >> >> His provided top stats: >> >> Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M >> Free >> ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M >> Other >> >> >> The big uma buckets (zio_buf_16384 and zio_data_buf_131072, 18.002GB and >> 28.802GB respectively) are nearly 0% free. >> > Still potentially accounts for 5.4GB of your 20GB inact. > > The rest could be malloc backed allocations? No. There are few reasons for that. The first one is that Inact constantly grows, and 20GiB you see were 50GiBs before we ran the script. (We have to run it periodically or else our production server will grow slower and slower) The second argumens is that our codebase is the same, the only thing that have changed is OS version. In the previous version Inact was dramatically much smaller: ~hundrets of megabytes. -- View this message in context: http://freebsd.1045724.n5.nabble.com/r273165-ZFS-ARC-possible-memory-leak-to-Inact-tp5962410p5962711.html Sent from the freebsd-current mailing list archive at Nabble.com.