From owner-freebsd-current@FreeBSD.ORG Tue Nov 4 12:19:27 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2F346D06 for ; Tue, 4 Nov 2014 12:19:27 +0000 (UTC) Received: from frv189.fwdcdn.com (frv189.fwdcdn.com [212.42.77.189]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E14A0808 for ; Tue, 4 Nov 2014 12:19:26 +0000 (UTC) Received: from [10.10.1.29] (helo=frv197.fwdcdn.com) by frv189.fwdcdn.com with esmtp ID 1XlcaF-000KS6-7X for freebsd-current@freebsd.org; Tue, 04 Nov 2014 13:47:35 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:To:Subject:From:Date; bh=dv6iBOxOGqClDfMZLMB74B+G8PWabZKZNrV5UBCKfhs=; b=njXCIHLI1SKkwK9UcihiQYwJCkuwIT0QLA6yMFvoIcMB/tt6g6wROcuiQbMpadv96p9Zz8Bqn8doUDiOjficTAWADnfq6qmcQDO854+OX5sh2UAAqc6s5/Btoi6XMekfjJ7wYNPQhgslE+4ulQj0uXQsokgE0/rKVyM0AO+J3uM=; Received: from [10.10.10.41] (helo=frv41.fwdcdn.com) by frv197.fwdcdn.com with smtp ID 1Xlca1-000Nj5-5h for freebsd-current@freebsd.org; Tue, 04 Nov 2014 13:47:21 +0200 Date: Tue, 04 Nov 2014 13:47:19 +0200 From: Dmitriy Makarov Subject: r273165. ZFS ARC: possible memory leak to Inact To: freebsd-current@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<1415098949.596412362.8vxee7kf@frv41.fwdcdn.com>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MIME-Version: 1.0 Received: from supportme@ukr.net by frv41.fwdcdn.com; Tue, 04 Nov 2014 13:47:19 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 12:19:27 -0000 Hi Current, It seems like there is constant flow (leak) of memory from ARC to Inact in FreeBSD 11.0-CURRENT #0 r273165. Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size very close to vfs.zfs.arc_max: Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M Free ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe enormous numbers of Inact memory in the top: Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other Funny thing is that when we manually allocate and release memory, using simple python script: #!/usr/local/bin/python2.7 import sys import time if len(sys.argv) != 2: print "usage: fillmem " sys.exit() count = int(sys.argv[1]) megabyte = (0,) * (1024 * 1024 / 8) data = megabyte * count as: # ./simple_script 10000 all those allocated megabyes 'migrate' from Inact to Free, and afterwards they are 'eaten' by ARC with no problem. Until Inact slowly grows back to the number it was before we ran the script. Current workaround is to periodically invoke this python script by cron. This is an ugly workaround and we really don't like it on our production To answer possible questions about ARC efficience: Cache efficiency drops dramatically with every GiB pushed off the ARC. Before upgrade: Cache Hit Ratio: 99.38% After upgrade: Cache Hit Ratio: 81.95% We believe that ARC misbehaves and we ask your assistance. ---------------------------------- Some values from configs. HW: 128GB RAM, LSI HBA controller with 36 disks (stripe of mirrors). top output: In /boot/loader.conf : vm.kmem_size="110G" vfs.zfs.arc_max="90G" vfs.zfs.arc_min="42G" vfs.zfs.txg.timeout="10" ----------------------------------- Thanks. Regards, Dmitriy