From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 18 21:30:44 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B05807C9; Tue, 18 Mar 2014 21:30:44 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 878B480; Tue, 18 Mar 2014 21:30:44 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 27774B946; Tue, 18 Mar 2014 17:30:43 -0400 (EDT) From: John Baldwin To: Karl Denninger Subject: Re: Tracking down what has inact pages locked up Date: Tue, 18 Mar 2014 17:30:02 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <53260B36.2070409@denninger.net> <201403181505.47349.jhb@freebsd.org> <5328A024.6050901@denninger.net> In-Reply-To: <5328A024.6050901@denninger.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201403181730.02471.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 18 Mar 2014 17:30:43 -0400 (EDT) Cc: Alan Cox , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 21:30:44 -0000 On Tuesday, March 18, 2014 3:36:04 pm Karl Denninger wrote: > > On 3/18/2014 2:05 PM, John Baldwin wrote: > > On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote: > >> Is there a reasonable way to determine who or what has that memory > >> locked up -- and thus why the vm system is not demoting that space into > >> the cache bucket so it can be freed (which, if my understanding is > >> correct, should be happening long before now!) > > I have a hackish thing (for 8.x, might work on 10.x) to let you figure out > > what is using up RAM. This should perhaps go into the base system at some > > point. > > > > Grab the bits at http://people.freebsd.org/~jhb/vm_objects/ > > > > You will want to build the kld first and use 'make load' to load it. It adds > > a new sysctl that dumps info about all the VM objects in the system. You can > > then build the 'vm_objects' tool and run it. It can take a while to run if > > you have NFS mounts, so I typically save its output to a file first and then > > use sort on the results. sort -n will show you the largest consumer of RAM, > > sort -n -k 3 will show you the largest consumer of inactive pages. Note > > that 'df' and 'ph' objects are anonymous, and that filename paths aren't > > always reliable, but this can still be useful. > > > Thanks. > > I suspect the cause of the huge inact consumption is a RAM leak in the > NAT code in IPFW. It was not occurring in 9.2-STABLE, but is on > 10.0-STABLE, and reverting to natd in userland stops it -- which > pretty-well isolates where it's coming from. Memory for in-kernel NAT should be wired pages, not inactive. -- John Baldwin