From owner-freebsd-fs@FreeBSD.ORG Thu Dec 30 20:27:57 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB102106566C for ; Thu, 30 Dec 2010 20:27:57 +0000 (UTC) (envelope-from chris@bab.cabstand.com) Received: from bab.cabstand.com (bab.cabstand.com [76.246.87.242]) by mx1.freebsd.org (Postfix) with ESMTP id 612FE8FC0A for ; Thu, 30 Dec 2010 20:27:55 +0000 (UTC) Received: from bab.cabstand.com (localhost [127.0.0.1]) by bab.cabstand.com (8.13.8/8.13.8) with ESMTP id oBUKRtBO008152 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Thu, 30 Dec 2010 12:27:55 -0800 (PST) (envelope-from chris@bab.cabstand.com) Received: (from chris@localhost) by bab.cabstand.com (8.13.8/8.13.8/Submit) id oBUKRtZC008151; Thu, 30 Dec 2010 12:27:55 -0800 (PST) (envelope-from chris) Date: Thu, 30 Dec 2010 12:27:55 -0800 From: Chris Peiffer To: John Baldwin Message-ID: <20101230202755.GB3256@cabstand.com> Mail-Followup-To: John Baldwin , freebsd-fs@freebsd.org References: <20101229221801.GA48661@cabstand.com> <4D1C9931.2060209@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D1C9931.2060209@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org Subject: Re: how big can kern.maxvnodes get? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Dec 2010 20:27:57 -0000 On Thu, Dec 30, 2010 at 09:37:37AM -0500, John Baldwin wrote: > Chris Peiffer wrote: > >I have a backend server running 8-2-PRERELEASE with lots of > >independent files that randomly grow and then get truncated to > >zero. (Think popserver.) > > > >Roughly 2.5 million inodes on each of 4 Intel SSD disks. 24 gb of RAM > >in the system. I want to maximize the buffer cache in RAM. > > > >I doubled kern.maxvnodes to 942108 and reads/second went down and > >memory use went up, (as I expected) but right now there's still about > >15g RAM marked as free. > > > >vfs.numvnodes crept up to 821704 and has hovered there. The file > >sizes range to 1 mb but most are in the range 0-10k. Since the server > >operations are so simple kern.openfiles hovers in the range 100-200. > > > >Obviously, all things being equal I'd like to give the filesystem > >buffer cache access to that free RAM by allowing more vnodes to stay > >cached. > > > >Can I increase kern.maxvnodes by another factor of 2? more? Are there > >any known problems with stepping it up, besides general memory > >exhaustion? With so much free RAM I'd like to turn the dial a little > >bit but I wonder if there are other linked things I should watch out > >for. > > You can increase it, but if numvnodes is less than maxvnodes then it > won't help you as the system hasn't had to recycle any vnodes yet. It > is already caching all the vnodes you have accessed in that case. > > If the files are frequently truncated then you may end up with a lot of > free RAM simply because there isn't enough recently used data to cache. > The VM system will cache everything that is accessed until either 1) > the pages become invalid (e.g. due to an unlink or truncate) or 2) free > memory runs low enough to trigger pagedaemon to harvest some inactive > pages. If you have 15G of free RAM, then 2) isn't happening and your > working set is less than your RAM. > Thanks John. The system has around 2m non-empty data files. I assumed that since the numvnodes quickly jumped to 820k and hovered, there must be some "min free" threshold below maxvnodes that an unloaded system would try to maintain. But I will investigate my exact working set before I tune maxvnodes up.