From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 15:28:15 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37520106566C for ; Mon, 3 Jan 2011 15:28:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id D7D398FC21 for ; Mon, 3 Jan 2011 15:28:14 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 7B3E046B4C; Mon, 3 Jan 2011 10:28:14 -0500 (EST) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7884D8A009; Mon, 3 Jan 2011 10:28:13 -0500 (EST) From: John Baldwin To: Chris Peiffer Date: Mon, 3 Jan 2011 10:19:50 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.3-CBSD-20101102; KDE/4.4.5; amd64; ; ) References: <20101229221801.GA48661@cabstand.com> <4D1C9931.2060209@FreeBSD.org> <20101230202755.GB3256@cabstand.com> In-Reply-To: <20101230202755.GB3256@cabstand.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201101031019.50236.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 03 Jan 2011 10:28:13 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.9 required=4.2 tests=BAYES_00 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: freebsd-fs@freebsd.org Subject: Re: how big can kern.maxvnodes get? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2011 15:28:15 -0000 On Thursday, December 30, 2010 3:27:55 pm Chris Peiffer wrote: > On Thu, Dec 30, 2010 at 09:37:37AM -0500, John Baldwin wrote: > > Chris Peiffer wrote: > > >I have a backend server running 8-2-PRERELEASE with lots of > > >independent files that randomly grow and then get truncated to > > >zero. (Think popserver.) > > > > > >Roughly 2.5 million inodes on each of 4 Intel SSD disks. 24 gb of RAM > > >in the system. I want to maximize the buffer cache in RAM. > > > > > >I doubled kern.maxvnodes to 942108 and reads/second went down and > > >memory use went up, (as I expected) but right now there's still about > > >15g RAM marked as free. > > > > > >vfs.numvnodes crept up to 821704 and has hovered there. The file > > >sizes range to 1 mb but most are in the range 0-10k. Since the server > > >operations are so simple kern.openfiles hovers in the range 100-200. > > > > > >Obviously, all things being equal I'd like to give the filesystem > > >buffer cache access to that free RAM by allowing more vnodes to stay > > >cached. > > > > > >Can I increase kern.maxvnodes by another factor of 2? more? Are there > > >any known problems with stepping it up, besides general memory > > >exhaustion? With so much free RAM I'd like to turn the dial a little > > >bit but I wonder if there are other linked things I should watch out > > >for. > > > > You can increase it, but if numvnodes is less than maxvnodes then it > > won't help you as the system hasn't had to recycle any vnodes yet. It > > is already caching all the vnodes you have accessed in that case. > > > > If the files are frequently truncated then you may end up with a lot of > > free RAM simply because there isn't enough recently used data to cache. > > The VM system will cache everything that is accessed until either 1) > > the pages become invalid (e.g. due to an unlink or truncate) or 2) free > > memory runs low enough to trigger pagedaemon to harvest some inactive > > pages. If you have 15G of free RAM, then 2) isn't happening and your > > working set is less than your RAM. > > > > Thanks John. > > The system has around 2m non-empty data files. I assumed that since > the numvnodes quickly jumped to 820k and hovered, there must be some > "min free" threshold below maxvnodes that an unloaded system would try > to maintain. But I will investigate my exact working set before I tune > maxvnodes up. Hmm, there is a minimum number of free vnodes (vfs.wantfreevnodes), however if the system needs more free vnodes (vfs.freevnodes is current count) it will just create more vnodes until it hits maxvnodes. Only then will it start to recycle used vnodes to create more free vnodes. -- John Baldwin