From owner-freebsd-fs@FreeBSD.ORG Wed Dec 29 22:35:00 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39B13106564A for ; Wed, 29 Dec 2010 22:35:00 +0000 (UTC) (envelope-from chris@bab.cabstand.com) Received: from bab.cabstand.com (bab.cabstand.com [76.246.87.242]) by mx1.freebsd.org (Postfix) with ESMTP id 0F55D8FC08 for ; Wed, 29 Dec 2010 22:34:59 +0000 (UTC) Received: from bab.cabstand.com (localhost [127.0.0.1]) by bab.cabstand.com (8.13.8/8.13.8) with ESMTP id oBTMI1Bl055057 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Wed, 29 Dec 2010 14:18:01 -0800 (PST) (envelope-from chris@bab.cabstand.com) Received: (from chris@localhost) by bab.cabstand.com (8.13.8/8.13.8/Submit) id oBTMI1BT055056 for freebsd-fs@freebsd.org; Wed, 29 Dec 2010 14:18:01 -0800 (PST) (envelope-from chris) Date: Wed, 29 Dec 2010 14:18:01 -0800 From: Chris Peiffer To: freebsd-fs@freebsd.org Message-ID: <20101229221801.GA48661@cabstand.com> Mail-Followup-To: freebsd-fs@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Subject: how big can kern.maxvnodes get? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Dec 2010 22:35:00 -0000 I have a backend server running 8-2-PRERELEASE with lots of independent files that randomly grow and then get truncated to zero. (Think popserver.) Roughly 2.5 million inodes on each of 4 Intel SSD disks. 24 gb of RAM in the system. I want to maximize the buffer cache in RAM. I doubled kern.maxvnodes to 942108 and reads/second went down and memory use went up, (as I expected) but right now there's still about 15g RAM marked as free. vfs.numvnodes crept up to 821704 and has hovered there. The file sizes range to 1 mb but most are in the range 0-10k. Since the server operations are so simple kern.openfiles hovers in the range 100-200. Obviously, all things being equal I'd like to give the filesystem buffer cache access to that free RAM by allowing more vnodes to stay cached. Can I increase kern.maxvnodes by another factor of 2? more? Are there any known problems with stepping it up, besides general memory exhaustion? With so much free RAM I'd like to turn the dial a little bit but I wonder if there are other linked things I should watch out for. Thanks. Here are some lines from vmstat -z that might be relevant: ITEM SIZE LIMIT USED FREE REQUESTS FAILURES VNODE: 472, 0, 779555, 69229, 163219829, 0 VNODEPOLL: 112, 0, 2, 64, 4, 0 S VFS Cache: 108, 0, 761856, 63606, 504696076, 0 L VFS Cache: 328, 0, 0, 228, 300, 0