From owner-freebsd-stable@FreeBSD.ORG Wed Sep 1 00:49:21 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CDCED16A4CE; Wed, 1 Sep 2004 00:49:21 +0000 (GMT) Received: from ganymede.hub.org (blk-222-46-91.eastlink.ca [24.222.46.91]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9923743D39; Wed, 1 Sep 2004 00:49:21 +0000 (GMT) (envelope-from scrappy@hub.org) Received: by ganymede.hub.org (Postfix, from userid 1000) id 6A6B435221; Tue, 31 Aug 2004 21:49:22 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id 5F7643506B; Tue, 31 Aug 2004 21:49:22 -0300 (ADT) Date: Tue, 31 Aug 2004 21:49:22 -0300 (ADT) From: "Marc G. Fournier" To: freebsd-stable@freebsd.org In-Reply-To: <20040831205907.O31538@ganymede.hub.org> Message-ID: <20040831214524.P31538@ganymede.hub.org> References: <20040831205907.O31538@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: das@FreeBSD.org Subject: Re: vnodes - is there a leak? where are they going? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Sep 2004 00:49:22 -0000 As a follow up, looking at vmstat -m .. specifically the work that David did on seperating the union vs regular vnodes: UNION mount 60 2K 3K204800K 162 0 0 32 undcac 0 0K 1K204800K343638713 0 0 16 unpath 13146 227K 1025K204800K 43541149 0 0 16,32,64,128 Export Host 1 1K 1K204800K 164 0 0 256 vnodes 141 7K 8K204800K 613 0 0 16,32,64,128,256 Why does 'vnodes' show only 141 InUse? Or, in this case, should I be looking at: FFS node496600124150K 127870K204800K401059293 0 0 256 496k FFS nodes, if I'm reading right? vs neptune, which is showing only: FFS node300433 75109K 80257K204800K 3875307 0 0 256 On Tue, 31 Aug 2004, Marc G. Fournier wrote: > > I have two servers, both running 4.10 of within a few days (Aug 5 for venus, > Aug 7 for neptune) ... both running jail environments ... one with ~60 > running, the other with ~80 ... the one with 60 has been running for ~25 days > now, and is at the border of running out of vnodes: > > Aug 31 20:58:00 venus root: debug.numvnodes: 519920 - debug.freevnodes: 11058 > - debug.vnlru_nowhere: 256463 - vlrup > Aug 31 20:59:01 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13155 > - debug.vnlru_nowhere: 256482 - vlrup > Aug 31 21:00:03 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13092 > - debug.vnlru_nowhere: 256482 - vlruwt > > while the other one has been up for ~1 days, but is using alot less, for more > processes: > > Aug 31 20:58:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: > 208655 - debug.vnlru_nowhere: 0 - vlruwt > Aug 31 20:59:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: > 208602 - debug.vnlru_nowhere: 0 - vlruwt > Aug 31 21:00:03 neptune root: debug.numvnodes: 344062 - debug.freevnodes: > 208319 - debug.vnlru_nowhere: 0 - vlruwt > > I've tried shutting down all of the VMs on venus, and umount'd all of the > unionfs mounts, as well as the one nfs mount we have ... the above #s are > after the VMs (and mounts are recreated ... > > Now, my understanding of the vnodes is that for every file opened, a vnode is > created ... in my case, since I'm using unionfs, there are two vnodes per > file ... if it possible that there are 'stale' vnodes that aren't being freed > up? Is there some way of 'viewing' the vnode structure? > > For instance, fstat shows: > > venus# fstat | wc -l > 19531 > > So, obviously it isn't just open files that I'm dealing with here, for even > if I double that, that is nowhere near 519920 ... > > So, where else are the vnodes going? Is there a 'leak'? What can I look at > to try and narrow this down / provide more information? > > Even some way of determining a specific process that is sucking back alot of > them, to move that to a different machine ... ? > > Help? > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664