From owner-freebsd-stable@FreeBSD.ORG Sat Oct 30 16:31:34 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 755E316A4CE for ; Sat, 30 Oct 2004 16:31:34 +0000 (GMT) Received: from hub.org (hub.org [200.46.204.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BD6643D2D for ; Sat, 30 Oct 2004 16:31:34 +0000 (GMT) (envelope-from scrappy@hub.org) Received: from localhost (unknown [200.46.204.144]) by hub.org (Postfix) with ESMTP id A0FC412B0E1 for ; Sat, 30 Oct 2004 13:31:32 -0300 (ADT) Received: from hub.org ([200.46.204.220]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 17578-05 for ; Sat, 30 Oct 2004 16:31:32 +0000 (GMT) Received: from ganymede.hub.org (blk-222-46-91.eastlink.ca [24.222.46.91]) by hub.org (Postfix) with ESMTP id 2B02112B0D4 for ; Sat, 30 Oct 2004 13:31:32 -0300 (ADT) Received: by ganymede.hub.org (Postfix, from userid 1000) id CB66A3C627; Sat, 30 Oct 2004 13:31:31 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id CA4C33C625 for ; Sat, 30 Oct 2004 13:31:31 -0300 (ADT) Date: Sat, 30 Oct 2004 13:31:31 -0300 (ADT) From: "Marc G. Fournier" To: freebsd-stable@freebsd.org In-Reply-To: <20041030131002.O6085@ganymede.hub.org> Message-ID: <20041030133044.O6085@ganymede.hub.org> References: <20041030131002.O6085@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by amavisd-new at hub.org Subject: Re: vnode 'leak' in 4.x ... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Oct 2004 16:31:34 -0000 Just to give an idea of what a second server, with less uptime, is looking like, with the approx. the same # of VMs on her: Oct 30 13:29:00 neptune root: debug.numvnodes: 462882 - debug.freevnodes: 132826 - debug.vnlru_nowhere: 0 - vlruwt Oct 30 13:30:00 neptune root: debug.numvnodes: 462882 - debug.freevnodes: 151976 - debug.vnlru_nowhere: 0 - vlruwt But she's only been up 7 days so far ... On Sat, 30 Oct 2004, Marc G. Fournier wrote: > > A little while ago, I reported a suspicion that vnodes just weren't being > freed up on long running servers ... after 55days of uptime on one of my > servers, here is what I'm dealing with ... > > 793 'samples' today (one every minute) > 786 with vnlru in a vlrup state > > I shutdown all of the VMs running on the large hard drive (the only place > unionfs is being used) and umount'd the drive ... there were some suggested > back then that this might/should free everything back up again ... but it > didn't: > > Oct 30 13:06:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57966 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:07:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57268 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:08:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 52335 > - debug.vnlru_nowhere: 209679 - vlruwt > Oct 30 13:09:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 50228 > - debug.vnlru_nowhere: 209682 - vlrup > Oct 30 13:10:01 venus root: debug.numvnodes: 522265 - debug.freevnodes: 44407 > - debug.vnlru_nowhere: 209690 - vlrup > Oct 30 13:11:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 35424 > - debug.vnlru_nowhere: 209697 - vlrup > Oct 30 13:12:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 34626 > - debug.vnlru_nowhere: 209708 - vlrup > Oct 30 13:13:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 29214 > - debug.vnlru_nowhere: 209727 - vlrup > Oct 30 13:14:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 24414 > - debug.vnlru_nowhere: 209746 - vlrup > Oct 30 13:15:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 26994 > - debug.vnlru_nowhere: 209766 - vlrup > > The 'vlruwt' states above are while I had everything shutdown ... the vlrup's > all started again after I mounted the drive and started to restart the VMs > themselves ... > > I expect a high # of vnodes to be used ... that isn't the issue ... the issue > is that even getting rid of the major mount point, so that only /, /tmp, > /usr, /var are left up, the large # of vnodes that are in use on that mount > point aren't being freed by vnlru :( > > I hate to reboot the server, but it looks like I've got no choice at this > point ... is there something else that I can do, in 50 days or so, to provide > more information? > > Thanks ... > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664