From owner-freebsd-stable@FreeBSD.ORG Sat Oct 30 16:20:17 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0C6EF16A4CE for ; Sat, 30 Oct 2004 16:20:17 +0000 (GMT) Received: from hub.org (hub.org [200.46.204.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id C91A343D3F for ; Sat, 30 Oct 2004 16:20:15 +0000 (GMT) (envelope-from scrappy@hub.org) Received: from localhost (unknown [200.46.204.144]) by hub.org (Postfix) with ESMTP id 0F2BB12B0E5 for ; Sat, 30 Oct 2004 13:20:10 -0300 (ADT) Received: from hub.org ([200.46.204.220]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 16868-02 for ; Sat, 30 Oct 2004 16:20:09 +0000 (GMT) Received: from ganymede.hub.org (blk-222-46-91.eastlink.ca [24.222.46.91]) by hub.org (Postfix) with ESMTP id A800F12B0E3 for ; Sat, 30 Oct 2004 13:20:09 -0300 (ADT) Received: by ganymede.hub.org (Postfix, from userid 1000) id ADF233D1FA; Sat, 30 Oct 2004 13:20:08 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id AC9113D185 for ; Sat, 30 Oct 2004 13:20:08 -0300 (ADT) Date: Sat, 30 Oct 2004 13:20:08 -0300 (ADT) From: "Marc G. Fournier" To: freebsd-stable@freebsd.org Message-ID: <20041030131002.O6085@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by amavisd-new at hub.org Subject: vnode 'leak' in 4.x ... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Oct 2004 16:20:17 -0000 A little while ago, I reported a suspicion that vnodes just weren't being freed up on long running servers ... after 55days of uptime on one of my servers, here is what I'm dealing with ... 793 'samples' today (one every minute) 786 with vnlru in a vlrup state I shutdown all of the VMs running on the large hard drive (the only place unionfs is being used) and umount'd the drive ... there were some suggested back then that this might/should free everything back up again ... but it didn't: Oct 30 13:06:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57966 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:07:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 57268 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:08:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 52335 - debug.vnlru_nowhere: 209679 - vlruwt Oct 30 13:09:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 50228 - debug.vnlru_nowhere: 209682 - vlrup Oct 30 13:10:01 venus root: debug.numvnodes: 522265 - debug.freevnodes: 44407 - debug.vnlru_nowhere: 209690 - vlrup Oct 30 13:11:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 35424 - debug.vnlru_nowhere: 209697 - vlrup Oct 30 13:12:02 venus root: debug.numvnodes: 522265 - debug.freevnodes: 34626 - debug.vnlru_nowhere: 209708 - vlrup Oct 30 13:13:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 29214 - debug.vnlru_nowhere: 209727 - vlrup Oct 30 13:14:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 24414 - debug.vnlru_nowhere: 209746 - vlrup Oct 30 13:15:00 venus root: debug.numvnodes: 522265 - debug.freevnodes: 26994 - debug.vnlru_nowhere: 209766 - vlrup The 'vlruwt' states above are while I had everything shutdown ... the vlrup's all started again after I mounted the drive and started to restart the VMs themselves ... I expect a high # of vnodes to be used ... that isn't the issue ... the issue is that even getting rid of the major mount point, so that only /, /tmp, /usr, /var are left up, the large # of vnodes that are in use on that mount point aren't being freed by vnlru :( I hate to reboot the server, but it looks like I've got no choice at this point ... is there something else that I can do, in 50 days or so, to provide more information? Thanks ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664