From owner-freebsd-bugs@FreeBSD.ORG Fri May 2 05:20:12 2003 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2730E37B401 for ; Fri, 2 May 2003 05:20:12 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id A388243FBF for ; Fri, 2 May 2003 05:20:10 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with ESMTP id h42CKAUp007467 for ; Fri, 2 May 2003 05:20:10 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.9/8.12.9/Submit) id h42CKA3Q007466; Fri, 2 May 2003 05:20:10 -0700 (PDT) Resent-Date: Fri, 2 May 2003 05:20:10 -0700 (PDT) Resent-Message-Id: <200305021220.h42CKA3Q007466@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Michael van Elst Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5103D37B401 for ; Fri, 2 May 2003 05:15:08 -0700 (PDT) Received: from sv1.dev.de.cw.net (sv1.dev.de.cw.net [141.1.23.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id C59D943F75 for ; Fri, 2 May 2003 05:15:07 -0700 (PDT) (envelope-from mlelstv@sv1.dev.de.cw.net) Received: by sv1.dev.de.cw.net (Postfix, from userid 10012) id 026FD8D7F5; Fri, 2 May 2003 14:15:05 +0200 (CEST) Message-Id: <20030502121505.026FD8D7F5@sv1.dev.de.cw.net> Date: Fri, 2 May 2003 14:13:46 +0200 (CEST) From: Michael van Elst To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: kern/51685: Unbounded inode allocation causes kernel to lock up X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Michael van Elst List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2003 12:20:12 -0000 >Number: 51685 >Category: kern >Synopsis: Unbounded inode allocation causes kernel to lock up >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri May 02 05:20:09 PDT 2003 >Closed-Date: >Last-Modified: >Originator: Michael van Elst >Release: FreeBSD 4.8-RELEASE i386 >Organization: C&W >Environment: System: FreeBSD pg6.dev.de.cw.net 4.8-RELEASE FreeBSD 4.8-RELEASE #0: Thu Apr 3 10:53:38 GMT 2003 root@freebsd-stable.sentex.ca:/usr/obj/usr/src/sys/GENERIC i386 >Description: For every cached filesystem object a 'vnode' and an associated 'inode' are allocated in memory. The inodes are allocated in the 'FFS node' memory region. When a vnode is no longer used it is either explicitely released to a free list or reclaimed by a kernel thread. The reclaim operation in vnlru_proc() also releases the associated filesystem data. The release operation done by calling vput() or vrele() on the other hand leave the associated filesystem data intact. It is released only when a new vnode needs to be allocated by recycling a vnode from the free list. The 'FFS node' region is limited to about 100MB, when it is filled up the kernel sleeps until some process has freed memory in that region. When no other process is allocating new vnodes this can only happen by vnlru_proc() pushing vnodes to the free list. vnlru_proc() however only runs when the number of vnodes that are not on the free list exceeds the system parameter 'kern.maxvnodes'. When the inode memory allocated for the remaining 'kern.maxvnodes' vnodes and for all vnodes on the free list that are still associated with filesystem data reaches the 100MB limit the system freezes. The vfs.vmiodirenable setting only amplifies the problem. Without directory caching you need to access more files to reach the limit. FreeBSD5 has the same issue, however the critical memory region is now handled by UMA and has no artificial limit. It is still limited to the size of a UMA zone. >How-To-Repeat: - run a FreeBSD4 system with vfs.vmiodirenable=1 - on a FFS filesystem create a directory with about one million files or subdirectories. - do a 'find -ls' over this directory or wait for some periodic task that does similar (e.g. weekly/310.locate). - watch the system slow down and lock up >Fix: No fix known, but some ideas: If vrele() and vput() would simply drop a vnode it could always be reclaimed by vnlru_proc(). Caching would be done only in the vnode layer but not also in the ffs layer. As an alternative vlnru_proc() could use a heuristic that prevents a resource shortage in the ffs layer, something like 'desiredusedvnodes' that is checked against the number of vnodes that carry filesystem specific data. >Release-Note: >Audit-Trail: >Unformatted: