From owner-freebsd-current@FreeBSD.ORG Thu Apr 12 04:55:02 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AAD5216A406 for ; Thu, 12 Apr 2007 04:55:02 +0000 (UTC) (envelope-from simokawa@freebsd.org) Received: from mail3.ecc.u-tokyo.ac.jp (mail3.ecc.u-tokyo.ac.jp [133.11.205.99]) by mx1.freebsd.org (Postfix) with ESMTP id 570DC13C455 for ; Thu, 12 Apr 2007 04:55:02 +0000 (UTC) (envelope-from simokawa@freebsd.org) Received: from mail1.ecc.u-tokyo.ac.jp (mail1.ecc.u-tokyo.ac.jp [133.11.50.203]) by mail3.ecc.u-tokyo.ac.jp (Postfix) with ESMTP id B1CA05B1489 for ; Thu, 12 Apr 2007 13:23:41 +0900 (JST) Received: from spam005.ecc.u-tokyo.ac.jp (spam005.ecc.u-tokyo.ac.jp [133.11.50.198]) by mail1.ecc.u-tokyo.ac.jp (Postfix) with ESMTP id D5FC82A2B07 for ; Thu, 12 Apr 2007 13:23:40 +0900 (JST) Received: from maru5.nunu.org (157.82.169.72 [157.82.169.72]) by spam005.ecc.u-tokyo.ac.jp (SpamBlock.pst 3.4.97) with ESMTP id <868xcyusp5.wl%simokawa@FreeBSD.ORG> for ; Thu, 12 Apr 2007 13:23:19 +0900 Date: Thu, 12 Apr 2007 13:23:18 +0900 Message-ID: <868xcyusp5.wl%simokawa@FreeBSD.ORG> From: Hidetoshi Shimokawa To: pjd@freebsd.org User-Agent: Wanderlust/2.15.2 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 Emacs/22.0.50 (i386-unknown-freebsd5.4) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-IP: 157.82.169.72 X-FROM-DOMAIN: freebsd.org X-FROM-EMAIL: simokawa@freebsd.org Cc: freebsd-current@freebsd.org Subject: ZFS vnode starvation with large kmem_size X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Apr 2007 04:55:02 -0000 Hi Pawel, I observed vnode starvation with large kmem_size in loader.conf. It seems that even if zfs eats vnodes up to maxvnodes, vnodes cannot be reclaimed and vnlru kthread will give up. As far as I understand, the problem comes from difference of name cache mechanism. I attached a patch for a workaround. How to repeat: arch: amd64 or i386 physical memory: 512MB vm.kmem_size=671088640 ( In this setting: kern.maxvnodes: 35077 vfs.zfs.dnlc.ncsize: 33095 ) Create zfs filesystem and extract ports.tar.gz(many files) on it. After a while you can see that the system is not responsive because of vnode starvation. Workarounds: 1. reduce vfs.zfs.dnlc.ncsize or 2. apply the following patch Index: dnlc.c =================================================================== RCS file: /home/ncvs/src/sys/contrib/opensolaris/uts/common/fs/dnlc.c,v retrieving revision 1.1 diff -u -r1.1 dnlc.c --- dnlc.c 6 Apr 2007 01:09:02 -0000 1.1 +++ dnlc.c 12 Apr 2007 04:18:34 -0000 @@ -703,6 +703,9 @@ dnlc_get(uchar_t namlen) { ncache_t *ncp; + extern int desiredvnodes; + extern int numvnodes; + static uint_t wantvnodes_reduce_percent = 10; if (dnlc_nentries > dnlc_max_nentries) { dnlc_max_nentries_cnt++; /* keep a statistic */ @@ -714,7 +717,10 @@ } ncp->namlen = namlen; atomic_add_32(&dnlc_nentries, 1); - dnlc_reduce_cache(NULL); + if (numvnodes > desiredvnodes) + dnlc_reduce_cache((void *)&wantvnodes_reduce_percent); + else + dnlc_reduce_cache(NULL); return (ncp); } I know this kmem_size must be too large but I think zfs should cooperate with existent vnode management. Except this problem, zfs works very well on my machines. Thank you for your great work! /\ Hidetoshi Shimokawa \/ simokawa@FreeBSD.ORG