From owner-freebsd-hackers Sat Sep 22 13:23:34 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id 55EDD37B418 for ; Sat, 22 Sep 2001 13:23:25 -0700 (PDT) Received: (from dillon@localhost) by earth.backplane.com (8.11.6/8.11.2) id f8MKMpU81911; Sat, 22 Sep 2001 13:22:51 -0700 (PDT) (envelope-from dillon) Date: Sat, 22 Sep 2001 13:22:51 -0700 (PDT) From: Matt Dillon Message-Id: <200109222022.f8MKMpU81911@earth.backplane.com> To: Seigo Tanimura , phk@critter.freebsd.dk, tanimura@r.dl.itc.u-tokyo.ac.jp, bright@wintelcom.net Cc: hackers@freebsd.org Subject: More on the cache_purgeleafdirs() routine References: <200104161024.f3GAO7Z34787@rina.r.dl.itc.u-tokyo.ac.jp> <1198.987417363@critter> <200104161223.f3GCNZZ51680@rina.r.dl.itc.u-tokyo.ac.jp> Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi guys. I've been tracking down issues with vnode recycling.. specifically, getnewvnode() deadlocks in -stable in large-memory configurations. I took a real good look at cache_purgeleafdirs() in -current to see if it was MFCable as a solution. But.... There are a number of issues... well, there is really one big issue, and that is the simple fact that there can be upwards of 260,000+ entries in the name cache and cache_purgeleafdirs() doesn't scale. It is an O(N*M) algorithm. Any system that requires a great deal of vnode recycling -- for example Yahoo's userbase lookup (one file per userid) would be terribly impacted by this algorithm. It seems to me that the best way to deal with this is to simply have getnewvnode() zap the namei-cache, and have vfs_lookup() requeue (for LRU purposes) any namei cache associated vnodes that are on the freelist. The patch is much less complex... here's is a preliminary patch (without the LRU requeueing, I haven't gotten to that yet, and I haven't completely removed the non-scaleable recycling code). This code seems to work quite well even without the LRU requeueing, especialy if you turn on vfs.vmiodirenable. I don't see any reason to be totally rabid about keeping top level directory entries in-core... at least not at the cost of the current algorithm in -current. We want to try to keep them in core, but I believe that LRU requeueing in vfs_lookup() may be sufficient for that... or possibly even something as simple as a flag in the namei caching structure that guarentees the first N directory levels are left in-core. Or something like that. But not a complete scan of the namei cache! -Matt Index: kern/vfs_cache.c =================================================================== RCS file: /home/ncvs/src/sys/kern/vfs_cache.c,v retrieving revision 1.61 diff -u -r1.61 vfs_cache.c --- kern/vfs_cache.c 2001/09/12 08:37:46 1.61 +++ kern/vfs_cache.c 2001/09/22 20:13:47 @@ -101,8 +101,10 @@ SYSCTL_ULONG(_debug, OID_AUTO, numcache, CTLFLAG_RD, &numcache, 0, ""); static u_long numcachehv; /* number of cache entries with vnodes held */ SYSCTL_ULONG(_debug, OID_AUTO, numcachehv, CTLFLAG_RD, &numcachehv, 0, ""); +#if 0 static u_long numcachepl; /* number of cache purge for leaf entries */ SYSCTL_ULONG(_debug, OID_AUTO, numcachepl, CTLFLAG_RD, &numcachepl, 0, ""); +#endif struct nchstats nchstats; /* cache effectiveness statistics */ static int doingcache = 1; /* 1 => enable the cache */ @@ -476,6 +478,20 @@ } /* + * Flush the namei cache references associated with a vnode. + * The vnode remains valid. + */ +void +cache_flush(vp) + struct vnode *vp; +{ + while (!LIST_EMPTY(&vp->v_cache_src)) + cache_zap(LIST_FIRST(&vp->v_cache_src)); + while (!TAILQ_EMPTY(&vp->v_cache_dst)) + cache_zap(TAILQ_FIRST(&vp->v_cache_dst)); +} + +/* * Flush all entries referencing a particular filesystem. * * Since we need to check it anyway, we will flush all the invalid @@ -499,6 +515,8 @@ } } +#if 0 + /* * Flush all dirctory entries with no child directories held in * the cache. @@ -554,6 +572,8 @@ } numcachepl++; } + +#endif /* * Perform canonical checks and cache lookup and pass on to filesystem Index: kern/vfs_subr.c =================================================================== RCS file: /home/ncvs/src/sys/kern/vfs_subr.c,v retrieving revision 1.319 diff -u -r1.319 vfs_subr.c --- kern/vfs_subr.c 2001/09/12 08:37:47 1.319 +++ kern/vfs_subr.c 2001/09/22 20:15:11 @@ -110,6 +110,8 @@ /* Number of vnodes in the free list. */ static u_long freevnodes = 0; SYSCTL_LONG(_debug, OID_AUTO, freevnodes, CTLFLAG_RD, &freevnodes, 0, ""); + +#if 0 /* Number of vnode allocation. */ static u_long vnodeallocs = 0; SYSCTL_LONG(_debug, OID_AUTO, vnodeallocs, CTLFLAG_RD, &vnodeallocs, 0, ""); @@ -125,6 +127,7 @@ /* Number of vnodes attempted to recycle at a time. */ static u_long vnoderecyclenumber = 3000; SYSCTL_LONG(_debug, OID_AUTO, vnoderecyclenumber, CTLFLAG_RW, &vnoderecyclenumber, 0, ""); +#endif /* * Various variables used for debugging the new implementation of @@ -556,8 +559,13 @@ * Don't recycle if active in the namecache or * if it still has cached pages or we cannot get * its interlock. + * + * XXX the namei cache can hold onto vnodes too long, + * causing us to run out of MALLOC space. Instead, we + * should make path lookups requeue any vnodes on the free + * list. */ - if (LIST_FIRST(&vp->v_cache_src) != NULL || + if (/* LIST_FIRST(&vp->v_cache_src) != NULL || */ (VOP_GETVOBJECT(vp, &object) == 0 && (object->resident_page_count || object->ref_count)) || !mtx_trylock(&vp->v_interlock)) { @@ -636,6 +644,7 @@ vfs_object_create(vp, td, td->td_proc->p_ucred); +#if 0 vnodeallocs++; if (vnodeallocs % vnoderecycleperiod == 0 && freevnodes < vnoderecycleminfreevn && @@ -643,6 +652,7 @@ /* Recycle vnodes. */ cache_purgeleafdirs(vnoderecyclenumber); } +#endif return (0); } Index: sys/vnode.h =================================================================== RCS file: /home/ncvs/src/sys/sys/vnode.h,v retrieving revision 1.157 diff -u -r1.157 vnode.h --- sys/vnode.h 2001/09/13 22:52:42 1.157 +++ sys/vnode.h 2001/09/22 20:18:45 @@ -558,8 +558,8 @@ int cache_lookup __P((struct vnode *dvp, struct vnode **vpp, struct componentname *cnp)); void cache_purge __P((struct vnode *vp)); +void cache_flush __P((struct vnode *vp)); void cache_purgevfs __P((struct mount *mp)); -void cache_purgeleafdirs __P((int ndir)); void cvtstat __P((struct stat *st, struct ostat *ost)); void cvtnstat __P((struct stat *sb, struct nstat *nsb)); int getnewvnode __P((enum vtagtype tag, To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message