Date: Fri, 14 Apr 2000 14:20:37 -0400 From: Dave Chapeskie <dchapes@borderware.com> To: freebsd-hackers@FreeBSD.ORG, freebsd-fs@FreeBSD.ORG Subject: vnode_free_list corruption [patch] Message-ID: <00Apr14.141908edt.117140@gateway.borderware.com>
next in thread | raw e-mail | index | archive | help
--GvXjxJ+pjyke8COw
Content-Type: text/plain; charset=us-ascii
Greetings.
I've been seeing a rash of "free vnode isn't" panics lately. Some
machines were panicing several times a day. Along with this we saw
occasional "object inconsistent state: RPC: %d, RC: %d" messages.
I was able to replicate the problem by running multiple (~8 on a
pentium 200 system with 32 MB of RAM) copies of each of the attached
simple shell scripts (with all output redirected to /dev/null). It
would often panic within 10-20 minutes.
I tracked the problem down to a race between getnewvnode() recycling a
vnode and vhold(). I found that vhold() was calling vbusy() for a vnode
with the VDOOMED flag set.
This is bad since getnewvnode() removes the vnode from the free list
before setting this flag so vbusy() is calling TAILQ_REMOVE for a vnode
that is not on the free list. This can easily result in corruption of
the free list pointers causing future getnewvnode() calls to find active
vnodes that it thinks are on the free list.
I added a panic in vbusy() if VDOOMED is set and this hit quite often
during my tests. Typically the call chain looked something like:
ffs_truncate
ffs_indirtrunc
getblk
bgetvp
vhold
vbusy
panic
With ffs_truncate often being called due to rename(2) or unlink(2).
I managed to solve the problem here by adding a VOP_ISLOCKED(vp) check
to getnewvnode() and skipping such vnodes instead of trying to recycle
them. From my searches of the mailing lists it appears I'm not the
first one to think of this but apparently this isn't guaranteed to work
for all files system types. I just know it works for the FFS problems I
was seeing.
At a minimum I'd highly recommend that someone commit a panic to vbusy()
for vnodes with VDOOMED set since letting it continue if that flag is
set can and does result in the corruption of the vnode_free_list. I'd
also recommend the addition of the VOP_ISLOCKED() check to getnewvnode()
even if it doesn't work for all file system types it will help in some
(most?) cases. A patch for CURRENT is attached.
--
Dave Chapeskie
Senior Software Engineer
Borderware Technologies Inc.
--GvXjxJ+pjyke8COw
Content-Type: application/x-sh
Content-Disposition: attachment; filename="cp_mv_loop.sh"
#!/bin/sh
cd /var/log
while true; do
mkdir copy; cp * copy
mkdir copy/2; mv copy/* copy/2
rm -rf copy
done
--GvXjxJ+pjyke8COw
Content-Type: application/x-sh
Content-Disposition: attachment; filename="find_loop.sh"
#!/bin/sh
FS="/tmp /var /usr"
while true; do
for f in $FS; do
find -x $f -type f -print | xargs -n 4 head -1
done
done
--GvXjxJ+pjyke8COw
Content-Type: application/x-sh
Content-Disposition: attachment; filename="ps_loop.sh"
#!/bin/sh
while true; do
ps -aux; ps -ajx
done
--GvXjxJ+pjyke8COw
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="vfs_subr.c.diff"
diff -u -t -r1.253 vfs_subr.c
--- kern/vfs_subr.c 2000/03/20 11:28:45 1.253
+++ kern/vfs_subr.c 2000/04/14 18:01:52
@@ -467,6 +467,8 @@
for (vp = TAILQ_FIRST(&vnode_tobefree_list); vp; vp = nvp) {
nvp = TAILQ_NEXT(vp, v_freelist);
+ if (!simple_lock_try(&vp->v_interlock))
+ continue;
TAILQ_REMOVE(&vnode_tobefree_list, vp, v_freelist);
if (vp->v_flag & VAGE) {
TAILQ_INSERT_HEAD(&vnode_free_list, vp, v_freelist);
@@ -478,6 +480,7 @@
if (vp->v_usecount)
panic("tobe free vnode isn't");
freevnodes++;
+ simple_unlock(&tvp->v_interlock);
}
if (wantfreevnodes && freevnodes < wantfreevnodes) {
@@ -507,6 +510,10 @@
/* Don't recycle if active in the namecache */
simple_unlock(&vp->v_interlock);
continue;
+ } else if (VOP_ISLOCKED(vp))
+ TAILQ_REMOVE(&vnode_free_list, vp, v_freelist);
+ TAILQ_INSERT_TAIL(&vnode_tmp_list, vp, v_freelist);
+ continue;
} else {
break;
}
@@ -2613,6 +2620,10 @@
int s;
s = splbio();
+ if (vp->v_flag & VDOOMED) {
+ vprint("vbusy", vp);
+ panic("vbusy on VDOOMED vnode");
+ }
simple_lock(&vnode_free_list_slock);
if (vp->v_flag & VTBFREE) {
TAILQ_REMOVE(&vnode_tobefree_list, vp, v_freelist);
--GvXjxJ+pjyke8COw--
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?00Apr14.141908edt.117140>