From owner-freebsd-current@FreeBSD.ORG Wed Jun 18 01:32:25 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1060537B401 for ; Wed, 18 Jun 2003 01:32:25 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 420EB43F3F for ; Wed, 18 Jun 2003 01:32:24 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9/8.12.9) with ESMTP id h5I8WHM7054008 for ; Wed, 18 Jun 2003 01:32:21 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <200306180832.h5I8WHM7054008@gw.catspoiler.org> Date: Wed, 18 Jun 2003 01:32:17 -0700 (PDT) From: Don Lewis To: current@FreeBSD.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Subject: fun with WITNESS and "pool mutex" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 08:32:25 -0000 When I was attempting to debug a system deadlock problem where the culprit process was sleeping on a "pool mutex", I noticed that "show witness" in ddb doesn't report anything about this particular mutex flavor. I discovered that witness doesn't monitor these mutexes because mtx_pool_setup() calls mtx_init with the MTX_NOWITNESS flag. These are a mutexes bit special, because they are supposed to be leaf mutexes and no other mutexes should be grabbed after them. The deadlock in question caused me to discover a violation of this restriction, so I wondered if there were more problems of this type in the code. I suspected there would be, since there haven't been any automatic checks of to verify that these mutexes are being used correctly. Just for grins, I removed the MTX_NOWITNESS flag from mtx_pool_setup() and quickly found the first violation during the boot sequence: Mounting root from ufs:/dev/da0s1a acquiring duplicate lock of same type: "pool mutex" 1st pool mutex @ /usr/src/sys/kern/vfs_syscalls.c:736 2nd pool mutex @ /usr/src/sys/kern/kern_lock.c:598 Stack backtrace: backtrace(c051f4df,c051c041,c051adcc,256,c05e4808) at backtrace+0x17 witness_lock(c05e0cf0,8,c051adcc,256,c05e2ae0) at witness_lock+0x697 _mtx_lock_flags(c05e0cf0,0,c051adcc,256,c641a248) at _mtx_lock_flags+0xb1 lockstatus(c641a304,0,e4b1ab24,c036f2e8,e4b1ab44) at lockstatus+0x3c vop_stdislocked(e4b1ab44,e4b1ab30,c0464f78,e4b1ab44,e4b1ab58) at vop_stdislocked +0x21 vop_defaultop(e4b1ab44,e4b1ab58,c0375a67,e4b1ab44,c05e4808) at vop_defaultop+0x1 8 ufs_vnoperate(e4b1ab44,c05e4808,c05740ac,c05b67e0,c641a248) at ufs_vnoperate+0x1 8 assert_vop_locked(c641a248,c05197fc,c05b74e0,c641a248,0) at assert_vop_locked+0x 47 VOP_GETVOBJECT(c641a248,0,c0524224,2e0,c05740ac) at VOP_GETVOBJECT+0x3f kern_open(c61c64c0,bfbff6b0,0,1,0) at kern_open+0x44f open(c61c64c0,e4b1ad10,c0537dc3,3fd,3) at open+0x30 syscall(2f,2f,2f,1,0) at syscall+0x26e Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (5), eip = 0x80529ef, esp = 0xbfbff16c, ebp = 0xbfbff208 --- The code in question: FILEDESC_LOCK(fdp); FILE_LOCK(fp); if (fp->f_count == 1) { KASSERT(fdp->fd_ofiles[indx] != fp, ("Open file descriptor lost all refs")); FILEDESC_UNLOCK(fdp); FILE_UNLOCK(fp); VOP_UNLOCK(vp, 0, td); vn_close(vp, flags & FMASK, fp->f_cred, td); fdrop(fp, td); td->td_retval[0] = indx; return 0; } /* assert that vn_open created a backing object if one is needed */ KASSERT(!vn_canvmio(vp) || VOP_GETVOBJECT(vp, NULL) == 0, ("open: vmio vnode has no backing object after vn_open")); fp->f_data = vp; fp->f_flag = flags & FMASK; fp->f_ops = &vnops; fp->f_type = (vp->v_type == VFIFO ? DTYPE_FIFO : DTYPE_VNODE); FILEDESC_UNLOCK(fdp); FILE_UNLOCK(fp); This one appears to be easily fixable by moving the second KASSERT down a few lines to below the FILE_UNLOCK() call. Any bets on how many other potential deadlock problems there are in the tree?