From owner-freebsd-current@FreeBSD.ORG Mon Feb 4 01:50:18 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E90ACC7; Mon, 4 Feb 2013 01:50:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 316A5658; Mon, 4 Feb 2013 01:50:17 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAEQTD1GDaFvO/2dsb2JhbAA4BAiGSLUPg3Vzgh8BAQQBHQYEUgUWDgoCAg0ZAlkGEYgNBq0skVaBI4tyAQQMgxmBEwOIZo05iVWGfIMaIYEwNQ X-IronPort-AV: E=Sophos;i="4.84,596,1355115600"; d="scan'208";a="14897909" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 03 Feb 2013 20:50:17 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 33857B3F26; Sun, 3 Feb 2013 20:50:17 -0500 (EST) Date: Sun, 3 Feb 2013 20:50:17 -0500 (EST) From: Rick Macklem To: Konstantin Belousov Message-ID: <1982774587.2641177.1359942617191.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20130203110759.GM2522@kib.kiev.ua> Subject: Re: panic: LK_RETRY set with incompatible flags MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: FreeBSD Current , Sergey Kandaurov , Andriy Gapon X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2013 01:50:19 -0000 Konstantin Belousov wrote: > On Sat, Feb 02, 2013 at 09:30:39PM -0500, Rick Macklem wrote: > > Andriy Gapon wrote: > > > on 31/01/2013 15:29 Sergey Kandaurov said the following: > > > > Hi. > > > > > > > > Got this assertion on idle NFS server while `ls -la > > > > /.zfs/shares/' > > > > issued on NFS client. > > > > kern/vfs_vnops.c:_vn_lock() > > > > KASSERT((flags & LK_RETRY) == 0 || error == 0, > > > > ("LK_RETRY set with incompatible flags > > > > (0x%x) or > > > > an error occured (%d)", > > > > > > > > panic: LK_RETRY set with incompatible flags (0x200400) or an > > > > error > > > > occured (11) > > > > > > > > What does that mean and how is it possible? As you can see, both > > > > parts > > > > of assertion failed. > > > > 11 is EDEADLK > > > > 0x200400: LK_RETRY & LK_UPGRADE > > > > > > LK_SHARED, not LK_UPGRADE. > > > Apparently the thread already holds an exlusive lock on the vnode, > > > which you > > > confirm below. > > > > > > > > > > Tracing pid 2943 tid 101532 td 0xfffffe004f5b7000 > > > > kdb_enter() at kdb_enter+0x3e/frame 0xffffff848e45ef50 > > > > vpanic() at vpanic+0x147/frame 0xffffff848e45ef90 > > > > kassert_panic() at kassert_panic+0x136/frame 0xffffff848e45f000 > > > > _vn_lock() at _vn_lock+0x70/frame 0xffffff848e45f070 > > > > zfs_lookup() at zfs_lookup+0x392/frame 0xffffff848e45f100 > > > > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame > > > > 0xffffff848e45f240 > > > > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xc2/frame > > > > 0xffffff848e45f260 > > > > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame > > > > 0xffffff848e45f2b0 > > > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc2/frame 0xffffff848e45f2d0 > > > > lookup() at lookup+0x548/frame 0xffffff848e45f350 > > > > nfsvno_namei() at nfsvno_namei+0x1a5/frame 0xffffff848e45f400 > > > > nfsrvd_lookup() at nfsrvd_lookup+0x13a/frame 0xffffff848e45f6b0 > > > > nfsrvd_dorpc() at nfsrvd_dorpc+0xca5/frame 0xffffff848e45f8a0 > > > > nfssvc_program() at nfssvc_program+0x482/frame > > > > 0xffffff848e45fa00 > > > > svc_run_internal() at svc_run_internal+0x1e9/frame > > > > 0xffffff848e45fba0 > > > > svc_thread_start() at svc_thread_start+0xb/frame > > > > 0xffffff848e45fbb0 > > > > fork_exit() at fork_exit+0x84/frame 0xffffff848e45fbf0 > > > > fork_trampoline() at fork_trampoline+0xe/frame > > > > 0xffffff848e45fbf0 > > > > --- trap 0xc, rip = 0x800883e9a, rsp = 0x7fffffffd488, rbp = > > > > 0x7fffffffd730 --- > > > > > > > > db> show lockedvnods > > > > Locked vnodes > > > > > > > > 0xfffffe02e21b11d8: tag zfs, type VDIR > > > > usecount 4, writecount 0, refcount 4 mountedhere 0 > > > > flags (VI_ACTIVE) > > > > v_object 0xfffffe02d9f2eb40 ref 0 pages 0 > > > > lock type zfs: EXCL by thread 0xfffffe004f5b7000 (pid 2943, > > > > nfsd, > > > > tid 101532) > > > > > > > > > > > > > > I just took a look at zfs_vnops.c and I am probably missing > > something, > > but I can't see how this ever worked for a lookup of ".." when at > > the > > root (unless ZFS doesn't do the ".." is the current directory when > > at > > the root). > > > > Here's the code snippet: > > 1442 if (error == 0 && (nm[0] != '.' || nm[1] != '\0')) { > > 1443 int ltype = 0; > > 1444 > > 1445 if (cnp->cn_flags & ISDOTDOT) { > > 1446 ltype = VOP_ISLOCKED(dvp); > > 1447 VOP_UNLOCK(dvp, 0); > > 1448 } > > 1449 ZFS_EXIT(zfsvfs); > > 1450 error = zfs_vnode_lock(*vpp, cnp->cn_lkflags); > > 1451 if (cnp->cn_flags & ISDOTDOT) > > 1452 vn_lock(dvp, ltype | LK_RETRY); > > 1453 if (error != 0) { > > 1454 VN_RELE(*vpp); > > 1455 *vpp = NULL; > > 1456 return (error); > > 1457 } > > > > Maybe line# 1451 should be changed to: > > if ((cnp->cn_flags & ISDOTDOT) && *vpp != dvp) > > > > I'm not at all familiar with ZFS, so I've probably way > > off the mark on this, rick > > ps: I hope kib and jhb don't mind being added as cc's, since > > they are familiar with this stuff, although maybe not ZFS > > specifics. > > VFS (should) never call VOP_LOOKUP for the dotdot and root vnode. > The logic in the lookup() prevents it. > Correcting my previous posts, like usual. If you look at the above snippet of code, it seems that zfs_lock_vnode() must lock the same vnode as *dvp. Notice that vn_lock() is only called when ISDOTDOT is set and the code does a VOP_UNLOCK(dvp, 0); for this case, just before the zfs_vnode_lock(). This assumes that the vn_lock() call at #1452 causes the panic. This is the only vn_lock() call in zfs_lookup(), so unless the compiler has done something weird, it seems the case. Sergey, do you have this kernel handy? If so, maybe you could check the line# for zfs_lookup+0x392. (If you haven't done this before, just email and I'll give you the steps. You just need the kernel.symbols file for the kernel that panic'd.) I'll take a look at zfs_dirlookup() to see if I can spot any case where it will return the same vnode as the dvp argument. rick > Might it be a missing check in the nfs server code ? (I did not looked > yet).