From owner-freebsd-current@FreeBSD.ORG Mon Feb 4 12:11:51 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 56848A8B; Mon, 4 Feb 2013 12:11:51 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qe0-f48.google.com (mail-qe0-f48.google.com [209.85.128.48]) by mx1.freebsd.org (Postfix) with ESMTP id DC5F31733; Mon, 4 Feb 2013 12:11:50 +0000 (UTC) Received: by mail-qe0-f48.google.com with SMTP id 3so2749030qea.21 for ; Mon, 04 Feb 2013 04:11:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=g0yF8AhO0/zzWP9jM/TLA4ViA7pflKtVnz8IcQosn0o=; b=Hpk0ushgfshW6Z0NgqUeNqLjEYed1fESdUojkMXUYMn6fiCod75sZ+rhQrSxiuk7A8 WLlZ7tycliNEIrmLQXq2y8L/W7XYZfN4hATKTaCfDCnQwS1KMTahkz1ODRR88xE/QrUZ fgrq/v/GL8qsCI/orw4eh5M4/zCpAusMBUUFzJlSKufen5h+LZLWr9o3wvvsBWp3NRnL bKIJRczgaCbKX7Gh4wQ7la4UgjrMhlhA2c3OUmn/7gKymwY9XnpXbdURF0TjHLS4p2p3 V+ErO6ch/ktiQ7vF0BDCaSbM6CqEeZUnmtlmxdbJZGqLPc6+mU4Kj+G+sgsCvRhWkDRA YHBA== MIME-Version: 1.0 X-Received: by 10.49.74.10 with SMTP id p10mr20756028qev.35.1359979910100; Mon, 04 Feb 2013 04:11:50 -0800 (PST) Received: by 10.229.105.201 with HTTP; Mon, 4 Feb 2013 04:11:49 -0800 (PST) In-Reply-To: <1982774587.2641177.1359942617191.JavaMail.root@erie.cs.uoguelph.ca> References: <20130203110759.GM2522@kib.kiev.ua> <1982774587.2641177.1359942617191.JavaMail.root@erie.cs.uoguelph.ca> Date: Mon, 4 Feb 2013 15:11:49 +0300 Message-ID: Subject: Re: panic: LK_RETRY set with incompatible flags From: Sergey Kandaurov To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , FreeBSD Current , Andriy Gapon X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2013 12:11:51 -0000 On 4 February 2013 05:50, Rick Macklem wrote: > Konstantin Belousov wrote: >> On Sat, Feb 02, 2013 at 09:30:39PM -0500, Rick Macklem wrote: >> > Andriy Gapon wrote: >> > > on 31/01/2013 15:29 Sergey Kandaurov said the following: >> > > > Hi. >> > > > >> > > > Got this assertion on idle NFS server while `ls -la >> > > > /.zfs/shares/' >> > > > issued on NFS client. >> > > > kern/vfs_vnops.c:_vn_lock() >> > > > KASSERT((flags & LK_RETRY) == 0 || error == 0, >> > > > ("LK_RETRY set with incompatible flags >> > > > (0x%x) or >> > > > an error occured (%d)", >> > > > >> > > > panic: LK_RETRY set with incompatible flags (0x200400) or an >> > > > error >> > > > occured (11) >> > > > >> > > > What does that mean and how is it possible? As you can see, both >> > > > parts >> > > > of assertion failed. >> > > > 11 is EDEADLK >> > > > 0x200400: LK_RETRY & LK_UPGRADE >> > > >> > > LK_SHARED, not LK_UPGRADE. >> > > Apparently the thread already holds an exlusive lock on the vnode, >> > > which you >> > > confirm below. >> > > >> > > >> > > > Tracing pid 2943 tid 101532 td 0xfffffe004f5b7000 >> > > > kdb_enter() at kdb_enter+0x3e/frame 0xffffff848e45ef50 >> > > > vpanic() at vpanic+0x147/frame 0xffffff848e45ef90 >> > > > kassert_panic() at kassert_panic+0x136/frame 0xffffff848e45f000 >> > > > _vn_lock() at _vn_lock+0x70/frame 0xffffff848e45f070 >> > > > zfs_lookup() at zfs_lookup+0x392/frame 0xffffff848e45f100 >> > > > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame >> > > > 0xffffff848e45f240 >> > > > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xc2/frame >> > > > 0xffffff848e45f260 >> > > > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame >> > > > 0xffffff848e45f2b0 >> > > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc2/frame 0xffffff848e45f2d0 >> > > > lookup() at lookup+0x548/frame 0xffffff848e45f350 >> > > > nfsvno_namei() at nfsvno_namei+0x1a5/frame 0xffffff848e45f400 >> > > > nfsrvd_lookup() at nfsrvd_lookup+0x13a/frame 0xffffff848e45f6b0 >> > > > nfsrvd_dorpc() at nfsrvd_dorpc+0xca5/frame 0xffffff848e45f8a0 >> > > > nfssvc_program() at nfssvc_program+0x482/frame >> > > > 0xffffff848e45fa00 >> > > > svc_run_internal() at svc_run_internal+0x1e9/frame >> > > > 0xffffff848e45fba0 >> > > > svc_thread_start() at svc_thread_start+0xb/frame >> > > > 0xffffff848e45fbb0 >> > > > fork_exit() at fork_exit+0x84/frame 0xffffff848e45fbf0 >> > > > fork_trampoline() at fork_trampoline+0xe/frame >> > > > 0xffffff848e45fbf0 >> > > > --- trap 0xc, rip = 0x800883e9a, rsp = 0x7fffffffd488, rbp = >> > > > 0x7fffffffd730 --- >> > > > >> > > > db> show lockedvnods >> > > > Locked vnodes >> > > > >> > > > 0xfffffe02e21b11d8: tag zfs, type VDIR >> > > > usecount 4, writecount 0, refcount 4 mountedhere 0 >> > > > flags (VI_ACTIVE) >> > > > v_object 0xfffffe02d9f2eb40 ref 0 pages 0 >> > > > lock type zfs: EXCL by thread 0xfffffe004f5b7000 (pid 2943, >> > > > nfsd, >> > > > tid 101532) >> > > > >> > > > >> > > > >> > I just took a look at zfs_vnops.c and I am probably missing >> > something, >> > but I can't see how this ever worked for a lookup of ".." when at >> > the >> > root (unless ZFS doesn't do the ".." is the current directory when >> > at >> > the root). >> > >> > Here's the code snippet: >> > 1442 if (error == 0 && (nm[0] != '.' || nm[1] != '\0')) { >> > 1443 int ltype = 0; >> > 1444 >> > 1445 if (cnp->cn_flags & ISDOTDOT) { >> > 1446 ltype = VOP_ISLOCKED(dvp); >> > 1447 VOP_UNLOCK(dvp, 0); >> > 1448 } >> > 1449 ZFS_EXIT(zfsvfs); >> > 1450 error = zfs_vnode_lock(*vpp, cnp->cn_lkflags); >> > 1451 if (cnp->cn_flags & ISDOTDOT) >> > 1452 vn_lock(dvp, ltype | LK_RETRY); >> > 1453 if (error != 0) { >> > 1454 VN_RELE(*vpp); >> > 1455 *vpp = NULL; >> > 1456 return (error); >> > 1457 } >> > >> > Maybe line# 1451 should be changed to: >> > if ((cnp->cn_flags & ISDOTDOT) && *vpp != dvp) >> > >> > I'm not at all familiar with ZFS, so I've probably way >> > off the mark on this, rick >> > ps: I hope kib and jhb don't mind being added as cc's, since >> > they are familiar with this stuff, although maybe not ZFS >> > specifics. >> >> VFS (should) never call VOP_LOOKUP for the dotdot and root vnode. >> The logic in the lookup() prevents it. >> > Correcting my previous posts, like usual. If you look at the above snippet of > code, it seems that zfs_lock_vnode() must lock the same vnode as > *dvp. Notice that vn_lock() is only called when ISDOTDOT is set and the > code does a VOP_UNLOCK(dvp, 0); for this case, just before the > zfs_vnode_lock(). > > This assumes that the vn_lock() call at #1452 causes the panic. > This is the only vn_lock() call in zfs_lookup(), so unless the compiler > has done something weird, it seems the case. > > Sergey, do you have this kernel handy? If so, maybe you could check the > line# for zfs_lookup+0x392. (If you haven't done this before, just email > and I'll give you the steps. You just need the kernel.symbols file for > the kernel that panic'd.) Yep, kgdb returned zfs_vnops.c:1453. -- wbr, pluknet