From owner-freebsd-current@FreeBSD.ORG Mon Feb 4 12:13:34 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 16A33BD4; Mon, 4 Feb 2013 12:13:34 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) by mx1.freebsd.org (Postfix) with ESMTP id AC5DE175F; Mon, 4 Feb 2013 12:13:33 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id b25so2623355qca.17 for ; Mon, 04 Feb 2013 04:13:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ApLUm/zW4HaVI/bUz3D/J0rfb6LV6YHiq+SzAg/sUak=; b=NuspONuU1saGFkE0TjwgynMdqDkrccK9aXDkiK1WzG38D2X8gMESwNOeK8CLygVxpo 5jLv3bmsSmL9OwR32Bg7AE5rZoUsHJW2cp1+JeYCeNkZuVRUB336Oql7CW3nvaSBn5kP Q6AGPflG5KEVhwL7w3ONdYkKF7SSzxrKionHCmlbhH0tw4U9wXGl3eIjf4sq31xmcdl5 qrvp4oomd0Qy2m1gKfUu/Fq1f/3/4JlGyg4MEzQt8KIDzh/1kM9Y7NwQ4Ayzdb9romGB jKOq4WoNwmGrL6IvtEr1eSD4qxCNoIKGXGrtTBrI1Mg6nkqtlyy9DivrzpmILmDxL8OS s4Xw== MIME-Version: 1.0 X-Received: by 10.224.27.77 with SMTP id h13mr18100913qac.23.1359980013098; Mon, 04 Feb 2013 04:13:33 -0800 (PST) Received: by 10.229.105.201 with HTTP; Mon, 4 Feb 2013 04:13:32 -0800 (PST) In-Reply-To: <510FA448.8020204@FreeBSD.org> References: <510E9877.5000701@FreeBSD.org> <1515954355.2640466.1359940065810.JavaMail.root@erie.cs.uoguelph.ca> <510FA448.8020204@FreeBSD.org> Date: Mon, 4 Feb 2013 15:13:32 +0300 Message-ID: Subject: Re: panic: LK_RETRY set with incompatible flags From: Sergey Kandaurov To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , Rick Macklem , FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2013 12:13:34 -0000 On 4 February 2013 16:06, Andriy Gapon wrote: > on 04/02/2013 13:49 Sergey Kandaurov said the following: >> Hi, Rick! Here is the requested info regarding witness, and a bit more. >> The triggered KASSERT is now different though. > > It's exactly the same problem though :-) > Do you have a crashdump? > If yes, please print **vpp. Yep, but it's in a bad form :( It has many bits optimized out (compiled with clang). I'll rebuild the kernel with -O or so and will try again. #8 0xffffffff808bc4ce in kdb_enter (why=0xffffffff80e7ed99 "panic", msg=) at cpufunc.h:63 #9 0xffffffff80888fb7 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:746 #10 0xffffffff80888e66 in kassert_panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:641 #11 0xffffffff808d2259 in witness_checkorder (lock=0xfffffe00a1b44240, ---Type to continue, or q to quit--- flags=1, file=0xffffffff81b2bb36 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c", line=1452, interlock=) at /usr/src/sys/kern/subr_witness.c:1122 #12 0xffffffff8086c11e in __lockmgr_args (lk=0xfffffe00a1b44240, flags=, ilk=0xfffffe00a1b44270, wmesg=0xffffffff81b2808d "zfs", pri=96, timo=51, file=0xffffffff80e8e407 "/usr/src/sys/kern/vfs_default.c", line=0) at /usr/src/sys/kern/kern_lock.c:511 #13 0xffffffff8091439c in vop_stdlock (ap=) at lockmgr.h:97 #14 0xffffffff80cb70c0 in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2022 #15 0xffffffff80932fbb in _vn_lock (vp=0xfffffe00a1b441d8, flags=, file=0xffffffff81b2bb36 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c", line=1452) at vnode_if.h:859 #16 0xffffffff81abd902 in zfs_lookup () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1452 #17 0xffffffff81abdc1d in zfs_freebsd_lookup (ap=0xffffff848e6c0270) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5864 ---Type to continue, or q to quit--- #18 0xffffffff80cb51b2 in VOP_CACHEDLOOKUP_APV (vop=, a=) at vnode_if.c:191 #19 0xffffffff8091177f in vfs_cache_lookup (ap=) at vnode_if.h:80 #20 0xffffffff80cb50a2 in VOP_LOOKUP_APV (vop=, a=) at vnode_if.c:125 #21 0xffffffff80919658 in lookup (ndp=0xffffff848e6c05d8) at vnode_if.h:54 #22 0xffffffff807dffe5 in nfsvno_namei (nd=0xffffff848e6c08c8, ndp=0xffffff848e6c05d8, dp=, islocked=, exp=, p=, retdirp=) at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:413 #23 0xffffffff807d8ffa in nfsrvd_lookup (nd=0xffffff848e6c08c8, isdgram=, dp=0xfffffe00a1b441d8, vpp=0xffffff848e6c0810, fhp=0xffffff848e6c07f4, p=0xfffffe00a1198000, exp=0xffffff848e6c07a0) at /usr/src/sys/fs/nfsserver/nfs_nfsdserv.c:517 #24 0xffffffff807cb845 in nfsrvd_dorpc (nd=0xffffff848e6c08c8, isdgram=0, p=0xfffffe00a1198000) at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:823 #25 0xffffffff807d7af2 in nfssvc_program (rqst=0xfffffe00a17bb000, xprt=0xfffffe00a124b200) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:347 #26 0xffffffff80a70659 in svc_run_internal (pool=0xfffffe00067db600, ismaster=0) at /usr/src/sys/rpc/svc.c:895 #27 0xffffffff80a7155b in svc_thread_start (arg=0xffffff848e6bfc50) ---Type to continue, or q to quit--- at /usr/src/sys/rpc/svc.c:1200 #28 0xffffffff80858944 in fork_exit ( callout=0xffffffff80a71550 , arg=0xfffffe00067db600, frame=0xffffff848e6c0c00) at /usr/src/sys/kern/kern_fork.c:991 #29 0xffffffff80bfa86e in fork_trampoline () at exception.S:602 #30 0x0000000000000080 in ?? () #31 0x00007fffffffd820 in ?? () #32 0x0000000000000001 in ?? () #33 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) p vpp Cannot access memory at address 0x0 (kgdb) p dvp Cannot access memory at address 0x0 > >> Full witness is at http://people.freebsd.org/~pluknet/witness-zfs-20130204.txt >> >> shared lock of (lockmgr) zfs @ >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1452 >> while exclusively locked from >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1747 >> panic: share->excl >> cpuid = 2 >> KDB: enter: panic >> [ thread pid 812 tid 100884 ] >> Stopped at kdb_enter+0x3e: movq $0,kdb_why >> >> The 1st line is at zfs_lookup(): >> if (error == 0 && (nm[0] != '.' || nm[1] != '\0')) { >> int ltype = 0; >> >> if (cnp->cn_flags & ISDOTDOT) { >> ltype = VOP_ISLOCKED(dvp); >> VOP_UNLOCK(dvp, 0); >> } >> ZFS_EXIT(zfsvfs); >> error = zfs_vnode_lock(*vpp, cnp->cn_lkflags); >> if (cnp->cn_flags & ISDOTDOT) >> ==> vn_lock(dvp, ltype | LK_RETRY); >> if (error != 0) { >> VN_RELE(*vpp); >> *vpp = NULL; >> return (error); >> } >> } else { >> ZFS_EXIT(zfsvfs); >> } >> >> The 2nd line is at zfs_vnode_lock(): >> int >> zfs_vnode_lock(vnode_t *vp, int flags) >> { >> int error; >> >> ASSERT(vp != NULL); >> >> error = vn_lock(vp, flags); >> return (error); >> } >> >> db> show locks >> exclusive lockmgr zfs (zfs) r = 0 (0xfffffe00a1b44240) locked @ >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1747 >> db> show alllocks >> Process 812 (nfsd) thread 0xfffffe00a1198000 (100884) >> exclusive lockmgr zfs (zfs) r = 0 (0xfffffe00a1b44240) locked @ >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1747 >> Process 750 (syslogd) thread 0xfffffe0015a4c480 (100706) >> exclusive lockmgr ufs (ufs) r = 0 (0xfffffe00a1962d50) locked @ >> /usr/src/sys/kern/vfs_syscalls.c:3433 >> Process 12 (intr) thread 0xfffffe0006813000 (100033) >> exclusive sleep mutex AAC I/O lock (AAC I/O lock) r = 0 >> (0xffffff8001bfb210) locked @ /usr/src/sys/dev/aac/aac.c:827 >> >> db> show lock 0xfffffe00a1b44240 >> class: lockmgr >> name: zfs >> state: XLOCK: 0xfffffe00a1198000 (tid 100884, pid 812, "nfsd") >> waiters: none >> spinners: none >> >> As KASSERT is different: >> >> db> bt >> Tracing pid 812 tid 100884 td 0xfffffe00a1198000 >> kdb_enter() at kdb_enter+0x3e/frame 0xffffff848e6bfd60 >> vpanic() at vpanic+0x147/frame 0xffffff848e6bfda0 >> kassert_panic() at kassert_panic+0x136/frame 0xffffff848e6bfe10 >> witness_checkorder() at witness_checkorder+0x289/frame 0xffffff848e6bfe90 >> __lockmgr_args() at __lockmgr_args+0x43e/frame 0xffffff848e6bffc0 >> vop_stdlock() at vop_stdlock+0x3c/frame 0xffffff848e6bffe0 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0xd0/frame 0xffffff848e6c0000 >> _vn_lock() at _vn_lock+0xab/frame 0xffffff848e6c0070 >> zfs_lookup() at zfs_lookup+0x392/frame 0xffffff848e6c0100 >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xffffff848e6c0240 >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xc2/frame 0xffffff848e6c0260 >> vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xffffff848e6c02b0 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc2/frame 0xffffff848e6c02d0 >> lookup() at lookup+0x548/frame 0xffffff848e6c0350 >> nfsvno_namei() at nfsvno_namei+0x1a5/frame 0xffffff848e6c0400 >> nfsrvd_lookup() at nfsrvd_lookup+0x13a/frame 0xffffff848e6c06b0 >> nfsrvd_dorpc() at nfsrvd_dorpc+0xca5/frame 0xffffff848e6c08a0 >> nfssvc_program() at nfssvc_program+0x482/frame 0xffffff848e6c0a00 >> svc_run_internal() at svc_run_internal+0x1e9/frame 0xffffff848e6c0ba0 >> svc_thread_start() at svc_thread_start+0xb/frame 0xffffff848e6c0bb0 >> fork_exit() at fork_exit+0x84/frame 0xffffff848e6c0bf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xffffff848e6c0bf0 >> --- trap 0xc, rip = 0x800883b7a, rsp = 0x7fffffffd6c8, rbp = 0x7fffffffd970 --- >> > > > -- > Andriy Gapon -- wbr, pluknet