From owner-freebsd-fs@freebsd.org Tue Aug 2 06:05:24 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0CF9FBACE14 for ; Tue, 2 Aug 2016 06:05:24 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vk0-x236.google.com (mail-vk0-x236.google.com [IPv6:2607:f8b0:400c:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF869189A for ; Tue, 2 Aug 2016 06:05:23 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-vk0-x236.google.com with SMTP id n129so114720457vke.3 for ; Mon, 01 Aug 2016 23:05:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=WWclWNCOjerJEXCBUuh9TM77KFDJbtfRn5Leji2MdLg=; b=08flCjU2qvc3vCYzK3b8wUtiatgWp9SRlwnFUGebvMHYYc3pMv8AfHkH5OIDdfxJQ3 TGfDAzszmT8cm3vrEHrMCZ28bUNg4/9qugJskUb6tVUwPMsfbMP1LnFAFVUMC+HY6ma6 IL+KRYw2s/XsR4uZ89IP0bGVXWLbcmjOvV6tqM8s2OfgikvTqCrl2l1Bv1aFj9ILCfh3 BN5XhOkUVFgklQvQUPuDxDLXYbHzxegjCNqCFIyURL2K32ImLQ5HWtFmD9nKEMaQW9cR L0qyWljlVn+OjHcqGn9Ls+m1hpq8FU6hOWBycHigMkqUae/hom37kx7F26qvqYWWzufy eptQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=WWclWNCOjerJEXCBUuh9TM77KFDJbtfRn5Leji2MdLg=; b=GMRq5wQ2YlDRNfKdUNmh+wx1LuEKxYUscVlmgLBlVN3+e5nNw0ISx1wBBi66h00AyQ J3nCGVkFdrltYROxp6zhRWW4P/6ZDdL+eMve0kj/S6R0BC2oz+91SaL/zBaqnfloiY5d XP1DTKf2BlfBP5Br3BKQ8Zrb438WM8rlNDm+D0P79MObtxt7ajiibBVx6cLSIFyPcmUK 4bANAtpMrsWOgPEQx5j4ZoODvjjITV4sWjgEu47RgAhbL35si8vFxZQXTgiZH3lRXvKT LbO1MthKoEbr038cOiHMdxicSRFTs7eCj8tON4hQzukF6TbbJXWMvEMdWsnBrEcG80my hVhA== X-Gm-Message-State: AEkoousmZp3J99cUrsVW+3GsJ95gQGixnzGt4FrUL36CqSBf88fUWOPKlFJsN9vKYa1uS7eNhQYPJvOpf7v3UA== X-Received: by 10.31.196.199 with SMTP id u190mr29121462vkf.116.1470117922568; Mon, 01 Aug 2016 23:05:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.6.5 with HTTP; Mon, 1 Aug 2016 23:05:22 -0700 (PDT) In-Reply-To: References: From: Zaphod Beeblebrox Date: Tue, 2 Aug 2016 02:05:22 -0400 Message-ID: Subject: Re: Crashes (with dumps) in zap_leaf.c To: freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 06:05:24 -0000 Replying to my own post and adding some detail. I didn't realize at first that was an l as-in the lowercase L. But looking at the macro, it is. (kgdb) p l $1 = (zap_leaf_t *) 0xfffff8008b3a3b00 (kgdb) p *l $2 = {l_dbu = {dbu_tqent = {tqent_task = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0, ta_context = 0x0}, tqent_func = 0, tqent_arg = 0x0}, dbu_evict_func = 0xffffffff81a482b0 }, l_rwlock = { lock_object = {lo_name = 0xffffffff81af06d9 "l->l_rwlock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, sx_lock = 18446735297203916800}, l_blkid = 1311, l_bs = 14, l_dbuf = 0xfffff806255e11b0} Now the macro starts by dereferencing l->l_phys->l_hash ... I can get it to spit out l->l_bs, but the debugger won't spit out l->l_phys. At this point ... I'm wondering if I'm confused and missing something. On Tue, Aug 2, 2016 at 1:37 AM, Zaphod Beeblebrox wrote: > I'm getting multiple crashes that look rather like: > > #0 0xffffffff8098e390 at kdb_backtrace+0x60 > #1 0xffffffff80951066 at vpanic+0x126 > #2 0xffffffff80950f33 at panic+0x43 > #3 0xffffffff80bcfa4c at vm_fault_hold+0x1b2c > #4 0xffffffff80bcded7 at vm_fault+0x77 > #5 0xffffffff80d5612c at trap_pfault+0x19c > #6 0xffffffff80d558fa at trap+0x47a > #7 0xffffffff80d3b8d2 at calltrap+0x8 > #8 0xffffffff81a49a5a at zap_entry_create+0x27a > #9 0xffffffff81a45eee at fzap_add_cd+0xde > #10 0xffffffff81a4c051 at zap_add+0x101 > #11 0xffffffff81a6bfb5 at zfs_link_create+0x415 > #12 0xffffffff81a87a34 at zfs_freebsd_rename+0xac4 > #13 0xffffffff80e81e1b at VOP_RENAME_APV+0xab > #14 0xffffffff80a03476 at kern_renameat+0x4a6 > #15 0xffffffff80d5694f at amd64_syscall+0x40f > #16 0xffffffff80d3bbbb at Xfast_syscall+0xfb > > Analyzing one of those crashes: > > #0 doadump (textdump=) at pcpu.h:219 > 219 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump (textdump=) at pcpu.h:219 > #1 0xffffffff80950cc2 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:486 > #2 0xffffffff809510a5 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:889 > #3 0xffffffff80950f33 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:818 > #4 0xffffffff80bcfa4c in vm_fault_hold (map=, > vaddr=, fault_type=, > fault_flags=, m_hold=) > at /usr/src/sys/vm/vm_fault.c:329 > #5 0xffffffff80bcded7 in vm_fault (map=0xfffff80002000000, > vaddr=, fault_type=1 '\001', fault_flags=0) > at /usr/src/sys/vm/vm_fault.c:273 > #6 0xffffffff80d5612c in trap_pfault (frame=0xfffffe0c56854320, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:757 > #7 0xffffffff80d558fa in trap (frame=0xfffffe0c56854320) > at /usr/src/sys/amd64/amd64/trap.c:447 > #8 0xffffffff80d3b8d2 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:236 > #9 0xffffffff81a494c5 in zap_leaf_array_create (l=0xfffff8008b3a3b00, > buf=, integer_size=1, > num_integers=) > ---Type to continue, or q to quit--- > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:198 > #10 0xffffffff81a49a5a in zap_entry_create (l=0xfffff8008b3a3b00, > zn=0xfffff8008bf19e00, cd=Cannot access memory at address 0x0 > ) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:644 > #11 0xffffffff81a45eee in fzap_add_cd (zn=0xfffff8008bf19e00, > integer_size=, num_integers=1, > val=0xfffffe0c568546d0, cd=4294967295, tx=0xfffff80787758c00) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:814 > #12 0xffffffff81a4c051 in zap_add (os=, > zapobj=, key=, > integer_size=, num_integers=, > val=, tx=) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1014 > #13 0xffffffff81a6bfb5 in zfs_link_create (dl=0xfffff800966cce00, > zp=, tx=0xfffff80787758c00, > flag=) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:767 > #14 0xffffffff81a87a34 in zfs_freebsd_rename (ap=) > ---Type to continue, or q to quit--- > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4067 > #15 0xffffffff80e81e1b in VOP_RENAME_APV (vop=, > a=) at vnode_if.c:1546 > #16 0xffffffff80a03476 in kern_renameat (td=, > oldfd=, old=, > newfd=, new=, > pathseg=) at vnode_if.h:636 > #17 0xffffffff80d5694f in amd64_syscall (td=0xfffff8048f7fd000, traced=0) > at subr_syscall.c:141 > #18 0xffffffff80d3bbbb in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:396 > #19 0x000000080381fbaa in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) frame 9 > #9 0xffffffff81a494c5 in zap_leaf_array_create (l=0xfffff8008b3a3b00, > buf=, integer_size=1, > num_integers=) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:198 > 198 ZAP_LEAF_CHUNK(l, chunk).l_free.lf_next; > (kgdb) p chunk > No symbol "chunk" in current context. > > ... now I don't debug kernels too often, but I don't know why I can't see > chunk. It's declared as an int in this function. > > Does this give anyone insight? > >