From nobody Sun Sep 4 12:34:54 2022 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MLB1V377Zz4bZ9n for ; Sun, 4 Sep 2022 12:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MLB1V12jPz3Ts1 for ; Sun, 4 Sep 2022 12:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MLB1V00nbzV8M for ; Sun, 4 Sep 2022 12:34:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 284CYrN7032906 for ; Sun, 4 Sep 2022 12:34:53 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 284CYrlZ032904 for fs@FreeBSD.org; Sun, 4 Sep 2022 12:34:53 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 266014] panic: on long running find (zfs issue) Date: Sun, 04 Sep 2022 12:34:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dpy@pobox.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662294894; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j6Fl1++cFrRQcfnR4V1wkCYYfKdwWzmF0w4fo0YXIHY=; b=HAZGZj3pLJ22adn0iBfvxJl4KaDr7Ug7GRe2oWFBOqhQDAFtKCucwN7QEw8D/YvRFLqPoZ Sao13tW/HRbFRStnc34A7FwTcgw877SylBlHho5zowQgDmS5p07zZxAvV2+ABduNQGZwAZ 3OO+f+YsKal9XVRWBz55+rWXKBWdJ5mAR+r2WGnSexpXIO9t9QpwVx7DGdp5fB80VuFlG7 yUXAqJnq4T6JJqd1RJtI5TKhqDEvTryTpHTPFisiJZu5NMO4006ZXEBnA+1mlU+bDYtUvq jUskKgKs3o4FB3Cgi/QG9fWumenhcD6ETCN/qIx4O8D2axMxmJ+HYr+8/9vbEQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1662294894; a=rsa-sha256; cv=none; b=IXdj1U2l6qZPie2oMpQztHt4juxLmAhLw3DZbQQoUcicO6ZKA3yh5WK/TZxg2YNiUnVbz/ 4Hik0GCLjlNemxbxEP+CWGiM40TVvajhJNZZTunch0VJt/0T8V0IZhKX9XTvh4eHmhofW8 jWWxWfZWkg71bRlTFE8QeCfjeQSzsTn2kQx3i5Jn/4/Ug9a+3UbATkQY9na9xIw2X/mA5Q Wt028IhTyxmfv5hC6A5NTy6Psv53JMrbhs2TH5M9Bw4lV/uMSbNoatSBteLlAwYtY/J1Er 8p5ZoY4GZHB0ghLjYf1WAYUXMYBSFd6TngG+6SZhRJXGmotlnLZv4W7/nHKHKw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D266014 --- Comment #4 from Duncan --- I have compiled my system from source and caused a panic (run periodic daily (daily_clean_disks_enabled will cause it) ). I turned off all the VMs and j= ails before causing the crash (remove extra complexity and reduce the size of the core.txt file). I also added a larger dump disk and now have a complete via= ble dump. I have attached the complete core.txt.0 file and the following is the starting portion: triple0.internal dumped core - see /var/crash/vmcore.0 Sun Sep 4 20:09:55 AEST 2022 FreeBSD triple0.internal 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 releng/13.1-n250158-752f813d6cc GENERIC amd64 panic: VERIFY3(0 =3D=3D sa_handle_get_from_db(zfsvfs->z_os, db, zp, SA_HDL_= SHARED, &zp->z_sa_hdl)) failed (0 =3D=3D 5) GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD] Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.1". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: panic: VERIFY3(0 =3D=3D sa_handle_get_from_db(zfsvfs->z_os, db, zp, SA_HDL_= SHARED, &zp->z_sa_hdl)) failed (0 =3D=3D 5) cpuid =3D 8 time =3D 1662284816 KDB: stack backtrace: #0 0xffffffff80c694a5 at kdb_backtrace+0x65 #1 0xffffffff80c1bb5f at vpanic+0x17f #2 0xffffffff8500bf4a at spl_panic+0x3a #3 0xffffffff85032c12 at zfs_znode_alloc+0x522 #4 0xffffffff85033c35 at zfs_zget+0x3b5 #5 0xffffffff8501e84b at zfs_dirent_lookup+0x16b #6 0xffffffff8501e91a at zfs_dirlook+0x7a #7 0xffffffff85030a10 at zfs_lookup+0x3d0 #8 0xffffffff8502c11b at zfs_freebsd_cachedlookup+0x6b #9 0xffffffff80cdc4ed at vfs_cache_lookup+0xad #10 0xffffffff80ce0cf0 at VOP_LOOKUP+0x30 #11 0xffffffff80ce0723 at cache_fplookup_noentry+0x1a3 #12 0xffffffff80cddfa6 at cache_fplookup+0x366 #13 0xffffffff80ce8cfa at namei+0x12a #14 0xffffffff80d06993 at kern_statat+0xf3 #15 0xffffffff80d0708f at sys_fstatat+0x2f #16 0xffffffff810b06ec at amd64_syscall+0x10c #17 0xffffffff81087e8b at fast_syscall_common+0xf8 Uptime: 56m13s (ada0:ahcich1:0:0:0): spin-down (ada1:ahcich2:0:0:0): spin-down (ada2:ahcich3:0:0:0): spin-down (ada3:ahcich4:0:0:0): spin-down Dumping 31966 out of 130858 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c1b75c in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:487 #3 0xffffffff80c1bbce in vpanic ( fmt=3D0xffffffff8527c012 "VERIFY3(0 =3D=3D sa_handle_get_from_db(zfsvfs= ->z_os, db, zp, SA_HDL_SHARED, &zp->z_sa_hdl)) failed (0 =3D=3D %lld)\n", ap=3D) at /usr/src/sys/kern/kern_shutdown.c:920 #4 0xffffffff8500bf4a in spl_panic (file=3D, func=3D, line=3D, fmt=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:107 #5 0xffffffff85032c12 in zfs_znode_sa_init (zfsvfs=3D0xfffff8027a9c5000, zp=3D0xfffff807415f9ce8, db=3D0xfffff80f7c49acb8, obj_type=3DDMU_OT_SA, sa_hdl=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:364 #6 zfs_znode_alloc (zfsvfs=3Dzfsvfs@entry=3D0xfffff8027a9c5000, db=3D0xfffff80f7c49acb8, blksz=3D3584, obj_type=3DDMU_OT_SA, hdl=3Dhdl@= entry=3D0x0) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:466 #7 0xffffffff85033c35 in zfs_zget (zfsvfs=3D, zfsvfs@entry=3D0xfffff8027a9c5000, obj_num=3D, zpp=3Dzpp@entry=3D0xfffffe03109916f0) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:1042 #8 0xffffffff8501e84b in zfs_dirent_lookup ( dzp=3Ddzp@entry=3D0xfffff80741474588, name=3D0xfffffe0310991860 "tb_pkm= eth.c", zpp=3Dzpp@entry=3D0xfffffe0310991740, flag=3Dflag@entry=3D2) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:191 #9 0xffffffff8501e91a in zfs_dirlook (dzp=3Ddzp@entry=3D0xfffff80741474588, name=3D, name@entry=3D0xfffffe0310991860 "tb_pkmeth.c", zpp=3Dzpp@entry=3D0xfffffe03109917e0) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:247 #10 0xffffffff85030a10 in zfs_lookup (dvp=3D, nm=3Dnm@entry=3D0xfffffe0310991860 "tb_pkmeth.c", vpp=3D, cnp=3Dcnp@entry=3D0xfffffe0310991c58, nameiop=3D0, cr=3D= , flags=3D0, cached=3D1) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:929 #11 0xffffffff8502c11b in zfs_freebsd_lookup (ap=3D0xfffffe0310991990, ap@entry=3D, cached=3D1) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:45= 93 #12 zfs_freebsd_cachedlookup (ap=3D0xfffffe0310991990, ap@entry=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:46= 01 #13 0xffffffff80cdc4ed in VOP_CACHEDLOOKUP (dvp=3D0xfffff80730da45b8, vpp=3D0xfffffe0310991a10, cnp=3D0xfffffe0310991c58) at ./vnode_if.h:99 #14 vfs_cache_lookup (ap=3D, ap@entry=3D) at /usr/src/sys/kern/vfs_cache.c:3069 #15 0xffffffff80ce0cf0 in VOP_LOOKUP (dvp=3Ddvp@entry=3D0xfffff80730da45b8, vpp=3D, vpp@entry=3D0xfffffe0310991a10, cnp=3D, cnp@entry=3D0xfffffe0310991c58) at ./vnode_if.h:65 #16 0xffffffff80ce0723 in cache_fplookup_noentry ( fpl=3Dfpl@entry=3D0xfffffe0310991a88) at /usr/src/sys/kern/vfs_cache.c:= 4928 #17 0xffffffff80cddfa6 in cache_fplookup_next (fpl=3D0xfffffe0310991a88) at /usr/src/sys/kern/vfs_cache.c:5284 #18 cache_fplookup_impl (dvp=3D, fpl=3D0xfffffe0310991a88) at /usr/src/sys/kern/vfs_cache.c:5932 #19 cache_fplookup (ndp=3Dndp@entry=3D0xfffffe0310991bd8, status=3Dstatus@entry=3D0xfffffe0310991b84, pwdp=3Dpwdp@entry=3D0xfffffe0310991b88) at /usr/src/sys/kern/vfs_cache.= c:6104 #20 0xffffffff80ce8cfa in namei (ndp=3Dndp@entry=3D0xfffffe0310991bd8) at /usr/src/sys/kern/vfs_lookup.c:570 #21 0xffffffff80d06993 in kern_statat (td=3D0xfffffe030c6b5000, flag=3D, fd=3D-100, path=3D, pathseg=3D, pathseg@entry=3DUIO_USERSPACE, sbp=3Dsbp@entry=3D0xfffffe0310991d18, ho= ok=3D0x0) at /usr/src/sys/kern/vfs_syscalls.c:2441 #22 0xffffffff80d0708f in sys_fstatat (td=3D, uap=3D0xfffffe030c6b53e8) at /usr/src/sys/kern/vfs_syscalls.c:2418 #23 0xffffffff810b06ec in syscallenter (td=3D0xfffffe030c6b5000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #24 amd64_syscall (td=3D0xfffffe030c6b5000, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1185 #25 #26 0x00000008011ad39a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffd588 (kgdb) ------------------------------------------------------------------------ --=20 You are receiving this mail because: You are the assignee for the bug.=