From owner-freebsd-current@freebsd.org Fri Aug 28 05:37:53 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 425F73CE909 for ; Fri, 28 Aug 2020 05:37:53 +0000 (UTC) (envelope-from ypankov@xsmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4Bd7fT0B70z4309 for ; Fri, 28 Aug 2020 05:37:53 +0000 (UTC) (envelope-from ypankov@xsmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 046A43CE68F; Fri, 28 Aug 2020 05:37:53 +0000 (UTC) Delivered-To: current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 042F13CE8AB for ; Fri, 28 Aug 2020 05:37:53 +0000 (UTC) (envelope-from ypankov@xsmail.com) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bd7fS1RXCz42nH; Fri, 28 Aug 2020 05:37:51 +0000 (UTC) (envelope-from ypankov@xsmail.com) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 5EA795C01DC; Fri, 28 Aug 2020 01:37:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 28 Aug 2020 01:37:51 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedruddviedgleekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepuffvfhfhkffffgggjggtgfesthejredttdefjeenucfhrhhomhepjghurhhi ucfrrghnkhhovhcuoeihphgrnhhkohhvseigshhmrghilhdrtghomheqnecuggftrfgrth htvghrnhephfdvvefhkeekkeehfeefveejveejudfhjefgjeefiedugeehueeuteehieeh uddvnecukfhppeeluddrvdegtddruddvgedrudeltdenucfuphgrmhfkphfpvghtfihorh hkpeeluddrvdegtddruddvgedrudeltdenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpeihphgrnhhkohhvseigshhmrghilhdrtghomh X-ME-Proxy: Received: from mercury.lan (unknown [91.240.124.190]) by mail.messagingengine.com (Postfix) with ESMTPA id AC78D328005D; Fri, 28 Aug 2020 01:37:50 -0400 (EDT) Subject: Re: panic in range_tree_seg64_compare() To: Matthew Macy Cc: current References: <2b89d19f-3fa9-8cf2-4ce2-91f1702172e4@xsmail.com> From: Yuri Pankov Message-ID: Date: Fri, 28 Aug 2020 08:37:48 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Bd7fS1RXCz42nH X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.06 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[xsmail.com:s=fm3,messagingengine.com:s=fm3]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.28:from]; FREEMAIL_FROM(0.00)[xsmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; NEURAL_HAM_LONG(-0.97)[-0.974]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_MEDIUM(-1.03)[-1.032]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[xsmail.com:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[xsmail.com,none]; NEURAL_HAM_SHORT(-0.95)[-0.952]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[xsmail.com]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[current]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2020 05:37:53 -0000 Matthew Macy wrote: > On Thu, Aug 27, 2020 at 6:34 PM Yuri Pankov wrote: >> >> Yet another issue I'm seeing after last update (currently running >> r364870), hit it 2 times today: >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 19; apic id = 0d >> fault virtual address = 0xfffff819e2ecdc40 >> fault code = supervisor read data, page not present >> instruction pointer = 0x20:0xffffffff8277fa64 >> stack pointer = 0x28:0xfffffe01f9ff2d90 >> frame pointer = 0x28:0xfffffe01f9ff2d90 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 48792 (blk-3:0-0) >> trap number = 12 >> panic: page fault >> cpuid = 19 >> time = 1598577675 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe01f9ff2a40 >> vpanic() at vpanic+0x182/frame 0xfffffe01f9ff2a90 >> panic() at panic+0x43/frame 0xfffffe01f9ff2af0 >> trap_fatal() at trap_fatal+0x387/frame 0xfffffe01f9ff2b50 >> trap_pfault() at trap_pfault+0x97/frame 0xfffffe01f9ff2bb0 >> trap() at trap+0x2ab/frame 0xfffffe01f9ff2cc0 >> calltrap() at calltrap+0x8/frame 0xfffffe01f9ff2cc0 >> --- trap 0xc, rip = 0xffffffff8277fa64, rsp = 0xfffffe01f9ff2d90, rbp = >> 0xfffffe01f9ff2d90 --- >> range_tree_seg64_compare() at range_tree_seg64_compare+0x4/frame >> 0xfffffe01f9ff2d90 >> zfs_btree_find() at zfs_btree_find+0x1bd/frame 0xfffffe01f9ff2df0 >> range_tree_find_impl() at range_tree_find_impl+0x6e/frame 0xfffffe01f9ff2e30 >> range_tree_find() at range_tree_find+0x1c/frame 0xfffffe01f9ff2e70 >> range_tree_contains() at range_tree_contains+0x9/frame 0xfffffe01f9ff2e80 >> dnode_block_freed() at dnode_block_freed+0x11d/frame 0xfffffe01f9ff2eb0 >> dbuf_read() at dbuf_read+0x70c/frame 0xfffffe01f9ff2fc0 >> dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x164/frame >> 0xfffffe01f9ff3030 >> dmu_read_impl() at dmu_read_impl+0xce/frame 0xfffffe01f9ff30c0 >> dmu_read() at dmu_read+0x45/frame 0xfffffe01f9ff3100 >> zvol_geom_bio_strategy() at zvol_geom_bio_strategy+0x2aa/frame >> 0xfffffe01f9ff3180 >> g_io_request() at g_io_request+0x2df/frame 0xfffffe01f9ff31b0 >> g_dev_strategy() at g_dev_strategy+0x155/frame 0xfffffe01f9ff31e0 >> physio() at physio+0x4f8/frame 0xfffffe01f9ff3270 >> devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe01f9ff32d0 >> dofileread() at dofileread+0x81/frame 0xfffffe01f9ff3320 >> kern_preadv() at kern_preadv+0x62/frame 0xfffffe01f9ff3360 >> sys_preadv() at sys_preadv+0x39/frame 0xfffffe01f9ff3390 >> amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe01f9ff34b0 >> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01f9ff34b0 >> --- syscall (289, FreeBSD ELF64, sys_preadv), rip = 0x8006fd89a, rsp = >> 0x7fffdfdfcf18, rbp = 0x7fffdfdfcfc0 --- >> Uptime: 4h13m43s > > >> >> Guessing on zvol_geom_bio_strategy(), it's volmode=dev zvol I'm using >> for bhyve VM. Anything known? > > Not really. A reproduction scenario would be very helpful. This was > seen once by someone at iX - I committed some additional asserts to > the truenas tree, but haven't heard further. > > +++ b/module/zfs/dbuf.c > @@ -3192,7 +3192,7 @@ > dbuf_dirty_leaf_with_existing_frontend(dbuf_dirty_state_t *dds) > * scheduled its write with its buffer, we must > * disassociate by replacing the frontend. > */ > - ASSERT(db->db_state & (DB_READ|DB_PARTIAL)); > + ASSERT3U(db->db_state, &, (DB_READ|DB_PARTIAL)); > ASSERT3U(db->db_dirtycnt, ==, 1); > dbuf_dirty_set_data(dds); > } else { > @@ -3238,18 +3238,24 @@ dbuf_dirty_record_create_leaf(dbuf_dirty_state_t *dds) > > dr = dbuf_dirty_record_create(dds); > > + /* > + * XXX - convert to ASSERT after dn_free_ranges fix > + */ > + VERIFY(db->db_level == 0); > + VERIFY(db->db_blkid != DMU_BONUS_BLKID); Can't find context for both chunks, there are simply no such functions in sys/contrib/openzfs/module/zfs/dbuf.c, and yes, note that I'm running the in-tree version.