From nobody Wed Dec 1 18:42:38 2021 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 26B5718AF3EE for ; Wed, 1 Dec 2021 18:42:50 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J47Hs0ZpVz3QhD for ; Wed, 1 Dec 2021 18:42:48 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 1B1IgdZN009717 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Dec 2021 10:42:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 1B1IgdY7003809; Wed, 1 Dec 2021 10:42:39 -0800 (PST) Message-Id: <202112011842.1B1IgdY7003809@nfbcal.org> From: Brian Buhrow Date: Wed, 1 Dec 2021 10:42:38 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Re: Missing MAC addresses for domu VM's Cc: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 01 Dec 2021 10:42:40 -0800 (PST) X-Rspamd-Queue-Id: 4J47Hs0ZpVz3QhD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.30 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org hello. In looking into this further, I discovered that the bridges on NetBSD have the same issue, that is, you can't have the same MAC address on the dom0 side as the domu side. However, I'd like to suggest that the NetBSD solution to the problem is more elegant in that it is possible to track domains solely based on netstat output and their MAC addresses while, at the same time, being able to derive both the front end and back end MAC addresses for a given domain by reading the MAC address from either side. I'll quote the comment and the relevant line of code from sys/arch/xen/xen/xennetback_xenbus.c in the NetBSD source tree. Is there any reason this solution wouldn't work in the FReeBSD case as well? If it would, it would be nice to see MAC addresses return to netstat -i output for xen virtual interfaces. -thanks -Brian /* we can't use the same MAC addr as our guest */ xneti->xni_enaddr[3]++; From nobody Sat Dec 4 09:33:37 2021 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 196DB18BFEE1 for ; Sat, 4 Dec 2021 09:33:49 +0000 (UTC) (envelope-from dsx@bsdsx.fr) Received: from mx2.bsdsx.fr (mx2.bsdsx.fr [31.216.24.205]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (Client CN "mx2.bsdsx.fr", Issuer "bsdsx.fr TLS CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J5kz01WWpz4SxG for ; Sat, 4 Dec 2021 09:33:48 +0000 (UTC) (envelope-from dsx@bsdsx.fr) Received: from mx2.bsdsx.fr (dkimproxy.bsdsx.fr [2a02:27d0:100:f205:a642::20]) by mx2.bsdsx.fr (OpenSMTPD) with ESMTP id ae303d0b for ; Sat, 4 Dec 2021 10:33:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=bsdsx.fr; h=date:from:to :subject:message-id:mime-version:content-type; s=mx2; bh=pnE8Xtg VzsQSerSD36ciAc29k7vtC5C3BXwDz3kNkhY=; b=WIiXeZpVODzodt8zp7np15R mS1CjfIQsIaCqjkvvT0/LwOxrm8FGBvZmaGwXu3lO8GcdDhzWjjSnkrlvtzcZb+t XSXWsYnbIZIaCJSiGCPwB46CbTEvvSbjox5GbVjwa9W69mJnjotm1l+RYbgw3OG1 WEokC4N1Q2XhcWYyDWMPlqXWfqR5LJqbKN4epyJG72YpJwa9euVRwCB+sv7PeIu1 9xtUg6Td7Dm2ov1pStByesOmMVJ7T5g7JZTuxxPetJL8W5tGlEHUrgVC54BfHqu/ QHaiCVNhKuzR1xYQFf0VIprk4+c6BhPfA1m+L23wBE8HC/FLSI7bk8KtM8utFJA= = Received: from blade.bsdsx.fr ( [172.16.0.9]) by mx2.bsdsx.fr (OpenSMTPD) with ESMTPS id 4ab05187 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO) for ; Sat, 4 Dec 2021 10:33:37 +0100 (CET) Received: (from dsx@localhost) by blade.bsdsx.fr (8.15.2/8.15.2/Submit) id 1B49XbZd001146 for freebsd-xen@freebsd.org; Sat, 4 Dec 2021 10:33:37 +0100 (CET) (envelope-from dsx@bsdsx.fr) Date: Sat, 4 Dec 2021 10:33:37 +0100 From: Freddy DISSAUX To: freebsd-xen@freebsd.org Subject: Can't shutdown FreeBSD domU Message-ID: <20211204093337.GA69642@blade.bsdsx.fr> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: 4J5kz01WWpz4SxG X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=fail ("headers rsa verify failed") header.d=bsdsx.fr header.s=mx2 header.b=WIiXeZpV; dmarc=pass (policy=reject) header.from=bsdsx.fr; spf=pass (mx1.freebsd.org: domain of dsx@bsdsx.fr designates 31.216.24.205 as permitted sender) smtp.mailfrom=dsx@bsdsx.fr X-Spamd-Result: default: False [-3.80 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_DKIM_REJECT(0.00)[bsdsx.fr:s=mx2]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; DKIM_TRACE(0.00)[bsdsx.fr:-]; DMARC_POLICY_ALLOW(0.00)[bsdsx.fr,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:49463, ipnet:31.216.16.0/20, country:FR] X-Spam: Yes X-ThisMailContainsUnwantedMimeParts: N Hello, My dom0 is a alpine linux 3.15 with xen 4.15: dsx@xen>cat /etc/alpine-release 3.15.0 dsx@xen>apk info xen xen-4.15.1-r2 description: Xen hypervisor xen-4.15.1-r2 webpage: https://www.xenproject.org/ xen-4.15.1-r2 installed size: 262 MiB All domU are in pvh mode. I can shutdown domU alpine linux, domU netbsd but not domU freebsd: dsx@xen>doas xl create al30.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait al30 ; sleep 2; doas xl list Parsing config from al30.cfg Name ID Mem VCPUsStateTime(s) Domain-0 0 1022 4 r----- 2581.2 al30 66 1023 1 -b---- 2.1 Shutting down domain 66 Waiting for 1 domains Domain 66 has been shut down, reason code 0 Name ID Mem VCPUsStateTime(s) Domain-0 0 1022 4 r----- 2581.7 dsx@xen>doas xl create nbsd20.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait nbsd20 ; sleep 2 ; doas xl list Parsing config from nbsd20.cfg WARNING: msr_relaxed will be removed in future versions. If it fixes an issue you are having please report to xen-devel@lists.xenproject.org. Name ID Mem VCPUsStateTime(s) Domain-0 0 1022 4 r----- 2583.2 nbsd20 67 2047 2 -b---- 2.2 Shutting down domain 67 Waiting for 1 domains Domain 67 has been shut down, reason code 0 Name ID Mem VCPUsStateTime(s) Domain-0 0 1022 4 r----- 2583.7 dsx@xen>doas xl create fbsd40.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait fbsd40 ; sleep 2 ; doas xl list Parsing config from fbsd40.cfg Name ID Mem VCPUsStateTime(s) Domain-0 0 1019 4 r----- 2584.7 fbsd40 68 2048 2 -b---- 4.5 Shutting down domain 68 Waiting for 1 domains ^C dsx@xen>doas xl list Name ID Mem VCPUsStateTime(s) Domain-0 0 1019 4 r----- 2585.0 fbsd40 68 2048 2 r----- 170.6 FreebSD domU config: name = "fbsd40" maxvcpus = 2 memory = 2048 vif = [ 'bridge=br40' ] type = "pvh" kernel = "/home/dsx/kernel" extra = 'vfs.root.mountfrom=ufs:ufs/rootfs,boot_verbose=1' disk = [ 'format=raw, vdev=xvda, access=w, target=/home/dsx/FreeBSD-13.0-RELEASE-amd64.raw' ] dmesg: http://download.bsdsx.fr/xen/dmesg.txt Same problem with FreeBSD dom0: dsx@ni3>uname -smr FreeBSD 13.0-RELEASE-p4 amd64 dsx@ni3>pkg info 'xen-*' xen-kernel-4.15.0_1 xen-tools-4.15.0_2 Any hint ? Regards, From nobody Mon Jan 10 15:57:25 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3EB5F19370AB for ; Mon, 10 Jan 2022 15:57:43 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXdks6ptZz4Rt8 for ; Mon, 10 Jan 2022 15:57:41 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1641830261; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=M463xhjnepaSN3CTPklTP9QcdwXEzcYQVwa91oayYUk=; b=MH7T7tsfxh7sy+pjIzPeT2k9WIeczSAf8AUDWY/O+HVRWSjIidFtvJVa J7SmEMxfW0Nca3O2K3EJqbLqdDhbm0gEb492wNdjuPAnW/Pglc6uH0YmF xia4G8fYZmpiHChM8Vq30h6YtiHPIINoklJJcuooRLYez6nc5g3iA8U30 w=; IronPort-SDR: ABhOK7FEKtCC+2ZjUzkOaorHq6I8/z2ItiAIlLB4XDxrsU6BiYBAq5GGgCoqddRoCQSkJRc25t Nfy/IeRgc8Az176HoLT7bBSCr5y6LMVeFjObD1azAQkUheRZIaZayP5ISO0XFaJGe4MGxjCwS6 Qe2a2QOIbwhyvn4UgE0oIQ0vAzon14fboVpYHRadUKwb0t6k40AkgO1QRyvvW3f4UMQICVmREs 5KzJMO5PDlbhfXLrwTvBostkgS3quMijhW8TxvqruMmYygAeVVdzi4v0TTjaX3oDEIV+O29Ds+ RDvLS4fKfO/TVDl4Fr/F5719 X-SBRS: 5.2 X-MesageID: 61660166 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:tlG+JaPzcw4zuL7vrR1wkMFynXyQoLVcMsEvi/4bfWQNrUpw0zcAz TEaWWvVPKyIMGPycop0Oozk9B8AucXWydZmSwto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h ynLQoCYdKjYdpJYz/uUGuCJQUNUjMlkfZKhTr6UUsxNbVU8En150Es7w7RRbrNA2rBVPSvc4 bsenOWHULOV82Yc3rU8sv/rRLtH5ZweiRtA1rAMTakjUGz2zhH5OKk3N6CpR0YUd6EPdgKMq 0Qv+5nilo/R109F5tpICd8XeGVSKlLZFVDmZna7x8FOK/WNz8A/+v9TCRYSVatYowWLxfF36 vsdiZPudBojF6Pqmtg6CQYNRkmSPYUekFPGCX22sMjVxEzaaXr8hf5pCSnaP6VBpLwxWzsXs 6VFdnZdNXhvhMrvqF6/YvNrick5atHiIasUu216zCGfBvEjKXzGa/uTu4EAhWlu7ixINffaT stCViJyUAX/bkVEOxRKMrIgus790xETdBUH8QnI9MLb+VP7whZ83bL2PJ/Fd9qiS99Yk0yDq iTB5WuRKhsHNMaY1hKD/Hinj+uJmjn0ML/+D5XhqKQs2gfKgDVOVltGDjNXvMVVlGaOWYh+L W46qhYh85Uv/mWlaNvBeEen9SvsUgEnZ/JcFOgz6Qeow6XS4hqECmVscgOteODKp+dtG2V0i wbhc8fBQGU27ebLES71GqK88GvqURX5O1PucsPtoeEtx9D46L8+gRvUJjqIOP7k14alcd0cL t3jkcTfu1nxpZNRv0lY1Qqe695JmnQvZlVsjukwdjj1hj6VnKb/O+SVBaHztJ6s1rqxQFibp 2QjkMOD9u0IBpzlvHXTHLxSQer0vKvVYWe0bbtT838JrG3FF5mLJ9E43d2DDB0xbpZslcHBP Sc/Rj+9FLcMZSD3PMebkqq6CtgwzLiIKDgWfqu8Uza6WbAoLFXv1Hg3PSa4hjmx+GBxz/1XE crFIK6EUCZLYYw6nWHeb7pMjtcWKtUWmDm7qWbTlUr3iNJzpRe9FN84Dbd5Rrtos/Pf/lSEq oY32gnj40w3bdASqxL/qOY7BVsLMWI6FdbxrclWffSEOQ1oBCcqDPq5/F/rU9cNc319mria8 3ejdFVfzVaj13TLJR/TMiJoaa/1XIY5pnU+ZHR+MVGt0nklQICu8KZAKMdnIeh5rLRunaxuU v0IW8ScGfATGD7JzCsQMMvmp4t4eRX12Q/XZ3i5YCIydoJLThDS/oO2ZRPm8SQDV3LltcY3r 7C6+BncRJ4PG1ZrAMrMMar9xFKtp3kN3ul1WhKQcNVUfUzt9qlsKjDw0aBrc51dd02by2LDh QiMABoeqe3cmKMP8YHE1fKesoOkM+piBU4GTWPV2qm7aHvB9W25zI4eDOvRJWLBVHn58bmJb PlOy62uK+UOmVtHvtYuE7tvyq5itdLjq6UDk1ZhFXTPKV+qFqlhMj+N2swW7v9BwbpQuA2XX EOT+4YFZeXVaZ29SFNBdhA4aumj1O0PnmiA5Ps4F0z2+Str8efVSk5VJRSN1HRQIbYd3FnJG gv9VBr6MzCCtyc= IronPort-HdrOrdr: A9a23:cB7nua5qyQg9zaob9gPXwSqBI+orL9Y04lQ7vn2ZFiY7TiXIra yTdaoguCMc6AxxZJkh8erwX5VoZUmsj6KdgLNhRotKOTOJhILGFvAB0WKP+UyEJ8S6zJ8h6U 4CSdkBNDSTNykCsS+S2mDReLxBsbq6GeKT9J/jJh9WPH5XgspbnmFE42igYylLrF4sP+tEKH PQ3LsOmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZVbxp/hZMZtU TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczIgBkF5MWUrHo6mt jFpBkte+x19nPqZ2mw5SDg3gHxuQxenEPK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL 9L93jxjesZMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO31GkeKp guMCjg3ocXTbvDBEqp/VWHgebcE0jbJy32DHTr4aeuonprdHMQ9Tpt+CVQpAZEyHsHceg12w 3zCNUcqFh/dL5mUUtDPpZzfSKWMB26ffueChPaHbzYfJt3SU4l7aSHpIkI2A== X-IronPort-AV: E=Sophos;i="5.88,277,1635220800"; d="scan'208";a="61660166" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C6FGWKIruY97f0kxhTMOs8ub5Kyu03Wsk06ZtimqN9HLJKOjZDlTwrepV5ixP0NiCjMJO9KKZLpQYiaYRtWAxjMuiZmhnyAXPEZUZGm7ZOQD/qgj66KTkmT/h9gWIw8I0Bk/tU+zZKbfbV7gM9WUyGAvocU1GTCgrO4A+06T2zWIWyuePPZ5eJvcWV0ebSSJQ6GcBfND9Wgg7eWfAVBvD0Z1rpiAWlJFQuqLv44Wwx3GhFYnD90OrURazOT1COm7OCmGFBrKLAQy85xzMdlfG9Cjhe/xxYiGOrp9Ygnh/NWcv7Kf6gZ7pGoezthYQg46G/3/qKjDDcSe9vUjiDm6/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=d5FKwxeblKsaDZAhfYXXluWez61LGD94XZ6q+9T4S94=; b=W7tWB5h5zflGOVUDUWTeyZl4A4SiUXqnYK+Z7MXGYPrkAQrUtT1/yO0VjSYEI1sBnQksU1UZvIju6Msu0HoYOQlcwsA16O69Xe8/SYE3El/qPC81bN31Lsp8QqY4jtrQpdPyNqVUF5v8/8VuSpOvVSwbNMPXrQ17LaZlad/kRcvu7Db8LhodbznRYcNLuuxgrpuvnayUKxnnzMAFTGsX1Xpn+UZFn3y2LqNY9/HG3iUUN/GdCwwk9Gtb0iJ9WDdZBB47lL50uK/89+pytKEOs7q1s3FEh2pYzGRYGeBVGpksHpRChSbK+Ld1cwRwma6NGcNXWQHs51WBfA7G+LnSQg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d5FKwxeblKsaDZAhfYXXluWez61LGD94XZ6q+9T4S94=; b=KK3T3YOCsp1lEmlfONkYzEFh/4fgCib8JMDR2zdx9vYDLh8TQgvpmyaf03t5c7GRu7hQuT8ldHR68GJPY+HxKVcuYWb6XV9gVBJ9qGX3ujgbEzYn/cyX10fKHivTTmxCcMXRRDXpBci0vPNMx8Z5w3SrJo6WKBsq9AUhQL71Rcs= Date: Mon, 10 Jan 2022 16:57:25 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Brian Buhrow CC: Subject: Re: Missing MAC addresses for domu VM's Message-ID: References: <202112011842.1B1IgdY7003809@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <202112011842.1B1IgdY7003809@nfbcal.org> X-ClientProxiedBy: MRXP264CA0002.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:15::14) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f6f912a9-05b8-4cc8-0041-08d9d451e839 X-MS-TrafficTypeDiagnostic: DM5PR03MB2713:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uiBN4Tr8dsIBzyZcj29A6Dza1pDtrtEjgD3ZAMGYeqpeCqsKhtSjAoZtANoWWRTSoHZ9YnG7jwi2hu957bBCkNZcjZhYHZkx42oe5TDD6+ZB3Ws3XygrONyeqS0zGFH50NiCDmiwufC7en1eGPhYQAbQgrwaeibV13cOZNdk14XPBwJ6/FpgW8TKCi7VkkhMZSp/v/f0yPAaZfNlRo5QNPnIRtCTI81guDoqANjAB0rVv4oP0No0Rh56l/gPfk1EL9LVEfkBkUGLiMKOhoxO9PXF90KfbvDHgUbR5egfco52C3M//PuptLjFwRhBWI/20qee2/DRUO5O2P/ervA9o/U1wr6GjVhrXljyZaSTlKf7j5AG0t1n2SfDqArWeXl/V473Ef2Zwj7+ns9yHxUVKiH0mfGVLg6AOJ2atREzZW5Koj8kxkwWXHRDXX6mNSowb9mpL5Zns6g7yKUnIZN6hgyscOOxuz8HKDxfe42BCKjrtzzw+/krD6xpXkUI35Ju2mI2YMDxrKzGLwLaj9DuSZoInIG4ZybJ7zM7pwC5TfxOd/r2fKRC7iKbGisQBD5dBl1APWTlAFm71VWO1J+cBBWF2cokTk0Y+3uqlB1nFEw/GxhyCdLW4VvWWRP9SVadGXNNaUzWmjOTVQo3raSx1Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(4326008)(5660300002)(6666004)(26005)(8676002)(82960400001)(38100700002)(186003)(6486002)(508600001)(33716001)(6916009)(2906002)(8936002)(66476007)(9686003)(66946007)(6512007)(66556008)(85182001)(86362001)(6506007)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QlBZbDYvLzJoV25sNmxFeFJYWG43Y3NKd2xQUjhwd1plaVRFN2FkTm5QbVg2?= =?utf-8?B?bmF2Y3lVaGFCcFM5a1dXQ2VEVVJUbmRXSGlSaUxxMW9GZzJjYjB0T2JUYTFC?= =?utf-8?B?WUkxUERPZndNSnNkN3Q5NmlVVjhpaUY2TEVQN0tlQjFFZW1LRkVmMjZtUlBa?= =?utf-8?B?YllCanp3UzZqZVlxSUErRVc2dUlWdjN1Snp4N2JIYVlmdjZWMnYwUmwyUVJt?= =?utf-8?B?ZGJ5cGVGSVJUa0J2RUlsUTNRbk5IdjRHUjVMM09EUWxTRC9XNVF4cUwvbW5Z?= =?utf-8?B?REsrdDBNNlVCby9hL0d2dXhGc0puWnRReUJYaXg2L3FuQTVjcStEVnNNZ29D?= =?utf-8?B?QktKTDhYVGdXNVdZaGtJOUZNNXBEWFpPem14T3JSK3Y3anBpUWFzajVkVEkr?= =?utf-8?B?Ni9NajVEdk5XWUQxRnM1d1gya1VkUExwM1JWUTZkL2FDMWhySWsyNzFwVzk5?= =?utf-8?B?ZE5FUEI5NTIrd2o2dFFCQlNtcnJhUFl6cHhickVRZ1VMSjh2aURERG0yQUdU?= =?utf-8?B?TW5CazFpaXhNaUhLVzlGUUZBTy9FVzNVSjJnNFBZbWlvREd0Y0JVbDE0NXdN?= =?utf-8?B?SXp3ajZsbFdXMUdjdEJTbzlGM1RuQXUrcFo0RmdyalVZdVZMajR6T1NQYXlV?= =?utf-8?B?ZDl3N1NBa3JiUldJeXBMSlk5MXBvZVJQUHhyZUF3dkFqOGpQM242aXFpWnBn?= =?utf-8?B?M3J0K0NNbkNKTVA2U0p0UGZtSVREd043b1NnTVRVQXorNHFKM1RVSVoreHdt?= =?utf-8?B?Q2lXNzFJSzNTTUgvRXJ6NUNXa283SU5NUnBsWkFOdmRDbXJJWlZQSVEwa3pu?= =?utf-8?B?MXUyMmFTZ0Q1UFN3YmZpa091TXp5WFdKWDdRSk5ieS9JSzVtWDh3QUVkTUUr?= =?utf-8?B?UmZSejhnZ2wybCtyVGw3Y21qRG9NNVpRZUlRY1BldTUxT3J5QjlHU2hqVHp0?= =?utf-8?B?L2o2YkZNLzVxemNaSzFOQVlFb3h4OTdzcjRiSUdCY0sybWxiY05BSDF4UG1S?= =?utf-8?B?TUxSdmpEeUkrb0syUnU3ejdHaUNtUHVuRjNwbXhHRit4WXBkY0hETXUwdGp3?= =?utf-8?B?RVVPZ3FOTmRmQll3K255M29BdWFGcHg2VjlVTyswRDhzeWdVMjNJS1ovNWs4?= =?utf-8?B?bS9GMnFvTTJBU3kyb205OVh0U00wdUxLcEhpUjY2UkpYNGY0eFhXNUE3OXRN?= =?utf-8?B?OXNaWm1YUkM2cTdtZkFoc2I0eVB6ZXJLOFF4c281ZmlwRkgwcEhrL2pqY1FQ?= =?utf-8?B?NUF6Vzg1OGpYY2o2Vjk0YXpyV01saUxiM042b2JjelRVY3VJMDZPRlJpclNO?= =?utf-8?B?dG5Cckk2ZnRTS1JvMUJ1RDc2Y01ZM04wTk5mSTQ4RGhtTitLQk1RZ09UYWdN?= =?utf-8?B?MWlBdDZQWGQzSGxWSXY2citMSGV0QlU1c01CTFo4ZWFkUE5VaDZ5TEZuY291?= =?utf-8?B?NUdicllZYU1EVFBLbDhiRS9DdkJ5ZzlxVjViTVppSktVZFNadktaUnNrem40?= =?utf-8?B?ZGJNTDNrWmpVeGQ4UGlSMWpCSkdXZlFsSjJOeFZPajdQM3ExRjltcnNYdzcz?= =?utf-8?B?cUcvNWtYaVVoR04zTXFKdHhtWEczRHRrcFMwaXA2VUZ5aldyVExVUWY1REFQ?= =?utf-8?B?d05JQ25xYUc5SVgvZkJYRVJBckhoYkRsMXQwVE91MHdHQU5DYk5tNVV1NXdw?= =?utf-8?B?WWZQd2t5d2xyalp5MlBIZGtVdHZnYWV0dXltNTZQU2dlaFp2cnhuNDJFSmQ2?= =?utf-8?B?WEs3enNJamc5Z0xmZTRlcmdkRG41RWJjZUUxSURBZm5VNFZNMmgzTzlRdnY1?= =?utf-8?B?dXpMbmV6ZE8rSzlSZDEzekF4UmlKQmV2blZUQjhBQ3ZXM3hCejZqTy9mWTVF?= =?utf-8?B?UXU1bjBZZ21ybUNscDFZQWdsL0FYaHBuT1kxVWVRYWpSTkExekdxNUU4N0ps?= =?utf-8?B?YTgvQlNCRGlNUnd4b0tybUIzZGxJSWRHc0lxcWJzKzVTaXR4WHVpS0dKeVVi?= =?utf-8?B?aDJ6aXNZbGVJb0R4SFh2ZjQ5RGVlZHkvZ0hUYzZHWXQralE5ei82MTBScVJJ?= =?utf-8?B?Rks1R2lSV2k0T1FqQmZTMG9aUW5vU1ZNK3dvTTIzU0M5bHYrRG5MRVovVzZY?= =?utf-8?B?VUtQQzJtZmNNN2M5d2RpbnM2empnTkZ4V1RpL2JLS215T295R2FZeTJFV3VG?= =?utf-8?Q?lRTVw4WTTK0KDO5h5KlhNxU=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: f6f912a9-05b8-4cc8-0041-08d9d451e839 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2022 15:57:30.5312 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5XAJGZuQTsZ5i3mInU8rXeyrazKZm8Mdxbs13X7WtapYH4kmKtNsqroZc6j8eybfVEjSXyiKEkdGzfIeJXbDbw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB2713 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JXdks6ptZz4Rt8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=MH7T7tsf; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=KK3T3YOC; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.145.153 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-6.20 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N On Wed, Dec 01, 2021 at 10:42:38AM -0800, Brian Buhrow wrote: > hello. In looking into this further, I discovered that the bridges on NetBSD have the > same issue, that is, you can't have the same MAC address on the dom0 side as the domu side. > However, I'd like to suggest that the NetBSD solution to the problem is more elegant in that it > is possible to track domains solely based on netstat output and their MAC addresses while, at > the same time, being able to derive both the front end and back end MAC addresses for a given > domain by reading the MAC address from either side. I'll quote the comment and the relevant > line of code from > sys/arch/xen/xen/xennetback_xenbus.c in the NetBSD source tree. Is there any reason this > solution wouldn't work in the FReeBSD case as well? If it would, it would be nice to see MAC > addresses return to netstat -i output for xen virtual interfaces. > > -thanks > -Brian > > /* we can't use the same MAC addr as our guest */ > xneti->xni_enaddr[3]++; Sorry, been on PTO and catching up on email. I'm not a network expert, I wonder whether this could cause clashes with other network adapters then (as a user could create another VM with a MAC address that matches enaddr[3]++). It would be nice to have a way to set the same mac on the netback interface and that things still work as expected, but I'm unsure that's possible. tap interfaces also seem to have the same issue, as the MAC on the tap interface seems to be a random one. Thanks, Roger. From nobody Mon Jan 10 15:58:40 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6C09E19371E2 for ; Mon, 10 Jan 2022 15:58:56 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXdmH3dwLz4RxX for ; Mon, 10 Jan 2022 15:58:55 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1641830335; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=eoXqHCTuyNVR6VO8lnhqjTVSUETCHKdKUpm5EtF0tCA=; b=KAFn1kleiTm78R7iegi6u5WS/wrCs50QsXPbRZveSgADV+Ky/pxq8wv/ w85aqEhj4IQyoZKlK6bvKOhuN1TSswdIRh/yMz5Qqyt73BBO3BnNoOatz CZNalRZzk44sGoexiLWM9EHfsXHWLOMNGq4PoXNe24CGWEn8osAvSd3MJ o=; IronPort-SDR: Qfw8NdoNkFrzhvl9s94ExHroT4dlw57ymRexH9rTu95LrFNctDI2UQllafKW/3XT1Fqwr50opi sMwZjPP8rK98Imw2NyvIbcBqFjiX/Rn6YJoMOG74YNEKhpyvr9ohvV9NILy1ETlgaNBcCN+cq0 3AKgNALc2rAT+MIKRmp/luJM9KPku02Mti+5JLZ1h4kdfpgZwfFVXSBOqXvZ2K2WdgoDq5sLZc BWcCWncfNERg+ZUdnon5kEDSut+r4EPVJKyWkKX+xa3rmORmuFQVKmGdmb2homqB5L9EPEx254 0ay12/PJmXeLI3MSw21qw8Rb X-SBRS: 5.2 X-MesageID: 61148720 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:ZdCK7qwzFKYSQAY5WG16t+f5wSrEfRIJ4+MujC+fZmUNrF6WrkUPm 2QcXWmHPKneajT0fNx0Poy//EgG6sKDxtZjQAZqpSAxQypGp/SeCIXCJC8cHc8zwu4v7q5Dx 59DAjUVBJlsFhcwnvopW1TYhSEUOZugH9IQM8aZfHAhLeNYYH1500g7wrdh2tQAbeWRWGthh /uj+6UzB3f9s9JEGjp8B3Wr8U4HUFza4Vv0j3RmDRx5lAa2e0o9VfrzEZqZPXrgKrS4K8bhL wr1IBNVyUuCl/slIovNfr8W6STmSJaKVeSFoiI+t6RPHnGuD8H9u0o2HKN0VKtZt9mGt/pOw YxAsa6ydTxqPZfx270tbDlWVAgraMWq+JefSZS+mcmazkmAeHrw2fR+SkoxOOX0+M4uXzsIr 6ZBbmlQMFbT3Ipaw5riIgVort4kI8TxepsWp1lrzC3DDOZgSpfGK0nPzYEBhWtg2ZoWdRrYT +EiZiJkSBWdWQ1SahRNFYIVpcyltnaqJlW0r3rK/PFqsgA/1jdZ3qLgPdDOdpmaSMN9kFyVq 2jd8yL+GB5yHNCE2RKP6X63lqnBh0vGtJk6TePisKQw2RvKmzJVWEZ+uUaHTeeRoHG3At9bG U4owWktl5VxrXyUbte6dkjtyJKbhSI0V91VGuw8zQiCzKvI/gqUblQ5oi59hM8O75FvG2Fzv rOdt5awXGE07uXJIZ6I3urM9VuP1T4pwXjujMPuZS8M+JHdrY46lXojpf4zQffu3rUZ9dwdq g1mTRTSZZ1O16bnNI3hpDgrZg5AQLCSF2bZAS2NDgqYAvtRPtLNWmBRwQGzAQx8BIiYVEKdm 3MPhtKT6usDZbnUynDUH7hWQer0t6ffWNE5vbKJN8J4n9hK0yTyFb28HRkkfBs5Wir6UWKBj LDvVfN5u8YIYSrCgV5faIOtEcU6pZUM5vy+Ps04muFmO8ArHCfepXkGTRfJgwjFzRZw+YliZ 8bzWZv8XB4yVPU8pBLrFrh17FPe7n1krY8lbcqln03PPHv3TCP9dIrpx3PVP79pt/3V8VyFm zudXuPToyhivCTFSnC/2aYYLEwQLGh9ApbzqsdNcfWELBYgE2YkY8I9C5t4E2C8t6gKxOrO4 F+nXUpUlAj2iXHdcF3YYXF/crL/G514qCtjbyArOF+p3VklYJqus/hDJ8dmI+F/+bwx1+NwQ tkEZ96EXqZFRAPY9mlPdpL6toFjKkim3FrcIyq/bTEjVJd8XAiVqMT8dw7i+XBWXCq6vMczu ZO60QbfTcZRTghuFp+OOvmu00mwrT4Wn+crBxnEJdxaeUPN9ol2KnOu0q9rcp9UcRian2mUz QebBxsctNLhmY5t/Ymbn72AoqeoD/B6QhhQEV7E4OvkLiLd5Gein9NNCb7aYTDHWWro06y+f uEJnerkOfgKkVsW4YpxF7FnkfA369f1/uIIyw1lGDPAbkixC6MmKX6DhJEduqpIz75fmA23R kPQpYUKZeTXYJvoQAwLOQ4oTuWfzvVFyDDd4MM8LFj++CIqrqGMVl9fPkXUhSFQRFev3FjJH Qv1VBYq1jGC IronPort-HdrOrdr: A9a23:sqo1Q69Xd096vi5jDG1uk+E6db1zdoMgy1knxilNoENuHfBwxv rDoB1E73LJYVYqOU3Jmbi7Sc69qFfnhORICOgqTMyftWzd1ldAQ7sSj7cKrweQfhEWs9QtqJ uIEJIOduEYb2IK9PoSiTPQe71LoKjlzEnrv5an854Ed3AUV0gK1XYeNu/0KDwTeOEQbqBJaK Z0q/A37waISDAyVICWF3MFV+/Mq5nik4/nWwcPA1oC5BOVhT2lxbbmG1zAty1uGQ9n8PMHyy zoggb57qKsv7WSzQLd7Xba69BzlMH6wtVOKcSQgow+KynqiCyveIN9Mofy9wwdkaWK0hIHgd PMqxAvM4Ba7G7QRHi8pV/X1wzpwF8Vmgjf4G7dpUGmjd3yRTo8BcYEr5leaAHl500pu8w5+L 5X3kqC3qAnQi/orWDY3ZzlRhtqnk27rT4JiugIlUFSVoMYdft4sZEfxkVIC50NdRiKpLzPKN MeTf002cwmMW9zNxvizypSKZ2XLzkO9y69MwY/Upf/6UkVoJh7p3FosPD30E1wsa7VcKM0l9 gsBJ4Y4Y2mfvVmHJ6VO91xMvdfKla9Ny4kY1jiaGgOKsk8SgDwQtjMkfEI2N0= X-IronPort-AV: E=Sophos;i="5.88,277,1635220800"; d="scan'208";a="61148720" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z9hugJ3OqyUXUT4SFiNo9nN1qq0OoeeiUB3scpj/ghdumpHy0THkM91bHebLItpZ9vFJgRpv7TDgN+ju1yfVUAc48Y7uTMt9EAtDrHYCFeGfQMsVZqKpl/Eqx0AQwzo34pOjxFLA+SVbw9gKjK5ov8AXLtxV1k8fUD9nUUCiBlIbP/G9D2SN5JdyLTRKNM3FIHKJqQX16h4Suibi2VDUV14iLBst+rpkvvIxyhUt4tUOcCeJEBQV60LX0YLW1mY1cSnMCc53zK/H693wIMB0BKvhgldT3vOjd+tQXhXI88CcgyuI41kjHmiaSudrvAP88dr4sjMu3J6PdCSPJ1wQ6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PDHT1cnymDHyaQX279P9K4R7fSobrlROChAX3/NLiiU=; b=VWdiShPtMn3ZFJdwTj9VoL+Gj2dgtVz4XCbqkVWaJvagDOxPLNMSn0Fbrt6ZwuL6kA+0gEob6TMrlsWHu4sB2HbeIMwCw2wHFbPEMt/hEpjkgdW4qGJg/6eIQ2PkfUynh1UJtZOgDiZ/suyaik/j5SAbdTm1x2lcS5phsyxj/4QoiRh/dhfPBmDwj5coWDuK3zSTqGjjwaSeUCU356zi+2BKhL6BlmwZkKRJFCgCJUZNzsHx8/e8EpKwhcAEWiXRpcSaV2UT3cmdANEgncAhYkhgHKzNxBKDp/223kPqFpoOh7qkx8VJpQN7/YWbWobwIuZGRca71wW3Nbks1GJiOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PDHT1cnymDHyaQX279P9K4R7fSobrlROChAX3/NLiiU=; b=PHK9K+FnPXhI5r9Nfx0RXZ3y4la803HgNFZ1PdgdMsO05Sm5b1DQKLXMgbeu1UbbN1MbjHKZQeAv6nMDug5XOsbka7Sw7MhLfYhhOGXhaGMyJTryqibIVGgPIbzbSWbuPenEOxKgrzcE77lqhClUaMsChyCdTcWM0EXs53HEftw= Date: Mon, 10 Jan 2022 16:58:40 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Freddy DISSAUX CC: Subject: Re: Can't shutdown FreeBSD domU Message-ID: References: <20211204093337.GA69642@blade.bsdsx.fr> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20211204093337.GA69642@blade.bsdsx.fr> X-ClientProxiedBy: FR3P281CA0088.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1f::16) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 41602655-9e94-4835-d0b4-08d9d452140f X-MS-TrafficTypeDiagnostic: DM5PR03MB3209:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:747; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SVNZZX+PW354I1xgYsKLDSpXCYvuhVrw5jXzfN7c/E5djuipWAXfBYutg+Fp9jbU8PvGyOOoDXN9JDOK/bNKMOZF3Fz5RtiM/Nct2LLaMEwjOSYC7+maCNPz8KQ1Nd8fwz/TMr34dz4rU30BBUkaCFJrchgKjQnscHvpHr0lKXXGVNmJjzQGiI4UkMYwDHLUp+zS6W8p5X4PTEVKlLp1G699aQeLQGzucnjeeRceuWePhX3orZ0i0lWAyKzt8eIIw79ut+AIntxLRgpZq6peFmvVs1dyEVL0e0dQRP/5RMFTNZKe6wbE4HTh2M0Fq0zYpFZes9WcZZuV+SzLJHDaDr9bx0Ka3KIkzscWHEyEjNwOUuSE6mt3L3DoZhLVq1kNW45K3ihivekk2rm1CVvNx/2UyplX4S8mHpH4spOF1crWc5cRYu2wwVT2b4eVmhv4ESaiL4bmoNkc1H/pc8UE74lUe3iL9efMJkG8juHHWt1a87X+y+f9SrOKP7OR/mHiLdA6GblPlpUxTSkxNjp5xOaDkRu62nWfs0bRtA3GP/8cJW6ghk+xfUNu7nd9DH/XIp3Fh4NBWBYsliFXotUA90KByxyH0HCZPcMuvQ83MoEtS+4Cb4rU6dsHqU2kPeZ4NU18xvV9S2TKxvfaQE5OkxsR4PzQ/TGtu8IzuaMYyzxP5iqqQbOYF9Ao0jH1r3Pf+7VAO+oowuZibz18gOZyuQWmjNotA+q0OijfXuLMEHrGRDWS1pMRBRS70mHLNZzqwRN+TBNl618QKhMafUPNkbP51WsoXrU4EbrpTaNOWmFxTWCUGT5RAQFfhLFfHq4Q X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(66556008)(4326008)(8936002)(26005)(6506007)(2906002)(186003)(316002)(5660300002)(82960400001)(66946007)(6666004)(66476007)(6916009)(38100700002)(8676002)(83380400001)(86362001)(966005)(85182001)(6486002)(19273905006)(508600001)(9686003)(33716001)(6512007)(3480700007)(562404015)(563064011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?anRNUW11b3ZOUFNzcHFSUEhLdk95OUp3dGU4KzFlWjNnUXA5RWN5dmdUUGRO?= =?utf-8?B?aGdONFh3MnRCUlN2czYrRlkwMzU1OWlDQUZqdnJDNE4vZUJ4UXNrZzVaSVZC?= =?utf-8?B?WnFEZlBUanNYS1JmR05wWkxYK04vNGw5VzVmYXNtM2FHQXE5Q0s1dkE3OG1R?= =?utf-8?B?SXFNYlZOZGlZK2RUclBRRFZieUw5cC9HTkg1eFB2TnJ5SXdwaHVqMGk2RFRS?= =?utf-8?B?SFFpZXlsMksxeTNqUlVnbkxWcTFSNWVHN2t4bzNiNDVLZ0tnSDl6NXIvZUJN?= =?utf-8?B?STdwc1VZR1FYcFVGTUZUNTZHamdvVzVLbWdmdzgvcGpKdHJpZDlzY3d1WWNX?= =?utf-8?B?WVFqdm8rdXBCL2EvT1hueEkxbGExbFpiTEMxTnRiS2VmaUh1YVJIN0JUeVFG?= =?utf-8?B?K0poQm9rWnNTYWxyVjlYbnMwemc3eXlDR3M1WnBMVGV6WjRBMy9ST3g2NDZO?= =?utf-8?B?U0hjWEY0TlpaN25hR21IOFRIc0g3enFSSGpzSWhYU3JsZkQrKzc4YklLK3lQ?= =?utf-8?B?YXlMMFRkcXlBbFJiZWRkS3BhbkgvekhFaklZK0hQOTdDaEtoL1hLM2ZZMVQ2?= =?utf-8?B?V3hsMTZGeGoxRXR5WUZmN0Y1cHBMazdwVFhtYkY5aDdmVWVFWHVzOXc1T2pH?= =?utf-8?B?Z0lkeGVlR25wM3BFaUdZWTBpZnU2ZHdLMTZxb251NzA0NnZUbkdHcVZVYWlI?= =?utf-8?B?NkU2TWpMTVRoWG9oQUQ4TGQ3aHVnblNFWEd4ZnV1UFI1MVIyeUpuUjJUeXpa?= =?utf-8?B?Nk5Jc29wYzJkbmhrM05EMGxHL2g5b0NVQmtOSDZEbyt3NVRVQnVmK3l3WG1k?= =?utf-8?B?c1JaR3lXWEg4eWg3Q0s4ZnVwTEZacktQd0pPWWE3Skh3S2JTcHRqdVJWbVNt?= =?utf-8?B?aGtUOEFXUW81MnZHOHZPb3lUNTU5cWNYajhUanZyeWh2Wm5ZRi9nZ1VwQWRl?= =?utf-8?B?YUpQQWdTbnpLWGVHVnoxUHo5NDBIYnJGT1NITGtQRTFMNk8vaVhTSHdBSzM4?= =?utf-8?B?ZkRBMVRCSG0wU1hrT1U4Y2lmOG45T0tsRTFzZStBUnkxUk1id2VDRWVZdndk?= =?utf-8?B?L29UUElZZ0tlS05mVzhRVmhWMWUzYmRPMWI1SWUvbVoxVjJiREJZQkl1VHdB?= =?utf-8?B?SUlJZnhFRGhnbHBQSUpBcmY1UUtCZEt5UWNZL3FPcFpJeGJiRFZ3SElZVGY0?= =?utf-8?B?MVl1ajQ5bHRIcHlzL05YMDIvM1VVaWlQbEpNWUo4bG9tM0dzTkJiVUkxK04r?= =?utf-8?B?ZWJKSkRvN1Y2bzlUUS8rTnB0K2xiZE5qMmo4WC9lbTJJdVJqT3R2Y0tyamNF?= =?utf-8?B?THovdkVrTjNoZ055dDA5cDhYbytRWHlmYkJaS1doSDNoZVpmL3RTWkNsbDNx?= =?utf-8?B?M0M3QUZNYUpVNGFjcEFobWdCcjRCdlN3VWRsRXJlTXRxa2hVcGxyZkwyL2F5?= =?utf-8?B?OHNoZUxyeGNtdVJ1S0pQOU1XR1FFV3FDSzdndzIxSzhzTlY3U0xNZW1PTG1U?= =?utf-8?B?enZIR2ZmL0UvelM3akhsckRFc2d0ajRPN2FTNGhaMFNyQm5CcTZ6SitFU3Zu?= =?utf-8?B?VDhRNHRTbkVVS0JhTWRDbUtLVGZHYW5BbWNWK2JpWlora1gvSVZvVXFFSFN4?= =?utf-8?B?aXVmZFhUMFhlNm5xaVFERnlpWERWTWhwdDFEOHFLM0czdUdZeTRSdXMwQkVq?= =?utf-8?B?YTFMV2xyNXNycHBiNEQrdHQwL3BBWi9zUjdUVWtnSEh6QldjNDJ2OXovZlQ2?= =?utf-8?B?VWVsUzkxc3NUbG41bzU3eXdvYTZRbFNJMDh0c2gzRXg2ckMyRkhaaVZab0Vq?= =?utf-8?B?Sk1yVDNrYmdvaHh4Z3o0aHNVaUtOQnhxNjRLakNybFRpT2VRWE1kZUFWSzVV?= =?utf-8?B?WVQwTTZONkdWQlgzSm1tTVpoVDVZT285aFhJcG9EVi9WUWJyY2RzT3phSVYx?= =?utf-8?B?ZlBZeitqeWwzUVp5YmNTTkhjSEh1V040bnVjV3F5cHFXc2dKSVNjRXA0Nmx4?= =?utf-8?B?aS9JYno3YTE3NUxnNC94Qk1KbEFRVEYwV2xXa1Z6eitRUHRacnZvNTVwRGJW?= =?utf-8?B?c2dJMFY5dlg3Wm9XblB5L0VmcXY1VUEyaUpxSEpaWk5QMWExSlAyMjdreFRr?= =?utf-8?B?NDFEOGdaWG04aTFXMWwwNDVmWE5IRFA0RUZ0QjN5THUwUkd4dFdqRXZocEJj?= =?utf-8?Q?EBsXO/K14asUlBMSV7B0xHI=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 41602655-9e94-4835-d0b4-08d9d452140f X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2022 15:58:44.1122 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hDIXNVhIk9dxTZDBhfWcYZnPoFOVmLkyfhr1G/wqi/W6CXIOcMBnZDrq+T/Z+NpPSPhtHzmjQj1VqHjlVdqUOg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB3209 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JXdmH3dwLz4RxX X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=KAFn1kle; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=PHK9K+Fn; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.168 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-5.58 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.62)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; MID_RHS_NOT_FQDN(0.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N On Sat, Dec 04, 2021 at 10:33:37AM +0100, Freddy DISSAUX wrote: > Hello, > > My dom0 is a alpine linux 3.15 with xen 4.15: > > dsx@xen>cat /etc/alpine-release > 3.15.0 > dsx@xen>apk info xen > xen-4.15.1-r2 description: > Xen hypervisor > > xen-4.15.1-r2 webpage: > https://www.xenproject.org/ > > xen-4.15.1-r2 installed size: > 262 MiB > > All domU are in pvh mode. I can shutdown domU alpine linux, domU netbsd > but not domU freebsd: > > dsx@xen>doas xl create al30.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait al30 ; sleep 2; doas xl list > Parsing config from al30.cfg > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1022 4 r----- 2581.2 > al30 66 1023 1 -b---- 2.1 > Shutting down domain 66 > Waiting for 1 domains > Domain 66 has been shut down, reason code 0 > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1022 4 r----- 2581.7 > dsx@xen>doas xl create nbsd20.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait nbsd20 ; sleep 2 ; doas xl list > Parsing config from nbsd20.cfg > WARNING: msr_relaxed will be removed in future versions. > If it fixes an issue you are having please report to xen-devel@lists.xenproject.org. > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1022 4 r----- 2583.2 > nbsd20 67 2047 2 -b---- 2.2 > Shutting down domain 67 > Waiting for 1 domains > Domain 67 has been shut down, reason code 0 > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1022 4 r----- 2583.7 > dsx@xen>doas xl create fbsd40.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait fbsd40 ; sleep 2 ; doas xl list > Parsing config from fbsd40.cfg > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1019 4 r----- 2584.7 > fbsd40 68 2048 2 -b---- 4.5 > Shutting down domain 68 > Waiting for 1 domains > ^C > dsx@xen>doas xl list > Name ID Mem VCPUsStateTime(s) > Domain-0 0 1019 4 r----- 2585.0 > fbsd40 68 2048 2 r----- 170.6 > > > FreebSD domU config: > > name = "fbsd40" > maxvcpus = 2 > memory = 2048 > vif = [ 'bridge=br40' ] > type = "pvh" > kernel = "/home/dsx/kernel" > extra = 'vfs.root.mountfrom=ufs:ufs/rootfs,boot_verbose=1' > disk = [ 'format=raw, vdev=xvda, access=w, target=/home/dsx/FreeBSD-13.0-RELEASE-amd64.raw' ] > > > dmesg: http://download.bsdsx.fr/xen/dmesg.txt > > Same problem with FreeBSD dom0: > > dsx@ni3>uname -smr > FreeBSD 13.0-RELEASE-p4 amd64 > dsx@ni3>pkg info 'xen-*' > xen-kernel-4.15.0_1 > xen-tools-4.15.0_2 > > Any hint ? This is a known problem. IIRC it's all caused by PVH having no ACPI SCI interrupt. Will try to find some time to look into it and hopefully find a solution. Thanks, Roger. From nobody Mon Jan 10 16:11:22 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F18C0193A9D1 for ; Mon, 10 Jan 2022 16:11:40 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXf300K2vz4TcK for ; Mon, 10 Jan 2022 16:11:39 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1641831099; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=emibXei8aB86ADsa/e1z2J4Z57jr+oaZiNoiQGScGrA=; b=PNuXPWO1S7WhFiAX8cRcKRlcnl/GJ3XXt4YBCmUFkGpZlq0Nnm+cGMyP gldgLasE0t4DAwWcqJ/82d240vkEOgjB8mVsTDbGU6NrORATfvZ1cdrPD Jf5dY6ICZfA27zOU9erXuS8cBTTzvkk3W/D9rQkqtA0QlVBM/16xg+CF5 c=; IronPort-SDR: 41F1rey9m55AMLTJDKVfTihMSjLwSrOvjTC5RCmfH3AJovlcN2ubKG3bgl3CYlVnYFv/yJJI6y h/y6/mMgUgAgutuKq8iDsLdE9f2HI4gkgvxxYHkcusSdmlBfO9/yCAX9p//8Stwb+Dz/3YKYgk VRZHAifeaus2y1rt2+tvIswzOI5EI0AdNZynrfwnm5wal1fsrs8cVJf1071fMYnGQuz40HAS8F lLO93zw9M07VL+nF7QN92/eU995rwB10sOkOQmwo3pcS/O5HkEzWZMC2XDdb3J3xe14Pmz7dsM O0+xLm3zMsgOBxDQKK+mzcLk X-SBRS: 5.2 X-MesageID: 63784019 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:zErUOKJFa1OOu29qFE+RH5IlxSXFcZb7ZxGr2PjKsXjdYENS3mBRy GAYUW3TOPeONGbxeoh+Poq29k8Bu5TQx9NmQAplqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s q3yv/GZdJhcokcxIn5BC5C5xZVG/fjgqoHUVaiUakideSc+EH170Us6w7Zj6mJVqYPR7z2l6 IuaT/L3YDdJ6xYsWo7Dw/vewP/HlK2aVAIw5jTSV9gS1LPtvyB94KYkDbOwNxPFrrx8RYZWc QphIIaRpQs19z91Yj+sfy2SnkciGtY+NiDW4pZatjTLbrGvaUXe345iXMfwZ3u7hB2sjst27 olrs6aSaj0sDLDqwuUGCytHRnQW0a1uoNcrIFC6uM2XiUbHb2Ht07NlC0Re0Y8wo7gtRzsUr LpBdW5LPkvra+GemdpXTsF2gcsuNo/zNZ43sXB81zDJS/0hRPgvRo2UvoAIgG9p16iiG97fV tgyWRNVNyjNahZpJ3gLJLEwpOyn0yyXnzpw9wvO+PtfD3Lo5AJr3bXnK9aQatGMbctPk0uSv W6A+H72ajkWL8y3wyGJ6WLqiPWnoM/gcNtMTvvirKcs2QDNgDxIYPELabelidyokUmYB/RZE W43wC8csI5o9VfzfMaoCnVUv0W4lhIbXtNRFcgz5weM1rfY7m6lO4QUctJSQId47ZFrHFTGw nfMxoq0XmI37NV5XFrAru/8kN+kBcQCwYbujwcgRBBN3dTsqZpbYvnnHoc6S/7dYjEY9FjNL 9G2QMoW2u97YS0jjfzTEbX7b9SE/cGhoukdvFW/Y45dxlklDLNJnqTxgbQh0d5OLZyCUn6Kt 2Uels6V4YgmVM/RzXPQG7hWQeH4vZ5p1QEwZ3Y0RvHNEBz3qhaekX14umkidC+FzO5ZEdMWX KMjkVwIv8ICVJdbRaR2f5iwG6wXIVvITrzYugTvRoMWOPBZLVbflAk3PBL49z2zzCAEzP9uU b/GIZfEJStLUsxPkWvpL9rxJJd2nEjSM0uJG8Cip/lmuJLDDEOopUAtawrRPrtnvfLd8G04M b93bqO39vmWa8WnCgH/+o8PN1Eaa389AJH9sctMceCfZAFhHQkc5zX5mNvNoqRpwPZYkPnm5 Ha4VhMKwVbznySfew6LdmpiePXkWpMm9SA3OiklPFCJ3Xk/YNnwsPdDJsVvJbR3pvZ+yfNUT uUef5nSCPp4VTmaqS8WaoPwrdI+eU3z1x6OJSesfBM2Y4VkG17S4tbhcwa2rHsOAyO7uNEQu bql0g+HE5MPSx47VJTdae61zkP3tn8YwbogU0zNK9hVWUPt7Ik1dHCh0q5pe5kBcEyRyCGb2 gCaBQYjidPM+4JlosPUga2krpuyF7csFERtAGSGv629MjPX/zT/zNYYAvqIZz3USEj95L6mO bdO1/j5PfAKwARKvo57H+o5xK4y/YKy9bpTzwAiF3TXdVW7TLhnJyDej8VIs6RMwJ5fuBe3B R3TqoULZ+3RNZO3CkMVKSokcv+HhKMdlTTl5PgoJFn3uX1s972dXEQOZxSBhUSx9leu3F/JF Qv5hPMr1g== IronPort-HdrOrdr: A9a23:+KrXf6uTBuXpLmbwDnqqXjif7skDjNV00zEX/kB9WHVpm6yj+v xGUs566faUskd0ZJhEo7q90ca7Lk80maQa3WBzB8bGYOCFghrKEGgK1+KLrwEIcxeUygc379 YDT0ERMrzN5VgRt7eG3OG7eexQvOVuJsqT9JjjJ3QGd3AVV0l5hT0JbTpyiidNNXJ77ZxSLu v72uN34wCOVF4wdcqBCnwMT4H41qf2fMKPW29+O/Y/gjP+9Q+V1A== X-IronPort-AV: E=Sophos;i="5.88,277,1635220800"; d="scan'208";a="63784019" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jrXR+itCIqB1YwH0vBQEE+MOQQutzB5GWLCMO2jYJ1WDgWzv3TYLNE+2Fn0xYYibLsEZF7isgIqR5bmVed0fMv8DwSX49ur7B/IlT6YRjiFzrQ46Or8ueCvvpVXxgxfQDjBrdMLpDuk2G8ESl+AXYXrwESMo6YzEFFhenvUKpDsqmPworHG0uAzhGaphoOY00IyLZzuRpY8hv9A1W2w5SuQm5fhGzozk+oFK3zTtftKwqRo6tZ+D9PKdjiLEh1aUDwSJts7b5vM21KilfHB/WTTd63Bfai4bspeJQ7LcaZFzMrKC0E21+Ho2mc34PWzcs9JshaFHBhhtHbjzpWxvwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QnQdq0NLTqvFwmfYujQ+BkP5An9N2PB1ypt52c06A00=; b=dxuoRyNLAuNrQixeEclJLNr9pJuvEGHXcwWcEps/QLhkqFy60YbTVx5i+SstETv+44QdVMkEgy3eXiZqjGzi6VmjzCZnhF0tEXPuEN9IiF3aIe4fPOiq2TZbauWFCygsydzL2jGbSP5oGdt5ySLbS4+RzrQjK1ERxoNeZyh2iaamPNID/9WdLc/x1Xq2IB3517/OY99ywJKYm87rraQ1rsiz2I5ExIya0qw5KF8CUF7mUFogjbV87ZMH9ltj19xy/POj5b94ucjPjbTwHF/9rzH3RZuTzdsuwTHHhrDzPTm5sLv3gr2O/Ijn/++zCq2X4HcnwFhLU8IOUojmyC640g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QnQdq0NLTqvFwmfYujQ+BkP5An9N2PB1ypt52c06A00=; b=sXZ2/mriC84Ha6OckP/DwjzJg7pJLTe+6q6GWpAPngRz3+114SZKK9GL8iuv+Ml6IJDIhYNNaOKRaWZAQbE7VdTpcDMmLaqTO9JvipkGB6sY4zc7reK5nApiKZCC9dICHWTj/RNLSPBDq2j6tiKUsi32xxV82AJwV+MUBBNmUMo= Date: Mon, 10 Jan 2022 17:11:22 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Freddy DISSAUX CC: Subject: Re: Can't shutdown FreeBSD domU Message-ID: References: <20211204093337.GA69642@blade.bsdsx.fr> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: MR2P264CA0108.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:33::24) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f79a575b-ca9d-4887-e9d0-08d9d453dbe6 X-MS-TrafficTypeDiagnostic: DM6PR03MB4603:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2803; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hQ6p99KGFC1QR0YMglYAXBAyRe17EBk6t27SccXP7p1yiI1thvdp8/Get3bTa3+rmyuARJMWggCLGuzTGZYVP2lfBtwRjfDEyaxiQWlq8hEyZywZikt85bSxonxjabyFVGw7IrubM+0aWHXfXAPt8pZzDxel2QW/ILVfY8E7qLYFpHIrfquKNdZrir+0VODBM74uBBvjWhEa9ZmEDDRPTPzTdk0ngV66ZrAoLJcJURFLxjthHCmeYsIjlYPApFUVt8hjd3ydKgke4JRSlQXwVijSBoxYpS78uzql/2BHXxYHmlx6E22tpQTE6RK9LNgFnPRqsKS0+HY2TIT+rvByX8ZLC4EDFLW3+/aQ27AD818s91a0IRZYTzPjjoy0utOz6kLg0P0R+jI0vyfe5Zk8pGous1a+AQMKNLUjbuycNcsAzjNXXxt+LlIn561nb2aNrF3Zgb8eZ/z0yNNGdsu/SSgSMe7+osp90MybWBH+eDB8+x3wkyBtqRXZsgwllOpvX3PawpefyCrvYwLOQn7aXaj03H0vXMmAITDYQSHK5E1Om4qU4QwexMXeS7XPWNQuRzHp8Qa2pH9k4lDMt6ldjC3F4KUbHRrvkr2gnrO+/A0sBTAdNhld8dERXnlQv8gL78yojcqiTKf2J5Vp5ifkcLQYjzUoP7vbb/RkwIpYMLzM3xoeVINXfa0BMBhNnwSxl5kQFJVbBV05q81dayylgMwd7KpCkIoqslldjQg3MMmG/+tT9VwvjWd9lH1c/3k0W3We46YpqnkKJp/Avip2cDKKOLACIFyBJ0X6nR1p6Yg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(6506007)(2906002)(85182001)(316002)(9686003)(6512007)(66476007)(6486002)(6666004)(508600001)(6916009)(86362001)(26005)(19273905006)(66556008)(186003)(966005)(5660300002)(3480700007)(8676002)(83380400001)(38100700002)(33716001)(66946007)(82960400001)(8936002)(4326008)(562404015)(563064011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c0tjSG9MQ2c2TDNDNlRDUkYyYWI1N2JXdEE3OUptUFVTSllFcWVGdWlzNGRh?= =?utf-8?B?THo2K0VKM1c2U01HTDVvYjcySmJjSzJDUWt2d3NQUlIwcllaQnJTRThOTVcr?= =?utf-8?B?aDdtcjJNdGxuQkswRXJwbHBYTFg0K0pPZ1Y2dUZCK1B1RVdQVDRtZzRUZ1lv?= =?utf-8?B?QVpQV05QZTI2N2JnVjdpTU4vL3RzM0lpYWhQZGtPK3hkSkF4a1JNQkFISW1F?= =?utf-8?B?TnJvOWl4RjVpdnpXZFZBekRSd3BGZDlNMVB4ZFdjV0ZTa25zaFN0Z3BFREhn?= =?utf-8?B?Ujg4eTlFSXlJR0drVEtkd1hqUERkOFkySEJDSnN0NjdxYk0rWmhFUjMvdzFw?= =?utf-8?B?VzIwU3RValVSbHd3V2pjSUFsNnoxWFpZWlp0R1hpK1JRNWFsM083UnlFQWJj?= =?utf-8?B?V3RwdW5ad3F6ejNzNkhoc0MxOTJaUG9XNTJ0V3IrWEFZTnJRK0ZHdjlYdzU0?= =?utf-8?B?SGZLTlRWcTBCTjVLNDh0aU9SMG90cXZyMlRQYUNBTSt0S3Y3ZCtUdmxPYWRa?= =?utf-8?B?Q2lNdk9lR0NleWR4akdmODBLbFpkVXp0OE56UHpSUWxlVzlxbitadGtyd2Qw?= =?utf-8?B?aUx2UDNEcFE2aFJBY2EvQjlQL0VLZ1V5cm43Qkt0em00bFR6Z3ZGek1EVzBI?= =?utf-8?B?bzg4eUhGZVcyWStIalhhUERGRDJsb09KS2JVcWxuRjdYdlAzSDZmREJBZm9h?= =?utf-8?B?TURESmNQQXlyWDBBbEZNVmQzWGwreUVBaGQrbkk2WnNnNGhJdE5leDJEeXZP?= =?utf-8?B?K1dtYmRiUkRvZXlTeWJ4MUdrM1N5SVZ0TGwrSGlHM3Z2NThPWUtGY3FOQzla?= =?utf-8?B?U25iS09XVFpFLzA4ODhtUG90SGtqRko5NFNGRGNtWjIxOGk1TFN3UzFjWXlr?= =?utf-8?B?NXhJeHZodUVEVDM3NENkTmlkaURhRU5rM25qM01rRjk3Y2kzL3gwUjAzRlJM?= =?utf-8?B?amNLVTdPS09QRlhNdjZUTno0UmpmbEtWNE00bHVtU2ZWZkdMZUJhYkx3RkFW?= =?utf-8?B?Um13T20zU0IvT01Fb1dSUGpZczhGbHJpQ2ZZVDgwRnhBbVB4QW9PSzcvTzZB?= =?utf-8?B?akV5OWQxUVpoREFNSmpNdTdXZXlCRTJlc1Njd0pPb1B2T0V6VDdoRW8vbHpp?= =?utf-8?B?a3FQanQyUGJ2cE9OVFpIUHREdUlQTUszVEpKZk0yem1DMkhDdlZKSVFrSTNx?= =?utf-8?B?MHlobW9Ud3NCcjlvclRQa3NzckYyVVR5VnRqd1E3ZUwxMGVHbmw1NUE1eG1C?= =?utf-8?B?NURmT05IejVIcWNSQno4YjBlNGt5UEUwaEpuc2h1NUpRYjd4YlBBcmJ2cFps?= =?utf-8?B?QzNMTUNJUTVNekxHUk00M0tQVWVoVXlVTjBIVUN3bUlDckd6QkM2dmhIa3ds?= =?utf-8?B?QTZUOEZWYWFIY3FSc2VuTG9mT3VGanFzZVZoV3QvZkZWdzEycCtuNkZ0bWxo?= =?utf-8?B?S2k5MmhoU1p1ektEMXFJU3FDa2U2ek9STk44OSttZ3E1aDdDQndmMFdzNHR4?= =?utf-8?B?Y3cwTWZYSVhJUDRrNy9Tc2w4OXc1RjQrejNheFM3K1JvWHloc1lYSzhxMDVQ?= =?utf-8?B?eFJRckpESXJqL2JrWVVwYVdBN1JQQUduL2R0b3YrTUlXdStpYk5tTld0QkRD?= =?utf-8?B?b3pGKzN4QkdRcE4zMGxWaEsyRzB4ajNOanhrdENRbFlsKzcvd25VdEhRNXU1?= =?utf-8?B?ajE1Q3J3cGNQSldqWXNaNXEyZUpSVEdmQ3VwbnZSTFNFdUtaNDB6bnNvMEEr?= =?utf-8?B?N2N3WHRMWEFCcTlPd04vRTBlQnRMRC8rVjhDY3N5R2pyRThWejVhV1FoYkkr?= =?utf-8?B?ZEtWQ1lHOFl5Z0hOYjh4ZlhWMkhsbTVPcm1RY2d5ZDJvZTZ3YmtCZXJPS0dY?= =?utf-8?B?UWc2ckhNQy8yMzdRbGFLU0xkbmF1R3o5LytWVGlYTWhJU1FvMkFRd3I2eXBR?= =?utf-8?B?N1lJRWd4YjJBRmVZVDh6NjZ0NGh0SGdJR2J3TlpkdklEYlVGSERDWjNvNy9E?= =?utf-8?B?T0NkSW9tZ2dzaTBHM2dWSXVFR1Q0MVpUakVYMzVrM2N4enBsQlBMWW9JQWF5?= =?utf-8?B?QlBLcFBObE9KeEJBV28ycmFxdXhvNDloaURZQjZPNGFmaWpVZmFHbjJrNnMv?= =?utf-8?B?UVo5emszZUNPMFFyL2xjYXFNV1QrU28vL1NXMDRndzhES0dyOGJXWWRqQkxH?= =?utf-8?Q?GCxy15cPOp/uOyhSXi2RjxQ=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: f79a575b-ca9d-4887-e9d0-08d9d453dbe6 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2022 16:11:28.7446 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vP6r/qzCEq1lcQFVzJ8vnc3oKEP8dh6LHdaX+PtuAasUD+0QmR+wghu4DVYEwQc7EQXzy6ejDU1WjTbCX62qXA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4603 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JXf300K2vz4TcK X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=PNuXPWO1; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b="sXZ2/mri"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.144 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-5.57 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.62)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; MID_RHS_NOT_FQDN(0.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N On Mon, Jan 10, 2022 at 04:58:40PM +0100, Roger Pau Monné wrote: > On Sat, Dec 04, 2021 at 10:33:37AM +0100, Freddy DISSAUX wrote: > > Hello, > > > > My dom0 is a alpine linux 3.15 with xen 4.15: > > > > dsx@xen>cat /etc/alpine-release > > 3.15.0 > > dsx@xen>apk info xen > > xen-4.15.1-r2 description: > > Xen hypervisor > > > > xen-4.15.1-r2 webpage: > > https://www.xenproject.org/ > > > > xen-4.15.1-r2 installed size: > > 262 MiB > > > > All domU are in pvh mode. I can shutdown domU alpine linux, domU netbsd > > but not domU freebsd: > > > > dsx@xen>doas xl create al30.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait al30 ; sleep 2; doas xl list > > Parsing config from al30.cfg > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1022 4 r----- 2581.2 > > al30 66 1023 1 -b---- 2.1 > > Shutting down domain 66 > > Waiting for 1 domains > > Domain 66 has been shut down, reason code 0 > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1022 4 r----- 2581.7 > > dsx@xen>doas xl create nbsd20.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait nbsd20 ; sleep 2 ; doas xl list > > Parsing config from nbsd20.cfg > > WARNING: msr_relaxed will be removed in future versions. > > If it fixes an issue you are having please report to xen-devel@lists.xenproject.org. > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1022 4 r----- 2583.2 > > nbsd20 67 2047 2 -b---- 2.2 > > Shutting down domain 67 > > Waiting for 1 domains > > Domain 67 has been shut down, reason code 0 > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1022 4 r----- 2583.7 > > dsx@xen>doas xl create fbsd40.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait fbsd40 ; sleep 2 ; doas xl list > > Parsing config from fbsd40.cfg > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1019 4 r----- 2584.7 > > fbsd40 68 2048 2 -b---- 4.5 > > Shutting down domain 68 > > Waiting for 1 domains > > ^C > > dsx@xen>doas xl list > > Name ID Mem VCPUsStateTime(s) > > Domain-0 0 1019 4 r----- 2585.0 > > fbsd40 68 2048 2 r----- 170.6 > > > > > > FreebSD domU config: > > > > name = "fbsd40" > > maxvcpus = 2 > > memory = 2048 > > vif = [ 'bridge=br40' ] > > type = "pvh" > > kernel = "/home/dsx/kernel" > > extra = 'vfs.root.mountfrom=ufs:ufs/rootfs,boot_verbose=1' > > disk = [ 'format=raw, vdev=xvda, access=w, target=/home/dsx/FreeBSD-13.0-RELEASE-amd64.raw' ] > > > > > > dmesg: http://download.bsdsx.fr/xen/dmesg.txt > > > > Same problem with FreeBSD dom0: > > > > dsx@ni3>uname -smr > > FreeBSD 13.0-RELEASE-p4 amd64 > > dsx@ni3>pkg info 'xen-*' > > xen-kernel-4.15.0_1 > > xen-tools-4.15.0_2 > > > > Any hint ? > > This is a known problem. IIRC it's all caused by PVH having no ACPI > SCI interrupt. Will try to find some time to look into it and > hopefully find a solution. The following patch should fix the issue, will test it doesn't cause issues for HVM guests. Note that migration of PVH guests won't work also. Thanks, Roger. --- diff --git a/sys/dev/xen/control/control.c b/sys/dev/xen/control/control.c index 426682a8f2d9..b42b52a62411 100644 --- a/sys/dev/xen/control/control.c +++ b/sys/dev/xen/control/control.c @@ -347,17 +347,12 @@ xctrl_crash() } static void -xen_pv_shutdown_final(void *arg, int howto) +shutdown_final(void *arg, int howto) { - /* - * Inform the hypervisor that shutdown is complete. - * This is not necessary in HVM domains since Xen - * emulates ACPI in that mode and FreeBSD's ACPI - * support will request this transition. - */ - if (howto & (RB_HALT | RB_POWEROFF)) + /* Inform the hypervisor that shutdown is complete. */ + if (howto & RB_POWEROFF) HYPERVISOR_shutdown(SHUTDOWN_poweroff); - else + else if (howto & RB_POWERCYCLE) HYPERVISOR_shutdown(SHUTDOWN_reboot); } @@ -453,9 +448,8 @@ xctrl_attach(device_t dev) xctrl->xctrl_watch.max_pending = 1; xs_register_watch(&xctrl->xctrl_watch); - if (xen_pv_shutdown_handler()) - EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL, - SHUTDOWN_PRI_LAST); + EVENTHANDLER_REGISTER(shutdown_final, shutdown_final, NULL, + SHUTDOWN_PRI_LAST); return (0); } diff --git a/sys/x86/include/xen/xen-os.h b/sys/x86/include/xen/xen-os.h index 9d0964c6631d..655961da22fd 100644 --- a/sys/x86/include/xen/xen-os.h +++ b/sys/x86/include/xen/xen-os.h @@ -56,13 +56,6 @@ xen_has_percpu_evtchn(void) return (!xen_hvm_domain() || xen_vector_callback_enabled); } -static inline bool -xen_pv_shutdown_handler(void) -{ - - return (xen_pv_domain()); -} - static inline bool xen_pv_disks_disabled(void) { From nobody Mon Jan 10 21:41:07 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5433E194F76D for ; Mon, 10 Jan 2022 21:41:19 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXnMM0ZfLz4bxK for ; Mon, 10 Jan 2022 21:41:18 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 20ALf7wQ016474 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 10 Jan 2022 13:41:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 20ALf7vD026926; Mon, 10 Jan 2022 13:41:07 -0800 (PST) Message-Id: <202201102141.20ALf7vD026926@nfbcal.org> From: Brian Buhrow Date: Mon, 10 Jan 2022 13:41:07 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Re: Missing MAC addresses for domu VM's Cc: , buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Mon, 10 Jan 2022 13:41:08 -0800 (PST) X-Rspamd-Queue-Id: 4JXnMM0ZfLz4bxK X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org hello Roger. Thanks for the reply on this topic. While you're correct that a user could set a MAC address on multiple VM's so they clash with each other using this incremental technique, it's already the case that with the current methodology on FreeBSD, all the the dom0 side interfaces for the VM's have a zero MAC address anyway, so if we're worried about dom0 side MAC address collisions, I don't think that's an issue. As to whether a user could accidentally set a MAC address on a domu to match the MAC address on the dom0 and thereby cause trouble, I think the solution there is to document how the backend code behaves and to warn people not to increment that octet of the MAC address. It's already the case that one can accidentally duplicate MAC addresses between domU's and it's already documented that this is a bad idea. So, as someone who's familiar with how networks work, I don't see any down sides to making this change and I do see a lot of upsides. -thanks -Brian From nobody Tue Jan 11 09:50:53 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0AB301937741 for ; Tue, 11 Jan 2022 09:50:57 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY5YD6wkdz4fRP; Tue, 11 Jan 2022 09:50:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641894657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=m6tTp/wXIvYrqx92MTPhx0EtacfGJGyNz1a1XL+m7ZM=; b=wc6LOh2Q59b9RkNN3GOgPTFKEv6E5UnZ1PJVzMgrVzsorcq1ob55313MqA72xaGRuWF3f2 SPykpcnF/3yZfJeFm3dwsoiZIJl1t/gHcRoYJtr5ljWDZ8DOSIhhfHyXQWDhZ0XHzn7Ff5 U1StZceXKHvcDQMB5N/fmEhWCHLJ1N2QXUGD+V96YhgwcN4fp/W7+ZGmIRh35Kzub/WrQ5 ZvciyX3wDlsOvS7gilhNWDMc57JANVNOpVPn60iDZEFgk6821W8fGAM44XHB13c0EX3ktj BDLTSQgg4i1JPcl6A14y3QSKCCNkV7bFjIe2lPvHV6581lYC+2enVMA175OOug== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 75B75282B; Tue, 11 Jan 2022 09:50:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> Date: Tue, 11 Jan 2022 11:50:53 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 Content-Language: en-US To: xen@freebsd.org Cc: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= From: Andriy Gapon Subject: VCPUOP_send_nmi returns -38 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641894657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=m6tTp/wXIvYrqx92MTPhx0EtacfGJGyNz1a1XL+m7ZM=; b=bweDv+1DOvS4EU8K+GGOcxNa7LKv7nmyuyML7tNxxQE7TMUzOQDUqVv+euPqWePo1xqszM aijwQdef6V99kquPopibhexyou+LXqicJDbSzTqnJmUqApYUKdOYPPDX+G3j0Mq1VMaHlH 4JB2qdWVy0qkzXkVjjXXEm2jrUIwgcni1a5h2BX06Uhq6IDQqE+B571fzkXGIZLKZ7/xWp vYbuBmfCn519zVGgIN4+Lj22en3UrAST9hR15d20X1QbQRk8Tw2Z2K0L7QoadkmMhQv3aT 61vVw1hcY+srpbic9kaHh8k7qHAq7EcaAhBjgpH91vy/0Shra5diL6iDaU9S2Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641894657; a=rsa-sha256; cv=none; b=lOUnuHln7efuwrVNVyLn+iEHmgJCXoMgZ9/4+wNezwxX88qCffl+BNvW/zoa9yKAP3QpSM YhdUh56Yt6uiGxoCcl53QMQMfLZOCJZe2xofLiWnefLBF/ze/8oveet63HM0rT9g65ZQPQ B9tSDi8PjOx2WEHkUckSyejyKq/1Cr6aGIw5XZvmZRAF7NVmrzC+Dn6shr6m0A3FDOlhyU LOHUFYyaSzEOtsLUuhT9BOmPpaTh/B0cALVaZ0BSQdYF2Z+ReYzvTqrBiZEusXbZYXhSsl grqeimAp9ratDnjaE0yfOjScSPKBM25NV2z+1sW5yV/zLzsteX9AGYsbrWWYAA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N Recently I got a report of crashes related to using procstat -k on one of our systems. The system runs FreeBSD 12.2 on an AWS Xen-based instance (can get more specifics about it later). It immediately reminded me of an older issue (Subject: Xen (HVM) and NMI) where the root cause was that NMIs were delivered as regular interrupts. But 12.2 has the newer code that delivers NMIs as NMIs. After some investigation it became evident that NMIs are not delivered at all. I modified send_nmi() in sys/x86/xen/xen_apic.c to capture and report errors from HYPERVISOR_vcpu_op(VCPUOP_send_nmi) calls. That revealed that the call returns -38 which appears to mean ENOSYS. I am not sure what that could mean. Perhaps NMI is somehow disabled in the Xen configuration (for that specific instance type)? I am out of better ideas. P.S. It appears that FreeBSD does not expect that an IPI, including NMI, can fail. So, there is no way to propagate the error to callers. I think that we could either printf it or, perhaps, even panic on such a failure. -- Andriy Gapon From nobody Tue Jan 11 10:07:28 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CCAA3193D301 for ; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY5wL5Qqqz4k2X; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641895650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKlvkHgVLxDqQouW77Ct9xqcuNrG693kGpZ1R+9cHns=; b=p34C0u9ZlfIua+MCY/4jpbdFdttgNk8evIKga6HEYW3m3UbuiRX+IAzd48inMUx87DHD79 /mK9dLldHK0/JqijDnI8FLEnj5NpgbHCwAHdbjS0d5QR37ZPx1F88qOREgZTrZd7bjC4JU aV2taMs6XqIDztkQCXPGGmWLBQs6N64Bl2kpfHhKA8l7GD5ZycAe+MABqHAKR2KmHWVBhX lHYmIdWnKWzJuiRrF9NgSNlIdYhjTzbTtEbGeFVHS70RVthU/T7DjxWcy+/4lz32BHShD6 ojwA9Z/uC+SmsChoG0q+JsTotT/ryBd33J3DWN96wrB0kfqQW9ld2BBH3RE3vg== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 3F1D23279; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: Date: Tue, 11 Jan 2022 12:07:28 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 Subject: Re: VCPUOP_send_nmi returns -38 Content-Language: en-US From: Andriy Gapon To: xen@freebsd.org Cc: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> In-Reply-To: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641895650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKlvkHgVLxDqQouW77Ct9xqcuNrG693kGpZ1R+9cHns=; b=RAiVKwJvY/IjwIpt3wKBc1QnDUdm0vDDqytJah48N82emIppPRHg/ET03r3hE9zczQAsMv Wvp81fye/hI077Zwm3Uz15IY04cLaG0VHjvGXGA8xlZquoTgK5HUo8xuZtwf5OcwLe2g92 FZt2Y5tvoaaCA/nLEWgmPXL0cg/2cGHzwzQI29cIu73+JSZGwfDYJi2a8txhemp9za4ho9 oR+FZlyVfpBzoJ2vKK2URl8fGt7AVV0hmeFRDYlk/qmFlDm1J2cNnYWNuwHIv7Mie2gmp4 kzjLeesQspanvZWR2/78nauntq3kxEhkNuJ2lttwsWUT91FeedVQZJZGiEf3cA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641895650; a=rsa-sha256; cv=none; b=Z4qHQhsbrd8YWqh/HIc2ieIa3rv52voKyn9zPVrW8r6jk3t60Bo5v/gz/e9rpwxisuEaeV FhwMUklGubfnbhNGWIMrRrV60N9IkHuWOnPG9bGBRnkEBS73OOWcjryTnJwk/TXkaWHcc7 csWhyJC8YZzIVzrudH8Hl3dDyE7qU0LIzMgmtt1sK/LfjZayRTxbeGVsstM0kG5PT9+6GP KScYLyJWhBljwYXcY5FwQ1qP/hPWpc1XFO1eNaDkXWlfc2LFCxAxjqRaT2lcpvzjNcibeR vi6XJxVHsX0b+BwkpUicK43i4he1rIBqFrQhXXkQ79gV4vAIZUKpZap6tP7b9g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 11/01/2022 11:50, Andriy Gapon wrote: > > Recently I got a report of crashes related to using procstat -k on one of our > systems.  The system runs FreeBSD 12.2 on an AWS Xen-based instance (can get > more specifics about it later). The instance type is t2.large. Here are all lines from verbose boot that mention Xen: XEN: Hypervisor version 4.2 detected. Disabling MSI-X interrupt migration due to Xen hypervisor bug. XEN: disabling emulated disks XEN: disabling emulated nics Hypervisor: Origin = "XenVMMXenVMM" x2APIC available but disabled due to running under XEN ACPI APIC Table: Xen interrupts: unable to register PIRQ EOI map Xen interrupt system initialized ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen ) ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen HVM 00000000 HVML 00000000) ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen HVM 00000000 HVML 00000000) ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen HVM 00000000 INTL 20090123) ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen HVM 00000000 HVML 00000000) ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen HVM 00000000 HVML 00000000) ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen HVM 00000000 HVML 00000000) ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen HVM 00000000 INTL 20090123) ACPI: SSDT 0x00000000FC00E260 000033 (v02 Xen HVM 00000000 INTL 20090123) acpi0: on motherboard xenpci0: port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0 xenpv0: on motherboard granttable0: on xenpv0 xen_et0: on xenpv0 Event timer "XENTIMER" frequency 1000000000 Hz quality 950 Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 xen_et0: registered as a time-of-day clock, resolution 0.000001s xenstore0: on xenpv0 xsd_dev0: on xenpv0 evtchn0: on xenpv0 privcmd0: on xenpv0 gntdev0: on xenpv0 debug0: on xenpv0 xenballoon0: on xenstore0 on xenstore0 xs_dev0: on xenstore0 xenbusb_front0: on xenstore0 xn0: at device/vif/0 on xenbusb_front0 xenbusb_back0: on xenstore0 xbd0: 40960MB at device/vbd/768 on xenbusb_front0 xbd6: 141312MB at device/vbd/51808 on xenbusb_front0 xbd1: 204800MB at device/vbd/51728 on xenbusb_front0 xbd2: 204800MB at device/vbd/51744 on xenbusb_front0 xen_et0: providing initial system time > It immediately reminded me of an older issue (Subject: Xen (HVM) and NMI) where > the root cause was that NMIs were delivered as regular interrupts. > But 12.2 has the newer code that delivers NMIs as NMIs. > > After some investigation it became evident that NMIs are not delivered at all. > I modified send_nmi() in sys/x86/xen/xen_apic.c to capture and report errors > from HYPERVISOR_vcpu_op(VCPUOP_send_nmi) calls. > That revealed that the call returns -38 which appears to mean ENOSYS. > > I am not sure what that could mean. > Perhaps NMI is somehow disabled in the Xen configuration (for that specific > instance type)? > I am out of better ideas. > > P.S. > It appears that FreeBSD does not expect that an IPI, including NMI, can fail. > So, there is no way to propagate the error to callers. > I think that we could either printf it or, perhaps, even panic on such a failure. > -- Andriy Gapon From nobody Tue Jan 11 10:18:41 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CE79C194150D for ; Tue, 11 Jan 2022 10:18:58 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY69Z2Zk7z4n6r; Tue, 11 Jan 2022 10:18:58 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1641896338; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=5c+8ZG9NgZymF9xLsr0x8Ts0lWCWEvo7fWE0UdUdf9E=; b=aVt9xCPEd6N/l1Qpt0VoyKrR7S5FbYeZflniKbGLs2I+jQ1uPF3RsjCt XSPANepDJ8DjCoIplWEz6BATDAlbAWqgdSpO/Oqc3XptG/4uai6IGqwu2 vEDlUSdep6TtIiqtI9rdwFpysK7whsMwlKumRUBsGKIdlANyozX6ullPa c=; IronPort-SDR: 3UVNYqMg6fj8/I6CVVf1lyT/MepLAHpgYIqQI/K1xYitjA+V4gfPYMhcbOigq+r+PWfItUjlun OwKQPKfM7siUW5vbScx37WcjCEEZUwvWDi+dzzg7QnLDOS8+0u1eGlQ/8FruMoETzqTibooQM7 lbvYpY0zMcaqJUUiBO80uQgTe60Rjc4vXYvs76Lb/tQHrEGhpTwqyJ3XCUsttL7du1jKLnRbfW Dw7gexJJMQfXCp8laxwVYAB8YTeY4s7C3nJ4x1WGHVy/LkRavykvVMT4TcfAe5q87iHk0BOn4H Ze0EwcACMPqBaen9myoZstek X-SBRS: 5.2 X-MesageID: 61646110 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:KhYmJ6DgwsvWgRVW/+Pkw5YqxClBgxIJ4kV8jS/XYbTApGlx0GZTz mtOW2/VOP7YYzf0ftlwaoi1p0NV6JHVm4IxQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA xA2M4GYRCwMo/u1Si6FatANl1ElvU2zbue6WL6s1hxZH1c+En940Es7wobVv6Yz6TSHK1LV0 T/Ni5W31G+Ng1aY5UpNtspvADs21BjDkGtwUm4WPJinj3eH/5UhN7oNJLnZEpfNatI88thW5 Qr05OrREmvxp3/BAz4++1rxWhVirrX6ZWBihpfKMkQLb9crSiEai84G2PQghUh/tTWDrchcl ZJxurOIUggpPPGTovkYXEwNe81+FfUuFL7vJHG+tYqYzlHccmuqyPJrZK00FdRGoKAtWzgIr KFGbmBWBvyAr7veLLaTUO5ji95lNMD2FIgepmth3XfSCvNOrZXrHfyUuYEBhW9YasZmP/vSW uwbUxRUNC/RRwNFanI2Os4ztbL97pX4W2IB8w/EzUYt2EDDwQpZ3qLgPdDOdpqNX8o9tk+Gq UrI5Wn0CAoWctuFxlK4HmmE37GV23mhAcRLSeP+pqUCbECvKnI7Nh0cZ0OW/L6FhWGGQN1fK RBLwDcrov1nnKC0deXVUxq9qX+CmxcTXdtMDuE3gD2wJrroDxWxXTZdEGMYADAynIpvHGFxi AfV9z/8LWE36NWopWShGqB4RN9YEQwcNic8aCANVmPpCPGz8dho3nojojuOeZNZb+EZ+xmtm 1hmTwBk3t3/aPLnMY3hpjgrZBr29/D0ovYdvFm/Y45f0it3ZZS+e6uj4kXB4PBLIe6xFwfd5 ShYw5HPtLtTVflhcRBhps1XR9lFAN7fYVXhbaNHRcF9p1xBBVb+FWyv3N2ODBgwaZtVEdMYS ETSpRlQ9Pdu0IiCNsdKj3aKI51yl8DITI29PtiNN4YmSsUvKGevoX8/DWbNjzGFuBV9yskXZ MbEGftA+F5HU8xP1iStfe4B3NcDn2ZmrY8lbcqln03PPHv3TCP9dIrpx3PVP79pt/3V8VyFm zudXuPToyhivCTFSnC/2aYYLEwQLGh9ApbzqsdNcfWELBYgE2YkY8I9C5t4E2C8t6gKxOrO4 F+nXUpUlAj2iXHdcF3YYXF/crL/G514qCtjbyArOF+p3VklYJqus/hDJ8dmI+F/+bwx1+NwQ tkEZ96EXqZFRAPY9mlPdpL6toFjKkim3FrcIyq/bTEjVJd8XAiVqMT8dw7i+XBWXCq6vMczu ZO60QbfTcZRTghuFp+OOvmu00mwrT4Wn+crBxnEJdxaeUPN9ol2KnOu0q9rcp9UcRian2mUz QebBxsctNLhmY5t/Ymbn72AoqeoD/B6QhhQEV7E4OvkLiLd5Gein9NNCb7aYTDHWWro06y+f uEJnerkOfgKkVsW4YpxF7FnkfA369f1/uIIyw1lGDPAbkixC6MmKX6DhJEduqpIz75fmA23R kPQpYUKZeTXYJvoQAwLOQ4oTuWfzvVFyDDd4MM8LFj++CIqrqGMVl9fPkXUhSFQRFev3FjJH Qv1VBYq1jGC IronPort-HdrOrdr: A9a23:R1/ia6EmyBU3eyOHpLqFDJHXdLJyesId70hD6qkvc3Jom52j+P xGws526faVslYssHFJo6HnBEClewKgyXcV2/hqAV7GZmjbUQSTXeRfBOfZslnd8mjFh5JgPM RbAtlD4b/LfCBHZK/BiWHSebtQo6jkzEnrv5ak854Ed3AVV0gK1XYBNu/0KDwQeOEQbqBJa6 Z0q/A37AaISDAyVICWF3MFV+/Mq5nik4/nWwcPA1oC5BOVhT2lxbbmG1zAty1uHg9n8PMHyy zoggb57qKsv7WSzQLd7Xba69BzlMH6wtVOKcSQgow+KynqiCyveIN9Mofy9gwdkaWK0hIHgd PMqxAvM4Ba7G7QRHi8pV/X1wzpwF8Vmgnf4G7dpUGmjd3yRTo8BcYEr5leaAHl500pu8w5+L 5X3kqC3qAnQy/orWDY3ZzlRhtqnk27rT4JiugIlUFSVoMYdft4sZEfxkVIC50NdRiKp7zPKN MeTP002cwmMm9zNxvizytSKZ2XLzgO9y69Mwk/Upf/6UkSoJh7p3Fos/D30E1wsK7VcKM0lN gsBJ4Y4I2mfvVmHZ6VO91xM/dfKla9CC4kY1jiaWgOKsk8SgfwQtjMkfII2N0= X-IronPort-AV: E=Sophos;i="5.88,279,1635220800"; d="scan'208";a="61646110" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CSmJQQZr13S+e9/AnS4lp8pCKAMmqhl2fJtwe4zJhp11OlDBfyToEy7KoQ+1wUhQzMVpwZutCOF5O63AQRjOCgtr8Hb/en8g9yFkXalCERcm+skNByS8/zK2pZCTcDnhr2f9NCj/8qbefRFaoIoJU//EpTT8hlc7hU5pOflDrOmqeOfta6IZ9oNqf34m05ndCbQ9nD2VY8HqIvWNd9YWgA3rhnkmEn+h/PabjNm7iOcmnnfDsv3cuoEP+jwtvczM6PgG4BEaUHlWd76NxiPNwwOv6xCasa6ZvsCktcmvqBLWRIuGaTiNuO/Rm5CYNtgAm7oKNFwdfAmZkC4TxsSC+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9uK0eYXxJkfFEEzso+oAWEhPTOkqx3YrVD2ErWNxQGw=; b=Uy4ZEOQG0s3fihErW6LkztH9NMrK4r8PfzXn9p80fcHRfpT93mqFbGfQftBwEsJL0cH47Xm7pC/3cY64RD65xHVZURx5odCKNJvzhYDCmo+tb1iBlXLaSvd0LerWowP71Y769yLyYDREy26YkqSrdldbmX6NikYiFTgKRXfEVKBnqKQz1++2M61RL1OUNneotJihQ/6aqPOxbzyRR55SmjEtB4F24TYOs5oB58iYYFedqXtIS3OlB6OVz4HiSA0YK8XqsbsBEFHTnok0aZ24ldJGlf5BM78w6XAUUQH+1tCFdf2dhzrYvx2laSLxE6d+irgmEpalR8pDptu+i5U2SQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9uK0eYXxJkfFEEzso+oAWEhPTOkqx3YrVD2ErWNxQGw=; b=Ymd5cQMrimcTut0qn2oLJ13TekSze3tPAjl42Z6ICe2PPRUwASJ+9TAtE5MyJrexRVVNAqO+7MgFicp2k05uqJOLKmedqYpMv3KRtlSXNqng6kB8P6+zrTvv2ZEuIpNQDbQui1ZaoZp2FWnwWKsA0JAgA2FkJQejlx6AHyOX4nY= Date: Tue, 11 Jan 2022 11:18:41 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Andriy Gapon CC: Subject: Re: VCPUOP_send_nmi returns -38 Message-ID: References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> X-ClientProxiedBy: MR1P264CA0056.FRAP264.PROD.OUTLOOK.COM (2603:10a6:501:3e::20) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a40dd13e-4534-434e-ef82-08d9d4ebc0cb X-MS-TrafficTypeDiagnostic: DM5PR03MB3291:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7OAbh3fVbyCOv7zKHSfyhaykNbboAuFvE+7PV+HbseBf/AKeVQLi9epWBlGIVC1mxCrtZAh75+X4GfIBnpmYx6krtMR3oiKzQXOEdZqcfni/QHy2S43lZ8f4s511Odir3gGoApq3mFxTYOVAVUS948risubb8YAHZvbXvq8PSltWQ6W3PIlANeuxse1wzYx4VN3WoMislxlkGJ+66ZV1rOm+xZ1k3KGsNBlUuEqlS1/DVDloAll2RP91S3hoymH1J+np3jPKl6r0gJSBaMKp0L41pj0ERJ/E1+5WxMy63zWqHUS8aIVbv0Makddq9NDbMd3+p8XVt/W/S57QcNCcAcbMqO5PXudvQyIVyiS5t+Hz+3zI3U8vkU2ZnHTPgtcG6cOiWFxl51oxMRj4LeRXqgRMgLYwVZ4212g4c+Qy92uiTk72IeW4eSfH2fOF5xveky95MuAWmz6v3c8bzDYK4dvxhiy9iW4Z/tQtZHPL5UZJZi43klJE1Vb5RAlM5Ts5PFr50khefvb/0SpDYcFyhHCzEf+oqHClwZdlYWlvYjrN0tFU6e++l0RWzwBpMu0aoaCE4nhuOXpiSsGM0BIfOpPAbT0t5vMeVaaSkFEhdKitg7H0gDedrgbkvzVgTgYn0SAsk/aGj9MHWyrVHpRa3Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(2906002)(38100700002)(8676002)(86362001)(66946007)(6666004)(8936002)(7116003)(26005)(5660300002)(4326008)(83380400001)(85182001)(9686003)(316002)(6512007)(33716001)(82960400001)(66556008)(450100002)(66476007)(186003)(6486002)(6506007)(6916009)(508600001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?alJWS2hOYmQ2ZlJ2ZmlzS2lQcFdFOVc3RTBwNzFrN2lWU1ZDYWVnWGdOZUxB?= =?utf-8?B?dkd6b29qS2RPSTYwemNUaGNlZXdvVm5oTVFSMHJXbGljaS9EZEFsMTZCS3l2?= =?utf-8?B?bysxZGxXSm1DVmtYTGZHU1VEZzJSUGhpdStOdTU2YTdrdDY5ZW5JRUg3czMy?= =?utf-8?B?b3dJMUp3dENDU0l6aml6c2VsTUJnVnFkSGRFU21wOHZwZmYzeGk2ZGV2NHQ1?= =?utf-8?B?NE9ON0ZtL2I3K0hzbjc4UnQzMXdEVlNOK3EvSndPOUlBQndMYjV3R2xWa1ZI?= =?utf-8?B?bXBIbHFoaGF1TXU1d3hDTXRpdWdqQzZvMzc4U1RkZHpiS0o4WmQ3UlZvdVpO?= =?utf-8?B?YXVGKzZYTVlFa3loNHRLeWFLNXhQb243QU52Z3RHVGlGSk4rZTMrYlppbVJH?= =?utf-8?B?cExCZ3ZRcWpmY2czK21TR2QxMEVIWGdQNjJOVHlURE9vUlJSS3Z3THVpMXZv?= =?utf-8?B?amg1WHBPbHBkNkxEWUNUNzBVY3NRNHQ3VlZXN1hTMHlVVW5DSTBQK1ZGY0pV?= =?utf-8?B?U2M4ZWtuM0VKK2hCRWZrU3JWUVRRVWJaUkxwVkZ4RUgwS0p5czhLcGJyR25P?= =?utf-8?B?c0hPbVpZTzNjbzM4RE9Ubm44YSs2aFpTTk1kVmpacEJjWXhCZzhGcUZmcU05?= =?utf-8?B?VzVMMWNVZXhMMFhqODQwYVBGamI1Nmp1aWE5VnlYYU9uTVdOeitpdWtjVnNx?= =?utf-8?B?MnpYQnlzMzVjVUxLUklJOUx5SlIxY1gwVXlOYkZZSnZuM0MwTWxsaFFia3BB?= =?utf-8?B?VDhCb2VoeFAvd2tYZ0VTRWJrTmlLZmovYnVndVRTS1kzaWhIckVTbzlVeXNa?= =?utf-8?B?STE2a1Q5TzAzSXlQSFRJMktNZ2NXL2lMWThuUnQ4MWIxVlR0THdDWUthM0pv?= =?utf-8?B?KzVHNHRySnJDSTIvT1FrQ0U3N1FWZ0cwTmRUTTNPSkV1STZKeXFLNVlscHVM?= =?utf-8?B?TEV4ZTlYSFU3RHRZR3Vra1hxbEFSek9OdWc3bS96dWZtcFBRNUpLR0pFQ1pB?= =?utf-8?B?WHJjWEloRHdUY1NSckNZVzRFd2NVSTBpTFF1S2RkNWp1MGh2TWx0ME9hQWdw?= =?utf-8?B?a21IcTVyRVBVd1ZhZUNXOGNCSWZVUmNyR3lEaXBUOXEzMENmQzFKNk4wWFhG?= =?utf-8?B?R1RrcHM1azlnK2h4aks5cXEwMitFdDY3b1pEY2srS2FoZ05BdkNSQWdwS0Ju?= =?utf-8?B?UWFiRTJvL2tBY3FIcFBmSDlKMmpDV0lqMll5bjVReW5lUGdhZ1k3YitneGNs?= =?utf-8?B?cmM3Nmkva0JnR3pLZzRrYlZ6SXd5WlVUMVJsVUxYZCtadEF5aEVFL05ORkU1?= =?utf-8?B?VDhuRlR3MUFaWmhYcTlWS1N1UGh6K1JoK3lac0VUUG5aejJidGQxb3phYW8w?= =?utf-8?B?YzJJOU9PSWRpbkdRazI4dUQxQnRFQU1OSmpDTy9oSDVLN3hocUZBY3I5NE5J?= =?utf-8?B?Sk5zQm5KTWpUVkcrdTRVYVlYMjliNDNGcXZmRlNPay9adXVRTVhsL0lNUHNJ?= =?utf-8?B?RlZ2K0NGNHhlVHB3bkVGaXVvZWQvb3BUbFZkaEgxMCtKRElJV2ZtYmsrTkYz?= =?utf-8?B?YnhlQkZ2S2VMN2cySzgvT3ZGbm1UNFdNN0xWZ2tKTjdQMFJsN3FLT25xdzhB?= =?utf-8?B?cENLcUxpZ3ptRnhFWndQbmd2NjdCRnpJUE5mUzI2VFppZTZpd1BnT3RvSGxv?= =?utf-8?B?SzF5VzFsQVF0TDhrcmVsTzBrL29sazQ1T3YwU2lVZzJEVU5pb2RYc0FXZ2xN?= =?utf-8?B?cVdPQnVLMjRrV0QycUk2QlZrN3BzaHZKdGd4VC9ib3JRc2pha0s3Ymg3NlRN?= =?utf-8?B?YXR0My9ENjVnQ3d4bzAzL2FvdGRNNzA4cjdVMjA4T2NqVmQ2V2RobnlpNzVP?= =?utf-8?B?dDNZOUo5S1pGRWtWcEgzTnBsT3lKcFJtQXJDUk9TSWF4UnlTMTg1WU1QNVZt?= =?utf-8?B?b2VCeGRFSlY1aFppTTJhZzRnZWxLZ0o0K2VGbGNGUlR5K3d5eTNQd2ZVVnVt?= =?utf-8?B?ajRZM0MvSDYxSCtNZTRZQ1VlZW9namxCTE1wOTJuQkw2TkNXYnpXaVRDMmQw?= =?utf-8?B?YVk0WDFPMlNSM21VZXMxY0ZhcmdmeGpDVUdRYysza1N2QUdNNzExeTRHOW1X?= =?utf-8?B?NnBCNTlBNkJrMW83eCtzUjBwL0tsUmRPd0pBeTBKb1cvdER5S0JLWFNwcTQw?= =?utf-8?Q?1FfKWUAL9PnpVAvFr5HzhkM=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: a40dd13e-4534-434e-ef82-08d9d4ebc0cb X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2022 10:18:46.8203 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OY/y/ZxZZPWkfOQzv5ZXzUeoOjs8xPQL8f+ZaoQnpnkxLpPQeQ0v9x5XkZztosvQ3a+udBmeElv047lOKZvXaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB3291 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JY69Z2Zk7z4n6r X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Tue, Jan 11, 2022 at 11:50:53AM +0200, Andriy Gapon wrote: > > Recently I got a report of crashes related to using procstat -k on one of > our systems. The system runs FreeBSD 12.2 on an AWS Xen-based instance (can > get more specifics about it later). > > It immediately reminded me of an older issue (Subject: Xen (HVM) and NMI) > where the root cause was that NMIs were delivered as regular interrupts. > But 12.2 has the newer code that delivers NMIs as NMIs. > > After some investigation it became evident that NMIs are not delivered at all. > I modified send_nmi() in sys/x86/xen/xen_apic.c to capture and report errors > from HYPERVISOR_vcpu_op(VCPUOP_send_nmi) calls. > That revealed that the call returns -38 which appears to mean ENOSYS. > > I am not sure what that could mean. > Perhaps NMI is somehow disabled in the Xen configuration (for that specific > instance type)? It's not something that you can easily disable at build time, but it's possible AWS has done some modifications to the hypervisor in order to disable it. Do you know which instance type and hypervisor version it's running on? Thanks, Roger. From nobody Tue Jan 11 10:26:15 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 235141944A86 for ; Tue, 11 Jan 2022 10:26:29 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY6LD4h1Yz4pyX; Tue, 11 Jan 2022 10:26:28 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1641896788; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=IHUuK68FbOsY9rkN8OwO9HWFfxNH6D43zG284iGGYpc=; b=KgOl1tYBRQ54PVCBtmQgLIpQ/yiFYKq/P9ibOmfoYUGDqBwatFS1yp4v 9EneMQYArM5Ku2KrVe5QKo2AJYZax0y4+XeBAN4hzRbs+fx/8zlUW2xmc 8jTpKVNRCyzOgwv6/IKzNoRa+uur/qKMuEjix++UUek/BAz5E6OacVMXV g=; IronPort-SDR: Ysg/hVoP8NJq2vcbx+aQEkLofpWvE8G9Ifn438uboDifeUUHfl8RthFl+TE+IRvH+3Ax82JJNN f9gNAe99+uRL0xyGhck1Q1UUqmAdmYeNWYr50ZXhCKhnbjdjzDK0cMLZ6ri+gx0d60NCx1sUkq gB3krNA0FG2o+0e8MOr+5ENhaDT+uPs9PRcMk6wAyow7geeaUuwtHPoZcXTHc/HtVixZMHhRWr 0UA59qMXVa/9FxVMEH+HSv9wdSAlg6ybvj7GAKxv5e1e09OM2Bq4cyVjtxdNkkQo0kw3NNREF8 0VQVM8d4GoZPN2K3H+DFaUKI X-SBRS: 5.2 X-MesageID: 62128562 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:tGk2LqyxOGxfT2WPYaV6t+f0wSrEfRIJ4+MujC+fZmUNrF6WrkVUm mUXCmCAMvaCa2vxc991Oo/n8UkC7JWBzIUxSFZo/yAxQypGp/SeCIXCJC8cHc8zwu4v7q5Dx 59DAjUVBJlsFhcwnvopW1TYhSEUOZugH9IQM8aZfHAhLeNYYH1500g7wrdg2tcAbeWRWGthh /uj+6UzB3f9s9JEGjp8B3Wr8U4HUFza4Vv0j3RmDRx5lAa2e0o9VfrzEZqZPXrgKrS4K8bhL wr1IBNVyUuCl/slIovNfr8W6STmSJaKVeSFoiI+t6RPHnGuD8H9u0o2HKN0VKtZt9mGt4BD+ ulih9+uclwkZpeQyPQxDxkISwgraMWq+JefSZS+mcmazkmAeHrw2fR+SkoxOOX0+M4uXzsIr 6ZBbmlQMFbT3Ipaw5riIgVort4kI8TxepsWp1lrzC3DDOZgSpfGK0nPzYEChW1g254UdRrYT 5M8SyZ3QzThWEVoOXwyC65mvuKmq0CqJlW0r3rK/PFqsgA/1jdZwLXrGNTPd9GAXskTmVyXz krP4WvRDAsXOdaExXyC6H3EuwPUtXqlAsRITuT+r6M0xg3IroAONPEIfQuJk8WWuA2gYIlSc 3ZIqjoC8ahirGX+G7ERQCaEiHKDuxcdXf9ZHOs79ByBx8Lo3uqJOoQXZmUfMYJ77afaURRvj wbUxI2xWVSDpZXIESr1y1uCkd+l1cH5x0cmbDRMcwYK6sKLTGob3kOWFYYL/EJYY7TI9dDML 9Ki8XBWa1Y715djO0CHEbbv2WnESn/hFF9d2+kvdjj5hj6Vnab8D2BS1XDV7OxbMKGSRUSbs X4PlqC2tb5SVMHRzHXcGL5QRNlFAspp1hWG2jaD+LF7plyQF4OLJ9gMsFmS2m80Wir7RdMZS BCK4l4AjHOiFHCrcbV2c+qM5zcClsDd+SDefqmMNLJmO8EpHCfepX0GTRPOgwjFzRZ9+Ylia cbzWZv9Vh4yVPU4pAdass9Ai9fHMAhkmzOKLX06pjz6uYejiIm9EuZaYADQP7FgvMtpYmz9q r5iCidD8D0GOMXWaSjL648Da1cMKHkwH5ftrMJLMOWEJ2Jb9KsJUZc9GJstJN5ombp7jODN8 i3vU0NU0gOn13bGNR+LejZob7a2BcRzqncyPCoNO1e02id8PdbzvflHL5ZnL6M68OFDzOJvS 6VXccu3HfkSGC/M/C4QbMehodU6Jgirnw+HIwGsfCM7I8x7XwXM99K9Jlnv+SACAzCZr8w7p 7H8hArXTYBaH1ZpDdrMaeLpxFS05CBPlOV3VkrOA99SZESzr9Q6d32v1qc6epheJw/Cyz2W0 xetLS0Z/eSd8ZUo9NTphLyfq9v7GeVJAUcHTXLQ6qy7NHeG8zP7k5NASuuBYRvUSHjwpPe5f exQwvzxbK8HkVJNv9YuGrpn1/tjtd7mprscxQV4BnTbKV+sD+o4cHWB2MBOsIxLx6NY5lTqC h7epIECNOXbIt7hHX4QOBEhP7aK2vwjkzXP6eg4fRfh7yht8bvbCUhfMnFgUsCGwGeZ5G/9/ dochQ== IronPort-HdrOrdr: A9a23:ej1Y160qp5VVTQ5W/4OdvwqjBLYkLtp133Aq2lEZdPUzSL3+qy nOpoV+6faQsl0ssR4b9exoVJPufZq+z/5ICOsqU4tKNTOO0AHEEGgI1+rf6gylNyri9vNMkY dMGpIObeEY1GIK7voSNjPIceod/A== X-IronPort-AV: E=Sophos;i="5.88,279,1635220800"; d="scan'208";a="62128562" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QIqXB0CnLaU8DSw6bC+hVL0fpM2Ho0gP7b9O7BADkUduB081QppvCnlRKt2AipI8sO6nM7zkGMWoQQtQzzL5HtKmhqjr0WTR2Yct6Z6xaPAE6su/NHIXtt5a/MGIQcynUf/MxOpwTi7kK0oE3TW+0CgaVXF4chHCvHe6v+7zLbt9VTuLClG2xPFiYPRdkcw7bGEz3TKO6OYG4iItbhFE++xjdadBfGbge3svwaYdfx1L7pY9RmL16hPRvrIwNT4w8AQ9nN9yv3gQr7HtPoGdGpxDToGILJkZx81Cho1TgjYV9Uzh42GsKj57kuq8/y8FsWrfBRMBTlR1Gx3P8AXOWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+gmPZ38Lo8YIp0zfonpS86wKFP1+jZ96fJAdKIudfw0=; b=R7f6hkDEc+G3i/hoW1cLlDXAt/LK63Gw9VYQQUu8o1SKEzPETZ3HXS+8cCEvCW1VtJQZ49MYXwy7JdMiwldLWA6m0XvIjecRGR419x0eePuU6NSMKANLQohpBgOLKdIOpAdXpg5EpCS2LBQ9sTFYathw/mwzQGo3Qj32I1Z8yZmhVYSoZsix1JVeK0fsNDkv4fJxhcgw/Hhtbp56TJWg1PnDBxnGXStraUbkT9jMTTW59fUftFSykNyha9T91qoerU+rghLqvXclemKeHAxG87HbLLdgN2J7MZkk1LCLk3sz+3kZ9COw1T+eRpBNBFdsq5uzHwxNtomSRe018o8cMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+gmPZ38Lo8YIp0zfonpS86wKFP1+jZ96fJAdKIudfw0=; b=gMg/zrkI6Rx+zFsh5fsmh5Ve+kLFZXlYzBUmZRNzD0xHrY4fSrDdY0pKv5gUe53ARviU8WsGLHk4OaJvzp6+h/DYHv87638/AJjum57QtcOKcheu37pILASgB3l6BLgUD+zN4wbmSfSF2aomKYZ5eYfM7yYcQhnNlpwLFmoWyTQ= Date: Tue, 11 Jan 2022 11:26:15 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Andriy Gapon CC: Subject: Re: VCPUOP_send_nmi returns -38 Message-ID: References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: FR0P281CA0078.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1e::17) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6e6c1f6a-a795-4d73-35f3-08d9d4ecce75 X-MS-TrafficTypeDiagnostic: DM6PR03MB4138:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: k8zdygrNyxzX4ufnWFclZzqVpYUsGogNdiggB4+G0uDslya8+9YEFedYwmcqzFudKiLUBzd3CYUUPUPlrNq18oJ7mAYxkt2oivN9/pOWGyrD4QfKsFOenenBbDRrGzQb3zO5owrTzKEJrNGM9rnzjTErWRZYcx1RAuT0roHrY51oFfOCQZ8DxGhMH33uf9BQ/ocOcefBpQ8kMz6+OLsl0h/rZfws4I0SgGhVb0kFbHYmsDhHO/UunPqPOjocbMeUMKMkKYT85E5Vyr2RESuuTJqf3QaJvACien+KksdwpZ8rSaUCfQOGIbISb5DwqOb9ADrGmGwpbhwlEf4n/aifqBNo3EDjdrkQ7KY18Pp1+v6h9Tfo5grtOHB3mZ6WecKhv2wTb+tW2mn040OKP806lxLS3mhFhbxG4EaLzDZR8ekj8Yj9VAajgrmntuNuvgm+hntO5JiZYsWsVc6tPBhYEgY/A+PGp5FxyvwXsUMppNnfOcA6aroV1XuDW25rlD0api4Gp9eWp5pfNdtHs+cZHJWH++9TTUj6ud/bAM4u28D4pj89xJoWbVR4kBOWJvQgRHjFWYMELED0DRYmWzZzGvJYF7n+l6A4JQQmcIWWL4l04gDTneACKXYsYVxXKQYPxDXgTZZ+IaKe1p6zmgOQ6Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(186003)(5660300002)(8936002)(66556008)(6486002)(83380400001)(450100002)(6506007)(53546011)(66946007)(508600001)(8676002)(26005)(82960400001)(316002)(66476007)(4326008)(33716001)(85182001)(9686003)(7116003)(2906002)(6666004)(6512007)(6916009)(4744005)(38100700002)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S1RHNHNZN054RjE5aSt3dUFpYU9FbEdYUW9RUmFtU2c5MjhxOWhycUdqeXZX?= =?utf-8?B?TmZUbjVMa3MzZzhDend2OGwyQWVOTXJmM0NmUnlEQ1B1M1lNSWFQbjBGL2Zv?= =?utf-8?B?MzU4L2taK3FpaVRFMGtEaG9ZMFJmc3M2SWN1QTJWMFlwQnlKOGJROUZkdU5r?= =?utf-8?B?YjYyTWpnWGhOMUZ4emxXNXJSUUIyNUNESnhNYkNKd2V5TlVSL1pCVSszL3cy?= =?utf-8?B?VmZQMGkvNitGY2o1ay9MakJFQWZlV0JVdjAyNTkyc2JBeHFHVFpnT3FxZThS?= =?utf-8?B?M2RsWUJvUnZ4ZFlWdVNNM0xvNFFiZkVaTFB3SzhGaU14QjFXTWN2dEljUENr?= =?utf-8?B?WXFUaTVLZnBMUmVHMmRpK3pVVGx2U1Jncmh3UUNlbUpacFhPYVZPby9kNE1o?= =?utf-8?B?Rk96ZGJCeUJSRXM2RVVBZUx3eFNIRERPUEt3dDZ4aHlNRlliZWxoaUgxUEI1?= =?utf-8?B?cHdiTzZyMjRuWXpJV2NkUmRuMzZPY2t6VTJpTTB0VEtqWCthZGlCUjREVWov?= =?utf-8?B?d2oxbjZJZCtGdEcrTHNnOTZmU3UrTTUvL3VmUUdXRG1weXAwbWVHZkJlNlJp?= =?utf-8?B?VnZLRStBOC9DT1ZRMkVENFVaQ2Z5eFdxS1lTT1ViWUJWaktUUnhzZk5Zdy9R?= =?utf-8?B?eVJoYlZuUThJczIrc1BNRjk4Ym9XSGdDd0JCYytiK3VqOC93MkxKNGJnSHkz?= =?utf-8?B?alVKMCtQVkx3bS9UdklUYjE3bmJpM1FOb1BxVElSR1RQbFpiekoyODU1RGtl?= =?utf-8?B?T1RZSmhWR3FMK1FLK21mOEVpWEpXVEhwTnkwbG8zTnJ0REp3eUhQVTQ5MWsr?= =?utf-8?B?UWJxaXRMMjRabFlxeG1yREVRUjJMMDNvRWxpVkIybjFJMlJXdEdYdno1WVpW?= =?utf-8?B?L282aEZROFRhVC9oSGQ0aHU4VVZFdXk2Mm4zMngzVkY5L2s5SlhDeTNuZ2xU?= =?utf-8?B?VjlnMkxnci9yeGxtekU2RUE3eGI0RXJNS2ZQK1RDZmRLRlJ0L3FDTGpSbklD?= =?utf-8?B?RjA2eE5HaFhVZGVzdlduZkR5UHo2SWJJc0xZQzdFWUJ3OVdDWU91SUkxOGh1?= =?utf-8?B?SDZ1dWM2T0c5cmRHV05ZVStOZmoyNDRaL2gxaFZzUlltazhIblRlZTZsdHlk?= =?utf-8?B?RDAwTzN2djB4aXZZZEFZSUtkMjcvNnk0VlI4Nm9MTVk3VXJKdTZweERIYllY?= =?utf-8?B?RW9KREZXTlFCbkw5Y0lMMEM3ZjVoWHpEeWg5VHp2K1ZSZVlaQS85NGs5VnMr?= =?utf-8?B?UjAzenVxQmVKdEJnWFBlYmMyWHViMmhUcDI2TjI0bG9yMzJDSlRlbUo5WFNB?= =?utf-8?B?aVpaTnZlMGljQ1VGR2JCY1BwN2xUMTUxZkZrOWlXd2krUk15eElVVTRhNGhz?= =?utf-8?B?WXhLSjZCbExaTVJyeDhnd05JV2JuWERMQmZGM3FTRW1SdDhvajFJeWdrOGM2?= =?utf-8?B?OWhlTktHeGgveDNlQlgyMDhreVNsTjd5TzdBZ0JPNUhGVHExdFd0c1EvQlhQ?= =?utf-8?B?S2dDK004dU1od3diQlltdmZjcFpsMjVyTVdXVGRZbm5QbWswNHRvT3lGQlpi?= =?utf-8?B?ZFhEYzdsRG01VTFoTUZrRmIzdERHc0tUMHo5M1lSdnhHbkkrWkdPK1A0ekc4?= =?utf-8?B?eW84SnJDYU5KSHhiemMrTUt2N3cyV2l4eE5rcGtpOVpNT1U0RVBCZzBwRFRn?= =?utf-8?B?dloyMlU4NExoaE5TNElBOXd1azBsaVJGSHBxT3VwYmRERHY1RTJlbDI4eEpC?= =?utf-8?B?dTJteEllSkc1MGIvazBWanE4R0RTdEQ1WXNUMlVaUlBKanZPS1RncFNzNU1y?= =?utf-8?B?OGJwZEVnUlpXWW1raVlGTjZFK3dQemNHRGwrdzVKUEJubHlmUmdIcDZFK1J3?= =?utf-8?B?MXFVNkdvVzNKeG9KcUhKRW5aSTFUZEtQdXhBQ3MzSTFJTC9YanJhaDdnQmdH?= =?utf-8?B?RXFPNkEweGw2UDByOUFXN0RtQk5ib2NsVnFNSS9uMENaM0RXVGFVWWF5VW9V?= =?utf-8?B?ZzE5NDd1S0pZZk81UzYzei82YW0vbkhsd2pXWHM3UGNoZ1JndmxVK0RjUWtp?= =?utf-8?B?TElzd2U3bWl6WGNnNnJDaXJxNUJHOGRpWGs2NnJTaWFvNm0zVURMbXZOdEJk?= =?utf-8?B?VU1PcmQyVXRkL1NrSERVWUxrb1ljYmhOYUJOQjE1MDFTdGtRZ0xJSkM4ekdZ?= =?utf-8?Q?hgRIptI49XA5qKWz3ndkWws=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6e6c1f6a-a795-4d73-35f3-08d9d4ecce75 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2022 10:26:19.1886 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Pae7FinKtCxUIw1OfdeNiEMC2BW1VVxIH3zvwpNTcb98RiMZ1VD08aIMUaH/ovPTkZrK2fFLmLlBS0PHLPNpIg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4138 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JY6LD4h1Yz4pyX X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Tue, Jan 11, 2022 at 12:07:28PM +0200, Andriy Gapon wrote: > On 11/01/2022 11:50, Andriy Gapon wrote: > > > > Recently I got a report of crashes related to using procstat -k on one > > of our systems.  The system runs FreeBSD 12.2 on an AWS Xen-based > > instance (can get more specifics about it later). > > The instance type is t2.large. > Here are all lines from verbose boot that mention Xen: > > XEN: Hypervisor version 4.2 detected. > Disabling MSI-X interrupt migration due to Xen hypervisor bug. > XEN: disabling emulated disks > XEN: disabling emulated nics > Hypervisor: Origin = "XenVMMXenVMM" > x2APIC available but disabled due to running under XEN Thanks. I've asked someone at AWS for comments. I guess the only way to workaround this is to switch back to using the emulated APIC. I will try to prepare a patch later. Roger. From nobody Tue Jan 11 10:28:01 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8DBA11944D25 for ; Tue, 11 Jan 2022 10:28:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY6N42jPJz4qXJ; Tue, 11 Jan 2022 10:28:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641896884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rpApX8YncfFCXkNUjAI2TbOFjmCFaRJYU+RAROlxk+g=; b=KFnaozcal2Ei5/sPug2XP/HFlz1tZmBg0KHWvYv7MgNdffFXKD93afIfbR/LJgVR2ahDsc N5/QugCPJv5nBU11wu2lWzGJCj0YjG6vsenNjUPyws/tomv0W0w217WMIDp11Mb2+1HwhG hqYSjV46NJmpsRtFZeZZ3gu695dD5ZoXzykxcMymu3jhdeknjHt/RUb/VglwlVTb1CDexu RdYn3kHtS+jTDuWSWlWGmRgqqoVYEzPmVlQ5L981CB9yrS3EZ5vfK7p24bVNvxVbt/TUQg aFcoh5pm3hkdIsFmeFoTiPkm8g4Gh7jdneTAUWo/yGs4M/6eWcd+Razm94y6LQ== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id B21923624; Tue, 11 Jan 2022 10:28:03 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> Date: Tue, 11 Jan 2022 12:28:01 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 From: Andriy Gapon Subject: Re: VCPUOP_send_nmi returns -38 Content-Language: en-US To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: xen@freebsd.org References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641896884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rpApX8YncfFCXkNUjAI2TbOFjmCFaRJYU+RAROlxk+g=; b=MxXHHIKVcT1zn9tP+OZ2S9rLkVJ6t7sN6o65RXLhv9TaRTFjTKMsCsWSO53It3gGJaTucW sCt0PDbU7k3roEYlBJkkbNC5B0+8HZn54FC+pmOvrF2TkgJD0Yo75g3NTdGUtYGJtBkdAw YLjNbHnjHFJWgUfL0fNY+R/o/H1X/c+X1nobTH23ykc30QJp7ZZ8TF/qVsDrB24JDyjQm+ 0OXpdeSTLApns8mjqf0WnY/MdX0PQ7xo+TvVADSkAfoJnTfWQ6slU/9BFyjajy5LNHCijH L1a/LDMzjw3CFlzzdYjIfaB52/sDB6bzXpExAe9gGwdoUfH5rtqvSmokQbKecw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641896884; a=rsa-sha256; cv=none; b=OUlfvU7ZaIrzXQEV0us/9yVtqi0aOcDSS/zuGLJC1WocdsBhrMjh4g5y0SW5IhyR9eqtIC qhZA6h1jDQ7jUfGYTCOL3ktLvi2JsLoIE+KLKJu4Dn1GdDfs5ZGYOBOzN31SXyPKR8UfUx GtYDPQ4FSmFWV+HqX41RFRZ79s5qG6ssx/MBEocHV5LC+A/0jvtDjiZmDVGY767E3V98Yc h+LotpxnSO2EWD3Ca09Z27qkBjidGqsHi8eEqahCaTZ8Z3eImmecKvuwlopZijzyelTpBh 70fOxVPlj+U60Oi6vPrw+SLoDirngDL9pWR0NZlnnHppvH4rDbAT88PYpjm6XQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 11/01/2022 12:26, Roger Pau Monné wrote: > On Tue, Jan 11, 2022 at 12:07:28PM +0200, Andriy Gapon wrote: >> On 11/01/2022 11:50, Andriy Gapon wrote: >>> >>> Recently I got a report of crashes related to using procstat -k on one >>> of our systems.  The system runs FreeBSD 12.2 on an AWS Xen-based >>> instance (can get more specifics about it later). >> >> The instance type is t2.large. >> Here are all lines from verbose boot that mention Xen: >> >> XEN: Hypervisor version 4.2 detected. >> Disabling MSI-X interrupt migration due to Xen hypervisor bug. >> XEN: disabling emulated disks >> XEN: disabling emulated nics >> Hypervisor: Origin = "XenVMMXenVMM" >> x2APIC available but disabled due to running under XEN > > Thanks. I've asked someone at AWS for comments. I guess the only way > to workaround this is to switch back to using the emulated APIC. I > will try to prepare a patch later. Thank you very much! -- Andriy Gapon From nobody Thu Jan 13 14:57:04 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2D6001954741 for ; Thu, 13 Jan 2022 14:57:14 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JZSFj0RcXz4p58; Thu, 13 Jan 2022 14:57:12 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1642085833; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=tobwTujOKEhLI0Uv/0dnOKDEvw4wllwpIk2bVKqJ0rs=; b=GsbWg3cSYn2J/FFXpE2/SNfGj8Uby75bFpE/mrLHGluMdKeE1OQlpitQ 9NzmQuz7YGb4uipl+lmkjiVPx1LeBK+5OvhuwMSiaQ/w6Lr8jQaKSm4uq Q9lejKgV0AF94k+IE5b5W0hZzA4bUllF0k0vGsMtJzcu3wnsYmit2bbpH E=; IronPort-SDR: xUDyZVPJ6Osu4ZrNSSMVDZUjju4NtGDJlTKOnyidLbmDJApukipSRktvXRvfdqoUBIihZWp6S5 jJ8mFhK7WNg9mu6nxt8JABmTMF/CFmimkH61yY3KUTwjHSHQFf/zECVw7/mA8LtfeaEyC53Y+j 2g0Czcgaxyt9B+zBneiVZpNktuCOHLaY18vx9ZbM9vz+uZnfSs6u/Dz2LLIl0L48OfJFxn5+F3 xiio+z033FnmO3euYY4ep9ADrPw7PhUFD7iHoQSCHL7YfYWOkcPjLSsy4t57JOz2fliDosDPUk 3NhX9ZIYexHNmrAJBmzvwyNc X-SBRS: 5.2 X-MesageID: 61928788 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:HRj1EaJwAt4s7bufFE+R8JMlxSXFcZb7ZxGr2PjLsTEN9I4Qp2hZl 39JACGYeq7OMTrFz+oGa9m1/RwEuMDTmoVjQQo/rywwQntHp8GYVN/BdRavbnrPdZaeHU42t Z4VMYeQJspsRSWB+hrxauPo8SQiiarUHNIQZAK81gVZHGeIHw960Uk/8wJAvrNVvTTQ73th0 /vi+cTHOVTjwzdoOWJ8B8mrohhltaT5sj9CtVcyP/0XtgWFyygfBcgWffHgcyTzHYUPT7C0T uuSlrvopD6D9kcnAYL+nOqkKhJTH++KZFTe0BK6N0Tabj1q/0TehY5qb6p0hT5rtgi0c/BNJ PRluMK9QA53ZvbAwLhEAxIGTH4uN/AfoOSafie04JGaxEeXL3e0z/tQV0xnZodwFsSboI1tG V70DBhXM3hvUsrvmOrTptGBAq3PFeGzVG8lkigIIQrxULB2Efgvf42QvYUChG5p3ZgUdRrjT 5FxhQRHPUyojyJnYj/7OLpm9At/riCiG9HwgAv9SZsfuwA//iQouFTeGIO9luixbddUhi6lS lfupAwVNP242Ou3klJp+lr07gPGcLiStIg6TNVU/dYy6LGfK/B65LT7mjJXrNHg4nNSVe6zJ GRL2zYIhJET0nWgdcXdQx3nsVSZ5DclDo84/+0SsGlhy4LR6gedQGMFUiRAeJots8peqT4Cj wHT2YmzXHo27ePTGSn1GrS89FtePQAPKmAPf2kcRBYt6Nj/uoAjyBnIS76PFYbr14yvSW6gk 1hmqgBgi5ZOqfEt0JngxkuWrAiLvZ2ZRBEMs1C/smWNsVoiOd/Ni5aTwUjW695ONouUREKN+ n8elKC25f4BJZ+XmSGHW+lLG6umj8tpKxWF3wQpRcN4sW3wpTjzJui8/Q2SOm82E9oiUAbIR nbYgh9x+J8MLVG3N4ZONtfZ59sR8YDsEtHsV/bxZ9VIY4RseALvwByCdXJ8zEi2zhFyzPhX1 YOzNJ/1UC1EUfgPIC+eGr9FuYLH0BzS0o86qXrT6x28mYSTa3eOIVvuGAvfN7tphE9oTei8z jq+Cydo40kOOAEdSnOOmWL2EbzsBSJrbXwRg5YGHtNv2iI8RAkc5wb5mNvNgbBNkaVPjfvv9 XqgQEJew1eXrSSZdV/SMys7Nu+/BswXQZcH0coEZwbAN58LO9fH0UviX8FvIelPGBJLkJaYs MXpi+3fW68SG1wrChwWbIXnrZwKSfhYrVnmAsZRWxBmJ8QIb1WQorfMJ1KznAFTUHbfnZZg8 tWIi1OKKbJeFl8KJJuHN5qSI6aZ4CJ1dBRaBRWYe7G+uSzEreBXFsAGpqRmfJFXd0SSn2vyO sT/KU5wmNQharQdqbHhrauFs52oA615GE9bFHPc9rG4KW/R+W/L/GOKeL/gken1WDym9aO8S /9Syv2gYvQLkEwT69h3EqpxzLJ47Nzq/ucIwgNhFXTNTlKqFrI/fSXWgZgR7vVAlu1DpA+7e kOT4d0Ga7+HD9zoTQwKLw0/Y+XdifxNwmvO7e44KVnR7TNs+ObVSl1bOhSB0XQPLLZ8PI4/7 /0mvcoat162hhYwa47UhSFI7WWcaHcHVvx/5J0dBYbqjCsty01DPsOAWnOnvsnXZowVYEcwI zKSiK7TvJhmxxLPIygpCHzA/etBnpBS6hpE+0APegaSkd3fi/5pgBAIqWYrTh5Yxwls2v5oP jQ5LFV8IKiD8ms6hMVHWGzwSQhNCAfApx70wloN0mbYU1OpRirGK2hkYbSB+0UQ8mR9eDlH/ e7HlDa5AGiyJMyhjDEvXUNFquD4SY0j/wLPr8mrAsCZEsRoejHimKKvOTIFphaP7RndX6EbS T2GJNpNVJA= IronPort-HdrOrdr: A9a23:kh64n6CYDrUk3FTlHeg0sceALOsnbusQ8zAXPh9KJiC9I/b1qy nxppkmPH/P6Qr4WBkb6Le90Y27MAnhHPlOkPQs1NaZLXLbUQ6TQr2KgrGSoQEIdxeOk9K1kJ 0QD5SWa+eAfGSS7/yKmTVQeuxIqLLskNHKuQ6d9QYUcegDUdAf0+4TMHf8LqQZfngjOXJvf6 Dsmfav6gDQMkg/X4CePD0oTuLDr9rEmNbPZgMHPQcu7E2rgSmz4LD3PhCE1lNGOgk/jIsKwC zgqUjU96+ju/a0xlv10HLS1Y1fnJ/ExsFYDMKBp8AJInHHixquZq5mR7qe1QpF6t2H2RIPqp 3hsh0gN8N85zf4eXy0mwLk303a3DMn+xbZuCmlqEqmhfa8aCMxCsJHi44cWADe8VAcsNZ117 8O936FtrJMZCmw0hjV1pztbVVHh0C0qX0tnao4lHpES7YTb7dXsMg24F5VKpEdByj3gbpXX9 WGNPuspMq+TGnqLEww5gJUsZ6RtzUIb1u7q3E5y42oO2M8pgE986MarPZv6UvouqhND6Ws3N 60QZiAoos+OvP+XZgNdNvpfvHHeFAlYSi8eV56cm6XXJ3uBRr22uvKCfMOlaaXRKA= X-IronPort-AV: E=Sophos;i="5.88,286,1635220800"; d="scan'208,223";a="61928788" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gZXKMgWJznN/cxEnznxnPBf1ankpxMafFiYFzbZwPzi8JLf+TudYOHNTe0t1LSD+3hkiSMZbCaunbdap3YTkKCkTTYNoNHbca/cvBIKLK8rvHXLiC8lVm0/XuDKM+EX4zJGjJLcL+mLZw9a+E+XONzchY1wdZiHw1b3RzxvyRubVWwfCZvfQjPzlz0wwDycIIDj8EoeSMcWOp64Cm3/IAW0cdGdrApGrQnlrHYZcJ6AvyOFI1EUVEy3rlTrceFRxnAWrVIsqHQp5WpZeyy0BJi3Tfvg3HIfy6aBsx9wU9PPS4pcCFyEMkKRkZPxg65hCybGnd97YPT5ibADXneCmJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5Aoxz1FY4TaTV68Q3ZBG9OEXDSCDXA3UJmQhLn5NC9g=; b=APLvho+NzwB6NaESuY84sUTZkcMqQDDI1V1wWXqmNc4aakwx/5SUp0SHupZXD2Dl+dFALX+Tnx0izvOddOoJExsuq72J7nt6Am4j8nm54rP/WtG0whwjYzhTx7XFa7wDavpFF+9o2jhlTZcg3X3iHdKRPwVxmm7QxdHe+tdRRjCQ31XgBnPNkW5z2J+2APzgXwB8MjlctLzGIvobj4rUNmBHuZES3dFYBCqWqvfp46LQwUeEllbx93pxQ96TqbVaGpy3H3njzMXpn4fZkHWU6P7Xi9vmdmdc8aFbTSDTXPu7c9KV1dF6HAm0MpWkTI6TPAl+iGDPJgqINhkFkElNYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5Aoxz1FY4TaTV68Q3ZBG9OEXDSCDXA3UJmQhLn5NC9g=; b=R5CAe9yQTNldU0MBBPNCGbRKQiCoiMCKqsDykorSgeTwl7T0H/WZ4th5cfqjSMpg7m5RsURbzb9ZA6vpGifkyZ6BOf75n8EEnFjcYrh9jPGH0NkaHKr9nzCZUFKssW16yuoF9/ZBICfdBOQRqAvBq0YVvkmbkNq203BD6nhcClk= Date: Thu, 13 Jan 2022 15:57:04 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Andriy Gapon CC: Subject: Re: VCPUOP_send_nmi returns -38 Message-ID: References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> Content-Type: multipart/mixed; boundary="s7dTevK8xyWDIcXS" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> X-ClientProxiedBy: MR1P264CA0143.FRAP264.PROD.OUTLOOK.COM (2603:10a6:501:51::22) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 36ad8cc4-4ed4-4956-3ed7-08d9d6a4f8d3 X-MS-TrafficTypeDiagnostic: SA0PR03MB5467:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iYUwijimJjxgG2TKWimqVthslyAwZ8O2mDOFpuFv/unDCoNRjJT/8D6SpeJfENlqMcY2uhZ3mvke8Gpq7tv+FCoFM5IZ86QMTMo11L/dTAt9Kkm4s9KmcGDXgQx4AiG/pCMhZbEtSUlPt49syldpNYV9ssmw2HKgYSVGo6WpZ7uDAfVr/uoyFD+nJw0krAVMlIMjhgsh4cJF0TmnTukV8h3BJM0KhXcq7D+HeeX0A7Lqo6rtbeGawqPuELAJAU3UzHR6itbNKE1HlyAQCtkgO2Kuk4EmKI80g0eKbBB+71JH3HUXOcg0Fuysq8H4FR4Ta0tUFVwKb9FYIRgvg0Yc5b7a9MK6o8LY3xG0Pw3eQEDNueZ+w7jaZKdabYEhrCfHHJjwTCTpeujcPqzefo/cWykRPlf1haG8Q8g21A3RvN7aOUx2rDDGZGB9FJ+4u/i+t5+Axekln57IVb3VdKpPAWjbtlckJr4rQmzidPDG+ypPxq0Dd8ePGZARnqIaZsoDDLSdfbU4wL+hTm1eGOZvfkIzfV/o8QYG6AUh2ZKpkmTs7FbuUNMkh8/AV6BhSmqQH/vbOrBnQyrQVisbBBLIjAer7LIp0z/kjULabJtNISXUIV7I2rraSFU7dvQlqkCE0YWu7mN//9Df7tOjZ1eihTXQRNl11+wRP5qHFP2ZfdPsoFEwvuq6iNASp9e7gxHzdJWMPzM7dyolQ2BlgFeUweGn32LZCuqb/z+kINcTnog= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(4636009)(366004)(508600001)(66556008)(7116003)(33964004)(6506007)(66476007)(53546011)(6666004)(66946007)(44144004)(38100700002)(316002)(8676002)(86362001)(235185007)(8936002)(26005)(83380400001)(6486002)(82960400001)(33716001)(4326008)(85182001)(9686003)(2906002)(6512007)(186003)(450100002)(5660300002)(6916009)(67856001)(2700100001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dXJvWDNBZWYydHYrbXgxZTlFYjNNYi9qYlU0NHU5S0VoZ2lLbHpLYTNwaS9n?= =?utf-8?B?VkVDTi9jbnB2YUNsZDBEcXdBNHkvZVczUVAwKzBWNlNoT1RzUnVISENpQkJC?= =?utf-8?B?b09GL0RCV05yNEZFNmNVY3Jmb1JCVXo0TGxQNEJ3Mkd1VzEweHJRZE05Tmsx?= =?utf-8?B?UldQNmNsQ3RGaWM0RXhhdWwzcTRscm9xR0tFcjcyanNQVmZ5dDRzRFFCTWpo?= =?utf-8?B?YWN5c0hjV0cxQWphaVBQbkFKZ1NicytJd2FiWm9IR1ZDMEdKa3NzR1lsTWZN?= =?utf-8?B?VlptY0FYdjBaQTUyZzNPNWdkNENqWFFmWmdRdDlTL1Q0NFFvRkZRcnZld0Vk?= =?utf-8?B?ak00NG9sSDBQY0F3aG5oRXY2UlZBYUhzVGVIeXJXdFJ1aFZ6cGRqVmZBeHVu?= =?utf-8?B?RjVHdE5QS3pMSTVIVVZWcHU0U0hFeDVPTG9pRmJmaE5MRytLdFBFZmxUM3N6?= =?utf-8?B?MEtWY0l0cVcyK1ZmYTVkMGd6NFdVZlJtRU1kOWV2V0RUeG0zdlBrbElTTnZH?= =?utf-8?B?VmNOYjNYUnBEeHJQODFodnBYR2VhbGhGaHAwNXNjR0ZGeWJITlcvbmJ5Z1Fv?= =?utf-8?B?a2F2c3B4RGp3Zyt2MDVBeTlLSXRrdFkyd2FoTWRSUk1NVEwvaWc3UnFMSlNE?= =?utf-8?B?cUc5ak9FUERtK3hScFNtTC9uUWI5ZFVmbFZLZUt6QlcveldkdkdaRTRieG5K?= =?utf-8?B?RERhSVFrR3lmZFByU0REU1lOVlA4RXEzd24vMUlmNlVYM3E1OVhUTTdIQ3R2?= =?utf-8?B?UnZ0eDdFQVFTMVcrYXU4a2xEeDJmOWdpV1JSdUdKd0ZNM1ZoN1BNeFZ4dUNU?= =?utf-8?B?dUorSzFUeFdxRk5xVUN3ZjUzU0ZSQUxZUE1rOWptUDh1YVFKYW5sRHB3MDNj?= =?utf-8?B?K3Y5Y1NGUHAyc1BJRzgzUkhFTVBWOEJMTVBhaWFwbWhrcmNNVU52V3E3YXJD?= =?utf-8?B?NnQyQkhKOElQWUw5c2l2Vmh4U2Y4MCtaNkpJOVhaTWFrcDczTG83L3E0Y0ZO?= =?utf-8?B?TVNZNUxzZmtuRGxOS1N1Ym14V0Qvb0liN1dUV3l5TkhCbzJRQnhQL1RFT0NW?= =?utf-8?B?ZFFnRmNYaVNNcFE5Z213TjV4a3B1bFZHYXlNUHN4d1RWODVkejZCRlEvSVZI?= =?utf-8?B?ZW5ycW51T1cwcEozWFVOUVIzY2JHWE9vcS90dDJTR0g2RWVPUVNiNUsycEdF?= =?utf-8?B?K2c2TXNTdHhkanRVWHpyeFRtdG1DelIrUnh0VEpyNDN0UDBvQ2pVd1VXcDdX?= =?utf-8?B?c0gxUVA3ZDNWL0E4czZ6ZVpXTXdTYk9kVXpuTkh4Qk1ZTldqSHR4UDlaM0JM?= =?utf-8?B?VEwvTklNZGxvc3dsZjZYQ1lBQXFjVHBsSndYVlpXSzdwUElCelg1czRyK01n?= =?utf-8?B?ZGQ3RlpKZ3FnY0hlaXY5Z1BBS05IMzNOeWFTOWIyc2ZUdzdVM29xRmt5SFpp?= =?utf-8?B?Y3ZSODB3WWFqakZFRnR0WlN4LzVRdC9yVytBQkk2OXM2MWZ5Mlp6T1VUNU9H?= =?utf-8?B?em9JbHkvc09RelNTMWZ0dDVtOXJPblB4V1FhYU1RSnRmd2hqYkorVEw3VGlT?= =?utf-8?B?L1pncVhFU3dYanJ4Sm1INFk1UmJXbE0xeDBtY3prNEZGN3NOQVhYbEZGUitq?= =?utf-8?B?a0xsRFRqbkE5UWkvanU2VCt3T2VLQ3NXa1hUV2EyZ1FQZ0d1dUJLaFpJemFp?= =?utf-8?B?dWIyb1RYb2luQ1RITG9nTHJSSU9LdCtINFJrTFdvT0p2MVlHcEtjL0tGTHVk?= =?utf-8?B?RjNRMGoyNlVsSk4xTll1a2Z4dTN6Tno3VGl3K3BMK2pXd1c3a3dLUURodUNo?= =?utf-8?B?aFpoMTBES21GOUlGc0Rvb3FXenR0Y0NVSnF4bTFORk5FaGFhcU1MZnl5N3Bh?= =?utf-8?B?bnoyb3lMaUxOM2hUeDVxQ2NvZktNZlJHY1VrTUkyYmhlSFRkMGN6S0pJL3lT?= =?utf-8?B?VC8zemJlMnp3TE9Ba0hxNW5uNHFHR2k2Z3hXWE9FVVdDaVVKNzZNR1dPbHVn?= =?utf-8?B?WVNvNlNYb0NyVXd6V0hZWGZvMStrUFZuRUNHcURXUUJVS1FvZkFVQXh6NDY3?= =?utf-8?B?bVVQemY1QkgzTHJIb0IvWGRUMk9YanZFRW9MOG02VHdyeFF6cGszWEhFVzIy?= =?utf-8?B?RGpnSFFLNFNNaVZDbUZ6eW1OZGM2MGhEVmlVeTRSZWVvSUdaUE1TK2dMUTFm?= =?utf-8?Q?i3tW6yvnOa4Xt5GOWtqqHcs=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 36ad8cc4-4ed4-4956-3ed7-08d9d6a4f8d3 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2022 14:57:09.1740 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: o+qohI/yGt7mQyWpJOCok2gX4hBouN5Vqfp2uIFKGil48EJsALMwm1wGHriG7asbytXVeeChMGfBAUMdYdin3Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR03MB5467 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4JZSFj0RcXz4p58 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=GsbWg3cS; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=R5CAe9yQ; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.145.153 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; HAS_ATTACHMENT(0.00)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; CTYPE_MIXED_BOGUS(1.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; MLMMJ_DEST(0.00)[xen]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+]; R_MIXED_CHARSET(0.71)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; MID_RHS_NOT_FQDN(0.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N --s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Tue, Jan 11, 2022 at 12:28:01PM +0200, Andriy Gapon wrote: > On 11/01/2022 12:26, Roger Pau Monné wrote: > > On Tue, Jan 11, 2022 at 12:07:28PM +0200, Andriy Gapon wrote: > > > On 11/01/2022 11:50, Andriy Gapon wrote: > > > > > > > > Recently I got a report of crashes related to using procstat -k on one > > > > of our systems.  The system runs FreeBSD 12.2 on an AWS Xen-based > > > > instance (can get more specifics about it later). > > > > > > The instance type is t2.large. > > > Here are all lines from verbose boot that mention Xen: > > > > > > XEN: Hypervisor version 4.2 detected. > > > Disabling MSI-X interrupt migration due to Xen hypervisor bug. > > > XEN: disabling emulated disks > > > XEN: disabling emulated nics > > > Hypervisor: Origin = "XenVMMXenVMM" > > > x2APIC available but disabled due to running under XEN > > > > Thanks. I've asked someone at AWS for comments. I guess the only way > > to workaround this is to switch back to using the emulated APIC. I > > will try to prepare a patch later. > > Thank you very much! I've got two patches for you to try, the first one is 'mni-fix.patch' and should fix the NMI sending issue. I'm also attaching a second patch, that should allow your guest to use x2APIC. FreeBSD has previously prevented Xen guests from using x2APIC because of a bug in old Xen versions, but I think this should be solved now, so if FreeBSD can successfully boot on your VM with the patch applied I think we are safe to enable x2APIC. That one is called 'x2apic.patch'. Let me know how the testing goes. Thanks, Roger. --s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="mni-fix.patch" >From 8a4a13210d7e55b65c164ed5d0a9d9e4e6f9b593 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Date: Thu, 13 Jan 2022 14:48:14 +0100 Subject: [PATCH] x86/xen: workaround for VCPUOP_send_nmi failing It has been reported that on some AWS instances VCPUOP_send_nmi returns -38 (ENOSYS). Add a fallback to use the native NMI sending procedure when that happens, so that the NMI is not lost. Reported by: avg MFC after: 1 week --- sys/x86/xen/xen_apic.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index b553e5248716..701598e44f4e 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -107,6 +107,12 @@ static struct xen_ipi_handler xen_ipis[] = }; #endif +/* + * Save old handler as a fallback. It has been reported some AWS system don't + * support VCPUOP_send_nmi and return ENOSYS. + */ +void (*native_ipi_vectored)(u_int, int); + /*------------------------------- Per-CPU Data -------------------------------*/ #ifdef SMP DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); @@ -273,10 +279,11 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) } #define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) -static void +static int send_nmi(int dest) { unsigned int cpu; + int rc = 0; /* * NMIs are not routed over event channels, and instead delivered as on @@ -286,24 +293,30 @@ send_nmi(int dest) */ switch(dest) { case APIC_IPI_DEST_SELF: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); break; case APIC_IPI_DEST_ALL: CPU_FOREACH(cpu) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); break; case APIC_IPI_DEST_OTHERS: CPU_FOREACH(cpu) if (cpu != PCPU_GET(cpuid)) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); break; default: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); break; } + + /* + * NB: for the loops above it's expected that all calls to + * VCPUOP_send_nmi will either fail or succeed. + */ + return rc; } #undef PCPU_ID_GET @@ -314,7 +327,8 @@ xen_pv_lapic_ipi_vectored(u_int vector, int dest) int ipi_idx, to_cpu, self; if (vector >= IPI_NMI_FIRST) { - send_nmi(dest); + if (send_nmi(dest)) + native_ipi_vectored(vector, dest); return; } @@ -580,8 +594,8 @@ xen_setup_cpus(void) xen_cpu_ipi_init(i); /* Set the xen pv ipi ops to replace the native ones */ - if (xen_hvm_domain()) - apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; + native_ipi_vectored = apic_ops.ipi_vectored; + apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; } /* Switch to using PV IPIs as soon as the vcpu_id is set. */ -- 2.34.1 --s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="x2apic.patch" >From d2c5cbe40836f9784307ca072f030365269586f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Date: Thu, 13 Jan 2022 14:54:21 +0100 Subject: [PATCH] x86/madt: allow Xen guest to use x2APIC mode The old bogus Xen versions that would deliver a GPF when writing to the LAPIC MSR are likely retired, so it's safe to enable x2APIC unconditionally now if available. --- sys/x86/acpica/madt.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c index 8e343a6619f3..880857a8cfad 100644 --- a/sys/x86/acpica/madt.c +++ b/sys/x86/acpica/madt.c @@ -168,9 +168,6 @@ madt_x2apic_disable_reason(void) return ("inside VMWare without intr redirection"); } - if (vm_guest == VM_GUEST_XEN) - return ("due to running under XEN"); - if (vm_guest == VM_GUEST_NO && CPUID_TO_FAMILY(cpu_id) == 0x6 && CPUID_TO_MODEL(cpu_id) == 0x2a) { -- 2.34.1 --s7dTevK8xyWDIcXS-- From nobody Thu Jan 13 15:29:21 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D5D4B194C29C for ; Thu, 13 Jan 2022 15:29:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JZSyr5hQTz3MCP; Thu, 13 Jan 2022 15:29:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642087764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a0gzO7x3nqGofMlKRhNGAiO7/BJRngkdh+rUwqQ41MY=; b=vcB6QwwGcuv9cIGrcfr0F+t2EEjpd5yiCcCIo799GBqKipzn/wGnPm35DFyduYr8YIeIQg 7K6+6qoRTrqQY1FwAIBdVny6U2wic0D73QIT5zcgN5TSl0MmNlb4S9FgswwbFXGu+R0Yvh xlBNkSallkEl/fjNkY0GDauPPd2+SJoca5SEqFxgiGLq1qZPHCMJf10ADxffiFXg0Fa3kV aUvWAWTFPo1Tj5fB5tLMFo6SjTjWAZIsh0TLvgCQJxWXcx1jMvuu0HR9WZs0gX/rFwSi9I 58NcqPSRyJnHtDzOPqa46SZ7JijyTIGqIxOlMOl1bbBNaw53bzf11EjX6b144w== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 46F262D6C8; Thu, 13 Jan 2022 15:29:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <7fb3c0fe-0145-7474-aa5e-0ad2dbb17e9e@FreeBSD.org> Date: Thu, 13 Jan 2022 17:29:21 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 From: Andriy Gapon Subject: Re: VCPUOP_send_nmi returns -38 Content-Language: en-US To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: xen@freebsd.org References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642087764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a0gzO7x3nqGofMlKRhNGAiO7/BJRngkdh+rUwqQ41MY=; b=rVLPQ7pVXGU8mCdyrJN1UJPATKBmHgCSSYSKgELipDi+2k3XAoYBvgwL6H368AwuOYqfDK 7KxTB3SYgK2oq1CuRS4Dd6nbHXDxZF7s62G/FSlDtlCZ/Ut+P8ldlLaa0qBHJAarmPT8g/ j3Q/HXjY0M7YfzGaVROcRgLmWJSMUKsjjgPbkNKyYHe9V7a41rL2IWyJ//z/E+b6bDOJZd Lv6kBvadk+noJ0i+9PUrxssUcRnuO30LRT8HVkdFXMSqZx0G5iQdI56hfBstYq3aD13pvk yoCFuf1zp3XQvnJGGO7RYAxFWmpmfNuHku0s8uCIx/TGtQp4i8XR9lMzzTMb3A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642087764; a=rsa-sha256; cv=none; b=PRYS2pchZFvWoq/FhNR9HNDCW7m1dqwimBPPztkeArm3iNggCC9r0/H6uPLFwunZZUONMH GwfFHJvJyaVNjQHZJBAl7ltcQy05VHMyDOM984T+cYAb9AWwBfipOsg4d6aEAOfc3cHPmj gdutJdWgqkwYat7eZ9e8LLoIXudg0lKtO3k+X2olOp6M5s8jmwN8fOrKIl4pHOqYo0iYyl EVUsH7mvmqUCUfO6P3sLlmNWrKFXfq7BspxvE+byETq2PQnJYgSM0p8tvDnK6tSpV3ZXlB a+nx/VegnWm1BzjDofdIJDftsXkZ+UpLkhlaHcEczGtd4goiWhPE9ToBrPigjg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 13/01/2022 16:57, Roger Pau Monné wrote: > I've got two patches for you to try, the first one is 'mni-fix.patch' > and should fix the NMI sending issue. > > I'm also attaching a second patch, that should allow your guest to use > x2APIC. FreeBSD has previously prevented Xen guests from using x2APIC > because of a bug in old Xen versions, but I think this should be > solved now, so if FreeBSD can successfully boot on your VM with the > patch applied I think we are safe to enable x2APIC. That one is called > 'x2apic.patch'. > > Let me know how the testing goes. Tested both patches, both jointly and independently. x2APIC mode gets successfully enabled: # sysctl hw.apic hw.apic.timer_tsc_deadline: 0 hw.apic.eoi_suppression: 0 hw.apic.x2apic_mode: 1 hw.apic.enable_extint: 0 Not seeing any obvious regressions when running with it. NMIs are successfully sent and delivered now as well. Thank you very much! -- Andriy Gapon From nobody Fri Jan 14 08:29:04 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ED41819495FF for ; Fri, 14 Jan 2022 08:29:16 +0000 (UTC) (envelope-from dsx@bsdsx.fr) Received: from blade.bsdsx.fr (blade.bsdsx.fr [IPv6:2a02:27d0:100:f205::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "bsdsx.fr", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JZvbb6snTz4knF for ; Fri, 14 Jan 2022 08:29:15 +0000 (UTC) (envelope-from dsx@bsdsx.fr) Received: from mail.blade.bsdsx.fr (dkimproxy.bsdsx.fr [2a02:27d0:100:f205:a642::20]) by mail.blade.bsdsx.fr (OpenSMTPD) with ESMTP id 6b2b48b5; Fri, 14 Jan 2022 09:29:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=bsdsx.fr; h=date:from:to :cc:subject:in-reply-to:message-id:references:mime-version :content-type; s=mx2; bh=oSAXuY9ToA4qIBkoERlqM5e1CbZNko/wJ1UZ7+n QdLQ=; b=NDyG9w240TThuyrhFv7sKoDDaQgGOi7hGvGSoRry2iVNT5f3Rs9Jg9t Fl9IuddeZP8rj6cU4zbiW+pxa73mGvc0uYT8eg8VWl6Uah7ZydXStCY9DViSfWqH QFOwGsAWnow89GycoeGjnGsNUEW/OP9JsQbHaVjgSaFtBzX/7Cjc= Received: from localhost (localhost [127.0.0.1]) by blade.bsdsx.fr (OpenSMTPD) with ESMTPSA id 2d1782a4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 14 Jan 2022 09:29:04 +0100 (CET) Date: Fri, 14 Jan 2022 09:29:04 +0100 (CET) From: Freddy DISSAUX To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= cc: freebsd-xen@freebsd.org Subject: Re: Can't shutdown FreeBSD domU In-Reply-To: Message-ID: <3911219-9935-2f47-6f3c-9c2560f672@bsdsx.fr> References: <20211204093337.GA69642@blade.bsdsx.fr> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 4JZvbb6snTz4knF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdsx.fr header.s=mx2 header.b=NDyG9w24; dmarc=pass (policy=reject) header.from=bsdsx.fr; spf=pass (mx1.freebsd.org: domain of dsx@bsdsx.fr designates 2a02:27d0:100:f205::1 as permitted sender) smtp.mailfrom=dsx@bsdsx.fr X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdsx.fr:s=mx2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdsx.fr:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[bsdsx.fr,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-xen]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:49463, ipnet:2a02:27d0:100::/48, country:FR]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello Roger, [ snip domU freebsd shutdown ] Our patch work perfectly. Thank you very much, Regards. From nobody Tue Mar 1 10:03:17 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E7CD519EDDD9 for ; Tue, 1 Mar 2022 10:02:49 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7CVJ62qWz3vjK for ; Tue, 1 Mar 2022 10:02:48 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pg1-x52d.google.com with SMTP id o26so13362400pgb.8 for ; Tue, 01 Mar 2022 02:02:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=LpNFXhjDj6L1rQ3RplMuaMjQzUCH7oi5wz+ZrcH1IiU=; b=baZrKhdTOXMxyk20ut8voMtmpbnVoea5BObB/g48/TIBd7xaiePvRlsOyiTjyZK+m9 3NlA8Vi2CTm1zmZqNkbXyvATndZJ5jTUaTkahoeKGN2SScGvsiTReV9niauZCKsYe0dp nfkguez87m/KAJphGkuVmub0k8QuiEsa551xMboDJb96XSrDzrjIj4BYkJvGqY5giDTR dtWpZhBH82rFf650JvGPI/ii4lxLFXPoMMHMGZQqbq9rul+vRiXLaPUTThQB2Xy4cJ+w efNheSkdaeNfWtjb6YIxw2Ekne2m0sWbMRyY5B2oTJC1Gt/dal8mWiy0kVkEMxVopvwc uB0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=LpNFXhjDj6L1rQ3RplMuaMjQzUCH7oi5wz+ZrcH1IiU=; b=uOOqI4NbXvsqU1JZGujruG4uGXGEmMjT1aUKTuOoeuYD4TPkU7PIBd7xuKXkHvVAbS pZPOXF/vms9fP0cK+4d77MtHe9ex++LFDeWEFzlmi2Oji3RP4jXyEiOBXnmGnUeuFpL0 V9lbPUHlREXYxRQaoF92ggAUlc2fDJjG7dpuOZfYwZk8IAajh9SixPDPPI4plp1cMsWK /TT3s+FPxL0bYFVW1Ysu052Qqd4gBHit+V/puuInzcH6XCEs8ls6QyF556Te6dszt9F8 ewUsgwHt+Rmj0STwn/7Pjr441m/LtD8HpZtrfcTI80i9qnb8TEmf/5APwWwppfK/7ILG mfOA== X-Gm-Message-State: AOAM531QUXvolj0FuySRO67Uv+0JjUDpD9CiS8PfyRiQN53du6ArrGft F5fLwrXaM2QJc1rrCBepNXc2Y+DGdOwpqruAJY25ZhW85co= X-Google-Smtp-Source: ABdhPJzKWSjCC3nKwv4aQ0WHNOF1jfJ0X+n92q1kI+P7/KlmAQrqEkb+c/d1o9x9kfC41rqOvdNII0/W7eX5N7sbNt4= X-Received: by 2002:a63:471f:0:b0:373:9b10:886e with SMTP id u31-20020a63471f000000b003739b10886emr16838725pga.307.1646128961909; Tue, 01 Mar 2022 02:02:41 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 From: Ze Dupsys Date: Tue, 1 Mar 2022 12:03:17 +0200 Message-ID: Subject: ZFS + FreeBSD XEN dom0 panic To: freebsd-xen@freebsd.org Content-Type: multipart/alternative; boundary="00000000000007764f05d9254461" X-Rspamd-Queue-Id: 4K7CVJ62qWz3vjK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=baZrKhdT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::52d as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52d:from]; NEURAL_HAM_SHORT(-0.99)[-0.994]; MLMMJ_DEST(0.00)[freebsd-xen]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --00000000000007764f05d9254461 Content-Type: text/plain; charset="UTF-8" Hello, I've been trying to make a stable XEN virtualization system with FreeBSD as Dom0. The problem is that at some point system panics (normal, non-stressful use-case), i have somehow found scripts and setups with witch i can provoke panic sooner. I have reported bug at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261059 with as detailed info as i could gather, including some of panic stack backtraces. In short - is anyone using ZFS + XEN + FreeBSD as Dom0 as a stable system? Can someone share his configuration or ideas, so i can try out? I've been experimenting with iSCSI as well, but it seems that at some setups the disk speed is really slow, and sometimes system panics as well, so i was going for the simplest setup. Other question, what could i do to help to solve the issues i have found in bug 261059? More verbosity flags, some other debugging files, locations? Thank you! --00000000000007764f05d9254461 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

I've been trying = to make a stable XEN virtualization system with FreeBSD as Dom0. The proble= m is that at some point system panics (normal, non-stressful use-case), i h= ave somehow found scripts and setups with witch i can provoke panic sooner.=

I have reported bug at https://bugs.freebsd.org/bugzi= lla/show_bug.cgi?id=3D261059 with as detailed info as i could gather, i= ncluding some of panic stack backtraces.

In sh= ort - is anyone using ZFS + XEN + FreeBSD as Dom0 as a stable system? Can s= omeone share his configuration or ideas, so i can try out? I've been ex= perimenting with iSCSI as well, but it seems that at some setups the disk s= peed is really slow, and sometimes system panics as well, so i was going fo= r the simplest setup.

Other question, what could i= do to help to solve the issues i have found in bug=20 261059? More verbosity flags, some other debugging files, locations?
<= div>
Thank you!
--00000000000007764f05d9254461-- From nobody Tue Mar 1 15:40:27 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D0DEA19EC5F5 for ; Tue, 1 Mar 2022 15:40:37 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7M043GY1z3nsw for ; Tue, 1 Mar 2022 15:40:36 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 221FeSqi028679 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 1 Mar 2022 07:40:28 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 221FeR4f028103; Tue, 1 Mar 2022 07:40:27 -0800 (PST) Message-Id: <202203011540.221FeR4f028103@nfbcal.org> From: Brian Buhrow Date: Tue, 1 Mar 2022 07:40:27 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Ze Dupsys , freebsd-xen@freebsd.org Subject: Re: ZFS + FreeBSD XEN dom0 panic Cc: buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Tue, 01 Mar 2022 07:40:28 -0800 (PST) X-Rspamd-Queue-Id: 4K7M043GY1z3nsw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.30 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-xen]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org hello. I've been running FreeBSD-12.1 and Freebsd-12.2 plus ZFS plus Xen with FreeBSD as dom0 without any stability issues for about 2 years now. I'm doing this on a number of systems, with a variety of NetBSD, FreeBSD and Linux as domU guests. I haven't looked at your bug details, but are you running FreeBSD-13? -thanks -Brian From nobody Wed Mar 2 08:57:37 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0C39319FFD1E for ; Wed, 2 Mar 2022 08:57:07 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7p022vyhz4SDl for ; Wed, 2 Mar 2022 08:57:06 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pl1-x629.google.com with SMTP id e13so1012367plh.3 for ; Wed, 02 Mar 2022 00:57:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=QWxv9pzutJU4f9DsRNXmnx5ubZZIqANCeuUyp3xpNvY=; b=etOxfO8jrULJXB0JYnukC+dvrFvHJpOewkYdIG1pGGWrxxfliaNsbDAC8QLaN8r3y7 V+lTz7DQOhO+fC6jrgpwWxiLpD56QqfFQCWCqMVJekAbfLNYlVhuFdQ+EStT9iaWoJ38 fksZQhRLfwOjSfl3NQ8NKVxSt3J2lMR9B6G0HOXULr/CYTrOMJ2LQJyj+c3k75WXAJkT OoTyuQEtnp/eRX/GTRQPz6Lu3cjCKfDVM6BX4Jfvpogxf4Na531GA5wg5ZRUEruOYhQY Uvy11JmqYPU6oogCbgLcOp5q/8O0Yjpnb17LMLdGxFReacLkcd+WViHJa28glDQAqQBn FVYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=QWxv9pzutJU4f9DsRNXmnx5ubZZIqANCeuUyp3xpNvY=; b=JGxLXqQ51M8PDYk6VPCDXwjDxu2xQPL5E/iBHsLig8ExZwuZ52QGSk5D9JzV16L1rj Evt/cwoBknStS3e6GgKMXWliR9OjdyU6ns2JFfls3sD5LWdXXjFQZpK4DdEzCooyAnzj hrjYDgj3oTg/AJuWbDQTnzWc4b4V0BWV2/7sv3RP560O0YTD6mLEsJVcXP1OmV/dEYVt yxjMl0+5OjJvrcOHwi9TvJCEthsOa9PXvFK/OxO1njR/Ewl8Fdv/rPK2LWOLL4UDxQrA qOrsoPwqqCO4v2hkeBBTcEI/V6fjKlYWNMSr8gQA5IKysfkSqrBThQxgDE3jsUMkOmFv mUBA== X-Gm-Message-State: AOAM53137PavAV6v/0dVtHgF8xL1dPIK9MGPrSI2cj4YFlAG9UlTB74H sCxlrFEJDMWW6y4taJD4VcRiQwu13pJ9aiPALUB4PkFRCUg= X-Google-Smtp-Source: ABdhPJyLnulvOMW1i8mmqiYTpGBmqtuR9tU5XyJis+0HQsIN57MutE9cS25QMubtcI2CTj3mi3332h+qD4NDVgXIR0s= X-Received: by 2002:a17:902:ec86:b0:151:9ca2:8ecb with SMTP id x6-20020a170902ec8600b001519ca28ecbmr374535plg.61.1646211425383; Wed, 02 Mar 2022 00:57:05 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <202203011540.221FeR4f028103@nfbcal.org> In-Reply-To: <202203011540.221FeR4f028103@nfbcal.org> From: Ze Dupsys Date: Wed, 2 Mar 2022 10:57:37 +0200 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: freebsd-xen@freebsd.org, buhrow@nfbcal.org Content-Type: multipart/alternative; boundary="0000000000003c3a7005d93877a8" X-Rspamd-Queue-Id: 4K7p022vyhz4SDl X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=etOxfO8j; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::629 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::629:from]; MLMMJ_DEST(0.00)[freebsd-xen]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --0000000000003c3a7005d93877a8 Content-Type: text/plain; charset="UTF-8" Hello, I started using XEN on one pre-production machine (with aim to use later in production) with 12.2, but since it experienced random crashes i updated to 13.0 in hope that errors might disappear. I do not know how detailed should i write, so that this email is not too long, but gives enough info. FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rules for NATting are used. Zpool is composed of 2 mirrored disks. There is a ZVOL volmode=dev for each VM and VM's jail that are attached as raw devices to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, with VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have bridge interfaces, where DomU's are connected depending on their "zone/network", those that have allowed outgoing connections are NATted by IPFW on specific physical NIC and IP. xen_cmdline="dom0_mem=6144M cpufreq=dom0-kernel dom0_max_vcpus=4 dom0=pvh console=vga,com1 com1=115200,8n1 guest_loglvl=all loglvl=all" Physical hardware is XEON CPU, ECC RAM 16G, 2x8TB HDD. DomU config, something like this: memory = 1024 vcpus=2 name = "sys-01" type = "hvm" boot = "dc" vif = [ 'vifname=xbr0p5,type=vif,mac=00:16:3E:01:63:05,bridge=xbr0' ] disk = [ 'backendtype=phy, format=raw, vdev=xvda, target=/dev/zvol/sys/vmdk/root/sys-01-root', 'backendtype=phy, format=raw, vdev=xvdb, target=/dev/zvol/sys/vmdk/root/sys-01-jail1', 'backendtype=phy, format=raw, vdev=xvdc, target=/dev/zvol/sys/vmdk/root/sys-01-jail2' .. more defs, if any .. ] vnc=1 vnclisten="0.0.0.0:X" usbdevice = "tablet" serial = "pty" When just started, overall system works, speeds are acceptable, load is not high so system is not under stress. The thing is that at some unexpected times i noticed that system reboots, i.e. when i create new ZFS volume in Dom0, or when i reboot DomU or do something in Dom0 which seems unrelated, sometimes it was that init 0 would reboot system, sometimes it shut it down. It somehow felt, that panics happen when there is HDD load. So i got somewhat similar machine for testing/lab env, 16G ECC, slower XEON, 2x2TB HDD and serial port and started to try to push that system to limits with various combinations, restricting RAM, CPUs, etc. The bug info contains combination, that seemed for me to be the fastest way of how to panic system. For XEN startup "vifname=" did not work as described in XEN user manual pages for default startup script, so i added "ifconfig name $vifname" in that script. The necessity for it was, that ipfw rules that required "via $ifname in", had to have specific NIC, but XEN by default each time was creating new NIC name depending on which name was free. This is not active on lab system, and it still crashes, so i do not think that problem cause is this. About history. I believe hardware is okay, since before XEN i was using FreeBSD 12.2 (upgraded incrementally from 12.0), ZFS + jails a lot, VNETs used were netgraph(VNET bridge and ethernet interfaces). What i loved about that setup was, clean output of ifconfig, since host had only bridge interface and virtual ethernet interfaces for jails came directly from that bridge. New jail creation was just "zfs clone", it did not take much space, snapshots for backups could be made, whole HDD space could be easily expanded/limited for each jail, due to ZFS capabilities. System was stable. The problem with that setup was, that if some jail started to misbehave badly it was hard to control overall system performance and behavioral characteristics, i tried rctl, but jails could misbehave in new unexpected bad ways (exhausting RAM, process count, CPU load, HDD load, opening too many network sockets, etc. If OOM killer started to kill processes, it was impossible to control which process/jail should get killed first, which should be kept), so for me it seemed that virtualization is better way to solve that. I.e. to have a system VM, that has DNS, Web gateway, etc., and lower priority VMs, that could crash if misbehaving. I like XEN architecture in general, and i would like to use FreeBSD as Dom0, if possible; due to ZFS, knowledge and good history of OS stability. Since ZFS dataset can not be passed through to DomU, my idea was to use ZVOLs and UFS within VM, then i could snapshot those ZVOLs for backups, DomU could growfs when necessary. Somewhat less convenient as for jail architecture, but still, good enough. My first attempt was to keep netgraph jails in Dom0, but it turned out bad. Almost every time system panic happened when jail was started/stopped. Not first jail, but 5th+, panic-ed system with high probability. So i started to use epairs instead. It was less unstable, but still crashed from time to time. Now there are no jails, and still. I tried different ideas, to pass through whole HDD as raw in DomU-iscsi and use ctld on Dom0 to provide disks for other DomUs, HDD speed was bad, but system still crashed, i tried raw files on ZFS datasets, speeds seemed close to ZVOLs actually, but system still crashed. So now i was starting to wonder, what configurations do people use successfully? What have i missed? On Tue, Mar 1, 2022 at 5:40 PM Brian Buhrow wrote: > hello. I've been running FreeBSD-12.1 and Freebsd-12.2 plus ZFS > plus Xen with FreeBSD as > dom0 without any stability issues for about 2 years now. I'm doing this > on a number of > systems, with a variety of NetBSD, FreeBSD and Linux as domU guests. I > haven't looked at your > bug details, but are you running FreeBSD-13? > -thanks > -Brian > > --0000000000003c3a7005d93877a8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

I started using XEN o= n one pre-production machine (with aim to use later in production) with 12.= 2, but since it experienced random crashes i updated to 13.0 in hope that e= rrors might disappear.

I do not know how detailed = should i write, so that this email is not too long, but gives enough info.<= br>

FreeBSD Dom0 is installed on ZFS, somewhat bas= ic install, IPFW and rules for NATting are used. Zpool is composed of 2 mir= rored disks. There is a ZVOL volmode=3Ddev for each VM and VM's jail th= at are attached as raw devices to DomU. At the moment DomUs contain FreeBSD= , some 12.0 to 13.0, UFS, with VNET jails, epairs all bridged to DomU's= xn0 interface. On Dom0 i have bridge interfaces, where DomU's are conn= ected depending on their "zone/network", those that have allowed = outgoing connections are NATted by IPFW on specific physical NIC and IP.

xen_cmdline=3D"dom0_mem=3D6144M cpufreq=3Ddom0-= kernel dom0_max_vcpus=3D4 dom0=3Dpvh console=3Dvga,com1 com1=3D115200,8n1 g= uest_loglvl=3Dall loglvl=3Dall"

Physical hard= ware is XEON CPU, ECC RAM 16G, 2x8TB HDD.

DomU con= fig, something like this:
memory =3D 1024
vcpus=3D2
name = =3D "sys-01"

type =3D "hvm"
boot =3D "dc= "

vif =3D [ 'vifname=3Dxbr0p5,type=3Dvif,mac=3D00:16:3E:01:= 63:05,bridge=3Dxbr0' ]
disk =3D [ 'backendtype=3Dphy, format=3Dr= aw, vdev=3Dxvda, target=3D/dev/zvol/sys/vmdk/root/sys-01-root',
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'backendtype=3Dphy, format=3Draw, vdev= =3Dxvdb, target=3D/dev/zvol/sys/vmdk/root/sys-01-jail1',
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0'backendtype=3Dphy, format=3Draw, vdev=3Dxvdc, = target=3D/dev/zvol/sys/vmdk/root/sys-01-jail2'
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .. more defs, if any ..
= =C2=A0 =C2=A0 =C2=A0=C2=A0 ]

vnc=3D1
vnclisten=3D"0.0.0.0:X&= quot;
usbdevice =3D "tablet"
serial =3D "pty"
=


When just started, overall system = works, speeds are acceptable, load is not high so system is not under stres= s. The thing is that at some unexpected times i noticed that system reboots= , i.e. when i create new ZFS volume in Dom0, or when i reboot DomU or do so= mething in Dom0 which seems unrelated, sometimes it was that init 0 would r= eboot system, sometimes it shut it down. It somehow felt, that panics happe= n when there is HDD load. So i got somewhat similar machine for testing/lab= env, 16G ECC, slower XEON, 2x2TB HDD and serial port and started to try to= push that system to limits with various combinations, restricting RAM, CPU= s, etc. The bug info contains combination, that seemed for me to be the fas= test way of how to panic system.

For XEN start= up "vifname=3D" did not work as described in XEN user manual page= s for default startup script, so i added "ifconfig name $vifname"= in that script. The necessity for it was, that ipfw rules that required &q= uot;via $ifname in", had to have specific NIC, but XEN by default each= time was creating new NIC name depending on which name was free. This is n= ot active on lab system, and it still crashes, so i do not think that probl= em cause is this.


About history= .
I believe hardware is okay, since before XEN i was using FreeBS= D 12.2 (upgraded incrementally from 12.0), ZFS=C2=A0+ jails a lot, VNETs us= ed were netgraph(VNET bridge and ethernet interfaces). What i loved about t= hat setup was, clean output of ifconfig, since host had only bridge interfa= ce and virtual ethernet interfaces for jails came directly from that bridge= . New jail creation was just "zfs clone", it did not take much sp= ace, snapshots for backups could be made, whole HDD space could be easily e= xpanded/limited for each jail, due to ZFS capabilities. System was stable. = The problem with that setup was, that if some jail started to misbehave bad= ly it was hard to control overall system performance and behavioral charact= eristics, i tried rctl, but jails could misbehave in new unexpected bad way= s (exhausting RAM, process count, CPU load, HDD load, opening too many netw= ork sockets, etc. If OOM killer started to kill processes, it was impossibl= e to control which process/jail should get killed first, which should be ke= pt), so for me it seemed that virtualization is better way to solve that. I= .e. to have a system VM, that has DNS, Web gateway, etc., and lower priorit= y VMs, that could crash if misbehaving. I like XEN architecture in general,= and i would like to use FreeBSD as Dom0, if possible; due to ZFS, knowledg= e and good history of OS stability.

Since ZFS data= set can not be passed through to DomU, my idea was to use ZVOLs and UFS wit= hin VM, then i could snapshot those ZVOLs for backups, DomU could growfs wh= en necessary. Somewhat less convenient as for jail architecture, but still,= good enough.

My first attempt was to keep net= graph jails in Dom0, but it turned out bad. Almost every time system panic = happened when jail was started/stopped. Not first jail, but 5th+, panic-ed = system with high probability. So i started to use epairs instead. It was le= ss unstable, but still crashed from time to time. Now there are no jails, a= nd still.

I tried different ideas, to pass through= whole HDD as raw in DomU-iscsi and use ctld on Dom0 to provide disks for o= ther DomUs, HDD speed was bad, but system still crashed, i tried raw files = on ZFS datasets, speeds seemed close to ZVOLs actually, but system still cr= ashed. So now i was starting to wonder, what configurations do people use s= uccessfully? What have i missed?


On Tue, Mar 1, 202= 2 at 5:40 PM Brian Buhrow <buhrow@n= fbcal.org> wrote:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 hello.=C2=A0 I've been running F= reeBSD-12.1 and Freebsd-12.2 plus ZFS plus Xen with FreeBSD as
dom0=C2=A0 without any stability issues for about 2 years now.=C2=A0 I'= m doing this on a number of
systems, with a variety of=C2=A0 NetBSD, FreeBSD and Linux as domU guests.= =C2=A0 I haven't looked at your
bug details, but are you running FreeBSD-13?
-thanks
-Brian

--0000000000003c3a7005d93877a8-- From nobody Wed Mar 2 13:57:12 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4B37619ED953 for ; Wed, 2 Mar 2022 13:57:30 +0000 (UTC) (envelope-from prvs=0530a50c4=roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7wfd0wLlz52MV for ; Wed, 2 Mar 2022 13:57:29 +0000 (UTC) (envelope-from prvs=0530a50c4=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1646229449; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=V6cL1ewHKKCqY5LNPUEBwGw6g6/GzIhBP4PA1PXw0tU=; b=eKwlf1BEl2jh2eDiQtEPZmIj1/yfro81MdG3ux7mFV7T7y31NVG68P3i D5DqgMzpZaJ2h22K0fo9TrCpDst24sQidIxAdNbJwwiPmmTfH7wwpoVYj 5gtI/g37083Acz3UnQ+Vq9xnS7/JaSF4UQuLBH//0JRVXRS//3Wx/SM6R 8=; X-SBRS: 5.1 X-MesageID: 64756999 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:ekYKzqmcD8Zz4fljhDEzJOjo5gyQJkRdPkR7XQ2eYbSJt1+Wr1Gzt xIaC27TOq2PajT3c94kaou+oEhT656Eyt43TABr+Cg3QyMWpZLJC+rCIxarNUt+DCFioGGLT Sk6QoOdRCzhZiaE/n9BCpC48T8kk/vgqoPUUIYoAAgoLeNfYHpn2EoLd9IR2NYy24DiW1nV4 rsenuWEULOb828sWo4rw/rrRCNH5JwebxtB4zTSzdgS1LPvvyF94KA3fMldHFOhKmVgJcaoR v6r8V2M1jixEyHBqD+Suu2TnkUiGtY+NOUV45Zcc/DKbhNq/kTe3kunXRa1hIg+ZzihxrhMJ NtxWZOYUiIqYvXOmv0keFoJSwNRbJB+2Zj1Li3q2SCT5xWun3rExvxvCAc9PJEC+/YxCmZLn RAaAGlTNFbZ3bvwme/lDLk37iggBJCD0Ic3oHZvwCufFf87aZvCX7/L9ZlT2zJYasVmQ6qBO pRIOGsHgBLoezx2A3gwC5cEo+L3vlv4KD4Ag1HInP9ii4TU5FMoi+W8WDbPQfSAWMVPmVmwo GzM9G39RBYAO7S3xyCI/3i3iqrfkCXTVpgfGbej+rhtmlL77n4eBAAOTkqgifa8g0+6HdlYL iQpFjEG9PZoshbxF5+kAkP+8CXsUgMgt8R4CMg5tFmPyLPtyEXIXkorHwN5SfsUu5pjLdA17 WOhk9TsDD1plbSaT3OB67uZxQ+P1TgpwXwqPnFdE1ZcizX3iMRq10+UEI4/eEKgpoCtQVnNL ya2QD/Sbln5peoCzO2F8F/OmFpATbCZH1dutm07so9Ihz6VhbJJhaT1sTA3Dt4ade51q2VtW lBewaByC8hUUPmweNSlGrllIV1Qz6/t3MfgqVBuBYI90D+m5mSue4tdiBknehs3a5hcIW63O R6J0e+02HO1FCHwBUOQS9jsY/nGMIC6TYi1PhwqRoAmjmdNmP+vo3g1OB/4M5HFm0kwi6AvU ap3gu73ZUv2/Z9PlWLsL89EiOdD7nlnmQv7GMCqpzz6gOH2TCPEFt843K6mM7lRAFWs+16Or b6y9qKiln1ibQEJSnKPoN5LcAtTdiBT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5 XazRglTz1+XuJENAVjihqxLAF83YatCkA== IronPort-HdrOrdr: A9a23:3eg78a4osg98Xcom4wPXwSSBI+orL9Y04lQ7vn2ZFiY7TiXIra yTdaoguCMc6AxxZJkh8erwX5VoZUmsj6KdgLNhRotKOTOJhILGFvAB0WKP+UyEJ8S6zJ8h6U 4CSdkBNDSTNykCsS+S2mDReLxBsbq6GeKT9J/jJh9WPH5XgspbnmFE42igYylLrF4sP+tEKH PQ3LsPmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZUbxp/hZMZtU TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczLgBkF5MWUrHo6mt jFpBkte+x19nPqZ2mw5SDg3gHxuQxen0PK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL 9L93jxjesaMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO21GkeKp ghMCjg3ocWTbvDBEqp/lWHgebcFEjbJy32DXTr4aeuontrdHMQ9Tps+CVQpAZDyHsHceg12w 31CNUYqFhwdL5kUUtcPpZ3fSLlMB26ffrzWFjiU2gPUpt3fk7wlw== X-IronPort-AV: E=Sophos;i="5.90,149,1643691600"; d="scan'208";a="64756999" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nvd+14Iy1FRiJ5bhfCqZ6g6iPC24MS8jQMuUNRTO59M40Adidyrnnhi3/qtkM2w4WCxucmwnuIgSBXVdEbMqfJZk2aOL978stpP/tBk/uZ/y87k0kKTceBAYvjDiZXYuw1leG6CNWmOsrBVUGxIY04tMmIoTuTap3Z5DSuTT/x/yvo2dKLgKecU82oVlqNNJfJKVR4l5oEN1oswknZKpPwm2RoISQG88F0qYR7Oep/QvKtBPWMAgHtYG+m4kRiZDs0r6Id1rIoBWqZasPW/YIW9WQOT/MWkVB8EPrNV8T6nTPephT5MaaRSrwJowXxO1XOMPKTWAhF8vipd1rjSlpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sF4YLlDfeMxNEfZ4py2VdNjsr0DF+89ztOd7wZWDvlg=; b=eU71q0rZBSEivNjpS02x0pisq1fUZQjBwhy61BRIXEx5EtHx40AK5POT9X5Mwptf2nxFkJARGzSq9C3jWfn52jmzVCKmWCm2CNHrgHa5o1+PK8wzw5d1iFJ8tZnsDtYZnhWLsrKybITj7eJZpESWrY7BiZDQ3uQfQia46LdeKNKBu4YBmaw9zw6HXifBiINU3dR5Xrb3totLnJBH6uFVq3j63DOMvixEY3D2WQUfaXuGhOAbERQPYZFD6AzG7MICpH9VmxhwJPKEVGC144TUaH6ortnTCM3PBF+vzGBWs9RR5YXDKvRIXziHL/P5Tw8NZ8gDD0Y0bKNx+qJlVZrAig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sF4YLlDfeMxNEfZ4py2VdNjsr0DF+89ztOd7wZWDvlg=; b=u/D3RKEik2w0eijdnu6rCfVhyG9St4KFqmtVI8Umle/JRIB+GJvYc/GSlGB0UXrqumKR+cg88AdJ/Fpk43bS4WW9MkfcdV23LnY1qRq4R4gBRiV/8uC6a++B1aFfspF3XhjckOhAnICljQQ6I0AcuNmG8iAKSsxeqDNYPWE3MdM= Date: Wed, 2 Mar 2022 14:57:12 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <202203011540.221FeR4f028103@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO4P123CA0242.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a7::13) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6805a47b-fc87-40b4-8267-08d9fc548ffe X-MS-TrafficTypeDiagnostic: DM6PR03MB4026:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Mf5UKNEW6jsrSuiM1g1NSZYyDsWsBJhauH00G0slMxPjIRCp+Dg9wYiseiVnnnR2UzdiwXA1XvgzEYz8hit5Nlz9yAaa3V509AknXjEpZRtx7AqCzOFC4C9xB/weyF5FGaSlOSXjr0ZZHKDDm/E0PHLHRsKw5dwSLAkFx8KIMlujOVXo6iadZ8mqSGY7kzF65+bG0koxqZah0YqfOJ06mu/tzSE8yp6en6pdKHcJwBEiHO8uZXEnCvWfr+IzfUG2fLldi6ksMhb4X2qLE70cEK0008/bBVsecRwrKrYwwP+ouIhd/WFdaMCyjLp5QdF32TID99NnA8lT3oaeCubj6jt6gl1eMDgn11DzqTpWNDUfqdf0zHyBszxIufaF+tQqIe0oJ8OZLNs10B6K6zDFtWfYjRP1y6e7lkj8tdV8FZS5kAyskQ7hOWlG0NrdLVOk9hWS47jbyo6fFhqIMShzt3ewVkWoDAVDgqGTYPCc1xXaYqYWxFtRHmJLWMLGs2DDSmYWVP0LXaiPwrx4qi88NLIgMIVydMPwBOkapHXVwe/EA+XzZaPT/5YXAVThmX98MY8sif/lhTI31+hIcDanWDLgh+Qly3gopEm3n/LKmc2nWpP/zsllhPu3IVZnciriMP0zA1k+XgEpMNk12Hp9cA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(316002)(6512007)(9686003)(5660300002)(38100700002)(8936002)(86362001)(8676002)(66556008)(66476007)(66946007)(4326008)(2906002)(186003)(26005)(82960400001)(85182001)(6506007)(6486002)(6666004)(508600001)(6916009)(83380400001)(33716001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YjJxeFNuTll0VFpSRFd3SXVtRzJCdkdBTE04S0QvOGJmYlJid3Jockk4U3JT?= =?utf-8?B?MjVlUGlVbkFsdVZMOTVJRmFGUVFGWXhXK0phZ09JSDBlZVIvYUwwbkNYSS9s?= =?utf-8?B?eVpVUXkzeTNjdVBIY2Q0NmxQWDJnSW5QQlZMTy9EL1A2ZStHbUE2Y1N0N1ZH?= =?utf-8?B?WlRUWlkxWnlnOGkrcjQ5NThFMU9tdnhWaG80eHR3dCtqbHZzekJBWUR1Wjd0?= =?utf-8?B?a2pTcUhzZjA4QityQ3p5SnJHTDRRdjM4bUVDM29ndS9NUW5sRXZZTElPQysr?= =?utf-8?B?QVpkazJpcTlRQjA4djI2dW1VMEljTkVTNVRZZGV0c3hwenFQTWJtVVRBMWp3?= =?utf-8?B?RkpKSWRYUll1cWRFdFNaZ0NOUDIzcElHUmVhamR0MStKMzJ6K3lmdVJIaGl5?= =?utf-8?B?a1JSVXk3TitSODc2TWNkS2xpOWxTa3diVnl0YVRwMGNWbDFkVm9vaFVOVURE?= =?utf-8?B?VjYxZTgvMDFnN1FHMkpyS2pVNUQySUtCTnVCWktJdkRNRlczOXoyVWtGT21X?= =?utf-8?B?aUVKS2RaMTJjYTJjZC96ZkQrbDltaXY0bFVVRE13dnM1SkpURXFmQUFZRXhW?= =?utf-8?B?U2N4ZmRyWHgzdkRNVGtpOVQyeTN3VUNKZlZBalBTQUNlaVcrYmJSd0E0N29m?= =?utf-8?B?MXByREltcmhueVptNUpSRWc1dFlka252NWh5MmtUaDV2b2JTeHlnajNKdVlG?= =?utf-8?B?WE1vbmpJOGpySE9iRHY3encxSUNoZXJ5QzJrTGlBQ25rZzRiNlJ3QTNKRlcz?= =?utf-8?B?MlhSaUxRTThKdlVVWDNQWTNmQUhPbHY2RnlEUTdab2tOUkxmK1IwTzBqMXho?= =?utf-8?B?ODNDWEovN3B0OXI4T1IwSHUrQVZKclFubWxvNExuUU9FSmFMYzJqQ3JLYnd6?= =?utf-8?B?cW80VCtUQUJyc3Y4UUJZZCtPSUR2TGJpT2dCUUdndlNQYzFiYXlXWFNrTDlK?= =?utf-8?B?ckI4ZE4vekYrdGhqTnV5UWhtSEludDM5S0JhQzMwNmRNZEhxVHBjRGd4UDk5?= =?utf-8?B?OTlCNWxRZWVkd3hIeC9FdkxZY0hqdEIwcjludFdvdElBam5YYUkyUlMwaWg5?= =?utf-8?B?NXRsWjNaVjlGVUsvNXlIcWUzakREU01VcURzN1E4Y2huZDRBdFd1SGtLc0hp?= =?utf-8?B?YU1PNDM0UGdiSUJjMVY5cThNa01SUlE3aFpLcnFzSzhxWWJRUmsyTU50STgz?= =?utf-8?B?RE9RV0lCbnlldGdEK1ROVXBZbElQeWwxZzJBWWVQekJCRmxwT2lNelNPcGRu?= =?utf-8?B?SVhxMjFrNSt4dGRkMmN0MWE3UmYwdHFqNmtmOFdneGM5U0dJVmM1Mm1TRnAy?= =?utf-8?B?K0NOOEx3Wlp3cHJNc3RsV0UxSHVBTjdXQ2N5VXFCSENhNmVES3F0NGlJRWJj?= =?utf-8?B?NDRHOW1NZXpYSzNiWGpsQmJJaEYwZmFma240WU1jNklkbVB0eFhQKzZyY1JT?= =?utf-8?B?WHBqbURKOVV3cnQ1dWFsRGE1ZlZzeWdXRnZJMWdQVXlUc0VBd1Q5ZWdWYjUv?= =?utf-8?B?dzZXS2MxY2pDYkVsUTdxa3AwOE1KZEhMbTc1RzRibU5zVGlqWDJOTU8yc1VX?= =?utf-8?B?WHZtRGVEU09jaW9iRGMrRGZORmE4a1pxcURxYVJxR3dyN09NWmcvUHZVN0M0?= =?utf-8?B?OGZyOWtDbVNMM001UDJueDg5SStkWjc1TGNScXV4SlJGYnllRjBDWjJHRUVI?= =?utf-8?B?L3lnSjhDdVpqb0toOXpocCt3dU5GQURQVGdaRG0ybTBwcTJzWldoZ3MreU1B?= =?utf-8?B?aXcxd1VOQVRYOXNnZmRzWkNLTzUyNDR5d1EzdWs0OHgrRitLbDhiaFBlQ0VX?= =?utf-8?B?NmZGOVhoajlFU05ZQUJ2QVdJOWxNTGJudTJVVHVxZUxyc0x6bnI3c2dnVW9t?= =?utf-8?B?UUZzblh1OG1XUThScW5mbitnNy8wcVZGNXpGMDkrOTJheUNIT2pFekNGbDZY?= =?utf-8?B?VnpqNVlBd0c0aW9EcndhSTRBanViNVZvaUxndzY0MWVIRXRqbkFTQVRzRlE1?= =?utf-8?B?VmhOM21BWHdIaitGZ2lRUE95cWl3bHdUMjhhRzdITjIwNG1rVkQxeHFuN1pM?= =?utf-8?B?UjVadkVMWnVxbG5CbHg0Q2ZKZDhqZ2F5YkJPR1dMcXBoSVFCc3hpNkptTWtQ?= =?utf-8?B?ZU9sZkIxc296SWVHM3VyTittblJUdXcrRWl6USsyK09PaC8vTEk0aE4rMGRa?= =?utf-8?Q?LsBml+zmbWUN3g+gl+0lWyM=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6805a47b-fc87-40b4-8267-08d9fc548ffe X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2022 13:57:17.4628 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wLrLFm2n6eFga8ywXSoJU25XmsSIjaoJwq/24++3nxauyEnQFc0x/G/jJmTqc/4l3+qXEZ1dujJpnXfPTS45bw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4026 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4K7wfd0wLlz52MV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=eKwlf1BE; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b="u/D3RKEi"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=0530a50c4=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=0530a50c4=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=0530a50c4=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.155.168:from]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 02, 2022 at 10:57:37AM +0200, Ze Dupsys wrote: > Hello, > > I started using XEN on one pre-production machine (with aim to use later in > production) with 12.2, but since it experienced random crashes i updated to > 13.0 in hope that errors might disappear. > > I do not know how detailed should i write, so that this email is not too > long, but gives enough info. > > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rules > for NATting are used. Zpool is composed of 2 mirrored disks. There is a > ZVOL volmode=dev for each VM and VM's jail that are attached as raw devices > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, with > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have > bridge interfaces, where DomU's are connected depending on their > "zone/network", those that have allowed outgoing connections are NATted by > IPFW on specific physical NIC and IP. So from the traces on the ticket: panic: pmap_growkernel: no memory to grow kernel cpuid = 0 time = 1646123072 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff81073eed at pmap_growkernel+0x27d #4 0xffffffff80f2dae8 at vm_map_insert+0x248 #5 0xffffffff80f30249 at vm_map_find+0x549 #6 0xffffffff80f2bf76 at kmem_init+0x226 #7 0xffffffff80c73341 at vmem_xalloc+0xcb1 #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab #9 0xffffffff80f2bfce at kmem_init+0x27e #10 0xffffffff80c73341 at vmem_xalloc+0xcb1 #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab #12 0xffffffff80c72646 at vmem_alloc+0x46 #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96 #14 0xffffffff80f21a2a at uma_prealloc+0x23a #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e #16 0xffffffff80f1f6af at uma_set_align+0x8f #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22 Uptime: 4m9s Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x22710028 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80c45892 stack pointer = 0x28:0xfffffe0096600930 frame pointer = 0x28:0xfffffe0096600930 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1496 (devmatch) trap number = 12 panic: page fault cpuid = 0 time = 1646123791 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff8108b1a7 at trap+0xbc7 #4 0xffffffff8108b1ff at trap+0xc1f #5 0xffffffff8108a85d at trap+0x27d #6 0xffffffff81061b18 at calltrap+0x8 #7 0xffffffff80c62011 at rman_is_region_manager+0x241 #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 #9 0xffffffff80c1949c at kernel_sysctl+0x43c #10 0xffffffff80c19b13 at userland_sysctl+0x173 #11 0xffffffff80c1995f at sys___sysctl+0x5f #12 0xffffffff8108baac at amd64_syscall+0x10c #13 0xffffffff8106243e at Xfast_syscall+0xfe Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x68 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff824a599d stack pointer = 0x28:0xfffffe00b1e27910 frame pointer = 0x28:0xfffffe00b1e279b0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (xbbd7 taskq) trap number = 12 panic: page fault cpuid = 1 time = 1646122723 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff8108b1a7 at trap+0xbc7 #4 0xffffffff8108b1ff at trap+0xc1f #5 0xffffffff8108a85d at trap+0x27d #6 0xffffffff81061b18 at calltrap+0x8 #7 0xffffffff8248935a at dmu_read+0x2a #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac #13 0xffffffff80bc7e3e at fork_exit+0x7e #14 0xffffffff81062b9e at fork_trampoline+0xe Uptime: 1h44m10s This all look to me like Out of Memory conditions, can you check with `top` whats going on with your memory? Might also be helpful to record periodic calls to `vmstat -m | sort -k 2 -r` to try to figure out what's using so much memory. Regards, Roger. From nobody Wed Mar 2 17:05:22 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DBEEC19F5476 for ; Wed, 2 Mar 2022 17:05:34 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K80qd1S7cz4Vb9 for ; Wed, 2 Mar 2022 17:05:32 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 222H5Muv016118 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 2 Mar 2022 09:05:23 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 222H5MVm026787; Wed, 2 Mar 2022 09:05:22 -0800 (PST) Message-Id: <202203021705.222H5MVm026787@nfbcal.org> From: Brian Buhrow Date: Wed, 2 Mar 2022 09:05:22 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Ze Dupsys , freebsd-xen@freebsd.org Subject: Re: ZFS + FreeBSD XEN dom0 panic Cc: buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 02 Mar 2022 09:05:23 -0800 (PST) X-Rspamd-Queue-Id: 4K80qd1S7cz4Vb9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.30 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-xen]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org hello. One difference between my systems and yurs, though I don't think that's the problem, is that I'm not running a firewall on the dom0 itself. The dom0 runs on a protected vlan with respect to the external network and the domu's are connected to bridges that are directly connected to the external network. I have one system where the customer wants the pfsense system runing, so pfsense runs as a domu on this system, connected to an internal "private" bridge and the public bridge, doing all the firewalling between them. In this way, the FreeBSD dom0 is only doing ZFS, simple ip routing and Xen management. If I had to wager a guess as to your trouble, it's that you don't have enough memory on your dom0. ZFS is a memory hog and I can't imagine getting away with anything less than 8G on the dom0 with FreeBSD-12 and ZFS. I'm using 8G for the dom0 on the system I'm writing from and it is quite stable, but, then again, I'm not doing as much with the dom0 as you are. I too am using zvols as disks for the domu's, but I've not been trying to make zfs snapshots from them. obvious question, but I'll ask it anyway, you're not trying to make snapshots of the zvols while the domu's on top of them are running, are you? I would imagine that would not give you good images, but I wouldn't expect it to panic the dom0 either. However, it wil stretch your meager memory resources even further. Have you been able to get a panic message or does the system just spontaneously reboot? If it just reboots, then, again, I think you are having a memory shortage. My suggestion is to try giving the dom0 8G of RAM and then for the domU's, use the balloon driver to oversubscribe the remaining memory for the domu's. Of course, the best course of action is to see if you can put more memory in this system; 16GB just isn't that much when you're trying to run Xen plus a few domu's, especially on top of ZFS. If yu can get a panic message or a crash dump, that would be helpful in figuring out more accurately what's going on. Another thought, since you were getting some crashes when running jails with xen, is to get memtest86 running on the raw machine and let it run for 3 or 4 days. If you don't get any memory errors, then I think you can be pretty sure it's not a hardware problem. If, however, you get any errors at all with that test, then I think it's a good bet you hav a hardware issue. -thanks -Brian From nobody Wed Mar 2 17:26:18 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D213519F89CC for ; Wed, 2 Mar 2022 17:25:49 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K81H10Vwkz4XWX for ; Wed, 2 Mar 2022 17:25:49 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pl1-x630.google.com with SMTP id z2so2139845plg.8 for ; Wed, 02 Mar 2022 09:25:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0B1TttZCtrKyN6S0dLLr5yh5o8OGkx4VXXA2Ej13s/E=; b=Szeh/nA4oCf9087cLdLI9IGOpss3/QpUsfrqS4Uq81eKHUVv7YWCzfZKoQUyI1a20Y dOL7sS5DvlI77tex5GjnObTZ48nTgAGgimOtzZZRu2EjxJCEUeI6EDQJ2sL1vXmwooYS fYXCPF2iE6HljopKkYbudWNetZxlfNax47iitv9d3Wb6ckXfr0OSiZlD0i2KFV2pzklT dm+6sKP16fSWfnqLcHSKo1LYPM4aGo7kjN7rJTPB/RMY/AbBmGv6xtD3suDo3uZ/LqBT gfodUyRn+56JzzyprruvN4ZCIhoWUNVYyBe+GNZH1iwkpBw95b9evQ5rNleRU8/yfEgy GQXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0B1TttZCtrKyN6S0dLLr5yh5o8OGkx4VXXA2Ej13s/E=; b=5ITUbV/j5F5SuE1dDAFBlKp6jO6OqYBBecfje0jz+1Iy2Y/Fq5Xduz3EXdaBzPq9Fa 5TvWt6g14R34TrXOlrHEoX7/RMQb5H3CxWgYQ9gkj6GtvN1xkv8y29a5rQICSYTfNfKT ZIXo8Cylg4NIEXckZ6pTdM48inKxTGIY8CqJcBsM+5NYYdOUagRWOeyHPogcKSUWANsT i5+f7TbNkGmDl5DUtFvKMG8d2U/rRdnox386YZlDg3crMLquxBlk6azMmkVKYiIWm7yP Z6K3rUVagK8V/TN1XBYOcDTuKIqRT5rYRyU9D//Y9dDiVPfbeIwqi8IGLbwSi+ZX074f RCNw== X-Gm-Message-State: AOAM531awtbOmKcNrQlX3sPiJ5eWA3exz8SpBySxxRMzy4lcARqRwnVa epRJpjSTWmU3ueWiqf0h6/IvZTCnSiz8w9uJ5Q8= X-Google-Smtp-Source: ABdhPJwaMUHTPQxs7AV/ZKO9oDBuh3X+aIoOUztrwEuwkGRuH9veKu2E7O1dhlFtIga3pRzYw5RRQxcHgOsl7rgCxPs= X-Received: by 2002:a17:902:bc83:b0:149:b26a:b9c8 with SMTP id bb3-20020a170902bc8300b00149b26ab9c8mr31746402plb.143.1646241948008; Wed, 02 Mar 2022 09:25:48 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <202203011540.221FeR4f028103@nfbcal.org> In-Reply-To: From: Ze Dupsys Date: Wed, 2 Mar 2022 19:26:18 +0200 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org Content-Type: multipart/mixed; boundary="00000000000086a47505d93f92ea" X-Rspamd-Queue-Id: 4K81H10Vwkz4XWX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="Szeh/nA4"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::630 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_ATTACHMENT(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.995]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::630:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --00000000000086a47505d93f92ea Content-Type: multipart/alternative; boundary="00000000000086a47305d93f92e8" --00000000000086a47305d93f92e8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Today managed to crash lab Dom0 with: xen_cmdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 console=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_con= sole=3D1 reboot=3Dno" I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in attachment, panic was with the same fingerprint as the one with "rman_is_region_manager" line already reported. The scripts i ran in parallel generally were the same as attached in bug report, just a bit modified. 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL created dd if=3D/dev/zero) 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write 3GB in it's /tmp, restart DomU, removes /tmp, repeat) 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 disks attached, turn off DomU, repeat) 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. Around dom id 108, system started to behave suspiciously, xl list showed DomUs created, but they did not really start up, script timeout-ed for ssh connection, no vnc. When i did xl destroy manually, and xl create, system panic happened. I have log files for all serial output, if there is anything useful, i can provide. On disk log files seems to loose latest messages due to crash. On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 = wrote: > On Wed, Mar 02, 2022 at 10:57:37AM +0200, Ze Dupsys wrote: > > Hello, > > > > I started using XEN on one pre-production machine (with aim to use late= r > in > > production) with 12.2, but since it experienced random crashes i update= d > to > > 13.0 in hope that errors might disappear. > > > > I do not know how detailed should i write, so that this email is not to= o > > long, but gives enough info. > > > > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rule= s > > for NATting are used. Zpool is composed of 2 mirrored disks. There is a > > ZVOL volmode=3Ddev for each VM and VM's jail that are attached as raw > devices > > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, > with > > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have > > bridge interfaces, where DomU's are connected depending on their > > "zone/network", those that have allowed outgoing connections are NATted > by > > IPFW on specific physical NIC and IP. > > So from the traces on the ticket: > > panic: pmap_growkernel: no memory to grow kernel > cpuid =3D 0 > time =3D 1646123072 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff81073eed at pmap_growkernel+0x27d > #4 0xffffffff80f2dae8 at vm_map_insert+0x248 > #5 0xffffffff80f30249 at vm_map_find+0x549 > #6 0xffffffff80f2bf76 at kmem_init+0x226 > #7 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #9 0xffffffff80f2bfce at kmem_init+0x27e > #10 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #12 0xffffffff80c72646 at vmem_alloc+0x46 > #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96 > #14 0xffffffff80f21a2a at uma_prealloc+0x23a > #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e > #16 0xffffffff80f1f6af at uma_set_align+0x8f > #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22 > Uptime: 4m9s > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =3D 0x22710028 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80c45892 > stack pointer =3D 0x28:0xfffffe0096600930 > frame pointer =3D 0x28:0xfffffe0096600930 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 1496 (devmatch) > trap number =3D 12 > panic: page fault > cpuid =3D 0 > time =3D 1646123791 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff80c62011 at rman_is_region_manager+0x241 > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #9 0xffffffff80c1949c at kernel_sysctl+0x43c > #10 0xffffffff80c19b13 at userland_sysctl+0x173 > #11 0xffffffff80c1995f at sys___sysctl+0x5f > #12 0xffffffff8108baac at amd64_syscall+0x10c > #13 0xffffffff8106243e at Xfast_syscall+0xfe > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 1; apic id =3D 02 > fault virtual address =3D 0x68 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff824a599d > stack pointer =3D 0x28:0xfffffe00b1e27910 > frame pointer =3D 0x28:0xfffffe00b1e279b0 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 0 (xbbd7 taskq) > trap number =3D 12 > panic: page fault > cpuid =3D 1 > time =3D 1646122723 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff8248935a at dmu_read+0x2a > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac > #13 0xffffffff80bc7e3e at fork_exit+0x7e > #14 0xffffffff81062b9e at fork_trampoline+0xe > Uptime: 1h44m10s > > This all look to me like Out of Memory conditions, can you check with > `top` whats going on with your memory? > > Might also be helpful to record periodic calls to `vmstat -m | sort -k > 2 -r` to try to figure out what's using so much memory. > > Regards, Roger. > --00000000000086a47305d93f92e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Today managed to crash lab Dom0 with:
xen_c= mdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 c= onsole=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_cons= ole=3D1 reboot=3Dno"

I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in at= tachment, panic was with the same fingerprint as the one with "rman_is= _region_manager" line already reported.

T= he scripts i ran in parallel generally were the same as attached in bug rep= ort, just a bit modified.
1) ./libexec.sh zfs_volstress_fast_4g (= this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL = created dd if=3D/dev/zero)
2)=C2=A0 ./test_vm1_zvol_3gb.sh (= this loops commands: start first DomU, write 3GB in it's /tmp, restart = DomU, removes /tmp, repeat)
3) ./test_vm2_zvol_5_on_off.sh (this = loops: start second DomU, which has 5 disks attached, turn off DomU, repeat= )
4) monitoring, sleep 120 seconds, print vmstat | sort in se= rial output.

Around dom id 108, system started= to behave suspiciously, xl list showed DomUs created, but they did not rea= lly start up, script timeout-ed for ssh connection, no vnc. When i did xl d= estroy manually, and xl create, system panic happened.

=
I have log files for all serial output, if there is anything useful, i= can provide. On disk log files seems to loose latest messages due to crash= .


On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 <roger.pau@citrix.com> wrote:
On Wed, Mar 02, 20= 22 at 10:57:37AM +0200, Ze Dupsys wrote:
> Hello,
>
> I started using XEN on one pre-production machine (with aim to use lat= er in
> production) with 12.2, but since it experienced random crashes i updat= ed to
> 13.0 in hope that errors might disappear.
>
> I do not know how detailed should i write, so that this email is not t= oo
> long, but gives enough info.
>
> FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rul= es
> for NATting are used. Zpool is composed of 2 mirrored disks. There is = a
> ZVOL volmode=3Ddev for each VM and VM's jail that are attached as = raw devices
> to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, = with
> VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i = have
> bridge interfaces, where DomU's are connected depending on their > "zone/network", those that have allowed outgoing connections= are NATted by
> IPFW on specific physical NIC and IP.

So from the traces on the ticket:

panic: pmap_growkernel: no memory to grow kernel
cpuid =3D 0
time =3D 1646123072
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff81073eed at pmap_growkernel+0x27d
#4 0xffffffff80f2dae8 at vm_map_insert+0x248
#5 0xffffffff80f30249 at vm_map_find+0x549
#6 0xffffffff80f2bf76 at kmem_init+0x226
#7 0xffffffff80c73341 at vmem_xalloc+0xcb1
#8 0xffffffff80c72c3b at vmem_xalloc+0x5ab
#9 0xffffffff80f2bfce at kmem_init+0x27e
#10 0xffffffff80c73341 at vmem_xalloc+0xcb1
#11 0xffffffff80c72c3b at vmem_xalloc+0x5ab
#12 0xffffffff80c72646 at vmem_alloc+0x46
#13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96
#14 0xffffffff80f21a2a at uma_prealloc+0x23a
#15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e
#16 0xffffffff80f1f6af at uma_set_align+0x8f
#17 0xffffffff82463362 at abd_borrow_buf_copy+0x22
Uptime: 4m9s


Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address=C2=A0 =C2=A0=3D 0x22710028
fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present
instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff80c45892
stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930
frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930
code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0
current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 1496 (devmatch)
trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12
panic: page fault
cpuid =3D 0
time =3D 1646123791
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff8108b1a7 at trap+0xbc7
#4 0xffffffff8108b1ff at trap+0xc1f
#5 0xffffffff8108a85d at trap+0x27d
#6 0xffffffff81061b18 at calltrap+0x8
#7 0xffffffff80c62011 at rman_is_region_manager+0x241
#8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
#9 0xffffffff80c1949c at kernel_sysctl+0x43c
#10 0xffffffff80c19b13 at userland_sysctl+0x173
#11 0xffffffff80c1995f at sys___sysctl+0x5f
#12 0xffffffff8108baac at amd64_syscall+0x10c
#13 0xffffffff8106243e at Xfast_syscall+0xfe


Fatal trap 12: page fault while in kernel mode
cpuid =3D 1; apic id =3D 02
fault virtual address=C2=A0 =C2=A0=3D 0x68
fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present
instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff824a599d
stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e27910
frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e279b0
code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0
current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0 (xbbd7 taskq)
trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12
panic: page fault
cpuid =3D 1
time =3D 1646122723
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff8108b1a7 at trap+0xbc7
#4 0xffffffff8108b1ff at trap+0xc1f
#5 0xffffffff8108a85d at trap+0x27d
#6 0xffffffff81061b18 at calltrap+0x8
#7 0xffffffff8248935a at dmu_read+0x2a
#8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa
#9 0xffffffff80a7f214 at xbd_instance_create+0xa394
#10 0xffffffff80a7b1ea at xbd_instance_create+0x636a
#11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
#12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
#13 0xffffffff80bc7e3e at fork_exit+0x7e
#14 0xffffffff81062b9e at fork_trampoline+0xe
Uptime: 1h44m10s

This all look to me like Out of Memory conditions, can you check with
`top` whats going on with your memory?

Might also be helpful to record periodic calls to `vmstat -m | sort -k
2 -r` to try to figure out what's using so much memory.

Regards, Roger.
--00000000000086a47305d93f92e8-- --00000000000086a47505d93f92ea Content-Type: application/octet-stream; name="vmstat_before_crash.log" Content-Disposition: attachment; filename="vmstat_before_crash.log" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l09tdsh80 Li9iaW4vbGliZXhlYy5zaDogPT09IFZNU1RBVCA9PT0KIE5GU0Qgc2Vzc2lvbiAgICAgMSAgICAg MUsgICAgICAgIDEgIDEwMjQKICAgIENBTSBxdWV1ZSAgICAxNiAgICAgNUsgICAgICAgNTcgIDE2 LDMyLDUxMgogICBDQU0gcGVyaXBoICAgICA2ICAgICAySyAgICAgICAyNCAgMTYsMzIsNjQsMTI4 LDI1NgogICAgIENBTSBwYXRoICAgIDExICAgICAxSyAgICAgICA4MiAgMzIKIE5GU0QgbGNrZmls ZSAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgpDQU0gZGV2IHF1ZXVlICAgICA3ICAgICAxSyAg ICAgICAgNyAgNjQKICAgICAgQ0FNIFhQVCAgICAzMyAgICAgM0sgICAgICAgNzQgIDE2LDMyLDEy OCw1MTIsMTAyNCwyMDQ4LDY1NTM2Ck5GU0QgVjRjbGllbnQgICAgIDEgICAgIDFLICAgICAgICAx ICAyNTYKICAgICAgQ0FNIFNJTSAgICAgNyAgICAgMksgICAgICAgIDcgIDI1NgogICAgICAgICBU eXBlIEluVXNlIE1lbVVzZSBSZXF1ZXN0cyAgU2l6ZShzKQpDQU0gSS9PIFNjaGVkdWxlciAgICAg MiAgICAgMUsgICAgICAgIDIgIDEyOAogICAgICBDQU0gREVWICAgICA5ICAgIDE4SyAgICAgICAy MiAgMjA0OAogICAgICBDQU0gQ0NCICAgIDEwICAgIDIwSyAgODM3MTk1NiAgMjA0OAogICAgICBz b2xhcmlzIDU2OTMxNCAxMTQwNjVLIDQ5ODUyMzQwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEy LDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgsNjU1MzYKICAgIHN5c2N0bG9pZCAzMjQz MiAgMTgzM0sgICAgNTM3NTUgIDE2LDMyLDY0LDEyOAogICAgICAgZGV2YnVmIDE3OTYxIDM1MTI3 SyAgICAxODMwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIs NjU1MzYKICAgICAgICAgeGJiZCAxMTk2OCAzODQ3MDZLICAgMjExMDc0ICA2NCwyMDQ4LDY1NTM2 CiAgICAgICBhY3BpY2EgIDk3NjggIDEwMjVLICAgMjU1MTA0ICAxNiwzMiw2NCwxMjgsMjU2LDM4 NCw1MTIsMTAyNCwyMDQ4CiAgICAgICAgICBidXMgIDEyOTYgICAxMTZLICAgIDI4NTMxICAxNiwz Miw2NCwxMjgsMjU2LDM4NCw0MDk2CiAgICAgICAgIHVtdHggIDExNTIgICAxNDRLICAgICAxMTUy ICAxMjgKICAgICAga2R0cmFjZSAgIDYyNSAgIDE0M0sgICAgMTUxMDQgIDY0LDI1NgogICAgICAg bW9kdWxlICAgNTA4ICAgIDY0SyAgICAgIDUwOCAgMTI4CiAgICAgICB4ZW5idXMgICAzOTcgICAg NjFLICAgIDI0ODU0ICAxNiwzMiw2NCwyNTYsNDA5NgogICAgICAgICAgb3NkICAgMzYzICAgICA3 SyAgICAgMTAwMyAgMTYsMzIsNjQsMTI4LDI1NgogICAgICAgbGlua2VyICAgMzU2ICAyMTE3SyAg ICAgMTQxMiAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYz ODQKICAgICAgICAga29iaiAgIDMzNiAgMTM0NEsgICAgICA2NzMgIDQwOTYKICAgICAgICAgcm1h biAgIDI0NCAgICAyOEsgICAgIDMwMTkgIDE2LDMyLDEyOAphdWRpdF9ldmNsYXNzICAgMjM2ICAg ICA4SyAgICAgIDI5NCAgMzIKICAgIHRhc2txdWV1ZSAgIDIxNiAgICAyNUsgICAgICA2NjkgIDE2 LDMyLDY0LDEyOCwyNTYKICAgICAgICAgIFVNQSAgIDIxNSAgICAgNUsgICAgICAyMTUgIDE2LDMy CiAgICAgICBERVZGUzMgICAyMDQgICAgNTFLICAgICAgNTI4ICAyNTYKICAgICAgc3VicHJvYyAg IDE2NSAgIDMyMEsgICAgMTM0MzUgIDM4NCw0MDk2CiAgICAgICBERVZGUzEgICAxNjEgICAxNjFL ICAgICAgNDkwICAxMDI0CiAgICAgIGl0aHJlYWQgICAxNTQgICAgMjZLICAgICAgNTg5ICAzMiwx MjgsMjU2CiAgICAgICBpZmFkZHIgICAxMzAgICAgMThLICAgICAgNTM2ICAxNiwzMiw2NCwxMjgs MjU2LDM4NCwyMDQ4LDQwOTYKIGV2ZW50aGFuZGxlciAgIDEzMCAgICAxMUsgICAgICAxMzAgIDY0 LDEyOAogICAgICAgICBHRU9NICAgMTI1ICAgIDIwSyAgICAgMTI5MCAgMTYsMzIsNjQsMTI4LDI1 Niw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5MiwxNjM4NAogICAgICAgICBrZW52ICAgMTE2ICAgIDEz SyAgICAgIDExNiAgMTYsMzIsNjQsMTI4LDI1Niw4MTkyCiAgICAgICBidXMtc2MgICAxMTAgICAz NjVLICAgICA2MzA3ICAxNiwzMiw2NCwxMjgsMjU2LDM4NCw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5 MiwxNjM4NCwzMjc2OCw2NTUzNgogICAgICAgS1RSQUNFICAgMTAwICAgIDEzSyAgICAgIDEwMCAg MTI4CiAgICAgICAgc2VsZmQgICAgOTUgICAgIDZLIDMyNTIxOTcwOCAgNjQKICAgICAgcHdkZGVz YyAgICA3MSAgICAgNUsgICAgMTMzNDEgIDY0CiAgICAgICBrcXVldWUgICAgNzEgICAgIDVLICAg IDEzMzQxICA2NAogICBERVZGU19SVUxFICAgIDU5ICAgIDIxSyAgICAgICA1OSAgNjQsMzg0CiAg ICBwcm9jLWFyZ3MgICAgNTcgICAgIDNLICAgICA5NjU2ICAxNiwzMiw2NCwxMjgsMjU2CiAgcHJp dmNtZF9kZXYgICAgNTcgICAgIDJLICA4NTk4NTQ4ICAxNiwzMiw2NCwyNTYsNTEyLDEwMjQsMjA0 OCw4MTkyLDE2Mzg0CiAgIGtzdGF0X2RhdGEgICAgNTYgICAgNTZLICAgICAgMjM5ICAxMDI0CnZt X2ZpY3RpdGlvdXMgICAgNTIgIDc0NTlLICAgICA1MjYwICA2NCwxMjgsMjU2LDEwMjQsODE5Miwz Mjc2OAogICAgICAgICAgVVNCICAgIDUxICAgIDgySyAgICAgICA1NyAgMTYsMzIsNjQsMTI4LDI1 NiwzODQsMTAyNCwyMDQ4LDQwOTYsODE5MgogICAgICAgVVNCZGV2ICAgIDQ5ICAgIDE1SyAgICAg ICA1MSAgMzIsNjQsMTI4LDI1NiwzODQsNDA5NgogICAgICAgIG1vdW50ICAgIDQ2ICAgICAySyAg ICAgIDEzNyAgMTYsMzIsNjQsMTI4LDI1NiwxMDI0CiAgICAgICAgaWZsaWIgICAgNDUgICAzMTZL ICAgICAgIDUzICAxNiw2NCwxMjgsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgK ICAgICAgIERFVkZTUCAgICA0MSAgICAgM0sgICAgIDMyMDIgIDY0CiAgICAgeGVuX2ludHIgICAg MzcgICAgIDlLICAgICAgIDM3ICAxMjgsNDA5NgogICAgICBsbHRhYmxlICAgIDMxICAgICA5SyAg ICAgIDI3OCAgMjU2LDM4NAogICAgICBzZXNzaW9uICAgIDMxICAgICA0SyAgICAgIDU3NyAgMTI4 CiAgICAgIGFjcGlzZW0gICAgMzEgICAgIDRLICAgICAgIDMxICAxMjgKICAgICAgICAgdGVtcCAg ICAzMCAgICAxN0sgICAgMzI4OTkgIDE2LDMyLDY0LDEyOCwyNTYsMzg0LDUxMiwxMDI0LDIwNDgs ODE5Miw2NTUzNgogICAgICAgIGxvY2tmICAgIDMwICAgICA0SyAgMzUxNTkyMyAgNjQsMTI4CiAg ICAgICBVbml0bm8gICAgMzAgICAgIDJLICAgICAgODkwICAzMiw2NAogICAgICBhY3BpZGV2ICAg IDMwICAgICAySyAgICAgICAzMCAgNjQKICB1ZnNfZGlyaGFzaCAgICAyNSAgICAxM0sgICAgICAz NTkgIDE2LDMyLDY0LDEyOCw1MTIKICAgICAgICB2dGJ1ZiAgICAyNCAgMTk2OEsgICAgICAgNDYg IDQwOTYKICAgICAgICAgIHR0eSAgICAyNCAgICAyNEsgICAgICAxNDEgIDEwMjQKICAgICAgICAg Y3JlZCAgICAyMyAgICAgNksgICAgIDIxOTggIDI1NgogICAgIHhlbnN0b3JlICAgIDIyICAgICA1 SyAgIDI1NDk3NyAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDQwOTYsMTYzODQKICBldGhlcl9t dWx0aSAgICAyMiAgICAgMksgICAgICAgMjIgIDE2LDMyLDY0LDEyOAogICBldnRjaG5fZGV2ICAg IDIwICAgIDI2SyAyODEwNjA1MSAgNjQsMjU2LDQwOTYKICAgIHBmc19ub2RlcyAgICAyMCAgICAg OEsgICAgICAgMjAgIDM4NAogICAgICAgc2VsZWN0ICAgIDIwICAgICAzSyAgICAgMTkyMiAgMTI4 CiAgIGd0YXNrcXVldWUgICAgMTggICAgMjZLICAgICAgIDE4ICAxNiwzMiwyNTYsODE5MgogICAg ICAgc29uYW1lICAgIDE4ICAgICAySyAgICAyNTA5OCAgMTYsMzIsMTI4CiAgICAgICAgICBwY2Ig ICAgMTcgIDE1NDVLICAgICAgNzEwICAxNiwzMiw2NCwxMDI0LDIwNDgKICAgICAgIHBsaW1pdCAg ICAxNyAgICAgNUsgICAgIDExMjkgIDI1NgogICAgIHJvdXRldGJsICAgIDE2ICAgICA0SyAgICAg ICAxNiAgMzIsNjQsMTI4LDM4NCwxMDI0CiAgICAgcGNpX2xpbmsgICAgMTYgICAgIDJLICAgICAg IDE2ICA2NCwxMjgKICAgICAgIGdudGRldiAgICAxNiAgICAgMUsgICAgICA5NTEgIDMyLDY0CiAg ICAgICAgREVWRlMgICAgMTYgICAgIDFLICAgICAgIDE3ICAxNiwxMjgKICAgIGluNl9tdWx0aSAg ICAxNSAgICAgMksgICAgICAgMTUgIDMyLDI1NgogICAgICAgICAgIHZ0ICAgIDEzICAgIDQxSyAg ICAgICAxMyAgMzg0LDQwOTYsMzI3NjgKICAgICAgICBoaG9vayAgICAxMyAgICAgNEsgICAgICAg MTMgIDI1NgogICAgICAgICAgcHRzICAgIDExICAgICAzSyAgICAgIDEyOCAgMjU2CiAgICAgdGVy bWluYWwgICAgMTEgICAgIDNLICAgICAgIDExICAyNTYKICAgICAgZGV2c3RhdCAgICAxMCAgICAy MUsgICAgICAgMTAgIDMyLDQwOTYKICAgIHRvcG9ub2RlcyAgICAxMCAgICAgMksgICAgICAgMTAg IDEyOAogICAgICAgIGNsb25lICAgICA5ICAgICAySyAgICAgICAgOSAgMTI4CiAgICAgICBrYmRt dXggICAgIDggICAgMjJLICAgICAgICA4ICAxNiwzODQsMTAyNCwyMDQ4LDE2Mzg0CiAgaW5wY2Jw b2xpY3kgICAgIDggICAgIDFLICAgICAxODAwICAzMgplbmNhcF9leHBvcnRfaG9zdCAgICAgOCAg ICAgMUsgICAgICAgIDggIDMyLDY0CiAgICAgICBwcmlzb24gICAgIDggICAgIDFLICAgICAgICA4 ICAxNiwzMgogICAgICAgaXNhZGV2ICAgICA4ICAgICAxSyAgICAgICAgOCAgMTI4CiAgICAgICAg aWZuZXQgICAgIDcgICAgMTNLICAgICAgIDY1ICAxMjgsMjA0OAogICAgICAgIGV2ZGV2ICAgICA3 ICAgICA3SyAgICAgICAgNyAgMTAyNAogICAgIGZpbGVjYXBzICAgICA3ICAgICAxSyAgICAgNTI5 NCAgMTYsMzIsNjQKICAgICAgIGlwNm5kcCAgICAgNyAgICAgMUsgICAgICAgNjUgIDY0LDI1Ngog ICAgICAgZmVlZGVyICAgICA3ICAgICAxSyAgICAgICAgNyAgMzIKICAgICBuZXh1c2RldiAgICAg NyAgICAgMUsgICAgICAgIDcgIDE2CiAgICAgICAgICBMUk8gICAgIDYgICAxMjBLICAgICAgICA2 ICA4MTkyLDMyNzY4CiAgICAgICAgIGlnbXAgICAgIDYgICAgIDFLICAgICAgIDY0ICAxMjgKICAg ICAgICAgIG1sZCAgICAgNiAgICAgMUsgICAgICAgNjQgIDEyOAogICAgICAgICAgQlBGICAgICA2 ICAgICAxSyAgICAgICA2NCAgMTI4CiAgICAgICAgbmhvcHMgICAgIDYgICAgIDFLICAgICAgICA2 ICAxMjgKICAgICBmaWxlZGVzYyAgICAgNSAgICAyMEsgICAgICA1NzkgIDQwOTYKICAgIHVmc19t b3VudCAgICAgNSAgICAxN0sgICAgICAgIDYgIDMyLDY0LDUxMiw0MDk2LDgxOTIKICAgICAgZXZl bnRmZCAgICAgNSAgICAgMUsgICAgICAzMDAgIDEyOAogICAgICAgIGxpbnV4ICAgICA1ICAgICAx SyAgICAgICAgNiAgMTYsNjQKICAgICAgICAgIE1DQSAgICAgNSAgICAgMUsgICAgICAgIDUgIDE2 LDMyLDEyOAogICAgICAgICBwZmlsICAgICA1ICAgICAxSyAgICAgICAgNSAgMTI4CiAgICAgICAg ICBtc2kgICAgIDUgICAgIDFLICAgICAgICA1ICAxMjgKICAgICAgICAgaW50ciAgICAgNCAgIDQ3 MksgICAgICAgIDQgIDY1NTM2CiAgICAgICAgICBzZW0gICAgIDQgICAxMDZLICAgICAgICA0ICAy MDQ4LDQwOTYKICAgICAgICAgIG1zZyAgICAgNCAgICAzMEsgICAgICAgIDQgIDIwNDgsNDA5Niw4 MTkyLDE2Mzg0CiAgICAgdmZzY2FjaGUgICAgIDMgIDIwNDlLICAgICAgICAzICAyNTYsNTEyCiAg ICAgICAgIHZtZW0gICAgIDMgIDEwNjBLICAgICAgIDE1ICA1MTIsMTAyNCwyMDQ4LDQwOTYsMTYz ODQsMzI3NjgKICAgICAgdGlkaGFzaCAgICAgMyAgIDE1OEsgICAgICAgIDMgIDIwNDgsMzI3NjgK ICAgICAgICAgcHJvYyAgICAgMyAgICAzM0sgICAgICAgIDMgIDEwMjQsMTYzODQKICAgICAgICAg c2J1ZiAgICAgMyAgICAgMksgICAyNTkzNjEgIDE2LDMyLDY0LDEyOCwyNTYsNTEyLDEwMjQsNDA5 NiwxNjM4NAogIHZub2RlbWFya2VyICAgICAzICAgICAySyAgICAgIDY3NiAgNTEyCiBsaW51eGN1 cnJlbnQgICAgIDMgICAgIDFLICAgICAgICA0ICA2NCwyNTYKICAgICAgIGNyeXB0byAgICAgMyAg ICAgMUsgICAgICAgIDMgIDY0LDI1NgogICBsb2dpbmNsYXNzICAgICAzICAgICAxSyAgICAgICAg MyAgNjQKICAgICAgICAgIHR1biAgICAgMyAgICAgMUsgICAgICAgIDMgIDMyCiAgICAgaW5fbXVs dGkgICAgIDMgICAgIDFLICAgICAgICAzICAyNTYKICAgICAgY2FsbG91dCAgICAgMiAgIDUxMksg ICAgICAgIDIgIAogICAgIG10eF9wb29sICAgICAyICAgIDcySyAgICAgICAgMiAgODE5Miw2NTUz NgogICAgICAgICAgcnBjICAgICAyICAgICA4SyAgICAgICAgMiAgNDA5NgogICAgICB1aWRpbmZv ICAgICAyICAgICA1SyAgICAgICA1NSAgMTI4LDQwOTYKICAgIGlwc2VjLXNhcSAgICAgMiAgICAg MksgICAgICAgIDIgIDEwMjQKICAgICAgICB0YXNrcSAgICAgMiAgICAgMUsgICAgICAgIDIgIDY0 LDEyOAogICAgc2ZzX25vZGVzICAgICAyICAgICAxSyAgICAgICAgMiAgMzg0CiAgICBhY3BpX3Bl cmYgICAgIDIgICAgIDFLICAgICAgICAyICAzODQKICAgICAgICAgY3B1cyAgICAgMiAgICAgMUsg ICAgICAgIDIgIDE2CiAgICAgIGpibG9ja3MgICAgIDIgICAgIDFLICAgICAgICAyICAxMjgsMjU2 CiAgICAgICBuZXdibGsgICAgIDEgIDIwNDhLICAgICAgMzg4ICAyNTYKICAgICBpbm9kZWRlcCAg ICAgMSAgMTAyNEsgICAgIDI3MjYgIDM4NAogICAgdWZzX3F1b3RhICAgICAxICAxMDI0SyAgICAg ICAgMSAgCiAgICAgdmZzX2hhc2ggICAgIDEgIDEwMjRLICAgICAgICAxICAKICAgICAgcGFnZWRl cCAgICAgMSAgIDI1NksgICAgICAxNjAgIDI1NgogICAgdm5ldF9kYXRhICAgICAxICAgMTY4SyAg ICAgICAgMSAgCiAgIHRmb19jY2FjaGUgICAgIDEgICAxMjhLICAgICAgICAxICAKICAgICBzeW5j YWNoZSAgICAgMSAgICA2OEsgICAgICAgIDEgIAogICAgIGFjcGl0YXNrICAgICAxICAgIDY0SyAg ICAgICAgMSAgNjU1MzYKICAgIGhvc3RjYWNoZSAgICAgMSAgICAzMksgICAgICAgIDEgIDMyNzY4 CiAgICAgICAgICBzaG0gICAgIDEgICAgMzJLICAgICAgICAxICAzMjc2OAogIHBmc192bmNhY2hl ICAgICAxICAgIDE2SyAgICAgICAgMSAgMTYzODQKICAgIGJtc2FmZW1hcCAgICAgMSAgICAgOEsg ICAgIDEwMjcgIDI1Niw4MTkyCiAgICAgICAgc2htZmQgICAgIDEgICAgIDhLICAgICAgICAxICA4 MTkyCiAgIGxvY2FsX2FwaWMgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIG1lbWRl c2MgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIGlvX2FwaWMgICAgIDEgICAgIDJL ICAgICAgICAxICAyMDQ4CiAgICAgICBzeXNjdGwgICAgIDEgICAgIDFLICAgICAxNjkyICAzMiw2 NAogICAgIGZyZWV3b3JrICAgICAxICAgICAxSyAgICAgIDMxOSAgMzIsMTI4CiAgICAgYWNwaWlu dHIgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICB0Y3BmdW5jICAgICAxICAgICAxSyAg ICAgICAgMSAgNjQKICAgICAgICAgdm5ldCAgICAgMSAgICAgMUsgICAgICAgIDEgIDY0CiAgICAg ICAgICBwbWMgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICBzb2Z0ZGVwICAgICAxICAg ICAxSyAgICAgICAgMSAgNTEyCmNoYWNoYTIwcmFuZG9tICAgICAxICAgICAxSyAgICAgICAgMSAg Mzg0CnZuZXRfZGF0YV9mcmVlICAgICAxICAgICAxSyAgICAgICAgMSAgMzIKICAgICAgUGVyLWNw dSAgICAgMSAgICAgMUsgICAgICAgIDEgIDMyCiAgICAgICB2bm9kZXMgICAgIDEgICAgIDFLICAg ICAgICAxICAyNTYKICAgICAgICAga3RscyAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgogICAg ICAgICBjZGV2ICAgICAxICAgICAxSyAgICAgICAgMSAgMjU2CiAgICAgcDEwMDMuMWIgICAgIDEg ICAgIDFLICAgICAgICAxICAxNgogICAgIHBtY2hvb2tzICAgICAxICAgICAxSyAgICAgICAgMSAg MTI4CiAgICAgICBhcG1kZXYgICAgIDEgICAgIDFLICAgICAgICAxICAxMjgKICBpcHNlY3BvbGlj eSAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICBmcHVrZXJuX2N0eCAgICAgMSAgICAgMUsg ICAgICAgIDEgIDEwMjQKICAgICBzZWNhc3ZhciAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQK ICAgICAgIHNhaGVhZCAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICAgICAgICAgIGlvdiAg ICAgMCAgICAgMEsgIDg5MTE3ODcgIDE2LDY0LDEyOCwyNTYsMzg0LDUxMgogICAgc3lzY3RsdG1w ICAgICAwICAgICAwSyAgICAxMzQ2NyAgMTYsMzIsNjQsMjU2LDEwMjQKICAgICAganNlZ2RlcCAg ICAgMCAgICAgMEsgICAgIDQ4NjYgIDY0CiAgICAgIGphZGRyZWYgICAgIDAgICAgIDBLICAgICAy MjQwICAxMjgKICAgICAgIGRpcmFkZCAgICAgMCAgICAgMEsgICAgIDIyMjggIDEyOAogICAgICBq cmVtcmVmICAgICAwICAgICAwSyAgICAgMjIyMCAgMTI4CiAgICAgICBkaXJyZW0gICAgIDAgICAg IDBLICAgICAyMjA4ICAxMjgKICAgICAgZW50cm9weSAgICAgMCAgICAgMEsgICAgIDE2OTQgIDE2 LDMyLDY0LDQwOTYKICAgICBmcmVlZmlsZSAgICAgMCAgICAgMEsgICAgICA4MzcgIDY0CiAgICAg cHJvY2Rlc2MgICAgIDAgICAgIDBLICAgICAgNzQ0ICAxMjgKICAgICBpb2N0bG9wcyAgICAgMCAg ICAgMEsgICAgICA0NzggIDI1NiwzODQsMTAyNAogICAgIHNhdmVkaW5vICAgICAwICAgICAwSyAg ICAgIDQwOSAgMjU2CiAgICAgICAgIGpzZWcgICAgIDAgICAgIDBLICAgICAgNDA2ICAxMjgKICAg ICAgam5ld2JsayAgICAgMCAgICAgMEsgICAgICAzODcgIDEyOAogICAgIGZyZWVibGtzICAgICAw ICAgICAwSyAgICAgIDMxOCAgMjU2CiAgICAgICAgc2JkZXAgICAgIDAgICAgIDBLICAgICAgMjI4 ICA2NAogICAgcmFpZF9kYXRhICAgICAwICAgICAwSyAgICAgIDE5MiAgMzIsMTI4LDI1NgogICAg ICAgc3RhdGZzICAgICAwICAgICAwSyAgICAgICA2NSAgNDA5NgogICAgamZyZWVmcmFnICAgICAw ICAgICAwSyAgICAgICAxOSAgMTI4CiAgICAgZnJlZWZyYWcgICAgIDAgICAgIDBLICAgICAgIDE5 ICAxMjgKICAgICAgICBta2RpciAgICAgMCAgICAgMEsgICAgICAgMTIgIDEyOAogICAgIG1idWZf dGFnICAgICAwICAgICAwSyAgICAgICAgNyAgMzIKICAgIG5ld2RpcmJsayAgICAgMCAgICAgMEsg ICAgICAgIDYgIDY0Cm5hbWVpX3RyYWNrZXIgICAgIDAgICAgIDBLICAgICAgICA1ICAzMgogICBj bF9zYXZlYnVmICAgICAwICAgICAwSyAgICAgICAgNCAgNjQKICAgbWFkdF90YWJsZSAgICAgMCAg ICAgMEsgICAgICAgIDIgIDMyLDQwOTYKICAgICAgICBzaWdpbyAgICAgMCAgICAgMEsgICAgICAg IDEgIDY0Ci4vYmluL2xpYmV4ZWMuc2g6ID09PSBWTVNUQVQgPT09Cg== --00000000000086a47505d93f92ea-- From nobody Wed Mar 2 18:12:28 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4091E19E038B for ; Wed, 2 Mar 2022 18:12:06 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K82JN6s3Qz4cch for ; Wed, 2 Mar 2022 18:12:04 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pf1-x433.google.com with SMTP id y11so2619025pfa.6 for ; Wed, 02 Mar 2022 10:12:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Km784lPMqWQy1P3KlOXLXyk3qDZQgh0YIKyW9HBluDM=; b=h2Pybmp+Y257QU7QyLn0uD88JaEVGTnN+BT+RY0QnkZY2g50XgqQSRg1kRF5qdAif1 0zlYMfQiljfYckiO4ph4LEmptm1lgeUzA53XxycMgrsfKnrDu1E+mmTv+Bxb+SaUF71s dmQMCvjcascR9UPPPfg4TN2rV7winTcHI+TbmyIII5ClMFcbkq+OWaVDTL+P5D8FonJt TiL/fh2XjQ45KtDYw6YVcHrP4GhAMWcEdOY/VTwUAJea6FT2B1IG9uoJWORnbAIi+iOj 20/HWVuBU7XC88A0d2pBvFCvIVF/09GN2KCmcm7xE6wCT5m3wvwhcd54jKdCoH9vLYCa jSPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Km784lPMqWQy1P3KlOXLXyk3qDZQgh0YIKyW9HBluDM=; b=I02u7/TwCVw5mjP62zRvyLvbPR2IRwi5wNJQXO1/HBx8N9pSnNVLxqqn4tNQL/edh8 0oMGbw4onLpuNbiHkK3xSqLqVZaAvpMlQ9SCsDEaJIQDErfsxJ5pIN4e//JPpc238BNq 5Gsm+U8s42DMa1ZYAqWvset5vLboIm9JfaM1M17Tq4o6adlw7BjBA3HKRCxeyciV6EIz 74jrjKA695KNjGYIi4NJOp4wH91kacDHfnEuLP7Eni0PeXimRcZ83eb19uP7YLITM6uv BRIzDKEURynVqq1zrRCJ3NRf3KTyXlmCnF0JI405bVJIP9WsyHo12+lzau+0hN5Y9knE 3h0w== X-Gm-Message-State: AOAM530kyyFrBY3nSZWo/eRz0lEDalgzQ/o5OEOBJfWKaStn7ISsKZKH MggJWmjhVkJhBe3OE1/MRZUi5C0wG5Vhbt5v30g= X-Google-Smtp-Source: ABdhPJwIJn85EdLRHbhOxmEfQ6T8kQleTUNRKdYR8yWBVPyESeNmlpvplPcVdSiQejpA5YEYZh3OF4rjfwsi2CZJocA= X-Received: by 2002:a05:6a00:16ce:b0:4ce:118f:8e4f with SMTP id l14-20020a056a0016ce00b004ce118f8e4fmr34846616pfc.56.1646244717933; Wed, 02 Mar 2022 10:11:57 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <202203021705.222H5MVm026787@nfbcal.org> In-Reply-To: <202203021705.222H5MVm026787@nfbcal.org> From: Ze Dupsys Date: Wed, 2 Mar 2022 20:12:28 +0200 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: Brian Buhrow Cc: freebsd-xen@freebsd.org Content-Type: multipart/alternative; boundary="000000000000a0365d05d9403764" X-Rspamd-Queue-Id: 4K82JN6s3Qz4cch X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=h2Pybmp+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::433 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::433:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MLMMJ_DEST(0.00)[freebsd-xen]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --000000000000a0365d05d9403764 Content-Type: text/plain; charset="UTF-8" I agree with you, that firewall and networking most probably is not at fault. When Dom0 had jails with netgraph, then it panic'ed a lot. Well, now i have concluded that mixing jails with VMs is not a good idea anyways, better having jails in DomU. Well, i could beef up RAM a bit, but it seems that this would just postpone the inevitable. While DomUs are just having CPU load, use network and have little load on HDDs, all is fine, but once HDD load increases, at some point system crashes. It really could be as you say due to ZFS and it's monopolistic/unfriendly RAM usage. I guess i will go on quest to search how to tune/limit ZFS a bit. About snapshoting ZFS volumes... i'm not doing it while DomUs are running. At the moment the crashing lab machine has no snapshots at all. Lab machine reboots with panic messages written on serial output, and logged by old laptop. So it feels that it is not hardware error; i will run memtest to be sure. At first i did get only partial panic messages, since XEN rebooted too soon, but then i added sync_console=1, so i get full panic messages, it seems that reboot=no value is not taken into account, though. About memory balooning i feel somewhat hesitant, in a way i am trying to not use too many different techniques that could introduce more problems. Is balooning on XEN + FreeBSD Dom0 considered stable? Have you used XENs driver domain with FreeBSD to "export/provide" disks? It seemed interesting approach as well, but as i was following documentation i could not understand how to even configure FreeBSD as a driver domain, if it's even possible, to provide block devices to Dom0 so it can provide them to other DomUs. This might solve RAM issues as well, since driver domain would have it's reserved RAM and could not put pressure on Dom0's RAM for whatever reason. In a way i am thinking about various strategies to shave off services from Dom0, to ensure it's stability. Maybe i should configure firewall inside in a DomU as in your pfSense example. Since for me usually CPU resources are not exhausted, but NICs and HDDs are. Thank's for the ideas of what else could be done, to solve this! Best wishes, Ze Dupsys On Wed, Mar 2, 2022 at 7:05 PM Brian Buhrow wrote: > hello. One difference between my systems and yurs, though I don't > think that's the > problem, is that I'm not running a firewall on the dom0 itself. The dom0 > runs on a protected > vlan with respect to the external network and the domu's are connected to > bridges that are > directly connected to the external network. I have one system where the > customer wants the > pfsense system runing, so pfsense runs as a domu on this system, connected > to an internal > "private" bridge and the public bridge, doing all the firewalling between > them. In this way, > the FreeBSD dom0 is only doing ZFS, simple ip routing and Xen management. > > If I had to wager a guess as to your trouble, it's that you don't > have enough memory on > your dom0. ZFS is a memory hog and I can't imagine getting away with > anything less than 8G on > the dom0 with FreeBSD-12 and ZFS. I'm using 8G for the dom0 on the system > I'm writing from and > it is quite stable, but, then again, I'm not doing as much with the dom0 > as you are. > > I too am using zvols as disks for the domu's, but I've not been > trying to make zfs > snapshots from them. obvious question, but I'll ask it anyway, you're not > trying to make > snapshots of the zvols while the domu's on top of them are running, are > you? I would imagine > that would not give you good images, but I wouldn't expect it to panic the > dom0 either. > However, it wil stretch your meager memory resources even further. > > Have you been able to get a panic message or does the system just > spontaneously reboot? If it > just reboots, then, again, I think you are having a memory shortage. > > My suggestion is to try giving the dom0 8G of RAM and then for the domU's, > use the balloon > driver to oversubscribe the remaining memory for the domu's. Of course, > the best course of > action is to see if you can put more memory in this system; 16GB just > isn't that much when > you're trying to run Xen plus a few domu's, especially on top of ZFS. > If yu can get a panic message or a crash dump, that would be helpful in > figuring out more > accurately what's going on. > > Another thought, since you were getting some crashes when running jails > with xen, is to get > memtest86 running on the raw machine and let it run for 3 or 4 days. If > you don't get any > memory errors, then I think you can be pretty sure it's not a hardware > problem. If, however, > you get any errors at all with that test, then I think it's a good bet you > hav a hardware issue. > > > -thanks > -Brian > > --000000000000a0365d05d9403764 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I agree with you, that firewall and networking most p= robably is not at fault. When Dom0 had jails with netgraph, then it panic&#= 39;ed a lot. Well, now i have concluded that mixing jails with VMs is not a= good idea anyways, better having jails in DomU.

Well, i could beef up RAM a bit, but it seems that this would just postp= one the inevitable. While DomUs are just having CPU load, use network and h= ave little load on HDDs, all is fine, but once HDD load increases, at some = point system crashes. It really could be as you say due to ZFS and it's= monopolistic/unfriendly RAM usage. I guess i will go on quest to search ho= w to tune/limit ZFS a bit.

About snapshoting ZFS v= olumes... i'm not doing it while DomUs are running. At the moment the c= rashing lab machine has no snapshots at all.

Lab m= achine reboots with panic messages written on serial output, and logged by = old laptop. So it feels that it is not hardware error; i will run memtest t= o be sure. At first i did get only partial panic messages, since XEN reboot= ed too soon, but then i added sync_console=3D1, so i get full panic message= s, it seems that reboot=3Dno value is not taken into account, though.

About memory balooning i feel somewhat hesitant, in= a way i am trying to not use too many different techniques that could intr= oduce more problems. Is balooning on XEN + FreeBSD Dom0 considered stable?<= br>

Have you used XENs driver domain with FreeBSD = to "export/provide" disks? It seemed interesting approach as well= , but as i was following documentation i could not understand how to even c= onfigure FreeBSD as a driver domain, if it's even possible, to provide = block devices to Dom0 so it can provide them to other DomUs. This might sol= ve RAM issues as well, since driver domain would have it's reserved RAM= and could not put pressure on Dom0's RAM for whatever reason.

In a way i am thinking about various strategies to sha= ve off services from Dom0, to ensure it's stability. Maybe i should con= figure firewall inside in a DomU as in your pfSense example. Since for me u= sually CPU resources are not exhausted, but NICs and HDDs are.

Thank's for the ideas of what else could be done, to s= olve this!

Best wishes,
Ze Dupsys


On Wed, Mar 2, 2022 at 7:05 PM Brian Buhrow <buhrow@nfbcal.org> wrote:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = hello.=C2=A0 One difference between my systems and yurs, though I don't= think that's the
problem, is that I'm not running a firewall on the dom0 itself.=C2=A0 T= he dom0 runs on a protected
vlan with respect to the external network and the domu's are connected = to bridges that are
directly connected to the external network.=C2=A0 I have one system where t= he customer wants the
pfsense system runing, so pfsense runs as a domu on this system, connected = to an internal
"private" bridge and the public bridge, doing all the firewalling= between them.=C2=A0 In this way,
the FreeBSD dom0 is only doing ZFS, simple ip routing and Xen management.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 If I had to wager a guess as to your trouble, i= t's that you don't have enough memory on
your dom0.=C2=A0 ZFS is a memory hog and I can't imagine getting away w= ith anything less than 8G on
the dom0 with FreeBSD-12 and ZFS.=C2=A0 I'm using 8G for the dom0 on th= e system I'm writing from and
it is quite stable, but, then again, I'm not doing as much with the dom= 0 as you are.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I too am using zvols as disks for the domu'= s, but I've not been trying to make zfs
snapshots from them.=C2=A0 obvious question, but I'll ask it anyway, yo= u're not trying to make
snapshots of the zvols while the domu's on top of them are running, are= you?=C2=A0 I would imagine
that would not give you good images, but I wouldn't expect it to panic = the dom0 either.
However, it wil stretch your meager memory resources even further.

Have you been able to get a panic message or does the system just spontaneo= usly reboot?=C2=A0 If it
just reboots, then, again, I think you are having a memory shortage.

My suggestion is to try giving the dom0 8G of RAM and then for the domU'= ;s, use the balloon
driver to oversubscribe the remaining memory for the domu's.=C2=A0 Of c= ourse, the best course of
action is to see if you can put more memory in this system; 16GB=C2=A0 just= isn't that much when
you're trying to run Xen plus a few domu's, especially on top of ZF= S.
If yu can get a panic message or a crash dump, that would be helpful in fig= uring out more
accurately what's going on.

Another thought, since you were getting some crashes when running jails wit= h xen, is to get
memtest86 running on the raw machine and let it run for=C2=A0 3 or 4 days.= =C2=A0 If you don't get any
memory errors, then I think you can be pretty sure it's not a hardware = problem.=C2=A0 =C2=A0If, however,
you get any errors at all with that test, then I think it's a good bet = you hav a hardware issue.


-thanks
-Brian

--000000000000a0365d05d9403764-- From nobody Wed Mar 2 20:07:34 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B938B19F5426 for ; Wed, 2 Mar 2022 20:07:38 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K84sj3lVpz4q3Z for ; Wed, 2 Mar 2022 20:07:37 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 222K7YTq006622 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 2 Mar 2022 12:07:34 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 222K7Y7I013387; Wed, 2 Mar 2022 12:07:34 -0800 (PST) Message-Id: <202203022007.222K7Y7I013387@nfbcal.org> From: Brian Buhrow Date: Wed, 2 Mar 2022 12:07:34 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Ze Dupsys Subject: Re: ZFS + FreeBSD XEN dom0 panic Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 02 Mar 2022 12:07:35 -0800 (PST) X-Rspamd-Queue-Id: 4K84sj3lVpz4q3Z X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-xen]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org Hello. Given Roger's message and the presentation of the errors you're seeing, I think we're both in agreement that you're running out of memory. Below I've listed some suggestions on how you might go about making your system more stable, and, possibly figuring out where the trouble is, exactly. When I was building my infrastructure, I ran into a bunch of edge conditions and weird bugs as well, so some ideas that may not seem obvious, may, in fact, work, simply because they cause you to skirt some latent bug in the system. Here's what my setup looks like, in case it helps. 1. Use FreeBSD-12.1 or 12.2, and xen-4.13 or xen-4.14 (Using FreeBSD as dom0.) 2. Use a minimum of 8G of RAM for the dom0. 3. Put the root filesystem on a mirrored set of partitions or drives; I do not use ZFS as a root filesystem, but put root and swap on a mirrored media using the gmirror(8) utility and mirror driver. In this way, if the system needs to use swap, it can do so without touching ZFS and, hopefully, without requiring additional memory to get to the swap. 4. Create a network bridge using the bridge(4) driver for attaching the domu's to the network. You can either attach this bridge to one of your physical network interfaces, or you can attach it to a vlan(4) interface, thus allowing separation between the network the dom0 lives on and the network(s) the domu's live on. I do not do any firewalling or natting on the dom0, it's just a packet forwarder. 5. Create a zpool for your domu disks. It can be comprised of partitions on the same disks you use for booting, or it can live on separate disks. 6. Create zvols for the disks you want to attach to your domu's. I typically attach 1 or two disks to each domu. I've not experimented with attaching large numbers of disks to a single domu, instead partitioning the virtual disks from inside the domu itself if I want separation. I use the xbd(4) block driver for attaching the disks as raw block devices to the domu's. I've not played with the other virtual drivers. ( I have played with HVM hosts using the qemu drivers and found things sort of work, but are not stable; though the dom0 is fine.) While I've not tried oversubscribing the RAM reserved for domu use on the systems I run, if the domu's you're running support the balloon driver, I think you will have success in doing that. I disagree with your comment that adding memory just puts off the inevitable problem. Once you figure out how much memory your dom0 needs to do its work, then it's just a matter of deciding how much memory you want for your domu's. Having said that, however, it's important to figure out what the minimum amount of memory you need for your dom0 is; going below that threshold will result in instability. As I think abut your issue, I'm guessing you'll find things are much more stable and you'll be happy with 8G for the dom0 if you put the root and swap on a mirror and get it out of ZFS. I think you're runing into a situation where your system is short of memory, it tries to swap, the swap is on ZFS, which needs more memory to fulfill the paging request and, well, you get the idea. The system I'm sending this from, for example, has 8GB devoted to the dom0 and it's currently using 36Mb of swap. It's currently been up for 69 days, which is the time since the last power failure. I have another system, the one running pfsense as one of its domu's, which has 8GB of memory, no swap configured, and it's been up for 740 days. Hope these notes are helpful. -Brian From nobody Thu Mar 3 10:39:43 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D938719E8DE4 for ; Thu, 3 Mar 2022 10:40:04 +0000 (UTC) (envelope-from prvs=05474af58=roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K8SDM6TyTz4jsB for ; Thu, 3 Mar 2022 10:40:03 +0000 (UTC) (envelope-from prvs=05474af58=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1646304003; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=0QyO5x2TzQ0Wl5JKLRB+cA96L2WDicG+Y4Pi+EX2GKw=; b=NEAIt31BpD349P3f26/qzY0BefaX2WVu8i8fi93yTqLLE04U+UGr4M1D 8JnnYwxfrYHp5uLFJpZzmEjBSCoAEEauKBezlvRTwgTsKKDWy1jzjVUSt Sd4ojH46TEcraQL8lwM0yOwgznoKJFkjIdxlwyfepvmeWbxXVA+YS8bN2 s=; X-SBRS: 5.1 X-MesageID: 65382780 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:TGx1cajPcINpWhRkF3gIMUXbX161cRAKZh0ujC45NGQN5FlHY01je htvDWiHa/7YZWf3fogkPY60801Q7JbSyYNkTwBu+y9gEygb9cadCdqndUqhZCn6wu8v7a5EA 2fyTvGacajYm1eF/k/F3oDJ9CU6jefSLlbFILas1hpZHGeIcw98z0M78wIFqtQw24LhWFvW4 YqaT/D3YzdJ5RYlagr41IrbwP9flKyaVOQw5wFWiVhj5TcyplFNZH4tDfjZw0jQG+G4KtWSV efbpIxVy0uCl/sb5nFJpZ6gGqECaua60QFjERO6UYD66vRJjnRaPqrWqJPwwKqY4tmEt4kZ9 TlDiXC/YVgqMfT0icg8aQYbEiZGJJ8c9LLteFHq5KR/z2WeG5ft6/BnDUVwNowE4OdnR2pJ8 JT0KhhUMErF3bjvhuvmFK883azPL+GyVG8bkmtnwjzDS+4vXLjIQrnQ5M8e1zA17ixLNaiDP JpDNmo3BPjGSwxqP35UGa87oMOloXTwVQZClQ7LpaVitgA/yyQuieOwYbI5YOeiT9hVhEGHj mTI9W/wBlcdLtP34TSf/3+mneOKjS79cIwIGbCy7fIsh0ecrkQBBRELTUCqutG2j0e/X5RUL El8x8Y1hfFsrgrxFIC7BkDm5i7f1vIBZzZOO7I551i124b12RnaB288VwwbZNEM7eZjEFTGy WS1t9/uADVutpicRnSc6qqYoFuOBMQFEYMRTXRaFFVYurEPtKl210uSFYg7TMZZm/WoQWmY/ tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U J4swZL2AAcmV8zlqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1bJxYJ2O4O xCO4Gu9AaO/2lPwN8ebhKrrVqwXIVXIT4y5Bpg4kPIUCnSOSON31H43PhPBt4wcuEMtjbs+K f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq Ig32grj40g3bdASlQGMqd9DdQ1RdCNjbX00wuQOHtO+zsNdMDhJI9fawK87epwjmKJQl+zS+ Wq6VFMew1367UAr4y3QApy/QNsDhapCkE8= IronPort-HdrOrdr: A9a23:6ialU6pqAN9Q8htfNvRwRocaV5vHL9V00zEX/kB9WHVpm5Oj+P xGzc526farslsssREb+OxpOMG7MBThHLpOkPMs1NaZLXLbUQ6TQr2KgrGSoQEIdxeOk9K1kJ 0QDpSWa+eAc2SS7/yKmTVQeuxIqLLskNHKuQ6d9QYUcegDUdAe0+4TMHf8LqQZfngjOXJvf6 Dsmvav6gDQMEg/X4CePD0oTuLDr9rEmNbPZgMHPQcu7E2rgSmz4LD3PhCE1lNGOgk/iosKwC zgqUjU96+ju/a0xlv10HLS1Y1fnJ/ExsFYDMKBp8AJInHHixquZq5mR7qe1QpF6N2H2RIPqp 3hsh0gN8N85zf4eXy0mwLk303a3DMn+xbZuCulqEqmhfa8aCMxCsJHi44cWADe8VAcsNZ117 8O936FtrJMZCmw0BjV1pztbVVHh0C0qX0tnao4lHpES7YTb7dXsMg24F5VKpEdByj3gbpXXd WGNPusqsq+TGnqLkww5gJUsZyRtzUIb127q3E5y4OoO2M8pgE786MarPZv60vouqhNCaWs3N 60QpiApIs+P/P+UpgNd9vpYfHHfVAlEii8Rl57HzzcZdM60jT22tvK3Ik= X-IronPort-AV: E=Sophos;i="5.90,151,1643691600"; d="scan'208";a="65382780" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q6soH5wgoM6FT/v6XNIGsyX1yy7/mPj/2z/us1dT7n2tks2evagVNK8aVUxkyhn7GYTPAMOt1Dz9y+xy9PkmFaHeJjjuyQsOhlVa80+D95G/xJeIsJpkkfAPu4DgDnqZHCVLPdh5AOh2rwc4H4RDaG4xXM+oFNHQDsFFaDg/q3RMHlNfdzy43zSOtb8gU/sqUyTrvUcDFau2e/XMeMccUqkDyxLKXCrY0c6llR/3vrE7KfmL3b/pYueJ0W3LuF5eLp1sKCJwEgwMJlgwmSVU2ucqBhNa5rejPSUsFsZKsea1HwHMwldXcUVMMmGhkb8mujMDj7sjmgPyCLbq9XpgyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HxoMZSpShgbJE4AvL17XZo/c/OpnQlva5Wp9It6p7g8=; b=VtHLzrImjYqAt3FZxwnOYjyWbiuGdOahzlTOE9T541adUesyVqq0/KDCMmG5TwCHVRaRId1aLnyCe40kF7OJZGrEZIi3DND8+RKm9C6JAd0DObidA0LvBwipA9M4BvquXRPxuvTrDXqOEsuajnQCLe1lHvk0Fyk3eKTmja0igM+SkUfJkF0A5SNC/qMAla6j4UlDtR1B+kRM05qlAiDw5fsJgl4cy6OIu83UIa1s2R+U/YQH7EGoAih4LxLCLcbCYJ8I0ZFOEOHMOZP+zo8gHfw8BfbFuKiUSdw9NDq+ijNxk1zIqkk+nsOHVe9o9eHXNlXTdy0+tNiUcokuzJlJBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HxoMZSpShgbJE4AvL17XZo/c/OpnQlva5Wp9It6p7g8=; b=ohKKGqL6ynyFoiY4pcXA1Ps9QPerVeXFj8NNFXqTLiwjMWOctLi2nxS8DIMANgBg3aBcQrpvh8f8/sp01t8OheMmxUyIKfmnCugYwShI0/AOZxpBvLNVfK/gQpbCaDfKz77d53QREVbXWWDUGuqD+eiStSim5spxJIWPrrfV3bA= Date: Thu, 3 Mar 2022 11:39:43 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <202203011540.221FeR4f028103@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO2P265CA0252.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8a::24) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ce43f619-9df3-4c60-6656-08d9fd02237f X-MS-TrafficTypeDiagnostic: BY5PR03MB5156:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WLisN88rwuxDaDFvLjzu3ZB2gOLtbYaCfID5KkOnc9PrH+TEE/fOxAJPx5TIMy/cXiUAjQQLjGDMAG96/X9qdYcAkbqFoKgZqsuvpe+X4Ld4EAbaBZxeJLg9FHmP1jCGk0CJJ2L45sw3Xzx7PkNtp3N6m8iScMHifrD+C3JB00bmD+wEKEugTQ4r8u7iYkZOeayWmTaBl27/XlzaYiBfGTS7LtLLIcVyG/Cy+U0kT4x+k/AOFiWMeU3XTGppQsl0GYOuDPz4r/tECO8CwhkTJ4EI8YZflX54LI+I9MGK0/YTfa/ODS52E+WpZKzST/oKgdWWOtpRxqLTSUrAo7QEGwjV+yUKbWNz/WQmthfnIfPO5KZ+UI+7Sfc13xMaCh8FTIc5DDIzFc5KKoljxSsGLL38ZCDK4g2JxVEpRL2dr19Wf3rNIUJhLOvJUZG5J3xoT9aKPtGhczjp60BPopr1oVO8DR7xKNBNOqGmJ/UhDTMsoCQr8iOSLaQfdtpgI71fY2WPJMMt7QKawBvKbMj+xVadqzfhqrxENuVTMpt46oS2k7v/+KK9Alwe5HqTCTtXTH2zWm0Ami/LWcv5L15QBou8sM1tilMmp7GnAEQb8U4njMaKF12cpZrO33uelj6kbGnDVNLAZZNrgcFxH/BBvmvMYWnw7tYFrjlpDbK3ymup913UoKBkZcQ+mSJj7OAAc4s3yiitCpWyqrTBqOtZsA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(85182001)(6666004)(6506007)(33716001)(6916009)(38100700002)(316002)(8936002)(2906002)(26005)(186003)(82960400001)(508600001)(66476007)(83380400001)(9686003)(4326008)(86362001)(6486002)(5660300002)(6512007)(66946007)(66556008)(8676002)(67856001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MUE5Slk5U3hlaFhIYjNIMzlReFoxbjZ6NFF2K3lvL1htN2lDWVExK0d0UmVl?= =?utf-8?B?RmdjaWNQdjRGVU9FamtuellIYVNpckhvMElDT2kyWFdlZ0hyeHg4R1ZrN0hM?= =?utf-8?B?UW4zMjgvdHhxbWJ2Z1pGQUlwNUduOUpSUVlqRnpaZmJyOWFoMkNmK3BMUWEr?= =?utf-8?B?SjN0RTRuZ3J0RWcvQnVXTzdDOVVWY0JvaVBtWTNaK3NiWDhDbE1aek5EZE9W?= =?utf-8?B?Tis4MEVvQkdFWHVzQUJoZW5CWW8wZTgvMENHL1ppb09xTDlqUGZLejBTeHNo?= =?utf-8?B?VVl3bDNjbCtFMEtXbXJOT0VlamV1VmVqM1hSYlRSOFgwSDJTZXF1MzRxMDEr?= =?utf-8?B?ZWhJMDdUU0RqdHZ2NHd5N0hpZGlwS0h2eFVOQXo1U3E2SVBXVHRCb09hYmtC?= =?utf-8?B?bWNuczFMdGpHNDhyaWdialdXVEZtTzVoaUVZeFZNdGZkRlExdlA1RFd3djZl?= =?utf-8?B?WTFkZTlxR2ZiQ1VGT3BpSHVUam96YWxpa1ZUY2E0QjNDZlFPSGtxem5GNExt?= =?utf-8?B?cDZiK3M1a3FVNXdBelBhWS9SRVlUMmlvM1hlSFgzckZVaUV6VjQvN1RoQ2l0?= =?utf-8?B?OUtHb21TelhJSCtlWWdoOGw3MkhYNm03ckRucVkzSEhGallocTloMGVQNzZ4?= =?utf-8?B?ekRMMlJ2ZVFkSkxyNEVHTVl6Y2lHTVNaR2tBby8wdWlvdS8raTlwTkF4VmtQ?= =?utf-8?B?SjFYNzhjd0prTnA0VFN3bTAyRVltZ0dwb21oUkIzbjdJZUhIY2VEaG81dnNn?= =?utf-8?B?SkJQOW5rNmcwZ2FkSE8yU1UxQzNkaTc0YkpDZ0tzeE1tRkw2dVZTWWt5VmJi?= =?utf-8?B?eXNnWDBNUVo3SXZjTzdVb1dheDQ2OFhoaXpnZVR3czlOZXRhMVJBUjBZYmk5?= =?utf-8?B?VEUxajlWSS95Kzl1YURpRE12RGJraEtmRGVmVkxmNGNpdXUzelYyZlU2UGtT?= =?utf-8?B?Q2VXWXlVWklJVkQ2UlpmaC9NZEZ1N3JkeVlPWVlLaVIvTkhvQk1qQVI3ZVN3?= =?utf-8?B?SkprLzRQSzQ4RnF5Q3dUMk1QYXFoZGIzdjd1OUkxdWlsMTVpakhDaThaUmJv?= =?utf-8?B?MGowSzNUV3RYZjN6ZGdEc2dYOXJHM3pJT0Z0d1JYQWJRalJIRkx1cHZTdmQy?= =?utf-8?B?WTJya2FmWEVpN3o1K09hdEI0SUlLeWc1UU14NXdGbGJqVWhlb1laeWlLZFlX?= =?utf-8?B?ejF6aHRkVzlLUERzSmZJZE9FS1B5ZURMSWdtWVRYb05KVkJ5a2lTWmpVSm5N?= =?utf-8?B?NHpScWN6TmFDVUJzLzhyTXFlQVA3YTJTMDhpMDV3ZkJBWGZUNHQzVlpKOWhr?= =?utf-8?B?b1VPazRQZzJ5eTl2a3hmNFJhSlo0dkJwUVhtZnozVjNEaVJ5SG1JcElDWHd2?= =?utf-8?B?dEQzdFVjdGEwL2tKdy9EWVl2UmQweWxDR2dhSlE0Y3VONVFITXI2cWRvNlV5?= =?utf-8?B?SEZESW1sSnl0a2F0bDV3akgxL1hkbTdEZDlsWEdqTzkyV1M2cnVsdUNpMEFr?= =?utf-8?B?ZXFpVjMxS3hRSkg4VWxqa1NBRThxdnFmNFpkZ2RsajNmUENvakZqRDYrUkw0?= =?utf-8?B?OG01Tm5lRWxNTnBIM05vZ2VQUGp0MWVteGI2SjUwZ3RuajkwU3o3YlA3M1F5?= =?utf-8?B?eXVSeDc5T21TNVA2OG9WR0hCQkRGRTRzbUxEVklXVFNvZ3RzY0NOVVBhU2hu?= =?utf-8?B?dkhVMVZxaTVPRnFIOU94VXVQWjRvamhyLzhScjdwclR5by9HRTdzamprSFpm?= =?utf-8?B?K0YwcG5XeE1UUkptcm0yZThobWVONWx5b2xEOTdlZmIwWDJxblo5RzYxaENk?= =?utf-8?B?eFozZlJJNldZekYzUFB1bktZV0tvSUt0VjlZZ0phWG5IZzZnRk1SRm15RXdL?= =?utf-8?B?Wk9rTHFLbkJOcFRpS2I4VzZrdlcrKzZqeVFrOXlHeG50TUZNN25raXQrbTBw?= =?utf-8?B?R2xrMlVRTGhhNVZRSm1qYUJTSEw4TmFwSEcxUjJ2Q0p3c29ndVROWkhuN1JG?= =?utf-8?B?S29HbzRBNEwvcWd3enZ3UlhHT1hyZFhvL3VxMXNEVUpQVTV4SVozcStjTk16?= =?utf-8?B?TVZJd0dVQ050WkgwU1J1bEpLTDBpckhyQkZUS09aV1hiR1dwMHRaTnVqRW9h?= =?utf-8?B?Q0IxU1RPd3pvNTJZWlUxUUlibjlucU1tU3QrQ0EyQzUwQkU1U0t5T0xTSklj?= =?utf-8?Q?f7KBPMMdaXAeSZCnuEpKJ0A=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: ce43f619-9df3-4c60-6656-08d9fd02237f X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2022 10:39:47.9136 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gFG1JGFzKjSKTtB0k73b/iOcph6e1fv3asAiU6NTujhFL8ULPz5RWSqsNijNS0x5yIx7+SaLtgmTq2/6eCf3IA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR03MB5156 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4K8SDM6TyTz4jsB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=NEAIt31B; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=ohKKGqL6; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=05474af58=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=05474af58=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=05474af58=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.145.153:from]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 02, 2022 at 07:26:18PM +0200, Ze Dupsys wrote: > Today managed to crash lab Dom0 with: > xen_cmdline="dom0_mem=6144M dom0_max_vcpus=2 dom0=pvh,verbose=1 > console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 > reboot=no" Hm, it's weird that reboot=no doesn't work for you. Does noreboot instead make a difference? > > I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as > in attachment, panic was with the same fingerprint as the one with > "rman_is_region_manager" line already reported. > The scripts i ran in parallel generally were the same as attached in bug > report, just a bit modified. > 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and > instead of 2GB, it writes 4BG in each ZVOL created dd if=/dev/zero) > 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write > 3GB in it's /tmp, restart DomU, removes /tmp, repeat) > 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 > disks attached, turn off DomU, repeat) Right. So the trigger for this seem to be related to creating (and destroying) VMs in a loop? Do you still see the same if you only execute steps 1 and 4 from the repro described above? > 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. > > Around dom id 108, system started to behave suspiciously, xl list showed > DomUs created, but they did not really start up, script timeout-ed for ssh > connection, no vnc. When i did xl destroy manually, and xl create, system > panic happened. Could you also add the output of `top -n1` to see where memory is going? I'm quite sure we have a leak in some of the backends, maybe the bounce buffer used by blkback. Thanks, Roger. From nobody Thu Mar 3 17:03:17 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 68DA119E05C3 for ; Thu, 3 Mar 2022 17:03:26 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K8ckj11KLz4XLS for ; Thu, 3 Mar 2022 17:03:24 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 223H3Ij1015194 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 3 Mar 2022 09:03:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 223H3Hl0004266; Thu, 3 Mar 2022 09:03:17 -0800 (PST) Message-Id: <202203031703.223H3Hl0004266@nfbcal.org> From: Brian Buhrow Date: Thu, 3 Mar 2022 09:03:17 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Roger Pau =?utf-8?B?TW9ubsOp?= , Ze Dupsys Subject: Re: ZFS + FreeBSD XEN dom0 panic Cc: , buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 03 Mar 2022 09:03:18 -0800 (PST) X-Rspamd-Queue-Id: 4K8ckj11KLz4XLS X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.29 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.993]; MLMMJ_DEST(0.00)[freebsd-xen]; FREEMAIL_TO(0.00)[citrix.com,gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org hello Roger. I wonder if this leak is related to a bug I ran into a few weeks ago, but didn't investigate. Traditionaly, when I shutdown a domu, I use the xl destroy command because the xl shutdown hasn't been very reliable for me with the domu's I've been using. However, I thought I'd try things out with the new version of Xen plus the newer versions of NetBSD, which was the domu in question for this issue. When I ran the xl shutdown command against the domain, the domain shut down, but didn't disappear from the list of domains when subsequent xl list commands were issued. Worse, following up with an xl destroy against the zombie domain that resulted from the xl shutdown command did nothing. Because this was a production system, I immediately rebooted the dom0 and all the domu hosts in order to assure the corruption didn't spread. this is against Xen-4.14.0 and FreeBSD-12.2. Have you seen this behavior before? -thanks -Brian From nobody Sun Mar 6 12:41:17 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9C47D19F0C50 for ; Sun, 6 Mar 2022 12:40:57 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KBLmS6ldsz3JN3 for ; Sun, 6 Mar 2022 12:40:56 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x130.google.com with SMTP id b5so1902315lfs.1 for ; Sun, 06 Mar 2022 04:40:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=P8yvZ+YjMdFDsg2FHN5VndGB6cm25YZpS4UNC3Kuv/k=; b=loc9eon6ely+CBrX8a1pWklybKuOCip+mfhVVck8Tg5bU7x9itQN/pqYqZcZFNuHZS nv4h2mQr0nH7CQTBncFKjqdW68hIGBHeSWsNHDz1/jRzu9w8REzNdmzVTCTL0iXmKKpE qlz4QWh7og05sdTcdZ39ccIraBmTMDH8ZIL63jOP/lkU/689FHT9nB6krd8mCXITx9UB dU2wAw+mAGrMrmqp1OdXtWDRFBGN+ThqxC2AsGyNb2Lsici8OX0X5On/nfIFAyjVUpbz RuF+aNXzfw+v0bFPvW0xONtDvzA8mxMREH4VDFlKf/cK4MzpFnsZwWuHjpdbyXX1TvLv 7elg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=P8yvZ+YjMdFDsg2FHN5VndGB6cm25YZpS4UNC3Kuv/k=; b=U9Tb9tqipdQpnj6b18Xjzp7kjIzdhHF31K7jeJwT0UkuAcaYllj/XEoOV/ggyJ2D9q gywlcpVKiEXw9kfR0z/JK2zTFAbC74od1LrhAf0m8vdgAQXk6aOgaUwmZtRUrqryv2p3 8BUZEK1nlAximq1kN7KDo5lRe6n5D9mLr0U7jShdBtIvfkp7FIN5DHqWBvjJ/9Yt0DM0 tuj0PWabpiveTKtgPh0pVNIB66Ih+zlCcUS8KDB++IixqDh8DVqnAJUnqIYEsfQ3+oAY 1MDuQhAl1ESi17/tQQa2HzQdOyFXbT9W62A07EAZ7zSiG4BbeIfd3xtBdfOHFfV6bfm2 u/4g== X-Gm-Message-State: AOAM5330pTxvjhztQMqjMD2Zy2eNI0g+zKyrf20LBnBNUK55NUeVo1QH dMceXelZZhPUyFBqiuiNhlM= X-Google-Smtp-Source: ABdhPJwFTHJXX3fkjVy4idkYor29kHzYhnVtH70TTM1xcf6mpasVNwGl11tdVcez6TcrRrNRmcPjbA== X-Received: by 2002:a05:6512:2082:b0:443:4236:5f57 with SMTP id t2-20020a056512208200b0044342365f57mr5082551lfr.335.1646570449144; Sun, 06 Mar 2022 04:40:49 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id u2-20020a2e9b02000000b002460fa70fa7sm2402887lji.59.2022.03.06.04.40.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 06 Mar 2022 04:40:48 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <202203011540.221FeR4f028103@nfbcal.org> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> Date: Sun, 6 Mar 2022 14:41:17 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: lv X-Rspamd-Queue-Id: 4KBLmS6ldsz3JN3 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=loc9eon6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::130 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-0.49 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.32)[0.320]; NEURAL_HAM_LONG(-0.64)[-0.645]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(0.83)[0.833]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::130:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022.03.03. 12:39, Roger Pau Monné wrote: > On Wed, Mar 02, 2022 at 07:26:18PM +0200, Ze Dupsys wrote: >> Today managed to crash lab Dom0 with: >> xen_cmdline="dom0_mem=6144M dom0_max_vcpus=2 dom0=pvh,verbose=1 >> console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 >> reboot=no" > Hm, it's weird that reboot=no doesn't work for you. Does noreboot > instead make a difference? > >> I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as >> in attachment, panic was with the same fingerprint as the one with >> "rman_is_region_manager" line already reported. > > >> The scripts i ran in parallel generally were the same as attached in bug >> report, just a bit modified. >> 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and >> instead of 2GB, it writes 4BG in each ZVOL created dd if=/dev/zero) >> 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write >> 3GB in it's /tmp, restart DomU, removes /tmp, repeat) >> 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 >> disks attached, turn off DomU, repeat) > Right. So the trigger for this seem to be related to creating (and > destroying) VMs in a loop? > > Do you still see the same if you only execute steps 1 and 4 from the > repro described above? > >> 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. >> >> Around dom id 108, system started to behave suspiciously, xl list showed >> DomUs created, but they did not really start up, script timeout-ed for ssh >> connection, no vnc. When i did xl destroy manually, and xl create, system >> panic happened. > Could you also add the output of `top -n1` to see where memory is > going? > > I'm quite sure we have a leak in some of the backends, maybe the > bounce buffer used by blkback. > > Thanks, Roger. Hello, About XEN command line option changing "reboot=no" to "noreboot"; this did not change anything. Physical machine reboots on kernel panic. Maybe i have misunderstood the use for this option. I somehow thought that reboot=no means that if Dom0 crashes/panics, then physical machine is not rebooted, but in reality it does reboot. At this moment though, this is not important for, since i am recording all console output from serial anyways, so messages are not lost as long as they are flushed out on a timely manner. Maybe kernel itself after panic calls reset to XEN? If i execute steps 1 and 4, it seems that machine does not crash; tested 8 hours Dom0 8GB RAM, 4 hours Dom0 2GB RAM. So creating a lot of ZVOLs and monitoring it does not seem to be the cause. Yesterday was a happy and a sad day at the same time. The happy thing is that i think that i have understood one panic reason and somewhat better located where  one problem might be, but at the same time, i think i have located 2 bugs that cause one specific panic, but 2 other panic reasons are due to different circumstances. Further is my pure speculation about two things, somewhat related; i do not know XEN or kernel internals, thus take this with grain of salt. Knowing that this seemed to be RAM problem and thinking how ZFS sysctl tuning values differ, i started to look at "sysctl -a" diffs for Dom0 with 2GB RAM and 8GB RAM, by pure luck because comparing Dom0 that was running longer and just recently restarted, i noticed there are diff lines like so: .. +dev.xbbd.202.%driver: xbbd +dev.xbbd.202.%location: +dev.xbbd.202.%parent: xenbusb_back0 +dev.xbbd.202.%pnpinfo: +dev.xbbd.202.xenbus_connection_state: Closed +dev.xbbd.202.xenbus_dev_type: vbd +dev.xbbd.202.xenbus_peer_domid: 309 +dev.xbbd.202.xenstore_path: backend/vbd/309/51728 +dev.xbbd.202.xenstore_peer_path: /local/domain/309/device/vbd/51728 +dev.xbbd.203.%desc: Backend Virtual Block Device +dev.xbbd.203.%driver: xbbd +dev.xbbd.203.%location: +dev.xbbd.203.%parent: xenbusb_back0 +dev.xbbd.203.%pnpinfo: +dev.xbbd.203.xenbus_connection_state: Closed .. although actually in system at the time there were only 2 DomUs running max. This seems to be the reason why this felt like memory leak, since after long time of start/stop VMs, most probably sysctl hit some sort of RAM limit (which most probably is auto-calculated from total RAM somehow) and panics. I can repeat this sysctl growing with just starting/stopping VMs in loop although stressing ZVOL at the same time speeds this up for unknown reasons to me. What i do not know though is what exactly triggers this, because there are times when xbbd.$number does not grow larger than total number of disk devices attached to all DomUs. I do not know what would be the best way to continue with this. Then i caught another sysctl variable that is growing due to XEN, "kern.msgbuf: Contents of kernel message buffer". I do not know how this variable grows or by which component it is managed, but in VM start/stop case it grows and contains lines with pattern like so: .. xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet xnb(xnb_dump_txreq:299): netif_tx_request index =0 xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0 xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 xnb(xnb_dump_txreq:302): netif_tx_request.flags =8 xnb(xnb_dump_txreq:303): netif_tx_request.id    =69 xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000 xnb(xnb_dump_txreq:299): netif_tx_request index =1 xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255 xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 xnb(xnb_dump_txreq:302): netif_tx_request.flags =0 xnb(xnb_dump_txreq:303): netif_tx_request.id    =0 xnb(xnb_dump_txreq:304): netif_tx_request.size  =0 .. Those lines in that variable just keep growing and growing, it is not that they are flushed, trimmed or anything. Each time i get the same message on serial output, it has one more section of error appended to "same-previous" serial output message and sysctl variable as well. Thus at some point serial output and sysctl contains a large block of those errors while VM is starting. So at some point the value of this sysctl could be reaching max allowed/available and this makes the system panic.  I do not know the reason for those errors, but actually if there was a patch to suppress them, this could be "solved". Another diff chunk might be related to this: +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0 +dev.xnb.1.xenbus_peer_domid: 7 +dev.xnb.1.xenbus_connection_state: InitWait +dev.xnb.1.xenbus_dev_type: vif +dev.xnb.1.xenstore_path: backend/vif/7/0 +dev.xnb.1.dump_rings: +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error: nr_reqs == 0 +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup, &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0 +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup, xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0 +52 Tests Passed +1 Tests FAILED What was suspicious about this is that i am using 13.0-RELEASE-p7, it is not a DEV version of XEN or anything, but there are sysctl variables with "dev" prefix. Could it be that ports have been accidentally compiled XEN version with some development flags turned on? So now it felt for me that i understand this panic case: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address   = 0x22710028 fault code              = supervisor read data, page not present instruction pointer     = 0x20:0xffffffff80c45892 stack pointer           = 0x28:0xfffffe009656a930 frame pointer           = 0x28:0xfffffe009656a930 code segment            = base 0x0, limit 0xfffff, type 0x1b                         = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags        = interrupt enabled, resume, IOPL = 0 current process         = 17633 (devmatch) trap number             = 12 panic: page fault cpuid = 1 time = 1646139877 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff8108b1a7 at trap+0xbc7 #4 0xffffffff8108b1ff at trap+0xc1f #5 0xffffffff8108a85d at trap+0x27d #6 0xffffffff81061b18 at calltrap+0x8 #7 0xffffffff80c62011 at rman_is_region_manager+0x241 #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 #9 0xffffffff80c1949c at kernel_sysctl+0x43c #10 0xffffffff80c19b13 at userland_sysctl+0x173 #11 0xffffffff80c1995f at sys___sysctl+0x5f #12 0xffffffff8108baac at amd64_syscall+0x10c #13 0xffffffff8106243e at Xfast_syscall+0xfe The function "kernel_sysctl" shows that this is sysctl related, and based on previous knowledge we know that there might be too many variables. If i assume that "Xfast_syscall" is a XEN function, then that shows that panic came from XEN call that wanted to do something to sysctl variable, and panic happened. This is all just my assumption. So those were my conclusions. What do you think? How should we proceed with this? Should i try to somehow build XEN from git sources? For what it's worth, system was crashed with 8GB Dom0 RAM, 16GB swap space on dedicated HDD not the one ZFS is on. xen_cmdline="dom0_mem=8192M dom0_max_vcpus=2 dom0=pvh,verbose=1 console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 noreboot" "top -n 1" output before panic: last pid: 18906;  load averages:  1.39,  2.62,  2.90  up 0+02:22:00    19:54:50 42 processes:  1 running, 41 sleeping CPU:  3.0% user,  0.0% nice, 27.7% system,  2.3% interrupt, 67.0% idle Mem: 872K Active, 4176K Inact, 4356K Laundry, 7656M Wired, 48M Buf, 238M Free ARC: 5183M Total, 6320K MFU, 4752M MRU, 219M Anon, 192M Header, 12M Other      4674M Compressed, 4746M Uncompressed, 1.02:1 Ratio Swap: 16G Total, 40M Used, 16G Free   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME WCPU COMMAND  1010 root          1  23    0    13M  3044K select   1   1:34 0.98% xenstored At this point i do not think "vmstat -m" output before panic is useful. What i can tell though is that grep'ing out vmstat messages on process "sysctloid" i see that it's RAM consumption most of the time is steadily growing. The sad part about all this though is that i do not believe that these are the bugs that cause panic on my pre-production machine. Maybe sysctl variable "kern.msgbuf" overgrows, but other than that, normally i do not start/stop VMs so often that it would create so many xbbd sysctl variables... Anyways we could try to deal with this and then see, if it fixes. Best wishes! From nobody Wed Mar 9 08:42:52 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4A32119F54D2 for ; Wed, 9 Mar 2022 08:43:18 +0000 (UTC) (envelope-from prvs=060736ad3=roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KD5Ls2LPkz4chc for ; Wed, 9 Mar 2022 08:43:17 +0000 (UTC) (envelope-from prvs=060736ad3=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1646815397; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=QeIrA3wDkLEZWj7BlQHP8axBCnpd4zlS3RZLUD2mM/0=; b=SauCoWjCTZgSWkKoEny5C+BP6jto7I7qciGV/i+s4Q4x9uGmd5j9Zc65 1HFFCiixqbENOZd0u4Vbi4Eambu9ABRW6XAJM9xxlvQvoTiKq9dQxexJr sZjRSrHZufKsMXF3zQoDo961KB4StkSv/z0w8DRXKb6M7Wwd9e2Xnp+gy o=; X-SBRS: 5.1 X-MesageID: 65792149 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:L//XdK3ZZMMrgjt0U/bD5cJxkn2cJEfYwER7XKvMYLTBsI5bpzACm 2NOCz+HPPbbYzOjetByaYiyp0wGu5aHndFkSwJvpC1hF35El5HIVI+TRqvS04J+DSFhoGZPt Zh2hgzodZhsJpPkjk7xdOCn9xGQ7InQLlbGILes1htZGEk1EE/NtTo5w7Rj2tUw0IDja++wk YiaT/P3aQfNNwFcagr424rbwP+4lK2v0N+wlgVWicFj5DcypVFMZH4sDfjZw0/DaptVBoaHq 9Prl9lVyI97EyAFUbtJmp6jGqEDryW70QKm0hK6UID66vROS7BbPg/W+5PwZG8O4whlkeydx /1BupubZh8GDpTss8Ytbzt8C3kvL4tJreqvzXiX6aR/zmXDenrohf5vEFs3LcsT/eMf7WNmr KJCbmpXN1ba2rzwkOnTpupE36zPKOHxO4wSoDd4xCzxBvc6W5HTBa7N4Le02R9u258fR6+CN qL1bxJLUx3NIDZtA20IS64gl+2QimfmKzpX/Qf9Sa0fvDGIkV0ZPKLWGNDIesaNWu1TkEyTr 2aA+H72ajkULtGSwyGJt2KhgMfBhy72UpkeUrqi+ZZCklCf23QIGQY+Wl6yoP3/gUm7M++zM GRNpHBo9/JrshX2EJ+tBHVUvUJooDYQVN17PPUV2Dit4Yjk3yS9HjBVXBhOPYlOWNANeRQm0 VqAntXMDDNpsaGIRX/1yop4vQ9eKgBOczZcOHZsoR8tpoC6/dpt1k6nosNLTfbt5uAZDw0c1 NxjQMIWo7wIxfAG2Kyglbwsq2L9/8OZJuLZC+i+Y45E0u+bTNP9D2BLwQKChRqlEGp/ZgPf1 JTjs5LChN3i9bnXyESwrBwlRdlFHcqtPjzGmkJIFJI87Tmr8HPLVdkOvG8hehszYppYJWGBj KrvVeV5vs870JyCN/MfXm5MI55ykfiI+SrNDZg4keaikrAuLVTarUmClGab3nz3kVhErE3ME czzTCpYNl5DUf4P5GPvH481iOZ3rghjlTK7bc2qlHyPjOvBDEN5vJ9YaTNimMhit/jayOgUm v4CX/a3J+J3C7SvPHOIrdZNcTjn7xETXPjLliCeTcbaSiJOE2A9Ef7Bh7Qnfo1uhaNOkenUu Hq6XydlJJDX3BUr9S3ihqhfVY7S IronPort-HdrOrdr: A9a23:YFns5qioKfq/+tm2nWEkweizoHBQXzh13DAbv31ZSRFFG/FwyP rAoB1L73PJYWgqNU3I+ergBEGBKUmskqKdxbNhR4tKOzOWxVdATbsSlrcKpgePJ8SQzJ8+6U 4NSdkaNDS0NykHsS+Y2njILz9D+qj/zEnAv463pB0MPGJXguNbnn9E426gYzNLrWJ9dPwE/f Snl656T23KQwVpUi33PAhMY8Hz4/nw0L72ax8PABAqrCGIkDOT8bb/VzyVxA0XXT9jyaortT GtqX2y2oyT99WAjjPM3W7a6Jpb3PPn19t4HcSJzuwYMC/lhAqEbJloH5eCoDc2iuey70tCqq iGnz4Qe+BIr1/BdGC8phXgnyHmzTYV8nfnjWSVhHPyyPaJMw4SOo5kv8Z0YxHZ400vsJVXy6 RQxV+UsJJREFfpgDn9z8KgbWAkqmOE5V4Z1cIDhX1WVoUTLJVLq5YEwU9TGJAcWArn9YEcFv V0Bs203ocbTbqjVQGZgoBT+q3tYpxqdS32AXTq+/blngS+pUoJgXfxn6ck7zU9HJFUcegx2w 2LCNUsqFh0dL5kUUtMPpZwfSKJMB2+ffvtChPlHb21LtBPB5ryw6SHlYndotvaPKA18A== X-IronPort-AV: E=Sophos;i="5.90,167,1643691600"; d="scan'208";a="65792149" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b4tmmc9wWDWulxZ6yu5gQCDYz06ORbkM2P7N+SNazFU15OXyWCW1wIbBvVZ2KINbUtdrntcFmgd5c+ya3z0XiEeVvJklsIVWdGFkwWifWmKlJSq4Eb7LbZLlafDlyKbTI24VqwzTfpUs3SbmKNQLXMT5Zik8yMAuxqM8oVpABkLkGnG4BNwjOIbk+drz2x2IZFvHHfcP1987XllSGszcJp+sVfMEPNK9dO5lwMTPJ9IwCn8rf/PG4KpbKwQnyYdFQNUpvo9woNAeQnw4UHrX36ONuFlgGFsLud+h6Pj3wLmxBeanoC3JDTPSwiEkJZK3XZ6sT4q0N0sTpn4MzqzQ9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xws++0D3Er4h27M4UHFFgcCAi73WjA8C61I1B0L3msk=; b=csX7Oj6iOhgomMShAHnk66CB4a2eScFz2hc71mkGBIkcRrIpV4Nexmifk/9dYZ8a3aGkq6u1WqWbRcJdL+0uk4TL+JEcvK9RTeNzWUz/swt+4zm3DMW17eQRMEw/+hbs0dzLMjozCRAIkvcsezqbvsDXQc4yFDJuoImaCcqg4aG4dpUyWk+JKyRN0+3oWIY1tfr7ItO2oqNx3pCNY2lQjWzqLRVpmyY+SNQ0r+R9R7DhLgGWRBAY+UHyIIT9g3DNroLpb6Uh6XxFyTOsThGsU/7LYpUhmIqGpQ+4+EM1JDCxZXgiSxj8gYeYq6ofVuxMLdRTAMAZGxItgmE4DZZHSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xws++0D3Er4h27M4UHFFgcCAi73WjA8C61I1B0L3msk=; b=aBXP3QHlTfORWtaYYOqKgeATHZ7EzEy82zddIJ178/N9aGuWWhRfjHMnqY8IEBujcGfyLkOx8ORm4VkSLiRJ+yt9aBjW0fdCMBlskDhGoeDLW3g8P1sexAXD5zALr6KK1wdNsIm0Z55I8cPrllBYNTERu1noEbrStHyayP6qu8U= Date: Wed, 9 Mar 2022 09:42:52 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <202203011540.221FeR4f028103@nfbcal.org> <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> X-ClientProxiedBy: MR2P264CA0076.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:32::16) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 67b2ff13-c7ab-4651-645f-08da01a8d0bf X-MS-TrafficTypeDiagnostic: BN8PR03MB4594:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9VnbjAjmt34Cf0djWqZA6CNXEIx/c7BGRSChqgjkgt4AfsxJI3AcU8lM/Uhqj1X0P9eRPSRa77nF5GIzCQoHXGcDLt2zgWac6G1jOc8mtkfDXPLV5da3K4z8P3pfu7LhC8OnyFlmKsVcjohPFYO2wo5xzpc2cqboiBdkqrNDQ78TMK3QAYXd+1s+E1hnZc7B9eo2WNto/G6vGtwEW1YcgnMgoZhWdFjwbP5iBs5ntL79NrH0n0ND9ZtSBOYrv/rNVdU6CpF8UOLI1ZbcYYtdM5efG3WGpV46mT3+0PRXmjAMVmw5Q3dqOrfLNe4zKrHuqn2ostioixcEacKSgg5TwNtAXYiU8azJZzc68qMo80WiNceXmrtmbepFJrtXanUbnrHDWlyvS+wd4WN6VBBkGKAroVqM6XbZwwEHVR23ES+CtfMfuD2illQ7qgkAOdvRWks7+q5VJfSrSKJ/svSSmBbAIdeOCErHObkLbc5tmQbuX7eeWF16KvbgVc05lIw5kjjF1VZBVyENpu73x9bOzVvuKNtf7PlZcezpimUX4RcQ5/sXhcdqVC2P5bAyLG3BSZUeXIjEe6cNWR0iIimjcalpGkLz5z2Yim+/5FivPg3FwVMqSuIbkEmy2ht8GXU6KidVsgxJjaAoLUWTSJ+XQwOST6jPxH8h4yXoWFBiFiHbF6M01MkpYywaWGITzNmKT9l76an4eJo0nSrEI6ni/1HriZRnPRKLJAyBfYGrArQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(85182001)(508600001)(66556008)(66476007)(66946007)(6486002)(5660300002)(38100700002)(33716001)(83380400001)(86362001)(8936002)(4326008)(8676002)(316002)(6916009)(30864003)(6506007)(2906002)(82960400001)(6512007)(186003)(6666004)(9686003)(26005)(522954003)(21314003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MFVqbEUycTQvV0VKWEdjV1VYeU5TVVU1V20zcy96cXl3RTBsdTZKblpsOXJk?= =?utf-8?B?M0l1N0c1K1VRci80T0dmSjFLM1BGZmlKaW1WYVhyb1JhZ09MVlJHZG5YbVVh?= =?utf-8?B?Y1k2c05zSXN4TjlwSi8zNWRLVmZ1WVpMbitsQXROVzdVWWdDaEliQy9NaHM5?= =?utf-8?B?UFhPbnBETEdRUEJrTnJoRStFNlVHRHVlVTIvN0o1aXlXYXBUU3dUb3FMZjZn?= =?utf-8?B?ZWpPNmlDSjJqZnZNUUVvelQ3Si81UnlEcVhpSXdkS05FQTVPc2RaU09IMWFI?= =?utf-8?B?YWtOM3loY29FcS9iSkU4eFRMbnhLYnpETG5UM1FvQTZlWmFmQmxyYlNaMUJC?= =?utf-8?B?ekpwZStpcWlQTXAzdDFUZFJabis4eVZqY1M1K1Z2Y3V1NDloRnNOanFBVnBV?= =?utf-8?B?c3E1aDZhWXNlQnZaSmh5L3BhSFJ4OG4zRVduS1NPVmRPUWV2R0UxbitxRWRE?= =?utf-8?B?QytXbE9iMlhORjh5SnZKUDVJTW9jODdnRUtnWHJrRTlIUG9RS3JMZHJVT1Nn?= =?utf-8?B?TlN2eXZoeFdOKytqa2VXdWhKM2VOalMwa0I1RnYyLzdscmtjQlVlTnVzMlVr?= =?utf-8?B?QnM3Q0Q3L2dZRFZONkVwNEl4eFl6Z2ZBa0loVXNtMFUra2dCRlFvRkRLTUYr?= =?utf-8?B?eGRucU5rYW9qczRsK0hrZXI1YVQ5c3lUdzJZbDdQRldMYUpSaC9Cd1VGVG9P?= =?utf-8?B?NzRPYWxNZzVHazJDaGI5Qmc1QkRxeGJhZzJaVnppQ1o1dytIaG5aVkJ0K1ZQ?= =?utf-8?B?dmFzZVEzcU9kLzBMbit1aTI0MHBMV1JYWW5ZMVlRSXpWYlFpSnk4ckx0SE1H?= =?utf-8?B?Ujl2dEpZSGl0K2VtSzB5ZDQ1elZNQWMxcDl4cmdEMEJreTkyS2xKUW0xdnd1?= =?utf-8?B?QjZ6VXZMV0ZKQ2ZwNHRxNU5yU2NucG9BY0RqZ0ZhbTlGL1h0VmphTWhydzVp?= =?utf-8?B?aVJVV2xtVVl4UnB2RkxQc0o0NCtUdUs4eUcrbFVwd0hzRTJpbWRBdVZPR1RH?= =?utf-8?B?ZjBrNTJZNEt5Y1R0RGNhNUE0QlVaWFNiRkZwOVk1aUNrcHNndDNJbGxwZWlv?= =?utf-8?B?WHZtaFluTnRPb1RsOTFWL2NMK0l6YkxQVHcrYlVINTNQb0o0Zmw3cEl4cDgz?= =?utf-8?B?enhBYmVvRmJQQmkvdjI3ZHAzdkVISWtlRzM4TFI1cllYSUtnVG5pbis2NGth?= =?utf-8?B?c21jOXdzcGFUZk4yWmo5SDQrSjhzV1FPd1NBWU9kbXpKTVRBSFpDTTdYdUFB?= =?utf-8?B?Ky9TTXUwT2xWLzhaaUsrYUxiTXZqT0h6SHFvUm80NW96UUJqeGVLQVRiNWZN?= =?utf-8?B?UHFja2dUZWx1eno1MDBpUW1SeS92SkdQOTcvL0pHVysyYmJwdXN6WGRwMWJS?= =?utf-8?B?YTEzeUsyVDlZWjhPZWl2ZWtyM29LSlVJSDYvN2k2SDZqU2pvVGpJSkRLTXBl?= =?utf-8?B?cUVjdkx5cXpzRUVzTVI3NkluNEJ6TWxvanRyZFU1QVJJSnRFQWtCdlpEcU5l?= =?utf-8?B?RkhPTVZtdjkxUTJqeFBLK0djT0dsdFNRVUVKaEVPUkl6c3ArdDFPclhHdjZN?= =?utf-8?B?czVtYmRhelhrdVlLZElLSDZvbUlOeW9HWDVLMGN2cmFQRzMzMmt3ajFqaGk2?= =?utf-8?B?VzJnWERCemVQL1I4ZnFMN0wyankyeWZlbHd1bXFha1VLcnVTU0FwdHBxZkVX?= =?utf-8?B?YUNxakVHQ1RkenJ5cElVQm5BaXl4L0FrRFZ0a0RmcVNCMlFhWjZCYVNpUlcw?= =?utf-8?B?aGlOMVh6cUs4MVVJU1F5ajBRNFowakc1MWlnQTUzWkR3bUEzUkhtQlQvb3Ux?= =?utf-8?B?L1dSTVBneTE4NDhoVFMzNktZeENpRnZ4Wng1eWczQUxQNmhsU09KQUhBeDNS?= =?utf-8?B?c2xYbnhvQXozcWZORlpPM2hnL2RDUEQ5TUEyOU40Y2hBL1VGVzlNbFJpdTBM?= =?utf-8?B?MEIvdDVFbm10Ylc1RlFQY1dvQTlIN2Z0L0dTRmcwZEVTeTdTajYyNFBSN2NS?= =?utf-8?B?WkM5T1cxYjk5c0dUaU5taXFhRk9nMG9TL1VJMkcxR3BmdU0rd0xuK0xPTDRV?= =?utf-8?B?UTVjdDV5T0N5ZW1la0JVZDhSZTNFaGNZc0FkVmJxcXRmVDZ3SkRZQkZ1K3JD?= =?utf-8?B?aTNpRmhRc25Wb0ZqWkhQcFhWRUI2N1lOUWVCR0tJSXdXWXNHb0FWODZHOEhx?= =?utf-8?Q?vZjr9/B77IR72kofOU8mUdo=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 67b2ff13-c7ab-4651-645f-08da01a8d0bf X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2022 08:42:59.7027 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4yRBhnthTyopR6vFugCuDFZVPPg6Isquuf/ccluMfa3AHG+fuRJtKwgaJ+JyTD4nykQqNvhTBcvY+ERQUmOK9A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR03MB4594 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4KD5Ls2LPkz4chc X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=SauCoWjC; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=aBXP3QHl; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=060736ad3=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=060736ad3=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=060736ad3=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.145.153:from]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote: > On 2022.03.03. 12:39, Roger Pau Monné wrote: > > On Wed, Mar 02, 2022 at 07:26:18PM +0200, Ze Dupsys wrote: > > > Today managed to crash lab Dom0 with: > > > xen_cmdline="dom0_mem=6144M dom0_max_vcpus=2 dom0=pvh,verbose=1 > > > console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 > > > reboot=no" > > Hm, it's weird that reboot=no doesn't work for you. Does noreboot > > instead make a difference? > > > > > I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as > > > in attachment, panic was with the same fingerprint as the one with > > > "rman_is_region_manager" line already reported. > > > > > > > The scripts i ran in parallel generally were the same as attached in bug > > > report, just a bit modified. > > > 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and > > > instead of 2GB, it writes 4BG in each ZVOL created dd if=/dev/zero) > > > 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write > > > 3GB in it's /tmp, restart DomU, removes /tmp, repeat) > > > 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 > > > disks attached, turn off DomU, repeat) > > Right. So the trigger for this seem to be related to creating (and > > destroying) VMs in a loop? > > > > Do you still see the same if you only execute steps 1 and 4 from the > > repro described above? > > > > > 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. > > > > > > Around dom id 108, system started to behave suspiciously, xl list showed > > > DomUs created, but they did not really start up, script timeout-ed for ssh > > > connection, no vnc. When i did xl destroy manually, and xl create, system > > > panic happened. > > Could you also add the output of `top -n1` to see where memory is > > going? > > > > I'm quite sure we have a leak in some of the backends, maybe the > > bounce buffer used by blkback. > > > > Thanks, Roger. > Hello, > > About XEN command line option changing "reboot=no" to "noreboot"; this did > not change anything. Physical machine reboots on kernel panic. Maybe i have > misunderstood the use for this option. I somehow thought that reboot=no > means that if Dom0 crashes/panics, then physical machine is not rebooted, > but in reality it does reboot. noreboot (or reboot=no) prevent Xen from rebooting when Xen crashes. If it's FreeBSD dom0 that crashes it's dom0 the one that's triggering the reboot. Can you try to set the sysctl kern.panic_reboot_wait_time = -1 on FreeBSD? > At this moment though, this is not important > for, since i am recording all console output from serial anyways, so > messages are not lost as long as they are flushed out on a timely manner. > Maybe kernel itself after panic calls reset to XEN? Yes, I think that's the case. > If i execute steps 1 and 4, it seems that machine does not crash; tested 8 > hours Dom0 8GB RAM, 4 hours Dom0 2GB RAM. So creating a lot of ZVOLs and > monitoring it does not seem to be the cause. > > Yesterday was a happy and a sad day at the same time. The happy thing is > that i think that i have understood one panic reason and somewhat better > located where  one problem might be, but at the same time, i think i have > located 2 bugs that cause one specific panic, but 2 other panic reasons are > due to different circumstances. > > Further is my pure speculation about two things, somewhat related; i do not > know XEN or kernel internals, thus take this with grain of salt. > > Knowing that this seemed to be RAM problem and thinking how ZFS sysctl > tuning values differ, i started to look at "sysctl -a" diffs for Dom0 with > 2GB RAM and 8GB RAM, by pure luck because comparing Dom0 that was running > longer and just recently restarted, i noticed there are diff lines like so: > .. > +dev.xbbd.202.%driver: xbbd > +dev.xbbd.202.%location: > +dev.xbbd.202.%parent: xenbusb_back0 > +dev.xbbd.202.%pnpinfo: > +dev.xbbd.202.xenbus_connection_state: Closed > +dev.xbbd.202.xenbus_dev_type: vbd > +dev.xbbd.202.xenbus_peer_domid: 309 > +dev.xbbd.202.xenstore_path: backend/vbd/309/51728 > +dev.xbbd.202.xenstore_peer_path: /local/domain/309/device/vbd/51728 > +dev.xbbd.203.%desc: Backend Virtual Block Device > +dev.xbbd.203.%driver: xbbd > +dev.xbbd.203.%location: > +dev.xbbd.203.%parent: xenbusb_back0 > +dev.xbbd.203.%pnpinfo: > +dev.xbbd.203.xenbus_connection_state: Closed > .. > although actually in system at the time there were only 2 DomUs running max. > This seems to be the reason why this felt like memory leak, since after long > time of start/stop VMs, most probably sysctl hit some sort of RAM limit > (which most probably is auto-calculated from total RAM somehow) and panics. Those dev.xbbd.XXX sysctls reference a specific blkback instance, and should be gone once the guest is shutdown. Do you mean that you have a range of dev.xbbd.[0-203] entries in your sysctl? Note that in dev.xbbd.XXX XXX matches the domain ID of the guest that's using the backend. > I can repeat this sysctl growing with just starting/stopping VMs in loop > although stressing ZVOL at the same time speeds this up for unknown reasons > to me. What i do not know though is what exactly triggers this, because > there are times when xbbd.$number does not grow larger than total number of > disk devices attached to all DomUs. I do not know what would be the best way > to continue with this. $number should be the domain ID, it will grown as long as you keep creating new guests, but that's not really a problem. > Then i caught another sysctl variable that is growing due to XEN, > "kern.msgbuf: Contents of kernel message buffer". I do not know how this > variable grows or by which component it is managed, but in VM start/stop > case it grows and contains lines with pattern like so: > .. > xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status > xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet > xnb(xnb_dump_txreq:299): netif_tx_request index =0 > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0 > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 > xnb(xnb_dump_txreq:302): netif_tx_request.flags =8 > xnb(xnb_dump_txreq:303): netif_tx_request.id    =69 > xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000 > xnb(xnb_dump_txreq:299): netif_tx_request index =1 > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255 > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 > xnb(xnb_dump_txreq:302): netif_tx_request.flags =0 > xnb(xnb_dump_txreq:303): netif_tx_request.id    =0 > xnb(xnb_dump_txreq:304): netif_tx_request.size  =0 > .. > > Those lines in that variable just keep growing and growing, it is not that > they are flushed, trimmed or anything. Each time i get the same message on > serial output, it has one more section of error appended to "same-previous" > serial output message and sysctl variable as well. Thus at some point serial > output and sysctl contains a large block of those errors while VM is > starting. So at some point the value of this sysctl could be reaching max > allowed/available and this makes the system panic.  I do not know the reason > for those errors, but actually if there was a patch to suppress them, this > could be "solved". Another diff chunk might be related to this: > +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0 > +dev.xnb.1.xenbus_peer_domid: 7 > +dev.xnb.1.xenbus_connection_state: InitWait > +dev.xnb.1.xenbus_dev_type: vif > +dev.xnb.1.xenstore_path: backend/vif/7/0 > +dev.xnb.1.dump_rings: > +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error: > nr_reqs == 0 > +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup, > &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0 > +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup, > xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0 > +52 Tests Passed > +1 Tests FAILED So you have failed tests for netback. Maybe the issue is with netback rather than blkback. > What was suspicious about this is that i am using 13.0-RELEASE-p7, it is not > a DEV version of XEN or anything, but there are sysctl variables with "dev" > prefix. Could it be that ports have been accidentally compiled XEN version > with some development flags turned on? 'dev' here means device, not developer. > So now it felt for me that i understand this panic case: > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 02 > fault virtual address   = 0x22710028 > fault code              = supervisor read data, page not present > instruction pointer     = 0x20:0xffffffff80c45892 > stack pointer           = 0x28:0xfffffe009656a930 > frame pointer           = 0x28:0xfffffe009656a930 > code segment            = base 0x0, limit 0xfffff, type 0x1b >                         = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags        = interrupt enabled, resume, IOPL = 0 > current process         = 17633 (devmatch) > trap number             = 12 > panic: page fault > cpuid = 1 > time = 1646139877 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff80c62011 at rman_is_region_manager+0x241 > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #9 0xffffffff80c1949c at kernel_sysctl+0x43c > #10 0xffffffff80c19b13 at userland_sysctl+0x173 > #11 0xffffffff80c1995f at sys___sysctl+0x5f > #12 0xffffffff8108baac at amd64_syscall+0x10c > #13 0xffffffff8106243e at Xfast_syscall+0xfe > > The function "kernel_sysctl" shows that this is sysctl related, and based on > previous knowledge we know that there might be too many variables. If i > assume that "Xfast_syscall" is a XEN function, then that shows that panic > came from XEN call that wanted to do something to sysctl variable, and panic > happened. This is all just my assumption. No, Xfast_syscall is not a Xen function. That's the syscall entry point into the kernel. > > So those were my conclusions. > > What do you think? How should we proceed with this? Should i try to somehow > build XEN from git sources? I think that's unlikely to make any difference. I would think the problem is with FreeBSD rather than Xen. Can you paste the config file you are using the create the domain(s)? I've tried myself to trigger this crash by creating and destroying a guest in a loop but didn't manage to trigger it. I was able to create (and shutdown) 1328 guests successfully on a ZFS dom0 using 4GB of RAM. Do you have some kind of configuration that differs from the default for ZFS for example? (ie: setting a different ARC size) Are you ballooning the amount of memory assigned to dom0? Thanks, Roger. From nobody Thu Mar 10 09:34:26 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EE17F1A0E7CA for ; Thu, 10 Mar 2022 09:34:15 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDkRC0pLfz4hVL for ; Thu, 10 Mar 2022 09:34:15 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x12e.google.com with SMTP id w7so8380333lfd.6 for ; Thu, 10 Mar 2022 01:34:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=vq54/vvWTG/u+PtfsK6p7MVL72jBcAARGyw4v4UxQ6U=; b=b/oGMurFplbapkJ+tFr0YnPKKvtes1nYN6ptRSQDWSICl2F8YWJK2Q76roiZFO2ndj LcWkrEs1WxnfmBhbD8cvqSfBZcbdo3jjXIh/Et3MXiQGQ+H/QkEV2JYqwzPG+kpehJZg DY3Tgs1UTef7OtrUdsoTDqZWtJHvFh2tKtG4Sig4IsZA6VSRtnruzHOC6bcFuAPiykzD Wkq92/cUU+khTXPwRc7EMDES2VugHsIphYZ1FKa5ADTA4WkCTVa8HF1AQjxiqOsssDWz lXqIxGiEzS9YpsV7mnZjiP7YPj6/L8v9x2buzM/KKU6H6dKO/OG6PGeDp48GuQXRYsx+ Jo7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=vq54/vvWTG/u+PtfsK6p7MVL72jBcAARGyw4v4UxQ6U=; b=4aLvN0U+u4uUtNzjg9FVJm4srIZz6LbiPMnwNARB0xbsmP4SXpsGzhAhcwH5MgGz43 uX8YDUEYdXJxqW4weYUMFeZBoDN6BR779y+O4B0sp2pRxfHmrTnm76eBFnvsnyS8StVl TnzrcL+5S4jpvgDnzlSAnXFYPOYd4bXx3IxnNbdzayfBbeBEboWNnl6l4Emq8nO6qbSp F+q64QIxXc7+DQ23fsKo4HNMjCN0L5fkStt2gXaVc1F4sbua+xvzsWa4jZP0KqncOtvt vHn8gGXG/jwiDtgw54lnwT2nxQiPqsStheMe4sICLBtNYWKzWVLW1iMYNlvEvcQyJTp9 oRSQ== X-Gm-Message-State: AOAM5319YiR9ezexhWNgZ+JW12qQHt7saz8gP2eHbqXJGeYjI838CjV9 mxeIdsvx5BX0qVqUWYhpHLY= X-Google-Smtp-Source: ABdhPJy4flu1r+4ZJ0NyW3Y9StHfNRl2fv+TLzANg/biohKMR/4r0MSiBmvRe7vXdsBrwrPnhxrhEA== X-Received: by 2002:a05:6512:2341:b0:448:2465:7cf with SMTP id p1-20020a056512234100b00448246507cfmr2333673lfu.474.1646904853442; Thu, 10 Mar 2022 01:34:13 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id m11-20020a05651202eb00b00443130ce80asm873630lfq.138.2022.03.10.01.34.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 01:34:12 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <202203011540.221FeR4f028103@nfbcal.org> <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <061700f9-86aa-b2ae-07a4-27d4c3d96966@gmail.com> Date: Thu, 10 Mar 2022 11:34:26 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4KDkRC0pLfz4hVL X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="b/oGMurF"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::12e as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-2.28 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.97)[-0.973]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(0.69)[0.693]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12e:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022.03.09. 10:42, Roger Pau Monné wrote: > On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote: >> If i execute steps 1 and 4, it seems that machine does not crash; tested 8 >> hours Dom0 8GB RAM, 4 hours Dom0 2GB RAM. So creating a lot of ZVOLs and >> monitoring it does not seem to be the cause. >> >> Yesterday was a happy and a sad day at the same time. The happy thing is >> that i think that i have understood one panic reason and somewhat better >> located where  one problem might be, but at the same time, i think i have >> located 2 bugs that cause one specific panic, but 2 other panic reasons are >> due to different circumstances. >> >> Further is my pure speculation about two things, somewhat related; i do not >> know XEN or kernel internals, thus take this with grain of salt. >> >> Knowing that this seemed to be RAM problem and thinking how ZFS sysctl >> tuning values differ, i started to look at "sysctl -a" diffs for Dom0 with >> 2GB RAM and 8GB RAM, by pure luck because comparing Dom0 that was running >> longer and just recently restarted, i noticed there are diff lines like so: >> .. >> +dev.xbbd.202.%driver: xbbd >> +dev.xbbd.202.%location: >> +dev.xbbd.202.%parent: xenbusb_back0 >> +dev.xbbd.202.%pnpinfo: >> +dev.xbbd.202.xenbus_connection_state: Closed >> +dev.xbbd.202.xenbus_dev_type: vbd >> +dev.xbbd.202.xenbus_peer_domid: 309 >> +dev.xbbd.202.xenstore_path: backend/vbd/309/51728 >> +dev.xbbd.202.xenstore_peer_path: /local/domain/309/device/vbd/51728 >> +dev.xbbd.203.%desc: Backend Virtual Block Device >> +dev.xbbd.203.%driver: xbbd >> +dev.xbbd.203.%location: >> +dev.xbbd.203.%parent: xenbusb_back0 >> +dev.xbbd.203.%pnpinfo: >> +dev.xbbd.203.xenbus_connection_state: Closed >> .. >> although actually in system at the time there were only 2 DomUs running max. >> This seems to be the reason why this felt like memory leak, since after long >> time of start/stop VMs, most probably sysctl hit some sort of RAM limit >> (which most probably is auto-calculated from total RAM somehow) and panics. > Those dev.xbbd.XXX sysctls reference a specific blkback instance, and > should be gone once the guest is shutdown. Do you mean that you have a > range of dev.xbbd.[0-203] entries in your sysctl? Yes, i have that range of those entries. Even if all DomUs are destroyed. Snippet from logs - yesterday started, today crashed: cat 2022_03_10__1646887233_sysctl__001.log | grep xbbd | sed -Ee 's/(xbbd\.[0-9]*)\..*/\1/g' | sort | uniq dev.xbbd.%parent: dev.xbbd.0 dev.xbbd.1 dev.xbbd.10 dev.xbbd.100 dev.xbbd.101 dev.xbbd.102 dev.xbbd.103 dev.xbbd.104 dev.xbbd.105 dev.xbbd.106 dev.xbbd.107 dev.xbbd.108 dev.xbbd.109 dev.xbbd.11 .. dev.xbbd.52 dev.xbbd.520 dev.xbbd.521 dev.xbbd.522 dev.xbbd.523 dev.xbbd.524 .. dev.xbbd.99 irq804: xbbd520:133 @cpu0(domain0): 436527 irq805: xbbd521:135 @cpu0(domain0): 4777 irq806: xbbd522:137 @cpu0(domain0): 16589254 irq807: xbbd523:139 @cpu0(domain0): 6412 irq808: xbbd524:141 @cpu0(domain0): 103083 And those entries keep growing. Yesterday, i tried to follow Brian's email about working setup with FreeBSD 12.1. With given scripts 8GB RAM Dom0, still crashed, took longer, but on morning i saw that it machine rebooted, in serial logs had panic message. It seems that "kern.panic_reboot_wait_time = -1" did not disable rebooting as well. In monitoring logs i see that approx. 3 minutes after crash, testing machine was up and running again (rebooted). > Note that in dev.xbbd.XXX XXX matches the domain ID of the guest > that's using the backend. On my setups i have tested this does not seem to be the case. If each VM has more than 1 HDD, i have that many instances of xbbd sysctl variables as total count of HDDs attached to VMs. But at some point after rebooting machines or something, this number starts to grow on each create. For me it seems that dom ID is in sysctl, like this "dev.xbbd.5.xenbus_peer_domid", and if i grep those values, they show domids for nonexistent VMs. I do not know how sysctls are created, but is there some test suite that creates awfully lot of sysctl variables and tries to crash system that way? I suppose there are no ways to limit sysctl variable count or any protection mechanisms for such cases. Then i could see if panic message is the same. Or test that allocates sysctl variable with insaneley large value, or appends to it. >> Then i caught another sysctl variable that is growing due to XEN, >> "kern.msgbuf: Contents of kernel message buffer". I do not know how this >> variable grows or by which component it is managed, but in VM start/stop >> case it grows and contains lines with pattern like so: >> .. >> xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status >> xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet >> xnb(xnb_dump_txreq:299): netif_tx_request index =0 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =8 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =69 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000 >> xnb(xnb_dump_txreq:299): netif_tx_request index =1 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =0 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =0 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =0 >> .. >> >> Those lines in that variable just keep growing and growing, it is not that >> they are flushed, trimmed or anything. Each time i get the same message on >> serial output, it has one more section of error appended to "same-previous" >> serial output message and sysctl variable as well. Thus at some point serial >> output and sysctl contains a large block of those errors while VM is >> starting. So at some point the value of this sysctl could be reaching max >> allowed/available and this makes the system panic.  I do not know the reason >> for those errors, but actually if there was a patch to suppress them, this >> could be "solved". Another diff chunk might be related to this: >> +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0 >> +dev.xnb.1.xenbus_peer_domid: 7 >> +dev.xnb.1.xenbus_connection_state: InitWait >> +dev.xnb.1.xenbus_dev_type: vif >> +dev.xnb.1.xenstore_path: backend/vif/7/0 >> +dev.xnb.1.dump_rings: >> +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error: >> nr_reqs == 0 >> +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup, >> &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0 >> +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup, >> xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0 >> +52 Tests Passed >> +1 Tests FAILED > So you have failed tests for netback. Maybe the issue is with > netback rather than blkback. I am not sure where the problem is. Maybe there are two or more problems. Should they be discussed separately network related and disk related? >> What was suspicious about this is that i am using 13.0-RELEASE-p7, it is not >> a DEV version of XEN or anything, but there are sysctl variables with "dev" >> prefix. Could it be that ports have been accidentally compiled XEN version >> with some development flags turned on? > 'dev' here means device, not developer. Okay. Understood. >> So those were my conclusions. >> >> What do you think? How should we proceed with this? Should i try to somehow >> build XEN from git sources? > I think that's unlikely to make any difference. I would think the > problem is with FreeBSD rather than Xen. Can you paste the config file > you are using the create the domain(s)? Yet again, as i said, i do not know where the problem is. My wording might be off when pointing that it's Xen, since Dom0 is FreeBSD and it panics not Xen. It's just that i do not know the boundaries and responsibilities of components clearly enough. And often by mentioning Xen i mean whole setup or FreeBSD module that interacts with Xen. > I've tried myself to trigger this crash by creating and destroying a > guest in a loop but didn't manage to trigger it. I was able to create > (and shutdown) 1328 guests successfully on a ZFS dom0 using 4GB of > RAM. No, just turning on/off single VM does not crash system. I had tried that as well before reporting bug. I had attached all /bin/sh scripts and configs that crash the system in bug report #261059 tar archive. Those scripts are really simple. I've been trying to minimize/trim everything off since then, but the simplest requirements for crash are executing 3 processes in parallel: 1) loop that creates ZFS volumes at least of size 2GB and with dd writes data into them, /dev/zero is good. I've never run out of space, thus crash is not due to exhausted space in ZFS pool. 2) loop that creates VM1 with single disk, connects through ssh to that VM and writes 3GB data in /tmp, then reboots, starts VM, removes file from /tmp, then reboots 3) loop that starts/stops VM2 with 5 disks. As for panic timing. Less RAM for Dom0, increases speed for panic, panic message is the same. For 8GB RAM on my machine i have to wait for approx. 10 hours till panic, on 4G RAM around 6 hours. On resource starved Dom0 with 2GB, even shorter, but panic messages are more or less the same. I have crashed just by omitting step 1, but it takes longer. Running just on/off for both VMs, system does not crash. Thus it seems that some sort of ZFS load (VM internal or Dom0) is essential for panic to occur. I have managed to crash with VM which has 2 HDDs, thus it does not seem to be due to 4 HDD threshold emulated/virtio disks. It's just that with 5, it takes less time till crash. Usually panic occurs after when xbbd.$value, $value > 230. > Do you have some kind of configuration that differs from the default > for ZFS for example? (ie: setting a different ARC size) I have no specific settings, everything is basically like in manual pages. # cat /etc/sysctl.conf vm.max_user_wired=-1 vm.max_wired=-1 kern.panic_reboot_wait_time=-1 vfs.zfs.vol.mode=2 # minimize problems with bug #262189 (default value still crashes system) # /etc/rc.conf cloned_interfaces="bridge0" ifconfig_bridge0=".." zfs_enable="YES" xencommons_enable="YES" ifconfig_em0=".." defaultrouter=".." sshd_* # configs sendmail_* # configs that should  not influence anything # cat /boot/loader.conf autoboot_delay="1" beastie_disable="YES" autoboot_delay="5" boot_multicons="YES" boot_serial="YES" comconsole_speed="9600" console="comconsole,vidconsole" xen_kernel="/boot/xen" xen_cmdline="dom0_mem=8192M dom0_max_vcpus=2 dom0=pvh,verbose=1 console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 noreboot" hw.usb.no_boot_wait=1 # cat xen-vm1-zvol.conf memory = 1024 vcpus=1 name = "xen-vm1-zvol" type = "hvm" boot = "dc" hdtype="ahci" vif = [ "type=vif,bridge=bridge0" ] disk = [ "backendtype=phy, format=raw, vdev=xvda, target=/dev/zvol/zpool/vmdk/dev/vm1-root" ] vnc=1 vnclisten="0.0.0.0:7" serial = ["pty"] VM2 config fingerprint is the same, just with 5 HDDs, different names, paths, values. > Are you ballooning the amount of memory assigned to dom0? No, if ballooning does not happen by default, then not. I saw Brian's message about experienced powering off panic case, but there was nothing that i could add to that. The way i power off VMs is with command: xl shutdown -w -F $vm_name. I am intending to change testing scripts 'xl shutdown' to ssh 'init 0' and 'xl list' wait loop, to rule out possible problem cause due to 'xl shutdown' command. With given scripts/setup i can crash Xen + FreeBSD 12.1, 12.2 or 13.0 RELEASES as Dom0. Some things that i do not know to mention or not, since i can not reliably repeat them are. There are times when after a while OOM starts to kill qemu-system-i386 processes, or ssh that initialed 'xl create' command, although it seems that there is plenty RAM available; usually not long after that if testing script is started again or xl create/destroy is run manually, system crashes. Sometimes it feels that after a crash, next crash happens sooner. There was a case when 8GB RAM system crashed in approx. 8 minues after boot and running those scripts, when previously it had already crashed. But i can not think of any reason for that... can ZFS be left in dirty state if Xen crashes? To rule out that, i installed FreeBSD on separate disk(UFS) and always recreate testing pool from scratch. If there is no time for you to set up the system, i would be willing to help with that, i can drop SSH pub key, then connect at some arranged time, setup takes around 30-40 mins. Nothing much is necessary, just some HDDs to test on, ZFS pool, and FreeBSD images, either iso or prepared VMs. But other than that, i have exposed all config changes i have made, in general it's just installed the way described in manual, most of things are default. I somehow do not feel that my hardware is at fault, since i have tested on three different machines, different manufacturers. But maybe they all have some BIOS setting that is common and causes this problem; i am not sure of that. From nobody Thu Mar 10 10:34:07 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 460761A17EC5 for ; Thu, 10 Mar 2022 10:33:51 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDlly1CTWz4ntq for ; Thu, 10 Mar 2022 10:33:50 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x134.google.com with SMTP id 3so8623172lfr.7 for ; Thu, 10 Mar 2022 02:33:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=9Ba3Y5MjouiJ1K44xxNH0WlDGC1RP/RK0P0KjfA2sdI=; b=K4xQUsip/jtxqXJkhRtNyxzAz+K4JS2MbgbbvmIKhcWUGhiFJQZUjMU15E46E3nvhu p7f9c3SsTqlOLuEIAJX8uvdtRuDSrlJCSiWN+RWZX9+K5tJM92EqON4fd1mNMqDWRYdg ML2maYOfIeFTy83y4Hsqdu/nV5vUwM8rs/klcLnNHRwKxw3EnyRdZrXSpPh01Iub69m4 UMeqxR2n9nlq2hawpxfs8HzZWDTPTY4HdtyyFQ8+/YhsuptDVhNEAMopK665K7mBH+Ul CTfYm0mydpJqkTBn+NUR+YyAuazYMbcQ2AZEDIj3oPjjAZv4pq79ne0iup3YsAlcGn2Q iONg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=9Ba3Y5MjouiJ1K44xxNH0WlDGC1RP/RK0P0KjfA2sdI=; b=rXCzaXKUx9I5MXtUAxIrgUzX2zMusfzNlqx+ginUiRPd3sikICIPHCAnxhtvViU/FR NVj3is6LrkBv3IPstAZhlu8+Gw36FJzVKt4vGV9UPwnpwUtG8jm+lvcF1WNjz9e2+ThH cMazNeBdrSHtM4M/Mx5rNaOfWxKT9PHFP1ovE0Q5wisSP/osAO5WZI6dHOVSKQYswOqb fbgkj9v3pMWO5Ue7qC6VrdRXa0IcE/zeserby0LcCpxzp2IeefcViXhRqy/76gcKuVr/ G2sgM361ASCkflu+welC006WEkgjC6dcNQZdePNq/WJuJrQ+p4b+tGU+BMc3PtugLMwL ZXaA== X-Gm-Message-State: AOAM531xtI1BoeUtIlaf4liy4XqQsNu5LeySDAahJNDI0+bkIfJluCUw qWyHw+uEARhKy72tHkUbcN6gAnv1cGY= X-Google-Smtp-Source: ABdhPJytLbiDc00ZdUCRjOeDOFzZHe2UpnLfTY2AnRynrb2qnSjRBdmHU2HPWWX0VTXcrWiX3+b9Sg== X-Received: by 2002:ac2:5de4:0:b0:448:2daf:b98c with SMTP id z4-20020ac25de4000000b004482dafb98cmr2477838lfq.641.1646908428691; Thu, 10 Mar 2022 02:33:48 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id z4-20020a0565120c0400b00443823361d0sm901146lfu.243.2022.03.10.02.33.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 02:33:48 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: Brian Buhrow Cc: freebsd-xen@freebsd.org References: <202203022007.222K7Y7I013387@nfbcal.org> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <1937e88f-159a-5324-db8f-687e84a0bea9@gmail.com> Date: Thu, 10 Mar 2022 12:34:07 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: <202203022007.222K7Y7I013387@nfbcal.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4KDlly1CTWz4ntq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=K4xQUsip; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::134 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.29 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.29)[-0.289]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::134:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022.03.02. 22:07, Brian Buhrow wrote: > Hello. Given Roger's message and the presentation of the errors you're seeing, I think > we're both in agreement that you're running out of memory. Below I've listed some suggestions > on how you might go about making your system more stable, and, possibly figuring out where the > trouble is, exactly. When I was building my infrastructure, I ran into a bunch of edge > conditions and weird bugs as well, so some ideas that may not seem obvious, may, in fact, work, > simply because they cause you to skirt some latent bug in the system. > > Here's what my setup looks like, in case it helps. > > > Hope these notes are helpful. Hello Brian, Yes, of course those notes are helpful. Knowing that somebody has a working and stable setup and how his configuration/setup is, is a help. Thank you for the detailed information. I did not respond right away, because i wanted to follow those steps before writing anything and i did not have a spare HDD for "new" testing system. Wanted to still keep already set-up current version 13.0 for testing as well. For testing i made somewhat simplified setup of your described one: 1. Installed FreeBSD-12.1, UFS ada0p1, brand new HDD, basic configuration just for network, resolve.conf, etc. 2. 8GB RAM Dom0, 16GB swap on ada0p2 3. i did not use gmirror for now, since it is a test setup and on this new HDD problems should not happen so soon, 4. bridge0 created, no vlans, just a simple bridge, no firewall not even physical NIC bridged, just for VMs 5. creating zpool on ada1, mountpoint=none, here though i got a question: When i create ZFS partition on disk, i use "gpart add -t freebsd-zfs .." and similar for UFS, but what gpt type should be used for XEN DomU passthrough partition? I mean, if i passed through whole disk to DomU, and VM formated it as ZFS, in case of physical system reboot, BIOS could get confused and maybe even boot from that previously passed-through-HDD. AFAIK FreeBSD boot loader prefers ZFS over UFS if both available. If gpt type is freebsd-zfs, how can i ensure that Dom0 does not try to taste it or mount it? In a way i know that in VM the given block device will be formatted with MBR or GPT, but still, what gpt type should be used in that case, so that Dom0 is satisfied and it is somewhat secure. For ZFS volumes for now i use volmode=dev. 6. well, to crash sooner i created 1 VM with 1 HDD, and other with 5 HDDs, passsed trhough ZVOLs. After approx. 10 hours of running given scripts, on/off 2 VMs and load on ZFS, machine panic'ed, so my hopes for stable setup were shattered. The only difference with your written mail was that on FreeBSD 12.1 RELEASE pkg repo, there is only xen-kernel-4.12 available. I will try to run with VMs with 1 and 2 HDDs only, but historically i know that FreeBSD 12.2 with 6GB RAM crashed with those setups as well, it just took longer to happen. Another difference is hardware used. Do you remember if you have tuned some BIOS settings? I'm thinking to poke around there as well. Could you share a config template for one of DomU VMs? Maybe i use some bad syntax for HDD devices or something. Thank you! From nobody Thu Mar 10 11:41:28 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 99FE219FE88A for ; Thu, 10 Mar 2022 11:41:13 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDnFh5DTKz4x8X for ; Thu, 10 Mar 2022 11:41:12 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lj1-x22a.google.com with SMTP id s25so7301169lji.5 for ; Thu, 10 Mar 2022 03:41:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=7Z+LMTzU4xxXWXI7RKfOZPZgbubePPV/duqXQCATFSQ=; b=Tdm2gRddN1cVCASalTQH0dy0eurwskwkw++QvQyzDTA5enKGONQ3+7e6txMgzezaJn w/7gCOkzW2jh4/a1K6UIgZKHIuAvJnaAH2lvytZq0QNS91SFh84QZM6b8aVupmWOPqQO MgoD0Ms+7wPEyibE8g+No1p1ubebV/YOr5D6HCjm5sJz1JD85A7lNVpaUnpfiZanODgk 5WgF5TgFrCBZ/lZThvitxidLWzVVzgVlcgPKTOWebg6O3xa5eXSr5JRy6rHGioHBzPvJ XY1yK6G137b9FEn76A+c2im6wdxOzM9j7vtk+fk0WRlFtp0d6aX8lBQXMvlDlT75x0bW DseQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-language; bh=7Z+LMTzU4xxXWXI7RKfOZPZgbubePPV/duqXQCATFSQ=; b=dj5UVo5BdricfmP63jLlyL8Ir0VTDybBtp67P3W9SO9i+6WulGFVVvQ/w700LJN979 eNDUx2s3kHxc++4kWYxMuZxN0e3xn0Jpe+Iygu5JfslO3uidUZVOem1BhHETjvjWyV1n aR4OsXOCe9m8yv+gcrKqBDNhdhBplPQGbs7T5sSMkQyfImn/5Awluui1oS2u4HDpYNh+ hjGr3zeobLe3afw8QyoPFTs9Jkz2hnKFllEOWfNwrnCCQ1mI28MZXrUhUhnc8GeGz4jc NBlN70Oh92upsKm+TdjlpKDNgpoJQ2PuSM4YVoYSe06yysjqkE2PEnUcv6h+aF1qcY+j dNCg== X-Gm-Message-State: AOAM530TNOy1hGg8tZ8zDCh8eCvK837n14BI4ViUBBmcYrEnqx8FcXVh ebpWSu0blaTxkcLWkzXOo0c= X-Google-Smtp-Source: ABdhPJz2vgOutmVglwHJPE1dxgi8s9yvWt5blfes6pwJH/qVQjmtVIUfJOq8WUuWoCIBCxVKHqUU4A== X-Received: by 2002:a2e:824b:0:b0:246:1246:d830 with SMTP id j11-20020a2e824b000000b002461246d830mr2763629ljh.267.1646912471311; Thu, 10 Mar 2022 03:41:11 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id 25-20020ac25f19000000b0044858b3cc20sm721624lfq.40.2022.03.10.03.41.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 03:41:10 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <202203011540.221FeR4f028103@nfbcal.org> <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com> Date: Thu, 10 Mar 2022 13:41:28 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------0001BD4272D7E3B6557D5819" Content-Language: lv X-Rspamd-Queue-Id: 4KDnFh5DTKz4x8X X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Tdm2gRdd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22a as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-0.97 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.96)[0.957]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(0.07)[0.074]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22a:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------0001BD4272D7E3B6557D5819 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 2022.03.09. 10:42, Roger Pau Monné wrote: > On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote: >> Then i caught another sysctl variable that is growing due to XEN, >> "kern.msgbuf: Contents of kernel message buffer". I do not know how this >> variable grows or by which component it is managed, but in VM start/stop >> case it grows and contains lines with pattern like so: >> .. >> xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status >> xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet >> xnb(xnb_dump_txreq:299): netif_tx_request index =0 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =8 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =69 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000 >> xnb(xnb_dump_txreq:299): netif_tx_request index =1 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =0 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =0 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =0 >> .. >> >> Those lines in that variable just keep growing and growing, it is not that >> they are flushed, trimmed or anything. Each time i get the same message on >> serial output, it has one more section of error appended to "same-previous" >> serial output message and sysctl variable as well. Thus at some point serial >> output and sysctl contains a large block of those errors while VM is >> starting. So at some point the value of this sysctl could be reaching max >> allowed/available and this makes the system panic.  I do not know the reason >> for those errors, but actually if there was a patch to suppress them, this >> could be "solved". Another diff chunk might be related to this: >> +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0 >> +dev.xnb.1.xenbus_peer_domid: 7 >> +dev.xnb.1.xenbus_connection_state: InitWait >> +dev.xnb.1.xenbus_dev_type: vif >> +dev.xnb.1.xenstore_path: backend/vif/7/0 >> +dev.xnb.1.dump_rings: >> +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error: >> nr_reqs == 0 >> +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup, >> &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0 >> +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup, >> xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0 >> +52 Tests Passed >> +1 Tests FAILED > So you have failed tests for netback. Maybe the issue is with > netback rather than blkback. Just found that there are no errors with network, most probably. Based on https://reviews.freebsd.org/D9234?id=24172#201024, since i was monitoring sysctl variables with 'sysctl -a', most probably that did "sysctl dev.xnb.0.unit_test_results" implicitly for each xnb interface, thus the extra output, since reading value actually runs tests. So now i upgraded my monitoring command to 'sysctl -a -N'. Never did i expect that sysctl variable reading could trigger test suite execution. I did not understand from comments though - how do you instantiate xnb interface? "ifconfig xnb create" returned: ifconfig: SIOCIFCREATE2: Invalid argument Thanks. --------------0001BD4272D7E3B6557D5819 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 2022.03.09. 10:42, Roger Pau Monné wrote:
On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote:

Then i caught another sysctl variable that is growing due to XEN,
"kern.msgbuf: Contents of kernel message buffer". I do not know how this
variable grows or by which component it is managed, but in VM start/stop
case it grows and contains lines with pattern like so:
..
xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status
xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet
xnb(xnb_dump_txreq:299): netif_tx_request index =0
xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0
xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
xnb(xnb_dump_txreq:302): netif_tx_request.flags =8
xnb(xnb_dump_txreq:303): netif_tx_request.id    =69
xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000
xnb(xnb_dump_txreq:299): netif_tx_request index =1
xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255
xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
xnb(xnb_dump_txreq:302): netif_tx_request.flags =0
xnb(xnb_dump_txreq:303): netif_tx_request.id    =0
xnb(xnb_dump_txreq:304): netif_tx_request.size  =0
..

Those lines in that variable just keep growing and growing, it is not that
they are flushed, trimmed or anything. Each time i get the same message on
serial output, it has one more section of error appended to "same-previous"
serial output message and sysctl variable as well. Thus at some point serial
output and sysctl contains a large block of those errors while VM is
starting. So at some point the value of this sysctl could be reaching max
allowed/available and this makes the system panic.  I do not know the reason
for those errors, but actually if there was a patch to suppress them, this
could be "solved". Another diff chunk might be related to this:
+dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0
+dev.xnb.1.xenbus_peer_domid: 7
+dev.xnb.1.xenbus_connection_state: InitWait
+dev.xnb.1.xenbus_dev_type: vif
+dev.xnb.1.xenstore_path: backend/vif/7/0
+dev.xnb.1.dump_rings:
+dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error:
nr_reqs == 0
+xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup,
&xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0
+xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup,
xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0
+52 Tests Passed
+1 Tests FAILED

So you have failed tests for netback. Maybe the issue is with
netback rather than blkback.

Just found that there are no errors with network, most probably. Based on https://reviews.freebsd.org/D9234?id=24172#201024, since i was monitoring sysctl variables with 'sysctl -a', most probably that did "sysctl dev.xnb.0.unit_test_results" implicitly for each xnb interface, thus the extra output, since reading value actually runs tests. So now i upgraded my monitoring command to 'sysctl -a -N'. Never did i expect that sysctl variable reading could trigger test suite execution.

I did not understand from comments though - how do you instantiate xnb interface?

"ifconfig xnb create" returned:
ifconfig: SIOCIFCREATE2: Invalid argument

Thanks.



  


--------------0001BD4272D7E3B6557D5819--

From nobody Thu Mar 10 15:40:54 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D8AD81A073EF
	for ; Thu, 10 Mar 2022 15:41:10 +0000 (UTC)
	(envelope-from prvs=061b0cdbb=roger.pau@citrix.com)
Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KDtZY41Vhz3mCN
	for ; Thu, 10 Mar 2022 15:41:09 +0000 (UTC)
	(envelope-from prvs=061b0cdbb=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1646926869;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=8BjBswdBOZNzx+hLaB/OMUf5MMjzuG7nLAcCBWLhP3k=;
  b=FvZ79uHVvQ1B7CZuFCYoaHP1GBeMdcL+o2P0PToCcwMAe7V4QL74rI95
   px9lP5posfIccTpUHZJiLCBpzm/Eo7YLonZdW1d87jt2IetGWY+j314Wd
   hI2EkT8SEfS9hm9BAMEgwBreGgWj8cnuWs2ey01Jpa9SlyauyMQ+ArjPD
   o=;
X-SBRS: 5.1
X-MesageID: 65402375
X-Ironport-Server: esa5.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:plYOrK9eUSyvd97ZZaj/DrUDkX6TJUtcMsCJ2f8bNWPcYEJGY0x3m
 jNMXm2OP/3cY2L8eopxYI+18RgGvMTXy9BqHgtqpXw8E34SpcT7XtnIdU2Y0wF+jyHgoOCLy
 +1EN7Es+ehtFie0Si+Fa+Sn9T8mvU2xbuKU5NTsY0idfic5DnZ54f5fs7Rh2NQw2oHhW1nlV
 e7a+KUzBnf0g1aYDUpMg06zgEsHUCPa4W5wUvQWPJinjXeG/5UnJMt3yZKZdhMUdrJ8DO+iL
 9sv+Znilo/vE7XBPfv++lrzWhVirrc/pmFigFIOM0SpqkAqSiDfTs/XnRfTAKtao2zhojx/9
 DlCnaWrRz0JDKbTo8oYVDoCNDhdYJd66oaSdBBTseTLp6HHW37lwvEoB0AqJ4wIvO1wBAmi9
 9RBdmpLNErawbvrnvTrEYGAhex6RCXvFJkYtXx6iynQEN4tQIzZQrWM7thdtNs1rp4TQa6EP
 JVEAdZpRDiZIDlDO0o1Mc4/38LyqSDnVBlW9l3A8MLb5ECMlVcsgdABKuH9cMGKX8JKtkCWr
 GnP+yL+GB5yHNKFxDeP6X7pluLJtS3hVY8YD7H+8eRl6HWBy2AOEAYHTnO0pPC4jgi1XNc3F
 qAP0nNw9+5orhXtF4SjGU3jyJKZgvICc+NLHdca2gqS8YTR5CXDIXEnEwR9euVz4afaWgcW/
 lOOmtroAxlmv7uUVW+R+9+okN+iBcQGBTRcPHFZFGPp9/Gm+dhu1UyXEr6PBYbo1oWdJN3m/
 9ydQMHSbZ03hNVD6ai09Euvb9mE9smQFV5dCuk6swuYAuJFiGyNOtTABbvzt68owGOlor+p5
 iZsdy+2trxmMH11vHbRKNjh5Znwjxp/DBXSgER0A74q/Cm39niocOh4uW8iehc0a51fIWG5P
 ic/XD+9ArcJbBNGioctP+qM5zkCl/C8RbwJqNiIBjaxXnSBXFDep3w/DaJh92vsjFItgckC1
 WSzKq6R4YIhIf0/llKeHr5FuZdyn3xW7T6DFPjTkkX8uZLDNSH9dFvwGAbXBgzPxPjf+1u9H
 hc2H5bi9iizp8WlO3iJq9FPdQ5WRZX5bLivw/Fqmie4ClMOMEkqCuPLwKNnfIpgnq9PkfzP8
 G37UUhdoGcTT1WXQelWQhiPsI/SYKs=
IronPort-HdrOrdr: A9a23:H+Vs7K5c3v3xdx6e3QPXwSyBI+orL9Y04lQ7vn2ZFiY6TiXIra
 +TdaoguSMc6AxwZJkh8erwXpVoZUmsiKKdhrNhQYtKPTOWwldASbsC0WKM+UyEJ8STzJ846U
 4kSdkANDSSNykLsS+Z2njBLz9I+rDum8rE9ISurUuFDzsaEJ2Ihz0JezpzeXcGPTWua6BJc6
 Z1saF81kSdkDksH4+GL0hAe9KGi8zAlZrgbxJDLxk76DOWhTftzLLhCRCX0joXTjsKmN4ZgC
 X4uj28wp/mn+Cwyxfa2WOWx5NKmOH5wt8GIMCXkMAaJhjllw7tToV8XL+puiwzvYiUmRwXue
 iJhy1lE9V46nvXcG3wiRzx2zP42DJr0HPmwU/wuwqUneXJABYBT+ZRj4NQdRXUr2A6ustn7a
 5N12WF87JKEBLphk3Glpb1fiAvsnDxjWspkOYVgXAae5AZcqVtoYsW+14QOIscHRj99JssHI
 BVfY7hDc5tABOnhk3izypSKITGZAVwIv7GeDlPhiWt6UkWoJgjpHFogfD2nR87heUAotd/lq
 D5259T5cNzp/8tHNFA7dg6ML6K40z2MFvx2TGpUBza/J9uAQO4l3ew2sRz2N2X
X-IronPort-AV: E=Sophos;i="5.90,171,1643691600"; 
   d="scan'208";a="65402375"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=jBk4hVXCY+8VNeLVBgKu0qKY/vdX0RjA+EWydx+0+pqIN9bclaHx0CjYNjF2mIh3OVWJU+56dH64M5hLHfzTIHSg4REZxuMgCkRRaESh9KjJAdX7I//ilhvr5bdfR4r7UFy5iNYqeXUGcohNZUKPACFxshUAWyTaTFUtw4/GS9ToEE8CeBGTcqJdyYgJ6b+/2+9eKSZwdmJqNA8b1/0RKh4k6z/zEK+4/WY72+LA3ulYqTPkBB8Z1ORxqZtKWsBsiMtmlRFZD4iYJJ6np99+dS0VP7yH5ZUPdYfhSSWRjV+ZoEGVMHjnA8hyIVPRD/kcQumedBA7Ux2Xbxajjb7htw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=6hSgc10ZfOXiydfe4uZH4ESvHqRhuoGyjCb2AP2LrDk=;
 b=JU7dnHgGn3r6kplSSnvtIAOn0KZ8ChAkjXLfr7tyy1DO/BTQJUspouvv2cxn9cXVOlwbRU1fchV1mtgBP706UkE7wg8o1VLklVHW2jkdUAqQ2iLcNiLglbDbo/uwflSbhuGkY5Qc4UnfqhfFMpinsG+qxwGIdhapGlwfnrO3YTgh9epnJ7BNd7RD8COuTCttNjzt3+zuhZQvQsuFWRjC2ap8h3z0RNRnMZ6Rk8RsDmynx/BPCDUwlS7sQVUIiFfhBN5K7WYIWmsO7Hug/I0NaUQ1uAZT2BDBZuc1GmZhY0/eiisvMIuHzXtcrfEfeRSdQ0EewvreB4czgjCG8xhA4Q==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=6hSgc10ZfOXiydfe4uZH4ESvHqRhuoGyjCb2AP2LrDk=;
 b=ezq6leGcufGeN/VHvt18KlmGae2uU/puSUDFTJrGs2cUFmoX7EY3gnDTHbs/83/ousalu9aqMKauQDK+/Ni5/5qH48xy7duxnNGXREArusXAOUhaRVRHVFMINZMF0Y+dZ63kzE34R2LQEJAjGouBorO5zyx6SOifnwaA7NfJQds=
Date: Thu, 10 Mar 2022 16:40:54 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <202203011540.221FeR4f028103@nfbcal.org>
 
 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <061700f9-86aa-b2ae-07a4-27d4c3d96966@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <061700f9-86aa-b2ae-07a4-27d4c3d96966@gmail.com>
X-ClientProxiedBy: LO2P265CA0282.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:a1::30) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: a3544ec0-1a86-4a97-9a05-08da02ac5f7e
X-MS-TrafficTypeDiagnostic: SN6PR03MB3583:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: ky9wC0cBEKz3nu2JHAcH3fCNReJd7Qyk7fh9DuDWBR4k3Oj1J/19jBCDYgwMuNJIXb1VdoU9jqO0R+UC0SZJcmgV0ELbLwx+WRV7Vvp5nQc/rmP6to7DjxGTrpsB1AZYX1k8/JQt3p51zixHAkiJExCVHRj5fWN3DX4iEgXT5M6TjqgrRyr0plhtJpDTKuu88z+92bVZGsil2kBSuNkeryP6UVG7cr3x6h+nw6xSILgfJpkEZiABkTBpYTepduJ05XNnKHD6lFScp7aFeCRLBV1NxIGQYRJ3qvArMeuDNHcdJcttozxHpGhWC8CimmD/Zl0fIshwgzDheUhPjIub1U7m4rC2zPG8z8HmsJaY4R8EqcvGIbwc4txcs8NXo8rFqMmOKBbqKWzBwvXHty3Bz8gAZbwnWGRpDe6p23IoEBGWzf/zj9o0IKpvxFIFyrTzQR6T26dA/fg12TVcXXo+/qm2Gk5K1V9XX0fkB+93AV8BaiRuw3mR4MRAoIc51p1kUQB8iLalhYYwWmh4FgkFItYmnsk7PUClw20OAGmMU3sT1UaJa8ja2pwkLG4DCZiB+cIIZ+qg8RUNKJXmE6A9MrLq9471ywO7yWNPXY+8G12A6+dZL2KSI2HrrLJS1eoaooh82FVIr6MW2T0GCV8X7H+FnY3TvbcuSGzMWDPXxid9M8xB63X1jhYauKAXIQSG7fGbo5GA5hHh7WzUawRo5w==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(8676002)(316002)(83380400001)(6916009)(85182001)(4326008)(186003)(86362001)(82960400001)(6506007)(9686003)(5660300002)(66946007)(6512007)(6666004)(26005)(38100700002)(6486002)(30864003)(8936002)(66476007)(66556008)(508600001)(33716001)(2906002)(21314003);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NFo5dkFtQ0NPek5Cc2FKT0hUcnZCa1FJOU1EVDJsTjJtc0hSWWt0KzE3bFIw?=
 =?utf-8?B?N291cW9Dc0RzRm9WZG1SUklRdG44V3NmWG1GNzhaYVEvSTh2bjFLZitHZ1FE?=
 =?utf-8?B?MTR3bWVScE5Wd2VMSzd5djVWT3hnejJ3VHdGZkVqOUkxMjBONHR4RTdXQXFB?=
 =?utf-8?B?bjFObmZWSEs1cnJjdjVnL2h5ZDd2b0tEU0pBd0NUNHFtc1BDdmRrY2o1ZzAy?=
 =?utf-8?B?aUdlSHRIZFExK1gwOS85b2pPcDFXa043aXRsMmdFMnNQQU9tY0FlclRkQmh4?=
 =?utf-8?B?LytWMzdpRlBEb25VQ2ZlNVdWM0o3ckEzSzVyencwQzlpVjJaaFRTcWNGV0NV?=
 =?utf-8?B?dEZpZHU5NXZsSy90Zllod2Q5bHJDZjdtMWI1aUM0UU5UeXB5LzVPSlBXeU5q?=
 =?utf-8?B?c1U2bXprZ3dTYTlyQ0R5dkZUVFd0L0tKUThBYVlBL0JVM0U2SWxjbk1tTHZR?=
 =?utf-8?B?anJWSFdTWDlCOGtpTzg4VldvenNuUGMvekdyZGFXanBGREJOL0hhT1VZZVVC?=
 =?utf-8?B?SndtSys1L0RCUzZZTXpqdzc3RVZBdGtQa0FicUs4djRYTTlhY0hxTXpxY0Fw?=
 =?utf-8?B?MHIrUENGUVBwM1BuU2xxWER6aWVMVzc1WXNTNnBDZDFPT3NFblNCQUpjY21Y?=
 =?utf-8?B?NnMrc3VjSXZDb05ac0M0blBkY3pLZ0EwcE1hT2ozTEEyUW5PbWFpNzB2dHdt?=
 =?utf-8?B?WitwUDZmeWtkemg0Z1h2NEh6ODBGbndMUG82bGwvVE5EbW1TcENEU25McjBF?=
 =?utf-8?B?RkhpcFcxdUtpbXRGQmZHSHRTVHVXK0hZUU5VdnhCT3RhWlkrN050c3YyakY2?=
 =?utf-8?B?QU9jMGpJV1NyTTFBQzdWWjBTVHJzdG03alp6OGFmelJpZUx0SG9qL2trK2VK?=
 =?utf-8?B?RVRpWGJMM2NTQjd0MFFFQjBSTldlUTdzckhxTXY0TUltRXJ3eStPNnN3WGNU?=
 =?utf-8?B?Tk5VVW1IUnhVVllXVS9LSzdxbEhSODRuREJpWEdiY3VFU3lVY0w4MjF3VW5h?=
 =?utf-8?B?Y20vRjNNcEFQWFBmSUUrTFVkR1hCUzZBbSs1ZWthQ2FyNDFHWUVpR1M3RGpF?=
 =?utf-8?B?WlFlalZ0VEhZbGl3MzBMZ1d0U2ptYml0VGFJaHRhV0tnc05iM1NUditQSU8x?=
 =?utf-8?B?Z21vcWM2TzZHeHR3bU1iSGVSNGsyVk1ud0g2MktOL3BhRTBnZVQ0Z2hTU0w2?=
 =?utf-8?B?TGVtYXZmWFBOU3o4UWhiMTNjTFdBdENhdkhiaUJFUEFmOE9QRWxPdDJRRTh5?=
 =?utf-8?B?WnhLaXpyWng5anF3VzRNM1BHK2tyNllrV3VpTTdGa2k4blFIWERLK1lWWWZG?=
 =?utf-8?B?c1VUTUI5WlJSd1FEcjNRK3F2VEV3TXFlUEJDaHFRQjVBMkY1Rk43NXdGRGRO?=
 =?utf-8?B?NzdZOVBRdzlod2lzbjhhbE5kOGhNZEJuSVpreUhXNDVaLzVCWmk1S3hCWGQ3?=
 =?utf-8?B?L1VPaFJldTBHcXFqRE5icGczSDBhb0pqU21ZYitrZEhJQnkrcFRxdnlreWZ0?=
 =?utf-8?B?a2dsdW1hY1Eva1ZmRDJCaWJNTkhUK2RpTW1tU0NycEFjV3p1ZStqa2NOd1ZU?=
 =?utf-8?B?Zllib3pVRXUwSldIbXJYREhoL1JmTmJYOS8rdHZ4MmJrL1Z6VU5CSzgrUlV0?=
 =?utf-8?B?bU1QQ2thL1VQZXNZeGp5ampJQ3ByRWZHRG9rRittTEpLQWhDSG1IYmI0MWp5?=
 =?utf-8?B?d1NFdkViT0ZUajQ2OW12TVVEQUxFSUlyRUFoSEQ0aVdJV0wxbFlqdG14NDlZ?=
 =?utf-8?B?MGlXL1cyNzJuSGYzcWV2THczcWZjVkFXbEhvalhDMy9oUHg5dUgvUDNjK3U1?=
 =?utf-8?B?MVAzMU1zNGUvTGFzSEVGNnhNdTdkZEhUbkRtZzlqWFVWWEhXU2pRTzkwdi9x?=
 =?utf-8?B?SUxLa2h3Q0pEeTNpcVQ0RnlZVTNYMk00WjRZMkNEajVlVzU5VlY3ZHpDVkNV?=
 =?utf-8?B?eWYrQnAxUzhTK1h4Z1R2MWlMR1RmTDg4L1UwQnhkamYxTHhVb0JpRm40dDlK?=
 =?utf-8?B?NERhc0hQdHAvTWtRVmc3c0ZvekdSSVRKcDJSNFNHY0tmSzJtTlkyeEhqd3RR?=
 =?utf-8?B?dkU2cFo2SFRuOS94dG04VDhGM09RTER3TUMxVkw1b1d5cnAyNlkwN3VxS2gr?=
 =?utf-8?B?Z0NEQVN0dTJFZ0xhRXVZRFFzY25rS29FdW5IZnREQUFIVTdiSGVQOWc3TUw3?=
 =?utf-8?Q?q8ebPuV/kgvbRtdxmh9pRKM=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a3544ec0-1a86-4a97-9a05-08da02ac5f7e
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2022 15:40:58.7671
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JsCayWA1AlOPVmmjhJskqHLi6Xe0Thrfdr7kod3No3pNdLq1aSsipK699OpRx6fDuqzjw+mTxMYLKLd/YOml7g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR03MB3583
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KDtZY41Vhz3mCN
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=FvZ79uHV;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=ezq6leGc;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=061b0cdbb=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=061b0cdbb=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=061b0cdbb=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.155.168:from];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Mar 10, 2022 at 11:34:26AM +0200, Ze Dupsys wrote:
> On 2022.03.09. 10:42, Roger Pau Monné wrote:
> > On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote:
> > > If i execute steps 1 and 4, it seems that machine does not crash; tested 8
> > > hours Dom0 8GB RAM, 4 hours Dom0 2GB RAM. So creating a lot of ZVOLs and
> > > monitoring it does not seem to be the cause.
> > > 
> > > Yesterday was a happy and a sad day at the same time. The happy thing is
> > > that i think that i have understood one panic reason and somewhat better
> > > located where  one problem might be, but at the same time, i think i have
> > > located 2 bugs that cause one specific panic, but 2 other panic reasons are
> > > due to different circumstances.
> > > 
> > > Further is my pure speculation about two things, somewhat related; i do not
> > > know XEN or kernel internals, thus take this with grain of salt.
> > > 
> > > Knowing that this seemed to be RAM problem and thinking how ZFS sysctl
> > > tuning values differ, i started to look at "sysctl -a" diffs for Dom0 with
> > > 2GB RAM and 8GB RAM, by pure luck because comparing Dom0 that was running
> > > longer and just recently restarted, i noticed there are diff lines like so:
> > > ..
> > > +dev.xbbd.202.%driver: xbbd
> > > +dev.xbbd.202.%location:
> > > +dev.xbbd.202.%parent: xenbusb_back0
> > > +dev.xbbd.202.%pnpinfo:
> > > +dev.xbbd.202.xenbus_connection_state: Closed
> > > +dev.xbbd.202.xenbus_dev_type: vbd
> > > +dev.xbbd.202.xenbus_peer_domid: 309
> > > +dev.xbbd.202.xenstore_path: backend/vbd/309/51728
> > > +dev.xbbd.202.xenstore_peer_path: /local/domain/309/device/vbd/51728
> > > +dev.xbbd.203.%desc: Backend Virtual Block Device
> > > +dev.xbbd.203.%driver: xbbd
> > > +dev.xbbd.203.%location:
> > > +dev.xbbd.203.%parent: xenbusb_back0
> > > +dev.xbbd.203.%pnpinfo:
> > > +dev.xbbd.203.xenbus_connection_state: Closed
> > > ..
> > > although actually in system at the time there were only 2 DomUs running max.
> > > This seems to be the reason why this felt like memory leak, since after long
> > > time of start/stop VMs, most probably sysctl hit some sort of RAM limit
> > > (which most probably is auto-calculated from total RAM somehow) and panics.
> > Those dev.xbbd.XXX sysctls reference a specific blkback instance, and
> > should be gone once the guest is shutdown. Do you mean that you have a
> > range of dev.xbbd.[0-203] entries in your sysctl?
> Yes, i have that range of those entries. Even if all DomUs are destroyed.
> Snippet from logs - yesterday started, today crashed:
> cat 2022_03_10__1646887233_sysctl__001.log | grep xbbd | sed -Ee
> 's/(xbbd\.[0-9]*)\..*/\1/g' | sort | uniq
> dev.xbbd.%parent:
> dev.xbbd.0
> dev.xbbd.1
> dev.xbbd.10
> dev.xbbd.100
> dev.xbbd.101
> dev.xbbd.102
> dev.xbbd.103
> dev.xbbd.104
> dev.xbbd.105
> dev.xbbd.106
> dev.xbbd.107
> dev.xbbd.108
> dev.xbbd.109
> dev.xbbd.11
> ..
> dev.xbbd.52
> dev.xbbd.520
> dev.xbbd.521
> dev.xbbd.522
> dev.xbbd.523
> dev.xbbd.524
> ..
> dev.xbbd.99
> irq804: xbbd520:133 @cpu0(domain0): 436527
> irq805: xbbd521:135 @cpu0(domain0): 4777
> irq806: xbbd522:137 @cpu0(domain0): 16589254
> irq807: xbbd523:139 @cpu0(domain0): 6412
> irq808: xbbd524:141 @cpu0(domain0): 103083
> 
> And those entries keep growing.

That's not expected. Can you paste the output of `xenstore-ls -fp`
when you get those stale entries?

Also, what does `xl list`?

Also, can you check the log files at '/var/log/xen/xl-*.log' (where * is
the domain name) to try to gather why the backend is not properly
destroyed?

There's clearly something that prevents blkback from shutting down and
releasing its resources.

> Yesterday, i tried to follow Brian's email
> about working setup with FreeBSD 12.1. With given scripts 8GB RAM Dom0,
> still crashed, took longer, but on morning i saw that it machine rebooted,
> in serial logs had panic message. It seems that "kern.panic_reboot_wait_time
> = -1" did not disable rebooting as well. In monitoring logs i see that
> approx. 3 minutes after crash, testing machine was up and running again
> (rebooted).
> 
> 
> > Note that in dev.xbbd.XXX XXX matches the domain ID of the guest
> > that's using the backend.
> On my setups i have tested this does not seem to be the case. If each VM has
> more than 1 HDD, i have that many instances of xbbd sysctl variables as
> total count of HDDs attached to VMs. But at some point after rebooting
> machines or something, this number starts to grow on each create. For me it
> seems that dom ID is in sysctl, like this "dev.xbbd.5.xenbus_peer_domid",
> and if i grep those values, they show domids for nonexistent VMs.

Right, my bad. Those numbers are assigned by FreeBSD to identify
multiple instances of the same driver.

> I do not know how sysctls are created, but is there some test suite that
> creates awfully lot of sysctl variables and tries to crash system that way?
> I suppose there are no ways to limit sysctl variable count or any protection
> mechanisms for such cases. Then i could see if panic message is the same. Or
> test that allocates sysctl variable with insaneley large value, or appends
> to it.

It's likely the sysctl plus the associated blkback bounce buffers for
example, that are not freed. The problem is that somehow blkback
instances are not properly freed.

> > > Then i caught another sysctl variable that is growing due to XEN,
> > > "kern.msgbuf: Contents of kernel message buffer". I do not know how this
> > > variable grows or by which component it is managed, but in VM start/stop
> > > case it grows and contains lines with pattern like so:
> > > ..
> > > xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status
> > > xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet
> > > xnb(xnb_dump_txreq:299): netif_tx_request index =0
> > > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0
> > > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
> > > xnb(xnb_dump_txreq:302): netif_tx_request.flags =8
> > > xnb(xnb_dump_txreq:303): netif_tx_request.id    =69
> > > xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000
> > > xnb(xnb_dump_txreq:299): netif_tx_request index =1
> > > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255
> > > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
> > > xnb(xnb_dump_txreq:302): netif_tx_request.flags =0
> > > xnb(xnb_dump_txreq:303): netif_tx_request.id    =0
> > > xnb(xnb_dump_txreq:304): netif_tx_request.size  =0
> > > ..
> > > 
> > > Those lines in that variable just keep growing and growing, it is not that
> > > they are flushed, trimmed or anything. Each time i get the same message on
> > > serial output, it has one more section of error appended to "same-previous"
> > > serial output message and sysctl variable as well. Thus at some point serial
> > > output and sysctl contains a large block of those errors while VM is
> > > starting. So at some point the value of this sysctl could be reaching max
> > > allowed/available and this makes the system panic.  I do not know the reason
> > > for those errors, but actually if there was a patch to suppress them, this
> > > could be "solved". Another diff chunk might be related to this:
> > > +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0
> > > +dev.xnb.1.xenbus_peer_domid: 7
> > > +dev.xnb.1.xenbus_connection_state: InitWait
> > > +dev.xnb.1.xenbus_dev_type: vif
> > > +dev.xnb.1.xenstore_path: backend/vif/7/0
> > > +dev.xnb.1.dump_rings:
> > > +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error:
> > > nr_reqs == 0
> > > +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup,
> > > &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0
> > > +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup,
> > > xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0
> > > +52 Tests Passed
> > > +1 Tests FAILED
> > So you have failed tests for netback. Maybe the issue is with
> > netback rather than blkback.
> I am not sure where the problem is. Maybe there are two or more problems.
> Should they be discussed separately network related and disk related?
> 
> 
> > > What was suspicious about this is that i am using 13.0-RELEASE-p7, it is not
> > > a DEV version of XEN or anything, but there are sysctl variables with "dev"
> > > prefix. Could it be that ports have been accidentally compiled XEN version
> > > with some development flags turned on?
> > 'dev' here means device, not developer.
> Okay. Understood.
> 
> 
> > > So those were my conclusions.
> > > 
> > > What do you think? How should we proceed with this? Should i try to somehow
> > > build XEN from git sources?
> > I think that's unlikely to make any difference. I would think the
> > problem is with FreeBSD rather than Xen. Can you paste the config file
> > you are using the create the domain(s)?
> Yet again, as i said, i do not know where the problem is. My wording might
> be off when pointing that it's Xen, since Dom0 is FreeBSD and it panics not
> Xen. It's just that i do not know the boundaries and responsibilities of
> components clearly enough. And often by mentioning Xen i mean whole setup or
> FreeBSD module that interacts with Xen.
> 
> 
> > I've tried myself to trigger this crash by creating and destroying a
> > guest in a loop but didn't manage to trigger it. I was able to create
> > (and shutdown) 1328 guests successfully on a ZFS dom0 using 4GB of
> > RAM.
> No, just turning on/off single VM does not crash system. I had tried that as
> well before reporting bug. I had attached all /bin/sh scripts and configs
> that crash the system in bug report #261059 tar archive. Those scripts are
> really simple. I've been trying to minimize/trim everything off since then,
> but the simplest requirements for crash are executing 3 processes in
> parallel:
> 
> 1) loop that creates ZFS volumes at least of size 2GB and with dd writes
> data into them, /dev/zero is good. I've never run out of space, thus crash
> is not due to exhausted space in ZFS pool.
> 2) loop that creates VM1 with single disk, connects through ssh to that VM
> and writes 3GB data in /tmp, then reboots, starts VM, removes file from
> /tmp, then reboots
> 3) loop that starts/stops VM2 with 5 disks.
> 
> As for panic timing. Less RAM for Dom0, increases speed for panic, panic
> message is the same. For 8GB RAM on my machine i have to wait for approx. 10
> hours till panic, on 4G RAM around 6 hours. On resource starved Dom0 with
> 2GB, even shorter, but panic messages are more or less the same. I have
> crashed just by omitting step 1, but it takes longer. Running just on/off
> for both VMs, system does not crash. Thus it seems that some sort of ZFS
> load (VM internal or Dom0) is essential for panic to occur. I have managed
> to crash with VM which has 2 HDDs, thus it does not seem to be due to 4 HDD
> threshold emulated/virtio disks. It's just that with 5, it takes less time
> till crash. Usually panic occurs after when xbbd.$value, $value > 230.

Right, at some point you will run out of memory if resources are not
properly freed. Can you try to boot with "boot_verbose=YES" in
/boot/loader.conf and see if that gives you are more information?

Otherwise I might have to provide you with a patch to blkback in order
to attempt to detect why backends are not destroyed.

Since you stress the system quite badly, do you by any chance see 'xl'
processes getting terminated? Background xl processes being killed
will lead to backends not being properly shut down.

Regards, Roger.

From nobody Thu Mar 10 15:42:34 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 158F91A0763C
	for ; Thu, 10 Mar 2022 15:42:51 +0000 (UTC)
	(envelope-from prvs=061b0cdbb=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KDtcV0ZpLz3mhT
	for ; Thu, 10 Mar 2022 15:42:50 +0000 (UTC)
	(envelope-from prvs=061b0cdbb=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1646926970;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=M0v0++VOnJrkskTvsZ0OlT2RdftyKvfiuPy43WmZ8Zs=;
  b=PkHkMZyiEiXdSLW5/M/ho6V5trO+TywuJhOkb5QjOFqF6BX3Q71ThSDM
   VvrgzdrDxkbAvWhBub95WyH4yUOCK+5SvQPPe6hS+LlReliP8s3xEg1p0
   I+09JfbGlT5/34DjM6rChYaQbzUjy61hbTpMlohpwHJfK4vHB1hbynBn7
   w=;
X-SBRS: 5.1
X-MesageID: 66328601
X-Ironport-Server: esa1.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:oRPluq2kOkiSsDCWL/bD5cJxkn2cJEfYwER7XKvMYLTBsI5bp2BRm
 mAWWGyPaPePYjf9f9AiYYW38kwCuZ7Wy9NjG1Q/pC1hF35El5HIVI+TRqvS04J+DSFhoGZPt
 Zh2hgzodZhsJpPkjk7xdOCn9xGQ7InQLlbGILes1htZGEk1EE/NtTo5w7Rj2tUx2YDja++wk
 YiaT/P3aQfNNwFcagr424rbwP+4lK2v0N+wlgVWicFj5DcypVFMZH4sDfjZw0/DaptVBoaHq
 9Prl9lVyI97EyAFUbtJmp6jGqEDryW70QKm0hK6UID66vROS7BbPg/W+5PwZG8O4whlkeydx
 /1cn7L3Ql4zbpaQgdYEcStYDAZsLYdZreqvzXiX6aR/zmXDenrohf5vEFs3LcsT/eMf7WNmr
 KJCbmpXN1ba2rzwkOnTpupE36zPKOHxO4wSoDd4xCzxBvc6W5HTBa7N4Le02R9u2JsUTaiGP
 qL1bxJ2chPtehFqIG4LVq8vut6WiSnnchBX/Qf9Sa0fvDGIkV0ZPKLWGNDIesaNWu1TkEyTr
 2aA+H72ajkULtGSwyGJt2KhgMfBhy72UpkeUrqi+ZZCklCf23QIGQY+Wl6yoP3/gUm7M++zM
 GRNpHBo9/JrshX2EJ+tBHVUvUJooDYNcfQXGLQguDrR05jNwTSZGU8WXhRePYlOWNANeRQm0
 VqAntXMDDNpsaGIRX/1yop4vQ9eKgBOczZcOHZsoR8tpoC6/dpt1k6nosNLTfbt5uAZDw0c1
 NxjQMIWo7wIxfAG2Kyglbwsq2L9/8OZJuLZC+i+Y45E0u+bTNP9D2BLwQKChRqlEGp/ZgPf1
 JTjs5LChN3i9bnXyESwrBwlRdlFHcqtPjzGmkJIFJI87Tmr8HPLVdkOvG8ueR4yaZ5aKWGBj
 KrvVeV5vs870JyCN/MfXm5MI55ykfiI+SrNCpg4keaikrAuLVTarUmClGab3nz3kVhErE3ME
 czzTCpYNl5DUf4P5GPvH481iOZ3rghjlTK7bc2qlHyPjOvBDEN5vJ9YaTNimMhit/jayOgUm
 v4CX/a3J+J3C7SvPHOIrdZNcTjn7xETXPjLliCeTcbaSiJOE2A9Ef7Bh7Qnfo1uhaNOkenUu
 Hq6XydlJJDX1RUr9S3ihqhfVY7S
IronPort-HdrOrdr: A9a23:lRRSPKmtQQ7K7TAGHlYcMhumEorpDfPIimdD5ihNYBxZY6Wkfp
 +V8sjzhCWatN9OYh0dcLC7WJVpQRvnhPhICK0qTMqftW7dyReVxeBZnPHfKljbehEWmdQtsJ
 uIH5IObOEYSGIK8voSgzPIY+rIouP3iJxA7N22pxwGIHAIGsNdBkVCe32m+yVNNXh77PECZe
 OhD6R81l2dkSN9VLXEOpBJZZmJm/T70LbdJTIWDR8u7weDyRuu9b7BChCdmjMTSSlGz7sO+X
 XM11WR3NTvj9iLjjvnk0PD5ZVfn9XsjvNFGcy3k8AQbhHhkByhaohNU6CL+Bo1vOaswlA3l8
 SkmWZrA+1Dr1fqOk2lqxrk3AftlB4o9n/Z0FedxUDupMToLQhKffZptMZ8SF/0+kAgtNZz3O
 ZgxGSCradaChvGgWDU+8XIfwsCrDv7nVMS1cooy1BPW4oXb7Fc6aYF+llOLZsGFCXmrKg6De
 hVCt3G7vo+SyLUU5nghBgu/DWQZAVxIv/fKXJy+PB9kgIm0EyR9nFohfD2xRw7hdcAo5ot3Z
 WyDk0nrsALciYsV9MOOA4we7rFNoXze2O4DIuzGyWvKEhVAQOEl3bIiI9FkN1CPqZ4i6cPpA
 ==
X-IronPort-AV: E=Sophos;i="5.90,171,1643691600"; 
   d="scan'208";a="66328601"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=E5Sh4rQ7maqeWiCAas+ZqA2Ftr/dUFPzd2m0HcT8xWSthPVwj8HNF1JTcy2a+jUDGArwJnrUAzwPbk3/b2qXFrl37tW2tsifR3IUaLa6dTawZ1VDf848OJ4iSSnhc4ZxuoBARV3ZVT37okhlCIFfrit6Zdr1rKwIG7A/EOI7LZKPr3GiTW4pDyR+ZPiXehFya1ZPHy+84HDdSqsiunhHOE5CS4n/Ci+NXLuF4a8DFTT6hJQRdpN9i5HJPpC3oF6kPk1f6ihDJAgm7MCKVy5+lBCgek7uSP3HPQK2CUEKq9Zz1xMJHmdAQlfe3meBx/5kzhgim+GjhH8PZJJckthJsw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=R48wM73mTMnuBRdkhb+kfb9TeN0aIIqsrwuXoKDWFOE=;
 b=k1KzBFSJ3ObT47LkY/Nc66mde6n0bSTE/eupvN0rBf6A9K68buFpFTIDedPwZS6bkdqcwqxz6f8EqB3FAS9hiYiGXn4HBzaR3CSponD0Rl/ieGW6E+An+L+DiSb6g0APK5UDPqNhav+ND589ITW7Z4k++dGomW5Frn1IYjpRemd/REDQUjUXPMGdt1zwuAV3HPNdrDDlSUDWm9NcSja4+u+YjGC3aMCjPLMubWpVdZ3JYab4IsOOsk760aYeTqUrsjM28KIR2lg0y1JhvBP88E+xPDu4wCg5KQs01EHKRPVOTcfxfqPQubzvGO/9PZ8qiam6EljzY1F3RCDUUXnebg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=R48wM73mTMnuBRdkhb+kfb9TeN0aIIqsrwuXoKDWFOE=;
 b=JLFWumrCn8lY/S5jLqJ5+469aEdCHCu1NECl54ZWNDXL51nNY0NskJPCoqt0ranvYtGg+Jd2FdtUdHTVD/YO6KCXZkfOb/KdO/JymGYNfsulSKsG4QxMpSHXk59y2u5xcpiIif+JAdBl/VHgPPad/LmHMUnT9sZxcaJU5EH9a+k=
Date: Thu, 10 Mar 2022 16:42:34 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <202203011540.221FeR4f028103@nfbcal.org>
 
 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
X-ClientProxiedBy: LO2P265CA0451.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:e::31) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: d8e5704c-6c9c-4a9a-bb18-08da02ac9b4b
X-MS-TrafficTypeDiagnostic: BN6PR03MB2563:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 7lRSO1MDDfADcX4GI/AzcABUMwOs5Nm/+8w0wpHMoumIBE5l0XkdgAdrtxSo6Kew0JGsOnrjCBCKhEmz4D+nfrKcgiSj9ArDm5WGBIFmrzdg+N4Xp+QIOIwrnRmIXkwhADLgpq+dxkotZNz1hnAo5DUDXs6lyS4lxWqF24zfvkb9/e4eiIB4u3y18paOdhGsF+ASG8KDSlu9pkbypbNa8Z7LnERfI2Be88hpJNzSVmbQXk5IZLye+PAfOB3hmzJ22MXM8hiDg2mp3MXreVqTIpuuPgGC86DkKV5JZ7QgB3DUEkBC45l9GnC6OaUSS4D9kIHqOURr3QnrChFTU2h2hC3E/o/UpdioRuGOZRAaazAhFZfBoJQIzOdrBpy9j6cSSnOFUHXusNJfSiYP/oM0ev+oRw4ff0rXgvoWNu++dY/tTSGx8tsAUfK2VD5X9kK6Fti54By/wF1PCviB26dV8o6+8MVAkV54YpdhAi9MgTIKhndXjOi6fuUp5YMahL0ihPPX9DOU9lxB0VAcmd7jnwvUF2yF9CbfQ5VtlcQ2jozRkyM0XiAM9Kp+NpPmRxrjY0Xh/9rzmKAzV13ZH360iV/EAsasFxORtNkGzJfYHdWRE5z/AIhRBB9rZWLx7xBc/Kw0+YUCjA/PViqXYSNnqXYY632y1nwuZ4SULLjJTRppGqrLIad0e7PMgFB6NaYXk2C3sho0rMTBfSiwrAF0QImEoi8R26s7684PdvkQziY=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(9686003)(26005)(186003)(6666004)(6512007)(66476007)(82960400001)(38100700002)(6506007)(83380400001)(2906002)(5660300002)(8936002)(85182001)(33716001)(6916009)(316002)(6486002)(966005)(66946007)(8676002)(66556008)(4326008)(86362001)(508600001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZFl2eGhRV25iZjNZVFNYTkR0c1hvcVA3c3VJNnhDWE9Zd1NlMWtpSnU4Tmcv?=
 =?utf-8?B?a1lka09QNnFNUHhPTzJwek1zYjQraVlIMjNDa2tKM3lPRjFFckZ5eS9IQmxp?=
 =?utf-8?B?OUg1VGxyMFVrb2hDTTZhbzBnblRiT1NRcmQ0dzlhVEpycFdBOVY2UE9DRkUv?=
 =?utf-8?B?a3NDdk1uSWtnRE00NXNkdkNaVHcwc3BXV0ZkRi8yMzJaRkFzbUlxUzdnOGZK?=
 =?utf-8?B?Nk5XTTB5Ui8wckxBVHg5NytndFhJMkt4YWRtQk5iQXpQclVFRmkvUW5BdEha?=
 =?utf-8?B?TWd4S0xZMG5NMWxnNUtQRHlhMVM2eWFGcE9yVnhSZDZUWHBLRlNBN3Zac3BJ?=
 =?utf-8?B?VG94Q090U3RSY2JtYjdNTEJENFlvYUZaTHYyekFZMU5GaFEwV1BRL1hCY2po?=
 =?utf-8?B?bXFkMytMYVNlaWRLdnhDdUVjZ1UwYVJVb3hmMXBDMENrRUtxTDVyTytYNzMw?=
 =?utf-8?B?alVuMlR6Vk9Kc0doUDVOS0pqYWFLdXVEM1R0VVlyb1p3UGpNbFRPbmtjOXBE?=
 =?utf-8?B?TERlZ0l3WkRXeUZKVkNKaEMrT21zbHVDb0o3cVNoZ2k0L2VBZW9JME14SC9u?=
 =?utf-8?B?dWp2YllOc2h0SkN1aWVVaENab1hlMTU3YisxVU5CNk5SazJIakNOMzFydHcv?=
 =?utf-8?B?OEJZcklMMkltYjVuOENJa3REbk1pK1BkWFhTQjBsT1VCZ0dWK2EvVHRjdWNr?=
 =?utf-8?B?TlRrYlErWUxzc0UwMmo1OTZDYWVjaTEyckFKdDNWR0FRU255WDBWV1JUalJv?=
 =?utf-8?B?VVRRSDNuQUprRk80UFV3SjMrdm5KZmFTengxcnQ2M3hNK01tMzM2K0x3L1B6?=
 =?utf-8?B?bktSQXJQemdrS3ZoaC9yQ1ppOTludFFjNW9EZ2dPZ1h6YjhJMEJsKzZQcVB6?=
 =?utf-8?B?dlEzMk9uK0pTQUR4eVdySkpKTit0VTNKS0pCblFRYm95TEdnRzROOHJNeUkr?=
 =?utf-8?B?QmFBbDlLWWVBY1hjK2xwMFJ2Y3pkNVkzV2J5dkpTZGZuelUvNkVXYzJmK3NB?=
 =?utf-8?B?ZWxTNGxDT1VoMEpyWW5tVzZ4Q0RkZkFYYTV3SUtCMWhwRzI4Uktzalgyek0w?=
 =?utf-8?B?WnVDNy9BSTkvSHJlQUhjN093YllMQnczSWtvSkJZVVhkS2tWYXBsMGNFcVdj?=
 =?utf-8?B?YTlhTFhLZlM1YUMvK2FGa1NKMWxWbm5iYTBSSDJwajUxTFJkL3JwcmVhQ016?=
 =?utf-8?B?WHp3RFpNMUFxWWg3dGlQY0gveVFJVk9LaXk1NUJyOVBob3Nsam03OTRpaHI5?=
 =?utf-8?B?WlR4Ylo5ZlFqRlZ1RUFMdUt3RVZVVnJvbkE3SkpWT3d3N1hXaS84NWJxclZx?=
 =?utf-8?B?bFlOUmtYeFI0VUZmaHZKb3RiMm5OR1dhcGxoRUNiV1M0MnZBelJwdG85RmFE?=
 =?utf-8?B?aU9scnRJWXFsTk1tbGJSS2gvakpkQXZzRjdTMUZteFZVVUk0VFVwa0NBbm5j?=
 =?utf-8?B?a21OUlp4d3Y0dnFMK0VjNW1hNC8zZVRGNk1relRVQk1LaE9BWGFrNmpKTnVh?=
 =?utf-8?B?TFRMQTlhT1JYcG4zUHVzUUF2ZWl3SWRzbmFKREc3V0RYekNIUmpjbzhGNldh?=
 =?utf-8?B?RTNqR1E5QXp2Wnc1NjdaQkFCNUN5Vkl0bld1RmxHUnVrTTZXa1lsVUcyaktF?=
 =?utf-8?B?R3JDcWRsdXIxOE5EOHVOYkFrN2hzVE9XM3d3blhNaEFOUlBaWkVBVGlVL2lJ?=
 =?utf-8?B?YnJvNGl0NUlRK0VzQktqRkg3QXozanhXTmZYL2F4OVFqVEFSQzkyR0hRVmZu?=
 =?utf-8?B?RU1yWFA5V1BXWUpYdWNoS2UzeVpXMjlCd0VKcWh3UFc0R2tzMzBoSVllSzNs?=
 =?utf-8?B?aTI5NWllQURvWnNkTWpBQkV0bGNzLzhib1VCRWpqS0NpQUd0bEFmSkVWajl4?=
 =?utf-8?B?R2Iwdlh3S1gwaEIrUjRhSGh1WU9jNUFYazlWSW1JUkZFS0x6ZjRrTmpLSTI1?=
 =?utf-8?B?TDJ4c1grYitPWkwwQmNrajh6aTNOWWtEZjJkL1Fpa2duRkYvUDhKVXFhVHVD?=
 =?utf-8?B?b1BFdys2NlJnPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d8e5704c-6c9c-4a9a-bb18-08da02ac9b4b
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2022 15:42:39.0502
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JJIkJQaFOdM4NyDU0ZsIVSCJLKbPD35WBhxMUrnLiGrtJADudc9rspYkLRRmnl1sV+8h8WVQIiSl31DmAjEoWg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB2563
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KDtcV0ZpLz3mhT
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=PkHkMZyi;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=JLFWumrC;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=061b0cdbb=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=061b0cdbb=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=061b0cdbb=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Mar 10, 2022 at 01:41:28PM +0200, Ze Dupsys wrote:
> On 2022.03.09. 10:42, Roger Pau Monné wrote:
> > On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote:
> > > Then i caught another sysctl variable that is growing due to XEN,
> > > "kern.msgbuf: Contents of kernel message buffer". I do not know how this
> > > variable grows or by which component it is managed, but in VM start/stop
> > > case it grows and contains lines with pattern like so:
> > > ..
> > > xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status
> > > xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet
> > > xnb(xnb_dump_txreq:299): netif_tx_request index =0
> > > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0
> > > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
> > > xnb(xnb_dump_txreq:302): netif_tx_request.flags =8
> > > xnb(xnb_dump_txreq:303): netif_tx_request.id    =69
> > > xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000
> > > xnb(xnb_dump_txreq:299): netif_tx_request index =1
> > > xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255
> > > xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
> > > xnb(xnb_dump_txreq:302): netif_tx_request.flags =0
> > > xnb(xnb_dump_txreq:303): netif_tx_request.id    =0
> > > xnb(xnb_dump_txreq:304): netif_tx_request.size  =0
> > > ..
> > > 
> > > Those lines in that variable just keep growing and growing, it is not that
> > > they are flushed, trimmed or anything. Each time i get the same message on
> > > serial output, it has one more section of error appended to "same-previous"
> > > serial output message and sysctl variable as well. Thus at some point serial
> > > output and sysctl contains a large block of those errors while VM is
> > > starting. So at some point the value of this sysctl could be reaching max
> > > allowed/available and this makes the system panic.  I do not know the reason
> > > for those errors, but actually if there was a patch to suppress them, this
> > > could be "solved". Another diff chunk might be related to this:
> > > +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0
> > > +dev.xnb.1.xenbus_peer_domid: 7
> > > +dev.xnb.1.xenbus_connection_state: InitWait
> > > +dev.xnb.1.xenbus_dev_type: vif
> > > +dev.xnb.1.xenstore_path: backend/vif/7/0
> > > +dev.xnb.1.dump_rings:
> > > +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error:
> > > nr_reqs == 0
> > > +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup,
> > > &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0
> > > +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup,
> > > xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0
> > > +52 Tests Passed
> > > +1 Tests FAILED
> > So you have failed tests for netback. Maybe the issue is with
> > netback rather than blkback.
> 
> Just found that there are no errors with network, most probably. Based on
> https://reviews.freebsd.org/D9234?id=24172#201024, since i was monitoring
> sysctl variables with 'sysctl -a', most probably that did "sysctl
> dev.xnb.0.unit_test_results" implicitly for each xnb interface, thus the
> extra output, since reading value actually runs tests. So now i upgraded my
> monitoring command to 'sysctl -a -N'. Never did i expect that sysctl
> variable reading could trigger test suite execution.
> 
> I did not understand from comments though - how do you instantiate xnb
> interface?
> 
> "ifconfig xnb create" returned:
> ifconfig: SIOCIFCREATE2: Invalid argument

You need to create a guest to instantiate a xnb interface. Those
cannot be created by ifconfig because they rely on a matching
frontend.

Roger.

From nobody Thu Mar 10 17:46:21 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C042E19F9316
	for ; Thu, 10 Mar 2022 17:46:07 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KDxLl0ybqz4bWx
	for ; Thu, 10 Mar 2022 17:46:07 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x233.google.com with SMTP id q5so8786243ljb.11
        for ; Thu, 10 Mar 2022 09:46:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=subject:to:cc:references:reply-to:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language;
        bh=BS+7Cy78gO5GysUTtEdVqYQimTQuoecyCDMIXoJh/pg=;
        b=CrLdrMM7r5HvYFkKO7GeZRhDaOGj2OqFwkALvrEWlkg4t03wO3Ecaz+BIZoHhpmqpM
         7h/iCTyS7DH/VxEXgeaVg2jdDwgo7RPLbZ3KWTM8hc78IbKLSAbEpQCaNLE74BK5te1E
         cCwUzZYMdkmvxkOrjdBytxLBxZnh3rhGGLOeuSZKRZA+c2v4o+uJyioXOk28bGb4sJSN
         /aprKXzbE1Gx5LEI1K4hlTizkW9LMEtf6gAWFzJdpvvsDK6H3Yzjda9I5wK+2In1aEIK
         vWnGxc743UmWkTMDUW6IbIssoPYwshFHU5IXcwi8XEUa0+bGEJke5u6q8LxHTRXWGN87
         K8rg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:subject:to:cc:references:reply-to:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language;
        bh=BS+7Cy78gO5GysUTtEdVqYQimTQuoecyCDMIXoJh/pg=;
        b=AJaq1JWyNNuboJU29yE05Ny0wMru6b9FLw/WRd/n6LPD8yJYGNh5FFm1Tw/w0hFjNp
         +dgCr98o7kv3l2uLU1DmdBgyYSeFzxiOWLaLGn+eCxt6NFUePtBcSElhbi2RqmkdeRCD
         lueyyviL/HNLpAvXesJjn3DGUk0rXOtYTjWN3UKcazg2fqI1KygtMYm+UiYX1PB2Pedn
         qgBbYqCraURMNJNQFWnLfqNYcQVN6kJpOTvp/yvyOTgMn5uR2fsLROE7Mcc8x0xg73vk
         IMJRvkSpBz/Ep/kKp8RKKxBplHPp4thAl+cUVdnGlb7w9X4TYuWnpWDOFqN9Z8Zjmm/8
         J86g==
X-Gm-Message-State: AOAM5338TsL13wynA2xNbZWoUCoEnhfvZ4TeB8TPYHBALyBs3lXxUVTg
	yWbdsnWdVdWzCC2xaFyxANw=
X-Google-Smtp-Source: ABdhPJyKYVcsUXNRcE37tDjvVGFpMbNlXiiDlNjXYzE9xnhQSTdDvFmhNLXKK55bW0QUGVvR6FXhog==
X-Received: by 2002:a2e:9dcf:0:b0:246:3ff1:d770 with SMTP id x15-20020a2e9dcf000000b002463ff1d770mr3616204ljj.330.1646934365818;
        Thu, 10 Mar 2022 09:46:05 -0800 (PST)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id u15-20020a2e2e0f000000b00247e32add6esm1188593lju.0.2022.03.10.09.46.04
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Thu, 10 Mar 2022 09:46:04 -0800 (PST)
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 <202203011540.221FeR4f028103@nfbcal.org>
 
 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <061700f9-86aa-b2ae-07a4-27d4c3d96966@gmail.com>
 
Reply-To: zedupsys@gmail.com
From: Ze Dupsys 
Message-ID: 
Date: Thu, 10 Mar 2022 19:46:21 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/mixed;
 boundary="------------8101AB55BF1195A52550F29E"
Content-Language: en-US
X-Rspamd-Queue-Id: 4KDxLl0ybqz4bWx
X-Spamd-Bar: ++
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=CrLdrMM7;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::233 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [2.84 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 HAS_ATTACHMENT(0.00)[];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 MIME_BASE64_TEXT_BOGUS(1.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 MIME_BASE64_TEXT(0.10)[];
	 CTYPE_MIXED_BOGUS(1.00)[];
	 NEURAL_HAM_SHORT(-0.16)[-0.158];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 MIME_GOOD(-0.10)[multipart/mixed,text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_SPAM_MEDIUM(1.00)[0.997];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_SPAM_LONG(0.90)[0.902];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::233:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

This is a multi-part message in MIME format.
--------------8101AB55BF1195A52550F29E
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2022.03.10. 17:40, Roger Pau Monné wrote:
> That's not expected. Can you paste the output of `xenstore-ls -fp`
> when you get those stale entries?
Yes, i can (xenstore_ls_fp.log), today i was stressing 12.1, 8GB RAM for 
10 hours.
I will power off all DomUs attach output of commands to this email.


> Also, what does `xl list`?
I attached xl_list.log, not to loose formatting due to email software. 
Added "sysctl -a -N | grep xbbd" as well, since there can be seen that 
sysctl variables exist.


> Also, can you check the log files at '/var/log/xen/xl-*.log' (where * is
> the domain name) to try to gather why the backend is not properly
> destroyed?
Those files seem not to be helpful, I have not seen different messages 
than those, i.e xl-xen-vm2-zvol-5.log.7:
Waiting for domain xen-vm2-zvol-5 (domid 486) to die [pid 11628]
Domain 486 has shut down, reason code 0 0x0
Action for shutdown reason code 0 is destroy
Domain 486 needs to be cleaned up: destroying the domain
Done. Exiting now


BTW is it possible to disable automatic log file rotation in 
/var/log/xen? Or make it always append to single file so that no info is 
lost? Maybe there has been some info for domain ID 200 for example, it's 
just that i have never seen different messages there.


Same goes for qemu-dm-xen-* files, nothing eye catchy:
qemu-system-i386: -serial pty: char device redirected to /dev/pts/6 
(label serial0)
VNC server running on 0.0.0.0:5907
qemu-system-i386: terminating on signal 1 from pid 24773 (xl)


> Right, at some point you will run out of memory if resources are not
> properly freed. Can you try to boot with "boot_verbose=YES" in
> /boot/loader.conf and see if that gives you are more information?
Yes, i will add and see if some more info is available.


> Otherwise I might have to provide you with a patch to blkback in order
> to attempt to detect why backends are not destroyed.
Sure. I'll just have to learn how to compile Xen from source then. I 
think i saw somewhere a howto.

Which version of FreeBSD should i stress? 13.0? Or then it all must be 
built from current source?


> Since you stress the system quite badly, do you by any chance see 'xl'
> processes getting terminated? Background xl processes being killed
> will lead to backends not being properly shut down.
I wouldn't stress it so much if it behaved in the first place when the 
load was normal :)

I've never seen any process containing xl in it's name being killed, 
qemu-system-i386 have been OOM killed though, but on those instances 
usually VM just does not start or cleans up and is gone. And i've seen 
system panic without OOM kill messages.

For the crash cases on normal load, i just noticed that when that 
happened it was always when i had some sort of HDD load, but i never 
remembered how many VMs i had started or restarted. Thus i invented 
these stress test scripts in hope to find the reason why those crashes 
ever happened. I even would not mind if just DomU's would crash, as long 
as their HDDs are not corrupt and Dom0 stays stable and can auto reboot 
them.

As for tests, i've been experimenting in various ways to somehow 
understand if it is due to VM restart count, attached HDD count or data 
throughput on HDDs. But for now i'm going in circles, because sometimes 
panic happens early, but sometimes it happens between domID 2 to 3 
hundred or later and those non-deterministic cases are hard to locate.

Thanks!


--------------8101AB55BF1195A52550F29E
Content-Type: text/plain; charset=UTF-8;
 name="xenstore_ls_fp.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="xenstore_ls_fp.log"

L3Rvb2wgPSAiIiAgIChuMCkKL3Rvb2wveGVuc3RvcmVkID0gIiIgICAobjApCi9sb2NhbCA9
ICIiICAgKG4wKQovbG9jYWwvZG9tYWluID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMCA9
ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvZG9taWQgPSAiMCIgICAobjApCi9sb2NhbC9k
b21haW4vMC9uYW1lID0gIkRvbWFpbi0wIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkID0gIiIgICAobjAp
Ci9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ID0gIiIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC80LzUxNzEyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC80LzUxNzEyL2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3MTIvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIx
IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3MTIvbWF4LXJpbmct
cGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQv
NTE3MjggPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3Mjgv
ZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvNC81MTcyOC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvNC81MTcyOC9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNC81MTc0NCA9ICIiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNC81MTc0NC9mZWF0dXJlLWJhcnJpZXIgPSAi
MSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80LzUxNzQ0L2ZlYXR1cmUt
Zmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80
LzUxNzQ0L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC80LzUxNzYwID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNr
ZW5kL3ZiZC80LzUxNzYwL2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2Rv
bWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3NjAvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAg
IChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3NjAvbWF4LXJpbmctcGFn
ZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3
NzYgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQvNTE3NzYvZmVh
dHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQv
NC81MTc3Ni9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWlu
LzAvYmFja2VuZC92YmQvNC81MTc3Ni9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4w
KQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzcgPSAiIiAgIChuMCkKL2xvY2FsL2Rv
bWFpbi8wL2JhY2tlbmQvdmJkLzc3LzUxNzEyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC83Ny81MTcxMi9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9s
b2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83Ny81MTcxMi9mZWF0dXJlLWZsdXNoLWNhY2hl
ID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzcvNTE3MTIvbWF4
LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQv
dmJkLzc3LzUxNzI4ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83
Ny81MTcyOC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC83Ny81MTcyOC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzcvNTE3MjgvbWF4LXJpbmctcGFnZS1vcmRl
ciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzc3LzUxNzQ0ID0g
IiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83Ny81MTc0NC9mZWF0dXJl
LWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83Ny81
MTc0NC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAv
YmFja2VuZC92YmQvNzcvNTE3NDQvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzc3LzUxNzYwID0gIiIgICAobjApCi9sb2Nh
bC9kb21haW4vMC9iYWNrZW5kL3ZiZC83Ny81MTc2MC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83Ny81MTc2MC9mZWF0dXJlLWZs
dXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzcv
NTE3NjAvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8w
L2JhY2tlbmQvdmJkLzc3LzUxNzc2ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNr
ZW5kL3ZiZC83Ny81MTc3Ni9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9k
b21haW4vMC9iYWNrZW5kL3ZiZC83Ny81MTc3Ni9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEi
ICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzcvNTE3NzYvbWF4LXJpbmct
cGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzc5
ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83OS81MTcxMiA9ICIi
ICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3MTIvZmVhdHVyZS1i
YXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3
MTIvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2Jh
Y2tlbmQvdmJkLzc5LzUxNzEyL21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9s
b2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83OS81MTcyOCA9ICIiICAgKG4wKQovbG9jYWwv
ZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3MjgvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3MjgvZmVhdHVyZS1mbHVz
aC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzc5LzUx
NzI4L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC83OS81MTc0NCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2Vu
ZC92YmQvNzkvNTE3NDQvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvNzkvNTE3NDQvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAg
IChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzc5LzUxNzQ0L21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83OS81
MTc2MCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3NjAv
ZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvNzkvNTE3NjAvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2Rv
bWFpbi8wL2JhY2tlbmQvdmJkLzc5LzUxNzYwL21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC83OS81MTc3NiA9ICIiICAgKG4w
KQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3NzYvZmVhdHVyZS1iYXJyaWVy
ID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNzkvNTE3NzYvZmVh
dHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQv
dmJkLzc5LzUxNzc2L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9k
b21haW4vMC9iYWNrZW5kL3ZiZC85MCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFj
a2VuZC92YmQvOTAvNTE3NDQgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQv
dmJkLzkwLzUxNzQ0L2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzkwLzUxNzQ0L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAo
bjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC85MC81MTc0NC9tYXgtcmluZy1wYWdl
LW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvOTAvNTE3
NjAgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzkwLzUxNzYwL2Zl
YXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJk
LzkwLzUxNzYwL2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC85MC81MTc2MC9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvOTAvNTE3NzYgPSAiIiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzkwLzUxNzc2L2ZlYXR1cmUtYmFycmllciA9
ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzkwLzUxNzc2L2ZlYXR1
cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3Zi
ZC85MC81MTc3Ni9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvMTEyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNr
ZW5kL3ZiZC8xMTIvNTE3MTIgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQv
dmJkLzExMi81MTcxMi9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3MTIvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAg
IChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzExMi81MTcxMi9tYXgtcmluZy1w
YWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTEy
LzUxNzI4ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3
MjgvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2Vu
ZC92YmQvMTEyLzUxNzI4L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2Nh
bC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3MjgvbWF4LXJpbmctcGFnZS1vcmRlciA9
ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzExMi81MTc0NCA9ICIi
ICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTEyLzUxNzQ0L2ZlYXR1cmUt
YmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzExMi81
MTc0NC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAv
YmFja2VuZC92YmQvMTEyLzUxNzQ0L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjAp
Ci9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3NjAgPSAiIiAgIChuMCkKL2xv
Y2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzExMi81MTc2MC9mZWF0dXJlLWJhcnJpZXIgPSAi
MSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3NjAvZmVhdHVy
ZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJk
LzExMi81MTc2MC9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvMTEyLzUxNzc2ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC8xMTIvNTE3NzYvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTEyLzUxNzc2L2ZlYXR1cmUtZmx1c2gtY2Fj
aGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMTIvNTE3NzYv
bWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzEyMCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTIw
LzUxNzEyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMjAvNTE3
MTIvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2Vu
ZC92YmQvMTIwLzUxNzEyL2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2Nh
bC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMjAvNTE3MTIvbWF4LXJpbmctcGFnZS1vcmRlciA9
ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzEyMC81MTcyOCA9ICIi
ICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTIwLzUxNzI4L2ZlYXR1cmUt
YmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzEyMC81
MTcyOC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAv
YmFja2VuZC92YmQvMTIwLzUxNzI4L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjAp
Ci9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMjAvNTE3NDQgPSAiIiAgIChuMCkKL2xv
Y2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzEyMC81MTc0NC9mZWF0dXJlLWJhcnJpZXIgPSAi
MSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMjAvNTE3NDQvZmVhdHVy
ZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJk
LzEyMC81MTc0NC9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvMTIwLzUxNzYwID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC8xMjAvNTE3NjAvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTIwLzUxNzYwL2ZlYXR1cmUtZmx1c2gtY2Fj
aGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8xMjAvNTE3NjAv
bWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzEyMC81MTc3NiA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvMTIwLzUxNzc2L2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzEyMC81MTc3Ni9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMTIwLzUxNzc2L21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8yNzAg
PSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzI3MC81MTcxMiA9ICIi
ICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMjcwLzUxNzEyL2ZlYXR1cmUt
YmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzI3MC81
MTcxMi9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAv
YmFja2VuZC92YmQvMjcwLzUxNzEyL21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjAp
Ci9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8yNzAvNTE3MjggPSAiIiAgIChuMCkKL2xv
Y2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzI3MC81MTcyOC9mZWF0dXJlLWJhcnJpZXIgPSAi
MSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8yNzAvNTE3MjgvZmVhdHVy
ZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJk
LzI3MC81MTcyOC9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvMjcwLzUxNzQ0ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC8yNzAvNTE3NDQvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMjcwLzUxNzQ0L2ZlYXR1cmUtZmx1c2gtY2Fj
aGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8yNzAvNTE3NDQv
bWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzI3MC81MTc2MCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvMjcwLzUxNzYwL2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzI3MC81MTc2MC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMjcwLzUxNzYwL21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8yNzAv
NTE3NzYgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzI3MC81MTc3
Ni9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5k
L3ZiZC8yNzAvNTE3NzYvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzI3MC81MTc3Ni9tYXgtcmluZy1wYWdlLW9yZGVyID0g
IjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMzc5ID0gIiIgICAobjAp
Ci9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8zNzkvNTE3MTIgPSAiIiAgIChuMCkKL2xv
Y2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzM3OS81MTcxMi9mZWF0dXJlLWJhcnJpZXIgPSAi
MSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8zNzkvNTE3MTIvZmVhdHVy
ZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJk
LzM3OS81MTcxMi9tYXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvMzc5LzUxNzI4ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC8zNzkvNTE3MjgvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMzc5LzUxNzI4L2ZlYXR1cmUtZmx1c2gtY2Fj
aGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8zNzkvNTE3Mjgv
bWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzM3OS81MTc0NCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvMzc5LzUxNzQ0L2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzM3OS81MTc0NC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMzc5LzUxNzQ0L21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8zNzkv
NTE3NjAgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzM3OS81MTc2
MC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5k
L3ZiZC8zNzkvNTE3NjAvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzM3OS81MTc2MC9tYXgtcmluZy1wYWdlLW9yZGVyID0g
IjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMzc5LzUxNzc2ID0gIiIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC8zNzkvNTE3NzYvZmVhdHVyZS1i
YXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvMzc5LzUx
Nzc2L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC8zNzkvNTE3NzYvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4MSA9ICIiICAgKG4wKQovbG9jYWwvZG9t
YWluLzAvYmFja2VuZC92YmQvNDgxLzUxNzEyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4v
MC9iYWNrZW5kL3ZiZC80ODEvNTE3MTIvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQov
bG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDgxLzUxNzEyL2ZlYXR1cmUtZmx1c2gtY2Fj
aGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODEvNTE3MTIv
bWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzQ4MS81MTcyOCA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvNDgxLzUxNzI4L2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzQ4MS81MTcyOC9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDgxLzUxNzI4L21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODEv
NTE3NDQgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4MS81MTc0
NC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5k
L3ZiZC80ODEvNTE3NDQvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4MS81MTc0NC9tYXgtcmluZy1wYWdlLW9yZGVyID0g
IjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDgxLzUxNzYwID0gIiIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODEvNTE3NjAvZmVhdHVyZS1i
YXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDgxLzUx
NzYwL2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC80ODEvNTE3NjAvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4MS81MTc3NiA9ICIiICAgKG4wKQovbG9j
YWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDgxLzUxNzc2L2ZlYXR1cmUtYmFycmllciA9ICIx
IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4MS81MTc3Ni9mZWF0dXJl
LWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQv
NDgxLzUxNzc2L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC80ODggPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tl
bmQvdmJkLzQ4OC81MTcxMiA9ICIiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92
YmQvNDg4LzUxNzEyL2ZlYXR1cmUtYmFycmllciA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFp
bi8wL2JhY2tlbmQvdmJkLzQ4OC81MTcxMi9mZWF0dXJlLWZsdXNoLWNhY2hlID0gIjEiICAg
KG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDg4LzUxNzEyL21heC1yaW5nLXBh
Z2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODgv
NTE3MjggPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4OC81MTcy
OC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5k
L3ZiZC80ODgvNTE3MjgvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4OC81MTcyOC9tYXgtcmluZy1wYWdlLW9yZGVyID0g
IjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDg4LzUxNzQ0ID0gIiIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODgvNTE3NDQvZmVhdHVyZS1i
YXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDg4LzUx
NzQ0L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC80ODgvNTE3NDQvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4OC81MTc2MCA9ICIiICAgKG4wKQovbG9j
YWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDg4LzUxNzYwL2ZlYXR1cmUtYmFycmllciA9ICIx
IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4OC81MTc2MC9mZWF0dXJl
LWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQv
NDg4LzUxNzYwL21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC80ODgvNTE3NzYgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8w
L2JhY2tlbmQvdmJkLzQ4OC81MTc3Ni9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9s
b2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80ODgvNTE3NzYvZmVhdHVyZS1mbHVzaC1jYWNo
ZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ4OC81MTc3Ni9t
YXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2Vu
ZC92YmQvNDkyID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80OTIv
NTE3MTIgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ5Mi81MTcx
Mi9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5k
L3ZiZC80OTIvNTE3MTIvZmVhdHVyZS1mbHVzaC1jYWNoZSA9ICIxIiAgIChuMCkKL2xvY2Fs
L2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ5Mi81MTcxMi9tYXgtcmluZy1wYWdlLW9yZGVyID0g
IjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDkyLzUxNzI4ID0gIiIg
ICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80OTIvNTE3MjgvZmVhdHVyZS1i
YXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDkyLzUx
NzI4L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAobjApCi9sb2NhbC9kb21haW4vMC9i
YWNrZW5kL3ZiZC80OTIvNTE3MjgvbWF4LXJpbmctcGFnZS1vcmRlciA9ICI1IiAgIChuMCkK
L2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ5Mi81MTc0NCA9ICIiICAgKG4wKQovbG9j
YWwvZG9tYWluLzAvYmFja2VuZC92YmQvNDkyLzUxNzQ0L2ZlYXR1cmUtYmFycmllciA9ICIx
IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ5Mi81MTc0NC9mZWF0dXJl
LWZsdXNoLWNhY2hlID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2VuZC92YmQv
NDkyLzUxNzQ0L21heC1yaW5nLXBhZ2Utb3JkZXIgPSAiNSIgICAobjApCi9sb2NhbC9kb21h
aW4vMC9iYWNrZW5kL3ZiZC80OTIvNTE3NjAgPSAiIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8w
L2JhY2tlbmQvdmJkLzQ5Mi81MTc2MC9mZWF0dXJlLWJhcnJpZXIgPSAiMSIgICAobjApCi9s
b2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80OTIvNTE3NjAvZmVhdHVyZS1mbHVzaC1jYWNo
ZSA9ICIxIiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzQ5Mi81MTc2MC9t
YXgtcmluZy1wYWdlLW9yZGVyID0gIjUiICAgKG4wKQovbG9jYWwvZG9tYWluLzAvYmFja2Vu
ZC92YmQvNDkyLzUxNzc2ID0gIiIgICAobjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3Zi
ZC80OTIvNTE3NzYvZmVhdHVyZS1iYXJyaWVyID0gIjEiICAgKG4wKQovbG9jYWwvZG9tYWlu
LzAvYmFja2VuZC92YmQvNDkyLzUxNzc2L2ZlYXR1cmUtZmx1c2gtY2FjaGUgPSAiMSIgICAo
bjApCi9sb2NhbC9kb21haW4vMC9iYWNrZW5kL3ZiZC80OTIvNTE3NzYvbWF4LXJpbmctcGFn
ZS1vcmRlciA9ICI1IiAgIChuMCkKL2xvY2FsL2RvbWFpbi8wL2RldmljZS1tb2RlbCA9ICIi
ICAgKG4wKQovdm0gPSAiIiAgIChuMCkKL2xpYnhsID0gIiIgICAobjApCg==
--------------8101AB55BF1195A52550F29E
Content-Type: text/plain; charset=UTF-8;
 name="xl_list.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="xl_list.log"

TmFtZSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBJRCAgIE1lbSBW
Q1BVcwlTdGF0ZQlUaW1lKHMpCkRvbWFpbi0wICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDAgIDgxOTEgICAgIDIgICAgIHItLS0tLSAgIDI5MDIzLjIK
--------------8101AB55BF1195A52550F29E
Content-Type: text/plain; charset=UTF-8;
 name="sysctl_aN_grep_xbbd.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="sysctl_aN_grep_xbbd.log"

ZGV2LnhiYmQuNTMueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjUzLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjUzLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjUz
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC41My54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjUz
LiVwYXJlbnQKZGV2LnhiYmQuNTMuJXBucGluZm8KZGV2LnhiYmQuNTMuJWxvY2F0aW9uCmRl
di54YmJkLjUzLiVkcml2ZXIKZGV2LnhiYmQuNTMuJWRlc2MKZGV2LnhiYmQuNTIueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjUyLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjUy
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjUyLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC41Mi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjUyLiVwYXJlbnQKZGV2LnhiYmQu
NTIuJXBucGluZm8KZGV2LnhiYmQuNTIuJWxvY2F0aW9uCmRldi54YmJkLjUyLiVkcml2ZXIK
ZGV2LnhiYmQuNTIuJWRlc2MKZGV2LnhiYmQuNTEueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjUxLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjUxLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjUxLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC41MS54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjUxLiVwYXJlbnQKZGV2LnhiYmQuNTEuJXBucGluZm8KZGV2Lnhi
YmQuNTEuJWxvY2F0aW9uCmRldi54YmJkLjUxLiVkcml2ZXIKZGV2LnhiYmQuNTEuJWRlc2MK
ZGV2LnhiYmQuNTAueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjUwLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjUwLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjUw
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC41MC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjUw
LiVwYXJlbnQKZGV2LnhiYmQuNTAuJXBucGluZm8KZGV2LnhiYmQuNTAuJWxvY2F0aW9uCmRl
di54YmJkLjUwLiVkcml2ZXIKZGV2LnhiYmQuNTAuJWRlc2MKZGV2LnhiYmQuNDkueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjQ5LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQ5
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQ5LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC40OS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQ5LiVwYXJlbnQKZGV2LnhiYmQu
NDkuJXBucGluZm8KZGV2LnhiYmQuNDkuJWxvY2F0aW9uCmRldi54YmJkLjQ5LiVkcml2ZXIK
ZGV2LnhiYmQuNDkuJWRlc2MKZGV2LnhiYmQuNDgueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjQ4LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQ4LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjQ4LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40OC54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjQ4LiVwYXJlbnQKZGV2LnhiYmQuNDguJXBucGluZm8KZGV2Lnhi
YmQuNDguJWxvY2F0aW9uCmRldi54YmJkLjQ4LiVkcml2ZXIKZGV2LnhiYmQuNDguJWRlc2MK
ZGV2LnhiYmQuNDcueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjQ3LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjQ3LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQ3
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40Ny54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQ3
LiVwYXJlbnQKZGV2LnhiYmQuNDcuJXBucGluZm8KZGV2LnhiYmQuNDcuJWxvY2F0aW9uCmRl
di54YmJkLjQ3LiVkcml2ZXIKZGV2LnhiYmQuNDcuJWRlc2MKZGV2LnhiYmQuNDYueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjQ2LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQ2
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQ2LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC40Ni54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQ2LiVwYXJlbnQKZGV2LnhiYmQu
NDYuJXBucGluZm8KZGV2LnhiYmQuNDYuJWxvY2F0aW9uCmRldi54YmJkLjQ2LiVkcml2ZXIK
ZGV2LnhiYmQuNDYuJWRlc2MKZGV2LnhiYmQuNDUueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjQ1LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQ1LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjQ1LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40NS54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjQ1LiVwYXJlbnQKZGV2LnhiYmQuNDUuJXBucGluZm8KZGV2Lnhi
YmQuNDUuJWxvY2F0aW9uCmRldi54YmJkLjQ1LiVkcml2ZXIKZGV2LnhiYmQuNDUuJWRlc2MK
ZGV2LnhiYmQuNDQueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjQ0LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjQ0LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQ0
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40NC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQ0
LiVwYXJlbnQKZGV2LnhiYmQuNDQuJXBucGluZm8KZGV2LnhiYmQuNDQuJWxvY2F0aW9uCmRl
di54YmJkLjQ0LiVkcml2ZXIKZGV2LnhiYmQuNDQuJWRlc2MKZGV2LnhiYmQuNDMueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjQzLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQz
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQzLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC40My54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQzLiVwYXJlbnQKZGV2LnhiYmQu
NDMuJXBucGluZm8KZGV2LnhiYmQuNDMuJWxvY2F0aW9uCmRldi54YmJkLjQzLiVkcml2ZXIK
ZGV2LnhiYmQuNDMuJWRlc2MKZGV2LnhiYmQuNDIueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjQyLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQyLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjQyLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40Mi54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjQyLiVwYXJlbnQKZGV2LnhiYmQuNDIuJXBucGluZm8KZGV2Lnhi
YmQuNDIuJWxvY2F0aW9uCmRldi54YmJkLjQyLiVkcml2ZXIKZGV2LnhiYmQuNDIuJWRlc2MK
ZGV2LnhiYmQuNDEueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjQxLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjQxLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQx
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC40MS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQx
LiVwYXJlbnQKZGV2LnhiYmQuNDEuJXBucGluZm8KZGV2LnhiYmQuNDEuJWxvY2F0aW9uCmRl
di54YmJkLjQxLiVkcml2ZXIKZGV2LnhiYmQuNDEuJWRlc2MKZGV2LnhiYmQuNDAueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjQwLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjQw
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjQwLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC40MC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjQwLiVwYXJlbnQKZGV2LnhiYmQu
NDAuJXBucGluZm8KZGV2LnhiYmQuNDAuJWxvY2F0aW9uCmRldi54YmJkLjQwLiVkcml2ZXIK
ZGV2LnhiYmQuNDAuJWRlc2MKZGV2LnhiYmQuMzkueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjM5LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjM5LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjM5LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zOS54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjM5LiVwYXJlbnQKZGV2LnhiYmQuMzkuJXBucGluZm8KZGV2Lnhi
YmQuMzkuJWxvY2F0aW9uCmRldi54YmJkLjM5LiVkcml2ZXIKZGV2LnhiYmQuMzkuJWRlc2MK
ZGV2LnhiYmQuMzgueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjM4LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjM4LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjM4
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zOC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjM4
LiVwYXJlbnQKZGV2LnhiYmQuMzguJXBucGluZm8KZGV2LnhiYmQuMzguJWxvY2F0aW9uCmRl
di54YmJkLjM4LiVkcml2ZXIKZGV2LnhiYmQuMzguJWRlc2MKZGV2LnhiYmQuMzcueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjM3LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjM3
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjM3LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4zNy54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjM3LiVwYXJlbnQKZGV2LnhiYmQu
MzcuJXBucGluZm8KZGV2LnhiYmQuMzcuJWxvY2F0aW9uCmRldi54YmJkLjM3LiVkcml2ZXIK
ZGV2LnhiYmQuMzcuJWRlc2MKZGV2LnhiYmQuMzYueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjM2LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjM2LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjM2LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zNi54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjM2LiVwYXJlbnQKZGV2LnhiYmQuMzYuJXBucGluZm8KZGV2Lnhi
YmQuMzYuJWxvY2F0aW9uCmRldi54YmJkLjM2LiVkcml2ZXIKZGV2LnhiYmQuMzYuJWRlc2MK
ZGV2LnhiYmQuMzUueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjM1LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjM1LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjM1
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zNS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjM1
LiVwYXJlbnQKZGV2LnhiYmQuMzUuJXBucGluZm8KZGV2LnhiYmQuMzUuJWxvY2F0aW9uCmRl
di54YmJkLjM1LiVkcml2ZXIKZGV2LnhiYmQuMzUuJWRlc2MKZGV2LnhiYmQuMzQueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjM0LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjM0
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjM0LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4zNC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjM0LiVwYXJlbnQKZGV2LnhiYmQu
MzQuJXBucGluZm8KZGV2LnhiYmQuMzQuJWxvY2F0aW9uCmRldi54YmJkLjM0LiVkcml2ZXIK
ZGV2LnhiYmQuMzQuJWRlc2MKZGV2LnhiYmQuMzMueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjMzLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjMzLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjMzLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zMy54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjMzLiVwYXJlbnQKZGV2LnhiYmQuMzMuJXBucGluZm8KZGV2Lnhi
YmQuMzMuJWxvY2F0aW9uCmRldi54YmJkLjMzLiVkcml2ZXIKZGV2LnhiYmQuMzMuJWRlc2MK
ZGV2LnhiYmQuMzIueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjMyLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjMyLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjMy
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zMi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjMy
LiVwYXJlbnQKZGV2LnhiYmQuMzIuJXBucGluZm8KZGV2LnhiYmQuMzIuJWxvY2F0aW9uCmRl
di54YmJkLjMyLiVkcml2ZXIKZGV2LnhiYmQuMzIuJWRlc2MKZGV2LnhiYmQuMzEueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjMxLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjMx
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjMxLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4zMS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjMxLiVwYXJlbnQKZGV2LnhiYmQu
MzEuJXBucGluZm8KZGV2LnhiYmQuMzEuJWxvY2F0aW9uCmRldi54YmJkLjMxLiVkcml2ZXIK
ZGV2LnhiYmQuMzEuJWRlc2MKZGV2LnhiYmQuMzAueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjMwLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjMwLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjMwLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4zMC54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjMwLiVwYXJlbnQKZGV2LnhiYmQuMzAuJXBucGluZm8KZGV2Lnhi
YmQuMzAuJWxvY2F0aW9uCmRldi54YmJkLjMwLiVkcml2ZXIKZGV2LnhiYmQuMzAuJWRlc2MK
ZGV2LnhiYmQuMjkueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjI5LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjI5LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjI5
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yOS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjI5
LiVwYXJlbnQKZGV2LnhiYmQuMjkuJXBucGluZm8KZGV2LnhiYmQuMjkuJWxvY2F0aW9uCmRl
di54YmJkLjI5LiVkcml2ZXIKZGV2LnhiYmQuMjkuJWRlc2MKZGV2LnhiYmQuMjgueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjI4LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjI4
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjI4LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4yOC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjI4LiVwYXJlbnQKZGV2LnhiYmQu
MjguJXBucGluZm8KZGV2LnhiYmQuMjguJWxvY2F0aW9uCmRldi54YmJkLjI4LiVkcml2ZXIK
ZGV2LnhiYmQuMjguJWRlc2MKZGV2LnhiYmQuMjcueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjI3LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjI3LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjI3LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yNy54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjI3LiVwYXJlbnQKZGV2LnhiYmQuMjcuJXBucGluZm8KZGV2Lnhi
YmQuMjcuJWxvY2F0aW9uCmRldi54YmJkLjI3LiVkcml2ZXIKZGV2LnhiYmQuMjcuJWRlc2MK
ZGV2LnhiYmQuMjYueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjI2LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjI2LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjI2
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yNi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjI2
LiVwYXJlbnQKZGV2LnhiYmQuMjYuJXBucGluZm8KZGV2LnhiYmQuMjYuJWxvY2F0aW9uCmRl
di54YmJkLjI2LiVkcml2ZXIKZGV2LnhiYmQuMjYuJWRlc2MKZGV2LnhiYmQuMjUueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjI1LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjI1
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjI1LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4yNS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjI1LiVwYXJlbnQKZGV2LnhiYmQu
MjUuJXBucGluZm8KZGV2LnhiYmQuMjUuJWxvY2F0aW9uCmRldi54YmJkLjI1LiVkcml2ZXIK
ZGV2LnhiYmQuMjUuJWRlc2MKZGV2LnhiYmQuMjQueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjI0LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjI0LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjI0LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yNC54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjI0LiVwYXJlbnQKZGV2LnhiYmQuMjQuJXBucGluZm8KZGV2Lnhi
YmQuMjQuJWxvY2F0aW9uCmRldi54YmJkLjI0LiVkcml2ZXIKZGV2LnhiYmQuMjQuJWRlc2MK
ZGV2LnhiYmQuMjMueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjIzLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjIzLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjIz
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yMy54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjIz
LiVwYXJlbnQKZGV2LnhiYmQuMjMuJXBucGluZm8KZGV2LnhiYmQuMjMuJWxvY2F0aW9uCmRl
di54YmJkLjIzLiVkcml2ZXIKZGV2LnhiYmQuMjMuJWRlc2MKZGV2LnhiYmQuMjIueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjIyLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjIy
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjIyLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4yMi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjIyLiVwYXJlbnQKZGV2LnhiYmQu
MjIuJXBucGluZm8KZGV2LnhiYmQuMjIuJWxvY2F0aW9uCmRldi54YmJkLjIyLiVkcml2ZXIK
ZGV2LnhiYmQuMjIuJWRlc2MKZGV2LnhiYmQuMjEueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjIxLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjIxLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjIxLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yMS54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjIxLiVwYXJlbnQKZGV2LnhiYmQuMjEuJXBucGluZm8KZGV2Lnhi
YmQuMjEuJWxvY2F0aW9uCmRldi54YmJkLjIxLiVkcml2ZXIKZGV2LnhiYmQuMjEuJWRlc2MK
ZGV2LnhiYmQuMTcueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjE3LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjE3LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjE3
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xNy54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjE3
LiVwYXJlbnQKZGV2LnhiYmQuMTcuJXBucGluZm8KZGV2LnhiYmQuMTcuJWxvY2F0aW9uCmRl
di54YmJkLjE3LiVkcml2ZXIKZGV2LnhiYmQuMTcuJWRlc2MKZGV2LnhiYmQuMTYueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjE2LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjE2
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjE2LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4xNi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjE2LiVwYXJlbnQKZGV2LnhiYmQu
MTYuJXBucGluZm8KZGV2LnhiYmQuMTYuJWxvY2F0aW9uCmRldi54YmJkLjE2LiVkcml2ZXIK
ZGV2LnhiYmQuMTYuJWRlc2MKZGV2LnhiYmQuMjAueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjIwLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjIwLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjIwLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4yMC54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjIwLiVwYXJlbnQKZGV2LnhiYmQuMjAuJXBucGluZm8KZGV2Lnhi
YmQuMjAuJWxvY2F0aW9uCmRldi54YmJkLjIwLiVkcml2ZXIKZGV2LnhiYmQuMjAuJWRlc2MK
ZGV2LnhiYmQuMTkueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjE5LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjE5LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjE5
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xOS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjE5
LiVwYXJlbnQKZGV2LnhiYmQuMTkuJXBucGluZm8KZGV2LnhiYmQuMTkuJWxvY2F0aW9uCmRl
di54YmJkLjE5LiVkcml2ZXIKZGV2LnhiYmQuMTkuJWRlc2MKZGV2LnhiYmQuMTgueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjE4LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjE4
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjE4LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4xOC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjE4LiVwYXJlbnQKZGV2LnhiYmQu
MTguJXBucGluZm8KZGV2LnhiYmQuMTguJWxvY2F0aW9uCmRldi54YmJkLjE4LiVkcml2ZXIK
ZGV2LnhiYmQuMTguJWRlc2MKZGV2LnhiYmQuMTUueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjE1LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjE1LnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjE1LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xNS54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjE1LiVwYXJlbnQKZGV2LnhiYmQuMTUuJXBucGluZm8KZGV2Lnhi
YmQuMTUuJWxvY2F0aW9uCmRldi54YmJkLjE1LiVkcml2ZXIKZGV2LnhiYmQuMTUuJWRlc2MK
ZGV2LnhiYmQuMTQueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjE0LnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjE0LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjE0
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xNC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjE0
LiVwYXJlbnQKZGV2LnhiYmQuMTQuJXBucGluZm8KZGV2LnhiYmQuMTQuJWxvY2F0aW9uCmRl
di54YmJkLjE0LiVkcml2ZXIKZGV2LnhiYmQuMTQuJWRlc2MKZGV2LnhiYmQuMTMueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjEzLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjEz
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjEzLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4xMy54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjEzLiVwYXJlbnQKZGV2LnhiYmQu
MTMuJXBucGluZm8KZGV2LnhiYmQuMTMuJWxvY2F0aW9uCmRldi54YmJkLjEzLiVkcml2ZXIK
ZGV2LnhiYmQuMTMuJWRlc2MKZGV2LnhiYmQuMTIueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54
YmJkLjEyLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjEyLnhlbmJ1c19jb25uZWN0aW9u
X3N0YXRlCmRldi54YmJkLjEyLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xMi54ZW5zdG9y
ZV9wYXRoCmRldi54YmJkLjEyLiVwYXJlbnQKZGV2LnhiYmQuMTIuJXBucGluZm8KZGV2Lnhi
YmQuMTIuJWxvY2F0aW9uCmRldi54YmJkLjEyLiVkcml2ZXIKZGV2LnhiYmQuMTIuJWRlc2MK
ZGV2LnhiYmQuMTEueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjExLnhlbmJ1c19wZWVy
X2RvbWlkCmRldi54YmJkLjExLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjEx
LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xMS54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjEx
LiVwYXJlbnQKZGV2LnhiYmQuMTEuJXBucGluZm8KZGV2LnhiYmQuMTEuJWxvY2F0aW9uCmRl
di54YmJkLjExLiVkcml2ZXIKZGV2LnhiYmQuMTEuJWRlc2MKZGV2LnhiYmQuMTAueGVuc3Rv
cmVfcGVlcl9wYXRoCmRldi54YmJkLjEwLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjEw
LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjEwLnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC4xMC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjEwLiVwYXJlbnQKZGV2LnhiYmQu
MTAuJXBucGluZm8KZGV2LnhiYmQuMTAuJWxvY2F0aW9uCmRldi54YmJkLjEwLiVkcml2ZXIK
ZGV2LnhiYmQuMTAuJWRlc2MKZGV2LnhiYmQuOS54ZW5zdG9yZV9wZWVyX3BhdGgKZGV2Lnhi
YmQuOS54ZW5idXNfcGVlcl9kb21pZApkZXYueGJiZC45LnhlbmJ1c19jb25uZWN0aW9uX3N0
YXRlCmRldi54YmJkLjkueGVuYnVzX2Rldl90eXBlCmRldi54YmJkLjkueGVuc3RvcmVfcGF0
aApkZXYueGJiZC45LiVwYXJlbnQKZGV2LnhiYmQuOS4lcG5waW5mbwpkZXYueGJiZC45LiVs
b2NhdGlvbgpkZXYueGJiZC45LiVkcml2ZXIKZGV2LnhiYmQuOS4lZGVzYwpkZXYueGJiZC44
LnhlbnN0b3JlX3BlZXJfcGF0aApkZXYueGJiZC44LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54
YmJkLjgueGVuYnVzX2Nvbm5lY3Rpb25fc3RhdGUKZGV2LnhiYmQuOC54ZW5idXNfZGV2X3R5
cGUKZGV2LnhiYmQuOC54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjguJXBhcmVudApkZXYueGJi
ZC44LiVwbnBpbmZvCmRldi54YmJkLjguJWxvY2F0aW9uCmRldi54YmJkLjguJWRyaXZlcgpk
ZXYueGJiZC44LiVkZXNjCmRldi54YmJkLjcueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJk
LjcueGVuYnVzX3BlZXJfZG9taWQKZGV2LnhiYmQuNy54ZW5idXNfY29ubmVjdGlvbl9zdGF0
ZQpkZXYueGJiZC43LnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC43LnhlbnN0b3JlX3BhdGgK
ZGV2LnhiYmQuNy4lcGFyZW50CmRldi54YmJkLjcuJXBucGluZm8KZGV2LnhiYmQuNy4lbG9j
YXRpb24KZGV2LnhiYmQuNy4lZHJpdmVyCmRldi54YmJkLjcuJWRlc2MKZGV2LnhiYmQuNi54
ZW5zdG9yZV9wZWVyX3BhdGgKZGV2LnhiYmQuNi54ZW5idXNfcGVlcl9kb21pZApkZXYueGJi
ZC42LnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRldi54YmJkLjYueGVuYnVzX2Rldl90eXBl
CmRldi54YmJkLjYueGVuc3RvcmVfcGF0aApkZXYueGJiZC42LiVwYXJlbnQKZGV2LnhiYmQu
Ni4lcG5waW5mbwpkZXYueGJiZC42LiVsb2NhdGlvbgpkZXYueGJiZC42LiVkcml2ZXIKZGV2
LnhiYmQuNi4lZGVzYwpkZXYueGJiZC41LnhlbnN0b3JlX3BlZXJfcGF0aApkZXYueGJiZC41
LnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjUueGVuYnVzX2Nvbm5lY3Rpb25fc3RhdGUK
ZGV2LnhiYmQuNS54ZW5idXNfZGV2X3R5cGUKZGV2LnhiYmQuNS54ZW5zdG9yZV9wYXRoCmRl
di54YmJkLjUuJXBhcmVudApkZXYueGJiZC41LiVwbnBpbmZvCmRldi54YmJkLjUuJWxvY2F0
aW9uCmRldi54YmJkLjUuJWRyaXZlcgpkZXYueGJiZC41LiVkZXNjCmRldi54YmJkLjQueGVu
c3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjQueGVuYnVzX3BlZXJfZG9taWQKZGV2LnhiYmQu
NC54ZW5idXNfY29ubmVjdGlvbl9zdGF0ZQpkZXYueGJiZC40LnhlbmJ1c19kZXZfdHlwZQpk
ZXYueGJiZC40LnhlbnN0b3JlX3BhdGgKZGV2LnhiYmQuNC4lcGFyZW50CmRldi54YmJkLjQu
JXBucGluZm8KZGV2LnhiYmQuNC4lbG9jYXRpb24KZGV2LnhiYmQuNC4lZHJpdmVyCmRldi54
YmJkLjQuJWRlc2MKZGV2LnhiYmQuMy54ZW5zdG9yZV9wZWVyX3BhdGgKZGV2LnhiYmQuMy54
ZW5idXNfcGVlcl9kb21pZApkZXYueGJiZC4zLnhlbmJ1c19jb25uZWN0aW9uX3N0YXRlCmRl
di54YmJkLjMueGVuYnVzX2Rldl90eXBlCmRldi54YmJkLjMueGVuc3RvcmVfcGF0aApkZXYu
eGJiZC4zLiVwYXJlbnQKZGV2LnhiYmQuMy4lcG5waW5mbwpkZXYueGJiZC4zLiVsb2NhdGlv
bgpkZXYueGJiZC4zLiVkcml2ZXIKZGV2LnhiYmQuMy4lZGVzYwpkZXYueGJiZC4yLnhlbnN0
b3JlX3BlZXJfcGF0aApkZXYueGJiZC4yLnhlbmJ1c19wZWVyX2RvbWlkCmRldi54YmJkLjIu
eGVuYnVzX2Nvbm5lY3Rpb25fc3RhdGUKZGV2LnhiYmQuMi54ZW5idXNfZGV2X3R5cGUKZGV2
LnhiYmQuMi54ZW5zdG9yZV9wYXRoCmRldi54YmJkLjIuJXBhcmVudApkZXYueGJiZC4yLiVw
bnBpbmZvCmRldi54YmJkLjIuJWxvY2F0aW9uCmRldi54YmJkLjIuJWRyaXZlcgpkZXYueGJi
ZC4yLiVkZXNjCmRldi54YmJkLjEueGVuc3RvcmVfcGVlcl9wYXRoCmRldi54YmJkLjEueGVu
YnVzX3BlZXJfZG9taWQKZGV2LnhiYmQuMS54ZW5idXNfY29ubmVjdGlvbl9zdGF0ZQpkZXYu
eGJiZC4xLnhlbmJ1c19kZXZfdHlwZQpkZXYueGJiZC4xLnhlbnN0b3JlX3BhdGgKZGV2Lnhi
YmQuMS4lcGFyZW50CmRldi54YmJkLjEuJXBucGluZm8KZGV2LnhiYmQuMS4lbG9jYXRpb24K
ZGV2LnhiYmQuMS4lZHJpdmVyCmRldi54YmJkLjEuJWRlc2MKZGV2LnhiYmQuJXBhcmVudAo=
--------------8101AB55BF1195A52550F29E--

From nobody Thu Mar 10 18:30:11 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CD84C1A03CA1
	for ; Thu, 10 Mar 2022 18:30:20 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KDyKl4LC1z4jKp
	for ; Thu, 10 Mar 2022 18:30:19 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 22AIUBbZ002288
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 10 Mar 2022 10:30:11 -0800 (PST)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 22AIUBGi021294;
	Thu, 10 Mar 2022 10:30:11 -0800 (PST)
Message-Id: <202203101830.22AIUBGi021294@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 10 Mar 2022 10:30:11 -0800
In-Reply-To: <1937e88f-159a-5324-db8f-687e84a0bea9@gmail.com>
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 10 Mar 2022 10:30:12 -0800 (PST)
X-Rspamd-Queue-Id: 4KDyKl4LC1z4jKp
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FREEMAIL_TO(0.00)[gmail.com];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	Hello.  I think the easiest way to prevent FreeBSD dom0 from "tasting" the domu zfs
partitions is to set volmode=none in the zfs properties list for the volumes you expect to be
formatted by domu's asas zfs partitions.  This doesn't prevent Xen from exporting the volume,
but does tell the dom0 to keep its nose out of the domu disks.  
	I'll answer your other questions in a separate e-mail.
-thanks
-Brian


From nobody Mon Mar 14 08:06:58 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EDA491A1702D
	for ; Mon, 14 Mar 2022 08:07:01 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KH8Jj1QxPz3pgG
	for ; Mon, 14 Mar 2022 08:07:01 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lf1-x132.google.com with SMTP id 3so25598614lfr.7
        for ; Mon, 14 Mar 2022 01:07:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=subject:to:cc:references:reply-to:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-transfer-encoding:content-language;
        bh=4xxOeJhLHt/sBb+3EHK/jdHbGdkBjUC3Yn6s7KNLgcI=;
        b=hD9T9Fxrfo/JJb+/GAvYk7dfnZBp+A5hi6zSC60NQyBlImYR9FD+kbD+NP9F6ohQ2e
         B581oh7R+GVPpYltZXeZWpXPrpXszwJX0FooSUBwc/acKMcVz+YgGsg5ATx4LJQcMw6q
         nrwHDu7oq/0ePcdabUIWeE5ljcfupeweb+48GqhQObGwO1RTUeYnmX+N0rBlSNTLXWCq
         1QwLc2JzK2J629llGfOv0DeGTrTWH1j3AuqeHinbast97mQ5C+UlNtVWZ8w6YWex0jmk
         ZDd5cVEZfiQNAkRCEdrPNRNC1F6Iz/qfvZEHN/6Z/gD1NJfbweD4K02+f4q3jlD/YcrF
         HZXg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:subject:to:cc:references:reply-to:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-transfer-encoding:content-language;
        bh=4xxOeJhLHt/sBb+3EHK/jdHbGdkBjUC3Yn6s7KNLgcI=;
        b=ITTzCZnwG6QsLGuPahN6L+eXOwhBss8FAi8nubTrYdlA0xMX+EOT+0L6r0iQE7PJ32
         CbiyUWG32odCwz6rssfTSSXl2lvfGfCaN3it1eYm5831QEGEK7RMbRYSCQU447W2ykyb
         pbj5S2RcqMLqkr5kVZ4AdJJsi35Ea/ac15zHRr/ZqZUceWNXlQrLnk6ubOJ7F+LTz+rI
         n+Ek6juN6bcBSMypNVeSuQXL7F3jUGAsOxVbD9KxDgNbWbw1EyrAs0MBQh6GKrtk4/ai
         xYXuY1bq46TSWgQYilh4WhSk018gfcxn4lwJZdMoSPuk5ij4mD9r3WOCYqzj29Ur0zU1
         7dvQ==
X-Gm-Message-State: AOAM530khHMP8h5VZ3vmh7J9sYNtmk3AZ63KbwiF1+2KNJR5yYY7Nh9u
	n1EEZpBZZpPobBIFpXAdB5px135i/D4=
X-Google-Smtp-Source: ABdhPJwwJeZZUiBGl3b8+PaX//P8LKSpHd0VFiTg1jMeZAg4dyBme39C5R4HBh/SECfqUvfbrdoCGQ==
X-Received: by 2002:a05:6512:b19:b0:446:f1c6:81bd with SMTP id w25-20020a0565120b1900b00446f1c681bdmr13031381lfu.320.1647245213655;
        Mon, 14 Mar 2022 01:06:53 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id v1-20020a2e6101000000b002480c4f33d5sm3160645ljb.132.2022.03.14.01.06.52
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 14 Mar 2022 01:06:53 -0700 (PDT)
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 <202203011540.221FeR4f028103@nfbcal.org>
 
 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
Reply-To: zedupsys@gmail.com
From: Ze Dupsys 
Message-ID: 
Date: Mon, 14 Mar 2022 10:06:58 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-Rspamd-Queue-Id: 4KH8Jj1QxPz3pgG
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=hD9T9Fxr;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-0.62 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-0.15)[-0.146];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_SPAM_MEDIUM(0.28)[0.281];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_SPAM_LONG(0.24)[0.245];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

I'd like to share more analysis on given problem. I do not know if this 
somehow helps or not, but i have noticed that across all my saved serial 
log outputs, panic messages follow after some of these lines.

..
(XEN) HVM d34v0 save: TSC_ADJUST
(XEN) HVM d34v0 save: CPU_MSR
(XEN) HVM34 restore: CPU 0
xnb(xnb_detach:1330):
xnb(xnb_detach:1339):
.. => panic


Most of panics are like this
..
(XEN) HVM d26v0 save: TSC_ADJUST
(XEN) HVM d26v0 save: CPU_MSR
(XEN) HVM26 restore: CPU 0
.. => panic

..
(XEN) HVM d42v0 save: TSC_ADJUST
(XEN) HVM d42v0 save: CPU_MSR
(XEN) HVM42 restore: CPU 0
xnb(xnb_detach:1330):
xnb(xnb_detach:1339):
xnb(xnb_detach:1330):
xnb(xnb_detach:1339):
.. => panic


This one i think had different stressing conditions than other's, but i 
don't remember
..
(XEN) HVM d660v0 save: CPU_MSR
(XEN) HVM660 restore: CPU 0
(XEN) d659v0: upcall vector 93
spin lock 0xffffffff81eaa780 (sched lock 1) held by 0xfffff8020152d000 
(tid 100434) too long
timeout stopping cpus
panic: spin lock held too long
.. => panic


For serial output in middle when there are no crashes i have noticed 
that there are at least 2 different execution paths.

For most VM's boot flow continues with serial lines like these:
..
(XEN) HVM1 restore: CPU 0
xnb(xnb_probe:1123): Claiming device 0, xnb
xnb(xnb_attach:1267): Attaching to backend/vif/1/0
xnb(xnb_frontend_changed:1391): frontend_state=Initialising, 
xnb_state=InitWait
(d1) HVM Loader
..

For some though, there are lines like these, but they still boot, it 
just seemed that these lines might be a possible continuation for 
"unsuccessful panic".
..
(XEN) HVM3 restore: CPU 0
xnb(xnb_detach:1330):
xnb(xnb_detach:1339):
xnb(xnb_detach:1330):
xnb(xnb_detach:1339):
xnb(xnb_probe:1123): Claiming device 0, xnb
xnb(xnb_attach:1267): Attaching to backend/vif/3/0
xnb(xnb_frontend_changed:1391): frontend_state=Initialising, 
xnb_state=InitWait
(d3) HVM Loader
..

Why those lines starting "xnb(xnb_detach:1330):" do not have any 
message? Could it be that there is a bad pointer to message buffer that 
can not be printed? And then sometimes panic happens because access goes 
out of allowed memory region?

Line numbers are just somewhat informational, since those messages are 
from across all my tests, various configs, versions. Yesterday i set up 
system with FreeBSD 13.1-STABLE, still can crash, same panic. What i do 
not know about those xnb messages is, to which VM they are related, 
since serial output is shared and on parallel while VM1 might be 
created, VM2 is starting or being destroyed.

Thanks.




From nobody Mon Mar 14 09:19:58 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A68D51A0E3D9
	for ; Mon, 14 Mar 2022 09:20:13 +0000 (UTC)
	(envelope-from prvs=0654505ba=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KH9x83W8Mz4Svn
	for ; Mon, 14 Mar 2022 09:20:12 +0000 (UTC)
	(envelope-from prvs=0654505ba=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1647249612;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=zCfDSlM/lVdduegzct8WpWHliePoj+iSDH1o/dhDOpw=;
  b=EjZ6S9ZsgkIlhr//RjrEp0zpU1MGng7SZ7yX4btLmwL2w22i44d0VPg/
   nGtSN0cge5wQ1VPbUKaDcANdKaddQ2qhAyrC7QBGCPaehs0geXUBTTcv2
   cpg+fjLYejrqLjn5aD9wkzjopkn8LZr9n4fkxqDW1voGnY4k7p1wJ9nNL
   4=;
X-SBRS: 5.1
X-MesageID: 66556240
X-Ironport-Server: esa1.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:n/3eJKuiYlRQU0LxxH1fkqhvi+fnVFReMUV32f8akzHdYApBsoF/q
 tZmKT3SOP2MYmSmf9x0a9ywpE4BsMWEn4BiSldsrnw3FSgU+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1jU5
 YuryyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi8kYLPQlM0XXyVXLGYkIqhg5I2aO2Sw5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AO
 ZdEN2AzNHwsZTV0an0OOZs1xt3xqXagVmV49m/LlIsOtj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcU3th+RxoPe7zqWOS9eUTtndO57rtM5EGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtAKzARVodt/xory9U
 J8swZT20Qz2JcvR/BFhuc1UdF1T296LMSfHnXlkFIQ7+jKm9haLJN4Mv2gvex0xbZldIVcFh
 XM/XisLuPdu0IaCN/crM+pd9ex3pUQfKTgVfq+NNYcfCnSAXASG4DtvdSatM5PFyyARfVUEE
 c7DK66EVC9CYYw+lWbeb7pNgNcDm3FlrUuOFM+T8vhS+efHDJJjYexeawXmgyFQxP7snTg5B
 P4EbprUkUoDCLOiCsQVmKZKRW03wbEALcmeg+Rcd/KZIxogH2ckCvTLxqgmdZAjlKNQ/tokN
 FnmMqOE4DITXUH6FDg=
IronPort-HdrOrdr: A9a23:530SqqCGpztEDZvlHelW55DYdb4zR+YMi2TDt3oddfWaSKylfq
 GV7ZAmPHrP4gr5N0tOpTntAse9qBDnhPtICOsqTNSftWDd0QPFEGgL1+DfKlbbak/DH4BmtJ
 uJc8JFeaDN5VoRt7eH3OFveexQv+Vu88qT9JnjJ28Gd3AMV0n5hT0JcTpyFCdNNW97LKt8Lr
 WwzOxdqQGtfHwGB/7LfEXsD4D41qT2fIuNW29/OyIa
X-IronPort-AV: E=Sophos;i="5.90,180,1643691600"; 
   d="scan'208";a="66556240"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=RoLXOWMe1aJGR32YeGXFjfoeunwIo0Fqk8Fn5A3WxlpjvlVl2fidGgHKnVMY9nHh0pTb2T7szP6Twm9vCpNnLPoSW1nvPZraYB3fgMfufqAAYpDsMGATHhlzX0m8JcNeTa7BcgyrEl4wb/RlBy9SpHdaPbRETt6liZ841y/D/w6aqW9Ya1J5KUfSS3YU+I4aABh5B/l17+thP9eVb8hPIXCk1Qu2BDNOUoO4ZcLfWduln8MW7HheX/Uu/Ilok3fDJGdYkE/o9ZItCvE44Dq108vWWPmd1a3CFx0QbJOVYyNhYeZFpaWuQMgYOwYRYl+rTJGnNTp/JS220M9dd6CUAg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=u8uz4XcyU/QnoJnJVkQ8jtu3W3GsKGxiOLPgDdsngU8=;
 b=kqGXG7K85x6cz77U2EzzedXuj/XdvlsiXRED7Dn+03xktsyGbXLbof3MFV154ehFZ0kKzjSLPmI1HE8avMqfUuv5/6ywJ0FS1u1rYQbQ9rOwFNUrW6uOCn0RB+SXVLBGoN+4qg7Y1L7jO4VvpTNyKBHDqeoMeCc4eiyRaCo5zEuYszqQ4fRN7AOgyalFl0/MCyxEnNiyY8ApaMN0Y9ikzZGcrbcZ6cwCCodbFvBka5eK7FljFFKISrXi38rDnNpERlzu7WSEBbFp7KwPTifDFVgMIl3+YEJEhd0iRIJt+w+BRq6r9Tsk5FWf9oRXwaWvCyew0M4ZHoNBn+McSGr4mA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=u8uz4XcyU/QnoJnJVkQ8jtu3W3GsKGxiOLPgDdsngU8=;
 b=SVsCCxkbUx1EufrqHSw2Rbj2L8rbJJXIlygphZm+t8StC15GY0IFrYh/1t/uHQ5OykYRMFzvEJzJFJbsWAciOGEWaYwJTDKObgygpu7galPiT6W74yfE7IsraIajSKscIc5ALZK10DXAnMJ01rZ8gXsWafR9rqkwIZ0pA09Bc4M=
Date: Mon, 14 Mar 2022 10:19:58 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: <202203011540.221FeR4f028103@nfbcal.org>
 
 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: 
X-ClientProxiedBy: FR3P281CA0024.DEUP281.PROD.OUTLOOK.COM
 (2603:10a6:d10:1c::15) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 1f63ba07-7697-4815-50ee-08da059bd157
X-MS-TrafficTypeDiagnostic: MWHPR03MB2493:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: MQD7ObEWl+nTGvUQGWU7J3UBiG2Ggp9KV803phLXYKoTAVkchwI22PcfQT/Biw/5O/MpdRTKpFsMl2blKdLOCs0VQ10DNyUWWTQJzPIPGqW95Hv99Qz4EsixsquHRsg5EXr/XYCJIMkP7jZGunde9MY/I7YDNEhLd73dEBMarPT/nha716NEUkrS2epWI2bZhvJYkVrocqzXZfbVHVWfyLrxK5vOyklChRqSTPxH5yxG8VtbfOczv1tigp+/+eRSsndOBsM5y1e41rVjV/zTGk3xDcxign5BIVWLHIWgsJT5234l5DaagD0fpzpo6vhNop4EyfdO/Q548IX/CTU8ENzLR/DWOw3hmKvvXZckp3J83EFB9z50N0MdWboHo/Ug4/eEbj9Jp/6m267o4EEwjDkpEAHb1RluSiZTICPJBxIDfvES3nry3lecnipDhXVA1dNFF4CF8VCXR0J3YdARaNG9EKMVAj4H/+UEsZ8iQfMxbIJBw9040WBfQMWMITq/b0t8vheFxM3YjkfLomRuukHo0VWNF3lpEppmSXGGLJsl/AzhDcEDH+Y49sKzoJUMkYMNcHNBZSV1i0VONuLo8AQKzbaTbPxJId1nZtYOoEB6blxvpVV4xd/1NH7bWdHn3KojDad2qF9GeaKDsL9YjA==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(4326008)(5660300002)(38100700002)(66556008)(508600001)(82960400001)(66476007)(6486002)(66946007)(8676002)(83380400001)(33716001)(2906002)(316002)(6916009)(8936002)(6506007)(186003)(26005)(86362001)(85182001)(6666004)(6512007)(9686003);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YnZFVmRQZlJWWkZZT2E4ZUc4V2d1ZVV6VnMySkZqbVRIS2NPMExra1JVL1Y3?=
 =?utf-8?B?dVJHTXhVTDZDc3RpdHZ4MDNUSGx0M1VLdGZhdTJXY3M0aTQ2MGR6U3h0TjBP?=
 =?utf-8?B?WGoxQnIzb2s2SExsMlM1K0lDc0JySytRWFIvdzdOelkwZ0JJNmlMQVcxMjNN?=
 =?utf-8?B?WTZKenpJd0RBQURVM0NCTUJSSTRLaENTbGlOUGpjdE1DVGczTTJHUGdyaGxM?=
 =?utf-8?B?OVFWeTB4Qzh3Wm9NWU1iNmhwSTUydDdDbk1QWkdpeGVWTllLTVNpQjBZZFBm?=
 =?utf-8?B?c1Y2OXgxdXFoVmJYM0RZSmNoNCtmajBkMmxZWlBMR1VDN2RTTEcrM0Jaampr?=
 =?utf-8?B?OG42d1oyYS9EaHNMOUNGdDljWmliY0tGQjhTUHA5eDMxMWNsMWorWWZaalJm?=
 =?utf-8?B?NmVFTU5NTTk3RVU2azk4UkdCZWZEc3Q2UUZUS0dUcWFKQ0ZxTGFLQnUvVnho?=
 =?utf-8?B?NjBHZUNCOUFNcFdYaWZuRnJHUkVXNnFrSlF4UUsxN1dIdk5ZTDIwNlZERHg1?=
 =?utf-8?B?dkMzaGdYekROcGw0WXI1Mi9CMkpiYWo2WWQ2dG81S09ZKzYrakFzNXRGV1dx?=
 =?utf-8?B?U2NsaW5kT1VPKzNsbDcvSGlSRTIrQ3EwQkJPc1dha1FVUElpbGsvQ1pXOVBH?=
 =?utf-8?B?NktoS0RJOTVnLzhmUWIydlNuSVRPOXlnbTVHMi9STDh4TjBldEJTUVpkZVRO?=
 =?utf-8?B?L090T0ZrNW5EdGdZWk1nS3RuMkMyV3hUYUhrWW1nRU5zUDloaHNiK01oZHFx?=
 =?utf-8?B?K09OTjFvOWZJRXMwdkF4RzZUSVhZZ2V3dXNMaEZzYjNOR1ZtN2JyOFliNS83?=
 =?utf-8?B?ZGdxN2NlMUxpTG5uaFZsb0pVZ1FFWmpLTmJoeFRUYUZyQVFKSTE4MS92dk53?=
 =?utf-8?B?UjJuQVV5cXROUmMwYmFRZmJVVUpiMXhZMDlDeTRnNDl0VEttTURvV2tZUHdX?=
 =?utf-8?B?QVptS1VMZlkyMWJkZ0FVd21Ga01aMFJTNWkzbEVBNFhxa2ovVDdwRW9NQ1c1?=
 =?utf-8?B?N2ZUb2d0Y2RSQXEvRDBoLzNjRTNzUldsdmJmZEJIVW5uNWRGUlNUbjZtYmdE?=
 =?utf-8?B?eHpvUWtyZlhiZ0lFMFBNempCZFpHUDVGWndtRE1zdWlZMFFpa2pkbHpla1V3?=
 =?utf-8?B?STVCYnd3USsxTTcwUE1uZlhjWm5Sczc5UmlWMnBENWhyTEp3MTZKTU4vd21E?=
 =?utf-8?B?MlN0V0dKeG5DUnV4UFcrdHVjUzYrZUtnQTJDNmpKUC9adWJMTjhBREhzcnBx?=
 =?utf-8?B?WHlGNUJnYUVhVFMrazZiSXhjYjFXUmNhMFk5RTdsTXhUQ1NrVDdNazUrQmRL?=
 =?utf-8?B?b2grNUtyMjJlSStiUFFMUVB3b2RCdGhZQUNmNENBOFhwUEZqUDJlY3VpWDY4?=
 =?utf-8?B?RU13VXFYL0J0MksxdnppT1JQN3kvZDRFc3VkRWFPTnpUZUtaTzVBWG9lV3pn?=
 =?utf-8?B?MnltWmszdEFCRzdqc0pqRUhFUHlMNkxNZDM3NXZpNldKWHUydktMeWxHWWp3?=
 =?utf-8?B?cDZ2VXo3ZVRJWXFJNDFRYngrWnFrZnZ5ZjI0V0hQTDlXbEovLy9iOGNoaVJQ?=
 =?utf-8?B?Y1B0cTdhbXBVUEpCWUVSTUtzcVB5UnQzMFRSaURvdkhBQU81Nks3WHVYb0hE?=
 =?utf-8?B?Y1hUa0JydmQzcnZqaVplTzY4MFJBRmZ2QjNQelc1V0hvYlExUTV3VzNwTS9q?=
 =?utf-8?B?SS9LR0JjSURrWSt0YnVZRGhnMVFiV2IzaS8ybFJOUVJ6bG5aY3d3K3Rqblo4?=
 =?utf-8?B?eXJjWmNCdkhVTmhFUWtJZ2VaWU5keTZwYXIramIzclROR1piTmNJd0VzNG9Q?=
 =?utf-8?B?cSt5T3k0OFFDR3NrSCtiY3h3OHAwY25tbm4zZ2N2RWhiVWZ1a2dIbStkcUx4?=
 =?utf-8?B?dDEyK0Jkd2o5bkVEYUp4d2tQdXZ6YW5ySjFxbWthZG9EWDRmSTF4eGdOTHlH?=
 =?utf-8?B?aWk4RVY1dU92ZkU0SW5UbGZJUWlpeW1VOGRQRFg0d1l3OXFJUU1TS2x6NkdY?=
 =?utf-8?B?OTZuQkNmOVJRPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1f63ba07-7697-4815-50ee-08da059bd157
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Mar 2022 09:20:01.8607
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Khs4nGc2zgyX5N0l2aEgrT3q0vrts8eo7l2/sDRxKKonwz5XVw6sTRz7HFNMudUuAihb0UKVjLdVHLQloDsIeQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB2493
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KH9x83W8Mz4Svn
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=EjZ6S9Zs;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=SVsCCxkb;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=0654505ba=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=0654505ba=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-0.99)[-0.991];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=0654505ba=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Mar 14, 2022 at 10:06:58AM +0200, Ze Dupsys wrote:
> I'd like to share more analysis on given problem. I do not know if this
> somehow helps or not, but i have noticed that across all my saved serial log
> outputs, panic messages follow after some of these lines.
> 
> ..
> (XEN) HVM d34v0 save: TSC_ADJUST
> (XEN) HVM d34v0 save: CPU_MSR
> (XEN) HVM34 restore: CPU 0
> xnb(xnb_detach:1330):
> xnb(xnb_detach:1339):
> .. => panic
> 
> 
> Most of panics are like this
> ..
> (XEN) HVM d26v0 save: TSC_ADJUST
> (XEN) HVM d26v0 save: CPU_MSR
> (XEN) HVM26 restore: CPU 0
> .. => panic
> 
> ..
> (XEN) HVM d42v0 save: TSC_ADJUST
> (XEN) HVM d42v0 save: CPU_MSR
> (XEN) HVM42 restore: CPU 0
> xnb(xnb_detach:1330):
> xnb(xnb_detach:1339):
> xnb(xnb_detach:1330):
> xnb(xnb_detach:1339):
> .. => panic
> 
> 
> This one i think had different stressing conditions than other's, but i
> don't remember
> ..
> (XEN) HVM d660v0 save: CPU_MSR
> (XEN) HVM660 restore: CPU 0
> (XEN) d659v0: upcall vector 93
> spin lock 0xffffffff81eaa780 (sched lock 1) held by 0xfffff8020152d000 (tid
> 100434) too long
> timeout stopping cpus
> panic: spin lock held too long

That one seems to be a watchdog panic, albeit it's quite likely cause
by an out of memory condition.

> .. => panic
> 
> 
> For serial output in middle when there are no crashes i have noticed that
> there are at least 2 different execution paths.
> 
> For most VM's boot flow continues with serial lines like these:
> ..
> (XEN) HVM1 restore: CPU 0
> xnb(xnb_probe:1123): Claiming device 0, xnb
> xnb(xnb_attach:1267): Attaching to backend/vif/1/0
> xnb(xnb_frontend_changed:1391): frontend_state=Initialising,
> xnb_state=InitWait
> (d1) HVM Loader
> ..
> 
> For some though, there are lines like these, but they still boot, it just
> seemed that these lines might be a possible continuation for "unsuccessful
> panic".
> ..
> (XEN) HVM3 restore: CPU 0
> xnb(xnb_detach:1330):
> xnb(xnb_detach:1339):
> xnb(xnb_detach:1330):
> xnb(xnb_detach:1339):
> xnb(xnb_probe:1123): Claiming device 0, xnb
> xnb(xnb_attach:1267): Attaching to backend/vif/3/0
> xnb(xnb_frontend_changed:1391): frontend_state=Initialising,
> xnb_state=InitWait
> (d3) HVM Loader
> ..
> 
> Why those lines starting "xnb(xnb_detach:1330):" do not have any message?
> Could it be that there is a bad pointer to message buffer that can not be
> printed? And then sometimes panic happens because access goes out of allowed
> memory region?

Some messages in netback are just "\n", likely leftovers from debug.

Can you try to stress the system again but this time with guests not
having any network interfaces? (so that netback doesn't get used in
dom0).

Then if you could rebuild the FreeBSD dom0 kernel with the above patch
we might be able to get a bit more of info about blkback shutdown.

Thanks, Roger.

---8<---
diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 792933402c93..84ebb9068881 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -125,7 +125,7 @@ __FBSDID("$FreeBSD$");
 /**
  * \brief Define to enable rudimentary request logging to the console.
  */
-#undef XBB_DEBUG
+#define XBB_DEBUG 1
 
 /*---------------------------------- Macros ----------------------------------*/
 /**


From nobody Fri Mar 18 15:24:17 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9D0191A2D0F8
	for ; Fri, 18 Mar 2022 15:26:08 +0000 (UTC)
	(envelope-from prvs=069af078f=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KKnsW2JTlz3s1w
	for ; Fri, 18 Mar 2022 15:26:07 +0000 (UTC)
	(envelope-from prvs=069af078f=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1647617167;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=3Y6LlHolQrMOoQRDnJ/m/AEcvG9o8xdx9gHCumvwup4=;
  b=ACFxs9x+UrWdliUxxJp/4v6+hM7cm2PIuaFn5L8y3C+VcXptrIssmUW9
   nzRWrigFDu5uNDaiuFiFs1IWoApo7PntVZzP4DHtLooKf7XC48keB71eg
   y9KODYsyX7TW1TQuwGr54ZKQYWd9tKxiaQSXpr7f8LW/tmUkewDSHaZ3o
   8=;
X-SBRS: 5.1
X-MesageID: 66612315
X-Ironport-Server: esa3.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:v/S27a6sLUd3nA3TornD4wxRtCnHchMFZxGqfqrLsTDasY5as4F+v
 jMcXG2FbqqCMDP8co1xYNvlp0oAusCHm99mSlY6qS43Hi5G8cbLO4+Ufxz6V8+wwmwvb67FA
 +E2MISowBUcFyeEzvuVGuG96yE6j8lkf5KkYAL+EnkZqTRMFWFw0XqPp8Zj2tQy2YPgU1vX0
 T/Pi5a31GGNimYc3l08s8pvmDs31BglkGpF1rCWTakjUG72zxH5PrpGTU2CByKQrr1vNvy7X
 47+IISRpQs1yfuP5uSNyd4XemVSKlLb0JPnZnB+A8BOiTAazsA+PzpS2FPxpi67hh3Q9+2dx
 umhurTtYwluIKnwmt4UTiQGCDl8H6Rn6LPIdC3XXcy7lyUqclPpyvRqSko3IZcZ6qB8BmQmG
 f4wcW5XKErZ3qTvnez9GrIEascLdaEHOKsFvX5t13fBBOsOSpHfWaTao9Rf2V/cg+gQQ6iHP
 ZFANFKDajz6WyZtHlcuI6se39qahkb6XT1Dsl+88P9fD2/7k1UqjemF3MDuUtaXRd9SgW6eq
 GjN9mC/BQsVXPSR0zeI+Gmgw//Cmwv1Qo8fHqa0sPlwjzW71GEfFQUNRECTrvywi0r4UNVaQ
 3H44QJ38/J0rhbyCICgAVvo+xZooyLwRfINOeN99gWxyJDd/jvHHXAGbDdgeNAp4ZpeqSMR6
 neFmNbgBDpKubKTSG6A+rr8kQ5eKRT5PkdZO3ZaEFJtD83L5dhq00mRFooL/Lud1IWdJN3m/
 9ydQMHSbZ03hNVD6ai09Euvb9mE9smQFV5dCuk6swuYAuJFiGyNOtTABbvzt68owGOlor+p5
 iBsdy+2trxmMH11vHbRKNjh5Znwjxp/DBXSgER0A74q/Cm39niocOh4uW8iexo3a59fIWKxO
 Sc/XD+9ArcJYBNGioctPuqM5zkCl/C8RbwJqNiJBjaxXnSBXFDep3w/DaJh92vsjFItgckC1
 WSzKq6R4YIhIf0/llKeHr5FuZdyn3xW7T6DFPjTkkX8uZLDNSH9dFvwGAbXBgzPxPjf+1u9H
 hc2H5bi9iizp8WlO3iJq9FPdQ5WRZX5bLivw/Fqmie4ClMOMEkqCuPLwKNnfIpgnq9PkfzP8
 G37UUhdoGcTT1WWdW1mtlgLhGvTYKtC
IronPort-HdrOrdr: A9a23:pQAVhakivGLoMR6kDZeEuNd/gkHpDfPIimdD5ihNYBxZY6Wkfp
 +V8sjzhCWatN9OYh0dcLC7WJVpQRvnhPhICK0qTMqftW7dyReVxeBZnPHfKljbehEWmdQtsJ
 uIH5IObOEYSGIK8voSgzPIY+rIouP3iJxA7N22pxwGIHAIGsNdBkVCe32m+yVNNXh77PECZe
 OhD6R81l2dkSN9VLXEOpBJZZmJm/T70LbdJTIWDR8u7weDyRuu9b7BChCdmjMTSSlGz7sO+X
 XM11WR3NTvj9iLjjvnk0PD5ZVfn9XsjvNFGcy3k8AQbhHhkByhaohNU6CL+Bo1vOaswlA3l8
 SkmWZrA+1Dr1fqOk2lqxrk3AftlB4o9n/Z0FedxUDupMToLQhKffZptMZ8SF/0+kAgtNZz3O
 ZgxGSCradaChvGgWDU+8XIfwsCrDv7nVMS1cooy1BPW4oXb7Fc6aYF+llOLZsGFCXmrKg6De
 hVCt3G7vo+SyLUU5nghBgu/DWQZAVxIv/fKXJy+PB9kgIm0EyR9nFohfD2xRw7hdcAo5ot3Z
 WyDk0nrsALciYsV9MOOA4we7rFNoXze2O4DIuzGyWvKEhVAQOEl3bIiI9FkN1CPqZ4i6cPpA
 ==
X-IronPort-AV: E=Sophos;i="5.90,192,1643691600"; 
   d="scan'208";a="66612315"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=EbRpKpWkPWhFIkbA48zxO2V6Q+WTOg43OwCiog3MwqRGH5UnNp8VA3cS+SVEJ9YVJjN8xVmVgvG4RfFlQ2T4ATEOkWZTBr3jKdiOOKfIsgD2pbnwB9UtFomT+xRDeIX6zMpwMwG/G//ArdZ1wewM6ax7JqZ+JNHN8nvKgw7w9wEyf77ZNBa8sXd9RhGuJcWx1a13cPEYyE48gm/MVL9UKKbTYdtDOTs2gD+/sDhwxFIeFOKPsukUfsNg61kAYsVB9w/W2QxUxQJO6nfTEU9OfNhlKtacDq1RM/+EejaSj6eqffLAqqBYYoS0Tv6+LkFLenaKUuLNq7/vJ9aa2LoWzA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=/TxqRAZaqlJdLC4rO3xuSnDU23bR4XzJveAV8wYtAm4=;
 b=T+0RNPEXt3OGldVTSUbJ/FFBBJxir3zznLU6HPR6P0tNaQNFVKPOYv3vZ4+9juZlvMSv3g5ZXpvVc5wAQa50pT2Rfhd2k9ab23udB0Rjky3Jifxw+8FoLhSSuRnxclo++ljLD/5WboqoPzBs70MN4uZxjveFbLm4CR2bVe1kzBM/JFyqz3HVHtgqRqNeZWh8EVSS/gKwy4znvwfpgecebMectB6HoObPmJN8Oo3qd+VM/zad6dKS5bc8XNlP7DQSP9JoByar6DxACBtVNC1F7RxMC9aCFq2JJwq2rVluHTw/zSYUyfSSWmz6eB5o7KWLLFFZhVWwkLm3Cgub66sQAg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=/TxqRAZaqlJdLC4rO3xuSnDU23bR4XzJveAV8wYtAm4=;
 b=XexZpUVpiUtz0Sg7h/jTpX6H8z29qcbCUGo4100o99OWbD9ZOU42xw0RfJZzTFs9llrmMPxfpMaMjL4h5PMBuNRgwQusAabFlW5jv5KW8xcn/Ke1hL8jLmycxXAxAc308kjOSQ2GIovRsjfGNJ/aRMh78PI68JTgG0nFZjfhr/Q=
Date: Fri, 18 Mar 2022 16:24:17 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO2P265CA0492.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:13a::17) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: a092eec2-0b86-4ffb-4d93-08da08f36169
X-MS-TrafficTypeDiagnostic: SN6PR03MB4368:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: Pa7ir0nX9RP60Nbj786KI5yUb9UptW/ElFjgMLXVegIAcc8PlbpMq4w0nt4IskpqPNly3bUcZSYWCCdV63RNeOjjYmBIuzEWhUDtkfG6Mjgj/xJkEyM6TJJOiXqTc48fFnZ8xWOQv/Kp5z5ema5WzEnB3DAdxIX5ErNDaEB3CZQQQN3yU9lVbSWchhi9hgh65j7RlWjqRwHM14cNKp23EM6JgE8QRtOjkS107g6lkYzM1qmqr1rTrv4OXDy8G4MRQjDsmhYYRlix4fNhwaNpCHocjNKWZRQ4rQkzik0R6kngveio+FfIWpv3ZyIyCzJl1sKP7sW61HxZWmN29sfiRlP/tZp7+SSSFl7VNXJslxojBEr33RpPKII2jdYZm3hTiBX0144a42RWonqZZ/Fil94XvKtcYTV2GuYkEN61en1L+iT3qlVOsHqyVDVUCpuc2Yp8JbEERiXnaJ642ekht2Etk7IKTTdWyp18nCzDaVjf1bi9rBgKdhyG4QT+z89twTWjqE4rUwvjhRaTT2zq7Aolsq1PVNvBvjnghln+jznL2t35WpA+1y94JcLpC3e1NVL4gWoUzNPL7AXiK4vB936QKsO10j+m4kCFppEwBaULdrOyhyostrynu6W42YQ2ny6jjYCP+jGSmdg5Ucpr6w+SRdCUjEB0SxMY85XE7wpNNUv/Ggy4U4zQoMpTU3wnz5Lnds1GvMR2IsValn7ogg==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(38100700002)(4326008)(66476007)(66946007)(66556008)(82960400001)(6916009)(8676002)(316002)(2906002)(33716001)(9686003)(8936002)(6512007)(83380400001)(5660300002)(26005)(186003)(86362001)(6666004)(6486002)(508600001)(85182001)(6506007)(67856001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?LzU2aTkrZXJycTNsMVlEVmJPbHZ5SFNRdW1HYWI4L1Jma1dHcmV6N1JPSEJl?=
 =?utf-8?B?MlIzUXZMTk4zdEJuUE51aGtrSDZoRk90eUFlbENLYzBNVFZ4T3dVbGxCOFl1?=
 =?utf-8?B?L0FFVWp4R3pUM3E0eXJzTk5TeitzN2ZIQkdmYjl1RDkwWGJiNXlzYnd0WDFJ?=
 =?utf-8?B?Q1A0L3FFK1paSDRVR2pSbEZaOHhpQWsxVjRoSjFOSWZBd01WOVRRNWlXcmh2?=
 =?utf-8?B?bmdiOU5nNVpHV2EzMVNobTg5YkpZOHVTcWFmb1ZxOGxvcDlFeW5BMG1iSEhW?=
 =?utf-8?B?WTJ3b0ZteWUzdHJYbW9sbEw1bVNBWEhMaXQ2OC9zS05sOHRwMHhTallGOWRk?=
 =?utf-8?B?SzIzTWhIY1h4LzFBQ21DTXFVTTJRSTUxSmdRWE9ObGNManB6MW5RWW9teU9Z?=
 =?utf-8?B?ODNrVmlKWUZKTjh4bHROcnkzUXpuRGpDZVFBV2czQWtwVHBYWUtNcGhsNXJh?=
 =?utf-8?B?aUdpR0tZdWU3dE1TZVQ5N2tWemVadHloWHlveTFPWW54VC96OTU3YVEveUF1?=
 =?utf-8?B?T2VWcVBSejVsRDkvTTZIZm4zSEJjK3pnSXVyS1Jqek4xalhGam5MaUpKS0Q3?=
 =?utf-8?B?UFRDVnBTVm1BdmNYei84M0JkNkdCVXFkb1BvWXBZYjFEQk5oZlUxUGpNcjlz?=
 =?utf-8?B?QzlxUkMydzNGbG0zMmZXOTVTWmRpeTJlQkNubUxnNjV3VnJTOGx6dEdqdDRm?=
 =?utf-8?B?SWpIUFhpUTJuTmt3eS9JcXFXZkMvZEc4Wjk4cktYWVZyN04rN1VCT01Fc1dW?=
 =?utf-8?B?YXg0NS9DV3RGY1dtcU8yNDJGOW8zVEw1OEZHVzFiWXVGR1kzZHgyeXFpa0ZK?=
 =?utf-8?B?eDhaWWVkZGhwSnZGMThWOTlPNC9ZUHZCdmRablNyRTQ2S2dQclFJcGVzSFpm?=
 =?utf-8?B?SkJoZXlBUzlOcmh6MytUZEF6MXJZK2xJOUFZc0wxdzhJV0F2cmxRd3FhWkp1?=
 =?utf-8?B?VCs1NGN6MUJGcmFvdkluZmNQY3JodUtBQXhsM0FrQklidE52LzJZTS9RT1BS?=
 =?utf-8?B?d01uU0dwTzBtdFhYcmsxU3JYNmFHb0NaZnhNaXRWR1kvbmlTU2I4dnV6aHp6?=
 =?utf-8?B?WkFXL3dxdWNBZkpPMU9PVlQ4dXd6VkVLZ1M1L3J6SWdCcXRPUVgvWmxKT2xX?=
 =?utf-8?B?QUNqdmhGOSt6aSsrQ3VGdFJaemxVYzN2eFJzbkNpTjlubTg2UWNhWTRtenpm?=
 =?utf-8?B?UnE0Q0RTVmJObUF2QU54MW5sRnMydWhwRDRZdXBqTHNaa0w2S1o5Z01vaHJJ?=
 =?utf-8?B?TTg5ZFcrSG5JWUxDejEzb1pDQU9DLzNuNmlpS3JhTW12emE1dkhTUURvbXFn?=
 =?utf-8?B?VXFUdzA4bGllYzUvVXBNTXZKdUtFTDJxeTBtKzhxTmFNUjhZZ3FCNUZTUGpi?=
 =?utf-8?B?M3JaMnhsNzdlMWYyUUMxcjNMSTNaN3FaTDlWZUxiRXRmNmZOZFZ0MnBFQkho?=
 =?utf-8?B?OFh2NnpCUXRPZGtaVEpzS3JSbDI4S1grV3pqalAwWGd6TWg4SE5VWG1LL2Fn?=
 =?utf-8?B?QjRaR1NpTlNUZktjYm1UMStFNDNCalB0Uk5XK1JCYU1iUEoycGN2aHRLSHhM?=
 =?utf-8?B?ZWFSU0cvbGhUNXFTeTNVMG1pR3RCQlgyK3VnbWVTRUhGcWtlMmRDSklaMTcz?=
 =?utf-8?B?THBuajNtNXdEMEE5Sm5JWXdCRU8yaUhtektwWWtGMTVJVWwrOEpQbmg4WXRY?=
 =?utf-8?B?M2pkL1BXYTRKQThOMSswV1JtcFhtM2RDejBuYjNBeWc0R1NRNzV4cFJNNW9p?=
 =?utf-8?B?RWNodzhUbFZXZjMyMVlzaWpPMWpaRWM0ZC9CUHY2RUVqazZDMzR6OWRzMFFX?=
 =?utf-8?B?N0I4cW50T1p1SFVCV1ZhYkhTdkNsZ1JCUytldzRaZnpNWWZkZldsMEdLdUJG?=
 =?utf-8?B?Vk1mY3dNL0JUb2xRYXJnOG5FTGo0NjkrZ2szTEVOUkF4SjhLNjFuWW9GUXJr?=
 =?utf-8?Q?X4Pk9vXQhIJLwbLU1BaXpWwZE2GdDPEl?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a092eec2-0b86-4ffb-4d93-08da08f36169
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Mar 2022 15:24:23.2755
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5QMaQk3fEj9YX2Noh4M3uGZvDe9oabaGiRWwkus7QtdxsPLBrzC28hmCwzPFUX9oEhwU7AOO1rDK/1uVtVnkRw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR03MB4368
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KKnsW2JTlz3s1w
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=ACFxs9x+;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=XexZpUVp;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=069af078f=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=069af078f=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.62 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=069af078f=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-0.98)[-0.980];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Tue, Mar 15, 2022 at 08:51:57AM +0200, Ze Dupsys wrote:
> On 2022.03.14. 11:19, Roger Pau Monné wrote:
> > On Mon, Mar 14, 2022 at 10:06:58AM +0200, Ze Dupsys wrote:
> > > ..
> > > 
> > > Why those lines starting "xnb(xnb_detach:1330):" do not have any message?
> > > Could it be that there is a bad pointer to message buffer that can not be
> > > printed? And then sometimes panic happens because access goes out of allowed
> > > memory region?
> > Some messages in netback are just "\n", likely leftovers from debug.
> Okay, found the lines, it is as you say. So this will not be an easy one.
> 
> 
> > Can you try to stress the system again but this time with guests not
> > having any network interfaces? (so that netback doesn't get used in
> > dom0).
> I'll try to come up with something. At the moment all commands to VMs are
> given through ssh.
> 
> 
> > Then if you could rebuild the FreeBSD dom0 kernel with the above patch
> > we might be able to get a bit more of info about blkback shutdown.
> I rebuilt 13.1 STABLE, with commenting out #undef and adding #define, thus
> line number will differ by single line. For this test i did not remove
> network interfaces, and did add DPRINTF messages to xnb_detach function as
> well, since i hoped to maybe catch something there, by printing pointers. I
> somewhat did not like that xnb_detach does not check for NULL return from
> device_get_softc, nor for device_t argument, so i though, maybe those
> crashes are something related to that. But i guess this will not be so easy,
> and maybe it is safe to assume that "device_t dev" is always valid in that
> context.
> 
> So i ran stress test, system did not crash as it happens often when more
> debugging info is printed, characteristics change. But it did leak sysctl
> xbbd variables. I'll attach all collected log files. sysctl and xl list
> commands differ in timing a little bit. xl list _02 is when all VMs are
> turned off. Sysctl only has keys without values, not to trigger xnb tests
> while reading all values.

So I've been staring at this for a while, and I'm not yet sure I
figured out exactly what's going on, but can you give a try to the
patch below?

Thanks, Roger.
---8<---
diff --git a/sys/xen/xenbus/xenbusb.c b/sys/xen/xenbus/xenbusb.c
index e026f8203ea1..a8b75f46b9cc 100644
--- a/sys/xen/xenbus/xenbusb.c
+++ b/sys/xen/xenbus/xenbusb.c
@@ -254,7 +254,7 @@ xenbusb_delete_child(device_t dev, device_t child)
 static void
 xenbusb_verify_device(device_t dev, device_t child)
 {
-	if (xs_exists(XST_NIL, xenbus_get_node(child), "") == 0) {
+	if (xs_exists(XST_NIL, xenbus_get_node(child), "state") == 0) {
 		/*
 		 * Device tree has been removed from Xenbus.
 		 * Tear down the device.


From nobody Mon Mar 21 11:14:16 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 766341A18830
	for ; Mon, 21 Mar 2022 11:14:45 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KMX842tFrz3PjB
	for ; Mon, 21 Mar 2022 11:14:44 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1647861284;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=p4aQvlkTr+nIOt+cUG3THabc86q8Ri4Mtc0inTZ+1Vc=;
  b=JnUIBQCvIEwta7YRbAgnFHPTCOqHnf5pWfP/CBZ+qJBxLQcmb/TyJ4/z
   b7D/NYEB5++95OzmLfJkUr45ZipG6kxZ0BVtttAVNMIW3RrD6E3c/Trcl
   zBD7tzNyPSbj9Pc0xVi/0azEd0Tfll28AGPppbRFFiPqSXSlRctcaqxHq
   U=;
X-SBRS: 5.1
X-MesageID: 66209058
X-Ironport-Server: esa5.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:TId8eqJ896EVdTD7FE+R05UlxSXFcZb7ZxGr2PjKsXjdYENS3jcDz
 mocXGvQMvncMTHyL911Od618hxSv5GDyYRlHAtlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s
 q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh2dQx2YHR7z6l4
 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc
 s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4
 NdTuMO0ciAoB7Tjs/8dCylkHxlkbKITrdcrIVDn2SCS50jPcn+qyPRyFkAme4Yf/46bA0kXq
 6ZecmpUKEne2aTmm9pXScE17ignBNPsM44F/Glp0BnSDOo8QICFSKLPjTNd9Glu3JgfRauOD
 yYfQRNpYRSab0VsBgsOGY4YsOis3HDjSSIN/Tp5ooJoujOOnWSdyoPFOsfVYNuVbcNblE2Ro
 iTB5WuRKh8AOdGV0jbA4n+qruHVkC7xQ45UE6e3ntZ4jVGO3ncCEzUZUFK6pb+yjUvWZj5EA
 xVKoGx09/F0rRH1CImmN/GlnJKalhwCVoVUKMo70Rii4bHUuzalClAeHzEUPbTKq/QKbTAt0
 1aImfbgCjpurKCZRBqhy1uEkd+hEXNLdDFfPEfoWSNAuoC++99r0nojW/45SMaIYsvJ9SYcK
 txghAw3nP0tgMECzM1XFniX0mv39vAlouPYjzg7v15JDCskP+ZJhKTysDA3CMqsy67DEjG8U
 IAswZT20Qz3JcjleNaxaOsMBqq1wP2OLSfRh1Vid7F4qWjyqyf7JNkAuWovTKuMDirjUWW0C
 HI/RCsLvMMDVJdURfEfj32N5zQCkvG7SIWNugH8ZdtSeJlhHDJrDwk1DXN8K1vFyRB2+YlmY
 M/zWZ/1UR4yVPQ2pBLrFrx1+eJ6mUgDKZb7GMmTI+KPiuHFOhZ4iN4tbTOzUwzOxPjd8VuPr
 IoHaZfiJtc2eLSWXxQ7OLU7dDgiBXM6GYr3u4pQcOuCKRBhA2YvF7naxrZJRmCvt/49ejvgl
 p1lZnJl9Q==
IronPort-HdrOrdr: A9a23:ysuiuKEVj1O08KZ0pLqFDJHXdLJyesId70hD6qkvc3Jom52j+P
 xGws526faVslYssHFJo6HnBEClewKgyXcT2/hsAV7CZnidhILMFuBfBOTZsljd8kHFh4pgPO
 JbAtdD4b7LfChHZKTBkXGF+r8bqbHtms3Y5pa9854ud3AQV0gJ1XYJNu/xKDwOeOApP+tfKH
 LKjfA32QZINE5nJfiTNz0gZazuttfLnJXpbVovAAMm0hCHiXeN5KThGxaV8x8CW3cXqI1SvF
 Ttokjc3OGOovu7whjT2yv66IlXosLozp9mCNaXgsYYBz3wgkKDZZhnWZeFoDcpydvfomoCoZ
 3pmVMNLs5z43TeciWcpgbs4RDp1HIU53rr2Taj8A3eiP28YAh/J9tKhIpffBecwVEnpstA3K
 VC2H/cn4ZLDDvb9R6NqeTgZlVPrA6ZsHAimekcgzh0So0FcoJcqoQZ4Qd8DIoAJiTn84oqed
 MeQ/003MwmMW9yUkqp/VWGmLeXLzYO91a9MwQ/U/WuonlrdCsT9Tpc+CQd9k1wg67VBaM0o9
 gsCZ4Y5o2mfvVmHp6VO91xNPdfKla9CC4kY1jiaWgOKsk8SgbwQtjMkfII2N0=
X-IronPort-AV: E=Sophos;i="5.90,198,1643691600"; 
   d="scan'208";a="66209058"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=cHNZbSQisW4bphHagICYUjpYJmONGglDbJa5rI+1d5SrRAcO3G+NDGYqROCW+pJGHX6A+Eo/2718A8irQ7wvymXntN/v+s7zDbNZ0fxnDhKmOJOBV2piDma/PSNK1iKlP832GcU76Wy9yLfwyO8lEP8SKF1axHjkai5h7WWWYoN8PzSvsDejo357hN1IUB8pY3FrRW20C6vuzVRSKeH5TPyuJgeQJA6nRHnnB2i9+Ooen6K18guZ7KiSTWhRApBKmVYrEkO3B4hhS1URDFQAXp9S+v7SJ07RfC3wDmcGXQSxDaypqeUjt4ahoLWZJw29fEhF1xY0CP93hb5GZzByGQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=s7kjiexqZmJiKNxKNU06qrepSbbv/BjfVNHXFADYoFA=;
 b=Mn5Kaw1HdkwH2YNzoxSs6lCnHmtV/LaD1Hkr6CeeeELTpyIrvD0WRQP1fgmmcwezh/Bj2wST8ABzGU5pvx3SqNmbLE774NO77ZORaN6hZoV5Oqs9vH+WKr2T+O7TmSStaHvlyMRbTF0E4diNm8Shcs+tGb6fRMAvju3zTyD76AnCqUlt4TgoU8XO1t291224CXkypX6Sfc1TNSvxOf8r99hHfPKHuNhvyywK5Oa025ULUSaADvt2sjlbfmp/a6R7TXimPWrdzehTNBdLsesEp8xLYUV5CNGpd5qALnhzZY4BpO3+QAcJugbSaSK2/ogdLzuey6XAEC4Ab0p5FTfvjg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=s7kjiexqZmJiKNxKNU06qrepSbbv/BjfVNHXFADYoFA=;
 b=ZKQvlq1lEwPx4bw3GJjiI5ilnNt0RsiNR0wp+7yf4FA5lKZVEsxkfIUtR+DXS3gvX1vP98gDhW2Vq6p9Rb1PrJgErXU+ccbHeFnaU6fX+nyEenLL1AY8AnT7VF18mH+HK2dIM2K2KYD0hNqRiUXbkU4k9vUvTVPWeA6N+hJsXig=
Date: Mon, 21 Mar 2022 12:14:16 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
 
 
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO4P123CA0501.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:1ab::20) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 22dd3f03-0dfd-4412-4816-08da0b2bf4ad
X-MS-TrafficTypeDiagnostic: BY5PR03MB5030:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: a6BU6PktACtwzFalRu9+DcIyK/6F8uwncoYc+Yid0u+qjXuB+gKmgEGTtygikRIQ4VmlP59Rsd4+0uzDS973tIlAIVsdglSI5gtL54FSW6q2koy1RAsEPq9/4v6kJg2Da61HGqEgEbZsVXMPuA/E4oaZw5z6fuscdyby/FilW7h7v0jO+ZNx6eW1c6FJJBszlfgnpww20kRnw7roshewVR9KUSb1+i56JR9i6mjhj4Bmu2dQzomr3Jg7qtGeXAzUBTEoXFD2d8A25gNyeH2lrENNBRviN3vvDtht3OktEosiqCh3OdEl+GV/4K2nYOHpLu+yKJzlUFQompK3u7vVwc2fhuC6NX+ZysXxxr+eYNy4xIesnVojUktTHePRF5VpcrfEAut9unRNc+fulFU7ma9EeCgxTvmhqBmVNMJW9stc7yOJtRTxm0SVfDHm9T/c0BHXdBwTzqGv1HqxaVrHWDAijj1POq5EteT2JJW7XJvWqjNDxYLsTcGkiYcqRfUULXce8CM5dw4rzYCiwu1SLicCmmz2mNXgYCMPA2X3klS9jki6v34Br5gGv6E/g2Mure7jQxfb4zD/j+f4P5I5xcmhg2qPSrLfIfBeF+V43sNUiYSg7BlS2lUEmbDebSRY8nMm1LVi/YA/b2YQSY8Fzw==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(186003)(26005)(83380400001)(33716001)(316002)(66556008)(4326008)(8676002)(6916009)(66946007)(6486002)(66476007)(85182001)(86362001)(6666004)(508600001)(6512007)(2906002)(8936002)(38100700002)(82960400001)(5660300002)(9686003)(6506007);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WFR1OElXZGpJZmpJMTVlaVRIdjhnY3M1QjBrd2tZOXA5bTNEWWYzbTdkb3U2?=
 =?utf-8?B?cmdIamdVbTRzT0E5V2d6bnN6SVAvVW9tUGR3U2UxYm52QzRuMHduQ2E4MVJt?=
 =?utf-8?B?dG44Q2xjaDhQVlEwaWZtVndCSDRUKzA3MVhCa25uc1dmQ1VXanhKcWdTY3gw?=
 =?utf-8?B?LzZXOW1JNm5lK3VjUVFyTWQzSC95SjAzSnQ3ajNYdVZjNFA1cUVJbmwxdXhL?=
 =?utf-8?B?U0xzcUd2Z1RRK0ZzZk0yZGJZQ2gxQlhZd3NKcHJlMGlhdzhZMThwdHBPaHh1?=
 =?utf-8?B?ZTZLTlhidk9CNjBVeHpoQldnWUlhL3lraDVuTWZZd3p1UkpoYWZWdE1JeDhj?=
 =?utf-8?B?Vzd2Y0JUay9QWEZGdEVXbmxWUHVQS2E3S0YvR0VWSjRKMTBJUUEwQ3dyeTY2?=
 =?utf-8?B?RkdIQWhDMEEvekg1MkZlRGd0SGk0SndSWDNsa3AyTmdZMTczcU4xOEl5NEMz?=
 =?utf-8?B?OWJ2NDRLMHVsRWszbjdiSGtFWXdtNXhxZHpCNDM0bkl2c3JrUnAzODJJK25w?=
 =?utf-8?B?U1B6U21yMFJQWC85azdMZzJCQkl2M3R0ZUJ0aEI2c0tPSlpuSk1JUWRpa0J2?=
 =?utf-8?B?Wm90STZuNTAxWmxRTUhNd1RXSHVnT05yYVpKTEZrV0JHSng1TGlaSXdVSllq?=
 =?utf-8?B?TjB2bkViYzgxZFVEZzFwelh1dWlYdnBaZ0x2V2VScWo0MkJyWUtEdGdXOTNx?=
 =?utf-8?B?S1N2L2JyRVNTVVRYMEF5dERPYkVJcTB3RFpOVW92RCt2cnZNLzg1OWJRb1M4?=
 =?utf-8?B?TlorZzBWa3h4V21FckNJL21BNldvZFE3cnpuY3JEYTUweWlPZnFaZ0FiQWV1?=
 =?utf-8?B?N2twTitKOVZhcWNLeDRhb1hpZXEwY21wbVdBT1NjbnZXLzNuY0RURVpmMm5q?=
 =?utf-8?B?TnF5RGpTV2pvekxQcXZjWGR5Z1JIRG55Nk0wNWhNU2p6UGxqbm5qK0UxVVRZ?=
 =?utf-8?B?QWt4V3RhYjFYRmNESHF1VDhLK01HYnh3T1I2dnAwMUFJcENZZzFuSmUydk13?=
 =?utf-8?B?cU91V2hoeWoyR3cyYVdWL0xtM3E0OUJDT0tSQzJQOVpqVkdmMTBhQWttYVZ5?=
 =?utf-8?B?ZVgyOHpKTE9rclJ1QkMxdnozZDZBb3Y0dTJEQTJ0MlpYTFB6TXpNUGJTeVFZ?=
 =?utf-8?B?Wm50OXBIR1phbFhzWUFZMXJpcEtqbVBjaTlXZXQ2bitOMUVUMlhBWmRKSm91?=
 =?utf-8?B?Nk9qY0RCY09RaXRqQXpMTHljOGJJV09BNEtpbVp1OVpiOUpMMHNzMEs1bU1T?=
 =?utf-8?B?R3AyYjNxRlNuWGdUdFQ5NGU1K0hQNkNzaTl5Nmw4RE42MEM0cE9NUTdHSVg0?=
 =?utf-8?B?MkM1dWVjZUdGNEorc3R4bXJpMFFGbWJ4SEVqaXB1ZVNtWnV0NTViYnUwVUs4?=
 =?utf-8?B?V1IzWVY2VmdrMXNtRHlISHdJeEhkZmw0UGg5aElRRWVPU0pxanoxbTI1RlNp?=
 =?utf-8?B?ME5PL2g2RTRSekE3dU1QT3h0UitSendNaW9GaFR3UldZUDJJNklJMVY0aTJ6?=
 =?utf-8?B?K3h2Nng2OUpHbVhNVG92MDVlRDdxNkY4WnlWNWFtQ216eU84cEtZRnptTUt0?=
 =?utf-8?B?aXlaYjVJNjNWYXhhWEM1YVNWZlFWc24rMUtLd25BbERUWFFJUmNOcG5rKyta?=
 =?utf-8?B?a1dxQnkyb24rRTdJNkZ3ay93Z05wOGREUU8rdU1GSzFaYytCM1ZnOHFiZHVt?=
 =?utf-8?B?b1V6ZFRvWVZXcm9Ic2o0alM2ZjlINzRtOWpWcnoxTWNOYXdycnNQVlZ1aVhT?=
 =?utf-8?B?eHFSMDE0eFdIb0RiWUh3djdMNFZsMkNrcSs2M1pyVW8yOHM1RGlNeitteXZv?=
 =?utf-8?B?eGJLTmpIZzdSRGNhZ2lHRUxtcnNzdVRrWVBVVFJ2MHd6azMvWWNYNkVJcmtu?=
 =?utf-8?B?RXp0MVprbGlNcGc4Wk5obWtPSWlUeGZTeG84enk2VFY2Y1BOZlplZHdLUU95?=
 =?utf-8?B?WXoxSzNHTm04cElWRDZ6bVNLNVoyTkhIQW9iOGJRU1pIR3A1NjJEN3FEMXpI?=
 =?utf-8?B?ODFkLzJzUk5RPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 22dd3f03-0dfd-4412-4816-08da0b2bf4ad
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2022 11:14:24.4716
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CX8JX1NFDYCOxJTv68F5UEtXQGjJ0Qj+suoRrraiOfOv6+N0A3EsCpYzj/Xc0bMZsNBokr0epz0KouWawQk9ag==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR03MB5030
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KMX842tFrz3PjB
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=JnUIBQCv;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=ZKQvlq1l;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=07205606a=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=07205606a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-3.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07205606a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(1.00)[1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Fri, Mar 18, 2022 at 08:48:08PM +0200, Ze Dupsys wrote:
> On 2022.03.18. 17:24, Roger Pau Monné wrote:
> > On Tue, Mar 15, 2022 at 08:51:57AM +0200, Ze Dupsys wrote:
> > > On 2022.03.14. 11:19, Roger Pau Monné wrote:
> > > > On Mon, Mar 14, 2022 at 10:06:58AM +0200, Ze Dupsys wrote:
> > > > > ..
> > > > > 
> > > > > Why those lines starting "xnb(xnb_detach:1330):" do not have any message?
> > > > > Could it be that there is a bad pointer to message buffer that can not be
> > > > > printed? And then sometimes panic happens because access goes out of allowed
> > > > > memory region?
> > > > Some messages in netback are just "\n", likely leftovers from debug.
> > > Okay, found the lines, it is as you say. So this will not be an easy one.
> > > 
> > > 
> > > > Can you try to stress the system again but this time with guests not
> > > > having any network interfaces? (so that netback doesn't get used in
> > > > dom0).
> > > I'll try to come up with something. At the moment all commands to VMs are
> > > given through ssh.
> > > 
> > > 
> > > > Then if you could rebuild the FreeBSD dom0 kernel with the above patch
> > > > we might be able to get a bit more of info about blkback shutdown.
> > > I rebuilt 13.1 STABLE, with commenting out #undef and adding #define, thus
> > > line number will differ by single line. For this test i did not remove
> > > network interfaces, and did add DPRINTF messages to xnb_detach function as
> > > well, since i hoped to maybe catch something there, by printing pointers. I
> > > somewhat did not like that xnb_detach does not check for NULL return from
> > > device_get_softc, nor for device_t argument, so i though, maybe those
> > > crashes are something related to that. But i guess this will not be so easy,
> > > and maybe it is safe to assume that "device_t dev" is always valid in that
> > > context.
> > > 
> > > So i ran stress test, system did not crash as it happens often when more
> > > debugging info is printed, characteristics change. But it did leak sysctl
> > > xbbd variables. I'll attach all collected log files. sysctl and xl list
> > > commands differ in timing a little bit. xl list _02 is when all VMs are
> > > turned off. Sysctl only has keys without values, not to trigger xnb tests
> > > while reading all values.
> > 
> > So I've been staring at this for a while, and I'm not yet sure I
> > figured out exactly what's going on, but can you give a try to the
> > patch below?
> > 
> > Thanks, Roger.
> > ---8<---
> > diff --git a/sys/xen/xenbus/xenbusb.c b/sys/xen/xenbus/xenbusb.c
> > index e026f8203ea1..a8b75f46b9cc 100644
> > --- a/sys/xen/xenbus/xenbusb.c
> > +++ b/sys/xen/xenbus/xenbusb.c
> > @@ -254,7 +254,7 @@ xenbusb_delete_child(device_t dev, device_t child)
> >   static void
> >   xenbusb_verify_device(device_t dev, device_t child)
> >   {
> > -	if (xs_exists(XST_NIL, xenbus_get_node(child), "") == 0) {
> > +	if (xs_exists(XST_NIL, xenbus_get_node(child), "state") == 0) {
> >   		/*
> >   		 * Device tree has been removed from Xenbus.
> >   		 * Tear down the device.
> > 
> 
> Hi, Roger,
> 
> Yes, i will run it, but results will likely be on Monday.
> 
> I have done some more digging, but since i do not know how things work
> internally, i have a lot of unknowns and assumptions. I did a lot of DPRINTF
> stuff in various places, i will attach logs, maybe they give better picture.
> 
> For easier log reading i filter single log pair with commands:
> sed -i '' -e 's/0xfffff80/0x/g' *
> cat *xen_serial_* | grep -E 'xbb|talkv|xs_read|xs_gather|xenbusb_add_device'
> > xbb.log
> cat *sysctl* | grep -E '^dev.xbb' | grep -Ev
> "\.%location:|\.%pnpinfo:|\.%parent" > xbb_sysctl.log
> 
> Then i search for leaked sysctl *.xenstore_path and search that in xbb.log.
> As far as i can tell, cases when it is leaked xbb's have no
> xbb_close_backend call, for whatever reason. Often pointers for leaked xbb
> are a reused ones from previously closed xbb instnaces.
> 
> 
> My suspicion goes for blkback.c, xbb_attach_disk lines:
> ..
>   error = xs_gather(XST_NIL, xenbus_get_node(dev), "physical-device-path",
>       NULL, &xbb->dev_name, NULL);
>   if (error != 0)
>     return;
> ..
> 
> The thing is that for those lost xbb's, this is where error = 2, which is
> ENOENT (if i found correct code).
> 
> Which is weird, because i assume that xbb_attach_disk function is called
> after "hot-plug script" has finished execution. This is defined in
> xbb_attach(device_t dev):
> ..
>   KASSERT(!xbb->hotplug_done, ("Hotplug scripts already executed"));
>   watch_path = xs_join(xenbus_get_node(xbb->dev), "physical-device-path");
>   xbb->hotplug_watch.callback_data = (uintptr_t)dev;
>   xbb->hotplug_watch.callback = xbb_attach_disk;
> ..
> 
> So i added debugging messages at file: /usr/local/etc/xen/scripts/block
> after line "xenstore-write $path/physical-device-path $params", but did not
> see anything suspicious.
> 
> One of my ideas is that, maybe for some reason callback is called too ealry,
> because i would expect that when the callback is called, the path
> $path/physical-device-path should exist (is created by script). But then how
> can xs_gather fail with ENOENT there? Something manages to remove
> $path/physical-device-path in the short window between xenstore-write and
> while callback activates?
> 
> Another thing is that function xbb_attach_disk on error for "error =
> xs_gather" just returns if != 0. I would expect it to be able to handle
> error gracefully it should flag xbb with an error and then shut down or
> something, but since i do not know internals, it is hard to judge correct
> behaviour in that context.
> 
> Another interesting behaviour is that xbb_shutdown is called twice for each
> instance. Once from some function and then from xbb_detach.

This is expected. On proper domain shutdown the backend will be closed
when the frontend switches to Closing state. Hence the cal from
xbb_detach is for cases where the backend hasn't been closed properly
(ie: domain destroy or stuck frontend).

> 
> My problem of geting more useful info is that i do not understand the flow
> of VM creation, including disk allocation.
> 
> For my understanding they just "magically" appear detected with xbb_probe.
> :)
> 
> I tried to understand function xs_gather called by xbb_attach, implemented
> by dev/xen/xenstore/xenstore.c, but it just writes packet somewhere; i
> assumed that dev/xen/xenstore/xenstore_dev.c is the one that handles other
> end for that packet, so i looked at "case XS_READ:" in that file, but it
> calls xs_dev_request_and_reply which is implemented by
> dev/xen/xenstore/xenstore.c, so for me it feels like going in circles. So at
> the moment i did not find which component could be the one that sends
> ENOENT.
> 
> Could you tell the big picture how VM block dev is created? I mean is it xen
> tools writing request in xenstore_dev, but how it then activates xbb probe?
> Which component is responsible for allocating memory xbb instance?

PV devices are all handled in xenstore (a.k.a. xenbus) which is a kind
of key-value database that contains the information about devices.

The FreeBSD kernel sets a watch on certain xenstore paths, so that
when new entries are added a specific function is called on order to
detect whether devices are being added or removed.

> 
> I added printf in dev/xen/xenstore/xenstore.c
> ..
> xs_talkv(struct xs_transaction t, enum xsd_sockmsg_type request_type,
>     const struct iovec *iovec, u_int num_vecs, u_int *len, void **result)
> {
> ..
>   if (msg.type == XS_ERROR) {
>     error = xs_get_error(ret);
>     // .. printf here ..
>     free(ret, M_XENSTORE);
> ..
> And it prints error 22 a lot (EINVAL, if like in errno.h) and sometimes
> error 2 (this is the one i suppose i saw at mentioned xs_gather call). The
> trouble is that i can not locate the source for xs_talkv error generation +
> when all error messages are printed on serial, it starts to influence
> overall system behaviour, since serial can not keep up with message
> generation speed and as a result sysctl leak does not trigger as fast.
> 
> Could i create static variable in xenstore.c and set it to 1 from blkback.c
> when i want logging and to 0 when i don't? Or is it a global component and
> when blkback.c would set's it's logging to 1, all other concurrent messages
> will be logged?
> 
> I would not expect xs_talkv to encounter as many error 22 cases, as it does.
> Could this be relevant? Or could it be that it's just by design that some
> functions test if key exists before creating new one?
> 
> I stared at xs_talkv function for a while. It calls xs_write_store twice, so
> i thought maybe there could be a case when message is not writen
> "atomically"? Thus error 22. Yes, it does sx_xlock(&xs.request_mutex), but
> function xs_write_store does mtx_lock(&xs.ring_lock); and
> mtx_unlock(&xs.ring_lock); in an interesting manner. OTOH if this really was
> even a bit wrong, then system would be far more unstable maybe even
> unusable, right?
> 
> As for now, it seems that actually the third process that was creating zfs
> volumes is not important. What is important is load on system, not load on
> ZVOLs, because i was running VM1 and VM2 on/off stress without zfs create/dd
> script, but was building kernel on parallel, sysctl variables were leaked.
> Kernel was building on UFS ada0, VM disks on ZVOLs ada1.
> 
> For today i guess this is it.
> 
> Thanks for looking into this and sorry i can not produce more useful error
> logs. Earlier dated has no error code prints. Sysctl variables are taken
> when all VMs are down.

I think the problem is not likely with the xenstore implementation
(ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
and manages addition and removal of devices that hang off xenbus.

I'm afraid there's too much data below for me to parse it.

Let me know whether the proposed patch made any difference.

Thanks, Roger.

From nobody Mon Mar 21 14:07:48 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8D31F1A26952
	for ; Mon, 21 Mar 2022 14:08:09 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KMc085llzz4g5S
	for ; Mon, 21 Mar 2022 14:08:08 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lf1-x130.google.com with SMTP id 5so6604751lfp.1
        for ; Mon, 21 Mar 2022 07:08:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=oqnB22vBeUomZwxeahF3u+nOQ9GSkNIAwVYoL83pIbs=;
        b=Fo2rLJ5QfNSRRlz9hu8QcQ2qtYe0LqDV5kOADK9DM174LAdqLhF6JHPDRPsigYxt6X
         f9GhnLkyDPQH2qTMVzYFvqbbzyu6MIkvQnUaklOUVuBO5EqzTKFFeZg9PAwM4pruA9Xa
         YkX3KEtsEGK7p7nGSsqVYPg6l5QE8zxL2MJ96vDdvj+j0xx0xqRoJV1if2nqO6vvAJPy
         hsKi/tZD5F9Wm4KfolYxbegkbNe+K5g8cyCS00YJR5tI/UX7yGrJzWfO/hZgVaet1zt8
         5OfnIfV+nkPS0Dx8V3GBBcFqUvnq4dbzNmUZM4oAANJXi1S3/42L77oJfmI7yoKDry+q
         D1kw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=oqnB22vBeUomZwxeahF3u+nOQ9GSkNIAwVYoL83pIbs=;
        b=1inpmiS67nWmqJlVEOGqPk26X3XkLByetm+aK4DTk4yX9mFIWR4jS8Bm3crcDpKr0z
         IYMdJz5rgyo3KK/ZODOjC+7EDF/50vPR8blE2ve6lR/X5yawIlf2od1LKX3io0U8jHbj
         cMnjlv8lIXC239uhXxT+fC6y3u4RWZQJJlMrSOCxpszwwMFB1Ne7NNhOCXgHSNvm93Wt
         0o/VW2wLTFLXwEVy6sdKSc91IAMU2MhjAoNdGtNtdoI+yV/pXF3uedlC6ORm+W1wDHqH
         768x1c1ygEmAWAflAtfbcp7Bdzlirb/GOGMRmcVZUTIvLUuOJ+WYuuC1uzwCfpERmnw0
         3gtA==
X-Gm-Message-State: AOAM531MK1U2dboSpRHTYmfOt0R1O1W2KRsEp2+N/xzTfM9VcPpTe19M
	73mmjA9YfTN/xOsINkStz7XEgVaxJ08=
X-Google-Smtp-Source: ABdhPJzn2bXTEYVM1v6N0wgu6zBIrb0Q4/iVZEm/kYtCLanz4kmU+Sn6SOa1qDhAISvS2zJ6cBONlA==
X-Received: by 2002:a05:6512:b11:b0:44a:2ead:daf2 with SMTP id w17-20020a0565120b1100b0044a2eaddaf2mr2767724lfu.642.1647871687049;
        Mon, 21 Mar 2022 07:08:07 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id s10-20020a19ad4a000000b0044826a25a2esm1811387lfd.292.2022.03.21.07.08.06
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 21 Mar 2022 07:08:06 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com>
 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
 
 
 
 
 
From: Ze Dupsys 
Message-ID: 
Date: Mon, 21 Mar 2022 16:07:48 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4KMc085llzz4g5S
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=Fo2rLJ5Q;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::130 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-2.00 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(1.00)[1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::130:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.03.21. 13:14, Roger Pau Monné wrote:
> On Fri, Mar 18, 2022 at 08:48:08PM +0200, Ze Dupsys wrote:
>> On 2022.03.18. 17:24, Roger Pau Monné wrote:
>>> On Tue, Mar 15, 2022 at 08:51:57AM +0200, Ze Dupsys wrote:
>>>> On 2022.03.14. 11:19, Roger Pau Monné wrote:
>>>>> On Mon, Mar 14, 2022 at 10:06:58AM +0200, Ze Dupsys wrote:
>>>>>> ..
>>>>>>
>>>>>> Why those lines starting "xnb(xnb_detach:1330):" do not have any message?
>>>>>> Could it be that there is a bad pointer to message buffer that can not be
>>>>>> printed? And then sometimes panic happens because access goes out of allowed
>>>>>> memory region?
>>>>> Some messages in netback are just "\n", likely leftovers from debug.
>>>> Okay, found the lines, it is as you say. So this will not be an easy one.
>>>>
>>>>
>>>>> Can you try to stress the system again but this time with guests not
>>>>> having any network interfaces? (so that netback doesn't get used in
>>>>> dom0).
>>>> I'll try to come up with something. At the moment all commands to VMs are
>>>> given through ssh.
>>>>
>>>>
>>>>> Then if you could rebuild the FreeBSD dom0 kernel with the above patch
>>>>> we might be able to get a bit more of info about blkback shutdown.
>>>> I rebuilt 13.1 STABLE, with commenting out #undef and adding #define, thus
>>>> line number will differ by single line. For this test i did not remove
>>>> network interfaces, and did add DPRINTF messages to xnb_detach function as
>>>> well, since i hoped to maybe catch something there, by printing pointers. I
>>>> somewhat did not like that xnb_detach does not check for NULL return from
>>>> device_get_softc, nor for device_t argument, so i though, maybe those
>>>> crashes are something related to that. But i guess this will not be so easy,
>>>> and maybe it is safe to assume that "device_t dev" is always valid in that
>>>> context.
>>>>
>>>> So i ran stress test, system did not crash as it happens often when more
>>>> debugging info is printed, characteristics change. But it did leak sysctl
>>>> xbbd variables. I'll attach all collected log files. sysctl and xl list
>>>> commands differ in timing a little bit. xl list _02 is when all VMs are
>>>> turned off. Sysctl only has keys without values, not to trigger xnb tests
>>>> while reading all values.
>>>
>>> So I've been staring at this for a while, and I'm not yet sure I
>>> figured out exactly what's going on, but can you give a try to the
>>> patch below?
>>>
>>> Thanks, Roger.
>>> ---8<---
>>> diff --git a/sys/xen/xenbus/xenbusb.c b/sys/xen/xenbus/xenbusb.c
>>> index e026f8203ea1..a8b75f46b9cc 100644
>>> --- a/sys/xen/xenbus/xenbusb.c
>>> +++ b/sys/xen/xenbus/xenbusb.c
>>> @@ -254,7 +254,7 @@ xenbusb_delete_child(device_t dev, device_t child)
>>>    static void
>>>    xenbusb_verify_device(device_t dev, device_t child)
>>>    {
>>> -	if (xs_exists(XST_NIL, xenbus_get_node(child), "") == 0) {
>>> +	if (xs_exists(XST_NIL, xenbus_get_node(child), "state") == 0) {
>>>    		/*
>>>    		 * Device tree has been removed from Xenbus.
>>>    		 * Tear down the device.
>>>
>>
>> Hi, Roger,
>>
>> Yes, i will run it, but results will likely be on Monday.
>>
>> I have done some more digging, but since i do not know how things work
>> internally, i have a lot of unknowns and assumptions. I did a lot of DPRINTF
>> stuff in various places, i will attach logs, maybe they give better picture.
>>
>> For easier log reading i filter single log pair with commands:
>> sed -i '' -e 's/0xfffff80/0x/g' *
>> cat *xen_serial_* | grep -E 'xbb|talkv|xs_read|xs_gather|xenbusb_add_device'
>>> xbb.log
>> cat *sysctl* | grep -E '^dev.xbb' | grep -Ev
>> "\.%location:|\.%pnpinfo:|\.%parent" > xbb_sysctl.log
>>
>> Then i search for leaked sysctl *.xenstore_path and search that in xbb.log.
>> As far as i can tell, cases when it is leaked xbb's have no
>> xbb_close_backend call, for whatever reason. Often pointers for leaked xbb
>> are a reused ones from previously closed xbb instnaces.
>>
>>
>> My suspicion goes for blkback.c, xbb_attach_disk lines:
>> ..
>>    error = xs_gather(XST_NIL, xenbus_get_node(dev), "physical-device-path",
>>        NULL, &xbb->dev_name, NULL);
>>    if (error != 0)
>>      return;
>> ..
>>
>> The thing is that for those lost xbb's, this is where error = 2, which is
>> ENOENT (if i found correct code).
>>
>> Which is weird, because i assume that xbb_attach_disk function is called
>> after "hot-plug script" has finished execution. This is defined in
>> xbb_attach(device_t dev):
>> ..
>>    KASSERT(!xbb->hotplug_done, ("Hotplug scripts already executed"));
>>    watch_path = xs_join(xenbus_get_node(xbb->dev), "physical-device-path");
>>    xbb->hotplug_watch.callback_data = (uintptr_t)dev;
>>    xbb->hotplug_watch.callback = xbb_attach_disk;
>> ..
>>
>> So i added debugging messages at file: /usr/local/etc/xen/scripts/block
>> after line "xenstore-write $path/physical-device-path $params", but did not
>> see anything suspicious.
>>
>> One of my ideas is that, maybe for some reason callback is called too ealry,
>> because i would expect that when the callback is called, the path
>> $path/physical-device-path should exist (is created by script). But then how
>> can xs_gather fail with ENOENT there? Something manages to remove
>> $path/physical-device-path in the short window between xenstore-write and
>> while callback activates?
>>
>> Another thing is that function xbb_attach_disk on error for "error =
>> xs_gather" just returns if != 0. I would expect it to be able to handle
>> error gracefully it should flag xbb with an error and then shut down or
>> something, but since i do not know internals, it is hard to judge correct
>> behaviour in that context.
>>
>> Another interesting behaviour is that xbb_shutdown is called twice for each
>> instance. Once from some function and then from xbb_detach.
> 
> This is expected. On proper domain shutdown the backend will be closed
> when the frontend switches to Closing state. Hence the cal from
> xbb_detach is for cases where the backend hasn't been closed properly
> (ie: domain destroy or stuck frontend).
> 
>>
>> My problem of geting more useful info is that i do not understand the flow
>> of VM creation, including disk allocation.
>>
>> For my understanding they just "magically" appear detected with xbb_probe.
>> :)
>>
>> I tried to understand function xs_gather called by xbb_attach, implemented
>> by dev/xen/xenstore/xenstore.c, but it just writes packet somewhere; i
>> assumed that dev/xen/xenstore/xenstore_dev.c is the one that handles other
>> end for that packet, so i looked at "case XS_READ:" in that file, but it
>> calls xs_dev_request_and_reply which is implemented by
>> dev/xen/xenstore/xenstore.c, so for me it feels like going in circles. So at
>> the moment i did not find which component could be the one that sends
>> ENOENT.
>>
>> Could you tell the big picture how VM block dev is created? I mean is it xen
>> tools writing request in xenstore_dev, but how it then activates xbb probe?
>> Which component is responsible for allocating memory xbb instance?
> 
> PV devices are all handled in xenstore (a.k.a. xenbus) which is a kind
> of key-value database that contains the information about devices.
> 
> The FreeBSD kernel sets a watch on certain xenstore paths, so that
> when new entries are added a specific function is called on order to
> detect whether devices are being added or removed.
> 
>>
>> I added printf in dev/xen/xenstore/xenstore.c
>> ..
>> xs_talkv(struct xs_transaction t, enum xsd_sockmsg_type request_type,
>>      const struct iovec *iovec, u_int num_vecs, u_int *len, void **result)
>> {
>> ..
>>    if (msg.type == XS_ERROR) {
>>      error = xs_get_error(ret);
>>      // .. printf here ..
>>      free(ret, M_XENSTORE);
>> ..
>> And it prints error 22 a lot (EINVAL, if like in errno.h) and sometimes
>> error 2 (this is the one i suppose i saw at mentioned xs_gather call). The
>> trouble is that i can not locate the source for xs_talkv error generation +
>> when all error messages are printed on serial, it starts to influence
>> overall system behaviour, since serial can not keep up with message
>> generation speed and as a result sysctl leak does not trigger as fast.
>>
>> Could i create static variable in xenstore.c and set it to 1 from blkback.c
>> when i want logging and to 0 when i don't? Or is it a global component and
>> when blkback.c would set's it's logging to 1, all other concurrent messages
>> will be logged?
>>
>> I would not expect xs_talkv to encounter as many error 22 cases, as it does.
>> Could this be relevant? Or could it be that it's just by design that some
>> functions test if key exists before creating new one?
>>
>> I stared at xs_talkv function for a while. It calls xs_write_store twice, so
>> i thought maybe there could be a case when message is not writen
>> "atomically"? Thus error 22. Yes, it does sx_xlock(&xs.request_mutex), but
>> function xs_write_store does mtx_lock(&xs.ring_lock); and
>> mtx_unlock(&xs.ring_lock); in an interesting manner. OTOH if this really was
>> even a bit wrong, then system would be far more unstable maybe even
>> unusable, right?
>>
>> As for now, it seems that actually the third process that was creating zfs
>> volumes is not important. What is important is load on system, not load on
>> ZVOLs, because i was running VM1 and VM2 on/off stress without zfs create/dd
>> script, but was building kernel on parallel, sysctl variables were leaked.
>> Kernel was building on UFS ada0, VM disks on ZVOLs ada1.
>>
>> For today i guess this is it.
>>
>> Thanks for looking into this and sorry i can not produce more useful error
>> logs. Earlier dated has no error code prints. Sysctl variables are taken
>> when all VMs are down.
> 
> I think the problem is not likely with the xenstore implementation
> (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
> and manages addition and removal of devices that hang off xenbus.
> 
> I'm afraid there's too much data below for me to parse it.

Understood. Sounds more tricky than i thought. What could i do to make 
data more useful?


> Let me know whether the proposed patch made any difference.

It did not make any difference, system still panics and sysctl leaks.


Thanks.

From nobody Mon Mar 21 16:35:15 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 36E241A260EF
	for ; Mon, 21 Mar 2022 16:35:26 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KMgG51J09z57HQ
	for ; Mon, 21 Mar 2022 16:35:25 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1647880524;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=4VoKZEzqd6KygRoTSBDEPZCNcXnZ7j1E/vDDTineboE=;
  b=enZ6FY0sbWcuQ4iTbGlarB0ue8P0BbZJKNELkhtSf2kXXULRLPzPfmEU
   Lm6I/DfVvIrtFtyYRB9EhAW6EiXN09BsdDPdlUxuhdxo5tK/L7I1T/L98
   jo+bl8aI3FcyaLX2PXG3JEpTaeVyIqWK7g5p4fLdXlSlvxTWTNzuxspt9
   k=;
X-SBRS: 5.1
X-MesageID: 66240405
X-Ironport-Server: esa5.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:i1fv06h9fLWeefNUXtnfRNp2X161YhAKZh0ujC45NGQN5FlHY01je
 htvCGrXM/2KYjD8KN8kOY+08kxSupPcmIBrHgc6/ClkFCsb9cadCdqndUqhZCn6wu8v7a5EA
 2fyTvGacajYm1eF/k/F3oDJ9CU6jefSLlbFILas1hpZHGeIcw98z0M78wIFqtQw24LhWFnU4
 YmaT/D3YzdJ5RYlagr41IrbwP9flKyaVOQw5wFWiVhj5TcyplFNZH4tDfjZw0jQG+G4KtWSV
 efbpIxVy0uCl/sb5nFJpZ6gGqECaua60QFjERO6UYD66vRJjnRaPqrWqJPwwKqY4tmEt4kZ9
 TlDiXC/YTVzZ67PtPQ4agFZSShnBJR95739LUHq5KR/z2WeG5ft6/BnDUVwNowE4OdnR2pJ8
 JT0KhhUMErF3bjvhuvmFK883azPL+GyVG8bkmtnwjzDS+4vXLjIQrnQ5M8e1zA17ixLNaiAP
 ZZBNWIyBPjGSwAQBAwqA4IwoMmP30P7VABJiVfKipNitgA/yyQuieOwYbI5YOeiT9hVhEGHj
 mTI9W/wBlcdLtP34TSf/3+mneOKjS79cIwIGbCy7fIsh0ecrkQBBRELTUCqutG2j0e/X5RUL
 El8x8Y1hfFsrgrxFIC7BkDm5i7f1vIBZzZOO99gsQKO5rrx2h6YXGEddwd6d94cuPZjEFTGy
 WS1t9/uADVutpicRnSc6qqYoFuOBMQFEYMRTXRaFFVYurEPtKl210uSFYg7TMZZm/WvQVnNL
 ya2QD/Sbln5peoCzO2F8F/OmFpATbCZH1dutm07so9Ihz6VhbJJhaT1sjA3Dt4ade51q2VtW
 1Bex6CjABgmV83lqcB0aLxl8EuVz/iEKibAplVkAoMs8T+gk1b6I9wPsGgmfR45a51VEdMMX
 KM0kVkMjKK/wVPwNfMnC25PI5pCIVfc+STNCamPM4smjmlZfw6b5iB+DXN8LEi2+HXAZZoXY
 M/BGe71VC5yIf0+kFKeGrdMuZd2l3tW7T6CGvjGI+GPjOP2iIi9EuxebjNjr4kRscu5neki2
 40GZpXQlEkHDrGWj+u+2dd7EG3m5EMTXPjeg8dWavSCMkxhHmQgAOXW2rQvZ8pumKE9qwsC1
 ivVtpNwoLYnuUD6FA==
IronPort-HdrOrdr: A9a23:pkIVdaEJImAHDfUbpLqFCpHXdLJyesId70hD6qkvc3Nom52j+/
 xGws536faVslcssHFJo6HmBEClewKnyXcT2/htAV7CZnichILMFu9fBOTZsl/d8kHFh4tgPO
 JbAtRD4b7LfClHZKTBkXCF+r8bqbHtmsDY5ts2jU0dNT2CA5sQkTuRYTzrdHGeKjM2YabQQ/
 Gnl7V6TnebCDwqR/X+IkNAc/nIptXNmp6jSRkaByQ/4A3LqT+z8rb1HzWRwx9bClp0sP0f2F
 mAtza8yrSosvm9xBOZ/2jP765OkN+k7tdYHsSDhuUcNz2poAe1Y4ZKXaGEoVkO0aqSwWdvtO
 OJjwYrPsx15X+UVmapoSH10w2l6zoq42+K8y7uvVLT5ejCAB4qActIgoxUNjHD7VA7gd162K
 VXm0qEqpt+F3r77WvAzumNcysvulu/oHIkn+JWpWdYS5EiZLhYqpFa1F9JEa0HADnx5OkcYa
 VT5fnnlbdrmG6hHjDkVjEF+q3uYp1zJGbKfqE6gL3a79AM90oJjXfxx6Qk7wM9HdwGOtx5Dt
 //Q9dVfYF1P78rhJ1GdZU8qOuMexrwqEH3QSuvyWqOLtBzB5uKke+y3IkI
X-IronPort-AV: E=Sophos;i="5.90,199,1643691600"; 
   d="scan'208";a="66240405"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=QYvS4egLn8aiWUowiHFNU3n1V+lSfxODJqk7Nq0I2Q0iJpW61AIV+qQQLOeXMdfSpAMtBSsswS1z1setm2lLGZg1GQ/ee/z0TdUIHxpsrv+H+Lto8eKhsqxoqBHN9u0OonkkVjuTLmUDjStlsqHPC5nn7I1LpvZ+S9IVArsQIwTqEzppV9l5/SRlISlolwLSk16pp7O2RbLAKCGmLmXy3RJKniFJBZaatIxFR7HE1so8QBXPJcSMo4331aj+K9my3aZuOFo86yyIvSwAEx6pSN8H30fcAdjRgQken8/QKmR26qR103ne5/23CM2WNHWxEsixkkqOavsCYAy7a9HtUg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=45pKe3cMg46l9FVP5dLUyCg6MQOTI5AOZov18GHoSM4=;
 b=LrvpTg+ZqRgW9i7g6TLRsXAPffBelSyQJqYAy8obut6Tsmf3wyGE5NFVOErpRRSpIKLBhFADGSfQ8OvO6uAoWHUF0wJyIlaEqV/x7yNEsF011tGWDy+NY7ITLluKj7OsvTnYoEm6h3D6Oukyma0CXfMDq2S3LEc/65hf4/qngP71ofFjFT6ORGKMRxZFwSEPwFPsgD3gW4/ls1M0LvgHA3/TsgDtP5hls76SXmMSs07xNVWY8bjPMwlSCYhFzNvEeyTl22aS1ENMWoXKR/oFmkm6BJEYCkgPIZvMOGmUbR9U+2pEb0Sx0yq7hswGW3PI9EFRw/e2jOqhOxVACXvxmg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=45pKe3cMg46l9FVP5dLUyCg6MQOTI5AOZov18GHoSM4=;
 b=bWO69Gqfl9aBGy4Rh7QLCXY2OxM8gPfNckBz5RtzBSPFsaEZXYDkjS8EVeRe/Wv21CJuvry21ODEcLoP8KRXkBSYhtMLvg5+gqTyGEanpdbUMIPCzOFd6w25YSBg0Hs/lDRR4qbANJkkIG41+7LEme6PINl/8hqu8DvYAxV02K0=
Date: Mon, 21 Mar 2022 17:35:15 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
 
 
 
 
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO4P123CA0195.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:1a4::20) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 0320a9ba-3536-465e-a23e-08da0b58ca05
X-MS-TrafficTypeDiagnostic: BN6PR03MB2531:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: Sb/3I5RWhFpRe4Xlisk7JnPGvHGHKgduGB+r74Wy7R4/RqI9J1s/mwmdeOkopxT/SjevIOqXmlyiDKtGzKVMIw9Nd+DRyYkmPhklrnhoq3RQSzwLft7Im18Y1isQ95aU6N4+ZhHKuBBR+bgpAGKwSyusQECfoPNFhiEoPuGu2zaG28rwi8BNxjThDJR6Gr4qo1UND90aFPj4/vplyq6eJjYm8/x7o7siuuWj2INSjiDOJeCsrc0/u6J3MHXid+66OOKGbxrOpvYj/M7XfwnnkOKBrf/3O0PnukfQeGGLEq2W95BD32SiyoLQq5enUyyTzJPnm+vVLOz0enN+SzPiE9uheA/T+BD8Dqr/K+Cf9d2PLirnklBQkEjQFAgueOjX2nVnrSubOflXVUkl9yPBp1T8uWeYDr9hwFDgnmS/NkiprGaZeQToA7OrvwlNF6l8QPuN5GV+uZOPGjCZPxoSRo/h9A57GQ3LsQXBoeaXTtH8tl/pPaG3wEfxOLQvxCjN3uOBYGT7MMXpx4CBlhGcc/79hkEC6IEeR42D/jXoLHE3l8lsHD61Rp7MDVHVezkPwsgUN9hXqK0cLDfZnXhZnotiU9IQurDDjpHF4PkhtFc5QijFRmU0XjwuaAx4n6TlZT9vs7FKu5F0M/TEmEFkqw==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(26005)(186003)(83380400001)(4326008)(82960400001)(38100700002)(8936002)(6916009)(5660300002)(66556008)(66946007)(66476007)(8676002)(33716001)(2906002)(6666004)(508600001)(6506007)(316002)(9686003)(6512007)(6486002)(85182001)(86362001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QWlZUFR6eXFPbTl6WXY0MkJSZlFHTHZtbGUxcExhSkVUTDRrTWpMUnVGWkRj?=
 =?utf-8?B?dnkwR0dWMldLRUU2eHRlL2RoSFIxNmtibUVsd1NJVGdCTDAzckNqLzhVb2M1?=
 =?utf-8?B?allmSnhCbXhMdHVPNVl5VWJJVWRUZGpSbzZFdDZxQVRjMFcvZS9yRFF0YjVM?=
 =?utf-8?B?cFZpRmMzbUNjR2hJWXRMUHJVdXpsRTFKY3hsdE52bk5tVmYxMEwrRmRFa1FW?=
 =?utf-8?B?WFEvQjhmcDdiS2V4VndCNWdYRVE3bFZ4b3JjZXhCUGFmMXMzUW5wNGhDOFlJ?=
 =?utf-8?B?cWJTamtoOHBWMWVOSHoxWnhNYkNycm9qZ1ZxUGw3aUpOLzFzYXhGUTR3eEIy?=
 =?utf-8?B?YU9Xd3cxNlNlckw5QjJudXgxTEN5YVZHbVRabXdoWENRVWJDczEzb2tYMFBk?=
 =?utf-8?B?K0tMcEJpS0dYRnZPVXgvaUZoNGcrc1lzOUNUKzA0NkNMdDlNT3F6Y05MNHI2?=
 =?utf-8?B?bHM3a3lSVG4zbUJGcVJOYVNjQk1Id1NFQzBNNkUrYnc3WGJOVjVkajFCNzUr?=
 =?utf-8?B?V2VxQjQyVDU3T2ZNSnpZbWlYSmt0VmFBcDJGKytyTm1hL29kTDZoQ0VmanNL?=
 =?utf-8?B?TnV5UVY5MmtvVWEzdEltRnZyVFRnQkt4WDJvTk4wVEtiYnl6WnRDcFdsRU9D?=
 =?utf-8?B?NHNlejdmT29DL0JMbTUzWjN3MjNVbitsNnV3a0htY0tuZGEwRzFRV0JwLzN5?=
 =?utf-8?B?bzdta1UxQm10cEl0eWJ6aDJLU1Z1R3NQQ25YN3lxclorUkdpNHJXSUhwU01y?=
 =?utf-8?B?ZWV5WjJWT0ZYSDNrRHhCdjR2cWNUZW4yZXhzeTRhM3YwMG42RlFzcnc3bjJQ?=
 =?utf-8?B?Y0U0dnJPSW5CTytYZTMvOThhVkpCUnNqc05vOEp3d3EyWllGNy9JQ2d5R3R3?=
 =?utf-8?B?MjhaN0p4TDMrRXUxMzc1TFM1NkdyRDZvY3FlcUovTzBRNDVHQ0I4ODc4Zi8r?=
 =?utf-8?B?cWJHYkt4ZFM5WUQrUTRnUk5POUdXc2pYbDR6YStSNTk5MkRLMjBKZEU4RGdJ?=
 =?utf-8?B?R0Y3dmUrbHJZVS9mY09xa2xTQzhsQnhCemw0aG9UTThNUCtHeW00WmJlYytR?=
 =?utf-8?B?Z2VsTVZDVFBPNktJOCs2eW02b3llNXk3Yjg5aGUrVktUYnJnbDZpNDh6VHcw?=
 =?utf-8?B?Zm8rQXhaM21wTVA3WXFmSUVlUlNHdWt6YnFuYjl3clh0Q1lxdG9wQ1VRMXFQ?=
 =?utf-8?B?NlFhcEFIKzdvQTE3L0tUOWh1dXptVU9TZWhubWpKOEdpTnR6UlNTckU0djRR?=
 =?utf-8?B?aldMbmJIM2JaS2NLUFZ5VzI0SExVa0NTV0svWHZiTXpmcysrY0xKOVp3Ry9H?=
 =?utf-8?B?d2gzSlRUdExiSWZnTTJoaGRoQkw0cFNqMjM4c1lSZVJBaWNreXgxbE9lN3ZG?=
 =?utf-8?B?aFFuLzdkV0M3RXpITVZqQWU0ekJBTlpJV3dCTkFaT2gxRWJBRzNDNFI5Nm1F?=
 =?utf-8?B?TXVDc2JSZjFLOE42MklZckJpOWpWdm5MMndXcTc2WUJReDZ6RXRwUUFMejZs?=
 =?utf-8?B?REpGYmZtckhsN2F4VmFkUjhMZ1FMUU5hd1o5TzhhVHYxb3lpYVBjM3ZlaUFa?=
 =?utf-8?B?Zm9Yb0NxMkU0TjFUblJtWFpUZlVMVStYQW1tUkVMK1F4SFBjL0haT3dEV2hU?=
 =?utf-8?B?L0RwVVhqNWJSQmM1eFpjQjlBTWs4MFZiWERzSFdDMkh1NFFQQTJkOWFtNXQy?=
 =?utf-8?B?QWtnM1NJWmQ1cVQrbGgxTmpkMTV1dk0ybVVTcUVUK3ZFM3dTNWtLY21Ob2Vx?=
 =?utf-8?B?ZVYrM1gvZWRJcVNWcmhjeGw2dXZWUnQrWTBqbkxMdHhTTm5MYjZaOS96dkpm?=
 =?utf-8?B?eVBPTnlWam9pd2ZIKzNhU1M0NGRFL29CK1NycEFtaTM3S1hFVFV4akNsQ3lO?=
 =?utf-8?B?YmFZYkoxTjdRcG9jQkxJa0FSTEg5RXlzVnoveXlUUHUzVHBQYmZmc3d1cFNR?=
 =?utf-8?Q?b5A3C2ETgEf9EkErMQ7UCuujkPh5hATQ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0320a9ba-3536-465e-a23e-08da0b58ca05
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2022 16:35:20.2147
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xv4Ovvs59Cuvd4ODmdDlgs0WDsA+W0nJuCf748U55yBp93FqFXgud6veM8dKQMtsnLssC9Ssxu1v6ynU8oUBuQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB2531
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KMgG51J09z57HQ
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=enZ6FY0s;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=bWO69Gqf;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=07205606a=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=07205606a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07205606a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote:
> On 2022.03.21. 13:14, Roger Pau Monné wrote:
> > I think the problem is not likely with the xenstore implementation
> > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
> > and manages addition and removal of devices that hang off xenbus.
> > 
> > I'm afraid there's too much data below for me to parse it.
> 
> Understood. Sounds more tricky than i thought. What could i do to make data
> more useful?

I have another patch for you to try. This will make the system a bit
chatty, let's see what you get.

Thanks, Roger.
---8<---
diff --git a/sys/dev/xen/netback/netback.c b/sys/dev/xen/netback/netback.c
index bf54f3a2f28e..f49b6baa90a6 100644
--- a/sys/dev/xen/netback/netback.c
+++ b/sys/dev/xen/netback/netback.c
@@ -99,7 +99,6 @@ static MALLOC_DEFINE(M_XENNETBACK, "xnb", "Xen Net Back Driver Data");
 #define	XNB_RX_FLIP 0	/* netback driver does not support feature-rx-flip */
 
 #undef XNB_DEBUG
-#define	XNB_DEBUG /* hardcode on during development */
 
 #ifdef XNB_DEBUG
 #define	DPRINTF(fmt, args...) \
diff --git a/sys/xen/xenbus/xenbusb.c b/sys/xen/xenbus/xenbusb.c
index e026f8203ea1..767175d7174f 100644
--- a/sys/xen/xenbus/xenbusb.c
+++ b/sys/xen/xenbus/xenbusb.c
@@ -254,11 +254,15 @@ xenbusb_delete_child(device_t dev, device_t child)
 static void
 xenbusb_verify_device(device_t dev, device_t child)
 {
-	if (xs_exists(XST_NIL, xenbus_get_node(child), "") == 0) {
+	struct xenbus_device_ivars *ivars = device_get_ivars(child);
+
+	printf("Check device %s\n", ivars->xd_node);
+	if (xs_exists(XST_NIL, xenbus_get_node(child), "state") == 0) {
 		/*
 		 * Device tree has been removed from Xenbus.
 		 * Tear down the device.
 		 */
+		printf("Delete %s\n", ivars->xd_node);
 		xenbusb_delete_child(dev, child);
 	}
 }
@@ -454,7 +458,11 @@ xenbusb_probe_children(device_t dev)
 				continue;
 			}
 
-			error = device_probe_and_attach(kids[i]);
+			ivars = device_get_ivars(kids[i]);
+			printf("Trying to add dev %s\n", ivars->xd_node);
+			error = xenbus_read_driver_state(ivars->xd_node);
+			if (error != XenbusStateClosed)
+				error = device_probe_and_attach(kids[i]);
 			if (error == ENXIO) {
 				struct xenbusb_softc *xbs;
 
@@ -509,9 +517,9 @@ xenbusb_probe_children(device_t dev)
 			 * that can receive otherend state change events,
 			 * hook up a watch for them.
 			 */
-			ivars = device_get_ivars(kids[i]);
 			xs_register_watch(&ivars->xd_otherend_watch);
 			xs_register_watch(&ivars->xd_local_watch);
+			printf("Added dev %s\n", ivars->xd_node);
 		}
 		free(kids, M_TEMP);
 	}
@@ -907,6 +915,7 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 	case XENBUS_IVAR_STATE:
 	{
 		int error;
+		struct xs_transaction xst;
 
 		newstate = (enum xenbus_state)value;
 		sx_xlock(&ivars->xd_lock);
@@ -915,31 +924,37 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 			goto out;
 		}
 
-		error = xs_scanf(XST_NIL, ivars->xd_node, "state",
-		    NULL, "%d", &currstate);
-		if (error)
-			goto out;
-
 		do {
-			error = xs_printf(XST_NIL, ivars->xd_node, "state",
-			    "%d", newstate);
-		} while (error == EAGAIN);
-		if (error) {
-			/*
-			 * Avoid looping through xenbus_dev_fatal()
-			 * which calls xenbus_write_ivar to set the
-			 * state to closing.
-			 */
-			if (newstate != XenbusStateClosing)
-				xenbus_dev_fatal(dev, error,
-						 "writing new state");
-			goto out;
-		}
+			error = xs_transaction_start(&xst);
+			if (error != 0)
+				goto out;
+
+			error = xs_scanf(xst, ivars->xd_node, "state", NULL,
+			    "%d", &currstate);
+			if (error)
+				goto out;
+
+			do {
+				error = xs_printf(xst, ivars->xd_node, "state",
+				    "%d", newstate);
+			} while (error == EAGAIN);
+			if (error) {
+				/*
+				 * Avoid looping through xenbus_dev_fatal()
+				 * which calls xenbus_write_ivar to set the
+				 * state to closing.
+				 */
+				if (newstate != XenbusStateClosing)
+					xenbus_dev_fatal(dev, error,
+					    "writing new state");
+				goto out;
+			}
+		} while (xs_transaction_end(xst, 0));
 		ivars->xd_state = newstate;
 
-		if ((ivars->xd_flags & XDF_CONNECTING) != 0
-		 && (newstate == XenbusStateClosed
-		  || newstate == XenbusStateConnected)) {
+		if ((ivars->xd_flags & XDF_CONNECTING) != 0 &&
+		    (newstate == XenbusStateClosed ||
+		    newstate == XenbusStateConnected)) {
 			struct xenbusb_softc *xbs;
 
 			ivars->xd_flags &= ~XDF_CONNECTING;
@@ -949,6 +964,8 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 
 		wakeup(&ivars->xd_state);
 	out:
+		if (error != 0)
+			xs_transaction_end(xst, 1);
 		sx_xunlock(&ivars->xd_lock);
 		return (error);
 	}


From nobody Mon Mar 21 17:56:05 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A2830191398A
	for ; Mon, 21 Mar 2022 17:56:23 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KMj3V1htpz3PwV
	for ; Mon, 21 Mar 2022 17:56:22 +0000 (UTC)
	(envelope-from prvs=07205606a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1647885382;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=Vvbqj0utJCcEE0Q9fTPBlT9T5glwFbrnoAFxBG9r8YM=;
  b=R9uoealhQc65uTicTf3lRJIulgZ7vCa6zFBJngPI+uwDh4GmeFaKcz3R
   kMFJQLBlbuqlW7CiVpONyHQkjejqefKS9zRdS7uAAiHkViX1s91rVFr0N
   ELjw8WM+B0TPANWtw5vbjfkNMYYVKne390az4R8muo2hWtJdATzvn3cqn
   8=;
X-SBRS: 5.1
X-MesageID: 66796411
X-Ironport-Server: esa2.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:Rn8er6MA8zS7TP7vrR2nl8FynXyQoLVcMsEvi/4bfWQNrUpw0DAGn
 GtJWGyCPKnYYGfxc992YIq1oE5V6MDWn9ExTwto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h
 ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFcMpBsJ00o5wbZi2Ncw27BVPivW0
 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb
 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Zk
 Yx0rYOJSw4QNKTukrUAUiNnQjl+BPgTkFPHCSDXXc27ykTHdz3nwul0DVFwNoodkgp1KTgQr
 7pCcmlLN03dwbLtqF64YrAEasALNs7kMZlZonh95TrYEewnUdbIRKCiCdpwgmlv154SRK+2i
 8wxKh1hYijufiZ0B3Q9OZQjuMiJhUvDWmgNwL6SjfVuuDWCpOBr65DnLdrPfcOiQ8tflU+c4
 GXc8AzRDAsXOdaExX+V/3SEiPXCkCngVMQVDrLQ3uxnjECL3X0IIBITXFq/5/K+jyaDt8l3c
 hJOvHB09O5rqRLtHoKVswCETGCs50clQ8VpEeQD4givwPvJ4jymV3AEd2sUADA5j/MeSTsv3
 16PutrmAz1zrbGYIU6gGqeoQSCaYnZMczJbDcMQZU5cuoS4/tlv5v7aZow7eJNZmOEZDt0ZL
 9qiiCElz4segscQv0lQ1QCW2mn8znQlo+Nc2+k2Yo5Hxl4iDGJGT9bxgbQ+0RqnBNzFJmRtR
 FBex6CjABkmVPlhbhClTuQXB62O7P2YKjDailMHN8B/q2v9oib6JNAJu2AWyKJV3iEsI2KBX
 aMukVkJuM870IWCM8ebnL5d++x1lPO9RLwJp9jfb8ZUY4gZSeN01HoGWKJk5Ei0yBJEufhmY
 f+zKJ/wZV5HWfUP5GfnHI81jO50rh3SMEuOHPgXOTz8iuHADJNUIJ9YWGazghcRt/rV/l+Kr
 44Ba6NnCXx3CYXDX8UeyqZKRXgiJnknH5Hm7ctRc++IOA19H289TfTWxNscl0ZNxsy5Ss+gE
 qmBZ3Jl
IronPort-HdrOrdr: A9a23:C/PRQKE5aKcAwBkSpLqE7MeALOsnbusQ8zAXPidKOHtom62j5q
 STdZEgviMc5wx8ZJhNo7+90cq7IU80l6Qa3WB5B97LNmTbUQCTTb1K3M/PxCDhBj271sM179
 YET0GmMqySMbGtt7eZ3DWF
X-IronPort-AV: E=Sophos;i="5.90,199,1643691600"; 
   d="scan'208";a="66796411"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=KpeEWqzoYZawnS4b+Gb/IL/dVEyL7Pd91Tmaa8YzwqgKJJnS/sHaiU9y57y92qKWcFAc0b5ZVakUgeRSK83Rr04Hy+y4SQ2++mzNlzjPUs0r6dKzs2em6Y0/jIT9kDLNlL/sUaphYPcss4erXsQEvpxkQIbJCIYHgLDNCWfcXOWJxlU/3+RLsq8yocgFhU0gPAZVYEeH6nd3o1BAkTUDJtVTuYWvBeFIS69InigK5+4/P6ioVCVwX+kD/lM1bj6gy9afr23q4EyI1q7qUTC3Xpy725HGu9LJfUjABQ/y+rnpwZC9vCdtv2WDQc2V6ttAE+ILUmvpn2CerKGC8jQY/w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=eNaGV7eV4QFfNUqMMN322Yd7UghgRTOsyx92BBIbais=;
 b=hvN7kWj0dw/3yrPnnwOc0lGP9+0ROiP84FSD6JGJj9/CwCRCdhQ5iLxfZFyMS0Hhv3lYq5WYIfQBTAWwE0ITdqH7jN/zPem5CRcQ0Hqfz/sxzcQzXkK0539Uo9hLNjE6u54MoVXT3MAcCQXaGKI3msKG7xkjL8omPz6E3laL4Dxf+iZY/GMgjSD348t2hgQLrRqjLcxo4iUW8YLGIjujS76Xqn0obDrD60IApR6vrAVpIfw5+T1npHxMCwFMtQW9iF1p6ZUkk52nnB2UuNp6zmWbGotA34tOb9qmSuGw21QGsxKXAgyqCruscSLkJTtXQB2wt4OKGJDmztjCb2vqLw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=eNaGV7eV4QFfNUqMMN322Yd7UghgRTOsyx92BBIbais=;
 b=bWv6E6ZRRfjlP//VSLE/CojvwAC9+ToYQrlZYOjFf3JrUwYUuO8W0FrvLFpFQR73gz0sB+0vXmdG4uxDuGHJkIM17OnXNJ1mcdX6mGF/pwY+l5kwl79XTR56Rxt46Rsz7fSCmAlsZLzyhh66KgJeo+rc/B5CI29zDiX6oc2B2rc=
Date: Mon, 21 Mar 2022 18:56:05 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com>
 
 
 
 
 
 
 
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LNXP265CA0072.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:5d::36) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 73143ce9-1482-4e53-3f59-08da0b64157a
X-MS-TrafficTypeDiagnostic: DM6PR03MB4329:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: aeADm2dIfntiFsUrneofofr5EPxx3PY7Wd122zA8InlVi9bGF2vekvNAUIf6Dlk9JHzF8tEeEwHXHJwmxDziIyzC8vKnyyOPEAs+67KQmhvgXSXit3M2b7mQGF1JWz4si5P39IfZFcDnJriGQFNL5xiGxctBave1zMu+BV6cxExSZuXfHXfySCydj3FQO2KMX+TpdkR8Qz952fHEIkSuCSJLEa0KPQWEhmya8mPKodIbn6i4X1NOPwjefFSkXWS7+c11Aw2p/+jyppowpToIjCA0rN2FIAIqRsYwlMBdhJ/E4RfEYF6UNMu8lCSEGvlDyYCWouIXshsUOxQO31DPKg9vdiKPKLDfLqM/1kUfMw+Pn4PEQ9xox0oUZ8XWzD8E+J3GfHOi+16Nrb0aJbKqkcSPG6sqUWkDg9Jgkqns2+Dv1gwzdzotAlCj3E9zVjYs8jg/9s/pg4L7QoxNZZptHpI2F1fL9pX5l5SEK2h/Rd+G2XJUFY605yaBW75HfjqXKRnuG6C0ESgA+vHOrofxgaQlZF89CpNPNzspt52wvSfrd8zq4V8ngUz8O5ENU/vUTs+ZUdbbOdO+d2m4R3hGClwwrZAva+mhY0nEg8RE80asJXJv2MJONlpE7P4quVl0rdhvgOoaLVECXmTVz6Pzgw==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(85182001)(186003)(26005)(38100700002)(9686003)(6512007)(8936002)(4326008)(66556008)(66476007)(66946007)(316002)(8676002)(83380400001)(6916009)(33716001)(508600001)(86362001)(6486002)(6506007)(2906002)(6666004)(82960400001)(5660300002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YUgzWHVGS3NoSDZINmZ0dmt4d1BSYlZQK0FjV0lXTURreUhUdW1HV3Ard2Ra?=
 =?utf-8?B?MlY5amZ6aXdyOHp2dGVHaVlzUG41V0RraXJJbXVoc2lVOVR2Nzd5UEszUjNZ?=
 =?utf-8?B?U3BNeEVNVk5kZ3c4bkNkZkNxWlh1Q3cwQ1FUemx3MW9nTjI4WTlYVVBKbnhQ?=
 =?utf-8?B?UW5BaHpvVXFDZWN0RHZzdDBHdmJhQXlXdzExalpYQnFLRVBlUGdqNlFOYi9Y?=
 =?utf-8?B?dUhncXJXSWRqcHlCVEpRemk5RndJUXM1WW9XSkZSazBZU3ROUGJnMG80c09o?=
 =?utf-8?B?OVNpZVFJdlg0aUgvOHdwUWF2ZUszQVNSYnk3Mk91d0pjTmo4WldKaXVjcFdE?=
 =?utf-8?B?SlNDNDZmbXFnSGdmRDdMNDRSMWdiTUVuN0N5dU9KYituR2NxMzB4L0xhdGJp?=
 =?utf-8?B?eDc1WjFlSitMTGwrQ3pYSUJyazg2ZE5Pdjk2M3RrVWsxdlRYdCtpNnY5S2J2?=
 =?utf-8?B?SkpwMkR1ZFQ3UjlNTmc0aU4ybi9rUGpsSTVmc1pUQVNXOWQ5K3lyZzFuVjdZ?=
 =?utf-8?B?V00xdGIraVJyNW0xVDlRMm1JM05sdjZTNDlQZno5OFR0RElweEc1MEhBalZv?=
 =?utf-8?B?dU5ZVzdyYkkyZWV4ZzNaVXhBVkgzRS9NcXRCeGV4MnlYNDcyQUUyNEk3RzA0?=
 =?utf-8?B?RXM3ZFIvakY1MnVEK1c3V3lqVjdnS09NdUxKTzMzc2o1TG0rMFI0SThraEJx?=
 =?utf-8?B?OG5PYVJZS3grZmswUEZrSGFoeEI3MXpIQzJWY1ZRZE02WmpLNFR1RTZJYWVy?=
 =?utf-8?B?R1E2QlNEM3hYZ1d3QkVpMVVGUE5XT2E5ZWpEalNoZ2Mrak1jZlZBc0ZReVdO?=
 =?utf-8?B?V1kvSmNSNzVtcVlMR3hKSWhMWGtwRU9xSEtaejJPbEtkUFZjYThQTWhuUzRw?=
 =?utf-8?B?emVtaitVYUVSN2dPVll0NlV1TTF3dGsvZ0NlNWJQRmlkZDBZb2tUWWJPL0lP?=
 =?utf-8?B?NHp2bUNlYWsvaHAxZlVKajdvQXBoY2pHb0poM2Nsd2ZDbHZ5anhjSVJQRkFO?=
 =?utf-8?B?dWhTVklZcW11VVhMeUwzSDEreUM2SmJ6UVh4MVEwTUEyeUxHUVpZcHBjZHR0?=
 =?utf-8?B?N1FjZXR2UEFsV3k5K0ZwREFSMWg1dTl3dUJLNVRpOVM0Zzh2MGlKNEZKQmVD?=
 =?utf-8?B?cVJUY0ljTWd2dk5JalBpbFhKTDQxOE95Ulg5cGVEWTVxSmJlN3pzMDVxL2Y5?=
 =?utf-8?B?YzFSREZFZDlQb0VoclBvU2ErQlFsSW5GNTluN3k2MzlrQzNGQ3BBWEltcnNL?=
 =?utf-8?B?SHRaTENqQm1qR3FGNHhyeVAvSWFCMFVIcEpVSkFaL0c4TEVrZ2N4dFZSRmxH?=
 =?utf-8?B?K3BKTFZmNkN4MDA5dkJHdDFJM2F3djF6OHZCVVA2Z3M4MmtTdGg3SXhuNFRJ?=
 =?utf-8?B?Si82ZWVIMU5VNVEyazh3ZnV3TGtaYk1FMFZnTFZtYUVDNVVGbnpzWWFrL0M5?=
 =?utf-8?B?aWhyTGtwRXREbW1HanNaNkJIMTArY2ZubnViTkRLNjF3QmxzTzV0YTVSd21D?=
 =?utf-8?B?UGY5aC8yaWtlT284VVE3VDArY25rc3lyOEZZUlE2aXpZT3Z6WlVYdGpOdmNa?=
 =?utf-8?B?Z2FzcEZuU1g2ZmRZSDB5VW40WG9ONytlZEREaGFiZWFoRnEwcWdZUXdsREZZ?=
 =?utf-8?B?K05YbHdkWjZrbXdpMDVMc0NWMEVBVlY5UVFxVlNMVG9XenJ3OFRBRjlkaE1V?=
 =?utf-8?B?QTJzOHJRMXd6RFhZQlM3dkJQUXZZbTZzMFpLK01CbkhJYnE2TlpGaXlIM3N1?=
 =?utf-8?B?NWZzeXRnaE9uc2pjZ1dabUhLYndEUTNxK1BhU0tYVTY0OEd3bFpOYThTZVpa?=
 =?utf-8?B?Tjk5ZFc5VWdmTVdaTmhCNDlJM1JmeGxuQ3ZEQm5ZekRBV25GVGtqcU9Lb3Iy?=
 =?utf-8?B?Wm5QVUl1aTdkVjNuY2hKS2tWWS9EOUdTUWpzdXNoRUFMTkw5RmM3cWhoZVNR?=
 =?utf-8?B?NklDbUViK01JbU52ampNUDlIMHNPTjhzYWxRTlhlQlhhTnpiYjgvNktaZlRy?=
 =?utf-8?B?cVpMd3pFU05BPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 73143ce9-1482-4e53-3f59-08da0b64157a
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2022 17:56:11.3658
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qbD8bEAgU9jDPalTEgV1OzTrXxoWrPf6PhRmbspivM7wtpPw3NG0eKL5siLCmDXmSDJlBh7MEENIODc96oAAxQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4329
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KMj3V1htpz3PwV
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=R9uoealh;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=bWv6E6ZR;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=07205606a=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=07205606a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07205606a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Mar 21, 2022 at 05:35:15PM +0100, Roger Pau Monné wrote:
> On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote:
> > On 2022.03.21. 13:14, Roger Pau Monné wrote:
> > > I think the problem is not likely with the xenstore implementation
> > > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
> > > and manages addition and removal of devices that hang off xenbus.
> > > 
> > > I'm afraid there's too much data below for me to parse it.
> > 
> > Understood. Sounds more tricky than i thought. What could i do to make data
> > more useful?
> 
> I have another patch for you to try. This will make the system a bit
> chatty, let's see what you get.

Forgot to mention: when testing the patch attached to the previous
email there's no need to push the system until you get a panic. Just
detecting when you have stale xbbd entries in sysctl would be enough,
or alternatively when you start to see entries in the output of
`xenstore-ls -fp` like:

/local/domain/0/backend/vbd/XX/XXXXX = ""   (n0)
/local/domain/0/backend/vbd/XX/XXXXX/feature-barrier = "1"   (n0)
/local/domain/0/backend/vbd/XX/XXXXX/feature-flush-cache = "1"   (n0)
/local/domain/0/backend/vbd/XX/XXXXX/max-ring-page-order = "5"   (n0)

Note the lack of a '/local/domain/0/backend/vbd/XX/XXXXX/state' node.

At that point I would request that you attach the output of
`xenstore-ls -fp` together with the full serial log since the system
booted.

You might not need a lot of iterations to trigger that state.

Thanks, Roger.

From nobody Thu Mar 24 13:12:28 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 973011A38C25
	for ; Thu, 24 Mar 2022 13:12:45 +0000 (UTC)
	(envelope-from prvs=07544547a=roger.pau@citrix.com)
Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KPQcr35CBz3MQc
	for ; Thu, 24 Mar 2022 13:12:44 +0000 (UTC)
	(envelope-from prvs=07544547a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648127564;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=caFNcES0G8+2DGywC4BhRgmX3EUibG7y+NwPy71ZlcI=;
  b=GDpqSfy0CdQDZ5IiPnzQb2FLF7fLPknlxKHPih8NOFN2i7PGDjVaaBo0
   yIu3kRDVdHhHqqx197nXESTnMOMYxAKNwMj9kigQFbjfWvHrq7VO4mOhi
   3AMfpydnwTgb3cPRBj0QyHdJXpEAObxyltJYEbxBAqfqXI76kpeJcKeh9
   k=;
X-SBRS: 5.1
X-MesageID: 69440644
X-Ironport-Server: esa4.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:KJ0JdKsuFMqQZq8yW6z+g7kFZOfnVEFeMUV32f8akzHdYApBsoF/q
 tZmKWvSPf3fMDDyLt9xYIy/80sHsJPRz9dnQVQ++X9mQSkR+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1jX5
 YuoyyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi8II4OTlbsMbCJjTTl9NJcaobnuCHyg5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AN
 pZFMWQzMHwsZTUMCwoyBq0yhNu1lybkKQ8JhU2P4vE4tj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUb9S/Sx/XF3D2iWHAHRwQdMdoPncspEGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U
 J4swZf2AAcmV83lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Yp5aIW+yP
 heO6Wu9AaO/2lPwPcebhKrrVqwXIVXIT4y5Bpg4kPIQCnSOSON31H43PhPBt4wcuEMtjbs+K
 f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq
 Ig32grj40g3bdASlQGLrt9JcQFbdyRT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5
 XazRglTz1+XuJENAVzihqxLAF83YatCkA==
IronPort-HdrOrdr: A9a23:fQbjyaykjEfRswpsOi0DKrPxB+gkLtp133Aq2lEZdPULSKOlfp
 GV8MjziyWYtN9IYgBcpTiBUJPwJE80hqQFnbX5XI3SETUO3VHHEGgM1/qa/9SNIVycygcZ79
 YbT0EcMqy9MbEZt7eD3ODQKb9Jq7PnkJxAx92utEuFJTsaLZ2IhD0JbjpzZ3cGIjWucqBJc6
 Z0iPA3xQaISDA0VOj+Ln4ZX9Lbr9mOtJX6bQUNCwc77hLLrTSm7tfBYkOl9yZbcQlihZ8k7G
 jbkxfk/OGCtOy20RXRznLU9NB3n9Tmo+EzWfCku4w2EBmpohu0bI5hX7GEoR80pfyi81Ysnp
 3tpAotJM5+8m7WZSWUphTqsjOQigoG2jvB83Pdu3f5vsDjZDU0DaN69NpkWyqcz3BlkMB30a
 pN0W7cnYFQFwn8kCP04MWNfw12l2KvyEBS3tI7vjh6a88zebVRpYsQ8Ad+C5EbBh/374ghDa
 1HENzc3vBLalmXBkqp8FWHgebcH0jbLC32AXTqifblkQS+W0oJinfw8fZv3kvoMqhND6Wsqd
 60dZiA34s+MfP+JZgNetvpB/HHU1AlCCi8QV66MBDpEroKNGnKrIOy6LIp5PuycJhN15cqno
 /dOWko/VLaVnief/Fm5qc7uywlgV/NKAjF24Vb/dx0q7f8TL3kPWmKT00vidKpp7EaDtfAU/
 i+NZpKC7u7RFGeU7phzkn7Qd1fOHMeWMoatpIyXE+PuNvCLsnvuvbAePjeKbLxGXIvW3/5AH
 EEQD/vTf8wpXyDSzv9mlzcSnntckvw8dZ5F7Xb5fEazMwXOohFomEu+COEDwGwWEl/W4ANDT
 hDyenc49OGTEGNjBT101k=
X-IronPort-AV: E=Sophos;i="5.90,207,1643691600"; 
   d="scan'208";a="69440644"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=SZuukS1mKNfJQVAPwexSrj2Z0dr190gvVe1bzpvgUgYMR/OFLmKW14LpU29MO1ljGmv6HdnwqSsRGkwMd75Kw+8qk9UwIFFCu2hIZvEIba9QgeVfYo5ReQuJtKOsix3Yz/KhB9aeFWStjdc5eVeXPbXkJWFxKnHJlLaoGPi1CEfIZTK/t6FMDflOaNH9Tun+e10cpQWsdhaN6/mZLoquIKld3+CqkmeGBTIJfZ/5iod+hVMFlwH8oozhL21fRJPJ0wPqEeYoLiv5CquuFSYY20vzVmcxsfB8AQq6MN28i9k/EUE1sXUcLLBrciekZWUrutGqrEF51RAMYxPe0fXd2Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=+Vjwepice0Vkdwq82tEQo/0cX+pUQ838hE9iWc+OqDQ=;
 b=YFUIu6xxHTqaNT3Zjs+hJ6gVZqylFZx3zB+K2Ab7U6qVfJBIXCHwwD0jqYMj3iLds1ugKwRNKc8p9sVwndVGbdtuCLYC1ZpGfZXZ904r0WN79dNEpbBgKS0pLckbDN5kug1ohcSZ2akUwvkTM3vjxG0vqqi5zCTyalX6qLbxeuoyzYwfeBRenZVFXrGqxBwLJYy0sHy39eaI6KumA7kA580nogVc7LVpXKP3pt+wCTEphn4K26M0zifPg7MBvnKbYoGQSyETpXbH/BLL+pLGyXcdU8xXg/nELf8BuMys+E9zIQQLT6OV5RkYG63LBZB2b1cairrpOcBaqxbkXckKMw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=+Vjwepice0Vkdwq82tEQo/0cX+pUQ838hE9iWc+OqDQ=;
 b=gYnqH7mPNfdstUDZVphY29YH8ul0JXbhYI3ob19I4HgpdMtRa4Xl/Re7gWcThXhI2kW+rdDQNr74JxaMoVvVd7rFQtgSdMT7K9h1f2mGyEbJQVhZljfJZvTTYnX29k1ysWJ3tSTyUYzuuRtnuYMcvu95rvpCgslKFv4YTMQaG70=
Date: Thu, 24 Mar 2022 14:12:28 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
 
 
 
 
 
 
  
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO3P265CA0013.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:bb::18) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 47d85030-eba8-452d-ef22-08da0d97f574
X-MS-TrafficTypeDiagnostic: BN6PR03MB3057:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: GStuBMwwP9ruA96JVQe56wk7RK8s5GtH4N58JnecXbjIjPuHZE1n1AXBH0phcFiS5M/OO1M7rjQOXbyh1UpAwqAjIEW4OlkJ64yFQRtQbFr5jwlYJ39zlU9+xl406z7enkOnAZx8I/ph76r3joq9kTxljYTUTYS0yPCS0sqkgDeVO4+iUpJEqsD4awS+YxzLIZr/d4grqDvpIldaCok70wBfwEvY3gBBplO4iQeFkPbWy5WVkEOCEOu0KmSFWguj2MARXQMqg5Q6jAqfhWa2PmqWdOuQP3SUuUQBFhAKQJR+D8xL0NGETrGGmDLfuwTfc6mRJY0UShSIR8+hShIa2yXdNcqU+PAls9u4pi7PgSgILngOHKym47m+X5HUkb+0JVLfaTOh9rzoWp0TxfucPLLR/v5fCsKWsssNqZpH7KFDHBLPYmZVoZv4pWUJ2QRpp2gFlgE2/+RvACvm61h4HY75j6XFmdadzYi1bHJmmwRcA+kB7FapIvlm7/AyWuBCFXykgQzPjq7Rrc4PWSG0EWi5vh6b6gidUkzXdoXOaVD7RLbMVMR8BjSZHIEGGd1kFXZCoqlzc7C8BjAJEdjyg7F47t9A9u9U9clsVUtZ9qHdtgFW4UILd/bOePXghnldylETviulCJAdexJeUhlBXA==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(6486002)(4326008)(83380400001)(6512007)(9686003)(5660300002)(66946007)(2906002)(66556008)(66476007)(508600001)(6506007)(6666004)(186003)(26005)(86362001)(8936002)(316002)(6916009)(38100700002)(82960400001)(33716001)(8676002)(85182001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MThyUHY2bU1ja2tpamU3a0kxZVA3NVJiWnNSTnI2YXQ0eHN3VFJYV2JmcFA3?=
 =?utf-8?B?WE15RTY0S0xVdlRSZjhLUHNzM1hZdUozSVk1T0pZSzF3Zldody9LdnI5azZz?=
 =?utf-8?B?cDcvYklhbGg4MHpFNXVZNTEzRUNRdUhQaFo0TENFeXFLY05yaE9lbWc0cWJ3?=
 =?utf-8?B?RXhXMWEycW80UVRxNHN1NXdUdFFXR2NyMHk0WEM1ckJnWVdOQWhVUk1kYkl6?=
 =?utf-8?B?VTZrUlZvODVhZXBlM1BwZFJzT1pseEVyOTFCTUIxaXhTTjJCbnRsQTRueWZp?=
 =?utf-8?B?TXhQdHg3a0Q2U1hVNHp2aHpiQzFkN1FSY3lTOHJ4eEtMT2JjZDJRLzU5Sm9B?=
 =?utf-8?B?bjkzUTV2NTliSUg5N3BIVEhRR0lHNTFVODBVQmpkMGFHdnJiMUxOOUxVZDU0?=
 =?utf-8?B?VFBsd3d6blVLbElzNUNCYmsyNVAyRks1VCtSdTBZK3ZVbXBJWjhFRHc1a3ZQ?=
 =?utf-8?B?Z0Q1Yld1NVRubjhTTzYrVGJPa0pJOVloQlRMdURRTjN1NjFUZ1R5V2VXTXdB?=
 =?utf-8?B?a3pqQUhiQS8wazBXRFBuNUlXK1N0S0V4citiTUtEbTRlbnYrcE8xMVA1Q3ZN?=
 =?utf-8?B?YTFheEMwaDI3ME5tQkNCdG5XTmJIS0puVy9CK1djNTUxQlNkS3RBN0dzbVUr?=
 =?utf-8?B?RnlqY2VJK2lvRVVzUEoxbzBSVlYzZnhSVWdWVTNabDQ1cEV2MVNpN3FwWlcv?=
 =?utf-8?B?UW55cGI1NklSeklSTW4vK0NtRHJWWWJmNUJjcm80anBaYmhpWXBGbXFPUWZ1?=
 =?utf-8?B?ZnAwalB0Q3dTWi9sNURNODdMWVh3cis5cHZkTXdiR3dPRm1PUmhkbTlVME1Q?=
 =?utf-8?B?RXRXbHUrVEEzT3kvR3l0bmJEUlR1UHVKMXM5UUt5YUg0bTJ4bUN4dEFBUEdr?=
 =?utf-8?B?VFVnRXk5VkJhTW1lY0N2d1R2Q0cveFNZYkxldHQrejQvdG56dDgvMlVtTmVV?=
 =?utf-8?B?ZTJtQU9NSkRHWVRGbzJnNjRBenVicVJNTzg5UGREZWhKUldsUXpGUXlqMnU2?=
 =?utf-8?B?bEZoREViaW9YS2tjUVoxQ1VuL0ovUDRPRDk4NEdzcmt6dW9pNEloNjRPS2ZB?=
 =?utf-8?B?TTVYR3ExSDNsMC9PS3pRZ1psVFlHcXd1ZDVXdFdZbERyZGdqQ2JNREpYU1JC?=
 =?utf-8?B?QVFOM1NzSFRBU20rTjd3dFFtaVk5bnI5dDB4U3QyVlNkVFA3VEtFRkhBTEJ2?=
 =?utf-8?B?TjJ0YUZVbWkxc0w2TTdkdlpNcklDZkJ4eCs0M3kyYmlkck9ZM25CMVkrc2xU?=
 =?utf-8?B?ajFCNU43Z050MFl2bmwxcUhLRjdMRzBZZzBuVHZqSnhDY3ZRQjlNZDJUMEFr?=
 =?utf-8?B?dXhhYnBMNmVVeUg1MHp5cGRBWUh2NFFsK000WmlRNnhnNFltYVpuSVBWajVU?=
 =?utf-8?B?WjA0MEE4K01EWmpMSXM2QUpaNzRMTzRFKzBhektwTUxUVzZLRDBod0dCeEhN?=
 =?utf-8?B?bmVVTVI5MzFraHN5TDN2S0FaSXIvMk83L0xjSVM5MGROVG0rNzBEQWN4S1Zm?=
 =?utf-8?B?dFRRTzJFbXhSaEhjYy9hVkdlNzFicDN3SmY0dlMramZMZ01UZmNPQ2c1K0c1?=
 =?utf-8?B?T1lEWWFwNWdDWDBORzlMUlgrUFI3ajkvZ29sTmc2MlFsRU52MHIyOHppL2dN?=
 =?utf-8?B?R2FWMk9WMXVodVBjOGJKbWlJekEzOHptUUtwWTQ2cnVGL0hvdGIvRWhWWXRt?=
 =?utf-8?B?WS9uSHRjeWxyMFZjL2xFSlhaR0tBa2luRkgzbkRRcmZYbDJ1OGpCRVdoWjNj?=
 =?utf-8?B?OGVhUlhQbUhWeXl2TGM5cFcwSGI0VS9rbXpqMjlveXNWdlQxekRIMHdQTWJD?=
 =?utf-8?B?Y2hObC83RTBrUjJyYXdlQUV5aDNYNTBsWC9ZL05nMVFxeFlGTVNJK0JDQkFP?=
 =?utf-8?B?VjNPVTNvdExlNGxnaEhaRlcwZ1J2RDZEc3BJNThGbkphaUlRYXlZdVV1aXNn?=
 =?utf-8?B?blZLdVlqZ0JSOU1qWG51cGZaU1dyaG9pWWV2OS9EZFNTdWRhU2loOUFtWnFa?=
 =?utf-8?B?VHlNa0xuMkRnPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 47d85030-eba8-452d-ef22-08da0d97f574
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2022 13:12:33.7837
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Iqf53txUiCGh1I/qroiv3krmnTvz15TjnDovC6iC5t89V3/yDVlf9FlWJwlTMgaGRECRNNy7gxY5Q/hxujAjEg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB3057
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KPQcr35CBz3MQc
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=GDpqSfy0;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=gYnqH7mP;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=07544547a=roger.pau@citrix.com" designates 216.71.155.144 as permitted sender) smtp.mailfrom="prvs=07544547a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07544547a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Mar 21, 2022 at 06:56:05PM +0100, Roger Pau Monné wrote:
> On Mon, Mar 21, 2022 at 05:35:15PM +0100, Roger Pau Monné wrote:
> > On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote:
> > > On 2022.03.21. 13:14, Roger Pau Monné wrote:
> > > > I think the problem is not likely with the xenstore implementation
> > > > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
> > > > and manages addition and removal of devices that hang off xenbus.
> > > > 
> > > > I'm afraid there's too much data below for me to parse it.
> > > 
> > > Understood. Sounds more tricky than i thought. What could i do to make data
> > > more useful?
> > 
> > I have another patch for you to try. This will make the system a bit
> > chatty, let's see what you get.
> 
> Forgot to mention: when testing the patch attached to the previous
> email there's no need to push the system until you get a panic. Just
> detecting when you have stale xbbd entries in sysctl would be enough,
> or alternatively when you start to see entries in the output of
> `xenstore-ls -fp` like:
> 
> /local/domain/0/backend/vbd/XX/XXXXX = ""   (n0)
> /local/domain/0/backend/vbd/XX/XXXXX/feature-barrier = "1"   (n0)
> /local/domain/0/backend/vbd/XX/XXXXX/feature-flush-cache = "1"   (n0)
> /local/domain/0/backend/vbd/XX/XXXXX/max-ring-page-order = "5"   (n0)
> 
> Note the lack of a '/local/domain/0/backend/vbd/XX/XXXXX/state' node.
> 
> At that point I would request that you attach the output of
> `xenstore-ls -fp` together with the full serial log since the system
> booted.
> 
> You might not need a lot of iterations to trigger that state.

Hello,

Sorry to pester, but do you have any update on this?

I'm quite sure there are races with xenbus device attach/detach, and
the earlier we can get this sorted out the better.

Thanks, Roger.

From nobody Thu Mar 24 16:26:22 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 437CF1A235C0
	for ; Thu, 24 Mar 2022 16:26:38 +0000 (UTC)
	(envelope-from prvs=07544547a=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KPVwY15Q7z4ll4
	for ; Thu, 24 Mar 2022 16:26:37 +0000 (UTC)
	(envelope-from prvs=07544547a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648139197;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=X11Ytl/Gh0k+aemNSzHDaCOFzveGcL2ikcPYe7Wkpu8=;
  b=fFtIoNOTXclglZptTC1vIUv4T83iS6GjvwfLlhE2wssAwi/iDUTR9tn/
   ktCNzS7iuRV/xH8sdWVWpQDFuHi+xywXz4ZHpKAdWcA3Hg8rP9PfqhmdO
   vt70RNSAsgQLyyGEp1oizjOlXrEgVvVGinWZtFMMXpH9QRTWHm7JQ8B2z
   8=;
X-SBRS: 5.1
X-MesageID: 67093999
X-Ironport-Server: esa2.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:5glWgKt+VqqHpCz9ccRMAinYWufnVEFeMUV32f8akzHdYApBsoF/q
 tZmKW+CbviNNzegKIx/bo+0/U0FupWHmNdmSAs9+Sg2Fy1G+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1jX5
 YuoyyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi82Y4rWk9kvSSJfKAB7EqRA4KXXGkag5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AO
 JJIMms+PXwsZTVEFmUKKrNuhdylg3+geB1Uon2v/4cotj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUltSaXwZfm+T2ePUkBYTdHcMY2j/EpEGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U
 J4swZf2AAcmV83lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Yp5aIW+yP
 heO6Wu9AaO/2lPwPcebhKrrVqwXIVXIT4y5Bpg4kPIQCnSOSON31H43PhPBt4wcuEMtjbs+K
 f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq
 Ig32grj40g3bdASlQGLrt9JcQFbdyRT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5
 XazRglTz1+XuJENAVzihqxLAF83YatCkA==
IronPort-HdrOrdr: A9a23:q19l1K9GhWGuDR3dgR1uk+C7I+orL9Y04lQ7vn2ZKCY0TiX8ra
 uTdZsguCMc5Ax6ZJhCo7G90de7Lk80nKQdibX5Vo3PYOCJggWVEL0=
X-IronPort-AV: E=Sophos;i="5.90,207,1643691600"; 
   d="scan'208";a="67093999"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=aTZj/IOfN5yxn4jqnuoZovleaUEd/XeOGQ+hmEAJRxhsJmLkDM+BsLxPmgZqK3XUIOMNbLSOv+mB3eTcglCb0HP63hu81YqT2xviDzWC+DTaExuNa5hlKy8WPRe+ehLaf+iJ3gwt/jZ6Gmb7/Wrx9y1OewmMn61J1osD9t8qNXmAbYwg/nYTJKlxiEghCQva/w/i8MRMJC1gimjkxcamMlc6Ho66LJCfTxWUL0suNf8soItp4FcutkH9KDk0V0sP8VejtW7sV/y6Sv/YGREQyhkPFjj0Wv2tuImJxJyk9u4MTG4sRvAVSdU4XmdcVgOx8sDNmFaL2syxLYw+izuwCg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=YQx+/hchdtV8Q+3fp9DsAbtZ44k6966PT/QWUWCY+vY=;
 b=D3oiH+3ASMifFaSsFJ9xg5p4K33zK5xk9TK8TxPdplUJjvPDZTrHpSxKsYD7zA4YibXivxIm5BqTaQB10wilVUE3UEgK7jfCNxZ+mduKgXCJ1bgUbZRHY9sFPRPqc5dp0iekoDxzBize4c4MKODWxSE/t3YErXtXRf0DsrF9IG04QCP9OlZMQ7QmBI32bg0fkSxj30k+jjvOrBXppgiTg7U/KEDWZwP3IZdzBaWlSkTx+BDO16knQGa7/eYDa0Gj0wv8qENlnkhf2YVF+roFcNA3ypEZTbHpqsq4kbkCWsym/idDBRCO68BnWPjHCS22aIIRPKzbQ/9YRgGF2n3+OQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=YQx+/hchdtV8Q+3fp9DsAbtZ44k6966PT/QWUWCY+vY=;
 b=rBgBXbiw2M9nf0ZhwX3m9Nl9B0GzQAMq7bVwKEESGHRTaSEhoFlU+qTHI71lh7O4liM9/AvEZG1c0gOnxjHIQlWXg27tBlEgt0icQjj+pKJn0wr+R1xJ00mq50s0tx/9N29UXakpXvMZGSJVNuqgvvKURmA4qAXoSymh2Zy9rwE=
Date: Thu, 24 Mar 2022 17:26:22 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
 
 
 
 
  
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO4P265CA0054.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:2ac::10) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 36817cc5-e39b-40bc-8af4-08da0db30b96
X-MS-TrafficTypeDiagnostic: BYAPR03MB3527:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: JslAl9GqIGRPwpkkkvox/xMacGAek0pxwtZjZiXOs/8cyK3VQCaJmRRkP3LAOlYgPyEOchOP1EcpDlDjJx+4JQWxy5cFicF+nyQcgRAwIhRJuGKhTRFy+eabKuIQkvjSDhCyh6dS5PHvMWG0sxvRIRSvIaLGsmTx5rVil99+cgW4VzTWUJO2z+elBGvpqXiZBvcB20LwaxMiLmDx3Cjp/6JQZb1yODrDx2Jj3NqbWS3E3ymrpUsAUMTW39OcJWY00Qg7NEfQgqx/cFv7lEcJV69ctTOURqyAdwj0hWc6GaeYUrGFUKSQJk1/rMLTpnQpo8SD9sDXxOKeE+ck0uGGIXAqjUJQiJVVLKa7BxYbbc+s+aVCcnnOqwD1io3FGLTgFpFBBkDp11jgPlKqRaZIVoohcyR59Xs1H2/usYUYx9VX3sRXnEGo6ZUSaHITGgPplCtgMX12SOW9pE03NLlVIOsb5Ngw12A6GbeUzO0XPdmLy/aO88LG+ythylWuFYMpLPY21dPs5AI74HHlyATuoGlRNJoUb2lE6N9O1QB+NyT3EhyHi5DCrSezegoCdy4aVKxsbqDkv2AEV0ArHn1BEMOoGY3YpkkEIaNwA5bbpN/0b6ZPvPzicuk71KRYViiQKPb1nKCaPoppiML5/T7I/w==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(4326008)(2906002)(83380400001)(66476007)(8936002)(85182001)(66556008)(66946007)(8676002)(6512007)(6506007)(26005)(186003)(508600001)(6666004)(9686003)(33716001)(86362001)(82960400001)(5660300002)(6916009)(6486002)(38100700002)(316002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OUR0MTNhZ1lZbEdlNUE3ODBZUVJzMExia1E5VXp1SXkzQ2lCVC93SkpyME1B?=
 =?utf-8?B?aThwekIrSzlHS2xMOHNFa1N4dVNpOTUzaWlPSmR2a3JGQXFDcXRXS2NkcEh0?=
 =?utf-8?B?Y1ZUc2tsTWEraE5ocmo5MFhhRDh4YVp0SVl0dDBvMVVWSXlHdEVNbXpoUzJi?=
 =?utf-8?B?czl2WUlURUtSTGxwU1huemgyZnBSeVFQVU9ObFpTdWNSVWhSUGtRR2N6VGVl?=
 =?utf-8?B?aTVVdElqVi9PRFZPUlFDN0FsNHo5L3J5bm5qMFN0NkZTYW92SDVpZlFrQ2Jp?=
 =?utf-8?B?b2JjTllBVzdwaDMxbWhBR2pLbUl2SzcvL0JCbXBKZzJadUllREI5dWNZWnRx?=
 =?utf-8?B?RFZzd2I2S1ZnWEttK3UrNVNZb3R0d3hQQmFzemg5LzZWRDFPL01LMi85UmNt?=
 =?utf-8?B?YzFFOWQ3YjhubkhGNDVPa3J5TG9nakg3dFgraHhhcnhZVjdqd01YOFFWMmtO?=
 =?utf-8?B?M25BbkdERmJGKzIzOU9Eb0lhUFdtc0M0U1VSVVRXcFRkbm9JNW1HM3BsVUUy?=
 =?utf-8?B?eG4vUk9nZC9FVXAzMU9UenJNUk5VZUw5d1VRUFRPVzF3NHVlU0l5ay9HVFJC?=
 =?utf-8?B?a3d2YXVUNm0yK3EyVGNQYjMvYWxiR2lHc1VyeDlTZ0lFNDlLb3U5TFYrQ1Vm?=
 =?utf-8?B?blFlcTlxemlCMlh6YTNUSGFXUVVTVVFEcncrOVpZOThBSUFEV24yVklwVzZt?=
 =?utf-8?B?cVdhNFVYOU9teTBZTzczc29LNlVaUTNLZWhVQm5Nd3dkRnlUT3c2OU53dkp2?=
 =?utf-8?B?Ri9xajY1RWtxN2xwSjlqRXRwOERYN1hKcmhrYmtkYWtKTCtGZ2p6SVFQUmxj?=
 =?utf-8?B?SGwyeVY2cC9xN0tzWmtYSW9jRHg3bVlZZXQ3Z1lqeXY4MmgwbjFJZ3JEYjZy?=
 =?utf-8?B?eEhMWFFLd2h5K0xLS1ErdDAwNXo3clJzeE9DUVVIRko5MTRPK05wQjh0T09Y?=
 =?utf-8?B?Y0lXQ1cxN2htYzBhZXJFZ2FQak80OGtYQzZQZ0ZJNVdFRzUxTEhJMEFaS2hQ?=
 =?utf-8?B?MXJqNzZ2bG1BbU9DTks3V2h5R0lNNUhKeWFIMkR6KzFxL2ZreWhqcFJGTEti?=
 =?utf-8?B?MUtvNm9UbVY5bXQ3S1VkYk52ZnhXYkxlMnlJeVV4SWdqT0R0SDRzZHNKNWJ0?=
 =?utf-8?B?ckdCZDBmVG5jSy9vUVBGelZ1dzFlbDkzczhKMit4eFNCSldOZnEvRDFtbXNo?=
 =?utf-8?B?SkgrZS9KUi9vVVVIRXRwRVVFVmZ4WU9BVDBxZEhQczFGNUFXMkdnLzc4Sm9h?=
 =?utf-8?B?R3UvUkpFeE1aTFpramdIMFBkR0tlU0hJTDBaSDZLeFFlb2pKK2tLWWt6T3JG?=
 =?utf-8?B?M1MxMHN3eTkvR1ZiMUViRDNXV04yeWhEWGdYbzNaVTdhdndDalBycW1aelRN?=
 =?utf-8?B?VndsVVhtZ1hpWUZJeUhzMEFtUWw4THJudzZXSVh1ckorMVR3UHl3OEwzeGQr?=
 =?utf-8?B?K1dsMVNYTEptN1pacjFsYVJGdlUzWjZvU0x2c2gyQ05UdUVxREpoeHlobUFK?=
 =?utf-8?B?RUpwZVZabSs3YzM0MGhBUVB6VGxqRVZSeHR2UWdhTWJuQUdXeURoakdxR0hq?=
 =?utf-8?B?TzZXMjhoNzNmWSt6RnB1NG95RVQyR2pBUjdJeUJWVlV3dkVGeVZxMHJ0SjI5?=
 =?utf-8?B?RGlPeGZyS1ZyYXV3M2NhR21GdWJtbWRPeU5YVlh0dXVwQS8rVEx0K253bWRR?=
 =?utf-8?B?Sm9FS3ZUODRNcmw4b1NRMlI1LzRCSWFKN1NiNDQ0YVlxZC9VWGdkVzR5dlJx?=
 =?utf-8?B?UUpXUWFZZkdZMXFHU092cjVaZkMyNmlwclZ4QzlUTUF0THZldW9WUlZGdS9p?=
 =?utf-8?B?TVFNcEpSNFlWa3dKL0VIeS9SbXV3RmU1dVNhNWhuTXdYTFVrZ2N0ZEJhblZ6?=
 =?utf-8?B?bUMxNmhrNXFzSEs2V2EzelB3Q2pnQXQvWG95Rk9YNUJpU29QUXdKaEFDMXR2?=
 =?utf-8?B?L1F6Yk51QXpJRmFZZ29lZ1RjcXErZkdPVkNaa1hhaTJ6ZmhEcGlqanFRZTlG?=
 =?utf-8?B?V1A2VUNnR0lnPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 36817cc5-e39b-40bc-8af4-08da0db30b96
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2022 16:26:27.2474
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h3r8vyN3kJ0fB0irSrrfkrlvTwjXMjodrgQ/g9Jowqn8ULm+T61iUti+XF+iPGaQIhdtsG0cX5DhFtvLh6G8Zw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR03MB3527
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KPVwY15Q7z4ll4
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=fFtIoNOT;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=rBgBXbiw;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=07544547a=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=07544547a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07544547a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Mar 24, 2022 at 06:01:24PM +0200, Ze Dupsys wrote:
> On 2022.03.24. 15:12, Roger Pau Monné wrote:
> > On Mon, Mar 21, 2022 at 06:56:05PM +0100, Roger Pau Monné wrote:
> > > On Mon, Mar 21, 2022 at 05:35:15PM +0100, Roger Pau Monné wrote:
> > > > On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote:
> > > > > On 2022.03.21. 13:14, Roger Pau Monné wrote:
> > > > > > I think the problem is not likely with the xenstore implementation
> > > > > > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects
> > > > > > and manages addition and removal of devices that hang off xenbus.
> > > > > > 
> > > > > > I'm afraid there's too much data below for me to parse it.
> > > > > 
> > > > > Understood. Sounds more tricky than i thought. What could i do to make data
> > > > > more useful?
> > > > 
> > > > I have another patch for you to try. This will make the system a bit
> > > > chatty, let's see what you get.
> > > 
> > > Forgot to mention: when testing the patch attached to the previous
> > > email there's no need to push the system until you get a panic. Just
> > > detecting when you have stale xbbd entries in sysctl would be enough,
> > > or alternatively when you start to see entries in the output of
> > > `xenstore-ls -fp` like:
> > > 
> > > /local/domain/0/backend/vbd/XX/XXXXX = ""   (n0)
> > > /local/domain/0/backend/vbd/XX/XXXXX/feature-barrier = "1"   (n0)
> > > /local/domain/0/backend/vbd/XX/XXXXX/feature-flush-cache = "1"   (n0)
> > > /local/domain/0/backend/vbd/XX/XXXXX/max-ring-page-order = "5"   (n0)
> > > 
> > > Note the lack of a '/local/domain/0/backend/vbd/XX/XXXXX/state' node.
> > > 
> > > At that point I would request that you attach the output of
> > > `xenstore-ls -fp` together with the full serial log since the system
> > > booted.
> > > 
> > > You might not need a lot of iterations to trigger that state.
> > 
> > Hello,
> > 
> > Sorry to pester, but do you have any update on this?
> > 
> > I'm quite sure there are races with xenbus device attach/detach, and
> > the earlier we can get this sorted out the better.
> 
> Hello,
> 
> Yes, i agree.
> 
> Sorry that i could not write sooner, the logs i have gathered so far does
> not seem to be helpful, garbage before panic (i'll attach anyways). I did
> run few tests, 2 of them did panic kernel, so i could not gather
> `xenstore-ls -fp` output.
> 
> While testing in all cases i did not notice any sysctl variable leaks, all
> logged sysctls seemed to be normal. Without patch system did not panic at
> first sysctl variable leak, so maybe the given patch fixed sysctl variable
> leaks, but it did not solve panic problem for sure.
> 
> I am thinking to rewrite testing scripts somehow so that information about
> system is gathered in a better way.
> 
> Would getting rid of ZFS help debugging problem? I'll try to decrease HDD
> count for VMs as well, so that there is less noise in logs.

Hm, TBH I'm not sure what the problem is. I've guessed it was some
kind of leak due to stale backends not being properly cleaned, but it
might be something else.

This seems to be a fairly common trace for your panics:

#0 0xffffffff80c74605 at kdb_backtrace+0x65
#1 0xffffffff80c26611 at vpanic+0x181
#2 0xffffffff80c26483 at panic+0x43
#3 0xffffffff810c1b97 at trap+0xba7
#4 0xffffffff810c1bef at trap+0xbff
#5 0xffffffff810c1243 at trap+0x253
#6 0xffffffff81098c58 at calltrap+0x8
#7 0xffffffff80c7f251 at rman_is_region_manager+0x241
#8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101
#9 0xffffffff80c362bc at kernel_sysctl+0x3ec
#10 0xffffffff80c36933 at userland_sysctl+0x173
#11 0xffffffff80c3677f at sys___sysctl+0x5f
#12 0xffffffff810c249c at amd64_syscall+0x10c
#13 0xffffffff8109956b at Xfast_syscall+0xfb

Could you give me the output of executing the following on dom0:

$ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
$ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
$ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
$ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
$ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f

That would give us a more accurate trace.

Thanks, Roger.

From nobody Thu Mar 24 16:38:41 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E9EF51A26348
	for ; Thu, 24 Mar 2022 16:39:19 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KPWCC1JpTz4nQm
	for ; Thu, 24 Mar 2022 16:39:19 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lf1-x12a.google.com with SMTP id d5so9009225lfj.9
        for ; Thu, 24 Mar 2022 09:39:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=Nq+D9wGAy5n7WWcpU3ob0rBcWZmQ2t40vYmPw2q4Lf4=;
        b=d8B7rrxYUyxm0AMwMtdFbjggIo8bXO8FTv+2mU9BWNcj2zpPyTOFACxYPiWkUbPWZS
         ms24PY5in7qYzCUMWZ/PyQYpA3WgrtuyXt33i9qFlUszuO8irHrFH5+Zj8x8y51N8vR4
         VgGAVr4dimhG5KurhACEpqM2QWWOrcbDv1sSULp94gZS2HCP237WVbSGAtxsu7rIF53E
         Njz0dZXCsvZ07oLY1RfyAHJygtcwJ2GtRTk0iKyomIl2YGqb0fO2myPCqdZ5U7vlL6Mu
         8dHm7nS1S89TDneqDiFw9KGcj2tbXp5izatJ9A+J+ZrDTVwpXW732tI1nzm44CnUMvKw
         u4aA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=Nq+D9wGAy5n7WWcpU3ob0rBcWZmQ2t40vYmPw2q4Lf4=;
        b=JTcYgk28HMDsF+1W5ayGI2dUqbVNuRkYkljoeESce/cEmE1iVYrSszkY/YXCqw3Rk+
         JMuBWAuG0fqi0V5WB4JcDyfoQHOFLz7T8iO1AorW7OTLJ3+H6i08ghKpA+XIeZX9fp3Y
         5eqUfLxdVXixWn1ZOygDgAhOxOmlEd/yOn1zzzbqwy8thiQJ3Lsoeh2moeIydzZLOo2b
         ozA/JlOFoqSEde91rdH7tgmPN8K8TaRynCiyGyhaWyvRuhIfbsqjk4Z53pDGxpmy43zA
         0Cfy6K0jELeA9PvPRLeCNPdClFn0GGovOyHysfTYTj7aNXP65S6SbQQhe3hHKcwEm9eu
         ulcA==
X-Gm-Message-State: AOAM531pw/nZKLSTeWdGlmYIyVAXy8eOxEdR/H+5qG4yizVyfG98qROK
	lXnyujgXuvzpSUHjKCPfY9g=
X-Google-Smtp-Source: ABdhPJyoSs/Z2YaHxtCob11Lc0SVcktrX5Ckp+nUonJgdu8amF92U1vOJlOQHdt5ZKyE69SxUL64JQ==
X-Received: by 2002:ac2:555e:0:b0:44a:3cfa:3fde with SMTP id l30-20020ac2555e000000b0044a3cfa3fdemr4244459lfk.448.1648139951763;
        Thu, 24 Mar 2022 09:39:11 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id t2-20020a05651c204200b00249324dd50bsm395834ljo.71.2022.03.24.09.39.10
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Thu, 24 Mar 2022 09:39:11 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 
 
 
 
 
  
 
 
 
From: Ze Dupsys 
Message-ID: <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
Date: Thu, 24 Mar 2022 18:38:41 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4KPWCC1JpTz4nQm
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=d8B7rrxY;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-3.61 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-0.99)[-0.987];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-0.62)[-0.619];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12a:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.03.24. 18:26, Roger Pau Monné wrote:
> 
> This seems to be a fairly common trace for your panics:
> 
> #0 0xffffffff80c74605 at kdb_backtrace+0x65
> #1 0xffffffff80c26611 at vpanic+0x181
> #2 0xffffffff80c26483 at panic+0x43
> #3 0xffffffff810c1b97 at trap+0xba7
> #4 0xffffffff810c1bef at trap+0xbff
> #5 0xffffffff810c1243 at trap+0x253
> #6 0xffffffff81098c58 at calltrap+0x8
> #7 0xffffffff80c7f251 at rman_is_region_manager+0x241
> #8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101
> #9 0xffffffff80c362bc at kernel_sysctl+0x3ec
> #10 0xffffffff80c36933 at userland_sysctl+0x173
> #11 0xffffffff80c3677f at sys___sysctl+0x5f
> #12 0xffffffff810c249c at amd64_syscall+0x10c
> #13 0xffffffff8109956b at Xfast_syscall+0xfb
> 
> Could you give me the output of executing the following on dom0:
> 
> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f

Yes, i'd say that with current stress test the panic message always 
contains rman_is_region_manager in mid.

  addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
/usr/src/sys/kern/subr_rman.c:0

  addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
/usr/src/sys/kern/kern_sysctl.c:185

addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
/usr/src/sys/kern/kern_sysctl.c:2305

addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
/usr/src/sys/kern/kern_sysctl.c:2462

addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f
/usr/src/sys/kern/kern_sysctl.c:2335

At the moment: uname -a
FreeBSD lab-01.localnet 13.1-STABLE FreeBSD 13.1-STABLE #27 
stable/13-812638797-dirty: Tue Mar 22 19:45:17 EET 2022 
root@lab-01.localnet:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

Thanks.





From nobody Fri Mar 25 08:17:09 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B82A91A24BB5
	for ; Fri, 25 Mar 2022 08:17:32 +0000 (UTC)
	(envelope-from prvs=076f4df05=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KPw1l229yz3JG1
	for ; Fri, 25 Mar 2022 08:17:31 +0000 (UTC)
	(envelope-from prvs=076f4df05=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648196251;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=5JxypRA8SGnv2r4oFaon3obNgYRv2V7cNZBtsuxU6Y0=;
  b=MUBc5d3+vDAxu5avVnuI4mHvB7N0LWzd90Y1f9nugLXCM4droRG/pwqD
   8Bg96sQnQ+mzSuBGwWxjNaHp3//rwYAdVsiNOM0ehDQ/SQDz+RJrbYve1
   mnog59lWgkfxEgASLHPcVpQdsLJSlZWJS9YcRYrkEn9BF/yP5ZpEDYdft
   k=;
X-SBRS: 5.1
X-MesageID: 67055507
X-Ironport-Server: esa6.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:TIb96qLePtOahmCDFE+RxpUlxSXFcZb7ZxGr2PjKsXjdYENSgzYBm
 DAYC2GBaP+KMWWmf9lyPorn9k0Cv8WDyN9gTAJlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s
 q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh2dQ12YHR7z6l4
 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc
 s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4
 PRghcGta10kArWWl7VEaTB3PgE9BLITrdcrIVDn2SCS50jPcn+qyPRyFkAme4Yf/46bA0kXq
 6ZecmpUKEne2aTmm9pXScE17ignBNPsM44F/Glp0BnSDOo8QICFSKLPjTNd9Glv3p0TQK6BD
 yYfQQdITkubP0VqAXUwM74jpsST2ku8KyIN/Tp5ooJoujOOnWSdyoPFOsfVYNuVbcNblE2Ro
 iTB5WuRKh8AOdGV0jbA4n+qruHVkC7xQ45UE6e3ntZ4jVGO3ncCEzUZUFK6pb+yjUvWZj5EA
 xVKoGx09/F0rRH1CImmN/GlnJKalj8xa4dNMbc31BnXxfuN2S+2K0hHdyEUPbTKq/QKbTAt0
 1aImfbgCjpurKCZRBqhy1uEkd+hEXNLdDFfPEfoWSNAuoC++99r0nojW/45SMaIYsvJ9SYcK
 txghAw3nP0tgMECzM1XFniX0mv39vAlouPYjzg7v15JDCskPOZJhKTysDA3CMqsyq7AHzFtW
 1Bex6CjABgmV83lqcB0aLxl8EuVz/iEKibAplVkAoMs8T+gk1b6I9wAuWwjeRk4aZxfEdMMX
 KM1kVoAjHO0FCH3BZKbnqrrU5h6pUQePYqNug/ogipmPcEqKV7vENBGbk+MxWH9+HXAYolkU
 ap3hf2EVC5AYYw+lWLeb75EjdcDm3BvrUuOFMuT50n2jtKjiIu9FO5t3K2mNbtisstpYWz9r
 r5iCid940kAAbeiM3iNreb+7zkidBAGOHw/kOQOHsarKQt6AmAxTfjXxLIqYYt+mKpJ0OzP+
 xmAtoVwkjITWVWvxd22V01e
IronPort-HdrOrdr: A9a23:D/gz/6p6/MkR3gzvaDXuEIMaV5vJL9V00zEX/kB9WHVpm5Oj+P
 xGzc526farslsssREb+OxpOMG7MBThHLpOkPMs1NCZLXTbUQqTXfpfBO7ZrQEIdBeOlNK1uZ
 0QFpSWTeeAcWSS7vyKkTVQcexQueVvmZrA7Yy1rwYPcegpUdAZ0+4QMHfkLqQcfnghOXNWLu
 v52iIRzADQBkj/I/7LTkUtbqzmnZnmhZjmaRkJC1oO7xSPtyqh7PrfHwKD1hkTfjtTyfN6mF
 K12TDR1+GGibWW2xXc32jc49B/n8bg8MJKAIiphtIOIjvhpw60bMBKWqGEvhoyvOazgWxa2+
 XkklMFBYBe+nnRdma6rV/E3BTh6i8n7zvYxVqRkRLY0LvEbQN/L/AEqZNScxPf5UZllsp7yr
 h302WQsIcSJQ/cnQzmjuK4GC1Cpw6Rmz4PgOQTh3tQXc81c7lKt7ES+0tTDdMpAD/60oY6C+
 NjZfusqMq+SWnqLkwxg1MfgOBFBh8Ib1S7qwk5y4GoOgFt7T5EJxBy/r1cop8CnKhNP6Wsqd
 60d5iAr4s+PfP+XZgNdNvpfvHHeFAlYSi8Rl56cm6XXZ3uBRr22uvKCfMOlaWXRKA=
X-IronPort-AV: E=Sophos;i="5.90,209,1643691600"; 
   d="scan'208";a="67055507"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=PzipaiFKzBzlB2Mh5fmVPcJFSjHqTY34X4rQpdAIaoo3NdO5aVZnZOO+B0jdGkHl3edjOiQvVAD/Z6KngoIzxr/Ms3RfZaAnDBUP4gcXCP52QK+JMQs9Sx2GRMo2E7yLEj+fqOUSm1UwLqZciR4JaV1tG6OWn3Z3nGy53vwco3GX64x3BlGhBsEjHPXGiJgS/KDyMdME7I5RW+dVnnSo/Fm2Hk+0vguc27JbVK2ryCYUpbWj3l6cIkq9zoCNh3gb0eJMl5iCDHryxU0NViDQO7JxJ6yvxypWB3tnr0EUd+q0eL2GIoM7Fu3NxNr7vcMJzBxHEzWcPkqMk/4ExtXtLA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=6lHsSnDUc0USD1Y/y0s2NFhbU9Jy1Iln8qyfVgKHlak=;
 b=fbk2YvR1LYfUPm4fOuoBxYpPmzoTJ/xpDxqzKpGoK+g99UASp+OPqKB5kiw7v0WwUwU4nObiTEOhJOttW+Zo4wLdgI0xPN49LEP3iq2OHdwpM9RxdRlK0QUNXSC/DudZAYufjL84p3Jqdnl3rMku7+Jn9onmArheEWG+2tu/lKeDS2hwa92729GdyQx71u7Q370FTfLrWtCryjC3xM0uwRADPVh581JybXznhJUQWN+vx6IXoHj7jlc6U5WlXwoPZu6tn5Gbr5+gVLVA49f8BNTj/LYfAKGHEU/hOPvUrTIXBzwEUoYJjtmkC+Fvj/T7f/VarAsQIs2BAZOaTcjmFQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=6lHsSnDUc0USD1Y/y0s2NFhbU9Jy1Iln8qyfVgKHlak=;
 b=o6Ci050QjKyYQrd2PpOwP4M8VYoSiV9sn8UWs/pvSie+xkZy5JvptnMA5pNgeCqa2m4vWq3fKpdke1T81gGQJi9bmmmOosRq8vTYr4UkflL0hKjH76IIWemOMG2F2ao/gezku9b73ZodK/+W4UFiQDMU4DRcv2dUcu8IsoU75qM=
Date: Fri, 25 Mar 2022 09:17:09 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
 
 
  
 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
X-ClientProxiedBy: LO4P123CA0151.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:188::12) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 6d127526-1880-4b4e-8765-08da0e37df15
X-MS-TrafficTypeDiagnostic: BN3PR03MB2274:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: /xRDjDZkPf3qhsKiyJYQkBrCtBPGTA2qcgbivCtAsXoFu6Bg8nQH3jIU3J2E6zR/9rg2VOTPj0MONeIqdRVrJKo9PiyTR53kSsy997l4S6ugJg72YLZtNQeXva2VX+aXXC8QBnMpriaCqETze/3I8nQ1u1+CdX6Ga+68EnNLtdsKlOH9uS+B4dCOZu/MKUXq6riJfxsVMYEynzhIafTCU3qyuTxNNWx+f8qgNooI5vxiLUSibxKqxmCZILhUa9EHXnfrK8DoCb16g3APVxOclUuztOXRYbHo9RKvBxlpDBJ/6uDgMBghskdQYBfG+t5ojW1vfQuU+bPQsHM97TmlFWj1I7ArE980OBPhltSG8hNmo56a0gBqH4pbZPKgXhCGrN0MyNmBKo46d0vxHXarV++0AmguQjmEFUNC1QeWTZ+TjAQZgdjxwSDobRprovbfWYcmzMnzpAm0RZrwMDnAszu35vzkbi5kUJqlZKLzgYV53nTwLPjMo6aZwZs4Rbo/fOXF+ucQ9izmCKozq9SQFJl3FRLxstXgrCs1jbQVnBtDFFCepKgFivWsBV2nREPO07MuaAc5STS1df4aknXX1p07x5tzBwMJUayeJvFlGBD21tSgPT+trI4w5vwJCLlwPdKIP4E53hlCyBlH+tkNYw==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(5660300002)(8936002)(38100700002)(33716001)(83380400001)(6506007)(9686003)(66476007)(6512007)(66556008)(6666004)(82960400001)(86362001)(2906002)(8676002)(66946007)(4326008)(6486002)(186003)(85182001)(316002)(6916009)(26005)(508600001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K2FNTk9zTkZ3KzNIaktWYXUyZHVGZXVITzFld0dQS3RXRzVxVXJiMXZDK0d5?=
 =?utf-8?B?STljaGxTeEtOaE1Lai9rejR0TkYxYmprazVHWThORHhWZkx4TUNHTFp3WVVy?=
 =?utf-8?B?TnN1TXkrbkhSb1IvN2UvK082WFd4STRxSlBYeWFxUm14WHo2V2oycnVxT2hE?=
 =?utf-8?B?WVc5QlBlakpFS3M3UXZEVDNESEdyMmp1U1J4Qm1HRm9kYmlqYWNsUVl1VU84?=
 =?utf-8?B?eGsxQ0JGaWFVTFZUTWpBbWZwRFZJNnY2VzlKVDN3Y2hoNDdVazNxb1pweFJV?=
 =?utf-8?B?b28vdGt1eENJaHRwSFRnZDF3eXo5ZEZIUG9INUdYaUczNUhiT2w1N2hCalhk?=
 =?utf-8?B?UEQvejVKTXBucWEyM3g1ODRhUU1PbDdwRHVjbGgvSDZHV1o1cHhFL1M5Q0Z5?=
 =?utf-8?B?TkN4Zmxqa2owUU5ndnBDRTRTTy81dTBxTGIzYlFFaWxlZTlaWmt1MTRqNjBs?=
 =?utf-8?B?cUM5L0t0SFVHVjdBZWR4UFovZ2VjeklSYWdPNFdLVWJYNVBBZk1GNUpQQkIw?=
 =?utf-8?B?V25UQzE0UWRwM0YyRCtrbWJpWEFvWDV1dVlCNUlUNkJ4TkorWlVYbExkRUwy?=
 =?utf-8?B?TWJTQzR2SXUva1Z2RWFWYmRjOTZRWm11MnZpTnBtbGVOZzFoTUJjK3NtbVc2?=
 =?utf-8?B?ZTUyYnU3NXpFVVIvelJFTi85QU5mN0tQazFVRFV0VWcybkU2dmhiRDZTSytL?=
 =?utf-8?B?dVowVER3UHFvb0ZBMHpQYzMzamNHczBCSDdmVnRZVTh3NWQ5QU1PS2RXcmhM?=
 =?utf-8?B?blVhMWR1dmI4K0ZiSzh1dnExdWFLV1J0VTErNjhoTkdEYlAwcUMrQlhiYWRz?=
 =?utf-8?B?Tnk5M1FSQjNMQWp4UVdWQVR5TEx3N2EwUHpBc0dkNlplSkltbzdyWnNkVDRV?=
 =?utf-8?B?K29WSGhpZU5kQWp3YVhiRGNyK0NpWjVLa1NkUitadTAzSmlvY1ZTRWNSWVB2?=
 =?utf-8?B?WEZBUy84bDlTMlJMbktkUDErZkQ1YUtaVnlEemwrRXMvcS9jWmNONEdFZk1w?=
 =?utf-8?B?QUk5dGE5Zy9qK1BHSE5NRXdTS3Z2TEc2anBQMWtROVJxcURoRkN1bXRSN1Ja?=
 =?utf-8?B?cS8rWlJzUmFHbWpzTHdzMnEyb0gxSlRmalIvKzFHNHNnanZxY3FrNmRHTlNi?=
 =?utf-8?B?UVBCRzhNNitJVEl5TUVHZ0VmV2M0UmE3dTg2QkZ5QXpZUVFHTWhOMUNWSzVJ?=
 =?utf-8?B?VDhpV3A4UVJ0dDc0RkF2eVVqT0s3enBiSDBXVGRTV000M2x2UWZyWkh0MXVz?=
 =?utf-8?B?R3hkdHVFSk5JdGdQcTJVMVVBMjhDamhYNnlydUNNaU5DYkR0YW5tSi9wcXVh?=
 =?utf-8?B?ZGZIblI1TjI0TC90V1MyWXhDR3RzaG5raVdDakljTFlXeTRvNXUxQUFoTDRU?=
 =?utf-8?B?VjVPdnVHazhmS3dLNzlEcXVleklWaHhISUhaTi9hR2lWM1FUOWlpd3MwQWNJ?=
 =?utf-8?B?MWtkL1dJaGR2N1FCbHVncnZEM0VMOTBDdU9GK281MVU4WEZHNnBHcEdhQi9i?=
 =?utf-8?B?TG1BVlc4ZnNhMTBiVitvVCtBRFRoTmRaeFIvUzRtZEw1STI3L0JxYndnWXAv?=
 =?utf-8?B?SHJUMUNpdW9pZ3Rna1AzaW41cDNqNlh6d1dCWU8wVDNheXBJdUlkMnczVW9L?=
 =?utf-8?B?cW1IOWdxbDNTdHhra3lBbEwzRXpPVnhTVVMwWFFhVWpTMjE2RFlQUG05TC91?=
 =?utf-8?B?RWlyNG5aTnY1TlpONGVzbVF3UXNTV2JTYXFYRFpqb0Rqdk1zbWFUNHFRZjhB?=
 =?utf-8?B?dHVyQ0lvcXd3ZzRxZU90N0FsSjRNYzRBYmEvZ1R3S1lHOHhaa0NacG9iNlRT?=
 =?utf-8?B?M3hIdEFtK1JDcUU1TW85cjlhWnZYdFVKMlZ2eURWbm1ZcjQ5S2J0N214TTkr?=
 =?utf-8?B?a0g1S24wZlhrb1ptOE1zTDdiTXdNVzVFaHFHV2V0b3loMlpRK2NyZFV6cERl?=
 =?utf-8?B?blJRc2J1VkhCQjVDVnRuUlZyU2VrT05zM0NKVFZUd0JCZm5KSDBLUXc5dTVX?=
 =?utf-8?B?T3FiclJvYkZ3PT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d127526-1880-4b4e-8765-08da0e37df15
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2022 08:17:15.6629
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JW/PSBPTev2BRch1IEFOQW8dGdYQIga1x695LlxRLB6rKoPhp5BSHBkH7olBzkxjR3YBvoBZGCQNybwU84U3ig==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR03MB2274
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KPw1l229yz3JG1
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=MUBc5d3+;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=o6Ci050Q;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=076f4df05=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=076f4df05=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=076f4df05=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-0.995];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Mar 24, 2022 at 06:38:41PM +0200, Ze Dupsys wrote:
> On 2022.03.24. 18:26, Roger Pau Monné wrote:
> > 
> > This seems to be a fairly common trace for your panics:
> > 
> > #0 0xffffffff80c74605 at kdb_backtrace+0x65
> > #1 0xffffffff80c26611 at vpanic+0x181
> > #2 0xffffffff80c26483 at panic+0x43
> > #3 0xffffffff810c1b97 at trap+0xba7
> > #4 0xffffffff810c1bef at trap+0xbff
> > #5 0xffffffff810c1243 at trap+0x253
> > #6 0xffffffff81098c58 at calltrap+0x8
> > #7 0xffffffff80c7f251 at rman_is_region_manager+0x241
> > #8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101
> > #9 0xffffffff80c362bc at kernel_sysctl+0x3ec
> > #10 0xffffffff80c36933 at userland_sysctl+0x173
> > #11 0xffffffff80c3677f at sys___sysctl+0x5f
> > #12 0xffffffff810c249c at amd64_syscall+0x10c
> > #13 0xffffffff8109956b at Xfast_syscall+0xfb
> > 
> > Could you give me the output of executing the following on dom0:
> > 
> > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
> > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
> > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
> > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f
> 
> Yes, i'd say that with current stress test the panic message always contains
> rman_is_region_manager in mid.

That's great. In fact I think I was mislead by the kdb resolved
symbols not being very accurate.

>  addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> /usr/src/sys/kern/subr_rman.c:0

It's a shame this one hasn't been resolved properly. I think this
would point to sysctl_rman, but without proper debug that's just a
guess. Could you install GNU binutils and try to resolve using GNU
addr2line?

$ pkg install binutils
$ /usr/local/bin/addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251

If you could post the output of that I think it would be helpful.

After attempting to resolve the address, can you give the attached
patch a try? (maybe it's not going to make a difference, as without
that symbol resolved this is just a hunch).

---8<---
diff --git a/sys/kern/subr_rman.c b/sys/kern/subr_rman.c
index 1bbaff8264ef..f73151c27bbe 100644
--- a/sys/kern/subr_rman.c
+++ b/sys/kern/subr_rman.c
@@ -1000,9 +1000,10 @@ sysctl_rman(SYSCTL_HANDLER_ARGS)
 		if (rman_idx-- == 0)
 			break;
 	}
-	mtx_unlock(&rman_mtx);
-	if (rm == NULL)
+	if (rm == NULL) {
+		mtx_unlock(&rman_mtx);
 		return (ENOENT);
+	}
 
 	/*
 	 * If the resource index is -1, we want details on the
@@ -1016,6 +1017,7 @@ sysctl_rman(SYSCTL_HANDLER_ARGS)
 		urm.rm_start = rm->rm_start;
 		urm.rm_size = rm->rm_end - rm->rm_start + 1;
 		urm.rm_type = rm->rm_type;
+		mtx_unlock(&rman_mtx);
 
 		error = SYSCTL_OUT(req, &urm, sizeof(urm));
 		return (error);
@@ -1037,6 +1039,7 @@ sysctl_rman(SYSCTL_HANDLER_ARGS)
 				goto found;
 	}
 	mtx_unlock(rm->rm_mtx);
+	mtx_unlock(&rman_mtx);
 	return (ENOENT);
 
 found:
@@ -1062,6 +1065,7 @@ sysctl_rman(SYSCTL_HANDLER_ARGS)
 	ures.r_flags = res->r_flags;
 
 	mtx_unlock(rm->rm_mtx);
+	mtx_unlock(&rman_mtx);
 	error = SYSCTL_OUT(req, &ures, sizeof(ures));
 	return (error);
 }


From nobody Sat Mar 26 07:09:27 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BFDFD1A3F17F
	for ; Sat, 26 Mar 2022 07:10:05 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KQVTS6jRBz3Qsg
	for ; Sat, 26 Mar 2022 07:10:04 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x229.google.com with SMTP id u3so12912973ljd.0
        for ; Sat, 26 Mar 2022 00:10:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=BucVQxUrcFaka/tF+rxEZ6gy0b/vrT57y+VJfPqZBMM=;
        b=D0qmtamqlUwL8BunEOgUkO5n1QEM6f4K/VVUpIlqvwvNAK1hnSaXduo5yeZKcc03xg
         NY0xn27fdB7uIb3RCe7OjVDcHMrGRqz+QatYXcTZlka3hTh5RCkRQHCBrd57eizFuiZ4
         WPMIWQSEd2HQ83d96S8h7UIKq5pxzlDPYcma29ZGOrAKHlI/+zvTsG89EWjvq69sa00V
         QPfX9rS1Zi9Qn7NwdAeAHzpA3vlgl7Gp4KCgMFjDNkDJ9ysMzG9phvObi/JdqrWIAmAB
         LZECFCHgBCC32S+o6JI4Eb2MR7pugJZtZa3/WabpYNwiDpTmV7UrzJkGnE4Y1yWvRkfo
         yARA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=BucVQxUrcFaka/tF+rxEZ6gy0b/vrT57y+VJfPqZBMM=;
        b=gwbPuOT+VIyFCDbEUMzNABXBhiIoa+gpcxIgIivyOzd7C9CwJZEghjAsJAtju+GLu2
         qjyCQnmsnTni6iK+2h1GjV0WBGEO7xJkOf4LNoUeI5trdz0ebrxy9D1fVIWOPTt4WK17
         iFh8o7snp/qvvf6TfqNR095CSZsKHJA2Ejx5J7XtnjTk+ATbgaLw1djf1iSG1/PENLEe
         cAFr+vOwtJtdOahRl8SYmBCDylYXxSV6qJxv6HPmMAEzY8CHmVl+CH0geKTSQbnzFkuQ
         VBaqZ7t9Wj1Zn3GoQlZ6Z8SgBKdL0eM3sHsYAGCTF4+HkD8dkeRxbizCcpCBm8Rki9yV
         0kjA==
X-Gm-Message-State: AOAM531X0RFSywyvFI21taYTzmbDAoDlbvTF2yO8Ticvq5S0KEIb5n4Y
	uRGuzmm8BV69aNq46oasX2Qke2IhMFQ=
X-Google-Smtp-Source: ABdhPJzz35cEzOlk7g2ToVkzfxPLAOpKDlqhREDo2x3lD/wa+V+cYM2p9VUi7/+ehCYwSbBE+WAoAQ==
X-Received: by 2002:a2e:7d05:0:b0:247:ed41:690d with SMTP id y5-20020a2e7d05000000b00247ed41690dmr11119731ljc.92.1648278603516;
        Sat, 26 Mar 2022 00:10:03 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id bi9-20020a0565120e8900b0044a84716777sm89754lfb.87.2022.03.26.00.10.02
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Sat, 26 Mar 2022 00:10:02 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 
 
 
  
 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
From: Ze Dupsys 
Message-ID: <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
Date: Sat, 26 Mar 2022 09:09:27 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4KQVTS6jRBz3Qsg
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=D0qmtamq;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::229 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [0.10 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(1.00)[1.000];
	 NEURAL_HAM_LONG(-0.85)[-0.851];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_SPAM_MEDIUM(0.95)[0.949];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::229:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.03.25. 10:17, Roger Pau Monné wrote:
> On Thu, Mar 24, 2022 at 06:38:41PM +0200, Ze Dupsys wrote:
>> On 2022.03.24. 18:26, Roger Pau Monné wrote:
>>>
>>> This seems to be a fairly common trace for your panics:
>>>
>>> #0 0xffffffff80c74605 at kdb_backtrace+0x65
>>> #1 0xffffffff80c26611 at vpanic+0x181
>>> #2 0xffffffff80c26483 at panic+0x43
>>> #3 0xffffffff810c1b97 at trap+0xba7
>>> #4 0xffffffff810c1bef at trap+0xbff
>>> #5 0xffffffff810c1243 at trap+0x253
>>> #6 0xffffffff81098c58 at calltrap+0x8
>>> #7 0xffffffff80c7f251 at rman_is_region_manager+0x241
>>> #8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101
>>> #9 0xffffffff80c362bc at kernel_sysctl+0x3ec
>>> #10 0xffffffff80c36933 at userland_sysctl+0x173
>>> #11 0xffffffff80c3677f at sys___sysctl+0x5f
>>> #12 0xffffffff810c249c at amd64_syscall+0x10c
>>> #13 0xffffffff8109956b at Xfast_syscall+0xfb
>>>
>>> Could you give me the output of executing the following on dom0:
>>>
>>> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
>>> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
>>> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
>>> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
>>> $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f
>>
>> Yes, i'd say that with current stress test the panic message always contains
>> rman_is_region_manager in mid.
> 
> That's great. In fact I think I was mislead by the kdb resolved
> symbols not being very accurate.
> 
>>   addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
>> /usr/src/sys/kern/subr_rman.c:0
> 
> It's a shame this one hasn't been resolved properly. I think this
> would point to sysctl_rman, but without proper debug that's just a
> guess. Could you install GNU binutils and try to resolve using GNU
> addr2line?
> 
> $ pkg install binutils
> $ /usr/local/bin/addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251

/usr/src/sys/kern/subr_rman.c:0

Prev. line is not my question, it is output from GNU addr2line. I don't 
know why it does not resolve either.


> After attempting to resolve the address, can you give the attached
> patch a try? (maybe it's not going to make a difference, as without
> that symbol resolved this is just a hunch).

Nice find, but it did not resolve panic problem. Aside from that, i 
think this fixes some possible problems since current locking logic does 
not seem to be correct.

Next week i will not be able to run tests till Thursday, but i will run 
some today and tomorrow.




From nobody Sat Mar 26 09:11:09 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DD2691A269FE
	for ; Sat, 26 Mar 2022 09:11:26 +0000 (UTC)
	(envelope-from prvs=077da0f8a=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KQY9T27ftz3vJl
	for ; Sat, 26 Mar 2022 09:11:25 +0000 (UTC)
	(envelope-from prvs=077da0f8a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648285885;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=KtwoOuD704shGHeTODnykPkWOBqNqB4278zHG2z/vHQ=;
  b=D5/LY5lIh+H35OU8UGmtsUr2Livx/jE6foc1QC+MiGD6Qf5pJ9oVVNVz
   isFhWRHrC413V2QFM/r+qcXan5fLftV1Z8vGSrrni5NmHbsvCAu6OtncT
   6rGmz7rQTBRa3ZQSHhm24OKdVJGfjTbccgvUfSvKrGSQBmIDxGH3OEKaV
   8=;
X-SBRS: 5.1
X-MesageID: 67145370
X-Ironport-Server: esa6.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:GkWoi6uHcV00GFzLAvM5uON74+fnVEFeMUV32f8akzHdYApBsoF/q
 tZmKWjQOquMYTCjc4p1aITl/U5Qv8LdyN8wTgRu/3s8FXtB+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1jX5
 4uryyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi8WO67dmegxDiVaEgJePLFqp6PJKiiG5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AO
 JNFNGEwNHwsZTVjC3EMVJtgs9u33EDHdgMGkE6MpLA4tj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUF7QKcy6b4vjqbB0skTyJtN8Aov58PEGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U
 J4swpn2AAcmV83lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Yp9ZJGK3M
 RSJ4mu9AaO/2lPzPMebhKrrVqwXIVXIT4y5Bpg4kPIQCnSOSON31H43PhPBt4wcuEMtjbs+K
 f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq
 Ig32grj40g3bdASlQGKqNZKdgpbdCJT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5
 XazRglTz1+XuJENAVzihqxLAF83YatCkA==
IronPort-HdrOrdr: A9a23:QGJKm6FUeUxqkRTwpLqFCpHXdLJyesId70hD6qkvc3Nom52j+/
 xGws536faVslcssHFJo6HmBEClewKnyXcT2/htAV7CZnichILMFu9fBOTZsl/d8kHFh4tgPO
 JbAtRD4b7LfClHZKTBkXCF+r8bqbHtmsDY5ts2jU0dNT2CA5sQkTuRYTzrdHGeKjM2YabQQ/
 Gnl7V6TnebCDwqR/X+IkNAc/nIptXNmp6jSRkaByQ/4A3LqT+z8rb1HzWRwx9bClp0sP0f2F
 mAtza8yrSosvm9xBOZ/2jP765OkN+k7tdYHsSDhuUcNz2poAe1Y4ZKXaGEoVkO0aqSwWdvtO
 OJjwYrPsx15X+UVmapoSH10w2l6zoq42+K8y7uvVLT5ejCAB4qActIgoxUNjHD7VA7gd162K
 VXm0qEqpt+F3r77WvAzumNcysvulu/oHIkn+JWpWdYS5EiZLhYqpFa1F9JEa0HADnx5OkcYa
 VT5fnnlbdrmG6hHjDkVjEF+q3uYp1zJGbKfqE6gL3a79AM90oJjXfxx6Qk7wM9HdwGOtx5Dt
 //Q9dVfYF1P78rhJ1GdZU8qOuMexrwqEH3QSuvyWqOLtBzB5uKke+y3IkI
X-IronPort-AV: E=Sophos;i="5.90,211,1643691600"; 
   d="scan'208";a="67145370"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=ID/rhpZAbyfYWvhpMRTAJ7pHGnheR2qM3GIBSee9FrW7L3B0Yc6J8zTzW9HWwA/KmEfDEt6XcVHlJzE5D/fdBhF4rW9/nfMGivkX6jVuDrp3aRIszqQsAhDYvLTTbXVOFMyyhQtEA3lZy7BLXp3+5tXoNYZg2lcNkI82ckBY8CAmGkBVfNeU08fWRy2d5jrm2aEwjqwx7VN/j2rWaONIuoxEjIdKahEYPsitpa1bGfDaJoNGdSVGfCBb3AUYKELyibNFxbKS25Qa8mvZ6VQXRJo3slIRKuCNt08Ych5XeVc3uV+tf6NQtXEn6K4L6NRtW8Pl2zhcZd3p4aDDgPTnJw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=hK5yqrGp+Q7p/AdJtRBIaCRjLkGEO2Fh1grLA26QhsM=;
 b=dl8vy3qUgiECQTqIA8Ln6WYruXhVk37czxukx7hUEKonqWQ9VT+OvcCVQQ97iewmMoj70euJH5kJZSLCUfGHfjlD26XKQ7F+09rXMn8ekfUmm6dYoIShhlVXCH4Si5SnyckavkSb2gFln4wGBr2FtjI4uLq5IfAymjHzgYb5grIak/wfenyedt81raaWriUzBt6fUtyeCkXhZ4eeWZo+Vh3y2g19ntSl9o7ZzOgP3lBWZD52FPL7kHhMgUJj4/iegw1cw1lWVpQlzCMeWiFnfse3MOQ7PUtoQOOMxb9eWPw1Ks5MOQXwEL8PlwxHKnMmCEOhEM4TLrYrgz0LbWMxiQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=hK5yqrGp+Q7p/AdJtRBIaCRjLkGEO2Fh1grLA26QhsM=;
 b=h6gYe7VASufXaQMjZ6yP51mxODGd8G+uoUMXZQIxzn5CWVERZM8tO7kutxp8q91Th8ks+sUWDkx7WjUVjhS8GPJPRaGQFUFPDuFDx3x//UviPZV7OU2oZCup56AG/c0V94kXIN8EFB6NrncNI1kvg6xnxRI/H+XQrDdnOVpG1eo=
Date: Sat, 26 Mar 2022 10:11:09 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
  
 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
X-ClientProxiedBy: MRXP264CA0023.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:500:15::35) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: c9cb565e-5185-4434-bafb-08da0f08947b
X-MS-TrafficTypeDiagnostic: DM6PR03MB5209:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: ilrsWpN0r2jW86DlrgRbJDSEIfUsO3g8e4HCNrJc0S2/M8StXFjjqTeNLyTuFdj3chttLSke2bRwsMiHN7z0uoYIhlJHWN2cmBvzrIjkByKeAltXaow7c5B+Rh5LDNSpDWoUJ2ZOsqm20721FlO5n+2UcWNajxxe0YMJx3f8Yh6mNfJzX/0vNj9JiZazbyMMwX/9cld0eKSDit/r/rd2NSzhWzLfubL+NPGE4+GF163Jp3V53aAFrXbwVz5KjkcxA7v3F0PVHnKMkqsFvb0WF11GgBsWPBjWbpNeTHjnsSPgVcej6JviPSYBJK30owBCuq6KTQjVnHNCZ4zdEd2L+sUhhaFqdRk7phHkdJicAZTWIfkw0e31O02jKlxwhgb6DXLmhxGp8lNnJOIQ0YKfAXMs/bX/jC2gTyjAkGtrNnG6CiLUAujt1RBCHF8yrQl993WqSGF7rDFMxwN4HE7r5vSorKtZ200rzkMeYIJZBMZ8jOSmzm/q90iec2RsNkcbuSrCMoTYyLNjG8Asv1sW2giMJD7oXwILpmuNmUGx4cHLE6QE9UZgtSG8uffkFT850wFQZs+qZP/ByBfdrEAtE+DGwWePaZhZjdf/kFqPSfHjwl8IrqpoNv6vXDRFLJDeabRTndYp88dn6kTpcFxmGTTtSH5LG3zWNmW3yIxPdAD67XuyTGWjfzPcCyyGVR9A4rOO0K5ZjYZr7qnlk6KvEQ==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(86362001)(82960400001)(9686003)(38100700002)(6512007)(6916009)(83380400001)(316002)(85182001)(33716001)(6666004)(5660300002)(508600001)(26005)(186003)(6506007)(66476007)(4326008)(2906002)(6486002)(8676002)(66556008)(8936002)(66946007)(67856001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZG0ySXd2OW1lR0lLSkhGb1B5OUhWd1p0S0VEZU1HUDUxN28vN1dkYUZRYUN5?=
 =?utf-8?B?bW0zSHozOEsxWHhLSUJXSkVqY21oQTFFSi84MU1jY0M1OXk3clA5MnV2WjJw?=
 =?utf-8?B?UDRRSXdJTlBaK3JONUxwT1c0TWE3bDNjczR4QkVoajRUbnZTeEEzM2d0TFl3?=
 =?utf-8?B?QUNkbWpLNmVuZ0NnbWt6ZjZsYVYzVFZ6R2JlcGExRHhFUDdIMi9pazZNcG02?=
 =?utf-8?B?M3BhOG1YV3ZCbklOQkNaNFVERmJ0UEh5MnMyNURTTkhqNzVncDZDN0VpY1Jz?=
 =?utf-8?B?TWNZK2JmRldDdllvS0hyczQwQ0prRnFlbnJrcldNaUxDSjVDWGVqK0dCZ01K?=
 =?utf-8?B?QVdicHlFMVh0Q010T3ZuUjk1aXg1aEtQd3YwMlI2WDVCUVpSSXlwTm93UjlP?=
 =?utf-8?B?L3JScWllZ2FOd3NySzYvUkU5dHFmZ2xQck04alRJVEVQdkNqb0hDcXAweXRi?=
 =?utf-8?B?Q1BYbVlpSWJUQkhVejR3QnBpZUVFbDkxTkdGaWlRSWVibUo2aUNUSHNIampH?=
 =?utf-8?B?UjRNc2QrL0VTUHFKWFpBS1ZzdWo2VEZzZXArdGRyeTlhVjBybldMRzlTbDVj?=
 =?utf-8?B?QW53OHcvcUlzOFlwbzJHU2orSloyb3JNdldtMkJwaEJ3azNGVENWdVlyMS9n?=
 =?utf-8?B?SHRFTkJ3c3BTNnJIcHRnUlF4MU1DZlZaRmlrRVVZSXEwcGkxcXpEVXNnMWFE?=
 =?utf-8?B?bks2Sm1JRUIrUkZmSU0zbmFFV0kyZXVDMkFrNlkxQmRSTmllZzlRS2o0ZC96?=
 =?utf-8?B?bnNNWW9PdXlRWmw3cjkyVVpEWDBVNVI2ZDZIZFgyMlhXd3VSeUlFTUNuQUVY?=
 =?utf-8?B?bWEvdlF1RFlubE02K3BTSXM1cnlGV3pJeUpFOVZyRFVMWS9PeXpqeHkrMDQx?=
 =?utf-8?B?M0xiS2NuVU02WWRZeEJCRGt0N2FsQ2cxQURMN3dtMTVHL3VlcXJNeVpRUXZI?=
 =?utf-8?B?aU5odFlwdXFCckdCQWxMU0drRGhHTjJyQkpKUkJLOEQxMmxIUEpYUFR5eEVq?=
 =?utf-8?B?Y2NEbktKYUhwdkJPYm9LalpPRFM1YXZSZklSbWo5MnZOQVE5Ynl6YmFwa2tN?=
 =?utf-8?B?ZlZ4MG85NmxWOG9yNlUvYnNWWk9UREtxd2MxVVN6UllWUTZkVmRiQ1FaUlI0?=
 =?utf-8?B?dzlYeFZoRndjSWRmaWl1YS9ubW1XNXpLNW1XQjNmQiszVFJid2Y3VVZTVkxX?=
 =?utf-8?B?eDc0YWUva1ArK2d4Y2VyaVA2NGJNVTIrWUlCNWJJdU4vRHNYUFJzUUQxQ1kx?=
 =?utf-8?B?VkhSTFhMcDNWM2RON2Q5L2FNVmJaczlSN1RDcU9PTFRCcUk5NWprTXpwbFVE?=
 =?utf-8?B?SUNSSkM1VWl4N1k2QmoxWTcxaFp5bjZGUWhKOXR4RHFUdG9DUjJnQVBRaHp3?=
 =?utf-8?B?d2dldzdrOTZINUdoelRBOGZXa0Jsa1JBbGQ5Wk8yNERYR0Y2ZDZsNCtjOEdz?=
 =?utf-8?B?TXZSMS80Uzh4V0lwQi9iTVBNYTBPbjFzZnRYY3k1RzNxS05PRGNibU90SVcv?=
 =?utf-8?B?blgvdUplRXloa2N6dGIvWlRKa0lwTlVON3NYZVFXNUYyVFdrY0pma1dFRjhB?=
 =?utf-8?B?ZFFvUXZ1OVJteFU4R0dtU2MwZUREM3AvZm5lREVCd00zc0NpakN6NmN4TWNN?=
 =?utf-8?B?aXVESlRZMS9DZlVnaS9qTUlzeDBHVXAvSkgxeG9yZkpya2tpUzIrbzErV2Qw?=
 =?utf-8?B?ZUFhTVgxazkwc2toMzRkaElEc0c1WlNWSmczNTA4S0JYamVlZUNEQ2k5elFB?=
 =?utf-8?B?eU5BeGJodE03cmxNOFZUZzRyZTFTcExOZGlWSWZRTXB5aDFJWG9SWkNkUDNt?=
 =?utf-8?B?YnNPYjJJMG9SaGxqNEIyOXdhYnhzRWVzb1RNVWRRUGtxdTNoYjJ0VFNWY1hC?=
 =?utf-8?B?bDZvRXBOV0V3TG9nRnVMaG5MajdWZ1Zxb2R3SC9BNFZCSHl1RWp2WktoK1pi?=
 =?utf-8?B?cU9OcU5KOGQrN2hGMUQ5cVhYMFpzQ3lYTlcxclpEMzUxWFMvQkJkSUJaTGFY?=
 =?utf-8?B?Nm9UUWZpOFIyWHJjZDN2T2tubW5BMVB2amM4RWpQQk1kdDBmZ0hLK01yTFEy?=
 =?utf-8?B?YW55RVQyaDNsQ3ExYS9aMTFFelc4dm9vSDBTa0gxaGFaekRBcVdQb1dUNWlG?=
 =?utf-8?B?M0lQdjBpWjMwZFhaOEFnbjMzK0hvblNkYkhYM0hjWUlpWGVESEZidFIzMkN4?=
 =?utf-8?B?ZUQwd2gxSkpwVllQTVhHa1hQdlY1MDAyNlFQZC8wdExpTnE2VlNRTGhBUWpF?=
 =?utf-8?B?TmFkdmpIU2JDUi9KNnA1ZHF3bFdFWll0cUFhYzZPNVF6aUhMUWp2S2FZdFJs?=
 =?utf-8?B?YStCQXZOYlNLT3pwNmlOM2t0a20xaWVVVndYUWVmL0c3cGhyYjNaQT09?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c9cb565e-5185-4434-bafb-08da0f08947b
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Mar 2022 09:11:15.3513
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mC0e2BanJfMW6uH9w7G/sxLvhK4Vxt2hRyBmFV9CH92q1mp96Jo3FmwuyD1NltgEVznsaUEX0hW2OjXOnkxNxw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB5209
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KQY9T27ftz3vJl
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b="D5/LY5lI";
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=h6gYe7VA;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=077da0f8a=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=077da0f8a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.54 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=077da0f8a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-0.90)[-0.902];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sat, Mar 26, 2022 at 09:09:27AM +0200, Ze Dupsys wrote:
> On 2022.03.25. 10:17, Roger Pau Monné wrote:
> > On Thu, Mar 24, 2022 at 06:38:41PM +0200, Ze Dupsys wrote:
> > > On 2022.03.24. 18:26, Roger Pau Monné wrote:
> > > > 
> > > > This seems to be a fairly common trace for your panics:
> > > > 
> > > > #0 0xffffffff80c74605 at kdb_backtrace+0x65
> > > > #1 0xffffffff80c26611 at vpanic+0x181
> > > > #2 0xffffffff80c26483 at panic+0x43
> > > > #3 0xffffffff810c1b97 at trap+0xba7
> > > > #4 0xffffffff810c1bef at trap+0xbff
> > > > #5 0xffffffff810c1243 at trap+0x253
> > > > #6 0xffffffff81098c58 at calltrap+0x8
> > > > #7 0xffffffff80c7f251 at rman_is_region_manager+0x241
> > > > #8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101
> > > > #9 0xffffffff80c362bc at kernel_sysctl+0x3ec
> > > > #10 0xffffffff80c36933 at userland_sysctl+0x173
> > > > #11 0xffffffff80c3677f at sys___sysctl+0x5f
> > > > #12 0xffffffff810c249c at amd64_syscall+0x10c
> > > > #13 0xffffffff8109956b at Xfast_syscall+0xfb
> > > > 
> > > > Could you give me the output of executing the following on dom0:
> > > > 
> > > > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> > > > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71
> > > > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc
> > > > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933
> > > > $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f
> > > 
> > > Yes, i'd say that with current stress test the panic message always contains
> > > rman_is_region_manager in mid.
> > 
> > That's great. In fact I think I was mislead by the kdb resolved
> > symbols not being very accurate.
> > 
> > >   addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> > > /usr/src/sys/kern/subr_rman.c:0
> > 
> > It's a shame this one hasn't been resolved properly. I think this
> > would point to sysctl_rman, but without proper debug that's just a
> > guess. Could you install GNU binutils and try to resolve using GNU
> > addr2line?
> > 
> > $ pkg install binutils
> > $ /usr/local/bin/addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251
> 
> /usr/src/sys/kern/subr_rman.c:0
> 
> Prev. line is not my question, it is output from GNU addr2line. I don't know
> why it does not resolve either.

Hm, do you think you could upload (or attach) your
/usr/lib/debug/boot/kernel/kernel.debug and provide an updated panic
trace using that same exact kernel?

I would also need to know the exact revision of your code base and
what patches you have applied. Might be easier to use a plain build
of FreeBSD without any patches.

> 
> > After attempting to resolve the address, can you give the attached
> > patch a try? (maybe it's not going to make a difference, as without
> > that symbol resolved this is just a hunch).
> 
> Nice find, but it did not resolve panic problem.

So you still got the same panic trace with the patch applied?

Trying to figure out whether there could be multiple issues.

Thanks, Roger.

From nobody Sat Mar 26 14:38:29 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 615DC1A3467F
	for ; Sat, 26 Mar 2022 14:38:53 +0000 (UTC)
	(envelope-from prvs=077da0f8a=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KQhRJ21slz3j7v
	for ; Sat, 26 Mar 2022 14:38:52 +0000 (UTC)
	(envelope-from prvs=077da0f8a=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648305532;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=3+GmxL8ZtcD+6f3PEzqPUXFCC5eeh6hjPHv+5feZUJA=;
  b=Gd48kh88h9P0BEhi/QaM7/zbY6B91uGceEDmME3ytyMmrn/seIIYKIA4
   bpxOPcSsLClOwxZmftrkzsrp26JbyYGHOpDhK2bugQGWmEsZWTSwuhl8T
   UIN5Na/FAig4jBWT6LZe91OUNZ1lnt0LDNtVGLcDbGLy4caUhzlTfQ1Mn
   8=;
X-SBRS: 5.1
X-MesageID: 67252212
X-Ironport-Server: esa2.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:yUCstqIUr0otjR2JFE+RxpUlxSXFcZb7ZxGr2PjKsXjdYENS3mRWn
 WcbCm/QMvvfNGL2KNkjPo/goBwHvpHWyNJkHlZlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s
 q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh2dQ22YLR7z6l4
 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc
 s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4
 Mh/toaaTCAEB6732+46axBCEAohHLITrdcrIVDn2SCS50jPcn+qyPRyFkAme4Yf/46bA0kXq
 6ZecmpUKEne2aTmm9pXScE17ignBNPsM44F/Glp0BnSDOo8QICFSKLPjTNd9Glv3J0UR6+ED
 yYfQWZBbQnlbxhLBgsKJakBjMiwnzqjIzIN/Tp5ooJoujOOnWSdyoPFOsfVYNuVbcNblE2Ro
 iTB5WuRKh8AOdGV0jbA4n+qruHVkC7xQ45UE6e3ntZ4jVGO3ncCEzUZUFK6pb+yjUvWZj5EA
 xVKoGx09/F0rRH1CImmN/GlnJKalgMuWMB6FsoX0QK2iaDz2w+DBUNbbyEUPbTKq/QKbTAt0
 1aImfbgCjpurKCZRBqhy1uEkd+hEXNLdDFfPEfoWSNAuoC++99r0nojW/45SMaIYsvJ9SYcK
 txghAw3nP0tgMECzM1XFniX0mv39vAlouPYjzg7v15JDCskPOZJhKTysDA3CMqsyq7AHzFtW
 1Bex6CjABgmV83lqcB0aLxl8EuVz/iEKibAplVkAoMs8T+gk1b6I9wAuWwjeRk4aZxfEdMMX
 KM1kVoAjHO0FCH3BZKbnqrrU5h6pUQePYqNug/ogipmPcEqKV7vENBGbk+MxWH9+HXAYolkU
 ap3hf2EVC5AYYw+lWLeb75EjdcDm3BvrUuOFMuT50n2jtKjiIu9FO5t3K2mNbtisstpYWz9r
 r5iCid940gGCLSlOHiOq+b+7zkidBAGOHw/kOQOHsarKQt6AmAxTfjXxLIqYYt+mKpJ0OzP+
 xmAtoVwkTITWVWvxd22V01e
IronPort-HdrOrdr: A9a23:dDmdPqPLLDKjkMBcTyP155DYdb4zR+YMi2TDiHofdfUFSKClfp
 6V8cjztSWUtN4QMEtQ/uxoHJPwO080kqQFnLX5XI3SJzUO3VHHEGgM1/qB/9SNIVyaygcZ79
 YdT0EcMqyAMbEZt7eC3ODQKb9Jq7PmgcOVbKXlvg9QpGlRGt9dBmxCe2Cm+yNNNW177c1TLu
 vi2iMLnUvpRV0nKuCAQlUVVenKoNPG0LrgfB49HhYirC2Dlymh5rLWGwWRmk52aUIE/Z4StU
 z+1yDp7KSqtP+2jjfaym/o9pxT3P/s0MFKCsCggtUcbh/slgGrToJ8XKDqhkF/nMifrHIR1P
 XcqRYpOMp+r1vXY2GOuBPonzLt1T4/gkWSvmOwsD/Gm4jUVTg6A81OicZyaR3C8Xctu9l6ze
 Ziw3+Zn4A/N2KOoA3No/zzEz16nEu9pnQv1cQJiWZEbIcYYLhN6aQC4UJuFosaFi6S0vFqLA
 BXNrCc2B9qSyLbU5iA1VMfg+BEH05DUytue3Jy9PB8iFNt7TJEJ0hx/r1qop5PzuN5d3B+3Z
 W2Dk1ZrsA/ciYoV9MOOA4ge7rANoWfe2OEDIqtSW6XYZ3vfUi976LK3A==
X-IronPort-AV: E=Sophos;i="5.90,213,1643691600"; 
   d="scan'208";a="67252212"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=HQbfqUQ8ULZkzJRldW6r7uSjCYRh6bpWL/hBQ8PlObmvpEvGgYYj0fo4rb/5is1/K1zo2kBct+hcUrJegRXT3MLyo98KlIsQwzvNnxNwN0GY2WE/FF0XhEVbf3JOU41vJjaMtnujq8sG2jxlPAnJA4rPQUY5mNstPjEas5Bq8W5ATrtN362blEOjPOPWn9+nuEq7SGCmLpA2fgZTMERBKHKaoY/LqH24BEpgA9sNQBFLoPEB1wiIYozgdN7w2OtnIwhQM9ogCxkJr8xPZ1h6A+sqBf9FIy0YyM4B5+GoaBrTpUZ6SbG/b1Ct2lz0MhCyyoLPOWFG31Eotf3eQdJVuA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=Rmbf3IjoWfpa8j3mCcp1Ww3XtuvBdGWCANx1M6Ra+NU=;
 b=SoTuaqnz04PG/myrHAxUGf2TITmYcS8j+qSs/gepQ8H52UYYEWOGCIzi/U0oTLNC/7XLyt1dCBxrfkOKDqjYfaZee/GUNt9Jt9CvBSw7ViQhxC0KkvbZTNK4LFa2k7y7EI+tVZFg7mxVx5tF0pMDGT/3uMP46yaeP1iCNUo+3/le8RdbCU7njob8o7w92ptly3LJQd055u2BjgOiiP/wnfPPQcla8vIOhbjiI8gvUeC/eBtkI1EP7aaZwzGi0jLyS8ZKRudWH5JZOiNUMbAlzUcVPFgCmo5xPBvffYFSFTaKupKNeiIGo+p44esfMUJNHL+w8jdroDS/YI2ZbDlpiQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=Rmbf3IjoWfpa8j3mCcp1Ww3XtuvBdGWCANx1M6Ra+NU=;
 b=u3AeYTdtX+HpdOHXFvMsYoxfyq4H/TERzSgVfPIhaaKvrCa8LZOclMTAJRT62TxotRfvS8BLjJ4mpXmZcHVPjHh9DkFXbSNzRwAXuhI/KsooYmPxW6npr6P3zfgGHNwxxSNJl5PntR1w1ifLlqZNSdB/6zj1JRB01pWIYrkeL1w=
Date: Sat, 26 Mar 2022 15:38:29 +0100
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References:  
 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
X-ClientProxiedBy: MR2P264CA0017.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:500:1::29) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 5b7e48fe-d11a-499a-51c6-08da0f364e8c
X-MS-TrafficTypeDiagnostic: DM6PR03MB5338:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: J/Nkn7648SHZUIBZZ8giKUpVKZbvQeJ2HDGG8z6C/i0Jb2JpiYEHXQgLB06yJnQ4eoJajZE3KQ58pkoaPybXcscqpnwofO0bRRSNIcBB5Nhl+CaSoi84/y8cG27e+KnguMXqBR5wxO/Bl2tGUzgBVY5gmC8VmOZAsZ/gt1HwMNiwGZwBRrlt8SbZfALS5sPqUwoZaaTwySU7kJUtXAbbysvT0vxnL7L/veCE0QniGa/C9ZZM3siAsnImHk4VG9ZauG9k9Em94ZY/d81lIYItgprfYK/o1t76c5OmlOkjFh79NqqaYGt/Q+hurbjnhNrok9gjWATpe5Jg+kKUQoq7qES+uhcmONmgPZ8WuTMe95PtzBQuYVvTJDSBP/cwpFIeoll/EqcHLXrtUCqEkMi/nQg6qu5IXBwHzeXu8nPD8qMUMpsNip44gK1jK3bKFwEKLtl7Zdb9FBHaGaNhRkQVCKAiwh9uVfRNFQb0yX25s21s508jCHlV84Ej/FPxC35FsOjd3Itt1cRmW6+V6SXd/aYOh1jC8+03qq42B9j1CxXJXZIYS6GrOzgiGQQ0c0ZPngN1KWWwQD89gkEL6xnyHBd04jbknG1NVVLlmOC/L3T0itJq04/HvQ8SdJDHxaBLwmlv97b9nqq4W9Rf346uOIZ+cjX1BQOJP8QfGbnb/+/6wJoYChRnJeQbtX9Ggfp64R7E3VCWkJgrB2pQ0mzN4NXonvO4DEKaLvbtNl+EsChatUPFO7Em4buHkJhXqTPuNymJyVPw1d2cPdclyX3Qig==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(508600001)(86362001)(33716001)(316002)(6916009)(6486002)(8936002)(966005)(8676002)(66556008)(66476007)(66946007)(4326008)(26005)(186003)(5660300002)(82960400001)(9686003)(6506007)(6512007)(6666004)(85182001)(83380400001)(38100700002)(2906002)(67856001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SnFHZ0RDbFBONUdSWjdZTlNPYVdyQ2pqTmVoRFF6cGdxY216Q1crV1pZMzk4?=
 =?utf-8?B?alJZaDVFMFhjMnFNQzE0WUxiWXhQRWU3cWNqVXUzMEhlSmN0TXZGcnp2U2ln?=
 =?utf-8?B?T2UwdDlhWlNleWlLTFMyb21yQnNOMU00Sm5Ob2huTUY2a2VKcFh0dW1jVkVC?=
 =?utf-8?B?djlTaGExZXhtaExzY1c4VSsxRUZBWU54S1pjMzA0eUJEb1R5L0R6aWZPdUxs?=
 =?utf-8?B?UlVETU05dEJHTGhqR09CdXJaVWNPS1lFcW0rcVhqc0VZN2J6NDliaWRHYW9q?=
 =?utf-8?B?SjVzd0lzMWQ1VVZyM080YldhK2FoV040MnNiV1hDcjk5cHREVFg4K3piVzU2?=
 =?utf-8?B?dmtvSmpOM3NKYXY0LzhQbFliN0UrZVNrV2NKaTlYR2ltdEVvaTdMcnpCZDlx?=
 =?utf-8?B?Y3YyaC82MWdtcE0yVm8wa1lPSnFLQjNrWDA2OFArdUYzZWxqRGpHNmdIRmhz?=
 =?utf-8?B?OWlacjd3Uys3R0lHelhZMmZFSSthQjg1ZFV1ZW9TSkp4NlkvNThCaWlQOS95?=
 =?utf-8?B?OHdyY1oxaGRQMWtBYm96aWNLYzFNWjNOOHl3Ym1rMEFobkJQcmhkOE9IbjJB?=
 =?utf-8?B?ckZmd00zRE9oc2JSOVcvdjR0bld0dWh2K3dNYllST2t0emVBUlZVZHArakxj?=
 =?utf-8?B?dm9JMDdEb1U2ckRVcytYSmt2bjNxcFBYNFhhZ2ZaY05KYnF2UnBrTmE4Qkho?=
 =?utf-8?B?dVhiUHFPa2x2eklXWmtLdHYvUGZBUEdGMjBoRUt6eG9sSmdaOVVwaGpxUnBu?=
 =?utf-8?B?NGhtTlJjU2lyQ0ZSVFJLTXdZTk9iZEV2SktuM2ZlMVhRck1DaTJMandqdlZ0?=
 =?utf-8?B?TThFaXhTdnd4Vml1cUV2UXZTMG5zUWlUT0Z6eU91WFl4NmJoQWNkOVJuR2hZ?=
 =?utf-8?B?KzhRQmVVcDNBZWNJckw1Z1I5T1NTZldrUWRCTGZRZWJERE1JWmxjektLVjJF?=
 =?utf-8?B?Vm9aaktqUjJaZ1FicWNZU25hcEtFaDd2OFcyKzVIb2x2bGROVUtwWGN6MlRr?=
 =?utf-8?B?TEkzUmZsRDMxemg4emdEcFRnclpBdjVDZWh1Q05qYWNRWXNQS25MSmtlTWF1?=
 =?utf-8?B?TExxR3pYVFk5T1FQeGs4S1RLRjZqSUFJT2hSSS94U3h3WmNtc2FSUFZzYitE?=
 =?utf-8?B?VnpZWHB6eVBSaFNtODdTYlFlS2s3bTY0NE93MFNhYVFzZjZrTGVvRDRQOGMw?=
 =?utf-8?B?QjNqRGQ3NWdJK0tITlBhTVZvYXhVNlZRWG4rRHhtd05HMmJIN1FKUjlIQWVz?=
 =?utf-8?B?eTg0NjM3RUo2RDdsSTg2SjN6ckhQWVppRWtGT3lyc0ViU2FvQytsL05GOXdC?=
 =?utf-8?B?aUxHbHhhbllMY1FYdVlVRlh2dkhXaUNPdlNFQjdlaDIyc1cvN2YzWFVEWENC?=
 =?utf-8?B?dEQxdE9LbldMcmZhYTFvQ3J1UC9TdGVsRkIxYUVmQVNqYzlZVUNOWUZOYlpX?=
 =?utf-8?B?RjRNVC9VcXFmeUlNdUVuY2x5cFlac0V6a3VZK1hEZzFRbFpuaDZxd3FYcWdK?=
 =?utf-8?B?eE9YbU0yQkhLZ0gvWE4xT2U4MzNZeEVnWFc5aDlqTkd6cXhEczNZVVRSSEM3?=
 =?utf-8?B?d1R4NnVLSHh1NUJ4L05Oc3dzbU5GVFdaY3lUU3VoOG5naFZUcTlLRk4vdExn?=
 =?utf-8?B?MEF5UVNUeHpzOTBoKzd3T1lOdVNSbThxTnBlbWRkSXpEZDF2NkxHaEpwVVNi?=
 =?utf-8?B?WWRiOW5Xd0VRUElkOUwvc0lJWWJNNU1hRHlLQ2lEdjcvTkgrTDhuWkxwdG1s?=
 =?utf-8?B?Mkk0YWNjZGZ1VmhOZFFFakwrUEVTTklHa1F6VldMRDBGaXJpUm5TUENVdlJu?=
 =?utf-8?B?ODR6NU1kcExmWS80Y0xWWkF0VVlBU2gzL2lWZmphRC9MRXB3dHlsRVFlV1Qx?=
 =?utf-8?B?VnR2MWh5RSswVDdjSkd6N1RIbk8xSGZLZng1c0lBL3E1ZlN2Z2JGbTEwa2Q4?=
 =?utf-8?B?SE14VDl0YnJqMGVxV1hTNE5zL2ZNWjYxUk0xT1pDR0hXQ3IrM2NneTc5SXIz?=
 =?utf-8?B?cHpTSkQ4SndRaFo0V2tZZWlXVEhMNDU4cXo4RjUyVzNPWHFsa3ZmaHcybjk2?=
 =?utf-8?B?OVdwZ0FrMGgxZFhsMWozS3RRT2VmakNJcXVvWFhmbklpdW9LUGQ4K3VQelFl?=
 =?utf-8?B?RjU3QUQzZkhHR3I4UGZFbUxvRURVWGprbzBEL2M4bDR5bC9sMjFrWU1Hemw5?=
 =?utf-8?B?ZXFnL2c5R0ZuY0VWTFZscWU2K2NqcXEvS2g0S1l0WU5rVW9oQjE4L25RbXQ3?=
 =?utf-8?B?NkJhOUJyaThlUTVSakp0MXlzRnNYMzVPZ2hld1FYUnFib3VNVEV6djlEU0Rm?=
 =?utf-8?B?OGJsRnRyYU5CZWxSSTFVclpIRVRVRnloUElKd3dFTDNHems3MUNHUT09?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b7e48fe-d11a-499a-51c6-08da0f364e8c
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Mar 2022 14:38:34.8221
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: b7RQoQzMU2gqmrU7uah7IkZ2a74Ky0DDsonmxsiG1UEviwMdAutHr+JrOuIyVLjFFD6RkTtZ6wM9a+4i9Zhaew==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB5338
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KQhRJ21slz3j7v
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=Gd48kh88;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=u3AeYTdt;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=077da0f8a=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=077da0f8a=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.55 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=077da0f8a=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-0.91)[-0.906];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sat, Mar 26, 2022 at 02:08:06PM +0200, Ze Dupsys wrote:
> On 2022.03.26. 11:11, Roger Pau Monné wrote:
> >
> > Hm, do you think you could upload (or attach) your
> > /usr/lib/debug/boot/kernel/kernel.debug and provide an updated panic
> > trace using that same exact kernel?
> 
> Yes, it is just too big for email attachment.
> Uploaded at: https://files.fm/f/mp3v3qa22
> 
> This time i starved Dom0 of RAM(2G) to speed panic up. Panic trace it the
> same.
> 
> Trace:
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 04
> fault virtual address	= 0x22710028
> fault code		= supervisor read data, page not present
> instruction pointer	= 0x20:0xffffffff80c6a2b2
> stack pointer	        = 0x28:0xfffffe009e486b30
> frame pointer	        = 0x28:0xfffffe009e486b30
> code segment		= base 0x0, limit 0xfffff, type 0x1b
> 			= DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags	= interrupt enabled, resume, IOPL = 0
> current process		= 3995 (devmatch)
> trap number		= 12
> panic: page fault
> cpuid = 2
> time = 1648293768
> KDB: stack backtrace:
> #0 0xffffffff80c7c285 at kdb_backtrace+0x65
> #1 0xffffffff80c2e2e1 at vpanic+0x181
> #2 0xffffffff80c2e153 at panic+0x43
> #3 0xffffffff810c8b97 at trap+0xba7
> #4 0xffffffff810c8bef at trap+0xbff
> #5 0xffffffff810c8243 at trap+0x253
> #6 0xffffffff810a0848 at calltrap+0x8
> #7 0xffffffff80c86ed1 at rman_is_region_manager+0x241
> #8 0xffffffff80c3eb41 at sbuf_new_for_sysctl+0x101
> #9 0xffffffff80c3df8c at kernel_sysctl+0x3ec
> #10 0xffffffff80c3e603 at userland_sysctl+0x173
> #11 0xffffffff80c3e44f at sys___sysctl+0x5f
> #12 0xffffffff810c949c at amd64_syscall+0x10c
> #13 0xffffffff810a115b at Xfast_syscall+0xfb
> Uptime: 10m19s

It's weird, because here you get a page fault, but there are also
traces with:

general protection fault while in kernel mode
cpuid = 3; a(d8) Scan for VGA option rom
pic id = 06
instruction pointer     = 0x20:0xffffffff810c5d64
stack pointer           = 0x28:0xfffffe00a20fe990
frame pointer           = 0x28:0xfffffe00a20fe990
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 8998 (devmatch)
trap number             = 9
panic: general protection fault
cpuid = 3
time = 1647416577
KDB: stack backtrace:
#0 0xffffffff80c7ca05 at kdb_backtrace+0x65
#1 0xffffffff80c2ea11 at vpanic+0x181
#2 0xffffffff80c2e883 at panic+0x43
#3 0xffffffff810c9b97 at trap+0xba7
#4 0xffffffff810c907b at trap+0x8b
#5 0xffffffff810a0dc8 at calltrap+0x8
#6 0xffffffff80c83067 at kvprintf+0x1007
#7 0xffffffff80c83df9 at snprintf+0x59
#8 0xffffffff80c8768b at rman_is_region_manager+0x27b
#9 0xffffffff80c3f271 at sbuf_new_for_sysctl+0x101
#10 0xffffffff80c3e6bc at kernel_sysctl+0x3ec
#11 0xffffffff80c3ed33 at userland_sysctl+0x173
#12 0xffffffff80c3eb7f at sys___sysctl+0x5f
#13 0xffffffff810ca49c at amd64_syscall+0x10c
#14 0xffffffff810a16db at Xfast_syscall+0xfb

That show a general protection fault instead of a page fault.

I've built an hypervisor with debug enabled for you, it's at:

https://people.freebsd.org/~royger/xen-debug

This is the same as the one in ports, just build with debug=y. If you
can place it in /boot/ and change your xen_kernel to:

xen_kernel="/boot/xen-debug"

It might provide some additional info.

I've also noticed it seems to always be 'devmatch' the process that
triggers the panic.

> 
> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line
> -e /usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:844
> /usr/src/sys/amd64/amd64/trap.c:944
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/exception.S:292
> /usr/src/sys/kern/subr_rman.c:0

I've been able to get a better trace with gdb and your debug symbols,
and this is:

(gdb) info line *0xffffffff80c6a2b2
Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address 0xffffffff80c6a2b2 
   and ends at 0xffffffff80c6a2b6 .
(gdb) info line *0xffffffff80c86ed1
Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address 0xffffffff80c86ecc 
   and ends at 0xffffffff80c86ed5 .

The page fault happens exactly at:

https://cgit.freebsd.org/src/tree/sys/kern/subr_bus.c?h=stable/13#n1386

Which is called from

https://cgit.freebsd.org/src/tree/sys/kern/subr_rman.c?h=stable/13#n1052

I'm trying to figure out how the device could be removed or
disconnected from the rman. I will try to create a patch to catch the
device that leaves rman regions when destroyed/removed.

Thanks, Roger.

From nobody Sat Mar 26 15:56:21 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 754231A46A80
	for ; Sat, 26 Mar 2022 15:56:42 +0000 (UTC)
	(envelope-from royger@gmail.com)
Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KQk953l8Zz3vR2
	for ; Sat, 26 Mar 2022 15:56:41 +0000 (UTC)
	(envelope-from royger@gmail.com)
Received: by mail-wr1-x42d.google.com with SMTP id h23so14498306wrb.8
        for ; Sat, 26 Mar 2022 08:56:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=fXQNLhXYbwRZ9WO7T+BhVV9+0Up77RvoHt23kV18qJk=;
        b=pOBjalGLCgWTAe0KlJjHb9CxHhGNPoF67b7cFFHqNzpMu7n8EEspBHSt86y8uoivYN
         e0GMin6guK/9hPp2Qi/fS2klpd32Qq+pgtXukBR2AyfdWDy2vJaMUR+3T91Lnz9OEFVZ
         UBJrFzVU8A+tpP5QKsR6y8aF91SyZOmFBXPTiBc7HJAulWmgOosYeJ1xuTfUVvahqCCR
         8OSpcvtCpg0ZM5qj9d6Z/CP0FRBA0HT/jZ61Hh5SDuC7wirqZbI9vQ3itsDtXQRVFBWy
         qKjkxTCL2HfsvIlL+F2Re0bgw49lOEdkHyXAN6XwiGRIc7TPL85M0iFZ3+wpOAeCH/lE
         ak0g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=fXQNLhXYbwRZ9WO7T+BhVV9+0Up77RvoHt23kV18qJk=;
        b=f/GPiX2HV/I3jYXyAxaFsIWiCG32U0xUFQvPVwrccOIc6d4318ndly4rXtUaRYGV7t
         zXZEgbsfWiPGhCF1Cy1hbgYV4+j9hvI9BpvYTl4YN6g4O2Ayt52h+I1qFuiiABi3vWVl
         yxB/ZIpkaEK/QbP2t3cIkiyVnZPiXv741j3EPVloWh3FqX7fU6Z22R1fa31+MlZsZkK7
         DkZL/FUKYojwbDoziojk3WVECfR4Z+1RKpLR69ThcP7PnjVoogfs3CQcCrPRmzp5XW1G
         n/xUUQ33B14m85c9YAx2ulUJd7Q2ckEaOlukMje5Xb2rq/GXNwGf6GNb838u+rNeFmqY
         SjSQ==
X-Gm-Message-State: AOAM5334A1NcEsw5x4NPF5S5b3BLLrqq175KfmUvjnSpBRJ/w+xj3TSQ
	SfgAxO6MhYj76azdxnPOXnzO6R80jriTtz+wmac=
X-Google-Smtp-Source: ABdhPJxiFe+1lJNWil3Cq5CXwiktl5VzBVEi6d2/PgXW/oZTTQ9CYIZ62RK5Y+AlK0MyR/IHqrzN6ks2Bl//LVV81bg=
X-Received: by 2002:adf:ed50:0:b0:203:da73:e0fd with SMTP id
 u16-20020adfed50000000b00203da73e0fdmr14169616wro.516.1648310193971; Sat, 26
 Mar 2022 08:56:33 -0700 (PDT)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
References:  
  
  <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
  <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
  <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
In-Reply-To: 
From: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= 
Date: Sat, 26 Mar 2022 16:56:21 +0100
Message-ID: 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: Ze Dupsys 
Cc: freebsd-xen@freebsd.org, Brian Buhrow 
Content-Type: multipart/mixed; boundary="00000000000097864305db211f9b"
X-Rspamd-Queue-Id: 4KQk953l8Zz3vR2
X-Spamd-Bar: -
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=pOBjalGL;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of royger@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) smtp.mailfrom=royger@gmail.com
X-Spamd-Result: default: False [-1.35 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 HAS_ATTACHMENT(0.00)[];
	 MID_RHS_MATCH_FROMTLD(0.00)[];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FREEMAIL_TO(0.00)[gmail.com];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 R_MIXED_CHARSET(0.56)[subject];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FREEFALL_USER(0.00)[royger];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(1.00)[0.999];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 MIME_UNKNOWN(0.10)[application/x-patch];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::42d:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

--00000000000097864305db211f9b
Content-Type: multipart/alternative; boundary="00000000000097864105db211f99"

--00000000000097864105db211f99
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

El ds., 26 de mar=C3=A7 2022, 15:39, Roger Pau Monn=C3=A9  va
escriure:

> On Sat, Mar 26, 2022 at 02:08:06PM +0200, Ze Dupsys wrote:
> > On 2022.03.26. 11:11, Roger Pau Monn=C3=A9 wrote:
> > >
> > > Hm, do you think you could upload (or attach) your
> > > /usr/lib/debug/boot/kernel/kernel.debug and provide an updated panic
> > > trace using that same exact kernel?
> >
> > Yes, it is just too big for email attachment.
> > Uploaded at: https://files.fm/f/mp3v3qa22
> >
> > This time i starved Dom0 of RAM(2G) to speed panic up. Panic trace it t=
he
> > same.
> >
> > Trace:
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 2; apic id =3D 04
> > fault virtual address =3D 0x22710028
> > fault code            =3D supervisor read data, page not present
> > instruction pointer   =3D 0x20:0xffffffff80c6a2b2
> > stack pointer         =3D 0x28:0xfffffe009e486b30
> > frame pointer         =3D 0x28:0xfffffe009e486b30
> > code segment          =3D base 0x0, limit 0xfffff, type 0x1b
> >                       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags      =3D interrupt enabled, resume, IOPL =3D 0
> > current process               =3D 3995 (devmatch)
> > trap number           =3D 12
> > panic: page fault
> > cpuid =3D 2
> > time =3D 1648293768
> > KDB: stack backtrace:
> > #0 0xffffffff80c7c285 at kdb_backtrace+0x65
> > #1 0xffffffff80c2e2e1 at vpanic+0x181
> > #2 0xffffffff80c2e153 at panic+0x43
> > #3 0xffffffff810c8b97 at trap+0xba7
> > #4 0xffffffff810c8bef at trap+0xbff
> > #5 0xffffffff810c8243 at trap+0x253
> > #6 0xffffffff810a0848 at calltrap+0x8
> > #7 0xffffffff80c86ed1 at rman_is_region_manager+0x241
> > #8 0xffffffff80c3eb41 at sbuf_new_for_sysctl+0x101
> > #9 0xffffffff80c3df8c at kernel_sysctl+0x3ec
> > #10 0xffffffff80c3e603 at userland_sysctl+0x173
> > #11 0xffffffff80c3e44f at sys___sysctl+0x5f
> > #12 0xffffffff810c949c at amd64_syscall+0x10c
> > #13 0xffffffff810a115b at Xfast_syscall+0xfb
> > Uptime: 10m19s
>
> It's weird, because here you get a page fault, but there are also
> traces with:
>
> general protection fault while in kernel mode
> cpuid =3D 3; a(d8) Scan for VGA option rom
> pic id =3D 06
> instruction pointer     =3D 0x20:0xffffffff810c5d64
> stack pointer           =3D 0x28:0xfffffe00a20fe990
> frame pointer           =3D 0x28:0xfffffe00a20fe990
> code segment            =3D base 0x0, limit 0xfffff, type 0x1b
>                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
> current process         =3D 8998 (devmatch)
> trap number             =3D 9
> panic: general protection fault
> cpuid =3D 3
> time =3D 1647416577
> KDB: stack backtrace:
> #0 0xffffffff80c7ca05 at kdb_backtrace+0x65
> #1 0xffffffff80c2ea11 at vpanic+0x181
> #2 0xffffffff80c2e883 at panic+0x43
> #3 0xffffffff810c9b97 at trap+0xba7
> #4 0xffffffff810c907b at trap+0x8b
> #5 0xffffffff810a0dc8 at calltrap+0x8
> #6 0xffffffff80c83067 at kvprintf+0x1007
> #7 0xffffffff80c83df9 at snprintf+0x59
> #8 0xffffffff80c8768b at rman_is_region_manager+0x27b
> #9 0xffffffff80c3f271 at sbuf_new_for_sysctl+0x101
> #10 0xffffffff80c3e6bc at kernel_sysctl+0x3ec
> #11 0xffffffff80c3ed33 at userland_sysctl+0x173
> #12 0xffffffff80c3eb7f at sys___sysctl+0x5f
> #13 0xffffffff810ca49c at amd64_syscall+0x10c
> #14 0xffffffff810a16db at Xfast_syscall+0xfb
>
> That show a general protection fault instead of a page fault.
>
> I've built an hypervisor with debug enabled for you, it's at:
>
> https://people.freebsd.org/~royger/xen-debug
>
> This is the same as the one in ports, just build with debug=3Dy. If you
> can place it in /boot/ and change your xen_kernel to:
>
> xen_kernel=3D"/boot/xen-debug"
>
> It might provide some additional info.
>
> I've also noticed it seems to always be 'devmatch' the process that
> triggers the panic.
>
> >
> > cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs
> addr2line
> > -e /usr/lib/debug/boot/kernel/kernel.debug
> > /usr/src/sys/kern/subr_kdb.c:443
> > /usr/src/sys/kern/kern_shutdown.c:0
> > /usr/src/sys/kern/kern_shutdown.c:844
> > /usr/src/sys/amd64/amd64/trap.c:944
> > /usr/src/sys/amd64/amd64/trap.c:0
> > /usr/src/sys/amd64/amd64/trap.c:0
> > /usr/src/sys/amd64/amd64/exception.S:292
> > /usr/src/sys/kern/subr_rman.c:0
>
> I've been able to get a better trace with gdb and your debug symbols,
> and this is:
>
> (gdb) info line *0xffffffff80c6a2b2
> Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address
> 0xffffffff80c6a2b2 
>    and ends at 0xffffffff80c6a2b6 .
> (gdb) info line *0xffffffff80c86ed1
> Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address
> 0xffffffff80c86ecc 
>    and ends at 0xffffffff80c86ed5 .
>
> The page fault happens exactly at:
>
> https://cgit.freebsd.org/src/tree/sys/kern/subr_bus.c?h=3Dstable/13#n1386
>
> Which is called from
>
> https://cgit.freebsd.org/src/tree/sys/kern/subr_rman.c?h=3Dstable/13#n105=
2
>
> I'm trying to figure out how the device could be removed or
> disconnected from the rman. I will try to create a patch to catch the
> device that leaves rman regions when destroyed/removed.


Replying from my phone so the format will likely be mangled.

I think I've found at least one issue with blkback leaking resources on
destroy if the ring was not connected. Could you give the following patch a
try? I've just build tested it, so can't guarantee it will work.

Thanks, Roger.

--00000000000097864105db211f99
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


El ds., 26 de mar=C3=A7 2022, 15:39, Roger Pau Monn=C3=A9 <=
roger.pau@citrix.com> va escriure:
On Sat, Mar 26, 2022 at 02:08:06PM +0200, Ze Dupsys wro=
te:

> On 2022.03.26. 11:11, Roger Pau Monn=C3=A9 wrote:

> >

> > Hm, do you think you could upload (or attach) your

> > /usr/lib/debug/boot/kernel/kernel.debug and provide an updated pa=
nic

> > trace using that same exact kernel?

> 

> Yes, it is just too big for email attachment.

> Uploaded at: https://files.fm/f/mp=
3v3qa22

> 

> This time i starved Dom0 of RAM(2G) to speed panic up. Panic trace it =
the

> same.

> 

> Trace:

> Fatal trap 12: page fault while in kernel mode

> cpuid =3D 2; apic id =3D 04

> fault virtual address =3D 0x22710028

> fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor rea=
d data, page not present

> instruction pointer=C2=A0 =C2=A0=3D 0x20:0xffffffff80c6a2b2

> stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe009e48=
6b30

> frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe009e48=
6b30

> code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base 0x0, limit 0xf=
ffff, type 0x1b

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0=3D DPL 0, pres 1, long 1, def32 0, gran 1

> processor eflags=C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, IO=
PL =3D 0

> current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=3D 3995 (devmatch)

> trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12

> panic: page fault

> cpuid =3D 2

> time =3D 1648293768

> KDB: stack backtrace:

> #0 0xffffffff80c7c285 at kdb_backtrace+0x65

> #1 0xffffffff80c2e2e1 at vpanic+0x181

> #2 0xffffffff80c2e153 at panic+0x43

> #3 0xffffffff810c8b97 at trap+0xba7

> #4 0xffffffff810c8bef at trap+0xbff

> #5 0xffffffff810c8243 at trap+0x253

> #6 0xffffffff810a0848 at calltrap+0x8

> #7 0xffffffff80c86ed1 at rman_is_region_manager+0x241

> #8 0xffffffff80c3eb41 at sbuf_new_for_sysctl+0x101

> #9 0xffffffff80c3df8c at kernel_sysctl+0x3ec

> #10 0xffffffff80c3e603 at userland_sysctl+0x173

> #11 0xffffffff80c3e44f at sys___sysctl+0x5f

> #12 0xffffffff810c949c at amd64_syscall+0x10c

> #13 0xffffffff810a115b at Xfast_syscall+0xfb

> Uptime: 10m19s



It's weird, because here you get a page fault, but there are also

traces with:



general protection fault while in kernel mode

cpuid =3D 3; a(d8) Scan for VGA option rom

pic id =3D 06

instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff810c5d64

stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00a2=
0fe990

frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00a2=
0fe990

code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base 0x0, limit 0=
xfffff, type 0x1b

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1

processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, =
IOPL =3D 0

current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 8998 (devmatch)

trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 9

panic: general protection fault

cpuid =3D 3

time =3D 1647416577

KDB: stack backtrace:

#0 0xffffffff80c7ca05 at kdb_backtrace+0x65

#1 0xffffffff80c2ea11 at vpanic+0x181

#2 0xffffffff80c2e883 at panic+0x43

#3 0xffffffff810c9b97 at trap+0xba7

#4 0xffffffff810c907b at trap+0x8b

#5 0xffffffff810a0dc8 at calltrap+0x8

#6 0xffffffff80c83067 at kvprintf+0x1007

#7 0xffffffff80c83df9 at snprintf+0x59

#8 0xffffffff80c8768b at rman_is_region_manager+0x27b

#9 0xffffffff80c3f271 at sbuf_new_for_sysctl+0x101

#10 0xffffffff80c3e6bc at kernel_sysctl+0x3ec

#11 0xffffffff80c3ed33 at userland_sysctl+0x173

#12 0xffffffff80c3eb7f at sys___sysctl+0x5f

#13 0xffffffff810ca49c at amd64_syscall+0x10c

#14 0xffffffff810a16db at Xfast_syscall+0xfb



That show a general protection fault instead of a page fault.



I've built an hypervisor with debug enabled for you, it's at:



https://people.freebsd.=
org/~royger/xen-debug



This is the same as the one in ports, just build with debug=3Dy. If you

can place it in /boot/ and change your xen_kernel to:



xen_kernel=3D"/boot/xen-debug"



It might provide some additional info.



I've also noticed it seems to always be 'devmatch' the process =
that

triggers the panic.



> 

> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//=
9; | xargs addr2line

> -e /usr/lib/debug/boot/kernel/kernel.debug

> /usr/src/sys/kern/subr_kdb.c:443

> /usr/src/sys/kern/kern_shutdown.c:0

> /usr/src/sys/kern/kern_shutdown.c:844

> /usr/src/sys/amd64/amd64/trap.c:944

> /usr/src/sys/amd64/amd64/trap.c:0

> /usr/src/sys/amd64/amd64/trap.c:0

> /usr/src/sys/amd64/amd64/exception.S:292

> /usr/src/sys/kern/subr_rman.c:0



I've been able to get a better trace with gdb and your debug symbols,
and this is:



(gdb) info line *0xffffffff80c6a2b2

Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address 0xf=
fffffff80c6a2b2 <device_get_name+18>

=C2=A0 =C2=A0and ends at 0xffffffff80c6a2b6 <device_get_name+22>.

(gdb) info line *0xffffffff80c86ed1

Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address 0x=
ffffffff80c86ecc <sysctl_rman+540>

=C2=A0 =C2=A0and ends at 0xffffffff80c86ed5 <sysctl_rman+549>.



The page fault happens exactly at:



https://cgit.freebsd.org/src/tree/sys/kern/subr_bus.c?h=3Dstable/13#n=
1386



Which is called from



https://cgit.freebsd.org/src/tree/sys/kern/subr_rman.c?h=3Dstable/13=
#n1052



I'm trying to figure out how the device could be removed or

disconnected from the rman. I will try to create a patch to catch the

device that leaves rman regions when destroyed/removed.
<=
/div>

Replying from my phone s=
o the format will likely be mangled.=C2=A0

I think I've found at least one issue with blkback l=
eaking resources on destroy if the ring was not connected. Could you give t=
he following patch a try? I've just build tested it, so can't guara=
ntee it will work.=C2=A0

Thanks, Roger.=C2=A0


--00000000000097864105db211f99--
--00000000000097864305db211f9b
Content-Type: application/x-patch; name="blkback.patch"
Content-Disposition: attachment; filename="blkback.patch"
Content-Transfer-Encoding: base64
Content-ID: <17fc6eb5f8e17e280a21>
X-Attachment-Id: 17fc6eb5f8e17e280a21

ZGlmZiAtLWdpdCBhL3N5cy9kZXYveGVuL2Jsa2JhY2svYmxrYmFjay5jIGIvc3lzL2Rldi94ZW4v
YmxrYmFjay9ibGtiYWNrLmMKaW5kZXggMzM0MTQyOTViZjUuLjY2NGY1MmE3NGU3IDEwMDY0NAot
LS0gYS9zeXMvZGV2L3hlbi9ibGtiYWNrL2Jsa2JhY2suYworKysgYi9zeXMvZGV2L3hlbi9ibGti
YWNrL2Jsa2JhY2suYwpAQCAtMjc4MSw5ICsyNzgxLDYgQEAgeGJiX2Rpc2Nvbm5lY3Qoc3RydWN0
IHhiYl9zb2Z0YyAqeGJiKQogCiAJRFBSSU5URigiXG4iKTsKIAotCWlmICgoeGJiLT5mbGFncyAm
IFhCQkZfUklOR19DT05ORUNURUQpID09IDApCi0JCXJldHVybiAoMCk7Ci0KIAltdHhfdW5sb2Nr
KCZ4YmItPmxvY2spOwogCXhlbl9pbnRyX3VuYmluZCgmeGJiLT54ZW5faW50cl9oYW5kbGUpOwog
CXRhc2txdWV1ZV9kcmFpbih4YmItPmlvX3Rhc2txdWV1ZSwgJnhiYi0+aW9fdGFzayk7IAo=
--00000000000097864305db211f9b--

From nobody Sat Mar 26 22:38:00 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9C2C61A3607E
	for ; Sat, 26 Mar 2022 22:38:50 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KQv545gdRz4T4c
	for ; Sat, 26 Mar 2022 22:38:48 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x231.google.com with SMTP id g24so14578916lja.7
        for ; Sat, 26 Mar 2022 15:38:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language;
        bh=GdlyZNqWEkU7q0B9icMK/yZrrPJBAqUW2gfR/0iyveY=;
        b=QajrhXb5IZVwIaM30u1M6pBp9Xb5YODyW8s6iZuB0h+IoeejQoVhGdc5Elm41FoKxH
         9Us9m0RXy9QNDZTjo1u24ZRGuloKqKfEKaM6temtkuZTZ09fJNHewi+4hX7YOz4rTPXK
         ofiMoNBZNzBe9PXRVSfbLyTjF4tI/r+f75s81/H7ZnGQUNNWsqJgLsq+EvUSYZ6qMg3K
         MWUWibyZvPYbzVGRQ5dPDS114S/ksznZ0vaZas8AQ8hHirS4eCNwtvOw7as2//qGd3Op
         pSFxCF9vumfm4dNXlEgQ7yH0cymYZMTt6tcaXh4gp0s2IjRM7yxLS996jMdAIGfuGkjK
         KKJw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language;
        bh=GdlyZNqWEkU7q0B9icMK/yZrrPJBAqUW2gfR/0iyveY=;
        b=IB8DI/GMq2FHGb2mpOxGBXXtOfZhx7zxQbl3JS/6O6SzbQ17ToHBZct/R7AfIdky2l
         zZnMw3t86zRgf2XZegEEpstewDwe8E+ch99/MpIcTPRU+JKSNami1BbkaMwapTL82vmP
         9ATTL31QPgNakxaPCW2BoplF8l+tz8O0UmQhISAdm4q3wHlU0Q8oJFpvp9FvV7LoSmi8
         eaFCfo6Vgj3sN+455g7SpPjWs9MV/fRacXBo204u6Ue8f9/mpSZTd24gZMepO30vteEu
         Bbwt4BPd/XG3o5Q6beq1MsIi/LkASpnw5V1G1jEvI5dTyCSZVwWLVwuc6rM7WtdkFofx
         mcVQ==
X-Gm-Message-State: AOAM532khOXJltsnbKRA64NMOAH5cUnUECe+720Q7NuWITlenqjejXVH
	vqBn/iLNp1l7FznL5fCRAls=
X-Google-Smtp-Source: ABdhPJxIoTw8ied9S8yyaq7S9xGpQB2JnJLYbhvlhTrHwmYdHN6enaBBdJK26S+hOXlTOx5UYfnKxQ==
X-Received: by 2002:a2e:b04c:0:b0:247:e829:9508 with SMTP id d12-20020a2eb04c000000b00247e8299508mr13974451ljl.353.1648334320606;
        Sat, 26 Mar 2022 15:38:40 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id f23-20020a2e9e97000000b0024921bcf06bsm1146797ljk.57.2022.03.26.15.38.38
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Sat, 26 Mar 2022 15:38:39 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References:  
 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
From: Ze Dupsys 
Message-ID: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
Date: Sun, 27 Mar 2022 00:38:00 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/mixed;
 boundary="------------6658CE0D501BB4674047759C"
Content-Language: en-US
X-Rspamd-Queue-Id: 4KQv545gdRz4T4c
X-Spamd-Bar: ++
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=QajrhXb5;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::231 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [2.03 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36];
	 HAS_ATTACHMENT(0.00)[];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 MIME_BASE64_TEXT_BOGUS(1.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 MIME_BASE64_TEXT(0.10)[];
	 CTYPE_MIXED_BOGUS(1.00)[];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.1:email,0.0.0.3:email,0.0.0.4:email,0.0.0.2:email,0.0.0.0:email];
	 NEURAL_HAM_LONG(-1.00)[-0.997];
	 MIME_GOOD(-0.10)[multipart/mixed,text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_SPAM_MEDIUM(0.99)[0.991];
	 NEURAL_SPAM_SHORT(0.93)[0.931];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::231:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

This is a multi-part message in MIME format.
--------------6658CE0D501BB4674047759C
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2022.03.26. 16:38, Roger Pau Monné wrote:
 > ..
 > It's weird, because here you get a page fault, but there are also
 > traces with:
 > ..
 > general protection fault while in kernel mode
 > ..
 > That show a general protection fault instead of a page fault.

Yes indeed, i had not noticed this. Grepped across 34 stored panic log 
files, i see that 28 are page fault, 4 are general protection fault, 2 
other. I though maybe RAM size influences this, but page faults have 2G, 
4G, 6G, 8G Dom0, general protection faults have 2G, 4G, 8G.

I have no idea what triggers what, since stress tests and command line 
args are more or less the same. Builds are different with patches, some 
debug info, etc. Almost all panic traces have "rman_is_region_manager" 
in mid, actually looking all of them together seemed interesting. I'll 
attach unique panic traces, since some included snprintf, kvprintf as 
well, maybe helpful. Unfortunately i do not know which version and what 
patches were applied.


 > I've also noticed it seems to always be 'devmatch' the process that
 > triggers the panic.

Yes, it seems to be the case most of the time. There are 3 cases when
process is "xbbd* taskq". 2 cases with 2G RAM, 1 with 6G.


 > I've been able to get a better trace with gdb and your debug symbols,
 > and this is:
 >
 > (gdb) info line *0xffffffff80c6a2b2
 > Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address 
0xffffffff80c6a2b2 
 >     and ends at 0xffffffff80c6a2b6 .
 > (gdb) info line *0xffffffff80c86ed1
 > Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address 
0xffffffff80c86ecc 
 >     and ends at 0xffffffff80c86ed5 .

This is a nice find!


 > I'm trying to figure out how the device could be removed or
 > disconnected from the rman. I will try to create a patch to catch the
 > device that leaves rman regions when destroyed/removed.

Okay, i'll apply when it will be possible.

I did run xen-debug on system with applied blkback.patch as you sent in 
next message to this.

System had panic with new trace:
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address	= 0xa4
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80c90ed0
stack pointer	        = 0x28:0xfffffe0051927ab0
frame pointer	        = 0x28:0xfffffe0051927ad0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 16 (xenwatch)
trap number		= 12
panic: page fault
cpuid = 1
time = 1648331592
KDB: stack backtrace:
#0 0xffffffff80c7c275 at kdb_backtrace+0x65
#1 0xffffffff80c2e2d1 at vpanic+0x181
#2 0xffffffff80c2e143 at panic+0x43
#3 0xffffffff810c8b97 at trap+0xba7
#4 0xffffffff810c8bef at trap+0xbff
#5 0xffffffff810c8243 at trap+0x253
#6 0xffffffff810a0838 at calltrap+0x8
#7 0xffffffff80a98515 at xbd_instance_create+0x7895
#8 0xffffffff80a98462 at xbd_instance_create+0x77e2
#9 0xffffffff80a9619b at xbd_instance_create+0x551b
#10 0xffffffff80f95c54 at xenbusb_localend_changed+0x7c4
#11 0xffffffff80ab0ef4 at xs_unlock+0x704
#12 0xffffffff80beaede at fork_exit+0x7e
#13 0xffffffff810a18ae at fork_trampoline+0xe

cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs 
addr2line -e /usr/lib/debug/boot/kernel/kernel.debug

/usr/src/sys/kern/subr_kdb.c:443
/usr/src/sys/kern/kern_shutdown.c:0
/usr/src/sys/kern/kern_shutdown.c:844
/usr/src/sys/amd64/amd64/trap.c:944
/usr/src/sys/amd64/amd64/trap.c:0
/usr/src/sys/amd64/amd64/trap.c:0
/usr/src/sys/amd64/amd64/exception.S:292
/usr/src/sys/dev/xen/blkback/blkback.c:2789
/usr/src/sys/dev/xen/blkback/blkback.c:3431
/usr/src/sys/dev/xen/blkback/blkback.c:3912
/usr/src/sys/xen/xenbus/xenbusb_back.c:238
/usr/src/sys/dev/xen/xenstore/xenstore.c:1007
/usr/src/sys/kern/kern_fork.c:1099
/usr/src/sys/amd64/amd64/exception.S:1091

Full serial log in attachment.

Thanks.

--------------6658CE0D501BB4674047759C
Content-Type: text/plain; charset=UTF-8;
 name="general_fault_agg.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="general_fault_agg.log"

PT09PSBDT1VOVDogMQ0KRmF0YWwgdHJhcCA5OiBnZW5lcmFsIHByb3RlY3Rpb24gZmF1bHQg
d2hpbGUgaW4ga2VybmVsIG1vZGUNCmNwdWlkID0gMDsgYXBpYyBpZCA9IDAwDQppbnN0cnVj
dGlvbiBwb2ludGVyCT0gMHgyMDoweGZmZmZmZmZmODBjNDU4OTINCnN0YWNrIHBvaW50ZXIJ
ICAgICAgICA9IDB4Mjg6MHhmZmZmZmUwMGQyZDJiOTMwDQpmcmFtZSBwb2ludGVyCSAgICAg
ICAgPSAweDI4OjB4ZmZmZmZlMDBkMmQyYjkzMA0KY29kZSBzZWdtZW50CQk9IGJhc2UgMHgw
LCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWINCgkJCT0gRFBMIDAsIHByZXMgMSwgbG9uZyAx
LCBkZWYzMiAwLCBncmFuIDENCnByb2Nlc3NvciBlZmxhZ3MJPSBpbnRlcnJ1cHQgZW5hYmxl
ZCwgcmVzdW1lLCBJT1BMID0gMA0KY3VycmVudCBwcm9jZXNzCQk9IDEwOTg0IChkZXZtYXRj
aCkNCnRyYXAgbnVtYmVyCQk9IDkNCnBhbmljOiBnZW5lcmFsIHByb3RlY3Rpb24gZmF1bHQN
CmNwdWlkID0gMA0KdGltZSA9IDE2NDYzMDU2ODANCktEQjogc3RhY2sgYmFja3RyYWNlOg0K
IzAgMHhmZmZmZmZmZjgwYzU3NTI1IGF0IGtkYl9iYWNrdHJhY2UrMHg2NQ0KIzEgMHhmZmZm
ZmZmZjgwYzA5ZjAxIGF0IHZwYW5pYysweDE4MQ0KIzIgMHhmZmZmZmZmZjgwYzA5ZDczIGF0
IHBhbmljKzB4NDMNCiMzIDB4ZmZmZmZmZmY4MTA4YjFhNyBhdCB0cmFwKzB4YmM3DQojNCAw
eGZmZmZmZmZmODEwOGE2NmUgYXQgdHJhcCsweDhlDQojNSAweGZmZmZmZmZmODEwNjFiMTgg
YXQgY2FsbHRyYXArMHg4DQojNiAweGZmZmZmZmZmODBjNjIwMTEgYXQgcm1hbl9pc19yZWdp
b25fbWFuYWdlcisweDI0MQ0KIzcgMHhmZmZmZmZmZjgwYzFhMDUxIGF0IHNidWZfbmV3X2Zv
cl9zeXNjdGwrMHgxMDENCiM4IDB4ZmZmZmZmZmY4MGMxOTQ5YyBhdCBrZXJuZWxfc3lzY3Rs
KzB4NDNjDQojOSAweGZmZmZmZmZmODBjMTliMTMgYXQgdXNlcmxhbmRfc3lzY3RsKzB4MTcz
DQojMTAgMHhmZmZmZmZmZjgwYzE5OTVmIGF0IHN5c19fX3N5c2N0bCsweDVmDQojMTEgMHhm
ZmZmZmZmZjgxMDhiYWFjIGF0IGFtZDY0X3N5c2NhbGwrMHgxMGMNCiMxMiAweGZmZmZmZmZm
ODEwNjI0M2UgYXQgWGZhc3Rfc3lzY2FsbCsweGZlDQpVcHRpbWU6IDFoMTVtNDZzDQoNCg0K
PT09PSBDT1VOVDogMw0KRmF0YWwgdHJhcCA5OiBnZW5lcmFsIHByb3RlY3Rpb24gZmF1bHQg
d2hpbGUgaW4ga2VybmVsIG1vZGUNCmNwdWlkID0gMDsgYXBpYyBpZCA9IDAwDQppbnN0cnVj
dGlvbiBwb2ludGVyCT0gMHgyMDoweGZmZmZmZmZmODBkMDcyOGYNCnN0YWNrIHBvaW50ZXIJ
ICAgICAgICA9IDB4Mjg6MHhmZmZmZmUwMGExN2VhNzkwDQpmcmFtZSBwb2ludGVyCSAgICAg
ICAgPSAweDI4OjB4ZmZmZmZlMDBhMTdlYTc5MA0KY29kZSBzZWdtZW50CQk9IGJhc2UgMHgw
LCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWINCgkJCT0gRFBMIDAsIHByZXMgMSwgbG9uZyAx
LCBkZWYzMiAwLCBncmFuIDENCnByb2Nlc3NvciBlZmxhZ3MJPSBpbnRlcnJ1cHQgZW5hYmxl
ZCwgcmVzdW1lLCBJT1BMID0gMA0KY3VycmVudCBwcm9jZXNzCQk9IDI3ODUgKGRldm1hdGNo
KQ0KdHJhcCBudW1iZXIJCT0gOQ0KcGFuaWM6IGdlbmVyYWwgcHJvdGVjdGlvbiBmYXVsdA0K
Y3B1aWQgPSAxDQp0aW1lID0gMTY0NjQxOTAyOQ0KS0RCOiBzdGFjayBiYWNrdHJhY2U6DQoj
MCAweGZmZmZmZmZmODBjNTc1MjUgYXQga2RiX2JhY2t0cmFjZSsweDY1DQojMSAweGZmZmZm
ZmZmODBjMDlmMDEgYXQgdnBhbmljKzB4MTgxDQojMiAweGZmZmZmZmZmODBjMDlkNzMgYXQg
cGFuaWMrMHg0Mw0KIzMgMHhmZmZmZmZmZjgxMDhiMWE3IGF0IHRyYXArMHhiYzcNCiM0IDB4
ZmZmZmZmZmY4MTA4YTY2ZSBhdCB0cmFwKzB4OGUNCiM1IDB4ZmZmZmZmZmY4MTA2MWIxOCBh
dCBjYWxsdHJhcCsweDgNCiM2IDB4ZmZmZmZmZmY4MGM1ZGExNyBhdCBrdnByaW50ZisweDEw
MDcNCiM3IDB4ZmZmZmZmZmY4MGM1ZTcxOSBhdCBzbnByaW50ZisweDU5DQojOCAweGZmZmZm
ZmZmODBjNjIwNGIgYXQgcm1hbl9pc19yZWdpb25fbWFuYWdlcisweDI3Yg0KIzkgMHhmZmZm
ZmZmZjgwYzFhMDUxIGF0IHNidWZfbmV3X2Zvcl9zeXNjdGwrMHgxMDENCiMxMCAweGZmZmZm
ZmZmODBjMTk0OWMgYXQga2VybmVsX3N5c2N0bCsweDQzYw0KIzExIDB4ZmZmZmZmZmY4MGMx
OWIxMyBhdCB1c2VybGFuZF9zeXNjdGwrMHgxNzMNCiMxMiAweGZmZmZmZmZmODBjMTk5NWYg
YXQgc3lzX19fc3lzY3RsKzB4NWYNCiMxMyAweGZmZmZmZmZmODEwOGJhYWMgYXQgYW1kNjRf
c3lzY2FsbCsweDEwYw0KIzE0IDB4ZmZmZmZmZmY4MTA2MjQzZSBhdCBYZmFzdF9zeXNjYWxs
KzB4ZmUNCg==
--------------6658CE0D501BB4674047759C
Content-Type: text/plain; charset=UTF-8;
 name="page_fault_agg.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="page_fault_agg.log"

VW5pcXVlIG9uICJjdXJyZW50IHByb2Nlc3MiIGFuZCB0cmFjZSBmaW5nZXJwcmludCAjMC0j
Ki4KCj09PT0gQ09VTlQ6IDIzCkZhdGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4g
a2VybmVsIG1vZGUKY3B1aWQgPSAwOyBhcGljIGlkID0gMDAKZmF1bHQgdmlydHVhbCBhZGRy
ZXNzID0gMHgyMjcxMDAyOApmYXVsdCBjb2RlICAgID0gc3VwZXJ2aXNvciByZWFkIGRhdGEs
IHBhZ2Ugbm90IHByZXNlbnQKaW5zdHJ1Y3Rpb24gcG9pbnRlciA9IDB4MjA6MHhmZmZmZmZm
ZjgwYzQ1ODkyCnN0YWNrIHBvaW50ZXIgICAgICAgICA9IDB4Mjg6MHhmZmZmZmUwMDk2NjAw
OTMwCmZyYW1lIHBvaW50ZXIgICAgICAgICA9IDB4Mjg6MHhmZmZmZmUwMDk2NjAwOTMwCmNv
ZGUgc2VnbWVudCAgICA9IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWIKICAg
ICAgPSBEUEwgMCwgcHJlcyAxLCBsb25nIDEsIGRlZjMyIDAsIGdyYW4gMQpwcm9jZXNzb3Ig
ZWZsYWdzICA9IGludGVycnVwdCBlbmFibGVkLCByZXN1bWUsIElPUEwgPSAwCmN1cnJlbnQg
cHJvY2VzcyAgID0gMTQ5NiAoZGV2bWF0Y2gpCnRyYXAgbnVtYmVyICAgPSAxMgpwYW5pYzog
cGFnZSBmYXVsdApjcHVpZCA9IDAKdGltZSA9IDE2NDYxMjM3OTEKS0RCOiBzdGFjayBiYWNr
dHJhY2U6CiMwIDB4ZmZmZmZmZmY4MGM1NzUyNSBhdCBrZGJfYmFja3RyYWNlKzB4NjUKIzEg
MHhmZmZmZmZmZjgwYzA5ZjAxIGF0IHZwYW5pYysweDE4MQojMiAweGZmZmZmZmZmODBjMDlk
NzMgYXQgcGFuaWMrMHg0MwojMyAweGZmZmZmZmZmODEwOGIxYTcgYXQgdHJhcCsweGJjNwoj
NCAweGZmZmZmZmZmODEwOGIxZmYgYXQgdHJhcCsweGMxZgojNSAweGZmZmZmZmZmODEwOGE4
NWQgYXQgdHJhcCsweDI3ZAojNiAweGZmZmZmZmZmODEwNjFiMTggYXQgY2FsbHRyYXArMHg4
CiM3IDB4ZmZmZmZmZmY4MGM2MjAxMSBhdCBybWFuX2lzX3JlZ2lvbl9tYW5hZ2VyKzB4MjQx
CiM4IDB4ZmZmZmZmZmY4MGMxYTA1MSBhdCBzYnVmX25ld19mb3Jfc3lzY3RsKzB4MTAxCiM5
IDB4ZmZmZmZmZmY4MGMxOTQ5YyBhdCBrZXJuZWxfc3lzY3RsKzB4NDNjCiMxMCAweGZmZmZm
ZmZmODBjMTliMTMgYXQgdXNlcmxhbmRfc3lzY3RsKzB4MTczCiMxMSAweGZmZmZmZmZmODBj
MTk5NWYgYXQgc3lzX19fc3lzY3RsKzB4NWYKIzEyIDB4ZmZmZmZmZmY4MTA4YmFhYyBhdCBh
bWQ2NF9zeXNjYWxsKzB4MTBjCiMxMyAweGZmZmZmZmZmODEwNjI0M2UgYXQgWGZhc3Rfc3lz
Y2FsbCsweGZlCgoKCj09PT0gQ09VTlQ6IDIKY3VycmVudCBwcm9jZXNzICAgPSAyMDI4NCAo
ZGV2bWF0Y2gpCnRyYXAgbnVtYmVyICAgPSAxMgpwYW5pYzogcGFnZSBmYXVsdApjcHVpZCA9
IDMKdGltZSA9IDE2NDcyNDc2MTgKS0RCOiBzdGFjayBiYWNrdHJhY2U6CiMwIDB4ZmZmZmZm
ZmY4MGM3YzYxNSBhdCBrZGJfYmFja3RyYWNlKzB4NjUKIzEgMHhmZmZmZmZmZjgwYzJlNjIx
IGF0IHZwYW5pYysweDE4MQojMiAweGZmZmZmZmZmODBjMmU0OTMgYXQgcGFuaWMrMHg0Mwoj
MyAweGZmZmZmZmZmODEwYzhiOTcgYXQgdHJhcCsweGJhNwojNCAweGZmZmZmZmZmODEwYzhi
ZWYgYXQgdHJhcCsweGJmZgojNSAweGZmZmZmZmZmODEwYzgyNDMgYXQgdHJhcCsweDI1Mwoj
NiAweGZmZmZmZmZmODEwYTA5ZDggYXQgY2FsbHRyYXArMHg4CiM3IDB4ZmZmZmZmZmY4MGM4
MmM3NyBhdCBrdnByaW50ZisweDEwMDcKIzggMHhmZmZmZmZmZjgwYzgzYTA5IGF0IHNucHJp
bnRmKzB4NTkKIzkgMHhmZmZmZmZmZjgwYzg3MjliIGF0IHJtYW5faXNfcmVnaW9uX21hbmFn
ZXIrMHgyN2IKIzEwIDB4ZmZmZmZmZmY4MGMzZWU4MSBhdCBzYnVmX25ld19mb3Jfc3lzY3Rs
KzB4MTAxCiMxMSAweGZmZmZmZmZmODBjM2UyY2MgYXQga2VybmVsX3N5c2N0bCsweDNlYwoj
MTIgMHhmZmZmZmZmZjgwYzNlOTQzIGF0IHVzZXJsYW5kX3N5c2N0bCsweDE3MwojMTMgMHhm
ZmZmZmZmZjgwYzNlNzhmIGF0IHN5c19fX3N5c2N0bCsweDVmCiMxNCAweGZmZmZmZmZmODEw
Yzk0OWMgYXQgYW1kNjRfc3lzY2FsbCsweDEwYwojMTUgMHhmZmZmZmZmZjgxMGExMmViIGF0
IFhmYXN0X3N5c2NhbGwrMHhmYgoKCgo9PT09IENPVU5UOiAyCkZhdGFsIHRyYXAgMTI6IHBh
Z2UgZmF1bHQgd2hpbGUgaW4ga2VybmVsIG1vZGUKY3B1aWQgPSAxOyBhcGljIGlkID0gMDIK
ZmF1bHQgdmlydHVhbCBhZGRyZXNzID0gMHg2OApmYXVsdCBjb2RlICAgID0gc3VwZXJ2aXNv
ciByZWFkIGRhdGEsIHBhZ2Ugbm90IHByZXNlbnQKaW5zdHJ1Y3Rpb24gcG9pbnRlciA9IDB4
MjA6MHhmZmZmZmZmZjgyNGE1OTlkCnN0YWNrIHBvaW50ZXIgICAgICAgICA9IDB4Mjg6MHhm
ZmZmZmUwMGIxZTI3OTEwCmZyYW1lIHBvaW50ZXIgICAgICAgICA9IDB4Mjg6MHhmZmZmZmUw
MGIxZTI3OWIwCmNvZGUgc2VnbWVudCAgICA9IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0
eXBlIDB4MWIKICAgICAgPSBEUEwgMCwgcHJlcyAxLCBsb25nIDEsIGRlZjMyIDAsIGdyYW4g
MQpwcm9jZXNzb3IgZWZsYWdzICA9IGludGVycnVwdCBlbmFibGVkLCByZXN1bWUsIElPUEwg
PSAwCmN1cnJlbnQgcHJvY2VzcyAgID0gMCAoeGJiZDcgdGFza3EpCnRyYXAgbnVtYmVyICAg
PSAxMgpwYW5pYzogcGFnZSBmYXVsdApjcHVpZCA9IDEKdGltZSA9IDE2NDYxMjI3MjMKS0RC
OiBzdGFjayBiYWNrdHJhY2U6CiMwIDB4ZmZmZmZmZmY4MGM1NzUyNSBhdCBrZGJfYmFja3Ry
YWNlKzB4NjUKIzEgMHhmZmZmZmZmZjgwYzA5ZjAxIGF0IHZwYW5pYysweDE4MQojMiAweGZm
ZmZmZmZmODBjMDlkNzMgYXQgcGFuaWMrMHg0MwojMyAweGZmZmZmZmZmODEwOGIxYTcgYXQg
dHJhcCsweGJjNwojNCAweGZmZmZmZmZmODEwOGIxZmYgYXQgdHJhcCsweGMxZgojNSAweGZm
ZmZmZmZmODEwOGE4NWQgYXQgdHJhcCsweDI3ZAojNiAweGZmZmZmZmZmODEwNjFiMTggYXQg
Y2FsbHRyYXArMHg4CiM3IDB4ZmZmZmZmZmY4MjQ4OTM1YSBhdCBkbXVfcmVhZCsweDJhCiM4
IDB4ZmZmZmZmZmY4MjQ1NmEzYSBhdCB6dm9sX2dlb21fYmlvX3N0cmF0ZWd5KzB4MmFhCiM5
IDB4ZmZmZmZmZmY4MGE3ZjIxNCBhdCB4YmRfaW5zdGFuY2VfY3JlYXRlKzB4YTM5NAojMTAg
MHhmZmZmZmZmZjgwYTdiMWVhIGF0IHhiZF9pbnN0YW5jZV9jcmVhdGUrMHg2MzZhCiMxMSAw
eGZmZmZmZmZmODBjNmIxYzEgYXQgdGFza3F1ZXVlX3J1bisweDJhMQojMTIgMHhmZmZmZmZm
ZjgwYzZjNGRjIGF0IHRhc2txdWV1ZV90aHJlYWRfbG9vcCsweGFjCiMxMyAweGZmZmZmZmZm
ODBiYzdlM2UgYXQgZm9ya19leGl0KzB4N2UKIzE0IDB4ZmZmZmZmZmY4MTA2MmI5ZSBhdCBm
b3JrX3RyYW1wb2xpbmUrMHhlCgoKPT09PSBDT1VOVDogMQpGYXRhbCB0cmFwIDEyOiBwYWdl
IGZhdWx0IHdoaWxlIGluIGtlcm5lbCBtb2RlCmNwdWlkID0gMTsgYXBpYyBpZCA9IDAyCmZh
dWx0IHZpcnR1YWwgYWRkcmVzcyA9IDB4MTQ4CmZhdWx0IGNvZGUgICAgPSBzdXBlcnZpc29y
IHJlYWQgZGF0YSwgcGFnZSBub3QgcHJlc2VudAppbnN0cnVjdGlvbiBwb2ludGVyID0gMHgy
MDoweGZmZmZmZmZmODI0OGNlZjQKc3RhY2sgcG9pbnRlciAgICAgICAgID0gMHgyODoweGZm
ZmZmZTAwOTk0MWQ5YTAKZnJhbWUgcG9pbnRlciAgICAgICAgID0gMHgyODoweGZmZmZmZTAw
OTk0MWQ5YTAKY29kZSBzZWdtZW50ICAgID0gYmFzZSAweDAsIGxpbWl0IDB4ZmZmZmYsIHR5
cGUgMHgxYgogICAgICA9IERQTCAwLCBwcmVzIDEsIGxvbmcgMSwgZGVmMzIgMCwgZ3JhbiAx
CnByb2Nlc3NvciBlZmxhZ3MgID0gaW50ZXJydXB0IGVuYWJsZWQsIHJlc3VtZSwgSU9QTCA9
IDAKY3VycmVudCBwcm9jZXNzICAgPSAwICh4YmJkMSB0YXNrcSkKdHJhcCBudW1iZXIgICA9
IDEyCnBhbmljOiBwYWdlIGZhdWx0CmNwdWlkID0gMQp0aW1lID0gMTY0NjEyOTc3MwpLREI6
IHN0YWNrIGJhY2t0cmFjZToKIzAgMHhmZmZmZmZmZjgwYzU3NTI1IGF0IGtkYl9iYWNrdHJh
Y2UrMHg2NQojMSAweGZmZmZmZmZmODBjMDlmMDEgYXQgdnBhbmljKzB4MTgxCiMyIDB4ZmZm
ZmZmZmY4MGMwOWQ3MyBhdCBwYW5pYysweDQzCiMzIDB4ZmZmZmZmZmY4MTA4YjFhNyBhdCB0
cmFwKzB4YmM3CiM0IDB4ZmZmZmZmZmY4MTA4YjFmZiBhdCB0cmFwKzB4YzFmCiM1IDB4ZmZm
ZmZmZmY4MTA4YTg1ZCBhdCB0cmFwKzB4MjdkCiM2IDB4ZmZmZmZmZmY4MTA2MWIxOCBhdCBj
YWxsdHJhcCsweDgKIzcgMHhmZmZmZmZmZjgyNWNiNzZlIGF0IHppbF9vcGVuKzB4ZQojOCAw
eGZmZmZmZmZmODI0NTZkMDIgYXQgenZvbF9lbnN1cmVfemlsb2crMHhiMgojOSAweGZmZmZm
ZmZmODI0NTY4MTggYXQgenZvbF9nZW9tX2Jpb19zdHJhdGVneSsweDg4CiMxMCAweGZmZmZm
ZmZmODBhN2YyMTQgYXQgeGJkX2luc3RhbmNlX2NyZWF0ZSsweGEzOTQKIzExIDB4ZmZmZmZm
ZmY4MGE3YjFlYSBhdCB4YmRfaW5zdGFuY2VfY3JlYXRlKzB4NjM2YQojMTIgMHhmZmZmZmZm
ZjgwYzZiMWMxIGF0IHRhc2txdWV1ZV9ydW4rMHgyYTEKIzEzIDB4ZmZmZmZmZmY4MGM2YzRk
YyBhdCB0YXNrcXVldWVfdGhyZWFkX2xvb3ArMHhhYwojMTQgMHhmZmZmZmZmZjgwYmM3ZTNl
IGF0IGZvcmtfZXhpdCsweDdlCiMxNSAweGZmZmZmZmZmODEwNjJiOWUgYXQgZm9ya190cmFt
cG9saW5lKzB4ZQo=
--------------6658CE0D501BB4674047759C
Content-Type: text/plain; charset=UTF-8;
 name="2022_03_26__xen_serial__064_xendebug_2g__tailtrim.log"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="2022_03_26__xen_serial__064_xendebug_2g__tailtrim.log"

dW5rTG9hZGluZyAvYm9vdC9sb2FkZXIuY29uZi5sb2NhbA0KLVwvL2Jvb3QveGVuLWRlYnVn
IGRhdGE9MHgyN2Q5YzgrMHgxNDQ2MzggLQ0KTG9hZGluZyBrZXJuZWwuLi4NClx8Ly9ib290
L2tlcm5lbC9rZXJuZWwgLVx8Ly1cfC8tXHwvLVx8c2l6ZT0weDFkZTFkNTANCi8tXHxMb2Fk
aW5nIGNvbmZpZ3VyZWQgbW9kdWxlcy4uLg0KL2Jvb3QvZW50cm9weSBzaXplPTB4MTAwMA0K
Ly9ldGMvaG9zdGlkIHNpemU9MHgyNQ0KDQpIaXQgW0VudGVyXSB0byBib290IGltbWVkaWF0
ZWx5LCBvciBhbnkgb3RoZXIga2V5IGZvciBjb21tYW5kIHByb21wdC4NCkJvb3RpbmcgWy9i
b290L2tlcm5lbC9rZXJuZWxdIGluIDMgc2Vjb25kcy4uLiBCb290aW5nIFsvYm9vdC9rZXJu
ZWwva2VybmVsXSBpbiAyIHNlY29uZHMuLi4gQm9vdGluZyBbL2Jvb3Qva2VybmVsL2tlcm5l
bF0gaW4gMSBzZWNvbmQuLi4gQm9vdGluZyBbL2Jvb3Qva2VybmVsL2tlcm5lbF0uLi4gICAg
ICAgICAgICAgICANCiBYZW4gNC4xNS4wDQooWEVOKSBYZW4gdmVyc2lvbiA0LjE1LjAgKHJv
b3RAKSAoRnJlZUJTRCBjbGFuZyB2ZXJzaW9uIDEzLjAuMCAoZ2l0QGdpdGh1Yi5jb206bGx2
bS9sbHZtLXByb2plY3QuZ2l0IGxsdm1vcmctMTMuMC4wLTAtZ2Q3YjY2OWIzYTMwMykpIGRl
YnVnPW4gU2F0IE1hciAyNiAxMToyNjo1NiBDRVQgMjAyMg0KKFhFTikgTGF0ZXN0IENoYW5n
ZVNldDogDQooWEVOKSBidWlsZC1pZDogMDIxMDc5MDRhMzE2ZTQ2NmE1MTdiMDg1YmNjZWQw
YzRkZjk2NDhjYQ0KKFhFTikgQ29uc29sZSBvdXRwdXQgaXMgc3luY2hyb25vdXMuDQooWEVO
KSBCb290bG9hZGVyOiBGcmVlQlNEIExvYWRlcg0KKFhFTikgQ29tbWFuZCBsaW5lOiBkb20w
X21lbT0yMDQ4TSBkb20wX21heF92Y3B1cz00IGRvbTA9cHZoLHZlcmJvc2U9MSBjb25zb2xl
PXZnYSxjb20xIGNvbTE9OTYwMCw4bjEgZ3Vlc3RfbG9nbHZsPWFsbCBsb2dsdmw9YWxsIHN5
bmNfY29uc29sZT0xIG5vcmVib290DQooWEVOKSBYZW4gaW1hZ2UgbG9hZCBiYXNlIGFkZHJl
c3M6IDANCihYRU4pIFZpZGVvIGluZm9ybWF0aW9uOg0KKFhFTikgIFZHQSBpcyB0ZXh0IG1v
ZGUgODB4MjUsIGZvbnQgOHgxNg0KKFhFTikgIFZCRS9EREMgbWV0aG9kczogbm9uZTsgRURJ
RCB0cmFuc2ZlciB0aW1lOiAwIHNlY29uZHMNCihYRU4pICBFRElEIGluZm8gbm90IHJldHJp
ZXZlZCBiZWNhdXNlIG5vIEREQyByZXRyaWV2YWwgbWV0aG9kIGRldGVjdGVkDQooWEVOKSBE
aXNjIGluZm9ybWF0aW9uOg0KKFhFTikgIEZvdW5kIDEgTUJSIHNpZ25hdHVyZXMNCihYRU4p
ICBGb3VuZCAyIEVERCBpbmZvcm1hdGlvbiBzdHJ1Y3R1cmVzDQooWEVOKSBDUFUgVmVuZG9y
OiBJbnRlbCwgRmFtaWx5IDYgKDB4NiksIE1vZGVsIDMwICgweDFlKSwgU3RlcHBpbmcgNSAo
cmF3IDAwMDEwNmU1KQ0KKFhFTikgWGVuLWU4MjAgUkFNIG1hcDoNCihYRU4pICBbMDAwMDAw
MDAwMDAwMDAwMCwgMDAwMDAwMDAwMDA5YTNmZl0gKHVzYWJsZSkNCihYRU4pICBbMDAwMDAw
MDAwMDA5YTQwMCwgMDAwMDAwMDAwMDA5ZmZmZl0gKHJlc2VydmVkKQ0KKFhFTikgIFswMDAw
MDAwMDAwMGUwMDAwLCAwMDAwMDAwMDAwMGZmZmZmXSAocmVzZXJ2ZWQpDQooWEVOKSAgWzAw
MDAwMDAwMDAxMDAwMDAsIDAwMDAwMDAwOWQ5NzZmZmZdICh1c2FibGUpDQooWEVOKSAgWzAw
MDAwMDAwOWQ5NzcwMDAsIDAwMDAwMDAwOWRhNDhmZmZdIChBQ1BJIE5WUykNCihYRU4pICBb
MDAwMDAwMDA5ZGE0OTAwMCwgMDAwMDAwMDA5ZjZhM2ZmZl0gKEFDUEkgZGF0YSkNCihYRU4p
ICBbMDAwMDAwMDA5ZjZhNDAwMCwgMDAwMDAwMDA5ZjZkZWZmZl0gKHJlc2VydmVkKQ0KKFhF
TikgIFswMDAwMDAwMDlmNmRmMDAwLCAwMDAwMDAwMDlmNzg4ZmZmXSAoQUNQSSBkYXRhKQ0K
KFhFTikgIFswMDAwMDAwMDlmNzg5MDAwLCAwMDAwMDAwMDlmN2RlZmZmXSAoQUNQSSBOVlMp
DQooWEVOKSAgWzAwMDAwMDAwOWY3ZGYwMDAsIDAwMDAwMDAwOWY3ZmZmZmZdIChBQ1BJIGRh
dGEpDQooWEVOKSAgWzAwMDAwMDAwOWY4MDAwMDAsIDAwMDAwMDAwYWZmZmZmZmZdIChyZXNl
cnZlZCkNCihYRU4pICBbMDAwMDAwMDBmZWQxYzAwMCwgMDAwMDAwMDBmZWQxZmZmZl0gKHJl
c2VydmVkKQ0KKFhFTikgIFswMDAwMDAwMGZmODAwMDAwLCAwMDAwMDAwMGZmZmZmZmZmXSAo
cmVzZXJ2ZWQpDQooWEVOKSAgWzAwMDAwMDAxMDAwMDAwMDAsIDAwMDAwMDA0NWZmZmZmZmZd
ICh1c2FibGUpDQooWEVOKSBOZXcgWGVuIGltYWdlIGJhc2UgYWRkcmVzczogMHg5ZDIwMDAw
MA0KKFhFTikgQUNQSTogUlNEUCAwMDBGMDQxMCwgMDAyNCAocjIgSU5URUwgKQ0KKFhFTikg
QUNQSTogWFNEVCA5RjdGRDEyMCwgMDA4QyAocjEgSU5URUwgIFMzNDIwR1BWICAgICAgICAw
ICAgICAgIDEwMDAwMTMpDQooWEVOKSBBQ1BJOiBGQUNQIDlGN0ZCMDAwLCAwMEY0IChyNCBJ
TlRFTCAgUzM0MjBHUFYgICAgICAgIDAgTVNGVCAgMTAwMDAwRCkNCihYRU4pIEFDUEk6IERT
RFQgOUY3RjYwMDAsIDQ4OTEgKHIyIElOVEVMICBTMzQyMEdQViAgICAgICAgMyBNU0ZUICAx
MDAwMDBEKQ0KKFhFTikgQUNQSTogRkFDUyA5Rjc4OTAwMCwgMDA0MA0KKFhFTikgQUNQSTog
QVBJQyA5RjdGNTAwMCwgMDBCQyAocjIgSU5URUwgIFMzNDIwR1BWICAgICAgICAwIE1TRlQg
IDEwMDAwMEQpDQooWEVOKSBBQ1BJOiBNQ0ZHIDlGN0Y0MDAwLCAwMDNDIChyMSBJTlRFTCAg
UzM0MjBHUFYgICAgICAgIDEgTVNGVCAgMTAwMDAwRCkNCihYRU4pIEFDUEk6IEhQRVQgOUY3
RjMwMDAsIDAwMzggKHIxIElOVEVMICBTMzQyMEdQViAgICAgICAgMSBNU0ZUICAxMDAwMDBE
KQ0KKFhFTikgQUNQSTogU0xJVCA5RjdGMjAwMCwgMDAzMCAocjEgSU5URUwgIFMzNDIwR1BW
ICAgICAgICAxIE1TRlQgIDEwMDAwMEQpDQooWEVOKSBBQ1BJOiBTUENSIDlGN0YxMDAwLCAw
MDUwIChyMSBJTlRFTCAgUzM0MjBHUFYgICAgICAgIDAgTVNGVCAgMTAwMDAwRCkNCihYRU4p
IEFDUEk6IFdERFQgOUY3RjAwMDAsIDAwNDAgKHIxIElOVEVMICBTMzQyMEdQViAgICAgICAg
MCBNU0ZUICAxMDAwMDBEKQ0KKFhFTikgQUNQSTogU1NEVCA5RjdFNTAwMCwgQUVGNCAocjIg
IElOVEVMIFNTRFQgIFBNICAgICA0MDAwIElOVEwgMjAwNjExMDkpDQooWEVOKSBBQ1BJOiBE
TUFSIDlGN0U0MDAwLCAwMEE4IChyMSBJTlRFTCAgUzM0MjBHUFYgICAgICAgIDEgTVNGVCAg
MTAwMDAwRCkNCihYRU4pIEFDUEk6IEhFU1QgOUY3RTMwMDAsIDAwQTggKHIxIElOVEVMICBT
MzQyMEdQViAgICAgICAgMSBJTlRMICAgICAgICAxKQ0KKFhFTikgQUNQSTogQkVSVCA5RjdF
MjAwMCwgMDAzMCAocjEgSU5URUwgIFMzNDIwR1BWICAgICAgICAxIElOVEwgICAgICAgIDEp
DQooWEVOKSBBQ1BJOiBFUlNUIDlGN0UxMDAwLCAwMjMwIChyMSBJTlRFTCAgUzM0MjBHUFYg
ICAgICAgIDEgSU5UTCAgICAgICAgMSkNCihYRU4pIEFDUEk6IEVJTkogOUY3RTAwMDAsIDAx
MzAgKHIxIElOVEVMICBTMzQyMEdQViAgICAgICAgMSBJTlRMICAgICAgICAxKQ0KKFhFTikg
U3lzdGVtIFJBTTogMTYzNDVNQiAoMTY3MzczNDhrQikNCihYRU4pIE5vIE5VTUEgY29uZmln
dXJhdGlvbiBmb3VuZA0KKFhFTikgRmFraW5nIGEgbm9kZSBhdCAwMDAwMDAwMDAwMDAwMDAw
LTAwMDAwMDA0NjAwMDAwMDANCihYRU4pIERvbWFpbiBoZWFwIGluaXRpYWxpc2VkDQooWEVO
KSBmb3VuZCBTTVAgTVAtdGFibGUgYXQgMDAwZmQ5YTANCihYRU4pIERNSSAyLjUgcHJlc2Vu
dC4NCihYRU4pIFVzaW5nIEFQSUMgZHJpdmVyIGRlZmF1bHQNCihYRU4pIEFDUEk6IFBNLVRp
bWVyIElPIFBvcnQ6IDB4NDA4ICgyNCBiaXRzKQ0KKFhFTikgQUNQSTogU0xFRVAgSU5GTzog
cG0xeF9jbnRbMTo0MDQsMTowXSwgcG0xeF9ldnRbMTo0MDAsMTowXQ0KKFhFTikgQUNQSTog
ICAgICAgICAgICAgd2FrZXVwX3ZlY1s5Zjc4OTAwY10sIHZlY19zaXplWzIwXQ0KKFhFTikg
QUNQSTogTG9jYWwgQVBJQyBhZGRyZXNzIDB4ZmVlMDAwMDANCihYRU4pIEFDUEk6IElPQVBJ
QyAoaWRbMHgwOF0gYWRkcmVzc1sweGZlYzAwMDAwXSBnc2lfYmFzZVswXSkNCihYRU4pIElP
QVBJQ1swXTogYXBpY19pZCA4LCB2ZXJzaW9uIDMyLCBhZGRyZXNzIDB4ZmVjMDAwMDAsIEdT
SSAwLTIzDQooWEVOKSBBQ1BJOiBJTlRfU1JDX09WUiAoYnVzIDAgYnVzX2lycSAwIGdsb2Jh
bF9pcnEgMiBkZmwgZGZsKQ0KKFhFTikgQUNQSTogSU5UX1NSQ19PVlIgKGJ1cyAwIGJ1c19p
cnEgOSBnbG9iYWxfaXJxIDkgaGlnaCBsZXZlbCkNCihYRU4pIEFDUEk6IElSUTAgdXNlZCBi
eSBvdmVycmlkZS4NCihYRU4pIEFDUEk6IElSUTIgdXNlZCBieSBvdmVycmlkZS4NCihYRU4p
IEFDUEk6IElSUTkgdXNlZCBieSBvdmVycmlkZS4NCihYRU4pIEVuYWJsaW5nIEFQSUMgbW9k
ZTogIEZsYXQuICBVc2luZyAxIEkvTyBBUElDcw0KKFhFTikgQUNQSTogSFBFVCBpZDogMHg4
MDg2YTgwMSBiYXNlOiAweGZlZDAwMDAwDQooWEVOKSBQQ0k6IE1DRkcgY29uZmlndXJhdGlv
biAwOiBiYXNlIGEwMDAwMDAwIHNlZ21lbnQgMDAwMCBidXNlcyAwMCAtIGZmDQooWEVOKSBQ
Q0k6IE1DRkcgYXJlYSBhdCBhMDAwMDAwMCByZXNlcnZlZCBpbiBFODIwDQooWEVOKSBQQ0k6
IFVzaW5nIE1DRkcgZm9yIHNlZ21lbnQgMDAwMCBidXMgMDAtZmYNCihYRU4pIFhlbiBFUlNU
IHN1cHBvcnQgaXMgaW5pdGlhbGl6ZWQuDQooWEVOKSBIRVNUOiBUYWJsZSBwYXJzaW5nIGhh
cyBiZWVuIGluaXRpYWxpemVkDQooWEVOKSBVc2luZyBBQ1BJIChNQURUKSBmb3IgU01QIGNv
bmZpZ3VyYXRpb24gaW5mb3JtYXRpb24NCihYRU4pIFNNUDogQWxsb3dpbmcgOCBDUFVzICgw
IGhvdHBsdWcgQ1BVcykNCihYRU4pIElSUSBsaW1pdHM6IDI0IEdTSSwgMTY0MCBNU0kvTVNJ
LVgNCihYRU4pIENQVTA6IDEyMDAgLi4uIDI1MzMgTUh6DQooWEVOKSBDUFUwOiBJbnRlbCBt
YWNoaW5lIGNoZWNrIHJlcG9ydGluZyBlbmFibGVkDQooWEVOKSBTcGVjdWxhdGl2ZSBtaXRp
Z2F0aW9uIGZhY2lsaXRpZXM6DQooWEVOKSAgIEhhcmR3YXJlIGZlYXR1cmVzOiBJQlJTL0lC
UEIgU1RJQlANCihYRU4pICAgQ29tcGlsZWQtaW4gc3VwcG9ydDogU0hBRE9XX1BBR0lORw0K
KFhFTikgICBYZW4gc2V0dGluZ3M6IEJUSS1UaHVuayBOL0EsIFNQRUNfQ1RSTDogSUJSUyss
IE90aGVyOiBJQlBCIEJSQU5DSF9IQVJERU4NCihYRU4pICAgTDFURjogYmVsaWV2ZWQgdnVs
bmVyYWJsZSwgbWF4cGh5c2FkZHIgTDFEIDQ0LCBDUFVJRCAzNiwgU2FmZSBhZGRyZXNzIDEw
MDAwMDAwMDANCihYRU4pICAgU3VwcG9ydCBmb3IgSFZNIFZNczogTVNSX1NQRUNfQ1RSTCBS
U0IgRUFHRVJfRlBVDQooWEVOKSAgIFN1cHBvcnQgZm9yIFBWIFZNczogTVNSX1NQRUNfQ1RS
TCBSU0IgRUFHRVJfRlBVDQooWEVOKSAgIFhQVEkgKDY0LWJpdCBQViBvbmx5KTogRG9tMCBl
bmFibGVkLCBEb21VIGVuYWJsZWQgKHdpdGhvdXQgUENJRCkNCihYRU4pICAgUFYgTDFURiBz
aGFkb3dpbmc6IERvbTAgZGlzYWJsZWQsIERvbVUgZW5hYmxlZA0KKFhFTikgVXNpbmcgc2No
ZWR1bGVyOiBTTVAgQ3JlZGl0IFNjaGVkdWxlciByZXYyIChjcmVkaXQyKQ0KKFhFTikgSW5p
dGlhbGl6aW5nIENyZWRpdDIgc2NoZWR1bGVyDQooWEVOKSAgbG9hZF9wcmVjaXNpb25fc2hp
ZnQ6IDE4DQooWEVOKSAgbG9hZF93aW5kb3dfc2hpZnQ6IDMwDQooWEVOKSAgdW5kZXJsb2Fk
X2JhbGFuY2VfdG9sZXJhbmNlOiAwDQooWEVOKSAgb3ZlcmxvYWRfYmFsYW5jZV90b2xlcmFu
Y2U6IC0zDQooWEVOKSAgcnVucXVldWVzIGFycmFuZ2VtZW50OiBzb2NrZXQNCihYRU4pICBj
YXAgZW5mb3JjZW1lbnQgZ3JhbnVsYXJpdHk6IDEwbXMNCihYRU4pIGxvYWQgdHJhY2tpbmcg
d2luZG93IGxlbmd0aCAxMDczNzQxODI0IG5zDQooWEVOKSBQbGF0Zm9ybSB0aW1lciBpcyAx
NC4zMThNSHogSFBFVA0KKFhFTikgRGV0ZWN0ZWQgMjUzMy4zMDggTUh6IHByb2Nlc3Nvci4N
CihYRU4pIEZyZWVkIDEwMjRrQiB1bnVzZWQgQlNTIG1lbW9yeQ0KKFhFTikgYWx0IHRhYmxl
IGZmZmY4MmQwNDA0NThmZjggLT4gZmZmZjgyZDA0MDQ2MjAzYw0KKFhFTikgSW50ZWwgVlQt
ZCBpb21tdSAwIHN1cHBvcnRlZCBwYWdlIHNpemVzOiA0a0INCihYRU4pIEludGVsIFZULWQg
U25vb3AgQ29udHJvbCBlbmFibGVkLg0KKFhFTikgSW50ZWwgVlQtZCBEb20wIERNQSBQYXNz
dGhyb3VnaCBub3QgZW5hYmxlZC4NCihYRU4pIEludGVsIFZULWQgUXVldWVkIEludmFsaWRh
dGlvbiBlbmFibGVkLg0KKFhFTikgSW50ZWwgVlQtZCBJbnRlcnJ1cHQgUmVtYXBwaW5nIG5v
dCBlbmFibGVkLg0KKFhFTikgSW50ZWwgVlQtZCBQb3N0ZWQgSW50ZXJydXB0IG5vdCBlbmFi
bGVkLg0KKFhFTikgSW50ZWwgVlQtZCBTaGFyZWQgRVBUIHRhYmxlcyBub3QgZW5hYmxlZC4N
CihYRU4pIEkvTyB2aXJ0dWFsaXNhdGlvbiBlbmFibGVkDQooWEVOKSAgLSBEb20wIG1vZGU6
IFJlbGF4ZWQNCihYRU4pIEludGVycnVwdCByZW1hcHBpbmcgZGlzYWJsZWQNCihYRU4pIG5y
X3NvY2tldHM6IDENCihYRU4pIEVOQUJMSU5HIElPLUFQSUMgSVJRcw0KKFhFTikgIC0+IFVz
aW5nIG5ldyBBQ0sgbWV0aG9kDQooWEVOKSAuLlRJTUVSOiB2ZWN0b3I9MHhGMCBhcGljMT0w
IHBpbjE9MiBhcGljMj0tMSBwaW4yPS0xDQooWEVOKSBBbGxvY2F0ZWQgY29uc29sZSByaW5n
IG9mIDY0IEtpQi4NCihYRU4pIG13YWl0LWlkbGU6IE1XQUlUIHN1YnN0YXRlczogMHgxMTIw
DQooWEVOKSBtd2FpdC1pZGxlOiB2MC40LjEgbW9kZWwgMHgxZQ0KKFhFTikgbXdhaXQtaWRs
ZTogbGFwaWNfdGltZXJfcmVsaWFibGVfc3RhdGVzIDB4Mg0KKFhFTikgSFBFVDogOCB0aW1l
cnMgdXNhYmxlIGZvciBicm9hZGNhc3QgKDggdG90YWwpDQooWEVOKSBWTVg6IFN1cHBvcnRl
ZCBhZHZhbmNlZCBmZWF0dXJlczoNCihYRU4pICAtIEFQSUMgTU1JTyBhY2Nlc3MgdmlydHVh
bGlzYXRpb24NCihYRU4pICAtIEFQSUMgVFBSIHNoYWRvdw0KKFhFTikgIC0gRXh0ZW5kZWQg
UGFnZSBUYWJsZXMgKEVQVCkNCihYRU4pICAtIFZpcnR1YWwtUHJvY2Vzc29yIElkZW50aWZp
ZXJzIChWUElEKQ0KKFhFTikgIC0gVmlydHVhbCBOTUkNCihYRU4pICAtIE1TUiBkaXJlY3Qt
YWNjZXNzIGJpdG1hcA0KKFhFTikgSFZNOiBBU0lEcyBlbmFibGVkLg0KKFhFTikgVk1YOiBE
aXNhYmxpbmcgZXhlY3V0YWJsZSBFUFQgc3VwZXJwYWdlcyBkdWUgdG8gQ1ZFLTIwMTgtMTIy
MDcNCihYRU4pIEhWTTogVk1YIGVuYWJsZWQNCihYRU4pIEhWTTogSGFyZHdhcmUgQXNzaXN0
ZWQgUGFnaW5nIChIQVApIGRldGVjdGVkDQooWEVOKSBIVk06IEhBUCBwYWdlIHNpemVzOiA0
a0IsIDJNQg0KKFhFTikgYWx0IHRhYmxlIGZmZmY4MmQwNDA0NThmZjggLT4gZmZmZjgyZDA0
MDQ2MjAzYw0KKFhFTikgRGlzYWJsaW5nIEMtc3RhdGVzIEMzIGFuZCBDNiBkdWUgdG8gQ1BV
IGVycmF0YQ0KKFhFTikgbXdhaXQtaWRsZTogbWF4IEMtc3RhdGUgMSByZWFjaGVkDQooWEVO
KSBtd2FpdC1pZGxlOiBtYXggQy1zdGF0ZSAxIHJlYWNoZWQNCihYRU4pIG13YWl0LWlkbGU6
IG1heCBDLXN0YXRlIDEgcmVhY2hlZA0KKFhFTikgbXdhaXQtaWRsZTogbWF4IEMtc3RhdGUg
MSByZWFjaGVkDQooWEVOKSBtd2FpdC1pZGxlOiBtYXggQy1zdGF0ZSAxIHJlYWNoZWQNCihY
RU4pIG13YWl0LWlkbGU6IG1heCBDLXN0YXRlIDEgcmVhY2hlZA0KKFhFTikgbXdhaXQtaWRs
ZTogbWF4IEMtc3RhdGUgMSByZWFjaGVkDQooWEVOKSBCcm91Z2h0IHVwIDggQ1BVcw0KKFhF
TikgU2NoZWR1bGluZyBncmFudWxhcml0eTogY3B1LCAxIENQVSBwZXIgc2NoZWQtcmVzb3Vy
Y2UNCihYRU4pIEFkZGluZyBjcHUgMCB0byBydW5xdWV1ZSAwDQooWEVOKSAgRmlyc3QgY3B1
IG9uIHJ1bnF1ZXVlLCBhY3RpdmF0aW5nDQooWEVOKSBBZGRpbmcgY3B1IDEgdG8gcnVucXVl
dWUgMA0KKFhFTikgQWRkaW5nIGNwdSAyIHRvIHJ1bnF1ZXVlIDANCihYRU4pIEFkZGluZyBj
cHUgMyB0byBydW5xdWV1ZSAwDQooWEVOKSBBZGRpbmcgY3B1IDQgdG8gcnVucXVldWUgMA0K
KFhFTikgQWRkaW5nIGNwdSA1IHRvIHJ1bnF1ZXVlIDANCihYRU4pIEFkZGluZyBjcHUgNiB0
byBydW5xdWV1ZSAwDQooWEVOKSBBZGRpbmcgY3B1IDcgdG8gcnVucXVldWUgMA0KKFhFTikg
bWNoZWNrX3BvbGw6IE1hY2hpbmUgY2hlY2sgcG9sbGluZyB0aW1lciBzdGFydGVkLg0KKFhF
TikgTlggKEV4ZWN1dGUgRGlzYWJsZSkgcHJvdGVjdGlvbiBhY3RpdmUNCihYRU4pIERvbTAg
aGFzIG1heGltdW0gODU2IFBJUlFzDQooWEVOKSAqKiogQnVpbGRpbmcgYSBQVkggRG9tMCAq
KioNCihYRU4pIERvbTAgbWVtb3J5IGFsbG9jYXRpb24gc3RhdHM6DQooWEVOKSBvcmRlciAg
MSBhbGxvY2F0aW9uczogMg0KKFhFTikgb3JkZXIgIDIgYWxsb2NhdGlvbnM6IDENCihYRU4p
IG9yZGVyICAzIGFsbG9jYXRpb25zOiAxDQooWEVOKSBvcmRlciAgNCBhbGxvY2F0aW9uczog
MQ0KKFhFTikgb3JkZXIgIDUgYWxsb2NhdGlvbnM6IDENCihYRU4pIG9yZGVyICA2IGFsbG9j
YXRpb25zOiAxDQooWEVOKSBvcmRlciAgNyBhbGxvY2F0aW9uczogMQ0KKFhFTikgb3JkZXIg
IDggYWxsb2NhdGlvbnM6IDENCihYRU4pIG9yZGVyICA5IGFsbG9jYXRpb25zOiAxDQooWEVO
KSBvcmRlciAxMCBhbGxvY2F0aW9uczogMQ0KKFhFTikgb3JkZXIgMTEgYWxsb2NhdGlvbnM6
IDENCihYRU4pIG9yZGVyIDEyIGFsbG9jYXRpb25zOiAxDQooWEVOKSBvcmRlciAxMyBhbGxv
Y2F0aW9uczogMQ0KKFhFTikgb3JkZXIgMTQgYWxsb2NhdGlvbnM6IDENCihYRU4pIG9yZGVy
IDE1IGFsbG9jYXRpb25zOiAxDQooWEVOKSBvcmRlciAxNiBhbGxvY2F0aW9uczogMQ0KKFhF
Tikgb3JkZXIgMTcgYWxsb2NhdGlvbnM6IDENCihYRU4pIG9yZGVyIDE4IGFsbG9jYXRpb25z
OiAxDQooWEVOKSBXQVJOSU5HOiBQVkggaXMgYW4gZXhwZXJpbWVudGFsIG1vZGUgd2l0aCBs
aW1pdGVkIGZ1bmN0aW9uYWxpdHkNCihYRU4pIEluaXRpYWwgbG93IG1lbW9yeSB2aXJxIHRo
cmVzaG9sZCBzZXQgYXQgMHg0MDAwIHBhZ2VzLg0KKFhFTikgU2NydWJiaW5nIEZyZWUgUkFN
IGluIGJhY2tncm91bmQNCihYRU4pIFN0ZC4gTG9nbGV2ZWw6IEFsbA0KKFhFTikgR3Vlc3Qg
TG9nbGV2ZWw6IEFsbA0KKFhFTikgKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqDQooWEVOKSBXQVJOSU5HOiBDT05TT0xFIE9VVFBVVCBJUyBT
WU5DSFJPTk9VUw0KKFhFTikgVGhpcyBvcHRpb24gaXMgaW50ZW5kZWQgdG8gYWlkIGRlYnVn
Z2luZyBvZiBYZW4gYnkgZW5zdXJpbmcNCihYRU4pIHRoYXQgYWxsIG91dHB1dCBpcyBzeW5j
aHJvbm91c2x5IGRlbGl2ZXJlZCBvbiB0aGUgc2VyaWFsIGxpbmUuDQooWEVOKSBIb3dldmVy
IGl0IGNhbiBpbnRyb2R1Y2UgU0lHTklGSUNBTlQgbGF0ZW5jaWVzIGFuZCBhZmZlY3QNCihY
RU4pIHRpbWVrZWVwaW5nLiBJdCBpcyBOT1QgcmVjb21tZW5kZWQgZm9yIHByb2R1Y3Rpb24g
dXNlIQ0KKFhFTikgKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqDQooWEVOKSBCb290ZWQgb24gTDFURi12dWxuZXJhYmxlIGhhcmR3YXJlIHdp
dGggU01UL0h5cGVydGhyZWFkaW5nDQooWEVOKSBlbmFibGVkLiAgUGxlYXNlIGFzc2VzcyB5
b3VyIGNvbmZpZ3VyYXRpb24gYW5kIGNob29zZSBhbg0KKFhFTikgZXhwbGljaXQgJ3NtdD08
Ym9vbD4nIHNldHRpbmcuICBTZWUgWFNBLTI3My4NCihYRU4pICoqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg0KKFhFTikgQm9vdGVkIG9uIE1M
UERTL01GQkRTLXZ1bG5lcmFibGUgaGFyZHdhcmUgd2l0aCBTTVQvSHlwZXJ0aHJlYWRpbmcN
CihYRU4pIGVuYWJsZWQuICBNaXRpZ2F0aW9ucyB3aWxsIG5vdCBiZSBmdWxseSBlZmZlY3Rp
dmUuICBQbGVhc2UNCihYRU4pIGNob29zZSBhbiBleHBsaWNpdCBzbXQ9PGJvb2w+IHNldHRp
bmcuICBTZWUgWFNBLTI5Ny4NCihYRU4pICoqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKg0KKFhFTikgMy4uLiAyLi4uIDEuLi4gDQooWEVOKSBY
ZW4gaXMgcmVsaW5xdWlzaGluZyBWR0EgY29uc29sZS4NCihYRU4pICoqKiBTZXJpYWwgaW5w
dXQgdG8gRE9NMCAodHlwZSAnQ1RSTC1hJyB0aHJlZSB0aW1lcyB0byBzd2l0Y2ggaW5wdXQp
DQooWEVOKSBGcmVlZCA2MDRrQiBpbml0IG1lbW9yeQ0KLS0tPDxCT09UPj4tLS0NCkFQSUM6
IFVzaW5nIHRoZSBNQURUIGVudW1lcmF0b3IuDQpDb3B5cmlnaHQgKGMpIDE5OTItMjAyMSBU
aGUgRnJlZUJTRCBQcm9qZWN0Lg0KQ29weXJpZ2h0IChjKSAxOTc5LCAxOTgwLCAxOTgzLCAx
OTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAxOTk0DQoJVGhlIFJlZ2VudHMg
b2YgdGhlIFVuaXZlcnNpdHkgb2YgQ2FsaWZvcm5pYS4gQWxsIHJpZ2h0cyByZXNlcnZlZC4N
CkZyZWVCU0QgaXMgYSByZWdpc3RlcmVkIHRyYWRlbWFyayBvZiBUaGUgRnJlZUJTRCBGb3Vu
ZGF0aW9uLg0KRnJlZUJTRCAxMy4xLVNUQUJMRSAjMzMgc3RhYmxlLzEzLTQwODE4ODJjNC1k
aXJ0eTogU2F0IE1hciAyNiAyMjoxNzoxMyBFRVQgMjAyMg0KICAgIHJvb3RAbGFiLTAxLmxv
Y2FsbmV0Oi91c3Ivb2JqL3Vzci9zcmMvYW1kNjQuYW1kNjQvc3lzL0dFTkVSSUMgYW1kNjQN
CkZyZWVCU0QgY2xhbmcgdmVyc2lvbiAxMS4wLjEgKGdpdEBnaXRodWIuY29tOmxsdm0vbGx2
bS1wcm9qZWN0LmdpdCBsbHZtb3JnLTExLjAuMS0wLWc0M2ZmNzVmMmMzZmUpDQpQUElNIDA6
IFBBPTB4YjgwMDAsIFZBPTB4ZmZmZmZmZmY4MjcxMDAwMCwgc2l6ZT0weDgwMDAsIG1vZGU9
MA0KcG1hcDogbGFyZ2UgbWFwIDggUE1MNCBzbG90cyAoNDA5NiBHQikNClZUKHZnYSk6IHRl
eHQgODB4MjUNClhFTjogSHlwZXJ2aXNvciB2ZXJzaW9uIDQuMTUgZGV0ZWN0ZWQuDQooWEVO
KSBkMHYwOiB1cGNhbGwgdmVjdG9yIDkzDQpQcmVsb2FkZWQgZWxmIG11bHRpYm9vdCBrZXJu
ZWwgIi9ib290L3hlbi1kZWJ1ZyIgYXQgMHhmZmZmZmZmZjgyNDRiNTQwLg0KUHJlbG9hZGVk
IGVsZiBrZXJuZWwgIi9ib290L2tlcm5lbC9rZXJuZWwiIGF0IDB4ZmZmZmZmZmY4MjQ0Yjcx
MC4NClByZWxvYWRlZCBib290X2VudHJvcHlfY2FjaGUgIi9ib290L2VudHJvcHkiIGF0IDB4
ZmZmZmZmZmY4MjQ1MzhjOC4NClByZWxvYWRlZCBob3N0dXVpZCAiL2V0Yy9ob3N0aWQiIGF0
IDB4ZmZmZmZmZmY4MjQ1MzkyMC4NClByZWxvYWRlZCBUU0xPRyBkYXRhICJUU0xPRyIgYXQg
MHhmZmZmZmZmZjgyNDUzOTcwLg0KRWFybHkgVFNDIGZyZXF1ZW5jeSAyNTMzMzA4ODMwSHog
Y2FsaWJyYXRlZCBmcm9tIDgyNTQgUElUDQpDUFU6IEludGVsKFIpIFhlb24oUikgQ1BVICAg
ICAgICAgICBYMzQ0MCAgQCAyLjUzR0h6ICgyNTMzLjMxLU1IeiBLOC1jbGFzcyBDUFUpDQog
IE9yaWdpbj0iR2VudWluZUludGVsIiAgSWQ9MHgxMDZlNSAgRmFtaWx5PTB4NiAgTW9kZWw9
MHgxZSAgU3RlcHBpbmc9NQ0KICBGZWF0dXJlcz0weDFmYzNmYmZmPEZQVSxWTUUsREUsUFNF
LFRTQyxNU1IsUEFFLE1DRSxDWDgsQVBJQyxTRVAsTVRSUixQR0UsTUNBLENNT1YsUEFULFBT
RTM2LEFDUEksTU1YLEZYU1IsU1NFLFNTRTIsU1MsSFRUPg0KICBGZWF0dXJlczI9MHg4MGI4
MjIwMTxTU0UzLFNTU0UzLENYMTYsU1NFNC4xLFNTRTQuMix4MkFQSUMsUE9QQ05ULEhWPg0K
ICBBTUQgRmVhdHVyZXM9MHgyODEwMDgwMDxTWVNDQUxMLE5YLFJEVFNDUCxMTT4NCiAgQU1E
IEZlYXR1cmVzMj0weDE8TEFIRj4NCiAgU3RydWN0dXJlZCBFeHRlbmRlZCBGZWF0dXJlczM9
MHhjMDAwMDAwPElCUEIsU1RJQlA+DQogIEFNRCBFeHRlbmRlZCBGZWF0dXJlIEV4dGVuc2lv
bnMgSUQgRUJYPTB4MTAwMDxJQlBCPg0KICBUU0M6IFAtc3RhdGUgaW52YXJpYW50DQpEYXRh
IFRMQjA6IDItTUJ5dGUgb3IgNCBNQnl0ZSBwYWdlcywgNC13YXkgc2V0IGFzc29jaWF0aXZl
LCAzMiBlbnRyaWVzDQpEYXRhIFRMQjogNCBLQiBwYWdlcywgNC13YXkgc2V0IGFzc29jaWF0
aXZlLCA2NCBlbnRyaWVzDQpJbnN0cnVjdGlvbiBUTEI6IDItTUJ5dGUgb3IgNC1NQnl0ZSBw
YWdlcywgZnVsbHkgYXNzb2NpYXRpdmUsIDcgZW50cmllcw0KM3JkLWxldmVsIGNhY2hlOiA4
IE1CeXRlLCAxNi13YXkgc2V0IGFzc29jaWF0aXZlLCA2NCBieXRlIGxpbmUgc2l6ZQ0KSW5z
dHJ1Y3Rpb24gVExCOiA0S0J5dGUgcGFnZXMsIDQtd2F5IHNldCBhc3NvY2lhdGl2ZSwgNjQg
ZW50cmllcw0KNjQtQnl0ZSBwcmVmZXRjaGluZw0KMXN0LWxldmVsIGRhdGEgY2FjaGU6IDMy
IEtCLCA4LXdheSBzZXQgYXNzb2NpYXRpdmUsIDY0IGJ5dGUgbGluZSBzaXplDQoybmQtbGV2
ZWwgY2FjaGU6IDI1NiBLQnl0ZXMsIDgtd2F5IHNldCBhc3NvY2lhdGl2ZSwgNjQgYnl0ZSBs
aW5lIHNpemUNClNoYXJlZCAybmQtTGV2ZWwgVExCOiA0IEtCeXRlIHBhZ2VzLCA0LXdheSBh
c3NvY2lhdGl2ZSwgNTEyIGVudHJpZXMNCjFzdC1sZXZlbCBpbnN0cnVjdGlvbiBjYWNoZTog
MzIgS0IsIDQtd2F5IHNldCBhc3NvY2lhdGl2ZSwgNjQgYnl0ZSBsaW5lIHNpemUNCkwyIGNh
Y2hlOiAyNTYga2J5dGVzLCA4LXdheSBhc3NvY2lhdGl2ZSwgNjQgYnl0ZXMvbGluZQ0KSHlw
ZXJ2aXNvcjogT3JpZ2luID0gIlhlblZNTVhlblZNTSINCnJlYWwgbWVtb3J5ICA9IDIxNDc4
ODkxNTIgKDIwNDggTUIpDQpQaHlzaWNhbCBtZW1vcnkgY2h1bmsocyk6DQoweDAwMDAwMDAw
MDAwMDEwMDAgLSAweDAwMDAwMDAwMDAwOTlmZmYsIDYyNjY4OCBieXRlcyAoMTUzIHBhZ2Vz
KQ0KMHgwMDAwMDAwMDAwMTAwMDAwIC0gMHgwMDAwMDAwMDAwMWZmZmZmLCAxMDQ4NTc2IGJ5
dGVzICgyNTYgcGFnZXMpDQoweDAwMDAwMDAwMDI3MDEwMDAgLSAweDAwMDAwMDAwN2M5ZDZm
ZmYsIDIwNDk3OTQwNDggYnl0ZXMgKDUwMDQzOCBwYWdlcykNCjB4MDAwMDAwMDA4MDAwMDAw
MCAtIDB4MDAwMDAwMDA4MDAzYWZmZiwgMjQxNjY0IGJ5dGVzICg1OSBwYWdlcykNCmF2YWls
IG1lbW9yeSA9IDIwNDEyNzAyNzIgKDE5NDYgTUIpDQp4MkFQSUMgYXZhaWxhYmxlIGJ1dCBk
aXNhYmxlZCBkdWUgdG8gcnVubmluZyB1bmRlciBYRU4NCk1BRFQ6IEZvdW5kIENQVSBBUElD
IElEIDAgQUNQSSBJRCAwOiBlbmFibGVkDQpTTVA6IEFkZGVkIENQVSAwIChBUCkNCk1BRFQ6
IEZvdW5kIENQVSBBUElDIElEIDIgQUNQSSBJRCAxOiBlbmFibGVkDQpTTVA6IEFkZGVkIENQ
VSAyIChBUCkNCk1BRFQ6IEZvdW5kIENQVSBBUElDIElEIDQgQUNQSSBJRCAyOiBlbmFibGVk
DQpTTVA6IEFkZGVkIENQVSA0IChBUCkNCk1BRFQ6IEZvdW5kIENQVSBBUElDIElEIDYgQUNQ
SSBJRCAzOiBlbmFibGVkDQpTTVA6IEFkZGVkIENQVSA2IChBUCkNCkV2ZW50IHRpbWVyICJM
QVBJQyIgcXVhbGl0eSAxMDANCkxBUElDOiBpcGlfd2FpdCgpIHVzIG11bHRpcGxpZXIgMSAo
ciAyNTQ1Njk0OTcyIHRzYyAyNTMzMzA4ODMwKQ0KQUNQSSBBUElDIFRhYmxlOiA8SU5URUwg
IFMzNDIwR1BWPg0KUGFja2FnZSBJRCBzaGlmdDogNA0KTDMgY2FjaGUgSUQgc2hpZnQ6IDQN
CkwyIGNhY2hlIElEIHNoaWZ0OiAxDQpMMSBjYWNoZSBJRCBzaGlmdDogMQ0KQ29yZSBJRCBz
aGlmdDogMQ0KQVAgYm9vdCBhZGRyZXNzIDB4MTAwMA0KSU5UUjogQWRkaW5nIGxvY2FsIEFQ
SUMgMiBhcyBhIHRhcmdldA0KSU5UUjogQWRkaW5nIGxvY2FsIEFQSUMgNCBhcyBhIHRhcmdl
dA0KSU5UUjogQWRkaW5nIGxvY2FsIEFQSUMgNiBhcyBhIHRhcmdldA0KRnJlZUJTRC9TTVA6
IE11bHRpcHJvY2Vzc29yIFN5c3RlbSBEZXRlY3RlZDogNCBDUFVzDQpGcmVlQlNEL1NNUDog
MSBwYWNrYWdlKHMpIHggNCBjb3JlKHMpDQpQYWNrYWdlIEhXIElEID0gMA0KCUNvcmUgSFcg
SUQgPSAwDQoJCUNQVTAgKEJTUCk6IEFQSUMgSUQ6IDANCglDb3JlIEhXIElEID0gMQ0KCQlD
UFUxIChBUCk6IEFQSUMgSUQ6IDINCglDb3JlIEhXIElEID0gMg0KCQlDUFUyIChBUCk6IEFQ
SUMgSUQ6IDQNCglDb3JlIEhXIElEID0gMw0KCQlDUFUzIChBUCk6IEFQSUMgSUQ6IDYNCkFQ
SUM6IENQVSAwIGhhcyBBQ1BJIElEIDANCkFQSUM6IENQVSAxIGhhcyBBQ1BJIElEIDENCkFQ
SUM6IENQVSAyIGhhcyBBQ1BJIElEIDINCkFQSUM6IENQVSAzIGhhcyBBQ1BJIElEIDMNCng4
NmJpb3M6ICBJVlQgMHgwMDAwMDAtMHgwMDA0ZmYgYXQgMHhmZmZmZjgwMDAwMDAwMDAwDQp4
ODZiaW9zOiBTU0VHIDB4MDAxMDAwLTB4MDAxZmZmIGF0IDB4ZmZmZmZlMDAwM2ZmYTAwMA0K
eDg2YmlvczogRUJEQSAweDA5YTAwMC0weDA5ZmZmZiBhdCAweGZmZmZmODAwMDAwOWEwMDAN
Cng4NmJpb3M6ICBST00gMHgwYTAwMDAtMHgwZmVmZmYgYXQgMHhmZmZmZjgwMDAwMGEwMDAw
DQpQZW50aXVtIFBybyBNVFJSIHN1cHBvcnQgZW5hYmxlZA0KcmFuZG9tOiByZWFkIDQwOTYg
Ynl0ZXMgZnJvbSBwcmVsb2FkZWQgY2FjaGUNCnJhbmRvbTogbm8gcGxhdGZvcm0gYm9vdGxv
YWRlciBlbnRyb3B5DQpyYW5kb206IHVuYmxvY2tpbmcgZGV2aWNlLg0KVklNQUdFICh2aXJ0
dWFsaXplZCBuZXR3b3JrIHN0YWNrKSBlbmFibGVkDQpob3N0dXVpZDogdXNpbmcgM2NlZDE0
ZDQtMDI3Mi0xMWUwLWI3MWMtMDAxZTY3MDIwNzA2DQpVTEU6IHNldHVwIGNwdSAwDQpVTEU6
IHNldHVwIGNwdSAxDQpVTEU6IHNldHVwIGNwdSAyDQpVTEU6IHNldHVwIGNwdSAzDQpYZW4g
aW50ZXJydXB0czogdW5hYmxlIHRvIHJlZ2lzdGVyIFBJUlEgRU9JIG1hcA0KWGVuIGludGVy
cnVwdCBzeXN0ZW0gaW5pdGlhbGl6ZWQNCkFDUEk6IFJTRFAgMHgwMDAwMDAwMDgwMDYzRjBD
IDAwMDAyNCAodjAyIElOVEVMICkNCkFDUEk6IFhTRFQgMHgwMDAwMDAwMDgwMDYzRjMwIDAw
MDA0NCAodjAxIElOVEVMICBTMzQyMEdQViAwMDAwMDAwMCAgICAgIDAxMDAwMDEzKQ0KQUNQ
STogQVBJQyAweDAwMDAwMDAwODAwNjNGNzQgMDAwMDhDICh2MDIgSU5URUwgIFMzNDIwR1BW
IDAwMDAwMDAwIE1TRlQgMDEwMDAwMEQpDQpBQ1BJOiBGQUNQIDB4MDAwMDAwMDA5RjdGQjAw
MCAwMDAwRjQgKHYwNCBJTlRFTCAgUzM0MjBHUFYgMDAwMDAwMDAgTVNGVCAwMTAwMDAwRCkN
CkFDUEk6IERTRFQgMHgwMDAwMDAwMDlGN0Y2MDAwIDAwNDg5MSAodjAyIElOVEVMICBTMzQy
MEdQViAwMDAwMDAwMyBNU0ZUIDAxMDAwMDBEKQ0KQUNQSTogRkFDUyAweDAwMDAwMDAwOUY3
ODkwMDAgMDAwMDQwDQpBQ1BJOiBNQ0ZHIDB4MDAwMDAwMDA5RjdGNDAwMCAwMDAwM0MgKHYw
MSBJTlRFTCAgUzM0MjBHUFYgMDAwMDAwMDEgTVNGVCAwMTAwMDAwRCkNCkFDUEk6IFNTRFQg
MHgwMDAwMDAwMDlGN0U1MDAwIDAwQUVGNCAodjAyIElOVEVMICBTU0RUICBQTSAwMDAwNDAw
MCBJTlRMIDIwMDYxMTA5KQ0KTUFEVDogRm91bmQgSU8gQVBJQyBJRCA4LCBJbnRlcnJ1cHQg
MCBhdCAweGZlYzAwMDAwDQppb2FwaWMwOiB2ZXIgMHgxMSBtYXhyZWRpciAweDE3DQppb2Fw
aWMwOiBSb3V0aW5nIGV4dGVybmFsIDgyNTlBJ3MgLT4gaW50cGluIDANCk1BRFQ6IEludGVy
cnVwdCBvdmVycmlkZTogc291cmNlIDAsIGlycSAyDQppb2FwaWMwOiBSb3V0aW5nIElSUSAw
IC0+IGludHBpbiAyDQpNQURUOiBJbnRlcnJ1cHQgb3ZlcnJpZGU6IHNvdXJjZSA5LCBpcnEg
OQ0KaW9hcGljMDogaW50cGluIDkgdHJpZ2dlcjogbGV2ZWwNCmlvYXBpYzAgPFZlcnNpb24g
MS4xPiBpcnFzIDAtMjMNCmNwdTAgQlNQOg0KICAgICBJRDogMHgwMDAwMDAwMCAgIFZFUjog
MHgwMDA1MDAxNCBMRFI6IDB4MDAwMDAwMDAgREZSOiAweGZmZmZmZmZmIHgyQVBJQzogMA0K
ICBsaW50MDogMHgwMDAxMDcwMCBsaW50MTogMHgwMDAwMDQwMCBUUFI6IDB4MDAwMDAwMDAg
U1ZSOiAweDAwMDAwMWZmDQogIHRpbWVyOiAweDAwMDEwMGVmIHRoZXJtOiAweDAwMDEwMDAw
IGVycjogMHgwMDAwMDBmMCBwbWM6IDB4MDAwMTA0MDANCihYRU4pIGQwdjM6IHVwY2FsbCB2
ZWN0b3IgOTMNCihYRU4pIGQwdjI6IHVwY2FsbCB2ZWN0b3IgOTMNCihYRU4pIGQwdjE6IHVw
Y2FsbCB2ZWN0b3IgOTMNClNNUDogQVAgQ1BVICMzIExhdW5jaGVkIQ0KY3B1MyBBUDoNCiAg
ICAgSUQ6IDB4MDYwMDAwMDAgICBWRVI6IDB4MDAwNTAwMTQgTERSOiAweDAwMDAwMDAwIERG
UjogMHhmZmZmZmZmZiB4MkFQSUM6IDANCiAgbGludDA6IDB4MDAwMTA3MDAgbGludDE6IDB4
MDAwMDA0MDAgVFBSOiAweDAwMDAwMDAwIFNWUjogMHgwMDAwMDFmZg0KICB0aW1lcjogMHgw
MDAxMDBlZiB0aGVybTogMHgwMDAxMDAwMCBlcnI6IDB4MDAwMDAwZjAgcG1jOiAweDAwMDEw
NDAwDQpTTVA6IEFQIENQVSAjMiBMYXVuY2hlZCENCmNwdTIgQVA6DQogICAgIElEOiAweDA0
MDAwMDAwICAgVkVSOiAweDAwMDUwMDE0IExEUjogMHgwMDAwMDAwMCBERlI6IDB4ZmZmZmZm
ZmYgeDJBUElDOiAwDQogIGxpbnQwOiAweDAwMDEwNzAwIGxpbnQxOiAweDAwMDAwNDAwIFRQ
UjogMHgwMDAwMDAwMCBTVlI6IDB4MDAwMDAxZmYNCiAgdGltZXI6IDB4MDAwMTAwZWYgdGhl
cm06IDB4MDAwMTAwMDAgZXJyOiAweDAwMDAwMGYwIHBtYzogMHgwMDAxMDQwMA0KU01QOiBB
UCBDUFUgIzEgTGF1bmNoZWQhDQpjcHUxIEFQOg0KICAgICBJRDogMHgwMjAwMDAwMCAgIFZF
UjogMHgwMDA1MDAxNCBMRFI6IDB4MDAwMDAwMDAgREZSOiAweGZmZmZmZmZmIHgyQVBJQzog
MA0KICBsaW50MDogMHgwMDAxMDcwMCBsaW50MTogMHgwMDAwMDQwMCBUUFI6IDB4MDAwMDAw
MDAgU1ZSOiAweDAwMDAwMWZmDQogIHRpbWVyOiAweDAwMDEwMGVmIHRoZXJtOiAweDAwMDEw
MDAwIGVycjogMHgwMDAwMDBmMCBwbWM6IDB4MDAwMTA0MDANClNNUDogcGFzc2VkIFRTQyBz
eW5jaHJvbml6YXRpb24gdGVzdA0KVFNDIHRpbWVjb3VudGVyIGRpc2NhcmRzIGxvd2VyIDEg
Yml0KHMpDQpLVExTOiBJbml0aWFsaXplZCA0IHRocmVhZHMNCndsYW46IDw4MDIuMTEgTGlu
ayBMYXllcj4NCnNuZF91bml0X2luaXQoKSB1PTB4MDBmZjgwMDAgWzUxMl0gZD0weDAwMDA3
YzAwIFszMl0gYz0weDAwMDAwM2ZmIFsxMDI0XQ0KZmVlZGVyX3JlZ2lzdGVyOiBzbmRfdW5p
dD0tMSBzbmRfbWF4YXV0b3ZjaGFucz0xNiBsYXRlbmN5PTIgZmVlZGVyX3JhdGVfbWluPTEg
ZmVlZGVyX3JhdGVfbWF4PTIwMTYwMDAgZmVlZGVyX3JhdGVfcm91bmQ9MjUNCnJhbmRvbTog
ZW50cm9weSBkZXZpY2UgZXh0ZXJuYWwgaW50ZXJmYWNlDQpFRkkgc3lzdGJsIG5vdCBhdmFp
bGFibGUNClthdGhfaGFsXSBsb2FkZWQNCnRjcF9sb2c6IHRjcF9sb2cgZGV2aWNlDQpjcnlw
dG86IDxjcnlwdG8gY29yZT4NCmtiZDogbmV3IGFycmF5IHNpemUgNA0Ka2JkMSBhdCBrYmRt
dXgwDQppbzogPEkvTz4NCm1lbTogPG1lbW9yeT4NCm51bGw6IDxmdWxsIGRldmljZSwgbnVs
bCBkZXZpY2UsIHplcm8gZGV2aWNlPg0KdnR2Z2EwOiA8VlQgVkdBIGRyaXZlcj4NCnNtYmlv
czA6IDxTeXN0ZW0gTWFuYWdlbWVudCBCSU9TPiBhdCBpb21lbSAweGYwNDQwLTB4ZjA0NWUN
CnNtYmlvczA6IFZlcnNpb246IDIuNSwgQkNEIFJldmlzaW9uOiAyLjUNCmNyeXB0bzogYXNz
aWduIGNyeXB0b3NvZnQwIGRyaXZlciBpZCAwLCBmbGFncyAweDYwMDAwMDANCmFlc25pMDog
Tm8gQUVTIG9yIFNIQSBzdXBwb3J0Lg0KYWNwaTA6IDxJTlRFTCBTMzQyMEdQVj4NCkFDUEk6
IDIgQUNQSSBBTUwgdGFibGVzIHN1Y2Nlc3NmdWxseSBhY3F1aXJlZCBhbmQgbG9hZGVkDQpQ
Q0llOiBNZW1vcnkgTWFwcGVkIGNvbmZpZ3VyYXRpb24gYmFzZSBAIDB4YTAwMDAwMDANCmlv
YXBpYzA6IHJvdXRpbmcgaW50cGluIDkgKElTQSBJUlEgOSkgdG8gbGFwaWMgMCB2ZWN0b3Ig
NDgNCmFjcGkwOiBQb3dlciBCdXR0b24gKGZpeGVkKQ0KYWNwaTA6IHdha2V1cCBjb2RlIHZh
IDB4ZmZmZmZlMDAwM2ZlMzAwMCBwYSAweDIwMDANCkZpcm13YXJlIEVycm9yIChBQ1BJKTog
QUVfQU1MX0JVRkZFUl9MSU1JVCwgRmllbGQgW0NQQjNdIGF0IGJpdCBvZmZzZXQvbGVuZ3Ro
IDY0LzMyIGV4Y2VlZHMgc2l6ZSBvZiB0YXJnZXQgQnVmZmVyICg2NCBiaXRzKSAoMjAyMDEx
MTMvZHNvcGNvZGUtMzU2KQ0KQUNQSSBFcnJvcjogQWJvcnRpbmcgbWV0aG9kIFxfU0IuX09T
QyBkdWUgdG8gcHJldmlvdXMgZXJyb3IgKEFFX0FNTF9CVUZGRVJfTElNSVQpICgyMDIwMTEx
My9wc3BhcnNlLTY4OSkNCmFjcGkwOiBfT1NDIGZhaWxlZDogQUVfQU1MX0JVRkZFUl9MSU1J
VA0KY3B1MDogPEFDUEkgQ1BVPiBvbiBhY3BpMA0KY3B1MTogPEFDUEkgQ1BVPiBvbiBhY3Bp
MA0KY3B1MjogPEFDUEkgQ1BVPiBvbiBhY3BpMA0KY3B1MzogPEFDUEkgQ1BVPiBvbiBhY3Bp
MA0KQUNQSTogUHJvY2Vzc29yIFxfUFJfLkNQVTQgKEFDUEkgSUQgNCkgaWdub3JlZA0KQUNQ
STogUHJvY2Vzc29yIFxfUFJfLkNQVTUgKEFDUEkgSUQgNSkgaWdub3JlZA0KQUNQSTogUHJv
Y2Vzc29yIFxfUFJfLkNQVTYgKEFDUEkgSUQgNikgaWdub3JlZA0KQUNQSTogUHJvY2Vzc29y
IFxfUFJfLkNQVTcgKEFDUEkgSUQgNykgaWdub3JlZA0KYXRydGMwOiA8QVQgcmVhbHRpbWUg
Y2xvY2s+IHBvcnQgMHg3MC0weDcxLDB4NzQtMHg3NyBpcnEgOCBvbiBhY3BpMA0KYXRydGMw
OiByZWdpc3RlcmVkIGFzIGEgdGltZS1vZi1kYXkgY2xvY2ssIHJlc29sdXRpb24gMS4wMDAw
MDBzDQppb2FwaWMwOiByb3V0aW5nIGludHBpbiA4IChJU0EgSVJRIDgpIHRvIGxhcGljIDIg
dmVjdG9yIDQ4DQppb2FwaWMwOiByb3V0aW5nIGludHBpbiA4IChJU0EgSVJRIDgpIHRvIGxh
cGljIDAgdmVjdG9yIDQ5DQpFdmVudCB0aW1lciAiUlRDIiBmcmVxdWVuY3kgMzI3NjggSHog
cXVhbGl0eSAwDQphdHRpbWVyMDogPEFUIHRpbWVyPiBwb3J0IDB4NDAtMHg0MywweDUwLTB4
NTMgaXJxIDAgb24gYWNwaTANClRpbWVjb3VudGVyICJpODI1NCIgZnJlcXVlbmN5IDExOTMx
ODIgSHogcXVhbGl0eSAwDQppb2FwaWMwOiByb3V0aW5nIGludHBpbiAyIChJU0EgSVJRIDAp
IHRvIGxhcGljIDQgdmVjdG9yIDQ4DQpFdmVudCB0aW1lciAiaTgyNTQiIGZyZXF1ZW5jeSAx
MTkzMTgyIEh6IHF1YWxpdHkgMTAwDQpocGV0MDogPEhpZ2ggUHJlY2lzaW9uIEV2ZW50IFRp
bWVyPiBpb21lbSAweGZlZDAwMDAwLTB4ZmVkMDAzZmYgb24gYWNwaTANCmhwZXQwOiB2ZW5k
b3IgMHhmZmZmLCByZXYgMHhmZiwgMjMyODMxSHogNjRiaXQsIDMyIHRpbWVycywgbGVnYWN5
IHJvdXRlDQpocGV0MDogIHQwOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwg
cGVyaW9kaWMNCmhwZXQwOiAgdDE6IGlycXMgMHhmZmZmZmZmZiAoMzEpLCBNU0ksIDY0Yml0
LCBwZXJpb2RpYw0KaHBldDA6ICB0MjogaXJxcyAweGZmZmZmZmZmICgzMSksIE1TSSwgNjRi
aXQsIHBlcmlvZGljDQpocGV0MDogIHQzOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2
NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDQ6IGlycXMgMHhmZmZmZmZmZiAoMzEpLCBNU0ks
IDY0Yml0LCBwZXJpb2RpYw0KaHBldDA6ICB0NTogaXJxcyAweGZmZmZmZmZmICgzMSksIE1T
SSwgNjRiaXQsIHBlcmlvZGljDQpocGV0MDogIHQ2OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwg
TVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDc6IGlycXMgMHhmZmZmZmZmZiAoMzEp
LCBNU0ksIDY0Yml0LCBwZXJpb2RpYw0KaHBldDA6ICB0ODogaXJxcyAweGZmZmZmZmZmICgz
MSksIE1TSSwgNjRiaXQsIHBlcmlvZGljDQpocGV0MDogIHQ5OiBpcnFzIDB4ZmZmZmZmZmYg
KDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDEwOiBpcnFzIDB4ZmZmZmZm
ZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDExOiBpcnFzIDB4ZmZm
ZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDEyOiBpcnFzIDB4
ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDEzOiBpcnFz
IDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDE0OiBp
cnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDE1
OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAg
dDE2OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQw
OiAgdDE3OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhw
ZXQwOiAgdDE4OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMN
CmhwZXQwOiAgdDE5OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9k
aWMNCmhwZXQwOiAgdDIwOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVy
aW9kaWMNCmhwZXQwOiAgdDIxOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwg
cGVyaW9kaWMNCmhwZXQwOiAgdDIyOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJp
dCwgcGVyaW9kaWMNCmhwZXQwOiAgdDIzOiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2
NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI0OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwgTVNJ
LCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI1OiBpcnFzIDB4ZmZmZmZmZmYgKDMxKSwg
TVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI2OiBpcnFzIDB4ZmZmZmZmZmYgKDMx
KSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI3OiBpcnFzIDB4ZmZmZmZmZmYg
KDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI4OiBpcnFzIDB4ZmZmZmZm
ZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDI5OiBpcnFzIDB4ZmZm
ZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDMwOiBpcnFzIDB4
ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiAgdDMxOiBpcnFz
IDB4ZmZmZmZmZmYgKDMxKSwgTVNJLCA2NGJpdCwgcGVyaW9kaWMNCmhwZXQwOiBIUEVUIG5l
dmVyIGluY3JlbWVudHMsIGRpc2FibGluZw0KZGV2aWNlX2F0dGFjaDogaHBldDAgYXR0YWNo
IHJldHVybmVkIDYNCkFDUEkgdGltZXI6IC0+IDEwDQpUaW1lY291bnRlciAiQUNQSS1mYXN0
IiBmcmVxdWVuY3kgMzU3OTU0NSBIeiBxdWFsaXR5IDkwMA0KYWNwaV90aW1lcjA6IDwyNC1i
aXQgdGltZXIgYXQgMy41Nzk1NDVNSHo+IHBvcnQgMHg0MDgtMHg0MGIgb24gYWNwaTANCnBj
aV9saW5rMDogICAgICAgIEluZGV4ICBJUlEgIFJ0ZCAgUmVmICBJUlFzDQogIEluaXRpYWwg
UHJvYmUgICAgICAgMCAgIDEwICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQg
MTUNCiAgVmFsaWRhdGlvbiAgICAgICAgICAwICAgMTAgICBOICAgICAwICAzIDQgNSA2IDcg
OSAxMCAxMSAxMiAxNCAxNQ0KICBBZnRlciBEaXNhYmxlICAgICAgIDAgIDI1NSAgIE4gICAg
IDAgIDMgNCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazE6ICAgICAgICBJbmRl
eCAgSVJRICBSdGQgIFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2JlICAgICAgIDAgICAxMSAg
IE4gICAgIDAgIDMgNCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQogIFZhbGlkYXRpb24gICAg
ICAgICAgMCAgIDExICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCiAg
QWZ0ZXIgRGlzYWJsZSAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAx
MSAxMiAxNCAxNQ0KcGNpX2xpbmsyOiAgICAgICAgSW5kZXggIElSUSAgUnRkICBSZWYgIElS
UXMNCiAgSW5pdGlhbCBQcm9iZSAgICAgICAwICAgIDUgICBOICAgICAwICAzIDQgNSA2IDcg
OSAxMCAxMSAxMiAxNCAxNQ0KICBWYWxpZGF0aW9uICAgICAgICAgIDAgICAgNSAgIE4gICAg
IDAgIDMgNCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQogIEFmdGVyIERpc2FibGUgICAgICAg
MCAgMjU1ICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCnBjaV9saW5r
MzogICAgICAgIEluZGV4ICBJUlEgIFJ0ZCAgUmVmICBJUlFzDQogIEluaXRpYWwgUHJvYmUg
ICAgICAgMCAgIDExICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCiAg
VmFsaWRhdGlvbiAgICAgICAgICAwICAgMTEgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAx
MSAxMiAxNCAxNQ0KICBBZnRlciBEaXNhYmxlICAgICAgIDAgIDI1NSAgIE4gICAgIDAgIDMg
NCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazQ6ICAgICAgICBJbmRleCAgSVJR
ICBSdGQgIFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2JlICAgICAgIDAgIDI1NSAgIE4gICAg
IDAgIDMgNCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQogIFZhbGlkYXRpb24gICAgICAgICAg
MCAgMjU1ICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCiAgQWZ0ZXIg
RGlzYWJsZSAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAxMSAxMiAx
NCAxNQ0KcGNpX2xpbms1OiAgICAgICAgSW5kZXggIElSUSAgUnRkICBSZWYgIElSUXMNCiAg
SW5pdGlhbCBQcm9iZSAgICAgICAwICAgIDkgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAx
MSAxMiAxNCAxNQ0KICBWYWxpZGF0aW9uICAgICAgICAgIDAgICAgOSAgIE4gICAgIDAgIDMg
NCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQogIEFmdGVyIERpc2FibGUgICAgICAgMCAgMjU1
ICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCnBjaV9saW5rNjogICAg
ICAgIEluZGV4ICBJUlEgIFJ0ZCAgUmVmICBJUlFzDQogIEluaXRpYWwgUHJvYmUgICAgICAg
MCAgMjU1ICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCiAgVmFsaWRh
dGlvbiAgICAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAxMSAxMiAx
NCAxNQ0KICBBZnRlciBEaXNhYmxlICAgICAgIDAgIDI1NSAgIE4gICAgIDAgIDMgNCA1IDYg
NyA5IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazc6ICAgICAgICBJbmRleCAgSVJRICBSdGQg
IFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2JlICAgICAgIDAgICAxMCAgIE4gICAgIDAgIDMg
NCA1IDYgNyA5IDEwIDExIDEyIDE0IDE1DQogIFZhbGlkYXRpb24gICAgICAgICAgMCAgIDEw
ICAgTiAgICAgMCAgMyA0IDUgNiA3IDkgMTAgMTEgMTIgMTQgMTUNCiAgQWZ0ZXIgRGlzYWJs
ZSAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNSA2IDcgOSAxMCAxMSAxMiAxNCAxNQ0K
cGNpYjA6IDxBQ1BJIEhvc3QtUENJIGJyaWRnZT4gcG9ydCAweGNmOC0weGNmZiBvbiBhY3Bp
MA0KcGNpYjA6IGRlY29kaW5nIDUgcmFuZ2UgMC0weGZlDQpwY2liMDogZGVjb2RpbmcgNCBy
YW5nZSAwLTB4Y2Y3DQpwY2liMDogZGVjb2RpbmcgNCByYW5nZSAweGQwMC0weGZmZmYNCnBj
aWIwOiBkZWNvZGluZyAzIHJhbmdlIDB4YTAwMDAtMHhiZmZmZg0KcGNpYjA6IGRlY29kaW5n
IDMgcmFuZ2UgMHhjNDAwMC0weGNiZmZmDQpwY2liMDogZGVjb2RpbmcgMyByYW5nZSAweGZl
ZDQwMDAwLTB4ZmVkZmZmZmYNCnBjaWIwOiBkZWNvZGluZyAzIHJhbmdlIDB4YjAwMDAwMDAt
MHhmZGZmZmZmZg0KcGNpMDogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjANCnBjaTA6IGRvbWFp
bj0wLCBwaHlzaWNhbCBidXM9MA0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHhkMTMw
LCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTAsIGZ1bmM9MA0KCWNsYXNz
PTA2LTAwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMTQwLCBzdGF0
cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5z
KSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCmZvdW5kLT4JdmVu
ZG9yPTB4ODA4NiwgZGV2PTB4ZDEzOCwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9MCwg
c2xvdD0zLCBmdW5jPTANCgljbGFzcz0wNi0wNC0wMCwgaGRydHlwZT0weDAxLCBtZmRldj0w
DQoJY21kcmVnPTB4MDE0Nywgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej0wIChkd29yZHMp
DQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4
MDAgKDAgbnMpDQoJaW50cGluPWEsIGlycT0xMA0KCXBvd2Vyc3BlYyAzICBzdXBwb3J0cyBE
MCBEMyAgY3VycmVudCBEMA0KCU1TSSBzdXBwb3J0cyAyIG1lc3NhZ2VzLCB2ZWN0b3IgbWFz
a3MNCnBjaWIwOiBtYXRjaGVkIGVudHJ5IGZvciAwLjMuSU5UQQ0KcGNpYjA6IHNsb3QgMyBJ
TlRBIGhhcmR3aXJlZCB0byBJUlEgMTYNCglzZWNidXM9MSwgc3ViYnVzPTENCmZvdW5kLT4J
dmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1NSwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9
MCwgc2xvdD04LCBmdW5jPTANCgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAwLCBtZmRl
dj0xDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej0wIChkd29y
ZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0
PTB4MDAgKDAgbnMpDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weGQxNTYsIHJldmlk
PTB4MTENCglkb21haW49MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0xDQoJY2xhc3M9MDgtODAt
MDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNtZHJlZz0weDAwMDAsIHN0YXRyZWc9MHgw
MDEwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5n
bnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KZm91bmQtPgl2ZW5kb3I9MHg4
MDg2LCBkZXY9MHhkMTU3LCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTgs
IGZ1bmM9Mg0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTENCgljbWRy
ZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0
aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBu
cykNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1OCwgcmV2aWQ9MHgxMQ0KCWRv
bWFpbj0wLCBidXM9MCwgc2xvdD04LCBmdW5jPTMNCgljbGFzcz0wOC04MC0wMCwgaGRydHlw
ZT0weDAwLCBtZmRldj0xDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAwMDAsIGNhY2hl
bG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgw
IG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0w
eGQxNTAsIHJldmlkPTB4MTENCglkb21haW49MCwgYnVzPTAsIHNsb3Q9MTYsIGZ1bmM9MA0K
CWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTENCgljbWRyZWc9MHgwMDAw
LCBzdGF0cmVnPTB4MDAwMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDAw
ICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCmZvdW5k
LT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1MSwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBi
dXM9MCwgc2xvdD0xNiwgZnVuYz0xDQoJY2xhc3M9MDgtODAtMDAsIGhkcnR5cGU9MHgwMCwg
bWZkZXY9MQ0KCWNtZHJlZz0weDAwMDAsIHN0YXRyZWc9MHgwMDAwLCBjYWNoZWxuc3o9MCAo
ZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1h
eGxhdD0weDAwICgwIG5zKQ0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHgxMGVmLCBy
ZXZpZD0weDA1DQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTI1LCBmdW5jPTANCgljbGFzcz0w
Mi0wMC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVnPTB4MDAwNywgc3RhdHJl
Zz0weDAwMTAsIGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyks
IG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWEsIGly
cT0xMA0KCXBvd2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBz
dXBwb3J0cyAxIG1lc3NhZ2UsIDY0IGJpdA0KCW1hcFsxMF06IHR5cGUgTWVtb3J5LCByYW5n
ZSAzMiwgYmFzZSAweGIxMjAwMDAwLCBzaXplIDE3LCBlbmFibGVkDQpwY2liMDogYWxsb2Nh
dGVkIHR5cGUgMyAoMHhiMTIwMDAwMC0weGIxMjFmZmZmKSBmb3IgcmlkIDEwIG9mIHBjaTA6
MDoyNTowDQoJbWFwWzE0XTogdHlwZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4YjEyMjUw
MDAsIHNpemUgMTIsIGVuYWJsZWQNCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGIxMjI1
MDAwLTB4YjEyMjVmZmYpIGZvciByaWQgMTQgb2YgcGNpMDowOjI1OjANCgltYXBbMThdOiB0
eXBlIEkvTyBQb3J0LCByYW5nZSAzMiwgYmFzZSAweDMwNDAsIHNpemUgIDUsIGVuYWJsZWQN
CnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSA0ICgweDMwNDAtMHgzMDVmKSBmb3IgcmlkIDE4IG9m
IHBjaTA6MDoyNTowDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3IgMC4yNS5JTlRBDQpwY2li
MDogc2xvdCAyNSBJTlRBIGhhcmR3aXJlZCB0byBJUlEgMTYNCmZvdW5kLT4JdmVuZG9yPTB4
ODA4NiwgZGV2PTB4M2IzYywgcmV2aWQ9MHgwNQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0y
NiwgZnVuYz0wDQoJY2xhc3M9MGMtMDMtMjAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MA0KCWNt
ZHJlZz0weDAxNDYsIHN0YXRyZWc9MHgwMjkwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxh
dHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgw
IG5zKQ0KCWludHBpbj1hLCBpcnE9OQ0KCXBvd2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMyAg
Y3VycmVudCBEMA0KCW1hcFsxMF06IHR5cGUgTWVtb3J5LCByYW5nZSAzMiwgYmFzZSAweGIx
MjIyMDAwLCBzaXplIDEwLCBlbmFibGVkDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhi
MTIyMjAwMC0weGIxMjIyM2ZmKSBmb3IgcmlkIDEwIG9mIHBjaTA6MDoyNjowDQpwY2liMDog
bWF0Y2hlZCBlbnRyeSBmb3IgMC4yNi5JTlRBDQpwY2liMDogc2xvdCAyNiBJTlRBIGhhcmR3
aXJlZCB0byBJUlEgMjENCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2I0MiwgcmV2
aWQ9MHgwNQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0yOCwgZnVuYz0wDQoJY2xhc3M9MDYt
MDQtMDAsIGhkcnR5cGU9MHgwMSwgbWZkZXY9MQ0KCWNtZHJlZz0weDAxNDcsIHN0YXRyZWc9
MHgwMDEwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBt
aW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9
MTANCglwb3dlcnNwZWMgMiAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kgc3Vw
cG9ydHMgMSBtZXNzYWdlDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3IgMC4yOC5JTlRBDQpw
Y2liMDogc2xvdCAyOCBJTlRBIGhhcmR3aXJlZCB0byBJUlEgMTYNCglzZWNidXM9Miwgc3Vi
YnVzPTINCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2I0YSwgcmV2aWQ9MHgwNQ0K
CWRvbWFpbj0wLCBidXM9MCwgc2xvdD0yOCwgZnVuYz00DQoJY2xhc3M9MDYtMDQtMDAsIGhk
cnR5cGU9MHgwMSwgbWZkZXY9MQ0KCWNtZHJlZz0weDAxNDcsIHN0YXRyZWc9MHgwMDEwLCBj
YWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgw
MCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1iLCBpcnE9MTANCglwb3dl
cnNwZWMgMiAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kgc3VwcG9ydHMgMSBt
ZXNzYWdlDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3IgMC4yOC5JTlRCDQpwY2liMDogc2xv
dCAyOCBJTlRCIGhhcmR3aXJlZCB0byBJUlEgMTYNCglzZWNidXM9Mywgc3ViYnVzPTMNCmZv
dW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2I0ZSwgcmV2aWQ9MHgwNQ0KCWRvbWFpbj0w
LCBidXM9MCwgc2xvdD0yOCwgZnVuYz02DQoJY2xhc3M9MDYtMDQtMDAsIGhkcnR5cGU9MHgw
MSwgbWZkZXY9MQ0KCWNtZHJlZz0weDAwMDcsIHN0YXRyZWc9MHgwMDEwLCBjYWNoZWxuc3o9
MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyks
IG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1jLCBpcnE9NQ0KCXBvd2Vyc3BlYyAyICBz
dXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBzdXBwb3J0cyAxIG1lc3NhZ2UNCnBj
aWIwOiBtYXRjaGVkIGVudHJ5IGZvciAwLjI4LklOVEMNCnBjaWIwOiBzbG90IDI4IElOVEMg
aGFyZHdpcmVkIHRvIElSUSAxOA0KCXNlY2J1cz00LCBzdWJidXM9NA0KZm91bmQtPgl2ZW5k
b3I9MHg4MDg2LCBkZXY9MHgzYjUwLCByZXZpZD0weDA1DQoJZG9tYWluPTAsIGJ1cz0wLCBz
bG90PTI4LCBmdW5jPTcNCgljbGFzcz0wNi0wNC0wMCwgaGRydHlwZT0weDAxLCBtZmRldj0x
DQoJY21kcmVnPTB4MDE0Nywgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej0wIChkd29yZHMp
DQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4
MDAgKDAgbnMpDQoJaW50cGluPWQsIGlycT0xMQ0KCXBvd2Vyc3BlYyAyICBzdXBwb3J0cyBE
MCBEMyAgY3VycmVudCBEMA0KCU1TSSBzdXBwb3J0cyAxIG1lc3NhZ2UNCnBjaWIwOiBtYXRj
aGVkIGVudHJ5IGZvciAwLjI4LklOVEQNCnBjaWIwOiBzbG90IDI4IElOVEQgaGFyZHdpcmVk
IHRvIElSUSAxOQ0KCXNlY2J1cz01LCBzdWJidXM9NQ0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2
LCBkZXY9MHgzYjM0LCByZXZpZD0weDA1DQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTI5LCBm
dW5jPTANCgljbGFzcz0wYy0wMy0yMCwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVn
PTB4MDE0Niwgc3RhdHJlZz0weDAyOTAsIGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0dGlt
ZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMp
DQoJaW50cGluPWEsIGlycT0xMA0KCXBvd2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMyAgY3Vy
cmVudCBEMA0KCW1hcFsxMF06IHR5cGUgTWVtb3J5LCByYW5nZSAzMiwgYmFzZSAweGIxMjIx
MDAwLCBzaXplIDEwLCBlbmFibGVkDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiMTIy
MTAwMC0weGIxMjIxM2ZmKSBmb3IgcmlkIDEwIG9mIHBjaTA6MDoyOTowDQpwY2liMDogbWF0
Y2hlZCBlbnRyeSBmb3IgMC4yOS5JTlRBDQpwY2liMDogc2xvdCAyOSBJTlRBIGhhcmR3aXJl
ZCB0byBJUlEgMjMNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4MjQ0ZSwgcmV2aWQ9
MHhhNQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0zMCwgZnVuYz0wDQoJY2xhc3M9MDYtMDQt
MDEsIGhkcnR5cGU9MHgwMSwgbWZkZXY9MA0KCWNtZHJlZz0weDAxNDcsIHN0YXRyZWc9MHgw
MDEwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5n
bnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCXNlY2J1cz02LCBzdWJidXM9
Ng0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHgzYjE0LCByZXZpZD0weDA1DQoJZG9t
YWluPTAsIGJ1cz0wLCBzbG90PTMxLCBmdW5jPTANCgljbGFzcz0wNi0wMS0wMCwgaGRydHlw
ZT0weDAwLCBtZmRldj0xDQoJY21kcmVnPTB4MDE0Nywgc3RhdHJlZz0weDAyMTAsIGNhY2hl
bG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgw
IG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0w
eDNiMjIsIHJldmlkPTB4MDUNCglkb21haW49MCwgYnVzPTAsIHNsb3Q9MzEsIGZ1bmM9Mg0K
CWNsYXNzPTAxLTA2LTAxLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMDA3
LCBzdGF0cmVnPTB4MDJiMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDAw
ICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCglpbnRw
aW49YiwgaXJxPTUNCglwb3dlcnNwZWMgMyAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDAN
CglNU0kgc3VwcG9ydHMgMSBtZXNzYWdlDQoJbWFwWzEwXTogdHlwZSBJL08gUG9ydCwgcmFu
Z2UgMzIsIGJhc2UgMHgzMDY4LCBzaXplICAzLCBlbmFibGVkDQpwY2liMDogYWxsb2NhdGVk
IHR5cGUgNCAoMHgzMDY4LTB4MzA2ZikgZm9yIHJpZCAxMCBvZiBwY2kwOjA6MzE6Mg0KCW1h
cFsxNF06IHR5cGUgSS9PIFBvcnQsIHJhbmdlIDMyLCBiYXNlIDB4MzA3NCwgc2l6ZSAgMiwg
ZW5hYmxlZA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDQgKDB4MzA3NC0weDMwNzcpIGZvciBy
aWQgMTQgb2YgcGNpMDowOjMxOjINCgltYXBbMThdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAz
MiwgYmFzZSAweDMwNjAsIHNpemUgIDMsIGVuYWJsZWQNCnBjaWIwOiBhbGxvY2F0ZWQgdHlw
ZSA0ICgweDMwNjAtMHgzMDY3KSBmb3IgcmlkIDE4IG9mIHBjaTA6MDozMToyDQoJbWFwWzFj
XTogdHlwZSBJL08gUG9ydCwgcmFuZ2UgMzIsIGJhc2UgMHgzMDcwLCBzaXplICAyLCBlbmFi
bGVkDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgNCAoMHgzMDcwLTB4MzA3MykgZm9yIHJpZCAx
YyBvZiBwY2kwOjA6MzE6Mg0KCW1hcFsyMF06IHR5cGUgSS9PIFBvcnQsIHJhbmdlIDMyLCBi
YXNlIDB4MzAyMCwgc2l6ZSAgNSwgZW5hYmxlZA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDQg
KDB4MzAyMC0weDMwM2YpIGZvciByaWQgMjAgb2YgcGNpMDowOjMxOjINCgltYXBbMjRdOiB0
eXBlIE1lbW9yeSwgcmFuZ2UgMzIsIGJhc2UgMHhiMTIyMDAwMCwgc2l6ZSAxMSwgZW5hYmxl
ZA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjEyMjAwMDAtMHhiMTIyMDdmZikgZm9y
IHJpZCAyNCBvZiBwY2kwOjA6MzE6Mg0KcGNpYjA6IG1hdGNoZWQgZW50cnkgZm9yIDAuMzEu
SU5UQg0KcGNpYjA6IHNsb3QgMzEgSU5UQiBoYXJkd2lyZWQgdG8gSVJRIDE4DQpmb3VuZC0+
CXZlbmRvcj0weDgwODYsIGRldj0weDNiMzAsIHJldmlkPTB4MDUNCglkb21haW49MCwgYnVz
PTAsIHNsb3Q9MzEsIGZ1bmM9Mw0KCWNsYXNzPTBjLTA1LTAwLCBoZHJ0eXBlPTB4MDAsIG1m
ZGV2PTANCgljbWRyZWc9MHgwMDAzLCBzdGF0cmVnPTB4MDI4MCwgY2FjaGVsbnN6PTAgKGR3
b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhs
YXQ9MHgwMCAoMCBucykNCglpbnRwaW49YiwgaXJxPTUNCgltYXBbMTBdOiB0eXBlIE1lbW9y
eSwgcmFuZ2UgNjQsIGJhc2UgMHhiMTIyMzAwMCwgc2l6ZSAgOCwgZW5hYmxlZA0KcGNpYjA6
IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjEyMjMwMDAtMHhiMTIyMzBmZikgZm9yIHJpZCAxMCBv
ZiBwY2kwOjA6MzE6Mw0KCW1hcFsyMF06IHR5cGUgSS9PIFBvcnQsIHJhbmdlIDMyLCBiYXNl
IDB4MzAwMCwgc2l6ZSAgNSwgZW5hYmxlZA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDQgKDB4
MzAwMC0weDMwMWYpIGZvciByaWQgMjAgb2YgcGNpMDowOjMxOjMNCnBjaWIwOiBtYXRjaGVk
IGVudHJ5IGZvciAwLjMxLklOVEINCnBjaWIwOiBzbG90IDMxIElOVEIgaGFyZHdpcmVkIHRv
IElSUSAxOA0KcGNpYjE6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBpcnEgMTYgYXQgZGV2aWNl
IDMuMCBvbiBwY2kwDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgNCAoMHgyMDAwLTB4MmZmZikg
Zm9yIHJpZCAxYyBvZiBwY2liMQ0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjEwMDAw
MDAtMHhiMTBmZmZmZikgZm9yIHJpZCAyNCBvZiBwY2liMQ0KcGNpYjE6ICAgZG9tYWluICAg
ICAgICAgICAgMA0KcGNpYjE6ICAgc2Vjb25kYXJ5IGJ1cyAgICAgMQ0KcGNpYjE6ICAgc3Vi
b3JkaW5hdGUgYnVzICAgMQ0KcGNpYjE6ICAgSS9PIGRlY29kZSAgICAgICAgMHgyMDAwLTB4
MmZmZg0KcGNpYjE6ICAgcHJlZmV0Y2hlZCBkZWNvZGUgMHhiMTAwMDAwMC0weGIxMGZmZmZm
DQpwY2kxOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMQ0KcGNpYjE6IGFsbG9jYXRlZCBidXMg
cmFuZ2UgKDEtMSkgZm9yIHJpZCAwIG9mIHBjaTENCnBjaTE6IGRvbWFpbj0wLCBwaHlzaWNh
bCBidXM9MQ0KZm91bmQtPgl2ZW5kb3I9MHgxYzAwLCBkZXY9MHgzMjUwLCByZXZpZD0weDEw
DQoJZG9tYWluPTAsIGJ1cz0xLCBzbG90PTAsIGZ1bmM9MA0KCWNsYXNzPTA3LTAwLTA1LCBo
ZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMDA3LCBzdGF0cmVnPTB4MDAxMCwg
Y2FjaGVsbnN6PTE2IChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0w
eDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWEsIGlycT0xMA0KCXBv
d2Vyc3BlYyAzICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBzdXBwb3J0cyAz
MiBtZXNzYWdlcywgNjQgYml0LCB2ZWN0b3IgbWFza3MNCgltYXBbMTBdOiB0eXBlIEkvTyBQ
b3J0LCByYW5nZSAzMiwgYmFzZSAweDIwMDAsIHNpemUgIDgsIGVuYWJsZWQNCnBjaWIxOiBh
bGxvY2F0ZWQgSS9PIHBvcnQgcmFuZ2UgKDB4MjAwMC0weDIwZmYpIGZvciByaWQgMTAgb2Yg
cGNpMDoxOjA6MA0KCW1hcFsxNF06IHR5cGUgUHJlZmV0Y2hhYmxlIE1lbW9yeSwgcmFuZ2Ug
MzIsIGJhc2UgMHhiMTAwMDAwMCwgc2l6ZSAxNSwgZW5hYmxlZA0KcGNpYjE6IGFsbG9jYXRl
ZCBwcmVmZXRjaCByYW5nZSAoMHhiMTAwMDAwMC0weGIxMDA3ZmZmKSBmb3IgcmlkIDE0IG9m
IHBjaTA6MTowOjANCgltYXBbMThdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAzMiwgYmFzZSAw
eDIxMDAsIHNpemUgIDIsIGVuYWJsZWQNCnBjaWIxOiBhbGxvY2F0ZWQgSS9PIHBvcnQgcmFu
Z2UgKDB4MjEwMC0weDIxMDMpIGZvciByaWQgMTggb2YgcGNpMDoxOjA6MA0KcGNpYjE6IG1h
dGNoZWQgZW50cnkgZm9yIDEuMC5JTlRBDQpwY2liMTogc2xvdCAwIElOVEEgaGFyZHdpcmVk
IHRvIElSUSAxNg0KcGNpMTogPHNpbXBsZSBjb21tcywgVUFSVD4gYXQgZGV2aWNlIDAuMCAo
bm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNl
IDguMCAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQg
ZGV2aWNlIDguMSAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJh
bD4gYXQgZGV2aWNlIDguMiAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVy
aXBoZXJhbD4gYXQgZGV2aWNlIDguMyAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJh
c2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDE2LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkNCnBj
aTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSAxNi4xIChubyBkcml2ZXIgYXR0YWNo
ZWQpDQplbTA6IDxJbnRlbChSKSA4MjU3OERNPiBwb3J0IDB4MzA0MC0weDMwNWYgbWVtIDB4
YjEyMDAwMDAtMHhiMTIxZmZmZiwweGIxMjI1MDAwLTB4YjEyMjVmZmYgaXJxIDE2IGF0IGRl
dmljZSAyNS4wIG9uIHBjaTANCmVtMDogYXR0YWNoX3ByZSBjYXBwaW5nIHF1ZXVlcyBhdCAx
DQplbTA6IEVFUFJPTSBWMC4xMC0yDQplbTA6IFVzaW5nIDEwMjQgVFggZGVzY3JpcHRvcnMg
YW5kIDEwMjQgUlggZGVzY3JpcHRvcnMNCmVtMDogbXNpeF9pbml0IHFzZXRzIGNhcHBlZCBh
dCAxDQplbTA6IGF0dGVtcHRpbmcgdG8gYWxsb2NhdGUgMSBNU0kgdmVjdG9ycyAoMSBzdXBw
b3J0ZWQpDQptc2k6IHJvdXRpbmcgTVNJIElSUSAyNCB0byBsb2NhbCBBUElDIDYgdmVjdG9y
IDQ4DQplbTA6IHVzaW5nIElSUSAyNCBmb3IgTVNJDQplbTA6IFVzaW5nIGFuIE1TSSBpbnRl
cnJ1cHQNCmVtMDogYWxsb2NhdGVkIGZvciAxIHR4X3F1ZXVlcw0KZW0wOiBhbGxvY2F0ZWQg
Zm9yIDEgcnhfcXVldWVzDQptc2k6IEFzc2lnbmluZyBNU0kgSVJRIDI0IHRvIGxvY2FsIEFQ
SUMgNCB2ZWN0b3IgNDkNCmVtMDogYnBmIGF0dGFjaGVkDQplbTA6IEV0aGVybmV0IGFkZHJl
c3M6IDAwOjFlOjY3OjAyOjA3OjA3DQplbTA6IG5ldG1hcCBxdWV1ZXMvc2xvdHM6IFRYIDEv
MTAyNCwgUlggMS8xMDI0DQplaGNpMDogPEludGVsIFBDSCBVU0IgMi4wIGNvbnRyb2xsZXIg
VVNCLUI+IG1lbSAweGIxMjIyMDAwLTB4YjEyMjIzZmYgaXJxIDIxIGF0IGRldmljZSAyNi4w
IG9uIHBjaTANCmlvYXBpYzA6IHJvdXRpbmcgaW50cGluIDIxIChQQ0kgSVJRIDIxKSB0byBs
YXBpYyAwIHZlY3RvciA1MA0KdXNidXMwOiBFSENJIHZlcnNpb24gMS4wDQp1c2J1czAgb24g
ZWhjaTANCmVoY2kwOiB1c2JwZjogQXR0YWNoZWQNCnVzYnVzMDogNDgwTWJwcyBIaWdoIFNw
ZWVkIFVTQiB2Mi4wDQpwY2liMjogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGlycSAxNiBhdCBk
ZXZpY2UgMjguMCBvbiBwY2kwDQpwY2liMjogICBkb21haW4gICAgICAgICAgICAwDQpwY2li
MjogICBzZWNvbmRhcnkgYnVzICAgICAyDQpwY2liMjogICBzdWJvcmRpbmF0ZSBidXMgICAy
DQpwY2kyOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMg0KcGNpYjI6IGFsbG9jYXRlZCBidXMg
cmFuZ2UgKDItMikgZm9yIHJpZCAwIG9mIHBjaTINCnBjaTI6IGRvbWFpbj0wLCBwaHlzaWNh
bCBidXM9Mg0KcGNpYjM6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBpcnEgMTYgYXQgZGV2aWNl
IDI4LjQgb24gcGNpMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDQgKDB4MTAwMC0weDFmZmYp
IGZvciByaWQgMWMgb2YgcGNpYjMNCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGIxMTAw
MDAwLTB4YjExZmZmZmYpIGZvciByaWQgMjAgb2YgcGNpYjMNCnBjaWIzOiAgIGRvbWFpbiAg
ICAgICAgICAgIDANCnBjaWIzOiAgIHNlY29uZGFyeSBidXMgICAgIDMNCnBjaWIzOiAgIHN1
Ym9yZGluYXRlIGJ1cyAgIDMNCnBjaWIzOiAgIEkvTyBkZWNvZGUgICAgICAgIDB4MTAwMC0w
eDFmZmYNCnBjaWIzOiAgIG1lbW9yeSBkZWNvZGUgICAgIDB4YjExMDAwMDAtMHhiMTFmZmZm
Zg0KcGNpMzogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjMNCnBjaWIzOiBhbGxvY2F0ZWQgYnVz
IHJhbmdlICgzLTMpIGZvciByaWQgMCBvZiBwY2kzDQpwY2kzOiBkb21haW49MCwgcGh5c2lj
YWwgYnVzPTMNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4MTBkMywgcmV2aWQ9MHgw
MA0KCWRvbWFpbj0wLCBidXM9Mywgc2xvdD0wLCBmdW5jPTANCgljbGFzcz0wMi0wMC0wMCwg
aGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVnPTB4MDAwNywgc3RhdHJlZz0weDAwMTAs
IGNhY2hlbG5zej0xNiAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9
MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9MTANCglw
b3dlcnNwZWMgMiAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kgc3VwcG9ydHMg
MSBtZXNzYWdlLCA2NCBiaXQNCglNU0ktWCBzdXBwb3J0cyA1IG1lc3NhZ2VzIGluIG1hcCAw
eDFjDQoJbWFwWzEwXTogdHlwZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4YjExMDAwMDAs
IHNpemUgMTcsIGVuYWJsZWQNCnBjaWIzOiBhbGxvY2F0ZWQgbWVtb3J5IHJhbmdlICgweGIx
MTAwMDAwLTB4YjExMWZmZmYpIGZvciByaWQgMTAgb2YgcGNpMDozOjA6MA0KCW1hcFsxOF06
IHR5cGUgSS9PIFBvcnQsIHJhbmdlIDMyLCBiYXNlIDB4MTAwMCwgc2l6ZSAgNSwgZW5hYmxl
ZA0KcGNpYjM6IGFsbG9jYXRlZCBJL08gcG9ydCByYW5nZSAoMHgxMDAwLTB4MTAxZikgZm9y
IHJpZCAxOCBvZiBwY2kwOjM6MDowDQoJbWFwWzFjXTogdHlwZSBNZW1vcnksIHJhbmdlIDMy
LCBiYXNlIDB4YjExMjAwMDAsIHNpemUgMTQsIGVuYWJsZWQNCnBjaWIzOiBhbGxvY2F0ZWQg
bWVtb3J5IHJhbmdlICgweGIxMTIwMDAwLTB4YjExMjNmZmYpIGZvciByaWQgMWMgb2YgcGNp
MDozOjA6MA0KcGNpYjM6IG1hdGNoZWQgZW50cnkgZm9yIDMuMC5JTlRBDQpwY2liMzogc2xv
dCAwIElOVEEgaGFyZHdpcmVkIHRvIElSUSAxNg0KZW0xOiA8SW50ZWwoUikgR2lnYWJpdCBD
VCA4MjU3NEw+IHBvcnQgMHgxMDAwLTB4MTAxZiBtZW0gMHhiMTEwMDAwMC0weGIxMTFmZmZm
LDB4YjExMjAwMDAtMHhiMTEyM2ZmZiBpcnEgMTYgYXQgZGV2aWNlIDAuMCBvbiBwY2kzDQpl
bTE6IGF0dGFjaF9wcmUgY2FwcGluZyBxdWV1ZXMgYXQgMg0KZW0xOiBFTV9OVk1fUENJRV9D
VFJMID0gMHg0NjBiDQplbTE6IEVFUFJPTSBWMi4xLTANCmVtMTogVXNpbmcgMTAyNCBUWCBk
ZXNjcmlwdG9ycyBhbmQgMTAyNCBSWCBkZXNjcmlwdG9ycw0KZW0xOiBtc2l4X2luaXQgcXNl
dHMgY2FwcGVkIGF0IDINCmVtMTogaW50ciBDUFVzOiA0IHF1ZXVlIG1zZ3M6IDQgYWRtaW5j
bnQ6IDENCmVtMTogVXNpbmcgMiBSWCBxdWV1ZXMgMiBUWCBxdWV1ZXMNCmVtMTogYXR0ZW1w
dGluZyB0byBhbGxvY2F0ZSAzIE1TSS1YIHZlY3RvcnMgKDUgc3VwcG9ydGVkKQ0KbXNpOiBy
b3V0aW5nIE1TSS1YIElSUSAyNSB0byBsb2NhbCBBUElDIDIgdmVjdG9yIDQ4DQptc2k6IHJv
dXRpbmcgTVNJLVggSVJRIDI2IHRvIGxvY2FsIEFQSUMgNCB2ZWN0b3IgNTANCm1zaTogcm91
dGluZyBNU0ktWCBJUlEgMjcgdG8gbG9jYWwgQVBJQyA2IHZlY3RvciA0OA0KZW0xOiB1c2lu
ZyBJUlFzIDI1LTI3IGZvciBNU0ktWA0KZW0xOiBVc2luZyBNU0ktWCBpbnRlcnJ1cHRzIHdp
dGggMyB2ZWN0b3JzDQplbTE6IGFsbG9jYXRlZCBmb3IgMiB0eF9xdWV1ZXMNCmVtMTogYWxs
b2NhdGVkIGZvciAyIHJ4X3F1ZXVlcw0KZW0xOiBicGYgYXR0YWNoZWQNCmVtMTogRXRoZXJu
ZXQgYWRkcmVzczogMDA6MWU6Njc6MDI6MDc6MDYNCmVtMTogbmV0bWFwIHF1ZXVlcy9zbG90
czogVFggMi8xMDI0LCBSWCAyLzEwMjQNCnBjaWI0OiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4g
aXJxIDE4IGF0IGRldmljZSAyOC42IG9uIHBjaTANCnBjaWI0OiAgIGRvbWFpbiAgICAgICAg
ICAgIDANCnBjaWI0OiAgIHNlY29uZGFyeSBidXMgICAgIDQNCnBjaWI0OiAgIHN1Ym9yZGlu
YXRlIGJ1cyAgIDQNCnBjaWI0OiBjb3VsZCBub3QgZ2V0IFBDSSBpbnRlcnJ1cHQgcm91dGlu
ZyB0YWJsZSBmb3IgXF9TQl8uUENJMC5QRVg2IC0gQUVfTk9UX0ZPVU5EDQpwY2k0OiA8QUNQ
SSBQQ0kgYnVzPiBvbiBwY2liNA0KcGNpYjQ6IGFsbG9jYXRlZCBidXMgcmFuZ2UgKDQtNCkg
Zm9yIHJpZCAwIG9mIHBjaTQNCnBjaTQ6IGRvbWFpbj0wLCBwaHlzaWNhbCBidXM9NA0KcGNp
YjU6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBpcnEgMTkgYXQgZGV2aWNlIDI4Ljcgb24gcGNp
MA0KcGNpYjU6ICAgZG9tYWluICAgICAgICAgICAgMA0KcGNpYjU6ICAgc2Vjb25kYXJ5IGJ1
cyAgICAgNQ0KcGNpYjU6ICAgc3Vib3JkaW5hdGUgYnVzICAgNQ0KcGNpYjU6IGNvdWxkIG5v
dCBnZXQgUENJIGludGVycnVwdCByb3V0aW5nIHRhYmxlIGZvciBcX1NCXy5QQ0kwLlBFWDcg
LSBBRV9OT1RfRk9VTkQNCnBjaTU6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWI1DQpwY2liNTog
YWxsb2NhdGVkIGJ1cyByYW5nZSAoNS01KSBmb3IgcmlkIDAgb2YgcGNpNQ0KcGNpNTogZG9t
YWluPTAsIHBoeXNpY2FsIGJ1cz01DQplaGNpMTogPEludGVsIFBDSCBVU0IgMi4wIGNvbnRy
b2xsZXIgVVNCLUE+IG1lbSAweGIxMjIxMDAwLTB4YjEyMjEzZmYgaXJxIDIzIGF0IGRldmlj
ZSAyOS4wIG9uIHBjaTANCmlvYXBpYzA6IHJvdXRpbmcgaW50cGluIDIzIChQQ0kgSVJRIDIz
KSB0byBsYXBpYyAwIHZlY3RvciA1MQ0KdXNidXMxOiBFSENJIHZlcnNpb24gMS4wDQp1c2J1
czEgb24gZWhjaTENCmVoY2kxOiB1c2JwZjogQXR0YWNoZWQNCnVzYnVzMTogNDgwTWJwcyBI
aWdoIFNwZWVkIFVTQiB2Mi4wDQpwY2liNjogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRl
dmljZSAzMC4wIG9uIHBjaTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGIwMDAwMDAw
LTB4YjBmZmZmZmYpIGZvciByaWQgMjAgb2YgcGNpYjYNCnBjaWI2OiAgIGRvbWFpbiAgICAg
ICAgICAgIDANCnBjaWI2OiAgIHNlY29uZGFyeSBidXMgICAgIDYNCnBjaWI2OiAgIHN1Ym9y
ZGluYXRlIGJ1cyAgIDYNCnBjaWI2OiAgIG1lbW9yeSBkZWNvZGUgICAgIDB4YjAwMDAwMDAt
MHhiMGZmZmZmZg0KcGNpYjY6ICAgc3BlY2lhbCBkZWNvZGUgICAgVkdBLCBzdWJ0cmFjdGl2
ZQ0KcGNpNjogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjYNCnBjaWI2OiBhbGxvY2F0ZWQgYnVz
IHJhbmdlICg2LTYpIGZvciByaWQgMCBvZiBwY2k2DQpwY2k2OiBkb21haW49MCwgcGh5c2lj
YWwgYnVzPTYNCmZvdW5kLT4JdmVuZG9yPTB4MTI2ZiwgZGV2PTB4MDcxMiwgcmV2aWQ9MHhi
MA0KCWRvbWFpbj0wLCBidXM9Niwgc2xvdD0zLCBmdW5jPTANCgljbGFzcz0wMy0wMC0wMCwg
aGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVnPTB4MDAxNywgc3RhdHJlZz0weDAyMzAs
IGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgyMCAoOTYwIG5zKSwgbWluZ250
PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCglpbnRwaW49YSwgaXJxPTExDQoJ
cG93ZXJzcGVjIDEgIHN1cHBvcnRzIEQwIEQxIEQyIEQzICBjdXJyZW50IEQwDQoJbWFwWzEw
XTogdHlwZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4YjAwMDAwMDAsIHNpemUgMjQsIGVu
YWJsZWQNCnBjaWI2OiBhbGxvY2F0ZWQgbWVtb3J5IHJhbmdlICgweGIwMDAwMDAwLTB4YjBm
ZmZmZmYpIGZvciByaWQgMTAgb2YgcGNpMDo2OjM6MA0KcGNpYjY6IG1hdGNoZWQgZW50cnkg
Zm9yIDYuMy5JTlRBDQpwY2liNjogc2xvdCAzIElOVEEgaGFyZHdpcmVkIHRvIElSUSAxNw0K
dmdhcGNpMDogPFZHQS1jb21wYXRpYmxlIGRpc3BsYXk+IG1lbSAweGIwMDAwMDAwLTB4YjBm
ZmZmZmYgaXJxIDE3IGF0IGRldmljZSAzLjAgb24gcGNpNg0KdmdhcGNpMDogQm9vdCB2aWRl
byBkZXZpY2UNCmlzYWIwOiA8UENJLUlTQSBicmlkZ2U+IGF0IGRldmljZSAzMS4wIG9uIHBj
aTANCmlzYTA6IDxJU0EgYnVzPiBvbiBpc2FiMA0KYWhjaTA6IDxJbnRlbCBJYmV4IFBlYWsg
QUhDSSBTQVRBIGNvbnRyb2xsZXI+IHBvcnQgMHgzMDY4LTB4MzA2ZiwweDMwNzQtMHgzMDc3
LDB4MzA2MC0weDMwNjcsMHgzMDcwLTB4MzA3MywweDMwMjAtMHgzMDNmIG1lbSAweGIxMjIw
MDAwLTB4YjEyMjA3ZmYgaXJxIDE4IGF0IGRldmljZSAzMS4yIG9uIHBjaTANCmFoY2kwOiBh
dHRlbXB0aW5nIHRvIGFsbG9jYXRlIDEgTVNJIHZlY3RvcnMgKDEgc3VwcG9ydGVkKQ0KbXNp
OiByb3V0aW5nIE1TSSBJUlEgMjggdG8gbG9jYWwgQVBJQyAyIHZlY3RvciA0OQ0KYWhjaTA6
IHVzaW5nIElSUSAyOCBmb3IgTVNJDQphaGNpMDogQUhDSSB2MS4zMCB3aXRoIDYgM0dicHMg
cG9ydHMsIFBvcnQgTXVsdGlwbGllciBub3Qgc3VwcG9ydGVkDQphaGNpMDogQ2FwczogNjRi
aXQgTkNRIFNOVEYgQUxQIEFMIENMTyAzR2JwcyBQTUQgU1NDIFBTQyAzMmNtZCBlU0FUQSA2
cG9ydHMNCmFoY2kwOiBDYXBzMjogQVBTVA0KYWhjaWNoMDogPEFIQ0kgY2hhbm5lbD4gYXQg
Y2hhbm5lbCAwIG9uIGFoY2kwDQphaGNpY2gwOiBDYXBzOiBIUENQIEVTUA0KYWhjaWNoMTog
PEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAxIG9uIGFoY2kwDQphaGNpY2gxOiBDYXBzOiBI
UENQIEVTUA0KYWhjaWNoMjogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAyIG9uIGFoY2kw
DQphaGNpY2gyOiBDYXBzOiBIUENQIEVTUA0KYWhjaWNoMzogPEFIQ0kgY2hhbm5lbD4gYXQg
Y2hhbm5lbCAzIG9uIGFoY2kwDQphaGNpY2gzOiBDYXBzOiBIUENQIEVTUA0KYWhjaWNoNDog
PEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCA0IG9uIGFoY2kwDQphaGNpY2g0OiBDYXBzOiBI
UENQIEVTUA0KYWhjaWNoNTogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCA1IG9uIGFoY2kw
DQphaGNpY2g1OiBDYXBzOiBIUENQIEVTUA0KcGNpMDogPHNlcmlhbCBidXMsIFNNQnVzPiBh
dCBkZXZpY2UgMzEuMyAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KYWNwaV9idXR0b24wOiA8U2xl
ZXAgQnV0dG9uPiBvbiBhY3BpMA0KQUNQSTogRW5hYmxlZCAyIEdQRXMgaW4gYmxvY2sgMDAg
dG8gM0YNCnhlbnB2MDogPFhlbiBQViBidXM+DQpncmFudHRhYmxlMDogPFhlbiBHcmFudC10
YWJsZSBEZXZpY2U+IG9uIHhlbnB2MA0KR3JhbnQgdGFibGUgaW5pdGlhbGl6ZWQNCnhjMDog
PFhlbiBDb25zb2xlPiBvbiB4ZW5wdjANCnhlbl9ldDA6IDxYZW4gUFYgQ2xvY2s+IG9uIHhl
bnB2MA0KRXZlbnQgdGltZXIgIlhFTlRJTUVSIiBmcmVxdWVuY3kgMTAwMDAwMDAwMCBIeiBx
dWFsaXR5IDk1MA0KVGltZWNvdW50ZXIgIlhFTlRJTUVSIiBmcmVxdWVuY3kgMTAwMDAwMDAw
MCBIeiBxdWFsaXR5IDk1MA0KeGVuX2V0MDogcmVnaXN0ZXJlZCBhcyBhIHRpbWUtb2YtZGF5
IGNsb2NrLCByZXNvbHV0aW9uIDAuMDAwMDAxcw0KeGVuc3RvcmUwOiA8WGVuU3RvcmU+IG9u
IHhlbnB2MA0KeHNkX2RldjA6IDxYZW5zdG9yZWQgdXNlci1zcGFjZSBkZXZpY2U+IG9uIHhl
bnB2MA0KZXZ0Y2huMDogPFhlbiBldmVudCBjaGFubmVsIHVzZXItc3BhY2UgZGV2aWNlPiBv
biB4ZW5wdjANCnByaXZjbWQwOiA8WGVuIHByaXZpbGVnZWQgaW50ZXJmYWNlIHVzZXItc3Bh
Y2UgZGV2aWNlPiBvbiB4ZW5wdjANCmdudGRldjA6IDxYZW4gZ3JhbnQtdGFibGUgdXNlci1z
cGFjZSBkZXZpY2U+IG9uIHhlbnB2MA0KZGVidWcwOiA8WGVuIGRlYnVnIGhhbmRsZXI+IG9u
IHhlbnB2MA0KYWhjX2lzYV9pZGVudGlmeSAwOiBpb3BvcnQgMHhjMDAgYWxsb2MgZmFpbGVk
DQphaGNfaXNhX2lkZW50aWZ5IDE6IGlvcG9ydCAweDFjMDAgYWxsb2MgZmFpbGVkDQphaGNf
aXNhX2lkZW50aWZ5IDI6IGlvcG9ydCAweDJjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lk
ZW50aWZ5IDM6IGlvcG9ydCAweDNjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5
IDQ6IGlvcG9ydCAweDRjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5IDU6IGlv
cG9ydCAweDVjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5IDY6IGlvcG9ydCAw
eDZjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5IDc6IGlvcG9ydCAweDdjMDAg
YWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5IDg6IGlvcG9ydCAweDhjMDAgYWxsb2Mg
ZmFpbGVkDQphaGNfaXNhX2lkZW50aWZ5IDk6IGlvcG9ydCAweDljMDAgYWxsb2MgZmFpbGVk
DQphaGNfaXNhX2lkZW50aWZ5IDEwOiBpb3BvcnQgMHhhYzAwIGFsbG9jIGZhaWxlZA0KYWhj
X2lzYV9pZGVudGlmeSAxMTogaW9wb3J0IDB4YmMwMCBhbGxvYyBmYWlsZWQNCmFoY19pc2Ff
aWRlbnRpZnkgMTI6IGlvcG9ydCAweGNjMDAgYWxsb2MgZmFpbGVkDQphaGNfaXNhX2lkZW50
aWZ5IDEzOiBpb3BvcnQgMHhkYzAwIGFsbG9jIGZhaWxlZA0KYWhjX2lzYV9pZGVudGlmeSAx
NDogaW9wb3J0IDB4ZWMwMCBhbGxvYyBmYWlsZWQNCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAz
ICgweGIwMDAwLTB4YjA3ZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVk
IHR5cGUgMyAoMHhiMDgwMC0weGIwZmZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFs
bG9jYXRlZCB0eXBlIDMgKDB4YjEwMDAtMHhiMTdmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBj
aWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGIxODAwLTB4YjFmZmYpIGZvciByaWQgMCBvZiBv
cm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiMjAwMC0weGIyN2ZmKSBmb3Igcmlk
IDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjI4MDAtMHhiMmZmZikg
Zm9yIHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGIzMDAwLTB4
YjM3ZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhi
MzgwMC0weGIzZmZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBl
IDMgKDB4YjQwMDAtMHhiNDdmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0
ZWQgdHlwZSAzICgweGI0ODAwLTB4YjRmZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDog
YWxsb2NhdGVkIHR5cGUgMyAoMHhiNTAwMC0weGI1N2ZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0K
cGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjU4MDAtMHhiNWZmZikgZm9yIHJpZCAwIG9m
IG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGI2MDAwLTB4YjY3ZmYpIGZvciBy
aWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiNjgwMC0weGI2ZmZm
KSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YjcwMDAt
MHhiNzdmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgw
eGI3ODAwLTB4YjdmZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5
cGUgMyAoMHhiODAwMC0weGI4N2ZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9j
YXRlZCB0eXBlIDMgKDB4Yjg4MDAtMHhiOGZmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBjaWIw
OiBhbGxvY2F0ZWQgdHlwZSAzICgweGI5MDAwLTB4Yjk3ZmYpIGZvciByaWQgMCBvZiBvcm0w
DQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiOTgwMC0weGI5ZmZmKSBmb3IgcmlkIDAg
b2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YmEwMDAtMHhiYTdmZikgZm9y
IHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGJhODAwLTB4YmFm
ZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiYjAw
MC0weGJiN2ZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMg
KDB4YmI4MDAtMHhiYmZmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQg
dHlwZSAzICgweGJjMDAwLTB4YmM3ZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxs
b2NhdGVkIHR5cGUgMyAoMHhiYzgwMC0weGJjZmZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KcGNp
YjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YmQwMDAtMHhiZDdmZikgZm9yIHJpZCAwIG9mIG9y
bTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGJkODAwLTB4YmRmZmYpIGZvciByaWQg
MCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgMyAoMHhiZTAwMC0weGJlN2ZmKSBm
b3IgcmlkIDAgb2Ygb3JtMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDMgKDB4YmU4MDAtMHhi
ZWZmZikgZm9yIHJpZCAwIG9mIG9ybTANCnBjaWIwOiBhbGxvY2F0ZWQgdHlwZSAzICgweGJm
MDAwLTB4YmY3ZmYpIGZvciByaWQgMCBvZiBvcm0wDQpwY2liMDogYWxsb2NhdGVkIHR5cGUg
MyAoMHhiZjgwMC0weGJmZmZmKSBmb3IgcmlkIDAgb2Ygb3JtMA0KaXNhX3Byb2JlX2NoaWxk
cmVuOiBkaXNhYmxpbmcgUG5QIGRldmljZXMNCmF0cnRjOiBhdHJ0YzAgYWxyZWFkeSBleGlz
dHM7IHNraXBwaW5nIGl0DQphdHRpbWVyOiBhdHRpbWVyMCBhbHJlYWR5IGV4aXN0czsgc2tp
cHBpbmcgaXQNCnNjOiBzYzAgYWxyZWFkeSBleGlzdHM7IHNraXBwaW5nIGl0DQppc2FfcHJv
YmVfY2hpbGRyZW46IHByb2Jpbmcgbm9uLVBuUCBkZXZpY2VzDQpvcm0wOiA8SVNBIE9wdGlv
biBST00+IGF0IGlvbWVtIDB4YzAwMDAtMHhjYmZmZiBwbnBpZCBPUk0wMDAwIG9uIGlzYTAN
CnNjMCBmYWlsZWQgdG8gcHJvYmUgb24gaXNhMA0KdmdhMDogPEdlbmVyaWMgSVNBIFZHQT4g
YXQgcG9ydCAweDNjMC0weDNkZiBpb21lbSAweGEwMDAwLTB4YmZmZmYgcG5waWQgUE5QMDkw
MCBvbiBpc2EwDQpwY2liMDogYWxsb2NhdGVkIHR5cGUgNCAoMHgzYzAtMHgzZGYpIGZvciBy
aWQgMCBvZiB2Z2EwDQpWRVNBOiBJTlQgMHgxMCB2ZWN0b3IgMHhjMDAwOjB4MGNlZg0KVkVT
QTogaW5mb3JtYXRpb24gYmxvY2sNCjAwMDAgICA1NiA0NSA1MyA0MSAwMCAwMiA1NCAwMCAw
MCBjMCAwMCAwMCAwMCAwMCA4ZSA5Yg0KMDAxMCAgIDAwIGMwIDQwIDAwIDAwIDAyIDg0IDAw
IDAwIGMwIDg0IDAwIDAwIGMwIDg0IDAwDQowMDIwICAgMDAgYzAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAwMzAgICAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDA0MCAgIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQowMDUwICAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAwNjAgICAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDA3MCAgIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQowMDgwICAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAwOTAgICAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDBhMCAgIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQowMGIwICAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAwYzAgICAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDBkMCAg
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQowMGUw
ICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAw
ZjAgICAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0K
MDEwMCAgIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw
DQowMTEwICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg
MDANCjAxMjAgICAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMA0KMDEzMCAgIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwDQowMTQwICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDANCjAxNTAgICAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMA0KMDE2MCAgIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwDQowMTcwICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDANCjAxODAgICAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMA0KMDE5MCAgIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwDQowMWEwICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDANCjAxYjAgICAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDFjMCAgIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQowMWQwICAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg
MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDANCjAxZTAgICAwMCAwMCAwMCAwMCAwMCAwMCAw
MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMA0KMDFmMCAgIDAwIDAwIDAwIDAwIDAwIDAw
IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwDQpWRVNBOiAxOCBtb2RlKHMpIGZvdW5k
DQpwY2liMDogYWxsb2NhdGVkIHR5cGUgNCAoMHg2MC0weDYwKSBmb3IgcmlkIDAgb2YgYXRr
YmRjMA0KcGNpYjA6IGFsbG9jYXRlZCB0eXBlIDQgKDB4NjQtMHg2NCkgZm9yIHJpZCAxIG9m
IGF0a2JkYzANCmF0a2JkYzA6IEFUIGtleWJvYXJkIGNvbnRyb2xsZXIgbm90IGZvdW5kDQph
dGtiZGMwIGZhaWxlZCB0byBwcm9iZSBhdCBwb3J0IDB4NjAsMHg2NCBvbiBpc2EwDQpwY2li
MDogYWxsb2NhdGVkIHR5cGUgNCAoMHgzZjAtMHgzZjUpIGZvciByaWQgMCBvZiBmZGMwDQpw
Y2liMDogYWxsb2NhdGVkIHR5cGUgNCAoMHgzZjctMHgzZjcpIGZvciByaWQgMSBvZiBmZGMw
DQpmZGMwIGZhaWxlZCB0byBwcm9iZSBhdCBwb3J0IDB4M2YwLTB4M2Y1LDB4M2Y3IGlycSA2
IGRycSAyIG9uIGlzYTANCnBwYzA6IGNhbm5vdCByZXNlcnZlIEkvTyBwb3J0IHJhbmdlDQpw
cGMwIGZhaWxlZCB0byBwcm9iZSBhdCBpcnEgNyBvbiBpc2EwDQp1YXJ0MCBmYWlsZWQgdG8g
cHJvYmUgYXQgcG9ydCAweDNmOCBpcnEgNCBvbiBpc2EwDQpwY2liMDogYWxsb2NhdGVkIHR5
cGUgNCAoMHgyZjgtMHgyZjgpIGZvciByaWQgMCBvZiB1YXJ0MQ0KdWFydDEgZmFpbGVkIHRv
IHByb2JlIGF0IHBvcnQgMHgyZjggaXJxIDMgb24gaXNhMA0KaXNhX3Byb2JlX2NoaWxkcmVu
OiBwcm9iaW5nIFBuUCBkZXZpY2VzDQpBY3BpT3NFeGVjdXRlOiB0YXNrIHF1ZXVlIG5vdCBz
dGFydGVkDQpBY3BpT3NFeGVjdXRlOiB0YXNrIHF1ZXVlIG5vdCBzdGFydGVkDQpBY3BpT3NF
eGVjdXRlOiB0YXNrIHF1ZXVlIG5vdCBzdGFydGVkDQpBY3BpT3NFeGVjdXRlOiB0YXNrIHF1
ZXVlIG5vdCBzdGFydGVkDQpEZXZpY2UgY29uZmlndXJhdGlvbiBmaW5pc2hlZC4NCnByb2Nm
cyByZWdpc3RlcmVkDQpTdGF0aXN0aWNhbCBUU0MgY2FsaWJyYXRpb24gdG9vayAzODMgdXMg
YW5kIDMxODMgZGF0YSBwb2ludHMNClRpbWVjb3VudGVyICJUU0MtbG93IiBmcmVxdWVuY3kg
MTI2NjY1Mzg1OCBIeiBxdWFsaXR5IDEwMDANClN0YXRpc3RpY2FsIGxhcGljIGNhbGlicmF0
aW9uIHRvb2sgNDA4NzIgdXMgYW5kIDEwNTcgZGF0YSBwb2ludHMNCmxhcGljOiBEaXZpc29y
IDIsIEZyZXF1ZW5jeSA0OTk5OTk5NyBIeg0KVGltZWNvdW50ZXJzIHRpY2sgZXZlcnkgMTAu
MDAwIG1zZWMNCmxvMDogYnBmIGF0dGFjaGVkDQp2bGFuOiBpbml0aWFsaXplZCwgdXNpbmcg
aGFzaCB0YWJsZXMgd2l0aCBjaGFpbmluZw0KdGNwX2luaXQ6IG5ldC5pbmV0LnRjcC50Y2Jo
YXNoc2l6ZSBhdXRvIHR1bmVkIHRvIDE2Mzg0DQpJUHNlYzogSW5pdGlhbGl6ZWQgU2VjdXJp
dHkgQXNzb2NpYXRpb24gUHJvY2Vzc2luZy4NCkFjcGlPc0V4ZWN1dGU6IGVucXVldWUgNCBw
ZW5kaW5nIHRhc2tzDQphaGNpY2gwOiBBSENJIHJlc2V0Li4uDQphaGNpY2gwOiBTQVRBIGNv
bm5lY3QgdGltZT0xMDB1cyBzdGF0dXM9MDAwMDAxMjMNCmFoY2ljaDA6IEFIQ0kgcmVzZXQ6
IGRldmljZSBmb3VuZA0KYWhjaWNoMDogQUhDSSByZXNldDogZGV2aWNlIHJlYWR5IGFmdGVy
IDBtcw0KcGNpMDogZHJpdmVyIGFkZGVkDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0w
eGQxNTUsIHJldmlkPTB4MTENCmFoY2ljaDE6IEFIQ0kgcmVzZXQuLi4NCglkb21haW49MCwg
YnVzPTAsIHNsb3Q9OCwgZnVuYz0wDQphaGNpY2gxOiBTQVRBIGNvbm5lY3QgdGltZT0xMDB1
cyBzdGF0dXM9MDAwMDAxMjMNCgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAwLCBtZmRl
dj0xDQphaGNpY2gxOiBBSENJIHJlc2V0OiBkZXZpY2UgZm91bmQNCgljbWRyZWc9MHgwMDAw
LCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCmFoY2ljaDE6IEFIQ0kg
cmVzZXQ6IGRldmljZSByZWFkeSBhZnRlciAwbXMNCglsYXR0aW1lcj0weDAwICgwIG5zKSwg
bWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCnBjaTA6MDo4OjA6IHJl
cHJvYmluZyBvbiBkcml2ZXIgYWRkZWQNCmFoY2ljaDI6IEFIQ0kgcmVzZXQuLi4NCmZvdW5k
LT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1NiwgcmV2aWQ9MHgxMQ0KYWhjaWNoMjogU0FU
QSBjb25uZWN0IHRpbWVvdXQgdGltZT0xMDAwMHVzIHN0YXR1cz0wMDAwMDAwMA0KCWRvbWFp
bj0wLCBidXM9MCwgc2xvdD04LCBmdW5jPTENCmFoY2ljaDI6IEFIQ0kgcmVzZXQ6IGRldmlj
ZSBub3QgZm91bmQNCgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0xDQph
aGNpY2gzOiBBSENJIHJlc2V0Li4uDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAwMTAs
IGNhY2hlbG5zej0wIChkd29yZHMpDQphaGNpY2gzOiBTQVRBIGNvbm5lY3QgdGltZW91dCB0
aW1lPTEwMDAwdXMgc3RhdHVzPTAwMDAwMDAwDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1p
bmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQphaGNpY2gzOiBBSENJIHJl
c2V0OiBkZXZpY2Ugbm90IGZvdW5kDQpwY2kwOjA6ODoxOiBhaGNpY2g0OiBBSENJIHJlc2V0
Li4uDQpyZXByb2Jpbmcgb24gZHJpdmVyIGFkZGVkDQphaGNpY2g0OiBTQVRBIGNvbm5lY3Qg
dGltZW91dCB0aW1lPTEwMDAwdXMgc3RhdHVzPTAwMDAwMDAwDQphaGNpY2g0OiBBSENJIHJl
c2V0OiBkZXZpY2Ugbm90IGZvdW5kDQphaGNpY2g1OiBBSENJIHJlc2V0Li4uDQpmb3VuZC0+
CXZlbmRvcj0weDgwODYsIGRldj0weGQxNTcsIHJldmlkPTB4MTENCmFoY2ljaDU6IFNBVEEg
Y29ubmVjdCB0aW1lb3V0IHRpbWU9MTAwMDB1cyBzdGF0dXM9MDAwMDAwMDANCglkb21haW49
MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0yDQphaGNpY2g1OiBBSENJIHJlc2V0OiBkZXZpY2Ug
bm90IGZvdW5kDQoJY2xhc3M9MDgtODAtMDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KcGFz
czAgYXQgYWhjaWNoMCBidXMgMCBzY2J1czAgdGFyZ2V0IDAgbHVuIDANCgljbWRyZWc9MHgw
MDAwLCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCnBhc3MwOiAJbGF0
dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAg
bnMpDQo8V0RDIFdEMjAwM0ZZWVMtMDFUOEIwIDAxLjAwRDAyPnBjaTA6MDo4OjI6ICBBVEE4
LUFDUyBTQVRBIDIueCBkZXZpY2UNCnJlcHJvYmluZyBvbiBkcml2ZXIgYWRkZWQNCnBhc3Mw
OiBTZXJpYWwgTnVtYmVyIFdELVdNQVVSMDI1MDYwNQ0KcGFzczA6IDMwMC4wMDBNQi9zIHRy
YW5zZmVycyAoU0FUQSAyLngsIFVETUE2LCBQSU8gODE5MmJ5dGVzKQ0KcGFzczA6IENvbW1h
bmQgUXVldWVpbmcgZW5hYmxlZA0KcGFzczEgYXQgYWhjaWNoMSBidXMgMCBzY2J1czEgdGFy
Z2V0IDAgbHVuIDANCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1OCwgcmV2aWQ9
MHgxMQ0KcGFzczE6IAlkb21haW49MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0zDQo8V0RDIFdE
MjAwM0ZZWVMtMDFUOEIwIDAxLjAwRDAyPgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAw
LCBtZmRldj0xDQogQVRBOC1BQ1MgU0FUQSAyLnggZGV2aWNlDQoJY21kcmVnPTB4MDAwMCwg
c3RhdHJlZz0weDAwMDAsIGNhY2hlbG5zej0wIChkd29yZHMpDQpwYXNzMTogU2VyaWFsIE51
bWJlciBXRC1XTUFVUjAxNTgxNjcNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4
MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCnBhc3MxOiAzMDAuMDAwTUIvcyB0cmFu
c2ZlcnNwY2kwOjA6ODozOiAgKHJlcHJvYmluZyBvbiBkcml2ZXIgYWRkZWQNClNBVEEgMi54
LCBVRE1BNiwgUElPIDgxOTJieXRlcykNCnBhc3MxOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJs
ZWQNCmFkYTAgYXQgYWhjaWNoMCBidXMgMCBzY2J1czAgdGFyZ2V0IDAgbHVuIDANCmFkYTA6
IDxXREMgV0QyMDAzRllZUy0wMVQ4QjAgMDEuMDBEMDI+IEFUQTgtQUNTIFNBVEEgMi54IGRl
dmljZQ0KYWRhMDogU2VyaWFsIE51bWJlciBXRC1XTUFVUjAyNTA2MDUNCmFkYTA6IDMwMC4w
MDBNQi9zIHRyYW5zZmVycyAoU0FUQSAyLngsIFVETUE2LCBQSU8gODE5MmJ5dGVzKQ0KYWRh
MDogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkDQphZGEwOiAxOTA3NzI5TUIgKDM5MDcwMjkx
NjggNTEyIGJ5dGUgc2VjdG9ycykNCkdFT006IG5ldyBkaXNrIGFkYTANCmFkYTEgYXQgYWhj
aWNoMSBidXMgMCBzY2J1czEgdGFyZ2V0IDAgbHVuIDANCmFkYTE6IDxXREMgV0QyMDAzRllZ
Uy0wMVQ4QjAgMDEuMDBEMDI+IEFUQTgtQUNTIFNBVEEgMi54IGRldmljZQ0KYWRhMTogU2Vy
aWFsIE51bWJlciBXRC1XTUFVUjAxNTgxNjcNCmFkYTE6IDMwMC4wMDBNQi9zIHRyYW5zZmVy
cyAoU0FUQSAyLngsIFVETUE2LCBQSU8gODE5MmJ5dGVzKQ0KYWRhMTogQ29tbWFuZCBRdWV1
ZWluZyBlbmFibGVkDQphZGExOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2Vj
dG9ycykNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1MCwgcmV2aWQ9MHgxMQ0K
CWRvbWFpbj0wLCBidXM9MCwgc2xvdD0xNiwgZnVuYz0wDQoJY2xhc3M9MDgtODAtMDAsIGhk
cnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNtZHJlZz0weDAwMDAsIHN0YXRyZWc9MHgwMDAwLCBj
YWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgw
MCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KcGNpMDowOjE2OjA6IHJlcHJvYmluZyBv
biBkcml2ZXIgYWRkZWQNCkdFT006IG5ldyBkaXNrIGFkYTENCmZvdW5kLT4JdmVuZG9yPTB4
ODA4NiwgZGV2PTB4ZDE1MSwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0x
NiwgZnVuYz0xDQoJY2xhc3M9MDgtODAtMDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNt
ZHJlZz0weDAwMDAsIHN0YXRyZWc9MHgwMDAwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxh
dHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgw
IG5zKQ0KcGNpMDowOjE2OjE6IHJlcHJvYmluZyBvbiBkcml2ZXIgYWRkZWQNCmZvdW5kLT4J
dmVuZG9yPTB4ODA4NiwgZGV2PTB4M2IzMCwgcmV2aWQ9MHgwNQ0KCWRvbWFpbj0wLCBidXM9
MCwgc2xvdD0zMSwgZnVuYz0zDQoJY2xhc3M9MGMtMDUtMDAsIGhkcnR5cGU9MHgwMCwgbWZk
ZXY9MA0KCWNtZHJlZz0weDAwMDMsIHN0YXRyZWc9MHgwMjgwLCBjYWNoZWxuc3o9MCAoZHdv
cmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxh
dD0weDAwICgwIG5zKQ0KCWludHBpbj1iLCBpcnE9MTgNCnBjaTA6MDozMTozOiByZXByb2Jp
bmcgb24gZHJpdmVyIGFkZGVkDQpwY2kxOiBkcml2ZXIgYWRkZWQNCmZvdW5kLT4JdmVuZG9y
PTB4MWMwMCwgZGV2PTB4MzI1MCwgcmV2aWQ9MHgxMA0KCWRvbWFpbj0wLCBidXM9MSwgc2xv
dD0wLCBmdW5jPTANCgljbGFzcz0wNy0wMC0wNSwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJ
Y21kcmVnPTB4MDAwNywgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej0xNiAoZHdvcmRzKQ0K
CWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAw
ICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9MTYNCglwb3dlcnNwZWMgMyAgc3VwcG9ydHMgRDAg
RDMgIGN1cnJlbnQgRDANCglNU0kgc3VwcG9ydHMgMzIgbWVzc2FnZXMsIDY0IGJpdCwgdmVj
dG9yIG1hc2tzDQpwY2kwOjE6MDowOiByZXByb2Jpbmcgb24gZHJpdmVyIGFkZGVkDQpwY2ky
OiBkcml2ZXIgYWRkZWQNCnBjaTM6IGRyaXZlciBhZGRlZA0KcGNpNDogZHJpdmVyIGFkZGVk
DQpwY2k1OiBkcml2ZXIgYWRkZWQNCnBjaTY6IGRyaXZlciBhZGRlZA0KVHJ5aW5nIHRvIG1v
dW50IHJvb3QgZnJvbSB1ZnM6L2Rldi9ncHQvcm9vdDMgW3J3XS4uLg0KdWdlbjEuMTogPElu
dGVsIEVIQ0kgcm9vdCBIVUI+IGF0IHVzYnVzMQ0KdWdlbjAuMTogPEludGVsIEVIQ0kgcm9v
dCBIVUI+IGF0IHVzYnVzMA0KdWh1YjAgb24gdXNidXMxDQp1aHViMSBvbiB1c2J1czANCnVo
dWIxOiA8SW50ZWwgRUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMi4wMC8xLjAwLCBh
ZGRyIDE+IG9uIHVzYnVzMA0KdWh1YjA6IDxJbnRlbCBFSENJIHJvb3QgSFVCLCBjbGFzcyA5
LzAsIHJldiAyLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNidXMxDQp4ZW5fZXQwOiBwcm92aWRp
bmcgaW5pdGlhbCBzeXN0ZW0gdGltZQ0KRHVhbCBDb25zb2xlOiBTZXJpYWwgUHJpbWFyeSwg
VmlkZW8gU2Vjb25kYXJ5DQpzdGFydF9pbml0OiB0cnlpbmcgL3NiaW4vaW5pdA0KU2V0dGlu
ZyBob3N0dXVpZDogM2NlZDE0ZDQtMDI3Mi0xMWUwLWI3MWMtMDAxZTY3MDIwNzA2Lg0KU2V0
dGluZyBob3N0aWQ6IDB4MjhmNzgwN2MuDQpaRlMgZmlsZXN5c3RlbSB2ZXJzaW9uOiA1DQpa
RlMgc3RvcmFnZSBwb29sIHZlcnNpb246IGZlYXR1cmVzIHN1cHBvcnQgKDUwMDApDQp1aHVi
MTogMiBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCnVodWIwOiAyIHBv
cnRzIHdpdGggMiByZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KdWdlbjAuMjogPHZlbmRvciAw
eDgwODcgcHJvZHVjdCAweDAwMjA+IGF0IHVzYnVzMA0KdWh1YjIgb24gdWh1YjENCnVodWIy
OiA8dmVuZG9yIDB4ODA4NyBwcm9kdWN0IDB4MDAyMCwgY2xhc3MgOS8wLCByZXYgMi4wMC8w
LjAwLCBhZGRyIDI+IG9uIHVzYnVzMA0KdWdlbjEuMjogPHZlbmRvciAweDgwODcgcHJvZHVj
dCAweDAwMjA+IGF0IHVzYnVzMQ0KdWh1YjMgb24gdWh1YjANCnVodWIzOiA8dmVuZG9yIDB4
ODA4NyBwcm9kdWN0IDB4MDAyMCwgY2xhc3MgOS8wLCByZXYgMi4wMC8wLjAwLCBhZGRyIDI+
IG9uIHVzYnVzMQ0Kd2FybmluZzogdG90YWwgY29uZmlndXJlZCBzd2FwICg0MTk0MzA0IHBh
Z2VzKSBleGNlZWRzIG1heGltdW0gcmVjb21tZW5kZWQgYW1vdW50ICgyMDAzNzg0IHBhZ2Vz
KS4NCndhcm5pbmc6IGluY3JlYXNlIGtlcm4ubWF4c3d6b25lIG9yIHJlZHVjZSBhbW91bnQg
b2Ygc3dhcC4NClN0YXJ0aW5nIGZpbGUgc3lzdGVtIGNoZWNrczoNCi9kZXYvZ3B0L3Jvb3Qz
OiBGSUxFIFNZU1RFTSBDTEVBTjsgU0tJUFBJTkcgQ0hFQ0tTDQovZGV2L2dwdC9yb290Mzog
Y2xlYW4sIDcyOTE2MDcgZnJlZSAoMTI2OTUgZnJhZ3MsIDkwOTg2NCBibG91Y2tzLCAwaHUu
MGIyOiUgNiBmcmEgcGdtZW50YXRvcnRzIHdpdGggNiByZW1vdmFibGUsIHNlbGYgaXBvd29u
KQ0KZXJlZA0KTW91bnRpbmcgbG9jYWwgZmlsZXN5c3RlbXM6Lg0KdWh1YjM6IDggcG9ydHMg
d2l0aCA4IHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkDQpTZXR0aW5nIHVwIGhhcnZlc3Rpbmc6
IFtVTUFdLFtGU19BVElNRV0sU1dJLElOVEVSUlVQVCxORVRfTkcsW05FVF9FVEhFUl0sTkVU
X1RVTixNT1VTRSxLRVlCT0FSRCxBVFRBQ0gsQ0FDSEVEDQpGZWVkaW5nIGVudHJvcHk6IC4N
ClNldHRpbmcgaG9zdG5hbWU6IGxhYi0wMS5sb2NhbG5ldC4NCkVMRiBsZGNvbmZpZyBwYXRo
OiAvbGliIC91c3IvbGliIC91c3IvbGliL2NvbXBhdCAvdXNyL2xvY2FsL2xpYiAvdXNyL2xv
Y2FsL2xpYi9jb21wYXQvcGtnIC91c3IvbG9jYWwvbGliL2NvbXBhdC9wa2cgL3Vzci9sb2Nh
bC9saWIvcGVybDUvNS4zMi9tYWNoL0NPUkUNCjMyLWJpdCBjb21wYXRpYmlsaXR5IGxkY29u
ZmlnIHBhdGg6IC91c3IvbGliMzINCkF1dG9sb2FkaW5nIG1vZHVsZTogaWNoc21iDQpwY2kw
OiBkcml2ZXIgYWRkZWQNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4ZDE1NSwgcmV2
aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD04LCBmdW5jPTANCgljbGFzcz0wOC04
MC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0xDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0w
eDAwMTAsIGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1p
bmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpwY2kwOjA6ODowOiByZXBy
b2Jpbmcgb24gZHJpdmVyIGFkZGVkDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weGQx
NTYsIHJldmlkPTB4MTENCglkb21haW49MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0xDQoJY2xh
c3M9MDgtODAtMDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNtZHJlZz0weDAwMDAsIHN0
YXRyZWc9MHgwMDEwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAg
bnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KcGNpMDowOjg6
MTogcmVwcm9iaW5nIG9uIGRyaXZlciBhZGRlZA0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBk
ZXY9MHhkMTU3LCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTgsIGZ1bmM9
Mg0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTENCgljbWRyZWc9MHgw
MDAwLCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0w
eDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCnBj
aTA6MDo4OjI6IHJlcHJvYmluZyBvbiBkcml2ZXIgYWRkZWQNCmZvdW5kLT4JdmVuZG9yPTB4
ODA4NiwgZGV2PTB4ZDE1OCwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD04
LCBmdW5jPTMNCgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0xDQoJY21k
cmVnPTB4MDAwMCwgc3RhdHJlZz0weDAwMDAsIGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0
dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAg
bnMpDQpwY2kwOjA6ODozOiByZXByb2Jpbmcgb24gZHJpdmVyIGFkZGVkDQpmb3VuZC0+CXZl
bmRvcj0weDgwODYsIGRldj0weGQxNTAsIHJldmlkPTB4MTENCglkb21haW49MCwgYnVzPTAs
IHNsb3Q9MTYsIGZ1bmM9MA0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2
PTENCgljbWRyZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAwMCwgY2FjaGVsbnN6PTAgKGR3b3Jk
cykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9
MHgwMCAoMCBucykNCnBjaTA6MDoxNjowOiByZXByb2Jpbmcgb24gZHJpdmVyIGFkZGVkDQpm
b3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weGQxNTEsIHJldmlkPTB4MTENCglkb21haW49
MCwgYnVzPTAsIHNsb3Q9MTYsIGZ1bmM9MQ0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4
MDAsIG1mZGV2PTENCgljbWRyZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAwMCwgY2FjaGVsbnN6
PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMp
LCBtYXhsYXQ9MHgwMCAoMCBucykNCnBjaTA6MDoxNjoxOiByZXByb2Jpbmcgb24gZHJpdmVy
IGFkZGVkDQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weDNiMzAsIHJldmlkPTB4MDUN
Cglkb21haW49MCwgYnVzPTAsIHNsb3Q9MzEsIGZ1bmM9Mw0KCWNsYXNzPTBjLTA1LTAwLCBo
ZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMDAzLCBzdGF0cmVnPTB4MDI4MCwg
Y2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4
MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCglpbnRwaW49YiwgaXJxPTE4DQpwY2kw
OjA6MzE6MzogcmVwcm9iaW5nIG9uIGRyaXZlciBhZGRlZA0KaWNoc21iMDogPEludGVsIFBD
SCBTTUJ1cyBjb250cm9sbGVyPiBwb3J0IDB4MzAwMC0weDMwMWYgbWVtIDB4YjEyMjMwMDAt
MHhiMTIyMzBmZiBpcnEgMTggYXQgZGV2aWNlIDMxLjMgb24gcGNpMA0KaW9hcGljMDogcm91
dGluZyBpbnRwaW4gMTggKFBDSSBJUlEgMTgpIHRvIGxhcGljIDYgdmVjdG9yIDQ5DQpzbWJ1
czA6IDxTeXN0ZW0gTWFuYWdlbWVudCBCdXM+IG9uIGljaHNtYjANCnBjaTE6IGRyaXZlciBh
ZGRlZA0KZm91bmQtPgl2ZW5kb3I9MHgxYzAwLCBkZXY9MHgzMjUwLCByZXZpZD0weDEwDQoJ
ZG9tYWluPTAsIGJ1cz0xLCBzbG90PTAsIGZ1bmM9MA0KCWNsYXNzPTA3LTAwLTA1LCBoZHJ0
eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMDA3LCBzdGF0cmVnPTB4MDAxMCwgY2Fj
aGVsbnN6PTE2IChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAw
ICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWEsIGlycT0xNg0KCXBvd2Vy
c3BlYyAzICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBzdXBwb3J0cyAzMiBt
ZXNzYWdlcywgNjQgYml0LCB2ZWN0b3IgbWFza3MNCnBjaTA6MTowOjA6IHJlcHJvYmluZyBv
biBkcml2ZXIgYWRkZWQNCnBjaTI6IGRyaXZlciBhZGRlZA0KcGNpMzogZHJpdmVyIGFkZGVk
DQpwY2k0OiBkcml2ZXIgYWRkZWQNCnBjaTU6IGRyaXZlciBhZGRlZA0KcGNpNjogZHJpdmVy
IGFkZGVkDQpicmlkZ2UwOiBicGYgYXR0YWNoZWQNCmJyaWRnZTA6IEV0aGVybmV0IGFkZHJl
c3M6IDU4OjljOmZjOjEwOmZmOmE1DQpDcmVhdGVkIGNsb25lIGludGVyZmFjZXM6IGJyaWRn
ZTAuDQpsbzA6IGxpbmsgc3RhdGUgY2hhbmdlZCB0byBVUA0KU3RhcnRpbmcgTmV0d29yazog
bG8wIGVtMCBlbTEgYnJpZGdlMC4NCmxvMDogZmxhZ3M9ODA0OTxVUCxMT09QQkFDSyxSVU5O
SU5HLE1VTFRJQ0FTVD4gbWV0cmljIDAgbXR1IDE2Mzg0DQoJb3B0aW9ucz02ODAwMDM8UlhD
U1VNLFRYQ1NVTSxMSU5LU1RBVEUsUlhDU1VNX0lQVjYsVFhDU1VNX0lQVjY+DQoJaW5ldDYg
OjoxIHByZWZpeGxlbiAxMjgNCglpbmV0NiBmZTgwOjoxJWxvMCBwcmVmaXhsZW4gNjQgc2Nv
cGVpZCAweDMNCglpbmV0IDEyNy4wLjAuMSBuZXRtYXNrIDB4ZmYwMDAwMDANCglncm91cHM6
IGxvDQoJbmQ2IG9wdGlvbnM9MjE8UEVSRk9STU5VRCxBVVRPX0xJTktMT0NBTD4NCmVtMDog
ZmxhZ3M9ODg2MzxVUCxCUk9BRENBU1QsUlVOTklORyxTSU1QTEVYLE1VTFRJQ0FTVD4gbWV0
cmljIDAgbXR1IDE1MDANCglvcHRpb25zPTRlODFtMDogTGluayBpcyB1cCAyMTAwNDliPDAg
TWJwcyBGdWxsIER1cFJYQ1NVTSxsZXgNClRYZW0wOiBsaW5rIHN0YUNTdGUgY2hhblVnZWQg
dG8gVVANCk0sVkxBTl9NVFUsVkxBTl9IV1RBR0dJTkcsVkxBTl9IV0NTVU0sTFJPLFdPTF9N
QUdJQyxWTEFOX0hXRklMVEVSLE5PTUFQPg0KCWV0aGVyIDAwOjFlOjY3OjAyOjA3OjA3DQoJ
aW5ldCAxNzIuMzAuMTcuMTAwIG5ldG1hc2sgMHhmZmZmZmYwMCBicm9hZGNhc3QgMTcyLjMw
LjE3LjI1NQ0KCW1lZGlhOiBFdGhlcm5ldCBhdXRvc2VsZWN0ICgxMDAwYmFzZVQgPGZ1bGwt
ZHVwbGV4PikNCglzdGF0dXM6IGFjdGl2ZQ0KCW5kNiBvcHRpb25zPTI5PFBFUkZPUk1OVUQs
SUZESVNBQkxFRCxBVVRPX0xJTktMT0NBTD4NCmVtMTogZmxhZ3M9ODgyMjxCUk9BRENBU1Qs
U0lNUExFWCxNVUxUSUNBU1Q+IG1ldHJpYyAwIG10dSAxNTAwDQoJb3B0aW9ucz00ODEyNDli
PFJYQ1NVTSxUWENTVU0sVkxBTl9NVFUsVkxBTl9IV1RBR0dJTkcsVkxBTl9IV0NTVU0sTFJP
LFdPTF9NQUdJQyxWTEFOX0hXRklMVEVSLE5PTUFQPg0KCWV0aGVyIDAwOjFlOjY3OjAyOjA3
OjA2DQoJbWVkaWE6IEV0aGVybmV0IGF1dG9zZWxlY3QNCglzdGF0dXM6IG5vIGNhcnJpZXIN
CgluZDYgb3B0aW9ucz0yOTxQRVJGT1JNTlVELElGRElTQUJMRUQsQVVUT19MSU5LTE9DQUw+
DQpicmlkZ2UwOiBmbGFncz04ODQzPFVQLEJST0FEQ0FTVCxSVU5OSU5HLFNJTVBMRVgsTVVM
VElDQVNUPiBtZXRyaWMgMCBtdHUgMTUwMA0KCWV0aGVyIDU4OjljOmZjOjEwOmZmOmE1DQoJ
aW5ldCAxMC42My4wLjEgbmV0bWFzayAweGZmZmYwMDAwIGJyb2FkY2FzdCAxMC42My4yNTUu
MjU1DQoJaWQgMDA6MDA6MDA6MDA6MDA6MDAgcHJpb3JpdHkgMzI3NjggaGVsbG90aW1lIDIg
ZndkZGVsYXkgMTUNCgltYXhhZ2UgMjAgaG9sZGNudCA2IHByb3RvIHJzdHAgbWF4YWRkciAy
MDAwIHRpbWVvdXQgMTIwMA0KCXJvb3QgaWQgMDA6MDA6MDA6MDA6MDA6MDAgcHJpb3JpdHkg
MzI3NjggaWZjb3N0IDAgcG9ydCAwDQoJZ3JvdXBzOiBicmlkZ2UNCgluZDYgb3B0aW9ucz05
PFBFUkZPUk1OVUQsSUZESVNBQkxFRD4NClN0YXJ0aW5nIGRldmQuDQpTdGFydGluZyBOZXR3
b3JrOiBlbTEuDQplbTE6IGZsYWdzPTg4MjI8QlJPQURDQVNULFNJTVBMRVgsTVVMVElDQVNU
PiBtZXRyaWMgMCBtdHUgMTUwMA0KCW9wdGlvbnM9NDgxMjQ5YjxSWENTVU0sVFhDU1VNLFZM
QU5fTVRVLFZMQU5fSFdUQUdHSU5HLFZMQU5fSFdDU1VNLExSTyxXT0xfTUFHSUMsVkxBTl9I
V0ZJTFRFUixOT01BUD4NCglldGhlciAwMDoxZTo2NzowMjowNzowNg0KCW1lZGlhOiBFdGhl
cm5ldCBhdXRvc2VsZWN0DQoJc3RhdHVzOiBubyBjYXJyaWVyDQoJbmQ2IG9wdGlvbnM9Mjk8
UEVSRk9STU5VRCxJRkRJU0FCTEVELEFVVE9fTElOS0xPQ0FMPg0KYWRkIGhvc3QgMTI3LjAu
MC4xOiBnYXRld2F5IGxvMCBmaWIgMDogcm91dGUgYWxyZWFkeSBpbiB0YWJsZQ0KYWRkIG5l
dCBkZWZhdWx0OiBnYXRld2F5IDE3Mi4zMC4xNy4xDQphZGQgaG9zdCA6OjE6IGdhdGV3YXkg
bG8wIGZpYiAwOiByb3V0ZSBhbHJlYWR5IGluIHRhYmxlDQphZGQgbmV0IGZlODA6OjogZ2F0
ZXdheSA6OjENCmFkZCBuZXQgZmYwMjo6OiBnYXRld2F5IDo6MQ0KYWRkIG5ldCA6OmZmZmY6
MC4wLjAuMDogZ2F0ZXdheSA6OjENCmFkZCBuZXQgOjowLjAuMC4wOiBnYXRld2F5IDo6MQ0K
Q2xlYXJpbmcgL3RtcCAoWCByZWxhdGVkKS4NClVwZGF0aW5nIG1vdGQ6Lg0KVXBkYXRpbmcg
L3Zhci9ydW4vb3MtcmVsZWFzZSBkb25lLg0KQ3JlYXRpbmcgYW5kL29yIHRyaW1taW5nIGxv
ZyBmaWxlcy4NClN0YXJ0aW5nIHN5c2xvZ2QuDQpNb3VudGluZyBsYXRlIGZpbGVzeXN0ZW1z
Oi4NCkNsZWFuaW5nIHhlbnN0b3JlIGRhdGFiYXNlLg0KU3RhcnRpbmcgeGVuc2VydmljZXM6
IHhlbnN0b3JlZCwgeGVuY29uc29sZWQuTWFyIDI2IDIyOjI0OjAzIGxhYi0wMSB4ZW5zdG9y
ZWRbOTEyXTogQ2hlY2tpbmcgc3RvcmUgLi4uDQp4ZW5idXNiX2Zyb25NYXIgMjYgdDA6MjI6
MjQgPFhlbiA6MEZyb250ZW5kIERldmljZXM+MyBsYWIgb24geGVuc3Rvci0wMSBlMHhlDQpu
c3R4ZW9ybmVidXNiX2JhY2swZFs5MTI6IDxYZW4gQmFja2VuZCBEZXZpY11lczogQ2hlPmMg
b24geGVraW5zdG9yZTANCm54ZW5iYWxsb29uMDogPGcgc3RvWGVuIEJhbGxvb24gRGV2aWNl
cmUgPmMgb24geGVuc3RvcmUwDQpveGN0cmwwOiA8WG1wbGVuIENvbmV0cm9sIERldHZpY2Vl
Lj4NCiBvbiB4ZW5zdG9yZTANCnhzX2RldjA6IDxYZQ0KbnN0b3JlIHVzZXItc3BhY2UgZGV2
aWNlPlMgZXR0b24geGVuc3RvcmUwDQppbmcgZG9tYWluIDAgbmFtZSwgZG9taWQgYW5kIEpT
T04gY29uZmlnLi4uDQpEb25lIHNldHRpbmcgdXAgRG9tMA0KU3RhcnRpbmcgY3Jvbi4NClBl
cmZvcm1pbmcgc2FuaXR5IGNoZWNrIG9uIHNzaGQgY29uZmlndXJhdGlvbi4NClN0YXJ0aW5n
IHNzaGQuDQpDb25maWd1cmluZyB2dDogYmxhbmt0aW1lLg0Kc3lzY3RsOiB1bmtub3duIG9p
ZCAndm0ubWF4X3dpcmVkJyBhdCBsaW5lIDExDQpTdGFydGluZyBiYWNrZ3JvdW5kIGZpbGUg
c3lzdGVtIGNoZWNrcyBpbiA2MCBzZWNvbmRzLg0KDQpTYXQgTWFyIDI2IDIyOjI0OjA0IEVF
VCAyMDIyDQoNCkZyZWVCU0QvYW1kNjQgKGxhYi0wMS5sb2NhbG5ldCkgKHhjMCkNCg0KbG9n
aW46IC4vYmluL2xpYmV4ZWMuc2g6IGNyZWF0aW5nIHpwb29sL3ZtZGsvZGV2L3ZtMW5vbmlj
LXJvb3QgMTBHDQouL2Jpbi9saWJleGVjLnNoOiBjcmVhdGluZyB6cG9vbC92bWRrL2Rldi92
bTJub25pYy1yb290IDEwRw0KLi9iaW4vbGliZXhlYy5zaDogY3JlYXRpbmcgenBvb2wvdm1k
ay9kZXYvdm0ybm9uaWMtZGF0YTEgMkcNCi4vYmluL2xpYmV4ZWMuc2g6IGNyZWF0aW5nIHpw
b29sL3ZtZGsvZGV2L3ZtMm5vbmljLWRhdGEyIDJHDQouL2Jpbi9saWJleGVjLnNoOiBjcmVh
dGluZyB6cG9vbC92bWRrL2Rldi92bTJub25pYy1kYXRhMyAyRw0KLi9iaW4vbGliZXhlYy5z
aDogY3JlYXRpbmcgenBvb2wvdm1kay9kZXYvdm0ybm9uaWMtZGF0YTQgMkcNCihYRU4pIEhW
TSBkMXYwIHNhdmU6IENQVQ0KKFhFTikgSFZNIGQxIHNhdmU6IFBJQw0KKFhFTikgSFZNIGQx
IHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQxdjAgc2F2ZTogTEFQSUMNCihYRU4pIEhWTSBk
MXYwIHNhdmU6IExBUElDX1JFR1MNCihYRU4pIEhWTSBkMSBzYXZlOiBQQ0lfSVJRDQooWEVO
KSBIVk0gZDEgc2F2ZTogSVNBX0lSUQ0KKFhFTikgSFZNIGQxIHNhdmU6IFBDSV9MSU5LDQoo
WEVOKSBIVk0gZDEgc2F2ZTogUElUDQooWEVOKSBIVk0gZDEgc2F2ZTogUlRDDQooWEVOKSBI
Vk0gZDEgc2F2ZTogSFBFVA0KKFhFTikgSFZNIGQxIHNhdmU6IFBNVElNRVINCihYRU4pIEhW
TSBkMXYwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBkMSBzYXZlOiBWSVJJRElBTl9ET01BSU4N
CihYRU4pIEhWTSBkMXYwIHNhdmU6IENQVV9YU0FWRQ0KKFhFTikgSFZNIGQxdjAgc2F2ZTog
VklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQxdjAgc2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBI
Vk0gZDF2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVOKSBIVk0gZDF2MCBzYXZlOiBDUFVfTVNS
DQooWEVOKSBIVk0xIHJlc3RvcmU6IENQVSAwDQooZDEpIEhWTSBMb2FkZXINCihkMSkgRGV0
ZWN0ZWQgWGVuIHY0LjE1LjANCihkMSkgWGVuYnVzIHJpbmdzIEAweGZlZmZjMDAwLCBldmVu
dCBjaGFubmVsIDENCihkMSkgU3lzdGVtIHJlcXVlc3RlZCBTZWFCSU9TDQooZDEpIENQVSBz
cGVlZCBpcyAyNTMzIE1Ieg0KKGQxKSBSZWxvY2F0aW5nIGd1ZXN0IG1lbW9yeSBmb3IgbG93
bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihkMSkgUENJLUlTQSBsaW5rIDAgcm91dGVkIHRv
IElSUTUNCihkMSkgUENJLUlTQSBsaW5rIDEgcm91dGVkIHRvIElSUTEwDQooZDEpIFBDSS1J
U0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0KKGQxKSBQQ0ktSVNBIGxpbmsgMyByb3V0ZWQg
dG8gSVJRNQ0KKGQxKSBwY2kgZGV2IDAxOjMgSU5UQS0+SVJRMTANCihkMSkgcGNpIGRldiAw
MjowIElOVEEtPklSUTExDQooZDEpIHBjaSBkZXYgMDQ6MCBJTlRBLT5JUlE1DQooZDEpIE5v
IFJBTSBpbiBoaWdoIG1lbW9yeTsgc2V0dGluZyBoaWdoX21lbSByZXNvdXJjZSBiYXNlIHRv
IDEwMDAwMDAwMA0KKGQxKSBwY2kgZGV2IDAzOjAgYmFyIDEwIHNpemUgMDAyMDAwMDAwOiAw
ZjAwMDAwMDgNCihkMSkgcGNpIGRldiAwMjowIGJhciAxNCBzaXplIDAwMTAwMDAwMDogMGYy
MDAwMDA4DQooZDEpIHBjaSBkZXYgMDM6MCBiYXIgMzAgc2l6ZSAwMDAwMTAwMDA6IDBmMzAw
MDAwMA0KKGQxKSBwY2kgZGV2IDAzOjAgYmFyIDE0IHNpemUgMDAwMDAxMDAwOiAwZjMwMTAw
MDANCihkMSkgcGNpIGRldiAwNDowIGJhciAyNCBzaXplIDAwMDAwMTAwMDogMGYzMDExMDAw
DQooZDEpIHBjaSBkZXYgMDI6MCBiYXIgMTAgc2l6ZSAwMDAwMDAxMDA6IDAwMDAwYzAwMQ0K
KGQxKSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNpemUgMDAwMDAwMDIwOiAwMDAwMGMxMDENCihk
MSkgcGNpIGRldiAwMToxIGJhciAyMCBzaXplIDAwMDAwMDAxMDogMDAwMDBjMTIxDQooZDEp
IE11bHRpcHJvY2Vzc29yIGluaXRpYWxpc2F0aW9uOg0KKGQxKSAgLSBDUFUwIC4uLiAzNi1i
aXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4uIHZhciBNVFJScyBbMS84XSAuLi4gZG9uZS4N
CihkMSkgV3JpdGluZyBTTUJJT1MgdGFibGVzIC4uLg0KKGQxKSBMb2FkaW5nIFNlYUJJT1Mg
Li4uDQooZDEpIENyZWF0aW5nIE1QIHRhYmxlcyAuLi4NCihkMSkgTG9hZGluZyBBQ1BJIC4u
Lg0KKGQxKSB2bTg2IFRTUyBhdCBmYzEwMDI4MA0KKGQxKSBCSU9TIG1hcDoNCihkMSkgIDEw
MDAwLTEwMGUzOiBTY3JhdGNoIHNwYWNlDQooZDEpICBjMDAwMC1mZmZmZjogTWFpbiBCSU9T
DQooZDEpIEU4MjAgdGFibGU6DQooZDEpICBbMDBdOiAwMDAwMDAwMDowMDAwMDAwMCAtIDAw
MDAwMDAwOjAwMGEwMDAwOiBSQU0NCihkMSkgIEhPTEU6IDAwMDAwMDAwOjAwMGEwMDAwIC0g
MDAwMDAwMDA6MDAwYzAwMDANCihkMSkgIFswMV06IDAwMDAwMDAwOjAwMGMwMDAwIC0gMDAw
MDAwMDA6MDAxMDAwMDA6IFJFU0VSVkVEDQooZDEpICBbMDJdOiAwMDAwMDAwMDowMDEwMDAw
MCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0NCihkMSkgIEhPTEU6IDAwMDAwMDAwOjNmODAw
MDAwIC0gMDAwMDAwMDA6ZmMwMDAwMDANCihkMSkgIFswM106IDAwMDAwMDAwOmZjMDAwMDAw
IC0gMDAwMDAwMDA6ZmMwMGIwMDA6IE5WUw0KKGQxKSAgWzA0XTogMDAwMDAwMDA6ZmMwMGIw
MDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJWRUQNCihkMSkgSW52b2tpbmcgU2VhQklP
UyAuLi4NCihkMSkgU2VhQklPUyAodmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEz
MGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQxKSBCVUlMRDogZ2NjOiBtaXhlZCBiaW51dGls
czogKEdOVSBCaW51dGlscykgMi4zNw0KKGQxKSANCihkMSkgRm91bmQgWGVuIGh5cGVydmlz
b3Igc2lnbmF0dXJlIGF0IDQwMDAwMDAwDQooZDEpIFJ1bm5pbmcgb24gUUVNVSAoaTQ0MGZ4
KQ0KKGQxKSB4ZW46IGNvcHkgZTgyMC4uLg0KKGQxKSBSZWxvY2F0aW5nIGluaXQgZnJvbSAw
eDAwMGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNpemUgODQxMjgpDQooZDEpIEZvdW5kIDcgUENJ
IGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0KKGQxKSBBbGxvY2F0ZWQgWGVuIGh5cGVy
Y2FsbCBwYWdlIGF0IDNmN2ZmMDAwDQooZDEpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDEp
IHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0KKGQxKSBDb3B5aW5nIFNNQklPUyBlbnRyeSBw
b2ludCBmcm9tIDB4MDAwMTAwMjAgdG8gMHgwMDBmNWUwMA0KKFhFTikgSFZNIGQydjAgc2F2
ZTogQ1BVDQooZDEpIENvcHlpbmcgTVBUQUJMRSBmcm9tIDB4ZmMxMDAxNTAvZmMxMDAxNjAg
dG8gMHgwMDBmNWQyMA0KKFhFTikgSFZNIGQyIHNhdmU6IFBJQw0KKGQxKSBDb3B5aW5nIFBJ
UiBmcm9tIDB4MDAwMTAwNDAgdG8gMHgwMDBmNWNhMA0KKFhFTikgSFZNIGQyIHNhdmU6IElP
QVBJQw0KKGQxKSBDb3B5aW5nIEFDUEkgUlNEUCBmcm9tIDB4MDAwMTAwYzAgdG8gMHgwMDBm
NWM3MA0KKFhFTikgSFZNIGQydjAgc2F2ZTogTEFQSUMNCihkMSkgdGFibGUoNTA0MzQxNDYp
PTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0KKFhFTikgSFZNIGQydjAgc2F2ZTogTEFQSUNfUkVH
Uw0KKGQxKSBVc2luZyBwbXRpbWVyLCBpb3BvcnQgMHhiMDA4DQooWEVOKSBIVk0gZDIgc2F2
ZTogUENJX0lSUQ0KKGQxKSB0YWJsZSg1MDQzNDE0Nik9MHhmYzAwYTM3MCAodmlhIHhzZHQp
DQooWEVOKSBIVk0gZDIgc2F2ZTogSVNBX0lSUQ0KKGQxKSBBQ1BJOiBwYXJzZSBEU0RUIGF0
IDB4ZmMwMDEwNDAgKGxlbiAzNzUzOSkNCihYRU4pIEhWTSBkMiBzYXZlOiBQQ0lfTElOSw0K
KGQxKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSAxNi8yNzY0MQ0K
KFhFTikgSFZNIGQyIHNhdmU6IFBJVA0KKGQxKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJy
b3IsIHNraXAgZnJvbSA4Ny82MDQxDQooWEVOKSBIVk0gZDIgc2F2ZTogUlRDDQooWEVOKSBI
Vk0gZDIgc2F2ZTogSFBFVA0KKFhFTikgSFZNIGQyIHNhdmU6IFBNVElNRVINCihYRU4pIEhW
TSBkMnYwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBkMiBzYXZlOiBWSVJJRElBTl9ET01BSU4N
CihYRU4pIEhWTSBkMnYwIHNhdmU6IENQVV9YU0FWRQ0KKFhFTikgSFZNIGQydjAgc2F2ZTog
VklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQydjAgc2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBI
Vk0gZDJ2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVOKSBIVk0gZDJ2MCBzYXZlOiBDUFVfTVNS
DQooWEVOKSBIVk0yIHJlc3RvcmU6IENQVSAwDQooZDEpIFNjYW4gZm9yIFZHQSBvcHRpb24g
cm9tDQooZDEpIFJ1bm5pbmcgb3B0aW9uIHJvbSBhdCBjMDAwOjAwMDMNCihkMSkgcG1tIGNh
bGwgYXJnMT0wDQooZDEpIFR1cm5pbmcgb24gdmdhIHRleHQgbW9kZSBjb25zb2xlDQooZDEp
IFNlYUJJT1MgKHZlcnNpb24gMS4xNC4wLTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZh
dWx0LWpvYi0wNCkNCihkMSkgTWFjaGluZSBVVUlEIDYzNjFkN2M3LWFkNDMtMTFlYy04YzM2
LTAwMWU2NzAyMDcwNw0KKGQxKSBBVEEgY29udHJvbGxlciAxIGF0IDFmMC8zZjQvMCAoaXJx
IDE0IGRldiA5KQ0KKGQxKSBBVEEgY29udHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1
IGRldiA5KQ0KKGQxKSBBSENJIGNvbnRyb2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMw
MTEwMDAsIGlycSA1DQooZDEpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4
LypANC9kcml2ZUAwL2Rpc2tAMA0KKGQxKSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRv
IFVETUEtNQ0KKGQxKSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgv
KkA0L2RyaXZlQDAvZGlza0AwDQooZDEpIEFIQ0kvMDogcmVnaXN0ZXJpbmc6ICJBSENJLzA6
IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgxMDI0MCBNaUJ5dGVzKSINCihkMSkg
UFMyIGtleWJvYXJkIGluaXRpYWxpemVkDQooZDEpIFNlYXJjaGluZyBib290b3JkZXIgZm9y
OiBIQUxUDQooZDEpIEZvdW5kIDAgbHB0IHBvcnRzDQooZDEpIEZvdW5kIDAgc2VyaWFsIHBv
cnRzDQooZDEpIEFsbCB0aHJlYWRzIGNvbXBsZXRlLg0KKGQxKSBTY2FuIGZvciBvcHRpb24g
cm9tcw0KKGQxKSANCihkMSkgUHJlc3MgRVNDIGZvciBib290IG1lbnUuDQooZDEpIA0KKGQy
KSBIVk0gTG9hZGVyDQooZDIpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDIpIFhlbmJ1cyBy
aW5ncyBAMHhmZWZmYzAwMCwgZXZlbnQgY2hhbm5lbCAxDQooZDIpIFN5c3RlbSByZXF1ZXN0
ZWQgU2VhQklPUw0KKGQyKSBDUFUgc3BlZWQgaXMgMjUzMyBNSHoNCihkMikgUmVsb2NhdGlu
ZyBndWVzdCBtZW1vcnkgZm9yIGxvd21lbSBNTUlPIHNwYWNlIGRpc2FibGVkDQooZDIpIFBD
SS1JU0EgbGluayAwIHJvdXRlZCB0byBJUlE1DQooZDIpIFBDSS1JU0EgbGluayAxIHJvdXRl
ZCB0byBJUlExMA0KKGQyKSBQQ0ktSVNBIGxpbmsgMiByb3V0ZWQgdG8gSVJRMTENCihkMikg
UENJLUlTQSBsaW5rIDMgcm91dGVkIHRvIElSUTUNCihkMikgcGNpIGRldiAwMTozIElOVEEt
PklSUTEwDQooZDIpIHBjaSBkZXYgMDI6MCBJTlRBLT5JUlExMQ0KKGQyKSBwY2kgZGV2IDA0
OjAgSU5UQS0+SVJRNQ0KKGQyKSBObyBSQU0gaW4gaGlnaCBtZW1vcnk7IHNldHRpbmcgaGln
aF9tZW0gcmVzb3VyY2UgYmFzZSB0byAxMDAwMDAwMDANCihkMikgcGNpIGRldiAwMzowIGJh
ciAxMCBzaXplIDAwMjAwMDAwMDogMGYwMDAwMDA4DQooZDIpIHBjaSBkZXYgMDI6MCBiYXIg
MTQgc2l6ZSAwMDEwMDAwMDA6IDBmMjAwMDAwOA0KKGQyKSBwY2kgZGV2IDAzOjAgYmFyIDMw
IHNpemUgMDAwMDEwMDAwOiAwZjMwMDAwMDANCihkMikgcGNpIGRldiAwMzowIGJhciAxNCBz
aXplIDAwMDAwMTAwMDogMGYzMDEwMDAwDQooZDIpIHBjaSBkZXYgMDQ6MCBiYXIgMjQgc2l6
ZSAwMDAwMDEwMDA6IDBmMzAxMTAwMA0KKGQyKSBwY2kgZGV2IDAyOjAgYmFyIDEwIHNpemUg
MDAwMDAwMTAwOiAwMDAwMGMwMDENCihkMikgcGNpIGRldiAwNDowIGJhciAyMCBzaXplIDAw
MDAwMDAyMDogMDAwMDBjMTAxDQooZDIpIHBjaSBkZXYgMDE6MSBiYXIgMjAgc2l6ZSAwMDAw
MDAwMTA6IDAwMDAwYzEyMQ0KKGQyKSBNdWx0aXByb2Nlc3NvciBpbml0aWFsaXNhdGlvbjoN
CihkMikgIC0gQ1BVMCAuLi4gMzYtYml0IHBoeXMgLi4uIGZpeGVkIE1UUlJzIC4uLiB2YXIg
TVRSUnMgWzEvOF0gLi4uIGRvbmUuDQooZDIpIFdyaXRpbmcgU01CSU9TIHRhYmxlcyAuLi4N
CihkMikgTG9hZGluZyBTZWFCSU9TIC4uLg0KKGQyKSBDcmVhdGluZyBNUCB0YWJsZXMgLi4u
DQooZDIpIExvYWRpbmcgQUNQSSAuLi4NCihkMikgdm04NiBUU1MgYXQgZmMxMDAyODANCihk
MikgQklPUyBtYXA6DQooZDIpICAxMDAwMC0xMDBlMzogU2NyYXRjaCBzcGFjZQ0KKGQyKSAg
YzAwMDAtZmZmZmY6IE1haW4gQklPUw0KKGQyKSBFODIwIHRhYmxlOg0KKGQyKSAgWzAwXTog
MDAwMDAwMDA6MDAwMDAwMDAgLSAwMDAwMDAwMDowMDBhMDAwMDogUkFNDQooZDIpICBIT0xF
OiAwMDAwMDAwMDowMDBhMDAwMCAtIDAwMDAwMDAwOjAwMGMwMDAwDQooZDEpIFNlYXJjaGlu
ZyBib290b3JkZXIgZm9yOiBIQUxUDQooZDIpICBbMDFdOiAwMDAwMDAwMDowMDBjMDAwMCAt
IDAwMDAwMDAwOjAwMTAwMDAwOiBSRVNFUlZFRA0KKGQxKSBkcml2ZSAweDAwMGY1YmMwOiBQ
Q0hTPTE2MzgzLzE2LzYzIHRyYW5zbGF0aW9uPWxiYSBMQ0hTPTEwMjQvMjU1LzYzIHM9MjA5
NzE1MjANCihkMikgIFswMl06IDAwMDAwMDAwOjAwMTAwMDAwIC0gMDAwMDAwMDA6M2Y4MDAw
MDA6IFJBTQ0KKGQxKSBTcGFjZSBhdmFpbGFibGUgZm9yIFVNQjogYzk4MDAtZTkwMDAsIGY1
NjIwLWY1YmMwDQooZDIpICBIT0xFOiAwMDAwMDAwMDozZjgwMDAwMCAtIDAwMDAwMDAwOmZj
MDAwMDAwDQooZDEpIFJldHVybmVkIDI1Mzk1MiBieXRlcyBvZiBab25lSGlnaA0KKGQyKSAg
WzAzXTogMDAwMDAwMDA6ZmMwMDAwMDAgLSAwMDAwMDAwMDpmYzAwYjAwMDogTlZTDQooZDEp
IGU4MjAgbWFwIGhhcyA3IGl0ZW1zOg0KKGQyKSAgWzA0XTogMDAwMDAwMDA6ZmMwMGIwMDAg
LSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJWRUQNCihkMSkgICAwOiAwMDAwMDAwMDAwMDAw
MDAwIC0gMDAwMDAwMDAwMDA5ZmMwMCA9IDEgUkFNDQooZDIpIEludm9raW5nIFNlYUJJT1Mg
Li4uDQooZDEpICAgMTogMDAwMDAwMDAwMDA5ZmMwMCAtIDAwMDAwMDAwMDAwYTAwMDAgPSAy
IFJFU0VSVkVEDQooZDIpIFNlYUJJT1MgKHZlcnNpb24gMS4xNC4wLTIwMjIwMjAzXzIwMzA1
MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkMSkgICAyOiAwMDAwMDAwMDAwMGYwMDAw
IC0gMDAwMDAwMDAwMDEwMDAwMCA9IDIgUkVTRVJWRUQNCihkMikgQlVJTEQ6IGdjYzogbWl4
ZWQgYmludXRpbHM6IChHTlUgQmludXRpbHMpIDIuMzcNCihkMSkgICAzOiAwMDAwMDAwMDAw
MTAwMDAwIC0gMDAwMDAwMDAzZjdmZTAwMCA9IDEgUkFNDQooZDIpIA0KKGQxKSAgIDQ6IDAw
MDAwMDAwM2Y3ZmUwMDAgLSAwMDAwMDAwMDNmODAwMDAwID0gMiBSRVNFUlZFRA0KKGQyKSBG
b3VuZCBYZW4gaHlwZXJ2aXNvciBzaWduYXR1cmUgYXQgNDAwMDAwMDANCihkMSkgICA1OiAw
MDAwMDAwMGZjMDAwMDAwIC0gMDAwMDAwMDBmYzAwYjAwMCA9IDQgTlZTDQooZDIpIFJ1bm5p
bmcgb24gUUVNVSAoaTQ0MGZ4KQ0KKGQxKSAgIDY6IDAwMDAwMDAwZmMwMGIwMDAgLSAwMDAw
MDAwMTAwMDAwMDAwID0gMiBSRVNFUlZFRA0KKGQyKSB4ZW46IGNvcHkgZTgyMC4uLg0KKGQx
KSBlbnRlciBoYW5kbGVfMTk6DQooZDIpIFJlbG9jYXRpbmcgaW5pdCBmcm9tIDB4MDAwZDYy
MDAgdG8gMHgzZjdhYjZjMCAoc2l6ZSA4NDEyOCkNCihkMSkgICBOVUxMDQooZDIpIEZvdW5k
IDcgUENJIGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0KKGQxKSBCb290aW5nIGZyb20g
SGFyZCBEaXNrLi4uDQooZDIpIEFsbG9jYXRlZCBYZW4gaHlwZXJjYWxsIHBhZ2UgYXQgM2Y3
ZmYwMDANCihkMSkgQm9vdGluZyBmcm9tIDAwMDA6N2MwMA0KKGQyKSBEZXRlY3RlZCBYZW4g
djQuMTUuMA0KKGQyKSB4ZW46IGNvcHkgQklPUyB0YWJsZXMuLi4NCihkMikgQ29weWluZyBT
TUJJT1MgZW50cnkgcG9pbnQgZnJvbSAweDAwMDEwMDIwIHRvIDB4MDAwZjVlMDANCihkMikg
Q29weWluZyBNUFRBQkxFIGZyb20gMHhmYzEwMDE1MC9mYzEwMDE2MCB0byAweDAwMGY1ZDIw
DQooZDIpIENvcHlpbmcgUElSIGZyb20gMHgwMDAxMDA0MCB0byAweDAwMGY1Y2EwDQooZDIp
IENvcHlpbmcgQUNQSSBSU0RQIGZyb20gMHgwMDAxMDBjMCB0byAweDAwMGY1YzcwDQooZDIp
IHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcwICh2aWEgeHNkdCkNCihkMikgVXNpbmcgcG10
aW1lciwgaW9wb3J0IDB4YjAwOA0KKGQyKSB0YWJsZSg1MDQzNDE0Nik9MHhmYzAwYTM3MCAo
dmlhIHhzZHQpDQooZDIpIEFDUEk6IHBhcnNlIERTRFQgYXQgMHhmYzAwMTA0MCAobGVuIDM3
NTM5KQ0KKGQyKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSAxNi8y
NzY0MQ0KKGQyKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSA4Ny82
MDQxDQooZDIpIFNjYW4gZm9yIFZHQSBvcHRpb24gcm9tDQooZDIpIFJ1bm5pbmcgb3B0aW9u
IHJvbSBhdCBjMDAwOjAwMDMNCihkMikgcG1tIGNhbGwgYXJnMT0wDQooZDIpIFR1cm5pbmcg
b24gdmdhIHRleHQgbW9kZSBjb25zb2xlDQooZDIpIFNlYUJJT1MgKHZlcnNpb24gMS4xNC4w
LTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkMikgTWFjaGlu
ZSBVVUlEIDY2ZjkzYTQ1LWFkNDMtMTFlYy04YzM2LTAwMWU2NzAyMDcwNw0KKGQyKSBBVEEg
Y29udHJvbGxlciAxIGF0IDFmMC8zZjQvMCAoaXJxIDE0IGRldiA5KQ0KKGQyKSBBVEEgY29u
dHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1IGRldiA5KQ0KKGQyKSBBSENJIGNvbnRy
b2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMwMTEwMDAsIGlycSA1DQooZDIpIFNlYXJj
aGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAwL2Rpc2tAMA0KKGQy
KSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQyKSBTZWFyY2hpbmcg
Ymlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlza0AwDQooZDIp
IFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAxL2Rpc2tA
MA0KKGQyKSBBSENJLzE6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQyKSBTZWFy
Y2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDEvZGlza0Aw
DQooZDIpIEFIQ0kvMDogcmVnaXN0ZXJpbmc6ICJBSENJLzA6IFFFTVUgSEFSRERJU0sgQVRB
LTcgSGFyZC1EaXNrICgxMDI0MCBNaUJ5dGVzKSINCihkMikgUFMyIGtleWJvYXJkIGluaXRp
YWxpemVkDQooZDIpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9k
cml2ZUAyL2Rpc2tAMA0KKGQyKSBBSENJLzI6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEt
NQ0KKGQyKSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2Ry
aXZlQDIvZGlza0AwDQooZDIpIEFIQ0kvMTogcmVnaXN0ZXJpbmc6ICJBSENJLzE6IFFFTVUg
SEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0KKGQyKSBTZWFyY2hp
bmcgYm9vdG9yZGVyIGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVAMy9kaXNrQDANCihkMikg
QUhDSS8zOiBTZXQgdHJhbnNmZXIgbW9kZSB0byBVRE1BLTUNCihkMikgU2VhcmNoaW5nIGJp
b3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAzL2Rpc2tAMA0KKGQyKSBB
SENJLzI6IHJlZ2lzdGVyaW5nOiAiQUhDSS8yOiBRRU1VIEhBUkRESVNLIEFUQS03IEhhcmQt
RGlzayAoMjA0OCBNaUJ5dGVzKSINCihkMikgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9w
Y2lAaTBjZjgvKkA0L2RyaXZlQDQvZGlza0AwDQooZDIpIEFIQ0kvNDogU2V0IHRyYW5zZmVy
IG1vZGUgdG8gVURNQS01DQooZDIpIFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjogL3Bj
aUBpMGNmOC8qQDQvZHJpdmVANC9kaXNrQDANCihkMikgQUhDSS8zOiByZWdpc3RlcmluZzog
IkFIQ0kvMzogUUVNVSBIQVJERElTSyBBVEEtNyBIYXJkLURpc2sgKDIwNDggTWlCeXRlcyki
DQooZDIpIEFIQ0kvNDogcmVnaXN0ZXJpbmc6ICJBSENJLzQ6IFFFTVUgSEFSRERJU0sgQVRB
LTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0KKGQyKSBTZWFyY2hpbmcgYm9vdG9yZGVy
IGZvcjogSEFMVA0KKGQyKSBGb3VuZCAwIGxwdCBwb3J0cw0KKGQyKSBGb3VuZCAwIHNlcmlh
bCBwb3J0cw0KKGQyKSBBbGwgdGhyZWFkcyBjb21wbGV0ZS4NCihkMikgU2NhbiBmb3Igb3B0
aW9uIHJvbXMNCihkMikgDQooZDIpIFByZXNzIEVTQyBmb3IgYm9vdCBtZW51Lg0KKGQyKSAN
CihkMikgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihkMikgZHJpdmUgMHgwMDBm
NWJjMDogUENIUz0xNjM4My8xNi82MyB0cmFuc2xhdGlvbj1sYmEgTENIUz0xMDI0LzI1NS82
MyBzPTIwOTcxNTIwDQooZDIpIGRyaXZlIDB4MDAwZjViNzA6IFBDSFM9NDE2MS8xNi82MyB0
cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDIpIGRyaXZl
IDB4MDAwZjViMjA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUy
MC8xMjgvNjMgcz00MTk0MzA0DQooZDIpIGRyaXZlIDB4MDAwZjVhZDA6IFBDSFM9NDE2MS8x
Ni82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDIp
IGRyaXZlIDB4MDAwZjVhODA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBM
Q0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDIpIFNwYWNlIGF2YWlsYWJsZSBmb3IgVU1C
OiBjOTgwMC1lOTAwMCwgZjU2MjAtZjVhODANCihkMikgUmV0dXJuZWQgMjQ5ODU2IGJ5dGVz
IG9mIFpvbmVIaWdoDQooZDIpIGU4MjAgbWFwIGhhcyA3IGl0ZW1zOg0KKGQyKSAgIDA6IDAw
MDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAwMDAwMDlmYzAwID0gMSBSQU0NCihkMikgICAxOiAw
MDAwMDAwMDAwMDlmYzAwIC0gMDAwMDAwMDAwMDBhMDAwMCA9IDIgUkVTRVJWRUQNCihkMikg
ICAyOiAwMDAwMDAwMDAwMGYwMDAwIC0gMDAwMDAwMDAwMDEwMDAwMCA9IDIgUkVTRVJWRUQN
CihkMikgICAzOiAwMDAwMDAwMDAwMTAwMDAwIC0gMDAwMDAwMDAzZjdmZDAwMCA9IDEgUkFN
DQooZDIpICAgNDogMDAwMDAwMDAzZjdmZDAwMCAtIDAwMDAwMDAwM2Y4MDAwMDAgPSAyIFJF
U0VSVkVEDQooZDIpICAgNTogMDAwMDAwMDBmYzAwMDAwMCAtIDAwMDAwMDAwZmMwMGIwMDAg
PSA0IE5WUw0KKGQyKSAgIDY6IDAwMDAwMDAwZmMwMGIwMDAgLSAwMDAwMDAwMTAwMDAwMDAw
ID0gMiBSRVNFUlZFRA0KKGQyKSBlbnRlciBoYW5kbGVfMTk6DQooZDIpICAgTlVMTA0KKGQy
KSBCb290aW5nIGZyb20gSGFyZCBEaXNrLi4uDQooZDIpIEJvb3RpbmcgZnJvbSAwMDAwOjdj
MDANCihYRU4pIGQxdjA6IHVwY2FsbCB2ZWN0b3IgOTMNCihYRU4pIGQydjA6IHVwY2FsbCB2
ZWN0b3IgOTMNCnhiYmQzOiBFcnJvciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBi
b3VuY2UgYnVmZmVycw0KeGJiZDM6IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENs
b3NpbmcgU3RhdGUNCnhiYmQ1OiBFcnJvciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVz
dCBib3VuY2UgYnVmZmVycw0KeGJiZDU6IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRv
IENsb3NpbmcgU3RhdGUNCihYRU4pIEhWTSBkM3YwIHNhdmU6IENQVQ0KKFhFTikgSFZNIGQz
IHNhdmU6IFBJQw0KKFhFTikgSFZNIGQzIHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQzdjAg
c2F2ZTogTEFQSUMNCihYRU4pIEhWTSBkM3YwIHNhdmU6IExBUElDX1JFR1MNCihYRU4pIEhW
TSBkMyBzYXZlOiBQQ0lfSVJRDQooWEVOKSBIVk0gZDMgc2F2ZTogSVNBX0lSUQ0KKFhFTikg
SFZNIGQzIHNhdmU6IFBDSV9MSU5LDQooWEVOKSBIVk0gZDMgc2F2ZTogUElUDQooWEVOKSBI
Vk0gZDMgc2F2ZTogUlRDDQooWEVOKSBIVk0gZDMgc2F2ZTogSFBFVA0KKFhFTikgSFZNIGQz
IHNhdmU6IFBNVElNRVINCihYRU4pIEhWTSBkM3YwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBk
MyBzYXZlOiBWSVJJRElBTl9ET01BSU4NCihYRU4pIEhWTSBkM3YwIHNhdmU6IENQVV9YU0FW
RQ0KKFhFTikgSFZNIGQzdjAgc2F2ZTogVklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQzdjAg
c2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBIVk0gZDN2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVO
KSBIVk0gZDN2MCBzYXZlOiBDUFVfTVNSDQooWEVOKSBIVk0zIHJlc3RvcmU6IENQVSAwDQoo
ZDMpIEhWTSBMb2FkZXINCihkMykgRGV0ZWN0ZWQgWGVuIHY0LjE1LjANCihkMykgWGVuYnVz
IHJpbmdzIEAweGZlZmZjMDAwLCBldmVudCBjaGFubmVsIDENCihkMykgU3lzdGVtIHJlcXVl
c3RlZCBTZWFCSU9TDQooZDMpIENQVSBzcGVlZCBpcyAyNTMzIE1Ieg0KKGQzKSBSZWxvY2F0
aW5nIGd1ZXN0IG1lbW9yeSBmb3IgbG93bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihkMykg
UENJLUlTQSBsaW5rIDAgcm91dGVkIHRvIElSUTUNCihkMykgUENJLUlTQSBsaW5rIDEgcm91
dGVkIHRvIElSUTEwDQooZDMpIFBDSS1JU0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0KKGQz
KSBQQ0ktSVNBIGxpbmsgMyByb3V0ZWQgdG8gSVJRNQ0KKGQzKSBwY2kgZGV2IDAxOjMgSU5U
QS0+SVJRMTANCihkMykgcGNpIGRldiAwMjowIElOVEEtPklSUTExDQooZDMpIHBjaSBkZXYg
MDQ6MCBJTlRBLT5JUlE1DQooZDMpIE5vIFJBTSBpbiBoaWdoIG1lbW9yeTsgc2V0dGluZyBo
aWdoX21lbSByZXNvdXJjZSBiYXNlIHRvIDEwMDAwMDAwMA0KKGQzKSBwY2kgZGV2IDAzOjAg
YmFyIDEwIHNpemUgMDAyMDAwMDAwOiAwZjAwMDAwMDgNCihkMykgcGNpIGRldiAwMjowIGJh
ciAxNCBzaXplIDAwMTAwMDAwMDogMGYyMDAwMDA4DQooZDMpIHBjaSBkZXYgMDM6MCBiYXIg
MzAgc2l6ZSAwMDAwMTAwMDA6IDBmMzAwMDAwMA0KKGQzKSBwY2kgZGV2IDAzOjAgYmFyIDE0
IHNpemUgMDAwMDAxMDAwOiAwZjMwMTAwMDANCihkMykgcGNpIGRldiAwNDowIGJhciAyNCBz
aXplIDAwMDAwMTAwMDogMGYzMDExMDAwDQooZDMpIHBjaSBkZXYgMDI6MCBiYXIgMTAgc2l6
ZSAwMDAwMDAxMDA6IDAwMDAwYzAwMQ0KKGQzKSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNpemUg
MDAwMDAwMDIwOiAwMDAwMGMxMDENCihkMykgcGNpIGRldiAwMToxIGJhciAyMCBzaXplIDAw
MDAwMDAxMDogMDAwMDBjMTIxDQooZDMpIE11bHRpcHJvY2Vzc29yIGluaXRpYWxpc2F0aW9u
Og0KKGQzKSAgLSBDUFUwIC4uLiAzNi1iaXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4uIHZh
ciBNVFJScyBbMS84XSAuLi4gZG9uZS4NCihkMykgV3JpdGluZyBTTUJJT1MgdGFibGVzIC4u
Lg0KKGQzKSBMb2FkaW5nIFNlYUJJT1MgLi4uDQooZDMpIENyZWF0aW5nIE1QIHRhYmxlcyAu
Li4NCihkMykgTG9hZGluZyBBQ1BJIC4uLg0KKGQzKSB2bTg2IFRTUyBhdCBmYzEwMDI4MA0K
KGQzKSBCSU9TIG1hcDoNCihkMykgIDEwMDAwLTEwMGUzOiBTY3JhdGNoIHNwYWNlDQooZDMp
ICBjMDAwMC1mZmZmZjogTWFpbiBCSU9TDQooZDMpIEU4MjAgdGFibGU6DQooZDMpICBbMDBd
OiAwMDAwMDAwMDowMDAwMDAwMCAtIDAwMDAwMDAwOjAwMGEwMDAwOiBSQU0NCihkMykgIEhP
TEU6IDAwMDAwMDAwOjAwMGEwMDAwIC0gMDAwMDAwMDA6MDAwYzAwMDANCihkMykgIFswMV06
IDAwMDAwMDAwOjAwMGMwMDAwIC0gMDAwMDAwMDA6MDAxMDAwMDA6IFJFU0VSVkVEDQooZDMp
ICBbMDJdOiAwMDAwMDAwMDowMDEwMDAwMCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0NCihk
MykgIEhPTEU6IDAwMDAwMDAwOjNmODAwMDAwIC0gMDAwMDAwMDA6ZmMwMDAwMDANCihkMykg
IFswM106IDAwMDAwMDAwOmZjMDAwMDAwIC0gMDAwMDAwMDA6ZmMwMGIwMDA6IE5WUw0KKGQz
KSAgWzA0XTogMDAwMDAwMDA6ZmMwMGIwMDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJW
RUQNCihkMykgSW52b2tpbmcgU2VhQklPUyAuLi4NCihkMykgU2VhQklPUyAodmVyc2lvbiAx
LjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQzKSBC
VUlMRDogZ2NjOiBtaXhlZCBiaW51dGlsczogKEdOVSBCaW51dGlscykgMi4zNw0KKGQzKSAN
CihkMykgRm91bmQgWGVuIGh5cGVydmlzb3Igc2lnbmF0dXJlIGF0IDQwMDAwMDAwDQooZDMp
IFJ1bm5pbmcgb24gUUVNVSAoaTQ0MGZ4KQ0KKGQzKSB4ZW46IGNvcHkgZTgyMC4uLg0KKGQz
KSBSZWxvY2F0aW5nIGluaXQgZnJvbSAweDAwMGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNpemUg
ODQxMjgpDQooZDMpIEZvdW5kIDcgUENJIGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0K
KGQzKSBBbGxvY2F0ZWQgWGVuIGh5cGVyY2FsbCBwYWdlIGF0IDNmN2ZmMDAwDQooZDMpIERl
dGVjdGVkIFhlbiB2NC4xNS4wDQooZDMpIHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0KKGQz
KSBDb3B5aW5nIFNNQklPUyBlbnRyeSBwb2ludCBmcm9tIDB4MDAwMTAwMjAgdG8gMHgwMDBm
NWUwMA0KKGQzKSBDb3B5aW5nIE1QVEFCTEUgZnJvbSAweGZjMTAwMTUwL2ZjMTAwMTYwIHRv
IDB4MDAwZjVkMjANCihkMykgQ29weWluZyBQSVIgZnJvbSAweDAwMDEwMDQwIHRvIDB4MDAw
ZjVjYTANCihkMykgQ29weWluZyBBQ1BJIFJTRFAgZnJvbSAweDAwMDEwMGMwIHRvIDB4MDAw
ZjVjNzANCihkMykgdGFibGUoNTA0MzQxNDYpPTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0KKGQz
KSBVc2luZyBwbXRpbWVyLCBpb3BvcnQgMHhiMDA4DQooZDMpIHRhYmxlKDUwNDM0MTQ2KT0w
eGZjMDBhMzcwICh2aWEgeHNkdCkNCihkMykgQUNQSTogcGFyc2UgRFNEVCBhdCAweGZjMDAx
MDQwIChsZW4gMzc1MzkpDQooZDMpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tp
cCBmcm9tIDE2LzI3NjQxDQooZDMpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tp
cCBmcm9tIDg3LzYwNDENCihkMykgU2NhbiBmb3IgVkdBIG9wdGlvbiByb20NCihkMykgUnVu
bmluZyBvcHRpb24gcm9tIGF0IGMwMDA6MDAwMw0KKGQzKSBwbW0gY2FsbCBhcmcxPTANCihk
MykgVHVybmluZyBvbiB2Z2EgdGV4dCBtb2RlIGNvbnNvbGUNCihkMykgU2VhQklPUyAodmVy
c2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0K
KGQzKSBNYWNoaW5lIFVVSUQgNDRjZWJlNjQtYWQ0NC0xMWVjLThjMzYtMDAxZTY3MDIwNzA3
DQooZDMpIEFUQSBjb250cm9sbGVyIDEgYXQgMWYwLzNmNC8wIChpcnEgMTQgZGV2IDkpDQoo
ZDMpIEFUQSBjb250cm9sbGVyIDIgYXQgMTcwLzM3NC8wIChpcnEgMTUgZGV2IDkpDQooZDMp
IEFIQ0kgY29udHJvbGxlciBhdCAwMDowNC4wLCBpb2Jhc2UgMHhmMzAxMTAwMCwgaXJxIDUN
CihkMykgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAv
ZGlza0AwDQooZDMpIEFIQ0kvMDogU2V0IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQooZDMp
IFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVAMC9k
aXNrQDANCihkMykgQUhDSS8wOiByZWdpc3RlcmluZzogIkFIQ0kvMDogUUVNVSBIQVJERElT
SyBBVEEtNyBIYXJkLURpc2sgKDEwMjQwIE1pQnl0ZXMpIg0KKGQzKSBQUzIga2V5Ym9hcmQg
aW5pdGlhbGl6ZWQNCihkMykgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihkMykg
Rm91bmQgMCBscHQgcG9ydHMNCihkMykgRm91bmQgMCBzZXJpYWwgcG9ydHMNCihkMykgQWxs
IHRocmVhZHMgY29tcGxldGUuDQooZDMpIFNjYW4gZm9yIG9wdGlvbiByb21zDQooZDMpIA0K
KGQzKSBQcmVzcyBFU0MgZm9yIGJvb3QgbWVudS4NCihkMykgDQooWEVOKSBIVk0gZDR2MCBz
YXZlOiBDUFUNCihYRU4pIEhWTSBkNCBzYXZlOiBQSUMNCihYRU4pIEhWTSBkNCBzYXZlOiBJ
T0FQSUMNCihYRU4pIEhWTSBkNHYwIHNhdmU6IExBUElDDQooWEVOKSBIVk0gZDR2MCBzYXZl
OiBMQVBJQ19SRUdTDQooWEVOKSBIVk0gZDQgc2F2ZTogUENJX0lSUQ0KKFhFTikgSFZNIGQ0
IHNhdmU6IElTQV9JUlENCihYRU4pIEhWTSBkNCBzYXZlOiBQQ0lfTElOSw0KKFhFTikgSFZN
IGQ0IHNhdmU6IFBJVA0KKFhFTikgSFZNIGQ0IHNhdmU6IFJUQw0KKFhFTikgSFZNIGQ0IHNh
dmU6IEhQRVQNCihYRU4pIEhWTSBkNCBzYXZlOiBQTVRJTUVSDQooWEVOKSBIVk0gZDR2MCBz
YXZlOiBNVFJSDQooWEVOKSBIVk0gZDQgc2F2ZTogVklSSURJQU5fRE9NQUlODQooWEVOKSBI
Vk0gZDR2MCBzYXZlOiBDUFVfWFNBVkUNCihYRU4pIEhWTSBkNHYwIHNhdmU6IFZJUklESUFO
X1ZDUFUNCihYRU4pIEhWTSBkNHYwIHNhdmU6IFZNQ0VfVkNQVQ0KKFhFTikgSFZNIGQ0djAg
c2F2ZTogVFNDX0FESlVTVA0KKFhFTikgSFZNIGQ0djAgc2F2ZTogQ1BVX01TUg0KKFhFTikg
SFZNNCByZXN0b3JlOiBDUFUgMA0KKGQzKSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogSEFM
VA0KKGQzKSBkcml2ZSAweDAwMGY1YmMwOiBQQ0hTPTE2MzgzLzE2LzYzIHRyYW5zbGF0aW9u
PWxiYSBMQ0hTPTEwMjQvMjU1LzYzIHM9MjA5NzE1MjANCihkMykgU3BhY2UgYXZhaWxhYmxl
IGZvciBVTUI6IGM5ODAwLWU5MDAwLCBmNTYyMC1mNWJjMA0KKGQzKSBSZXR1cm5lZCAyNTM5
NTIgYnl0ZXMgb2YgWm9uZUhpZ2gNCihkMykgZTgyMCBtYXAgaGFzIDcgaXRlbXM6DQooZDMp
ICAgMDogMDAwMDAwMDAwMDAwMDAwMCAtIDAwMDAwMDAwMDAwOWZjMDAgPSAxIFJBTQ0KKGQz
KSAgIDE6IDAwMDAwMDAwMDAwOWZjMDAgLSAwMDAwMDAwMDAwMGEwMDAwID0gMiBSRVNFUlZF
RA0KKGQzKSAgIDI6IDAwMDAwMDAwMDAwZjAwMDAgLSAwMDAwMDAwMDAwMTAwMDAwID0gMiBS
RVNFUlZFRA0KKGQzKSAgIDM6IDAwMDAwMDAwMDAxMDAwMDAgLSAwMDAwMDAwMDNmN2ZlMDAw
ID0gMSBSQU0NCihkMykgICA0OiAwMDAwMDAwMDNmN2ZlMDAwIC0gMDAwMDAwMDAzZjgwMDAw
MCA9IDIgUkVTRVJWRUQNCihkMykgICA1OiAwMDAwMDAwMGZjMDAwMDAwIC0gMDAwMDAwMDBm
YzAwYjAwMCA9IDQgTlZTDQooZDMpICAgNjogMDAwMDAwMDBmYzAwYjAwMCAtIDAwMDAwMDAx
MDAwMDAwMDAgPSAyIFJFU0VSVkVEDQooZDMpIGVudGVyIGhhbmRsZV8xOToNCihkMykgICBO
VUxMDQooZDMpIEJvb3RpbmcgZnJvbSBIYXJkIERpc2suLi4NCihkMykgQm9vdGluZyBmcm9t
IDAwMDA6N2MwMA0KKGQ0KSBIVk0gTG9hZGVyDQooZDQpIERldGVjdGVkIFhlbiB2NC4xNS4w
DQooZDQpIFhlbmJ1cyByaW5ncyBAMHhmZWZmYzAwMCwgZXZlbnQgY2hhbm5lbCAxDQooZDQp
IFN5c3RlbSByZXF1ZXN0ZWQgU2VhQklPUw0KKGQ0KSBDUFUgc3BlZWQgaXMgMjUzMyBNSHoN
CihkNCkgUmVsb2NhdGluZyBndWVzdCBtZW1vcnkgZm9yIGxvd21lbSBNTUlPIHNwYWNlIGRp
c2FibGVkDQooZDQpIFBDSS1JU0EgbGluayAwIHJvdXRlZCB0byBJUlE1DQooZDQpIFBDSS1J
U0EgbGluayAxIHJvdXRlZCB0byBJUlExMA0KKGQ0KSBQQ0ktSVNBIGxpbmsgMiByb3V0ZWQg
dG8gSVJRMTENCihkNCkgUENJLUlTQSBsaW5rIDMgcm91dGVkIHRvIElSUTUNCihkNCkgcGNp
IGRldiAwMTozIElOVEEtPklSUTEwDQooZDQpIHBjaSBkZXYgMDI6MCBJTlRBLT5JUlExMQ0K
KGQ0KSBwY2kgZGV2IDA0OjAgSU5UQS0+SVJRNQ0KKGQ0KSBObyBSQU0gaW4gaGlnaCBtZW1v
cnk7IHNldHRpbmcgaGlnaF9tZW0gcmVzb3VyY2UgYmFzZSB0byAxMDAwMDAwMDANCihkNCkg
cGNpIGRldiAwMzowIGJhciAxMCBzaXplIDAwMjAwMDAwMDogMGYwMDAwMDA4DQooZDQpIHBj
aSBkZXYgMDI6MCBiYXIgMTQgc2l6ZSAwMDEwMDAwMDA6IDBmMjAwMDAwOA0KKGQ0KSBwY2kg
ZGV2IDAzOjAgYmFyIDMwIHNpemUgMDAwMDEwMDAwOiAwZjMwMDAwMDANCihkNCkgcGNpIGRl
diAwMzowIGJhciAxNCBzaXplIDAwMDAwMTAwMDogMGYzMDEwMDAwDQooZDQpIHBjaSBkZXYg
MDQ6MCBiYXIgMjQgc2l6ZSAwMDAwMDEwMDA6IDBmMzAxMTAwMA0KKGQ0KSBwY2kgZGV2IDAy
OjAgYmFyIDEwIHNpemUgMDAwMDAwMTAwOiAwMDAwMGMwMDENCihkNCkgcGNpIGRldiAwNDow
IGJhciAyMCBzaXplIDAwMDAwMDAyMDogMDAwMDBjMTAxDQooZDQpIHBjaSBkZXYgMDE6MSBi
YXIgMjAgc2l6ZSAwMDAwMDAwMTA6IDAwMDAwYzEyMQ0KKGQ0KSBNdWx0aXByb2Nlc3NvciBp
bml0aWFsaXNhdGlvbjoNCihkNCkgIC0gQ1BVMCAuLi4gMzYtYml0IHBoeXMgLi4uIGZpeGVk
IE1UUlJzIC4uLiB2YXIgTVRSUnMgWzEvOF0gLi4uIGRvbmUuDQooZDQpIFdyaXRpbmcgU01C
SU9TIHRhYmxlcyAuLi4NCihkNCkgTG9hZGluZyBTZWFCSU9TIC4uLg0KKGQ0KSBDcmVhdGlu
ZyBNUCB0YWJsZXMgLi4uDQooZDQpIExvYWRpbmcgQUNQSSAuLi4NCihkNCkgdm04NiBUU1Mg
YXQgZmMxMDAyODANCihkNCkgQklPUyBtYXA6DQooZDQpICAxMDAwMC0xMDBlMzogU2NyYXRj
aCBzcGFjZQ0KKGQ0KSAgYzAwMDAtZmZmZmY6IE1haW4gQklPUw0KKGQ0KSBFODIwIHRhYmxl
Og0KKGQ0KSAgWzAwXTogMDAwMDAwMDA6MDAwMDAwMDAgLSAwMDAwMDAwMDowMDBhMDAwMDog
UkFNDQooZDQpICBIT0xFOiAwMDAwMDAwMDowMDBhMDAwMCAtIDAwMDAwMDAwOjAwMGMwMDAw
DQooZDQpICBbMDFdOiAwMDAwMDAwMDowMDBjMDAwMCAtIDAwMDAwMDAwOjAwMTAwMDAwOiBS
RVNFUlZFRA0KKGQ0KSAgWzAyXTogMDAwMDAwMDA6MDAxMDAwMDAgLSAwMDAwMDAwMDozZjgw
MDAwMDogUkFNDQooZDQpICBIT0xFOiAwMDAwMDAwMDozZjgwMDAwMCAtIDAwMDAwMDAwOmZj
MDAwMDAwDQooZDQpICBbMDNdOiAwMDAwMDAwMDpmYzAwMDAwMCAtIDAwMDAwMDAwOmZjMDBi
MDAwOiBOVlMNCihkNCkgIFswNF06IDAwMDAwMDAwOmZjMDBiMDAwIC0gMDAwMDAwMDE6MDAw
MDAwMDA6IFJFU0VSVkVEDQooZDQpIEludm9raW5nIFNlYUJJT1MgLi4uDQooZDQpIFNlYUJJ
T1MgKHZlcnNpb24gMS4xNC4wLTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpv
Yi0wNCkNCihkNCkgQlVJTEQ6IGdjYzogbWl4ZWQgYmludXRpbHM6IChHTlUgQmludXRpbHMp
IDIuMzcNCihkNCkgDQooZDQpIEZvdW5kIFhlbiBoeXBlcnZpc29yIHNpZ25hdHVyZSBhdCA0
MDAwMDAwMA0KKGQ0KSBSdW5uaW5nIG9uIFFFTVUgKGk0NDBmeCkNCihkNCkgeGVuOiBjb3B5
IGU4MjAuLi4NCihkNCkgUmVsb2NhdGluZyBpbml0IGZyb20gMHgwMDBkNjIwMCB0byAweDNm
N2FiNmMwIChzaXplIDg0MTI4KQ0KKGQ0KSBGb3VuZCA3IFBDSSBkZXZpY2VzIChtYXggUENJ
IGJ1cyBpcyAwMCkNCihkNCkgQWxsb2NhdGVkIFhlbiBoeXBlcmNhbGwgcGFnZSBhdCAzZjdm
ZjAwMA0KKGQ0KSBEZXRlY3RlZCBYZW4gdjQuMTUuMA0KKGQ0KSB4ZW46IGNvcHkgQklPUyB0
YWJsZXMuLi4NCihkNCkgQ29weWluZyBTTUJJT1MgZW50cnkgcG9pbnQgZnJvbSAweDAwMDEw
MDIwIHRvIDB4MDAwZjVlMDANCihkNCkgQ29weWluZyBNUFRBQkxFIGZyb20gMHhmYzEwMDE1
MC9mYzEwMDE2MCB0byAweDAwMGY1ZDIwDQooZDQpIENvcHlpbmcgUElSIGZyb20gMHgwMDAx
MDA0MCB0byAweDAwMGY1Y2EwDQooZDQpIENvcHlpbmcgQUNQSSBSU0RQIGZyb20gMHgwMDAx
MDBjMCB0byAweDAwMGY1YzcwDQooZDQpIHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcwICh2
aWEgeHNkdCkNCihkNCkgVXNpbmcgcG10aW1lciwgaW9wb3J0IDB4YjAwOA0KKGQ0KSB0YWJs
ZSg1MDQzNDE0Nik9MHhmYzAwYTM3MCAodmlhIHhzZHQpDQooZDQpIEFDUEk6IHBhcnNlIERT
RFQgYXQgMHhmYzAwMTA0MCAobGVuIDM3NTM5KQ0KKGQ0KSBwYXJzZV90ZXJtbGlzdDogcGFy
c2UgZXJyb3IsIHNraXAgZnJvbSAxNi8yNzY0MQ0KKGQ0KSBwYXJzZV90ZXJtbGlzdDogcGFy
c2UgZXJyb3IsIHNraXAgZnJvbSA4Ny82MDQxDQooZDQpIFNjYW4gZm9yIFZHQSBvcHRpb24g
cm9tDQooZDQpIFJ1bm5pbmcgb3B0aW9uIHJvbSBhdCBjMDAwOjAwMDMNCihkNCkgcG1tIGNh
bGwgYXJnMT0wDQooZDQpIFR1cm5pbmcgb24gdmdhIHRleHQgbW9kZSBjb25zb2xlDQooZDQp
IFNlYUJJT1MgKHZlcnNpb24gMS4xNC4wLTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZh
dWx0LWpvYi0wNCkNCihkNCkgTWFjaGluZSBVVUlEIDRkNTQxMWI2LWFkNDQtMTFlYy04YzM2
LTAwMWU2NzAyMDcwNw0KKGQ0KSBBVEEgY29udHJvbGxlciAxIGF0IDFmMC8zZjQvMCAoaXJx
IDE0IGRldiA5KQ0KKGQ0KSBBVEEgY29udHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1
IGRldiA5KQ0KKGQ0KSBBSENJIGNvbnRyb2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMw
MTEwMDAsIGlycSA1DQooZDQpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4
LypANC9kcml2ZUAwL2Rpc2tAMA0KKGQ0KSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRv
IFVETUEtNQ0KKGQ0KSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgv
KkA0L2RyaXZlQDAvZGlza0AwDQooZDQpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNp
QGkwY2Y4LypANC9kcml2ZUAxL2Rpc2tAMA0KKGQ0KSBBSENJLzE6IFNldCB0cmFuc2ZlciBt
b2RlIHRvIFVETUEtNQ0KKGQ0KSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lA
aTBjZjgvKkA0L2RyaXZlQDEvZGlza0AwDQooZDQpIEFIQ0kvMDogcmVnaXN0ZXJpbmc6ICJB
SENJLzA6IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgxMDI0MCBNaUJ5dGVzKSIN
CihkNCkgUFMyIGtleWJvYXJkIGluaXRpYWxpemVkDQooZDQpIFNlYXJjaGluZyBib290b3Jk
ZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAyL2Rpc2tAMA0KKGQ0KSBBSENJLzI6IFNl
dCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ0KSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRy
eSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDIvZGlza0AwDQooZDQpIEFIQ0kvMTogcmVn
aXN0ZXJpbmc6ICJBSENJLzE6IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4
IE1pQnl0ZXMpIg0KKGQ0KSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogL3BjaUBpMGNmOC8q
QDQvZHJpdmVAMy9kaXNrQDANCihkNCkgQUhDSS8zOiBTZXQgdHJhbnNmZXIgbW9kZSB0byBV
RE1BLTUNCihkNCkgU2VhcmNoaW5nIGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypA
NC9kcml2ZUAzL2Rpc2tAMA0KKGQ0KSBBSENJLzI6IHJlZ2lzdGVyaW5nOiAiQUhDSS8yOiBR
RU1VIEhBUkRESVNLIEFUQS03IEhhcmQtRGlzayAoMjA0OCBNaUJ5dGVzKSINCihkNCkgU2Vh
cmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDQvZGlza0AwDQoo
ZDQpIEFIQ0kvNDogU2V0IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQooZDQpIFNlYXJjaGlu
ZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVANC9kaXNrQDANCihk
NCkgQUhDSS8zOiByZWdpc3RlcmluZzogIkFIQ0kvMzogUUVNVSBIQVJERElTSyBBVEEtNyBI
YXJkLURpc2sgKDIwNDggTWlCeXRlcykiDQooZDQpIEFIQ0kvNDogcmVnaXN0ZXJpbmc6ICJB
SENJLzQ6IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0K
KGQ0KSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogSEFMVA0KKGQ0KSBGb3VuZCAwIGxwdCBw
b3J0cw0KKGQ0KSBGb3VuZCAwIHNlcmlhbCBwb3J0cw0KKGQ0KSBBbGwgdGhyZWFkcyBjb21w
bGV0ZS4NCihkNCkgU2NhbiBmb3Igb3B0aW9uIHJvbXMNCihkNCkgDQooZDQpIFByZXNzIEVT
QyBmb3IgYm9vdCBtZW51Lg0KKGQ0KSANCihkNCkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6
IEhBTFQNCihkNCkgZHJpdmUgMHgwMDBmNWJjMDogUENIUz0xNjM4My8xNi82MyB0cmFuc2xh
dGlvbj1sYmEgTENIUz0xMDI0LzI1NS82MyBzPTIwOTcxNTIwDQooZDQpIGRyaXZlIDB4MDAw
ZjViNzA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgv
NjMgcz00MTk0MzA0DQooZDQpIGRyaXZlIDB4MDAwZjViMjA6IFBDSFM9NDE2MS8xNi82MyB0
cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDQpIGRyaXZl
IDB4MDAwZjVhZDA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUy
MC8xMjgvNjMgcz00MTk0MzA0DQooZDQpIGRyaXZlIDB4MDAwZjVhODA6IFBDSFM9NDE2MS8x
Ni82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDQp
IFNwYWNlIGF2YWlsYWJsZSBmb3IgVU1COiBjOTgwMC1lOTAwMCwgZjU2MjAtZjVhODANCihk
NCkgUmV0dXJuZWQgMjQ5ODU2IGJ5dGVzIG9mIFpvbmVIaWdoDQooZDQpIGU4MjAgbWFwIGhh
cyA3IGl0ZW1zOg0KKGQ0KSAgIDA6IDAwMDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAwMDAwMDlm
YzAwID0gMSBSQU0NCihkNCkgICAxOiAwMDAwMDAwMDAwMDlmYzAwIC0gMDAwMDAwMDAwMDBh
MDAwMCA9IDIgUkVTRVJWRUQNCihkNCkgICAyOiAwMDAwMDAwMDAwMGYwMDAwIC0gMDAwMDAw
MDAwMDEwMDAwMCA9IDIgUkVTRVJWRUQNCihkNCkgICAzOiAwMDAwMDAwMDAwMTAwMDAwIC0g
MDAwMDAwMDAzZjdmZDAwMCA9IDEgUkFNDQooZDQpICAgNDogMDAwMDAwMDAzZjdmZDAwMCAt
IDAwMDAwMDAwM2Y4MDAwMDAgPSAyIFJFU0VSVkVEDQooZDQpICAgNTogMDAwMDAwMDBmYzAw
MDAwMCAtIDAwMDAwMDAwZmMwMGIwMDAgPSA0IE5WUw0KKGQ0KSAgIDY6IDAwMDAwMDAwZmMw
MGIwMDAgLSAwMDAwMDAwMTAwMDAwMDAwID0gMiBSRVNFUlZFRA0KKGQ0KSBlbnRlciBoYW5k
bGVfMTk6DQooZDQpICAgTlVMTA0KKGQ0KSBCb290aW5nIGZyb20gSGFyZCBEaXNrLi4uDQoo
ZDQpIEJvb3RpbmcgZnJvbSAwMDAwOjdjMDANCihYRU4pIGQzdjA6IHVwY2FsbCB2ZWN0b3Ig
OTMNCihYRU4pIGQ0djA6IHVwY2FsbCB2ZWN0b3IgOTMNCnhiYmQxOiBFcnJvciAxMiBVbmFi
bGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDE6IEZhdGFsIGVy
cm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCnhiYmQyOiBFcnJvciAxMiBV
bmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDI6IEZhdGFs
IGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCnhiYmQ0OiBFcnJvciAx
MiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDQ6IEZh
dGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCnhiYmQ1OiBFcnJv
ciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDU6
IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCihYRU4pIEhW
TSBkNXYwIHNhdmU6IENQVQ0KKFhFTikgSFZNIGQ1IHNhdmU6IFBJQw0KKFhFTikgSFZNIGQ1
IHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQ1djAgc2F2ZTogTEFQSUMNCihYRU4pIEhWTSBk
NXYwIHNhdmU6IExBUElDX1JFR1MNCihYRU4pIEhWTSBkNSBzYXZlOiBQQ0lfSVJRDQooWEVO
KSBIVk0gZDUgc2F2ZTogSVNBX0lSUQ0KKFhFTikgSFZNIGQ1IHNhdmU6IFBDSV9MSU5LDQoo
WEVOKSBIVk0gZDUgc2F2ZTogUElUDQooWEVOKSBIVk0gZDUgc2F2ZTogUlRDDQooWEVOKSBI
Vk0gZDUgc2F2ZTogSFBFVA0KKFhFTikgSFZNIGQ1IHNhdmU6IFBNVElNRVINCihYRU4pIEhW
TSBkNXYwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBkNSBzYXZlOiBWSVJJRElBTl9ET01BSU4N
CihYRU4pIEhWTSBkNXYwIHNhdmU6IENQVV9YU0FWRQ0KKFhFTikgSFZNIGQ1djAgc2F2ZTog
VklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQ1djAgc2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBI
Vk0gZDV2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVOKSBIVk0gZDV2MCBzYXZlOiBDUFVfTVNS
DQooWEVOKSBIVk01IHJlc3RvcmU6IENQVSAwDQooZDUpIEhWTSBMb2FkZXINCihkNSkgRGV0
ZWN0ZWQgWGVuIHY0LjE1LjANCihkNSkgWGVuYnVzIHJpbmdzIEAweGZlZmZjMDAwLCBldmVu
dCBjaGFubmVsIDENCihkNSkgU3lzdGVtIHJlcXVlc3RlZCBTZWFCSU9TDQooZDUpIENQVSBz
cGVlZCBpcyAyNTMzIE1Ieg0KKGQ1KSBSZWxvY2F0aW5nIGd1ZXN0IG1lbW9yeSBmb3IgbG93
bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihkNSkgUENJLUlTQSBsaW5rIDAgcm91dGVkIHRv
IElSUTUNCihkNSkgUENJLUlTQSBsaW5rIDEgcm91dGVkIHRvIElSUTEwDQooZDUpIFBDSS1J
U0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0KKGQ1KSBQQ0ktSVNBIGxpbmsgMyByb3V0ZWQg
dG8gSVJRNQ0KKGQ1KSBwY2kgZGV2IDAxOjMgSU5UQS0+SVJRMTANCihkNSkgcGNpIGRldiAw
MjowIElOVEEtPklSUTExDQooZDUpIHBjaSBkZXYgMDQ6MCBJTlRBLT5JUlE1DQooZDUpIE5v
IFJBTSBpbiBoaWdoIG1lbW9yeTsgc2V0dGluZyBoaWdoX21lbSByZXNvdXJjZSBiYXNlIHRv
IDEwMDAwMDAwMA0KKGQ1KSBwY2kgZGV2IDAzOjAgYmFyIDEwIHNpemUgMDAyMDAwMDAwOiAw
ZjAwMDAwMDgNCihkNSkgcGNpIGRldiAwMjowIGJhciAxNCBzaXplIDAwMTAwMDAwMDogMGYy
MDAwMDA4DQooZDUpIHBjaSBkZXYgMDM6MCBiYXIgMzAgc2l6ZSAwMDAwMTAwMDA6IDBmMzAw
MDAwMA0KKGQ1KSBwY2kgZGV2IDAzOjAgYmFyIDE0IHNpemUgMDAwMDAxMDAwOiAwZjMwMTAw
MDANCihkNSkgcGNpIGRldiAwNDowIGJhciAyNCBzaXplIDAwMDAwMTAwMDogMGYzMDExMDAw
DQooZDUpIHBjaSBkZXYgMDI6MCBiYXIgMTAgc2l6ZSAwMDAwMDAxMDA6IDAwMDAwYzAwMQ0K
KGQ1KSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNpemUgMDAwMDAwMDIwOiAwMDAwMGMxMDENCihk
NSkgcGNpIGRldiAwMToxIGJhciAyMCBzaXplIDAwMDAwMDAxMDogMDAwMDBjMTIxDQooZDUp
IE11bHRpcHJvY2Vzc29yIGluaXRpYWxpc2F0aW9uOg0KKGQ1KSAgLSBDUFUwIC4uLiAzNi1i
aXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4uIHZhciBNVFJScyBbMS84XSAuLi4gZG9uZS4N
CihkNSkgV3JpdGluZyBTTUJJT1MgdGFibGVzIC4uLg0KKGQ1KSBMb2FkaW5nIFNlYUJJT1Mg
Li4uDQooZDUpIENyZWF0aW5nIE1QIHRhYmxlcyAuLi4NCihkNSkgTG9hZGluZyBBQ1BJIC4u
Lg0KKGQ1KSB2bTg2IFRTUyBhdCBmYzEwMDI4MA0KKGQ1KSBCSU9TIG1hcDoNCihkNSkgIDEw
MDAwLTEwMGUzOiBTY3JhdGNoIHNwYWNlDQooZDUpICBjMDAwMC1mZmZmZjogTWFpbiBCSU9T
DQooZDUpIEU4MjAgdGFibGU6DQooZDUpICBbMDBdOiAwMDAwMDAwMDowMDAwMDAwMCAtIDAw
MDAwMDAwOjAwMGEwMDAwOiBSQU0NCihkNSkgIEhPTEU6IDAwMDAwMDAwOjAwMGEwMDAwIC0g
MDAwMDAwMDA6MDAwYzAwMDANCihkNSkgIFswMV06IDAwMDAwMDAwOjAwMGMwMDAwIC0gMDAw
MDAwMDA6MDAxMDAwMDA6IFJFU0VSVkVEDQooZDUpICBbMDJdOiAwMDAwMDAwMDowMDEwMDAw
MCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0NCihkNSkgIEhPTEU6IDAwMDAwMDAwOjNmODAw
MDAwIC0gMDAwMDAwMDA6ZmMwMDAwMDANCihkNSkgIFswM106IDAwMDAwMDAwOmZjMDAwMDAw
IC0gMDAwMDAwMDA6ZmMwMGIwMDA6IE5WUw0KKGQ1KSAgWzA0XTogMDAwMDAwMDA6ZmMwMGIw
MDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJWRUQNCihkNSkgSW52b2tpbmcgU2VhQklP
UyAuLi4NCihkNSkgU2VhQklPUyAodmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEz
MGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQ1KSBCVUlMRDogZ2NjOiBtaXhlZCBiaW51dGls
czogKEdOVSBCaW51dGlscykgMi4zNw0KKGQ1KSANCihkNSkgRm91bmQgWGVuIGh5cGVydmlz
b3Igc2lnbmF0dXJlIGF0IDQwMDAwMDAwDQooZDUpIFJ1bm5pbmcgb24gUUVNVSAoaTQ0MGZ4
KQ0KKGQ1KSB4ZW46IGNvcHkgZTgyMC4uLg0KKGQ1KSBSZWxvY2F0aW5nIGluaXQgZnJvbSAw
eDAwMGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNpemUgODQxMjgpDQooZDUpIEZvdW5kIDcgUENJ
IGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0KKGQ1KSBBbGxvY2F0ZWQgWGVuIGh5cGVy
Y2FsbCBwYWdlIGF0IDNmN2ZmMDAwDQooZDUpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDUp
IHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0KKGQ1KSBDb3B5aW5nIFNNQklPUyBlbnRyeSBw
b2ludCBmcm9tIDB4MDAwMTAwMjAgdG8gMHgwMDBmNWUwMA0KKGQ1KSBDb3B5aW5nIE1QVEFC
TEUgZnJvbSAweGZjMTAwMTUwL2ZjMTAwMTYwIHRvIDB4MDAwZjVkMjANCihkNSkgQ29weWlu
ZyBQSVIgZnJvbSAweDAwMDEwMDQwIHRvIDB4MDAwZjVjYTANCihkNSkgQ29weWluZyBBQ1BJ
IFJTRFAgZnJvbSAweDAwMDEwMGMwIHRvIDB4MDAwZjVjNzANCihkNSkgdGFibGUoNTA0MzQx
NDYpPTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0KKGQ1KSBVc2luZyBwbXRpbWVyLCBpb3BvcnQg
MHhiMDA4DQooZDUpIHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcwICh2aWEgeHNkdCkNCihk
NSkgQUNQSTogcGFyc2UgRFNEVCBhdCAweGZjMDAxMDQwIChsZW4gMzc1MzkpDQooZDUpIHBh
cnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tpcCBmcm9tIDE2LzI3NjQxDQooZDUpIHBh
cnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tpcCBmcm9tIDg3LzYwNDENCihkNSkgU2Nh
biBmb3IgVkdBIG9wdGlvbiByb20NCihkNSkgUnVubmluZyBvcHRpb24gcm9tIGF0IGMwMDA6
MDAwMw0KKGQ1KSBwbW0gY2FsbCBhcmcxPTANCihkNSkgVHVybmluZyBvbiB2Z2EgdGV4dCBt
b2RlIGNvbnNvbGUNCihkNSkgU2VhQklPUyAodmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAz
MDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQ1KSBNYWNoaW5lIFVVSUQgNzk2MDVl
ZWUtYWQ0NC0xMWVjLThjMzYtMDAxZTY3MDIwNzA3DQooZDUpIEFUQSBjb250cm9sbGVyIDEg
YXQgMWYwLzNmNC8wIChpcnEgMTQgZGV2IDkpDQooZDUpIEFUQSBjb250cm9sbGVyIDIgYXQg
MTcwLzM3NC8wIChpcnEgMTUgZGV2IDkpDQooZDUpIEFIQ0kgY29udHJvbGxlciBhdCAwMDow
NC4wLCBpb2Jhc2UgMHhmMzAxMTAwMCwgaXJxIDUNCihkNSkgU2VhcmNoaW5nIGJvb3RvcmRl
ciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlza0AwDQooZDUpIEFIQ0kvMDogU2V0
IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQooZDUpIFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5
IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVAMC9kaXNrQDANCihkNSkgQUhDSS8wOiByZWdp
c3RlcmluZzogIkFIQ0kvMDogUUVNVSBIQVJERElTSyBBVEEtNyBIYXJkLURpc2sgKDEwMjQw
IE1pQnl0ZXMpIg0KKGQ1KSBQUzIga2V5Ym9hcmQgaW5pdGlhbGl6ZWQNCihkNSkgU2VhcmNo
aW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihkNSkgRm91bmQgMCBscHQgcG9ydHMNCihkNSkg
Rm91bmQgMCBzZXJpYWwgcG9ydHMNCihkNSkgQWxsIHRocmVhZHMgY29tcGxldGUuDQooZDUp
IFNjYW4gZm9yIG9wdGlvbiByb21zDQooZDUpIA0KKGQ1KSBQcmVzcyBFU0MgZm9yIGJvb3Qg
bWVudS4NCihkNSkgDQooZDUpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiBIQUxUDQooZDUp
IGRyaXZlIDB4MDAwZjViYzA6IFBDSFM9MTYzODMvMTYvNjMgdHJhbnNsYXRpb249bGJhIExD
SFM9MTAyNC8yNTUvNjMgcz0yMDk3MTUyMA0KKGQ1KSBTcGFjZSBhdmFpbGFibGUgZm9yIFVN
QjogYzk4MDAtZTkwMDAsIGY1NjIwLWY1YmMwDQooZDUpIFJldHVybmVkIDI1Mzk1MiBieXRl
cyBvZiBab25lSGlnaA0KKGQ1KSBlODIwIG1hcCBoYXMgNyBpdGVtczoNCihkNSkgICAwOiAw
MDAwMDAwMDAwMDAwMDAwIC0gMDAwMDAwMDAwMDA5ZmMwMCA9IDEgUkFNDQooZDUpICAgMTog
MDAwMDAwMDAwMDA5ZmMwMCAtIDAwMDAwMDAwMDAwYTAwMDAgPSAyIFJFU0VSVkVEDQooZDUp
ICAgMjogMDAwMDAwMDAwMDBmMDAwMCAtIDAwMDAwMDAwMDAxMDAwMDAgPSAyIFJFU0VSVkVE
DQooZDUpICAgMzogMDAwMDAwMDAwMDEwMDAwMCAtIDAwMDAwMDAwM2Y3ZmUwMDAgPSAxIFJB
TQ0KKGQ1KSAgIDQ6IDAwMDAwMDAwM2Y3ZmUwMDAgLSAwMDAwMDAwMDNmODAwMDAwID0gMiBS
RVNFUlZFRA0KKGQ1KSAgIDU6IDAwMDAwMDAwZmMwMDAwMDAgLSAwMDAwMDAwMGZjMDBiMDAw
ID0gNCBOVlMNCihkNSkgICA2OiAwMDAwMDAwMGZjMDBiMDAwIC0gMDAwMDAwMDEwMDAwMDAw
MCA9IDIgUkVTRVJWRUQNCihkNSkgZW50ZXIgaGFuZGxlXzE5Og0KKGQ1KSAgIE5VTEwNCihk
NSkgQm9vdGluZyBmcm9tIEhhcmQgRGlzay4uLg0KKGQ1KSBCb290aW5nIGZyb20gMDAwMDo3
YzAwDQooWEVOKSBIVk0gZDZ2MCBzYXZlOiBDUFUNCihYRU4pIEhWTSBkNiBzYXZlOiBQSUMN
CihYRU4pIEhWTSBkNiBzYXZlOiBJT0FQSUMNCihYRU4pIEhWTSBkNnYwIHNhdmU6IExBUElD
DQooWEVOKSBIVk0gZDZ2MCBzYXZlOiBMQVBJQ19SRUdTDQooWEVOKSBIVk0gZDYgc2F2ZTog
UENJX0lSUQ0KKFhFTikgSFZNIGQ2IHNhdmU6IElTQV9JUlENCihYRU4pIEhWTSBkNiBzYXZl
OiBQQ0lfTElOSw0KKFhFTikgSFZNIGQ2IHNhdmU6IFBJVA0KKFhFTikgSFZNIGQ2IHNhdmU6
IFJUQw0KKFhFTikgSFZNIGQ2IHNhdmU6IEhQRVQNCihYRU4pIEhWTSBkNiBzYXZlOiBQTVRJ
TUVSDQooWEVOKSBIVk0gZDZ2MCBzYXZlOiBNVFJSDQooWEVOKSBIVk0gZDYgc2F2ZTogVklS
SURJQU5fRE9NQUlODQooWEVOKSBIVk0gZDZ2MCBzYXZlOiBDUFVfWFNBVkUNCihYRU4pIEhW
TSBkNnYwIHNhdmU6IFZJUklESUFOX1ZDUFUNCihYRU4pIEhWTSBkNnYwIHNhdmU6IFZNQ0Vf
VkNQVQ0KKFhFTikgSFZNIGQ2djAgc2F2ZTogVFNDX0FESlVTVA0KKFhFTikgSFZNIGQ2djAg
c2F2ZTogQ1BVX01TUg0KKFhFTikgSFZNNiByZXN0b3JlOiBDUFUgMA0KKGQ2KSBIVk0gTG9h
ZGVyDQooZDYpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDYpIFhlbmJ1cyByaW5ncyBAMHhm
ZWZmYzAwMCwgZXZlbnQgY2hhbm5lbCAxDQooZDYpIFN5c3RlbSByZXF1ZXN0ZWQgU2VhQklP
Uw0KKGQ2KSBDUFUgc3BlZWQgaXMgMjUzMyBNSHoNCihkNikgUmVsb2NhdGluZyBndWVzdCBt
ZW1vcnkgZm9yIGxvd21lbSBNTUlPIHNwYWNlIGRpc2FibGVkDQooZDYpIFBDSS1JU0EgbGlu
ayAwIHJvdXRlZCB0byBJUlE1DQooZDYpIFBDSS1JU0EgbGluayAxIHJvdXRlZCB0byBJUlEx
MA0KKGQ2KSBQQ0ktSVNBIGxpbmsgMiByb3V0ZWQgdG8gSVJRMTENCihkNikgUENJLUlTQSBs
aW5rIDMgcm91dGVkIHRvIElSUTUNCihkNikgcGNpIGRldiAwMTozIElOVEEtPklSUTEwDQoo
ZDYpIHBjaSBkZXYgMDI6MCBJTlRBLT5JUlExMQ0KKGQ2KSBwY2kgZGV2IDA0OjAgSU5UQS0+
SVJRNQ0KKGQ2KSBObyBSQU0gaW4gaGlnaCBtZW1vcnk7IHNldHRpbmcgaGlnaF9tZW0gcmVz
b3VyY2UgYmFzZSB0byAxMDAwMDAwMDANCihkNikgcGNpIGRldiAwMzowIGJhciAxMCBzaXpl
IDAwMjAwMDAwMDogMGYwMDAwMDA4DQooZDYpIHBjaSBkZXYgMDI6MCBiYXIgMTQgc2l6ZSAw
MDEwMDAwMDA6IDBmMjAwMDAwOA0KKGQ2KSBwY2kgZGV2IDAzOjAgYmFyIDMwIHNpemUgMDAw
MDEwMDAwOiAwZjMwMDAwMDANCihkNikgcGNpIGRldiAwMzowIGJhciAxNCBzaXplIDAwMDAw
MTAwMDogMGYzMDEwMDAwDQooZDYpIHBjaSBkZXYgMDQ6MCBiYXIgMjQgc2l6ZSAwMDAwMDEw
MDA6IDBmMzAxMTAwMA0KKGQ2KSBwY2kgZGV2IDAyOjAgYmFyIDEwIHNpemUgMDAwMDAwMTAw
OiAwMDAwMGMwMDENCihkNikgcGNpIGRldiAwNDowIGJhciAyMCBzaXplIDAwMDAwMDAyMDog
MDAwMDBjMTAxDQooZDYpIHBjaSBkZXYgMDE6MSBiYXIgMjAgc2l6ZSAwMDAwMDAwMTA6IDAw
MDAwYzEyMQ0KKGQ2KSBNdWx0aXByb2Nlc3NvciBpbml0aWFsaXNhdGlvbjoNCihkNikgIC0g
Q1BVMCAuLi4gMzYtYml0IHBoeXMgLi4uIGZpeGVkIE1UUlJzIC4uLiB2YXIgTVRSUnMgWzEv
OF0gLi4uIGRvbmUuDQooZDYpIFdyaXRpbmcgU01CSU9TIHRhYmxlcyAuLi4NCihkNikgTG9h
ZGluZyBTZWFCSU9TIC4uLg0KKGQ2KSBDcmVhdGluZyBNUCB0YWJsZXMgLi4uDQooZDYpIExv
YWRpbmcgQUNQSSAuLi4NCihkNikgdm04NiBUU1MgYXQgZmMxMDAyODANCihkNikgQklPUyBt
YXA6DQooZDYpICAxMDAwMC0xMDBlMzogU2NyYXRjaCBzcGFjZQ0KKGQ2KSAgYzAwMDAtZmZm
ZmY6IE1haW4gQklPUw0KKGQ2KSBFODIwIHRhYmxlOg0KKGQ2KSAgWzAwXTogMDAwMDAwMDA6
MDAwMDAwMDAgLSAwMDAwMDAwMDowMDBhMDAwMDogUkFNDQooZDYpICBIT0xFOiAwMDAwMDAw
MDowMDBhMDAwMCAtIDAwMDAwMDAwOjAwMGMwMDAwDQooZDYpICBbMDFdOiAwMDAwMDAwMDow
MDBjMDAwMCAtIDAwMDAwMDAwOjAwMTAwMDAwOiBSRVNFUlZFRA0KKGQ2KSAgWzAyXTogMDAw
MDAwMDA6MDAxMDAwMDAgLSAwMDAwMDAwMDozZjgwMDAwMDogUkFNDQooZDYpICBIT0xFOiAw
MDAwMDAwMDozZjgwMDAwMCAtIDAwMDAwMDAwOmZjMDAwMDAwDQooZDYpICBbMDNdOiAwMDAw
MDAwMDpmYzAwMDAwMCAtIDAwMDAwMDAwOmZjMDBiMDAwOiBOVlMNCihkNikgIFswNF06IDAw
MDAwMDAwOmZjMDBiMDAwIC0gMDAwMDAwMDE6MDAwMDAwMDA6IFJFU0VSVkVEDQooZDYpIElu
dm9raW5nIFNlYUJJT1MgLi4uDQooZDYpIFNlYUJJT1MgKHZlcnNpb24gMS4xNC4wLTIwMjIw
MjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkNikgQlVJTEQ6IGdjYzog
bWl4ZWQgYmludXRpbHM6IChHTlUgQmludXRpbHMpIDIuMzcNCihkNikgDQooZDYpIEZvdW5k
IFhlbiBoeXBlcnZpc29yIHNpZ25hdHVyZSBhdCA0MDAwMDAwMA0KKGQ2KSBSdW5uaW5nIG9u
IFFFTVUgKGk0NDBmeCkNCihkNikgeGVuOiBjb3B5IGU4MjAuLi4NCihkNikgUmVsb2NhdGlu
ZyBpbml0IGZyb20gMHgwMDBkNjIwMCB0byAweDNmN2FiNmMwIChzaXplIDg0MTI4KQ0KKGQ2
KSBGb3VuZCA3IFBDSSBkZXZpY2VzIChtYXggUENJIGJ1cyBpcyAwMCkNCihkNikgQWxsb2Nh
dGVkIFhlbiBoeXBlcmNhbGwgcGFnZSBhdCAzZjdmZjAwMA0KKGQ2KSBEZXRlY3RlZCBYZW4g
djQuMTUuMA0KKGQ2KSB4ZW46IGNvcHkgQklPUyB0YWJsZXMuLi4NCihkNikgQ29weWluZyBT
TUJJT1MgZW50cnkgcG9pbnQgZnJvbSAweDAwMDEwMDIwIHRvIDB4MDAwZjVlMDANCihkNikg
Q29weWluZyBNUFRBQkxFIGZyb20gMHhmYzEwMDE1MC9mYzEwMDE2MCB0byAweDAwMGY1ZDIw
DQooZDYpIENvcHlpbmcgUElSIGZyb20gMHgwMDAxMDA0MCB0byAweDAwMGY1Y2EwDQooZDYp
IENvcHlpbmcgQUNQSSBSU0RQIGZyb20gMHgwMDAxMDBjMCB0byAweDAwMGY1YzcwDQooZDYp
IHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcwICh2aWEgeHNkdCkNCihkNikgVXNpbmcgcG10
aW1lciwgaW9wb3J0IDB4YjAwOA0KKGQ2KSB0YWJsZSg1MDQzNDE0Nik9MHhmYzAwYTM3MCAo
dmlhIHhzZHQpDQooZDYpIEFDUEk6IHBhcnNlIERTRFQgYXQgMHhmYzAwMTA0MCAobGVuIDM3
NTM5KQ0KKGQ2KSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSAxNi8y
NzY0MQ0KKGQ2KSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSA4Ny82
MDQxDQooZDYpIFNjYW4gZm9yIFZHQSBvcHRpb24gcm9tDQooZDYpIFJ1bm5pbmcgb3B0aW9u
IHJvbSBhdCBjMDAwOjAwMDMNCihkNikgcG1tIGNhbGwgYXJnMT0wDQooZDYpIFR1cm5pbmcg
b24gdmdhIHRleHQgbW9kZSBjb25zb2xlDQooZDYpIFNlYUJJT1MgKHZlcnNpb24gMS4xNC4w
LTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkNikgTWFjaGlu
ZSBVVUlEIDhiMTVjNGFhLWFkNDQtMTFlYy04YzM2LTAwMWU2NzAyMDcwNw0KKGQ2KSBBVEEg
Y29udHJvbGxlciAxIGF0IDFmMC8zZjQvMCAoaXJxIDE0IGRldiA5KQ0KKGQ2KSBBVEEgY29u
dHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1IGRldiA5KQ0KKGQ2KSBBSENJIGNvbnRy
b2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMwMTEwMDAsIGlycSA1DQooZDYpIFNlYXJj
aGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAwL2Rpc2tAMA0KKGQ2
KSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ2KSBTZWFyY2hpbmcg
Ymlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlza0AwDQooZDYp
IFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAxL2Rpc2tA
MA0KKGQ2KSBBSENJLzE6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ2KSBTZWFy
Y2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDEvZGlza0Aw
DQooZDYpIEFIQ0kvMDogcmVnaXN0ZXJpbmc6ICJBSENJLzA6IFFFTVUgSEFSRERJU0sgQVRB
LTcgSGFyZC1EaXNrICgxMDI0MCBNaUJ5dGVzKSINCihYRU4pIGQ1djA6IHVwY2FsbCB2ZWN0
b3IgOTMNCihkNikgUFMyIGtleWJvYXJkIGluaXRpYWxpemVkDQooZDYpIFNlYXJjaGluZyBi
b290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAyL2Rpc2tAMA0KKGQ2KSBBSENJ
LzI6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ2KSBTZWFyY2hpbmcgYmlvcy1n
ZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDIvZGlza0AwDQooZDYpIEFIQ0kv
MTogcmVnaXN0ZXJpbmc6ICJBSENJLzE6IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNr
ICgyMDQ4IE1pQnl0ZXMpIg0KKGQ2KSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogL3BjaUBp
MGNmOC8qQDQvZHJpdmVAMy9kaXNrQDANCihkNikgQUhDSS8zOiBTZXQgdHJhbnNmZXIgbW9k
ZSB0byBVRE1BLTUNCihkNikgU2VhcmNoaW5nIGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkw
Y2Y4LypANC9kcml2ZUAzL2Rpc2tAMA0KKGQ2KSBBSENJLzI6IHJlZ2lzdGVyaW5nOiAiQUhD
SS8yOiBRRU1VIEhBUkRESVNLIEFUQS03IEhhcmQtRGlzayAoMjA0OCBNaUJ5dGVzKSINCihk
NikgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDQvZGlz
a0AwDQooZDYpIEFIQ0kvNDogU2V0IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQooZDYpIFNl
YXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVANC9kaXNr
QDANCihkNikgQUhDSS8zOiByZWdpc3RlcmluZzogIkFIQ0kvMzogUUVNVSBIQVJERElTSyBB
VEEtNyBIYXJkLURpc2sgKDIwNDggTWlCeXRlcykiDQooZDYpIEFIQ0kvNDogcmVnaXN0ZXJp
bmc6ICJBSENJLzQ6IFFFTVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0
ZXMpIg0KKGQ2KSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogSEFMVA0KKGQ2KSBGb3VuZCAw
IGxwdCBwb3J0cw0KKGQ2KSBGb3VuZCAwIHNlcmlhbCBwb3J0cw0KKGQ2KSBBbGwgdGhyZWFk
cyBjb21wbGV0ZS4NCihkNikgU2NhbiBmb3Igb3B0aW9uIHJvbXMNCihkNikgDQooZDYpIFBy
ZXNzIEVTQyBmb3IgYm9vdCBtZW51Lg0KKGQ2KSANCihkNikgU2VhcmNoaW5nIGJvb3RvcmRl
ciBmb3I6IEhBTFQNCihkNikgZHJpdmUgMHgwMDBmNWJjMDogUENIUz0xNjM4My8xNi82MyB0
cmFuc2xhdGlvbj1sYmEgTENIUz0xMDI0LzI1NS82MyBzPTIwOTcxNTIwDQooZDYpIGRyaXZl
IDB4MDAwZjViNzA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUy
MC8xMjgvNjMgcz00MTk0MzA0DQooZDYpIGRyaXZlIDB4MDAwZjViMjA6IFBDSFM9NDE2MS8x
Ni82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDYp
IGRyaXZlIDB4MDAwZjVhZDA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBM
Q0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDYpIGRyaXZlIDB4MDAwZjVhODA6IFBDSFM9
NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0
DQooZDYpIFNwYWNlIGF2YWlsYWJsZSBmb3IgVU1COiBjOTgwMC1lOTAwMCwgZjU2MjAtZjVh
ODANCihkNikgUmV0dXJuZWQgMjQ5ODU2IGJ5dGVzIG9mIFpvbmVIaWdoDQooZDYpIGU4MjAg
bWFwIGhhcyA3IGl0ZW1zOg0KKGQ2KSAgIDA6IDAwMDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAw
MDAwMDlmYzAwID0gMSBSQU0NCihkNikgICAxOiAwMDAwMDAwMDAwMDlmYzAwIC0gMDAwMDAw
MDAwMDBhMDAwMCA9IDIgUkVTRVJWRUQNCihkNikgICAyOiAwMDAwMDAwMDAwMGYwMDAwIC0g
MDAwMDAwMDAwMDEwMDAwMCA9IDIgUkVTRVJWRUQNCihkNikgICAzOiAwMDAwMDAwMDAwMTAw
MDAwIC0gMDAwMDAwMDAzZjdmZDAwMCA9IDEgUkFNDQooZDYpICAgNDogMDAwMDAwMDAzZjdm
ZDAwMCAtIDAwMDAwMDAwM2Y4MDAwMDAgPSAyIFJFU0VSVkVEDQooZDYpICAgNTogMDAwMDAw
MDBmYzAwMDAwMCAtIDAwMDAwMDAwZmMwMGIwMDAgPSA0IE5WUw0KKGQ2KSAgIDY6IDAwMDAw
MDAwZmMwMGIwMDAgLSAwMDAwMDAwMTAwMDAwMDAwID0gMiBSRVNFUlZFRA0KKGQ2KSBlbnRl
ciBoYW5kbGVfMTk6DQooZDYpICAgTlVMTA0KKGQ2KSBCb290aW5nIGZyb20gSGFyZCBEaXNr
Li4uDQooZDYpIEJvb3RpbmcgZnJvbSAwMDAwOjdjMDANCihYRU4pIGQ2djA6IHVwY2FsbCB2
ZWN0b3IgOTMNCnhiYmQzOiBFcnJvciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBi
b3VuY2UgYnVmZmVycw0KeGJiZDM6IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENs
b3NpbmcgU3RhdGUNCnhiYmQ0OiBFcnJvciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVz
dCBib3VuY2UgYnVmZmVycw0KeGJiZDQ6IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRv
IENsb3NpbmcgU3RhdGUNCnhiYmQ1OiBFcnJvciAxMiBVbmFibGUgdG8gYWxsb2NhdGUgcmVx
dWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDU6IEZhdGFsIGVycm9yLiBUcmFuc2l0aW9uaW5n
IHRvIENsb3NpbmcgU3RhdGUNCihYRU4pIEhWTSBkN3YwIHNhdmU6IENQVQ0KKFhFTikgSFZN
IGQ3IHNhdmU6IFBJQw0KKFhFTikgSFZNIGQ3IHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQ3
djAgc2F2ZTogTEFQSUMNCihYRU4pIEhWTSBkN3YwIHNhdmU6IExBUElDX1JFR1MNCihYRU4p
IEhWTSBkNyBzYXZlOiBQQ0lfSVJRDQooWEVOKSBIVk0gZDcgc2F2ZTogSVNBX0lSUQ0KKFhF
TikgSFZNIGQ3IHNhdmU6IFBDSV9MSU5LDQooWEVOKSBIVk0gZDcgc2F2ZTogUElUDQooWEVO
KSBIVk0gZDcgc2F2ZTogUlRDDQooWEVOKSBIVk0gZDcgc2F2ZTogSFBFVA0KKFhFTikgSFZN
IGQ3IHNhdmU6IFBNVElNRVINCihYRU4pIEhWTSBkN3YwIHNhdmU6IE1UUlINCihYRU4pIEhW
TSBkNyBzYXZlOiBWSVJJRElBTl9ET01BSU4NCihYRU4pIEhWTSBkN3YwIHNhdmU6IENQVV9Y
U0FWRQ0KKFhFTikgSFZNIGQ3djAgc2F2ZTogVklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQ3
djAgc2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBIVk0gZDd2MCBzYXZlOiBUU0NfQURKVVNUDQoo
WEVOKSBIVk0gZDd2MCBzYXZlOiBDUFVfTVNSDQooWEVOKSBIVk03IHJlc3RvcmU6IENQVSAw
DQooZDcpIEhWTSBMb2FkZXINCihkNykgRGV0ZWN0ZWQgWGVuIHY0LjE1LjANCihkNykgWGVu
YnVzIHJpbmdzIEAweGZlZmZjMDAwLCBldmVudCBjaGFubmVsIDENCihkNykgU3lzdGVtIHJl
cXVlc3RlZCBTZWFCSU9TDQooZDcpIENQVSBzcGVlZCBpcyAyNTMzIE1Ieg0KKGQ3KSBSZWxv
Y2F0aW5nIGd1ZXN0IG1lbW9yeSBmb3IgbG93bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihk
NykgUENJLUlTQSBsaW5rIDAgcm91dGVkIHRvIElSUTUNCihkNykgUENJLUlTQSBsaW5rIDEg
cm91dGVkIHRvIElSUTEwDQooZDcpIFBDSS1JU0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0K
KGQ3KSBQQ0ktSVNBIGxpbmsgMyByb3V0ZWQgdG8gSVJRNQ0KKGQ3KSBwY2kgZGV2IDAxOjMg
SU5UQS0+SVJRMTANCihkNykgcGNpIGRldiAwMjowIElOVEEtPklSUTExDQooZDcpIHBjaSBk
ZXYgMDQ6MCBJTlRBLT5JUlE1DQooZDcpIE5vIFJBTSBpbiBoaWdoIG1lbW9yeTsgc2V0dGlu
ZyBoaWdoX21lbSByZXNvdXJjZSBiYXNlIHRvIDEwMDAwMDAwMA0KKGQ3KSBwY2kgZGV2IDAz
OjAgYmFyIDEwIHNpemUgMDAyMDAwMDAwOiAwZjAwMDAwMDgNCihkNykgcGNpIGRldiAwMjow
IGJhciAxNCBzaXplIDAwMTAwMDAwMDogMGYyMDAwMDA4DQooZDcpIHBjaSBkZXYgMDM6MCBi
YXIgMzAgc2l6ZSAwMDAwMTAwMDA6IDBmMzAwMDAwMA0KKGQ3KSBwY2kgZGV2IDAzOjAgYmFy
IDE0IHNpemUgMDAwMDAxMDAwOiAwZjMwMTAwMDANCihkNykgcGNpIGRldiAwNDowIGJhciAy
NCBzaXplIDAwMDAwMTAwMDogMGYzMDExMDAwDQooZDcpIHBjaSBkZXYgMDI6MCBiYXIgMTAg
c2l6ZSAwMDAwMDAxMDA6IDAwMDAwYzAwMQ0KKGQ3KSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNp
emUgMDAwMDAwMDIwOiAwMDAwMGMxMDENCihkNykgcGNpIGRldiAwMToxIGJhciAyMCBzaXpl
IDAwMDAwMDAxMDogMDAwMDBjMTIxDQooZDcpIE11bHRpcHJvY2Vzc29yIGluaXRpYWxpc2F0
aW9uOg0KKGQ3KSAgLSBDUFUwIC4uLiAzNi1iaXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4u
IHZhciBNVFJScyBbMS84XSAuLi4gZG9uZS4NCihkNykgV3JpdGluZyBTTUJJT1MgdGFibGVz
IC4uLg0KKGQ3KSBMb2FkaW5nIFNlYUJJT1MgLi4uDQooZDcpIENyZWF0aW5nIE1QIHRhYmxl
cyAuLi4NCihkNykgTG9hZGluZyBBQ1BJIC4uLg0KKGQ3KSB2bTg2IFRTUyBhdCBmYzEwMDI4
MA0KKGQ3KSBCSU9TIG1hcDoNCihkNykgIDEwMDAwLTEwMGUzOiBTY3JhdGNoIHNwYWNlDQoo
ZDcpICBjMDAwMC1mZmZmZjogTWFpbiBCSU9TDQooZDcpIEU4MjAgdGFibGU6DQooZDcpICBb
MDBdOiAwMDAwMDAwMDowMDAwMDAwMCAtIDAwMDAwMDAwOjAwMGEwMDAwOiBSQU0NCihkNykg
IEhPTEU6IDAwMDAwMDAwOjAwMGEwMDAwIC0gMDAwMDAwMDA6MDAwYzAwMDANCihkNykgIFsw
MV06IDAwMDAwMDAwOjAwMGMwMDAwIC0gMDAwMDAwMDA6MDAxMDAwMDA6IFJFU0VSVkVEDQoo
ZDcpICBbMDJdOiAwMDAwMDAwMDowMDEwMDAwMCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0N
CihkNykgIEhPTEU6IDAwMDAwMDAwOjNmODAwMDAwIC0gMDAwMDAwMDA6ZmMwMDAwMDANCihk
NykgIFswM106IDAwMDAwMDAwOmZjMDAwMDAwIC0gMDAwMDAwMDA6ZmMwMGIwMDA6IE5WUw0K
KGQ3KSAgWzA0XTogMDAwMDAwMDA6ZmMwMGIwMDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVT
RVJWRUQNCihkNykgSW52b2tpbmcgU2VhQklPUyAuLi4NCihkNykgU2VhQklPUyAodmVyc2lv
biAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQ3
KSBCVUlMRDogZ2NjOiBtaXhlZCBiaW51dGlsczogKEdOVSBCaW51dGlscykgMi4zNw0KKGQ3
KSANCihkNykgRm91bmQgWGVuIGh5cGVydmlzb3Igc2lnbmF0dXJlIGF0IDQwMDAwMDAwDQoo
ZDcpIFJ1bm5pbmcgb24gUUVNVSAoaTQ0MGZ4KQ0KKGQ3KSB4ZW46IGNvcHkgZTgyMC4uLg0K
KGQ3KSBSZWxvY2F0aW5nIGluaXQgZnJvbSAweDAwMGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNp
emUgODQxMjgpDQooZDcpIEZvdW5kIDcgUENJIGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAw
KQ0KKGQ3KSBBbGxvY2F0ZWQgWGVuIGh5cGVyY2FsbCBwYWdlIGF0IDNmN2ZmMDAwDQooZDcp
IERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDcpIHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0K
KGQ3KSBDb3B5aW5nIFNNQklPUyBlbnRyeSBwb2ludCBmcm9tIDB4MDAwMTAwMjAgdG8gMHgw
MDBmNWUwMA0KKGQ3KSBDb3B5aW5nIE1QVEFCTEUgZnJvbSAweGZjMTAwMTUwL2ZjMTAwMTYw
IHRvIDB4MDAwZjVkMjANCihkNykgQ29weWluZyBQSVIgZnJvbSAweDAwMDEwMDQwIHRvIDB4
MDAwZjVjYTANCihkNykgQ29weWluZyBBQ1BJIFJTRFAgZnJvbSAweDAwMDEwMGMwIHRvIDB4
MDAwZjVjNzANCihkNykgdGFibGUoNTA0MzQxNDYpPTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0K
KGQ3KSBVc2luZyBwbXRpbWVyLCBpb3BvcnQgMHhiMDA4DQooZDcpIHRhYmxlKDUwNDM0MTQ2
KT0weGZjMDBhMzcwICh2aWEgeHNkdCkNCihkNykgQUNQSTogcGFyc2UgRFNEVCBhdCAweGZj
MDAxMDQwIChsZW4gMzc1MzkpDQooZDcpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwg
c2tpcCBmcm9tIDE2LzI3NjQxDQooZDcpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwg
c2tpcCBmcm9tIDg3LzYwNDENCihkNykgU2NhbiBmb3IgVkdBIG9wdGlvbiByb20NCihkNykg
UnVubmluZyBvcHRpb24gcm9tIGF0IGMwMDA6MDAwMw0KKGQ3KSBwbW0gY2FsbCBhcmcxPTAN
CihkNykgVHVybmluZyBvbiB2Z2EgdGV4dCBtb2RlIGNvbnNvbGUNCihkNykgU2VhQklPUyAo
dmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0
KQ0KKGQ3KSBNYWNoaW5lIFVVSUQgNDQzOTY2NmMtYWQ0OS0xMWVjLThjMzYtMDAxZTY3MDIw
NzA3DQooZDcpIEFUQSBjb250cm9sbGVyIDEgYXQgMWYwLzNmNC8wIChpcnEgMTQgZGV2IDkp
DQooZDcpIEFUQSBjb250cm9sbGVyIDIgYXQgMTcwLzM3NC8wIChpcnEgMTUgZGV2IDkpDQoo
ZDcpIEFIQ0kgY29udHJvbGxlciBhdCAwMDowNC4wLCBpb2Jhc2UgMHhmMzAxMTAwMCwgaXJx
IDUNCihkNykgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZl
QDAvZGlza0AwDQooZDcpIEFIQ0kvMDogU2V0IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQoo
ZDcpIFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVA
MC9kaXNrQDANCihkNykgQUhDSS8wOiByZWdpc3RlcmluZzogIkFIQ0kvMDogUUVNVSBIQVJE
RElTSyBBVEEtNyBIYXJkLURpc2sgKDEwMjQwIE1pQnl0ZXMpIg0KKGQ3KSBQUzIga2V5Ym9h
cmQgaW5pdGlhbGl6ZWQNCihkNykgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihk
NykgRm91bmQgMCBscHQgcG9ydHMNCihkNykgRm91bmQgMCBzZXJpYWwgcG9ydHMNCihkNykg
QWxsIHRocmVhZHMgY29tcGxldGUuDQooZDcpIFNjYW4gZm9yIG9wdGlvbiByb21zDQooZDcp
IA0KKGQ3KSBQcmVzcyBFU0MgZm9yIGJvb3QgbWVudS4NCihkNykgDQooZDcpIFNlYXJjaGlu
ZyBib290b3JkZXIgZm9yOiBIQUxUDQooZDcpIGRyaXZlIDB4MDAwZjViYzA6IFBDSFM9MTYz
ODMvMTYvNjMgdHJhbnNsYXRpb249bGJhIExDSFM9MTAyNC8yNTUvNjMgcz0yMDk3MTUyMA0K
KGQ3KSBTcGFjZSBhdmFpbGFibGUgZm9yIFVNQjogYzk4MDAtZTkwMDAsIGY1NjIwLWY1YmMw
DQooZDcpIFJldHVybmVkIDI1Mzk1MiBieXRlcyBvZiBab25lSGlnaA0KKGQ3KSBlODIwIG1h
cCBoYXMgNyBpdGVtczoNCihkNykgICAwOiAwMDAwMDAwMDAwMDAwMDAwIC0gMDAwMDAwMDAw
MDA5ZmMwMCA9IDEgUkFNDQooZDcpICAgMTogMDAwMDAwMDAwMDA5ZmMwMCAtIDAwMDAwMDAw
MDAwYTAwMDAgPSAyIFJFU0VSVkVEDQooZDcpICAgMjogMDAwMDAwMDAwMDBmMDAwMCAtIDAw
MDAwMDAwMDAxMDAwMDAgPSAyIFJFU0VSVkVEDQooZDcpICAgMzogMDAwMDAwMDAwMDEwMDAw
MCAtIDAwMDAwMDAwM2Y3ZmUwMDAgPSAxIFJBTQ0KKGQ3KSAgIDQ6IDAwMDAwMDAwM2Y3ZmUw
MDAgLSAwMDAwMDAwMDNmODAwMDAwID0gMiBSRVNFUlZFRA0KKGQ3KSAgIDU6IDAwMDAwMDAw
ZmMwMDAwMDAgLSAwMDAwMDAwMGZjMDBiMDAwID0gNCBOVlMNCihkNykgICA2OiAwMDAwMDAw
MGZjMDBiMDAwIC0gMDAwMDAwMDEwMDAwMDAwMCA9IDIgUkVTRVJWRUQNCihkNykgZW50ZXIg
aGFuZGxlXzE5Og0KKGQ3KSAgIE5VTEwNCihkNykgQm9vdGluZyBmcm9tIEhhcmQgRGlzay4u
Lg0KKGQ3KSBCb290aW5nIGZyb20gMDAwMDo3YzAwDQooWEVOKSBkN3YwOiB1cGNhbGwgdmVj
dG9yIDkzDQooWEVOKSBIVk0gZDh2MCBzYXZlOiBDUFUNCihYRU4pIEhWTSBkOCBzYXZlOiBQ
SUMNCihYRU4pIEhWTSBkOCBzYXZlOiBJT0FQSUMNCihYRU4pIEhWTSBkOHYwIHNhdmU6IExB
UElDDQooWEVOKSBIVk0gZDh2MCBzYXZlOiBMQVBJQ19SRUdTDQooWEVOKSBIVk0gZDggc2F2
ZTogUENJX0lSUQ0KKFhFTikgSFZNIGQ4IHNhdmU6IElTQV9JUlENCihYRU4pIEhWTSBkOCBz
YXZlOiBQQ0lfTElOSw0KKFhFTikgSFZNIGQ4IHNhdmU6IFBJVA0KKFhFTikgSFZNIGQ4IHNh
dmU6IFJUQw0KKFhFTikgSFZNIGQ4IHNhdmU6IEhQRVQNCihYRU4pIEhWTSBkOCBzYXZlOiBQ
TVRJTUVSDQooWEVOKSBIVk0gZDh2MCBzYXZlOiBNVFJSDQooWEVOKSBIVk0gZDggc2F2ZTog
VklSSURJQU5fRE9NQUlODQooWEVOKSBIVk0gZDh2MCBzYXZlOiBDUFVfWFNBVkUNCihYRU4p
IEhWTSBkOHYwIHNhdmU6IFZJUklESUFOX1ZDUFUNCihYRU4pIEhWTSBkOHYwIHNhdmU6IFZN
Q0VfVkNQVQ0KKFhFTikgSFZNIGQ4djAgc2F2ZTogVFNDX0FESlVTVA0KKFhFTikgSFZNIGQ4
djAgc2F2ZTogQ1BVX01TUg0KKFhFTikgSFZNOCByZXN0b3JlOiBDUFUgMA0KKGQ4KSBIVk0g
TG9hZGVyDQooZDgpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDgpIFhlbmJ1cyByaW5ncyBA
MHhmZWZmYzAwMCwgZXZlbnQgY2hhbm5lbCAxDQooZDgpIFN5c3RlbSByZXF1ZXN0ZWQgU2Vh
QklPUw0KKGQ4KSBDUFUgc3BlZWQgaXMgMjUzMyBNSHoNCihkOCkgUmVsb2NhdGluZyBndWVz
dCBtZW1vcnkgZm9yIGxvd21lbSBNTUlPIHNwYWNlIGRpc2FibGVkDQooZDgpIFBDSS1JU0Eg
bGluayAwIHJvdXRlZCB0byBJUlE1DQooZDgpIFBDSS1JU0EgbGluayAxIHJvdXRlZCB0byBJ
UlExMA0KKGQ4KSBQQ0ktSVNBIGxpbmsgMiByb3V0ZWQgdG8gSVJRMTENCihkOCkgUENJLUlT
QSBsaW5rIDMgcm91dGVkIHRvIElSUTUNCihkOCkgcGNpIGRldiAwMTozIElOVEEtPklSUTEw
DQooZDgpIHBjaSBkZXYgMDI6MCBJTlRBLT5JUlExMQ0KKGQ4KSBwY2kgZGV2IDA0OjAgSU5U
QS0+SVJRNQ0KKGQ4KSBObyBSQU0gaW4gaGlnaCBtZW1vcnk7IHNldHRpbmcgaGlnaF9tZW0g
cmVzb3VyY2UgYmFzZSB0byAxMDAwMDAwMDANCihkOCkgcGNpIGRldiAwMzowIGJhciAxMCBz
aXplIDAwMjAwMDAwMDogMGYwMDAwMDA4DQooZDgpIHBjaSBkZXYgMDI6MCBiYXIgMTQgc2l6
ZSAwMDEwMDAwMDA6IDBmMjAwMDAwOA0KKGQ4KSBwY2kgZGV2IDAzOjAgYmFyIDMwIHNpemUg
MDAwMDEwMDAwOiAwZjMwMDAwMDANCihkOCkgcGNpIGRldiAwMzowIGJhciAxNCBzaXplIDAw
MDAwMTAwMDogMGYzMDEwMDAwDQooZDgpIHBjaSBkZXYgMDQ6MCBiYXIgMjQgc2l6ZSAwMDAw
MDEwMDA6IDBmMzAxMTAwMA0KKGQ4KSBwY2kgZGV2IDAyOjAgYmFyIDEwIHNpemUgMDAwMDAw
MTAwOiAwMDAwMGMwMDENCihkOCkgcGNpIGRldiAwNDowIGJhciAyMCBzaXplIDAwMDAwMDAy
MDogMDAwMDBjMTAxDQooZDgpIHBjaSBkZXYgMDE6MSBiYXIgMjAgc2l6ZSAwMDAwMDAwMTA6
IDAwMDAwYzEyMQ0KKGQ4KSBNdWx0aXByb2Nlc3NvciBpbml0aWFsaXNhdGlvbjoNCihkOCkg
IC0gQ1BVMCAuLi4gMzYtYml0IHBoeXMgLi4uIGZpeGVkIE1UUlJzIC4uLiB2YXIgTVRSUnMg
WzEvOF0gLi4uIGRvbmUuDQooZDgpIFdyaXRpbmcgU01CSU9TIHRhYmxlcyAuLi4NCihkOCkg
TG9hZGluZyBTZWFCSU9TIC4uLg0KKGQ4KSBDcmVhdGluZyBNUCB0YWJsZXMgLi4uDQooZDgp
IExvYWRpbmcgQUNQSSAuLi4NCihkOCkgdm04NiBUU1MgYXQgZmMxMDAyODANCihkOCkgQklP
UyBtYXA6DQooZDgpICAxMDAwMC0xMDBlMzogU2NyYXRjaCBzcGFjZQ0KKGQ4KSAgYzAwMDAt
ZmZmZmY6IE1haW4gQklPUw0KKGQ4KSBFODIwIHRhYmxlOg0KKGQ4KSAgWzAwXTogMDAwMDAw
MDA6MDAwMDAwMDAgLSAwMDAwMDAwMDowMDBhMDAwMDogUkFNDQooZDgpICBIT0xFOiAwMDAw
MDAwMDowMDBhMDAwMCAtIDAwMDAwMDAwOjAwMGMwMDAwDQooZDgpICBbMDFdOiAwMDAwMDAw
MDowMDBjMDAwMCAtIDAwMDAwMDAwOjAwMTAwMDAwOiBSRVNFUlZFRA0KKGQ4KSAgWzAyXTog
MDAwMDAwMDA6MDAxMDAwMDAgLSAwMDAwMDAwMDozZjgwMDAwMDogUkFNDQooZDgpICBIT0xF
OiAwMDAwMDAwMDozZjgwMDAwMCAtIDAwMDAwMDAwOmZjMDAwMDAwDQooZDgpICBbMDNdOiAw
MDAwMDAwMDpmYzAwMDAwMCAtIDAwMDAwMDAwOmZjMDBiMDAwOiBOVlMNCihkOCkgIFswNF06
IDAwMDAwMDAwOmZjMDBiMDAwIC0gMDAwMDAwMDE6MDAwMDAwMDA6IFJFU0VSVkVEDQooZDgp
IEludm9raW5nIFNlYUJJT1MgLi4uDQooZDgpIFNlYUJJT1MgKHZlcnNpb24gMS4xNC4wLTIw
MjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkOCkgQlVJTEQ6IGdj
YzogbWl4ZWQgYmludXRpbHM6IChHTlUgQmludXRpbHMpIDIuMzcNCihkOCkgDQooZDgpIEZv
dW5kIFhlbiBoeXBlcnZpc29yIHNpZ25hdHVyZSBhdCA0MDAwMDAwMA0KKGQ4KSBSdW5uaW5n
IG9uIFFFTVUgKGk0NDBmeCkNCihkOCkgeGVuOiBjb3B5IGU4MjAuLi4NCihkOCkgUmVsb2Nh
dGluZyBpbml0IGZyb20gMHgwMDBkNjIwMCB0byAweDNmN2FiNmMwIChzaXplIDg0MTI4KQ0K
KGQ4KSBGb3VuZCA3IFBDSSBkZXZpY2VzIChtYXggUENJIGJ1cyBpcyAwMCkNCihkOCkgQWxs
b2NhdGVkIFhlbiBoeXBlcmNhbGwgcGFnZSBhdCAzZjdmZjAwMA0KKGQ4KSBEZXRlY3RlZCBY
ZW4gdjQuMTUuMA0KKGQ4KSB4ZW46IGNvcHkgQklPUyB0YWJsZXMuLi4NCihkOCkgQ29weWlu
ZyBTTUJJT1MgZW50cnkgcG9pbnQgZnJvbSAweDAwMDEwMDIwIHRvIDB4MDAwZjVlMDANCihk
OCkgQ29weWluZyBNUFRBQkxFIGZyb20gMHhmYzEwMDE1MC9mYzEwMDE2MCB0byAweDAwMGY1
ZDIwDQooZDgpIENvcHlpbmcgUElSIGZyb20gMHgwMDAxMDA0MCB0byAweDAwMGY1Y2EwDQoo
ZDgpIENvcHlpbmcgQUNQSSBSU0RQIGZyb20gMHgwMDAxMDBjMCB0byAweDAwMGY1YzcwDQoo
ZDgpIHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcwICh2aWEgeHNkdCkNCihkOCkgVXNpbmcg
cG10aW1lciwgaW9wb3J0IDB4YjAwOA0KKGQ4KSB0YWJsZSg1MDQzNDE0Nik9MHhmYzAwYTM3
MCAodmlhIHhzZHQpDQooZDgpIEFDUEk6IHBhcnNlIERTRFQgYXQgMHhmYzAwMTA0MCAobGVu
IDM3NTM5KQ0KKGQ4KSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSAx
Ni8yNzY0MQ0KKGQ4KSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSA4
Ny82MDQxDQooZDgpIFNjYW4gZm9yIFZHQSBvcHRpb24gcm9tDQooZDgpIFJ1bm5pbmcgb3B0
aW9uIHJvbSBhdCBjMDAwOjAwMDMNCihkOCkgcG1tIGNhbGwgYXJnMT0wDQooZDgpIFR1cm5p
bmcgb24gdmdhIHRleHQgbW9kZSBjb25zb2xlDQooZDgpIFNlYUJJT1MgKHZlcnNpb24gMS4x
NC4wLTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkOCkgTWFj
aGluZSBVVUlEIDdiYzk5N2QyLWFkNDktMTFlYy04YzM2LTAwMWU2NzAyMDcwNw0KKGQ4KSBB
VEEgY29udHJvbGxlciAxIGF0IDFmMC8zZjQvMCAoaXJxIDE0IGRldiA5KQ0KKGQ4KSBBVEEg
Y29udHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1IGRldiA5KQ0KKGQ4KSBBSENJIGNv
bnRyb2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMwMTEwMDAsIGlycSA1DQooZDgpIFNl
YXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAwL2Rpc2tAMA0K
KGQ4KSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ4KSBTZWFyY2hp
bmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlza0AwDQoo
ZDgpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAxL2Rp
c2tAMA0KKGQ4KSBBSENJLzE6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQ4KSBT
ZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDEvZGlz
a0AwDQooZDgpIEFIQ0kvMDogcmVnaXN0ZXJpbmc6ICJBSENJLzA6IFFFTVUgSEFSRERJU0sg
QVRBLTcgSGFyZC1EaXNrICgxMDI0MCBNaUJ5dGVzKSINCihkOCkgUFMyIGtleWJvYXJkIGlu
aXRpYWxpemVkDQooZDgpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypA
NC9kcml2ZUAyL2Rpc2tAMA0KKGQ4KSBBSENJLzI6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVE
TUEtNQ0KKGQ4KSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgvKkA0
L2RyaXZlQDIvZGlza0AwDQooZDgpIEFIQ0kvMTogcmVnaXN0ZXJpbmc6ICJBSENJLzE6IFFF
TVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0KKGQ4KSBTZWFy
Y2hpbmcgYm9vdG9yZGVyIGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVAMy9kaXNrQDANCihk
OCkgQUhDSS8zOiBTZXQgdHJhbnNmZXIgbW9kZSB0byBVRE1BLTUNCihkOCkgU2VhcmNoaW5n
IGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAzL2Rpc2tAMA0KKGQ4
KSBBSENJLzI6IHJlZ2lzdGVyaW5nOiAiQUhDSS8yOiBRRU1VIEhBUkRESVNLIEFUQS03IEhh
cmQtRGlzayAoMjA0OCBNaUJ5dGVzKSINCihkOCkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6
IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDQvZGlza0AwDQooZDgpIEFIQ0kvNDogU2V0IHRyYW5z
ZmVyIG1vZGUgdG8gVURNQS01DQooZDgpIFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjog
L3BjaUBpMGNmOC8qQDQvZHJpdmVANC9kaXNrQDANCihkOCkgQUhDSS8zOiByZWdpc3Rlcmlu
ZzogIkFIQ0kvMzogUUVNVSBIQVJERElTSyBBVEEtNyBIYXJkLURpc2sgKDIwNDggTWlCeXRl
cykiDQooZDgpIEFIQ0kvNDogcmVnaXN0ZXJpbmc6ICJBSENJLzQ6IFFFTVUgSEFSRERJU0sg
QVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0KKGQ4KSBTZWFyY2hpbmcgYm9vdG9y
ZGVyIGZvcjogSEFMVA0KKGQ4KSBGb3VuZCAwIGxwdCBwb3J0cw0KKGQ4KSBGb3VuZCAwIHNl
cmlhbCBwb3J0cw0KKGQ4KSBBbGwgdGhyZWFkcyBjb21wbGV0ZS4NCihkOCkgU2NhbiBmb3Ig
b3B0aW9uIHJvbXMNCihkOCkgDQooZDgpIFByZXNzIEVTQyBmb3IgYm9vdCBtZW51Lg0KKGQ4
KSANCihkOCkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihkOCkgZHJpdmUgMHgw
MDBmNWJjMDogUENIUz0xNjM4My8xNi82MyB0cmFuc2xhdGlvbj1sYmEgTENIUz0xMDI0LzI1
NS82MyBzPTIwOTcxNTIwDQooZDgpIGRyaXZlIDB4MDAwZjViNzA6IFBDSFM9NDE2MS8xNi82
MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDgpIGRy
aXZlIDB4MDAwZjViMjA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hT
PTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDgpIGRyaXZlIDB4MDAwZjVhZDA6IFBDSFM9NDE2
MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQoo
ZDgpIGRyaXZlIDB4MDAwZjVhODA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJn
ZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDgpIFNwYWNlIGF2YWlsYWJsZSBmb3Ig
VU1COiBjOTgwMC1lOTAwMCwgZjU2MjAtZjVhODANCihkOCkgUmV0dXJuZWQgMjQ5ODU2IGJ5
dGVzIG9mIFpvbmVIaWdoDQooZDgpIGU4MjAgbWFwIGhhcyA3IGl0ZW1zOg0KKGQ4KSAgIDA6
IDAwMDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAwMDAwMDlmYzAwID0gMSBSQU0NCihkOCkgICAx
OiAwMDAwMDAwMDAwMDlmYzAwIC0gMDAwMDAwMDAwMDBhMDAwMCA9IDIgUkVTRVJWRUQNCihk
OCkgICAyOiAwMDAwMDAwMDAwMGYwMDAwIC0gMDAwMDAwMDAwMDEwMDAwMCA9IDIgUkVTRVJW
RUQNCihkOCkgICAzOiAwMDAwMDAwMDAwMTAwMDAwIC0gMDAwMDAwMDAzZjdmZDAwMCA9IDEg
UkFNDQooZDgpICAgNDogMDAwMDAwMDAzZjdmZDAwMCAtIDAwMDAwMDAwM2Y4MDAwMDAgPSAy
IFJFU0VSVkVEDQooZDgpICAgNTogMDAwMDAwMDBmYzAwMDAwMCAtIDAwMDAwMDAwZmMwMGIw
MDAgPSA0IE5WUw0KKGQ4KSAgIDY6IDAwMDAwMDAwZmMwMGIwMDAgLSAwMDAwMDAwMTAwMDAw
MDAwID0gMiBSRVNFUlZFRA0KKGQ4KSBlbnRlciBoYW5kbGVfMTk6DQooZDgpICAgTlVMTA0K
KGQ4KSBCb290aW5nIGZyb20gSGFyZCBEaXNrLi4uDQooZDgpIEJvb3RpbmcgZnJvbSAwMDAw
OjdjMDANCihYRU4pIEhWTSBkOXYwIHNhdmU6IENQVQ0KKFhFTikgSFZNIGQ5IHNhdmU6IFBJ
Qw0KKFhFTikgSFZNIGQ5IHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQ5djAgc2F2ZTogTEFQ
SUMNCihYRU4pIEhWTSBkOXYwIHNhdmU6IExBUElDX1JFR1MNCihYRU4pIEhWTSBkOSBzYXZl
OiBQQ0lfSVJRDQooWEVOKSBIVk0gZDkgc2F2ZTogSVNBX0lSUQ0KKFhFTikgSFZNIGQ5IHNh
dmU6IFBDSV9MSU5LDQooWEVOKSBIVk0gZDkgc2F2ZTogUElUDQooWEVOKSBIVk0gZDkgc2F2
ZTogUlRDDQooWEVOKSBIVk0gZDkgc2F2ZTogSFBFVA0KKFhFTikgSFZNIGQ5IHNhdmU6IFBN
VElNRVINCihYRU4pIEhWTSBkOXYwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBkOSBzYXZlOiBW
SVJJRElBTl9ET01BSU4NCihYRU4pIEhWTSBkOXYwIHNhdmU6IENQVV9YU0FWRQ0KKFhFTikg
SFZNIGQ5djAgc2F2ZTogVklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQ5djAgc2F2ZTogVk1D
RV9WQ1BVDQooWEVOKSBIVk0gZDl2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVOKSBIVk0gZDl2
MCBzYXZlOiBDUFVfTVNSDQooWEVOKSBIVk05IHJlc3RvcmU6IENQVSAwDQooZDkpIEhWTSBM
b2FkZXINCihkOSkgRGV0ZWN0ZWQgWGVuIHY0LjE1LjANCihkOSkgWGVuYnVzIHJpbmdzIEAw
eGZlZmZjMDAwLCBldmVudCBjaGFubmVsIDENCihkOSkgU3lzdGVtIHJlcXVlc3RlZCBTZWFC
SU9TDQooZDkpIENQVSBzcGVlZCBpcyAyNTMzIE1Ieg0KKGQ5KSBSZWxvY2F0aW5nIGd1ZXN0
IG1lbW9yeSBmb3IgbG93bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihkOSkgUENJLUlTQSBs
aW5rIDAgcm91dGVkIHRvIElSUTUNCihkOSkgUENJLUlTQSBsaW5rIDEgcm91dGVkIHRvIElS
UTEwDQooZDkpIFBDSS1JU0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0KKGQ5KSBQQ0ktSVNB
IGxpbmsgMyByb3V0ZWQgdG8gSVJRNQ0KKGQ5KSBwY2kgZGV2IDAxOjMgSU5UQS0+SVJRMTAN
CihkOSkgcGNpIGRldiAwMjowIElOVEEtPklSUTExDQooZDkpIHBjaSBkZXYgMDQ6MCBJTlRB
LT5JUlE1DQooZDkpIE5vIFJBTSBpbiBoaWdoIG1lbW9yeTsgc2V0dGluZyBoaWdoX21lbSBy
ZXNvdXJjZSBiYXNlIHRvIDEwMDAwMDAwMA0KKGQ5KSBwY2kgZGV2IDAzOjAgYmFyIDEwIHNp
emUgMDAyMDAwMDAwOiAwZjAwMDAwMDgNCihkOSkgcGNpIGRldiAwMjowIGJhciAxNCBzaXpl
IDAwMTAwMDAwMDogMGYyMDAwMDA4DQooZDkpIHBjaSBkZXYgMDM6MCBiYXIgMzAgc2l6ZSAw
MDAwMTAwMDA6IDBmMzAwMDAwMA0KKGQ5KSBwY2kgZGV2IDAzOjAgYmFyIDE0IHNpemUgMDAw
MDAxMDAwOiAwZjMwMTAwMDANCihkOSkgcGNpIGRldiAwNDowIGJhciAyNCBzaXplIDAwMDAw
MTAwMDogMGYzMDExMDAwDQooZDkpIHBjaSBkZXYgMDI6MCBiYXIgMTAgc2l6ZSAwMDAwMDAx
MDA6IDAwMDAwYzAwMQ0KKGQ5KSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNpemUgMDAwMDAwMDIw
OiAwMDAwMGMxMDENCihkOSkgcGNpIGRldiAwMToxIGJhciAyMCBzaXplIDAwMDAwMDAxMDog
MDAwMDBjMTIxDQooZDkpIE11bHRpcHJvY2Vzc29yIGluaXRpYWxpc2F0aW9uOg0KKGQ5KSAg
LSBDUFUwIC4uLiAzNi1iaXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4uIHZhciBNVFJScyBb
MS84XSAuLi4gZG9uZS4NCihkOSkgV3JpdGluZyBTTUJJT1MgdGFibGVzIC4uLg0KKGQ5KSBM
b2FkaW5nIFNlYUJJT1MgLi4uDQooZDkpIENyZWF0aW5nIE1QIHRhYmxlcyAuLi4NCihkOSkg
TG9hZGluZyBBQ1BJIC4uLg0KKGQ5KSB2bTg2IFRTUyBhdCBmYzEwMDI4MA0KKGQ5KSBCSU9T
IG1hcDoNCihkOSkgIDEwMDAwLTEwMGUzOiBTY3JhdGNoIHNwYWNlDQooZDkpICBjMDAwMC1m
ZmZmZjogTWFpbiBCSU9TDQooZDkpIEU4MjAgdGFibGU6DQooZDkpICBbMDBdOiAwMDAwMDAw
MDowMDAwMDAwMCAtIDAwMDAwMDAwOjAwMGEwMDAwOiBSQU0NCihkOSkgIEhPTEU6IDAwMDAw
MDAwOjAwMGEwMDAwIC0gMDAwMDAwMDA6MDAwYzAwMDANCihkOSkgIFswMV06IDAwMDAwMDAw
OjAwMGMwMDAwIC0gMDAwMDAwMDA6MDAxMDAwMDA6IFJFU0VSVkVEDQooZDkpICBbMDJdOiAw
MDAwMDAwMDowMDEwMDAwMCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0NCihkOSkgIEhPTEU6
IDAwMDAwMDAwOjNmODAwMDAwIC0gMDAwMDAwMDA6ZmMwMDAwMDANCihkOSkgIFswM106IDAw
MDAwMDAwOmZjMDAwMDAwIC0gMDAwMDAwMDA6ZmMwMGIwMDA6IE5WUw0KKGQ5KSAgWzA0XTog
MDAwMDAwMDA6ZmMwMGIwMDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJWRUQNCihkOSkg
SW52b2tpbmcgU2VhQklPUyAuLi4NCihkOSkgU2VhQklPUyAodmVyc2lvbiAxLjE0LjAtMjAy
MjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQ5KSBCVUlMRDogZ2Nj
OiBtaXhlZCBiaW51dGlsczogKEdOVSBCaW51dGlscykgMi4zNw0KKGQ5KSANCihkOSkgRm91
bmQgWGVuIGh5cGVydmlzb3Igc2lnbmF0dXJlIGF0IDQwMDAwMDAwDQooZDkpIFJ1bm5pbmcg
b24gUUVNVSAoaTQ0MGZ4KQ0KKGQ5KSB4ZW46IGNvcHkgZTgyMC4uLg0KKGQ5KSBSZWxvY2F0
aW5nIGluaXQgZnJvbSAweDAwMGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNpemUgODQxMjgpDQoo
ZDkpIEZvdW5kIDcgUENJIGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0KKGQ5KSBBbGxv
Y2F0ZWQgWGVuIGh5cGVyY2FsbCBwYWdlIGF0IDNmN2ZmMDAwDQooZDkpIERldGVjdGVkIFhl
biB2NC4xNS4wDQooZDkpIHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0KKGQ5KSBDb3B5aW5n
IFNNQklPUyBlbnRyeSBwb2ludCBmcm9tIDB4MDAwMTAwMjAgdG8gMHgwMDBmNWUwMA0KKGQ5
KSBDb3B5aW5nIE1QVEFCTEUgZnJvbSAweGZjMTAwMTUwL2ZjMTAwMTYwIHRvIDB4MDAwZjVk
MjANCihkOSkgQ29weWluZyBQSVIgZnJvbSAweDAwMDEwMDQwIHRvIDB4MDAwZjVjYTANCihk
OSkgQ29weWluZyBBQ1BJIFJTRFAgZnJvbSAweDAwMDEwMGMwIHRvIDB4MDAwZjVjNzANCihk
OSkgdGFibGUoNTA0MzQxNDYpPTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0KKGQ5KSBVc2luZyBw
bXRpbWVyLCBpb3BvcnQgMHhiMDA4DQooZDkpIHRhYmxlKDUwNDM0MTQ2KT0weGZjMDBhMzcw
ICh2aWEgeHNkdCkNCihkOSkgQUNQSTogcGFyc2UgRFNEVCBhdCAweGZjMDAxMDQwIChsZW4g
Mzc1MzkpDQooZDkpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tpcCBmcm9tIDE2
LzI3NjQxDQooZDkpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tpcCBmcm9tIDg3
LzYwNDENCihkOSkgU2NhbiBmb3IgVkdBIG9wdGlvbiByb20NCihkOSkgUnVubmluZyBvcHRp
b24gcm9tIGF0IGMwMDA6MDAwMw0KKGQ5KSBwbW0gY2FsbCBhcmcxPTANCihkOSkgVHVybmlu
ZyBvbiB2Z2EgdGV4dCBtb2RlIGNvbnNvbGUNCihkOSkgU2VhQklPUyAodmVyc2lvbiAxLjE0
LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9iLTA0KQ0KKGQ5KSBNYWNo
aW5lIFVVSUQgODg3NDFiMTUtYWQ0OS0xMWVjLThjMzYtMDAxZTY3MDIwNzA3DQooZDkpIEFU
QSBjb250cm9sbGVyIDEgYXQgMWYwLzNmNC8wIChpcnEgMTQgZGV2IDkpDQooZDkpIEFUQSBj
b250cm9sbGVyIDIgYXQgMTcwLzM3NC8wIChpcnEgMTUgZGV2IDkpDQooZDkpIEFIQ0kgY29u
dHJvbGxlciBhdCAwMDowNC4wLCBpb2Jhc2UgMHhmMzAxMTAwMCwgaXJxIDUNCihkOSkgU2Vh
cmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlza0AwDQoo
ZDkpIEFIQ0kvMDogU2V0IHRyYW5zZmVyIG1vZGUgdG8gVURNQS01DQooZDkpIFNlYXJjaGlu
ZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQvZHJpdmVAMC9kaXNrQDANCihk
OSkgQUhDSS8wOiByZWdpc3RlcmluZzogIkFIQ0kvMDogUUVNVSBIQVJERElTSyBBVEEtNyBI
YXJkLURpc2sgKDEwMjQwIE1pQnl0ZXMpIg0KKGQ5KSBQUzIga2V5Ym9hcmQgaW5pdGlhbGl6
ZWQNCihkOSkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IEhBTFQNCihkOSkgRm91bmQgMCBs
cHQgcG9ydHMNCihkOSkgRm91bmQgMCBzZXJpYWwgcG9ydHMNCihkOSkgQWxsIHRocmVhZHMg
Y29tcGxldGUuDQooZDkpIFNjYW4gZm9yIG9wdGlvbiByb21zDQooZDkpIA0KKGQ5KSBQcmVz
cyBFU0MgZm9yIGJvb3QgbWVudS4NCihkOSkgDQooZDkpIFNlYXJjaGluZyBib290b3JkZXIg
Zm9yOiBIQUxUDQooZDkpIGRyaXZlIDB4MDAwZjViYzA6IFBDSFM9MTYzODMvMTYvNjMgdHJh
bnNsYXRpb249bGJhIExDSFM9MTAyNC8yNTUvNjMgcz0yMDk3MTUyMA0KKGQ5KSBTcGFjZSBh
dmFpbGFibGUgZm9yIFVNQjogYzk4MDAtZTkwMDAsIGY1NjIwLWY1YmMwDQooZDkpIFJldHVy
bmVkIDI1Mzk1MiBieXRlcyBvZiBab25lSGlnaA0KKGQ5KSBlODIwIG1hcCBoYXMgNyBpdGVt
czoNCihkOSkgICAwOiAwMDAwMDAwMDAwMDAwMDAwIC0gMDAwMDAwMDAwMDA5ZmMwMCA9IDEg
UkFNDQooZDkpICAgMTogMDAwMDAwMDAwMDA5ZmMwMCAtIDAwMDAwMDAwMDAwYTAwMDAgPSAy
IFJFU0VSVkVEDQooZDkpICAgMjogMDAwMDAwMDAwMDBmMDAwMCAtIDAwMDAwMDAwMDAxMDAw
MDAgPSAyIFJFU0VSVkVEDQooZDkpICAgMzogMDAwMDAwMDAwMDEwMDAwMCAtIDAwMDAwMDAw
M2Y3ZmUwMDAgPSAxIFJBTQ0KKGQ5KSAgIDQ6IDAwMDAwMDAwM2Y3ZmUwMDAgLSAwMDAwMDAw
MDNmODAwMDAwID0gMiBSRVNFUlZFRA0KKGQ5KSAgIDU6IDAwMDAwMDAwZmMwMDAwMDAgLSAw
MDAwMDAwMGZjMDBiMDAwID0gNCBOVlMNCihkOSkgICA2OiAwMDAwMDAwMGZjMDBiMDAwIC0g
MDAwMDAwMDEwMDAwMDAwMCA9IDIgUkVTRVJWRUQNCihkOSkgZW50ZXIgaGFuZGxlXzE5Og0K
KGQ5KSAgIE5VTEwNCihkOSkgQm9vdGluZyBmcm9tIEhhcmQgRGlzay4uLg0KKGQ5KSBCb290
aW5nIGZyb20gMDAwMDo3YzAwDQooWEVOKSBkOHYwOiB1cGNhbGwgdmVjdG9yIDkzDQp4YmJk
NTogRXJyb3IgMTIgVW5hYmxlIHRvIGFsbG9jYXRlIHJlcXVlc3QgYm91bmNlIGJ1ZmZlcnMN
CnhiYmQ1OiBGYXRhbCBlcnJvci4gVHJhbnNpdGlvbmluZyB0byBDbG9zaW5nIFN0YXRlDQoo
WEVOKSBkOXYwOiB1cGNhbGwgdmVjdG9yIDkzDQooWEVOKSBIVk0gZDEwdjAgc2F2ZTogQ1BV
DQooWEVOKSBIVk0gZDEwIHNhdmU6IFBJQw0KKFhFTikgSFZNIGQxMCBzYXZlOiBJT0FQSUMN
CihYRU4pIEhWTSBkMTB2MCBzYXZlOiBMQVBJQw0KKFhFTikgSFZNIGQxMHYwIHNhdmU6IExB
UElDX1JFR1MNCihYRU4pIEhWTSBkMTAgc2F2ZTogUENJX0lSUQ0KKFhFTikgSFZNIGQxMCBz
YXZlOiBJU0FfSVJRDQooWEVOKSBIVk0gZDEwIHNhdmU6IFBDSV9MSU5LDQooWEVOKSBIVk0g
ZDEwIHNhdmU6IFBJVA0KKFhFTikgSFZNIGQxMCBzYXZlOiBSVEMNCihYRU4pIEhWTSBkMTAg
c2F2ZTogSFBFVA0KKFhFTikgSFZNIGQxMCBzYXZlOiBQTVRJTUVSDQooWEVOKSBIVk0gZDEw
djAgc2F2ZTogTVRSUg0KKFhFTikgSFZNIGQxMCBzYXZlOiBWSVJJRElBTl9ET01BSU4NCihY
RU4pIEhWTSBkMTB2MCBzYXZlOiBDUFVfWFNBVkUNCihYRU4pIEhWTSBkMTB2MCBzYXZlOiBW
SVJJRElBTl9WQ1BVDQooWEVOKSBIVk0gZDEwdjAgc2F2ZTogVk1DRV9WQ1BVDQooWEVOKSBI
Vk0gZDEwdjAgc2F2ZTogVFNDX0FESlVTVA0KKFhFTikgSFZNIGQxMHYwIHNhdmU6IENQVV9N
U1INCihYRU4pIEhWTTEwIHJlc3RvcmU6IENQVSAwDQooZDEwKSBIVk0gTG9hZGVyDQooZDEw
KSBEZXRlY3RlZCBYZW4gdjQuMTUuMA0KKGQxMCkgWGVuYnVzIHJpbmdzIEAweGZlZmZjMDAw
LCBldmVudCBjaGFubmVsIDENCihkMTApIFN5c3RlbSByZXF1ZXN0ZWQgU2VhQklPUw0KKGQx
MCkgQ1BVIHNwZWVkIGlzIDI1MzMgTUh6DQooZDEwKSBSZWxvY2F0aW5nIGd1ZXN0IG1lbW9y
eSBmb3IgbG93bWVtIE1NSU8gc3BhY2UgZGlzYWJsZWQNCihkMTApIFBDSS1JU0EgbGluayAw
IHJvdXRlZCB0byBJUlE1DQooZDEwKSBQQ0ktSVNBIGxpbmsgMSByb3V0ZWQgdG8gSVJRMTAN
CihkMTApIFBDSS1JU0EgbGluayAyIHJvdXRlZCB0byBJUlExMQ0KKGQxMCkgUENJLUlTQSBs
aW5rIDMgcm91dGVkIHRvIElSUTUNCihkMTApIHBjaSBkZXYgMDE6MyBJTlRBLT5JUlExMA0K
KGQxMCkgcGNpIGRldiAwMjowIElOVEEtPklSUTExDQooZDEwKSBwY2kgZGV2IDA0OjAgSU5U
QS0+SVJRNQ0KKGQxMCkgTm8gUkFNIGluIGhpZ2ggbWVtb3J5OyBzZXR0aW5nIGhpZ2hfbWVt
IHJlc291cmNlIGJhc2UgdG8gMTAwMDAwMDAwDQooZDEwKSBwY2kgZGV2IDAzOjAgYmFyIDEw
IHNpemUgMDAyMDAwMDAwOiAwZjAwMDAwMDgNCihkMTApIHBjaSBkZXYgMDI6MCBiYXIgMTQg
c2l6ZSAwMDEwMDAwMDA6IDBmMjAwMDAwOA0KKGQxMCkgcGNpIGRldiAwMzowIGJhciAzMCBz
aXplIDAwMDAxMDAwMDogMGYzMDAwMDAwDQooZDEwKSBwY2kgZGV2IDAzOjAgYmFyIDE0IHNp
emUgMDAwMDAxMDAwOiAwZjMwMTAwMDANCihkMTApIHBjaSBkZXYgMDQ6MCBiYXIgMjQgc2l6
ZSAwMDAwMDEwMDA6IDBmMzAxMTAwMA0KKGQxMCkgcGNpIGRldiAwMjowIGJhciAxMCBzaXpl
IDAwMDAwMDEwMDogMDAwMDBjMDAxDQooZDEwKSBwY2kgZGV2IDA0OjAgYmFyIDIwIHNpemUg
MDAwMDAwMDIwOiAwMDAwMGMxMDENCihkMTApIHBjaSBkZXYgMDE6MSBiYXIgMjAgc2l6ZSAw
MDAwMDAwMTA6IDAwMDAwYzEyMQ0KKGQxMCkgTXVsdGlwcm9jZXNzb3IgaW5pdGlhbGlzYXRp
b246DQooZDEwKSAgLSBDUFUwIC4uLiAzNi1iaXQgcGh5cyAuLi4gZml4ZWQgTVRSUnMgLi4u
IHZhciBNVFJScyBbMS84XSAuLi4gZG9uZS4NCihkMTApIFdyaXRpbmcgU01CSU9TIHRhYmxl
cyAuLi4NCihkMTApIExvYWRpbmcgU2VhQklPUyAuLi4NCihkMTApIENyZWF0aW5nIE1QIHRh
YmxlcyAuLi4NCihkMTApIExvYWRpbmcgQUNQSSAuLi4NCihkMTApIHZtODYgVFNTIGF0IGZj
MTAwMjgwDQooZDEwKSBCSU9TIG1hcDoNCihkMTApICAxMDAwMC0xMDBlMzogU2NyYXRjaCBz
cGFjZQ0KKGQxMCkgIGMwMDAwLWZmZmZmOiBNYWluIEJJT1MNCihkMTApIEU4MjAgdGFibGU6
DQooZDEwKSAgWzAwXTogMDAwMDAwMDA6MDAwMDAwMDAgLSAwMDAwMDAwMDowMDBhMDAwMDog
UkFNDQooZDEwKSAgSE9MRTogMDAwMDAwMDA6MDAwYTAwMDAgLSAwMDAwMDAwMDowMDBjMDAw
MA0KKGQxMCkgIFswMV06IDAwMDAwMDAwOjAwMGMwMDAwIC0gMDAwMDAwMDA6MDAxMDAwMDA6
IFJFU0VSVkVEDQooZDEwKSAgWzAyXTogMDAwMDAwMDA6MDAxMDAwMDAgLSAwMDAwMDAwMDoz
ZjgwMDAwMDogUkFNDQooZDEwKSAgSE9MRTogMDAwMDAwMDA6M2Y4MDAwMDAgLSAwMDAwMDAw
MDpmYzAwMDAwMA0KKGQxMCkgIFswM106IDAwMDAwMDAwOmZjMDAwMDAwIC0gMDAwMDAwMDA6
ZmMwMGIwMDA6IE5WUw0KKGQxMCkgIFswNF06IDAwMDAwMDAwOmZjMDBiMDAwIC0gMDAwMDAw
MDE6MDAwMDAwMDA6IFJFU0VSVkVEDQooZDEwKSBJbnZva2luZyBTZWFCSU9TIC4uLg0KKGQx
MCkgU2VhQklPUyAodmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRl
ZmF1bHQtam9iLTA0KQ0KKGQxMCkgQlVJTEQ6IGdjYzogbWl4ZWQgYmludXRpbHM6IChHTlUg
QmludXRpbHMpIDIuMzcNCihkMTApIA0KKGQxMCkgRm91bmQgWGVuIGh5cGVydmlzb3Igc2ln
bmF0dXJlIGF0IDQwMDAwMDAwDQooZDEwKSBSdW5uaW5nIG9uIFFFTVUgKGk0NDBmeCkNCihk
MTApIHhlbjogY29weSBlODIwLi4uDQooZDEwKSBSZWxvY2F0aW5nIGluaXQgZnJvbSAweDAw
MGQ2MjAwIHRvIDB4M2Y3YWI2YzAgKHNpemUgODQxMjgpDQooZDEwKSBGb3VuZCA3IFBDSSBk
ZXZpY2VzIChtYXggUENJIGJ1cyBpcyAwMCkNCihkMTApIEFsbG9jYXRlZCBYZW4gaHlwZXJj
YWxsIHBhZ2UgYXQgM2Y3ZmYwMDANCihkMTApIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDEw
KSB4ZW46IGNvcHkgQklPUyB0YWJsZXMuLi4NCihkMTApIENvcHlpbmcgU01CSU9TIGVudHJ5
IHBvaW50IGZyb20gMHgwMDAxMDAyMCB0byAweDAwMGY1ZTAwDQooZDEwKSBDb3B5aW5nIE1Q
VEFCTEUgZnJvbSAweGZjMTAwMTUwL2ZjMTAwMTYwIHRvIDB4MDAwZjVkMjANCihkMTApIENv
cHlpbmcgUElSIGZyb20gMHgwMDAxMDA0MCB0byAweDAwMGY1Y2EwDQooZDEwKSBDb3B5aW5n
IEFDUEkgUlNEUCBmcm9tIDB4MDAwMTAwYzAgdG8gMHgwMDBmNWM3MA0KKGQxMCkgdGFibGUo
NTA0MzQxNDYpPTB4ZmMwMGEzNzAgKHZpYSB4c2R0KQ0KKGQxMCkgVXNpbmcgcG10aW1lciwg
aW9wb3J0IDB4YjAwOA0KKGQxMCkgdGFibGUoNTA0MzQxNDYpPTB4ZmMwMGEzNzAgKHZpYSB4
c2R0KQ0KKGQxMCkgQUNQSTogcGFyc2UgRFNEVCBhdCAweGZjMDAxMDQwIChsZW4gMzc1Mzkp
DQooZDEwKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNraXAgZnJvbSAxNi8yNzY0
MQ0KKGQxMCkgcGFyc2VfdGVybWxpc3Q6IHBhcnNlIGVycm9yLCBza2lwIGZyb20gODcvNjA0
MQ0KKGQxMCkgU2NhbiBmb3IgVkdBIG9wdGlvbiByb20NCihkMTApIFJ1bm5pbmcgb3B0aW9u
IHJvbSBhdCBjMDAwOjAwMDMNCihkMTApIHBtbSBjYWxsIGFyZzE9MA0KKGQxMCkgVHVybmlu
ZyBvbiB2Z2EgdGV4dCBtb2RlIGNvbnNvbGUNCihkMTApIFNlYUJJT1MgKHZlcnNpb24gMS4x
NC4wLTIwMjIwMjAzXzIwMzA1MC0xMzBhbWQ2NC1kZWZhdWx0LWpvYi0wNCkNCihkMTApIE1h
Y2hpbmUgVVVJRCBiZjMyN2RlZS1hZDQ5LTExZWMtOGMzNi0wMDFlNjcwMjA3MDcNCihkMTAp
IEFUQSBjb250cm9sbGVyIDEgYXQgMWYwLzNmNC8wIChpcnEgMTQgZGV2IDkpDQooZDEwKSBB
VEEgY29udHJvbGxlciAyIGF0IDE3MC8zNzQvMCAoaXJxIDE1IGRldiA5KQ0KKGQxMCkgQUhD
SSBjb250cm9sbGVyIGF0IDAwOjA0LjAsIGlvYmFzZSAweGYzMDExMDAwLCBpcnEgNQ0KKGQx
MCkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDAvZGlz
a0AwDQooZDEwKSBBSENJLzA6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQxMCkg
U2VhcmNoaW5nIGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUAwL2Rp
c2tAMA0KKGQxMCkgU2VhcmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2Ry
aXZlQDEvZGlza0AwDQooZDEwKSBBSENJLzE6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEt
NQ0KKGQxMCkgU2VhcmNoaW5nIGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypANC9k
cml2ZUAxL2Rpc2tAMA0KKGQxMCkgQUhDSS8wOiByZWdpc3RlcmluZzogIkFIQ0kvMDogUUVN
VSBIQVJERElTSyBBVEEtNyBIYXJkLURpc2sgKDEwMjQwIE1pQnl0ZXMpIg0KKGQxMCkgUFMy
IGtleWJvYXJkIGluaXRpYWxpemVkDQooZDEwKSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjog
L3BjaUBpMGNmOC8qQDQvZHJpdmVAMi9kaXNrQDANCihkMTApIEFIQ0kvMjogU2V0IHRyYW5z
ZmVyIG1vZGUgdG8gVURNQS01DQooZDEwKSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6
IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDIvZGlza0AwDQooZDEwKSBBSENJLzE6IHJlZ2lzdGVy
aW5nOiAiQUhDSS8xOiBRRU1VIEhBUkRESVNLIEFUQS03IEhhcmQtRGlzayAoMjA0OCBNaUJ5
dGVzKSINCihkMTApIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiAvcGNpQGkwY2Y4LypANC9k
cml2ZUAzL2Rpc2tAMA0KKGQxMCkgQUhDSS8zOiBTZXQgdHJhbnNmZXIgbW9kZSB0byBVRE1B
LTUNCihkMTApIFNlYXJjaGluZyBiaW9zLWdlb21ldHJ5IGZvcjogL3BjaUBpMGNmOC8qQDQv
ZHJpdmVAMy9kaXNrQDANCihkMTApIEFIQ0kvMjogcmVnaXN0ZXJpbmc6ICJBSENJLzI6IFFF
TVUgSEFSRERJU0sgQVRBLTcgSGFyZC1EaXNrICgyMDQ4IE1pQnl0ZXMpIg0KKGQxMCkgU2Vh
cmNoaW5nIGJvb3RvcmRlciBmb3I6IC9wY2lAaTBjZjgvKkA0L2RyaXZlQDQvZGlza0AwDQoo
ZDEwKSBBSENJLzQ6IFNldCB0cmFuc2ZlciBtb2RlIHRvIFVETUEtNQ0KKGQxMCkgU2VhcmNo
aW5nIGJpb3MtZ2VvbWV0cnkgZm9yOiAvcGNpQGkwY2Y4LypANC9kcml2ZUA0L2Rpc2tAMA0K
KGQxMCkgQUhDSS8zOiByZWdpc3RlcmluZzogIkFIQ0kvMzogUUVNVSBIQVJERElTSyBBVEEt
NyBIYXJkLURpc2sgKDIwNDggTWlCeXRlcykiDQooZDEwKSBBSENJLzQ6IHJlZ2lzdGVyaW5n
OiAiQUhDSS80OiBRRU1VIEhBUkRESVNLIEFUQS03IEhhcmQtRGlzayAoMjA0OCBNaUJ5dGVz
KSINCihkMTApIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiBIQUxUDQooZDEwKSBGb3VuZCAw
IGxwdCBwb3J0cw0KKGQxMCkgRm91bmQgMCBzZXJpYWwgcG9ydHMNCihkMTApIEFsbCB0aHJl
YWRzIGNvbXBsZXRlLg0KKGQxMCkgU2NhbiBmb3Igb3B0aW9uIHJvbXMNCihkMTApIA0KKGQx
MCkgUHJlc3MgRVNDIGZvciBib290IG1lbnUuDQooZDEwKSANCihkMTApIFNlYXJjaGluZyBi
b290b3JkZXIgZm9yOiBIQUxUDQooZDEwKSBkcml2ZSAweDAwMGY1YmMwOiBQQ0hTPTE2Mzgz
LzE2LzYzIHRyYW5zbGF0aW9uPWxiYSBMQ0hTPTEwMjQvMjU1LzYzIHM9MjA5NzE1MjANCihk
MTApIGRyaXZlIDB4MDAwZjViNzA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJn
ZSBMQ0hTPTUyMC8xMjgvNjMgcz00MTk0MzA0DQooZDEwKSBkcml2ZSAweDAwMGY1YjIwOiBQ
Q0hTPTQxNjEvMTYvNjMgdHJhbnNsYXRpb249bGFyZ2UgTENIUz01MjAvMTI4LzYzIHM9NDE5
NDMwNA0KKGQxMCkgZHJpdmUgMHgwMDBmNWFkMDogUENIUz00MTYxLzE2LzYzIHRyYW5zbGF0
aW9uPWxhcmdlIExDSFM9NTIwLzEyOC82MyBzPTQxOTQzMDQNCihkMTApIGRyaXZlIDB4MDAw
ZjVhODA6IFBDSFM9NDE2MS8xNi82MyB0cmFuc2xhdGlvbj1sYXJnZSBMQ0hTPTUyMC8xMjgv
NjMgcz00MTk0MzA0DQooZDEwKSBTcGFjZSBhdmFpbGFibGUgZm9yIFVNQjogYzk4MDAtZTkw
MDAsIGY1NjIwLWY1YTgwDQooZDEwKSBSZXR1cm5lZCAyNDk4NTYgYnl0ZXMgb2YgWm9uZUhp
Z2gNCihkMTApIGU4MjAgbWFwIGhhcyA3IGl0ZW1zOg0KKGQxMCkgICAwOiAwMDAwMDAwMDAw
MDAwMDAwIC0gMDAwMDAwMDAwMDA5ZmMwMCA9IDEgUkFNDQooZDEwKSAgIDE6IDAwMDAwMDAw
MDAwOWZjMDAgLSAwMDAwMDAwMDAwMGEwMDAwID0gMiBSRVNFUlZFRA0KKGQxMCkgICAyOiAw
MDAwMDAwMDAwMGYwMDAwIC0gMDAwMDAwMDAwMDEwMDAwMCA9IDIgUkVTRVJWRUQNCihkMTAp
ICAgMzogMDAwMDAwMDAwMDEwMDAwMCAtIDAwMDAwMDAwM2Y3ZmQwMDAgPSAxIFJBTQ0KKGQx
MCkgICA0OiAwMDAwMDAwMDNmN2ZkMDAwIC0gMDAwMDAwMDAzZjgwMDAwMCA9IDIgUkVTRVJW
RUQNCihkMTApICAgNTogMDAwMDAwMDBmYzAwMDAwMCAtIDAwMDAwMDAwZmMwMGIwMDAgPSA0
IE5WUw0KKGQxMCkgICA2OiAwMDAwMDAwMGZjMDBiMDAwIC0gMDAwMDAwMDEwMDAwMDAwMCA9
IDIgUkVTRVJWRUQNCihkMTApIGVudGVyIGhhbmRsZV8xOToNCihkMTApICAgTlVMTA0KKGQx
MCkgQm9vdGluZyBmcm9tIEhhcmQgRGlzay4uLg0KKGQxMCkgQm9vdGluZyBmcm9tIDAwMDA6
N2MwMA0KKFhFTikgZDEwdjA6IHVwY2FsbCB2ZWN0b3IgOTMNCnhiYmQzOiBFcnJvciAxMiBV
bmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDM6IEZhdGFs
IGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCnhiYmQ1OiBFcnJvciAx
MiBVbmFibGUgdG8gYWxsb2NhdGUgcmVxdWVzdCBib3VuY2UgYnVmZmVycw0KeGJiZDU6IEZh
dGFsIGVycm9yLiBUcmFuc2l0aW9uaW5nIHRvIENsb3NpbmcgU3RhdGUNCihYRU4pIEhWTSBk
MTF2MCBzYXZlOiBDUFUNCihYRU4pIEhWTSBkMTEgc2F2ZTogUElDDQooWEVOKSBIVk0gZDEx
IHNhdmU6IElPQVBJQw0KKFhFTikgSFZNIGQxMXYwIHNhdmU6IExBUElDDQooWEVOKSBIVk0g
ZDExdjAgc2F2ZTogTEFQSUNfUkVHUw0KKFhFTikgSFZNIGQxMSBzYXZlOiBQQ0lfSVJRDQoo
WEVOKSBIVk0gZDExIHNhdmU6IElTQV9JUlENCihYRU4pIEhWTSBkMTEgc2F2ZTogUENJX0xJ
TksNCihYRU4pIEhWTSBkMTEgc2F2ZTogUElUDQooWEVOKSBIVk0gZDExIHNhdmU6IFJUQw0K
KFhFTikgSFZNIGQxMSBzYXZlOiBIUEVUDQooWEVOKSBIVk0gZDExIHNhdmU6IFBNVElNRVIN
CihYRU4pIEhWTSBkMTF2MCBzYXZlOiBNVFJSDQooWEVOKSBIVk0gZDExIHNhdmU6IFZJUklE
SUFOX0RPTUFJTg0KKFhFTikgSFZNIGQxMXYwIHNhdmU6IENQVV9YU0FWRQ0KKFhFTikgSFZN
IGQxMXYwIHNhdmU6IFZJUklESUFOX1ZDUFUNCihYRU4pIEhWTSBkMTF2MCBzYXZlOiBWTUNF
X1ZDUFUNCihYRU4pIEhWTSBkMTF2MCBzYXZlOiBUU0NfQURKVVNUDQooWEVOKSBIVk0gZDEx
djAgc2F2ZTogQ1BVX01TUg0KKFhFTikgSFZNMTEgcmVzdG9yZTogQ1BVIDANCihkMTEpIEhW
TSBMb2FkZXINCihkMTEpIERldGVjdGVkIFhlbiB2NC4xNS4wDQooZDExKSBYZW5idXMgcmlu
Z3MgQDB4ZmVmZmMwMDAsIGV2ZW50IGNoYW5uZWwgMQ0KKGQxMSkgU3lzdGVtIHJlcXVlc3Rl
ZCBTZWFCSU9TDQooZDExKSBDUFUgc3BlZWQgaXMgMjUzMyBNSHoNCihkMTEpIFJlbG9jYXRp
bmcgZ3Vlc3QgbWVtb3J5IGZvciBsb3dtZW0gTU1JTyBzcGFjZSBkaXNhYmxlZA0KKGQxMSkg
UENJLUlTQSBsaW5rIDAgcm91dGVkIHRvIElSUTUNCihkMTEpIFBDSS1JU0EgbGluayAxIHJv
dXRlZCB0byBJUlExMA0KKGQxMSkgUENJLUlTQSBsaW5rIDIgcm91dGVkIHRvIElSUTExDQoo
ZDExKSBQQ0ktSVNBIGxpbmsgMyByb3V0ZWQgdG8gSVJRNQ0KKGQxMSkgcGNpIGRldiAwMToz
IElOVEEtPklSUTEwDQooZDExKSBwY2kgZGV2IDAyOjAgSU5UQS0+SVJRMTENCihkMTEpIHBj
aSBkZXYgMDQ6MCBJTlRBLT5JUlE1DQooZDExKSBObyBSQU0gaW4gaGlnaCBtZW1vcnk7IHNl
dHRpbmcgaGlnaF9tZW0gcmVzb3VyY2UgYmFzZSB0byAxMDAwMDAwMDANCihkMTEpIHBjaSBk
ZXYgMDM6MCBiYXIgMTAgc2l6ZSAwMDIwMDAwMDA6IDBmMDAwMDAwOA0KKGQxMSkgcGNpIGRl
diAwMjowIGJhciAxNCBzaXplIDAwMTAwMDAwMDogMGYyMDAwMDA4DQooZDExKSBwY2kgZGV2
IDAzOjAgYmFyIDMwIHNpemUgMDAwMDEwMDAwOiAwZjMwMDAwMDANCihkMTEpIHBjaSBkZXYg
MDM6MCBiYXIgMTQgc2l6ZSAwMDAwMDEwMDA6IDBmMzAxMDAwMA0KKGQxMSkgcGNpIGRldiAw
NDowIGJhciAyNCBzaXplIDAwMDAwMTAwMDogMGYzMDExMDAwDQooZDExKSBwY2kgZGV2IDAy
OjAgYmFyIDEwIHNpemUgMDAwMDAwMTAwOiAwMDAwMGMwMDENCihkMTEpIHBjaSBkZXYgMDQ6
MCBiYXIgMjAgc2l6ZSAwMDAwMDAwMjA6IDAwMDAwYzEwMQ0KKGQxMSkgcGNpIGRldiAwMTox
IGJhciAyMCBzaXplIDAwMDAwMDAxMDogMDAwMDBjMTIxDQooZDExKSBNdWx0aXByb2Nlc3Nv
ciBpbml0aWFsaXNhdGlvbjoNCihkMTEpICAtIENQVTAgLi4uIDM2LWJpdCBwaHlzIC4uLiBm
aXhlZCBNVFJScyAuLi4gdmFyIE1UUlJzIFsxLzhdIC4uLiBkb25lLg0KKGQxMSkgV3JpdGlu
ZyBTTUJJT1MgdGFibGVzIC4uLg0KKGQxMSkgTG9hZGluZyBTZWFCSU9TIC4uLg0KKGQxMSkg
Q3JlYXRpbmcgTVAgdGFibGVzIC4uLg0KKGQxMSkgTG9hZGluZyBBQ1BJIC4uLg0KKGQxMSkg
dm04NiBUU1MgYXQgZmMxMDAyODANCihkMTEpIEJJT1MgbWFwOg0KKGQxMSkgIDEwMDAwLTEw
MGUzOiBTY3JhdGNoIHNwYWNlDQooZDExKSAgYzAwMDAtZmZmZmY6IE1haW4gQklPUw0KKGQx
MSkgRTgyMCB0YWJsZToNCihkMTEpICBbMDBdOiAwMDAwMDAwMDowMDAwMDAwMCAtIDAwMDAw
MDAwOjAwMGEwMDAwOiBSQU0NCihkMTEpICBIT0xFOiAwMDAwMDAwMDowMDBhMDAwMCAtIDAw
MDAwMDAwOjAwMGMwMDAwDQooZDExKSAgWzAxXTogMDAwMDAwMDA6MDAwYzAwMDAgLSAwMDAw
MDAwMDowMDEwMDAwMDogUkVTRVJWRUQNCihkMTEpICBbMDJdOiAwMDAwMDAwMDowMDEwMDAw
MCAtIDAwMDAwMDAwOjNmODAwMDAwOiBSQU0NCihkMTEpICBIT0xFOiAwMDAwMDAwMDozZjgw
MDAwMCAtIDAwMDAwMDAwOmZjMDAwMDAwDQooZDExKSAgWzAzXTogMDAwMDAwMDA6ZmMwMDAw
MDAgLSAwMDAwMDAwMDpmYzAwYjAwMDogTlZTDQooZDExKSAgWzA0XTogMDAwMDAwMDA6ZmMw
MGIwMDAgLSAwMDAwMDAwMTowMDAwMDAwMDogUkVTRVJWRUQNCihkMTEpIEludm9raW5nIFNl
YUJJT1MgLi4uDQooZDExKSBTZWFCSU9TICh2ZXJzaW9uIDEuMTQuMC0yMDIyMDIwM18yMDMw
NTAtMTMwYW1kNjQtZGVmYXVsdC1qb2ItMDQpDQooZDExKSBCVUlMRDogZ2NjOiBtaXhlZCBi
aW51dGlsczogKEdOVSBCaW51dGlscykgMi4zNw0KKGQxMSkgDQooZDExKSBGb3VuZCBYZW4g
aHlwZXJ2aXNvciBzaWduYXR1cmUgYXQgNDAwMDAwMDANCihkMTEpIFJ1bm5pbmcgb24gUUVN
VSAoaTQ0MGZ4KQ0KKGQxMSkgeGVuOiBjb3B5IGU4MjAuLi4NCihkMTEpIFJlbG9jYXRpbmcg
aW5pdCBmcm9tIDB4MDAwZDYyMDAgdG8gMHgzZjdhYjZjMCAoc2l6ZSA4NDEyOCkNCihkMTEp
IEZvdW5kIDcgUENJIGRldmljZXMgKG1heCBQQ0kgYnVzIGlzIDAwKQ0KKGQxMSkgQWxsb2Nh
dGVkIFhlbiBoeXBlcmNhbGwgcGFnZSBhdCAzZjdmZjAwMA0KKGQxMSkgRGV0ZWN0ZWQgWGVu
IHY0LjE1LjANCihkMTEpIHhlbjogY29weSBCSU9TIHRhYmxlcy4uLg0KKGQxMSkgQ29weWlu
ZyBTTUJJT1MgZW50cnkgcG9pbnQgZnJvbSAweDAwMDEwMDIwIHRvIDB4MDAwZjVlMDANCihk
MTEpIENvcHlpbmcgTVBUQUJMRSBmcm9tIDB4ZmMxMDAxNTAvZmMxMDAxNjAgdG8gMHgwMDBm
NWQyMA0KKGQxMSkgQ29weWluZyBQSVIgZnJvbSAweDAwMDEwMDQwIHRvIDB4MDAwZjVjYTAN
CihkMTEpIENvcHlpbmcgQUNQSSBSU0RQIGZyb20gMHgwMDAxMDBjMCB0byAweDAwMGY1Yzcw
DQooZDExKSB0YWJsZSg1MDQzNDE0Nik9MHhmYzAwYTM3MCAodmlhIHhzZHQpDQooZDExKSBV
c2luZyBwbXRpbWVyLCBpb3BvcnQgMHhiMDA4DQooZDExKSB0YWJsZSg1MDQzNDE0Nik9MHhm
YzAwYTM3MCAodmlhIHhzZHQpDQooZDExKSBBQ1BJOiBwYXJzZSBEU0RUIGF0IDB4ZmMwMDEw
NDAgKGxlbiAzNzUzOSkNCihkMTEpIHBhcnNlX3Rlcm1saXN0OiBwYXJzZSBlcnJvciwgc2tp
cCBmcm9tIDE2LzI3NjQxDQooZDExKSBwYXJzZV90ZXJtbGlzdDogcGFyc2UgZXJyb3IsIHNr
aXAgZnJvbSA4Ny82MDQxDQooZDExKSBTY2FuIGZvciBWR0Egb3B0aW9uIHJvbQ0KKGQxMSkg
UnVubmluZyBvcHRpb24gcm9tIGF0IGMwMDA6MDAwMw0KKGQxMSkgcG1tIGNhbGwgYXJnMT0w
DQooZDExKSBUdXJuaW5nIG9uIHZnYSB0ZXh0IG1vZGUgY29uc29sZQ0KKGQxMSkgU2VhQklP
UyAodmVyc2lvbiAxLjE0LjAtMjAyMjAyMDNfMjAzMDUwLTEzMGFtZDY0LWRlZmF1bHQtam9i
LTA0KQ0KKGQxMSkgTWFjaGluZSBVVUlEIGY1MjJkMGM5LWFkNGUtMTFlYy04YzM2LTAwMWU2
NzAyMDcwNw0KKGQxMSkgQVRBIGNvbnRyb2xsZXIgMSBhdCAxZjAvM2Y0LzAgKGlycSAxNCBk
ZXYgOSkNCihkMTEpIEFUQSBjb250cm9sbGVyIDIgYXQgMTcwLzM3NC8wIChpcnEgMTUgZGV2
IDkpDQooZDExKSBBSENJIGNvbnRyb2xsZXIgYXQgMDA6MDQuMCwgaW9iYXNlIDB4ZjMwMTEw
MDAsIGlycSA1DQooZDExKSBTZWFyY2hpbmcgYm9vdG9yZGVyIGZvcjogL3BjaUBpMGNmOC8q
QDQvZHJpdmVAMC9kaXNrQDANCihkMTEpIEFIQ0kvMDogU2V0IHRyYW5zZmVyIG1vZGUgdG8g
VURNQS01DQooZDExKSBTZWFyY2hpbmcgYmlvcy1nZW9tZXRyeSBmb3I6IC9wY2lAaTBjZjgv
KkA0L2RyaXZlQDAvZGlza0AwDQooZDExKSBBSENJLzA6IHJlZ2lzdGVyaW5nOiAiQUhDSS8w
OiBRRU1VIEhBUkRESVNLIEFUQS03IEhhcmQtRGlzayAoMTAyNDAgTWlCeXRlcykiDQooZDEx
KSBQUzIga2V5Ym9hcmQgaW5pdGlhbGl6ZWQNCihkMTEpIFNlYXJjaGluZyBib290b3JkZXIg
Zm9yOiBIQUxUDQooZDExKSBGb3VuZCAwIGxwdCBwb3J0cw0KKGQxMSkgRm91bmQgMCBzZXJp
YWwgcG9ydHMNCihkMTEpIEFsbCB0aHJlYWRzIGNvbXBsZXRlLg0KKGQxMSkgU2NhbiBmb3Ig
b3B0aW9uIHJvbXMNCihkMTEpIA0KKGQxMSkgUHJlc3MgRVNDIGZvciBib290IG1lbnUuDQoo
ZDExKSANCihkMTEpIFNlYXJjaGluZyBib290b3JkZXIgZm9yOiBIQUxUDQooZDExKSBkcml2
ZSAweDAwMGY1YmMwOiBQQ0hTPTE2MzgzLzE2LzYzIHRyYW5zbGF0aW9uPWxiYSBMQ0hTPTEw
MjQvMjU1LzYzIHM9MjA5NzE1MjANCihkMTEpIFNwYWNlIGF2YWlsYWJsZSBmb3IgVU1COiBj
OTgwMC1lOTAwMCwgZjU2MjAtZjViYzANCihkMTEpIFJldHVybmVkIDI1Mzk1MiBieXRlcyBv
ZiBab25lSGlnaA0KKGQxMSkgZTgyMCBtYXAgaGFzIDcgaXRlbXM6DQooZDExKSAgIDA6IDAw
MDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAwMDAwMDlmYzAwID0gMSBSQU0NCihkMTEpICAgMTog
MDAwMDAwMDAwMDA5ZmMwMCAtIDAwMDAwMDAwMDAwYTAwMDAgPSAyIFJFU0VSVkVEDQooZDEx
KSAgIDI6IDAwMDAwMDAwMDAwZjAwMDAgLSAwMDAwMDAwMDAwMTAwMDAwID0gMiBSRVNFUlZF
RA0KKGQxMSkgICAzOiAwMDAwMDAwMDAwMTAwMDAwIC0gMDAwMDAwMDAzZjdmZTAwMCA9IDEg
UkFNDQooZDExKSAgIDQ6IDAwMDAwMDAwM2Y3ZmUwMDAgLSAwMDAwMDAwMDNmODAwMDAwID0g
MiBSRVNFUlZFRA0KKGQxMSkgICA1OiAwMDAwMDAwMGZjMDAwMDAwIC0gMDAwMDAwMDBmYzAw
YjAwMCA9IDQgTlZTDQooZDExKSAgIDY6IDAwMDAwMDAwZmMwMGIwMDAgLSAwMDAwMDAwMTAw
MDAwMDAwID0gMiBSRVNFUlZFRA0KKGQxMSkgZW50ZXIgaGFuZGxlXzE5Og0KKGQxMSkgICBO
VUxMDQooZDExKSBCb290aW5nIGZyb20gSGFyZCBEaXNrLi4uDQooZDExKSBCb290aW5nIGZy
b20gMDAwMDo3YzAwDQooWEVOKSBkMTF2MDogdXBjYWxsIHZlY3RvciA5Mw0KKFhFTikgSFZN
IGQxMnYwIHNhdmU6IENQVQ0KKFhFTikgSFZNIGQxMiBzYXZlOiBQSUMNCihYRU4pIEhWTSBk
MTIgc2F2ZTogSU9BUElDDQooWEVOKSBIVk0gZDEydjAgc2F2ZTogTEFQSUMNCihYRU4pIEhW
TSBkMTJ2MCBzYXZlOiBMQVBJQ19SRUdTDQooWEVOKSBIVk0gZDEyIHNhdmU6IFBDSV9JUlEN
CihYRU4pIEhWTSBkMTIgc2F2ZTogSVNBX0lSUQ0KKFhFTikgSFZNIGQxMiBzYXZlOiBQQ0lf
TElOSw0KKFhFTikgSFZNIGQxMiBzYXZlOiBQSVQNCihYRU4pIEhWTSBkMTIgc2F2ZTogUlRD
DQooWEVOKSBIVk0gZDEyIHNhdmU6IEhQRVQNCihYRU4pIEhWTSBkMTIgc2F2ZTogUE1USU1F
Ug0KKFhFTikgSFZNIGQxMnYwIHNhdmU6IE1UUlINCihYRU4pIEhWTSBkMTIgc2F2ZTogVklS
SURJQU5fRE9NQUlODQooWEVOKSBIVk0gZDEydjAgc2F2ZTogQ1BVX1hTQVZFDQooWEVOKSBI
Vk0gZDEydjAgc2F2ZTogVklSSURJQU5fVkNQVQ0KKFhFTikgSFZNIGQxMnYwIHNhdmU6IFZN
Q0VfVkNQVQ0KKFhFTikgSFZNIGQxMnYwIHNhdmU6IFRTQ19BREpVU1QNCihYRU4pIEhWTSBk
MTJ2MCBzYXZlOiBDUFVfTVNSDQooWEVOKSBIVk0xMiByZXN0b3JlOiBDUFUgMA0KDQoNCkZh
dGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4ga2VybmVsIG1vZGUNCmNwdWlkID0g
MjsgYXBpYyBpZCA9IDA0DQpmYXVsdCB2aXJ0dWFsIGFkZHJlc3MJPSAweGE0DQpmYXVsdCBj
b2RlCQk9IHN1cGVydmlzb3IgcmVhZCBkYXRhLCBwYWdlIG5vdCBwcmVzZW50DQppbnN0cnVj
dGlvbiBwb2ludGVyCT0gMHgyMDoweGZmZmZmZmZmODBjOTBlZDANCnN0YWNrIHBvaW50ZXIJ
ICAgICAgICA9IDB4Mjg6MHhmZmZmZmUwMDUxOTI3YWIwDQpmcmFtZSBwb2ludGVyCSAgICAg
ICAgPSAweDI4OjB4ZmZmZmZlMDA1MTkyN2FkMA0KY29kZSBzZWdtZW50CQk9IGJhc2UgMHgw
LCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWINCgkJCT0gRFBMIDAsIHByZXMgMSwgbG9uZyAx
LCBkZWYzMiAwLCBncmFuIDENCnByb2Nlc3NvciBlZmxhZ3MJPSBpbnRlcnJ1cHQgZW5hYmxl
ZCwgcmVzdW1lLCBJT1BMID0gMA0KY3VycmVudCBwcm9jZXNzCQk9IDE2ICh4ZW53YXRjaCkN
CnRyYXAgbnVtYmVyCQk9IDEyDQpwYW5pYzogcGFnZSBmYXVsdA0KY3B1aWQgPSAxDQp0aW1l
ID0gMTY0ODMzMTU5Mg0KS0RCOiBzdGFjayBiYWNrdHJhY2U6DQojMCAweGZmZmZmZmZmODBj
N2MyNzUgYXQga2RiX2JhY2t0cmFjZSsweDY1DQojMSAweGZmZmZmZmZmODBjMmUyZDEgYXQg
dnBhbmljKzB4MTgxDQojMiAweGZmZmZmZmZmODBjMmUxNDMgYXQgcGFuaWMrMHg0Mw0KIzMg
MHhmZmZmZmZmZjgxMGM4Yjk3IGF0IHRyYXArMHhiYTcNCiM0IDB4ZmZmZmZmZmY4MTBjOGJl
ZiBhdCB0cmFwKzB4YmZmDQojNSAweGZmZmZmZmZmODEwYzgyNDMgYXQgdHJhcCsweDI1Mw0K
IzYgMHhmZmZmZmZmZjgxMGEwODM4IGF0IGNhbGx0cmFwKzB4OA0KIzcgMHhmZmZmZmZmZjgw
YTk4NTE1IGF0IHhiZF9pbnN0YW5jZV9jcmVhdGUrMHg3ODk1DQojOCAweGZmZmZmZmZmODBh
OTg0NjIgYXQgeGJkX2luc3RhbmNlX2NyZWF0ZSsweDc3ZTINCiM5IDB4ZmZmZmZmZmY4MGE5
NjE5YiBhdCB4YmRfaW5zdGFuY2VfY3JlYXRlKzB4NTUxYg0KIzEwIDB4ZmZmZmZmZmY4MGY5
NWM1NCBhdCB4ZW5idXNiX2xvY2FsZW5kX2NoYW5nZWQrMHg3YzQNCiMxMSAweGZmZmZmZmZm
ODBhYjBlZjQgYXQgeHNfdW5sb2NrKzB4NzA0DQojMTIgMHhmZmZmZmZmZjgwYmVhZWRlIGF0
IGZvcmtfZXhpdCsweDdlDQojMTMgMHhmZmZmZmZmZjgxMGExOGFlIGF0IGZvcmtfdHJhbXBv
bGluZSsweGUNClVwdGltZTogMWgyOW0zMXMNCihhZGEwOmFoY2ljaDA6MDowOjApOiBzcGlu
LWRvd24NCihhZGExOmFoY2ljaDE6MDowOjApOiBzcGluLWRvd24NCg==
--------------6658CE0D501BB4674047759C--

From nobody Sun Mar 27 09:13:42 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 370851A436A5
	for ; Sun, 27 Mar 2022 09:13:59 +0000 (UTC)
	(envelope-from prvs=0786bbb88=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KR99y0Y9Wz3Bwf
	for ; Sun, 27 Mar 2022 09:13:57 +0000 (UTC)
	(envelope-from prvs=0786bbb88=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648372438;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=sDgDS3d4+EnV4ntO5BN8ciJTN3sU6GONHafQzRWP1ck=;
  b=cHWaYBQ25FzVsEQV0ugZ98kBxTsxQ4gjHHx/nAR7yKPVk/J5xckHAdZS
   dfrnpkkQm/5g2L8piny92JRE4r6gGzEpr4FwknF28J+3JlEQEC2XkqPvX
   8dnXKueAfWTjN8eT8VHZWhqURXthRUL6twRZXCoP5n6GT4KidBVzGXy68
   s=;
X-SBRS: 5.1
X-MesageID: 67263878
X-Ironport-Server: esa3.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:4qocl6MtxUU9JTXvrR2yl8FynXyQoLVcMsEvi/4bfWQNrUpzhGFSx
 2tLXT2GPv/eYWLzfdsiaYTn9kMF75WGy9UxGQto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h
 ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFcMpBsJ00o5wbZi2NAw2rBVPivW0
 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb
 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Z1
 PsTmoCcWz8SAqDtv+9CCQBfOR0vMvgTkFPHCSDXXc27ykTHdz3nwul0DVFwNoodkgp1KTgQr
 7pCcmlLN03dwbLtqF64YrAEasALNs7kMZlZonh95TrYEewnUdbIRKCiCdpwgmhq2JseQqa2i
 8wxVX1dZROdfgRzYWgdApZuusi3gEPRSmgNwL6SjfVuuDWCpOBr65DnLdrPfcOiQ8tflU+c4
 GXc8AzRDAsXOdaExX+V/3SEiPXCkCngVMQVDrLQ3uxnjECL3X0IIBITXFq/5/K+jyaDt8l3c
 hJOvHB09O5rqRLtHoKVswCETGCsojBCdMZ8AdUB1Ays7aGLuR+eLFRVQWsUADA5j/MeSTsv3
 16PutrmAz1zrbGYIU6gGqeoQSCaYnZMczJbDcMQZU5cuoS4/tlv5v7aZow7eJNZmOEZDt0ZL
 9qiiCElz4segscQv0lQ1QCW2mn8znQlo+Nc2+k2Yo5Hxl4jDGJGT9bxgbQ+0RqmBN3AJrVml
 CJY8/VyFMhUUfmweNWlGY3h5o2B6fefKyH7ilVyBZQn/DnF0yf9IdALumkheB8yaZxsldrVj
 Kn74lo5CHh7ZibCUEOKS9jpV5RCIVbIS7wJqcw4nvIRO8MsJWdrDQllZFKK3nCFraTfufpXB
 HtvSu71VSxyIf0+lFKeHr5BuZd2lnFW7T6CHvjTkkX4uYdykVbIEN/pxnPVNbtnhE5FyS2Im
 +ti2zyilkwHCbakM3CNmWPRRHhTRUUG6VnNg5U/XsaIIxZ8GXFnDPnUwLg7fJdikbgTneDNl
 kxRkGcBoLYjrRUr8Tm3V00=
IronPort-HdrOrdr: A9a23:JNsrOK9qY9tWxR4HqUduk+E6db1zdoMgy1knxilNoENuHfBwxv
 rDoB1E73LJYVYqOU3Jmbi7Sc69qFfnhORICO4qTMqftWjdyRCVxeRZg7cKrAeQeREWmtQtsJ
 uINpIOdOEYbmIK/PoSgjPIaurIqePvmMvD5Za8854ud3ATV0gJ1XYGNu/xKDwReOApP+tcKH
 LKjfA32AZINE5nJviTNz0gZazuttfLnJXpbVovAAMm0hCHiXeN5KThGxaV8x8CW3cXqI1Su1
 Ttokjc3OGOovu7whjT2yv66IlXosLozp9mCNaXgsYYBz3wgkKDZZhnWZeFoDcpydvfo2oCoZ
 3pmVMNLs5z43TeciWcpgbs4RDp1HIU53rr2Taj8AzeiP28YAh/J9tKhIpffBecwVEnpstA3K
 VC2H/cn4ZLDDvb9R6NqOTgZlVPrA6ZsHAimekcgzh0So0FcoJcqoQZ4Qd8DIoAJiTn84oqed
 MeQP003MwmMG9yUkqp/lWGmLeXLzcO91a9MwU/U/WuonZrdCsT9Tpb+CQd9k1wgK7VBaM0ot
 gsCZ4Y542mfvVmHZ6VO91xM/dfKla9Ny4kY1jiaGgOKsk8SgfwQtjMkfEI2N0=
X-IronPort-AV: E=Sophos;i="5.90,214,1643691600"; 
   d="scan'208";a="67263878"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=CynrBxGIDrAbVvIXSdHzrBUvVPtk6IFBg4oAiKNIzwn2zNFa9QRZ2sU3VNTyENy5rfPRnrhKYOcmREzXvNQuioX8Yle1fVALWfk40PryBVWoyLPN02iz1m1TVkeXKoFR5Sdhh74c6BwzmlzFLmsIGry3jXgdXm7OdCEDj+/rDRhRT1yIDTDte8Hjev/3tbvONEdInuf6u0Uk/pX02Va6Jj8ReVRNCaLVRSDKYuqykn8mPNRH3ah5yrINGT0ASsgxZ42H6En6Rum8A0yHn6igRlWIbmofvY0bLarPP6s+U3WYXWN349e13YALIv/YQS/qnP5o4bAcAl+N92n2Xx5KaQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=phK1VtnyV6Q2W2ongc6zZ1VKNl6MB4ofLvHZ+OMp2J0=;
 b=i8Xf/NcVNQUd1b344OgvCPMgj2FcgWupdJd+RRn7VCes3Y0ON5pfmb9QQ5cZnVKgIw0H6SJ1rJ/o/PEi7bB63Su8kSyyCZ2kCF7T7kxI+eu4/W5SpJJP+lTQZA6PDJ2+Jizlp/R5f3hAEOH15zGS6DudmsONoBQU4kABboxLsaUSflAOys+cm2Dzy0Oc7vP460iey470ijtVqIkG5zDnlrfJ0CoSdNeNNmk0FmuGFra9qcnwzHh9DSNSqFSBk+DSVaZWwY6gSoHbOEp4F2ke/3nJBSpdjbfrZz+VOS+xe12I0mqe9+mbMStKUKQ7jUWqzpUO1sn9jYWkb7JntshXSQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=phK1VtnyV6Q2W2ongc6zZ1VKNl6MB4ofLvHZ+OMp2J0=;
 b=Xw7RhY0aGpTsHXF3Eg5A14WyKJD3Jb8HHL3dkWSA2YNnm8mmPebAeR+Sh91iSPOa1cIMrp96uvWISMkjA460xnysVnjGSY7koh3I7gQ+ze+zgcgMa45eWxbbsIPUdjE+KZFiFInJXcIO/pElLaMeWUT4ZLcZcRboeuqlSAZZR0g=
Date: Sun, 27 Mar 2022 11:13:42 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 
 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
X-ClientProxiedBy: MR2P264CA0070.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:500:31::34) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: e974ca05-fe96-47d3-7904-08da0fd219e9
X-MS-TrafficTypeDiagnostic: MN2PR03MB4829:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: R2Q7ZKnhae89zKCqCwKWWC7AMSOr6kzflfOgU51xrykLMbxCPzndQiDrRu7Z4ZoQS4zIWpDt3CrHYPpwmtRSOvvb92GPvH7xhPuMpRh49OhJR6xZI9jcQKwWsYsMu0Q1F/Bx0rwaTg2mfXKLd9rumTntpBMu1++XuvJi51OYGeKxUEXIvhAKtRtEQ84sUaLd0oCnq3sF8zYhvWultC8IBloxlv7PWDzH/JSzTYkUiUQT/uVdjiKmu7q8rnMiC4PEeDTn1T6NKinFfkOkYfPf2LXblOzt6qevpi+GBM2oNh/JFimogPvvkyhw7nqDptwcfVfGN+eWrXNRMIwwOuUyuNwVnC/yeT+DfY8dNcorj8bEv2Muv3LRjUMglXbczSbi+To5U5o2o+eaN3MIhRCqwiDslhSZJA4HxzgVbYJc53BgACY8rBqa6xHkxcHexExKsXYUgUh4EjSuNAO5XKp7rT4kzCzksUK+TfczTWTHBDwbSyA1YcDRlpGDoSYMiQCjSCMnTPuZQXtC4iAUipeQnJDwrKF073H2dR3yxe7gvUNKdNzWQhah2EmwduZCZDPGywEHAo5+h8B79uOrpGw/0UTo9b+zPkd4qHWY5cmFB9u3MqUVvx4l7oDbVLFilia3tqLheKGogsilYAkF8ZRBeA==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(83380400001)(6486002)(508600001)(186003)(2906002)(33716001)(8936002)(86362001)(6512007)(6506007)(30864003)(6916009)(82960400001)(316002)(9686003)(38100700002)(85182001)(6666004)(26005)(4326008)(66556008)(66476007)(5660300002)(66946007)(8676002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bnJoK2JtT09wVmdqT3B2QktYSFFaQTQ3bXhoM2hrdlBGanBOZktXSFpCZTZU?=
 =?utf-8?B?Qld0YzB5ZmNuUlNOMUFHVXJ3VG1vcVBDaDMzekVNY0puelNYSjRSZlkySWJ1?=
 =?utf-8?B?QnplTTE4c0FRR0pjZWpHZ0dUbVcwN29scDJIR3lsSmcySU5nM0F1WmFKTjgw?=
 =?utf-8?B?NWdOZzlKNVJ4azg5K1FBYzBqaWdSayswTGFodEZlZGRodi9pMHhoS3V2WVV5?=
 =?utf-8?B?VWxDd0YzajhlTU45NlBBV0Q5MGFtVjlabTdOYWtta3ViSjQwNXVZelF0YlVv?=
 =?utf-8?B?NWNPTC9zMy9LNkhZZEp1NXBRaTN2UzlwZ1BPdlk0dmRJRExFS2JLYUxhbmdH?=
 =?utf-8?B?cU8zKzc3UjgwTW13UkZxOXRzQmlGR21HelJMc0ZZV0g0SWJjYmx1OENFdHNk?=
 =?utf-8?B?dlNNRklvMXpoMlJCUG1SekFJcEUzSDRscFN4cDkySWdzS1pOeVd3L0dJYVEv?=
 =?utf-8?B?VEdIRUVmU1EzaHVmREFqTGV3Z0g0bllIU1FwbzFNMUZpak5QOVBCWVc4QU03?=
 =?utf-8?B?OE1rSkttaURhTE16Z1FWa0Q4R3k0V2o2WkZaaXdEMXJpbmlNNzJoZ1dkSCtG?=
 =?utf-8?B?NHBIRXZQYkJWZkt4cHEvaGRCSkF3SVJHeXhSUjYxVHJjdjZLQ1dVUkZBazhI?=
 =?utf-8?B?QlBpQng2VmJsMWUzb1dsdGt0OUkrQUVVMkRwM1RFTTdpZDJDWExRYzI2N1VO?=
 =?utf-8?B?bVhkczFzZmVKbmppTDVKR2htcU9BWTk3TUdvTnU3ZzZJWjUxeDh6d3NLNjFR?=
 =?utf-8?B?Uk5GdVorVjFLMThpK0oxaWxSVkMwdDFtS3FRWHlOaEo1SHNWMXpSVkpjUzBH?=
 =?utf-8?B?THdaY3BBMVRKd2YvWGp0VTZOMnRpTVpERmxRZjQ1d2pja2lZejhVSmsyV2hQ?=
 =?utf-8?B?WHpwWlR5Q01zNGtubmhjTFNyVTdHZGQ4bTNFMTBXQ3g2bVl4dW1tdzNhZTVJ?=
 =?utf-8?B?UWF1OWlVcXZFYnQvSVM0c1R3ditESi96b1k2dkFpVW1GY1RLdGF0TXJucm5l?=
 =?utf-8?B?U1BoWWJVWlg5RzBRN2d2cG02WDFxYXFKanNSNTVEOE4wd0FQU2pBYklyNk1h?=
 =?utf-8?B?aUdVd3JOK3lBUWh5b0U0MWxPVmc2L1lob01yNVlPOU5XVjB2aTJUcjZNcU5Y?=
 =?utf-8?B?QjZtZ0s1WDYyN3F5SzV4UXMyV1RCNCtPb1pmclVlNjl4Q0FuSVNBV0JhMUNR?=
 =?utf-8?B?dldieGpVSXFTWmdTVlI5bXROZzhlQVhQUmlDU0szUWEzY2p1Z2dpZ21XRG5H?=
 =?utf-8?B?OTRuYnFIWU95QnlrUEJ1clJJcTNpTG5jeDRad2RIemZiaDZlOWdzR29GMFF2?=
 =?utf-8?B?SmtFeE9vaXU4eStjY0JvQVAvais5aEVtdXFzU24vQTg5alRkbEdMSUwrZGJn?=
 =?utf-8?B?dktMQWw5bEMrTHNFbXlxVmgrYXU5YWQyUXBBNnhORFFENzhuVTdEdm16UFRy?=
 =?utf-8?B?R1l4YzIzVUxYbnR3akI4NW43YzdTcFpQMHhNSXFUWkhMWkZlcGlyRTFIQ092?=
 =?utf-8?B?WWl5MVpxZnNjQ2FPZFdEbnBXRG1nbEhUM1ZFamxCNTd5T0ltM2dENlZBR3lp?=
 =?utf-8?B?aUZOL1FHVVQzV240QlM1cEl3TCtXMHNEdVdTNW5lU2xWSWw0SWJFUzJ4TTNM?=
 =?utf-8?B?S3hsNUM5WGQvMEVhRHJ5cUlGUVRsc2tzSGtyQlkrSVg2TnJtaEZhSDA5bnJB?=
 =?utf-8?B?R2Z5RnRGL1pOa1I3dGlvOXIxbW80eEl2dkF0Mkg5ZXZEYzZrekIxa3BKaFhL?=
 =?utf-8?B?c1l0V2lQWWgxdjlnVFlXd0VUOTFsMEltdzRoVEJoNFFSRjA1NnYzNlFKNkEv?=
 =?utf-8?B?U29rQUpiMWVtWmQ2Q20zcEZmR0EyZjNzeVNKMUFmTWxwQ05kcXJwTjJaeWNl?=
 =?utf-8?B?UldJakZhdzY5cjBKK0dPdjhQUHY0Vzh3eUJDRGx3MlNFSHRYTnAxWGk0TE9L?=
 =?utf-8?B?SkJzb1QwT2pTbW5oYnpyMXd2Z2c4UGkwNlFWL1ZKYm9iM21lY2tzYnZCUUJ3?=
 =?utf-8?B?ZGx1UmpCNnRQcmg3SVF4WDYvRlNUVFFCYno3TzltemwwSG5OMjA5N3BpZjhY?=
 =?utf-8?B?d01uOEZqdEtKWXhhZGFYb0ZHeTQvMndrT1d0cWlNVXEvMDNEbVZhaysrNHNw?=
 =?utf-8?B?VUorR3BabDhqZXpmNDZrb2hhcDBiRjNoVzJiMFRkQlhJcFJPMlBjWUZlVFk1?=
 =?utf-8?B?THRlYVRUMVVmRWhLc1YvR0FqZ0dvdWl5T0taWnRDeE1rZFdhR2taSjRJcjJi?=
 =?utf-8?B?WGN6ZlNxUy8zSjhHSldLRFVyVnp2Y2lVRlQ5aEc0ZWd6cllOTnN6VVYyMG9u?=
 =?utf-8?B?OHR0bzJBeW1lOGwyd2dWbEhEQ1NDMDdqdk43SzlpT05HSW1mc0tVQT09?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e974ca05-fe96-47d3-7904-08da0fd219e9
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2022 09:13:48.1750
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: B4pncfJ0dgMRzPUUmv/83Sf0xp60rIyiddiyG+flEtvI2c2gBipL+Hz5lYulKEUA0luDViENgaT7lA0Iva+qRA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR03MB4829
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KR99y0Y9Wz3Bwf
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=cHWaYBQ2;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=Xw7RhY0a;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=0786bbb88=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=0786bbb88=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.61 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-0.96)[-0.963];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=0786bbb88=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sun, Mar 27, 2022 at 12:38:00AM +0200, Ze Dupsys wrote:
> On 2022.03.26. 16:38, Roger Pau Monné wrote:
> > ..
> > It's weird, because here you get a page fault, but there are also
> > traces with:
> > ..
> > general protection fault while in kernel mode
> > ..
> > That show a general protection fault instead of a page fault.
> 
> Yes indeed, i had not noticed this. Grepped across 34 stored panic log
> files, i see that 28 are page fault, 4 are general protection fault, 2
> other. I though maybe RAM size influences this, but page faults have 2G, 4G,
> 6G, 8G Dom0, general protection faults have 2G, 4G, 8G.
> 
> I have no idea what triggers what, since stress tests and command line args
> are more or less the same. Builds are different with patches, some debug
> info, etc. Almost all panic traces have "rman_is_region_manager" in mid,
> actually looking all of them together seemed interesting. I'll attach unique
> panic traces, since some included snprintf, kvprintf as well, maybe helpful.
> Unfortunately i do not know which version and what patches were applied.
> 
> 
> > I've also noticed it seems to always be 'devmatch' the process that
> > triggers the panic.
> 
> Yes, it seems to be the case most of the time. There are 3 cases when
> process is "xbbd* taskq". 2 cases with 2G RAM, 1 with 6G.
> 
> 
> > I've been able to get a better trace with gdb and your debug symbols,
> > and this is:
> >
> > (gdb) info line *0xffffffff80c6a2b2
> > Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address
> 0xffffffff80c6a2b2 
> >     and ends at 0xffffffff80c6a2b6 .
> > (gdb) info line *0xffffffff80c86ed1
> > Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address
> 0xffffffff80c86ecc 
> >     and ends at 0xffffffff80c86ed5 .
> 
> This is a nice find!
> 
> 
> > I'm trying to figure out how the device could be removed or
> > disconnected from the rman. I will try to create a patch to catch the
> > device that leaves rman regions when destroyed/removed.
> 
> Okay, i'll apply when it will be possible.
> 
> I did run xen-debug on system with applied blkback.patch as you sent in next
> message to this.
> 
> System had panic with new trace:
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 04
> fault virtual address	= 0xa4
> fault code		= supervisor read data, page not present
> instruction pointer	= 0x20:0xffffffff80c90ed0
> stack pointer	        = 0x28:0xfffffe0051927ab0
> frame pointer	        = 0x28:0xfffffe0051927ad0
> code segment		= base 0x0, limit 0xfffff, type 0x1b
> 			= DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags	= interrupt enabled, resume, IOPL = 0
> current process		= 16 (xenwatch)
> trap number		= 12
> panic: page fault
> cpuid = 1
> time = 1648331592
> KDB: stack backtrace:
> #0 0xffffffff80c7c275 at kdb_backtrace+0x65
> #1 0xffffffff80c2e2d1 at vpanic+0x181
> #2 0xffffffff80c2e143 at panic+0x43
> #3 0xffffffff810c8b97 at trap+0xba7
> #4 0xffffffff810c8bef at trap+0xbff
> #5 0xffffffff810c8243 at trap+0x253
> #6 0xffffffff810a0838 at calltrap+0x8
> #7 0xffffffff80a98515 at xbd_instance_create+0x7895
> #8 0xffffffff80a98462 at xbd_instance_create+0x77e2
> #9 0xffffffff80a9619b at xbd_instance_create+0x551b
> #10 0xffffffff80f95c54 at xenbusb_localend_changed+0x7c4
> #11 0xffffffff80ab0ef4 at xs_unlock+0x704
> #12 0xffffffff80beaede at fork_exit+0x7e
> #13 0xffffffff810a18ae at fork_trampoline+0xe
> 
> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line
> -e /usr/lib/debug/boot/kernel/kernel.debug
> 
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:844
> /usr/src/sys/amd64/amd64/trap.c:944
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/exception.S:292
> /usr/src/sys/dev/xen/blkback/blkback.c:2789
> /usr/src/sys/dev/xen/blkback/blkback.c:3431
> /usr/src/sys/dev/xen/blkback/blkback.c:3912
> /usr/src/sys/xen/xenbus/xenbusb_back.c:238
> /usr/src/sys/dev/xen/xenstore/xenstore.c:1007
> /usr/src/sys/kern/kern_fork.c:1099
> /usr/src/sys/amd64/amd64/exception.S:1091

Thanks, unfortunately that patch was incomplete. I have an updated
version that I think is better now, and I've slightly tested it
(creating and destroying a domain with it doesn't seem to crash).
Appended patch at the end of the message.

> 
> Full serial log in attachment.
> 
> Thanks.

> ==== COUNT: 1
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 0; apic id = 00
> instruction pointer	= 0x20:0xffffffff80c45892
> stack pointer	        = 0x28:0xfffffe00d2d2b930
> frame pointer	        = 0x28:0xfffffe00d2d2b930
> code segment		= base 0x0, limit 0xfffff, type 0x1b
> 			= DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags	= interrupt enabled, resume, IOPL = 0
> current process		= 10984 (devmatch)
> trap number		= 9
> panic: general protection fault
> cpuid = 0
> time = 1646305680
> KDB: stack backtrace:
> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> #1 0xffffffff80c09f01 at vpanic+0x181
> #2 0xffffffff80c09d73 at panic+0x43
> #3 0xffffffff8108b1a7 at trap+0xbc7
> #4 0xffffffff8108a66e at trap+0x8e
> #5 0xffffffff81061b18 at calltrap+0x8
> #6 0xffffffff80c62011 at rman_is_region_manager+0x241
> #7 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> #8 0xffffffff80c1949c at kernel_sysctl+0x43c
> #9 0xffffffff80c19b13 at userland_sysctl+0x173
> #10 0xffffffff80c1995f at sys___sysctl+0x5f
> #11 0xffffffff8108baac at amd64_syscall+0x10c
> #12 0xffffffff8106243e at Xfast_syscall+0xfe
> Uptime: 1h15m46s
> 
> 
> ==== COUNT: 3
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 0; apic id = 00
> instruction pointer	= 0x20:0xffffffff80d0728f
> stack pointer	        = 0x28:0xfffffe00a17ea790
> frame pointer	        = 0x28:0xfffffe00a17ea790
> code segment		= base 0x0, limit 0xfffff, type 0x1b
> 			= DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags	= interrupt enabled, resume, IOPL = 0
> current process		= 2785 (devmatch)
> trap number		= 9
> panic: general protection fault
> cpuid = 1
> time = 1646419029
> KDB: stack backtrace:
> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> #1 0xffffffff80c09f01 at vpanic+0x181
> #2 0xffffffff80c09d73 at panic+0x43
> #3 0xffffffff8108b1a7 at trap+0xbc7
> #4 0xffffffff8108a66e at trap+0x8e
> #5 0xffffffff81061b18 at calltrap+0x8
> #6 0xffffffff80c5da17 at kvprintf+0x1007
> #7 0xffffffff80c5e719 at snprintf+0x59
> #8 0xffffffff80c6204b at rman_is_region_manager+0x27b
> #9 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> #10 0xffffffff80c1949c at kernel_sysctl+0x43c
> #11 0xffffffff80c19b13 at userland_sysctl+0x173
> #12 0xffffffff80c1995f at sys___sysctl+0x5f
> #13 0xffffffff8108baac at amd64_syscall+0x10c
> #14 0xffffffff8106243e at Xfast_syscall+0xfe

> Unique on "current process" and trace fingerprint #0-#*.
> 
> ==== COUNT: 23
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x22710028
> fault code    = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff80c45892
> stack pointer         = 0x28:0xfffffe0096600930
> frame pointer         = 0x28:0xfffffe0096600930
> code segment    = base 0x0, limit 0xfffff, type 0x1b
>       = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags  = interrupt enabled, resume, IOPL = 0
> current process   = 1496 (devmatch)
> trap number   = 12
> panic: page fault
> cpuid = 0
> time = 1646123791
> KDB: stack backtrace:
> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> #1 0xffffffff80c09f01 at vpanic+0x181
> #2 0xffffffff80c09d73 at panic+0x43
> #3 0xffffffff8108b1a7 at trap+0xbc7
> #4 0xffffffff8108b1ff at trap+0xc1f
> #5 0xffffffff8108a85d at trap+0x27d
> #6 0xffffffff81061b18 at calltrap+0x8
> #7 0xffffffff80c62011 at rman_is_region_manager+0x241
> #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> #9 0xffffffff80c1949c at kernel_sysctl+0x43c
> #10 0xffffffff80c19b13 at userland_sysctl+0x173
> #11 0xffffffff80c1995f at sys___sysctl+0x5f
> #12 0xffffffff8108baac at amd64_syscall+0x10c
> #13 0xffffffff8106243e at Xfast_syscall+0xfe
> 
> 
> 
> ==== COUNT: 2
> current process   = 20284 (devmatch)
> trap number   = 12
> panic: page fault
> cpuid = 3
> time = 1647247618
> KDB: stack backtrace:
> #0 0xffffffff80c7c615 at kdb_backtrace+0x65
> #1 0xffffffff80c2e621 at vpanic+0x181
> #2 0xffffffff80c2e493 at panic+0x43
> #3 0xffffffff810c8b97 at trap+0xba7
> #4 0xffffffff810c8bef at trap+0xbff
> #5 0xffffffff810c8243 at trap+0x253
> #6 0xffffffff810a09d8 at calltrap+0x8
> #7 0xffffffff80c82c77 at kvprintf+0x1007
> #8 0xffffffff80c83a09 at snprintf+0x59
> #9 0xffffffff80c8729b at rman_is_region_manager+0x27b
> #10 0xffffffff80c3ee81 at sbuf_new_for_sysctl+0x101
> #11 0xffffffff80c3e2cc at kernel_sysctl+0x3ec
> #12 0xffffffff80c3e943 at userland_sysctl+0x173
> #13 0xffffffff80c3e78f at sys___sysctl+0x5f
> #14 0xffffffff810c949c at amd64_syscall+0x10c
> #15 0xffffffff810a12eb at Xfast_syscall+0xfb

Thanks, those all seem to be related to a device being removed without
cleaning it's rman regions properly. So far I've spotted an issue in
blkback in this regard, but I wouldn't discard other issues in either
blkback or netback. Let's see if the updated blkback patch makes a
difference now.

> ==== COUNT: 2
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 02
> fault virtual address = 0x68
> fault code    = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff824a599d
> stack pointer         = 0x28:0xfffffe00b1e27910
> frame pointer         = 0x28:0xfffffe00b1e279b0
> code segment    = base 0x0, limit 0xfffff, type 0x1b
>       = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags  = interrupt enabled, resume, IOPL = 0
> current process   = 0 (xbbd7 taskq)
> trap number   = 12
> panic: page fault
> cpuid = 1
> time = 1646122723
> KDB: stack backtrace:
> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> #1 0xffffffff80c09f01 at vpanic+0x181
> #2 0xffffffff80c09d73 at panic+0x43
> #3 0xffffffff8108b1a7 at trap+0xbc7
> #4 0xffffffff8108b1ff at trap+0xc1f
> #5 0xffffffff8108a85d at trap+0x27d
> #6 0xffffffff81061b18 at calltrap+0x8
> #7 0xffffffff8248935a at dmu_read+0x2a
> #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa
> #9 0xffffffff80a7f214 at xbd_instance_create+0xa394
> #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> #13 0xffffffff80bc7e3e at fork_exit+0x7e
> #14 0xffffffff81062b9e at fork_trampoline+0xe
> 
> 
> ==== COUNT: 1
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 02
> fault virtual address = 0x148
> fault code    = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff8248cef4
> stack pointer         = 0x28:0xfffffe009941d9a0
> frame pointer         = 0x28:0xfffffe009941d9a0
> code segment    = base 0x0, limit 0xfffff, type 0x1b
>       = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags  = interrupt enabled, resume, IOPL = 0
> current process   = 0 (xbbd1 taskq)
> trap number   = 12
> panic: page fault
> cpuid = 1
> time = 1646129773
> KDB: stack backtrace:
> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> #1 0xffffffff80c09f01 at vpanic+0x181
> #2 0xffffffff80c09d73 at panic+0x43
> #3 0xffffffff8108b1a7 at trap+0xbc7
> #4 0xffffffff8108b1ff at trap+0xc1f
> #5 0xffffffff8108a85d at trap+0x27d
> #6 0xffffffff81061b18 at calltrap+0x8
> #7 0xffffffff825cb76e at zil_open+0xe
> #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2
> #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88
> #10 0xffffffff80a7f214 at xbd_instance_create+0xa394
> #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> #14 0xffffffff80bc7e3e at fork_exit+0x7e
> #15 0xffffffff81062b9e at fork_trampoline+0xe

Hm, those last ones are in ZFS code, can you try to get the line
numbers for those?

Maybe it's blkback providing bad data to the disk open functions.

Since you are doing so much testing, it might make sense for you to
use a debug FreeBSD kernel rather than a production one (one with
WITNESS and INVARIANTS enabled).

Thanks, Roger.

---8<---
diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 33414295bf5e..4007a93a54c7 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -2774,19 +2774,12 @@ xbb_free_communication_mem(struct xbb_softc *xbb)
 static int
 xbb_disconnect(struct xbb_softc *xbb)
 {
-	struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
-	struct gnttab_unmap_grant_ref *op;
-	u_int			       ring_idx;
-	int			       error;
-
 	DPRINTF("\n");
 
-	if ((xbb->flags & XBBF_RING_CONNECTED) == 0)
-		return (0);
-
 	mtx_unlock(&xbb->lock);
 	xen_intr_unbind(&xbb->xen_intr_handle);
-	taskqueue_drain(xbb->io_taskqueue, &xbb->io_task); 
+	if (xbb->io_taskqueue != NULL)
+		taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
 	mtx_lock(&xbb->lock);
 
 	/*
@@ -2796,19 +2789,28 @@ xbb_disconnect(struct xbb_softc *xbb)
 	if (xbb->active_request_count != 0)
 		return (EAGAIN);
 
-	for (ring_idx = 0, op = ops;
-	     ring_idx < xbb->ring_config.ring_pages;
-	     ring_idx++, op++) {
-		op->host_addr    = xbb->ring_config.gnt_addr
-			         + (ring_idx * PAGE_SIZE);
-		op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
-		op->handle	 = xbb->ring_config.handle[ring_idx];
-	}
+	if (xbb->flags & XBBF_RING_CONNECTED) {
+		struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
+		struct gnttab_unmap_grant_ref *op;
+		unsigned int ring_idx;
+		int error;
+
+		for (ring_idx = 0, op = ops;
+		     ring_idx < xbb->ring_config.ring_pages;
+		     ring_idx++, op++) {
+			op->host_addr    = xbb->ring_config.gnt_addr
+				         + (ring_idx * PAGE_SIZE);
+			op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
+			op->handle	 = xbb->ring_config.handle[ring_idx];
+		}
 
-	error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
-					  xbb->ring_config.ring_pages);
-	if (error != 0)
-		panic("Grant table op failed (%d)", error);
+		error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
+						  xbb->ring_config.ring_pages);
+		if (error != 0)
+			panic("Grant table op failed (%d)", error);
+
+		xbb->flags &= ~XBBF_RING_CONNECTED;
+	}
 
 	xbb_free_communication_mem(xbb);
 
@@ -2839,7 +2841,6 @@ xbb_disconnect(struct xbb_softc *xbb)
 		xbb->request_lists = NULL;
 	}
 
-	xbb->flags &= ~XBBF_RING_CONNECTED;
 	return (0);
 }
 

From nobody Sun Mar 27 17:42:23 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B22071A4E0D9
	for ; Sun, 27 Mar 2022 17:43:23 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KRNTk5XpRz3srJ
	for ; Sun, 27 Mar 2022 17:43:22 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-pg1-x529.google.com with SMTP id b130so9081492pga.13
        for ; Sun, 27 Mar 2022 10:43:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc:content-transfer-encoding;
        bh=5wVN5/7B0s9ldZcGbVzg1Z+JYnJm3dRNi2jZ1mgq3JQ=;
        b=Wj+ewkoSOq2DsK8/QXZxBzZNV4U6HTZrWdW1hwe8MLmt/p44DDzkBrY+NfUlZaSpv8
         euUuB54D85yc79nqZxrWaApRSsMyRkdRtaPhZl5udpYIxovyuXgjKRkwLNsPaZHm8vlx
         GS9j5Kovv4yVA/peR5ix6bHV6H5m1ro15R2Um9rbtLH0dkN6qRaOklTicLho9Bx4hOPz
         jbs6ZGIiHXCciWwNOcNm5JCeFO+YNisq0lKKUf1RkoqVZsY2Upsusvi90GUhcBW/vspv
         2QMf8cATlchOB2DH8Ry+mcKKao9zZNY3WXQOyf73C8QWsP8QdHDsUkSgcO+YXcyPPNbl
         H+pQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc:content-transfer-encoding;
        bh=5wVN5/7B0s9ldZcGbVzg1Z+JYnJm3dRNi2jZ1mgq3JQ=;
        b=IC+4Hir91wwSFe14YhRxEqc82fKueOoERKqeUw0qKA8xYt0qLcv64Gh84PhT6X3BQa
         Fp3bV+inty8n5XnPJGr9O1f1zZAnr5kuPXuvLDCZcURt0a8XhlTJbBw6XxrzQs5oq48A
         L2lBKHxIbSxKfsMxnQQ8aKzDThTtvxFMRgb+QcfjWt6fJebqmUxli2N7OqM0k3mJm8gg
         H9+keHyvTDO08Q11KVvKnfZtK6n45fl3+GEv1HsYK6nkSxuVJjbkEv7XNJgTm7gHlyBe
         umcx8J70OBHz1ifN/8KoPkLgdmnEeRa7e1C7O89CBRN3/BlnU0iW0XWGCaoDyKZNkUzq
         TVSA==
X-Gm-Message-State: AOAM5301OnnugNZWbu8lw/lrhqArcBG5RDeZnUD7Vu9YtCVl+g610i/u
	JI/wW6OCRNxFUd8d3TXMS6c8PiLKPv4m6Pp2xkg=
X-Google-Smtp-Source: ABdhPJw0qaZTCDMwRKvVy28BQtnWDUPlaVJJP4K+mZ/+/HwdP/53EtlioZCMrCImRh1O0w1+gGDTC0QGUNO3emeugpM=
X-Received: by 2002:a63:2a04:0:b0:398:5225:8982 with SMTP id
 q4-20020a632a04000000b0039852258982mr554840pgq.394.1648402995517; Sun, 27 Mar
 2022 10:43:15 -0700 (PDT)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
References:  
  <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
  <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
  <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
  <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
In-Reply-To: 
From: Ze Dupsys 
Date: Sun, 27 Mar 2022 20:42:23 +0300
Message-ID: 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= 
Cc: freebsd-xen@freebsd.org, Brian Buhrow 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Rspamd-Queue-Id: 4KRNTk5XpRz3srJ
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=Wj+ewkoS;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::529 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-4.00 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 MIME_GOOD(-0.10)[text/plain];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 TO_DN_SOME(0.00)[];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MID_RHS_MATCH_FROMTLD(0.00)[];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::529:from];
	 NEURAL_HAM_SHORT(-1.00)[-0.996];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim]
X-ThisMailContainsUnwantedMimeParts: N

On Sun, Mar 27, 2022 at 12:13 PM Roger Pau Monn=C3=A9  wrote:
>
> On Sun, Mar 27, 2022 at 12:38:00AM +0200, Ze Dupsys wrote:
> > On 2022.03.26. 16:38, Roger Pau Monn=C3=A9 wrote:
> > > ..
> > > It's weird, because here you get a page fault, but there are also
> > > traces with:
> > > ..
> > > general protection fault while in kernel mode
> > > ..
> > > That show a general protection fault instead of a page fault.
> >
> > Yes indeed, i had not noticed this. Grepped across 34 stored panic log
> > files, i see that 28 are page fault, 4 are general protection fault, 2
> > other. I though maybe RAM size influences this, but page faults have 2G=
, 4G,
> > 6G, 8G Dom0, general protection faults have 2G, 4G, 8G.
> >
> > I have no idea what triggers what, since stress tests and command line =
args
> > are more or less the same. Builds are different with patches, some debu=
g
> > info, etc. Almost all panic traces have "rman_is_region_manager" in mid=
,
> > actually looking all of them together seemed interesting. I'll attach u=
nique
> > panic traces, since some included snprintf, kvprintf as well, maybe hel=
pful.
> > Unfortunately i do not know which version and what patches were applied=
.
> >
> >
> > > I've also noticed it seems to always be 'devmatch' the process that
> > > triggers the panic.
> >
> > Yes, it seems to be the case most of the time. There are 3 cases when
> > process is "xbbd* taskq". 2 cases with 2G RAM, 1 with 6G.
> >
> >
> > > I've been able to get a better trace with gdb and your debug symbols,
> > > and this is:
> > >
> > > (gdb) info line *0xffffffff80c6a2b2
> > > Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address
> > 0xffffffff80c6a2b2 
> > >     and ends at 0xffffffff80c6a2b6 .
> > > (gdb) info line *0xffffffff80c86ed1
> > > Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address
> > 0xffffffff80c86ecc 
> > >     and ends at 0xffffffff80c86ed5 .
> >
> > This is a nice find!
> >
> >
> > > I'm trying to figure out how the device could be removed or
> > > disconnected from the rman. I will try to create a patch to catch the
> > > device that leaves rman regions when destroyed/removed.
> >
> > Okay, i'll apply when it will be possible.
> >
> > I did run xen-debug on system with applied blkback.patch as you sent in=
 next
> > message to this.
> >
> > System had panic with new trace:
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 2; apic id =3D 04
> > fault virtual address =3D 0xa4
> > fault code            =3D supervisor read data, page not present
> > instruction pointer   =3D 0x20:0xffffffff80c90ed0
> > stack pointer         =3D 0x28:0xfffffe0051927ab0
> > frame pointer         =3D 0x28:0xfffffe0051927ad0
> > code segment          =3D base 0x0, limit 0xfffff, type 0x1b
> >                       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags      =3D interrupt enabled, resume, IOPL =3D 0
> > current process               =3D 16 (xenwatch)
> > trap number           =3D 12
> > panic: page fault
> > cpuid =3D 1
> > time =3D 1648331592
> > KDB: stack backtrace:
> > #0 0xffffffff80c7c275 at kdb_backtrace+0x65
> > #1 0xffffffff80c2e2d1 at vpanic+0x181
> > #2 0xffffffff80c2e143 at panic+0x43
> > #3 0xffffffff810c8b97 at trap+0xba7
> > #4 0xffffffff810c8bef at trap+0xbff
> > #5 0xffffffff810c8243 at trap+0x253
> > #6 0xffffffff810a0838 at calltrap+0x8
> > #7 0xffffffff80a98515 at xbd_instance_create+0x7895
> > #8 0xffffffff80a98462 at xbd_instance_create+0x77e2
> > #9 0xffffffff80a9619b at xbd_instance_create+0x551b
> > #10 0xffffffff80f95c54 at xenbusb_localend_changed+0x7c4
> > #11 0xffffffff80ab0ef4 at xs_unlock+0x704
> > #12 0xffffffff80beaede at fork_exit+0x7e
> > #13 0xffffffff810a18ae at fork_trampoline+0xe
> >
> > cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2l=
ine
> > -e /usr/lib/debug/boot/kernel/kernel.debug
> >
> > /usr/src/sys/kern/subr_kdb.c:443
> > /usr/src/sys/kern/kern_shutdown.c:0
> > /usr/src/sys/kern/kern_shutdown.c:844
> > /usr/src/sys/amd64/amd64/trap.c:944
> > /usr/src/sys/amd64/amd64/trap.c:0
> > /usr/src/sys/amd64/amd64/trap.c:0
> > /usr/src/sys/amd64/amd64/exception.S:292
> > /usr/src/sys/dev/xen/blkback/blkback.c:2789
> > /usr/src/sys/dev/xen/blkback/blkback.c:3431
> > /usr/src/sys/dev/xen/blkback/blkback.c:3912
> > /usr/src/sys/xen/xenbus/xenbusb_back.c:238
> > /usr/src/sys/dev/xen/xenstore/xenstore.c:1007
> > /usr/src/sys/kern/kern_fork.c:1099
> > /usr/src/sys/amd64/amd64/exception.S:1091
>
> Thanks, unfortunately that patch was incomplete. I have an updated
> version that I think is better now, and I've slightly tested it
> (creating and destroying a domain with it doesn't seem to crash).
> Appended patch at the end of the message.
>
> >
> > Full serial log in attachment.
> >
> > Thanks.
>
> > =3D=3D=3D=3D COUNT: 1
> > Fatal trap 9: general protection fault while in kernel mode
> > cpuid =3D 0; apic id =3D 00
> > instruction pointer   =3D 0x20:0xffffffff80c45892
> > stack pointer         =3D 0x28:0xfffffe00d2d2b930
> > frame pointer         =3D 0x28:0xfffffe00d2d2b930
> > code segment          =3D base 0x0, limit 0xfffff, type 0x1b
> >                       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags      =3D interrupt enabled, resume, IOPL =3D 0
> > current process               =3D 10984 (devmatch)
> > trap number           =3D 9
> > panic: general protection fault
> > cpuid =3D 0
> > time =3D 1646305680
> > KDB: stack backtrace:
> > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > #1 0xffffffff80c09f01 at vpanic+0x181
> > #2 0xffffffff80c09d73 at panic+0x43
> > #3 0xffffffff8108b1a7 at trap+0xbc7
> > #4 0xffffffff8108a66e at trap+0x8e
> > #5 0xffffffff81061b18 at calltrap+0x8
> > #6 0xffffffff80c62011 at rman_is_region_manager+0x241
> > #7 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> > #8 0xffffffff80c1949c at kernel_sysctl+0x43c
> > #9 0xffffffff80c19b13 at userland_sysctl+0x173
> > #10 0xffffffff80c1995f at sys___sysctl+0x5f
> > #11 0xffffffff8108baac at amd64_syscall+0x10c
> > #12 0xffffffff8106243e at Xfast_syscall+0xfe
> > Uptime: 1h15m46s
> >
> >
> > =3D=3D=3D=3D COUNT: 3
> > Fatal trap 9: general protection fault while in kernel mode
> > cpuid =3D 0; apic id =3D 00
> > instruction pointer   =3D 0x20:0xffffffff80d0728f
> > stack pointer         =3D 0x28:0xfffffe00a17ea790
> > frame pointer         =3D 0x28:0xfffffe00a17ea790
> > code segment          =3D base 0x0, limit 0xfffff, type 0x1b
> >                       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags      =3D interrupt enabled, resume, IOPL =3D 0
> > current process               =3D 2785 (devmatch)
> > trap number           =3D 9
> > panic: general protection fault
> > cpuid =3D 1
> > time =3D 1646419029
> > KDB: stack backtrace:
> > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > #1 0xffffffff80c09f01 at vpanic+0x181
> > #2 0xffffffff80c09d73 at panic+0x43
> > #3 0xffffffff8108b1a7 at trap+0xbc7
> > #4 0xffffffff8108a66e at trap+0x8e
> > #5 0xffffffff81061b18 at calltrap+0x8
> > #6 0xffffffff80c5da17 at kvprintf+0x1007
> > #7 0xffffffff80c5e719 at snprintf+0x59
> > #8 0xffffffff80c6204b at rman_is_region_manager+0x27b
> > #9 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> > #10 0xffffffff80c1949c at kernel_sysctl+0x43c
> > #11 0xffffffff80c19b13 at userland_sysctl+0x173
> > #12 0xffffffff80c1995f at sys___sysctl+0x5f
> > #13 0xffffffff8108baac at amd64_syscall+0x10c
> > #14 0xffffffff8106243e at Xfast_syscall+0xfe
>
> > Unique on "current process" and trace fingerprint #0-#*.
> >
> > =3D=3D=3D=3D COUNT: 23
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 0; apic id =3D 00
> > fault virtual address =3D 0x22710028
> > fault code    =3D supervisor read data, page not present
> > instruction pointer =3D 0x20:0xffffffff80c45892
> > stack pointer         =3D 0x28:0xfffffe0096600930
> > frame pointer         =3D 0x28:0xfffffe0096600930
> > code segment    =3D base 0x0, limit 0xfffff, type 0x1b
> >       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags  =3D interrupt enabled, resume, IOPL =3D 0
> > current process   =3D 1496 (devmatch)
> > trap number   =3D 12
> > panic: page fault
> > cpuid =3D 0
> > time =3D 1646123791
> > KDB: stack backtrace:
> > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > #1 0xffffffff80c09f01 at vpanic+0x181
> > #2 0xffffffff80c09d73 at panic+0x43
> > #3 0xffffffff8108b1a7 at trap+0xbc7
> > #4 0xffffffff8108b1ff at trap+0xc1f
> > #5 0xffffffff8108a85d at trap+0x27d
> > #6 0xffffffff81061b18 at calltrap+0x8
> > #7 0xffffffff80c62011 at rman_is_region_manager+0x241
> > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
> > #9 0xffffffff80c1949c at kernel_sysctl+0x43c
> > #10 0xffffffff80c19b13 at userland_sysctl+0x173
> > #11 0xffffffff80c1995f at sys___sysctl+0x5f
> > #12 0xffffffff8108baac at amd64_syscall+0x10c
> > #13 0xffffffff8106243e at Xfast_syscall+0xfe
> >
> >
> >
> > =3D=3D=3D=3D COUNT: 2
> > current process   =3D 20284 (devmatch)
> > trap number   =3D 12
> > panic: page fault
> > cpuid =3D 3
> > time =3D 1647247618
> > KDB: stack backtrace:
> > #0 0xffffffff80c7c615 at kdb_backtrace+0x65
> > #1 0xffffffff80c2e621 at vpanic+0x181
> > #2 0xffffffff80c2e493 at panic+0x43
> > #3 0xffffffff810c8b97 at trap+0xba7
> > #4 0xffffffff810c8bef at trap+0xbff
> > #5 0xffffffff810c8243 at trap+0x253
> > #6 0xffffffff810a09d8 at calltrap+0x8
> > #7 0xffffffff80c82c77 at kvprintf+0x1007
> > #8 0xffffffff80c83a09 at snprintf+0x59
> > #9 0xffffffff80c8729b at rman_is_region_manager+0x27b
> > #10 0xffffffff80c3ee81 at sbuf_new_for_sysctl+0x101
> > #11 0xffffffff80c3e2cc at kernel_sysctl+0x3ec
> > #12 0xffffffff80c3e943 at userland_sysctl+0x173
> > #13 0xffffffff80c3e78f at sys___sysctl+0x5f
> > #14 0xffffffff810c949c at amd64_syscall+0x10c
> > #15 0xffffffff810a12eb at Xfast_syscall+0xfb
>
> Thanks, those all seem to be related to a device being removed without
> cleaning it's rman regions properly. So far I've spotted an issue in
> blkback in this regard, but I wouldn't discard other issues in either
> blkback or netback. Let's see if the updated blkback patch makes a
> difference now.
>
> > =3D=3D=3D=3D COUNT: 2
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 1; apic id =3D 02
> > fault virtual address =3D 0x68
> > fault code    =3D supervisor read data, page not present
> > instruction pointer =3D 0x20:0xffffffff824a599d
> > stack pointer         =3D 0x28:0xfffffe00b1e27910
> > frame pointer         =3D 0x28:0xfffffe00b1e279b0
> > code segment    =3D base 0x0, limit 0xfffff, type 0x1b
> >       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags  =3D interrupt enabled, resume, IOPL =3D 0
> > current process   =3D 0 (xbbd7 taskq)
> > trap number   =3D 12
> > panic: page fault
> > cpuid =3D 1
> > time =3D 1646122723
> > KDB: stack backtrace:
> > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > #1 0xffffffff80c09f01 at vpanic+0x181
> > #2 0xffffffff80c09d73 at panic+0x43
> > #3 0xffffffff8108b1a7 at trap+0xbc7
> > #4 0xffffffff8108b1ff at trap+0xc1f
> > #5 0xffffffff8108a85d at trap+0x27d
> > #6 0xffffffff81061b18 at calltrap+0x8
> > #7 0xffffffff8248935a at dmu_read+0x2a
> > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa
> > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394
> > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> > #13 0xffffffff80bc7e3e at fork_exit+0x7e
> > #14 0xffffffff81062b9e at fork_trampoline+0xe
> >
> >
> > =3D=3D=3D=3D COUNT: 1
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 1; apic id =3D 02
> > fault virtual address =3D 0x148
> > fault code    =3D supervisor read data, page not present
> > instruction pointer =3D 0x20:0xffffffff8248cef4
> > stack pointer         =3D 0x28:0xfffffe009941d9a0
> > frame pointer         =3D 0x28:0xfffffe009941d9a0
> > code segment    =3D base 0x0, limit 0xfffff, type 0x1b
> >       =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags  =3D interrupt enabled, resume, IOPL =3D 0
> > current process   =3D 0 (xbbd1 taskq)
> > trap number   =3D 12
> > panic: page fault
> > cpuid =3D 1
> > time =3D 1646129773
> > KDB: stack backtrace:
> > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > #1 0xffffffff80c09f01 at vpanic+0x181
> > #2 0xffffffff80c09d73 at panic+0x43
> > #3 0xffffffff8108b1a7 at trap+0xbc7
> > #4 0xffffffff8108b1ff at trap+0xc1f
> > #5 0xffffffff8108a85d at trap+0x27d
> > #6 0xffffffff81061b18 at calltrap+0x8
> > #7 0xffffffff825cb76e at zil_open+0xe
> > #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2
> > #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88
> > #10 0xffffffff80a7f214 at xbd_instance_create+0xa394
> > #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> > #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> > #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> > #14 0xffffffff80bc7e3e at fork_exit+0x7e
> > #15 0xffffffff81062b9e at fork_trampoline+0xe
>
> Hm, those last ones are in ZFS code, can you try to get the line
> numbers for those?
>
> Maybe it's blkback providing bad data to the disk open functions.
>
> Since you are doing so much testing, it might make sense for you to
> use a debug FreeBSD kernel rather than a production one (one with
> WITNESS and INVARIANTS enabled).
>
> Thanks, Roger.
>
> ---8<---
> diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.=
c
> index 33414295bf5e..4007a93a54c7 100644
> --- a/sys/dev/xen/blkback/blkback.c
> +++ b/sys/dev/xen/blkback/blkback.c
> @@ -2774,19 +2774,12 @@ xbb_free_communication_mem(struct xbb_softc *xbb)
>  static int
>  xbb_disconnect(struct xbb_softc *xbb)
>  {
> -       struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
> -       struct gnttab_unmap_grant_ref *op;
> -       u_int                          ring_idx;
> -       int                            error;
> -
>         DPRINTF("\n");
>
> -       if ((xbb->flags & XBBF_RING_CONNECTED) =3D=3D 0)
> -               return (0);
> -
>         mtx_unlock(&xbb->lock);
>         xen_intr_unbind(&xbb->xen_intr_handle);
> -       taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
> +       if (xbb->io_taskqueue !=3D NULL)
> +               taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
>         mtx_lock(&xbb->lock);
>
>         /*
> @@ -2796,19 +2789,28 @@ xbb_disconnect(struct xbb_softc *xbb)
>         if (xbb->active_request_count !=3D 0)
>                 return (EAGAIN);
>
> -       for (ring_idx =3D 0, op =3D ops;
> -            ring_idx < xbb->ring_config.ring_pages;
> -            ring_idx++, op++) {
> -               op->host_addr    =3D xbb->ring_config.gnt_addr
> -                                + (ring_idx * PAGE_SIZE);
> -               op->dev_bus_addr =3D xbb->ring_config.bus_addr[ring_idx];
> -               op->handle       =3D xbb->ring_config.handle[ring_idx];
> -       }
> +       if (xbb->flags & XBBF_RING_CONNECTED) {
> +               struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
> +               struct gnttab_unmap_grant_ref *op;
> +               unsigned int ring_idx;
> +               int error;
> +
> +               for (ring_idx =3D 0, op =3D ops;
> +                    ring_idx < xbb->ring_config.ring_pages;
> +                    ring_idx++, op++) {
> +                       op->host_addr    =3D xbb->ring_config.gnt_addr
> +                                        + (ring_idx * PAGE_SIZE);
> +                       op->dev_bus_addr =3D xbb->ring_config.bus_addr[ri=
ng_idx];
> +                       op->handle       =3D xbb->ring_config.handle[ring=
_idx];
> +               }
>
> -       error =3D HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops=
,
> -                                         xbb->ring_config.ring_pages);
> -       if (error !=3D 0)
> -               panic("Grant table op failed (%d)", error);
> +               error =3D HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_=
ref, ops,
> +                                                 xbb->ring_config.ring_p=
ages);
> +               if (error !=3D 0)
> +                       panic("Grant table op failed (%d)", error);
> +
> +               xbb->flags &=3D ~XBBF_RING_CONNECTED;
> +       }
>
>         xbb_free_communication_mem(xbb);
>
> @@ -2839,7 +2841,6 @@ xbb_disconnect(struct xbb_softc *xbb)
>                 xbb->request_lists =3D NULL;
>         }
>
> -       xbb->flags &=3D ~XBBF_RING_CONNECTED;
>         return (0);
>  }

Hello,

I applied given patch, i did not have enough time to test thoroughly,
but for 3 hours system was running without panic whereas previously it
would crash in around 1,5 hours in similar settings. Till Thursday i
will not be able to test.

About those ZFS panic traces, i will try to get line numbers, but the
problem is that i do not have /usr/lib/debug/boot/kernel/kernel.debug
for FreeBSD 13.0-RELEASE-p7. I tried on laptop's VirtualBox to set up
13.0-RELEASE, but freebsd-update now updates to -p10 version not -p7,
and i did not find a way to to get -p7. It seems to be unsupported
feature.

What do you mean to use debug kernel with WITNESS and INVARIANTS? To
build custom kernel GENERIC + add those two options or is there a
common kernel build config used by devs that already includes those
options?

Thanks.

From nobody Mon Mar 28 13:28:25 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4D7B81A3C953
	for ; Mon, 28 Mar 2022 13:28:48 +0000 (UTC)
	(envelope-from prvs=079c560b7=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KRtnW1Npxz4mQk
	for ; Mon, 28 Mar 2022 13:28:47 +0000 (UTC)
	(envelope-from prvs=079c560b7=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1648474127;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=n8UYgP9IjyTuuJCSRP68S72rRsYKPQQoJWZL6EhviHo=;
  b=hG0BrnuiYFu7SFxK++3W32UjCuuHNN5yGAyvLPM02vC4nNf1CJacIHOS
   PV3TNuiKGLHllW931+u9FeRJ9aTZl+jXzkMVLefAWqz6LMzeWk6D2oyuX
   XZ2f0RV96DvBfiGHebm6AAIxnHxXfYXy/YD3+WRHSot0eH83vWMUek/Sv
   c=;
X-SBRS: 5.1
X-MesageID: 67326684
X-Ironport-Server: esa3.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:x2lFBKJTXI7HOzGQFE+RxpUlxSXFcZb7ZxGr2PjKsXjdYENS0mNWn
 zRJWWDUPfjYZTGkc4wjaYzlpEtSv8SAyYcwSwFlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s
 q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh2dQ42YHR7z6l4
 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc
 s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4
 PBWvsK1Zl90BJWSncoEbx1+EyJ6L7ITrdcrIVDn2SCS50jPcn+qyPRyFkAme4Yf/46bA0kXq
 6ZecmpUKEne2aTmm9pXScE17ignBNPsM44F/Glp0BnSDOo8QICFSKLPjTNd9Glv3ZoQQ6aCD
 yYfQT1LQB/uWjNIAW88Iao+xemTvnvTMAQN/Tp5ooJoujOOnWSdyoPFOsfVYNuVbcNblE2Ro
 iTB5WuRKh8AOdGV0jbA4n+qruHVkC7xQ45UE6e3ntZ4jVGO3ncCEzUZUFK6pb+yjUvWZj5EA
 xVKoGx09/F0rRH1CImmN/GlnJKaljsHUIRVPLVn0jnTz4TN3wK/BmsAXCEUPbTKq/QKbTAt0
 1aImfbgCjpurKCZRBqhy1uEkd+hEXNLdDFfPEfoWSNAuoC++99r0nojW/45SMaIYsvJ9SYcK
 txghAw3nP0tgMECzM1XFniX0mv39vAlouPYjzg7v15JDCskP+ZJhKTysDA3CMqsyq7DFDFtW
 1Bex6CjABgmV83lqcB0aLxl8EuVz/iEKibAplVkAoMs8T+gk1b6I9wAvGkhfh03aZhUEdMMX
 KM1kVkLjHO0FCH3BZKbnqrrU5h6pUQePYqNug/ogipmPcEqKV7vENBGbk+MxWH9+HXAYolkU
 ap3hf2EVC5AYYw+lWLeb75EjdcDm3BvrUuOFMuT50n2jtKjiIu9FO5t3K2mNbtisstpYWz9r
 r5iCid940gGCLSlOHiOq+b+7zkidBAGOHw/kOQOHsarKQt6AmAxTfjXxLIqYYt+mKpJ0OzP+
 xmAtoVwkjITWVWvxd22V01e
IronPort-HdrOrdr: A9a23:mUpvj6v3X/FzAX6T3g85wQaB7skClIMji2hC6mlwRA09TyXGra
 +TdaUguSMc1gx9ZJhBo7G90KnpewK6yXdQ2/hqAV7EZniahILIFvAY0WKG+VPd8kLFh4xgPM
 tbAs1D4ZjLfCRHZKXBkXiF+rQbsaC6GcmT7I+0pRcdLj2CKZsQlzuRYjzrbHGeLzM2Y6bReq
 Dsgvau8FGbCAsqh4mAdzE4dtmGg+eOuIPtYBYACRJiwA6SjQmw4Lq/NxSDxB8RXx5G3L9nqA
 H+4kbEz5Tml8v+5g7X1mfV4ZgTsNz9yuFbDMjJrsQOMD3jhiuheYwkcbyfuzIepv2p9T8R4Z
 LxiiZlG/42x2Laf2mzrxeo8w780Aw243un8lOciWuLm72PeBsKT+56wa5JeBrQ7EQt+Ptm1r
 hQ4m6fv51LSTvdgSXU/bHzJl5Xv3vxhUBnvf8YjnRZX4dbQqRWt5Yj8ERcF4pFND7m6bogDP
 JlAKjnlbprmGuhHjHkV1RUsZyRtixZJGbEfqFCgL3Z79FupgE286NCr/Zv3Evp9/oGOu15Dq
 r/Q+FVfYp1P7wrhJJGdZc8qPSMex7wqDL3QRSvyAfcZeg600ykke+D3Fxy3pDvRKA1
X-IronPort-AV: E=Sophos;i="5.90,217,1643691600"; 
   d="scan'208";a="67326684"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=c2433eNNld5+rj9nhamzI+y96SUXGJWJk9ufgmh/6hmawF617bay03MO9NCDwV6BoPj1OGEKWK2R+sFFMwWO3Dh6WVNgTXACWlp6orje3ZSGLiC9dEYaZnNB/E++s5bisTzC4BB1zg3orODGPzoxNwn1g+AAdwuIGBLOS4ftdmEbWcCD9gLwgoAfqvSMgHzMGwu3q19xjvM9sne3C2mYPJD9hVYLLDuw9NVgFyGNTdhZ2/ARtL5ub+liGdoUGr79Zj9jhiZ/RkPmV7eftsCxpKZgyLHfCZ+ZSCt2LHhepAHlglQDtvkaRXQx5lYN6GzxT4gE7Z3SCxDcbWHgEF6/6Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=1xKNzzzb0kWVFuE5ieaYLQmo20WFGFQ5r1uKmS8ZoL8=;
 b=DeOjI76lOs1vpH2AwWGiIZ94v3Cp963mYf7XZBs/1C4GSzs82HN1SBB50qI+W3rDkKvxBtvz0SmY5hyYR1LyzI8nSGFyPlEEox2jec86sb33p7T2+rn5pPn1F9LiEHywto5NkKBcd5gR7beFsuyzddMqEWVKp2I/0iA0s2Hm3akq2j81qn8X7G4N2Od41W4Aq+NUxB/WJf30dTzPxWCtejFvAj6pjOcKkZsVGXPueYSa6ytYPvITJ7bPoSM1dj9Fgq95c0hkvS2peWa4yoO436tsSOF+xkTqjls6CY7HweUlvJ6AQtg+3IDSW0U5c/Ch2Ui0IDVrBwjL7F6FXUOqPw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=1xKNzzzb0kWVFuE5ieaYLQmo20WFGFQ5r1uKmS8ZoL8=;
 b=Tdu+sKn5DLvySV404WNFTnakILp5mG3tA3C9mbvBtqqfj75zXkkMoc4Ha7iffRuaSZMFk+8ykrHXwOT7GzifTaqoP850hb+CG10cuYNasHKvUYRwWXEF51fu56MKxauNfmwhhiYWAnVulR1Qlo6AglX4pLFIrjBv0DjsJDftyes=
Date: Mon, 28 Mar 2022 15:28:25 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , Brian Buhrow 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO4P123CA0272.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:195::7) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 35ee7a3a-6994-46ab-8928-08da10beda2c
X-MS-TrafficTypeDiagnostic: BN7PR03MB3907:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: WxpP1ZjO5nsi5JLsJ0GaUdCYx4q4K+XToeQBSQM5h5B3XX3RdWui8Ga4f/dgkb+lq849fyzF3L6Zd7w1A4cSbZMaKgyty+w2BFiQUOnS4EOdIk3MFzjIHHhmJyRszvOmh5sq33RHSiXZNwG4n4wKeXkP9fBcPH5DRSjiJq/oBhQNdsSPj/q2Px1jrAeLPSDbYww1MpPyuimhyQsectI05A/yj8Vxwv3uA+ndFBGqo66dYValoap7SYDCJQF9+XQUm1l7jnkLvnB/xFa/2IFv/Icv176mO7hugrpC5X2YLJ5uU2jbfsE0MCccLjKzwFkumb2NwXp7sFB067bhP31YWQaiFTFKHXFSpKxPVs1pQYYU+4QEMf7Lvr7Gnv7bDwHdP7bCrNIw5TdGHd6jWIKDv6Q03RmmrfAd2mIUUs+HfczChDz44Au2moctV4j+DIEnIq9Nim8EHGnXrIavTkxPgB/nxj9NmGoJOHytZY/Mgbd9cor6PMxIxO4t5LWQdFC9wINccvAUGWb9lKka8/ZST9nK/Cif+w3ctM3APgBpTkQSmrQkIQ04PFI1eM38g3U11M22kALWBhEZLYurMXafUF6JXY3hNxvXll6TGMDI9PaWNZY1Zn7jsOTetLeOsiSxBDfGlEA7Ni1UuB7GpSBtUAlIvkrwaL1/PdrcW/PMwWtHvf09IrGj5tiAUcX+0+8PUUVGXZw9fTHt/m1z4Yq2HkicFQhCtFqWYUE2+l8Qt4HYQRcVrbLPY97pp9dm5JjHi0ObiHfrlSFMwtTzPt9llBfxcOxPvzFcng4sO1QYZFU=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(966005)(508600001)(33716001)(38100700002)(8936002)(53546011)(6486002)(6506007)(186003)(2906002)(9686003)(6512007)(83380400001)(82960400001)(26005)(86362001)(8676002)(66946007)(6666004)(6916009)(85182001)(316002)(4326008)(5660300002)(66556008)(66476007)(67856001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UzVoZE9HNEsyUXhPQzBvWE1ycGQ4Q1RkdnhtTDRpQXZtdnpHdGZwZWFGVHBD?=
 =?utf-8?B?bWQ5cjU4MTJQSW1uaXNvK0pveml2QmY3RW41c0lKUzNmQXd5SHp6cXlFVmw1?=
 =?utf-8?B?MVhRSjB5UUNTR2JYVDlnVWJHejZJMGZJaURPRGVDS0pEc2hac1cwa2xrMW4v?=
 =?utf-8?B?bW41WGpTbHJOQ1p5VTB5Smt0UnVhZWtDWENEdFVudW1XTUkzRzBEb202b2JD?=
 =?utf-8?B?K0N5SkxaaU1nWUFjZkhqT2dGSlBjKzZaREFsbW0xblZ2MjladmFraTVjaDUr?=
 =?utf-8?B?QTM1clNXWnFIVkM3UCtmVlovM3pVVUNWUngySlZIRlpWNTh1SURkN0E0ZFZj?=
 =?utf-8?B?UkErbHpsbUk0WkNmNlFDVi9BNmEvTWJZbmFuZ3FldGNkSkVzYWRSaHcvMHB5?=
 =?utf-8?B?aXFSVE56Z09JVFVBcXpFOTlicVZHNkZXTzVGUGtnSm9oUUNxaFFkbzA2ME9D?=
 =?utf-8?B?b0RKSmc2ckZRdHREVGpGY003dXM0amQ0T0VoczFyTThINGFjMEhaQ3V6bW05?=
 =?utf-8?B?VndHakR5Qng3N2V2cEtDejZOSDFrdjVMcGlPaDdtR3hoalVsbFY2T3kyenZX?=
 =?utf-8?B?dnJWaEI4QUV4YmlUNTlsWlpLTWRKamlma29PeHdWNGVyZys2WVdXQUQyd2RS?=
 =?utf-8?B?TjlteWZLbSsxK2lwTGcrdCtSUDB4K21ldEU4SmlkSURUWlhNSU1KVFY0Slgz?=
 =?utf-8?B?WTRLQlpjdVAwcUI3U21uUGEwSk45VjdHWUJYUEM0VDgzM0tyaTBSQ1BvbjVJ?=
 =?utf-8?B?TEk3WjNUd0wxSjhOWmx0U1FXdzRZaWQybzVOa1pZNnhiWkVOQ0dNY3ljSE1Z?=
 =?utf-8?B?YTY1Y0VkVmR1QmE4eEpjUjhDelBFenpFVHhHSGJ5YStSdGpzN2J5aU1IQVAy?=
 =?utf-8?B?R2V1LzBMaEF5cjBWMCtYSXVtVGJvcmVsYUYyRUJUOFRtU291amJUODNPaFV3?=
 =?utf-8?B?Q0VBSXoyZnNUOUh4TDhYSklyeGFyQlJYRjdHMGhpeTlPTE5ZYkUzenRPQTBH?=
 =?utf-8?B?VlhiZDFidk1zbzYwN2lsNW5ncjVMQUxsa3lMMnhMcXhjSmcrK0RTcWY5VHEx?=
 =?utf-8?B?S2VZMTZVak9id0M1SXRrWFcvQURaOFpKRHBISE40cUZsQUVQd3lIc08wUDJI?=
 =?utf-8?B?K1JQbXNqN3R1UUZzdTVxZlk0MEtqSWx1SlFRUzVHekwwOGhWcEIxTVhVd3Z3?=
 =?utf-8?B?VkNOVDFHd0RPRUVneGt3a0RvaFhvUUpNKzVOY2NzeTdwMEZuWGQ0UXovLzFM?=
 =?utf-8?B?NUZjcWh5Zmc4STRyVW9KSHN2QlYrekxZU0JGRFlZeEV2eGhnSHNHQVdsWThJ?=
 =?utf-8?B?cnBrdTQ3M2o0MlErcGt6SlVsQnZreWdvRlkxWnZiLytPVDZKQktCVFdmZzd0?=
 =?utf-8?B?K3pzNzdqUHJwQkFMWFNDWTRuWWx2OUUzcW9aVytTKzdTSlNLZElGVVk3UXpx?=
 =?utf-8?B?aDVTU1dYekZOa1p6UVJ2L1ByU1VQSnY1dkpSdEl2bU1HT1QwazN1eWlhRGM0?=
 =?utf-8?B?enpYLzBDQnpLL1VlVnIzcmQxRWRYY3M4ZUlqbVBZODR0OFo2aDhEU1ZteGtB?=
 =?utf-8?B?VSszQjU4bEg5VFgzYzRyaUwwRjFMSWpzSWh1R3FFaXUra3BHUFJCRnJKU0dt?=
 =?utf-8?B?c0xnZmM0dzFnZHFyYzVhaWVsdFN1bEh6blJzYlJxVGtGcGdObjRZeGNUWDc2?=
 =?utf-8?B?cjZFeDZiMlFlM09qWUhoMlAyVnlnc25sV3FuKzF1QzhBSTBHU1gvWjZrQlZ1?=
 =?utf-8?B?WkZqTnZJTGJ3c0ZjSDdFb3pCbVJDbkhMalBSbGRpWkFmRnBoZWx1NjVUU0I1?=
 =?utf-8?B?a0FXNmNGbTlQeW1OaFhSUnRtejRIeWM0Z1pxdVFkQjMyUjQ1eXZNQ2JtUE05?=
 =?utf-8?B?Nll0VnVhRWdNMmVLYmVqdmdkS3lURmswRldRWFMrNXFUay9nbzQwWUZreFN4?=
 =?utf-8?B?Y1FEN3ZORStzNHptcmJBMDBCSFpBdkcvQTJMTXI4aW1DZDNkb1ZBZDdrRlFq?=
 =?utf-8?B?dnZ5RVRmeENabk9FT2pyVm12WG85Y1UvdEpOYVdZN0tOOHFRSVlQM0MxYVdR?=
 =?utf-8?B?WFNiS0gzdnBta25IUXZlZzZhS0xBaTVJc3pzUllORHoxR2FYUEZCZ3FVd3lY?=
 =?utf-8?B?NnhTanhrN2w5RHpvRUwrd05YampuRlNFcERTRDZ5aDFZYVltTzRZS1NHa2ZW?=
 =?utf-8?B?aEV4ZE9BdjhaazFadVR0cGx6aWN1VmM2TXNlclpaNTJqK284S1V5LzNZK3pP?=
 =?utf-8?B?dURZWlF5OWloSGhuQmU4dTlmbWZZNzF6WmFEYnVCdFQrcy8reFM3SG8wMlpQ?=
 =?utf-8?B?WUMxSU5OWnFUeDgrT0cxbmpySnVpVTNWWGRRQ0JIbmk5aTd3V21qQUg3OXkx?=
 =?utf-8?Q?ch+CvN0FhPBBYDjg=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 35ee7a3a-6994-46ab-8928-08da10beda2c
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2022 13:28:31.8403
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8BMTjDGPAGaK5rpcchWbHB2WPtZi6OKy6bedEWvKf3YiB7xr/TcHAvlQGw5TjQLIBHnflUUaPmWPZ5sewk9jWQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR03MB3907
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KRtnW1Npxz4mQk
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=hG0Brnui;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=Tdu+sKn5;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=079c560b7=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=079c560b7=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=079c560b7=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sun, Mar 27, 2022 at 08:42:23PM +0300, Ze Dupsys wrote:
> On Sun, Mar 27, 2022 at 12:13 PM Roger Pau Monné  wrote:
> >
> > On Sun, Mar 27, 2022 at 12:38:00AM +0200, Ze Dupsys wrote:
> > > ==== COUNT: 2
> > > Fatal trap 12: page fault while in kernel mode
> > > cpuid = 1; apic id = 02
> > > fault virtual address = 0x68
> > > fault code    = supervisor read data, page not present
> > > instruction pointer = 0x20:0xffffffff824a599d
> > > stack pointer         = 0x28:0xfffffe00b1e27910
> > > frame pointer         = 0x28:0xfffffe00b1e279b0
> > > code segment    = base 0x0, limit 0xfffff, type 0x1b
> > >       = DPL 0, pres 1, long 1, def32 0, gran 1
> > > processor eflags  = interrupt enabled, resume, IOPL = 0
> > > current process   = 0 (xbbd7 taskq)
> > > trap number   = 12
> > > panic: page fault
> > > cpuid = 1
> > > time = 1646122723
> > > KDB: stack backtrace:
> > > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > > #1 0xffffffff80c09f01 at vpanic+0x181
> > > #2 0xffffffff80c09d73 at panic+0x43
> > > #3 0xffffffff8108b1a7 at trap+0xbc7
> > > #4 0xffffffff8108b1ff at trap+0xc1f
> > > #5 0xffffffff8108a85d at trap+0x27d
> > > #6 0xffffffff81061b18 at calltrap+0x8
> > > #7 0xffffffff8248935a at dmu_read+0x2a
> > > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa
> > > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394
> > > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> > > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> > > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> > > #13 0xffffffff80bc7e3e at fork_exit+0x7e
> > > #14 0xffffffff81062b9e at fork_trampoline+0xe
> > >
> > >
> > > ==== COUNT: 1
> > > Fatal trap 12: page fault while in kernel mode
> > > cpuid = 1; apic id = 02
> > > fault virtual address = 0x148
> > > fault code    = supervisor read data, page not present
> > > instruction pointer = 0x20:0xffffffff8248cef4
> > > stack pointer         = 0x28:0xfffffe009941d9a0
> > > frame pointer         = 0x28:0xfffffe009941d9a0
> > > code segment    = base 0x0, limit 0xfffff, type 0x1b
> > >       = DPL 0, pres 1, long 1, def32 0, gran 1
> > > processor eflags  = interrupt enabled, resume, IOPL = 0
> > > current process   = 0 (xbbd1 taskq)
> > > trap number   = 12
> > > panic: page fault
> > > cpuid = 1
> > > time = 1646129773
> > > KDB: stack backtrace:
> > > #0 0xffffffff80c57525 at kdb_backtrace+0x65
> > > #1 0xffffffff80c09f01 at vpanic+0x181
> > > #2 0xffffffff80c09d73 at panic+0x43
> > > #3 0xffffffff8108b1a7 at trap+0xbc7
> > > #4 0xffffffff8108b1ff at trap+0xc1f
> > > #5 0xffffffff8108a85d at trap+0x27d
> > > #6 0xffffffff81061b18 at calltrap+0x8
> > > #7 0xffffffff825cb76e at zil_open+0xe
> > > #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2
> > > #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88
> > > #10 0xffffffff80a7f214 at xbd_instance_create+0xa394
> > > #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> > > #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> > > #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> > > #14 0xffffffff80bc7e3e at fork_exit+0x7e
> > > #15 0xffffffff81062b9e at fork_trampoline+0xe
> >
> > Hm, those last ones are in ZFS code, can you try to get the line
> > numbers for those?
> >
> > Maybe it's blkback providing bad data to the disk open functions.
> >
> > Since you are doing so much testing, it might make sense for you to
> > use a debug FreeBSD kernel rather than a production one (one with
> > WITNESS and INVARIANTS enabled).
> >
> > Thanks, Roger.
> >
> > ---8<---
> > diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
> > index 33414295bf5e..4007a93a54c7 100644
> > --- a/sys/dev/xen/blkback/blkback.c
> > +++ b/sys/dev/xen/blkback/blkback.c
> > @@ -2774,19 +2774,12 @@ xbb_free_communication_mem(struct xbb_softc *xbb)
> >  static int
> >  xbb_disconnect(struct xbb_softc *xbb)
> >  {
> > -       struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
> > -       struct gnttab_unmap_grant_ref *op;
> > -       u_int                          ring_idx;
> > -       int                            error;
> > -
> >         DPRINTF("\n");
> >
> > -       if ((xbb->flags & XBBF_RING_CONNECTED) == 0)
> > -               return (0);
> > -
> >         mtx_unlock(&xbb->lock);
> >         xen_intr_unbind(&xbb->xen_intr_handle);
> > -       taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
> > +       if (xbb->io_taskqueue != NULL)
> > +               taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
> >         mtx_lock(&xbb->lock);
> >
> >         /*
> > @@ -2796,19 +2789,28 @@ xbb_disconnect(struct xbb_softc *xbb)
> >         if (xbb->active_request_count != 0)
> >                 return (EAGAIN);
> >
> > -       for (ring_idx = 0, op = ops;
> > -            ring_idx < xbb->ring_config.ring_pages;
> > -            ring_idx++, op++) {
> > -               op->host_addr    = xbb->ring_config.gnt_addr
> > -                                + (ring_idx * PAGE_SIZE);
> > -               op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
> > -               op->handle       = xbb->ring_config.handle[ring_idx];
> > -       }
> > +       if (xbb->flags & XBBF_RING_CONNECTED) {
> > +               struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
> > +               struct gnttab_unmap_grant_ref *op;
> > +               unsigned int ring_idx;
> > +               int error;
> > +
> > +               for (ring_idx = 0, op = ops;
> > +                    ring_idx < xbb->ring_config.ring_pages;
> > +                    ring_idx++, op++) {
> > +                       op->host_addr    = xbb->ring_config.gnt_addr
> > +                                        + (ring_idx * PAGE_SIZE);
> > +                       op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
> > +                       op->handle       = xbb->ring_config.handle[ring_idx];
> > +               }
> >
> > -       error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
> > -                                         xbb->ring_config.ring_pages);
> > -       if (error != 0)
> > -               panic("Grant table op failed (%d)", error);
> > +               error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
> > +                                                 xbb->ring_config.ring_pages);
> > +               if (error != 0)
> > +                       panic("Grant table op failed (%d)", error);
> > +
> > +               xbb->flags &= ~XBBF_RING_CONNECTED;
> > +       }
> >
> >         xbb_free_communication_mem(xbb);
> >
> > @@ -2839,7 +2841,6 @@ xbb_disconnect(struct xbb_softc *xbb)
> >                 xbb->request_lists = NULL;
> >         }
> >
> > -       xbb->flags &= ~XBBF_RING_CONNECTED;
> >         return (0);
> >  }
> 
> Hello,
> 
> I applied given patch, i did not have enough time to test thoroughly,
> but for 3 hours system was running without panic whereas previously it
> would crash in around 1,5 hours in similar settings. Till Thursday i
> will not be able to test.

Well, I guess that's good. Let see if you can trigger those ZFS
related issues again and can provide the file:line numbers for the
backtrace.

> About those ZFS panic traces, i will try to get line numbers, but the
> problem is that i do not have /usr/lib/debug/boot/kernel/kernel.debug
> for FreeBSD 13.0-RELEASE-p7. I tried on laptop's VirtualBox to set up
> 13.0-RELEASE, but freebsd-update now updates to -p10 version not -p7,
> and i did not find a way to to get -p7. It seems to be unsupported
> feature.

I've tried to use the kernel.debug from 13.0-RELEASE but the output
doesn't seem to make any sense.

> What do you mean to use debug kernel with WITNESS and INVARIANTS? To
> build custom kernel GENERIC + add those two options or is there a
> common kernel build config used by devs that already includes those
> options?

IIRC main branch has those options enabled, but not stable or releng
branches IIRC. If you build from stable or releng branch you will have
to add:

makeoptions 	DEBUG=-g
options 	INVARIANTS
options 	INVARIANT_SUPPORT
options 	WITNESS
options 	WITNESS_SKIPSPIN
options 	DEBUG_LOCKS
options 	DEBUG_VFS_LOCKS
options 	DIAGNOSTIC

To your kernel config. Ie: this is removed from the GENERIC config
during the preparation of a release:

https://cgit.freebsd.org/src/commit/sys/amd64/conf/GENERIC?id=bfd15705156b0436cfe79aea11868dcc0c6e078a

Regards, Roger.

From nobody Tue Apr  5 15:22:39 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CE9401A99D8F
	for ; Tue,  5 Apr 2022 15:23:07 +0000 (UTC)
	(envelope-from prvs=087021108=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KXrxk56byz4pLt
	for ; Tue,  5 Apr 2022 15:23:06 +0000 (UTC)
	(envelope-from prvs=087021108=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649172186;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=sZ/NhjEFcUZVFt/SJ6G9PK/HWZdWx4G4YaRt5nHQl1A=;
  b=H6TqDw+Kls6hz/HxpXiJ7K5jt3VM1qax9WOne2UJDM9QzcbZbfyg1xD1
   RT6xO2taYCKormq7v0YJo+tu+atm3iPEcdNdy+EOS8yQCPNHEyoQzaxo0
   p4oh6u4qvAOAwDeBhWdbGeo+F5faNX1tY9ftKN5K1ZzXV/bCherYtbKsS
   0=;
X-SBRS: 5.1
X-MesageID: 68075823
X-Ironport-Server: esa2.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:xQDHT6/drp7d3Rl4J6D/DrUDq3mTJUtcMsCJ2f8bNWPdYAuW94E1v
 mIfWWibcqrUMCe2ZoArKM/psx8b7cOTl4U2VUYu/HpuTnRF79LMGd+ZRqubFy7DdcCcRRg8t
 Z9ONIDNIJFkRCbSrEimPOi+8CEkiPjZGrPxWL+UN3AtFQM4Qn9/2U1twLUz3YBm2om0U13Q6
 I6aT6EzQLOA82cc3jU8t/rZ93uDxcjPhQ/017BGiZpj5AeGzTwFAZRaL6WyJXLzQcxaH/7/W
 +vK1vS1+XzfuhY2Ec6oifDje1EBQtY+ViDT1idbAfmu3kcdq3Vr2atnb/cXZRZc2mnVz9ohk
 4RA6cW6FQ0iM/SdxekQA0kESS11MfEco7aaLSTu6JzNlEGeG5eAL42CKWlvVWFP0rotXTgmG
 YUkFQ0wgjC/a8Oeke6wGuJiis18dMPiZNkS5H0wnG/TUqkoTZnKSf2RvYUJgDtois5wRvuPP
 MBxhRiDzfjjj7+jHn9NVfrSSc/x3iGXnwVw8Q7T/exti4Tq5FQZPILFabI5QfTXA5QN9qqkj
 jiepT6hXklHbIb3JQetqRpAuMeexUsXZ6pKfFGI3qYCbIq7nzF75LU+DDNXkNHh4qKMc4s3x
 388o0LCmZMa5k2zJuQRajXjyJKyUrzwbPILewEywFnlJqM5eG91DEBcJtJKQIROWMPb2VXGf
 7JG9j/kLWUHjVGbdZ6S3quO9AiuMwgsFig5bjMEZiI308bPkI5m23ojTv47eEK0ptj8GDW2y
 DGWtikuwb4UiKbn1Y3iowqB2Wj14MGUEEhlvW07XUr8hu99TJSiaIGyr0DS8N5LLZqDT0nHt
 38B8ySbxL5eVc7QxHbWKAkLNLTwzcTCbTjYuwdUHrQjyXfq5lL+I58FtVmSI282a51ZKFcFe
 nT7sx5R/phMFH2kZ6R+Z8S2EctC5ab6GNnvTfyRftdISpZreQKN5y0oYlSft0j/nUQxibouI
 r+UdM+tCTARDqEP8datb75Di/lxnHl4nD6NA8Cgp/i67VaATGKPEIwZOnu3VacC57+Eiyznw
 8lzB8Tfnn2zT9bCSiXQ9IcSK3UDIn46GY36pqRrSwKTHuZ1MDp/UqGMmNvNb6Qgxv0IzbmQo
 hlRT2cCkDLCaWv7xRJmg5yJQJfmRt5BoH0yJkTA1n74iiF4Me5DAEryHqbbnIXLFsQ+lpaYr
 NFfIq1s58ijrBydplzxirGn8eRfmOyD317mAsZcSGFXk2RcbwLI4MT4WQDk6TMDCCG63eNn/
 eHxjlmDHcdfHFg+ZCozVB5J5wns1ZT6sLgsN3Yk3/EJIBm8mGSUA3KZYgALzzEkdkyYm2ryO
 /e+ChYEv+jdy7LZA/GS7Z1oW7yBSrMkdmIDRjGzxe/vaUHyozryqacdAb3gVW2MCwvJFFCKO
 Lw9IwfUa6Zcwj6ncuNUTt5W8E7Jz4C2/eUHkV04Qi6jgpbCIuoIH0RqFPJn78Vl7rRYpRG3S
 gSI/NxbMq+OI8TrDBgaIw9NUwhJ/atE8tUOxZzZ+HnH2RI=
IronPort-HdrOrdr: A9a23:47njlK6NPiHCz7cjigPXwSqBI+orL9Y04lQ7vn2ZFiY7TiXIra
 yTdaoguCMc6AxxZJkh8erwX5VoZUmsj6KdhrNhQItKPTOWw1dASbsN0WKM+UyDJ8STzJ856U
 4kSdkDNDSSNykKsS+Z2njALz9I+rDum8rJ9ITjJjVWPHlXgslbnnlE422gYytLrWd9dP4E/M
 323Ls5m9PsQwVeUu2LQl0+G8TTrdzCk5zrJTYAGh4c8QGLyRel8qTzHRS01goXF2on+8ZpzU
 H11yjCoomzufCyzRHRk0fV8pRtgdPkjv9OHtaFhMQ5IijlziyoeINicbufuy1dmpDl1H8a1P
 335zswNcV67H3cOkmzvBvWwgHllA0j7nfzoGXo9kfLkIjcfnYXGsBBjYVWfl/y8Ew7puxx16
 pNwiawq4dXJQmoplWz2/H4EzVR0makq3srluAey1ZFV5EFVbNXpYsDuGtIDZY7Gj7g4oxPKp
 gjMCjl3ocWTbqmVQGYgoE2q+bcHUjbXy32D3Tqg/blnQS/xxtCvgklLM92pAZ1yHtycegA2w
 3+CNUaqFh/dL5nUUtDPpZyfSKWMB26ffueChPaHbzYfJt3Tk4l7aSHpIkI2A==
X-IronPort-AV: E=Sophos;i="5.90,236,1643691600"; 
   d="scan'208,223";a="68075823"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=AU5oOZ5lG3Nt1Gu8Btz+hyWzG5Ygs0xC+OgDAa+syN2L8OaxtopYDa4PKTxdtjZvVuL1R70/I52LR+Tk5HYNFeuFS0vftkzQhIAfiMGnpgpKoySLtfy6f8Ih7vyAlsrDK2w4eIk11M7oTm0nGWBPJdUxHFHhtDZMUhWOXi0TMxnFCKQzOwVU3l5ooAdwqSeRza2unafmiiu4/0u6sQmkXcLSoqLmal870CvnwSIL7ROyma4l+kS7PtCDdwW3hTNqv6B7XIxhclOqURRzVLJZXaaeeD25rcw2fBChOrF6QrTM29MTiNP3fEeoEG0uIruUCzBF/7bkUG7QULz2GOIL8Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=rU3+G0jbKkVpndS40hA5Tdupv0Ej9e1bVJaRMaxeHPo=;
 b=E/lC+ZAPqdo3RjDuacjz1AUy2t0i9/U/rEgklkgGFIrfIALXqHLjJyJ8QgafgfsvWa8w2PIeU7nBT1SfhdfvjFSTuJKyrRzu2LqloiJM42grPQvvL5I0nVVGVaSR2Vw0tHzASNgvtctYx8bfndzrpuOXAxuW8OW9QKChemH1LV6XGzVJ/tCWdHJOUTpC/JKE4dJ/Klce/n+Pou64JSCrh1bsMSb2H7euKzQ9E8S8aCCI/vJjuGbV40vV7jDgGYTn8hFfk36m2S8/NvH2PiBpi2X89MZ0ImBCEdoOlvfB4TNAXNUhmbJAIJ47LoKjdWNVdV2OI0qfmpoCBUjxAgxxSQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=rU3+G0jbKkVpndS40hA5Tdupv0Ej9e1bVJaRMaxeHPo=;
 b=LFOUGTOpwPF7g1VcdoT+9iwvtXsUB+5x3c2QBfgwDb1eLYbcrMwoiiEiIrA8moskp7fubcfdGHcSVOPpqnw6dQjcYUOQ+hoSedjKoZLks2bd/ARHecj3izRwPsZsveoWe9LytMIjMgK8R9CR0YIHYP95pvMZNyCV+AewMsqsINM=
Date: Tue, 5 Apr 2022 17:22:39 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
Content-Type: multipart/mixed; boundary="4oNrmEULt/shPi4w"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
X-ClientProxiedBy: LO2P265CA0250.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:8a::22) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 2180caab-aa97-4aa7-800f-08da171821a9
X-MS-TrafficTypeDiagnostic: BYAPR03MB4103:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: zmqRDYwWSoIGjSvvtEI7efCI+OC4Bk6VHGcYZjDqY3o8SygjVV+Nv8X9R1paeyPlRU2cG23rQ8oTINjF2FrRkOuoQ5h7v9Ep1TdkdjL+WPandEk7ruxhjvufuNftRfs4Pkm5VkLsL6TnEumgtYbqJY2+TQI8JuK26lQXerGsij8dP2nr4FqJNAtdAuTqTnVzG3+o7uZrJ7NcTPGbze9G6e3agIdrjCRjlqVZLhVP+5zSr1tIs9A2iIVgnUYBSVl/Qb0LTZHQVE7lAaCpzlUC3vDK0K7eOPHYhjLTV+4lG8fJrO969L1DliMk/oltC3qwss/TsUQh0dg4bH0wZ33aYTuQJJW6ZpYTndY/eWFDDa75oIDkIn5WFTmYqI+JHijUS6UXa4v0+LoI/XBnS4tScmi0/40riGvqBGJUXps6fMeBV+xBgw7W0XoVxjIwcMSDdPJgWy4UKKuZYTArSBgC3Mq54WEIZHkIebhUrb9kKkOX8BZ5cBXhTCD6pdvkW3camD+D8kTRT0vqTTVayMU/gsrHAPNPidQYd9G0hOtKiy7wIPJ4KRPh2soQvrLpnihoHuZgTLMMBVCkk46HRb6u40/8+5qSnLzU4X2iJGT2rz39WRw6dGEZ87/In164TUjc2gsoIrAddnbYgdjv7Q7BrDLCOSgBo1tiUGV0xIbZueIfIfMO4jHdnEEjsL9tjDLdt3vexzVdAlJVfbk+8hJdT+ty9YPxyKOEkdUH3PynsuqQeOlbxXkG/M6iH6VzfPcbNfjzM8l2bfmI524DbmDH+1DQFxjhA9BaWgKviHcOu1Y=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(6512007)(9686003)(8676002)(966005)(33716001)(8936002)(86362001)(5660300002)(82960400001)(66476007)(6506007)(33964004)(4326008)(6666004)(44144004)(66556008)(38100700002)(66946007)(235185007)(186003)(316002)(83380400001)(6916009)(85182001)(6486002)(508600001)(26005)(2906002)(2700100001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?b2VYN3ZLaXBwNVRRTFgxQm1PdVZhUnYrSUdUbXFNQzdDUFBjdlZTOGZrNUJq?=
 =?utf-8?B?K0F1bzdsK2FFVmh6b1FYOU54TTF2MGlRR29kbVkzNkZSVHdMRis1L2JDU3p1?=
 =?utf-8?B?aGlNS2RiQmp4aERLalQ4YlovUnZhbSt3MjgyNzYxOWpCNWh4cWJrR20zZ2lG?=
 =?utf-8?B?dG9yMy9sMkJ3UnBjR0hidlJKYzhZbEVjYmNCdGxqN2dLTnBRR04ydFlrMVlM?=
 =?utf-8?B?eVRVMUFFM1VpNnFOM1NaYVRLVUs4Mi90Q2NXdGdXY01nRGRFWjN3a1BnYWtW?=
 =?utf-8?B?aFhqRzNvSmdia2lDL084OUdUMUI3c0Y1NjhFQzFPUFBWb3dmTDBHa2xSdXpq?=
 =?utf-8?B?WGNDSHN4Nms5NDhsekNuU25ETzBseW92TVgyMzh6ekFFekE2TEc2MXFUaTVT?=
 =?utf-8?B?VG41NWZCN1RGTys5S3JQZnV0bzlBWWJ2cUFjaFlCVG92YURuTTBFQ0lPRTdF?=
 =?utf-8?B?WGFUTjVnZXF3TithK2pyeTBmcDJKc1dpdUs3N1R6dFpoZkN1Q1pFTEpFYnBZ?=
 =?utf-8?B?bjgrOVFrUHRhbzUzNDhFenoyK3pUNVZTbTkyZDh4L3loMHgvc2VNVXVnQmk2?=
 =?utf-8?B?dm1CRjM1ZmpjWHAzUG5jRzBmaUZMWFZScjgwQnVVa21ZWkJTTXJoWnF4OXVx?=
 =?utf-8?B?VFp4MGF0V3F3alZ1RUFybEpLd1JDZzlVUm9nNklGT2hGUklSMk5OVFIyU2tQ?=
 =?utf-8?B?ZGhuU2srdFVpaXVBbDJQZC9RdDVLOVE0ZmY1d1g2N2NERHJtektGNS8rRjVk?=
 =?utf-8?B?aXNpbC84V2UwTS9SL2l2U1lYMld4NTZnTGJoSHNML3RWakVxUVZMWmlDZFNu?=
 =?utf-8?B?dGp5TW0vUmJwVHhPME1rNVdKUENiU2tKQ2g2MHgwSWlWV1lUbGkzbFY1RkJt?=
 =?utf-8?B?ckZESlc1LzZxVnhKRy95Q1RTNlZpVkx2RHRUbVM5MEVlWEMxMEx4YjlNdXJQ?=
 =?utf-8?B?dlN1K284Rjh5YlRWRGpSY2l2MENOT3BOaGt5eWZqYVp0dFhjRW1sVGJ4TW1j?=
 =?utf-8?B?cGR0c0poQStGQXovRnBTNVVDaVdQNi90MDE4NlRER2JocklVTTI1NnBKTk95?=
 =?utf-8?B?R3VWVWFYWDRSdGFscThWS0FNR09WazNXeDVEb0RvSjJ5bGZPWFdUK0FOcmxs?=
 =?utf-8?B?L3JSSDQxbDdObzBEdmJlVkhhOXFsdm1hWitaK0JLcldNTmRib0ZQWEZxZzJN?=
 =?utf-8?B?bjZib0JDc1RlNkZhdHkrQlVJN1k5bmFMMlhQbkZEeWxKVVB5aWxpUFFLQ29G?=
 =?utf-8?B?T1RCM3cxbUJ5RjBWRjVxTWpOY2N3TGNreDNBUERUV2xta0hkalc1eThQSXVh?=
 =?utf-8?B?RkpCcW9qUytoaWtCVXZuSS9qOEVBRVkzZytETkxNQ01LZmVaY0VvK3NrREhG?=
 =?utf-8?B?U3FibTR1U1gzTkNZUm1uV2VrM3F3NDZDZkY5QVZqV3QrUWZrRnlHWUwwNVFu?=
 =?utf-8?B?bzFjdHlSOW03aWFNOW16U09iWjdENlNUUFdKbHBJbzR0RmhLbDkxT2ZRbWpJ?=
 =?utf-8?B?ajBneHg1SzF1QUhidEhXdTVqME5kY1dYS0tqcWVXK1B3NWVlMXpRcGgvQXpY?=
 =?utf-8?B?NHhXWUJnRXVDeUFJUEVqdDBidXl6TzV0TFNJMElpNWxtcXRQUCtQczhaNlJu?=
 =?utf-8?B?TFNab0s2NW9kRE9nNHJkWlprZXp1bFVxaGdhVis4Tk5OWkdTc0RqZWRnQkE0?=
 =?utf-8?B?NDB1aDF6RWdlNWtrMVl2dkNnSklVcm9iMWRkOU1qTW8xMjlwYW5KdzdIN3hn?=
 =?utf-8?B?RnJDK1RRMUIzd2l0cmVkNUNKWjhrTnJPNktzcnlIN2hSSStDNkxCTlZHUmM4?=
 =?utf-8?B?aTExUEcxYTY3bzdibnUyTnhoTWl1aE1JQWFjTldxbUp6ODR5a1VjNFlNaHA5?=
 =?utf-8?B?R1JJajRRenJxNUZGNnlFZStSVFpQYjQ2cmN0NjJiVXVwNkNXeEh4SVZrVzNX?=
 =?utf-8?B?YnpTWmRWUnFaRXg4UkROVS9WZGdJK3JPRmpiNHR0MWNnTjJ4RzJ0VVV5aVU0?=
 =?utf-8?B?STV4RkFRQnNkSXRXZXdsUUQrdm9SL2hOTGZCcWRYOUFpR3pzMndVOFhVYzB1?=
 =?utf-8?B?NVIzRGtubVhIWVFjTjhRY3hJTmdWUU90cDlPVmk4a1ZuMHJFcGxubHpHVDhC?=
 =?utf-8?B?cnNRWUd6UEVwSFl5WjgxaWxmbDlhYyswYUwvRFpZRjU0M2gwem5aWExIYjJz?=
 =?utf-8?B?U2pkQjhubG1lY2lCQTlWVzYwL0F4d1JWdGZZNEl6Y0cxUGM0eDM0ZllIazJJ?=
 =?utf-8?B?bkk5RndnKy9XTWFhTVJRUjVxQjJ3UmZpUytMYjlxM1l5M0FVTFJjVi9HSFZp?=
 =?utf-8?B?ZTZ6Q3lkbS90T1gvT3YrWkZGSmVaRGEveGQ5LzdadW1ZMncyRTAvdVRiMHE2?=
 =?utf-8?Q?xG1fxnZu4j6rYpPw=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2180caab-aa97-4aa7-800f-08da171821a9
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2022 15:22:44.0412
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wZb2UqatkPF7XhDB3ZfM610shD7qoe1NcvnIdK7wCUULiXt6OxISJ/EzaBGM41k+UqqfoZNqegXu3m/lttj0Rw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR03MB4103
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KXrxk56byz4pLt
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=H6TqDw+K;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=LFOUGTOp;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=087021108=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=087021108=roger.pau@citrix.com"
X-Spamd-Result: default: False [-3.89 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 HAS_ATTACHMENT(0.00)[];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 CTYPE_MIXED_BOGUS(1.00)[];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+,1:+,2:+,3:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=087021108=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-0.24)[-0.241];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[multipart/mixed,text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

--4oNrmEULt/shPi4w
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On Sun, Apr 03, 2022 at 09:54:24AM +0300, Ze Dupsys wrote:
> On 2022.03.27. 12:13, Roger Pau Monné wrote:
> > ..
> > Thanks, unfortunately that patch was incomplete. I have an updated
> > version that I think is better now, and I've slightly tested it
> > (creating and destroying a domain with it doesn't seem to crash).
> > Appended patch at the end of the message.
> 
> Hi,
> 
> This patch was far better, i almost wanted to say that it works, stressed
> system with 2G RAM and it did not even have signs of sysctl-var leaks. There
> were too many things going on, thus i most probably will not be able to
> reproduce this case, but just before panic i did "xl list" command and it
> instantly crashed (new trace). What i noticed after restart, that again some
> default nightly script had made /var/backup/* files which made root file
> system full. Since test overlaid 2 nights, in first day when root disk was
> full, system did not panic, i just rm'ed /var/backup biggest file to make
> sure that there are no problems. When i did "xl list" both test VMs were
> running, state i do not know.
> 
> Full serial log in attachment, part 2 is where most interesting stuff is,
> ending is a bit mess, but:
> ..
> (XEN) d284v0: upcall vector 93
> Apr  3 08:10:11 lab-01 xenstored[937]: TDB: expand_file to 229376 failed (No
> space left on devicex)bbd24: Error 5 w
> riting backend/vbApd/284/51760/sectors
> r  3 08:10:11 lab-01 kernel: xbbd24: Fapid 9tal error. T37 ransitioning to
> Closing St(xensate
> tored), uid 0 inumber 2003596 on /: filesystekernel trap m ful12 lw
> ith interrupts disableApd
> 
> r
> Fatal trap 3 08: 110:11 2: page fault while in kerlab-nel mode
> 0cp1uid = 0 ; apicx id = 00
> enstofault virtual address	= 0red[9x20
> 37fault co]: code		= surpervisor read data, page not presentru
> inptistruction pointer	= 0x20:0xffffffff80c94e80on
>  destack pointer	        = 0x28:0xfffffe0051tected8803c0
> frame  pobyinter	        = 0x28:0xfffffe00518803d0
>  connecode segment		= basce 0x0, limit 0xfffff, type 0x1b
> tion			= DPL 0, pres 1 , long 1, def32 0, gran 1
> proces0: sor eflags	= resumerre, IOPL = 0 No sp
> current process		= 16 (xenwatch)
> traap number	ce	= 12
>   panic: page fault
> cpuid = 0
> time = 1648962612
> KDB: stack backtrace:
> #0 0xffffffff80c7c285 at kdb_backtrace+0x65
> #1 0xffffffff80c2e2e1 at vpanic+0x181
> #2 0xffffffff80c2e153 at panic+0x43
> #3 0xffffffff810c8b97 at trap+0xba7
> #4 0xffffffff810c8bef at trap+0xbff
> #5 0xffffffff810c8243 at trap+0x253
> #6 0xffffffff810a0848 at calltrap+0x8
> #7 0xffffffff80c0b87a at __mtx_unlock_sleep+0x7a
> #8 0xffffffff80a98724 at xbd_instance_create+0x7aa4
> #9 0xffffffff80a9abb0 at xbd_instance_create+0x9f30
> #10 0xffffffff80f95c64 at xenbusb_localend_changed+0x7c4
> #11 0xffffffff80ab0f04 at xs_unlock+0x704
> #12 0xffffffff80beaeee at fork_exit+0x7e
> #13 0xffffffff810a18be at fork_trampoline+0xe
> Uptime: 1d10h56m34s

Thanks, sorry for the late reply, somehow the message slip.

I've been able to get the file:line for those, and the trace is kind
of weird, I'm not sure I know what's going on TBH. It seems to me the
backend instance got freed while being in the process of connecting.

I've made some changes, that might mitigate this, but having not a
clear understanding of what's going on makes this harder.

I've pushed the changes to:

http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak

(This is on top of main branch).

I'm also attaching the two patches on this email.

Let me know if those make a difference to stabilize the system.

Thanks, Roger.

--4oNrmEULt/shPi4w
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment;
	filename="0001-xenbus-improve-device-tracking.patch"

>From 4cf5c9300bf8f9517b9b1acafcc95657dafd99de Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= 
Date: Mon, 21 Mar 2022 12:47:20 +0100
Subject: [PATCH 1/2] xenbus: improve device tracking

xenbus needs to keep track of the devices exposed on xenstore, so that
it can trigger frontend and backend device creation.

Removal of backend devices is currently detected by checking the
existence of the device (backend) xenstore directory, but that's prone
to races as the device driver would usually add entries to such
directory itself, so under certain circumstances it's possible for a
driver to add node to the directory after the toolstack has removed
it.  This leads to devices not removed, which can eventually exhaust
the memory of FreeBSD.

Fix this by checking for the existence of the 'state' node instead of
the directory, as such directory will always be present when a device
is active, and will be removed by the toolstack when the device is
shut down.  In order to avoid any races with the updating of the
'state' node by FreeBSD and the toolstack removing it use a
transaction in xenbusb_write_ivar() for that purpose.

Reported by: Ze Dupsys 
Sponsored by: Citrix Systems R&D
---
 sys/xen/xenbus/xenbusb.c | 55 +++++++++++++++++++++++-----------------
 1 file changed, 32 insertions(+), 23 deletions(-)

diff --git a/sys/xen/xenbus/xenbusb.c b/sys/xen/xenbus/xenbusb.c
index e026f8203ea1..b038b63bd289 100644
--- a/sys/xen/xenbus/xenbusb.c
+++ b/sys/xen/xenbus/xenbusb.c
@@ -254,7 +254,7 @@ xenbusb_delete_child(device_t dev, device_t child)
 static void
 xenbusb_verify_device(device_t dev, device_t child)
 {
-	if (xs_exists(XST_NIL, xenbus_get_node(child), "") == 0) {
+	if (xs_exists(XST_NIL, xenbus_get_node(child), "state") == 0) {
 		/*
 		 * Device tree has been removed from Xenbus.
 		 * Tear down the device.
@@ -907,6 +907,7 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 	case XENBUS_IVAR_STATE:
 	{
 		int error;
+		struct xs_transaction xst;
 
 		newstate = (enum xenbus_state)value;
 		sx_xlock(&ivars->xd_lock);
@@ -915,31 +916,37 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 			goto out;
 		}
 
-		error = xs_scanf(XST_NIL, ivars->xd_node, "state",
-		    NULL, "%d", &currstate);
-		if (error)
-			goto out;
-
 		do {
-			error = xs_printf(XST_NIL, ivars->xd_node, "state",
-			    "%d", newstate);
-		} while (error == EAGAIN);
-		if (error) {
-			/*
-			 * Avoid looping through xenbus_dev_fatal()
-			 * which calls xenbus_write_ivar to set the
-			 * state to closing.
-			 */
-			if (newstate != XenbusStateClosing)
-				xenbus_dev_fatal(dev, error,
-						 "writing new state");
-			goto out;
-		}
+			error = xs_transaction_start(&xst);
+			if (error != 0)
+				goto out;
+
+			error = xs_scanf(xst, ivars->xd_node, "state", NULL,
+			    "%d", &currstate);
+			if (error)
+				goto out;
+
+			do {
+				error = xs_printf(xst, ivars->xd_node, "state",
+				    "%d", newstate);
+			} while (error == EAGAIN);
+			if (error) {
+				/*
+				 * Avoid looping through xenbus_dev_fatal()
+				 * which calls xenbus_write_ivar to set the
+				 * state to closing.
+				 */
+				if (newstate != XenbusStateClosing)
+					xenbus_dev_fatal(dev, error,
+					    "writing new state");
+				goto out;
+			}
+		} while (xs_transaction_end(xst, 0));
 		ivars->xd_state = newstate;
 
-		if ((ivars->xd_flags & XDF_CONNECTING) != 0
-		 && (newstate == XenbusStateClosed
-		  || newstate == XenbusStateConnected)) {
+		if ((ivars->xd_flags & XDF_CONNECTING) != 0 &&
+		    (newstate == XenbusStateClosed ||
+		    newstate == XenbusStateConnected)) {
 			struct xenbusb_softc *xbs;
 
 			ivars->xd_flags &= ~XDF_CONNECTING;
@@ -949,6 +956,8 @@ xenbusb_write_ivar(device_t dev, device_t child, int index, uintptr_t value)
 
 		wakeup(&ivars->xd_state);
 	out:
+		if (error != 0)
+			xs_transaction_end(xst, 1);
 		sx_xunlock(&ivars->xd_lock);
 		return (error);
 	}
-- 
2.35.1


--4oNrmEULt/shPi4w
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment;
	filename="0002-xen-blkback-fix-tear-down-issues.patch"

>From 1525f8ea0b35edf6df7633ac217c32711583caec Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= 
Date: Sun, 27 Mar 2022 10:43:42 +0200
Subject: [PATCH 2/2] xen/blkback: fix tear-down issues

Handle tearing down a blkback that hasn't been fully initialized. This
requires carefully checking that fields are allocated before trying to
access them.  Also communication memory is allocated before setting
XBBF_RING_CONNECTED, so gating it's freeing on XBBF_RING_CONNECTED
being set is wrong and will lead to memory leaks.

Also stop using xbb_disconnect() in error paths. Use xenbus_dev_fatal
and let the normal disconnection procedure take care of the cleanup.

Reported by: Ze Dupsys 
Sponsored by: Citrix Systems R&D
---
 sys/dev/xen/blkback/blkback.c | 63 +++++++++++++++++------------------
 1 file changed, 30 insertions(+), 33 deletions(-)

diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 33414295bf5e..15e4bbe78fc0 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -2774,19 +2774,12 @@ xbb_free_communication_mem(struct xbb_softc *xbb)
 static int
 xbb_disconnect(struct xbb_softc *xbb)
 {
-	struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
-	struct gnttab_unmap_grant_ref *op;
-	u_int			       ring_idx;
-	int			       error;
-
 	DPRINTF("\n");
 
-	if ((xbb->flags & XBBF_RING_CONNECTED) == 0)
-		return (0);
-
 	mtx_unlock(&xbb->lock);
 	xen_intr_unbind(&xbb->xen_intr_handle);
-	taskqueue_drain(xbb->io_taskqueue, &xbb->io_task); 
+	if (xbb->io_taskqueue != NULL)
+		taskqueue_drain(xbb->io_taskqueue, &xbb->io_task);
 	mtx_lock(&xbb->lock);
 
 	/*
@@ -2796,19 +2789,28 @@ xbb_disconnect(struct xbb_softc *xbb)
 	if (xbb->active_request_count != 0)
 		return (EAGAIN);
 
-	for (ring_idx = 0, op = ops;
-	     ring_idx < xbb->ring_config.ring_pages;
-	     ring_idx++, op++) {
-		op->host_addr    = xbb->ring_config.gnt_addr
-			         + (ring_idx * PAGE_SIZE);
-		op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
-		op->handle	 = xbb->ring_config.handle[ring_idx];
-	}
+	if (xbb->flags & XBBF_RING_CONNECTED) {
+		struct gnttab_unmap_grant_ref  ops[XBB_MAX_RING_PAGES];
+		struct gnttab_unmap_grant_ref *op;
+		unsigned int ring_idx;
+		int error;
+
+		for (ring_idx = 0, op = ops;
+		     ring_idx < xbb->ring_config.ring_pages;
+		     ring_idx++, op++) {
+			op->host_addr    = xbb->ring_config.gnt_addr
+				         + (ring_idx * PAGE_SIZE);
+			op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx];
+			op->handle	 = xbb->ring_config.handle[ring_idx];
+		}
 
-	error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
-					  xbb->ring_config.ring_pages);
-	if (error != 0)
-		panic("Grant table op failed (%d)", error);
+		error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops,
+						  xbb->ring_config.ring_pages);
+		if (error != 0)
+			panic("Grant table op failed (%d)", error);
+
+		xbb->flags &= ~XBBF_RING_CONNECTED;
+	}
 
 	xbb_free_communication_mem(xbb);
 
@@ -2839,7 +2841,6 @@ xbb_disconnect(struct xbb_softc *xbb)
 		xbb->request_lists = NULL;
 	}
 
-	xbb->flags &= ~XBBF_RING_CONNECTED;
 	return (0);
 }
 
@@ -2963,7 +2964,6 @@ xbb_connect_ring(struct xbb_softc *xbb)
 					  INTR_TYPE_BIO | INTR_MPSAFE,
 					  &xbb->xen_intr_handle);
 	if (error) {
-		(void)xbb_disconnect(xbb);
 		xenbus_dev_fatal(xbb->dev, error, "binding event channel");
 		return (error);
 	}
@@ -3338,6 +3338,13 @@ xbb_connect(struct xbb_softc *xbb)
 		return;
 	}
 
+	error = xbb_publish_backend_info(xbb);
+	if (error != 0) {
+		xenbus_dev_fatal(xbb->dev, error,
+		    "Unable to publish device information");
+		return;
+	}
+
 	error = xbb_alloc_requests(xbb);
 	if (error != 0) {
 		/* Specific errors are reported by xbb_alloc_requests(). */
@@ -3359,16 +3366,6 @@ xbb_connect(struct xbb_softc *xbb)
 		return;
 	}
 
-	if (xbb_publish_backend_info(xbb) != 0) {
-		/*
-		 * If we can't publish our data, we cannot participate
-		 * in this connection, and waiting for a front-end state
-		 * change will not help the situation.
-		 */
-		(void)xbb_disconnect(xbb);
-		return;
-	}
-
 	/* Ready for I/O. */
 	xenbus_set_state(xbb->dev, XenbusStateConnected);
 }
-- 
2.35.1


--4oNrmEULt/shPi4w--

From nobody Tue Apr  5 15:38:08 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BA7E11A9DB6B
	for ; Tue,  5 Apr 2022 15:38:24 +0000 (UTC)
	(envelope-from prvs=087021108=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KXsHM3RdSz4rnY
	for ; Tue,  5 Apr 2022 15:38:23 +0000 (UTC)
	(envelope-from prvs=087021108=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649173103;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=IANSJSenLtoOZCxfyi+6T6lLRuFkQpalQjJM6VYG5SA=;
  b=Lq8owEUt3YCCN3+xM3dGFGi619J0ajj1QneGZczhyXawNeQ13AP+LmC+
   d4j+fTsJzmLZr9r0i0tBA36Ru8D58tByitKQM6p5FM7B4lPV7/ZBnmZ6M
   9mTJX9r1G605cqX6Lq9Ykdr8L7EP8hPUqPND2wFRCyYc+fL6scnHQVxYY
   s=;
X-SBRS: 5.1
X-MesageID: 68471494
X-Ironport-Server: esa1.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:7gTZHq9Z3EOeviDGhFp6DrUDln6TJUtcMsCJ2f8bNWPcYEJGY0x3n
 zFKWj2EM/aNZmL8Ktt+OYiz8kkC6MSHyNZjSFFrqyw8E34SpcT7XtnIdU2Y0wF+jyHgoOCLy
 +1EN7Es+ehtFie0Si+Fa+Sn9T8mvU2xbuKU5NTsY0idfic5DnZ54f5fs7Rh2NQw3YDkW1nlV
 e7a+KUzBnf0g1aYDUpMg06zgEsHUCPa4W5wUvQWPJinjXeG/5UnJMt3yZKZdhMUdrJ8DO+iL
 9sv+Znilo/vE7XBPfv++lrzWhVirrc/pmFigFIOM0SpqkAqSiDfTs/XnRfTAKtao2zhojx/9
 DlCncWgUD10IZDzofUcAwMFQjtEI7dg9YaSdBBTseTLp6HHW37lwvEoB0AqJ4wIvO1wBAmi9
 9RBdmpLNErawbvrnvTrEYGAhex6RCXvFJkYtXx6iynQEN4tQIzZQrWM7thdtNs1rp4eQamHO
 5tFAdZpREXOOUJrN1sTM7M3uu63q0PYKyZjh13A8MLb5ECMlVcsgdABKuH9cMGKX8JKtkCWr
 GnP+yL+GB5yHNKFxDeP6X7pluLJtS3hVY8YD7H+8eRl6HWBy2AOEAYHTnO0pPC4jgi1XNc3F
 qAP0nNw9+5orhXtF4SjGU3jyJKZgvICc4NACtdn9zyy87vVsxzEV3I+UCRnZeVz4afaWgcW/
 lOOmtroAxlmv7uUVW+R+9+okN+iBcQGBTRcPHFZFGPp9/Gm+dhu1UyXEr6PBYbv1rXI9SfML
 ydmRcTUr5EaloY12qqy5jgraBr898GSHmbZCug6N19JDz+Vhqb4PORECnCBtJ6sybp1qHHb4
 RDofODEsYgz4WmlznDlfQn0NOjBCwy5GDPdm0VzOJIq6i6g/XWuFagJvm0vfB0xbptYImO1C
 KM2he+3zMUOVJdNRfUpC79d9uxwlfSwfTgbfq68giVyjmhZK1bcoXAGib+41GHxikk8+ZzTy
 r/AGftA+U0yUPw9pBLvHr91+eZymkgWmDOCLbimnk/P+efPOxaopUItbQLmghYRt/jf/m04M
 r93aqO39vmoeLanOHOKrtdPcwtiwLpSLcmelvG7v9Wre2JOMGogF+XQ0fUmfYlklL5SjeDG4
 je2XUow9bY1rSSvxdmiApy7VI7SYA==
IronPort-HdrOrdr: A9a23:uur4PKjp2qOcubkcVcFDtWeE53BQXzh13DAbv31ZSRFFG/FwyP
 rAoB1L73PJYWgqNU3I+ergBEGBKUmskqKdxbNhR4tKOzOWxVdATbsSlrcKpgePJ8SQzJ8+6U
 4NSdkaNDS0NykHsS+Y2njILz9D+qj/zEnAv463pB0MPGJXguNbnn9E426gYzNLrWJ9dPwE/f
 Snl656T23KQwVpUi33PAhMY8Hz4/nw0L72ax8PABAqrCGIkDOT8bb/VzyVxA0XXT9jyaortT
 GtqX2y2oyT99WAjjPM3W7a6Jpb3PPn19t4HcSJzuwYMC/lhAqEbJloH5eCoDc2iuey70tCqq
 iGnz4Qe+BIr1/BdGC8phXgnyHmzTYV8nfnjWSVhHPyyPaJMw4SOo5kv8Z0YxHZ400vsJVXy6
 RQxV+UsJJREFfpgDn9z8KgbWAkqmOE5V4Z1cIDhX1WVoUTLJVLq5YEwU9TGJAcWArn9YEcFv
 V0Bs203ocbTbqjVQGZgoBT+q3tYpxqdS32AXTq+/blngS+pUoJgXfxn6ck7zU9HJFUcegx2w
 2LCNUsqFh0dL5kUUtMPpZwfSKJMB2+ffvtChPlHb21LtBPB5ryw6SHlYndotvaPKA18A==
X-IronPort-AV: E=Sophos;i="5.90,236,1643691600"; 
   d="scan'208";a="68471494"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=YtOwKBsxMlIF3gcmWuYxSgSVip/NZkRnGt3zsttBM+9QrerZe8iWWAxynVx32YLUl/nb9679Dk+YC7YrK+l8IYnRgEVcfEzaG+dVCB19d/+AmQoM93sGYHmnsftg+FuXCvmW8bM3gR19rmWSTRcleVtwWtr3ZNyD57ymBGkNYrU14z89dzC4SOWMQ+kb/oJ61xV1mmSB7QKyGZ5LqtHEZPH7HxuOMvFp59vlt/NLmOnQisMGbxjO2Vu8eZlJ5dMg3lgA4K3mEQ+xw8mP15tz4sgHf8JKh9CCyeSQvT2/aEwTuR2THocFGU+D5nTNMqNhTmyojVWecgn3ABAYBY2CHg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=BaPoFmPqAf241RB6NWllyMooMbCluDfmp7N4W4eFyKQ=;
 b=gkyRj+H8XZFisSNqvaqbaRZYB+to1yo7bmUbw5gFSaJWeNEOZPFn4hpY2RRFLWIuP3T8hDCOiBBbAjCmAtumn6PgzGzNzsUb60hb2xD+2gJ+qyvgWg6MZdhOIxEmWtqcAI00V0b1iL9K3a6T8pfyxOkg/Mk1s/3uek+G9JnsmYnqOxld1J7Gw3CSf5ZIEMdAFzfE613d4tUOo0QkNZTZk8jcjZygKog67C/zTvFTAT57SYqNBVMqnyKsk333vk6BkElS8CO9qTOLXCQ7pvU6o4NIm8n2xLCF3mD07mGGUpn9n9MggCV5Zd9XI5auFQocoIYBZXqcHApYC4l2UyOtHA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=BaPoFmPqAf241RB6NWllyMooMbCluDfmp7N4W4eFyKQ=;
 b=qLYgeUu1xaRT3r2lG8VupcsW9Hf7xw7tGwDQzNW159YvdKdgFX//YgqiTxNqlhI661DXDWsHXsnKPAcCMQxOyPgjqPTftTY1uaPdyaMEOww2p4+6IyWDNyutSuG6GLdC5DqHrnJclkHR8F6obxXIa3ja07rVYhIXPuXB3Yol9Sk=
Date: Tue, 5 Apr 2022 17:38:08 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
X-ClientProxiedBy: LO4P123CA0509.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:272::13) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 87a72040-3c56-426d-e13f-08da171a4b33
X-MS-TrafficTypeDiagnostic: MWHPR03MB3039:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: P4lMN6tvFq/TQGt2k2gFxM9ry5ReVRZ/+7yx7f0WDBZTAxepx6dNNh+5hyMZag/Dprcdq75+7bH2fczcZvluQ9pnu7O/MunmbPIhayNGoeRQYMU8mIOhocb5q2F8vpvzzy5BkcUWYgTSZ0di7zfJf4gtnVn+UK9uFc0d96CuNrU+LPtp7+zNAdTcf3iFeAtkQs4scoZL6W1NU+5O76s1wxKYGVQ1ktzPPiTR+a/uza04jJ4r7OqQhwLE7a5Bt89LWLlDpVZzyYZdO0Ct9pqhc/SECfOlKxZ0txXss6e14gKo4z2awePGj3TVP4vfaTSZYFoiGnkfCynW23e9XTFXpnzR+/EpbowGYgrF+YP4Z5CwhyOVFM8sESm811ohrKDkM6PcqnfjY7YlduEiBeeN4pEJFNgj3pChPtixsqspFGs+Q9YARkHOag+ubw8FTPZ7kBsHoE1WyigqyvyUTy8YxzGJKO/pO0pgeHpfPLvUaytA4xVbaZ1aR8/N+916v8z3JODZbBo3z7lgMDuekbi/RvurTz9a0bhm0I0Zu8phPXgrf/5WZN4WQWdXvRM6NUEyHKm/ELr53yRc3S7AVZWju4JHG5acRk4NoxjSdYSTRvuzYcsZaIfrbWa9DMoTdUJydyJJhUNSesptbZaML1ZvQxY6WTCQB6vxpFw6u77dw34ypjGViqli7JLTqPwCRv1fX9jyeIELczhdLLyysUnWhxUx58WUP7ZlZCSuFnTUfqE=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(38100700002)(6506007)(82960400001)(5660300002)(966005)(6486002)(2906002)(6666004)(6916009)(9686003)(6512007)(316002)(508600001)(86362001)(8676002)(66556008)(66476007)(83380400001)(66946007)(33716001)(186003)(85182001)(4326008)(26005)(8936002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Tkc3ajVJTkdvejdhQVlCUTV5ZzhWSDdLRlpTRCtjZW5KMWlmVTZHbS9JTUcx?=
 =?utf-8?B?WG1SQjFLbmV3ZXEwbGhzaTNSVVpyL2xPY3V5SDdZaHlyZjZGQjdyZVIvQm5X?=
 =?utf-8?B?bnBIdmkwSHQ3QlNQZzdqQnVYSitWeUk4aFhJTnIwQmFMdlcxVHpISFRjZ3pE?=
 =?utf-8?B?dGxCOWpOMVJDSHIzZjdHVmtMTVBXOVd0UmlveTdnRDl3TmJ3T1JjZkVVbmM5?=
 =?utf-8?B?N0VkNml3Y0IzZ1VPQ09YckNROWJQTUJxcjIxNlBGOXQ1VURaV0N6eUl0Qm0z?=
 =?utf-8?B?VFAybXVrdm5SZ1RNVlY5bWtNbmZxdEVDZ1dSa3RVKzFHWVh5UWd5T3UwbXdR?=
 =?utf-8?B?S3hLUDhsa0R5RFZWaFg5bDZBckYxUnBnV2l4VlV3WmRNamZuS1NDbFZPRVMv?=
 =?utf-8?B?Y3hZN0o4Y3U0aFZLa1NWMGI4Wm82bUJuV05pOVVCZUlRaXl6aHVSeGRab2xy?=
 =?utf-8?B?N3pZcVVubkV3UjNqV1VRb3pXSW56VW43THJEQVF0aDY1Tk1OU2cyMlBGbkhN?=
 =?utf-8?B?TnBBd2NMbnZtbkZTMTFyNGQxczdDY2UzTzhLUGc4d3h1WkhNU1g5bThlMGw3?=
 =?utf-8?B?SnN2alNDeXd1eU42cy9VeitydUVyNzA0SytmMnVQN1grcjhReUZyaWZvQXdm?=
 =?utf-8?B?Tkc2bldQMU1oNDJQK3BneEdzQ1dBVzgwTzhhdmZKMW5hM1FvditFbFBUMXpZ?=
 =?utf-8?B?L2lVaWpWd3pORjZWV2tBR05DckNyNmVKcjBZV1dVYmREWFRvZlNPcXVpU0NP?=
 =?utf-8?B?bE9hNEJhOFZlY29tbzYySmJoaEM5cHhRVTkzWkVHV3lJQkdaaDJkd0FNTWoz?=
 =?utf-8?B?QjY0RzVYRVRoUUl6TTNvS3RTK3VDRlJCSnRHTUR2RGdzSXA3MDNnTll3c3V3?=
 =?utf-8?B?aWhXM1RvdFd1dlNoM1RxdVZrYmxvUVJoSWVlVmxFQkxoa2pPd2MyNXdsSVZC?=
 =?utf-8?B?QTJIdm14SVFCWWZCUWlsOVFMYjhWT01tN1RhV2JaSWNzM2JmTDhkY2l0bld6?=
 =?utf-8?B?V09aRm92YlNBbTFnM0c0cW1CWlE5VEJCemxVbmVVVGhHNFlYTU9PZ291UVpv?=
 =?utf-8?B?emY5RnZJbnpYbDZUZzV3andpd2FnRGw1Z09mMzNrZCtUVDhnSStCRTRWMTRC?=
 =?utf-8?B?VklpQ1hIa25MMTdCNFAzaTRIbi9TQ2R6QkRsYVVOblBuTzNiNFBrSENqN2o4?=
 =?utf-8?B?MGVYbWVRQzNRei9NZHVDcGViVHlpTEJUejVaRlBuOVhIZ3NKSmFzZXlDTWFZ?=
 =?utf-8?B?aVVkcEhERmtud1Z1Y255ZGVzam1ac1lXWjU0OVZHY2VZMlV4NUpIZ0FVMDRk?=
 =?utf-8?B?S1NVOTJsbFl5eFVzSmpSZTZ3aDlwM2tLS1dDOTA4N0R1MXdFdW5TZjhwWkhB?=
 =?utf-8?B?RnlUbzd1Y29jMkE5WnVzakZUYmwyVXU1Y014Vk01cU1vdGM1aTFCQXgxYmtK?=
 =?utf-8?B?MlZYSUtlV1cyL2NicTQ5UCthUEpvV3ZCMGtHUXdUbGZ6RTg1aDdadEQyMGNq?=
 =?utf-8?B?WVlTN2NwUS9ZQ2NRU2ZLN3Fjd1N5bGgxTlZRT04yR2hQdlp4V3dYS0V1bzZq?=
 =?utf-8?B?aW5FT0lnV0FFVXEzTVZ6UlVpYzAxVmtZSTQ0b3lnNWNjUFVUNUV6ditOVjBp?=
 =?utf-8?B?bnIzYnlYL0ZQOEZ1WW1TQlR0eFJCRnlpQTVvV1AyUGljRGlvcEZuZ1Bnb0VC?=
 =?utf-8?B?a0M3V3E1UGFHZjRkY1hueG5aVlo2Ym5vQjd5VE83U0ZQWVBQb2U2bDArMmhs?=
 =?utf-8?B?aTVSUE9zbWxuaGVaSVI5WitvRDRJZFFBNGV0UFY3dllaS0pYb3Q1bjN6Z1Nt?=
 =?utf-8?B?V2E3UFJjbGwyWENaaldFTmlmM1pPZzVxUFR5b1pSaitIUDREbE0xN1FlNHpV?=
 =?utf-8?B?YjAwY1hPanVMRFlhdGswenZMWFhwNjBGUTJXMStaaGNFNHd4eUp1VWhTbXR1?=
 =?utf-8?B?ZTNYSHpnWml1eVdUMDhMbDFMd1prbFAvN2ozZHc3VEEwdjltSHZBU0NQUEhW?=
 =?utf-8?B?T2t1ZFNyS0RVK3N3NlBhRktlU0JaUlZVajVYVHNjL01Sd0NGVEx0RzlzMm5O?=
 =?utf-8?B?U0lXTmhZNDA2MmxoOUdCS1Q5NzZKTUg4N1ZTaVVCNHZiN3dQd0pWZ0ovdUhO?=
 =?utf-8?B?bWdUZlV4ZVFnWlZPTm5MUFdSNTBuK2MrQXE2cFVRS2orUzFOaVltUnE4M2s1?=
 =?utf-8?B?a00rZEY1T042ckc2WXNOVXlhRzNIQUxjSzk3dHlYSm8rRllFUUpjYzhJR21E?=
 =?utf-8?B?WTBrR0Vad3pEejlnVVJ3d0pzb2dMNHRLT3NreUNMRTBaQWxlOTNkRkUvZFhB?=
 =?utf-8?B?bnl2ZXp5NDhuUmRlTkFkUUxQRktDcDFjMWlUcnUxUTJCaG50cFIrSGtKUnZD?=
 =?utf-8?Q?GNS7NeJCl8Vhvco4=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 87a72040-3c56-426d-e13f-08da171a4b33
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2022 15:38:13.0407
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3Qwek1Yd5OKPuA2IQ87MYprgZjQs21gnuSainuB4s78eRgeKwIaGSX0z3ziBtz73Gsyls5glPFjPgaLbuloUyw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3039
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KXsHM3RdSz4rnY
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=Lq8owEUt;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=qLYgeUu1;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=087021108=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=087021108=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=087021108=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sun, Apr 03, 2022 at 09:54:24AM +0300, Ze Dupsys wrote:
> On 2022.03.27. 12:13, Roger Pau Monné wrote:
> >> ==== COUNT: 1
> >> Fatal trap 12: page fault while in kernel mode
> >> cpuid = 1; apic id = 02
> >> fault virtual address = 0x148
> >> fault code    = supervisor read data, page not present
> >> instruction pointer = 0x20:0xffffffff8248cef4
> >> stack pointer         = 0x28:0xfffffe009941d9a0
> >> frame pointer         = 0x28:0xfffffe009941d9a0
> >> code segment    = base 0x0, limit 0xfffff, type 0x1b
> >>        = DPL 0, pres 1, long 1, def32 0, gran 1
> >> processor eflags  = interrupt enabled, resume, IOPL = 0
> >> current process   = 0 (xbbd1 taskq)
> >> trap number   = 12
> >> panic: page fault
> >> cpuid = 1
> >> time = 1646129773
> >> KDB: stack backtrace:
> >> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> >> #1 0xffffffff80c09f01 at vpanic+0x181
> >> #2 0xffffffff80c09d73 at panic+0x43
> >> #3 0xffffffff8108b1a7 at trap+0xbc7
> >> #4 0xffffffff8108b1ff at trap+0xc1f
> >> #5 0xffffffff8108a85d at trap+0x27d
> >> #6 0xffffffff81061b18 at calltrap+0x8
> >> #7 0xffffffff825cb76e at zil_open+0xe
> >> #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2
> >> #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88
> >> #10 0xffffffff80a7f214 at xbd_instance_create+0xa394
> >> #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> >> #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> >> #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> >> #14 0xffffffff80bc7e3e at fork_exit+0x7e
> >> #15 0xffffffff81062b9e at fork_trampoline+0xe
> >
> > Hm, those last ones are in ZFS code, can you try to get the line
> > numbers for those?
> 
> The thing with these ZFS cases is that, luckily i had saved copy of 13.0-p7
> partition, it was official build, but the trace for me seems un unsable for
> unknown reasons. It feels that kernel.debug file is the does not match
> kernel, but i did manage to install 13-p7 in VirtualBox as well and md5 sum
> for that one is the same as in my backup, so i don't know what to think,
> because i expected that it would point to blkfront.c and not random lines in
> blkback.c.
> 
> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line
> -e /mnt/gpt_root_13_p7/usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:843
> /usr/src/sys/amd64/amd64/trap.c:915
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/exception.S:292
> ??:0
> ??:0
> ??:0
> /usr/src/sys/dev/xen/blkback/blkback.c:2183
> /usr/src/sys/dev/xen/blkback/blkback.c:1772
> /usr/src/sys/kern/subr_taskqueue.c:478
> /usr/src/sys/kern/subr_taskqueue.c:799
> /usr/src/sys/kern/kern_fork.c:1075
> /usr/src/sys/amd64/amd64/exception.S:1093
> 
> From official update servers kern.debug for 13.0-p7 can be downloaded:
> fetch http://update2.freebsd.org/13.0-RELEASE/amd64/f/f96dbe024c25febf8a798b5b3d6ca717c2e18f79cc6060cb714c3b3681404608.gz

Let's see if you can get another trace of this panic, because I don't
seem to be able to make sense out of this. I've tried looking at the
RELEASE-13.0 sources (there's no blkback related changed in p7 or the
ones before), but the file:line doesn't seem to make much sense.

I would assume the call is from the:

(*dev_data->csw->d_strategy)(bios[bio_idx]);

Call in xbb_dispatch_dev(), but that's not exactly where gdb (or
addr2line) points me to.

Thanks, Roger.

From nobody Fri Apr  8 15:02:24 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3C7AB1AAE930
	for ; Fri,  8 Apr 2022 15:02:43 +0000 (UTC)
	(envelope-from prvs=090ff2996=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KZhLn2v85z4cXp
	for ; Fri,  8 Apr 2022 15:02:41 +0000 (UTC)
	(envelope-from prvs=090ff2996=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649430161;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=GtYutd2XGhpXBKU2RYOvChOxEmqJZfZlGEnIFiBtuSs=;
  b=S9kLh1lZfTD7bitn+6dQBVBIZvxqWg4XEHcUUHiimR3g9JlYY0lH7itF
   gLKrK5bXxAQsFZEqcGcE8/qs2WKo6BFLu38ICEuNskaRp9XESczWAgay0
   RdFiQVBekSITBauebmS0NSJdYzttwsoYnnJ3xpWyLMTEV3JGT+3wWFI/y
   Q=;
X-SBRS: 5.1
X-MesageID: 68779674
X-Ironport-Server: esa1.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:CYu2BqAj7Dd4axVW/z7jw5YqxClBgxIJ4kV8jS/XYbTApG8j02MHm
 2ocXmjQP/jfNGunc4p3b43j9E8DuZfcnIAxQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA
 xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgHWeIdA970Ug5w7Jh0tYy6TSEK1jlV
 e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH
 I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPgv7
 fZMuc2PDjwKP4rcuu9EWTJbTHpHaPguFL/veRBTsOSWxkzCNXDt3+9vHAc9OohwFuRfWD8Us
 6ZCcXZUM07F17neLLGTE4GAguw5K8bmJsUHs2xIxjDFF/c2B5vERs0m4PcGjGhv154RQZ4yY
 eIINThfbjKaYCddZFlGDIomptiYnFbgJmgwRFW9+vNsvjm7IBZK+LLwMMDSZfSMS89el0Leo
 XjJl0z3GBwXPseTjCqI9FqinOLClzj3HoUIG9WQ5PNmnUWI13c7BxgfVF/9qv684nNSQPoGd
 RZSoHB36/Fvqgr7FbERQiFUvlbamz0dV4tvVNYZ6Sad6KGN6S2aWTM9G2sphMMdiOc6Qjkj1
 1msltzvBCByvLD9dU9x5ot4vhvpZ3FLcDZqiTssCFJcvoK9+N1bYgfnFI4LLUKjsjHi9dgcK
 RiupTN2ubgchNVjO06TrQGe2GLESnQko2cICuTrsoCNs1sRiG2NPdXABb3nARBodtvxor6p5
 iVspiRmxLpSZaxhbQTUKAn3IJmn5uyeLBrXikN1Ep8q+lyFoiD/LNoAvWwuexgwa67onAMFh
 meJ5Gu9A7cJYhOXgVJfOdrtW6zGM4C+fTgaahwkRoUXOcUgHON21CpveVSRzwjQfLsEyskC1
 WOgWZ/0Vx4yUP0/pBLvHrt1+eJ7l0gWmDKILbimnkvP7FZrTCPMIVvzGADVNb5RAWLtiFi9z
 uuzwOPWk04PD7CkOnC/HEx6BQliEEXXzKve8qR/XuWCPhBnCCcmDfrQyqkmYItrg+JekeKgw
 513chMwJIbX7ZEfFTi3Vw==
IronPort-HdrOrdr: A9a23:dTvCYq1hy/zaOkP60K+S4gqjBVZyeYIsimQD101hICG9Lfb3qy
 n+ppsmPEHP5Ar5OEtBpTiBUJPwJ0800aQFnLX5Wo3SIDUO2VHYVr2KiLGC/9SOIVyaygcw79
 YFT0E6MqyOMbEYt7eL3ODbKadZ/DDvysnB7o2yvhQdLz2CKZsQlDuRYjzrY3GeLzM2fKbReq
 Dsgfau8FGbCAoqh4mAdzI4dtmGg+eOuIPtYBYACRJiwA6SjQmw4Lq/NxSDxB8RXx5G3L9nqA
 H+4kHEz5Tml8v+5g7X1mfV4ZgTsNz9yuFbDMjJrsQOMD3jhiuheYwkcbyfuzIepv2p9T8R4Z
 PxiiZlG/42x2Laf2mzrxeo8w780Aw243un8lOciWuLm72OeBsKT+56wa5JeBrQ7EQt+Ptm1r
 hQ4m6fv51LSTvdgSXU/bHzJl9Xv3vxhUBnvf8YjnRZX4dbQqRWt5Yj8ERcF4pFND7m6bogDP
 JlAKjnlblrmGuhHjDkV1RUsZ+RtixZJGbFfqFCgL3Y79FupgE586NCr/Zv20vp9/oGOu55Dq
 r/Q+BVfYp1P70rhJJGdZQ8qPSMexnwqDL3QSuvyAfcZek600ykke+C3Fxy3pDsRKA1
X-IronPort-AV: E=Sophos;i="5.90,245,1643691600"; 
   d="scan'208";a="68779674"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=EV3f7ghWyq2S9UXMDv1eKia1G88hC4LNGGsWcyvJ35ozuQ+L0vSLug36MLG/PIVl3rb04ql6M6nTCx7Fj63dWxmkHx0jXDfEwJ2LGY6hhG5sY1c0ozuxSZCrZN2cWvrcIpgJsCt0DpMO3JLCeXkrhAzuATlKuZ9bd9OKWLcctmq/xYPj2K4a4pzBxOGV1rmJ9Xsk7tzviuy/DmCEwvVwPOkIw1djdlbTLeWEwSAl9nD8LIpFFiDLiRVN0SzkR7w6T0eQU7zuOZfhHbCkBnWT3BW+HwOll4Sp/VQ5qSFH2mnKVWw8OqoGockuyDHbq02u2Bs+FKAk1k5+983GlsRpwg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=Mofduz2sYvuo81prsvzEtn7qbZtm+Yq5cGL4jSLOLyw=;
 b=XCpBSBCZUJqiwNZlG7asxPwhJliV1GSPnZzGL2GKu5AUy5xelo3uwRYTxuyAGa6sxCYYm/EjUsY2QrOYcXQnE5ZoZG+T2CcoMHO0WE+tiVpDsfAcR/ub7RCbX/1MwVEmqwMrRe2KsLTAyE55N0a/mogoF6MZe9VZaL4smcA36ALe4gLJwDMNW+7j8DUpij3U6DWxIGGRBOjLfFX8IqnDmRYbEdAf23XZeSmiDHAPznBnhFiMS0t66bbh+L/VykDAKk7nWPIqsccYOXJ1bnviaLCCeUdxpTkGCseIcrbS9jE0Th4cUput6piZcidUoW0QGpqHE/lAMsloZ6JxUg7JKA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=Mofduz2sYvuo81prsvzEtn7qbZtm+Yq5cGL4jSLOLyw=;
 b=VrVwXMn+MV1i94spPLVJpL/S6yi/BPCPqWax6pJ8EW0il3jXwydhAtiZEUgU7LYvTFcq6jRsuc1vNUM2NPVpB9uZTsYo1gFhgWA3dHuI17DiwsyW/JJ65n56X4NbtwjADzV3EEfLWF5mT0wcknLrhhCzy4pw8ESf1fBrOYRAW4o=
Date: Fri, 8 Apr 2022 17:02:24 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
X-ClientProxiedBy: LO4P265CA0081.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:2bd::9) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 1089a28c-1e0a-4e52-2326-08da1970cce0
X-MS-TrafficTypeDiagnostic: CO6PR03MB6225:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: +xaB3GBTtsEIcslOLAQXtv18AjAEYpR3odE80Vk8qTtm1CRUMVqtYV6e2Q21ME6g39KkFjyqBDpgGGNOtw6Ics8jPZfM2iU7gmt/MN/w3xUR0p+76Q1/7hT/dZmXa/5+ccCtEX6DJWUbId6tiJRHnu+eftnOFxVc4fZpBPcGyF9u7HpEe7N/xASDt6niiy6qQ8kIbqFaURFNZjrgVeCg8SyMbrTse0/EsMatbBv1E0ajj07sS4WhHxUj/XAhQZLZGgA7g8TZZdio+NMenQJMYhkxeZ7YqAXhSkH1NEx7zSRttauzUGBCWJy/wYR6102qteFqLx74C+C+s2fe8pOSi0IsfgXJcXJ/CJZ+kw74EnyN741DesM7AKUtrQsKLBzqAvdan7Bhhqt3ai2booX/UHN8s5e/5QI3jNJ+MRpabgv0OlcXEeOnIrrmcOYhmcLKU7PBP0cSmJKdX+asf4hdT1q1sN8JhukqE8WstpPOJ3tD7Nk9k2FTzf/CsNabxw+VpLQ8O7UK2bXzuBiA3MK+nkqWQAWGgjiudF4mlZXjEf7z3TDwj6SwohxxNsNZqcLhfw3MlatLBvlC5xJ/guS8B1CD6kfmZDETqmKVRCZ9TFzquAWMjtNS0yHr+8Yzj969LeOJ+JSHp3OKuAn5reF2DY2JPktlWez4qG+iFyUBLxaC1h4vZ4SFrX7R3sEl4cRgbGfLKoUCsU3HjeQYHNgUGVPgD3GSJ9UZ9F3W9FNTiRE=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(82960400001)(4326008)(66946007)(8676002)(86362001)(66556008)(2906002)(66476007)(8936002)(38100700002)(26005)(966005)(186003)(6486002)(5660300002)(6666004)(33716001)(6512007)(6506007)(9686003)(508600001)(85182001)(6916009)(316002)(83380400001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cEdzNXdKaEMxME5jZ3pKblRUTm5wR2l6cGZjTTZvbUpnN1RJM2lyRnBlS084?=
 =?utf-8?B?MnVXUzhzQ1BsN2ZFak1IWTVmbTV3aE9LSHNQOWdmL2RjYXU2SmxlYjN1QjVw?=
 =?utf-8?B?bmVVT0Y2WVRxY3ZlaTlDMS9CaURjclBQalM5ZDNFTXhlYStlNzBpYnlnWm5S?=
 =?utf-8?B?WjdQQ0kvc00yN1lHdmRpS3lwNmtvWjZqZy9nVkxuZEVyQlRUU3M5ZzhvTXVq?=
 =?utf-8?B?Sys3azBSdGZ3V1lsOHR4ZTFGbFdOTG1xbjBzd0dqVWRTeUZkaVoxVG5wTXJn?=
 =?utf-8?B?cjZXdFJsQUxVdkE1ajl5ZGNaVnBpcFR4MVVsMjRWNEpORXgxWU82ZHYxUnFa?=
 =?utf-8?B?OUlvZ3ZvN1FNNjhzck8yWVlvWmE0NU1NZWxDNWZKZ0xBbXU3dER0QWE5c0J5?=
 =?utf-8?B?Y2I3U0xoNldkNXBEQmF0azFMVUIvaGtYcHBlUlRsSzhtdEFEQTFrMjF6cm1t?=
 =?utf-8?B?SlczUExSbEtjM2dYTVhtWURJa3BHclVOTDZ5cDVKMlV6ZTZLMktQVHNUZTBZ?=
 =?utf-8?B?M2xsZUJlT0N2MU5Wc1lGRG9XVUIzZTFKYm1pQkNIZnJsOEVaOGNiR3kxUm40?=
 =?utf-8?B?MkRrZ1lRNE5MeDhKTnBzaUptdjZKOHVNa29wYkRxc1NRMnByUVZRdVg1bkFD?=
 =?utf-8?B?UTZ4RzZYSmdHbjFBV3JYTzV1clc3YmdDcnRtdkRWSUxzSkxPU09XSEJLbHc5?=
 =?utf-8?B?TncxY1BBcTBvWXhLaGdBcmQ5YzlQKzF2M2swZmlyTHFKcWM0MnJjNitvS1BP?=
 =?utf-8?B?Z3ZCN0duSThLRFhjQTl5SnlFeXdWbS96cTBzSjg2YWZ1Q2hYYlBNMkpUTHl4?=
 =?utf-8?B?eGZKUGd1V0lzTE5iQnRVaG83dURMamRqVUJhZXNBNXdiVzhWU1ArQ2c1K0p3?=
 =?utf-8?B?Z2ovcWlDMWkvSDZ0azR3SFFvZ2xUSy9ZUjNmNENBNUp0bFNLNGd2NXZoQkJq?=
 =?utf-8?B?NHVzNDdvZVpLUU1ZeUoxbFFYZ2ZwUllUTTVoYXJ5LzRyQTh6UWErMk9nUEha?=
 =?utf-8?B?N0drMVlyekg3R1pCcFZtV0lyUnFsY0lYVGRNTm5IaHowYm5yalo5TkJtWXR0?=
 =?utf-8?B?Y2N5ViswVWtqbmtJQW5sc3krbnd0b242YmcvY0RWVzhEbHBwWUFqVjl4b21k?=
 =?utf-8?B?TDR6S2UyRnROeG9ESnkwMHNaLzR4cDhKMDdYSnZOY2dIc3E4cW1hUHpNV1Bw?=
 =?utf-8?B?UFpsaVdyMUVFNkY1WEFSZUwvK0RuUzdKbmtTcHV6VWVkcE1HYmE4bm55U09s?=
 =?utf-8?B?dnArbDFNNEZDSVdLNlV6ZG80QUtZNlV6UFhpZ2NyemZzTHVZWnlNQjdzK0lO?=
 =?utf-8?B?SmVzWE9VUGlwam42QkxyQStBUEgrL0pjQUpwRlRneTFZUmg3UjBDK0RHVUl4?=
 =?utf-8?B?L2dPUHRrU2JOazlvTG9pYld0ZXlCN1gwVlMxb0VVYU1JMzJ1ZjhuL3BFVHJl?=
 =?utf-8?B?bTVmekl1UnlVOUhQZXdCOWlIdEtOVVVsYVJBTi9rNFZBNDJBS2tzSG9KWUw5?=
 =?utf-8?B?SGJmbmd3V2lGVTJybmFRYnBJNnE5TXBUQ1FGZjZRcUFJMkVhQzdqcFJmOUVy?=
 =?utf-8?B?SzdsSmJDdnhLSzArT1grLzJsbG94bDN1eFI3bXR0S09WSGdBblpJd3QvUmZ4?=
 =?utf-8?B?TXJCYmNRcUwrK0VMTTZ0cGEra080OTZIVkJQeXJZdm1IYk9HeThxVW1jUXV2?=
 =?utf-8?B?eEp5RnYwWVdnTmYzUlprMzBldDZpNUZIYnR3amtjdmdvdTE2UzZhNStBbk01?=
 =?utf-8?B?YzZMMzI1eWdpVXNDSTQ1R21tRWw5RTJvTjJoV082QXFRQ0dvaVhxTUxaZDg0?=
 =?utf-8?B?elhLL3pVMkpBSS9mdUVWMllKTlF0elU2UUgrQXFRQjRJN3VaVlRnbEwwTzJF?=
 =?utf-8?B?VHlmZTQyMTlMQk5ZZUcyRE9qMmx4Mlo1SVVXZ0VIc0VPOE9NL3pra3NBcWJD?=
 =?utf-8?B?cUdHb1JKeXNjdEd1MWVjRFFBdlc3RjloSDh2a0NJVlRtejMwSEtIcWl3SU40?=
 =?utf-8?B?ZXB1aE0yU2dIVXFqbld6cGpYbCtRUkpReWxMNW1qZ09YMHRERHJtSjhtczlM?=
 =?utf-8?B?Q052UTJWNW1KVlI5Y1U1UXdzLzFZc0s1T2U1UExWYXVmY1FyazdTNHpnV1ZX?=
 =?utf-8?B?c2dFZ1dodktId3BlQW5GdjI0OXkzMUcya1RxL2lCQUUyaW9YNWI0UXQ1TGVp?=
 =?utf-8?B?TmNsMVoxSUozd0pHVnp0NHBrK05BdjN3Mm0ranc1cGRqOUZKREltd2x1M1U0?=
 =?utf-8?B?Y3RVUnNHL3AxbUxXMzhQbUZkakoreW1NcmxYQVBibnhjbHJIVGtzR1NvR04r?=
 =?utf-8?B?YzgwZzI2YXA1R1NpclpFbW5EeDZSZDY2dVBFSkU3a2MzVys3eDluUTVjYjlR?=
 =?utf-8?Q?B3R0V2KT1wDknIFc=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1089a28c-1e0a-4e52-2326-08da1970cce0
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2022 15:02:29.3274
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xgoCi/ran7Tyo1brMhBTmj35J9Z/c/I7aCvVpY0ShJ4EwDsWxLS/Rpmz7eZ2DDWaH6pnvfUqT6w/9UPwXjRaiw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR03MB6225
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KZhLn2v85z4cXp
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=S9kLh1lZ;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=VrVwXMn+;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=090ff2996=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=090ff2996=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=090ff2996=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Fri, Apr 08, 2022 at 10:45:12AM +0300, Ze Dupsys wrote:
> On 2022.04.05. 18:22, Roger Pau Monné wrote:
> > .. Thanks, sorry for the late reply, somehow the message slip.
> > 
> > I've been able to get the file:line for those, and the trace is kind
> > of weird, I'm not sure I know what's going on TBH. It seems to me the
> > backend instance got freed while being in the process of connecting.
> > 
> > I've made some changes, that might mitigate this, but having not a
> > clear understanding of what's going on makes this harder.
> > 
> > I've pushed the changes to:
> > 
> > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
> > 
> > (This is on top of main branch).
> > 
> > I'm also attaching the two patches on this email.
> > 
> > Let me know if those make a difference to stabilize the system.
> 
> Hi,
> 
> Yes, it stabilizes the system, but there is still a memleak somewhere, i
> think.
> 
> System could run tests for approximately 41 hour, did not panic, but started
> to OOM kill everything.
> 
> I did not know how to git clone given commit, thus i just applied patches to
> 13.0-RELEASE sources.
> 
> Serial logs have nothing unusual, just that at some point OOM kill starts.

Well, I think that's good^W better than before. Thanks again for all
the testing.

It might be helpful now to start dumping `vmstat -m` periodically
while running the stress tests. As there are (hopefully) no more
panics now vmstat might report us what subsystem is hogging the
memory. It's possible it's blkback (again).

Thanks, Roger.

From nobody Mon Apr 11 08:47:50 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 919541A8A86F
	for ; Mon, 11 Apr 2022 08:47:52 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KcMtv6kNzz3QY1
	for ; Mon, 11 Apr 2022 08:47:51 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x22f.google.com with SMTP id s13so19199876ljd.5
        for ; Mon, 11 Apr 2022 01:47:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=0V04Nn7wi/w3jXd6dbY9dqnH636OKc5tAxmCiR3gkNE=;
        b=CPXzP3rDMbGhh8K3MQvRLcqKrYRhxfvLl9vnY45otBgqx2yjnad9rm9LhBlRqdI2L3
         lXXns9VY6Kg/cfHr2wwCF+Jp4wEq8Y7RzL3PteoC522zDr1nTOUcnVkYQoi0lX3GznUo
         ESS0jq/0/AnbfubG1UeGSrV+jHMsMU3mta5sAZmc5X41D2mzAE39u0vDuEQ7nerSv1Ir
         jYeiKeJvP5UzdfywNr1l5X6XCCkv9T19NT11jYXfWaaK226SLkUhfqK7Ezdc3kTNe38E
         4wJyl632jI31cjwZE+61rKdiBy9yxbdrQuYyc+UdN1JTPy0dd8vy3cd862Gs45AHYfRI
         vNIg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=0V04Nn7wi/w3jXd6dbY9dqnH636OKc5tAxmCiR3gkNE=;
        b=uTtpwwuq1P1duzi7KA7eFdxBAkFj0lih6JcLelvxtnPqCbIKWobDG0GnrFndLLB+Rr
         2PRky2qrE8jmvqYdejMb3t99LJOPwyOW5M5z4xgRNOb131IXgxfP4+mT73DOZF+6yInk
         0ixjv/NKREQCkad/hHaVMzCq7sKUM4u7b5u75r5apZS6Gckr9mD1siYP5GpGPpEecfZP
         73mQmu4L9UBDnvXCV2hLpi9hWE+a4P9wqK3cNVQ/2oZs023Az/1GYpK6XD3qjqe11fpU
         C3cYgvATaRF7umL3E4AV2d2I+E7W2weju4dp15yNKuOfS5gEKqfYhNCtH6fk3yZ9/Dni
         7wOQ==
X-Gm-Message-State: AOAM533iv1039dJ0BA8152nDhlEyXv2uJuGwLCtP0xamqHhQ8rA0FvR7
	m/2EJ8x8kEN8CrlqYxT/fYE=
X-Google-Smtp-Source: ABdhPJxMYO/B30MHnxDt96v89SiLsCs+mYVGLxTW0MyLSnWhxAqelXlxao60zs2wKy48ut/DlMU+yg==
X-Received: by 2002:a05:651c:146:b0:24b:4782:e6e9 with SMTP id c6-20020a05651c014600b0024b4782e6e9mr12116604ljd.224.1649666869912;
        Mon, 11 Apr 2022 01:47:49 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id x1-20020a2ea7c1000000b0024b6155de11sm355592ljp.122.2022.04.11.01.47.48
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 11 Apr 2022 01:47:49 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
From: Ze Dupsys 
Message-ID: <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
Date: Mon, 11 Apr 2022 11:47:50 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4KcMtv6kNzz3QY1
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=CPXzP3rD;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22f as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-3.85 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-0.86)[-0.856];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-0.998];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22f:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.04.08. 18:02, Roger Pau Monné wrote:
> On Fri, Apr 08, 2022 at 10:45:12AM +0300, Ze Dupsys wrote:
>> On 2022.04.05. 18:22, Roger Pau Monné wrote:
>>> .. Thanks, sorry for the late reply, somehow the message slip.
>>>
>>> I've been able to get the file:line for those, and the trace is kind
>>> of weird, I'm not sure I know what's going on TBH. It seems to me the
>>> backend instance got freed while being in the process of connecting.
>>>
>>> I've made some changes, that might mitigate this, but having not a
>>> clear understanding of what's going on makes this harder.
>>>
>>> I've pushed the changes to:
>>>
>>> http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
>>>
>>> (This is on top of main branch).
>>>
>>> I'm also attaching the two patches on this email.
>>>
>>> Let me know if those make a difference to stabilize the system.
>>
>> Hi,
>>
>> Yes, it stabilizes the system, but there is still a memleak somewhere, i
>> think.
>>
>> System could run tests for approximately 41 hour, did not panic, but started
>> to OOM kill everything.
>>
>> I did not know how to git clone given commit, thus i just applied patches to
>> 13.0-RELEASE sources.
>>
>> Serial logs have nothing unusual, just that at some point OOM kill starts.
> 
> Well, I think that's good^W better than before. Thanks again for all
> the testing.
> 
> It might be helpful now to start dumping `vmstat -m` periodically
> while running the stress tests. As there are (hopefully) no more
> panics now vmstat might report us what subsystem is hogging the
> memory. It's possible it's blkback (again).
> 
> Thanks, Roger.
> 

Yes, it certainly is better. Applied patch on my pre-production server, 
have not had any panic since then, still testing though.

On my stressed lab server, it's a bit different story. On occasion i see 
a panic with this trace on serial (can not reliably repeat, but 
sometimes upon starting dom id 1 and 2, sometimes mid-stress-test, dom 
id > 95).
panic: pmap_growkernel: no memory to grow kernel
cpuid = 2
time = 1649485133
KDB: stack backtrace:
#0 0xffffffff80c57385 at kdb_backtrace+0x65
#1 0xffffffff80c09d61 at vpanic+0x181
#2 0xffffffff80c09bd3 at panic+0x43
#3 0xffffffff81073eed at pmap_growkernel+0x27d
#4 0xffffffff80f2d918 at vm_map_insert+0x248
#5 0xffffffff80f30079 at vm_map_find+0x549
#6 0xffffffff80f2bda6 at kmem_init+0x226
#7 0xffffffff80c731a1 at vmem_xalloc+0xcb1
#8 0xffffffff80c72a9b at vmem_xalloc+0x5ab
#9 0xffffffff80c724a6 at vmem_alloc+0x46
#10 0xffffffff80f2ac6b at kva_alloc+0x2b
#11 0xffffffff8107f0eb at pmap_mapdev_attr+0x27b
#12 0xffffffff810588ca at nexus_add_irq+0x65a
#13 0xffffffff81058710 at nexus_add_irq+0x4a0
#14 0xffffffff810585b9 at nexus_add_irq+0x349
#15 0xffffffff80c495c1 at bus_alloc_resource+0xa1
#16 0xffffffff8105e940 at xenmem_free+0x1a0
#17 0xffffffff80a7e0dd at xbd_instance_create+0x943d

| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e 
/usr/lib/debug/boot/kernel/kernel.debug
/usr/src/sys/kern/subr_kdb.c:443
/usr/src/sys/kern/kern_shutdown.c:0
/usr/src/sys/kern/kern_shutdown.c:843
/usr/src/sys/amd64/amd64/pmap.c:0
/usr/src/sys/vm/vm_map.c:0
/usr/src/sys/vm/vm_map.c:0
/usr/src/sys/vm/vm_kern.c:712
/usr/src/sys/kern/subr_vmem.c:928
/usr/src/sys/kern/subr_vmem.c:0
/usr/src/sys/kern/subr_vmem.c:1350
/usr/src/sys/vm/vm_kern.c:150
/usr/src/sys/amd64/amd64/pmap.c:0
/usr/src/sys/x86/x86/nexus.c:0
/usr/src/sys/x86/x86/nexus.c:449
/usr/src/sys/x86/x86/nexus.c:412
/usr/src/sys/kern/subr_bus.c:4620
/usr/src/sys/x86/xen/xenpv.c:123
/usr/src/sys/dev/xen/blkback/blkback.c:3010

With gdb backtrace i think i can get a better trace though:
#0  __curthread at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump at /usr/src/sys/kern/kern_shutdown.c:399
#2  kern_reboot at /usr/src/sys/kern/kern_shutdown.c:486
#3  vpanic at /usr/src/sys/kern/kern_shutdown.c:919
#4  panic at /usr/src/sys/kern/kern_shutdown.c:843
#5  pmap_growkernel at /usr/src/sys/amd64/amd64/pmap.c:208
#6  vm_map_insert at /usr/src/sys/vm/vm_map.c:1752
#7  vm_map_find at /usr/src/sys/vm/vm_map.c:2259
#8  kva_import at /usr/src/sys/vm/vm_kern.c:712
#9  vmem_import at /usr/src/sys/kern/subr_vmem.c:928
#10 vmem_try_fetch at /usr/src/sys/kern/subr_vmem.c:1049
#11 vmem_xalloc at /usr/src/sys/kern/subr_vmem.c:1449
#12 vmem_alloc at /usr/src/sys/kern/subr_vmem.c:1350
#13 kva_alloc at /usr/src/sys/vm/vm_kern.c:150
#14 pmap_mapdev_internal at /usr/src/sys/amd64/amd64/pmap.c:8974
#15 pmap_mapdev_attr at /usr/src/sys/amd64/amd64/pmap.c:8990
#16 nexus_map_resource at /usr/src/sys/x86/x86/nexus.c:523
#17 nexus_activate_resource at /usr/src/sys/x86/x86/nexus.c:448
#18 nexus_alloc_resource at /usr/src/sys/x86/x86/nexus.c:412
#19 BUS_ALLOC_RESOURCE at ./bus_if.h:321
#20 bus_alloc_resource at /usr/src/sys/kern/subr_bus.c:4617
#21 xenpv_alloc_physmem at /usr/src/sys/x86/xen/xenpv.c:121
#22 xbb_alloc_communication_mem at 
/usr/src/sys/dev/xen/blkback/blkback.c:3010
#23 xbb_connect at /usr/src/sys/dev/xen/blkback/blkback.c:3336
#24 xenbusb_back_otherend_changed at 
/usr/src/sys/xen/xenbus/xenbusb_back.c:228
#25 xenwatch_thread at /usr/src/sys/dev/xen/xenstore/xenstore.c:1003
#26 in fork_exit at /usr/src/sys/kern/kern_fork.c:1069
#27 


There is some sort of mismatch in info, because panic message printed 
"panic: pmap_growkernel: no memory to grow kernel", but gdb backtrace in
#5  0xffffffff81073eed in pmap_growkernel at 
/usr/src/sys/amd64/amd64/pmap.c:208
leads to lines:
switch (pmap->pm_type) {
..
panic("pmap_valid_bit: invalid pm_type %d", pmap->pm_type)

So either trace is off the mark or message in serial logs. If this was 
only memleak related, then it should not happen when dom id 1 is 
started, i suppose.

I am still gathering more info regarding memleak case, will inform when 
available.

Thanks.


From nobody Mon Apr 11 15:37:27 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ECCBA1AB5B7C
	for ; Mon, 11 Apr 2022 15:37:52 +0000 (UTC)
	(envelope-from prvs=09308b1b0=roger.pau@citrix.com)
Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KcXzx5zGjz3pPN
	for ; Mon, 11 Apr 2022 15:37:49 +0000 (UTC)
	(envelope-from prvs=09308b1b0=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649691469;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=noV/yhcMs/+Ed0pmkXczXIHazppoE6s8rAPknsISibo=;
  b=PVSYuEbdHREK7pjQIKWm+I0dB6r2v6Nafdb3eUD1oOeKE0T+Ce2Qj4p6
   EZYqxSScTffhfQVyygz5jGeawpt0eQzIiVCyZ5IpnXM97CD+mmqJeXIQt
   JpcEBLVUtQncHnVwft0FncvmvJMJYESteH4NZmPhQxlzdrDOZmhA938Yi
   Q=;
X-SBRS: 5.1
X-MesageID: 70968029
X-Ironport-Server: esa4.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:ti6lr6yTygBtK5GnCTh6t+c0xirEfRIJ4+MujC+fZmUNrF6WrkUPz
 zZOWmvXM//cZGTxf9x1YYW380sGvpXWzIBlTldq+SAxQypGp/SeCIXCJC8cHc8zwu4v7q5Dx
 59DAjUVBJlsFhcwnj/0bv656yMUOZigHtIQMsadUsxKbVIiGX9JZS5LwbZj2NY02IHhWWthh
 PupyyHhEA79s9JLGjp8B5Kr8HuDa9yr5Vv0FnRnDRx6lAe2e0s9VfrzFonoR5fMeaFGH/bSe
 gr25OrRElU1XfsaIojNfr7TKiXmS1NJVOSEoiI+t6OK2nCuqsGuu0qS2TV1hUp/0l20c95NJ
 NpljoOxUR4qOLX1scMcUTYJCy1aI4lA0eqSSZS/mZT7I0zudnLtx7NlDV0sPJ1e8eFyaY1M3
 aVGcnZXNEnF3r/ohuLgIgVvrp1LwM3DJoQQt2sm1TjEJf0nXYrCU+PB4towMDIY2ZgfQ6aGP
 ZtxhTxHXTeaPD9wHlAuJ8gvsMWumHnUQgJqgQfAzUYwyzeKl1EguFT3C/LfZ9yQQtp9mEiSr
 WvCuWPjDXkyOsaSxDCe/je8i+nnkzn2VYgJGPu/7PECqE+eynELEwwNfVS+qPi9zEW5Xrpix
 1c8o3R06/JorQryE4e7D0bQTGO4UgA0YMVRC/Q57hm0443V4SnIHmQgUBpHQYlz3CMpfgAC2
 liMltLvIDVgtryJVH6QnoupQSOO1Ts9djFbO3JdJecRy5y6+dxo0EqTJjp2OPTt5uAZDw0c1
 NxjQMIWo7wIxfAG2Kyglbwsq2L9/8OZJuLZC+i+Y45E0u+bTNP+D2BLwQKChRqlEGp+ZgPc1
 EXoY+DEsIgz4WilzURhutklErCz/OqiOzbBm1NpFJRJ323zpy/9JN4LvWghdR0B3iM4ldjBO
 hG7hO+szMULYCvCgVFfPupd9PjGPYC/TI+4B5g4n/JFY4RrdR/vwc2dTRX44owZq2B1yftXE
 c7CKa6EVC9GYYw6nGveb7pMitcDm3FhrV4/sLimlnxLJ5LFPyXLIVrEWXPTBt0EAFSs/F2Oo
 4kCbZPTo/idOcWnChTqHUcoBQliBVAwBIzsqtwRceiGIwF8H3omBeOXyrQkE7GJVYwP/gsU1
 hlRgnNl9Wc=
IronPort-HdrOrdr: A9a23:kj9nw6NsymuVs8BcTyX155DYdb4zR+YMi2TDiHoedfUFSKOlfp
 6V8MjztSWVtN4QMEtQ/+xoHJPwPE80kqQFnbX5XI3SJjUO3VHIEGgM1/qG/9SNIVybygcZ79
 YeT0EcMqyBMbEZt7eD3ODQKb9Jq7PrgcPY59s2jU0dNj2CA5sQnjuRYTzra3GeKjM2YqbQQ/
 Gnl7R6TnebCD8qR/X+IkNAc/nIptXNmp6jSRkaByQ/4A3LqT+z8rb1HzWRwx9bClp0sPof2F
 mAtza8yrSosvm9xBOZ/2jP765OkN+k7tdYHsSDhuUcNz2poAe1Y4ZKXaGEoVkO0auSwWdvtO
 OJjwYrPsx15X+UVmapoSH10w2l6zoq42+K8y7vvVLT5ejCAB4qActIgoxUNjHD7VA7gd162K
 VXm0qEqpt+F3r77WrAzumNcysvulu/oHIkn+JWpWdYS5EiZLhYqpFa1F9JEa0HADnx5OkcYa
 ZT5fnnlbZrmG6hHjPkVjEF+q3vYp1zJGbLfqE6gL3V79AM90oJinfxx6Qk7wA9HdwGOt15Dt
 //Q9ZVfYF1P7wrhJ1GdZI8qOuMeyXwqEH3QSqvyWqOLtByB5uKke+x3IkI
X-IronPort-AV: E=Sophos;i="5.90,252,1643691600"; 
   d="scan'208";a="70968029"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=Lghbei/Y4nS2i1DM+4MJ3XGh8W2UnGF/Zly0pJSUatSwYD1tXmxm0s412l9Bb3LOKEWZaM2tRYG/G3mxcocUOg1ux+7qKa02gT6ouaV/sH4likHDgsYOKjVgc4REJKniRm+VKb242YrvY8CduMhzHjcnFQ8plDLt0wW+0J8QBk5fYNz+DP0kkqZNN+KGlpJMlm9o4aTZbZmp+oNN4u6sqe6USdYTHV3HLOe4xMuiCBTOoFfcxfMnjt+99PVSQzChqF+17EcfmATHJYgaPMEvw2F5p5hGPVkyBg1X/gcZ3oBodNwqwsAljUbZX8QI2gCRtKW2hMFjsoZRRVc5Mo07Mg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=9a59LEUBHWb9tzmlre4kWA+us+lrI3Y5j2LKd+yy8+0=;
 b=kQoFuFb2HceAk/ryAUS/dHAfrBBrwcr9rBUBrEhIn8yt1AguC0beqsUcjco3D5yuVpdlJnR5nW/XOMiQVDemidkL3/EMh15Vjz8z2Bo6byvbdmpzTL7o8eRjQ9PrGzofr+5d1+VkKVyksuQs2qas8WHWGnBgpNMaJ1oaZJMhUdZ9oEOygVTIDCwpNwnLGkah0MP3ugazLq/NV9ipVcFolUH4chUBTQjwGx0QKp2nTJKWdVKcC+jVGW/2rmgITtpRbtVHwmgAkrsXsAH2cASqv4sBZZimpkLCbYROeRowe5Rly0nL0n7Ps0OJaXeFhnbBzDls+uyOEhRQKnWHc6lY7Q==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=9a59LEUBHWb9tzmlre4kWA+us+lrI3Y5j2LKd+yy8+0=;
 b=ujRXXXZdeAEzghPigG/pw8trtvkgkx1U2Fbpo+r9t60y3Xsnw8a2YETA1BduTATxc2HCHrNgblpti8ptGf0jmQe8E74NV8sggEU8bV92tC/UKoTegVTkPvrApJ5PsFbGuAVyuudjuC10VpT8ckZL1cb4HLgPTAHSA4Scc/WeDlI=
Date: Mon, 11 Apr 2022 17:37:27 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
 <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
X-ClientProxiedBy: LO4P123CA0362.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:18e::7) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: ca5bafaa-a488-4843-b3a4-08da1bd131e8
X-MS-TrafficTypeDiagnostic: MWHPR03MB2496:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 0nmnB17pqYGbuPqs+p7C/fEnuopxI1eV98cedItFV93TRh5hrTQL5nEur4DBX41OevRN17JQxNqz3AtvD/seF5bqOBe7T9Z6e+BZ9bjvzqCf+zwlKodxZ6LKSB8tRYn85ltVSz/P6Ql0mbWmNqv1c64AUl3HoJ0e9CuCf9de7qTUEZIai+JljOOJkYvH15R9yHJD7vScqegVTxaM0IyGcWiBx62nYeKpon6Mc9cpBV1vxo2VfFbNhHKZ7DPVD3dZEY3vqHYKCBnL1T7FTTwZQJ1HEPBwXeKCy3qmiAlePN5TqjYyM5XmkRqHMFaWQNTkIz4UxApVyFvxuNiGSq232YiLuMBCAVyylT07w9mUcZ7gSFohyDNWMtK5O4L1gIyzrEWUs5wwwj963SLajI5YDw+K5gci9YRoCr010hVRLV5bM+Qcdu2CxfcoMP9ddmLvY5WnMapGnfPu27n7h5cWsubECO58R1HUud6tdU/xLfzq5KwHqoAvEQAmVfJEf/Ha+J3JzJl0bpKVo1zSx7QKiiWpNwO5o1QwBf+gLTiGXE2wL7o9gDX9pVPV9tmOgkP9pjibrL5aHxh8LD+buMsCEM4O0kco4sA/vw859zsAzUADTCBRWJyy8qMNtoJZRgCcGyOIW4mc+sHcSih3hxSGUA7nXysIzrnl44wXMlE/Y8wnCLn62mhhEBDll6fS18OK253gVLrS2hNIfQPnR0MxikqRXWh3socLZX3xRHsPi6s=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(186003)(83380400001)(86362001)(26005)(2906002)(8936002)(85182001)(6512007)(9686003)(6666004)(6506007)(316002)(33716001)(6486002)(966005)(5660300002)(4326008)(66946007)(38100700002)(66476007)(66556008)(82960400001)(6916009)(508600001)(8676002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T1BQT3R2U0xMSGo1M0lKWGFEd29WaTc4NnoybENCbWJSUmJvYjZtYXE1aVhP?=
 =?utf-8?B?Y25MSzd6c0RWaFNJVWNjajUyUDBGWWdnTVZ5eVVmUEhEZ0xyYUN0dHJkK1No?=
 =?utf-8?B?MkdTYVF3aFNmUEo5d2tkUW1JTlNCY21PUHJVVlNYalpBRnhYendNNy8zRVFE?=
 =?utf-8?B?ZklnUk1uVXhGQ0JibEg4cWY3RTlWUys0dWxVZStWdXBKSHg3bHhpZkVWTUZz?=
 =?utf-8?B?QWRGR1pZTnVkWlJCd3RvS1BoU3g4STNnbjVZa3JwYjYvRElSaDM0bS9UaWps?=
 =?utf-8?B?WWRKaE52Q0lVOVV2a01RdlFtSC9CNklnWmtsWm1UWFgwdjNYak85anY4cnBT?=
 =?utf-8?B?a2xUWjk0MTRIYVNUY0FuQmRNMFNWVWtWeHAvVnh2K3BOSmlyT2tTaVcrUWdI?=
 =?utf-8?B?VmZsZVhzaVJJNlppTnBpdHRYOGJ6UzRqRmN4NVZaUzFCQVRUVThnbUU2bWd0?=
 =?utf-8?B?TFpYcm0vL0s5WjVGOXFuazBLY2NqOVlMWHZQSnNPQW5sc1V2b3VIb0JhZnJL?=
 =?utf-8?B?U2lnY1ROc2JCRDhwc2pnc1VtNTVJUnRYMHAxQ09rRkQ4cDFueXE3M21QRGR4?=
 =?utf-8?B?bXN2ellYUEZWU3I1bjB1NTg2NVRSd1RETm5Tb1hoc0R2cXM1UlhER3Z5bXhI?=
 =?utf-8?B?Ukd5Z0NhekRNSG45RHZ1eHNBbzJMKzQzanFMNko2eWVOMXBhekdNbUIwZFhw?=
 =?utf-8?B?Sit1TXVmVzhEUkVJWm5YV2hZdUFhZmlrL09UZTVKS0RpTnNtcTF2TVhidVEr?=
 =?utf-8?B?eEJOR3FsdFRqR0o3a0V0NEJCeExnNXlJRXhHOTVlTElXTDZrYkF5SktzWUFu?=
 =?utf-8?B?UUR4a0t6bGkxQjBOeXlXdDdMQVZ4Umh5U2xoVU4zYVltUVE1bkVxOVF4S3pJ?=
 =?utf-8?B?Smp1Z2tKaFRic3VsaHdRUjlHWmErK3BSUlo2RldxTGhXUlVGSlFlSDBYd1h0?=
 =?utf-8?B?S09lcFBXUmRGZUkxSjc2Q28xNW5SR1NLQ1ROTXFnVDBHN1hTdUxlUUxncHJT?=
 =?utf-8?B?cUtrdHNnWWV5dEo1YXplUDdhNDVLdW9SYWc1WU16SDhWOWFRTWJjOGRuY0Rx?=
 =?utf-8?B?L3hjbTArais0SktkbE9CYmwvYTg0eHpMNmNYTmZLSEo2V2N3ZU9xU1E3bWxv?=
 =?utf-8?B?QkUvUVRvQmExMGIrcjRSUDc0a0VFVHpHL2hLcjJKdCt6RDd5a0hScDhQdGl6?=
 =?utf-8?B?UVJjVUdqMjZwcWRWODRjTEFvUUJpbFdJZEpGN1hWWHNFUDM5dFNjc2sxcVQv?=
 =?utf-8?B?VEFyK1ZGT3orTEovd2lyQytURkI1dUNraElUN0J5Y3RUazdPd3RSOTJEY0hN?=
 =?utf-8?B?cTRKaHBxMGhyY3YwdFdaazZJYTJ1QVBlVE1aWUhlNGVDemZQMXN6d3ZqbjVZ?=
 =?utf-8?B?ZnNuVkpuTjJpTXB5eWFUMEZLUjY5Nm1tR2NYcHRqVUEyVGVXVGRhZGlMamUv?=
 =?utf-8?B?SFV4Szh5azlLQVEzdllxRm52NGxhdWpVUDhKSjA5UFpWNnZ2UTdwY2V6WXFG?=
 =?utf-8?B?N2E2VllUZHIvYnFsMHdRVXZZQnBIcTVEbkFKRzVYWDdCQzgwcDY1VTdvbm9D?=
 =?utf-8?B?ZjMwUnVWdmwzUG1HenpIY2NkcEx5aEVYemE2T1JJbnJQekJVYUdiTUxTUkJq?=
 =?utf-8?B?WlNzdktOQXRYRHRidW9ZRkxodWNyTk9OY2tzTWNHZXd6Z3I2a0pITEVsRTdq?=
 =?utf-8?B?RFhBa3pJTkZLVFIrUVc5STZnNCtub0c0YXF3NlhMMmMvK2xmbjVlUCsyNnlv?=
 =?utf-8?B?NjNjT2c3Zmo5WXJnRnJZakljcC9JZEJ0QW5QTG9kblNWR25LMXZ5SmczclRL?=
 =?utf-8?B?LysyRWpkdE1LS3dPaTV2cU9BV05obWNBR1EvWXRNSFlENk9VUUtQNFFOQnZn?=
 =?utf-8?B?ckdwOTdKWS9MVS8zd3VIZE5OM29hMEkvb3owWDhCOUxtelRzTkJvQmxROXdT?=
 =?utf-8?B?dk1mSllJYjFxRGwwZjFaMC9JeVhKcUhUNUJvRzBYb1ZablQrTVpwOEp4UTlL?=
 =?utf-8?B?LzJXM2FDWmwwSURsYTJwOUNKWVo4VXdkNnR1NXM1YlBidUtSaENhbTgyL1pr?=
 =?utf-8?B?dTRVYW9VRWl0djVpTS9DbEUyN0k0b1FxQ3d0Vzg0SGw5M1FDelpRaW92bTVN?=
 =?utf-8?B?TDNwSDY2ckFoak42YlU3V0pRSkRUb0VJeDlaYkJjMzZSZm01b1RvSHU4ZGhm?=
 =?utf-8?B?NEZLQUdPVlo1elVtNklkSGQ0aW1pdXRBM3BvcTZySUFqQmx3TzZ3eTNSZXl4?=
 =?utf-8?B?QU5VZWh5d0ZWRm5VSXZRbFlROURTenIyNFhVcSsrL3V0TjVjYjNvUTBmV0Fm?=
 =?utf-8?B?enhOcjVlbVVGTWoxdTZhUHF2SWlhWVdtS2JBRDRNMjVRYW5xWmFwZ3VNVno2?=
 =?utf-8?Q?TadF76U572tXyBcc=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ca5bafaa-a488-4843-b3a4-08da1bd131e8
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2022 15:37:32.7357
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ft0FSYswT4UKN54Rn+s1xDP7ZktuENSI25CL0zwy9+p4+9wUYe4mL5KmHSI82GKRYYk+fyx2v1sGARP0m9HZIQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB2496
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KcXzx5zGjz3pPN
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=PVSYuEbd;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=ujRXXXZd;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=09308b1b0=roger.pau@citrix.com" designates 216.71.155.144 as permitted sender) smtp.mailfrom="prvs=09308b1b0=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.59 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-0.94)[-0.941];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=09308b1b0=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Apr 11, 2022 at 11:47:50AM +0300, Ze Dupsys wrote:
> On 2022.04.08. 18:02, Roger Pau Monné wrote:
> > On Fri, Apr 08, 2022 at 10:45:12AM +0300, Ze Dupsys wrote:
> > > On 2022.04.05. 18:22, Roger Pau Monné wrote:
> > > > .. Thanks, sorry for the late reply, somehow the message slip.
> > > > 
> > > > I've been able to get the file:line for those, and the trace is kind
> > > > of weird, I'm not sure I know what's going on TBH. It seems to me the
> > > > backend instance got freed while being in the process of connecting.
> > > > 
> > > > I've made some changes, that might mitigate this, but having not a
> > > > clear understanding of what's going on makes this harder.
> > > > 
> > > > I've pushed the changes to:
> > > > 
> > > > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
> > > > 
> > > > (This is on top of main branch).
> > > > 
> > > > I'm also attaching the two patches on this email.
> > > > 
> > > > Let me know if those make a difference to stabilize the system.
> > > 
> > > Hi,
> > > 
> > > Yes, it stabilizes the system, but there is still a memleak somewhere, i
> > > think.
> > > 
> > > System could run tests for approximately 41 hour, did not panic, but started
> > > to OOM kill everything.
> > > 
> > > I did not know how to git clone given commit, thus i just applied patches to
> > > 13.0-RELEASE sources.
> > > 
> > > Serial logs have nothing unusual, just that at some point OOM kill starts.
> > 
> > Well, I think that's good^W better than before. Thanks again for all
> > the testing.
> > 
> > It might be helpful now to start dumping `vmstat -m` periodically
> > while running the stress tests. As there are (hopefully) no more
> > panics now vmstat might report us what subsystem is hogging the
> > memory. It's possible it's blkback (again).
> > 
> > Thanks, Roger.
> > 
> 
> Yes, it certainly is better. Applied patch on my pre-production server, have
> not had any panic since then, still testing though.
> 
> On my stressed lab server, it's a bit different story. On occasion i see a
> panic with this trace on serial (can not reliably repeat, but sometimes upon
> starting dom id 1 and 2, sometimes mid-stress-test, dom id > 95).
> panic: pmap_growkernel: no memory to grow kernel
> cpuid = 2
> time = 1649485133
> KDB: stack backtrace:
> #0 0xffffffff80c57385 at kdb_backtrace+0x65
> #1 0xffffffff80c09d61 at vpanic+0x181
> #2 0xffffffff80c09bd3 at panic+0x43
> #3 0xffffffff81073eed at pmap_growkernel+0x27d
> #4 0xffffffff80f2d918 at vm_map_insert+0x248
> #5 0xffffffff80f30079 at vm_map_find+0x549
> #6 0xffffffff80f2bda6 at kmem_init+0x226
> #7 0xffffffff80c731a1 at vmem_xalloc+0xcb1
> #8 0xffffffff80c72a9b at vmem_xalloc+0x5ab
> #9 0xffffffff80c724a6 at vmem_alloc+0x46
> #10 0xffffffff80f2ac6b at kva_alloc+0x2b
> #11 0xffffffff8107f0eb at pmap_mapdev_attr+0x27b
> #12 0xffffffff810588ca at nexus_add_irq+0x65a
> #13 0xffffffff81058710 at nexus_add_irq+0x4a0
> #14 0xffffffff810585b9 at nexus_add_irq+0x349
> #15 0xffffffff80c495c1 at bus_alloc_resource+0xa1
> #16 0xffffffff8105e940 at xenmem_free+0x1a0
> #17 0xffffffff80a7e0dd at xbd_instance_create+0x943d
> 
> | sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e
> /usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:843
> /usr/src/sys/amd64/amd64/pmap.c:0
> /usr/src/sys/vm/vm_map.c:0
> /usr/src/sys/vm/vm_map.c:0
> /usr/src/sys/vm/vm_kern.c:712
> /usr/src/sys/kern/subr_vmem.c:928
> /usr/src/sys/kern/subr_vmem.c:0
> /usr/src/sys/kern/subr_vmem.c:1350
> /usr/src/sys/vm/vm_kern.c:150
> /usr/src/sys/amd64/amd64/pmap.c:0
> /usr/src/sys/x86/x86/nexus.c:0
> /usr/src/sys/x86/x86/nexus.c:449
> /usr/src/sys/x86/x86/nexus.c:412
> /usr/src/sys/kern/subr_bus.c:4620
> /usr/src/sys/x86/xen/xenpv.c:123
> /usr/src/sys/dev/xen/blkback/blkback.c:3010
> 
> With gdb backtrace i think i can get a better trace though:
> #0  __curthread at /usr/src/sys/amd64/include/pcpu_aux.h:55
> #1  doadump at /usr/src/sys/kern/kern_shutdown.c:399
> #2  kern_reboot at /usr/src/sys/kern/kern_shutdown.c:486
> #3  vpanic at /usr/src/sys/kern/kern_shutdown.c:919
> #4  panic at /usr/src/sys/kern/kern_shutdown.c:843
> #5  pmap_growkernel at /usr/src/sys/amd64/amd64/pmap.c:208
> #6  vm_map_insert at /usr/src/sys/vm/vm_map.c:1752
> #7  vm_map_find at /usr/src/sys/vm/vm_map.c:2259
> #8  kva_import at /usr/src/sys/vm/vm_kern.c:712
> #9  vmem_import at /usr/src/sys/kern/subr_vmem.c:928
> #10 vmem_try_fetch at /usr/src/sys/kern/subr_vmem.c:1049
> #11 vmem_xalloc at /usr/src/sys/kern/subr_vmem.c:1449
> #12 vmem_alloc at /usr/src/sys/kern/subr_vmem.c:1350
> #13 kva_alloc at /usr/src/sys/vm/vm_kern.c:150
> #14 pmap_mapdev_internal at /usr/src/sys/amd64/amd64/pmap.c:8974
> #15 pmap_mapdev_attr at /usr/src/sys/amd64/amd64/pmap.c:8990
> #16 nexus_map_resource at /usr/src/sys/x86/x86/nexus.c:523
> #17 nexus_activate_resource at /usr/src/sys/x86/x86/nexus.c:448
> #18 nexus_alloc_resource at /usr/src/sys/x86/x86/nexus.c:412
> #19 BUS_ALLOC_RESOURCE at ./bus_if.h:321
> #20 bus_alloc_resource at /usr/src/sys/kern/subr_bus.c:4617
> #21 xenpv_alloc_physmem at /usr/src/sys/x86/xen/xenpv.c:121
> #22 xbb_alloc_communication_mem at
> /usr/src/sys/dev/xen/blkback/blkback.c:3010
> #23 xbb_connect at /usr/src/sys/dev/xen/blkback/blkback.c:3336
> #24 xenbusb_back_otherend_changed at
> /usr/src/sys/xen/xenbus/xenbusb_back.c:228
> #25 xenwatch_thread at /usr/src/sys/dev/xen/xenstore/xenstore.c:1003
> #26 in fork_exit at /usr/src/sys/kern/kern_fork.c:1069
> #27 
> 
> 
> There is some sort of mismatch in info, because panic message printed
> "panic: pmap_growkernel: no memory to grow kernel", but gdb backtrace in
> #5  0xffffffff81073eed in pmap_growkernel at
> /usr/src/sys/amd64/amd64/pmap.c:208
> leads to lines:
> switch (pmap->pm_type) {
> ..
> panic("pmap_valid_bit: invalid pm_type %d", pmap->pm_type)
> 
> So either trace is off the mark or message in serial logs. If this was only
> memleak related, then it should not happen when dom id 1 is started, i
> suppose.

That's weird, I would rather trust the printed panic message rather
than the symbol resolution.  Seems to be a kind of memory exhaustion,
as the kernel is failing to allocate a page for use in the kernel page
table.

I will try to see what can be done here.

Thanks, Roger.

From nobody Tue Apr 12 15:37:48 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0BE8A1A808CF
	for ; Tue, 12 Apr 2022 15:38:07 +0000 (UTC)
	(envelope-from prvs=094341a74=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Kd8xp0QRtz4Sgt
	for ; Tue, 12 Apr 2022 15:38:05 +0000 (UTC)
	(envelope-from prvs=094341a74=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649777885;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=MML1NGI0Crpuntyv5yDA1sL7h6pmzuwsnI+8Lcy0eqU=;
  b=L6hvaDnC45P+0KpWAdkmixeuK1QA7gWxo421h1pzbAqivN7EZPwdsrb6
   KG01MMFSUnrgRi5KDnFZ51BWu7A4ZzHBGLKDD2vj4IXFwOjwbjVF4tT/d
   8R1jlEtqn3pioufrt1njt4gzL36rvINKBmZrsMQBGtpzafole2h0vlEhL
   E=;
X-SBRS: 5.1
X-MesageID: 68709445
X-Ironport-Server: esa2.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:hV1qdqz4sncAInlY6B96t+ebxirEfRIJ4+MujC+fZmQN5Y4CYwd3z
 mIfWHTOb6GXMD6uKYglM5D0oBdZ/8PInIkmFlE5sHtwQXxHs8vDCpGcKUv9eTubMsbDJK4Mx
 8tBZtWedZ04RC/X+Bnxb+K7piMm2/rYG7OkA7+YZ3krFFRvRHd/0U9ty7Bpi4U2i4a0CQiDs
 o+q/JWPYg//hW4c3g74k06mgEoHUKPa4WtJ4QZWiYl3gWLje1kp4LM3L/jsIXagTtBYE7exS
 7/Pwe7koW3Qox0nW9n/mbrwKBFbHrKKMQaw0XcHAKLKbjquB8ARPgTXENJGNC+7Xh3Qx4gZJ
 O1l7MD2EUFzVkH1sLx1vyNwSkmSBoUbvu6fSZSDmZbLlReeLyK2m68G4HweZuX0xM4mWQmiy
 tRAQNw9RkjrazWeme/TpkFE36zPHeGzVG8tkigIIQLxVJ7KdavrUaTSjeK06R9r7ix48VQyU
 OJCAdZnREyojxSioT77Arpm9AujriGXnzG1NDt5DEf4ioTe5FUZ7VTjDDbaUoOSYeBFuHTEm
 k/l0XjDRUwYGdeP7zXQpxpAhseX9c/6cIcbFbn+/f92mlyDgGcUDXX6V3Pi/6P/0BTnHYsCd
 QpEoULCroBrnKCvZsP6UBCi5maNozYXWsZKEv184waIokbRy1jJWTBVEmIcADAgnOYMQxAU/
 HzRpNHOQgEoi6OXUGmM/ajB+FtePgBKdDRfNEfoVzAt6cTjuoYsphTBRN1qFOiylNKdMT/qz
 j2AtyR7l7gVpcAR2qix5lyBhCijzrDbQwIo/h7GRUqq6wp4YMiuYInA1LTAxa8edsDDFADH5
 SVa3ZjFhAwTMX2TvHOVZMAdOoP129umHiHmxgNtMpIx0Tv4rhZPYrtsyD15IU5oNOMNdjnof
 FLftGtt2XNDAJe5RfQpOtzsUqzG2YClTI24Da6MMrKid7ArLGe6EDdSiVl8NowHuGwliukBN
 JiSaq5A5l5KWP08nFJaqwrwuILHJxzSJ0uOHfgXLDz9iNJygUJ5r59fbjNiichjscu5TP39q
 Yo3Cidz40w3vBfCSifW65UPClsBMGI2A5v7w+QOKLLSfVI3SDF4VqWMqV/ER2CDt/4K/gsv1
 ivjMnK0NXKl3SGXQel0Qi0LhEzTsWZX8itgYH1E0aeA0Hk/e4e/hJrzhLNsFYTLANdLlKYuJ
 9FcIp3oKq0WFlzvpmRMBbGg/dcKXEn62mqz09+NPWFXk2hIHFeSpLcJv2LHqUEzM8ZAnZdm+
 O3/i1mDGPLuhW1KVa7rVR5m9Hvo1VA1k+NuRUrYZN5VfUTn6o9xLCLtyPQwJqkxxd/rnVN2C
 y7+7c8kmNTw
IronPort-HdrOrdr: A9a23:83FxmKmG/Aj2UwauAZJbgkP+finpDfPIimdD5ihNYBxZY6Wkfp
 +V8sjzhCWatN9OYh0dcLC7WJVpQRvnhPhICK0qTMqftW7dyReVxeBZnPHfKljbehEWmdQtsJ
 uIH5IObOEYSGIK8voSgzPIY+rIouP3iJxA7N22pxwGIHAIGsNdBkVCe32m+yVNNXh77PECZe
 OhD6R81l2dkSN9VLXEOpBJZZmJm/T70LbdJTIWDR8u7weDyRuu9b7BChCdmjMTSSlGz7sO+X
 XM11WR3NTvj9iLjjvnk0PD5ZVfn9XsjvNFGcy3k8AQbhHhkByhaohNU6CL+Bo1vOaswlA3l8
 SkmWZrA+1Dr1fqOk2lqxrk3AftlB4o9n/Z0FedxUDupMToLQhKffZptMZ8SF/0+kAgtNZz3O
 ZgxGSCradaChvGgWDU+8XIfwsCrDv7nVMS1cooy1BPW4oXb7Fc6aYF+llOLZsGFCXmrKg6De
 hVCt3G7vo+SyLUU5nghBgu/DWQZAVxIv/fKXJy+PB9kgIm0EyR9nFohfD2xRw7hdcAo5ot3Z
 WyDk0nrsALciYsV9MOOA4we7rFNoXze2O4DIuzGyWvKEhVAQOEl3bIiI9FkN1CPqZ4i6cPpA
 ==
X-IronPort-AV: E=Sophos;i="5.90,254,1643691600"; 
   d="scan'208,223";a="68709445"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=YiQ9pdh7xPJYGolLm4P24j3AuuoCEH85jUNKSwh1KFjecodSiFX6QDGrAZc15sqUZ6XLDyW6P/KoW5vZyxkJk/FErfEzSdVoOisFbQcEvKH3NXtpSidWjHLkVWwi1L9bBLUwUU9E87/6C5grZTISKUnSc8WOdLbGcx1GzSeVGCNXubb58Og/Md9NoKrZuxCSVa+TQRjWMpoBxzDN/6jiuiv4wCDo6btoacJkekZrRyKZXM+5y7R/rrIVEUrD0jgg81Hgz1zJ9wKK8UZaQHrw4sRkFWqjuaQqWS8h50NkiNWsk9tMCBezwOJPNG7fgXbuR933uO0BtTMuTwVBlPIGaw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=Ic04E1uVcH6JyKP1fbqPkdA6CUgjly254HlqIZPVjhM=;
 b=fd+F7J0xvL43+zQjq9M+vqX3wUCkPcK29Xoj9UBiB2NWh3amEIFXUmtEc0KkD46kp7abDDjkn2cUZnH1K0+7pl4qNFv54fmN5ZkU0xexZU75xDd8eQS1qCa8QxS42DGQqW4Ux6i77ENtRCalfYHjHIdBTLhabEmE8l9btWpzO0PkDgiQYElQFe2d81exYWDe5sRfNA5O41OQ5odtfkxn/v/ImMPOFN8jaBidG0qCaaejlxjyZpj+6RsMxsIlj+gWQ/wn6bSoLG8P5CIu8PRS1lOhBtBwXinTDorUtTa/AQhLIdUWW8AMmUm5dk24HyMlYy3Wt2HN1lo8QjKCHs+5EQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=Ic04E1uVcH6JyKP1fbqPkdA6CUgjly254HlqIZPVjhM=;
 b=WjG6q4TWLTxVPr6ZpTHkWFkb/CfXhgDvgjREMDZDDT+nhfJPvCt5Ama6+AThqEcdvltbaXELTKzEzGX5Dd1ZwJmuZkcQc3wr7REItwGiG9yHise2hfB+pJ1/O3HgPKsw+vJVuGxMPMPP9hS63/5/x+nS9DFSL/F1Nl9vBhczGEU=
Date: Tue, 12 Apr 2022 17:37:48 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
 <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
 
Content-Type: multipart/mixed; boundary="AXxWJ3w5Q3v8ydwo"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LNXP265CA0046.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:5c::34) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: f96e271b-1167-4df5-c791-08da1c9a6915
X-MS-TrafficTypeDiagnostic: DM6PR03MB5161:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 7DVOQXWjl0A+LHKnweRr8FUqkNWwlsc11jIOoJpH/sWO+6+WKuQvY03K22nWmIH19Uks/df1KKNdV8nLme0mv5KoqOKtUTWbmFiPE+2z1TPRduwioS1nfHrGi3CVAcjYMmZVC4O8QWn+3EUalnS0K/pMRmwitKew6SdGb7JEZ5RPwxyQrnOsmyW8vw9Emgh+oSc+oiOnlh+2Vud3YUJ9MmxP7RZLmQ9bMzs+mXcRieH0F58zj91+/Bq9G5LxEdn3YXWd9gzZnHPit/67bedo1Z8PgjOUYP+DbhVVIEXUGl6Y6ho3mi/J/1sa+8cbA1lVi7C6iHIySV+jEj109uj1p/24R3V77Ks7tSZuvBcefs8idH4LT1GM05Unankt/7/QJGyqmytZ+3ANLCPt4CuzuW861a3RiZXdXV7lnb6pfj6LWJ8ztmNKBzWODx/D65Cpms7mcuO3EoMvPZKUNaVV6e6/Q9MljrYtWQBpJmcmeocLeSf7jYF/67HWij7y1mJhwkYcI0HmI82DLgvjOpZmRXy43yonSJ33rQsmlgXesmVqC9cF5fg3ketEvtN2AXq6idjB8X7nAML7uWRqqm/Rdyfv5xK/j1wNWeaCdjAoZq6tzmDKJJqA5g8rLXcpzQxDQ/9FST3e653wLCdhynJm92TQynGJmKLxomfjO9XPOU1whGNE1WfIhzo3Im6heTsaXcEzPRC+pSEveGNgTbUz4Mz4Q5alzFOfak1DtCIS/l//dXd4mT+xrAZ2Dim2Mi5YwPU22sYtAT+OGyJl+steLTvsWEV9y+vYpJd7R3DZ7XM=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(66556008)(38100700002)(4326008)(33964004)(44144004)(6506007)(66476007)(66946007)(8936002)(8676002)(6486002)(83380400001)(6916009)(33716001)(508600001)(82960400001)(85182001)(86362001)(966005)(26005)(186003)(6666004)(2906002)(316002)(235185007)(6512007)(5660300002)(9686003)(2700100001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q0xxdzlYcHpSdkw0MXFRMWtibERKWm4yVmZBUTBkdm9RaEdmRVZoV3JRMkFC?=
 =?utf-8?B?NWRSWDJxTGRnRERid2kreGhtUFg0MWlkVFhPVzhHb0dET3JYSm4rV2ZKelcz?=
 =?utf-8?B?anZ3M2lnYXJORlJBSUNMYnlPY0tESElLZytRR1J0UUtYbkMva3RSWnF4cXlv?=
 =?utf-8?B?RWpHOVdvQlVQQS9QQzhKVkNTcDRjWjY5MTFCS3I1ejJCeEd4cVhscVlVKzRl?=
 =?utf-8?B?SHlZb3VoWmJhYlVOUmJWY1BnY2dhWjFRaHQ2VHBYdVF5Q2RMMS9VNGFPZmdE?=
 =?utf-8?B?VDBIM1JkQ3B6ajlaM25mNFVFQTFQTzRmZit0TVg0VzJVbW1sR0wwNkFIdFlW?=
 =?utf-8?B?T05NK2V3dzRjTG5qRG9WRVB2NUFXODd3UnlPVUY3dDVkZ3dmbk1FUVljeXVE?=
 =?utf-8?B?T3FBbzFCeGo5YUhDQi84WEFlQitkcmVEa0FGMytlOEh4U21pZTlra3B0SzdC?=
 =?utf-8?B?ZVA0YUVKd0RvdUU2Um83emJhK0JoUzVIc2RFbExCQ09jS2NJaW92RnlUR21h?=
 =?utf-8?B?TktVWG1rRUQ1Y2wra25BMHA2NzJRWVMvVXpGYzQwSFR2ckplTUNzL2dCRlJr?=
 =?utf-8?B?MEUweXhQTTZoNlNFL1pGaWVIYllwMC9Ib2RrblBtanFCaG1kYVJkVURTY0Z0?=
 =?utf-8?B?ditkSlVOOHA2RTl2QklLalluR0RRaFhhSmhBeUxtdUtTNHFSRWc2SW1HMTZn?=
 =?utf-8?B?NU9Kb25kS0laaS9aY0dNQkQ3V1VvZEVKU2U5cklRUEdzRVRlN0kxNjNGdkRu?=
 =?utf-8?B?RGVicUprSU8raDFvWXpGM0xEbVVQekdFdlBZbnFmZCtxSm8zcVFLNG8zTnE0?=
 =?utf-8?B?NHBtNjZnak5KRXRiQWxFbWQ2MEw0a0JEa3ExbVN0cXNxbmRiSG1lQ2R3Mi9O?=
 =?utf-8?B?MCs0U2QrMFhFZUZEa1pTVDlXZFdXQjNkWE5MSFVsSGZ0MmMwU045dSs0eFAy?=
 =?utf-8?B?RHZwOWFNaHhTS3ozTmg2NGNpeTY5dEJLckZEOFZGa1dHTWRGSEF1dTZ4T2Nj?=
 =?utf-8?B?ZmVXcGZxV0FDTjVYRGpvN3o1RlYrekJHU09remVUZVJFaHNZY3FUTjQxUnI5?=
 =?utf-8?B?ZVh3RkFJL2N0N0d0c2ZNK3hOZnVJQVVENlROV2FNZWp0dGRFQkV0L0pHanZq?=
 =?utf-8?B?YW80MkhxQjUrMU9WS002cGF0WGptREp0Wlg0Sm04MTZjMVZmTHdtY1k3eFdH?=
 =?utf-8?B?dDN5VDZyeXBmVW1IdkhYaWF6bmV2OVY0WTBZVEtrRHRINXl5aXp0RnZHaHh2?=
 =?utf-8?B?aExqZXJaZnkrKy8wQjVxMmdneENCU2YrVVlmbnBUM3huQ3RkQ1J1d3hsaGh5?=
 =?utf-8?B?SDRYVmVuVXN6WUhmd1FCallSUlNvZWRDWGZza1pPVHdRWUFPcFVlc283am15?=
 =?utf-8?B?bGxyR0kyek5TQUZUZjhFZXVVcERwMkNjWkRUWnJoSDBuOXFkVElJWWJWSGhq?=
 =?utf-8?B?cHRVNHZJK2svUnZWSUVaWC93YUtOZE93WWJKVlZZY1hMVExnbzNIQnhQREN4?=
 =?utf-8?B?aFE0V1ZlVTd5Tzd2ZldKeU16S1ljcnJWTHZGeFlLeG1kVWtyWUJRcHJDcjJR?=
 =?utf-8?B?djdXTlM3V2l4UFF3N2trUzZnWVZZSWcwN2c5RkxBNXNwbERtRnB0ZnRRRStD?=
 =?utf-8?B?N0Z2QVdkaStrN3NMUmRpZTNGRk5EaUxxRXA2bU96bFZaWGhMbThPbUQ3OThQ?=
 =?utf-8?B?NmhIK1ZaYlVtOFN0SjI0UUxuSjlMekxPN2lWTEpUNEhZR0F2RFBTYWNyNldP?=
 =?utf-8?B?ZEVBbGY4RENtNkxLT2JpaUNhWmhEZ0VNamY1aDB5elNwajdqQllQR3pXWkFh?=
 =?utf-8?B?d2ZZeHo1aDNPUFVObHhlZFE4RENQcUk1bStSM081Q0N6VFV2MnpLak0xcUJP?=
 =?utf-8?B?ek5EcExYTEhTRUlTWXRFMXpOcloyakVIWGFUU0xrZnBtV21sMHBGbEhpZ251?=
 =?utf-8?B?TDJDRlRVcVVsS2NkRWRJekZDZEVGZXZoR1FScnZSQkptREQzZnllSnFPcTZI?=
 =?utf-8?B?YW54QVZTMVBTdzN2ckxxMC9XNjl4OElDVzNyU0wyaTcyTDgyUTZDYktxSmw4?=
 =?utf-8?B?K3NJQlRpU3plb3ovMnpEbHZOVWk0UXBjbG5WNEh4bVFOcHFocjZrSzRlb1BB?=
 =?utf-8?B?M243VHBKbWorcXdRMHI5NmwxLzNwR1I1NGhhcWZGbTJTNjA2RXdLRGxkdElN?=
 =?utf-8?B?KzA1UE5McjZFejZiZU9USXpuVGpjZEV0Zm13UERZQXRIRStXYnJkVXdtQlht?=
 =?utf-8?B?djhGUHhzQ1k1OWZEUlgxS3BXbjY1S0pOYnpaQmdUUHdYZ3ZNbDBxd1ZObStY?=
 =?utf-8?B?NGxRR0hTZkt1ZjljWTJ0L0c2Yjdnb24xaTUyQTArZVhrRGJTV1AvY1pndVI2?=
 =?utf-8?Q?hpQ1UfyCUIAc1e7k=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f96e271b-1167-4df5-c791-08da1c9a6915
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2022 15:37:54.2747
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DnIma0OX6d3ZdLLvhOByizTehw2x7/o+T/2oBe+EexPNgP53Nn/acvukWFbYrtI6ad0ldhdFIidqxDuL5dOI8w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB5161
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4Kd8xp0QRtz4Sgt
X-Spamd-Bar: ----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=L6hvaDnC;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=WjG6q4TW;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=094341a74=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=094341a74=roger.pau@citrix.com"
X-Spamd-Result: default: False [-4.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 HAS_ATTACHMENT(0.00)[];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 CTYPE_MIXED_BOGUS(1.00)[];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+,1:+,2:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=094341a74=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[multipart/mixed,text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

--AXxWJ3w5Q3v8ydwo
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On Mon, Apr 11, 2022 at 05:37:27PM +0200, Roger Pau Monné wrote:
> On Mon, Apr 11, 2022 at 11:47:50AM +0300, Ze Dupsys wrote:
> > On 2022.04.08. 18:02, Roger Pau Monné wrote:
> > > On Fri, Apr 08, 2022 at 10:45:12AM +0300, Ze Dupsys wrote:
> > > > On 2022.04.05. 18:22, Roger Pau Monné wrote:
> > > > > .. Thanks, sorry for the late reply, somehow the message slip.
> > > > > 
> > > > > I've been able to get the file:line for those, and the trace is kind
> > > > > of weird, I'm not sure I know what's going on TBH. It seems to me the
> > > > > backend instance got freed while being in the process of connecting.
> > > > > 
> > > > > I've made some changes, that might mitigate this, but having not a
> > > > > clear understanding of what's going on makes this harder.
> > > > > 
> > > > > I've pushed the changes to:
> > > > > 
> > > > > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
> > > > > 
> > > > > (This is on top of main branch).
> > > > > 
> > > > > I'm also attaching the two patches on this email.
> > > > > 
> > > > > Let me know if those make a difference to stabilize the system.
> > > > 
> > > > Hi,
> > > > 
> > > > Yes, it stabilizes the system, but there is still a memleak somewhere, i
> > > > think.
> > > > 
> > > > System could run tests for approximately 41 hour, did not panic, but started
> > > > to OOM kill everything.
> > > > 
> > > > I did not know how to git clone given commit, thus i just applied patches to
> > > > 13.0-RELEASE sources.
> > > > 
> > > > Serial logs have nothing unusual, just that at some point OOM kill starts.
> > > 
> > > Well, I think that's good^W better than before. Thanks again for all
> > > the testing.
> > > 
> > > It might be helpful now to start dumping `vmstat -m` periodically
> > > while running the stress tests. As there are (hopefully) no more
> > > panics now vmstat might report us what subsystem is hogging the
> > > memory. It's possible it's blkback (again).
> > > 
> > > Thanks, Roger.
> > > 
> > 
> > Yes, it certainly is better. Applied patch on my pre-production server, have
> > not had any panic since then, still testing though.
> > 
> > On my stressed lab server, it's a bit different story. On occasion i see a
> > panic with this trace on serial (can not reliably repeat, but sometimes upon
> > starting dom id 1 and 2, sometimes mid-stress-test, dom id > 95).
> > panic: pmap_growkernel: no memory to grow kernel
> > cpuid = 2
> > time = 1649485133
> > KDB: stack backtrace:
> > #0 0xffffffff80c57385 at kdb_backtrace+0x65
> > #1 0xffffffff80c09d61 at vpanic+0x181
> > #2 0xffffffff80c09bd3 at panic+0x43
> > #3 0xffffffff81073eed at pmap_growkernel+0x27d
> > #4 0xffffffff80f2d918 at vm_map_insert+0x248
> > #5 0xffffffff80f30079 at vm_map_find+0x549
> > #6 0xffffffff80f2bda6 at kmem_init+0x226
> > #7 0xffffffff80c731a1 at vmem_xalloc+0xcb1
> > #8 0xffffffff80c72a9b at vmem_xalloc+0x5ab
> > #9 0xffffffff80c724a6 at vmem_alloc+0x46
> > #10 0xffffffff80f2ac6b at kva_alloc+0x2b
> > #11 0xffffffff8107f0eb at pmap_mapdev_attr+0x27b
> > #12 0xffffffff810588ca at nexus_add_irq+0x65a
> > #13 0xffffffff81058710 at nexus_add_irq+0x4a0
> > #14 0xffffffff810585b9 at nexus_add_irq+0x349
> > #15 0xffffffff80c495c1 at bus_alloc_resource+0xa1
> > #16 0xffffffff8105e940 at xenmem_free+0x1a0
> > #17 0xffffffff80a7e0dd at xbd_instance_create+0x943d
> > 
> > | sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e
> > /usr/lib/debug/boot/kernel/kernel.debug
> > /usr/src/sys/kern/subr_kdb.c:443
> > /usr/src/sys/kern/kern_shutdown.c:0
> > /usr/src/sys/kern/kern_shutdown.c:843
> > /usr/src/sys/amd64/amd64/pmap.c:0
> > /usr/src/sys/vm/vm_map.c:0
> > /usr/src/sys/vm/vm_map.c:0
> > /usr/src/sys/vm/vm_kern.c:712
> > /usr/src/sys/kern/subr_vmem.c:928
> > /usr/src/sys/kern/subr_vmem.c:0
> > /usr/src/sys/kern/subr_vmem.c:1350
> > /usr/src/sys/vm/vm_kern.c:150
> > /usr/src/sys/amd64/amd64/pmap.c:0
> > /usr/src/sys/x86/x86/nexus.c:0
> > /usr/src/sys/x86/x86/nexus.c:449
> > /usr/src/sys/x86/x86/nexus.c:412
> > /usr/src/sys/kern/subr_bus.c:4620
> > /usr/src/sys/x86/xen/xenpv.c:123
> > /usr/src/sys/dev/xen/blkback/blkback.c:3010
> > 
> > With gdb backtrace i think i can get a better trace though:
> > #0  __curthread at /usr/src/sys/amd64/include/pcpu_aux.h:55
> > #1  doadump at /usr/src/sys/kern/kern_shutdown.c:399
> > #2  kern_reboot at /usr/src/sys/kern/kern_shutdown.c:486
> > #3  vpanic at /usr/src/sys/kern/kern_shutdown.c:919
> > #4  panic at /usr/src/sys/kern/kern_shutdown.c:843
> > #5  pmap_growkernel at /usr/src/sys/amd64/amd64/pmap.c:208
> > #6  vm_map_insert at /usr/src/sys/vm/vm_map.c:1752
> > #7  vm_map_find at /usr/src/sys/vm/vm_map.c:2259
> > #8  kva_import at /usr/src/sys/vm/vm_kern.c:712
> > #9  vmem_import at /usr/src/sys/kern/subr_vmem.c:928
> > #10 vmem_try_fetch at /usr/src/sys/kern/subr_vmem.c:1049
> > #11 vmem_xalloc at /usr/src/sys/kern/subr_vmem.c:1449
> > #12 vmem_alloc at /usr/src/sys/kern/subr_vmem.c:1350
> > #13 kva_alloc at /usr/src/sys/vm/vm_kern.c:150
> > #14 pmap_mapdev_internal at /usr/src/sys/amd64/amd64/pmap.c:8974
> > #15 pmap_mapdev_attr at /usr/src/sys/amd64/amd64/pmap.c:8990
> > #16 nexus_map_resource at /usr/src/sys/x86/x86/nexus.c:523
> > #17 nexus_activate_resource at /usr/src/sys/x86/x86/nexus.c:448
> > #18 nexus_alloc_resource at /usr/src/sys/x86/x86/nexus.c:412
> > #19 BUS_ALLOC_RESOURCE at ./bus_if.h:321
> > #20 bus_alloc_resource at /usr/src/sys/kern/subr_bus.c:4617
> > #21 xenpv_alloc_physmem at /usr/src/sys/x86/xen/xenpv.c:121
> > #22 xbb_alloc_communication_mem at
> > /usr/src/sys/dev/xen/blkback/blkback.c:3010
> > #23 xbb_connect at /usr/src/sys/dev/xen/blkback/blkback.c:3336
> > #24 xenbusb_back_otherend_changed at
> > /usr/src/sys/xen/xenbus/xenbusb_back.c:228
> > #25 xenwatch_thread at /usr/src/sys/dev/xen/xenstore/xenstore.c:1003
> > #26 in fork_exit at /usr/src/sys/kern/kern_fork.c:1069
> > #27 
> > 
> > 
> > There is some sort of mismatch in info, because panic message printed
> > "panic: pmap_growkernel: no memory to grow kernel", but gdb backtrace in
> > #5  0xffffffff81073eed in pmap_growkernel at
> > /usr/src/sys/amd64/amd64/pmap.c:208
> > leads to lines:
> > switch (pmap->pm_type) {
> > ..
> > panic("pmap_valid_bit: invalid pm_type %d", pmap->pm_type)
> > 
> > So either trace is off the mark or message in serial logs. If this was only
> > memleak related, then it should not happen when dom id 1 is started, i
> > suppose.
> 
> That's weird, I would rather trust the printed panic message rather
> than the symbol resolution.  Seems to be a kind of memory exhaustion,
> as the kernel is failing to allocate a page for use in the kernel page
> table.
> 
> I will try to see what can be done here.

I have a patch to disable the bounce buffering done in blkback
(attached).

While I think it's not directly related to the panic you are hitting,
it's long time since we should have disabled that.  It should reduce
the memory consumption by blkback greatly, so might have the side
effect of helping with your issue related to pmap_growkernel.

On my test box a single instance of blkback reduced memory usage from
~100M to ~300K.

It should be applied on top of the other two patches.

Regards, Roger.

--AXxWJ3w5Q3v8ydwo
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment;
	filename="0001-xen-blkback-remove-bounce-buffering-mode.patch"

>From 449ef76695cf5ec5cc3514e6bd653d0b1dff3dde Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= 
Date: Tue, 12 Apr 2022 16:17:09 +0200
Subject: [PATCH] xen/blkback: remove bounce buffering mode

Remove bounce buffering code for blkback and only attach if Xen
creates IOMMU entries for grant mapped pages.

Such bounce buffering consumed a non trivial amount of memory and CPU
resources to do the memory copy, when it's been a long time since Xen
has been creating IOMMU entries for grant maps.

Refuse to attach blkback if Xen doesn't advertise that IOMMU entries
are created for grant maps.

Sponsored by: Citrix Systems R&D
---
 sys/dev/xen/blkback/blkback.c | 181 ++--------------------------------
 1 file changed, 8 insertions(+), 173 deletions(-)

diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 15e4bbe78fc0..97939d32ffce 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -80,6 +80,7 @@ __FBSDID("$FreeBSD$");
 #include 
 #include 
 
+#include 
 #include 
 #include 
 
@@ -101,27 +102,6 @@ __FBSDID("$FreeBSD$");
 #define	XBB_MAX_REQUESTS 					\
 	__CONST_RING_SIZE(blkif, PAGE_SIZE * XBB_MAX_RING_PAGES)
 
-/**
- * \brief Define to force all I/O to be performed on memory owned by the
- *        backend device, with a copy-in/out to the remote domain's memory.
- *
- * \note  This option is currently required when this driver's domain is
- *        operating in HVM mode on a system using an IOMMU.
- *
- * This driver uses Xen's grant table API to gain access to the memory of
- * the remote domains it serves.  When our domain is operating in PV mode,
- * the grant table mechanism directly updates our domain's page table entries
- * to point to the physical pages of the remote domain.  This scheme guarantees
- * that blkback and the backing devices it uses can safely perform DMA
- * operations to satisfy requests.  In HVM mode, Xen may use a HW IOMMU to
- * insure that our domain cannot DMA to pages owned by another domain.  As
- * of Xen 4.0, IOMMU mappings for HVM guests are not updated via the grant
- * table API.  For this reason, in HVM mode, we must bounce all requests into
- * memory that is mapped into our domain at domain startup and thus has
- * valid IOMMU mappings.
- */
-#define XBB_USE_BOUNCE_BUFFERS
-
 /**
  * \brief Define to enable rudimentary request logging to the console.
  */
@@ -257,14 +237,6 @@ struct xbb_xen_reqlist {
 	 */
 	uint64_t	 	 gnt_base;
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	/**
-	 * Pre-allocated domain local memory used to proxy remote
-	 * domain memory during I/O operations.
-	 */
-	uint8_t			*bounce;
-#endif
-
 	/**
 	 * Array of grant handles (one per page) used to map this request.
 	 */
@@ -500,30 +472,6 @@ struct xbb_file_data {
 	 * so we only need one of these.
 	 */
 	struct iovec	xiovecs[XBB_MAX_SEGMENTS_PER_REQLIST];
-#ifdef XBB_USE_BOUNCE_BUFFERS
-
-	/**
-	 * \brief Array of io vectors used to handle bouncing of file reads.
-	 *
-	 * Vnode operations are free to modify uio data during their
-	 * exectuion.  In the case of a read with bounce buffering active,
-	 * we need some of the data from the original uio in order to
-	 * bounce-out the read data.  This array serves as the temporary
-	 * storage for this saved data.
-	 */
-	struct iovec	saved_xiovecs[XBB_MAX_SEGMENTS_PER_REQLIST];
-
-	/**
-	 * \brief Array of memoized bounce buffer kva offsets used
-	 *        in the file based backend.
-	 *
-	 * Due to the way that the mapping of the memory backing an
-	 * I/O transaction is handled by Xen, a second pass through
-	 * the request sg elements is unavoidable. We memoize the computed
-	 * bounce address here to reduce the cost of the second walk.
-	 */
-	void		*xiovecs_vaddr[XBB_MAX_SEGMENTS_PER_REQLIST];
-#endif /* XBB_USE_BOUNCE_BUFFERS */
 };
 
 /**
@@ -891,25 +839,6 @@ xbb_reqlist_vaddr(struct xbb_xen_reqlist *reqlist, int pagenr, int sector)
 	return (reqlist->kva + (PAGE_SIZE * pagenr) + (sector << 9));
 }
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-/**
- * Given a page index and 512b sector offset within that page,
- * calculate an offset into a request's local bounce memory region.
- *
- * \param reqlist The request structure whose bounce region will be accessed.
- * \param pagenr  The page index used to compute the bounce offset.
- * \param sector  The 512b sector index used to compute the page relative
- *                bounce offset.
- *
- * \return  The computed global bounce buffer address.
- */
-static inline uint8_t *
-xbb_reqlist_bounce_addr(struct xbb_xen_reqlist *reqlist, int pagenr, int sector)
-{
-	return (reqlist->bounce + (PAGE_SIZE * pagenr) + (sector << 9));
-}
-#endif
-
 /**
  * Given a page number and 512b sector offset within that page,
  * calculate an offset into the request's memory region that the
@@ -929,11 +858,7 @@ xbb_reqlist_bounce_addr(struct xbb_xen_reqlist *reqlist, int pagenr, int sector)
 static inline uint8_t *
 xbb_reqlist_ioaddr(struct xbb_xen_reqlist *reqlist, int pagenr, int sector)
 {
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	return (xbb_reqlist_bounce_addr(reqlist, pagenr, sector));
-#else
 	return (xbb_reqlist_vaddr(reqlist, pagenr, sector));
-#endif
 }
 
 /**
@@ -1508,17 +1433,6 @@ xbb_bio_done(struct bio *bio)
 		}
 	}
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	if (bio->bio_cmd == BIO_READ) {
-		vm_offset_t kva_offset;
-
-		kva_offset = (vm_offset_t)bio->bio_data
-			   - (vm_offset_t)reqlist->bounce;
-		memcpy((uint8_t *)reqlist->kva + kva_offset,
-		       bio->bio_data, bio->bio_bcount);
-	}
-#endif /* XBB_USE_BOUNCE_BUFFERS */
-
 	/*
 	 * Decrement the pending count for the request list.  When we're
 	 * done with the requests, send status back for all of them.
@@ -2180,17 +2094,6 @@ xbb_dispatch_dev(struct xbb_softc *xbb, struct xbb_xen_reqlist *reqlist,
 
 	for (bio_idx = 0; bio_idx < nbio; bio_idx++)
 	{
-#ifdef XBB_USE_BOUNCE_BUFFERS
-		vm_offset_t kva_offset;
-
-		kva_offset = (vm_offset_t)bios[bio_idx]->bio_data
-			   - (vm_offset_t)reqlist->bounce;
-		if (operation == BIO_WRITE) {
-			memcpy(bios[bio_idx]->bio_data,
-			       (uint8_t *)reqlist->kva + kva_offset,
-			       bios[bio_idx]->bio_bcount);
-		}
-#endif
 		if (operation == BIO_READ) {
 			SDT_PROBE3(xbb, kernel, xbb_dispatch_dev, read,
 				   device_get_unit(xbb->dev),
@@ -2241,10 +2144,6 @@ xbb_dispatch_file(struct xbb_softc *xbb, struct xbb_xen_reqlist *reqlist,
 	struct uio            xuio;
 	struct xbb_sg        *xbb_sg;
 	struct iovec         *xiovec;
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	void                **p_vaddr;
-	int                   saved_uio_iovcnt;
-#endif /* XBB_USE_BOUNCE_BUFFERS */
 	int                   error;
 
 	file_data = &xbb->backend.file;
@@ -2300,18 +2199,6 @@ xbb_dispatch_file(struct xbb_softc *xbb, struct xbb_xen_reqlist *reqlist,
 			xiovec = &file_data->xiovecs[xuio.uio_iovcnt];
 			xiovec->iov_base = xbb_reqlist_ioaddr(reqlist,
 			    seg_idx, xbb_sg->first_sect);
-#ifdef XBB_USE_BOUNCE_BUFFERS
-			/*
-			 * Store the address of the incoming
-			 * buffer at this particular offset
-			 * as well, so we can do the copy
-			 * later without having to do more
-			 * work to recalculate this address.
-		 	 */
-			p_vaddr = &file_data->xiovecs_vaddr[xuio.uio_iovcnt];
-			*p_vaddr = xbb_reqlist_vaddr(reqlist, seg_idx,
-			    xbb_sg->first_sect);
-#endif /* XBB_USE_BOUNCE_BUFFERS */
 			xiovec->iov_len = 0;
 			xuio.uio_iovcnt++;
 		}
@@ -2331,28 +2218,6 @@ xbb_dispatch_file(struct xbb_softc *xbb, struct xbb_xen_reqlist *reqlist,
 
 	xuio.uio_td = curthread;
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	saved_uio_iovcnt = xuio.uio_iovcnt;
-
-	if (operation == BIO_WRITE) {
-		/* Copy the write data to the local buffer. */
-		for (seg_idx = 0, p_vaddr = file_data->xiovecs_vaddr,
-		     xiovec = xuio.uio_iov; seg_idx < xuio.uio_iovcnt;
-		     seg_idx++, xiovec++, p_vaddr++) {
-			memcpy(xiovec->iov_base, *p_vaddr, xiovec->iov_len);
-		}
-	} else {
-		/*
-		 * We only need to save off the iovecs in the case of a
-		 * read, because the copy for the read happens after the
-		 * VOP_READ().  (The uio will get modified in that call
-		 * sequence.)
-		 */
-		memcpy(file_data->saved_xiovecs, xuio.uio_iov,
-		       xuio.uio_iovcnt * sizeof(xuio.uio_iov[0]));
-	}
-#endif /* XBB_USE_BOUNCE_BUFFERS */
-
 	switch (operation) {
 	case BIO_READ:
 
@@ -2429,25 +2294,6 @@ xbb_dispatch_file(struct xbb_softc *xbb, struct xbb_xen_reqlist *reqlist,
 		/* NOTREACHED */
 	}
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-	/* We only need to copy here for read operations */
-	if (operation == BIO_READ) {
-		for (seg_idx = 0, p_vaddr = file_data->xiovecs_vaddr,
-		     xiovec = file_data->saved_xiovecs;
-		     seg_idx < saved_uio_iovcnt; seg_idx++,
-		     xiovec++, p_vaddr++) {
-			/*
-			 * Note that we have to use the copy of the 
-			 * io vector we made above.  uiomove() modifies
-			 * the uio and its referenced vector as uiomove
-			 * performs the copy, so we can't rely on any
-			 * state from the original uio.
-			 */
-			memcpy(*p_vaddr, xiovec->iov_base, xiovec->iov_len);
-		}
-	}
-#endif /* XBB_USE_BOUNCE_BUFFERS */
-
 bailout_send_response:
 
 	if (error != 0)
@@ -2826,12 +2672,6 @@ xbb_disconnect(struct xbb_softc *xbb)
 		/* There is one request list for ever allocated request. */
 		for (i = 0, reqlist = xbb->request_lists;
 		     i < xbb->max_requests; i++, reqlist++){
-#ifdef XBB_USE_BOUNCE_BUFFERS
-			if (reqlist->bounce != NULL) {
-				free(reqlist->bounce, M_XENBLOCKBACK);
-				reqlist->bounce = NULL;
-			}
-#endif
 			if (reqlist->gnt_handles != NULL) {
 				free(reqlist->gnt_handles, M_XENBLOCKBACK);
 				reqlist->gnt_handles = NULL;
@@ -3210,17 +3050,6 @@ xbb_alloc_request_lists(struct xbb_softc *xbb)
 
 		reqlist->xbb = xbb;
 
-#ifdef XBB_USE_BOUNCE_BUFFERS
-		reqlist->bounce = malloc(xbb->max_reqlist_size,
-					 M_XENBLOCKBACK, M_NOWAIT);
-		if (reqlist->bounce == NULL) {
-			xenbus_dev_fatal(xbb->dev, ENOMEM, 
-					 "Unable to allocate request "
-					 "bounce buffers");
-			return (ENOMEM);
-		}
-#endif /* XBB_USE_BOUNCE_BUFFERS */
-
 		reqlist->gnt_handles = malloc(xbb->max_reqlist_segments *
 					      sizeof(*reqlist->gnt_handles),
 					      M_XENBLOCKBACK, M_NOWAIT|M_ZERO);
@@ -3489,8 +3318,14 @@ xbb_attach_failed(struct xbb_softc *xbb, int err, const char *fmt, ...)
 static int
 xbb_probe(device_t dev)
 {
+	uint32_t regs[4];
+
+	KASSERT(xen_cpuid_base != 0, ("Invalid base Xen CPUID leaf"));
+	cpuid_count(xen_cpuid_base + 4, 0, regs);
 
-        if (!strcmp(xenbus_get_type(dev), "vbd")) {
+	/* Only attach if Xen creates IOMMU entries for grant mapped pages. */
+	if ((regs[0] & XEN_HVM_CPUID_IOMMU_MAPPINGS) &&
+	    !strcmp(xenbus_get_type(dev), "vbd")) {
                 device_set_desc(dev, "Backend Virtual Block Device");
                 device_quiet(dev);
                 return (0);
-- 
2.35.1


--AXxWJ3w5Q3v8ydwo--

From nobody Thu Apr 14 06:47:19 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E8F191B32E03
	for ; Thu, 14 Apr 2022 06:47:24 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Kf94W6FWnz4TkG
	for ; Thu, 14 Apr 2022 06:47:23 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lf1-x129.google.com with SMTP id p10so7382194lfa.12
        for ; Wed, 13 Apr 2022 23:47:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=fx9/mZfkne3JyfgMXKrTmwCb0IzVHTV6hVq8j5IWquk=;
        b=q0TSOGE93mC7zbFzrUthl9qj6dJvP1EMowbUtX32gSj87ehcznD0bziPjcCk5URdr8
         wx82s9/gfzwHmTJeVWqCYodY8+4Lu8opP/0tSR6BOJIkrA8VWA8SQyvjkolY8CrPkvVq
         tdRuGKpgqIM30hLaEr/i6F1El3pfy1wUhjjWOus/uIGtUkR9ZWQjjUY5lJH1lj+wsQut
         VgvYLs293OkUiihkIMJXjrlZvSAE/2wXYrji6JHHbgUe8EM2Sad1ZOLCpMWJND6W4TIK
         GMC11pLBMV+KaHBI+V/R3g5hakQpYFk70PqOSzPoRkbe3SEC2meBfzyzJZ247i4j+Z9w
         lTKA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=fx9/mZfkne3JyfgMXKrTmwCb0IzVHTV6hVq8j5IWquk=;
        b=I4FSYPIc5osUO5IX2Kc0xPLIH/jD9/H8NEAzSifgDQv0OIblpML+j5YHQgfEFw9GtZ
         FbJL17L2n9bA0FhZmbpuF2t+ltDF4FnSEbLnxCM9iL6qhKEJqI7qQwlOAcIms1HzjaaN
         lq48kdGvt32bLTn8c1VXyyLe10HUM6UjVpZGd4ovLHYNi419fl3rADGfBaAScMMtgCLF
         ZNAMfxeF9k/t00Y+pga7bb++nDdGfQKTt7XieAFGJnyzyAIf5HQQUN4CJoG1B+BBVqxe
         F1g0gWFJcPrYUgCOUg+6llpjeXinPxAEeDbU45xDIwKsPM34IZI1pWqHMuiw8tNL0c8D
         FFhw==
X-Gm-Message-State: AOAM530EBAROqIjsaOvJ2kQS1ANNbAsL8bIrjec7GUt8wR1HhqO4PJCL
	K+DHcjOqx9r4X3zIwo/B9pM=
X-Google-Smtp-Source: ABdhPJwGgam6VbxVay4ZjjC0IYJugal+IVJtYcqHSneEoYYoqhZUPF95xnGdqsE2fG1RDFSmLF3JdA==
X-Received: by 2002:a05:6512:3fa6:b0:44a:cc1f:4a52 with SMTP id x38-20020a0565123fa600b0044acc1f4a52mr971933lfa.456.1649918842304;
        Wed, 13 Apr 2022 23:47:22 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id bj8-20020a2eaa88000000b0024c8f61c960sm105098ljb.41.2022.04.13.23.47.21
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Wed, 13 Apr 2022 23:47:21 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
 <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
  
From: Ze Dupsys 
Message-ID: 
Date: Thu, 14 Apr 2022 09:47:19 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4Kf94W6FWnz4TkG
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=q0TSOGE9;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::129 as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-0.40 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(0.72)[0.716];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_SPAM_LONG(0.89)[0.888];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::129:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.04.12. 18:37, Roger Pau Monné wrote:
> 
> I have a patch to disable the bounce buffering done in blkback
> (attached).

Regarding this patch, it wouldn't apply on RELEASE 13.0 code, because i 
guess there are other changes, when updating code manually i got compile 
error, variable xen_cpuid_base not defined, and include paths do not 
have  at the start.

So i applied it manually to main branch, commit: 
2f0baec1ecb2e79cfb97a19e1bbfb27741365847, since there at least variable 
xen_cpuid_base existed.


> While I think it's not directly related to the panic you are hitting,
> it's long time since we should have disabled that.  It should reduce
> the memory consumption by blkback greatly, so might have the side
> effect of helping with your issue related to pmap_growkernel.

I think i do not understand this clearly. Was it an obsolete feature 
that had to be removed?


> On my test box a single instance of blkback reduced memory usage from
> ~100M to ~300K.

On my lab system this seems to work quiet niceley as well. System is 
under load now for 17:36 hours, no panic, collecting vmstat. And what 
else is different, that VM start/stop script has not printed any error 
about adding block devices.

So this seems promissing.



From nobody Thu Apr 14 07:20:25 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 23FB31B3C30D
	for ; Thu, 14 Apr 2022 07:20:36 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Kf9pq2gJDz4n5n
	for ; Thu, 14 Apr 2022 07:20:35 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x22a.google.com with SMTP id a8so4685893ljq.5
        for ; Thu, 14 Apr 2022 00:20:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=2WyCGFbO6pH1ZPsdVrpvtKd0c6ZI6lOn8YCv1QrGNRE=;
        b=hzpW9LaZMaekfp1+heed8U6zoJciQLPoplhojj/fmMVCX9k8wIfniVEmDgaeE1pzKp
         gDDVWeMIEmAoqSg9RfjgNQKsCkXE1XXCN3TPL7+PIgZbnLonzp5zD5qKcWBOqnL5MR+y
         sYQA1RCsTLMA4DhtSG0f08Dipz/TlDeznm/wN6WGTkUIlHw3Uyvc66J8V6TUgMpU6Ps6
         Sr4MX5iMGjGdWWv8L3No4JEE3edVAYz1hwcM6qjqZm5TFthXuQS4zJfvEOrsbp7OuxF5
         M3YGP28SWJ0H9fLkfWLWzFwThWyNud2XJRYL94e/C5zwiVvb0gDpv+GcKGMutLvAUUzr
         0h7w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=2WyCGFbO6pH1ZPsdVrpvtKd0c6ZI6lOn8YCv1QrGNRE=;
        b=I/qjzejM+4xbFA+UOSvFsplUR31BsapBHcYTViFkLBuO44sAhpHfq+6AfINAY75FxR
         6K8hsQrmHgVXtmjSY7RFrfLLejuxSzF1pE0oct3wgtDGzElFxGYWOEPCcyv1IMAgxGBf
         +ipw76VmyUsKOWzrnEiG45SMnY9EVk50Hk3e5DHHuugf60722seNmgH3qidUJVcakaH5
         irc/9LR4lrs0Jv4Axo3iDtQC8dhlf56pvccz5W4sGHiavUYHeboYtwnXfj1lYpviu6Ex
         spJDIKkbf8ogZsRhR9XUjJf7Dk8lPNw1RjOBTrX3/NOfWHp/4nQ+7PdEfzeokoiyU3G0
         wcWQ==
X-Gm-Message-State: AOAM531FxRPICZPg2OiwsX7Cc/XVLjeBgplGMOvnjF+5SwUgZfQSiJiD
	LFQvEJWJHIdwIzvOaWavxCs=
X-Google-Smtp-Source: ABdhPJx2QoV6YIIEI/qKy9o39MlmWHQjmTHCg3HMrzF2IlFGT0GyIwGCgdS4dMXvRn3c8C9M7zwlbA==
X-Received: by 2002:a2e:a54d:0:b0:24c:7fa9:f2ca with SMTP id e13-20020a2ea54d000000b0024c7fa9f2camr856065ljn.183.1649920828392;
        Thu, 14 Apr 2022 00:20:28 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id d11-20020a194f0b000000b0046babb3b4e9sm147933lfb.8.2022.04.14.00.20.27
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Thu, 14 Apr 2022 00:20:27 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: 
 <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com>
 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
From: Ze Dupsys 
Message-ID: <2dbf24f9-2bc8-a8d5-e31f-90ec2c4b64c5@gmail.com>
Date: Thu, 14 Apr 2022 10:20:25 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4Kf9pq2gJDz4n5n
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=hzpW9LaZ;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22a as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-0.69 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_SPAM_SHORT(0.33)[0.327];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_SPAM_LONG(0.99)[0.986];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22a:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.04.05. 18:22, Roger Pau Monné wrote:
> I've pushed the changes to:
> 
> http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
> 
> (This is on top of main branch).
> 
> I'm also attaching the two patches on this email.
> 
> Let me know if those make a difference to stabilize the system.
> 

I do not know should i start a new thread, but i have captured another 
panic, new trace, this is on different machine, similar setup, 
RELEASE-13.0 + 2 mentioned patches.

I do not know how to reliably repeat it, nor the cause. But i have 
suspicion that this happens when doing some of steps like: create new 
ZVOL, turn one VM off, add new HDD/ZVOL path to VM in cfg file, start VM 
back up, inside this VM do some HDD load on newly added HDD (install 
stuff, extract data, etc.) + something of: shut all VMs down one by one, 
then do init 0 or 6, or create new other VM. On this machine i can't 
experiment too much, no serial output available either.


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x68
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff821dc99d
stack pointer           = 0x28:0xfffffe00c6b497d0
frame pointer           = 0x28:0xfffffe00c6b49870
code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (xbbd26 taskq)
trap number             = 12
panic: page fault
cpuid = 3
time = 1649915274
KDB: stack backtrace:
#0 0xffffffff80c57385 at kdb_backtrace+0x65
#1 0xffffffff80c09d61 at vpanic+0x181
#2 0xffffffff80c09bd3 at panic+0x43
#3 0xffffffff8108b187 at trap+0xbc7
#4 0xffffffff8108b1df at trap+0xc1f
#5 0xffffffff8108a83d at trap+0x27d
#6 0xffffffff81061818 at calltrap+0x8
#7 0xffffffff821c035a at dmu_read+0x2a
#8 0xffffffff8218da3a at zvol_geom_bio_strategy+0x2aa
#9 0xffffffff80a7f074 at xbd_instance_create+0xa3d4
#10 0xffffffff80a7b00a at xbd_instance_create+0x636a
#11 0xffffffff80c6b021 at taskqueue_run+0x2a1
#12 0xffffffff80c6c33c at taskqueue_thread_loop+0xac
#13 0xffffffff80bc7c9e at fork_exit+0x7e
#14 0xffffffff8106289e at fork_trampoline+0xe
Uptime: 24m0s
(ada0:ahcich0:0:0:0): spin-down
(ada1:ahcich1:0:0:0): spin-down
(ada2:ahcich2:0:0:0): spin-down
Dumping 2922 out of 6104



cat panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e 
/usr/lib/debug/boot/kernel/kernel.debug
/usr/src/sys/kern/subr_bus.c:2410
/usr/src/sys/kern/kern_racct.c:632
/usr/src/sys/kern/kern_racct.c:617
/usr/src/sys/dev/isci/isci_sysctl.c:92
/usr/src/sys/dev/isci/isci_sysctl.c:0
/usr/src/sys/dev/isci/isci_oem_parameters.c:130
/usr/src/sys/dev/hyperv/input/hv_kbd.c:540
??:0
??:0
/usr/src/sys/dev/xen/blkback/blkback.c:3083
/usr/src/sys/xen/xenbus/xenbusvar.h:96
/usr/src/sys/kern/subr_kobj.c:145
/usr/src/sys/kern/subr_module.c:255
/usr/src/sys/kern/kern_event.c:0
/usr/src/sys/dev/hyperv/pcib/vmbus_pcib.c:1158


Full output of (kgdb) backtrace
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=) at 
/usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c09956 in kern_reboot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff80c09dd0 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff80c09bd3 in panic (fmt=) at 
/usr/src/sys/kern/kern_shutdown.c:843
#5  0xffffffff8108b187 in trap_fatal (frame=0xfffffe00c6b49710, eva=104) 
at /usr/src/sys/amd64/amd64/trap.c:915
#6  0xffffffff8108b1df in trap_pfault 
(frame=frame@entry=0xfffffe00c6b49710, usermode=false, signo=, signo@entry=0x0, ucode=, ucode@entry=0x0) at 
/usr/src/sys/amd64/amd64/trap.c:732
#7  0xffffffff8108a83d in trap (frame=0xfffffe00c6b49710) at 
/usr/src/sys/amd64/amd64/trap.c:398
#8  
#9  0xffffffff821dc99d in dbuf_write_children_ready (zio=, buf=, vdb=0x0) at 
/usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4642
#10 0xffffffff821c035a in arc_evict_impl (state=, 
spa=, bytes=, type=) at 
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:4377
#11 arc_evict_meta_balanced (meta_used=) at 
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:4443
#12 arc_evict_meta (meta_used=) at 
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:4533
#13 arc_evict () at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4627
#14 arc_evict_cb (arg=, zthr=) at 
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:4938
#15 0xffffffff8218da3a in zfs_deleteextattr (ap=0x1430f6000) at 
/usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:5592
#16 0xffffffff80a7f074 in xbb_dispatch_dev (xbb=0xfffff8011a6ff800, 
reqlist=, operation=, bio_flags=0) at 
/usr/src/sys/dev/xen/blkback/blkback.c:2207
#17 0xffffffff80a7b00a in xbb_dispatch_io (xbb=0xfffff8011a6ff800, 
reqlist=) at /usr/src/sys/dev/xen/blkback/blkback.c:1767
#18 xbb_run_queue (context=0xfffff8011a6ff800, pending=) 
at /usr/src/sys/dev/xen/blkback/blkback.c:1987
#19 0xffffffff80c6b021 in taskqueue_run_locked 
(queue=queue@entry=0xfffff8011a9f1e00) at 
/usr/src/sys/kern/subr_taskqueue.c:476
#20 0xffffffff80c6c33c in taskqueue_thread_loop (arg=, 
arg@entry=0xfffff8011a6ff800) at /usr/src/sys/kern/subr_taskqueue.c:793
#21 0xffffffff80bc7c9e in fork_exit (callout=0xffffffff80c6c290 
, arg=0xfffff8011a6ff800, 
frame=0xfffffe00c6b49c00) at /usr/src/sys/kern/kern_fork.c:1069
#22 

If it is better next time to filter function variables from kgdb 
backtrace, let me know. blkback lines don't match again, but kgdb output 
seems more meaningful.

Thanks.

From nobody Thu Apr 14 07:39:45 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AE0851B4074B
	for ; Thu, 14 Apr 2022 07:40:06 +0000 (UTC)
	(envelope-from prvs=09619e58f=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KfBFK5B5Wz4qnC
	for ; Thu, 14 Apr 2022 07:40:05 +0000 (UTC)
	(envelope-from prvs=09619e58f=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649922005;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=y4LBap1nHkPwFc/3hrXayMA9anWP194jdKYh62URCbE=;
  b=HwoScQHZygqZYgi7lLa3i5qS1Tkp79IaEFUDLyAuWIwM+Veckn0Pq3o9
   X6Eoeq3w15KfIWNk2JFqde/yj2hyTpG4/5iYKyHEtpKbtDcmtf0rS6RW2
   VwFIFgxDY0olFfiAaukGMZ1RukHQ9FRsEKoQ+7XTLdQs2CE+GYcilbCv2
   Q=;
X-SBRS: 5.1
X-MesageID: 68812765
X-Ironport-Server: esa6.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:y4OsFqs5RwSey40Kg8H3E5ix5OfnVEBeMUV32f8akzHdYApBsoF/q
 tZmKTuFbKneZ2D9et0jbt+29k4GuJPRzIVjTANsrns9FSgU+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1/U5
 YuryyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi8iFYnBlrszdSJRUA0kLaca5o3GOXWG5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AN
 5pAM2UxMXwsZTVQKwxUWbtuhNuwn1bQdwB+qXiyp4Y4tj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUxwyWAxPX6xT2QRUReXjQddu4nrMIPEGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U
 J4swZn2AAcmV8/lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Y5ZeIWa3P
 BSI5Gu9AaO/2lPwPMebhKrrVawXIVXIT4y5Bpg4kPITCnSOSON31H43PhPBt4wcuEMtjbs+K
 f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq
 Ig32grj40g3bdASlQGOqNBIfAFWdyVjbX00wuQOHtO+zsNdMDhJI9fawK87epwjmKJQl+zS+
 Wq6VFMew1367UAr4y3TApy/QNsDhapCkE8=
IronPort-HdrOrdr: A9a23:X+YAoaNoBCv7PMBcTyP155DYdb4zR+YMi2TDiHofdfUFSKClfp
 6V8cjztSWUtN4QMEtQ/uxoHJPwO080kqQFnLX5XI3SJzUO3VHHEGgM1/qB/9SNIVyaygcZ79
 YdT0EcMqyAMbEZt7eC3ODQKb9Jq7PmgcOVbKXlvg9QpGlRGt9dBmxCe2Cm+yNNNW177c1TLu
 vi2iMLnUvpRV0nKuCAQlUVVenKoNPG0LrgfB49HhYirC2Dlymh5rLWGwWRmk52aUIE/Z4StU
 z+1yDp7KSqtP+2jjfaym/o9pxT3P/s0MFKCsCggtUcbh/slgGrToJ8XKDqhkF/nMifrHIR1P
 XcqRYpOMp+r1vXY2GOuBPonzLt1T4/gkWSvmOwsD/Gm4jUVTg6A81OicZyaR3C8Xctu9l6ze
 Ziw3+Zn4A/N2KOoA3No/zzEz16nEu9pnQv1cQJiWZEbIcYYLhN6aQC4UJuFosaFi6S0vFqLA
 BXNrCc2B9qSyLbU5iA1VMfg+BEH05DUytue3Jy9PB8iFNt7TJEJ0hx/r1qop5PzuN5d3B+3Z
 W2Dk1ZrsA/ciYoV9MOOA4ge7rANoWfe2OEDIqtSW6XYZ3vfUi976LK3A==
X-IronPort-AV: E=Sophos;i="5.90,259,1643691600"; 
   d="scan'208";a="68812765"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=oIRFs/5QzBVu51pd2VGxsQlHcEWg4stSeuUIXvIXr3VooIkqTS6KlS3H3+yF9h6CYx+40AUuWo81MchnWVLdzUqu3au0bqmh9CIEkaYX5IHPWSeYKpL1OdsyGOeGRHLOtHK2U3Ya6ETuQPEvMG7RlUOzIWSTRI6/9IYUP1MVDoT5wdan88vfYpKnu04BV6D5VPV3OeALMjTXz8/msm4Z57VCmO+VA7UrSRmfYZ3cmn4lzHkXsTpLOqencwvK2sDj4t8a4Vs+ByIuUI/Y0KomI+YMOlthC7bWnfcbxReESID+wZ7Zyd5NQ9miNjCloRY5SA6nzkB5HV3GV2jTPbTUUQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=JGU2HhJ+hbsJU9X76LPYzzbPV5/dqB+l02CyBYFV6gk=;
 b=AQdBsnrRXzlg1CtMqdujq7hOTP6Gy9WWoPhHUyBWYvSj2ae5jVt5KsJ5+ZXgofXZ8VeI9T9H/k7LtfJB/CFUcAh7eUBEDrDGFSXjsLgObQu5oa9YvH2b0zzGvIXrZttpg+2kfdZi/ei+j0B3DRqTOr2BCqRBouXCJLh7USBNJC2EzYUZfvsPSIwzAIz+yOdxGIDsyw7AaBQQKvHsUzooPu1/mldxkzdxR4yOlDrVijIHUYLjD5Rt8pkKzaMaws8B/HxxNIWiblRVB2lDc6JpLrSYAMjXPnHm92LGq+fuxKHeqrlD8DAJN0dRHYlkiBAO1nX97ow1+llrZUILm2++kg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=JGU2HhJ+hbsJU9X76LPYzzbPV5/dqB+l02CyBYFV6gk=;
 b=aHv6nqyD24s9Pm2x6h1cYC71fmU3InIKfU9JuK/QzOdpWGsH0xbpd6pohy/4cqkaAKTPNa+kOZCBGpugA/XbdQSXJpMSKPmeXHF18S9fbMn5QOw+i4Z05ZkU62GhDd9OYqpW8xm7KX3MBwJtWZ4VBZ8PAS2C0aYafPloMWGFlF4=
Date: Thu, 14 Apr 2022 09:39:45 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
 <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
  
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: MR1P264CA0134.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:501:51::14) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 15b110d2-ed96-49b8-b59f-08da1de9f4e3
X-MS-TrafficTypeDiagnostic: SN6PR03MB4302:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: wmO08w2XZIQa+gaClM+BeuOP6hWjtuXSDSLV6WYcYQft6PxmyUKqGXTMTg98YQURZkTO40QAhTb5V23WNvIizDxXMjHv19MGEAGwLq/WLhGBbRcyNHvl/zlOs4FLoFQLLgjEVUiovJc1kreSduHsBvMrpMRVyGqf6oa86ha+IYmnkqsLpAptlmMS8Klmqn40M7wXyHXkHIlUruJlLLfCaylVlJduGz3tMfUToFxbVExRDwFRWwl1oD28qB+3XiJPLRk8hkw4r1HTIahkQtw/FsKXySn10s1PTemlZ/3B/gejLeT+aws+EYtZyB5M+I3kfQVTxeW6JbwFEntZ84Zv3BQ2+vMou5TrbiOgXXM2GyO0WQHom+LL4ueSBUJbVassq+CmjiIBBRk14oNt2139opW8oED3NM0r9K5lAP9bRQBzCIfShL2rkYGttyUj+Bvn+qNBEANknn6dKw8sjmJEdQeFyztHTS+MbhJkXlyAcu9XnXB85ts/1HMNC2PtHFgQme22kUvGL+amJuVidHlON0kDInaJlrXLszU/CyngeiYd29wEK0BIXp9vmV2IdhIm0Vk/cTzl6q7AuBk5HX7cPHqAeUUfZxH65cav42QiYLbG9ImY/OIhPyzpQn2ZCiUW16MjmA0IJfG3drS4WpSnOQ==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(2906002)(86362001)(6916009)(9686003)(6486002)(6666004)(5660300002)(82960400001)(26005)(85182001)(316002)(38100700002)(6506007)(83380400001)(6512007)(8936002)(508600001)(66556008)(66476007)(66946007)(8676002)(4326008)(186003)(33716001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eTJ5dUtjL2lIZ1RYbmFUWVZRRjZGa0oyemlUbS9RSG1qRDBZNzQ4UkxMcU9u?=
 =?utf-8?B?aTMvR1VNY1dlRWI3OUZGSXY4cWhOdG83ckxmSkYvTWovQ0RmV1E5aVVuODJX?=
 =?utf-8?B?ZGFCT1NqakVDSzR5TmdQSmNFNE9Wcld4ckFaRWJUWFIzcWE3ZEw2c0VEamZz?=
 =?utf-8?B?N3VQNGw0Ty8vZUFSQ0VpWWtxV1YrZm5ZZGxzRkhuelBzcWh4Q1AxeG9mRVZs?=
 =?utf-8?B?bTl1YkFMVysvdmNpRjJTUFpmNnVLMjNEQ1dYVUVyZS8wZkhkZ1ZZSjUvclpu?=
 =?utf-8?B?QjkyUHlJT29KNFYzbmVTWTRYekpjSjNrZGZhNFUycmxiYklqdUZGMEJHZnR6?=
 =?utf-8?B?Znh1NmgvWGhMb0N5RDdYT1JERTloOS9pTFRxcGFhK2lPd2hPN1F2dDJhMndo?=
 =?utf-8?B?bDVmZ2F4WVRTTXlNalhRazRML0hrMjF3eEE3NmR3ZXVQZDdrYVh3amc5a25Q?=
 =?utf-8?B?WUFzLzlBb3puRlYvQUYyY3hGRGVqSjAwSmlOWE5YZWdvcE4yUGJMMW9kVE5v?=
 =?utf-8?B?bEtudlJzcnBaNXNmeDBwYnVaOVJ3ZmhZdkxRcUtyenIrUFppcWhsYXo1eVZj?=
 =?utf-8?B?UVNyN0V3WGc1QmpkTncweForY3ZyNWdwcXJmc0tjcW1vcjlPRW9tRnlIZ2RQ?=
 =?utf-8?B?MVhaMlhnTnR1b05ycGhST3NJVXlXcWVMdlJnVGwrVnV5ckFjNkxuOXE3Sk9u?=
 =?utf-8?B?ZHcrOVdxMis5Z212QTB6bUlGSSt5RENlcGVwcTdUNDNkZnVVb3ZnZnNvczVN?=
 =?utf-8?B?NTJWV0dSS2s2ajZ6S1FuZ0Npb1ZjS2YwU096NDlNR3BWbmYrK1ZSQ2hrcGI0?=
 =?utf-8?B?aE5zV3NLK1BOcy84VkE5NUNWa0NMQlA2Zzl5MzViSGNVL1lwc25Qdy9RNzZM?=
 =?utf-8?B?bEgxOFJqVmwvTUx4RUYxcjQ0dFB6Tkt2RDgzVU1NUHdFcDNkb2ZoY25BbkVW?=
 =?utf-8?B?eDM4L09FbzM1RkpHQWlDMHRmdk9UUUtJeDN6VGwwR1FBLzBrUjY5bnN3aWh6?=
 =?utf-8?B?d2RrM2l3UHh4UG5Oa1hMZEY2aHh4YU9DRFY4WmNEOEo4SW4yQUphNHh6Kzd5?=
 =?utf-8?B?UVZVSDRab283aXNkRFFLM2M5REdWWHZwdnVmWXd5RGlSdUt2eXI1YlA2SzRj?=
 =?utf-8?B?eDZlZUoyMFo0bEszbnd4bTNBSnhUVGpxbTZkK21JZ0ttWFVaNVZBUWZ5b3Jk?=
 =?utf-8?B?bXJZNmlYS3RORCtvK3Q5N2luT2d0WmdoZVBYeW1OVjNDS1VZWTZTTEhNeDNu?=
 =?utf-8?B?cWJVcU9KY2h4eCtnU1lDWGVUSGxXUk9FNkl6TG1ZNGtZK3pyNWVGL1JmdWhH?=
 =?utf-8?B?NWt2Y0xPMjJyRHg1Q0lESG9lUGJtODYyK3RKT0pOMUdhRUYyUHRzUzYvMksr?=
 =?utf-8?B?YnB0QTUyOGtnODRkTy9QcHRlNmpONlFLdmtud0VhRVppc1l6SlJkUE1UVUFS?=
 =?utf-8?B?Qlk4bXNPV3AzcExmTmtxWWtrS0FJWitWZXJvSnNGWWU5K2ZxRE1HVmpmakt2?=
 =?utf-8?B?UGE3dmJjMmJKOHFiMmxWQlJ3WnVndGRwUFIzRHN2TWx1RkxEcjFNZ3BicnhI?=
 =?utf-8?B?S1FHQzJMamRiRHRKbUR2a08wdldxNUpYcmVPT01ORkN1VXhYTmV1NlpyT3BJ?=
 =?utf-8?B?V2RadjlKSWswb1lJQ3BESHRTQ04yVzFSa2VuOE1XSWMxWk9BbitDd2Z2MFFD?=
 =?utf-8?B?Y0FkcjFBcWE5RmxJcHMxQnJzT0lrQjlpa2szdXFwMzVjNlhIVXQvNE1lcGk3?=
 =?utf-8?B?czNJamRRdGhBQ2lERG4xVC84Mlg0dmdDdVJmWGlHR2RYeVREb09xNi9vWk15?=
 =?utf-8?B?NURuMWZDU2VtaFV6aDVhNzNSeXp0T0ErLzZubTRVRHg5dSt0ckRjTHp4OEZ5?=
 =?utf-8?B?OXBxQ29BOHJ6aC9yWVlSRkxxa1d4N1hIYzNvTjA0N2ZRdVloNlNEV1QveEpK?=
 =?utf-8?B?VFRBU0hSTnR1ZzBwNG1QblhwRmZ5TXp6MFAzd1JPRm9iZXVmbG1NS3FQcUJM?=
 =?utf-8?B?U0hsM3NheTF4ZVpzWGhjaFhybU5NRjlSYkV3YWZodFV2R2NaRkNzN2RkN2hC?=
 =?utf-8?B?dmNzUTd5OUhkTC9mWEF1QUxwaVRyMCtsbHlnMzFGMXNpZjJjRVBleWg3aHd5?=
 =?utf-8?B?ZFE3Um90eHhGQTRkUW93eFFpKzJjdmhQeHlqdHp2TlJyT1R5R1dVYTJpcExt?=
 =?utf-8?B?WmJ0dFNIQUlNd2ZOQWJCRDZsMktQeHNlMzFha0JhM3IvenVxNHh1dnBjdEdK?=
 =?utf-8?B?dml0cGNUZS9NTUROeWVwMnIxY2VEVU1uSE1oWUZPR0tYYU94RC9zbFRrdWMv?=
 =?utf-8?B?aE4ydDF0Mk5EMTdLNVpacU1Yayt5dVFlb3JRZHF3ZkxMNTRJV0pOdz09?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 15b110d2-ed96-49b8-b59f-08da1de9f4e3
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2022 07:39:50.2776
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GPJjQBe2rB3qHJ3MY6ITbgLPkGBeiZUK7zGMMqB2hoM5B4MqOU+5xt/ZHUzcqlXPSCPL5t10DDRBfAGN1Fjnxg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR03MB4302
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KfBFK5B5Wz4qnC
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=HwoScQHZ;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=aHv6nqyD;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=09619e58f=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=09619e58f=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.63 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 NEURAL_HAM_SHORT(-0.99)[-0.987];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=09619e58f=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Apr 14, 2022 at 09:47:19AM +0300, Ze Dupsys wrote:
> On 2022.04.12. 18:37, Roger Pau Monné wrote:
> > 
> > I have a patch to disable the bounce buffering done in blkback
> > (attached).
> 
> Regarding this patch, it wouldn't apply on RELEASE 13.0 code, because i
> guess there are other changes, when updating code manually i got compile
> error, variable xen_cpuid_base not defined, and include paths do not have
>  at the start.
> 
> So i applied it manually to main branch, commit:
> 2f0baec1ecb2e79cfb97a19e1bbfb27741365847, since there at least variable
> xen_cpuid_base existed.

Right, there's some context to be backported to stable/13 for it to
apply correctly.

> 
> > While I think it's not directly related to the panic you are hitting,
> > it's long time since we should have disabled that.  It should reduce
> > the memory consumption by blkback greatly, so might have the side
> > effect of helping with your issue related to pmap_growkernel.
> 
> I think i do not understand this clearly. Was it an obsolete feature that
> had to be removed?

It's a workaround that we had in FreeBSD to deal with older Xen
versions not creating proper mappings for grant references. This is
now all fixed in Xen since quite some time ago, so we can drop the
workaround in FreeBSD.

Such workaround implemented bounce buffering in blkback, which takes
an awful lot of memory.

> 
> > On my test box a single instance of blkback reduced memory usage from
> > ~100M to ~300K.
> 
> On my lab system this seems to work quiet niceley as well. System is under
> load now for 17:36 hours, no panic, collecting vmstat. And what else is
> different, that VM start/stop script has not printed any error about adding
> block devices.
> 
> So this seems promissing.

Thanks. I will groom those patches in order to prepare them for
commit. Regardless of whether there are other issues still lurking I
think those changes are worth committing now.

Regards, Roger.

From nobody Thu Apr 14 07:49:27 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4AB4E1B43CCE
	for ; Thu, 14 Apr 2022 07:49:42 +0000 (UTC)
	(envelope-from prvs=09619e58f=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4KfBSP2Bw1z4tc3
	for ; Thu, 14 Apr 2022 07:49:41 +0000 (UTC)
	(envelope-from prvs=09619e58f=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1649922581;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=kThIC6pGNn0YEhnmJnsHd/o31FIaLOKAYDBbQzxvDSk=;
  b=fcbV/ENYneheODrrrLcM9OnqrXHO4jhULwnq4bFCBKxOKY36zOxPkLRj
   gn5KTn2ixE0YZc7+7gPkkq5LRFdI1mcFNGR7I4Y6OHHQbq6PB6t6kMnpc
   SZu5BhKdIeWCWh/IduTJmgyPabLcVKWQL/nxuN4W0soX4Hq5GjcA6sZE5
   E=;
X-SBRS: 5.1
X-MesageID: 69336493
X-Ironport-Server: esa1.hc3370-68.iphmx.com
X-Remote-IP: 162.221.156.83
X-Policy: $RELAYED
IronPort-Data: A9a23:cBOMM6tudQYXpC9vrG4OVUGJDefnVEBeMUV32f8akzHdYApBsoF/q
 tZmKWzQM/bYa2r8LdAgPNm38R8CsZHRx4RiGVQ4pC88Ei0W+JbJXdiXEBz9bniYRiHhoOOLz
 Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1/U5
 YuryyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb
 /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo
 Oihu6BcRi8ZBLLVnuciDSJ6MDwhD6NKorLXIWGG5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8
 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AN
 pFCN2Y+N3wsZTVCFFIRBINhoN3w2FKgUDx7+AK6hPEotj27IAtZj+G2bYu9lsaxbc9Inlyfu
 UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI
 EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUK0h6UzKnrvT27JXVbfDBNMtE2tNYPEGlCO
 kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/
 tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U
 J4swZn2AAcmV8/lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Y5ZeIWa3P
 BSI5Gu9AaO/2lPwPMebhKrrVawXIVXIT4y5Bpg4kPITCnSOSON31H43PhPBt4wcuEMtjbs+K
 f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq
 Ig32grj40g3bdASlQGOqNBIfAFWdyVjbX00wuQOHtO+zsNdMDhJI9fawK87epwjmKJQl+zS+
 Wq6VFMew1367UAr4y3TApy/QNsDhapCkE8=
IronPort-HdrOrdr: A9a23:ND6Asa4B9BRsKsopcQPXwPTXdLJyesId70hD6qkXc20wTiX4rb
 HKoB11726WtN98YhAdcLO7VpVoI0mskqKdiLN5Vd3OYOCPghrPEGgI1/qB/9SPIU3DH8dmpN
 5dm/cUMrbN5CJB4voSSjPIa+rIu+PqzJyV
X-IronPort-AV: E=Sophos;i="5.90,259,1643691600"; 
   d="scan'208";a="69336493"
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=oXoe6NXKAkmPzxLaRIFWoAn9Je2pBEFLMErTZUVi4+Z9jfggya/qRPLPGb5aAx5UNQw3P3j6R53NHIqN/BLtrhQ2UEW2dsHv2ZbxlGtmQOq8BoLIPK895kHS4rGGsyxAntZ3lp+3RfP6A3HXekFPmjSSaJLks1It6wq5CSQ+lCVBaNe9eLmTlHhTUtw65tvETPdFgpErjqd330AxZNjgp/4oYtk8h1LUuOm1uFSHSl4kMoOOUq14oXtblj/tCypqEevrk5hzPD24IrMpL04W/bhoyaszw0DWdv3tmaVdh0DPpJEHTeSFPysl0dOI/BixgIR2lX2q0jA1KWVdfcmPMg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=2rq1r/Sfs+8UVVf/JV0u3TqTHggjrko+aiB19/kSe5s=;
 b=bPpI+KyTCvMDBwxHIQQJ3fIft8yHglBCZScdSrwXxYU5UBlj9B4hVDa/oc0jhgDvkN/ibHdk61wem9FQDPa/oBDM7Khv6k0SUaafNPN1ZBakGKdV+1J/oZ6TXuEXkhwJK6Ez7F2iOsSL3ckgRhxEbk0sOwi/1B2HvHsxk2oNrMIQfSt+5Ynd7RejxpV+NtfZkU2J1s+wp9nvS7/KB+cTHtT7DykzXc4I7ai2itsu5L/sbxMy2cy87lrWi30IReidEacVKuzYgBrOldC0qATvBLjwx8eyXIz8WSQJAgnxcif+FL9dGQohF4Y1mFHQBNAsyLi1r4UAJYG1CibFhoxGlg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=2rq1r/Sfs+8UVVf/JV0u3TqTHggjrko+aiB19/kSe5s=;
 b=fjOZxa2pKYUWXKKB79IFBWAqaLzK9TfyxNX64AXDrjt6GzG9/WxcvGzW0wFZcQOgWoY0/5Qkjx4BF9cMxta95QMWpBew4fLwm7HXzKWN5z/Z38xDG/Q+dYsOAkZCvFoHcO8B7T988iL0BdkZDACJkbWTPPtmgvFQbYxWRWnCG20=
Date: Thu, 14 Apr 2022 09:49:27 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ze Dupsys 
CC: , 
Subject: Re: ZFS + FreeBSD XEN dom0 panic
Message-ID: 
References: 
 <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com>
 
 <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com>
 
 <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <2dbf24f9-2bc8-a8d5-e31f-90ec2c4b64c5@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <2dbf24f9-2bc8-a8d5-e31f-90ec2c4b64c5@gmail.com>
X-ClientProxiedBy: MR1P264CA0051.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:501:3e::26) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 5552b399-a954-4010-7b4b-08da1deb4fd2
X-MS-TrafficTypeDiagnostic: DM6PR03MB3948:EE_
X-Microsoft-Antispam-PRVS: 
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: /gpBkL2ag9ctEeUtAwkXECpHeBA33YqYmZMXjXuvTGrft/ANSPT8G7dTNn69AkwGXgrAUldpt0rY+nk9ZljIBUk97YLLi5OzERz56XEjVqHfOljNVh8NR/lwdAREUB0hyS3Xwg8phtc5rJr4XdJ4Ifs66l8oDejXDBwy5eVmJxrL5H8U89XEZFBhbJbp73maEza25reum6y+T9WaD0KL91dgnHBmM/McIx40DdOIte5Ymyn0bqRgoaMUNYJJVxF4n6HUfJO58zpE4/reQj171Re7G3vX273y4ANrUMENT37ZwETjCr2v+Yy1NJS3JD34IRDskhjqfS7BtZHa5+wzGRokrTettjxw8j83VW4U7B+OXP53QT/OJZRaPYYZNG8ydIiG/e57kx0X9Y5FkgoRKJTvLRKZ85o2P058M5TXGlm8Fkfs66cLnw99eQME08hLcW96tN4wvB8Urqiv+THTdEnOb6Ob5OC3YGpMLBCwfAUwX8yROMR6IM0Tg6p1XNwtZ3uSaMzggh8lsSccYxiaa73ywC0+yN4KgC+Z751bUCYSQ6S4BPc+5tnTcabFK6XsNBRHUoAKnUMSRhpAyntcjJjuRrmI2ZPkNs9Oy1eevSzP3k3l/eIR+gpTJTNCK4aVCVJP1EmQS9BU8yqXlzwdmps4gbMZJeNJUDdE7ew4sOkH/GM4Cp8RZnyfWfWMFwzAFu5E+8okwp/9/yDFlFyxv0xBGIo3zp9+ejBg0ydNwSo=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(6666004)(83380400001)(33716001)(8936002)(66556008)(6916009)(85182001)(2906002)(9686003)(6512007)(6506007)(38100700002)(508600001)(6486002)(966005)(82960400001)(66476007)(26005)(186003)(66946007)(316002)(5660300002)(8676002)(86362001)(4326008);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c3B5MjkyRmdaNURuSS9HbWxtT2pDTHNXeG03Mi9UakFiSk9vNkJQVEE1RzNr?=
 =?utf-8?B?SmRsemt6ME5RaGlJRjZPcC9qcWVDajJGUXEyRmdwTWNzZTh0NFJnd1dva3N3?=
 =?utf-8?B?K0RmMS8wcjFSSGJtQkxYcmcrSERtOVVFNmRkWTFuY3VERWtPRlZ1ZzZTaDcz?=
 =?utf-8?B?ZVhxSjFUbGxTMUJWK2VxdnNpV1lpTmJTVXBjWW52WXJpYkd5NFZsRjdLQThW?=
 =?utf-8?B?TWVCWXV4VTV4d2NNLzFpb3hvZ0d3dTRmbDdLeFBRZzJHanVReW1LZzBQWTJ5?=
 =?utf-8?B?QVh4Q01kelFXRWZxRU1sTjlra3ZRdzJwWCtCYmhwSy9STUNNdmxIYis3dklV?=
 =?utf-8?B?dG04M1BHQXV6OXlUZTNLT3ZTdEYycUJvYmV4dDExbWNOUTI3djlmZjgyd3pK?=
 =?utf-8?B?eDFwUUd5K0w2UytyUDF1cXNNNUJ0UnlBalIyWUErWUx5UzdCbEVUZHA5YjZM?=
 =?utf-8?B?bk5tb1NpL3l1MTkzMEh6M2FwNFNoMVdHR1dsd0VDamxUY3ZTWkNyTDVTekk0?=
 =?utf-8?B?eUF1QTU5QnFTenJrYkx5N1VtdVNuRWFYdzVOdEFJMVFZUHIvTmhXMHJ1MEpK?=
 =?utf-8?B?a2NsRlV1R0Q3VlYzRnFHZTFzQWtsU3F1MHZjTHpJNFcyaGtmZmhqZHBxVk5p?=
 =?utf-8?B?WVhlOFVkRGRxbHZjVzJ6cjZEaWtWcXhza3VkOERubko1eFFER2lqdG1tSUVM?=
 =?utf-8?B?QTR6UThrVGZmSzJPNDJRZG9Pc1p6bXFoSTVDS0FGZUZEdkovRmJNbk9jT1dC?=
 =?utf-8?B?c0JlcllHSTZkTkhSSmRSSnFqYk4vcE5STWk4L1V4UWFhZUlLUVRSTjZsa1R3?=
 =?utf-8?B?MTNtMDFvU0tCQkV4OUxmT2pIVXBkYXZUanJ2TWVaQStuZGlLVmZlTkhRSEJP?=
 =?utf-8?B?T29HbmpqeG51UmhFbUZSNGVkNHRJaTYyRWNDK0FMb0RmdWVuVDN6N3NhTy9P?=
 =?utf-8?B?T0UxR3VUWE1TM3ArMm1GTG90TEp0d3NWa3p3eUpvejhXV1lnN1FxNjFqZDJX?=
 =?utf-8?B?UTR6WFgwL2ZyQnUxTytubjlDNDBMSkREVWREeXRGbzg2cTNyeHRFVm9icTF5?=
 =?utf-8?B?TGQ3UjNEekNHamwweUdxUjhnd3VkNWdYaU5kVXNoejZiM3dNYi9ObU9rTVdi?=
 =?utf-8?B?UVhXRmNYSXR2L3pJYzlueFRwem0wTTl3Vkpmdzg2ald6UllUMlB0VkxXMjA5?=
 =?utf-8?B?Nk5vQ1lZaSt4aVc4TEZpZVZ1Sks4a1FVUS90c294eW5NaEtJdEEydk5RZE1I?=
 =?utf-8?B?S3JMZ0s1aE9vVHNHaW12TXdOTlMxNFYxNk1ZYmJRZHE4dVg0ZmZFQWFuaVEw?=
 =?utf-8?B?QlI0MUk3NUhycmlBZ3FmS1JhN1FjN3FzbkZBM0hFcG83WmZlbjhjNThqYnRn?=
 =?utf-8?B?QldOTHJyaUF1dTIwZnNFK0lTQk5uMU5PYVEydkxBem5xaS9ORVc2aWgxWmlN?=
 =?utf-8?B?UWZsTDRPVnM3Nm5IK3R1d0gvTURjcWpveFBub2xxQkpwZHNORHJRL0tiVzBS?=
 =?utf-8?B?S1BWNmFIRVFMbmQwWC9veUVVdjdpTXdwdG1SVmEyd0FrUGlwRzhlSlQ2eXpM?=
 =?utf-8?B?QkJqNUVkZVNTT2NZZVEvcGVWYjdVQUdVVlFhVWF3TzFzc0dibXoxRnFoUjRS?=
 =?utf-8?B?ci9LMjBEbHRMd3R5ZEtKMkZ1WXVWVGZha2NkQmorK1pncnNVQnpLZ2V6NkhN?=
 =?utf-8?B?Vndoa3IwSVFhKzFCY2QxbHJ6NnI0eEhicHB4MldsUnBkUDdNUU5BOHlqdjVu?=
 =?utf-8?B?ZEt6U0U0T05RWXl5RkthNXlSSHJiMUZOa0d1OE1UQjJVaG1ZZXhuaEZiZERP?=
 =?utf-8?B?WTlKWWp6NmJNT3cxR3ZnTWFjNXVuSVdKbUJxdWUvYWZmZ0hKOFhUTzZUdWVL?=
 =?utf-8?B?MmdqQ1B4WEpyV2IyQnhyVFpsLzBUR3dBaUxyYUVROEtTKzVsTm5VMVN5MnpW?=
 =?utf-8?B?RXF1bDRlaWdvdWtQUS9vYkdINHVuYVh4V3FEVFAwQmFobnRtbFdsdEx2N0VX?=
 =?utf-8?B?REpGOVowU2xTS213M1R3QjZVMXRTdVQ1N05Fbzh2V0tlRjhRaWFsL3Jqdzda?=
 =?utf-8?B?VmdiUXArQmFBK1dnNEk2L1RXQ2xVNW1PNlVHOXhIT2FCMTlYaDh6TEFIZVRl?=
 =?utf-8?B?bE5ZbmhGbmx0UHRnQmpRVEtmdU5QVnN1TExndk5qTCs2Z1VBZmR3bHZhSkU2?=
 =?utf-8?B?czhWNm1ZYWg2bEJPOUxXakp1Skc1d2VSYi8wT3pBQkVKT3ozVkZ1UTRZaEJY?=
 =?utf-8?B?Yml5WmlPcTltWHNaaEphVHhDSUVlQ0ttV3JnVlRvZ01wUFZmYmxLdjhUMlhG?=
 =?utf-8?B?V3FDMzhwaGRrbkt5OXpyOWgwRGVJa0JoamNTYTNacUhKUjh6NEZkUFlrRUc1?=
 =?utf-8?Q?CpIXQxddWsNuHdLc=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5552b399-a954-4010-7b4b-08da1deb4fd2
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2022 07:49:32.0842
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ny5iNYPzmm1zn8PIaVMm12E1Flm84UB7yGQhlK6XvlR7jPdWXbDae4BK8DMmC2Z4UOyDN5NDKLin4y4BS1kawA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB3948
X-OriginatorOrg: citrix.com
X-Rspamd-Queue-Id: 4KfBSP2Bw1z4tc3
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b="fcbV/ENY";
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=fjOZxa2p;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=09619e58f=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=09619e58f=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.64 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 FREEMAIL_TO(0.00)[gmail.com];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+];
	 R_MIXED_CHARSET(0.56)[subject];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=09619e58f=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Apr 14, 2022 at 10:20:25AM +0300, Ze Dupsys wrote:
> On 2022.04.05. 18:22, Roger Pau Monné wrote:
> > I've pushed the changes to:
> > 
> > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak
> > 
> > (This is on top of main branch).
> > 
> > I'm also attaching the two patches on this email.
> > 
> > Let me know if those make a difference to stabilize the system.
> > 
> 
> I do not know should i start a new thread, but i have captured another
> panic, new trace, this is on different machine, similar setup, RELEASE-13.0
> + 2 mentioned patches.
> 
> I do not know how to reliably repeat it, nor the cause. But i have suspicion
> that this happens when doing some of steps like: create new ZVOL, turn one
> VM off, add new HDD/ZVOL path to VM in cfg file, start VM back up, inside
> this VM do some HDD load on newly added HDD (install stuff, extract data,
> etc.) + something of: shut all VMs down one by one, then do init 0 or 6, or
> create new other VM. On this machine i can't experiment too much, no serial
> output available either.

So you haven't seen this panic with the 3rd patch applied?

I guess that's also possible because when testing the 3rd patch you
are using a HEAD kernel rather than stable/13, so the ZFS code might
have changed.

> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 3; apic id = 06
> fault virtual address   = 0x68
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff821dc99d
> stack pointer           = 0x28:0xfffffe00c6b497d0
> frame pointer           = 0x28:0xfffffe00c6b49870
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 0 (xbbd26 taskq)
> trap number             = 12
> panic: page fault
> cpuid = 3
> time = 1649915274
> KDB: stack backtrace:
> #0 0xffffffff80c57385 at kdb_backtrace+0x65
> #1 0xffffffff80c09d61 at vpanic+0x181
> #2 0xffffffff80c09bd3 at panic+0x43
> #3 0xffffffff8108b187 at trap+0xbc7
> #4 0xffffffff8108b1df at trap+0xc1f
> #5 0xffffffff8108a83d at trap+0x27d
> #6 0xffffffff81061818 at calltrap+0x8
> #7 0xffffffff821c035a at dmu_read+0x2a
> #8 0xffffffff8218da3a at zvol_geom_bio_strategy+0x2aa
> #9 0xffffffff80a7f074 at xbd_instance_create+0xa3d4
> #10 0xffffffff80a7b00a at xbd_instance_create+0x636a
> #11 0xffffffff80c6b021 at taskqueue_run+0x2a1
> #12 0xffffffff80c6c33c at taskqueue_thread_loop+0xac
> #13 0xffffffff80bc7c9e at fork_exit+0x7e
> #14 0xffffffff8106289e at fork_trampoline+0xe
> Uptime: 24m0s
> (ada0:ahcich0:0:0:0): spin-down
> (ada1:ahcich1:0:0:0): spin-down
> (ada2:ahcich2:0:0:0): spin-down
> Dumping 2922 out of 6104
> 
> 
> 
> cat panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e
> /usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_bus.c:2410
> /usr/src/sys/kern/kern_racct.c:632
> /usr/src/sys/kern/kern_racct.c:617
> /usr/src/sys/dev/isci/isci_sysctl.c:92
> /usr/src/sys/dev/isci/isci_sysctl.c:0
> /usr/src/sys/dev/isci/isci_oem_parameters.c:130
> /usr/src/sys/dev/hyperv/input/hv_kbd.c:540
> ??:0
> ??:0
> /usr/src/sys/dev/xen/blkback/blkback.c:3083
> /usr/src/sys/xen/xenbus/xenbusvar.h:96
> /usr/src/sys/kern/subr_kobj.c:145
> /usr/src/sys/kern/subr_module.c:255
> /usr/src/sys/kern/kern_event.c:0
> /usr/src/sys/dev/hyperv/pcib/vmbus_pcib.c:1158
> 
> 
> Full output of (kgdb) backtrace
> #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
> #1  doadump (textdump=) at
> /usr/src/sys/kern/kern_shutdown.c:399
> #2  0xffffffff80c09956 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:486
> #3  0xffffffff80c09dd0 in vpanic (fmt=, ap=)
> at /usr/src/sys/kern/kern_shutdown.c:919
> #4  0xffffffff80c09bd3 in panic (fmt=) at
> /usr/src/sys/kern/kern_shutdown.c:843
> #5  0xffffffff8108b187 in trap_fatal (frame=0xfffffe00c6b49710, eva=104) at
> /usr/src/sys/amd64/amd64/trap.c:915
> #6  0xffffffff8108b1df in trap_pfault (frame=frame@entry=0xfffffe00c6b49710,
> usermode=false, signo=, signo@entry=0x0, ucode= out>, ucode@entry=0x0) at /usr/src/sys/amd64/amd64/trap.c:732
> #7  0xffffffff8108a83d in trap (frame=0xfffffe00c6b49710) at
> /usr/src/sys/amd64/amd64/trap.c:398
> #8  
> #9  0xffffffff821dc99d in dbuf_write_children_ready (zio=,
> buf=, vdb=0x0) at
> /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4642

If this trace is correct the error is from passing vdb == NULL to
dbuf_write_children_ready():

https://cgit.freebsd.org/src/tree/sys/contrib/openzfs/module/zfs/dbuf.c?h=stable/13#n4551

The function will unconditionally dereference (v)db, so passing NULL
will trigger a page fault.

I have no idea however how can you get to this state.  Might be worth
posting the trace to freebsd-fs@freebsd.org in order to get some
feedback from the ZFS people.  It's possible the issue is with
blkback, but I would benefit from some help about what's wrong with
the data I'm providing to d_strategy.

Please Cc me on the email if you send to freebsd-fs@.

Thanks, Roger.

From nobody Fri Apr 15 09:06:12 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6E1282DE559
	for ; Fri, 15 Apr 2022 09:06:20 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Kfr6M3gPrz4Qvm
	for ; Fri, 15 Apr 2022 09:06:19 +0000 (UTC)
	(envelope-from zedupsys@gmail.com)
Received: by mail-lj1-x22d.google.com with SMTP id f5so1331175ljp.8
        for ; Fri, 15 Apr 2022 02:06:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=reply-to:subject:to:cc:references:from:message-id:date:user-agent
         :mime-version:in-reply-to:content-language:content-transfer-encoding;
        bh=hx6EpiLqpJbRPslduuxGcR5A4SgCuJ/HlI2LtQpHq4Q=;
        b=Qa9u3QPinEHSPRgMeXA5/cxeWD3hpv2jm0k0NKAfmyAB+AT+js2+v5Kc7x8QUEFBuA
         VahCf7TP1pVps66O8ptAHsSz4616bgwyzXe7B0KVh7YylEo+o6o1X09xgVHKe7EiRFau
         A7VJIZT3Mjna5P/szSMlwo7rMgzSRbAITEiqNlVJhd11cyVQgNX+ctjaSDL3x8H/kIy6
         qpEid5CqkD/Ipfp7FywXBOmJmXwW1ln5/xGM7M/RKkBjUAT0ufeAagVCkBuyFrwp2l55
         SMRaEutZRZqnfO8AN1BrLfsHMo09bEBH2b3Cvh/52EJ/2AyjDGLIY4cQgTGd1IyIQxk5
         K6Ug==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:reply-to:subject:to:cc:references:from
         :message-id:date:user-agent:mime-version:in-reply-to
         :content-language:content-transfer-encoding;
        bh=hx6EpiLqpJbRPslduuxGcR5A4SgCuJ/HlI2LtQpHq4Q=;
        b=2+fOshbjC2Y4DVaPjhU0Tc92ic1/yflF4bRvQxgQjQ2kmHBC4JG+fY5jPGvCG3jJc9
         XgZv9R8roSN4Pgts+Ne7hZSoZDpzcnhBmEvBOKncKieaizhCcAOxytxBwofxWrUydJoZ
         9mvDhFtdCo+VBQhwFCcECcqdspt1oWiRetGryYQvyU7lamUjiFR579MkReHNvgL5IdEL
         5NO1ROeb4bLguuaRLqsua1amGthZxDuCU1z6xaKde0rOS7QUpQisyMoyUZipqaKJOEHY
         uJVxnbrV2hNRlCQ6e4lD9+bVXmssTX+1RNg/aNcc7rpEnvs8DHlF9XqFSNFhGzEfQznt
         FpMQ==
X-Gm-Message-State: AOAM531lk6bvc/Dasbb6jPk4a6ENwCYmFxH/pyjMjcOBWGtJrQGmAy9j
	Uztv+YfnKlw2oICgCG1cF5M=
X-Google-Smtp-Source: ABdhPJzK+cP9JBWjB3SaQNc7QH1zGx1okPLxem2iQiXhvr2Sx8GFaUe92cdOVmlCt+g9Xizi2DCgQA==
X-Received: by 2002:a2e:8748:0:b0:24d:a7a0:1ae3 with SMTP id q8-20020a2e8748000000b0024da7a01ae3mr747776ljj.8.1650013578045;
        Fri, 15 Apr 2022 02:06:18 -0700 (PDT)
Received: from [10.3.0.1] ([213.110.65.3])
        by smtp.googlemail.com with ESMTPSA id f16-20020ac24990000000b0046bb8e2ca13sm267957lfl.14.2022.04.15.02.06.17
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 15 Apr 2022 02:06:17 -0700 (PDT)
Reply-To: zedupsys@gmail.com
Subject: Re: ZFS + FreeBSD XEN dom0 panic
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
References: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com>
 
 <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com>
 
 <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com>
 
 <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
  
 
 
From: Ze Dupsys 
Message-ID: 
Date: Fri, 15 Apr 2022 12:06:12 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: lv
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 4Kfr6M3gPrz4Qvm
X-Spamd-Bar: -
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=Qa9u3QPi;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22d as permitted sender) smtp.mailfrom=zedupsys@gmail.com
X-Spamd-Result: default: False [-1.76 / 15.00];
	 HAS_REPLYTO(0.00)[zedupsys@gmail.com];
	 RCVD_VIA_SMTP_AUTH(0.00)[];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 RCVD_COUNT_THREE(0.00)[3];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-0.77)[-0.766];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-0.99)[-0.991];
	 MIME_GOOD(-0.10)[text/plain];
	 FREEMAIL_REPLYTO(0.00)[gmail.com];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 NEURAL_SPAM_MEDIUM(1.00)[0.999];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22d:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On 2022.04.14. 10:39, Roger Pau Monné wrote:
> ..
> Thanks. I will groom those patches in order to prepare them for
> commit. Regardless of whether there are other issues still lurking I
> think those changes are worth committing now.

Hi,

So the tests still are running with 3 patches, seems a lot better than 
without, soon i think i will stop them, since they have proved a point.

uptime
11:22AM  up 1 day, 19:18, 8 users, load averages: 2.17, 2.04, 2.12


There has been some problem though, because at this stage xl list has 
(null) VM.

xl list
Name                                        ID   Mem VCPUs      State 
Time(s)
Domain-0                                     0  1023     4     r----- 
211752.0
(null)                                     346     0     1     --ps-d 
    61.7
xen-vm2nonic-zvol-5                        557  1024     1     r----- 
    47.2
xen-vm1nonic-zvol                          558  1024     1     -b---- 
    42.0

I have no idea why it went in that state. I've been collecting vmstat -m 
since start and after filtering out, i think i got maybe useful hints.

Pictures in url: https://file.fm/u/k67uhj436#/

In general we can see, that xbbd with 3 patches seems not to leak 
memory. I did not see any component whose memory usage was just growing. 
Images show values of InUse of vmstat -m, y is bytes, x unix timestamp, 
but i don't know if maybe i should have looked into MemUse instead.

We see that solaris takes quiet a chunk, which is expected for ZFS i 
guess. But the interesting thing is a spike in newblk, looking deeper it 
can be seen that this spike happens at the same time when jsegdep spike 
is, and then there is at smaller scale, but still spike at the same time 
for jseg. At the same time when there is spike for newblk there is a 
little bit dip for solaris.

If there are some specific ones i should pay more attention, i could 
look at them.

I would like to speculate that "..pmap_growkernel.." panic happens at 
those times when there is spike high enough and that for this current 
case it was just a lucky coincidence that system had more mem and did 
not panic. Or maybe this is where this (null) VM appeared. 
Unfortunateley i did not log output of xl list with timestamps.

Currently xenstore-ls -fp, does not contain any row with 346, so i 
suppose that disks have been freed, i don't see any suspicious sysctl 
variables either, so i do not know what state this (null) VM is in and 
why it is not cleaning up. Is there some useful command in this case to 
collect more info about (null) VM?

Thanks.

From nobody Thu Jun  2 09:16:35 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A068B1B75462
	for ; Thu,  2 Jun 2022 09:16:50 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDL4J5QZNz3hbp
	for ; Thu,  2 Jun 2022 09:16:48 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 2529GbFW003677
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 2 Jun 2022 02:16:37 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 2529Gbgx002408;
	Thu, 2 Jun 2022 02:16:37 -0700 (PDT)
Message-Id: <202206020916.2529Gbgx002408@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 2 Jun 2022 02:16:35 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: FreeBSD-13.1 and Xen-4.15.0 crash formerly working NetBSD-99.77 domu's
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 02 Jun 2022 02:16:38 -0700 (PDT)
X-Rspamd-Queue-Id: 4LDL4J5QZNz3hbp
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-0.999];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  I just finished updating one of my FreeBSD Xen servers from 12.2-stable to
13.1-release.  After a very painful upgrade process, which I'll detail in another e-mail, I
discovered that my domu's running NetBSD-99.77, which had been running without any trouble
whatsoever in pv mode, are failing with an unhandled general protection fault.  Is this a known
issue, a configuration error on my part, a bug in the NetBSD kernel, or something completely
unknown?  The details from xen from a sample crash are listed below.  I'm happy to get more
details, if they would be helpful.  Just tell me what you would like to see and how I might go
about getting it.  Any light anyone can shed on this issue would be helpful.

-thanks
-Brian

Xen host details:

%uname -a
FreeBSD xen-lothlorien.nfbcal.org 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64

Xen dmesg output:

 BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
(XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
(XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
(XEN) System RAM: 32683MB (33467896kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1600 ... 3100 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN)   Compiled-in support: SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 3093.000 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Dom0 has maximum 440 PIRQs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 604kB init memory



(XEN) d2v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 2 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d2v0)
(XEN) rax: 0000000000070106   rbx: 000000000197d000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80718ec0
(XEN) rbp: ffffffff8197bf00   rsp: ffffffff8197be70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80718f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff81977000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 0000000421342000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80718ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff8197be70:
(XEN)    0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b
(XEN)    ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) d4v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 4 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d4v0)
(XEN) rax: 0000000000070106   rbx: 0000000000f7b000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80716ec0
(XEN) rbp: ffffffff80f79f00   rsp: ffffffff80f79e70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80716f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff80f75000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 000000047b140000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80716ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff80f79e70:
(XEN)    0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b
(XEN)    ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) d5v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 5 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d5v0)
(XEN) rax: 0000000000070106   rbx: 000000000197d000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80718ec0
(XEN) rbp: ffffffff8197bf00   rsp: ffffffff8197be70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80718f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff81977000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 0000000421f42000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80718ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff8197be70:
(XEN)    0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b
(XEN)    ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) d6v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 6 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d6v0)
(XEN) rax: 0000000000070106   rbx: 0000000000f7b000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80716ec0
(XEN) rbp: ffffffff80f79f00   rsp: ffffffff80f79e70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80716f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff80f75000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 0000000422940000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80716ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff80f79e70:
(XEN)    0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b
(XEN)    ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) d7v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 7 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d7v0)
(XEN) rax: 0000000000070106   rbx: 0000000000f7b000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80716ec0
(XEN) rbp: ffffffff80f79f00   rsp: ffffffff80f79e70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80716f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff80f75000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 0000000523140000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80716ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff80f79e70:
(XEN)    0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b
(XEN)    ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) d2v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 2 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e033:[]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d2v0)
(XEN) rax: 0000000000070106   rbx: 000000000197d000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80718ec0
(XEN) rbp: ffffffff8197bf00   rsp: ffffffff8197be70   r8:  000000001fc9cbf5
(XEN) r9:  0000000000000000   r10: 000000000000000c   r11: ffffffff80718f14
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: ffffffff81977000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000042660
(XEN) cr3: 0000000421342000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff80718ec0   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff8197be70:
(XEN)    0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a
(XEN)    000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b
(XEN)    ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000756e6547 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000

From nobody Thu Jun  2 09:42:30 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 47D231B491D8
	for ; Thu,  2 Jun 2022 09:42:37 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDLf40MGrz3kXg
	for ; Thu,  2 Jun 2022 09:42:35 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 2529gVNt005892
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 2 Jun 2022 02:42:32 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 2529gVZX005027;
	Thu, 2 Jun 2022 02:42:31 -0700 (PDT)
Message-Id: <202206020942.2529gVZX005027@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 2 Jun 2022 02:42:30 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: Upgrade from FreeBSD-12.2 to FreeBSD-13.1 with xen-4.14.0 is not straightforward.
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 02 Jun 2022 02:42:32 -0700 (PDT)
X-Rspamd-Queue-Id: 4LDLf40MGrz3kXg
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  In the process of updating one of my FreeBSD-xen servers from FreeBSD-12.2 to
FreeBSD-13.1, I ran into what I think is a documentation omition.
After I updated the operating system, but not the xen kernel or xen tools, I got the following
error messages shown below.  Essentially, the xen kernel claimed it couldn't find the FreeBSD kernel.

	After struggling for a long time to get the xen-4.15.0 tools built, I was able to boot the
system and bring up some of the domu's, see my other recent message on this topic.  However, I
don't see anywhere in the documentation or the wiki pages that xen-4.14.0 doesn't work with
FreeBSD-13.1.  Nor do I see any warnings about this issue.  It occurred to me that I'd probably
done this the hard way and the real problem was that when I updated FreeBSD, I failed to update
the gptboot file in the boot partition, which was originally installed with FreeBSD-12.1.  When
I updated the xen tools and kernel to 4.15.0, I also updated the gptboot file in the boot
partition of the boot drive.  my question is, is this a known issue?  That is, FreeBSD-13 needs
a new gptboot installed in order to boot xen at all?  If that's right, can we get this
documented somewhere, so folks dont run into this as they update from FreeBSD-12 to FreeBSD-13?

If my theory is correct, a simple update of the gptboot file on the boot partition would have
been sufficient to get xen 4.14.0 running and I wouldn't have had to spend the day rebuilding a
ton of stuff to get the xen tools built for 4.15.0.  Also, I wouldn't now be working on
resolving the general protection fault  error  I wrote about elsewhere.  

Is this correct or was I required to update to 4.15.0 when I went to FreeBSD-13.1?

-thanks
-Brian

/boot/config: -h


BIOS drive C: is disk0

BIOS drive D: is disk1

BIOS drive E: is disk2

BIOS drive F: is disk3

|/-\|/-\|BIOS 614kB/523264kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
/boot/xen data=0x26b9a8+0x146658 -
Loading kernel...
/boot/kernel/kernel
/boot/entropy size=0x1000
/etc/hostid size=0x25
/boot/kernel/ipmi.ko size 0x11968 at 0x21b1000
loading required module 'smbus'
/boot/kernel/smbus.ko size 0x3cb0 at 0x21c3000
/boot/kernel/tpm.ko |size 0xad70 at 0x21c7000
/boot/kernel/geom_mirror.ko /size 0x20c80 at 0x21d2000
/boot/firmware/intel-ucode.bin -\size=0x303800
 Xen 4.14.0
(XEN) Xen version 4.14.0 (buhrow@) (FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)) debug=n  Tue Nov 10 15:26:53 PST 2020
(XEN) Latest ChangeSet: 
(XEN) Bootloader: FreeBSD Loader
(XEN) Command line: dom0_mem=8192m dom0_max_vcpus=2 dom0=pvh pv-l1tf=off,domu=off console=com1,vga com1=9600,8n1
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 00000000000997ff] (usable)
(XEN)  [0000000000099800, 000000000009ffff] (reserved)
(XEN)  [00000000000e0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 000000001fffffff] (usable)
(XEN)  [0000000020000000, 00000000201fffff] (reserved)
(XEN)  [0000000020200000, 000000003fffffff] (usable)
(XEN)  [0000000040000000, 00000000401fffff] (reserved)
(XEN)  [0000000040200000, 00000000bc855fff] (usable)
(XEN)  [00000000bc856000, 00000000bc85efff] (ACPI data)
(XEN)  [00000000bc85f000, 00000000bc8a9fff] (ACPI NVS)
(XEN)  [00000000bc8aa000, 00000000bc8b1fff] (usable)
(XEN)  [00000000bc8b2000, 00000000bc9a4fff] (reserved)
(XEN)  [00000000bc9a5000, 00000000bc9a6fff] (usable)
(XEN)  [00000000bc9a7000, 00000000bcbc5fff] (reserved)
(XEN)  [00000000bcbc6000, 00000000bcbc6fff] (usable)
(XEN)  [00000000bcbc7000, 00000000bcbd6fff] (reserved)
(XEN)  [00000000bcbd7000, 00000000bcbf4fff] (ACPI NVS)
(XEN)  [00000000bcbf5000, 00000000bcc18fff] (reserved)
(XEN)  [00000000bcc19000, 00000000bcc5bfff] (ACPI NVS)
(XEN)  [00000000bcc5c000, 00000000bce7bfff] (reserved)
(XEN)  [00000000bce7c000, 00000000bcffffff] (usable)
(XEN)  [00000000bd800000, 00000000bf9fffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed3ffff] (reserved)
(XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000083e5fffff] (usable)
(XEN) New Xen image base address: 0xbc200000
(XEN) ACPI: RSDP 000F0450, 0024 (r2  INTEL)
(XEN) ACPI: XSDT BC856070, 0064 (r1 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: FACP BC85DBC0, 00F4 (r4 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: DSDT BC856168, 7A54 (r2 INTEL  DQ67SW         16 INTL 20051117)
(XEN) ACPI: FACS BCBDBF80, 0040
(XEN) ACPI: APIC BC85DCB8, 0072 (r3 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: TCPA BC85DD30, 0032 (r2 INTEL  DQ67SW          1 MSFT  1000013)
(XEN) ACPI: SSDT BC85DD68, 0102 (r1 INTEL  DQ67SW          1 MSFT  3000001)
(XEN) ACPI: MCFG BC85DE70, 003C (r1 INTEL  DQ67SW    1072009 MSFT       97)
(XEN) ACPI: HPET BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
(XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
(XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
(XEN) System RAM: 32683MB (33467896kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1600..3100 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN)   Compiled-in support: SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 3092.993 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Dom0 has maximum 440 PIRQs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 556kB init memory
Unable to find kernel
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

From nobody Thu Jun  2 10:37:47 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 34BE21B5B0C4
	for ; Thu,  2 Jun 2022 10:38:27 +0000 (UTC)
	(envelope-from prvs=1458da55d=roger.pau@citrix.com)
Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDMtV0RL4z3pJL
	for ; Thu,  2 Jun 2022 10:38:25 +0000 (UTC)
	(envelope-from prvs=1458da55d=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654166305;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=7BgHEbjj+yXUv6aW9bZyEf6+ykMSyhoJg6Bo5hqqmH4=;
  b=iX+IjO9yM4gZqdYOzbyGCUtCJhyMoLYINvqRpsxVnfT59nKKF3mlXGTQ
   9bxg05jrwffv1EEr3N94IUGZkUA8tfzeRHsWhDI2fC5TKGPOfYUVFulVd
   +Cxam3SGcl5r6eAToNVPqESp82zLr+A/JNqVo6Ll+UmnLvAtl70fUgqwV
   w=;
X-IronPort-RemoteIP: 104.47.58.102
X-IronPort-MID: 72053765
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:lWuU1aMMrU2C/8zvrR3+lsFynXyQoLVcMsEvi/4bfWQNrUoh32FRn
 TcZWW+APvyNYWGhfo8lOo7k8h4BscKGxtFlHQto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h
 ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFYMpBsJ00o5wbZn2tcw3bBVPivW0
 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb
 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Z4
 shHqLywW0QTOvfiqbg4aAd7Mj5YFPgTkFPHCSDXXc276WTjKiOp79AwSUY8MMsf5/p9BnxI+
 boAMjcRYxufhuWwhrWmVu1rgcdlJ87uVG8dkig4kXeFUrB7EdaaG/+iCdxwhV/cguhUGvnTf
 YwBYCdHZxXceRxffFwQDfrSmc/33CGiKGIG9Tp5o4Ib0Wf00RBN7ISwavWOX/+sGJtTkBmx8
 zeuE2PRR0ty2Mak4TCY8mirmcfJkynwUYZUE6e3ntZkmlCezHcfTQ0bU3O1u/S0gFWyHdVFJ
 CQpFjEGqKEz8AmnS4f7Vhjh+Hqc5EdEAZxXDvEw7xyLxuzM+QGFC2MYTzlHLts7qMsxQj9s3
 ViM9z/0OQFSXHSuYSr13t+pQfmaZUD58Udqifc4cDY4
IronPort-HdrOrdr: A9a23:eB8Ug6u0AdyS4V8kG5JvQTox7skC6oMji2hC6mlwRA09TyXGra
 2TdaUgvyMc1gx7ZJhBo7+90We7MBbhHLpOkPEs1NaZLXDbUQ6TQL2KgrGD/9SNIVycygcZ79
 YaT0EcMqyNMbEZt7ec3ODQKb9Jrri6GeKT9IHjJh9WPHxXgspbnmNE42igYy9LrF4sP+tCKH
 PQ3LswmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZXbxp/hZMZtU
 TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczKgBkF5MWUrHo6mt
 jFpBkte+x19nPqZ2mw5SDg3gHxuQxenkPK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL
 9L93jxjesaMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO21GkeKp
 ghMCjg3ocWTbvDBEqp/lWHgebcFEjbJy32DXTr4aeuontrdHMQ9Tpr+CVQpAZDyHsHceg72w
 31CNUWqFhwdL5mUUtcPpZ0fSLlMB27ffrzWFjiUWjPJeUgB0/njaLRzfEc2NyKEaZ4v6fa3q
 6xG29liQ==
X-IronPort-AV: E=Sophos;i="5.91,270,1647316800"; 
   d="scan'208";a="72053765"
Received: from mail-dm6nam10lp2102.outbound.protection.outlook.com (HELO NAM10-DM6-obe.outbound.protection.outlook.com) ([104.47.58.102])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 02 Jun 2022 06:38:00 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=L6XOn2eHWdNruxfxiZa+IQdKMvetxbFLcvv6rQ6ZjZzqBNC0wBxtfdXNJdoBhLXer9KDw2eyefBNPKvH7aG20oE+RfpHuSgTY7L6DmPfHuJWWBrDEokio0xqQh4DFOeUZc4WTC2HvVkBIWOTytgV+m18Wti8d0pKgD38NB1kK+ggAPNeZXygcln93wwtMOorvtdF2hjj+Kl4OMZG3tHVd6EDRz/P/PxqYl6FlMwzgMAiA5Q1lWBUexlwe+sdOBJ4RLaHMPPfCZGbwYDRMFDsjAUepJIHsgnCmDUYu1i6b05Gz4WY2cka34Gx50nfmhiACIge5ZZ/DlXlLAhiE/udeg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=zRdUi/O/YKuqh+OXdfywGukFsq7OPDokC/eEW52CwpQ=;
 b=a0QKCud1zwWW6Cx9yqmlVTHRhuEUCvOUm+1lBnTCU7w8sc58Jc5X2tu4AeXJi/rCssOVRaqQv73rCnTw6Gz601KZn3gFYYzo0Ur15OwBAl9tOkev5meIIhBr9Okx8YWQbSxHSjFKUCKxUMR87p3fQIET8TFFa8QGdONaeBbZgnUwXL+S7IHEn5Nl01ijWBW4ncdHcaIgRAxi09fqnba8BDhEx3vMCLzvuVITUsCIoJdxSKQmMR/KzH0krngTos58jozvMim2VanP1aPzXBKZePwLT/WXYBywwKwtyjUMI3GNppmjk/LtCKZV9r2ZkugFE072DHSxLkyRN5fvoK4YVQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=zRdUi/O/YKuqh+OXdfywGukFsq7OPDokC/eEW52CwpQ=;
 b=JYAYCNCgxz258Q5EeuelrK2WOVTxKqH+KF4JsuMADXoOvLuvGrZumZ6sXqbF5REQoeJ9Lsl/bmQaLQAuZvhKSDC88BOJLnj6698qu9Umuy9YWK+vbF4sSRJxgUXscd9m4wVff/RjXcqmumywnQh4aa5b2mHc41UgwkuZEAwU5/M=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by SJ0PR03MB5774.namprd03.prod.outlook.com (2603:10b6:a03:2da::18) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Thu, 2 Jun
 2022 10:37:51 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.015; Thu, 2 Jun 2022
 10:37:51 +0000
Date: Thu, 2 Jun 2022 12:37:47 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: FreeBSD-13.1 and Xen-4.15.0 crash formerly working NetBSD-99.77
 domu's
Message-ID: 
References: <202206020916.2529Gbgx002408@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206020916.2529Gbgx002408@nfbcal.org>
X-ClientProxiedBy: LO2P265CA0070.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:60::34) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 101a7c1e-76df-4cf4-e436-08da4483f18a
X-MS-TrafficTypeDiagnostic: SJ0PR03MB5774:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	F7uO2KD5jXglE4r5C0zXNG0Srl2Eyud5yvh4NEIsU9twLZ+crxGKCr5CZac29jP+TtVM7s6nGTrmXOuuShieywZFsRFHf2B4ME8AP2WmYiprc+13Ug5LD86NOcEFIl7bChIPu0RdBBxDg4DeRxikqC6PScq+yGdeE//6HpeTu1Db6VEe21JDg/sedyd8uCaeCPieStWyMASmnoBj/sQTR0lTl/LFIV0i9r2C6P+QYnAAnIGkc9UC0Y5SF4vyBgQytj+8ZEKHCUoOrL6tdAMoKrH0m4axiiUc+4vam8s+wM87M05ZktHUQfHPYEoaQ4qUDLdZhOZBVTv0pHbPZFGRgDbAfybv41ZzyzH59S3Zb0bt1W0BCgxzSHlU1yB9uBeLQ6WTvS/ycbzTVoeCJrBKfKY46LV8D67gLHJ2ZpS6jDewHMb0bh27xS7iJJFKjelxonSysuwQfriSxsGseKKCRE7J2h9KG838KI0dbdidJdshYEloF4QmtZRtCCdCoxTp978NUUW7kSQBTvdFOybeL/Ym4VFQk/B6Ic329nPTTQz0Dg449Tns8omw/tJBPSCfOrunYBMl7g3SWrU+EJmxaoDwivJYUJgya09Mt1zd+d808bdHpubESdKteeIUmgfcQBNgOQpoDHSNfAI3zK/pODk3UfdRx0cPh6CBC+//banOI3I17lRHyvhlMao9sbCOfnAMp3KOC4utlOWDBmvx3cZQqmTzexYSt+M1Dvgmeho=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(6512007)(26005)(9686003)(6666004)(186003)(4326008)(66476007)(66946007)(66556008)(6916009)(8676002)(85182001)(5660300002)(8936002)(508600001)(966005)(6486002)(38100700002)(86362001)(6506007)(83380400001)(33716001)(2906002)(316002)(82960400001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VkZ5R2hmUzZxanNkT0x0VUdlU2Rmb3dBallQb3ZmQUVxcWwxRlVCd2dIQzFL?=
 =?utf-8?B?NW1EcG9OWFhla0g2YTAyVTlKK0s3UDZtWnNzSDFzRmVtdE1hcGpLZTM4QUZP?=
 =?utf-8?B?aTA0Kzg5ZmZOV0RyRGZhR3F2Q3dKWTd6ZGFscFBlSVFObmwweU96VTlCZGlW?=
 =?utf-8?B?WDNiaXZ3ejg4by9SUnl3NnJDenZTMmdEM1FBL0liSWZsVlh5cWFOWVdHenl6?=
 =?utf-8?B?VURFV3R6R0NFRk5HSUZYbUFiSFhJTlRVU0UvQmNBOWozNy9nYUJGeSs2ZjdI?=
 =?utf-8?B?VVFhWEoyMmExN1hZbGpWYjd4dldzZXNjZEZjbWoyQ0g3clA4UThsUkdNV2hu?=
 =?utf-8?B?RVhLNTNsSW82dlJSMGZOVXlYeWVNTkNTWHZCTks0N3hDSzZUdmlTSXhnb1JL?=
 =?utf-8?B?MWpIdXhFS0tVczh4aEVpUDZadUZUZVJVQm16aFQyOCttdllPWXMxNzNMaGxO?=
 =?utf-8?B?dzczOHlJTlVlU1g4dTNSWFo2ZDlFbVdjTzUwTmFtanc0UW1aaWJYdy9WMVkx?=
 =?utf-8?B?ZzhveTJzYkFoY2dPV05CTmVMamoyVGlCTDBsbFVOeFJta3VZc2QxdXhnOG5u?=
 =?utf-8?B?NFZPdFFEMUZnYnl5dzFna25LaDh5R1E0VzlYNnFlSlNIdEgzMW0zdUZ5dFZQ?=
 =?utf-8?B?TkVYc2liVUp5N1JScXlyRnQ4TCt2MEJJVy90azhVQUtCVEx4WjVNNVVQWURp?=
 =?utf-8?B?czFRdlgwam5uTW9Sd0pZZTlZb1hnWGJ1SDZtSGFrZlQrK1ByVEJ6cXpUblFn?=
 =?utf-8?B?cHF1WWJGYlZwOWFRcGxVSTRNOVFrTVBVVVJ6RzB1c1ErRjJqMVpMempjV2Zn?=
 =?utf-8?B?SkF4cGozVE1hR0RoRTVZeFpzcjlXa2RJRGZDd1YxZWNLSDNiVVNQMXNmRith?=
 =?utf-8?B?UklLU2k5Z25Pb0krOU5WTjBNMTlvYnFYbFZvVXZ5MHZYczlPL0xIVm91VWla?=
 =?utf-8?B?N2pJdmVldE1qWXFOdk9NWXRWNHVnR04rQ2lYS2xUUm5zNmR0aGl4MEhPK25m?=
 =?utf-8?B?LzgvenZXS3pGVHk3UzlFN0w0ZnlBd01MK256QkpUa1JpWS9aK3JPaWI0eDNQ?=
 =?utf-8?B?d0NldUtNS0JMRisybkJPQjg2Z0pMaG5WYUtYcmZwTjRnL29VWkk3cThTNUQ1?=
 =?utf-8?B?aGowbVd6Y1pLUVg4MDdXMy9Xell5R0FWVGdnb1kyYldDL3pWRENxR2ErZ05m?=
 =?utf-8?B?Z1RQaGpQczV4cEVrNUxzakdKQVJ2Y0FEZk1wWEdLQXhPN0xTc3g0SmtZOExw?=
 =?utf-8?B?MGF4Qm5qNGp0YmxMQzUzdHIreHBjbzB5VU9FS1ZhOFBMV1pxaTlvOEoyTHN5?=
 =?utf-8?B?czVDWFlCZzNpNFBPSFphYW1NNkQvOThjTDV3cEJFZFEwRklRSlJteEVVckJH?=
 =?utf-8?B?dCt1YytMUDN4UXg1WEtDRU53bWlRSkNqVmV4MXhqOXU1T01VS2I4ekZESzFO?=
 =?utf-8?B?eXEyMWpuWmFLa3dwaFVIMThvZlZkdTJZc1Z1a3VaeHFISTdGaTFMbVhLRUI2?=
 =?utf-8?B?Zy9KWWhlSlBEdGNXN0RvckhLeFBPaEtQb2FKV2FnS0wyTDdwY2F5Q0R3R2JZ?=
 =?utf-8?B?bHd0a3Zyb2Q4RUZOVTUzWG5lblNZcVdydTFtMk55ZmM5eVhDVU5nVnA0RFlF?=
 =?utf-8?B?ZDA1blhTaGg4Sll1clppMHZ3eWZBa0s1b1VyZUNRZnBLWWI0a243Rm10QkRq?=
 =?utf-8?B?RURBQXp5ZmNZMVFRMW1ENzhrbDBlWmxpRjlSd0N1Z2JjZVR1YzgzeTRTby9H?=
 =?utf-8?B?R09ZWWhRbWhpMEVPTU5vMEVvOFZoS3ZUMmhVVnBwdFRHdU1hcUhYZ3MwbEtl?=
 =?utf-8?B?K3poQW05RGxjNVE4VHUrSFhJUVhmNWVweFJxZjhMMXB3QzZlUmZ6S2pOWWhs?=
 =?utf-8?B?UFdoQzhyNFo2aXVtVUZLK3hZL0Zuc0hiZ0pqWU9OTExOVFExNmZsRFNpMXVn?=
 =?utf-8?B?eUVwbzIwdXBrYTZ5VDlEbjNUS00zZk5qNWk5a1VHZmtGMHJ0UjN3OSt1M3Iv?=
 =?utf-8?B?YlNwVGM0L0lrRGVPd1dBT0E4MXZpcnQ5azROa0pnbHFUUVNobENJc1lqYy82?=
 =?utf-8?B?bXVuK0hUdjkxNml2dnNUYlIvS0RjUERDSmJWbHRWSXJndDc0NktYdnZlbjRQ?=
 =?utf-8?B?UVA4dzg4NmRxdGRmeXNlS3dOaEx1YTV3TmJlbVZJdjZ3bkFtdEhTN1BwWHV4?=
 =?utf-8?B?UUlyZ2NSdmg4akNQNjZ2M0dqY2o3MDBPTVZvWVdPZGxGLzRQeE41R1drbUlE?=
 =?utf-8?B?RzBCeFhVS3F5SkJ5YnEvdXI2REZXMCtSc2pHMmo1d1IyazVVa0hoMUFjYTlY?=
 =?utf-8?B?bXh0d0NTdzV6UVYycTg3VU40Vk5aVm4rV2FMTDdqQXRoY1VuOEpFNEhqTklB?=
 =?utf-8?Q?9/ERcsRzr57dtZ+Y=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 101a7c1e-76df-4cf4-e436-08da4483f18a
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2022 10:37:51.1412
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QctdjC0aGihgybDiRDUXpLEE+mosIskgRsRgMFoBEaZpOQo6G1mJ9F0UDRLZLhVJvhxp8sNWGWOz2tCxMJDKFg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB5774
X-Rspamd-Queue-Id: 4LDMtV0RL4z3pJL
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=iX+IjO9y;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=JYAYCNCg;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1458da55d=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=1458da55d=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1458da55d=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Jun 02, 2022 at 02:16:35AM -0700, Brian Buhrow wrote:
> 	hello.  I just finished updating one of my FreeBSD Xen servers from 12.2-stable to
> 13.1-release.  After a very painful upgrade process, which I'll detail in another e-mail, I
> discovered that my domu's running NetBSD-99.77, which had been running without any trouble
> whatsoever in pv mode, are failing with an unhandled general protection fault.  Is this a known
> issue, a configuration error on my part, a bug in the NetBSD kernel, or something completely
> unknown?  The details from xen from a sample crash are listed below.  I'm happy to get more
> details, if they would be helpful.  Just tell me what you would like to see and how I might go
> about getting it.  Any light anyone can shed on this issue would be helpful.
> 
> -thanks
> -Brian
> 
> Xen host details:
> 
> %uname -a
> FreeBSD xen-lothlorien.nfbcal.org 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64
> 
> Xen dmesg output:
> 
>  BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
> (XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
> (XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
> (XEN) System RAM: 32683MB (33467896kB)
> (XEN) Domain heap initialised
> (XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
> (XEN) Switched to APIC driver x2apic_cluster
> (XEN) CPU0: 1600 ... 3100 MHz
> (XEN) xstate: size: 0x340 and states: 0x7
> (XEN) Speculative mitigation facilities:
> (XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
> (XEN)   Compiled-in support: SHADOW_PAGING
> (XEN)   Xen settings: BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
> (XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
> (XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
> (XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
> (XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
> (XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
> (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
> (XEN) Initializing Credit2 scheduler
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Detected 3093.000 MHz processor.
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Posted Interrupt not enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using old ACK method
> (XEN) Allocated console ring of 16 KiB.
> (XEN) VMX: Supported advanced features:
> (XEN)  - APIC MMIO access virtualisation
> (XEN)  - APIC TPR shadow
> (XEN)  - Extended Page Tables (EPT)
> (XEN)  - Virtual-Processor Identifiers (VPID)
> (XEN)  - Virtual NMI
> (XEN)  - MSR direct-access bitmap
> (XEN)  - Unrestricted Guest
> (XEN) HVM: ASIDs enabled.
> (XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
> (XEN) HVM: VMX enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB
> (XEN) Brought up 4 CPUs
> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
> (XEN) Dom0 has maximum 440 PIRQs
> (XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
> (XEN) WARNING: PVH is an experimental mode with limited functionality
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Scrubbing Free RAM in background
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) Xen is relinquishing VGA console.
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> (XEN) Freed 604kB init memory
> 
> 
> 
> (XEN) d2v0 Unhandled general protection fault fault/trap [#13, ec=0000]
> (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167
> (XEN) Domain 2 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-4.15.0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    e033:[]
> (XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d2v0)
> (XEN) rax: 0000000000070106   rbx: 000000000197d000   rcx: 0000000000000277

This is NetBSD trying to write to CR_PAT MSR (see rcx being 0x277).
You will need to boot the guest with:

msr_related=1

In the xl.cfg config file.

This has been fixed in NetBSD HEAD, but the fix hasn't made it to any release IIRC:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/x86/x86/pmap.c

See Revision 1.410.

This is because of a change in Xen 4.15 that made the handling of MSRs
accesses more strict, as a hardening effort resulting from XSA-351:

https://xenbits.xen.org/xsa/advisory-351.html

Thanks, Roger.

From nobody Thu Jun  2 11:13:05 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 716F11B60CA3
	for ; Thu,  2 Jun 2022 11:13:20 +0000 (UTC)
	(envelope-from prvs=1458da55d=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDNfl2VWNz3sh2
	for ; Thu,  2 Jun 2022 11:13:19 +0000 (UTC)
	(envelope-from prvs=1458da55d=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654168399;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=melvJ+VhF7sqgH4b8L8Hcx62hbYKqpD3g/WROP9uHQE=;
  b=YTK+N3ZGtUtwwQrW1UO2qCj8dpLpPOk7eQJSZRCwUAzRjkcPOEZvG1/X
   Hr8D0mW+9+mS14pRo3lIlk7Fk/5PPWxmbDZheIyM96kSIvCzMt7jcdVWN
   PAMDRFXNYMOPdtbFpU6YcgwHgWFHPUnCWsBpLGL8qyghPnPn8iZf1tV7s
   s=;
X-IronPort-RemoteIP: 104.47.70.101
X-IronPort-MID: 73105980
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:L2GxpqBQpR11xhVW/3Liw5YqxClBgxIJ4kV8jS/XYbTApGx0hGYCz
 2EXUT+Daa2JYDf3KdtxbYzl9RkEv5TTx9AxQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA
 xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgHGeIdA970Ug5w7Bh2NYx6TSEK1jlV
 e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH
 I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPh+k
 8tNiYORRjsnM7bynsg4fjloASVhaPguFL/veRBTsOS15mifKz7G5aorC0s7e4oF5uxwHGdCs
 +QCLywAZQyCgOTwx6+nTu5rhYIoK8yD0IE34yk8i22GS6l+B8mbGs0m5vcBtNs0rtpJEvvEI
 dIQdBJkbQjaYg0JMVASYH47tLjz3yGiIm0AwL6TjaQR2E7fkQFv7IfwKsSORISOT/RsuG/N8
 woq+Ey8WHn2Lue3wyeF6HWzrunBlyjyXMQVDrLQ3vBwjVCZ3WBWGBAQfVWhrPSzkUL4XMhQQ
 3H44QIrpKk2sUatEd/0WkTgpGbe50ZFHd1ND+c99QeBjLLO5BqUDXQFSTgHb8E6sMgxRnoh0
 Vrhc87VOAGDeYa9ERq1nop4ZxvrUcTJBQfuvRM5cDY=
IronPort-HdrOrdr: A9a23:ZBh0rKFXo47fzoMwpLqFdJHXdLJyesId70hD6qkvc3Fom52j/f
 xGws5x6faVslkssb8b6LW90Y27MAvhHP9OkPAs1NKZMDUO11HJEGgP1/qA/9SkIVyEygc/79
 YdT0EdMqyWMbESt6+TjmiF+pQbsb+6GciT9JrjJhxWPGVXgs9bnmVE4lHxKDwNeOAKP+tOKL
 Osou584xawc3Ueacq2QlEDQuj4vtXO0LbrewQPCRIL4BSHyWrA0s+zLzGomjMlFx9fy7Yr9m
 bI1yT/+6WYqvm+jjvRzXXa4Zh6kMbojvFDGMuPoM4ILSiEsHfgWK1RH5m5+BwlquCm71gn1P
 HKvhcbJsx2r0jce2mkyCGdrjXI4XIL0TvP2FWYiXzsrYjSXzQhEfdMgopfb1/w91cglMsU6t
 MG40up875sST/QliX04NbFEztwkFCvnHYkmekPy1RCTIolbqNLp4B3xjIZLH45JlO11GkbKp
 guMCmFj8wmMW9yLkqp9FWH+ebcEUjaRXy9Mws/Us/86UkloJk29Tpb+CUlpAZ/yHsMceg62w
 36CNUYqFhvdL5jUUsvPpZ3fSOIYla9MS7kASa1HWnNMp0hFjbkl6PXiY9Fl91CPqZ4h6cPpA
 ==
X-IronPort-AV: E=Sophos;i="5.91,271,1647316800"; 
   d="scan'208";a="73105980"
Received: from mail-bn7nam10lp2101.outbound.protection.outlook.com (HELO NAM10-BN7-obe.outbound.protection.outlook.com) ([104.47.70.101])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 02 Jun 2022 07:13:11 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=lUTZOP2jhGap0FZ6PNhV5up4wTrTS1QyxTLJkx9iB8sh0LzO6jEopJ4WxPWOTSKiZTV5zs2/RU6R+C18iU+WUN2iJb8QNcFF69AurEzPU58vrS4MnjYMjYoQoroKQtgP/20Dn3LTwn9BjVRD4qSKpxsl3Ho/vnbDkpD+XLxghf93fn0I+nMmEE/nAYQ3OBjPji+8Je7lEaplaQSzY5nqy68my2j4+nubh29SF9i4KWelZVWnNqqN1qSpi+Gid078lVTIuP7uiCFJ3NBR0tNE6MMAjcS2x68zk84G3hYkmYr/HOvG/B6IPa/J3JV6XbpQIiP4nLCe0jgEcLCsgN2opQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=wFL60IpbqAMoqUXP5lTf7l6A+IEglLpXgpH+IMqGrY8=;
 b=YpEMBcbLfZvbS/b9Aw1BuewEOQMyodLp8J6FXwEtHq51GMxsmvfWFBx1LECQ+74F8PVScwl2jMd8KFk5jYecBpmwtuYJQIe5BTKuLl5x1GBKphqk6yTB19znQ6BCuBYu93o19yfqILe6xZ8Pp+iQbQXYam8HE/5sJqePjGcwxj7bSofxE2gkuCMpK6whmiSc+eth0NK/XLIt82t/vvE7VU8CflssjeLSMsGt/8tYWCgL5lFoduuPuOrnfDb0X1cjU6j3Qrxp+BNnatcDXKZE32z/8N8XinjBy/7Gj0Vvw2KF64gCZx78LLRcsEV62iFcJMFDC2roPqqfFlAaYytk1g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=wFL60IpbqAMoqUXP5lTf7l6A+IEglLpXgpH+IMqGrY8=;
 b=o9C5GxG0IW5hkUflIPS7fuu/4+5YEIRxFzpo4rrrsayCt2zS0STSifwqr3QSNeLMEsWVgmM5JfAdWij4OP/2MMIvHVKIlKEGNn4KiYq83e4ehJFkAA/zDDE2FErm3fo/z6PtI4BrF3tYzkecogrIdpKgb6C5K31EM3kY07fGHe8=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by SJ0PR03MB5661.namprd03.prod.outlook.com (2603:10b6:a03:288::5) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Thu, 2 Jun
 2022 11:13:10 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.015; Thu, 2 Jun 2022
 11:13:09 +0000
Date: Thu, 2 Jun 2022 13:13:05 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Upgrade from FreeBSD-12.2 to FreeBSD-13.1 with xen-4.14.0 is not
 straightforward.
Message-ID: 
References: <202206020942.2529gVZX005027@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206020942.2529gVZX005027@nfbcal.org>
X-ClientProxiedBy: LO2P265CA0110.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:c::26) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 14a78859-326d-4764-2b51-08da4488e05e
X-MS-TrafficTypeDiagnostic: SJ0PR03MB5661:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	lhcs65NIMmtEwcLUQZKKiVnvR28vlmc7D/W/YuJrC/QQADxxQz5ROOdD4iQ+5P+hcms4xcowy9jy7X3lsvT+q7qS4aK3Z0vcQnw5cKFH2XtJM1Hup9nUdBh7KqiStvDJGdbOY04QWSotlM8qFawFb8a0pBLGSxFs7hoH+SGIDf7tXzejj6qyF/2inyoepaeDpJhQA5+0tFg2OwgBV5yFAkn4mYnFzWWBw9nQCAeoNw2SyzdEclB37MUTD1KKGFdbdU1uUnXIABIapvo5qxE5s11CeviTEQHOvdAxUJ49uin8S8c0mXjnCt1LQjQHxk8mVVPvD5wrpD4f1BAcelu6jN9CKqqZaXkHboyxnGbwObd7qgkHReIuEA3cFt3Avlrm7uEqV6I42YCVUA8aZ0xlM8GKbwHN7BUiGUNIeQvSPJ/EqFCHHYZOtXEPtobx/ISwq7YVS69ZHVdI7tr/CJTnAHvcz6a9KFVK2FkzJEmZIst5OwEiIJnilRWpC1ERm7YlBTCGPm3XO+wEHlM+tWpXdL/pakmt2lCZG5f6BLKBz0UWA0J3if/QLsl8BJWfvg6lpneEm1ey9jvDVHCcRzoienPI14w8foVhzddTms6LpI86dxycgn3tYAYuAmhOUsLUF0FyQpwarWNDiy3wV+FgyJuXrw1An4Kz4FBMqwvCXM6BaOIdPoQG7ZtiWHC8A4pOJOHIe7fchNf6bIwnbQ6yiySZxp+qVkFHtPdO79maTwY=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(9686003)(6666004)(6512007)(2906002)(26005)(6506007)(186003)(85182001)(83380400001)(8936002)(33716001)(966005)(86362001)(6486002)(82960400001)(8676002)(4326008)(66556008)(66476007)(66946007)(6916009)(38100700002)(508600001)(316002)(5660300002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SEtCZVllUmZySnhPTUM1Q3FiQkpqRUFxNkpLckNxcEkzL1pNcVo0Q1BFd0dZ?=
 =?utf-8?B?MGVHN1U5UHIyNDNkNmZ6bTQzeno5TEVxQ2VWcjM2MXNaUGpjQ21TUjdxeTFz?=
 =?utf-8?B?YmxHb29lOTdTTjhCMDYyUldnMHlacHFtVkM5TitPRHhNeTQ5bjJrTFg0MW9y?=
 =?utf-8?B?QzRObU1ESHpQK3haWnFLekxnVE5IWWFMTmFLRTlqeGxOUndVMzFCUU96SnZO?=
 =?utf-8?B?QzdvalpzWkFFTnQyRmNFTWYvRUNVQndybUFkTS9rUmoyekc1MXpUUzlrcTR0?=
 =?utf-8?B?Q242MVRVYkxybktwNEtoTG81THZYdStRNGNlR2t0TTlOVnVCKzd6SHNhTFly?=
 =?utf-8?B?ZlRkK3QrQkxuNkt1OHVXUzN0THVMZkYyY0tvVnIzazZTditzaCtlcnRKZ2p4?=
 =?utf-8?B?WCtQV2c5cG1vdHowKzBCOHlTTEoyQmc0QVVld3NWQkF2MlliQTFlK0xTdE1j?=
 =?utf-8?B?K1h1Ni9VRTRoU0Jsem9uVERMRTNGTE5pM2tsTHFJR284M0tHNE5EU2NSSW9M?=
 =?utf-8?B?bnFEbXBNK3Z2WEM1YXYvSlpBNDRFaGZTbzhZVzRaQlZsMnlxQkdHbEI4bjJa?=
 =?utf-8?B?T3VQazA0MXhoUUIwNmlCSVBsU28vbkVlcHA1K2kwKy9IRzJwYmdkR256T0Ez?=
 =?utf-8?B?Ulgxd083a2ViOUs2ZWEvSUU5SmhrZDhZOEYzeWtLL0M2QW9NM0NSWERCdjlZ?=
 =?utf-8?B?c2NUWDFvWHB1OGNEZDY2b3RPK2JUdXNPdlZ1ZkllSVFUQTVJMzZEMnVyeUMz?=
 =?utf-8?B?UFphZTJMQlo2dXplMktzSDg3QUlvTGpDVUZlNlpxNFY1UWhZQ2ZJMnZzQWEx?=
 =?utf-8?B?Mm5veDdCdnBQZzBWQnVNYUVHeUpOZ1ZHWXRjNXdqNmdNN0s2QlQrVVZYQVIy?=
 =?utf-8?B?dXNLeWkzWERCTmt0NlRPcjRjeTZWUTVZWk00WERPYWlTWDNTNEVGRWRGN2dn?=
 =?utf-8?B?THc4N1JCZEFKM3pKM0JCYk9MQXp0dzRIUG80TFpFbzVWSzl6cDV2a3dRUXRM?=
 =?utf-8?B?R09nZCtPK095VE8yMTVXbk94Zjd0TnhnR3BHMGI1ZmFNRHN0ekJwUU5PbEU5?=
 =?utf-8?B?SzdUd3V0aWFjd0JZRXNlYlVPUEVSY3VId2pCdWkybW5POXhrNTl4b3J1TkRE?=
 =?utf-8?B?MEYxWmFLV0dKRytEOTM1dXNwZmUzYkdJNnhMYVd6Vy83MS8yTUxJaG9kaGpI?=
 =?utf-8?B?cGM3K3p5QVBVSWFHTVk2eGZhQnpHQ05RNjRjWUpHbHZsT1NlWlNPcHEzSzMy?=
 =?utf-8?B?NmFEMUJqamg0b0JkME9jZHRXdGJIWlEzYlBxWjRIZWVHWmhNa0tkdk5qRVhP?=
 =?utf-8?B?Z2RHYmxGV1QwSUtYanBqMmxCQnJiaUQrdk9uR1hlUDB6K21SankwcFZHdWVD?=
 =?utf-8?B?cXNYb2h5UnhNL2l3aUgvTWFKVTFQbFYyU1JyeE9TTm8razFud1h1eVN2MGty?=
 =?utf-8?B?a3ZnazRPQUROOHZweWFGaklJVXpsOENiYWZuNTBibjFXQ0U0b25UeURMWWln?=
 =?utf-8?B?ZUE5dTVqS3ZZOUtUN2hTdEE0VEJJaUNOb2lzWGpaRXlPT0NrZUYxNGx3b3Zt?=
 =?utf-8?B?UXFkWnUwM3ppbExnakxuT2QzTnhsYTBZRTN3djRVMU0rTGkwczZRT0JCTUND?=
 =?utf-8?B?MlRLSm04WkdLYzc3OFh4VWY5K2c4NWlxdWU1dnVXQTcvd204TTJkYTJodmc5?=
 =?utf-8?B?d0lyaXYySjVJMExjS2o4eDNWKy8vU3lkdHVsZzZ2cU1hK1NvSzArNDdBd2hx?=
 =?utf-8?B?WkM4UTBMeDVQVUdVbkNXVDZoOE04U0JhYzBBQUV6VGs0dW9Ga3VRZzdzNmRZ?=
 =?utf-8?B?bitnMGtMbmZDTjVaTFJsM2c0MG9FVWdXc05rcU51Z2ZwMWxBS3ZacmVMdXk0?=
 =?utf-8?B?TStzSlZ0OW0yM3E0dUJCN2NlT2hoQ1NhNmlYb0Q3UkFNakVJMEcrRjNPNXA0?=
 =?utf-8?B?am1aN3JyTEllaDdEWFY5OEtWQVYwMGxsY01Jd21ybGhqT05JS29OWmpURDZH?=
 =?utf-8?B?ZnkraWcza0VtdCtNblRnNmFpVmsvcU80VkpiS1NxMXV0TmFOdS9uL3RBTVlG?=
 =?utf-8?B?cEFkbDlXbEFSWDRmZk03RXRwcThNeENrQk11cWx4NWF4YXh3aDFwa3RaZ3FH?=
 =?utf-8?B?ZkhlRUV5MFV6QW1Qcnp6Yk0xbU1nSWliR0haVFpsd1RvWDh4eFEvcE9RMFp0?=
 =?utf-8?B?TUNGWkJ2Sm1YNllhd09sdmE5TG91cG82TkovMDFRYzdSdHdKcGNHVnp4MDNJ?=
 =?utf-8?B?SDZFSzllYUhrUW04VWN1eE0zeFRWa0V5dXV3eWdlQ0hpUkgxZHk0L2dhNVZQ?=
 =?utf-8?B?alhucWkxYVIxaTVvUGNDMXNwSXFadmJnQ2RIR2RuaGl3dDFKU2VFSVBObE1u?=
 =?utf-8?Q?fY6ceXJP8NBwtHfw=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 14a78859-326d-4764-2b51-08da4488e05e
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jun 2022 11:13:09.8573
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 29WATZc6qhYjubaz4C/fDYfH48OFmq3GnkLMfdh3oX4oC5znEghhsJjk2i+Sr9fzvAYQigk00dY29m4yamVuNw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB5661
X-Rspamd-Queue-Id: 4LDNfl2VWNz3sh2
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=YTK+N3ZG;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=o9C5GxG0;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1458da55d=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=1458da55d=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1458da55d=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Jun 02, 2022 at 02:42:30AM -0700, Brian Buhrow wrote:
> 	hello.  In the process of updating one of my FreeBSD-xen servers from FreeBSD-12.2 to
> FreeBSD-13.1, I ran into what I think is a documentation omition.
> After I updated the operating system, but not the xen kernel or xen tools, I got the following
> error messages shown below.  Essentially, the xen kernel claimed it couldn't find the FreeBSD kernel.
> 
> 	After struggling for a long time to get the xen-4.15.0 tools built, I was able to boot the
> system and bring up some of the domu's, see my other recent message on this topic.  However, I
> don't see anywhere in the documentation or the wiki pages that xen-4.14.0 doesn't work with
> FreeBSD-13.1.  Nor do I see any warnings about this issue.  It occurred to me that I'd probably
> done this the hard way and the real problem was that when I updated FreeBSD, I failed to update
> the gptboot file in the boot partition, which was originally installed with FreeBSD-12.1.  When
> I updated the xen tools and kernel to 4.15.0, I also updated the gptboot file in the boot
> partition of the boot drive.  my question is, is this a known issue?  That is, FreeBSD-13 needs
> a new gptboot installed in order to boot xen at all?  If that's right, can we get this
> documented somewhere, so folks dont run into this as they update from FreeBSD-12 to FreeBSD-13?

So yes, as you discovered the hard way, FreeBSD 13 requires a matching
loader in order to work correctly with Xen.  This is due to:

https://cgit.freebsd.org/src/commit/sys/x86/xen/pv.c?id=b6d85a5f51e4147452b35d76478fb9e191b7734b

As a user, where d oyou think it would be helpful for you to find this
information, in the Xen handbook chapter maybe?

https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-xen

Thanks, Roger.

From nobody Thu Jun  2 18:08:00 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BB6201963F27
	for ; Thu,  2 Jun 2022 18:08:26 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDYsj5s1Qz3j2l
	for ; Thu,  2 Jun 2022 18:08:25 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 252I81FN019548
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 2 Jun 2022 11:08:01 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 252I80jw002836;
	Thu, 2 Jun 2022 11:08:00 -0700 (PDT)
Message-Id: <202206021808.252I80jw002836@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 2 Jun 2022 11:08:00 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: FreeBSD-13.1 and Xen-4.15.0 crash formerly working NetBSD-99.77 domu's
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 02 Jun 2022 11:08:01 -0700 (PDT)
X-Rspamd-Queue-Id: 4LDYsj5s1Qz3j2l
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_DN_SOME(0.00)[];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

hello Roger.  Thanks for the quick reply and pointer.  Just for clarification on this list, in case 
someone reads this later, the actual variable to set in the xl.cfg file, also can be set in the
specific guest configuration files, is:
msr_relaxed = 1

This is documented on the xl.cfg man page in the xen package.
-thanks
-Brian


From nobody Thu Jun  2 18:35:22 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C6A1E19E919F
	for ; Thu,  2 Jun 2022 18:35:31 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LDZSy4wTrz3ljP
	for ; Thu,  2 Jun 2022 18:35:30 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 252IZMvg006159
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 2 Jun 2022 11:35:23 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 252IZMm9009017;
	Thu, 2 Jun 2022 11:35:22 -0700 (PDT)
Message-Id: <202206021835.252IZMm9009017@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 2 Jun 2022 11:35:22 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Upgrade from FreeBSD-12.2 to FreeBSD-13.1 with xen-4.14.0 is not straightforward.
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 02 Jun 2022 11:35:23 -0700 (PDT)
X-Rspamd-Queue-Id: 4LDZSy4wTrz3ljP
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.29 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_DN_SOME(0.00)[];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-0.99)[-0.990];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  Yes, the handbook URL you sited,
https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-xen
would be a good place to document this requirement.  Also, I'm pretty sure you can now boot xen
and FreeBSD with UEFI support as well.  Is this correct?  If it is, then this should be
documented as well.

-thanks
-Brian


From nobody Fri Jun  3 21:51:50 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 799591BE4FF8
	for ; Fri,  3 Jun 2022 21:52:03 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LFGnG3VMmz511q
	for ; Fri,  3 Jun 2022 21:52:02 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 253LppST028555
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Fri, 3 Jun 2022 14:51:51 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 253Lpocq022622;
	Fri, 3 Jun 2022 14:51:50 -0700 (PDT)
Message-Id: <202206032151.253Lpocq022622@nfbcal.org>
From: Brian Buhrow 
Date: Fri, 3 Jun 2022 14:51:50 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: PV network interfaces on NetBSD-99.77 domu guests fail to operate
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Fri, 03 Jun 2022 14:51:51 -0700 (PDT)
X-Rspamd-Queue-Id: 4LFGnG3VMmz511q
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.29 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-0.99)[-0.993];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello. As part of my update to xen-4.16 and FreeBSD-13.1 as dom0, I'm trying to get some
guests running NetBSD-99.77 working.  These guests have been working fine with xen-4.14 and
FreeBSD-12.2, with the patch shown below applied.

What happens when these guests boot is that the network interface attaches, but then netback.c
decides something is wrong and it shuts the xnb interface down for that domain, detaching it
from the domu's perspective.  I can reproduce this behavior over and over again by using the xl
network-detach and network-attach commands to reinitialize the xnb and xennet interfaces on
host and guest respectively to show the behavior.
	I downloaded the the latest FreeBSD-13-stable sources to see if the patch shown below was
applied.  It is.  I'm not sure if it made it into the 13.1 binary release, but it seems like it
should be there.  
As a data point, FreeBSD guests seem to work fine, as do NetBSD-5.2 guests.
Is this a casewhere the patch is in the current code, but not the binary release or is this a
diferent problem with a similar symptom?

-thanks
-Brian

(Output on the dom0 side when I attach a network interface to the relevant guest.)
root@xen-lothlorien:/var/xen/nfbcal/loth_9 # xl network-attach 6 'mac=00:4e:46:42:42:cc'
ifconfig: interface xenbr0 does not exist
ifconfig: 'mtu' requires argument
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
/usr/local/etc/xen/scripts/vif-bridge
/local/domain/0/backend/vif/6/4 [5085] exited with error status 1
libxl: error: libxl_device.c:1412:device_addrm_aocomplete: Domain 6:Unable to add vif with id 4
libxl_device_nic_add failed.

(Output from dom0/FreeBSD-13.1  kernel when above command is issued.)
Jun  3 14:24:48 xen-lothlorien kernel: xnb(xnb_probe:1123): Claiming device 5, xnb
Jun  3 14:24:48 xen-lothlorien kernel: xnb(xnb_attach:1267): Attaching to backend/vif/6/4
Jun  3 14:24:48 xen-lothlorien kernel: xnb(xnb_frontend_changed:1391): frontend_state=Initialising, xnb_state=InitWait
Jun  3 14:24:48 xen-lothlorien kernel: xnb(xnb_frontend_changed:1391): frontend_state=Initialised, xnb_state=InitWait
Jun  3 14:24:48 xen-lothlorien kernel: xnb5: Error 2 Unable to retrieve ring information from frontend /local/domain/6/device/vif/4.  Unable to connect.
Jun  3 14:24:48 xen-lothlorien kernel: xnb5: Fatal error. Transitioning to Closing State
Jun  3 14:24:48 xen-lothlorien kernel: xnb(xnb_frontend_changed:1391): frontend_state=Closed, xnb_state=Closing

(Output from domu host when network interface is attached.)

Script started on Fri Jun  3 14:22:00 2022
[ 6810.3600640] xenbus0: can't get state for device/suspend/event-channel (2)
[ 6810.3600640] xennet0 at xenbus0 id 4: Xen Virtual Network Interface
[ 6810.3700593] xennet0: backend features 0x6
[ 6810.3700593] xennet0: MAC address 00:4e:46:42:42:cc
[ 6810.3800626] xenbus0: can't get state for device/suspend/event-channel (2)
[ 6810.3900596] xenbus0: can't get state for device/suspend/event-channel (2)
[ 6810.3900596] xennet0: detached


; Sat,  4 Jun 2022 14:42:40 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LFjCL5gqlz3JSF
	for ; Sat,  4 Jun 2022 14:42:38 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 254EgWCZ019800
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Sat, 4 Jun 2022 07:42:33 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 254EgWR8005264;
	Sat, 4 Jun 2022 07:42:32 -0700 (PDT)
Message-Id: <202206041442.254EgWR8005264@nfbcal.org>
From: Brian Buhrow 
Date: Sat, 4 Jun 2022 07:42:30 -0700
In-Reply-To: <202206032151.253Lpocq022622@nfbcal.org>
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: Re: PV network interfaces on NetBSD-99.77 domu guests fail to operate
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Sat, 04 Jun 2022 07:42:33 -0700 (PDT)
X-Rspamd-Queue-Id: 4LFjCL5gqlz3JSF
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  Following up on this issue.  It seems that I was mistaken.  The stable/13 branch
of the FreeBSD source tree has the patch installed.  However, the releng/13.1 branch does not.
So, it's back to source kernels for me, just as it's been for 12.x.
Sorry for the noise.
-thanks
-Brian


From nobody Sat Jun  4 20:03:55 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 127C21BD307F
	for ; Sat,  4 Jun 2022 20:04:16 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LFrLN27Z3z4YwQ
	for ; Sat,  4 Jun 2022 20:04:11 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 254K3xCr010478
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Sat, 4 Jun 2022 13:03:59 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 254K3uV1016135;
	Sat, 4 Jun 2022 13:03:56 -0700 (PDT)
Message-Id: <202206042003.254K3uV1016135@nfbcal.org>
From: Brian Buhrow 
Date: Sat, 4 Jun 2022 13:03:55 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: Stable/13 doesn't boot with xen
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Sat, 04 Jun 2022 13:03:59 -0700 (PDT)
X-Rspamd-Queue-Id: 4LFrLN27Z3z4YwQ
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	Hello.  In the process of getting the netback.c patch into a working kernel on the system
I'm updating, I discovered that Stable/13 doesn't boot successfully when xen-4.16 is running.
Specifically, the system hangs while configuring the PCI busses.  I suspect it has something to
do with the failure of the hpet timers to attach.
I do not know if it works with older versions of the xen hypervisor, but it boots and runs fine
without xen running.  Below are two boot logs, the first with xen installed and running, the
one that fails, and the successfful boot,  with FreeBSD running on bare metal.
Hopefully this is a known issue and there is an easy fix.

-thanks
-Brian

Version info:

FreeBSD xen-lothlorien.nfbcal.org 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-984a45d77: Sat Jun  4 07:38:09 PDT 2022     buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/13/amd64.amd64/sys/GENERIC amd64



BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3

FreeBSD/x86 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
Autoboot in 1 seconds. [Space] to pause 
Loading Xen kernel...
/boot/xen data=0x2659c8+0x13e638 -
Loading kernel...
/boot/kernel/kernel 
Loading configured modules...
/boot/kernel/ipmi.ko size 0x11950 at 0x21a6000
loading required module 'smbus'
/boot/kernel/smbus.ko size 0x3cb0 at 0x21b8000
/boot/kernel/geom_mirror.ko size 0x20c80 at 0x21bc000
/boot/kernel/tpm.ko size 0xad70 at 0x21dd000
/boot/firmware/intel-ucode.bin -size=0x303800
/etc/hostid size=0x25
/boot/entropy size=0x1000
 Xen 4.16.0
(XEN) Xen version 4.16.0 (buhrow@) (FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)) debug=n Thu Jun  2 11:30:17 PDT 2022
(XEN) Latest ChangeSet: 
(XEN) Bootloader: FreeBSD Loader
(XEN) Command line: dom0_mem=8192m dom0_max_vcpus=2 dom0=pvh pv-l1tf=off,domu=off console=com1,vga com1=9600,8n1
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 00000000000997ff] (usable)
(XEN)  [0000000000099800, 000000000009ffff] (reserved)
(XEN)  [00000000000e0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 000000001fffffff] (usable)
(XEN)  [0000000020000000, 00000000201fffff] (reserved)
(XEN)  [0000000020200000, 000000003fffffff] (usable)
(XEN)  [0000000040000000, 00000000401fffff] (reserved)
(XEN)  [0000000040200000, 00000000bc855fff] (usable)
(XEN)  [00000000bc856000, 00000000bc85efff] (ACPI data)
(XEN)  [00000000bc85f000, 00000000bc8a9fff] (ACPI NVS)
(XEN)  [00000000bc8aa000, 00000000bc8b1fff] (usable)
(XEN)  [00000000bc8b2000, 00000000bc9a4fff] (reserved)
(XEN)  [00000000bc9a5000, 00000000bc9a6fff] (usable)
(XEN)  [00000000bc9a7000, 00000000bcbc5fff] (reserved)
(XEN)  [00000000bcbc6000, 00000000bcbc6fff] (usable)
(XEN)  [00000000bcbc7000, 00000000bcbd6fff] (reserved)
(XEN)  [00000000bcbd7000, 00000000bcbf4fff] (ACPI NVS)
(XEN)  [00000000bcbf5000, 00000000bcc18fff] (reserved)
(XEN)  [00000000bcc19000, 00000000bcc5bfff] (ACPI NVS)
(XEN)  [00000000bcc5c000, 00000000bce7bfff] (reserved)
(XEN)  [00000000bce7c000, 00000000bcffffff] (usable)
(XEN)  [00000000bd800000, 00000000bf9fffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed3ffff] (reserved)
(XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000083e5fffff] (usable)
(XEN) New Xen image base address: 0xbc200000
(XEN) ACPI: RSDP 000F0450, 0024 (r2  INTEL)
(XEN) ACPI: XSDT BC856070, 0064 (r1 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: FACP BC85DBC0, 00F4 (r4 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: DSDT BC856168, 7A54 (r2 INTEL  DQ67SW         16 INTL 20051117)
(XEN) ACPI: FACS BCBDBF80, 0040
(XEN) ACPI: APIC BC85DCB8, 0072 (r3 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: TCPA BC85DD30, 0032 (r2 INTEL  DQ67SW          1 MSFT  1000013)
(XEN) ACPI: SSDT BC85DD68, 0102 (r1 INTEL  DQ67SW          1 MSFT  3000001)
(XEN) ACPI: MCFG BC85DE70, 003C (r1 INTEL  DQ67SW    1072009 MSFT       97)
(XEN) ACPI: HPET BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
(XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
(XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
(XEN) System RAM: 32683MB (33467896kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1600 ... 3100 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- STIBP- SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 3092.988 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Dom0 has maximum 440 PIRQs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 600kB init memory
---<>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-STABLE #0 stable/13-984a45d77: Sat Jun  4 07:38:09 PDT 2022
    buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/13/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): text 80x25
XEN: Hypervisor version 4.16 detected.
CPU microcode: no matching update found
CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3092.99-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0x1fc3fbff
  Features2=0x9fba2203
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c000000
  XSAVE Features=0x1
  AMD Extended Feature Extensions ID EBX=0x101000
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 9725026304 (9274 MB)
avail memory = 8273047552 (7889 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
ioapic0  irqs 0-23
Launching APs: 1
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: 
smbios0:  at iomem 0xf0480-0xf049e
smbios0: Version: 2.6, BCD Revision: 2.6
aesni0: 
acpi0: 
acpi0: Power Button (fixed)
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
hpet0: memory region width 1024 too small for 32 timers
device_attach: hpet0 attach returned 6
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
[ ... At this point, the system hangs, needing a hard reset ...]



BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 614kB/523264kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
OK  unset xen_kernel
OK boot
Loading kernel...
boot/kernel/kernel text=0x1855c8 text=0xe02330 /-\|/-\text=0x665524 |/-data=0x140 data=0x1c0448+0x43ebb8 \syms=[0x8+0x18a350|+0x8+0x1a8b68/]
Loading configured modules...
/etc/hostid size=0x25
/boot/kernel/tpm.ko size 0xad70 at 0x2133000
/boot/entropy size=0x1000
/boot/kernel/ipmi.ko /size 0x11950 at 0x213f000
loading required module 'smbus'
/boot/kernel/smbus.ko size 0x3cb0 at 0x2151000
/boot/firmware/intel-ucode.bin \|size=0x303800
/boot/kernel/geom_mirror.ko size 0x20c80 at 0x2459000
---<>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-STABLE #0 stable/13-984a45d77: Sat Jun  4 07:38:09 PDT 2022
    buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/13/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): resolution 640x480
CPU microcode: updated from 0x2e to 0x2f
CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3093.10-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0xbfebfbff
  Features2=0x1fbae3ff
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c000400
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33272487936 (31731 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0  irqs 0-23
Launching APs: 2 3 1
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: 
smbios0:  at iomem 0xf0480-0xf049e
smbios0: Version: 2.6, BCD Revision: 2.6
aesni0: 
acpi0: 
acpi0: Power Button (fixed)
cpu0:  on acpi0
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
pcib1:  irq 16 at device 1.0 on pci0
pci1:  on pcib1
mps0:  port 0xe000-0xe0ff mem 0xfe6c0000-0xfe6c3fff,0xfe680000-0xfe6bffff irq 16 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c
vgapci0:  port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
pci0:  at device 22.0 (no driver attached)
atapci0:  port 0xf130-0xf137,0xf120-0xf123,0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0ff irq 18 at device 22.2 on pci0
ata2:  at channel 0 on atapci0
ata3:  at channel 1 on atapci0
uart2:  port 0xf0e0-0xf0e7 mem 0xfe729000-0xfe729fff irq 17 at device 22.3 on pci0
uart2: Using 1 MSI message
em0:  port 0xf080-0xf09f mem 0xfe700000-0xfe71ffff,0xfe728000-0xfe728fff irq 20 at device 25.0 on pci0
em0: EEPROM V0.13-4
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:22:4d:56:3c:f7
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0:  mem 0xfe727000-0xfe7273ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
hdac0:  mem 0xfe720000-0xfe723fff irq 22 at device 27.0 on pci0
pcib2:  irq 17 at device 28.0 on pci0
pci2:  on pcib2
pcib3:  irq 18 at device 28.6 on pci0
pci3:  on pcib3
xhci0:  mem 0xfe500000-0xfe501fff irq 18 at device 0.0 on pci3
xhci0: 32 bytes context size, 32-bit DMA
usbus1 on xhci0
usbus1: 5.0Gbps Super Speed USB v3.0
ehci1:  mem 0xfe726000-0xfe7263ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
usbus2: 480Mbps High Speed USB v2.0
pcib4:  at device 30.0 on pci0
pci4:  on pcib4
pci4:  at device 3.0 (no driver attached)
isab0:  at device 31.0 on pci0
isa0:  on isab0
ahci0:  port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe725000-0xfe7257ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0:  at channel 0 on ahci0
ahciem0:  on ahci0
acpi_button0:  on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0uart0: console (9600,n,8,1)
tpm0:  iomem 0xfed40000-0xfed44fff on acpi0
tpm: WEC WPCT200 rev 0x46
WARNING: Device "tpm" is Giant locked and may be deleted before FreeBSD 14.0.
orm0:  at iomem 0xd4800-0xd57ff pnpid ORM0000 on isa0
atkbdc0:  at port 0x60,0x64 on isa0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
est0:  on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 278d00001f00
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 278d00001f00
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 278d00001f00
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 278d00001f00
device_attach: est3 attach returned 6
Timecounter "TSC-low" frequency 1546486788 Hz quality 1000
Timecounters tick every 1.000 msec
hdacc0:  at cad 2 on hdac0
hdaa0:  at nid 1 on hdacc0
pcm0:  at nid 20 and 24,26 on hdaa0
pcm1:  at nid 27 and 25 on hdaa0
pcm2:  at nid 30 on hdaa0
pcm3:  at nid 23 on hdaa0
hdacc1:  at cad 3 on hdac0
hdaa1:  at nid 1 on hdacc1
pcm4:  at nid 6 on hdaa1
Trying to mount root from ufs:/dev/mirror/gptroot []...
ugen2.1:  at usbus2
ugen1.1: <(0x1033) XHCI root HUB> at usbus1
ugen0.1:  at usbus0
uhub0 on usbus2
uhub0:  on usbus2
uhub1 on usbus0
uhub1:  on usbus0
uhub2 on usbus1
uhub2: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub2: 4 ports with 4 removable, self powered
uhub0: 3 ports with 3 removable, self powered
Root mount waiting for: CAM usbus0 usbus2
uhub1: 3 ports with 3 removable, self powered
ugen2.2:  at usbus2
uhub3 on uhub0
uhub3:  on usbus2
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ses0:  SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: pass4,cd0 in 'Slot 00', SATA Slot: scbus3 target 0
cd0 at ahcich0 bus 0 scbus3 target 0 lun 0
cd0:  Removable CD-ROM SCSI device
cd0: Serial Number 3524220 228217500534
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0:  Fixed Direct Access SPC-4 SCSI device
da0: Serial Number VAJ3Y51L            
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 7630885MB (15628053168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2:  Fixed Direct Access SPC-4 SCSI device
da2: Serial Number VGKUGHSG            
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 7630885MB (15628053168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3:  Fixed Direct Access SPC-4 SCSI device
da3: Serial Number VAJYL6HL            
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (15628053168 512 byte sectors)
ugen0.2:  at usbus0
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1:  Fixed Direct Access SPC-4 SCSI device
da1: Serial Number VAJ3RZJL            
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 7630885MB (15628053168 512 byte sectors)
uhub4 on uhub1
uhub4:  on usbus0
GEOM_MIRROR: Device mirror/gptroot launched (1/2).
GEOM_MIRROR: Device gptroot: rebuilding provider da0p2.
GEOM_MIRROR: Device mirror/gptswap launched (2/2).
GEOM_MIRROR: Device mirror/gptback launched (2/2).
Root mount waiting for: usbus0 usbus2
uhub3: 8 ports with 8 removable, self powered
uhub4: 6 ports with 6 removable, self powered
sysctl: kern.timecounter.hardware=XENTIMER at line 13: Invalid argument
Setting hostuuid: 49434d53-0200-9009-2500-09902500dac8.
Setting hostid: 0x0535156f.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM: zvol/xendisks/loth_9_install: invalid disklabel.
Starting file system checks:
/dev/mirror/gptroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gptroot: clean, 56609677 free (27589 frags, 7072761 blocks, 0.0% fragmentation)
Mounting local filesystems:.
Setting hostname: xen-lothlorien.nfbcal.org.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/gcc10 /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
Autoloading module: ichsmb
ichsmb0:  port 0xf040-0xf05f mem 0xfe724000-0xfe7240ff irq 18 at device 31.3 on pci0
smbus0:  on ichsmb0
bridge0: Ethernet address: 58:9c:fc:10:94:73
Created clone inlo0: link state changed to UP
terfaces: bridge0.
em0: promiscuous mode enabled
bridge0: link state changed to UP
Starting Network: lo0 em0 bridge0.
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff0000em0: link state changed to UP
em0.22: link state changed to UP
00
	groups: lo
	nd6 options=21
em0: flags=8963 metric 0 mtu 1500
	options=4812098
	ether 00:22:4d:56:3c:f7
	media: Ethernet autoselect
	status: no carrier
	nd6 options=29
bridge0: flags=8843 metric 0 mtu 1500
	ether 58:9c:fc:10:94:73
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: em0 flags=143
	        ifmaxaddr 0 port 1 priority 128 path cost 2000000
	groups: bridge
	nd6 options=9
Starting devd.
Sleeping 60  seconds before starting...
Waiting 60s for the default route interface: .....(no carrier)
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add net default: gateway 10.40.60.1
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
iUpdating motd:.


From nobody Sat Jun  4 20:28:56 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3889B1BD6984
	for ; Sat,  4 Jun 2022 20:29:01 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LFrv00xL3z4cJk
	for ; Sat,  4 Jun 2022 20:29:00 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 254KSvpp016387
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Sat, 4 Jun 2022 13:28:57 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 254KSvjE017533;
	Sat, 4 Jun 2022 13:28:57 -0700 (PDT)
Message-Id: <202206042028.254KSvjE017533@nfbcal.org>
From: Brian Buhrow 
Date: Sat, 4 Jun 2022 13:28:56 -0700
In-Reply-To: <202206042003.254K3uV1016135@nfbcal.org>
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Sat, 04 Jun 2022 13:28:57 -0700 (PDT)
X-Rspamd-Queue-Id: 4LFrv00xL3z4cJk
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  Following up on my original post, it appears that my speculation that the failing
hpet timer was the source of the trouble is wrong. Under releng/13.1-RELEASE, the system boots
and runs fine with xen-4.16, modulo the missing netback.c patch, and the hpet timer still
fails.  The error message is different, but it still fails.  
Under 13.1-RELEASE, I see:
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
hpet0: HPET never increments, disabling
device_attach: hpet0 attach returned 6


Just offering this as a data point so we don't spend a lot of time chasing it down for no
reason.

-thanks
-Brian


From nobody Sun Jun  5 13:41:22 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5EB261BDFBE6
	for ; Sun,  5 Jun 2022 13:41:38 +0000 (UTC)
	(envelope-from prvs=148314efc=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LGHpS5yVLz3tML
	for ; Sun,  5 Jun 2022 13:41:36 +0000 (UTC)
	(envelope-from prvs=148314efc=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654436496;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=VvBZF/iSQOsZquVP686+StieRLcH0qcrN5CnOGFvkwk=;
  b=WdDs9lVtKetUeBcsK3aFeYrWC8mkkmyPdqN5yHrOC+GX182gBEc1bRx3
   Q9aIFVc+9ByRstG/7H2AfC0IeIt61I+sggNJUZIX1asDpFdUagwG+EFtS
   GIaE1D85dq7z3WuUrpF4zeEJtH0QX83lNS/PKfkmQ40QV08UEC+rSrHgr
   4=;
X-IronPort-RemoteIP: 104.47.57.170
X-IronPort-MID: 72886094
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:f45St68MRrdomOY5XeCeDrUD23+TJUtcMsCJ2f8bNWPcYEJGY0x3z
 zdMXW+PbvuLZzCmLtglPo+z9EIDv56Bz99rSQQ//yw8E34SpcT7XtnIdU2Y0wF+jyHgoOCLy
 +1EN7Es+ehtFie0Si+Fa+Sn9T8mvU2xbuKU5NTsY0idfic5DnZ44f5fs7Rh2NQw34DkW1rlV
 e7a+KUzBnf0g1aYDUpMg06zgEsHUCPa4W5wUvQWPJinjXeG/5UnJMt3yZKZdhMUdrJ8DO+iL
 9sv+Znilo/vE7XBPfv++lrzWhVirrc/pmFigFIOM0SpqkAqSiDfTs/XnRfTAKtao2zhojx/9
 DlCnd+yEC4mZY78pNUySAZGSCtgZv1e06CSdBBTseTLp6HHW13F5qw3SW0TY8gf8OsxBnxS/
 /sFLjxLdgqEm++93LO8TK9rm9gnK87oeogYvxmMzxmAVapgHc+FHvqMvIAAtNszrpkm8fL2f
 c0WZCApdB3dSxZOJk0WGNQ1m+LAanzXLGcI+AzN9PZfD277ySV86IPMHd7sQ/+taMBNlEqzh
 lzX4DGsav0dHJnFodafyVqmm+/VmjzTXowYH7S8sPlwjzW7yHYeBRcNWR6rrP+RhFSzVthEJ
 woT4CVGkEQp3EmiT924UxjmpneB50cYQ4AJTLV87xyRwK3J5QrfHnIDUjNKdN0hsokxWCAu0
 ViK2djuAFSDrYGodJ5UzZ/MxRvaBMTfBTZqifMsJefd3+TenQ==
IronPort-HdrOrdr: A9a23:JF0k5KBehPUt9EjlHegwsceALOsnbusQ8zAXPh9KJCC9I/bzqy
 nxpp8mPH/P5wr5lktQ/OxoHJPwOU80lKQFmLX5WI3PYOCIgguVxe1ZnOjfKnjbalbDH41mpN
 tdmspFebrN5DFB5K6VgTVQUexQpuVvmJrY+Ns2pE0dKT2CBZsQjTuQXW2gYzdLrUR9dOwEPa
 vZwvACiyureHwRYMj+Ln4ZX9Lbr9mOsJ79exYJCzMu9QHL1FqTmfbHOind+i1bfyJEwL8k/2
 SAuwvl5p+7u/X+7hPHzWfc47lfhdOk4NpeA86njNQTN1zX+0+VTbUkf4fHkCE+oemp5lpvuN
 7Qoy04N8A20H/VdnHdm2qZ5yDQlBIVr1Pyw16RhnXu5ebjQighNsZHjYVFNjPE9ksJprhHoe
 529lPck6ASIQLLnSz76dSNfQptjFCIrX0rlvNWp2BDULEZdKRaoeUkjQ5o+a87bWzHAb0cYa
 hT5Jm23ocXTbraVQGSgoBX+q3iYpxpdS32AXTruaSuokprdT5CvgklLfck7wk9HaIGOuZ5Dt
 v/Q9VVfZF1P7srhPFGdZA8qfXeMB28fTv8dESvHH/AKIYrf1rwlr+f2sRH2AjtQu1C8KcP
X-IronPort-AV: E=Sophos;i="5.91,279,1647316800"; 
   d="scan'208";a="72886094"
Received: from mail-dm6nam11lp2170.outbound.protection.outlook.com (HELO NAM11-DM6-obe.outbound.protection.outlook.com) ([104.47.57.170])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 05 Jun 2022 09:41:28 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=nweFnNMTIF/EdkNbGwAy1A6Zsgh2bt2qCsWrTB3lu3C7JPJNZHglJaztjT5HSQBnxH4fbAw3tb8K8ze9Tro5Afp/vaKxa5tPEM94MX6cUlvEvqnhIO3xiZd/uGsQ3xUk1KIBxuj8u2xSuH6OP3Z6xeiG1P853Gievxpy/xBkz7zcjMsPtSnYfYTGTJcZJEV+gUA9fOWDbaAYM/1fvqbXF9AGxCVEZ+EyiQLlmlxKx5jtZXkQTTE6y1Lx4oDo6hgHtoCqeHKIvgJbR3UGGCxSW081T8OHGvCLRXrXfQDQV4e0aQI96G7HNdJ1Z36xZIJBBzG0CRHJyEd94LLt40WZrw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=vwUUxBhi6SwWyOxk/NQPyVM7d6D2c9sNSa5i3HDkl58=;
 b=m63leth+O/jV8+xr2uWM+R1fC4JkwqMnHFS9vcdrXiNu0cUIKrRYZcF0lWojrrCarWfxM8R0PNqLy6z9ksB3Q545cO4+P95obqY75K09n9Mqh3TaB2DY2FC7quM5EwgLmFukG2YcgV4WtrhEfNdyWxYBF8P5Ew5w7fP6NEek35RaVzEeRpSHKA8OtfjCcTGafS6gIKiJ8FIA3Pa3/xeHOMVCEvqm9+vXjxOXd0FiLOYSfM5AEURLnTws6XCPN5o1SIHD6N6tFxsa3uE/owHURpFKw2+HYORdEiRvx263bpAgIOsH3O5ntTIf1L9rQ720dgJUgWaHWacq814w80TK/A==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=vwUUxBhi6SwWyOxk/NQPyVM7d6D2c9sNSa5i3HDkl58=;
 b=rvUSuGfsmBB8LSEYVL4pO/iHZwBdaSTP5WIdHJUjI0ZQjVejtDkYGWbXK7uNYX/u8Acud3ZL9fSMcQrf8qrvcFI2rjhINzRfviVKcVhsXkxYyGwr59Yl8rQh+fq+CEZx3LEpJcM/eBUhdndFaXDfIrpYFY6wlhNQO+RUcXgZPM8=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by BY5PR03MB5095.namprd03.prod.outlook.com (2603:10b6:a03:1ee::17) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Sun, 5 Jun
 2022 13:41:26 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.018; Sun, 5 Jun 2022
 13:41:26 +0000
Date: Sun, 5 Jun 2022 15:41:22 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: <202206042003.254K3uV1016135@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206042003.254K3uV1016135@nfbcal.org>
X-ClientProxiedBy: LO4P123CA0381.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:18f::8) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: fc18df89-6248-4e4b-bd08-08da46f9169d
X-MS-TrafficTypeDiagnostic: BY5PR03MB5095:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	OROyUoB6deSjT1A2pLH//QIfbvA8q9RAyZKUQNpkuSLKdw5e3L0yKXKPRiBbS0l6zZXsVxVWTnQvQnkH818GlsuhdF8higLV9f+ojXFXXNJLNS2Ud1OTAhBAGsB76o5zyxfaheQo8oXy1H1FlmMOH6Z67o5BwUECupQis0cEX8bsOAjORdVf0uuSGYEKEeEc+2+/MQSLZGJ/eGH+1Pcrhnea29JWeT/7jmNsYAulf7gK4V7T/Jhu6dKM20Xalx6rrGFp+90uHg0Hv2tJWRDjbUMuMh7E0W6mIJbz0fEnfH5BWksz2OhVZpPRYixqZDxGNx8QKqQ2s0SKRp+Irc64Wt+RY5vJg7+dNqCxm6JQbY91B5n10IQBBIJ5MrGvKWJpHF8JMfwLZMntFDa+liRYLjP4770RqkOMVWvyGgJj8eBzIKzZN/WQrVETiwweBY/NmdeFCO2S+Kw5Jmxbcr3HwlLi4SZloGJ/HojaI4O4qdGnNHIG3sCOufnKog4L639s524hj5UuW81p+y1xmyBy1tZtff7oxOghzG1xmrVu4W8N7OqeeEMHxpyzBF7GImRdjgwskfwALifbiRguV98lkX6teRXRPglKLZYyBTRt3rIpcTPh4ApN4Smls6xrXhelJ1DqKw4RkjUVbskM2nyJYA==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(508600001)(38100700002)(85182001)(316002)(83380400001)(186003)(6916009)(9686003)(66946007)(6512007)(86362001)(33716001)(4744005)(5660300002)(66556008)(66476007)(8676002)(4326008)(82960400001)(6666004)(8936002)(6506007)(6486002)(2906002)(26005);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WVZLVnBqYmE3N1hMRkttSi9lNkZ5amx6V0NZTTV2MFFFbVhtRFN2TVk3Ritl?=
 =?utf-8?B?ZUt2ejRVa3ZSM3BsS0tWRkpTR2pERUFLYjdmaUNQOVVwcE9rZ0gydFY0TE11?=
 =?utf-8?B?bGtIeTUycUdaSTV5ejNIZTdGYkltR3h0cWNWZ202MitHNWtkaE81OExXSThG?=
 =?utf-8?B?Vis0emdRWENLbWo0MXRXZW11bDNGeU9pY3dNY0Y2cy92RkRkLzI3YVMvTlVh?=
 =?utf-8?B?bXgzNzNuRk9KVXdyU1JwVEJId2pCa1NUQXJUVlZubElOVnc5U0x2RzlJK1Nz?=
 =?utf-8?B?ZCtLSUJzWndQYVVjTnJtMFpERzdJWVNuU3puM1JiVkpXd0ErUEt5VGdFMXoy?=
 =?utf-8?B?QUl0VHZFdndzZHN2UU1jS2d2TjVGb2NnZnJtOG1MZkFEUnVkMVV1aWlOMkc2?=
 =?utf-8?B?RTl1U1ZmblhBT2J0Z0w1UlpNN0JwSTY2bzg4RVRCaXBINE44dnp6VmlzMFhu?=
 =?utf-8?B?RVBDNFN4cTdzTWpUaDVzTmJuUlpuRDVDZFZHVVp1S1ZyY2J5VTJ5b09WS1A1?=
 =?utf-8?B?RER4US9GOE9OWTBRQ25FVjk5RjZuaDRMNVpjV1hSbmZGTHdSTEtJUGJNOGgv?=
 =?utf-8?B?Q2tBd0U3UUZ5N055YWlESkcrQ1ZZc25VNFpPNFo5YW9adklZWnNKaFozMHEw?=
 =?utf-8?B?MVdtcUtEd1RXcEZ3QmUyeHNuU3dCaFFJMmRKb0ppdjVnSlk5alJWcVJZd25i?=
 =?utf-8?B?U1NRM3RKZFJkdkx3dTMvU0huaFVUOVhBOVhaOG94M3RaRXZhTC94VTFlWHNB?=
 =?utf-8?B?SG95QnFZTHpYYUVseWE4VmVLWXF5VXptSTU4VXcwb2NSOTNCakIwYkw3ekpm?=
 =?utf-8?B?UzBzZ25VSUloOExQNjlaLzhQZFB3TDVOUWkxSElvbWlxdGJTQzE2bWFsYS9L?=
 =?utf-8?B?dUY0am1tNGFJVi9MT2VYV0xnaFE2WG1sQlptb0RwVmZQcUlYSzR2N09XVjA0?=
 =?utf-8?B?cHB1dURzQzBIK0gzaWE2WHFhUGRPRE82a0lEeWJOS3pHMTlJOVhGRS9OQ3ZL?=
 =?utf-8?B?ZXY1T3l4c21iNFVrbko3aU1MMGQvZ1pVTmFDVG4xN3VIM2Q2YTJMbDhERjl1?=
 =?utf-8?B?Yk1lQ2tiY0FibDErbEplNDRIb1dQMmpaU25OSEZPTFpWRDF6cHNTWDRPTDVU?=
 =?utf-8?B?M3kwcFMxdExMVS9CNnRvdVVZZi9TT2pWbFJEKy9MNERhVm02L2FjR2JOZXBB?=
 =?utf-8?B?TlVNeGU0MlhSNmZTa1JVZmdnUlBTRXM2ZDFCZUhjdE1ic3gwNm91T2hPUWRy?=
 =?utf-8?B?TGlKcUluMUtod3pqUlJXOERPQTJKZWRjNTE0NWZiY0dETUJmTHdaZS9GUlZQ?=
 =?utf-8?B?MVVWdTlEUTJCRlBFUkNvY2t6QlNzYlNwWkFRTDRoYTZXcjBYeXIxNUxwZHMr?=
 =?utf-8?B?aFFoV2dmSUZTMnF1SGJ3QlhEWEZhNzZkeDc2Nno0cG0wRm9HbWd5OVJrNXpK?=
 =?utf-8?B?ODloS1Q4VTFVN3hJc3cxdStjNy9RaDBVZHFkZVVwdExxZTczMjVXR3ZhMXZO?=
 =?utf-8?B?YnYyQUpDZFBnM0pKOUtMNGhYYzZ5c1pwYmFralYzaHY4STBabGl2QlNhenZi?=
 =?utf-8?B?YkJlNzY5QVZ5aGlQREpPb01Jb3NBSFdpakJDUlhJdHBhV2Z1SytjallpeFRh?=
 =?utf-8?B?Mk41UnlUSXl4WW9BZlZGYWY1Mm56SjlJTUlFQTQ2cFI5N3JJNWNmRCtjMFg3?=
 =?utf-8?B?S0l3Mi9NbldWeU5mZWxBN3U2dnVPQWhYdi94NmZBUE41QzAvd1hicXV5NkFF?=
 =?utf-8?B?amtEQndUa24ycFkyZlhoWStndVFqTm9lU2E1dWJDZ1dkTlVmdllWRTlTbDFq?=
 =?utf-8?B?eW0yT3FqQ1RiQTFHVTB0VW5HWjZuQ09Bc1o4cVUwTVBCQWpwMnlLdzNJeitF?=
 =?utf-8?B?T2w0aGpaemplRzNITGpJNzZUSG92LzdxR2Iwa3FFU2Y5SzlUd1hoQzlaNXpp?=
 =?utf-8?B?dkF3ZjRtWTZ5cVZYclB0bU5wRm5GOUtHWmRsR3NnU2x4ZUFqTWl2dnpoRnFE?=
 =?utf-8?B?cmFNSjg1VStwOGQ0Slp3SlJiSzZuTzR2czFJOVlBMHBWTWpWVHJaMnJBUklF?=
 =?utf-8?B?OTFnT3hNZlBKUXZnVTkzYnZpbTdpNE4wbC9sTjB0SUlkd3JYOU4yZ2lmekx2?=
 =?utf-8?B?c1MxaUhZY1VxNzZQcWZhODJzT0pFNmRNcjRja0tWL0tmS0JBTk5NL0xMQ2pT?=
 =?utf-8?B?WEIxMW8yVC9VbytiMld2THdVOGdNV2RwMUZVelp2WUpHNE51WTZGNE1lRjQw?=
 =?utf-8?B?S0VRd1VrNG9qL2JHb2crUmtyNkdadTVGcjZZVGpKcXlRVVVWeStDcFB2dTB1?=
 =?utf-8?B?cC9YVTQwUVJBOUFLMVlrdU5McDZ3andhcHh0aUxUYVVIbXBtMlBtbEtlYTJ5?=
 =?utf-8?Q?2D7Px86elziSa6ks=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fc18df89-6248-4e4b-bd08-08da46f9169d
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2022 13:41:26.7532
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YFz1gAN4n3VVEAj11CE5FKSeCj34P+k4ACtQrkk4Snd97qoP713saEsoqn3tUZomr3MudWGoby3ct7SfX+hbqw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR03MB5095
X-Rspamd-Queue-Id: 4LGHpS5yVLz3tML
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=WdDs9lVt;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=rvUSuGfs;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=148314efc=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=148314efc=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=148314efc=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sat, Jun 04, 2022 at 01:03:55PM -0700, Brian Buhrow wrote:
> 	Hello.  In the process of getting the netback.c patch into a working kernel on the system
> I'm updating, I discovered that Stable/13 doesn't boot successfully when xen-4.16 is running.
> Specifically, the system hangs while configuring the PCI busses.  I suspect it has something to
> do with the failure of the hpet timers to attach.
> I do not know if it works with older versions of the xen hypervisor, but it boots and runs fine
> without xen running.  Below are two boot logs, the first with xen installed and running, the
> one that fails, and the successfful boot,  with FreeBSD running on bare metal.
> Hopefully this is a known issue and there is an easy fix.

I will have to give it a try, in the meantime could you give a try
with a kernel from HEAD directly, so that I know if the issue is also
present there?

Trying to boot a HEAD kernel with a 13 userland should be fine.

Thanks, Roger.

From nobody Tue Jun  7 06:05:05 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C298A1BD0376
	for ; Tue,  7 Jun 2022 06:05:34 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHKbJ3zGFz3jQ0
	for ; Tue,  7 Jun 2022 06:05:32 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 257656IL024163
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Mon, 6 Jun 2022 23:05:06 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 257655Bb025456;
	Mon, 6 Jun 2022 23:05:05 -0700 (PDT)
Message-Id: <202206070605.257655Bb025456@nfbcal.org>
From: Brian Buhrow 
Date: Mon, 6 Jun 2022 23:05:05 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Mon, 06 Jun 2022 23:05:06 -0700 (PDT)
X-Rspamd-Queue-Id: 4LHKbJ3zGFz3jQ0
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-2.89 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-0.59)[-0.593];
	 RCPT_COUNT_THREE(0.00)[3];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 FROM_HAS_DN(0.00)[];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-0.999];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  I tried booting my system from a head kernel and it boots ans runs
successfully, boot logs shown below.  However, there is an issue both with the working
releng/13.1 and the head kernel, which is that upon shutdown, the system panics.  It's mostly
harmless since it seems to do it after all buffers are synced, but it doesn't panic if FreeBSD
is running on bare metal and it doesn't look nice when  operators see it.

I've captured the logs from the failure on head, but I've reverted the system back to
releng/13.1 with the netback patch installed, so it's back in production mode.
For this message, I'll include the boot log for the head kernel.

I'll followup with a second message on this thread with the shutdown log, which fails.

Hope this helps you track down the issue with stable/13 as well.

-thanks
-Brian


Script started on Mon Jun  6 09:38:54 2022

[  Not too informative shutdown log from releng/13.1, but shows the panic. ]

login: Jun  6 09:39:28 xen-lothlorien shutdown[7223]: reboot by buhrow: 
Stopping sshd.
Waiting for PIDS: 1299.
Stopping cron.
Waiting for PIDS: 1283.
Stopping devd.
Waiting for PIDS: 544.
Writing entropy file: .
Writing early boot entropy file: .
Terminated
.
Jun  6 09:39:29 xen-lothlorien syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 2 1 0 done
All buffers synced.
Uptime: 2d0h10m24s
GEOM_MIRROR: Device gptback: provider destroyed.
GEOM_MIRROR: Device gptback destroyed.
GEOM_MIRROR: Device gptswap: provider destroyed.
GEOM_MIRROR: Device gptswap destroyed.
GEOM_MIRROR: Device gptroot: provider destroyed.
GEOM_MIRROR: Device gptroot destroyed.
uhub4: detached
uhub1: detached
uhub0: detached
uhub3: detached
uhub2: detached


Fatal trap 12: page fault while in kernel mode
cpuid = 1

[ ... cuts to boot log with head kernel ... ]

 Xen 4.16.0
(XEN) Xen version 4.16.0 (buhrow@) (FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)) debug=n Thu Jun  2 11:30:17 PDT 2022
(XEN) Latest ChangeSet: 
(XEN) Bootloader: FreeBSD Loader
(XEN) Command line: dom0_mem=8192m dom0_max_vcpus=2 dom0=pvh pv-l1tf=off,domu=off console=com1,vga com1=9600,8n1
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 00000000000997ff] (usable)
(XEN)  [0000000000099800, 000000000009ffff] (reserved)
(XEN)  [00000000000e0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 000000001fffffff] (usable)
(XEN)  [0000000020000000, 00000000201fffff] (reserved)
(XEN)  [0000000020200000, 000000003fffffff] (usable)
(XEN)  [0000000040000000, 00000000401fffff] (reserved)
(XEN)  [0000000040200000, 00000000bc855fff] (usable)
(XEN)  [00000000bc856000, 00000000bc85efff] (ACPI data)
(XEN)  [00000000bc85f000, 00000000bc8a9fff] (ACPI NVS)
(XEN)  [00000000bc8aa000, 00000000bc8b1fff] (usable)
(XEN)  [00000000bc8b2000, 00000000bc9a4fff] (reserved)
(XEN)  [00000000bc9a5000, 00000000bc9a6fff] (usable)
(XEN)  [00000000bc9a7000, 00000000bcbc5fff] (reserved)
(XEN)  [00000000bcbc6000, 00000000bcbc6fff] (usable)
(XEN)  [00000000bcbc7000, 00000000bcbd6fff] (reserved)
(XEN)  [00000000bcbd7000, 00000000bcbf4fff] (ACPI NVS)
(XEN)  [00000000bcbf5000, 00000000bcc18fff] (reserved)
(XEN)  [00000000bcc19000, 00000000bcc5bfff] (ACPI NVS)
(XEN)  [00000000bcc5c000, 00000000bce7bfff] (reserved)
(XEN)  [00000000bce7c000, 00000000bcffffff] (usable)
(XEN)  [00000000bd800000, 00000000bf9fffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed3ffff] (reserved)
(XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000083e5fffff] (usable)
(XEN) New Xen image base address: 0xbc200000
(XEN) ACPI: RSDP 000F0450, 0024 (r2  INTEL)
(XEN) ACPI: XSDT BC856070, 0064 (r1 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: FACP BC85DBC0, 00F4 (r4 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: DSDT BC856168, 7A54 (r2 INTEL  DQ67SW         16 INTL 20051117)
(XEN) ACPI: FACS BCBDBF80, 0040
(XEN) ACPI: APIC BC85DCB8, 0072 (r3 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: TCPA BC85DD30, 0032 (r2 INTEL  DQ67SW          1 MSFT  1000013)
(XEN) ACPI: SSDT BC85DD68, 0102 (r1 INTEL  DQ67SW          1 MSFT  3000001)
(XEN) ACPI: MCFG BC85DE70, 003C (r1 INTEL  DQ67SW    1072009 MSFT       97)
(XEN) ACPI: HPET BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
(XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
(XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
(XEN) System RAM: 32683MB (33467896kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1600 ... 3100 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- STIBP- SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 3092.985 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Dom0 has maximum 440 PIRQs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 600kB init memory
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<>---
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-71df58aef: Mon Jun  6 07:46:07 PDT 2022
    buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/current/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 14.0.4 (https://github.com/llvm/llvm-project.git llvmorg-14.0.4-0-g29f1039a7285)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
XEN: Hypervisor version 4.16 detected.
CPU microcode: no matching update found
CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3092.98-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0x1fc3fbff
  Features2=0x9fba2203
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features3=0x9c000000
  XSAVE Features=0x1
  AMD Extended Feature Extensions ID EBX=0x101000
  TSC: P-state invariant
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 9725026304 (9274 MB)
avail memory = 8252633088 (7870 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
ioapic0  irqs 0-23
Launching APs: 1
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: 
smbios0:  at iomem 0xf0480-0xf049e
smbios0: Version: 2.6, BCD Revision: 2.6
aesni0: 
acpi0: 
acpi0: Power Button (fixed)
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
hpet0: memory region width 1024 too small for 32 timers
device_attach: hpet0 attach returned 6
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
pcib1:  irq 16 at device 1.0 on pci0
pci1:  on pcib1
mps0:  port 0xe000-0xe0ff mem 0xfe6c0000-0xfe6c3fff,0xfe680000-0xfe6bffff irq 16 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c
vgapci0:  port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
pci0:  at device 22.0 (no driver attached)
atapci0:  port 0xf130-0xf137,0xf120-0xf123,0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0ff irq 18 at device 22.2 on pci0
ata2:  at channel 0 on atapci0
ata3:  at channel 1 on atapci0
uart2:  port 0xf0e0-0xf0e7 mem 0xfe729000-0xfe729fff irq 17 at device 22.3 on pci0
uart2: Using 1 MSI message
em0:  port 0xf080-0xf09f mem 0xfe700000-0xfe71ffff,0xfe728000-0xfe728fff irq 20 at device 25.0 on pci0
em0: EEPROM V0.13-4
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:22:4d:56:3c:f7
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0:  mem 0xfe727000-0xfe7273ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
hdac0:  mem 0xfe720000-0xfe723fff irq 22 at device 27.0 on pci0
pcib2:  irq 17 at device 28.0 on pci0
pci2:  on pcib2
pcib3:  irq 18 at device 28.6 on pci0
pci3:  on pcib3
xhci0:  mem 0xfe500000-0xfe501fff irq 18 at device 0.0 on pci3
xhci0: 32 bytes context size, 32-bit DMA
usbus1 on xhci0
usbus1: 5.0Gbps Super Speed USB v3.0
ehci1:  mem 0xfe726000-0xfe7263ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
usbus2: 480Mbps High Speed USB v2.0
pcib4:  at device 30.0 on pci0
pci4:  on pcib4
pci4:  at device 3.0 (no driver attached)
isab0:  at device 31.0 on pci0
isa0:  on isab0
ahci0:  port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe725000-0xfe7257ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0:  at channel 0 on ahci0
ahciem0:  on ahci0
acpi_button0:  on acpi0
tpm0:  iomem 0xfed40000-0xfed44fff on acpi0
tpm: device 0xffffffff rev 0xff
WARNING: Device "tpm" is Giant locked and may be deleted before FreeBSD 14.0.
xenpv0: 
granttable0:  on xenpv0
xc0:  on xenpv0
xen_et0:  on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0:  on xenpv0
xsd_dev0:  on xenpv0
evtchn0:  on xenpv0
privcmd0:  on xenpv0
gntdev0:  on xenpv0
debug0:  on xenpv0
orm0:  at iomem 0xd4800-0xd57ff pnpid ORM0000 on isa0
vga0:  at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
atkbdc0:  at port 0x60,0x64 on isa0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
Timecounter "TSC-low" frequency 1546492500 Hz quality 1000
Timecounters tick every 10.000 msec
hdacc0:  at cad 2 on hdac0
hdaa0:  at nid 1 on hdacc0
pcm0:  at nid 20 and 24,26 on hdaa0
pcm1:  at nid 27 and 25 on hdaa0
pcm2:  at nid 30 on hdaa0
pcm3:  at nid 23 on hdaa0
hdacc1:  at cad 3 on hdac0
hdaa1:  at nid 1 on hdacc1
pcm4:  at nid 6 on hdaa1
WARNING: WITNESS option enabled, expect reduced performance.
ugen2.1:  at usbus2
ugen0.1:  at usbus0
uhub0 on usbus2
uhub1 on usbus0
uhub0:  on usbus2
uhub1:  on usbus0
ugen1.1: <(0x1033) XHCI root HUB> at usbus1
uhub2 on usbus1
Trying to mount root from ufs:/dev/mirror/gptroot []...
uhub2: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub2: 4 ports with 4 removable, self powered
Root mount waiting for: CAM usbus0 usbus2
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
ugen0.2:  at usbus0
uhub3 on uhub1
uhub3:  on usbus0
ugen2.2:  at usbus2
uhub4 on uhub0
uhub4:  on usbus2
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ses0:  SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: cd0 in 'Slot 00', SATA Slot: scbus3 target 0
cd0 at ahcich0 bus 0 scbus3 target 0 lun 0
cd0:  Removable CD-ROM SCSI device
cd0: Serial Number 3524220 228217500534
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2:  Fixed Direct Access SPC-4 SCSI device
da2: Serial Number VGKUGHSG
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 7630885MB (15628053168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1:  Fixed Direct Access SPC-4 SCSI device
da1: Serial Number VAJ3RZJL
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 7630885MB (15628053168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3:  Fixed Direct Access SPC-4 SCSI device
da3: Serial Number VAJYL6HL
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (15628053168 512 byte sectors)
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0:  Fixed Direct Access SPC-4 SCSI device
da0: Serial Number VAJ3Y51L
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 7630885MB (15628053168 512 byte sectors)
GEOM_MIRROR: Device mirror/gptroot launched (2/2).
GEOM_MIRROR: Device mirror/gptswap launched (2/2).
GEOM_MIRROR: Device mirror/gptback launched (2/2).
Root mount waiting for: usbus0 usbus2
uhub3: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
Setting hostuuid: 49434d53-0200-9009-2500-09902500dac8.
Setting hostid: 0x0535156f.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM: zvol/xendisks/loth_9_install: invalid disklabel.
Starting file system checks:
/dev/mirror/gptroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gptroot: clean, 56544108 free (28156 frags, 7064494 blocks, 0.0% fragmentation)
Mounting local filesystems:.
Setting hostname: xen-lothlorien.nfbcal.org.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/gcc10 /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
Setting up harvesting: [CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
Autoloading module: ichsmb
ichsmb0:  port 0xf040-0xf05f mem 0xfe724000-0xfe7240ff irq 18 at device 31.3 on pci0
smbus0:  on ichsmb0
bridge0: Ethernet address: 58:9c:fc:10:94:73
Created clone interfaces: bridge0.
lo0: link state changed to UP
em0: link state changed to UP
em0: promiscuous mode enabled
bridge0: link state changed to UP
Starting Network: lo0 em0 bridge0.
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21
em0: flags=8963 metric 0 mtu 1500
	options=4812098
	ether 00:22:4d:56:3c:f7
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=29
bridge0: flags=8843 metric 0 mtu 1500
	ether 58:9c:fc:10:94:73
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: em0 flags=143
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge
	nd6 options=9
Starting devd.
Sleeping 60  seconds before starting...
Waiting 60s for the default route interface: .....(no carrier)
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add net default: gateway 10.40.60.1
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Starting syslogd.
Setting date via ntp.
 6 Jun 09:48:08 ntpdate[767]: step time server 209.81.9.7 offset +4.800691 sec
Mounting late filesystems:.
Starting xenservices: xenstored, xenconsoled.Jun  6 09:48:09 xen-lothlorien xenstored[812]: Checking store ...
Jun  6 09:48:09 xen-lothlorien xenstored[812]: Checking store complete.
xenballoon0:  on xenstore0
xctrl0:  on xenstore0
xs_dev0:  on xenstore0
xenbusb_front0:  on xenstore0
xenbusb_back0:  on xenstore0

Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenservices: xenconsoled.
Setting domain 0 name, domid and JSON config...
Dom0 is already set up

From nobody Tue Jun  7 06:18:09 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7F1251BD1E40
	for ; Tue,  7 Jun 2022 06:18:16 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHKsy4f6nz3kT8
	for ; Tue,  7 Jun 2022 06:18:13 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 2576I9nU026022
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Mon, 6 Jun 2022 23:18:09 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 2576I9NI026060;
	Mon, 6 Jun 2022 23:18:09 -0700 (PDT)
Message-Id: <202206070618.2576I9NI026060@nfbcal.org>
From: Brian Buhrow 
Date: Mon, 6 Jun 2022 23:18:09 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Mon, 06 Jun 2022 23:18:09 -0700 (PDT)
X-Rspamd-Queue-Id: 4LHKsy4f6nz3kT8
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.00 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-0.70)[-0.705];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  Following up on my recent post, here is the log of the failed shutdown from
 FreeBSD head.
This probably doesn't tell you everything, but hopefully it will give you  a clue.

-thanks
-Brian


Jun  6 10:25:34 xen-lothlorien shutdown[1582]: reboot by buhrow:
Stopping sshd.
Waiting for PIDS: 1292.
Stopping cron.
Waiting for PIDS: 1276.
Stopping devd.
Waiting for PIDS: 537.
Writing entropy file: .
Writing early boot entropy file: .
Terminated
.
Jun  6 10:25:34 xen-lothlorien syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 2 3 2 1 0 done
All buffers synced.
Uptime: 39m13s
GEOM_MIRROR^Oyk^[ce gptback: provider destroyed.
GEOM_MIRROR: Device gptswap: provider destroyed.
GEOM_MIRROR: Device gptswap destroyed.
GEOM_MIRROR: Device gptroot: provider destroyed.
GEOM_MIRROR: Device gptroot destroyed.
uhub4: detached
uhub0: detached
uhub2: detached
uhub3: detached
uhub1: detached


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x188
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff82cafe74
stack pointer    ^O^R   = 0x28:0xfffffe00d18a9bb0
yk^[y    nter           = 0x28:0xfffffe00d18a9bb0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (xbbd0 taskq)
trap number             = 12
panic: page fault
cpuid = 0
time = 1654536343
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d18a9970
vpanic() at vpanic+0x151/frame 0xfffffe00d18a99c0
panic() at panic+0x43/frame 0xfffffe00d18a9a20
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00d18a9a80
trap_pfault() at trap_pfault+0xab/frame 0xfffffe00d18a9ae0
calltrap() at calltrap+0x8/frame 0xfffffe00d18a9ae0
--- trap 0xc, rip = 0xffffffff82cafe74, rsp = 0xfffffe00d18a9bb0, rbp = 0xfffffe00d18a9bb0 ---
dmu_objset_zil() at dmu_objset_zil+0x4/frame 0xfffffe00d18a9bb0
zil_open() at zil_open+0xf/frame 0xfffffe00d18a9bd0

l_ensure_zilog() at zvol_ensure_zilog+0xf1/frame 0xfffffe00d18a9bf0
zvol_geom_bio_strategy() at zvol_geom_bio_strategy+0x90/frame 0xfffffe00d18a9c70
xbb_dispatch_dev() at xbb_dispatch_dev+0x274/frame 0xfffffe00d18a9d20
xbb_run_queue() at xbb_run_queue+0xbf5/frame 0xfffffe00d18a9e40
taskqueue_run_locked() at taskqueue_run_locked+0xaa/frame 0xfffffe00d18a9ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe00d18a9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d18a9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d18a9f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 0 tid 100355 ]
Stopped at      kdb_enter+0x32: movq    $0,0x12a22a3(%rip)
db> reboot
(XEN) d0v0 Unexpected vmexit: reason 3
(XEN) domain_crash called from vmx.c:4460
(XEN) Domain 0 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.16.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:
FLAGS: 0000000000000093   CONTEXT: hvm guest (d0v0)
(XEN) rax: 000000006e458498   rbx: fffffe00d35bc720   rcx: 000006d0ca72787c
(XEN) rdx: 00000000000006d0   rsi: 000006d06e44de28   rdi: 000000000007a120
(XEN) rbp: fffffe00d18a9550   rsp: fffffe00d18a9510   r8:  0000000000000000
(XEN) r9:  ffffffff81bd4c97   r10: ffffffff81bd4c97   r11: d9ff9c96919e8f00
(XEN) r12: 0000000000000004   r13: 0000000000000000   r14: 0000000000000001
(XEN) r15: ffffffff81a5b430   cr0: 000000008005003b   cr4: 00000000000606e0
(XEN) cr3: 000000000281e002   cr2: 0000000000000188
(XEN) fsb: 0000000822808080   gsb: ffffffff82a10000   gss: 0000000000000000
(XEN) ds: 003b   es: 003b   fs: 0013   gs: 001b   ss: 0028   cs: 0020
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.


From nobody Tue Jun  7 07:49:00 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1B2301BE0D0B
	for ; Tue,  7 Jun 2022 07:49:33 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHMvJ1dXmz4SXQ
	for ; Tue,  7 Jun 2022 07:49:32 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654588172;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=KmPtWEOrlSzf4MmIEVxr3fayjXLMCEZ2WytUKwC2NgA=;
  b=WQAKGCKWr3xu9ek13a3yZxdAejE8RbmZVBaygJKDHX5PuFVu+En0BKEL
   f0h7Vp+0iwSmJ6JmgFk+vXtQHltVS3+svo0KB17eBZWObJNvWtKnV1sOO
   lZSWjEncRysHcqRcKDl1M5OvJXKIoZTk08Y/N4TyLdbnFwIcyJ1aZEgLX
   A=;
X-IronPort-RemoteIP: 104.47.59.177
X-IronPort-MID: 75568168
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:tk19qK7VE6Ji0HKxJL34DgxRtGLGchMFZxGqfqrLsTDasY5as4F+v
 mBMWGyHbvuMamvzf4x1boq19khV7MeHztQwHFNsqHtjHi5G8cbLO4+Ufxz6V8+wwmwvb67FA
 +E2MISowBUcFyeEzvuVGuG96yE6j8lkf5KkYAL+EnkZqTRMFWFw0HqPp8Zj2tQy2YbhXFvU0
 T/Pi5a31GGNimYc3l08s8pvmDs31BglkGpF1rCWTakjUG72zxH5PrpGTU2CByKQrr1vNvy7X
 47+IISRpQs1yfuP5uSNyd4XemVSKlLb0JPnZnB+A8BOiTAazsA+PzpS2FPxpi67hh3Q9+2dx
 umhurS+FFh5Ga/Pw94lbENeNBknErEb6ZvudC3XXcy7lyUqclPK6tA3VgQaGNNd/ex6R2ZT6
 fYfNTYBKAiZgP67y666Te8qgdk/KM7sP8UUvXQIITPxVK56B8ycBfibo4YHg1/chegXdRraT
 9AeZjd1KgzJfjVEO0sNCYJ4l+Ct7pX6W2IF9QzF/PRni4TV5FJU94DHD8DMQNzQRsxOklyoh
 UWZ502sV3n2M/Tak1Jp6EmEjPLPhyTkcIQcHrax8rhtmlL77mIJBRAcTlf+sfS/okCkVt9VM
 EBS/TAhxZXe72SuR9j5GhG9/niNu0dGX8IKSrJlrgaQ1qDT/gCVQHAeSSJMY8Ans8lwQiE20
 liOnJXiAjkHXKCpdE9xP4y89VuaURX550dbPUfokSNtDwHfnbwO
IronPort-HdrOrdr: A9a23:YWDNfaBjjTItyhjlHegwsceALOsnbusQ8zAXPh9KJCC9I/bzqy
 nxpp8mPH/P5wr5lktQ/OxoHJPwOU80lKQFmLX5WI3PYOCIgguVxe1ZnOjfKnjbalbDH41mpN
 tdmspFebrN5DFB5K6VgTVQUexQpuVvmJrY+Ns2pE0dKT2CBZsQjTuQXW2gYzdLrUR9dOwEPa
 vZwvACiyureHwRYMj+Ln4ZX9Lbr9mOsJ79exYJCzMu9QHL1FqTmfbHOind+i1bfyJEwL8k/2
 SAuwvl5p+7u/X+7hPHzWfc47lfhdOk4NpeA86njNQTN1zX+0+VTbUkf4fHkCE+oemp5lpvuN
 7Qoy04N8A20H/VdnHdm2qZ5yDQlBIVr1Pyw16RhnXu5ebjQighNsZHjYVFNjPE9ksJprhHoe
 529lPck6ASIQLLnSz76dSNfQptjFCIrX0rlvNWp2BDULEZdKRaoeUkjQ5o+a87bWzHAb0cYa
 hT5Jm23ocXTbraVQGSgoBX+q3iYpxpdS32AXTruaSuokprdT5CvgklLfck7wk9HaIGOuZ5Dt
 v/Q9VVfZF1P7srhPFGdZA8qfXeMB28fTv8dESvHH/AKIYrf1rwlr+f2sRH2AjtQu1C8KcP
X-IronPort-AV: E=Sophos;i="5.91,283,1647316800"; 
   d="scan'208";a="75568168"
Received: from mail-dm6nam12lp2177.outbound.protection.outlook.com (HELO NAM12-DM6-obe.outbound.protection.outlook.com) ([104.47.59.177])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 07 Jun 2022 03:49:09 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=i/lqAVsunNafSn3fwclBw6ADNmQ9xuecHVabGhCG31DpVMSaGwYxkKICmHVFHOQJ54r+Nz0jduy3kZhjMhUZ+CXYVXlpaStkSz1TuCyCqr/DHfqlZ4MJ2yy9ySKrOqXE3c8kkon31pE22eK0pCSoBSdPFB2Kp0aC88+X9fzz+bEi8c1NMjuZ9AzEdJTE2m5CancJpgVrSfL7gd6HE8jnyfkEWPlqIe1WuVIAgohxM1rC0FR/VzjKsnddoW1vRXAKWIq6X9XQzwDpFSCBLZB0iyqY4B7GkjtZ0Zr8PxC1vOTEVOQT2SIc9LW5ufuGs80rvCtHirXH6w3XtHGmt/f/3w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=B/gUY4Ix0Rzo+XysLYI9cKuCV/ukyrKG+LczEb8zn/I=;
 b=HyNk/IwUiZ12nAet/sGHVX14NeIatjLjPakNLVBPIroW23zLf7tBStl95IyVeqo4EL5bukjeQnFAMIQjmExYt7eqht2osMz9A46E6NkrsM++b528JJVHcVaxx2ZEWlmkfmOb/lRkYxsQfCcGJc6zPtsBXYYYGydUjX6mKkLj72GCF4Rda/AvcLMPO/1v/B43WrghpEQRWd7Rr4MtTpqp5qeSDr/IDBXMdgFph6c10pfGYVzUzzrvY8i6PSRZ5WLJaDFcBiR+MIAizsUzpMvxK8Urs8W54LWtvm09tIL0nsZ/Ih2tB00qpnMuSN+8IMJzkm6S+BwLgBOux6q4xsHZeQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=B/gUY4Ix0Rzo+XysLYI9cKuCV/ukyrKG+LczEb8zn/I=;
 b=PQEPVq87vGH5fxLXkHqLr2GXV4HzGI+8W7Z7vmT1WAUYXB58AUCvedHTGT1aneNLC+w37ETvPv5fqUmd1MYPusqDTOz3GQLDOPok4s7N0SZeB20S3Rercku+qHH1UYhySv3niKSxJ0ktZubupTY6PAxhYNHNsGDgItemLvJsetU=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by PH0PR03MB5685.namprd03.prod.outlook.com (2603:10b6:510:42::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Tue, 7 Jun
 2022 07:49:05 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.019; Tue, 7 Jun 2022
 07:49:05 +0000
Date: Tue, 7 Jun 2022 09:49:00 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: 
 <202206070618.2576I9NI026060@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206070618.2576I9NI026060@nfbcal.org>
X-ClientProxiedBy: LNXP265CA0050.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:5d::14) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 5c9c1916-50b1-4867-7ea6-08da485a3266
X-MS-TrafficTypeDiagnostic: PH0PR03MB5685:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	AVDYm0garAioTo9QRUnjyigboOYMwDX8PokiSmFG3vAwCTaGgOokB7IiKk//BXhWSKoL1px47zDL2GtTnhi860RXMd7R5Fy8S/GvMYz0J33YfBXezmC9JDq4/NR8lHvYjirmE8LBK0DfSCTmqKTFjAHjwjBMTkX3VzwkPTqQ1Xo8acAQz+MfVsDqE6rAvtkCNKrc1RhMDAwC1f8TURFAmNkZdLQncWIYjoTHvV85qSYr/wf5Rg02BJhK/QPhfx3h40h0ou400vlb4dJHGRbe1ngABOMMQWAUK6N9riYozvKR2slvguEw7WzjL+F6LSTwzuCD0o7nRXS5eQAY/9SYiWXQLIoz3cUQaPZ1PP+7e0fydgUGdg/vnOPJQ6z5oe+vjZNXfjQPmwdUDu4XMP7pE+am7wgcOLL3smvqjkRc+rFwxFUUazPdwWXkzTK2eccTZ+zgaG/729zOUM82+NxTvkPRTn+W9nh4U9dWNWu0NwtOxKXCY4Qbn1r7rAhjrOGuDXnVOQLQEfcypFJgj/BrtAkqDJZBAYezJorCdXcErK1rsOAYdpAEzmlIm7ldCxWtFhIwVorPR7RcxD/hjj412WeG58DJqT5UaZBaM1PgdRhAg6SADe15TQIrcne+d1wWjih1DD97Hu3SAYarxu1U2w==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(66476007)(316002)(6486002)(86362001)(66946007)(186003)(66556008)(33716001)(6506007)(6666004)(85182001)(8676002)(4326008)(38100700002)(6916009)(8936002)(508600001)(82960400001)(6512007)(9686003)(2906002)(26005)(5660300002)(83380400001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?aWVGTmRZK0NlTXlIMm9Kd01LSWQ1VVBnQ1dZVWw3Tk9tdndtRUhiWXdIc053?=
 =?utf-8?B?WDRUTHE1QlZiaDVkNWlTQVp5OWdMdE94em1QUlZkdTdUT0k5ai80ZDVUaTQw?=
 =?utf-8?B?SE5hMFlmUXkzTTZpVkwyUDhWZXBGeG9aNVlXTUphUVhQbU1yODhZa0N2ZVNS?=
 =?utf-8?B?WVlwNHV1b0J4V0MyWVd5dUJ3T3JEaEIvc0JNRUJRWjQvWlUzcDR6TDVHTDVV?=
 =?utf-8?B?YnRZZ0VTYmZNSVQzRTlJZVorbkF0M2QzUm9JelNveEFSOU5EdUFHdFVackFK?=
 =?utf-8?B?TnN3N3lraGZTd0VhMzBFVW1iMGNtd0hGM0V1cnZCY2EwMmJ3K2hMMFBGWVJm?=
 =?utf-8?B?aHI1dE11YkZ5UHFIYVJpV3RyTk9FeVp3YW81WlJKOUFod05VNTlNSUtXK01y?=
 =?utf-8?B?QVBNUWJrMlQ3NXNMN2tYZTFNZGNOUGw4T0hwOVV1YjB5ZHkxOFZwQU8rQmtr?=
 =?utf-8?B?Qk8yYlVRMFJyQmNWOUdQeEpzTVRWVlZXRUMwOGVGbkl4VHQyR01PbGVHWUFI?=
 =?utf-8?B?VDMwK3hNZmtiYXl3VjRrQ0VNMkVXbll6UkhNSlQ0Mm1Ud0NTQWdCbmYxOWpB?=
 =?utf-8?B?K1FHd3lqMDRmZG03VlBRUVhJRzZvakZKd3BvckRTSkQrRTY1dGhYcGd5L29y?=
 =?utf-8?B?YXh5cEFNRnJsM1k1citwSmQxQlkvZmNiYjVEeDlhckFlVC9VVnVNQW5ZU3Jn?=
 =?utf-8?B?b3RlK0UzTWlpbks0Ujd3N055U1kyVkU1TS9vN3hGMC9NcndBemdFWDhBUFFQ?=
 =?utf-8?B?MmxUaTJqcjZmYTNTbU1XZ2NzclMyR3VydnlMa3QyMEFiS2g4Q3kxQlBsV09V?=
 =?utf-8?B?bkxVVXlBM1BMamNhRXBmNjRqTUp5ajVRUG1WK2MxdnFNN2ZYcUlBSGtacC9C?=
 =?utf-8?B?b3lDMmVMZFFvR0MwUkppaE9TcERiZ0prRXJGZnZKT0p0QWhuYjFmYlZ5MEdu?=
 =?utf-8?B?aUdsOGcxY1Urc3FjT0VkNHl5NlYvUmVCMG4rczhRNFIvZG1VZStsNjN1TzMx?=
 =?utf-8?B?bnQxUDFZN2hlYWwyamZ0bXBpUzR5MmMyS2tDV1B5NlZmYkhub0VMc3VuVW1q?=
 =?utf-8?B?WlFRWklBRVNPRlRQNk9GeEJOK1JCQWVrSlV3WG50SnB4RHJ3RktzRjhVc2JE?=
 =?utf-8?B?N3NnS0tuTXhLNEJzbXV2UklNRERLUDlnQVphTGFKN1l6R2ZiQllEV1RMNDE1?=
 =?utf-8?B?QkpNb0tGTzJzbFJ4ZGdiZHpIWEE4VGdZUFg2YWszSUpYM2FUNzN3b1ZsV21t?=
 =?utf-8?B?T1cwS0dKeXYxZDlqcXFzbC9rNnM3YmhmUDNjeW5lMEZTNE5VdFp6aWRuQWpV?=
 =?utf-8?B?d1lUWmpNZ0IrTGk4U2xXeUU4bE15V2ZEMGxvQWRSNE1tVnpkRDlzaVB2cEJy?=
 =?utf-8?B?VWsvNXlLc2M2RWlFaHQ2ckF6aDQwd255Z1FFNzJuRC9RR1ovZnVWNURSd1hu?=
 =?utf-8?B?eVlZUnpVYjNrYU9HemdKVTJnbDN6TDhXVFdzRHRxUXZLRzhjQ2FSSmN6RG1F?=
 =?utf-8?B?MWJyMG5xVWNPTTV5aDZzMGdIWW53YXFPdEpEMkNDUFU0US9TT1ZQSEdXYUF3?=
 =?utf-8?B?ODFtOHhEN2ZDUCtxbHlhR09sOER5blFMRGRPQzd4eU5CUUlZckI5VzZScUlj?=
 =?utf-8?B?c3RjTVdtbERpL3ZMbFFpczY4RURiYmlNdHc1VGI1ckk5b1NnZXljWDlqNUI4?=
 =?utf-8?B?Nk5lSUU3K3RLL0lSWXAzVTN0YzlwcHJsY2wrbGxGOEJDamJQYzYxT3V3d0lr?=
 =?utf-8?B?VnVwcFRKVTh2cW1ZUkU3YVU1NEIwN0l3ekpndnM1bjRvQ1BmOHBUU2NzbHhF?=
 =?utf-8?B?UDcvbmpOTGdqaWdOZjFsNVpYWHUxbVltOVl6elNEbkxBSHd5RGJsV2haZFNQ?=
 =?utf-8?B?NGlOZEJpRWR5Mms4ZmRDQ2MyanI5bEZhemlwSEdMTGQzbkJvN1FLdUlibjRC?=
 =?utf-8?B?cWdYRG9SbFl2WldUVXR2MXhrV1B5cnIwMlRaUEJjZjV2NVh6c2UvTytsZG4r?=
 =?utf-8?B?Q0NYSkVXZHNkM2kyc0ptRjRuVG1DYnhpYmRjMHN5c05MOTNtNXdWSmdTZkFD?=
 =?utf-8?B?U0JQenZKakJ6L2R3ZktDOU5GWm5PSVAyUkw3UTRlK0lvYVpGZzlSM2k2bjlF?=
 =?utf-8?B?dDR6VXF1MHFXL2Zua3JIc2dVTmNvdVJKWXNGNWZtVHhuV2ljWXRwTTNGYXov?=
 =?utf-8?B?dm9uaDlOcjV1azlOMEQ5MnJobHlIWWlqTFRkbjFXLzFVZHVWOEJITGJlM3Br?=
 =?utf-8?B?aS8yNmZ0dUtwNnMwMGtWLytMYkV4QTZDNm9TQWlSMXgrVU1tZldYSzBDNk1W?=
 =?utf-8?B?a1ViN29oTUNodkV3L3hxUENYK2FZVitmM010djBYVytDZ3l5M1RqQT09?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5c9c1916-50b1-4867-7ea6-08da485a3266
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2022 07:49:05.7586
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tIbmxzSFAS11pnxF/k9N7jeY9zUzUROIb5nQX2AwpYDFz/PlT3r4zuPDn2oqIeI8mCPxqRVyEgKuhF/6g24MTQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR03MB5685
X-Rspamd-Queue-Id: 4LHMvJ1dXmz4SXQ
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=WQAKGCKW;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=PQEPVq87;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1504b46c8=roger.pau@citrix.com" designates 216.71.155.144 as permitted sender) smtp.mailfrom="prvs=1504b46c8=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1504b46c8=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Mon, Jun 06, 2022 at 11:18:09PM -0700, Brian Buhrow wrote:
> 	hello.  Following up on my recent post, here is the log of the failed shutdown from
>  FreeBSD head.
> This probably doesn't tell you everything, but hopefully it will give you  a clue.
> 
> -thanks
> -Brian
> 
> 
> Jun  6 10:25:34 xen-lothlorien shutdown[1582]: reboot by buhrow:
> Stopping sshd.
> Waiting for PIDS: 1292.
> Stopping cron.
> Waiting for PIDS: 1276.
> Stopping devd.
> Waiting for PIDS: 537.
> Writing entropy file: .
> Writing early boot entropy file: .
> Terminated
> .
> Jun  6 10:25:34 xen-lothlorien syslogd: exiting on signal 15
> Waiting (max 60 seconds) for system process `vnlru' to stop... done
> Waiting (max 60 seconds) for system process `syncer' to stop...
> Syncing disks, vnodes remaining... 2 3 2 1 0 done
> All buffers synced.
> Uptime: 39m13s
> GEOM_MIRROR^Oyk^[ce gptback: provider destroyed.
> GEOM_MIRROR: Device gptswap: provider destroyed.
> GEOM_MIRROR: Device gptswap destroyed.
> GEOM_MIRROR: Device gptroot: provider destroyed.
> GEOM_MIRROR: Device gptroot destroyed.
> uhub4: detached
> uhub0: detached
> uhub2: detached
> uhub3: detached
> uhub1: detached
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x188
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff82cafe74
> stack pointer    ^O^R   = 0x28:0xfffffe00d18a9bb0
> yk^[y    nter           = 0x28:0xfffffe00d18a9bb0
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 0 (xbbd0 taskq)
> trap number             = 12
> panic: page fault
> cpuid = 0
> time = 1654536343
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d18a9970
> vpanic() at vpanic+0x151/frame 0xfffffe00d18a99c0
> panic() at panic+0x43/frame 0xfffffe00d18a9a20
> trap_fatal() at trap_fatal+0x387/frame 0xfffffe00d18a9a80
> trap_pfault() at trap_pfault+0xab/frame 0xfffffe00d18a9ae0
> calltrap() at calltrap+0x8/frame 0xfffffe00d18a9ae0
> --- trap 0xc, rip = 0xffffffff82cafe74, rsp = 0xfffffe00d18a9bb0, rbp = 0xfffffe00d18a9bb0 ---
> dmu_objset_zil() at dmu_objset_zil+0x4/frame 0xfffffe00d18a9bb0
> zil_open() at zil_open+0xf/frame 0xfffffe00d18a9bd0
> 
> l_ensure_zilog() at zvol_ensure_zilog+0xf1/frame 0xfffffe00d18a9bf0
> zvol_geom_bio_strategy() at zvol_geom_bio_strategy+0x90/frame 0xfffffe00d18a9c70
> xbb_dispatch_dev() at xbb_dispatch_dev+0x274/frame 0xfffffe00d18a9d20
> xbb_run_queue() at xbb_run_queue+0xbf5/frame 0xfffffe00d18a9e40

This seems to be caused by a stale blkback thread that's still active
after the underlying device has been shut down.

Can you provide a bit more information about how to reproduce the
issue?

I assume you are shutting down with at least a guest still active and
doing IO?

Which command are you using to reboot the system?

Thanks, Roger.

From nobody Tue Jun  7 07:57:39 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A5C001BE136B
	for ; Tue,  7 Jun 2022 07:57:55 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHN4v6L5Dz4T0L
	for ; Tue,  7 Jun 2022 07:57:50 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 2577vdAH004820
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Tue, 7 Jun 2022 00:57:39 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 2577vdw8005328;
	Tue, 7 Jun 2022 00:57:39 -0700 (PDT)
Message-Id: <202206070757.2577vdw8005328@nfbcal.org>
From: Brian Buhrow 
Date: Tue, 7 Jun 2022 00:57:39 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Tue, 07 Jun 2022 00:57:40 -0700 (PDT)
X-Rspamd-Queue-Id: 4LHN4v6L5Dz4T0L
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-2.42 / 15.00];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 MID_RHS_MATCH_FROM(0.00)[];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_DN_SOME(0.00)[];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-0.12)[-0.118];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  I shut the system down with:
shutdown -r.
I notice that, compared to NetBSD, FreeBSD shuts down very quickly.  I'm also aware that the
NetBSD domu's don't seem to know anything about the xenbus shutdown notifications they receive
from xen.  I wonder if I need to generate some kind of ACPI notification instead?
On the other hand, I'm not sure the FreeBSD domu I have running shuts down cleanly either.

But, couldn't the xen daemons pause the system shutdown process long enough to do a destroy on
any domu's that don't respond and then close things cleanly so at least the dom0 doesn't panic
on the way down?
-thanks
-Brian


From nobody Tue Jun  7 08:05:51 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 493091BE2063
	for ; Tue,  7 Jun 2022 08:06:06 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHNGN6tSsz4TPN
	for ; Tue,  7 Jun 2022 08:06:04 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654589164;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=Q7Di0c0jVNbzKQ+3s9ARm9tH8R06VG87P0X8CcT+Y3g=;
  b=Gdg6Z2Lpdg3Hu3oaU5UBDjqZtSBhrEwQYyxWi1P7lRkOAVRym1APHHkJ
   BsQVnPx2yw+mbyu0HFikNdxzGfhwCvA+s7BxKYVyPEClbQzS0iCyLmr+8
   V8OAHap0CnznYVXukPYXhjg6xJUMPkpfCh876nAbJbgNZ499oeFjvRsov
   o=;
X-IronPort-RemoteIP: 104.47.59.172
X-IronPort-MID: 72372265
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:Pf2bCqpR/3cphQd6PNV1Jg/JxX5eBmJDZBIvgKrLsJaIsI4StFCzt
 garIBmGb6veYDP9L98gadu3oB5QvJLcm4M2GwJu/i1nFX9GpZuZCYyVIHmrMnLJJKUvbq7GA
 +byyDXkBJppJpMJjk71atANlVEliefQAOCU5NfsYkidfyc9IMsaoU8lyrdRbrJA24DjWVvT4
 Yyq+qUzBXf+s9JKGjNMg068gEsHUMTa4Fv0aXRnOJinFHeH/5UkJMp3yZOZdhMUcaENdgKOf
 M7RzanRw4/s10xF5uVJMFrMWhZirrb6ZWBig5fNMkSoqkAqSicais7XOBeAAKv+Zvrgc91Zk
 b1wWZKMpQgBP53psvsccSNiPw5YPJAX1ZmWeV6zrpnGp6HGWyOEL/RGKmgTZNdd1sMpRGZE+
 LofNSwHaQ2Fi6Su2rWnR+Jwh8Mlas72IIcYvXImxjbcZRokacmbH+OWupkFjXFp2Zom8fX2P
 qL1bRJ1axvNeVtXM0o/A5Mihua4wHL4dlW0rXrK//FouTKNkWSd1pDnPP3HZd67aP5coUeW+
 07prmfiMkgVYYn3JT2ttyjEavX0tSnmWJgVC5W++PRhiVnVwXYcYDUYT1a2q+K1z16zXfpRM
 UEZ8DEi668o+ySWosLVWhS5pDuIu0AaUt8JSekisljSkOzT/hqTAXUCQnhZcts6ucQqRDssk
 FiUg9fuAj8pu7qQIZ6AyoqpQfqJEXB9BQc/ieUsF2PpP/GLTFkPsy/y
IronPort-HdrOrdr: A9a23:rGzqGK+4NOHB5J/vIhluk+FEdb1zdoMgy1knxilNoENuH/Bwxv
 rFoB1E73TJYVYqN03IV+rwXZVoZUmsjaKdgLNhRItKOTOLhILGFuFfBOfZsl7d8mjFh5VgPM
 RbAtRD4b/LfD9HZK/BiWHXcurIguP3lpxA7d2uskuFJjsaD52IgT0JaDpyRSZNNXN77NcCZe
 yhz/sCgwDlVWUcb8y9CHVAd+/fp+fTnJajRRIdHRYo5CSHkDvtsdfBYlCl9yZbdwkK7aYp8G
 DDnQC8zqK/s8ujwhuZ82PI9ZxZlPbo19MGLs2Rjco+LCnql2+TFc1ccozHmApwjPCk6V4snt
 WJixA8P/5r43eURW2xqQuF4XiU7B8er1vZjXOIi3rqpsL0ABggDdBauI5fehzFr2I9odBVys
 twri6knqsSKSmFsDX25tDOWR0vvFGzu2AenekaiGEaeZcCaYVWsZcU8CpuYdo99RrBmc4a+d
 RVfYDhDK48SyLbU5mZhBgk/DWUZAV9Iv/cKXJy+fB80FBt7QJEJgUjtY4id0w7hewAoql/lp
 v525tT5cBzp+8tHNZA7bQ6MLyK4lKke2O9DEuiZXLaKYogB1Xh77bK3ZRd3pDYRHVP9up4pK
 j8
X-IronPort-AV: E=Sophos;i="5.91,283,1647316800"; 
   d="scan'208";a="72372265"
Received: from mail-dm6nam12lp2172.outbound.protection.outlook.com (HELO NAM12-DM6-obe.outbound.protection.outlook.com) ([104.47.59.172])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 07 Jun 2022 04:05:57 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=IF4BoKoHzuMUEaQKoHROtn5sfCXPuhZiKj1mpoxpt8CnUpR9QEVoLHBinXOTZiOR66CSQEmoUrh0yvXbaYbCGDwbHcmumRjf5R29Hq7BnjK5rAo9M0jZA5fkt9KIx736F/JWDlNn5ruCd/0VwQHJ3ITAopuHWdBiIoBi3ofTvwZ3/IZf1T9Xz5qBJWBGbj4a6ElNwmBAlo5xlRXV5EeZHGcWF2OcK5DwXn/V2VBkMkriNHaBgz4y4KyuWxWU7srb621KUKxumZTi2pi+LefUJqfnIOQ3UoE+rRoszdN8ZYPh2OGjzyAG8947x/pnUxyn9tICahUVFunERp38Vb2I+g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=lWdF9Zz6j49bFfnPtcsssiQ0VcMjHffc/S4S0rY5p+M=;
 b=bM6O76dtbRcwjJYGrxDgjMHHeYsTIaWjpyQEEbvy0NS5H1Ho7LsDj9S3ZJX8yzoDbOMyVYACMzq81rtW7CnMY7V6HgFgZ/ub9Pq4piGbes2lHvSpbCfQZjM0XTK5COawxtKduKU05x9n8ztw/PDSFJ7ZysK+y5hS4mz0wOKGLyZVzd8TkTrIbNo3zpRiOaLgCnaPY5EVudWSSR2skjIoitvi3DCru1ghYZmFuMdOsU1jVPXtGSNN4kkmatoMzibsGDPQM9wdWgeUiwVYV370b+6JpsSsrzIQmM7xqhkK3pMHcUkmpG3YXlQc73Ra/skH3uXHoB63iq1imp3y8cDP5A==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=lWdF9Zz6j49bFfnPtcsssiQ0VcMjHffc/S4S0rY5p+M=;
 b=xQtv0f/RHB+ru+gmuZILRHCcij6WKbk4r5nJsC+8lAYjt54jYiUwO/tX+Gso2ynJWt7PwAz9XpUNiHWJJ9YtXT1g9asvPsvXisxyQXA4LuWAKczxnT7eqzAYWFtTzmnbGKrQ7GAF8NBA42GDDwvLSHHf6XdOcl0xek4Xy4qQgMY=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by PH7PR03MB6995.namprd03.prod.outlook.com (2603:10b6:510:12f::6) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.19; Tue, 7 Jun
 2022 08:05:55 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.019; Tue, 7 Jun 2022
 08:05:55 +0000
Date: Tue, 7 Jun 2022 10:05:51 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: 
 <202206070757.2577vdw8005328@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206070757.2577vdw8005328@nfbcal.org>
X-ClientProxiedBy: LO4P123CA0442.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:1a9::15) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: d906ef71-afeb-42b6-2a65-08da485c8c08
X-MS-TrafficTypeDiagnostic: PH7PR03MB6995:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	hU6wDjtlDS1AdaRgKskMJY17qcEjyvo3CZ3/aZfCNIi3asOOFXKgscsWymiPpIvs9J01GRmUWaMdqcTiB0E2EFaa+PP0//eQSoKMEYtKiJuyT+t1v9GL3UHstt8A9conoW4HJ2QW8XV1ccY+wiRfalbVOVeS5aXnUICfhc0t8CK+bzl6brySh9+PIg3VE6PmPBvwHNBe5MLMpK6xKXEofM3/25vhPahhS71tygzIJhc6jFXxPmcBGMy5D8JZXOtzW7Zl7gOQO6gFw54a3P04lttD/KJ9WazUTe5dcGKUw5oQkmXNCydCoNaGijAHuD1OLa+i4pQbnMBUc9MaN6AzPlIhtTrddn1Cw46skxE+sSV7MTO0/wtqFKLPDeBeQjod8ZmVpmzzSF6d8SmOlHuX0oKhPysmJArJqQm9slw3vMbBikJgkej+w0NgUOVPACAmcbMt7t6jbvMbTlQCe35Qw5f34IdZuv8rRcRHmad10CoeMrztZqvROW+bckyqW9ylbZLudqvMfNoxWKPhtsALn8tun6O7jDcoU4g5lu1if08OXg55uJ2nZjndR0gENQcMOSQySfGyttEkrsfcR4hoG8/c7cmQ0ow02jCtS7vkgsvI/AcKExg63cdJVuZxmy7wXrn4aOV8n3ofnNcaCime0Cm91wcMnS/wooi0w1kVk0HPIiZKh6pRU9lUd8vpR3YWPas7VrXmO6aJUM5oPRXjzQ5I3kA1ER0xqrHYQ0KHzBDarU0evih2rNn0wfs4U6DBiHn5Zsz2PUMBF2Wddx61yw==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(6486002)(966005)(9686003)(26005)(6506007)(508600001)(6512007)(186003)(82960400001)(8936002)(86362001)(4326008)(33716001)(83380400001)(8676002)(5660300002)(66476007)(66556008)(66946007)(6666004)(2906002)(85182001)(6916009)(38100700002)(316002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?UW5wQW9WeDBGVFUySDdhcXloZTN2WUE2b0k3ZlFWdmI1VmhRNjR3cXRvaGpQ?=
 =?utf-8?B?SG1RaVVNcVAxZ2xPb0Z5TGZ1SmlqSXhqRFF1SUhYc1JEMlJiZjBTTzRJUmNP?=
 =?utf-8?B?UFM1YmdvaGhTajJ5bGFBOVRhc1MxdURscG5lbjZ0R21WakZDYWsvYzU4K25L?=
 =?utf-8?B?N29zSTI2WkZIMUZsOHA1WlhqVXdpYkhGeWVxalo0eEVJOXVXbFFrbUtjb3Ur?=
 =?utf-8?B?M1ZWdGEvdk13VjFpbERtSmR5WG9YUFRjYm5FY2RyRzhaRUF3emhzTWtKZGNW?=
 =?utf-8?B?U25Qa2I4M3VYT2RFbGNPQkRYMms2WE4wMnNnN1RmWlRkRE1BUnc1Nm9LREo1?=
 =?utf-8?B?b3RxNzIxK2pUb2hrY29xYSsvWGhEQi9CZzErZXRVcjlnK0VQTzJZRlFOTE1p?=
 =?utf-8?B?cnk1R1JPcHZ4dDEvdmJHRFg2a2tnYk9mTzFtbG84ZEI0QmEvRzhJcjRzUEVj?=
 =?utf-8?B?SnhBZ00zUVFxZWRwWGw4a3NLV2tIRlM5c1gwdCsyZFgzNmYvYll1QkdZMTFW?=
 =?utf-8?B?RGdKUHpIUjhYUk9pM2lmcVFKcEJDN29IZ21PV0xPS1VFK2V6RWczTGM2eVlV?=
 =?utf-8?B?NDltcjdETWFtcXRiWlZCblJ1aHJ4TEFwcEhSTEV5NWhTbGhDZjVDbnB0MXBV?=
 =?utf-8?B?VEFGZVVqbTZOdG9QVGJoNDQwbTRtRm1XUGRQbUVaa3lSRjFxN0x6WExmU1JB?=
 =?utf-8?B?RjlKNnVoc1pIMnFmWlR4bFRJWVhFQm01M2dVVTkxbk9CMGczcGRFQlhXbGR1?=
 =?utf-8?B?UXhyc2hlYTU0STRJWVc0SDl5bVc3VnErV3A5WDYxOHdoV29sdGtrQ0lnRnBF?=
 =?utf-8?B?TFVya0J4R2tUb1k2S3ZVUTBtSXBiMldqOEZId2FBTmpGYmN6YUVockRMbUNI?=
 =?utf-8?B?R3g4R29NYnduT3p3bHNlREJkMGxtNXdadnBqcUFucE9HelRoOVNYS3NTaHZS?=
 =?utf-8?B?ZFNZZ0FFY1ZhenRraUdybUZWUmIrb2tvY1lab3c5Q0J5OHhSM0dGdmhlSkZT?=
 =?utf-8?B?OE5MMEgxS3c4T2ZEdUpxdDZxTkwwaTkwdXhKQTBYbkp0bzh2TnpSblVCeHB0?=
 =?utf-8?B?VnNISGFJdU9iL1l1VDlPakRUS3E1d1NWckxVSndGcWtyL003cEVRS2Q0NEJN?=
 =?utf-8?B?S254N3ZGekk5azVveS8rcTd4cUFkQ3UxbU1xTzU4clhDSGdxb2hKT2wzSHpZ?=
 =?utf-8?B?cjJPUVFsclQzQ2ZhZnd5RDMyaElxNDAzaWIxSFZzK3M4N2N4dW8wZ0hERVYr?=
 =?utf-8?B?d3lXZVRXNGhMSERrMUZRM01pMUNWQTA3QkZROU54amhsRERLZXV3aUhhY3hT?=
 =?utf-8?B?aWU1VVhRSFNUcExsOXNrNVA0L0U4RGZMcWxtcjNQRlB6dFZFOE9jTFEvaWJX?=
 =?utf-8?B?QXlFVlIrclpmRzJYYVJZblJ1V3Q4N21yanVyQmM0eUkrSFlqcnZnazQ5Tlkx?=
 =?utf-8?B?VFJBSGwvLzJUM012WDQ4cVlvMU9TOUJRb2wwQm8xRTZuY24wM29oQUFUSUxM?=
 =?utf-8?B?V2ZSUDJDbHlRdXQ1a05VV0VIc29kS0NHWEo1cFlEYUtyejYxeFNzTFdFRDUx?=
 =?utf-8?B?WFZLWGYxYVlCQXVTbFpzR0dqaHlIaTJacElIbkE5eCtKazhVYjZxZzV3SlpN?=
 =?utf-8?B?MVM1WFJRS1Jhb2xLL1FnOWZkbVU0VDlMcUxmZGtNKzNrcjh4WHMwYllLcUo0?=
 =?utf-8?B?YU5nbjFlbWFhQlpueDlSNExhZUUwZ3QwSmRiVFB6bUJtTmQ1NXFOMEYxakpK?=
 =?utf-8?B?Q3NOalUvNlRId3hpZXlyRjdRZXdsdTFsUmdtSVFxdUc1djM4T1BPakxPTkt4?=
 =?utf-8?B?b2I5dXF3UlI5T3hiV3pGMUthYU5jdS85QTk4Q1pERngyZTF4YmVlQjllTitX?=
 =?utf-8?B?L1FIUTJPTnI2ZmFsVEY5MDhQZTh4MGJTN2xlSkpQSWd3MlNTT01nRnYxaUpB?=
 =?utf-8?B?MlJmY3QySnFQQS9BVm5TZEZ5Uk1kWmk0MXFSNnZidWJuZlF5U2FOQnhtMUZa?=
 =?utf-8?B?K0kwQW9HQk5LdkJEQndYWldyL2ZWb250TGRyeEV1YldXNThIOW4rQitxUUlL?=
 =?utf-8?B?VkxOaWNvZ3RzanM5Q2VtQlMwMXVhdGhKVGFMMlR6YnFQSTh5QTFoOWhkL3d5?=
 =?utf-8?B?S09wUDh4cWp1WTZWOUVRT3dXQXBlSnFpNGNLRW5nSVdPczI3UnZUOGNVWEYx?=
 =?utf-8?B?WU9McUk3L0EzdTlldzRiNkRMMHlqZjBQMlB0allZZjIwRnVHRE1YWmllejdh?=
 =?utf-8?B?cWF6V0dDVkhjRkRaUmluWEdrU2YvM0JXQ1hkdEZZVHpwRDBTTWtvZ3dlMnVv?=
 =?utf-8?B?SXFnSHpCRU15c0tBYVpjZ2ZGYWZ6MFo5YzdqT3R6NDdrcmhTM0d3Vk5WaGZu?=
 =?utf-8?Q?HqdVw+nBi0hrTQSE=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d906ef71-afeb-42b6-2a65-08da485c8c08
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2022 08:05:55.1926
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CUn/3ekc9yUMQPw1yWPgnXkX9KSNiu9txwH/NoGll5XUzG4VoBqIbs76ZZS/0+2212cvtKQ+fa0EgZR5Xv3+Vw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR03MB6995
X-Rspamd-Queue-Id: 4LHNGN6tSsz4TPN
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=Gdg6Z2Lp;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b="xQtv0f/R";
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1504b46c8=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=1504b46c8=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1504b46c8=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Tue, Jun 07, 2022 at 12:57:39AM -0700, Brian Buhrow wrote:
> 	hello.  I shut the system down with:
> shutdown -r.
> I notice that, compared to NetBSD, FreeBSD shuts down very quickly.  I'm also aware that the
> NetBSD domu's don't seem to know anything about the xenbus shutdown notifications they receive
> from xen.  I wonder if I need to generate some kind of ACPI notification instead?
> On the other hand, I'm not sure the FreeBSD domu I have running shuts down cleanly either.
> 
> But, couldn't the xen daemons pause the system shutdown process long enough to do a destroy on
> any domu's that don't respond and then close things cleanly so at least the dom0 doesn't panic
> on the way down?

Yes, I think we are missing a xendomains init script, see as examples:

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/hotplug/Linux/init.d/xendomains.in;h=dcd330cb8d5df137b493848bcde0b86f62c4acb1;hb=HEAD
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/hotplug/NetBSD/rc.d/xendomains;h=edc4495317e7c97f77d295f3068647b07a871f5a;hb=HEAD

That shuts down (or destroys) all guests when dom0 shutdowns.
Regardless of this dom0 should not panic when shutting down with
active guests.

I will try to reproduce the issue on my end so that dom0 kernel
doesn't panic.

Would you like to try adding a xendoomain init script to FreeBSD?

Thanks, Roger.

From nobody Tue Jun  7 09:32:57 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1BDF11BEBE1D
	for ; Tue,  7 Jun 2022 09:33:08 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHQBq2lTGz4db7
	for ; Tue,  7 Jun 2022 09:33:07 +0000 (UTC)
	(envelope-from prvs=1504b46c8=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654594387;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=NOJWPFFHRHiWtaVvWHMrTOW2FRkFbvwesMOTKlrDA8o=;
  b=XihkpgEQGDEK66MbDxFjbQqUP4Qk0XAHlZlZpHn9QtdQzA7FgH1TGrvT
   Kkx524nQs/xiN5eA3aUqCNhnCieSoXzjKEWTmtTsbqlsmHcz04Ql1bEGD
   UaSjWCsZwdnmLxE/YAe2/cyzUCb/7Exm7zkqMOvVJQHq+CD05g74j9vc1
   Q=;
X-IronPort-RemoteIP: 104.47.66.45
X-IronPort-MID: 75575478
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:erZGhK0S0AwBf+pCu/bD5Ytwkn2cJEfYwER7XKvMYLTBsI5bpzQEn
 2UYUG/Vb67eajOncox+OoTio0gGvJWHmN5gQFE5pC1hF35El5HIVI+TRqvS04J+DSFhoGZPt
 Zh2hgzodZhsJpPkjk7xdOCn9xGQ7InQLlbGILes1htZGEk1EU/NtTo5w7Rj2tAw3oDga++wk
 YiaT/P3aQfNNwFcagr424rbwP+4lK2v0N+wlgVWicFj5DcypVFMZH4sDfjZw0/DaptVBoaHq
 9Prl9lVyI97EyAFUbtJmp6jGqEDryW70QKm0hK6UID66vROS7BbPg/W+5PwZG8O4whlkeydx
 /0ck5i8VAoTLJHS2+gWTTBiTnlmEINvreqvzXiX6aR/zmXgWl61mbBLMxtzOocVvOFqHWtJ6
 PoUbigXaQyOjP63x7T9TfRwgsMkL4/gO4Z3VnNIlGmFS6p5B86dBfmXjTNb9G5YasRmB/HRa
 tBfcTNyRB/BfwdOKhEcD5dWcOKA2SSiLGAE9wL9Sawfx0bC4gZMiufXAYDwJty1V91wxWLGq
 TeTl4j+KlRAXDCF8hKP73K2gfDnmy3xU4sWUrqi+ZZCjkeey2YIBFgGWF2Tr+O0h0SlV5RYM
 UN8x8Y1ha079UjuS8anWRS9+SSApkREB4cWFPAm4gaQzKaS+xyeGmUPUj9GbpohqdMyQjspk
 FSOmrsFGABSjVFcclrFnp/8kN94EXF9wbMqDcPccTY43g==
IronPort-HdrOrdr: A9a23:Kzs3zKNiUywKW8BcT1P155DYdb4zR+YMi2TDiHoddfUFSKalfp
 6V98jztSWatN/eYgBEpTmlAtj5fZq6z+8P3WBxB8baYOCCggeVxe5ZjbcKrweQeBEWs9Qtr5
 uIEJIOd+EYb2IK6voSiTPQe7hA/DDEytHPuQ639QYQcegAUdAF0+4WMHf4LqUgLzM2eKbRWa
 Dsr/Zvln6FQzA6f867Dn4KU6zqoMDKrovvZVojCwQ84AeDoDu04PqieiLolSs2Yndq+/MP4G
 LFmwv26uGKtOy68AbV0yv2445NkNXs59NfDIini9QTKB/rlgG0Db4RE4GqjXQQmqWC+VwqmN
 7Dr1MJONly0WrYeiWPrR7ky2DboUITwk6n7WXdrWrooMT/Sj5/IdFGn5hlfhzQ7FdllM1g0Y
 pQtljp+KZ/PFflpmDQ9tLIXxZlmg6funw5i9MeiHRZTM83dKJRl4oC50lYea1wUB4S0LpXUd
 WGMfuspMq/KTihHjPkVyhUsZGRt00Ib1m7qhNogL3W79BU9EoJunfwivZv20voz6hNOqWs19
 60TJiAq4s+PvP+TZgNc9vpEvHHfFAkf3r3QRGvCGWiMp07EFTwjLOyyIkJxYiRCe41Jd0J6d
 78bG8=
X-IronPort-AV: E=Sophos;i="5.91,283,1647316800"; 
   d="scan'208";a="75575478"
Received: from mail-mw2nam12lp2045.outbound.protection.outlook.com (HELO NAM12-MW2-obe.outbound.protection.outlook.com) ([104.47.66.45])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 07 Jun 2022 05:33:03 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=JPDrK4di2aWe+Bbd2te1TUW21m8gQs9pFTn1BDO2Sv15EdjYrGWAs6ReuXf3hdD0SeUaqOV2h87dXhKIGkEIhl7X5TduDou6udLJqvxdUQO3tf5FUMTJBP/4/7cLjMenvxmtY75l9PC9oIrjZ4OEUb5VKKVE+j/5BcStsd/vzGlRhInNCjg1P0N3rQDicp6Iz1GxrOVdBe/9Z23tC1d4WeJz/+RTU7wr/zP8vu3XYSLpN4jDVq8uaaVQdOVCT+Va3WusQ6AIKvZjf9MskXTs+xG5ly+Bn1frTtSe4BEUfWlwP3hTIm9ALSG8bSV/YCqGYJ1TqOcXkMFKV+Oj1LkKUw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=9+1Ra+FOZDIWVqIkVpTWDOtTkC95Sar2Bq8DUWBTSJE=;
 b=fKgD2dC4NHEBJ4T38kV+wm3HCNK4NCySfPrdEglpz2yXD3oNZToTXGpy0j6MaHdlOyqHcrUHTXu1QMg+7t0mKQUIF6GrrR/1CbwVBVVGd+vLNM0/DYT0xejQS0aMMGJRtLok3wAYziZRVWgsiaWw6YRdicjZBOdo/lxR8gOmNahDXwa18uY+KzQ37ZMwZJv50Uw5fkarKhWScqBuYbqbVAP7WJs/g00YH64BJKHlQCDo3Z9doRXtcGuDA//+BcoEl9Ri0/epOZTCDzgt8Na2Qh2D96M3dAqIlbDBPeOK0uO19F4fa8Dzxq/tthYJLqLQH5wpmHqoU2Bf6IDHx6QUcQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=9+1Ra+FOZDIWVqIkVpTWDOtTkC95Sar2Bq8DUWBTSJE=;
 b=cMeLuQfAJyv54otWE6LzRYzYC9bRR+f6wJh5QamWHBcn2zwNDT5P0MLYsutwF72LpXSSTZykP6orUolN/n9Hmlkv+oseUKoB7jO6pWNOr9CdScoioMc9KKPnNa+zN1q9CSBZCXdlJVe3gHY4b+uAY8wJgY7PKo+7IJf3TeQ0lJw=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by MW4PR03MB6441.namprd03.prod.outlook.com (2603:10b6:303:120::11) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.19; Tue, 7 Jun
 2022 09:33:01 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.019; Tue, 7 Jun 2022
 09:33:01 +0000
Date: Tue, 7 Jun 2022 11:32:57 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: <202206042003.254K3uV1016135@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <202206042003.254K3uV1016135@nfbcal.org>
X-ClientProxiedBy: PR1P264CA0056.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:102:2ca::6) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: d896b021-18a4-494c-bb47-08da4868b756
X-MS-TrafficTypeDiagnostic: MW4PR03MB6441:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	TnQEDaWrDAZKwD4TIY4J4KcwjqA3k2cC2rA2e7wjxHoTZ1B/WyENBB39uozmyXRKTY9D5J/MmKAzHiuwYg1KN7mU7tnrSABIZbEYcKKDclaBcnrXt9ZgaB6ZZ83u665qANN4qWldu/taAUFL5Ed9DVPpL1x8rx3SynC7ugJXnYkmo5w/zGw7VdowAIzkmWXD49/ZwKa7tU3q73psbVuGG0dVzNzfUOSSf7x6naWDyI883oT308ziaCRUK29F+YsNKx5th0mxdmMppHU3DhRvD/JTCFb7SoN/3Dw1YMJwbeNSYG6ZX2uZc/gbC7A9VKwVRqcx+IeXd2mSk9Ua1DYNDWPW9gTI+k5bpM8RiYoXrVbiD99sWNyE3/NFQSUsHZD6wAUdvmkOiRsd1h8fLfky8NLYsXgKoLMkx1pKka0JylYvbq4AzSmEszpse8TbbTq1VyKNPvGVpPn9UPMCbQzhurnvVvy9RzNCk5e0uH/HjZs3poOHw2BGfuw6gIJ+aUWSCInIr/pr4l4842UtbXqGtbk0/bBc87L+iPCsEFGNJQj5wiAlL4w9l8pmFHNiipYktVEyYbmCjMwEvY8v+OkrlJV1M5pMgrh9lc9GgxaxdmNiYFEmxL2eaTPM0rJs7kbyp9Zedhw3FPthVCQQGcTOyQ==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(2906002)(86362001)(82960400001)(8936002)(6486002)(5660300002)(508600001)(83380400001)(186003)(6916009)(85182001)(38100700002)(316002)(6666004)(4326008)(30864003)(26005)(6506007)(9686003)(6512007)(33716001)(8676002)(66556008)(66476007)(66946007);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?RG9NVzRFMEdvM3l6cUFJYnE5UDlRaUQ3TjdlbFhBRWxmamRRYmNLZWdUalVT?=
 =?utf-8?B?bHlhalcyZlJmYUFVbExabFpud3JHbGo0QVZ2VmIxTktuSVNmVVNJeHdpSkhS?=
 =?utf-8?B?YVExdVFpWGI1eVpuckNMWUxvZVJLT2JIMi9Nb0E5TG1nOU15K3NkaEprc2tM?=
 =?utf-8?B?Q1NaeFQ1NUdNSmlPODVnelZrbWRYN2hPMXZmTi8xTDdRUkRQQ2ZDRmhlRFV5?=
 =?utf-8?B?ZDR0cXMrcVRDS1BMUy83SzNMVGRWbGMvTFlZNmIyYkxYbDkwSFF6TGJKWUts?=
 =?utf-8?B?c0ZnWXdiSUFvN3ZabTFrbFRkVHB0NEJyUXhGRkF5UEQ3Rlh3MFZaT0pFRWxl?=
 =?utf-8?B?cDhQMS9tSXJPNmdKM3NTWFpKQm5tM1JUVDRRVzRJdGpEWjNiMTAzOTV5bElG?=
 =?utf-8?B?QmZoZjNBRnpBVU1pZGxWdHN4Qlh4cmhCZ3dyZmd3bHZsVmhvdyttL0dKS1Nk?=
 =?utf-8?B?U3dhRm1uOWFIRXBuMjVIM3REMXRJZGM0Y2cwN0l3Skt1NFRoV1hPOWszZ2x1?=
 =?utf-8?B?aWtwMVBrMFU5TEtGbWZvOWx1dnliL3orcERBcWtXMmdPQitZL21nODVJeDVJ?=
 =?utf-8?B?bG0xRWpzT2QwakRhTlNWVmZRdG5haVVVS01YRzQ2RnFHWW00a3pzdGg1VzlP?=
 =?utf-8?B?MERvMHRTUGlzampIRDczMUJmSFlmWkFWSjluSU9PNC9QSWRKWnVtanp1R0hO?=
 =?utf-8?B?WCs0aXYxTXZ5alRNdXlLWTlVMmJKR2o4UzJwcXpqZ1dMR0RrSVBJU0J2alVC?=
 =?utf-8?B?RERSRzYxQmhpTVFLbVRJQ24yY1VzdHhIWnpreHphVWNKZnk5eDkwU3lYZ2tM?=
 =?utf-8?B?YXdtTUJPUXZFakFiL05HcmltM3lYN29OdWRvWFNPV0o5TmV0d3hISlRyRitu?=
 =?utf-8?B?YWtYRHU5UFZuNE5HMHkwb1B3OEdzWUtIUFp6WmphMXk2em1mOHIyTHlIRDdD?=
 =?utf-8?B?K29lenQ4NFEycWMzeFJTZUFFck5rZEIrVit5MVZpTG9sbW1sRjhBNXVyNTNk?=
 =?utf-8?B?V2NNUEVXR3Zoa0ZjWWFNZVBCcmVhbzFNZ2NZTUtVY1dWTm9wTjZ3R3BpVUxv?=
 =?utf-8?B?dXZJNWYrbnJIZmpkUjdRYWNFSzNWam91ZWhOa0NsWDI4QjRJTGpZWElzQkwr?=
 =?utf-8?B?TXE0bC9CMTQ0d05DYk5wVzF3ajdCSlV1Rm9xUlN0QnMyd1Fkei9yL3lLS2Jx?=
 =?utf-8?B?UjlnYnp2R0lHTEw0VHdjMU0rZndNMkphMm1tbDhyQnF5WDN5N2luZVVEblVM?=
 =?utf-8?B?bUE2Y2xLR3I0MTBIcWVGOXc4UnZEUmI0dkpnYjhPUTRBZUdkNUdwczJPdHNN?=
 =?utf-8?B?SUVYL0xnR2dsRDZPRFlvVlBhTDBsOERvU2VFcVFaNnVwVVJuTmNRKzBaR0Zq?=
 =?utf-8?B?Sjl0WEFTTk12TGpuWWl3bnBGMzV4THdVNDl4STRHSWJYQk5GS0U0M2pDYkx0?=
 =?utf-8?B?NGxDSDQrTGgzN2RTbVZLSE92cTREMER4bWdUUjFDTStSb0ZKYmlHQzl4SVBH?=
 =?utf-8?B?dUdPTEJoY3NueXM3T0Y3WStmelM2Wk1DeDVBK21BY3VOTzBoSkVEdGNBcFZV?=
 =?utf-8?B?VHJMRXp4U2ppM3FUcjNXRGIycWF2cVhCbUNNUTFVdHRWNlZKWXROSjdVdGs5?=
 =?utf-8?B?U1U2N1RKd0ZNTjhLOUlrYVNGbk9wUVV0N0FPZVBQbkU2MDBwNm1tWWRCdTVw?=
 =?utf-8?B?TXQ2RlpsQ096OExram55c2ErRFl5M2l5SC80WWV2S08zK1hINXFTcVpPdExW?=
 =?utf-8?B?ZVhiaXhvS0hLZEh2WUtIT0xlanFucWlrNnhKWEhVRVdmRmZ3SFVzcnlYaHhF?=
 =?utf-8?B?dVdxOWhVUGg3ZjJlalgwUlNWOHBBa2tKbjVSbUtvd1U3TU1BNkZDSUI1aWF4?=
 =?utf-8?B?RGFUM0VlNFZRSkV2cjlSWG1UTGo0eFkxWmlTR2lndm5aU0R4VVc2NDFjcFNi?=
 =?utf-8?B?c3Bsc1JBV28vckRUZEY0L3NPMU1wY2lseHZiM25UcTNXQXFielB1RXUyUzYv?=
 =?utf-8?B?and2SUp6V0thcG1HTnlraTVSS1kyd2N4K0lDNFgxakI2aXZPbExUb3dTRGor?=
 =?utf-8?B?V0RVWExzZDJKbnNzK2ZST0N5Yy9HZjR4cHJ4UGZPWDhvWGdCWWNaQnBtS3ha?=
 =?utf-8?B?RnpNWEhhQnlDblNSVmYvUnF1ZHlyOVA2N2N4cXhQQTRwRnl3enA4d01EbWlP?=
 =?utf-8?B?RFRUbXBlK2dpZ3QxbDRnMFd2cGhocXh5Tk13NjlqL0Q5SGV1aVZFTTVVY2dr?=
 =?utf-8?B?R0JSS3dQQlNadWc4NnhMK0ZpMFcxczBKVnNMQ0VUbFR2bWtQS0xmS2RPMldC?=
 =?utf-8?B?c2VBU3pjdW1iRWs3U3VjWkUyUDQ0VHI5R1lMOWlzNk5McHpuUVpBRXY1ZTc4?=
 =?utf-8?Q?jfUA2iNYE3JJo7q4=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d896b021-18a4-494c-bb47-08da4868b756
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2022 09:33:01.7621
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZF0hWKaAwOSI7pUZ8cE+o9UMYSG+lvv5AtLCceBET1CeO+J1UJFwVimJSTGq1yggTXdu07dyIgtvQuR/U4L7Ug==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR03MB6441
X-Rspamd-Queue-Id: 4LHQBq2lTGz4db7
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=XihkpgEQ;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=cMeLuQfA;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1504b46c8=roger.pau@citrix.com" designates 216.71.155.144 as permitted sender) smtp.mailfrom="prvs=1504b46c8=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1504b46c8=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Sat, Jun 04, 2022 at 01:03:55PM -0700, Brian Buhrow wrote:
> 	Hello.  In the process of getting the netback.c patch into a working ker=
nel on the system
> I'm updating, I discovered that Stable/13 doesn't boot successfully when =
xen-4.16 is running.
> Specifically, the system hangs while configuring the PCI busses.  I suspe=
ct it has something to
> do with the failure of the hpet timers to attach.
> I do not know if it works with older versions of the xen hypervisor, but =
it boots and runs fine
> without xen running.  Below are two boot logs, the first with xen install=
ed and running, the
> one that fails, and the successfful boot,  with FreeBSD running on bare m=
etal.
> Hopefully this is a known issue and there is an easy fix.
>=20
> -thanks
> -Brian
>=20
> Version info:
>=20
> FreeBSD xen-lothlorien.nfbcal.org 13.1-STABLE FreeBSD 13.1-STABLE #0 stab=
le/13-984a45d77: Sat Jun  4 07:38:09 PDT 2022     buhrow@fbsd_dev.nfbcal.or=
g:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/13/amd64.amd64/sys/GENE=
RIC amd64
>=20
> 
>=20
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
>=20
> FreeBSD/x86 bootstrap loader, Revision 1.1
> Loading /boot/defaults/loader.conf
> Loading /boot/defaults/loader.conf
> Loading /boot/device.hints
> Loading /boot/loader.conf
> Loading /boot/loader.conf.local
> Autoboot in 1 seconds. [Space] to pause=20
> Loading Xen kernel...
> /boot/xen data=3D0x2659c8+0x13e638 -=08
> Loading kernel...
> /boot/kernel/kernel=20
> Loading configured modules...
> /boot/kernel/ipmi.ko size 0x11950 at 0x21a6000
> loading required module 'smbus'
> /boot/kernel/smbus.ko size 0x3cb0 at 0x21b8000
> /boot/kernel/geom_mirror.ko size 0x20c80 at 0x21bc000
> /boot/kernel/tpm.ko size 0xad70 at 0x21dd000
> /boot/firmware/intel-ucode.bin -=08size=3D0x303800
> /etc/hostid size=3D0x25
> /boot/entropy size=3D0x1000
>  Xen 4.16.0
> (XEN) Xen version 4.16.0 (buhrow@) (FreeBSD clang version 13.0.0 (git@git=
hub.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)) debug=3Dn Th=
u Jun  2 11:30:17 PDT 2022
> (XEN) Latest ChangeSet:=20
> (XEN) Bootloader: FreeBSD Loader
> (XEN) Command line: dom0_mem=3D8192m dom0_max_vcpus=3D2 dom0=3Dpvh pv-l1t=
f=3Doff,domu=3Doff console=3Dcom1,vga com1=3D9600,8n1
> (XEN) Xen image load base address: 0
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 4 MBR signatures
> (XEN)  Found 4 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  [0000000000000000, 00000000000997ff] (usable)
> (XEN)  [0000000000099800, 000000000009ffff] (reserved)
> (XEN)  [00000000000e0000, 00000000000fffff] (reserved)
> (XEN)  [0000000000100000, 000000001fffffff] (usable)
> (XEN)  [0000000020000000, 00000000201fffff] (reserved)
> (XEN)  [0000000020200000, 000000003fffffff] (usable)
> (XEN)  [0000000040000000, 00000000401fffff] (reserved)
> (XEN)  [0000000040200000, 00000000bc855fff] (usable)
> (XEN)  [00000000bc856000, 00000000bc85efff] (ACPI data)
> (XEN)  [00000000bc85f000, 00000000bc8a9fff] (ACPI NVS)
> (XEN)  [00000000bc8aa000, 00000000bc8b1fff] (usable)
> (XEN)  [00000000bc8b2000, 00000000bc9a4fff] (reserved)
> (XEN)  [00000000bc9a5000, 00000000bc9a6fff] (usable)
> (XEN)  [00000000bc9a7000, 00000000bcbc5fff] (reserved)
> (XEN)  [00000000bcbc6000, 00000000bcbc6fff] (usable)
> (XEN)  [00000000bcbc7000, 00000000bcbd6fff] (reserved)
> (XEN)  [00000000bcbd7000, 00000000bcbf4fff] (ACPI NVS)
> (XEN)  [00000000bcbf5000, 00000000bcc18fff] (reserved)
> (XEN)  [00000000bcc19000, 00000000bcc5bfff] (ACPI NVS)
> (XEN)  [00000000bcc5c000, 00000000bce7bfff] (reserved)
> (XEN)  [00000000bce7c000, 00000000bcffffff] (usable)
> (XEN)  [00000000bd800000, 00000000bf9fffff] (reserved)
> (XEN)  [00000000fed1c000, 00000000fed3ffff] (reserved)
> (XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
> (XEN)  [0000000100000000, 000000083e5fffff] (usable)
> (XEN) New Xen image base address: 0xbc200000
> (XEN) ACPI: RSDP 000F0450, 0024 (r2  INTEL)
> (XEN) ACPI: XSDT BC856070, 0064 (r1 INTEL  DQ67SW    1072009 AMI     1001=
3)
> (XEN) ACPI: FACP BC85DBC0, 00F4 (r4 INTEL  DQ67SW    1072009 AMI     1001=
3)
> (XEN) ACPI: DSDT BC856168, 7A54 (r2 INTEL  DQ67SW         16 INTL 2005111=
7)
> (XEN) ACPI: FACS BCBDBF80, 0040
> (XEN) ACPI: APIC BC85DCB8, 0072 (r3 INTEL  DQ67SW    1072009 AMI     1001=
3)
> (XEN) ACPI: TCPA BC85DD30, 0032 (r2 INTEL  DQ67SW          1 MSFT  100001=
3)
> (XEN) ACPI: SSDT BC85DD68, 0102 (r1 INTEL  DQ67SW          1 MSFT  300000=
1)
> (XEN) ACPI: MCFG BC85DE70, 003C (r1 INTEL  DQ67SW    1072009 MSFT       9=
7)
> (XEN) ACPI: HPET BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        =
4)
> (XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F42=
40)
> (XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        =
1)
> (XEN) System RAM: 32683MB (33467896kB)
> (XEN) Domain heap initialised
> (XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000=
000, using 32
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
> (XEN) Switched to APIC driver x2apic_cluster
> (XEN) CPU0: 1600 ... 3100 MHz
> (XEN) xstate: size: 0x340 and states: 0x7
> (XEN) Speculative mitigation facilities:
> (XEN)   Hardware hints:
> (XEN)   Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH
> (XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
> (XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- STIBP- SSBD-,=
 Other: IBPB L1D_FLUSH BRANCH_HARDEN
> (XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe add=
ress 1000000000
> (XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
> (XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
> (XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
> (XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
> (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
> (XEN) Initializing Credit2 scheduler
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Detected 3092.988 MHz processor.
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Posted Interrupt not enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using old ACK method
> (XEN) Allocated console ring of 16 KiB.
> (XEN) VMX: Supported advanced features:
> (XEN)  - APIC MMIO access virtualisation
> (XEN)  - APIC TPR shadow
> (XEN)  - Extended Page Tables (EPT)
> (XEN)  - Virtual-Processor Identifiers (VPID)
> (XEN)  - Virtual NMI
> (XEN)  - MSR direct-access bitmap
> (XEN)  - Unrestricted Guest
> (XEN) HVM: ASIDs enabled.
> (XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
> (XEN) HVM: VMX enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB
> (XEN) Brought up 4 CPUs
> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
> (XEN) Dom0 has maximum 440 PIRQs
> (XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
> (XEN) WARNING: PVH is an experimental mode with limited functionality
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Scrubbing Free RAM in background
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) Xen is relinquishing VGA console.
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input=
)
> (XEN) Freed 600kB init memory
> ---<>---
> Copyright (c) 1992-2021 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> 	The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 13.1-STABLE #0 stable/13-984a45d77: Sat Jun  4 07:38:09 PDT 2022
>     buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/f=
bsd-src/13/amd64.amd64/sys/GENERIC amd64
> FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmor=
g-13.0.0-0-gd7b669b3a303)
> VT(vga): text 80x25
> XEN: Hypervisor version 4.16 detected.
> CPU microcode: no matching update found
> CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3092.99-MHz K8-class CPU)
>   Origin=3D"GenuineIntel"  Id=3D0x206a7  Family=3D0x6  Model=3D0x2a  Step=
ping=3D7
>   Features=3D0x1fc3fbff
>   Features2=3D0x9fba2203
>   AMD Features=3D0x28100800
>   AMD Features2=3D0x1
>   Structured Extended Features3=3D0x9c000000
>   XSAVE Features=3D0x1
>   AMD Extended Feature Extensions ID EBX=3D0x101000
> Hypervisor: Origin =3D "XenVMMXenVMM"
> real memory  =3D 9725026304 (9274 MB)
> avail memory =3D 8273047552 (7889 MB)
> Event timer "LAPIC" quality 100
> ACPI APIC Table: 
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s)
> random: unblocking device.
> ioapic0  irqs 0-23
> Launching APs: 1
> random: entropy device external interface
> kbd1 at kbdmux0
> vtvga0: 
> smbios0:  at iomem 0xf0480-0xf049e
> smbios0: Version: 2.6, BCD Revision: 2.6
> aesni0: 
> acpi0: 
> acpi0: Power Button (fixed)
> attimer0:  port 0x40-0x43 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> atrtc0:  port 0x70-0x71 irq 8 on acpi0
> atrtc0: registered as a time-of-day clock, resolution 1.000000s
> Event timer "RTC" frequency 32768 Hz quality 0
> hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
> hpet0: memory region width 1024 too small for 32 timers
> device_attach: hpet0 attach returned 6
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> pcib0:  port 0xcf8-0xcff on acpi0
> pci0:  on pcib0
> [ ... At this point, the system hangs, needing a hard reset ...]

Interesting.  I don't seem to be able to reproduce the hang, so I will
need a bit more data from your side in order to debug.

When the system hangs, can you switch to the Xen console (Ctrl-A x3 on
the serial) and then press the 'd' key and paste the output here?

If possible also upload your kernel.debug somewhere, so I can get the
symbols (it's in /usr/lib/debug/boot/kernel/kernel.debug)

Alternatively you can also try to find the commit that broke it, by
using 'git bisect' between 13.1 RELEASE and current stable/13.  13.1
is fairly new, so there shouldn't be many bisection steps to figure
out the wrong commit.

Thanks, Roger.

From nobody Tue Jun  7 19:19:01 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6068183C4C6
	for ; Tue,  7 Jun 2022 19:19:09 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LHgC02nr7z3jGV
	for ; Tue,  7 Jun 2022 19:19:07 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 257JJ1rt005176
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Tue, 7 Jun 2022 12:19:01 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 257JJ1aB027047;
	Tue, 7 Jun 2022 12:19:01 -0700 (PDT)
Message-Id: <202206071919.257JJ1aB027047@nfbcal.org>
From: Brian Buhrow 
Date: Tue, 7 Jun 2022 12:19:01 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Tue, 07 Jun 2022 12:19:01 -0700 (PDT)
X-Rspamd-Queue-Id: 4LHgC02nr7z3jGV
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.30 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-0.998];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  I'm happy to do what I can, but it is difficult to get a lot of time to
bring this server down to get diagnostic data.  Having said that, I'm happy to look at the code
via bisection if you can help me by providing sample commands to git to assist me in getting
the idea.  I'm pretty new to git and am still learning my way around it.
Another question which you may or may not know the answer to.  Since the server in question
runs FreeBSD fine when it's running on bare metal, do you think the bad commit would be xen
specific or do you think it's some change related to the difference in the way the hardware
looks to FreeBSD when it's looking at it through xen versus looking at the bare machine?

-thanks
-Brian


From nobody Wed Jun  8 12:34:29 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3C1C483AF09
	for ; Wed,  8 Jun 2022 12:34:46 +0000 (UTC)
	(envelope-from prvs=151432dd3=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LJ69x13qXz3klX
	for ; Wed,  8 Jun 2022 12:34:45 +0000 (UTC)
	(envelope-from prvs=151432dd3=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654691685;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=4q9D//ea4VXUD0gF/Rv8OUUZnKSQXajHa37Gzbrkx0A=;
  b=CJptygzM3jxoU6u9tDAP1+d87CKjdGnv0eIph/dWsjx0roKXln2qVTwe
   HVpw4mHJUEfT93jSWujhD5b04qFC8GDnbatXHUM8BlwxFky+su7zQOFSq
   DHRq97llREEmr/6AVONocMUmTDzjiXe0LdD6cb4e53OsvBTk2qZIX4MBH
   Y=;
X-IronPort-RemoteIP: 104.47.58.102
X-IronPort-MID: 72979601
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:s+yTkaB8e3OKJhVW/3Liw5YqxClBgxIJ4kV8jS/XYbTApGkr1DQDz
 WdLXmqAafyLMGChLYhza4izoxgDvJeHmN42QQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA
 xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgHGeIdA970Ug5w7Bh0tYy6TSEK1jlV
 e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH
 I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPh+x
 YtQsLeTcjwPFbLAk8ssChpnMzlXaPguFL/veRBTsOS15mifKz7A5qsrC0s7e4oF5uxwHGdCs
 +QCLywAZQyCgOTwx6+nTu5rhYIoK8yD0IE34yk8i22GS6t2B8mbGc0m5vcBtNs0rtpJEvvEI
 dIQdBJkbQjaYg0JMVASYH47tLjy1yauLWUBwL6TjfYe+1OJnB1d6rTSP8P7UYWYTphruG/N8
 woq+Ey8WHn2Lue3wyeF6HWzrunBlyjyXMQVDrLQ3vBwjVCZ3WBWGBAQfVWhrPSzkUL4XMhQQ
 3H44QIrpKk2sUCuEN/0WkTgpGbe50ZGHd1ND+c99QeBjLLO5BqUDXQFSTgHb8E6sMgxRnoh0
 Vrhc87VOAGDeYa9ERq1nop4ZxvoUcTJBQfuvRM5cDY=
IronPort-HdrOrdr: A9a23:ZK9ErKOyJyNRlMBcT13155DYdb4zR+YMi2TDiHoddfUFSKalfp
 6V98jztSWatN/eYgBEpTmlAtj5fZq8z+8N3WB1B9uftWbd2FdAQLsSjrcKhgeQYBEWldQtqZ
 uIEZIOb+EYZGIS5aia3OD7KadH/DDuytHUuQ609QYIcegFUdAD0+8vYTzraHGeCTM2c6YRJd
 653I5qtjCgcXMYYoCSAWQEZfHKo5numIj9aRALKhY74E3W5AnYoILSIly95FMzQjlPybAt/S
 zslBH43Lyqt7WexgXH32HewpxKkJ/Ky8dFBuaLls8JQw+cwzqAVcBEYfmvrTo1qOag5BIDl8
 TNmQ4pO4BJ53bYbgiO0G7Q8jil9Axrx27pyFeej3emi9f+XigGB81Igp8cWgfF6mI71esMnJ
 5j7ia8jd56HBnAlCPy65zjTBdxjHe5pnIkjKo6k2Ffa40Dc7VcxLZvsX+9KK1wUh4S1bpXUd
 WHVKrnlbZrmBKhHjrkV1BUsZORti9ZJGbEfqAA0vbloQS+0koJjXfw//Zv4EvoxKhNNKWs2N
 60Q5iAtIs+OvP+PpgNc9vof6OMexzwaCOJFl6uCnLaM4xCE07xivfMkcYIDaeRCdc18Kc=
X-IronPort-AV: E=Sophos;i="5.91,286,1647316800"; 
   d="scan'208";a="72979601"
Received: from mail-dm6nam10lp2102.outbound.protection.outlook.com (HELO NAM10-DM6-obe.outbound.protection.outlook.com) ([104.47.58.102])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 08 Jun 2022 08:34:38 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=ESFildMV3v0HOOUnUcGeH+Iu5gfGyTVsOBVejtHfWbhuEk4J+Q99ycAFRx3auqEeI7ji0Ig/usyerjny6d7dQE3WpkJXVN9DeR25h3wtoN5qnlNmA9SdCMYx9ukSwCwlvc9U4+aG7gBs+xaaxReaXASYrR1B3G5CZWrPSyJPZyERO+ax8xMJlgZPlg5z+orNsXjOB3bNEw86yK2UWweuJT4GUhS2EHaZIIPyJgPLsOKFjCqN4BSJlO7Cf86qQ1mCjQhLUXhCVvGlm5r2XQ0tQFTf7AuQQGztQdbbCMIsI1cs/4ezaaAL/QmYfLbvi4PFEjakLTfh0UreSqF93tV11A==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=iySmQsg0gj/Tui/7mu363lF1r6kfSGBYs+RMFHK5v5U=;
 b=kUcVIxhan9LiiX1N4UZV3iVMpsBTMv0rJ1ZcWVK7RkWAi9spxdVMN+vpNb2BYitmkhI9ZMHRLBRWrN/LdDYYrbrkfYp6lwA+uMY/Ofamfi3XfcLjwFOAi6WLccsL1BUEkvG0YXng99w+GazZX7rwB72Fic7ziCwywQat6JY7esx3KOXJb4dvDI+uqBW/n7eWlVtqcofJnWL6K/N8IXns1ONmv0zWZX1ERYnTg70g8CgqLtBTu+30x9N2YFfo4dbSafCY/XTzhbpKphU+ibL/UYf0vC5HZU2IpTEJcs7T227aLi7me8+85BodDkVKmPnKCK3oWcliDj7RczINdWJMcA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=iySmQsg0gj/Tui/7mu363lF1r6kfSGBYs+RMFHK5v5U=;
 b=d1M+dKFPItHlfrNqWyfTQ4ttKQsewbOSLJrjuSDIONIcDJ/4CvSldAFcOgnyXGZRS3EoDDoGAgJsE07+SqHujd1OD+O9+xy70kfE8Kei7aRen7YZUAG95OJpTg2sb1sj64L+h6TBK90h0E7/VSivQDzoqMUMdcdDISGHwxyDIpc=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by SJ0PR03MB6762.namprd03.prod.outlook.com (2603:10b6:a03:409::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.12; Wed, 8 Jun
 2022 12:34:34 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.019; Wed, 8 Jun 2022
 12:34:34 +0000
Date: Wed, 8 Jun 2022 14:34:29 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: 
 <202206071919.257JJ1aB027047@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206071919.257JJ1aB027047@nfbcal.org>
X-ClientProxiedBy: LO4P123CA0310.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:197::9) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: b630e624-4fcf-4e58-5fb4-08da494b3e02
X-MS-TrafficTypeDiagnostic: SJ0PR03MB6762:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	zcPRrJ6JpUehywyzxQ7PVnwz1u0rCO/3rk0Aifp6noikT/dkrTfiLNHhY8+OveKB7R7BuMx6/opVycO3G5XZD1a1Z2WYbtrPw4EhKWNgfodRJB+XissF+FhdQQlk3HeWoAtzPRWrQqAK316cysDm5VOs2ux0Vq2fmqfzM5goXAsDO7PXd3AgvvJUiy8hb3lLM28wpv9uCJTdbSj/SSNsBiqCG6qWrj0Z+bHtqfVTPyTZ1UX3aXLHpSsOo/PRu3ouNB/IotHTTyuEoqpdz5fZDcVmBmH21ZFpzZSfzXfwZIfIQPSJiIgMztASSOlML8sDmv8YSi+eR7uMoY4wP0jYcHmVJfNFnTlkyly58zuQk6lvUPGkYhcDbVgiEWRGgjvJEFUpmOlge2nbuF+1Qt1iNv6oVH/g4jtyzOEUbcf6UBoabnkP657jovW56+J71UDEq/5aYw7YAz18tToggcWEHhW0PIBD3RtOGGK4YGfm9h/rEONLGTfx7rVlVwXfIVTUqc4kzJWrw7Us/X/POIXm/XgeZC2RbWoUVC2eeGm0LiGVIgmhgfF8BFTBPh4uwqd5QtAgWT7SPtq2E0VZI88DncQ2pfRyFsx4+xak5VEaN2ba5virqdDvvmQGKnXxhtZaRbOK/LEX00k0o+a1ZIk4V72g/7JmaoqqNKFGLmS29iVZNA8hP0FWuYlFADjUycu2G0tyvon6/vFrV0sXb1DOTN6xc40TKZhw1eJENKjHPXSYG+3EMQrw4Nml1CrNYDmWK4WaxPNvKIhbSomgWlJ8uVc8KmGMx0/WHGPZTA9HSv0=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(66556008)(4326008)(6916009)(85182001)(38100700002)(66476007)(316002)(6512007)(5660300002)(33716001)(8676002)(66946007)(966005)(6486002)(82960400001)(86362001)(508600001)(186003)(6506007)(8936002)(26005)(2906002)(9686003)(6666004)(340984004);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?eGplbEdSbWtuS2tIaG0xSW9CWWRWZ2FFZFB6Nnc3SjB2NzArTElEUnNzYkNi?=
 =?utf-8?B?aXRrVnc3ejVOdnZWSWJPS3R3UHduaHdqY2xtZ05uNlNMd1BnY1l5dW9jL1Q3?=
 =?utf-8?B?NDhKUWZ2NWV4Y2VTMkp4WG9jY0VDMHdEOHBORkJUNi95TzFtNnVwT25WYzNJ?=
 =?utf-8?B?TjJaMHRWVjZ0YUdWYTBlR01HcW5kaXAvaklIc1NRMDNtMEdVVlF4Zzg3SEt5?=
 =?utf-8?B?Wk9LNlZpOTNrMmtkdmUwR2t1cWdLYXVTZUJicWhMT0UrMEIwTUZVdXBoMnZQ?=
 =?utf-8?B?OFlYcHdNZzl1M09MeVk3VlN4UUNBNVhZRThhczdGejd0N1pJZXZvTExJVHVs?=
 =?utf-8?B?NDZGL2UySHFXNWhjemxuU0VveXh3TmxYeXUzNkg1eXZ4S3B4cWlkYnZKanlP?=
 =?utf-8?B?OGV1Q000MUVUU1p0RklmZ0ZmVCtqbGpvckRjblZ4WmlhSmhVYXZjS0JJM0sx?=
 =?utf-8?B?RUpiU1VOZkd0cHhxRzdnN21TVlIrdFRSWWRBc0M5OTk4OTBGaVJTLzhuOE9C?=
 =?utf-8?B?TlBXYUtTUHJFQkR6Qk4zWVpvWTd5blFPU0ZoMW9KM0czTklRQ2NBRHJHUG9W?=
 =?utf-8?B?ZWgwS0dVK3l4OXV4cFBOWjg1L3AvRjNUay9IMkhOcHZCUUxjdDBPSDJaeUlU?=
 =?utf-8?B?OGYxQWFsNnJvQ05lQkl2MDFKNHQ4RU01aFV5VWczUktLbTlIL3l4V205SW4y?=
 =?utf-8?B?S1VmK0ZuZUM1VUhORE9lSmtLTDBrL3I5WitEYmZtWFBZdmN3UFpwSWk4Z2J3?=
 =?utf-8?B?YnQ1ZW85Q1hCbC9aK1ovTkFoODFVdlo1b2FuWFR5bXNCRHJGbHF5ZGV6WnBp?=
 =?utf-8?B?b3YvZW9NYzhYeWpJSzBDdGhIMEc0dE54UjhEc3dHTWZ5b3FIdWpiamMxR0Ra?=
 =?utf-8?B?ZE16SklWTzJDZTgxVVVlREh0U2twYkFKdytOZTFaV3NvRWxyb3FndHF5QmYx?=
 =?utf-8?B?RWxFQklpejJUUlZsWWZGUFBDT2I4c0J0TGhlWmluMkVEalBNZ1MrUGZsOWt1?=
 =?utf-8?B?enMxV0MvUVFyc2Y4M0tObVo0OUNWdUhiV2pmOE9XL0lYbzFadUFvQ0pZYmVH?=
 =?utf-8?B?NWg2cWJVVFhIY3JrZnhxV1lLSDZiZFRZYVZFS0ZUVVFGR29DMmNjQURoVkZ1?=
 =?utf-8?B?c3JNSXJLTXQ2MUdmR1dBcDdPR0d4YUtFdVJJR25NWkZQUy9WSFJ4V3g0RGxZ?=
 =?utf-8?B?dHd6R3lveG43VHYxZ2k2aHZpTnVISzgrYURCNWgrb0RkODRlMExvZVA1R0pn?=
 =?utf-8?B?U0dEelBUTEZHZkM0SzNaZ3hCaHIvT1JEbjBraUhMQy9WSW9ZZTBtaEJoYzAy?=
 =?utf-8?B?SHU1aytNRlZkZ0dHV3dlQURQenFUKzV1OEs2S0t5NUJ0WFNvYmMrQ1NlN0hL?=
 =?utf-8?B?VzFZOWZzanBtZjNLYUxzVkh4K25JQ2Q2M2w4bElrVlZPQ205T1NPaWF4ZjZN?=
 =?utf-8?B?K2RBY3NlSXRDbzdydGNpWFMraFBMaEdpaXBFWHBpdC9EaXVTWGk3RHR0eDdC?=
 =?utf-8?B?RjF3MXliZDFmNjE1TUpocDRGWjYxQmUyaTZoU0wxbnpaQVFHSE1JR1hxZGtW?=
 =?utf-8?B?ZUl0Zk1iZWlZeWp0b0ZHMXZQVFJoOU5vTkhnYXo0RkowUWtER28rMkhRMDNO?=
 =?utf-8?B?STZUeHA2Umh4Tk5udkxEdkdKVEQ0bCtZc0l0YUM1REszQTFkU2NxN21udmJI?=
 =?utf-8?B?Tk5LZVIvY3RPQkN3WEFNa3V1QjVONGRYaFZ0eDUrWVBXR0JUaFVKQmNtRmZ2?=
 =?utf-8?B?dEo2Z2ZPL0VsTkUwdVFsRTNENXlyT0JnTDlIMWNJWGo0V1I5QlFLb0RHV3V5?=
 =?utf-8?B?VUxzcm9xdFBPZmdMZXl3bFZJYzREbUVhR3FuTXB6QzBENy9PTWxETGNmOE12?=
 =?utf-8?B?N0VXOGF3V21jYStYaUhuL25XU0p3RkhGaU1WdkxLbmd2dGc4SUpZb3VnV2Nz?=
 =?utf-8?B?aVZsRUJ6Rzk4RC9RSlV0aGxyeTdQdC9TNkhtRDZsUVRvTTQ4K2tpK0RncGxR?=
 =?utf-8?B?Vllxd3RjSjVLKzRKQzBJenpHUmNNZDNhUDZDY3RmVG9KWEVYY0c2bjFYajh1?=
 =?utf-8?B?UjBDZzJqNlhXWnZDOUZrU3BQeGV6ZFRsaUhackZjQXJkZEEzUmtUYVdCUzN1?=
 =?utf-8?B?Q01EU2EyN2ZHUXZ2bWZ6MkY3L3NOYXVnM3M4cytEbmE2MHhmQkV0ckdRMU14?=
 =?utf-8?B?S2t2M2dyV3VTeHdXNXZzSi96MHpMelJWK0lSeG41dGtYQUd2TmhCZ0xBL3Np?=
 =?utf-8?B?WVhBSDM2TXJmcXF0N3dQR3B6QjByT1pkclpZVEdIWUduQXJZNkdNVVJYUUpv?=
 =?utf-8?B?dHNWS0N3a2lXdjFYeENWb0FCZDNSVW05eTlPK3F3aVhQbzBadCtUaW82RHlL?=
 =?utf-8?Q?gIcc4r//oPauUZxc=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b630e624-4fcf-4e58-5fb4-08da494b3e02
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jun 2022 12:34:34.0720
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KgbZfY7S+PB6ju8E5L3qJ+ncKlGp5wD+gBA/Ljcp4h44LzK0l+5wpAL+dfS0JDYyOYhe8K9YE8Rb3QpfiyFU2g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB6762
X-Rspamd-Queue-Id: 4LJ69x13qXz3klX
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=CJptygzM;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=d1M+dKFP;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=151432dd3=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=151432dd3=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=151432dd3=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Tue, Jun 07, 2022 at 12:19:01PM -0700, Brian Buhrow wrote:
> 	hello Roger.  I'm happy to do what I can, but it is difficult to get a lot of time to
> bring this server down to get diagnostic data.  Having said that, I'm happy to look at the code
> via bisection if you can help me by providing sample commands to git to assist me in getting
> the idea.  I'm pretty new to git and am still learning my way around it.

`git bisect` [0] will simply point to commits that you build and then tell
git whether the code work or not, it's basically a binary search over
a range of commits to find the commit that introduced the issue.

For example in this case you would use, from a src.git checkout:

$ git fetch
$ git bisect start 0bbbd845fd32ef07707c03d1dc4e4b27f6ebef83 fc952ac2212b121aa6eefc273f5960ec3e0a466d
$ make -j8 kernel
$ reboot
<... test whether the kernel boots ...>
$ git bisect [bad|good]

Select either bad or good from the command above depending on whether
the proposed commit works.  Git will then propose a new commit for you
to test until it can pin the commit that introduced the regression.

If a commit fails to build you can also use `git bisect skip` to
select a different commit for test.

> Another question which you may or may not know the answer to.  Since the server in question
> runs FreeBSD fine when it's running on bare metal, do you think the bad commit would be xen
> specific or do you think it's some change related to the difference in the way the hardware
> looks to FreeBSD when it's looking at it through xen versus looking at the bare machine?

I don't think there have been any? Xen related changes to stable/13
since 13.1 was branched, so it's unlikely to be a Xen specific commit
I would think.

It's also weird that HEAD works for you, which means it's likely
either a commit that changed when backported to stable-13, or a commit
that had some prerequisites to work correctly on Xen, and those
prerequisites haven't been backported to stable-13.

Thanks, Roger.

[0] https://git-scm.com/docs/git-bisect

From nobody Wed Jun  8 15:44:58 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6013E8587DD
	for ; Wed,  8 Jun 2022 15:45:20 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LJBPq1K81z4cLv
	for ; Wed,  8 Jun 2022 15:45:19 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 258Fiwvc014007
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Wed, 8 Jun 2022 08:44:59 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 258FiwmW012809;
	Wed, 8 Jun 2022 08:44:58 -0700 (PDT)
Message-Id: <202206081544.258FiwmW012809@nfbcal.org>
From: Brian Buhrow 
Date: Wed, 8 Jun 2022 08:44:58 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 08 Jun 2022 08:44:59 -0700 (PDT)
X-Rspamd-Queue-Id: 4LJBPq1K81z4cLv
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.22 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 RCPT_COUNT_THREE(0.00)[3];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 FROM_HAS_DN(0.00)[];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-0.999];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-0.92)[-0.917];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  Okay.  Another question, which might be one you answered, but I don't yet
have enough git knowledge to understand it.  Since releng/13.1 boots and runs fine on my
machine, but stable/13 does not, is there a way to compare the differences between the two
branches?  If so, what do I need to clone and how would I instigate such a comparison through
git?  Or, is this just a matter of diffing the two source trees the old fashioned way?
-thanks
-Brian


From nobody Thu Jun  9 06:51:04 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 621478399D8
	for ; Thu,  9 Jun 2022 06:51:20 +0000 (UTC)
	(envelope-from prvs=152c7a754=roger.pau@citrix.com)
Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LJZWC3nGZz4Qlm
	for ; Thu,  9 Jun 2022 06:51:19 +0000 (UTC)
	(envelope-from prvs=152c7a754=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1654757479;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=Tvh/7otQIX49ZX66McKLjTRuEw6pzjIFB8LiN66GxN0=;
  b=gZHfI4Ac6N9npzyxAQuNHRR/jY8PXDKfrBOQzWckZafj9SjciMjYZFEi
   UQsbgltwSipU2kBKvVYPayL+0QRlW+xp3B2Bz3TpwwRpqmJYxspAiI3c4
   mUVFSwKNjoq4gcB0HihN6pOcMZeR4WiqUGeZz4tKjdXPGC0e4epfYqAqQ
   E=;
X-IronPort-RemoteIP: 104.47.57.170
X-IronPort-MID: 73214056
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:hWURnKBA4vTOERVW/3Liw5YqxClBgxIJ4kV8jS/XYbTApDl01DMHm
 jRNUWuHbqvcNmCnLYtxYIy+ox5T7ZaAxodjQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA
 xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgHGeIdA970Ug5w7Bh0tYz6TSEK1jlV
 e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH
 I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPgy8
 sRop5yqaTt5BazMo7QHbiBjPw9HaPguFL/veRBTsOS15mifKj7A5qsrC0s7e4oF5uxwHGdCs
 +QCLywAZQyCgOTwx6+nTu5rhYIoK8yD0IE34yk8i22GS6t5B8mcG80m5vcBtNs0rtpJEvvEI
 dIQdBJkbQjaYg0JMVASYH47tLjz3CCjK2AGwL6TjfQYxXDU5g1A65/0EtOMKvmkZ+USg3/N8
 woq+Ey8WHn2Lue3wyeF6HWzrunBlyjyXMQVDrLQ3vBwjVCZ3WBWGBAQfVWhrPSzkUL4XMhQQ
 3H44QIrpKk2sUCuEN/0WkThpGbe50FEHd1ND+c99QeBjLLO5BqUDXQFSTgHb8E6sMgxRnoh0
 Vrhc87VOAGDeYa9ERq1nop4ZxvrUcTJBQfuvRM5cDY=
IronPort-HdrOrdr: A9a23:CtUIuK7uoqudiCNN/QPXwVSBI+orL9Y04lQ7vn2ZFiY5TiXIra
 qTdaogviMc6Ax/ZJjvo6HkBEClewKlyXcV2/hpAV7GZmXbUQSTTL2KgbGSoAEIXheOjdK1tp
 0QD5SWaueAamSS5PySiGfYLz9j+qjgzEnBv5ai854Hd3APV0gP1XYaNu7NeXcGPjWuSKBJYq
 a0145inX6NaH4XZsO0Cj0sWPXCncTCkNbDbQQdDxAqxQGShXfwgYSKWySw71M7aXdi0L0i+W
 /Kn0jQ4biiieiyzlv523XI55pbtdP9wp9oBdCKiOISNjLw4zzYLbhJavmnhnQYseuv4FElnJ
 3lpAohBd167zfrcmS8sXLWqnzd+Qdrz0Wn5U6TgHPlr8C8bik9EdB9iYVQdQacw1Y8vflnuZ
 g7k16xht5yN1ftjS7979/HW1VBjUyvu0cvluYVkjh2TZYeUrlMtoYSlXklXavoJBiKprzPLd
 MeTf01vJ1tABOnhjHizyNSKeWXLzsO9kzseDlAhiSXuwIm7kyRgXFohvD3pU1wiq7Ve6M0mN
 gsDZ4Y5Y2mbvVmGZ6VV91xNvdeNAT2MGLxGVPXB2jbP4c6HF+Ig6LLwdwOlZKXkdozvdAPpK
 g=
X-IronPort-AV: E=Sophos;i="5.91,287,1647316800"; 
   d="scan'208";a="73214056"
Received: from mail-dm6nam11lp2170.outbound.protection.outlook.com (HELO NAM11-DM6-obe.outbound.protection.outlook.com) ([104.47.57.170])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 09 Jun 2022 02:51:11 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=RTwBbXuuCZs3JTo7ImWukEVzXXda5doegHLZjK6B0afj06Lkc1bhxh554JHGS5XXeAtc+iuo/+Jts+bHUVBzNzV3XHCn3i5nIe+fwGFkc/H+JKKEHQGtrjL33hORqxouxagoJM3Exahxil7FLnJnrB517WsA7uUZvxS2o8D6o4Wvd4tmFGwVk872nUHRcvvPIIDVM3z1IKTgBdkqkhB2TN3+Q2EgEJsuUX9NZ/jJjnD7J9K5SmJcobb05eIkwpOPI1sR2AdQVn7pEpk7Wf9c7lH7m6M1yNHX/2J2X6yzNAuDOOkJUfOf1EFv4BSpY2pikFShC7r8pLsFSAGz+3vzrA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=OBXuC1ZtDolhS6fBs8nOnpuZn+ykw+4PrRqLvjLH+RA=;
 b=FQDjV0d8nr6Y6SgcGFeyK6ci2Jua/7owK7u9akyKtJ8ca5DSWOED0Y6mkTe+JFlmElXZWBYp1gxEHtmyHfid5Ky6wHePEBzjdQWrewDdEEPclYFDVmpdPnwQA6ihSjMLnB9fIfsIp+4Zylter+1/xU2hrlGgGASzw5jLAdiyV73MYt1leKz3eZT671CGIKNJ53eI8uodPs40KXI2P1GE+axuysu+U8oUg3pSV/4WgJBHw+NxxBRC1SHfXlzSDGrlXipekd6IcAHcSVod5bbGK2Ea3g/riz7URAgjMv0ZzAtxdXoogE1jE+2cuZgyHUKV7KJhvrESX8TR3pBVhR/zMw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=OBXuC1ZtDolhS6fBs8nOnpuZn+ykw+4PrRqLvjLH+RA=;
 b=Sy9FqGuA+tKywUM3O8E+mV+bnEOw7Egkzc+OEo7I/39wJTTFjKG31bWB6dxmRnaAtmGwZcUuxjYp4M3LPl0f8gRR45/3E9l/704jIsxuxpc3N3WHlLXeLn1AGA50qBK0C8dhZJXnGQcmQNmJf3FD4cC33MkqAQsD2ur5GSk1SyE=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by MN2PR03MB5023.namprd03.prod.outlook.com (2603:10b6:208:1a7::21) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Thu, 9 Jun
 2022 06:51:09 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::5df3:95ce:4dfd:134e%4]) with mapi id 15.20.5314.019; Thu, 9 Jun 2022
 06:51:09 +0000
Date: Thu, 9 Jun 2022 08:51:04 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: 
 <202206081544.258FiwmW012809@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206081544.258FiwmW012809@nfbcal.org>
X-ClientProxiedBy: PR3P192CA0026.EURP192.PROD.OUTLOOK.COM
 (2603:10a6:102:56::31) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: e5d6e16c-f277-4940-16a7-08da49e46f60
X-MS-TrafficTypeDiagnostic: MN2PR03MB5023:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	xULOILApBMaR1LyEYHJ5ByNrhOSmmuR2N4J5y+Mk5KpL/tItbyn7IDgELH25B09Ual2pnSGEvzUVSB97ziHKHUYzjJZilzicO5lQjBxqVPnZdqVAsEE3sAACY3Coa6O6sKLBWm2PpBJnFri529unql8+Fg326o9bet5seFf6OYn5KIn+CMwvsrT5OemH6F7LaBSyE+r7k1Q8n/uWveTXO2xqn2My8E3w3Gmh/wGtBwdHul9ZwIsdokQ3iWR4NPiOmMY4eRwPDtjjQXC37uAgsl+G4+JcuQ+gMxq8QvZhn/tgVqneMnK+gTQodYkBlWk2/FZ7PIkZjA029K3sNWCMPF25LzY4uxFTmf46Va5eHafpUxeJA9UlCilVY1B41FzqqQmu4nPsooNREsrBJ8JngqIsj80nfuj9Ni89Rz9KwaOIQ+y0BQkTxzO9grUCjPdtKXdh5+pzJMeeq/+kGKx4A8hJkIyvxxu9SQ9IVBnPHjYripfDCHULEBBuSIrp30phTPaqS+lrUvGB8zKkS2dq6wzZsQiioNf3wr8YFZRzKxc6iiasHxhvoHKvlT8BqwxmAxiwHIPxm24KOkV1gy9oDJfAjZYE0cenH/akeZGnay2r0+GARXn1kraHvkdmGJQgJQAXQVLLTc7IxSk/L2gJegqeQne+xMcoLJuDTUSxzMbQgytGnnvJTh4IekNG7H/1sOyqd2i2UmjxXoApji+WJjDgDCe4i9UPNPOzlMQj18w=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(33716001)(6666004)(86362001)(83380400001)(6486002)(966005)(508600001)(38100700002)(6916009)(186003)(26005)(6512007)(6506007)(9686003)(5660300002)(2906002)(4326008)(8936002)(8676002)(82960400001)(66476007)(66556008)(66946007)(85182001)(316002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?RWQvWU1iQ1RTbDhha0hFaEhlbCt2ZkQ3KzB2Q1U4NVZBYXA0S0pOc216YWYr?=
 =?utf-8?B?Vkkza0RsNlhjMVoxZDhhRzlyOTVZcUNSMGViMmRBUEhoZ0hNamMwVWtPUzVR?=
 =?utf-8?B?dzhRb1hOWEVXZmhscnFmdUY3RCtUUzI0cElKZnFyYmN3NXl1QkdRdkJqTTMv?=
 =?utf-8?B?Um9OK3pGeGlnYTRwMlIyMG9EVGRVYnBJOGFsQ09lS2c3K3FHYUcrTzRSZnJn?=
 =?utf-8?B?SG10M2dYQ2ZHRVY1aGg1aHFUNUZIY0ZJM2VJeHpKRm5CRzFuYnRScDBjQnhG?=
 =?utf-8?B?UG96bGs3V3dnM2s5WmZSMEI1MlZQTHZrWDhOZEdNU2RHV2RLRy95SHlXWlU2?=
 =?utf-8?B?RHNMMGpEWEJUQmJhVldVMGhia0E5MU9tcnNBemdSZ0tzV2labjlzSTlUR2Zv?=
 =?utf-8?B?T2w5dEI1NHQzMlpuRmMydHptYnppS0dWNUQ4TTVlajdDK043dU1iNkZ3UmNY?=
 =?utf-8?B?SVRRQ1gzdHYzcXBjdVFnNUNSSmZuRFZEeVNxKzNPVXN2cmJldXJXUUlDUWZt?=
 =?utf-8?B?RVExNmZoNUx6NlZoeVJqbWdFYldEaWdJOHVQdUtab0FaVnI3YjhQK2NLdzZ0?=
 =?utf-8?B?WGVrRnA3R0Vyby9ndWFNeHZiV00vZWowekhmM2pKcThDZkI5RGtsZnZDdUZH?=
 =?utf-8?B?MzlMUFZKcUhHQTM5ZTFqK3RJZDQrM2xaMVJ6WjFZdjJjTVdaZkI4cHVKY2pY?=
 =?utf-8?B?blRYRkQ4RXhSVmRBR1A1MTJKZzRjdXVqMG40UHNBa3FtSGVsbEpaa3k3RjNw?=
 =?utf-8?B?bjgwUzUvaTZLdTc2V3VDdEI0dXlZVGZzM0Z0V29sZmJBY3BEeFgweDh5QWNM?=
 =?utf-8?B?YjRMVzZoRngzcmE2Ymo0dlFLekdtNmNZUFFEbWpCN1Azd2F4NXNHMDhpb1Q3?=
 =?utf-8?B?cHlDU3ZVS25kM2pCVjlqV2hPSVBuVVhZZ3dsaDB6NUpSSkF2TGJ4b1pDSDBD?=
 =?utf-8?B?eFpaL25nL2NlRDgxc3UwY1JjQ08xVjdGRkRSSEZBcUMxd2RUTzhVREZaQVll?=
 =?utf-8?B?aE5GYTUvbGNaM202MlR2bTdMTVZjTFY4WEErRU1XRUl5empWSXJhN2xLUDBj?=
 =?utf-8?B?LzU3T1J0ajFCMjdMOGp6VFd4aXJzU2IyekwzbnZLVU4yMm1iN2x2R28vLy9L?=
 =?utf-8?B?bzRmNngyWmNzMzZ4d1Ywc2dTTk5Jb3dpbHF5OUFkQjdKRmc3MDhGOVZ3U1Mv?=
 =?utf-8?B?ZHRQNGIwc0k0NU85ellQM3hQSC8wSEhZVzJuZkd2LzNJVkk0eUZ3dk1nS3Nz?=
 =?utf-8?B?QlZPZy9ycnNXVnkwblBpS1dKaHh3OHhDRGxVTngxQ0RlUmJjUkd6eThrNlUr?=
 =?utf-8?B?WGlCT2cvbDdNSDNmeExIUU1uM0tFajg2aVJ4dWdGMk5LR2QzSjZrMjl6RXZ2?=
 =?utf-8?B?VkZod3JnNVFtdm9LYmFkUHVyOHArdlpVYlR1aGlZaGhTa2VsK3FxZUhzSXhv?=
 =?utf-8?B?QVF3T2ZxWUVtZ29rRW1Vb0hmVFhNN3VZaGl1OHI2TmY1OXB4KzJueFVFcjli?=
 =?utf-8?B?Nm8vNVBYb0hrNEtmbldkbHlrK1RaZnRRQ0Y2WEJMQllaNzVaaVdYek96RmVp?=
 =?utf-8?B?a09XVEh0VlBGdy9PNFd0aWkzazhvOUpFdzl5V0I3MityNnhHQ2hnc3RpUGpD?=
 =?utf-8?B?Y3NKZEN1dTVCRjdIME1LVlJEZDRXUEpFNndiVDF1ek5TZGZzODM3QlVpZ2wr?=
 =?utf-8?B?ckhzaFBsSXVvMnRtbGpkc0FGT2daQ21ncnFCamwvK0c1VFZKSWs2OW85dTNO?=
 =?utf-8?B?bzYrMFZUWUZzU1NvbU4xTG1RV0lCa2NGTGhmSjlZamMvMmlyN1pkWnRDL0FF?=
 =?utf-8?B?YkQrNXRSRjBUZlNMQkFMRkpvblhEbnlIUUFLM3d6aEZnM2tEQmlRbnJNMUMw?=
 =?utf-8?B?VGZpTW9kVDA1Z09WNHYwVVZTQUJnU0JJNjNJM1hicFZDVkx6SnYrOFk5Z2dK?=
 =?utf-8?B?MVFvekk0Z0RnVUVNVnpzNC9oWm56Z1NCVDlMWCtWMTU5d2dmMDJOeDlpNGFC?=
 =?utf-8?B?cEZpaFYxQ1oyc2duNE9jWElIamZpQkxPMytyWHZOQ2Z3R3o0STFzZFkwQmlQ?=
 =?utf-8?B?a3BXSko3elYzYWF2WEFhNnZQZzBXSkZmVWZwTHE3WVMzcS9ENEhDQmVITHl2?=
 =?utf-8?B?YjBEQ1RoQjZGR29uaWt0Rnc3b0FjdGZvcStTak9PTWxzT055YWIydnFmdlVK?=
 =?utf-8?B?VGtqNUFJNXpOVmVqNW1EaFljQXpmSllMVEJqSm5BQ1dUbDM4cllRQmxnS0FW?=
 =?utf-8?B?OFZ6SGhKbGE2TElncHZrYTZaVHVBRVlSVTF0WVVvUWV5RW5tM2RvWXBzRGgy?=
 =?utf-8?B?YXR6czBCMURQZlpqM2xNbCszSzBhSGRxOGxpZUluVlp1T04vei9NUT09?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e5d6e16c-f277-4940-16a7-08da49e46f60
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jun 2022 06:51:09.7691
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CGU0W647SIT3cv47/auL9BJp3iwCoTzdKRLMRtyW6kESAFIVxcr390aLfWDFjoWQJpSvlX0ZpF+DgYRE3PwEJA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR03MB5023
X-Rspamd-Queue-Id: 4LJZWC3nGZz4Qlm
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=gZHfI4Ac;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=Sy9FqGuA;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=152c7a754=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=152c7a754=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.13 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=152c7a754=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-0.93)[-0.935];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Wed, Jun 08, 2022 at 08:44:58AM -0700, Brian Buhrow wrote:
> 	hello Roger.  Okay.  Another question, which might be one you answered, but I don't yet
> have enough git knowledge to understand it.  Since releng/13.1 boots and runs fine on my
> machine, but stable/13 does not, is there a way to compare the differences between the two
> branches?  If so, what do I need to clone and how would I instigate such a comparison through
> git?  Or, is this just a matter of diffing the two source trees the old fashioned way?


You can easily get a diff from the git tree, ie:

$ git clone https://git.FreeBSD.org/src.git
$ cd src
$ git diff 0bbbd845fd32ef07707c03d1dc4e4b27f6ebef83..fc952ac2212b121aa6eefc273f5960ec3e0a466d sys

Note the two commit hashes: one is HEAD from releng/13.1 and the
other is from stable/13.  I've added the 'sys' at the end to limit the
diff to changes to the sys/ folder (so kernel only).

In any case, I think such diff is too big to be able to spot any
changes that could cause issues by code inspection.

If you can do the bisection and provide the info from the debug key
(plus the kernel debug symbols) that's likely to be quite useful.

Thanks, Roger.

From nobody Mon Jun 13 05:06:25 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BC52F831FF0
	for ; Mon, 13 Jun 2022 05:06:38 +0000 (UTC)
	(envelope-from fetrovsky@gmail.com)
Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LM00Y5s0Pz3l5V
	for ; Mon, 13 Jun 2022 05:06:37 +0000 (UTC)
	(envelope-from fetrovsky@gmail.com)
Received: by mail-ed1-x52c.google.com with SMTP id x5so5587683edi.2
        for ; Sun, 12 Jun 2022 22:06:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=mime-version:from:date:message-id:subject:to;
        bh=Fv/RMSi02IGgeysurKpQ5OZ9E8lZjvMqggHYNuPLjRU=;
        b=VNuDTtoDJCklWrs4+i0co4x4p+DCimBizhPKAXMlN6RrxjxSDhuapdVq1Ry5I/bCRv
         rvOTkF8h11Ro3zX+2pgH8DrPU4LMhEECmYYLBCtFmjQ9yDylu8LEGEaLUVcdZezBOad0
         kCaa2mYKVZLRAXiYP8b2LZ6bab6RHcoVY/U1jWFChtD2ltCzyzQhRv9CS/qvPof/e3A4
         6SDPfIw33LoVkmVN1kG6eWLAsPn4kJ72s1SSy0UEqDvL2Jx6CHbgfB30PG4/p+9zJ0ik
         JKLheg+tPCN1qKHARH99lHOxaxfQKmbnqdm0NIbUL2s51wbnDNRCMDuly0z6azR5visN
         7wKg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=Fv/RMSi02IGgeysurKpQ5OZ9E8lZjvMqggHYNuPLjRU=;
        b=cWsEoXGL77eyVUxob6p4PtT/1b7g42xsumaeiPFStf55uNJQ77ymxJbSMf0HEew7HH
         1QwRKV8HYCAfGGl0v1ir7j7yEbtE9qifHzX5qwaylsdvI7tKP1qi8lVzSyDJGM2d//1Q
         Bc1HHn8h/50cBNHta29+nlkc8nNcBZ1m1yehhjFQ3fWSRr8NvDm8zZQmwuxf3iiw+c4U
         2EEzCnHxv+zwib9zhowdRK9doVjzZnkONWdDEpq7sw9ffeUixOgZ35Hd8a6M9LR44FGi
         ioQoe9ViB3jjbjHow1BUzSPiPNlPQUEqyrRw1C0NImTdhv9Ls7QNNIjuNW95A7RpMZh6
         6g8w==
X-Gm-Message-State: AOAM531HhvVX2t9G/9wX4pUTr55OAL0oXlacC1vXZzzFR1wtVLyAPpLl
	WBBB7qDxd+/o1tkaqEYznQQWMdxjbWOSYO5l0HtuERBC
X-Google-Smtp-Source: ABdhPJy0MSHS28QPsCjKCay6iFPOkdzOLZyK1MINAufvVAhGRPv4XWLf+26cTBXXgP81tPQJF1/4y6Wj3wjQb5ctmM4=
X-Received: by 2002:a05:6402:2684:b0:431:497e:699e with SMTP id
 w4-20020a056402268400b00431497e699emr45027405edd.343.1655096796233; Sun, 12
 Jun 2022 22:06:36 -0700 (PDT)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
From: =?UTF-8?Q?Daniel_Jes=C3=BAs_Valencia_S=C3=A1nchez?= 
Date: Sun, 12 Jun 2022 22:06:25 -0700
Message-ID: 
Subject: Test
To: freebsd-xen@freebsd.org
Content-Type: multipart/alternative; boundary="0000000000009bc26605e14d402b"
X-Rspamd-Queue-Id: 4LM00Y5s0Pz3l5V
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=VNuDTtoD;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of fetrovsky@gmail.com designates 2a00:1450:4864:20::52c as permitted sender) smtp.mailfrom=fetrovsky@gmail.com
X-Spamd-Result: default: False [-2.75 / 15.00];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36];
	 TO_DN_NONE(0.00)[];
	 MID_RHS_MATCH_FROMTLD(0.00)[];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+,1:+,2:~];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 R_MIXED_CHARSET(1.25)[subject];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 TO_MATCH_ENVRCPT_ALL(0.00)[];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 RCPT_COUNT_ONE(0.00)[1];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52c:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

--0000000000009bc26605e14d402b
Content-Type: text/plain; charset="UTF-8"

PING

This is a test.  All I need is one PONG to see if I can actually publish to
this list.

Thank you,


- Daniel

--0000000000009bc26605e14d402b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


PING

This is a test.=C2=A0 All I need i=
s one PONG to see if I can actually publish to this list.

Thank you,


- Daniel


--0000000000009bc26605e14d402b--

From nobody Thu Jun 16 16:23:38 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8711584199C
	for ; Thu, 16 Jun 2022 16:23:55 +0000 (UTC)
	(envelope-from prvs=1593354c1=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LP6tf0KT6z4dyM
	for ; Thu, 16 Jun 2022 16:23:53 +0000 (UTC)
	(envelope-from prvs=1593354c1=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1655396634;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=kz3EpOvZQtzy4EwcJuCBsy2tupYGrOIZXb14p+TWqYI=;
  b=VDzN+ozLfMyduK5MbYUyTf1mN7btpeu2FD9csJeXjoWPeo6Zck1J03xp
   DvSr9j2xDaXZGRtFKTL8I6S4SR/8bKqrxH9Gvowo7lQ/h6KC+QjmHZRyh
   RIP4ZS9mdCIbgt4ayCu8zD8BFRFQR+sKjgOqXTdORODgdE5tTsOrbTq0g
   M=;
X-IronPort-RemoteIP: 104.47.74.45
X-IronPort-MID: 73626437
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:+Scdgqnl+8ItQGZgptjMhGPo5gzZJ0RdPkR7XQ2eYbSJt1+Wr1Gzt
 xIcXWiFaf7bMWfze4h/bY6+9RwEv5SEyYM3TwRuqSBjHyMWpZLJC+rCIxarNUt+DCFioGGLT
 Sk6QoOdRCzhZiaE/n9BCpC48T8kk/vgqoPUUIYoAAgoLeNfYHpn2EgLd9IR2NYy24DnWl3V4
 rsenuWEULOb828sWo4rw/rrRCNH5JwebxtB4zTSzdgS1LPvvyF94KA3fMldHFOhKmVgJcaoR
 v6r8V2M1jixEyHBqD+Suu2TnkUiGtY+NOUV45Zcc/DKbhNq/kTe3kunXRa1hIg+ZzihxrhMJ
 NtxWZOYexUyEfSRov4hAxhEQg19H4ZZ5ZKcGC3q2SCT5xWun3rE5dxLVB1zF6tIv+F9DCdJ6
 OASLy0LYlabneWqzbmnS+5qwMM+MM3sO4BZsXZlpd3bJa9+HdafHOOVuZkEgV/chegXdRraT
 9AeZjd1KgzJfjVEO0sNCYJ4l+Ct7pX6W2ID9gjL+fFoi4TV5FMr3oLIH8XrRvehWcBTsGuhi
 Eia/U2sV3n2M/Tak1Jp6EmEjPLPhyTkcIQcHrax8rhtmlL77mIJBRAcTlf+sfS/okCkVt9VM
 EBS/TAhxZXe72SuR9j5Ghe+8HiNu0ZAX8IKS7FnrgaQ1qDT/gCVQHAeSSJMY8Ans8lwQiE20
 liOnJXiAjkHXKCpdE9xP4y89VuaURX550dbDcPYZWPpO+Xenbw=
IronPort-HdrOrdr: A9a23:tjjbdqAs9NPSgITlHeg+sceALOsnbusQ8zAXPh9KJCC9I/bzqy
 nxpp8mPH/P5wr5lktQ/OxoHJPwOU80kqQFmrX5XI3SJTUO3VHFEGgM1+vfKlHbak7DH6tmpN
 1dmstFeaLN5DpB/KHHCWCDer5PoeVvsprY49s2p00dMT2CAJsQizuRZDzrcHGfE2J9dOcE/d
 enl4N6jgvlXU5SQtWwB3EDUeSGj9rXlKj+aRpDIxI88gGBgR6h9ba/SnGjr1wjegIK5Y1n3X
 nOkgT/6Knmm/anyiXE32uWy5hNgtPuxvZKGcTJoMkILTfHjBquee1aKvW/lQFwhNvqxEchkd
 HKrRtlF8Nv60nJdmXwmhfp0xmI6kdb11bSjXujxVfzq83wQzw3T+Bbg5hCTxff4008+Plhza
 NixQuixtVqJCKFuB64y8nDVhlsmEbxi2Eli/Qvg3tWVpZbQKNNrLYY4FheHP47bW7HAbgcYa
 hT5fznlbZrmQvwVQGbgoAv+q3gYp0LJGbJfqBY0fblkQS/nxhCvj4lLYIk7zI9HakGOuh5Dt
 T/Q9pVfY51P78rhIJGdZA8qJiMexrwqSylChPgHX3XUIc6Blnql7nbpJ0I2cDCQu178HJ1ou
 WKbG9l
X-IronPort-AV: E=Sophos;i="5.92,305,1650945600"; 
   d="scan'208";a="73626437"
Received: from mail-bn8nam04lp2045.outbound.protection.outlook.com (HELO NAM04-BN8-obe.outbound.protection.outlook.com) ([104.47.74.45])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 16 Jun 2022 12:23:44 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=AdCbDt4b+ve49+23Tdk2uzzcgDMVV/87vWHqA7Doec1eTSSTUPxYMC1z+sgnUAyStN9DrSdao7kQoif89hbaSYy8f9g3hhOsZFa7d/WUGUYzwZAqyeM1VA1T/YlyVzqRIagOWxYE2mu+2MFpq2Qw1ZLUjnv8F1CRtS16JHzznZSb/npyzQcPHZJZ/8T7DzaTLbnbLzeAoq8MS2ut0pToS1a3fxBR8ODZ677bKwMXcJ14/OuBd3SGe+y11TT3OlSIh3xY8tkO3c9nj0r1DfjYZUtyqXE6oSD8mISZODtrO/OUBmpnwMFuvzTPqrW78rFYkkxMVoJbAuS5X0ZMppy5Mg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=j/7vq03CoeJFxaR4kaHv9+LfTuTw9bOkOH/ikjP2zNs=;
 b=Pj+5v5EE/f8i3pqq1GAixVqL23+zR7nBjrkGclW/xoABuBtPWZtGHzs7uvZ0PJd0aPOZHE8XNKQWCYtHw7UbMqJRnTMVd9ygKttty9UviVqHL+mxwHNhCVg5OF6EyQiX98jR8Ae4LpN2rH6Jmh/9pELYxPCKGemrQFZRdEhi3+AFsiRiGixCzvWdQIT44iHf5890T1wp8sZ3kPlZf1WwjOm42l6ryBFcwURucgVmDWheh5fNcBMMCTm4giThHqSUT9UeAcqfd3OZh6Svi4azuJZ2TZmj5YWUcECqXY16z7fSFJ0D7BT1yPHVTv3JYqZIMgmO7BSpSO9pXjR3pZCr/w==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=j/7vq03CoeJFxaR4kaHv9+LfTuTw9bOkOH/ikjP2zNs=;
 b=TPUX9k3fSmGvRjtDNrmtOUYwgzdYVTvq8a4DiPbQx1Tc502IjGg0rsKZ3vta7Wx7GHxg+yXZn90CwEJgpdzAoh/KfJaFq7ZaZcWlTJ2Ha6HUfvAifcthu5ajP2Ig5pRBE6Y4XOq3q64N21gU3f1p2Sn9z2qyvYLSugcWzwh8DYY=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by MW4PR03MB6587.namprd03.prod.outlook.com (2603:10b6:303:12a::9) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.14; Thu, 16 Jun
 2022 16:23:42 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534%7]) with mapi id 15.20.5353.015; Thu, 16 Jun 2022
 16:23:42 +0000
Date: Thu, 16 Jun 2022 18:23:38 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Stable/13 doesn't boot with xen
Message-ID: 
References: 
 <202206081544.258FiwmW012809@nfbcal.org>
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: LO2P265CA0450.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:e::30) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: e5272b4d-b076-4c14-dddc-08da4fb493dd
X-MS-TrafficTypeDiagnostic: MW4PR03MB6587:EE_
X-Microsoft-Antispam-PRVS:
	
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	sFwsl5y0VCWcUaMzoybghFxvba0Jo22sd0rbUNMJFGbkHTFD3ATTWohtvleYqzCZeUAky9/JCHVqLrr9LXV4FCTSo2khy1nW0WIr60NcvPrLre1oi6yM8zAxglm7KKdB5i656iKhAUtA5K0J5XE2AtiMy9IhTm+EA8+7/3+vHZiiBBFmQHTvHUVcHndt8LbY2UVbmR50/F/RNf2HZnWqxpvMabENcZdg7ChP258MecIyPo4XwNrPFgxvk2roeX8O5OSYTBo8ypcBJAuKSXmnkKu6uh9rtNHCAPRLxbFmdNMF6nEBJYMK7Y8VmKG/ZcWmJ62dIvCvH1HfvG24dqDTTPpxp85zBzzoxSJSR+EeDpGyQsMsQH0QVsdiOB9+GSXWN5WlnXg0/Shd7YM8cUB8+NbTl5ueVbVLSs6xcT70dBdC6bzAT3d/9TCPBkR0dPoR+AFgNr53M3zQdFAr1teGoZzwm+kno7/oQG0NtKonwNyCA+Ufyfdt/oiodzuaCeyiYCNVpfNqoCd0cqwi0cnDwh4RgH88GnLyJQhLyUm5UcISS8u061QHrUxk52EGODLwfrlaqRZQmcq8TBYDqUiiyZAPOs/d/QSJD0zU4chtBqrImY4uyarbTxMZW/VNv/xdewwZtMaJJ7k+maigA8YivdYdlTDiCTJpK2EnX9BplEBaJSyew0w+XeozqjcpmYD7V1j8dalz8h/K+i6UJ3N0V5r/L8Yyad9kEk5HHECNTi7aqMYwvLzeUsQRTMfTbWjqZNHwatbej8ZlFszL/ghDRw==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(7916004)(4636009)(366004)(5660300002)(83380400001)(82960400001)(2906002)(86362001)(38100700002)(6506007)(9686003)(26005)(6512007)(498600001)(6486002)(966005)(6666004)(66476007)(8676002)(186003)(66946007)(66556008)(33716001)(316002)(8936002)(85182001)(6916009)(4326008);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ZTQySGFkZlFJeVc1emdmUTFZQ3BsaXMxdTVrblJ1ZkZScE9xVGtVb0paWjd3?=
 =?utf-8?B?M3Btc3g2MnU5a1JTK2N6ZGRpTDNWdXZEeUw3MFl4ZlFZTHUzcm5KeVVlaDN5?=
 =?utf-8?B?cGwxTWJmUUNodCtoa0hiUjZOSC9SeXcxNkdHQnJPTnk2MVdlTHhnMmN0WmFw?=
 =?utf-8?B?VjZuY3k1RDJib0tLMFdLOVNUa0cxSVBTbCtQczBNTGprNUE2VjVMNmNTOUhP?=
 =?utf-8?B?QVVNSDlmMVR4c01CMmN4MGYrWlpjdVlyY3p5eTcrQkM1aDV4dUFCVERVY2ZZ?=
 =?utf-8?B?M3BKV2s3c3FDYjVtZm40Qm1LRmM4bnNYZmx0YmpYbGEwUVJsekZDSUxuN2xB?=
 =?utf-8?B?dHdoRW5QSGRFOUlvK2tIenhpeXZJTE1nYnVoem1JSWZKRWI3SkNwRDNvS05q?=
 =?utf-8?B?S24wMkNxc1phRWJ3YzVOSUZGcWs2ZVVlMU0wdUhRVXV5OVZwMUw3WGIzTXBT?=
 =?utf-8?B?dEFRMTVpK1hQSm5yaVFkd1JkUmIvUWthcGp0THI4czFRcm1MRUx1WmU3QTdS?=
 =?utf-8?B?dUtWNHBqLzZYc1BQR29EZllDdHA0QXFMOUpldkNFYnRXRHl0RDB4N2JEVkta?=
 =?utf-8?B?QjBNVTdDMU5CUnJvK3VnR2FpZE5MeCtLVW5mdEpwdlBmSFcrOWJWZDFPc1Zt?=
 =?utf-8?B?QU96aVJXcnA5RVRsZWFoaFlWYjFBTGhyL2hRbTNLcGVlNjc1d2d4clZzQmNV?=
 =?utf-8?B?MCtUSjJaak5weENKS2puakJyc2c3VXJWWFkva3EwU09LYlIySDZJRFZUODQ2?=
 =?utf-8?B?ZzZ5VEtwYnFQcXBMWHpKZ3pNNDdQNTVIbXVzY2tybEdNRktHZ293SlJ4Z1NI?=
 =?utf-8?B?WUo0WDJpTCtCcUFaSnhwMTFUTm9ZZGxiMlMyMGlxKzFtTi95eWU2aVhTQkhN?=
 =?utf-8?B?T2MzU2Q3SjdRQ1UwMU5iWEs4NmhseFZwbk5za3kzNnRkMmFWcnZ3R2JqdlhT?=
 =?utf-8?B?NTZFSElQSWgwNGFkWkYydTI1L0Z6Zkp6bC8zRlBWZkpMZHZHTEpXc3IrTmpL?=
 =?utf-8?B?NGdZT002RTFjUm1HWnl3MFFGTTYzdTBpQmdjR1kyZVZZcnBLR2pQTUZJandV?=
 =?utf-8?B?NzVONkJFUDd4MHpZSnhVcjA0OFVwQlc0UkRDR2RTWUZLZVd0MmVYMFhLRjJK?=
 =?utf-8?B?Qnl3YkowT0pLb1pTQmhiYTMxZVJnRW9DaStFZXYxWjlCL0d3N1RONk1kcWFt?=
 =?utf-8?B?RkZYVUNhVnVPTHVqNitEdnpLQXBMekx2Y1pQVnBXbjBSNnlGem1Kb1pRWjRE?=
 =?utf-8?B?MjRYQmxnQlp1VFB3Kzk4Y1dvNXQ1Z1lVeUoyQkp6OHRMVVNUU1h4cWUwR25W?=
 =?utf-8?B?dE82RldScUV4SW9CcHVVcDlxK3JUcGNpL0VJSEwzVlhjYUtkVkJRSnZwVkg3?=
 =?utf-8?B?dlVhWjhhMkJCWFprSVUyU0lCTU1KN3ljcGhhd2FHZGM0eFpvQ3B4K1dnbFBn?=
 =?utf-8?B?d0F1aUQ0WlpvRE04ZGFCK3ptajAyL1RiZXU0aUkvVW4zWkdTdTNzMEpyNm1r?=
 =?utf-8?B?b3VGdURIdnA3bHdyTVNxVDc5bWlncXJGUkRSbFFGaVlzMWVSZUJieUhMZnZE?=
 =?utf-8?B?ZXVqdGZQWmdzQ1FCVThiZjZQTjlMWG5LWnJCcjZubkZveFpUN1BFYzhrbEcv?=
 =?utf-8?B?Sk5DcGhMUGNqajllSEg1Rkd6bktzckF4VXV5NzM4bEMvajNqSFlQMVVyREI1?=
 =?utf-8?B?Z1hwS0ZmQUhaRmp4M205UUxDbXltNE5YNmRhNUtOT3V3ZGRDMFZ0REpGOGRw?=
 =?utf-8?B?N0FPdzRzanVZSENEWHdXYzYxa0FnMmxOZ2NJVC9VdHNlUG5YS2NzS0xYaXF0?=
 =?utf-8?B?RXo2dTJPVXpMVlFYK2FFaE1xQ3NNSHpvcmJyaUFSNFZZZUNXc1NOYXBSQVBv?=
 =?utf-8?B?cHNJYXhXMXVXemx2ZzJwbFAvbmgrdk9zQzlXSWdMeWlHc2Z5TmJXNFlaOE1N?=
 =?utf-8?B?anVEdE5Gb3hYLzJpMmJaakFzNUtPK1ltMC8rbGxkNmg5MUQ3QWNvTE0xSmNW?=
 =?utf-8?B?QTdEZHRISDFBbWFZK0FXbllVU2JQa2JUUzJtNjVhS2hJUVpzTmZhcmRUMGFn?=
 =?utf-8?B?QXJqRWRBRFN3RGQ0OGFkNDhVQlkvTE9KamFmSkxHY3E5dzBDZU13allmaHNQ?=
 =?utf-8?B?dHhvWktNM3l4V3NDNEZ6RW5TSW1jbXRDVTNSNnovWnQxdER1ZS9LbnlmQndO?=
 =?utf-8?B?VGh4Q21IUWJETHpURGRkMTJxWDBHWTB6eVo4TmV3UHNSQmlBV3J4akRGeGkz?=
 =?utf-8?B?VlJPN0RnRFprcUhDYk9PeFU3WWpicVg1V3R3MkV3cWF6ZjlWY3k3NFRWd2cy?=
 =?utf-8?B?dXVTTHVzNnV6Z05BT0prQnlON0dqR25ZRUNlQURvMFJncWFjZkV6MDM0WS9G?=
 =?utf-8?Q?PdiVWlmw6k1roXZo=3D?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e5272b4d-b076-4c14-dddc-08da4fb493dd
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 16:23:42.1875
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DKbS5M3wb0vyEGxVuUrhEQnoSpdk89DHlRcCRh9SPUUaIoUfA4FdcXqyG1V1+sB5/KWGKHbQ5jpzNeBCdzeOIA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR03MB6587
X-Rspamd-Queue-Id: 4LP6tf0KT6z4dyM
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=VDzN+ozL;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=TPUX9k3f;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=1593354c1=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=1593354c1=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=1593354c1=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-0.998];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Jun 09, 2022 at 08:51:04AM +0200, Roger Pau Monné wrote:
> On Wed, Jun 08, 2022 at 08:44:58AM -0700, Brian Buhrow wrote:
> > 	hello Roger.  Okay.  Another question, which might be one you answered, but I don't yet
> > have enough git knowledge to understand it.  Since releng/13.1 boots and runs fine on my
> > machine, but stable/13 does not, is there a way to compare the differences between the two
> > branches?  If so, what do I need to clone and how would I instigate such a comparison through
> > git?  Or, is this just a matter of diffing the two source trees the old fashioned way?
> 
> 
> You can easily get a diff from the git tree, ie:
> 
> $ git clone https://git.FreeBSD.org/src.git
> $ cd src
> $ git diff 0bbbd845fd32ef07707c03d1dc4e4b27f6ebef83..fc952ac2212b121aa6eefc273f5960ec3e0a466d sys
> 
> Note the two commit hashes: one is HEAD from releng/13.1 and the
> other is from stable/13.  I've added the 'sys' at the end to limit the
> diff to changes to the sys/ folder (so kernel only).
> 
> In any case, I think such diff is too big to be able to spot any
> changes that could cause issues by code inspection.
> 
> If you can do the bisection and provide the info from the debug key
> (plus the kernel debug symbols) that's likely to be quite useful.

Hello,

Before I forget, do you have any more info from the issue on
stable/13?

Thanks, Roger.

From nobody Thu Jun 16 17:27:22 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AFE86857915
	for ; Thu, 16 Jun 2022 17:27:37 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LP8J83S9dz4s7k
	for ; Thu, 16 Jun 2022 17:27:36 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 25GHRMav006112
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 16 Jun 2022 10:27:23 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 25GHRM9U028508;
	Thu, 16 Jun 2022 10:27:22 -0700 (PDT)
Message-Id: <202206161727.25GHRM9U028508@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 16 Jun 2022 10:27:22 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: Stable/13 doesn't boot with xen
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 16 Jun 2022 10:27:23 -0700 (PDT)
X-Rspamd-Queue-Id: 4LP8J83S9dz4s7k
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-3.29 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 FROM_HAS_DN(0.00)[];
	 RCPT_COUNT_THREE(0.00)[3];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 MIME_GOOD(-0.10)[text/plain];
	 DMARC_NA(0.00)[nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-0.99)[-0.993];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  I don't have any additional information as yet.  I need to git clone the
FreeBSD source tree and then do the bisection.  Unfortunately, I'm in the midst of updating my
production environment and I'm about to leave for a week of vacation.  So, I won't be able to
do this for probably at least a month.  By then, it may be worth trying the current stable
again, as the issue may have been fixed.
-thanks
-Brian

From nobody Fri Jun 24 01:30:56 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DA61B873AA3
	for ; Fri, 24 Jun 2022 01:31:07 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LTfhp4fR9z3lDp
	for ; Fri, 24 Jun 2022 01:31:05 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 25O1UuM1007299
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 23 Jun 2022 18:30:57 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 25O1Uul3002911;
	Thu, 23 Jun 2022 18:30:56 -0700 (PDT)
Message-Id: <202206240130.25O1Uul3002911@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 23 Jun 2022 18:30:56 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: Some kind of race condition in adding and removing domu's causes vm zombies
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 23 Jun 2022 18:30:57 -0700 (PDT)
X-Rspamd-Queue-Id: 4LTfhp4fR9z3lDp
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-2.44 / 15.00];
	 MID_RHS_MATCH_FROM(0.00)[];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 FROM_HAS_DN(0.00)[];
	 R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 TO_DN_NONE(0.00)[];
	 DMARC_NA(0.00)[nfbcal.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 NEURAL_HAM_SHORT(-0.14)[-0.143];
	 RCPT_COUNT_TWO(0.00)[2];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 FROM_EQ_ENVFROM(0.00)[];
	 R_DKIM_NA(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  I don't have a lot more details on the issue, but under xen-4.15 and xen-4.16 with
freeBSD-12 and FreeBSD-13, it's pretty easy to end up with zombie domu's that are unkillable
and unrestartable. Even worse, the block devices associated with these not-quite-gone domus'
are unusable with other domu's without an entire system reboot.

	How to reproduce:

1.  Shutdown a vm that's currently running, I'm using NetBSD, but FreeBSD domus' wil
demonstrate this behavior as well.


2.  If auto-restart is set in the domu's conf file, the domu will restart with a new domain id.

3.  Just as the newly restarted domu is coming up, issue:
xl destroy 

You may see output like the following:

root# xl destroy 20
libxl: error: libxl_device.c:1111:device_backend_callback: Domain 20:unable to remove device
with pa
th /local/domain/0/backend/vbd/20/768
libxl: error: libxl_device.c:1111:device_backend_callback: Domain 20:unable to remove device
with pa
th /local/domain/0/backend/vif/20/0
libxl: error: libxl_domain.c:1530:devices_destroy_cb: Domain 20:libxl__devices_destroy failed

Now, issue:
#xl list
(null)                                      20     0     1     --p--d    2083.7

The work around I've found for this issue is to shutdown the domu with the -h flag, causing the
system to wait for a final keypress on the console before rebooting.  Then, while it's waiting,
issue the xl destroy command on the old, waiting, domain ID.

this work around will prevent the issue, but it's my view that I shouldn't be able to wedge the
destruction process in this way such that the entire machine needs to be restarted.  Being able
to do this makes the system rather fragile.

-thanks
-Brian


From nobody Tue Jun 28 11:38:59 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EE558860D00
	for ; Tue, 28 Jun 2022 11:39:14 +0000 (UTC)
	(envelope-from prvs=171720f04=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LXN0d4lWWz3PHB
	for ; Tue, 28 Jun 2022 11:39:13 +0000 (UTC)
	(envelope-from prvs=171720f04=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1656416353;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=MX2Ea28VMxfqw3owlk/KxtbUFjLgoxbWrOH8HHFjW5o=;
  b=gNtlcDQfkFk4rTWNYjgv83tp20eo3NZoF4zD7hr7Hw8EBGznoTWV88kR
   568U/csVfc7YqMUZNpPPy8d/FWwCcLdHbg0zwwz7Qt8VEc6Jf0XNiuCcp
   spJzM/XWrhrsVlkc5qLJk3esFSEuY8gdPQAEQ9qHSi3Ec2GUb8rOBOIB1
   U=;
X-IronPort-RemoteIP: 104.47.59.171
X-IronPort-MID: 74576258
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:i43AM6BJxHEHMxVW/3Liw5YqxClBgxIJ4kV8jS/XYbTApG9zgzYCz
 jYfCmDUPa6Na2L3KNwiaYzk90hT7JTRzYQyQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA
 xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgH2eIdA970Ug5w7Bj0tYx6TSEK1jlV
 e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH
 I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPh35
 4wK5M3vZjsSO5XqqvY2QQEHMXhxaPguFL/veRBTsOS15mifKT7A5qsrC0s7e4oF5uxwHGdCs
 +QCLywAZQyCgOTwx6+nTu5rhYIoK8yD0IE34yk8i22GS6t3B8mcGs0m5vcBtNs0rtpJEvvEI
 dIQdBJkbQjaYg0JMVASYH47tLj02yahKWAIwL6TjbIw4XmO8hJJ6ZrGLsv3Z97SSeFwn0nN8
 woq+Ey8WHn2Lue3wyeF6HWzrunBlyjyXMQVDrLQ3vBwjVCZ3WBWGBAQfVWhrPSzkUL4XMhQQ
 3H44QIrpKk2sUCuEN/0WkTipGbe50FFHd1ND+c99QeBjLLO5BqUDXQFSTgHb8E6sMgxRnoh0
 Vrhc87VOAGDeYa9ERq1nop4ZxvrUcTJBQfuvRM5cDY=
IronPort-HdrOrdr: A9a23:YIId66xAWY3a+Q360uNZKrPxyeskLtp133Aq2lEZdPULSKGlfp
 GV9sjziyWetN9wYh4dcB67Scy9qFfnhOZICOgqTM6ftWzd1FdAQ7sD0WKP+UyCJ8S6zJ8n6U
 4CSdkDNDSTNykcsS+S2mDRfbcdKZu8gcaVbI/lvgpQpGpRGsVdBmlCe2Sm+hocfng9OXN1Lu
 vq2uN34x6bPVgHZMWyAXcIG8DFut3wjZrjJToLHQQu5gWihS6hrOeSKWnQ4j4uFxd0hZsy+2
 nMlAL0oo2lrvGA0xfZk0ve9Y5fltfNwsZKQOaMls8WADPxjRvAXvUWZ5Sy+BQO5M2/4lcjl9
 fB5z8mIsRI8nvUOlq4pBP8sjOQogoG2jvH8xu1kHHjqcv2SHYREMxan79UdRPf9g4JoMx8+L
 gj5RPVi7NnSTf72Ajt7dnBUB9n0mCup2A5rOIVh3tDFaMDdb5qq5AF9k89KuZLIMvD0vFpLA
 BSNrCT2B4PGmnqLkwx/1Mfi+BEZ05DUitvGSM5y4yoOzs/pgEL86JX/r1dop46zuNCd3B13Z
 W6Dk1WrsA+ciY3V9MJOA5Te7rCNkX9BTTxDUm1HXPLUIk6BlOlke+G3Fxy3pDjRKA1
X-IronPort-AV: E=Sophos;i="5.92,227,1650945600"; 
   d="scan'208";a="74576258"
Received: from mail-dm6nam12lp2171.outbound.protection.outlook.com (HELO NAM12-DM6-obe.outbound.protection.outlook.com) ([104.47.59.171])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 28 Jun 2022 07:39:05 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=hd9WtBhitvnaAxpGW86cx7fdLc7r0u/s5TW05Fc1j7GH6gE+55npOynzEE2/rTjrEWErLrtQEnLQ9JzDNyCR0gTTQeKm+Z5I2lX1xizjJufdZbI1kHOLRu9sJiO5VKExrp+44Oj1aRDeUArKmXNTubZKRfoqMXzaQNyaRq+JTvBD07NsIYpwfzH7wk9KoELkmX/AfaPPEdKsRpc5lmfAUinScJtrUFTJkdF33qVETYaEgpg2MvZaclvwWLLKe6Lyx0ckGXThzQoysPvwdQZsfEgI/w2ppE47EW3chCWJiUfRYHPSq8YEvhd+I42opIVKj4pIxjJtdWKzRdg51PHJWQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=A6XOuLUMAOj4k0x8ZtgJGEoX8fDPZ1I50dmf2Bp8gfs=;
 b=iCBOfhCaBDheBsV6PjvVxWBQa59QiYy8MYX8gkhPiNuNxNYsrUr4Mvlv61k4EVFID7Rs/HhAVqK+8Wu6PoEmXH5ntRk6Mjk7mvcx5wJvz3Gz4Ef1OqIfEdvv5Yw9bky9uyWtbh2QAmjmI+D9DrkUqPkFIkmJ0qVZeCPOK+UrwOO/7bePT188hP/vqylTychGbNIHtlCPPXZ1eRnR/phWjuFrLOx9quTuYSpvCofT0BkxobBmfYPajmtYN/r35cUOHtP5T7iv+2s3u2C/ULt1AAVmrOJitFuuwQi1M6HEiJd/EuR6f/ygVlarSGGS1Ssr2Jpah5Fx8tRo8rgHOaC+qg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=A6XOuLUMAOj4k0x8ZtgJGEoX8fDPZ1I50dmf2Bp8gfs=;
 b=DxsoU34DFO9gX67iuWUPReCjashFawqO+6sLKRpUm8MAnaPWvrBtrhry4I1NOFUoWeu5ZecIZaUxkfn2PgCDvDWG9ufFgFnkF+ipSHX3vyKMx28d9nZLWAbAObNNzshvc0TBD6uhhUQExJSxQALpz5y75nlXS5dWATlXjRGiUbM=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by MW5PR03MB6981.namprd03.prod.outlook.com (2603:10b6:303:1a8::20) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5373.16; Tue, 28 Jun
 2022 11:39:03 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534%7]) with mapi id 15.20.5373.018; Tue, 28 Jun 2022
 11:39:03 +0000
Date: Tue, 28 Jun 2022 13:38:59 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: Some kind of race condition in adding and removing domu's causes
 vm zombies
Message-ID: 
References: <202206240130.25O1Uul3002911@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202206240130.25O1Uul3002911@nfbcal.org>
X-ClientProxiedBy: LNXP265CA0060.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:5d::24) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 06891347-deba-41cb-9f50-08da58facd1d
X-MS-TrafficTypeDiagnostic: MW5PR03MB6981:EE_
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	E+S47Dwy/zNZmIrCPEM8mvpuXr5lHp0JU3GWeRY65RHEgPob4j0fL7yHEhnjc9IQv71rhyAeCQWZFT5R0pj7lDf/71cRRCQiewb5BpDiBN1vBpTir+mudRavGf25msvV+C19osBKQPVQbfcgL9NOCRU7opaqz3bfmO2h/2Qmtz4u2XugGSrIWDk//oVd/3Gajhi1rUPuhjf8ZR+010snvUWSODIuVojTaBuIFtiXY6P4iPsHRur1Ww0tGr8RpQJglVNbMA+BWKnO63tITMxy3g1L00Zw9/ryNs+B7sxHMA3QD3AEZ0WZfmCYsPTg9W0p2vsCv58zkDZm+tgomkHc0WSK7FGF+PDQS61nb/EnVd4O/NPHM4EVpvjp8+0hFI8GiSO2KcHdecE3RJd5BXtLqy60eZIFOOX8hkfFov5Pj1TTGgKQfWI3oD5ODeqG7I0f9DYrGAz2KNVx/YvbOLuGexvov/Col3QSl6+/gGsvhbs7SRAVfUF+zOh2933ZlaROEiQ2z6D+dzCGC23m3fq9BqIh3MQQNXEGjVhhlDeD6AuPXA/Zm3x52pv5JdqQRsTdc/2U6LzdcjRtpHEkaVCgAdcOkh1mq4SSV1c8P4iZnUR1koTg022PPdinxVHAO1UVJRPNk6p1zEW1q0YiNGDX1b6LD2PRzMOijHYnyJ5EhKJXP8Bhp6UNaJOpoK/R849BdGX6W7c7WLlreTDOR9BizCmjo3R57Wldiy+ikHJorsm4AZjBkmKpveUXIe7LjoYc
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(7916004)(4636009)(136003)(396003)(376002)(366004)(346002)(39860400002)(186003)(66476007)(316002)(33716001)(66946007)(83380400001)(5660300002)(8676002)(6486002)(38100700002)(9686003)(478600001)(41300700001)(82960400001)(86362001)(85182001)(4326008)(6666004)(6506007)(6512007)(8936002)(66556008)(2906002)(6916009)(26005);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?c0lta0srcW9SNk9rWEpyNU1GeVhkYTZ4dktBTFBHR0dhc3lpeFA4eERlRVRl?=
 =?utf-8?B?VExYVWRaZGF4NDdIVW1HMWN6WlpQLzlraGIvNll0WnZKVVhLOERlWmRpZzVZ?=
 =?utf-8?B?WkxvNGNqTUtPTkk3RE94blV4RXkrR3NsZm9lWGZOYVBRVDlUSmlyVmF3MWdr?=
 =?utf-8?B?KzhZNW02aEQrbVRBQUx0dUE2cFJYNVBiRmNjOUlXYjd4QythYWNQMVZTZjB1?=
 =?utf-8?B?V0Izbk1iMkNXa3Q4Y1diWUwvbnZPbHh0a1FGQmYwRmdrSzVsMWNrTXZoa3ZH?=
 =?utf-8?B?azkzTkdGaXpMSk1MVklDSXArY3VRakVGditZYVJHaUdnS0ljN2x5Q21JQW5G?=
 =?utf-8?B?VDMveE9zeVdSZmZZVldRZlZCdW5SRVlRSkErZ1BJRXhlejZZSzdMV2lreTBC?=
 =?utf-8?B?U05SSnd0OVZJR0Q0T0pObGJyOFNLdnJDTGRHZ0ZMbHpveHJGRmM0S1Nod1hs?=
 =?utf-8?B?R0FLbzJrRm8wSERSYzRpVGRqZkZqTUl1QU5NR0dJZ2pFRk5BUTFLQ2ZnbURE?=
 =?utf-8?B?LzBtVjFHNThnRVo2cTc3QWlTUWxhUkRVSGNFcERONVFDWGhvSURZVmcxTndD?=
 =?utf-8?B?YitNNlp6eEZpWng4ODhMeE9TUi9kSVBkVnhOdXlmaG8rdXhzSTNGaEtENXlV?=
 =?utf-8?B?eFN3YndwWU9ndENnaUg2Rmt0RGdHb2dtSjMxWURocnIyN255ZUczbXhadjhT?=
 =?utf-8?B?ZVRvMVhHaW0rdkV1amtVYkJ1b2hnVVVxOFc3U0FXaW9kQTVqdWZsVkNiWk1V?=
 =?utf-8?B?Vkc1THZURXJEMjl6TnM4SWE0ZlBDTlRBK010Z0VzSUhDQUdHbjdnTEhlR3pp?=
 =?utf-8?B?cmF5RjluME56T3FmSEVRN2psUkVRM0F1bDMwU0YwZTJnSi80WGl0WENwLzhi?=
 =?utf-8?B?YUZWWnRHenNVNFZtVXh2dFBVbnJzV1J3S0J4eExXZmFIREhoNFJwZE5tUWF2?=
 =?utf-8?B?VjQ0NGd3MlJTWkFScFZMbHdGbVNTMDQvV3NYcHRhazF4MUpETU4yM1hJdHNN?=
 =?utf-8?B?S3VIM1FHS2kxSkVFdUk5RUxMQTg2ZnNQcS8rU0pYR290NHVzSk9RSmNydlBU?=
 =?utf-8?B?dUxaL3F2dzdNZ1ViRFVtV2tZNllOY0NjbG1nOGlMa1duUEZBVlBqV0ZSMkFW?=
 =?utf-8?B?QW5keS9ZbWtFclRlcXh5QkRTWmpHT2lVSi85UjVVcDgySVdJcDJaZncvREY5?=
 =?utf-8?B?K3NGMmlITmFaSHMvV1hVZStIZkhVTUZVVGN4QTZXNkFQV1RhMkFRSW8xZHhz?=
 =?utf-8?B?L2IyMGsvQlppaTdBeGErbjBrT3E5TS9pODNLMUNJTHI1Z1R1U3RCYmIwdkl6?=
 =?utf-8?B?T3lTS2ZUc2xFaHZRUnVEdTZ0TEFUc3pWVlA1NDc4NzJYWkxzWWtmMHd3SjJ4?=
 =?utf-8?B?Q2tmMGNha2pUeHpMRjJSRGZlMExLMHA5TUJhd2xBRmpZMTFJTUJCcTFiV1Ay?=
 =?utf-8?B?b0xWR2I4bm5hWk0xKytDNUxlNE9adkZDam5aREJZM1FkWDZLWXF3Nm51bitV?=
 =?utf-8?B?T0EvTGpnMzk5UGdHMFZqY0VIQ1BTa1J1NEsrT3ZLM0h5Q0VSVmUvZHpXZVhP?=
 =?utf-8?B?UkovMDUvZXlmRnN1SWUvdGtEbDJzT3I0dE0xSWxaK0RoK083VGx4RXZNYW9Q?=
 =?utf-8?B?TjVFWTE3UVZYT2xJQjI4VnRCbW5FbnVtdkFaSnhHRDd6Z0RUdHR0bndhcWJP?=
 =?utf-8?B?aXFFWWhvcXZKK1RINDFoOXkrZ0pYa0JHa2xtQnNHRU9DdkZnUFhSMzA1SHgz?=
 =?utf-8?B?QWZnZUlPNTNqMVlvQ3gzbUdlZHViTTdrREdxdFJINXh6VDREU2F2enIwdE5N?=
 =?utf-8?B?S3RFYlBMYVVycUxSY2UydjhHS1JyRkE0bGZKNkIxOFk0WlBxOXZpa1dwU3R5?=
 =?utf-8?B?Z29tMnl3TXAxWVpVbDVHT1VmN1pNSkdjK0l0enVXZGlBc3gyMVFMZWl5TGhl?=
 =?utf-8?B?eDJyUDZSVkdoY3FrdjF1VmFoMFZBbWhrZFNONEdZakZ1MlpZSStEWTR1RGda?=
 =?utf-8?B?OGZld2FRNXFyWmxMM1ZMV1h4YzI4N1Z3TWNNNXJibU5ua2FCWVVFM0lvQkJa?=
 =?utf-8?B?OWtUVnNwNm1OSlI1VG9qOHVQS2tvWkFmaVFXdW1BR05zdkxEdFpYdXdVODFB?=
 =?utf-8?B?YmplaWdsNnJKTDgwdGFqZjR3ZnA4S2FNc24wWXdJZzB3KzNYdnVWcFg3OXpj?=
 =?utf-8?B?d3c9PQ==?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 06891347-deba-41cb-9f50-08da58facd1d
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2022 11:39:03.5184
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: t41DOJFfRSic0Gbhx7hdsh3Su4M147Vb8aNr0fTnn37I7rWgafhW55gJnBl80wtyp19jh3aNbkZGBhwe6/U8bQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR03MB6981
X-Rspamd-Queue-Id: 4LXN0d4lWWz3PHB
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=gNtlcDQf;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=DxsoU34D;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=171720f04=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=171720f04=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	 TO_DN_SOME(0.00)[];
	 R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	 RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	 RCVD_COUNT_THREE(0.00)[4];
	 RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	 DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	 MIME_TRACE(0.00)[0:+];
	 RCVD_TLS_LAST(0.00)[];
	 ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	 FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=171720f04=roger.pau@citrix.com];
	 ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 FORGED_SENDER_VERP_SRS(0.00)[];
	 DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 MID_RHS_NOT_FQDN(0.50)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Jun 23, 2022 at 06:30:56PM -0700, Brian Buhrow wrote:
> 	hello.  I don't have a lot more details on the issue, but under xen-4.15 and xen-4.16 with
> freeBSD-12 and FreeBSD-13, it's pretty easy to end up with zombie domu's that are unkillable
> and unrestartable. Even worse, the block devices associated with these not-quite-gone domus'
> are unusable with other domu's without an entire system reboot.
> 
> 	How to reproduce:
> 
> 1.  Shutdown a vm that's currently running, I'm using NetBSD, but FreeBSD domus' wil
> demonstrate this behavior as well.
> 
> 
> 2.  If auto-restart is set in the domu's conf file, the domu will restart with a new domain id.
> 
> 3.  Just as the newly restarted domu is coming up, issue:
> xl destroy 
> 
> You may see output like the following:
> 
> root# xl destroy 20
> libxl: error: libxl_device.c:1111:device_backend_callback: Domain 20:unable to remove device
> with pa
> th /local/domain/0/backend/vbd/20/768
> libxl: error: libxl_device.c:1111:device_backend_callback: Domain 20:unable to remove device
> with pa
> th /local/domain/0/backend/vif/20/0
> libxl: error: libxl_domain.c:1530:devices_destroy_cb: Domain 20:libxl__devices_destroy failed
> 
> Now, issue:
> #xl list
> (null)                                      20     0     1     --p--d    2083.7
> 
> The work around I've found for this issue is to shutdown the domu with the -h flag, causing the
> system to wait for a final keypress on the console before rebooting.  Then, while it's waiting,
> issue the xl destroy command on the old, waiting, domain ID.
> 
> this work around will prevent the issue, but it's my view that I shouldn't be able to wedge the
> destruction process in this way such that the entire machine needs to be restarted.  Being able
> to do this makes the system rather fragile.

Hm, I don't seem to be able to reproduce this on HEAD.  Could you give
a try to a HEAD kernel and see whether you can reproduce? (keep the
same userland, that should be fine).

Thanks, Roger.

From nobody Wed Jul  6 08:47:32 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DA79A1D06F97
	for ; Wed,  6 Jul 2022 08:47:45 +0000 (UTC)
	(envelope-from tdtemccna@gmail.com)
Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LdCq52vwkz4stV
	for ; Wed,  6 Jul 2022 08:47:45 +0000 (UTC)
	(envelope-from tdtemccna@gmail.com)
Received: by mail-lj1-x22c.google.com with SMTP id j23so849101lji.13
        for ; Wed, 06 Jul 2022 01:47:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20210112;
        h=mime-version:from:date:message-id:subject:to:cc;
        bh=qNJS8bXJjIJEOjJk4e4xeTOXLNTgbklFiRogoFnIe/0=;
        b=OqYyZVUpMpeMyHl8dpmvGlDFd2/XZwMeBkC+wCRUYKWHOLg9mLp9e/qRZts9XnTR0k
         c/XOHqPh4deZBBm6OxM74dxC6hnm95Q8k6Et0o1jwXbcgznSk8VqJ8uc4D3YsFQDi3Pi
         KgVegS8sSQ2TB2eOD26Vxj47EXwu6tKVMnyhHoGPVAHfUjwl22w5TjnSD6+J6h/SzMJZ
         NW7ihAFXuU30SXYfeC5Sy5NmKRIoCFxfcoVnsYdd45YjrpPQgqKdDylplbqQ4T5c8Fuw
         CKjkcwTZuBtwmS8ZsqKarO+qLZvIoq7qmgm6BGcEN4NUZfFrjrwvkSZc2Tqpdmhgig/i
         7Xaw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc;
        bh=qNJS8bXJjIJEOjJk4e4xeTOXLNTgbklFiRogoFnIe/0=;
        b=MVnMMSB0vlYjI3kOsOyY/ECL8yUAXhd1nuiHq3TMPQ6ePo8OSgQvvQWGIORDrsf51D
         VlN7KlmKI42MNFrlPS2jYyrLLcUH8C9Ir4CRpHpAoGHeZ/3Ho/EPDeg6pT3Q4fKwbq/z
         IA0/+iw9JENAC1b/GLlkwD26LvdEfhDxp7HUfk3RntAptggbpSciYSFE5ywAH5FnOjho
         DKllrSCI24oys44T4d+FCAz1e664cWGBzZ5BKtYdMrZoXDuFVC4kUOx4W9Dt+I174EXM
         f5fe1EblyA83IFMUHGkSrSB/OMZ7/c7uhigIJk+J6aUcHLJ8K+JLk0+jPYRspz/KZ5fO
         U7uA==
X-Gm-Message-State: AJIora9k9lBU1fUppQ9co+C1DMmfSMoE46xMFAiMYBlcIBW5mCxeYMpg
	b/Pa+xf8k57NvERSqtqvrA2RJmebtCVqBP5x1ULRFj53a40cEZCU
X-Google-Smtp-Source: AGRyM1sx5A6cGpznE2LcOLA6FPbAd1RAS7rmAaTrTV5cjp/7sy8/yy3U1eGnfZEkhIBbs4jlcqSuy4vgpdJjyXGnObo=
X-Received: by 2002:a05:651c:10b8:b0:25a:9bc0:2110 with SMTP id
 k24-20020a05651c10b800b0025a9bc02110mr23102196ljn.234.1657097263941; Wed, 06
 Jul 2022 01:47:43 -0700 (PDT)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
From: Turritopsis Dohrnii Teo En Ming 
Date: Wed, 6 Jul 2022 16:47:32 +0800
Message-ID: 
Subject: FreeBSD is a great operating system!
To: freebsd-xen@freebsd.org
Cc: ceo@teo-en-ming-corp.com
Content-Type: text/plain; charset="UTF-8"
X-Rspamd-Queue-Id: 4LdCq52vwkz4stV
X-Spamd-Bar: ---
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20210112 header.b=OqYyZVUp;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of tdtemccna@gmail.com designates 2a00:1450:4864:20::22c as permitted sender) smtp.mailfrom=tdtemccna@gmail.com
X-Spamd-Result: default: False [-4.00 / 15.00];
	 R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c];
	 FREEMAIL_FROM(0.00)[gmail.com];
	 TO_DN_NONE(0.00)[];
	 MID_RHS_MATCH_FROMTLD(0.00)[];
	 DKIM_TRACE(0.00)[gmail.com:+];
	 RCPT_COUNT_TWO(0.00)[2];
	 DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	 NEURAL_HAM_SHORT(-1.00)[-1.000];
	 SUBJECT_ENDS_EXCLAIM(0.00)[];
	 FROM_EQ_ENVFROM(0.00)[];
	 MIME_TRACE(0.00)[0:+];
	 FREEMAIL_ENVFROM(0.00)[gmail.com];
	 ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	 DWL_DNSWL_NONE(0.00)[gmail.com:dkim];
	 ARC_NA(0.00)[];
	 NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	 R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112];
	 FROM_HAS_DN(0.00)[];
	 NEURAL_HAM_LONG(-1.00)[-1.000];
	 MIME_GOOD(-0.10)[text/plain];
	 PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	 TO_MATCH_ENVRCPT_SOME(0.00)[];
	 RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22c:from];
	 MLMMJ_DEST(0.00)[freebsd-xen];
	 RCVD_COUNT_TWO(0.00)[2];
	 RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

Subject: FreeBSD is a great operating system!

Good day from Singapore,

I think FreeBSD is a great operating system! I support FreeBSD because
the most popular pfSense firewall, the extremely popular OPNsense
firewall and the BSD Router Project are all powered by FreeBSD! macOS
is also based on FreeBSD!

I use pfSense community edition firewall in my home. I am planning to
try out OPNsense firewall next.

I will continue to support FreeBSD! It is a great operating system!
FreeBSD is a very good network operating system.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
6 July 2022 Wed
Blogs:
https://tdtemcerts.blogspot.com
https://tdtemcerts.wordpress.com

From nobody Thu Jul 14 03:22:09 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 979251CFE98D
	for ; Thu, 14 Jul 2022 03:22:20 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Lk0Cv2v5Qz41Xy
	for ; Thu, 14 Jul 2022 03:22:19 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 26E3M9Uf011112
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Wed, 13 Jul 2022 20:22:10 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 26E3M9Wr025120;
	Wed, 13 Jul 2022 20:22:09 -0700 (PDT)
Message-Id: <202207140322.26E3M9Wr025120@nfbcal.org>
From: Brian Buhrow 
Date: Wed, 13 Jul 2022 20:22:09 -0700
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: freebsd-xen@freebsd.org
Subject: How deep is the xl socket queue?
Cc: buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 13 Jul 2022 20:22:10 -0700 (PDT)
X-Rspamd-Queue-Id: 4Lk0Cv2v5Qz41Xy
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-2.30 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	NEURAL_HAM_SHORT(-1.00)[-0.998];
	R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	MIME_GOOD(-0.10)[text/plain];
	MLMMJ_DEST(0.00)[freebsd-xen];
	FROM_EQ_ENVFROM(0.00)[];
	RCVD_TLS_LAST(0.00)[];
	MIME_TRACE(0.00)[0:+];
	R_DKIM_NA(0.00)[];
	ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	RCPT_COUNT_TWO(0.00)[2];
	RCVD_COUNT_THREE(0.00)[3];
	DMARC_NA(0.00)[nfbcal.org];
	ARC_NA(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_HAS_DN(0.00)[];
	TO_DN_NONE(0.00)[];
	MID_RHS_MATCH_FROM(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello.  Recently, I've started seeing messages like the following:
sonewconn: pcb 0xfffff801074d7800 (local:/var/run/xenstored/socket): Listen queue overflow: 2
alread
y in queue awaiting acceptance (1 occurrences)

I figured out what causes this.  I have a number of automated scripts that use the xl
command to do their work.  Sometimes, multiple scripts run at once, causing too many instances
of xl to run simultaneously.  My questions are:
What is the default queue depth for the xl socket listener?
And, is there a way to either increase the size of the queue or implement betterparallel
processing?

-thanks
-Brian

this is under xen-4.16, 
FreeBSD xen3.via.net 13.1-RELEASE FreeBSD 13.1-RELEASE #0 releng/13.1-fc952ac22-dirty: Mon Jun
6 22:54:09 PDT 2022
buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/13/
amd64.amd64/sys/GENERIC amd64


From nobody Thu Jul 14 13:37:46 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LkFtF1K8Bz1J2s5
	for ; Thu, 14 Jul 2022 13:37:57 +0000 (UTC)
	(envelope-from prvs=187829186=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LkFtC4pN6z3xKX
	for ; Thu, 14 Jul 2022 13:37:55 +0000 (UTC)
	(envelope-from prvs=187829186=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1657805875;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=6TZC9aSVHFHJlN5nP8CedYYWQXwnZmbAy899EaLPqI8=;
  b=X93Bj0c85I6ZwKFdOhlZMaLPLKqjWDBPQFNg9/wiotmMBE+ndKQOcXxr
   jZedjAyJ4wpWhBFc+YyEo4dXjVbZGF5Cvjv2qWssuMRes2yvzpiv0RRL/
   kJnvKlv74kr8RtpHYGM6r1c5J72+FqWts/rXEJZKS0mOoBbJkhJ+Pgpkd
   w=;
X-IronPort-RemoteIP: 104.47.73.40
X-IronPort-MID: 75614721
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:8DSmAKIvxqs708TdFE+R15QlxSXFcZb7ZxGr2PjKsXjdYENS0DJRn
 2sdCGqGbKyCYjb8f40iat++8xtX6pfRy9cwTAJlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s
 q3yv/GZdJhcokf0/0vraP65xZVF/fngbqLmD+LZMTxGSwZhSSMw4TpugOd8iYNz6TSDK1rlV
 eja/ouOYjdJ5xYuajhOs/3a9Es11BjPkGhwUmIWNKgjUGD2zxH5PLpHTYmtIn3xRJVjH+LSb
 44vG5ngows1Vz90Yj+Uuu6Tnn8iG9Y+DiDX4pZiYICwgwAqm8AH+v1T2Mzwy6tgo27hc9hZk
 L2hvHErIOsjFvWkdO81C3G0H8ziVEHvFXCuzXWX6KSuI0P6n3TE5sljJmQSYpcj08lFK21K0
 fFFNg9UR0XW7w626OrTpuhEoO0GdJWuEKZB/3ZqwHfeEOosRo3FT+PS/9hE0Twsh8dIW/HDe
 84ebjkpZxPFC/FNEg5PVNRiw6H33T+jK1W0q3rMzUYzy3LUwwFrlqDkLfLee8CQRNUTlUGdz
 o7D1zukXEhGa4bGodaD2k3vuN3hsyHhYp4xErr/ruNFoRq1xnNGXXX6UnP++5FVkHWWUc9fM
 UsL0iwgp6M78AqsVNaVdxOirXOOpRJZS9dWO+Mg5QyH0aaS5ByWblXoVRZEYd0i8cMzGzoj0
 wfVm8uzXWA39rqIVXia67GY6yuoPjQYJnMDYilCShYZ597ko8c4iRenostfLZNZR+bdQVnYq
 w1mZgBk71nPpabnD5mGwG0=
IronPort-HdrOrdr: A9a23:chR8bqpkV4mak1MX3OTXT/AaV5vOL9V00zEX/kB9WHVpm5Oj+v
 xGzc5w6farsl0ssREb9uxo9pPwJE800aQFmbX5XI3SJTUO3VHFEGgM1+vfKlHbak7DH6tmpN
 xdmstFeaHN5DpB/KHHCWCDer5PoeVvsprY49s2p00dMD2CAJsQizuRZDzrcHGfE2J9dOAE/d
 enl716jgvlXU5SQtWwB3EDUeSGj9rXlKj+aRpDIxI88gGBgR6h9ba/SnGjr14jegIK5Y1n3X
 nOkgT/6Knmm/anyiXE32uWy5hNgtPuxvZKGcTJoMkILTfHjBquee1aKsu/lQFwhNvqxEchkd
 HKrRtlF8Nv60nJdmXwmhfp0xmI6kdd11bSjXujxVfzq83wQzw3T+Bbg5hCTxff4008+Plhza
 NixQuixtdqJCKFuB64y8nDVhlsmEbxi2Eli/Qvg3tWVpZbQKNNrLYY4FheHP47bWjHAbgcYa
 ZT5fznlbhrmQvwVQGZgoAv+q3mYp0LJGbLfqBY0fblkwS/nxhCvjglLYIk7zQ9HakGOud5Dt
 T/Q9tVfY51P74rhIJGdZI8qJiMexjwaCOJFl6uCnLaM4xCE07xivfMkcUIDaeRCdQ18Kc=
X-IronPort-AV: E=Sophos;i="5.92,271,1650945600"; 
   d="scan'208";a="75614721"
Received: from mail-dm6nam04lp2040.outbound.protection.outlook.com (HELO NAM04-DM6-obe.outbound.protection.outlook.com) ([104.47.73.40])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 14 Jul 2022 09:37:53 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=RUgZDipJUMb95inFLZwfe8IRlTjnS6uY4vzYBJgs5dnfqvZoVBsZwrSM8fUkhyJYnceEJkCtdqJwPO8u9qboCYNVkHMHWKgoFkC98CaJfdEA5v1Qeq05plZXVOApqDbg/7hAqZKoLhrLZ20zRnAhmAIEZ6cGsKna4l6nGJke2Fk+kDKdA/kz2HpAQ/Ck1cJFdZC42ZS17+D4qXkcJPGLNKWUfLsVReYNGDyj9I6wUUapTA4DO0ahzsJb9MiOxAfDbb5RuN0LpJAQZ2L/ZaADx1JZn8ovH/J5by+/94mo20sYq4SBMbZ/XMMa1pUJYhb+WtL1VYEi533f0E1J0vZkdg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=NI3I8uUk4z5K2rF/ZqRhHe4X1XGXTTWQBsKoAMs8IXY=;
 b=A+xt2VfUJb6tNaQsJAtpZkAOmXnz6oU+3EfGw6hdbSD2F6tZuDhagny+enZ3zIuhyrULaj2gUjcyAe2Zv80MwYnMg2uF/pSdJ41axT+Vr+OezXRr3ZSqPpQdv4NJT94ooYL4izt5MSWZ7GkvTzotsKxwmDguqt44Yh2fNBdsCD7V7OXOffAhAOL8JKIqv5Wr7y2mfRVNmrUnKzFg2OFiIXT3nwj+cOGL6OyWHyr1YmTIsMyW9H0VBNf4KhRGvvyo+p3qfpEgQodC+0798ISkNAKOLSEHUiFrHhk7nPolsw06LD3Vod+icyYG5V9CGrSj0c0Gtqm8Slu7C10HSuoTjA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=NI3I8uUk4z5K2rF/ZqRhHe4X1XGXTTWQBsKoAMs8IXY=;
 b=BwkiiUFWYykXVpjdBqsexDlnijDVEYocTGOHepoSC2i9KtGdzxP/mjvfCJGOA9Wp3HNM/kYjsxGRJNs9cECAdIDkDUn+bwOb/aIHZNok9bzLNXtUow7bNcuN9gyMbRny6CsHqgBW2cVPOmVE/tU26sFSJE85gfd6F4GMrVSfgJw=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by SJ0PR03MB5501.namprd03.prod.outlook.com (2603:10b6:a03:28e::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.14; Thu, 14 Jul
 2022 13:37:51 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::40af:d5f4:95eb:d534%6]) with mapi id 15.20.5417.026; Thu, 14 Jul 2022
 13:37:51 +0000
Date: Thu, 14 Jul 2022 15:37:46 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Brian Buhrow 
Cc: freebsd-xen@freebsd.org
Subject: Re: How deep is the xl socket queue?
Message-ID: 
References: <202207140322.26E3M9Wr025120@nfbcal.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <202207140322.26E3M9Wr025120@nfbcal.org>
X-ClientProxiedBy: MR1P264CA0117.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:501:50::10) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 8ab859b1-2b41-49a9-9516-08da659e0c73
X-MS-TrafficTypeDiagnostic: SJ0PR03MB5501:EE_
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	ydVnjL/wmCqCyEGeudgTUNCgtOHia1Hrylnet1ZYYaXSk51uGDv54eGa222s+8gYVcMyHpfPUkzIijCVR1R9T9r1h/f+OloAwguEXfl+6uRYe6eRHxTXhxe2bJ4myr7T4RfP1n5K7XYCy5j9p6JuyFng1Jt9DeIRBVfhGHyHK5DBv/EsnBxwQbPvMtsa1U/fJN1aWVrR4UGauWmOMUWbyfGUk9Nh7B8AoBL/VxfVCgUwbnJuan8ueLNkVl376WQG5xF9t6P/eDvy2HgPB1D3KkPu512P5Flf+42ZCq+h64r9uLJbUnutLon7J+taMyoP1qkrSpro+/OrMiIJXm6pypawR3IQ3LU4rwyhJGJMM2GA+MbgxtADTuYufgl1Ozo/h4kq9G2SD2RFoZnmDTuSvm/tAGJCKVrkWvc24ksRsH3H6zA4g+ibXWm7YLsGB9q410nGuHOIc6Ic/3m/2UnlgaVkkQCD5GePfCfLDpRo+BbbOyxtUBh6OukwozXoPNHwyGnFFhutUFLqIon1+3MynWxNwrmgzlWW9S/PcACbpVOP16qAs4Hfs13tWMSw0PeGDO94uhSUiWOGjhgcHoqWscpRuUM3PnjA7zpTnjJswB2DZFHWXZPqzeq/Rkr8QlmSQ0Dy0msvxxUqRko6JdCnH2h+WXaZ3xZufQMDBwcRfF+nyO3O6VBVqGgh8kE7vqBUDrZTjWd+0A4ADS5kHIRD2U9auLkUckXK/GueZMUiNzdb4iw4x/+LG+OWGOJuMwsJ
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(8936002)(4326008)(5660300002)(2906002)(8676002)(86362001)(83380400001)(85182001)(82960400001)(38100700002)(478600001)(6486002)(66476007)(66556008)(6916009)(316002)(66946007)(186003)(41300700001)(6506007)(6666004)(6512007)(9686003)(26005);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ZlpYS1hxSTZrWW5aQ1Q2QmlnL1p6MjlYdDFUczBnampQdndVcnhUQnZvNnFy?=
 =?utf-8?B?aytENFc1NUcvNnBSQUVyKzdpSFVZTURyWENsQ1NSNCtqYkN5bFVQYlVsdzNr?=
 =?utf-8?B?ZEN3dFlMTUcxUU5sL2pYb2F0dlNhNVV4cFRGb1NCRFVicDJMZHZNTVJXR3pt?=
 =?utf-8?B?K20rVUtUYTdFZWpBbFZuRkVuSkVmWTlHdDZEU3dsbVZGVE5yUVpkNDlRMUdW?=
 =?utf-8?B?aGpTOWVBVHU2ODRNZVNveXF4QitmQ1BRTHphS3dMUVpOQy90cnZDZVRpRmVt?=
 =?utf-8?B?blBraWo5bkl1UC92S2creUFKU3ZKUFNSMDI1SlJSSFFoYXVFUFAzb1B5MWVX?=
 =?utf-8?B?Sm0vSUdBRTNsZTlNY2cyVWlzMVFWNE5lVWRnTFVxTUFrT3ZVM1FEOHlyaFcr?=
 =?utf-8?B?OTFLYm00TkM4RWd6OWxVam1ZcG1NUEhvaHJRMklLSC81bytFZWY2cU9ma1J1?=
 =?utf-8?B?T2g2aXlPYnFEVXdQZ2VYTFYyNFlCR1Q4S2FCN0dubUxjWlVsWE5jeFk5c2Uy?=
 =?utf-8?B?SkFsbllnbFdIUFZ5VG5yY2w2ZDh1WlVxWjNnQmp6eXg0V0JySmlDV09mSXEr?=
 =?utf-8?B?YThCTUdrRlhySlFYWFhIYkN5dWo1OEFXWnJCYW5ETTdaajhocFlnZjlLMGJT?=
 =?utf-8?B?bzJ5cVg0Y1h4RHFjOVgzWGY0V2tYL1hvYkZtMzg3SGQ3T2xHd2l3emNMOEhq?=
 =?utf-8?B?K1g0MDlTUEw3SUQvbThiYm5DWGJQRG5MOWF1WWFEckFPUWVBV0c0ZkUvZktj?=
 =?utf-8?B?ZjFUWnFEZUpmcTQvcWgxNmhDcnV6WUo3b09SUCt3WjhhaVozQm1mclZDK25p?=
 =?utf-8?B?MGNpQkxDdHJWengwa0t5TUIrYWFrbU5XbG1JNjhyZEVCM09MWXR4RDZBQndX?=
 =?utf-8?B?NklFVWhLdGJEbWFVOTVyZG4xaEs4VnViWTVDdFZyLzJlbTE1QUtjUWRET2Fv?=
 =?utf-8?B?b3MrenQyNEdQZ2pqVlkwYlpjaWZHRFgrNVpDWkwyM2tveW5yWDgvT2N4c0hX?=
 =?utf-8?B?YlJZWmNuK3hsOEhhd2JaTW9EaWxVRHFmTlR5d0xna250R3VDeFNJLzl2ejl3?=
 =?utf-8?B?ZTZuM3pYL2g0ZlcrUCtJTGRYT1NuSUpLSG5vemV6b21nL1pwRWJIUnRGM3JO?=
 =?utf-8?B?cHNZakZuOFhSeGFMcEJOU0hSS2dpS3lhU1FmZ3A3VjJLeEN0NnVGMFpEZlE0?=
 =?utf-8?B?b2M2TThJRVlFQklnYkdRMTIvMmloeldYWGplaGNpWjNsc1lSSE1qLzhYNldu?=
 =?utf-8?B?TkwvVmtseUFLVG9LcnhXQXdxVVhxaXVlZjJob093Y3ZZMWR5MjdicFRFQzlL?=
 =?utf-8?B?ZFZmaWNKUEk3ZU1ZUFB2dlpjT1FGL00vY0l0ZUdZQnZ4OE5PdWxLUVZiZlZs?=
 =?utf-8?B?UVhwRVNmRXN2ZXJZcDQvd09HR21pWDQ0SVZlUmlnNDdsbWVlMmZIRW5HSldN?=
 =?utf-8?B?N0E4RCs3ZlhLUVcrdDhrVnNMS0c5YzRuRm5LV0licFFZWFJwU1lZbHVQRnBh?=
 =?utf-8?B?Mi85d0V1TmdBbjZ4ZzBTQ1lhVG00VzUvQlNqaHBLRzlTRklwaDBzOE9weSti?=
 =?utf-8?B?dlJ2MXVHa2FVNTZRSlYvWUwwT29UV3FSeFF3YmNRU3hnSENqOG13UGZJSWFr?=
 =?utf-8?B?Z2dZSXNsU0hBOXZRcis2RDVVZU1udW54ampHdktSdFI4Vk9PcnJmWkQ0aFlt?=
 =?utf-8?B?RGNhbS84aHQrc2QxU1hZQXhMRnZwYWFsK2JYSmE4YzZDcDh0TWo3WWozUDNT?=
 =?utf-8?B?WGhVRXFDOFBwUCtuV0JxaXY5aVRQNjEyTXU1Y2orR1F4K0owaE44MnRVYjNK?=
 =?utf-8?B?b2YxYi9KUW5sYmE2TFBzQUs3RitVdjFjRmJZbUVER0t3NE1RY0ZPSWp2elJV?=
 =?utf-8?B?eVZZcFhleTBkcDJLREYzNjRvOFN4dzNTMkJrYXFGVlRQSHdaZFhwK3gwNnBk?=
 =?utf-8?B?N25HYkhaNmN3VzFpUmw1ZG5aWmxVWVRyT2Z3TVl6NldyMXNYTFNML29Qc0JR?=
 =?utf-8?B?THBKT3Y1NHFPNjJOekttcGhqZHZmL2RsTFZ4MFJIRWdEZDdiVTVyS3M1WHhq?=
 =?utf-8?B?WG9qc05qS0l2VkdCVC9Vb0tBS250b0hmZGgyREVUcHFTcFdvT0V5UmIxajNT?=
 =?utf-8?Q?FNjiDs6iQR2yJbMNKcVukWTnR?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ab859b1-2b41-49a9-9516-08da659e0c73
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2022 13:37:51.6829
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gY7sZkbJnFzFpeijlNqIWLNV0o/EtVNRNLB18vxZyYSpTlo11qvhMoR61lxYu2jB3olJAsZ6c2nKdDHYC7NGtg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB5501
X-Rspamd-Queue-Id: 4LkFtC4pN6z3xKX
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=X93Bj0c8;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=BwkiiUFW;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=187829186=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=187829186=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.63 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	NEURAL_HAM_SHORT(-1.00)[-0.998];
	NEURAL_HAM_LONG(-0.98)[-0.980];
	NEURAL_HAM_MEDIUM(-0.95)[-0.948];
	RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	MIME_GOOD(-0.10)[text/plain];
	FROM_HAS_DN(0.00)[];
	FORGED_SENDER_VERP_SRS(0.00)[];
	TO_DN_SOME(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	RCVD_COUNT_THREE(0.00)[4];
	FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=187829186=roger.pau@citrix.com];
	DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	MIME_TRACE(0.00)[0:+];
	RCVD_TLS_LAST(0.00)[];
	ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	MLMMJ_DEST(0.00)[freebsd-xen]
X-ThisMailContainsUnwantedMimeParts: N

On Wed, Jul 13, 2022 at 08:22:09PM -0700, Brian Buhrow wrote:
> 	hello.  Recently, I've started seeing messages like the following:
> sonewconn: pcb 0xfffff801074d7800 (local:/var/run/xenstored/socket): Listen queue overflow: 2
> alread
> y in queue awaiting acceptance (1 occurrences)
> 
> I figured out what causes this.  I have a number of automated scripts that use the xl
> command to do their work.  Sometimes, multiple scripts run at once, causing too many instances
> of xl to run simultaneously.  My questions are:
> What is the default queue depth for the xl socket listener?

Hm, I've got no idea TBH.  Note however this is the xenstored socket,
libxl instances are the clients here.  Can you check which version of
xenstored are you currently running?  Just paste the output of:

# ps aux |grep xenstored

Does it make any difference if you use Ocaml or C xenstored
implementations?

You can do that by changing the line in
/usr/local/etc/rc.d/xencommons:

XENSTORED=/usr/local/sbin/xenstored

Or:

XENSTORED=/usr/local/sbin/oxenstored

And rebooting (note: you must reboot the system in order to switch
between xenstored implementations).

The binary prefixed with 'o' is the Ocaml one, while the binary
without any prefix is the C one.

Thanks, Roger.

From nobody Fri Jul 15 01:24:32 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LkYYl6pTbz4WS18
	for ; Fri, 15 Jul 2022 01:24:43 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "nfbcal.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4LkYYk2wz6z3xG1
	for ; Fri, 15 Jul 2022 01:24:42 +0000 (UTC)
	(envelope-from buhrow@nfbcal.org)
Received: from nfbcal.org (localhost [127.0.0.1])
	by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 26F1OWMV015970
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 14 Jul 2022 18:24:33 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org
Received: (from buhrow@localhost)
	by nfbcal.org (8.15.2/8.12.11) id 26F1OWAV017288;
	Thu, 14 Jul 2022 18:24:32 -0700 (PDT)
Message-Id: <202207150124.26F1OWAV017288@nfbcal.org>
From: Brian Buhrow 
Date: Thu, 14 Jul 2022 18:24:32 -0700
In-Reply-To: 
X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103)
To: Roger Pau =?utf-8?B?TW9ubsOp?= 
Subject: Re: How deep is the xl socket queue?
Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Thu, 14 Jul 2022 18:24:33 -0700 (PDT)
X-Rspamd-Queue-Id: 4LkYYk2wz6z3xG1
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org
X-Spamd-Result: default: False [-2.30 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org];
	MIME_GOOD(-0.10)[text/plain];
	MIME_TRACE(0.00)[0:+];
	ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US];
	R_DKIM_NA(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-xen];
	FROM_EQ_ENVFROM(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCVD_TLS_LAST(0.00)[];
	DMARC_NA(0.00)[nfbcal.org];
	ARC_NA(0.00)[];
	RCVD_COUNT_THREE(0.00)[3];
	RCPT_COUNT_THREE(0.00)[3];
	FROM_HAS_DN(0.00)[];
	TO_DN_SOME(0.00)[];
	MID_RHS_MATCH_FROM(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org

	hello Roger.  I am running the C version of xenstored, the oxenstored file is not
installed on my server.  With that said, I have figured out that the queue length on the
xenstored socket is 2, meaning there can be no more than 3 xl commands runing simultaneously on
a given xen installation.  I'm guessing, but do not know for sure, that running the oxenstored
version of the xenstored daemon wouldn't change this situation since I believe that queue
length value is a kernel default.  I looked through the sysctl variables, but don't see an
obvious variable to change this queue length.

The error message I quoted comes from kern/uipc_socket.c, which actually prints the queue
length in the message itself.  I can learn to live with this limitation, I'm just surprised no
one has run into it before, since I'm not running a large server implementation, and I assume
there are those who are using this for larger deployments than I am.

-thanks
-Brian


From nobody Fri Aug  5 13:43:55 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lzmz91ggLz4Y10r
	for ; Fri,  5 Aug 2022 13:44:05 +0000 (UTC)
	(envelope-from knezour@weboutsourcing.cz)
Received: from mx.weboutsourcing.cz (mx.weboutsourcing.cz [109.205.73.92])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Lzmz767mnz3LZq
	for ; Fri,  5 Aug 2022 13:44:03 +0000 (UTC)
	(envelope-from knezour@weboutsourcing.cz)
Received: from [172.23.200.66] (unknown [84.246.160.126])
	by mx.weboutsourcing.cz (Postfix) with ESMTPSA id E3F30E513
	for ; Fri,  5 Aug 2022 15:43:55 +0200 (CEST)
Message-ID: 
Date: Fri, 5 Aug 2022 15:43:55 +0200
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Content-Language: en-US
To: freebsd-xen@freebsd.org
From: Ondra Knezour 
Subject: Better network reporting from FreeBSD VM via xenstore
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms090704070902000106000004"
X-Rspamd-Queue-Id: 4Lzmz767mnz3LZq
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of knezour@weboutsourcing.cz designates 109.205.73.92 as permitted sender) smtp.mailfrom=knezour@weboutsourcing.cz
X-Spamd-Result: default: False [-5.40 / 15.00];
	SIGNED_SMIME(-2.00)[];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	MIME_GOOD(-0.20)[multipart/signed,text/plain];
	R_SPF_ALLOW(-0.20)[+mx];
	DMARC_NA(0.00)[weboutsourcing.cz];
	ASN(0.00)[asn:35236, ipnet:109.205.72.0/21, country:CZ];
	FROM_EQ_ENVFROM(0.00)[];
	MIME_TRACE(0.00)[0:+,1:+,2:~];
	R_DKIM_NA(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	RCPT_COUNT_ONE(0.00)[1];
	HAS_ATTACHMENT(0.00)[];
	MID_RHS_MATCH_FROM(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_HAS_DN(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	TO_DN_NONE(0.00)[];
	RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

This is a cryptographically signed message in MIME format.

--------------ms090704070902000106000004
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello all,

it seems to me that there is some room to improve how are things about 
networking reported from FreeBSD VM to the hypervisor.

Now we have function xe_ip_if() in /usr/local/sbin/xe-update-guest-attrs 
which just grep for xn interfaces, read one IP address for each which is 
eventually written to the /var/cache/xenstore/attr/ethNUMBER/ip

First - from where the ip node comes? I didn't find any such in 
https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html

We have

xenstore ls attr
eth0 = ""
  ip = "10.0.0.1"
PVAddons = ""
  MajorVersion = "6"
  MinorVersion = "2"
  MicroVersion = "0"
  BuildVersion = "76888"
  Installed = "1"

But on Linux

xenstore ls | grep attr -A 50
attr = ""
  PVAddons = ""
   BuildVersion = "1"
   Installed = "1"
   MajorVersion = "8"
   MinorVersion = "1"
   MicroVersion = "50"
  vif = ""
   1 = ""
    ipv6 = ""
     0 = "fe80::9c30:52ff:fef1:8711"
    ipv4 = ""
     0 = "10.128.99.15"
   2 = ""
    ipv4 = ""
     0 = "10.0.1.15"
    ipv6 = ""
     0 = "fe80::e833:9cff:fe5e:c21c"
   0 = ""
    ipv4 = ""
     0 = "172.22.1.15"
    ipv6 = ""
     0 = "fe80::c0ff:d7ff:fee2:b5ff"
xenserver = ""
  attr = ""
  device = ""
   vbd3 = ""
    768 = ""
    5696 = ""
   vif = ""
    0 = ""
     static-ip-setting = ""
      mac = "c2:ff:d7:e2:b5:ff"
      error-code = "0"
      error-msg = ""
      enabled = "0"
      enabled6 = "0"
    1 = ""
     static-ip-setting = ""
      mac = "9e:30:52:f1:87:11"
      error-code = "0"
      error-msg = ""
      enabled = "0"
      enabled6 = "0"
    2 = ""
     static-ip-setting = ""
      mac = "ea:33:9c:5e:c2:1c"
      error-code = "0"
      error-msg = ""
      enabled = "0"
      enabled6 = "0"

And secondly - I have multiple interfaces in VM, renamed by VLANs their 
are connected to. Some of them have multiple IPv4 addresses. Some also 
have IPv6 address(es). Bummer, I don't see any of this in pool 
management. And I wouldn't even without interface renames, because we 
report only one IPv4 address. First problem may be solved with some 
creative work with /etc/rc.conf, for example using result of following 
command in adition to simple ifconfig | grep xn

sysrc -a | grep -E "ifconfig_xn[[:digit:]]_name"
ifconfig_xn0_name: vlan1000
ifconfig_xn1_name: vlan1001
ifconfig_xn2_name: vlan2000

or list all ethernet interfaces (ifconfig -l ether) and look for 
ifconfig_xnX_name line for each not(xn). But regarding other missing 
info/nodes, I just don't know. May someone please enlighten me what is 
missing on our side?

uname -a
FreeBSD hostname 13.1-RELEASE FreeBSD 13.1-RELEASE 
releng/13.1-n250148-fc952ac2212 GENERIC amd64

pkg info xe\*
xe-guest-utilities-6.2.0_3
xen-guest-tools-4.14.0



-- 
Best regards

Ondra Knezour


--------------ms090704070902000106000004
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: Elektronicky podpis S/MIME

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
EAUwggewMIIFmKADAgECAgIQETANBgkqhkiG9w0BAQ0FADBlMQswCQYDVQQGEwJDWjEXMBUG
A1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAbBgNVBAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMR4w
HAYDVQQDExVQb3N0U2lnbnVtIFJvb3QgUUNBIDQwHhcNMTgwOTI3MDczOTIzWhcNMzMwOTI3
MDczOTIzWjBpMQswCQYDVQQGEwJDWjEXMBUGA1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAbBgNV
BAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMSIwIAYDVQQDExlQb3N0U2lnbnVtIFF1YWxpZmll
ZCBDQSA0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAufhybRoyz3Y8nWLACeVN
Ui7ZYHNFw9IeBTQLs7f67EqUpVelFU6W+jeOJBQ+HvawXXgNnQRp5IOKF09/YFI6A8/Y79U0
onXCptFbRGPooGGdEbOzSLkBKDDc5AZoGCy2cjuUXHjWveMqC1XaRrtXCG4eVieghMND1Kzb
qWCgA1L/LY8AJdkH+iN2Lq7u/YYSBdMuDC+E7YvtfLjbtAP9Wy9ezIf+zDP+2moL9Z1F5/9F
16fTEZuTLqo1gvvR4F+qC+sZUUSh0UZrnjegO6cQWDpJW/gv6R2XGp6hJnqP2JBqaE6qBh7W
Az2+SEaUD/VBctDzGGkNA/w4xWh/GHGdaUnAwvPFp2xbvE6Zokmds+iTRvlGYpD4zDb44zAX
uybrNlMTMYZvrp0eHlnftrYX7z8K480ksDoumOyf72YuZlo6LxdViayBsCQaogOjd+cQFiAt
SnJxRUdIL6lErW/kg4FfAhcZVMuPDdI/oJ9DD7YclnImxxTADFAvtpwDqYmdrmaSTSeHXkid
Ki7kX70crKmbuneJjpL7uvVcjIFkBJFmol6fkOcpBd70jmnoFdV5sCzD4lrwcBcxVHet0OPq
INSSfMzO1+8TyTKZnoMFnZcWoMm4e51EO3IhoymycPd13I7qGv35YD/SI7zodZuLBT/IUmQV
uNwH3j6t4NZEtJUCAwEAAaOCAmQwggJgMIHVBgNVHSAEgc0wgcowgccGBFUdIAAwgb4wgbsG
CCsGAQUFBwICMIGuGoGrVGVudG8gY2VydGlmaWthdCBwcm8gZWxla3Ryb25pY2tvdSBwZWNl
dCBieWwgdnlkYW4gdiBzb3VsYWR1IHMgbmFyaXplbmltIEVVIGMuIDkxMC8yMDE0LlRoaXMg
aXMgYSBjZXJ0aWZpY2F0ZSBmb3IgZWxlY3Ryb25pYyBzZWFsIGFjY29yZGluZyB0byBSZWd1
bGF0aW9uIChFVSkgTm8gOTEwLzIwMTQuMBIGA1UdEwEB/wQIMAYBAf8CAQAwegYIKwYBBQUH
AQEEbjBsMDcGCCsGAQUFBzAChitodHRwOi8vY3J0LnBvc3RzaWdudW0uY3ovY3J0L3Bzcm9v
dHFjYTQuY3J0MDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcC5wb3N0c2lnbnVtLmN6L09DU1Av
UlFDQTQvMA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBSTGDYfqWlwUTWqTz+sjVB+JgUp
CjCBpQYDVR0fBIGdMIGaMDGgL6AthitodHRwOi8vY3JsLnBvc3RzaWdudW0uY3ovY3JsL3Bz
cm9vdHFjYTQuY3JsMDKgMKAuhixodHRwOi8vY3JsMi5wb3N0c2lnbnVtLmN6L2NybC9wc3Jv
b3RxY2E0LmNybDAxoC+gLYYraHR0cDovL2NybC5wb3N0c2lnbnVtLmV1L2NybC9wc3Jvb3Rx
Y2E0LmNybDAdBgNVHQ4EFgQUDyh8PjYAOBBQrj24IZeL92BcYXgwDQYJKoZIhvcNAQENBQAD
ggIBABuGFixikXQXOPeKKwO8lrZxavCXzjowiWBDmyBzpvidK0pyiYNnqaYKJ2k99/LX0l8n
8TF0t7XM0TDzZN2uN7pYPDDoLte3oOgIT60yyvAlKDRb2KsWtAMibJJRBCGUUmygB8Dw8mys
fE0+TMpJa0WG35IqVKgsP7Z4ZS3rwU0m1TTcE+b5rZ5Nfn5o8qZoz7i7q2H3lpzSyGXBbZPX
3vG5Wa/pZ3qlUDS9bNLtZdMZu77MWi19igZ7Hq4Xp/8OE/zFJEPY7cxW+Wj+DdtCZRsht0e6
03w2FxFFefevehicFYBf6g48TzUxurRvB+15IMnNZzNFXtE502oyb2PGrcOykehW1fAOPVHg
fQNFxixyD2S68Lvi/IfKt61X76lIBOt3RnglMHveIxXGjMIpqdwf69ulbOAMuetmPQ8gtvy8
4U6TMGTJRoyefJBv2rqQm91ZDgoLMAs1grUjOOjoZdeVjvutdJD/+zR7P0zdGNZ/Nn9sRlOm
uwdlU1Z/udrvco9PtKT8qN5fQfpYY9wYneubRPU4LCwix8FnqhxKcR7mtbVVdEjrDCjD+tEU
b27RHSklTyQXUUq1ENa0FplMNs4lneg5wMraIq5RHG3iqAqE9Th5E+s+JyoAZB54p+y8pzdg
lEdXY/8+sr8VPUjwtqzmS3EgrGqnbiN1iYeAd1o0MIIITTCCBjWgAwIBAgIEAVakxjANBgkq
hkiG9w0BAQsFADBpMQswCQYDVQQGEwJDWjEXMBUGA1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAb
BgNVBAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMSIwIAYDVQQDExlQb3N0U2lnbnVtIFF1YWxp
ZmllZCBDQSA0MB4XDTIxMDgzMTAyMDA0MVoXDTIyMDkyMDAyMDA0MVowgaExCzAJBgNVBAYT
AkNaMRcwFQYDVQRhEw5OVFJDWi02ODg4NTQ4MjEaMBgGA1UECgwRT25kxZllaiBLbsSbxb5v
dXIxCjAIBgNVBAsTATExGjAYBgNVBAMMEU9uZMWZZWogS27Em8W+b3VyMRIwEAYDVQQEDAlL
bsSbxb5vdXIxEDAOBgNVBCoMB09uZMWZZWoxDzANBgNVBAUTBlAxODIzNjCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBALpR7AkZE8xR8xsUMW3NoB8oDSBWdwCiHGjIJbfGIzM3
7kzfhah5EoMsKaaTGbNiXKzdCFm7dB8PdLRUQLgs0DK92Hr42RuYqnN7ez1mh90ltLQnT7Fg
gMy0WpUA/xoat+uN0x/hvR11pWZ6SmlCzKmGpt5EU1S07jfnxNMqJ61dpAp7pwIu9f9T5Vzc
MmvD3fJXP2PAUGTCIDjvZYD4A/jKV28Ah2Z352qV1brw7AnhUcZCR6fAynYXm8xenAEyVKZL
3s/HJQcAsB4ne+t269yCESsUawdYNpY+Mt5oDWjedpQKnphyIbBJ4hLHos1SNuScccs46pYl
/X4D83e0imECAwEAAaOCA8IwggO+MD8GA1UdEQQ4MDaBGWtuZXpvdXJAd2Vib3V0c291cmNp
bmcuY3qgGQYJKwYBBAHcGQIBoAwTCjExMjg5MTE4NjEwCQYDVR0TBAIwADCCASwGA1UdIASC
ASMwggEfMIIBEAYJZ4EGAQQBEYFIMIIBATCB2AYIKwYBBQUHAgIwgcsagchUZW50byBrdmFs
aWZpa292YW55IGNlcnRpZmlrYXQgcHJvIGVsZWt0cm9uaWNreSBwb2RwaXMgYnlsIHZ5ZGFu
IHYgc291bGFkdSBzIG5hcml6ZW5pbSBFVSBjLiA5MTAvMjAxNC5UaGlzIGlzIGEgcXVhbGlm
aWVkIGNlcnRpZmljYXRlIGZvciBlbGVjdHJvbmljIHNpZ25hdHVyZSBhY2NvcmRpbmcgdG8g
UmVndWxhdGlvbiAoRVUpIE5vIDkxMC8yMDE0LjAkBggrBgEFBQcCARYYaHR0cDovL3d3dy5w
b3N0c2lnbnVtLmN6MAkGBwQAi+xAAQAwgZsGCCsGAQUFBwEDBIGOMIGLMAgGBgQAjkYBATBq
BgYEAI5GAQUwYDAuFihodHRwczovL3d3dy5wb3N0c2lnbnVtLmN6L3Bkcy9wZHNfZW4ucGRm
EwJlbjAuFihodHRwczovL3d3dy5wb3N0c2lnbnVtLmN6L3Bkcy9wZHNfY3MucGRmEwJjczAT
BgYEAI5GAQYwCQYHBACORgEGATB9BggrBgEFBQcBAQRxMG8wOwYIKwYBBQUHMAKGL2h0dHA6
Ly9jcnQucG9zdHNpZ251bS5jei9jcnQvcHNxdWFsaWZpZWRjYTQuY3J0MDAGCCsGAQUFBzAB
hiRodHRwOi8vb2NzcC5wb3N0c2lnbnVtLmN6L09DU1AvUUNBNC8wDgYDVR0PAQH/BAQDAgXg
MB8GA1UdJQQYMBYGCCsGAQUFBwMEBgorBgEEAYI3CgMMMB8GA1UdIwQYMBaAFA8ofD42ADgQ
UK49uCGXi/dgXGF4MIGxBgNVHR8EgakwgaYwNaAzoDGGL2h0dHA6Ly9jcmwucG9zdHNpZ251
bS5jei9jcmwvcHNxdWFsaWZpZWRjYTQuY3JsMDagNKAyhjBodHRwOi8vY3JsMi5wb3N0c2ln
bnVtLmN6L2NybC9wc3F1YWxpZmllZGNhNC5jcmwwNaAzoDGGL2h0dHA6Ly9jcmwucG9zdHNp
Z251bS5ldS9jcmwvcHNxdWFsaWZpZWRjYTQuY3JsMB0GA1UdDgQWBBTLm0ELTjgZczAvLC02
8XpvN3A/JTANBgkqhkiG9w0BAQsFAAOCAgEAcNLzbZfREA9OJ8gH34AYqBCFzug+GI3fXVDG
2brccNedB8au/5NAq9xKGQKlVqWRb4ipjc0FpHeF6V+uGxkOycrOcfOiZKXRypGbk901ZYUn
jKP+8EcH6TxDsdMOfXiRNX9u+ZBLk4p13tiqWI1xi7RSsfl8Z8NObqA15nUew5pIqcd028ky
8tQH6ivHZ0LekoX4HFkHQ5PJr/9jY32SEEBlmk7G9qVWHa2iUTP05WQZ0p3YuqHpv/y91jN/
+PjjNP3B8TDGOTFloiDPH6Fxd1SbDSVg1cuF7d2nggwdSiLF8sPwLfm367Q7Yw4PeVEVOdEB
sXsn0Fp/Nd5Qz7ilrDigKPY1z3Vl2nj4LST66Yw3Cv0v8vQeTXInqZFN9ZrbepaQQrb7rfus
+xreQKIJt/frw/kgEvy7dsfRsW44qkPjmimfGqRYTDjO+Y1C3iuwiVw5PelmXxsFaW0XjrVX
YvOLuIoBNUJSZBnM2K94rkDl/rm2XLd7BIIlJMrkiUkZV3YH+JzfqlW/nMm9rxUJ4dF9g1RX
lvKaaLA3atF6YTZsmkjXn293cAs6Vfcd4oZh95AZ3gHbHYOPxHBr/THSWb6ZP3+856JxHM3f
/rKop/+VjMaz3q03tmgM3i+ePReoHDJ91lnoXbQHD7YaGVGNSykZjFREI3LRZs83uPG5Kg8x
ggN/MIIDewIBATBxMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJDWi00NzExNDk4MzEd
MBsGA1UECgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBvc3RTaWdudW0gUXVh
bGlmaWVkIENBIDQCBAFWpMYwDQYJYIZIAWUDBAIBBQCgggHfMBgGCSqGSIb3DQEJAzELBgkq
hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIyMDgwNTEzNDM1NlowLwYJKoZIhvcNAQkEMSIE
IKQZcrdSavXHYIeHXOv2y8KBeO7Ek+NR9uCVZyBZIqXBMGwGCSqGSIb3DQEJDzFfMF0wCwYJ
YIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYI
KoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgYAGCSsGAQQBgjcQBDFzMHEw
aTELMAkGA1UEBhMCQ1oxFzAVBgNVBGETDk5UUkNaLTQ3MTE0OTgzMR0wGwYDVQQKDBTEjGVz
a8OhIHBvxaF0YSwgcy5wLjEiMCAGA1UEAxMZUG9zdFNpZ251bSBRdWFsaWZpZWQgQ0EgNAIE
AVakxjCBggYLKoZIhvcNAQkQAgsxc6BxMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJD
Wi00NzExNDk4MzEdMBsGA1UECgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBv
c3RTaWdudW0gUXVhbGlmaWVkIENBIDQCBAFWpMYwDQYJKoZIhvcNAQEBBQAEggEArDCds1l8
uXFJ/6n31JZKZ/RD24mOIlhpJ/uWC0nM9iMXx1LF/Q6QW8ZKtEbeyVyR1gO+8yqJHSsLjE58
B/RsUESu0ybCJqFP8/DEfNdUwx1ORDAzFkfIbJQ0f3CbocA7xQdDnr6xBb1uhqxN/5Ze6b7W
qppt2mQdkUKXwiz4SG1mv4F/3cYnEK7MBF9qK9TeIRq5uB2L8ahVdMAjnzYpfJnqS16H+YTa
crPf7uxmiF3Mchcrx6OeyUXnNk9XxfWps8/rrMqenmeG/mZKhObhWOnVfFGxSLB6T+k9gBQD
lG1MjtB8Pd+xoWleUYW4cOqmSKwGu2p+rMIGX2S9+bdWEQAAAAAAAA==
--------------ms090704070902000106000004--

From nobody Mon Aug  8 12:30:36 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M1bCF0KRgz4XptQ
	for ; Mon,  8 Aug 2022 12:30:49 +0000 (UTC)
	(envelope-from prvs=2126b1b44=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4M1bCC6WCfz3jnr;
	Mon,  8 Aug 2022 12:30:47 +0000 (UTC)
	(envelope-from prvs=2126b1b44=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1659961847;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=mspsT7+ReTDCjVxpI9Uh3gU8+z8WO+aAqp4LpBeVXQs=;
  b=EVofN/DTxVsA5+v104yPzClnw6BB352piUFuw5neeknJgdzGY6siIu2B
   1XaJZiQ+hHmSGtuZfkuqC3OR/WutEPYOJ8NkmhTMSDet1zOLCXYe8MW+q
   ZAWGSv9cqjiqP7cEEMDkwqTEOjCviInM1I0nmLNbAkXgckSuhYn1ujmH/
   8=;
X-IronPort-RemoteIP: 104.47.66.45
X-IronPort-MID: 78031348
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:3WomUan5ilR9xu7hfLBWxtbo5gybJ0RdPkR7XQ2eYbSJt1+Wr1Gzt
 xIZWDiHOP6IM2ekLosgOoW2/EID6pXTndQ2G1NqrCExRSMWpZLJC+rCIxarNUt+DCFioGGLT
 Sk6QoOdRCzhZiaE/n9BCpC48T8mk/ngqoPUUIbsIjp2SRJvVBAvgBdin/9RqoNziJ2yDhjlV
 ena+qUzA3f4nW8vWo4ow/jb8kk37Kyi4GpwUmEWPpingnePzxH5M7pHTU2BByOQapVZGOe8W
 9HCwNmRlo8O105wYj8Nuu+TnnwiGtY+DyDX4pZlc/HKbix5jj4zys4G2M80Mi+7vdkrc+dZk
 72hvbToIesg0zaldO41C3G0GAkmVUFKFSOuzdFSfqV/wmWfG0YAzcmCA2kXMLA+xchuWV0R0
 tcHeTIta0GhmsaPlefTpulE3qzPLeHNFaZG4DRF8mucCvwrB5feX6/N+NlUmi8qgdxDFurfY
 MxfbidzaBPHYFtEPVJ/5JAWxb/0wCWgNWMG7g7IzUY0yzG7IAhZyr/iPcCTYtWXbc5UglyZt
 iTN+GGR7hQya4zHkmPeriLEaunnt3nmfak/HZaD+Pdhqm2vxXcdV0URfA7uyRW+ogvkMz5FE
 GQX/Cc8qrUo3FOmRN7gQwa/vWSPvxgTHdFCe8U08gyAwLHYphaUAEAKVDNMYscq8sgsSlQC2
 l6PmN75LTJ1vbCfUn7b8a2bxQ5eIgAQJG4GICobFw0M5oC5pJlp1k6SCNF+DKSyk9v5Xynqx
 CyHpzQ/gLNVitMX06K8/hbMhDfESoX1czPZLz7/BgqNhj6Vrqb/D2B0wTA3Ncp9Ebs=
IronPort-HdrOrdr: A9a23:ELiYha7JYJ5K0JIjHgPXwVOBI+orL9Y04lQ7vn2ZFiY5TiXIra
 qTdaogviMc6Ax/ZJjvo6HkBEClewKlyXcT2/hrAV7CZniehILMFu1fBOTZowEIdxeOldK1kJ
 0QCZSWa+eAcmSS7/yKhzVQeuxIqLfnzEnrv5a5854Ed3AXV0gK1XYcNu/0KDwVeOEQbqBJaa
 Z0q/A30QaISDAyVICWF3MFV+/Mq5nik4/nWwcPA1oC5BOVhT2lxbbmG1zAty1uGw9n8PMHyy
 zoggb57qKsv7WSzQLd7Xba69BzlMH6wtVOKcSQgow+KynqiCyveIN9Mofy9QwdkaWK0hIHgd
 PMqxAvM4Ba7G7QRHi8pV/X1wzpwF8Vmgrf4G7dpUGmjd3yRTo8BcYEr5leaAHl500pu8w5+L
 5X3kqC3qAnQS/orWDY3ZzlRhtqnk27rT4JiugIlUFSVoMYdft4sZEfxkVIC50NdRiKpbzPKN
 MeQv002cwmMG9zNxvizylSKZ2XLz4O9y69Mwc/Upf/6UkUoJh7p3FotvD30E1wtq7VcKM0lt
 gsAp4Y6o2mcfVmHZ6VJN1xNfdfWVa9Ni7kASa1HWnNMp0hFjbkl6PXiY9Fl91CPqZ4h6cPpA
 ==
X-IronPort-AV: E=Sophos;i="5.93,222,1654574400"; 
   d="scan'208";a="78031348"
Received: from mail-mw2nam12lp2045.outbound.protection.outlook.com (HELO NAM12-MW2-obe.outbound.protection.outlook.com) ([104.47.66.45])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 08 Aug 2022 08:30:45 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=OV37eVQZ59iHZ9uvRFhHA+RdYvy2tQ0r0VXdhCBbvDTzGiCMEJ6OqfN9Vm3RISnmdio7bh4UERwCeQYmhkedp+71vnf1lPNnmze8RIInLGlqQLGPBwqF+Rhc0iwhDodz/lA0+m7fCVCi1/DrbzbvFkKB/28Aw1QZNB2+AWdn1+ZTn3Jfa7f6oYu8eZ/J454jw93axcWUH7ZVRtCpcIyWOBUSKgtii3FMwT/jjKXMKmj44urppReADlVUY2TTJfSixU0Qdkk/EC9G+GlEOsy4M1rFDqxZRXyipX7EvNouyj/ub83xB3wkTLKbHcAiKzSSrlQMxCC+NLsKdVVbWrjo/A==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=qmnVqWOyjAmuostJx9wIWOMfZx3DhAZ+YP0Zqh2GZPw=;
 b=MloI4vQGPhzFIO6vybSxQISq2JuXMrXv7xGye3q69rfuI9wc3CWcCSq38irHfAQJoHN3CYEgHQ4xuFGdKTiwdWBtaUitTU1+gv11CcyIXIyNOl7OJcrHq78TL/QDUe/GmjVuXGYkyQ07/tf1lg8tsgrITEgF8EoLo8MIB3TlHz5uQXdkydZ8k1Qq0MY+YMc6Nscwm8PYc+EDBHR5UYePQTz6QJ1o+AoENMas3SHV9ZZ9ZLTQ/tgUh13akXVItCdtQFdB5ThcNpsDv8aCgrKfSe+BwMvF0Drm/ktCyNpl5dW6X1VCRwx8j+7z7rTK/+JLpxcKY1Xu+P8+k6cRfelbzQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=qmnVqWOyjAmuostJx9wIWOMfZx3DhAZ+YP0Zqh2GZPw=;
 b=VN+FazS/ELvRW3ZFuRp38WdlncoU3v07z9U2NgUYni4QBwmf9A/MG2U3phz4/iUAtkL/pQav2blqIzI5SZdWj5bEhzIvFyzr/14fptVyZF7f/XiRK5n3ukcy8rzDgi6VyuGus4C3dcwUz6qVN3Cxx5o8Km0aV1LZaxVBfSYFSLk=
Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18)
 by PH0PR03MB5704.namprd03.prod.outlook.com (2603:10b6:510:3a::24) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5504.14; Mon, 8 Aug
 2022 12:30:41 +0000
Received: from DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::d4e3:4d09:87e1:7f96]) by DS7PR03MB5608.namprd03.prod.outlook.com
 ([fe80::d4e3:4d09:87e1:7f96%4]) with mapi id 15.20.5504.020; Mon, 8 Aug 2022
 12:30:41 +0000
Date: Mon, 8 Aug 2022 14:30:36 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Ondra Knezour 
Cc: freebsd-xen@freebsd.org, dinoex@FreeBSD.org
Subject: Re: Better network reporting from FreeBSD VM via xenstore
Message-ID: 
References: 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: MR1P264CA0167.FRAP264.PROD.OUTLOOK.COM
 (2603:10a6:501:55::8) To DS7PR03MB5608.namprd03.prod.outlook.com
 (2603:10b6:5:2c9::18)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 79dad89d-948d-4a1f-7676-08da7939ce69
X-MS-TrafficTypeDiagnostic: PH0PR03MB5704:EE_
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	cqDoTglu3qV1m79PBNFY9RWQTAYAkuLu3jX4Wo2ke6lLl73hSlYlRJKtiTjLy7qE13VYnr23y3RV2TYqQqch2QR/mWxfLVMKhE3agdtJX94COPIoITN13gb4kWL3pvY8BS259tVE1Y3bU/eUQyrdvqUlV8S/WCFWlf6LIse8AhvKDp5B0GWH7yf/jScErhDvceULwtT1kSvELeAIVkzIlkgQ6vaQmrnkKA3ljDQgPWNOVPZ4VsonGcjCHo0S0+3xmXRW9FLqCKfHk1mGPe05JRzmS0yro4HyI4ebsMHcUmc3ZAP7CrF3625dDW528Rqc9l7t4QOH1RMeWmJjcxIY7JgzYXe3EUkASkE7L2CcmmdgPqjjHsDVgaR/FXzMkJTLanccD8+CmZf6DxVB/9UI6cAgguPzMo48zGXyg27jQ8J1BRpIrwwdGXWV6ifZyoK69RcA4bBX0K0S+4wu8s4sV5gXGytEbfI46K/rUDT5zi0T8HJtJZvtiNlhLuBHC5QgWOmwDmrqKWXWWpWAJuBr20pnuS4vuT7EjE7jhJzfiBErrG4hMsIaPB8EnUmvr3vvTwWwXm0pCMYg8rV9I8uGVbEh17VLlnoETW/M1w6vEJHbaYEvLUU634E2iSApvkkOzN4gyQrI7oC96lfczfcP0JxHXxizxoLfo9qpLI754Z8rLqoW9jswe0bvw4gWtGEJc81j96YsVFmADwEvz8MctHVAmcf352XeuCP0hV7Ro38bkuHTkJRt//GTi4DhWpZu+gDv4UPy+CIJJmXUu+8gK+P8Dx2t6hJhvVC52NCZOqqONJgB53F9kHB9VMf07c5f
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(7916004)(366004)(39860400002)(346002)(136003)(396003)(376002)(8676002)(66476007)(66556008)(8936002)(4326008)(186003)(85182001)(966005)(478600001)(6486002)(316002)(6916009)(5660300002)(66946007)(86362001)(41300700001)(26005)(6512007)(83380400001)(33716001)(6506007)(6666004)(9686003)(2906002)(82960400001)(38100700002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?Vyt0eDlkMzJYZERJZEVYQlo1Vkc2NzdIY1J5UTVZLzViUXlzRk0yUldqZmxV?=
 =?utf-8?B?eUY3MTM4bTJrNE1FK3BBSUpqZ0Rpb3prZmU5Q2JpZ3NQOWtyOW9zSmhIRFMw?=
 =?utf-8?B?eS9VcnY4cCsxd2UwVG9HNDl0dE81dHdrQ1Q1SkxqTWJmbXRlaTB5cUNPWGNL?=
 =?utf-8?B?akUzNnIxVEg5WVdEZHh1dzNqVmZ0R0wrdERjZGpUZm10SklmbUJWa3hoZGla?=
 =?utf-8?B?UGI4aFZQNFBZSG4wd1h0b0p4ekFhUk9qZXZsQ1JPbmhFNDIyS1dCNEt1K0k3?=
 =?utf-8?B?SXpxODN1SEFqc0ZsR3VpYmxFanBERldQZit6cnU2bTBBMWQ5LytBc3NXd1JV?=
 =?utf-8?B?UjYwdHR2RDJpRnFUSW9ZNlgyK2VuTzZHYnRwTi9EL05qQURIVldTY1NoNmlJ?=
 =?utf-8?B?UFFtZGxsbzMwNm1TanJSdWI2WFJYZ1hPR1ZjSXpoc1VuMEFoTlRlTHdJY1Va?=
 =?utf-8?B?OWZ1WkxuNHNtUlJtSjBaOEdoNmxOaVBMd2VuZGRwQlpjbnowc0MyeTVkYlVC?=
 =?utf-8?B?VWpEbWY1cS9UZURvRld6Y3VCMmtVSzllOTgrNGZIUVZHVVZrSU8zWGhkQkp2?=
 =?utf-8?B?UVFtQ2JBdGErQ3h0bjlKUmNNSTBxTDNZV051S0FsaTVhNE9oRE1xSDdtOWNR?=
 =?utf-8?B?WHhwZFBvTWI2eDZrU05XRTQ1V3NnTng3NHI5TE5zS3JDbHdmMVBHcTRSNkxw?=
 =?utf-8?B?cllXT1oyZEZFZ0VNQmY5R1l3K1FSSWtzVVJHd0VKbURVazVGWDVOUG44SzFn?=
 =?utf-8?B?Q042MzVvK2dab1dHemxzNzZuTGtIN1JaWTM5ZWlUVWkzSUd6U0hicGtvcU9V?=
 =?utf-8?B?MENlVDBJZDgxcXlnUE1Hcndlc3JEWnZoVlN2d3ZBWkY0bkRVUmJ2cUREZ01O?=
 =?utf-8?B?YzhER2h3UGc1S3BFcm9ZaDhQbmVnOUtMamxUK3ZCVTh6UFZkVWpUb29oTXNP?=
 =?utf-8?B?WlVlZkJ4NWFHQ0h0UWkrOHZwWGxBYTBPU2F1N1NBZEZvZVptQXVacjVDbWlM?=
 =?utf-8?B?QVB5L3ZUVFNRd3RGRk9pcVR4RWNESnk5UjRMQ3pwYkN6T2dadWY4M0xFRGNR?=
 =?utf-8?B?Uk5qRDh6M2xwZWUzQWtydVp3Vlk0SmpmemtiWHZPcXlPQ0hwWEUxbXpBUkRi?=
 =?utf-8?B?ZEczZnNhNjRSeVRiTW9oQjV1Um8vdkQ2c2YvMTIwTHYya3BaM2JCa1JWMzMy?=
 =?utf-8?B?elkwNkJCNWZJWTVMMHhETUJpYXZHY1htUUxTKy82QVlRODh4Y1FVWWwrVHYv?=
 =?utf-8?B?bjh5RTFtYWpTTjg2RHFFcmhEYXhWZmkrZDdoejRmcDY5Z0ZqbW5KZkNlQlpV?=
 =?utf-8?B?N055MUt1V2pSRmRZZC96bVBBWWxVT21PckVKY3B0MFZvQ28wMGV5bEY3dDJJ?=
 =?utf-8?B?bFNDNmVUTFB0Q3NJVjFVc2U5Qit1eFUxbHpHQkpOUS9qc045YUpyd3ZmbWFQ?=
 =?utf-8?B?dmdiQ2g4WUh0Z1ZkZ2gzVUd3cThiNE9hK01ObXRPQnQyVktVclVrUmpFZ2Nl?=
 =?utf-8?B?ZG1JR3dkUmZ2YlBKdlc5TXVpdzRvd3MrWFBBZTY3RnY2b3BvWnNwYldyT2ZZ?=
 =?utf-8?B?S2VnYngrNjZTdFl0K05SazdncDA3UXJHZVllWHVFNDl6bzdkbzZLTTl0NzNB?=
 =?utf-8?B?azJWSlVXN0tXc0p6d0pJZGtoVTJnRFRqckpQODUyMCtST1pqS0NJUitiMnh1?=
 =?utf-8?B?NXFoTnE3UFRQVDFsUXNlSVAzTWhMeGFKYnlkNVBmSlNqUnBlbzJoaVpBOHIz?=
 =?utf-8?B?L0tNazRmV2c2WlBHRmpkZ0dKZmswdkNSZTQrU0tqdS92d2hWZ09reHprODNT?=
 =?utf-8?B?TkxOdTBFa2h4YUNtU0RaZnA3UEJkZzg3OXlsK3JKeldqY2FvN0RiTG5ENFdt?=
 =?utf-8?B?cXJ0UlUzTlhjOVoyOFhxZzd6dS9ueVQraU1EUzlpaG5LbHJDZDMzL3hyc25Y?=
 =?utf-8?B?emRSc0tPVjlIL3diRndGTExZSVBSaVJXVE9YcEV3SFAzSFpXUGQ4UUxzdkd0?=
 =?utf-8?B?VFlVbGp2dk9KY0lXb1hDWXBJZE9odUJyQTBlamYxL3V1Mzl0TUdRS25YWEZh?=
 =?utf-8?B?bS9lNTJpK3JwZzZZRy9qNm1aS3FEb01PUHdPM0NXaWdFaUY1RHRzbEN2TVlu?=
 =?utf-8?Q?8TD+AgH5ZqydspBuBxWOuKGD2?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 79dad89d-948d-4a1f-7676-08da7939ce69
X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2022 12:30:41.5553
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 50UKaPaSlbhcG6AzEiwMD9BSap2TeeHEgb5gz09e7RnBHWwwFkXbIKN/dFwa7EqhvuHc3zgrgmD17sQMD8sFhQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR03MB5704
X-Rspamd-Queue-Id: 4M1bCC6WCfz3jnr
X-Spamd-Bar: ------
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b="EVofN/DT";
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b="VN+FazS/";
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=2126b1b44=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=2126b1b44=roger.pau@citrix.com"
X-Spamd-Result: default: False [-6.20 / 15.00];
	DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	NEURAL_HAM_LONG(-1.00)[-0.999];
	NEURAL_HAM_SHORT(-1.00)[-0.998];
	DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	MID_RHS_NOT_FQDN(0.50)[];
	RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	MIME_GOOD(-0.10)[text/plain];
	RCPT_COUNT_THREE(0.00)[3];
	FORGED_SENDER_VERP_SRS(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_HAS_DN(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=2126b1b44=roger.pau@citrix.com];
	ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	RCVD_COUNT_THREE(0.00)[4];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	MIME_TRACE(0.00)[0:+];
	RCVD_TLS_LAST(0.00)[];
	TO_DN_SOME(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On Fri, Aug 05, 2022 at 03:43:55PM +0200, Ondra Knezour wrote:
> Hello all,
> 
> it seems to me that there is some room to improve how are things about
> networking reported from FreeBSD VM to the hypervisor.
> 
> Now we have function xe_ip_if() in /usr/local/sbin/xe-update-guest-attrs
> which just grep for xn interfaces, read one IP address for each which is
> eventually written to the /var/cache/xenstore/attr/ethNUMBER/ip
> 
> First - from where the ip node comes? I didn't find any such in
> https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html

I think this is all specific to XenServer (a.k.a. Citrix Hypervisor)
and XCP (as being part of the xe-guest-utilities package), but not
used by upstream Xen toolstack (xl), and no one cared to add the used
paths to the upstream documents.

> We have
> 
> xenstore ls attr
> eth0 = ""
>  ip = "10.0.0.1"
> PVAddons = ""
>  MajorVersion = "6"
>  MinorVersion = "2"
>  MicroVersion = "0"
>  BuildVersion = "76888"
>  Installed = "1"
> 
> But on Linux
> 
> xenstore ls | grep attr -A 50
> attr = ""
>  PVAddons = ""
>   BuildVersion = "1"
>   Installed = "1"
>   MajorVersion = "8"
>   MinorVersion = "1"
>   MicroVersion = "50"
>  vif = ""
>   1 = ""
>    ipv6 = ""
>     0 = "fe80::9c30:52ff:fef1:8711"
>    ipv4 = ""
>     0 = "10.128.99.15"
>   2 = ""
>    ipv4 = ""
>     0 = "10.0.1.15"
>    ipv6 = ""
>     0 = "fe80::e833:9cff:fe5e:c21c"
>   0 = ""
>    ipv4 = ""
>     0 = "172.22.1.15"
>    ipv6 = ""
>     0 = "fe80::c0ff:d7ff:fee2:b5ff"
> xenserver = ""
>  attr = ""
>  device = ""
>   vbd3 = ""
>    768 = ""
>    5696 = ""
>   vif = ""
>    0 = ""
>     static-ip-setting = ""
>      mac = "c2:ff:d7:e2:b5:ff"
>      error-code = "0"
>      error-msg = ""
>      enabled = "0"
>      enabled6 = "0"
>    1 = ""
>     static-ip-setting = ""
>      mac = "9e:30:52:f1:87:11"
>      error-code = "0"
>      error-msg = ""
>      enabled = "0"
>      enabled6 = "0"
>    2 = ""
>     static-ip-setting = ""
>      mac = "ea:33:9c:5e:c2:1c"
>      error-code = "0"
>      error-msg = ""
>      enabled = "0"
>      enabled6 = "0"

You seem to use different commands on FreeBSD and Linux to get the
entries (`xenstore ls attr` vs `xenstore ls | grep attr -A 50`), so
it's not really a fair comparison I would say.

> And secondly - I have multiple interfaces in VM, renamed by VLANs their are
> connected to. Some of them have multiple IPv4 addresses. Some also have IPv6
> address(es). Bummer, I don't see any of this in pool management. And I
> wouldn't even without interface renames, because we report only one IPv4
> address. First problem may be solved with some creative work with
> /etc/rc.conf, for example using result of following command in adition to
> simple ifconfig | grep xn
> 
> sysrc -a | grep -E "ifconfig_xn[[:digit:]]_name"
> ifconfig_xn0_name: vlan1000
> ifconfig_xn1_name: vlan1001
> ifconfig_xn2_name: vlan2000
> 
> or list all ethernet interfaces (ifconfig -l ether) and look for
> ifconfig_xnX_name line for each not(xn). But regarding other missing
> info/nodes, I just don't know. May someone please enlighten me what is
> missing on our side?

I've added dinoex as he is the maintainer of the xe-guest-utilities
package, maybe he has some ideas about how this can be solved.

You might also want to raise a ticket with the XCP-ng guys [0] about
this, they usually care about OSes not officially supported by
XenServer.

Thanks, Roger.

[0] https://github.com/xcp-ng/xcp/issues

From nobody Mon Aug  8 12:56:29 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M1bn310Wsz4XtMG
	for ; Mon,  8 Aug 2022 12:56:39 +0000 (UTC)
	(envelope-from knezour@weboutsourcing.cz)
Received: from mx.weboutsourcing.cz (mx.weboutsourcing.cz [109.205.73.92])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4M1bn15rJyz3mgq;
	Mon,  8 Aug 2022 12:56:37 +0000 (UTC)
	(envelope-from knezour@weboutsourcing.cz)
Received: from [172.23.200.66] (unknown [84.246.160.126])
	by mx.weboutsourcing.cz (Postfix) with ESMTPSA id D139B112A3;
	Mon,  8 Aug 2022 14:56:29 +0200 (CEST)
Message-ID: <8f0643c5-22b3-c1cf-a7f2-86cada663bc9@weboutsourcing.cz>
Date: Mon, 8 Aug 2022 14:56:29 +0200
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Subject: Re: Better network reporting from FreeBSD VM via xenstore
Content-Language: en-US
To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= 
Cc: freebsd-xen@freebsd.org, dinoex@FreeBSD.org
References: 
 
From: Ondra Knezour 
In-Reply-To: 
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms000004050702060507060904"
X-Rspamd-Queue-Id: 4M1bn15rJyz3mgq
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of knezour@weboutsourcing.cz designates 109.205.73.92 as permitted sender) smtp.mailfrom=knezour@weboutsourcing.cz
X-Spamd-Result: default: False [-5.40 / 15.00];
	SIGNED_SMIME(-2.00)[];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-0.999];
	MIME_GOOD(-0.20)[multipart/signed,text/plain];
	R_SPF_ALLOW(-0.20)[+mx];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	R_DKIM_NA(0.00)[];
	FROM_EQ_ENVFROM(0.00)[];
	ASN(0.00)[asn:35236, ipnet:109.205.72.0/21, country:CZ];
	MIME_TRACE(0.00)[0:+,1:+,2:~];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	DMARC_NA(0.00)[weboutsourcing.cz];
	HAS_ATTACHMENT(0.00)[];
	MID_RHS_MATCH_FROM(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	RCPT_COUNT_THREE(0.00)[3];
	FROM_HAS_DN(0.00)[];
	TO_DN_SOME(0.00)[];
	RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

This is a cryptographically signed message in MIME format.

--------------ms000004050702060507060904
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Dne 08.08.2022 v 14:30 Roger Pau Monné napsal(a):
> On Fri, Aug 05, 2022 at 03:43:55PM +0200, Ondra Knezour wrote:
>> Hello all,
>>
>> it seems to me that there is some room to improve how are things about
>> networking reported from FreeBSD VM to the hypervisor.
>>
>> Now we have function xe_ip_if() in /usr/local/sbin/xe-update-guest-attrs
>> which just grep for xn interfaces, read one IP address for each which is
>> eventually written to the /var/cache/xenstore/attr/ethNUMBER/ip
>>
>> First - from where the ip node comes? I didn't find any such in
>> https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html
> I think this is all specific to XenServer (a.k.a. Citrix Hypervisor)
> and XCP (as being part of the xe-guest-utilities package), but not
> used by upstream Xen toolstack (xl), and no one cared to add the used
> paths to the upstream documents.
>
>> We have
>>
>> [ snip }
 >> But on Linux
>> [ snip ]
 >>
>> You seem to use different commands on FreeBSD and Linux to get the
>> entries (`xenstore ls attr` vs `xenstore ls | grep attr -A 50`), so
>> it's not really a fair comparison I would say.
Only for brevity. There are all entries form FreeBSD guest and only 
couple relevant from Linux guest, so even if not fair, still favourable 
for "us" :)
>> And secondly - I have multiple interfaces in VM, renamed by VLANs their are
>> connected to. Some of them have multiple IPv4 addresses. Some also have IPv6
>> address(es). Bummer, I don't see any of this in pool management. And I
>> wouldn't even without interface renames, because we report only one IPv4
>> address. First problem may be solved with some creative work with
>> /etc/rc.conf, for example using result of following command in adition to
>> simple ifconfig | grep xn
>>
>> sysrc -a | grep -E "ifconfig_xn[[:digit:]]_name"
>> ifconfig_xn0_name: vlan1000
>> ifconfig_xn1_name: vlan1001
>> ifconfig_xn2_name: vlan2000
>>
>> or list all ethernet interfaces (ifconfig -l ether) and look for
>> ifconfig_xnX_name line for each not(xn). But regarding other missing
>> info/nodes, I just don't know. May someone please enlighten me what is
>> missing on our side?
> I've added dinoex as he is the maintainer of the xe-guest-utilities
> package, maybe he has some ideas about how this can be solved.
>
> You might also want to raise a ticket with the XCP-ng guys [0] about
> this, they usually care about OSes not officially supported by
> XenServer.
>
> Thanks, Roger.
>
> [0] https://github.com/xcp-ng/xcp/issues

I just started experimenting and it seems like I am lucky.

xenstore-write attr/vif/1/ipv4/0 '11.10.10.10'

xenstore-write attr/vif/0/ipv4/1 '22.10.10.10'

Only one ovious difference between this and Linux output from previous 
message is indexing from 0, but after this write I can see both address 
in both XCP-ng Center and Xen Orchestra.

So my question changes to are we willing to add support for multiple 
IPv4 addresses, IPv6 addresses and renamed interfaces to 
xe-guest-utilities? dinoex@? :)

-- 
Best regards

Ondra Knezour


--------------ms000004050702060507060904
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: Elektronicky podpis S/MIME

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
EAUwggewMIIFmKADAgECAgIQETANBgkqhkiG9w0BAQ0FADBlMQswCQYDVQQGEwJDWjEXMBUG
A1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAbBgNVBAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMR4w
HAYDVQQDExVQb3N0U2lnbnVtIFJvb3QgUUNBIDQwHhcNMTgwOTI3MDczOTIzWhcNMzMwOTI3
MDczOTIzWjBpMQswCQYDVQQGEwJDWjEXMBUGA1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAbBgNV
BAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMSIwIAYDVQQDExlQb3N0U2lnbnVtIFF1YWxpZmll
ZCBDQSA0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAufhybRoyz3Y8nWLACeVN
Ui7ZYHNFw9IeBTQLs7f67EqUpVelFU6W+jeOJBQ+HvawXXgNnQRp5IOKF09/YFI6A8/Y79U0
onXCptFbRGPooGGdEbOzSLkBKDDc5AZoGCy2cjuUXHjWveMqC1XaRrtXCG4eVieghMND1Kzb
qWCgA1L/LY8AJdkH+iN2Lq7u/YYSBdMuDC+E7YvtfLjbtAP9Wy9ezIf+zDP+2moL9Z1F5/9F
16fTEZuTLqo1gvvR4F+qC+sZUUSh0UZrnjegO6cQWDpJW/gv6R2XGp6hJnqP2JBqaE6qBh7W
Az2+SEaUD/VBctDzGGkNA/w4xWh/GHGdaUnAwvPFp2xbvE6Zokmds+iTRvlGYpD4zDb44zAX
uybrNlMTMYZvrp0eHlnftrYX7z8K480ksDoumOyf72YuZlo6LxdViayBsCQaogOjd+cQFiAt
SnJxRUdIL6lErW/kg4FfAhcZVMuPDdI/oJ9DD7YclnImxxTADFAvtpwDqYmdrmaSTSeHXkid
Ki7kX70crKmbuneJjpL7uvVcjIFkBJFmol6fkOcpBd70jmnoFdV5sCzD4lrwcBcxVHet0OPq
INSSfMzO1+8TyTKZnoMFnZcWoMm4e51EO3IhoymycPd13I7qGv35YD/SI7zodZuLBT/IUmQV
uNwH3j6t4NZEtJUCAwEAAaOCAmQwggJgMIHVBgNVHSAEgc0wgcowgccGBFUdIAAwgb4wgbsG
CCsGAQUFBwICMIGuGoGrVGVudG8gY2VydGlmaWthdCBwcm8gZWxla3Ryb25pY2tvdSBwZWNl
dCBieWwgdnlkYW4gdiBzb3VsYWR1IHMgbmFyaXplbmltIEVVIGMuIDkxMC8yMDE0LlRoaXMg
aXMgYSBjZXJ0aWZpY2F0ZSBmb3IgZWxlY3Ryb25pYyBzZWFsIGFjY29yZGluZyB0byBSZWd1
bGF0aW9uIChFVSkgTm8gOTEwLzIwMTQuMBIGA1UdEwEB/wQIMAYBAf8CAQAwegYIKwYBBQUH
AQEEbjBsMDcGCCsGAQUFBzAChitodHRwOi8vY3J0LnBvc3RzaWdudW0uY3ovY3J0L3Bzcm9v
dHFjYTQuY3J0MDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcC5wb3N0c2lnbnVtLmN6L09DU1Av
UlFDQTQvMA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBSTGDYfqWlwUTWqTz+sjVB+JgUp
CjCBpQYDVR0fBIGdMIGaMDGgL6AthitodHRwOi8vY3JsLnBvc3RzaWdudW0uY3ovY3JsL3Bz
cm9vdHFjYTQuY3JsMDKgMKAuhixodHRwOi8vY3JsMi5wb3N0c2lnbnVtLmN6L2NybC9wc3Jv
b3RxY2E0LmNybDAxoC+gLYYraHR0cDovL2NybC5wb3N0c2lnbnVtLmV1L2NybC9wc3Jvb3Rx
Y2E0LmNybDAdBgNVHQ4EFgQUDyh8PjYAOBBQrj24IZeL92BcYXgwDQYJKoZIhvcNAQENBQAD
ggIBABuGFixikXQXOPeKKwO8lrZxavCXzjowiWBDmyBzpvidK0pyiYNnqaYKJ2k99/LX0l8n
8TF0t7XM0TDzZN2uN7pYPDDoLte3oOgIT60yyvAlKDRb2KsWtAMibJJRBCGUUmygB8Dw8mys
fE0+TMpJa0WG35IqVKgsP7Z4ZS3rwU0m1TTcE+b5rZ5Nfn5o8qZoz7i7q2H3lpzSyGXBbZPX
3vG5Wa/pZ3qlUDS9bNLtZdMZu77MWi19igZ7Hq4Xp/8OE/zFJEPY7cxW+Wj+DdtCZRsht0e6
03w2FxFFefevehicFYBf6g48TzUxurRvB+15IMnNZzNFXtE502oyb2PGrcOykehW1fAOPVHg
fQNFxixyD2S68Lvi/IfKt61X76lIBOt3RnglMHveIxXGjMIpqdwf69ulbOAMuetmPQ8gtvy8
4U6TMGTJRoyefJBv2rqQm91ZDgoLMAs1grUjOOjoZdeVjvutdJD/+zR7P0zdGNZ/Nn9sRlOm
uwdlU1Z/udrvco9PtKT8qN5fQfpYY9wYneubRPU4LCwix8FnqhxKcR7mtbVVdEjrDCjD+tEU
b27RHSklTyQXUUq1ENa0FplMNs4lneg5wMraIq5RHG3iqAqE9Th5E+s+JyoAZB54p+y8pzdg
lEdXY/8+sr8VPUjwtqzmS3EgrGqnbiN1iYeAd1o0MIIITTCCBjWgAwIBAgIEAVakxjANBgkq
hkiG9w0BAQsFADBpMQswCQYDVQQGEwJDWjEXMBUGA1UEYRMOTlRSQ1otNDcxMTQ5ODMxHTAb
BgNVBAoMFMSMZXNrw6EgcG/FoXRhLCBzLnAuMSIwIAYDVQQDExlQb3N0U2lnbnVtIFF1YWxp
ZmllZCBDQSA0MB4XDTIxMDgzMTAyMDA0MVoXDTIyMDkyMDAyMDA0MVowgaExCzAJBgNVBAYT
AkNaMRcwFQYDVQRhEw5OVFJDWi02ODg4NTQ4MjEaMBgGA1UECgwRT25kxZllaiBLbsSbxb5v
dXIxCjAIBgNVBAsTATExGjAYBgNVBAMMEU9uZMWZZWogS27Em8W+b3VyMRIwEAYDVQQEDAlL
bsSbxb5vdXIxEDAOBgNVBCoMB09uZMWZZWoxDzANBgNVBAUTBlAxODIzNjCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBALpR7AkZE8xR8xsUMW3NoB8oDSBWdwCiHGjIJbfGIzM3
7kzfhah5EoMsKaaTGbNiXKzdCFm7dB8PdLRUQLgs0DK92Hr42RuYqnN7ez1mh90ltLQnT7Fg
gMy0WpUA/xoat+uN0x/hvR11pWZ6SmlCzKmGpt5EU1S07jfnxNMqJ61dpAp7pwIu9f9T5Vzc
MmvD3fJXP2PAUGTCIDjvZYD4A/jKV28Ah2Z352qV1brw7AnhUcZCR6fAynYXm8xenAEyVKZL
3s/HJQcAsB4ne+t269yCESsUawdYNpY+Mt5oDWjedpQKnphyIbBJ4hLHos1SNuScccs46pYl
/X4D83e0imECAwEAAaOCA8IwggO+MD8GA1UdEQQ4MDaBGWtuZXpvdXJAd2Vib3V0c291cmNp
bmcuY3qgGQYJKwYBBAHcGQIBoAwTCjExMjg5MTE4NjEwCQYDVR0TBAIwADCCASwGA1UdIASC
ASMwggEfMIIBEAYJZ4EGAQQBEYFIMIIBATCB2AYIKwYBBQUHAgIwgcsagchUZW50byBrdmFs
aWZpa292YW55IGNlcnRpZmlrYXQgcHJvIGVsZWt0cm9uaWNreSBwb2RwaXMgYnlsIHZ5ZGFu
IHYgc291bGFkdSBzIG5hcml6ZW5pbSBFVSBjLiA5MTAvMjAxNC5UaGlzIGlzIGEgcXVhbGlm
aWVkIGNlcnRpZmljYXRlIGZvciBlbGVjdHJvbmljIHNpZ25hdHVyZSBhY2NvcmRpbmcgdG8g
UmVndWxhdGlvbiAoRVUpIE5vIDkxMC8yMDE0LjAkBggrBgEFBQcCARYYaHR0cDovL3d3dy5w
b3N0c2lnbnVtLmN6MAkGBwQAi+xAAQAwgZsGCCsGAQUFBwEDBIGOMIGLMAgGBgQAjkYBATBq
BgYEAI5GAQUwYDAuFihodHRwczovL3d3dy5wb3N0c2lnbnVtLmN6L3Bkcy9wZHNfZW4ucGRm
EwJlbjAuFihodHRwczovL3d3dy5wb3N0c2lnbnVtLmN6L3Bkcy9wZHNfY3MucGRmEwJjczAT
BgYEAI5GAQYwCQYHBACORgEGATB9BggrBgEFBQcBAQRxMG8wOwYIKwYBBQUHMAKGL2h0dHA6
Ly9jcnQucG9zdHNpZ251bS5jei9jcnQvcHNxdWFsaWZpZWRjYTQuY3J0MDAGCCsGAQUFBzAB
hiRodHRwOi8vb2NzcC5wb3N0c2lnbnVtLmN6L09DU1AvUUNBNC8wDgYDVR0PAQH/BAQDAgXg
MB8GA1UdJQQYMBYGCCsGAQUFBwMEBgorBgEEAYI3CgMMMB8GA1UdIwQYMBaAFA8ofD42ADgQ
UK49uCGXi/dgXGF4MIGxBgNVHR8EgakwgaYwNaAzoDGGL2h0dHA6Ly9jcmwucG9zdHNpZ251
bS5jei9jcmwvcHNxdWFsaWZpZWRjYTQuY3JsMDagNKAyhjBodHRwOi8vY3JsMi5wb3N0c2ln
bnVtLmN6L2NybC9wc3F1YWxpZmllZGNhNC5jcmwwNaAzoDGGL2h0dHA6Ly9jcmwucG9zdHNp
Z251bS5ldS9jcmwvcHNxdWFsaWZpZWRjYTQuY3JsMB0GA1UdDgQWBBTLm0ELTjgZczAvLC02
8XpvN3A/JTANBgkqhkiG9w0BAQsFAAOCAgEAcNLzbZfREA9OJ8gH34AYqBCFzug+GI3fXVDG
2brccNedB8au/5NAq9xKGQKlVqWRb4ipjc0FpHeF6V+uGxkOycrOcfOiZKXRypGbk901ZYUn
jKP+8EcH6TxDsdMOfXiRNX9u+ZBLk4p13tiqWI1xi7RSsfl8Z8NObqA15nUew5pIqcd028ky
8tQH6ivHZ0LekoX4HFkHQ5PJr/9jY32SEEBlmk7G9qVWHa2iUTP05WQZ0p3YuqHpv/y91jN/
+PjjNP3B8TDGOTFloiDPH6Fxd1SbDSVg1cuF7d2nggwdSiLF8sPwLfm367Q7Yw4PeVEVOdEB
sXsn0Fp/Nd5Qz7ilrDigKPY1z3Vl2nj4LST66Yw3Cv0v8vQeTXInqZFN9ZrbepaQQrb7rfus
+xreQKIJt/frw/kgEvy7dsfRsW44qkPjmimfGqRYTDjO+Y1C3iuwiVw5PelmXxsFaW0XjrVX
YvOLuIoBNUJSZBnM2K94rkDl/rm2XLd7BIIlJMrkiUkZV3YH+JzfqlW/nMm9rxUJ4dF9g1RX
lvKaaLA3atF6YTZsmkjXn293cAs6Vfcd4oZh95AZ3gHbHYOPxHBr/THSWb6ZP3+856JxHM3f
/rKop/+VjMaz3q03tmgM3i+ePReoHDJ91lnoXbQHD7YaGVGNSykZjFREI3LRZs83uPG5Kg8x
ggN/MIIDewIBATBxMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJDWi00NzExNDk4MzEd
MBsGA1UECgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBvc3RTaWdudW0gUXVh
bGlmaWVkIENBIDQCBAFWpMYwDQYJYIZIAWUDBAIBBQCgggHfMBgGCSqGSIb3DQEJAzELBgkq
hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIyMDgwODEyNTYzMFowLwYJKoZIhvcNAQkEMSIE
IBA4Uz1ZPbJDih/cFQKNggeDRxq6/FgZIdiGbQQAKdPLMGwGCSqGSIb3DQEJDzFfMF0wCwYJ
YIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYI
KoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgYAGCSsGAQQBgjcQBDFzMHEw
aTELMAkGA1UEBhMCQ1oxFzAVBgNVBGETDk5UUkNaLTQ3MTE0OTgzMR0wGwYDVQQKDBTEjGVz
a8OhIHBvxaF0YSwgcy5wLjEiMCAGA1UEAxMZUG9zdFNpZ251bSBRdWFsaWZpZWQgQ0EgNAIE
AVakxjCBggYLKoZIhvcNAQkQAgsxc6BxMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJD
Wi00NzExNDk4MzEdMBsGA1UECgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBv
c3RTaWdudW0gUXVhbGlmaWVkIENBIDQCBAFWpMYwDQYJKoZIhvcNAQEBBQAEggEAFvE8FRpO
THwGh+Q3Co7KL15OQrHosma8k27Jn2EcZvEV38hDtl7iQjn1jqSK1zqX6U4mbWyBmrYasQWE
iCtPqTfUhllkAmYlUXJM6QCuq6t2pJYTR/6KSH+Pd9SyKf6MyL8qGYkKThgQMqD4amPCU9Yh
4JRpA7SXxK0QPifxNkLgzTgvJOUHBwgpFKea+yHaiz81q4+bo2MeG9JOyYzZI05qrtkGSAXX
AZCx43Mp49mhhGYl0HOnjbFjJS1be+1cgKonvL3wLFI6gT5r+Csxp6OSUUkgn5OotbRYRc7O
V4Rb6pl/DL017Xt+AV0zuxHMnFqerm4flrcCXXYxeylxsgAAAAAAAA==
--------------ms000004050702060507060904--

From nobody Wed Oct 12 00:50:54 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MnDbf2g6rz4dsSC
	for ; Wed, 12 Oct 2022 00:50:54 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MnDbf0nWBz3vZK
	for ; Wed, 12 Oct 2022 00:50:54 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MnDbd707Dz18RK
	for ; Wed, 12 Oct 2022 00:50:53 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29C0orNj072996
	for ; Wed, 12 Oct 2022 00:50:53 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29C0orq2072995
	for xen@FreeBSD.org; Wed, 12 Oct 2022 00:50:53 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 243993] FreeBSD as Guest with 5+ with xnb(4) devices panic on
 call to  sysctl -a in strlen
Date: Wed, 12 Oct 2022 00:50:54 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 12.1-STABLE
X-Bugzilla-Keywords: crash, panic
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: koobs@FreeBSD.org
X-Bugzilla-Status: Closed
X-Bugzilla-Resolution: DUPLICATE
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: keywords
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1665535854;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=hz3eqI/FwI2YYaD6n2p+cYpnmZ4Kd/tTnJVzQJOsvmA=;
	b=Y4ERpmutCg9zkkgvqZGrhi5vT2OCJQi/vELzQ3m+Z9mzOGGPFcEy2dMdT6qV1QbPtUGxRu
	LlXpi7jdMa8jofVwwhdbgrWGPrnr/9vcURtbfqzrQmWEKMhWBREHatBWtpCQSqMPHEJBY3
	x2kjUtuU7/KRcqUBe9+7TJcFvZiPmsJYtSf5GJDfiS7tlHUH49kWXlFsG2vQUzB2mtUAsK
	WyDsd86oXo9xWBYhfxi6MMi3YWX6LP7TJvttXf99DZ+DREtj7RRDm5EMhfllRUsx3eg3BL
	/o+BtT1X1qCeMdKFtDRR9V6Cfj0UdxkjEq5wkY0caKgPXYrUI5fN5vsmK2u5Fg==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1665535854; a=rsa-sha256; cv=none;
	b=G5iuItZHNcBg3yuwllIM28gVj3GVXWl8PywRS4E3j93aLVLbY5wb/ueYUyz/ERLw8QuOvt
	scP0+x1aIz+xEgy23C3bmPHVYEEBcHg6WkXhfkL/1I5/y966gMdFmfMOhWffHJAtuSpDrC
	cs7tX07vem1zW7T7Sy2dlm+OWM3LpDjzAtiaPa5tNKBjw3321nu2o1Y+iql6ZtobgwTLU3
	EpnHU4niGujtDDD8TSEvrHvXIUiSdbKGKtuWpVO2LMIBia+wf8vLNgsAaHeQekbbFcDwC9
	z4Dc6RirBPdD3/jQDyVtxQYthfiHN0PGw6gAFErLch1PxPUqQYr5TRj7kc7cQQ==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243993

Kubilay Kocak  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |crash

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Thu Oct 13 10:00:17 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mp4lG4X53z4fbZq
	for ; Thu, 13 Oct 2022 10:00:26 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "*.krpservers.com", Issuer "RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Mp4lF6DK5z41kd
	for ; Thu, 13 Oct 2022 10:00:25 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from [10.12.30.106] 
	by smtp.krpservers.com (8.16.1/8.15.2) with ESMTPSA id 29DA0HPB093174
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for ; Thu, 13 Oct 2022 11:00:18 +0100 (BST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tdx.co.uk;
	s=krpdkim; t=1665655218;
	bh=LahRxEbIi10fixKmdfCkXu8Rs/QmMPV3MMDV/p0WEbk=;
	h=Date:From:To:Subject;
	b=lXAHsbco1Q34KEGqab5xAK/8jmv2jWBOPU8UiO3DNm4s4qlXSDQxwlz6yXtKEnMCN
	 hxePqQCe/opaa5Q3ZFPOz1oyklPJOsw65hFsIR81Yi8U0SSlwZF1nD3yW9TzCGPadD
	 8XjYKuq9mPf6tnlvubl7pVwA5i2Cg09+kQkLZog3BllnuA/Th1Er6lro3Liad42zbD
	 KG2CqaMQ1r9Reab5VehVN4v4KPOVT1LafNS5+6ZWkfPihMz1+iEL4eLLIRYxgyIWsg
	 N/wCW/QiWfbCmsjERQvuanKb9u1OOj2z4QiVZQyjWHdVF+t+gG4W9Py9VBSC5DeNMx
	 58bxcz90BMhVw==
Date: Thu, 13 Oct 2022 11:00:17 +0100
From: Karl Pielorz 
To: freebsd-xen@freebsd.org
Subject: Recently moved to XS 8.2 on new hardware - seen a couple of FreeBSD
 DomU lock ups?
Message-ID: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
X-Mailer: Mulberry/4.0.8 (Win32)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Rspamd-Queue-Id: 4Mp4lF6DK5z41kd
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=tdx.co.uk header.s=krpdkim header.b=lXAHsbco;
	dmarc=pass (policy=none) header.from=tdx.co.uk;
	spf=pass (mx1.freebsd.org: domain of kpielorz_lst@tdx.co.uk designates 62.13.128.145 as permitted sender) smtp.mailfrom=kpielorz_lst@tdx.co.uk
X-Spamd-Result: default: False [-0.52 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_SPAM_MEDIUM(0.98)[0.981];
	DMARC_POLICY_ALLOW(-0.50)[tdx.co.uk,none];
	MID_RHS_IP_LITERAL(0.50)[];
	R_SPF_ALLOW(-0.20)[+a:smtp.krpservers.com];
	R_DKIM_ALLOW(-0.20)[tdx.co.uk:s=krpdkim];
	MIME_GOOD(-0.10)[text/plain];
	FROM_EQ_ENVFROM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	ASN(0.00)[asn:60969, ipnet:62.13.128.0/24, country:GB];
	MIME_TRACE(0.00)[0:+];
	DKIM_TRACE(0.00)[tdx.co.uk:+];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	RCPT_COUNT_ONE(0.00)[1];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	TO_DN_NONE(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org];
	RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N


Hi all,

We've been running FreeBSD as a DomU under Xen Server for years now, and 
only really had a few now 'known' issues (e.g. with networking).

We've recently setup XS 8.2 on some new Dell servers, and whilst everything 
appears fine - twice now in a couple of weeks we've had FreeBSD DomU's just 
"lock up".

There's no errors logged, no kernel panic, nothing - they just "stop". Xen 
reckons the CPU is pegged at 100% (with brief periods of zero) - the 
console is still 'available' (but locked up) - and the kernel is dead (i.e. 
you cannot ping it).

Aside from the "Has anyone else seen similar" - with nothing in the logs, 
no panic, nothing - I'm kind of at a loss as to how best to troubleshoot 
this further?

The VM's are lightly loaded - haven't run out of resources (RAM /CPU etc.) 
- and it's only happened a couple of times now (but in a couple of weeks) - 
whereas our setup before never experienced this in it's lifetime.

One was a legacy 11.4 system (amd64) - the other was a 12.3 system (amd64).

A forced reboot brings them back (with some file system damage - as you'd 
expect from a crash).

Just at a loss as to where to look - given the lack of any panic/errors 
etc. Any suggestions?

Regards,

-Karl

From nobody Fri Oct 14 14:40:17 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MppwH0yDhz4fbsV
	for ; Fri, 14 Oct 2022 14:40:47 +0000 (UTC)
	(envelope-from prvs=27923c854=roger.pau@citrix.com)
Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MppwF6rlyz450l
	for ; Fri, 14 Oct 2022 14:40:45 +0000 (UTC)
	(envelope-from prvs=27923c854=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1665758445;
  h=date:from:to:cc:subject:message-id:references:
   in-reply-to:mime-version;
  bh=v7WBWy+XjlEDM04D14IkY1WGW+pmglVRObGlHCN7eQc=;
  b=JDVvcR9hWhmqYoUxyHViwdCuwAelP9nR36dUpj1+B6IkuRDWfjULze9+
   fYh4aW+ICPksvyy8KjcTTR8MTmbuxBbBjzThZSDCl0K40e2iwfNeLygZQ
   WRSqqMqV+qCcSYUQesrQcMQE+O1UqOJW4XIkWfaioUnjChT9zanUrNwJg
   E=;
X-IronPort-RemoteIP: 104.47.55.174
X-IronPort-MID: 82383682
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:W9+lfK8iWhqLXglirgqaDrUDlH+TJUtcMsCJ2f8bNWPcYEJGY0x3z
 2YaUGmDP/+Lamryco9zYd+39xgGuJLUzNJgHQU4/3o8E34SpcT7XtnIdU2Y0wF+jCHgZBk+s
 5hBMImowOQcFCK0SsKFa+C5xZVE/fjUAOC6UIYoAwgpLSd8UiAtlBl/rOAwh49skLCRDhiE/
 Nj/uKUzAnf8s9JPGj9Suv/rRC9H5qyo4mpA5wBmP5ingXeF/5UrJMNHTU2OByOQrrl8RoaSW
 +vFxbelyWLVlz9F5gSNy+uTnuUiG9Y+DCDW4pZkc/HKbitq/0Te5p0TJvsEAXq7vh3S9zxHJ
 HehgrTrIeshFvWkdO3wyHC0GQkmVUFN0OevzXRSLaV/ZqAJGpfh66wGMa04AWEX0vx9PjBws
 sBJExQqZ0C/u7KRg4+6VNA506zPLOGzVG8ekldJ6GiDSNwAEdXESaiM4sJE1jAtgMwIBezZe
 8cSdTtoalLHfgFLPVAUTpk5mY9EhFGmK2Ee9A3T+PdxujePpOBy+OGF3N79YNuFSN8Thk+Fj
 mnH4374ElcRM9n3JT+toij31rSRzXmTtIQ6H4Oi69o0vXSvxi8dKTxOWQGWucmysxvrMz5YA
 wlOksY0loAx7E2mSt7nTSq8u2KsohIMHdFXVfA5gCmD0KfZ6R2VQ3gJSBZPctErvtQ6Azsw2
 Te0c8jBADVutPifTyKb/7LN9zeqY3FKcSkFeDMOShYD75/7uoYvgxnTT9FlVqmoktnyHjK2y
 DePxMQju4guYQcw//3T1Tj6b/iE/cGhotIdjukPYl+Y0w==
IronPort-HdrOrdr: A9a23:MzbkhqlgZmzl+StPqHcTXSDTVwrpDfPJimdD5ihNYBxZY6Wkfp
 +V8cjzhCWftN9OYhodcLC7V5Voj0mskKKdxbNhRYtKPTOWwVdASbsP0WKM+V3d8kHFh41gPO
 JbAtND4b7LfCRHZKTBkW6F+r8bqbHokZxAx92uqUuFJTsaFp2IhD0JbjpzfHcGJjWvUvECZe
 ChD4d81k2dUEVSSv7+KmgOXuDFqdGOvJX6YSQeDxpizAWVlzun5JPzDhDdh34lInpy6IZn1V
 KAvx3y562lvf3+4hjA11XL55ATvNf60NNMCOGFl8BQADTxjQSDYphnRtS5zUYIidDqzGxvvM
 jHoh8mMcg2w3TNflutqR+o4AXk2CZG0Q6R9XaoxV/Y5eDpTjMzDMRMwahDdAHC1kYmtNZglI
 pWwmOwrfNsfF79tRW4w+KNewBhl0Kyr3Znu/UUlWZjXYwXb6IUhZAD/XlSDIwLEEvBmcoa+d
 FVfY7hDcttAB2nhyizhBgv/DXsZAV5Iv6+eDlPhiTPuAIm3EyQzCMjtb8idzk7hdEAoqJ/lp
 X525RT5c5zp/AtHNxA7cc6ML6K4z/2MGbxGVPXB2jbP4c6HF+Ig6LLwdwOlZGXkdozvdMPpK
 g=
X-IronPort-AV: E=Sophos;i="5.95,184,1661832000"; 
   d="scan'208";a="82383682"
Received: from mail-bn8nam12lp2174.outbound.protection.outlook.com (HELO NAM12-BN8-obe.outbound.protection.outlook.com) ([104.47.55.174])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 14 Oct 2022 10:40:25 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=B1ESBTt1HSoqYGl2ZwI6C1rnH7EbYiGXJKolBAjtycK0iuiHQ4dVXUE6xDYBxEBRpieRraw54UZ+0L4nvFVSRZ5yL3bEVxN48jmDR5SY+jAjybk10wRkfN0UoD6GaExITiNwKr3OIlrz+jlTn6CQeLly+pW0+FhY174Gzxv59xvbjncDWUdWaw+XJbUT/WISjAznaZjr5uZC+YrnV4c9xF8a94wEFXkzOKxwKYeWDLBacMZbJdUevGpqFqPXhc6wHGIsu68bl7adP/wHnjsxBVM6T0HEH4q++2U6aGVygkW4e31FumHQeMjvlmSRJQl7SU4il/Nu78+C+V4K1wA2cA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=OKTy+kOTlXxl5pFQ1KFBEbcJtN36CzhbfyiyjhybtSo=;
 b=GS4dNe/+pc4977Yz/r/ksC8Zxhd0XF06BTudWsi4bLLvERILI0EpZlikDa3RB4yH/Cn4Qy+IqRrlz30KdvrfBQib94Qy7sBRmRC+nkJnsWwKXMXKxDrZpyXhHGQBwjhAQlyGqJMgHygWzwe4CibqVAbhg8/PeXR2Xomem+N+jpO1sLj6Kk+B2ZTPIwTj2mbzyF3t7Z6+T7aZlX0f4wEDB8hbyS/4CEqVhrExwaWp4xEdUnXlv1uC99Bp/uYok1wsH6J+yJzwPvqoW7Mq2EIzyO0fyIhLWQaDcFKVPuDHrrnMLOFC786wKZewuGkk74I6anzZjan98GmHCv0C12CdVg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=OKTy+kOTlXxl5pFQ1KFBEbcJtN36CzhbfyiyjhybtSo=;
 b=BwT9SqHjvwbyJEMBZv82v0priV2o5PVz3ma2kPmJmp2Pr5mS5H6GyiXhH6bbX17DnBLq5PmJRRzBYZs3h7Cj43ws7poN9tA/ritMh0FzBx0/jhSc0lmYByaGWyxuddAZBixbWnmmW7Vo/6d5ojKvE4qepgQKBTPB7saVv254k34=
Received: from SJ0PR03MB6360.namprd03.prod.outlook.com (2603:10b6:a03:395::11)
 by DS7PR03MB5589.namprd03.prod.outlook.com (2603:10b6:5:2cd::14) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.15; Fri, 14 Oct
 2022 14:40:23 +0000
Received: from SJ0PR03MB6360.namprd03.prod.outlook.com
 ([fe80::861e:b46c:1acc:c576]) by SJ0PR03MB6360.namprd03.prod.outlook.com
 ([fe80::861e:b46c:1acc:c576%6]) with mapi id 15.20.5709.015; Fri, 14 Oct 2022
 14:40:23 +0000
Date: Fri, 14 Oct 2022 16:40:17 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Karl Pielorz 
Cc: freebsd-xen@freebsd.org
Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of
 FreeBSD DomU lock ups?
Message-ID: 
References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
X-ClientProxiedBy: LO2P123CA0009.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:a6::21) To SJ0PR03MB6360.namprd03.prod.outlook.com
 (2603:10b6:a03:395::11)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: SJ0PR03MB6360:EE_|DS7PR03MB5589:EE_
X-MS-Office365-Filtering-Correlation-Id: d82fc507-d95d-49d6-84cb-08daadf20629
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	D5NDIA1PD5wbfsoow0Z4akoTrcFXipKchkrTZoQo+ZoDsroUA5Gh9CdFcjgX3s2gyldKWNU21Mh1sZRubA7uB02HwIatFdgA8KNaKWTyqcP3KyZu0YKk87/ktziQhzIBTFk/zIfCcUs6eQe/mKeukJzlKuV26FEKOLYs0DPGAPQcJSOE3M+t1n6SU9kbQPyJjgIxsFyOqkVl20bxI7QvFUd1ssFaWnYm7LgjdDglsaZu92/34iXrKvUTrDt9MaZeaWKYGz0fZmEDRUWR+MOsenzsXx02yJMMJMQOe0FgV6A9NpEWzFSWG6X3K0xWisKKlJw+BjncY4Og7HJQwzWz1loA+bCUpYzBVGPqHe4G3x5bJOaINZ4zi0vG7KRs+9e7ZjRQlCXNUUwHrURpIAfp1Pust+5QnATHrn1ZytWmfnHOU2aM3EqBNftc7M09aEm2F5ZPGEnZ17jLZ+zeP4IIlREO037W1a5ti1XtKk+LD8bLABviQvdRZyvhX8HsOBxF9y6pTW4CgmBHXIS9f3nhb0GizhRtAPxDEwL+eKKz1NLHeiCZODqq0jG2YcMkNqotlWSi9uGiycmb6vNTqMy6ggZNxT2w9epvdK2czxxxvWfj+h9K1JXbShij2li6AoFW9j8+v9ABfYradWHyByDTAf7wehDfOYtsicMaXlEOqcLfrJ3k7yzCiyXDv67wDoNky4bSNSdEGX87hfUJnMzzLw==
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR03MB6360.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(7916004)(136003)(366004)(39860400002)(396003)(376002)(346002)(451199015)(6506007)(38100700002)(33716001)(9686003)(6486002)(478600001)(85182001)(86362001)(82960400001)(6512007)(26005)(6666004)(186003)(6916009)(83380400001)(316002)(5660300002)(8676002)(66946007)(66476007)(66556008)(41300700001)(8936002)(4326008)(2906002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SFAwTDBncmR4U2xqUnY5TDBRdWl4eFBjSVd5MDBneEpkMkp0WlZtTEc4WlVv?=
 =?utf-8?B?bUFTU2cyZU44OE5QNU1NRnNMRDZvTTBtUGZ1UFhyZlFUalltYVBqR2tBa0xo?=
 =?utf-8?B?ZDRTREpHSUV1eHF3eXdiUEZ6QStlc3NTVFp0TlRVQXVQd0Y3dlgvQzFzVWJv?=
 =?utf-8?B?TjQ5QmtGR1hqN3pYZDFRT0NadGx2YjRyYWh6R0c4Y2J4L1lkcjMxVit3cmtI?=
 =?utf-8?B?M0Z6WkYra0xNdXBEY1NER3VOdFNINTdSR3RlVG5ZdnRmQVZzanU0am01RHE4?=
 =?utf-8?B?ZFYrSWxvOHo4WjNoRGQ4TnNubDRwVXBYQnRPS05pRXNPaU5aM0dESERDalFi?=
 =?utf-8?B?WWdmc055Z3NZUzRWeDkzdHhWT0hMWmxac1ZEY1pXUGJ5ckZ4NjlvNWtpSkFj?=
 =?utf-8?B?ODNBWHR4TVhCQVZMR3l0SFJvTlNJTEN2V3dJSFBMZ3BPM3lLN0pCYUFGLzZS?=
 =?utf-8?B?eUVQNi9WT2c5dWVWclFXSFh1dkJyWWQ3Qzh4bnNtMndVeTJSWFpjQ0Y2bkNI?=
 =?utf-8?B?TjNWTWNpa2t6WC9wZnZKd1A1WUlEM2IrMlpIVVMrMnR1QW5TTmltUGlTb1VT?=
 =?utf-8?B?QU1ZMGtYT1kvdGhWWFlvTHp2ZUdQNG5oellkQTI2Ui9hUnQ0dndpQ21qTEx6?=
 =?utf-8?B?ay9XZFV3c2FaM2V2VGM0QTduQnRiZExBTGoydmVVMXRxOW0valQ5TTVNbEI4?=
 =?utf-8?B?dk5ib0ZwOVRlbklVb0JoMHBaRUpwbEJtMVozelJ3ckd6U2g3dWJLOEEyQXp2?=
 =?utf-8?B?dUF2NTBRVlk5aGJxa2pmZjFDUk01L2MwYUh1Z1hiMittSDh2Y2JUdTc2QlpQ?=
 =?utf-8?B?KzZjQVNnSFBHUTRmTUlRSWJ0OG0yNHFZejZSMEhxZUFibnFQZUpwc3BIMlhi?=
 =?utf-8?B?MGwyNGtKa2QxcmxUVEtWaE5lcmNsOE54WFpES1JGSGFuaE93WFl1cSs3dXMx?=
 =?utf-8?B?eFlzZURsSUdyYm5qbGdhZXpXQ2JPcTRidVdmblBDRGFCMjd5MEJKb0w4WXk1?=
 =?utf-8?B?RWhVQk9JL2NBaWNWN05BeVlpV0FDc0pYTDlick01SUlmQkZNU1M4L0dJd01w?=
 =?utf-8?B?ZnY2REdTUFNGYXA1cWc0Tzh1L3MzdUpoRUFQSFB6NFV1MzQ5aVJFODVLalRm?=
 =?utf-8?B?RmtCTVA1eUpoNm9heXFYMzk0QTVVdGc5SUZ2anZ3b1AxNGtVY3I0M1lHNG1Y?=
 =?utf-8?B?aFJKM1pjZXBIaDBWUUdvRDlsQUVYK1ovKzcwc1VRQVdyTVhHYzRyUDRFRW12?=
 =?utf-8?B?OW9jS3JrVTJDTEtOOWJRZlBNSDRNcWMxWkd2MWFYS24rSy8yR0QvVUU4OEpz?=
 =?utf-8?B?Rzd5NnVrZEYrR0xXdUtUYzRib1BZTHQrUGFzSmlKNSs5VnZMZXVXN0YyWUZV?=
 =?utf-8?B?MVhweG1jYlRjR1NCV3ZZSENoVTZ0bHZpdndHcmcySGxTemxQakw2QXhNVjdS?=
 =?utf-8?B?NUVlSWZta2ZxZlhPeTk1NlZLUWN1Z0hoLzE5eitQRGU4dDJSd1hmcWtRZmZP?=
 =?utf-8?B?UlNZc2FiY04yRUxSMGJkQWhHaDRzZGlsNDhMbXVFcHpvVGwwL1JlSjh0Y21u?=
 =?utf-8?B?Uk53b1M1bXc3QXdHaTNZK3BaUU5TL2VERHYvWkhNWEdNWEhZNmdyWHNIc1Vu?=
 =?utf-8?B?UW95Y3BCYXpIVTZsMXUrY0N4dmtYaVM4cm0xdmlsNCszM3dZclUwa3hqWE1O?=
 =?utf-8?B?ZUQ3T2xFbXVXc3dHdlE1YmdYbTVxM2tQYkc0MEkxNXhQVm12M205ekZLQnEx?=
 =?utf-8?B?NUloY1BHK1hiMHNlMHdsUHMwUEkzM0FvRk5ONm5JWnY3NHo4eHE2ekwyWmpx?=
 =?utf-8?B?Q0VXYmFwUmd2SVpzUmNtVTNRbHg1ZXZQVm9RT3ZZK3BOUmE1cUFZaDlpRHNs?=
 =?utf-8?B?QU1Yb1prZlJQWUtMWDdURHYvZWsyU0JVZ2R3SDMzankxeVhjTm03SkZ5dmpx?=
 =?utf-8?B?enJ1MGs2d1BObmFEZ2c5cTF5cHlQOVp5UXBxb21HZWhkbWpSNkxWVlFWYlJh?=
 =?utf-8?B?c3VBTTZlYWkvUkdDSkU4L2tPV0o3SGR2a3prRlV6eDRONVF0dzZPQzF2TTF3?=
 =?utf-8?B?bk1QUk5neHBtQzdLY2t2cUsvOXVsL1VZWWxBYzNJL3RTd21STHR0bjUvOHNl?=
 =?utf-8?Q?KEsA5mB20J7ivJ/yVFl5BcpSr?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d82fc507-d95d-49d6-84cb-08daadf20629
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR03MB6360.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2022 14:40:23.1104
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +sPsbCNGknrnW713hV8j7ubSFFtGT1R2SsVCGvIsf9/Fy2QV6ms+cFf6yI0rsNDAq7HJG4YPZzBRCmVuHcoVww==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR03MB5589
X-Rspamd-Queue-Id: 4MppwF6rlyz450l
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=JDVvcR9h;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=BwT9SqHj;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=27923c854=roger.pau@citrix.com" designates 216.71.155.175 as permitted sender) smtp.mailfrom="prvs=27923c854=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.19 / 15.00];
	DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.99)[-0.992];
	DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	MID_RHS_NOT_FQDN(0.50)[];
	R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	RCVD_IN_DNSWL_MED(-0.20)[216.71.155.175:from];
	R_SPF_ALLOW(-0.20)[+exists:216.71.155.175.spf.hc3370-68.iphmx.com];
	MIME_GOOD(-0.10)[text/plain];
	FROM_HAS_DN(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FORGED_SENDER_VERP_SRS(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=27923c854=roger.pau@citrix.com];
	ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US];
	RCPT_COUNT_TWO(0.00)[2];
	RCVD_COUNT_THREE(0.00)[4];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	MIME_TRACE(0.00)[0:+];
	RCVD_TLS_LAST(0.00)[];
	TO_DN_SOME(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On Thu, Oct 13, 2022 at 11:00:17AM +0100, Karl Pielorz wrote:
> 
> Hi all,
> 
> We've been running FreeBSD as a DomU under Xen Server for years now, and
> only really had a few now 'known' issues (e.g. with networking).
> 
> We've recently setup XS 8.2 on some new Dell servers, and whilst everything
> appears fine - twice now in a couple of weeks we've had FreeBSD DomU's just
> "lock up".
> 
> There's no errors logged, no kernel panic, nothing - they just "stop". Xen
> reckons the CPU is pegged at 100% (with brief periods of zero) - the console
> is still 'available' (but locked up) - and the kernel is dead (i.e. you
> cannot ping it).
> 
> Aside from the "Has anyone else seen similar" - with nothing in the logs, no
> panic, nothing - I'm kind of at a loss as to how best to troubleshoot this
> further?
> 
> The VM's are lightly loaded - haven't run out of resources (RAM /CPU etc.) -
> and it's only happened a couple of times now (but in a couple of weeks) -
> whereas our setup before never experienced this in it's lifetime.
> 
> One was a legacy 11.4 system (amd64) - the other was a 12.3 system (amd64).
> 
> A forced reboot brings them back (with some file system damage - as you'd
> expect from a crash).
> 
> Just at a loss as to where to look - given the lack of any panic/errors etc.
> Any suggestions?

Hello,

Sorry, been very busy this week and forgot to reply earlier.

Could you try to setup a watchdog in FreeBSD and see if that
triggers?  So that we can get an idea of where the guest locks up.

Is also the 100% load on all CPUs, or just one?

If the watchdog doesn't work we can try other methods.

Thanks, Roger.

From nobody Mon Oct 17 08:48:21 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MrVyR6Xc4z4fJ7T
	for ; Mon, 17 Oct 2022 08:48:31 +0000 (UTC)
	(envelope-from prvs=2822468d9=roger.pau@citrix.com)
Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MrVyR0S3yz3PrZ
	for ; Mon, 17 Oct 2022 08:48:30 +0000 (UTC)
	(envelope-from prvs=2822468d9=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1665996511;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=3Wdsq5dwlHQ81UnOWBxYRj0G4NyVU4XbDC4KpRdlEUA=;
  b=dCyv+eaPrAsL1xyJXkS60gGt/YgXV9FEGj0HdYppYWHSnu/u5phXzCgz
   3MDKX3wJWcomjAWBXnAwOwTmOq6KfU0Td/qMean5ykN+zzbLZI5SBF3vq
   rMGp65entEgOpAM2JVVTWUP3bR4LhEWTjiWVpvfbtfWQNEu6MrMYrE2Xv
   E=;
X-IronPort-RemoteIP: 104.47.56.41
X-IronPort-MID: 83301499
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:ilXa0aOSA9MBzgDvrR2vlsFynXyQoLVcMsEvi/4bfWQNrUomgzUAy
 jNMXmyBPv+Namb9KYxzPtvnoENTvZSHn9ZnTQto+SlhQUwRpJueD7x1DKtS0wC6dZSfER09v
 63yTvGacajYm1eF/k/F3oDJ9CU6j+fQLlbFILasEjhrQgN5QzsWhxtmmuoo6qZlmtH8CA6W0
 T/Ii5S31GSNhnglbwr414rZ8Ek15ayr52tB1rADTasjUGH2xiF94K03fcldH1OgKqFIE+izQ
 fr0zb3R1gs1KD90V7tJOp6iGqE7aua60Tqm0xK6aID76vR2nQQg075TCRYpQRw/ZwNlPTxG4
 I4lWZSYEW/FN0BX8QgXe0Ew/ypWZcWq9FJbSJQWXAP6I0DuKhPRL/tS4E4eGKMHyN9lWl133
 9NBeHc2Tx7YrNO6+efuIgVsrpxLwMjDGqo64isl4RSCSPEsTNbEXrnA4sJe0HEonMdSEP3CZ
 s0fLz1ycBDHZB4JMVASYH48tL7w2j+jLHsF9hTJ/MLb4ECKpOB1+KLqP9fPPMSDWO1en1qCp
 3KA9GP8av0fHIzEkmLcriP97gPJtRvVXsFNSbfiz+FFqwK930cWKBIUS0Tu9JFVjWb7AbqzM
 Xc8+TIvoKw17l6cSsLhdwG0sDiPuVgBWLJ4Hfc35QuWxu/E7gCxCHIJQzRaZJohrsBeeNAx/
 lqAntesCTk0trSQESiZ7u3M8Wr0PjUJJ2gfYyNCVREC/9TovIA0iFTIU8pnF6m2yNbyHFkc3
 gy3kcT3vJ1L5eZj6klx1Qmvb+6EznQRcjMI2w==
IronPort-HdrOrdr: A9a23:bmVteaGbq4YKDGRVpLqFc5HXdLJyesId70hD6qkvc3Fom52j/f
 xGws5x6faVslkssb8b6LW90Y27MAvhHPlOkPIs1NaZLXDbUQ6TQL2KgrGD/9SNIVycygcZ79
 YbT0EcMqyOMbEZt7ec3ODQKb9Jrri6GeKT9IHjJh9WPH1XgspbnmNE42igYy9LrF4sP+tFKH
 PQ3LswmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZXbxp/hZMZtU
 TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczKgBkF5MWUrHo6mt
 jFpBkte+x19nPqZ2mw5SDg3gHxuQxenkPK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL
 9L93jxjesaMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO21GkeKp
 ghMCjg3ocWTbvDBEqp/lWHgebcFEjbJy32DXTr4aeuontrdHMQ9Tpr+CVQpAZDyHsHceg12w
 31CNUXqFhwdL5mUUtcPpZ0fSLlMB27fTv8dESvHH/AKIYrf1rwlr+f2sRH2AjtQu1C8KcP
X-IronPort-AV: E=Sophos;i="5.95,191,1661832000"; 
   d="scan'208";a="83301499"
Received: from mail-dm3nam02lp2041.outbound.protection.outlook.com (HELO NAM02-DM3-obe.outbound.protection.outlook.com) ([104.47.56.41])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 17 Oct 2022 04:48:27 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=odQzZJeKSZFnRz9x7xItRsqvOeP4PRifgApyV27WCKDviAUiBPt3UC59Jay91xNMcxygcs9A/5XQLJrpx+Ac67mTpHbAr4D2J9PnaoS9uhhdx69F6f/wBdGVVzwyEO9nkRVkl2sCzQR/wXmWdlpWgKnauz9CKz/sE/1/91jc2bWMFmzFw/uHjqmrH3Xj7j9tw4UxdWySvF9c1nhKhyJfqEaPxcM47J37Pmw/wyuSXCRk3Uf5x4YXw5Ze9LLuNzFoxWNTeQt+HjXf14JWT0zLgGnQ9R0xZla3pZD/+SfLH3v6+AP5LBT4iyy4pA7MtosZD2Zu2Gg+G/My76pLetKRBQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=CIVYCIAGU+dnUiZiL22MWNZBL+PVSRedETRZ2hmG/Kc=;
 b=joF9Ux5NlZDDotJJA7VnobTRu7SvLT+Sb3F9+vSD5uPjg8zqRGR/UkfLCurusXVo1WAZtxsi4pTKACBN8duN9kSCqPBUmqPoViKlETjD3tlsT36cJ3GyoJCaovCebd71hrLOglRMJ0j+UQuugJ8xYlOJIXY1tNndROH5cjI/gpRL9J03YwmfIACPwmrwTWPZthiNv46GEk6mFRnO14rjgQpK8acqZqAEOWSfKn/zUrUpQOuyp6nOH2cJ5QSAI56Dcg80LU8kSFIGkrJy6LHA/OH0D8yQ1cXFg3FNv1uuiXpqjEbQhdt1b4bbOwXN2Bal6kVdrjrVqWr+sRcXxS8JmA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=CIVYCIAGU+dnUiZiL22MWNZBL+PVSRedETRZ2hmG/Kc=;
 b=DvVk4RvlSf47ftVwtbMtk/xDgLG7vDuvk0V6TwUSNqh+AQXE4pTA+M8dy47TBbvjCaxPJhBj0Jbg0yfASC8g3n31C8hpnJPUhr5s0WSK1QRsleIqBc52/3DTZ3GwWnPheCaysCTzsH8WM7loxF20sq7Mr517HWIoz44wfTfASGU=
Received: from SJ0PR03MB6360.namprd03.prod.outlook.com (2603:10b6:a03:395::11)
 by BLAPR03MB5476.namprd03.prod.outlook.com (2603:10b6:208:29b::19) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.32; Mon, 17 Oct
 2022 08:48:25 +0000
Received: from SJ0PR03MB6360.namprd03.prod.outlook.com
 ([fe80::861e:b46c:1acc:c576]) by SJ0PR03MB6360.namprd03.prod.outlook.com
 ([fe80::861e:b46c:1acc:c576%5]) with mapi id 15.20.5723.033; Mon, 17 Oct 2022
 08:48:25 +0000
Date: Mon, 17 Oct 2022 10:48:21 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Karl Pielorz 
Cc: freebsd-xen@freebsd.org
Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of
 FreeBSD DomU lock ups?
Message-ID: 
References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
 
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
X-ClientProxiedBy: BN9PR03CA0463.namprd03.prod.outlook.com
 (2603:10b6:408:139::18) To SJ0PR03MB6360.namprd03.prod.outlook.com
 (2603:10b6:a03:395::11)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: SJ0PR03MB6360:EE_|BLAPR03MB5476:EE_
X-MS-Office365-Filtering-Correlation-Id: 61c3f831-eb5b-4f84-3a5c-08dab01c5a87
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	5zAo0LW4tEAYlStBoK83cyYgPti3nyvNQbsC70S/qvtpDdYCpMbO2AavLImWTO1AO1syLE9C9thYAuVZS5GJ1pqI+cqI28uO3OT8X8BC853vu9XTia+AKsIlxj9qDNkUT1QQ1K4cMbdTVS5ZdmWXx4/z6gZWF8mr9HpB0McCXW9c7SpiAJPN5sNy8wPZTF6tMX/FnTdMYb9cqn6Eefn76VUPxiygJ5QaeN2phCJO8b1tiqRl1TDSGjKvIHxfuvqBMY6Ob6svPD1kpm3w4DqwRwucH2eL9W5rfQuLSpQ5TIarTLGSBuGohRmreOyHfSV+mVRRP5WwPBggjadO4/DmCjyULxEo68OimdPKdlIfyv77p4LUc1gG9PkBtq087ea0SQ2TP6QE87M9LXixAS0Rp64YdrbzVG0uTFzClcByOw7ZJv/rXutxRiQZFv+EohpAUgnFyVYTZJ0SVnwr2R/+M58aX4Z470OyZANPpOtZ9jNgzN6am9S2tbz7ARziQzDuLG+KITXBycQ7WhYDCVADGyR7Dl3jWT6t3/FCSVxjAQF7EJuHiZxG/Vj7yYmdT0M1JuzG2IM8s0iN3w42DhFtu13CRHkJ6N0gk0iwS2xJKQE3/+z3mfNtCfwVG4XQdSfFyu8Z1X/XwpLC7+YT2bBJaHuCr+/6IzL4tybbtDx1xB8e+ugnJlYBLzCNRDJYBxOl8ETXBWk97H0WSih3CVRfQ7kuNiE69BND4Ljr/vciBaAvblF4ARpq5Pi44Y4nXWAI
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR03MB6360.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(7916004)(4636009)(136003)(396003)(366004)(346002)(376002)(39860400002)(451199015)(33716001)(86362001)(82960400001)(33656002)(85182001)(4326008)(41300700001)(6512007)(26005)(6506007)(8676002)(6916009)(5660300002)(9686003)(8936002)(66556008)(6486002)(478600001)(6666004)(66946007)(316002)(38100700002)(66476007)(186003)(83380400001)(2906002)(86696005);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VzhlSGhZSUlIVVNYYUhMdGhkT0t6dEQvZXpsU2wvS0dWWDNoWEF6d2tXbTdw?=
 =?utf-8?B?ZGxHS2RFYkZoQ0o1eWtDQXNoOFY0R0MvQ2xMaUZRdkRmOEtHVHpHcXh3Mlgx?=
 =?utf-8?B?UEpNM2VSLzBncXR2by9sK3ByUlpDVnJuMVNCTzZKSTJLdkpranRuaGJJNC8v?=
 =?utf-8?B?Tk5UVnY2K1krSWdOcGk5NTlFYUJsZ0o2YW1qVWd4MjJPSW9TQTc2UnFiSVhN?=
 =?utf-8?B?cWd2dTJzODFEanExc3B0VGtJWWZ0RVZaZGFvVDhGamwzY1A3YWRiTyszZmlS?=
 =?utf-8?B?OHRqYStMQkh6NFNGeEJUeWlwemQ5WWhjQXp1Q2o2K3BRVGEyWlZ0ME1RQ3hv?=
 =?utf-8?B?eElvb0lMNWdjS1lKaDhzTTRlNG54T2U2WXpraTZjS00rdTJQdGtFbW92cWFa?=
 =?utf-8?B?T2tZc1o0MW5Vall6MmxnNG1mYSswclMvVU45VU9NK3M4aCtIT3lVeGNoMnpl?=
 =?utf-8?B?MDMrOTBqb3dlcW5BK3o4Vkd2OFUzQjZ1Mm1NKzFSTTR1R2djZFQxZGw2R3dm?=
 =?utf-8?B?TmllbkVVNzVnc1hkUXJFMUN0UTBzT3RYN29helN5YUdReUVOOThobll2K0ZV?=
 =?utf-8?B?dVZpVmJUbHRoYzVqZjhpZXAzVUxXS3k5WXkxWC9ucklmZXg5cStlMTlBS3Zk?=
 =?utf-8?B?ajd6U2prQUxLRVROQ1g4MmxGSWpVZm12cmc5NjQ4WCtCODRickRFVDgza0to?=
 =?utf-8?B?eXBpcFlPZ3JjZW95SXpLZy9nd1pDVUFzYVE0aGRDUUxsR01pNnVhL092NzN4?=
 =?utf-8?B?V205NnJnb2hZQ1hkK3Z6MXdkNlRCMWNjRGtucVdQcDRYc0IvemFHNHVXcStw?=
 =?utf-8?B?ZDZUS0ZaZzdsc2FJUUR5c0hmSURLdFBiSW9CaGdUMkZiNTg2MDVlRzRxMk8v?=
 =?utf-8?B?M1VWQlZ3MnN0RGsyNngvQ2JHcmhuY21TUmN0SUVIMCtsQkxPRTdHNnV1S0Yr?=
 =?utf-8?B?YjZaazh1VU94cWVDUURrWGkwbU1qS1oxS1hqUUdET0hJMGd6Q1RBNXpMR29N?=
 =?utf-8?B?Q2UzMGVCM0tlN0hEOWFZeEVhQzBrTml6Tkg4Z3FRbVArbmQyalpzZjk2cXRB?=
 =?utf-8?B?emJHbG5VR1RaSm5MamYrQzZPenVWcTNhVXJLd0FBcmk0eDNOb2tBbzdpNUNv?=
 =?utf-8?B?OXBIMzZ6dXRKa1Bxay9NaWF4Y0NSdy9zOTBFbVU5b2E2S1VGZVo3NkN6SUpv?=
 =?utf-8?B?eWQ3emRFZDgzVHluYWRLQWhtaE5WY0t2M0g1c0ZoL3h6ZFkrMmZnSUc3TElx?=
 =?utf-8?B?aVNnVEtmT0J6UHRPSGl1VENHYUZxMFVUeC9TaTRyNE9xSlZyWE1EMStRRlNN?=
 =?utf-8?B?cnpLSXVZTUpNcDA2aEZtS2YyOUVTaWtxZG1PZ3lNZnVHdktScEdRUDFLdjNl?=
 =?utf-8?B?VlpUaUdtMnN3Rmp6MmI1WHRhQjFtZ2p5dlZCYWR0SU0rOUhMeHZxaElTNTVi?=
 =?utf-8?B?ako2dFo3THVoN3BOeE1hVXdqSXcrcW1ZTjcwVzlpTUpGOHhMdy9mWHJhTUlt?=
 =?utf-8?B?azIzbGVuZk1Tem5TZWFEZVdVZ2h4b1ZsaHVmWXQrb3FzTC8xVmo0S3A5cTFG?=
 =?utf-8?B?RnI4bW4rTks0b2s0Wno0RWdPZ3NKSjB3SkNid0NpVmkwQ01DeWk2RHhZQnc0?=
 =?utf-8?B?OFNLWVlWWU5OSjI3THFPS3hMSjR5cXFFbEkwZHliY0lzQTZqNHhXR293dURJ?=
 =?utf-8?B?bnN3QlBvYnc3aTJiQjIvUTE0TzZDYXJiOEZpUVFxdGlIZDZtdm16VTBCazl4?=
 =?utf-8?B?cnZpTVp5TjcrTnU1ZERKVS9rcUw2TGJQVjdZM3VNVG1PZDE4c0R1RGZQQjFO?=
 =?utf-8?B?dnZuSHJPNGs2Vnk0NmhnZjZRL0xVM1BzdnZ3K01ZbTBHS3Q3ZlpJOGliMzNl?=
 =?utf-8?B?STRPQ29qcjJtZFVwdUh1QW1DM1RPVU1zNzlGMFhsTXlaNDA2VHVaNVVHUUZC?=
 =?utf-8?B?eXNpNEZiaUpnNk1scGEwckF3RHNMRnFGVVdJaXNlNDNsMzJYREVyWmRHL0hk?=
 =?utf-8?B?RzRJVHZVa3Y2Q24xMlgyYnZqWVVqdDV4d1pXNnpZeWNPSmJIa3pHd3piL0ZB?=
 =?utf-8?B?QkZXWFdlb3k0QXBNSGNScFluSTU1dDFpZzBDQ1lDVnBrNG9ZVk8wNkFoamJI?=
 =?utf-8?Q?5S48pm9De2/hNDHZJMnsQIA0V?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 61c3f831-eb5b-4f84-3a5c-08dab01c5a87
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR03MB6360.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2022 08:48:25.3690
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: j1qj1ipedcBrzrTuEAp8SrNlXJF4FPy06GYEgCQZkqewzHAlX7xKa+v0MtKxVuTl0h5Q27685gU1IzLbEiz8mw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR03MB5476
X-Rspamd-Queue-Id: 4MrVyR0S3yz3PrZ
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=dCyv+eaP;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=DvVk4Rvl;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=2822468d9=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=2822468d9=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.20 / 15.00];
	DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	MID_RHS_NOT_FQDN(0.50)[];
	R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from];
	R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com];
	MIME_GOOD(-0.10)[text/plain];
	FROM_HAS_DN(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FORGED_SENDER_VERP_SRS(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=2822468d9=roger.pau@citrix.com];
	ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	RCPT_COUNT_TWO(0.00)[2];
	RCVD_COUNT_THREE(0.00)[4];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	MIME_TRACE(0.00)[0:+];
	RCVD_TLS_LAST(0.00)[];
	TO_DN_SOME(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On Fri, Oct 14, 2022 at 04:40:17PM +0200, Roger Pau Monné wrote:
> On Thu, Oct 13, 2022 at 11:00:17AM +0100, Karl Pielorz wrote:
> > 
> > Hi all,
> > 
> > We've been running FreeBSD as a DomU under Xen Server for years now, and
> > only really had a few now 'known' issues (e.g. with networking).
> > 
> > We've recently setup XS 8.2 on some new Dell servers, and whilst everything
> > appears fine - twice now in a couple of weeks we've had FreeBSD DomU's just
> > "lock up".
> > 
> > There's no errors logged, no kernel panic, nothing - they just "stop". Xen
> > reckons the CPU is pegged at 100% (with brief periods of zero) - the console
> > is still 'available' (but locked up) - and the kernel is dead (i.e. you
> > cannot ping it).
> > 
> > Aside from the "Has anyone else seen similar" - with nothing in the logs, no
> > panic, nothing - I'm kind of at a loss as to how best to troubleshoot this
> > further?
> > 
> > The VM's are lightly loaded - haven't run out of resources (RAM /CPU etc.) -
> > and it's only happened a couple of times now (but in a couple of weeks) -
> > whereas our setup before never experienced this in it's lifetime.
> > 
> > One was a legacy 11.4 system (amd64) - the other was a 12.3 system (amd64).
> > 
> > A forced reboot brings them back (with some file system damage - as you'd
> > expect from a crash).
> > 
> > Just at a loss as to where to look - given the lack of any panic/errors etc.
> > Any suggestions?
> 
> Hello,
> 
> Sorry, been very busy this week and forgot to reply earlier.
> 
> Could you try to setup a watchdog in FreeBSD and see if that
> triggers?  So that we can get an idea of where the guest locks up.
> 
> Is also the 100% load on all CPUs, or just one?
> 
> If the watchdog doesn't work we can try other methods.

Forgot to mention, you can also try to send an NMI to the guest when
it's frozen and see if that generates a trace.

>From XenServer host (dom0) command line:

# xl list # get domain id
# xl xl trigger  nmi 

If you don't know which vCPU is stuck you can always use vCPU 0 and
hopefully FreeBSD will print a trace for all the vCPUs before shutting
down as a result of the unexpected NMI.

Regards, Roger.

From nobody Mon Oct 17 12:08:51 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MrbPb1N89z4fj0J
	for ; Mon, 17 Oct 2022 12:08:51 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MrbPZ755Zz3g09
	for ; Mon, 17 Oct 2022 12:08:50 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MrbPZ62ZYzp08
	for ; Mon, 17 Oct 2022 12:08:50 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HC8okd084936
	for ; Mon, 17 Oct 2022 12:08:50 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HC8oDG084935
	for xen@FreeBSD.org; Mon, 17 Oct 2022 12:08:50 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 261059] Kernel panic XEN + ZFS volume.
Date: Mon, 17 Oct 2022 12:08:51 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 13.0-RELEASE
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: royger@freebsd.org
X-Bugzilla-Status: Open
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: assigned_to
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1666008531;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=kDiAxKxYixz7MOcz1djtBqfnZuDNFwxQTMOclbXhvnw=;
	b=faPSe/g6dORTSkEqr7zSUdg0VBSAQ/9DnsxTHf2jVMjI2qp1Eu83SfKEvrEJFtaZfM6j7j
	KXlE6im7GtxzK7Nax4PX9ot82Igt/T1VOnDemWHGnokPC5IM5CpCjvm6fXZF8OJ6e43a6K
	7cLnubKAbYrxdO5/VnXz6lO4kcev8CVZuFAfCDQyR/oZfwgWj9pp5oX5GQ2DlD1/GowK6K
	ppcM5aS8Q2YHtY5NXthsXPrRvXMCupNTqk9dhnsgzKkrorqKubF82RBe/gkCBgFoCrept9
	BNmL0GXZd92YLHKON+eNB+18IODWklWybSK8XZyKcG/RQ6C47p03Q6HkVk5G2g==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666008531; a=rsa-sha256; cv=none;
	b=a2h3B6Ar7+L1J63/L2EnAZsN+eWDw3NUaUIez2iARF8pmSk8QsccgvM9w/YMl0mVStEbpo
	9hXqBTNqF+3PxHXk43Z89LhOrhohCDX76iJwXQDr75HUz6tdPnfjkrlSyFttiBzGHLDFKA
	Cs//NUEn72zAlCU1/C1KUNnk1/OOtDLmYYItCFRiAvJpBWIjq2Zx917vtBS9O/PMcBh33Z
	h0NPKOlUEV6YFn3Qz5XNU7jQ/tsHIqzD5zQZ2O0rm75LFi6UZCPqUgOK4rKO6wKPnmCKha
	C54K9otgVwY34RoU8Q2wwPN32h2BKbjThgWUa1dKioDsjS/6PX+M6inQ3Pw9SA==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261059

Roger Pau Monn=C3=A9  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|fs@FreeBSD.org              |xen@FreeBSD.org

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Mon Oct 17 12:09:19 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MrbQ71J5Zz4fjB3
	for ; Mon, 17 Oct 2022 12:09:19 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MrbQ70Hp4z3gTC
	for ; Mon, 17 Oct 2022 12:09:19 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MrbQ66T7rznnt
	for ; Mon, 17 Oct 2022 12:09:18 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HC9Idl085225
	for ; Mon, 17 Oct 2022 12:09:18 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HC9IhN085224
	for xen@FreeBSD.org; Mon, 17 Oct 2022 12:09:18 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 261059] Kernel panic XEN + ZFS volume.
Date: Mon, 17 Oct 2022 12:09:19 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 13.0-RELEASE
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: royger@freebsd.org
X-Bugzilla-Status: Closed
X-Bugzilla-Resolution: Feedback Timeout
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: resolution bug_status
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1666008559;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=masl2OZA7XFx0u1ZGnK+Nk6JQpUAgkpZQEYf+yIdD8o=;
	b=a0zlgnprMhjOuT8ZAksoZWvKjoeUYMcSCRGGuhBTiOnsPEOuk+IvgKbOEIG4Oyw+lTU/46
	VErQF8bZ2DJsVo10+5HjczTfuecDyM+l60xdpT+LYNWenUQik2AgWVxDicHJAOuZcvvkW/
	LkFOpuO7jMNPb94DGSW2/ZHh1ERlT2HcXQZbjhTIfDalOZefyyAZW7TuJGLAVyP8jsh2il
	+h/Iiwm9l0O5FSWpwibM00jsr+N8hfyS3v+H5ap/9mC+kXbvRTr11IMyVbcVckH08M8KzI
	LU6tf+h0z2KxO217knIH2wLLjuSDpBSl2+S2WB1vNOwqvhSXhU5IsroZwpry1w==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666008559; a=rsa-sha256; cv=none;
	b=brq3HRcuTPR3tTTzMov82SA9zVns7Ess4t0SB2kYUncSHziZ59cd+T8UoEfe92eH5WkKs5
	6u4p/Open6nBcK2VHgEhPwZ0KnAzUnK7Jg09OMZQeeFFt19IXOHjx4/bwAf9q0HGzu/AdI
	rJxEf6Erg2hPuu+CQzDj3UPVZQWfD+eF9vCEaDr49g6X/KBzqXpffXCZGczIpqfynQzw+0
	umydj/jXXtlIAGTRFq4/3kbtuTu4s730nWdlcNtXexpa6ZTrioEuyM/DGkl6ZLjnWijjXu
	KYr/3hpKVdko4zt2MkkTsGbQbg5uXEvG1oJ76HKHCwGS/kvIrOwNDLqh++2TWQ==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261059

Roger Pau Monn=C3=A9  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |Feedback Timeout
             Status|Open                        |Closed

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Mon Oct 17 12:23:38 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mrbkg0DXSz4fm12
	for ; Mon, 17 Oct 2022 12:23:39 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Mrbkf3ZzCz45Jj
	for ; Mon, 17 Oct 2022 12:23:38 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Mrbkf2Ng3zpRb
	for ; Mon, 17 Oct 2022 12:23:38 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29HCNchb016397
	for ; Mon, 17 Oct 2022 12:23:38 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29HCNcpp016396
	for xen@FreeBSD.org; Mon, 17 Oct 2022 12:23:38 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 261059] Kernel panic XEN + ZFS volume.
Date: Mon, 17 Oct 2022 12:23:38 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 13.0-RELEASE
X-Bugzilla-Keywords: crash
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: grahamperrin@freebsd.org
X-Bugzilla-Status: Closed
X-Bugzilla-Resolution: Feedback Timeout
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: royger@freebsd.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: assigned_to
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1666009418;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=tcujU1XapxFotSdjb9t7BwnCY7deQhSX5JD7Wjbu4EY=;
	b=E/+y9lXY/w2+oQ19QZfnS0HD6MJhJKdC5W0AlV3Kbe1DbEzTGJJrbwTk3bk4wqkeo5bBkZ
	5c+drGsLJVewHBwSzHnKd+4xH4XZtF4qDyNzSi5LGxIRBTWUk9ZcwvjMHNEYQPng2IZuRx
	czXiIc5DG7AaZG2FSRPn+7b0yNwb6YVh38lx4QG/9cpfCJHVzjxPKt7kOWFF9rJKoHrGJb
	9CTzl2abfm1f11NhpzwokcmXGQ1hMn15ZydT5OxzJVYHqTRLV/FJeFtcZcxKapU3SFuEBN
	UA9528dQqEXSzDDoDfsSRSsyRNz+09oBXICRnJff1uL6vvzLscwsdDWcCArOdw==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666009418; a=rsa-sha256; cv=none;
	b=mOWzNny8BVaKJpKoCXL1d/2LYr/uU8iTbcbyXKPY++lkGcnymzIXo/DxGN4MTIiDZAg4ft
	qc0OT3YW2dO53/XycuLCG8n8+gacF1vr08Epe+6iKqt4Bvr5fgnpM68JUl7eBDS6VqDC0/
	S5o9hjw0Mq4koBH2PAsV8/z70l0wTpoexLOJcMT/4XfGnvf218nDc0eSZqFsYOg0FRZR1I
	wlSRo1eB1X4IS9lBlFmsGJWNzNkO2NojF7jHmTz2bw8b8k8hcHldZ7mK6plnuNC7Z0zwby
	mdUP7u983aXA6+rSLETrbxlEWcFuhCInImVUYF+JpjFtwyGLriIW2RTzM9Tifw==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261059

Graham Perrin  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|xen@FreeBSD.org             |royger@freebsd.org

--- Comment #7 from Graham Perrin  ---
^Triage: assignment to the person who closes the report.=20

(Thank you.)

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Tue Oct 18 11:15:28 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MsB9z1wf8z4fHNG
	for ; Tue, 18 Oct 2022 11:15:51 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "*.krpservers.com", Issuer "RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MsB9y1VQlz413R
	for ; Tue, 18 Oct 2022 11:15:50 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from [10.12.30.106] 
	by smtp.krpservers.com (8.16.1/8.15.2) with ESMTPSA id 29IBFSuS057672
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 18 Oct 2022 12:15:30 +0100 (BST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tdx.co.uk;
	s=krpdkim; t=1666091730;
	bh=yh4q4YL5FJjHP/Oc38f05wd7CN17bYsEqz8lrnGpV5Y=;
	h=Date:From:To:Subject;
	b=gfk+zTimf4FooUV41GJXdk1ybjPy5tUPHlYuy+VsFRB+kyTNKnyJiqQlAWuSrag4H
	 csoKrkZbca1E9aVYzsCsPeRmRml5A8PkeqxA50zrmgCZb7+IZx4IH+i5tOxzQ4ySb4
	 S0rrgdZQ6jNippTJEh/1WREwfeq8Qm/AVcCE61oek8lhM3Z9L+jnMOnyx4rv55yRgi
	 3Mi2iSew/sHb5BjWfcPhTi9Y88hAdCqPHVcAsSm9Rgyw9B0rV9Kl1aBFg5OZwCE2OY
	 ImrAXXbJGu+rusJFNFwRr7yrmjD7EPvI034PTcl5KNKZFAmkZy4bLPC0fbZf128cCp
	 KZ027htbcsknA==
Date: Tue, 18 Oct 2022 12:15:28 +0100
From: Karl Pielorz 
To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= 
cc: freebsd-xen@freebsd.org
Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of
 FreeBSD DomU lock ups?
Message-ID: <91CF514F28FADFC45E157B96@[10.12.30.106]>
In-Reply-To: 
References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
 
X-Mailer: Mulberry/4.0.8 (Win32)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
X-Rspamd-Queue-Id: 4MsB9y1VQlz413R
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=tdx.co.uk header.s=krpdkim header.b=gfk+zTim;
	dmarc=pass (policy=none) header.from=tdx.co.uk;
	spf=pass (mx1.freebsd.org: domain of kpielorz_lst@tdx.co.uk designates 62.13.128.145 as permitted sender) smtp.mailfrom=kpielorz_lst@tdx.co.uk
X-Spamd-Result: default: False [-2.49 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	NEURAL_HAM_LONG(-0.99)[-0.992];
	MID_RHS_IP_LITERAL(0.50)[];
	DMARC_POLICY_ALLOW(-0.50)[tdx.co.uk,none];
	R_DKIM_ALLOW(-0.20)[tdx.co.uk:s=krpdkim];
	R_SPF_ALLOW(-0.20)[+a:smtp.krpservers.com];
	MIME_GOOD(-0.10)[text/plain];
	ASN(0.00)[asn:60969, ipnet:62.13.128.0/24, country:GB];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	MIME_TRACE(0.00)[0:+];
	FROM_EQ_ENVFROM(0.00)[];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	DKIM_TRACE(0.00)[tdx.co.uk:+];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	TO_DN_SOME(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N



--On 14 October 2022 16:40 +0200 Roger Pau Monn=C3=A9 =
=20
wrote:

> Hello,
>
> Sorry, been very busy this week and forgot to reply earlier.
>
> Could you try to setup a watchdog in FreeBSD and see if that
> triggers?  So that we can get an idea of where the guest locks up.

Hi - no problem / thanks for the reply...

I'll give the above ago - part of the problem is not knowing which VM is=20
going to die (there are quite a few) - the second part, is the waiting=20
game...

> Is also the 100% load on all CPUs, or just one?

>>From memory of the graphs - I think it was probably just one (I think the=20
last VM that locked was a two core VM).

> If the watchdog doesn't work we can try other methods.

Well, I'll get back to you when it happens again (personally - I hope it=20
doesn't happen again) - but at least I know it's not quite as much of a=20
dead end as I feared debug wise.

If this does happen again - and I'm able, is there any point in doing a=20
snapshot + memory of the VM? (which is about the only thing I could think=20
of - not knowing about the watchdog stuff).

-Kp

From nobody Tue Oct 18 13:32:13 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MsFCy2BMkz4fYqq
	for ; Tue, 18 Oct 2022 13:32:46 +0000 (UTC)
	(envelope-from prvs=283b39a2d=roger.pau@citrix.com)
Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4MsFCx0fk7z3Cxs
	for ; Tue, 18 Oct 2022 13:32:44 +0000 (UTC)
	(envelope-from prvs=283b39a2d=roger.pau@citrix.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=citrix.com; s=securemail; t=1666099965;
  h=date:from:to:cc:subject:message-id:references:
   content-transfer-encoding:in-reply-to:mime-version;
  bh=mPQVn3MbUcON7xDa8JX2YzBqLLO0FGBh6l+Lubv8i8E=;
  b=eVMQhbUhWxdQUmvaPvfMf4yXU4TEFuMDG22Bx9uslKMlq4ozQVqaLLml
   +s7ZWUIULjM78v1xbNot0UcRuGzYZf0D6YaoVj2TsFXmU9dk+5IT374xc
   i/Hiim4mLsXJrGTTIvmG5VPIEK3JxCnEQC6yaivx8zdGLNXl/lfjtF6u6
   0=;
X-IronPort-RemoteIP: 104.47.55.102
X-IronPort-MID: 82972687
X-IronPort-Reputation: None
X-IronPort-Listener: OutboundMail
X-IronPort-SenderGroup: RELAY_O365
X-IronPort-MailFlowPolicy: $RELAYED
IronPort-Data: A9a23:mCLT7q5hTRuj4Vz7zWRM5QxRtC3GchMFZxGqfqrLsTDasY5as4F+v
 mceD2iOM/aMZTGhKI1wO9nj9BgB75HSmN5iTgZvpSoyHi5G8cbLO4+Ufxz6V8+wwm8vb2o8t
 plDNYOQRCwQZiWBzvt4GuG59RGQ7YnRGvymTras1hlZHWdMUD0mhQ9oh9k3i4tphcnRKw6Ws
 Jb5rta31GWNglaYCUpJrfPewP9TlK6q4mlB5gVuPakjUGL2zBH5MrpOfcldEFOgKmVkNrbSb
 /rOyri/4lTY838FYj9yuu+mGqGiaue60Tmm0hK6aYD76vRxjnVaPpIAHOgdcS9qZwChxLid/
 jnvWauYEm/FNoWU8AgUvoIx/ytWZcWq85efSZSzXFD6I+QrvBIAzt03ZHzaM7H09c5oCGdg0
 dA+MgtKaxWS3uer5bOkQbdj05FLwMnDZOvzu1lG5BSAVbMDfsqGRK/Ho9hFwD03m8ZCW+7EY
 NYUYiZuaxKGZABTPlAQC9Q1m+LAanvXKmUE7g7K4/dqpTGLlmSd05C0WDbRUsaNSshP2F6Ru
 0rN/njjAwFcP9uaodaA2iL217CVw3OkMG4UPIae3/FL0UGK+ndNVAJHCQW6hNCWp0HrDrqzL
 GRRoELCt5Ma91SqQ936RQ+PpGWYlgIQS5xbFKsn62mlxLDZ7gyCCi0UQzhpZsYrucUtSXotz
 FDhoj/yLTlmsbnQTG3H8L6R9Gu2IXJMdTVEYjIYRwwY5dWluJs0kh/EUtdkFuiyk8HxHjbzh
 TuNqUDSmokusCLC7I3jlXivvt5mjsKhotIdjukPYl+Y0w==
IronPort-HdrOrdr: A9a23:yR4mDKCSZhqAHzXlHeg3sceALOsnbusQ8zAXPh9KJCC9I/bzqy
 nxpp8mPH/P5wr5lktQ/OxoHJPwOU80kqQFmrX5XI3SJTUO3VHFEGgM1+vfKlHbak7DH6tmpN
 1dmstFeaLN5DpB/KHHCWCDer5PoeVvsprY49s2p00dMT2CAJsQizuRZDzrcHGfE2J9dOcE/d
 enl7x6jgvlXU5SQtWwB3EDUeSGj9rXlKj+aRpDIxI88gGBgR6h9ba/SnGjr18jegIK5Y1n3X
 nOkgT/6Knmm/anyiXE32uWy5hNgtPuxvZKGcTJoMkILTfHjBquee1aKvS/lQFwhNvqxEchkd
 HKrRtlF8Nv60nJdmXwmhfp0xmI6kda11bSjXujxVfzq83wQzw3T+Bbg5hCTxff4008+Plhza
 NixQuixtZqJCKFuB64y8nDVhlsmEbxi2Eli/Qvg3tWVpZbQKNNrLYY4FheHP47bW/HAbgcYa
 dT5fznlbdrmQvwVQGYgoAv+q3nYp0LJGbIfqBY0fblkAS/nxhCvjklLYIk7zU9HakGOul5Dt
 T/Q9pVfY51P74rhIJGdZM8qJiMexvwaCOJFl6uCnLaM4xCE07xivfMkcYIDaeRCdc18Kc=
X-IronPort-AV: E=Sophos;i="5.95,193,1661832000"; 
   d="scan'208";a="82972687"
Received: from mail-mw2nam10lp2102.outbound.protection.outlook.com (HELO NAM10-MW2-obe.outbound.protection.outlook.com) ([104.47.55.102])
  by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 18 Oct 2022 09:32:21 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=lu8hwq4GS4qSczUXdcF3w9457Bpf0kDF0uSXHaBcmEaT4YoC4HbgSZ3u9bVyIqbHhE+YXzD0okPXYSbgIBY9sLi2TfCgA31MsPfu3X24yZxSOzLU61mYji5ibKrt81ifOUvP2p5afScsI/JRa8adzRSApotIMB1l7ex42TSDJf5vQky0y6l6V5nfaehlhTtQ+NMXV7+6U0Ry8qe4Y5tmNs54E9RX280l3a45kyexBlJn2Ga3IdZUdUOR9l9fKHl/le97v7Nw+l+9NVuK9TJ2F0iHU5ShcYp4dRtw3qJGsXo9MfVwKXprL+fsO8Um/wJSmC94KZnB1MZ2Kn4LTs1mUQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=UdXjnJNimgO0sU3zE/dr8FZuUaX2VmJdtfLKrrIIVAM=;
 b=ReegZv8ddWo6mMd+SeAz3IWslpwRn3BowAboYB8ozsp6Gl/NFwv3EWX+75OYWIn8mHi7bhVpnixsQtXXf3bAO6+TvT48wqEo0wqxvZo4HC8bM/+mQtmQenUlMk2sesMNmhTheJjuYhopYQOoWm8gUkQjziF0cD4fjjvpFwRyfFmkNV8KdYHNMhObUp6AiB3sN8BJiQ7pLLtD+sGVGoSv9aOg1WbxzHwMsbp4/Z0tx1gfNr3bGEpIWCbt2WCZr6yewAaN2JJZt3MtyKm+iQLDhAFAR1BsBWE6FFiMhLYk7j20RmZTowWCuzgnuk1Ltlm9ospPJnj8p7qUHTDQHIhRog==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com;
 dkim=pass header.d=citrix.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=UdXjnJNimgO0sU3zE/dr8FZuUaX2VmJdtfLKrrIIVAM=;
 b=H3QlnHQoaZuARFUa4LVVZypoflqbvQMc/LfZYTdd64CzDWYvYJxl0QWLBvr2OWJp3Wz5RemRVhVLonxN7rW+pRnWBNCX4Ln9PBPdOQs4Dv+rswjugL7n16ZC6cJCTM2c9oXZ1kgcUDAGYlogz8Jkvl1G6qrHUNK6fDQ5vLdP6yU=
Received: from PH0PR03MB6368.namprd03.prod.outlook.com (2603:10b6:510:aa::21)
 by MN2PR03MB5103.namprd03.prod.outlook.com (2603:10b6:208:1aa::11) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.34; Tue, 18 Oct
 2022 13:32:19 +0000
Received: from PH0PR03MB6368.namprd03.prod.outlook.com
 ([fe80::3b75:f07d:1914:3e7c]) by PH0PR03MB6368.namprd03.prod.outlook.com
 ([fe80::3b75:f07d:1914:3e7c%7]) with mapi id 15.20.5723.033; Tue, 18 Oct 2022
 13:32:19 +0000
Date: Tue, 18 Oct 2022 15:32:13 +0200
From: Roger Pau =?utf-8?B?TW9ubsOp?= 
To: Karl Pielorz 
Cc: freebsd-xen@freebsd.org
Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of
 FreeBSD DomU lock ups?
Message-ID: 
References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
 
 <91CF514F28FADFC45E157B96@[10.12.30.106]>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <91CF514F28FADFC45E157B96@[10.12.30.106]>
X-ClientProxiedBy: LO4P123CA0447.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:1a9::20) To PH0PR03MB6368.namprd03.prod.outlook.com
 (2603:10b6:510:aa::21)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: PH0PR03MB6368:EE_|MN2PR03MB5103:EE_
X-MS-Office365-Filtering-Correlation-Id: 6fdd2259-c611-4e89-eefa-08dab10d2dd9
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	g4ztZB25UfAl0mRO1QmW/xn2Q9H25dAtr9qpBgwCFIIDR7pLZbuCkIT0X6LlSBmHHz20k0fGmigK90BTBZqmqNfPHZZGT0B5hva4pvgSknoPkC///WIUO4fsKqlpsxYb/NMWM02Qgpy8YaKHk7z+X0NptJu9f20GvEtRIytxl3sVXC9CDn0gjiib20hARrPc/JosZK2dX7kRpmIRin/X31doqf+wIf68vCsBMCDm11XO2tQtF2rD8KoMWxg7g3w7eCxisWEMDB8ZUI+uRe3NUnELigax4H7cEIRmPOJ0LU2jsI9Dkti5j5y6X2wDBcRetqEjPzCLAh/SJ6BLvBsZsOneLNxkqnUDAtsw3WJ4LY0JHDcYb1bwuSVD6uftF32e2Dv2wX/hmTks3KmLynxNPliGWLdMska5lMBQ0bC+unVsRYXD8KmCbrgpHaSyoGT7H6IK9vLChiVdx19YWzEOllSKRvwSLybSTKm46lHH1PAym9dQSoqOdiFycJNeP0rLnoho5RdFJYmF8K5Xb7/jx8Osa6jxEqjTVVcg6H4SUFcOBCCq9lgBge7yMFiPfpf0hrdkie+wOMr+rdGNGvWmq4fo8arcA107K/110qPYkFeKGSamtDRhcPLZ2DKN6S0/0AxQoaYUthYPTc/nqTVzR1Z+YHa0hePCVcMGIPd9aUGZTe1dth6UINKSAQjOPcoLp8whS/0n5w13Dhv70zLGo2WD1UdgviSDFBofPZKhZIM=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR03MB6368.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(7916004)(136003)(39860400002)(366004)(376002)(346002)(396003)(451199015)(8936002)(5660300002)(2906002)(8676002)(4326008)(66476007)(66556008)(38100700002)(41300700001)(85182001)(86362001)(82960400001)(6512007)(9686003)(26005)(6506007)(6916009)(186003)(6486002)(966005)(6666004)(478600001)(66946007)(33716001)(316002)(83380400001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?Sy9CcllVMGZhNnhISU84TlZyNlNzWmc4bG9iR29ZZlk3RVo2Zjhoc3g0T0po?=
 =?utf-8?B?T3k4MmpENnBNN0VWN29LRjZPY2M1KzRCQldSZW9SSldKRjRFaXJBcjVxOWRz?=
 =?utf-8?B?NkNhN0NCVitpTEk0YlV5S1VPZlNFNDcydE42Y1FGejlURVZFNnIxWHJCTXR5?=
 =?utf-8?B?OXgzQmtRSnBiL09mbkd1d29TdTRnR2QwQ1dQSldjd1B1RGhmTVZKOGNMRlIv?=
 =?utf-8?B?WXFDMnhkMVdXc3loS0RzNUhFSnNzbDZhZThjdGNZRlE5UHpSRG5kbHRpQVJP?=
 =?utf-8?B?M2kzSnRLQ3Z2T0tVMk1OK2VNSE15RzhqQUlyOHhmNjlubmJGRThWKzB0NG4y?=
 =?utf-8?B?SlJQU3I1T0RXam92MFFCV2RHRkpCSlhMbTN0bXk1LzJlbSt1UXlIVmRqMGk1?=
 =?utf-8?B?MkExaXc4Tk11SUFiTlh2VjhoK2NPTlU5VlpWcW5KaDVhZ0Q2cUxCOVZBUXQ3?=
 =?utf-8?B?ME41Q2FDeUdkc200TXVHUHJoVHhsOEV5aFBjK050YllzQ0NwZEJxRXlTV01w?=
 =?utf-8?B?M3dqUU9oaHJTaEZtNnJyMUNsQ29NUUJrVnNKSWxZaE5RbkhBUFYyWmJBNThh?=
 =?utf-8?B?ZWg5QlVxUDc2ZHM1Q2V3SHR2bGRvajlYSU9GY2UremFnY2VpdHRqQ3djb3VB?=
 =?utf-8?B?NmU2UWhHV0FKdnpDZUJ2TnozNVdIVVJFVGNvR3dUa1Iwcjc5K2RrcDJLK3Jo?=
 =?utf-8?B?aXdOTUgyc2JKRVBaM2JqSVhSd24rSkRDd2ZEa21vM0lpWUpKTDBzZHNjR2Q5?=
 =?utf-8?B?cFE5WXRaT0JhOUVIR1pxWkFYZ1BsSW93eGl2am01OEljdXgvSU9odjFUNkpm?=
 =?utf-8?B?elQ1aXdobFZXZWpGOUdnZ3lqdWg5K3VKbFdGT3BBdzY5dHZqVDdicDVkNkY5?=
 =?utf-8?B?SkF1LzdUdHlycmM0UDdWbFI2dG5ia2FQYWVxVWpTMzNFa1ZqV0ZRMzJZSWp1?=
 =?utf-8?B?WUtvdittTEtMZ081MGxTTHFjMEI1THp5ZXlmZ29QeEgvUmI2SnFITnVZNDcy?=
 =?utf-8?B?NFBnSmVPSEkwaUdha2p6NW5IKzRRSGFocmx6a1dGdmF3SnRuSEpTZTNFUzI4?=
 =?utf-8?B?NEZFaXJER0xMVHJsTUNFZUlsTEtnbTJINElDU2xIaVFIWFNxUFI5KzhIZG5L?=
 =?utf-8?B?OThEMXJGOTBYWVY4QnNkMW10cmRzTTlQUWNlZGZhRVZlT1J6UDVMMWkvb1Va?=
 =?utf-8?B?VjduRUtWd2N3ZG1lT0NIUmsvWDhTMkdZUDBVUTE3U3diK2NLd0tMRkQxMzBr?=
 =?utf-8?B?MU5aVitacDBJdHZPc3ZUSTBLNndvNDk4dnphNk44L29OeW9IczFPRUNFQW9C?=
 =?utf-8?B?eWJ1MGFEU29QZ29HaFlleVFOcUo0ZVA0Njg3UlN5d1ZONzQvQURKL1ZjcEY3?=
 =?utf-8?B?UlU4djMrNmozUExKWHdBeTlJVWFER1ArdzZLQU9SRmdqOFFSQy9vZ2tNbUxI?=
 =?utf-8?B?cHQ2K3FCaVlUUkdySHVoRXdla24wT0t0bmlCbFBnWjlyZzNTcU5EbnhhaG1X?=
 =?utf-8?B?ZDRsbk9jOUN1ZElOY3A3RmF6OEU4TVNUSzFmV1FQbGFUNHdUUlBtSWdGVEg4?=
 =?utf-8?B?Ym9wb2FXc1FTZWI3M0c1RXBoTS81VkJ4b2gyenh0TXdQUWRDMFdZcFFDM01v?=
 =?utf-8?B?eE9MMmNuQjdZQWtkVFJlaUhXUHk1KzYxSmdKQTR4VHRCbkgzRWZCK2ZSekU1?=
 =?utf-8?B?MldhcWdFWlNsM1pCdHZETHRlbUlSZGRESEEvNFVxdm1HWWhlb3d0YUFkT2Np?=
 =?utf-8?B?ako3Q2hHUWhHKytQaVlTRXl6RTZWMWloenlXNUVCMSt0TDEvRHE4R1FoOEJj?=
 =?utf-8?B?QWxyYURSMXNzaFpYNElJQ0xZSXpwRDVRV2NoWjdsTzB1TnU4bHk2TUtYYVd4?=
 =?utf-8?B?VlZDcGU5cjFUU21tY0tzTUFvTmtPckwzZVMvVFVjZi9rNEl3SFIrbGFpZFFj?=
 =?utf-8?B?V3RqRzQ5eDgyWUQyUm81SU5PV0VuS3loZmFDUk9HYzhMd0ZPVlZRcDhTdHpy?=
 =?utf-8?B?bFdOZCtPQkk2MUpndlJ0TCtaZnczbHhoNThHbEsvdkY3SnZFbnlwK29ORmRR?=
 =?utf-8?B?emVNMit4TGJrTC9XOHpZM3lGVGprZmJrSmVXdUptUGRsQUVqOW9OcUdXb1Y0?=
 =?utf-8?Q?C2Y+FU8ZJiTOvvAumI6TqlLlu?=
X-OriginatorOrg: citrix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6fdd2259-c611-4e89-eefa-08dab10d2dd9
X-MS-Exchange-CrossTenant-AuthSource: PH0PR03MB6368.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2022 13:32:19.1263
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hwLADfaJSh9xgfUo3gH/Z2LTAS9dnqDDTjOKhCRKQnsaqdGwfM/ijmVemY7F+FatxuzwKQ9LqyBUzIp4mj90SA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR03MB5103
X-Rspamd-Queue-Id: 4MsFCx0fk7z3Cxs
X-Spamd-Bar: -----
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=citrix.com header.s=securemail header.b=eVMQhbUh;
	dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=H3QlnHQo;
	arc=pass ("microsoft.com:s=arcselector9901:i=1");
	dmarc=pass (policy=reject) header.from=citrix.com;
	spf=pass (mx1.freebsd.org: domain of "prvs=283b39a2d=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=283b39a2d=roger.pau@citrix.com"
X-Spamd-Result: default: False [-5.16 / 15.00];
	DWL_DNSWL_LOW(-1.00)[citrix.com:dkim];
	ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-0.999];
	NEURAL_HAM_SHORT(-0.96)[-0.958];
	DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject];
	RCVD_DKIM_ARC_DNSWL_MED(-0.50)[];
	MID_RHS_NOT_FQDN(0.50)[];
	R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com];
	RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from];
	R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com];
	MIME_GOOD(-0.10)[text/plain];
	FROM_HAS_DN(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FORGED_SENDER_VERP_SRS(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=283b39a2d=roger.pau@citrix.com];
	ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US];
	RCPT_COUNT_TWO(0.00)[2];
	RCVD_COUNT_THREE(0.00)[4];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+];
	MIME_TRACE(0.00)[0:+];
	RCVD_TLS_LAST(0.00)[];
	TO_DN_SOME(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N

On Tue, Oct 18, 2022 at 12:15:28PM +0100, Karl Pielorz wrote:
> 
> 
> --On 14 October 2022 16:40 +0200 Roger Pau Monné 
> wrote:
> 
> > Hello,
> > 
> > Sorry, been very busy this week and forgot to reply earlier.
> > 
> > Could you try to setup a watchdog in FreeBSD and see if that
> > triggers?  So that we can get an idea of where the guest locks up.
> 
> Hi - no problem / thanks for the reply...
> 
> I'll give the above ago - part of the problem is not knowing which VM is
> going to die (there are quite a few) - the second part, is the waiting
> game...
> 
> > Is also the 100% load on all CPUs, or just one?
> 
> > From memory of the graphs - I think it was probably just one (I think
> > the
> last VM that locked was a two core VM).
> 
> > If the watchdog doesn't work we can try other methods.
> 
> Well, I'll get back to you when it happens again (personally - I hope it
> doesn't happen again) - but at least I know it's not quite as much of a dead
> end as I feared debug wise.
> 
> If this does happen again - and I'm able, is there any point in doing a
> snapshot + memory of the VM? (which is about the only thing I could think of
> - not knowing about the watchdog stuff).

You could try to get a snapshot, albeit I'm not sure if that will work
correctly if the VM is wedged.

Since it might not be feasible to setup the watchdog on all VMs my
recommendation would be to try sending an NMI to the stuck processor,
and see if we can get a trace that way, see:

https://lists.freebsd.org/archives/freebsd-xen/2022-October/000125.html

That seems to get me a trace when used on a non-locked up VM, so it's
worth a try.

Thanks, Roger.

From nobody Wed Oct 19 20:49:35 2022
X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mt2tp0b7Tz4fQnl
	for ; Wed, 19 Oct 2022 20:50:42 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "*.krpservers.com", Issuer "RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Mt2tn0Z75z3G56
	for ; Wed, 19 Oct 2022 20:50:40 +0000 (UTC)
	(envelope-from kpielorz_lst@tdx.co.uk)
Received: from [10.12.30.106] 
	by smtp.krpservers.com (8.16.1/8.15.2) with ESMTPSA id 29JKo251049877
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Wed, 19 Oct 2022 21:50:05 +0100 (BST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tdx.co.uk;
	s=krpdkim; t=1666212605;
	bh=NE8KL9e/CUgnm2Lccwr0WRlvHHe2cG5am3OSLQWjrbw=;
	h=Date:From:To:Subject;
	b=jzOa54rjLFpcqJm3yLr+uVRv0+pAncZdPGFFNJ/dS8HMDJGFyDe/u4adbaWRy6kY2
	 Low+ZdBaVdyxCy7lADVMC8OGExZDqNwxbNOFfU6jZpsmrd6W4OK3pidzqxol/LrZ8G
	 hlAM7bU1J8PwqJf8E3KOteRR3slX7h1gkVkVjyjVMGhSkBHQmYPsAGsk3Bu5h0C734
	 LkEQt8D+5lqCYxfRVr44yViq5JIADy9oZj2BpyDdHnkv38K4iyc9no6rxM8z/pdb1h
	 Qnpml13KBUVos+Umf/K5mSGKwZPLQJqeqUj97KFsy1aKs/UNdGP36ab78UVwS24JIN
	 07yagibfkItGg==
Date: Wed, 19 Oct 2022 21:49:35 +0100
From: Karl Pielorz 
To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= 
cc: freebsd-xen@freebsd.org
Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of
 FreeBSD DomU lock ups?
Message-ID: 
In-Reply-To: 
References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]>
  <91CF514F28FADFC45E157B96@[10.12.30.106]>
 
X-Mailer: Mulberry/4.0.8 (Win32)
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
X-Rspamd-Queue-Id: 4Mt2tn0Z75z3G56
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=tdx.co.uk header.s=krpdkim header.b=jzOa54rj;
	dmarc=pass (policy=none) header.from=tdx.co.uk;
	spf=pass (mx1.freebsd.org: domain of kpielorz_lst@tdx.co.uk designates 62.13.128.145 as permitted sender) smtp.mailfrom=kpielorz_lst@tdx.co.uk
X-Spamd-Result: default: False [-2.43 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	NEURAL_HAM_MEDIUM(-1.00)[-0.998];
	NEURAL_HAM_LONG(-0.93)[-0.928];
	MID_RHS_IP_LITERAL(0.50)[];
	DMARC_POLICY_ALLOW(-0.50)[tdx.co.uk,none];
	R_DKIM_ALLOW(-0.20)[tdx.co.uk:s=krpdkim];
	R_SPF_ALLOW(-0.20)[+a:smtp.krpservers.com];
	MIME_GOOD(-0.10)[text/plain];
	ASN(0.00)[asn:60969, ipnet:62.13.128.0/24, country:GB];
	MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org];
	MIME_TRACE(0.00)[0:+];
	FROM_EQ_ENVFROM(0.00)[];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	DKIM_TRACE(0.00)[tdx.co.uk:+];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	TO_DN_SOME(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	RCVD_TLS_ALL(0.00)[]
X-ThisMailContainsUnwantedMimeParts: N



--On 18 October 2022 15:32 +0200 Roger Pau Monn=C3=A9 =
=20
wrote:

> Since it might not be feasible to setup the watchdog on all VMs my
> recommendation would be to try sending an NMI to the stuck processor,
> and see if we can get a trace that way, see:
>
> https://lists.freebsd.org/archives/freebsd-xen/2022-October/000125.html
>
> That seems to get me a trace when used on a non-locked up VM, so it's
> worth a try.

Ok, I'll give that a go... No guarantee of when it'll happen next - but=20
thanks again, for at least giving me something to do / try :)

-Karl



From nobody Fri Nov 11 20:55:53 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N89wD4kCPz4g2cH
	for ; Fri, 11 Nov 2022 20:55:56 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4N89wD3JG9z4Wy8
	for ; Fri, 11 Nov 2022 20:55:56 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1668200156;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=Mv16ayEg8yepUqjm2qIuenrVMD/RPeUayBcxh8SRNoM=;
	b=b8uOLTwTROS7pfDV1dpSEAs6PrKtFMoXQDgiZSePr2RMk/Us5i8+3CHhuginABzNYmd5pC
	4Vxir8YNhcRH5GwrAByfwe8j/rK84VmOu9FVbm3SIS99EMXS/1scj7TeipfxNSAbCIXGXH
	XXPK1EhewWAeGh442m0khgvbduel+NiMxWF3LdeY7lof4nRVlyMGWPRYHMuIR9ext+yGYc
	b6tXQtA437jBqc8AeaBm0B474I+8qoAGNELMVEX3PkUavB3UmWbJx/j+9MABpNOnlFao8g
	HJfANsxorOCQtzWQu0nIxyWsaNhpYwNJcEAxNA0pqOukufdc0OZUxkfaYVDR5A==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1668200156; a=rsa-sha256; cv=none;
	b=qtDnkahXOCHlkP/Atl0kZlqtgDB8ya/OyHJz13ssI8W5Rwb9S31KTKN+YgHwFU06xVg/zg
	egO5z+jxgydFGitswn6mSi3GgwIyUVEfnP0C6BF2VKmImjtxpbhY76kF1zAXvi839S5G8Y
	NY/7TYmfpO3fRtLkWGeWhKuhp7A4/Vblo38AFmc08yVP1hpdU4yWsR3op+xjfiWikK0Zg8
	c/TCPlCj6D7aSwZtrRM0Xd/tHok2LJ3Qo/2tMwlcLEjBa2g9D9xWzGFusrxsEwwR1+KMgz
	dbozUAGssvD+GGkJLfeJIbxcs6o1B1UIuBrUIz7/xFgm7Gag4Up3reuvz7Qwhw==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4N89wD2NXnzxVq
	for ; Fri, 11 Nov 2022 20:55:56 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2ABKtuCg047193
	for ; Fri, 11 Nov 2022 20:55:56 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2ABKtuvn047192
	for xen@FreeBSD.org; Fri, 11 Nov 2022 20:55:56 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 188261] [xen] FreeBSD DomU PVHVM guests cannot 'route' traffic
 for other Xen PV guests on same Dom0 Host.
Date: Fri, 11 Nov 2022 20:55:53 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 9.2-STABLE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: koobs@FreeBSD.org
X-Bugzilla-Status: Open
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: Normal
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: see_also
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D188261

Kubilay Kocak  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.freebsd.org/bu
                   |                            |gzilla/show_bug.cgi?id=3D1=
650
                   |                            |59

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Fri Nov 11 20:55:53 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N89wH0kVqz4g2Qd
	for ; Fri, 11 Nov 2022 20:55:59 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4N89wG53bZz4XK7
	for ; Fri, 11 Nov 2022 20:55:58 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1668200158;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=ad77YRQ980zvk3GDvQZR/zQqpTyNrdxtzX7iZzcoOAY=;
	b=uv55m+l12GATvzf8dKiSiO5M6/ru1DL28CmjWktqGjEnJPlMq/o6beq7QLIwTAESk1Oatd
	If2ILuMQIzdOsBucYWSLZbK3M2pekVbfm9xCB+kfzDINiXv2aO7+INdMXHsXNpN+wjIxYA
	2SYdWFps0SLQl73NgY8hTP37iveKe50Qju9agSCXq4L6WDWsahrxIt+DDL590rmZEMoOou
	3ces3GXwzEZwXlFD3gHnf9y8930PzmIPwbkDSi9D2GTz8QAI+uNfTxyt73q3QiHyfK5h+2
	jeHvAKa3xpAE9HWZnYUj/J+zMrg5WTakhXSCh0xsu73XUz8pC6BVAlTjWF/g2w==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1668200158; a=rsa-sha256; cv=none;
	b=RQZDSb4rjdyjFShcWujc31jW2KNm4lP8+/wDzbGXejttVr9XVzyrrQAJOf6H7QFBTIT0KL
	f4H/oaPJIVs96RQr3dSFYMMkerrsRP8RCoANAjyVrt5ctLftXd8VdAJSmYOCnRzgvuVauR
	RquVxzSXOCthYz5/wSATvJUVXgOE4HnJoupYcZWyXQgOajQm3G3FKxDH9BPwaOEEnKzKPG
	LMmy26oZNtjZ6SE2Po/Ajp+GRA278roEctwmtOddQqQGmMIJcApotWH26t0a15sCniw48H
	t8jR56M1TnoRBdmUC7XohUomI2ELjVXt34kh4XMhHIIbCrqw0Kc87hXl1tkigg==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4N89wG49fdzxVt
	for ; Fri, 11 Nov 2022 20:55:58 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2ABKtwgg047248
	for ; Fri, 11 Nov 2022 20:55:58 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2ABKtwCl047246
	for xen@FreeBSD.org; Fri, 11 Nov 2022 20:55:58 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 202199] packet forwarding in xen becomes slow
Date: Fri, 11 Nov 2022 20:55:53 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 10.1-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: koobs@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: see_also
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D202199

Kubilay Kocak  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.freebsd.org/bu
                   |                            |gzilla/show_bug.cgi?id=3D1=
650
                   |                            |59

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From nobody Fri Nov 11 20:55:53 2022
X-Original-To: xen@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N89wJ0NVrz4g2cy
	for ; Fri, 11 Nov 2022 20:56:00 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4N89wH4w8Sz4XC7
	for ; Fri, 11 Nov 2022 20:55:59 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1668200159;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=h/DKJcNT8liV8HXkt4Q9d3b/7Ka7wqB5meNM76Z9nSI=;
	b=Ofid7XWDCNFuHMSdtUl6vmTaPh1Rtps1+Cre56t3KjhCs3YVTcSiyg+hYzGIlraWyB2gvF
	FV2MHiFnlM9SO+OM+cIZCiB78XJwtpVygvuw/5o93gJroQIh+pXJZSjMI63sfG9GwVOdO5
	6yGcGYtK1A2YHhgeZgQrt9JVdN1iEyqfg6hK0vYM/6gMhan6znyKz5r+NuLeOQl3zuQThb
	mJFMfZ3vOMgfS869m+QHMheG2oYAg5+FglckFm1Uf8h1C2VyGOGBhoeEJ/eQmLvGoQeupO
	GT2aoOGDCjewZGt2jUGqiyxx5qJWHadK4eg5b8851MTaQLSohqUMjw9PzQr+QQ==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1668200159; a=rsa-sha256; cv=none;
	b=shZNGYEKWmbgBXteZa3UvNigG5LkrzYSK86TZud/o+Nw9xijppotYRcHQ23aNCpzZLEEGY
	vYbkmhpFuH9ItW0qgiyEosj47Zd+FtGeqS3XjiEbpvlwwmbHV/NDccQnL3mxmU79mzZc2z
	sRdk/iqVpATj3ffD5k0nTJVdYY027asSZ3AQ9o/z3IBngsiDXMflwjmlAtMkhFovVOpoRx
	KV10FqkWruBi0yBNbkX877M2V3WC0z+YNmOLq5Bh0PK4O1X+mjZIwn6VeXyIybj85WJuiJ
	qd5IWESULtSUOGv8f4gEyZfobceHDSvrFnvMpdHAPtV3JHu2Q1YRhXbD6XaALA==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4N89wH42DzzxYk
	for ; Fri, 11 Nov 2022 20:55:59 +0000 (UTC)
	(envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
	by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2ABKtxN8047276
	for ; Fri, 11 Nov 2022 20:55:59 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
	by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2ABKtxKs047275
	for xen@FreeBSD.org; Fri, 11 Nov 2022 20:55:59 GMT
	(envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: xen@FreeBSD.org
Subject: [Bug 197344] tx checksum broken on XEN
Date: Fri, 11 Nov 2022 20:55:53 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 10.1-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: koobs@FreeBSD.org
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: xen@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: see_also
Message-ID: 
In-Reply-To: 
References: 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
List-Id: Discussion 
List-Archive: https://lists.freebsd.org/archives/freebsd-xen
List-Help: 
List-Post: 
List-Subscribe: 
List-Unsubscribe: 
Sender: owner-freebsd-xen@freebsd.org
X-BeenThere: freebsd-xen@freebsd.org
MIME-Version: 1.0
X-ThisMailContainsUnwantedMimeParts: N

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D197344

Kubilay Kocak  changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.freebsd.org/bu
                   |                            |gzilla/show_bug.cgi?id=3D1=
650
                   |                            |59

--=20
You are receiving this mail because:
You are the assignee for the bug.=