From owner-freebsd-xen@freebsd.org Tue Nov 17 09:21:16 2020 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9A33A2D5F09 for ; Tue, 17 Nov 2020 09:21:16 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cb0mp69byz3tkn for ; Tue, 17 Nov 2020 09:21:14 +0000 (UTC) (envelope-from roger.pau@citrix.com) IronPort-SDR: pEeNhIHZAsZJqxQDhrhSpu25RZbhBB+9z1WJdn6cz7BS7AU5kAYyLMIp2miSguF8aiBqvN351n XGV7H3NJK4lrQoJfh8f3K9FoxEdNRFk9zeTRne+Ha13XC/PoqmzceiG7EPZBGJ8SIXrJ0+Q7IC vDGLOVcWA918abOhrO0vcz9oplkzKm7ERrz9a9yoFg65fTXJ7ul9B+MJFZcn0P0a0nFCn1Civi KO6bh6LY+tUlujR0sMIc2GTntdIlAdx+Q4lGM0qP2DTMiLmF8gJiXn8KQp24CBHL5BFX2/klRp bjE= X-SBRS: None X-MesageID: 32456694 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.77,485,1596513600"; d="scan'208";a="32456694" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YXJgW42AgAmjl2leHlVeEbYr5n8o6Sod6Xe6FIDvuyB6vi3GxSzgwhoskdP3rJb99HOQXonxU5MEC9s+TCUpzSo/kx3GEyt7wDH5vtGWJJoW3IvbwAOrkYOX6sec+06MdLqQYFVrrYaTk75YiJAkErvUn6R2NSeloZCtUrG6gwa7uGTFaV62Q9YwcTlOkJsYshTPPS5oun9m8OCqme24uXSgXGOe0d7/36YYdaenMKgn8xbNQbJbMWXKBAD8qURuJRb0HXT6/rGuRSQ88WWf2krbIpZnRai+sz0zDSM1Sa2vb9AOLMhT+sw1WgsH11wHuID4+Qkxjhxe1WB8RUVBiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/gslbG8md5SpLub0LyqoxWz12rftVvyktD73oJm9OHs=; b=ViUglvVujHTOYrD4TEiWgN2k6AT6HGmkjW65gN/o+yCtoA1Y2rgnYFQuDddPGp+CqrZxBg64oMuXs3x4SDi5iQyHkb7ufgcDeOpYt++1EmjKe3s2p3GpPD6MCIshAn+FPSaSZAYxe3FWT83hFiwvFKYalTaQj0KLjwTzJZV96Oube0Q5M+Gk2NwKEJJguIzcYrg3LNhhqQ6Yn56ovHf1fyvcMLf3gnlNqXsq5tyXJMqaIuYjRKbxKw8XpUnVGLYeXMvFt/hdmO0Vd5MbP6zTyzgFaeR1s/bQdhl+479ys/kOduByXHeDkEXCu53GE0ohLf7ZA3MV2h+i9XvbinVyoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none Date: Tue, 17 Nov 2020 10:21:04 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Brian Buhrow CC: Subject: Re: ZFS corruption using zvols as backingstore for hvm VM's Message-ID: <20201117092104.usulqtvcvyv7mcji@Air-de-Roger> References: <20201111095009.6lcik5y3s7wrsh5k@Air-de-Roger> <202011130518.0AD5IxtF013960@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <202011130518.0AD5IxtF013960@nfbcal.org> X-ClientProxiedBy: LO2P265CA0446.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:e::26) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 75267baf-1b09-4a42-ddee-08d88ada1df2 X-MS-TrafficTypeDiagnostic: DM5PR03MB2922: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: krv3Mm4fiSeJiI67hNmS2AwR0U6XhomH4RELexT1VmY2jfFB9chMeiMrRgj/moo9J53+KcPUIXKK9hsgg2NdjpsbeJ2SVAMAl+k7M0fK2QqEwkb1XdKcExNB1GMKnAgYU47DD2wnc79L64bUEj/MMYsYk81fxuHyGr2jeNwiWVOzqFxss/cv5zy8uJMAngukjVjdbqsCeXjYDbdkDUTQ6DP+WPCQZP+1TCFMDKLbb7/15JStWZGdIXAXQWv1wFgRw2Nkv3xpBXj5bNsgmh0Zddn44dauk4bqiq0VXPabbV/LaJUD2Yc2mWs67Xpj4UgrobRb25cIqVRtqV2kTI0Ciclym8KiQDANoP9nHNoennVEZB4XDAXxQW3+D0rlsCeGnX9UTqoQ+qtUWNZPX1BuSA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS7PR03MB5608.namprd03.prod.outlook.com; PTR:; CAT:NONE; SFS:(7916004)(4636009)(376002)(39860400002)(136003)(366004)(396003)(346002)(8676002)(6916009)(33716001)(8936002)(956004)(16526019)(85182001)(4326008)(6666004)(66556008)(66946007)(5660300002)(186003)(316002)(86362001)(2906002)(1076003)(66476007)(83380400001)(26005)(478600001)(9686003)(966005)(6496006)(6486002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: ABMos6djtMoQM2HGF7s2E95K/ZMwk2BRX3KvnJqscDb2EIo1l1uO28L4jJaDWjwnbIZIIOJ935UzidcMJaZdj8pvs7MKP0Dny+z0uPf055Oh8PieDj9CQHBqmSpaNrHaaF75ooRNpeBwbUtacbEhgkUL0BRKB7TfoYxsCUKWIgZc6nyVkCb0TxHBlle+IBmztT3lZz90BA2/9hJ8e5UiKGUPju2tMcHUyRAtfNl/tFQFh3RQDwfPTxeUdy2Fsl/WCfY+ctTPhrksezrtBiFSHcf3t0DyWvfPyueSG2Qo13esg2BhU0WRgnJsFRQaGt+FlEtg/ulFKswXJrdP3SbdJpHUAM6C1j5HA6ZemKF4se95yR8XQ8SjjwaSMrU2l2KtUO87pSTKYjy8DZN8xMV70L29G0OFt3PQd8e8Hb3hYae6B3sIxECtxQBHYkL7TV3PNGq9BSZj8PfxyavwubRQ+L/2jlV/OO74C+jXp3XJ6/IGO0txlxBfBYnYBisKPPv2ZHX9/dyTHca65OnE0F/Yr8PgM4+HAvdJ+cLatFKj2Vzgur3con6wX2qQtQc6xoThZ76W+W5KMyZwFxDg41vh0ZwqdgKE5c9rhYV38tqztbnPmztpIiriyl4b+pH0imXga33meQzP0dDMHG88Ve2kMEK/Puf9u5BK76mIVKhUCjYzNmvPgfYg91iDwUmMAXnG50Wc3x6bYJJVUQXTa6Yva4G1/LiHBZo+ybFMzZPI6qRfBhS0XjsziC4atjiRu4CGIl7uN844p6dGkeM9pnbqYXsmPAlSpSjCcHxIMhEzt1wfZ7HaxbCP7DCEcvkELFQvgmdr5WpgoBNgJt4M+4fb4cG9UkfRCgpMGo3b+3pJIozfMWyg4cwO/8RkV7jpHuvBs0vQc5RIjJH8211WyBO/ow== X-MS-Exchange-CrossTenant-Network-Message-Id: 75267baf-1b09-4a42-ddee-08d88ada1df2 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2020 09:21:08.4607 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cycnSLAih7uzrJYgP+3525FcP3tVjPCyc+htBVi3HyML+ax8PduXFC7YKUI2SuUNXQEQp4A4vwhkbAP8nwhNmg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB2922 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4Cb0mp69byz3tkn X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.20 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RWL_MAILSPIKE_VERYGOOD(0.00)[216.71.155.144:from]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.144.spf.hc3370-68.iphmx.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.144:from]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-xen] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2020 09:21:16 -0000 On Thu, Nov 12, 2020 at 09:18:59PM -0800, Brian Buhrow wrote: > hello Roger. thanks for engaging with me on this issue. I think > I've made progress on the issue and have a better handle on what's going > wrong. There seem to be a cascade of bugs here, which I'll try to > enumerate. > > 1. The disk corruption issue seems to be a bug in qemu whereby the > emulated IDE disk controller issues partial writes instead of full writes > or no writes with appropriate failure to the disk. The IDE driver in > NetBSD-5.2 doesn't play well with this behavior, in fact, NetBSD until May > of 2020, doesn't play well with this behavior > See: > http://mail-index.NetBSD.org/source-changes/2020/05/24/msg117668.html Oh great, so it's something specific to NetBSD. This means you are running NetBSD in HVM mode? > 2. This causes memory corruption in the OS itself, which can trigger a xen > server crash! (In my view, no matter how badly behaved the guest OS is, it > shouldn't be able to bring down the xen server.) Right, we really need the trace from this crash. Do you have a serial hooked up to the box so that you can provide the panic message? > 3. Running NetBSD-5.2/i386 as a domu, which works flawlessly under xen3, gets a > panic: HYPERVISOR_mmu_update failed This would imply that you are running NetBSD in PV mode, in which case it won't be using the emulated hard disk drive, and hence the commit you referenced above would be unrelated. Can you assert whether you are running NetBSD in PV or HVM mode? > I suspect this can be worked around using some of the command line > options under Xen, xpti=true,domu=false, perhaps? > Are there others I should consider? Maybe. I think you should report this to xen-devel and NetBSD/Xen mailing lists, with a full trace of the crash and the guest config file. Roger.