From owner-freebsd-xen@freebsd.org Tue Mar 16 11:56:47 2021 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 824E45A9DF0 for ; Tue, 16 Mar 2021 11:56:47 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID SSL CA G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0BbL375Wz4S9X for ; Tue, 16 Mar 2021 11:56:46 +0000 (UTC) (envelope-from roger.pau@citrix.com) IronPort-SDR: WQmSEtYJjTenuZnjXHvBfyQdfZDoO+WPMpdbghcLIpCbAr2P4jftNZAQTHRdKFSHmQsrhfHs39 76UexnZLGYL3ZLaJ0i0qkLNtxB/sA8zJ3zEFBKUVTp6TzmX79sUh1by/c98GR+u7LaCHzHmjDB a2/N6LcbkalErl2jGbbqxK8FrX+R3M3Lj3mWdl1YsjpeHNN+7kiA+F5+Yk0p8gyWOM+nH2LmKf mLCevTq1BlOBp4a6A9uqJHPjwIzG8LBZACXt84P/ARr1g6auPeohVyam5jn0SVw1WrN2EMqz7G GZI= X-SBRS: 5.2 X-MesageID: 39276960 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-HdrOrdr: A9a23:CWZWfK7LrWKNVblMBQPXwTWEI+orLtY04lQ7vn1ZYSd+NuSFis Gjm+ka3xfoiDAXHEotg8yEJbPoexzh3LZPy800Ma25VAfr/FGpIoZr8Jf4z1TbdxHW3tV2kZ 1te60WMrDNJHBnkMf35xS5Gd48wN+BtJuln/va0m0Fd2BXQotLhj0JbTqzOEtwWQVAGN4VFI CE4NBGujqnfh0sH76GL1MCWPXOoMCOqYL+bXc9dmUawSStrRft0r71FBCEwgwTOgkg/Z4O+X XI+jaJhJmLnOq8znbnpgruxrRQ3ODs095SQPGL4/JlVwnEriaNSMBfV6aZvDYzydvfmWoCtN XXuR8vM4BSxhrqDxiIiCDg0QXhzzoigkWKoTTz4QqB0L/EbQk3BMZbiYVSfgGx0TtZgPhG3L 9WxGXcjpJLDHr77VrAzuLVXBJnnFfcmwtFrccviRVkIO8jQY4Uh4ke8ERJKYwHDSL35as2ed Mec/301bJ4d0iXYGveuXQq6NuwXm4rFhPDeUQavNeJugIm1kxR/g89/ogyj30A/JUyR91t4P nFCL1hkPVrQtUNZaxwKe8dSaKMeyHwaCOJFFjXDUXsFakBNX6IgYXw+q8J6Oajf4FN5Icumb zaOWko+1IaSgbLM4mjzZdL+hfCTCGWRjL20PxT4JB/p/nVWKfrCyueU1oj+vHQ7Mk3M4n+Yb KeKZhWC/jsIS/FAoBSxTDzXJFUND0wS8sQltEnW0+fg87CJ4Hw39arMMr7Ff7IK3IJS2n/Cn wMUHzYP8Nb9H2mXXf+nVzwVhrWCw/C1KM1NJKf0/kYyYALOIEJmBMSk06F6saCLiAHlqQ3eU B5MY72i6/Tnxj1wU/4q0FSfjZNBEdc57vtF1lQoxURDk/yebEf//KSZH5Vx3nCAhNkVcvZHE p+qj1MiO6KBq3V4RpnJ8OsM2qcgXdWjmmNVY0glqqK4tqgXIg5AJYgUKlYDh7KCBRxpAZvpA 54GU45b36aMgmrpbSujZQSCu2aXcJ7mh2XLcldrm+aiV+Rvvg1RnwQXyenVOmehQpGfUsQun RBt4skxJaQkzemLmUyxMA1KkdFZmisDLVaNwidf4lPlrf3eAZ/cHeSiVWh+mEOU1uv039Xqn 3qLCWSd/2OOFZbt3xC+ovB8V9/dAymDgtNQ0E/lbc4OXXNu3513+POW7G61HGJbEAehssHNi veXDcUKgRy5ty+2RKPggyeHXE+yphGBJ2YMJ0TN5XonlWjJ42DmftYQ7t6/JN5OMvvteFOe+ SFYAORJC75DeRs+wH9nAdRBABE7F0f1dXv01nZyUL9+lgVK//bOk5nSLEWOMv01RmTe9+4lL FCyesotu6xOFjrYtGIyavrfydOQymj01KefqUNk9Ropqo8u7t4IonDXRbJ3H9B2g8iLM2crj JrfI1LpJTAMJRoZcocZmZw+Ucojs2GKCIQw0TLK957WVEmlHnAOdyVp5LOtLo0G0WE4C/9I0 OW/SEY3/DLWUK4pPUnIpN1BWRdc04n7nt+uMuEao3LEQ2vM9h5w2DSCA7LTJZtDI6fGbsRqR 5m49aH28+vHhCIpDz4jH9cOaJB82GuXMWoJhmDcNQ4tOCHBQ== X-IronPort-AV: E=Sophos;i="5.81,251,1610427600"; d="scan'208";a="39276960" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FdNJNWDz04tE52I9R3EbRXaWbzxOVR1SbaCBojcgNjyYZkLZWMxO/ruVArAdH7z1i4BxFynUERI5xvbKSHq7VrTs+WuM9yXmVGSWGl5zawgiMNOWxhRCarUNWM3NYLKaCRXm9OGzXkny3V97oxcFGI6n/BCdbvkO64N5uGiUGaXjDTog+hXVt5QXT7lohJIPD/jpEcGTtRYvCj9aenHI8dC9sDz64+jhWYOgxlxem52zcv/pwuWdxw+abdbVaxGuSJclmjMjYbWVJgsodsimgeNulq9CCuW/1+SIeBhXyU1fHp0kH6ZeMdSjnSGMGKXbiQFGcV6NTOZp5pOWGCCEWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mzA0Cl0RkWeP/gUp7V2Pz3yRoU2HIjN+bfxEabGDWeQ=; b=jENj8jFc2DsGKVSTMRxdYIKPVJJNwM2fMDfk1K3NYYuUI6CPkiydVj1ytcBr7GFOURfsghCKknQHAJO8enLkIU4kkkUrVmWx5gaX2QXv4WUMBxeyXWRuJmKoSRcF7J9QIhiv+nY5cKYn68Eozx/2qY5nN06gBSYFnQoUGYWDBiWycVuw4GGK3zX/iG6or31VQHxevgq/YZ3KcqSlyoUXN0xyTjz4jI4t8bd9+TgOpWXLbQsXC/RQzsaD2QaiMwL7b3XfB3iS6dNrl+sR/OFz898IYRQTH+gbP9hlyr9dV0ypbJI7Qn3vy1SMF8GjY6rNRpxK8Id1mlF5CE9PJk/6+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none Date: Tue, 16 Mar 2021 12:56:33 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Brian Buhrow CC: Subject: Re: Corruption in xenstored tdb file? Message-ID: References: <202103152124.12FLOeeg004469@nfbcal.org> <202103160706.12G76WLQ014062@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <202103160706.12G76WLQ014062@nfbcal.org> X-ClientProxiedBy: MR2P264CA0109.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:33::25) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9ca1b186-8231-4c97-40b1-08d8e8728fd9 X-MS-TrafficTypeDiagnostic: DM5PR03MB3292: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8GUelkzOsm+NbYkofztN0IHLyzaoj+nH7RHptTJ7G1vcj20LQC+hOJNhWrO4QO06r4Qoj3lwLFqpamfK34huw80a3dWgDszYi+G/gDx05KOwpNiNb7Lgo2QO1PhwVyJMb69BFwL1l+0Ep6Selswej7GM6lncuCPRF60kYNShbjxN2K4VgAHHhV1p67NfOIhBEgwyYOBocbqkoHquEQBdcpEytONtSI+aRbfbCn8XvoFen/xDWYguLnzJHRCvYkzrxKMY3+lIhJmjz3+ifiEd2VwqqBimjRAxUktWDuJ5IiQ3JxN3ZSpRCmgBIcpqxfXSeneV3ex1zlIRnw9isODVRabETBg4j/enOIoTEr5Bkc5zhfm0yWgMYTQNl2heboa3723ihHfXr7qPWJ9ec0VhvWgnwmsVUqHHV1rHXJYdmG4aRXIr6Jm8img+RYERs1Tai+6rvDjCWRMAvLezw6cbwb7P/ANtHUUvPDVNNzNmBkQDBUivae62oEPHAjYqllfhiW7T4wSfb7jkWJmnMmSpwA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS7PR03MB5608.namprd03.prod.outlook.com; PTR:; CAT:NONE; SFS:(7916004)(4636009)(396003)(136003)(346002)(366004)(39860400002)(376002)(6486002)(316002)(6496006)(9686003)(85182001)(186003)(478600001)(33716001)(956004)(2906002)(86362001)(5660300002)(6916009)(6666004)(26005)(8676002)(83380400001)(66476007)(66556008)(66946007)(4326008)(8936002)(16526019); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?SHBtK1F6U2RXd3d4clU5dHBnUUNLM0Z2Nk1FQTdCTVBra1lJTmhsU0lodmZX?= =?utf-8?B?WkZPR0RsQjlHZU51VVF6M0hacWVaajM0T1JxdVZMQVpRNytNZ0dVd1Q5TkpT?= =?utf-8?B?U2dJMWpZTldnSmdhNWtudVVSYTZUS0J5WkVLbEhnRUdMU1hKQTVzMWtFYVlk?= =?utf-8?B?b2p0L2JrMFdjeENwbW45NXBES1NhZ295SDNvY1dScFZKY1gyR1pwSjlOSEhT?= =?utf-8?B?OFpZeHlKYWhzS1FNT01WbmJOVHlSZVQ3a0VoaTFIMXREenZnTmNWd3RxRDVn?= =?utf-8?B?R2hyeU1NK2trRy8wQkloYUQrcFpxY0JpRDRiek5GNTBxV0E2SURLL2k4MnJT?= =?utf-8?B?dW1UL0RqaGNkK09NL1BocWJoTDRGQnhTZnhCYXQ3ekxscWtnZm80MmZwNVl5?= =?utf-8?B?Qk91ZkRVNjlDMCt2SGRHZldNZGhnZlY1K25rODNoakxOcWZpWmUyMEJLM3JL?= =?utf-8?B?UjF1VDFiSGRudWtMdkNHQmh6SlMyZkcyNCt1b1VsbWNZWmQreExLOE5EQWsv?= =?utf-8?B?SmUzNUtId2JMK3YzdzdIWDRpMkhYR2F6TWd4aEo1VG5JSzdBdS9TYlovVVBz?= =?utf-8?B?RDBUQ05JSk5DbEFKUTZVbi9lMkllQzh0YlI2aXRNdnhYQUhxK0VDQlpXbU1J?= =?utf-8?B?ZFNXMGtFYklCY0ZVWElPeXN5VlJ1ZUZaanh2WlJHMEdzQ1BlTW1TS3UwKzND?= =?utf-8?B?T2kzUUJCbHRZRDFTM0RmRHF2UW1GTW5ndTJqVjVmQ0hkTU9WWmliSWs5Y0tz?= =?utf-8?B?YWJYWWhJbEZ0TEtEQ1d4T01BODZvdDJUWVNoZVNQSGo4QlBTTkNCYXZ3UEdl?= =?utf-8?B?azRIcWg2akNvMUxkTlBMZVhjcUVkU3VxWEh4OC9aOFZUVS9lTmdaVndzUjhI?= =?utf-8?B?NHlkTGlxOGZJQ2RpcHZlRWh0Q25rWng0Nm1hSzd2UWs3eXRIR3dnbDhIOFZ2?= =?utf-8?B?OGM5UGp5WG1raHdJcm5CYzVKV1lPZG1qWHRnQTBwZDFRWVZXQmVrYTZHZ2Zh?= =?utf-8?B?QTRLK2ZDUmVocGlkSkxDZVlEOG4wa2tJd3Q5WkJUL1BIYk9XNVJpUGxPTWNI?= =?utf-8?B?eWcwQTc1OTVUc2p5UWp2dGo5b3FXRkdwTE1uaVZ3Q2xBRFZLN0lCWm1PRTU1?= =?utf-8?B?Z1M4RGNETUpzaUJFQUtYL2l0elFieGkrQzFsdWNPbEVid0IzL21SL1l1YlN6?= =?utf-8?B?cTZKS1hybm9zNzJZZWdtWFgyNS9lNlpMdC8yNXI2Y1QwNWlWOE5WK1ZOR1Ey?= =?utf-8?B?R3FTYXJneFB6QzNjSTlVVFdHeUhPU09kcnlFWVVrWFpUbllMZHFKaWMwdlg2?= =?utf-8?B?QThBTUM3ckVCUTVzdy9KSXNPSGV4WTZuTU02K3B0M0xhako3bmMwdGpoRzNK?= =?utf-8?B?YWpjc2tvSEx4Z0pYakg1dU9PRWMwcmhsNjVBdUxKai96Tks3dWRyOEUxdkxP?= =?utf-8?B?SHR1dE5HLzZURmFINVlSNmt5Nk9hWm16R1k1UmFWQ1dHaDRYV3A2WmoveDEx?= =?utf-8?B?SVczaWxUaVRRL2g0RnZNbzJyMi9ra3JzVnVtUmsvSGxucTgyL3k4emc5NXRN?= =?utf-8?B?Nlh6ajV0NTRicGYrOVFHYitkUFBvMzU1UTZmd2dOTkphUWt1RmxsY0ozWC8z?= =?utf-8?B?VUxHYzJHQ0xiUnRtaTNpcXlBdFJBRzZTd003dWM4U1hqQmNxdk41ZHlQWlQz?= =?utf-8?B?Mkt4SkprZGlnZjBQQ2grb281VU1uNTkySWUvalNNK01tQkRGODJGdHBGYW5S?= =?utf-8?Q?cww6pfiQLZ9EILEF6H3dZsBOIsEEs5lRHODedST?= X-MS-Exchange-CrossTenant-Network-Message-Id: 9ca1b186-8231-4c97-40b1-08d8e8728fd9 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2021 11:56:41.3088 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cEE/Xr0PBhdyhK/Tya5BUkn8sl1YyzUveMPvipoHeg/dnN1LBhsU/TXMGzBCuhjWaIpeSVjW0BEz2OV76J8MAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB3292 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4F0BbL375Wz4S9X X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.64 / 15.00]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[216.71.155.168:from]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-xen]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 11:56:47 -0000 On Tue, Mar 16, 2021 at 12:06:32AM -0700, Brian Buhrow wrote: > hello. Following up on this further, it seems there may be a timing issue related to this > after all. If I bring up a NetBSD-5.2 VM, the VM comes up without a problem and xennet0 works > just as it should. I can do this time and time again without any trouble. > However, if I bring up a NetBSD-99.77 (current as of January 28 2021), I get the behavior I > described in the previous message. It's obviously some kind of race condition, since if I > reboot the NetBSD-current VM several times, I can get it to come up with a network interface > occasionally. However, not enough to make it usable. > Also, since I wrote last, I updated to 12.2-release--p4, just to see if that made things > better. It did not. I suspect, but don't know for sure, that the issue is that NetBSD-current > is issuing commands on the xenbus faster than it did in NetBSD-5. If that's true, then I think > the problem lies with FreeBSD, as, in my view, a VM guest shouldn't be able to trigger a race > condition in the host side of the server, which is what appears to be happening here. > Is there a way to get a trace of the communications between the domU's and the dom0 so I > can see the differences between what NetBSD used to do and what it does today? So I've taken a look and it seems NetBSD now switches to the XenbusStateInitialised state without having written some of the configuration data required by netback. It's not clear to me whether this is a bug in NetBSD, or a bug in FreeBSD netback. In any case the patch below should fix it, can you apply it to your kernel sources, recompile and test? The above fix changes the behavior of FreeBSD netback to only try to fetch the data when the frontend switches to the Connected state, this seems to be inline with what Linux netback does, so in any case I think it's a change worth making. Thanks, Roger. ---8<--- diff --git a/sys/dev/xen/netback/netback.c b/sys/dev/xen/netback/netback.c index 44159f60d996..29efd76430c7 100644 --- a/sys/dev/xen/netback/netback.c +++ b/sys/dev/xen/netback/netback.c @@ -1392,8 +1392,8 @@ xnb_frontend_changed(device_t dev, XenbusState frontend_state) switch (frontend_state) { case XenbusStateInitialising: - break; case XenbusStateInitialised: + break; case XenbusStateConnected: xnb_connect(xnb); break;