From owner-freebsd-xen@freebsd.org Fri Oct 19 16:40:10 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77B69FCFDCB for ; Fri, 19 Oct 2018 16:40:10 +0000 (UTC) (envelope-from Brian.McCarthy@netapp.com) Received: from NAM05-BY2-obe.outbound.protection.outlook.com (mail-eopbgr710081.outbound.protection.outlook.com [40.107.71.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D116183D5E for ; Fri, 19 Oct 2018 16:40:09 +0000 (UTC) (envelope-from Brian.McCarthy@netapp.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.onmicrosoft.com; s=selector1-netapp-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J7JPU2dYvYmYCXYxJYV7arIuW4LyBcknBdDkh6FmLOA=; b=MUUsh+q4u2LMdpmBlcvYkzHvTBF/HR5cJFlz9phzIcA4AfKqC7frrltSqRqGNPJ4LSDdOto4WOWNb+GI4NA2uLYF2TbqZQ5eGrMURxC2xgGJz4FhK/g0pSrnYwX4iJD1pu08CpFCuezOsfJktvN2dx82n7jbbZo7B+jU1thBUCc= Received: from BL0PR06MB4323.namprd06.prod.outlook.com (10.167.181.204) by BL0PR06MB4369.namprd06.prod.outlook.com (10.167.181.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1250.20; Fri, 19 Oct 2018 16:40:08 +0000 Received: from BL0PR06MB4323.namprd06.prod.outlook.com ([fe80::d9b6:1e26:2f92:2e8f]) by BL0PR06MB4323.namprd06.prod.outlook.com ([fe80::d9b6:1e26:2f92:2e8f%4]) with mapi id 15.20.1228.020; Fri, 19 Oct 2018 16:40:07 +0000 From: "McCarthy, Brian" To: "freebsd-xen@freebsd.org" , "McCarthy, Brian" Subject: Bug in XEN support with IPIs? Thread-Topic: Bug in XEN support with IPIs? Thread-Index: AdRnyaMmzZu57y2aSQ6r82Acwm4uMw== Date: Fri, 19 Oct 2018 16:40:07 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-dg-ref: PG1ldGE+PGF0IG5tPSJib2R5Lmh0bWwiIHA9ImM6XHVzZXJzXG1icmlhblxhcHBkYXRhXHJvYW1pbmdcMDlkODQ5YjYtMzJkMy00YTQwLTg1ZWUtNmI4NGJhMjllMzViXG1zZ3NcbXNnLTlkZjY2OGFhLWQzYmQtMTFlOC1hZGIwLTVjYzVkNDYwY2JmMFxhbWUtdGVzdFw5ZGY2NjhhYy1kM2JkLTExZTgtYWRiMC01Y2M1ZDQ2MGNiZjBib2R5Lmh0bWwiIHN6PSIyMzE5IiB0PSIxMzE4NDQ0MDgwNjA5NjQ0NzEiIGg9Imw1L0YyT3ZEeFRtc0pTQUI1YmpudnNCVVNIaz0iIGlkPSIiIGJsPSIwIiBibz0iMSIvPjwvbWV0YT4= authentication-results: spf=none (sender IP is ) smtp.mailfrom=Brian.McCarthy@netapp.com; x-originating-ip: [2601:191:8500:8acd:d558:e4d8:a7d4:bf1a] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BL0PR06MB4369; 6:0B8ksCQ1SzATcDHn+AI0ProU1gWg6Swt7WNA6zzMZHZLAm+Z+dH57hS0+lvOokquuYTJDX54RJuiYhKUSpGI9ZpWvrJig+7XkRr1JZai6GBXFD1nKt5zIup1aTNVNiItKFIOtpinBa4EZcLtnxD+ONTy42102h67chja6/c/435e5h3MOJOTfkmoJThnxqf1aZsPHNlgFklR7jtTjd4OOCFLwTz0yRK80Pc5IFcCIiKc6Iq/YpfBSRRzHVK5EVlnmDFzRnAa1oCAOPEcoG0pAb2mhM35Ti12zGtPuse5vXGTGWm4FemPrxQJ57kKNd/w6gI+bLYof5PSvagdN5PG6J2zrikpeVNn7XMmp0EaJd2aKTJ6tebjUdSYMqDFH5SF5WfynDrrMcZWs0OiFJ6cEhJiBT1+tvPd39iTWEHfjRXuEGbfB1KwRohdWGU3sCEKzV6OW820D78PobkJgpj05A==; 5:xa2Bw0cREVLpoTH5IFmqrT/7Udgh082rPt2E9USo+cLcBEAsy5230lOnqTCMuCyJVbalVmtzXOjZam7KkJ8jgSkx1+o7NBTqAArPK0EBpaOm4Iak21qI6VEjc9FFDqU19aP1v3expNrjD0Ps5zPRI3nj8REFlsXTjKTW6iXXyIU=; 7:yB38bM0dWs2SWiBCoR0ZBW/5/yTqhEhrUqRBKsfmvEPC+hX+rROrLLyaBw4+BoyTS8FgMrJDObzIHrclOs+A1pY6B+84OqB5KZxbfUDWFbG+Wb1HV2UwWaFe5twmRwa+4PoFB7K5Vv+1Gf+zoAFr6zRIJKTKnnukle3TZ2oDh89DzYrSNo6ho8T+/6aZZKyHnugqkdPd+w5jRSTJRCXFWMQlH7G4v1Ct9sD0u717EDTndh8VXo5fN/FCc5KLFy3g x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10009020)(346002)(366004)(396003)(376002)(39860400002)(136003)(199004)(189003)(6116002)(316002)(105586002)(478600001)(25786009)(790700001)(6636002)(99286004)(106356001)(486006)(476003)(74316002)(33656002)(2906002)(72206003)(19609705001)(55016002)(68736007)(86362001)(5660300001)(2900100001)(46003)(110136005)(6306002)(71200400001)(6506007)(8936002)(2501003)(7696005)(71190400001)(256004)(102836004)(97736004)(14454004)(5250100002)(81156014)(6436002)(53936002)(186003)(81166006)(8676002)(9686003)(54896002)(7736002); DIR:OUT; SFP:1101; SCL:1; SRVR:BL0PR06MB4369; H:BL0PR06MB4323.namprd06.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; x-ms-office365-filtering-correlation-id: ba9f92c8-3edc-47a7-0753-08d635e18776 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7193020); SRVR:BL0PR06MB4369; x-ms-traffictypediagnostic: BL0PR06MB4369: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(21748063052155)(28532068793085)(190501279198761)(227612066756510); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231355)(944501410)(52105095)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123560045)(20161123562045)(20161123564045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:BL0PR06MB4369; BCL:0; PCL:0; RULEID:; SRVR:BL0PR06MB4369; x-forefront-prvs: 0830866D19 received-spf: None (protection.outlook.com: netapp.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: Fw2ff/pB3REb/l9cGXCODF5uo4BBXvw1sREo7zJa0yjLuAH+ABVeCVL9KZnmBj5QyZOBleacI+LS+gB5ZqDGd61qCclQQfVN3CgH/PXyqnGipYyRvztYg4E1vPbvQKkYIXj4/yG93njyWeVRTeLf8X2sd0oT/6GKVC5Yi7eRK28U6lE4C0X0mQ81mRL0Bgce4LGAigChQ6fzYzLyHExs8AdlGWg00VOJW61nYKfIzFm6p7Mf4y9T9kodkZtk4tSFUITn6ekAG1/Kw3N67dSscYzx6EsJXtjfawWEogCMEpSRz4LEqbnEQryqgkONlnreeSAXcZQ16X8fcWMbUZtJxpS6q4RS9kiwoBmcq9ez+pM= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba9f92c8-3edc-47a7-0753-08d635e18776 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Oct 2018 16:40:07.5716 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR06MB4369 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2018 16:40:10 -0000 it appears, that for FreeBSD XEN guests with SMP support, The IPIs get mess= ed up by intr_shuffle_irqs. The symptom is a smp_ipi_mtx timeout during TLB shootdown. I've seen this mentioned in e-mails, with the= solution (without explanation) being to enable EARLY_AP_STARTUP. The reason this works in BSD11 is that int_shuffle_irqs is conditionalixed = out on EARLY_AP_STARTUP. The real solution seems to be that xi_cpu and/or ie_cpu should be set. I'm trying to figure out where that wou= ld be (for our somewhat modified FreeBSD kernel). -Brian McCarthy, NetApp From owner-freebsd-xen@freebsd.org Fri Oct 19 18:34:45 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE7FCFD8572 for ; Fri, 19 Oct 2018 18:34:44 +0000 (UTC) (envelope-from 010001668d9ba04f-4e7c54d1-4b98-46b8-b2bb-4095fd574057-000000@amazonses.com) Received: from a8-176.smtp-out.amazonses.com (a8-176.smtp-out.amazonses.com [54.240.8.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6331E88C0D for ; Fri, 19 Oct 2018 18:34:44 +0000 (UTC) (envelope-from 010001668d9ba04f-4e7c54d1-4b98-46b8-b2bb-4095fd574057-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=dqtolf56kk3wpt62c3jnwboqvr7iedax; d=tarsnap.com; t=1539974078; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=1OB4jT1SdqpDzPTSj6Mp9oVc9v72DMKerJRcrzkM/cQ=; b=QKmey5qN6m6K9ATi9R0n4PFGMoy3H/YQGjUPoa6XJKX8oW9BrJvVy1DF7myTUa9l z34/B6+LUiX1YOUkfojW8OAyhAw0FmA9vMf0KxJb/UPu6qcZJ0iDh652qCIgeiFe9QF 5FWSkep60fb3fbYFdsX2TjD5uw4b31SqomYHXH3Q= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1539974078; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=1OB4jT1SdqpDzPTSj6Mp9oVc9v72DMKerJRcrzkM/cQ=; b=WoHWEDnce8DH1wXy7zq/tCw+pp0etnpJ1Y/wc2PycONtGCFiok7DKCg9xbeCEH+U 6wV7kQaXUpYOVe9XAetbOvQpo9KZPvmblnszNjtdWspWTLZ2WnDLsuiXJLVCMCrFuaj lv0kPNkS2ktmsI4fKIm533CsH7tqo0PDk3BxAQ74= Subject: Re: Bug in XEN support with IPIs? To: "McCarthy, Brian" , "freebsd-xen@freebsd.org" References: From: Colin Percival Openpgp: preference=signencrypt Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= xsDhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgSzSVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+wmEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEzsFNBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFwkkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <010001668d9ba04f-4e7c54d1-4b98-46b8-b2bb-4095fd574057-000000@email.amazonses.com> Date: Fri, 19 Oct 2018 18:34:38 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2018.10.19-54.240.8.176 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2018 18:34:45 -0000 On 10/19/18 9:40 AM, McCarthy, Brian wrote: > it appears, that for FreeBSD XEN guests with SMP support, The IPIs get messed up by intr_shuffle_irqs. The symptom is a smp_ipi_mtx > timeout during TLB shootdown. I've seen this mentioned in e-mails, with the solution (without explanation) being to enable EARLY_AP_STARTUP. > The reason this works in BSD11 is that int_shuffle_irqs is conditionalixed out on EARLY_AP_STARTUP. The real solution seems to be that > xi_cpu and/or ie_cpu should be set. I'm trying to figure out where that would be (for our somewhat modified FreeBSD kernel). Which version are you looking at? We fixed a bug along these lines just about a month ago -- see r338624-338631. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid