From owner-freebsd-xen@freebsd.org Thu Nov 7 18:08:40 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E44541BB973 for ; Thu, 7 Nov 2019 18:08:40 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478BGw0D0yz43mf for ; Thu, 7 Nov 2019 18:08:39 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f169.google.com with SMTP id v8so3324547ljh.5 for ; Thu, 07 Nov 2019 10:08:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:openpgp:autocrypt:message-id :date:user-agent:mime-version:content-language :content-transfer-encoding; bh=Ra+KF8lOk80OYB4+A8BYHkaJ9eEQNledGmXzMWN72hg=; b=HblF3q8cFRLBpYr6f/s3ScIniu0pHqyLcTiM6yOpuxBWi9YNflB8lj3tRGY4TV1lRy 9gRxVnmK+WuiJC3DsViigRAS9erZ26vaiU0dkuyXilx0GaWItH7YCyUkYQqHciIrFNE9 56vmd7HboDJsmw/lmt4yIweAVyT/5kiJ//KepRpOXHQ2peIDmXGPMhwowQbX0miGNiTS 0Be7e7+ZBNHI0Sh99orHxMgBdSZ1U1L6yYqwIZij0LltPxD9+kly7rYX4izGMmA3/9au xoj368zYWBalPH/qZIxRJAlX1PvG74wC3Y9ehfTErFTLDYyyNpaaSG7YmgotXXeANIL6 yQqA== X-Gm-Message-State: APjAAAVwXeaAE0yMlv+Ny0K+RmldQasvgcrDlp8VqADHKRDW6hbpPN49 dFPCd4as+KMLapTBglNe8RxpcVsRebA= X-Google-Smtp-Source: APXvYqw13SfJCWaDUO3ocg9yNInlp1Rq82DsYjpOiAD8KMkmXKTrBiJb7QvNViXCaW3LE/arAwBFCQ== X-Received: by 2002:a2e:5451:: with SMTP id y17mr3325079ljd.98.1573150117844; Thu, 07 Nov 2019 10:08:37 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 3sm1300100lfq.55.2019.11.07.10.08.36 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Nov 2019 10:08:36 -0800 (PST) To: freebsd-xen@freebsd.org From: Andriy Gapon Subject: Xen (HVM) and NMI Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Thu, 7 Nov 2019 20:08:36 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 478BGw0D0yz43mf X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.15)[ip: (-0.46), ipnet: 209.85.128.0/17(-3.20), asn: 15169(-2.01), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[169.208.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[169.208.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2019 18:08:41 -0000 Does Xen provide any support for non-maskable interrupts interrupt delivery to a guest? I am trying to debug a particular crash in a Xen guest, but it seems that the FreeBSD crash code (stop_cpus specifically) cannot interrupt some processors and, thus, take control of them and collect their context. For CPUs that do get interrupted I see stack traces like: cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9 intr_event_handle+0x8b intr_execute_handlers+0x58 xen_intr_handle_upcall+0x15a xen_intr_upcall_u+0x96 ... So, it looks like the NMI is delivered by the same mechanism as normal interrupts. If a processor has interrupts disabled then the NMI would not get delivered? Is there anything we could do to improve this? Thank you! -- Andriy Gapon From owner-freebsd-xen@freebsd.org Fri Nov 8 08:04:02 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 764D71AAEDD for ; Fri, 8 Nov 2019 08:04:02 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478Xpn1JZVz4245 for ; Fri, 8 Nov 2019 08:04:00 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f46.google.com with SMTP id f24so705834lfh.10 for ; Fri, 08 Nov 2019 00:04:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=tTWueceQbgTASw/LThdinVv33gK6WCDlme6M8YaF4AY=; b=MFGeSpxNZQJXwgmBh50Nv/hM7mPoUq5Ho0Oucv1CV4OaqCTle831x5mOVRTZ42CTDS F0/oo+9Op9LG/K6lLAkbbhjsFTQGZq6mY9nvmns91uQdyiVqD0ojNGGY7I0B+x5T0D2g pV8e6Qhq7XURhm+DZfqZmdlTdMk48ouvEO2SQENf3uUuJKURpeV2DdB60k9ZP9/oLGhC JpoHNKX0ucvTM/YElbQhYISf9OYCfJlfIK8aWKCGPWolSeLm1kBFzyxjJxOvfuUHrz0x iTNfrIHZby+/k0y4amFYnr01c7Ib/WnrJBHDcAGctNioA6Jv4qf4cq1HkMZEzJXBZXMX Aphg== X-Gm-Message-State: APjAAAV7M90d2etfWRN4gHboILBuxBfC+lGl5qUocJlAx+7hOIlYnLVQ X1ppyDQsISyywuEYay/GNGNmgH/Amyg= X-Google-Smtp-Source: APXvYqwDOrnwGAgpFCij4FwFjX1QofREpd/yePh029AZprN0Twvq+cuIV5qasrvKjcZ7UqYyk3ZKVA== X-Received: by 2002:a19:5e0e:: with SMTP id s14mr5522067lfb.30.1573200238618; Fri, 08 Nov 2019 00:03:58 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id o5sm2451168lfn.78.2019.11.08.00.03.56 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Nov 2019 00:03:57 -0800 (PST) Subject: Re: Xen (HVM) and NMI From: Andriy Gapon To: freebsd-xen@freebsd.org References: Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> Date: Fri, 8 Nov 2019 10:03:55 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 478Xpn1JZVz4245 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.16 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.16)[ip: (-0.52), ipnet: 209.85.128.0/17(-3.20), asn: 15169(-2.01), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[46.167.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[46.167.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 08:04:02 -0000 On 07/11/2019 20:08, Andriy Gapon wrote: > For CPUs that do get interrupted I see stack traces like: > cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9 > intr_event_handle+0x8b intr_execute_handlers+0x58 xen_intr_handle_upcall+0x15a > xen_intr_upcall_u+0x96 ... > So, it looks like the NMI is delivered by the same mechanism as normal > interrupts. If a processor has interrupts disabled then the NMI would not get > delivered? > > Is there anything we could do to improve this? I found this in Linux code: HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); It's in xen_send_IPI_one(). I wonder if that's that or if there is more to this than meets the eye. Any help and advice is appreciated! -- Andriy Gapon From owner-freebsd-xen@freebsd.org Fri Nov 8 08:19:06 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 781FA1AB170 for ; Fri, 8 Nov 2019 08:19:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478Y893dyfz42VN for ; Fri, 8 Nov 2019 08:19:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f65.google.com with SMTP id f5so3775497lfp.1 for ; Fri, 08 Nov 2019 00:19:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=uqxSuXwTGNAWJYQ+slwitWyYvaypPDCZEKbo7uEYTdM=; b=Tom8XnC/d6l+6AF4S6GBPFuLiZQkWuciTGZ9bmXaYMt/9gdnqXk8A+fURMxBstfhzp IZGkSbjRXmGcoqgf5JoYeT6c0km9LXK8LcE0xQzJetjhSu+baoWF65uNUrQWlo8DEShu J/vHOTFJIQNIAfGkgliGWSkkMKtN1uJOykuGVMU4pR+swdKyQQbzDfR5nnyk4GiP641B fKekTdu3y/mfcmQeiPCl/jYRyyRFVJ0Snlu+IK600Wbzh/Xsie5ECu5DR1+uTH7bW94I AYl64Kutzr7MXcDvETG960LAIe0yQB3xmZWq8EQ1wuVePeCLwLwjFAta0tjzDTR6Ta3E RWxA== X-Gm-Message-State: APjAAAXMuF62fbgpUVeuKwLS3klVacWc2VU5kZISZkdSiBC84wiSOsMK SIqbr7QdTnaEfCRTP7EQKRXQ2BFIKEE= X-Google-Smtp-Source: APXvYqxwfQoywse/ELhFHZq5QCVmHdjjybK+SKv+QLgguG7eFxsLuf7xNbkfGU96pcKA4GhlSqbTPA== X-Received: by 2002:a19:1ce:: with SMTP id 197mr5727095lfb.16.1573201143543; Fri, 08 Nov 2019 00:19:03 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id y17sm2036131lfy.58.2019.11.08.00.19.02 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Nov 2019 00:19:02 -0800 (PST) Subject: Re: Xen (HVM) and NMI From: Andriy Gapon To: freebsd-xen@freebsd.org References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Fri, 8 Nov 2019 10:19:01 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 478Y893dyfz42VN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.65 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.11 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.11)[ip: (-0.30), ipnet: 209.85.128.0/17(-3.20), asn: 15169(-2.01), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[65.167.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[65.167.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 08:19:06 -0000 On 08/11/2019 10:03, Andriy Gapon wrote: > On 07/11/2019 20:08, Andriy Gapon wrote: >> For CPUs that do get interrupted I see stack traces like: >> cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9 >> intr_event_handle+0x8b intr_execute_handlers+0x58 xen_intr_handle_upcall+0x15a >> xen_intr_upcall_u+0x96 ... >> So, it looks like the NMI is delivered by the same mechanism as normal >> interrupts. If a processor has interrupts disabled then the NMI would not get >> delivered? >> >> Is there anything we could do to improve this? > > I found this in Linux code: > HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); > It's in xen_send_IPI_one(). > I wonder if that's that or if there is more to this than meets the eye. I also found this in an old post. Ian Campbell wrote: > You need to register a callback with CALLBACKOP_register > CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive > that callback. IIRC you also need to arrange that returning from an NMI > is always done with HYPERVISOR_iret and not optimised to a direct iret > as it can be otherwise. This is to allow the hypervisor to implement NMI > masking correctly. But I could not find a single use of CALLBACKTYPE_nmi in the Linux code (it's defined but not referenced). I did find this: hypercall_iret = hypercall_page + __HYPERVISOR_iret * 32 ENTRY(xen_iret) pushq $0 jmp hypercall_iret -- Andriy Gapon From owner-freebsd-xen@freebsd.org Fri Nov 8 09:58:03 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A87A1AD421 for ; Fri, 8 Nov 2019 09:58:03 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa2.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478bLM1jsmz480d; Fri, 8 Nov 2019 09:58:02 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1573207083; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=eWNaOUNbSLq1KV0tLVKLg/k29eTMRE2PS+T7kW82Kfw=; b=HmJtRgK4P9q/oljcF2kY4EHW9udXHYC7PYsNpajB4auDHWg5LwSlWPvk Lywk3mD0vN20MYIQuFemLMw1e2C9vA4snILqWS+VmWG8BOgQmJgovanml E1ATWEs/t+xPP/C6U/xBOOSvHjOLOoonJ5l1+tVEx/L6pS6vaaaRqRYrx 4=; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: 0Sv6LbfvmYX7pHomwrE6A7Gk+4+eq0JTe15Q+TdKe6l5oOrPNrdXn3m67NOSkCBqBhI1WF//fK 2ZhbgiPdAybizDchmgtdlS7Iu86F3tVLxPRc0txfhLc2yfhDBgqRzdwgwgu0G4OvqmNadDfvbV MUsdPiwrRhO9ykw+T50USQ0rr0s2GIuPHGHWBZEMeylTPM5tAZmic5QrAUo4I6ULpz0NmSfSwl 8hVdMC0JM1BQLvT9Zjx2grE0IzVGzo4cJCBSSWUpnUkbPXrNPf17BEPEYRt4SLZcIiD0f8IGbb jOI= X-SBRS: 2.7 X-MesageID: 8052046 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,281,1569297600"; d="scan'208";a="8052046" Date: Fri, 8 Nov 2019 10:57:54 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Andriy Gapon CC: Subject: Re: Xen (HVM) and NMI Message-ID: <20191108095754.GV14005@Air-de-Roger> References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL03.citrite.net (10.69.22.127) X-Rspamd-Queue-Id: 478bLM1jsmz480d X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 09:58:03 -0000 On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote: > On 08/11/2019 10:03, Andriy Gapon wrote: > > On 07/11/2019 20:08, Andriy Gapon wrote: > >> For CPUs that do get interrupted I see stack traces like: > >> cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9 > >> intr_event_handle+0x8b intr_execute_handlers+0x58 xen_intr_handle_upcall+0x15a > >> xen_intr_upcall_u+0x96 ... > >> So, it looks like the NMI is delivered by the same mechanism as normal > >> interrupts. If a processor has interrupts disabled then the NMI would not get > >> delivered? Yes, sorry, I certainly didn't code this correctly regarding NMI handling. > >> Is there anything we could do to improve this? > > > > I found this in Linux code: > > HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); > > It's in xen_send_IPI_one(). > > I wonder if that's that or if there is more to this than meets the eye. Yes, something along this lines should work, we could even use the native NMI signaling using the local APIC, but the hypercall shortcut should be faster in most cases. > I also found this in an old post. > Ian Campbell wrote: > > You need to register a callback with CALLBACKOP_register > > CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive > > that callback. IIRC you also need to arrange that returning from an NMI > > is always done with HYPERVISOR_iret and not optimised to a direct iret > > as it can be otherwise. This is to allow the hypervisor to implement NMI > > masking correctly. That's AFAIK for PV guests which use a completely different mechanism in order to receive interrupts. None of this is needed for FreeBSD because there's no classic PV support. Can you try the patch below? Roger. ---8<--- diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index 7d254ef3f734..7a6964d60cdb 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache; static driver_filter_t xen_ipi_bitmap_handler; static driver_filter_t xen_cpustop_handler; static driver_filter_t xen_cpususpend_handler; -static driver_filter_t xen_cpustophard_handler; #endif /*---------------------------------- Macros ----------------------------------*/ @@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] = [IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b" }, [IPI_TO_IDX(IPI_STOP)] = { xen_cpustop_handler, "st" }, [IPI_TO_IDX(IPI_SUSPEND)] = { xen_cpususpend_handler, "sp" }, - [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler, "sth" }, }; #endif @@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) XEN_APIC_UNSUPPORTED; } +#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) +static void +send_nmi(int dest) +{ + unsigned int cpu; + + /* + * NMIs are not routed over event channels, and instead delivered as on + * native using the exception vector (#2). Triggering them can be done + * using the local APIC, or an hypercall as a shortcut like it's done + * below. + */ + switch(dest) { + case APIC_IPI_DEST_SELF: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + break; + case APIC_IPI_DEST_ALL: + CPU_FOREACH(cpu) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + case APIC_IPI_DEST_OTHERS: + CPU_FOREACH(cpu) + if (cpu != PCPU_GET(cpuid)) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + default: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(dest, vcpu_id), + NULL); + break; + } +} +#undef PCPU_ID_GET + static void xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + if (vector >= IPI_NMI_FIRST) { + send_nmi(dest); + return; + } + ipi_idx = IPI_TO_IDX(vector); if (ipi_idx >= nitems(xen_ipis)) panic("IPI out of range"); @@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg) return (FILTER_HANDLED); } -static int -xen_cpustophard_handler(void *arg) -{ - - ipi_nmi_handler(); - return (FILTER_HANDLED); -} - /*----------------------------- XEN PV IPI setup -----------------------------*/ /* * Those functions are provided outside of the Xen PV APIC implementation From owner-freebsd-xen@freebsd.org Fri Nov 8 13:15:46 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 21B801B2192 for ; Fri, 8 Nov 2019 13:15:46 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478gkT2GRMz4LRl for ; Fri, 8 Nov 2019 13:15:45 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f65.google.com with SMTP id j14so4426459lfb.8 for ; Fri, 08 Nov 2019 05:15:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=x/XmkAzMcX+Uqv250KaJAjn0OBj5HINcVFq5b2EsX9o=; b=Qh6vffhpc9CcEspVh14ouv9WGM6gFq7s9lYajJb+/CX7mwBCO3YsvhunAKNTOujbve FMOEkqvdVcr+ylxtPpWzfpNEdUhbqMiI017Wxhxh7X4+lL4Q/bIgbHokY0a9sVdIBGFz J3CJEU/iJlv0O4ISyV5qZCx+Ese5HOU/Ne1Bx0M3/mnc4Tn3o7WOQCvqKRFqKqoTuUmB avc32isIqUZJQeeG44NqOcV00G1/NZqJguevFrclGGjrzLHnngvtcFGBd8Oc/lRzFgXf xN26Pa/vjPtT18XKOSX7DRYKM/Mosdz0xrXARy0mRmks6EJMCWN86ebUobPWSFWf7z8e HxbA== X-Gm-Message-State: APjAAAVWa7djJsBz4d471Sd+JBvLe58UE6bCRFvqdCX3IT0QCRgetVHo R1MAkFO7LLnRNZzna/tbJD5AjEEYgB0= X-Google-Smtp-Source: APXvYqywNBylDIxo8Bs41kUut4V9gKbgQ5VSTCzRTLJnGhqlFWejec73LCAAggTCgaG2W/MJTk0JYA== X-Received: by 2002:a19:6759:: with SMTP id e25mr6387736lfj.80.1573218943032; Fri, 08 Nov 2019 05:15:43 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id g7sm2572464lfb.4.2019.11.08.05.15.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Nov 2019 05:15:42 -0800 (PST) Subject: Re: Xen (HVM) and NMI To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@FreeBSD.org References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> <20191108095754.GV14005@Air-de-Roger> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <760f4fe2-8a5b-87a7-f84f-f250bb4eb31c@FreeBSD.org> Date: Fri, 8 Nov 2019 15:15:40 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20191108095754.GV14005@Air-de-Roger> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 478gkT2GRMz4LRl X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.65 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.11 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[65.167.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-1.11)[ip: (-0.28), ipnet: 209.85.128.0/17(-3.20), asn: 15169(-2.01), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[65.167.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 13:15:46 -0000 On 08/11/2019 11:57, Roger Pau Monné wrote: > On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote: >>> I found this in Linux code: >>> HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); >>> It's in xen_send_IPI_one(). >>> I wonder if that's that or if there is more to this than meets the eye. > > Yes, something along this lines should work, we could even use the > native NMI signaling using the local APIC, but the hypercall shortcut > should be faster in most cases. > >> I also found this in an old post. >> Ian Campbell wrote: >>> You need to register a callback with CALLBACKOP_register >>> CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive >>> that callback. IIRC you also need to arrange that returning from an NMI >>> is always done with HYPERVISOR_iret and not optimised to a direct iret >>> as it can be otherwise. This is to allow the hypervisor to implement NMI >>> masking correctly. > > That's AFAIK for PV guests which use a completely different mechanism > in order to receive interrupts. None of this is needed for FreeBSD > because there's no classic PV support. Perhaps HYPERVISOR_iret is still needed even for us? This is said with zero knowledge of the topic. > Can you try the patch below? I tried it and, unfortunately, it was not a success. And I don't have much diagnostic. My test was: - run a busy process, I used dd if=/dev/random of=/dev/null bs=16k - run procstat -kk on that process After this the test system locked up hard and I could not get anything out of it. It being an EC2 instance did not help either. Unfortunately, I do not have any local Xen deployment. -- Andriy Gapon From owner-freebsd-xen@freebsd.org Fri Nov 8 14:01:00 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1AA8D1B3386 for ; Fri, 8 Nov 2019 14:01:00 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478hkg5SK3z4P05; Fri, 8 Nov 2019 14:00:59 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1573221659; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=3UeKj+UIKN5+9af1AK2WbLLDuX/65FLNlUDYXTLjKRU=; b=QdpxKKVlUvlof5JwRmuK+jvsKBLS/MMSqw04+5n/nvcloAUT271Qnnxq KFMBFHO2ucw22kFKKSEX94NwgM0JleqOwSwieqDs6N2wBDuvdafW0a+bJ J2FqIwLj6NKfrRHPzgzQCOaxqpWL2+WP+I5DXjPThDQwZ/+RIu2gpIHOj U=; Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: OK8jDI/oaOgxpkylYPOrL9Roi8UAfYokLrTa3Wyz7XLZjKjMoHg3qxahEaMtDKlUUo6Z4tqLHu snFYlFriQTAuVDtJNlNHU057OSzohq38tYBfYs/iJQR6JIa+VgkUivXoUdCazbgQTHDB/AAFKX 5xunCr4/EqeTKTvYFNXSZWD6pz0www7UdOpGN8JHVOk7TxLkibBJDc7SVMD9GTF2CNQSWJuGZC Np8eU3f0BzjaGKFbsc8rAIdaFogf/Wmx6F3y4W9SVOPbBYthmdwma+MKqtLyW//8GJxz+0/6+x alo= X-SBRS: 2.7 X-MesageID: 8573630 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,281,1569297600"; d="scan'208";a="8573630" Date: Fri, 8 Nov 2019 15:00:49 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Andriy Gapon CC: Subject: Re: Xen (HVM) and NMI Message-ID: <20191108140049.GW14005@Air-de-Roger> References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> <20191108095754.GV14005@Air-de-Roger> <760f4fe2-8a5b-87a7-f84f-f250bb4eb31c@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <760f4fe2-8a5b-87a7-f84f-f250bb4eb31c@FreeBSD.org> User-Agent: Mutt/1.12.2 (2019-09-21) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL03.citrite.net (10.69.22.127) X-Rspamd-Queue-Id: 478hkg5SK3z4P05 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 14:01:00 -0000 On Fri, Nov 08, 2019 at 03:15:40PM +0200, Andriy Gapon wrote: > On 08/11/2019 11:57, Roger Pau Monné wrote: > > On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote: > >>> I found this in Linux code: > >>> HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); > >>> It's in xen_send_IPI_one(). > >>> I wonder if that's that or if there is more to this than meets the eye. > > > > Yes, something along this lines should work, we could even use the > > native NMI signaling using the local APIC, but the hypercall shortcut > > should be faster in most cases. > > > >> I also found this in an old post. > >> Ian Campbell wrote: > >>> You need to register a callback with CALLBACKOP_register > >>> CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive > >>> that callback. IIRC you also need to arrange that returning from an NMI > >>> is always done with HYPERVISOR_iret and not optimised to a direct iret > >>> as it can be otherwise. This is to allow the hypervisor to implement NMI > >>> masking correctly. > > > > That's AFAIK for PV guests which use a completely different mechanism > > in order to receive interrupts. None of this is needed for FreeBSD > > because there's no classic PV support. > > Perhaps HYPERVISOR_iret is still needed even for us? No, that's just for classic PV. > This is said with zero knowledge of the topic. > > > Can you try the patch below? > > I tried it and, unfortunately, it was not a success. And I don't have much > diagnostic. My bad, that's what happens when you send untested patches, the previous patch was missing an apic_cpuid and was hitting a page-fault in the kernel. Patch below should work fine. Roger. ---8<--- diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index 7d254ef3f734..8bf2158dbec2 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache; static driver_filter_t xen_ipi_bitmap_handler; static driver_filter_t xen_cpustop_handler; static driver_filter_t xen_cpususpend_handler; -static driver_filter_t xen_cpustophard_handler; #endif /*---------------------------------- Macros ----------------------------------*/ @@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] = [IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b" }, [IPI_TO_IDX(IPI_STOP)] = { xen_cpustop_handler, "st" }, [IPI_TO_IDX(IPI_SUSPEND)] = { xen_cpususpend_handler, "sp" }, - [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler, "sth" }, }; #endif @@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) XEN_APIC_UNSUPPORTED; } +#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) +static void +send_nmi(int dest) +{ + unsigned int cpu; + + /* + * NMIs are not routed over event channels, and instead delivered as on + * native using the exception vector (#2). Triggering them can be done + * using the local APIC, or an hypercall as a shortcut like it's done + * below. + */ + switch(dest) { + case APIC_IPI_DEST_SELF: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + break; + case APIC_IPI_DEST_ALL: + CPU_FOREACH(cpu) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + case APIC_IPI_DEST_OTHERS: + CPU_FOREACH(cpu) + if (cpu != PCPU_GET(cpuid)) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + default: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); + break; + } +} +#undef PCPU_ID_GET + static void xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + if (vector >= IPI_NMI_FIRST) { + send_nmi(dest); + return; + } + ipi_idx = IPI_TO_IDX(vector); if (ipi_idx >= nitems(xen_ipis)) panic("IPI out of range"); @@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg) return (FILTER_HANDLED); } -static int -xen_cpustophard_handler(void *arg) -{ - - ipi_nmi_handler(); - return (FILTER_HANDLED); -} - /*----------------------------- XEN PV IPI setup -----------------------------*/ /* * Those functions are provided outside of the Xen PV APIC implementation