From owner-freebsd-virtualization@FreeBSD.ORG Fri Sep 27 09:17:17 2013 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5DEBF1B6; Fri, 27 Sep 2013 09:17:17 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 633AB2D9F; Fri, 27 Sep 2013 09:17:15 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.90,991,1371081600"; d="scan'208";a="57983762" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 27 Sep 2013 09:17:08 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Fri, 27 Sep 2013 05:17:08 -0400 Message-ID: <52454D15.8020407@citrix.com> Date: Fri, 27 Sep 2013 11:17:09 +0200 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Julian Elischer Subject: Re: Xen (and others Hypervisors) how do they handle IPIs? References: <5242F9B8.9090300@freebsd.org> <5242FB81.4090002@citrix.com> <5243927C.9080609@freebsd.org> <5244036D.7060004@citrix.com> <52453747.6090706@freebsd.org> In-Reply-To: <52453747.6090706@freebsd.org> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 8bit X-DLP: MIA2 Cc: virtualization@FreeBSD.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 09:17:17 -0000 On 27/09/13 09:44, Julian Elischer wrote: > On 9/26/13 5:50 PM, Roger Pau Monné wrote: >> On 26/09/13 03:48, Julian Elischer wrote: >>> On 9/25/13 11:04 PM, Roger Pau Monné wrote: >>>> On 25/09/13 16:56, Julian Elischer wrote: >>>>> If CPUs are mapped around, how are IPIs handled? I assume they must be >>>>> emulated? >>>>> >>>>> I've noticed that under Xen (on both Amazon EC2 and a Redhat server) >>>>> whenever you schedule a thread it always sits on the run queue for 20 >>>>> uSecs before it starts running. It looks to me like it's the IPI >>>>> taking >>>>> a long time to be emulated. >>>> This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs >>>> instead of the emulated ones, see r255331. It should be faster than the >>>> previous emulated implementation. >>>> >>> I missed that.. thanks! >>> Do you (or anyone else) know if this can be used on Amazon EC2? >>> And do you need a specific version/configuration of Xen to be able to >>> use it? > > I ran a new GENERIC kernel on an Amazon Xen 4.2 system and it performed > a bit better. > delays in the running of threads reduced from a constant 20uSecs to a > variable amount between 1.5uSecs and 9uSecs. > It's still very slow when compared with real hardware. > (timing viewed using Schedgraph) > > Is there a way to see whether PV IPIs are active? > (it looks it from the results but that could be so many other things too). The easiest way to see if PV IPIs are active is to run vmstat -i inside of the guest and search for the following lines: irq836: cpu6:r 2 0 irq839: cpu6:irg 2530 70 irq841: cpu6:b 2175 60 This is the number of the different kinds of PV IPIs received on each CPU. You can find the correlation between this names and the IPIs on top of the file x86/xen/hvm.c. Roger.