From owner-freebsd-virtualization@FreeBSD.ORG  Wed Sep 25 14:57:08 2013
Return-Path: <owner-freebsd-virtualization@FreeBSD.ORG>
Delivered-To: virtualization@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTP id 85EAF7A8
 for <virtualization@FreeBSD.org>; Wed, 25 Sep 2013 14:57:08 +0000 (UTC)
 (envelope-from julian@freebsd.org)
Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 59F762B75
 for <virtualization@FreeBSD.org>; Wed, 25 Sep 2013 14:57:05 +0000 (UTC)
Received: from Julian-MBP3.local
 (ppp121-45-245-177.lns20.per2.internode.on.net [121.45.245.177])
 (authenticated bits=0)
 by vps1.elischer.org (8.14.6/8.14.6) with ESMTP id r8PEv08c032599
 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO)
 for <virtualization@FreeBSD.org>; Wed, 25 Sep 2013 07:57:03 -0700 (PDT)
 (envelope-from julian@freebsd.org)
Message-ID: <5242F9B8.9090300@freebsd.org>
Date: Wed, 25 Sep 2013 22:56:56 +0800
From: Julian Elischer <julian@freebsd.org>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: virtualization@FreeBSD.org
Subject: Xen (and others Hypervisors) how do they handle IPIs?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-virtualization@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "Discussion of various virtualization techniques FreeBSD supports."
 <freebsd-virtualization.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-virtualization>
List-Post: <mailto:freebsd-virtualization@freebsd.org>
List-Help: <mailto:freebsd-virtualization-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Sep 2013 14:57:08 -0000

If CPUs are mapped around, how are IPIs handled? I assume they must be 
emulated?

I've noticed that under Xen (on both Amazon EC2 and a Redhat server) 
whenever you schedule a thread it always sits on the run queue for 20 
uSecs before it starts running. It looks to me like it's the IPI 
taking a long time to be emulated.

We have some workloads where there is a lot of flipping back and forth 
between threads and they are slowed down by an order of magnitude due 
to this..

turning off NOADAPTIVE(mumble) seems to help a bit as some of the 
reschedules go away, but it's still a problem.

Does anyone know if BHyVe or HyperV also have this problem? I have not 
yet gone to the source of Xen to see what it does but it would be 
educational to know what the other Hypervisors do.