From owner-freebsd-virtualization@FreeBSD.ORG  Sun Apr 27 15:21:06 2014
Return-Path: <owner-freebsd-virtualization@FreeBSD.ORG>
Delivered-To: freebsd-virtualization@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id C2551B8F
 for <freebsd-virtualization@freebsd.org>; Sun, 27 Apr 2014 15:21:06 +0000 (UTC)
Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12])
 by mx1.freebsd.org (Postfix) with ESMTP id 8582E1DC2
 for <freebsd-virtualization@freebsd.org>; Sun, 27 Apr 2014 15:21:06 +0000 (UTC)
Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57])
 by alto.onthenet.com.au (Postfix) with ESMTPS id 32CE012579;
 Mon, 28 Apr 2014 01:20:59 +1000 (EST)
Received: from Peters-MacBook-Pro.local (m208-191.dsl.rawbw.com
 [198.144.208.191]) by dommail.onthenet.com.au (MOS 4.2.4-GA)
 with ESMTP id BTQ84030 (AUTH peterg@ptree32.com.au);
 Mon, 28 Apr 2014 01:20:57 +1000
Message-ID: <535D2055.6060107@freebsd.org>
Date: Sun, 27 Apr 2014 08:20:53 -0700
From: Peter Grehan <grehan@freebsd.org>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:24.0) Gecko/20100101 Thunderbird/24.3.0
MIME-Version: 1.0
To: Oleg Ginzburg <olevole@olevole.ru>
Subject: Re: bhyve behavior under cpuset_setaffinity
References: <11465971.5E2sZFzEpK@kde4.my.domain>
In-Reply-To: <11465971.5E2sZFzEpK@kde4.my.domain>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-virtualization@freebsd.org
X-BeenThere: freebsd-virtualization@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "Discussion of various virtualization techniques FreeBSD supports."
 <freebsd-virtualization.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-virtualization/>
List-Post: <mailto:freebsd-virtualization@freebsd.org>
List-Help: <mailto:freebsd-virtualization-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 27 Apr 2014 15:21:06 -0000

Hi Oleg,

> For example by executing follow command on bhyve pid:
>
> % cpuset -l 2 -p 3476

>    3476 vcpu 0   RET   ioctl 0
  ...
>    3476 vcpu 1   RET   ioctl 0

  Looks like you have a 2 vCPU guest, but the cpuset only contains a 
single host CPU. What happens here under load is that you will start 
hitting "lock-holder preemption", where a vCPU can be preempted while 
holding a lock, and the other vCPU will spin trying to acquire it.

  Using the "-P" option with bhyve will help this somewhat, though I'd 
recommend using >= 2 CPUs with the cpuset command.

later,

Peter.