From owner-freebsd-virtualization@freebsd.org Wed Feb 14 19:27:17 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34A7AF195BF for ; Wed, 14 Feb 2018 19:27:17 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2A8469152 for ; Wed, 14 Feb 2018 19:27:16 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w1EJANsW054511; Wed, 14 Feb 2018 11:10:23 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w1EJAMQi054510; Wed, 14 Feb 2018 11:10:22 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201802141910.w1EJAMQi054510@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve and contention In-Reply-To: To: Shane Ambler Date: Wed, 14 Feb 2018 11:10:22 -0800 (PST) CC: tech-lists , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2018 19:27:17 -0000 [ Charset UTF-8 unsupported, converting... ] > On 14/02/2018 10:18, Peter Grehan wrote: > >>> 2. > >>> In the following context, the server is the same but this time all five > >>> guests have -c 4 per guest, so bhyve is asking 12 more cores than that > >>> existing in hardware. Does the guest fail to load, do either guest or > >>> server crash? > >> > >> The is core over commit, very common in the virtualization world, > >> bhyve does its best effort to give the guests cores as needed. > > > > ?To add to what Rod said - bhyve uses a thread for each vCPU. It's up to > > the FreeBSD scheduler to determine where/when these threads run. > > > > ?It is possible for a guest to fail, for example if a spinlock times out > > due to vCPUs not being able to run to release a lock. This is a general > > problem with virtualization and can occur even on VMWare ESXi with heavy > > oversubscription. > > There are some bhyve options you need to check - > > You want to use the -H option which prevents the guest keeping 100% cpu. Very true, this yields the vCPU when a halt is executed. > Be sure not to pass the -S option to bhyve as it wires the guest memory. > This was seen recently as sysutils/chyves uses it by default. This should be evaluated, if you do not pass -S you run into the possible performance impact of swapping guest memory, which does infact allow you more over commit, but at a pretty large performance penality. You need swap space if you are going to not pass -S and over commit, if you do not have enough swap space for the over commit your system runs out of swap and usually ends up killing random processes, and random usually means the most important things you are running! > https://www.mail-archive.com/freebsd-virtualization@freebsd.org/msg05665.html > > If you use zfs, you may want to set the arc_max. This is not a may, if you use ZFS you must restrict its buffer cache, or expect unstable systems. ZFS's default situation is to use up to 95% of memory, that leaves almost nothing for a running VM. > One thing that I have noticed, as a guest allocates ram, the host > allocates ram to its bhyve instance, but when the guest releases it, the > host is unaware of the ram now being free. Apparently there is work on > getting the guest to notify the host of released ram. Until that is > available you may want to restart guests that temporarily allocate large > amounts of ram. Correct, we need an implementation of what is called a memory ballon driver so that the guest can pass back to the host un used memory. > -- > FreeBSD - the place to B...Sharing Devices > Shane Ambler -- Rod Grimes rgrimes@freebsd.org