From nobody Wed Dec 1 23:56:03 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0CE9F18A980C for ; Wed, 1 Dec 2021 23:56:14 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J4GFT5kDmz3C22; Wed, 1 Dec 2021 23:56:13 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 1B1Nu5bk071304 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Dec 2021 15:56:05 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 1B1Nu3Zt071300; Wed, 1 Dec 2021 15:56:03 -0800 (PST) (envelope-from jmg) Date: Wed, 1 Dec 2021 15:56:03 -0800 From: John-Mark Gurney To: Miroslav Lachman <000.fbsd@quip.cz> Cc: Oleg Ginzburg , rgrimes@freebsd.org, bsdlists@jld3.net, jbo@insane.engineer, freebsd-virtualization@freebsd.org Subject: Re: bhyve vCPU limit Message-ID: <20211201235603.GP35602@funkthat.com> Mail-Followup-To: Miroslav Lachman <000.fbsd@quip.cz>, Oleg Ginzburg , rgrimes@freebsd.org, bsdlists@jld3.net, jbo@insane.engineer, freebsd-virtualization@freebsd.org References: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz> <8a27cc73-4c68-a643-5809-d6ec87201eb2@quip.cz> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8a27cc73-4c68-a643-5809-d6ec87201eb2@quip.cz> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Wed, 01 Dec 2021 15:56:05 -0800 (PST) X-Rspamd-Queue-Id: 4J4GFT5kDmz3C22 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Miroslav Lachman wrote this message on Wed, Dec 01, 2021 at 23:15 +0100: > On 01/12/2021 19:13, Oleg Ginzburg wrote: > > Hi > > > > On Wed, Dec 1, 2021 at 7:52 PM Miroslav Lachman <000.fbsd@quip.cz> wrote: > >> I was playing with VMs in VirtualBox and Bhyve and compared > >> performance with increasing vCPU count. The more cores VM get the slower > > > > How many physical cores you have on the host? This is a characteristic > > behavior when overcoming. > > But without overcommit I observe productivity growth: > > I tested it on 2 machines. One had 6 cores 12 threads, the second 4 > cores 8 threads. Even if the host machine was almost idle, only one VM > was running the slowness with more than 1 vCPU was so significant. Let's > say 0.01 sec for 1 vCPU, 0.3 s for 2 vCPU, 0.9 s for 4 vCPU. (on machine > with real 4 cores / 8 threads). > With overcommit of vCPUs it was like 3 seconds with 6 vCPU and 8 seconds > for 8 vCPU. Still the same task on the same idle machine. > > On the other hand I can start 5 VMs with 2 vCPUs each and "everything > seems normal" even if the total count of vCPUs are 10 on machine with 4 > physical / 8 threads. > In fact I tested with total count of 15 vCPUs distributed between 5+ VMs > but the performance problem was always visible on VM with more than 2 vCPU. Yeah, overcommitting vCPUs is really bad, because a number of "optimizations" that FreeBSD uses is to spin on a cpu waiting for another CPU to unlock a lock.. If the cpu that is holding the lock gets descheduled (stops running), then the spinning cpu will wait for a very long time when it would have been better to just put the processor to sleep (or do other work), allowing the original cpu to run again, and do it's work... If I remember correctly, adding these options: options NO_ADAPTIVE_RWLOCKS options NO_ADAPTIVE_SX can improve performance due to this problem. This also points out that wall time isn't just the only measure of benchmarking, but the amount of CPU time used to get the work done.. Another option that has been discussed, but I don't think has made any headway is only running all the VM's vCPUs at once, but that would require a lot of scheduler work. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."