Date: Wed, 1 Dec 2021 15:56:03 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: Miroslav Lachman <000.fbsd@quip.cz> Cc: Oleg Ginzburg <olevole@olevole.ru>, rgrimes@freebsd.org, bsdlists@jld3.net, jbo@insane.engineer, freebsd-virtualization@freebsd.org Subject: Re: bhyve vCPU limit Message-ID: <20211201235603.GP35602@funkthat.com> In-Reply-To: <8a27cc73-4c68-a643-5809-d6ec87201eb2@quip.cz> References: <PigdsByvTXmOLg46mIkWprP1GQQPuxEiHn55uKNYuSBIzBFFe-CVGYdJ2FuzYSd5OebhMlSpRGMIisaN07yzjSSaWz8JQ7LeXDeINIZg_D8=@insane.engineer> <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz> <CAMsb%2BmbY4ZbtFvBHd15naGT-BdXAVKg6BAOTpnBNg7hNcg2Czw@mail.gmail.com> <8a27cc73-4c68-a643-5809-d6ec87201eb2@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
Miroslav Lachman wrote this message on Wed, Dec 01, 2021 at 23:15 +0100:
> On 01/12/2021 19:13, Oleg Ginzburg wrote:
> > Hi
> >
> > On Wed, Dec 1, 2021 at 7:52 PM Miroslav Lachman <000.fbsd@quip.cz> wrote:
> >> I was playing with VMs in VirtualBox and Bhyve and compared
> >> performance with increasing vCPU count. The more cores VM get the slower
> >
> > How many physical cores you have on the host? This is a characteristic
> > behavior when overcoming.
> > But without overcommit I observe productivity growth:
>
> I tested it on 2 machines. One had 6 cores 12 threads, the second 4
> cores 8 threads. Even if the host machine was almost idle, only one VM
> was running the slowness with more than 1 vCPU was so significant. Let's
> say 0.01 sec for 1 vCPU, 0.3 s for 2 vCPU, 0.9 s for 4 vCPU. (on machine
> with real 4 cores / 8 threads).
> With overcommit of vCPUs it was like 3 seconds with 6 vCPU and 8 seconds
> for 8 vCPU. Still the same task on the same idle machine.
>
> On the other hand I can start 5 VMs with 2 vCPUs each and "everything
> seems normal" even if the total count of vCPUs are 10 on machine with 4
> physical / 8 threads.
> In fact I tested with total count of 15 vCPUs distributed between 5+ VMs
> but the performance problem was always visible on VM with more than 2 vCPU.
Yeah, overcommitting vCPUs is really bad, because a number of
"optimizations" that FreeBSD uses is to spin on a cpu waiting for
another CPU to unlock a lock.. If the cpu that is holding the lock
gets descheduled (stops running), then the spinning cpu will wait for
a very long time when it would have been better to just put the
processor to sleep (or do other work), allowing the original cpu to
run again, and do it's work...
If I remember correctly, adding these options:
options NO_ADAPTIVE_RWLOCKS
options NO_ADAPTIVE_SX
can improve performance due to this problem.
This also points out that wall time isn't just the only measure of
benchmarking, but the amount of CPU time used to get the work done..
Another option that has been discussed, but I don't think has made any
headway is only running all the VM's vCPUs at once, but that would
require a lot of scheduler work.
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20211201235603.GP35602>