Date: Tue, 24 Jul 2018 18:09:26 -0700 (PDT) From: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-virtualization@freebsd.org" <virtualization@freebsd.org> Subject: Re: Overcommitting CPUs with BHyve? Message-ID: <201807250109.w6P19QVf038767@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <CAOtMX2hEPnBaaKk25YWe6mYW-gCA6pGssEHTBB4GJ-C6sQU%2Btg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> What are people's experiences with overcommitting CPUs in BHyve? I have an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs without > problems. But today I got greedy. I assigned 8 cores to one VM for a big > build job. Obviously, some of those were shared with the host. I also > assigned it 8GB of RAM (out of 16 total). Build performance fell through > the floor, even though the host was idle. Eventually I killed the build > and restarted it with a more modest 2 make jobs (but the VM still had 8 > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. > > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? > 2) Could this be related to the pfault hang, even though the guest was idle > at the time? I on occasion do over commit vCPU's in bhyve, but I do so with a few specific conditions: 1) I count CPU's as real Cores, not Hyperthread cores, I do not expect hyperthreading to work well in over commit. 2) I always wire my VM's memory, I NEVER overcommit memory, that just leads to bad and ugly. (-S option to bhyveload and bhyve) This totally takes ARC issues out of the picture, but you may not be able to start your VM's if you dont decrease the ARC. 3) Watch out for host side disk drive IOP saturation, you can easily stahly your guests if your trying to do to much I/O, they usually recover from this on there own, though it can make things go pretty slow for a time. Firing off 16 VM's doing "nightly" on a single spindle host is a sure way to have some very long runs. My work load runs from an always running 6 vCPU light load, to an occasion guest running make -j4 buildworlds (total of 10 vCPU load). My host has 4 cores, 8 threads. The 10vCPU load usually drives the host to a load average of 5. The 6 vCPU always present light load very rarely drives the load above 1. I think the secret sauce is wired memory :-) -- Rod Grimes rgrimes@freebsd.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201807250109.w6P19QVf038767>