Date: Wed, 01 Dec 2021 16:03:35 -0700 From: John Doherty via freebsd-virtualization <freebsd-virtualization@freebsd.org> To: "Miroslav Lachman" <000.fbsd@quip.cz> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve vCPU limit Message-ID: <F74A6A29-449A-470E-AB83-314972782FB0@jld3.net> In-Reply-To: <30e4454c-414a-833f-3829-586a450e7205@quip.cz> References: <PigdsByvTXmOLg46mIkWprP1GQQPuxEiHn55uKNYuSBIzBFFe-CVGYdJ2FuzYSd5OebhMlSpRGMIisaN07yzjSSaWz8JQ7LeXDeINIZg_D8=@insane.engineer> <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed 2021-12-01 09:52 AM MST -0700, <000.fbsd@quip.cz> wrote: > On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote: >> That limitation appears to still exist in FreeBSD 13.0-RELEASE: >> >> [root@grit] # freebsd-version -k ; grep 'VM_MAXCPU' >> /usr/src/sys/amd64/include/vmm.h >> 13.0-RELEASE >> #define VM_MAXCPU 16 /* maximum >> virtual cpus */ >> >> I ran into this in May 2021 and with some help from folks on this >> list was able to increase it. The simplest (if not minimalist) way to >> do that is: >> >> 1. edit /usr/src/sys/amd64/include/vmm.h to increase that value: I >> used 48 >> 2. make buildworld >> 3. make installworld >> >> The increased value has been working fine for me since I did that. I >> run a couple of VMs with 24 vCPUs each and several others with >> smaller numbers all the time and have run others with as many as 48 >> temporarily. No problems that I have seen. > > I am sorry for hijacking this thread but your information is very > interesting. I was playing with VMs in VirtualBox and Bhyve and > compared performance with increasing vCPU count. The more cores VM get > the slower was even a simple single threaded task like loading PF > rules from /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I > tested ULE and 4BSD schedulers. Maybe it was somewhat HW related but > it always shows VMs with more than 2 v CPUs significantly slower. VMs > with 6+ vCPU was almost unusable (loading of PF ruleset takes about 8 > seconds instead of fraction on single vCPU VM). > > Do you have any special tunning to have so large number of vCPU > without this penalty? I did not do anything special other than the steps described above. I did do some other stuff while sort of stumbling toward the eventual solution, but that's neither here nor there anymore. The steps above are what I used to build the primary system where I use bhyve. The physical host has two Xeon E5-2690 v4 CPUs, 14 cores/28 threads each so 28 cores/56 threads total. I have not seen but neither have I tried to measure any problems like you describe. bhyve works very well for me and I especially like it in combination with the vm-bhyve package, which I'm using to manage the VMs. Of the various virtualization systems I've used or tried over the years, I like this combination more than any other. It's simple, clean, integrates well with ZFS, and is a pleasure to use, at least by my lights. Haven't had any trouble either before or after increasing the VM_MAXCPU value.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F74A6A29-449A-470E-AB83-314972782FB0>