Date: Wed, 1 Dec 2021 10:13:12 -0800 From: Paul Vixie via freebsd-virtualization <freebsd-virtualization@freebsd.org> To: freebsd-virtualization@freebsd.org Cc: bsdlists@jld3.net, jbo@insane.engineer, Miroslav Lachman <000.fbsd@quip.cz> Subject: Re: bhyve vCPU limit Message-ID: <83a44a1b-c7a8-31e6-e212-7a9a04af3fb2@redbarn.org> In-Reply-To: <30e4454c-414a-833f-3829-586a450e7205@quip.cz> References: <PigdsByvTXmOLg46mIkWprP1GQQPuxEiHn55uKNYuSBIzBFFe-CVGYdJ2FuzYSd5OebhMlSpRGMIisaN07yzjSSaWz8JQ7LeXDeINIZg_D8=@insane.engineer> <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
Miroslav Lachman wrote on 2021-12-01 08:52: > On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote: >> ... > > I am sorry for hijacking this thread but your information is very > interesting. I was playing with VMs in VirtualBox and Bhyve and compared > performance with increasing vCPU count. The more cores VM get the slower > was even a simple single threaded task like loading PF rules from > /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I tested ULE and > 4BSD schedulers. Maybe it was somewhat HW related but it always shows > VMs with more than 2 v CPUs significantly slower. VMs with 6+ vCPU was > almost unusable (loading of PF ruleset takes about 8 seconds instead of > fraction on single vCPU VM). > > ... loading a PF ruleset requires a fair bit of locking and unlocking of kernel data structures for each system call, per rule. while pfctl is single threaded, the acquisition process of those kernel locks probably requires a memory buffer flush to guaranty atomicity, and the lock's domain may overlap with other non-PF kernel activities that different hypervisors virtualize differently. this makes loading a PF ruleset a poor benchmark for hypervisors, unless that activity is so common that the unusable slowness is interfering with other work. it could be debugged or optimized in that case, but how often do you really need to add a PF ruleset? -- vixie
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?83a44a1b-c7a8-31e6-e212-7a9a04af3fb2>