From nobody Wed Dec 1 18:13:12 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2ADB018C14E9 for ; Wed, 1 Dec 2021 18:13:19 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from util.redbarn.org (util.redbarn.org [24.104.150.212]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.redbarn.org", Issuer "RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J46dq00yyz3G4L for ; Wed, 1 Dec 2021 18:13:18 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from family.redbarn.org (family.redbarn.org [24.104.150.213]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by util.redbarn.org (Postfix) with ESMTPS id 1905D1B242A; Wed, 1 Dec 2021 18:13:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=redbarn.org; s=util; t=1638382391; bh=7O54ULx00xRjzQRkBqOTlyQFlbiSL20wx2XvEyqftAE=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=GC3LoL57dsxPXe00kw98tx8YO6aAWCp3JIALVLclCYDEKZzWHijF0/i47qLOqkeo4 SqG2jYsvUDSEeOq9cK90dspNxquEUSpLZHnwWuMO4rFjx/PjVQeF3lBiJMGPQlyV8K vBzU3G4ufTj21m3rLT7/d6l5PcQhSORoBjT+WBjA= Received: from [IPv6:2001:559:8000:c9:3129:49f8:14c7:f25d] (unknown [IPv6:2001:559:8000:c9:3129:49f8:14c7:f25d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 02F397597E; Wed, 1 Dec 2021 18:13:11 +0000 (UTC) Subject: Re: bhyve vCPU limit To: freebsd-virtualization@freebsd.org Cc: bsdlists@jld3.net, jbo@insane.engineer, Miroslav Lachman <000.fbsd@quip.cz> References: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz> Message-ID: <83a44a1b-c7a8-31e6-e212-7a9a04af3fb2@redbarn.org> Date: Wed, 1 Dec 2021 10:13:12 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.52 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 In-Reply-To: <30e4454c-414a-833f-3829-586a450e7205@quip.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4J46dq00yyz3G4L X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Reply-To: paul@redbarn.org From: Paul Vixie via freebsd-virtualization X-Original-From: Paul Vixie X-ThisMailContainsUnwantedMimeParts: N Miroslav Lachman wrote on 2021-12-01 08:52: > On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote: >> ... > > I am sorry for hijacking this thread but your information is very > interesting. I was playing with VMs in VirtualBox and Bhyve and compared > performance with increasing vCPU count. The more cores VM get the slower > was even a simple single threaded task like loading PF rules from > /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I tested ULE and > 4BSD schedulers. Maybe it was somewhat HW related but it always shows > VMs with more than 2 v CPUs significantly slower. VMs with 6+ vCPU was > almost unusable (loading of PF ruleset takes about 8 seconds instead of > fraction on single vCPU VM). > > ... loading a PF ruleset requires a fair bit of locking and unlocking of kernel data structures for each system call, per rule. while pfctl is single threaded, the acquisition process of those kernel locks probably requires a memory buffer flush to guaranty atomicity, and the lock's domain may overlap with other non-PF kernel activities that different hypervisors virtualize differently. this makes loading a PF ruleset a poor benchmark for hypervisors, unless that activity is so common that the unusable slowness is interfering with other work. it could be debugged or optimized in that case, but how often do you really need to add a PF ruleset? -- vixie