From owner-freebsd-virtualization@freebsd.org Tue May 18 20:56:57 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A73A46575C2 for ; Tue, 18 May 2021 20:56:57 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fl7bX53c9z3HQV for ; Tue, 18 May 2021 20:56:56 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qv1-xf2a.google.com with SMTP id a7so1387149qvf.11 for ; Tue, 18 May 2021 13:56:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9ruPdqkyXBgHu+l2P9LlTPCw8TsOgn8ONWKvbVdQWcg=; b=LfXwYzy8U5O9Xt5Czl74cQizY8xnQIO0Hm4VCHiRGLeTyMCRvvzW66QKA1g4ic34Xk eivYLJAJ7VndegAeQ2jnKu4tAVQovcZGVpVLpAS9fU3+uO4LK9FfS+mdV8e2U0xUAzx4 CIBivgPWpnq5b8G8no5o1AAD8OuDB21ghAJy2QPmlADTwfegMyHz8AM+LSQxDn1fIkH+ yFQuFC3GgVsAA3tat0v5003wEPFVK5eH01bMQa8FgyzYLSF1rR0kv/y6ikWyWe8+ygOm lxiZY5bts9EUUzlq2baOGXGOtLUI2t8t+ogtaDPT0zhTNFVsfLNZTRk5ssVhcd5aG48W ljWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9ruPdqkyXBgHu+l2P9LlTPCw8TsOgn8ONWKvbVdQWcg=; b=RIkLdfbe5241sXsQhhA0Sc2vJeVFOnzfEWWgTLv1xawcqfedikLzHtP4JOKLdhgR1i PI552ffCvkIOo/0Cs/+aREuNrZzW7gW2tnOJ2nvsBwYEJdnvLRy8UNCOkcv8GZAmsmp1 Xdp26j8jmrnWGY3jspauR6hbr/OX8DVjWcSjdDcUaoY+QWvwXBHTyF/PJz7CFHfth4Ts HV0cYU6H7RY3OrGkK6/iGfhV5K3W+jhZzsPmLsVFStuKt1DQSeGlwj4cSwbSKQYBePB5 r7B9mRxlIajK0OQE/44Ngfc5wX0B2FAMiExSU2E3dXgflpWAUeI12b0M2u5T2QxrmKLk JeFA== X-Gm-Message-State: AOAM531QFVGeks86QxMFg2rBkKwHAg7Fl9vTXI+6qhWd7jUseaqjZnnC MHCdzl/xf4EqpKJQqeEUEE3rFtPnnjoQOrZo X-Google-Smtp-Source: ABdhPJz54+Zzdf0gaPxCED0qp/g5OPH2zkVwn17BicW367i9c0kFIOgny+uTXPDtYjjchC/m0g5kAQ== X-Received: by 2002:ad4:43c7:: with SMTP id o7mr8359977qvs.42.1621371415745; Tue, 18 May 2021 13:56:55 -0700 (PDT) Received: from mutt-hbsd ([38.140.209.219]) by smtp.gmail.com with ESMTPSA id r10sm1888632qtm.6.2021.05.18.13.56.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 May 2021 13:56:54 -0700 (PDT) Date: Tue, 18 May 2021 16:56:55 -0400 From: Shawn Webb To: John Doherty Cc: freebsd-virtualization@freebsd.org Subject: Re: increasing bhyve VM_MAXCPU Message-ID: <20210518205655.2nnjt4yfdt43atby@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <7C6FC30B-BADC-4D32-AA2B-5F00B5E874F6@jld3.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="wszted6l6snkae6g" Content-Disposition: inline In-Reply-To: <7C6FC30B-BADC-4D32-AA2B-5F00B5E874F6@jld3.net> X-Rspamd-Queue-Id: 4Fl7bX53c9z3HQV X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=LfXwYzy8; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::f2a as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-4.99 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::f2a:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.893]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::f2a:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2a:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 May 2021 20:56:57 -0000 --wszted6l6snkae6g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via freebsd-virtuali= zation wrote: > I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD > 12.2-RELEASE. >=20 > As I understand it, this limit is defined as VM_MAXCPU in > /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two > years ago, it should be possible to increase it beyond 21[1]. >=20 > I began working on a machine with only eight CPU cores/threads, so rather > than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to > work fine. My procedure was: >=20 > edit /usr/src/sys/amd64/include/vmm.h > cd /usr/src/lib/libvmmapi ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install > cd /usr/src ; make buildkernel KERNCONF=3DJLD ; make installkernel > KERNCONF=3DJLD > reboot >=20 > where JLD is a kernel config file that differs from GENERIC only in the > "ident" line. >=20 > I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs > with a number of vCPUs less than or equal to the max booted fine and that > VMs with more didn't. >=20 > So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores= /28 > threads each, using the same procedure. I set VM_MAXCPU to 28 and configu= red > a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20 > and booting a VM with 18 vCPUs but that didn't work either. In all cases, > VMs with 16 or fewer vCPUs continued to boot fine. >=20 > All the tests were done with VMs created before doing any of this stuff. I > am using vm-bhyve to manage VMs. The VMs in these tests were running one = of > FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing. >=20 > So it seems like I'm overlooking something but I don't know what, or maybe > this just isn't going to work. Anybody have experience actually doing thi= s? >=20 > Thanks very much for any advice or further information. HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our 14-CURRENT/amd64 package building VM is set up with 30 (out of 36 available) vCPUs (`vmrun.sh -c 30`). We've completed one full package build (so, 31,000+ ports) without issue. The one package building VM we have on the system (so no other VMs) is running HardenedBSD 14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs. [0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30 Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --wszted6l6snkae6g Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmCkKhQACgkQ/y5nonf4 4frv4g//WYnnfpb3LRxuTxRgwqKWZ9EGAH8TdKXbCKlQat4kcww9P78dzWNiR4/T EGm2r2IXMzY+f+tqECr1wu8Wuy+mJUTDotTm7ZP1+TdKJQsulBEDIZU5Quf+E/0e UfGegvaDySDD1cJ+dGV7H5dF8RRejZEwlZfyDXVpilL2jMaF6sjFI9H++9GHwOUF tj8ppTVxw65dviAWiu5Vw1/VgWDBfgif2cMf2iS+P/rna5Pcjt6KQbKgJBfm6zBy TgLtlf3YzmhJ94OveBoGmtoDUs5SP6wATni0JK5OsJgSRiGMGGpnBUCTNe7SEOJy ipbltjLtDojsmUwCeaMKQ9C2hR6moyCS2D5LNlSqE/hfG6FoNmKEAzgfmwaffxgb F7LqPbktzdj6RHdcLSI2bniLIQHat4EZTZ+SQU6HIqunKT1A3b9lfXJ0du9+Qk3Z HqUahFc0PNV+P2RD+rdhFHS21qR4y92kuWTz7buH6qjx5KV46TqDry+MooBGVRRg yD5IZA4EjeP84jsdEZEETctTX/D+4f6Axve2CjFYusafwSGROoHmJElEKhnReWBk 0CbskProCspXDa6yi9G3nIV35HBXWXR2xFuepCmCKO/qKtdCFPy3aS3vAP4dFoSq RtJJs/ATWdHD1uE3bTvOnCVTKjs9Nq2wtiKrdryNP7ZMmzX4iVE= =U1bc -----END PGP SIGNATURE----- --wszted6l6snkae6g--