From nobody Tue Jan 13 18:50:58 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4drJM50WWcz6NY2Y for ; Tue, 13 Jan 2026 18:54:25 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4drJM437Hkz3xfH for ; Tue, 13 Jan 2026 18:54:24 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Mp+cDb+U; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bogorodskiy@gmail.com designates 2a00:1450:4864:20::62f as permitted sender) smtp.mailfrom=bogorodskiy@gmail.com Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-b8716197f3eso403412166b.1 for ; Tue, 13 Jan 2026 10:54:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1768330457; x=1768935257; darn=freebsd.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=ehixk452swFhBjEMJcnrisyBjAkvUR9mVaQeq410Zng=; b=Mp+cDb+UJlpKluZ407ripZyMWJyw1g5A3M67IXLV45NtyJ/o+bfeII20K4UYn9ntof 9IgHtY7Ako5OQese7DE7FbklMZ3rMZcon/b76gllsWUQM4u5KuhwNFlHySrsLbgPst4S 7DC9Qi5diF5VIwetNHz9IfW21zUcfMKJ78H0SkN9rwSJyD+w6Z7RAL5/8Ks7y8gJ6h9J UyetIciqKuXL3z5WFJecwWOKbu6KWLR7p3RBd2K29+CD7tp5y6GBcfods8gDTdVTZ5zR XatC4lrteA/GaBP9LRmPQePhC4KkiF7gPYqUAGIFTj8T5us2tnipqA7Rmzs9B7jCvpu2 Ab2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768330457; x=1768935257; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ehixk452swFhBjEMJcnrisyBjAkvUR9mVaQeq410Zng=; b=H8pH07rn8CEt3ZKQ5DRu04uZ1VK6oLTr050heqgJBPsoNc+UFMjMV8e6TvIV736qPg AGoHeE0nUghQPmpyB+LDbki/nsl24jxbIj6jP+eQ5pg4fMXYPNT6Q4eZ4mEwa15C+LRO S3ztiv1WTpYoPAcBiuCxeYRE2LOgJVTAmVu7EQQJ4/tQ7hsSMaQ/TpxdSsWhMCcwj8af 2aNNynvgmZ4921J5dynxoy8dfJVsvWrcCeACkcFpysQ58dq77q06VAqVbWl8t3S5KLoJ bJj9VZTF+lx9aIDnwrYc8Su5PKzbYJtmhcvffM67C8GJkYKm9WGGSEJA/dkCeffwV6G6 hVnw== X-Gm-Message-State: AOJu0YxsgPaSSgf2EmLqFPeZx+NB6eFMEXSv8DgXhyGUI2xnqZXne4U5 cTYYN0KrDWeattQiIbhwalmvBFM07ESY99Ku6JITXTd25A+CGNKGlH52lk5+xriK X-Gm-Gg: AY/fxX7BAZG0GQrTb9me/wCAssVGxp2Eqrt8AhDjV4AC7r2lYiyy/cEjJANVWIw5og4 6JWzPaE4Y6xMxMkbsUe1VO0ByCFgyDExM7afAIrWABY0/VQ/UsNLkCQOQAAeCy1HhU5sC8ESB+b jQHLjbaC7/vYdSK76x2p2iZsW+7cXu9GX2kIOHz+eltJ80r9lmGVxxVdP6qKUnLV6uMZZM3CbM5 ZWU8HPRMzChOSm7f0y24mpdsJxM5AKR93uEaOv5EZyR/ZemwTVFbMV0nXIsf6Y6TyNRe1aJQksc OG7/ur+jb6BYbRjb7HIGHV+DjjzDLX8zA4fRSJRHXHror4VfLEGB4YOpuFDgx24FqWg8ufJncsn vJUE6SZkUMcG+gvJpBo9KV6XS2p4jwtG71cNpRjS7bZMRpGA2wnTGbeFvUelVJh54XaPSemPKE8 tcgKu7yn6Fntv7RzW+pgrirRBLjhYbU29pk/kZUNW1IwWDZamRyDny/EK9I6BQNNoP5qBw3QzdM f3ptg== X-Received: by 2002:a17:907:86a4:b0:b87:173f:61b with SMTP id a640c23a62f3a-b8760fe0ab2mr9782266b.9.1768330455822; Tue, 13 Jan 2026 10:54:15 -0800 (PST) Received: from tulp (2001-1c02-1a15-3000-c218-03ff-feb5-6cc4.cable.dynamic.v6.ziggo.nl. [2001:1c02:1a15:3000:c218:3ff:feb5:6cc4]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b842a27c760sm2342992666b.24.2026.01.13.10.54.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jan 2026 10:54:15 -0800 (PST) Date: Tue, 13 Jan 2026 19:50:58 +0100 From: Roman Bogorodskiy To: Bojan =?utf-8?B?Tm92a292acSH?= Cc: freebsd-virtualization@freebsd.org Subject: Re: RFC: ACPI-based PCI device hotplugging for bhyve Message-ID: References: <66a1c389-6b87-4062-aa2d-a1123f9edc97@freebsd.org> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <66a1c389-6b87-4062-aa2d-a1123f9edc97@freebsd.org> X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.41 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.81)[-0.807]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[novel@freebsd.org,bogorodskiy@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[novel@freebsd.org,bogorodskiy@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62f:from] X-Rspamd-Queue-Id: 4drJM437Hkz3xfH Bojan Novković wrote: > Hi everyone, > > Over the last couple of months I've been working on implementing PCI device > hotplugging in bhyve and would like to get some feedback on the changes. > The patch series from D54641 to D54656 implements an ACPI-based hotplugging > mechanism > that is accessible through an nvlist(9)-based IPC protocol. > Users can add to or remove an emulated PCI device from a running virtual > machine > by sending an appropriately crafted nvlist(9) object to a UNIX socket > located at '/var/run/bhyve/'. > The patch series also adds new command-line options to 'bhyvectl' > ('--pci-{add,remove}') to facilitate these operations. > Since the implementation ended up touching a lot of files and adding a lot > of > changes to the PCI emulation layer, I'll try to give a brief overview of the > problems each change is meant to solve and outline the major changes in this > patch series. > > Aside from implementing the ACPI-based hotplugging mechanism, adding PCI > hotplugging to bhyve required solving the following problems: >  1) bhyve's PCI BAR allocator can't allocate and free BARs during runtime, >  2) bhyve is sandboxed, which makes it impossible to initialize certain > emulated >     PCI devices during runtime since they might need to open files >     during initialization, >  3) Allowing users to request hotplug or hotremove operations from a running >     bhyve instance. > > The first problem was solved using 'libuvmem(3)', a userspace port of the > 'vmem(9)' general-purpose allocator. This allows us to manage resources > owned by > each virtual PCI bus during runtime. The second problem was solved by > delegating > parts of the PCI device initialization to 'bhyvectl'. Under this scheme, > 'bhyvectl' > opens all required files and sends the appropriate file descriptors to bhyve > over > a UNIX socket. To avoid unnecessary code duplication, parts of the bhyve's > device-specific initialization routines were moved to 'libbhyve', a new > private library. > The third problem was solved by generalizing bhyve's VM snapshotting IPC > thread. > This allows end users to invoke bhyve's PCI hotplugging routines through an > nvlist(9)-based IPC protocol. > > In summary, here's a brief overview of the changes made by this patch > series. > - Replacing bhyve's static PCI BAR allocation algorithm with 'libuvmem(3)' > (D54641), > - Adding ACPI methods that allow us to inject hotplugging events into the > guest (D54642 - D54646), > - Generalizing bhyve's IPC thread and introducing an nvlist(9)-based IPC > protocol (D54650), > - Refactoring and moving parts of the device initialization and > configuration code to 'libbhyve' (D54648), > - Adding hotplugging IPC commands to 'bhyve' and the corresponding > '--pci-{add,remove}' options to 'bhyvectl' (D54651 - D54653), > - Making the 'e1000' and 'virtio-net' network devices hotpluggable - (D54654 > - D54656). > > The network devices were chosen as a proof of concept. Other devices will > be added after this patch series gets reviewed. > Please feel free to leave comments on any of the reviews from D54641 to > D54656. > > Thanks, > Bojan > Hi Bojan, This is an exciting feature for bhyve! Do you have a public branch with all these changes? It's a bit cumbersome to pull them from arcanist. Roman