From nobody Sun Jan 11 13:39:26 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dpxSh3ZGrz6P0JP for ; Sun, 11 Jan 2026 13:39:32 +0000 (UTC) (envelope-from bojan.novkovic@kset.org) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dpxSf6bFZz458h for ; Sun, 11 Jan 2026 13:39:30 +0000 (UTC) (envelope-from bojan.novkovic@kset.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bojan.novkovic@kset.org designates 209.85.208.50 as permitted sender) smtp.mailfrom=bojan.novkovic@kset.org Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-64b9cb94ff5so8060426a12.2 for ; Sun, 11 Jan 2026 05:39:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768138769; x=1768743569; h=content-transfer-encoding:autocrypt:subject:from:to :content-language:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m9n1Na0JEad421pf8ONEHpF3x/EgndO4yR/VQH1I4PA=; b=EZ6l6oMmdg6fymSxSuJvpdQcOpjmH3TJllGupP2piUT8QLV6VlOuQ4eUlyYpcJCkgA QpuWz7baBTr/oWs4LHOnv0oXJPtn6/QyONAl1xYd0s4pMSrPhyA3a5NkoqaNb4uK7fnN qhawH/5G9CCC2n+lT1DP1qpqSJQxl7DiErbEEy9yoULhQhUY52R2rGjoCi8ynCgIl4Ac wWlSt7bRtAoOuNHp5dLBW0IZuTJyOsjhtPj//9OWgmzH3EiH9Lbmc2JekU6OxXO9GWRv iOJshuVvs7wlnzl8DiFccDFGkYcfcP/6zCNn+doJt3KmpBoLyekTKO4lw/yTBmEOCjGa tJuA== X-Gm-Message-State: AOJu0YycRQP7OLUWVuKaL+GtX+mZ72zhkcC4YADZj9HcwqtbYonQlbj2 ViyR66Snu5MHbkcE+IOEfmapg0+iqeI0CWTxA0YSZRmVHW/l0+an+Uh9IiWRd0Po+KCSBbWMoU8 3bsBB X-Gm-Gg: AY/fxX5VSIuZkcnZ4XCfCd/WE9EFq7dnWTlDJDGDAwCiRKKfj9hEhCsLYF201SKLJiL RrKdByPKyI8WDnnmdJhjURmiGKOYFLzwqBdOd3OsejsIuR9bv7vCt4Re4Hr6JJlpbuCNJE5dCfD LXBW/eZKx3OYa6/eRrIZAah6EnkGl+HP8FgQ30i1vnCy3H9w0uBAGES35U37zTqu8r0JnfydbDv fbA0C9Tf2GKpgxF8gi7Deh9zNnggu2bOYfPhtsSRUu3fg5lQWjACLD1Tc53tKcUGhvvyMcUGxmG xvm1GHVAuNy+STDcr/Lf53N/PMwGA8uQlxqVssPfrpzjWqRBeLxslnMV2EZYGgZ9jNuUHKFEHp5 K2HuojZHgJuvpZTMbt1P5HtovNzcTuV1fn2ULuw+7kksTmqm9nAHIGfZSEBo0m4MGj7+0mqlBS/ GiAVK0 X-Google-Smtp-Source: AGHT+IExuWJUOjKFwbpDKOPN1+wS+31MX5UhXpyo7pnoRDhsnINX5ZvzgZ3Qd+9+MUAjFT0QeVAUsQ== X-Received: by 2002:a05:6402:13ce:b0:64d:1a0f:694b with SMTP id 4fb4d7f45d1cf-65097de79a8mr13642313a12.10.1768138768971; Sun, 11 Jan 2026 05:39:28 -0800 (PST) Received: from [192.168.0.91] ([188.252.173.22]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-6507bf6d5d4sm14996533a12.32.2026.01.11.05.39.26 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 11 Jan 2026 05:39:27 -0800 (PST) Message-ID: <66a1c389-6b87-4062-aa2d-a1123f9edc97@freebsd.org> Date: Sun, 11 Jan 2026 14:39:26 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: freebsd-virtualization@freebsd.org From: =?UTF-8?Q?Bojan_Novkovi=C4=87?= Subject: RFC: ACPI-based PCI device hotplugging for bhyve Autocrypt: addr=bnovkov@freebsd.org; keydata= xjMEZdIQsxYJKwYBBAHaRw8BAQdA/V2uVmdN7VY2Ps8wDgLlWU3b9+kPdg9bf+FHgEEX49TN JUJvamFuIE5vdmtvdmnEhyA8Ym5vdmtvdkBGcmVlQlNELm9yZz7CmQQTFgoAQRYhBLAb6L2d hfD6hKflVB43npi7IZ8rBQJl0hCzAhsDBQkFo5qABQsJCAcCAiICBhUKCQgLAgQWAgMBAh4H AheAAAoJEB43npi7IZ8rzb0A/0aY3c/XubbtQzNyA0xzyKNZlDc9zesxEMVi6rOAZNz/AQC2 QmBTBEcbyOKDfJ5m02LpudVi9thZxlrr2n0ZX9kgA844BGXSELMSCisGAQQBl1UBBQEBB0Dn 3m+8g7KTp3yC4CLICis/CIonFfNqQcJOVv6Gd73adQMBCAfCfgQYFgoAJhYhBLAb6L2dhfD6 hKflVB43npi7IZ8rBQJl0hCzAhsMBQkFo5qAAAoJEB43npi7IZ8ruPcBAJM5wq5j64RFu4sc zrryK4FeCTt/Xhfyn3UhT2hHuYkPAQDWHDN6XZ097C5wUkWUr8ywHDlMM5gWIDbr9TMUudoc Aw== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: - X-Spamd-Result: default: False [-1.54 / 15.00]; NEURAL_HAM_MEDIUM(-0.88)[-0.881]; NEURAL_HAM_LONG(-0.59)[-0.594]; FORGED_SENDER(0.30)[bnovkov@freebsd.org,bojan.novkovic@kset.org]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_SHORT(-0.16)[-0.163]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.50:from]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; TO_DOM_EQ_FROM_DOM(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[bnovkov@freebsd.org,bojan.novkovic@kset.org]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.50:from] X-Rspamd-Queue-Id: 4dpxSf6bFZz458h Hi everyone, Over the last couple of months I've been working on implementing PCI device hotplugging in bhyve and would like to get some feedback on the changes. The patch series from D54641 to D54656 implements an ACPI-based hotplugging mechanism that is accessible through an nvlist(9)-based IPC protocol. Users can add to or remove an emulated PCI device from a running virtual machine by sending an appropriately crafted nvlist(9) object to a UNIX socket located at '/var/run/bhyve/'. The patch series also adds new command-line options to 'bhyvectl' ('--pci-{add,remove}') to facilitate these operations. Since the implementation ended up touching a lot of files and adding a lot of changes to the PCI emulation layer, I'll try to give a brief overview of the problems each change is meant to solve and outline the major changes in this patch series. Aside from implementing the ACPI-based hotplugging mechanism, adding PCI hotplugging to bhyve required solving the following problems:  1) bhyve's PCI BAR allocator can't allocate and free BARs during runtime,  2) bhyve is sandboxed, which makes it impossible to initialize certain emulated     PCI devices during runtime since they might need to open files     during initialization,  3) Allowing users to request hotplug or hotremove operations from a running     bhyve instance. The first problem was solved using 'libuvmem(3)', a userspace port of the 'vmem(9)' general-purpose allocator. This allows us to manage resources owned by each virtual PCI bus during runtime. The second problem was solved by delegating parts of the PCI device initialization to 'bhyvectl'. Under this scheme, 'bhyvectl' opens all required files and sends the appropriate file descriptors to bhyve over a UNIX socket. To avoid unnecessary code duplication, parts of the bhyve's device-specific initialization routines were moved to 'libbhyve', a new private library. The third problem was solved by generalizing bhyve's VM snapshotting IPC thread. This allows end users to invoke bhyve's PCI hotplugging routines through an nvlist(9)-based IPC protocol. In summary, here's a brief overview of the changes made by this patch series. - Replacing bhyve's static PCI BAR allocation algorithm with 'libuvmem(3)' (D54641), - Adding ACPI methods that allow us to inject hotplugging events into the guest (D54642 - D54646), - Generalizing bhyve's IPC thread and introducing an nvlist(9)-based IPC protocol (D54650), - Refactoring and moving parts of the device initialization and configuration code to 'libbhyve' (D54648), - Adding hotplugging IPC commands to 'bhyve' and the corresponding '--pci-{add,remove}' options to 'bhyvectl' (D54651 - D54653), - Making the 'e1000' and 'virtio-net' network devices hotpluggable - (D54654 - D54656). The network devices were chosen as a proof of concept. Other devices will be added after this patch series gets reviewed. Please feel free to leave comments on any of the reviews from D54641 to D54656. Thanks, Bojan