From nobody Thu Mar 26 14:56:13 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fhRg75tsBz6RdKv for ; Thu, 26 Mar 2026 14:56:19 +0000 (UTC) (envelope-from Sean.fagan@klarasystems.com) Received: from YT5PR01CU002.outbound.protection.outlook.com (mail-canadacentralazon11021108.outbound.protection.outlook.com [40.107.192.108]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (secp384r1) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fhRg66G9qz3kyS; Thu, 26 Mar 2026 14:56:18 +0000 (UTC) (envelope-from Sean.fagan@klarasystems.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klarasystems.com header.s=selector2 header.b="As1pB/AC"; dmarc=pass (policy=none) header.from=klarasystems.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (mx1.freebsd.org: domain of Sean.fagan@klarasystems.com designates 40.107.192.108 as permitted sender) smtp.mailfrom=Sean.fagan@klarasystems.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FiccPPL2vlto/BopWGtJ8UvQHVvCEZC8vf+LqVPqrH9g+6PBcLL8CtDddN4iUXUcsHhfQn3bRpC9bb8s9MN53Gjsrg3x+yacsLG5f0XU4JgeudB3yRpXu9Lw4NIYxQ4uFZhSezLgL/EHlyafJ27mtyO7Uy5TY2vxQZLp1AsDGuvMUd0D65yXvgZ/+bybAziOOsQRWc/Fzze2/NVtxjNCeCedfRg8XznFAzOqG5wvGN0GsrRHre4MLSImRIB50rUuQCHmWAy/1KWYPHVVH47xp4PxgL8E3bB6Cfbsnw5R23zABDvwAjUioNffCBArSs/ooMwHxmSX5hMdtpepzwh8uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fkDpSc5Y3uk9rzeYA3yaXUq5yOfKWiBwVBo43Zg0AbU=; b=TX74Mvy5DoQduqN3ft5rOHWn7LRCLiRXv0c7YNwdpmVtpuFIsnDfU/sWlbNsrGdf6AkYUCQJA+Xl7DtH6l0YFu94pqAQhT9CU/H09YTb52CHrhBAhYBVSYlxTvpob0qN/edTc1JvRffrhKLhn47eRaxsVbHp23twYce9eqMy1qhWYtLGiHMRI3mh5WEt9unQXS7cQ3Om8HuPM9u8KH7zSKSjCSFHm/5LPPgxlhM4MOj5r/5daz1vjdF5JC3ulls4eXgOt+1dr3gOMEWrZq9sBOQd/yNoqhbe+3HOYGJyGK64C62Np8POKcBKQORovBcKbzA/T9Er5PbHQm8mVr6CQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=klarasystems.com; dmarc=pass action=none header.from=klarasystems.com; dkim=pass header.d=klarasystems.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klarasystems.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fkDpSc5Y3uk9rzeYA3yaXUq5yOfKWiBwVBo43Zg0AbU=; b=As1pB/AC0dn/UnBatm6MvWAaRfhBGnrZW4JwYtgI+YLwLQxYav6LEAKR34r2kuSO8oSMklja5KKrTHJ8jb2HU7Ce5hyMAt47aas/bFQtjrlq3gQ6EZADUKuUyoKjZyLB//Z8FT8eE92AjOM/7sHb/NO407WaHDhwwVsjmmfxRIk= Received: from YQBPR0101MB8191.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:52::12) by TO1PPF888705B67.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b08::665) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9745.21; Thu, 26 Mar 2026 14:56:14 +0000 Received: from YQBPR0101MB8191.CANPRD01.PROD.OUTLOOK.COM ([fe80::e2fb:33b5:f99f:938f]) by YQBPR0101MB8191.CANPRD01.PROD.OUTLOOK.COM ([fe80::e2fb:33b5:f99f:938f%5]) with mapi id 15.20.9745.022; Thu, 26 Mar 2026 14:56:14 +0000 From: Sean Eric Fagan To: "freebsd-virtualization@freebsd.org" CC: "markj@freebsd.org" Subject: bhyve and controlled errors Thread-Topic: bhyve and controlled errors Thread-Index: AQHcvTCwg3lgnvbuPUKskmJ49vMC6g== Date: Thu, 26 Mar 2026 14:56:13 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: YQBPR0101MB8191:EE_|TO1PPF888705B67:EE_ x-ms-office365-filtering-correlation-id: b2604d69-c53e-49d1-cfb1-08de8b47d36a x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|376014|366016|1800799024|38070700021|18002099003|56012099003; x-microsoft-antispam-message-info: lgZ26h6VMouaw6G55GA+RywSuVEfuJfcjtuAhrcmZZnVLefz8izPNDBhvoJipssfqYJEdBqG+piF2+EGNjIvZ3fE3/XGNsDJta1BLvyBAx7ph/pmaDShufJeiky3cbdmRTT5Ct5u7yJADyv5Y7UWXHGvP0O0Z3HV0CPsIZqNIKpHqGhSaKpc0nj99V0Y+6sFW0OG/PpQ0HsxyxuGkEuvCLM+xvoeVJtnVAbS5T/347hSOSGsma183cLV72Gfax8Hy67s6U6b+VZztEMwCPLfMuC0MGujSm6KkbIvANATmcRA7gFroOzAw+mSv2+Gz57QilTsYJw7ctgGhLX1m7A+uGL8LHQKMzC5J/R5sjaKxtjbmFFQnxCDHfqHuZxRnA7+s8lpDvv0Gqh24lbohxg9Z3u5xd2aYF274D0//aS/fdh7N7E/EDHH79P73UcczbXWihElUNumi0cBonpTwoT2/e0/7yhaECOcrevetHIddyMNhcIXm5H6SEHPIca7lqMWfw4JoZyZ3QUNUySPxMDsA+qrzB47YTbv2aRGpKW+Cr2GKIzlWSbwydlS3bPos8La6rdckl1ayYbwHE1qPa6pRgRK+W1b4+oRxwTeJ4MwyfFU5YI6klG0MT6maOk0knuh5AZwB426JrgmhHJU59zmzgSEnKp7u3XImuWuTkMmobbRw+Iy6TDgO1juOmNg8lraVwK1m1S6yRk+xWxnb1giAe1v9djrtJ2fW53L7jGxEF6BfeRKClb7pUG/hIwwduwWOU/HXMWtDCDCZgJgFYzGuBOzuh8C/Xy1lam5ZDu7nSM= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB8191.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024)(38070700021)(18002099003)(56012099003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?YlUzdkYrbTBNcTgvU0Z3ejBVM1RNQUVnYlhNVGxxS1B3R1BhSTJLWGY3NmlM?= =?utf-8?B?ZkhlRDdRRzQ1bnRjN1E1Q3c1L1pCRjViSTdsbU55TFZVYlRRTnU1bU04eXND?= =?utf-8?B?Uld6UXI3aWJDT0x0K1F0VjRjcHc0bytHZXU2U3JVQ0hsNkUwdXNDNmZJVWlz?= =?utf-8?B?SEpKV0dMTU1aSElOT1ltVWpiaEk0WCtKaW9DeUM3WmttVER5aDczRDhuYlV1?= =?utf-8?B?UVV5LzVsbkN1S05aTTVVS3RUREduTW5VbUxQRTRQMFVaUm5hd3d6SjJLR2Vt?= =?utf-8?B?N0QyYWNnZHJlTmR5T2NkWVppV05GVGhmTk45UFpTdnpyUmJwRXV2YjlpTFo2?= =?utf-8?B?R280SHJmeHhCaTRZQ2hPOTh2aGZQcHE2Sy9oY25ETFpScGd0RHZOY1pXVGxq?= =?utf-8?B?MGs0bHVzMml6Ump0M3RFYVFkdVZYVGovUHNJS1RqOVR4N2V2NnIvaWpvK21K?= =?utf-8?B?R2pDNUR4eERwM2NET1RmYk92WUxaYllldG5hMjJVYmlJc2hwQVRDK0s0SE5x?= =?utf-8?B?akkveXdLK2ZsZzVMTUhXZTAwRjE1SzI2aGtzK0ZPc0pQSy9nalBCV0VxQitH?= =?utf-8?B?YjVQQ3h1Q2lVNzRYQVdMam14dDN5K1dkN01oUWZ3b2lCcDZmeVliY0hLd01Z?= =?utf-8?B?MXFvUUN4TEd2OEYreTdxVm9PRHZnc1pVWEhQRmdJclAyc1dFSmNBVlJSa1gx?= =?utf-8?B?VnVNK08rREJlSk5PWHJVM0l4TjJQcDhWWmFHdEVJekpWaWd1cGIzRlVIcmNm?= =?utf-8?B?Q3VmN0J6M0c2SitoODBjK1ljMWY2dnFISjZGVU1KZDNmQlFhSkwzTldMU3Ja?= =?utf-8?B?aEJ2Z21OdTNPVThXNGhYd0MvQi95NW9sRFVuMk5mcXpLSDZTV0hlSDg1U1Ey?= =?utf-8?B?SzdJZkNabUhTZXg0REE4TWtlVHF0QnhIM3ZtYzRwUktkYjZiSEI4SGx2bzhm?= =?utf-8?B?aWhsYVd0dU1wdWdDYTBzYlo4NWlFcnU5UGhpSy9UOXBSVU82Y2tTVExXLy9J?= =?utf-8?B?bURaNmd6TTAwMXVQUnNXRXgzcy9QMGFRMXdzU2k0QzU2WjdwN1Rsa1BFSDJa?= =?utf-8?B?MWZJejRhd1dSajF5T1ZiSEZQRHkvMDFlb0lQMnRqTUIyd1E2TjlJaDZKUFdm?= =?utf-8?B?QnNCbFJoaXBEVFU2SjFLVTVHbThXZ1plVjFOd1FMdnpwc0p6QWRma0FCaGps?= =?utf-8?B?ZVRhRjczTXhNYW5QTFVTaTdrZitRcXR5dm5qdnkzTWFYK3p0K0F2SjBHM0Ra?= =?utf-8?B?T0g5OTlVYXFQUDZMSk5UNnBUN2J0Ujd6Ym0za0ovb1ZwdmZiNjdPclVHdU5o?= =?utf-8?B?d0hZdHgwKzJWNGFqcUlzMWZrbVhRQVNtT1dEQmlIRVVkQU1KaW9kYjNVNVVW?= =?utf-8?B?dlZVY1NNY1RIMzVWVXpVYWlzQmFuU2JMWGwwajZGajhiSEczdTJvQ1hIMTll?= =?utf-8?B?QVBHclV4YkxML1Mwa3lhR1IyTDFHNm9WbUI4SWhHTUcrR3I5b1VpV2ZWNXkr?= =?utf-8?B?elpxd09ucmkrc3VVMklCMnR5c1lENFpGV2pQNEJvOHNORk13eWdJUkNTd0Rv?= =?utf-8?B?eDBvUGdjQUlwc2NxdE5wbWI1NTB0cTRuU0FCUVdaV1JYYXBtWFpkbFIvVXUy?= =?utf-8?B?NlVhMXdTZWVZamZ5WXFYNnZ1a0tPRlVDQ2hnaDRrSHhDMkY2cVI0YTc2a1do?= =?utf-8?B?Y0lJcWRUNS9LUmpMbXBrbzdmeUhhZmRac3VlblZUUWhqbzhEV0hNU1hidDZM?= =?utf-8?B?TTJ3QVEwVkhsaTludkxuaWd5WGw2M0Zmb3RWcGptejFmaW1qa2VtdkNSUHFw?= =?utf-8?B?RTU2YnhJZ1JxbEVOT1dJUTg0bDVKc0NwajRvZks2ZGxPbW1uRGdmS1BUUlRC?= =?utf-8?B?d1lKVE1Cc25qVmppTCtjT0JVM1JkVDl2YmRMQXlkT05UL1kvYTZHTS8yaC8w?= =?utf-8?B?MitOY1BuWmpGR2ZRVHJNbjVRYVhxT3lIV0hFSFZta205UlF1MFZyWjhObmti?= =?utf-8?B?bGJPazZBaFZlcGhiUjdxWDZnSGlqVGJ0NitnTnNwTGhteWJYQWEvQ3lzYWM1?= =?utf-8?B?L0NuaG1tSjhFNVJQS2xVaTJpT1lSYXd6NDhuY05YZURaK1lRRUZpTUtpOFZx?= =?utf-8?B?R0V1QnFnOUg5SVNWeVg4YWJTbWRFMVFsQ08wczNRT0p3M2ROUEZvMWhtSWpO?= =?utf-8?B?VGJwZ3NSZ0lENVBvY0pqdmZQTkFpZ2VpQzZqOVJEUzBpLzdwcXovZ21KbThH?= =?utf-8?B?SnBnclRvd3UycmwvdWJBNyswbzlyQ29kTVNGSE9BZnRXUjFDMXh1V0o4Zloz?= =?utf-8?B?TXlKeG5xVkdOZzY1K0ZzbDZMKzFLSHRaZDB3d0syS3JReUlDWHVTMGMyTStV?= =?utf-8?Q?t1sXIN9nmeMDaXKo=3D?= Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 X-OriginatorOrg: klarasystems.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB8191.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b2604d69-c53e-49d1-cfb1-08de8b47d36a X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Mar 2026 14:56:13.8690 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 317a6cb0-9c76-4599-b7e2-d6ff3424fa73 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 68Af20/JLC2ceDnn9ir3uqq01R8qjIwVCue/2r7qS0vkX49kdQgHo/EyfYtGeA0sC6np1LUMlRAK+BjqqEXzssOOfngI9KLayMUGFDCcC2s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: TO1PPF888705B67 X-Spamd-Result: default: False [-4.89 / 15.00]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector10001:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; DMARC_POLICY_ALLOW(-0.50)[klarasystems.com,none]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[klarasystems.com:s=selector2]; MIME_GOOD(-0.10)[text/plain]; MIME_BASE64_TEXT(0.10)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.192.108:from]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[40.107.192.108:from]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[klarasystems.com:+] X-Rspamd-Queue-Id: 4fhRg66G9qz3kyS X-Spamd-Bar: ---- SeKAmWQgYXNrZWQgTWFyayBhYm91dCB0aGlzIGFuZCBoZSBzdWdnZXN0ZWQgSSBicmluZyBpdCB1 cCBvbiB0aGUgbGlzdDoNCg0KSGFzIGFueW9uZSB0aG91Z2h0IGFib3V0IGltcGxlbWVudGluZyBj b250cm9scyAvIHR1bmFibGVzIHRvIGJoeXZlIHRvIGludHJvZHVjZSBlcnJvcnM/IEkgYW0sIG1v c3Qgc3BlY2lmaWNhbGx5IHJpZ2h0IG5vdywgdGhpbmtpbmcgYWJvdXQgY2F1c2luZyBkaXNrIEkv TyBlcnJvcnMsIG9yIGhhdmluZyByZXF1ZXN0cyBkcm9wcGVkIGJ5IHRoZSAiaGFyZHdhcmUsIiB0 byB0ZXN0IGVycm9yIGhhbmRsaW5nLg0KDQpMYXN0IHRpbWUgSSBsb29rZWQgYXQgdGhlIGJoeXZl IHNvdXJjZSBjb2RlLCBpdCBzY2FyZWQgbWUsIHNvIGlmIHNvbWVvbmUgZWxzZSBoYXMgbG9va2Vk IGF0IHRoaXMsIEnigJltIGxvdmUgdG8ga25vdy4NCg0KVGhhbmtzLA0KDQpTZWFuLg0KDQotLS0N ClNlYW4gRXJpYyBGYWdhbg0KU3lzdGVtcyBFbmdpbmVlciBJSSB8IEtsYXJhIEluYy4NCkUtbWFp bDogU2Vhbi5mYWdhbkBrbGFyYXN5c3RlbXMuY29tDQpXZWI6IGtsYXJhc3lzdGVtcy5jb208aHR0 cHM6Ly9rbGFyYXN5c3RlbXMuY29tPg0KDQpEaXNjbGFpbWVyDQoNClRoaXMgZW1haWwgYW5kIGFu eSBhdHRhY2htZW50cyBtYXkgY29udGFpbiBjb25maWRlbnRpYWwgb3IgcHJpdmlsZWdlZCBpbmZv cm1hdGlvbiBpbnRlbmRlZCBvbmx5IGZvciB0aGUgbmFtZWQgcmVjaXBpZW50KHMpLiBJZiB5b3Ug aGF2ZSByZWNlaXZlZCB0aGlzIG1lc3NhZ2UgaW4gZXJyb3IsIHBsZWFzZSBub3RpZnkgdGhlIHNl bmRlciBhbmQgZGVsZXRlIGl0IGltbWVkaWF0ZWx5LiBBbnkgdW5hdXRob3JpemVkIHJldmlldywg dXNlLCBkaXNjbG9zdXJlLCBvciBkaXN0cmlidXRpb24gaXMgcHJvaGliaXRlZC4NCg0KDQo= From nobody Thu Mar 26 15:17:38 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fhS891kcGz6RgB6 for ; Thu, 26 Mar 2026 15:18:01 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from fhigh-b5-smtp.messagingengine.com (fhigh-b5-smtp.messagingengine.com [202.12.124.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4fhS886c6vz3ppZ; Thu, 26 Mar 2026 15:18:00 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id AC9017A0202; Thu, 26 Mar 2026 11:17:59 -0400 (EDT) Received: from phl-imap-17 ([10.202.2.105]) by phl-compute-06.internal (MEProxy); Thu, 26 Mar 2026 11:17:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1774538279; x=1774624679; bh=KqRtCzvfMP K8Iqf96OpRfWLYJJqZ6m6mHrlTTs0lW18=; b=GT8Q3D0/R67Gxah3qsC0B0q/10 vwx17yFVlMB3eTQIblBGY3gkXu/T6eOC8X6EbqHfn736sZJ0EquEm1H/pXI1LPWw 2J05VYMd2+mbh6cSfCuDsCzHwfm/o9taKsp0GiM5OW8Eocfs7DFGYavxXv5A7kjj RSMr33DZZlLe7vjds/oKvIPDF2jJjILR/uHtqzCUalHCDpCDHQ8J0LnYaC20upRP oQgkuqYHarqMat/s15hnKa5kY5maZIMSb+bFocuekhdzH/isFFgMp4MmJPo0s5pU ULMbW5aiypSuVHCMfXOn8JBhKedixM8W1HSOEMqSMK9IcBJWv2/nblkol10A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1774538279; x=1774624679; bh=KqRtCzvfMPK8Iqf96OpRfWLYJJqZ6m6mHrl TTs0lW18=; b=Gnl3MY+iD4BPJ/UofeigdRhZ1OWDqeAwkOSU79BhiY19HHc/+sg Qi2HlWXUoA4nLJZdOwXNF+zSsMs4KyBE9XVeLAoaUtefGm3Lip/GWfsTjigruHGL KruypgvUFxR7XZ2HwJsu0aFcx/kLl/nUMCdHKaTsSsIcNWcl7Dx9gJE80C8yzoIg BvrKrhq1I0E+a1JtQoLjNbnGXM/YDjdt2osWLqnnraiB2l6Pn3yLCPQyXwwsnCC7 /ZVmX0X1kgobr/xMs8Jn0cgZnREEq1tZv9+/nAUQo/jvphC2ZAQDAyVgM2FyBlWt VkD2n0F+3aZunfr/+X63cDfpUzAD55uzNtQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefgedrtddtgdefvdejjedvucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepofggfffhvfevkfgjfhfutgesrgdtreerredtjeenucfhrhhomhepfdevhhhutghk ucfvuhhffhhlihdfuceotghhuhgtkhesthhufhhflhhirdhnvghtqeenucggtffrrghtth gvrhhnpedvuefgudehleeuheehheetteelgfejiedtvdejheelvdekiedutefhieeigffg gfenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegthhhutghksehtuhhffhhlihdrnhgvthdp nhgspghrtghpthhtohepfedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvg gvsghsugdqvhhirhhtuhgrlhhiiigrthhiohhnsehfrhgvvggsshgurdhorhhgpdhrtghp thhtohepmhgrrhhkjhesfhhrvggvsghsugdrohhrghdprhgtphhtthhopehsvggrnhdrfh grghgrnheskhhlrghrrghshihsthgvmhhsrdgtohhm X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 1AD97C40071; Thu, 26 Mar 2026 11:17:59 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 X-ThreadId: Af1tmuC_I1Fw Date: Thu, 26 Mar 2026 08:17:38 -0700 From: "Chuck Tuffli" To: "Sean Eric Fagan" , "FreeBSD virtualization" Cc: "markj@freebsd.org" Message-Id: <0de221d3-40e5-431c-826f-4ba22c20ca17@app.fastmail.com> In-Reply-To: References: Subject: Re: bhyve and controlled errors Content-Type: multipart/alternative; boundary=1e641218b7464c31ade0e26c26006071 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:151847, ipnet:202.12.124.0/24, country:AU] X-Rspamd-Queue-Id: 4fhS886c6vz3ppZ X-Spamd-Bar: ---- --1e641218b7464c31ade0e26c26006071 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Thu, Mar 26, 2026, at 7:56 AM, Sean Eric Fagan wrote: > I=E2=80=99d asked Mark about this and he suggested I bring it up on th= e list: >=20 > Has anyone thought about implementing controls / tunables to bhyve to = introduce errors? I am, most specifically right now, thinking about caus= ing disk I/O errors, or having requests dropped by the "hardware," to te= st error handling. >=20 > Last time I looked at the bhyve source code, it scared me, so if someo= ne else has looked at this, I=E2=80=99m love to know. I prototyped something like this a few years back https://papers.freebsd.org/2019/bsdcan/tuffli-frankensteins_disk_drive/ Happy to discuss it more if you are interested.=20 =E2=80=94chuck=20 --1e641218b7464c31ade0e26c26006071 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable


On Thu, Mar 26, 2026, at 7:56 AM, Sean Eric Fagan wrot= e:
I=E2=80=99d = asked Mark about this and he suggested I bring it up on the list:
<= div>
Has anyone thought about implementing controls / tuna= bles to bhyve to introduce errors? I am, most specifically right now, th= inking about causing disk I/O errors, or having requests dropped by the = "hardware," to test error handling.

Last time I= looked at the bhyve source code, it scared me, so if someone else has l= ooked at this, I=E2=80=99m love to know.

=
I prototyped something like this a few years back
<= br>

Happy to discuss = it more if you are interested. 

=E2=80=94c= huck 
--1e641218b7464c31ade0e26c26006071-- From nobody Fri Mar 27 17:35:16 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fj7BH34q5z6WdPN for ; Fri, 27 Mar 2026 17:37:11 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fj7BG4r5hz3WyN for ; Fri, 27 Mar 2026 17:37:10 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20251104 header.b=jMqPObpA; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bogorodskiy@gmail.com designates 2a00:1450:4864:20::52a as permitted sender) smtp.mailfrom=bogorodskiy@gmail.com Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-66a9a2187a5so3720937a12.0 for ; Fri, 27 Mar 2026 10:37:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774633024; x=1775237824; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=I9lpsUfRyjIyljx3qx7wmYGfnmTnNOX5hS0i46s+Qdc=; b=jMqPObpAnoqRSdaWgxHO9iEsCy0DEKKJeLqLKc77pnK8f5lU8UH4Mq/sseTMhqUGUW 9incrhp8uG16DoC+xEBnlMUgUbvh6hnbZvUtoIzRbuc29Rgs1ALH/10vj5dZXitR3KEc 4+ym9BcFkIrrb1d8czUkTkHtLJ02Jt3yqnjuH7nrT0GMnxkqnd/tiUPCSzaON7RyoDJR oK8vGJ4SPjqP79SxMyEAm7Wz+059vEnRHaoXd0n2xtFGmwzRXJIfJ9qQ37ApVIjRBWlG J2grFr1/rLkglcGzqIoFugIzqc5NTJmhanpQMqegeuVXujEwij4vdELapHb/EBq6wc6z IhnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774633024; x=1775237824; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=I9lpsUfRyjIyljx3qx7wmYGfnmTnNOX5hS0i46s+Qdc=; b=BPM4k0bd417e/nlsAy5Z5ba0DlLtZxS0q1cCZh6gTuGgRn5ae+nbA3l9zukVdKoq0d 9id+yvCwUyRSvrj7Y1s8ZOWfI4LyeVaOwDcAFC6HiLPWO1Dkio1QXSQEXOoHOezyjuOJ 89rmB9TzXZVSscLulta8grWV7ZC/P50zFK6fjKT+4bUuUU1F78cqJtmup1G6sDqIzBki W6bghgrk743UU/URcdqpB47IzgTN/cQJMUZeugpJ8ulqWHh+KIEJU8IXcF0VFufnVzbN HWi3Prc/mNSBewhUiJyU2ugJzrjUsG0jf+edJI3/JGeb42zpjBerVBshJMPLxe7uuRcb d8ww== X-Gm-Message-State: AOJu0Yy34ftLz+o5Lw4d0B6o1Y4YI6jsVbO6zVDPFB7e4NShSIfLWEd0 pk/qG+eFfI0M/AeTYM+cqdgpwaavL3okTjUqYMBOlqy9SzlMaZoanPLUm0HSdZ8C X-Gm-Gg: ATEYQzwpOIBKR5zipOSFxiKRQl5jAn5BXV5+Z9aQ+9vPOL8waVbXEW3GgKCFNHe8zAx VhcWznChX+uY3+RiYRGX1/ZNhtb2WsXJPvbxIiUnL5BZhiiE8SgNO1gqDJC9xRzRl4AVpVTXyyw sTTx4Sc86HGyChKK6+PS1kjKV47dbN+CfLTg6VQCsYvmLBoGA9U889nE3/5/t1agAoPSRM8+VXH OSrdTfl/q2TnhXyhhauoppqje6EFpZ3oUMbit14tRNaXYtk3JrMollnX6UoR6dpubhWn7JpSpkE RnwyojaCZXxPad+NNoGmDINEUPSMOsIbS1Fob4gLrtEyy3MEMCR9MaznrEO3YbJ6UluI4nZCt+E +L47rmfTSV26yjXKQ6O4gZggHHi2HS7NE6Lkl+VCl/qHeIg4zyqnvbvjcV9EBRao6GawteWc7QF X/jYBP1+frSbJ+ X-Received: by 2002:a17:907:971e:b0:b8e:796a:fd5c with SMTP id a640c23a62f3a-b9b507b20camr242510966b.27.1774633023944; Fri, 27 Mar 2026 10:37:03 -0700 (PDT) Received: from tulp ([2001:1c02:1a15:3000:ee82:4536:a8f2:9e22]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b9b203ef86csm271568966b.49.2026.03.27.10.37.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Mar 2026 10:37:03 -0700 (PDT) Date: Fri, 27 Mar 2026 18:35:16 +0100 From: Roman Bogorodskiy To: freebsd-virtualization@freebsd.org Subject: bhyve(load) does not start in jail on -CURRENT Message-ID: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [-2.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.89)[-0.888]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[novel@freebsd.org,bogorodskiy@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20251104]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[novel@freebsd.org,bogorodskiy@gmail.com]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52a:from] X-Rspamd-Queue-Id: 4fj7BG4r5hz3WyN X-Spamd-Bar: -- Hi, I'm using jails to run some tests with bhyve and this setup works for me with 15.0-RELEASE, but does not with 16.0-CURRENT. The setup is as follows: jail.conf: testrunnergit { enforce_statfs = 2; devfs_ruleset = 44; exec.clean; exec.consolelog = /var/log/bastille/testrunnergit_console.log; exec.start = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown'; host.hostname = testrunnergit; mount.devfs; mount.fstab = /usr/local/bastille/jails/testrunnergit/fstab; path = /usr/local/bastille/jails/testrunnergit/root; osrelease = 16.0-CURRENT; vnet; vnet.interface = e0b_bastille1; exec.prestart += "jib addm bastille1 vtnet0"; exec.prestart += "ifconfig e0a_bastille1 description \"vnet0 host interface for Bastille jail testrunnergit\""; exec.poststop += "ifconfig e0a_bastille1 destroy"; allow.chflags = 1; allow.raw_sockets = 1; allow.vmm = 1; allow.mount; allow.mount.devfs; allow.mount.fdescfs; allow.mount.procfs; mount.fdescfs; mount.procfs; } When I try to run bhyveload: root@testrunnergit:~ # bhyveload -m 1024 -d /var/cache/libvirt-tck/os-x86_64-hvm/disk-freebsd-11.1.img tck32424 bhyveload: vm_create: Operation not permitted (An unprivileged user must run VMs in monitor mode) root@testrunnergit:~ # root@freebsd16-current:~ # jls -j testrunnergit allow.vmm true root@freebsd16-current:~ # devfs -m /usr/local/bastille/jails/testrunnergit/root/dev rule show 100 include 4 200 path vmmctl unhide 300 path vmm unhide 400 path vmm/* unhide 500 path tap* unhide 600 path mem unhide 700 path kmem unhide 800 path nmdm* unhide 900 path pci unhide 1000 path io unhide 1100 path pf unhide root@freebsd16-current:~ # Am I missing anything? Thanks, Roman From nobody Sat Mar 28 08:42:28 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fjWH64hbwz6W7vw for ; Sat, 28 Mar 2026 08:42:42 +0000 (UTC) (envelope-from markj@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fjWH648xhz49jj; Sat, 28 Mar 2026 08:42:42 +0000 (UTC) (envelope-from markj@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774687362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vgTIG6GA6WpePbHj0E6bQZkpjMPnAAoa791qQN/tQao=; b=TA2p4J5rtpwm/JexCvEmR1x4hq76qMS6ZpNMHlWTsSG07ttqtVqIBObGKQWG1iePQtciEq YLqERJWsnwc04l1hpMoX0kGnPHAr45/678HuAUj7R69t+bw+pD8D6WvZj+ffIW0UbNy3Mn Xl7U/Q+11xt9/uPXB81kPCxjCvVpcmiVTYAmdbBpA87OS1Vkwo6whdp3ZDUFfKdja3P9VU 8rCnrwwBq1zv7rhMS3UHJPxlGr+SNVhdX8kDIy0BuubtP2czx8tUyVZbgROdIfJMRa4vbf N/94DUWzGmLqVZ/WmPEwnXAC0vUoFBwcvlBMfPv0QSOnh5Qp8tDhFv79ky5/rA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1774687362; a=rsa-sha256; cv=none; b=AJJh3PuoNKpm/bA2Wl1CVgyR7fyznrx92++Bw0CAlFv32z5ANqIz2sGLTFau/ivZWMXpIs NK9xNBIdxz3S6z1GOcuTfyqxvyQosaaUPmdYTaWZGAJXHq9EaWAndwpKWB86YDw4kl8DsV YhBay1FNwdl/U3D7ABTdKvFcpoEl2DeTH7zMwRx4LO10Q8mj2+rUntZM6wl/emoL62xW2C 588LEBWCfGOf1xcZ4LOO73DiSBvI8VQGdjVhRHPOKJ8n4fqg+iFwfRqQLA0wxT2h9WWHeP AU2sRDBUXRS1KVkc1rCDDyYr006UEOijSQ38RyM+iBfs1iS7fU5EViB1Cbglsg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774687362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vgTIG6GA6WpePbHj0E6bQZkpjMPnAAoa791qQN/tQao=; b=o0WLikNPu2U5W9WYEzzs+6WE2cGyLA9dPv4iulStP+lK21SG8bntJ9lOCaxdY4InVuVzAh McaILiQKII+x/GHu1JUYgZALQ1D7SSNV/UCpaxNovVH/asYoRZi28xG+uXo9PHo/OVUrJV Pzmh29cPluUf61MnAYR2zyaNDOD6dSZiUdlQoB/txyrgEpNVZPhHV2uQQQgKJ0P+d7YqjS ocAP4y0ir1derETDXv22NHEcbvrC8fMNMO+fM0P+MLqbWvJ+vJioWPd/SMyS05+7B/vQjo doYAz4lHLsancp1ZAf9Ykf4r0TvyBqzdfot9FPtBf/798RO0ZL9QZWnstiZPOQ== Received: from framework (218-161-101-195.hinet-ip.hinet.net [218.161.101.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: markj) by smtp.freebsd.org (Postfix) with ESMTPSA id 4fjWH548Ghz81D; Sat, 28 Mar 2026 08:42:41 +0000 (UTC) (envelope-from markj@freebsd.org) Date: Sat, 28 Mar 2026 04:42:28 -0400 From: Mark Johnston To: Roman Bogorodskiy Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve(load) does not start in jail on -CURRENT Message-ID: References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Mar 27, 2026 at 06:35:16PM +0100, Roman Bogorodskiy wrote: > Hi, > > I'm using jails to run some tests with bhyve and this setup works for me > with 15.0-RELEASE, but does not with 16.0-CURRENT. > > The setup is as follows: > > jail.conf: > > testrunnergit { > enforce_statfs = 2; > devfs_ruleset = 44; > exec.clean; > exec.consolelog = /var/log/bastille/testrunnergit_console.log; > exec.start = '/bin/sh /etc/rc'; > exec.stop = '/bin/sh /etc/rc.shutdown'; > host.hostname = testrunnergit; > mount.devfs; > mount.fstab = /usr/local/bastille/jails/testrunnergit/fstab; > path = /usr/local/bastille/jails/testrunnergit/root; > osrelease = 16.0-CURRENT; > > vnet; > vnet.interface = e0b_bastille1; > exec.prestart += "jib addm bastille1 vtnet0"; > exec.prestart += "ifconfig e0a_bastille1 description \"vnet0 host interface for Bastille jail testrunnergit\""; > exec.poststop += "ifconfig e0a_bastille1 destroy"; > allow.chflags = 1; > allow.raw_sockets = 1; > allow.vmm = 1; > allow.mount; > allow.mount.devfs; > allow.mount.fdescfs; > allow.mount.procfs; > mount.fdescfs; > mount.procfs; > } > > When I try to run bhyveload: > > root@testrunnergit:~ # bhyveload -m 1024 -d /var/cache/libvirt-tck/os-x86_64-hvm/disk-freebsd-11.1.img tck32424 > bhyveload: vm_create: Operation not permitted (An unprivileged user must run VMs in monitor mode) > root@testrunnergit:~ # > > root@freebsd16-current:~ # jls -j testrunnergit allow.vmm > true > root@freebsd16-current:~ # devfs -m /usr/local/bastille/jails/testrunnergit/root/dev rule show > 100 include 4 > 200 path vmmctl unhide > 300 path vmm unhide > 400 path vmm/* unhide > 500 path tap* unhide > 600 path mem unhide > 700 path kmem unhide > 800 path nmdm* unhide > 900 path pci unhide > 1000 path io unhide > 1100 path pf unhide > root@freebsd16-current:~ # > > Am I missing anything? I think this is a regression. Could you please try the patch here? https://reviews.freebsd.org/D56119 From nobody Sat Mar 28 09:22:00 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fjXBh6r4Mz6WCl8 for ; Sat, 28 Mar 2026 09:23:56 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fjXBh3Xv0z4Hjt for ; Sat, 28 Mar 2026 09:23:56 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-b982518b73fso464112366b.1 for ; Sat, 28 Mar 2026 02:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774689830; x=1775294630; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=C9HVRu6nJXYe8exEQP+dMA6Pp9/0hRYFh5razImZF1I=; b=UF9Uz3JjGO7b+9AIhqHuXQIIebH4OBQqnqL0SriUPsFBaAdYdwzpBAj9TI5JiWEi0r X8yMfLQqevqXAEC8LKJNL8Mw29voQYu410HyEKZf5q+fE2LqNaM3Actc6YBDMncL+CMW BXo/RvzCghz+7EQ+kyAUL1P6GA8Iru239xHaT6FinXWLJCYbvp90PumkJONB/pPj3F3T ph9cf2IeqQJE/RGLHIdZyr3XRH9B+oA16/uZnNIQXOacwVNYjj1pYfrE/pXJYcPoA5IB yAgxFWprrZyiO6f7bIS8DBR8ObZa+y/1an1YXHSNjU7PHeZEA1e2Bpvc431Ld+TXWGo0 rzLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774689830; x=1775294630; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=C9HVRu6nJXYe8exEQP+dMA6Pp9/0hRYFh5razImZF1I=; b=Y6jXpTzcsvD/+v5cOs/Uqw51iqRKXXcsuXBMq0PQxe2HHfpK0ZnS7NNoHb4W9bIAe0 4/4g/fF4mpgdhGvOoEoWlHKQ/iqUJUQHtyYZ85LP3kJurtcqMNoCwngKZyrMK+XL1vLl eGPTZGPANt5R7zcXW3lV+vM0pkhspkjIqkaOK6Bgpm4ZLtX8Lm4DV/Ev1I6lfHCKS84j EZxGil3jOs0i3omSNr3d5D/kPXkX7B8+tVUaM+Rm1UiZcowzCSghInn8T38Ki0q1oPT9 z+TnweVG+oOq5nqVcibNw2/ORglK0LhXxiDh6RgWZs9mx3auGuFXqIGQMMMpYs6ceDhw dg4w== X-Gm-Message-State: AOJu0YwxlCcdGdQyCvTgQ5zbZbjH/5IWJDidFnxPpS/Mb34J/FwFHmL8 KeF/E+30DVSFeKlZLIlpcCyCueQdkstzvhswciePDmoLU9VR4Stkczj2 X-Gm-Gg: ATEYQzxCb7Gl3HAaNKOgy8xoZ5WiPKMOan6T+6hlw4b+fadNSaSy51q88yKfn57FRe6 j3KkJfh7URrmvnB5OxBggzdk7xuKYKqxIvLo3EqaPhi/FKopJH6DKhKxMZ4uHMyLHwzm+wHypJo nxH1B+tFrp079/PIiyG2gA3UHNsr5ODBM4ZIlCPzYq5DfNm1FNAYHnUhJBV5I2qisKAKgv7/5ra 6MxI/QXcNFU5BLi/6WVAy6vvcHcGUjDAFEX8MAodcP2nc74WfCePIJRmLjtNo+vrEr405qV1rt5 A5PNWZjZOwvRqZvRCimYSr6Z8zqyd3jiwWpG3TedrXAJDtvUQXA6EmdPOeUThsHngvy328AozXR gLBimyOilTYEy0p/KA9NU1txskacWFViKEGCBiEuuMFyTi5cVyOM0EWfsgrxMABoDtKxnQUyvDJ EBbH5+nQKfVNNPTbAHHZVApUx3+TNSw236Xj4sZNn6bgK8yjpwityeJWhcTTxb6PRA2aELbVFLb tuSZY6s6u7iEH+zco8= X-Received: by 2002:a17:907:1c25:b0:b94:a1d4:ceff with SMTP id a640c23a62f3a-b9b508f62efmr332052766b.35.1774689829711; Sat, 28 Mar 2026 02:23:49 -0700 (PDT) Received: from tulp (2001-1c02-1a15-3000-ee82-4536-a8f2-9e22.cable.dynamic.v6.ziggo.nl. [2001:1c02:1a15:3000:ee82:4536:a8f2:9e22]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b9b7ae52064sm56593966b.23.2026.03.28.02.23.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Mar 2026 02:23:49 -0700 (PDT) Date: Sat, 28 Mar 2026 10:22:00 +0100 From: Roman Bogorodskiy To: Mark Johnston Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve(load) does not start in jail on -CURRENT Message-ID: References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4fjXBh3Xv0z4Hjt X-Spamd-Bar: ---- Mark Johnston wrote: > On Fri, Mar 27, 2026 at 06:35:16PM +0100, Roman Bogorodskiy wrote: > > Hi, > > > > I'm using jails to run some tests with bhyve and this setup works for me > > with 15.0-RELEASE, but does not with 16.0-CURRENT. > > > > The setup is as follows: > > > > jail.conf: > > > > testrunnergit { > > enforce_statfs = 2; > > devfs_ruleset = 44; > > exec.clean; > > exec.consolelog = /var/log/bastille/testrunnergit_console.log; > > exec.start = '/bin/sh /etc/rc'; > > exec.stop = '/bin/sh /etc/rc.shutdown'; > > host.hostname = testrunnergit; > > mount.devfs; > > mount.fstab = /usr/local/bastille/jails/testrunnergit/fstab; > > path = /usr/local/bastille/jails/testrunnergit/root; > > osrelease = 16.0-CURRENT; > > > > vnet; > > vnet.interface = e0b_bastille1; > > exec.prestart += "jib addm bastille1 vtnet0"; > > exec.prestart += "ifconfig e0a_bastille1 description \"vnet0 host interface for Bastille jail testrunnergit\""; > > exec.poststop += "ifconfig e0a_bastille1 destroy"; > > allow.chflags = 1; > > allow.raw_sockets = 1; > > allow.vmm = 1; > > allow.mount; > > allow.mount.devfs; > > allow.mount.fdescfs; > > allow.mount.procfs; > > mount.fdescfs; > > mount.procfs; > > } > > > > When I try to run bhyveload: > > > > root@testrunnergit:~ # bhyveload -m 1024 -d /var/cache/libvirt-tck/os-x86_64-hvm/disk-freebsd-11.1.img tck32424 > > bhyveload: vm_create: Operation not permitted (An unprivileged user must run VMs in monitor mode) > > root@testrunnergit:~ # > > > > root@freebsd16-current:~ # jls -j testrunnergit allow.vmm > > true > > root@freebsd16-current:~ # devfs -m /usr/local/bastille/jails/testrunnergit/root/dev rule show > > 100 include 4 > > 200 path vmmctl unhide > > 300 path vmm unhide > > 400 path vmm/* unhide > > 500 path tap* unhide > > 600 path mem unhide > > 700 path kmem unhide > > 800 path nmdm* unhide > > 900 path pci unhide > > 1000 path io unhide > > 1100 path pf unhide > > root@freebsd16-current:~ # > > > > Am I missing anything? > > I think this is a regression. Could you please try the patch here? > https://reviews.freebsd.org/D56119 Yes, this patch fixes it for me. Thanks, Roman From nobody Sun Mar 29 13:45:54 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fkFyX3stbz6X6t7 for ; Sun, 29 Mar 2026 13:45:56 +0000 (UTC) (envelope-from danilo@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fkFyX3Ny0z43xk for ; Sun, 29 Mar 2026 13:45:56 +0000 (UTC) (envelope-from danilo@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774791956; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mUJLNzY5tHUoWzWCc2uGCl/ypkCcCqbuVoNzTdv+VSY=; b=SX/GSj+6VK69anLRCMcwW9Uvxfoit6AolFIgCqfl6km8EZhch4v3SN3dbiObgOOGIHpCvQ v6S+sO0xbdwsqG3j2WIFA0nvQYxFinLoAyWxU8d4aOP/5EYusLHgUpBTUukHdjIPkc/BOA cQvVWtgttJ7/aqJ4Sf/mBYjpTMKpbThZjReeykYjCHhZU+4ydfZD3Oozfd0QmHnnYO3DuY uMpFY6W/PEKdvcqMan9cp1t7BiwSsWeikutXUIqUqhEkw6n8V30XDxZPnHfuozWSMyJVmR +Swh/zP9H0njXmG0KLv1VWGL4c+KKcr3sz7NBjYOS8xDJGMNPGsXI7PsL6yWvQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1774791956; a=rsa-sha256; cv=none; b=gpLaQ/r+L7eghfo/5U2x7Z3weECwnI+fYq0V6V83QLKBZEaLN1FSEsjpti2E1cc0kgZ+rR xiY7+a6RVAKz5wyuIa8BRhteKJ8olDaX/rbCJu4uRxfDEshhnEGvcvktovrq+AgUiyAe4y +CZyg7j+2jXdzevmzHkjt1ARXBz7ia3HJ3Gfcm9DxZHfHkGtOQGsRNME8y6NDMYeVZDZEL 0NLXq8JzlOA/3FW1WMId/jPFoQEN4D9m5gXoXyuTNQ8jg9qCp8lxAwf7x3RDDdy7oVlyT8 0xPkqs1wRZaB9yfmLfhcIuN79y7bbokDmwHJR0Q1gnzHzbvj9j51cJ0aIq0VZQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774791956; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mUJLNzY5tHUoWzWCc2uGCl/ypkCcCqbuVoNzTdv+VSY=; b=UL19ZlfgClsKkaRVf0fJmj4q1AQNvxHxJ6f877jc5OOj2QelNHl0ndEP9ilvWcXhzWcfuF GYBoIyWpQ9mwy2LNQu0T8zlp6e/Fio2lfASPo8eaW8JCyjybGff88Ilt6/yZeOHTkdPBBg vS7wIKVJjaBw29Ytr0poOBKbXtZC6dXZLxDo6BGt7YAoUPscdehxeor4OFnOCVu37nLptW hVyDPJs7U/AbMJ/WYz80yJWpLMl5LLbfzbdkgBQ52b1McfrC5lPeRe+7C57f7ryQb07+57 24tXPoh5pTePLiZn85sgqjeL6bvzNOZrAoa9zzG714GQ6oOGfyyYrWjUBvPQ/w== Received: from [192.168.1.126] (unknown [37.228.200.108]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: danilo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4fkFyX168qz134P for ; Sun, 29 Mar 2026 13:45:56 +0000 (UTC) (envelope-from danilo@FreeBSD.org) Message-ID: Date: Sun, 29 Mar 2026 14:45:54 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: virtualization@FreeBSD.org From: Danilo Egea Gondolfo Subject: Incus 6.23 released with official support for FreeBSD guests Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi all, Incus is a container and VM manager for Linux. They recently released version 6.23 with official support for FreeBSD 14 and 15 guests. That means that you can now launch FreeBSD VMs in an Incus cluster with a single command: $ incus launch images:freebsd/14.4 freebsd-14-vm -c security.secureboot=false --vm Launching freebsd-14-vm They also ported their incus-agent so now it works on FreeBSD guests. It allows you to use the Incus client to access the VM, copy files and etc: $ incus shell freebsd-14-vm root@freebsd-14-vm:~ # uname -a FreeBSD freebsd-14-vm 14.4-RELEASE FreeBSD 14.4-RELEASE releng/14.4-n273675-a456f852d145 GENERIC amd64 They are still building the new images with an embedded agent (I built and copied it to the VM myself for the example), so it may take a few days. Without the agent you can access the VM with: $ incus console freebsd-14-vm This is the announcement: https://discuss.linuxcontainers.org/t/incus-6-23-has-been-released/26447#p-89931-freebsd-vm-support-6 There is a Github issue to track the support for FreeBSD: https://github.com/lxc/incus/issues/3069 I just pushed sysutils/incus-client to the Ports tree. You can use it to manage Incus servers from FreeBSD. This is how you configure the server and the client: https://linuxcontainers.org/incus/docs/main/howto/server_expose/#server-expose Check it out :) From nobody Sun Mar 29 21:00:22 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fkRbq2QCyz6XN28 for ; Sun, 29 Mar 2026 21:00:23 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fkRbp5w0Hz3wKf for ; Sun, 29 Mar 2026 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774818022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wf+yKo/D4RvfB8RddBo1Ekcl7OEQSuzc0ZsoPYQFC6U=; b=WDAdY1nn5OdGqh8wdgx89NbpqZTIZlCrCtUdFT+yokv/RBuagk+1GTMMsk6ebI6MSyVCpF cfNP3RLQ29WzFSanq+cVdtNBinXSsLSseWBkygB3ADTHo+8HK5GPTReGVjzZwRwpb1nv1N jATf4P2bPhlhqWnTDWzuR+6MYiArYeeJlJ0t+ssc3pACFpbCnd0C8GGAW+iz/wP6kJJKZp +28p3eoVKaucExL4NPuRzk97ZvJk4ZzblMPtdWjOiAnDUdy4cY9wzv7Ras65V1pLjSrDgX Q3FhLMJAV3KESDqeFAhROKhRkbrbQdt2Y1mtUEnvBBTr+Pb7XZRwkoGLQaAhBw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1774818022; a=rsa-sha256; cv=none; b=DtqOVuVJDm4feezdEebv3PquPLdkNUX/g93ZKdmH+WNZLghqF0RbSRUjv8nkfRqXwwb+ki E/5UGc3fTV7FsFGcH+YIv15L2y/VqLSAxPaTyBQPB7RRUOefuyzYRpos/EcQd5GfHHbG43 f4RD+IxxsYlH5xitdwRyUKLEDEH7vImEbgUVjp1aucemTn8EceiImIY8zjfyDBM42sf6/+ vvca9DPb0MK/hFX4sRVs5Lqfd6PWS24EsWHFFi+v0LsbgWTSxmTbcVLld2EQKcYTuokkFC KLZVpY/qpGvCIH78cOeHODTcBildfCPde4xlKrVSGCj3ir8HP2QdYjtoxbDXBA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1774818022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wf+yKo/D4RvfB8RddBo1Ekcl7OEQSuzc0ZsoPYQFC6U=; b=u5wN8WOHJlkTtPmjBMfQMoPIwK9vPa79VmItZ5P3xk+OwZilKlyHBOM08tWZ+pntDahWdG 09cwKpcLL046pohune3fY1BdEzUPeQIrVRzRcHFLkL9OFN8NUbCkZrWK5MpT9tDndxaTfb 2OX+9ncS9V+8PlUCNFjYKCxu30EIadWk/VDnitbTUdzPeTAM6HyjfJPywKLQXgRpfHZpBQ yNYfzxZV5gQglT/SMYumCjUrfop/4oTIA/B3ESu09BKe1VKLY7KOnc/itR+5+HShL06KWC WViQtjSm7O9B4FFNVA1pY0sDjfJ0GRt8wRN5sbbzMeD5Gj1tGbT1kmHN/h2P3A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4fkRbp4VCdz1RS6 for ; Sun, 29 Mar 2026 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 62TL0MxR047272 for ; Sun, 29 Mar 2026 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 62TL0MCY047271 for virtualization@FreeBSD.org; Sun, 29 Mar 2026 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202603292100.62TL0MCY047271@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 29 Mar 2026 21:00:22 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17748180224.15c8cD.44644" Content-Transfer-Encoding: 7bit --17748180224.15c8cD.44644 Date: Sun, 29 Mar 2026 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 289848 | Enable by default BHYVE_SNAPSHOT 1 problems total for which you should take action. --17748180224.15c8cD.44644 Date: Sun, 29 Mar 2026 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    289848 | Enable by default BHYVE_SNAPSHOT

1 problems total for which you should take action.
--17748180224.15c8cD.44644--