From nobody Mon Feb 12 00:32:23 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TY55D4hRsz592tM for ; Mon, 12 Feb 2024 00:32:32 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TY55C5gQbz4kxn for ; Mon, 12 Feb 2024 00:32:31 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=b9rLtMP0; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41C0WNLx085792 for ; Sun, 11 Feb 2024 18:32:23 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1707697943; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=91HXrawuZ2ozGGPfq5b4WksfP/YOUsz9YoedrK3osko=; b=b9rLtMP0WUaP+uv1DAKUT1h1a4QhTHpwLUpAigFufW6KMDHk4oJ5d3zdmvYv4I5dJudbOj PClikiwuDWe4KFnQ1vW5bKOSFURawq/SeRZ519QBfRO3tEgsmBK930P6IYqDVBK5+C7+1y mm233edCHlYjtU+LoTZFnxYFeLZVGe8lttTDCIa8YnvMlvKAS408xjubOrAFFwTxPgs8iu XReFv73S/Bk8CJoSsrgMnIrp7oviJRWrmqrkv05y/kfaAZX7XcET4ZB5b/5X4i1qLNwXM4 fwHBuipqXRQWLXRZR8I+N4W5IRV1Rj+5U3Wr40Kald6RsmIkjODVoirF9sLNNg== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id BE04D3AB37 for ; Sun, 11 Feb 2024 18:32:23 -0600 (CST) Message-ID: <678ac1d2-d00c-450e-bc2e-f9d979d19340@shrew.net> Date: Sun, 11 Feb 2024 18:32:23 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: FreeBSD virtualization Content-Language: en-US From: Matthew Grooms Subject: 14R bhyve RHEL9+NVME Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.981]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[shrew.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TY55C5gQbz4kxn Hey All, Has anyone had experience installing either Rocky or Alma Linux 9.3 using virtual NVME disks on a FreeBSD 14R host? I have no problem using VirtIO block devices or if I downgrade to an 8.9 variant. However, using NVME with a 9.3 variant will cause the installer to consistently fail on my system. I'm not sure how to gather more data at this point as the installer displays a wall of text that I'm not sure how to extract. I've already burned a few hours on this so I thought I'd reach out to the list to see if anyone else has already dug into this. If not, I'll see what I can do to gather more info. For reference, here are the ISOs I've tried installing from so far ... AlmaLinux-8.9-x86_64-boot.iso AlmaLinux-9.3-x86_64-boot.iso Rocky-9.3-x86_64-minimal.iso Thanks, -Matthew From nobody Mon Feb 12 01:25:59 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TY6H84nKbz597dR for ; Mon, 12 Feb 2024 01:26:12 +0000 (UTC) (envelope-from jason@tubnor.net) Received: from mail.tubnor.net (mail.tubnor.net [103.236.162.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TY6H73nWcz4pVn for ; Mon, 12 Feb 2024 01:26:11 +0000 (UTC) (envelope-from jason@tubnor.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tubnor.net; s=20220915; t=1707701160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+A27OVeYZFbUa4RF8mL5rz6jAAe6wb7h1qTmJCaXe8U=; b=TCbsGMcZjQh/keTDXJX3nLmRvTh8bBBL1mx8pWFkZQBGnndhvPHK1MSkxczi/1j0VNf0bD DuMGEXHLOLTiVaUIqtywphT75aBzFMvLXN4O8JgsPk+f7WdMjOMM2KKsSa3/zf8S2cAZmM rox7N68aM7CCVAc/83Ia5xFjaja6/24= Received: from drive.tubnor.net ( [2403:5812:73e6:2::22:1]) by mel01.ar18.net (OpenSMTPD) with ESMTPSA id ec149df5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 12 Feb 2024 12:25:59 +1100 (AEDT) From: Jason Tubnor To: FreeBSD virtualization , Matthew Grooms Cc: Subject: Re: 14R bhyve RHEL9+NVME References: <678ac1d2-d00c-450e-bc2e-f9d979d19340@shrew.net> In-Reply-To: <678ac1d2-d00c-450e-bc2e-f9d979d19340@shrew.net> Message-ID: <20240212012559.Horde.aH5FMCkNv4j7GfULz1fq5rn@drive.tubnor.net> User-Agent: Horde Application Framework 5 Date: Mon, 12 Feb 2024 01:25:59 +0000 Content-Type: text/plain; charset=utf-8 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:133159, ipnet:103.236.162.0/23, country:AU] X-Rspamd-Queue-Id: 4TY6H73nWcz4pVn X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated Hi Matthew,   There is some underlying bug post 9.1 when it comes to Anaconda and the presented NVMe storage. I can't reproduce it with newer version of Anaconda that is present in Fedora.   Uplifting in place from 9.1 to 9.3, there are no issues once the operating system is up and running, so it isn't a RHEL thing, just the installer.   Cheers,   Jason.   "Matthew Grooms" mgrooms@shrew.net – 12 February 2024 11:32 AM > Hey All, >  > Has anyone had experience installing either Rocky or Alma Linux 9.3  > using virtual NVME disks on a FreeBSD 14R host? I have no problem using  > VirtIO block devices or if I downgrade to an 8.9 variant. However, using  > NVME with a 9.3 variant will cause the installer to consistently fail on  > my system. I'm not sure how to gather more data at this point as the  > installer displays a wall of text that I'm not sure how to extract. I've  > already burned a few hours on this so I thought I'd reach out to the  > list to see if anyone else has already dug into this. If not, I'll see  > what I can do to gather more info. For reference, here are the ISOs I've  > tried installing from so far ... >  > AlmaLinux-8.9-x86_64-boot.iso > AlmaLinux-9.3-x86_64-boot.iso > Rocky-9.3-x86_64-minimal.iso >  > Thanks, >  > -Matthew >  > From nobody Mon Feb 12 04:10:05 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TY9wv2CJZz59Py3 for ; Mon, 12 Feb 2024 04:10:39 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from wfhigh4-smtp.messagingengine.com (wfhigh4-smtp.messagingengine.com [64.147.123.155]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TY9wt35Spz529b for ; Mon, 12 Feb 2024 04:10:38 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.west.internal (Postfix) with ESMTP id 4A3461800086; Sun, 11 Feb 2024 23:10:36 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute7.internal (MEProxy); Sun, 11 Feb 2024 23:10:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1707711035; x=1707797435; bh=O981ebh/Zg cLEj514crP1+BGzlT1GRewEisDLhIpaSM=; b=bUiU63FECuKPYDigxsqxTHj1+0 j93u5QJGkrd5YXo60Tpgq0klIlqT5nrFp1NzO/5eQ1o9/V8t6rSjVm3TcUHJBD0L YZCEAOc5u+iCFbdU3/jLSq9NKF9BtQwQJB974s+2rVCZaUB+P4hokjT0l1aZxVhU C6FxWBNDnqVX+kHE3PCtWbz+xXzZ058vaxi+0SOZJCdWnhSI34xeXsbSQL0iODNd fB0DpL9f2N0u8fo1QertPf9cHbHRFkSPcdUb4RMhr+IDWF/RFTk4dkzgf4Z4+lKK +uyvESaGor/YxVPKGxFmeTqhAom2GxRelDCvX4FK01R63FiPRQVU681MBEPQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1707711035; x=1707797435; bh=O981ebh/ZgcLEj514crP1+BGzlT1 GRewEisDLhIpaSM=; b=LcEiheWKh4m4eHaaLkKNSMeriowmYk6HqLCavvwWL9p6 Y8qsx77DoVAa8tC0c+/lVzIgKLCD9DtvKDBeEl2zafdf6BfhERaV/l22+WbdUs92 w8uo95YBFOtK5JUSjzqdHtBZGOz5C61+icbanUHjnLL/wXctljbaBT3MeC/zeEiM SFVPZGz8KocOPOYhd9H3O5gwTtlLXjkCYS652MiG+zo5BjFKFIpCsReWXXzCEkOK iihyyJtTn2hy8AOETN6Mf8QHBpYsYgRS3098u4oIO3WhbW3e8NBjRv/3iP6fgZGX TU9jcCRERjrtAclKThCaLL8Faqb6TosYvynQmFCBrg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddvgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefofgggkfgjfhffhffvufgtsegrtderreerreejnecuhfhrohhmpedfvehhuhgt khcuvfhufhhflhhifdcuoegthhhutghksehtuhhffhhlihdrnhgvtheqnecuggftrfgrth htvghrnhepveejvedttdetffekieeghfevkeejhfetuddttdeigfejgfffjeehvedttefh fedtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheptg hhuhgtkhesthhufhhflhhirdhnvght X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 493D3B6008D; Sun, 11 Feb 2024 23:10:35 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Message-Id: <6c9bd61c-a3ca-4470-895c-494297af0956@app.fastmail.com> In-Reply-To: <20240212012559.Horde.aH5FMCkNv4j7GfULz1fq5rn@drive.tubnor.net> References: <678ac1d2-d00c-450e-bc2e-f9d979d19340@shrew.net> <20240212012559.Horde.aH5FMCkNv4j7GfULz1fq5rn@drive.tubnor.net> Date: Sun, 11 Feb 2024 20:10:05 -0800 From: "Chuck Tuffli" To: "Jason Tubnor" , "FreeBSD virtualization" , "Matthew Grooms" Subject: Re: 14R bhyve RHEL9+NVME Content-Type: multipart/alternative; boundary=d7044c208f3a48baa3daca6471f24497 X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US] X-Rspamd-Queue-Id: 4TY9wt35Spz529b X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --d7044c208f3a48baa3daca6471f24497 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sun, Feb 11, 2024, at 5:25 PM, Jason Tubnor wrote: > Hi Matthew, > =20 > There is some underlying bug post 9.1 when it comes to Anaconda and th= e presented NVMe storage. I can't reproduce it with newer version of Ana= conda that is present in Fedora. > =20 > Uplifting in place from 9.1 to 9.3, there are no issues once the opera= ting system is up and running, so it isn't a RHEL thing, just the instal= ler. > =20 > Cheers, > =20 > Jason. > =20 > "Matthew Grooms" mgrooms@shrew.net =E2=80=93 12 February 2024 11:32 AM > > Hey All, > >=20 > > Has anyone had experience installing either Rocky or Alma Linux 9.3=20 > > using virtual NVME disks on a FreeBSD 14R host? I have no problem us= ing=20 > > VirtIO block devices or if I downgrade to an 8.9 variant. However, u= sing=20 > > NVME with a 9.3 variant will cause the installer to consistently fai= l on=20 > > my system. I'm not sure how to gather more data at this point as the=20 > > installer displays a wall of text that I'm not sure how to extract. = I've=20 > > already burned a few hours on this so I thought I'd reach out to the=20 > > list to see if anyone else has already dug into this. If not, I'll s= ee=20 > > what I can do to gather more info. For reference, here are the ISOs = I've=20 > > tried installing from so far ... I have heard that anaconda has a debug mode activated by some kernel boo= t option. Maybe that could shed some light.=20 >From the NVMe side, there aren=E2=80=99t any errors or =E2=80=9Cweird=E2= =80=9D commands. My guess is the emulation is returning something anacon= da doesn=E2=80=99t like, but it=E2=80=99s hard to tell from the Python e= xception what that might be.=20 Note the text/rescue installer uses tmux which gives better visibility i= nto the logs.=20 --d7044c208f3a48baa3daca6471f24497 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable

=
On Sun, Feb 11, 2024, at 5:25 PM, Jason Tubnor wrote:
=
Hi Matthew,
 
There is some underlying bug post 9.1 w= hen it comes to Anaconda and the presented NVMe storage. I can't reprodu= ce it with newer version of Anaconda that is present in Fedora.
 
Uplifting in place from 9.1 to 9.3, there are= no issues once the operating system is up and running, so it isn't a RH= EL thing, just the installer.
 
Cheers,=
 
Jason.
 
"Matthew Grooms" mgrooms@shrew.net =E2=80=93 12 February 2024 11:32 AM
> Hey All,
> 
> Has anyone = had experience installing either Rocky or Alma Linux 9.3 
=
> using virtual NVME disks on a FreeBSD 14R host? I have no prob= lem using 
> VirtIO block devices or if I downgrad= e to an 8.9 variant. However, using 
> NVME with a= 9.3 variant will cause the installer to consistently fail on 
<= /div>
> my system. I'm not sure how to gather more data at this p= oint as the 
> installer displays a wall of text t= hat I'm not sure how to extract. I've 
> already b= urned a few hours on this so I thought I'd reach out to the 
> list to see if anyone else has already dug into this. If no= t, I'll see 
> what I can do to gather more info. = For reference, here are the ISOs I've 
> tried ins= talling from so far ...

I have= heard that anaconda has a debug mode activated by some kernel boot opti= on. Maybe that could shed some light.

From= the NVMe side, there aren=E2=80=99t any errors or =E2=80=9Cweird=E2=80=9D= commands. My guess is the emulation is returning something anaconda doe= sn=E2=80=99t like, but it=E2=80=99s hard to tell from the Python excepti= on what that might be. 

Note the text/= rescue installer uses tmux which gives better visibility into the logs.&= nbsp;
--d7044c208f3a48baa3daca6471f24497-- From nobody Mon Feb 12 09:13:53 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TYJgY57kXz59P3c for ; Mon, 12 Feb 2024 09:14:33 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TYJgX2NSBz4GY2 for ; Mon, 12 Feb 2024 09:14:32 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=fCaRkvqd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::62c as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a28a6cef709so389418666b.1 for ; Mon, 12 Feb 2024 01:14:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707729271; x=1708334071; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=acIZ9XA9Jvv9Ug4Xrn49LHb1v9nYDYtYhvXcxBOdaEI=; b=fCaRkvqdCAMi4TGguYH5hktS+oszVj+HV89a5EGJwfaR0OCLf4jd1C9Rmvt7ZHDTLT vR3oA8PW0fCgdnF5C+dso3e+iS1iogMnZOdhtm4cjdhUi0ilbxx82pQSzOdIDLnKO9SJ kVj53aKWUi6cR4zEzrikel0umo6ZTsgzMNINtAgZra5QPGf0gZF3aHevv5nPMtQ+85a5 ohUjxHw1mdLB84F/OxIyRmQdvm799DhhT2mReDq/tnW7jcG7AXv/srZFtMUbW7o+EiBr WqCxGVFAwzgOmeGEadQB05JCrxXD1T4B9liIF/oCplocDeD5rS2Kh66KAlDiXmXNpYde RokQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707729271; x=1708334071; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=acIZ9XA9Jvv9Ug4Xrn49LHb1v9nYDYtYhvXcxBOdaEI=; b=S2L+9RHvQFKjDQKGpT8GiBdh+mq146lQSJLVegLyX1Yr0fGIiPYayjZPby6rz0otCi 9gTgBJS5veS5C9H9R6teU/yXcLrSwMtKt15IAVoBqhvZkuo6wdKkrZbmAIDxU8arPSUy aCXe/knvZ3gRv3lbW9+G2jc3xmtFq0XiOrq6braBn7stsSUxDhbSmy/9jdDW2JwTy/Fk SxcBzFzz5e2pKHmoisRIF1CkoiPzIPJvQGCBzD7J3M6uGQnkgARJdPaWPHH17dP4R+4J mMOt85FdL98EYg6jwHzTUiFJFaFJdBjvTLvTjeRajmEEATMIjOGfQtGhmOpTwxXyb6kl zYOA== X-Gm-Message-State: AOJu0Yw6XVdlFW0p/krRu/D1sodb+s8oHUHi/Fw/4nJC71vZFTBhF5Gz AtiW6GEcX6Gyh4ary+j1ORhLxZEKSYUEBKw4+Trl+Hmg3sBrc0mmSryzBwYo6p3pUM8XGK1Ofgl 7uq8CSBpwIVUPu7v6bey0Op4Y4TC1oJYvXtwMfg== X-Google-Smtp-Source: AGHT+IF1gBgI9HfvEk+YmaEzKtJItfzK34Bf4wZzpHs+SRWff1Odv76WM91KA6yOFVz7+TCInqJU6835vrpQeL4eJHY= X-Received: by 2002:a17:906:b218:b0:a3c:8f4c:b1d with SMTP id p24-20020a170906b21800b00a3c8f4c0b1dmr1912874ejz.63.1707729270396; Mon, 12 Feb 2024 01:14:30 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 12 Feb 2024 10:13:53 +0100 Message-ID: Subject: Re: OpenBSD vm troubles To: Odhiambo Washington Cc: FreeBSD virtualization Content-Type: multipart/alternative; boundary="00000000000088dc3306112bb4ba" X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62c:from] X-Rspamd-Queue-Id: 4TYJgX2NSBz4GY2 --00000000000088dc3306112bb4ba Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Try this : bhyve -S -c sockets=3D4,cores=3D2,threads=3D2 -m 8G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-hd,/home/marietto/Desktop/Files/OS/BSD/ISO/install74.img,bootindex= =3D1 \ -s 2,ahci-hd,/mnt/zroot2/zroot2/OS/BSD/openbsd74.img \ -s 3,fbuf,tcp=3D0.0.0.0:5915,w=3D1600,h=3D950 \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm15 < /dev/null & sleep 2 && vncviewer 0:15 the installer starts,but you will not have the network interface. Unfortunately,when I add this parameter : -s 10,virtio-net,tap15 \ It produced this error : Assertion failed: (error =3D=3D 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 637. probably a bug with the virtio-driver and bhyve on openbsd. On Sun, Feb 11, 2024 at 7:59=E2=80=AFAM Odhiambo Washington wrote: > I am not sure what it is that I am doing wrong, but I am unable to instal= l > OpenBSD. > > Here is how I am invoking the installer: > ``` > bhyve -A -D -H -P -S -u -w -c 8 -m 4G \ > -s 0,hostbridge \ > -s 3,ahci-hd,/usr/local/bhyve-vms/.iso/miniroot74.img \ > -s 4,virtio-blk,/usr/local/bhyve-vms/OpenBSD/openbsd74.img \ > -s 5,virtio-net,tap1 \ > -s 8,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 20,virtio-rnd \ > -s 29,fbuf,tcp=3D0.0.0.0:5901,w=3D1680,h=3D1050,wait \ > -s 30,xhci,tablet \ > -s 31,lpc -l com1,stdio \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > openbsd74 > ``` > > I end up with: > ``` > probing: pc0 com0 com1 mem[640K 3049M 128K 16M 3M 1024M] > disk: hd0 hd1* > >> OpenBSD/amd64 BOOTX64 3.65 > boot> > cannot open hd0a:/etc/random.seed: No such file or directory > booting hd0a:/bsd: 17163596read text > failed(0). will try /bsd > boot> > cannot open hd0a:/etc/random.seed: No such file or directory > booting hd0a:/bsd: 17163596read text > failed(0). will try /bsd > Turning timeout off. > boot> > ``` > Not sure what I need to do for the installer to run. > > > > -- > Best regards, > Odhiambo WASHINGTON, > Nairobi,KE > +254 7 3200 0004/+254 7 2274 3223 > In an Internet failure case, the #1 suspect is a constant: DNS. > "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) > [How to ask smart questions: > http://www.catb.org/~esr/faqs/smart-questions.html] > --=20 Mario. --00000000000088dc3306112bb4ba Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Try this :

bhyve -S -c sockets=3D4,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/home/marietto/Desktop/Files/OS/BSD/ISO/install74.img,boot= index=3D1 \
-s 2,ahci-hd,/mnt/zroot2/zroot2/OS/BSD/openbsd74.img \
-s 3,fbuf,tcp=3D0.0.0.0:5915,w=3D16= 00,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm15 < /dev/null & sleep 2 && vncviewer 0:15

the installer starts,but you will not have the= network interface. Unfortunately,when I add this parameter :

-s 10,virtio-net,ta= p15 \
It produced this error :

Assertion failed: (error =3D=3D 0), fu= nction modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emu= l.c, line 637.

probably a bug with the virtio-dri= ver and bhyve on openbsd.

On Sun, Feb 11, 2024 at 7:59=E2= =80=AFAM Odhiambo Washington <odhi= ambo@gmail.com> wrote:
I am not sure what it is that I am doing wro= ng, but I am unable to install OpenBSD.

Here is how I am= invoking the installer:
```
bhyve -A -D -H -P -S -u -w= -c 8 -m 4G \
-s 0,hostbridge \
-s 3,ahci-hd,/usr/local/bhyve-vms/.is= o/miniroot74.img \
-s 4,virtio-blk,/usr/local/bhyve-vms/OpenBSD/openbsd7= 4.img \
-s 5,virtio-net,tap1 \
-s 8,hda,play=3D/dev/dsp,rec=3D/dev/ds= p \
-s 20,virtio-rnd \
-s 29,fbuf,tcp=3D0.0.0.0:5901,w=3D1680,h=3D1050,wait \
-s 30,xhc= i,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi= -firmware/BHYVE_UEFI.fd \
openbsd74
```

I end up with:
```
probing: pc0 com0 com1 mem[640K 3= 049M 128K 16M 3M 1024M]
disk: hd0 hd1*
>> OpenBSD/amd64 BOOTX64= 3.65
boot>
cannot open hd0a:/etc/random.seed: No such file or dir= ectory
booting hd0a:/bsd: 17163596read text
=C2=A0failed(0). will try= /bsd
boot>
cannot open hd0a:/etc/random.seed: No such file or dir= ectory
booting hd0a:/bsd: 17163596read text
=C2=A0failed(0). will try= /bsd
Turning timeout off.
boot>
```
Not sure = what I need to do for the installer to run.



--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7= 3200 0004/+254 7 2274 3223
= =C2=A0In=C2=A0an Internet failur= e case, the #1 suspect is a constant: DNS.
"Oh, the cruft.&quo= t;,=C2=A0egrep -v '^$|^.*#'= =C2=A0=C2=AF\_(=E3=83=84)_/=C2=AF=C2=A0:-)
[How to ask smart questions:=C2=A0http://www.catb.org/~esr/faqs/smart-questions.html]


--
Mario.
--00000000000088dc3306112bb4ba-- From nobody Tue Feb 13 17:05:12 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZ74L13KDz59Gqt for ; Tue, 13 Feb 2024 17:05:22 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZ74K2mQpz4tcJ for ; Tue, 13 Feb 2024 17:05:21 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=ErpAX3fu; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41DH5DSD097086; Tue, 13 Feb 2024 11:05:13 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1707843913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4k7tMTDsHWyeZFYvyA5HAx5GT3J6K5juDfnZttX9bBs=; b=ErpAX3fuGTXJwCnUsj6tzj8U/bQZ+LwAb04uNK9jnjv0fhksT/sOsvjhOFZ8FUvqaPyWO+ qjgsATVrUwDZqbauI37QYsgyP7r+GRSLmRgJ6DuNhDxJPBYZLfomJs9dzaEO5kVyfWhyuj 8HDL+Ydp5T4ZxO12Z/wsisXel1HOpjb7V6UzTeMyqyX7z0/7qMdx6ghLc3Yb0rO1yVE0T2 8u5q7gAaNHbr+fPcaVCDPEtGDBhY1ASbGfA/4puePqzMVBeBexoY+zvyQr3UIMXn8YPIfv ZSLcnKtb5Pwm8G83+uzcw2V7Q2hbXFylLH2ww3fUMyTJlTARQkomi/V4w0ruZg== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 7641F3B578; Tue, 13 Feb 2024 11:05:13 -0600 (CST) Message-ID: <0e7fb763-43e1-4b79-8e1e-ee97e664ac29@shrew.net> Date: Tue, 13 Feb 2024 11:05:12 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 14R bhyve RHEL9+NVME Content-Language: en-US To: Jason Tubnor , FreeBSD virtualization References: <678ac1d2-d00c-450e-bc2e-f9d979d19340@shrew.net> <20240212012559.Horde.aH5FMCkNv4j7GfULz1fq5rn@drive.tubnor.net> From: Matthew Grooms In-Reply-To: <20240212012559.Horde.aH5FMCkNv4j7GfULz1fq5rn@drive.tubnor.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; DMARC_NA(0.00)[shrew.net]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TZ74K2mQpz4tcJ On 2/11/24 19:25, Jason Tubnor wrote: > Hi Matthew, > > There is some underlying bug post 9.1 when it comes to Anaconda and the presented NVMe storage. I can't reproduce it with newer version of Anaconda that is present in Fedora. > > Uplifting in place from 9.1 to 9.3, there are no issues once the operating system is up and running, so it isn't a RHEL thing, just the installer. > > Cheers, > > Jason. > > "Matthew Grooms" mgrooms@shrew.net – 12 February 2024 11:32 AM >> Hey All, >> >> Has anyone had experience installing either Rocky or Alma Linux 9.3 >> using virtual NVME disks on a FreeBSD 14R host? I have no problem using >> VirtIO block devices or if I downgrade to an 8.9 variant. However, using >> NVME with a 9.3 variant will cause the installer to consistently fail on >> my system. I'm not sure how to gather more data at this point as the >> installer displays a wall of text that I'm not sure how to extract. I've >> already burned a few hours on this so I thought I'd reach out to the >> list to see if anyone else has already dug into this. If not, I'll see >> what I can do to gather more info. For reference, here are the ISOs I've >> tried installing from so far ... >> >> AlmaLinux-8.9-x86_64-boot.iso >> AlmaLinux-9.3-x86_64-boot.iso >> Rocky-9.3-x86_64-minimal.iso >> >> Thanks, >> >> -Matthew Thanks for the info. I was able to install with 9.1 and upgrade to 9.3 without an issue. -Matthew From nobody Wed Feb 14 12:18:49 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZch16lDKz59fw4 for ; Wed, 14 Feb 2024 12:19:29 +0000 (UTC) (envelope-from odhiambo@gmail.com) Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZch11WFwz4Xxm for ; Wed, 14 Feb 2024 12:19:29 +0000 (UTC) (envelope-from odhiambo@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="hjEBJCV/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of odhiambo@gmail.com designates 2a00:1450:4864:20::22d as permitted sender) smtp.mailfrom=odhiambo@gmail.com Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2d0f1ec376bso41952231fa.1 for ; Wed, 14 Feb 2024 04:19:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707913166; x=1708517966; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=ewyuBTiXrLJUhjBf8aYyLYTKR01y6d50YhJs8gOjTwU=; b=hjEBJCV/5Z7VNoUzbhOLB+uaPz1l2SSofs2tR+SvVQZPqHHguzAzQBzR2SrGa78Ec/ Wh6tpuGEiPbrvdhWiuOfmbItGfS+9Qiok/VVkqI9XBwBFpSh2Yh6iP5h09g7v+wjF6LT H6+K7HT9pU8HeuvSr5Lo7zZetBQCm0ox+khic1RArsSVFe1Nr1rMyLMcJFZTRb7D9uTG PoVIeAUoHdLAbdVJFh4d/VAO6Z738QtpGk8/VgmNolNfVsNMvXhGxyZ+C2Sxa0rC003Q PQ3C+ZXKbH7jMwA/ULZUcNDP6M2yrIG0F1fAo/syig9GA/FPkIxXEHAYP73HokVyDxgW TeUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913166; x=1708517966; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ewyuBTiXrLJUhjBf8aYyLYTKR01y6d50YhJs8gOjTwU=; b=s+6HyCHVNQev+JMRD0kfBfFMgaG5YpQR3vGWtvo2ASpd6GEMyq+o8J9313XdOydlbt 4WWYyzlBa6GdKqThl5tNJMjxPaPDROiUZPzYUrr8Qw6p4yi5gKEuGAgBDDbCWZp5cQvz 4MZRsTRPQyLt9s4sOVBG0cLfgMP6rIYD3DAaRPT2ymfEuZCRJequ5q1FwuSPvpvMFlpo wv0vxeP7jDymz1GQ10JR0kTOKHx5JkIpV0SC62BUS+OfoDxA1ybr5ulNbc4k+sywgNtv LaTDbWzAAFn4b5Dw0qX8TtPXN+3n1U8k+T0tapsDYDetrQ+0Zc/oWwKyMqSq724f0EHE +36Q== X-Gm-Message-State: AOJu0YyqnAWc1cS6sZfa1tDtfkpIGc8f5GeAO/4P9uMhPi7x61CXqKSv A/UxIAIgEaQr19H8TEF5R787hceUROXw74q2QxujA9SuFkLA1QDS5pwG0WfWoR4fCs2rEobOdf2 GfzYS6VpL7gRrRevKGRXKacv8k/I0zvDe+l6wQ4NI X-Google-Smtp-Source: AGHT+IH5vENz2Uxfzvy03CngZfKsERvFgNnVgA/HDFMA8jJLcWLGZaNO1DkR/Cz14Yd72PHmgToibgvl7mrPez1cVc8= X-Received: by 2002:a2e:b00c:0:b0:2d0:f62b:cdca with SMTP id y12-20020a2eb00c000000b002d0f62bcdcamr2076172ljk.1.1707913166245; Wed, 14 Feb 2024 04:19:26 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Odhiambo Washington Date: Wed, 14 Feb 2024 15:18:49 +0300 Message-ID: Subject: VNC Viewer on Windows: Unable to access VMs GUI To: FreeBSD virtualization Content-Type: multipart/alternative; boundary="00000000000094d37d06115685ba" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22d:from] X-Rspamd-Queue-Id: 4TZch11WFwz4Xxm --00000000000094d37d06115685ba Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hellos, When I was still on FreeBSD 13.2-RELEASE I was able to access some Linux VMs GUI using VNC Viewer on my Windows PC. After I upgraded to 14-RELEASE, this changed: I can connect to the VMs, but the GUI desktop is frozen. The mouse cursor isn't moving at all and the Desktop doesn't load like before. I start all my VMs the same way: bhyve -c N -m NG -w -H \ -s 0,hostbridge \ -s 4,virtio-blk,/usr/local/bhyve-vms/DISTRONAME/distroname.img \ -s 5,virtio-net,tapN \ -s 8,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ -s 29,fbuf,tcp=3D0.0.0.0:59XX,w=3D1680,h=3D1050 \ -s 30,xhci,tablet \ -s 31,lpc -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ distroname I would then connect to a.b.c.d:59xx using VNC Viewer and I'd get the Linux/Windows Desktop. Not anymore! Is there something I need to do to fix the issue? --=20 Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] --00000000000094d37d06115685ba Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hellos,

When I was still on FreeBSD 13.= 2-RELEASE I was able to access some Linux VMs GUI using=C2=A0VNC Viewer on = my Windows PC.
After I upgraded to 14-RELEASE, this changed: I ca= n connect to the VMs, but the GUI desktop is frozen. The mouse cursor isn&#= 39;t moving at all and the Desktop doesn't load like before.
= I start all my VMs the same way:

bhyve -c N -m NG = -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/usr/local/bhyve-vms/DISTRO= NAME/distroname.img \
-s 5,virtio-net,tapN \
-s 8,hda,play=3D/dev/dsp= ,rec=3D/dev/dsp \
-s 29,fbuf,tcp=3D0.0.0.0:59XX,w=3D1680,h=3D1050 \
-= s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/sh= are/uefi-firmware/BHYVE_UEFI.fd \
distroname

I = would then connect to a.b.c.d:59xx using VNC Viewer and I'd get the Lin= ux/Windows Desktop.
Not anymore!

Is there something I need to do to fix the=C2=A0issue?


--
Best regards,
Odhiambo WASHINGTON,Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
=C2=A0In=C2=A0an Internet failure case, the #1 suspect is a constant: DNS.
&= quot;Oh, the cruft.",=C2=A0egrep= -v '^$|^.*#'=C2=A0=C2=AF\_(=E3=83=84)_/= =C2=AF=C2=A0:-)
[How to ask smart questions:=C2=A0http://www.catb.org/~esr/faqs/smart-que= stions.html]
--00000000000094d37d06115685ba-- From nobody Wed Feb 14 12:47:09 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZdJh0xx6z59kn6 for ; Wed, 14 Feb 2024 12:47:48 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZdJg5qMcz4c72 for ; Wed, 14 Feb 2024 12:47:47 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a3cb228b90bso339830666b.3 for ; Wed, 14 Feb 2024 04:47:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707914866; x=1708519666; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=DEKVc/Hmkwsw1JEyY9eS8jMfKCLxqLT2LOWO0rFhqpQ=; b=JebrBs3dvuBx3b7iMMjqlBh+qZngBUw5ZpNCxSagnfebyc9xr/XscByBURkDwZ1Ijc qc4PUuQnoEOte5Cl49cPiNwqKx0XR3+q+pkqSn9Lc4LXNw0pCOPkB+wgJ3YIowQmcYfd IXU3KRrrknMmuIs3YXWHdPvs/zRjS/IaX3QcYGR1WH/ANQTALUUgpajVMOFb8ro1ThyH MAStN+9ounpgTMN5YTtxWQiOQMdqk9S4wFQNXkVx6RZ5wiiTr8qmW3GFvUkP2gFSNGth M0XX+pi+HnMzhyLu2SXT42y3Wvq+IhBQ/wq7QSTuNbjKyu+1Iaz7RoVXMrpOaaa8LI5F 3Ijg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707914866; x=1708519666; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=DEKVc/Hmkwsw1JEyY9eS8jMfKCLxqLT2LOWO0rFhqpQ=; b=crrOX5I5lmS70sEbneqJd585xB99hvXkqUu7pVCVcPKtkv8bB4K1RiyoNpgDTY3hY/ 4VCxXx4zFP+uM3yuOY5fT5aJxV7T9tLqyGPZQV7TodZiX1CC+oFhb0NWcqSVbsTvODDC zDQiGLWX5bXvclC5Bpr9cQghfI6v3j9dLj2lLzVqU0r8iF+4rAyC8svyMDCBqGJF5T6V hBmS9Xh0HtUTgHCSK9nu+ocIXmbABM3ZKzTl3o5QflkfxZ3PlJKc1kM1ybog5iFpS7/f m95V3iIWDpk705vvVjF3xL5viC8tL03EYGOOp37chyfkARdBTGIMH2QkCR9lD06AgsLf 7gPg== X-Gm-Message-State: AOJu0YxJTkkIcCpkbGlJZPATO5vSZmYF8NOxjnnBee7vjvk4tjPik7c0 3wWKLgww7jedP0dNs59uGABAN1WxWlZeNVZYxqWWauoZtSmH2zTgRqT5Jhn8qPKDtvw8feHD3po g697r/M9EZQJcXRnDTu92daCE3oo= X-Google-Smtp-Source: AGHT+IHGeomfAcDXRkrpcCQj1hmEpe/AAla+HhHGkPnPiuDlHWKejy/qxXnyJtUzf0uuXgra4bVVbfV06kUENN33nOU= X-Received: by 2002:a17:906:f299:b0:a3c:fcb5:fbb7 with SMTP id gu25-20020a170906f29900b00a3cfcb5fbb7mr1704808ejb.28.1707914865966; Wed, 14 Feb 2024 04:47:45 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Wed, 14 Feb 2024 13:47:09 +0100 Message-ID: Subject: Re: VNC Viewer on Windows: Unable to access VMs GUI To: Odhiambo Washington Cc: FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000e4851c061156ea28" X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TZdJg5qMcz4c72 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --000000000000e4851c061156ea28 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable -s 3,fbuf,tcp=3D0.0.0.0:59 *15*,w=3D1600,h=3D950 \ -s 10,virtio-net,tap*15* \ *vm15* < /dev/null & sleep 2 && vncviewer *0:15* On Wed, Feb 14, 2024 at 1:19=E2=80=AFPM Odhiambo Washington wrote: > Hellos, > > When I was still on FreeBSD 13.2-RELEASE I was able to access some Linux > VMs GUI using VNC Viewer on my Windows PC. > After I upgraded to 14-RELEASE, this changed: I can connect to the VMs, > but the GUI desktop is frozen. The mouse cursor isn't moving at all and t= he > Desktop doesn't load like before. > I start all my VMs the same way: > > bhyve -c N -m NG -w -H \ > -s 0,hostbridge \ > -s 4,virtio-blk,/usr/local/bhyve-vms/DISTRONAME/distroname.img \ > -s 5,virtio-net,tapN \ > -s 8,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 29,fbuf,tcp=3D0.0.0.0:59XX,w=3D1680,h=3D1050 \ > -s 30,xhci,tablet \ > -s 31,lpc -l com1,stdio \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > distroname > > I would then connect to a.b.c.d:59xx using VNC Viewer and I'd get the > Linux/Windows Desktop. > Not anymore! > > Is there something I need to do to fix the issue? > > > -- > Best regards, > Odhiambo WASHINGTON, > Nairobi,KE > +254 7 3200 0004/+254 7 2274 3223 > In an Internet failure case, the #1 suspect is a constant: DNS. > "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) > [How to ask smart questions: > http://www.catb.org/~esr/faqs/smart-questions.html] > --=20 Mario. --000000000000e4851c061156ea28 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
-s 3,fbuf,tcp= =3D0.0.0.0:5915,w=3D1600,h=3D950 \
-s= 10,virtio-net,tap15 \
vm15 < /dev/null & sleep 2 && vncviewer <= b>0:15

On Wed, Feb 14, 2024 at 1:19=E2=80=AFPM Odhiambo Washingto= n <odhiambo@gmai= l.com> wrote:
Hellos,

When I was still on FreeBS= D 13.2-RELEASE I was able to access some Linux VMs GUI using=C2=A0VNC Viewe= r on my Windows PC.
After I upgraded to 14-RELEASE, this changed:= I can connect to the VMs, but the GUI desktop is frozen. The mouse cursor = isn't moving at all and the Desktop doesn't load like before.
=
I start all my VMs the same way:

bhyve -c N -= m NG -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/usr/local/bhyve-vms/D= ISTRONAME/distroname.img \
-s 5,virtio-net,tapN \
-s 8,hda,play=3D/de= v/dsp,rec=3D/dev/dsp \
-s 29,fbuf,tcp=3D0.0.0.0:59XX,w=3D1680,h=3D1050 \=
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/loc= al/share/uefi-firmware/BHYVE_UEFI.fd \
distroname

I would then connect to a.b.c.d:59xx using VNC Viewer and I'd get th= e Linux/Windows Desktop.
Not anymore!

Is there something I need to do to fix the=C2=A0issue?

=

--
<= div dir=3D"ltr" class=3D"gmail_signature">
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 00= 04/+254 7 2274 3223
=C2=A0In= =C2=A0an Internet failure case, = the #1 suspect is a constant: DNS.
"Oh, the cruft.",=C2= =A0egrep -v '^$|^.*#'=C2=A0= =C2=AF\_(=E3=83=84)_/=C2=AF=C2=A0:-)
[How to ask smart questions:=C2=A0http://www.catb.org/~esr/faqs/smart-questions.html]


--
Mario.
--000000000000e4851c061156ea28-- From nobody Wed Feb 14 12:59:45 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZdZX2XCxz59mDx for ; Wed, 14 Feb 2024 12:59:48 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZdZX1zyNz4ch6; Wed, 14 Feb 2024 12:59:48 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707915588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=fRDfSLctFZGN0E3UCHLoW3wIoYLosZNO8Y2yRlYQSuE=; b=YYvVSrfTGKE256pgynYBtljnFdH3+n7IM0pJNe+I4HJWIPt0BCYnIbX7+SWtUIe0sRnTf/ uSQ/155kO4LVazLsuiqLV2bq6QMbFr4UcERjLHvOsOBiiCYUD0jYbEIssmfBwxuLcyzbmO vVYSLsdARt1W/2zBnwdXCYsa3kOstRWHNrp9pOcvujFazNLH1p9Uh7McWWBkFiGa3lfTCE m5FGuC8/HYSmEXBMSen4S9RcTQn+LlV8iNfPm2MAK2OG2YyAoJXURD+NazKzBWPbLT/Kp8 RqCMhe6QmTDqZ+p1F9rEABt2N2IZ9DnO9ln3rNk0LFJA7KDucnYR256ec9SHdQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707915588; a=rsa-sha256; cv=none; b=vj+js473jR6DofJgguUNJkwj3VzCcBjq9F3QYz7BYy703jBYdzoXLsvqGHjK2PEx0wSQFF K4qFUH+Saxyuy84AnvtWg/Ip5aTxBvOTVCXhY0T3FL43B6OdaGxfQgMs4Nolmo3aw2GekW C8sVVL+LfIiNG0IqeDYNRJFGcHxM7s2eJNYr2lorpWGmuiCN8EzM0NrclMEa3WFIxud0Z6 Diqg8H2g/dUiOhuAPdE7uxWrtq1m/QiN/+EZ3fulmdQc2j3fP8WDhY3CVZ97c4/Mq2RANG zs+AnamEZBKfS732bT4xk7wVcfDb0rZ6NtWwjpJsPGvYfoOrIRIUbKYT4hkvqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707915588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=fRDfSLctFZGN0E3UCHLoW3wIoYLosZNO8Y2yRlYQSuE=; b=hp6KEohF+nAL7SOg6HXSpwKLh5uO+LkblKb/54TWkV4/eQO1NKWHioXm/fMr7WcmQTBxDl 7Dka7HW3r0nbs2hs0ldDgdWyrSXi8s8j6l+Nbdb3CdWqpnN8rIecEyyKwdAdYOafAvMHnn IbwQgqfBK9T8CkF+JrcncQWDHWvFaK53PbrxYX/uT6sZXm2sFSzx17gQjT1sH+usfh43xX WsZkotB/5uouUDWHce9eiOVXNnhihc6I3ft20oKiWq9GaaKVyZUTBUHDw98cmMBMRqWM+1 Fm6trbzE3UuRj7E3wu46lS1GCbhnD8o/VulnH3Yn7/TL6/nsG2BEXy4EICYUgg== Received: from [IPv6:2001:9e8:dc1c:c100:c0cb:55f1:f8e9:7b1e] (unknown [IPv6:2001:9e8:dc1c:c100:c0cb:55f1:f8e9:7b1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TZdZW5RlWzLYV; Wed, 14 Feb 2024 12:59:47 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: Subject: Re: VNC Viewer on Windows: Unable to access VMs GUI From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Odhiambo Washington , FreeBSD virtualization Date: Wed, 14 Feb 2024 13:59:45 +0100 In-Reply-To: References: Autocrypt: addr=corvink@FreeBSD.org; prefer-encrypt=mutual; keydata=mQINBGNjZaIBEADDTrDNf+0pwiuRPBdClcnZW83dH1UhuOi0u+A1J2SatEBbNaFVtXXAavewCTuyV/ZbNidjlhq3R/pWyiKjFKvs5dj7PMCw+3z2D5OWpMdHg7TrB+fbdFPOEsu0zQVKNaO+pSKCfN0Re0m7bL3wuvl7PXvBufRwA3Guo1P4j3TXWaEkuso7VupTvE25zVGg9ONHrGOjA9RUy+Yg4Se3NLgtUdjBgA21SBQTDvRQV4fDmVenlwvWeE0Xm8FcDcpQb6sJTihaDku78mi3Ux1HCk7rTcepVEB0xIB6qmFxv0sLlDmVv6Z6qg1y/Q5m23Pgz60o3TulMPV4F+3Itm8ifU+wgVSzBZbD29GYkd7LKqMkFbhvfSBk+5db3vbYY5OD//+LTM5AV7e2AhXuXMvG1UNBqXqSJTTSy6KZz+qmPQO0zos0dq46p8o82lKiBEGD2Hu0p+u0OyV+MmRYo1NIBFVbOPXp2MvUVl5II0UIJ3+N9gLBmfGA+HEpVO8PnvdoT/5NQ7m8JK1rQHzjiDub/iDPAYMqKH4C0eZ/7zO0fuY5FeRNtuNtpH1Bw/+7/5RJH7bcKkfGHHEp15FJUrGHgWNydoDLB9QBprwQc8FEldDXBjzOMXIgh6FGKLNu6DswvIPGy6M3u7DXwDakCXz+c9Ym0oFihLzZxWntrsxdswD/CwARAQABtCdDb3J2aW4gS8O2aG5lIDxjb3J2aW4ua29laG5lQGdtYWlsLmNvbT6JAlQEEwEIAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCZArUIwIbAwUJCWYBgAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRDYVNpWMV4CalcID/44k2i/mqSSi4W6FAobSF1nFLtP/pfcRNJriWKx2UF7cfFMKyg7Nilg7FhLb5FDB1umUW2nFfchFPTUp4FfKzgRvPzIMg4RIRcVtTpYbl3z7zs9ZXD8q S//iibbiUG3quncm6tO2x1jLZD3ORC+8MuLGXhYQIa4O5vVF2SBHdb/U6P+wsrF+U+OpRdEdQ/4Xu9S02kltzBGgArjcexdhUqEqW01KCCSH0+qgfN1NE+9L934ZOB+cai2b9apPbCOGuV6KcUKMj4z0RWInhlXIyMqtmhdix/P/GjrQ1REVNdp74JeweSSedM15wwc6YLMXPrtOnnExyZ1gyNFYaub+Mdo3ZQ+83863B0C9IlpzEW0K8bYlZrl4WBNdcMOyByETAxgQmBgP6ZFErTtaeUOH1nX2FyR6o2GLSahRgngQmnREzScTpPbBEkMwJMDAO+rbjjGxjeKSUwD1WOfbI6QZj+MS/uBk2p08kgN7fQaCEwj6jqML/IIE+FZ9IIn6TNG0hChD384VHO+YioLBno1Atgi4Q7JUWSRIHQXZW+StQajFrWPPyKQwCe1MwqpKoMpX/q8IZBlzwJgZS8ShLeFZjtzOt1jgM99TD91Neonf9OzjTSbfo7sJviSWoICMhB/MvDZjj+naMVF86uGFxnIEsjVKyBxlJd4TRHnLYPTIHFKBLQjQ29ydmluIEvDtmhuZSA8Y29ydmlua0BGcmVlQlNELm9yZz6JAlQEEwEKAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCY2NlogIbAwUJCWYBgAULCQgHAwUVCgkICwUWAwIBAAIeAQIXgAAKCRDYVNpWMV4CavfDEACCFnXpR7H9eOgP+GJMNPtK6i9/xnqdyXi8uCZIN0hYwjN4Xzo9SMLOf4UUlQEveOB+bGqbRfHd/fGKnrlXiPd0SGpKWJC21gqL/DsIH0J8I3Whth+O8tfPWeFy0oCsvBaaGFLIrDfoIgHF9i/gqEe48xhN42weB02Z3mdR1L0d7ME/BLwS0mCXe9Zh3uHw63S6xYB3Wsjptxe/ph6TpQDUKWtRJkjC6BqXPBdThpbbfIRWmjZbp2fKEJPvtRXS14+gbUqWeJ 4xCvprA3+ae7vtrp91X775yngyW3XTw5cmDiJIjykH8+zhEIoNQXNBpFrehkQDYrcM+WoE6NGSJo+3VJvSRWhUGWDVrxdTYNkbIjmTNlkI12NINC007DiuV7OF9XHWgrbbylvuZvbODmbJRdhTFy9upAUygX1/xUAQEIMqMiJmyTdv8i0IbZ611WElQx9XHgGeZgM6+39/laN8FwspM6gE/4NzZHIZN0LEBOerZqoF+Il6eccQpoEWx7nb/RilJp3dUUyvkBnJWg+AJByosg857kvvmDnZ3UB+bejWpcfFvnbkiKPUBPDO7tWPb5r2yFDpDe/Vg5x+sRbkkXGUFD6Rx0p/ZiRIneVVg8emTzhDR0IL/BZkq/uVJkdaphZli1F/31cNgf6ZJYnjxlk86uiYSySZQR0dKLDqq7QlQ29ydmluIEvDtmhuZSA8Yy5rb2VobmVAYmVja2hvZmYuY29tPokCVAQTAQgAPhYhBIL0UpWt5trfx9lPRthU2lYxXgJqBQJkCtScAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJENhU2lYxXgJqI+QQAML5PTR7KpUFV3SLG60LQJGEOHUfDmJYczxBFbIAq1U4hIbivopu1AdLty7oDDrIjCVoa2/Cy34dd99O7lhLvUmZFB/zDSUtbUg2zhDkU0YSZ11FdrlWzky2tFaQRgxpDvWlUP0baa3Pd4dPDRiIUI6AOSR2SL6XANk6sJh56gLVM6G8yyafGsxSyDYg6Z78EEMFejHwB+KP2DdsahupzM+F97HeC1+bOHYxtqN+2hEkPLtQWizyumPqNg5FvZhwe7yO8V95hF3RhuDO+9aJT+WLLvcZEb/L1bI04IvZ5FWgCLI7Levd/DuOtZI8gWapHhqGZRbXB2fuJkCoKCl6V67h/7aWhU3LjFTsC5siJyrxPjapKcIk8a7PqZDswNCKR+24LJ5D59mPgEOnsiCCVpik1WE /kgD+rOu9dQxjpjKwuKowf4EJP1KYNkYtoy1HthzyTOqPwqXC3IUl0GVPO3xw6MhUM3irCVFruC+ecVrv85Rd37vrduT6JvgW92xjWegMsamtkDZH8Ik/cmYhH0K/qEc6OZVNea/4PTEZe0uxODJ6pbMd80AJyGqDPPVeAgWJtEIG4k6IS8XyD5v1QJtlpDVpPwP/bbFnVc3h3Oatfn8Etm1KAqYvNwyO+om2PkF1p732uapDZdLwksVmgc9s79+9pSpeP5DbIeMzhrIKuQINBGNjZaIBEAC4V8zlnLa957NAFPmOuW6cL1W1/E3pMtoxNYMaZmOtEDaOLV645qfie2XXh2Bn44hzN3vZ3ZaWV9FKipGTxCTNL6Im6o8ghKX4cIBiACeSbcAcIdsxCGnFLO46lPm7NYbGGfU532A8QfvpYeO4ue8H+qNWw9lWXCU1djoPwbo9McfyJ7CA3reT9wgPO4/nAo1StfeiYvkOWoxYwpiNstzUZMmd6dRCJhDtHyy639VB2YsvhyLYVB9yQdv5M2VPk2q+oodiTK/uZvaoubsIqkVlL/fqBdx+bZOG6eSogqTjTLFN5S6EjL4usCY1Vv19uDhWwuvADuMChu3jPNm4PC8pI6O4DPiWAqt+Aw4WDfKM2ie8JqzCtUXf/Iv+aSiMhNMT0qGn+Ybq98yWXs1k67M7PheurWO2hfYtMQJtpHYHqz3T7VC0F4bAPl3rDRL4PJ2Vr9eoo5upVPbZN1JXAA5oEX7coA1BQz/18LlTBhNmHk2wsi5omYZOnBoZelA7kpNx/8zc2zanOnO7NW0dJLq/o4GlfP56UFV8I1MWNyI351BAkIJyThrjv7aMxLhpNny6uYoms7X2oWf2R//QIMA/0jkqsGirksV4CW+7xhuQVwxGIHR2JskZYaPSjJaXTvoGxu/+SwqT00xnF64ZvwDUGiw7yB70s/LolEOZ/5JqgQARAQABiQI8BBgBCgAmFiEEgvRS la3m2t/H2U9G2FTaVjFeAmoFAmNjZaICGwwFCQlmAYAACgkQ2FTaVjFeAmrhSw/+NqYqv6oHppWZ7hpt+2Df+qIw2kOgvo0ecU8orastt7OfiJpRzlDFPK2nhok5t4+1PZCi4jcR5Ub22Ddy4O00FOCRAq70haA+cNNiZ0XlD5cDv+CxmT0NkD337ls5wz8zyOX7n7Z4jG8ghiJEkcLQbyp2qYaggKrz2sGWKUjByS7jySRCotPOO+0W6Iz2dw9215ZQ1F3uZwdRlpXA7ypzUBEvIJxc563fFuPetbZAIavGMT844hovsMXW2Q/MS3HI03USkgeVaqANbSOUFAdt2tgTrvUw/vXBckp4T+vATNdQH0WieBIX4nRQQ6SjfmaI82QxJuJjb5NJ6bgS+HPIUH8J1Iw958y/Rs5svzKW+/YYoZBDuhtbAeoJdiy7a7wtK6pBv+xMdpmKLBgVVXOeX6sucwJ/K68tb5aOmbuPLAaIoKgm/9IF8dqiI23JoM2ZhpYZjpVMpakGIxV6R4Kp2hrqC4oNpuVLJ7LQSMuocXduguvWYdYvVXSdpC1Ed+fLtFXA0h5fhDqHFDCDM2CgQX2DfzXe/rV7vChm61fQYn+85md1vPiefnsaunEh3+cbcFfIshNmIRfAKwA2//75eqgBuC2D5ZIr93LsEUCcabnQZdaZKGSo/ruWNZdPyjj/b6MhAifkoFFrkN4/dwIqYIev8wMbh3+7dcZIRza0foM= Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-6KtyKHRMF/9zEbadV+xC" User-Agent: Evolution 3.50.3 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 --=-6KtyKHRMF/9zEbadV+xC Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2024-02-14 at 15:18 +0300, Odhiambo Washington wrote: > Hellos, >=20 > When I was still on FreeBSD 13.2-RELEASE I was able to access some > Linux VMs GUI using=C2=A0VNC Viewer on my Windows PC. > After I upgraded to 14-RELEASE, this changed: I can connect to the > VMs, but the GUI desktop is frozen. The mouse cursor isn't moving at > all and the Desktop doesn't load like before. > I start all my VMs the same way: >=20 > bhyve -c N -m NG -w -H \ > -s 0,hostbridge \ > -s 4,virtio-blk,/usr/local/bhyve-vms/DISTRONAME/distroname.img \ > -s 5,virtio-net,tapN \ > -s 8,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 29,fbuf,tcp=3D0.0.0.0:59XX,w=3D1680,h=3D1050 \ > -s 30,xhci,tablet \ > -s 31,lpc -l com1,stdio \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > distroname Could you please try to add the `-A` flag to your cmdline? >=20 > I would then connect to a.b.c.d:59xx using VNC Viewer and I'd get the > Linux/Windows Desktop. > Not anymore! >=20 > Is there something I need to do to fix the=C2=A0issue? >=20 >=20 > --=20 > Best regards, > Odhiambo WASHINGTON, > Nairobi,KE > +254 7 3200 0004/+254 7 2274 3223 > =C2=A0In=C2=A0an Internet failure case, the #1 suspect is a constant: DNS= . > "Oh, the cruft.",=C2=A0egrep -v '^$|^.*#'=C2=A0=C2=AF\_(=E3=83=84)_/=C2= =AF=C2=A0:-) > [How to ask smart > questions:=C2=A0http://www.catb.org/~esr/faqs/smart-questions.html] --=20 Kind regards, Corvin --=-6KtyKHRMF/9zEbadV+xC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmXMuUEACgkQ2FTaVjFe Amp+sA//bheiqAOhA3VLHnP3a17RZYDOzVUNtfSv5C0XjRNPDnDyGE89qEW4FvR4 4cqoFjTl4LasJJ7wPay2gZkWFtVEb9PrLf3LkVNdjhkmZ5WY7UG3T3XpR6YAP0s5 h96wy26LSCtzL+e3rXFyyNe6fn5uze28h1zwuUaUFvxb/uu9d5x9nIZDe8s70Hgg cEy6uJDE6DHGEfKuV/+FiE0Hz57OBd9sZb86b+Ipg1nVgp2h4LtUDgvAYw3iwkeZ W25F+vK43Ttlx7obyFYKSt9LAID0OqKpskYrrDXau615zK8x4OQFUBjI5+CcvVnP JEVw33wh6kdjWwmhcDDL8W5d0M6DbZjyjEcxZzXF3GttYfy/AAaLoEhHSo7zkHdE zZzfOYUZm6FLBxoGTcJ79Qqf93ikkKoe9GiroPk5LJ0zrJygWDjYzfg42ZNsjscg +S9iEIt9iKSl+IFmuqlWuib6Wq/eobsOSeK/6kgUdZ/dE3ER7rP7M321Lg0KAZrS HZP4ZD7VmIg7tyMLmug5EASl7mB17vEzZNqlE4p1+aKjGB8WViQt3ASvfsQAVAMt g6xY3MdJHyVzEJxNZkZMEq1JpQvTcUsZuP9Xsq7lkA5Ri4K2uRaHdDWZiUBpTFx/ 2XuR909KXxSdWW/7N2m0xNwnFcb6O6s9B2XWYUdTdhFv5+5J40k= =x9Qf -----END PGP SIGNATURE----- --=-6KtyKHRMF/9zEbadV+xC-- From nobody Wed Feb 14 15:19:38 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZhhh6X6yz5B7Td for ; Wed, 14 Feb 2024 15:20:20 +0000 (UTC) (envelope-from odhiambo@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZhhh462Xz40xH for ; Wed, 14 Feb 2024 15:20:20 +0000 (UTC) (envelope-from odhiambo@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2d0e5212559so53742431fa.0 for ; Wed, 14 Feb 2024 07:20:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707924016; x=1708528816; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=21sJAzeQ0G/x/WnerVawC7DyOQhhAIJTezq0q44hiYg=; b=iCGL5uvHtfFdD7XJZfh3TTB1jReo4yD39o7Ko7+1B+NGVVxYS7gUrUdRm1QfFenm7O Pd9+uChvwQP/LEMFdjCE8gf+Fbwgw6skWrTrVYNg2axKZAZhYpN4cgxn9Xg6Icj+jgm/ AQsBZt/4+GRGFHYCS5fxhRH9yvJUIQdpsw8FOw5FimPBPQDwpoJRtorpLAdT5FHz27vd d/+hRBLyGXs+CAE0/CdvVL1Nqy6sefswlQJC0jNP5wALCfEB2+KZstA7z7NJzbadwCfL UxVUv+Fo2Q54Hole1DcYdiDHB91DuxIHs2YSdC8K9DA1Wg8WxA4cubUoSxCgt16uL06g 1q+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707924016; x=1708528816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=21sJAzeQ0G/x/WnerVawC7DyOQhhAIJTezq0q44hiYg=; b=rnJQh2/oBLujzC8Y3r0WkUNdBQ3gQx0C8tv2ykNPSBBHS7I3rXGnxXB//D7zfJU+q+ JSPkSx08uNrxAeCX6zxd7WBcFTiceIdssZh0VgqELHdwgYonZvY8CiAVArtRRjuDoXY6 ZFHAI4BpttmoEw5SjsXJswINHRTQx9hXpitB9qfb+p1iz79gAEbJLwjHKH7ECKplBCAk VRvQ/Mi0uFYMwdA/K7wDbP9xC5+xjzYV0hjG16HtlbOtWEPeiqGMBb43nr5yu0YaeJ6M aE12NVTBw5g5kaPkfKS0JPir2hORwf9ceRSdTDUDk64Z+9ALwTJqAa8ZVFIyhPdig55r NMtw== X-Gm-Message-State: AOJu0YxQqUTOCEUdoFR3HrsacBB4UDZ6wR7q/YmaVpR5F3h7VIA3DRDj B1jQ/D6KBijCLrePzGnbfeAOj5e+wsQ5tpAi4UzFv8sGwdTVUYeQ0ETv5q1OBHd5nRpT88i+YZ2 2D3ZvHtG5osOj82kaCq7aekTrjMokpVNMT6NrvhJP X-Google-Smtp-Source: AGHT+IFR/hGc40XGd1KT9T5vKi8rAZ3y+a0YPGhb4oJbcPdXgzYqu653RJuWPVEaq3gBhHuIrg3pnewE+eKywdqBpj4= X-Received: by 2002:a2e:b741:0:b0:2d0:b003:95b7 with SMTP id k1-20020a2eb741000000b002d0b00395b7mr1987560ljo.13.1707924016039; Wed, 14 Feb 2024 07:20:16 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Odhiambo Washington Date: Wed, 14 Feb 2024 18:19:38 +0300 Message-ID: Subject: Re: VNC Viewer on Windows: Unable to access VMs GUI To: FreeBSD virtualization Cc: Mario Marietto Content-Type: multipart/alternative; boundary="000000000000478e890611590c24" X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TZhhh462Xz40xH X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --000000000000478e890611590c24 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 14, 2024 at 3:47=E2=80=AFPM Mario Marietto wrote: > -s 3,fbuf,tcp=3D0.0.0.0:59 *15*,w=3D1600,h=3D950 \ > -s 10,virtio-net,tap*15* \ > *vm15* < /dev/null & sleep 2 && vncviewer *0:15* > PS: Sorry, I have to post at the bottom always. --=20 Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] --000000000000478e890611590c24 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Feb 14, 2024 at 3:47=E2=80=AFPM Mario Mari= etto <marietto2008@gmail.com> wrote:
-s 10= ,virtio-net,tap15 \
vm15 < /dev/null & sleep 2 && vncviewer 0= :15

PS: = Sorry, I have to post at the bottom always.

--
Best regards,<= br>Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223<= /div>
=C2=A0In=C2=A0an Internet failure case, the #1 suspect is a c= onstant: DNS.
"Oh, the cruf= t.",=C2=A0egrep -v '^$|^.*#'=C2=A0=C2=AF\_(=E3=83=84)_/=C2=AF=C2= =A0:-)
[How to ask smart = questions:=C2=A0http://www.ca= tb.org/~esr/faqs/smart-questions.html]
--000000000000478e890611590c24-- From nobody Thu Feb 15 20:37:37 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TbRhV4vVJz5B8Zv for ; Thu, 15 Feb 2024 20:37:46 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TbRhT5P0cz4HQQ for ; Thu, 15 Feb 2024 20:37:45 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=fdThKkxx; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:From:References:To:Subject:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=6Pv5Uw32lT6EJLu8V5U9pE+C6gs4OGItLR2bMy1L238=; b=fdThKkxxneuFRpPBUMLSjbhlac O+G3j9+1F+MJudOpbxQIIfKBS8DVe+KtmBLaQLpGETfzXxLACkFdCWfJzpH0m3R0NDvFoI0h/+86T Bkq6TcZnN0ic4iU2QwdPJPovYnlgqjQzV8k6g6O48RqP8ECc7r45NNWyhZW9uJofQK10eaM8njrwq 20dd19lGaSI/ZRibEyRz/Ngf6QgROZ8hXTz7zfaF5z+xtti+iXtN9uXCuZiCQvbiJCfLy2SpPKehT odzzXLrJ1wfXFcNZItsgrtqtDt0+7fBvJu6pxVqu8j7X6gHiYlmlAa7Gb/GQ34wEXdvNglNNk4gFg zKg+QaCg==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1raiUE-000Mmz-0k for virtualization@freebsd.org; Thu, 15 Feb 2024 21:37:38 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1raiUD-000ShO-T8 for virtualization@freebsd.org; Thu, 15 Feb 2024 21:37:37 +0100 Message-ID: Date: Thu, 15 Feb 2024 21:37:37 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Won't contribute KVM build :-( (was: Contributing the build of a KVM image) Content-Language: en-US To: virtualization@freebsd.org References: From: Jo Durchholz In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27186/Thu Feb 15 10:23:30 2024) X-Rspamd-Queue-Id: 4TbRhT5P0cz4HQQ X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; HAS_X_AS(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[durchholz.org:+] On 29.01.24 23:41, Jo Durchholz wrote: > Hi all, > > where do I turn to if I want to contribute to the FreeBSD build/release > process so that a KVM image will be built? > > This is not actually about virtualization per se but about the > build&release process, so I guess I'd be wrong here. I'm sorry to report that I can't do it, due to lack of feedback and incomplete/outdated documentation. Current roadblock is that I don't see a list of valid value for TARGET and TARGET_ARCH. After much searching, I found that "make targets" will give me that. Trying "make.py targets" -> complains about unset MAKEOBJDIRPREFIX. Some Internet searches later I know it's just a temporary build directory, so I put it in ~/obj, export MAKEOBJDIRPREFIX=~/obj -> complains: Could not infer value for $XCC. Either set $XCC or pass --cross-bindir=/cross/compiler/dir/bin "which clang" gives me /usr/bin/clang, so why does it not find clang? Now trying src/tools/build/make.py --cross-bindir=/usr/bin targets Could not infer value for $XCPP: /usr/bin/clang-cpp Ehm... sure, no, there is no such thing as a clang-cpp. I have a clang-cpp-14. Not sure why there's no clang-cpp, https://clang.llvm.org/docs/UsersManual.html says it should be there, but then again the Debian package maintainer may have run into naming conflicts and decided to give a version-specific name. I could obviously use XCPP=/usr/bin/clang-cpp, but given that I haven't gotten an answer to my previous question and that I have to expect more roadbumps ahead, plus I'll have to learn the freebsd Arcanist tooling just to submit a patch... well, it's been death by a thousand papercuts, and sorry, that's more work than I'm willing to invest, and it's a KO anyway since I'm making something highly security-critical and I'm not confident I can learn enough of FreeBSD to make no stupid mistakes. So... Linux it will be. It's sad, because FreeBSD has a pretty impressive security track record that I'd really like to leverage, but if I can't achieve the traction to bring that advantage to the road, it's not useful for me - I'd need a competent FreeBSD person for that, and I don't have one, and my own competence seems insufficient. Please not that THIS IS NOT A CRITICISM OF FREEBSD. It's just a single data point, for anybody who wants to know what kinds of things hold back FreeBSD adoption; if there's no need for that (possibly because other things have higher priority, or a gazillion of other possible reason), please pretend I never wrote this. And, thanks for making FreeBSD. I know it's a lot of work. Regards, Jo From nobody Fri Feb 16 10:37:33 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TbpKZ17pRz5BWqw for ; Fri, 16 Feb 2024 10:37:38 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TbpKY0Tvrz4CN2 for ; Fri, 16 Feb 2024 10:37:36 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=TamU6skX; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:From:References:To:Subject:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=E6OxibPjYaKGl/ZHf7Caq0QqRBWTtQTAOSXGnlcWCvI=; b=TamU6skXUKFHx46SlWUCwcUbKN Bbl1VDQgJ5VloI93hM9fLBRQleiU4HhkmJQTGXxrHWhpXSdJ/wyKpxcIRpa4B289DK2EhzFfJYGts e255yoDiDf49OxSms1O4KXjZb0vYUMWmKjz85be4ybiiH6d6A/BeFu6u/+BbXeAOveRPGGGvImc7g C3IQ1QDq2dhafr20MifBmp5qdAEgxU4CNMfC29uh0O/8jxC3QnUZnmTuI/gzgWCZHxbbq8Z9yK/gX CuuRaDwnDCczv2BH2SSVm0sIQydSWWQkW3TYUs99YptLbF+GxLVw3WEQsfZSrB7hWdYR/fie427fC RFJEHHTQ==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ravb4-000Gum-6n for virtualization@freebsd.org; Fri, 16 Feb 2024 11:37:34 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ravb4-000Aq0-2g for virtualization@freebsd.org; Fri, 16 Feb 2024 11:37:34 +0100 Message-ID: <47e48541-15df-4c06-834a-6113694b3dba@durchholz.org> Date: Fri, 16 Feb 2024 11:37:33 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Contributing the build of a KVM image To: virtualization@freebsd.org References: <8d266dd2-5ebc-40db-995f-4a4bfb9707eb@nomadlogic.org> <60f81e4d-8117-4410-b76b-e4ce3e076379@durchholz.org> <21ddd0c9-3b5d-a8ba-edaa-ebefc174172a@ShaneWare.Biz> Content-Language: en-US From: Jo Durchholz In-Reply-To: <21ddd0c9-3b5d-a8ba-edaa-ebefc174172a@ShaneWare.Biz> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27187/Fri Feb 16 10:23:45 2024) X-Rspamd-Queue-Id: 4TbpKY0Tvrz4CN2 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.35 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.86)[-0.860]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; HAS_X_AS(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[durchholz.org:+] > So what is it that you want to change that the current images don't > provide? In hindsight, I found out there's already a qcow2 image and the only thing left to do would be a Vagrant configuration that used that image. > You may want to use a standard iso image to install FreeBSD > into a virtual machine and then do your modified build within that and > install over the standard system install. This is not for a modified build actually, it's for automated tests. So I don't need to do the build, except I wanted to study it to know where to insert the Vagrant configuration - I can't send a patch unless I know it builds cleanly, can I? I know it's probably possible to take shortcuts here, if the Vagrant configs are created during "make release", but I don't now enough about FreeBSD's build system to confidently do that, and I ultimately stopped the effort when the cross-build setup failed. > https://computingforgeeks.com/how-to-install-freebsd-on-kvm-virtualbox/ Good to know there's a ready-made set of instructions for interactive setup. Some KVM parameters aren't easy to understand, and would have required experimentation. It's based on an interactive tool, so I'll need to replace that with scripting for automated testing, and it's using an install image so there's a lot of interactive steps, but I found the for-VM images, including a qcow2 one, so I can whip something up I believe. I won't be able to use Vagrant, and I won't have the expectation that this specific kind of image will be available in future releases, but I guess there are no guarantees for that kind of stuff anyway. > You can also find a list of pre-built virtual disk images at the bottom > of the release announcement. > > https://www.freebsd.org/releases/14.0R/announce/ I found the images directly :-) >> Onwards to the next question: >> What would be the right place to ask questions about how to run `make`? >> >> In case this is already the right place, here they are: >> >> 1) I'm worried that make will write files to directories that I'm not >> aware of, or even overwrite stuff it has no business overwriting. >> How to I make sure that it does not write anywhere except >> ~/projects/freebsd (which is the place where I want to experiment with >> the freebsd build)? > > Running make buildworld will keep all build output within MAKEOBJDIRPREFIX I figured that much, but it's good to have a confirmation from somebody with more experience. >> 3) I need to run make on a Linux box. > > Aahh, so you are trying to cross compile FreeBSD on Linux. Seems this > should be easier than in the past, have you looked at > > https://docs.freebsd.org/en/books/handbook/cutting-edge/#building-on-non-freebsd-hosts Yeah, it fails on today's Debian. Autodetection does not find clang's cpp, possibly due to a quirk in Debian's clang setup, and even getting to that point was a lot of experimentation. I stopped trying that approach, since it was too much of "with each hurdle passed, another one comes up", which Isn't Worth It(tm) if there's an alternative approach available. > It is a few years old, but this creates a docker image to build FreeBSD > > https://github.com/sandvine/freebsd-cross-build That's for building FreeBSD inside a Docker container. It does some interesting things: - Download the gcc toolchain (at a specific version). - Configure it with some very specific option and build it. - Prepare things so the toolchain will be picked up by make.py. It does not start the actual build, which would (hopefully) be "just call make.py, the environment is already set up for this". I don't know why it uses a Docker container. One reason might be to allow doing the build on Windows, another one might be to isolate the build from the host to avoid any risk of environment pollution. It's using the gcc toolchain instead of clang, and I don't know of any justification for that, except maybe for better toolchain knowledge. Still, it's a useful baseline for those who want to run a build in a Docker contains. Regards, Jo From nobody Fri Feb 16 17:19:40 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TbzFg37hFz53b3x for ; Fri, 16 Feb 2024 17:19:51 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TbzFd6nB1z52vq for ; Fri, 16 Feb 2024 17:19:49 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=UfxduSPy; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41GHJgjl035859 for ; Fri, 16 Feb 2024 11:19:42 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708103982; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=8hxDWyBHvUACXcCSNXQf5Ue7XndQs1pno8K0b8SuGFQ=; b=UfxduSPy4NReXzOok1dPqxRB/gdead/gMz8jo5XzlfNLoi9RACAUqIOrXe6KxMWuTFasY0 9yBW3JsJlnnoXIwBKgSGUKbhME0rPqq15MwIPZzSSCBS3EyE1LApovgUhDjb07Qq3NpSRK rOUBPzNMZFeqTjjcPAFOUHZ8HwgJ2RbnWbV24vdBCT2AzJ/s1CxcNxxBMIuViAaly6bOvw U1cB3sxqPlqoKJ1tczxirwEx086DPHZX2Np+2QBB00eY10otaW3I0IpqKxiw+65jSd3LsA UjA1GuAJ+I9Z4hEh9g0lcdktS/c/gAUe8jXCPgB1b1FCKE4BkM0DoK6RNocNqQ== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 6E2FF3B587 for ; Fri, 16 Feb 2024 11:19:42 -0600 (CST) Content-Type: multipart/alternative; boundary="------------1LSnpPp4X6zx2rrYZP7xUqyR" Message-ID: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> Date: Fri, 16 Feb 2024 11:19:40 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: FreeBSD virtualization Content-Language: en-US From: Matthew Grooms Subject: bhyve disk performance issue X-Rspamd-Queue-Id: 4TbzFd6nB1z52vq X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.42 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.93)[-0.931]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[shrew.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[shrew.net:+] This is a multi-part message in MIME format. --------------1LSnpPp4X6zx2rrYZP7xUqyR Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi All, I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve testing different guest options such as nvme vs virtio block storage. Unfortunately I hit a road block due to a performance issue that I can't explain and would like to bounce it off the list. Here are the hardware specs for the systems ... Intel Xeon 6338 CPU ( 32c/64t ) 256G 2400 ECC RAM 16x 4TB Samsung SATA3 SSDs Avago 9361-16i ( mrsas - HW RAID10 ) Avago 9305-16i ( mpr - zpool RAID10 ) I started by performing some bonnie++ benchmarks on the host system running AlmaLinux 9.3 and FreeBSD 14 to get a baseline using HW RAID10. The disk controllers are 8x PCIe v3 but that should be adequate considering the 6gbit disk interfaces ... RHEL9 + EXT4 ---------------------------------------------------------------------------------------------------- Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP localhost.loca 502G 2224k  99  2.4g  96  967m  33 3929k  93 1.6g  33 +++++ +++ Latency              4403us   30844us   69444us   27015us 22675us    8754us Version  2.00       ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency               118us     108us     829us 101us       6us     393us FreeBSD14 + UFS ---------------------------------------------------------------------------------------------------- Version  1.98       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP test.shrew. 523440M  759k  99  2.0g  99  1.1g  61 1945k  99 1.3g  42 264.8  99 Latency             11106us   31930us     423ms    4824us 321ms   12881us Version  1.98       ------Sequential Create------ --------Random Create-------- test.shrew.lab      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16 13095.927203  20 +++++ +++ 25358.227072 24 13573.129095  19 +++++ +++ 25354.222712  23 Latency              4382us      13us      99us 3125us       5us      67us Good enough. The next thing I tried was running the same benchmark in a RHEL9 guest to test the different storage config options, but that's when I started to encounter difficulty repeating tests that produced consistent results. At first the results appeared somewhat random but, after a few days of trial and error, I started to identify a pattern. The guest would sometimes perform well for a while, usually after a restart, and then hit a sharp drop off in performance over time. For example: Version  2.00 ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP linux-blk    63640M  694k  99  1.6g  99  737m  76  985k  99 1.3g  69 +++++ +++ Latency             11579us     535us   11889us    8597us 21819us    8238us Version  2.00       ------Sequential Create------ --------Random Create-------- linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              7620us     126us    1648us     151us 15us     633us --------------------------------- speed drop --------------------------------- Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP linux-blk    63640M  676k  99  451m  99  314m  93  951k  99 402m  99 15167 530 Latency             11902us    8959us   24711us   10185us 20884us    5831us Version  2.00       ------Sequential Create------ --------Random Create-------- linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16     0  96 +++++ +++ +++++ +++     0  96 +++++ +++     0  75 Latency               343us     165us    1636us     113us 55us    1836us The above test ran 6 times for roughly 20 mins producing higher speed results before slowing to the lower speed result. The time to complete the benchmark also increased from a about 2.5 mins to about 8 minutes. To ensure I didn't miss something in my baseline, I repeated that benchmark on the host system in a loop for about an hour but the output was consistent with my original testing. To ensure performance didn't bounce back after it slowed, I repeated the benchmark in a loop on the guest for about 4 hours but the output was also consistent. I then tried switching between block device, img on ufs, img on zfs dataset and zvols as well as switching between virtio block and nvme in the guest. All of these options appeared to suffer from the same problem albeit with slightly different performance numbers. I also tried swapping out the storage controller and running some benchmarks using a zpool over the disk array to see if that was any better. Same issue. I also tried pinning the guest CPU to specific cores ( -p x:y ). No improvement. Here is a list of a few other things I'd like to try: 1) Wiring guest memory ( unlikely as it's 32G of 256G ) 2) Downgrading the host to 13.2-RELEASE 3) Test a different guest OSs other than RHEL8 & RHEL9 4) Test a different model of RAID/SAS controller 5) Test xen vs bhyve disk performance At this point I thought it prudent to post here for some help. Does anyone have an idea of what might cause this issue? Does anyone have experience testing bhyve with an SSD disk array of this size or larger? I'm happy to provide more data points on request. Thanks, -Matthew --------------1LSnpPp4X6zx2rrYZP7xUqyR Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi All,

I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve testing different guest options such as nvme vs virtio block storage. Unfortunately I hit a road block due to a performance issue that I can't explain and would like to bounce it off the list. Here are the hardware specs for the systems ...

Intel Xeon 6338 CPU ( 32c/64t )
256G 2400 ECC RAM
16x 4TB Samsung SATA3 SSDs
Avago 9361-16i ( mrsas - HW RAID10 )
Avago 9305-16i ( mpr - zpool RAID10 )

I started by performing some bonnie++ benchmarks on the host system running AlmaLinux 9.3 and FreeBSD 14 to get a baseline using HW RAID10. The disk controllers are 8x PCIe v3 but that should be adequate considering the 6gbit disk interfaces ...

RHEL9 + EXT4
----------------------------------------------------------------------------------------------------
Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.loca 502G 2224k  99  2.4g  96  967m  33 3929k  93  1.6g  33 +++++ +++
Latency              4403us   30844us   69444us   27015us   22675us    8754us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               118us     108us     829us     101us       6us     393us

FreeBSD14 + UFS
----------------------------------------------------------------------------------------------------
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
test.shrew. 523440M  759k  99  2.0g  99  1.1g  61 1945k  99  1.3g  42 264.8  99
Latency             11106us   31930us     423ms    4824us     321ms   12881us
Version  1.98       ------Sequential Create------ --------Random Create--------
test.shrew.lab      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 13095.927203  20 +++++ +++ 25358.227072  24 13573.129095  19 +++++ +++ 25354.222712  23
Latency              4382us      13us      99us    3125us       5us      67us

Good enough. The next thing I tried was running the same benchmark in a RHEL9 guest to test the different storage config options, but that's when I started to encounter difficulty repeating tests that produced consistent results. At first the results appeared somewhat random but, after a few days of trial and error, I started to identify a pattern. The guest would sometimes perform well for a while, usually after a restart, and then hit a sharp drop off in performance over time. For example:

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  694k  99  1.6g  99  737m  76  985k  99  1.3g  69 +++++ +++
Latency             11579us     535us   11889us    8597us   21819us    8238us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7620us     126us    1648us     151us      15us     633us

--------------------------------- speed drop ---------------------------------

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  676k  99  451m  99  314m  93  951k  99  402m  99 15167 530
Latency             11902us    8959us   24711us   10185us   20884us    5831us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  96 +++++ +++ +++++ +++     0  96 +++++ +++     0  75
Latency               343us     165us    1636us     113us      55us    1836us

The above test ran 6 times for roughly 20 mins producing higher speed results before slowing to the lower speed result. The time to complete the benchmark also increased from a about 2.5 mins to about 8 minutes. To ensure I didn't miss something in my baseline, I repeated that benchmark on the host system in a loop for about an hour but the output was consistent with my original testing. To ensure performance didn't bounce back after it slowed, I repeated the benchmark in a loop on the guest for about 4 hours but the output was also consistent. I then tried switching between block device, img on ufs, img on zfs dataset and zvols as well as switching between virtio block and nvme in the guest. All of these options appeared to suffer from the same problem albeit with slightly different performance numbers. I also tried swapping out the storage controller and running some benchmarks using a zpool over the disk array to see if that was any better. Same issue. I also tried pinning the guest CPU to specific cores ( -p x:y ). No improvement.

Here is a list of a few other things I'd like to try:

1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE
3) Test a different guest OSs other than RHEL8 & RHEL9
4) Test a different model of RAID/SAS controller
5) Test xen vs bhyve disk performance

At this point I thought it prudent to post here for some help. Does anyone have an idea of what might cause this issue? Does anyone have experience testing bhyve with an SSD disk array of this size or larger? I'm happy to provide more data points on request.

Thanks,

-Matthew

--------------1LSnpPp4X6zx2rrYZP7xUqyR-- From nobody Fri Feb 16 17:42:35 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TbzmL5JXGz53tbk for ; Fri, 16 Feb 2024 17:42:58 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TbzmL0ClYz57wC for ; Fri, 16 Feb 2024 17:42:58 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id E362411400B4; Fri, 16 Feb 2024 12:42:56 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute7.internal (MEProxy); Fri, 16 Feb 2024 12:42:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1708105376; x=1708191776; bh=ncRK1GaDUL RsBm3ERSjSEvhW98FhfG9w2XNIHJD44Ps=; b=drLypW1B0Av4C8jFnG6reuAuau DVUyofctx7n2rU8l0sm+jsTyxO80UGoasC4imIv7k8EqVX3wauVuQ5+XtemIMzRD C/+G3TxJmSR0/Wgtvge7KubMYFhgn80v3ci4dfkX7k0XIxfyZiJrrfRt3JAIDQPR Wqg2yC5kOhfxnXUyawhFwawRsI5x7O4U3vYx2ij3Snr1pYBGUZm9YTqbeqy55Krt 4JnGivPDVV+Q2Q53O6w1Wv3q9xkJjxPJbm76IDkqFEw2vm/bwL6sMnaWzFgtrNAy pCzEIKhyH7JNo8QvaFBB94XeSwm8H7iixo83CgSMHUpztUpNdWidwqjcscZg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708105376; x=1708191776; bh=ncRK1GaDULRsBm3ERSjSEvhW98Fh fG9w2XNIHJD44Ps=; b=MulIyMR2CFO17I7jngZfQfDTdpU1FjB7/eG+nhMOo/wy +49kHGsf+dJ350yweIWoW1Ww2vzxq/QfrGC/ITlwrj75aLT/e0IRz/1t+Zlc7g8m pb34+hzXbHLGe1iTDKPA+CVPdp4XziUZOKxmkqyBj04+V8OdO4GvbnicsUfu+yQz hJn72/+wCLHM9f7rw5fhkFFUMbM9XK7h8EuWhsKjJjtUW4SkRNC1Cl6zpnHa0pxI GdCNxwsJrQc4lkl6644xJtKzrR7dI5q+NuKr6l+UnInx3rZsQcfBxNwHyz0HqS/n qdJFW/k2o8QHF7nJctI6S4TN0ZRRf5WqCWrhhlSgQA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvgddutdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdevhhhu tghkucfvuhhffhhlihdfuceotghhuhgtkhesthhufhhflhhirdhnvghtqeenucggtffrrg htthgvrhhnpeevjeevtddtteffkeeigefhveekjefhtedutddtiefgjefgffejheevtdet hfeftdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gthhhutghksehtuhhffhhlihdrnhgvth X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 393EEB60092; Fri, 16 Feb 2024 12:42:56 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Message-Id: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> In-Reply-To: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> Date: Fri, 16 Feb 2024 09:42:35 -0800 From: "Chuck Tuffli" To: "Matthew Grooms" , "FreeBSD virtualization" Subject: Re: bhyve disk performance issue Content-Type: multipart/alternative; boundary=4a4f83f7edf3401cad2a2c472cf96382 X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4TbzmL0ClYz57wC X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US] --4a4f83f7edf3401cad2a2c472cf96382 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: > Hi All, >=20 >=20 >=20 > I'm in the middle of a project that involves building out a handful of= servers to host virtual Linux instances. Part of that includes testing = bhyve to see how it performs. The intent is to compare host storage opti= ons such as raw vs zvol block devices and ufs vs zfs disk images using h= ardware raid vs zfs managed disks. It would also involve >=20 >=20 =E2=80=A6 > Here is a list of a few other things I'd like to try: >=20 >=20 > 1) Wiring guest memory ( unlikely as it's 32G of 256G ) > 2) Downgrading the host to 13.2-RELEASE FWIW we recently did a similar exercise and saw significant performance = differences on ZFS backed disk images when comparing 14.0 and 13.2. We d= idn=E2=80=99t have time to root cause the difference, so it could simply= be some tuning difference needed for 14.=20 =E2=80=94chuck --4a4f83f7edf3401cad2a2c472cf96382 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Fri, Feb 16,= 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the mi= ddle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80= =A6

Here is a list of a few other things I'd l= ike to try:


1) Wiring guest memory (= unlikely as it's 32G of 256G )
2) Downgrading the host t= o 13.2-RELEASE

FWIW we recentl= y did a similar exercise and saw significant performance differences on = ZFS backed disk images when comparing 14.0 and 13.2. We didn=E2=80=99t h= ave time to root cause the difference, so it could simply be some tuning= difference needed for 14. 

=E2=80=94c= huck
--4a4f83f7edf3401cad2a2c472cf96382-- From nobody Fri Feb 16 17:45:47 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tbzqk2gjjz53twY for ; Fri, 16 Feb 2024 17:45:54 +0000 (UTC) (envelope-from pprocacci@gmail.com) Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tbzqk0tfTz592y for ; Fri, 16 Feb 2024 17:45:54 +0000 (UTC) (envelope-from pprocacci@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-a3ddc13bbb3so165523566b.0 for ; Fri, 16 Feb 2024 09:45:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708105552; x=1708710352; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=N7mnvjMwUoxZoKGfYbsM0FM+oh1q+JWqQJ0z+LLz2wc=; b=AeyPeuYtw6iuZZz4U2sizeM4YPfzos4B96KencUwtil20RmNR39Ix/Fg1kdxVVleLn Q59IA/JhA/KbjcVlS6s9615o9MGaQhJ0KaQI/kNKk/C3AUp7EhGpaaMh1FmpqBmAVlFw 5GwzR1zosMEq02DWp0YLz0J4S2Dc2VMqEXmG4YlpR94N+gNx0KUJ5hP5BYfIaSixjxXf ixx8z6HYm0Qpr7gvL23kod7ci0kXTAtiLL7xEfmbZRzzBVULHP0CI5X9WCJ675fKaQiA M11ez+SZ4z9okLuyP+F4/BUKCcfFeIQym0TiBTHVbn7NfFxaWjcnVyT98umfdUoNXWaB Sfpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708105552; x=1708710352; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=N7mnvjMwUoxZoKGfYbsM0FM+oh1q+JWqQJ0z+LLz2wc=; b=RTKKnFz9d7Iwkdu8RPQbCfj58Iu08Ma4GlvLI0rpcezJ1DVNR0aE8a74BlQZzHLAY3 OoyA9vPrUWjsFjDD+EC1vk3r83MjNFePtLmBdEGA/usPbI4+h9LdfFoJEqPTKXHAGG5t mCggJ32v1UgrNgsae63beq1yckawxEi1WK0jvaXNi1c4q9QsyM91LWEV6a9vdprpjEl2 Z2F/lj1YRR0ibob676BH5DRgp69H28spqskXuBaCHytz1lvP/UU01DelGLSpYsDoBatf lnTR57sDj9RyZ3slh89N3v1/FJ5yL8FBhDXsDVe6FQP3KuQmala+lh1j5/GAECiUI58t Ojlw== X-Forwarded-Encrypted: i=1; AJvYcCXjQKqOz+JSuzDaJiRc3bqCJT/IOPih6cUruhBh9opZQ1pJtB2OTrVIR3+Um7bmYzUIoyyi0aYJcccTq62+avK5x8sgaxr75AN3PqKGUqHyfvuQ X-Gm-Message-State: AOJu0Yxfm2wzx9MhQ4VWOJazpY3fdqu26wyQp/We4PVU0yNev6ThhK8X xozFYxGRBhb9BoIiF9DkdLFSLb9ylok476UL/405Uup82Z5eBfL2qrFEXV0LD4Ue0Sh12zyTuG3 IADO2XQ4i62yyZfy1WaXTlp1DO+WPJTDjXg== X-Google-Smtp-Source: AGHT+IGlFOx4f6b5Qn6EkhCuk4JQN2E+zuCd+wMYvq9TV5bKsvw+mh4wT3l1mW7RZopYjF9rkH3ZCKUuMJbD+13ebTA= X-Received: by 2002:a17:906:5f84:b0:a3c:1e4d:725a with SMTP id a4-20020a1709065f8400b00a3c1e4d725amr6725980eju.37.1708105552217; Fri, 16 Feb 2024 09:45:52 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> In-Reply-To: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> From: Paul Procacci Date: Fri, 16 Feb 2024 12:45:47 -0500 Message-ID: Subject: Re: bhyve disk performance issue To: Chuck Tuffli Cc: Matthew Grooms , FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000adcbaa06118350cb" X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tbzqk0tfTz592y X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --000000000000adcbaa06118350cb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli wr= ote: > On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: > > Hi All, > > > I'm in the middle of a project that involves building out a handful of > servers to host virtual Linux instances. Part of that includes testing > bhyve to see how it performs. The intent is to compare host storage optio= ns > such as raw vs zvol block devices and ufs vs zfs disk images using hardwa= re > raid vs zfs managed disks. It would also involve > > > =E2=80=A6 > > Here is a list of a few other things I'd like to try: > > 1) Wiring guest memory ( unlikely as it's 32G of 256G ) > 2) Downgrading the host to 13.2-RELEASE > > > FWIW we recently did a similar exercise and saw significant performance > differences on ZFS backed disk images when comparing 14.0 and 13.2. We > didn=E2=80=99t have time to root cause the difference, so it could simply= be some > tuning difference needed for 14. > > =E2=80=94chuck > I myself am actually doing something very very similar. I was seeing atrocious disk performance until I set the disk type to nvme. Now it's screaming fast. disk0_type=3D"nvme" Not sure what yours is set at, but it might be worth looking into. ~Paul --=20 __________________ :(){ :|:& };: --000000000000adcbaa06118350cb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Feb 16, 2024 at 12:43=E2= =80=AFPM Chuck Tuffli <chuck@tuffli.= net> wrote:
On Fri, Feb 16, = 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm i= n the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80=A6=

Here is a list of a few other things I'= d like to try:


1) Wiring guest memory (= unlikely as it's 32G of 256G )
2) Downgrading the host = to 13.2-RELEASE

FWIW we recently = did a similar exercise and saw significant performance differences on ZFS b= acked disk images when comparing 14.0 and 13.2. We didn=E2=80=99t have time= to root cause the difference, so it could simply be some tuning difference= needed for 14.=C2=A0

=E2=80=94chuck

I myself am actuall= y doing something very very similar.
I was seeing atrocious disk = performance until I set the disk type to nvme.
Now it's s= creaming fast.

disk0_type=3D"nvme"

Not s= ure what yours is set at, but it might be worth looking into.
~Paul
-- <= /span>
__________________
=
:(){ :|:& };:
--000000000000adcbaa06118350cb-- From nobody Fri Feb 16 17:55:27 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc0394jlNz53wM1 for ; Fri, 16 Feb 2024 17:55:49 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc0392j1sz3x8f for ; Fri, 16 Feb 2024 17:55:49 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 680925C00AE; Fri, 16 Feb 2024 12:55:48 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute7.internal (MEProxy); Fri, 16 Feb 2024 12:55:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1708106148; x=1708192548; bh=N6shIJCxz7 cMT0xOjez70/QlT/pqNM+UlU8mhfCYi8Y=; b=rfmyFIL28tddL3XX2mfV2r8Xo6 GZUtO7qdEE6V6Pz/havVF2P1ziWvA4IKTNPebzVosHQLE36tySG8Mne5fSOX9B+N qpPHfdWcgybvz5gxKZvJGnbOwKCEtUtj4sk+OLEDSPufmJWrFDSG03qxnveGUFBx ipT2fO2GCM1hyxTfpOt4vKcHXEPE8UF4DmWb1f5A+OHZUK/RSINUi+z0ffpqFiSz LYZsLw+WPd7MvznMubnLW+xcvUypwqju3ZEAUaJteLwtkGmxfQW1Y7KQZdKeViax 9tyftKuy/8S5AF8GiXaXlV3zb7vtAtvayVNH4afjlbaSG5wm82bDQcpyAB8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708106148; x=1708192548; bh=N6shIJCxz7cMT0xOjez70/QlT/pq NM+UlU8mhfCYi8Y=; b=d7CKlPvdUYqdpmWESjDmpS1n7LG3MaYj29oMmyHVBrO/ pHJHCJ7rZmI0KeiNFmpDqkwD9/ENb3YvbZug+8IzZxccWBMsPMUg/sNEcRB3oTx9 Bm8oP3cXvSAYLgEo4okJrwnYOaBRFqLJLkw8ewRahDZpDtOkuphOUaSmNOQANuEJ eVr39pMladn9+qEz3BHjF6tPR6oPHSJs46fcC0mvNnpPI7557UegGvKmV8YX/ZyL e7ResoZ378Eo58vrZ/mvjbyiDaBtK8xfRxACKGKjSshzkCFxowMMt6dpqOMN45V6 7GFTQIqoBi2Wha5731xtvfInRUVS/SFW5eKd80sLhA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvgddutdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerreejnecuhfhrohhmpedfvehh uhgtkhcuvfhufhhflhhifdcuoegthhhutghksehtuhhffhhlihdrnhgvtheqnecuggftrf grthhtvghrnhepudeuteehiefgtdehheduveevgeehieevveeuvddthedvueevjeehfeet heefjeffnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh eptghhuhgtkhesthhufhhflhhirdhnvght X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 0E4F9B6008D; Fri, 16 Feb 2024 12:55:47 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Message-Id: <5fc8b9d9-da94-4694-9134-d0cc22df2eaf@app.fastmail.com> In-Reply-To: References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> Date: Fri, 16 Feb 2024 09:55:27 -0800 From: "Chuck Tuffli" To: "Paul Procacci" Cc: "Matthew Grooms" , "FreeBSD virtualization" Subject: Re: bhyve disk performance issue Content-Type: multipart/alternative; boundary=edaaeeb4c44c4b3f8141d4116aa98709 X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tc0392j1sz3x8f X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] --edaaeeb4c44c4b3f8141d4116aa98709 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024, at 9:45 AM, Paul Procacci wrote: >=20 >=20 > On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli wrote: >> __ >> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>> Hi All, >>>=20 >>>=20 >>>=20 >>> I'm in the middle of a project that involves building out a handful = of servers to host virtual Linux instances. Part of that includes testin= g bhyve to see how it performs. The intent is to compare host storage op= tions such as raw vs zvol block devices and ufs vs zfs disk images using= hardware raid vs zfs managed disks. It would also involve >>>=20 >>>=20 >> =E2=80=A6 >>> Here is a list of a few other things I'd like to try: >>>=20 >>>=20 >>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>> 2) Downgrading the host to 13.2-RELEASE >>=20 >> FWIW we recently did a similar exercise and saw significant performan= ce differences on ZFS backed disk images when comparing 14.0 and 13.2. W= e didn=E2=80=99t have time to root cause the difference, so it could sim= ply be some tuning difference needed for 14.=20 >>=20 >> =E2=80=94chuck > I myself am actually doing something very very similar. > I was seeing atrocious disk performance until I set the disk type to n= vme. > Now it's screaming fast. >=20 > disk0_type=3D"nvme" > Not sure what yours is set at, but it might be worth looking into. Similar to Matthew, we were testing both virtio and nvme and saw perform= ance differences for both emulation types between 13 and 14.=20 --edaaeeb4c44c4b3f8141d4116aa98709 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Fri, Feb 16,= 2024, at 9:45 AM, Paul Procacci wrote:


On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli <= ;chuck@tuffli.net> wrote:
=

On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,=


I'm in the middle of a project that involves bui= lding out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80= =A6

Here is a list of a few othe= r things I'd like to try:


1) Wiring gu= est memory ( unlikely as it's 32G of 256G )
2) Downgrading= the host to 13.2-RELEASE

FWIW= we recently did a similar exercise and saw significant performance diff= erences on ZFS backed disk images when comparing 14.0 and 13.2. We didn=E2= =80=99t have time to root cause the difference, so it could simply be so= me tuning difference needed for 14. 

=E2= =80=94chuck
I myself = am actually doing something very very similar.
I was seein= g atrocious disk performance until I set the disk type to nvme.
Now it's screaming fast.

disk0_t= ype=3D"nvme"
Not sure what yours is set at, but it m= ight be worth looking into.
Similar to = Matthew, we were testing both virtio and nvme and saw performance differ= ences for both emulation types between 13 and 14. 

--edaaeeb4c44c4b3f8141d4116aa98709-- From nobody Fri Feb 16 18:00:13 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc08H4g3Fz53wwV for ; Fri, 16 Feb 2024 18:00:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc08H3hf2z3xW2 for ; Fri, 16 Feb 2024 18:00:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41GI0DH3036228; Fri, 16 Feb 2024 12:00:13 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708106413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7acgBZXoWjZKLVXFxT2S150WoyWhqM5Ex951cfRkd5g=; b=B2u76vRtq0FpOauZhYxJSXo1zkWtHKimJ83sDhFuTrFJepCh2Hi7tzqzpIcK9SRtsgYmDb ex+OmM9tj6Q8Jp7g20rbelDSjnauLaX5P0+cATHdgRMBz+Xv83ns4i4FDOn/JwQhXvoyu1 1gd9rwtK9SKzIciACYxq2sITAQbNk5YuF8+QFi/3Nh0vNLtFKqEaiTMquZ0atUxqk2yhxb eXFUvnDImuc30gTJWsmQ0xAm623BXW6Z0j1dXNZTQTLccyYubOzSx0ZKqMRLzySf6bd55U ZQjetNaWyHidKm6vLAxgIK1I68Rt+poLYXS4KckC3acItsffFm0NYp/LneDXTw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id CB83D3B587; Fri, 16 Feb 2024 12:00:13 -0600 (CST) Content-Type: multipart/alternative; boundary="------------OUO03Djdqszf3XOd8PWHDNYK" Message-ID: <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> Date: Fri, 16 Feb 2024 12:00:13 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Chuck Tuffli , FreeBSD virtualization References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> From: Matthew Grooms In-Reply-To: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tc08H3hf2z3xW2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US] This is a multi-part message in MIME format. --------------OUO03Djdqszf3XOd8PWHDNYK Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/16/24 11:42, Chuck Tuffli wrote: > On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >> >> Hi All, >> >> >> I'm in the middle of a project that involves building out a handful >> of servers to host virtual Linux instances. Part of that includes >> testing bhyve to see how it performs. The intent is to compare host >> storage options such as raw vs zvol block devices and ufs vs zfs disk >> images using hardware raid vs zfs managed disks. It would also involve >> >> > … >> >> Here is a list of a few other things I'd like to try: >> >> >> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >> 2) Downgrading the host to 13.2-RELEASE > > FWIW we recently did a similar exercise and saw significant > performance differences on ZFS backed disk images when comparing 14.0 > and 13.2. We didn’t have time to root cause the difference, so it > could simply be some tuning difference needed for 14. Hi Chuck, That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here. Thanks again, -Matthew --------------OUO03Djdqszf3XOd8PWHDNYK Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.

Thanks again,

-Matthew

--------------OUO03Djdqszf3XOd8PWHDNYK-- From nobody Sat Feb 17 00:04:55 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc8Fr3H0Nz5B5Ds for ; Sat, 17 Feb 2024 00:05:36 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc8Fp5bnTz4qQw for ; Sat, 17 Feb 2024 00:05:34 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Jp1rEpUS; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::62e as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-a2f22bfb4e6so326729266b.0 for ; Fri, 16 Feb 2024 16:05:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708128332; x=1708733132; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=1vrI8UXsTWtiMeiyU1J5jDjx8hk48z+UXeqw4Efm/Pc=; b=Jp1rEpUSw1JP/BFrAyLGqCie1qYXiZGe4Bu9bhK35Vcu1/6Ev7OF1urIpNXDXC40WH WLxVEzJMHa2USgrlRrts8dy3/RGkqBXo0vEjTysgrpo4KQ4gQqOwUkfGzLsDJGHPr8q6 eEamurwXE566nCwGJVNsYUIXt/QqIkwaR59j0pZN88YrP1KCVA/HT+A8pcwdzA9RS/g8 EmenLaMJWpspKVoBoNB7MVl07bTEkxW5norZyvyfvfY6zYudvtE3HIh1yF1NxDUxzazp uIy7jGIYQ9UinQ2YmyQ1VQfFOXiUP2TtPFEjxpz5zaEmdfqG2KhstcAOmVab08tz8Wpj o8Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708128332; x=1708733132; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1vrI8UXsTWtiMeiyU1J5jDjx8hk48z+UXeqw4Efm/Pc=; b=geXAhbH7PGp5n+rGc6Z5+v4Ut+345wc4zmzuBIrm4R90SnpPhqxZK7Ue/y8UGN3y3g HzuDvlE0w7YpRRma2Q2DnkwHQMcy5FdceYYq22MbkKCBsrIlFxH+CAaWG8mg1lnwfiEJ Y3XDFyBbPtYl+oGt4QHSqzS+bRi99YQfh/2w2FOIUzuCNrl9vTJrj7nTdLNDRk2FMN/z AUXFq/srlYROussUNYeWk3HpOZIGcTkQRb/EMlsaXH/NlNzBPcxY7iz9yl+mYKq/Hv5G Dyr60UUA+A5S5bgNoo93Xjf3iZo1V9Ue44y11sgynUwJQmep1vrABb9SvUCeoW57/vc6 ep9Q== X-Gm-Message-State: AOJu0YxD7u+X8fVgEQC7jNJQ4YOzf6maA0hyPTs9U8WwKEDnX3Q3n4U6 zb34KJfepxJrCBzCzrdDzey9gAa04nu0u+G2XMk1UpWYmHkHdK48KOuMs6XCKO2f4rScWrQbWfj iYtrwEL4z/hEjrcHYPToV4GeKNffyr76XmTIEkw== X-Google-Smtp-Source: AGHT+IFjXlTFwP4ujALqw5/IvMRdWR65gp+ZyszzhETRhdBoOazjm7qmHizNz7YFnsd0KFd6x3ps2VMAIoCD3+v2W/g= X-Received: by 2002:a17:906:278f:b0:a3d:aa78:af02 with SMTP id j15-20020a170906278f00b00a3daa78af02mr3762181ejc.19.1708128331774; Fri, 16 Feb 2024 16:05:31 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Sat, 17 Feb 2024 01:04:55 +0100 Message-ID: Subject: How to send a bhyve vm totally in background,suppressing any message produced by the bhyve script that I use. To: FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000723ff70611889e88" X-Rspamd-Queue-Id: 4Tc8Fp5bnTz4qQw X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62e:from] --000000000000723ff70611889e88 Content-Type: text/plain; charset="UTF-8" Hello. I want to boot a bhyve vm that does not produce any error message or warnings on the screen and that it stays in the background. This is how I launch my Linux vm : bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootindex=1 \ -s 11,hda,play=/dev/dsp,rec=/dev/dsp \ -s 13,virtio-net,tap19 \ -s 14,virtio-9p,sharename=/ \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:19 > /dev/null & I've thought that "> /dev/null" means that every message produced by the script would disappear,but,instead,it produces the following messages that I want to suppress : marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0 wrmsr to register 0x140(0) on vcpu 2 wrmsr to register 0x140(0) on vcpu 4 wrmsr to register 0x140(0) on vcpu 6 wrmsr to register 0x140(0) on vcpu 1 wrmsr to register 0x140(0) on vcpu 3 wrmsr to register 0x140(0) on vcpu 5 wrmsr to register 0x140(0) on vcpu 7 rdmsr to register 0x64e on vcpu 1 rdmsr to register 0x34 on vcpu 1 rdmsr to register 0xc0011029 on vcpu 7 Unhandled ps2 mouse command 0xe1 Unhandled ps2 mouse command 0x0a Unhandled ps2 mouse command 0x01 Unhandled ps2 mouse command 0x41 Unhandled ps2 mouse command 0x88 rdmsr to register 0x64d on vcpu 5 rdmsr to register 0x64d on vcpu 0 at this point it hangs. I should press a key to have the ability to write again on the terminal. I don't want it. Someone can help me to refine the script ? thanks. -- Mario. --000000000000723ff70611889e88 Content-Type: text/html; charset="UTF-8"
Hello.

I want to boot a bhyve vm that does not produce any error message or warnings on the screen and that it stays in the background. This is how I launch my Linux vm :

bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootindex=1 \
-s 11,hda,play=/dev/dsp,rec=/dev/dsp \
-s 13,virtio-net,tap19 \
-s 14,virtio-9p,sharename=/ \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19 > /dev/null &

I've thought that "> /dev/null" means that every message produced by the script would disappear,but,instead,it produces the following messages that I want to suppress :

marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0
wrmsr to register 0x140(0) on vcpu 2
wrmsr to register 0x140(0) on vcpu 4
wrmsr to register 0x140(0) on vcpu 6
wrmsr to register 0x140(0) on vcpu 1
wrmsr to register 0x140(0) on vcpu 3
wrmsr to register 0x140(0) on vcpu 5
wrmsr to register 0x140(0) on vcpu 7
rdmsr to register 0x64e on vcpu 1
rdmsr to register 0x34 on vcpu 1
rdmsr to register 0xc0011029 on vcpu 7
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
rdmsr to register 0x64d on vcpu 5
rdmsr to register 0x64d on vcpu 0

at this point it hangs.

I should press a key to have the ability to write again on the terminal. I don't want it. Someone can help me to refine the script ? thanks.

--
Mario.
--000000000000723ff70611889e88-- From nobody Sat Feb 17 00:17:50 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc8XC4tRQz5B6KQ for ; Sat, 17 Feb 2024 00:18:03 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc8XC38bLz4rP0 for ; Sat, 17 Feb 2024 00:18:03 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a3d0d26182dso282800366b.1 for ; Fri, 16 Feb 2024 16:18:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708129082; x=1708733882; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vMeFg43fMrkZyunHrDRXdLrw0ByenSUY5/pNznikAaY=; b=Dq/5o49k4GTygaGONuH+FnyIECn18W0dmcPq30DWQVnL1SqU+IEHCdRIdVEgnxA3rk ulVEqfBzhAvl/5aueJ5N3Oem1pFGIuR39KOSPUaezyiVLUKmEVTymKLmeWOMmX1Cvwfp 5OM4QRXs2w0jvbpUD9+k9iVddePQiCblxTVwCvZ+Ml96+tqza1CkuobcNUnRYGUjU1an WpCM/QB3qlFbtkAKxE45zylbQg9EDwINsl3TAmQ5LW862RO7wX21PtdGyaRnPUvT4KWP YO0x2SZ4PH0waI/PpTu+V2Ksyd0Fj4LyKPMPKR/ITovFHgDBjQqLVM5WMAg3JrUTrhqn lukg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708129082; x=1708733882; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vMeFg43fMrkZyunHrDRXdLrw0ByenSUY5/pNznikAaY=; b=w8/m4nhDHSBjIn2IRHUvkHYW2hVfIITCoBjIQ8d0ub7cJA6CJ0Z1IgGBaBzf0vP1dy YHuihYcutdYLL6Cx7s4CDQApa/M22HQFnwyt63oXEQiSEuFBzJLbhBSPUQ/yauzJKt3n AjTZeXT0FDlOmqFnD0WTN9a+voMjvdNu+iGIh071VjC2lkHqvk18DpW1KzrQvVPSZGGa LRdXvG2SGvfUTA67y/ud7GnP36orOxHbj+0ZMZvfZoW7CmWmnqEBANId8dVP58p1+Hc4 FIFEmnhIHLFJMetS40RyemR0e75/xWdjFs+9lV/kKGAkj88n7WIPz8gzrcG1dMPSKRF8 JXfw== X-Gm-Message-State: AOJu0YxHcgfJhIKvMNnHLPDFYb2WzP4MICObbOQKyLDu61O6zyxrNezh ZtWTu6LRi9n/nwa4vFAhqNUt1TgDPsW2sSH87sMIEICVrm1V+S2NvrHQq6C3Hod6GoLJHeE46f4 mhbD2dCcbmLWbQ7APsfQC+Eu58x8= X-Google-Smtp-Source: AGHT+IGONsm55FhzS/WjYPfOELo9G5fIR3waRJp0O9RDx+PK6pI+twzc+3RMHi5P3LlvmySh47gT6gh/gZZtNr49mmo= X-Received: by 2002:a17:906:1149:b0:a3d:f94:ebe2 with SMTP id i9-20020a170906114900b00a3d0f94ebe2mr4290923eja.76.1708129081541; Fri, 16 Feb 2024 16:18:01 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Aryeh Friedman Date: Fri, 16 Feb 2024 19:17:50 -0500 Message-ID: Subject: Re: How to send a bhyve vm totally in background,suppressing any message produced by the bhyve script that I use. To: Mario Marietto Cc: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tc8XC38bLz4rP0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto wrote: > > Hello. > > I want to boot a bhyve vm that does not produce any error message or warn= ings on the screen and that it stays in the background. This is how I launc= h my Linux vm : > > bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \ > -s 0,hostbridge \ > -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootind= ex=3D1 \ > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 13,virtio-net,tap19 \ > -s 14,virtio-9p,sharename=3D/ \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > vm0:19 > /dev/null & > > > I've thought that "> /dev/null" means that every message produced by the = script would disappear,but,instead,it produces the following messages that = I want to suppress : > > marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0 > wrmsr to register 0x140(0) on vcpu 2 > wrmsr to register 0x140(0) on vcpu 4 > wrmsr to register 0x140(0) on vcpu 6 > wrmsr to register 0x140(0) on vcpu 1 > wrmsr to register 0x140(0) on vcpu 3 > wrmsr to register 0x140(0) on vcpu 5 > wrmsr to register 0x140(0) on vcpu 7 > rdmsr to register 0x64e on vcpu 1 > rdmsr to register 0x34 on vcpu 1 > rdmsr to register 0xc0011029 on vcpu 7 > Unhandled ps2 mouse command 0xe1 > Unhandled ps2 mouse command 0x0a > Unhandled ps2 mouse command 0x01 > Unhandled ps2 mouse command 0x41 > Unhandled ps2 mouse command 0x88 > rdmsr to register 0x64d on vcpu 5 > rdmsr to register 0x64d on vcpu 0 > > at this point it hangs. > > > I should press a key to have the ability to write again on the terminal. = I don't want it. Someone can help me to refine the script ? thanks. Here is a fully tested/debugged hand made script to do what you requested (added bonus is the passthrough): #!/bin/sh # # Generated by PetiteCloud 0.2.6 # # nvidia1030.rom ifconfig tap1 destroy ifconfig tap1 create ifconfig tap1 up ifconfig bridge0 addm tap1 up echo net done sleep 5 bhyvectl --destroy --vm=3D6222f881ad268c17 nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s 0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s 29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -s 5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd 6222f881ad268c17&>/dev/null echo final sleep sleep 10 > > -- > Mario. --=20 Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org From nobody Sat Feb 17 00:36:41 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc8yT1LWyz5B90C for ; Sat, 17 Feb 2024 00:37:21 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc8yS0yY8z4tJ7 for ; Sat, 17 Feb 2024 00:37:20 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hUfhaaZ6; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::529 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-563ed3d221aso1801516a12.0 for ; Fri, 16 Feb 2024 16:37:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708130238; x=1708735038; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=VrkOTtD/K7nZKDFwtER4+lCPQ2lTUgMBnImFUslaadw=; b=hUfhaaZ6RrLHSrH2dXK+UP2L33yLUDZ7m/TjIGcU1spl/seWYkKZgjAuh5u4xV59Lq GPbvgCHAFZYwX23vORkaZN+PJM4okTfUMILiKH2X5a3orc5N7MJgYwypPJOchir7EZrv mjT5wKVS8w4sh+gapPtd9ObUDxsXIK1m5wWl8zF6ORNAH2q4J5XsklZqVMqEHOYvriVr ECKDEVeouC5kaw0zGNby/lEYiIl/XSv2f6ujxPQ5FUrk07icwmBFnuv1c2Mq9FcFyoAQ 9eXo1KU067sK0izPS3dn0H/tpdtf0/yCJMlOn9CbVPaV1AvBOp35GaJZ1FpK/+lSPhab EFBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708130238; x=1708735038; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VrkOTtD/K7nZKDFwtER4+lCPQ2lTUgMBnImFUslaadw=; b=dF+T4qiC+vThXYd03+PK2yVLTZBEBO9K1cCPrr88GLKYEd0V2XvV/6iA0jvS1rCeRD 7f4vuZ+1w7A1BjPRwbXCb+7rZKpJdetrcGpl2tEddLD8oXg6QLD1Lbv4Na+SWf23VPy/ VHPumdxG8/ZHHM43SenEchhgK6BJt5Qho+QGTpXkPExuIaQD/gYiQAma8raUYg3gv+0r 752mJdXz1q5YZGnpoUGBt6a8hH3fcPJYly5yQxTN+RNSM4z7kUY6S86UtPqLb89sqGFo NvXRDcIpoaa9rV4J+OMNIJYvrfJP7F0Gw0gn+9eCSrLLd4svgeEkhSFWwdl7dTSBVc5w svgA== X-Gm-Message-State: AOJu0YyfhmVKGkTTUQP7AplPgasWRI+PNPBn53TrDVGVuYh6kmCukS5E DwtGv2TVYVcHLYJOxBcJsqE2HclGvh2mA6LzUPdk2+Qc2BQFT/9OSG0PFOgg59uMLgYfNWdLTW1 vAMnJpK9m+MI2YZdn1zMPcTPk8LI= X-Google-Smtp-Source: AGHT+IHCq6DzPzt3+hkmcE9Fb3NNCIdV9eZLqKzAICF5zUqAposM0FoR0n22KTVN3TV/52KdBdhWdSh5tL15bOF7en4= X-Received: by 2002:a17:906:6617:b0:a3d:5058:50ff with SMTP id b23-20020a170906661700b00a3d505850ffmr5577723ejp.2.1708130238057; Fri, 16 Feb 2024 16:37:18 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 17 Feb 2024 01:36:41 +0100 Message-ID: Subject: Re: How to send a bhyve vm totally in background,suppressing any message produced by the bhyve script that I use. To: Aryeh Friedman Cc: FreeBSD virtualization Content-Type: multipart/alternative; boundary="00000000000011d37c06118910aa" X-Rspamd-Queue-Id: 4Tc8yS0yY8z4tJ7 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.98 / 15.00]; URI_COUNT_ODD(1.00)[5]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.976]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::529:from] --00000000000011d37c06118910aa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable This is how looks mine script after having applied your suggestions : nohup /usr/bin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A= \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex=3D= 1 \ -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ -s 13,virtio-net,tap19 \ -s 14,virtio-9p,sharename=3D/ \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:19&>/dev/null well,it does not work : the vm does not start,I don't see its process as active. output : final sleep appending output to nohup.out root@marietto:/bhyve # ps ax | grep bhyve 20220 1 S+ 0:00.00 grep bhyve On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedman wrote: > On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto > wrote: > > > > Hello. > > > > I want to boot a bhyve vm that does not produce any error message or > warnings on the screen and that it stays in the background. This is how I > launch my Linux vm : > > > > bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \ > > -s 0,hostbridge \ > > -s > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootindex= =3D1 \ > > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > > -s 13,virtio-net,tap19 \ > > -s 14,virtio-9p,sharename=3D/ \ > > -s 30,xhci,tablet \ > > -s 31,lpc \ > > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > > vm0:19 > /dev/null & > > > > > > I've thought that "> /dev/null" means that every message produced by th= e > script would disappear,but,instead,it produces the following messages tha= t > I want to suppress : > > > > marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0 > > wrmsr to register 0x140(0) on vcpu 2 > > wrmsr to register 0x140(0) on vcpu 4 > > wrmsr to register 0x140(0) on vcpu 6 > > wrmsr to register 0x140(0) on vcpu 1 > > wrmsr to register 0x140(0) on vcpu 3 > > wrmsr to register 0x140(0) on vcpu 5 > > wrmsr to register 0x140(0) on vcpu 7 > > rdmsr to register 0x64e on vcpu 1 > > rdmsr to register 0x34 on vcpu 1 > > rdmsr to register 0xc0011029 on vcpu 7 > > Unhandled ps2 mouse command 0xe1 > > Unhandled ps2 mouse command 0x0a > > Unhandled ps2 mouse command 0x01 > > Unhandled ps2 mouse command 0x41 > > Unhandled ps2 mouse command 0x88 > > rdmsr to register 0x64d on vcpu 5 > > rdmsr to register 0x64d on vcpu 0 > > > > at this point it hangs. > > > > > > I should press a key to have the ability to write again on the terminal= . > I don't want it. Someone can help me to refine the script ? thanks. > > Here is a fully tested/debugged hand made script to do what you > requested (added bonus is the passthrough): > > #!/bin/sh > # > # Generated by PetiteCloud 0.2.6 > # > > # nvidia1030.rom > > ifconfig tap1 destroy > ifconfig tap1 create > ifconfig tap1 up > ifconfig bridge0 addm tap1 up > echo net done > sleep 5 > bhyvectl --destroy --vm=3D6222f881ad268c17 > nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s > 0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s > 29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -s > 5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd > 6222f881ad268c17&>/dev/null > echo final sleep > sleep 10 > > > > > -- > > Mario. > > > > -- > Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org > --=20 Mario. --00000000000011d37c06118910aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
This is how looks mine script after having applied yo= ur suggestions :

nohup /usr/bin/bhyve -S -c so= ckets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
= -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex= =3D1 \
-s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 13,virtio-net,ta= p19 \
-s 14,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \
-s 31,lp= c \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19&>/dev/null

well,it does not work = : the vm does not start,I don't see its process as active.

output :

final sleep
appending output to nohup.out
=C2=A0
root@marietto:/bhyve # ps ax | grep bhyve
20220 =C2=A01 =C2=A0S+ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00:00.00 grep= bhyve

On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedm= an <aryeh.friedman@gmail.com= > wrote:
= On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> = wrote:
>
> Hello.
>
> I want to boot a bhyve vm that does not produce any error message or w= arnings on the screen and that it stays in the background. This is how I la= unch my Linux vm :
>
> bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
> -s 0,hostbridge \
> -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,boot= index=3D1 \
> -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
> -s 13,virtio-net,tap19 \
> -s 14,virtio-9p,sharename=3D/ \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
> vm0:19 > /dev/null &
>
>
> I've thought that "> /dev/null" means that every mess= age produced by the script would disappear,but,instead,it produces the foll= owing messages that I want to suppress :
>
> marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0
> wrmsr to register 0x140(0) on vcpu 2
> wrmsr to register 0x140(0) on vcpu 4
> wrmsr to register 0x140(0) on vcpu 6
> wrmsr to register 0x140(0) on vcpu 1
> wrmsr to register 0x140(0) on vcpu 3
> wrmsr to register 0x140(0) on vcpu 5
> wrmsr to register 0x140(0) on vcpu 7
> rdmsr to register 0x64e on vcpu 1
> rdmsr to register 0x34 on vcpu 1
> rdmsr to register 0xc0011029 on vcpu 7
> Unhandled ps2 mouse command 0xe1
> Unhandled ps2 mouse command 0x0a
> Unhandled ps2 mouse command 0x01
> Unhandled ps2 mouse command 0x41
> Unhandled ps2 mouse command 0x88
> rdmsr to register 0x64d on vcpu 5
> rdmsr to register 0x64d on vcpu 0
>
> at this point it hangs.
>
>
> I should press a key to have the ability to write again on the termina= l. I don't want it. Someone can help me to refine the script ? thanks.<= br>
Here is a fully tested/debugged hand made script to do what you
requested (added bonus is the passthrough):

#!/bin/sh
#
# Generated by PetiteCloud 0.2.6
#

# nvidia1030.rom

ifconfig tap1 destroy
ifconfig tap1 create
ifconfig tap1 up
ifconfig bridge0 addm tap1 up
echo net done
sleep 5
bhyvectl --destroy --vm=3D6222f881ad268c17
nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s
0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s
29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -s
5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
6222f881ad268c17&>/dev/null
echo final sleep
sleep 10

>
> --
> Mario.



--
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org


--
Mario.
--00000000000011d37c06118910aa-- From nobody Sat Feb 17 00:46:54 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc9BG17y1z5BBQf for ; Sat, 17 Feb 2024 00:47:34 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc9BF25Yvz4v3B for ; Sat, 17 Feb 2024 00:47:33 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Af9eQSws; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::52a as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-563e6131140so1566101a12.2 for ; Fri, 16 Feb 2024 16:47:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708130851; x=1708735651; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=58d9O57aavNMF3q/voTAptMSmNSsrA1pJADpLCtrSeg=; b=Af9eQSwsaX0jCsupxwDO7cujkd90PPU7RRWk7sZB6dl/SIEAMQc6tYM5V0KYszZOAu CA0ZD2s6aKsGxRhAFAjXRddMrN2KpZXllcAdrdp8+5OWc/PAznJ0J+8z2bryV4ff26J+ jjfVvIO4UI25iMriHqxa1TP4KdDhxNLLzvIOf16RR8jN53JhoElnA6+zJqNS1jJgU/Q8 Rt1eofoLT1sTbn6JHpXJJkMEEtH0PUiBPtWItPO975kMzlg4JztIlka389d0F/jFSBYG LZBdu2Ynty4h4ptd/9FuPim8uRZqp6vzegQ266cw+Ohmenm6AHQ3G07FsUE4/ey/TZ5E uRyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708130851; x=1708735651; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=58d9O57aavNMF3q/voTAptMSmNSsrA1pJADpLCtrSeg=; b=QCjuBPIP4g77iccG2EEjTHvuxZF9LDkxqpIXw3tNCG3EzcMMb5tYBWzbkKUxfXESGi Zy++TPR/z5vnh368kxtxxpV627Bia39+PoS9ZAWpG7YB7vx4b1mPD/MpUAVARTvG12k9 JpmIrtBI6KAkdDkt1fixYOgdYnLxuD38xZvEx0pC6/VeK8zu/BsEooA8iOAm3ZjDGoH2 E8pwDz9CLnu4Nhvdku3mvJp4o1m3mn4Z3yED4BRQZWu1LPX4gBYm9xrn8OpQvBlccBpC laTYYUtoDCjjOU0tsxZBjpNOoIAZuvmx2eK+p9Luipoa2xXda4QCLgu6KEKuna3LfCdJ 7VWA== X-Gm-Message-State: AOJu0YzpS7ij3SQ5HbxMR19CvTKr5KNF9U1mBEkQTWfdSoijpf3H/2Lx oAeZsNOaPmv6vhhHLD7P4Ur0RGydKZoGCfGqQdmE4PBI6PqXEMHmv8h3UwV5bg4xlY5vq/cICBD aueaQCU8lWDAyfUnnsgKLKUFMVyRbifS1kk0wXw== X-Google-Smtp-Source: AGHT+IEZ1L9ua1Xrb3WgQTm9HC7YkHuGvxmDXvG7dUv+iz+qoloawNhe4IBTkSQCoVe18h3ief2wOw61fkKJbw3QHoo= X-Received: by 2002:a17:907:175b:b0:a38:537:9598 with SMTP id lf27-20020a170907175b00b00a3805379598mr4343408ejc.17.1708130851198; Fri, 16 Feb 2024 16:47:31 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 17 Feb 2024 01:46:54 +0100 Message-ID: Subject: Re: How to send a bhyve vm totally in background,suppressing any message produced by the bhyve script that I use. To: Aryeh Friedman Cc: FreeBSD virtualization Content-Type: multipart/alternative; boundary="0000000000009d9c80061189347b" X-Rspamd-Queue-Id: 4Tc9BF25Yvz4v3B X-Spamd-Bar: - X-Spamd-Result: default: False [-1.98 / 15.00]; URI_COUNT_ODD(1.00)[5]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.976]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52a:from] --0000000000009d9c80061189347b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Errata corrige. The vm didn't start because my bhyve executable is on /usr/sbin ; but you keep it on /usr/bin ; So,the vm starts and it is stored in the background. Only one problem remains. The message "appending output to nohup.out" is displayed on the screen. Is there a method to suppress it ? thanks. On Sat, Feb 17, 2024 at 1:36=E2=80=AFAM Mario Marietto wrote: > This is how looks mine script after having applied your suggestions : > > nohup /usr/bin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H = -A \ > -s 0,hostbridge \ > -s > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex= =3D1 \ > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 13,virtio-net,tap19 \ > -s 14,virtio-9p,sharename=3D/ \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > vm0:19&>/dev/null > > well,it does not work : the vm does not start,I don't see its process as > active. > > output : > > final sleep > appending output to nohup.out > > root@marietto:/bhyve # ps ax | grep bhyve > 20220 1 S+ 0:00.00 grep bhyve > > On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedman > wrote: > >> On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto >> wrote: >> > >> > Hello. >> > >> > I want to boot a bhyve vm that does not produce any error message or >> warnings on the screen and that it stays in the background. This is how = I >> launch my Linux vm : >> > >> > bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \ >> > -s 0,hostbridge \ >> > -s >> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootindex= =3D1 \ >> > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ >> > -s 13,virtio-net,tap19 \ >> > -s 14,virtio-9p,sharename=3D/ \ >> > -s 30,xhci,tablet \ >> > -s 31,lpc \ >> > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ >> > vm0:19 > /dev/null & >> > >> > >> > I've thought that "> /dev/null" means that every message produced by >> the script would disappear,but,instead,it produces the following message= s >> that I want to suppress : >> > >> > marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0 >> > wrmsr to register 0x140(0) on vcpu 2 >> > wrmsr to register 0x140(0) on vcpu 4 >> > wrmsr to register 0x140(0) on vcpu 6 >> > wrmsr to register 0x140(0) on vcpu 1 >> > wrmsr to register 0x140(0) on vcpu 3 >> > wrmsr to register 0x140(0) on vcpu 5 >> > wrmsr to register 0x140(0) on vcpu 7 >> > rdmsr to register 0x64e on vcpu 1 >> > rdmsr to register 0x34 on vcpu 1 >> > rdmsr to register 0xc0011029 on vcpu 7 >> > Unhandled ps2 mouse command 0xe1 >> > Unhandled ps2 mouse command 0x0a >> > Unhandled ps2 mouse command 0x01 >> > Unhandled ps2 mouse command 0x41 >> > Unhandled ps2 mouse command 0x88 >> > rdmsr to register 0x64d on vcpu 5 >> > rdmsr to register 0x64d on vcpu 0 >> > >> > at this point it hangs. >> > >> > >> > I should press a key to have the ability to write again on the >> terminal. I don't want it. Someone can help me to refine the script ? >> thanks. >> >> Here is a fully tested/debugged hand made script to do what you >> requested (added bonus is the passthrough): >> >> #!/bin/sh >> # >> # Generated by PetiteCloud 0.2.6 >> # >> >> # nvidia1030.rom >> >> ifconfig tap1 destroy >> ifconfig tap1 create >> ifconfig tap1 up >> ifconfig bridge0 addm tap1 up >> echo net done >> sleep 5 >> bhyvectl --destroy --vm=3D6222f881ad268c17 >> nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s >> 0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s >> 29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -= s >> 5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l >> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd >> 6222f881ad268c17&>/dev/null >> echo final sleep >> sleep 10 >> >> > >> > -- >> > Mario. >> >> >> >> -- >> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org >> > > > -- > Mario. > --=20 Mario. --0000000000009d9c80061189347b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Errata corrige. The vm didn't start because my bh= yve executable is on /usr/sbin ; but you keep it on /usr/bin ;
So= ,the vm starts and it is stored in the background. Only one problem remains= . The message "appending output to nohup= .out" is displayed on the screen. Is there a method to suppress= it ? thanks.

On Sat, Feb 17, 2024 at 1:36=E2=80=AFAM Mario= Marietto <marietto2008@gmail.= com> wrote:
This is how looks mine script after having applied= your suggestions :

nohup /usr/bin/bhyve -S -c= sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \<= br>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootin= dex=3D1 \
-s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 13,virtio-net= ,tap19 \
-s 14,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \
-s 31= ,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19&>/dev/null

well,it does not wo= rk : the vm does not start,I don't see its process as active.

output :

final sleep
appending output to nohup.out
=C2=A0
root@marietto:/bhyve # ps ax | grep bhyve
20220 =C2=A01 =C2=A0S+ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00:00.00 grep= bhyve

On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedm= an <aryeh.= friedman@gmail.com> wrote:
On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto &l= t;marietto2008@= gmail.com> wrote:
>
> Hello.
>
> I want to boot a bhyve vm that does not produce any error message or w= arnings on the screen and that it stays in the background. This is how I la= unch my Linux vm :
>
> bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
> -s 0,hostbridge \
> -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,boot= index=3D1 \
> -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
> -s 13,virtio-net,tap19 \
> -s 14,virtio-9p,sharename=3D/ \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
> vm0:19 > /dev/null &
>
>
> I've thought that "> /dev/null" means that every mess= age produced by the script would disappear,but,instead,it produces the foll= owing messages that I want to suppress :
>
> marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0
> wrmsr to register 0x140(0) on vcpu 2
> wrmsr to register 0x140(0) on vcpu 4
> wrmsr to register 0x140(0) on vcpu 6
> wrmsr to register 0x140(0) on vcpu 1
> wrmsr to register 0x140(0) on vcpu 3
> wrmsr to register 0x140(0) on vcpu 5
> wrmsr to register 0x140(0) on vcpu 7
> rdmsr to register 0x64e on vcpu 1
> rdmsr to register 0x34 on vcpu 1
> rdmsr to register 0xc0011029 on vcpu 7
> Unhandled ps2 mouse command 0xe1
> Unhandled ps2 mouse command 0x0a
> Unhandled ps2 mouse command 0x01
> Unhandled ps2 mouse command 0x41
> Unhandled ps2 mouse command 0x88
> rdmsr to register 0x64d on vcpu 5
> rdmsr to register 0x64d on vcpu 0
>
> at this point it hangs.
>
>
> I should press a key to have the ability to write again on the termina= l. I don't want it. Someone can help me to refine the script ? thanks.<= br>
Here is a fully tested/debugged hand made script to do what you
requested (added bonus is the passthrough):

#!/bin/sh
#
# Generated by PetiteCloud 0.2.6
#

# nvidia1030.rom

ifconfig tap1 destroy
ifconfig tap1 create
ifconfig tap1 up
ifconfig bridge0 addm tap1 up
echo net done
sleep 5
bhyvectl --destroy --vm=3D6222f881ad268c17
nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s
0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s
29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -s
5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
6222f881ad268c17&>/dev/null
echo final sleep
sleep 10

>
> --
> Mario.



--
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org


--
Mario.


--
Mario.
--0000000000009d9c80061189347b-- From nobody Sat Feb 17 01:19:57 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc9wQ4tc4z5BGB1 for ; Sat, 17 Feb 2024 01:20:38 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc9wP51dTz40R7 for ; Sat, 17 Feb 2024 01:20:37 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=eRwNcpzB; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-a3122b70439so307707666b.3 for ; Fri, 16 Feb 2024 17:20:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708132834; x=1708737634; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5vTdbDK66ks1pk0XJ/5FhW3q8s7BDRepCnUDIrSIQXo=; b=eRwNcpzBhP4uLYkny3/wkiRHr0IZkncJYhsQ5wUWltBPRvXQPxz3uiA4OYkuki9dov kQSZbHsjdReMfbcnnYH6XzMLQ6XSCnLMGUg4XyeBJT+h5YDChGkgOO4xabD+T04Y2vqT tDGZU1jB7WipqfKfPAFRDQCWytDyz3k1yFQDA1dWZm6CNEmi6HVa9iGnRqZuh1rjRs2W PKKJ0QfBlcudy3MR9D+9+OK0n4HWXR5kwAdLko+CVsWsVFnzA6WdM7fyv9Rc/22DdInV mof4Vu1/nnpoHGDRRagYXnb8kcZBG/VjALVVBUjS8GdCDX5xDUKiBfr7a8BJEcFYaKce nKIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708132834; x=1708737634; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5vTdbDK66ks1pk0XJ/5FhW3q8s7BDRepCnUDIrSIQXo=; b=mTb8NJ0Wsbc0h2QC5/bzUXVk3dRv064Td0a+3w6coFGnPMnX/nYUmbDdzSVqu/EOCF nMNweOttxaIyOR5wsswYDPXZfnxOFwtXOfEYk06pWkcMPBJ/YYEQX9cq43WVzC4RFfuk hICNmFy0OlLQju9sAfsX8Q9cyq0u2GBZryzQ0EL+OKEu9aYTJsp0BKJs3fKCxQgdK/lh +l+OUpyJSM772JHVC7iTU0BI7+84bTwlBuqAtgeTxsS6pLi6i8lhuZKAdqzbT4cLXxmI 1L7hW8GP9cyeFFLHy3TzdJlhKvWehs9svpY/59/iMj1ZcU/QSLoUIInjn6nQNiIKZ7a1 TKUg== X-Gm-Message-State: AOJu0Ywn/XcVY+n/fcOf+MEqDXBZmy1E7jENBvW9BystS+jYtE0fyZD2 qGaFfmkqDUZE8J7Hn1xPa3/mKJvOuJ0an2oliQ8JJQpRwQAKL7kV88jFgWf4F6hH2alFygpCeGY 3Xr0uMfIPE51vqxm8twoqR6PRD5dfCXklJNt85w== X-Google-Smtp-Source: AGHT+IHAV/jDQSJ0sKpq2s7jYcfRHa69IDUqW2omYYJz1kEYQCwbfbInxs8ldXHcZBHwdc0UuoISHJAGWkbWyDRg31U= X-Received: by 2002:a17:906:3757:b0:a3d:9e6b:2776 with SMTP id e23-20020a170906375700b00a3d9e6b2776mr4310143ejc.17.1708132834236; Fri, 16 Feb 2024 17:20:34 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 17 Feb 2024 02:19:57 +0100 Message-ID: Subject: Re: How to send a bhyve vm totally in background,suppressing any message produced by the bhyve script that I use. To: Aryeh Friedman Cc: FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000d05f91061189aa42" X-Rspamd-Queue-Id: 4Tc9wP51dTz40R7 X-Spamd-Bar: - X-Spamd-Result: default: False [-2.00 / 15.00]; URI_COUNT_ODD(1.00)[5]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; RCVD_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] --000000000000d05f91061189aa42 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I fixed the script and now it works exactly as I want : nohup /usr/sbin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -= A \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex=3D= 1 \ -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ -s 13,virtio-net,tap19 \ -s 14,virtio-9p,sharename=3D/ \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:19/dev/null 2>&1 & if test -f nohup.out; then rm -r nohup.out fi very thanks for your help. On Sat, Feb 17, 2024 at 1:46=E2=80=AFAM Mario Marietto wrote: > Errata corrige. The vm didn't start because my bhyve executable is on > /usr/sbin ; but you keep it on /usr/bin ; > So,the vm starts and it is stored in the background. Only one problem > remains. The message "appending output to nohup.out" is displayed on the > screen. Is there a method to suppress it ? thanks. > > On Sat, Feb 17, 2024 at 1:36=E2=80=AFAM Mario Marietto > wrote: > >> This is how looks mine script after having applied your suggestions : >> >> nohup /usr/bin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H= -A \ >> -s 0,hostbridge \ >> -s >> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex= =3D1 \ >> -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ >> -s 13,virtio-net,tap19 \ >> -s 14,virtio-9p,sharename=3D/ \ >> -s 30,xhci,tablet \ >> -s 31,lpc \ >> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ >> vm0:19&>/dev/null >> >> well,it does not work : the vm does not start,I don't see its process as >> active. >> >> output : >> >> final sleep >> appending output to nohup.out >> >> root@marietto:/bhyve # ps ax | grep bhyve >> 20220 1 S+ 0:00.00 grep bhyve >> >> On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedman >> wrote: >> >>> On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto >>> wrote: >>> > >>> > Hello. >>> > >>> > I want to boot a bhyve vm that does not produce any error message or >>> warnings on the screen and that it stays in the background. This is how= I >>> launch my Linux vm : >>> > >>> > bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \ >>> > -s 0,hostbridge \ >>> > -s >>> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,bootinde= x=3D1 \ >>> > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ >>> > -s 13,virtio-net,tap19 \ >>> > -s 14,virtio-9p,sharename=3D/ \ >>> > -s 30,xhci,tablet \ >>> > -s 31,lpc \ >>> > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ >>> > vm0:19 > /dev/null & >>> > >>> > >>> > I've thought that "> /dev/null" means that every message produced by >>> the script would disappear,but,instead,it produces the following messag= es >>> that I want to suppress : >>> > >>> > marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0 >>> > wrmsr to register 0x140(0) on vcpu 2 >>> > wrmsr to register 0x140(0) on vcpu 4 >>> > wrmsr to register 0x140(0) on vcpu 6 >>> > wrmsr to register 0x140(0) on vcpu 1 >>> > wrmsr to register 0x140(0) on vcpu 3 >>> > wrmsr to register 0x140(0) on vcpu 5 >>> > wrmsr to register 0x140(0) on vcpu 7 >>> > rdmsr to register 0x64e on vcpu 1 >>> > rdmsr to register 0x34 on vcpu 1 >>> > rdmsr to register 0xc0011029 on vcpu 7 >>> > Unhandled ps2 mouse command 0xe1 >>> > Unhandled ps2 mouse command 0x0a >>> > Unhandled ps2 mouse command 0x01 >>> > Unhandled ps2 mouse command 0x41 >>> > Unhandled ps2 mouse command 0x88 >>> > rdmsr to register 0x64d on vcpu 5 >>> > rdmsr to register 0x64d on vcpu 0 >>> > >>> > at this point it hangs. >>> > >>> > >>> > I should press a key to have the ability to write again on the >>> terminal. I don't want it. Someone can help me to refine the script ? >>> thanks. >>> >>> Here is a fully tested/debugged hand made script to do what you >>> requested (added bonus is the passthrough): >>> >>> #!/bin/sh >>> # >>> # Generated by PetiteCloud 0.2.6 >>> # >>> >>> # nvidia1030.rom >>> >>> ifconfig tap1 destroy >>> ifconfig tap1 create >>> ifconfig tap1 up >>> ifconfig bridge0 addm tap1 up >>> echo net done >>> sleep 5 >>> bhyvectl --destroy --vm=3D6222f881ad268c17 >>> nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s >>> 0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s >>> 29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc = -s >>> 5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l >>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd >>> 6222f881ad268c17&>/dev/null >>> echo final sleep >>> sleep 10 >>> >>> > >>> > -- >>> > Mario. >>> >>> >>> >>> -- >>> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org >>> >> >> >> -- >> Mario. >> > > > -- > Mario. > --=20 Mario. --000000000000d05f91061189aa42 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I fixed the script and now it works exactly as I want= :

nohup /usr/sbin/b= hyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,booti= ndex=3D1 \
-s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 13,virtio-net,tap19 \
-s 14,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19</dev/null >/dev/null 2>&1 &
if test -f nohup.out; then rm -r nohup.out
fi

very thanks for your help.

On Sat, Feb 17, 2024 at 1:46=E2=80=AFAM Mario Marietto <= marietto2008@gmail.com> wr= ote:
Errata corrige. The vm didn't start because my bhyve executa= ble is on /usr/sbin ; but you keep it on /usr/bin ;
So,the vm sta= rts and it is stored in the background. Only one problem remains. The messa= ge "appending output to nohup.out= " is displayed on the screen. Is there a method to suppress it ? thank= s.

On Sat, Feb 17, 2024 at 1:36=E2=80=AFAM Mario Marietto &= lt;marietto2008= @gmail.com> wrote:
This is how looks mine script after having= applied your suggestions :

nohup /usr/bin/bhy= ve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostb= ridge \
-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.im= g,bootindex=3D1 \
-s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 13,vi= rtio-net,tap19 \
-s 14,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \<= br>-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE= .fd \
vm0:19&>/dev/null

well,it doe= s not work : the vm does not start,I don't see its process as active.

ou= tput :

final sleep
appending output to nohup.out
=C2=A0
root@marietto:/bhyve # ps ax | grep bhyve
20220 =C2=A01 =C2=A0S+ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00:00.00 grep= bhyve

On Sat, Feb 17, 2024 at 1:18=E2=80=AFAM Aryeh Friedm= an <aryeh.= friedman@gmail.com> wrote:
On Fri, Feb 16, 2024 at 7:05=E2=80=AFPM Mario Marietto &l= t;marietto2008@= gmail.com> wrote:
>
> Hello.
>
> I want to boot a bhyve vm that does not produce any error message or w= arnings on the screen and that it stays in the background. This is how I la= unch my Linux vm :
>
> bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
> -s 0,hostbridge \
> -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2304.img,boot= index=3D1 \
> -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
> -s 13,virtio-net,tap19 \
> -s 14,virtio-9p,sharename=3D/ \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
> vm0:19 > /dev/null &
>
>
> I've thought that "> /dev/null" means that every mess= age produced by the script would disappear,but,instead,it produces the foll= owing messages that I want to suppress :
>
> marietto@marietto:/bhyve # wrmsr to register 0x140(0) on vcpu 0
> wrmsr to register 0x140(0) on vcpu 2
> wrmsr to register 0x140(0) on vcpu 4
> wrmsr to register 0x140(0) on vcpu 6
> wrmsr to register 0x140(0) on vcpu 1
> wrmsr to register 0x140(0) on vcpu 3
> wrmsr to register 0x140(0) on vcpu 5
> wrmsr to register 0x140(0) on vcpu 7
> rdmsr to register 0x64e on vcpu 1
> rdmsr to register 0x34 on vcpu 1
> rdmsr to register 0xc0011029 on vcpu 7
> Unhandled ps2 mouse command 0xe1
> Unhandled ps2 mouse command 0x0a
> Unhandled ps2 mouse command 0x01
> Unhandled ps2 mouse command 0x41
> Unhandled ps2 mouse command 0x88
> rdmsr to register 0x64d on vcpu 5
> rdmsr to register 0x64d on vcpu 0
>
> at this point it hangs.
>
>
> I should press a key to have the ability to write again on the termina= l. I don't want it. Someone can help me to refine the script ? thanks.<= br>
Here is a fully tested/debugged hand made script to do what you
requested (added bonus is the passthrough):

#!/bin/sh
#
# Generated by PetiteCloud 0.2.6
#

# nvidia1030.rom

ifconfig tap1 destroy
ifconfig tap1 create
ifconfig tap1 up
ifconfig bridge0 addm tap1 up
echo net done
sleep 5
bhyvectl --destroy --vm=3D6222f881ad268c17
nohup /usr/sbin/bhyve -S -c cores=3D8 -m 8192 -AI -H -P -w -s
0:0,hostbridge -s 1,virtio-net,tap1 -s 2,ahci-hd,/dev/ada0 -s
29,fbuf,tcp=3D0.0.0.0:6048,w=3D800,h=3D600 -s 30,xhci,tablet -s 31,lpc -s
5,passthru,37/0/0 -s 6,passthru,37/0/1 -l com1,/dev/nmdm1B -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
6222f881ad268c17&>/dev/null
echo final sleep
sleep 10

>
> --
> Mario.



--
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org


--
Mario.


--
Mario.


--
Mario.
--000000000000d05f91061189aa42-- From nobody Sat Feb 17 21:53:10 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TcjGm2CVDz59ntP for ; Sat, 17 Feb 2024 21:53:20 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TcjGl2DKCz4xw9 for ; Sat, 17 Feb 2024 21:53:19 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=hEJv+ixX; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41HLrBKl062450 for ; Sat, 17 Feb 2024 15:53:11 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708206791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=eLmpFBk/n8cFe/W+Qcl5WTr01GVr3tnc4szjIB4dzKY=; b=hEJv+ixXFFelKKtRkN/boD5CEo9h6XbW3kslYyz+tMLfg7rg3VAn+nkHvp0vM7AcwSH+5a IBLn6aFovHtxaWpuNkUo25NxUfgnXakv9ZGF0Z7zTYLFjO4duWshXeLOzMC3A0DUPNg46k 4xtpiNJaNJzMFGKQSjL8k2TCVJeyUTS5//Zv4NxdrSZkPF96YYDcC/CYfMbZPtp9JSxMYW UITv9WO39lkz+HMlDlciqE6i58iBzHgscWabQGmWil97IiSnYUFbTGYqB6T9SW2TjwLvo6 Dyx7cZCcGE3R4dtLPUYgN9yyqP8Ybi/N73f/SOydpZpKhaSYXWLlBa5FQuMAPw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 68E713AB37 for ; Sat, 17 Feb 2024 15:53:11 -0600 (CST) Content-Type: multipart/alternative; boundary="------------rzBtn40afBj06nBx93HWKb1k" Message-ID: <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> Date: Sat, 17 Feb 2024 15:53:10 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> From: Matthew Grooms In-Reply-To: <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> X-Rspamd-Queue-Id: 4TcjGl2DKCz4xw9 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] This is a multi-part message in MIME format. --------------rzBtn40afBj06nBx93HWKb1k Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/16/24 12:00, Matthew Grooms wrote: > On 2/16/24 11:42, Chuck Tuffli wrote: >> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>> >>> Hi All, >>> >>> >>> I'm in the middle of a project that involves building out a handful >>> of servers to host virtual Linux instances. Part of that includes >>> testing bhyve to see how it performs. The intent is to compare host >>> storage options such as raw vs zvol block devices and ufs vs zfs >>> disk images using hardware raid vs zfs managed disks. It would also >>> involve >>> >>> >> … >>> >>> Here is a list of a few other things I'd like to try: >>> >>> >>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>> 2) Downgrading the host to 13.2-RELEASE >> >> FWIW we recently did a similar exercise and saw significant >> performance differences on ZFS backed disk images when comparing 14.0 >> and 13.2. We didn’t have time to root cause the difference, so it >> could simply be some tuning difference needed for 14. > > Hi Chuck, > > That's very helpful feedback. I'll start by downgrading the host to > 13.2 and report back here. > Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better. ================================================================================ Begin @ Sat Feb 17 11:00:01 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g 68 +++++ +++ Latency             11759us   29114us    8098us    8649us 23413us    4540us Version  2.00       ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              7791us     131us    1671us     464us 15us    1811us End @ Sat Feb 17 11:03:13 CST 2024 ================================================================================ Begin @ Sat Feb 17 11:10:01 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m 99 16204 563 Latency             12147us    1079us   24470us    8795us 17853us    4384us Version  2.00       ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++ Latency               118us     159us     269us     164us 54us     844us End @ Sat Feb 17 11:18:43 CST 2024 --------------rzBtn40afBj06nBx93HWKb1k Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/16/24 12:00, Matthew Grooms wrote:
On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.


Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better.

================================================================================
Begin @ Sat Feb 17 11:00:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g  68 +++++ +++
Latency             11759us   29114us    8098us    8649us   23413us    4540us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7791us     131us    1671us     464us      15us    1811us

End @ Sat Feb 17 11:03:13 CST 2024
================================================================================
Begin @ Sat Feb 17 11:10:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m  99 16204 563
Latency             12147us    1079us   24470us    8795us   17853us    4384us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++
Latency               118us     159us     269us     164us      54us     844us

End @ Sat Feb 17 11:18:43 CST 2024

--------------rzBtn40afBj06nBx93HWKb1k-- From nobody Sun Feb 18 12:48:09 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Td57G3CtJz59YpW for ; Sun, 18 Feb 2024 12:48:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Td57G1CHhz4hLw for ; Sun, 18 Feb 2024 12:48:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708260490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jGdX6BGuNZhEMsuHHEKrv5RxnhwBMxu9fo0xrD96bmc=; b=LrClIeRi87T04TW7tJcAJ8QoWJJDIo4iWoZD8mW8b+sFdOAy4kFNefahfZkm0z6AtWERXM zTfRAOWGiC6nJf6Fu/rkg8B9QaW4s2khUiURSLXIqSTn1/JRf+LfLCqold0SR7LCv7A561 7BJghR4afWNFsnmdDXfShpBhBCiYBqHBss4pfVZ1yhLWLx+z5/Lmv85W5jy3LGY57/3P7w boOnRLq0RJ2t6jmWYVmZgnXzH+F4UHw7zzrTK5ah2n1k5IeyZa/jECvoHkLcRi1bx/umfp EHePPFnPGJqnffnzJJu14ol6QicIki334LSnL3u9hr6Z3q441p0Vaaawunyk8g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708260490; a=rsa-sha256; cv=none; b=oQtvhtiw2s2/My1tjSL11V0lL0L6y+sjX7UGsaVwWRKcApaIaQ6GzldjX/d09QPREV0wcC fOmwWSlAseUz92JyZzXEqsGqBfMKJ8ggJ8uhzneVJdx9qXDCdDT/FcP6zfmHwPlDIfokRf +5MsoHSy4h/CwFDG6FyUoLCMkv8tKZ/RlkoD5L92RjEnELthFhe5v0K0X37FV4C5raOYbr lcGLWdPC5w4ueherTMfu99h+FFx2haGZjxTrmM/gELo6yagds/cBAGGOdgkPKWH4BfI7+8 6oag7OhnZuwhjCTEpu+njPD8kLHbQ+9jQd4nmjwGQZbonqhFhLEg+MX75+wWlg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Td57G08wgzrJ8 for ; Sun, 18 Feb 2024 12:48:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41ICm9oG046000 for ; Sun, 18 Feb 2024 12:48:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41ICm9dm045999 for virtualization@FreeBSD.org; Sun, 18 Feb 2024 12:48:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277128] IPV6 public address lost after networking driver reload OR attaching a new network interface Date: Sun, 18 Feb 2024 12:48:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277128 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |virtualization@FreeBSD.org --- Comment #1 from Mark Linimon --- ^Triage: note that this is about EC2 instances. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 18 15:21:10 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Td8Wt3Znlz59xJt for ; Sun, 18 Feb 2024 15:21:14 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Td8Ws3mKvz46pl for ; Sun, 18 Feb 2024 15:21:13 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=t+KSVmxs; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41IFLBht071852 for ; Sun, 18 Feb 2024 09:21:11 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708269671; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=mLckGg/35HCYCGmXjDSvMooeZYNmSYcHT94WvQeeyDo=; b=t+KSVmxs6VncGiqjLFO/wNMv3CLAoDIoGBcq3We8NyPu8aYtVLskyGk8Y+hJ3cupZSrUAx jsHCVqPuCiNpq2o08GQ/AtCZ0kvO6JzS+Nnb9QoCzMHiVax18lWC32BojIJuaySje/BRRq dMVSgp+jN96t5G6SEv6VeA93fpabd1CvXr0mMa1Fb3RZH0ChOoNr4Owjw2OjVVM91cTLiI hopN9s0Jh+ahBC2mznyLXM2na+x7RgA3DYpNFwBCvAjB99xXp0oxbeMwLJ1BTBNNmim6d1 SgIxW1SwpVsaAL9do9WfJ9v1ejMxj22ORTJHFxWsyEB+avBF8+dOoh41lCYu3w== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 4B6193B553 for ; Sun, 18 Feb 2024 09:21:11 -0600 (CST) Content-Type: multipart/alternative; boundary="------------aSmCDs7yhrXrJYYlTx0uyb8U" Message-ID: <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> Date: Sun, 18 Feb 2024 09:21:10 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> From: Matthew Grooms In-Reply-To: <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> X-Rspamd-Queue-Id: 4Td8Ws3mKvz46pl X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] This is a multi-part message in MIME format. --------------aSmCDs7yhrXrJYYlTx0uyb8U Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/17/24 15:53, Matthew Grooms wrote: > On 2/16/24 12:00, Matthew Grooms wrote: >> On 2/16/24 11:42, Chuck Tuffli wrote: >>> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>>> >>>> Hi All, >>>> >>>> >>>> I'm in the middle of a project that involves building out a handful >>>> of servers to host virtual Linux instances. Part of that includes >>>> testing bhyve to see how it performs. The intent is to compare host >>>> storage options such as raw vs zvol block devices and ufs vs zfs >>>> disk images using hardware raid vs zfs managed disks. It would also >>>> involve >>>> >>>> >>> … >>>> >>>> Here is a list of a few other things I'd like to try: >>>> >>>> >>>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>>> 2) Downgrading the host to 13.2-RELEASE >>> >>> FWIW we recently did a similar exercise and saw significant >>> performance differences on ZFS backed disk images when comparing >>> 14.0 and 13.2. We didn’t have time to root cause the difference, so >>> it could simply be some tuning difference needed for 14. >> >> Hi Chuck, >> >> That's very helpful feedback. I'll start by downgrading the host to >> 13.2 and report back here. >> > > Unfortunately same story with 13.2. I'm going to try and downgrade to > 12.4 and see if it gets any better. > > ================================================================================ > Begin @ Sat Feb 17 11:00:01 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP > localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  68 > +++++ +++ > Latency             11759us   29114us    8098us    8649us 23413us    > 4540us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP >                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency              7791us     131us    1671us     464us 15us    1811us > > End @ Sat Feb 17 11:03:13 CST 2024 > ================================================================================ > Begin @ Sat Feb 17 11:10:01 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP > localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  99 > 16204 563 > Latency             12147us    1079us   24470us    8795us 17853us    > 4384us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP >                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ > +++++ +++ > Latency               118us     159us     269us     164us 54us     844us > > End @ Sat Feb 17 11:18:43 CST 2024 > I wasn't able to get a working 12.4 system in place due to lack of packages. However, I did fire up a FreeBSD 14 VM and let it run overnight on the same SSD array. It consistently ran at a much higher speed for 50+ runs @ 10m intervals ... ================================================================================ Begin @ Sun Feb 18 15:00:00 UTC 2024 Version  1.98       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99  1.1g 42 +++++ +++ Latency             13447us   68490us     207ms    7187us 195ms   17665us Version  1.98       ------Sequential Create------ --------Random Create-------- freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency             18225us      18us      28us   18812us 18us      25us End @ Sun Feb 18 15:03:11 UTC 2024 I used identical options to run both that VM and the Linux VM I've been testing. The backing store for each VM has a 1TB partition and the guest interfaces are NVME. Now I'm really scratching my head. Chuck, were you testing disk performance in Linux VMs or only FreeBSD? Anyone have ideas on why Linux disk performance would drop off a cliff over time? Thanks, Thanks, -Matthew --------------aSmCDs7yhrXrJYYlTx0uyb8U Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/17/24 15:53, Matthew Grooms wrote:
On 2/16/24 12:00, Matthew Grooms wrote:
On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.


Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better.

================================================================================
Begin @ Sat Feb 17 11:00:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g  68 +++++ +++
Latency             11759us   29114us    8098us    8649us   23413us    4540us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7791us     131us    1671us     464us      15us    1811us

End @ Sat Feb 17 11:03:13 CST 2024
================================================================================
Begin @ Sat Feb 17 11:10:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m  99 16204 563
Latency             12147us    1079us   24470us    8795us   17853us    4384us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++
Latency               118us     159us     269us     164us      54us     844us

End @ Sat Feb 17 11:18:43 CST 2024

I wasn't able to get a working 12.4 system in place due to lack of packages. However, I did fire up a FreeBSD 14 VM and let it run overnight on the same SSD array. It consistently ran at a much higher speed for 50+ runs @ 10m intervals ...

================================================================================
Begin @ Sun Feb 18 15:00:00 UTC 2024

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99  1.1g  42 +++++ +++
Latency             13447us   68490us     207ms    7187us     195ms   17665us
Version  1.98       ------Sequential Create------ --------Random Create--------
freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             18225us      18us      28us   18812us      18us      25us

End @ Sun Feb 18 15:03:11 UTC 2024

I used identical options to run both that VM and the Linux VM I've been testing. The backing store for each VM has a 1TB partition and the guest interfaces are NVME. Now I'm really scratching my head.

Chuck, were you testing disk performance in Linux VMs or only FreeBSD?

Anyone have ideas on why Linux disk performance would drop off a cliff over time?

Thanks,

Thanks,

-Matthew

--------------aSmCDs7yhrXrJYYlTx0uyb8U-- From nobody Sun Feb 18 20:06:00 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TdGrS6YnTz5Bd9x for ; Sun, 18 Feb 2024 20:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TdGrS49b6z4nBM for ; Sun, 18 Feb 2024 20:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708286760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MtAesYtSwQsK2Vp7Ts3E58579pPRLsRfO1G8RDIHMJU=; b=K0xyONnv0qRmEfi86BkYLLYyAn8Oj6E/cOfBAPnSoynl/6IP0CL58hAXUqVuK118tTOHTy 0Gwu5grSPh2xS61Bj8TWG4sUi/k8vceJutzGCmXdD3FBnBy9JGznIOe11o454mQzCeXkyD WeyekM9bSM/pbVbLRyqJCYwPD2FN0rkSaP+gxMPqor2+g1/tzUHdM3MvFfo27t7nMsBHzC CNRidHIZi+6zWNS8+xdiw3MvM2lIhgVA4W1XG1gyVWnpKYAMG0FrUpUngwjNROShpJhi0N 0gZXwy0P6Pdo8dO2W9mXFjVJAsdLEJC1BfKaGs79iJfHrpFP78i4LN+y7pELVw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708286760; a=rsa-sha256; cv=none; b=cDpUcfdBnDYInsTpkVHF01ydkKey7RvXLCO+kItTJ80dxdKuEQszmXfHOjERPMcO4nZ2lc 3Kf5d7yvKu0P6vX1mzX54lqh9wNYwy18Dp2KQ4fCeLZbY+mIJGV+WDS0waiFHfjNNKoUK5 FSOzr38K4lH1iGS8khsXLQfx0CcQWqMWDY1/NBa4SiHZp1dIAevQwDNqJ1k/Rm/ZS3IuCu UOssUFzwE4PRMC3lr2MfFMILMbHKJvYpcu2jVHcgZBTC9AISUohNPLwbEJ+B/DISJtHVmE fd7w1e9aCZDWPjY8Sd0SSaLcL1UR5iYOLLNihWuDfHJkMSMbKAH6C4bGVhDhkQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TdGrS38Ckz14FH for ; Sun, 18 Feb 2024 20:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41IK6074025457 for ; Sun, 18 Feb 2024 20:06:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41IK60DY025452 for virtualization@FreeBSD.org; Sun, 18 Feb 2024 20:06:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277128] IPV6 public address lost after networking driver reload OR attaching a new network interface Date: Sun, 18 Feb 2024 20:06:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: osamaabb@amazon.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277128 --- Comment #2 from Osama Abboud --- (In reply to Osama Abboud from comment #0) PS:The ------Reboot------ in issue No.2 was meant to be kldunload & kldload --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 18 21:00:14 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TdJ33274Gz5Bl1R for ; Sun, 18 Feb 2024 21:00:15 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TdJ324Vsgz4tc3 for ; Sun, 18 Feb 2024 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708290014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kiuomM+fkgGc2wedI0dNv9xJ9AutRlUrW69P2YgWyhE=; b=KTKBu22i5nhmlFXFiJYeHxozy2o2Z/7ZlRF56Dg8+Vlp66ilZbhnE8f4oxljUXiCGYShWM n+WM1rA2rj2s2iU+CWn8sNMJrYg4s1GgUUysLl2FZaYQHXYb9pRkPT4IdU4vWlfcHaVtUD 793cJh9vSrQzgzXtueRw39eXnFGoJXjustLDbf293h/ohgk8XRR/lkSr7EZRpDDopU4Vsy NDjUblZJ+Cmg4Elb8/uJ76Z/hyQyvpz84almzMMReyTBEbP527W9cEnqo3TmTA+CY981kd Xz3A4Myhb6RfiswPBP9mV9RYQK0rcGrbnsBzyrUCyRksQabDKDB0AuQJhO5zyQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708290014; a=rsa-sha256; cv=none; b=rsJD67Fz3cEq/oCfy8RjdgY+Its6X6z9a7+wv0rXfCfPku2Bm2R+wvXmRLmi9ig4xXh4Fi 53Jx/4nC/JoHMJOKOb7KjJT3lBSxNXOaB8whgbYAbs+r2kR/e9fLrnleD5ZJ8zcOMRI2kv OohGUZ89VzAYBqfa2k2vut1ElS2yUOtMlbIL//Vqw7krbLi2UeOTNKaG6bZ6imMa3YzyY8 6yZypltTbvvePXB8JNno43lxa7CbgAtURKyQ9BtcHtRYvI1mIglvLrpRTGg0rn7J67NLfE aAxHHTTLHDNgl1e4WGBVVaM5edV2imNmNOsgaVp4Kzoq314sPwTfQHWK7X7HXw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TdJ323bLpz15wM for ; Sun, 18 Feb 2024 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41IL0EsO000054 for ; Sun, 18 Feb 2024 21:00:14 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41IL0EJ5000052 for virtualization@FreeBSD.org; Sun, 18 Feb 2024 21:00:14 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202402182100.41IL0EJ5000052@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 18 Feb 2024 21:00:14 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17082900142.fDC97c.96481" Content-Transfer-Encoding: 7bit --17082900142.fDC97c.96481 Date: Sun, 18 Feb 2024 21:00:14 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 264253 | hyperv: ifconfig media autoselect on FreeBSD 13.0 Open | 264267 | UEFI Booting on Azure Generation 2 VMs crashes 2 problems total for which you should take action. --17082900142.fDC97c.96481 Date: Sun, 18 Feb 2024 21:00:14 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    264253 | hyperv: ifconfig media autoselect on FreeBSD 13.0
Open        |    264267 | UEFI Booting on Azure Generation 2 VMs crashes

2 problems total for which you should take action.
--17082900142.fDC97c.96481--