From nobody Thu Jun 15 21:42:35 2023
X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qhwkc18zpz4fXnC
	for <freebsd-virtualization@mlmmj.nyi.freebsd.org>; Thu, 15 Jun 2023 21:42:48 +0000 (UTC)
	(envelope-from jason@tubnor.net)
Received: from mail.tubnor.net (mail.tubnor.net [103.236.162.16])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Qhwkb21jxz4LhN
	for <freebsd-virtualization@FreeBSD.org>; Thu, 15 Jun 2023 21:42:47 +0000 (UTC)
	(envelope-from jason@tubnor.net)
Authentication-Results: mx1.freebsd.org;
	none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tubnor.net; s=20220915;
	t=1686865354;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=Wuexlp/Nfy/DlbZrEGtuDM6mvExFm5rwOSSCmAxh5Dk=;
	b=V08mvtcJSz8MDFO+PBYrXhytSfv3QcRIruag1mad5vKZwVa+6q7KFPSjDF9lrA5Ny5Wxv3
	e6RAGoElnc6E8EXQNf9F7qdogoJXqQNQC6INioj64my7JA7n6gdZhI5KOOBOSXGZ7soh9z
	SxV8h2TsY84IP5GMCBBa2ZY7mLSkW8E=
Received: from THEMASTER (<unknown> [2403:5812:73e6:1:6807:b315:69c1:b336])
	by mel01.ar18.net (OpenSMTPD) with ESMTPSA id 99be6499 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO);
	Fri, 16 Jun 2023 07:42:34 +1000 (AEST)
From: <jason@tubnor.net>
To: "'Sean McBride'" <sean@rogue-research.com>,
	<freebsd-virtualization@FreeBSD.org>
References: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com>
In-Reply-To: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com>
Subject: RE: Ubuntu in bhyve
Date: Fri, 16 Jun 2023 07:42:35 +1000
Message-ID: <002401d99fd2$4c6d2470$e5476d50$@tubnor.net>
List-Id: Discussion <freebsd-virtualization.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization
List-Help: <mailto:virtualization+help@freebsd.org>
List-Post: <mailto:virtualization@freebsd.org>
List-Subscribe: <mailto:virtualization+subscribe@freebsd.org>
List-Unsubscribe: <mailto:virtualization+unsubscribe@freebsd.org>
Sender: owner-freebsd-virtualization@freebsd.org
X-BeenThere: freebsd-virtualization@freebsd.org
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0025_01D9A026.1E19F7C0"
X-Mailer: Microsoft Outlook 16.0
Content-Language: en-au
Thread-Index: AQG7j4KiCHDB8k9TVed3dRnx5vsdbK/IhHmA
X-Rspamd-Queue-Id: 4Qhwkb21jxz4LhN
X-Spamd-Bar: ----
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:133159, ipnet:103.236.162.0/23, country:AU]
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-ThisMailContainsUnwantedMimeParts: N

This is a multipart message in MIME format.

------=_NextPart_000_0025_01D9A026.1E19F7C0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Make sure your guests (and host) are syncing to a reliable time source. I
have heard time to time of some having this issue, but we haven't seen it
though we do use OpenNTPD across our BSD/*nix guests (windows has its own
time sync service).

 

Cheers,

 

Jason.

 

From: owner-freebsd-virtualization@freebsd.org
<owner-freebsd-virtualization@freebsd.org> On Behalf Of Sean McBride
Sent: Friday, June 16, 2023 5:20 AM
To: freebsd-virtualization@FreeBSD.org
Subject: Ubuntu in bhyve

 

Hi all,

I am running bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04 and
22.04 guests.

One of them rather regularly seizes up hard, where I cannot ssh to it, then
after a few minutes works again. This happens repeatedly and on and off. In
the Ubuntu logs I see messages like:

clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as
unstable because the skew is too large:
clocksource:                       'hpet' wd_nsec: 536417782 wd_now:
638cb3ff wd_last: 63036152 mask: ffffffff
clocksource:                       'tsc' cs_nsec: 536821277 cs_now:
225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff
clocksource:                       'tsc' is current clocksource.
tsc: Marking TSC unstable due to clocksource watchdog
TSC found unstable after boot, most likely due to broken BIOS. Use
'tsc=unstable'.
sched_clock: Marking unstable (12899572906362, 9192186981)<-(12908811630850,
-46250627)
clocksource: Checking clocksource tsc synchronization from CPU 2 to CPUs
0-1,3.
clocksource: Switched to clocksource hpet
loop8: detected capacity change from 0 to 114000

Although I don't fully grok these messages, they sound like something that
could be bhyve's fault. Anyone know if that may indeed be the case?

There is a thread about this on the TrueNAS forum too, but with no answers
really:

 
<https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.1083
76/>
https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.10837
6/

Thanks,

Sean


------=_NextPart_000_0025_01D9A026.1E19F7C0
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 =
(filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Consolas;
	panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
code
	{mso-style-priority:99;
	font-family:"Courier New";}
pre
	{mso-style-priority:99;
	mso-style-link:"HTML Preformatted Char";
	margin:0cm;
	font-size:10.0pt;
	font-family:"Courier New";}
span.HTMLPreformattedChar
	{mso-style-name:"HTML Preformatted Char";
	mso-style-priority:99;
	mso-style-link:"HTML Preformatted";
	font-family:"Consolas",serif;}
span.EmailStyle22
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;
	mso-ligatures:none;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-AU link=3Dblue =
vlink=3Dpurple style=3D'word-wrap:break-word'><div =
class=3DWordSection1><p class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'>Make sure your guests (and host) =
are syncing to a reliable time source. I have heard time to time of some =
having this issue, but we haven&#8217;t seen it though we do use =
OpenNTPD across our BSD/*nix guests (windows has its own time sync =
service).<o:p></o:p></span></p><p class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'>Cheers,<o:p></o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'>Jason.<o:p></o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><div =
style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm =
4.0pt'><div><div style=3D'border:none;border-top:solid #E1E1E1 =
1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DMsoNormal><b><span =
lang=3DEN-US>From:</span></b><span lang=3DEN-US> =
owner-freebsd-virtualization@freebsd.org =
&lt;owner-freebsd-virtualization@freebsd.org&gt; <b>On Behalf Of =
</b>Sean McBride<br><b>Sent:</b> Friday, June 16, 2023 5:20 =
AM<br><b>To:</b> freebsd-virtualization@FreeBSD.org<br><b>Subject:</b> =
Ubuntu in bhyve<o:p></o:p></span></p></div></div><p =
class=3DMsoNormal><o:p>&nbsp;</o:p></p><div><div><p><span =
style=3D'font-family:"Arial",sans-serif'>Hi =
all,<o:p></o:p></span></p><p><span =
style=3D'font-family:"Arial",sans-serif'>I am running bhyve (via TrueNAS =
Core 13) and have some Ubuntu 20.04 and 22.04 =
guests.<o:p></o:p></span></p><p><span =
style=3D'font-family:"Arial",sans-serif'>One of them rather regularly =
seizes up hard, where I cannot ssh to it, then after a few minutes works =
again. This happens repeatedly and on and off. In the Ubuntu logs I see =
messages like:<o:p></o:p></span></p><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7;border-radius:=
 5px 5px 5px 5px;overflow-x: auto;max-width:90vw'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource: timekeeping =
watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew =
is too large:</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource:&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 'hpet' wd_nsec: 536417782 =
wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource:&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 'tsc' cs_nsec: 536821277 cs_now: =
225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource:&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 'tsc' is current =
clocksource.</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>tsc: Marking TSC unstable due =
to clocksource watchdog</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>TSC found unstable after boot, =
most likely due to broken BIOS. Use 'tsc=3Dunstable'.</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>sched_clock: Marking unstable =
(12899572906362, 9192186981)&lt;-(12908811630850, -46250627)</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource: Checking =
clocksource tsc synchronization from CPU 2 to CPUs 0-1,3.</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>clocksource: Switched to =
clocksource hpet</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre =
style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma=
rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span =
style=3D'color:black;background:#F7F7F7'>loop8: detected capacity change =
from 0 to 114000</span><span =
style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><p><span =
style=3D'font-family:"Arial",sans-serif'>Although I don't fully grok =
these messages, they sound like something that could be bhyve's fault. =
Anyone know if that may indeed be the =
case?<o:p></o:p></span></p><p><span =
style=3D'font-family:"Arial",sans-serif'>There is a thread about this on =
the TrueNAS forum too, but with no answers =
really:<o:p></o:p></span></p><p><span =
style=3D'font-family:"Arial",sans-serif'><a =
href=3D"https://www.truenas.com/community/threads/debian-vms-under-bhyve-=
clock.108376/"><span =
style=3D'color:#3983C4'>https://www.truenas.com/community/threads/debian-=
vms-under-bhyve-clock.108376/</span></a><o:p></o:p></span></p><p><span =
style=3D'font-family:"Arial",sans-serif'>Thanks,<o:p></o:p></span></p><p>=
<span =
style=3D'font-family:"Arial",sans-serif'>Sean<o:p></o:p></span></p></div>=
</div></div></div></body></html>
------=_NextPart_000_0025_01D9A026.1E19F7C0--