Date: Thu, 15 Jun 2023 18:22:19 -0400 From: Sean McBride <sean@rogue-research.com> To: jason@tubnor.net Cc: freebsd-virtualization@FreeBSD.org Subject: Re: Ubuntu in bhyve Message-ID: <FC7170CD-82DA-4AE1-9D35-56982D071B36@rogue-research.com> In-Reply-To: <002401d99fd2$4c6d2470$e5476d50$@tubnor.net> References: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com> <002401d99fd2$4c6d2470$e5476d50$@tubnor.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_MailMate_C7833C97-3B71-4A0C-A381-0BD751521718_=
Content-Type: text/plain; format=flowed
Jason,
Thanks for your reply.
We run NTP on our router (a pfsense) and both the host (TrueNAS) and
guests are using it, correctly I believe.
On TrueNAS:
# ntpdate -q time.rogue-research.com
server 10.xx.xx.1, stratum 2, offset -0.000149, delay 0.02580
15 Jun 18:19:11 ntpdate[1239]: adjust time server 10.xx.xx.1 offset
-0.000149 sec
On Ubuntu:
$ timedatectl
Local time: Thu 2023-06-15 18:17:02 EDT
Universal time: Thu 2023-06-15 22:17:02 UTC
RTC time: Thu 2023-06-15 22:17:02
Time zone: America/Toronto (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
I believe both indicate that it's working...
Sean
On 15 Jun 2023, at 17:42, jason@tubnor.net wrote:
> Make sure your guests (and host) are syncing to a reliable time
> source. I
> have heard time to time of some having this issue, but we haven't seen
> it
> though we do use OpenNTPD across our BSD/*nix guests (windows has its
> own
> time sync service).
>
>
>
> Cheers,
>
>
>
> Jason.
>
>
>
> From: owner-freebsd-virtualization@freebsd.org
> <owner-freebsd-virtualization@freebsd.org> On Behalf Of Sean McBride
> Sent: Friday, June 16, 2023 5:20 AM
> To: freebsd-virtualization@FreeBSD.org
> Subject: Ubuntu in bhyve
>
>
>
> Hi all,
>
> I am running bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04
> and
> 22.04 guests.
>
> One of them rather regularly seizes up hard, where I cannot ssh to it,
> then
> after a few minutes works again. This happens repeatedly and on and
> off. In
> the Ubuntu logs I see messages like:
>
> clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc'
> as
> unstable because the skew is too large:
> clocksource: 'hpet' wd_nsec: 536417782 wd_now:
> 638cb3ff wd_last: 63036152 mask: ffffffff
> clocksource: 'tsc' cs_nsec: 536821277 cs_now:
> 225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff
> clocksource: 'tsc' is current clocksource.
> tsc: Marking TSC unstable due to clocksource watchdog
> TSC found unstable after boot, most likely due to broken BIOS. Use
> 'tsc=unstable'.
> sched_clock: Marking unstable (12899572906362,
> 9192186981)<-(12908811630850,
> -46250627)
> clocksource: Checking clocksource tsc synchronization from CPU 2 to
> CPUs
> 0-1,3.
> clocksource: Switched to clocksource hpet
> loop8: detected capacity change from 0 to 114000
>
> Although I don't fully grok these messages, they sound like something
> that
> could be bhyve's fault. Anyone know if that may indeed be the case?
>
> There is a thread about this on the TrueNAS forum too, but with no
> answers
> really:
>
>
> <https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.1083
> 76/>
> https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.10837
> 6/
>
> Thanks,
>
> Sean
--=_MailMate_C7833C97-3B71-4A0C-A381-0BD751521718_=
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"=
>
</head>
<body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty=
le=3D"white-space: normal;">
<p dir=3D"auto">Jason,</p>
<p dir=3D"auto">Thanks for your reply.</p>
<p dir=3D"auto">We run NTP on our router (a pfsense) and both the host (T=
rueNAS) and guests are using it, correctly I believe.</p>
<p dir=3D"auto">On TrueNAS:</p>
<h2>ntpdate -q time.rogue-research.com</h2>
<p dir=3D"auto">server 10.xx.xx.1, stratum 2, offset -0.000149, delay 0.0=
2580<br>
15 Jun 18:19:11 ntpdate[1239]: adjust time server 10.xx.xx.1 offset -0.00=
0149 sec</p>
<p dir=3D"auto">On Ubuntu:</p>
<p dir=3D"auto">$ timedatectl<br>
Local time: Thu 2023-06-15 18:17:02 EDT<br>
Universal time: Thu 2023-06-15 22:17:02 UTC<br>
RTC time: Thu 2023-06-15 22:17:02<br>
Time zone: America/Toronto (EDT, -0400)<br>
System clock synchronized: yes<br>
NTP service: active<br>
RTC in local TZ: no</p>
<p dir=3D"auto">I believe both indicate that it's working...</p>
<p dir=3D"auto">Sean</p>
<p dir=3D"auto">On 15 Jun 2023, at 17:42, <a href=3D"mailto:jason@tubnor.=
net" style=3D"color: #3983C4;">jason@tubnor.net</a> wrote:</p>
</div><blockquote class=3D"embedded" style=3D"margin: 0 0 5px; padding-le=
ft: 5px; border-left: 2px solid #777777; color: #777777;"><div id=3D"385C=
24F8-818F-4661-910E-FEEBF47434BA">
<div lang=3D"EN-AU" link=3D"blue" vlink=3D"purple" style=3D"word-wrap:bre=
ak-word">
<div class=3D"WordSection1" style=3D"page: WordSection1;">
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Make su=
re your guests (and host) are syncing to a reliable time source. I have h=
eard time to time of some having this issue, but we haven=E2=80=99t seen =
it though we do use OpenNTPD across our BSD/*nix guests (windows has its =
own time sync service).</span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"> <=
/span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Cheers,=
</span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"> <=
/span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Jason.<=
/span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"> <=
/span></p>
<div style=3D"border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0c=
m 4.0pt">
<div>
<div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0c=
m 0cm 0cm">
<p class=3D"MsoNormal"><b><span lang=3D"EN-US">From:</span></b> <span lan=
g=3D"EN-US">owner-freebsd-virtualization@freebsd.org <owner-freebsd-vi=
rtualization@freebsd.org> <b>On Behalf Of</b> Sean McBride<br>
<b>Sent:</b> Friday, June 16, 2023 5:20 AM<br>
<b>To:</b> freebsd-virtualization@FreeBSD.org<br>
<b>Subject:</b> Ubuntu in bhyve</span></p>
</div>
</div>
<p class=3D"MsoNormal"> </p>
<div>
<div>
<p><span style=3D"font-family:"Arial",sans-serif">Hi all,</span=
></p>
<p><span style=3D"font-family:"Arial",sans-serif">I am running =
bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04 and 22.04 guests.<=
/span></p>
<p><span style=3D"font-family:"Arial",sans-serif">One of them r=
ather regularly seizes up hard, where I cannot ssh to it, then after a fe=
w minutes works again. This happens repeatedly and on and off. In the Ubu=
ntu logs I see messages like:</span></p>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7; border-radius: 5p=
x 5px 5px 5px; overflow-x: auto; max-width: 90vw;"><code style=3D"mso-sty=
le-priority: 99; font-family: 'Courier New';"><span style=3D"color:black;=
background:#F7F7F7">clocksource: timekeeping watchdog on CPU2: Marking cl=
ocksource 'tsc' as unstable because the skew is too large:</span></code><=
/pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">clocksource: 'hpet' wd_ns=
ec: 536417782 wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff</span></c=
ode></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">clocksource: 'tsc' cs_nse=
c: 536821277 cs_now: 225a9f9e1250b cs_last: 225a9b3891749 mask: fffffffff=
fffffff</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">clocksource: 'tsc' is cur=
rent clocksource.</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">tsc: Marking TSC unstable due to clocksource wa=
tchdog</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">TSC found unstable after boot, most likely due =
to broken BIOS. Use 'tsc=3Dunstable'.</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">sched_clock: Marking unstable (12899572906362, =
9192186981)<-(12908811630850, -46250627)</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">clocksource: Checking clocksource tsc synchroni=
zation from CPU 2 to CPUs 0-1,3.</span></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">clocksource: Switched to clocksource hpet</span=
></code></pre>
<pre style=3D"mso-style-priority: 99; mso-style-link: 'HTML Preformatted =
Char'; margin: 0cm; font-size: 10.0pt; font-family: 'Courier New'; mso-ma=
rgin-top-alt: 0cm; margin-right: 11.25pt; margin-bottom: 0cm; margin-left=
: 11.25pt; margin-bottom: .0001pt; background: #F7F7F7;"><code style=3D"m=
so-style-priority: 99; font-family: 'Courier New';"><span style=3D"color:=
black;background:#F7F7F7">loop8: detected capacity change from 0 to 11400=
0</span></code></pre>
<p><span style=3D"font-family:"Arial",sans-serif">Although I do=
n't fully grok these messages, they sound like something that could be bh=
yve's fault. Anyone know if that may indeed be the case?</span></p>
<p><span style=3D"font-family:"Arial",sans-serif">There is a th=
read about this on the TrueNAS forum too, but with no answers really:</sp=
an></p>
<p><span style=3D"font-family:"Arial",sans-serif"><a href=3D"ht=
tps://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.1083=
76/"><span style=3D"color:#3983C4">https://www.truenas.com/community/thre=
ads/debian-vms-under-bhyve-clock.108376/</span></a></span></p>
<p><span style=3D"font-family:"Arial",sans-serif">Thanks,</span=
></p>
<p><span style=3D"font-family:"Arial",sans-serif">Sean</span></=
p>
</div>
</div>
</div>
</div>
</div></div></blockquote>
<div class=3D"markdown" style=3D"white-space: normal;">
</div>
</div>
</body>
</html>
--=_MailMate_C7833C97-3B71-4A0C-A381-0BD751521718_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FC7170CD-82DA-4AE1-9D35-56982D071B36>