Date: Fri, 16 Jun 2023 07:42:35 +1000 From: <jason@tubnor.net> To: "'Sean McBride'" <sean@rogue-research.com>, <freebsd-virtualization@FreeBSD.org> Subject: RE: Ubuntu in bhyve Message-ID: <002401d99fd2$4c6d2470$e5476d50$@tubnor.net> In-Reply-To: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Make sure your guests (and host) are syncing to a reliable time source. I have heard time to time of some having this issue, but we haven't seen it though we do use OpenNTPD across our BSD/*nix guests (windows has its own time sync service). Cheers, Jason. From: owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualization@freebsd.org> On Behalf Of Sean McBride Sent: Friday, June 16, 2023 5:20 AM To: freebsd-virtualization@FreeBSD.org Subject: Ubuntu in bhyve Hi all, I am running bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04 and 22.04 guests. One of them rather regularly seizes up hard, where I cannot ssh to it, then after a few minutes works again. This happens repeatedly and on and off. In the Ubuntu logs I see messages like: clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large: clocksource: 'hpet' wd_nsec: 536417782 wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff clocksource: 'tsc' cs_nsec: 536821277 cs_now: 225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff clocksource: 'tsc' is current clocksource. tsc: Marking TSC unstable due to clocksource watchdog TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. sched_clock: Marking unstable (12899572906362, 9192186981)<-(12908811630850, -46250627) clocksource: Checking clocksource tsc synchronization from CPU 2 to CPUs 0-1,3. clocksource: Switched to clocksource hpet loop8: detected capacity change from 0 to 114000 Although I don't fully grok these messages, they sound like something that could be bhyve's fault. Anyone know if that may indeed be the case? There is a thread about this on the TrueNAS forum too, but with no answers really: <https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.1083 76/> https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.10837 6/ Thanks, Sean [-- Attachment #2 --] <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} code {mso-style-priority:99; font-family:"Courier New";} pre {mso-style-priority:99; mso-style-link:"HTML Preformatted Char"; margin:0cm; font-size:10.0pt; font-family:"Courier New";} span.HTMLPreformattedChar {mso-style-name:"HTML Preformatted Char"; mso-style-priority:99; mso-style-link:"HTML Preformatted"; font-family:"Consolas",serif;} span.EmailStyle22 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1" /> </o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Make sure your guests (and host) are syncing to a reliable time source. I have heard time to time of some having this issue, but we haven’t seen it though we do use OpenNTPD across our BSD/*nix guests (windows has its own time sync service).<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Cheers,<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Jason.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualization@freebsd.org> <b>On Behalf Of </b>Sean McBride<br><b>Sent:</b> Friday, June 16, 2023 5:20 AM<br><b>To:</b> freebsd-virtualization@FreeBSD.org<br><b>Subject:</b> Ubuntu in bhyve<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p><span style='font-family:"Arial",sans-serif'>Hi all,<o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'>I am running bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04 and 22.04 guests.<o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'>One of them rather regularly seizes up hard, where I cannot ssh to it, then after a few minutes works again. This happens repeatedly and on and off. In the Ubuntu logs I see messages like:<o:p></o:p></span></p><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7;border-radius: 5px 5px 5px 5px;overflow-x: auto;max-width:90vw'><code><span style='color:black;background:#F7F7F7'>clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large:</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>clocksource: 'hpet' wd_nsec: 536417782 wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>clocksource: 'tsc' cs_nsec: 536821277 cs_now: 225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>clocksource: 'tsc' is current clocksource.</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>tsc: Marking TSC unstable due to clocksource watchdog</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>sched_clock: Marking unstable (12899572906362, 9192186981)<-(12908811630850, -46250627)</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>clocksource: Checking clocksource tsc synchronization from CPU 2 to CPUs 0-1,3.</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>clocksource: Switched to clocksource hpet</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><pre style='mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;margin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span style='color:black;background:#F7F7F7'>loop8: detected capacity change from 0 to 114000</span><span style='background:#F7F7F7'><o:p></o:p></span></code></pre><p><span style='font-family:"Arial",sans-serif'>Although I don't fully grok these messages, they sound like something that could be bhyve's fault. Anyone know if that may indeed be the case?<o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'>There is a thread about this on the TrueNAS forum too, but with no answers really:<o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'><a href="https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.108376/"><span style='color:#3983C4'>https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.108376/</span></a><o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'>Thanks,<o:p></o:p></span></p><p><span style='font-family:"Arial",sans-serif'>Sean<o:p></o:p></span></p></div></div></div></div></body></html>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?002401d99fd2$4c6d2470$e5476d50$>