Date: Fri, 16 Jun 2023 07:42:35 +1000 From: <jason@tubnor.net> To: "'Sean McBride'" <sean@rogue-research.com>, <freebsd-virtualization@FreeBSD.org> Subject: RE: Ubuntu in bhyve Message-ID: <002401d99fd2$4c6d2470$e5476d50$@tubnor.net> In-Reply-To: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com> References: <1AC64CF0-2CCF-4B72-9E44-75104E813CF6@rogue-research.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multipart message in MIME format. ------=_NextPart_000_0025_01D9A026.1E19F7C0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Make sure your guests (and host) are syncing to a reliable time source. I have heard time to time of some having this issue, but we haven't seen it though we do use OpenNTPD across our BSD/*nix guests (windows has its own time sync service). Cheers, Jason. From: owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualization@freebsd.org> On Behalf Of Sean McBride Sent: Friday, June 16, 2023 5:20 AM To: freebsd-virtualization@FreeBSD.org Subject: Ubuntu in bhyve Hi all, I am running bhyve (via TrueNAS Core 13) and have some Ubuntu 20.04 and 22.04 guests. One of them rather regularly seizes up hard, where I cannot ssh to it, then after a few minutes works again. This happens repeatedly and on and off. In the Ubuntu logs I see messages like: clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large: clocksource: 'hpet' wd_nsec: 536417782 wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff clocksource: 'tsc' cs_nsec: 536821277 cs_now: 225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff clocksource: 'tsc' is current clocksource. tsc: Marking TSC unstable due to clocksource watchdog TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. sched_clock: Marking unstable (12899572906362, 9192186981)<-(12908811630850, -46250627) clocksource: Checking clocksource tsc synchronization from CPU 2 to CPUs 0-1,3. clocksource: Switched to clocksource hpet loop8: detected capacity change from 0 to 114000 Although I don't fully grok these messages, they sound like something that could be bhyve's fault. Anyone know if that may indeed be the case? There is a thread about this on the TrueNAS forum too, but with no answers really: <https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.1083 76/> https://www.truenas.com/community/threads/debian-vms-under-bhyve-clock.10837 6/ Thanks, Sean ------=_NextPart_000_0025_01D9A026.1E19F7C0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} code {mso-style-priority:99; font-family:"Courier New";} pre {mso-style-priority:99; mso-style-link:"HTML Preformatted Char"; margin:0cm; font-size:10.0pt; font-family:"Courier New";} span.HTMLPreformattedChar {mso-style-name:"HTML Preformatted Char"; mso-style-priority:99; mso-style-link:"HTML Preformatted"; font-family:"Consolas",serif;} span.EmailStyle22 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-AU link=3Dblue = vlink=3Dpurple style=3D'word-wrap:break-word'><div = class=3DWordSection1><p class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'>Make sure your guests (and host) = are syncing to a reliable time source. I have heard time to time of some = having this issue, but we haven’t seen it though we do use = OpenNTPD across our BSD/*nix guests (windows has its own time sync = service).<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'>Cheers,<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'>Jason.<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'mso-fareast-language:EN-US'><o:p> </o:p></span></p><div = style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm = 4.0pt'><div><div style=3D'border:none;border-top:solid #E1E1E1 = 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DMsoNormal><b><span = lang=3DEN-US>From:</span></b><span lang=3DEN-US> = owner-freebsd-virtualization@freebsd.org = <owner-freebsd-virtualization@freebsd.org> <b>On Behalf Of = </b>Sean McBride<br><b>Sent:</b> Friday, June 16, 2023 5:20 = AM<br><b>To:</b> freebsd-virtualization@FreeBSD.org<br><b>Subject:</b> = Ubuntu in bhyve<o:p></o:p></span></p></div></div><p = class=3DMsoNormal><o:p> </o:p></p><div><div><p><span = style=3D'font-family:"Arial",sans-serif'>Hi = all,<o:p></o:p></span></p><p><span = style=3D'font-family:"Arial",sans-serif'>I am running bhyve (via TrueNAS = Core 13) and have some Ubuntu 20.04 and 22.04 = guests.<o:p></o:p></span></p><p><span = style=3D'font-family:"Arial",sans-serif'>One of them rather regularly = seizes up hard, where I cannot ssh to it, then after a few minutes works = again. This happens repeatedly and on and off. In the Ubuntu logs I see = messages like:<o:p></o:p></span></p><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7;border-radius:= 5px 5px 5px 5px;overflow-x: auto;max-width:90vw'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: timekeeping = watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew = is too large:</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: &n= bsp; &nb= sp; 'hpet' wd_nsec: 536417782 = wd_now: 638cb3ff wd_last: 63036152 mask: ffffffff</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: &n= bsp; &nb= sp; 'tsc' cs_nsec: 536821277 cs_now: = 225a9f9e1250b cs_last: 225a9b3891749 mask: ffffffffffffffff</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: &n= bsp; &nb= sp; 'tsc' is current = clocksource.</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>tsc: Marking TSC unstable due = to clocksource watchdog</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>TSC found unstable after boot, = most likely due to broken BIOS. Use 'tsc=3Dunstable'.</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>sched_clock: Marking unstable = (12899572906362, 9192186981)<-(12908811630850, -46250627)</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: Checking = clocksource tsc synchronization from CPU 2 to CPUs 0-1,3.</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>clocksource: Switched to = clocksource hpet</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><pre = style=3D'mso-margin-top-alt:0cm;margin-right:11.25pt;margin-bottom:0cm;ma= rgin-left:11.25pt;margin-bottom:.0001pt;background:#F7F7F7'><code><span = style=3D'color:black;background:#F7F7F7'>loop8: detected capacity change = from 0 to 114000</span><span = style=3D'background:#F7F7F7'><o:p></o:p></span></code></pre><p><span = style=3D'font-family:"Arial",sans-serif'>Although I don't fully grok = these messages, they sound like something that could be bhyve's fault. = Anyone know if that may indeed be the = case?<o:p></o:p></span></p><p><span = style=3D'font-family:"Arial",sans-serif'>There is a thread about this on = the TrueNAS forum too, but with no answers = really:<o:p></o:p></span></p><p><span = style=3D'font-family:"Arial",sans-serif'><a = href=3D"https://www.truenas.com/community/threads/debian-vms-under-bhyve-= clock.108376/"><span = style=3D'color:#3983C4'>https://www.truenas.com/community/threads/debian-= vms-under-bhyve-clock.108376/</span></a><o:p></o:p></span></p><p><span = style=3D'font-family:"Arial",sans-serif'>Thanks,<o:p></o:p></span></p><p>= <span = style=3D'font-family:"Arial",sans-serif'>Sean<o:p></o:p></span></p></div>= </div></div></div></body></html> ------=_NextPart_000_0025_01D9A026.1E19F7C0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?002401d99fd2$4c6d2470$e5476d50$>