From owner-freebsd-virtualization@freebsd.org Sun Oct 15 01:50:18 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADF9CE34379 for ; Sun, 15 Oct 2017 01:50:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C69065007 for ; Sun, 15 Oct 2017 01:50:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9F1oI5D043731 for ; Sun, 15 Oct 2017 01:50:18 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 222996] FreeBSD 11.1 on Hyper-V with PCI Express Pass Through Date: Sun, 15 Oct 2017 01:50:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sepherosa@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2017 01:50:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222996 Sepherosa Ziehau changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sepherosa@gmail.com --- Comment #1 from Sepherosa Ziehau --- Do you mean pass-through or SR-IOV? For SR-IOV, we have got mlx4_en work (connectx-3), which is also used in Az= ure. Last time we tested, ixgbe's VF didn't not work. PCI pass-through requires some extra per-VM configuration through powershel= l, I will check w/ Dexuan next Monday. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sun Oct 15 05:35:48 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 076DCE3AF28 for ; Sun, 15 Oct 2017 05:35:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DCB626BA32 for ; Sun, 15 Oct 2017 05:35:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9F5Zl8Y022911 for ; Sun, 15 Oct 2017 05:35:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 222996] FreeBSD 11.1 on Hyper-V with PCI Express Pass Through Date: Sun, 15 Oct 2017 05:35:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dmitry_kuleshov@ukr.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2017 05:35:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222996 --- Comment #2 from Dmitry --- No, i mean not SR-IOV. I am already have extra configuration through powershell for PCI Passthroug= h, and it's works for other OS in Generation 2 Hyper-V VM, except FreeBSD. I have reconfigured VM for legacy Generation 1, and PCI Passtrought works g= reat in FreeBSD 11.1, but in new UEFI Generation 2 VM Passthrought didn't work. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sun Oct 15 07:34:13 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79A88E3C842 for ; Sun, 15 Oct 2017 07:34:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 67CE76DC96 for ; Sun, 15 Oct 2017 07:34:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9F7YDn2046002 for ; Sun, 15 Oct 2017 07:34:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 222996] FreeBSD 11.1 on Hyper-V with PCI Express Pass Through Date: Sun, 15 Oct 2017 07:34:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sepherosa@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Oct 2017 07:34:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222996 --- Comment #3 from Sepherosa Ziehau --- OK, I see. Please post the bootverbose dmesg, if possible. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Oct 16 15:03:32 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17013E3CAD7 for ; Mon, 16 Oct 2017 15:03:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0449D83D32 for ; Mon, 16 Oct 2017 15:03:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9GF3S1K007628 for ; Mon, 16 Oct 2017 15:03:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Mon, 16 Oct 2017 15:03:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: abeesh@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2017 15:03:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #15 from Abi --- Affecting me too. Started since pfsense 2.4 uses FreeBSD 11 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Oct 20 17:20:14 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA459E3B472; Fri, 20 Oct 2017 17:20:14 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward102j.mail.yandex.net (forward102j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7BDDD837C4; Fri, 20 Oct 2017 17:20:11 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback13j.mail.yandex.net (mxback13j.mail.yandex.net [IPv6:2a02:6b8:0:1619::88]) by forward102j.mail.yandex.net (Yandex) with ESMTP id C1179560560C; Fri, 20 Oct 2017 20:20:07 +0300 (MSK) Received: from smtp1j.mail.yandex.net (smtp1j.mail.yandex.net [2a02:6b8:0:801::ab]) by mxback13j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id CTJIjMd24C-K7faKnAg; Fri, 20 Oct 2017 20:20:07 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508520007; bh=gP3qVzWMC8Hr5UFe9Rdetfpko3h0tO5kTA5xixVF2bU=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=dhPIuXJD99ty+usMvKJ7yJ9TMzcr++ppeHJlRuZW3AlBfFAyo0wKcBXlqmCIItiOw 6L2RoAlsnUB8UTX5nwe/At0e/Du2KyefTsscSvnwmWX4185kqMuX2Vo0+q/NHANxpu fzJxx2auTe+2ocFrLccxeNgFy77U1Pyb3plChoVc= Received: by smtp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id vNW5MEB0Yn-K7w0FMN1; Fri, 20 Oct 2017 20:20:07 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508520007; bh=gP3qVzWMC8Hr5UFe9Rdetfpko3h0tO5kTA5xixVF2bU=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=dhPIuXJD99ty+usMvKJ7yJ9TMzcr++ppeHJlRuZW3AlBfFAyo0wKcBXlqmCIItiOw 6L2RoAlsnUB8UTX5nwe/At0e/Du2KyefTsscSvnwmWX4185kqMuX2Vo0+q/NHANxpu fzJxx2auTe+2ocFrLccxeNgFy77U1Pyb3plChoVc= Authentication-Results: smtp1j.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> From: Boris Samorodov Message-ID: <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> Date: Fri, 20 Oct 2017 20:20:06 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508517160.1383.63.camel@freebsd.org> Content-Type: text/plain; charset=koi8-r Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 17:20:15 -0000 (CC to freebsd-virtualization@) 20.10.2017 19:32, Ian Lepore ΠΙΫΕΤ: > On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote: >> 20.10.2017 18:31, Boris Samorodov ΠΙΫΕΤ: >>> >>> 20.10.2017 18:12, Ian Lepore ΠΙΫΕΤ: >>>> >>>> On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote: >>>>> >>>>> Hi All, >>>>> >>>>> I have got a host: >>>>> --- >>>>> bhyve-host% uname -a >>>>> FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug >>>>> 25 05:25:26 MSK 2017 >>>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FASTššamd64 amd64 >>>>> --- >>>>> >>>>> And a bhyve vm: >>>>> --- >>>>> bhyve-vm: uname -a >>>>> FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri >>>>> Oct 20 05:12:17 MSK 2017 >>>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64Xššamd64 amd64 >>>>> --- >>>>> >>>>> The only difference at kernel configs is a colored console. :-) >>>>> >>>>> And here I get some weird (is it?) result at the VM (I expect ntpd to be >>>>> more stable): >>>>> --- >>>>> bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done >>>>> šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset >>>>> jitter >>>>> ============================================================================== >>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš9ššš64šššš3šššš0.605ššš-1.202 >>>>> 316.407 >>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš7ššš64šššš7šššš0.605ššš-1.202 >>>>> 358.395 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš5ššš64ššš17šššš0.615šš-328.42 >>>>> 181.405 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64ššš37šššš0.615šš-328.42 >>>>> 214.868 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64ššš37šššš0.615šš-328.42 >>>>> 214.868 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš63ššš64ššš77šššš0.615šš-328.42 >>>>> 268.618 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64šš177šššš0.615šš-328.42 >>>>> 333.175 >>>>> šXX.XX.XX.1šššššš.STEP.šššššššššš16 u 1910ššš64šššš0šššš0.000šššš0.000 >>>>> 0.000 >>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš27ššš64šššš1šššš0.703šš-262.63 >>>>> 0.004 >>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš31ššš64šššš1šššš0.649šš-331.43 >>>>> 68.800 >>>>> --- >>>>> >>>>> At the same time host's results are very stable: >>>>> --- >>>>> bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done >>>>> šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset >>>>> jitter >>>>> ============================================================================== >>>>> >>>>> >>>>> >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš1ššš64šššš1šššš0.401šššš0.176 >>>>> 0.106 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš6ššš64šššš3šššš0.401šššš0.176 >>>>> 0.459 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64šššš7šššš0.401šššš0.176 >>>>> 0.940 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64šššš7šššš0.401šššš0.176 >>>>> 0.940 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš64ššš64ššš17šššš0.401šššš0.176 >>>>> 1.566 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64ššš37šššš0.448šššš1.275 >>>>> 1.739 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš55ššš64ššš77šššš0.448šššš1.275 >>>>> 2.365 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš53ššš64šš177šššš0.448šššš1.275 >>>>> 3.110 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš50ššš64šš377šššš0.448šššš1.275 >>>>> 3.929 >>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš45ššš64šš377šššš0.443šššš8.750 >>>>> 4.722 >>>>> --- >>>>> >>>>> The network is organized via bridge -- host igb and vm tap interfaces >>>>> are members of one bridge. >>>>> >>>>> Are those results expected? Does it smell like a bug? Should I dig >>>>> furter? >>>>> >>>> So it is repeatedly stepping the clock in the VM? (Set >>>> kern.timecounter.stepwarnings=1 to log steps). >>> No kernel/ntpd messages for 20 minutes after setting this sysctl. >>> >>>> >>>> šThat is usually a sign >>>> that the chosen timecounter is running at a different frequency than it >>>> claimed to be when it registered itself -- the host may not be >>>> emulating the timer hardware properly in the guest. šWhat is the output >>>> of sysctl kern.timecounter in the vm? >>> --- >>> bhyve-vm% sysctl kern.timecounter >>> >>> kern.timecounter.tsc_shift: 1 >>> kern.timecounter.smp_tsc_adjust: 0 >>> kern.timecounter.smp_tsc: 0 >>> kern.timecounter.invariant_tsc: 1 >>> kern.timecounter.fast_gettime: 1 >>> kern.timecounter.tick: 1 >>> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100) >>> dummy(-1000000) >>> kern.timecounter.hardware: HPET >>> kern.timecounter.alloweddeviation: 5 >>> kern.timecounter.stepwarnings: 1 >>> kern.timecounter.tc.ACPI-fast.quality: 900 >>> kern.timecounter.tc.ACPI-fast.frequency: 3579545 >>> kern.timecounter.tc.ACPI-fast.counter: 4161213491 >>> kern.timecounter.tc.ACPI-fast.mask: 4294967295 >>> kern.timecounter.tc.HPET.quality: 950 >>> kern.timecounter.tc.HPET.frequency: 10000000 >>> kern.timecounter.tc.HPET.counter: 3518036865 >>> kern.timecounter.tc.HPET.mask: 4294967295 >>> kern.timecounter.tc.i8254.quality: 0 >>> kern.timecounter.tc.i8254.frequency: 1193182 >>> kern.timecounter.tc.i8254.counter: 47597 >>> kern.timecounter.tc.i8254.mask: 65535 >>> kern.timecounter.tc.TSC-low.quality: -100 >>> kern.timecounter.tc.TSC-low.frequency: 1199886114 >>> kern.timecounter.tc.TSC-low.counter: 1274338278 >>> kern.timecounter.tc.TSC-low.mask: 4294967295 >>> --- >> BTW, here is the host kern.timecounter: >> --- >> kern.timecounter.tsc_shift: 1 >> kern.timecounter.smp_tsc_adjust: 0 >> kern.timecounter.smp_tsc: 1 >> kern.timecounter.invariant_tsc: 1 >> kern.timecounter.fast_gettime: 1 >> kern.timecounter.tick: 1 >> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000) >> dummy(-1000000) >> kern.timecounter.hardware: TSC-low >> kern.timecounter.alloweddeviation: 5 >> kern.timecounter.stepwarnings: 0 >> kern.timecounter.tc.ACPI-fast.quality: 900 >> kern.timecounter.tc.ACPI-fast.frequency: 3579545 >> kern.timecounter.tc.ACPI-fast.counter: 9047194 >> kern.timecounter.tc.ACPI-fast.mask: 16777215 >> kern.timecounter.tc.HPET.quality: 950 >> kern.timecounter.tc.HPET.frequency: 14318180 >> kern.timecounter.tc.HPET.counter: 2232552795 >> kern.timecounter.tc.HPET.mask: 4294967295 >> kern.timecounter.tc.i8254.quality: 0 >> kern.timecounter.tc.i8254.frequency: 1193182 >> kern.timecounter.tc.i8254.counter: 43410 >> kern.timecounter.tc.i8254.mask: 65535 >> kern.timecounter.tc.TSC-low.quality: 1000 >> kern.timecounter.tc.TSC-low.frequency: 1200067168 >> kern.timecounter.tc.TSC-low.counter: 2463146362 >> kern.timecounter.tc.TSC-low.mask: 4294967295 >> --- >> >>> >>>> >>>> Also, what is the output of ntptime(8) in the vm? >>> --- >>> bhyve-vm% ntptime >>> >>> ntp_gettime() returns code 0 (OK) >>> š time dd94930f.20ea2900ššFri, Oct 20 2017 18:21:51.128, (.128573699), >>> š maximum error 1309110 us, estimated error 3 us, TAI offset 37 >>> ntp_adjtime() returns code 0 (OK) >>> š modes 0x0 (), >>> š offset 0.000 us, frequency 0.000 ppm, interval 1 s, >>> š maximum error 1309110 us, estimated error 3 us, >>> š status 0x2001 (PLL,NANO), >>> š time constant 6, precision 0.001 us, tolerance 496 ppm, >>> --- >>> >>> Ian, thank you for your help! >>> >> > > It seems odd to me that the frequency of the host HPET is 14.3mhz and > that of the guest is 10.0mhz, but maybe that's a normal condition for > bhyve. šI did find some google hits[1] for bhyve guest timekeeping > trouble with the HPET timer which was solved by switching to a > different timecounter. šTimecounter choices can't be controlled from > loader.conf, so I guess a sysctl.conf entry of > kern.timecounter.hardware="ACPI-fast" is the only way to fix that. šYou > can also just do that command interactively first and see if it stops > the time steps and ntp settles down. The process seems to become more monotonic. But steps nevertheless: --- *XX.1 XX.245 4 u 20 64 1 0.717 -12.771 4.193 *XX.1 XX.245 4 u 28 64 3 0.751 -41.970 32.342 *XX.1 XX.245 4 u 23 64 7 0.748 -59.505 46.624 *XX.1 XX.245 4 u 18 64 17 0.699 -75.164 56.482 *XX.1 XX.245 4 u 14 64 37 0.669 -90.112 63.767 *XX.1 XX.245 4 u 11 64 77 0.605 -10.567 60.914 *XX.1 XX.245 4 u 7 64 177 0.591 -169.54 116.762 *XX.1 XX.245 4 u 3 64 377 0.591 -169.54 102.107 *XX.1 XX.245 4 u 68 64 377 0.591 -169.54 102.107 *XX.1 XX.245 4 u 63 64 377 0.591 -169.54 88.424 *XX.1 XX.245 4 u 58 64 377 0.591 -169.54 92.949 *XX.1 XX.245 4 u 55 64 377 0.591 -169.54 111.512 *XX.1 XX.245 4 u 50 64 377 0.591 -169.54 140.827 *XX.1 XX.245 4 u 45 64 377 0.591 -169.54 177.360 *XX.1 XX.245 4 u 43 64 377 0.591 -169.54 219.057 XX.1 .STEP. 16 u 588 64 0 0.000 0.000 0.000 --- > This would be a workaround, not a fix per se. šIf the time steps go > away, then something in bhyve's emulation of HPET (maybe only on some > hardware?) must be buggy. > > > [1] https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-April/003492.html Also thanks for the link. Unfortunately the problem seems to persist. -- WBR, Boris Samorodov (bsam) FreeBSD Committer, http://www.FreeBSD.org The Power To Serve From owner-freebsd-virtualization@freebsd.org Fri Oct 20 18:04:33 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F2B3E3C3DD for ; Fri, 20 Oct 2017 18:04:33 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from pmta2.delivery6.ore.mailhop.org (pmta2.delivery6.ore.mailhop.org [54.200.129.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3F352E92 for ; Fri, 20 Oct 2017 18:04:32 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 040a06dc-b5c1-11e7-b50b-53dc5ecda239 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound2.ore.mailhop.org (Halon) with ESMTPSA id 040a06dc-b5c1-11e7-b50b-53dc5ecda239; Fri, 20 Oct 2017 18:03:46 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v9KI4TbW011615; Fri, 20 Oct 2017 12:04:29 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1508522667.1383.69.camel@freebsd.org> Subject: Re: host, bhyve vm and ntpd From: Ian Lepore To: Boris Samorodov , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Date: Fri, 20 Oct 2017 12:04:27 -0600 In-Reply-To: <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> Content-Type: text/plain; charset="koi8-r" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 18:04:33 -0000 On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote: > (CC to freebsd-virtualization@) > > 20.10.2017 19:32, Ian Lepore ΠΙΫΕΤ: > > > > On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote: > > > > > > 20.10.2017 18:31, Boris Samorodov ΠΙΫΕΤ: > > > > > > > > > > > > 20.10.2017 18:12, Ian Lepore ΠΙΫΕΤ: > > > > > > > > > > > > > > > On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote: > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > I have got a host: > > > > > > --- > > > > > > bhyve-host% uname -a > > > > > > FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug > > > > > > 25 05:25:26 MSK 2017 > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FASTššamd64 amd64 > > > > > > --- > > > > > > > > > > > > And a bhyve vm: > > > > > > --- > > > > > > bhyve-vm: uname -a > > > > > > FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri > > > > > > Oct 20 05:12:17 MSK 2017 > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64Xššamd64 amd64 > > > > > > --- > > > > > > > > > > > > The only difference at kernel configs is a colored console. :-) > > > > > > > > > > > > And here I get some weird (is it?) result at the VM (I expect ntpd to be > > > > > > more stable): > > > > > > --- > > > > > > bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done > > > > > > šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset > > > > > > jitter > > > > > > ============================================================================== > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš9ššš64šššš3šššš0.605ššš-1.202 > > > > > > 316.407 > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš7ššš64šššš7šššš0.605ššš-1.202 > > > > > > 358.395 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš5ššš64ššš17šššš0.615šš-328.42 > > > > > > 181.405 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64ššš37šššš0.615šš-328.42 > > > > > > 214.868 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64ššš37šššš0.615šš-328.42 > > > > > > 214.868 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš63ššš64ššš77šššš0.615šš-328.42 > > > > > > 268.618 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64šš177šššš0.615šš-328.42 > > > > > > 333.175 > > > > > > šXX.XX.XX.1šššššš.STEP.šššššššššš16 u 1910ššš64šššš0šššš0.000šššš0.000 > > > > > > 0.000 > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš27ššš64šššš1šššš0.703šš-262.63 > > > > > > 0.004 > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš31ššš64šššš1šššš0.649šš-331.43 > > > > > > 68.800 > > > > > > --- > > > > > > > > > > > > At the same time host's results are very stable: > > > > > > --- > > > > > > bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done > > > > > > šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset > > > > > > jitter > > > > > > ============================================================================== > > > > > > > > > > > > > > > > > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš1ššš64šššš1šššš0.401šššš0.176 > > > > > > 0.106 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš6ššš64šššš3šššš0.401šššš0.176 > > > > > > 0.459 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64šššš7šššš0.401šššš0.176 > > > > > > 0.940 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64šššš7šššš0.401šššš0.176 > > > > > > 0.940 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš64ššš64ššš17šššš0.401šššš0.176 > > > > > > 1.566 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64ššš37šššš0.448šššš1.275 > > > > > > 1.739 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš55ššš64ššš77šššš0.448šššš1.275 > > > > > > 2.365 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš53ššš64šš177šššš0.448šššš1.275 > > > > > > 3.110 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš50ššš64šš377šššš0.448šššš1.275 > > > > > > 3.929 > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš45ššš64šš377šššš0.443šššš8.750 > > > > > > 4.722 > > > > > > --- > > > > > > > > > > > > The network is organized via bridge -- host igb and vm tap interfaces > > > > > > are members of one bridge. > > > > > > > > > > > > Are those results expected? Does it smell like a bug? Should I dig > > > > > > furter? > > > > > > > > > > > So it is repeatedly stepping the clock in the VM? (Set > > > > > kern.timecounter.stepwarnings=1 to log steps). > > > > No kernel/ntpd messages for 20 minutes after setting this sysctl. > > > > > > > > > > > > > > > > > > > šThat is usually a sign > > > > > that the chosen timecounter is running at a different frequency than it > > > > > claimed to be when it registered itself -- the host may not be > > > > > emulating the timer hardware properly in the guest. šWhat is the output > > > > > of sysctl kern.timecounter in the vm? > > > > --- > > > > bhyve-vm% sysctl kern.timecounter > > > > > > > > kern.timecounter.tsc_shift: 1 > > > > kern.timecounter.smp_tsc_adjust: 0 > > > > kern.timecounter.smp_tsc: 0 > > > > kern.timecounter.invariant_tsc: 1 > > > > kern.timecounter.fast_gettime: 1 > > > > kern.timecounter.tick: 1 > > > > kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100) > > > > dummy(-1000000) > > > > kern.timecounter.hardware: HPET > > > > kern.timecounter.alloweddeviation: 5 > > > > kern.timecounter.stepwarnings: 1 > > > > kern.timecounter.tc.ACPI-fast.quality: 900 > > > > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > > > > kern.timecounter.tc.ACPI-fast.counter: 4161213491 > > > > kern.timecounter.tc.ACPI-fast.mask: 4294967295 > > > > kern.timecounter.tc.HPET.quality: 950 > > > > kern.timecounter.tc.HPET.frequency: 10000000 > > > > kern.timecounter.tc.HPET.counter: 3518036865 > > > > kern.timecounter.tc.HPET.mask: 4294967295 > > > > kern.timecounter.tc.i8254.quality: 0 > > > > kern.timecounter.tc.i8254.frequency: 1193182 > > > > kern.timecounter.tc.i8254.counter: 47597 > > > > kern.timecounter.tc.i8254.mask: 65535 > > > > kern.timecounter.tc.TSC-low.quality: -100 > > > > kern.timecounter.tc.TSC-low.frequency: 1199886114 > > > > kern.timecounter.tc.TSC-low.counter: 1274338278 > > > > kern.timecounter.tc.TSC-low.mask: 4294967295 > > > > --- > > > BTW, here is the host kern.timecounter: > > > --- > > > kern.timecounter.tsc_shift: 1 > > > kern.timecounter.smp_tsc_adjust: 0 > > > kern.timecounter.smp_tsc: 1 > > > kern.timecounter.invariant_tsc: 1 > > > kern.timecounter.fast_gettime: 1 > > > kern.timecounter.tick: 1 > > > kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000) > > > dummy(-1000000) > > > kern.timecounter.hardware: TSC-low > > > kern.timecounter.alloweddeviation: 5 > > > kern.timecounter.stepwarnings: 0 > > > kern.timecounter.tc.ACPI-fast.quality: 900 > > > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > > > kern.timecounter.tc.ACPI-fast.counter: 9047194 > > > kern.timecounter.tc.ACPI-fast.mask: 16777215 > > > kern.timecounter.tc.HPET.quality: 950 > > > kern.timecounter.tc.HPET.frequency: 14318180 > > > kern.timecounter.tc.HPET.counter: 2232552795 > > > kern.timecounter.tc.HPET.mask: 4294967295 > > > kern.timecounter.tc.i8254.quality: 0 > > > kern.timecounter.tc.i8254.frequency: 1193182 > > > kern.timecounter.tc.i8254.counter: 43410 > > > kern.timecounter.tc.i8254.mask: 65535 > > > kern.timecounter.tc.TSC-low.quality: 1000 > > > kern.timecounter.tc.TSC-low.frequency: 1200067168 > > > kern.timecounter.tc.TSC-low.counter: 2463146362 > > > kern.timecounter.tc.TSC-low.mask: 4294967295 > > > --- > > > > > > > > > > > > > > > > > > > > > > > > > > Also, what is the output of ntptime(8) in the vm? > > > > --- > > > > bhyve-vm% ntptime > > > > > > > > ntp_gettime() returns code 0 (OK) > > > > š time dd94930f.20ea2900ššFri, Oct 20 2017 18:21:51.128, (.128573699), > > > > š maximum error 1309110 us, estimated error 3 us, TAI offset 37 > > > > ntp_adjtime() returns code 0 (OK) > > > > š modes 0x0 (), > > > > š offset 0.000 us, frequency 0.000 ppm, interval 1 s, > > > > š maximum error 1309110 us, estimated error 3 us, > > > > š status 0x2001 (PLL,NANO), > > > > š time constant 6, precision 0.001 us, tolerance 496 ppm, > > > > --- > > > > > > > > Ian, thank you for your help! > > > > > > It seems odd to me that the frequency of the host HPET is 14.3mhz and > > that of the guest is 10.0mhz, but maybe that's a normal condition for > > bhyve. šI did find some google hits[1] for bhyve guest timekeeping > > trouble with the HPET timer which was solved by switching to a > > different timecounter. šTimecounter choices can't be controlled from > > loader.conf, so I guess a sysctl.conf entry of > > kern.timecounter.hardware="ACPI-fast" is the only way to fix that. šYou > > can also just do that command interactively first and see if it stops > > the time steps and ntp settles down. > The process seems to become more monotonic. But steps nevertheless: > --- > *XX.1ššššššXX.245ššššš4 uššš20ššš64šššš1šššš0.717šš-12.771ššš4.193 > *XX.1ššššššXX.245ššššš4 uššš28ššš64šššš3šššš0.751šš-41.970šš32.342 > *XX.1ššššššXX.245ššššš4 uššš23ššš64šššš7šššš0.748šš-59.505šš46.624 > *XX.1ššššššXX.245ššššš4 uššš18ššš64ššš17šššš0.699šš-75.164šš56.482 > *XX.1ššššššXX.245ššššš4 uššš14ššš64ššš37šššš0.669šš-90.112šš63.767 > *XX.1ššššššXX.245ššššš4 uššš11ššš64ššš77šššš0.605šš-10.567šš60.914 > *XX.1ššššššXX.245ššššš4 ušššš7ššš64šš177šššš0.591šš-169.54 116.762 > *XX.1ššššššXX.245ššššš4 ušššš3ššš64šš377šššš0.591šš-169.54 102.107 > *XX.1ššššššXX.245ššššš4 uššš68ššš64šš377šššš0.591šš-169.54 102.107 > *XX.1ššššššXX.245ššššš4 uššš63ššš64šš377šššš0.591šš-169.54šš88.424 > *XX.1ššššššXX.245ššššš4 uššš58ššš64šš377šššš0.591šš-169.54šš92.949 > *XX.1ššššššXX.245ššššš4 uššš55ššš64šš377šššš0.591šš-169.54 111.512 > *XX.1ššššššXX.245ššššš4 uššš50ššš64šš377šššš0.591šš-169.54 140.827 > *XX.1ššššššXX.245ššššš4 uššš45ššš64šš377šššš0.591šš-169.54 177.360 > *XX.1ššššššXX.245ššššš4 uššš43ššš64šš377šššš0.591šš-169.54 219.057 > šXX.1šššššš.STEP.šššš16 ušš588ššš64šššš0šššš0.000šššš0.000ššš0.000 > --- > > > > > This would be a workaround, not a fix per se. šIf the time steps go > > away, then something in bhyve's emulation of HPET (maybe only on some > > hardware?) must be buggy. > > > > > > [1] https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-April/003492.html > Also thanks for the link. Unfortunately the problem seems to persist. > Hmm, that almost looks normal... it noticed the clock was drifting fast, so it went into a frequency-training cycle (at the point where the offset stopped changing at -169.54) and then once it had figured out the frequency it did a step to get realigned, and (presumably) adjusted the frequency of the kernel clock. šThe output of ntptime before and after the step would show that... before the step the frequency would show as zero (which was the case in your ntptime output earlier), and after the step it would be non-zero. šNo more steps would occur after the first one, if that's what is happening here. -- Ian From owner-freebsd-virtualization@freebsd.org Fri Oct 20 18:15:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AF34E3C7AE; Fri, 20 Oct 2017 18:15:49 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward105j.mail.yandex.net (forward105j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946A3157C; Fri, 20 Oct 2017 18:15:48 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback9g.mail.yandex.net (mxback9g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:170]) by forward105j.mail.yandex.net (Yandex) with ESMTP id DA1A2183DC6; Fri, 20 Oct 2017 21:15:45 +0300 (MSK) Received: from smtp1j.mail.yandex.net (smtp1j.mail.yandex.net [2a02:6b8:0:801::ab]) by mxback9g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id sWvRtAedPA-FjDKSv2v; Fri, 20 Oct 2017 21:15:45 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508523345; bh=EkROp2IuGx5O7R+XmYyYVamjxLmExSwjYxyW6PRYLpM=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=MLOpgLEsIFucNhXK7/tx+8Z7ncNiCKAYtxtNLoq3xXclnGP+OlXKyIIIb5Fu8Y0Q8 TC2pFUenw7gacb9DPpRfIlV0AjTyGRHw8pRfDnQnmG3rVeUj9hvRQhZEaGSM9gZ+EP 4kSm8D7svRc3Wgko3BZyoRQimDV3XHdXkTvEe2xQ= Received: by smtp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 8WICKTfZc9-FiwmYCAN; Fri, 20 Oct 2017 21:15:44 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508523344; bh=EkROp2IuGx5O7R+XmYyYVamjxLmExSwjYxyW6PRYLpM=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=Ss/KBtWVkBMaCITjfZmP2FoGH4na4+PtzvpfW2mkpOnMv/V2PYDZ8E4SzEIGVO33u Q7k9IIkxbaqHCI9Nru5PZ4eS7AqEslGRSzZHQ/7468oFTeJj5ysh+vzpacqHaj9Ujp VLwCjy2qA1H10nU8JC5YsoBO3fo3nMHTYSIlyHbg= Authentication-Results: smtp1j.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> From: Boris Samorodov Message-ID: <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> Date: Fri, 20 Oct 2017 21:15:44 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508522667.1383.69.camel@freebsd.org> Content-Type: text/plain; charset=koi8-r Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 18:15:49 -0000 20.10.2017 21:04, Ian Lepore ΠΙΫΕΤ: > On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote: >> (CC to freebsd-virtualization@) >> >> 20.10.2017 19:32, Ian Lepore ΠΙΫΕΤ: >>> >>> On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote: >>>> >>>> 20.10.2017 18:31, Boris Samorodov ΠΙΫΕΤ: >>>>> >>>>> >>>>> 20.10.2017 18:12, Ian Lepore ΠΙΫΕΤ: >>>>>> >>>>>> >>>>>> On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote: >>>>>>> >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I have got a host: >>>>>>> --- >>>>>>> bhyve-host% uname -a >>>>>>> FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug >>>>>>> 25 05:25:26 MSK 2017 >>>>>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FASTššamd64 amd64 >>>>>>> --- >>>>>>> >>>>>>> And a bhyve vm: >>>>>>> --- >>>>>>> bhyve-vm: uname -a >>>>>>> FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri >>>>>>> Oct 20 05:12:17 MSK 2017 >>>>>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64Xššamd64 amd64 >>>>>>> --- >>>>>>> >>>>>>> The only difference at kernel configs is a colored console. :-) >>>>>>> >>>>>>> And here I get some weird (is it?) result at the VM (I expect ntpd to be >>>>>>> more stable): >>>>>>> --- >>>>>>> bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done >>>>>>> šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset >>>>>>> jitter >>>>>>> ============================================================================== >>>>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš9ššš64šššš3šššš0.605ššš-1.202 >>>>>>> 316.407 >>>>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš7ššš64šššš7šššš0.605ššš-1.202 >>>>>>> 358.395 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš5ššš64ššš17šššš0.615šš-328.42 >>>>>>> 181.405 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64ššš37šššš0.615šš-328.42 >>>>>>> 214.868 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64ššš37šššš0.615šš-328.42 >>>>>>> 214.868 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš63ššš64ššš77šššš0.615šš-328.42 >>>>>>> 268.618 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64šš177šššš0.615šš-328.42 >>>>>>> 333.175 >>>>>>> šXX.XX.XX.1šššššš.STEP.šššššššššš16 u 1910ššš64šššš0šššš0.000šššš0.000 >>>>>>> 0.000 >>>>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš27ššš64šššš1šššš0.703šš-262.63 >>>>>>> 0.004 >>>>>>> šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš31ššš64šššš1šššš0.649šš-331.43 >>>>>>> 68.800 >>>>>>> --- >>>>>>> >>>>>>> At the same time host's results are very stable: >>>>>>> --- >>>>>>> bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done >>>>>>> šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset >>>>>>> jitter >>>>>>> ============================================================================== >>>>>>> >>>>>>> >>>>>>> >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš1ššš64šššš1šššš0.401šššš0.176 >>>>>>> 0.106 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš6ššš64šššš3šššš0.401šššš0.176 >>>>>>> 0.459 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64šššš7šššš0.401šššš0.176 >>>>>>> 0.940 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64šššš7šššš0.401šššš0.176 >>>>>>> 0.940 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš64ššš64ššš17šššš0.401šššš0.176 >>>>>>> 1.566 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64ššš37šššš0.448šššš1.275 >>>>>>> 1.739 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš55ššš64ššš77šššš0.448šššš1.275 >>>>>>> 2.365 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš53ššš64šš177šššš0.448šššš1.275 >>>>>>> 3.110 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš50ššš64šš377šššš0.448šššš1.275 >>>>>>> 3.929 >>>>>>> *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš45ššš64šš377šššš0.443šššš8.750 >>>>>>> 4.722 >>>>>>> --- >>>>>>> >>>>>>> The network is organized via bridge -- host igb and vm tap interfaces >>>>>>> are members of one bridge. >>>>>>> >>>>>>> Are those results expected? Does it smell like a bug? Should I dig >>>>>>> furter? >>>>>>> >>>>>> So it is repeatedly stepping the clock in the VM? (Set >>>>>> kern.timecounter.stepwarnings=1 to log steps). >>>>> No kernel/ntpd messages for 20 minutes after setting this sysctl. >>>>> >>>>>> >>>>>> >>>>>> šThat is usually a sign >>>>>> that the chosen timecounter is running at a different frequency than it >>>>>> claimed to be when it registered itself -- the host may not be >>>>>> emulating the timer hardware properly in the guest. šWhat is the output >>>>>> of sysctl kern.timecounter in the vm? >>>>> --- >>>>> bhyve-vm% sysctl kern.timecounter >>>>> >>>>> kern.timecounter.tsc_shift: 1 >>>>> kern.timecounter.smp_tsc_adjust: 0 >>>>> kern.timecounter.smp_tsc: 0 >>>>> kern.timecounter.invariant_tsc: 1 >>>>> kern.timecounter.fast_gettime: 1 >>>>> kern.timecounter.tick: 1 >>>>> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100) >>>>> dummy(-1000000) >>>>> kern.timecounter.hardware: HPET >>>>> kern.timecounter.alloweddeviation: 5 >>>>> kern.timecounter.stepwarnings: 1 >>>>> kern.timecounter.tc.ACPI-fast.quality: 900 >>>>> kern.timecounter.tc.ACPI-fast.frequency: 3579545 >>>>> kern.timecounter.tc.ACPI-fast.counter: 4161213491 >>>>> kern.timecounter.tc.ACPI-fast.mask: 4294967295 >>>>> kern.timecounter.tc.HPET.quality: 950 >>>>> kern.timecounter.tc.HPET.frequency: 10000000 >>>>> kern.timecounter.tc.HPET.counter: 3518036865 >>>>> kern.timecounter.tc.HPET.mask: 4294967295 >>>>> kern.timecounter.tc.i8254.quality: 0 >>>>> kern.timecounter.tc.i8254.frequency: 1193182 >>>>> kern.timecounter.tc.i8254.counter: 47597 >>>>> kern.timecounter.tc.i8254.mask: 65535 >>>>> kern.timecounter.tc.TSC-low.quality: -100 >>>>> kern.timecounter.tc.TSC-low.frequency: 1199886114 >>>>> kern.timecounter.tc.TSC-low.counter: 1274338278 >>>>> kern.timecounter.tc.TSC-low.mask: 4294967295 >>>>> --- >>>> BTW, here is the host kern.timecounter: >>>> --- >>>> kern.timecounter.tsc_shift: 1 >>>> kern.timecounter.smp_tsc_adjust: 0 >>>> kern.timecounter.smp_tsc: 1 >>>> kern.timecounter.invariant_tsc: 1 >>>> kern.timecounter.fast_gettime: 1 >>>> kern.timecounter.tick: 1 >>>> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000) >>>> dummy(-1000000) >>>> kern.timecounter.hardware: TSC-low >>>> kern.timecounter.alloweddeviation: 5 >>>> kern.timecounter.stepwarnings: 0 >>>> kern.timecounter.tc.ACPI-fast.quality: 900 >>>> kern.timecounter.tc.ACPI-fast.frequency: 3579545 >>>> kern.timecounter.tc.ACPI-fast.counter: 9047194 >>>> kern.timecounter.tc.ACPI-fast.mask: 16777215 >>>> kern.timecounter.tc.HPET.quality: 950 >>>> kern.timecounter.tc.HPET.frequency: 14318180 >>>> kern.timecounter.tc.HPET.counter: 2232552795 >>>> kern.timecounter.tc.HPET.mask: 4294967295 >>>> kern.timecounter.tc.i8254.quality: 0 >>>> kern.timecounter.tc.i8254.frequency: 1193182 >>>> kern.timecounter.tc.i8254.counter: 43410 >>>> kern.timecounter.tc.i8254.mask: 65535 >>>> kern.timecounter.tc.TSC-low.quality: 1000 >>>> kern.timecounter.tc.TSC-low.frequency: 1200067168 >>>> kern.timecounter.tc.TSC-low.counter: 2463146362 >>>> kern.timecounter.tc.TSC-low.mask: 4294967295 >>>> --- >>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> Also, what is the output of ntptime(8) in the vm? >>>>> --- >>>>> bhyve-vm% ntptime >>>>> >>>>> ntp_gettime() returns code 0 (OK) >>>>> š time dd94930f.20ea2900ššFri, Oct 20 2017 18:21:51.128, (.128573699), >>>>> š maximum error 1309110 us, estimated error 3 us, TAI offset 37 >>>>> ntp_adjtime() returns code 0 (OK) >>>>> š modes 0x0 (), >>>>> š offset 0.000 us, frequency 0.000 ppm, interval 1 s, >>>>> š maximum error 1309110 us, estimated error 3 us, >>>>> š status 0x2001 (PLL,NANO), >>>>> š time constant 6, precision 0.001 us, tolerance 496 ppm, >>>>> --- >>>>> >>>>> Ian, thank you for your help! >>>>> >>> It seems odd to me that the frequency of the host HPET is 14.3mhz and >>> that of the guest is 10.0mhz, but maybe that's a normal condition for >>> bhyve. šI did find some google hits[1] for bhyve guest timekeeping >>> trouble with the HPET timer which was solved by switching to a >>> different timecounter. šTimecounter choices can't be controlled from >>> loader.conf, so I guess a sysctl.conf entry of >>> kern.timecounter.hardware="ACPI-fast" is the only way to fix that. šYou >>> can also just do that command interactively first and see if it stops >>> the time steps and ntp settles down. >> The process seems to become more monotonic. But steps nevertheless: >> --- >> *XX.1ššššššXX.245ššššš4 uššš20ššš64šššš1šššš0.717šš-12.771ššš4.193 >> *XX.1ššššššXX.245ššššš4 uššš28ššš64šššš3šššš0.751šš-41.970šš32.342 >> *XX.1ššššššXX.245ššššš4 uššš23ššš64šššš7šššš0.748šš-59.505šš46.624 >> *XX.1ššššššXX.245ššššš4 uššš18ššš64ššš17šššš0.699šš-75.164šš56.482 >> *XX.1ššššššXX.245ššššš4 uššš14ššš64ššš37šššš0.669šš-90.112šš63.767 >> *XX.1ššššššXX.245ššššš4 uššš11ššš64ššš77šššš0.605šš-10.567šš60.914 >> *XX.1ššššššXX.245ššššš4 ušššš7ššš64šš177šššš0.591šš-169.54 116.762 >> *XX.1ššššššXX.245ššššš4 ušššš3ššš64šš377šššš0.591šš-169.54 102.107 >> *XX.1ššššššXX.245ššššš4 uššš68ššš64šš377šššš0.591šš-169.54 102.107 >> *XX.1ššššššXX.245ššššš4 uššš63ššš64šš377šššš0.591šš-169.54šš88.424 >> *XX.1ššššššXX.245ššššš4 uššš58ššš64šš377šššš0.591šš-169.54šš92.949 >> *XX.1ššššššXX.245ššššš4 uššš55ššš64šš377šššš0.591šš-169.54 111.512 >> *XX.1ššššššXX.245ššššš4 uššš50ššš64šš377šššš0.591šš-169.54 140.827 >> *XX.1ššššššXX.245ššššš4 uššš45ššš64šš377šššš0.591šš-169.54 177.360 >> *XX.1ššššššXX.245ššššš4 uššš43ššš64šš377šššš0.591šš-169.54 219.057 >> šXX.1šššššš.STEP.šššš16 ušš588ššš64šššš0šššš0.000šššš0.000ššš0.000 >> --- >> >>> >>> This would be a workaround, not a fix per se. šIf the time steps go >>> away, then something in bhyve's emulation of HPET (maybe only on some >>> hardware?) must be buggy. >>> >>> >>> [1] https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-April/003492.html >> Also thanks for the link. Unfortunately the problem seems to persist. >> > > Hmm, that almost looks normal... it noticed the clock was drifting > fast, so it went into a frequency-training cycle (at the point where > the offset stopped changing at -169.54) and then once it had figured > out the frequency it did a step to get realigned, and (presumably) > adjusted the frequency of the kernel clock. šThe output of ntptime > before and after the step would show that... before the step the > frequency would show as zero (which was the case in your ntptime output > earlier), and after the step it would be non-zero. šNo more steps would > occur after the first one, if that's what is happening here. Those were following steps: --- XX.1 XX.245 4 u 15 64 1 0.597 -51.802 7.547 XX.1 XX.245 4 u 17 64 1 0.597 -51.802 29.526 XX.1 XX.245 4 u 19 64 1 0.597 -51.802 52.760 XX.1 XX.245 4 u 19 64 3 0.597 -51.802 77.354 XX.1 XX.245 4 u 16 64 7 0.597 -51.802 103.182 XX.1 XX.245 4 u 12 64 17 0.597 -51.802 138.976 *XX.1 XX.245 4 u 9 64 37 0.700 -200.07 109.275 XX.1 .STEP. 16 u 1100 64 0 0.000 0.000 0.002 XX.1 XX.245 4 u 63 64 1 0.645 -3.620 0.002 XX.1 XX.245 4 u 34 64 1 0.645 -3.620 66.907 XX.1 XX.245 4 u 36 64 1 0.643 -115.30 85.082--- No stability. :-) But I see your point. I'll leave it till morning with hope it may stabilize. -- WBR, Boris Samorodov (bsam) FreeBSD Committer, http://www.FreeBSD.org The Power To Serve From owner-freebsd-virtualization@freebsd.org Fri Oct 20 18:21:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB445E3CAA0 for ; Fri, 20 Oct 2017 18:21:39 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A6231BD0 for ; Fri, 20 Oct 2017 18:21:39 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 8c86b86c-b5c3-11e7-a938-4f970e858fdb X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id 8c86b86c-b5c3-11e7-a938-4f970e858fdb; Fri, 20 Oct 2017 18:21:54 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v9KILaK3011644; Fri, 20 Oct 2017 12:21:36 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1508523696.1383.75.camel@freebsd.org> Subject: Re: host, bhyve vm and ntpd From: Ian Lepore To: Boris Samorodov , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Date: Fri, 20 Oct 2017 12:21:36 -0600 In-Reply-To: <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> Content-Type: text/plain; charset="koi8-r" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 18:21:40 -0000 On Fri, 2017-10-20 at 21:15 +0300, Boris Samorodov wrote: > 20.10.2017 21:04, Ian Lepore ΠΙΫΕΤ: > > > > On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote: > > > > > > (CC to freebsd-virtualization@) > > > > > > 20.10.2017 19:32, Ian Lepore ΠΙΫΕΤ: > > > > > > > > > > > > On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote: > > > > > > > > > > > > > > > 20.10.2017 18:31, Boris Samorodov ΠΙΫΕΤ: > > > > > > > > > > > > > > > > > > > > > > > > 20.10.2017 18:12, Ian Lepore ΠΙΫΕΤ: > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > I have got a host: > > > > > > > > --- > > > > > > > > bhyve-host% uname -a > > > > > > > > FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug > > > > > > > > 25 05:25:26 MSK 2017 > > > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FASTššamd64 amd64 > > > > > > > > --- > > > > > > > > > > > > > > > > And a bhyve vm: > > > > > > > > --- > > > > > > > > bhyve-vm: uname -a > > > > > > > > FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri > > > > > > > > Oct 20 05:12:17 MSK 2017 > > > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64Xššamd64 amd64 > > > > > > > > --- > > > > > > > > > > > > > > > > The only difference at kernel configs is a colored console. :-) > > > > > > > > > > > > > > > > And here I get some weird (is it?) result at the VM (I expect ntpd to be > > > > > > > > more stable): > > > > > > > > --- > > > > > > > > bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done > > > > > > > > šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset > > > > > > > > jitter > > > > > > > > ============================================================================== > > > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš9ššš64šššš3šššš0.605ššš-1.202 > > > > > > > > 316.407 > > > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš7ššš64šššš7šššš0.605ššš-1.202 > > > > > > > > 358.395 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš5ššš64ššš17šššš0.615šš-328.42 > > > > > > > > 181.405 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64ššš37šššš0.615šš-328.42 > > > > > > > > 214.868 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64ššš37šššš0.615šš-328.42 > > > > > > > > 214.868 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš63ššš64ššš77šššš0.615šš-328.42 > > > > > > > > 268.618 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64šš177šššš0.615šš-328.42 > > > > > > > > 333.175 > > > > > > > > šXX.XX.XX.1šššššš.STEP.šššššššššš16 u 1910ššš64šššš0šššš0.000šššš0.000 > > > > > > > > 0.000 > > > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš27ššš64šššš1šššš0.703šš-262.63 > > > > > > > > 0.004 > > > > > > > > šXX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš31ššš64šššš1šššš0.649šš-331.43 > > > > > > > > 68.800 > > > > > > > > --- > > > > > > > > > > > > > > > > At the same time host's results are very stable: > > > > > > > > --- > > > > > > > > bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done > > > > > > > > šššššremotešššššššššššrefidššššššst t when poll reachšššdelayšššoffset > > > > > > > > jitter > > > > > > > > ============================================================================== > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš1ššš64šššš1šššš0.401šššš0.176 > > > > > > > > 0.106 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš6ššš64šššš3šššš0.401šššš0.176 > > > > > > > > 0.459 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 ušššš3ššš64šššš7šššš0.401šššš0.176 > > > > > > > > 0.940 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš67ššš64šššš7šššš0.401šššš0.176 > > > > > > > > 0.940 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš64ššš64ššš17šššš0.401šššš0.176 > > > > > > > > 1.566 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš60ššš64ššš37šššš0.448šššš1.275 > > > > > > > > 1.739 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš55ššš64ššš77šššš0.448šššš1.275 > > > > > > > > 2.365 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš53ššš64šš177šššš0.448šššš1.275 > > > > > > > > 3.110 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš50ššš64šš377šššš0.448šššš1.275 > > > > > > > > 3.929 > > > > > > > > *XX.XX.XX.1ššššššXX.XX.XX.245ššššš4 uššš45ššš64šš377šššš0.443šššš8.750 > > > > > > > > 4.722 > > > > > > > > --- > > > > > > > > > > > > > > > > The network is organized via bridge -- host igb and vm tap interfaces > > > > > > > > are members of one bridge. > > > > > > > > > > > > > > > > Are those results expected? Does it smell like a bug? Should I dig > > > > > > > > furter? > > > > > > > > > > > > > > > So it is repeatedly stepping the clock in the VM? (Set > > > > > > > kern.timecounter.stepwarnings=1 to log steps). > > > > > > No kernel/ntpd messages for 20 minutes after setting this sysctl. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > šThat is usually a sign > > > > > > > that the chosen timecounter is running at a different frequency than it > > > > > > > claimed to be when it registered itself -- the host may not be > > > > > > > emulating the timer hardware properly in the guest. šWhat is the output > > > > > > > of sysctl kern.timecounter in the vm? > > > > > > --- > > > > > > bhyve-vm% sysctl kern.timecounter > > > > > > > > > > > > kern.timecounter.tsc_shift: 1 > > > > > > kern.timecounter.smp_tsc_adjust: 0 > > > > > > kern.timecounter.smp_tsc: 0 > > > > > > kern.timecounter.invariant_tsc: 1 > > > > > > kern.timecounter.fast_gettime: 1 > > > > > > kern.timecounter.tick: 1 > > > > > > kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100) > > > > > > dummy(-1000000) > > > > > > kern.timecounter.hardware: HPET > > > > > > kern.timecounter.alloweddeviation: 5 > > > > > > kern.timecounter.stepwarnings: 1 > > > > > > kern.timecounter.tc.ACPI-fast.quality: 900 > > > > > > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > > > > > > kern.timecounter.tc.ACPI-fast.counter: 4161213491 > > > > > > kern.timecounter.tc.ACPI-fast.mask: 4294967295 > > > > > > kern.timecounter.tc.HPET.quality: 950 > > > > > > kern.timecounter.tc.HPET.frequency: 10000000 > > > > > > kern.timecounter.tc.HPET.counter: 3518036865 > > > > > > kern.timecounter.tc.HPET.mask: 4294967295 > > > > > > kern.timecounter.tc.i8254.quality: 0 > > > > > > kern.timecounter.tc.i8254.frequency: 1193182 > > > > > > kern.timecounter.tc.i8254.counter: 47597 > > > > > > kern.timecounter.tc.i8254.mask: 65535 > > > > > > kern.timecounter.tc.TSC-low.quality: -100 > > > > > > kern.timecounter.tc.TSC-low.frequency: 1199886114 > > > > > > kern.timecounter.tc.TSC-low.counter: 1274338278 > > > > > > kern.timecounter.tc.TSC-low.mask: 4294967295 > > > > > > --- > > > > > BTW, here is the host kern.timecounter: > > > > > --- > > > > > kern.timecounter.tsc_shift: 1 > > > > > kern.timecounter.smp_tsc_adjust: 0 > > > > > kern.timecounter.smp_tsc: 1 > > > > > kern.timecounter.invariant_tsc: 1 > > > > > kern.timecounter.fast_gettime: 1 > > > > > kern.timecounter.tick: 1 > > > > > kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000) > > > > > dummy(-1000000) > > > > > kern.timecounter.hardware: TSC-low > > > > > kern.timecounter.alloweddeviation: 5 > > > > > kern.timecounter.stepwarnings: 0 > > > > > kern.timecounter.tc.ACPI-fast.quality: 900 > > > > > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > > > > > kern.timecounter.tc.ACPI-fast.counter: 9047194 > > > > > kern.timecounter.tc.ACPI-fast.mask: 16777215 > > > > > kern.timecounter.tc.HPET.quality: 950 > > > > > kern.timecounter.tc.HPET.frequency: 14318180 > > > > > kern.timecounter.tc.HPET.counter: 2232552795 > > > > > kern.timecounter.tc.HPET.mask: 4294967295 > > > > > kern.timecounter.tc.i8254.quality: 0 > > > > > kern.timecounter.tc.i8254.frequency: 1193182 > > > > > kern.timecounter.tc.i8254.counter: 43410 > > > > > kern.timecounter.tc.i8254.mask: 65535 > > > > > kern.timecounter.tc.TSC-low.quality: 1000 > > > > > kern.timecounter.tc.TSC-low.frequency: 1200067168 > > > > > kern.timecounter.tc.TSC-low.counter: 2463146362 > > > > > kern.timecounter.tc.TSC-low.mask: 4294967295 > > > > > --- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also, what is the output of ntptime(8) in the vm? > > > > > > --- > > > > > > bhyve-vm% ntptime > > > > > > > > > > > > ntp_gettime() returns code 0 (OK) > > > > > > š time dd94930f.20ea2900ššFri, Oct 20 2017 18:21:51.128, (.128573699), > > > > > > š maximum error 1309110 us, estimated error 3 us, TAI offset 37 > > > > > > ntp_adjtime() returns code 0 (OK) > > > > > > š modes 0x0 (), > > > > > > š offset 0.000 us, frequency 0.000 ppm, interval 1 s, > > > > > > š maximum error 1309110 us, estimated error 3 us, > > > > > > š status 0x2001 (PLL,NANO), > > > > > > š time constant 6, precision 0.001 us, tolerance 496 ppm, > > > > > > --- > > > > > > > > > > > > Ian, thank you for your help! > > > > > > > > > > It seems odd to me that the frequency of the host HPET is 14.3mhz and > > > > that of the guest is 10.0mhz, but maybe that's a normal condition for > > > > bhyve. šI did find some google hits[1] for bhyve guest timekeeping > > > > trouble with the HPET timer which was solved by switching to a > > > > different timecounter. šTimecounter choices can't be controlled from > > > > loader.conf, so I guess a sysctl.conf entry of > > > > kern.timecounter.hardware="ACPI-fast" is the only way to fix that. šYou > > > > can also just do that command interactively first and see if it stops > > > > the time steps and ntp settles down. > > > The process seems to become more monotonic. But steps nevertheless: > > > --- > > > *XX.1ššššššXX.245ššššš4 uššš20ššš64šššš1šššš0.717šš-12.771ššš4.193 > > > *XX.1ššššššXX.245ššššš4 uššš28ššš64šššš3šššš0.751šš-41.970šš32.342 > > > *XX.1ššššššXX.245ššššš4 uššš23ššš64šššš7šššš0.748šš-59.505šš46.624 > > > *XX.1ššššššXX.245ššššš4 uššš18ššš64ššš17šššš0.699šš-75.164šš56.482 > > > *XX.1ššššššXX.245ššššš4 uššš14ššš64ššš37šššš0.669šš-90.112šš63.767 > > > *XX.1ššššššXX.245ššššš4 uššš11ššš64ššš77šššš0.605šš-10.567šš60.914 > > > *XX.1ššššššXX.245ššššš4 ušššš7ššš64šš177šššš0.591šš-169.54 116.762 > > > *XX.1ššššššXX.245ššššš4 ušššš3ššš64šš377šššš0.591šš-169.54 102.107 > > > *XX.1ššššššXX.245ššššš4 uššš68ššš64šš377šššš0.591šš-169.54 102.107 > > > *XX.1ššššššXX.245ššššš4 uššš63ššš64šš377šššš0.591šš-169.54šš88.424 > > > *XX.1ššššššXX.245ššššš4 uššš58ššš64šš377šššš0.591šš-169.54šš92.949 > > > *XX.1ššššššXX.245ššššš4 uššš55ššš64šš377šššš0.591šš-169.54 111.512 > > > *XX.1ššššššXX.245ššššš4 uššš50ššš64šš377šššš0.591šš-169.54 140.827 > > > *XX.1ššššššXX.245ššššš4 uššš45ššš64šš377šššš0.591šš-169.54 177.360 > > > *XX.1ššššššXX.245ššššš4 uššš43ššš64šš377šššš0.591šš-169.54 219.057 > > > šXX.1šššššš.STEP.šššš16 ušš588ššš64šššš0šššš0.000šššš0.000ššš0.000 > > > --- > > > > > > > > > > > > > > > This would be a workaround, not a fix per se. šIf the time steps go > > > > away, then something in bhyve's emulation of HPET (maybe only on some > > > > hardware?) must be buggy. > > > > > > > > > > > > [1] https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-April/003492.html > > > Also thanks for the link. Unfortunately the problem seems to persist. > > > > > Hmm, that almost looks normal... it noticed the clock was drifting > > fast, so it went into a frequency-training cycle (at the point where > > the offset stopped changing at -169.54) and then once it had figured > > out the frequency it did a step to get realigned, and (presumably) > > adjusted the frequency of the kernel clock. šThe output of ntptime > > before and after the step would show that... before the step the > > frequency would show as zero (which was the case in your ntptime output > > earlier), and after the step it would be non-zero. šNo more steps would > > occur after the first one, if that's what is happening here. > Those were following steps: > --- > šXX.1ššššššXX.245ššššš4 uššš15ššš64šššš1šššš0.597šš-51.802ššš7.547 > šXX.1ššššššXX.245ššššš4 uššš17ššš64šššš1šššš0.597šš-51.802šš29.526 > šXX.1ššššššXX.245ššššš4 uššš19ššš64šššš1šššš0.597šš-51.802šš52.760 > šXX.1ššššššXX.245ššššš4 uššš19ššš64šššš3šššš0.597šš-51.802šš77.354 > šXX.1ššššššXX.245ššššš4 uššš16ššš64šššš7šššš0.597šš-51.802 103.182 > šXX.1ššššššXX.245ššššš4 uššš12ššš64ššš17šššš0.597šš-51.802 138.976 > *XX.1ššššššXX.245ššššš4 ušššš9ššš64ššš37šššš0.700šš-200.07 109.275 > šXX.1šššššš.STEP.šššš16 u 1100ššš64šššš0šššš0.000šššš0.000ššš0.002 > šXX.1ššššššXX.245ššššš4 uššš63ššš64šššš1šššš0.645ššš-3.620ššš0.002 > šXX.1ššššššXX.245ššššš4 uššš34ššš64šššš1šššš0.645ššš-3.620šš66.907 > šXX.1ššššššXX.245ššššš4 uššš36ššš64šššš1šššš0.643šš-115.30šš85.082--- > > No stability. :-) But I see your point. I'll leave it till morning > with hope it may stabilize. > I don't think it will. šntpd doesn't need more than one cycle to figure out the frequency of a clock and adjust it (and record the amount in ntpd.drift for future use). šEither the clock is drifting at more than the max 500ppm rate, or it isn't really drifting steadily, it's still jumping around in an uneven manner. It might be interesting to try one of the lower-quality clocks such asš i8254, or try TSC-low, although there was a comment in that mail thread about how TSC might be a bad choice. Beyond that, I'm not sure what else to try. šIt might be necessary to get some bhyve developers involved (I know almost nothing about it). -- Ian From owner-freebsd-virtualization@freebsd.org Sat Oct 21 21:07:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06F0BE3A482; Sat, 21 Oct 2017 21:07:49 +0000 (UTC) (envelope-from mvoorhis@atom.mcvau.net) Received: from atom.mcvau.net (unknown [IPv6:2602:100:615f:bdb7:5054:ff:fe0a:d781]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "atomvm.mcvau.net", Issuer "atomvm.mcvau.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B053B6BDF0; Sat, 21 Oct 2017 21:07:48 +0000 (UTC) (envelope-from mvoorhis@atom.mcvau.net) Received: from atom.mcvau.net (localhost [127.0.0.1]) by atom.mcvau.net (8.15.2/8.15.2) with ESMTPS id v9LL7f4i027709 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 21 Oct 2017 17:07:41 -0400 (EDT) (envelope-from mvoorhis@atom.mcvau.net) Received: (from mvoorhis@localhost) by atom.mcvau.net (8.15.2/8.15.2/Submit) id v9LL7ejq021852; Sat, 21 Oct 2017 17:07:40 -0400 (EDT) (envelope-from mvoorhis) From: Michael Voorhis MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Message-ID: <23019.46875.929719.481108@atom.mcvau.net> Date: Sat, 21 Oct 2017 17:07:39 -0400 To: Ian Lepore Cc: Boris Samorodov , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Subject: Re: host, bhyve vm and ntpd In-Reply-To: <1508523696.1383.75.camel@freebsd.org> References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> X-Mailer: VM 8.2.0b under 25.3.1 (amd64-portbld-freebsd11.1) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on atom.mcvau.net X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Oct 2017 21:07:49 -0000 Ian Lepore writes: > Beyond that, I'm not sure what else to try. =A0It might be necessary = to > get some bhyve developers involved (I know almost nothing about it). NTPD behaves more normally on uniprocessor VMs. A FreeBSD bhyve-guest running on a freebsd host will select a different timecounter depending on whether it is a multiprocessor or a uniprocessor. My uniprocessor bhyve-vm selected TSC-low as the best timecounter in a uniprocessor. NTP functions there as expected. kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0= ) dummy(-1000000) kern.timecounter.hardware: TSC-low The very same VM, when given two total CPUs, selected HPET (if I recall) and the timekeeping with NTPD was unreliable, with many step-resets to the clock. From owner-freebsd-virtualization@freebsd.org Sat Oct 21 22:16:08 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F877E3B651 for ; Sat, 21 Oct 2017 22:16:08 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06F406D69A for ; Sat, 21 Oct 2017 22:16:07 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 695d617b-b6ad-11e7-a893-25625093991c X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 695d617b-b6ad-11e7-a893-25625093991c; Sat, 21 Oct 2017 22:15:58 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v9LMFrKh014285; Sat, 21 Oct 2017 16:15:53 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1508624153.1383.107.camel@freebsd.org> Subject: Re: host, bhyve vm and ntpd From: Ian Lepore To: Michael Voorhis Cc: Boris Samorodov , freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Date: Sat, 21 Oct 2017 16:15:53 -0600 In-Reply-To: <23019.46875.929719.481108@atom.mcvau.net> References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> Content-Type: multipart/mixed; boundary="=-85jbhbwt4hlyeaS0wt1C" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Oct 2017 22:16:08 -0000 --=-85jbhbwt4hlyeaS0wt1C Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 8bit On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: > Ian Lepore writes: > > > > Beyond that, I'm not sure what else to try.  It might be necessary to > > get some bhyve developers involved (I know almost nothing about it). > NTPD behaves more normally on uniprocessor VMs. > > A FreeBSD bhyve-guest running on a freebsd host will select a > different timecounter depending on whether it is a multiprocessor or a > uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best > timecounter in a uniprocessor.  NTP functions there as expected. > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) > kern.timecounter.hardware: TSC-low > > The very same VM, when given two total CPUs, selected HPET (if I > recall) and the timekeeping with NTPD was unreliable, with many > step-resets to the clock. > Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it looks right.  I wonder if this is just a simple roundoff error in converting between 10.0MHz and SBT units?  If so, that could be wished away easily by using a power-of-2 frequency for the virtual HPET.  I wonder if the attached patch is all that's needed? -- Ian --=-85jbhbwt4hlyeaS0wt1C Content-Disposition: inline; filename="temp.diff" Content-Type: text/x-patch; name="temp.diff"; charset="us-ascii" Content-Transfer-Encoding: 7bit Index: sys/amd64/vmm/io/vhpet.c =================================================================== --- sys/amd64/vmm/io/vhpet.c (revision 324176) +++ sys/amd64/vmm/io/vhpet.c (working copy) @@ -51,7 +51,7 @@ __FBSDID("$FreeBSD$"); static MALLOC_DEFINE(M_VHPET, "vhpet", "bhyve virtual hpet"); -#define HPET_FREQ 10000000 /* 10.0 Mhz */ +#define HPET_FREQ 16777216 /* 16.7 (2^24) Mhz */ #define FS_PER_S 1000000000000000ul /* Timer N Configuration and Capabilities Register */ --=-85jbhbwt4hlyeaS0wt1C--