From owner-freebsd-virtualization@freebsd.org Sun Oct 22 08:28:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7F7FE4CB8C; Sun, 22 Oct 2017 08:28:34 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward101o.mail.yandex.net (forward101o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B82E826BC; Sun, 22 Oct 2017 08:28:34 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback8g.mail.yandex.net (mxback8g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:169]) by forward101o.mail.yandex.net (Yandex) with ESMTP id 0C1E31342875; Sun, 22 Oct 2017 11:28:22 +0300 (MSK) Received: from smtp4o.mail.yandex.net (smtp4o.mail.yandex.net [2a02:6b8:0:1a2d::28]) by mxback8g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id QlXmCuw6DD-SLr0Wk23; Sun, 22 Oct 2017 11:28:21 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508660902; bh=oGywilQWzKTcb12LDAoF07lJi5+pL41ukz0b8jZ+SB8=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=SXcAF0mWWsIh5BccT3WbdNl8ZK/9OYHePhuFJoQLevpbxth26B95CixjAcwto7RJ3 WzYkbOY9AKGDGRE99qMqWISDN3zBHA40adgXj51NCyFiLPkPXJ4XnRPtUGGD5zqpZT 5z2iCxWs6OC02Qa5lm2Vn1Ny0JLnQ5bMrLJNDWRQ= Received: by smtp4o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id dIZPnZo0zk-SKpOU7Yt; Sun, 22 Oct 2017 11:28:21 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508660901; bh=oGywilQWzKTcb12LDAoF07lJi5+pL41ukz0b8jZ+SB8=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=IDQKtO4tJvKhJYD4QhEKlUJGbrunD9Eisbdf5AE965kSWSBDz6n/+rqwzVgmd55C8 kbwrv6h4kXsqSHAU5mZNmAh4e70/3zVFImqPzvNcqimUcdHLTVxK699jHcu2WYq6+u KGmUuv1mkZr3+HC/1gTJkORAPmMpELPyv29dRoP0= Authentication-Results: smtp4o.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Michael Voorhis , Ian Lepore Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> From: Boris Samorodov Message-ID: <1f33d52f-c087-9cc2-1d88-4f1019ea2c31@passap.ru> Date: Sun, 22 Oct 2017 11:28:20 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <23019.46875.929719.481108@atom.mcvau.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 08:28:35 -0000 22.10.2017 00:07, Michael Voorhis пишет: > Ian Lepore writes: >> Beyond that, I'm not sure what else to try.  It might be necessary to >> get some bhyve developers involved (I know almost nothing about it). > > NTPD behaves more normally on uniprocessor VMs. > > A FreeBSD bhyve-guest running on a freebsd host will select a > different timecounter depending on whether it is a multiprocessor or a > uniprocessor. My uniprocessor bhyve-vm selected TSC-low as the best > timecounter in a uniprocessor. NTP functions there as expected. > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) > kern.timecounter.hardware: TSC-low > > The very same VM, when given two total CPUs, selected HPET (if I > recall) and the timekeeping with NTPD was unreliable, with many > step-resets to the clock. Yep, the same here. I've switched to TSC-low at Bhyve guest and there is no stepping per 24 hours. -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Sun Oct 22 08:31:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C45C4E4CD49; Sun, 22 Oct 2017 08:31:26 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward105p.mail.yandex.net (forward105p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7794182929; Sun, 22 Oct 2017 08:31:26 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback9g.mail.yandex.net (mxback9g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:170]) by forward105p.mail.yandex.net (Yandex) with ESMTP id 110F74082344; Sun, 22 Oct 2017 11:31:24 +0300 (MSK) Received: from smtp1j.mail.yandex.net (smtp1j.mail.yandex.net [2a02:6b8:0:801::ab]) by mxback9g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id R6e03ZTspc-VND8iQgw; Sun, 22 Oct 2017 11:31:24 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508661084; bh=ePzbgkEZTdPXldLnns2AnQ86wNpHyWYplqQJZziLmpU=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=glh8EbTV/dTfq46Km6J0pXzn6VlYGMtHiPl3NPgYEJLnebTYEevGKrY/SgQC2Vaou VYG9a/EJE36tmKjiZTRCFmH7E/NHSanpMaAzjnoEtNRmlwdYdHNIeXK8opN4O4zXmQ zUOMqRdt6BQDDAL5ef83kjpV/fP07/XVe22OjZvA= Received: by smtp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id h3FGSjBdJ3-VNZePpNo; Sun, 22 Oct 2017 11:31:23 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508661083; bh=ePzbgkEZTdPXldLnns2AnQ86wNpHyWYplqQJZziLmpU=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=oJDo3wsGbRyOqnFL/Y/x2tGB4ije831ApS1ct2DPPg2wJRl/biWi3dHLx6sEyYgC5 GJkRvu8W+CE/tvuxP8TNQotWXdd9WcpBOE9zJT8CPZEuucdGjaExMIfVTN91xyYLH4 1eiTb7YEYaLMmTkPxUw8g5mPF1+WHIXNXmnTIlJI= Authentication-Results: smtp1j.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore , Michael Voorhis Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> From: Boris Samorodov Message-ID: <6dc189a0-e7bc-6c3b-2be6-80154e7ece42@passap.ru> Date: Sun, 22 Oct 2017 11:31:22 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508624153.1383.107.camel@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Language: ru-RU Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 08:31:26 -0000 22.10.2017 01:15, Ian Lepore пишет: > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: >> Ian Lepore writes: >>> >>> Beyond that, I'm not sure what else to try.  It might be necessary to >>> get some bhyve developers involved (I know almost nothing about it). >> NTPD behaves more normally on uniprocessor VMs. >> >> A FreeBSD bhyve-guest running on a freebsd host will select a >> different timecounter depending on whether it is a multiprocessor or a >> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best >> timecounter in a uniprocessor.  NTP functions there as expected. >> >> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) >> kern.timecounter.hardware: TSC-low >> >> The very same VM, when given two total CPUs, selected HPET (if I >> recall) and the timekeeping with NTPD was unreliable, with many >> step-resets to the clock. >> > > Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it > looks right.  I wonder if this is just a simple roundoff error in > converting between 10.0MHz and SBT units?  If so, that could be wished > away easily by using a power-of-2 frequency for the virtual HPET.  I > wonder if the attached patch is all that's needed? I've tried the patch (at bhyve guest) and nothing has changed. Should the patched system be tested at bhyve guest or bhyve host? -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Sun Oct 22 15:22:20 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BC5CE54BEA; Sun, 22 Oct 2017 15:22:20 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30FEC69DC7; Sun, 22 Oct 2017 15:22:19 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v9MFMHwG039119; Sun, 22 Oct 2017 08:22:17 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v9MFMGi6039118; Sun, 22 Oct 2017 08:22:16 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201710221522.v9MFMGi6039118@pdx.rh.CN85.dnsmgr.net> Subject: Re: host, bhyve vm and ntpd In-Reply-To: <6dc189a0-e7bc-6c3b-2be6-80154e7ece42@passap.ru> To: Boris Samorodov Date: Sun, 22 Oct 2017 08:22:16 -0700 (PDT) CC: Ian Lepore , Michael Voorhis , freebsd-current@freebsd.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 15:22:20 -0000 [ Charset UTF-8 unsupported, converting... ] > 22.10.2017 01:15, Ian Lepore ?????: > > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: > >> Ian Lepore writes: > >>> > >>> Beyond that, I'm not sure what else to try. ?It might be necessary to > >>> get some bhyve developers involved (I know almost nothing about it). > >> NTPD behaves more normally on uniprocessor VMs. > >> > >> A FreeBSD bhyve-guest running on a freebsd host will select a > >> different timecounter depending on whether it is a multiprocessor or a > >> uniprocessor.??My uniprocessor bhyve-vm selected TSC-low as the best > >> timecounter in a uniprocessor.??NTP functions there as expected. > >> > >> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) > >> kern.timecounter.hardware: TSC-low > >> > >> The very same VM, when given two total CPUs, selected HPET (if I > >> recall) and the timekeeping with NTPD was unreliable, with many > >> step-resets to the clock. > >> > > > > Hmm, I just had glance at the code in?sys/amd64/vmm/io/vhpet.c and it > > looks right. ?I wonder if this is just a simple roundoff error in > > converting between 10.0MHz and SBT units? ?If so, that could be wished > > away easily by using a power-of-2 frequency for the virtual HPET. ?I > > wonder if the attached patch is all that's needed? > I've tried the patch (at bhyve guest) and nothing has changed. Should > the patched system be tested at bhyve guest or bhyve host? I believe the suggested patch would have to be made to the bhyve host. Also on the host and guest what are the values of sysctl kern.timecounter.tc.HPET sysctl kern.timecounter.tc.i8254 Getting good ntpd behavior in a VM guest of any kind is sometimes a non trivial thing to do. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Sun Oct 22 15:38:59 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69623E54FA1; Sun, 22 Oct 2017 15:38:59 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward106j.mail.yandex.net (forward106j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02C126A573; Sun, 22 Oct 2017 15:38:58 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback13g.mail.yandex.net (mxback13g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:92]) by forward106j.mail.yandex.net (Yandex) with ESMTP id 273BB18056D0; Sun, 22 Oct 2017 18:38:46 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback13g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ZKbttz1MG3-cjvKQAvG; Sun, 22 Oct 2017 18:38:46 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508686726; bh=FYZqtmDoaEhbr53uk4NqahytsU9FFTmtNB/8unK74IM=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=KFpevbm7iZ5n75/+Mhspd9NnH149MZoIqgnEG+LXERsrpOdWqsvNMWzwOfoWEToOj l0YBIRsCVoPeISxZLJyKOmHQGyQ4ZPC/Sdnt98kgZ5x3+kr0eO1EDMmfXShYIE/cLf JYW15JSwwlgcs0slZ77VN0m5d6Ovx8XNYdxLjefA= Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id hCirSmNIQ7-cihSflGe; Sun, 22 Oct 2017 18:38:44 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508686724; bh=FYZqtmDoaEhbr53uk4NqahytsU9FFTmtNB/8unK74IM=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=qEqlZvL7NsLnM5PfXTiPvqW75XcQWk+ZuOZWlC9WxfJPrTAYD9uMwOEpPyMA9bfIG OA9Glq8XVAIATvBoYpMeqEtbmZHqBwZaB04kmHYemR31+vxBegQ/6t5DBrvLIQYJjk KMU+uSrJoHnyYNv1ivni5UXsuECa/Ly0u6PLFgS4= Authentication-Results: smtp3p.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: "Rodney W. Grimes" Cc: Ian Lepore , Michael Voorhis , freebsd-current@freebsd.org, freebsd-virtualization@freebsd.org References: <201710221522.v9MFMGi6039118@pdx.rh.CN85.dnsmgr.net> From: Boris Samorodov Message-ID: <6ddef049-3e24-5cac-cccc-d99db9a835af@passap.ru> Date: Sun, 22 Oct 2017 18:38:44 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <201710221522.v9MFMGi6039118@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 15:38:59 -0000 22.10.2017 18:22, Rodney W. Grimes пишет: > [ Charset UTF-8 unsupported, converting... ] >> 22.10.2017 01:15, Ian Lepore ?????: >>> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: >>>> Ian Lepore writes: >>>>> >>>>> Beyond that, I'm not sure what else to try. ?It might be necessary to >>>>> get some bhyve developers involved (I know almost nothing about it). >>>> NTPD behaves more normally on uniprocessor VMs. >>>> >>>> A FreeBSD bhyve-guest running on a freebsd host will select a >>>> different timecounter depending on whether it is a multiprocessor or a >>>> uniprocessor.??My uniprocessor bhyve-vm selected TSC-low as the best >>>> timecounter in a uniprocessor.??NTP functions there as expected. >>>> >>>> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) >>>> kern.timecounter.hardware: TSC-low >>>> >>>> The very same VM, when given two total CPUs, selected HPET (if I >>>> recall) and the timekeeping with NTPD was unreliable, with many >>>> step-resets to the clock. >>>> >>> >>> Hmm, I just had glance at the code in?sys/amd64/vmm/io/vhpet.c and it >>> looks right. ?I wonder if this is just a simple roundoff error in >>> converting between 10.0MHz and SBT units? ?If so, that could be wished >>> away easily by using a power-of-2 frequency for the virtual HPET. ?I >>> wonder if the attached patch is all that's needed? >> I've tried the patch (at bhyve guest) and nothing has changed. Should >> the patched system be tested at bhyve guest or bhyve host? > > I believe the suggested patch would have to be made to the bhyve > host OK, I'd do it tomorrow and report back. >. Also on the host and guest what are the values of > sysctl kern.timecounter.tc.HPET > sysctl kern.timecounter.tc.i8254 Here they are: --- bhyve-host% sysctl kern.timecounter.tc.HPET kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 14318180 kern.timecounter.tc.HPET.counter: 2138094157 kern.timecounter.tc.HPET.mask: 4294967295 bhyve-host% sysctl kern.timecounter.tc.i8254 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 54883 kern.timecounter.tc.i8254.mask: 65535 --- bhyve-guest% sysctl kern.timecounter.tc.HPET kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 10000000 kern.timecounter.tc.HPET.counter: 969429421 kern.timecounter.tc.HPET.mask: 4294967295 bhyve-guest% sysctl kern.timecounter.tc.i8254 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 39893 kern.timecounter.tc.i8254.mask: 65535 --- > Getting good ntpd behavior in a VM guest of any kind is sometimes a > non trivial thing to do. As a side note, I have a CentOS-7 bhyve VM at the same host. And it was enough to run chronyd with default config. Which stepped twice and is stable (no messages) for several days, current log: --- Oct 19 16:01:03 c.vpn systemd[1]: Starting NTP client/server... Oct 19 16:01:03 c.vpn chronyd[27043]: chronyd version 3.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG) Oct 19 16:01:03 c.vpn chronyd[27043]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift Oct 19 16:01:03 c.vpn systemd[1]: Started NTP client/server. Oct 19 16:01:07 c.vpn chronyd[27043]: Selected source XX.XX.XX.1 Oct 19 16:01:07 c.vpn chronyd[27043]: System clock wrong by -44.392782 seconds, adjustment started Oct 19 16:00:23 c.vpn chronyd[27043]: System clock was stepped by -44.392782 seconds Oct 19 16:00:34 c.vpn chronyd[27043]: System clock was stepped by 0.000001 seconds --- -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Sun Oct 22 16:02:35 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32B8AE5568B for ; Sun, 22 Oct 2017 16:02:35 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B31106AFAD for ; Sun, 22 Oct 2017 16:02:34 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 67749f99-b742-11e7-a893-25625093991c X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 67749f99-b742-11e7-a893-25625093991c; Sun, 22 Oct 2017 16:02:30 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v9MG2OBX001409; Sun, 22 Oct 2017 10:02:25 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1508688144.1383.114.camel@freebsd.org> Subject: Re: host, bhyve vm and ntpd From: Ian Lepore To: Boris Samorodov , Michael Voorhis Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Date: Sun, 22 Oct 2017 10:02:24 -0600 In-Reply-To: <6dc189a0-e7bc-6c3b-2be6-80154e7ece42@passap.ru> References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> <6dc189a0-e7bc-6c3b-2be6-80154e7ece42@passap.ru> Content-Type: text/plain; charset="koi8-r" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 16:02:35 -0000 On Sun, 2017-10-22 at 11:31 +0300, Boris Samorodov wrote: > 22.10.2017 01:15, Ian Lepore ÐÉÛÅÔ: > > > > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: > > > > > > Ian Lepore writes: > > > > > > > > > > > > Beyond that, I'm not sure what else to try. šIt might be necessary to > > > > get some bhyve developers involved (I know almost nothing about it). > > > NTPD behaves more normally on uniprocessor VMs. > > > > > > A FreeBSD bhyve-guest running on a freebsd host will select a > > > different timecounter depending on whether it is a multiprocessor or a > > > uniprocessor.ššMy uniprocessor bhyve-vm selected TSC-low as the best > > > timecounter in a uniprocessor.ššNTP functions there as expected. > > > > > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) > > > kern.timecounter.hardware: TSC-low > > > > > > The very same VM, when given two total CPUs, selected HPET (if I > > > recall) and the timekeeping with NTPD was unreliable, with many > > > step-resets to the clock. > > > > > Hmm, I just had glance at the code inšsys/amd64/vmm/io/vhpet.c and it > > looks right. šI wonder if this is just a simple roundoff error in > > converting between 10.0MHz and SBT units? šIf so, that could be wished > > away easily by using a power-of-2 frequency for the virtual HPET. šI > > wonder if the attached patch is all that's needed? > I've tried the patch (at bhyve guest) and nothing has changed. Should > the patched system be tested at bhyve guest or bhyve host? > Oh, I'm sorry, I should have mentioned that's for the host side. -- Ian From owner-freebsd-virtualization@freebsd.org Sun Oct 22 16:16:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B746E55DA6; Sun, 22 Oct 2017 16:16:46 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward100p.mail.yandex.net (forward100p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E16C86C409; Sun, 22 Oct 2017 16:16:45 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback5o.mail.yandex.net (mxback5o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1f]) by forward100p.mail.yandex.net (Yandex) with ESMTP id BEFE45103A95; Sun, 22 Oct 2017 19:16:42 +0300 (MSK) Received: from smtp1j.mail.yandex.net (smtp1j.mail.yandex.net [2a02:6b8:0:801::ab]) by mxback5o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id W3U1j44byJ-GgHWOQ8b; Sun, 22 Oct 2017 19:16:42 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508689002; bh=xQCui/wG8ltvuXaxMBNG3fAMoJRanYRvllWtwATHlNo=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=actaExz1izBcKqZb53PTFO2uYncumFdblXnqBs3HVChZVz+8P0ZNgvRQfHM3x/onO Z/zXZhHVev2lrEtWWy3nVW0EXrHtx2Hq+Nwbfer6q8z61FZzKqZ68bR82NevjxDrml 9EyCpATExrF4Y9nquUkCjK/QpstO9hoDtp8Fypzs= Received: by smtp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id crX7XuOydV-GgZGaJF9; Sun, 22 Oct 2017 19:16:42 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508689002; bh=xQCui/wG8ltvuXaxMBNG3fAMoJRanYRvllWtwATHlNo=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=actaExz1izBcKqZb53PTFO2uYncumFdblXnqBs3HVChZVz+8P0ZNgvRQfHM3x/onO Z/zXZhHVev2lrEtWWy3nVW0EXrHtx2Hq+Nwbfer6q8z61FZzKqZ68bR82NevjxDrml 9EyCpATExrF4Y9nquUkCjK/QpstO9hoDtp8Fypzs= Authentication-Results: smtp1j.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> <6dc189a0-e7bc-6c3b-2be6-80154e7ece42@passap.ru> <1508688144.1383.114.camel@freebsd.org> From: Boris Samorodov Message-ID: <4494cefa-5fe9-8482-e0de-14d7e7c09e75@passap.ru> Date: Sun, 22 Oct 2017 19:16:41 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508688144.1383.114.camel@freebsd.org> Content-Type: text/plain; charset=koi8-r Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Oct 2017 16:16:46 -0000 22.10.2017 19:02, Ian Lepore ÐÉÛÅÔ: > On Sun, 2017-10-22 at 11:31 +0300, Boris Samorodov wrote: >> 22.10.2017 01:15, Ian Lepore ÐÉÛÅÔ: >>> >>> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: >>>> >>>> Ian Lepore writes: >>>>> >>>>> >>>>> Beyond that, I'm not sure what else to try. šIt might be necessary to >>>>> get some bhyve developers involved (I know almost nothing about it). >>>> NTPD behaves more normally on uniprocessor VMs. >>>> >>>> A FreeBSD bhyve-guest running on a freebsd host will select a >>>> different timecounter depending on whether it is a multiprocessor or a >>>> uniprocessor.ššMy uniprocessor bhyve-vm selected TSC-low as the best >>>> timecounter in a uniprocessor.ššNTP functions there as expected. >>>> >>>> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) >>>> kern.timecounter.hardware: TSC-low >>>> >>>> The very same VM, when given two total CPUs, selected HPET (if I >>>> recall) and the timekeeping with NTPD was unreliable, with many >>>> step-resets to the clock. >>>> >>> Hmm, I just had glance at the code inšsys/amd64/vmm/io/vhpet.c and it >>> looks right. šI wonder if this is just a simple roundoff error in >>> converting between 10.0MHz and SBT units? šIf so, that could be wished >>> away easily by using a power-of-2 frequency for the virtual HPET. šI >>> wonder if the attached patch is all that's needed? >> I've tried the patch (at bhyve guest) and nothing has changed. Should >> the patched system be tested at bhyve guest or bhyve host? >> > > Oh, I'm sorry, I should have mentioned that's for the host side. NP, that's OK. However, the host is busy now, and I'll have an opportunity to test host only tomorrow evening. Ian, thank you for your help! -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Tue Oct 24 21:44:10 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3AA7E2F8C5; Tue, 24 Oct 2017 21:44:10 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward100o.mail.yandex.net (forward100o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::600]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4298872409; Tue, 24 Oct 2017 21:44:09 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback13g.mail.yandex.net (mxback13g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:92]) by forward100o.mail.yandex.net (Yandex) with ESMTP id 5A26E2A20377; Wed, 25 Oct 2017 00:43:57 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback13g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id isWUQvay8k-hvvS5KMY; Wed, 25 Oct 2017 00:43:57 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508881437; bh=RybvAS57SQL0OlWyT48G8jgsB5KnBTAJw8AokO4jpDE=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=fISIJ3lLtL4Wfd8bMcikB/sNYmdoMov0bewoct/quOsWD5tdydtof/JZh99isl4jc 6sW8O5dM2fjY+0XvM0aTM/2/tMOLWL7gRbGf8i/2Ikvg8aEKJYvXgLXUskfY9Xm0Up lr2VsEA0JOKZ4jy+gIYV0didQtLxmDpUiacyJdrY= Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 4fevxQ2hI4-hugGLx6g; Wed, 25 Oct 2017 00:43:56 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508881436; bh=RybvAS57SQL0OlWyT48G8jgsB5KnBTAJw8AokO4jpDE=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=eoCyv7nbmCCPvMMlygvjslnGgL4kWxeV3RsN4Ncjci/EY9eWr5adiXjJAsDnlRSzi 1A43f6wBQ04Vhy9ae1SQbjJ9i/bNCKr8KhWjELaT+bZFRyq8HQygFhsLp7i6Qp/qdX oNpeN3As7Wj0aB6wNGDdhc/8uZ4hL2mSvrito8+Q= Authentication-Results: smtp3p.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore , Michael Voorhis Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> From: Boris Samorodov Message-ID: Date: Wed, 25 Oct 2017 00:43:56 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508624153.1383.107.camel@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2017 21:44:10 -0000 Hi Ian, All! 22.10.2017 01:15, Ian Lepore пишет: > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: >> Ian Lepore writes: >>> >>> Beyond that, I'm not sure what else to try.  It might be necessary to >>> get some bhyve developers involved (I know almost nothing about it). >> NTPD behaves more normally on uniprocessor VMs. >> >> A FreeBSD bhyve-guest running on a freebsd host will select a >> different timecounter depending on whether it is a multiprocessor or a >> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best >> timecounter in a uniprocessor.  NTP functions there as expected. >> >> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) >> kern.timecounter.hardware: TSC-low >> >> The very same VM, when given two total CPUs, selected HPET (if I >> recall) and the timekeeping with NTPD was unreliable, with many >> step-resets to the clock. >> > > Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it > looks right.  I wonder if this is just a simple roundoff error in > converting between 10.0MHz and SBT units?  If so, that could be wished > away easily by using a power-of-2 frequency for the virtual HPET.  I > wonder if the attached patch is all that's needed? I suppose the answer is "yes", the patch helped. Here are two samples (host for bhyve VM without your patch and after patching): --- https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.10000000.txt https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.16777216.txt --- The command was: % for t in `jot 1000`; do ntpq -pn; sleep 64; done The patch made the system to stabilize the process. Ian, thank you! -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Tue Oct 24 22:14:11 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE034E30314 for ; Tue, 24 Oct 2017 22:14:11 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6DF5731D1 for ; Tue, 24 Oct 2017 22:14:11 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: b36b2837-b908-11e7-a938-4f970e858fdb X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id b36b2837-b908-11e7-a938-4f970e858fdb; Tue, 24 Oct 2017 22:14:28 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v9OME8TV002019; Tue, 24 Oct 2017 16:14:08 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1508883248.34364.52.camel@freebsd.org> Subject: Re: host, bhyve vm and ntpd From: Ian Lepore To: Boris Samorodov , Michael Voorhis Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org Date: Tue, 24 Oct 2017 16:14:08 -0600 In-Reply-To: References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> Content-Type: text/plain; charset="koi8-r" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2017 22:14:12 -0000 On Wed, 2017-10-25 at 00:43 +0300, Boris Samorodov wrote: > Hi Ian, All! > > 22.10.2017 01:15, Ian Lepore ÐÉÛÅÔ: > > > > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote: > > > > > > Ian Lepore writes: > > > > > > > > > > > > Beyond that, I'm not sure what else to try. šIt might be necessary to > > > > get some bhyve developers involved (I know almost nothing about it). > > > NTPD behaves more normally on uniprocessor VMs. > > > > > > A FreeBSD bhyve-guest running on a freebsd host will select a > > > different timecounter depending on whether it is a multiprocessor or a > > > uniprocessor.ššMy uniprocessor bhyve-vm selected TSC-low as the best > > > timecounter in a uniprocessor.ššNTP functions there as expected. > > > > > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000) > > > kern.timecounter.hardware: TSC-low > > > > > > The very same VM, when given two total CPUs, selected HPET (if I > > > recall) and the timekeeping with NTPD was unreliable, with many > > > step-resets to the clock. > > > > > Hmm, I just had glance at the code inšsys/amd64/vmm/io/vhpet.c and it > > looks right. šI wonder if this is just a simple roundoff error in > > converting between 10.0MHz and SBT units? šIf so, that could be wished > > away easily by using a power-of-2 frequency for the virtual HPET. šI > > wonder if the attached patch is all that's needed? > I suppose the answer is "yes", the patch helped. Here are two samples > (host for bhyve VM without your patch and after patching): > --- > https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.10000000.txt > https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.16777216.txt > --- > > The command was: > % for t in `jot 1000`; do ntpq -pn; sleep 64; done > > The patch made the system to stabilize the process. > Ian, thank you! > Hmmm. šThe startup behavior wasn't great, it took a long time and several clock steps for it to figure out the frequency error and get the clock under control. šBut, as you say, it did eventually stabilize this time. Can you show the /var/db/ntpd.drift file contents of the host and guest? šIdeally, now that it's stable, the two values should be very close. šIf they're not, maybe this isn't the right fix. -- Ian From owner-freebsd-virtualization@freebsd.org Tue Oct 24 22:21:55 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60767E3074A; Tue, 24 Oct 2017 22:21:55 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward105j.mail.yandex.net (forward105j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11C0873564; Tue, 24 Oct 2017 22:21:55 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback1j.mail.yandex.net (mxback1j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10a]) by forward105j.mail.yandex.net (Yandex) with ESMTP id 191C61869E7; Wed, 25 Oct 2017 01:21:43 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback1j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id GfXTJKe01P-LhqSvngk; Wed, 25 Oct 2017 01:21:43 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508883703; bh=QJNHS16WgoL1aX1yymJAc/L18Md3Wn9m9xobLqFT2N8=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=cUPY+0nZu86bOPPktv3qYM7SDldP5ZgNhOjCsuFoKkTv2sFKIIyB7deItD5x5saJq N3lg5Ay/8j8OC8KD+neeoeh6NIixu94O2lAw7LKXnq7FCB5bK+FWRTuNj1P9k875pH LItR2AMO/jnwwY2RYHUQrin7737pmRvy10gfQ1Do= Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id NllA6FgQJ6-LggCeksW; Wed, 25 Oct 2017 01:21:42 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1508883702; bh=QJNHS16WgoL1aX1yymJAc/L18Md3Wn9m9xobLqFT2N8=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=fEThMalPD1UIglIFDAXhGXE/hkVhSU7vz3yErTZ7P4bW+ZzikHFYHUfevl8BvgXJv r2OYl7SGKEPLS9vmmDH0AJCwdR7UsbcjUdtSCzVvUkZaCUXF9gqeqE3uyDJiZUoyA7 S9jgEeh5odoUrnLxHM7nUqpqc3mL6ZsOoVRNI4lc= Authentication-Results: smtp3p.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: host, bhyve vm and ntpd To: Ian Lepore , Michael Voorhis Cc: freebsd-current@FreeBSD.org, freebsd-virtualization@FreeBSD.org References: <2931f1cc-6574-b58d-4b94-5f77fa5cdb85@passap.ru> <1508512327.1383.55.camel@freebsd.org> <39bf2426-2edf-d485-7c81-519e931154be@passap.ru> <1508517160.1383.63.camel@freebsd.org> <76ff7afb-3d3a-96f6-1275-89472ff5683d@passap.ru> <1508522667.1383.69.camel@freebsd.org> <30992c14-7b78-ab9f-5693-931e6ca41f1b@passap.ru> <1508523696.1383.75.camel@freebsd.org> <23019.46875.929719.481108@atom.mcvau.net> <1508624153.1383.107.camel@freebsd.org> <1508883248.34364.52.camel@freebsd.org> From: Boris Samorodov Message-ID: <9dad7aa4-3adc-8667-655b-e3083e041d50@passap.ru> Date: Wed, 25 Oct 2017 01:21:42 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <1508883248.34364.52.camel@freebsd.org> Content-Type: text/plain; charset=koi8-r Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2017 22:21:55 -0000 25.10.2017 01:14, Ian Lepore ÐÉÛÅÔ: > Can you show the /var/db/ntpd.drift file contents of the host and > guest? šIdeally, now that it's stable, the two values should be very > close. šIf they're not, maybe this isn't the right fix. Sorry, no. :-( I experimented with the host and bhyve vm now has unstable ntp values (stepping). I'll try to revert all my changes and report back in a couple of days. -- WBR, bsam From owner-freebsd-virtualization@freebsd.org Fri Oct 27 14:57:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25A34E476B9 for ; Fri, 27 Oct 2017 14:57:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 139B8688A0 for ; Fri, 27 Oct 2017 14:57:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9REvf0U040676 for ; Fri, 27 Oct 2017 14:57:41 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 222937] [byhve] Severe RAM corruption after PciPassThrough-guest shutdown Date: Fri, 27 Oct 2017 14:57:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 14:57:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222937 --- Comment #2 from commit-hook@freebsd.org --- A commit references this bug: Author: jhb Date: Fri Oct 27 14:57:15 UTC 2017 New revision: 325039 URL: https://svnweb.freebsd.org/changeset/base/325039 Log: Rework pass through changes in r305485 to be safer. Specifically, devices that do not support PCI-e FLR and were not gracefully shutdown by the guest OS could continue to issue DMA requests after the VM was terminated. The changes in r305485 meant that those DMA requests were completed against the host's memory which could result in random memory corruption. Instead, leave ppt devices that are not attached to a VM disabled in the IOMMU and only restore the devices to the host domain if the ppt(4) driver is detached from a device. As an added safety belt, disable busmastering for a pass-through device when before adding it to the host domain during ppt(4) detach. PR: 222937 Tested by: Harry Schmalzbauer Reviewed by: grehan MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D12661 Changes: head/sys/amd64/vmm/io/iommu.c head/sys/amd64/vmm/io/ppt.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Oct 27 15:02:47 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CA56E47AA0 for ; Fri, 27 Oct 2017 15:02:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A3B769313 for ; Fri, 27 Oct 2017 15:02:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9RF2kNm063793 for ; Fri, 27 Oct 2017 15:02:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 222937] [byhve] Severe RAM corruption after PciPassThrough-guest shutdown Date: Fri, 27 Oct 2017 15:02:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jhb@FreeBSD.org X-Bugzilla-Flags: mfc-stable11? X-Bugzilla-Changed-Fields: cc flagtypes.name assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 15:02:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222937 John Baldwin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jhb@FreeBSD.org Flags| |mfc-stable11? Assignee|freebsd-virtualization@Free |jhb@FreeBSD.org |BSD.org | --=20 You are receiving this mail because: You are the assignee for the bug.=