Date: Mon, 11 Apr 2016 21:35:50 +0200 From: Andrea Brancatelli <abrancatelli@schema31.it> To: freebsd-virtualization@freebsd.org Cc: Ilario Febi <ifebi@schema31.it> Subject: bhyve clock jitter Message-ID: <3d1e9bbda3b402e8b185fd27cc4dceee@schema31.it>
next in thread | raw e-mail | index | archive | help
Hi ...
I have a strange issue about the clock of some FreeBSD Bhyve guests,
maybe someone of you can help me to diagnose it and work on a solution.
In detail, sometimes (without an apparent regularity) one of four bhyve
guest's clock goes exactly 7 minutes and 10 seconds in the future.
This clock stays unaligned for about 20 minutes, then, without any human
intervention, the clock realign to proper condition.
My configuration is:
envrm02 (bhyve HOST, 10.2-RELEASE-p14)
-> og-qa (bhyve guest - 10.2-RELEASE-p14)
-> og-ci (bhyve guest - 10.2-RELEASE-p14)
-> og-test (bhyve guest - 10.2-RELEASE-p14)
-> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)
All the og-* machines just have a running copy of tomcat.
I built a simple test to catch the error with python/fabric... the test
is:
fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'
which simply run the command "date" on all hosts.
So, the result is:
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 17:38:00 CEST 2016 [envrm02] out
Thu Apr 7 17:38:00 CEST 2016 [og-qa] out
Thu Apr 7 17:38:00 CEST 2016 [og-ci] out
Thu Apr 7 17:38:00 CEST 2016 [og-test] out
Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:38:00 CEST 2016 [envrm02] out
Thu Apr 7 18:45:10 CEST 2016 [og-qa] out
Thu Apr 7 18:38:00 CEST 2016 [og-ci] out
Thu Apr 7 18:38:00 CEST 2016 [og-test] out
Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:41:05 CEST 2016 [envrm02] out
Thu Apr 7 18:48:15 CEST 2016 [og-qa] out
Thu Apr 7 18:41:05 CEST 2016 [og-ci] out
Thu Apr 7 18:41:05 CEST 2016 [og-test] out
Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:46:51 CEST 2016 [envrm02] out
Thu Apr 7 18:46:52 CEST 2016 [og-qa] out
Thu Apr 7 18:46:52 CEST 2016 [og-ci] out
Thu Apr 7 18:46:53 CEST 2016 [og-test] out
Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out
The configuration is the same for all hosts and is this:
@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
[envrm02] out: ntpd_enable="YES"
[og-qa] out: ntpd_enable="YES"
[og-qa] out: ntpd_sync_on_start="YES"
[og-ci] out: ntpd_enable="YES"
[og-ci] out: ntpd_sync_on_start="YES"
[og-test] out: ntpd_enable="YES"
[og-test] out: ntpd_sync_on_start="YES"
[og-prod-rm] out: ntpd_enable="YES"
[og-prod-rm] out: ntpd_sync_on_start="YES"
@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v
'out: #'
[envrm02] Executing task '<remainder>'
[envrm02] run: cat /etc/ntp.conf
[envrm02] out:
[envrm02] out: server 0.freebsd.pool.ntp.org iburst
[envrm02] out: server 1.freebsd.pool.ntp.org iburst
[envrm02] out: server 2.freebsd.pool.ntp.org iburst
[envrm02] out:
[envrm02] out: restrict default limited kod nomodify notrap nopeer
noquery
[envrm02] out: restrict -6 default limited kod nomodify notrap nopeer
noquery
[envrm02] out: restrict 127.0.0.1
[envrm02] out: restrict -6 ::1
[envrm02] out: restrict 127.127.1.0
[og-qa] Executing task '<remainder>'
[og-qa] run: cat /etc/ntp.conf
[og-qa] out:
[og-qa] out: server 0.freebsd.pool.ntp.org iburst
[og-qa] out: server 1.freebsd.pool.ntp.org iburst
[og-qa] out: server 2.freebsd.pool.ntp.org iburst
[og-qa] out:
[og-qa] out: restrict default limited kod nomodify notrap nopeer noquery
[og-qa] out: restrict -6 default limited kod nomodify notrap nopeer
noquery
[og-qa] out: restrict 127.0.0.1
[og-qa] out: restrict -6 ::1
[og-qa] out: restrict 127.127.1.0
[og-ci] Executing task '<remainder>'
[og-ci] run: cat /etc/ntp.conf
[og-ci] out:
[og-ci] out: server 0.freebsd.pool.ntp.org iburst
[og-ci] out: server 1.freebsd.pool.ntp.org iburst
[og-ci] out: server 2.freebsd.pool.ntp.org iburst
[og-ci] out:
[og-ci] out: restrict default limited kod nomodify notrap nopeer noquery
[og-ci] out: restrict -6 default limited kod nomodify notrap nopeer
noquery
[og-ci] out: restrict 127.0.0.1
[og-ci] out: restrict -6 ::1
[og-ci] out: restrict 127.127.1.0
[og-test] Executing task '<remainder>'
[og-test] run: cat /etc/ntp.conf
[og-test] out:
[og-test] out: server 0.freebsd.pool.ntp.org iburst
[og-test] out: server 1.freebsd.pool.ntp.org iburst
[og-test] out: server 2.freebsd.pool.ntp.org iburst
[og-test] out:
[og-test] out: restrict default limited kod nomodify notrap nopeer
noquery
[og-test] out: restrict -6 default limited kod nomodify notrap nopeer
noquery
[og-test] out: restrict 127.0.0.1
[og-test] out: restrict -6 ::1
[og-test] out: restrict 127.127.1.0
Some observation:
* The clock of envrm02 (the HOST) always seems to be correct.
* The issue happens on a (apparently) random guest (not always the same
one).
* The issue happens always on one host at a time.
The bhyve command line is, roughly, this one:
/usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img
$machine
/usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s
2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap
-s 31,lpc -l com1,stdio $machine
Thanks in advance.
--
Andrea Brancatelli
Schema31 S.p.a.
Responsabile IT
ROMA - BO - FI - PA
ITALY
Tel: +39.06.98.358.472
Cell: +39.331.2488468
Fax: +39.055.71.880.466
Società del Gruppo SC31 ITALIA
From owner-freebsd-virtualization@freebsd.org Thu Apr 14 06:03:35 2016
Return-Path: <owner-freebsd-virtualization@freebsd.org>
Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F7ECB10E89
for <freebsd-virtualization@mailman.ysv.freebsd.org>;
Thu, 14 Apr 2016 06:03:35 +0000 (UTC)
(envelope-from abrancatelli@schema31.it)
Received: from stricnina.roma.schema31.it (stricnina.schema31.it
[2.228.74.189])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "stricnina.roma.schema31.it",
Issuer "stricnina.roma.schema31.it" (not verified))
by mx1.freebsd.org (Postfix) with ESMTPS id 285A619CD
for <freebsd-virtualization@freebsd.org>; Thu, 14 Apr 2016 06:03:33 +0000 (UTC)
(envelope-from abrancatelli@schema31.it)
Received: from smtp.schema31.it (localhost [127.0.0.1])
by stricnina.roma.schema31.it (8.14.9/8.14.9) with ESMTP id u3DAa6aA022418;
Wed, 13 Apr 2016 12:36:06 +0200 (CEST)
(envelope-from abrancatelli@schema31.it)
Subject: Fwd: bhyve clock jitter
References: <3d1e9bbda3b402e8b185fd27cc4dceee@schema31.it>
From: Andrea Brancatelli <abrancatelli@schema31.it>
Mime-Version: 1.0 (1.0)
X-Mailer: iPad Mail (13E238)
Message-Id: <0D35099D-C959-4713-999D-2E5F2AD3A7C9@schema31.it>
Date: Wed, 13 Apr 2016 12:36:01 +0200
Cc: Ilario Febi <ifebi@schema31.it>
To: freebsd-virtualization@freebsd.org
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: freebsd-virtualization@freebsd.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: "Discussion of various virtualization techniques FreeBSD supports."
<freebsd-virtualization.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-virtualization>,
<mailto:freebsd-virtualization-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-virtualization/>;
List-Post: <mailto:freebsd-virtualization@freebsd.org>
List-Help: <mailto:freebsd-virtualization-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>,
<mailto:freebsd-virtualization-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2016 06:03:35 -0000
Hi ...
I have a strange issue about the clock of some FreeBSD Bhyve guests, maybe s=
omeone of you can help me to diagnose it and work on a solution.
In detail, sometimes (without an apparent regularity) one of four bhyve gues=
t's clock goes exactly 7 minutes and 10 seconds in the future.
=20
This clock stays unaligned for about 20 minutes, then, without any human int=
ervention, the clock realign to proper condition.
My configuration is:
envrm02 (bhyve HOST, 10.2-RELEASE-p14)
-> og-qa (bhyve guest - 10.2-RELEASE-p14)
-> og-ci (bhyve guest - 10.2-RELEASE-p14)
-> og-test (bhyve guest - 10.2-RELEASE-p14)
-> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)
All the og-* machines just have a running copy of tomcat.
I built a simple test to catch the error with python/fabric... the test is:
fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'
which simply run the command "date" on all hosts.
So, the result is:
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 17:38:00 CEST 2016 [envrm02] out
Thu Apr 7 17:38:00 CEST 2016 [og-qa] out
Thu Apr 7 17:38:00 CEST 2016 [og-ci] out
Thu Apr 7 17:38:00 CEST 2016 [og-test] out
Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:38:00 CEST 2016 [envrm02] out
Thu Apr 7 18:45:10 CEST 2016 [og-qa] out
Thu Apr 7 18:38:00 CEST 2016 [og-ci] out
Thu Apr 7 18:38:00 CEST 2016 [og-test] out
Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:41:05 CEST 2016 [envrm02] out
Thu Apr 7 18:48:15 CEST 2016 [og-qa] out
Thu Apr 7 18:41:05 CEST 2016 [og-ci] out
Thu Apr 7 18:41:05 CEST 2016 [og-test] out
Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out
@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr 7 18:46:51 CEST 2016 [envrm02] out
Thu Apr 7 18:46:52 CEST 2016 [og-qa] out
Thu Apr 7 18:46:52 CEST 2016 [og-ci] out
Thu Apr 7 18:46:53 CEST 2016 [og-test] out
Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out
The configuration is the same for all hosts and is this:
@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
[envrm02] out: ntpd_enable=3D"YES"
[og-qa] out: ntpd_enable=3D"YES"
[og-qa] out: ntpd_sync_on_start=3D"YES"
[og-ci] out: ntpd_enable=3D"YES"
[og-ci] out: ntpd_sync_on_start=3D"YES"
[og-test] out: ntpd_enable=3D"YES"
[og-test] out: ntpd_sync_on_start=3D"YES"
[og-prod-rm] out: ntpd_enable=3D"YES"
[og-prod-rm] out: ntpd_sync_on_start=3D"YES"
@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'out:=
#'
[envrm02] Executing task '<remainder>'
[envrm02] run: cat /etc/ntp.conf
[envrm02] out:
[envrm02] out: server 0.freebsd.pool.ntp.org iburst
[envrm02] out: server 1.freebsd.pool.ntp.org iburst
[envrm02] out: server 2.freebsd.pool.ntp.org iburst
[envrm02] out:
[envrm02] out: restrict default limited kod nomodify notrap nopeer noquery
[envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noquer=
y
[envrm02] out: restrict 127.0.0.1
[envrm02] out: restrict -6 ::1
[envrm02] out: restrict 127.127.1.0
[og-qa] Executing task '<remainder>'
[og-qa] run: cat /etc/ntp.conf
[og-qa] out:
[og-qa] out: server 0.freebsd.pool.ntp.org iburst
[og-qa] out: server 1.freebsd.pool.ntp.org iburst
[og-qa] out: server 2.freebsd.pool.ntp.org iburst
[og-qa] out:
[og-qa] out: restrict default limited kod nomodify notrap nopeer noquery
[og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[og-qa] out: restrict 127.0.0.1
[og-qa] out: restrict -6 ::1
[og-qa] out: restrict 127.127.1.0
[og-ci] Executing task '<remainder>'
[og-ci] run: cat /etc/ntp.conf
[og-ci] out:
[og-ci] out: server 0.freebsd.pool.ntp.org iburst
[og-ci] out: server 1.freebsd.pool.ntp.org iburst
[og-ci] out: server 2.freebsd.pool.ntp.org iburst
[og-ci] out:
[og-ci] out: restrict default limited kod nomodify notrap nopeer noquery
[og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[og-ci] out: restrict 127.0.0.1
[og-ci] out: restrict -6 ::1
[og-ci] out: restrict 127.127.1.0
[og-test] Executing task '<remainder>'
[og-test] run: cat /etc/ntp.conf
[og-test] out:
[og-test] out: server 0.freebsd.pool.ntp.org iburst
[og-test] out: server 1.freebsd.pool.ntp.org iburst
[og-test] out: server 2.freebsd.pool.ntp.org iburst
[og-test] out:
[og-test] out: restrict default limited kod nomodify notrap nopeer noquery
[og-test] out: restrict -6 default limited kod nomodify notrap nopeer noquer=
y
[og-test] out: restrict 127.0.0.1
[og-test] out: restrict -6 ::1
[og-test] out: restrict 127.127.1.0
Some observation:=20
* The clock of envrm02 (the HOST) always seems to be correct.
* The issue happens on a (apparently) random guest (not always the same one)=
.
* The issue happens always on one host at a time.
The bhyve command line is, roughly, this one:
/usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $mach=
ine
/usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s 2,ahci-hd,/san_stor=
age/VMfs/$machine/$machine.img -s 4,virtio-net,$tap -s 31,lpc -l com1,stdio $=
machine
Thanks in advance.
=20
--=20
Andrea Brancatelli
Schema31 S.p.a.
Responsabile IT
ROMA - BO - FI - PA=20
ITALY
Tel: +39.06.98.358.472
Cell: +39.331.2488468
Fax: +39.055.71.880.466
Societ=C3=A0 del Gruppo SC31 ITALIA
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3d1e9bbda3b402e8b185fd27cc4dceee>