From owner-freebsd-virtualization@freebsd.org Tue Apr 26 14:51:05 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB5ADB1C7CE for ; Tue, 26 Apr 2016 14:51:05 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-pf0-x234.google.com (mail-pf0-x234.google.com [IPv6:2607:f8b0:400e:c00::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9F211D04 for ; Tue, 26 Apr 2016 14:51:05 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by mail-pf0-x234.google.com with SMTP id c189so7757878pfb.3 for ; Tue, 26 Apr 2016 07:51:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-transfer-encoding :message-id:cc:from:subject:date:to; bh=HUsdCvya0fD6ZZ7RhWqmjBEAI3Y7gOWW9JKqkrg5esc=; b=cSovj48Is81GyT6z93C/5bYB6CaltjKMJLxubGhUH03VcLSHqEJckOPGuKijlsIqQ8 mBbXSqTSNW5KeAMn9ovuBbm9FD6CR7dUhOr9ncoFzAXPZLg/wrdtqV3cZ5gMNTDsXBWJ F8rkMagobB2pQFlYTFX00wFs1OwdaSAEfgA7+heSyu8iZd3vgG/Z0BrhXNi0UI9aG9kZ QE7ydaP+2IjjyhWpWMUrO1I31p0UoIeJVuZ/iOiXJdKF3vdhUibrUn7JyjdRC8E/eR3I SqBVhZ5RRLvAGbFddetpxW6kFSwT8RRd7B10dQ4m3Z0rXjg2tsBp+h0SOi4sU2VvE0NP zGLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=HUsdCvya0fD6ZZ7RhWqmjBEAI3Y7gOWW9JKqkrg5esc=; b=dE3GG7Sq0zqg17oqgeOSZi1Tdcnur6WUQSqfhyJ7gMgr4oBPC/nyL4dmFxebNWroc8 B6b8hstkHsfdut8id60zkAmv5ItdtwnZrHdJEa1y2dcbN9oXEGWJiEFuFQmNPoMNc3Zi 60K56yjBvPRul8gSytXuWL1zFFhkaVBdmrgT7fg7KO9ZL5nTH1HLRy5uRpVdJ5hdaTPT hma+/UjmafDmWDuUO9UWXyO5Dho9/dlwTVua8f4XF3TJXiqYeBT/CI8cDsyLk1kAIJ5F fzeNv2+g7grUrK1FDFJHPxhdi1ZlxYJ8zcTOJKRslt0J+z6XXTbFsjXmMsntdfENHH0d nvyg== X-Gm-Message-State: AOPr4FXeV8Mvef/WocGqmX569+Ph90xkCdcQcPl1EJslsORD5AbKjh6IoyuVXGFeKDPNcQ== X-Received: by 10.98.49.134 with SMTP id x128mr4157184pfx.45.1461682265211; Tue, 26 Apr 2016 07:51:05 -0700 (PDT) Received: from ?IPv6:2602:306:ce40:d9e0:202a:8fdf:ec12:4557? ([2602:306:ce40:d9e0:202a:8fdf:ec12:4557]) by smtp.gmail.com with ESMTPSA id h88sm5906287pfd.10.2016.04.26.07.51.03 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Apr 2016 07:51:03 -0700 (PDT) References: <8EE098BA-6C8F-41F4-B9CD-88F3D587B10F@gmail.com> <571F7127.1050100@schema31.it> Mime-Version: 1.0 (1.0) In-Reply-To: <571F7127.1050100@schema31.it> Message-Id: <647BCA7C-B915-4FAE-870F-CA914B38B8B5@gmail.com> Cc: "abrancatelli@schema31.it" , "freebsd-virtualization@freebsd.org" X-Mailer: iPhone Mail (11D257) From: Neel Natu Subject: Re: bhyve clock issue Date: Tue, 26 Apr 2016 07:50:59 -0700 To: Ilario Febi Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2016 14:51:06 -0000 What was the timecounter being used when you were seeing the time skew? Best Neel > On Apr 26, 2016, at 6:46 AM, Ilario Febi wrote: >=20 > Yes,=20 >=20 > I found errors in /var/log/messages and fix adding:=20 >=20 > kern.timecounter.hardware=3DTSC-low > into /etc/sysctl.conf file >=20 > Thanks for support. >=20 >=20 >=20 > Il 14/04/16 18:13, Neel Natu ha scritto: >> Hi, >>=20 >> Did you see anything interesting in the ntpd log file around the time the= large skew happened? >>=20 >> Best >> Neel >>=20 >>> On Apr 13, 2016, at 3:44 AM, Ilario Febi wrote: >>>=20 >>> Hi All >>>=20 >>> I have a strange issue about the clock of some FreeBSD Bhyve guests, >>> maybe someone of you can help me to diagnose it and work on a >>> solution. >>>=20 >>> In detail, sometimes (without an apparent regularity) one of four >>> bhyve guest's clock goes exactly 7 minutes and 10 seconds in the >>> future. >>>=20 >>> This clock stays unaligned for about 20 minutes, then, without any >>> human intervention, the clock realign to proper condition. >>>=20 >>> My configuration is: >>>=20 >>> envrm02 (bhyve HOST, 10.2-RELEASE-p14) >>> -> og-qa (bhyve guest - 10.2-RELEASE-p14) >>> -> og-ci (bhyve guest - 10.2-RELEASE-p14) >>> -> og-test (bhyve guest - 10.2-RELEASE-p14) >>> -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14) >>> All the og-* machines just have a running copy of tomcat. >>>=20 >>> I built a simple test to catch the error with python/fabric... the test i= s: >>>=20 >>> fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}' >>>=20 >>> which simply run the command "date" on all hosts. >>>=20 >>> So, the result is: >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 17:38:00 CEST 2016 [envrm02] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-qa] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-ci] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-test] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out >>>=20 >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:38:00 CEST 2016 [envrm02] out >>> Thu Apr 7 18:45:10 CEST 2016 [og-qa] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-ci] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-test] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out >>>=20 >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:41:05 CEST 2016 [envrm02] out >>> Thu Apr 7 18:48:15 CEST 2016 [og-qa] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-ci] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-test] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out >>>=20 >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:46:51 CEST 2016 [envrm02] out >>> Thu Apr 7 18:46:52 CEST 2016 [og-qa] out >>> Thu Apr 7 18:46:52 CEST 2016 [og-ci] out >>> Thu Apr 7 18:46:53 CEST 2016 [og-test] out >>> Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out >>>=20 >>> The configuration is the same for all hosts and is this: >>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd >>> [envrm02] out: ntpd_enable=3D"YES" >>> [og-qa] out: ntpd_enable=3D"YES" >>> [og-qa] out: ntpd_sync_on_start=3D"YES" >>> [og-ci] out: ntpd_enable=3D"YES" >>> [og-ci] out: ntpd_sync_on_start=3D"YES" >>> [og-test] out: ntpd_enable=3D"YES" >>> [og-test] out: ntpd_sync_on_start=3D"YES" >>> [og-prod-rm] out: ntpd_enable=3D"YES" >>> [og-prod-rm] out: ntpd_sync_on_start=3D"YES" >>>=20 >>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v '= out: #' >>> [envrm02] Executing task '' >>> [envrm02] run: cat /etc/ntp.conf >>> [envrm02] out: >>> [envrm02] out: server 0.freebsd.pool.ntp.org iburst >>> [envrm02] out: server 1.freebsd.pool.ntp.org iburst >>> [envrm02] out: server 2.freebsd.pool.ntp.org iburst >>> [envrm02] out: >>> [envrm02] out: restrict default limited kod nomodify notrap nopeer noque= ry >>> [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer no= query >>> [envrm02] out: restrict 127.0.0.1 >>> [envrm02] out: restrict -6 ::1 >>> [envrm02] out: restrict 127.127.1.0 >>>=20 >>> [og-qa] Executing task '' >>> [og-qa] run: cat /etc/ntp.conf >>> [og-qa] out: >>> [og-qa] out: server 0.freebsd.pool.ntp.org iburst >>> [og-qa] out: server 1.freebsd.pool.ntp.org iburst >>> [og-qa] out: server 2.freebsd.pool.ntp.org iburst >>> [og-qa] out: >>> [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery= >>> [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noqu= ery >>> [og-qa] out: restrict 127.0.0.1 >>> [og-qa] out: restrict -6 ::1 >>> [og-qa] out: restrict 127.127.1.0 >>>=20 >>> [og-ci] Executing task '' >>> [og-ci] run: cat /etc/ntp.conf >>> [og-ci] out: >>> [og-ci] out: server 0.freebsd.pool.ntp.org iburst >>> [og-ci] out: server 1.freebsd.pool.ntp.org iburst >>> [og-ci] out: server 2.freebsd.pool.ntp.org iburst >>> [og-ci] out: >>> [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery= >>> [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noqu= ery >>> [og-ci] out: restrict 127.0.0.1 >>> [og-ci] out: restrict -6 ::1 >>> [og-ci] out: restrict 127.127.1.0 >>>=20 >>> [og-test] Executing task '' >>> [og-test] run: cat /etc/ntp.conf >>> [og-test] out: >>> [og-test] out: server 0.freebsd.pool.ntp.org iburst >>> [og-test] out: server 1.freebsd.pool.ntp.org iburst >>> [og-test] out: server 2.freebsd.pool.ntp.org iburst >>> [og-test] out: >>> [og-test] out: restrict default limited kod nomodify notrap nopeer noque= ry >>> [og-test] out: restrict -6 default limited kod nomodify notrap nopeer no= query >>> [og-test] out: restrict 127.0.0.1 >>> [og-test] out: restrict -6 ::1 >>> [og-test] out: restrict 127.127.1.0 >>>=20 >>> Some observation: >>> * The clock of envrm02 (the HOST) always seems to be correct. >>> * The issue happens on a (apparently) random guest (not always the same o= ne). >>> * The issue happens always on one host at a time. >>>=20 >>> The bhyve command line is, roughly, this one: >>>=20 >>> /usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $= machine >>> /usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s >>> 2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap >>> -s 31,lpc -l com1,stdio $machine >>>=20 >>> Thanks in advance. >>> _______________________________________________ >>> freebsd-virtualization@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre= ebsd.org" >=20 > --=20 > Ing. Ilario Febi >=20 > Schema 31 S.p.a. Socio Unico > System Administrator >=20 > BO - FI - ROMA - PA=20 > ITALY > Tel: +39. 06.98.358.472 > Cell: +39. 366.69.45.384 > Fax: +39. 055.71.880.466 > Societ=C3=A0 del Gruppo SC31 ITALIA