From owner-freebsd-virtualization@freebsd.org Thu Apr 14 16:13:32 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8B5DB0F10A for ; Thu, 14 Apr 2016 16:13:32 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-pa0-x230.google.com (mail-pa0-x230.google.com [IPv6:2607:f8b0:400e:c03::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AFF4214B8 for ; Thu, 14 Apr 2016 16:13:32 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by mail-pa0-x230.google.com with SMTP id ot11so46178376pab.1 for ; Thu, 14 Apr 2016 09:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-transfer-encoding :message-id:cc:from:subject:date:to; bh=Key+BaEmrtqraOGdwYv0ozDPOvUjYw+isc272eCat4Y=; b=qvQqafE52FZC1R2U0nQ9L+zJYxW5lLuQRxR4nQdFFNxWb/h5sJq3FH00F/qn3xYvi+ Km8niSeB5xGBx+pvw4SfS8rAJ2GRgyQSWFUl/OI/O7MkxE+2+BleuGuqXD/9Fl2OZH6r B+kcuvLSkGD/pfM7FHA4zSOPdSr7c3zbanKEB8EXCtBV6y1CoCZbnUp1LudsU3SBIyl5 hk9y6xwl284TRYbtVBwJZVYY6rIHyI+KZlP6KcAHBGGQV2U11zu/eEWts6cWzD1EQ4L4 C28CeUzn7jC+siY/rF4XVL+YZ9xCBQFAFsY9US/I3p+INXb+B6aPFvcRepdYRyCc89d0 24JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=Key+BaEmrtqraOGdwYv0ozDPOvUjYw+isc272eCat4Y=; b=HOKO+4putVuGyt/ad6g6FyekdYcLH+PYTN9GH2TWsAvjLlo9T1aRDJ1hPWHDlow2Dx vg9X1AsKZ74FQ7ikOOjaqjJ8ZpfxHIpHorXbNvEP2ZseSOxJKgaLgI49SUoxplrI03DW hhOl0ayuWltfFjv5Z9eag/jgIShyHFxBTVLnWmt9E7rDLwysVbVqz/adFrxy9qT7WSRI /02iTzfUnysps2gDLzKJh5USOScSy1EKwLSa4wHTKyPlGUKEgYqy1Erfzud9YNtLwcKG qo1Ag0kWFObgk+AsAkJPavWxpzUXaepNGTfPqcD1/hN8kw5Q/fp7oNt7+dp08erfq5ny 90kA== X-Gm-Message-State: AOPr4FWfW+Bhk5fQdUb6304AHzqwt+xTRbG2Rm7J27TCFZ+RwHZevBvvV8qPSL/xk6sKJg== X-Received: by 10.66.66.198 with SMTP id h6mr22102603pat.112.1460650412066; Thu, 14 Apr 2016 09:13:32 -0700 (PDT) Received: from ?IPv6:2602:306:ce40:d9e0:9559:faaa:281a:19e7? ([2602:306:ce40:d9e0:9559:faaa:281a:19e7]) by smtp.gmail.com with ESMTPSA id g28sm58940548pfd.25.2016.04.14.09.13.30 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 14 Apr 2016 09:13:30 -0700 (PDT) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <8EE098BA-6C8F-41F4-B9CD-88F3D587B10F@gmail.com> Cc: "freebsd-virtualization@freebsd.org" X-Mailer: iPhone Mail (11D257) From: Neel Natu Subject: Re: bhyve clock issue Date: Thu, 14 Apr 2016 09:13:26 -0700 To: Ilario Febi X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Apr 2016 16:13:33 -0000 Hi, Did you see anything interesting in the ntpd log file around the time the la= rge skew happened? Best Neel > On Apr 13, 2016, at 3:44 AM, Ilario Febi wrote: >=20 > Hi All >=20 > I have a strange issue about the clock of some FreeBSD Bhyve guests, > maybe someone of you can help me to diagnose it and work on a > solution. >=20 > In detail, sometimes (without an apparent regularity) one of four > bhyve guest's clock goes exactly 7 minutes and 10 seconds in the > future. >=20 > This clock stays unaligned for about 20 minutes, then, without any > human intervention, the clock realign to proper condition. >=20 > My configuration is: >=20 > envrm02 (bhyve HOST, 10.2-RELEASE-p14) > -> og-qa (bhyve guest - 10.2-RELEASE-p14) > -> og-ci (bhyve guest - 10.2-RELEASE-p14) > -> og-test (bhyve guest - 10.2-RELEASE-p14) > -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14) > All the og-* machines just have a running copy of tomcat. >=20 > I built a simple test to catch the error with python/fabric... the test is= : >=20 > fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}' >=20 > which simply run the command "date" on all hosts. >=20 > So, the result is: > @Mac-mini ~/p/f/> ./check_clock.sh > Thu Apr 7 17:38:00 CEST 2016 [envrm02] out > Thu Apr 7 17:38:00 CEST 2016 [og-qa] out > Thu Apr 7 17:38:00 CEST 2016 [og-ci] out > Thu Apr 7 17:38:00 CEST 2016 [og-test] out > Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out >=20 > @Mac-mini ~/p/f/> ./check_clock.sh > Thu Apr 7 18:38:00 CEST 2016 [envrm02] out > Thu Apr 7 18:45:10 CEST 2016 [og-qa] out > Thu Apr 7 18:38:00 CEST 2016 [og-ci] out > Thu Apr 7 18:38:00 CEST 2016 [og-test] out > Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out >=20 > @Mac-mini ~/p/f/> ./check_clock.sh > Thu Apr 7 18:41:05 CEST 2016 [envrm02] out > Thu Apr 7 18:48:15 CEST 2016 [og-qa] out > Thu Apr 7 18:41:05 CEST 2016 [og-ci] out > Thu Apr 7 18:41:05 CEST 2016 [og-test] out > Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out >=20 > @Mac-mini ~/p/f/> ./check_clock.sh > Thu Apr 7 18:46:51 CEST 2016 [envrm02] out > Thu Apr 7 18:46:52 CEST 2016 [og-qa] out > Thu Apr 7 18:46:52 CEST 2016 [og-ci] out > Thu Apr 7 18:46:53 CEST 2016 [og-test] out > Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out >=20 > The configuration is the same for all hosts and is this: > @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd > [envrm02] out: ntpd_enable=3D"YES" > [og-qa] out: ntpd_enable=3D"YES" > [og-qa] out: ntpd_sync_on_start=3D"YES" > [og-ci] out: ntpd_enable=3D"YES" > [og-ci] out: ntpd_sync_on_start=3D"YES" > [og-test] out: ntpd_enable=3D"YES" > [og-test] out: ntpd_sync_on_start=3D"YES" > [og-prod-rm] out: ntpd_enable=3D"YES" > [og-prod-rm] out: ntpd_sync_on_start=3D"YES" >=20 > @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'ou= t: #' > [envrm02] Executing task '' > [envrm02] run: cat /etc/ntp.conf > [envrm02] out: > [envrm02] out: server 0.freebsd.pool.ntp.org iburst > [envrm02] out: server 1.freebsd.pool.ntp.org iburst > [envrm02] out: server 2.freebsd.pool.ntp.org iburst > [envrm02] out: > [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery= > [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noqu= ery > [envrm02] out: restrict 127.0.0.1 > [envrm02] out: restrict -6 ::1 > [envrm02] out: restrict 127.127.1.0 >=20 > [og-qa] Executing task '' > [og-qa] run: cat /etc/ntp.conf > [og-qa] out: > [og-qa] out: server 0.freebsd.pool.ntp.org iburst > [og-qa] out: server 1.freebsd.pool.ntp.org iburst > [og-qa] out: server 2.freebsd.pool.ntp.org iburst > [og-qa] out: > [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery > [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquer= y > [og-qa] out: restrict 127.0.0.1 > [og-qa] out: restrict -6 ::1 > [og-qa] out: restrict 127.127.1.0 >=20 > [og-ci] Executing task '' > [og-ci] run: cat /etc/ntp.conf > [og-ci] out: > [og-ci] out: server 0.freebsd.pool.ntp.org iburst > [og-ci] out: server 1.freebsd.pool.ntp.org iburst > [og-ci] out: server 2.freebsd.pool.ntp.org iburst > [og-ci] out: > [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery > [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquer= y > [og-ci] out: restrict 127.0.0.1 > [og-ci] out: restrict -6 ::1 > [og-ci] out: restrict 127.127.1.0 >=20 > [og-test] Executing task '' > [og-test] run: cat /etc/ntp.conf > [og-test] out: > [og-test] out: server 0.freebsd.pool.ntp.org iburst > [og-test] out: server 1.freebsd.pool.ntp.org iburst > [og-test] out: server 2.freebsd.pool.ntp.org iburst > [og-test] out: > [og-test] out: restrict default limited kod nomodify notrap nopeer noquery= > [og-test] out: restrict -6 default limited kod nomodify notrap nopeer noqu= ery > [og-test] out: restrict 127.0.0.1 > [og-test] out: restrict -6 ::1 > [og-test] out: restrict 127.127.1.0 >=20 > Some observation: > * The clock of envrm02 (the HOST) always seems to be correct. > * The issue happens on a (apparently) random guest (not always the same on= e). > * The issue happens always on one host at a time. >=20 > The bhyve command line is, roughly, this one: >=20 > /usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $ma= chine > /usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s > 2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap > -s 31,lpc -l com1,stdio $machine >=20 > Thanks in advance. > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freeb= sd.org"