From owner-freebsd-virtualization@freebsd.org Wed Apr 13 10:44:23 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 629B7B0F478 for ; Wed, 13 Apr 2016 10:44:23 +0000 (UTC) (envelope-from ifebi@schema31.it) Received: from mail-io0-x243.google.com (mail-io0-x243.google.com [IPv6:2607:f8b0:4001:c06::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 38A641A6A for ; Wed, 13 Apr 2016 10:44:22 +0000 (UTC) (envelope-from ifebi@schema31.it) Received: by mail-io0-x243.google.com with SMTP id u185so6178179iod.2 for ; Wed, 13 Apr 2016 03:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=schema31-it.20150623.gappssmtp.com; s=20150623; h=mime-version:date:message-id:subject:from:to; bh=f0n5qr1UEVJj3CcPhGv3IvaPYVxrhuOBvq8V4aPYw9Q=; b=HNJIgyojqrlt3UPF8Qe5R/F5pRe81azfC8qcgw/Cp3sD+v/PmyP2IZBX0pCJeZlMvQ mJ0aExS8Q5GxvcWebMyqaHpxSc5bExT94qx+8sahudgan4MyptUpBbuLIwc8QXnZ+/YD chA+rhBrWr2CnFtgS7lPPAkW7XlYN66d2NHgpm/ziPeQWIQR4FyPqXvd1KL2DUWvE1eG t1ePYRTyxGIOVUSP1HOxQatVq8MkiINhOf4H+hgxnPBj357AdP7iGkx0frRp9Hap8Xfp eR2HAWrGhscIwoOf/h9s2J3nKkscDhU21cCP/whwvne5JxowChkeF5bdPtd00v3WlV+x KBBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=f0n5qr1UEVJj3CcPhGv3IvaPYVxrhuOBvq8V4aPYw9Q=; b=AM0PkMAC+QW25Q63WkqzoWl9u87It3Yowwpd87RxdxksBFL+1h3tM3t2+nDj5V0uTo GiIRvkW7WMsgcUdoL18JV+BqnT+TFbNZonFBb/XKwULFEcxfkY+tt5ClBfnFN+ZuWuo9 jqOnuXO+q6m0V9NEG7ZByf2bjLDEbf8I3NNgnTP6Cmsg1GcD7EfaYow0uXOLF8sVGrB7 2cbEaM7apxvdpCkGuOyA7JQ7ezCcuVQ7lug3r9EE/wejjQiJE3PFXyzn4PXseT1CnTgP U4uZv8Z6590V7xgZAtCTOg9dYQZjZ/fdYXTDDycxM7rNpu89Ofutx7dzjSYPZhzYSrF4 0U/Q== X-Gm-Message-State: AOPr4FVRsb4DDyywhJmN4VA4PCW6zsHMkIUBDOAj7T6p1nfn5Y0vX1S7yW8RxaxV/KCUPn3MrS2MZEOvQhnZ/A== MIME-Version: 1.0 X-Received: by 10.107.26.201 with SMTP id a192mr9548411ioa.13.1460544262200; Wed, 13 Apr 2016 03:44:22 -0700 (PDT) Received: by 10.36.127.132 with HTTP; Wed, 13 Apr 2016 03:44:22 -0700 (PDT) X-Originating-IP: [2001:470:28:12b::7] Date: Wed, 13 Apr 2016 12:44:22 +0200 Message-ID: Subject: bhyve clock issue From: Ilario Febi To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 10:44:23 -0000 Hi All I have a strange issue about the clock of some FreeBSD Bhyve guests, maybe someone of you can help me to diagnose it and work on a solution. In detail, sometimes (without an apparent regularity) one of four bhyve guest's clock goes exactly 7 minutes and 10 seconds in the future. This clock stays unaligned for about 20 minutes, then, without any human intervention, the clock realign to proper condition. My configuration is: envrm02 (bhyve HOST, 10.2-RELEASE-p14) -> og-qa (bhyve guest - 10.2-RELEASE-p14) -> og-ci (bhyve guest - 10.2-RELEASE-p14) -> og-test (bhyve guest - 10.2-RELEASE-p14) -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14) All the og-* machines just have a running copy of tomcat. I built a simple test to catch the error with python/fabric... the test is: fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}' which simply run the command "date" on all hosts. So, the result is: @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 17:38:00 CEST 2016 [envrm02] out Thu Apr 7 17:38:00 CEST 2016 [og-qa] out Thu Apr 7 17:38:00 CEST 2016 [og-ci] out Thu Apr 7 17:38:00 CEST 2016 [og-test] out Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:38:00 CEST 2016 [envrm02] out Thu Apr 7 18:45:10 CEST 2016 [og-qa] out Thu Apr 7 18:38:00 CEST 2016 [og-ci] out Thu Apr 7 18:38:00 CEST 2016 [og-test] out Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:41:05 CEST 2016 [envrm02] out Thu Apr 7 18:48:15 CEST 2016 [og-qa] out Thu Apr 7 18:41:05 CEST 2016 [og-ci] out Thu Apr 7 18:41:05 CEST 2016 [og-test] out Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:46:51 CEST 2016 [envrm02] out Thu Apr 7 18:46:52 CEST 2016 [og-qa] out Thu Apr 7 18:46:52 CEST 2016 [og-ci] out Thu Apr 7 18:46:53 CEST 2016 [og-test] out Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out The configuration is the same for all hosts and is this: @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd [envrm02] out: ntpd_enable="YES" [og-qa] out: ntpd_enable="YES" [og-qa] out: ntpd_sync_on_start="YES" [og-ci] out: ntpd_enable="YES" [og-ci] out: ntpd_sync_on_start="YES" [og-test] out: ntpd_enable="YES" [og-test] out: ntpd_sync_on_start="YES" [og-prod-rm] out: ntpd_enable="YES" [og-prod-rm] out: ntpd_sync_on_start="YES" @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'out: #' [envrm02] Executing task '' [envrm02] run: cat /etc/ntp.conf [envrm02] out: [envrm02] out: server 0.freebsd.pool.ntp.org iburst [envrm02] out: server 1.freebsd.pool.ntp.org iburst [envrm02] out: server 2.freebsd.pool.ntp.org iburst [envrm02] out: [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noquery [envrm02] out: restrict 127.0.0.1 [envrm02] out: restrict -6 ::1 [envrm02] out: restrict 127.127.1.0 [og-qa] Executing task '' [og-qa] run: cat /etc/ntp.conf [og-qa] out: [og-qa] out: server 0.freebsd.pool.ntp.org iburst [og-qa] out: server 1.freebsd.pool.ntp.org iburst [og-qa] out: server 2.freebsd.pool.ntp.org iburst [og-qa] out: [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-qa] out: restrict 127.0.0.1 [og-qa] out: restrict -6 ::1 [og-qa] out: restrict 127.127.1.0 [og-ci] Executing task '' [og-ci] run: cat /etc/ntp.conf [og-ci] out: [og-ci] out: server 0.freebsd.pool.ntp.org iburst [og-ci] out: server 1.freebsd.pool.ntp.org iburst [og-ci] out: server 2.freebsd.pool.ntp.org iburst [og-ci] out: [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-ci] out: restrict 127.0.0.1 [og-ci] out: restrict -6 ::1 [og-ci] out: restrict 127.127.1.0 [og-test] Executing task '' [og-test] run: cat /etc/ntp.conf [og-test] out: [og-test] out: server 0.freebsd.pool.ntp.org iburst [og-test] out: server 1.freebsd.pool.ntp.org iburst [og-test] out: server 2.freebsd.pool.ntp.org iburst [og-test] out: [og-test] out: restrict default limited kod nomodify notrap nopeer noquery [og-test] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-test] out: restrict 127.0.0.1 [og-test] out: restrict -6 ::1 [og-test] out: restrict 127.127.1.0 Some observation: * The clock of envrm02 (the HOST) always seems to be correct. * The issue happens on a (apparently) random guest (not always the same one). * The issue happens always on one host at a time. The bhyve command line is, roughly, this one: /usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $machine /usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s 2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap -s 31,lpc -l com1,stdio $machine Thanks in advance.