From owner-freebsd-cloud@freebsd.org Thu Jan 19 01:56:26 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08B06CB76B8 for ; Thu, 19 Jan 2017 01:56:26 +0000 (UTC) (envelope-from bilbo@hobbiton.org) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B44831625 for ; Thu, 19 Jan 2017 01:56:25 +0000 (UTC) (envelope-from bilbo@hobbiton.org) Received: by mail-wm0-x235.google.com with SMTP id c85so269355732wmi.1 for ; Wed, 18 Jan 2017 17:56:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hobbiton-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=S4yuqu5/JAJ0X6d6LbYRwnjZQhSvPfuj35yg0fCJE+0=; b=KDh+qYqfCq4jZUlRCCan3RkyeBLfkHeDVc7oAXOCwjy2HK+DLkYNe0eAlfzHHbrUpL TjVP9CLn7aL1wJpiQZ425yS/G6nNhZmZ4Ds96gQ5AWH5flhGvR/rnMfJYdJjsiIeSrtI 1M2spFgPxx0LqzwpG3XtWpNiB15DMWco/kb7ARL9A3oCsAidwkFOvgmxlFuMkWfxt+ND TWk6Mb7AJKRWqWPmjrDuTn0B7OlhcFm27YkWoQOJjw6i1U/uR/3f+xt80H5j9beHQAYS wQu8nHqflj3qVbMfWZPIRjgjBAOJSKCe2ON3wDnaH2kfdjmqDt83T+UhfsJEwhmz4crN zA+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=S4yuqu5/JAJ0X6d6LbYRwnjZQhSvPfuj35yg0fCJE+0=; b=cLoBP9EN8uk9mKbfq119QiM3MzMd9Cjs1NR4gT6//a52nqPRMjFcskp/SKavop79OQ HvkTKvQ9/cO9LBsGk0MBAe6kcwROxLqbooQFLGWDbexfcQblQZEyqqu1EAcbr7jsYSBo mlgcumTlLCpDHmnPLfu7bNwKAspnIBXvfEigTZoKk+31GmXHNlGiPWwab+tVeuFN34O5 mmW9jKotvvJbQlfrZFEU4mvwqQ4FCM5JuJtlC1VYxWWBNQBsHt/0mPpll7W87kiRyg4a OYVDWeKCAk5gUZtCA7nBEUO4S+1otdm8Y/Dl6vH0cT3zFKD4yDCnzKh4g478dsQS6ESb aYHA== X-Gm-Message-State: AIkVDXJww81IhSi+XX+EsIcBEnNq+has7hxZ3mRYGmGXlXBVE8tlQ3GC6EfbIX+InwEfO0vmxcKjuRqDb1Zcjw== X-Received: by 10.28.131.132 with SMTP id f126mr12763887wmd.61.1484790983809; Wed, 18 Jan 2017 17:56:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.68.84 with HTTP; Wed, 18 Jan 2017 17:56:23 -0800 (PST) X-Originating-IP: [96.3.205.236] Received: by 10.28.68.84 with HTTP; Wed, 18 Jan 2017 17:56:23 -0800 (PST) In-Reply-To: References: From: Leif Pedersen Date: Wed, 18 Jan 2017 19:56:23 -0600 Message-ID: Subject: Re: GCE: significant clock drift - the solution To: Bruce Walker Cc: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jan 2017 01:56:26 -0000 Just for others to learn from my head-pounding, heh: On 10.3, I've been using kern.timecounter.hardware=ACPI-safe I set it with sysctl after booting, then recorded it in sysctl.conf. No loader tweaking was necessary. Besides speculating from experience, I don't know which is best or why. I arrived at that by trying all the options until one kept the clock stable for me, after other suggestions didn't work consistently. I forgot which, but one option actually seemed to work for days, then made my VMs appear to hang because the virtual clock stopped ticking completely. Upon rebooting the clock seemed fine for a few more days. It was repeatable, and changing the setting decidedly stopped the problem. The takeaway I'm getting at is, picking the wrong thing can (or at least did) make for very weird symptoms. So it's something to suspect if you find that your VMs appear "unstable" in mysterious ways. I can imagine that things get pretty weird if the clock abruptly stops, leaving only IO events left to drive the kernel. Again, I'm not *advising* anyone to use my setting. But if you find you have seemingly unrelated stability problems or are just into jolly bizarre experiments, there's something to see here by selecting different hardware clocks. And it may vary by provider. I observed this on Google Compute Engine. -Leif On Jan 18, 2017 7:10 PM, "Bruce Walker" wrote: Hi gang, first message. I am developing on Google Cloud Engine, I'm using the FreeBSD community image, and so far it's been pretty painless. Thank you! But I noticed that the system clock on one of my two instances was drifting fast over time, and I mean really fast. I estimate about 1 second ahead for every 20-30 seconds of elapsed time. So I would stop and restart ntpd and that would reset the time. Then the creep again ... Long story short: I searched the Google gce-discussion group and found another FreeBSD user with the same issue, and a solution was provided by one Andy Carrel. In a nutshell two system boot files need tweaks: /boot/loader.conf machdep.disable_tsc_calibration=1 kern.timecounter.invariant_tsc=1 /etc/sysctl.conf kern.timecounter.hardware=TSC-low Andy further explains: "The loader.conf changes instruct the kernel to not try to do calibration of the TSC, since this calibration could give bad results due to the virtualization, and to ignore the fact that the "invariant TSC" feature is not advertised by the CPU and just assume the CPU has this feature. The sysctl.conf change instructs the kernel to use the TSC-low timecounter which was made available by the loader.conf changes." For reference, here's a link to the posting: https://groups.google.com/forum/#!msg/gce-discussion/ NKhl1QOVucQ/EDyLd_FxCAAJ Maybe these can make it back into the community cloud image. -- -bmw _______________________________________________ freebsd-cloud@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-cloud To unsubscribe, send any mail to "freebsd-cloud-unsubscribe@freebsd.org"