From owner-freebsd-stable@freebsd.org Tue Feb 13 00:29:53 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7774F0805A for ; Tue, 13 Feb 2018 00:29:53 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf0-x235.google.com (mail-lf0-x235.google.com [IPv6:2a00:1450:4010:c07::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23F4F770E1 for ; Tue, 13 Feb 2018 00:29:53 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf0-x235.google.com with SMTP id 37so3952912lfs.7 for ; Mon, 12 Feb 2018 16:29:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:cc; bh=jifWi7SV1ewisCKkLy1Fh6xy7qhKF2mBqgZJHCzwVaw=; b=E4OGo1+yw5iK9lrPicc9TpXNORnSQP0gVvuOE7+LqS1EMxjDitwfMAE5NISW+r+Ka0 kM9VwvOgaN3jhnp7/gCuxGvbLBaA8k1/FWXyl+pB2iMBk5XnED0B3Avwz4ICbce9D/FR B/I1GRpl6J0bhewLoicSk1WGnfX+NyS2ofRMQeMKNK2pMgQB/wM9qL5FKr216vvyY1nC Tcthb+9CQ9I4DYhxGVkR6YGggbfFixr4gFOU5T/KA0WSq09eiKgc2gP+4l7UUaWc4bQF 4bC1oAlnxikSL9ECsH8UrY021VGzon78JOI8UFgRjwBlfMZns9YKJgtNr9LPpSUbCxSj 1jlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:cc; bh=jifWi7SV1ewisCKkLy1Fh6xy7qhKF2mBqgZJHCzwVaw=; b=OvSjdxl9R4FU7H90Hi2VakxOJUbl1wN7qVcdd2zxP5PTQcLqeJdYqbK9YKZq9aTD6d RoclOtzVnaHs/Lm/KJKjPZh6di+VTPjXMd5IPrVCn5kDgTOn6iwG9wsAWei1iADFZv+H Ulj27RSg6Zt5Dda6ffvRTaHklrdPrqK+cg7PWpraW+ZBKc4dQHp24U7Md53ZRA0HuNQD h49FD+tqlvhwZ2YQPd1Z22crRJbic2tmxzdKO94DYD8JuINtPOdXbhqeO9uGUOjqnURM D0yMJMQZ+CowyKbynwD0yT4Kk+p0zOjILLdG418pmozYQ1QTsc85mZZnytF20PmemGCa VD/A== X-Gm-Message-State: APf1xPBhcbM3vZPbEgKLyZAZJIw4ynOIRPbtCPIdXkO4o+F6FsuIPsLl zyd7+RZDAj5M03yexy7cU7MzxQ8Cqle9gjOvhiI= X-Received: by 10.25.229.12 with SMTP id c12mt8918184lfh.106.1518481791490; Mon, 12 Feb 2018 16:29:51 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.179.67.243 with HTTP; Mon, 12 Feb 2018 16:29:50 -0800 (PST) In-Reply-To: References: <0b170dae-b816-ea49-3516-40bfd1deaa2a@bsquare.com> From: Alan Somers Date: Mon, 12 Feb 2018 17:29:50 -0700 X-Google-Sender-Auth: f7duSSYWVRzL6PLCfiwp62XIC5k Message-ID: Subject: Re: Clock occasionally jumps backwards on 11.1-RELEASE Cc: Mike Pumford , FreeBSD Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2018 00:29:53 -0000 On Tue, Jan 23, 2018 at 8:40 AM, Alan Somers wrote: > On Tue, Jan 23, 2018 at 3:48 AM, Mike Pumford > wrote: > >> On 22/01/2018 17:07, Alan Somers wrote: >> >>> Since upgrading my jail server to 11.1-RELEASE, the clock occasionally >>> jumps backwards by 5-35 minutes for no apparent reason. Has anybody seen >>> something like this? >>> >>> Details >>> ===== >>> >>> * Happens about once a day on my jail server, and has happened at least >>> once on a separate bhyve server. >>> >>> * The jumps almost always happen between 1 and 3 AM, but I've also seen >>> them happen at 06:30 and 20:15. >>> >>> That's the window when the period scripts are run which if you have a >> default configuration and a lot of jails will put the system under a lot of >> stress. >> > > That did not fail to escape my notice. However, none of the jails' > periodic jobs involve the clock in any way. And I wouldn't think that a > high CPU load could cause clock drift, could it? This isn't Windows XP, > after all. > > >> * I'm using the default ntp.conf file. >>> >>> Are you running ntpd inside the jail or on the jail host? On my jail >> systems (which are 10.3 and 11.1) I run ntpd out the jail host (outside all >> jails) and not inside the jails and the jails then get the accurate time as >> the underlying host has accurate time. >> > > Only on the host. > > New info: there is a possibility that my NFS server is hanging for > awhile. That would explain my problem's timing. However, ntpd shouldn't > be accessing any NFS shares, and I wouldn't think that a hung NFS server > should be able to pause the clock. I'm doing a new experiment that should > be more informative. But I'll have to wait until the problem recurs to > learn anything. > I have a little more data now. The problem happens much more frequently than I originally realized, but usually for just a few seconds at a time. It looks like the system is hanging for awhile and then recovering. Or at least, the clocks are hanging. The only other possibility would be for both the realtime _and_ monotonic clocks to jump backwards. In any case, the problem is not ntpd's fault. I don't know what could cause a system to hang for up to 30 minutes without crashing, and I'm not sure how to tell unless it happens during working hours. I'll send another update if I learn more. -Alan