From owner-freebsd-arm@freebsd.org Tue Nov 14 11:29:10 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E441DBB5DB for ; Tue, 14 Nov 2017 11:29:10 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-121.reflexion.net [208.70.210.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 05DE07AAB0 for ; Tue, 14 Nov 2017 11:29:09 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 5415 invoked from network); 14 Nov 2017 10:29:08 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 14 Nov 2017 10:29:08 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Tue, 14 Nov 2017 05:29:08 -0500 (EST) Received: (qmail 15897 invoked from network); 14 Nov 2017 10:29:07 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 14 Nov 2017 10:29:07 -0000 Received: from [192.168.1.25] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 28F78EC8F04; Tue, 14 Nov 2017 02:29:07 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: PINE64 - 12.0-CURRENT r324563 - ntpd can't keep time From: Mark Millard In-Reply-To: <23D8D3DB-CC10-4A61-924A-D81FF22E0250@dsl-only.net> Date: Tue, 14 Nov 2017 02:29:06 -0800 Cc: freebsd-arm@FreeBSD.org, Andreas Schwarz , ian@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2B53AA8A-7AA1-45F9-AA3C-144A69A51C0A@dsl-only.net> References: <4BF75B1E-318C-414A-B5D4-4BA7D6578316@dsl-only.net> <1509029871.56824.49.camel@freebsd.org> <4af740148ca.47a474e3@mail.schwarzes.net> <04b67007-a95a-9e40-28b4-764adf8b2ded@restart.be> <23D8D3DB-CC10-4A61-924A-D81FF22E0250@dsl-only.net> To: Henri Hennebert X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2017 11:29:10 -0000 [Top post of an FYI and other short notes.] FYI: My Pine64+ 2GB context uses an external USB3 powered hub with a USB3 SSD on one of the Pine64+ 2GB USB2 ports in order to hold the UFS root file system. (I can boot with just the sdcard but rarely do.) (I've only used ZFS in a couple of contexts, all with more than 2 GiBytes of RAM.) The -r320599 'works' point means that a (slow) bisect may be possible in your context to narrow the range of what changes in head might be at issue in your context. What is the smallest -r?????? figure above -r320599 that you have seen the problem with? Could you test some approximate mid-point between the two? (A difficulty with this can be avoiding other known problems in the history.) [End top post] On 2017-Nov-14, at 1:39 AM, Mark Millard wrote: > On 2017-Nov-13, at 11:06 PM, Henri Hennebert = wrote: >=20 >>> . . . >>=20 >> I believe that the clock of the Pine64+ is going too fast and that = the 2 servers where polled and so show this offset/jitter. In an other = occurrence of this problem, if I wait long enough, all servers display = huge offset. >> In a old version of Freebsd12, when dev.cpu.0.freq was accessible, = the problem appear when I force the frequency to 1200. >=20 > May be the time is just being damaged on occasion > or read-incorrectly sometimes? (Once seen as wrong, > the adjustment process will gradually change time > towards what it calculates as a supposedly better > time.) >=20 > Good to know that eventually all the servers show > a large jitter(?) at the same time. (Jitter reflects > more history and so may be a better reference.) >=20 >> In the same network, other computers (amd64) get no problem with this = ntpd configuration. >=20 > Good to know. It helps localize the issue. >=20 >>> It looks to me like you need to avoid those 2 severs, >>> substituting some others or some such. If some >>> communication channel(s) involved are corrupting data >>> then simply switching servers might not avoid the >>> issue? >>> I finally got a hold of the rpi3 and updated it to >>> -r325700 from back a -r320123. it has been up 9 hr >>> 40 min and date is still showing the correct time, >>> no evidence of huge offsets or huge jitter. >>> The Pine64+ 2GB is also at -r325700 now and has been >>> up for 2 days. It also is working fine. Again no >>> evidence of huge offsets or huge jitter. >>=20 >> Did you run heavy jobs running on this system? My Pine64+ is managing = my connection to internet (mpd5), named, dhcp, my mail (sendmail + = cyrus-imap) and dspam with postgres as db. The problem crops up when for = example I run a port upgrade. >=20 > I did a complete, from-scratch -j4 buildworld buildkernel > on the Pine64+ 2GB, keeping it busy for a little under > 18 hr + 10 min as I remember. top showed that it used some > swapspace during this. I included: >=20 > WITH_CLANG=3D > WITH_CLANG_IS_CC=3D > WITH_CLANG_FULL=3D > WITH_CLANG_EXTRAS=3D > WITH_LLD_BOOTSTRAP=3D > WITH_LLD=3D > WITH_LLD_IS_LD=3D > WITH_LLDB=3D > WITH_BOOT=3D >=20 > But it did claim to reuse the system clang instead > of building a bootstrap clang compiler. Yet the > WITH_LLD_BOOTSTRAP=3D did lead to a bunch of > bootstrap activity anyway, so two builds of lld > related materials. (The purpose of the build > was to see if it would complete or not, extra work > made for a better test.) >=20 > It was on an earlier version than -r325700=20 > that I last rebuilt my ports but rebuilding the > ports has never caused a time problem either. > The ports builds were via poudriere. >=20 > But I do not use the Pine64+ 2GB for any of those > other things that you list as using yours for. I > do ssh over ethernet into the Pine64+ to work on > it but it is not a service provider at all, just > a machine for me to do build experiments with > (native and cross builds) and other forms of problem > investigations. nfs is present but little/rarely > used, and only temporarily in use at that. >=20 > As a step in potential isolation of what contributes > vs. what does not, could you temporarily revert to > not using/running the Pine64+ 2GB for the other > activities and try the ports builds and/or buildworld > buildkernel? Does it still have the problem in > something like my simpler context? >=20 > If time no long jumps, then you could add back an > activity and try again until the additions lead to > the time problem. That might give a clue what could > be interfering with time. >=20 > Also: I do not know if you have access to a 2nd > Pine64 to check for a device-specific problem by > switching devices temporarily. Such might also > allow keeping your services active while checking > the simpler context. >=20 > If Pine64s had a general time problem, there would > be a general problem for everyone else. I think the > stage of investigation is still: try to isolate a > smaller context sufficient to have the problem but > for which an even smaller context is observed to > not have the problem. Try to narrow the range of > differences between observed-working and > observed-failing to gain a clue about what > contributes to the problem vs. what does not. >=20 >=20 >=20 > FYI: Sitting idle the Pine64+ 2GB here shows > in top: >=20 > sshd: @pts/0 (sshd) > sshd: [priv] (sshd) > /usr/sbin/sshd > /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/ntpd.pid -f = /var/db/ntpd.drift > sendmail: accepting connections (sendmail) > sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) > /sbin/devd > top -CawPosize > login [pam] () > su () > top -CawPosize > /usr/sbin/mountd -r > su (sh) > -sh (sh) > -sh () > nfsd: master (nfsd) > dhclient: awg0 (dhclient) > /usr/sbin/cron -s () > dhclient: awg0 [priv] (dhclient) > /usr/sbin/syslogd -s > /usr/sbin/rpcbind > nfsd: server (nfsd) >=20 > (I do not normally use sendmail for anything. It > just sits there while the Pine64+ 2GB is up. nfs > is normally idle but is used on occasion.) >=20 > FYI: The installed ports list (far more is built > by poudriere): >=20 > # pkg info > atf-0.21 C, C++ and shell libraries to write = ATF-compliant test programs > binutils-2.28,1 GNU binary tools > bonnie-2.0.6_1 Performance Test of Filesystem I/O > bonnie++-1.97.3 Performance Test of Filesystem I/O > ca_root_nss-3.32.1 Root certificate bundle from the = Mozilla Project > curl-7.56.1 Command line tool and library for = transferring data with URLs > dialog4ports-0.1.6 Console Interface to configure ports > dtrace-toolkit-1.0_1 Collection of useful scripts for DTrace > dwarfdump-20161124 Tool to display DWARF debugging = information in ELF files > expat-2.2.1 XML 1.0 parser written in C > freebsd-release-manifests-20171003 FreeBSD release manifests > gcc7-7.2.0_2 GNU Compiler Collection 7 > gdb-8.0.1_1 GNU GDB of newer version than comes = with the system > gettext-runtime-0.19.8.1_1 GNU gettext runtime libraries and = programs > git-lite-2.14.3 Distributed source code management tool = (lite package) > gmp-6.1.2 Free library for arbitrary precision = arithmetic > indexinfo-0.3 Utility to regenerate the GNU info page = index > iorate-3.05 General purpose storage I/O = benchmarking tool > iozone-3.457 Performance Test of Sequential File I/O > kyua-0.13_4,3 Testing framework for infrastructure = software > libedit-3.1.20170329_2,1 Command line editor library > libidn2-2.0.4 Implementation of IDNA2008 = internationalized domain names > libnghttp2-1.27.0 HTTP/2.0 C Library > libunistring-0.9.7 Unicode string library > lua52-5.2.4 Small, compilable scripting language = providing easy access to C code > lutok-0.4_6 Lightweight C++ API for Lua > mpc-1.0.3 Library of complex numbers with = arbitrarily high precision > mpfr-3.1.6 Library for multiple-precision = floating-point computations > patch-2.7.5 GNU patch utility > pcre-8.40_1 Perl Compatible Regular Expressions = library > perl5-5.24.3 Practical Extraction and Report = Language > pkg-1.10.1 Package manager > portlint-2.17.13 Verifier for FreeBSD port directory > portmaster-3.17.10 Manage your ports without external = databases or languages > poudriere-devel-3.1.99.20171028 Port build and test system > randomio-1.4 Multithreaded disk i/o microbenchmark > readline-7.0.3_1 Library for editing command lines as = they are typed > rsync-3.1.2_7 Network file = distribution/synchronization utility > sqlite3-3.20.1_2 SQL database engine in a C library > stress-1.0.4 Tool to impose load on and stress test = Unix-like systems > sudo-1.8.21p2 Allow others to run commands as root > unzip-6.0_7 List, test, and extract compressed = files from a ZIP archive > wget-1.19.2 Retrieve files from the Net via HTTP(S) = and FTP > zip-3.0_1 Create/update ZIP files compatible with = PKZIP >=20 > So it is enough to keep the Pine64+ 2GB busy > for a a notable time when rebuilt. In particular, > gcc7-7.2.0_2 (and what it requires) takes a fair > time to build. >=20 > The poudriere build of the ports is currently the > primary thing the ports are used for: build tests > to catch build failures, should they occur. =3D=3D=3D Mark Millard markmi at dsl-only.net