From owner-freebsd-arm@freebsd.org Tue Sep 10 14:41:31 2019 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC19ED88AB for ; Tue, 10 Sep 2019 14:41:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46SSQg46vNz4Wct; Tue, 10 Sep 2019 14:41:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x8AEf1Ef064224 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 10 Sep 2019 17:41:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x8AEf1Ef064224 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x8AEexjW064220; Tue, 10 Sep 2019 17:40:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 10 Sep 2019 17:40:59 +0300 From: Konstantin Belousov To: Ian Lepore Cc: "O'Connor, Daniel" , "freebsd-arm@FreeBSD.org" Subject: Re: Is it a good idea to use a usb-serial adapter for PPS input? Yes, it is. Message-ID: <20190910144059.GA2559@kib.kiev.ua> References: <61B1AAF3-40F6-47BC-8F05-7491C13BF288@dons.net.au> <9E142F1A-5E8C-4410-91F5-7C80B3D0A15B@dons.net.au> <9D2ACA87-C2DB-40D9-9638-B0E215A4EEC0@dons.net.au> <0A7796DA-508B-4FE6-B5C0-391EC5F46C86@dons.net.au> <20190908134205.GO2559@kib.kiev.ua> <25BF53F1-23CF-4619-AB13-110566D6DC82@dons.net.au> <20190909164501.GT2559@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46SSQg46vNz4Wct X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Sep 2019 14:41:31 -0000 On Mon, Sep 09, 2019 at 04:19:48PM -0600, Ian Lepore wrote: > On Mon, 2019-09-09 at 19:45 +0300, Konstantin Belousov wrote: > > On Mon, Sep 09, 2019 at 05:20:25PM +0930, O'Connor, Daniel wrote: > > > > > > > > > > On 8 Sep 2019, at 23:12, Konstantin Belousov > > > > wrote: > > > > > I suppose it would be good to change it to the same structure > > > > > as the feed forward clock stuff, that way it is much easier to > > > > > change the number of hands at compile time.. > > > > > > > > > > > > > The reason to not increase it by default is the same as the > > > > reason why it > > > > was reduced. But since I did still not provided the analysis why > > > > increasing > > > > timehands count helps for Ian' and your case, I think that making > > > > it easy > > > > to increase the timehands number is due. > > > > > > I am a bit worried (based on your commit log for r303383) that the > > > code now relies on it being 2 for correct function, although given > > > I increased it to 10 and this system works fine perhaps not :) > > > > > > > It means that the sliding window where consumer should reach the > > writer > > to read the current timehands is larger. I think that in reality the > > cases where the latency of get*time*(9) KPI increased are very rare. > > > > Still the question is why your system have the negative impact from > > reducing the number of timehands. Interrupt should never interrupt > > tc_windup() because it is protected by spinlock. Silly question, are > > spinlocks functional on this machine ? > > > > > > Yep, spinlocks work fine. > > The problem sequence that happens, as I remember it, is that the system > is sleeping in cpu_idle() which has stopped the single core with the > WFI (wait-for-interrupt) instruction. The wakeup from WFI is an > eventtimer event that was scheduled to handle state->nexthard to keep > the timecounter updated. As part of handling that event, the system > calls the timecounter's tc_poll_pps function (which is dmtpps_poll() in > arm/ti/am335x/am335x_dmtpps.c). That captures the pps event time using > the current timehands, then schedules a taskqueue task to finish > processing the captured data. By the time the taskqueue task runs, > tc_windup() has been called more times than there are timehands, so > pps_event() rejects the data because th->th_generation captured in > dmtpps_poll() does not match th_generation in that set of timehands > anymore. Is it really needed to schedule task for pps_event() ? We don't for tc_windup(), and pps_event() is quite similar. And for pps_event(), we might use same opportunistic locking as for tc_windup(). > > When I was trying to track this down a couple years ago, part of why I > gave up was that I couldn't point my finger at anything that was > happening and say "here is the problem". It's odd to me that > tc_windup() gets called 3 or 4 times so quickly like that, but odd is > not the same as wrong. > > Ironically, a very busy system doesn't suffer this problem. The bad > sequence happens only when the system is very quiet, spending most of > its time in cpu_idle(). It's something about that path where you come > out of cpu_idle() to process a hardclock event followed by a taskqueue > task that causes the multiple tc_windup calls to happen. If hardclock > events are interrupting other processes, you somehow don't get as many > calls to tc_windup between the pps_capture() at interrupt time and the > pps_event() call from the taskqueue. > > The problem also doesn't happen on multi-core systems, perhaps because > the taskqueue event begins running sooner on a different core (that is > a wild guess). It requires the combo of single core, mostly-idle, and > pps-capture driver. > > -- Ian