From owner-freebsd-arm@freebsd.org Sun Aug 18 20:53:59 2019 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 74D75D0B53 for ; Sun, 18 Aug 2019 20:53:59 +0000 (UTC) (envelope-from per@hedeland.org) Received: from outbound1g.eu.mailhop.org (outbound1g.eu.mailhop.org [52.28.6.212]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46BTn22cdsz3x9G for ; Sun, 18 Aug 2019 20:53:57 +0000 (UTC) (envelope-from per@hedeland.org) ARC-Seal: i=1; a=rsa-sha256; t=1566161636; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=q0CE8OGVCIvhuf+XoZZloas76o/M9JbnX0ech+8QAfPHVeUxFNHZ4ocQvfXEe6LmuSNzEsD3SPr6j tZ4x/ZyWzJjYawDHydUNi2QvaxK+HNXKRitk7XHFh32EBkIy5wZjvIyxXs/LBVWl369EoZm9hZYqXR 6Rkmo6NFJO5zckFEjza43FcBfOSNvgPF9rD8Q8ACtTY3eki5+VCI6vjqLjo/TVJCAx3T6hjwQ898JT NznjdXMXV1+Dbynht4EKnnPATMyBRm0iQZacf5F4JyswjKNMJGsjn05Zu1R4uGAoI09uSpIPqdRpjv Qy2UgubWobqLMa2me9M8+xwv2bShrcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:content-type:in-reply-to:mime-version:date: message-id:from:references:cc:to:subject:dkim-signature:from; bh=x6CcadSo+QBxpRQVFrjeX9264AjLDFgAmEgeynUGGls=; b=jvtBVFiVjO4d2XHqh8bDNGNzB6bDaXDnWWQleXoRBch2DjLbpAhiinlUm2HnjdrkYT9xTM3ZUChAO IQEaf7p8UhbPrLiI2GyNPQm+GVGkNe81mTYyWAlglM7/yGY3gQ+jvjK5fEs7AMhNDB6ldXi0nJxzyI VR0PZ44K3bJ8NutNbZ3L6fd4XQ3ajoPmsDHwZkwLWNvponh8iglm5o8upPUHVuzhnY2TszcmVmsFCp f8/iP+dz61B1AMpy7GTOrMFfqVoBVCRqsrGTOEmP1jxfEl3ttVetPUJG7ayfOIUZ4NQDvoe8mdsIPo 2q51DrLV2lWfERKqL4W2W5/TR35+1XQ== ARC-Authentication-Results: i=1; outbound3.eu.mailhop.org; spf=none smtp.mailfrom=hedeland.org smtp.remote-ip=81.228.157.209; dmarc=none header.from=hedeland.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:content-type:in-reply-to:mime-version:date: message-id:from:references:cc:to:subject:from; bh=x6CcadSo+QBxpRQVFrjeX9264AjLDFgAmEgeynUGGls=; b=sirbGGddlkJdCfxNCOb9vbYyrefBwpy/WuCXsdrH/fW8PWXHZPfiisQ6ULY7n6F5q7WfzUxGoJ7Mr F9paQPtHhhLjpQMm4uYRv/RwaYL4/ReLcjr/cRIDuMeZ2p+z53OxpvXNmdCn4fIs/KvG7l3q+jRLD+ Rl8RWCiX+OeYLuQxMXoXlN3Zo1nDHUYpZEZPUmsc3J0jGcPZNISrdEmxXEynEr8uCs6PNz7du8FTdv AjFGNC81IYkjw94ty60SgxujtNKvQVTdPT0gDM9zQ+lWMk7TqmiIxUQaes3TP2JMgdlmQLkTg0/77o 8tviMw4g5ru1CSTq8gZWmFtdH9y7qJA== X-MHO-RoutePath: cGVyaGVkZWxhbmQ= X-MHO-User: 48e63aec-c1fa-11e9-98b8-25195f42ee1c X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 81.228.157.209 X-Mail-Handler: DuoCircle Outbound SMTP Received: from hedeland.org (unknown [81.228.157.209]) by outbound3.eu.mailhop.org (Halon) with ESMTPSA id 48e63aec-c1fa-11e9-98b8-25195f42ee1c; Sun, 18 Aug 2019 20:53:52 +0000 (UTC) Received: from pluto.hedeland.org (pluto.hedeland.org [10.1.1.5]) by tellus.hedeland.org (8.15.2/8.15.2) with ESMTPS id x7IKrpv1054253 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 18 Aug 2019 22:53:51 +0200 (CEST) (envelope-from per@hedeland.org) Subject: Re: Is it a good idea to use a usb-serial adapter for PPS? Yes, it is. To: Ian Lepore Cc: freebsd-arm@freebsd.org References: <69a9bed3-4d0a-f8f6-91af-a8f7d84ee307@hedeland.org> <345bae77417c2495f55799b4c7ca2784f4ece9ed.camel@freebsd.org> <7312032d-2908-9414-0445-6b442c3a02e5@hedeland.org> <523b6f0a0fa5f2aeec298fa74df25d3c4af66acc.camel@freebsd.org> <0426fc8b-5398-d8ab-561e-7823c24403a5@hedeland.org> <24b0eaf25b64d6098b390df092866c69e352d859.camel@freebsd.org> <16c91be1-6f2a-b26d-22c7-be8e4ba8eec0@hedeland.org> <72a964c78cbfc36be2345919633ca2196f0783e3.camel@freebsd.org> From: Per Hedeland Message-ID: Date: Sun, 18 Aug 2019 22:53:51 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <72a964c78cbfc36be2345919633ca2196f0783e3.camel@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46BTn22cdsz3x9G X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=outbound.mailhop.org header.s=dkim-high header.b=sirbGGdd; dmarc=none; spf=none (mx1.freebsd.org: domain of per@hedeland.org has no SPF policy when checking 52.28.6.212) smtp.mailfrom=per@hedeland.org X-Spamd-Result: default: False [-5.53 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[outbound.mailhop.org:s=dkim-high]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[hedeland.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[outbound.mailhop.org:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[212.6.28.52.list.dnswl.org : 127.0.20.0]; NEURAL_HAM_SHORT(-0.97)[-0.975,0]; RECEIVED_SPAMHAUS_PBL(0.00)[209.157.228.81.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-1.26)[ipnet: 52.28.0.0/16(-4.88), asn: 16509(-1.35), country: US(-0.05)]; ASN(0.00)[asn:16509, ipnet:52.28.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Aug 2019 20:53:59 -0000 On 2019-08-18 21:27, Ian Lepore wrote: > On Thu, 2019-08-15 at 23:05 +0200, Per Hedeland wrote: >> On 2019-08-15 17:49, Ian Lepore wrote: >>> On Thu, 2019-08-15 at 13:46 +0200, Per Hedeland wrote: >>>> On 2019-08-09 22:17, Ian Lepore wrote: >>>>> [...] >>>> >>>> I have a theory that your making the kernel clock be based on the 10 >>>> MHz clock also ended up locking the USB poll frequency to that clock, >>>> and thus to the PPS signal - this would certainly explain the result. >>>> Do you think this is a possibility? Would it be possible for you to >>>> re-run the test without modifying the kernel clock? (I do understand >>>> that the results will be harder to interpret with the drift, and >>>> ntpd's correction of it, coming into play.) >>>> >>>> --Per >>>> >>> >>> I'm not sure what you mean by "modifying the kernel clock". The kernel >>> clock always runs on some frequency source. Typically it's derived >>> from the cheap 24 MHz crystal that clocks the SoC, sometimes after >>> being scaled up to 66 MHz by a phase-fractional PLL within the SoC. I >>> arranged to use a very stable nearly-drift-free frequency source >>> instead of a cheap crystal for counting time in the kernel. >>> >>> The kernel clock has nothing to do with usb, including polling >>> intervals; the usb controller hardware handles that, and the root >>> source clock for that is the cheap 24 MHz crystal. >> >> The thing that made me hypothesize that the kernel clock *could* have >> *something* to do with the USB polling frequency was this observation >> in https://blog.dan.drown.org/pps-over-usb (link provided by one of >> the posters in the newsgroup, though he didn't refer specifically to >> this): >> >> Looking closer at the USB latency, you can see the PPS drifting >> relative to the host schedule of polling the USB device for its >> status. The system clock error was 2.215ppm during this time >> period, and this drift matches that error exactly. This probably >> means USB on this system shares the same clock as the system >> clock. This hardware is a Raspberry Pi 2, and I suspect it won't be >> true for other platforms. >> >> So at least on RPi 2, there appears to be a relation between the >> "normal" system/kernel clock and the USB polling frequency. But I have >> no idea if there is such a relation on the system you used, and even >> in that case, *I* certainly can't see how using a different source for >> the kernel clock could affect the USB polling frequency, which is why >> asked if you thought that it was a possibility... >> > > I probably should have been clearer that I meant there was no > correlation between the kernel clock and the usb polling on the system > I was using as a testbed. On most SoCs, and probably even modern x86 > systems, the same frequency source (typically a 24MHz crystal) will be > the root clock for both the usb controller hardware and the timer > hardware from which the kernel clock is derived. However, the kernel > clock is numerically steered to be more stable in frequency and > accurate in phase, so once ntpd has been running for long enough to > capture and disipline the kernel clock, the situation will change. The > usb polling will still be happening at the drifting frequency of the > underlying crystal, while the kernel timestamps used to mark the PPS > pulse time will not be drifting at that rate. Understood. What I don't understand is if, and if so how, your "replacing" the kernel clock with an "exact" frequency from your 10 MHz clock might affect the USB polling. > I have a hard time understanding how the measurements were made in that > pps-over-usb page you cited. There is mention of a STM32F103 > microcontroller, but it's not clear to me what role it plays. There is > also mention of a usb irq and something about a message in a buffer. My understanding is that the "STM32F103 devboard" actually *implements* the USB-to-serial adapter. This means that the author has detailed insight into the workings of the adapter, including the possibilty to modify its firmware - something that is obviously not the case for an off-the-shelf adapter. The "PPS IRQ" is thus something that happens *inside* the "adapter". > For the measurements I made, I was using FTDI usb-serial devices > directly connected to the usb bus on the Wandboard I was using to make > measurements. When the DCD pin changes at the ftdi chip, the chip > internally notes that it has a line-status change that must be > communicated upstream at the next opportunity. When the time comes to > send the data, it sends a 2-byte packet which contains the modem and > line status register bits. (If there are any routine uart data bytes > in the buffer, they also get transferred, but I'm not doing any data > transfer on the adapters I'm using for this test, in fact the only pins > connected are ground and DCD.) When the input packet arrives, the > uftdi driver sees the change in the DCD bit and captures a pps event. > For ftdi chips, all of the foregoing is done with usb BULK-IN > transfers, not control or interrupt transfers. > > I need to run the same tests with some other brands of usb-serial > adapters. I think I may have a cable laying around based on Prolific > PL2303 chipset. If I can find it. I should just go buy a few other- > brand breakout boards and test them. FWIW, I did a bleak replica of your setup, using a "noname" USB-to-serial adapter that I had laying around, which was actually identified as pi kernel: uplcom0: on usbus0 - and since the uplcom driver has "support for Prolific PL-2303/2303X/2303HX", I assume it is one of those. Since I don't have a "real" PPS source, I simulated one with a simple program running on an RPi 3, that generated a pulse on a gpio pin at the turn of the second. This pin was then connected to a gpio pin on an RPi B, and to the DCD pin on the above adapter, also connected to the RPi B - notably without any ttl-to-rs232 converter (since I also don't have one of those). I then set up ntpd on the RPi B to use gpiopps plus the PPS from the uplcom driver: server iburst prefer server 127.127.22.0 minpoll 4 maxpoll 4 fudge 127.127.22.0 refid gpio server 127.127.22.1 minpoll 4 maxpoll 4 noselect fudge 127.127.22.1 refid usb The result was very nice for gpiopps, but the offset for the uplcom PPS varied way more than the 1 ms that could be expected even from a constant 1 ms polling interval (this is a 1.1 device), more like a 5-6 ms interval - some ntpq samples approximately 16 s apart: remote refid st t when poll reach delay offset jitter ============================================================================== * 194.58.205.148 2 u 40 64 377 0.996 -0.049 0.034 oPPS(0) .gpio. 0 l 14 16 377 0.000 -0.004 0.004 PPS(1) .usb. 0 l 13 16 377 0.000 -7.090 3.273 remote refid st t when poll reach delay offset jitter ============================================================================== * 194.58.205.148 2 u 56 64 377 0.996 -0.049 0.034 oPPS(0) .gpio. 0 l 14 16 377 0.000 0.000 0.004 PPS(1) .usb. 0 l 13 16 377 0.000 -2.957 2.567 remote refid st t when poll reach delay offset jitter ============================================================================== * 194.58.205.148 2 u 6 64 377 0.996 -0.049 0.034 oPPS(0) .gpio. 0 l 15 16 377 0.000 -0.001 0.004 PPS(1) .usb. 0 l 14 16 377 0.000 -8.627 4.871 This *could* be taken to imply that there was also some polling going on *in* the adapter towards the DCD pin - especially since the above was with a 10 ms pulse, while if I shortened the pulse to 1 ms, the variation went down to ~ 1 ms, but almost half the pulses were missed. Or it might be due to shaky detection due to the lack of a ttl-to-rs232 converter. In any case pretty inconclusive, other than the observation that it's certainly possible to mess things up...:-) >>> I think people are massively confused by usb. A usb 2.0 bus runs at >>> 480MHz. That means the time to transmit a packet describing a usb >>> serial pin-change event takes literally a dozen or so nanoseconds. The >>> time it takes to transmit an entire sector of disk data is 2 >>> microseconds; even if continuous disk data is flowing, the usb serial >>> adapter gets its round-robin opportunity to send a packet on the bus in >>> between them. >> >> Yes, the transmission speed is obviously not a problem, the question >> is about varying latency due to the polling. >> >>> A USB 2.0 bus spends most of its time idle. The >>> devices on the bus are polled, but the polling happens in time slots >>> that are 125 microseconds wide. There's just no reason for a lot of >>> jitter or latency. >> >> In the newsgroup it was claimed that the polling frequency was 1 kHz >> for USB 1.1 and 4 kHz for USB 2.0, but it seems it should indeed be 8 >> kHz for 2.0 "high" speed. And your test used one USB 1.1 device and >> one 2.0 device. >> >> And "a lot" is a bit subjective, but for any polling at a frequency >> that isn't an exact integral number of periods per second, there will >> be a latency between the start of the PPS pulse and the detection in >> the host that *varies* in an interval the size of the polling >> interval. I believe that interval should thus be expected to be 1000 >> microseconds for 1.1 and 125 microseconds for 2.0. >> > > I think there is some confusion around the concept of usb 1.x devices > on a usb 2.0 bus. I think there may even be some confusion when a 1.x > bus is involved. And then adding to the confusion is the likelyhood > that different usb-serial adapters use different usb transfer types > (bulk vs interrupt)_to communicate line-state changes. > > A usb 1.x bus is divided into 1ms frames. A 2.0 bus is divided into > 125us (micro-)frames. For interrupt endpoints, a usb 1.x bus limits > devices to 1 interrupt transfer per frame, and that may imply that > there is up to 1ms of latency for reporting a DCD change on such a 1.x > bus. A 2.0 bus allows up to 3 interrupt transfers per microframe, > implying latency of up to 125us. > > However, there is no limit on either 1.x or 2.0 busses for how many > bulk transfers can happen to a given endpoint during a frame. The > controller needs to fill a frame with transactions in a way that first > provides all the g'teed bandwidth that is promised to control, > interrupt, and isochronous transfers. It is then free to fill all the > remaining time with bulk transfers. > > To me, this implies that you may end up with nearly no latency (and > negligible jitter) if you have a usb 2.0 bus that has just one or two > devices on it which are communicating via bulk-transfer endpoints. The > controller would be continuously sending BULK IN tokens to the one or > two devices, so that as soon as one of them has data, it gets an > opportunity to deliver it almost immediately (meaning within a few > microseconds). > > The results I see with FTDI usb-serial adapters which use bulk > transfers provide some evidence that my theory may be correct. I think > the bus looks like this: > > BULK IN token to usb 1.x device (do you have anything to say?) > 1.x device NAKs > BULK IN token to usb 2.0 device > 2.0 device NAKs > > BULK IN token to usb 1.x device > 1.x device NAKs > ... (repeat forever) > > In other words, the device(s) aren't getting 1 chance per frame to > transfer data, they are getting many thousands of chances per second. > I think the bus overhead of the BULK-IN token followed by a NAK from > the device, along with the various framing bits and crc and all that > probably adds up to less than 64 bytes per poll. But assuming it took > as much as 64 bytes to do that, if there was one usb-serial device on a > usb 2.0 bus, it would be getting asked about 1 million times per second > whether it had anything to say. This is extremely interesting - if it really is the case that the host will poll "as fast as it can", as opposed to always doing it with a fixed frequency, it would definitely change the picture. Unfortunately I haven't seen any documentation to support that this is the case. > I'd welcome input from low-level USB gurus about the bus and controller > behavior in this regard. I'm afraid we have lost freebsd-usb@ in this sub-thread (it was actually the case before my first comment, but it would probably have happened anyway, since I'm not subscribed to that list). Unless you know that "low-level USB gurus" are also subscribed to freebsd-arm@, it might make sense to forward your message to freebsd-usb@. >> Your ntpq output showed an offset close to 200 microseconds for both >> devices, and I *assumed* that it was more or less constant and thus >> ntpd could trivially be told to correct for it - but maybe that >> assumption was incorrect, there was only one instance of ntpq output? >> >> If it actually varied in an interval per above, I would expect the >> jitter to be significantly higher though. And if it *is* more or less >> constant, can you explain how this is possible? Even the 2.0 125 >> microsecond case should be clearly visible in the offset reported by >> ntpd across a sequence of ntpq requests. >> >>> I'm not on a crusade to change the minds of people who make judgements >>> based on gut feelings and reject objective measurements. I put the >>> measurements out there, and I described the measurement methodology. >>> (Precision timing is what I do for a living, btw.) I'm perfectly >>> willing to explain the methodology in more detail or help interpret the >>> results, but I'm not going to butt heads with people who just reject >>> data they don't like for emotional reasons. >> >> Well, I guess a problem here is that it's my confused head that is >> butted between yours and those of the supposedly-experts that >> participate in the NTP newsgroup/maillist:-) - you already declined to >> participate there, and I don't expect that any of them will take the >> trouble to participate here. Maybe we'll just have to leave it at >> that... > > I had a brief look at whether I could get posting access to the ntp > newsgroup and didn't find anything easy to set up and use, and I'm > reluctant to get an nntp provider and install a newsreader for one > conversation. Understood - there are free nntp providers, but you do need a newsreader of some kind. The newsgroup is sort-of gatewayed to the questions@lists.ntp.org mailing list, but the gatewaying is pretty broken - posts to the mailing list do not appear at all in the newsgroup, and posts to the newsgroup only appear on the mailing list after manual approval by a moderator (at least unless you are subscribed to the mailing list). Thus I believe most participants use the newsgroup. > (The 1980s me would be astounded to hear that future-me > would have any reluctance to get involved in usenet.) Well, there isn't much value there anymore, but there are some groups that refuse to die.:-) --Per