From owner-freebsd-arm@freebsd.org Sat Sep 19 19:18:55 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D14B73EC74C for ; Sat, 19 Sep 2020 19:18:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bv0qf6nHYz4N6X for ; Sat, 19 Sep 2020 19:18:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: UeCunM0VM1kYoboDWTqh68AXJnIUx5y4VBMzEw_WZdyNSmJuGJ4U1sDLu849ZtB EHz6BTGvmyC8vlDQvZ9dpqVTDawaJ00VMWW_ESJDMgo.hKFfCxVEefxd0uDS__7z3ULEkhipuQVH 6dIqBXsqr7GEXpn6ozj6JLw.0Cnj7oybZq9S2fGcO8XX5C5VcJajflYSWgezq7a2Ioq9ZOdtHNFr XAd32lzydIjR1_5ATOsZ4yZEIZoegrAYOnFUBC7.ZAnOjscqf7jG6nhQUlPAYvL8e5xTNjY3jbHA N_KGOCbdlTyh8jAqX.I3J5VQq4747WKbDvaepd_4McJOOe9FF3LdAuDtpMHcJakEDqsbwcX7LqmL 2.zbD_mh7zzPOfhjPzkHzdrmeRgd.__a2CGcp19lfvVeTwJnseJUXOrpYXothqxdKC3YCTe_d4lu qtD1GNJkP6MtoFNRQZU30HiKmXuupbmauAmJWFPrrYatF3sSNFhqIMzfetT_77vmoVaSx0qge1WZ zkoKSbbXpZNAF0wzF4hPnXjmzmzutKMpijt4lNpSsMzPalp875lN.EKma07CeqMyluH9rBj0aMGU bipU8FODDdDv4PDskmxEVDev_JjDqyLG3igigJTY4cOJqYz0pzq0Yl47ynoGPWyeqttY7u8k2Zzr bQVoSayisey_VGlZ_hO3hhnYJL3NZSOjqgTi3RiNWR.rz7IdQdFzV.5URV5UGlTH3tuJghpLnSKI www4mvjgLF5zc8e3tvwXvaTKIwcp6cYKPJuz.5g0yLU9_hv8qe2gunxDo20lvaqdcmBV6jxo7Bid nJpGKFT9B77cRI8RoEHZS8c06FU8ffZ6lY66naMrZgE8_neLkSNNE3V8hZGnWncyOkKCtHcilxFr ffPPgD29JtutDhDVooiHFm8qYDv2l2SMHh0QeQec8D7NZb8nmNsYT6u7uJ2MPgqei4eKirJrgCAa wvPJifsUsISqzOy.wubZnn.OoWBVnAJAPrTV3fPC8kz3KinjdMuTHVD6lcmXwEX1a2CTELcgL7x0 uIikZ8UhzSCUIJbOTcBk4O8iI8vn4DEhl.oGN.HEBpWudjD99YjWuCgGE38supSRp6yXR09urnrX bYqgANNFbdyKinJ8_1lB360dqRNi50snCMiurqF4SBIaieOPm2aiUAk.jK0S9Fmhxcs2S7G.lhPZ jjWS_R7EsIXZBwhZFEZEWdlJt_RewRQ8JG5AwJ9GuHf8dIF.cooTORqfoCP5KK6KmWO2GJHSPpRd Wyfrf.WRsi4E3KhxQWf3qqbsYE1pPKJzfqzh4zUxLZ.EcX6jxpkoc_P8s0eFxrFdZAuj_1iU599W CQzMIT9wOf2ABLSRyvZh.EbX6D7wytaI8fvm1HNucm8a2HP4CbGpAy3oEZasZg8rwQZOgoShtI.Z 9dzioB25IxK3E6QZowAjTCN1aISfQby_CGlPadWIU6Fg.dRhh.Pd6rWDRWb3A..7ZohnihxEsFNZ VfMq9.aYAyqhrpxeboMfMweQ149zxAhjj4F6c4umTiSxtXqo5GwibiWaJYbdEqS3bv8Z5Ysx8t6Z hrdAPLhoaaKk176ADZSaHPuDxxA.Hiqi2hA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Sep 2020 19:18:52 +0000 Received: by smtp401.mail.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID ee9a2c4f9bc944b2d10ac9f074e682aa; Sat, 19 Sep 2020 19:18:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: Comment #135 for bugzilla 237666 : a USB3-handling problem with a investigatory fix for a cortex-a72 context From: Mark Millard In-Reply-To: <723E6915-94F5-417C-B4AF-EEEBFBDF6162@yahoo.com> Date: Sat, 19 Sep 2020 12:18:50 -0700 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <565258A0-BEE1-48F8-9851-E6C7CF7ADAE7@yahoo.com> References: <6E618C3D-12DF-429E-A249-5BAB90FC6B15.ref@yahoo.com> <6E618C3D-12DF-429E-A249-5BAB90FC6B15@yahoo.com> <866bd652-5a8b-7500-b1e0-7528c035048b@selasky.org> <578faf26-6042-91ec-c639-2bca17105fae@selasky.org> <723E6915-94F5-417C-B4AF-EEEBFBDF6162@yahoo.com> To: Hans Petter Selasky X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Rspamd-Queue-Id: 4Bv0qf6nHYz4N6X X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.61 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.06)[-1.065]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.029]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.020]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.148:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Sep 2020 19:18:55 -0000 In the below I've cut out text without always indicating where so that code looks complete that is complete. I've also included some text from another reply of yours that fills in some specifics of what you were after. On 2020-Sep-19, at 04:46, Hans Petter Selasky = wrote: >=20 >>>> . . . >>>>=20 >>>> Your finding indicate a problem in usb_pc_cpu_flush() and >>>>=20 >>>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE); >>>>=20 >>>> Try to put the dsb only after dmamap_sync. >>>>=20 >>>> void >>>> usb_pc_cpu_flush(struct usb_page_cache *pc) >>>> { >>>> if (pc->page_offset_end =3D=3D pc->page_offset_buf) { >>>> /* nothing has been loaded into this page cache! */ >>>> return; >>>> } >>>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE); >>>> } >>>> . . . >>> After adding a couple of more DSB ST instructions and rebuilding >>> (cross build), installing, and booting, I started a self-hosted, >>> from-scratch, buildworld buildlkernel and intend to installkernel >>> installworld and reboot if it appears to go well. (Better than >>> just a boot test for if things seem at least sufficient, even if >>> overkill.) But it will be hours before buildworld build kernel is >>> done (-j3). (I'll sleep during much of that time.) The buildworld buildkernel, installkernel installworld, reboot sequence seems to have gone fine. >>> . . . >>> phwr =3D buf_res.buffer; >>> addr =3D buf_res.physaddr; >>> addr +=3D (uintptr_t)&((struct xhci_hw_root = *)0)->hwr_events[0]; >>> /* reset hardware root structure */ >>> memset(phwr, 0, sizeof(*phwr)); >>> phwr->hwr_ring_seg[0].qwEvrsTablePtr =3D = htole64(addr); >>> phwr->hwr_ring_seg[0].dwEvrsTableSize =3D = htole32(XHCI_MAX_EVENTS); >>> DPRINTF("ERDP(0)=3D0x%016llx\n", (unsigned long = long)addr); >>> #ifdef __aarch64__ >>> __asm __volatile("dsb st" : : : "memory"); >>> #endif >>> XWRITE4(sc, runt, XHCI_ERDP_LO(0), (uint32_t)addr); >>> XWRITE4(sc, runt, XHCI_ERDP_HI(0), (uint32_t)(addr >> 32)); >>> addr =3D buf_res.physaddr; >>> DPRINTF("ERSTBA(0)=3D0x%016llx\n", (unsigned long = long)addr); >>> XWRITE4(sc, runt, XHCI_ERSTBA_LO(0), (uint32_t)addr); >>> XWRITE4(sc, runt, XHCI_ERSTBA_HI(0), (uint32_t)(addr >> 32)); >> . . . >> Probably same bug in the USB invalidate function. >> On 2020-09-19 11:31, Mark Millard wrote: >>> #ifdef __aarch64__ >>> __asm __volatile("dsb ld" : : : "memory"); >>> #endif >>> temp =3D le32toh(phwr->hwr_events[i].dwTrb3); >>> =20 >>=20 >>=20 >> If you look a few lines up you'll find the: >>=20 >> usb_pc_cpu_invalidate(&sc->sc_hw.root_pc); >>=20 >> That's where this instruction belongs. >> Try to patch those generic flush/invalidate functions instead. I've = been very careful about those. >=20 > Sure. >=20 >>> temp =3D le32toh(phwr->hwr_events[i].dwTrb3); >>> k =3D (temp & XHCI_TRB_3_CYCLE_BIT) ? 1 : 0; >>> if (j !=3D k) >>> break; >>> always took the break because temp was always zero, k was always = zero, >>> and j was 1. I expect this is because the wrong qwEvrsTablePtr and >>> dwEvrsTableSize content was in use by the xhci (fixed by the DSB = ST). >>> . . . >>=20 >> It smells like a generic problem in the busdma synchronize code. >>=20 >>> (Pe means "executing processor".) >>> I expect the removal of that specific DSB ST to result in >>> xhci_interrupt_poll's code again taking the break every time, >>> just based on event ring behavior alone. >>=20 >> Again, try to patch both >> usb_pc_cpu_flush() >> and >> usb_pc_cpu_invalidate() >=20 > Sure. >=20 Unfortunately, it fails the same old way as far as what is seen on the console. So the steps used were: I reverted /usr/src/sys/dev/usb/controller/xhci.c . Then changed things so that: # svnlite diff /usr/src/sys/dev/usb/usb_busdma.c Index: /usr/src/sys/dev/usb/usb_busdma.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/dev/usb/usb_busdma.c (revision 363590) +++ /usr/src/sys/dev/usb/usb_busdma.c (working copy) @@ -737,6 +737,9 @@ */ bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD); bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD); +#ifdef __aarch64__ +__asm __volatile("dsb ld" : : : "memory"); +#endif } =20 = /*------------------------------------------------------------------------= * @@ -750,6 +753,9 @@ return; } bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE); +#ifdef __aarch64__ +__asm __volatile("dsb st" : : : "memory"); +#endif } =20 = /*------------------------------------------------------------------------= * As I expected, the resulting build fails to boot the same old way: a sequence messages that repeats indefinitely and involves "xhci0: Resetting controller" and no root file system mount happens. So I put back the just last change that had previously lead to the successful booting (given the prior DSB ST / DSB LD additions). I did so exactly as I originally had it: # svnlite diff /usr/src/sys/dev/usb/controller/xhci.c Index: /usr/src/sys/dev/usb/controller/xhci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/dev/usb/controller/xhci.c (revision 363590) +++ /usr/src/sys/dev/usb/controller/xhci.c (working copy) @@ -406,6 +406,9 @@ =20 addr =3D buf_res.physaddr; =20 +#ifdef __aarch64__ +__asm __volatile("dsb st" : : : "memory"); +#endif XWRITE4(sc, oper, XHCI_DCBAAP_LO, (uint32_t)addr); XWRITE4(sc, oper, XHCI_DCBAAP_HI, (uint32_t)(addr >> 32)); XWRITE4(sc, oper, XHCI_DCBAAP_LO, (uint32_t)addr); The result still fails. So more is required. I'll do some exploring to try to find a combination that has fewer DSB ST / DSB LD changes in usb/controller/xhci.c than my original A72 success, leaving the usb/usb_busdma.c patch in place. It may take a while for me to have a successful result to report. Side note: The from-scratch buildworld buildkernel reported: World built in 37469 seconds, ncpu: 4, make -j3 Kernel(s) GENERIC-NODBG built in 2474 seconds, ncpu: 4, make -j3 which totals to somewhat over 1hr 50min faster than the last time I did such a -j3 from-scratch rebuild on the same RPi4B: World built in 44034 seconds, ncpu: 4, make -j3 Kernel(s) GENERIC-NODBG built in 2895 seconds, ncpu: 4, make -j3 Both were based on a head -r363590 variation rebuilding itself but the newer build is based on everything being -mcpu=3Dcortex-a72 based already in the running system. (Same over_voltage=3D6 and arm_freq=3D2000 setting in use both times.) (I can cross build, which does not take long.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)