From owner-freebsd-arm@freebsd.org Sat Sep 19 22:19:48 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 95B1B3F113A for ; Sat, 19 Sep 2020 22:19:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bv4rM5Tmwz4Zdb for ; Sat, 19 Sep 2020 22:19:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 12s3WOIVM1nysYnYzCc5xfyErOvNURCWXwb.Yhy7HVAHplBkaSpYXYv3s4rmGuJ 9zZJ6zlkKxFbMPxOACgBT9R4dT_QjhV9quniLkPAP6C4YXAP.c75X6uXLDHGt952Zjme2BYrqk_M 9nIyczf.JxdDZ_opC1e_ncOybBMJmL4M84b6vzRoYLDKZrTdnJ7X0kkgrV9fwfLKQ2zGTCGy91yT 5dl08.vHlUWqsQvf5gBrHn5M.PRLEM9ZhlAA0sD6CRsf.VCqnlwspWRrwu6S1KjsPszqAWFMYk7W CnTVBKxJWrwO8ExqIDAOBc12oHLjdeDA2cXgQN3mrTz6XMIkcOFykP14pO5TMzRGJ9qJjdS3gyMk 5hXaAnOwzVT3wLvn8U2Sk3ZY_X.03_TICVqqbs6PdCf5szwtnFG1_VCg_rdhMhHWHY_eNFErB7oE ElVUw2pfDFJFYPFyGlhxiytz32PU_R6mhidX6SBhQ6KLCSPu2dA36Lkre98XXTKfifGqociAPOoc VVRubPuE8FN9YDl9mfvarmymbm_32ArBOCC2WIVQsfZTpG66kVMplTphmVemu5mK36MNScbvUqa2 lyQniup1H8WosrVZNHjhwVwPqJRd_CigwaogkW.rL0jaML06pDMpUdfeJ0XsrdZomCozFb.nCsOo JhwbOEq5X6WjdAsg55SJF4Eq8G_Wf8ZVjJxGhnRJz6M1BU75fQCeEQPTUMgFOrHGzJu2C508Ntav qU0Fcmq5wLlFFbyWhQ217a9XTOZG8bTeOWo7flAoopet14aqSb_TDFNMq40T092m7br9QrMtAGdx rhBFqEcZxOp5r.hEpUlGylXOQwz40rSKZQNeUblO4KAnE4TIrTb45DI59OwGbN2edDHFO_0BMhXL TmfCc73smdCu19LzI6bpYQ.AORxC4ViKdDlgcG0OzAmi7GJfnDSglFCpeQsZSeOFTjDzVFPud0k9 t.r8MczsiZsXTMHcWCQjuxv.H.Nd.yG2jMuovM4dbOtQ4Ua3BWwicbJ.Cu9K_RuYfLv76XS6QOnz B1jCvqZpcwZhvuGFfBLgEO5ZnJXLb3_m5agkaOnODF10np.HkDhlu6OGRL1hY7i.0O9vxTYUTu.0 F..bDm7B22LrOOBGoRPcnltvbbobhgleLIlXqKDUkIOc2OyuTKsITiFPPwxL5ZO8_SzHSYYCjp3O ngQd0m1jI2F8GdlUSAtr3vFvfyVPpQM_B1lV1IUKTjWyOfaB_iS0yRn0EBOItCiMRm5VRcX.5R0c .Wa_02r4ar5JKvZl7rN3tFGOcRbThD5re3WnExPc_VNLKUGALernfpy2P8TOVlcYV3e2NZqPul9o Zt7TKNe.9uTtdcy0PigadcS_5yVsNfRNMHGapyVgOpJFM0I8qtfsScZkgRuDbIQdIIL47lMk0brE z9g7ymVm6ujtQP7m8cjHiiKkeJbOeZnuUTRAUgmhRN5_HNEDrjr1YThpCIhXhRRyKtUL1aWeGY_h 4Ti4HKIRkSnyZK13aCwrNhqlk.Ho6v2EXPW0GYAuSLtXt7kl1YZFCIAbs9R.f_7gcohs7yeYdgNx z Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Sep 2020 22:19:45 +0000 Received: by smtp418.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 979d2853e5fecc080629c81695e86217; Sat, 19 Sep 2020 22:19:44 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: Comment #135 for bugzilla 237666 : a USB3-handling problem with a investigatory fix for a cortex-a72 context From: Mark Millard In-Reply-To: <7_E2XXmpIwdiLPwD5tkXUAFlHsAhrFIbs87-JJnE57wRg4vrRcqCL1qSToBN_52_YjcPvt7HQSrzA0v6fWDAYIoN348pYVc62bTUXNxudBU=@protonmail.com> Date: Sat, 19 Sep 2020 15:19:42 -0700 Cc: Hans Petter Selasky , freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <3D0CFAD6-A93B-401E-82CD-831E22BD8D7A@yahoo.com> References: <6E618C3D-12DF-429E-A249-5BAB90FC6B15.ref@yahoo.com> <723E6915-94F5-417C-B4AF-EEEBFBDF6162@yahoo.com> <565258A0-BEE1-48F8-9851-E6C7CF7ADAE7@yahoo.com> <75af04ec-0021-3575-40bf-c5ab9b6d4703@selasky.org> <9cf87718-9d4a-60ca-004f-5818371c937b@selasky.org> <47D6CA1E-F842-47B6-97E0-C87B33610C64@yahoo.com> <8BAF3798-4BB4-4C5E-87FC-ECD1458910A2@yahoo.com> <7_E2XXmpIwdiLPwD5tkXUAFlHsAhrFIbs87-JJnE57wRg4vrRcqCL1qSToBN_52_YjcPvt7HQSrzA0v6fWDAYIoN348pYVc62bTUXNxudBU=@protonmail.com> To: Robert Crowston X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Rspamd-Queue-Id: 4Bv4rM5Tmwz4Zdb X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.57 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.04)[-1.044]; FREEMAIL_TO(0.00)[protonmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.01)[-1.008]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.02)[-1.022]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.147:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Sep 2020 22:19:48 -0000 On 2020-Sep-19, at 14:49, Robert Crowston = wrote: >=20 > I=E2=80=99ve been working with aarch64 with mcpu=3Dcortex72 on the = rpi4 with u-boot. I did not need this extra barrier or flush. This = suggests the problem is slightly more nuanced. (I do not object to the = extra barrier if it fixes your problem, I=E2=80=99m just adding extra = information.) >=20 > My question, which may be irrelevant or misguided: The flags field in = the dma tag has an option for specifying if the hardware is cache = coherent (BUS_DMA_COHERENT). Does the UEFI-derived tag passed through to = the xhci driver have this bit set? >=20 > I was slightly confused that explicitly generating a barrier after = calling bus_dmamap_sync(9) had any effect. I understood that the purpose = of the sync function was to generate whatever barrier or fence the = underlying bus required, if any, to ensure consistency. But, if those = barriers were not required in the end, perhaps that is a red herring? You are just somewhat behind on the investigation/testing. All my explicit DSB ST / DSB LD use has been eliminated. (I warned originally that what started working need not be minimal. Only one turned out to be contributing to allowing correct operation in my A72 context.) The one DSB that needed to be added (for aarch64 contexts) is now covered by a call to: usb_bus_mem_flush_all so that the change is machine independent at this level and picks up machine dependent code as needed: # svnlite diff /usr/src/sys/dev/usb/controller/xhci.c Index: /usr/src/sys/dev/usb/controller/xhci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/dev/usb/controller/xhci.c (revision 363590) +++ /usr/src/sys/dev/usb/controller/xhci.c (working copy) @@ -431,6 +431,17 @@ phwr->hwr_ring_seg[0].qwEvrsTablePtr =3D htole64(addr); phwr->hwr_ring_seg[0].dwEvrsTableSize =3D = htole32(XHCI_MAX_EVENTS); + /* + * For bugzilla 237666: + * According to extensible-host-controler-interface-usb-xhci.pdf = , + * the later XWRITE4's to XHCI_ERSTBA_LO and _HI lead to the = xhci + * needing to copy the qwEvrsTablePtr and dwEvrsTableSize + * values above at that time (as the xhci initializes its event + * ring support). This is before the event ring starts to pay + * attention to Run/Stop. Thus, make sure the values are + * observable to the xhci before that point. + */ + usb_bus_mem_flush_all(&sc->sc_bus, &xhci_iterate_hw_softc); DPRINTF("ERDP(0)=3D0x%016llx\n", (unsigned long long)addr); (I'm not the one that identified usb_bus_mem_flush_all as the appropriate FreeBSD way to express the criteria in a machine independent way. I just did the testing and earlier evidence gathering in a A72 context that is an example of the more general bugzilla 237666 issue where non-arm platforms also show odd behavior. I'm hopeful that the above will fix them all.) On Sat, Sep 19, 2020 at 22:18, Mark Millard via freebsd-arm = wrote: > On 2020-Sep-19, at 13:54, Mark Millard wrote: >=20 > . . . > > > > As stands, in my head -r363590 based context, the patch set for this > > overall currently looks like (up to E-mail variability in spaces): > > > > # svnlite diff /usr/src/sys/dev/usb/usb_busdma.c = /usr/src/sys/dev/usb/controller/xhci.c > > Index: /usr/src/sys/dev/usb/usb_busdma.c > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > --- /usr/src/sys/dev/usb/usb_busdma.c (revision 363590) > > +++ /usr/src/sys/dev/usb/usb_busdma.c (working copy) > > @@ -737,6 +737,9 @@ > > */ > > bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD); > > bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD); > > +#ifdef __aarch64__ > > +__asm __volatile("dsb ld" : : : "memory"); > > +#endif > > } > > > > = /*------------------------------------------------------------------------= * > > @@ -750,6 +753,9 @@ > > return; > > } > > bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE); > > +#ifdef __aarch64__ > > +__asm __volatile("dsb st" : : : "memory"); > > +#endif > > } > > >=20 > Looking at the lower level code it looked like DSB SY is used for > the above two automatically and it should be a valid sustitute > for DSB ST and DSB LD, even if overkill. So I'm testing with > dev/usb/usb_busdma.c reverted. (See later below.) >=20 > > = /*------------------------------------------------------------------------= * > > Index: /usr/src/sys/dev/usb/controller/xhci.c > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > --- /usr/src/sys/dev/usb/controller/xhci.c (revision 363590) > > +++ /usr/src/sys/dev/usb/controller/xhci.c (working copy) > > @@ -431,6 +431,7 @@ > > > > phwr->hwr_ring_seg[0].qwEvrsTablePtr =3D htole64(addr); > > phwr->hwr_ring_seg[0].dwEvrsTableSize =3D htole32(XHCI_MAX_EVENTS); > > + usb_bus_mem_flush_all(&sc->sc_bus, &xhci_iterate_hw_softc); > > > > DPRINTF("ERDP(0)=3D0x%016llx\n", (unsigned long long)addr); > > > > >=20 > The test booted just fine, no "Resetting controller" notices or the > like. So now I have just: >=20 > # svnlite diff /usr/src/sys/dev/usb/controller/xhci.c > Index: /usr/src/sys/dev/usb/controller/xhci.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/dev/usb/controller/xhci.c (revision 363590) > +++ /usr/src/sys/dev/usb/controller/xhci.c (working copy) > @@ -431,6 +431,7 @@ >=20 > phwr->hwr_ring_seg[0].qwEvrsTablePtr =3D htole64(addr); > phwr->hwr_ring_seg[0].dwEvrsTableSize =3D htole32(XHCI_MAX_EVENTS); > + usb_bus_mem_flush_all(&sc->sc_bus, &xhci_iterate_hw_softc); >=20 > DPRINTF("ERDP(0)=3D0x%016llx\n", (unsigned long long)addr); >=20 >=20 > for the issue. (So my earlier powerpc* SYNC comments are likely junk.) >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) >=20 > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)