From owner-freebsd-arm@freebsd.org Sun Oct 4 08:15:58 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 289543FE8EB for ; Sun, 4 Oct 2020 08:15:58 +0000 (UTC) (envelope-from crowston@protonmail.com) Received: from mail-40133.protonmail.ch (mail-40133.protonmail.ch [185.70.40.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "SwissSign Server Gold CA 2014 - G22" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C3xPm0z2Vz4PB3 for ; Sun, 4 Oct 2020 08:15:55 +0000 (UTC) (envelope-from crowston@protonmail.com) Date: Sun, 04 Oct 2020 08:15:46 +0000 To: Mark Millard From: Robert Crowston Cc: freebsd-arm Reply-To: Robert Crowston Subject: Re: RPi4B 3 GiByte pcie limitation: The following change survived my huge-file duplicate-and-diff tests so far Message-ID: In-Reply-To: <447B28E3-3964-4498-A91D-528EA5923793@yahoo.com> References: <7506FD70-D814-4F64-BFAF-CCCE9A0D71A3@yahoo.com> <447B28E3-3964-4498-A91D-528EA5923793@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=10.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM shortcircuit=no autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mailout.protonmail.ch X-Rspamd-Queue-Id: 4C3xPm0z2Vz4PB3 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.16 / 15.00]; HAS_REPLYTO(0.00)[crowston@protonmail.com]; FREEMAIL_FROM(0.00)[protonmail.com]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[protonmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[protonmail.com,quarantine]; NEURAL_HAM_SHORT(-0.20)[-0.203]; FREEMAIL_TO(0.00)[yahoo.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[protonmail.com]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.933]; R_DKIM_ALLOW(-0.20)[protonmail.com:s=protonmail]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.03)[-1.027]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[protonmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.40.133:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[185.70.40.133:from]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2020 08:15:58 -0000 I have experimented with the segment size. I cannot have tested every scena= rio, but I did not find it made any difference, even if I went down to segm= ents of only a page. I understand the original Linux driver used a bounce buffer where each DMA = transaction was limited to a few MB. However that design was dropped when t= he driver was merged into the mainline kernel. =E2=80=94 RHC. =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Saturday, 3 October 2020 03:35, Mark Millard wrote: > [Operator error note and another note.] > > On 2020-Oct-2, at 13:56, Mark Millard wrote: > > > It appears to me that 3 GiByte works for lowaddr and > > maxsize but that the maxsegsz needs to be limited to > > 1 GiByte [or so, maybe (1 Gi - 1) Bytes]. > > I did the following based on all those notes that I > > sent out, that included the ?_TXFR_LEN being limited > > 30 bits for (normal) DMA engines (0-6) and the DMA4 > > engines (11-14) [but not DMA LITE (7-10)]: > > Ignore: I had the wrong kernel build installed. > In fact the below fails when the intended kernel > is tested. > > > svnlite diff /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c /usr/src/s= ys/arm/broadcom/bcm2835/bcm2838_xhci.c > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > Index: /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > > > --- /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c (revision 365932) > > +++ /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c (working copy) > > @@ -105,7 +105,8 @@ > > * > > > > - Whatever the true maximum address, 960 MiB works. > > */ > > -#define DMA_HIGH_LIMIT 0x3c000000 > > +#define DMA_SEG_HIGH_LIMIT 0x3c000000 > > +#define DMA_TOTAL_HIGH_LIMIT 0xc0000000 > > #define MAX_MEMORY_LOG2 0x21 > > #define REG_VALUE_DMA_WINDOW_LOW (MAX_MEMORY_LOG2 - 0xf) > > #define REG_VALUE_DMA_WINDOW_HIGH 0x0 > > @@ -642,12 +643,12 @@ > > / > > error =3D bus_dma_tag_create(bus_get_dma_tag(dev), / parent / > > 1, 0, / alignment, bounds */ > > > > - DMA_HIGH_LIMIT,=09=09=09/* lowaddr */ > > > > > > > > - DMA_TOTAL_HIGH_LIMIT,=09=09/* lowaddr */ > > BUS_SPACE_MAXADDR,=09=09=09/* highaddr */ > > NULL, NULL,=09=09=09=09/* filter, filterarg */ > > > > > > > > - DMA_HIGH_LIMIT,=09=09=09/* maxsize */ > > > > > > > > - DMA_TOTAL_HIGH_LIMIT,=09=09/* maxsize */ > > BUS_SPACE_UNRESTRICTED,=09=09/* nsegments */ > > > > > > > > - DMA_HIGH_LIMIT,=09=09=09/* maxsegsize */ > > > > > > > > - DMA_SEG_HIGH_LIMIT,=09=09=09/* maxsegsize */ > > 0, =09=09=09=09=09/* flags */ > > NULL, NULL,=09=09=09=09/* lockfunc, lockarg */ > > &sc->dmat); > > > > > > > > Then, with such a kernel installed, I used: > > -rw-r--r-- 1 root wheel 11570948096 Jul 18 18:32:37 2020 clang-armv7-on= -aarch64.tar > > (so much larger than the 8 GiByte RAM) and did the following with > > the file: > > > > cp -aRx clang-armv7-on-aarch64.tar clang-armv7-on-aarch64.alt_tar > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > diff clang-armv7-on-aarch64.tar clang-armv7-on-aarch64.alt_tar > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > =3D=3D > > > > Such tests have been passing. > > I do not know what your test procedures were. But if the above > > is suggestive, you might want to try testing something like the > > above via your test procedures. > > Going in a different direction for potential maxsegsz figures > (and the like): xhci instead of bcm2711 material . . . > > extensible-host-controler-interface-usb-xhci.pdf reports that > normal TRB Transfer Length fields have bits 16:0 and "Valid > values are 0 to 64K". (Transfer Event TRB Transfer Lengths > have 23:0 but above 0x10000 as a value is only for Event Data > flag being 1 or for condition code is stopped - short packet.) > > I've not experimented with this. > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar)