Date: Thu, 25 Apr 2019 19:24:59 -0400 From: Ryan Stone <rysto32@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: Tycho Nightingale <tychon@freebsd.org>, src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, Ed Maste <emaste@freebsd.org>, scottl@freebsd.org Subject: Re: svn commit: r346386 - in head/sys: dev/bge dev/pci dev/twa x86/iommu Message-ID: <CAFMmRNzouUCbU1n4xMNd4WbQP3=2wxQEHtbTG77J%2B4fKXX8EQA@mail.gmail.com> In-Reply-To: <5c43013c-1fc6-57c2-6dec-1fdfc5213fb3@FreeBSD.org> References: <201904191343.x3JDhYVF010453@repo.freebsd.org> <CAFMmRNxaJ6gWaFpxMFdYx_T2%2BjTRGrvZpvptXFgDr=Z-Zj2=Eg@mail.gmail.com> <CAFMmRNywSPckXRi0oATi3AL%2BBX2MBUQd7t36TCncWiPf06%2BbkQ@mail.gmail.com> <5c43013c-1fc6-57c2-6dec-1fdfc5213fb3@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
+scottl@, who I believe explained this to us in the first place. As I recall, it had something to do with 64-bit DMA being expressed as segment base + 32-bit offset. DMA engines that blindly try to cross a 32-bit boundary end up back at the start of the segment and read/write the wrong memory location. On Thu, Apr 25, 2019 at 4:37 PM John Baldwin <jhb@freebsd.org> wrote: > > I had looked for the aac change, but wasn't able to find it, perhaps because I > looked at tags created in aac.c rather than aac_pci.c. I agree aac will need to > be re-patched. I'm not really certain how many other devices are actually broken. > They would all be due to a firmware bug, nothing inherent in PCI. > > I believe twa(4) and bge(4) issues predated aac(4) FWIW. > > Unfortunately, the main bit of discussion about moving the limit into the PCI bus > itself seems to be an IRC discussion on 2/28/12 that resulted in revision r232267 > as a quick MFC'able fix, but I don't have a log of that conversation. :( I > couldn't find anything in e-mail either that was definitive for why this might have > been inherent in PCI-e vs a few firmware writers having similar bugs. > > On 4/25/19 12:20 PM, Ryan Stone wrote: > > Following up, this is what will have to be re-instated in the aac driver: > > > > http://svn.freebsd.org/changeset/base/232260 > > > > However, my biggest concern is that we have no idea how many new > > devices with the broken behaviour might have been introduced since we > > fixed the problem in general. How does Linux handle the issue? > > > > On Thu, Apr 25, 2019 at 3:17 PM Ryan Stone <rysto32@gmail.com> wrote: > >> > >> This change makes me *very* uncomfortable. It was originally brought > >> in due to issues with Adaptec RAID cards using the aac(9) driver. The > >> symptoms of the bug included silent corruption of data as it was > >> written to disk. Are we sure that this change is a good idea, given > >> how catastrophic it is when a device gets this wrong? > >> > >> On Fri, Apr 19, 2019 at 9:43 AM Tycho Nightingale <tychon@freebsd.org> wrote: > >>> > >>> Author: tychon > >>> Date: Fri Apr 19 13:43:33 2019 > >>> New Revision: 346386 > >>> URL: https://svnweb.freebsd.org/changeset/base/346386 > >>> > >>> Log: > >>> remove the 4GB boundary requirement on PCI DMA segments > >>> > >>> Reviewed by: kib > >>> Discussed with: jhb > >>> Sponsored by: Dell EMC Isilon > >>> Differential Revision: https://reviews.freebsd.org/D19867 > >>> > >>> Modified: > >>> head/sys/dev/bge/if_bgereg.h > >>> head/sys/dev/pci/pci.c > >>> head/sys/dev/pci/pcivar.h > >>> head/sys/dev/twa/tw_osl.h > >>> head/sys/dev/twa/tw_osl_freebsd.c > >>> head/sys/x86/iommu/intel_ctx.c > >>> > >>> Modified: head/sys/dev/bge/if_bgereg.h > >>> ============================================================================== > >>> --- head/sys/dev/bge/if_bgereg.h Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/dev/bge/if_bgereg.h Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -3067,3 +3067,11 @@ struct bge_softc { > >>> #define BGE_LOCK_ASSERT(_sc) mtx_assert(&(_sc)->bge_mtx, MA_OWNED) > >>> #define BGE_UNLOCK(_sc) mtx_unlock(&(_sc)->bge_mtx) > >>> #define BGE_LOCK_DESTROY(_sc) mtx_destroy(&(_sc)->bge_mtx) > >>> + > >>> +#ifdef BUS_SPACE_MAXADDR > >>> +#if (BUS_SPACE_MAXADDR > 0xFFFFFFFF) > >>> +#define BGE_DMA_BOUNDARY (0x100000000) > >>> +#else > >>> +#define BGE_DMA_BOUNDARY 0 > >>> +#endif > >>> +#endif > >>> > >>> Modified: head/sys/dev/pci/pci.c > >>> ============================================================================== > >>> --- head/sys/dev/pci/pci.c Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/dev/pci/pci.c Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -4343,9 +4343,6 @@ pci_attach_common(device_t dev) > >>> { > >>> struct pci_softc *sc; > >>> int busno, domain; > >>> -#ifdef PCI_DMA_BOUNDARY > >>> - int error, tag_valid; > >>> -#endif > >>> #ifdef PCI_RES_BUS > >>> int rid; > >>> #endif > >>> @@ -4365,23 +4362,7 @@ pci_attach_common(device_t dev) > >>> if (bootverbose) > >>> device_printf(dev, "domain=%d, physical bus=%d\n", > >>> domain, busno); > >>> -#ifdef PCI_DMA_BOUNDARY > >>> - tag_valid = 0; > >>> - if (device_get_devclass(device_get_parent(device_get_parent(dev))) != > >>> - devclass_find("pci")) { > >>> - error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, > >>> - PCI_DMA_BOUNDARY, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, > >>> - NULL, NULL, BUS_SPACE_MAXSIZE, BUS_SPACE_UNRESTRICTED, > >>> - BUS_SPACE_MAXSIZE, 0, NULL, NULL, &sc->sc_dma_tag); > >>> - if (error) > >>> - device_printf(dev, "Failed to create DMA tag: %d\n", > >>> - error); > >>> - else > >>> - tag_valid = 1; > >>> - } > >>> - if (!tag_valid) > >>> -#endif > >>> - sc->sc_dma_tag = bus_get_dma_tag(dev); > >>> + sc->sc_dma_tag = bus_get_dma_tag(dev); > >>> return (0); > >>> } > >>> > >>> > >>> Modified: head/sys/dev/pci/pcivar.h > >>> ============================================================================== > >>> --- head/sys/dev/pci/pcivar.h Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/dev/pci/pcivar.h Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -693,14 +693,6 @@ int pcie_link_reset(device_t port, int pcie_location); > >>> > >>> void pci_print_faulted_dev(void); > >>> > >>> -#ifdef BUS_SPACE_MAXADDR > >>> -#if (BUS_SPACE_MAXADDR > 0xFFFFFFFF) > >>> -#define PCI_DMA_BOUNDARY 0x100000000 > >>> -#else > >>> -#define PCI_DMA_BOUNDARY 0 > >>> -#endif > >>> -#endif > >>> - > >>> #endif /* _SYS_BUS_H_ */ > >>> > >>> /* > >>> > >>> Modified: head/sys/dev/twa/tw_osl.h > >>> ============================================================================== > >>> --- head/sys/dev/twa/tw_osl.h Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/dev/twa/tw_osl.h Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -57,6 +57,12 @@ > >>> #define TW_OSLI_MAX_NUM_IOS (TW_OSLI_MAX_NUM_REQUESTS - 2) > >>> #define TW_OSLI_MAX_NUM_AENS 0x100 > >>> > >>> +#ifdef PAE > >>> +#define TW_OSLI_DMA_BOUNDARY (1u << 31) > >>> +#else > >>> +#define TW_OSLI_DMA_BOUNDARY ((bus_size_t)((uint64_t)1 << 32)) > >>> +#endif > >>> + > >>> /* Possible values of req->state. */ > >>> #define TW_OSLI_REQ_STATE_INIT 0x0 /* being initialized */ > >>> #define TW_OSLI_REQ_STATE_BUSY 0x1 /* submitted to CL */ > >>> > >>> Modified: head/sys/dev/twa/tw_osl_freebsd.c > >>> ============================================================================== > >>> --- head/sys/dev/twa/tw_osl_freebsd.c Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/dev/twa/tw_osl_freebsd.c Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -551,7 +551,7 @@ tw_osli_alloc_mem(struct twa_softc *sc) > >>> /* Create the parent dma tag. */ > >>> if (bus_dma_tag_create(bus_get_dma_tag(sc->bus_dev), /* parent */ > >>> sc->alignment, /* alignment */ > >>> - 0, /* boundary */ > >>> + TW_OSLI_DMA_BOUNDARY, /* boundary */ > >>> BUS_SPACE_MAXADDR, /* lowaddr */ > >>> BUS_SPACE_MAXADDR, /* highaddr */ > >>> NULL, NULL, /* filter, filterarg */ > >>> > >>> Modified: head/sys/x86/iommu/intel_ctx.c > >>> ============================================================================== > >>> --- head/sys/x86/iommu/intel_ctx.c Fri Apr 19 13:23:41 2019 (r346385) > >>> +++ head/sys/x86/iommu/intel_ctx.c Fri Apr 19 13:43:33 2019 (r346386) > >>> @@ -130,7 +130,7 @@ ctx_tag_init(struct dmar_ctx *ctx, device_t dev) > >>> maxaddr = MIN(ctx->domain->end, BUS_SPACE_MAXADDR); > >>> ctx->ctx_tag.common.ref_count = 1; /* Prevent free */ > >>> ctx->ctx_tag.common.impl = &bus_dma_dmar_impl; > >>> - ctx->ctx_tag.common.boundary = PCI_DMA_BOUNDARY; > >>> + ctx->ctx_tag.common.boundary = 0; > >>> ctx->ctx_tag.common.lowaddr = maxaddr; > >>> ctx->ctx_tag.common.highaddr = maxaddr; > >>> ctx->ctx_tag.common.maxsize = maxaddr; > >>> > > > > > -- > John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFMmRNzouUCbU1n4xMNd4WbQP3=2wxQEHtbTG77J%2B4fKXX8EQA>