From owner-svn-src-head@FreeBSD.ORG Mon Nov 23 13:38:39 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37BBD106568B; Mon, 23 Nov 2009 13:38:39 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id EF79C8FC2F; Mon, 23 Nov 2009 13:38:38 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 91C8B46B2C; Mon, 23 Nov 2009 08:38:38 -0500 (EST) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id C652B8A01B; Mon, 23 Nov 2009 08:38:37 -0500 (EST) From: John Baldwin To: Scott Long Date: Mon, 23 Nov 2009 08:38:19 -0500 User-Agent: KMail/1.9.7 References: <200911222050.nAMKoRYh029141@svn.freebsd.org> <0298EC7E-26D5-460A-9EBE-5ABEAE8B21BD@samsco.org> In-Reply-To: <0298EC7E-26D5-460A-9EBE-5ABEAE8B21BD@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200911230838.20217.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Mon, 23 Nov 2009 08:38:37 -0500 (EST) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Pyun YongHyeon Subject: Re: svn commit: r199670 - head/sys/dev/bge X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2009 13:38:39 -0000 On Sunday 22 November 2009 6:48:18 pm Scott Long wrote: > By definition, PCIe can't transfer across 4GB boundaries. It's not a > bug specific to Broadcom. If you're loading dynamic buffers (i.e. > mbufs), setting an appropriate boundary value in the tag will allow > busdma to take care of this. If you're allocating static buffers, > busdma won't honor this. But what you've done here is best anyways; > control buffers that are going to be frequently transferred are best > kept in the lower 4GB of the address space. It simplifies PCIe > handling, and it's significantly faster on PCI/PCI-X. So I'd suggest > making this the rule rather than the exception in the driver. Should we enforce an implicit 4GB boundary in bus_dma then? Perhaps Host-PCI bridge drivers should create a tag with a 4GB boundary that devices inherit via bus_get_dma_tag(). For i386/PAE we might should always enforce a 4GB boundary as well? > Scott > > On Nov 22, 2009, at 1:50 PM, Pyun YongHyeon wrote: > > > Author: yongari > > Date: Sun Nov 22 20:50:27 2009 > > New Revision: 199670 > > URL: http://svn.freebsd.org/changeset/base/199670 > > > > Log: > > Fix two long standing bugs on bge(4). Most pre BCM5755 controllers > > have a DMA bug when buffer address crosses a multiple of the 4GB > > boundary(e.g. 4GB, 8GB, 12GB etc). Limit DMA address to be within > > 4GB address for these controllers. The second DMA bug limits DMA > > address to be within 40bit address space. This bug applies to > > BCM5714 and BCM5715 and 5708(bce(4) controller). This is not > > actually a MAC controller bug but an issue with the embedded PCIe > > to PCI-X bridge in the device. So for BCM5714/BCM5715 controllers > > also limit the DMA address to be within 40bit address space. > > Special thanks to davidch@ who gave me detailed errata information. > > I think this change will fix long standing bge(4) instability > > issues on systems with more than 4GB memory. > > > > Reviewed by: davidch > > > > Modified: > > head/sys/dev/bge/if_bge.c > > head/sys/dev/bge/if_bgereg.h > > > > Modified: head/sys/dev/bge/if_bge.c > > = > > = > > = > > = > > = > > = > > = > > = > > ====================================================================== > > --- head/sys/dev/bge/if_bge.c Sun Nov 22 20:45:15 2009 (r199669) > > +++ head/sys/dev/bge/if_bge.c Sun Nov 22 20:50:27 2009 (r199670) > > @@ -2104,15 +2104,21 @@ bge_dma_alloc(device_t dev) > > { > > struct bge_dmamap_arg ctx; > > struct bge_softc *sc; > > + bus_addr_t lowaddr; > > int i, error; > > > > sc = device_get_softc(dev); > > > > + lowaddr = BUS_SPACE_MAXADDR; > > + if ((sc->bge_flags & BGE_FLAG_40BIT_BUG) != 0) > > + lowaddr = BGE_DMA_MAXADDR; > > + if ((sc->bge_flags & BGE_FLAG_4G_BNDRY_BUG) != 0) > > + lowaddr = BUS_SPACE_MAXADDR_32BIT; > > /* > > * Allocate the parent bus DMA tag appropriate for PCI. > > */ > > error = bus_dma_tag_create(bus_get_dma_tag(sc->bge_dev), > > - 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, > > + 1, 0, lowaddr, BUS_SPACE_MAXADDR, NULL, > > NULL, BUS_SPACE_MAXSIZE_32BIT, 0, BUS_SPACE_MAXSIZE_32BIT, > > 0, NULL, NULL, &sc->bge_cdata.bge_parent_tag); > > > > @@ -2566,6 +2572,16 @@ bge_attach(device_t dev) > > sc->bge_flags |= BGE_FLAG_BER_BUG; > > } > > > > + /* > > + * All controllers that are not 5755 or higher have 4GB > > + * boundary DMA bug. > > + * Whenever an address crosses a multiple of the 4GB boundary > > + * (including 4GB, 8Gb, 12Gb, etc.) and makes the transition > > + * from 0xX_FFFF_FFFF to 0x(X+1)_0000_0000 an internal DMA > > + * state machine will lockup and cause the device to hang. > > + */ > > + if (BGE_IS_5755_PLUS(sc) == 0) > > + sc->bge_flags |= BGE_FLAG_4G_BNDRY_BUG; > > > > /* > > * We could possibly check for BCOM_DEVICEID_BCM5788 in bge_probe() > > @@ -2729,6 +2745,13 @@ bge_attach(device_t dev) > > #ifdef DEVICE_POLLING > > ifp->if_capabilities |= IFCAP_POLLING; > > #endif > > + /* > > + * The 40bit DMA bug applies to the 5714/5715 controllers and is > > + * not actually a MAC controller bug but an issue with the embedded > > + * PCIe to PCI-X bridge in the device. Use 40bit DMA workaround. > > + */ > > + if (BGE_IS_5714_FAMILY(sc) && (sc->bge_flags & BGE_FLAG_PCIX)) > > + sc->bge_flags |= BGE_FLAG_40BIT_BUG; > > > > /* > > * 5700 B0 chips do not support checksumming correctly due > > > > Modified: head/sys/dev/bge/if_bgereg.h > > = > > = > > = > > = > > = > > = > > = > > = > > ====================================================================== > > --- head/sys/dev/bge/if_bgereg.h Sun Nov 22 20:45:15 2009 (r199669) > > +++ head/sys/dev/bge/if_bgereg.h Sun Nov 22 20:50:27 2009 (r199670) > > @@ -2484,6 +2484,13 @@ struct bge_gib { > > #define BGE_NSEG_JUMBO 4 > > #define BGE_NSEG_NEW 32 > > > > +/* Maximum DMA address for controllers that have 40bit DMA address > > bug. */ > > +#if (BUS_SPACE_MAXADDR < 0xFFFFFFFFFF) > > +#define BGE_DMA_MAXADDR BUS_SPACE_MAXADDR > > +#else > > +#define BGE_DMA_MAXADDR 0xFFFFFFFFFF > > +#endif > > + > > /* > > * Ring structures. Most of these reside in host memory and we tell > > * the NIC where they are via the ring control blocks. The exceptions > > @@ -2600,6 +2607,8 @@ struct bge_softc { > > #define BGE_FLAG_5714_FAMILY 0x00004000 > > #define BGE_FLAG_575X_PLUS 0x00008000 > > #define BGE_FLAG_5755_PLUS 0x00010000 > > +#define BGE_FLAG_40BIT_BUG 0x00020000 > > +#define BGE_FLAG_4G_BNDRY_BUG 0x00040000 > > #define BGE_FLAG_RX_ALIGNBUG 0x00100000 > > #define BGE_FLAG_NO_3LED 0x00200000 > > #define BGE_FLAG_ADC_BUG 0x00400000 > > -- John Baldwin