Date: Mon, 14 Dec 2009 18:44:24 +0000 (UTC) From: Pyun YongHyeon <yongari@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r200525 - head/sys/dev/vge Message-ID: <200912141844.nBEIiOZa034749@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: yongari Date: Mon Dec 14 18:44:23 2009 New Revision: 200525 URL: http://svn.freebsd.org/changeset/base/200525 Log: Overhaul bus_dma(9) usage and fix various things. o Separate TX/RX buffer DMA tag from TX/RX descriptor ring DMA tag. o Separate RX buffer DMA tag from common buffer DMA tag. RX DMA tag has different restriction compared to TX DMA tag. o Add 40bit DMA address support. o Adjust TX/RX descriptor ring alignment to 64 bytes from 256 bytes as documented in datasheet. o Added check to ensure TX/RX ring reside within a 4GB boundary. Since TX/RX ring shares the same high address register they should have the same high address. o TX/RX side bus_dmamap_load_mbuf_sg(9) support. o Add lock assertion to vge_setmulti(). o Add RX spare DMA map to recover from DMA map load failure. o Add optimized RX buffer handler, vge_discard_rxbuf which is activated when vge(4) sees bad frames. o Don't blindly update VGE_RXDESC_RESIDUECNT register. Datasheet says the register should be updated only when number of available RX descriptors are multiple of 4. o Use __NO_STRICT_ALIGNMENT instead of defining VGE_FIXUP_RX which is only set for i386 architecture. Previously vge(4) also performed expensive copy operation to align IP header on amd64. This change should give RX performance boost on amd64 architecture. o Don't reinitialize controller if driver is already running. This should reduce number of link state flipping. o Since vge(4) drops a driver lock before passing received frame to upper layer, make sure vge(4) is still running after re-acquiring driver lock. o Add second argument count to vge_rxeof(). The argument will limit number of packets could be processed in RX handler. o Rearrange vge_rxeof() not to allocate RX buffer if received frame was bad packet. o Removed if_printf that prints DMA map failure. This type of message shouldn't be used in fast path of driver. o Reduce number of allowed TX buffer fragments to 6 from 7. A TX descriptor allows 7 fragments of a frame. However the CMZ field of descriptor has just 3bits and the controller wants to see fragment + 1 in the field. So if we have 7 fragments the field value would be 0 which seems to cause unexpected results under certain conditions. This change should fix occasional TX hang observed on vge(4). o Simplify vge_stat_locked() and add number of available TX descriptor check. o vge(4) controllers lack padding short frames. Make sure to fill zero for the padded bytes. This closes unintended information disclosure. o Don't set VGE_TDCTL_JUMBO flag. Datasheet is not clear whether this bit should be set by driver or write-back status bit after transmission. At least vendor's driver does not set this bit so remove it. Without this bit vge(4) still can send jumbo frames. o Don't start driver when vge(4) know there are not enough RX buffers. o Remove volatile keyword in RX descriptor structure. This should be handled by bus_dma(9). o Collapse two 16bits member of TX/RX descriptor into single 32bits member. o Reduce number of RX descriptors to 252 from 256. The VGE_RXDESCNUM is 16bits register but only lower 8bits are valid. So the maximum number of RX descriptors would be 255. However the number of should be multiple of 4 as controller wants to update 4 RX descriptors at a time. This limits the maximum number of RX descriptor to be 252. Tested by: Dewayne Geraghty (dewayne.geraghty <> heuristicsystems dot com dot au) Carey Jones (m.carey.jones <> gmail dot com) Yoshiaki Kasahara (kasahara <> nc dor kyushu-u dot ac dotjp) Modified: head/sys/dev/vge/if_vge.c head/sys/dev/vge/if_vgereg.h head/sys/dev/vge/if_vgevar.h Modified: head/sys/dev/vge/if_vge.c ============================================================================== --- head/sys/dev/vge/if_vge.c Mon Dec 14 18:43:27 2009 (r200524) +++ head/sys/dev/vge/if_vge.c Mon Dec 14 18:44:23 2009 (r200525) @@ -140,22 +140,21 @@ static int vge_probe (device_t); static int vge_attach (device_t); static int vge_detach (device_t); -static int vge_encap (struct vge_softc *, struct mbuf *, int); +static int vge_encap (struct vge_softc *, struct mbuf **); -static void vge_dma_map_addr (void *, bus_dma_segment_t *, int, int); -static void vge_dma_map_rx_desc (void *, bus_dma_segment_t *, int, - bus_size_t, int); -static void vge_dma_map_tx_desc (void *, bus_dma_segment_t *, int, - bus_size_t, int); -static int vge_allocmem (device_t, struct vge_softc *); -static int vge_newbuf (struct vge_softc *, int, struct mbuf *); +static void vge_dmamap_cb (void *, bus_dma_segment_t *, int, int); +static int vge_dma_alloc (struct vge_softc *); +static void vge_dma_free (struct vge_softc *); +static void vge_discard_rxbuf (struct vge_softc *, int); +static int vge_newbuf (struct vge_softc *, int); static int vge_rx_list_init (struct vge_softc *); static int vge_tx_list_init (struct vge_softc *); -#ifdef VGE_FIXUP_RX +static void vge_freebufs (struct vge_softc *); +#ifndef __NO_STRICT_ALIGNMENT static __inline void vge_fixup_rx (struct mbuf *); #endif -static int vge_rxeof (struct vge_softc *); +static int vge_rxeof (struct vge_softc *, int); static void vge_txeof (struct vge_softc *); static void vge_intr (void *); static void vge_tick (void *); @@ -547,6 +546,8 @@ vge_setmulti(sc) struct ifmultiaddr *ifma; u_int32_t h, hashes[2] = { 0, 0 }; + VGE_LOCK_ASSERT(sc); + ifp = sc->vge_ifp; /* First, zot all the multicast entries. */ @@ -662,249 +663,326 @@ vge_probe(dev) return (ENXIO); } -static void -vge_dma_map_rx_desc(arg, segs, nseg, mapsize, error) - void *arg; - bus_dma_segment_t *segs; - int nseg; - bus_size_t mapsize; - int error; -{ - - struct vge_dmaload_arg *ctx; - struct vge_rx_desc *d = NULL; - - if (error) - return; - - ctx = arg; - - /* Signal error to caller if there's too many segments */ - if (nseg > ctx->vge_maxsegs) { - ctx->vge_maxsegs = 0; - return; - } - - /* - * Map the segment array into descriptors. - */ - - d = &ctx->sc->vge_ldata.vge_rx_list[ctx->vge_idx]; - - /* If this descriptor is still owned by the chip, bail. */ - - if (le32toh(d->vge_sts) & VGE_RDSTS_OWN) { - device_printf(ctx->sc->vge_dev, - "tried to map busy descriptor\n"); - ctx->vge_maxsegs = 0; - return; - } - - d->vge_buflen = htole16(VGE_BUFLEN(segs[0].ds_len) | VGE_RXDESC_I); - d->vge_addrlo = htole32(VGE_ADDR_LO(segs[0].ds_addr)); - d->vge_addrhi = htole16(VGE_ADDR_HI(segs[0].ds_addr) & 0xFFFF); - d->vge_sts = 0; - d->vge_ctl = 0; - - ctx->vge_maxsegs = 1; - - return; -} - -static void -vge_dma_map_tx_desc(arg, segs, nseg, mapsize, error) - void *arg; - bus_dma_segment_t *segs; - int nseg; - bus_size_t mapsize; - int error; -{ - struct vge_dmaload_arg *ctx; - struct vge_tx_desc *d = NULL; - struct vge_tx_frag *f; - int i = 0; - - if (error) - return; - - ctx = arg; - - /* Signal error to caller if there's too many segments */ - if (nseg > ctx->vge_maxsegs) { - ctx->vge_maxsegs = 0; - return; - } - - /* Map the segment array into descriptors. */ - - d = &ctx->sc->vge_ldata.vge_tx_list[ctx->vge_idx]; - - /* If this descriptor is still owned by the chip, bail. */ - - if (le32toh(d->vge_sts) & VGE_TDSTS_OWN) { - ctx->vge_maxsegs = 0; - return; - } - - for (i = 0; i < nseg; i++) { - f = &d->vge_frag[i]; - f->vge_buflen = htole16(VGE_BUFLEN(segs[i].ds_len)); - f->vge_addrlo = htole32(VGE_ADDR_LO(segs[i].ds_addr)); - f->vge_addrhi = htole16(VGE_ADDR_HI(segs[i].ds_addr) & 0xFFFF); - } - - /* Argh. This chip does not autopad short frames */ - - if (ctx->vge_m0->m_pkthdr.len < VGE_MIN_FRAMELEN) { - f = &d->vge_frag[i]; - f->vge_buflen = htole16(VGE_BUFLEN(VGE_MIN_FRAMELEN - - ctx->vge_m0->m_pkthdr.len)); - f->vge_addrlo = htole32(VGE_ADDR_LO(segs[0].ds_addr)); - f->vge_addrhi = htole16(VGE_ADDR_HI(segs[0].ds_addr) & 0xFFFF); - ctx->vge_m0->m_pkthdr.len = VGE_MIN_FRAMELEN; - i++; - } - - /* - * When telling the chip how many segments there are, we - * must use nsegs + 1 instead of just nsegs. Darned if I - * know why. - */ - i++; - - d->vge_sts = ctx->vge_m0->m_pkthdr.len << 16; - d->vge_ctl = ctx->vge_flags|(i << 28)|VGE_TD_LS_NORM; - - if (ctx->vge_m0->m_pkthdr.len > ETHERMTU + ETHER_HDR_LEN) - d->vge_ctl |= VGE_TDCTL_JUMBO; - - ctx->vge_maxsegs = nseg; - - return; -} - /* * Map a single buffer address. */ +struct vge_dmamap_arg { + bus_addr_t vge_busaddr; +}; + static void -vge_dma_map_addr(arg, segs, nseg, error) +vge_dmamap_cb(arg, segs, nsegs, error) void *arg; bus_dma_segment_t *segs; - int nseg; + int nsegs; int error; { - bus_addr_t *addr; + struct vge_dmamap_arg *ctx; - if (error) + if (error != 0) return; - KASSERT(nseg == 1, ("too many DMA segments, %d should be 1", nseg)); - addr = arg; - *addr = segs->ds_addr; + KASSERT(nsegs == 1, ("%s: %d segments returned!", __func__, nsegs)); - return; + ctx = (struct vge_dmamap_arg *)arg; + ctx->vge_busaddr = segs[0].ds_addr; } static int -vge_allocmem(dev, sc) - device_t dev; - struct vge_softc *sc; +vge_dma_alloc(sc) + struct vge_softc *sc; { - int error; - int nseg; - int i; - - /* - * Allocate map for RX mbufs. - */ - nseg = 32; - error = bus_dma_tag_create(sc->vge_parent_tag, ETHER_ALIGN, 0, - BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, - NULL, MCLBYTES * nseg, nseg, MCLBYTES, BUS_DMA_ALLOCNOW, - NULL, NULL, &sc->vge_ldata.vge_mtag); - if (error) { - device_printf(dev, "could not allocate dma tag\n"); - return (ENOMEM); + struct vge_dmamap_arg ctx; + struct vge_txdesc *txd; + struct vge_rxdesc *rxd; + bus_addr_t lowaddr, tx_ring_end, rx_ring_end; + int error, i; + + lowaddr = BUS_SPACE_MAXADDR; + +again: + /* Create parent ring tag. */ + error = bus_dma_tag_create(bus_get_dma_tag(sc->vge_dev),/* parent */ + 1, 0, /* algnmnt, boundary */ + lowaddr, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + BUS_SPACE_MAXSIZE_32BIT, /* maxsize */ + 0, /* nsegments */ + BUS_SPACE_MAXSIZE_32BIT, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_ring_tag); + if (error != 0) { + device_printf(sc->vge_dev, + "could not create parent DMA tag.\n"); + goto fail; } - /* - * Allocate map for TX descriptor list. - */ - error = bus_dma_tag_create(sc->vge_parent_tag, VGE_RING_ALIGN, - 0, BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, - NULL, VGE_TX_LIST_SZ, 1, VGE_TX_LIST_SZ, BUS_DMA_ALLOCNOW, - NULL, NULL, &sc->vge_ldata.vge_tx_list_tag); - if (error) { - device_printf(dev, "could not allocate dma tag\n"); - return (ENOMEM); + /* Create tag for Tx ring. */ + error = bus_dma_tag_create(sc->vge_cdata.vge_ring_tag,/* parent */ + VGE_TX_RING_ALIGN, 0, /* algnmnt, boundary */ + BUS_SPACE_MAXADDR, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + VGE_TX_LIST_SZ, /* maxsize */ + 1, /* nsegments */ + VGE_TX_LIST_SZ, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_tx_ring_tag); + if (error != 0) { + device_printf(sc->vge_dev, + "could not allocate Tx ring DMA tag.\n"); + goto fail; } - /* Allocate DMA'able memory for the TX ring */ - - error = bus_dmamem_alloc(sc->vge_ldata.vge_tx_list_tag, - (void **)&sc->vge_ldata.vge_tx_list, BUS_DMA_NOWAIT | BUS_DMA_ZERO, - &sc->vge_ldata.vge_tx_list_map); - if (error) - return (ENOMEM); + /* Create tag for Rx ring. */ + error = bus_dma_tag_create(sc->vge_cdata.vge_ring_tag,/* parent */ + VGE_RX_RING_ALIGN, 0, /* algnmnt, boundary */ + BUS_SPACE_MAXADDR, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + VGE_RX_LIST_SZ, /* maxsize */ + 1, /* nsegments */ + VGE_RX_LIST_SZ, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_rx_ring_tag); + if (error != 0) { + device_printf(sc->vge_dev, + "could not allocate Rx ring DMA tag.\n"); + goto fail; + } - /* Load the map for the TX ring. */ + /* Allocate DMA'able memory and load the DMA map for Tx ring. */ + error = bus_dmamem_alloc(sc->vge_cdata.vge_tx_ring_tag, + (void **)&sc->vge_rdata.vge_tx_ring, + BUS_DMA_WAITOK | BUS_DMA_ZERO | BUS_DMA_COHERENT, + &sc->vge_cdata.vge_tx_ring_map); + if (error != 0) { + device_printf(sc->vge_dev, + "could not allocate DMA'able memory for Tx ring.\n"); + goto fail; + } - error = bus_dmamap_load(sc->vge_ldata.vge_tx_list_tag, - sc->vge_ldata.vge_tx_list_map, sc->vge_ldata.vge_tx_list, - VGE_TX_LIST_SZ, vge_dma_map_addr, - &sc->vge_ldata.vge_tx_list_addr, BUS_DMA_NOWAIT); + ctx.vge_busaddr = 0; + error = bus_dmamap_load(sc->vge_cdata.vge_tx_ring_tag, + sc->vge_cdata.vge_tx_ring_map, sc->vge_rdata.vge_tx_ring, + VGE_TX_LIST_SZ, vge_dmamap_cb, &ctx, BUS_DMA_NOWAIT); + if (error != 0 || ctx.vge_busaddr == 0) { + device_printf(sc->vge_dev, + "could not load DMA'able memory for Tx ring.\n"); + goto fail; + } + sc->vge_rdata.vge_tx_ring_paddr = ctx.vge_busaddr; - /* Create DMA maps for TX buffers */ + /* Allocate DMA'able memory and load the DMA map for Rx ring. */ + error = bus_dmamem_alloc(sc->vge_cdata.vge_rx_ring_tag, + (void **)&sc->vge_rdata.vge_rx_ring, + BUS_DMA_WAITOK | BUS_DMA_ZERO | BUS_DMA_COHERENT, + &sc->vge_cdata.vge_rx_ring_map); + if (error != 0) { + device_printf(sc->vge_dev, + "could not allocate DMA'able memory for Rx ring.\n"); + goto fail; + } - for (i = 0; i < VGE_TX_DESC_CNT; i++) { - error = bus_dmamap_create(sc->vge_ldata.vge_mtag, 0, - &sc->vge_ldata.vge_tx_dmamap[i]); - if (error) { - device_printf(dev, "can't create DMA map for TX\n"); - return (ENOMEM); - } + ctx.vge_busaddr = 0; + error = bus_dmamap_load(sc->vge_cdata.vge_rx_ring_tag, + sc->vge_cdata.vge_rx_ring_map, sc->vge_rdata.vge_rx_ring, + VGE_RX_LIST_SZ, vge_dmamap_cb, &ctx, BUS_DMA_NOWAIT); + if (error != 0 || ctx.vge_busaddr == 0) { + device_printf(sc->vge_dev, + "could not load DMA'able memory for Rx ring.\n"); + goto fail; } + sc->vge_rdata.vge_rx_ring_paddr = ctx.vge_busaddr; - /* - * Allocate map for RX descriptor list. - */ - error = bus_dma_tag_create(sc->vge_parent_tag, VGE_RING_ALIGN, - 0, BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, - NULL, VGE_TX_LIST_SZ, 1, VGE_TX_LIST_SZ, BUS_DMA_ALLOCNOW, - NULL, NULL, &sc->vge_ldata.vge_rx_list_tag); - if (error) { - device_printf(dev, "could not allocate dma tag\n"); - return (ENOMEM); + /* Tx/Rx descriptor queue should reside within 4GB boundary. */ + tx_ring_end = sc->vge_rdata.vge_tx_ring_paddr + VGE_TX_LIST_SZ; + rx_ring_end = sc->vge_rdata.vge_rx_ring_paddr + VGE_RX_LIST_SZ; + if ((VGE_ADDR_HI(tx_ring_end) != + VGE_ADDR_HI(sc->vge_rdata.vge_tx_ring_paddr)) || + (VGE_ADDR_HI(rx_ring_end) != + VGE_ADDR_HI(sc->vge_rdata.vge_rx_ring_paddr)) || + VGE_ADDR_HI(tx_ring_end) != VGE_ADDR_HI(rx_ring_end)) { + device_printf(sc->vge_dev, "4GB boundary crossed, " + "switching to 32bit DMA address mode.\n"); + vge_dma_free(sc); + /* Limit DMA address space to 32bit and try again. */ + lowaddr = BUS_SPACE_MAXADDR_32BIT; + goto again; + } + + /* Create parent buffer tag. */ + error = bus_dma_tag_create(bus_get_dma_tag(sc->vge_dev),/* parent */ + 1, 0, /* algnmnt, boundary */ + VGE_BUF_DMA_MAXADDR, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + BUS_SPACE_MAXSIZE_32BIT, /* maxsize */ + 0, /* nsegments */ + BUS_SPACE_MAXSIZE_32BIT, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_buffer_tag); + if (error != 0) { + device_printf(sc->vge_dev, + "could not create parent buffer DMA tag.\n"); + goto fail; } - /* Allocate DMA'able memory for the RX ring */ + /* Create tag for Tx buffers. */ + error = bus_dma_tag_create(sc->vge_cdata.vge_buffer_tag,/* parent */ + 1, 0, /* algnmnt, boundary */ + BUS_SPACE_MAXADDR, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + MCLBYTES * VGE_MAXTXSEGS, /* maxsize */ + VGE_MAXTXSEGS, /* nsegments */ + MCLBYTES, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_tx_tag); + if (error != 0) { + device_printf(sc->vge_dev, "could not create Tx DMA tag.\n"); + goto fail; + } - error = bus_dmamem_alloc(sc->vge_ldata.vge_rx_list_tag, - (void **)&sc->vge_ldata.vge_rx_list, BUS_DMA_NOWAIT | BUS_DMA_ZERO, - &sc->vge_ldata.vge_rx_list_map); - if (error) - return (ENOMEM); + /* Create tag for Rx buffers. */ + error = bus_dma_tag_create(sc->vge_cdata.vge_buffer_tag,/* parent */ + VGE_RX_BUF_ALIGN, 0, /* algnmnt, boundary */ + BUS_SPACE_MAXADDR, /* lowaddr */ + BUS_SPACE_MAXADDR, /* highaddr */ + NULL, NULL, /* filter, filterarg */ + MCLBYTES, /* maxsize */ + 1, /* nsegments */ + MCLBYTES, /* maxsegsize */ + 0, /* flags */ + NULL, NULL, /* lockfunc, lockarg */ + &sc->vge_cdata.vge_rx_tag); + if (error != 0) { + device_printf(sc->vge_dev, "could not create Rx DMA tag.\n"); + goto fail; + } - /* Load the map for the RX ring. */ + /* Create DMA maps for Tx buffers. */ + for (i = 0; i < VGE_TX_DESC_CNT; i++) { + txd = &sc->vge_cdata.vge_txdesc[i]; + txd->tx_m = NULL; + txd->tx_dmamap = NULL; + error = bus_dmamap_create(sc->vge_cdata.vge_tx_tag, 0, + &txd->tx_dmamap); + if (error != 0) { + device_printf(sc->vge_dev, + "could not create Tx dmamap.\n"); + goto fail; + } + } + /* Create DMA maps for Rx buffers. */ + if ((error = bus_dmamap_create(sc->vge_cdata.vge_rx_tag, 0, + &sc->vge_cdata.vge_rx_sparemap)) != 0) { + device_printf(sc->vge_dev, + "could not create spare Rx dmamap.\n"); + goto fail; + } + for (i = 0; i < VGE_RX_DESC_CNT; i++) { + rxd = &sc->vge_cdata.vge_rxdesc[i]; + rxd->rx_m = NULL; + rxd->rx_dmamap = NULL; + error = bus_dmamap_create(sc->vge_cdata.vge_rx_tag, 0, + &rxd->rx_dmamap); + if (error != 0) { + device_printf(sc->vge_dev, + "could not create Rx dmamap.\n"); + goto fail; + } + } - error = bus_dmamap_load(sc->vge_ldata.vge_rx_list_tag, - sc->vge_ldata.vge_rx_list_map, sc->vge_ldata.vge_rx_list, - VGE_TX_LIST_SZ, vge_dma_map_addr, - &sc->vge_ldata.vge_rx_list_addr, BUS_DMA_NOWAIT); +fail: + return (error); +} - /* Create DMA maps for RX buffers */ +static void +vge_dma_free(sc) + struct vge_softc *sc; +{ + struct vge_txdesc *txd; + struct vge_rxdesc *rxd; + int i; - for (i = 0; i < VGE_RX_DESC_CNT; i++) { - error = bus_dmamap_create(sc->vge_ldata.vge_mtag, 0, - &sc->vge_ldata.vge_rx_dmamap[i]); - if (error) { - device_printf(dev, "can't create DMA map for RX\n"); - return (ENOMEM); + /* Tx ring. */ + if (sc->vge_cdata.vge_tx_ring_tag != NULL) { + if (sc->vge_cdata.vge_tx_ring_map) + bus_dmamap_unload(sc->vge_cdata.vge_tx_ring_tag, + sc->vge_cdata.vge_tx_ring_map); + if (sc->vge_cdata.vge_tx_ring_map && + sc->vge_rdata.vge_tx_ring) + bus_dmamem_free(sc->vge_cdata.vge_tx_ring_tag, + sc->vge_rdata.vge_tx_ring, + sc->vge_cdata.vge_tx_ring_map); + sc->vge_rdata.vge_tx_ring = NULL; + sc->vge_cdata.vge_tx_ring_map = NULL; + bus_dma_tag_destroy(sc->vge_cdata.vge_tx_ring_tag); + sc->vge_cdata.vge_tx_ring_tag = NULL; + } + /* Rx ring. */ + if (sc->vge_cdata.vge_rx_ring_tag != NULL) { + if (sc->vge_cdata.vge_rx_ring_map) + bus_dmamap_unload(sc->vge_cdata.vge_rx_ring_tag, + sc->vge_cdata.vge_rx_ring_map); + if (sc->vge_cdata.vge_rx_ring_map && + sc->vge_rdata.vge_rx_ring) + bus_dmamem_free(sc->vge_cdata.vge_rx_ring_tag, + sc->vge_rdata.vge_rx_ring, + sc->vge_cdata.vge_rx_ring_map); + sc->vge_rdata.vge_rx_ring = NULL; + sc->vge_cdata.vge_rx_ring_map = NULL; + bus_dma_tag_destroy(sc->vge_cdata.vge_rx_ring_tag); + sc->vge_cdata.vge_rx_ring_tag = NULL; + } + /* Tx buffers. */ + if (sc->vge_cdata.vge_tx_tag != NULL) { + for (i = 0; i < VGE_TX_DESC_CNT; i++) { + txd = &sc->vge_cdata.vge_txdesc[i]; + if (txd->tx_dmamap != NULL) { + bus_dmamap_destroy(sc->vge_cdata.vge_tx_tag, + txd->tx_dmamap); + txd->tx_dmamap = NULL; + } } + bus_dma_tag_destroy(sc->vge_cdata.vge_tx_tag); + sc->vge_cdata.vge_tx_tag = NULL; + } + /* Rx buffers. */ + if (sc->vge_cdata.vge_rx_tag != NULL) { + for (i = 0; i < VGE_RX_DESC_CNT; i++) { + rxd = &sc->vge_cdata.vge_rxdesc[i]; + if (rxd->rx_dmamap != NULL) { + bus_dmamap_destroy(sc->vge_cdata.vge_rx_tag, + rxd->rx_dmamap); + rxd->rx_dmamap = NULL; + } + } + if (sc->vge_cdata.vge_rx_sparemap != NULL) { + bus_dmamap_destroy(sc->vge_cdata.vge_rx_tag, + sc->vge_cdata.vge_rx_sparemap); + sc->vge_cdata.vge_rx_sparemap = NULL; + } + bus_dma_tag_destroy(sc->vge_cdata.vge_rx_tag); + sc->vge_cdata.vge_rx_tag = NULL; + } + + if (sc->vge_cdata.vge_buffer_tag != NULL) { + bus_dma_tag_destroy(sc->vge_cdata.vge_buffer_tag); + sc->vge_cdata.vge_buffer_tag = NULL; + } + if (sc->vge_cdata.vge_ring_tag != NULL) { + bus_dma_tag_destroy(sc->vge_cdata.vge_ring_tag); + sc->vge_cdata.vge_ring_tag = NULL; } - - return (0); } /* @@ -961,25 +1039,7 @@ vge_attach(dev) */ vge_read_eeprom(sc, (caddr_t)eaddr, VGE_EE_EADDR, 3, 0); - /* - * Allocate the parent bus DMA tag appropriate for PCI. - */ -#define VGE_NSEG_NEW 32 - error = bus_dma_tag_create(NULL, /* parent */ - 1, 0, /* alignment, boundary */ - BUS_SPACE_MAXADDR_32BIT,/* lowaddr */ - BUS_SPACE_MAXADDR, /* highaddr */ - NULL, NULL, /* filter, filterarg */ - MAXBSIZE, VGE_NSEG_NEW, /* maxsize, nsegments */ - BUS_SPACE_MAXSIZE_32BIT,/* maxsegsize */ - BUS_DMA_ALLOCNOW, /* flags */ - NULL, NULL, /* lockfunc, lockarg */ - &sc->vge_parent_tag); - if (error) - goto fail; - - error = vge_allocmem(dev, sc); - + error = vge_dma_alloc(sc); if (error) goto fail; @@ -1051,7 +1111,6 @@ vge_detach(dev) { struct vge_softc *sc; struct ifnet *ifp; - int i; sc = device_get_softc(dev); KASSERT(mtx_initialized(&sc->vge_mtx), ("vge mutex not initialized")); @@ -1084,97 +1143,93 @@ vge_detach(dev) if (ifp) if_free(ifp); - /* Unload and free the RX DMA ring memory and map */ + vge_dma_free(sc); + mtx_destroy(&sc->vge_mtx); - if (sc->vge_ldata.vge_rx_list_tag) { - bus_dmamap_unload(sc->vge_ldata.vge_rx_list_tag, - sc->vge_ldata.vge_rx_list_map); - bus_dmamem_free(sc->vge_ldata.vge_rx_list_tag, - sc->vge_ldata.vge_rx_list, - sc->vge_ldata.vge_rx_list_map); - bus_dma_tag_destroy(sc->vge_ldata.vge_rx_list_tag); - } - - /* Unload and free the TX DMA ring memory and map */ - - if (sc->vge_ldata.vge_tx_list_tag) { - bus_dmamap_unload(sc->vge_ldata.vge_tx_list_tag, - sc->vge_ldata.vge_tx_list_map); - bus_dmamem_free(sc->vge_ldata.vge_tx_list_tag, - sc->vge_ldata.vge_tx_list, - sc->vge_ldata.vge_tx_list_map); - bus_dma_tag_destroy(sc->vge_ldata.vge_tx_list_tag); - } - - /* Destroy all the RX and TX buffer maps */ - - if (sc->vge_ldata.vge_mtag) { - for (i = 0; i < VGE_TX_DESC_CNT; i++) - bus_dmamap_destroy(sc->vge_ldata.vge_mtag, - sc->vge_ldata.vge_tx_dmamap[i]); - for (i = 0; i < VGE_RX_DESC_CNT; i++) - bus_dmamap_destroy(sc->vge_ldata.vge_mtag, - sc->vge_ldata.vge_rx_dmamap[i]); - bus_dma_tag_destroy(sc->vge_ldata.vge_mtag); - } + return (0); +} - if (sc->vge_parent_tag) - bus_dma_tag_destroy(sc->vge_parent_tag); +static void +vge_discard_rxbuf(sc, prod) + struct vge_softc *sc; + int prod; +{ + struct vge_rxdesc *rxd; + int i; - mtx_destroy(&sc->vge_mtx); + rxd = &sc->vge_cdata.vge_rxdesc[prod]; + rxd->rx_desc->vge_sts = 0; + rxd->rx_desc->vge_ctl = 0; - return (0); + /* + * Note: the manual fails to document the fact that for + * proper opration, the driver needs to replentish the RX + * DMA ring 4 descriptors at a time (rather than one at a + * time, like most chips). We can allocate the new buffers + * but we should not set the OWN bits until we're ready + * to hand back 4 of them in one shot. + */ + if ((prod % VGE_RXCHUNK) == (VGE_RXCHUNK - 1)) { + for (i = VGE_RXCHUNK; i > 0; i--) { + rxd->rx_desc->vge_sts = htole32(VGE_RDSTS_OWN); + rxd = rxd->rxd_prev; + } + sc->vge_cdata.vge_rx_commit += VGE_RXCHUNK; + } } static int -vge_newbuf(sc, idx, m) +vge_newbuf(sc, prod) struct vge_softc *sc; - int idx; - struct mbuf *m; + int prod; { - struct vge_dmaload_arg arg; - struct mbuf *n = NULL; - int i, error; - - if (m == NULL) { - n = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); - if (n == NULL) - return (ENOBUFS); - m = n; - } else - m->m_data = m->m_ext.ext_buf; - - -#ifdef VGE_FIXUP_RX - /* - * This is part of an evil trick to deal with non-x86 platforms. - * The VIA chip requires RX buffers to be aligned on 32-bit - * boundaries, but that will hose non-x86 machines. To get around - * this, we leave some empty space at the start of each buffer - * and for non-x86 hosts, we copy the buffer back two bytes - * to achieve word alignment. This is slightly more efficient - * than allocating a new buffer, copying the contents, and - * discarding the old buffer. + struct vge_rxdesc *rxd; + struct mbuf *m; + bus_dma_segment_t segs[1]; + bus_dmamap_t map; + int i, nsegs; + + m = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); + if (m == NULL) + return (ENOBUFS); + /* + * This is part of an evil trick to deal with strict-alignment + * architectures. The VIA chip requires RX buffers to be aligned + * on 32-bit boundaries, but that will hose strict-alignment + * architectures. To get around this, we leave some empty space + * at the start of each buffer and for non-strict-alignment hosts, + * we copy the buffer back two bytes to achieve word alignment. + * This is slightly more efficient than allocating a new buffer, + * copying the contents, and discarding the old buffer. */ - m->m_len = m->m_pkthdr.len = MCLBYTES - VGE_ETHER_ALIGN; - m_adj(m, VGE_ETHER_ALIGN); -#else m->m_len = m->m_pkthdr.len = MCLBYTES; -#endif + m_adj(m, VGE_RX_BUF_ALIGN); - arg.sc = sc; - arg.vge_idx = idx; - arg.vge_maxsegs = 1; - arg.vge_flags = 0; - - error = bus_dmamap_load_mbuf(sc->vge_ldata.vge_mtag, - sc->vge_ldata.vge_rx_dmamap[idx], m, vge_dma_map_rx_desc, - &arg, BUS_DMA_NOWAIT); - if (error || arg.vge_maxsegs != 1) { - if (n != NULL) - m_freem(n); - return (ENOMEM); + if (bus_dmamap_load_mbuf_sg(sc->vge_cdata.vge_rx_tag, + sc->vge_cdata.vge_rx_sparemap, m, segs, &nsegs, 0) != 0) { + m_freem(m); + return (ENOBUFS); } + KASSERT(nsegs == 1, ("%s: %d segments returned!", __func__, nsegs)); + + rxd = &sc->vge_cdata.vge_rxdesc[prod]; + if (rxd->rx_m != NULL) { + bus_dmamap_sync(sc->vge_cdata.vge_rx_tag, rxd->rx_dmamap, + BUS_DMASYNC_POSTREAD); + bus_dmamap_unload(sc->vge_cdata.vge_rx_tag, rxd->rx_dmamap); + } + map = rxd->rx_dmamap; + rxd->rx_dmamap = sc->vge_cdata.vge_rx_sparemap; + sc->vge_cdata.vge_rx_sparemap = map; + bus_dmamap_sync(sc->vge_cdata.vge_rx_tag, rxd->rx_dmamap, + BUS_DMASYNC_PREREAD); + rxd->rx_m = m; + + rxd->rx_desc->vge_sts = 0; + rxd->rx_desc->vge_ctl = 0; + rxd->rx_desc->vge_addrlo = htole32(VGE_ADDR_LO(segs[0].ds_addr)); + rxd->rx_desc->vge_addrhi = htole32(VGE_ADDR_HI(segs[0].ds_addr) | + (VGE_BUFLEN(segs[0].ds_len) << 16) | VGE_RXDESC_I); /* * Note: the manual fails to document the fact that for @@ -1184,73 +1239,127 @@ vge_newbuf(sc, idx, m) * but we should not set the OWN bits until we're ready * to hand back 4 of them in one shot. */ - -#define VGE_RXCHUNK 4 - sc->vge_rx_consumed++; - if (sc->vge_rx_consumed == VGE_RXCHUNK) { - for (i = idx; i != idx - sc->vge_rx_consumed; i--) - sc->vge_ldata.vge_rx_list[i].vge_sts |= - htole32(VGE_RDSTS_OWN); - sc->vge_rx_consumed = 0; + if ((prod % VGE_RXCHUNK) == (VGE_RXCHUNK - 1)) { + for (i = VGE_RXCHUNK; i > 0; i--) { + rxd->rx_desc->vge_sts = htole32(VGE_RDSTS_OWN); + rxd = rxd->rxd_prev; + } + sc->vge_cdata.vge_rx_commit += VGE_RXCHUNK; } - sc->vge_ldata.vge_rx_mbuf[idx] = m; - - bus_dmamap_sync(sc->vge_ldata.vge_mtag, - sc->vge_ldata.vge_rx_dmamap[idx], - BUS_DMASYNC_PREREAD); - return (0); } static int vge_tx_list_init(sc) - struct vge_softc *sc; + struct vge_softc *sc; { - bzero ((char *)sc->vge_ldata.vge_tx_list, VGE_TX_LIST_SZ); - bzero ((char *)&sc->vge_ldata.vge_tx_mbuf, - (VGE_TX_DESC_CNT * sizeof(struct mbuf *))); - - bus_dmamap_sync(sc->vge_ldata.vge_tx_list_tag, - sc->vge_ldata.vge_tx_list_map, BUS_DMASYNC_PREWRITE); - sc->vge_ldata.vge_tx_prodidx = 0; - sc->vge_ldata.vge_tx_considx = 0; - sc->vge_ldata.vge_tx_free = VGE_TX_DESC_CNT; + struct vge_ring_data *rd; + struct vge_txdesc *txd; + int i; + + VGE_LOCK_ASSERT(sc); + + sc->vge_cdata.vge_tx_prodidx = 0; + sc->vge_cdata.vge_tx_considx = 0; + sc->vge_cdata.vge_tx_cnt = 0; + + rd = &sc->vge_rdata; + bzero(rd->vge_tx_ring, VGE_TX_LIST_SZ); + for (i = 0; i < VGE_TX_DESC_CNT; i++) { + txd = &sc->vge_cdata.vge_txdesc[i]; + txd->tx_m = NULL; + txd->tx_desc = &rd->vge_tx_ring[i]; + } + + bus_dmamap_sync(sc->vge_cdata.vge_tx_ring_tag, + sc->vge_cdata.vge_tx_ring_map, + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); return (0); } static int vge_rx_list_init(sc) - struct vge_softc *sc; + struct vge_softc *sc; { + struct vge_ring_data *rd; + struct vge_rxdesc *rxd; int i; - bzero ((char *)sc->vge_ldata.vge_rx_list, VGE_RX_LIST_SZ); - bzero ((char *)&sc->vge_ldata.vge_rx_mbuf, - (VGE_RX_DESC_CNT * sizeof(struct mbuf *))); + VGE_LOCK_ASSERT(sc); - sc->vge_rx_consumed = 0; + sc->vge_cdata.vge_rx_prodidx = 0; + sc->vge_cdata.vge_head = NULL; + sc->vge_cdata.vge_tail = NULL; + sc->vge_cdata.vge_rx_commit = 0; + rd = &sc->vge_rdata; + bzero(rd->vge_rx_ring, VGE_RX_LIST_SZ); for (i = 0; i < VGE_RX_DESC_CNT; i++) { - if (vge_newbuf(sc, i, NULL) == ENOBUFS) + rxd = &sc->vge_cdata.vge_rxdesc[i]; + rxd->rx_m = NULL; + rxd->rx_desc = &rd->vge_rx_ring[i]; + if (i == 0) + rxd->rxd_prev = + &sc->vge_cdata.vge_rxdesc[VGE_RX_DESC_CNT - 1]; + else + rxd->rxd_prev = &sc->vge_cdata.vge_rxdesc[i - 1]; + if (vge_newbuf(sc, i) != 0) return (ENOBUFS); } - /* Flush the RX descriptors */ + bus_dmamap_sync(sc->vge_cdata.vge_rx_ring_tag, + sc->vge_cdata.vge_rx_ring_map, + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); - bus_dmamap_sync(sc->vge_ldata.vge_rx_list_tag, - sc->vge_ldata.vge_rx_list_map, - BUS_DMASYNC_PREWRITE|BUS_DMASYNC_PREREAD); - - sc->vge_ldata.vge_rx_prodidx = 0; - sc->vge_rx_consumed = 0; - sc->vge_head = sc->vge_tail = NULL; + sc->vge_cdata.vge_rx_commit = 0; return (0); } -#ifdef VGE_FIXUP_RX +static void +vge_freebufs(sc) + struct vge_softc *sc; +{ + struct vge_txdesc *txd; + struct vge_rxdesc *rxd; + struct ifnet *ifp; + int i; + + VGE_LOCK_ASSERT(sc); + + ifp = sc->vge_ifp; + /* + * Free RX and TX mbufs still in the queues. + */ + for (i = 0; i < VGE_RX_DESC_CNT; i++) { + rxd = &sc->vge_cdata.vge_rxdesc[i]; + if (rxd->rx_m != NULL) { + bus_dmamap_sync(sc->vge_cdata.vge_rx_tag, + rxd->rx_dmamap, BUS_DMASYNC_POSTREAD); + bus_dmamap_unload(sc->vge_cdata.vge_rx_tag, + rxd->rx_dmamap); + m_freem(rxd->rx_m); + rxd->rx_m = NULL; + } + } + + for (i = 0; i < VGE_TX_DESC_CNT; i++) { + txd = &sc->vge_cdata.vge_txdesc[i]; + if (txd->tx_m != NULL) { + bus_dmamap_sync(sc->vge_cdata.vge_tx_tag, + txd->tx_dmamap, BUS_DMASYNC_POSTWRITE); + bus_dmamap_unload(sc->vge_cdata.vge_tx_tag, + txd->tx_dmamap); + m_freem(txd->tx_m); + txd->tx_m = NULL; + ifp->if_oerrors++; + } + } +} + +#ifndef __NO_STRICT_ALIGNMENT static __inline void vge_fixup_rx(m) struct mbuf *m; @@ -1265,8 +1374,6 @@ vge_fixup_rx(m) *dst++ = *src++; m->m_data -= ETHER_ALIGN; - - return; } #endif @@ -1275,49 +1382,39 @@ vge_fixup_rx(m) * been fragmented across multiple 2K mbuf cluster buffers. */ static int -vge_rxeof(sc) +vge_rxeof(sc, count) struct vge_softc *sc; + int count; { struct mbuf *m; struct ifnet *ifp; - int i, total_len; - int lim = 0; + int prod, prog, total_len; + struct vge_rxdesc *rxd; struct vge_rx_desc *cur_rx; - u_int32_t rxstat, rxctl; + uint32_t rxstat, rxctl; VGE_LOCK_ASSERT(sc); - ifp = sc->vge_ifp; - i = sc->vge_ldata.vge_rx_prodidx; - - /* Invalidate the descriptor memory */ - - bus_dmamap_sync(sc->vge_ldata.vge_rx_list_tag, - sc->vge_ldata.vge_rx_list_map, - BUS_DMASYNC_POSTREAD); - - while (!VGE_OWN(&sc->vge_ldata.vge_rx_list[i])) { -#ifdef DEVICE_POLLING - if (ifp->if_capenable & IFCAP_POLLING) { - if (sc->rxcycles <= 0) - break; - sc->rxcycles--; - } -#endif + ifp = sc->vge_ifp; *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200912141844.nBEIiOZa034749>