Date: Tue, 10 Jul 2007 18:57:35 +0100 From: Tom Judge <tom@tomjudge.com> To: Tom Judge <tom@tomjudge.com> Cc: Sepherosa Ziehau <sepherosa@gmail.com>, freebsd-net <freebsd-net@freebsd.org>, David Christensen <davidch@broadcom.com> Subject: Re: Problems with BCE network adapter (Dell PE2950) Message-ID: <4693C88F.8050204@tomjudge.com> In-Reply-To: <46938E75.80900@tomjudge.com> References: <46680DB1.9050905@tomjudge.com> <09BFF2FA5EAB4A45B6655E151BBDD9030414B1EC@NT-IRVA-0750.brcm.ad.broadcom.com> <466873FA.9030800@tomjudge.com> <09BFF2FA5EAB4A45B6655E151BBDD9030423EE13@NT-IRVA-0750.brcm.ad.broadcom.com> <46823A78.7020501@tomjudge.com> <4683C578.6070009@tomjudge.com> <09BFF2FA5EAB4A45B6655E151BBDD90304571430@NT-IRVA-0750.brcm.ad.broadcom.com> <09BFF2FA5EAB4A45B6655E151BBDD903045714EF@NT-IRVA-0750.brcm.ad.broadcom.com> <4684D5C0.3040709@tomjudge.com> <ea7b9c170706300244s5c93851fn6a8e63f2d3b05da4@mail.gmail.com> <09BFF2FA5EAB4A45B6655E151BBDD90304571BF1@NT-IRVA-0750.brcm.ad.broadcom.com> <468A2D55.60301@tomjudge.com> <46937D97.7030507@tomjudge.com> <46938E75.80900@tomjudge.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------060500010901090709040408 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Tom Judge wrote: > Tom Judge wrote: >> Tom Judge wrote: >>> David Christensen wrote: >>>>> Sorry for the top post, please try following patch: >>>>> http://people.freebsd.org/~sephe/if_bce.c.diff >>>>> >>>>> This is probably the cause; I noticed it when bce(4) was ported to >>>>> DragonFly. >>>>> >>>> >>>> Thanks Sephe, I think you're on to something. I have some >>>> debug code in the driver to simulate mbuf allocation >>>> failures and when I enable that I start receiving the same >>>> error messages Tom reported (along with various kernel >>>> panics), but when I include your change the system seems >>>> to keep humming along. I'll certainly add your code into an update >>>> shortly. >>>> >>>> Dave >>>> >>> >>> I'm not going to have a chance to test this patch until next week but >>> I will let you know what the results are. >>> >>> Tom >> >> >> So here goes, after 2 days testing we have come up with the following >> data. >> >> The configuration >> >> [PE[12]950] ----> [PowerConnect 5324] >> >> The system is running 8192 byte Jumbo Frames. >> >> sultan# ifconfig bce0 >> bce0: flags=8847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST> mtu 8192 >> options=3b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU> >> inet 172.31.0.28 netmask 0xffffff00 broadcast 172.31.0.255 >> inet 172.31.0.163 netmask 0xffffffff broadcast 172.31.0.163 >> ether 00:19:b9:e4:4d:cc >> media: Ethernet autoselect (1000baseTX <full-duplex>) >> status: active >> >> >> After applying both David and Sephe's patches I have yet to get a >> system in a state where it is stable with jumbo frames enabled, the >> systems crash almost immediately after the switch changes the port >> state (Spanning tree) from LEARNING to FORWARDING. The output from >> this crash can be found attached as crash-1.txt.gz. >> >> If the frame size is left at 1500 then the interface seems stable, >> however I can't fully test this as the interface is connected to a >> GigE only network with an mtu of 8192. >> >> If BCE_DEBUG is remove from if_bcereg.h then the system just exhibits >> the original problem and may or may not crash. >> >> The next test was to try the kernel with BCE_DEBUG and with the >> following extra patch (so that the driver does not jump to the >> breakpoint when an unexpected mbuf is found in the rx buffer). >> >> --- if_bce.c (revision 62) >> +++ if_bce.c (revision 66) >> @@ -4050,7 +4050,8 @@ >> DBRUNIF((!(rxbd->rx_bd_flags & RX_BD_FLAGS_END)), >> BCE_PRINTF("%s(%d): Unexpected mbuf >> found in rx_bd[0x%04X]!\n", >> __FILE__, __LINE__, sw_chain_cons); >> - bce_breakpoint(sc)); >> + bce_dump_mbuf(sc, m)); >> +// bce_breakpoint(sc)); >> >> /* >> * ToDo: If the received packet is small enough >> >> >> With this patch the system boots and does not crash straight away, >> however it is almost completely unusable. The output with this kernel >> can be found attached as crash-2.txt.gz. Also this causes the >> following new error message: >> >> fgrep -n leak crash-2.txt >> 3194:bce0: /usr/src/sys/dev/bce/if_bce.c(3842): Memory leak! Lost 114 >> mbufs from rx chain! >> >> Has no one else come across this problem, or are Jumbo frames not >> widely used? >> >> Tom >> > It would seem that the crash can be simulated just by increasing the MTU > above 1500 (tested in single user mode). > Ok so I think I have fix the problem with the rx_bd tracking. I have ported rboyer's patch to NetBSD's bnx driver to FreeBSD (patch attached). The patch seems to get rid of two problems: 1) Unexpected mbuf in rx_bd 2) Too many free rx_bd's However I am still faced with the problem of frames with missing ethernet headers: bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(0), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:7B69AC00, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86F76000 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F76000, ext_size = 9216, type = EXT_JUMBO9 bce0: discard frame w/o leading ethernet header (len 4294967292 pkt len 4294967292) bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(0), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:5EB48B00, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86F73000 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F73000, ext_size = 9216, type = EXT_JUMBO9 bce0: discard frame w/o leading ethernet header (len 4294967292 pkt len 4294967292) bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(27745), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:5E9DDC00, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86EF8000 0x00: 2C 6F 75 3D 50 65 72 73 6F 6E 61 6C 2C 6F 75 3D 0x10: 47 72 6F 75 70 73 2C 6F 3D 4D 69 6E 74 65 6C 30 0x20: 28 30 11 04 02 63 6E 31 0B 04 09 63 62 75 74 74 0x30: 72 6F 73 65 30 13 04 09 67 69 64 4E 75 6D 62 65 0x40: 72 31 06 04 04 31 31 30 38 30 5E 02 01 02 64 59 0x50: 04 31 63 6E 3D 6D 63 61 68 6D 2C 6F 75 3D 4C 6F 0x60: 6E 64 6F 6E 2C 6F 75 3D 50 65 72 73 6F 6E 61 6C 0x70: 2C 6F 75 3D 47 72 6F 75 70 73 2C 6F 3D 4D 69 6E bce0: - m_pkthl er0ror 9 67 69 64 4E 75 6D 62 65 72 31 06 04 04 31 30 0x70: 33 37 30 62 02 01 02 64 5D 04 33 63 6E 3D 72 63 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86E8C000, ext_size = 9216, type = EXT_JUMBO9 bce0: /usr/src/sys/dev/bce/if_bce.c(4081): Unexpected mbuf found in rx_bd[0x002A]! bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(28515), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:5AB4C800, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86F28000 0x00: 30 0E 04 02 63 6E 31 08 04 06 63 6F 68 61 72 61 0x10: 30 13 04 09 67 69 64 4E 75 6D 62 65 72 31 06 04 0x20: 04 31 30 37 32 30 65 02 01 02 64 60 04 35 63 6E 0x30: 3D 6A 70 69 65 6B 61 72 73 2C 6F 75 3D 43 68 69 0x40: 63 61 67 6F 2C 6F 75 3D 50 65 72 73 6F 6E 61 6C 0x50: 2C 6F 75 3D 47 72 6F 75 70 73 2C 6F 3D 4D 69 6E 0x60: 74 65 6C 30 27 30 10 04 02 63 6E 31 0A 04 08 6A 0x70: 70 69 65 6B 61 72 73 30 13 04 09 67 69 64 4E 75 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F28000, ext_size = 9216, type = EXT_JUMBO9 bce0: /usr/src/sys/dev/bce/if_bce.c(4081): Unexpected mbuf found in rx_bd[0x002E]! bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(28460), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:5EB9F200, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86F70000 0x00: 04 32 63 6E 3D 69 6E 65 73 73 2C 6F 75 3D 43 68 0x10: 69 63 61 67 6F 2C 6F 75 3D 50 65 72 73 6F 6E 61 0x20: 6C 2C 6F 75 3D 47 72 6F 75 70 73 2C 6F 3D 4D 69 0x30: 6E 74 65 6C 30 24 30 0D 04 02 63 6E 31 07 04 05 0x40: 69 6E 65 73 73 30 13 04 09 67 69 64 4E 75 6D 62 0x50: 65 72 31 06 04 04 31 31 34 32 30 67 02 01 02 64 0x60: 62 04 36 63 6E 3D 70 6D 63 6E 61 6D 61 72 61 2C 0x70: 6F 75 3D 43 68 69 63 61 67 6F 2C 6F 75 3D 50 65 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F70000, ext_size = 9216, type = EXT_JUMBO9 bce0: /usr/src/sys/dev/bce/if_bce.c(4081): Unexpected mbuf found in rx_bd[0x0032]! bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(28787), Max(9022) bce0: mbuf: vaddr = 0xFFFFFF00:5AB4CA00, m_len = 9216, m_flags = ( M_EXT M_PKTHDR ) m_data = 0xFFFFFFFF:86F6D000 0x00: 02 01 02 64 57 04 30 63 6E 3D 73 70 79 65 2C 6F 0x10: 75 3D 4C 6F 6E 64 6F 6E 2C 6F 75 3D 50 65 72 73 0x20: 6F 6E 61 6C 2C 6F 75 3D 47 72 6F 75 70 73 2C 6F 0x30: 3D 4D 69 6E 74 65 6C 30 23 30 0C 04 02 63 6E 31 0x40: 06 04 04 73 70 79 65 30 13 04 09 67 69 64 4E 75 0x50: 6D 62 65 72 31 06 04 04 31 32 30 39 30 59 02 01 0x60: 02 64 54 04 2F 63 6E 3D 71 61 2C 6F 75 3D 43 68 0x70: 69 63 61 67 6F 2C 6F 75 3D 50 65 72 73 6F 6E 61 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F6D000, ext_size = 9216, type = EXT_JUMBO9 bce0: /usr/src/sys/dev/bce/if_bce.c(4128): Unusual frame size found. Min(60), Actual(12855), Max(9022) bce0: mbuf: vaddr =0 67 02 01 02 64 0x60: 62 04 36 63 6E 3D 70 6D 63 6E 61 6D 61 72 61 2C 0x70: 6F 75 3D 43 68 69 63 61 67 6F 2C 6F 75 3D 50 65 bce0: - m_pkthdr: flags = ( ) csum_flags = ( ) bce0: - m_ext: vaddr = 0xFFFFFFFF:86F70000, ext_size = 9216, type = EXT_JUMBO9 if_bnx.c - 1.4 -> 1.5 LOG: RX buffers are malloced memory of 9216 bytes. This can require from 1 to 4 DMA memory segments, depending on how the buffer is in memory. When receiving a packet, we allocate a new one to remplace the one we've used. It can need more segments than the one it remplace, leading to corrution of the RX descriptors, and a panic in bus_dmamap_sync() (DIAGNOSTIC kernels) or possibly memory corruption. Fix: - bce_get_buf() allocates as many buffer as possible, checking the number of free RX descriptors. Because one receive buffer is not guaranteed to be remplaced on receive, call bce_get_buf() from bce_tick() too. This also improve error handling from bce_get_buf(). - use MCLGET() instead of MEXTMALLOC() if we're running with the standard ethernet MTU. This gives us more receive buffers and waste less memory. Seem to be moving in the right direction slowly. Tom --------------060500010901090709040408 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" Index: if_bce.c =================================================================== --- if_bce.c (revision 67) +++ if_bce.c (revision 74) @@ -732,7 +732,7 @@ ifp->if_capenable = ifp->if_capabilities; /* Assume a standard 1500 byte MTU size for mbuf allocations. */ - sc->mbuf_alloc_size = MCLBYTES; +// sc->mbuf_alloc_size = MCLBYTES; #ifdef DEVICE_POLLING ifp->if_capabilities |= IFCAP_POLLING; #endif @@ -3475,10 +3475,12 @@ bus_dma_segment_t segs[4]; struct mbuf *m_new = NULL; struct rx_bd *rxbd; - int i, nsegs, error, rc = 0; + int i, nsegs, rc = 0; #ifdef BCE_DEBUG u16 debug_chain_prod = *chain_prod; #endif + u16 first_chain_prod; + u16 min_free_bd; DBPRINT(sc, (BCE_VERBOSE_RESET | BCE_VERBOSE_RECV), "Entering %s()\n", __FUNCTION__); @@ -3491,114 +3493,146 @@ DBPRINT(sc, BCE_VERBOSE_RECV, "%s(enter): prod = 0x%04X, chain_prod = 0x%04X, " "prod_bseq = 0x%08X\n", __FUNCTION__, *prod, *chain_prod, *prod_bseq); - if (m == NULL) { + /* try to get in as many mbufs as possible */ + if (sc->mbuf_alloc_size == MCLBYTES) + min_free_bd = (MCLBYTES + PAGE_SIZE - 1) / PAGE_SIZE; + else + min_free_bd = (BCE_MAX_MTU + PAGE_SIZE - 1) / PAGE_SIZE; + while (sc->free_rx_bd >= min_free_bd) { + if (m == NULL) { + DBRUNIF(DB_RANDOMTRUE(bce_debug_mbuf_allocation_failure), + BCE_PRINTF("%s(%d): Simulating mbuf allocation failure.\n", + __FILE__, __LINE__); + sc->mbuf_alloc_failed++; + rc = ENOBUFS; + goto bce_get_buf_exit); - DBRUNIF(DB_RANDOMTRUE(bce_debug_mbuf_allocation_failure), - BCE_PRINTF("%s(%d): Simulating mbuf allocation failure.\n", - __FILE__, __LINE__); - sc->mbuf_alloc_failed++; - rc = ENOBUFS; - goto bce_get_buf_exit); + /* This is a new mbuf allocation. */ + MGETHDR(m_new, M_DONTWAIT, MT_DATA); + if (m_new == NULL) { + DBPRINT(sc, BCE_WARN, + "%s(%d): RX mbuf header allocation failed!\n", + __FILE__, __LINE__); +// + DBRUNIF(1, sc->mbuf_alloc_failed++); +// - /* This is a new mbuf allocation. */ - MGETHDR(m_new, M_DONTWAIT, MT_DATA); - if (m_new == NULL) { + rc = ENOBUFS; + goto bce_get_buf_exit; + } +// + DBRUNIF(1, sc->rx_mbuf_alloc++); + if (sc->mbuf_alloc_size == MCLBYTES) + MCLGET(m_new, M_DONTWAIT); + else + m_cljget(m_new, M_DONTWAIT, sc->mbuf_alloc_size); + if (!(m_new->m_flags & M_EXT)) { + DBPRINT(sc, BCE_WARN, + "%s(%d): RX mbuf chain allocation failed!\n", + __FILE__, __LINE__); - DBPRINT(sc, BCE_WARN, "%s(%d): RX mbuf header allocation failed!\n", - __FILE__, __LINE__); + m_freem(m_new); +// + DBRUNIF(1, sc->rx_mbuf_alloc--); + DBRUNIF(1, sc->mbuf_alloc_failed++); - DBRUNIF(1, sc->mbuf_alloc_failed++); + rc = ENOBUFS; + goto bce_get_buf_exit; + } - rc = ENOBUFS; - goto bce_get_buf_exit; - } + } else { + m_new = m; + m = NULL; + m_new->m_data = m_new->m_ext.ext_buf; + } + m_new->m_len = m_new->m_pkthdr.len = sc->mbuf_alloc_size; +// - DBRUNIF(1, sc->rx_mbuf_alloc++); - m_cljget(m_new, M_DONTWAIT, sc->mbuf_alloc_size); - if (!(m_new->m_flags & M_EXT)) { - - DBPRINT(sc, BCE_WARN, "%s(%d): RX mbuf chain allocation failed!\n", - __FILE__, __LINE__); - + /* Map the mbuf cluster into device memory. */ + map = sc->rx_mbuf_map[*chain_prod]; + first_chain_prod = *chain_prod; + if (bus_dmamap_load_mbuf_sg(sc->rx_mbuf_tag, map, m_new, segs, &nsegs, BUS_DMA_NOWAIT)) { + BCE_PRINTF("%s(%d): Error mapping mbuf into RX chain!\n", + __FILE__, __LINE__); +// m_freem(m_new); DBRUNIF(1, sc->rx_mbuf_alloc--); - DBRUNIF(1, sc->mbuf_alloc_failed++); rc = ENOBUFS; goto bce_get_buf_exit; } - m_new->m_len = m_new->m_pkthdr.len = sc->mbuf_alloc_size; - } else { - m_new = m; - m_new->m_len = m_new->m_pkthdr.len = sc->mbuf_alloc_size; - m_new->m_data = m_new->m_ext.ext_buf; - } + bus_dmamap_sync(sc->rx_mbuf_tag, map, BUS_DMASYNC_PREREAD); +// - /* Map the mbuf cluster into device memory. */ - map = sc->rx_mbuf_map[*chain_prod]; - error = bus_dmamap_load_mbuf_sg(sc->rx_mbuf_tag, map, m_new, - segs, &nsegs, BUS_DMA_NOWAIT); - if (error) { - BCE_PRINTF("%s(%d): Error mapping mbuf into RX chain!\n", - __FILE__, __LINE__); + /* Watch for overflow. */ + DBRUNIF((sc->free_rx_bd > USABLE_RX_BD), + BCE_PRINTF("%s(%d): Too many free rx_bd (0x%04X > 0x%04X)!\n", + __FILE__, __LINE__, sc->free_rx_bd, (u16) USABLE_RX_BD)); - m_freem(m_new); + /* Update some debug statistic counters */ + DBRUNIF((sc->free_rx_bd < sc->rx_low_watermark), + sc->rx_low_watermark = sc->free_rx_bd); + DBRUNIF((sc->free_rx_bd == 0), sc->rx_empty_count++); - DBRUNIF(1, sc->rx_mbuf_alloc--); + /* + * Setup the rx_bd for the first segment + */ + rxbd = &sc->rx_bd_chain[RX_PAGE(*chain_prod)][RX_IDX(*chain_prod)]; - rc = ENOBUFS; - goto bce_get_buf_exit; - } + rxbd->rx_bd_haddr_lo = htole32(BCE_ADDR_LO(segs[0].ds_addr)); + rxbd->rx_bd_haddr_hi = htole32(BCE_ADDR_HI(segs[0].ds_addr)); + rxbd->rx_bd_len = htole32(segs[0].ds_len); + rxbd->rx_bd_flags = htole32(RX_BD_FLAGS_START); + *prod_bseq += segs[0].ds_len; + bus_dmamap_sync(sc->rx_mbuf_tag, + sc->rx_bd_chain_map[RX_PAGE(*chain_prod)], + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); + for (i = 1; i < nsegs; i++) { + *prod = NEXT_RX_BD(*prod); + *chain_prod = RX_CHAIN_IDX(*prod); - /* Watch for overflow. */ - DBRUNIF((sc->free_rx_bd > USABLE_RX_BD), - BCE_PRINTF("%s(%d): Too many free rx_bd (0x%04X > 0x%04X)!\n", - __FILE__, __LINE__, sc->free_rx_bd, (u16) USABLE_RX_BD)); + rxbd = &sc->rx_bd_chain[RX_PAGE(*chain_prod)][RX_IDX(*chain_prod)]; + rxbd->rx_bd_haddr_lo = htole32(BCE_ADDR_LO(segs[i].ds_addr)); + rxbd->rx_bd_haddr_hi = htole32(BCE_ADDR_HI(segs[i].ds_addr)); + rxbd->rx_bd_len = htole32(segs[i].ds_len); + rxbd->rx_bd_flags = 0; + *prod_bseq += segs[i].ds_len; - /* Update some debug statistic counters */ - DBRUNIF((sc->free_rx_bd < sc->rx_low_watermark), - sc->rx_low_watermark = sc->free_rx_bd); - DBRUNIF((sc->free_rx_bd == 0), sc->rx_empty_count++); + bus_dmamap_sync(sc->rx_mbuf_tag, + sc->rx_bd_chain_map[RX_PAGE(*chain_prod)], + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); + } - /* Setup the rx_bd for the first segment. */ - rxbd = &sc->rx_bd_chain[RX_PAGE(*chain_prod)][RX_IDX(*chain_prod)]; + rxbd->rx_bd_flags |= htole32(RX_BD_FLAGS_END); + bus_dmamap_sync(sc->rx_mbuf_tag, + sc->rx_bd_chain_map[RX_PAGE(*chain_prod)], + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); - rxbd->rx_bd_haddr_lo = htole32(BCE_ADDR_LO(segs[0].ds_addr)); - rxbd->rx_bd_haddr_hi = htole32(BCE_ADDR_HI(segs[0].ds_addr)); - rxbd->rx_bd_len = htole32(segs[0].ds_len); - rxbd->rx_bd_flags = htole32(RX_BD_FLAGS_START); - *prod_bseq += segs[0].ds_len; + /* + * Save the mbuf, ajust the map pointer (swap map for first and + * last rx_bd entry to that rx_mbuf_ptr and rx_mbuf_map matches) + * and update counter. + */ + sc->rx_mbuf_ptr[*chain_prod] = m_new; + sc->rx_mbuf_map[first_chain_prod] = sc->rx_mbuf_map[*chain_prod]; + sc->rx_mbuf_map[*chain_prod] = map; + sc->free_rx_bd -= nsegs; - for (i = 1; i < nsegs; i++) { - *prod = NEXT_RX_BD(*prod); - *chain_prod = RX_CHAIN_IDX(*prod); + DBRUN(BCE_VERBOSE_RECV, bce_dump_rx_mbuf_chain(sc, debug_chain_prod, + nsegs)); + *prod = NEXT_RX_BD(*prod); + *chain_prod = RX_CHAIN_IDX(*prod); + } - rxbd = &sc->rx_bd_chain[RX_PAGE(*chain_prod)][RX_IDX(*chain_prod)]; - rxbd->rx_bd_haddr_lo = htole32(BCE_ADDR_LO(segs[i].ds_addr)); - rxbd->rx_bd_haddr_hi = htole32(BCE_ADDR_HI(segs[i].ds_addr)); - rxbd->rx_bd_len = htole32(segs[i].ds_len); - rxbd->rx_bd_flags = 0; - *prod_bseq += segs[i].ds_len; - } - - rxbd->rx_bd_flags |= htole32(RX_BD_FLAGS_END); - - /* Save the mbuf and update our counter. */ - sc->rx_mbuf_ptr[*chain_prod] = m_new; - sc->free_rx_bd -= nsegs; - - DBRUN(BCE_VERBOSE_RECV, bce_dump_rx_mbuf_chain(sc, debug_chain_prod, - nsegs)); - +bce_get_buf_exit: DBPRINT(sc, BCE_VERBOSE_RECV, "%s(exit): prod = 0x%04X, chain_prod = 0x%04X, " "prod_bseq = 0x%08X\n", __FUNCTION__, *prod, *chain_prod, *prod_bseq); -bce_get_buf_exit: DBPRINT(sc, (BCE_VERBOSE_RESET | BCE_VERBOSE_RECV), "Exiting %s()\n", __FUNCTION__); @@ -3773,16 +3807,11 @@ /* Allocate mbuf clusters for the rx_bd chain. */ prod = prod_bseq = 0; - while (prod < TOTAL_RX_BD) { - chain_prod = RX_CHAIN_IDX(prod); - if (bce_get_buf(sc, NULL, &prod, &chain_prod, &prod_bseq)) { - BCE_PRINTF("%s(%d): Error filling RX chain: rx_bd[0x%04X]!\n", - __FILE__, __LINE__, chain_prod); - rc = ENOBUFS; - break; - } - prod = NEXT_RX_BD(prod); - } + chain_prod = RX_CHAIN_IDX(prod); + if (bce_get_buf(sc, NULL, &prod, &chain_prod, &prod_bseq)) { + BCE_PRINTF( + "Error filling RX chain: rx_bd[0x%04X]!\n", chain_prod); + } /* Save the RX chain producer index. */ sc->rx_prod = prod; @@ -4045,13 +4074,14 @@ /* The mbuf is stored with the last rx_bd entry of a packet. */ if (sc->rx_mbuf_ptr[sw_chain_cons] != NULL) { - +#ifdef BCE_DEBUG /* Validate that this is the last rx_bd. */ - DBRUNIF((!(rxbd->rx_bd_flags & RX_BD_FLAGS_END)), + if ((rxbd->rx_bd_flags & RX_BD_FLAGS_END) == 0) { BCE_PRINTF("%s(%d): Unexpected mbuf found in rx_bd[0x%04X]!\n", __FILE__, __LINE__, sw_chain_cons); - bce_breakpoint(sc)); - +// bce_breakpoint(sc)); + } +#endif /* * ToDo: If the received packet is small enough * to fit into a single, non-M_EXT mbuf, @@ -4117,7 +4147,7 @@ } m = NULL; - goto bce_rx_int_next_rx; + continue; } /* @@ -4143,7 +4173,7 @@ } m = NULL; - goto bce_rx_int_next_rx; + continue; } /* Skip over the l2_fhdr when passing the data up the stack. */ @@ -4215,7 +4245,6 @@ /* Pass the mbuf off to the upper layers. */ ifp->if_ipackets++; -bce_rx_int_next_rx: sw_prod = NEXT_RX_BD(sw_prod); } @@ -4475,9 +4504,15 @@ bce_set_mac_addr(sc); /* Calculate and program the Ethernet MTU size. */ - ether_mtu = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN + ifp->if_mtu + - ETHER_CRC_LEN; + if (ifp->if_mtu <= ETHERMTU) { + ether_mtu = BCE_MAX_STD_ETHER_MTU_VLAN; + sc->mbuf_alloc_size = MCLBYTES; + } else { + ether_mtu = BCE_MAX_JUMBO_ETHER_MTU_VLAN; + sc->mbuf_alloc_size = BCE_MAX_MTU; + } + DBPRINT(sc, BCE_INFO_MISC, "%s(): setting mtu = %d\n",__FUNCTION__, ether_mtu); /* @@ -4488,10 +4523,8 @@ if (ether_mtu > ETHER_MAX_LEN + ETHER_VLAN_ENCAP_LEN) { REG_WR(sc, BCE_EMAC_RX_MTU_SIZE, min(ether_mtu, BCE_MAX_JUMBO_ETHER_MTU) | BCE_EMAC_RX_MTU_SIZE_JUMBO_ENA); - sc->mbuf_alloc_size = MJUM9BYTES; } else { REG_WR(sc, BCE_EMAC_RX_MTU_SIZE, ether_mtu); - sc->mbuf_alloc_size = MCLBYTES; } /* Calculate the RX Ethernet frame size for rx_bd's. */ @@ -5640,6 +5673,8 @@ struct bce_softc *sc = xsc; struct mii_data *mii; struct ifnet *ifp; + u16 prod, chain_prod; + u32 prod_bseq; ifp = sc->bce_ifp; @@ -5675,6 +5710,13 @@ } bce_tick_locked_exit: + /* try to get more RX buffers, just in case */ + prod = sc->rx_prod; + prod_bseq = sc->rx_prod_bseq; + chain_prod = RX_CHAIN_IDX(prod); + bce_get_buf(sc, NULL, &prod, &chain_prod, &prod_bseq); + sc->rx_prod = prod; + sc->rx_prod_bseq = prod_bseq; return; } Index: if_bcereg.h =================================================================== --- if_bcereg.h (revision 67) +++ if_bcereg.h (revision 74) @@ -4682,7 +4682,7 @@ #define BCE_MAX_JUMBO_ETHER_MTU 9018 #define BCE_MAX_JUMBO_ETHER_MTU_VLAN 9022 -// #define BCE_MAX_MTU ETHER_MAX_LEN_JUMBO + ETHER_VLAN_ENCAP_LEN /* 9022 */ +#define BCE_MAX_MTU 9216 /****************************************************************************/ /* BCE Device State Data Structure */ --------------060500010901090709040408--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4693C88F.8050204>