Date: Tue, 2 Jan 2007 11:31:02 -0800 From: "David Christensen" <davidch@broadcom.com> To: "Bruce Evans" <bde@zeta.org.au>, "Doug Barton" <dougb@freebsd.org> Cc: net@freebsd.org Subject: RE: [Fwd: Re: bge Ierr rate increase from 5.3R -> 6.1R] Message-ID: <09BFF2FA5EAB4A45B6655E151BBDD90302B2E59D@NT-IRVA-0750.brcm.ad.broadcom.com> In-Reply-To: <20061230233343.U745@epsplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> These happen under loads that can't be handled, and generally cause > thousands of input errors every second. The hardware records dropped > packets separately from other input errors, but unfortunately=20 > all types > of input errors are counted together in if_ierrors, and I haven't done > more than muck around in ddb to separate them. There are a couple places I can suggest you look to understand if there are any resource limitations being encountered by the controller. The first is register 0x4450 (bits 15:0 for 5700 to 5704 and bits 8:0 for other controllers) which shows the number of internal=20 free RX MBUFs on the controller. (In this case an MBUF is an internal=20 128 byte data structure used by the controller.) This value is especially=20 important on 5700 to 5704 devices since those devices share a pool of=20 MBUFs between the RX and TX data paths, unlike the 5705 and later devices which use separate MBUF pools for RX and TX data. As the value approaches=20 0, the controller will either start to generate pause frames or simply=20 drop packets as specified by the values in registers 0x4410, 0x4414, and 0x4418. If you see ifInDiscards incrementing then this is a definite possibility. You would also see bit 4 of register 0x4400 set if this occurs. I also noticed on -CURRENT that register 0x2c18 was set to a rather high value (0x40). This register controls how many bge_rx_bd's will be fetched at one time. A high value will reduce PCI bus utilization as it will fetch more BD's in a single transaction, but it will increase latency and could potentially deplete the available internal MBUFs as the controller waits for the driver to populate additional bge_rx_bd's. Try using a smaller=20 value such as 0x08. It would be really interesting if you could add a sysctl that would bring out the hardware statistics maintained by the device, similar to what exists in the "bce" driver. With this information we could focus on individual blocks to see where the packet loss is occurring and may be able to come up with more ideas on tuning the controller. > bge also has a "mini" rx ring which FreeBSD doesn't use. I=20 > don't really > understand this or the interaction of the separate rx rings, but hope > that the mini ring can be used to handle small packets and would only > need an mbuf (not an mbuf cluster) for each packet, and with=20 > it and the > jumbo ring the total hardware buffering would be 1024(mini) +=20 > 512(std) + > 256(jumbo), with the 1024-entry event ring only used to communicate > with the host and its size not really limiting buffering. Meeting a > latency requirement of 1024+512 tiny-packet times is much easier than > meeting one of 192 or 20 tiny-packet times. (I only actually saw the > limits of 20 and 192 for full-sized (non jumbo) packets.) Mini rings are only supported by 57XX controllers that support external SSRAM, and the only controller that supports that is the 5700. This=20 feature was never used in production so I wouldn't consider it an option since you'll never find the right hardware to support it. Dave
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?09BFF2FA5EAB4A45B6655E151BBDD90302B2E59D>