Date: Wed, 7 Mar 2012 19:43:45 -0800 From: YongHyeon PYUN <pyunyh@gmail.com> To: Eugene Grosbein <egrosbein@rdtc.ru> Cc: marius@freebsd.org, yongari@freebsd.org, "net@freebsd.org" <net@freebsd.org> Subject: Re: suboptimal bge(4) BCM5704 performance in RELENG_8 Message-ID: <20120308034345.GD9436@michelle.cdnetworks.com> In-Reply-To: <4F571870.3090902@rdtc.ru> References: <4F5608EA.6080705@rdtc.ru> <20120307202914.GB9436@michelle.cdnetworks.com> <4F571870.3090902@rdtc.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 07, 2012 at 03:12:32PM +0700, Eugene Grosbein wrote: > 08.03.2012 03:29, YongHyeon PYUN пишет: > > On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote: > >> Hi! > >> > >> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64 > >> running busy icecast2 server in hope it can saturate 1G bge(4) link. > >> > >> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704). > > > > Would you show me the output of dmesg(bge(4) and brgphy(4) related > > ones)? > > # egrep 'bge|brgphy' /var/run/dmesg.boot > bge0: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf70000-0xfdf7ffff irq 25 at device 2.0 on pci2 > bge0: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X > miibus0: <MII bus> on bge0 > brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bge0: Ethernet address: 00:18:fe:86:ab:f4 > bge0: [ITHREAD] > bge1: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf60000-0xfdf6ffff irq 26 at device 2.1 on pci2 > bge1: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X > miibus1: <MII bus> on bge1 > brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bge1: Ethernet address: 00:18:fe:86:ab:f3 > bge1: [ITHREAD] > It seems your controller is BCM5704 B2 and it is running in PCI-X mode. There is one known BCM5704 issue but I don't think you're seeing this one. To confirm that could you show me both the output of "pciconf -lcbv" and "devinfo -rv"? > There are bge0 and bge1 but bge1 is not used and is down. > > >> Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300? > >> > > > > Yes. > > [skip] > > >> I've glanced at bge(4) code and see it uses lots of hardcoded constants > >> for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue. > > > > Firmware has a fixed number of descriptors so increasing them > > wouldn't give better numbers. > > What it the limit? > Both TX and RX have 512 descriptors. > >> And no loader tunnables/sysctls like em/igb have. > > > > Show me the output of "sysctl dev.bge.0.stats". > > # sysctl dev.bge.0.stats > dev.bge.0.stats.FramesDroppedDueToFilters: 0 > dev.bge.0.stats.DmaWriteQueueFull: 84072 > dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 > dev.bge.0.stats.NoMoreRxBDs: 0 > dev.bge.0.stats.InputDiscards: 0 > dev.bge.0.stats.InputErrors: 30 > dev.bge.0.stats.RecvThresholdHit: 745400662 > dev.bge.0.stats.DmaReadQueueFull: 2020586592 > dev.bge.0.stats.DmaReadHighPriQueueFull: 0 > dev.bge.0.stats.SendDataCompQueueFull: 0 > dev.bge.0.stats.RingSetSendProdIndex: 2832885493 > dev.bge.0.stats.RingStatusUpdate: 899990835 > dev.bge.0.stats.Interrupts: 899990835 > dev.bge.0.stats.AvoidedInterrupts: 0 > dev.bge.0.stats.SendThresholdHit: 0 > dev.bge.0.stats.rx.ifHCInOctets: 491268800 > dev.bge.0.stats.rx.Fragments: 234 > dev.bge.0.stats.rx.UnicastPkts: 1977202324 > dev.bge.0.stats.rx.MulticastPkts: 0 > dev.bge.0.stats.rx.FCSErrors: 341 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You have multiple FCS and Input errors. Check signal quality(i.e. UTP cable). > dev.bge.0.stats.rx.AlignmentErrors: 0 > dev.bge.0.stats.rx.xonPauseFramesReceived: 0 > dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 > dev.bge.0.stats.rx.ControlFramesReceived: 0 > dev.bge.0.stats.rx.xoffStateEntered: 0 > dev.bge.0.stats.rx.FramesTooLong: 0 > dev.bge.0.stats.rx.Jabbers: 0 > dev.bge.0.stats.rx.UndersizePkts: 0 > dev.bge.0.stats.rx.inRangeLengthError: 0 > dev.bge.0.stats.rx.outRangeLengthError: 0 > dev.bge.0.stats.tx.ifHCOutOctets: 683592718 > dev.bge.0.stats.tx.Collisions: 0 > dev.bge.0.stats.tx.XonSent: 0 > dev.bge.0.stats.tx.XoffSent: 0 > dev.bge.0.stats.tx.flowControlDone: 0 > dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 > dev.bge.0.stats.tx.SingleCollisionFrames: 0 > dev.bge.0.stats.tx.MultipleCollisionFrames: 0 > dev.bge.0.stats.tx.DeferredTransmissions: 0 > dev.bge.0.stats.tx.ExcessiveCollisions: 0 > dev.bge.0.stats.tx.LateCollisions: 0 > dev.bge.0.stats.tx.UnicastPkts: 3292778353 > dev.bge.0.stats.tx.MulticastPkts: 0 > dev.bge.0.stats.tx.BroadcastPkts: 147 > dev.bge.0.stats.tx.CarrierSenseErrors: 0 > dev.bge.0.stats.tx.Discards: 0 > dev.bge.0.stats.tx.Errors: 0 > > >> Should I try to play with constants in the code and if so, what are limits of this chip? > >> Or it will never be capable of utilizing full gigabit speed? > > > > BCM5704 is old controller but it should have no problems to > > saturate the link. I suspect it could be related with DMA > > configuration but needs more information. > > I'll supply any information. It's also possible to try kernel patches, if any :-) Does netperf benchmark also show 540Mbps on bge(4)? > > Eugene Grosbein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120308034345.GD9436>