From owner-freebsd-net@FreeBSD.ORG Wed Mar 7 08:12:37 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E402C106564A; Wed, 7 Mar 2012 08:12:36 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) by mx1.freebsd.org (Postfix) with ESMTP id 1573E8FC16; Wed, 7 Mar 2012 08:12:35 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.5/8.14.5) with ESMTP id q278CW9O004159; Wed, 7 Mar 2012 15:12:33 +0700 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4F571870.3090902@rdtc.ru> Date: Wed, 07 Mar 2012 15:12:32 +0700 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: pyunyh@gmail.com References: <4F5608EA.6080705@rdtc.ru> <20120307202914.GB9436@michelle.cdnetworks.com> In-Reply-To: <20120307202914.GB9436@michelle.cdnetworks.com> Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 8bit Cc: marius@freebsd.org, yongari@freebsd.org, "net@freebsd.org" Subject: Re: suboptimal bge(4) BCM5704 performance in RELENG_8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2012 08:12:37 -0000 08.03.2012 03:29, YongHyeon PYUN пишет: > On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote: >> Hi! >> >> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64 >> running busy icecast2 server in hope it can saturate 1G bge(4) link. >> >> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704). > > Would you show me the output of dmesg(bge(4) and brgphy(4) related > ones)? # egrep 'bge|brgphy' /var/run/dmesg.boot bge0: mem 0xfdf70000-0xfdf7ffff irq 25 at device 2.0 on pci2 bge0: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:18:fe:86:ab:f4 bge0: [ITHREAD] bge1: mem 0xfdf60000-0xfdf6ffff irq 26 at device 2.1 on pci2 bge1: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge1: Ethernet address: 00:18:fe:86:ab:f3 bge1: [ITHREAD] There are bge0 and bge1 but bge1 is not used and is down. >> Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300? >> > > Yes. [skip] >> I've glanced at bge(4) code and see it uses lots of hardcoded constants >> for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue. > > Firmware has a fixed number of descriptors so increasing them > wouldn't give better numbers. What it the limit? >> And no loader tunnables/sysctls like em/igb have. > > Show me the output of "sysctl dev.bge.0.stats". # sysctl dev.bge.0.stats dev.bge.0.stats.FramesDroppedDueToFilters: 0 dev.bge.0.stats.DmaWriteQueueFull: 84072 dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 dev.bge.0.stats.NoMoreRxBDs: 0 dev.bge.0.stats.InputDiscards: 0 dev.bge.0.stats.InputErrors: 30 dev.bge.0.stats.RecvThresholdHit: 745400662 dev.bge.0.stats.DmaReadQueueFull: 2020586592 dev.bge.0.stats.DmaReadHighPriQueueFull: 0 dev.bge.0.stats.SendDataCompQueueFull: 0 dev.bge.0.stats.RingSetSendProdIndex: 2832885493 dev.bge.0.stats.RingStatusUpdate: 899990835 dev.bge.0.stats.Interrupts: 899990835 dev.bge.0.stats.AvoidedInterrupts: 0 dev.bge.0.stats.SendThresholdHit: 0 dev.bge.0.stats.rx.ifHCInOctets: 491268800 dev.bge.0.stats.rx.Fragments: 234 dev.bge.0.stats.rx.UnicastPkts: 1977202324 dev.bge.0.stats.rx.MulticastPkts: 0 dev.bge.0.stats.rx.FCSErrors: 341 dev.bge.0.stats.rx.AlignmentErrors: 0 dev.bge.0.stats.rx.xonPauseFramesReceived: 0 dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 dev.bge.0.stats.rx.ControlFramesReceived: 0 dev.bge.0.stats.rx.xoffStateEntered: 0 dev.bge.0.stats.rx.FramesTooLong: 0 dev.bge.0.stats.rx.Jabbers: 0 dev.bge.0.stats.rx.UndersizePkts: 0 dev.bge.0.stats.rx.inRangeLengthError: 0 dev.bge.0.stats.rx.outRangeLengthError: 0 dev.bge.0.stats.tx.ifHCOutOctets: 683592718 dev.bge.0.stats.tx.Collisions: 0 dev.bge.0.stats.tx.XonSent: 0 dev.bge.0.stats.tx.XoffSent: 0 dev.bge.0.stats.tx.flowControlDone: 0 dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 dev.bge.0.stats.tx.SingleCollisionFrames: 0 dev.bge.0.stats.tx.MultipleCollisionFrames: 0 dev.bge.0.stats.tx.DeferredTransmissions: 0 dev.bge.0.stats.tx.ExcessiveCollisions: 0 dev.bge.0.stats.tx.LateCollisions: 0 dev.bge.0.stats.tx.UnicastPkts: 3292778353 dev.bge.0.stats.tx.MulticastPkts: 0 dev.bge.0.stats.tx.BroadcastPkts: 147 dev.bge.0.stats.tx.CarrierSenseErrors: 0 dev.bge.0.stats.tx.Discards: 0 dev.bge.0.stats.tx.Errors: 0 >> Should I try to play with constants in the code and if so, what are limits of this chip? >> Or it will never be capable of utilizing full gigabit speed? > > BCM5704 is old controller but it should have no problems to > saturate the link. I suspect it could be related with DMA > configuration but needs more information. I'll supply any information. It's also possible to try kernel patches, if any :-) Eugene Grosbein