Date: Wed, 07 Mar 2012 17:59:15 +0700 From: Eugene Grosbein <egrosbein@rdtc.ru> To: pyunyh@gmail.com Cc: marius@freebsd.org, "net@freebsd.org" <net@freebsd.org>, yongari@freebsd.org Subject: Re: suboptimal bge(4) BCM5704 performance in RELENG_8 Message-ID: <4F573F83.5090502@rdtc.ru> In-Reply-To: <20120308034345.GD9436@michelle.cdnetworks.com> References: <4F5608EA.6080705@rdtc.ru> <20120307202914.GB9436@michelle.cdnetworks.com> <4F571870.3090902@rdtc.ru> <20120308034345.GD9436@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
08.03.2012 10:43, YongHyeon PYUN пишет:
>>>> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64
>>>> running busy icecast2 server in hope it can saturate 1G bge(4) link.
>>>>
>>>> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704).
>>>
>>> Would you show me the output of dmesg(bge(4) and brgphy(4) related
>>> ones)?
[skip]
>
> It seems your controller is BCM5704 B2 and it is running in PCI-X
> mode. There is one known BCM5704 issue but I don't think you're
> seeing this one. To confirm that could you show me both the output
> of "pciconf -lcbv" and "devinfo -rv"?
# pciconf -lcbv
hostb0@pci0:0:0:0: class=0x060000 card=0x32000e11 chip=0x35908086 rev=0x0c hdr=0x00
vendor = 'Intel Corporation'
device = 'E7520 Server Memory Controller Hub'
class = bridge
subclass = HOST-PCI
cap 09[40] = vendor (length 5) Intel cap 4 version 1
pcib1@pci0:0:2:0: class=0x060400 card=0x00000000 chip=0x35958086 rev=0x0c hdr=0x01
vendor = 'Intel Corporation'
device = 'E752x Memory Controller Hub PCIe Port A0'
class = bridge
subclass = PCI-PCI
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 05[58] = MSI supports 2 messages
cap 10[64] = PCI-Express 1 root port max data 256(256) link x0(x8)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
pcib2@pci0:0:4:0: class=0x060400 card=0x00000000 chip=0x35978086 rev=0x0c hdr=0x01
vendor = 'Intel Corporation'
device = 'E752x Memory Controller Hub PCIe Port B0'
class = bridge
subclass = PCI-PCI
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 05[58] = MSI supports 2 messages
cap 10[64] = PCI-Express 1 root port max data 256(256) link x8(x8)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
pcib5@pci0:0:6:0: class=0x060400 card=0x00000000 chip=0x35998086 rev=0x0c hdr=0x01
vendor = 'Intel Corporation'
device = 'E752x Memory Controller Hub PCIe Port C0'
class = bridge
subclass = PCI-PCI
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 05[58] = MSI supports 2 messages
cap 10[64] = PCI-Express 1 root port max data 256(256) link x0(x8)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
pcib6@pci0:0:28:0: class=0x060400 card=0x00000000 chip=0x25ae8086 rev=0x02 hdr=0x01
vendor = 'Intel Corporation'
device = 'Hub Interface to PCI-X Bridge (6300ESB)'
class = bridge
subclass = PCI-PCI
cap 07[50] = PCI-X 64-bit bridge
uhci0@pci0:0:29:0: class=0x0c0300 card=0x32010e11 chip=0x25a98086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'USB 1.1 UHCI Controller *1 (6300ESB)'
class = serial bus
subclass = USB
bar [20] = type I/O Port, range 32, base 0x2000, size 32, enabled
uhci1@pci0:0:29:1: class=0x0c0300 card=0x32010e11 chip=0x25aa8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'USB 1.1 UHCI Controller *2 (6300ESB)'
class = serial bus
subclass = USB
bar [20] = type I/O Port, range 32, base 0x2020, size 32, enabled
none0@pci0:0:29:4: class=0x088000 card=0x32010e11 chip=0x25ab8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Watchdog Timer (6300ESB)'
class = base peripheral
bar [10] = type Memory, range 32, base 0xfbef0000, size 16, enabled
ioapic0@pci0:0:29:5: class=0x080020 card=0x32010e11 chip=0x25ac8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '6300ESB I/O Advanced Programmable Interrupt Controller'
class = base peripheral
subclass = interrupt controller
cap 07[50] = PCI-X 64-bit supports 512 burst read, 1 split transaction
ehci0@pci0:0:29:7: class=0x0c0320 card=0x32010e11 chip=0x25ad8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'USB 2.0 EHCI Controller (6300ESB)'
class = serial bus
subclass = USB
bar [10] = type Memory, range 32, base 0xfbee0000, size 1024, enabled
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 0a[58] = EHCI Debug Port at offset 0x80 in map 0x14
pcib7@pci0:0:30:0: class=0x060400 card=0x00000000 chip=0x244e8086 rev=0x0a hdr=0x01
vendor = 'Intel Corporation'
device = '82801 Family (ICH2/3/4/5/6/7/8/9,63xxESB) Hub Interface to PCI Bridge'
class = bridge
subclass = PCI-PCI
isab0@pci0:0:31:0: class=0x060100 card=0x00000000 chip=0x25a18086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '6300ESB LPC Inteface Controller'
class = bridge
subclass = PCI-ISA
atapci0@pci0:0:31:1: class=0x01018a card=0x32010e11 chip=0x25a28086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'IDE Controller (6300ESB)'
class = mass storage
subclass = ATA
bar [10] = type I/O Port, range 32, base 0x1f0, size 8, enabled
bar [14] = type I/O Port, range 32, base 0x3f4, size 1, enabled
bar [18] = type I/O Port, range 32, base 0x170, size 8, enabled
bar [1c] = type I/O Port, range 32, base 0x374, size 1, enabled
bar [20] = type I/O Port, range 32, base 0x500, size 16, enabled
pcib3@pci0:6:0:0: class=0x060400 card=0x00000000 chip=0x03298086 rev=0x09 hdr=0x01
vendor = 'Intel Corporation'
device = 'PCI Express-to-PCI Express Bridge A (6700PXH)'
class = bridge
subclass = PCI-PCI
cap 10[44] = PCI-Express 1 PCI bridge max data 256(256) link x8(x8)
cap 05[5c] = MSI supports 1 message, 64 bit
cap 01[6c] = powerspec 2 supports D0 D3 current D0
cap 07[d8] = PCI-X bridge
ecap 0001[100] = AER 1 1 fatal 0 non-fatal 0 corrected
ecap 0004[300] = unknown 1
pcib4@pci0:6:0:2: class=0x060400 card=0x00000000 chip=0x032a8086 rev=0x09 hdr=0x01
vendor = 'Intel Corporation'
device = 'PCI Express-to-PCI Express Bridge B (6700PXH)'
class = bridge
subclass = PCI-PCI
cap 10[44] = PCI-Express 1 PCI bridge max data 256(256) link x8(x8)
cap 05[5c] = MSI supports 1 message, 64 bit
cap 01[6c] = powerspec 2 supports D0 D3 current D0
cap 07[d8] = PCI-X bridge
ecap 0001[100] = AER 1 1 fatal 0 non-fatal 0 corrected
ecap 0004[300] = unknown 1
ciss0@pci0:2:1:0: class=0x010400 card=0x40910e11 chip=0x00460e11 rev=0x01 hdr=0x00
vendor = 'Compaq Computer Corp (Now owned by Hewlett-Packard)'
device = 'Smart Array 6400 Controller (N/A)'
class = mass storage
subclass = RAID
bar [10] = type Memory, range 64, base 0xfdff0000, size 8192, enabled
bar [18] = type I/O Port, range 32, base 0x4000, size 256, enabled
bar [1c] = type Memory, range 64, base 0xfdf80000, size 262144, enabled
cap 01[d0] = powerspec 2 supports D0 D1 D3 current D0
cap 07[dc] = PCI-X 64-bit supports 133MHz, 2048 burst read, 8 split transactions
cap 03[f0] = VPD
bge0@pci0:2:2:0: class=0x020000 card=0x00d00e11 chip=0x164814e4 rev=0x10 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme Dual Gigabit Adapter (BCM5704)'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xfdf70000, size 65536, enabled
cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
bge1@pci0:2:2:1: class=0x020000 card=0x00d00e11 chip=0x164814e4 rev=0x10 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme Dual Gigabit Adapter (BCM5704)'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xfdf60000, size 65536, enabled
cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
vgapci0@pci0:1:3:0: class=0x030000 card=0x001e0e11 chip=0x47521002 rev=0x27 hdr=0x00
vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.'
device = 'ATI On-Board VGA for HP Proliant 350 G3 (Rage XL PCI)'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xfc000000, size 16777216, enabled
bar [14] = type I/O Port, range 32, base 0x3000, size 256, enabled
bar [18] = type Memory, range 32, base 0xfbff0000, size 4096, enabled
cap 01[5c] = powerspec 2 supports D0 D1 D2 D3 current D0
none1@pci0:1:4:0: class=0x088000 card=0xb2060e11 chip=0xb2030e11 rev=0x01 hdr=0x00
vendor = 'Compaq Computer Corp (Now owned by Hewlett-Packard)'
device = 'Integrated Lights Out Processor (iLo)'
class = base peripheral
bar [10] = type I/O Port, range 32, base 0x1800, size 256, enabled
bar [14] = type Memory, range 32, base 0xfbfe0000, size 512, enabled
cap 01[f0] = powerspec 2 supports D0 D3 current D0
none2@pci0:1:4:2: class=0x088000 card=0xb2060e11 chip=0xb2040e11 rev=0x01 hdr=0x00
vendor = 'Compaq Computer Corp (Now owned by Hewlett-Packard)'
device = 'Integrated Lights Out Processor (iLo)'
class = base peripheral
bar [10] = type I/O Port, range 32, base 0x3400, size 256, enabled
bar [14] = type Memory, range 32, base 0xfbfd0000, size 2048, enabled
bar [18] = type Memory, range 32, base 0xfbfc0000, size 8192, enabled
bar [1c] = type Memory, range 32, base 0xfbf00000, size 524288, enabled
cap 01[f0] = powerspec 2 supports D0 D3 current D0
# devinfo -rv
nexus0
cryptosoft0
apic0
I/O memory addresses:
0xfec00000-0xfec0001f
0xfec10000-0xfec1001f
0xfec82000-0xfec8201f
0xfec82400-0xfec8241f
0xfee00000-0xfee003ff
ram0
I/O memory addresses:
0x0-0x9f3ff
0x100000-0x7fff2fff
acpi0
Interrupt request lines:
9
I/O ports:
0x10-0x1f
0x20-0x3f
0x50-0x53
0x70-0x77
0x90-0x9f
0xa0-0xbf
0x2f8-0x2ff
0x408-0x40f
0x4d0-0x4d1
0x700-0x71f
0x800-0x83f
0x900-0x97f
0xc80-0xc83
0xcd4-0xcd7
0xf50-0xf58
I/O memory addresses:
0xe0000000-0xefffffff
cpu0 pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU0
p4tcc0
cpufreq0
cpu1 pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU1
p4tcc1
cpufreq1
cpu2 pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU2
p4tcc2
cpufreq2
cpu3 pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU3
p4tcc3
cpufreq3
unknown pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU4
unknown pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU5
unknown pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU6
unknown pnpinfo _HID=none _UID=0 at handle=\_PR_.CPU7
pcib0 pnpinfo _HID=PNP0A03 _UID=0 at handle=\_SB_.PCI0
pci0
I/O memory addresses:
0xfbef0000-0xfbef000f
hostb0 pnpinfo vendor=0x8086 device=0x3590 subvendor=0x0e11 subdevice=0x3200 class=0x060000 at slot=0 function=0 handle=\_SB_.PCI0.CFG0
pcib1 pnpinfo vendor=0x8086 device=0x3595 subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=2 function=0 handle=\_SB_.PCI0.PTA0
pci13
pcib2 pnpinfo vendor=0x8086 device=0x3597 subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=4 function=0 handle=\_SB_.PCI0.PTB0
pci6
pcib3 pnpinfo vendor=0x8086 device=0x0329 subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=0 function=0 handle=\_SB_.PCI0.PTB0.PCXA
pci7
pcib4 pnpinfo vendor=0x8086 device=0x032a subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=0 function=2 handle=\_SB_.PCI0.PTB0.PCXB
pci10
pcib5 pnpinfo vendor=0x8086 device=0x3599 subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=6 function=0 handle=\_SB_.PCI0.PTC0
pci3
pcib6 pnpinfo vendor=0x8086 device=0x25ae subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=28 function=0 handle=\_SB_.PCI0.ICHR
pci2
I/O ports:
0x4000-0x40ff
I/O memory addresses:
0xfdf80000-0xfdfbffff
ciss0 pnpinfo vendor=0x0e11 device=0x0046 subvendor=0x0e11 subdevice=0x4091 class=0x010400 at slot=1 function=0
Interrupt request lines:
24
I/O memory addresses:
0xfdff0000-0xfdff1fff
bge0 pnpinfo vendor=0x14e4 device=0x1648 subvendor=0x0e11 subdevice=0x00d0 class=0x020000 at slot=2 function=0
Interrupt request lines:
25
I/O memory addresses:
0xfdf70000-0xfdf7ffff
miibus0
brgphy0 pnpinfo oui=0x818 model=0x19 rev=0x0 at phyno=1
bge1 pnpinfo vendor=0x14e4 device=0x1648 subvendor=0x0e11 subdevice=0x00d0 class=0x020000 at slot=2 function=1
Interrupt request lines:
26
I/O memory addresses:
0xfdf60000-0xfdf6ffff
miibus1
brgphy1 pnpinfo oui=0x818 model=0x19 rev=0x0 at phyno=1
uhci0 pnpinfo vendor=0x8086 device=0x25a9 subvendor=0x0e11 subdevice=0x3201 class=0x0c0300 at slot=29 function=0
Interrupt request lines:
16
I/O ports:
0x2000-0x201f
usbus0
uhub0
uhci1 pnpinfo vendor=0x8086 device=0x25aa subvendor=0x0e11 subdevice=0x3201 class=0x0c0300 at slot=29 function=1
Interrupt request lines:
19
I/O ports:
0x2020-0x203f
usbus1
uhub1
unknown pnpinfo vendor=0x8086 device=0x25ab subvendor=0x0e11 subdevice=0x3201 class=0x088000 at slot=29 function=4
ioapic0 pnpinfo vendor=0x8086 device=0x25ac subvendor=0x0e11 subdevice=0x3201 class=0x080020 at slot=29 function=5
ehci0 pnpinfo vendor=0x8086 device=0x25ad subvendor=0x0e11 subdevice=0x3201 class=0x0c0320 at slot=29 function=7
Interrupt request lines:
23
I/O memory addresses:
0xfbee0000-0xfbee03ff
usbus2
uhub2
pcib7 pnpinfo vendor=0x8086 device=0x244e subvendor=0x0000 subdevice=0x0000 class=0x060400 at slot=30 function=0 handle=\_SB_.PCI0.IP2P
pci1
I/O ports:
0x1800-0x18ff
0x3000-0x30ff
0x3400-0x34ff
I/O memory addresses:
0xfbf00000-0xfbf7ffff
0xfbfc0000-0xfbfc1fff
0xfbfd0000-0xfbfd07ff
0xfbfe0000-0xfbfe01ff
0xfbff0000-0xfbff0fff
0xfc000000-0xfcffffff
vgapci0 pnpinfo vendor=0x1002 device=0x4752 subvendor=0x0e11 subdevice=0x001e class=0x030000 at slot=3 function=0
drm0
unknown pnpinfo vendor=0x0e11 device=0xb203 subvendor=0x0e11 subdevice=0xb206 class=0x088000 at slot=4 function=0 handle=\_SB_.PCI0.IP2P.ASMD
unknown pnpinfo vendor=0x0e11 device=0xb204 subvendor=0x0e11 subdevice=0xb206 class=0x088000 at slot=4 function=2
isab0 pnpinfo vendor=0x8086 device=0x25a1 subvendor=0x0000 subdevice=0x0000 class=0x060100 at slot=31 function=0 handle=\_SB_.PCI0.IBRG
isa0
orm0
I/O memory addresses:
0xc0000-0xc7fff
0xc8000-0xcbfff
0xee000-0xeffff
ichwd0
sc0
vga0
I/O ports:
0x3c0-0x3df
I/O memory addresses:
0xa0000-0xbffff
atrtc0
Interrupt request lines:
8
ACPI I/O ports:
0x70-0x71
ppc0
uart1
Interrupt request lines:
3
ACPI I/O ports:
0x2f8-0x2ff
atapci0 pnpinfo vendor=0x8086 device=0x25a2 subvendor=0x0e11 subdevice=0x3201 class=0x01018a at slot=31 function=1
I/O ports:
0x170-0x177
0x1f0-0x1f7
0x376
0x3f6
0x500-0x50f
ata0 at channel=0
Interrupt request lines:
14
acd0
ata1 at channel=1
Interrupt request lines:
15
acpi_sysresource0 pnpinfo _HID=PNP0C02 _UID=0 at handle=\_SB_.PCI0.IBRG.MOMB
attimer0 pnpinfo _HID=PNP0100 _UID=0 at handle=\_SB_.PCI0.IBRG.TIME
atdma0 pnpinfo _HID=PNP0200 _UID=0 at handle=\_SB_.PCI0.IBRG.DMA0
unknown pnpinfo _HID=PNP0800 _UID=0 at handle=\_SB_.PCI0.IBRG.BEEP
atkbdc0 pnpinfo _HID=PNP0303 _UID=0 at handle=\_SB_.PCI0.IBRG.KBD_
I/O ports:
0x60
0x64
atkbd0
Interrupt request lines:
1
psm0
Interrupt request lines:
12
psmcpnp0 pnpinfo _HID=PNP0F13 _UID=0 at handle=\_SB_.PCI0.IBRG.PS2M
unknown pnpinfo _HID=PNP0A06 _UID=0 at handle=\_SB_.PCI0.IBRG.S417
uart0 pnpinfo _HID=PNP0501 _UID=0 at handle=\_SB_.PCI0.IBRG.S417.COMA
Interrupt request lines:
4
I/O ports:
0x3f8-0x3ff
fdc0 pnpinfo _HID=PNP0700 _UID=0 at handle=\_SB_.PCI0.IBRG.S417.FDC0
Interrupt request lines:
6
DMA request lines:
2
I/O ports:
0x3f2-0x3f5
0x3f7
fd0
pci_link0 pnpinfo _HID=PNP0C0F _UID=1 at handle=\_SB_.LNKA
pci_link1 pnpinfo _HID=PNP0C0F _UID=2 at handle=\_SB_.LNKB
pci_link2 pnpinfo _HID=PNP0C0F _UID=3 at handle=\_SB_.LNKC
pci_link3 pnpinfo _HID=PNP0C0F _UID=4 at handle=\_SB_.LNKD
pci_link4 pnpinfo _HID=PNP0C0F _UID=5 at handle=\_SB_.LNKE
pci_link5 pnpinfo _HID=PNP0C0F _UID=6 at handle=\_SB_.LNKF
pci_link6 pnpinfo _HID=PNP0C0F _UID=7 at handle=\_SB_.LNKG
pci_link7 pnpinfo _HID=PNP0C0F _UID=8 at handle=\_SB_.LNKH
acpi_tz0 pnpinfo _HID=none _UID=0 at handle=\_TZ_.THM0
acpi_timer0 pnpinfo unknown at unknown
ACPI I/O ports:
0x908-0x90b
>> # sysctl dev.bge.0.stats
>> dev.bge.0.stats.FramesDroppedDueToFilters: 0
>> dev.bge.0.stats.DmaWriteQueueFull: 84072
>> dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
>> dev.bge.0.stats.NoMoreRxBDs: 0
>> dev.bge.0.stats.InputDiscards: 0
>> dev.bge.0.stats.InputErrors: 30
>> dev.bge.0.stats.RecvThresholdHit: 745400662
>> dev.bge.0.stats.DmaReadQueueFull: 2020586592
>> dev.bge.0.stats.DmaReadHighPriQueueFull: 0
>> dev.bge.0.stats.SendDataCompQueueFull: 0
>> dev.bge.0.stats.RingSetSendProdIndex: 2832885493
>> dev.bge.0.stats.RingStatusUpdate: 899990835
>> dev.bge.0.stats.Interrupts: 899990835
>> dev.bge.0.stats.AvoidedInterrupts: 0
>> dev.bge.0.stats.SendThresholdHit: 0
>> dev.bge.0.stats.rx.ifHCInOctets: 491268800
>> dev.bge.0.stats.rx.Fragments: 234
>> dev.bge.0.stats.rx.UnicastPkts: 1977202324
>> dev.bge.0.stats.rx.MulticastPkts: 0
>> dev.bge.0.stats.rx.FCSErrors: 341
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> You have multiple FCS and Input errors. Check signal
> quality(i.e. UTP cable).
Thanks, I'll check.
> Does netperf benchmark also show 540Mbps on bge(4)?
I'll try to run benchmarks if I find a peer for netperf.
Eugene Grosbein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F573F83.5090502>