Date: Thu, 16 Aug 2007 18:24:18 +0900 From: KAWAGUTI Ginga <ginga-freebsd@ginganet.org> To: freebsd-net@freebsd.org Cc: ginga-freebsd@ginganet.org Subject: Some Broadcom GbE NIC(bge driver) suffers packet loss at receiving Message-ID: <20070816092418.GH6523@ginganet.org>
next in thread | raw e-mail | index | archive | help
Hi.
Some revision of Broadcom GbE NIC with bge driver
seems to loss 1 packet per 1000 packet at receiving on some occasion.
Sending doesn't have this problem(receiving only).
This loss doesn't always happen,
but when happens, it seems that loss is exactly 1 per 1000,
as the last 3 digits of lost packets' sequence number are all same.
(i.e. example of lost packet seq-No. are:
3284, 4284, 5284, 8284, 9284, ...)
I've tested with:
* tcpdump capturing of SmartBits,
* ports/net/rude(network traffic generator/collector software)
* owping(one-way-ping)
This loss happens with:
* BCM5704(?) @ FreeBSD/i386 6-stable (Aug. 2007)
(6-stable of some months ago also suffered with this problem)
* same type hardware
(I tested with 2 HP DL385 servers, and happend with both H/W.)
This loss doesn't happen with:
* same hardware with Linux(Fedora-Core 5)
* same server with different NIC (Intel GbE with em driver)
also with same version of FreeBSD/i386 6-stable
* same FreeBSD/i386 6-stable with some other Broadcom cards(Broadcom BCM5752)
(not same PC/server, but other PC hardware)
-------------------------------------------------------
More precise hardware information is given below:
This happens with BCM5704 of this revision:
(on-board with HP DL385 Opteron Server)
bge0: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7df0000-0xf7dfffff irq 28 at device 6.0 on pci3
bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7df0000
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0: OUI 0x000818, model 0x0019, rev. 0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: bpf attached
bge0: Ethernet address: 00:17:a4:8f:22:bc
ioapic2: routing intpin 0 (PCI IRQ 28) to vector 53
bge0: [MPSAFE]
bge1: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7de0000-0xf7deffff irq 29 at device 6.1 on pci3
bge1: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7de0000
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1: OUI 0x000818, model 0x0019, rev. 0
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: bpf attached
bge1: Ethernet address: 00:17:a4:8f:22:bb
ioapic2: routing intpin 1 (PCI IRQ 29) to vector 54
bge1: [MPSAFE]
pciconf -lvc:
bge0@pci3:6:0: class=0x020000 card=0x00d00e11 chip=0x164814e4 rev=0x10 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'BCM5704 NetXtreme Dual Gigabit Adapter'
class = network
subclass = ethernet
cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
Other bge cards that doesn't suffer with this problem:
(HP xw8400 Workstation on-board)
bge0: <Broadcom BCM5752 A1, ASIC rev. 0x6001> mem 0xfb600000-0xfb60ffff irq 17 at device 0.0 on pci31
bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xfb600000
bge0: Disabling fastboot
bge0: Disabling fastboot
miibus0: <MII bus> on bge0
brgphy0: <BCM5752 10/100/1000baseTX PHY> on miibus0
brgphy0: OUI 0x000818, model 0x0010, rev. 0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: bpf attached
bge0: Ethernet address: 00:17:08:2a:a3:a3
ioapic0: routing intpin 17 (PCI IRQ 17) to vector 49
bge0: [MPSAFE]
pciconf -lvc:
bge0@pci31:0:0: class=0x020000 card=0x3015103c chip=0x160014e4 rev=0x01 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'Broadcom NetXtreme Gigabit Ethernet'
class = network
subclass = ethernet
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
cap 10[d0] = PCI-Express 1 endpoint
-------------------------------------------------------
#######################################################
Example usage of ports/net/rude (and crude) to test this.
#######################################################
(At packet sender:)
sender% cat > rude.cfg << EOFEOF
START NOW
1000 0030 ON 3002 10.1.1.1:10001 CONSTANT 10000 200
11000 0030 OFF
EOFEOF
sender%
# Above config will send 10000 packets/sec for 10 secs.
# ==> 100001 packets will be generated.
# (+1 comes from rude's spec(bug?))
sender% rude -s rude.cfg (Run packet generator)
rude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo
rude comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under GNU GENERAL PUBLIC LICENSE Version 2.
F_ID: F_START: F_STOP: F_SPORT: F_DADD: F_DPORT: F_err: F_suc: F_seq: F_TYPE: [+ type params]
30 1187254729.594905 1187254739.594905 3002 129.60.75.70 10001 0 100001 100001 CBR [r:10000 s:200]
sender%
#######################################################
(At packet receiver; which have bge cards)
receiver% crude -s 30
crude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo
crude comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under GNU GENERAL PUBLIC LICENSE Version 2.
^C (Press Ctrl+C after rude@sender has finished)
Runtime statistics results:
---------------------------
Flow_ID=30
Packets: received=100001 out-of-seq=0 lost(est)=0
Total bytes received=20000200
Sequence numbers: first=0 last=100000
Delay: average = -0.001333 jitter=0.000069 seconds
Absolute maximum jitter=0.008760 seconds
Throughput=2.00001e+06 Bps (from first to last packet received)
receiver%
-------------------------------------------------------
For good GbE-cards, packet received count should be equal with
sender's packet count.
For some bge cards mentioned above, lost packet percentage
comes ~ 0.1%, and when looking to sequence numbers,
the loss seem to happen every 1000 packets.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070816092418.GH6523>