Date: Wed, 12 Nov 2008 23:36:02 GMT From: Aurélien Méré <freebsd@amc-os.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/128833: Network packets corrupted when bge card is in 64-bit PCI slot Message-ID: <200811122336.mACNa2nj027464@www.freebsd.org> Resent-Message-ID: <200811122340.mACNe1f7077030@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 128833
>Category: kern
>Synopsis: Network packets corrupted when bge card is in 64-bit PCI slot
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Nov 12 23:40:00 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Aurélien Méré
>Release: 7.1-STABLE
>Organization:
AMC-OS Development Team
>Environment:
FreeBSD vodka.adriana.amc-os.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Tue Nov 11 19:41:15 CET 2008 root@vodka.adriana.amc-os.com:/mnt/usr/obj/mnt/usr/src/sys/VODKA-1.0.5 i386
>Description:
Server is based on an Asus A7M766-D motherboard with 2 Athlon MP processors.
When plugged in a 66MHz 64-bit port, the 3Com 3C996 1000-SX PCI-X (bge) network card works incorrectly, while everything is fine in a classic 32-bit port. I tried disabling specific card features, like TX and RX checksums but the result is worse, no traffic goes outside. Problem is that some packets have their content changed, leading to incorrect decoding by applications mostly ending with "protocol error".
In a tcpdump -XX, that's the kind of change that is observable, just in the middle of the packet (here in an SSH initiation), at 0x78 for example :
SERVER OK SEND :
0x0000: 000a 5e62 3282 0060 97a0 11cd 0800 4500 ..^b2..`......E.
0x0010: 0314 8b24 4000 4006 1ea1 c0a8 0606 c0a8 ...$@.@.........
0x0020: 06c8 0016 febc 4f32 b113 8da3 9cf9 8018 ......O2........
0x0030: 2086 4bf8 0000 0101 080a 2c06 63de 0004 ..K.......,.c...
0x0040: 7de2 0000 02dc 0a14 871a a2ba 5a09 5823 }...........Z.X#
0x0050: 7431 4bb8 32f0 08a3 0000 007e 6469 6666 t1K.2......~diff
0x0060: 6965 2d68 656c 6c6d 616e 2d67 726f 7570 ie-hellman-group
0x0070: 2d65 7863 6861 6e67 652d 7368 6132 3536 -exchange-sha256
0x0080: 2c64 6966 6669 652d 6865 6c6c 6d61 6e2d ,diffie-hellman-
0x0090: 6772 6f75 702d 6578 6368 616e 6765 2d73 group-exchange-s
0x00a0: 6861 312c 6469 6666 6965 2d68 656c 6c6d ha1,diffie-hellm
0x00b0: 616e 2d67 726f 7570 3134 2d73 6861 312c an-group14-sha1,
0x00c0: 6469 6666 6965 2d68 656c 6c6d 616e 2d67 diffie-hellman-g
0x00d0: 726f 7570 312d 7368 6131 0000 0007 7373 roup1-sha1....ss
0x00e0: 682d 6473 7300 0000 9d61 6573 3132 382d h-dss....aes128-
0x00f0: 6362 632c 3364 6573 2d63 6263 2c62 6c6f cbc,3des-cbc,blo
0x0100: 7766 6973 682d 6362 632c 6361 7374 3132 wfish-cbc,cast12
0x0110: 382d 6362 632c 6172 6366 6f75 7231 3238 8-cbc,arcfour128
SERVER NOK RECEIVE :
0x0000: 000a 5e62 3282 0060 97a0 11cd 0800 4500 ..^b2..`......E.
0x0010: 0314 8b24 4000 4006 1ea1 c0a8 0606 c0a8 ...$@.@.........
0x0020: 06c8 0016 febc 4f32 b113 8da3 9cf9 8018 ......O2........
0x0030: 2086 4bf8 0000 0101 080a 2c06 63de 0004 ..K.......,.c...
0x0040: 7de2 0000 02dc 0a14 871a a2ba 5a09 5823 }...........Z.X#
0x0050: 7431 4bb8 32f0 08a3 0000 007e 6469 6666 t1K.2......~diff
0x0060: 6965 2d68 656c 6c6d 616e 2d67 726f 7570 ie-hellman-group
0x0070: 2d65 7863 6861 6e67 2c64 6966 6669 652d -exchang,diffie-
0x0080: 2c64 6966 6669 652d 6865 6c6c 6d61 6e2d ,diffie-hellman-
0x0090: 6772 6f75 702d 6578 6368 616e 6765 2d73 group-exchange-s
0x00a0: 6861 312c 6469 6666 6965 2d68 656c 6c6d ha1,diffie-hellm
0x00b0: 616e 2d67 726f 7570 3134 2d73 6861 312c an-group14-sha1,
0x00c0: 6469 6666 6965 2d68 656c 6c6d 616e 2d67 diffie-hellman-g
0x00d0: 726f 7570 312d 7368 6131 0000 0007 7373 roup1-sha1....ss
0x00e0: 682d 6473 7300 0000 9d61 6573 3132 382d h-dss....aes128-
0x00f0: 6362 632c 3364 6573 7766 6973 682d 6362 cbc,3deswfish-cb
0x0100: 7766 6973 682d 6362 632c 6361 7374 3132 wfish-cbc,cast12
0x0110: 382d 6362 632c 6172 6366 6f75 7231 3238 8-cbc,arcfour128
There seems to be some buffer issue on the reception side. The problems appear only on the RX, packets are correctly received (well identical at the tcpdump level anyway) on the destination server.
Follows ifconfig, dmesg and pciconf
Thanks for your help,
Aurélien
--
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0a:5e:62:32:82
inet 192.168.6.200 netmask 0xffffff00 broadcast 192.168.6.255
media: Ethernet autoselect (1000baseSX <full-duplex>)
status: active
FreeBSD 7.1-PRERELEASE #2: Tue Nov 11 19:41:15 CET 2008
root@vodka.adriana.amc-os.com:/mnt/usr/obj/mnt/usr/src/sys/VODKA-1.0.5
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(TM) MP 2000+ (1666.74-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x662 Stepping = 2
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
AMD Features=0xc0480800<SYSCALL,MP,MMX+,3DNow!+,3DNow!>
real memory = 805306368 (768 MB)
avail memory = 778317824 (742 MB)
MPTable: <ASUS PROD00000000>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Assuming intbase of 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
smbios0: <System Management BIOS> at iomem 0xf3b60-0xf3b7e on motherboard
smbios0: Version: 2.3, BCD Revision: 2.3
cryptosoft0: <software crypto> on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
agp0: <AMD 762 host to AGP bridge> on hostb0
pcib1: <MPTable PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd800-0xd8ff mem 0xf4000000-0xf5ffffff,0xf8000000-0xf9ffffff irq 16 at device 5.0 on pci1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <AMD 768 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb800-0xb80f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
amdpm0: <AMD 756/766/768/8111 Power Management Controller> port 0xe4e0-0xe4ff at device 7.3 on pci0
smbus0: <System Management Bus> on amdpm0
smb0: <SMBus generic I/O> on smbus0
bge0: <3Com Gigabit Fiber-SX Server NIC, ASIC rev. 0x105> mem 0xf3800000-0xf380ffff irq 16 at device 8.0 on pci0
bge0: Ethernet address: 00:0a:5e:62:32:82
bge0: [ITHREAD]
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xb400-0xb4ff mem 0xf3000000-0xf30000ff irq 17 at device 9.0 on pci0
re0: Chip rev. 0x04000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: Ethernet address: 00:18:4d:79:72:65
re0: [FILTER]
pcib2: <MPTable PCI-PCI bridge> at device 16.0 on pci0
pci2: <PCI bus> on pcib2
atapci1: <SiI SiI 3114 SATA150 controller> port 0xa800-0xa807,0xa400-0xa403,0xa000-0xa007,0x9800-0x9803,0x9400-0x940f mem 0xf2000000-0xf20003ff irq 18 at device 5.0 on pci2
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
atapci2: <VIA 6421 SATA150 controller> port 0x9000-0x900f,0x8800-0x880f,0x8400-0x840f,0x8000-0x800f,0x7800-0x781f,0x7400-0x74ff irq 17 at device 6.0 on pci2
atapci2: [ITHREAD]
ata6: <ATA channel 0> on atapci2
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci2
ata7: [ITHREAD]
ata8: <ATA channel 2> on atapci2
ata8: [ITHREAD]
atapci3: <Promise PDC20268 UDMA100 controller> port 0x7000-0x7007,0x6800-0x6803,0x6400-0x6407,0x6000-0x6003,0x5800-0x580f mem 0xf1800000-0xf1803fff irq 19 at device 8.0 on pc i2
atapci3: [ITHREAD]
ata9: <ATA channel 0> on atapci3
ata9: [ITHREAD]
ata10: <ATA channel 1> on atapci3
ata10: [ITHREAD]
cpu0 on motherboard
cpu1 on motherboard
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcc7ff,0xd0000-0xd27ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0700> can't assign resources (port)
unknown: <PNP0c01> can't assign resources (memory)
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0c02> can't assign resources (port)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging unlimited
ad0: 15488MB <SMI MODEL 20071116> at ata0-master WDMA2
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719407
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719420
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719423
ad4: 305245MB <Hitachi HDT725032VLA360 V54OA52A> at ata2-master SATA150
ad8: 238475MB <Hitachi HDT725025VLA380 V5DOA7EA> at ata4-master SATA150
ad10: 305245MB <Hitachi HDT725032VLA360 V54OA52A> at ata5-master SATA150
ad12: 953869MB <WDC WD10EACS-00D6B0 01.01A01> at ata6-master SATA150
ad14: 476940MB <Hitachi HDP725050GLA360 GM4OA52A> at ata7-master SATA150
ad18: 190782MB <HDT722520DLAT80 V44OA70A> at ata9-master UDMA100
ad20: 19077MB <HITACHI DK23DA-20 00J2A0A1> at ata10-master UDMA100
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719423
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719423
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719423
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=31719423
SMP: AP CPU #1 Launched!
hostb0@pci0:0:0:0: class=0x060000 card=0x00000000 chip=0x700c1022 rev=0x11 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-762 CPU to PCI Bridge (SMP chipset)'
class = bridge
subclass = HOST-PCI
cap 02[a0] = AGP 2x 1x SBA disabled
pcib1@pci0:0:1:0: class=0x060400 card=0x00000000 chip=0x700d1022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-762 CPU to PCI Bridge (AGP 4x)'
class = bridge
subclass = PCI-PCI
isab0@pci0:0:7:0: class=0x060100 card=0x80441043 chip=0x74401022 rev=0x05 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-768 (Opus) PCI to ISA/LPC Bridge'
class = bridge
subclass = PCI-ISA
atapci0@pci0:0:7:1: class=0x01018a card=0x74411022 chip=0x74411022 rev=0x04 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-768 (Opus) EIDE Controller'
class = mass storage
subclass = ATA
amdpm0@pci0:0:7:3: class=0x068000 card=0x80441043 chip=0x74431022 rev=0x03 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-768 (Opus) ACPI Controller'
class = bridge
bge0@pci0:0:8:0: class=0x020000 card=0x100410b7 chip=0x164514e4 rev=0x15 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'BCM5701 NetXtreme BCM5701 Gigabit Ethernet'
class = network
subclass = ethernet
cap 07[40] = PCI-X 64-bit supports 133MHz, 512 burst read, 1 split transaction
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
re0@pci0:0:9:0: class=0x020000 card=0x311a1385 chip=0x816910ec rev=0x10 hdr=0x00
vendor = 'Realtek Semiconductor'
device = 'RTL8110SB Single-Chip Gigabit LOM Ethernet Controller'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D1 D2 D3 current D0
pcib2@pci0:0:16:0: class=0x060400 card=0x00000000 chip=0x74481022 rev=0x05 hdr=0x01
vendor = 'Advanced Micro Devices (AMD)'
device = 'AMD-768 (Opus) PCI Bridge'
class = bridge
subclass = PCI-PCI
vgapci0@pci0:1:5:0: class=0x030000 card=0x0030121a chip=0x0005121a rev=0x01 hdr=0x00
vendor = '3dfx Interactive Inc'
device = 'Voodoo3 All Voodoo3 chips, 3000'
class = display
subclass = VGA
cap 02[54] = AGP 2x 1x SBA disabled
cap 01[60] = powerspec 1 supports D0 D3 current D0
atapci1@pci0:2:5:0: class=0x010400 card=0x61141095 chip=0x31141095 rev=0x02 hdr=0x00
vendor = 'Silicon Image Inc (Was: CMD Technology Inc)'
device = 'Sil 3114 SATALink/SATARaid Controller'
class = mass storage
subclass = RAID
cap 01[60] = powerspec 2 supports D0 D1 D2 D3 current D0
atapci2@pci0:2:6:0: class=0x010400 card=0x32491106 chip=0x32491106 rev=0x50 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'VT6421 VIA VT6421 RAID Controller'
class = mass storage
subclass = RAID
cap 01[e0] = powerspec 2 supports D0 D3 current D0
atapci3@pci0:2:8:0: class=0x018085 card=0x4d68105a chip=0x4d68105a rev=0x02 hdr=0x00
vendor = 'Promise Technology Inc'
device = 'PDC20268 Ultra100 TX2 EIDE Controller'
class = mass storage
cap 01[60] = powerspec 1 supports D0 D1 D3 current D0
>How-To-Repeat:
Any kind of protocol with large packets (ICMP seems to look fine for example) but dns, http, ssh communication mostly fails with "protocol error", "packet error" or so.
Other example with HTTP (Location URL corrupted) :
root@vodka:~> telnet www.freebsd.org 80
Trying 69.147.83.33...
Connected to www.freebsd.org.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.0 301 Moved Permanently
Connection: close
Locttp://wwttp://www.freebsd.org/
Content-Length: 0
Date: Wed, 12 Nov 2008 23:33:39 GMT
Server: httpd/1.4.x LaHonda
Connection closed by foreign host.
>Fix:
Putting the PCI card in a classic 32 bit port fixes the problem. Here are dmesg and pciconf changes :
bge0@pci0:2:8:0: class=0x020000 card=0x100410b7 chip=0x164514e4 rev=0x15 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'BCM5701 NetXtreme BCM5701 Gigabit Ethernet'
class = network
subclass = ethernet
cap 07[40] = PCI-X 64-bit supports 133MHz, 512 burst read, 1 split transaction
cap 01[48] = powerspec 2 supports D0 D3 current D0
cap 03[50] = VPD
cap 05[58] = MSI supports 8 messages, 64 bit
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xb400-0xb4ff mem 0xf3800000-0xf38000ff irq 17 at device 9.0 on pci0
re0: Chip rev. 0x04000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: Ethernet address: 00:18:4d:79:72:65
re0: [FILTER]
pcib2: <MPTable PCI-PCI bridge> at device 16.0 on pci0
pci2: <PCI bus> on pcib2
atapci1: <SiI SiI 3114 SATA150 controller> port 0xa800-0xa807,0xa400-0xa403,0xa000-0xa007,0x9800-0x9803,0x9400-0x940f mem 0xf2800000-0xf28003ff irq 18 at device 5.0 on pci2
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
atapci2: <VIA 6421 SATA150 controller> port 0x9000-0x900f,0x8800-0x880f,0x8400-0x840f,0x8000-0x800f,0x7800-0x781f,0x7400-0x74ff irq 17 at device 6.0 on pci2
atapci2: [ITHREAD]
ata6: <ATA channel 0> on atapci2
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci2
ata7: [ITHREAD]
ata8: <ATA channel 2> on atapci2
ata8: [ITHREAD]
bge0: <3Com Gigabit Fiber-SX Server NIC, ASIC rev. 0x105> mem 0xf2000000-0xf200ffff irq 19 at device 8.0 on pci2
bge0: Ethernet address: 00:0a:5e:62:32:82
bge0: [ITHREAD]
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200811122336.mACNa2nj027464>