Date: Sun, 20 Apr 2008 10:36:32 GMT From: Neil Hoggarth <neil@hoggarth.me.uk> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/122928: em net interface watchdog timeouts and stops receiving packets Message-ID: <200804201036.m3KAaWHV095134@www.freebsd.org> Resent-Message-ID: <200804201040.m3KAe28F066364@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 122928
>Category: kern
>Synopsis: em net interface watchdog timeouts and stops receiving packets
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sun Apr 20 10:40:02 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Neil Hoggarth
>Release: 7.0-STABLE
>Organization:
>Environment:
FreeBSD neilhoggarth-2.dsl.easynet.co.uk 7.0-STABLE FreeBSD 7.0-STABLE #0: Thu Apr 3 00:57:21 BST 2008 root@neilhoggarth-2.dsl.easynet.co.uk:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
I have a recently built system using an ASRock AM2NF3-VSTA motherboard
and an AMD Athlon X2 BE-2400 dual-core processor, running 7-STABLE (amd64),
with the GENERIC kernel.
This system uses an Intel PRO/1000 MT PCI Gigabit Ethernet adaptor
(with a 82540EM chip). The same adaptor was used for several years in
my previous system, a uniprocessor AthlonXP system running 6-STALE
(i386), where it worked without any visible issues.
The Intel PCI card provides interface em0, which is connected to my
home LAN (a 3Com 3C1670500 OfficeConnect Gigabit switch). The
motherboard's built-in ethernet provides interface nfe0, which is
connected to my ADSL router. The system uses ipf(5) for filtering
incoming packets on nfe0, and ipnat(5) to NAT connections from other
machines on the LAN to the public Internet.
The machine frequently gets into a state where traffic no longer
reliably passes through the em0 interface (devices on the LAN can no
longer communicate with the machine itself or with the public
Internet), and the kernel starts logging an endless succession of
messages of the form:
Apr 12 07:38:50 neilhoggarth-2 kernel: em0: watchdog timeout -- resetting
Apr 12 07:38:50 neilhoggarth-2 kernel: em0: link state changed to DOWN
Apr 12 07:38:52 neilhoggarth-2 kernel: em0: link state changed to UP
Apr 12 07:38:57 neilhoggarth-2 kernel: em0: watchdog timeout -- resetting
Apr 12 07:38:57 neilhoggarth-2 kernel: em0: link state changed to DOWN
Apr 12 07:39:01 neilhoggarth-2 kernel: em0: link state changed to UP
As far as I can tell from using tcpdump on the machine itself and a
laptop connected to the internal LAN, the machine still sends packets
out through the em0 interface, but does not receive incoming
packets. The only way that I have found of recovering normal function
is to reboot the machine.
The same problem was present when I first installed 7.0-RELEASE.
I have tried a variety of different things in an attempt to gather
more information or find a workaround, including building a kernel
with WITNESS and INVARIANTS (which didn't seem to provide any extra
diagnostics), building a kernel with DEVICE_POLLING and enabling
polling on the em0 interface, building a kernel without PREEMPTION,
and building a kernel without the SMP option. I have also experimented
with running the interface at various other speeds (all the way down
to 10baseT half-duplex) rather than allowing it to autonegotiate
1000baseTX. The problem still occurs under all the circumstances that
I've tried.
Any suggestions on how I can try to debug this further would be
appreciated.
pciconf -lv:
============
hostb0@pci0:0:0:0: class=0x060000 card=0x00e11849 chip=0x00e110de rev=0xa1 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 Host/PCI Bridge'
class = bridge
subclass = HOST-PCI
isab0@pci0:0:1:0: class=0x060100 card=0x00e01849 chip=0x00e010de rev=0xa2 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 LPC Interface Bridge'
class = bridge
subclass = PCI-ISA
none0@pci0:0:1:1: class=0x0c0500 card=0x00e41849 chip=0x00e410de rev=0xa1 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 PCI System Management'
class = serial bus
subclass = SMBus
ohci0@pci0:0:2:0: class=0x0c0310 card=0x00e71849 chip=0x00e710de rev=0xa1 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 OpenHCD USB Controller'
class = serial bus
subclass = USB
ohci1@pci0:0:2:1: class=0x0c0310 card=0x00e71849 chip=0x00e710de rev=0xa1 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 OpenHCD USB Controller'
class = serial bus
subclass = USB
ehci0@pci0:0:2:2: class=0x0c0320 card=0x00e81849 chip=0x00e810de rev=0xa2 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 Enhanced PCI to USB Controller'
class = serial bus
subclass = USB
nfe0@pci0:0:5:0: class=0x068000 card=0x00df1849 chip=0x00df10de rev=0xa2 hdr=0x00
vendor = 'Nvidia Corp'
device = 'Marvell 88E1111 Network adapter'
class = bridge
atapci0@pci0:0:8:0: class=0x01018a card=0x00e51849 chip=0x00e510de rev=0xa2 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 Parallel ATA Controller'
class = mass storage
subclass = ATA
atapci1@pci0:0:10:0: class=0x010185 card=0x00e31849 chip=0x00e310de rev=0xa2 hdr=0x00
vendor = 'Nvidia Corp'
device = 'nForce3 250 Serial ATA Controller'
class = mass storage
subclass = ATA
pcib1@pci0:0:11:0: class=0x060400 card=0x00000000 chip=0x00e210de rev=0xa2 hdr=0x01
vendor = 'Nvidia Corp'
device = 'nForce3 250 AGP Host to PCI Bridge'
class = bridge
subclass = PCI-PCI
pcib2@pci0:0:14:0: class=0x060400 card=0x00000000 chip=0x00ed10de rev=0xa2 hdr=0x01
vendor = 'Nvidia Corp'
device = 'nForce3 250 PCI-PCI Bridge'
class = bridge
subclass = PCI-PCI
hostb1@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration'
class = bridge
subclass = HOST-PCI
hostb2@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron Address Map'
class = bridge
subclass = HOST-PCI
hostb3@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron DRAM Controller'
class = bridge
subclass = HOST-PCI
hostb4@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron Miscellaneous Control'
class = bridge
subclass = HOST-PCI
vgapci0@pci0:1:0:0: class=0x030000 card=0x2062148c chip=0x59611002 rev=0x01 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'RV280 ATI RADEON 9200 se agp'
class = display
subclass = VGA
vgapci1@pci0:1:0:1: class=0x038000 card=0x2063148c chip=0x59411002 rev=0x01 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'RV280 ATI Radeon 9200 - Secondary'
class = display
atapci2@pci0:2:8:0: class=0x010400 card=0x00011103 chip=0x00081103 rev=0x07 hdr=0x00
vendor = 'Triones Technologies Inc. (HighPoint)'
device = 'HPT374 Rocket 154x/1640, RocketRAID 154x/1640 RAID EIDE Controller'
class = mass storage
subclass = RAID
atapci3@pci0:2:8:1: class=0x010400 card=0x00011103 chip=0x00081103 rev=0x07 hdr=0x00
vendor = 'Triones Technologies Inc. (HighPoint)'
device = 'HPT374 Rocket 154x/1640, RocketRAID 154x/1640 RAID EIDE Controller'
class = mass storage
subclass = RAID
em0@pci0:2:9:0: class=0x020000 card=0x002e8086 chip=0x100e8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82540EM Gigabit Ethernet Controller'
class = network
subclass = ethernet
vmstat -i:
==========
interrupt total rate
irq14: ata0 525 0
irq15: ata1 246 0
irq16: drm0 172525 48
irq17: atapci2+ 132203 36
irq18: em0 72904 20
irq20: nfe0 ohci0 13790 3
irq21: ohci1+ 1 0
irq22: ehci0 1149 0
cpu0: timer 7146437 1999
cpu1: timer 7138376 1997
Total 14678156 4106
dmesg:
======
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #0: Thu Apr 3 00:57:21 BST 2008
root@neilhoggarth-2.dsl.easynet.co.uk:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) X2 Dual Core Processor BE-2400 (2331.98-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x60fb2 Stepping = 2
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x2001<SSE3,CX16>
AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch>
Cores per package: 2
usable memory = 8575258624 (8178 MB)
avail memory = 8289517568 (7905 MB)
ACPI APIC Table: <A_M_I OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <A_M_I OEMRSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, d5f00000 (3) failed
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
powernow0: <PowerNow! K8> on cpu0
device_attach: powernow0 attach returned 6
cpu1: <ACPI CPU> on acpi0
powernow1: <PowerNow! K8> on cpu1
device_attach: powernow1 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <NVIDIA nForce3-250 AGP Controller> on hostb0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfebff000-0xfebfffff irq 20 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 4 ports with 4 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfebfe000-0xfebfefff irq 21 at device 2.1 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0, legacy support
usb1: SMM does not respond, resetting
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 4 ports with 4 removable, self powered
ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem 0xfebfdc00-0xfebfdcff irq 22 at device 2.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: <nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
uhub2: 8 ports with 8 removable, self powered
uhub3: <vendor 0x0409 product 0x0059, class 9/0, rev 2.00/1.00, addr 2> on uhub2
uhub3: single transaction translator
uhub3: 4 ports with 4 removable, self powered
uscanner0: <EPSON EPSON Scanner 010F, class 255/0, rev 1.10/1.00, addr 3> on uhub3
uhid0: <Mega World USB 2-Axis 8-Button Gamepad, class 0/0, rev 1.10/1.01, addr 4> on uhub3
ukbd0: <vendor 0x1241 USB Keyboard, class 0/0, rev 1.10/2.80, addr 5> on uhub3
kbd2 at ukbd0
uhid1: <vendor 0x1241 USB Keyboard, class 0/0, rev 1.10/2.80, addr 5> on uhub3
nfe0: <NVIDIA nForce3 MCP7 Networking Adapter> port 0xec00-0xec07 mem 0xfebfc000-0xfebfcfff irq 20 at device 5.0 on pci0
miibus0: <MII bus> on nfe0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
nfe0: Ethernet address: 00:19:66:47:21:9d
nfe0: [FILTER]
atapci0: <nVidia nForce3 Pro UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 8.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce3 Pro SATA150 controller> port 0xf80-0xf87,0xf00-0xf03,0xe80-0xe87,0xe00-0xe03,0xe000-0xe00f,0xd800-0xd87f irq 21 at device 10.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xa000-0xa0ff mem 0xe8000000-0xefffffff,0xfe9f0000-0xfe9fffff irq 16 at device 0.0 on pci1
vgapci1: <VGA-compatible display> mem 0xe0000000-0xe7ffffff,0xfe9e0000-0xfe9effff at device 0.1 on pci1
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
atapci2: <HighPoint HPT374 (channel 0+1) UDMA133 controller> port 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc000-0xc0ff irq 17 at device 8.0 on pci2
atapci2: [ITHREAD]
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
atapci3: <HighPoint HPT374 (channel 2+3) UDMA133 controller> port 0xc400-0xc407,0xbc00-0xbc03,0xb880-0xb887,0xb800-0xb803,0xb400-0xb4ff irq 17 at device 8.1 on pci2
atapci3: [ITHREAD]
ata6: <ATA channel 0> on atapci3
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci3
ata7: [ITHREAD]
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0xb080-0xb0bf mem 0xfeac0000-0xfeadffff,0xfeaa0000-0xfeabffff irq 18 at device 9.0 on pci2
em0: Ethernet address: 00:0e:0c:06:c2:3a
em0: [FILTER]
acpi_button0: <Power Button> on acpi0
sio0: configured irq 3 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 3 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <Generic IRDA-compatible device> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sio1: configured irq 4 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 4 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 on acpi0
sio1: type 16550A
sio1: [FILTER]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xccfff,0xcd000-0xce7ff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
uaudio0: <vendor 0x0ccd USB Audio, class 0/0, rev 1.10/1.00, addr 2> on uhub0
uaudio0: audio rev 1.00
pcm0: <USB Audio> on uaudio0
uhid2: <vendor 0x0ccd USB Audio, class 0/0, rev 1.10/1.00, addr 2> on uhub0
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounters tick every 1.000 msec
ad0: DMA limited to UDMA33, device found non-ATA66 cable
ad0: FAILURE - SET_MULTI status=51<READY,DSC,ERROR> error=4<ABORTED>
ad0: 976MB <TRANSCEND 2.0> at ata0-master UDMA33
ZFS filesystem version 6
ZFS storage pool version 6
acd0: DVDR <PIONEER DVD-RW DVR-109/1.40> at ata1-master UDMA66
ad8: 715404MB <Seagate ST3750640AS 3.AAK> at ata4-master UDMA133
ad10: 715404MB <Seagate ST3750640AS 3.AAK> at ata5-master UDMA133
acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x48 0x00 0x01
acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x48 0x00 0x01
SMP: AP CPU #1 Launched!
cd0 at ata1 bus 0 target 0 lun 0
cd0: <PIONEER DVD-RW DVR-109 1.40> Removable CD-ROM SCSI-0 device
cd0: 66.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from zfs:newtank
IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled
em0: link state changed to UP
>How-To-Repeat:
The problem happens intermittently, typically with a frequency on the order of 2 or 3 times a week, and I have not found a way to repeatably trigger it.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200804201036.m3KAaWHV095134>