From owner-freebsd-bugs@FreeBSD.ORG Tue May 22 05:30:11 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D33C1065678; Tue, 22 May 2012 05:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 15CB38FC18; Tue, 22 May 2012 05:30:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4M5UAmf099238; Tue, 22 May 2012 05:30:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4M5UAkF099235; Tue, 22 May 2012 05:30:10 GMT (envelope-from gnats) Resent-Date: Tue, 22 May 2012 05:30:10 GMT Resent-Message-Id: <201205220530.q4M5UAkF099235@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Cc: nemoliu@FreeBSD.org, d@delphij.net Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Xin LI Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 44D4B1065670; Tue, 22 May 2012 05:29:55 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 247EC8FC12; Tue, 22 May 2012 05:29:55 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4M5TtDQ098770; Tue, 22 May 2012 05:29:55 GMT (envelope-from delphij@freefall.freebsd.org) Received: (from delphij@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4M5TtFu098769; Tue, 22 May 2012 05:29:55 GMT (envelope-from delphij) Message-Id: <201205220529.q4M5TtFu098769@freefall.freebsd.org> Date: Tue, 22 May 2012 05:29:55 GMT From: Xin LI To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 X-GNATS-Notify: nemoliu@FreeBSD.org, d@delphij.net Cc: davidch@FreeBSD.org, yongari@FreeBSD.org Subject: kern/168217: Watchdog timeouts with bce(4) on BCM5716 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Xin LI List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2012 05:30:11 -0000 >Number: 168217 >Category: kern >Synopsis: Watchdog timeouts with bce(4) on BCM5716 >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue May 22 05:30:10 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Xin LI >Release: FreeBSD 9.0-STABLE i386 >Organization: The FreeBSD Project >Environment: System: FreeBSD sanji.chzip.com 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Tue May 22 01:53:17 CST 2012 delphij@sanji.chzip.com:/usr/obj/usr/src/sys/GENERIC amd64 >Description: The system sometimes stop to respond network traffic with the following in system log: May 22 07:40:01 sanji kernel: Limiting closed port RST response from 235 to 200 packets/sec May 22 09:41:41 sanji kernel: bce1: /usr/src/sys/dev/bce/if_bce.c(7628): Watchdog timeout occurred, resetting! May 22 09:41:41 sanji kernel: bce1: link state changed to DOWN May 22 09:41:43 sanji kernel: bce1: discard frame w/o leading ethernet header (len 0 pkt len 0) This also sometimes lead to panic: May 22 12:42:21 sanji kernel: bce0: discard frame w/o leading ethernet header (len 0 pkt len 0) May 22 12:42:21 sanji kernel: bce0: discard frame w/o leading ethernet header (len 0 pkt len 0) May 22 12:42:21 sanji kernel: May 22 12:42:21 sanji kernel: May 22 12:42:21 sanji kernel: Fatal trap 12: page fault while in kernel mode May 22 12:42:21 sanji kernel: cpuid = 13; apic id = 13 May 22 12:42:21 sanji kernel: fault virtual address = 0x18 May 22 12:42:21 sanji kernel: fault code = supervisor read data, page not present May 22 12:42:21 sanji kernel: instruction pointer = 0x20:0xffffffff80403a46 (this is RELENG_9_0 sys/dev/bce/bce.c:6449) May 22 12:42:21 sanji kernel: stack pointer = 0x28:0xffffff84601e5a80 May 22 12:42:21 sanji kernel: frame pointer = 0x28:0xffffff84601e5b40 May 22 12:42:21 sanji kernel: code segment = base 0x0, limit 0xfffff, type 0x1b May 22 12:42:21 sanji kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 May 22 12:42:21 sanji kernel: processor eflags = interrupt enabled, resume, IOPL = 0 May 22 12:42:21 sanji kernel: current process = 12 (irq256: bce0) Note that the system does not have jumbo enabled, however, split_hdr is enabled. Looking at the code, it was a hardwired value. The system used to run stock FreeBSD 8.2-RELEASE and now run a patched 9.0-RELEASE. The host system is a Dell PowerEdge R410. bce0: mem 0xd6000000-0xd7ffffff irq 36 at device 0.0 on pci1 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: 78:2b:cb:74:82:a6 bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:0); Flags (MSI|MFW); MFW (NCSI 2.0.11) bce1: mem 0xd8000000-0xd9ffffff irq 48 at device 0.1 on pci1 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce1: Ethernet address: 78:2b:cb:74:82:a7 bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:0); Flags (MSI|MFW); MFW (NCSI 2.0.11) bce2: mem 0xda000000-0xdbffffff irq 38 at device 0.0 on pci3 miibus2: on bce2 brgphy2: PHY 1 on miibus2 brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce2: Ethernet address: 00:10:18:bc:f6:30 bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:0); Flags (MSI) bce3: mem 0xdc000000-0xddffffff irq 45 at device 0.1 on pci3 miibus3: on bce3 brgphy3: PHY 1 on miibus3 brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce3: Ethernet address: 00:10:18:bc:f6:32 bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:0); Flags (MSI) bce1: Gigabit link up! bce1: Gigabit link up! bce0: Gigabit link up! bce0: Gigabit link up! bce0@pci0:1:0:0: class=0x020000 card=0x028c1028 chip=0x163b14e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5716 Gigabit Ethernet' class = network subclass = ethernet bce1@pci0:1:0:1: class=0x020000 card=0x028c1028 chip=0x163b14e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5716 Gigabit Ethernet' class = network subclass = ethernet mpt0@pci0:2:0:0: class=0x010000 card=0x1f0f1028 chip=0x00581000 rev=0x08 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'SAS1068E PCI-Express Fusion-MPT SAS' class = mass storage subclass = SCSI bce2@pci0:3:0:0: class=0x020000 card=0x090714e4 chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet bce3@pci0:3:0:1: class=0x020000 card=0x090714e4 chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet More information will be available upon request. >How-To-Repeat: >Fix: Not known at this time. We are testing with split_hdr disabled. >Release-Note: >Audit-Trail: >Unformatted: