From owner-freebsd-questions@FreeBSD.ORG Thu Feb 20 16:34:39 2014 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFB4F570 for ; Thu, 20 Feb 2014 16:34:39 +0000 (UTC) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 839381499 for ; Thu, 20 Feb 2014 16:34:38 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [172.17.17.101]) by email2.allantgroup.com (8.14.7/8.14.7) with ESMTP id s1KGR2S8085555 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 20 Feb 2014 10:27:02 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.7/8.14.6) with ESMTP id s1KGR1X5003074 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 20 Feb 2014 10:27:01 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.8/8.14.7/Submit) id s1KGR1nG003073; Thu, 20 Feb 2014 10:27:01 -0600 (CST) (envelope-from dan) Date: Thu, 20 Feb 2014 10:27:01 -0600 From: Dan Nelson To: Brett Glass Subject: Re: Instability in re driver? Message-ID: <20140220162701.GD80443@dan.emsphone.com> References: <201402200339.UAA09587@mail.lariat.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201402200339.UAA09587@mail.lariat.net> X-OS: FreeBSD 9.2-STABLE User-Agent: Mutt/1.5.22 (2013-10-16) X-Virus-Scanned: clamav-milter 0.98.1 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (email2.allantgroup.com [172.17.19.78]); Thu, 20 Feb 2014 10:27:02 -0600 (CST) X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on email2.allantgroup.com X-Scanned-By: MIMEDefang 2.73 Cc: questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Feb 2014 16:34:39 -0000 In the last episode (Feb 19), Brett Glass said: > We've been experiencing occasional crashes on heavily loaded machines that > have Realtek gigabit Ethernet ports built into their motherboards. This > evening, at a time of peak usage, one of the machines showed the log > messages > > Feb 19 18:44:14 server kernel: re0: watchdog timeout > Feb 19 18:44:14 server kernel: re0: link state changed to DOWN > Feb 19 18:44:18 server kernel: re0: link state changed to UP > > even though we did not take the port down or up. A few minutes later, the > entire machine locked up solid and required a power cycle. > > Here are the boot time messages from the same server (as I rebooted it > following the crash): > > Feb 19 19:08:22 server kernel: re0: port 0xd800-0xd8ff mem 0xfeadf000-0xfeadffff,0xfdefc000-0xfdefffff irq 16 at device 0.0 on pci1 > Feb 19 19:08:22 server kernel: re0: Using 1 MSI-X message > Feb 19 19:08:22 server kernel: re0: Chip rev. 0x28000000 > Feb 19 19:08:22 server kernel: re0: MAC rev. 0x00000000 > Feb 19 19:08:22 server kernel: miibus0: on re0 > Feb 19 19:08:22 server kernel: rgephy0: PHY 1 on miibus0 > Feb 19 19:08:22 server kernel: rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow > > This problem is occurring on several servers built with the same > motherboard/chipset. Are there known bugs, hangs, or memory leaks that > might have led to this condition? Are there recent fixes which address > it? (The machines that are crashing are all running the latest security > patch level of FreeBSD 9.1-RELEASE.) I have had a similar persistent problem with a re0 interface in one of my machines, where medium UDP traffic sometimes causes a problem with the clock timer. Packets stop flowing, calls to nanosleep() never return, etc. I have watchdogd enabled, so the system ends up resetting on its own. Without that, it would just sit in a half-frozen state indefinitely. The machine is an old Dell Studio 540s, and I've seen the problem on FreeBSD 7, 8 and 9, i386 and amd64. I assumed it was just a problem with my motherboard since nobody else has ever reported a similar problem, and it only resets once or twice a month, so it doesn't really affect me. Neither disabling MSI nor using a different timecounter seems to help. re0: port 0xe800-0xe8ff mem 0xfebff000-0xfebfffff,0xfdff0000-0xfdffffff irq 17 at device 0.0 on pci3 re0: MSI count : 1 re0: MSI-X count : 2 re0: attempting to allocate 1 MSI-X vectors (2 supported) msi: routing MSI-X IRQ 259 to local APIC 0 vector 58 re0: using IRQ 259 for MSI-X re0: Using 1 MSI-X message re0: Chip rev. 0x3c000000 re0: MAC rev. 0x00400000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: OUI 0x00e04c, model 0x0011, rev. 2 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow -- Dan Nelson dnelson@allantgroup.com