From owner-freebsd-stable@FreeBSD.ORG Thu Jan 9 05:14:55 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 354DDC87 for ; Thu, 9 Jan 2014 05:14:55 +0000 (UTC) Received: from maildrop2.v6ds.occnc.com (maildrop2.v6ds.occnc.com [IPv6:2001:470:88e6:3::232]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C319D1A59 for ; Thu, 9 Jan 2014 05:14:54 +0000 (UTC) Received: from harbor3.ipv6.occnc.com (harbor3.v6ds.occnc.com [IPv6:2001:470:88e6:3::239]) (authenticated bits=128) by maildrop2.v6ds.occnc.com (8.14.7/8.14.7) with ESMTP id s095Eo0c061022; Thu, 9 Jan 2014 00:14:50 -0500 (EST) (envelope-from curtis@ipv6.occnc.com) Message-Id: <201401090514.s095Eo0c061022@maildrop2.v6ds.occnc.com> To: pyunyh@gmail.com From: Curtis Villamizar Subject: Re: regression: msk0 watchdog timeout and interrupt storm In-reply-to: Your message of "Thu, 09 Jan 2014 10:12:35 +0900." <20140109011235.GA2813@michelle.cdnetworks.com> Date: Thu, 09 Jan 2014 00:14:50 -0500 Cc: freebsd-stable@freebsd.org, Curtis Villamizar X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: curtis@ipv6.occnc.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jan 2014 05:14:55 -0000 In message <20140109011235.GA2813@michelle.cdnetworks.com> Yonghyeon PYUN writes: > On Tue, Jan 07, 2014 at 10:11:32PM -0500, Curtis Villamizar wrote: > > > > In message <20140107084938.GA1361@michelle.cdnetworks.com> > > Yonghyeon PYUN writes: > > > > > On Mon, Jan 06, 2014 at 10:20:40AM -0500, Curtis Villamizar wrote: > > > > > > [...] > > > > > > > Here are some relevant parts of dmesg. Is there anything else you want? > > > > > > > > real memory = 2147483648 (2048 MB) > > > > avail memory = 2061438976 (1965 MB) > > > > Event timer "LAPIC" quality 400 > > > > ACPI APIC Table: > > > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > > > > FreeBSD/SMP: 1 package(s) x 2 core(s) > > > > cpu0 (BSP): APIC ID: 0 > > > > cpu1 (AP): APIC ID: 1 > > > > > > > > pcib2: irq 19 at device 7.0 on pci0 > > > > pci2: on pcib2 > > > > on pci1 > > > > pcib2: irq 19 at device 7.0 on pci0 > > > > pci2: on pcib2 > > > > mskc0: port 0xe800-0xe8ff mem > > > > 0xfebfc000-0xfebfffff irq 19 at device 0.0 on pci2 > > > > msk0: > > > > on mskc0 > > > > msk0: Ethernet address: c8:9c:dc:56:38:ef > > > > miibus0: on msk0 > > > > e1000phy0: PHY 0 on miibus0 > > > > e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > > > > 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, > > > > auto, auto-flow > > > > > > > > > > Thank you for the info. > > > > > > > The computer is a Lenovo ThinkCenter (small tower) and not an uncommon > > > > machine so others are likely to run into this. > > > > > > > > > > Please let me know what I could do to help debug this. > > > > > > > > > > > > > > > > If you have more than 4GB memory, try reducing the amount of > > > > > memory(e.g. 3G) in /boot/loader.conf and let me know whether that > > > > > makes any difference for you. > > > > > Note, in order to test this you have to back out your local > > > > > changes. > > > > > > > > Only have 2 GB memory. > > > > > > > > > > Ok, that means my wild guess was not right. :-( > > > > > > > > > [...] > > > > > > > > I'm under the impression that the controller may have additional > > > > > DMA addressing limitation where TX/RX and status LEs should have > > > > > the same high DMA address. Due to the lack of documentation I'm > > > > > not sure about that. If the issue does not happen with 3GB memory, > > > > > we have to use 32bit DMA addressing. > > > > > > > > We have 2 GB memory so the problem with the original code does happen > > > > with less than 4 GB memory. Everything has the same high address of > > > > zero. > > > > > > > > > > Right. > > > > > > > Is there anything else you want me to try? > > > > > > msk(4) uses 4KB alignment for status/TX/RX rings. Your local change > > > will reduce the number of status LEs to be 1024. Stock msk(4) will > > > use 2048 entries for status LEs and you said the cons variable is > > > stuck with 1024 in this case. I have no idea this can happen at > > > this moment. > > > Did msk(4) ever work on your box? If the answer is yes, would you > > > back out both r258780 and your local change? > > > > This host worked for a few years under FreeBSD 8.x and FreeBSD 9.x, > > most recently 9.2. I have other machines running stable_10 at about > > the 10.0.beta3 vintage. I had mostly good luck building the ports I > > use (except openoffice never seems to build). > > > > I transferred a bunch of small stuff over after upgrading to > > 10.0.beta3 on this machine but then with the big move of a tar backup > > through the GbE, it locked up consisitently. > > > > I tried my patch and symptom gone. > > > > > I have a small local diff which was made after seeing r258780. But > > > I'm not sure whether it makes any difference. > > > > So it seems what you want me to do is: > > > > 1. verify whether just backing out r258780 on if_mskreg.h fixes this. > > > > 2. if so, then put back r258780 and try your patch below and see if > > that fixes it. > > > > Correct. Bad news: backing out didn't help. the patch didn't help. Worse news: putting the code back to where it was didn't help. It seems this thing gets wedged in a state where the LE stat entries get written in 0x0 to 0x200 or 0x400, either way to the halfway point in the ring, and then start writing at zero again. Even though the writes seem to start over at zero, the counter continues to increment past the halfway mark. Rebooting many times can sometime unwedge it. Any thoughts on how this could occur? I'm assuming this is a chip problem because once wedged it is mostly unaffected by reboots. A short power cycle yields a better chance of coming up with no problem but still a poor chance. The best way to unwedge it seems to be to put in the code to make msk_stat_count end up as 2048, reboot, and then change the code so that max_stat_count ends up as 1024 and reboot again. Once unwedged msk0 works fine indefinitely until a reboot. It may work for a few reboots, then gets wedged again. This could be a hint as to finding a sequence of writes, perhaps to STAT_LAST_IDX so it consistently comes up unwedged. Maybe a DELAY(1) after the reset (which is shortly before writing STAT_LAST_IDX). If you want debug traces showing the contents of msk_stat_ring when the watchdog occurs I can send some. Can I do this without rebooting to save time? Maybe config out msk and use kldload and kldunload? I'll experiment a bit later, maybe tomorrow. Right now things are running and I have other work to do. Does anyone have any insights into the code after the write to STAT_LAST_IDX and how the watermark and timer values were picked? > > I think I can find some time to do this maybe immediately or at least > > very soon. After doing that I will report back. Please stand by. > > > > Thank you. Thanks for the help with this. > > > > Curtis > > > > > > > > btw - I added someone from Marvell on the Bcc in case he wants to join > > > > in on the conversation or give us a hint in private email.