From owner-freebsd-current@FreeBSD.ORG Wed Nov 28 20:43:03 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E503116A419 for ; Wed, 28 Nov 2007 20:43:02 +0000 (UTC) (envelope-from bofh@terranova.net) Received: from tog.net (tog.net [216.89.226.5]) by mx1.freebsd.org (Postfix) with ESMTP id A5AD213C442 for ; Wed, 28 Nov 2007 20:43:02 +0000 (UTC) (envelope-from bofh@terranova.net) Received: from [216.89.228.170] (host-216-89-228-170.wireless.terranova.net [216.89.228.170]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tog.net (Postfix) with ESMTP id 8804E29B5F1 for ; Wed, 28 Nov 2007 15:43:01 -0500 (EST) Message-ID: <474DD2D0.7010909@terranova.net> Date: Wed, 28 Nov 2007 15:42:56 -0500 From: Travis Mikalson Organization: TerraNovaNet Internet Services User-Agent: Thunderbird 1.5.0.13 (Windows/20070809) MIME-Version: 1.0 To: freebsd-current@freebsd.org References: <73807.10710.qm@web63912.mail.re1.yahoo.com> <200711280842.09340.jhb@freebsd.org> <474D726A.8080807@deepcore.dk> <200711280938.38545.jhb@freebsd.org> In-Reply-To: <200711280938.38545.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: Any successful installs on a Broadcom HT1000 chipset? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2007 20:43:03 -0000 John Baldwin wrote: > On Wednesday 28 November 2007 08:51:38 am Søren Schmidt wrote: >> John Baldwin wrote: >>> On Wednesday 28 November 2007 02:45:16 am Søren Schmidt wrote: >>> >>>> John Baldwin wrote: >>>> >>>>> FYI, I've seen weird in-memory corruption with machines with the HT1000_S1 >>>>> atapci device. In all the cases I've seen so far, a single page is corrupted >>>>> with garbage and the page happens to be used by UMA to hold credentials >>>>> including proc0's credentials. I've seen this corruption (trashed creds for >>>>> proc0 and other creds in that page) on many of the same boxes (Dell 1435's >>>>> IIRC) running on 6.2. I've tried switching the HT1000_S1 to use SWKSMIO >>>>> rather SWKS100 as I mentioned to you in an earlier e-mail (the Linux driver >>>>> uses equivalent of SWKSMIO FWIW) but don't have any conclusive tests on that. >>>>> >>>>> >>>>> >>>> OK, seems the chipset has some real problems, I have digged through all >>>> the (very little) docs and info I got from serverworks back when, and >>>> the only thing I can find is that the chips doesn't support MSI in any >>>> shape or fashion or it will do really strange things. >>>> Now on my system it seems to be disabled but I'm not sure yet how its >>>> determined to be that way. Would be worth for you guys to check what the >>>> sysctl's "hw.pci.enable_msi" and "hw.pci.enable_msix" are set to. >>>> I havn't looked into this yet, but I'm pretty sure we added MSI support >>>> in the 6.2 -> 7.0 timeframe, so that might have uncovered this chipset >>>> bug, and possibly the Promise data corruption one as well. >>>> >>> The ata driver doesn't use MSI (no calls to pci_msi_count or pci_msi_alloc, >>> etc.), so this isn't an issue. Also, the boxes I've seen the corruption on >>> already have MSI disabled (it's still disabled by default in 6.x). >>> >> OK, its must be *totally* disabled not just for ATA but for everything >> on those chipsets or they'll barf all over the place. >> If we do that already we need to look into other places. >> However, if we are dealing with in-memory corruption this is going to >> get "interesting".... >> Does that also happen if nothing uses DMA ? > > Again, on the machines I'm seeing this on it was totally disabled. I don't think > I can totally disable DMA (NICs etc. must use DMA) on the machines and since they > are in production and I only see the corruption as an after-effect when the boxes > panic or deadlock for another reason I'm not easily able to reproduce this. Also, > we do disable MSI for devices behind HT2000 chipsets because of a chip bug, but > not on HT1000 currently. However, MSI isn't on on 6.x anyway. It does make sense that the problem with RELENG_7 is a memory corruption issue, mine just seems to be triggered when I start using my PCI-X ata devices. At boot random processes will start sig11'ing all over the place randomly not to mention horrible corruption of my data. Here's some testing I've just done with various FreeBSD versions on my HT1000 Tyan S3950 motherboard, all went pretty badly... Hm I guess we never tried 6.2 on this. It works way worse than 7.0. All this testing is with a disk attached to the onboard SATA controller, no Marvell PCI-X controller involved. 7.0-BETA3 amd64 and i386, onboard SATA controller causes massive data corruption and things sig11 (like init and the "sh" that's running /etc/rc), but system runs well when you use a SiI3114 4-port SATA 32-bit PCI card. 6.2-R amd64, wont boot past "Timecounters tick every 1.000 msec" with ACPI enabled. With ACPI disabled, after "Timecounters tick" it'll spew stray irq 7, stray irq 7, not logging any more, and then it'll run at a snail's pace. In 3-4 minutes it didn't manage to finish reading the mfsroot and start sysinstall so I didn't go any further. I tried with the SATA controller disabled and with the on-board "em" NICs disabled, no effect on that problem. 6.3-BETA2 amd64, boots with ACPI enabled, sysinstall console is sluggish indicating it's having the same problem, it's just not as severe. Sysinstall almost made it through, but finally failed with a g_vfs_alloc error while trying to write out the kernel distributions from CD to hard drive. top was showing interrupt usage at 70%. I booted that almost-completed 6.3-BETA2 installation up with the SMP kernel for the hell of it, top showed 35% interrupt usage but ran a little faster... things sig11'ed and there was massive data corruption on the disk. fsck (not background fsck) even went crazy and ran out of swap while trying to fsck /usr, etc. etc. I tried to boot HEAD from today but unfortunately it hung at the boot loader and wouldn't boot the kernel, so not much to say on that. -- TerraNovaNet Internet Services - Key Largo, FL Voice: (305)453-4011 x101 Fax: (305)451-5991 http://www.terranova.net/ ---------------------------------------------- Life's not fair, but the root password helps.