From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 18 09:01:52 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C8EC106564A for ; Wed, 18 Feb 2009 09:01:52 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from lorca.tdx.co.uk (lorca.tdx.co.uk [62.13.128.6]) by mx1.freebsd.org (Postfix) with ESMTP id 79AA38FC12 for ; Wed, 18 Feb 2009 09:01:51 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from Works64.dmpriest.net.uk (thebrick.dmpriest.net.uk [62.13.130.30]) by lorca.tdx.co.uk (8.14.0/8.14.0/Kp) with ESMTP id n1I91nfR042575; Wed, 18 Feb 2009 09:01:49 GMT Date: Wed, 18 Feb 2009 09:01:19 +0000 From: Karl Pielorz To: Max Laier , freebsd-hackers@freebsd.org, Yamagi Burmeister Message-ID: <626FC588D3A7707FC1186924@Works64.dmpriest.net.uk> In-Reply-To: <4FF0CBDE2E90DAFEDE59D777@Octa64> References: <12320CD678FB9B76CA7A29F1@Octa64> <200902132008.38110.max@love2party.net> <4FF0CBDE2E90DAFEDE59D777@Octa64> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: Re: Tyan S2895 7.1 amd64 >8Gb RAM support? - resolved X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2009 09:01:52 -0000 Hi, I finally resolved this... (posted to the list for completeness / incase someone else hits this issue). Brief Solution: Reduce dram timing in the bios from DDR400 to DDR333. Gory details: Having taken quite a trip through mptables, bioses, pulling / pushing DIMM's etc. - all the DIMM's in the machine are DDR400 rated - all do test OK. Also remember this machine was used for years under WinXP64/Vista (including using things like a 6Gb RAM drive for video production) - if it had been corrupting data as much as under FreeBSD it would have been noticed :( Anyway, it transpires earlier Opterons were either incapable of running DDR400, or had a 'caveat' that when running lots of memory modules they recommended dropping the timing from DDR400 downwards. This was 'apparently' fixed in later Opterons (my system has 2 * E Stepping 285's). To counter this the BIOS would detect which CPU was in use, and drop the maximum memory speed accordingly vs. installed DIMM slots. This restriction was apparently lifted for Opteron 285's (at least for the stepping I'm using) as their onboard memory controller should be capable of driving all memory banks, at DDR400 speeds - the BIOS reflects this by defaulting to DDR400 timing for my system. Obviously, this works on this system under Windows memory usage patterns / loading / stress - but not for FreeBSD's memory usage / loading. With the RAM backed down to DDR333 - the system once again runs flawlessly. RAM tests didn't catch this - as I could only effectively test 4Gb of RAM 'at a time' - I also doubt that the memory usage patterns software such as memtest86 use would 'load' the memory system enough to cause the timing problem. Max Laier did say he'd had to make memory timing adjustments on his S2895 system, but I presumed that with DDR400 rated memory, and the BIOS using the SPD details on the chips (all branded RAM) that it wouldn't be that (and the fact windows gave years of service with the current settings). -Kp [Who now knows far more than I ever wanted to about mptables, APIC, ASL and hopefully a whole bunch of other stuff I can forget now ;)]