From owner-freebsd-hardware@FreeBSD.ORG Mon Jul 16 11:08:58 2012 Return-Path: Delivered-To: freebsd-hardware@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2FC72106566B for ; Mon, 16 Jul 2012 11:08:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F38E68FC18 for ; Mon, 16 Jul 2012 11:08:57 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q6GB8vMC093989 for ; Mon, 16 Jul 2012 11:08:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q6GB8tDj093986 for freebsd-hardware@FreeBSD.org; Mon, 16 Jul 2012 11:08:55 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 16 Jul 2012 11:08:55 GMT Message-Id: <201207161108.q6GB8tDj093986@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-hardware@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-hardware@FreeBSD.org X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2012 11:08:58 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/156241 hardware [mfi] 'zfs send' does not prevents disks to suspend if 1 problem total. From owner-freebsd-hardware@FreeBSD.ORG Mon Jul 16 23:45:19 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 73238106566C for ; Mon, 16 Jul 2012 23:45:19 +0000 (UTC) (envelope-from ayoung@mosaicarchive.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3695D8FC08 for ; Mon, 16 Jul 2012 23:45:19 +0000 (UTC) Received: by obbun3 with SMTP id un3so12767559obb.13 for ; Mon, 16 Jul 2012 16:45:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=qovnb/06Xo3Qckuo16lkv3YcgOOwPxLNBNikqUlIwhY=; b=KRVX+UoZPCnpiMj6IukeMSjwYqcsR4m9CoqhWy/0Fop5W+OcoFY1YiJUokTt/d/7SV HK4rI9qO0nYW2SkmdfY7U+FiqQRP7oM/UgochYhPU1Gr5s5Jxa+QTTp0R3ooiy4XYCZx rYm37e35/vsHIzp/MP1FTipndh2oorvQa9Fskc5qYGo6qwUPb8ugGrcv+rCM6B3+Xsov Xka3H//zpP5j5E/eCyQPjSpU2A7mSF5JCOzjho6zhJgA1QhcuwFs3DS+ieNgkos+VHw/ eH4IKKe4kZ0TzGRpZyTP4NZLgcJn6XWa5uCE5A5IPpvi3vVv5NBdzlFTVj1Pbm0YUvZ0 KcTA== MIME-Version: 1.0 Received: by 10.182.167.101 with SMTP id zn5mr318768obb.60.1342482318461; Mon, 16 Jul 2012 16:45:18 -0700 (PDT) Received: by 10.76.79.165 with HTTP; Mon, 16 Jul 2012 16:45:18 -0700 (PDT) X-Originating-IP: [96.237.242.243] Date: Mon, 16 Jul 2012 19:45:18 -0400 Message-ID: From: Andy Young To: freebsd-hardware@freebsd.org X-Gm-Message-State: ALoCoQng2E/cfo5UbfnmdRir4edwxJPi5p2Xgl7RTLfngRv/B01dHveA3n9ZLwNvLrzRBzLRSTc+ Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2012 23:45:19 -0000 I am having trouble with one of our servers and I'm not sure what to try next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD processors and two memory banks, one for each processor. When I originally built it, I only had one processor and 40 GB of ram. Everything worked awesome. I recently upgraded it, adding another processor and another 40 GB of ram. It was incredibly unstable and constantly rebooted within minute or two of uptime, sometimes it wouldn't even boot all the way before crashing and rebooting again. Seemed like a memory issue so I scaled it back to two processors and 32 GB (4x8GB) of ram. Worked well so I added the remaining 8 GB sticks I had, bringing it up to 64 GB. Still worked great. The sticks I had left were a mix and match variety of 8GB and 4GB sticks. Thinking maybe there was some problem with mixing them, I ordered more 8GB memory just like the ones in the box. While waiting for the new memory, the machine performed great with no issues. New memory arrived and I added two more 8GB sticks. Immediately the constant crashing returned. It seems really unlikely that I got bad memory in two separate orders. Does anyone have any other ideas? Again, its perfectly stable with two processors and 64 GB of memory but goes nuts when I more. I really appreciate the help!! Motherboard: Supermicro H8DGi-F CPU: 2 x AMD 6274 (2.2 Ghz 16-core) Memory: Kingston 8GB DDR3 1333 -- Andrew Young Mosaic Storage Systems, Inc http://www.mosaicarchive.com/ Follow us on: Twitter , Facebook , Google Plus , Pinterest From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 00:49:24 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9747106566C for ; Tue, 17 Jul 2012 00:49:24 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 1FB0F8FC0A for ; Tue, 17 Jul 2012 00:49:23 +0000 (UTC) Received: from server.rulingia.com (c220-239-248-69.belrs5.nsw.optusnet.com.au [220.239.248.69]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id q6H0nGiY065369 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 17 Jul 2012 10:49:16 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id q6H0n9Ro068152 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 17 Jul 2012 10:49:10 +1000 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id q6H0n967068151; Tue, 17 Jul 2012 10:49:09 +1000 (EST) (envelope-from peter) Date: Tue, 17 Jul 2012 10:49:09 +1000 From: Peter Jeremy To: Andy Young Message-ID: <20120717004909.GB66913@server.rulingia.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kfjH4zxOES6UT95V" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 00:49:24 -0000 --kfjH4zxOES6UT95V Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-Jul-16 19:45:18 -0400, Andy Young wrote: >I am having trouble with one of our servers and I'm not sure what to try >next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD >processors and two memory banks, one for each processor. When I originally >built it, I only had one processor and 40 GB of ram. Everything worked >awesome. I recently upgraded it, adding another processor and another 40 GB >of ram. It was incredibly unstable and constantly rebooted within minute or >two of uptime, sometimes it wouldn't even boot all the way before crashing >and rebooting again. =2E.. >other ideas? Again, its perfectly stable with two processors and 64 GB of >memory but goes nuts when I more. Have you checked the motherboard notes to ensure that your configuration is supported? Is the BIOS up to date? What version of FreeBSD is this? And I presume it's amd64 rather than i386+PAE. Have you tried running memtest86 or memtest86+? (You might like to run both because ISTR only the former handles SMP). Can you capture the output from a verbose boot with all the memory installed? The SMAP and/or physical memory layout might offer a clue as to what is going wrong. --=20 Peter Jeremy --kfjH4zxOES6UT95V Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlAEtoUACgkQ/opHv/APuIcu8gCdG4tIaomcl+dIj2JrTFanqxX2 K9gAn0yWlrARLw9tNqKQ8UKMabk6I0kw =Z1P6 -----END PGP SIGNATURE----- --kfjH4zxOES6UT95V-- From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 00:57:32 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99EDE1065670 for ; Tue, 17 Jul 2012 00:57:32 +0000 (UTC) (envelope-from erichfreebsdlist@ovitrap.com) Received: from alogreentechnologies.com (alogreentechnologies.com [67.212.224.110]) by mx1.freebsd.org (Postfix) with ESMTP id 602248FC12 for ; Tue, 17 Jul 2012 00:57:32 +0000 (UTC) Received: from amd620.ovitrap.com ([49.128.188.2]) (authenticated bits=0) by alogreentechnologies.com (8.13.1/8.13.1) with ESMTP id q6H0vRWL015395; Mon, 16 Jul 2012 18:57:30 -0600 From: Erich Dollansky To: freebsd-hardware@freebsd.org Date: Tue, 17 Jul 2012 07:59:44 +0700 User-Agent: KMail/1.13.7 (FreeBSD/8.3-STABLE; KDE/4.7.4; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201207170759.44995.erichfreebsdlist@ovitrap.com> Cc: Andy Young Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 00:57:32 -0000 Hi, On Tuesday 17 July 2012 06:45:18 Andy Young wrote: > I am having trouble with one of our servers and I'm not sure what to try > next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD > processors and two memory banks, one for each processor. When I originally > built it, I only had one processor and 40 GB of ram. Everything worked > awesome. I recently upgraded it, adding another processor and another 40 GB > of ram. It was incredibly unstable and constantly rebooted within minute or > two of uptime, sometimes it wouldn't even boot all the way before crashing > and rebooting again. Seemed like a memory issue so I scaled it back to two > processors and 32 GB (4x8GB) of ram. Worked well so I added the remaining 8 > GB sticks I had, bringing it up to 64 GB. Still worked great. The sticks I > had left were a mix and match variety of 8GB and 4GB sticks. Thinking maybe > there was some problem with mixing them, I ordered more 8GB memory just > like the ones in the box. While waiting for the new memory, the machine > performed great with no issues. New memory arrived and I added two more 8GB > sticks. Immediately the constant crashing returned. It seems really > unlikely that I got bad memory in two separate orders. Does anyone have any > other ideas? Again, its perfectly stable with two processors and 64 GB of > memory but goes nuts when I more. > could it be caused by the power supply? Did you run a memory test? If possible, try different power supplies. > I really appreciate the help!! > > Motherboard: Supermicro H8DGi-F > CPU: 2 x AMD 6274 (2.2 Ghz 16-core) > Memory: Kingston 8GB DDR3 1333 No ECC? Erich From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 06:38:40 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBE69106566B for ; Tue, 17 Jul 2012 06:38:40 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 608598FC0A for ; Tue, 17 Jul 2012 06:38:39 +0000 (UTC) X-RZG-AUTH: :IWUHfUGtd9+6EujMWHx57N4dWae4bmTL/JIGbzkGUoozgkO4q1xDEhkgOJDsXNs= X-RZG-CLASS-ID: mo00 Received: from fuckner2.delnet ([85.183.0.195]) by smtp.strato.de (jorabe mo82) (RZmta 29.19 AUTH) with ESMTPA id J07e49o6H3rp4q for ; Tue, 17 Jul 2012 08:38:37 +0200 (CEST) Message-ID: <500507C6.9030606@fuckner.net> Date: Tue, 17 Jul 2012 08:35:50 +0200 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: freebsd-hardware@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 06:38:40 -0000 On 07/17/2012 01:45 AM, Andy Young wrote: > Motherboard: Supermicro H8DGi-F > CPU: 2 x AMD 6274 (2.2 Ghz 16-core) > Memory: Kingston 8GB DDR3 1333 > Hi all, if it is a memory problem it will probably logged via ipmi or dmi. Try ipmitool sel list- or if there are logs in bios. We typicially use 8 identical modules DDR3- ECC Reg on this board. Regards, Michael! From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 07:31:16 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA6E4106564A for ; Tue, 17 Jul 2012 07:31:16 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from rack.patpro.net (rack.patpro.net [193.30.227.216]) by mx1.freebsd.org (Postfix) with ESMTP id 331268FC16 for ; Tue, 17 Jul 2012 07:31:16 +0000 (UTC) Received: from rack.patpro.net (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP id A529A1CC020; Tue, 17 Jul 2012 09:23:11 +0200 (CEST) X-Virus-Scanned: amavisd-new at patpro.net Received: from amavis-at-patpro.net ([127.0.0.1]) by rack.patpro.net (rack.patpro.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hgDzpWmgb2ju; Tue, 17 Jul 2012 09:23:05 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP; Tue, 17 Jul 2012 09:23:05 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/signed; boundary=Apple-Mail-13--869634846; protocol="application/pkcs7-signature"; micalg=sha1 From: Patrick Proniewski In-Reply-To: Date: Tue, 17 Jul 2012 09:23:05 +0200 Message-Id: <7E5394D4-4212-4B83-8554-2ABB59D36467@patpro.net> References: To: Andy Young X-Mailer: Apple Mail (2.1084) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 07:31:16 -0000 --Apple-Mail-13--869634846 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 17 juil. 2012, at 01:45, Andy Young wrote: > New memory arrived and I added two more 8GB > sticks. Immediately the constant crashing returned. It seems really > unlikely that I got bad memory in two separate orders. Does anyone = have any > other ideas? Again, its perfectly stable with two processors and 64 GB = of > memory but goes nuts when I more. what about a defective memory slot? Testing another power supply as = suggested is a good idea, too. patpro= --Apple-Mail-13--869634846-- From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 16:50:56 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12215106566C for ; Tue, 17 Jul 2012 16:50:56 +0000 (UTC) (envelope-from ayoung@mosaicarchive.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id BB4D88FC16 for ; Tue, 17 Jul 2012 16:50:55 +0000 (UTC) Received: by yhfs35 with SMTP id s35so740598yhf.13 for ; Tue, 17 Jul 2012 09:50:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=+d/pNYUBegpct44wZ8+/BrDXLQ5jg4R8t+sHZPV7gWM=; b=R4QNK7ydr18h2Y11IJjGd/xBq8U6jqqoLIbkhjxwA6/dJR+WMlqgrIaIjDOQ+AEamq 1TZokOcUm7mmtHkMiyKmYv6H/9+YK1t5mFd+rtqR1ZKMZ0FriRbVW7ygU2snbYuj7yuw XEOGjSjiH4x6yN8CcSTgEkAJR1ZYaquNDTy1Kwr4mT9vAYtNMHTX1GByKCLN/UekSgXJ 8nBK5EIEH38yykAYvCEm4xPdQHgFJWDAyafADXZ0JBDPkaX6zaMCuqQSTnHD7W4VXrqG GDm3UM/XgHu/vp2AfuiaUp6zoQAuRAnLaRl4xxpJBVtmCtQT+QMtz+f5kW7s9cF9KZc7 zE+Q== MIME-Version: 1.0 Received: by 10.60.28.162 with SMTP id c2mr4437857oeh.3.1342543855084; Tue, 17 Jul 2012 09:50:55 -0700 (PDT) Received: by 10.76.79.165 with HTTP; Tue, 17 Jul 2012 09:50:55 -0700 (PDT) X-Originating-IP: [75.147.53.134] In-Reply-To: <20120717004909.GB66913@server.rulingia.com> References: <20120717004909.GB66913@server.rulingia.com> Date: Tue, 17 Jul 2012 12:50:55 -0400 Message-ID: From: Andy Young To: Peter Jeremy X-Gm-Message-State: ALoCoQl7V8ZI4q5lj0MWZmkIESILW01c5lGRiWR7JXN4LDSBSkj253RzTXWaZ8iiGvR7I5sOrzJN Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 16:50:56 -0000 Hi Peter, I will check the BIOS firmware. I haven't tried that yet. I'm running FreeBSD 9-RELEASE-p3. Yes it is AMD64. I ran memtest on the first 32 gb or memory where the machine was initially stable. Once I put over 64 GB in, I can't get the machine to stay up for long enough to even try. I'll try the verbose boot idea too. Thanks! On Mon, Jul 16, 2012 at 8:49 PM, Peter Jeremy wrote: > On 2012-Jul-16 19:45:18 -0400, Andy Young > wrote: > >I am having trouble with one of our servers and I'm not sure what to try > >next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD > >processors and two memory banks, one for each processor. When I originally > >built it, I only had one processor and 40 GB of ram. Everything worked > >awesome. I recently upgraded it, adding another processor and another 40 > GB > >of ram. It was incredibly unstable and constantly rebooted within minute > or > >two of uptime, sometimes it wouldn't even boot all the way before crashing > >and rebooting again. > ... > >other ideas? Again, its perfectly stable with two processors and 64 GB of > >memory but goes nuts when I more. > > Have you checked the motherboard notes to ensure that your configuration > is supported? Is the BIOS up to date? > > What version of FreeBSD is this? And I presume it's amd64 rather than > i386+PAE. > > Have you tried running memtest86 or memtest86+? (You might like to > run both because ISTR only the former handles SMP). > > Can you capture the output from a verbose boot with all the memory > installed? The SMAP and/or physical memory layout might offer a > clue as to what is going wrong. > > -- > Peter Jeremy > -- Andrew Young Mosaic Storage Systems, Inc http://www.mosaicarchive.com/ Follow us on: Twitter , Facebook , Google Plus , Pinterest From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 16:54:44 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C56B9106566C for ; Tue, 17 Jul 2012 16:54:44 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5673F8FC24 for ; Tue, 17 Jul 2012 16:54:44 +0000 (UTC) X-RZG-AUTH: :IWUHfUGtd9+4Du6KUGxoqde+AFhxnvkTDzh0c7ueojHnW/eNeq6A82NJ3vfS7BrFpOouOw== X-RZG-CLASS-ID: mo00 Received: from c64.rebootking.de (e176131079.adsl.alicedsl.de [85.176.131.79]) by smtp.strato.de (joses mo15) (RZmta 29.19 DYNA|AUTH) with ESMTPA id U01bd4o6HGoA6J for ; Tue, 17 Jul 2012 18:54:43 +0200 (CEST) Message-ID: <500598D5.9060307@fuckner.net> Date: Tue, 17 Jul 2012 18:54:45 +0200 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: freebsd-hardware@freebsd.org References: <20120717004909.GB66913@server.rulingia.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 16:54:44 -0000 On 07/17/2012 06:50 PM, Andy Young wrote: > Hi Peter, > > I will check the BIOS firmware. I haven't tried that yet. please also check IPMI-Firmware since IPMI controlls memory refresh etc. Should be 2.50. > > I'm running FreeBSD 9-RELEASE-p3. Yes it is AMD64. > > I ran memtest on the first 32 gb or memory where the machine was initially > stable. Once I put over 64 GB in, I can't get the machine to stay up for > long enough to even try. > can you tell us about the type of memory you are using- is it Reg Memory? Regards, Michael! From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 17:16:33 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3DB106564A for ; Tue, 17 Jul 2012 17:16:33 +0000 (UTC) (envelope-from ayoung@mosaicarchive.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 784838FC08 for ; Tue, 17 Jul 2012 17:16:33 +0000 (UTC) Received: by obbun3 with SMTP id un3so1124473obb.13 for ; Tue, 17 Jul 2012 10:16:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=VM2tNM2WhNSTjiwjbTLYlTH5px0fcysrbzGo0f9ndd0=; b=hge4NvHkwzB4TaOQLdmplViYv3ksweGyW7Mw51bWGfWX4W7CFqnlM6yiwGGMs0Gfzy PyBx0a8XjdQfLvyOKDpSqfGvkGLhRmH3bZakTgjntEpvzkL40oxFuhklXMRCmJPFxtB2 RlhXW5wye+Dv7u+M18tsbkRG7npVo2u16/SJJx+fwcblPfsh7V6t8AOyvxQRvrvolDV/ UNp6eEjzCITtXcc+KPaqkJwENMOepySP/LmmRAJgh4WIxyyi1mUX9y1L32Dq0+xxD/MQ io1DlIlxgmPon2QeDjqH/Ne6aSRHHJgBGStfbKaruATB2IdrR6Z7AX+aB4EanbFX2BeV OoFA== MIME-Version: 1.0 Received: by 10.182.167.101 with SMTP id zn5mr4470593obb.60.1342545393048; Tue, 17 Jul 2012 10:16:33 -0700 (PDT) Received: by 10.76.79.165 with HTTP; Tue, 17 Jul 2012 10:16:33 -0700 (PDT) X-Originating-IP: [75.147.53.134] In-Reply-To: <201207170759.44995.erichfreebsdlist@ovitrap.com> References: <201207170759.44995.erichfreebsdlist@ovitrap.com> Date: Tue, 17 Jul 2012 13:16:33 -0400 Message-ID: From: Andy Young To: Erich Dollansky X-Gm-Message-State: ALoCoQlDm4J7U+6sc26DC48wZ9SeKlAxb5g4Ps1/AbfR8kYQLB8wmp8maKfwsg524+QqclyRvlVA Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 17:16:33 -0000 Hi Erich, Why would the power supply be suspect since the machine is perfectly stable with 64 GB of memory in it? The server won't stay up long enough to run memtest. Andy On Mon, Jul 16, 2012 at 8:59 PM, Erich Dollansky < erichfreebsdlist@ovitrap.com> wrote: > Hi, > > On Tuesday 17 July 2012 06:45:18 Andy Young wrote: > > I am having trouble with one of our servers and I'm not sure what to try > > next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD > > processors and two memory banks, one for each processor. When I > originally > > built it, I only had one processor and 40 GB of ram. Everything worked > > awesome. I recently upgraded it, adding another processor and another 40 > GB > > of ram. It was incredibly unstable and constantly rebooted within minute > or > > two of uptime, sometimes it wouldn't even boot all the way before > crashing > > and rebooting again. Seemed like a memory issue so I scaled it back to > two > > processors and 32 GB (4x8GB) of ram. Worked well so I added the > remaining 8 > > GB sticks I had, bringing it up to 64 GB. Still worked great. The sticks > I > > had left were a mix and match variety of 8GB and 4GB sticks. Thinking > maybe > > there was some problem with mixing them, I ordered more 8GB memory just > > like the ones in the box. While waiting for the new memory, the machine > > performed great with no issues. New memory arrived and I added two more > 8GB > > sticks. Immediately the constant crashing returned. It seems really > > unlikely that I got bad memory in two separate orders. Does anyone have > any > > other ideas? Again, its perfectly stable with two processors and 64 GB of > > memory but goes nuts when I more. > > > could it be caused by the power supply? > > Did you run a memory test? > > If possible, try different power supplies. > > > I really appreciate the help!! > > > > Motherboard: Supermicro H8DGi-F > > CPU: 2 x AMD 6274 (2.2 Ghz 16-core) > > Memory: Kingston 8GB DDR3 1333 > > No ECC? > > Erich > -- Andrew Young Mosaic Storage Systems, Inc http://www.mosaicarchive.com/ Follow us on: Twitter , Facebook , Google Plus , Pinterest From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 18:34:47 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 01F13106566B for ; Tue, 17 Jul 2012 18:34:47 +0000 (UTC) (envelope-from dieterbsd@engineer.com) Received: from mailout-us.gmx.com (mailout-us.gmx.com [74.208.5.67]) by mx1.freebsd.org (Postfix) with SMTP id 99A688FC0A for ; Tue, 17 Jul 2012 18:34:46 +0000 (UTC) Received: (qmail 1461 invoked by uid 0); 17 Jul 2012 18:34:40 -0000 Received: from 67.206.185.131 by rms-us005 with HTTP Content-Type: text/plain; charset="utf-8" Date: Tue, 17 Jul 2012 14:34:36 -0400 From: "Dieter BSD" Message-ID: <20120717183438.298400@gmx.com> MIME-Version: 1.0 To: freebsd-hardware@freebsd.org X-Authenticated: #74169980 X-Flags: 0001 X-Mailer: GMX.com Web Mailer x-registered: 0 Content-Transfer-Encoding: 8bit X-GMX-UID: adV3cPQV3zOlNR3dAHAhP8t+IGRvb0Cg Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 18:34:47 -0000 > It seems really unlikely that I got bad memory in two separate orders. Test the new memory by itself, keeping at or below the 64 GB you know works. Then you will know if the board is happy with the new memory or not. If the mainboard supported configurations allow, test all slots while keeping <= 64 GB. If there is a bad slot, visually inspect for cold solder joints, shorts, etc. IIRC, FreeBSD has some knob that allows limiting the memory used. Try having > 64GB of memory plugged in, but limit how much you actually use. Some systems are very very picky about memory. Published required specs for memory might not cover everything. The technical support folks might have learned more what actually works or not. From owner-freebsd-hardware@FreeBSD.ORG Tue Jul 17 20:08:39 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1998D1065672 for ; Tue, 17 Jul 2012 20:08:39 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 9B8AC8FC0A for ; Tue, 17 Jul 2012 20:08:38 +0000 (UTC) Received: from server.rulingia.com (c220-239-248-69.belrs5.nsw.optusnet.com.au [220.239.248.69]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id q6HK8aTb069965 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 18 Jul 2012 06:08:36 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id q6HK18cb085537 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 18 Jul 2012 06:01:08 +1000 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id q6HK17RC085536; Wed, 18 Jul 2012 06:01:07 +1000 (EST) (envelope-from peter) Date: Wed, 18 Jul 2012 06:01:07 +1000 From: Peter Jeremy To: Andy Young Message-ID: <20120717200107.GB72689@server.rulingia.com> References: <20120717004909.GB66913@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ADZbWkCsHQ7r3kzd" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2012 20:08:39 -0000 --ADZbWkCsHQ7r3kzd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-Jul-17 12:50:55 -0400, Andy Young wrote: >I ran memtest on the first 32 gb or memory where the machine was initially >stable. Once I put over 64 GB in, I can't get the machine to stay up for >long enough to even try. This pretty well clears FreeBSD then. As others have suggested, I'd try all the RAM in smaller blocks and talk to Supermicro's technical support. It's also possible that your second CPU is bad. --=20 Peter Jeremy --ADZbWkCsHQ7r3kzd Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlAFxIMACgkQ/opHv/APuIe9EgCgsHQRpNoNp8SmVTIiI1cagGaP o0EAn08P8Rr0I9/HWO3dopt80rYO0/2h =al1y -----END PGP SIGNATURE----- --ADZbWkCsHQ7r3kzd-- From owner-freebsd-hardware@FreeBSD.ORG Wed Jul 18 00:35:02 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A1F7F1065672 for ; Wed, 18 Jul 2012 00:35:02 +0000 (UTC) (envelope-from erichfreebsdlist@ovitrap.com) Received: from alogreentechnologies.com (alogreentechnologies.com [67.212.224.110]) by mx1.freebsd.org (Postfix) with ESMTP id 68D848FC22 for ; Wed, 18 Jul 2012 00:35:02 +0000 (UTC) Received: from amd620.ovitrap.com ([49.128.188.2]) (authenticated bits=0) by alogreentechnologies.com (8.13.1/8.13.1) with ESMTP id q6HNluVl013113; Tue, 17 Jul 2012 17:48:13 -0600 From: Erich Dollansky To: Andy Young Date: Wed, 18 Jul 2012 06:50:10 +0700 User-Agent: KMail/1.13.7 (FreeBSD/8.3-STABLE; KDE/4.7.4; amd64; ; ) References: <201207170759.44995.erichfreebsdlist@ovitrap.com> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Message-Id: <201207180650.11035.erichfreebsdlist@ovitrap.com> Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 00:35:02 -0000 Hi, On Wednesday 18 July 2012 00:16:33 Andy Young wrote: > > Why would the power supply be suspect since the machine is perfectly stable > with 64 GB of memory in it? > because the machine needs more electricity with the extra modules. If it is at the limits without, it could go behind with the additional modules installed. > The server won't stay up long enough to run memtest. Also when you boot directly into the memory test? You did not answer the question regarding ECC. Did you mix the modules? Put modules back in reversed order. Can you insert modules only into the sockets which seem to fail and leave all other empty? If you can and the machine works then, I assume it is caused by the power supply. If the machine fails then, it is the motherboard. At least it looks like this from distance as you said that it is most unlikely that the modules are faulty. Erich > > Andy > > On Mon, Jul 16, 2012 at 8:59 PM, Erich Dollansky < > erichfreebsdlist@ovitrap.com> wrote: > > > Hi, > > > > On Tuesday 17 July 2012 06:45:18 Andy Young wrote: > > > I am having trouble with one of our servers and I'm not sure what to try > > > next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD > > > processors and two memory banks, one for each processor. When I > > originally > > > built it, I only had one processor and 40 GB of ram. Everything worked > > > awesome. I recently upgraded it, adding another processor and another 40 > > GB > > > of ram. It was incredibly unstable and constantly rebooted within minute > > or > > > two of uptime, sometimes it wouldn't even boot all the way before > > crashing > > > and rebooting again. Seemed like a memory issue so I scaled it back to > > two > > > processors and 32 GB (4x8GB) of ram. Worked well so I added the > > remaining 8 > > > GB sticks I had, bringing it up to 64 GB. Still worked great. The sticks > > I > > > had left were a mix and match variety of 8GB and 4GB sticks. Thinking > > maybe > > > there was some problem with mixing them, I ordered more 8GB memory just > > > like the ones in the box. While waiting for the new memory, the machine > > > performed great with no issues. New memory arrived and I added two more > > 8GB > > > sticks. Immediately the constant crashing returned. It seems really > > > unlikely that I got bad memory in two separate orders. Does anyone have > > any > > > other ideas? Again, its perfectly stable with two processors and 64 GB of > > > memory but goes nuts when I more. > > > > > could it be caused by the power supply? > > > > Did you run a memory test? > > > > If possible, try different power supplies. > > > > > I really appreciate the help!! > > > > > > Motherboard: Supermicro H8DGi-F > > > CPU: 2 x AMD 6274 (2.2 Ghz 16-core) > > > Memory: Kingston 8GB DDR3 1333 > > > > No ECC? > > > > Erich > > > > > > -- > Andrew Young > Mosaic Storage Systems, Inc > http://www.mosaicarchive.com/ > > Follow us on: > Twitter , > Facebook > , Google Plus > , Pinterest > From owner-freebsd-hardware@FreeBSD.ORG Wed Jul 18 02:21:10 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1895C1065678 for ; Wed, 18 Jul 2012 02:21:10 +0000 (UTC) (envelope-from lnb@freebsdsystems.com) Received: from panda.servaris.com (panda.servaris.com [107.6.50.5]) by mx1.freebsd.org (Postfix) with ESMTP id A64BA8FC14 for ; Wed, 18 Jul 2012 02:21:09 +0000 (UTC) Received: (qmail 68505 invoked by uid 89); 18 Jul 2012 02:21:08 -0000 Received: from unknown (HELO ?192.168.0.55?) (lnb@freebsdsystems.com@99.238.64.55) by panda.servaris.com with ESMTPA; 18 Jul 2012 02:21:08 -0000 Message-ID: <50061D65.7040605@freebsdsystems.com> Date: Tue, 17 Jul 2012 22:20:21 -0400 From: Lanny Baron Organization: Freedom Technologies Corp. FreeBSD Systems User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: freebsd-hardware@freebsd.org References: <201207170759.44995.erichfreebsdlist@ovitrap.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 02:21:10 -0000 Hi Andy, Sounds to me like you have 1) a flakey board, 2) the memory is not identical. We never use kingston for a variety of reasons, but you really should see if the part numbers are identical. The timings on the drams is critical. I would never mix capacities. 3) Make sure the memory is all the same i.e. registered e.c.c. or non registered e.c.c. I don't think its the power supply, but it can be. Regards, Lanny http://www.servaris.com or http://www.freebsdsystems.com On 7/17/2012 1:16 PM, Andy Young wrote: > Hi Erich, > > Why would the power supply be suspect since the machine is perfectly stable > with 64 GB of memory in it? > > The server won't stay up long enough to run memtest. > > Andy > > On Mon, Jul 16, 2012 at 8:59 PM, Erich Dollansky < > erichfreebsdlist@ovitrap.com> wrote: > >> Hi, >> >> On Tuesday 17 July 2012 06:45:18 Andy Young wrote: >>> I am having trouble with one of our servers and I'm not sure what to try >>> next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD >>> processors and two memory banks, one for each processor. When I >> originally >>> built it, I only had one processor and 40 GB of ram. Everything worked >>> awesome. I recently upgraded it, adding another processor and another 40 >> GB >>> of ram. It was incredibly unstable and constantly rebooted within minute >> or >>> two of uptime, sometimes it wouldn't even boot all the way before >> crashing >>> and rebooting again. Seemed like a memory issue so I scaled it back to >> two >>> processors and 32 GB (4x8GB) of ram. Worked well so I added the >> remaining 8 >>> GB sticks I had, bringing it up to 64 GB. Still worked great. The sticks >> I >>> had left were a mix and match variety of 8GB and 4GB sticks. Thinking >> maybe >>> there was some problem with mixing them, I ordered more 8GB memory just >>> like the ones in the box. While waiting for the new memory, the machine >>> performed great with no issues. New memory arrived and I added two more >> 8GB >>> sticks. Immediately the constant crashing returned. It seems really >>> unlikely that I got bad memory in two separate orders. Does anyone have >> any >>> other ideas? Again, its perfectly stable with two processors and 64 GB of >>> memory but goes nuts when I more. >>> >> could it be caused by the power supply? >> >> Did you run a memory test? >> >> If possible, try different power supplies. >> >>> I really appreciate the help!! >>> >>> Motherboard: Supermicro H8DGi-F >>> CPU: 2 x AMD 6274 (2.2 Ghz 16-core) >>> Memory: Kingston 8GB DDR3 1333 >> >> No ECC? >> >> Erich >> > > > From owner-freebsd-hardware@FreeBSD.ORG Thu Jul 19 15:08:25 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E5E1106566B for ; Thu, 19 Jul 2012 15:08:25 +0000 (UTC) (envelope-from ayoung@mosaicarchive.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id BCE908FC17 for ; Thu, 19 Jul 2012 15:08:24 +0000 (UTC) Received: by obbun3 with SMTP id un3so5025092obb.13 for ; Thu, 19 Jul 2012 08:08:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=qGLNSv/SUwEL3E1EaQeQC8B2+7x0/yWNsfEEf7UE2Aw=; b=Esq75x0/jw58PdevNAtXpiGP+L8iD4c2eilZPdo+kfEUTqWb1i2YN4rM+/37CV/zsx ooZg0ADx8JXukTJP0CgTGdBbOg2hrBqGTC1kDWhD1wbAW8sbRRlqP2W6FY8tNY9tvK9K 5Br2JWjSbluS3UFYAjnF0dPiDnSGoCzkrtB2kwY4TXd+HVT4EMJZTn/jNcBXWqa0C8FH tgtSVn1an5ZvN8bKGN43kdIwzMjEzluO+DAEXGPXgeuTObvjFQk/Gw+1x+HaHjmDXQ1j SVCLo5zE8lpqm/zfGsRDFhu3x/FlELrE3YRaiRePt/dbX/VwPXhOGRpjHn/sfFYwudwy nTsA== MIME-Version: 1.0 Received: by 10.182.16.3 with SMTP id b3mr3174043obd.72.1342710504006; Thu, 19 Jul 2012 08:08:24 -0700 (PDT) Received: by 10.76.79.165 with HTTP; Thu, 19 Jul 2012 08:08:23 -0700 (PDT) X-Originating-IP: [75.147.53.134] In-Reply-To: <201207180650.11035.erichfreebsdlist@ovitrap.com> References: <201207170759.44995.erichfreebsdlist@ovitrap.com> <201207180650.11035.erichfreebsdlist@ovitrap.com> Date: Thu, 19 Jul 2012 11:08:23 -0400 Message-ID: From: Andy Young To: Erich Dollansky X-Gm-Message-State: ALoCoQmMuKUX0BXU/AL5bTAhhV7iLSUqxJvddKvDhugXyGSMT32AB4s8lN/6qq8ha+7k/Qu50Kld Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2012 15:08:25 -0000 Hi Erich, > because the machine needs more electricity with the extra modules. If it is at the limits without, it could go behind > with the additional modules installed. Interesting. The chassis has dual 900W power supplies. Apart from simply replacing them, I am not sure how I can verify whether the power supply is the issue. > You did not answer the question regarding ECC. The memory modules I added are listed on Newegg as Kingston 8 GB 240-Pin DDRS SDRAM ECC Registered DDR3 1333 Server Memory. So yes they have ECC. > Did you mix the modules? Yes. There is a mix of modules in there. The original 32 GB of memory that I put in to begin with are not the exact same module. They are Hynix 8GB PC3-10600 DDR3-1333MHz ECC Registered CL9 240-Pin DIMM Dual that came from the hardware integrator we bought the Supermicro chassis from. At this point there are 32 of the Hynix and 32 of the Kingston and it is working ok so simply mixing them isn't causing an issue. > Can you insert modules only into the sockets which seem to fail and leave all other empty? Ok. I can try that. Thanks for the help!! Andy On Tue, Jul 17, 2012 at 7:50 PM, Erich Dollansky < erichfreebsdlist@ovitrap.com> wrote: > Hi, > > On Wednesday 18 July 2012 00:16:33 Andy Young wrote: > > > > Why would the power supply be suspect since the machine is perfectly > stable > > with 64 GB of memory in it? > > > because the machine needs more electricity with the extra modules. If it > is at the limits without, it could go behind with the additional modules > installed. > > > The server won't stay up long enough to run memtest. > > Also when you boot directly into the memory test? > > You did not answer the question regarding ECC. > > Did you mix the modules? > > Put modules back in reversed order. > > Can you insert modules only into the sockets which seem to fail and leave > all other empty? > > If you can and the machine works then, I assume it is caused by the power > supply. If the machine fails then, it is the motherboard. > > At least it looks like this from distance as you said that it is most > unlikely that the modules are faulty. > > Erich > > > > Andy > > > > On Mon, Jul 16, 2012 at 8:59 PM, Erich Dollansky < > > erichfreebsdlist@ovitrap.com> wrote: > > > > > Hi, > > > > > > On Tuesday 17 July 2012 06:45:18 Andy Young wrote: > > > > I am having trouble with one of our servers and I'm not sure what to > try > > > > next. It has a Supermicro H8DGi-F motherboard with two 16-core AMD > > > > processors and two memory banks, one for each processor. When I > > > originally > > > > built it, I only had one processor and 40 GB of ram. Everything > worked > > > > awesome. I recently upgraded it, adding another processor and > another 40 > > > GB > > > > of ram. It was incredibly unstable and constantly rebooted within > minute > > > or > > > > two of uptime, sometimes it wouldn't even boot all the way before > > > crashing > > > > and rebooting again. Seemed like a memory issue so I scaled it back > to > > > two > > > > processors and 32 GB (4x8GB) of ram. Worked well so I added the > > > remaining 8 > > > > GB sticks I had, bringing it up to 64 GB. Still worked great. The > sticks > > > I > > > > had left were a mix and match variety of 8GB and 4GB sticks. Thinking > > > maybe > > > > there was some problem with mixing them, I ordered more 8GB memory > just > > > > like the ones in the box. While waiting for the new memory, the > machine > > > > performed great with no issues. New memory arrived and I added two > more > > > 8GB > > > > sticks. Immediately the constant crashing returned. It seems really > > > > unlikely that I got bad memory in two separate orders. Does anyone > have > > > any > > > > other ideas? Again, its perfectly stable with two processors and 64 > GB of > > > > memory but goes nuts when I more. > > > > > > > could it be caused by the power supply? > > > > > > Did you run a memory test? > > > > > > If possible, try different power supplies. > > > > > > > I really appreciate the help!! > > > > > > > > Motherboard: Supermicro H8DGi-F > > > > CPU: 2 x AMD 6274 (2.2 Ghz 16-core) > > > > Memory: Kingston 8GB DDR3 1333 > > > > > > No ECC? > > > > > > Erich > > > > > > > > > > > -- > > Andrew Young > > Mosaic Storage Systems, Inc > > http://www.mosaicarchive.com/ > > > > Follow us on: > > Twitter , > > Facebook > > , Google Plus< > https://plus.google.com/b/102077382489657821832/https://plus.google.com/b/104681960235222388167/104681960235222388167/posts > > > > , Pinterest > > > -- Andrew Young Mosaic Storage Systems, Inc http://www.mosaicarchive.com/ Follow us on: Twitter , Facebook , Google Plus , Pinterest From owner-freebsd-hardware@FreeBSD.ORG Thu Jul 19 15:23:08 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 35A40106566B for ; Thu, 19 Jul 2012 15:23:08 +0000 (UTC) (envelope-from erichfreebsdlist@ovitrap.com) Received: from alogreentechnologies.com (alogreentechnologies.com [67.212.224.110]) by mx1.freebsd.org (Postfix) with ESMTP id BAD128FC1A for ; Thu, 19 Jul 2012 15:23:07 +0000 (UTC) Received: from amd620.ovitrap.com ([49.128.188.2]) (authenticated bits=0) by alogreentechnologies.com (8.13.1/8.13.1) with ESMTP id q6JFN2C4003445; Thu, 19 Jul 2012 09:23:05 -0600 From: Erich Dollansky To: Andy Young Date: Thu, 19 Jul 2012 22:25:24 +0700 User-Agent: KMail/1.13.7 (FreeBSD/8.3-STABLE; KDE/4.7.4; amd64; ; ) References: <201207180650.11035.erichfreebsdlist@ovitrap.com> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Message-Id: <201207192225.24876.erichfreebsdlist@ovitrap.com> Cc: freebsd-hardware@freebsd.org Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2012 15:23:08 -0000 Hi, On Thursday 19 July 2012 22:08:23 Andy Young wrote: > > > because the machine needs more electricity with the extra modules. If it > is at the limits without, it could go behind > with the additional modules > installed. > > Interesting. The chassis has dual 900W power supplies. Apart from simply > replacing them, I am not sure how I can verify whether the power supply is > the issue. the next question is if the machine can run with only one. If it is so, then remove one when you have the failing configuration. If it still fails, remove the other one an bring the first one back. > > > You did not answer the question regarding ECC. > > The memory modules I added are listed on Newegg as Kingston 8 GB 240-Pin > DDRS SDRAM ECC Registered DDR3 1333 Server Memory. So yes they have ECC. > > > Did you mix the modules? > > Yes. There is a mix of modules in there. The original 32 GB of memory that > I put in to begin with are not the exact same module. They are Hynix 8GB > PC3-10600 DDR3-1333MHz ECC Registered CL9 240-Pin DIMM Dual that came from This must work as both are registered and ECC. In addition you should have a BIOS option like 'scrup' the RAM when booting. This takes some time but might shows already the problem. Erich From owner-freebsd-hardware@FreeBSD.ORG Thu Jul 19 19:50:29 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B067E106564A for ; Thu, 19 Jul 2012 19:50:29 +0000 (UTC) (envelope-from dieterbsd@engineer.com) Received: from mailout-us.gmx.com (mailout-us.gmx.com [74.208.5.67]) by mx1.freebsd.org (Postfix) with SMTP id 55C388FC1D for ; Thu, 19 Jul 2012 19:50:29 +0000 (UTC) Received: (qmail 25295 invoked by uid 0); 19 Jul 2012 19:10:21 -0000 Received: from 67.206.184.2 by rms-us002 with HTTP Content-Type: text/plain; charset="utf-8" Date: Thu, 19 Jul 2012 15:10:19 -0400 From: "Dieter BSD" Message-ID: <20120719191020.298420@gmx.com> MIME-Version: 1.0 To: freebsd-hardware@freebsd.org X-Authenticated: #74169980 X-Flags: 0001 X-Mailer: GMX.com Web Mailer x-registered: 0 Content-Transfer-Encoding: 8bit X-GMX-UID: yz56cPYV3zOlNR3dAHAhhrx+IGRvb4Aw Subject: Re: Server memory problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2012 19:50:29 -0000 >> because the machine needs more electricity with the extra modules. If it >> is at the limits without, it could go behind > with the additional modules >> installed. > > Interesting. The chassis has dual 900W power supplies. Apart from simply > replacing them, I am not sure how I can verify whether the power supply is > the issue. As a simple test, you could measure the various Voltages with a DC Voltmeter and see if they are within spec. Finding problems like noise and ripple require an oscilloscope. In theory, you should be able to get specs on the maximum power required by the mainboard from the various Voltage rails.  Add the power requirements from any expansion cards, disks, etc. See if any of the totals exceed the specs of the power supply. In practice, Tyan would not tell me the power requirements for my mainboard, just "use one of our recommended power supplies". Which doesn't tell me how many disks I can add.  And sure enough, eventually I added enough disks that I started seeing problems and had to add a second power supply. Perhaps Supermicro will tell you the power requirements for your mainboard? If you can find out the power requirements for the memory, you could add some other load that uses the same or more power from the same rails instead of the extra memory. If the system runs fine with the dummy load, then you aren't running out of power. > The memory modules I added are listed on Newegg as Kingston 8 GB 240-Pin > DDRS SDRAM ECC Registered DDR3 1333 Server Memory. So yes they have ECC. Good. Look through the firmware/bios and see if there are any options for turning ecc on/off. There might be an option for scrubbing.