From owner-freebsd-hackers@FreeBSD.ORG  Wed Mar 18 14:58:32 2015
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id CF2B1DDE
 for <freebsd-hackers@freebsd.org>; Wed, 18 Mar 2015 14:58:32 +0000 (UTC)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 3CAEF7AF
 for <freebsd-hackers@freebsd.org>; Wed, 18 Mar 2015 14:58:32 +0000 (UTC)
Received: from tom.home (kostik@localhost [127.0.0.1])
 by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2IEwQiv047642
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
 Wed, 18 Mar 2015 16:58:26 +0200 (EET)
 (envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2IEwQiv047642
Received: (from kostik@localhost)
 by tom.home (8.14.9/8.14.9/Submit) id t2IEwQFW047641;
 Wed, 18 Mar 2015 16:58:26 +0200 (EET)
 (envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
 using -f
Date: Wed, 18 Mar 2015 16:58:26 +0200
From: Konstantin Belousov <kostikbel@gmail.com>
To: Michael Fuckner <michael@fuckner.net>
Subject: Re: Server with 3TB Crashing at boot
Message-ID: <20150318145826.GU2379@kib.kiev.ua>
References: <20150316103140.GA2379@kib.kiev.ua> <5506B23F.20400@fuckner.net>
 <20150316105301.GB2379@kib.kiev.ua> <5506E8D6.30703@fuckner.net>
 <20150316154022.GD2379@kib.kiev.ua> <55072195.40609@fuckner.net>
 <20150316185602.GJ2379@kib.kiev.ua> <5507E525.4000209@fuckner.net>
 <20150317220644.GQ2379@kib.kiev.ua> <55097FB7.5080306@fuckner.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <55097FB7.5080306@fuckner.net>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
 DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no
 autolearn_force=no version=3.4.0
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home
Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>,
 Ryan Stone <rysto32@gmail.com>, Steven Hartland <killing@multiplay.co.uk>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Mar 2015 14:58:32 -0000

On Wed, Mar 18, 2015 at 02:37:59PM +0100, Michael Fuckner wrote:
> 
> >>
> >> http://dedi3.fuckner.net/~molli123/temp/head-ixgbe-gas.txt
> > So machine booted, right ?
> >
> > Do network and non-nvme HBAs function correctly ?
> yes!
> 
> http://dedi3.fuckner.net/~molli123/temp/head-nvme.log

The faults reported for NVMe devices are writes to the bus address zero.
I am not sure what is it, the reason could be bug in the DMAR code, bug in
the NVMe driver, or even a hardware issue (I listed this in order from
more likely to less likely).

I will try to find a machine with ix and nvme to do some debugging.