From owner-freebsd-stable@freebsd.org Tue Oct 6 15:18:18 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37DFD9D004C for ; Tue, 6 Oct 2015 15:18:18 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 096FE1383 for ; Tue, 6 Oct 2015 15:18:17 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id B44F9202A5 for ; Tue, 6 Oct 2015 11:18:10 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute1.internal (MEProxy); Tue, 06 Oct 2015 11:18:10 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=smkelly.org; h=cc :content-type:date:from:message-id:mime-version:subject:to :x-sasl-enc:x-sasl-enc; s=mesmtp; bh=/ynJDqQXC2WGnMC3+PlSdXhkQNc =; b=BRqPoWb+SYCXS0RTNSsrWgKOVEKtFkIBZMpzBu/6CmH4tONrDE1BYueVv1M 6Rslx6W4ySBK9g4mTiCwHJuTOfL5HgyFyX5GAvz/g5HckFaqnYuJCOs5dWNrzToU 11AEUMt8RcM5UdxdYvA87j2zttf5BsxBWZiRbByb7dOvPhvs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:message-id :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=/y nJDqQXC2WGnMC3+PlSdXhkQNc=; b=etX82f4+l39+bHqMJRtUgIjccCNipipci4 Kqr6MpxAs5Y8cIsePsIbOgw+XIEDIXV1m8O943FrMXgRIS5SoWUcjE6MczJ9MnJc 3/vWfv2cjNIit/86x4IN7f2d7KtBlQ/xwN3XQccU6iLkZmoINFFMRqlxEbaeY/G2 v3Ero8KFU= X-Sasl-enc: 0cf9XXQCEhCIS+nODszNdj6KgDCPutR2XdqMugohvx6+ 1444144690 Received: from [172.16.32.220] (unknown [38.100.147.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 612666800C5; Tue, 6 Oct 2015 11:18:10 -0400 (EDT) From: Sean Kelly Subject: Dell NVMe issues Date: Tue, 6 Oct 2015 10:18:11 -0500 Message-Id: Cc: Jim Harris To: FreeBSD-STABLE Mailing List Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3093\)) X-Mailer: Apple Mail (2.3093) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2015 15:18:18 -0000 Back in May, I posted about issues I was having with a Dell PE R630 with = 4x800GB NVMe SSDs. I would get kernel panics due to the inability to = assign all the interrupts because of = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321 = . Jim Harris = helped fix this issue so I bought several more of these servers, = Including ones with 4x1.6TB drives=E2=80=A6 while the new servers with 4x800GB drives still work, the ones with = 4x1.6TB drives do not. When I do a zpool create tank mirror nvd0 nvd1 mirror nvd2 nvd3 the command never returns and the kernel logs: nvme0: resetting controller nvme0: controller ready did not become 0 within 2000 ms I=E2=80=99ve tried several different things trying to understand where = the actual problem is. WORKS: dd if=3D/dev/nvd0 of=3D/dev/null bs=3D1m WORKS: dd if=3D/dev/zero of=3D/dev/nvd0 bs=3D1m WORKS: newfs /dev/nvd0 FAILS: zpool create tank mirror nvd[01] FAILS: gpart add -t freebsd-zfs nvd[01] && zpool create tank mirror = nvd[01]p1 FAILS: gpart add -t freebsd-zfs -s 1400g nvd[01[ && zpool create tank = nvd[01]p1 WORKS: gpart add -t freebsd-zfs -s 800g nvd[01] && zpool create tank = nvd[01]p1 NOTE: The above commands are more about getting the point across, not = validity. I wiped the disk clean between gpart attempts and used GPT. So it seems like zpool works if I don=E2=80=99t cross past ~800GB. But = other things like dd and newfs work. When I get the kernel messages about the controller resetting and then = not responding, the NVMe subsystem hangs entirely. Since my boot disks = are not NVMe, the system continues to work but no more NVMe stuff can be = done. Further, attempting to reboot hangs and I have to do a power = cycle. Any thoughts on what the deal may be here? 10.2-RELEASE-p5 nvme0@pci0:132:0:0: class=3D0x010802 card=3D0x1f971028 = chip=3D0xa820144d rev=3D0x03 hdr=3D0x00 vendor =3D 'Samsung Electronics Co Ltd' class =3D mass storage subclass =3D NVM --=20 Sean Kelly smkelly@smkelly.org http://smkelly.org