Date: Tue, 6 Oct 2015 10:35:57 -0500 From: Sean Kelly <smkelly@smkelly.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, Jim Harris <jim.harris@gmail.com> Subject: Re: Dell NVMe issues Message-ID: <27228FE7-5FF9-4F58-9E23-42A66806C374@smkelly.org> In-Reply-To: <20151006152955.GA16596@zxy.spb.ru> References: <BC5F191D-FEB2-4ADC-9D6B-240C80B2301C@smkelly.org> <20151006152955.GA16596@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Oct 6, 2015, at 10:29 AM, Slawa Olhovchenkov <slw@zxy.spb.ru> = wrote: >=20 > On Tue, Oct 06, 2015 at 10:18:11AM -0500, Sean Kelly wrote: >=20 >> Back in May, I posted about issues I was having with a Dell PE R630 = with 4x800GB NVMe SSDs. I would get kernel panics due to the inability = to assign all the interrupts because of = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321 = <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321>; = <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321 = <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321>>. Jim = Harris helped fix this issue so I bought several more of these servers, = Including ones with 4x1.6TB drives... >>=20 >> while the new servers with 4x800GB drives still work, the ones with = 4x1.6TB drives do not. When I do a >> zpool create tank mirror nvd0 nvd1 mirror nvd2 nvd3 >> the command never returns and the kernel logs: >> nvme0: resetting controller >> nvme0: controller ready did not become 0 within 2000 ms >>=20 >> I've tried several different things trying to understand where the = actual problem is. >> WORKS: dd if=3D/dev/nvd0 of=3D/dev/null bs=3D1m >> WORKS: dd if=3D/dev/zero of=3D/dev/nvd0 bs=3D1m >> WORKS: newfs /dev/nvd0 >> FAILS: zpool create tank mirror nvd[01] >> FAILS: gpart add -t freebsd-zfs nvd[01] && zpool create tank mirror = nvd[01]p1 >> FAILS: gpart add -t freebsd-zfs -s 1400g nvd[01[ && zpool create tank = nvd[01]p1 >> WORKS: gpart add -t freebsd-zfs -s 800g nvd[01] && zpool create tank = nvd[01]p1 >>=20 >> NOTE: The above commands are more about getting the point across, not = validity. I wiped the disk clean between gpart attempts and used GPT. >=20 > Just for purity of the experiment: do you try zpool on raw disk, w/o > GPT? I.e. zpool create tank mirror nvd0 nvd1 >=20 Yes, that was actually what I tried first. I headed down the path of GPT = because it allowed me a way to restrict how much disk zpool touched. = zpool on the bare NVMe disks also triggers the issue. >> So it seems like zpool works if I don't cross past ~800GB. But other = things like dd and newfs work. >>=20 >> When I get the kernel messages about the controller resetting and = then not responding, the NVMe subsystem hangs entirely. Since my boot = disks are not NVMe, the system continues to work but no more NVMe stuff = can be done. Further, attempting to reboot hangs and I have to do a = power cycle. >>=20 >> Any thoughts on what the deal may be here? >>=20 >> 10.2-RELEASE-p5 >>=20 >> nvme0@pci0:132:0:0: class=3D0x010802 card=3D0x1f971028 = chip=3D0xa820144d rev=3D0x03 hdr=3D0x00 >> vendor =3D 'Samsung Electronics Co Ltd' >> class =3D mass storage >> subclass =3D NVM >>=20 >> --=20 >> Sean Kelly >> smkelly@smkelly.org >> http://smkelly.org >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org <mailto:freebsd-stable@freebsd.org> = mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable = <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>; >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = <mailto:freebsd-stable-unsubscribe@freebsd.org>"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?27228FE7-5FF9-4F58-9E23-42A66806C374>