Date: Thu, 28 Jul 2016 10:25:29 -0700 From: Jim Harris <jim.harris@gmail.com> To: Borja Marcos <borjam@sarenet.es> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: Intel NVMe troubles? Message-ID: <CAJP=Hc-KdmScZtCRDcF=CTpNcMkn2brXiPx4XwJA0aTYgkxm%2Bg@mail.gmail.com> In-Reply-To: <CBC304D0-AA57-4EF5-A2DD-1888FB88DE12@sarenet.es> References: <CBC304D0-AA57-4EF5-A2DD-1888FB88DE12@sarenet.es>
next in thread | previous in thread | raw e-mail | index | archive | help
--001a113d0042a8e15b0538b56b3f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, Jul 28, 2016 at 3:29 AM, Borja Marcos <borjam@sarenet.es> wrote: > Hi :) > > Still experimenting with NVMe drives and FreeBSD, and I have ran into > problems, I think. > > I=C2=B4ve got a server with 10 Intel DC P3500 NVMe drives. Right now, run= ning > 11-BETA2. > > I have updated the firmware in the drives to the latest version (8DV10174= ) > using the Data Center Tools. > And I=E2=80=99ve formatted them for 4 KB blocks (LBA format #3) > > nvmecontrol identify nvme0ns1 > Size (in LBAs): 488378646 (465M) > Capacity (in LBAs): 488378646 (465M) > Utilization (in LBAs): 488378646 (465M) > Thin Provisioning: Not Supported > Number of LBA Formats: 7 > Current LBA Format: LBA Format #03 > LBA Format #00: Data Size: 512 Metadata Size: 0 > LBA Format #01: Data Size: 512 Metadata Size: 8 > LBA Format #02: Data Size: 512 Metadata Size: 16 > LBA Format #03: Data Size: 4096 Metadata Size: 0 > LBA Format #04: Data Size: 4096 Metadata Size: 8 > LBA Format #05: Data Size: 4096 Metadata Size: 64 > LBA Format #06: Data Size: 4096 Metadata Size: 128 > > > ZFS properly detects the 4 KB block size and sets the correct ashift (12)= . > But I=E2=80=99ve found these error messages > generated while I created a pool (zpool create tank raidz2 /dev/nvd[0-8] > spare /dev/nvd9) > > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:63 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:63 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:62 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:62 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:61 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:61 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:60 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:60 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:59 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:59 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:58 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:58 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:57 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:57 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:56 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:56 cdw0:0 > > And the same for the rest of the drives [0-9]. > > Should I worry? > Yes, you should worry. Normally we could use the dump_debug sysctls to help debug this - these sysctls will dump the NVMe I/O submission and completion queues. But in this case the LBA data is in the payload, not the NVMe submission entries, so dump_debug will not help as much as dumping the NVMe DSM payload directly. Could you try the attached patch and send output after recreating your pool= ? -Jim Thanks! > > > > > Borja. > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" --001a113d0042a8e15b0538b56b3f Content-Type: application/octet-stream; name="delete_debug.patch" Content-Disposition: attachment; filename="delete_debug.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ir6l980m0 ZGlmZiAtLWdpdCBhL3N5cy9kZXYvbnZtZS9udm1lX25zLmMgYi9zeXMvZGV2L252bWUvbnZtZV9u cy5jCmluZGV4IDc1NGQwNzQuLjI5M2RkMjUgMTAwNjQ0Ci0tLSBhL3N5cy9kZXYvbnZtZS9udm1l X25zLmMKKysrIGIvc3lzL2Rldi9udm1lL252bWVfbnMuYwpAQCAtNDYxLDYgKzQ2MSw3IEBAIG52 bWVfbnNfYmlvX3Byb2Nlc3Moc3RydWN0IG52bWVfbmFtZXNwYWNlICpucywgc3RydWN0IGJpbyAq YnAsCiAJCSAgICBicC0+YmlvX2Jjb3VudC9udm1lX25zX2dldF9zZWN0b3Jfc2l6ZShucyk7CiAJ CWRzbV9yYW5nZS0+c3RhcnRpbmdfbGJhID0KIAkJICAgIGJwLT5iaW9fb2Zmc2V0L252bWVfbnNf Z2V0X3NlY3Rvcl9zaXplKG5zKTsKKwkJbnZtZV9wcmludGYobnMtPmN0cmxyLCAibGVuZ3RoPSVq dSBsYmE9JWp1XG4iLCAodWludG1heF90KWRzbV9yYW5nZS0+bGVuZ3RoLCAodWludG1heF90KWRz bV9yYW5nZS0+c3RhcnRpbmdfbGJhKTsKIAkJYnAtPmJpb19kcml2ZXIyID0gZHNtX3JhbmdlOwog CQllcnIgPSBudm1lX25zX2NtZF9kZWFsbG9jYXRlKG5zLCBkc21fcmFuZ2UsIDEsCiAJCQludm1l X25zX2Jpb19kb25lLCBicCk7Cg== --001a113d0042a8e15b0538b56b3f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJP=Hc-KdmScZtCRDcF=CTpNcMkn2brXiPx4XwJA0aTYgkxm%2Bg>