Date: Thu, 28 Jul 2016 10:25:29 -0700 From: Jim Harris <jim.harris@gmail.com> To: Borja Marcos <borjam@sarenet.es> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: Intel NVMe troubles? Message-ID: <CAJP=Hc-KdmScZtCRDcF=CTpNcMkn2brXiPx4XwJA0aTYgkxm%2Bg@mail.gmail.com> In-Reply-To: <CBC304D0-AA57-4EF5-A2DD-1888FB88DE12@sarenet.es> References: <CBC304D0-AA57-4EF5-A2DD-1888FB88DE12@sarenet.es>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Thu, Jul 28, 2016 at 3:29 AM, Borja Marcos <borjam@sarenet.es> wrote: > Hi :) > > Still experimenting with NVMe drives and FreeBSD, and I have ran into > problems, I think. > > I´ve got a server with 10 Intel DC P3500 NVMe drives. Right now, running > 11-BETA2. > > I have updated the firmware in the drives to the latest version (8DV10174) > using the Data Center Tools. > And I’ve formatted them for 4 KB blocks (LBA format #3) > > nvmecontrol identify nvme0ns1 > Size (in LBAs): 488378646 (465M) > Capacity (in LBAs): 488378646 (465M) > Utilization (in LBAs): 488378646 (465M) > Thin Provisioning: Not Supported > Number of LBA Formats: 7 > Current LBA Format: LBA Format #03 > LBA Format #00: Data Size: 512 Metadata Size: 0 > LBA Format #01: Data Size: 512 Metadata Size: 8 > LBA Format #02: Data Size: 512 Metadata Size: 16 > LBA Format #03: Data Size: 4096 Metadata Size: 0 > LBA Format #04: Data Size: 4096 Metadata Size: 8 > LBA Format #05: Data Size: 4096 Metadata Size: 64 > LBA Format #06: Data Size: 4096 Metadata Size: 128 > > > ZFS properly detects the 4 KB block size and sets the correct ashift (12). > But I’ve found these error messages > generated while I created a pool (zpool create tank raidz2 /dev/nvd[0-8] > spare /dev/nvd9) > > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:63 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:63 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:62 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:62 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:61 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:61 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:60 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:60 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:59 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:59 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:58 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:58 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:57 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:57 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:56 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:56 cdw0:0 > > And the same for the rest of the drives [0-9]. > > Should I worry? > Yes, you should worry. Normally we could use the dump_debug sysctls to help debug this - these sysctls will dump the NVMe I/O submission and completion queues. But in this case the LBA data is in the payload, not the NVMe submission entries, so dump_debug will not help as much as dumping the NVMe DSM payload directly. Could you try the attached patch and send output after recreating your pool? -Jim Thanks! > > > > > Borja. > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" [-- Attachment #2 --] diff --git a/sys/dev/nvme/nvme_ns.c b/sys/dev/nvme/nvme_ns.c index 754d074..293dd25 100644 --- a/sys/dev/nvme/nvme_ns.c +++ b/sys/dev/nvme/nvme_ns.c @@ -461,6 +461,7 @@ nvme_ns_bio_process(struct nvme_namespace *ns, struct bio *bp, bp->bio_bcount/nvme_ns_get_sector_size(ns); dsm_range->starting_lba = bp->bio_offset/nvme_ns_get_sector_size(ns); + nvme_printf(ns->ctrlr, "length=%ju lba=%ju\n", (uintmax_t)dsm_range->length, (uintmax_t)dsm_range->starting_lba); bp->bio_driver2 = dsm_range; err = nvme_ns_cmd_deallocate(ns, dsm_range, 1, nvme_ns_bio_done, bp);
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJP=Hc-KdmScZtCRDcF=CTpNcMkn2brXiPx4XwJA0aTYgkxm%2Bg>