From owner-freebsd-stable@freebsd.org Thu Jul 28 17:25:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E5EABA7970 for ; Thu, 28 Jul 2016 17:25:31 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-ua0-x229.google.com (mail-ua0-x229.google.com [IPv6:2607:f8b0:400c:c08::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18B3B12EF for ; Thu, 28 Jul 2016 17:25:31 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: by mail-ua0-x229.google.com with SMTP id 35so44662408uap.1 for ; Thu, 28 Jul 2016 10:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=e2rRJ2pNVxVr6PkCCNkgOkHHZpjHSY39M6MBYe2xywI=; b=L1UFp/p9Xqjn9Sg0hF2mXc/0V4MtII7am5pQnxW4gmfXnrF4nk0ohDay1zA+V9wDYb uGjmuNCBpwtjW8aPII/3dQ+DilxcAQoBqPX64I+v9v4cjcytpy6p8UtnImrZ446jiVDM ff85kRWM/ZaLjHB/yR475whHbAu48HY9P+0CLryMuATTb4jHZ+x1XHcHvgPgPwz4zqAm mbFH9xWlN40rWk2SVaLmzKKPA4QGbvh1pidEPRTJg3S2+uNeuToi8gY9azW6FyXWDcts UohZYLi/ZH2QAYiwS68oP4TYF3+/L9hVHUwWfvWiyzOrcaipGfaxhwKEaC/3C6PuC2Il bejw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=e2rRJ2pNVxVr6PkCCNkgOkHHZpjHSY39M6MBYe2xywI=; b=iuWIxCEb3XRlTIjA7CmceCrQjQFwbKxRK8fUbfFb25az6aFRt7f46L1aclYAVVqDrZ w8aS+wuEVE9ub19HrBq+Or4R5YI0JfWFXENcqWicd+Q1cPqNadMovhxcuSshDeC+Su70 n4sKOzaj7rLaUh4UUV+gJNk0WTDYh8zVa5BVvYMsKbdpsezeqhQagIOkOocAqZ+peY78 WV53hNZI/GR+4pqFSkh4B0yRRt9biIZTZXwXvqEaxdlh/eIKtPJoij5Pvmuo36eDkpAu hy7aj1nFFSf4jfp1/BdZP0w2qQRv4aoR7qHg+V+NfmdJ+0HKyJih2aSOXI1Q1PECJZFg zMZw== X-Gm-Message-State: AEkooutxnKcWwDHGl8VPZjybP3LZrqqq6U+s7lMPnA7uRcKA7u6NdWy2NsjuF2HXc/1tf9uJHF8gvVjCG3VHPg== X-Received: by 10.159.34.55 with SMTP id 52mr15818030uad.2.1469726730010; Thu, 28 Jul 2016 10:25:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.4.229 with HTTP; Thu, 28 Jul 2016 10:25:29 -0700 (PDT) In-Reply-To: References: From: Jim Harris Date: Thu, 28 Jul 2016 10:25:29 -0700 Message-ID: Subject: Re: Intel NVMe troubles? To: Borja Marcos Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/mixed; boundary=001a113d0042a8e15b0538b56b3f X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jul 2016 17:25:31 -0000 --001a113d0042a8e15b0538b56b3f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, Jul 28, 2016 at 3:29 AM, Borja Marcos wrote: > Hi :) > > Still experimenting with NVMe drives and FreeBSD, and I have ran into > problems, I think. > > I=C2=B4ve got a server with 10 Intel DC P3500 NVMe drives. Right now, run= ning > 11-BETA2. > > I have updated the firmware in the drives to the latest version (8DV10174= ) > using the Data Center Tools. > And I=E2=80=99ve formatted them for 4 KB blocks (LBA format #3) > > nvmecontrol identify nvme0ns1 > Size (in LBAs): 488378646 (465M) > Capacity (in LBAs): 488378646 (465M) > Utilization (in LBAs): 488378646 (465M) > Thin Provisioning: Not Supported > Number of LBA Formats: 7 > Current LBA Format: LBA Format #03 > LBA Format #00: Data Size: 512 Metadata Size: 0 > LBA Format #01: Data Size: 512 Metadata Size: 8 > LBA Format #02: Data Size: 512 Metadata Size: 16 > LBA Format #03: Data Size: 4096 Metadata Size: 0 > LBA Format #04: Data Size: 4096 Metadata Size: 8 > LBA Format #05: Data Size: 4096 Metadata Size: 64 > LBA Format #06: Data Size: 4096 Metadata Size: 128 > > > ZFS properly detects the 4 KB block size and sets the correct ashift (12)= . > But I=E2=80=99ve found these error messages > generated while I created a pool (zpool create tank raidz2 /dev/nvd[0-8] > spare /dev/nvd9) > > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:63 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:63 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:62 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:62 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:61 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:61 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:60 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:60 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:59 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:59 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:58 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:58 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:57 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:57 cdw0:0 > Jul 28 13:16:11 nvme2 kernel: nvme0: DATASET MANAGEMENT sqid:6 cid:56 > nsid:1 > Jul 28 13:16:11 nvme2 kernel: nvme0: LBA OUT OF RANGE (00/80) sqid:6 > cid:56 cdw0:0 > > And the same for the rest of the drives [0-9]. > > Should I worry? > Yes, you should worry. Normally we could use the dump_debug sysctls to help debug this - these sysctls will dump the NVMe I/O submission and completion queues. But in this case the LBA data is in the payload, not the NVMe submission entries, so dump_debug will not help as much as dumping the NVMe DSM payload directly. Could you try the attached patch and send output after recreating your pool= ? -Jim Thanks! > > > > > Borja. > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" --001a113d0042a8e15b0538b56b3f Content-Type: application/octet-stream; name="delete_debug.patch" Content-Disposition: attachment; filename="delete_debug.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ir6l980m0 ZGlmZiAtLWdpdCBhL3N5cy9kZXYvbnZtZS9udm1lX25zLmMgYi9zeXMvZGV2L252bWUvbnZtZV9u cy5jCmluZGV4IDc1NGQwNzQuLjI5M2RkMjUgMTAwNjQ0Ci0tLSBhL3N5cy9kZXYvbnZtZS9udm1l X25zLmMKKysrIGIvc3lzL2Rldi9udm1lL252bWVfbnMuYwpAQCAtNDYxLDYgKzQ2MSw3IEBAIG52 bWVfbnNfYmlvX3Byb2Nlc3Moc3RydWN0IG52bWVfbmFtZXNwYWNlICpucywgc3RydWN0IGJpbyAq YnAsCiAJCSAgICBicC0+YmlvX2Jjb3VudC9udm1lX25zX2dldF9zZWN0b3Jfc2l6ZShucyk7CiAJ CWRzbV9yYW5nZS0+c3RhcnRpbmdfbGJhID0KIAkJICAgIGJwLT5iaW9fb2Zmc2V0L252bWVfbnNf Z2V0X3NlY3Rvcl9zaXplKG5zKTsKKwkJbnZtZV9wcmludGYobnMtPmN0cmxyLCAibGVuZ3RoPSVq dSBsYmE9JWp1XG4iLCAodWludG1heF90KWRzbV9yYW5nZS0+bGVuZ3RoLCAodWludG1heF90KWRz bV9yYW5nZS0+c3RhcnRpbmdfbGJhKTsKIAkJYnAtPmJpb19kcml2ZXIyID0gZHNtX3JhbmdlOwog CQllcnIgPSBudm1lX25zX2NtZF9kZWFsbG9jYXRlKG5zLCBkc21fcmFuZ2UsIDEsCiAJCQludm1l X25zX2Jpb19kb25lLCBicCk7Cg== --001a113d0042a8e15b0538b56b3f--