From owner-freebsd-stable@freebsd.org Fri Jul 29 15:44:52 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9BE5BA8674 for ; Fri, 29 Jul 2016 15:44:52 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-ua0-x22c.google.com (mail-ua0-x22c.google.com [IPv6:2607:f8b0:400c:c08::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9606D1B3A for ; Fri, 29 Jul 2016 15:44:52 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: by mail-ua0-x22c.google.com with SMTP id 35so64573496uap.1 for ; Fri, 29 Jul 2016 08:44:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=TG2aDVjFULqGLib0P/EgSgsskZ8njWg9if9O7BFCyQk=; b=fcJRX7yn2owhFt0OZMoIpWM94wA6cwlWRApjRqqSNjV3Lejj3kXwdNQ/NLi91WawOV xEzwH6WW9/bF6eOowkd01zRU1ORTcXgwrlMYc332EM3kHSUTqlUAzF8q/N+aRZef43CS SuyD6ZiMevptHPiQtCUl1BLDS0iuXI/VZ8jI9YVwqVjnnuW4ZHHPZ3h7+YmuqPARCpgm ZmbORcvoCaipi4iIuKoWIpTNUcWmLovx+0/2AlZLCGt3Zjz2hyNxqrAEQgSJvV3/QINb igfWTgKACqW3oZqgENAZACtdM377aywIC7sGwhhojMuUM10dsh4sRG6n/3REvBZqyyZK KdWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=TG2aDVjFULqGLib0P/EgSgsskZ8njWg9if9O7BFCyQk=; b=L67BY0kR7mdFR/9GEKM57HNkiGO2HfmzuM9dSOzhvROAYezCGa6mhms2wl0T3oydEX 6mCYG+weiYgaMhdlSjYU8gy1gfA59e1TWNBoan6zL6rXy8yoiwh/qW/6CbBdMIvggQlX pEPQvbQJ4JQZ2ZbV+jBjzcL5dlkOjMLJp2Hes14NHzZAajaJjvePLzYV4tRXmYlUThV1 54RnCCtf9gNrhtFwFoUprlG8weWx5JbQypNAsCxKiR7DP9tM4mGxA9s8IB1fd+IkgYBV sD3lubAxlgWvPC3eHsGTI4nc1N99f+pLTEVCLp2KZly0tM3pdzCfU9sxukRXmLxOS+Aq ADeQ== X-Gm-Message-State: AEkoousfpFwyeZ6ypke0RVB4XhcXUSS1XcVXEHBdrllD6Tn3z9+OLZ0uRlRxXW8rE4Q/rjDZgWOy6qhGQgiXhA== X-Received: by 10.176.0.208 with SMTP id 74mr19580871uaj.64.1469807091223; Fri, 29 Jul 2016 08:44:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.4.229 with HTTP; Fri, 29 Jul 2016 08:44:50 -0700 (PDT) In-Reply-To: References: From: Jim Harris Date: Fri, 29 Jul 2016 08:44:50 -0700 Message-ID: Subject: Re: Intel NVMe troubles? To: Borja Marcos Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/mixed; boundary=001a113ad34a8fabbe0538c82130 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 15:44:53 -0000 --001a113ad34a8fabbe0538c82130 Content-Type: text/plain; charset=UTF-8 On Fri, Jul 29, 2016 at 1:10 AM, Borja Marcos wrote: > > > On 28 Jul 2016, at 19:25, Jim Harris wrote: > > > > Yes, you should worry. > > > > Normally we could use the dump_debug sysctls to help debug this - these > > sysctls will dump the NVMe I/O submission and completion queues. But in > > this case the LBA data is in the payload, not the NVMe submission > entries, > > so dump_debug will not help as much as dumping the NVMe DSM payload > > directly. > > > > Could you try the attached patch and send output after recreating your > pool? > > Just in case the evil anti-spam ate my answer, sent the results to your > Gmail account. > > Thanks Borja. It looks like all of the TRIM commands are formatted properly. The failures do not happen until about 10 seconds after the last TRIM to each drive was submitted, and immediately before TRIMs start to the next drive, so I'm assuming the failures are for the the last few TRIM commands but cannot say for sure. Could you apply patch v2 (attached) which will dump the TRIM payload contents inline with the failure messages? Thanks, -Jim --001a113ad34a8fabbe0538c82130 Content-Type: application/octet-stream; name="delete_debug_v2.patch" Content-Disposition: attachment; filename="delete_debug_v2.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ir7x08cv0 ZGlmZiAtLWdpdCBhL3N5cy9kZXYvbnZtZS9udm1lX25zLmMgYi9zeXMvZGV2L252bWUvbnZtZV9u cy5jCmluZGV4IDc1NGQwNzQuLjI5M2RkMjUgMTAwNjQ0Ci0tLSBhL3N5cy9kZXYvbnZtZS9udm1l X25zLmMKKysrIGIvc3lzL2Rldi9udm1lL252bWVfbnMuYwpAQCAtNDYxLDYgKzQ2MSw3IEBAIG52 bWVfbnNfYmlvX3Byb2Nlc3Moc3RydWN0IG52bWVfbmFtZXNwYWNlICpucywgc3RydWN0IGJpbyAq YnAsCiAJCSAgICBicC0+YmlvX2Jjb3VudC9udm1lX25zX2dldF9zZWN0b3Jfc2l6ZShucyk7CiAJ CWRzbV9yYW5nZS0+c3RhcnRpbmdfbGJhID0KIAkJICAgIGJwLT5iaW9fb2Zmc2V0L252bWVfbnNf Z2V0X3NlY3Rvcl9zaXplKG5zKTsKKwkJbnZtZV9wcmludGYobnMtPmN0cmxyLCAibGVuZ3RoPSVq dSBsYmE9JWp1XG4iLCAodWludG1heF90KWRzbV9yYW5nZS0+bGVuZ3RoLCAodWludG1heF90KWRz bV9yYW5nZS0+c3RhcnRpbmdfbGJhKTsKIAkJYnAtPmJpb19kcml2ZXIyID0gZHNtX3JhbmdlOwog CQllcnIgPSBudm1lX25zX2NtZF9kZWFsbG9jYXRlKG5zLCBkc21fcmFuZ2UsIDEsCiAJCQludm1l X25zX2Jpb19kb25lLCBicCk7CmRpZmYgLS1naXQgYS9zeXMvZGV2L252bWUvbnZtZV9xcGFpci5j IGIvc3lzL2Rldi9udm1lL252bWVfcXBhaXIuYwppbmRleCA5MmZlNjcyLi42ZDM2ZDMzIDEwMDY0 NAotLS0gYS9zeXMvZGV2L252bWUvbnZtZV9xcGFpci5jCisrKyBiL3N5cy9kZXYvbnZtZS9udm1l X3FwYWlyLmMKQEAgLTMxOSw2ICszMTksMTMgQEAgbnZtZV9xcGFpcl9jb21wbGV0ZV90cmFja2Vy KHN0cnVjdCBudm1lX3FwYWlyICpxcGFpciwgc3RydWN0IG52bWVfdHJhY2tlciAqdHIsCiAKIAlp ZiAoZXJyb3IgJiYgcHJpbnRfb25fZXJyb3IpIHsKIAkJbnZtZV9xcGFpcl9wcmludF9jb21tYW5k KHFwYWlyLCAmcmVxLT5jbWQpOworCQlpZiAocXBhaXItPmlkID4gMCAmJiByZXEtPmNtZC5vcGMg PT0gTlZNRV9PUENfREFUQVNFVF9NQU5BR0VNRU5UKSB7CisJCQlzdHJ1Y3QgbnZtZV9kc21fcmFu Z2UgKmRzbV9yYW5nZTsKKworCQkJZHNtX3JhbmdlID0gcmVxLT51LnBheWxvYWQ7CisJCQludm1l X3ByaW50ZihxcGFpci0+Y3RybHIsICJ0cmltIGZhaWxlZDogbGVuPSVqdSBsYmE9JWp1XG4iLAor CQkJCSAgICAodWludG1heF90KWRzbV9yYW5nZS0+bGVuZ3RoLCAodWludG1heF90KWRzbV9yYW5n ZS0+c3RhcnRpbmdfbGJhKTsKKwkJfQogCQludm1lX3FwYWlyX3ByaW50X2NvbXBsZXRpb24ocXBh aXIsIGNwbCk7CiAJfQogCg== --001a113ad34a8fabbe0538c82130--