From owner-freebsd-stable@freebsd.org Mon Aug 1 15:32:13 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C47ABAB230 for ; Mon, 1 Aug 2016 15:32:13 +0000 (UTC) (envelope-from mloftis@wgops.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F8D1128E for ; Mon, 1 Aug 2016 15:32:12 +0000 (UTC) (envelope-from mloftis@wgops.com) Received: by mail-wm0-x233.google.com with SMTP id o80so250126219wme.1 for ; Mon, 01 Aug 2016 08:32:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wgops.com; s=gm01; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=rAPxKbi4+eFNw1zHxpujEK4MkxNSPMRp0vcrAoF4zTc=; b=Jz2Vn+KtjBW7GVIiOViN3l1NtAPIj0xmFjH9R+uQiyVvlNKfZu/qF/YgL31gBgWe7x VfnyNFBj/1TNuo/7bt3dLWLYkzFxbG6xdKSinRizEiLwOmHRmsMCdQU2yNjLzFlHptH8 pQrKmd1FOIzb+v2q5l5Ns3aw615F9ocfMRnSs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=rAPxKbi4+eFNw1zHxpujEK4MkxNSPMRp0vcrAoF4zTc=; b=h03vI72XdyUrtL1OW+i2NaPESAtB4dLz8BNNF7/Ai5Q3ceQyGW0swHeeRp023cLg5P mHDekDoIYkgvzlgvk2tIA3idcCQzpeKdzqR+yo8zUPZaQUB4litmwzReN+DwH6iZVZmd YXqPcDMSTZj4P350H1d2M2eHj2aW45JqB4FOTMSpLiDYXxnywQfWt2MJvu8KaPyGQJ/4 LTq+wV60XaE7K8NqaSCCyFMHFEVsKXKVn97wVBUsj+1Xn8t8oZXxvBbIa3cZAFg74FLM J0YatrHBqNS4/0eJSCvfLRJ1TOA8EPk9L+d8dMNG4nm4WI+5BaggZXQfNvViz/9UbGj5 /5xw== X-Gm-Message-State: AEkooutv9m36R7hb8tzUHN0JRT5/XhpkgFmzBTgiGfe4luiLU2JP9VUv4ggolax9zKqbl0iXrQLuK1y7h1fRGw== X-Received: by 10.28.98.135 with SMTP id w129mr55712023wmb.38.1470065530888; Mon, 01 Aug 2016 08:32:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.17.167 with HTTP; Mon, 1 Aug 2016 08:32:10 -0700 (PDT) In-Reply-To: <4996AF96-76BA-47F1-B328-D4FE7AC777EE@sarenet.es> References: <4996AF96-76BA-47F1-B328-D4FE7AC777EE@sarenet.es> From: Michael Loftis Date: Mon, 1 Aug 2016 08:32:10 -0700 Message-ID: Subject: Re: Intel NVMe troubles? To: Borja Marcos Cc: Jim Harris , FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 15:32:13 -0000 FWIW I've had similar issues with Intel 750 PCIe NVMe drives when attempting to use 4K blocks on Linux with EXT4 on top of MD RAID1 (software mirror). I didn't dig much into because too many layers to reduce at the time but it looked like the drive misreported the number of blocks and a subsequent TRIM command or write of the last sector then errored. I mention it because despite the differences the similarities (Intel NVMe, LBA#3/4K) and error writing to a nonexistent block. Might give someone enough info to figure it out fully. On Monday, August 1, 2016, Borja Marcos wrote: > > > On 29 Jul 2016, at 17:44, Jim Harris > wrote: > > > > > > > > On Fri, Jul 29, 2016 at 1:10 AM, Borja Marcos > wrote: > > > > > On 28 Jul 2016, at 19:25, Jim Harris > wrote: > > > > > > Yes, you should worry. > > > > > > Normally we could use the dump_debug sysctls to help debug this - the= se > > > sysctls will dump the NVMe I/O submission and completion queues. But > in > > > this case the LBA data is in the payload, not the NVMe submission > entries, > > > so dump_debug will not help as much as dumping the NVMe DSM payload > > > directly. > > > > > > Could you try the attached patch and send output after recreating you= r > pool? > > > > Just in case the evil anti-spam ate my answer, sent the results to your > Gmail account. > > > > > > Thanks Borja. > > > > It looks like all of the TRIM commands are formatted properly. The > failures do not happen until about 10 seconds after the last TRIM to each > drive was submitted, and immediately before TRIMs start to the next drive= , > so I'm assuming the failures are for the the last few TRIM commands but > cannot say for sure. Could you apply patch v2 (attached) which will dump > the TRIM payload contents inline with the failure messages? > > Sure, this is the complete /var/log/messages starting with the system > boot. Before booting I destroyed the pool > so that you could capture what happens when booting, zpool create, etc. > > Remember that the drives are in LBA format #3 (4 KB blocks). As far as I > know that=E2=80=99s preferred to the old 512 byte blocks. > > Thank you very much and sorry about the belated response. > > > > > > Borja. > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > " --=20 "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler