Date: Fri, 12 Apr 2019 21:45:00 +0800 (GMT+08:00) From: kelly <kelly@rsw-power.com> To: <stable@freebsd.org> Subject: =?UTF-8?B?UmU6IEhhbmRoZWxkIFByaW50ZXIgTWFudWZhY3R1cmVy?= Message-ID: <AOYAnwCrCNkIlPm54PwdlqpY.1.1555060634525.Hmail.kelly@rsw-power.com>
next in thread | raw e-mail | index | archive | help
RGVhciBtYW5hZ2VyLAogCldlIHdhbnQgdG8gaW50cm9kdWNlIHlvdSBhIG5ldyBwcm9kdWN0LS0t SGFuZGhlbGQgSW5ramV0IFByaW50ZXIgd2l0aCAzLjUgaW5jaCBUb3VjaCBTY3JlZW4uIEl0J3Mg dmVyeSBob3Qgc2FsZSBub3cuCiAKWW91IGNhbiBwcmludCBkaWZmZXJlbnQgY29udGVudHM6IGxl dHRlcnMsIG51bWJlcnMsIHN5bWJvbHMsIFFSIGNvZGUsIHNjYW4gY29kZSwgYmFyY29kZSwgYmF0 Y2ggY29kZSwgZXhwaXJ5IGRhdGUsIHRpbWUsIGNvdW50ZXIsIGxvdCBudW1iZXIsIGxvZ28gYW5k IG1hcmtzIHZpYSB0b3VjaCBzY3JlZW4gb3IgaW1wb3J0ZWQgdGhlIHByaW50aW5nIGNvbnRlbnQg dmlhIFVTQiBmbGFzayBkaXNrLgogCk91ciBoYW5kaGVsZCBpbmtqZXQgcHJpbnRlciB3aXRoIHNv bHZlbnQgaW5rIGNhbiBwcmludCBvbiBkaWZmZXJlbnQga2luZHMgb2YgbWF0ZXJpYWxzLCBzdWNo IGFzIHBhcGVyLCBjYXJ0b24sIGxhYmVsLCBwbGFzdGljLCBQRSBiYWcsIGdsYXNzLCB3b29kLCBz dGVlbCBwaXBlLCB0dWJlLCBQVkMsIG1ldGFsLCB3YWxsLCBzdG9uZSBib2FyZCwgZmFicmljLCB0 ZXh0aWxlLCBmb2lsLCBmaWxtIG1hdGVyaWFsLCBldGMuCgogCk91ciBoYW5kaGVsZCBpbmtqZXQg cHJpbnRlciBpcyB2ZXJ5IGNyZWF0aXZlLCBwb3J0YWJsZSBhbmQgY2FuIGJlIHdpZGVseSB1c2Vk IGluIGRpZmZlcmVudCBpbmR1c3RyaWVzLgogCkNvbnRhY3QgdXMgaWYgeW91IG5lZWQsIGxldCdz IHRhbGsgZGV0YWlscy4gb3IgcGxlYXNlIGZvcndhcmQgdGhpcyBlbWFpbCB0byB3aG8gaW50ZXJl c3RlZCBpbiBpdC4KIApLaW5kIHJlZ2FyZHMsCi0tCktlbGx5IENoZW4gCk06IDg2LTEzMjY2Njg1 NjY5ClNreXBlOiBrZWxseV8zMTY3CldlYjp3d3cuYmVzaGVuZ3ByaW50ZXJzLmNvbQoKCgoNCg0K From owner-freebsd-stable@freebsd.org Fri Apr 12 14:43:58 2019 Return-Path: <owner-freebsd-stable@freebsd.org> Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14DCA157E3EA for <freebsd-stable@mailman.ysv.freebsd.org>; Fri, 12 Apr 2019 14:43:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 007BC760FE for <freebsd-stable@freebsd.org>; Fri, 12 Apr 2019 14:43:56 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x834.google.com with SMTP id k14so11538333qtb.0 for <freebsd-stable@freebsd.org>; Fri, 12 Apr 2019 07:43:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PPaEkKifsEsxR2THc1YsOuRdQpQu4pb8stt5pd9IwOA=; b=P60NcOy7R3VEdzIe5ymJv0h+HLqHuJuX8TI6epGIff9T3i9muRX4ZOpmAg8CuLSDQl cLm6nswgdk6itO4x/XJfpCq3iWsCBVXpO8KVBcWtY/+MLbjgGNtakDtH8iYH6wPDoaiz puyUc+vgpwMFFZH3OYEbhr4P1kok9aa3/xX4Tu7UiaUrscxN1FnDR/O1b/eVSXzzBOhY /7IqIMxjDcX6r2Wq/4xOORiZO3ba37MXnikLUThJKocO+5VFCbhFqo7D+HBQXiSNvdjr 3TgJwzb6PjznKhIHZisDWvsmhqKM/ZbKBUY4lKeyhhrrwoXNa6WKLvhcTl9xmibL3hNA i6XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PPaEkKifsEsxR2THc1YsOuRdQpQu4pb8stt5pd9IwOA=; b=H0trFBEsUp+e/Pnna9ubUVcbwxWVQ0Gg3lXnlK63wx9QtXyhWWMs0WlbdB6URUhPsf qyXL6glnqGUK56luDTg9ukR/0N9LJ+Jg/yzmc9fOyunb5PBRYc3QXNn5AwUfpI8F/Ert FV4SKhq/lt03f31Aj3v8UsDRzO+Nyq67kPMzCE0QpYPV9+pRAJ6V7YqyYqq4QMQDkL/t g40oN5aPLEcSiogvxf/npyG0XwHzoJN9nFNrUwkF2ZrkJwjjhqULwtnHJVe7/71HjQHY Hjsok04XORnpP5yX4NsD9SPGjYk3ZY6V8a2/X3QuB/WUdnFDbDdvarmsyF5jReopf0mm 61sw== X-Gm-Message-State: APjAAAWKJNFROZwTE2erGPF6fTrzmvU8wQw3XkpaMakRdoTfYWdu7mbb kAeJ+0yIB0IlvWateT3WoxUnT5K6wvRsX/5ylbbxQIAe X-Google-Smtp-Source: APXvYqwgA/b/WwTmZqk2EN8F1SZ+XrgUwIkbWUXgdGcMLygSAmC28gMCh4uNnLERdj7TjOzrmrEqyq9SMnq6jcUsc1Y= X-Received: by 2002:a0c:afa2:: with SMTP id s31mr46307791qvc.118.1555080236302; Fri, 12 Apr 2019 07:43:56 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <CF2365AE-23EA-4F18-9520-C998216155D5@punkt.de> <CANCZdfoPZ9ViQzZ2k8GT5pNw5hjso3rzmYxzU=s+3K=ze+LZwg@mail.gmail.com> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <CANCZdfpeZ-MMKB3Sh=3vhsjJcmFkGG7Jq8nW52D5S45PL3menA@mail.gmail.com> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> In-Reply-To: <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> From: Warner Losh <imp@bsdimp.com> Date: Fri, 12 Apr 2019 08:43:45 -0600 Message-ID: <CANCZdfrcnRwqDPXMyT6xNKUZ5nX8x9Fj6DHbCnh+Q4mWzx0vGQ@mail.gmail.com> Subject: Re: NVME aborting outstanding i/o and controller resets To: "Patrick M. Hausen" <hausen@punkt.de> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> X-Rspamd-Queue-Id: 007BC760FE X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=P60NcOy7 X-Spamd-Result: default: False [-5.48 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.71)[-0.708,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[ALT1.aspmx.l.google.com,aspmx.l.google.com,ALT2.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.76)[ip: (-8.63), ipnet: 2607:f8b0::/32(-2.94), asn: 15169(-2.18), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>; List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Fri, 12 Apr 2019 14:43:58 -0000 On Fri, Apr 12, 2019 at 6:00 AM Patrick M. Hausen <hausen@punkt.de> wrote: > Hi all, > > my problems seem not to be TRIM related after all =E2=80=A6 and I can now > quickly reproduce it. > > =3D=3D=3D=3D=3D > root@freenas01[~]# sysctl vfs.zfs.trim.enabled > vfs.zfs.trim.enabled: 0 > =3D=3D=3D=3D=3D > root@freenas01[~]# cd /mnt/zfs > root@freenas01[/mnt/zfs]# dd if=3D/dev/urandom of=3Dhurz bs=3D10m > ^C =E2=80=94 system freezes temporarily > This does one I/O at a time to the filesystem, which then repackages the I/Os such that multiple I/Os are going on with the NVMe card. > =3D=3D=3D=3D=3D > Apr 12 13:42:16 freenas01 nvme6: resetting controller > OK. This means that whatever I/O workload we've done has caused the NVME card to stop responding for 30s, so we reset it. > Apr 12 13:42:16 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:42:16 freenas01 nvme6: WRITE sqid:1 cid:117 nsid:1 lba:98182510= 4 > len:176 > Apr 12 13:42:16 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:117 cdw0:0 > But only one request was in flight... And we keep doing it over and over again, but to different LBAs suggesting that we're stuttering: a few go through and then things wedge again. This happens every 30ish seconds. > Apr 12 13:42:49 freenas01 nvme6: resetting controller > Apr 12 13:42:50 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:42:50 freenas01 nvme6: WRITE sqid:1 cid:127 nsid:1 lba:98410793= 6 > len:96 > Apr 12 13:42:50 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:127 cdw0:0 > Apr 12 13:43:35 freenas01 nvme6: resetting controller > Apr 12 13:43:35 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:43:35 freenas01 nvme6: WRITE sqid:1 cid:112 nsid:1 lba:97617203= 2 > len:176 > Apr 12 13:43:35 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:112 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: resetting controller > And then this one goes wonkies. > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:111 nsid:1 lba:97619917= 6 > len:248 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:111 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:102 nsid:1 lba:97619943= 2 > len:248 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:102 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:112 nsid:1 lba:97619968= 0 > len:8 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:112 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:105 nsid:1 lba:97619975= 2 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:105 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:122 nsid:1 lba:97619981= 6 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:122 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:103 nsid:1 lba:97619968= 8 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:103 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:126 nsid:1 lba:97620013= 6 > len:56 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:126 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:106 nsid:1 lba:97620019= 2 > len:8 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:106 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:107 nsid:1 lba:97620020= 0 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:107 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:127 nsid:1 lba:97620026= 4 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:127 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:113 nsid:1 lba:97620032= 8 > len:120 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:113 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:108 nsid:1 lba:97620044= 8 > len:72 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:108 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:116 nsid:1 lba:97620052= 0 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:116 cdw0:0 > =3D=3D=3D=3D=3D > root@freenas01[~]# nvmecontrol identify nvme6 > Controller Capabilities/Features > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > Vendor ID: 8086 > Subsystem Vendor ID: 8086 > Serial Number: BTLJ90230EC61P0FGN > Model Number: INTEL SSDPE2KX010T8 > So it's an intel card. > Firmware Version: VDV10131 > Recommended Arb Burst: 0 > IEEE OUI Identifier: e4 d2 5c > Multi-Interface Cap: 00 > Max Data Transfer Size: 131072 > Controller ID: 0x00 > > Admin Command Set Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > Security Send/Receive: Not Supported > Format NVM: Supported > Firmware Activate/Download: Supported > Namespace Managment: Supported > Abort Command Limit: 4 > Async Event Request Limit: 4 > Number of Firmware Slots: 1 > Firmware Slot 1 Read-Only: No > Per-Namespace SMART Log: No > Error Log Page Entries: 64 > Number of Power States: 1 > > NVM Command Set Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > Submission Queue Entry Size > Max: 64 > Min: 64 > Completion Queue Entry Size > Max: 16 > Min: 16 > Number of Namespaces: 1 > Compare Command: Not Supported > Write Uncorrectable Command: Supported > Dataset Management Command: Supported > Volatile Write Cache: Not Present > > Namespace Drive Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > NVM total cap: 1000204886016 > NVM unallocated cap: 0 > =3D=3D=3D=3D=3D > root@freenas01[~]# zpool status > pool: freenas-boot > state: ONLINE > scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Sun Apr 7 > 03:45:03 2019 > config: > > NAME STATE READ WRITE CKSUM > freenas-boot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > nvd0p2 ONLINE 0 0 0 > nvd1p2 ONLINE 0 0 0 > > errors: No known data errors > > pool: zfs > state: ONLINE > scan: scrub repaired 0 in 0 days 00:01:53 with 0 errors on Fri Mar 22 > 19:53:37 2019 > config: > > NAME STATE READ > WRITE CKSUM > zfs ONLINE 0 > 0 0 > raidz2-0 ONLINE 0 > 0 0 > gptid/97d0a7ce-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98053880-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/983a9468-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/987100f2-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98aa6e88-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98f20b8c-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > > errors: No known data errors > =3D=3D=3D=3D=3D > > The problem only appears on the data pool built from 6 INTEL > SSDPE2KX010T8. The system > pool built from two KXG50ZNV256G TOSHIBA does not show any problem with > write load. > OK. That suggests Intel has a problem with their firmware. > All the Intel drives have the latest firmware according to the Intel > support website. > > Could it possibly help to tweak dev.nvme.7.ioq0.num_entries and similar > entries? > It might. Maybe. I'm unsure what the intel firmware bug is. > What about switching to the nda device instead of nvd? > I doubt that would have any effect. They both throw as much I/O onto the card as possible in the default config. There's been some minor improvements in -current here. Any chance you could experimentally try that with this test? You won't get as many I/O abort errors (since we don't print those), and we have a few more workarounds for the reset path (though honestly, it's still kinda stinky). Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AOYAnwCrCNkIlPm54PwdlqpY.1.1555060634525.Hmail.kelly>