From owner-freebsd-stable@freebsd.org Wed Nov 13 18:41:35 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0AC61B831A for ; Wed, 13 Nov 2019 18:41:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Ctk61Stbz4P07 for ; Wed, 13 Nov 2019 18:41:33 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x842.google.com with SMTP id o3so3684509qtj.8 for ; Wed, 13 Nov 2019 10:41:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cXS0zB+IPcPPtJtyjKAqVcdaGr+zUEOgTIP9hG7PsdE=; b=uCxyTD+A9WOtuzuD8vGfMFB5wbAhfnvOuIkWPu4N8YkGown1dQ0N9g0qTvEKcNMowg WtnjTigKBO7L7yaBgAHWv3L0gBI0/e+LhfzrhiJSxSw57t6G7e+tlRrRjm+FG3ZUnu+s ma3gi5B0axAg1OCa2iVNX465Vpb3emk6UY58ARGWLDutdNWrmuLY1nj33zAR4bHVFdcJ 8mihO5PlMRgsf621ercpZF+SkApQ1xISu/0nj1HSOfLR+PTi0ZU1NF6ADrfMplJHyk23 T+AnYoB5pB40W7S9i15qgtwhEFsDRmEqOxLE7Kzx5mjc4fJczIAxYHxnZWN1iaROz5bt gJOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cXS0zB+IPcPPtJtyjKAqVcdaGr+zUEOgTIP9hG7PsdE=; b=MrCGMrvDqr55hjJahgFm68ujQYluWSqj684UCUqGuqLj2heBFPZu1vhW8lUxqSsbk9 eryeuVXB1IdWNICRfhVx9PeMrFAwHgZS7s7KpY/pw0Wfu160l/YPY8DHp3K7jQ01BKG+ 5BVYMon1Eez8G8fvWCUX3Jh65JRbtPy0IFNw/DoW7WctoL/wuzGuP8TwZw41jHyVBxlE Xukp1/SOxH9YVm7jHbuVJbkMwIOerHF/1kUIeVh1QB8DO+nboDGs/7urs8BvdxkpMopi AMAfXGpO4VONO0kVogfU/n1jT+LeMX36cLOiIe1nSNWQifpKcNzNedShaNFMwpScE/I7 zCnw== X-Gm-Message-State: APjAAAWWgOGAjNgtR3rubvR2S4Tpe/0Ncfoe1H9UQOiFYIYN3IEak93E OmAqR2V14+e+HEtPxrGp4SjnHbKWytfzcK8Qc5e11A== X-Google-Smtp-Source: APXvYqxcSd63AV4sBUZFroluoLl6DPxn7UO7EERBDdNPDX1zjWE+6F5CKlyZVJB6Y0RbvyMyoG2vTuCklDDE0/VCmMc= X-Received: by 2002:ac8:6697:: with SMTP id d23mr3956012qtp.32.1573670492673; Wed, 13 Nov 2019 10:41:32 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 13 Nov 2019 11:41:21 -0700 Message-ID: Subject: Re: NVMe timeout / aborting outstanding i/o (again) To: Theron Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 47Ctk61Stbz4P07 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=uCxyTD+A; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::842) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.44 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.44)[ip: (2.18), ipnet: 2607:f8b0::/32(-2.33), asn: 15169(-1.99), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2019 18:41:36 -0000 On Wed, Nov 13, 2019 at 11:34 AM Theron wrote: > With latest 12.1-STABLE (r354687), I have file access 30 second hang > problem again when resuming from suspend, with dmesg: > > nvme0: Resetting controller due to a timeout. > nvme0: resetting controller > nvme0: aborting outstanding i/o > nvme0: aborting outstanding i/o > (...) > nvme0: aborting outstanding i/o > nvme0: aborting outstanding i/o > nvme0: nvme0: aborting outstanding i/o > async event occurred (type 0x0, info 0x00, page 0x01) > nvme0: aborting outstanding i/o > nvme0: aborting outstanding i/o > (...) > > I thought this was fixed with r351914 "MFC r351747: Implement nvme > suspend / resume for pci attachment." > There are three causes of timeouts this fixed: First, a prior MFC fixed a missed interrupt due to some difference in how some drives implemented read modify write of the MSI registers. Second, during suspend, we weren't properly shutting down the controller. Finally, there was a restoration of the controller that might have had power removed issue the commit also fixed. > Latest change, r354074 "MFC r352630: Make nvme(4) driver some more NUMA > aware.", looks suspicious, I'll test before vs. after that change when I > can. > > Is anyone else seeing this? > I've had no other reports of this, so it's a good one to test. I don't think it will matter, but I'm not sure of that. Warner