From nobody Tue Jul 5 23:04:50 2022 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 925A31D00343 for ; Tue, 5 Jul 2022 23:04:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LcytV2G3Yz4f00 for ; Tue, 5 Jul 2022 23:04:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 292F46962 for ; Tue, 5 Jul 2022 23:04:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 265N4otp093866 for ; Tue, 5 Jul 2022 23:04:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 265N4oVA093864 for fs@FreeBSD.org; Tue, 5 Jul 2022 23:04:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 264141] nvme(4): Heavy load to SSD wedges 13.1 system: Controller in fatal status, resetting ... Resetting controller due to a timeout and possible hot unplug. Date: Tue, 05 Jul 2022 23:04:50 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: needs-qa, regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: imp@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657062290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/4dp0F9ls/qbEf5ttv0IvTohE4fegWOBWWwJiywEd48=; b=NUb/++AZEspl6yjby6og5Ux0eme5OEIKLdebSm5Q5CF5a18zmxWsfUABeeveKX2UZmQ7VB QEXH5EWZ2gJCwXmTFmvkSGKx5Y3Uy0HbVJPx7mzVyfdGlXVp1UlmehvSCRL5wYhLH+X6a/ m8rxC3/vqSHl926+KkUKxQhlsdmmMEn2Gb3pkGANmLq8nN37gvYC+eLO+67Nkw6odFdS19 4kA3aDqDyZhZ6yKlqryHvFcM5LamsyGN69g5liJLfY1CcFkMMUOUHSV2O3Meuw6NICfi+6 9YZf/KtcC5/lZjuwlOVOAsrlx6dto/97Ec23zpNGey3n4FlT+P/yBI1/6cogJg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1657062290; a=rsa-sha256; cv=none; b=yfog3L3Xpwjl3CigAuI7vcuhn+JzYN5grT7Q3OG+wznQWJkm0YDAYhA68rQnYF3JfpOiEM 1MKNkAUEwS/THB1bVrrtYnl0Qe+lahEaomZWJ13RzGoUL2bTDEoKM1nGlH56fv69AyNHhL wHRwemSgvsHLJqdBcUa9+pvI7OifDsKmVETHPrN1RMS62eIVfGueZb8o4tzeXhvjRFTwl/ nZIypv1UWj+RYTngE5TLQBakJaE3qaNoip4rdYW1xQT/Za0849NsWSOoR4xw2aAebxZLjF EZwk8EfGhKLxoPJIaHRHtWWz+Hh6TLWURKWBGKpwoMZuiENu4LbYk5Yqtq6hrQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264141 --- Comment #23 from Warner Losh --- (In reply to dgilbert from comment #22) > theory: FreeBSD is stomping on the host DRAM reserved for the NVME There's no host ram reserved for nvme, per se. The driver will optionally allocate memory for the drive to use, however. Do you have "nvmeX: Allocated %lluMB host memory buffer" in your dmesg? Without it, you're not using nvme memory. You can set the tunable hw.nvme.hmb_max=3D0 as well to disable usin= g host memory for the DRAM-less cards at the cost of some additional latency if you think that this is the cause of the problem. This would rule it out as a problem. There may be some cards that lose their minds when this is enabled= as well, though I've not seen reports of that in Linux world (I could easily h= ave missed them). Ruling this in/out would be useful... But corrupting host memory seems unlikely to be a cause given that the card drops off the bus and has its memory BARs reset so it isn't decoding anythi= ng (which is what's indicated by the possible hotplug messages). This indicates some kind of power or connection issue to the card, a faulty power controll= er on the card or wonky firmware in the cases that I've diagnosed. There might= be a possible additional cause that's still unknown, but absent better evidence I'm at a loss for where to look. --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.=