From nobody Tue Feb 25 16:48:18 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Z2NpB4f5Cz5qBf3; Tue, 25 Feb 2025 16:48:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Z2NpB3y6Rz3QwF; Tue, 25 Feb 2025 16:48:18 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1740502098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NiSU14r71Q5XUroEcCU21G93G+cgtVgy8kxjydR6OI4=; b=c8CaLpUOy8LMeQlumn611xann+NQ6CaRjNEdoq9rMuzc//uAeYGmuj1eqiPGdimsxDgQfR vMVW0OU5OP9jIbrIJUVFH0Jn70sRULrgbYT3UgoyTw/fcsoapA9BtdwYtmfC20HwoNtY05 cVTNRBGWJlF8B79oakDMYRxGrUYwFidScEb7xb/fPz9jCjK7rjdobrDkLYsdf/88hmHEWM TpJqNS5rtspPm4g7MHRh7HgtVKopTBpNMv3tUuyygAM5AUPg+j8nHRQjWHjXBmrXLEfYCo fPIgYZwxtnKwptSuLwhWvEfmvwXJR4B2numz8r6vIwPjuX/m19+aZPVePDbRdg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1740502098; a=rsa-sha256; cv=none; b=UY56jgJFXvDsAzqKIxNxJZZIomZk5TYeHhfYTh0mHacyId3k6P+WrFI2a/J014kfqceRzc pQAwAGuDrl67UYB2qg1njnzeP6MaUBWOfmtgFL4BeZPMABWG6YHyBRfMcwdguRxx62tJwj wdIQGFHRfveJuCbMTNkWW665P6NWGaR6rzTzcTMgLnAQ0Fhk9An+adcOYcDIqphnD3+Gpf /OlE0AumITTBS0yVj2VgjGrauYzdwQElb8puRJ7RpDej8PXTimcT3qYlJngFyjdTGJKPv3 BEuVdGP15ngdFCIVtQ2veWjJdiANVCn3ZReN/1V0EvspyHwyDSL2uPdHxr2adQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1740502098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NiSU14r71Q5XUroEcCU21G93G+cgtVgy8kxjydR6OI4=; b=pLDngdlGsgBcEpKmemVWIIyFAamFeMJTN7f5IX+bmSeQ6qnyWKIT5o7UPVWF6DDENSpDEW 7NK1nENb1EByozqCpeCqJDBlETPIwvDpKHInznTQ6M+MFLynBM0NgPMULewjkPrD4bKoND EgeIxFm9S91G5KgtwW8sVTpiiwJa7KLJNNxeV03YuRIDKhFmYviKKlK/ICCzu2GDCAm3Xw PvMblocIMi5fRg+vAMftMsKiJ6RGrble4W2cs6hZ7ImSq0rYopynP+7UNe1PluW8rK4DWi J/zlctkmXdQWBzXfusp+G55/ZxLSVeSiUbVYD73gmxKx92G8+Btvo8xDWEzywg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Z2NpB38LmzZqB; Tue, 25 Feb 2025 16:48:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 51PGmIBo013176; Tue, 25 Feb 2025 16:48:18 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 51PGmISF013173; Tue, 25 Feb 2025 16:48:18 GMT (envelope-from git) Date: Tue, 25 Feb 2025 16:48:18 GMT Message-Id: <202502251648.51PGmISF013173@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: dc95228d9847 - main - nvme: Fix hotplug on one of the amazon platforms List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: dc95228d98474aba940e3885164912b419c5579d Auto-Submitted: auto-generated The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=dc95228d98474aba940e3885164912b419c5579d commit dc95228d98474aba940e3885164912b419c5579d Author: Warner Losh AuthorDate: 2025-02-25 16:29:14 +0000 Commit: Warner Losh CommitDate: 2025-02-25 16:36:53 +0000 nvme: Fix hotplug on one of the amazon platforms Amazon EC2 m7i cloud instances use PCI hotplug rather than ACPI hotplug. The card is removed and detach is called to remove the drive from the system. The hardware is no longer present at this point, but the bridge doesn't translate the now-missing hardware reads to all ff's leading us to conclude the hardware is there and we need to do a proper shutdown of it. Fix this oversight by asking the bridge if the device is still present as well. We need both tests since some systems one cane remove the card w/o a hotplug event and we want to fail-safe in those cases. Convert gone to a bool while I'm here and update a comment about shutting down the controller and why that's important. Tested by: cperciva Sponsored by: Netflix --- sys/dev/nvme/nvme_ctrlr.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index 98a9e62f851b..c47e7ece5a04 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -1612,7 +1612,8 @@ nvme_ctrlr_construct(struct nvme_controller *ctrlr, device_t dev) void nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev) { - int gone, i; + int i; + bool gone; ctrlr->is_dying = true; @@ -1622,10 +1623,16 @@ nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev) goto noadminq; /* - * Check whether it is a hot unplug or a clean driver detach. - * If device is not there any more, skip any shutdown commands. + * Check whether it is a hot unplug or a clean driver detach. If device + * is not there any more, skip any shutdown commands. Some hotplug + * bridges will return zeros instead of ff's when the device is + * departing, so ask the bridge if the device is gone. Some systems can + * remove the drive w/o the bridge knowing its gone (they don't really + * do hotplug), so failsafe with detecting all ff's (impossible with + * this hardware) as the device being gone. */ - gone = (nvme_mmio_read_4(ctrlr, csts) == NVME_GONE); + gone = bus_child_present(dev) == 0 || + (nvme_mmio_read_4(ctrlr, csts) == NVME_GONE); if (gone) nvme_ctrlr_fail(ctrlr, true); else @@ -1653,17 +1660,17 @@ nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev) nvme_admin_qpair_destroy(&ctrlr->adminq); /* - * Notify the controller of a shutdown, even though this is due to - * a driver unload, not a system shutdown (this path is not invoked - * during shutdown). This ensures the controller receives a - * shutdown notification in case the system is shutdown before - * reloading the driver. + * Notify the controller of a shutdown, even though this is due to a + * driver unload, not a system shutdown (this path is not invoked uring + * shutdown). This ensures the controller receives a shutdown + * notification in case the system is shutdown before reloading the + * driver. Some NVMe drives need this to flush their cache to stable + * media and consider it a safe shutdown in SMART stats. */ - if (!gone) + if (!gone) { nvme_ctrlr_shutdown(ctrlr); - - if (!gone) nvme_ctrlr_disable(ctrlr); + } noadminq: if (ctrlr->taskqueue)