From nobody Tue Apr 22 11:22:51 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Zhfwq73JHz5sTfk; Tue, 22 Apr 2025 11:22:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Zhfwq29Wvz3Bwb; Tue, 22 Apr 2025 11:22:51 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745320971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yf4DYsPjN51r/omlmdTVDePbtGHDsjPBdx5Q2AnfpZo=; b=lOZY7O8cROFGChOuiUnjdSlQ1DU/4aIWOBqZK7taXQcBIjirCR9jB261rpG2lpZu3N0dKu kk7MgtLayO4cWTCbQskwSocpOSbIfhdU6gTZwimQmGF3JSy1tpFzUnz0tMpkY+1EOeP6m9 +cnJmNwiERxjd25hHEf5AJ2OmsPVNV6GdEq0dl/c9JHQ15ocHRzWE+E0TKvdNVew0/snB5 v4iDR7CrHWFzReeS420+4KOUMj/HhiwOu3ZsIV98hQr6PBru4LNso7sJpR05P/bNfQBrWC Ix7snAKPuKjW1UOWE61jRwwSlwQUa/itX4RTRxrYzGURZsVUyBENaWGMZax+MA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1745320971; a=rsa-sha256; cv=none; b=lq14OuVW2aGpU+AlbjGFzhJzKVA4xIJpVGcoQhIzX8pjt2dKZfkPi4taiAXxuWHMTZ1Q50 e67L5D05bNc62dQSPJSbPhLPcRPt9s0r1ZN8bM21m3ivyuSfJ8/SKPuIB07MYPofhl0FZx pqZppy57PmaGq4HbGa1SYtnBwguoufQKQgBoXlh+qxIFZ9Axbx5/+ZVHSPMZUwiMfY1M2S iN+2HRWQwpYmwwIXYtxfKQdd4vuHgkOXjAmCpDD0vZFqvL08kiTbc9/HPJo93IKS9ypfr1 nHwWFyj1HdwiNrNVaFHI2PZZpgRwP5CY/My45TudKMvwGU6WREiGMzwaxIGD2Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745320971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yf4DYsPjN51r/omlmdTVDePbtGHDsjPBdx5Q2AnfpZo=; b=xQCI5E41uM1WH/xQT0KTEmh+Th9mx3W2Kor7ZmDW+RL6Sr9L6skIYP839TB2hQg0ZwLTyf p+FdvWIYR2Zm1qWj7bHBMlR3Od0PPBJYFpyBrPdFMqFpihqPGe2KejCfXWAyzr7RnHqwDm +s2u69keY7K4otP6KM8XS2cdOmuMuuCwKNks5ch7/0+u46V+/k26h96gxxdwdj3gghCCLJ gM7EwUBLZ5GdN4WI8CjZG55L0V1afc0u+azcdyt/Gux6uaE0V/6Ybis5Z3mbe/KQUAUbAe XnlxAjhCqeo5l73Lew6vY/E02HuY/qjdwo2ZmnkfVxaoCSRr1R75+6CUDp+Tcw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Zhfwq1VM4z2Vh; Tue, 22 Apr 2025 11:22:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53MBMp2p026226; Tue, 22 Apr 2025 11:22:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53MBMp2S026223; Tue, 22 Apr 2025 11:22:51 GMT (envelope-from git) Date: Tue, 22 Apr 2025 11:22:51 GMT Message-Id: <202504221122.53MBMp2S026223@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Navdeep Parhar Subject: git: ca9d5b10ebfe - stable/14 - cxgbe(4): Perform Conventional Reset instead of FLR on the device. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: np X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: ca9d5b10ebfe810aeb644fb7402523993c5d6434 Auto-Submitted: auto-generated The branch stable/14 has been updated by np: URL: https://cgit.FreeBSD.org/src/commit/?id=ca9d5b10ebfe810aeb644fb7402523993c5d6434 commit ca9d5b10ebfe810aeb644fb7402523993c5d6434 Author: Navdeep Parhar AuthorDate: 2024-12-07 08:00:49 +0000 Commit: Navdeep Parhar CommitDate: 2025-04-22 11:12:46 +0000 cxgbe(4): Perform Conventional Reset instead of FLR on the device. The driver uses bus_reset_child on its parent to reset itself but that performs an FLR whereas the hardware needs a Conventional Reset[1] for full re-initialization. Add routines that perform conventional hot reset and use them instead. The available reset mechanisms are: * PCIe secondary bus reset (default) * PCIe link bounce hw.cxgbe.reset_method can be used to override the default. The internal PL_RST is also available but is for testing only. [1] 6.6.1 in PCI Express® Base Specification 5.0 version 1.0 Sponsored by: Chelsio Communications (cherry picked from commit 011e3d0b8b90a4330f14b2cb7da45ed7b805ed10) --- sys/dev/cxgbe/t4_main.c | 137 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 118 insertions(+), 19 deletions(-) diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c index 4bfb4b007a4f..1f08a1611f1d 100644 --- a/sys/dev/cxgbe/t4_main.c +++ b/sys/dev/cxgbe/t4_main.c @@ -633,6 +633,10 @@ static int t4_reset_on_fatal_err = 0; SYSCTL_INT(_hw_cxgbe, OID_AUTO, reset_on_fatal_err, CTLFLAG_RWTUN, &t4_reset_on_fatal_err, 0, "reset adapter on fatal errors"); +static int t4_reset_method = 1; +SYSCTL_INT(_hw_cxgbe, OID_AUTO, reset_method, CTLFLAG_RWTUN, &t4_reset_method, + 0, "reset method: 0 = PL_RST, 1 = PCIe secondary bus reset, 2 = PCIe link bounce"); + static int t4_clock_gate_on_suspend = 0; SYSCTL_INT(_hw_cxgbe, OID_AUTO, clock_gate_on_suspend, CTLFLAG_RWTUN, &t4_clock_gate_on_suspend, 0, "gate the clock on suspend"); @@ -2546,40 +2550,135 @@ t4_reset_post(device_t dev, device_t child) return (0); } -static int -reset_adapter_with_pci_bus_reset(struct adapter *sc) -{ - int rc; - - mtx_lock(&Giant); - rc = BUS_RESET_CHILD(device_get_parent(sc->dev), sc->dev, 0); - mtx_unlock(&Giant); - return (rc); -} - static int reset_adapter_with_pl_rst(struct adapter *sc) { - suspend_adapter(sc); - /* This is a t4_write_reg without the hw_off_limits check. */ MPASS(sc->error_flags & HW_OFF_LIMITS); bus_space_write_4(sc->bt, sc->bh, A_PL_RST, F_PIORSTMODE | F_PIORST | F_AUTOPCIEPAUSE); pause("pl_rst", 1 * hz); /* Wait 1s for reset */ + return (0); +} - resume_adapter(sc); +static int +reset_adapter_with_pcie_sbr(struct adapter *sc) +{ + device_t pdev = device_get_parent(sc->dev); + device_t gpdev = device_get_parent(pdev); + device_t *children; + int rc, i, lcap, lsta, nchildren; + uint32_t v; - return (0); + rc = pci_find_cap(gpdev, PCIY_EXPRESS, &v); + if (rc != 0) { + CH_ERR(sc, "%s: pci_find_cap(%s, pcie) failed: %d\n", __func__, + device_get_nameunit(gpdev), rc); + return (ENOTSUP); + } + lcap = v + PCIER_LINK_CAP; + lsta = v + PCIER_LINK_STA; + + nchildren = 0; + device_get_children(pdev, &children, &nchildren); + for (i = 0; i < nchildren; i++) + pci_save_state(children[i]); + v = pci_read_config(gpdev, PCIR_BRIDGECTL_1, 2); + pci_write_config(gpdev, PCIR_BRIDGECTL_1, v | PCIB_BCR_SECBUS_RESET, 2); + pause("pcie_sbr1", hz / 10); /* 100ms */ + pci_write_config(gpdev, PCIR_BRIDGECTL_1, v, 2); + pause("pcie_sbr2", hz); /* Wait 1s before restore_state. */ + v = pci_read_config(gpdev, lsta, 2); + if (pci_read_config(gpdev, lcap, 2) & PCIEM_LINK_CAP_DL_ACTIVE) + rc = v & PCIEM_LINK_STA_DL_ACTIVE ? 0 : ETIMEDOUT; + else if (v & (PCIEM_LINK_STA_TRAINING_ERROR | PCIEM_LINK_STA_TRAINING)) + rc = ETIMEDOUT; + else + rc = 0; + if (rc != 0) + CH_ERR(sc, "%s: PCIe link is down after reset, LINK_STA 0x%x\n", + __func__, v); + else { + for (i = 0; i < nchildren; i++) + pci_restore_state(children[i]); + } + free(children, M_TEMP); + + return (rc); +} + +static int +reset_adapter_with_pcie_link_bounce(struct adapter *sc) +{ + device_t pdev = device_get_parent(sc->dev); + device_t gpdev = device_get_parent(pdev); + device_t *children; + int rc, i, lcap, lctl, lsta, nchildren; + uint32_t v; + + rc = pci_find_cap(gpdev, PCIY_EXPRESS, &v); + if (rc != 0) { + CH_ERR(sc, "%s: pci_find_cap(%s, pcie) failed: %d\n", __func__, + device_get_nameunit(gpdev), rc); + return (ENOTSUP); + } + lcap = v + PCIER_LINK_CAP; + lctl = v + PCIER_LINK_CTL; + lsta = v + PCIER_LINK_STA; + + nchildren = 0; + device_get_children(pdev, &children, &nchildren); + for (i = 0; i < nchildren; i++) + pci_save_state(children[i]); + v = pci_read_config(gpdev, lctl, 2); + pci_write_config(gpdev, lctl, v | PCIEM_LINK_CTL_LINK_DIS, 2); + pause("pcie_lnk1", 100 * hz / 1000); /* 100ms */ + pci_write_config(gpdev, lctl, v | PCIEM_LINK_CTL_RETRAIN_LINK, 2); + pause("pcie_lnk2", hz); /* Wait 1s before restore_state. */ + v = pci_read_config(gpdev, lsta, 2); + if (pci_read_config(gpdev, lcap, 2) & PCIEM_LINK_CAP_DL_ACTIVE) + rc = v & PCIEM_LINK_STA_DL_ACTIVE ? 0 : ETIMEDOUT; + else if (v & (PCIEM_LINK_STA_TRAINING_ERROR | PCIEM_LINK_STA_TRAINING)) + rc = ETIMEDOUT; + else + rc = 0; + if (rc != 0) + CH_ERR(sc, "%s: PCIe link is down after reset, LINK_STA 0x%x\n", + __func__, v); + else { + for (i = 0; i < nchildren; i++) + pci_restore_state(children[i]); + } + free(children, M_TEMP); + + return (rc); } static inline int reset_adapter(struct adapter *sc) { - if (vm_guest == 0) - return (reset_adapter_with_pci_bus_reset(sc)); - else - return (reset_adapter_with_pl_rst(sc)); + int rc; + const int reset_method = vm_guest == VM_GUEST_NO ? t4_reset_method : 0; + + rc = suspend_adapter(sc); + if (rc != 0) + return (rc); + + switch (reset_method) { + case 1: + rc = reset_adapter_with_pcie_sbr(sc); + break; + case 2: + rc = reset_adapter_with_pcie_link_bounce(sc); + break; + case 0: + default: + rc = reset_adapter_with_pl_rst(sc); + break; + } + if (rc == 0) + rc = resume_adapter(sc); + return (rc); } static void