From nobody Thu Apr 3 00:52:03 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZSjr3511Yz5s9RD; Thu, 03 Apr 2025 00:52:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZSjr24kTQz3L5P; Thu, 03 Apr 2025 00:52:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 5330q3F6047267; Thu, 3 Apr 2025 03:52:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 5330q3F6047267 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 5330q31E047266; Thu, 3 Apr 2025 03:52:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 3 Apr 2025 03:52:03 +0300 From: Konstantin Belousov To: Navdeep Parhar Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 011e3d0b8b90 - main - cxgbe(4): Perform Conventional Reset instead of FLR on the device. Message-ID: References: <202503160235.52G2ZIgC026408@gitrepo.freebsd.org> List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <202503160235.52G2ZIgC026408@gitrepo.freebsd.org> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on tom.home X-Spamd-Result: default: False [-0.32 / 15.00]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_HAM_LONG(-0.85)[-0.849]; NEURAL_HAM_SHORT(-0.47)[-0.465]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[dev-commits-src-all@freebsd.org,dev-commits-src-main@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_TLS_LAST(0.00)[]; HAS_XAW(0.00)[] X-Rspamd-Queue-Id: 4ZSjr24kTQz3L5P X-Spamd-Bar: / On Sun, Mar 16, 2025 at 02:35:18AM +0000, Navdeep Parhar wrote: > The branch main has been updated by np: > > URL: https://cgit.FreeBSD.org/src/commit/?id=011e3d0b8b90a4330f14b2cb7da45ed7b805ed10 > > commit 011e3d0b8b90a4330f14b2cb7da45ed7b805ed10 > Author: Navdeep Parhar > AuthorDate: 2024-12-07 08:00:49 +0000 > Commit: Navdeep Parhar > CommitDate: 2025-03-16 01:16:42 +0000 > > cxgbe(4): Perform Conventional Reset instead of FLR on the device. > > The driver uses bus_reset_child on its parent to reset itself but that > performs an FLR whereas the hardware needs a Conventional Reset[1] for > full re-initialization. Add routines that perform conventional hot > reset and use them instead. The available reset mechanisms are: > * PCIe secondary bus reset (default) > * PCIe link bounce > > hw.cxgbe.reset_method can be used to override the default. The internal > PL_RST is also available but is for testing only. > > [1] 6.6.1 in PCI Express® Base Specification 5.0 version 1.0 > > MFC after: 1 month > Sponsored by: Chelsio Communications > --- > sys/dev/cxgbe/t4_main.c | 137 +++++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 118 insertions(+), 19 deletions(-) > > diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c > index 6ee839151db0..20df6a97aa87 100644 > --- a/sys/dev/cxgbe/t4_main.c > +++ b/sys/dev/cxgbe/t4_main.c > @@ -633,6 +633,10 @@ static int t4_reset_on_fatal_err = 0; > SYSCTL_INT(_hw_cxgbe, OID_AUTO, reset_on_fatal_err, CTLFLAG_RWTUN, > &t4_reset_on_fatal_err, 0, "reset adapter on fatal errors"); > > +static int t4_reset_method = 1; > +SYSCTL_INT(_hw_cxgbe, OID_AUTO, reset_method, CTLFLAG_RWTUN, &t4_reset_method, > + 0, "reset method: 0 = PL_RST, 1 = PCIe secondary bus reset, 2 = PCIe link bounce"); > + > static int t4_clock_gate_on_suspend = 0; > SYSCTL_INT(_hw_cxgbe, OID_AUTO, clock_gate_on_suspend, CTLFLAG_RWTUN, > &t4_clock_gate_on_suspend, 0, "gate the clock on suspend"); > @@ -2535,40 +2539,135 @@ t4_reset_post(device_t dev, device_t child) > return (0); > } > > -static int > -reset_adapter_with_pci_bus_reset(struct adapter *sc) > -{ > - int rc; > - > - mtx_lock(&Giant); > - rc = BUS_RESET_CHILD(device_get_parent(sc->dev), sc->dev, 0); > - mtx_unlock(&Giant); > - return (rc); > -} > - > static int > reset_adapter_with_pl_rst(struct adapter *sc) > { > - suspend_adapter(sc); > - > /* This is a t4_write_reg without the hw_off_limits check. */ > MPASS(sc->error_flags & HW_OFF_LIMITS); > bus_space_write_4(sc->bt, sc->bh, A_PL_RST, > F_PIORSTMODE | F_PIORST | F_AUTOPCIEPAUSE); > pause("pl_rst", 1 * hz); /* Wait 1s for reset */ > + return (0); > +} > > - resume_adapter(sc); > +static int > +reset_adapter_with_pcie_sbr(struct adapter *sc) > +{ > + device_t pdev = device_get_parent(sc->dev); > + device_t gpdev = device_get_parent(pdev); > + device_t *children; > + int rc, i, lcap, lsta, nchildren; > + uint32_t v; > > - return (0); > + rc = pci_find_cap(gpdev, PCIY_EXPRESS, &v); > + if (rc != 0) { > + CH_ERR(sc, "%s: pci_find_cap(%s, pcie) failed: %d\n", __func__, > + device_get_nameunit(gpdev), rc); > + return (ENOTSUP); > + } > + lcap = v + PCIER_LINK_CAP; > + lsta = v + PCIER_LINK_STA; > + > + nchildren = 0; > + device_get_children(pdev, &children, &nchildren); > + for (i = 0; i < nchildren; i++) > + pci_save_state(children[i]); > + v = pci_read_config(gpdev, PCIR_BRIDGECTL_1, 2); > + pci_write_config(gpdev, PCIR_BRIDGECTL_1, v | PCIB_BCR_SECBUS_RESET, 2); > + pause("pcie_sbr1", hz / 10); /* 100ms */ > + pci_write_config(gpdev, PCIR_BRIDGECTL_1, v, 2); > + pause("pcie_sbr2", hz); /* Wait 1s before restore_state. */ > + v = pci_read_config(gpdev, lsta, 2); > + if (pci_read_config(gpdev, lcap, 2) & PCIEM_LINK_CAP_DL_ACTIVE) > + rc = v & PCIEM_LINK_STA_DL_ACTIVE ? 0 : ETIMEDOUT; > + else if (v & (PCIEM_LINK_STA_TRAINING_ERROR | PCIEM_LINK_STA_TRAINING)) > + rc = ETIMEDOUT; > + else > + rc = 0; > + if (rc != 0) > + CH_ERR(sc, "%s: PCIe link is down after reset, LINK_STA 0x%x\n", > + __func__, v); > + else { > + for (i = 0; i < nchildren; i++) > + pci_restore_state(children[i]); > + } > + free(children, M_TEMP); > + > + return (rc); > +} > + > +static int > +reset_adapter_with_pcie_link_bounce(struct adapter *sc) We already have pcie_link_reset() that is used for BUS_RESET_CHILD() on pcib. See e.g. mlx5_core/mlx5_fwdump.c::mlx5_fw_reset() for its use. We needed exactly link bounce for that.