From nobody Wed Apr 19 16:51:53 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q1mzG0GHMz46BX0; Wed, 19 Apr 2023 16:51:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q1mzF42QKz3P2t; Wed, 19 Apr 2023 16:51:53 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681923113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5p6Up1Yr37X8ywqpLlbNVCVtSIjtZWhXD/g9VDM689o=; b=kF1ayd9/qthOmOqFu1JAqYrpgLQbukNWYK3K3c3I7EP4iXarmA37R2yBjtFT5Hit7q1NqK XVDJwaj9C9hpedTX3tQ7Yb9we/mK416Nobdacb8PMDV8WNjU0bNiJW5OpWxhxeqJAziKNQ LEo+3fAEW1DlDo92lwcAzwxMKnQSyYT0BMkm3J1+WB0bzXMZ6uJSFpBMG1fgzSCN/L3NXD ptGVk4zhQEUXaBB7EAZ8X4RpcMjWGQJ60r+1uxs3tNWCCw2CDUWRsHX4T9lSTLVu5DBtqE TBseY75BnCL6lCiGHcaxolOvBAAvecTHTHgs7aBdMEfboFLuK7/X54Qg7VOtXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681923113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5p6Up1Yr37X8ywqpLlbNVCVtSIjtZWhXD/g9VDM689o=; b=ClL5cSE7M7jVIWazalGwqD7RU8nndyBMmEbdQJ8NHPEVAUyr2b4f1hjFdyKm+1FxD6PVIy da5KOG6khPLwFZdsSR/acJyxWCMacqFgBLQK8r7pJDZVSMmY5gzYIVZskW9Ov9XavlqT2B mV+II6KfjiNx/xbXxsXIBEzVnZCJQzFfo3P7PVU+AdT4+X5vpZBIdlnKQCV6ksAm3E23NE ixxbl/lk4XXAFnbKlcU5ozu6EQ5i9o6zJJE5H/ul0NuDVo9IfpYDCE3a4qc58VmhzqEk82 Va28+EwDba1y8QN8PHvCEXtfWAnZNeVZe9R6Ptn/0aV2WgRfzrRNg2GIP3zBmQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1681923113; a=rsa-sha256; cv=none; b=PepWuWIqZIT0Xkh0d4IdNvSRB4I5oCBdPNZJu0glvu2mtFCNwCYMhc58GpeVozzpyr3Su3 KkHkqy/g7IFxAXClEupunAO6t9ajzorw1eHDzUCrrk66WmpujUGfWuNa8S8SuKSFB8vDhJ 5PC1xNLjZ/OODLJVSeG6cN9cmLauqPbdiDD1I7f4Xy++ypX+fJdGAqMLKtiKfc0mGSMB07 y9QM0dORceIFokmMsQPiF1n50smEyl3QZ5Vmh4rKfFIlLuU9VxgwlTmkyj5iMF9JvEbQma INPN/3nhRNvy6aY98MBZ6TdYmb+zNZbVSkZ3ALmHp+iZVeS5nEOgH8ZrpIUfOg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q1mzF37mNz12GC; Wed, 19 Apr 2023 16:51:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 33JGprL4035164; Wed, 19 Apr 2023 16:51:53 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 33JGpr7O035163; Wed, 19 Apr 2023 16:51:53 GMT (envelope-from git) Date: Wed, 19 Apr 2023 16:51:53 GMT Message-Id: <202304191651.33JGpr7O035163@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Navdeep Parhar Subject: git: 7adf138ba9b9 - main - cxgbe/iw_cxgbe: debug routines to dump STAG (steering tag) entries. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: np X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7adf138ba9b90db5ed0f1e8cd68e787fb63fa760 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by np: URL: https://cgit.FreeBSD.org/src/commit/?id=7adf138ba9b90db5ed0f1e8cd68e787fb63fa760 commit 7adf138ba9b90db5ed0f1e8cd68e787fb63fa760 Author: Navdeep Parhar AuthorDate: 2023-04-06 17:52:15 +0000 Commit: Navdeep Parhar CommitDate: 2023-04-19 16:38:14 +0000 cxgbe/iw_cxgbe: debug routines to dump STAG (steering tag) entries. t4_dump_stag to dump hw state for a known STAG. t4_dump_all_stag to dump hw state for all valid STAGs. This routine walks the entire STAG region looking for valid entries and this can take a while for some configurations. MFC after: 1 week Sponsored by: Chelsio Communications --- sys/dev/cxgbe/iw_cxgbe/ev.c | 82 ++++++++++++++++++++++++++++++++++++--- sys/dev/cxgbe/iw_cxgbe/iw_cxgbe.h | 2 + 2 files changed, 79 insertions(+), 5 deletions(-) diff --git a/sys/dev/cxgbe/iw_cxgbe/ev.c b/sys/dev/cxgbe/iw_cxgbe/ev.c index 7c42354644e0..a1475554c08f 100644 --- a/sys/dev/cxgbe/iw_cxgbe/ev.c +++ b/sys/dev/cxgbe/iw_cxgbe/ev.c @@ -41,6 +41,82 @@ __FBSDID("$FreeBSD$"); #include "iw_cxgbe.h" +static void print_tpte(struct adapter *sc, const u32 stag, + const struct fw_ri_tpte *tpte) +{ + const __be64 *p = (const void *)tpte; + + CH_ERR(sc, "stag idx 0x%x valid %d key 0x%x state %d pdid %d " + "perm 0x%x ps %d len 0x%016llx va 0x%016llx\n", + stag & 0xffffff00, + G_FW_RI_TPTE_VALID(ntohl(tpte->valid_to_pdid)), + G_FW_RI_TPTE_STAGKEY(ntohl(tpte->valid_to_pdid)), + G_FW_RI_TPTE_STAGSTATE(ntohl(tpte->valid_to_pdid)), + G_FW_RI_TPTE_PDID(ntohl(tpte->valid_to_pdid)), + G_FW_RI_TPTE_PERM(ntohl(tpte->locread_to_qpid)), + G_FW_RI_TPTE_PS(ntohl(tpte->locread_to_qpid)), + (long long)(((u64)ntohl(tpte->len_hi) << 32) | ntohl(tpte->len_lo)), + (long long)(((u64)ntohl(tpte->va_hi) << 32) | ntohl(tpte->va_lo_fbo))); + CH_ERR(sc, "stag idx 0x%x: %016llx %016llx %016llx %016llx\n", + stag & 0xffffff00, + (long long)be64_to_cpu(p[0]), (long long)be64_to_cpu(p[1]), + (long long)be64_to_cpu(p[2]), (long long)be64_to_cpu(p[3])); +} + +void t4_dump_stag(struct adapter *sc, const u32 stag) +{ + struct fw_ri_tpte tpte __aligned(sizeof(__be64)) = {0}; + const u32 offset = sc->vres.stag.start + ((stag >> 8) * 32); + + if (offset > sc->vres.stag.start + sc->vres.stag.size - 32) { + CH_ERR(sc, "stag 0x%x is invalid for current configuration.\n", + stag); + return; + } + read_via_memwin(sc, 0, offset, (u32 *)&tpte, 32); + print_tpte(sc, stag, &tpte); +} + +void t4_dump_all_stag(struct adapter *sc) +{ + struct fw_ri_tpte tpte __aligned(sizeof(__be64)) = {0}; + const u32 first = sc->vres.stag.start; + const u32 last = first + sc->vres.stag.size - 32; + u32 offset, i; + + for (i = 0, offset = first; offset <= last; i++, offset += 32) { + tpte.valid_to_pdid = 0; + read_via_memwin(sc, 0, offset, (u32 *)&tpte, 4); + if (tpte.valid_to_pdid != 0) { + read_via_memwin(sc, 0, offset, (u32 *)&tpte, 32); + print_tpte(sc, i << 8, &tpte); + } + } +} + +static void dump_err_cqe(struct c4iw_dev *dev, struct t4_cqe *err_cqe) +{ + struct adapter *sc = dev->rdev.adap; + __be64 *p = (void *)err_cqe; + + CH_ERR(sc, "AE qpid 0x%x opcode %d status 0x%x " + "type %d wrid.hi 0x%x wrid.lo 0x%x\n", + CQE_QPID(err_cqe), CQE_OPCODE(err_cqe), + CQE_STATUS(err_cqe), CQE_TYPE(err_cqe), + CQE_WRID_HI(err_cqe), CQE_WRID_LOW(err_cqe)); + CH_ERR(sc, "%016llx %016llx %016llx %016llx\n", + (long long)be64_to_cpu(p[0]), (long long)be64_to_cpu(p[1]), + (long long)be64_to_cpu(p[2]), (long long)be64_to_cpu(p[3])); + + /* + * Ingress WRITE and READ_RESP errors provide + * the offending stag, so parse and log it. + */ + if (RQ_TYPE(err_cqe) && (CQE_OPCODE(err_cqe) == FW_RI_RDMA_WRITE || + CQE_OPCODE(err_cqe) == FW_RI_READ_RESP)) + t4_dump_stag(sc, CQE_WRID_STAG(err_cqe)); +} + static void post_qp_event(struct c4iw_dev *dev, struct c4iw_cq *chp, struct c4iw_qp *qhp, struct t4_cqe *err_cqe, @@ -58,11 +134,7 @@ static void post_qp_event(struct c4iw_dev *dev, struct c4iw_cq *chp, return; } - printf("AE qpid 0x%x opcode %d status 0x%x " - "type %d wrid.hi 0x%x wrid.lo 0x%x\n", - CQE_QPID(err_cqe), CQE_OPCODE(err_cqe), - CQE_STATUS(err_cqe), CQE_TYPE(err_cqe), - CQE_WRID_HI(err_cqe), CQE_WRID_LOW(err_cqe)); + dump_err_cqe(dev, err_cqe); if (qhp->attr.state == C4IW_QP_STATE_RTS) { attrs.next_state = C4IW_QP_STATE_TERMINATE; diff --git a/sys/dev/cxgbe/iw_cxgbe/iw_cxgbe.h b/sys/dev/cxgbe/iw_cxgbe/iw_cxgbe.h index 8e33e2d04597..149393a71c25 100644 --- a/sys/dev/cxgbe/iw_cxgbe/iw_cxgbe.h +++ b/sys/dev/cxgbe/iw_cxgbe/iw_cxgbe.h @@ -982,4 +982,6 @@ u32 c4iw_get_qpid(struct c4iw_rdev *rdev, struct c4iw_dev_ucontext *uctx); void c4iw_put_qpid(struct c4iw_rdev *rdev, u32 qid, struct c4iw_dev_ucontext *uctx); void c4iw_ev_dispatch(struct c4iw_dev *dev, struct t4_cqe *err_cqe); +void t4_dump_stag(struct adapter *sc, const u32 stag); +void t4_dump_all_stag(struct adapter *sc); #endif