From nobody Tue Feb 1 15:24:09 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 455DB1914F6F; Tue, 1 Feb 2022 15:24:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jp7y26dqpz3FvJ; Tue, 1 Feb 2022 15:24:10 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=CB4on7o7M35QH8BBTxI7o8CcXmc70QRUXcqIaRnLhiA=; b=chfHOR6YNoyKO+f75ItMMU/JVvUnPHgrY36iSDc+m7H7+jwujWpaY0cnpv0wmF5DnKkskO VuHZ3SQMFcbameZzGTFP00a4828EKcY2bF39IpD2Xi6GZMBW2KFr3clk/mecWjWsgwYILc aYt5cRYqQZvrtqy3g1S/8llgH370JkFh9Ph64RtIg+8zCeSF7M15Xqw1bZVj3CGp5XQPzi 6wWQ0NLBbW5H4aAAowu4PULxrErJS6NREggSOuezvNuedM9bzoEjSdxtgrxP+ztm2BV1lY X8guiWl7mWyR/SvpRgt9AzHWRurxsg9M0YgVg9Pl9d1NAU/Ek4t2qRnuullJNA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 086CE13423; Tue, 1 Feb 2022 15:24:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 211FO9Ku050641; Tue, 1 Feb 2022 15:24:09 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 211FO9Fh050640; Tue, 1 Feb 2022 15:24:09 GMT (envelope-from git) Date: Tue, 1 Feb 2022 15:24:09 GMT Message-Id: <202202011524.211FO9Fh050640@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Hans Petter Selasky Subject: git: 27b778ae5537 - main - mlx5en: Implement dummy receive queue, RQ, for dropping packets. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 27b778ae55374a195ebb889f40200cd5986e9c15 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=CB4on7o7M35QH8BBTxI7o8CcXmc70QRUXcqIaRnLhiA=; b=Oj7JQaK4lo/ua3f0ODOvMw5Ba4yGkeMHRqrXEIHXqF+VMyg51a322uurXBiFRlIhc/ts6C MIiDxQ9vRfgfotptI0Jl8EKpghZLl3YuYADZstraPfrTEloQFLEl0Q6/bNM3J6nxwjLMRd 7iC9jBp+mF9WfRWnxP1FjMQV9Ilu5vWviP7bC4m9kGZ0PaAzVBhv6iey2kjnnm30PzI9yl nQJR2YKTo2ZOKETBFKCRTs15IrLZ/vrWQjSlhKEcjJ4Yv9x+KiYQsEHqskuq/tLluL8FIS tR9OG/DFy0CgfltakTyq03pJOuApRLSAzoDJ+xOY5msHtJiTw3uPp3DwQ4Aq3A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643729051; a=rsa-sha256; cv=none; b=EIDvftc+nSyQYPluhoZMelXiDOWw3BPdprlI/4q40knFY6aSMoK64JcUlRGEHEecGaXsFD MJM+IXMuQghHAFDnPgZ7GZeF0EGPbXe5IJAA5Ktct2wZKzugT+xRkp2+RmY8h+ZjLChFUR Z3m5l+RMTREwNT3ATOMfivbDpy8hWZsbp/ng/hiCkt/7AnNNaFMfN5f1q/VluPPHOVQ3YG qigH/I2SXmg7980vsfqkk8q/vXViO0IELzY6vGX4AH77dwXQJNmoz6tQttj2sFH2fgFzoD hJD/UL9FWytyFHASWD+LkCXAHB69OlzlWiuUWRVuY3nRzDMjv2NCRcNxXxq2LQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=27b778ae55374a195ebb889f40200cd5986e9c15 commit 27b778ae55374a195ebb889f40200cd5986e9c15 Author: Hans Petter Selasky AuthorDate: 2022-02-01 15:20:06 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-01 15:21:14 +0000 mlx5en: Implement dummy receive queue, RQ, for dropping packets. What is a drop RQ and why is it needed? The RSS indirection table, also called the RQT, selects the destination RQ based on the receive queue number, RQN. The RQT is frequently referred to by flow steering rules to distribute traffic among multiple RQs. The problem is that the RQs cannot be destroyed before the RQT referring them is destroyed too. Further, TLS RX rules may still be referring to the RQT even if the link went down. Because there is no magic RQN for dropping packets, we create a dummy RQ, also called drop RQ, which sole purpose is to drop all received packets. When the link goes down this RQN is filled in all RQT entries, of the main RQT, so the real RQs which are about to be destroyed can be released and the TLS RX rules can be sustained. MFC after: 1 week Sponsored by: NVIDIA Networking --- sys/dev/mlx5/mlx5_en/en.h | 1 + sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 92 +++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index 4eb771d5c5c6..293d7cb4d05d 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -1018,6 +1018,7 @@ struct mlx5e_priv { #define PRIV_LOCKED(priv) sx_xlocked(&(priv)->state_lock) #define PRIV_ASSERT_LOCKED(priv) sx_assert(&(priv)->state_lock, SA_XLOCKED) struct sx state_lock; /* Protects Interface state */ + struct mlx5e_rq drop_rq; u32 pdn; u32 tdn; struct mlx5_core_mkey mr; diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index 4477d5fd53dc..c785335e243c 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -1479,6 +1479,88 @@ mlx5e_close_rq_wait(struct mlx5e_rq *rq) mlx5e_destroy_rq(rq); } +/* + * What is a drop RQ and why is it needed? + * + * The RSS indirection table, also called the RQT, selects the + * destination RQ based on the receive queue number, RQN. The RQT is + * frequently referred to by flow steering rules to distribute traffic + * among multiple RQs. The problem is that the RQs cannot be destroyed + * before the RQT referring them is destroyed too. Further, TLS RX + * rules may still be referring to the RQT even if the link went + * down. Because there is no magic RQN for dropping packets, we create + * a dummy RQ, also called drop RQ, which sole purpose is to drop all + * received packets. When the link goes down this RQN is filled in all + * RQT entries, of the main RQT, so the real RQs which are about to be + * destroyed can be released and the TLS RX rules can be sustained. + */ +static void +mlx5e_open_drop_rq_comp(struct mlx5_core_cq *mcq __unused, struct mlx5_eqe *eqe __unused) +{ +} + +static int +mlx5e_open_drop_rq(struct mlx5e_priv *priv, + struct mlx5e_rq *drop_rq) +{ + struct mlx5e_cq_param param_cq = {}; + struct mlx5e_rq_param param_rq = {}; + void *rqc_wq = MLX5_ADDR_OF(rqc, param_rq.rqc, wq); + int err; + + /* set basic CQ parameters needed */ + MLX5_SET(cqc, param_cq.cqc, log_cq_size, 0); + MLX5_SET(cqc, param_cq.cqc, uar_page, priv->mdev->priv.uar->index); + + /* open receive completion queue */ + err = mlx5e_open_cq(priv, ¶m_cq, &drop_rq->cq, + &mlx5e_open_drop_rq_comp, 0); + if (err) + goto err_done; + + /* set basic WQ parameters needed */ + MLX5_SET(wq, rqc_wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST); + MLX5_SET(wq, rqc_wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN); + MLX5_SET(wq, rqc_wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe) + sizeof(struct mlx5_wqe_data_seg))); + MLX5_SET(wq, rqc_wq, log_wq_sz, 0); + MLX5_SET(wq, rqc_wq, pd, priv->pdn); + + param_rq.wq.linear = 1; + + err = mlx5_wq_ll_create(priv->mdev, ¶m_rq.wq, rqc_wq, &drop_rq->wq, + &drop_rq->wq_ctrl); + if (err) + goto err_close_cq; + + err = mlx5e_enable_rq(drop_rq, ¶m_rq); + if (err) + goto err_wq_destroy; + + err = mlx5e_modify_rq(drop_rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY); + if (err) + goto err_disable_rq; + + return (err); + +err_disable_rq: + mlx5e_disable_rq(drop_rq); +err_wq_destroy: + mlx5_wq_destroy(&drop_rq->wq_ctrl); +err_close_cq: + mlx5e_close_cq(&drop_rq->cq); +err_done: + return (err); +} + +static void +mlx5e_close_drop_rq(struct mlx5e_rq *drop_rq) +{ + mlx5e_modify_rq(drop_rq, MLX5_RQC_STATE_RDY, MLX5_RQC_STATE_ERR); + mlx5e_disable_rq(drop_rq); + mlx5_wq_destroy(&drop_rq->wq_ctrl); + mlx5e_close_cq(&drop_rq->cq); +} + void mlx5e_free_sq_db(struct mlx5e_sq *sq) { @@ -4450,6 +4532,12 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) goto err_rl_init; } + err = mlx5e_open_drop_rq(priv, &priv->drop_rq); + if (err) { + if_printf(ifp, "%s: mlx5e_open_drop_rq failed\n", __func__); + goto err_tls_init; + } + /* set default MTU */ mlx5e_set_dev_port_mtu(ifp, ifp->if_mtu); @@ -4577,6 +4665,9 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) return (priv); +err_tls_init: + mlx5e_tls_cleanup(priv); + err_rl_init: mlx5e_rl_cleanup(priv); @@ -4672,6 +4763,7 @@ mlx5e_destroy_ifp(struct mlx5_core_dev *mdev, void *vpriv) ifmedia_removeall(&priv->media); ether_ifdetach(ifp); + mlx5e_close_drop_rq(&priv->drop_rq); mlx5e_tls_cleanup(priv); mlx5e_rl_cleanup(priv);