From owner-svn-src-all@freebsd.org Thu Aug 2 08:55:22 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0DF8105FC68; Thu, 2 Aug 2018 08:55:21 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4D157E563; Thu, 2 Aug 2018 08:55:21 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 817E27784; Thu, 2 Aug 2018 08:55:21 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w728tLFc062110; Thu, 2 Aug 2018 08:55:21 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w728tKZp062103; Thu, 2 Aug 2018 08:55:20 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201808020855.w728tKZp062103@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Thu, 2 Aug 2018 08:55:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r337114 - in stable/11/sys/dev/mlx5: . mlx5_core mlx5_en X-SVN-Group: stable-11 X-SVN-Commit-Author: hselasky X-SVN-Commit-Paths: in stable/11/sys/dev/mlx5: . mlx5_core mlx5_en X-SVN-Commit-Revision: 337114 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 08:55:22 -0000 Author: hselasky Date: Thu Aug 2 08:55:19 2018 New Revision: 337114 URL: https://svnweb.freebsd.org/changeset/base/337114 Log: MFC r336450: Do not inline transmit headers and use HW VLAN tagging if supported by mlx5en(4). Query the minimal inline mode supported by the card. When creating a send queue, cache the queried mode and optimize the transmit if no inlining is required. In this case, we can avoid touching the headers cache line and avoid dirtying several more lines by copying headers into the send WQEs. Also, if no inline headers are used, hardware assists in the VLAN tag framing. Submitted by: kib@, slavash@ Sponsored by: Mellanox Technologies Modified: stable/11/sys/dev/mlx5/device.h stable/11/sys/dev/mlx5/mlx5_core/mlx5_vport.c stable/11/sys/dev/mlx5/mlx5_en/en.h stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_main.c stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c stable/11/sys/dev/mlx5/qp.h stable/11/sys/dev/mlx5/vport.h Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/dev/mlx5/device.h ============================================================================== --- stable/11/sys/dev/mlx5/device.h Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/device.h Thu Aug 2 08:55:19 2018 (r337114) @@ -1091,6 +1091,13 @@ enum { MLX5_CMD_HCA_CAP_MIN_WQE_INLINE_MODE_NOT_REQUIRED = 0x2 }; +enum mlx5_inline_modes { + MLX5_INLINE_MODE_NONE, + MLX5_INLINE_MODE_L2, + MLX5_INLINE_MODE_IP, + MLX5_INLINE_MODE_TCP_UDP, +}; + enum { MLX5_QUERY_VPORT_STATE_OUT_STATE_FOLLOW = 0x2, }; Modified: stable/11/sys/dev/mlx5/mlx5_core/mlx5_vport.c ============================================================================== --- stable/11/sys/dev/mlx5/mlx5_core/mlx5_vport.c Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/mlx5_core/mlx5_vport.c Thu Aug 2 08:55:19 2018 (r337114) @@ -208,6 +208,58 @@ int mlx5_vport_query_out_of_rx_buffer(struct mlx5_core return err; } +int mlx5_query_nic_vport_min_inline(struct mlx5_core_dev *mdev, + u16 vport, u8 *min_inline) +{ + u32 out[MLX5_ST_SZ_DW(query_nic_vport_context_out)] = {0}; + int err; + + err = mlx5_query_nic_vport_context(mdev, vport, out, sizeof(out)); + if (!err) + *min_inline = MLX5_GET(query_nic_vport_context_out, out, + nic_vport_context.min_wqe_inline_mode); + return err; +} +EXPORT_SYMBOL_GPL(mlx5_query_nic_vport_min_inline); + +void mlx5_query_min_inline(struct mlx5_core_dev *mdev, + u8 *min_inline_mode) +{ + switch (MLX5_CAP_ETH(mdev, wqe_inline_mode)) { + case MLX5_CAP_INLINE_MODE_L2: + *min_inline_mode = MLX5_INLINE_MODE_L2; + break; + case MLX5_CAP_INLINE_MODE_VPORT_CONTEXT: + mlx5_query_nic_vport_min_inline(mdev, 0, min_inline_mode); + break; + case MLX5_CAP_INLINE_MODE_NOT_REQUIRED: + *min_inline_mode = MLX5_INLINE_MODE_NONE; + break; + } +} +EXPORT_SYMBOL_GPL(mlx5_query_min_inline); + +int mlx5_modify_nic_vport_min_inline(struct mlx5_core_dev *mdev, + u16 vport, u8 min_inline) +{ + u32 in[MLX5_ST_SZ_DW(modify_nic_vport_context_in)] = {0}; + int inlen = MLX5_ST_SZ_BYTES(modify_nic_vport_context_in); + void *nic_vport_ctx; + + MLX5_SET(modify_nic_vport_context_in, in, + field_select.min_wqe_inline_mode, 1); + MLX5_SET(modify_nic_vport_context_in, in, vport_number, vport); + MLX5_SET(modify_nic_vport_context_in, in, other_vport, 1); + + nic_vport_ctx = MLX5_ADDR_OF(modify_nic_vport_context_in, + in, nic_vport_context); + MLX5_SET(nic_vport_context, nic_vport_ctx, + min_wqe_inline_mode, min_inline); + + return mlx5_modify_nic_vport_context(mdev, in, inlen); +} +EXPORT_SYMBOL_GPL(mlx5_modify_nic_vport_min_inline); + int mlx5_query_nic_vport_mac_address(struct mlx5_core_dev *mdev, u16 vport, u8 *addr) { Modified: stable/11/sys/dev/mlx5/mlx5_en/en.h ============================================================================== --- stable/11/sys/dev/mlx5/mlx5_en/en.h Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/mlx5_en/en.h Thu Aug 2 08:55:19 2018 (r337114) @@ -450,6 +450,8 @@ struct mlx5e_params { u32 rx_pauseframe_control __aligned(4); u32 tx_priority_flow_control __aligned(4); u32 rx_priority_flow_control __aligned(4); + u16 tx_max_inline; + u8 tx_min_inline_mode; }; #define MLX5E_PARAMS(m) \ @@ -612,6 +614,9 @@ struct mlx5e_sq { u32 sqn; u32 bf_buf_size; u32 mkey_be; + u16 max_inline; + u8 min_inline_mode; + u8 vlan_inline_cap; /* control path */ struct mlx5_wq_ctrl wq_ctrl; @@ -909,5 +914,6 @@ void mlx5e_drain_sq(struct mlx5e_sq *); void mlx5e_modify_tx_dma(struct mlx5e_priv *priv, uint8_t value); void mlx5e_modify_rx_dma(struct mlx5e_priv *priv, uint8_t value); void mlx5e_resume_sq(struct mlx5e_sq *sq); +u8 mlx5e_params_calculate_tx_min_inline(struct mlx5_core_dev *mdev); #endif /* _MLX5_EN_H_ */ Modified: stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_main.c ============================================================================== --- stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_main.c Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_main.c Thu Aug 2 08:55:19 2018 (r337114) @@ -1069,6 +1069,9 @@ mlx5e_create_sq(struct mlx5e_channel *c, sq->ifp = priv->ifp; sq->priv = priv; sq->tc = tc; + sq->max_inline = priv->params.tx_max_inline; + sq->min_inline_mode = priv->params.tx_min_inline_mode; + sq->vlan_inline_cap = MLX5_CAP_ETH(mdev, wqe_vlan_insert); /* check if we should allocate a second packet buffer */ if (priv->params_ethtool.tx_bufring_disable == 0) { @@ -2906,6 +2909,16 @@ mlx5e_check_required_hca_cap(struct mlx5_core_dev *mde return (0); } +static u16 +mlx5e_get_max_inline_cap(struct mlx5_core_dev *mdev) +{ + int bf_buf_size = (1 << MLX5_CAP_GEN(mdev, log_bf_reg_size)) / 2; + + return bf_buf_size - + sizeof(struct mlx5e_tx_wqe) + + 2 /*sizeof(mlx5e_tx_wqe.inline_hdr_start)*/; +} + static void mlx5e_build_ifp_priv(struct mlx5_core_dev *mdev, struct mlx5e_priv *priv, @@ -2941,6 +2954,8 @@ mlx5e_build_ifp_priv(struct mlx5_core_dev *mdev, priv->params.num_tc = 1; priv->params.default_vlan_prio = 0; priv->counter_set_id = -1; + priv->params.tx_max_inline = mlx5e_get_max_inline_cap(mdev); + mlx5_query_min_inline(mdev, &priv->params.tx_min_inline_mode); /* * hw lro is currently defaulted to off. when it won't anymore we @@ -3199,6 +3214,20 @@ mlx5e_modify_rx_dma(struct mlx5e_priv *priv, uint8_t v else mlx5e_enable_rx_dma(priv->channel[i]); } +} + +u8 +mlx5e_params_calculate_tx_min_inline(struct mlx5_core_dev *mdev) +{ + u8 min_inline_mode; + + min_inline_mode = MLX5_INLINE_MODE_L2; + mlx5_query_min_inline(mdev, &min_inline_mode); + if (min_inline_mode == MLX5_INLINE_MODE_NONE && + !MLX5_CAP_ETH(mdev, wqe_vlan_insert)) + min_inline_mode = MLX5_INLINE_MODE_L2; + + return (min_inline_mode); } static void Modified: stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c ============================================================================== --- stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c Thu Aug 2 08:55:19 2018 (r337114) @@ -137,7 +137,43 @@ mlx5e_select_queue(struct ifnet *ifp, struct mbuf *mb) static inline u16 mlx5e_get_inline_hdr_size(struct mlx5e_sq *sq, struct mbuf *mb) { - return (MIN(MLX5E_MAX_TX_INLINE, mb->m_len)); + + switch(sq->min_inline_mode) { + case MLX5_INLINE_MODE_NONE: + /* + * When inline mode is NONE, we do not need to copy + * headers into WQEs, except when vlan tag framing is + * requested. Hardware might offload vlan tagging on + * transmit. This is a separate capability, which is + * known to be disabled on ConnectX-5 due to a hardware + * bug RM 931383. If vlan_inline_cap is not present and + * the packet has vlan tag, fall back to inlining. + */ + if ((mb->m_flags & M_VLANTAG) != 0 && + sq->vlan_inline_cap == 0) + break; + return (0); + case MLX5_INLINE_MODE_L2: + /* + * Due to hardware limitations, when trust mode is + * DSCP, the hardware may request MLX5_INLINE_MODE_L2 + * while it really needs all L2 headers and the 4 first + * bytes of the IP header (which include the + * TOS/traffic-class). + * + * To avoid doing a firmware command for querying the + * trust state and parsing the mbuf for doing + * unnecessary checks (VLAN/eth_type) in the fast path, + * we are going for the worth case (22 Bytes) if + * the mb->m_pkthdr.len allows it. + */ + if (mb->m_pkthdr.len > ETHER_HDR_LEN + + ETHER_VLAN_ENCAP_LEN + 4) + return (MIN(sq->max_inline, ETHER_HDR_LEN + + ETHER_VLAN_ENCAP_LEN + 4)); + break; + } + return (MIN(sq->max_inline, mb->m_pkthdr.len)); } static int @@ -275,37 +311,47 @@ mlx5e_sq_xmit(struct mlx5e_sq *sq, struct mbuf **mbp) sq->mbuf[pi].num_bytes = max_t (unsigned int, mb->m_pkthdr.len, ETHER_MIN_LEN - ETHER_CRC_LEN); } - if (mb->m_flags & M_VLANTAG) { - struct ether_vlan_header *eh = - (struct ether_vlan_header *)wqe->eth.inline_hdr_start; - - /* Range checks */ - if (ihs > (MLX5E_MAX_TX_INLINE - ETHER_VLAN_ENCAP_LEN)) - ihs = (MLX5E_MAX_TX_INLINE - ETHER_VLAN_ENCAP_LEN); - else if (ihs < ETHER_HDR_LEN) { - err = EINVAL; - goto tx_drop; + if (ihs == 0) { + if ((mb->m_flags & M_VLANTAG) != 0) { + wqe->eth.vlan_cmd = htons(0x8000); /* bit 0 CVLAN */ + wqe->eth.vlan_hdr = htons(mb->m_pkthdr.ether_vtag); + } else { + wqe->eth.inline_hdr_sz = 0; } - m_copydata(mb, 0, ETHER_HDR_LEN, (caddr_t)eh); - m_adj(mb, ETHER_HDR_LEN); - /* Insert 4 bytes VLAN tag into data stream */ - eh->evl_proto = eh->evl_encap_proto; - eh->evl_encap_proto = htons(ETHERTYPE_VLAN); - eh->evl_tag = htons(mb->m_pkthdr.ether_vtag); - /* Copy rest of header data, if any */ - m_copydata(mb, 0, ihs - ETHER_HDR_LEN, (caddr_t)(eh + 1)); - m_adj(mb, ihs - ETHER_HDR_LEN); - /* Extend header by 4 bytes */ - ihs += ETHER_VLAN_ENCAP_LEN; } else { - m_copydata(mb, 0, ihs, wqe->eth.inline_hdr_start); - m_adj(mb, ihs); + if ((mb->m_flags & M_VLANTAG) != 0) { + struct ether_vlan_header *eh = (struct ether_vlan_header + *)wqe->eth.inline_hdr_start; + + /* Range checks */ + if (ihs > (MLX5E_MAX_TX_INLINE - ETHER_VLAN_ENCAP_LEN)) + ihs = (MLX5E_MAX_TX_INLINE - + ETHER_VLAN_ENCAP_LEN); + else if (ihs < ETHER_HDR_LEN) { + err = EINVAL; + goto tx_drop; + } + m_copydata(mb, 0, ETHER_HDR_LEN, (caddr_t)eh); + m_adj(mb, ETHER_HDR_LEN); + /* Insert 4 bytes VLAN tag into data stream */ + eh->evl_proto = eh->evl_encap_proto; + eh->evl_encap_proto = htons(ETHERTYPE_VLAN); + eh->evl_tag = htons(mb->m_pkthdr.ether_vtag); + /* Copy rest of header data, if any */ + m_copydata(mb, 0, ihs - ETHER_HDR_LEN, (caddr_t)(eh + + 1)); + m_adj(mb, ihs - ETHER_HDR_LEN); + /* Extend header by 4 bytes */ + ihs += ETHER_VLAN_ENCAP_LEN; + } else { + m_copydata(mb, 0, ihs, wqe->eth.inline_hdr_start); + m_adj(mb, ihs); + } + wqe->eth.inline_hdr_sz = cpu_to_be16(ihs); } - wqe->eth.inline_hdr_sz = cpu_to_be16(ihs); - ds_cnt = sizeof(*wqe) / MLX5_SEND_WQE_DS; - if (likely(ihs > sizeof(wqe->eth.inline_hdr_start))) { + if (ihs > sizeof(wqe->eth.inline_hdr_start)) { ds_cnt += DIV_ROUND_UP(ihs - sizeof(wqe->eth.inline_hdr_start), MLX5_SEND_WQE_DS); } Modified: stable/11/sys/dev/mlx5/qp.h ============================================================================== --- stable/11/sys/dev/mlx5/qp.h Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/qp.h Thu Aug 2 08:55:19 2018 (r337114) @@ -240,8 +240,16 @@ struct mlx5_wqe_eth_seg { u8 swp_flags; __be16 mss; __be32 rsvd2; - __be16 inline_hdr_sz; - u8 inline_hdr_start[2]; + union { + struct { + __be16 inline_hdr_sz; + u8 inline_hdr_start[2]; + }; + struct { + __be16 vlan_cmd; + __be16 vlan_hdr; + }; + }; }; struct mlx5_wqe_xrc_seg { Modified: stable/11/sys/dev/mlx5/vport.h ============================================================================== --- stable/11/sys/dev/mlx5/vport.h Thu Aug 2 08:52:49 2018 (r337113) +++ stable/11/sys/dev/mlx5/vport.h Thu Aug 2 08:55:19 2018 (r337114) @@ -29,6 +29,13 @@ #define __MLX5_VPORT_H__ #include + +enum { + MLX5_CAP_INLINE_MODE_L2, + MLX5_CAP_INLINE_MODE_VPORT_CONTEXT, + MLX5_CAP_INLINE_MODE_NOT_REQUIRED, +}; + int mlx5_vport_alloc_q_counter(struct mlx5_core_dev *mdev, int client_id, u16 *counter_set_id); int mlx5_vport_dealloc_q_counter(struct mlx5_core_dev *mdev, int client_id, @@ -79,6 +86,11 @@ int mlx5_modify_nic_vport_mac_address(struct mlx5_core u16 vport, u8 mac[ETH_ALEN]); int mlx5_set_nic_vport_current_mac(struct mlx5_core_dev *mdev, int vport, bool other_vport, u8 *addr); +int mlx5_query_nic_vport_min_inline(struct mlx5_core_dev *mdev, + u16 vport, u8 *min_inline); +void mlx5_query_min_inline(struct mlx5_core_dev *mdev, u8 *min_inline); +int mlx5_modify_nic_vport_min_inline(struct mlx5_core_dev *mdev, + u16 vport, u8 min_inline); int mlx5_modify_nic_vport_port_guid(struct mlx5_core_dev *mdev, u32 vport, u64 port_guid); int mlx5_modify_nic_vport_node_guid(struct mlx5_core_dev *mdev,