From owner-svn-src-head@freebsd.org Tue Jul 17 09:11:55 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 162F31032075; Tue, 17 Jul 2018 09:11:55 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B881C7CA53; Tue, 17 Jul 2018 09:11:54 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9AFD614E4; Tue, 17 Jul 2018 09:11:54 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6H9BscF010824; Tue, 17 Jul 2018 09:11:54 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6H9BreP010820; Tue, 17 Jul 2018 09:11:53 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201807170911.w6H9BreP010820@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Tue, 17 Jul 2018 09:11:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r336372 - in head/sys: dev/mlx4/mlx4_ib dev/mlx5/mlx5_ib ofed/drivers/infiniband/core ofed/include/rdma X-SVN-Group: head X-SVN-Commit-Author: hselasky X-SVN-Commit-Paths: in head/sys: dev/mlx4/mlx4_ib dev/mlx5/mlx5_ib ofed/drivers/infiniband/core ofed/include/rdma X-SVN-Commit-Revision: 336372 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jul 2018 09:11:55 -0000 Author: hselasky Date: Tue Jul 17 09:11:53 2018 New Revision: 336372 URL: https://svnweb.freebsd.org/changeset/base/336372 Log: Add support for prio-tagged traffic for RDMA in ibcore. When receiving a PCP change all GID entries are reloaded. This ensures the relevant GID entries use prio tagging, by setting VLAN present and VLAN ID to zero. The priority for prio tagged traffic is set using the regular rdma_set_service_type() function. Fake the real network device to have a VLAN ID of zero when prio tagging is enabled. This is logic is hidden inside the rdma_vlan_dev_vlan_id() function which must always be used to retrieve the VLAN ID throughout all of ibcore and the infiniband network drivers. The VLAN presence information then propagates through all of ibcore and so incoming connections will have the VLAN bit set. The incoming VLAN ID is then checked against the return value of rdma_vlan_dev_vlan_id(). MFC after: 1 week Sponsored by: Mellanox Technologies Modified: head/sys/dev/mlx4/mlx4_ib/mlx4_ib_ah.c head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c head/sys/ofed/drivers/infiniband/core/ib_roce_gid_mgmt.c head/sys/ofed/drivers/infiniband/core/ib_verbs.c head/sys/ofed/include/rdma/ib_addr.h Modified: head/sys/dev/mlx4/mlx4_ib/mlx4_ib_ah.c ============================================================================== --- head/sys/dev/mlx4/mlx4_ib/mlx4_ib_ah.c Tue Jul 17 09:09:17 2018 (r336371) +++ head/sys/dev/mlx4/mlx4_ib/mlx4_ib_ah.c Tue Jul 17 09:11:53 2018 (r336372) @@ -96,8 +96,7 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, return ERR_PTR(ret); eth_zero_addr(ah->av.eth.s_mac); if (gid_attr.ndev) { - if (is_vlan_dev(gid_attr.ndev)) - vlan_tag = vlan_dev_vlan_id(gid_attr.ndev); + vlan_tag = rdma_vlan_dev_vlan_id(gid_attr.ndev); memcpy(ah->av.eth.s_mac, IF_LLADDR(gid_attr.ndev), ETH_ALEN); dev_put(gid_attr.ndev); } Modified: head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c ============================================================================== --- head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c Tue Jul 17 09:09:17 2018 (r336371) +++ head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c Tue Jul 17 09:11:53 2018 (r336372) @@ -285,14 +285,16 @@ static void ib_gid_to_mlx5_roce_addr(const union ib_gi source_l3_address); void *mlx5_addr_mac = MLX5_ADDR_OF(roce_addr_layout, mlx5_addr, source_mac_47_32); + u16 vlan_id; if (!gid) return; ether_addr_copy(mlx5_addr_mac, IF_LLADDR(attr->ndev)); - if (is_vlan_dev(attr->ndev)) { + vlan_id = rdma_vlan_dev_vlan_id(attr->ndev); + if (vlan_id != 0xffff) { MLX5_SET_RA(mlx5_addr, vlan_valid, 1); - MLX5_SET_RA(mlx5_addr, vlan_id, vlan_dev_vlan_id(attr->ndev)); + MLX5_SET_RA(mlx5_addr, vlan_id, vlan_id); } switch (attr->gid_type) { Modified: head/sys/ofed/drivers/infiniband/core/ib_roce_gid_mgmt.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_roce_gid_mgmt.c Tue Jul 17 09:09:17 2018 (r336371) +++ head/sys/ofed/drivers/infiniband/core/ib_roce_gid_mgmt.c Tue Jul 17 09:11:53 2018 (r336372) @@ -402,7 +402,20 @@ static struct notifier_block nb_inetaddr = { .notifier_call = inetaddr_event }; +static eventhandler_tag eh_ifnet_event; + static void +roce_ifnet_event(void *arg, struct ifnet *ifp, int event) +{ + if (event != IFNET_EVENT_PCP || is_vlan_dev(ifp)) + return; + + /* make sure GID table is reloaded */ + roce_gid_delete_all_event(ifp); + roce_gid_queue_scan_event(ifp); +} + +static void roce_rescan_device_handler(struct work_struct *_work) { struct roce_rescan_work *work = @@ -445,11 +458,18 @@ int __init roce_gid_mgmt_init(void) */ register_netdevice_notifier(&nb_inetaddr); + eh_ifnet_event = EVENTHANDLER_REGISTER(ifnet_event, + roce_ifnet_event, NULL, EVENTHANDLER_PRI_ANY); + return 0; } void __exit roce_gid_mgmt_cleanup(void) { + + if (eh_ifnet_event != NULL) + EVENTHANDLER_DEREGISTER(ifnet_event, eh_ifnet_event); + unregister_inetaddr_notifier(&nb_inetaddr); unregister_netdevice_notifier(&nb_inetaddr); Modified: head/sys/ofed/drivers/infiniband/core/ib_verbs.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_verbs.c Tue Jul 17 09:09:17 2018 (r336371) +++ head/sys/ofed/drivers/infiniband/core/ib_verbs.c Tue Jul 17 09:11:53 2018 (r336372) @@ -402,12 +402,8 @@ static bool find_gid_index(const union ib_gid *gid, if (ctx->gid_type != gid_attr->gid_type) return false; - - if ((!!(ctx->vlan_id != 0xffff) == !is_vlan_dev(gid_attr->ndev)) || - (is_vlan_dev(gid_attr->ndev) && - vlan_dev_vlan_id(gid_attr->ndev) != ctx->vlan_id)) + if (rdma_vlan_dev_vlan_id(gid_attr->ndev) != ctx->vlan_id) return false; - return true; } @@ -484,7 +480,7 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 po if (rdma_protocol_roce(device, port_num)) { struct ib_gid_attr dgid_attr; - const u16 vlan_id = wc->wc_flags & IB_WC_WITH_VLAN ? + const u16 vlan_id = (wc->wc_flags & IB_WC_WITH_VLAN) ? wc->vlan_id : 0xffff; if (!(wc->wc_flags & IB_WC_GRH)) Modified: head/sys/ofed/include/rdma/ib_addr.h ============================================================================== --- head/sys/ofed/include/rdma/ib_addr.h Tue Jul 17 09:09:17 2018 (r336371) +++ head/sys/ofed/include/rdma/ib_addr.h Tue Jul 17 09:11:53 2018 (r336372) @@ -166,6 +166,8 @@ static inline u16 rdma_vlan_dev_vlan_id(const struct n { uint16_t tag; + if (dev->if_pcp != IFNET_PCP_NONE) + return 0x0000; /* prio-tagged traffic */ if (VLAN_TAG(__DECONST(struct ifnet *, dev), &tag) != 0) return 0xffff; return tag; @@ -345,8 +347,10 @@ static inline u16 rdma_get_vlan_id(union ib_gid *dgid) return vid < 0x1000 ? vid : 0xffff; } -static inline struct net_device *rdma_vlan_dev_real_dev(const struct net_device *dev) +static inline struct net_device *rdma_vlan_dev_real_dev(struct net_device *dev) { + if (dev->if_pcp != IFNET_PCP_NONE) + return dev; /* prio-tagged traffic */ return VLAN_TRUNKDEV(__DECONST(struct ifnet *, dev)); }