From owner-svn-src-head@freebsd.org Sun Sep 9 07:20:16 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6EC2108D846; Sun, 9 Sep 2018 07:20:16 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59DC189D26; Sun, 9 Sep 2018 07:20:16 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4EC9E168C9; Sun, 9 Sep 2018 07:20:16 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w897KGHU014193; Sun, 9 Sep 2018 07:20:16 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w897KFXL014189; Sun, 9 Sep 2018 07:20:15 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201809090720.w897KFXL014189@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Sun, 9 Sep 2018 07:20:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r338541 - in head/sys/ofed: drivers/infiniband/core include/rdma X-SVN-Group: head X-SVN-Commit-Author: hselasky X-SVN-Commit-Paths: in head/sys/ofed: drivers/infiniband/core include/rdma X-SVN-Commit-Revision: 338541 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Sep 2018 07:20:17 -0000 Author: hselasky Date: Sun Sep 9 07:20:15 2018 New Revision: 338541 URL: https://svnweb.freebsd.org/changeset/base/338541 Log: Introduce and use sgid_index in CM requests in ibcore. For RoCE, when CM requests are received for RC and UD connections, netdevice of the incoming request is unavailable. Because of that CM requests are always forwarded to init_net namespace. Now that we have the GID index available, introduce SGID index in incoming CM requests and refer to the netdevice of it. While at it fix some incorrect uses of init_net and make sure the rdma_create_id() function stores the VNET it is passed. Based on linux commit: cee104334c98dd04e9dd4d9a4fa4784f7f6aada9 MFC after: 3 days Approved by: re (gjb) Sponsored by: Mellanox Technologies Modified: head/sys/ofed/drivers/infiniband/core/ib_cm.c head/sys/ofed/drivers/infiniband/core/ib_cma.c head/sys/ofed/drivers/infiniband/core/ib_uverbs_marshall.c head/sys/ofed/include/rdma/ib_addr.h head/sys/ofed/include/rdma/ib_cm.h Modified: head/sys/ofed/drivers/infiniband/core/ib_cm.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_cm.c Sun Sep 9 06:30:15 2018 (r338540) +++ head/sys/ofed/drivers/infiniband/core/ib_cm.c Sun Sep 9 07:20:15 2018 (r338541) @@ -1453,6 +1453,7 @@ static void cm_format_req_event(struct cm_work *work, param->retry_count = cm_req_get_retry_count(req_msg); param->rnr_retry_count = cm_req_get_rnr_retry_count(req_msg); param->srq = cm_req_get_srq(req_msg); + param->ppath_sgid_index = cm_id_priv->av.ah_attr.grh.sgid_index; work->cm_event.private_data = &req_msg->private_data; } @@ -3137,6 +3138,7 @@ out: EXPORT_SYMBOL(ib_send_cm_sidr_req); static void cm_format_sidr_req_event(struct cm_work *work, + const struct cm_id_private *rx_cm_id, struct ib_cm_id *listen_id) { struct cm_sidr_req_msg *sidr_req_msg; @@ -3150,6 +3152,7 @@ static void cm_format_sidr_req_event(struct cm_work *w param->service_id = sidr_req_msg->service_id; param->bth_pkey = cm_get_bth_pkey(work); param->port = work->port->port_num; + param->sgid_index = rx_cm_id->av.ah_attr.grh.sgid_index; work->cm_event.private_data = &sidr_req_msg->private_data; } @@ -3203,7 +3206,7 @@ static int cm_sidr_req_handler(struct cm_work *work) cm_id_priv->id.service_id = sidr_req_msg->service_id; cm_id_priv->id.service_mask = ~cpu_to_be64(0); - cm_format_sidr_req_event(work, &cur_cm_id_priv->id); + cm_format_sidr_req_event(work, cm_id_priv, &cur_cm_id_priv->id); cm_process_work(cm_id_priv, work); cm_deref_id(cur_cm_id_priv); return 0; Modified: head/sys/ofed/drivers/infiniband/core/ib_cma.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_cma.c Sun Sep 9 06:30:15 2018 (r338540) +++ head/sys/ofed/drivers/infiniband/core/ib_cma.c Sun Sep 9 07:20:15 2018 (r338541) @@ -733,6 +733,10 @@ struct rdma_cm_id *rdma_create_id(struct vnet *net, { struct rdma_id_private *id_priv; +#ifdef VIMAGE + if (net == NULL) + return ERR_PTR(-EINVAL); +#endif id_priv = kzalloc(sizeof *id_priv, GFP_KERNEL); if (!id_priv) return ERR_PTR(-ENOMEM); @@ -751,7 +755,7 @@ struct rdma_cm_id *rdma_create_id(struct vnet *net, INIT_LIST_HEAD(&id_priv->listen_list); INIT_LIST_HEAD(&id_priv->mc_list); get_random_bytes(&id_priv->seq_num, sizeof id_priv->seq_num); - id_priv->id.route.addr.dev_addr.net = TD_TO_VNET(curthread); + id_priv->id.route.addr.dev_addr.net = net; return &id_priv->id; } @@ -1375,6 +1379,26 @@ static bool validate_net_dev(struct net_device *net_de } } +static struct net_device * +roce_get_net_dev_by_cm_event(struct ib_device *device, u8 port_num, + const struct ib_cm_event *ib_event) +{ + struct ib_gid_attr sgid_attr; + union ib_gid sgid; + int err = -EINVAL; + + if (ib_event->event == IB_CM_REQ_RECEIVED) { + err = ib_get_cached_gid(device, port_num, + ib_event->param.req_rcvd.ppath_sgid_index, &sgid, &sgid_attr); + } else if (ib_event->event == IB_CM_SIDR_REQ_RECEIVED) { + err = ib_get_cached_gid(device, port_num, + ib_event->param.sidr_req_rcvd.sgid_index, &sgid, &sgid_attr); + } + if (err) + return (NULL); + return (sgid_attr.ndev); +} + static struct net_device *cma_get_net_dev(struct ib_cm_event *ib_event, const struct cma_req_info *req) { @@ -1390,8 +1414,14 @@ static struct net_device *cma_get_net_dev(struct ib_cm if (err) return ERR_PTR(err); - net_dev = ib_get_net_dev_by_params(req->device, req->port, req->pkey, - gid, listen_addr); + if (rdma_protocol_roce(req->device, req->port)) { + net_dev = roce_get_net_dev_by_cm_event(req->device, req->port, + ib_event); + } else { + net_dev = ib_get_net_dev_by_params(req->device, req->port, + req->pkey, + gid, listen_addr); + } if (!net_dev) return ERR_PTR(-ENODEV); @@ -1526,10 +1556,6 @@ static struct rdma_id_private *cma_id_from_event(struc if (IS_ERR(*net_dev)) { if (PTR_ERR(*net_dev) == -EAFNOSUPPORT) { /* Assuming the protocol is AF_IB */ - *net_dev = NULL; - } else if (cma_protocol_roce_dev_port(req.device, req.port)) { - /* TODO find the net dev matching the request parameters - * through the RoCE GID table */ *net_dev = NULL; } else { return ERR_CAST(*net_dev); Modified: head/sys/ofed/drivers/infiniband/core/ib_uverbs_marshall.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_uverbs_marshall.c Sun Sep 9 06:30:15 2018 (r338540) +++ head/sys/ofed/drivers/infiniband/core/ib_uverbs_marshall.c Sun Sep 9 07:20:15 2018 (r338541) @@ -146,7 +146,7 @@ void ib_copy_path_rec_from_user(struct ib_sa_path_rec dst->packet_life_time_selector = src->packet_life_time_selector; memset(dst->dmac, 0, sizeof(dst->dmac)); - dst->net = NULL; + dst->net = TD_TO_VNET(curthread); dst->ifindex = 0; dst->gid_type = IB_GID_TYPE_IB; } Modified: head/sys/ofed/include/rdma/ib_addr.h ============================================================================== --- head/sys/ofed/include/rdma/ib_addr.h Sun Sep 9 06:30:15 2018 (r338540) +++ head/sys/ofed/include/rdma/ib_addr.h Sun Sep 9 07:20:15 2018 (r338541) @@ -221,7 +221,11 @@ static inline void iboe_addr_get_sgid(struct rdma_dev_ struct net_device *dev; struct ifaddr *ifa; - dev = dev_get_by_index(&init_net, dev_addr->bound_dev_if); +#ifdef VIMAGE + if (dev_addr->net == NULL) + return; +#endif + dev = dev_get_by_index(dev_addr->net, dev_addr->bound_dev_if); if (dev) { CK_STAILQ_FOREACH(ifa, &dev->if_addrhead, ifa_link) { if (ifa->ifa_addr == NULL || Modified: head/sys/ofed/include/rdma/ib_cm.h ============================================================================== --- head/sys/ofed/include/rdma/ib_cm.h Sun Sep 9 06:30:15 2018 (r338540) +++ head/sys/ofed/include/rdma/ib_cm.h Sun Sep 9 07:20:15 2018 (r338541) @@ -125,6 +125,13 @@ struct ib_cm_req_event_param { struct ib_sa_path_rec *primary_path; struct ib_sa_path_rec *alternate_path; + /* + * SGID index of the primary path. Currently only + * useful for RoCE. Alternate path GID attributes + * are not yet supported. + */ + u8 ppath_sgid_index; + __be64 remote_ca_guid; u32 remote_qkey; u32 remote_qpn; @@ -231,6 +238,13 @@ struct ib_cm_apr_event_param { struct ib_cm_sidr_req_event_param { struct ib_cm_id *listen_id; __be64 service_id; + + /* + * SGID index of the request. Currently only + * useful for RoCE. + */ + u8 sgid_index; + /* P_Key that was used by the GMP's BTH header */ u16 bth_pkey; u8 port;