From owner-svn-src-all@freebsd.org Fri Apr 22 18:16:14 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 397FFB1822C; Fri, 22 Apr 2016 18:16:14 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08C521393; Fri, 22 Apr 2016 18:16:13 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u3MIGDX6024921; Fri, 22 Apr 2016 18:16:13 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u3MIGCgK024913; Fri, 22 Apr 2016 18:16:12 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <201604221816.u3MIGCgK024913@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Fri, 22 Apr 2016 18:16:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r298486 - in head/sys: ofed/drivers/infiniband/core ofed/drivers/infiniband/hw/mlx4 ofed/include/rdma sys X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Apr 2016 18:16:14 -0000 Author: hselasky Date: Fri Apr 22 18:16:12 2016 New Revision: 298486 URL: https://svnweb.freebsd.org/changeset/base/298486 Log: More fixes for using IPv6 addresses with RDMA: - Added check that the SCOPE ID is only restored for IPv6 linklocal addresses. - Changes made by r237263 in the "cma_bind_addr()" function did not check if the socket address was of type IPv6 and used the IPv4 socket address for IPv6 addresses. This caused the function to fail. Fixed this. - In the "rdma_gid2ip()" function and some other places the "sin6_len" and "sin6_scope_id" fields were not set for IPv6 socket addresses. Fixed this. - The scope ID is not stored as part of the GID entries and must be passed as an argument to "rdma_gid2ip()". - Added new method to "struct ib_device" which returns a pointer to the network interface which belongs to the given infiniband device. This is needed to be able to get the scope ID for IPv6 addresses via the associated ethernet interface. - Added convenience function, "rdma_get_ipv6_scope_id()", to get the scope ID for IPv6 addresses. - Implemented new "get_netdev" method for mlx4ib. Other IB controller drivers which want to support IPv6 addresses needs to implement this aswell. - Bumped the FreeBSD version due to changing "struct ib_device". Sponsored by: Mellanox Technologies MFC after: 1 week Modified: head/sys/ofed/drivers/infiniband/core/addr.c head/sys/ofed/drivers/infiniband/core/cma.c head/sys/ofed/drivers/infiniband/core/uverbs_cmd.c head/sys/ofed/drivers/infiniband/core/verbs.c head/sys/ofed/drivers/infiniband/hw/mlx4/main.c head/sys/ofed/include/rdma/ib_addr.h head/sys/ofed/include/rdma/ib_verbs.h head/sys/sys/param.h Modified: head/sys/ofed/drivers/infiniband/core/addr.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/addr.c Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/drivers/infiniband/core/addr.c Fri Apr 22 18:16:12 2016 (r298486) @@ -43,6 +43,7 @@ #include #include #include +#include MODULE_AUTHOR("Sean Hefty"); @@ -417,7 +418,8 @@ done: #ifdef INET6 if (scope_id < 256) { sin6 = (struct sockaddr_in6 *)src_in; - SCOPE_ID_RESTORE(scope_id, sin6); + if (IN6_IS_SCOPE_LINKLOCAL(&sin6->sin6_addr)) + SCOPE_ID_RESTORE(scope_id, sin6); sin6 = (struct sockaddr_in6 *)dst_in; SCOPE_ID_RESTORE(scope_id, sin6); } @@ -554,7 +556,7 @@ static void resolve_cb(int status, struc } int rdma_addr_find_dmac_by_grh(union ib_gid *sgid, union ib_gid *dgid, u8 *dmac, - u16 *vlan_id) + u16 *vlan_id, u32 scope_id) { int ret = 0; struct rdma_dev_addr dev_addr; @@ -568,11 +570,11 @@ int rdma_addr_find_dmac_by_grh(union ib_ } sgid_addr, dgid_addr; - ret = rdma_gid2ip(&sgid_addr._sockaddr, sgid); + ret = rdma_gid2ip(&sgid_addr._sockaddr, sgid, scope_id); if (ret) return ret; - ret = rdma_gid2ip(&dgid_addr._sockaddr, dgid); + ret = rdma_gid2ip(&dgid_addr._sockaddr, dgid, scope_id); if (ret) return ret; @@ -598,7 +600,23 @@ int rdma_addr_find_dmac_by_grh(union ib_ } EXPORT_SYMBOL(rdma_addr_find_dmac_by_grh); -int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id) +u32 rdma_get_ipv6_scope_id(struct ib_device *ib, u8 port_num) +{ +#ifdef INET6 + struct ifnet *ifp; + if (ib->get_netdev == NULL) + return (-1U); + ifp = ib->get_netdev(ib, port_num); + if (ifp == NULL) + return (-1U); + return (in6_getscopezone(ifp, IPV6_ADDR_SCOPE_LINKLOCAL)); +#else + return (-1U); +#endif +} + +int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id, + u32 scope_id) { int ret = 0; struct rdma_dev_addr dev_addr; @@ -608,8 +626,7 @@ int rdma_addr_find_smac_by_sgid(union ib struct sockaddr_in6 _sockaddr_in6; } gid_addr; - ret = rdma_gid2ip(&gid_addr._sockaddr, sgid); - + ret = rdma_gid2ip(&gid_addr._sockaddr, sgid, scope_id); if (ret) return ret; memset(&dev_addr, 0, sizeof(dev_addr)); Modified: head/sys/ofed/drivers/infiniband/core/cma.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/cma.c Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/drivers/infiniband/core/cma.c Fri Apr 22 18:16:12 2016 (r298486) @@ -689,8 +689,11 @@ static int cma_modify_qp_rtr(struct rdma == RDMA_TRANSPORT_IB && rdma_port_get_link_layer(id_priv->id.device, id_priv->id.port_num) == IB_LINK_LAYER_ETHERNET) { - ret = rdma_addr_find_smac_by_sgid(&sgid, qp_attr.smac, NULL); + u32 scope_id = rdma_get_ipv6_scope_id(id_priv->id.device, + id_priv->id.port_num); + ret = rdma_addr_find_smac_by_sgid(&sgid, qp_attr.smac, NULL, + scope_id); if (ret) goto out; } @@ -931,11 +934,13 @@ static void cma_save_net_info(struct rdm ip4->sin_family = listen4->sin_family; ip4->sin_addr.s_addr = dst->ip4.addr; ip4->sin_port = listen4->sin_port; + ip4->sin_len = sizeof(struct sockaddr_in); ip4 = (struct sockaddr_in *) &addr->dst_addr; ip4->sin_family = listen4->sin_family; ip4->sin_addr.s_addr = src->ip4.addr; ip4->sin_port = port; + ip4->sin_len = sizeof(struct sockaddr_in); break; case 6: listen6 = (struct sockaddr_in6 *) &listen_addr->src_addr; @@ -943,11 +948,15 @@ static void cma_save_net_info(struct rdm ip6->sin6_family = listen6->sin6_family; ip6->sin6_addr = dst->ip6; ip6->sin6_port = listen6->sin6_port; + ip6->sin6_len = sizeof(struct sockaddr_in6); + ip6->sin6_scope_id = listen6->sin6_scope_id; ip6 = (struct sockaddr_in6 *) &addr->dst_addr; ip6->sin6_family = listen6->sin6_family; ip6->sin6_addr = src->ip6; ip6->sin6_port = port; + ip6->sin6_len = sizeof(struct sockaddr_in6); + ip6->sin6_scope_id = listen6->sin6_scope_id; break; default: break; @@ -1431,16 +1440,19 @@ static int cma_req_handler(struct ib_cm_ goto err3; if (is_iboe && !is_sidr) { + u32 scope_id = rdma_get_ipv6_scope_id(cm_id->device, + ib_event->param.req_rcvd.port); + if (ib_event->param.req_rcvd.primary_path != NULL) rdma_addr_find_smac_by_sgid( &ib_event->param.req_rcvd.primary_path->sgid, - psmac, NULL); + psmac, NULL, scope_id); else psmac = NULL; if (ib_event->param.req_rcvd.alternate_path != NULL) rdma_addr_find_smac_by_sgid( &ib_event->param.req_rcvd.alternate_path->sgid, - palt_smac, NULL); + palt_smac, NULL, scope_id); else palt_smac = NULL; } @@ -2296,22 +2308,47 @@ static int cma_bind_addr(struct rdma_cm_ { if (!src_addr || !src_addr->sa_family) { src_addr = (struct sockaddr *) &id->route.addr.src_addr; - if ((src_addr->sa_family = dst_addr->sa_family) == AF_INET6) { + src_addr->sa_family = dst_addr->sa_family; +#ifdef INET6 + if (dst_addr->sa_family == AF_INET6) { ((struct sockaddr_in6 *) src_addr)->sin6_scope_id = ((struct sockaddr_in6 *) dst_addr)->sin6_scope_id; } +#endif } if (!cma_any_addr(src_addr)) return rdma_bind_addr(id, src_addr); else { - struct sockaddr_in addr_in; + union { +#ifdef INET + struct sockaddr_in in; +#endif +#ifdef INET6 + struct sockaddr_in6 in6; +#endif + } addr; - memset(&addr_in, 0, sizeof addr_in); - addr_in.sin_family = dst_addr->sa_family; - addr_in.sin_len = sizeof addr_in; - return rdma_bind_addr(id, (struct sockaddr *) &addr_in); + switch(dst_addr->sa_family) { +#ifdef INET + case AF_INET: + memset(&addr.in, 0, sizeof(addr.in)); + addr.in.sin_family = dst_addr->sa_family; + addr.in.sin_len = sizeof(addr.in); + return rdma_bind_addr(id, (struct sockaddr *)&addr.in); +#endif +#ifdef INET6 + case AF_INET6: + memset(&addr.in6, 0, sizeof(addr.in6)); + addr.in6.sin6_family = dst_addr->sa_family; + addr.in6.sin6_len = sizeof(addr.in6); + addr.in6.sin6_scope_id = + ((struct sockaddr_in6 *)dst_addr)->sin6_scope_id; + return rdma_bind_addr(id, (struct sockaddr *)&addr.in6); +#endif + default: + return -EINVAL; + } } - } int rdma_resolve_addr(struct rdma_cm_id *id, struct sockaddr *src_addr, Modified: head/sys/ofed/drivers/infiniband/core/uverbs_cmd.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/uverbs_cmd.c Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/drivers/infiniband/core/uverbs_cmd.c Fri Apr 22 18:16:12 2016 (r298486) @@ -2094,11 +2094,11 @@ static ssize_t __uverbs_modify_qp(struct } else { ret = rdma_addr_find_dmac_by_grh(&sgid, dgid, attr->ah_attr.dmac, - &attr->vlan_id); + &attr->vlan_id, -1U); if (ret) goto out; ret = rdma_addr_find_smac_by_sgid(&sgid, attr->smac, - NULL); + NULL, -1U); if (ret) goto out; } Modified: head/sys/ofed/drivers/infiniband/core/verbs.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/verbs.c Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/drivers/infiniband/core/verbs.c Fri Apr 22 18:16:12 2016 (r298486) @@ -207,8 +207,10 @@ int ib_init_ah_from_wc(struct ib_device memcpy(ah_attr->dmac, wc->smac, ETH_ALEN); ah_attr->vlan_id = wc->vlan_id; } else { + u32 scope_id = rdma_get_ipv6_scope_id(device, port_num); ret = rdma_addr_find_dmac_by_grh(&grh->dgid, &grh->sgid, - ah_attr->dmac, &ah_attr->vlan_id); + ah_attr->dmac, &ah_attr->vlan_id, + scope_id); if (ret) return ret; } Modified: head/sys/ofed/drivers/infiniband/hw/mlx4/main.c ============================================================================== --- head/sys/ofed/drivers/infiniband/hw/mlx4/main.c Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/drivers/infiniband/hw/mlx4/main.c Fri Apr 22 18:16:12 2016 (r298486) @@ -1620,6 +1620,12 @@ free: kfree(gw); } +static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, u8 port_num) +{ + struct mlx4_ib_dev *ibdev = to_mdev(device); + return mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num); +} + static void reset_gids_task(struct work_struct *work) { struct update_gid_work *gw = @@ -2353,6 +2359,7 @@ static void *mlx4_ib_add(struct mlx4_dev ibdev->ib_dev.attach_mcast = mlx4_ib_mcg_attach; ibdev->ib_dev.detach_mcast = mlx4_ib_mcg_detach; ibdev->ib_dev.process_mad = mlx4_ib_process_mad; + ibdev->ib_dev.get_netdev = mlx4_ib_get_netdev; ibdev->ib_dev.ioctl = mlx4_ib_ioctl; ibdev->ib_dev.query_values = mlx4_ib_query_values; Modified: head/sys/ofed/include/rdma/ib_addr.h ============================================================================== --- head/sys/ofed/include/rdma/ib_addr.h Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/include/rdma/ib_addr.h Fri Apr 22 18:16:12 2016 (r298486) @@ -105,9 +105,10 @@ void rdma_addr_cancel(struct rdma_dev_ad int rdma_copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev, const unsigned char *dst_dev_addr); -int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id); +int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id, + u32 scope_id); int rdma_addr_find_dmac_by_grh(union ib_gid *sgid, union ib_gid *dgid, u8 *smac, - u16 *vlan_id); + u16 *vlan_id, u32 scope_id); static inline int ip_addr_size(struct sockaddr *addr) { @@ -164,7 +165,8 @@ static inline int rdma_ip2gid(struct soc } /* Important - sockaddr should be a union of sockaddr_in and sockaddr_in6 */ -static inline int rdma_gid2ip(struct sockaddr *out, union ib_gid *gid) +static inline int rdma_gid2ip(struct sockaddr *out, union ib_gid *gid, + uint32_t scope_id) { if (ipv6_addr_v4mapped((struct in6_addr *)gid)) { struct sockaddr_in *out_in = (struct sockaddr_in *)out; @@ -175,12 +177,18 @@ static inline int rdma_gid2ip(struct soc } else { struct sockaddr_in6 *out_in = (struct sockaddr_in6 *)out; memset(out_in, 0, sizeof(*out_in)); + out_in->sin6_len = sizeof(*out_in); out_in->sin6_family = AF_INET6; memcpy(&out_in->sin6_addr.s6_addr, gid->raw, 16); + if (scope_id < 256 && + IN6_IS_SCOPE_LINKLOCAL(&out_in->sin6_addr)) + out_in->sin6_scope_id = scope_id; } return 0; } +u32 rdma_get_ipv6_scope_id(struct ib_device *ib, u8 port_num); + /* This func is called only in loopback ip address (127.0.0.1) * case in which sgid is not relevant */ Modified: head/sys/ofed/include/rdma/ib_verbs.h ============================================================================== --- head/sys/ofed/include/rdma/ib_verbs.h Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/ofed/include/rdma/ib_verbs.h Fri Apr 22 18:16:12 2016 (r298486) @@ -1655,6 +1655,14 @@ struct ib_device { struct ib_port_attr *port_attr); enum rdma_link_layer (*get_link_layer)(struct ib_device *device, u8 port_num); + /* When calling get_netdev, the HW vendor's driver should return the + * net device of device @device at port @port_num. The function + * is called in rtnl_lock. The HW vendor's device driver must guarantee + * to return NULL before the net device has reached + * NETDEV_UNREGISTER_FINAL state. + */ + struct net_device *(*get_netdev)(struct ib_device *device, + u8 port_num); int (*query_gid)(struct ib_device *device, u8 port_num, int index, union ib_gid *gid); Modified: head/sys/sys/param.h ============================================================================== --- head/sys/sys/param.h Fri Apr 22 18:05:34 2016 (r298485) +++ head/sys/sys/param.h Fri Apr 22 18:16:12 2016 (r298486) @@ -58,7 +58,7 @@ * in the range 5 to 9. */ #undef __FreeBSD_version -#define __FreeBSD_version 1100105 /* Master, propagated to newvers */ +#define __FreeBSD_version 1100106 /* Master, propagated to newvers */ /* * __FreeBSD_kernel__ indicates that this system uses the kernel of FreeBSD,