From owner-freebsd-virtualization@freebsd.org Thu Jul 30 00:33:53 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E17DC3704DD for ; Thu, 30 Jul 2020 00:33:53 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BHBH511zZz4SFf for ; Thu, 30 Jul 2020 00:33:52 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: by mail-ed1-x532.google.com with SMTP id l23so4999101edv.11 for ; Wed, 29 Jul 2020 17:33:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli-net.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=H0AyB++eKp9FjZg0v+bEdY5YniT+hW7QYXUNMA8D/sI=; b=PeHBk/7ZQmV903Mf3iHi3KOyPCC2skY9Qfge7qd2QlDN71/coePXdfWXlWimRvpO0L X4YShYrjaYo2yXpfm8akU7GK074SEiOXLr/tQqmm8ooFQMyktxJ8nH8j8qkuWdJ/1T0p 5WmFAzFMu6SvtB44r/e2yg+TiYAYobKfXBLHHRgeuU8oM+Z5ooTvqX8RlycOJ8epPPX3 gVemEi8pTkjzwyGXCVhgf8ASWhTNssWwLm3sL9NoBsaD7826/GfwqpCnABxUQXGHVkvi nIp0n576/dDbZzk8TSEtz7Q/9c9RsWrTkYY+ECgR2Dis6+qIZVK4IQFu33HRg86tH8QJ UFWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=H0AyB++eKp9FjZg0v+bEdY5YniT+hW7QYXUNMA8D/sI=; b=MsKWj+LaU2CjOSLwdL8Zmv5Y3OuATipibIOJAWbcKXMuOD9ybquTQrK+ddMv7kqoJs 0IeKiKTMioxWNx/Crs/Yc/1/V4uu7unkhqRGF3NYMKa2JAw+FGtg4lJlMVAECzNLSZaU UzpMK7U+9GTBT5RGqbYHD3T8kJkn4XxOVu4cXHTt/PNMbdtNaq0XNwc60YBytV+W8PU4 qg/lrx7XfwgeB/G5Iwm40JhfrZIR3Ry+cCsUzm1DVa2FF7MLfviTH6Elfm40vUSeDtGY 4LhCsek+IpxOKdgafrPmTXOSVvRLR6YzNhr4vT79Q8hkdGFnXRnQ7v2eEe++oE86bcSk U5HA== X-Gm-Message-State: AOAM532dI3O0aXiRbSZ0tLVKI8XO+UNM+x9uNyicC3Nn6A7AD/QYwY1q YvYPc16QJDSqROYJLdIzSlK+QnLfEOXCjwcRXe6FL2um X-Google-Smtp-Source: ABdhPJy96t1Lu1KR+9hQ5Wfa4nljDCUYZW5to/GjpQRm7a2CbmUWqffiyiooEaQpwehhmj0/iMnckxBoH8xFK0A8h7I= X-Received: by 2002:a05:6402:1e2:: with SMTP id i2mr348325edy.70.1596069231335; Wed, 29 Jul 2020 17:33:51 -0700 (PDT) MIME-Version: 1.0 From: Chuck Tuffli Date: Wed, 29 Jul 2020 17:33:40 -0700 Message-ID: Subject: bhyve passthrough of SR-IOV To: freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: 4BHBH511zZz4SFf X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuffli-net.20150623.gappssmtp.com header.s=20150623 header.b=PeHBk/7Z; dmarc=none; spf=permerror (mx1.freebsd.org: domain of chuck@tuffli.net uses mechanism not recognized by this client) smtp.mailfrom=chuck@tuffli.net X-Spamd-Result: default: False [-0.61 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.17)[-0.173]; R_DKIM_ALLOW(-0.20)[tuffli-net.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.75)[-0.755]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[tuffli.net]; NEURAL_SPAM_SHORT(0.62)[0.621]; R_SPF_PERMFAIL(0.00)[empty SPF record]; DKIM_TRACE(0.00)[tuffli-net.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::532:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2020 00:33:53 -0000 I'm new to the world of SR-IOV and could use some guidance. Does anyone have experience doing PCI passthrough of SR-IOV virtual functions on Mellanox CNA's to bhyve? Are there any known limitations? For example, does RDMA not work? The reason I ask is, I have a setup running FreeBSD 12-stable with a Mellanox ConnectX-5 configured to support 16 VFs on each port. All VFs are configured to be passthrough. If I pass through a physical function to the Linux VM, the RDMA-based application works correctly. But if I pass through a virtual function, the RDMA-based application starts complaining about "ibv_context" (i.e. RDMA) failures. It's worth noting that even in this case, I can assign an IP address to the device in the VM and ping addresses on the other side of the wire. So I believe passthrough (mostly?) worked. Any suggestions? TIA! --chuck