Date: Sat, 23 Sep 2023 08:14:30 +0000 From: Benoit Chesneau <benoitc@enki-multimedia.eu> To: "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org> Subject: how to setup the host interfacce with sriov and mlx5en(4) driver Message-ID: <QQYOMF0aCNNuAHisQ5RWN94trnvu79VtRcbtikpRnFo2Y93LOrgdB0cROK2eUAHger9EHaXuEFMLR2tui3VFE2WDX8V7pOHGFbRkgLji0bo=@enki-multimedia.eu>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hi all,
I am trying to understand how works sriov with the mlx5en(4) driver.
In particularly should Isetup the VF-0 as passthrough false? On intel NIC i would have to set one of the VF not passthrough for the host. It seems that sr-iov is behaving differently on mellanox: even if all VFs are set with `passthrough=true` I can still configure the interface mce0 on the host.
Is this expected? Does it mean the VF-0 is usable too?
I have setup sr-iov thiis way:
```
$ more /etc/iov/mce0.conf
PF {
device: "mlx5_core0";
num_vfs: 8;
}
DEFAULT {
passthrough: true;
}
# VF used by host
#VF-0 {
# passthrough: false;#}
```
In rc.conf i have the line `iovctl_files="/etc/iov/mce0.conf" ` to enable it.
I can see sriov is workig using `pciconf -lev` which return a numbe rof ppt devices.
````
mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx]'
class = network
subclass = ethernet
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx]'
class = network
subclass = ethernet
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
--
vendor = 'Adaptec'
device = 'Smart Storage PQI SAS'
class = mass storage
subclass = SAS
ppt0@pci0:17:0:2: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt1@pci0:17:0:3: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt2@pci0:17:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt3@pci0:17:0:5: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt4@pci0:17:0:6: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt5@pci0:17:0:7: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt6@pci0:17:0:8: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt7@pci0:17:0:9: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network subclass = ethernet
````
I can still set the derive mce0 using ifconfig on the host and I have access to all ppt in bhyve:
host:
```
ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=7eed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
ether 9c:dc:71:4c:84:f0
inet6 fe80::9edc:71ff:fe4c:84f0%mce0 prefixlen 64 scopeid 0x4
inet6 2a0e:e701:110b:102::12 prefixlen 64
media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```
In bhybe I have setup as passthrough: 17/0/3=2:0 so the third ppt.
Benoît
[-- Attachment #2 --]
<div style="font-family: Arial, sans-serif; font-size: 14px;">Hi all,</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">I am trying to understand how works sriov with the <span>mlx5en(4) </span> driver.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"> In particularly should Isetup the VF-0 as passthrough false? <span style="text-decoration: none; display: inline !important; background-color: rgb(255, 255, 255);">On intel NIC i would have to set one of the VF not passthrough for the host.</span> It seems that sr-iov is behaving differently on mellanox: even if all VFs are set with `passthrough=true` I can still configure the interface mce0 on the host. </div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Is this expected? Does it mean the VF-0 is usable too?</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">I have setup sr-iov thiis way:</div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span>$ more /etc/iov/mce0.conf</span><div><span>PF {</span></div><div><span> device: "mlx5_core0";</span></div><div><span> num_vfs: 8;</span></div><div><span>}</span></div><div><br></div><div><span>DEFAULT {</span></div><div><span> passthrough: true;</span></div><div><span>}</span></div><div><br></div><div><span># VF used by host</span></div><div><span>#VF-0 {</span></div><div><span># passthrough: false;</span></div><span>#}</span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">In rc.conf i have the line `iovctl_files="/etc/iov/mce0.conf" ` to enable it.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">I can see sriov is workig using `pciconf -lev` which return a numbe rof ppt devices.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">````</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span>mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4</span><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> PCI-e errors = Correctable Error Detected</span></div><div><span> Unsupported Request Detected</span></div><div><span> Corrected = Advisory Non-Fatal Error</span></div><div><span>mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> PCI-e errors = Correctable Error Detected</span></div><div><span> Unsupported Request Detected</span></div><div><span>--</span></div><div><span> vendor = 'Adaptec'</span></div><div><span> device = 'Smart Storage PQI SAS'</span></div><div><span> class = mass storage</span></div><div><span> subclass = SAS</span></div><div><span>ppt0@pci0:17:0:2: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt1@pci0:17:0:3: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt2@pci0:17:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt3@pci0:17:0:5: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt4@pci0:17:0:6: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt5@pci0:17:0:7: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt6@pci0:17:0:8: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span>ppt7@pci0:17:0:9: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'</span></div><div><span> class = network</span></div><span> subclass = ethernet</span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">````</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">I can still set the derive mce0 using ifconfig on the host and I have access to all ppt in bhyve:</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">host:</div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span>ifconfig mce0</span><div><span>mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000</span></div><div><span> options=7eed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO></span></div><div><span> ether 9c:dc:71:4c:84:f0</span></div><div><span> inet6 fe80::9edc:71ff:fe4c:84f0%mce0 prefixlen 64 scopeid 0x4</span></div><div><span> inet6 2a0e:e701:110b:102::12 prefixlen 64</span></div><div><span> media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause></span></div><div><span> status: active</span></div><div><span> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL></span></div></div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">In bhybe I have setup as passthrough: <span>17/0/3=2:0 so the third ppt. </span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div class="protonmail_signature_block" style="font-family: Arial, sans-serif; font-size: 14px;">
<div class="protonmail_signature_block-user">
<div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Helvetica; font-size: 12px; color: rgb(0, 0, 0);">Benoît</div></div>
</div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?QQYOMF0aCNNuAHisQ5RWN94trnvu79VtRcbtikpRnFo2Y93LOrgdB0cROK2eUAHger9EHaXuEFMLR2tui3VFE2WDX8V7pOHGFbRkgLji0bo=>