Date: Sun, 10 Jul 2022 13:33:38 +0000 From: Benoit Chesneau <benoitc@enki-multimedia.eu> To: "freebsd-net@FreeBSD.org" <freebsd-net@FreeBSD.org> Subject: qlnxe driver and SR-iOV ? VF driver unknown Message-ID: <zY9JaoLuM9_Uu9IPsxba2kUCZdFXyY0VScJgfvXnxElRqd_e91qp2z115Jl73c-QiL6GIMf_9XY1dic-oBtSdeG0zvSuFAaOovC57wUINg0=@enki-multimedia.eu>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
I have an HPE branded 2x25G qlogic card (HPE Eth 1G 2p 368i/10G 2p 568FLR-MMSFP+ Adptr 866467-B21) which support SR-IOV. I have enabled it using the following configuration below. I confirmed SR-IOV is enabled and when running the pciconf command I see the VFs. Unfortunately the VI I set with passthrough = false is not recognized. I get get the following message in dmesg:
```
ql0: qlnx_iov_add_vf: mac_addr = 1e:4f:d8:c5:92:ab
pci4: <network, ethernet> at device 0.16 (no driver attached)
```
Any idea of what I am missing?
/etc/iov/ql0.conf
```
PF {
device: "ql0";
num_vfs: 8;
}
DEFAULT {
passthrough: true;
allow-set-mac: true;
allow-promisc: true;
}
VF-0 {
passthrough: false;
mac-addr: "1e:4f:d8:c5:92:ab";}
```
I can confirm it is enabled:
```
# pciconf -lvc ql0
ql0@pci0:17:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8070 subvendor=0x1590 subdevice=0x021d
vendor = 'QLogic Corp.'
device = 'FastLinQ QL41000 Series 10/25/40/50GbE Controller'
class = network
subclass = ethernet
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS
max read 4096
link x8(x8) speed 8.0(8.0) ClockPM disabled
cap 11[b0] = MSI-X supports 129 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0002[148] = VC 1 max VC0
ecap 0003[168] = Serial 1 0000000000000000
ecap 0004[178] = Power Budgeting 1
ecap 000e[188] = ARI 1
ecap 0019[198] = PCIe Sec 1 lane errors 0
ecap 0010[1b8] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled
8 VFs configured out of 96 supported
First VF RID Offset 0x0010, VF RID Stride 0x0001
VF Device ID 0x8090
Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 4194304
iov bar [1dc] = type Prefetchable Memory, range 64, base 0xc7ffc800000, size 32768, enabled
iov bar [1e4] = type Prefetchable Memory, range 64, base 0xc7ffc840000, size 262144, enabled
iov bar [1ec] = type Prefetchable Memory, range 64, base 0xc7ffca40000, size 8192, enabled
ecap 0017[1f8] = TPH Requester 1
ecap 0018[284] = LTR 1
ecap 000b[28c] = Vendor [1] ID 0002 Rev 3 Length 256
ecap 000b[38c] = Vendor [1] ID 0001 Rev 1 Length 56
ecap 001f[3c4] = Precision Time Measurement 1
ecap 000b[3d0] = Vendor [1] ID 0003 Rev 1 Length 84 ecap 0015[424] = Resizable BAR 1
```
The VF with passthrough attached has no driver:
```
none88@pci0:17:0:16: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8090 subvendor=0x1590 subdevice=0x021d
vendor = 'QLogic Corp.'
device = 'FastLinQ QL41000 Series Gigabit Ethernet Controller (SR-IOV VF)'
class = network
subclass = ethernet
cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS
max read 4096
link x0(x8) speed 0.0(8.0) ClockPM disabled
cap 11[b0] = MSI-X supports 12 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
ecap 000e[100] = ARI 1 ecap 0017[110] = TPH Requester 1
```
loader.conf:
```
more /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
hw.vga.textmode=1
net.fibs="16"
boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
#console="comconsole,vidconsole"
cryptodev_load="YES"
zfs_load="YES"
if_ixl_updated_load="YES"
if_qlnxe_load="YES"
ipmi_load="YES"
vmm_load="YES"nmdm_load="YES"
```
Benoît Chesneau, Enki Multimedia
—
t. +33608655490
Sent with [Proton Mail](https://proton.me/) secure email.
[-- Attachment #2 --]
<div style="font-family: arial; font-size: 14px;">I have an HPE branded 2x25G qlogic card (<span style="font-size:16px;caret-color:rgb(51, 51, 51);color:rgb(51, 51, 51);font-family:Metric, Arial, sans-serif;text-align:left;background-color:rgb(208, 243, 235);display:inline !important">HPE Eth 1G 2p 368i/10G 2p 568FLR-MMSFP+ Adptr <span style="caret-color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);display:inline !important">866467-B21) which support SR-IOV. I have enabled it using the following configuration below. I confirmed SR-IOV is enabled and when running the pciconf command I see the VFs. Unfortunately the VI I set with passthrough = false is not recognized. I get get the following message in dmesg:</span></span></div><div style="font-family: arial; font-size: 14px;"><span style="font-size:16px;caret-color:rgb(51, 51, 51);color:rgb(51, 51, 51);font-family:Metric, Arial, sans-serif;text-align:left;background-color:rgb(208, 243, 235);display:inline !important"><span style="caret-color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);display:inline !important"><br></span></span></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);">```</span></font></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);"><span>ql0: qlnx_iov_add_vf: mac_addr = 1e:4f:d8:c5:92:ab</span><br><span>pci4: <network, ethernet> at device 0.16 (no driver attached)</span><br></span></font></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);">```</span></font></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);"><br></span></font></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);">Any idea of what I am missing? </span></font></div><div><font color="#333333" face="Metric, Arial, sans-serif" size="3"><span style="caret-color: rgb(51, 51, 51);"><br></span></font></div><div style="font-family: arial; font-size: 14px;"><span style="font-size:16px;caret-color:rgb(51, 51, 51);color:rgb(51, 51, 51);font-family:Metric, Arial, sans-serif;text-align:left;background-color:rgb(208, 243, 235);display:inline !important"><span style="caret-color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);display:inline !important">/etc/iov/ql0.conf</span></span></div><div style="font-family: arial; font-size: 14px;"><span style="font-size:16px;caret-color:rgb(51, 51, 51);color:rgb(51, 51, 51);font-family:Metric, Arial, sans-serif;text-align:left;background-color:rgb(208, 243, 235);display:inline !important"><span style="caret-color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);display:inline !important"><span>```</span></span></span></div><div style="font-family: arial; font-size: 14px;"><span style="font-size:16px;caret-color:rgb(51, 51, 51);color:rgb(51, 51, 51);font-family:Metric, Arial, sans-serif;text-align:left;background-color:rgb(208, 243, 235);display:inline !important"><span style="caret-color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);display:inline !important"><span>PF {</span><div><span> device: "ql0";</span></div><div><span> num_vfs: 8;</span></div><div><span>}</span></div><div><br></div><div><span>DEFAULT {</span></div><div><span> passthrough: true;</span></div><div><span> allow-set-mac: true;</span></div><div><span> allow-promisc: true;</span></div><div><span>}</span></div><div><br></div><div><span>VF-0 {</span></div><div><span> passthrough: false;</span></div><div><span> mac-addr: "1e:4f:d8:c5:92:ab";</span></div><span>}</span><br></span></span></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><br></div><div style="font-family: arial; font-size: 14px;">I can confirm it is enabled:</div><div style="font-family: arial; font-size: 14px;"><br></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><span># pciconf -lvc ql0</span><div><span>ql0@pci0:17:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8070 subvendor=0x1590 subdevice=0x021d</span></div><div><span> vendor = 'QLogic Corp.'</span></div><div><span> device = 'FastLinQ QL41000 Series 10/25/40/50GbE Controller'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> cap 01[40] = powerspec 3 supports D0 D3 current D0</span></div><div><span> cap 05[50] = MSI supports 8 messages, 64 bit, vector masks</span></div><div><span> cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS</span></div><div><span> max read 4096</span></div><div><span> link x8(x8) speed 8.0(8.0) ClockPM disabled</span></div><div><span> cap 11[b0] = MSI-X supports 129 messages, enabled</span></div><div><span> Table in map 0x20[0x0], PBA in map 0x20[0x1000]</span></div><div><span> cap 03[d0] = VPD</span></div><div><span> ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected</span></div><div><span> ecap 0002[148] = VC 1 max VC0</span></div><div><span> ecap 0003[168] = Serial 1 0000000000000000</span></div><div><span> ecap 0004[178] = Power Budgeting 1</span></div><div><span> ecap 000e[188] = ARI 1</span></div><div><span> ecap 0019[198] = PCIe Sec 1 lane errors 0</span></div><div><span> ecap 0010[1b8] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled</span></div><div><span> 8 VFs configured out of 96 supported</span></div><div><span> First VF RID Offset 0x0010, VF RID Stride 0x0001</span></div><div><span> VF Device ID 0x8090</span></div><div><span> Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 4194304</span></div><div><span> iov bar [1dc] = type Prefetchable Memory, range 64, base 0xc7ffc800000, size 32768, enabled</span></div><div><span> iov bar [1e4] = type Prefetchable Memory, range 64, base 0xc7ffc840000, size 262144, enabled</span></div><div><span> iov bar [1ec] = type Prefetchable Memory, range 64, base 0xc7ffca40000, size 8192, enabled</span></div><div><span> ecap 0017[1f8] = TPH Requester 1</span></div><div><span> ecap 0018[284] = LTR 1</span></div><div><span> ecap 000b[28c] = Vendor [1] ID 0002 Rev 3 Length 256</span></div><div><span> ecap 000b[38c] = Vendor [1] ID 0001 Rev 1 Length 56</span></div><div><span> ecap 001f[3c4] = Precision Time Measurement 1</span></div><div><span> ecap 000b[3d0] = Vendor [1] ID 0003 Rev 1 Length 84</span></div><span> ecap 0015[424] = Resizable BAR 1</span><br></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><span><br></span></div><div style="font-family: arial; font-size: 14px;">The VF with passthrough attached has no driver:</div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><span>none88@pci0:17:0:16: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8090 subvendor=0x1590 subdevice=0x021d</span><div><span> vendor = 'QLogic Corp.'</span></div><div><span> device = 'FastLinQ QL41000 Series Gigabit Ethernet Controller (SR-IOV VF)'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> cap 10[70] = PCI-Express 2 endpoint max data 256(512) FLR NS</span></div><div><span> max read 4096</span></div><div><span> link x0(x8) speed 0.0(8.0) ClockPM disabled</span></div><div><span> cap 11[b0] = MSI-X supports 12 messages</span></div><div><span> Table in map 0x20[0x0], PBA in map 0x20[0x1000]</span></div><div><span> ecap 000e[100] = ARI 1</span></div><span> ecap 0017[110] = TPH Requester 1</span><br></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><br></div><div style="font-family: arial; font-size: 14px;">loader.conf:</div><div style="font-family: arial; font-size: 14px;"><br></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><span>more /boot/loader.conf</span><div><span>kern.geom.label.disk_ident.enable="0"</span></div><div><span>kern.geom.label.gptid.enable="0"</span></div><div><span>hw.vga.textmode=1</span></div><div><span>net.fibs="16"</span></div><div><span>boot_multicons="YES"</span></div><div><span>boot_serial="YES"</span></div><div><span>comconsole_speed="115200"</span></div><div><span>#console="comconsole,vidconsole"</span></div><div><span>cryptodev_load="YES"</span></div><div><span>zfs_load="YES"</span></div><div><span>if_ixl_updated_load="YES"</span></div><div><span>if_qlnxe_load="YES"</span></div><div><span>ipmi_load="YES"</span></div><div><span>vmm_load="YES"</span></div><span>nmdm_load="YES"</span><br></div><div style="font-family: arial; font-size: 14px;">```</div><div style="font-family: arial; font-size: 14px;"><br></div>
<div class="protonmail_signature_block" style="font-family: arial; font-size: 14px;">
<div class="protonmail_signature_block-user">
<div style="font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(0,0,0);font-family:Helvetica;font-size:12px;">Benoît Chesneau, Enki Multimedia<br></div><div style="font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(0,0,0);font-family:Helvetica;font-size:12px;">—<br></div><div style="font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;color:rgb(0,0,0);font-family:Helvetica;font-size:12px;">t. +33608655490 <br></div>
</div>
<div style="font-family: arial; font-size: 14px;"><br></div>
<div class="protonmail_signature_block-proton">
Sent with <a target="_blank" href="https://proton.me/" rel="noopener noreferrer">Proton Mail</a> secure email.
</div>
</div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?zY9JaoLuM9_Uu9IPsxba2kUCZdFXyY0VScJgfvXnxElRqd_e91qp2z115Jl73c-QiL6GIMf_9XY1dic-oBtSdeG0zvSuFAaOovC57wUINg0=>