Date: Sat, 26 Aug 2023 22:31:33 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 273372] SR-IOV Networking in Bhyve Causes Chelsio T520-SO-CR to Fail on Host, Kernel Panic if Reset Message-ID: <bug-273372-27103@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D273372 Bug ID: 273372 Summary: SR-IOV Networking in Bhyve Causes Chelsio T520-SO-CR to Fail on Host, Kernel Panic if Reset Product: Base System Version: 13.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: mark@markmcb.com If I setup SR-IOV for bhyve using a Chelsio T520-SO-CR network adapter, sta= rt the install process for a FreeBSD guest, but shortly (maybe a minute) after= I load the driver for the pci device in the guest, the network card stops responding in the guest and host. # Relevant /boot/loader.conf vmm_load=3D"YES" nmdm_load=3D"YES" t5fw_cfg_load=3D"YES" if_cxgbe_load=3D"YES" if_cxgbev_load=3D"YES" # Relevant /etc/rc.conf iovctl_files=3D"/etc/iov/cxl1.conf" vm_enable=3D"YES" vm_dir=3D"zfs:zapps/bhyve" vm_list=3D"" vm_delay=3D"5" # Relevant /etc/iov/cxl1.conf PF { device : "cxl1";=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 num_vfs : 14; } DEFAULT { passthrough : false; } # ... # VFs for bhyve VF-10 { mac-addr : "aa:88:44:00:02:20"; passthrough : true; } # ... ### pciconf -lvbc in host ppt0@pci0:2:0:49: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xdd00a000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xdd060000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xdd194000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 0 corrected ecap 000e[140] =3D ARI 1 ecap 0017[150] =3D TPH Requester 1 Use vm install freebsd-test FreeBSD-13.2-RELEASE-amd64-bootonly.iso Config line for passthru: passthru0=3D"2/0/49" Connect with vm console freebsd-test Installer starts, choose shell. ### pciconf -lvbc in guest, prior to driver load none0@pci0:0:5:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xc000e000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xc0000000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xc000c000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks # kldload cxlv t5vf0: <Chelsio T520-SO VF> mem 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc000c000-0xc000dfff at device= 5.0 on pci0 t5vf0: 1 ports, 2 MSI-X interrupts, 3 eq, 2 iq cxlv0: <port 0> on t5vf0 cxlv0: 1 txq, 1 rxq (NIC) ### pciconf -lvbc in guest, after driver load t5vf0@pci0:0:5:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xc000e000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xc0000000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xc000c000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages, enabled Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks Shortly after loading the driver, I lose networking on the host. dmesg shows nothing after event # ifconfig looks normal cxl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS= UM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOM= AP> ether 00:07:43:36:bc:80 inet 10.0.1.201 netmask 0xffffff00 broadcast 10.0.1.255 media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause> status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # ifconfig cxl0 down # ifconfig cxl0 up # Aug 26 15:21:11 core18 kernel: t5nex0: command 0x16 in mbox 4 timed out (0x4014c010). Aug 26 15:21:11 core18 kernel: t5nex0: mbox 4 cmdsent 16a0094400000001 05dc050000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 26 15:21:11 core18 kernel: t5nex0: mbox 4 current 16a0094400000001 05dc050000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 26 15:21:11 core18 kernel: t5nex0: encountered fatal error, adapter sto= pped (1). --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-273372-27103>