Date: Sat, 05 Jan 2019 00:06:25 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234624] FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Message-ID: <bug-234624-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234624 Bug ID: 234624 Summary: FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Product: Base System Version: 12.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: misc Assignee: bugs@FreeBSD.org Reporter: freebsd@aoeu.uk Hi, Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performance dropped significantly. Seems to be related to TSO, or at least disabling it resolves the problem for me. Setup is a physical FreeBSD 12 host connected to a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit switch. 23:58:18 root@filer:/root/ uname -a FreeBSD filer.<censored> 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 iperf3 tests from physical FreeBSD to virtual Debian: 23:32:39 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.0 KBytes 572 Kbits/sec 8 17.5 KBytes [ 5] 1.00-2.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 2.00-3.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 3.00-4.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 4.00-5.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 5.00-6.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 6.00-7.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 7.00-8.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 8.00-9.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 9.00-10.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 411 KBytes 336 Kbits/sec 86 sender [ 5] 0.00-10.00 sec 376 KBytes 308 Kbits/sec receiver iperf Done. 23:32:51 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 3.00-4.00 sec 1.13 GBytes 9.71 Gbits/sec [ 5] 4.00-5.00 sec 1.13 GBytes 9.73 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.84 Gbits/sec receiver iperf Done. Turning off TSO: 23:33:17 root@filer:/root/ ifconfig ix0 -tso Re-running iperf3: 23:33:57 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver iperf Done. 23:34:12 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.14 GBytes 9.81 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec receiver iperf Done. Network configuration is not entirely trivial. From /etc/rc.conf (ipv6 global addresses censored): ipv6_activate_all_interfaces="YES" ifconfig_igb0="up mtu 9000" ifconfig_igb1="up mtu 9000" ifconfig_ix0="up mtu 9000" ifconfig_ix1="up mtu 9000" cloned_interfaces="lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35" ifconfig_lagg0="laggproto failover laggport ix0 laggport igb0" ifconfig_lagg1="laggproto failover laggport ix1 laggport igb1" ifconfig_vlan10="vlan 10 vlandev lagg0 mtu 9000" ifconfig_vlan10_alias0="inet 10.21.10.2/24" ifconfig_vlan10_ipv6="inet6 fd9c:bee6:9d33:10::2 prefixlen 64" ifconfig_vlan11="vlan 11 vlandev lagg1 mtu 9000" ifconfig_vlan11_alias0="inet 10.21.11.2/24" ifconfig_vlan11_ipv6="inet6 fd9c:bee6:9d33:11::2 prefixlen 64" ifconfig_vlan30="vlan 30 vlandev lagg0 mtu 9000" ifconfig_vlan30_alias0="inet 10.21.30.3/24" ifconfig_vlan30_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" ifconfig_vlan34="vlan 34 vlandev lagg1 mtu 9000" ifconfig_vlan34_alias0="inet 10.21.34.2/24" ifconfig_vlan34_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64" ifconfig_vlan35="vlan 35 vlandev lagg1 mtu 1500" ifconfig_vlan35_alias0="inet 10.21.35.3/24" ifconfig_vlan35_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" defaultrouter="10.21.34.1" ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::1" 23:46:03 root@filer:/root/ ifconfig ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 hwaddr bc:5f:f4:c7:d5:58 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 hwaddr bc:5f:f4:c7:d5:56 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=5<MASTER,ACTIVE> laggport: igb0 flags=0<> groups: lagg media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7 laggproto failover lagghash l2,l3,l4 laggport: ix1 flags=5<MASTER,ACTIVE> laggport: igb1 flags=0<> groups: lagg media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fd9c:bee6:9d33:10::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8 inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255 groups: vlan vlan: 10 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fd9c:bee6:9d33:11::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9 inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255 groups: vlan vlan: 11 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255 groups: vlan vlan: 30 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan34: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255 groups: vlan vlan: 34 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan35: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255 groups: vlan vlan: 35 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 23:53:21 root@filer:/root/ netstat -I vlan10 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll vlan1 9000 <Link#8> a0:36:9f:72:35:94 1957078 0 0 2672967 6 0 vlan1 - fd9c:bee6:9d3 fd9c:bee6:9d33:10 451401 - - 372777 - - vlan1 - fe80::%vlan10 fe80::a236:9fff:f 0 - - 4 - - vlan1 - 10.21.10.0/24 10.21.10.2 1505540 - - 2301252 - - 23:53:25 root@filer:/root/ netstat -I lagg0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll lagg0 9000 <Link#6> a0:36:9f:72:35:94 6043751 0 0 4156297 15 0 lagg0 - fe80::%lagg0/ fe80::a236:9fff:f 0 - - 3 - - 23:53:31 root@filer:/root/ netstat -I ix0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 9000 <Link#1> a0:36:9f:72:35:94 6043052 0 0 4156335 0 0 No seemingly relevant warnings or messages in dmesg or /var/log/messages. 23:56:26 root@filer:/root/ dmesg | grep "igb\|ix\|lagg" acpi0: Power Button (fixed) ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe020-0xe03f mem 0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix0: using 4 rx queues 4 tx queues ix0: Using MSIX interrupts with 5 vectors ix0: allocated for 4 queues ix0: allocated for 4 rx queues ix0: Ethernet address: a0:36:9f:72:35:94 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe000-0xe01f mem 0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1 ix1: using 2048 tx descriptors and 2048 rx descriptors ix1: msix_init qsets capped at 16 ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix1: using 4 rx queues 4 tx queues ix1: Using MSIX interrupts with 5 vectors ix1: allocated for 4 queues ix1: allocated for 4 rx queues ix1: Ethernet address: a0:36:9f:72:35:96 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5 igb0: attach_pre capping queues at 4 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 4 igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb0: using 4 rx queues 4 tx queues igb0: Using MSIX interrupts with 5 vectors igb0: allocated for 4 tx_queues igb0: allocated for 4 rx_queues igb0: Ethernet address: bc:5f:f4:c7:d5:58 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9 igb1: attach_pre capping queues at 4 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 4 igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb1: using 4 rx queues 4 tx queues igb1: Using MSIX interrupts with 5 vectors igb1: allocated for 4 tx_queues igb1: allocated for 4 rx_queues igb1: Ethernet address: bc:5f:f4:c7:d5:56 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: link state changed to UP lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to prevent IPv6 address scope violation. lagg0: link state changed to UP lagg0: IPv6 addresses on igb0 have been removed before adding it as a member to prevent IPv6 address scope violation. ix1: link state changed to UP lagg1: IPv6 addresses on ix1 have been removed before adding it as a member to prevent IPv6 address scope violation. lagg1: link state changed to UP lagg1: IPv6 addresses on igb1 have been removed before adding it as a member to prevent IPv6 address scope violation. ix0: link state changed to DOWN lagg0: link state changed to DOWN ix1: link state changed to DOWN lagg1: link state changed to DOWN ix0: link state changed to UP lagg0: link state changed to UP ix1: link state changed to UP lagg1: link state changed to UP igb0: link state changed to UP igb1: link state changed to UP I'll perform some further tests over the weekend, see if failing over to the 1gbit igb interfaces or testing against a non-vlan-tagged interface makes any difference. Let me know if any further info could be useful. Kind regards, Nick -- You are receiving this mail because: You are the assignee for the bug.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-234624-227>