From owner-freebsd-bugs@freebsd.org Sat Jan 5 00:06:28 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEC57143F575 for ; Sat, 5 Jan 2019 00:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 88ABB8AEF3 for ; Sat, 5 Jan 2019 00:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4BA3F143F56F; Sat, 5 Jan 2019 00:06:27 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18009143F56E for ; Sat, 5 Jan 2019 00:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A8B778AEF1 for ; Sat, 5 Jan 2019 00:06:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id E13B713C47 for ; Sat, 5 Jan 2019 00:06:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x0506PDN006266 for ; Sat, 5 Jan 2019 00:06:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x0506PUL006257 for bugs@FreeBSD.org; Sat, 5 Jan 2019 00:06:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234624] FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Date: Sat, 05 Jan 2019 00:06:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd@aoeu.uk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jan 2019 00:06:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234624 Bug ID: 234624 Summary: FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Product: Base System Version: 12.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: misc Assignee: bugs@FreeBSD.org Reporter: freebsd@aoeu.uk Hi, Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performan= ce dropped significantly. Seems to be related to TSO, or at least disabling it resolves the problem for me. Setup is a physical FreeBSD 12 host connected= to a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit switch. 23:58:18 root@filer:/root/ uname -a FreeBSD filer. 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC= =20 amd64 iperf3 tests from physical FreeBSD to virtual Debian: 23:32:39 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.0 KBytes 572 Kbits/sec 8 17.5 KBytes [ 5] 1.00-2.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 2.00-3.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 3.00-4.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 4.00-5.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 5.00-6.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 6.00-7.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 7.00-8.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 8.00-9.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 9.00-10.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 411 KBytes 336 Kbits/sec 86 sender [ 5] 0.00-10.00 sec 376 KBytes 308 Kbits/sec recei= ver iperf Done. 23:32:51 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 3.00-4.00 sec 1.13 GBytes 9.71 Gbits/sec [ 5] 4.00-5.00 sec 1.13 GBytes 9.73 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.84 Gbits/sec recei= ver iperf Done. Turning off TSO: 23:33:17 root@filer:/root/ ifconfig ix0 -tso Re-running iperf3: 23:33:57 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec recei= ver iperf Done. 23:34:12 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.14 GBytes 9.81 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec recei= ver iperf Done. Network configuration is not entirely trivial. From /etc/rc.conf (ipv6 glo= bal addresses censored): ipv6_activate_all_interfaces=3D"YES" ifconfig_igb0=3D"up mtu 9000" ifconfig_igb1=3D"up mtu 9000" ifconfig_ix0=3D"up mtu 9000" ifconfig_ix1=3D"up mtu 9000" cloned_interfaces=3D"lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35" ifconfig_lagg0=3D"laggproto failover laggport ix0 laggport igb0" ifconfig_lagg1=3D"laggproto failover laggport ix1 laggport igb1" ifconfig_vlan10=3D"vlan 10 vlandev lagg0 mtu 9000" ifconfig_vlan10_alias0=3D"inet 10.21.10.2/24" ifconfig_vlan10_ipv6=3D"inet6 fd9c:bee6:9d33:10::2 prefixlen 64" ifconfig_vlan11=3D"vlan 11 vlandev lagg1 mtu 9000" ifconfig_vlan11_alias0=3D"inet 10.21.11.2/24" ifconfig_vlan11_ipv6=3D"inet6 fd9c:bee6:9d33:11::2 prefixlen 64" ifconfig_vlan30=3D"vlan 30 vlandev lagg0 mtu 9000" ifconfig_vlan30_alias0=3D"inet 10.21.30.3/24" ifconfig_vlan30_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" ifconfig_vlan34=3D"vlan 34 vlandev lagg1 mtu 9000" ifconfig_vlan34_alias0=3D"inet 10.21.34.2/24" ifconfig_vlan34_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64" ifconfig_vlan35=3D"vlan 35 vlandev lagg1 mtu 1500" ifconfig_vlan35_alias0=3D"inet 10.21.35.3/24" ifconfig_vlan35_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" defaultrouter=3D"10.21.34.1" ipv6_defaultrouter=3D"2a01:xxxx:xxxx:xxxx::1" 23:46:03 root@filer:/root/ ifconfig ix0: flags=3D8843 metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:94 media: Ethernet autoselect (10Gbase-Twinax ) status: active nd6 options=3D21 ix1: flags=3D8843 metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:96 media: Ethernet autoselect (10Gbase-Twinax ) status: active nd6 options=3D21 igb0: flags=3D8843 metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:94 hwaddr bc:5f:f4:c7:d5:58 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D21 igb1: flags=3D8843 metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:96 hwaddr bc:5f:f4:c7:d5:56 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D21 lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 lagg0: flags=3D8843 metric 0 mtu 90= 00 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:94 inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=3D5 laggport: igb0 flags=3D0<> groups: lagg media: Ethernet autoselect status: active nd6 options=3D21 lagg1: flags=3D8843 metric 0 mtu 90= 00 =20=20=20=20=20=20=20 options=3De524bb ether a0:36:9f:72:35:96 inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7 laggproto failover lagghash l2,l3,l4 laggport: ix1 flags=3D5 laggport: igb1 flags=3D0<> groups: lagg media: Ethernet autoselect status: active nd6 options=3D21 vlan10: flags=3D8843 metric 0 mtu 9= 000 options=3D600403 ether a0:36:9f:72:35:94 inet6 fd9c:bee6:9d33:10::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8 inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255 groups: vlan vlan: 10 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=3D21 vlan11: flags=3D8843 metric 0 mtu 9= 000 options=3D600403 ether a0:36:9f:72:35:96 inet6 fd9c:bee6:9d33:11::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9 inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255 groups: vlan vlan: 11 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21 vlan30: flags=3D8843 metric 0 mtu 9= 000 options=3D600403 ether a0:36:9f:72:35:94 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255 groups: vlan vlan: 30 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=3D21 vlan34: flags=3D8843 metric 0 mtu 9= 000 options=3D600403 ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255 groups: vlan vlan: 34 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21 vlan35: flags=3D8843 metric 0 mtu 1= 500 options=3D600403 ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255 groups: vlan vlan: 35 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21 23:53:21 root@filer:/root/ netstat -I vlan10 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll vlan1 9000 a0:36:9f:72:35:94 1957078 0 0 2672967 = 6 0 vlan1 - fd9c:bee6:9d3 fd9c:bee6:9d33:10 451401 - - 372777 = - - vlan1 - fe80::%vlan10 fe80::a236:9fff:f 0 - - 4 = - - vlan1 - 10.21.10.0/24 10.21.10.2 1505540 - - 2301252 = - - 23:53:25 root@filer:/root/ netstat -I lagg0=20 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll lagg0 9000 a0:36:9f:72:35:94 6043751 0 0 4156297 = 15 0 lagg0 - fe80::%lagg0/ fe80::a236:9fff:f 0 - - 3 = - - 23:53:31 root@filer:/root/ netstat -I ix0=20=20 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll ix0 9000 a0:36:9f:72:35:94 6043052 0 0 4156335 = 0 0 No seemingly relevant warnings or messages in dmesg or /var/log/messages. 23:56:26 root@filer:/root/ dmesg | grep "igb\|ix\|lagg" acpi0: Power Button (fixed) ix0: port 0xe020-0xe03f mem 0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix0: using 4 rx queues 4 tx queues ix0: Using MSIX interrupts with 5 vectors ix0: allocated for 4 queues ix0: allocated for 4 rx queues ix0: Ethernet address: a0:36:9f:72:35:94 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: port 0xe000-0xe01f mem 0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1 ix1: using 2048 tx descriptors and 2048 rx descriptors ix1: msix_init qsets capped at 16 ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix1: using 4 rx queues 4 tx queues ix1: Using MSIX interrupts with 5 vectors ix1: allocated for 4 queues ix1: allocated for 4 rx queues ix1: Ethernet address: a0:36:9f:72:35:96 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 igb0: port 0xb000-0xb01f mem 0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5 igb0: attach_pre capping queues at 4 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 4 igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb0: using 4 rx queues 4 tx queues igb0: Using MSIX interrupts with 5 vectors igb0: allocated for 4 tx_queues igb0: allocated for 4 rx_queues igb0: Ethernet address: bc:5f:f4:c7:d5:58 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: port 0xa000-0xa01f mem 0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9 igb1: attach_pre capping queues at 4 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 4 igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb1: using 4 rx queues 4 tx queues igb1: Using MSIX interrupts with 5 vectors igb1: allocated for 4 tx_queues igb1: allocated for 4 rx_queues igb1: Ethernet address: bc:5f:f4:c7:d5:56 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: link state changed to UP lagg0: IPv6 addresses on ix0 have been removed before adding it as a member= to prevent IPv6 address scope violation. lagg0: link state changed to UP lagg0: IPv6 addresses on igb0 have been removed before adding it as a membe= r to prevent IPv6 address scope violation. ix1: link state changed to UP lagg1: IPv6 addresses on ix1 have been removed before adding it as a member= to prevent IPv6 address scope violation. lagg1: link state changed to UP lagg1: IPv6 addresses on igb1 have been removed before adding it as a membe= r to prevent IPv6 address scope violation. ix0: link state changed to DOWN lagg0: link state changed to DOWN ix1: link state changed to DOWN lagg1: link state changed to DOWN ix0: link state changed to UP lagg0: link state changed to UP ix1: link state changed to UP lagg1: link state changed to UP igb0: link state changed to UP igb1: link state changed to UP I'll perform some further tests over the weekend, see if failing over to the 1gbit igb interfaces or testing against a non-vlan-tagged interface makes a= ny difference. Let me know if any further info could be useful. Kind regards, Nick --=20 You are receiving this mail because: You are the assignee for the bug.=