Date: Sat, 05 Jan 2019 00:06:25 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234624] FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Message-ID: <bug-234624-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234624 Bug ID: 234624 Summary: FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO Product: Base System Version: 12.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: misc Assignee: bugs@FreeBSD.org Reporter: freebsd@aoeu.uk Hi, Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performan= ce dropped significantly. Seems to be related to TSO, or at least disabling it resolves the problem for me. Setup is a physical FreeBSD 12 host connected= to a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit switch. 23:58:18 root@filer:/root/ uname -a FreeBSD filer.<censored> 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC= =20 amd64 iperf3 tests from physical FreeBSD to virtual Debian: 23:32:39 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.0 KBytes 572 Kbits/sec 8 17.5 KBytes [ 5] 1.00-2.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 2.00-3.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 3.00-4.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 4.00-5.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 5.00-6.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 6.00-7.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 7.00-8.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 8.00-9.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 9.00-10.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 411 KBytes 336 Kbits/sec 86 sender [ 5] 0.00-10.00 sec 376 KBytes 308 Kbits/sec recei= ver iperf Done. 23:32:51 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 3.00-4.00 sec 1.13 GBytes 9.71 Gbits/sec [ 5] 4.00-5.00 sec 1.13 GBytes 9.73 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.84 Gbits/sec recei= ver iperf Done. Turning off TSO: 23:33:17 root@filer:/root/ ifconfig ix0 -tso Re-running iperf3: 23:33:57 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec recei= ver iperf Done. 23:34:12 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.14 GBytes 9.81 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec recei= ver iperf Done. Network configuration is not entirely trivial. From /etc/rc.conf (ipv6 glo= bal addresses censored): ipv6_activate_all_interfaces=3D"YES" ifconfig_igb0=3D"up mtu 9000" ifconfig_igb1=3D"up mtu 9000" ifconfig_ix0=3D"up mtu 9000" ifconfig_ix1=3D"up mtu 9000" cloned_interfaces=3D"lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35" ifconfig_lagg0=3D"laggproto failover laggport ix0 laggport igb0" ifconfig_lagg1=3D"laggproto failover laggport ix1 laggport igb1" ifconfig_vlan10=3D"vlan 10 vlandev lagg0 mtu 9000" ifconfig_vlan10_alias0=3D"inet 10.21.10.2/24" ifconfig_vlan10_ipv6=3D"inet6 fd9c:bee6:9d33:10::2 prefixlen 64" ifconfig_vlan11=3D"vlan 11 vlandev lagg1 mtu 9000" ifconfig_vlan11_alias0=3D"inet 10.21.11.2/24" ifconfig_vlan11_ipv6=3D"inet6 fd9c:bee6:9d33:11::2 prefixlen 64" ifconfig_vlan30=3D"vlan 30 vlandev lagg0 mtu 9000" ifconfig_vlan30_alias0=3D"inet 10.21.30.3/24" ifconfig_vlan30_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" ifconfig_vlan34=3D"vlan 34 vlandev lagg1 mtu 9000" ifconfig_vlan34_alias0=3D"inet 10.21.34.2/24" ifconfig_vlan34_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64" ifconfig_vlan35=3D"vlan 35 vlandev lagg1 mtu 1500" ifconfig_vlan35_alias0=3D"inet 10.21.35.3/24" ifconfig_vlan35_ipv6=3D"inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" defaultrouter=3D"10.21.34.1" ipv6_defaultrouter=3D"2a01:xxxx:xxxx:xxxx::1" 23:46:03 root@filer:/root/ ifconfig ix0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> ix1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> igb0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 hwaddr bc:5f:f4:c7:d5:58 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> igb1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 hwaddr bc:5f:f4:c7:d5:56 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 90= 00 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=3D5<MASTER,ACTIVE> laggport: igb0 flags=3D0<> groups: lagg media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> lagg1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 90= 00 =20=20=20=20=20=20=20 options=3De524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7 laggproto failover lagghash l2,l3,l4 laggport: ix1 flags=3D5<MASTER,ACTIVE> laggport: igb1 flags=3D0<> groups: lagg media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vlan10: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9= 000 options=3D600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fd9c:bee6:9d33:10::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8 inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255 groups: vlan vlan: 10 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vlan11: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9= 000 options=3D600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fd9c:bee6:9d33:11::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9 inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255 groups: vlan vlan: 11 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vlan30: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9= 000 options=3D600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255 groups: vlan vlan: 30 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vlan34: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9= 000 options=3D600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255 groups: vlan vlan: 34 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vlan35: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 options=3D600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255 groups: vlan vlan: 35 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> 23:53:21 root@filer:/root/ netstat -I vlan10 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll vlan1 9000 <Link#8> a0:36:9f:72:35:94 1957078 0 0 2672967 = 6 0 vlan1 - fd9c:bee6:9d3 fd9c:bee6:9d33:10 451401 - - 372777 = - - vlan1 - fe80::%vlan10 fe80::a236:9fff:f 0 - - 4 = - - vlan1 - 10.21.10.0/24 10.21.10.2 1505540 - - 2301252 = - - 23:53:25 root@filer:/root/ netstat -I lagg0=20 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll lagg0 9000 <Link#6> a0:36:9f:72:35:94 6043751 0 0 4156297 = 15 0 lagg0 - fe80::%lagg0/ fe80::a236:9fff:f 0 - - 3 = - - 23:53:31 root@filer:/root/ netstat -I ix0=20=20 Name Mtu Network Address Ipkts Ierrs Idrop Opkts O= errs Coll ix0 9000 <Link#1> a0:36:9f:72:35:94 6043052 0 0 4156335 = 0 0 No seemingly relevant warnings or messages in dmesg or /var/log/messages. 23:56:26 root@filer:/root/ dmesg | grep "igb\|ix\|lagg" acpi0: Power Button (fixed) ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe020-0xe03f mem 0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix0: using 4 rx queues 4 tx queues ix0: Using MSIX interrupts with 5 vectors ix0: allocated for 4 queues ix0: allocated for 4 rx queues ix0: Ethernet address: a0:36:9f:72:35:94 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe000-0xe01f mem 0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1 ix1: using 2048 tx descriptors and 2048 rx descriptors ix1: msix_init qsets capped at 16 ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix1: using 4 rx queues 4 tx queues ix1: Using MSIX interrupts with 5 vectors ix1: allocated for 4 queues ix1: allocated for 4 rx queues ix1: Ethernet address: a0:36:9f:72:35:96 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5 igb0: attach_pre capping queues at 4 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 4 igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb0: using 4 rx queues 4 tx queues igb0: Using MSIX interrupts with 5 vectors igb0: allocated for 4 tx_queues igb0: allocated for 4 rx_queues igb0: Ethernet address: bc:5f:f4:c7:d5:58 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9 igb1: attach_pre capping queues at 4 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 4 igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb1: using 4 rx queues 4 tx queues igb1: Using MSIX interrupts with 5 vectors igb1: allocated for 4 tx_queues igb1: allocated for 4 rx_queues igb1: Ethernet address: bc:5f:f4:c7:d5:56 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: link state changed to UP lagg0: IPv6 addresses on ix0 have been removed before adding it as a member= to prevent IPv6 address scope violation. lagg0: link state changed to UP lagg0: IPv6 addresses on igb0 have been removed before adding it as a membe= r to prevent IPv6 address scope violation. ix1: link state changed to UP lagg1: IPv6 addresses on ix1 have been removed before adding it as a member= to prevent IPv6 address scope violation. lagg1: link state changed to UP lagg1: IPv6 addresses on igb1 have been removed before adding it as a membe= r to prevent IPv6 address scope violation. ix0: link state changed to DOWN lagg0: link state changed to DOWN ix1: link state changed to DOWN lagg1: link state changed to DOWN ix0: link state changed to UP lagg0: link state changed to UP ix1: link state changed to UP lagg1: link state changed to UP igb0: link state changed to UP igb1: link state changed to UP I'll perform some further tests over the weekend, see if failing over to the 1gbit igb interfaces or testing against a non-vlan-tagged interface makes a= ny difference. Let me know if any further info could be useful. Kind regards, Nick --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-234624-227>