Date: Tue, 5 Nov 2019 22:44:10 +0100 From: "Jose G. Juanino" <jjuanino@gmail.com> To: freebsd-stable@freebsd.org Subject: High Send-Q values in netstat after upgrading to 12.1-RELEASE: system is unusable Message-ID: <20191105214410.xef4fwksfwqkjwx3@riemann.ust-global.com>
next in thread | raw e-mail | index | archive | help
Hi all,
today I have upgraded to 12.1-RELEASE from 12.0-RELEASE-p10 via
freebsd-update. The host is a VMware guest with vmx net interface. The
system runs with 12.0-RELEASE perfectly, with no issue, reliably for months.
I have boot environments enabled (thanks god) and therefore I am able to
switch between both 12.0 and 12.1 versions.
Just after boot the new 12.1 release, I noticed that my apache did not
serve pages with the previous speed, but rather the opposite: any web
operation from clients hangs.
After a single inspection in the FreeBSD 12.1 server, I see that Send-Q
in netstat is the root cause of the issue. While in 12.0 Send-Q is
almost always zero or close to zero in every socket, in 12.1 I get the
following (snipped output):
$ netstat -x -4
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime
tcp4 0 31848 XXXXXXXXXXXXX.http YYYYYYYYYYYY.57220 0 12 0 12 66350 33175 1 2048 0 48128 530800 265400 62.90 0.00 7069.49 0.00 0.00 1.09
tcp4 0 36 XXXXXXXXXXXXX.ssh YYYYYYYYYYYY.57189 0 1 0 0 66350 33175 1 2048 0 256 530800 265400 0.28 0.00 6836.19 0.00 0.00 0.00
As you can see, Send-Q is huge.
But after this revelation, I have to admit that I am stuck. I have no
idea whether the cause of this high Send-Q values are related with the
vmx driver, with the upgrade itself, or with something related with the
hypervisor.
Please, help me to make a deeper troubleshooting. As I have said before,
my boot environment let me to start cleanly with both versions.
Other relevant fact may be also the difference between netstat -m and
ifconfig outputs:
In 12.0 netstat -m outputs:
268/1517/1785 mbufs in use (current/cache/total)
260/766/1026/250998 mbuf clusters in use (current/cache/total/max)
260/499 mbuf+clusters out of packet secondary zone in use (current/cache)
0/2/2/125499 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/37184 9k jumbo clusters in use (current/cache/total/max)
0/0/0/20916 16k jumbo clusters in use (current/cache/total/max)
587K/1919K/2506K bytes allocated to network (current/cache/total)
while in 12.1 outputs:
572/1213/1785 mbufs in use (current/cache/total)
515/515/1030/250998 mbuf clusters in use (current/cache/total/max)
4/502 mbuf+clusters out of packet secondary zone in use (current/cache)
24/2/26/125499 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/37184 9k jumbo clusters in use (current/cache/total/max)
0/0/0/20916 16k jumbo clusters in use (current/cache/total/max)
1269K/1341K/2610K bytes allocated to network (current/cache/total)
In 12.0 ifconfig vmx0 outputs (ip address and mac obfuscated):
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether TTTTTTTTTTTTTTTTT
inet DD.DD.DD.DD netmask 0xffffff00 broadcast BB.BB.BB.BB
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
while in 12.1 outputs (notices JUMBO frames enabled):
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether TTTTTTTTTTTTTTTTT
inet DDDDDDDDDDDD netmask 0xffffff00 broadcast DDDDDDDDDDDDD
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Thanks in advanced, regards
--
Jose G. Juanino
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20191105214410.xef4fwksfwqkjwx3>