Date: Thu, 26 Dec 2019 21:04:24 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: freebsd-net <freebsd-net@FreeBSD.org> Subject: vmx: strange issue, related to to tso? Message-ID: <67dc1ce9-274c-7e70-30dc-97e2d5767237@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
Maybe someone would have any pointers for me with the following problem. This happens with CURRENT as of the beginning of September. I connect via ssh to a VM running on VMware, it has a single vmx0 interface. The problem is that when I print a moderately large amount of text to the terminal (e.g., tail -100 /var/log/messages) I literally see it printed in chunks with noticeable pauses between chunks. It takes several seconds for all lines to get shown. This happens every time I do it. There is an interesting twist. If I disable TSO with ifconfig vmx0 -tso and print the same output in the same ssh session, then the output is smooth and fast as I would expect it. The lines scroll by almost instantly. If then I re-enable TSO and again produce the same output in the same ssh, then it is still fast. It appears that the TCP connection gets tuned to some very sub-optimal parameters when TSO is enabled. When I disable TSO, the parameters get re-tuned to better values and the values stick when I re-enable TSO. This is just a conjecture, of course. I have some tcpdump captures, but I do not see anything that would really stand out. One difference is that in the slow case only "full sized" packets are sent while in the fast case there are shorter packets with push flag. Some packets for the slow case: 00:00:00.453202 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq 37:1485, ack 36, win 128, options [nop,nop,TS val 1403195134 ecr 4966311], length 1448 00:00:00.096859 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 1485, win 1026, options [nop,nop,TS val 4966864 ecr 1403195134], length 0 00:00:00.442963 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq 1485:2933, ack 36, win 128, options [nop,nop,TS val 1403195664 ecr 4966864], length 1448 00:00:00.092677 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 2933, win 1026, options [nop,nop,TS val 4967400 ecr 1403195664], length 0 00:00:00.437336 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq 2933:4381, ack 36, win 128, options [nop,nop,TS val 1403196194 ecr 4967400], length 1448 00:00:00.097190 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 4381, win 1026, options [nop,nop,TS val 4967934 ecr 1403196194], length 0 Some packets after the TSO dance: 00:00:00.000450 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq 4077:5525, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 1448 00:00:00.000016 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq 5525:6097, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 572 00:00:00.000009 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 5525, win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 00:00:00.000303 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq 6097:7545, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 1448 00:00:00.000019 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq 7545:8117, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 572 00:00:00.000013 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 7545, win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 00:00:00.000162 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq 8117:9565, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 1448 00:00:00.000012 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq 9565:10137, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], length 572 00:00:00.000007 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 9565, win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 What else can I examine to debug the problem further? Thank you! -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?67dc1ce9-274c-7e70-30dc-97e2d5767237>