From owner-freebsd-net@freebsd.org Fri Dec 27 13:34:50 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 871A21ED444 for ; Fri, 27 Dec 2019 13:34:50 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47knqt2l25z46WV; Fri, 27 Dec 2019 13:34:50 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id 4B1E911B00; Fri, 27 Dec 2019 13:34:50 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-qk1-f179.google.com with SMTP id x129so21511004qke.8; Fri, 27 Dec 2019 05:34:50 -0800 (PST) X-Gm-Message-State: APjAAAWt2MaFUP0emGqpZfq8fjo8NlMV2jKnYJScnSYgXp2I53G1E+Lr 57BtSayLcwVUjVIKCQtaba4fSbfZzkVFNqmVcFo= X-Google-Smtp-Source: APXvYqzj9Uplacb7ITQNwy7w7QnP+O14YBbf2Zvy5hRkoGSHZnxNauGugRKYTZikyGRyYS85raJwgWScrimXDuo4BRE= X-Received: by 2002:a05:620a:782:: with SMTP id 2mr43748174qka.169.1577453689538; Fri, 27 Dec 2019 05:34:49 -0800 (PST) MIME-Version: 1.0 References: <67dc1ce9-274c-7e70-30dc-97e2d5767237@FreeBSD.org> In-Reply-To: <67dc1ce9-274c-7e70-30dc-97e2d5767237@FreeBSD.org> From: Vincenzo Maffione Date: Fri, 27 Dec 2019 14:34:38 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: vmx: strange issue, related to to tso? To: Andriy Gapon Cc: freebsd-net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 13:34:50 -0000 It may be useful to check what happens if you replace the vmx0 interface with an em0. In this way you would know if the issue is vmx-specific or not. Cheers, Vincenzo Il giorno gio 26 dic 2019 alle ore 20:04 Andriy Gapon ha scritto: > > Maybe someone would have any pointers for me with the following problem. > This happens with CURRENT as of the beginning of September. > I connect via ssh to a VM running on VMware, it has a single vmx0 > interface. > The problem is that when I print a moderately large amount of text to the > terminal (e.g., tail -100 /var/log/messages) I literally see it printed in > chunks with noticeable pauses between chunks. It takes several seconds > for all > lines to get shown. This happens every time I do it. > There is an interesting twist. If I disable TSO with ifconfig vmx0 -tso > and > print the same output in the same ssh session, then the output is smooth > and > fast as I would expect it. The lines scroll by almost instantly. > If then I re-enable TSO and again produce the same output in the same ssh, > then > it is still fast. > > It appears that the TCP connection gets tuned to some very sub-optimal > parameters when TSO is enabled. When I disable TSO, the parameters get > re-tuned > to better values and the values stick when I re-enable TSO. > This is just a conjecture, of course. > > I have some tcpdump captures, but I do not see anything that would really > stand > out. One difference is that in the slow case only "full sized" packets > are sent > while in the fast case there are shorter packets with push flag. > > Some packets for the slow case: > 00:00:00.453202 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 37:1485, ack 36, win 128, options [nop,nop,TS val 1403195134 ecr 4966311], > length 1448 > 00:00:00.096859 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack > 1485, > win 1026, options [nop,nop,TS val 4966864 ecr 1403195134], length 0 > 00:00:00.442963 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 1485:2933, ack 36, win 128, options [nop,nop,TS val 1403195664 ecr > 4966864], > length 1448 > 00:00:00.092677 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack > 2933, > win 1026, options [nop,nop,TS val 4967400 ecr 1403195664], length 0 > 00:00:00.437336 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 2933:4381, ack 36, win 128, options [nop,nop,TS val 1403196194 ecr > 4967400], > length 1448 > 00:00:00.097190 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack > 4381, > win 1026, options [nop,nop,TS val 4967934 ecr 1403196194], length 0 > > Some packets after the TSO dance: > 00:00:00.000450 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 4077:5525, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 1448 > 00:00:00.000016 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 5525:6097, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 572 > 00:00:00.000009 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack > 5525, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 > 00:00:00.000303 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 6097:7545, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 1448 > 00:00:00.000019 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 7545:8117, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 572 > 00:00:00.000013 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack > 7545, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 > 00:00:00.000162 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 8117:9565, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 1448 > 00:00:00.000012 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 9565:10137, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr > 21706510], > length 572 > 00:00:00.000007 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack > 9565, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 > > What else can I examine to debug the problem further? > Thank you! > -- > Andriy Gapon > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >