From owner-freebsd-net@freebsd.org Fri Dec 27 22:01:06 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1304E1D0C1A for ; Fri, 27 Dec 2019 22:01:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47l1411p7Mz4cXb; Fri, 27 Dec 2019 22:01:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f182.google.com with SMTP id j26so28077897ljc.12; Fri, 27 Dec 2019 14:01:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=pTyyU9OXqIJSoBS/qYtTEGUftk9CKzcBCr6YHgbctvc=; b=Bhd0H81fIrF5tJw3lnN2fhXvJLbbYSrN9XNKzfji7fH3VUdHBpbSYVBRUhxcTP1Quq NEk9rsHw+X/KDF9J4GQI5qcq+lS9Q6QkUKKL0XBvzdKyNpl/olXTcIPxuZIapt60sZxY D0/LgFh8TQCHZBMg2sXKo9sShrl3ZA8svV+CfE4AXYvM/irxTx/rEUCZihal0Xs7d6jp 2TSUCjBFEe69NJdvmeeIf0yfiZ8A3lK5Wc1I/ZpN2nBIv60VSvm25vu8a1p+ATXYjmZl iQw6Vv/Ic+HolXohJLCWAFGQSY8pxaeWSwE15r14YWmsHVI0TCRWhRPqU7t4B8otXSoV sH3A== X-Gm-Message-State: APjAAAUXVBRp8/YYzpwTwkMIYS0ZwGbS+OR2G8E6kczWElQl/MfJoB6d Cysqndcd6hP7S450//PDocrv+DqD X-Google-Smtp-Source: APXvYqy9BiS6ijJ6bZ+y3LBM/l8LVvjaIbk87Co+e0ioe9BedbjkyS0oIFb4iZilYRVu7pmm4fqJJg== X-Received: by 2002:a2e:86c4:: with SMTP id n4mr29400546ljj.97.1577484062682; Fri, 27 Dec 2019 14:01:02 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id p136sm15289531lfa.8.2019.12.27.14.01.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Dec 2019 14:01:01 -0800 (PST) Subject: Re: vmx: strange issue, related to to tso? To: Vincenzo Maffione Cc: freebsd-net References: <67dc1ce9-274c-7e70-30dc-97e2d5767237@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <963e3042-90b4-4de2-e18c-3e29627a25a9@FreeBSD.org> Date: Sat, 28 Dec 2019 00:01:00 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47l1411p7Mz4cXb X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.182 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.11 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_TLS_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.11)[ip: (-0.50), ipnet: 209.85.128.0/17(-3.11), asn: 15169(-1.88), country: US(-0.05)]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[182.208.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[182.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 22:01:06 -0000 On 27/12/2019 15:34, Vincenzo Maffione wrote: > It may be useful to check what happens if you replace the vmx0 interface with an > em0. > In this way you would know if the issue is vmx-specific or not. I'll put this on my to-do, can't test right now. But one thing I noticed when comparing the TCP control block of the connection before and after the "TSO dance" is that TF_TSO gets cleared after any outgoing traffic while TSO is disabled on the interface. And the flag does not come back after TSO is reenabled. Any new connections get the flag, of course. So, I indeed suspect that there is a problem with vmx TSO. As another data point, an older system from before vmx->iflib conversion does not exhibit the problem. > Il giorno gio 26 dic 2019 alle ore 20:04 Andriy Gapon > ha scritto: > > > Maybe someone would have any pointers for me with the following problem. > This happens with CURRENT as of the beginning of September. > I connect via ssh to a VM running on VMware, it has a single vmx0 interface. > The problem is that when I print a moderately large amount of text to the > terminal (e.g., tail -100 /var/log/messages) I literally see it printed in > chunks with noticeable pauses between chunks.  It takes several seconds for all > lines to get shown.  This happens every time I do it. > There is an interesting twist.  If I disable TSO with ifconfig vmx0 -tso and > print the same output in the same ssh session, then the output is smooth and > fast as I would expect it.  The lines scroll by almost instantly. > If then I re-enable TSO and again produce the same output in the same ssh, then > it is still fast. > > It appears that the TCP connection gets tuned to some very sub-optimal > parameters when TSO is enabled.  When I disable TSO, the parameters get re-tuned > to better values and the values stick when I re-enable TSO. > This is just a conjecture, of course. > > I have some tcpdump captures, but I do not see anything that would really stand > out.  One difference is that in the slow case only "full sized" packets are sent > while in the fast case there are shorter packets with push flag. > > Some packets for the slow case: >  00:00:00.453202 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 37:1485, ack 36, win 128, options [nop,nop,TS val 1403195134 ecr 4966311], > length 1448 >  00:00:00.096859 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 1485, > win 1026, options [nop,nop,TS val 4966864 ecr 1403195134], length 0 >  00:00:00.442963 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 1485:2933, ack 36, win 128, options [nop,nop,TS val 1403195664 ecr 4966864], > length 1448 >  00:00:00.092677 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 2933, > win 1026, options [nop,nop,TS val 4967400 ecr 1403195664], length 0 >  00:00:00.437336 IP 10.180.106.180.22 > 10.180.1.29.25490: Flags [.], seq > 2933:4381, ack 36, win 128, options [nop,nop,TS val 1403196194 ecr 4967400], > length 1448 >  00:00:00.097190 IP 10.180.1.29.25490 > 10.180.106.180.22: Flags [.], ack 4381, > win 1026, options [nop,nop,TS val 4967934 ecr 1403196194], length 0 > > Some packets after the TSO dance: >  00:00:00.000450 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 4077:5525, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 1448 >  00:00:00.000016 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 5525:6097, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 572 >  00:00:00.000009 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 5525, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 >  00:00:00.000303 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 6097:7545, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 1448 >  00:00:00.000019 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 7545:8117, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 572 >  00:00:00.000013 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 7545, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 >  00:00:00.000162 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [.], seq > 8117:9565, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 1448 >  00:00:00.000012 IP 10.180.106.180.22 > 10.180.1.29.25369: Flags [P.], seq > 9565:10137, ack 36, win 128, options [nop,nop,TS val 2124310129 ecr 21706510], > length 572 >  00:00:00.000007 IP 10.180.1.29.25369 > 10.180.106.180.22: Flags [.], ack 9565, > win 1003, options [nop,nop,TS val 21706510 ecr 2124310129], length 0 > > What else can I examine to debug the problem further? > Thank you! > -- > Andriy Gapon > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > -- Andriy Gapon