From owner-freebsd-stable@freebsd.org Tue Nov 5 23:38:01 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 151CF1A6908 for ; Tue, 5 Nov 2019 23:38:01 +0000 (UTC) (envelope-from jjuanino@gmail.com) Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4775gq72v8z4CK0 for ; Tue, 5 Nov 2019 23:37:59 +0000 (UTC) (envelope-from jjuanino@gmail.com) Received: by mail-wm1-x336.google.com with SMTP id v3so1015170wmh.1 for ; Tue, 05 Nov 2019 15:37:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=T43JL8XSQ8S7babs2V/g/ybTVcUPELoXC7Ev3qHaNO8=; b=e4CgYISYOUHSmG8+Uoi2syJWMkR3ePj0V+9rNFN4KMC8rI1hX9gJVTXl1flLwsGdvi ugh0vV45y4zBrwu2R4eahqkhh3KvZuTS4ajpx8iscAOwA7XySn9Gx6hp97PoWgYFaGG9 8ADFPEd/lFGP4o2jHN8mKyKxsf4PoH9BXMthBFMjdufVIB2/VLvB4CdWSkx9mVIY7llO bkS6sfi3vLWswMKdhVQgeEljtKUtr205QCAGYwXNBk4w32KIA04nVmo7805fL6C7s3uC yipr08szSWzfQcj1k5NAWNhScQTOP3weI5kaVkaye1Wh+hhO3E9dSoumXPTY2PqApxs3 6I1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=T43JL8XSQ8S7babs2V/g/ybTVcUPELoXC7Ev3qHaNO8=; b=NRMiWuxKeCLT18Q/TgVBoym/LtjA5Ps2Q/XLZRy0ZwXBEsLe677Pf00Ruo0HQyOW/A dpv2WcU4OOkz+Fa7oOb+1o2XO1TAqdfHvBq3bn7+d/2Dne4JaEsGymN4zV27xgPr8bO7 hmrwhPnqY4JXYyt6xVONdRIK5+flZpQAVhXG2Qp2N7DDAJGCUGX+dLdSeQ6ArqOPPp5B /fg6aSrA/GfnX5TAFsbTJKOpcNelXLd4QE2lYTR54LgoGb3w3QWacT9IqijWheXs13Ex B+0Ho4Cnu8R1tlbDnr+21JNU5YeLx9h8UEnVuqVU8966CiWfHoZE0cDhXJfoIC6e2CSg 0KFg== X-Gm-Message-State: APjAAAU0vBdfIPOEUI6HtRcgh489cFiB8l7qlDj42OIdFMdlUu6eqC2l dUdVUKw/WxcujfrDjrtGchBK+gJD X-Google-Smtp-Source: APXvYqwYx9jiQ27k6gO7cm498jyRhycOPySpCy7gDXtN5P54CBT3vuI/1TyZ5zihKsW/PB6C4iT4iA== X-Received: by 2002:a1c:39c2:: with SMTP id g185mr1239835wma.88.1572997077994; Tue, 05 Nov 2019 15:37:57 -0800 (PST) Received: from riemann.ust-global.com (2.153.135.60.dyn.user.ono.com. [2.153.135.60]) by smtp.gmail.com with ESMTPSA id s13sm840789wmc.28.2019.11.05.15.37.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Nov 2019 15:37:57 -0800 (PST) Date: Wed, 6 Nov 2019 00:37:50 +0100 From: "Jose G. Juanino" To: freebsd-stable@freebsd.org Subject: Re: High Send-Q values in netstat after upgrading to 12.1-RELEASE: system is unusable Message-ID: <20191105233750.4nojttru53i2j6cc@riemann.ust-global.com> References: <20191105214410.xef4fwksfwqkjwx3@riemann.ust-global.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191105214410.xef4fwksfwqkjwx3@riemann.ust-global.com> X-Operating-System: FreeBSD 12.1-RELEASE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 4775gq72v8z4CK0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=e4CgYISY; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jjuanino@gmail.com designates 2a00:1450:4864:20::336 as permitted sender) smtp.mailfrom=jjuanino@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RECEIVED_SPAMHAUS_PBL(0.00)[60.135.153.2.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.00)[ip: (-9.38), ipnet: 2a00:1450::/32(-2.77), asn: 15169(-2.02), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[6.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2019 23:38:01 -0000 On Tuesday, November 05 at 22:44:10 CET, Jose G. Juanino wrote: > Hi all, > > today I have upgraded to 12.1-RELEASE from 12.0-RELEASE-p10 via > freebsd-update. The host is a VMware guest with vmx net interface. The > system runs with 12.0-RELEASE perfectly, with no issue, reliably for months. > > I have boot environments enabled (thanks god) and therefore I am able to > switch between both 12.0 and 12.1 versions. > > Just after boot the new 12.1 release, I noticed that my apache did not > serve pages with the previous speed, but rather the opposite: any web > operation from clients hangs. > > After a single inspection in the FreeBSD 12.1 server, I see that Send-Q > in netstat is the root cause of the issue. While in 12.0 Send-Q is > almost always zero or close to zero in every socket, in 12.1 I get the > following (snipped output): > > [ .... ] > Hi again, after doing a further research, I have noticed that in 12.0 version dev.vmx.0.txq0.hstats.tso_packets is always increasing (especially when doing large transfers), but in 12.1 is statically equal to zero, never increases. As workaround, I have disabled TCP segment offloading by setting the net.inet.tcp.tso=0 sysctl in 12.1 version and the host again performs almost as well as before in 12.0. I hope this helps others suffering the same issue. Regards -- Jose G. Juanino