From owner-freebsd-stable@freebsd.org Wed Nov 6 01:28:44 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81F701B10B3 for ; Wed, 6 Nov 2019 01:28:44 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail.karels.net (mail.karels.net [216.160.39.52]) by mx1.freebsd.org (Postfix) with ESMTP id 47787b3kbRz4Hpl for ; Wed, 6 Nov 2019 01:28:43 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail.karels.net (localhost [127.0.0.1]) by mail.karels.net (8.15.2/8.15.2) with ESMTP id xA61SfVr065464; Tue, 5 Nov 2019 19:28:41 -0600 (CST) (envelope-from mike@karels.net) Message-Id: <201911060128.xA61SfVr065464@mail.karels.net> To: "Jose G. Juanino" cc: freebsd-stable@freebsd.org From: Mike Karels Reply-to: mike@karels.net Subject: Re: High Send-Q values in netstat after upgrading to 12.1-RELEASE: system is unusable In-reply-to: Your message of Wed, 06 Nov 2019 00:37:50 +0100. <20191105233750.4nojttru53i2j6cc@riemann.ust-global.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <65462.1573003721.1@mail.karels.net> Content-Transfer-Encoding: quoted-printable Date: Tue, 05 Nov 2019 19:28:41 -0600 X-Rspamd-Queue-Id: 47787b3kbRz4Hpl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 216.160.39.52 as permitted sender) smtp.mailfrom=mike@karels.net X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[mike@karels.net]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.160.39.52]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[karels.net]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.86)[ip: (-6.23), ipnet: 216.160.0.0/15(-3.00), asn: 209(-0.02), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:209, ipnet:216.160.0.0/15, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2019 01:28:44 -0000 > On Tuesday, November 05 at 22:44:10 CET, Jose G. Juanino wrote: > > Hi all, > > = > > today I have upgraded to 12.1-RELEASE from 12.0-RELEASE-p10 via > > freebsd-update. The host is a VMware guest with vmx net interface. The > > system runs with 12.0-RELEASE perfectly, with no issue, reliably for m= onths. > > = > > I have boot environments enabled (thanks god) and therefore I am able = to > > switch between both 12.0 and 12.1 versions. > > = > > Just after boot the new 12.1 release, I noticed that my apache did not > > serve pages with the previous speed, but rather the opposite: any web > > operation from clients hangs. > > = > > After a single inspection in the FreeBSD 12.1 server, I see that Send-= Q > > in netstat is the root cause of the issue. While in 12.0 Send-Q is > > almost always zero or close to zero in every socket, in 12.1 I get the > > following (snipped output): > > > > [ .... ] > > > Hi again, after doing a further research, I have noticed that in 12.0 > version dev.vmx.0.txq0.hstats.tso_packets is always increasing > (especially when doing large transfers), but in 12.1 is statically equal > to zero, never increases. > As workaround, I have disabled TCP segment offloading by setting the > net.inet.tcp.tso=3D0 sysctl in 12.1 version and the host again performs > almost as well as before in 12.0. Have you tried disabling jumbo rather than TSO? If jumbo is enabled on this system, but not on the client (or next hop), all the segments sent from the server that are over 1500 bytes will be lost, and data will pile up in the send queue waiting for acknowledgement. > I hope this helps others suffering the same issue. > Regards > -- = > Jose G. Juanino Mike