From owner-freebsd-net@freebsd.org Fri Jun 21 21:23:58 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9226A15BF920 for ; Fri, 21 Jun 2019 21:23:58 +0000 (UTC) (envelope-from igor-fbsdnet@grinchenko.org) Received: from sun.grinchenko.org (sun.grinchenko.org [96.78.165.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CC46681683; Fri, 21 Jun 2019 21:23:56 +0000 (UTC) (envelope-from igor-fbsdnet@grinchenko.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=grinchenko.org; s=myselector; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=OKU7amMvdfCAcr7ZmnVP3NqseEXIMq/aiF12ZCCdsSQ=; b=bTK3J7LGGkj7jjxvDLhJCPZB6 VVc9FyVwJLhVyKz/OnvGEB+1IbfXC7Meh5mCLCOryflvniQFRth21XvMeL0nf91kRdPd40V8xaOoO EY+k7ExGHxXzez2OFoeMM4OzZj; Received: from localhost ([127.0.0.1] helo=sun.grinchenko.org) by sun.grinchenko.org with esmtp (Exim 4.90_1 (FreeBSD)) (envelope-from ) id 1heR0k-00017Y-6q; Fri, 21 Jun 2019 14:23:54 -0700 Date: Fri, 21 Jun 2019 14:23:54 -0700 From: Igor Grinchenko To: Michael Tuexen Cc: freebsd-net@freebsd.org Subject: Re: unexpected TCP resets (RST) in 12.0-RELEASE Message-ID: <20190621212354.GP94573@sun.grinchenko.org> References: <20190621081941.GM94573@sun.grinchenko.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: CC46681683 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=grinchenko.org header.s=myselector header.b=bTK3J7LG; spf=pass (mx1.freebsd.org: domain of igor-fbsdnet@grinchenko.org designates 96.78.165.85 as permitted sender) smtp.mailfrom=igor-fbsdnet@grinchenko.org X-Spamd-Result: default: False [-4.12 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[grinchenko.org:s=myselector]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:96.78.165.85]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grinchenko.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-0.74)[ipnet: 96.64.0.0/11(-3.67), asn: 7922(0.03), country: US(-0.06)]; MX_GOOD(-0.01)[cached: sun.grinchenko.org]; DKIM_TRACE(0.00)[grinchenko.org:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.88)[-0.878,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Jun 2019 21:23:58 -0000 On Fri, Jun 21, 2019 at 12:26:46PM +0200, Michael Tuexen wrote: > > On 21. Jun 2019, at 10:19, Igor Grinchenko wrote: <..> > > > > It doesn't seem to matter what app that is, doesn't matter what event mechanism is used (kqueue or select). TCP stack just refuses to handshake from time to time. The rate of these increases with the rate of incoming connections. 12.0-RELEASE-p6 seems to be producing fewer of these, it could be due to the fix in https://www.freebsd.org/security/advisories/FreeBSD-EN-19:11.net.asc . While my guess is not very scientific, it seems to be related to the new epoch(9) based synchronization. A 11.2-RELEASE host, serving the same exact traffic, which I kept for baselining is rock-solid and doesn't produce TCP resets like these. > Hi Igor, > > could you do > sudo sysctl net.inet.tcp.log_debug=1 > on the host sending the RST segments and see if you get some messages in /var/log/messages. > Do you see these messages? If yes, what is logged? > > Do you have a way to reproduce this issue? > > Best regards > Michael > > Michael, first of all, thank you for the MIB pointer, very useful, it should really be promoted for bugreports/mailing list postings. Unfortunately, setting net.inet.tcp.log_debug=1 doesn't result in any log entries for this particular use case. I see logging about other legitimate resets but not that one. I have been trying to find an easy way to reproduce it, but haven't been able to do it yet. It is a moderately loaded server running FPM accepting and serving 300-500 requests(each on a new TCP connection) per second. It seems like I'm getting significantly fewer resets on more powerful hardware(Intel Gold 6140 vs E5-2670 v3), all other things being equal. Enabling net.inet.tcp.syncookies_only=1 doesn't seem to help, either. is there anything else I can run to get a better insight into what might be happening? -- Igor