From owner-freebsd-stable@freebsd.org Mon Oct 10 14:29:44 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56009C0B253 for ; Mon, 10 Oct 2016 14:29:44 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1392B1BB; Mon, 10 Oct 2016 14:29:44 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1btbaH-000C3n-Ro; Mon, 10 Oct 2016 17:29:41 +0300 Date: Mon, 10 Oct 2016 17:29:41 +0300 From: Slawa Olhovchenkov To: Julien Charbon Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Subject: Re: 11.0 stuck on high network load Message-ID: <20161010142941.GV54003@zxy.spb.ru> References: <20160925124626.GI2840@zxy.spb.ru> <20160926172159.GA54003@zxy.spb.ru> <62453d9c-b1e4-1129-70ff-654dacea37f9@gmail.com> <20160928115909.GC54003@zxy.spb.ru> <20161006111043.GH54003@zxy.spb.ru> <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2016 14:29:44 -0000 On Mon, Oct 10, 2016 at 04:03:39PM +0200, Julien Charbon wrote: > > Hi Slawa, > > On 10/10/16 3:32 PM, Slawa Olhovchenkov wrote: > > On Mon, Oct 10, 2016 at 01:26:12PM +0200, Julien Charbon wrote: > >> On 10/6/16 1:10 PM, Slawa Olhovchenkov wrote: > >>> On Thu, Oct 06, 2016 at 09:28:06AM +0200, Julien Charbon wrote: > >>> > >>>> 2. thread1: In tcp_close() the inp is marked with INP_DROPPED flag, the > >>>> process continues and calls INP_WUNLOCK() here: > >>>> > >>>> https://github.com/freebsd/freebsd/blob/releng/11.0/sys/netinet/tcp_subr.c#L1568 > >>> > >>> Look also to sys/netinet/tcp_timewait.c:488 > >>> > >>> And check other locks from r160549 > >> > >> You are right, and here the a fix proposal for this issue: > >> > >> Fix a double-free when an inp transitions to INP_TIMEWAIT state after > >> having been dropped > >> https://reviews.freebsd.org/D8211 > >> > >> It basically enforces in_pcbdrop() logic in tcp_input(): A INP_DROPPED > >> inpcb should never be proceed further. > >> > >> Slawa, as you are the only one to reproduce this issue currently, could > >> test this patch? (And remove the temporary patch I did provided to you > >> before). > >> > >> I will wait for your tests results before pushing further. > >> > >> Thanks! > >> > >> diff --git a/sys/netinet/tcp_input.c b/sys/netinet/tcp_input.c > >> index c72f01f..37f27e0 100644 > >> --- a/sys/netinet/tcp_input.c > >> +++ b/sys/netinet/tcp_input.c > >> @@ -921,6 +921,16 @@ findpcb: > >> goto dropwithreset; > >> } > >> INP_WLOCK_ASSERT(inp); > >> + /* > >> + * While waiting for inp lock during the lookup, another thread > >> + * can have droppedt the inpcb, in which case we need to loop back > >> + * and try to find a new inpcb to deliver to. > >> + */ > >> + if (inp->inp_flags & INP_DROPPED) { > >> + INP_WUNLOCK(inp); > >> + inp = NULL; > >> + goto findpcb; > > > > Are you sure about this goto? > > Can this cause infinite loop by found same inpcb? > > May be drop packet is more correct? > > Good question: Infinite loop is not possible here, as the next TCP > hash lookup will return NULL or a fresh new and not dropped inp. You I am not expert in this api and don't see cause of this: I am assume hash lookup don't remove from hash returned args and I am don't see any removing of this inp. Why hash lookup don't return same inp? (assume this input patch interrupt callout code on the same CPU core). > can check the current other usages of goto findpcb in tcp_input(). The > rational here being: > > - Behavior before the patch: If the inp we found was deleted then goto > findpcb. > - Behavior after the patch: If the inp we found was deleted or dropped > then goto findpcb. > > I just prefer having the same behavior applied everywhere: If > tcp_input() loses the inp lock race and the inp was deleted or dropped > then retry to find a new inpcb to deliver to. > > But you are right dropping the packet here will also fix the issue. > > Then the review process becomes quite helpful because people can argue: > Dropping here is better because "blah", or goto findpcb is better > because "bluh", etc. And at the review end you have a nice final patch. > > https://reviews.freebsd.org/D8211 I am not sure, I am see to sys/netinet/in_pcb.h:#define INP_DROPPED 0x04000000 /* protocol drop flag */ and think this is a flag 'all packets must be droped'