From owner-freebsd-stable@freebsd.org Wed Oct 12 09:29:47 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4DCBC0EE98 for ; Wed, 12 Oct 2016 09:29:47 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8317BB0A; Wed, 12 Oct 2016 09:29:47 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1buFr7-000I2l-Dr; Wed, 12 Oct 2016 12:29:45 +0300 Date: Wed, 12 Oct 2016 12:29:45 +0300 From: Slawa Olhovchenkov To: Julien Charbon Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Subject: Re: 11.0 stuck on high network load Message-ID: <20161012092945.GB57714@zxy.spb.ru> References: <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <20161012084045.GA57714@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 09:29:47 -0000 On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote: > > if INP_WLOCK is like spinlock -- this is dead lock. > > if INP_WLOCK is like mutex -- thread1 resheduled. > > Thanks, I understand you question now. No an interrupt cannot bypass a > lock: Here INP_WLOCK is like mutex -- thread1 resheduled. Thanks, nice. > >>> As I remeber race created by call tcp_twstart() at time of end > >>> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected > >>> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp_twstart() > >> > >> Exactly, thus the current fix is: If you already have the INP_DROPPED > >> flag set you are not allowed to call tcp_twstart(), actually it is a > >> good candidate for a new INVARIANT. Let me add that. > >> > >>> After check source code I am found invocation of tcp_twstart() in > >>> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c, > >>> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c. > >>> > >>> Invocation from sys/netinet/tcp_stacks/fastpath.c and > >>> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and now > >>> will be OK. > >>> > >>> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and > >>> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independed > >>> INP_WLOCK. Is this OK? > >>> > >>> Can be thread A wants do_peer_close() directed from chelsio IRQ > >>> handler, bypass tcp_input()? > >> > >> If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and > >> t4_cpl_io.c before calling tcp_twstart(). > > > > Yes, and you remeber: sys/netinet/tcp_subr.c > > > > 1535 struct tcpcb * > > 1536 tcp_close(struct tcpcb *tp) > > 1537 { > > ... > > 1569 INP_WUNLOCK(inp); > > 1570 ACCEPT_LOCK(); > > 1571 SOCK_LOCK(so); > > 1572 so->so_state &= ~SS_PROTOREF; > > 1573 sofree(so); > > 1574 return (NULL); > > > > sofree() call tcp_usr_detach() and in tcp_usr_detach() we have > > unexpected INP_TIMEWAIT. > > I see, thus just for the context: The TCP stack in sys/dev/cxgb* is a > TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a > separate/side TCP stack that is used only with TCP_OFFLOAD option. > > This TOE TCP stack actually has its own set of detach()/input() > functions and seems to check INP_DROPPED flag properly. I guess @np > check fixes in socket TCP stack and decides which one can also impact > the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack, some > are only in TOE TCP stack. I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TOE TCP stack and impact this to tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path.