Date: Wed, 12 Oct 2016 12:29:45 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Julien Charbon <jch@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Subject: Re: 11.0 stuck on high network load Message-ID: <20161012092945.GB57714@zxy.spb.ru> In-Reply-To: <f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org> References: <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org> <20161012084045.GA57714@zxy.spb.ru> <f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote:
> > if INP_WLOCK is like spinlock -- this is dead lock.
> > if INP_WLOCK is like mutex -- thread1 resheduled.
>
> Thanks, I understand you question now. No an interrupt cannot bypass a
> lock: Here INP_WLOCK is like mutex -- thread1 resheduled.
Thanks, nice.
> >>> As I remeber race created by call tcp_twstart() at time of end
> >>> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected
> >>> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp_twstart()
> >>
> >> Exactly, thus the current fix is: If you already have the INP_DROPPED
> >> flag set you are not allowed to call tcp_twstart(), actually it is a
> >> good candidate for a new INVARIANT. Let me add that.
> >>
> >>> After check source code I am found invocation of tcp_twstart() in
> >>> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c,
> >>> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c.
> >>>
> >>> Invocation from sys/netinet/tcp_stacks/fastpath.c and
> >>> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and now
> >>> will be OK.
> >>>
> >>> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and
> >>> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independed
> >>> INP_WLOCK. Is this OK?
> >>>
> >>> Can be thread A wants do_peer_close() directed from chelsio IRQ
> >>> handler, bypass tcp_input()?
> >>
> >> If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and
> >> t4_cpl_io.c before calling tcp_twstart().
> >
> > Yes, and you remeber: sys/netinet/tcp_subr.c
> >
> > 1535 struct tcpcb *
> > 1536 tcp_close(struct tcpcb *tp)
> > 1537 {
> > ...
> > 1569 INP_WUNLOCK(inp);
> > 1570 ACCEPT_LOCK();
> > 1571 SOCK_LOCK(so);
> > 1572 so->so_state &= ~SS_PROTOREF;
> > 1573 sofree(so);
> > 1574 return (NULL);
> >
> > sofree() call tcp_usr_detach() and in tcp_usr_detach() we have
> > unexpected INP_TIMEWAIT.
>
> I see, thus just for the context: The TCP stack in sys/dev/cxgb* is a
> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a
> separate/side TCP stack that is used only with TCP_OFFLOAD option.
>
> This TOE TCP stack actually has its own set of detach()/input()
> functions and seems to check INP_DROPPED flag properly. I guess @np
> check fixes in socket TCP stack and decides which one can also impact
> the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack, some
> are only in TOE TCP stack.
I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TOE
TCP stack and impact this to
tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161012092945.GB57714>