From owner-freebsd-stable@freebsd.org Wed Oct 12 12:07:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB9C9C0EAFD for ; Wed, 12 Oct 2016 12:07:23 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-lf0-f49.google.com (mail-lf0-f49.google.com [209.85.215.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C07939C for ; Wed, 12 Oct 2016 12:07:23 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-lf0-f49.google.com with SMTP id x79so74525093lff.0 for ; Wed, 12 Oct 2016 05:07:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=tnbnF+N97MKQZJlClhdG556ibwmuS29jsy8bIiIHUsw=; b=knIDDxXbf5OsPQbR2aq1B4IeX4e/dc1VWRtLSH668zVPVuk4ThzYd+WiFq/zdTB1y+ xO1xfwjnVjMEloCPhjmzQsiwXdrzzNkrjK81ygSE0UWQLF2NXebQlWiivn7n3bRgjs27 19755Xyp4RBDxxFAyXZdcRdpQZ2FIWkhDY+Ed26dChv3gJzEzZA33AHQIivtBCUqIaSP ExG+HrZ2pTvAzs76GxXf0Slmg0rVRgszgDeYwz8HngS69NhXugmTfHvfbsPTGhkYXw6y NOzJjDjr3IDGxnueI6G8VkNEvU+RDoHc3yLtMayydGTkVGt9kxvv5I3OrPqm2OhpjlDo RKJg== X-Gm-Message-State: AA6/9RlK9iS8OD4fB4Xx6Z2nqL4oLYccMxEPHVoxZSUeIQZ3CGcVdawL8WpqypxbBcq/Mg== X-Received: by 10.25.76.7 with SMTP id z7mr804771lfa.126.1476274035012; Wed, 12 Oct 2016 05:07:15 -0700 (PDT) Received: from [10.100.64.21] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id p3sm2069434lfe.1.2016.10.12.05.07.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Oct 2016 05:07:14 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <20161012084045.GA57714@zxy.spb.ru> <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: Date: Wed, 12 Oct 2016 14:06:59 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161012095233.GC57714@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 12:07:24 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0 Content-Type: multipart/mixed; boundary="S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: Subject: Re: 11.0 stuck on high network load References: <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <20161012084045.GA57714@zxy.spb.ru> <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> In-Reply-To: <20161012095233.GC57714@zxy.spb.ru> --S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 10/12/16 11:52 AM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 11:42:38AM +0200, Julien Charbon wrote: >> On 10/12/16 11:29 AM, Slawa Olhovchenkov wrote: >>> On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote: >>> >>>>> if INP_WLOCK is like spinlock -- this is dead lock. >>>>> if INP_WLOCK is like mutex -- thread1 resheduled. >>>> >>>> Thanks, I understand you question now. No an interrupt cannot bypa= ss a >>>> lock: Here INP_WLOCK is like mutex -- thread1 resheduled. >>> >>> Thanks, nice. >>> >>>>>>> As I remeber race created by call tcp_twstart() at time of end >>>>>>> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected >>>>>>> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in t= cp_twstart() >>>>>> >>>>>> Exactly, thus the current fix is: If you already have the INP_DR= OPPED >>>>>> flag set you are not allowed to call tcp_twstart(), actually it is= a >>>>>> good candidate for a new INVARIANT. Let me add that. >>>>>> >>>>>>> After check source code I am found invocation of tcp_twstart() in= >>>>>>> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c, >>>>>>> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c= =2E >>>>>>> >>>>>>> Invocation from sys/netinet/tcp_stacks/fastpath.c and >>>>>>> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and = now >>>>>>> will be OK. >>>>>>> >>>>>>> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and >>>>>>> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see indepe= nded >>>>>>> INP_WLOCK. Is this OK? >>>>>>> >>>>>>> Can be thread A wants do_peer_close() directed from chelsio IRQ >>>>>>> handler, bypass tcp_input()? >>>>>> >>>>>> If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and >>>>>> t4_cpl_io.c before calling tcp_twstart(). >>>>> >>>>> Yes, and you remeber: sys/netinet/tcp_subr.c >>>>> >>>>> 1535 struct tcpcb * >>>>> 1536 tcp_close(struct tcpcb *tp) >>>>> 1537 { >>>>> ... >>>>> 1569 INP_WUNLOCK(inp); >>>>> 1570 ACCEPT_LOCK(); >>>>> 1571 SOCK_LOCK(so); >>>>> 1572 so->so_state &=3D ~SS_PROTOREF; >>>>> 1573 sofree(so); >>>>> 1574 return (NULL); >>>>> >>>>> sofree() call tcp_usr_detach() and in tcp_usr_detach() we have >>>>> unexpected INP_TIMEWAIT. >>>> >>>> I see, thus just for the context: The TCP stack in sys/dev/cxgb* i= s a >>>> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a >>>> separate/side TCP stack that is used only with TCP_OFFLOAD option. >>>> >>>> This TOE TCP stack actually has its own set of detach()/input() >>>> functions and seems to check INP_DROPPED flag properly. I guess @np= >>>> check fixes in socket TCP stack and decides which one can also impac= t >>>> the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack, = some >>>> are only in TOE TCP stack. >>> >>> I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TO= E >>> TCP stack and impact this to >>> tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path. >> >> I see, I expect no problem on this side as tcp_timer_2msl() checks th= e >> INP_TIMEWAIT flag and do not call tcp_close() if set. >=20 > I am about case when at time of first INP_WUNLOCK() tcp_timer_2msl() > don't see INP_TIMEWAIT, call tcp_close(), tcp_close() do INP_WUNLOCK() > and now Chelsio TOE take INP_WLOCK, do tcp_twstart() and set > INP_TIMEWAIT. After this tcp_timer_2msl resume and have unexpected > INP_TIMEWAIT in tcp_usr_detach(). Sure, basically the same bug that in classic TCP stack. If you think it can happen, send an email describing that to np@ and he will check and fix that. He is a TOE TCP stack expert and I am not. In all cases, if this issue is possible in TOE TCP stack context, the patch will be straightforward: If the INP_DROPPED flag is set do not call tcp_twstart(= ). The current patch focuses only on the classic TCP stack. -- Julien --S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD-- --QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX/idwAAoJEKVlQ5Je6dhxHcoIALKEpseITVVpDlpMkteujLge nKwhtpaTyxAh1S7CDbvuGoHSDcvmy8ajHW928lD+Y7zIT325Zk2LCBjxfNqcMoRz rTXrFcph7nhEhHawEXCeKHhxP3VLmy76axfm13SG9stT0vWYOFwbdiEF1BiGrIZb 7Qu3JPumAGsW8CGPlhFu1kTXx7msVCxnCBC4N84UmhU/oVL6oKKyDlvNZ7barKXM kOknO5B2JDujT/WispMGtcKjqwLbtrAlgustfPGBzxSKpMw4DyUT+ps8yZRY/Zyn T8RduPXnvzwvTY5CufUoc3nYySvb6U8Nbxk9fmrlUsTNFIvkHlBs5WW3ezwgJPY= =uWUU -----END PGP SIGNATURE----- --QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0--