From owner-freebsd-stable@freebsd.org Wed Oct 12 09:42:47 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41D6BC0C32A for ; Wed, 12 Oct 2016 09:42:47 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B763C371 for ; Wed, 12 Oct 2016 09:42:46 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-f44.google.com with SMTP id o81so20408958wma.1 for ; Wed, 12 Oct 2016 02:42:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=SnqEA7S/x6xjmwHd0nT05P+nhQ1Qft7XAVGTipkB/g8=; b=GH2K3DSnWuwRRkPGmRM0wYqNQY7Me+oU0Nc1PRqdFTq+U42fZhzDNUtC+QhjXSkVtT gshHTxOV91+BJzEY67O3WYTt83bs07m7kfk6Ycqrlk4tENvctWx8XT+RVyJKGOMy3H0v ZEesm8T/U1US8QVuxMXIia19gnKOwrrKedxaq22+cnVY34t4f52a4Zn/kBQ0vrAxtgNM 2Nz1LtakTu2i/bwn8l6lbduBkVDgf66dXHkgU31sGEH+Tz66w19nm+cAJLWOKgh5cG7B 182zRb74HUqUeCYbqA9VETCIi4x91AA9nYF9lOghV3pa+hZ3FSUSm1txDqOL0XWSoWd9 fGmQ== X-Gm-Message-State: AA6/9RkbiFOLOhTIR9t59JDfargEKA5E/O1DI1S1jTIBLmHbMJaI54Kc/xADtKG9+quAew== X-Received: by 10.194.222.225 with SMTP id qp1mr154337wjc.161.1476265364687; Wed, 12 Oct 2016 02:42:44 -0700 (PDT) Received: from [10.100.64.21] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id af4sm1982414wjc.17.2016.10.12.02.42.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Oct 2016 02:42:43 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <20161012084045.GA57714@zxy.spb.ru> <20161012092945.GB57714@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> Date: Wed, 12 Oct 2016 11:42:38 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161012092945.GB57714@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="N9Ce2544eUHm39KOxGT1kCICLAd7kvFl0" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 09:42:47 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --N9Ce2544eUHm39KOxGT1kCICLAd7kvFl0 Content-Type: multipart/mixed; boundary="fJwW0kuse1wwXC9OHGT2rDC3o0fcKpDKb"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> Subject: Re: 11.0 stuck on high network load References: <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <20161012084045.GA57714@zxy.spb.ru> <20161012092945.GB57714@zxy.spb.ru> In-Reply-To: <20161012092945.GB57714@zxy.spb.ru> --fJwW0kuse1wwXC9OHGT2rDC3o0fcKpDKb Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 10/12/16 11:29 AM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote: >=20 >>> if INP_WLOCK is like spinlock -- this is dead lock. >>> if INP_WLOCK is like mutex -- thread1 resheduled. >> >> Thanks, I understand you question now. No an interrupt cannot bypass= a >> lock: Here INP_WLOCK is like mutex -- thread1 resheduled. >=20 > Thanks, nice. >=20 >>>>> As I remeber race created by call tcp_twstart() at time of end >>>>> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected >>>>> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp= _twstart() >>>> >>>> Exactly, thus the current fix is: If you already have the INP_DROP= PED >>>> flag set you are not allowed to call tcp_twstart(), actually it is a= >>>> good candidate for a new INVARIANT. Let me add that. >>>> >>>>> After check source code I am found invocation of tcp_twstart() in >>>>> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c, >>>>> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c. >>>>> >>>>> Invocation from sys/netinet/tcp_stacks/fastpath.c and >>>>> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and no= w >>>>> will be OK. >>>>> >>>>> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and >>>>> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independ= ed >>>>> INP_WLOCK. Is this OK? >>>>> >>>>> Can be thread A wants do_peer_close() directed from chelsio IRQ >>>>> handler, bypass tcp_input()? >>>> >>>> If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and >>>> t4_cpl_io.c before calling tcp_twstart(). >>> >>> Yes, and you remeber: sys/netinet/tcp_subr.c >>> >>> 1535 struct tcpcb * >>> 1536 tcp_close(struct tcpcb *tp) >>> 1537 { >>> ... >>> 1569 INP_WUNLOCK(inp); >>> 1570 ACCEPT_LOCK(); >>> 1571 SOCK_LOCK(so); >>> 1572 so->so_state &=3D ~SS_PROTOREF; >>> 1573 sofree(so); >>> 1574 return (NULL); >>> >>> sofree() call tcp_usr_detach() and in tcp_usr_detach() we have >>> unexpected INP_TIMEWAIT. >> >> I see, thus just for the context: The TCP stack in sys/dev/cxgb* is = a >> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a >> separate/side TCP stack that is used only with TCP_OFFLOAD option. >> >> This TOE TCP stack actually has its own set of detach()/input() >> functions and seems to check INP_DROPPED flag properly. I guess @np >> check fixes in socket TCP stack and decides which one can also impact >> the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack, so= me >> are only in TOE TCP stack. >=20 > I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TOE > TCP stack and impact this to > tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path. I see, I expect no problem on this side as tcp_timer_2msl() checks the INP_TIMEWAIT flag and do not call tcp_close() if set. -- Julien --fJwW0kuse1wwXC9OHGT2rDC3o0fcKpDKb-- --N9Ce2544eUHm39KOxGT1kCICLAd7kvFl0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX/gWRAAoJEKVlQ5Je6dhxF4YH/3ttxM1WJJuEt5FgXKizUk2/ uKT4mPGegoTIslwUC6CJzs/OjCkWKzKwNCWN5fK630gdV1kgQygPaXINR4QDI90y /VqSVzPdhWltR/+xuziuOdaPeGxrEV51i7poHQYpM9v4vXN/yEj2QiP40lZlUsUi j8LatszYKcEq6u2VHdbzWgxzL7xWqz3PGZZfQaNTxh/yqAK18VUXlmiCqcWzfjz+ rL7kRRAwjpoMtkF0gLLEteGwMnRRNCrZ8aBX3kfcKJr+d32nVHZm9s7TC9uwV7FY iZPJIBiOLURF1QGepbhxQ/ldvJVPNRZtH4rfEXuoOhlX2El+pI9gMEt6foVo9ww= =PK4z -----END PGP SIGNATURE----- --N9Ce2544eUHm39KOxGT1kCICLAd7kvFl0--