From owner-freebsd-stable@freebsd.org Mon Oct 10 11:26:35 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1413C0B327 for ; Mon, 10 Oct 2016 11:26:35 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-lf0-f66.google.com (mail-lf0-f66.google.com [209.85.215.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62F43F65 for ; Mon, 10 Oct 2016 11:26:34 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-lf0-f66.google.com with SMTP id p80so9498663lfp.1 for ; Mon, 10 Oct 2016 04:26:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=XrCGbOil6wNQ9qC18JMmhcCIxEHRAiMvA9AjtRBeojQ=; b=MjZEcrIrjsa8otFMMulGhaj+977DMpb6quPSYgc1Km5KCfVQYozVm6Rhc7Jqfx3pNb RjY2bYK5xcoa+r4PFBEhNrxywk1FJj61A64mPMiX/KJSm65YDmkSWZ9FVa37nbO0cKrS 84Uoj2axR+E8LkrpNi6IDmRbrVfiZJmQAfbyvYk8/jgxEbotClzYgLtU6MajB5FkEv+e Wy17d/WFJMJh9MGJedrM5zxMIn3qQAOMbd4MDRCzsA9UFInbJ0Z6i/Y6G0lpUgOnSfHn WwJmuOp7itqjrW3rbxzzCseCflZ+F5cvCbKK9WsDACNHOjGhbEYqlOFVnkVFthZKAPB2 TC9Q== X-Gm-Message-State: AA6/9RnP2l91WOb/KKuB2saWa8yjS7DOzVU7OFLpMC6i1UKnutkryN4rn5nhdQUbMNXUUA== X-Received: by 10.25.15.169 with SMTP id 41mr12323935lfp.19.1476098786694; Mon, 10 Oct 2016 04:26:26 -0700 (PDT) Received: from [10.100.64.17] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id f23sm623801lji.12.2016.10.10.04.26.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Oct 2016 04:26:25 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20160921195155.GW2840@zxy.spb.ru> <20160923200143.GG2840@zxy.spb.ru> <20160925124626.GI2840@zxy.spb.ru> <20160926172159.GA54003@zxy.spb.ru> <62453d9c-b1e4-1129-70ff-654dacea37f9@gmail.com> <20160928115909.GC54003@zxy.spb.ru> <20161006111043.GH54003@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> Date: Mon, 10 Oct 2016 13:26:12 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161006111043.GH54003@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OCKtAMIJv6mron2xficHhijhFvEF3eE3r" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2016 11:26:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --OCKtAMIJv6mron2xficHhijhFvEF3eE3r Content-Type: multipart/mixed; boundary="xUDws4JJQNipMCDbiexlEg21759fQGK8C"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> Subject: Re: 11.0 stuck on high network load References: <20160921195155.GW2840@zxy.spb.ru> <20160923200143.GG2840@zxy.spb.ru> <20160925124626.GI2840@zxy.spb.ru> <20160926172159.GA54003@zxy.spb.ru> <62453d9c-b1e4-1129-70ff-654dacea37f9@gmail.com> <20160928115909.GC54003@zxy.spb.ru> <20161006111043.GH54003@zxy.spb.ru> In-Reply-To: <20161006111043.GH54003@zxy.spb.ru> --xUDws4JJQNipMCDbiexlEg21759fQGK8C Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, On 10/6/16 1:10 PM, Slawa Olhovchenkov wrote: > On Thu, Oct 06, 2016 at 09:28:06AM +0200, Julien Charbon wrote: >=20 >> 2. thread1: In tcp_close() the inp is marked with INP_DROPPED flag, t= he >> process continues and calls INP_WUNLOCK() here: >> >> https://github.com/freebsd/freebsd/blob/releng/11.0/sys/netinet/tcp_su= br.c#L1568 >=20 > Look also to sys/netinet/tcp_timewait.c:488 >=20 > And check other locks from r160549 You are right, and here the a fix proposal for this issue: Fix a double-free when an inp transitions to INP_TIMEWAIT state after having been dropped https://reviews.freebsd.org/D8211 It basically enforces in_pcbdrop() logic in tcp_input(): A INP_DROPPED inpcb should never be proceed further. Slawa, as you are the only one to reproduce this issue currently, could test this patch? (And remove the temporary patch I did provided to you before). I will wait for your tests results before pushing further. Thanks! diff --git a/sys/netinet/tcp_input.c b/sys/netinet/tcp_input.c index c72f01f..37f27e0 100644 --- a/sys/netinet/tcp_input.c +++ b/sys/netinet/tcp_input.c @@ -921,6 +921,16 @@ findpcb: goto dropwithreset; } INP_WLOCK_ASSERT(inp); + /* + * While waiting for inp lock during the lookup, another thread + * can have droppedt the inpcb, in which case we need to loop ba= ck + * and try to find a new inpcb to deliver to. + */ + if (inp->inp_flags & INP_DROPPED) { + INP_WUNLOCK(inp); + inp =3D NULL; + goto findpcb; + } if ((inp->inp_flowtype =3D=3D M_HASHTYPE_NONE) && (M_HASHTYPE_GET(m) !=3D M_HASHTYPE_NONE) && ((inp->inp_socket =3D=3D NULL) || @@ -981,6 +991,10 @@ relocked: if (in_pcbrele_wlocked(inp)) { inp =3D NULL; goto findpcb; + } else if (inp->inp_flags & INP_DROPPED) = { + INP_WUNLOCK(inp); + inp =3D NULL; + goto findpcb; } } else ti_locked =3D TI_RLOCKED; @@ -1040,6 +1054,10 @@ relocked: if (in_pcbrele_wlocked(inp)) { inp =3D NULL; goto findpcb; + } else if (inp->inp_flags & INP_DROPPED) = { + INP_WUNLOCK(inp); + inp =3D NULL; + goto findpcb; } goto relocked; } else -- Julien --xUDws4JJQNipMCDbiexlEg21759fQGK8C-- --OCKtAMIJv6mron2xficHhijhFvEF3eE3r Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX+3rgAAoJEKVlQ5Je6dhxQ40H/0dYh5hPqNQX1r15Z0x1sE9q 9/Lh6Zn6cLM+cxH2Me5rKeVxmX28bpTIug00fbqk6CI0ZlRHS+R4/iP3w2yl40g1 FUGysS8Cvh3EErzsoKHNwscrbNI8DWLgftW0L+el+srGRcVupoHA12AIhMTNCxQ+ Y990PZKWmuOuxCNxkCbm+yadaQbaOsrGoI0uyEoLDovE/rHKr2ObrypFadrXxg64 VL9xegpLzXnVMBUc3b/FbGAyq33KZnAsqc1Thi7pXEm7Lk6rT/m5mq3XC5jcPt9r MIPV9/pNj2Dy7FCQV/K/714O/F8tpCWjtp69KWVB9tcQGVtmd5Fsnh2dMVBH47c= =x0Tb -----END PGP SIGNATURE----- --OCKtAMIJv6mron2xficHhijhFvEF3eE3r--