From owner-freebsd-stable@freebsd.org Thu Oct 13 15:06:18 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90133C0E985 for ; Thu, 13 Oct 2016 15:06:18 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47675F27 for ; Thu, 13 Oct 2016 15:06:17 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-qk0-f178.google.com with SMTP id n189so95296169qke.0 for ; Thu, 13 Oct 2016 08:06:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=rh1+rbd2zgGGduc6WjISZr8tOL4qyXq3XWCYXd7DlAI=; b=j4QZ41insy6LLmsOmI1Ry5f5D+avtcmwOCRUMOIWAAfLRCJQBFYaMBKcj4vs+2qoar EiOLNyVH+wbJ+fBlZ1GHNJUt4LRDIcpm1s82WcTNBJHPuwGNGkkDZwYM+Wo/0WQWyi2L JE6+5vPxfYSp2B7LrJO76ldgw1dBXcE0a/f8V5rOqZJltLD7uzvfO6w0GoGhKaZ2DDij gxDmu4PkiFJQy4V0RO9GzrCJcRxjUT67rAYfVOBYMkrzOAw+lsJXdaaRFshLNXiIIkm+ oFnlYNzEX7MqtmDVIksFP64wfOCdVQ/AsrexXbnQqOUz22YOPNZ9iz0PXeT2/COOtTQb L3Ow== X-Gm-Message-State: AA6/9RmeSNIW6bqMr4L9gPqQaRJd0OZNZEyx//lYJAQNt91xh3Aav9HBidnhb1IGFgA33A== X-Received: by 10.194.141.233 with SMTP id rr9mr7087673wjb.18.1476371163975; Thu, 13 Oct 2016 08:06:03 -0700 (PDT) Received: from [10.100.64.26] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id y2sm23172921wjx.20.2016.10.13.08.06.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Oct 2016 08:06:02 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> <20161012121322.GB57876@zxy.spb.ru> <62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org> <20161012130103.GD57714@zxy.spb.ru> <20161012154229.GC57876@zxy.spb.ru> <20161013143825.GK57714@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: <33ab0bfc-7009-95a7-7752-c2c439092e85@freebsd.org> Date: Thu, 13 Oct 2016 17:06:00 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161013143825.GK57714@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="a85w2Ti06SRN3wnPEEmo3Nli51ICXgk05" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2016 15:06:18 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --a85w2Ti06SRN3wnPEEmo3Nli51ICXgk05 Content-Type: multipart/mixed; boundary="eS5gOKKhWQfOwIqLPd6aU4a7ATAckBMFo"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: <33ab0bfc-7009-95a7-7752-c2c439092e85@freebsd.org> Subject: Re: 11.0 stuck on high network load References: <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> <20161012121322.GB57876@zxy.spb.ru> <62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org> <20161012130103.GD57714@zxy.spb.ru> <20161012154229.GC57876@zxy.spb.ru> <20161013143825.GK57714@zxy.spb.ru> In-Reply-To: <20161013143825.GK57714@zxy.spb.ru> --eS5gOKKhWQfOwIqLPd6aU4a7ATAckBMFo Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 10/13/16 4:38 PM, Slawa Olhovchenkov wrote: > On Thu, Oct 13, 2016 at 01:56:21PM +0200, Julien Charbon wrote: >>>> Something like: >>> >>> Yes, thanks! >> >> Proposed changes added in the review: >> >> https://reviews.freebsd.org/D8211 >> >> tell me when you have three days without issue with this change. >> >>>> tcp_detach() { >>>> >>>> ... >>>> if (inp->inp_flags & INP_TIMEWAIT) { >>>> >>>> ... >>>> if (inp->inp_flags & INP_DROPPED) { >>>> >>>> in_pcbdetach(inp); >>>> if (__predict_true(tp =3D=3D NULL)) { >>>> in_pcbfree(inp); >>>> } else { >>>> #ifdef INVARIANTS >>>> panic("tcp_detach: tp !=3D NULL, That's not good because 'bl= ah'\n"); >>>> #else >>>> log(LOG_ERR, "tcp_detach: tp !=3D NULL, That's not good beca= use >>>> 'blah'\n"); >>> >>> May be some more info in log can help to detect root cause of issuse?= >>> I am don't know what info, may be flags or number of references? >> >> For this kind of issue, the useful part is the stacktrace. INVARIANT= >=20 > Like this? >=20 > #ifdef KDB > kdb_backtrace(); > #endif >=20 > as found in sys/netgraph/ng_base.c It is overkill dtrace can do that. >> will give you that trace in the core, and without INVARIANT then it is= >> better to use dtrace: >> >> $ cat tcp-twstart-dropped.d >> fbt::tcp_twstart:entry >> /args[0]->t_inpcb->inp_flags & 0x04000000/ >> { >> stack(); >> printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_flags= ); >> } >=20 > Same code may be insert there too, IMHO. Hmm, I don't think so: - If you have INVARIANT, the kernel will panic in tcp_twstart() or tcp_detach() and you will have everything you need to debug. - If you don't, dtrace is the right tool to use in all cases anyway. -- Julien --eS5gOKKhWQfOwIqLPd6aU4a7ATAckBMFo-- --a85w2Ti06SRN3wnPEEmo3Nli51ICXgk05 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX/6LcAAoJEKVlQ5Je6dhxlUAH/jHhMvvTBIoxoK8eodIPzE8m +5quGdpDvht+3mRfY6fGpYGCTT60rT3xMsN2L0xNJ8i6qXiY9oB48nTO8P9359Nk 5oolU1L7WCFeKAeTkmOCWim9YxTGT9f+4MQxVliKsMa3uDlR09RMj1JLucVLAXZ1 tbc6u96+AlhTZu21EzGiejv4otp+KbLEbDCmFS1jbufLd8tnN2A3S/olMzv4kadi plhwDu+wAXk5cu0hK8ETDwjlmVV+MXY7r2yAZp3jk5QXvh3wSbDK+2Re8WQpnaze Ee2N7zdibXVpOTCdACK02srQZlAASC8DwRy41cCLVV8mvmEoG8ijJxKYLXL0KBw= =oU3g -----END PGP SIGNATURE----- --a85w2Ti06SRN3wnPEEmo3Nli51ICXgk05--