From owner-freebsd-stable@freebsd.org Mon Sep 19 20:32:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44A49BE1D4C for ; Mon, 19 Sep 2016 20:32:23 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CEC15A36 for ; Mon, 19 Sep 2016 20:32:22 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-f41.google.com with SMTP id l132so30114855wmf.1 for ; Mon, 19 Sep 2016 13:32:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=TIfrkBxfolzizDHPx/PMBOmQOg8Sxg9+F77OLRxcUy8=; b=XSkrEq1Z4L78gF8LQNVEKiJ0TViLKPedLRvC4RDSAFP6/musYW+HupP5G212Vt9pym YS6/0/8SwqpO7SG59erzXUAhpdpBT1fUwQmkYtVdQN/Xx04nWAcS2yK93Y925R/grZh2 XMGe0HtvVtk/UeIn2/8mixDN8McoQJkH4loKiqIH8Nm5GY8FuQg/HTvz4tALm4VUTAJP 6ERhGt1GL/1MP34s35VaUPpS19xsyZHKELfx7Z+UIbiGihC5FV/l/J31WgD5R/lEXWtP /99pGDV5Z36hciY6OI4zugQ/A5jy+jFSXpumUtFTcMQ1ilLZez1uIbQaTa1eziF5xyrK l2Pg== X-Gm-Message-State: AE9vXwPaYYmPgY5kqItzpAP/uYnTVNjUzW1CAO0LIwPIp6osqRHHVGl85donsAC5BLvZpw== X-Received: by 10.28.13.66 with SMTP id 63mr15820wmn.113.1474317140206; Mon, 19 Sep 2016 13:32:20 -0700 (PDT) Received: from [192.168.0.12] (217-162-163-184.dynamic.hispeed.ch. [217.162.163.184]) by smtp.gmail.com with ESMTPSA id ir9sm24803965wjb.16.2016.09.19.13.32.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Sep 2016 13:32:19 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov , hiren panchasara References: <20160904215739.GC22212@zxy.spb.ru> <20160905014612.GA42393@strugglingcoder.info> <20160914213503.GJ2840@zxy.spb.ru> <20160915085938.GN38409@kib.kiev.ua> <20160915090633.GS2840@zxy.spb.ru> <20160916181839.GC2960@zxy.spb.ru> <20160916183053.GL9397@strugglingcoder.info> <20160916190330.GG2840@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org From: Julien Charbon Message-ID: <78cbcdc9-f565-1046-c157-2ddd8fcccc62@freebsd.org> Date: Mon, 19 Sep 2016 22:32:13 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160916190330.GG2840@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="MikMWBCTJh2GK09JwCirqxaop1nePIMxP" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2016 20:32:23 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --MikMWBCTJh2GK09JwCirqxaop1nePIMxP Content-Type: multipart/mixed; boundary="mgBFvcdIF2aRPu3PPAA6UmtpRvir4s5j4"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov , hiren panchasara Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org Message-ID: <78cbcdc9-f565-1046-c157-2ddd8fcccc62@freebsd.org> Subject: Re: 11.0 stuck on high network load References: <20160904215739.GC22212@zxy.spb.ru> <20160905014612.GA42393@strugglingcoder.info> <20160914213503.GJ2840@zxy.spb.ru> <20160915085938.GN38409@kib.kiev.ua> <20160915090633.GS2840@zxy.spb.ru> <20160916181839.GC2960@zxy.spb.ru> <20160916183053.GL9397@strugglingcoder.info> <20160916190330.GG2840@zxy.spb.ru> In-Reply-To: <20160916190330.GG2840@zxy.spb.ru> --mgBFvcdIF2aRPu3PPAA6UmtpRvir4s5j4 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 9/16/16 9:03 PM, Slawa Olhovchenkov wrote: > On Fri, Sep 16, 2016 at 11:30:53AM -0700, hiren panchasara wrote: >=20 >> On 09/16/16 at 09:18P, Slawa Olhovchenkov wrote: >>> On Thu, Sep 15, 2016 at 12:06:33PM +0300, Slawa Olhovchenkov wrote: >>> >>>> On Thu, Sep 15, 2016 at 11:59:38AM +0300, Konstantin Belousov wrote:= >>>> >>>>> On Thu, Sep 15, 2016 at 12:35:04AM +0300, Slawa Olhovchenkov wrote:= >>>>>> On Sun, Sep 04, 2016 at 06:46:12PM -0700, hiren panchasara wrote: >>>>>> >>>>>>> On 09/05/16 at 12:57P, Slawa Olhovchenkov wrote: >>>>>>>> I am try using 11.0 on Dual E5-2620 (no X2APIC). >>>>>>>> Under high network load and may be addtional conditional system = go to >>>>>>>> unresponsible state -- no reaction to network and console (USB I= PMI >>>>>>>> emulation). INVARIANTS give to high overhad. Is this exist some = way to >>>>>>>> debug this? >>>>>>> >>>>>>> Can you panic it from console to get to db> to get backtrace and = other >>>>>>> info when it goes unresponsive? >>>>>> >>>>>> ipmi console don't respond (chassis power diag don't react) >>>>>> login on sol console stuck on *tcp. >>>>> >>>>> Is 'login' you reference is the ipmi client state, or you mean logi= n(1) >>>>> on the wedged host ? >>>> >>>> on the wedged host >>>> >>>>> If BMC stops responding simultaneously with the host, I would suspe= ct >>>>> the hardware platform issues instead of a software problem. Do you= have >>>>> dedicated LAN port for BMC ? >>>> >>>> Yes. >>>> But BMC emulate USB keyboard and this is may be lock inside USB >>>> system. >>>> "ipmi console don't respond" must be read as "ipmi console runnnig a= nd >>>> attached but system don't react to keypress on this console". >>>> at the sime moment system respon to `enter` on ipmi sol console, but= >>>> after enter `root` stuck in login in the '*tcp' state (I think this = is >>>> NIS related). >>> >>> ~^B don't break to debuger. >>> But I can login to sol console. >> >> You can probably: >> debug.kdb.enter: set to enter the debugger >> >> or force a panic and get vmcore: >> debug.kdb.panic: set to panic the kernel >=20 > I am reset this host. > PMC samples collected and decoded: >=20 > @ CPU_CLK_UNHALTED_CORE [4653445 samples] >=20 > 51.86% [2413083] lock_delay @ /boot/kernel.VSTREAM/kernel > 100.0% [2413083] __rw_wlock_hard > 100.0% [2413083] tcp_tw_2msl_scan > 99.99% [2412958] pfslowtimo > 100.0% [2412958] softclock_call_cc > 100.0% [2412958] softclock > 100.0% [2412958] intr_event_execute_handlers > 100.0% [2412958] ithread_loop > 100.0% [2412958] fork_exit > 00.01% [125] tcp_twstart > 100.0% [125] tcp_do_segment > 100.0% [125] tcp_input > 100.0% [125] ip_input > 100.0% [125] swi_net > 100.0% [125] intr_event_execute_handlers > 100.0% [125] ithread_loop > 100.0% [125] fork_exit The only write lock tcp_tw_2msl_scan() tries to get is a INP_WLOCK(inp). Thus here, tcp_tw_2msl_scan() seems to be stuck spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls tcp_tw_2msl_scan() at high rate but this will be quite unexpected). Thus my hypothesis is that something is holding the INP_WLOCK and not releasing it, and tcp_tw_2msl_scan() is spinning on it. If you can, could you compile the kernel with below options: options DDB # Support DDB. options DEADLKRES # Enable the deadlock resolver options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed And once the issue is reproduce, run in ddb run the below commands: show pcpu show allpcpu show locks show alllocks show lockchain show allchains show all trace This is to see if the contention is indeed on the tcp_tw_2msl_scan's INP_WLOCK. -- Julien --mgBFvcdIF2aRPu3PPAA6UmtpRvir4s5j4-- --MikMWBCTJh2GK09JwCirqxaop1nePIMxP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX4EtSAAoJEKVlQ5Je6dhx7mkIAJo089e6oC5ZUq4bLa9dOiGP NGf5Qq9DzbrrGQXI9AXCtcYUio8dSnBaK3Gkv5POxAW+7ppq9kaiznydugIZMfMC L0A1cLmiFesvZ1BxGFxeE/P+sn1iTIgiKXFbMW4MmesmKEVwTKJ/7BgIQKX7TYUA ZbuNLuL4yWS9HPQitb3I23pcfJ6tBRHCKORB9An6OJA76ETizodyp0mUPp9UxEXP xKGXdmJPKpnts8eUuMOCGdBTqOghzP9gwEnq8J+0cGjcE45zErxnI5FvMx/VeGnr xiwBb6E3/CvKURcvsDc2SDlZ6lokNM21drLzIBOX8WN++F+JG0ayKwHcxtdLebA= =FcSq -----END PGP SIGNATURE----- --MikMWBCTJh2GK09JwCirqxaop1nePIMxP--