From owner-freebsd-stable@freebsd.org Wed Sep 21 21:25:28 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1543BE37FD for ; Wed, 21 Sep 2016 21:25:28 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6642B86C for ; Wed, 21 Sep 2016 21:25:28 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-x232.google.com with SMTP id l132so292168327wmf.0 for ; Wed, 21 Sep 2016 14:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=k5CLCtVpIkmqkgjvo2JIE8j8wgkatI9oCXcqIs/iwEA=; b=qTR8atZmRpnJMcGNfW6bYHIc1IfduguNQ08QwS0/ARK8yxcL6Tr7pTnbMYB5D8RzMl yNc8CauKu/EBSnPlv3pdKGj54DDlcTMfxZlhgtajHgPwWUDI2DAF0zBEd0ePU5qpEN4s frNSlrln/n4YeRteqb63vN4Wksx9miFLerwAVKq+fVr6WR6wy+ZE0sf3Y+nhcs+P+PbB /ZBR5cqWXqe4zy1SEnWkMY4IdTyOsOdJkhiReoGJ/9nt99M8oNf81mD0BSg5iJ3o0pFY C4ShfC+jFcwRQVFdzSeycDoX+rS6apA0EoYeP9qOJVijZW2X37Ahe9ETlL7z3XidvwAH NODA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=k5CLCtVpIkmqkgjvo2JIE8j8wgkatI9oCXcqIs/iwEA=; b=O2dmMMwwUTBvv0h4cqGZaiYLTgKF770O1UG+oylyJJpugGm9A8hG5kRWlzIbKQH76Z PhW6cg9eUn2dvEehLkklafHZG7yphXWF++od1gCtzr0g93fcSdAyHaB16bS6/rgGQefU rWexNkAZZ4ZbkYkgT/Ko1emDGbIM9eg139/dsFVhscCHHK4nvBfEfwZ3N6k3fowLGbUv pKHCK0jb7B5UQ5s/PSnlwMbahQzUHQ3sfORu6VR7XlYGsslTDRnQ9EL0exHnjZbGFeEb qw+JAFK+tvVfLX71wRfBpKk8QK3kU0rcxocaFvYkGuij3tDR3T+d7cozdArNYWQIKENC q0dQ== X-Gm-Message-State: AE9vXwOquAmtPIOfgZHBVNjTcSy0XmrMK+87CF72RSLiP/J4G9bbjYdGVyY47BYXJ5qGZw== X-Received: by 10.28.66.6 with SMTP id p6mr4737960wma.59.1474493126961; Wed, 21 Sep 2016 14:25:26 -0700 (PDT) Received: from [192.168.0.14] (217-162-163-184.dynamic.hispeed.ch. [217.162.163.184]) by smtp.gmail.com with ESMTPSA id xb6sm35418168wjb.30.2016.09.21.14.25.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Sep 2016 14:25:26 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20160915085938.GN38409@kib.kiev.ua> <20160915090633.GS2840@zxy.spb.ru> <20160916181839.GC2960@zxy.spb.ru> <20160916183053.GL9397@strugglingcoder.info> <20160916190330.GG2840@zxy.spb.ru> <78cbcdc9-f565-1046-c157-2ddd8fcccc62@freebsd.org> <20160919204328.GN2840@zxy.spb.ru> <8ba75d6e-4f01-895e-0aed-53c6c6692cb9@freebsd.org> <20160920202633.GQ2840@zxy.spb.ru> <20160921195155.GW2840@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: Date: Wed, 21 Sep 2016 23:25:18 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160921195155.GW2840@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="PMIcB2RuhcdqOnqGRO71Tkne4QgWf9AS6" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Sep 2016 21:25:29 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --PMIcB2RuhcdqOnqGRO71Tkne4QgWf9AS6 Content-Type: multipart/mixed; boundary="31rbt2q54Ws8RPnw0ruU6la6KWx342Bne"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: Subject: Re: 11.0 stuck on high network load References: <20160915085938.GN38409@kib.kiev.ua> <20160915090633.GS2840@zxy.spb.ru> <20160916181839.GC2960@zxy.spb.ru> <20160916183053.GL9397@strugglingcoder.info> <20160916190330.GG2840@zxy.spb.ru> <78cbcdc9-f565-1046-c157-2ddd8fcccc62@freebsd.org> <20160919204328.GN2840@zxy.spb.ru> <8ba75d6e-4f01-895e-0aed-53c6c6692cb9@freebsd.org> <20160920202633.GQ2840@zxy.spb.ru> <20160921195155.GW2840@zxy.spb.ru> In-Reply-To: <20160921195155.GW2840@zxy.spb.ru> --31rbt2q54Ws8RPnw0ruU6la6KWx342Bne Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 9/21/16 9:51 PM, Slawa Olhovchenkov wrote: > On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote: >> You can also use Dtrace and lockstat (especially with the lockstat -s= >> option): >> >> https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks >> https://www.freebsd.org/cgi/man.cgi?query=3Dlockstat&manpath=3DFreeBSD= +11.0-RELEASE >> >> But I am less familiar with Dtrace/lockstat tools. >=20 > I am still use old kernel and got lockdown again. > Try using lockstat (I am save more output), interesting may be next: >=20 > R/W writer spin on writer: 190019 events in 1.070 seconds (177571 event= s/sec) >=20 > -----------------------------------------------------------------------= -------- > Count indv cuml rcnt nsec Lock Caller = =20 > 140839 74% 74% 0.00 24659 tcpinp tcp_tw_2msl_scan+= 0xc6 =20 >=20 > nsec ------ Time Distribution ------ count Stack = =20 > 4096 | 913 tcp_twstart+0xa3 = =20 > 8192 |@@@@@@@@@@@@ 58191 tcp_do_segment+0x2= 01f =20 > 16384 |@@@@@@ 29594 tcp_input+0xe1c = =20 > 32768 |@@@@ 23447 ip_input+0x15f = =20 > 65536 |@@@ 16197 =20 > 131072 |@ 8674 =20 > 262144 | 3358 =20 > 524288 | 456 =20 > 1048576 | 9 =20 > -----------------------------------------------------------------------= -------- > Count indv cuml rcnt nsec Lock Caller = =20 > 49180 26% 100% 0.00 15929 tcpinp tcp_tw_2msl_scan+0= xc6 =20 >=20 > nsec ------ Time Distribution ------ count Stack = =20 > 4096 | 157 pfslowtimo+0x54 = =20 > 8192 |@@@@@@@@@@@@@@@ 24796 softclock_call_cc+= 0x179=20 > 16384 |@@@@@@ 11223 softclock+0x44 = =20 > 32768 |@@@@ 7426 intr_event_execute= _handlers+0x95 > 65536 |@@ 3918 =20 > 131072 | 1363 =20 > 262144 | 278 =20 > 524288 | 19 =20 > -----------------------------------------------------------------------= -------- This is interesting, it seems that you have two call paths competing for INP locks here: - pfslowtimo()/tcp_tw_2msl_scan(reuse=3D0) and - tcp_input()/tcp_twstart()/tcp_tw_2msl_scan(reuse=3D1) These paths can indeed compete for the same INP lock, as both tcp_tw_2msl_scan() calls always start with the first inp found in twq_2msl list. But in both cases, this first inp should be quickly used and its lock released anyway, thus that could explain your situation it that the TCP stack is doing that all the time, for example: - Let say that you are running out completely and constantly of tcptw, and then all connections transitioning to TIME_WAIT state are competing with the TIME_WAIT timeout scan that tries to free all the expired tcptw. If the stack is doing that all the time, it can appear like "live" locked. This is just an hypothesis and as usual might be a red herring. Anyway, could you run: $ vmstat -z | head -2; vmstat -z | grep -E 'tcp|sock' Ideally, once when everything is ok, and once when you have the issue to see the differences (if any). If it appears your are quite low in tcptw, and if you have enough memory, could you try increase the tcptw limit using sysctl net.inet.tcp.maxtcptw? And actually see if it improve (or not) your performance. My 2 cents. -- Julien --31rbt2q54Ws8RPnw0ruU6la6KWx342Bne-- --PMIcB2RuhcdqOnqGRO71Tkne4QgWf9AS6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX4vrEAAoJEKVlQ5Je6dhxDDAH/1ZVvUOHAOrR83zbYHmPgaGZ NgNAj8jsXSf8Q37bnl/m4QwF25Dg9srpEv8VXmgKFO7MlpoIvNcf9I/exILbAJTO TnHB48CKL82SvXtVdb7DBmRC51fOZZJCZ5FQwjAuJSz9GOxbrxrARjWXH+jrNEx6 GjA/4CfcnaM7Uq5Y7jsAnxjYyBDdBdaTPqc03asUySKtdEEPHa9bonK/YTsHElik W8V4OIsarjeLrF6RLaxW9oYTTIWLsDEG8Hvw/M/PEFvxggHYNXfOx78L24PW/twF cXLoh0+U1UOUR4dob+fprdRARbDcTm53VS16DSSn/0IZ4jLcbPeKt0Ng/UgJh0g= =aRb8 -----END PGP SIGNATURE----- --PMIcB2RuhcdqOnqGRO71Tkne4QgWf9AS6--