From owner-freebsd-net@FreeBSD.ORG Wed May 28 16:44:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03C97DEF for ; Wed, 28 May 2014 16:44:18 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B23902AF5 for ; Wed, 28 May 2014 16:44:17 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1Wpgwt-0000rT-6T for freebsd-net@freebsd.org; Wed, 28 May 2014 18:43:31 +0200 Received: from h87.s239.verisign.com ([216.168.239.87]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 May 2014 18:43:31 +0200 Received: from jcharbon by h87.s239.verisign.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 May 2014 18:43:31 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-net@freebsd.org From: Julien Charbon Subject: Re: TCP stack lock contention with short-lived connections Date: Wed, 28 May 2014 18:42:49 +0200 Lines: 59 Message-ID: <53861209.2000306@verisign.com> References: <537F39DF.1090900@verisign.com> <537FB51D.2060401@verisign.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: h87.s239.verisign.com User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 In-Reply-To: <537FB51D.2060401@verisign.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2014 16:44:18 -0000 Hi, On 23/05/14 22:52, Julien Charbon wrote: > On 23/05/14 14:06, Julien Charbon wrote: >> On 27/02/14 11:32, Julien Charbon wrote: >>> On 07/11/13 14:55, Julien Charbon wrote: >>>> On Mon, 04 Nov 2013 22:21:04 +0100, Julien Charbon >>>> wrote: >>>>> I have put technical and how-to-repeat details in below PR: >>>>> >>>>> kern/183659: TCP stack lock contention with short-lived connections >>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=183659 >>>>> >>>>> We are currently working on this performance improvement effort; it >>>>> will impact only the TCP locking strategy not the TCP stack logic >>>>> itself. We will share on freebsd-net the patches we made for >>>>> reviewing and improvement propositions; anyway this change might also >>>>> require enough eyeballs to avoid tricky race conditions introduction >>>>> in TCP stack. > > Joined the two cumulative patches (tcp-scale-inp-list-v1.patch and > tcp-scale-pcbinfo-rlock-v1.patch) we discussed the most at BSDCan 2014. At BSDCan 2014 we were also asked to provide flame graph [1][2] to highlight impacts of these TCP changes. The Dtrace sampling was done on a NIC receive queue IRQ bound core. o First CPU flame graph on 10.0-RELENG at 40k TCP connection/secs: https://googledrive.com/host/0BwwgoN552srvQi1JWG42TklfQ28/releng10-40k.html Note: - __rw_wlock_hard on ipi_lock contention is clear as usual. o Second, same test with all our patches applied (thus from 10.0-next branch [3]): https://googledrive.com/host/0BwwgoN552srvQi1JWG42TklfQ28/tcp-scale-40k.html Note: - Almost all __rw_wlock_hard on ipi_lock contention is converted in idle time. o Third, still using 10.0-next branch, the flame graph when doubling the rate to 80k TCP connection/sec: https://googledrive.com/host/0BwwgoN552srvQi1JWG42TklfQ28/tcp-scale-80k.html My 2 cents. [1] http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html [2] https://wiki.freebsd.org/201405DevSummit/NetworkStack [3] https://github.com/verisign/freebsd/commits/share/10.0-next -- Julien