From owner-freebsd-net@FreeBSD.ORG Thu Mar 6 21:57:46 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2BB2A6B for ; Thu, 6 Mar 2014 21:57:45 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BDC86FF8 for ; Thu, 6 Mar 2014 21:57:45 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s26LvhIY094513 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 6 Mar 2014 13:57:44 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s26LvhZi094512; Thu, 6 Mar 2014 13:57:43 -0800 (PST) (envelope-from jmg) Date: Thu, 6 Mar 2014 13:57:43 -0800 From: John-Mark Gurney To: Julien Charbon Subject: Re: TCP stack lock contention with short-lived connections Message-ID: <20140306215743.GB47921@funkthat.com> Mail-Followup-To: Julien Charbon , freebsd-net@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 06 Mar 2014 13:57:44 -0800 (PST) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2014 21:57:46 -0000 Julien Charbon wrote this message on Thu, Feb 27, 2014 at 11:32 +0100: > Obviously, to mitigate this lock contention there are various solutions: > > - Introduce a new time-wait lock as proposed in joined patch > - Call tcp_tw_2msl_scan() more often in case of high workload > - Use INP_INFO_TRY_WLOCK() in tcp_tw_2msl_scan() to clean-up time-wait > objects only when nobody else handles INP_INFO lock > - Etc. > > The strategy being to prioritize packet reception over time-wait > objects cleaned-up as: > > - we hate dropping packet in reception when the bandwidth is far from > being full > - the maximum of used time-wait objects is configurable > (net.inet.tcp.maxtcptw) > - in case of time-wait objects memory exhaustion, the current behavior > is already optimal: The oldest time-wait object is recycled and > directly reused. > > We picked the time-wait lock way because it suits well our long-term > strategy to completely mitigate the INP_INFO lock contention everywhere > in TCP stack. > > Any thoughts on this particular behavior? One thing that I noticed is that you now lock/unlock the tw and inp lock a lot... Have you thought about grabing the TW lock once, grabbing some/all of the ones necessary to process and then process them in a second step? If the bulk processing is still an issue, then you could process them in blocks of 50 or so, that way the number of lock/unlock cycles is reduced... > +/* > + * Drop a refcount on an tw elevated using tw_pcbref(). If it is > + * valid, we return with the tw lock held. > + */ I assume you mean that you return with the tw lock unlocked? at least that's what the code reads to me... [...] > +static int > +tw_pcbrele(struct tcptw *tw) > +{ > + TW_WLOCK_ASSERT(V_tw_lock); > + KASSERT(tw->tw_refcount > 0, ("%s: refcount 0", __func__)); > + > + if (!refcount_release(&tw->tw_refcount)) { > + TW_WUNLOCK(V_tw_lock); > + return (0); > + } > + > + uma_zfree(V_tcptw_zone, tw); > + TW_WUNLOCK(V_tw_lock); > + return (1); > +} [...] Otherwise looks like a good patch... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."