From owner-freebsd-net@FreeBSD.ORG Thu Jan 2 17:23:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3975E221; Thu, 2 Jan 2014 17:23:05 +0000 (UTC) Received: from aussmtpmrkpc120.us.dell.com (aussmtpmrkpc120.us.dell.com [143.166.82.159]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CEA7D1E3C; Thu, 2 Jan 2014 17:23:04 +0000 (UTC) X-Loopcount0: from 64.238.244.148 X-IronPort-AV: E=Sophos;i="4.95,592,1384322400"; d="scan'208";a="66401734" Message-ID: <52C5A020.9010404@vangyzen.net> Date: Thu, 2 Jan 2014 11:21:36 -0600 From: Eric van Gyzen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Adrian Chadd , FreeBSD Net Subject: Re: ipv6 lock contention with parallel socket io References: In-Reply-To: Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jan 2014 17:23:05 -0000 On 12/31/2013 11:53, Adrian Chadd wrote: > On 30 December 2013 23:35, Adrian Chadd wrote: >> Hi, >> >> I've noticed a bunch of lock contention occurs when doing highly >> parallel (> 4096 sockets) TCP IPv6 traffic. >> >> The contention is here: >> > [snip] > >> .. it's the IF_ADATA lock surrounding the lla_lookup() calls. >> >> Now: >> >> * is there any reason this isn't an rmlock? >> * the instance early on in nd6_output_lle() isn't taking the read >> lock, it's taking the full lock. Is there any reason for this? >> >> I don't have much experience or time to spend on optimising the ipv6 >> code to scale better but this seems like one of those things that'll >> bite us in the ass as the amount of ipv6 deployed increases. >> >> Does anyone have any ideas/suggestions on how we could improve things? > This improves things quite a bit - from 1.9gbyte/sec @ 100% cpu usage > to 2.7gbyte/sec @ 85% CPU usage. It's not perfect - the lock > contention is still there - but it's much less of an issue now. > > Are there any issues with it? > > Index: sys/netinet6/nd6.c > =================================================================== > --- sys/netinet6/nd6.c (revision 259475) > +++ sys/netinet6/nd6.c (working copy) > @@ -1891,9 +1891,9 @@ > flags = ((m != NULL) || (lle != NULL)) ? LLE_EXCLUSIVE : 0; > if (ln == NULL) { > retry: > - IF_AFDATA_LOCK(ifp); > + IF_AFDATA_RLOCK(ifp); > ln = lla_lookup(LLTABLE6(ifp), flags, (struct sockaddr *)dst); > - IF_AFDATA_UNLOCK(ifp); > + IF_AFDATA_RUNLOCK(ifp); > if ((ln == NULL) && nd6_is_addr_neighbor(dst, ifp)) { > /* > * Since nd6_is_addr_neighbor() internally > calls nd6_lookup(), This change looks safe, since flags can only contain LLE_EXCLUSIVE. However, the assertion in in6_lltable_lookup() seems insufficient. It asserts any lock on afdata. I think it should also assert a write-lock in the LLE_CREATE case. Granted, this is not strictly related to your change. Eric