From owner-freebsd-net@FreeBSD.ORG Sat Aug 24 16:36:25 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D7983DD8; Sat, 24 Aug 2013 16:36:25 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id B991E275D; Sat, 24 Aug 2013 16:36:25 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 1D5031A3E2D; Sat, 24 Aug 2013 09:36:25 -0700 (PDT) Message-ID: <5218E108.6090901@mu.org> Date: Sat, 24 Aug 2013 09:36:24 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Robert Watson Subject: Re: [rfc] migrate lagg to an rmlock References: <5218AA36.1080807@ipfw.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Adrian Chadd , freebsd-current , "Alexander V. Chernikov" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 16:36:25 -0000 On 8/24/13 7:16 AM, Robert Watson wrote: > On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: > >> On 24.08.2013 00:54, Adrian Chadd wrote: >>> >>> I'd like to commit this to -10. It migrates the if_lagg locking >>> from a rw lock to a rm lock. We see a bit of contention between the >>> transmit and >> >> We're running lagg with rmlock on several hundred heavily loaded >> machines, it really works better. However, there should not be any >> contention between receive and transmit side since there is actually >> no _real_ need to lock RX (and even use lagg receive code at all): >> >> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html > > We should distinguish "lock contention" from "line contention". When > acquiring a rwlock on multiple CPUs concurrently, the cache lines used > to implement the lock are contended, as they must bounce between > caches via the cache coherence protocol, also referred to as > "contention". In the if_lagg code, I assume that the read-only > acquire of the rwlock (and perhaps now rmlock) is for data stability > rather than mutual exclusion -- e.g., to allow processing to > completion against a stable version of the lagg configuration. As > such, indeed, there should be no lock contention unless a > configuration update takes place, and any line contention is a > property of the locking primitive rather than data model. > > There are a number of other places in the kernel where migration to an > rmlock makes sense -- however, some care must be taken for four > reasons: (1) while read locks don't experience line contention, write > locking becomes observably e.g., rmlocks might not be suitable for > tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not > suitable for all rwlock line contention spots -- implement reader > priority propagation, so you must reason about; and (3) historically, > rmlocks have not fully implemented WITNESS so you may get less good > debugging output. if_lagg is a nice place to use rmlocks, as > reconfigurations are very rare, and it's really all about long-term > data stability. > > Robert > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? -Alfred -- Alfred Perlstein