From owner-freebsd-current@FreeBSD.ORG Sat Aug 24 18:14:38 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0E85EF93; Sat, 24 Aug 2013 18:14:38 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id DE6D82BE2; Sat, 24 Aug 2013 18:14:37 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 0C7851A3E38; Sat, 24 Aug 2013 11:14:28 -0700 (PDT) Message-ID: <5218F803.7000405@mu.org> Date: Sat, 24 Aug 2013 11:14:27 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: "Robert N. M. Watson" Subject: Re: [rfc] migrate lagg to an rmlock References: <5218AA36.1080807@ipfw.ru> <5218E108.6090901@mu.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd , freebsd-current , "Alexander V. Chernikov" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 18:14:38 -0000 On 8/24/13 10:47 AM, Robert N. M. Watson wrote: > On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: > >>> We should distinguish "lock contention" from "line contention". When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as "contention". In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. >>> >>> There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. >> Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? > Could you be more specific about the proposal you're making? > > Robert The lagg patch referred to in the thread seems to indicate that zero locking is needed if we just switched to counter(9), that makes me wonder if we could do better with locking in other places if we switched to counter(9) while we have the chance. This is the thread: http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html >/ />/ Perfect solution would be to convert ifnet(9) to counters(9), but this />/ requires much more work, and unfortunately ABI change, so temporarily />/ patch lagg(4) manually. />/ />/ We store counters in the softc, and once per second push their values />/ to legacy ifnet counters./ -- Alfred Perlstein