From owner-freebsd-current@FreeBSD.ORG Sat Aug 24 17:47:13 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 286849FC; Sat, 24 Aug 2013 17:47:13 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 015F92AA7; Sat, 24 Aug 2013 17:47:13 +0000 (UTC) Received: from [10.0.1.16] (host86-155-36-56.range86-155.btcentralplus.com [86.155.36.56]) by cyrus.watson.org (Postfix) with ESMTPSA id 645DB46B35; Sat, 24 Aug 2013 13:47:10 -0400 (EDT) Subject: Re: [rfc] migrate lagg to an rmlock Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: "Robert N. M. Watson" In-Reply-To: <5218E108.6090901@mu.org> Date: Sat, 24 Aug 2013 18:47:08 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <5218AA36.1080807@ipfw.ru> <5218E108.6090901@mu.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1283) Cc: FreeBSD Net , Adrian Chadd , freebsd-current , "Alexander V. Chernikov" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 17:47:13 -0000 On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: >> We should distinguish "lock contention" from "line contention". When = acquiring a rwlock on multiple CPUs concurrently, the cache lines used = to implement the lock are contended, as they must bounce between caches = via the cache coherence protocol, also referred to as "contention". In = the if_lagg code, I assume that the read-only acquire of the rwlock (and = perhaps now rmlock) is for data stability rather than mutual exclusion = -- e.g., to allow processing to completion against a stable version of = the lagg configuration. As such, indeed, there should be no lock = contention unless a configuration update takes place, and any line = contention is a property of the locking primitive rather than data = model. >>=20 >> There are a number of other places in the kernel where migration to = an rmlock makes sense -- however, some care must be taken for four = reasons: (1) while read locks don't experience line contention, write = locking becomes observably e.g., rmlocks might not be suitable for = tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable = for all rwlock line contention spots -- implement reader priority = propagation, so you must reason about; and (3) historically, rmlocks = have not fully implemented WITNESS so you may get less good debugging = output. if_lagg is a nice place to use rmlocks, as reconfigurations are = very rare, and it's really all about long-term data stability. >=20 > Robert, what do you think about a quick swap of the ifnet structures = to counter before 10.x? Could you be more specific about the proposal you're making? Robert=