From owner-freebsd-net@FreeBSD.ORG Sat Aug 24 16:12:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 91738F4; Sat, 24 Aug 2013 16:12:19 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B5A2E25FF; Sat, 24 Aug 2013 16:12:18 +0000 (UTC) Received: by mail-wg0-f49.google.com with SMTP id y10so1514388wgg.16 for ; Sat, 24 Aug 2013 09:12:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=VieMGFY2jPZAeEUd3qHXiXNBGXUzf94J3Sk/FSgubSY=; b=pVDDiZik4WuCvijZKvZ41VJapFKQ7yNLBs0v7LIOjajXyeec2qbZkyiWbnvxbyCphq 5BMBxo5yb3iJC06w+VV3RYJSCHjWY2MP5JIUq1qwHMzopcVvUOsJO/bmvrANr3b/3Iej cYpWwX+8Ha33ZoOcul4hG4B9ScayVIsfVg3IDrNXwIwcH18OlT0+F5wsuvv7Yo2FT04k 8fiY/dKPXtpjKG64lL18urdnHX6fu1QDmOQeBGLK2BYWzvlg0VqKQOiKA48uLf+3MDTl nhlyUI8dPtwmU2kwtt7fx4YZKBxMEb/nBtzqmKT8P0/FEkQfEinAImziQ5ZvkJ/LgRAN XEuA== MIME-Version: 1.0 X-Received: by 10.180.36.133 with SMTP id q5mr1960687wij.0.1377360737044; Sat, 24 Aug 2013 09:12:17 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.116.136 with HTTP; Sat, 24 Aug 2013 09:12:16 -0700 (PDT) In-Reply-To: References: <5218AA36.1080807@ipfw.ru> Date: Sat, 24 Aug 2013 09:12:16 -0700 X-Google-Sender-Auth: byUiCc2zTVAddHrpkga8GwMXMFQ Message-ID: Subject: Re: [rfc] migrate lagg to an rmlock From: Adrian Chadd To: Robert Watson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , freebsd-current , "Alexander V. Chernikov" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 16:12:19 -0000 Sorry, I meant "line contention" rather than "lock contention". Yes, you're right. -adrian On 24 August 2013 07:16, Robert Watson wrote: > On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: > > On 24.08.2013 00:54, Adrian Chadd wrote: >> >>> >>> I'd like to commit this to -10. It migrates the if_lagg locking >>> from a rw lock to a rm lock. We see a bit of contention between the >>> transmit and >>> >> >> We're running lagg with rmlock on several hundred heavily loaded >> machines, it really works better. However, there should not be any >> contention between receive and transmit side since there is actually no >> _real_ need to lock RX (and even use lagg receive code at all): >> >> http://lists.freebsd.org/**pipermail/svn-src-all/2013-**April/067570.html >> > > We should distinguish "lock contention" from "line contention". When > acquiring a rwlock on multiple CPUs concurrently, the cache lines used to > implement the lock are contended, as they must bounce between caches via > the cache coherence protocol, also referred to as "contention". In the > if_lagg code, I assume that the read-only acquire of the rwlock (and > perhaps now rmlock) is for data stability rather than mutual exclusion -- > e.g., to allow processing to completion against a stable version of the > lagg configuration. As such, indeed, there should be no lock contention > unless a configuration update takes place, and any line contention is a > property of the locking primitive rather than data model. > > There are a number of other places in the kernel where migration to an > rmlock makes sense -- however, some care must be taken for four reasons: > (1) while read locks don't experience line contention, write locking > becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) > rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock > line contention spots -- implement reader priority propagation, so you must > reason about; and (3) historically, rmlocks have not fully implemented > WITNESS so you may get less good debugging output. if_lagg is a nice place > to use rmlocks, as reconfigurations are very rare, and it's really all > about long-term data stability. > > Robert >