From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 21 14:27:01 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CB7D816A501 for ; Thu, 21 Dec 2006 14:27:01 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.185]) by mx1.freebsd.org (Postfix) with ESMTP id 3F6A613C44E for ; Thu, 21 Dec 2006 14:27:01 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id x37so2868786nfc for ; Thu, 21 Dec 2006 06:27:00 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=chfiFJtcDe1ZmCzdtjKFhZGF9b0L9re7eWjAT2nMXfsKhwM6ni8jCHVXLKYb1cg+Qa5toYlTV7/P8tcgpdSERKSUZO7BeZtkFZAyWYIL/rDhpaaPstRiwnMQUM50TQEjWV8hfkgRktIud/7L7E/+oB56jVBjIg1gb5ZKUOC0Ues= Received: by 10.82.105.13 with SMTP id d13mr1911565buc.1166709482541; Thu, 21 Dec 2006 05:58:02 -0800 (PST) Received: by 10.82.178.4 with HTTP; Thu, 21 Dec 2006 05:58:02 -0800 (PST) Message-ID: <3bbf2fe10612210558m66795673kd352a385a98f6e2b@mail.gmail.com> Date: Thu, 21 Dec 2006 14:58:02 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Suleiman Souhlal" In-Reply-To: <458A249D.3030502@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20061220041843.GA10511@dwpc.dwlabs.ca> <3bbf2fe10612200414j4c1c01ecr7b37e956b70b01fa@mail.gmail.com> <458A249D.3030502@FreeBSD.org> X-Google-Sender-Auth: d0c1054090421291 Cc: freebsd-hackers@freebsd.org, Duane Whitty Subject: Re: Locking fundamentals X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2006 14:27:01 -0000 2006/12/21, Suleiman Souhlal : > Attilio Rao wrote: > > 2006/12/20, Duane Whitty : > > > >> Hello again, > >> > >> It seems to me that understanding locking holds the key to > >> understanding fbsd internals. > >> > >> Could someone review my understanding of fbsd locking fundamentals. > >> (No assertions here, just questions) > >> > >> lock_mgr > >> -------------------- > >> mutexes|sx_lock > >> ------------------- ^ > >> atomic | mem barriers | > > > > > > Our current locking hierarchy is basically different: > > > > III level: lockmgr - sema - sx > > II level: mutex (sleep/spin/pool) - rwlock - refcount - cv - msleep > > I level: atomic instructions - memory barriers - sleepqueues/turnstiles > > > > (a lower lever means that the upper layer primitives use it as a base. > > ie: sx locks are build using 1 pool > > mutex and 2 condition variables). > > > > This scheme is far from being perfect due to the presence of 'level 3 > > primitives' which should never exist. > > Currently, there is an ongoing efforts to take all the top layer > > primitives to the level II. > > > > On the other side, level I primitives should never be used directly by > > kernel code, but should only be used as a bottom layer for > > syncronizing primitives. All you need to care is in the layer 2 and 3 > > (and possibly should switch to layer 2). > > I disagree. There are many uses of atomic operations in the kernel that are not for locks or refcounts. It's a bad idea to use locks if you can achieve the same thing locklessly, with atomic operations. I can agree with you about this but atomic instructions/memory barriers should be used very carefully (and if you know what you are going to do). It is very simple to write down a wrong semantic using them. > I would personally also add "critical sections" (critical_enter()/critical_exit()) at level I. They can be used instead of locks when you know your data will only be accessed on one CPU, and you only need to protect it from (non-FAST) interrupt handlers. >From this point of view, we would also add sched_pin()/sched_unpin() which are used in order to avoid thread migration between CPUs (particulary helpfull in the case we have to access safely to some per-CPU datas). However, probabilly one of the most important usage we do of critical_section is in the spin mutex implementation (which linked to interrupt disabling would avoid deadlock in spin mutex code). Attilio -- Peace can only be achieved by understanding - A. Einstein