From owner-freebsd-hackers@freebsd.org Wed Apr 12 11:56:55 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD77AD39CE6 for ; Wed, 12 Apr 2017 11:56:55 +0000 (UTC) (envelope-from ablacktshirt@gmail.com) Received: from mail-pg0-x241.google.com (mail-pg0-x241.google.com [IPv6:2607:f8b0:400e:c05::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D80DB18 for ; Wed, 12 Apr 2017 11:56:55 +0000 (UTC) (envelope-from ablacktshirt@gmail.com) Received: by mail-pg0-x241.google.com with SMTP id 79so4917595pgf.0 for ; Wed, 12 Apr 2017 04:56:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=nxeN4e5qP1PDoFMH8fkHZssfH3T8E3KVVH241a9rSBg=; b=LNLPD/WmNWQtmgoKvmxgt86srQQ0XLlc9iY7hCfuj5CLSE6g6/J7EZBB9UgA1s9/M+ LBY8PYw4l1npYhGw1kkAEB2RLT3MVSJpVVzwY1wTEPoB7UXJgmCPDdeYL9u803ztaai/ r9GmAKh45Za6NL19X5qRJ33hf6TmCMFDpbq6azIYxoEcN+hZ18w8hfm7wLrNqpFVmrsN YTmZExkNP6HaLGwghpkyHtBS89342QdtV0GducSBJSi8/s12tr1FkPMz3mZvuXizTt8V AVOLfo6waeXkbfAmM4JzeA2gYxmmpwnGe59S0sFdF9KJtfhK+TbmB+60WjQT90NFtQfM COEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=nxeN4e5qP1PDoFMH8fkHZssfH3T8E3KVVH241a9rSBg=; b=CGEsVTesrtebwm6gNBdRbTIylCgrX6jx6ytPmuOYFXgExwIhIKhVfglyI8nZojoHTa YrfAH8Xz1dmgNUQdcp2LyvQC0aeSUh21rpksusrEBKyrzS71WNdtGog+GUq0A7vLMXEF BPaq8KY2ps7B0y8NjbXTKlZWN3F1KRD2eJUOjPoYRMnbyr6OICinzVT/v0KxuVf5X7Pl BlGsyWm3/tQC4r7zz2+VSWNVirg6xJO6bLIRRQfacSe2cGB6pbzr/JT4PV8vbO2UzuJQ z6R+KnV3O/IuMzYFbzua9AdzFzGpzqrn2gLkp0CJFGCqU3utHe2fQM/8wDUHcoJk0lbj u8/w== X-Gm-Message-State: AFeK/H3QTuy1NCKMfizJSWdEuH64lNB3H5hKxSsFUa1tIOP1ikdsFrYwXBgBIwTCKR1wAA== X-Received: by 10.84.212.8 with SMTP id d8mr82135131pli.152.1491998215161; Wed, 12 Apr 2017 04:56:55 -0700 (PDT) Received: from [192.168.2.211] ([116.56.129.146]) by smtp.gmail.com with ESMTPSA id b10sm5238515pfc.27.2017.04.12.04.56.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Apr 2017 04:56:54 -0700 (PDT) Subject: Re: Understanding the FreeBSD locking mechanism To: Chris Torek , imp@bsdimp.com References: <201704120755.v3C7tYUH016699@elf.torek.net> Cc: ed@nuxi.nl, freebsd-hackers@freebsd.org, rysto32@gmail.com From: Yubin Ruan Message-ID: Date: Wed, 12 Apr 2017 19:56:50 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <201704120755.v3C7tYUH016699@elf.torek.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Apr 2017 11:56:55 -0000 On 2017年04月12日 15:55, Chris Torek wrote: >> clear. How can I distinguish these two conditions? I mean, whether I >> am using my own state/stack or borrowing others' state. > > You choose it when you establish your interrupt handler. If you > say you are a filter interrupt, then you *are* one, and the rest > of your code must be written as one. Unless you know what you > are doing, don't do this, and then you *aren't* one and the rest > of your code can be written using the much more relaxed model. > >> What do you mean by "must take MTX_SPIN locks 'inside' any outer >> MTX_DEF locks? > > This means that any code path that is going to hold a spin-type > lock must obtain it while already holding any applicable non-spin > locks. For instance, if we look at we find these: > > #define PROC_STATLOCK(p) mtx_lock_spin(&(p)->p_statmtx) > #define PROC_ITIMLOCK(p) mtx_lock_spin(&(p)->p_itimmtx) > #define PROC_PROFLOCK(p) mtx_lock_spin(&(p)->p_profmtx) > > Let's find a bit of code that uses one, such as in kern_time.c: > > https://github.com/freebsd/freebsd/blob/master/sys/kern/kern_time.c#L338 > > (kern_clock_gettime()). This code reads: > > case CLOCK_PROF: > PROC_LOCK(p); > PROC_STATLOCK(p); > calcru(p, &user, &sys); > PROC_STATUNLOCK(p); > PROC_UNLOCK(p); > timevaladd(&user, &sys); > TIMEVAL_TO_TIMESPEC(&user, ats); > break; > > Note that the call to PROC_LOCK comes first, then the call to > PROC_STATLOCK. This is because PROC_LOCK > > https://github.com/freebsd/freebsd/blob/master/sys/sys/proc.h#L825 > > is defined as: > > #define PROC_LOCK(p) mtx_lock(&(p)->p_mtx) > > If you obtain the locks in the other order -- i.e., if you grab > the PROC_STATLOCK first, then try to lock PROC_LOCK -- you are > trying to take a spin-type mutex while holding a default mutex, Is this a typo? I guess you mean something like "you are trying to take a blocking mutex while holding spin-type mutex". > and this is not allowed (can cause deadlock). But taking the > PROC_LOCK first (which may block), then taking the PROC_STATLOCK > (a spin lock) "inside" the outer PROC_LOCK default mutex, is OK. I think I get your point: if you take a spin-type mutex, you already disable interrupt, which in effect means that no other code can preempt you. Under this circumstance, if you continue to take a blocking mutex, you may get blocked. Since you already disable interrupt and nobody can interrupt/preempt you, you are blocked on that CPU, not being able to do anything, which is pretty much a "deadlock" (actually this is not a deadlock, but, it is similar) Regards, Yubin Ruan > (This is one of my mild objections to macros like PROC_LOCK and > PROC_STATLOCK: they hide whether the mutex in question is a spin > lock.) > > Incidentally, any time you take *any* lock while holding any > other lock (e.g., lock A, then lock B while holding A), you have > created a "lock order" in which A predeces B. If some other > code path locks B first, then while holding B, attempts to lock > A, you get a deadlock if both code paths are running at the same > time. The WITNESS code dynamically discovers these various orders > and warns you at run time if you have a "lock order reversal" > (a case where one code path does A-then-B while another does > B-then-A). > > (This is, in a sense, the same problem as discovering whether > there is a loop in a directed graph, or whether this directed > graph is acyclic. If you can force the graph to take the shape of > a tree, rather than the more general graph, there will never be > any loops in it, and you will never have lock order reversals. > And of course if you have only *one* lock for some data, there is > nothing to be reversed. Not all lock order reversals are > guaranteed to lead to deadlock, but sorting out which ones are > really OK, and which are not, is ... challenging.) > > Chris >