From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 1 15:36:00 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id ECEA1715; Mon, 1 Apr 2013 15:36:00 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id CAD90DC6; Mon, 1 Apr 2013 15:36:00 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id F406DB91C; Mon, 1 Apr 2013 11:35:59 -0400 (EDT) From: John Baldwin To: freebsd-acpi@freebsd.org Subject: Re: call suspend_cpus() under smp_ipi_mtx Date: Mon, 1 Apr 2013 10:52:18 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <5114AB2E.2050909@FreeBSD.org> <514D7A82.9000105@FreeBSD.org> In-Reply-To: <514D7A82.9000105@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201304011052.18370.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 01 Apr 2013 11:36:00 -0400 (EDT) Cc: freebsd-hackers@freebsd.org, FreeBSD Current , Andriy Gapon X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2013 15:36:01 -0000 On Saturday, March 23, 2013 5:48:50 am Andriy Gapon wrote: > > Looks like this issue needs more thinking and discussing. > > The basic idea is that suspend_cpus() must be called with smp_ipi_mtx held (on > SMP systems). > This is for exactly the same reasons as to why we first take smp_ipi_mtx before > calling stop_cpus() in the shutdown path. Essentially one CPU could be holding > smp_ipi_mtx (and thus with interrupts disabled[*]) and waiting for an > acknowledgement from other CPUs (e.g. in smp_rendezvous or in a TLB shootdown), > while another CPU could be with interrupts disabled (explicitly - like in the > shutdown or ACPI suspend paths) and trying to deliver an IPI to other CPUs. > > In my opinion, we must consistently use the same lock, smp_ipi_mtx, for all > regular (non-NMI) synchronous IPI-based communication between CPUs. Otherwise a > deadlock is quite possible. > > Some obstacles for just going ahead and making the suggested change: > > - acpi_sleep_machdep() calls intr_suspend() with interrupts disabled; currently > witness(9) is not aware of that, but if smp_ipi_mtx spin-lock is used, then we > would have to make intr_table_lock and msi_lock the spin-locks as well; > - AcpiLeaveSleepStatePrep() (from ACPICA) is called with interrupts disabled and > currently it performs an action that requires memory allocation; again, with > interrupts disabled via intr_disable() this fact is not visible to witness, etc, > but with smp_ipi_mtx it needs to be somehow handled. > > I talked to ACPICA guys about the last issue and they told me that what is > currently done in AcpiLeaveSleepStatePrep does not need to be with interrupts > disabled and can be moved to AcpiLeaveSleepState. This is after the _BFS and > _GTS support was removed. > > What do you think? > Thank you. Hmm, I think intr_table_lock used to be a spin lock at some point. I don't remember why we changed it to a regular mutex. It may be that there was a lock order reason for that. :( -- John Baldwin