From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 23 09:48:56 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1BC84BFD; Sat, 23 Mar 2013 09:48:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E8C2FD32; Sat, 23 Mar 2013 09:48:54 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA15528; Sat, 23 Mar 2013 11:48:53 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1UJL4G-000Dtf-O0; Sat, 23 Mar 2013 11:48:52 +0200 Message-ID: <514D7A82.9000105@FreeBSD.org> Date: Sat, 23 Mar 2013 11:48:50 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130321 Thunderbird/17.0.4 MIME-Version: 1.0 To: FreeBSD Current , freebsd-hackers@FreeBSD.org, freebsd-acpi@FreeBSD.org Subject: Re: call suspend_cpus() under smp_ipi_mtx References: <5114AB2E.2050909@FreeBSD.org> In-Reply-To: <5114AB2E.2050909@FreeBSD.org> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=x-viet-vps Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2013 09:48:56 -0000 Looks like this issue needs more thinking and discussing. The basic idea is that suspend_cpus() must be called with smp_ipi_mtx held (on SMP systems). This is for exactly the same reasons as to why we first take smp_ipi_mtx before calling stop_cpus() in the shutdown path. Essentially one CPU could be holding smp_ipi_mtx (and thus with interrupts disabled[*]) and waiting for an acknowledgement from other CPUs (e.g. in smp_rendezvous or in a TLB shootdown), while another CPU could be with interrupts disabled (explicitly - like in the shutdown or ACPI suspend paths) and trying to deliver an IPI to other CPUs. In my opinion, we must consistently use the same lock, smp_ipi_mtx, for all regular (non-NMI) synchronous IPI-based communication between CPUs. Otherwise a deadlock is quite possible. Some obstacles for just going ahead and making the suggested change: - acpi_sleep_machdep() calls intr_suspend() with interrupts disabled; currently witness(9) is not aware of that, but if smp_ipi_mtx spin-lock is used, then we would have to make intr_table_lock and msi_lock the spin-locks as well; - AcpiLeaveSleepStatePrep() (from ACPICA) is called with interrupts disabled and currently it performs an action that requires memory allocation; again, with interrupts disabled via intr_disable() this fact is not visible to witness, etc, but with smp_ipi_mtx it needs to be somehow handled. I talked to ACPICA guys about the last issue and they told me that what is currently done in AcpiLeaveSleepStatePrep does not need to be with interrupts disabled and can be moved to AcpiLeaveSleepState. This is after the _BFS and _GTS support was removed. What do you think? Thank you. -- Andriy Gapon