From owner-freebsd-arch@FreeBSD.ORG Sat Nov 15 10:58:26 2014 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48EE5D8B; Sat, 15 Nov 2014 10:58:26 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFE1D9B1; Sat, 15 Nov 2014 10:58:25 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sAFAwJF3044371 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 15 Nov 2014 12:58:19 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sAFAwJF3044371 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id sAFAwJVR044370; Sat, 15 Nov 2014 12:58:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 15 Nov 2014 12:58:19 +0200 From: Konstantin Belousov To: Andriy Gapon Subject: Re: suspending threads before devices [Was: svn commit: r233249 - head/sys/amd64/acpica] Message-ID: <20141115105819.GJ17068@kib.kiev.ua> References: <201203202037.q2KKbNfK037014@svn.freebsd.org> <201203211502.14353.jkim@FreeBSD.org> <4F6AF1CB.80902@FreeBSD.org> <201203220748.49635.jhb@freebsd.org> <20120322141436.GC2358@deviant.kiev.zoral.com.ua> <54666FD5.6080705@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54666FD5.6080705@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: "src-committers@freebsd.org" , Jung-uk Kim , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Nov 2014 10:58:26 -0000 On Fri, Nov 14, 2014 at 11:10:45PM +0200, Andriy Gapon wrote: > On 22/03/2012 16:14, Konstantin Belousov wrote: > > I already noted this to Jung-uk, I think that current suspend handling > > is (somewhat) wrong. We shall not stop other CPUs for suspension when > > they are executing some random kernel code. Rather, CPUs should be safely > > stopped at the kernel->user boundary, or at sleep point, or at designated > > suspend point like idle loop. > > > > We already are engaged into somewhat doubtful actions like restoring of %cr2, > > since we might, for instance, preemt page fault handler with suspend IPI. > > I recently revisited this issue in the context of some suspend+resume problems > that I am having with radeonkms driver. What surprised me is that the driver's > suspend code has no synchronization whatsoever with its other code paths. So, I > looked first at the Linux code and then at the illumos code to see how suspend > is implemented there. > As far as I can see, those kernels do exactly what you suggest that we do. > Before suspending devices they first suspend all threads except for one that > initiates the suspend. For userland threads a signal-like mechanism is used to > put them in a state similar to SIGSTOP-ed one. With the kernel threads > mechanisms are different between the kernels. Also, illumos freezes kernel > threads after suspending the devices, not before. > > I think that we could start with only the userland threads initially. Do you > think the SIGSTOP-like approach would be hard to implement for us? We have most, if not all, parts of the stopping code already implemented. I mean the single-threading code, see thread_single(SINGLE_BOUNDARY). The code ensures that other threads in the current process are stopped either at the kernel->user boundary, or at the safe kernel sleep point. This is not immediately applicable, since the caller is supposed to be a thread in the suspended process, but modifications to allow external process to do the same are really small comparing with the complexity of the code. I suspect that all what is needed is change of while/if (remaining != 1) to while/if ((p == curproc && remaining != 1) || (p != curproc && remaining != 0)) together with explicit passing of struct proc *p to thread_single. > > References: > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/cpr/cpr_main.c#425 > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/cpr/cpr_uthread.c#80 > > http://lxr.free-electrons.com/source/kernel/power/suspend.c#L388 > http://lxr.free-electrons.com/source/kernel/power/suspend.c#L207 > http://lxr.free-electrons.com/source/kernel/power/power.h#L235 > http://lxr.free-electrons.com/source/kernel/power/process.c#L118 > http://lxr.free-electrons.com/source/kernel/power/process.c#L27 > http://lxr.free-electrons.com/source/kernel/freezer.c#L115 > > -- > Andriy Gapon