From owner-freebsd-arch@freebsd.org Thu Dec 15 21:50:58 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19B27C8230D for ; Thu, 15 Dec 2016 21:50:58 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EDAA435F for ; Thu, 15 Dec 2016 21:50:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by mail.baldwin.cx (Postfix) with ESMTPSA id C83A610AA27; Thu, 15 Dec 2016 16:50:55 -0500 (EST) From: John Baldwin To: Roger Pau =?ISO-8859-1?Q?Monn=E9?= Cc: freebsd-arch@freebsd.org Subject: Re: Order of device suspend/resume Date: Thu, 15 Dec 2016 13:38:11 -0800 Message-ID: <7469755.xT5lfhErkd@ralph.baldwin.cx> User-Agent: KMail/4.14.10 (FreeBSD/11.0-PRERELEASE; KDE/4.14.10; amd64; ; ) In-Reply-To: <20161215114033.r33nt3fqhnfi7hqw@dhcp-3-221.uk.xensource.com> References: <20161215114033.r33nt3fqhnfi7hqw@dhcp-3-221.uk.xensource.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Thu, 15 Dec 2016 16:50:55 -0500 (EST) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2016 21:50:58 -0000 On Thursday, December 15, 2016 11:40:33 AM Roger Pau Monn=E9 wrote: > Hello, >=20 > I'm currently dealing with a bug in the Xen suspend/resume sequence, = and I've=20 > found that lacking a way to order device priority during suspend/resu= me is=20 > proving quite harmful for Xen (and maybe other systems too). The curr= ent=20 > suspend/resume code simply scans the root bus, and suspends/resumes e= very device=20 > based on the order they are attached to their parents. The problem he= re is that=20 > there's no way to tell that some devices should be resumed before oth= ers, for=20 > example the event timers/time counters/uarts should definitely be res= ume before=20 > other devices, but that's seems to happens mostly out of chance. >=20 > Currently most time related devices are attached directly to the nexu= s, which=20 > means they will get resumed first, but for example the uart is curren= tly=20 > attached to the pci bus IIRC, which means it gets resumed quite late.= On Xen=20 > systems, this is even worse. The Xen PV bus (that contains all Xen-re= lated=20 > devices) is attached the last one (because it tends to pick up unused= memory=20 > regions for it's own usage) and this bus also contains the PV timecou= nter which=20 > should be resumed _before_ other devices, or else timecounting will b= e=20 > completely screwed and things can get stuck in indefinitely long loop= s (due to=20 > the fact that the timecounter is implemented based on the uptime of t= he host,=20 > and that changes from host-to-host). >=20 > In order to solve this I could add a hack to the Xen resume process (= which is=20 > already different from the ACPI one), but this looks gross. I could a= lso attach=20 > the Xen PV timer to the nexus directly (as it was done before), but I= also=20 > prefer to keep all Xen-related devices in the same bus for coherency.= Last=20 > option would be to add some kind of suspend/resume priorities to the = devices,=20 > and do more than one suspend/resume pass. This is more complex and re= quires more=20 > changes, so I would like to know if it would be helpful for other sys= tems, or if=20 > someone has already attempted to do it. I think Justin Hibbits had some patches to make use of the boot-time ne= w-bus passes for suspend and resume which I think would help with this. You = suspend things in the reverse order of boot and resume operates in the same ord= er as boot. --=20 John Baldwin