From owner-freebsd-arch@freebsd.org Fri Dec 16 05:25:07 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1E21C80965 for ; Fri, 16 Dec 2016 05:25:07 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B33CAE58 for ; Fri, 16 Dec 2016 05:25:07 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x234.google.com with SMTP id p42so89158048ioo.1 for ; Thu, 15 Dec 2016 21:25:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=AxIreycXh+suxG3cP8Cy+ivz1G83Gj9U63TnPZ2wbSY=; b=vLkMar2TEum5Yh4QHK3W71Q6CpMi7kHLUZCLVe9oWGp9DQqe+uK3Gz+oQel3JguAEa D7GE3kRp8PBJZuyIVhEr2euemJLvFCm5zJO2OZg9Z9ybVzRJyc+ARzaWuOFDuhMR31N8 ELfaE7ZvgE1IsYHJl4vPTS6N0gvo0ViUtLZA8/64OdwhhL1JBXYivz4fttyHSIJ2pMl/ o4Ef6L7uGwvXV8pAGWiYSdguw22VHMdOegSc0STW802frSjWGkLTijs9kKwV5vECbrMr XYat6MRg4ifcy/wRuuE8AEHj1jbEIkzQJfHK/n12wjeLxj8hmJ3q1hV5bFbCthorBRxU lukA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=AxIreycXh+suxG3cP8Cy+ivz1G83Gj9U63TnPZ2wbSY=; b=f8sxsgky/83sRGuTAIVYi8jCsnAlRo8QhI78jLXUUFN08fOCIAqW88XlKma6sYqSNb yVtfRZko4FPWIx52M7HF30EVBu99OCPEgRqM6ezcWh4Eu5Sr5IujsbFI+g5iHw9LoaTy EpVhDw1PQqJMLSayPow9gLlUNTJrkanJhHkLmoqirOV06/mCbMdw6+Rcn9/cTYMuquuc h2NYXImdnPFSDMG2AvBx3u2itBG8PkyFrt+ojHokOrcJTJfkA63gWWx5pDcRDhn6zDiY NmgcTugrTfhKdDvh+xu/7sNR0WzmKStDOhNo9qyWKoWJvKvIuPZFupkaZA3nOqnFAELK Urdg== X-Gm-Message-State: AIkVDXLCDmhINoLIjcNeRAXTUqVehallJrpq81zkrE7GxwTODp18/5HL3K7DOH5fSK++fUGFaTsPZ4K7bw32tw== X-Received: by 10.107.132.74 with SMTP id g71mr1183275iod.19.1481865905903; Thu, 15 Dec 2016 21:25:05 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.22.135 with HTTP; Thu, 15 Dec 2016 21:25:05 -0800 (PST) X-Originating-IP: [69.53.245.200] In-Reply-To: References: <20161215114033.r33nt3fqhnfi7hqw@dhcp-3-221.uk.xensource.com> <7469755.xT5lfhErkd@ralph.baldwin.cx> From: Warner Losh Date: Thu, 15 Dec 2016 21:25:05 -0800 X-Google-Sender-Auth: Ad4JEYgOFgBNQCNzJ56OTOqkOSs Message-ID: Subject: Re: Order of device suspend/resume To: Justin Hibbits Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , FreeBSD Arch Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2016 05:25:08 -0000 On Thu, Dec 15, 2016 at 8:34 PM, Justin Hibbits wrot= e: > > On Dec 15, 2016, at 3:38 PM, John Baldwin wrote: > >> On Thursday, December 15, 2016 11:40:33 AM Roger Pau Monn=C3=A9 wrote: >>> >>> Hello, >>> >>> I'm currently dealing with a bug in the Xen suspend/resume sequence, an= d >>> I've >>> found that lacking a way to order device priority during suspend/resume >>> is >>> proving quite harmful for Xen (and maybe other systems too). The curren= t >>> suspend/resume code simply scans the root bus, and suspends/resumes eve= ry >>> device >>> based on the order they are attached to their parents. The problem here >>> is that >>> there's no way to tell that some devices should be resumed before other= s, >>> for >>> example the event timers/time counters/uarts should definitely be resum= e >>> before >>> other devices, but that's seems to happens mostly out of chance. >>> >>> Currently most time related devices are attached directly to the nexus, >>> which >>> means they will get resumed first, but for example the uart is currentl= y >>> attached to the pci bus IIRC, which means it gets resumed quite late. O= n >>> Xen >>> systems, this is even worse. The Xen PV bus (that contains all >>> Xen-related >>> devices) is attached the last one (because it tends to pick up unused >>> memory >>> regions for it's own usage) and this bus also contains the PV timecount= er >>> which >>> should be resumed _before_ other devices, or else timecounting will be >>> completely screwed and things can get stuck in indefinitely long loops >>> (due to >>> the fact that the timecounter is implemented based on the uptime of the >>> host, >>> and that changes from host-to-host). >>> >>> In order to solve this I could add a hack to the Xen resume process >>> (which is >>> already different from the ACPI one), but this looks gross. I could als= o >>> attach >>> the Xen PV timer to the nexus directly (as it was done before), but I >>> also >>> prefer to keep all Xen-related devices in the same bus for coherency. >>> Last >>> option would be to add some kind of suspend/resume priorities to the >>> devices, >>> and do more than one suspend/resume pass. This is more complex and >>> requires more >>> changes, so I would like to know if it would be helpful for other >>> systems, or if >>> someone has already attempted to do it. >> >> >> I think Justin Hibbits had some patches to make use of the boot-time >> new-bus >> passes for suspend and resume which I think would help with this. You >> suspend >> things in the reverse order of boot and resume operates in the same orde= r >> as >> boot. >> >> -- >> John Baldwin > > > John is right. I have a (somewhat abandoned due to time and focus) branc= h, > https://svnweb.freebsd.org/base/projects/pmac_pmu/ which has the necessar= y > code working mostly on PowerPC. The diff can be found at > https://reviews.freebsd.org/D203 too. Cool. Does it have a mechanism similar to the attach code that lets you run again at each pass? Warner