From owner-freebsd-arm@FreeBSD.ORG Sun Jan 19 03:23:54 2014 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2412E94D; Sun, 19 Jan 2014 03:23:54 +0000 (UTC) Received: from smtpauth4.wiscmail.wisc.edu (wmauth4.doit.wisc.edu [144.92.197.145]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DE716190E; Sun, 19 Jan 2014 03:23:53 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from avs-daemon.smtpauth4.wiscmail.wisc.edu by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MZM00600OYGMO00@smtpauth4.wiscmail.wisc.edu>; Sat, 18 Jan 2014 21:23:53 -0600 (CST) X-Spam-PmxInfo: Server=avs-4, Version=6.0.3.2322014, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.19.31515, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from wanderer.tachypleus.net (adsl-76-208-68-77.dsl.mdsnwi.sbcglobal.net [76.208.68.77]) by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MZM0080HPFQZ500@smtpauth4.wiscmail.wisc.edu>; Sat, 18 Jan 2014 21:23:52 -0600 (CST) Message-id: <52DB4546.504@freebsd.org> Date: Sat, 18 Jan 2014 21:23:50 -0600 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 To: Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> <52D89DC9.7050303@freebsd.org> <52DB1138.6010804@linaro.org> <3AE8EDE6-D931-4F93-9BF7-ABFB297B5B96@bsdimp.com> <52DB3FFD.2070503@freebsd.org> <8C16C019-B9AF-417B-9B02-C016A202AAC7@bsdimp.com> In-reply-to: <8C16C019-B9AF-417B-9B02-C016A202AAC7@bsdimp.com> X-Enigmail-Version: 1.6 Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org, roger.pau@citrix.com X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jan 2014 03:23:54 -0000 On 01/18/14 21:08, Warner Losh wrote: > On Jan 18, 2014, at 8:01 PM, Nathan Whitehorn wrote: > >> On 01/18/14 20:44, Warner Losh wrote: >>> On Jan 18, 2014, at 4:41 PM, Julien Grall wrote: >>> >>>> Hello Nathan, >>>> >>>> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote: >>>>> On 01/16/14 18:36, Julien Grall wrote: >>>>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: >>>>>> As I understand, only the simple bus code (see simplebus_attach) is >>>>>> translating the interrupts in the device on a resource. >>>>>> So if you have a node directly attached to the root node with >>>>>> interrupts and MMIO, the driver won't be able to retrieve and >>>>>> translate the interrupts via bus_alloc_resources. >>>>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this. >>>> I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ. >>>> >>>> This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306). >>>> >>>> With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD: >>>> >>>> / { >>>> >>>> mybus { >>>> compatible = "simple-bus"; >>>> >>>> mynode { >>>> interrupt-parent = &gic; >>>> interrupts = <...>; >>>> }; >>>> >>>> gic: gic@xxxx { >>>> interrupt-controller; >>>> } >>>> }; >>>> }; >>>> >>>> The node "mynode" will have to move after the GIC to be able to work correctly. >>> This stems from a difference in enumeration between FreeBSD and Linux. FreeBSD enumerates the devices in DTB order, while Linux does a partial ordering based on dependencies. >>> >>> Warner >> Enumerating in some other order doesn't necessarily help: since the >> interrupt and bus trees are independent, circular dependencies can >> happen. This is not a hypothetical: on most powermacs, the main >> interrupt controller is a functional unit on a PCI device -- a PCI >> device whose other units have interrupt lines that eventually connect >> back to itself. There is no way to fix that with ordering. So I think we >> still need to defer interrupt setup. It's not that bad -- PPC already >> does this to handle the powermac case. > I guess I've looked at simpler cases where interrupts and GPIO pins need to be up before anything can work on Atmel... We kinda fake it now, but there's some ordering issues that are solved in this way. But I've not finished the work on bringing Atmel into the FDT world yet. Deferred setup isn't always an option, but I'll keep that in mind in case I hit that case... > > The other way to cope is to use the multi-pass enumeration stuff that John Baldwin put into the tree a while ago. In that case, you could enumerate bridges, interrupt controllers, gpio providers, etc first, and then do a second pass that catches the rest of the devices and the interrupt processing for the first pass devices. This is a variation on the deferred enumeration stuff you are talking about, so that might be a better, more general solution to these sorts of problems. > > Warner > > I'm still not sure that helps. Take the powermac case. The PCI parent bridge assigns and routes interrupts as part of its probe stage. One device (called mac-io) is the one that actually has the interrupt controller, along with other things like ATA and sound. It has several interrupts to support e.g. ATA. How can the PCI bus attachment sensibly attach the macio device? If it delays attaching it until the interrupt controller registers, the bus probing will deadlock, since the interrupt controller is itself a child of macio! If it attaches it immediately, it will not be able to route the interrupts of the macio device. It's a catch-22. The only solution we were able to come up with when this situation arose was to treat the bus and interrupt trees as entirely separate things, built independently, which allows the kernel to break the loop. GPIOs can have similar problems. The basic issue is that newbus ultimately assumes that the system topology can be described by a single tree. Interconnections between the branches -- especially if it isn't even a DAG -- fundamentally break the model and multipass doesn't necessarily help. -Nathan