From owner-freebsd-sparc64@freebsd.org Sat Sep 19 23:05:45 2015 Return-Path: Delivered-To: freebsd-sparc64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6EFF9CF7F1 for ; Sat, 19 Sep 2015 23:05:45 +0000 (UTC) (envelope-from mark.cave-ayland@ilande.co.uk) Received: from s16892447.onlinehome-server.info (s16892447.onlinehome-server.info [82.165.15.123]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0A21964; Sat, 19 Sep 2015 23:05:44 +0000 (UTC) (envelope-from mark.cave-ayland@ilande.co.uk) Received: from cpc2-slam8-2-0-cust642.2-4.cable.virginm.net ([82.24.206.131] helo=[192.168.0.5]) by s16892447.onlinehome-server.info with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1ZdRCM-0004gM-Jp; Sun, 20 Sep 2015 00:05:41 +0100 Message-ID: <55FDEA3C.1010804@ilande.co.uk> Date: Sun, 20 Sep 2015 00:05:32 +0100 From: Mark Cave-Ayland User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.8.0 MIME-Version: 1.0 To: Marius Strobl CC: Alexey Dokuchaev , "freebsd-sparc64@freebsd.org" References: <55EDFE00.9090109@ilande.co.uk> <20150913022143.GA7862@alchemy.franken.de> <20150913103940.GA60101@FreeBSD.org> <20150913180126.GC7862@alchemy.franken.de> <55F89861.1030107@ilande.co.uk> <20150916031030.GA6711@FreeBSD.org> <55F9C2B8.7030605@ilande.co.uk> <20150916211914.GD18789@alchemy.franken.de> <20150917082817.GA71811@FreeBSD.org> <55FBB662.4080708@ilande.co.uk> <20150919211420.GK18789@alchemy.franken.de> In-Reply-To: <20150919211420.GK18789@alchemy.franken.de> Content-Type: multipart/mixed; boundary="------------060504080405090800060005" X-SA-Exim-Connect-IP: 82.24.206.131 X-SA-Exim-Mail-From: mark.cave-ayland@ilande.co.uk Subject: Re: PCI range checking under qemu-system-sparc64 X-SA-Exim-Version: 4.2.1 (built Sun, 08 Jan 2012 02:45:44 +0000) X-SA-Exim-Scanned: No (on s16892447.onlinehome-server.info); Unknown failure X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Sep 2015 23:05:45 -0000 This is a multi-part message in MIME format. --------------060504080405090800060005 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit On 19/09/15 22:14, Marius Strobl wrote: > On Fri, Sep 18, 2015 at 07:59:46AM +0100, Mark Cave-Ayland wrote: >> On 17/09/15 09:28, Alexey Dokuchaev wrote: >> >>> On Wed, Sep 16, 2015 at 11:19:15PM +0200, Marius Strobl wrote: >>>> [...] >>>> Which suggest that the next thing to investigate is the CMD646 >>>> emulation. Is there a particular reason why QEMU emulates a >>>> CMD646U rather than a plain CMD646 as found in the real sun4u >>>> machines of the USIIe/i era? >>>> >>>> Alexey, does building the port with CDROM_DMA disabled make >>>> a difference? >>> >>> Ironically I had it already disabled prior to your question; but I've >>> rebuilt the port enabling it for completeness' sake. It did not make >>> a difference. >>> >>> Then I've disabled all CAM/ATA stuff (scbus, ata, umass, etc.) in the >>> kernel config and that's what I see now (this is with CDROM_DMA=on): >> >> What does the CAM/ATA stuff do here? Does this mean it may not >> necessarily be an interrupt issue if you can get to mounting the root fs >> with CDROM_DMA=on? > > I think we are looking at multiple issues here. First off, based > on the interrupt-map provided by OpenBIOS, we route the intpin of > the ATA controller to INO 20 (which uses the interrupt mapping > register at offset 0xc28). If I additionally enable the code for > debugging interrupt routing problems, which just clears all > interrupts handle by the Sabre and then in all its interrupt > mapping registers enables all INOs by writing the valid bit, > Sabre IGN and CPU module ID there, I see 5 interrupts for INO > 20 at the time the kernel hangs under QEMU. I don't get any > stray interrupts for INOs that we're not actively routing. > Based on what Alexey wrote, he doesn't see interrupts for INO 20, > i. e. the ATA controller, with a kernel not including the debug > code. That doesn't make sense to me. Without the debug code, we > still enable the INOs for which drivers request them when they > attach to devices the same way as described above, just not > unconditionally for all interrupts handle by the Sabre. So > there should be no difference for correctly routed interrupts. > > Second, even when I see interrupts for the ATA controller, > enumeration of storage devices hangs somehow. On a real > machine when booting verbose, this looks like (mainly output > from ata_generic_reset(): > <...> > IPsec: Initialized Security Association Processing. > lo0: bpf attached > ata2: reset tp1 mask=00 ostat0=ff ostat1=ff > ata3: reset tp1 mask=03 ostat0=50 ostat1=00 > ata3: stat0=0x80 err=0x80 lsb=0x80 msb=0x80 > ata3: stat0=0x50 err=0x01 lsb=0x00 msb=0x00 > ata3: stat1=0x00 err=0x01 lsb=0x14 msb=0xeb > ata3: reset tp2 stat0=50 stat1=00 devices=0x20001 > GEOM: new disk cd0 > cd0 at ata3 bus 0 scbus1 target 1 lun 0 > cd0: Removable CD-ROM SCSI device > cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) > cd0: Attempt to query device size failed: NOT READY, Medium not present > <...> > > With QEMU: > <...> > IPsec: Initialized Security Association Processing. > lo0: bpf attached > ata2: reset tp1 mask=03 ostat0=00 ostat1=00 > ata2: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 > ata2: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 > ata2: reset tp2 stat0=00 stat1=00 devices=0x0 > ata3: reset tp1 mask=03 ostat0=50 ostat1=00 > ata3: stat0=0x00 err=0x01 lsb=0x14 msb=0xeb > ata3: stat1=0x00 err=0x00 lsb=0xff msb=0xff > ata3: reset tp2 stat0=00 stat1=00 devices=0x10000 > > <...> > db> show intrcnt > pil3: ithrd 5 > vec2004: atapci0 5 > QEMU: Terminated > <...> > > So after a reset, real iron additionally sets the READY and > SERVICE bits in the ATA status register but that doesn't > explain the hang. Code that is waiting for READY to get set > also uses a timeout. > > I patched QEMU to identify the ATA controller as a plain > CMD646 one so the UDMA isn't tried in the first place. I > also limited the ATA mode used by the kernel to PIO4 at > maximum. Neither made a difference, i. e. the hang still > occurs. > > Another striking thing is that the interrupt statistics > show to CPU tick interrupts. I'm unsure at which point > the kernel actually enables them but they really should > have been engaged at the time the hang occurs, otherwise > scheduling won't work properly, which also might be an > explanation for the hang, i. e. the kernel might never > switch to the CAM thread(s). > > All in all, interrupts in QEMU seem buggy in one way > or another. Thanks for looking into this in detail Marius - plenty of information to start debugging this further. While I don't have any insight on the CPU tick interrupt yet, my initial feeling is that the ATA hang could be related to the PCI interrupt clearing issue that I started looking into a while back. Although it isn't a complete fix, does the attached patch against QEMU help at all? Otherwise it will require a deeper dive into the QEMU interrupt emulation. ATB, Mark. --------------060504080405090800060005 Content-Type: text/x-diff; name="apb-no-clear.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="apb-no-clear.patch" diff --git a/hw/pci-host/apb.c b/hw/pci-host/apb.c index 599768e..714e654 100644 --- a/hw/pci-host/apb.c +++ b/hw/pci-host/apb.c @@ -454,9 +454,10 @@ static void apb_config_writel (void *opaque, hwaddr addr, break; case 0x1400 ... 0x14ff: /* PCI interrupt clear */ if (addr & 4) { - unsigned int ino = (addr & 0xff) >> 5; - if ((s->irq_request / 4) == ino) { - pbm_clear_request(s, s->irq_request); + unsigned int ino = (addr & 0xff) >> 3; + if (s->irq_request == ino) { + s->pci_irq_in &= ~(1ULL << ino); + s->irq_request = NO_IRQ_REQUEST; pbm_check_irqs(s); } } @@ -465,7 +466,8 @@ static void apb_config_writel (void *opaque, hwaddr addr, if (addr & 4) { unsigned int ino = ((addr & 0xff) >> 3) | 0x20; if (s->irq_request == ino) { - pbm_clear_request(s, ino); + s->pci_irq_in &= ~(1ULL << ino); + s->irq_request = NO_IRQ_REQUEST; pbm_check_irqs(s); } } --------------060504080405090800060005--