From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 14:42:28 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19248106566C for ; Wed, 7 Jan 2009 14:42:28 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.157]) by mx1.freebsd.org (Postfix) with ESMTP id 9ED5E8FC1B for ; Wed, 7 Jan 2009 14:42:27 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so2726264fgb.35 for ; Wed, 07 Jan 2009 06:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=zCJl8CeBMqWSjGeQnOK9bDg21Dy8pYt2LQid6VYu1h4=; b=e3Jiq5FihX4PFNLSdKULzK+FC5mBG7SkLogTT3Dfz0sUm24TZpIHJtFCJk+N/wJ0pa xyLjHilHM7Ih2E9VMKWk87cRWUxWgr5AJ43OZaLKxarpoi9eRFEVV34FeCxN7Y0NNv5B v5957gU2W9I6lELiJBqqWXchcRgm0OQB5jY8o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=KyplsJPyEMpkGJOo1J3rn20VxK8iO15ANhS+qGDcsqpDpVKXPBaBkKdJRvKT+/kmQs VxWTBvCfNNHHoHSuSBpTiZkP4vaMHiKOeLFCftxUOfjzmcfbAO9+dI9a4D9bW7Ot1uQi 6wcbKpgOJQxU11JcTUPzZX9xzi4MwtoPJSKUs= Received: by 10.86.23.18 with SMTP id 18mr13534807fgw.6.1231337993150; Wed, 07 Jan 2009 06:19:53 -0800 (PST) Received: by 10.86.95.15 with HTTP; Wed, 7 Jan 2009 06:19:53 -0800 (PST) Message-ID: Date: Wed, 7 Jan 2009 16:19:53 +0200 From: "Jacques Fourie" To: freebsd-arm@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: hwpmc on xscale (pxa255) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 14:42:28 -0000 I am interested in making hwpmc work on my gumstix (pxa255 cpu). Does anyone have working examples of pmc_save_kernel_callchain() and pmc_save_user_callchain() for arm platforms that they are willing to share? From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 17:39:21 2009 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C260106566B for ; Wed, 7 Jan 2009 17:39:21 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id 3DA0C8FC12 for ; Wed, 7 Jan 2009 17:39:21 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from DSPAM-Daemon (localhost [127.0.0.1]) by mx0.deglitch.com (Postfix) with SMTP id 3F8EC8FC52 for ; Wed, 7 Jan 2009 20:26:40 +0300 (MSK) Received: from orion.SpringDaemons.com (drsun1.dialup.corbina.ru [85.21.245.235]) by mx0.deglitch.com (Postfix) with ESMTPA id 6CD128FC4E; Wed, 7 Jan 2009 20:26:38 +0300 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id CFDC9398F3; Wed, 7 Jan 2009 20:29:06 +0300 (MSK) Date: Wed, 7 Jan 2009 20:29:01 +0300 From: Stanislav Sedov To: Hans Petter Selasky Message-Id: <20090107202901.21df29e3.stas@FreeBSD.org> In-Reply-To: <200901071751.49494.hselasky@c2i.net> References: <200901040012.n040C2gH040928@svn.freebsd.org> <200901071519.23311.hselasky@c2i.net> <20090107172310.bf12fe39.stas@FreeBSD.org> <200901071751.49494.hselasky@c2i.net> Organization: The FreeBSD Project X-XMPP: ssedov@jabber.ru X-Voice: +7 916 849 20 23 X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Jan 7 20:26:39 2009 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 4964e5cf967009398168836 Cc: Alfred Perlstein , arm@FreeBSD.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 17:39:21 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 7 Jan 2009 17:51:48 +0100 Hans Petter Selasky mentioned: > > cpu_idcache_wbinv_all(); /* and this line */ > > bus_dmamap_sync(pc->tag, pc->map, > BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); > } > > Then everything worked. > > I think the ones that have been working on busdma & cache sync issues recently > are to blame. I will try to investigate more. > Hmm, this might be the same problem with duplicate dcache entries raj reported recently on arm@. I tried replacing cpu_idcache_wbinv_all with busdma PREREAD and PREWRITE sync, but this doesn't work. BTW, even with cpu_idcache_wbinv_all change I'm still observing some CAM request retires under high load, so we really seem to have a generic cache issue somewhere. - -- Stanislav Sedov ST4096-RIPE -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAklk5mIACgkQK/VZk+smlYHKYQCfeuMdzjTVOu+PcZVfPNae+ZSg GE4AmwQlzD66bQZ8HB5xqThsbwzGaCvB =nK+S -----END PGP SIGNATURE----- !DSPAM:4964e5cf967009398168836! From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 18:46:18 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39C271065670; Wed, 7 Jan 2009 18:46:18 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 99EBE8FC24; Wed, 7 Jan 2009 18:46:17 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.2/8.14.2) with ESMTP id n07IkEWw053150; Wed, 7 Jan 2009 12:46:14 -0600 (CST) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1231353974; bh=76jF7uCw1DEW/GJs6/9ceFEWG8YuX3ZbWKmtNbj LfH4=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=G8ul4OsL 9a8GLImJZO1oboWbSgJN+WQf8u4T96xDnkYRa/NvG9kZZcj9easokJJJBNKl3YBaOTn Dy6ADVqV8tUnrcw7RBzpLXIY1n80o99Ibr/EMDFmfAQBg0hkY1j7pfKoLe1BAYg0WpQ 2A0TA1HfuCr0tn4Xm7UytM7O4S3EA= Received: (from tinguely@localhost) by casselton.net (8.14.2/8.14.2/Submit) id n07IkE41053149; Wed, 7 Jan 2009 12:46:14 -0600 (CST) (envelope-from tinguely) Date: Wed, 7 Jan 2009 12:46:14 -0600 (CST) From: Mark Tinguely Message-Id: <200901071846.n07IkE41053149@casselton.net> To: hselasky@c2i.net, stas@freebsd.org In-Reply-To: <20090107202901.21df29e3.stas@FreeBSD.org> Cc: arm@freebsd.org, alfred@freebsd.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 18:46:18 -0000 > On Wed, 7 Jan 2009 17:51:48 +0100 > Hans Petter Selasky mentioned: > > > > > cpu_idcache_wbinv_all(); /* and this line */ > > > > bus_dmamap_sync(pc->tag, pc->map, > > BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); > > } > > > > Then everything worked. > > > > I think the ones that have been working on busdma & cache sync issues recently > > are to blame. I will try to investigate more. > > > > Hmm, this might be the same problem with duplicate dcache entries raj reported > recently on arm@. I tried replacing cpu_idcache_wbinv_all with busdma PREREAD > and PREWRITE sync, but this doesn't work. > > BTW, even with cpu_idcache_wbinv_all change I'm still observing some CAM > request retires under high load, so we really seem to have a generic cache > issue somewhere. > > - -- > Stanislav Sedov > ST4096-RIPE It could be the same problem if there is a duplicate kernel mapping. If there is no duplicate kernel mapping, then there is another cache leak somewhere. Because a duplicate kernel mapping currently does not turn off the caches in either mapping, even the wb/inv trick is a cache russian rollette situation. If someone wants to look at very rough code - I don't have a current box to even compile it, I don't know anyone besides myself and Olivier Houchard that is up on the ARM pmap code, but I am looking for someone who will give this an eyeball. There can be consequences to adding pv_entrys where none existed before. I don't want people to blindly download and attempt to run it. My fix idea is to manage all mapped pages, with the expection of ficticous pages. I don't think fictiicous pages can be mapped twice. We need to manage them to turn caching off on all the mappings when there is multiple writers or a writer/reader case. --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 19:30:06 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26D451065674 for ; Wed, 7 Jan 2009 19:30:06 +0000 (UTC) (envelope-from raj@semihalf.com) Received: from semihalf.com (semihalf.com [206.130.101.55]) by mx1.freebsd.org (Postfix) with ESMTP id EFF028FC1E for ; Wed, 7 Jan 2009 19:30:05 +0000 (UTC) (envelope-from raj@semihalf.com) Received: from mail.semihalf.com (mail.semihalf.com [83.15.139.206]) by semihalf.com (8.13.1/8.13.1) with ESMTP id n07Iwjol009735; Wed, 7 Jan 2009 11:58:46 -0700 Received: from [77.113.128.244] (apn-77-113-128-244.gprs.plus.pl [77.113.128.244]) by mail.semihalf.com (Postfix) with ESMTP id 2BF65149A2; Wed, 7 Jan 2009 20:18:38 +0100 (CET) Message-ID: <4964FB53.8040607@semihalf.com> Date: Wed, 07 Jan 2009 19:58:27 +0100 From: Rafal Jaworowski Organization: Semihalf MIME-Version: 1.0 To: Mark Tinguely References: <200901071846.n07IkE41053149@casselton.net> In-Reply-To: <200901071846.n07IkE41053149@casselton.net> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 7bit Cc: arm@freebsd.org, alfred@freebsd.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 19:30:06 -0000 Mark Tinguely wrote: >> On Wed, 7 Jan 2009 17:51:48 +0100 >> Hans Petter Selasky mentioned: >> >> > >> > cpu_idcache_wbinv_all(); /* and this line */ >> > >> > bus_dmamap_sync(pc->tag, pc->map, >> > BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); >> > } >> > >> > Then everything worked. >> > >> > I think the ones that have been working on busdma & cache sync issues recently >> > are to blame. I will try to investigate more. >> > >> >> Hmm, this might be the same problem with duplicate dcache entries raj reported >> recently on arm@. I tried replacing cpu_idcache_wbinv_all with busdma PREREAD >> and PREWRITE sync, but this doesn't work. >> >> BTW, even with cpu_idcache_wbinv_all change I'm still observing some CAM >> request retires under high load, so we really seem to have a generic cache >> issue somewhere. >> >> - -- >> Stanislav Sedov >> ST4096-RIPE > > It could be the same problem if there is a duplicate kernel mapping. If > there is no duplicate kernel mapping, then there is another cache leak > somewhere. > > Because a duplicate kernel mapping currently does not turn off the > caches in either mapping, even the wb/inv trick is a cache russian rollette > situation. > > If someone wants to look at very rough code - I don't have a current box > to even compile it, I don't know anyone besides myself and Olivier Houchard > that is up on the ARM pmap code, but I am looking for someone who will > give this an eyeball. There can be consequences to adding pv_entrys where none > existed before. I don't want people to blindly download and attempt to run it. Mark, I'd like to look at your code, I guess also Grzegorz (CC'd) who knows ARM pmap code very well will be much interested in your thoughts on resolving the multiple virtual mappings issue, once and for all. How can we access your said proto/rough attempts? Rafal From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 19:32:13 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63878106566C for ; Wed, 7 Jan 2009 19:32:13 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id 05F058FC1A for ; Wed, 7 Jan 2009 19:32:13 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from DSPAM-Daemon (localhost [127.0.0.1]) by mx0.deglitch.com (Postfix) with SMTP id E02978FC51 for ; Wed, 7 Jan 2009 22:32:10 +0300 (MSK) Received: from orion.SpringDaemons.com (drsun1.dialup.corbina.ru [85.21.245.235]) by mx0.deglitch.com (Postfix) with ESMTPA id 2B9CF8FC18; Wed, 7 Jan 2009 22:32:08 +0300 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id A737D398F3; Wed, 7 Jan 2009 22:34:34 +0300 (MSK) Date: Wed, 7 Jan 2009 22:34:28 +0300 From: Stanislav Sedov To: Mark Tinguely Message-Id: <20090107223428.93bf0942.stas@FreeBSD.org> In-Reply-To: <200901071846.n07IkE41053149@casselton.net> References: <20090107202901.21df29e3.stas@FreeBSD.org> <200901071846.n07IkE41053149@casselton.net> Organization: The FreeBSD Project X-XMPP: ssedov@jabber.ru X-Voice: +7 916 849 20 23 X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Jan 7 22:32:10 2009 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 4965033a967001106815188 Cc: arm@freebsd.org, alfred@freebsd.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 19:32:13 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 7 Jan 2009 12:46:14 -0600 (CST) Mark Tinguely mentioned: > It could be the same problem if there is a duplicate kernel mapping. If > there is no duplicate kernel mapping, then there is another cache leak > somewhere. > Yeah, it seems it leaks somewhere as invalidating the entire wb cache in case of coherent mapping in sys/arm/arm/busdma_machdep.c:_bus_dmamap_sync, where currently it simply returns solves the problem. In fact, all usb memory is mapped as coherent and thus should not require cache flush, but something goes wrong. Could additional entries established by arm_remap_nocache in case of coherent mapping interfere with previous mappings and thus create problems? Is it the same case as duplicate mappings? Thanks! - -- Stanislav Sedov ST4096-RIPE -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAkllA8oACgkQK/VZk+smlYHUFQCcD8Ho1E/aFQj6Xm5dOVEwlggW aHMAnR7W3ahfhgJdRpnABLp/c0EuPoa6 =HB2P -----END PGP SIGNATURE----- !DSPAM:4965033a967001106815188! From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 19:55:47 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60DD01065673; Wed, 7 Jan 2009 19:55:47 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1C19F8FC17; Wed, 7 Jan 2009 19:55:47 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.2/8.14.2) with ESMTP id n07JthqZ057053; Wed, 7 Jan 2009 13:55:43 -0600 (CST) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1231358144; bh=fPwGK5+Jar3yFijT6Ptko4Ed1gVqE2t99VANQV8 6/gk=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=T3b92K5L ygY2TRiyRZbTZ9Cy1/8zWaS/VyQU5O+6RC4UPwSCbxWhNz1/id5TpBuwFWPtfKYpQtK j7lLQHKMeQbryqKgtzvMrSiV8gJeoRD3YjdL1clLVBFFxTAWiw50PjnpGfsXTxd2U1t OpfcHEu0mdvG/qJmbIcr2Cy/IF6UQ= Received: (from tinguely@localhost) by casselton.net (8.14.2/8.14.2/Submit) id n07Jthqu057051; Wed, 7 Jan 2009 13:55:43 -0600 (CST) (envelope-from tinguely) Date: Wed, 7 Jan 2009 13:55:43 -0600 (CST) From: Mark Tinguely Message-Id: <200901071955.n07Jthqu057051@casselton.net> To: raj@semihalf.com, tinguely@casselton.net In-Reply-To: <4964FB53.8040607@semihalf.com> Cc: arm@freebsd.org, alfred@freebsd.org Subject: Re: svn commit: r186730 - in head... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 19:55:47 -0000 > Mark, > I'd like to look at your code, I guess also Grzegorz (CC'd) who knows ARM pmap > code very well will be much interested in your thoughts on resolving the > multiple virtual mappings issue, once and for all. How can we access your said > proto/rough attempts? > > Rafal Here is the "svn diff" from a week or so ago. The url is below in case the inclusion adds some extra characters. http://www.casselton.net/~tinguely/arm_pmap_unmanage.diff pmap_fix_cache() - and the equivalent FreeBSD 6/7 and NetBSD pmap_vac_me_XXX routines should be called when a mapping occurs. It will check for situations where either multiple writers or readers/writer can occur in the current mapping. I won't go into the side cases. Anyway, if there is a writer/reader or multiple writers in a mapping, we have to write back/invalidate and turn off the cache in this mapping. The problem is the kernel mappings (via pmap_enter, pmap_qenter and pmap_kenter) are not managed with pv_entrys. In pmap_fix_cache() we count pv_entrys to detect multiple writers and write/read situations, so it is possible a writable unmanage page may get missed as is. I think the real problem is bigger, for entries mapped with pmap_qenter and pmap_kenter routines, the caches are left on. What my code does is manage all kernel mappings and therefore the pmap_fix_code can do its job. This patch uses more pv_entrys and there were new locks that had to be added for the pv_entry allocation routines. arm_pmap_unmanage.diff Index: arm/arm/pmap.c =================================================================== --- arm/arm/pmap.c (revision 186394) +++ arm/arm/pmap.c (working copy) @@ -407,7 +407,7 @@ #define pmap_is_current(pm) ((pm) == pmap_kernel() || \ curproc->p_vmspace->vm_map.pmap == (pm)) -static uma_zone_t pvzone; +static uma_zone_t pvzone = NULL; uma_zone_t l2zone; static uma_zone_t l2table_zone; static vm_offset_t pmap_kernel_l2dtable_kva; @@ -2713,8 +2713,8 @@ cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); for (pv = TAILQ_FIRST(&pmap->pm_pvlist); pv; pv = npv) { - if (pv->pv_flags & PVF_WIRED) { - /* The page is wired, cannot remove it now. */ + if (pv->pv_flags & PVF_WIRED || pv->pv_flags & PVF_UNMAN) { + /* The page is wired or unmanaged, cannot remove it now. */ npv = TAILQ_NEXT(pv, pv_plist); continue; } @@ -2824,10 +2824,12 @@ struct l2_bucket *l2b; pt_entry_t *pte; pt_entry_t opte; + struct pv_entry *pve; + vm_page_t m; + PDEBUG(1, printf("pmap_kenter: va = %08x, pa = %08x\n", (uint32_t) va, (uint32_t) pa)); - l2b = pmap_get_l2_bucket(pmap_kernel(), va); if (l2b == NULL) l2b = pmap_grow_l2_bucket(pmap_kernel(), va); @@ -2837,10 +2839,10 @@ PDEBUG(1, printf("pmap_kenter: pte = %08x, opte = %08x, npte = %08x\n", (uint32_t) pte, opte, *pte)); if (l2pte_valid(opte)) { - cpu_dcache_wbinv_range(va, PAGE_SIZE); - cpu_l2cache_wbinv_range(va, PAGE_SIZE); - cpu_tlb_flushD_SE(va); - cpu_cpwait(); + /* remove the old mapping + * note, pmap_kremove() has a lot of duplicated code + */ + pmap_kremove(va); } else { if (opte == 0) l2b->l2b_occupancy++; @@ -2852,6 +2854,27 @@ if (flags & KENTER_USER) *pte |= L2_S_PROT_U; PTE_SYNC(pte); + + /* krenel direct mappings can be shared, so use a pv_entry + * to ensure proper caching. + * + * with the old UMA pv allocation method, it is possible + * that this routine is called before UMA is initialized. + * Early allocation should not be shared. + * This can change with "chunk" style pv_entrys. + */ + if (pvzone != NULL) { + if ((pve = pmap_get_pv_entry()) == NULL) + panic("pmap_kenter_internal: no pv entries"); + + vm_page_lock_queues(); + PMAP_LOCK(pmap_kernel()); + m = PHYS_TO_VM_PAGE(vtophys(va)); + pmap_enter_pv(m, pve, pmap_kernel(), va, PVF_WRITE | PVF_UNMAN); + pmap_fix_cache(m, pmap_kernel(), va); + PMAP_UNLOCK(pmap_kernel()); + vm_page_unlock_queues(); + } } void @@ -2888,6 +2911,9 @@ { struct l2_bucket *l2b; pt_entry_t *pte, opte; + struct pv_entry *pve; + vm_page_t m; + vm_offset_t pa; l2b = pmap_get_l2_bucket(pmap_kernel(), va); if (!l2b) @@ -2896,6 +2922,25 @@ pte = &l2b->l2b_kva[l2pte_index(va)]; opte = *pte; if (l2pte_valid(opte)) { + /* pa = vtophs(va) taken from pmap_extract() */ + switch (opte & L2_TYPE_MASK) { + case L2_TYPE_L: + pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); + break; + default: + pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); + break; + } + /* note: should never have to remove an allocation + * before the pvzone is initialized. + */ + vm_page_lock_queues(); + PMAP_LOCK(pmap_kernel()); /* XXX only PVC_UNMAN? XXX */ + if ((m = PHYS_TO_VM_PAGE(pa)) && + (pve = pmap_remove_pv(m, pmap_kernel(), va))) + pmap_free_pv_entry(pve); + PMAP_UNLOCK(pmap_kernel()); + vm_page_unlock_queues(); cpu_dcache_wbinv_range(va, PAGE_SIZE); cpu_l2cache_wbinv_range(va, PAGE_SIZE); cpu_tlb_flushD_SE(va); @@ -2917,7 +2962,7 @@ * update '*virt' with the first usable address after the mapped * region. */ -vm_offset_t +vm_offset_t pmap_map(vm_offset_t *virt, vm_offset_t start, vm_offset_t end, int prot) { #ifdef ARM_USE_SMALL_ALLOC @@ -3123,6 +3168,11 @@ pmap_remove_write(m); curpm = vmspace_pmap(curproc->p_vmspace); while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { + /* don't expect to remove a unmanaged page */ + if (pv->pv_flag & PVC_UNMAN) { + printf("pmap_remove_all: Unmanaged page %08x\n", + (u_int32_t) m); + } if (flush == FALSE && (pv->pv_pmap == curpm || pv->pv_pmap == pmap_kernel())) flush = TRUE; @@ -3418,16 +3468,16 @@ * Replacing an existing mapping with a new one. * It is part of our managed memory so we * must remove it from the PV list + * + * unmanaged kernel memory also will have a pv_entry. */ pve = pmap_remove_pv(opg, pmap, va); - if (m && (m->flags & (PG_UNMANAGED | PG_FICTITIOUS)) && - pve) + if (m && (m->flags & PG_FICTITIOUS) && pve) pmap_free_pv_entry(pve); - else if (!pve && - !(m->flags & (PG_UNMANAGED | PG_FICTITIOUS))) + else if (!pve && !(m->flags & PG_FICTITIOUS)) pve = pmap_get_pv_entry(); - KASSERT(pve != NULL || m->flags & (PG_UNMANAGED | - PG_FICTITIOUS), ("No pv")); + KASSERT(pve != NULL || m->flags & PG_FICTITIOUS), + ("No pv")); oflags = pve->pv_flags; /* @@ -3436,8 +3486,7 @@ * initially) then make sure to frob * the cache. */ - if ((oflags & PVF_NC) == 0 && - l2pte_valid(opte)) { + if ((oflags & PVF_NC) == 0 && l2pte_valid(opte)) { if (PV_BEEN_EXECD(oflags)) { pmap_idcache_wbinv_range(pmap, va, PAGE_SIZE); @@ -3448,13 +3497,15 @@ (oflags & PVF_WRITE) == 0); } } - } else if (m && !(m->flags & (PG_UNMANAGED | PG_FICTITIOUS))) + } else if (m && !(m->flags & PG_FICTITIOUS)) if ((pve = pmap_get_pv_entry()) == NULL) { panic("pmap_enter: no pv entries"); } - if (m && !(m->flags & (PG_UNMANAGED | PG_FICTITIOUS))) { + if (m && !(m->flags & PG_FICTITIOUS)) { KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, ("pmap_enter: managed mapping within the clean submap")); + if (m->flags & PG_UNMANAGED) + nflags |= PVF_UNMAN; pmap_enter_pv(m, pve, pmap, va, nflags); } } Index: arm/include/pmap.h =================================================================== --- arm/include/pmap.h (revision 186394) +++ arm/include/pmap.h (working copy) @@ -495,6 +495,7 @@ #define PVF_EXEC 0x10 /* mapping is executable */ #define PVF_NC 0x20 /* mapping is non-cacheable */ #define PVF_MWC 0x40 /* mapping is used multiple times in userland */ +#define PVF_UNMAN 0x80 /* mapping is unmanaged */ void vector_page_setprot(int); From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 21:53:22 2009 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 662601065937 for ; Wed, 7 Jan 2009 21:53:22 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [78.110.53.255]) by mx1.freebsd.org (Postfix) with ESMTP id CFD478FC16 for ; Wed, 7 Jan 2009 21:53:21 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from DSPAM-Daemon (localhost [127.0.0.1]) by mx0.deglitch.com (Postfix) with SMTP id 368E28FC4F for ; Thu, 8 Jan 2009 00:48:07 +0300 (MSK) Received: from orion.SpringDaemons.com (drsun1.dialup.corbina.ru [85.21.245.235]) by mx0.deglitch.com (Postfix) with ESMTPA id 7AF438FC18; Thu, 8 Jan 2009 00:48:05 +0300 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id D1AAB398F3; Thu, 8 Jan 2009 00:50:32 +0300 (MSK) Date: Thu, 8 Jan 2009 00:50:28 +0300 From: Stanislav Sedov To: Hans Petter Selasky Message-Id: <20090108005028.3c31d5eb.stas@FreeBSD.org> In-Reply-To: <200901071819.26446.hselasky@c2i.net> References: <200901071819.26446.hselasky@c2i.net> Organization: The FreeBSD Project X-XMPP: ssedov@jabber.ru X-Voice: +7 916 849 20 23 X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Jan 8 00:48:06 2009 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 49652316967001025050683 Cc: raj@freebsd.org, jhb@freebsd.org, Alfred Perlstein , thompsa@freebsd.org, arm@FreeBSD.org Subject: Re: USB2 reveals cache sync problems on AT91RM9200 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 21:53:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 7 Jan 2009 18:19:25 +0100 Hans Petter Selasky mentioned: > Hi, > > I'm writing this e-mail to you because some of you have changed something in: > > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/arm/arm/busdma_machdep.c > > I'm experiencing a problem where the CPU is not seeing the data written to > memory by the OHCI using DMA. This does not happen all the time. > > Adding a "cpu_idcache_wbinv_all()" to my "cpu_invalidate" function solves the > problem 100% reliably. I think Stanislav should be able to reproduce this. > > Any ideas? > > I feel pretty sure that this is not an USB2 problem. I have tested that data > is correctly bounced on x86 and assume that if the data is correctly bounced, > it will also be correctly synched. Using bounce buffers by lowering > the "lowaddr" in the DMA tag, on my AT91RM9200, does not solve the problem. > > --HPS > > Hardware: > > ## Starting application at 0x200000E0 ... > KDB: debugger backends: ddb > KDB: current backend: ddb > Copyright (c) 1992-2009 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.0-CURRENT #33: Wed Jan 7 17:52:20 CET 2009 > hans_other@server0.selasky.org:/usr/obj/arm/usr/8-current/src/sys/custom > CPU: ARM920T rev 0 (ARM9TDMI core) > DC enabled IC enabled WB enabled LABT > 16KB/32B 64-way Instruction cache > 16KB/32B 64-way write-back-locking-A Data cache > > My patch that makes things work again: > > #include > > and: > > /*------------------------------------------------------------------------* > * usb2_pc_cpu_invalidate - invalidate CPU cache > *------------------------------------------------------------------------*/ > void > usb2_pc_cpu_invalidate(struct usb2_page_cache *pc) > { > if (pc->page_offset_end == pc->page_offset_buf) { > /* nothing has been loaded into this page cache! */ > return; > } > > cpu_idcache_wbinv_all(); /* and this line */ > > bus_dmamap_sync(pc->tag, pc->map, > BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); > } > Well, further research showed that what is really not synchronizing correctly is ohci ed descriptors. I've added the following code into ohci2.c and clearly shows that ed descriptor contents differ before and after cpu_idcache_wbinv_all despite it should have been syncronized by usb2_pc_cpu_invalidate. usb2_pc_cpu_invalidate(ed->page_cache); ed_flags = le32toh(ed->ed_flags); ed_headp = le32toh(ed->ed_headp); ed_tailp = le32toh(ed->ed_tailp); + printf("OHCI: ed flags was: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp)); + cpu_idcache_wbinv_all(); + printf("OHCI: ed flags now: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp)); Moving cpu_idcache_wbinv_all just after usb2_pc_cpu_invalidate before ed descriptor access fixes umass storage on arm for me. - -- Stanislav Sedov ST4096-RIPE -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAkllI6gACgkQK/VZk+smlYG0ewCcD5VlJMoLas6hrx9sUC2y7/e5 cfQAnjBKaOwvmMBxt1nMecjB3pvemk7a =86fe -----END PGP SIGNATURE----- !DSPAM:49652316967001025050683! From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 21:53:27 2009 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56BB51065A93; Wed, 7 Jan 2009 21:53:27 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id D409B8FC2E; Wed, 7 Jan 2009 21:53:26 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.2/8.14.2) with ESMTP id n07LrNi6063947; Wed, 7 Jan 2009 15:53:23 -0600 (CST) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1231365204; bh=4qo/87vFqSreuNfqFKkxkQC3FnWhLtKGFnSknLA aG6k=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=lnSIfk4K 0M9mfkDV2MA79FZb3vQX95Z0srDTvtKw+Qk/aZOtmxEzoLR1hescSMH83yZGap6Ngi8 +kMcaDG700IDCT8JKFtOdXgk/eVHqUqlSQffgU6g3Cm3pZtZaHFVJUktgCquJSPI71s jKrgBv6wGH7QqhKE0LQnthNTHuytQ= Received: (from tinguely@localhost) by casselton.net (8.14.2/8.14.2/Submit) id n07LrNZU063946; Wed, 7 Jan 2009 15:53:23 -0600 (CST) (envelope-from tinguely) Date: Wed, 7 Jan 2009 15:53:23 -0600 (CST) From: Mark Tinguely Message-Id: <200901072153.n07LrNZU063946@casselton.net> To: stas@FreeBSD.org, tinguely@casselton.net In-Reply-To: <20090107223428.93bf0942.stas@FreeBSD.org> Cc: arm@FreeBSD.org, alfred@FreeBSD.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 21:53:29 -0000 > On Wed, 7 Jan 2009 12:46:14 -0600 (CST) > Mark Tinguely mentioned: > > > It could be the same problem if there is a duplicate kernel mapping. If > > there is no duplicate kernel mapping, then there is another cache leak > > somewhere. > > > > Yeah, it seems it leaks somewhere as invalidating the entire wb cache > in case of coherent mapping in sys/arm/arm/busdma_machdep.c:_bus_dmamap_sync, > where currently it simply returns solves the problem. In fact, all usb memory > is mapped as coherent and thus should not require cache flush, but something > goes wrong. Could additional entries established by arm_remap_nocache in > case of coherent mapping interfere with previous mappings and thus create > problems? Is it the same case as duplicate mappings? > > Thanks! > - -- > Stanislav Sedov > ST4096-RIPE Coherent in ARM is a duplicate mapping with a special VA above the KVM. The new kernel entry is mapped with cache turned off (pmap_kenter_nocache()), but the original virtual mapping still has caching turned on. In fact, pmap_kenter_nocache() invalidates/writebacks the new special VA, but does not writeback/invalidate the original virtual mapping - that is where your problem originates and why your writeback/invalidates works. If you trust that the other mapping will not modify memory - which in this case, I believe we can - a quick solution would be to add a writeback/invalidate (pmap_fix_cache) in the second for loop of arm_remap_nocache(). But I would have to make sure the pmap_fix_cache counts the new un-managed kernel mapping before I recommend that. When I wrote the patch, I grepped for pmap_kenter_* routines, and noticed as side effect of managing the cohert code is also fixed and in fact it can be simplified greatly. At the time - before the usb2 import, I grepping the sources and there were only a couple network adapters using BUS_DMA_COHERENT, so I was not too concerned. --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Wed Jan 7 22:24:10 2009 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74B8A1065C87 for ; Wed, 7 Jan 2009 22:24:10 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id 1B36D8FC18 for ; Wed, 7 Jan 2009 22:24:10 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from DSPAM-Daemon (localhost [127.0.0.1]) by mx0.deglitch.com (Postfix) with SMTP id B1D488FC51 for ; Thu, 8 Jan 2009 01:24:08 +0300 (MSK) Received: from orion.SpringDaemons.com (drsun1.dialup.corbina.ru [85.21.245.235]) by mx0.deglitch.com (Postfix) with ESMTPA id D98508FC18; Thu, 8 Jan 2009 01:24:05 +0300 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id BEF29398F3; Thu, 8 Jan 2009 01:26:33 +0300 (MSK) Date: Thu, 8 Jan 2009 01:26:27 +0300 From: Stanislav Sedov To: Mark Tinguely Message-Id: <20090108012627.4ed3e907.stas@FreeBSD.org> In-Reply-To: <200901072153.n07LrNZU063946@casselton.net> References: <20090107223428.93bf0942.stas@FreeBSD.org> <200901072153.n07LrNZU063946@casselton.net> Organization: The FreeBSD Project X-XMPP: ssedov@jabber.ru X-Voice: +7 916 849 20 23 X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Jan 8 01:24:08 2009 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 49652b88967002281716519 Cc: arm@FreeBSD.org, alfred@FreeBSD.org Subject: Re: svn commit: r186730 - in head: lib/libusb20 sys/dev/usb2/controller sys/dev/usb2/core sys/dev/usb2/ethernet sys/dev/usb2/image sys/dev/usb2/include sys/dev/usb2/serial sys/dev/usb2/sound sys/dev/us... X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2009 22:24:15 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 7 Jan 2009 15:53:23 -0600 (CST) Mark Tinguely mentioned: > Coherent in ARM is a duplicate mapping with a special VA above the KVM. The > new kernel entry is mapped with cache turned off (pmap_kenter_nocache()), but > the original virtual mapping still has caching turned on. In fact, > pmap_kenter_nocache() invalidates/writebacks the new special VA, but does > not writeback/invalidate the original virtual mapping - that is where your > problem originates and why your writeback/invalidates works. > > If you trust that the other mapping will not modify memory - which in this > case, I believe we can - a quick solution would be to add a writeback/invalidate > (pmap_fix_cache) in the second for loop of arm_remap_nocache(). But I would > have to make sure the pmap_fix_cache counts the new un-managed kernel mapping > before I recommend that. > > When I wrote the patch, I grepped for pmap_kenter_* routines, and noticed > as side effect of managing the cohert code is also fixed and in fact it can > be simplified greatly. At the time - before the usb2 import, I grepping the > sources and there were only a couple network adapters using BUS_DMA_COHERENT, > so I was not too concerned. It looks like I've found the place where the mapping could not be invalidated because of this. In ohci2.c we need the ed descriptor to be synchronized before we access it, but busdma fails to sync it. I tried to do what busdma does by hand, but this doesn't make "ed" to be consistent as well: cpu_idcache_wbinv_range((vm_offset_t)ed->page_cache->map->buffer, ed->page_cache->map->len); After calling cpu_idcache_wbinv_all at this point (which invalidates and cleans all entries) ed contains correct entries. Thus it seems that because of double mappings cpu_idcache_wbinv_range fails to clean the entry (it seem to clear the old one instead). I'll think about your points here, need to study pmap and busdma code a bit first. Thanks for clearing this out! - -- Stanislav Sedov ST4096-RIPE -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAkllLBkACgkQK/VZk+smlYERswCggfXhIHlTAZZ0nymRwwRPQKwf NaQAn2tHgT4ez2eHfRyC/JMX37eKiuLU =jzsU -----END PGP SIGNATURE----- !DSPAM:49652b88967002281716519! From owner-freebsd-arm@FreeBSD.ORG Thu Jan 8 09:22:12 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3873B1065676; Thu, 8 Jan 2009 09:22:12 +0000 (UTC) (envelope-from gjb@semihalf.com) Received: from semihalf.com (semihalf.com [206.130.101.55]) by mx1.freebsd.org (Postfix) with ESMTP id 1004C8FC13; Thu, 8 Jan 2009 09:22:11 +0000 (UTC) (envelope-from gjb@semihalf.com) Received: from mail.semihalf.com (mail.semihalf.com [83.15.139.206]) by semihalf.com (8.13.1/8.13.1) with ESMTP id n089M0Vd003202; Thu, 8 Jan 2009 02:22:01 -0700 Message-ID: <4965C5D8.6020805@semihalf.com> Date: Thu, 08 Jan 2009 10:22:32 +0100 From: Grzegorz Bernacki MIME-Version: 1.0 To: Stanislav Sedov References: <200901071819.26446.hselasky@c2i.net> <20090108005028.3c31d5eb.stas@FreeBSD.org> In-Reply-To: <20090108005028.3c31d5eb.stas@FreeBSD.org> Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: raj@freebsd.org, Mark Tinguely , jhb@freebsd.org, Alfred Perlstein , thompsa@freebsd.org, arm@freebsd.org Subject: Re: USB2 reveals cache sync problems on AT91RM9200 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2009 09:22:13 -0000 Stanislav Sedov wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Wed, 7 Jan 2009 18:19:25 +0100 > Hans Petter Selasky mentioned: > >> Hi, >> >> I'm writing this e-mail to you because some of you have changed something in: >> >> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/arm/arm/busdma_machdep.c >> >> I'm experiencing a problem where the CPU is not seeing the data written to >> memory by the OHCI using DMA. This does not happen all the time. >> >> Adding a "cpu_idcache_wbinv_all()" to my "cpu_invalidate" function solves the >> problem 100% reliably. I think Stanislav should be able to reproduce this. >> >> Any ideas? >> >> I feel pretty sure that this is not an USB2 problem. I have tested that data >> is correctly bounced on x86 and assume that if the data is correctly bounced, >> it will also be correctly synched. Using bounce buffers by lowering >> the "lowaddr" in the DMA tag, on my AT91RM9200, does not solve the problem. >> >> --HPS >> >> Hardware: >> >> ## Starting application at 0x200000E0 ... >> KDB: debugger backends: ddb >> KDB: current backend: ddb >> Copyright (c) 1992-2009 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 8.0-CURRENT #33: Wed Jan 7 17:52:20 CET 2009 >> hans_other@server0.selasky.org:/usr/obj/arm/usr/8-current/src/sys/custom >> CPU: ARM920T rev 0 (ARM9TDMI core) >> DC enabled IC enabled WB enabled LABT >> 16KB/32B 64-way Instruction cache >> 16KB/32B 64-way write-back-locking-A Data cache >> >> My patch that makes things work again: >> >> #include >> >> and: >> >> /*------------------------------------------------------------------------* >> * usb2_pc_cpu_invalidate - invalidate CPU cache >> *------------------------------------------------------------------------*/ >> void >> usb2_pc_cpu_invalidate(struct usb2_page_cache *pc) >> { >> if (pc->page_offset_end == pc->page_offset_buf) { >> /* nothing has been loaded into this page cache! */ >> return; >> } >> >> cpu_idcache_wbinv_all(); /* and this line */ >> >> bus_dmamap_sync(pc->tag, pc->map, >> BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); >> } >> > > Well, further research showed that what is really not synchronizing > correctly is ohci ed descriptors. I've added the following code into > ohci2.c and clearly shows that ed descriptor contents differ before > and after cpu_idcache_wbinv_all despite it should have been syncronized > by usb2_pc_cpu_invalidate. > > usb2_pc_cpu_invalidate(ed->page_cache); > ed_flags = le32toh(ed->ed_flags); > ed_headp = le32toh(ed->ed_headp); > ed_tailp = le32toh(ed->ed_tailp); > + printf("OHCI: ed flags was: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp)); > + cpu_idcache_wbinv_all(); > + printf("OHCI: ed flags now: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp)); > > Moving cpu_idcache_wbinv_all just after usb2_pc_cpu_invalidate before ed descriptor > access fixes umass storage on arm for me. > Hi, Some time ago we had a problem with EHCI. The problem was that after transfer CSW was overwritten with some old data from CSW of previous transfer. We saw it only when write-allocate feature was turned off. The problem was caused by specific handling of BUS_DMASYNC_POSTREAD in ARM bus dma code. If you look at the code you see that if synced region is not cache aligned we copy it into temporary location, invalidate the region and copy back saved data. But if write-allocate is turned on, writing data back causes allocating cache line and store the data in it. So we end up with dirty cache line contains stale data which should be invalidated. And those data could be flushed back at random time whenever cache line is recycled. We fixed the problem using BUS_DMASYNC_PREREAD command before using updated CSW. I don't know if you use write-allocate, but if you do and if the problem disappears when you turn it off it's probably the same problem. You can also check the thread when Rafal described our problem: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=198997+0+archive/2008/freebsd-usb/20080203.freebsd-usb correct place for fix mentioned there is: http://people.freebsd.org/~raj/patches/misc/committed/usbdi-bus_dma-fix.diff pozdrawiam, Grzesiek From owner-freebsd-arm@FreeBSD.ORG Fri Jan 9 09:52:52 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1859106566C for ; Fri, 9 Jan 2009 09:52:52 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: from mail-ew0-f21.google.com (mail-ew0-f21.google.com [209.85.219.21]) by mx1.freebsd.org (Postfix) with ESMTP id 4F6578FC13 for ; Fri, 9 Jan 2009 09:52:51 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: by ewy14 with SMTP id 14so11258817ewy.19 for ; Fri, 09 Jan 2009 01:52:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=zNXvTVncGfXF899ESPNqeshIan5ZRCTt1a8Vjr26gpQ=; b=vsl+cNbHf3oFUjcBbPVvIAI6lnQi3vy49wiJ4PcKkUXJVamb7tYmxcgv8pyFjXbf6p fxV61z/PLCv3kVjZOt9syCmQEiHoA86tHTvFvkC5YqmdOCi+91609IPoV5YGFsb2S6aG 9Dx8RVbGfXnarKwOGyjlb6KDg9inol8/gAH28= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=mbQDVVxyG88qLKcr7IEiN/cf+9NqxjXStw0RgP8hw1fvmrxg6DVOVE8aES2upixXz4 YeFJN619JgxDUObPkXb3uIOiCf6p8w03LosrNiIPbm/N0Jp9ei/0SNmB/mKvrKKoPNG3 MycstD/3ADvCu1FMkNHx8VR0UGswbRnaNz3AE= Received: by 10.210.127.10 with SMTP id z10mr6675077ebc.106.1231494771228; Fri, 09 Jan 2009 01:52:51 -0800 (PST) Received: from kaskar.trispen.com (dsl-244-23-69.telkomadsl.co.za [41.244.23.69]) by mx.google.com with ESMTPS id 33sm50774355nfu.70.2009.01.09.01.52.49 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 09 Jan 2009 01:52:50 -0800 (PST) Message-ID: <49671DFB.8010800@gmail.com> Date: Fri, 09 Jan 2009 11:50:51 +0200 From: Jacques Fourie User-Agent: Thunderbird 2.0.0.17 (X11/20081113) MIME-Version: 1.0 To: freebsd-arm@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: if_smc patch X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2009 09:52:53 -0000 Hi, I'm not sure who the maintainer of the if_smc code is, but here is a simple patch that addresses what seems like typos in the original code. diff -u -r smc.old/if_smc.c smc.new/if_smc.c --- smc.old/if_smc.c 2009-01-09 10:31:56.000000000 +0200 +++ smc.new/if_smc.c 2009-01-09 11:45:01.000000000 +0200 @@ -494,7 +494,7 @@ * Work out how many 256 byte "pages" we need. We have to include the * control data for the packet in this calculation. */ - npages = (len * PKT_CTRL_DATA_LEN) >> 8; + npages = (len + PKT_CTRL_DATA_LEN) >> 8; if (npages == 0) npages = 1; diff -u -r smc.old/if_smcreg.h smc.new/if_smcreg.h --- smc.old/if_smcreg.h 2009-01-09 10:31:56.000000000 +0200 +++ smc.new/if_smcreg.h 2009-01-09 11:45:01.000000000 +0200 @@ -138,7 +138,7 @@ #define GPR 0xa /* Bank 1, Offset 0xc: Control Register */ -#define CTR 0xa +#define CTR 0xc #define CTR_STORE 0x0001 /* Store registers to EEPROM */ #define CTR_RELOAD 0x0002 /* Reload registers from EEPROM */ #define CTR_EEPROM_SELECT 0x0004 /* Select registers to store/reload */ From owner-freebsd-arm@FreeBSD.ORG Fri Jan 9 21:15:00 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 027C4106566C for ; Fri, 9 Jan 2009 21:15:00 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id B90CA8FC1D for ; Fri, 9 Jan 2009 21:14:59 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id n09KtDBk081518 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 9 Jan 2009 12:55:13 -0800 (PST) (envelope-from sam@freebsd.org) Message-ID: <4967B9B1.3000503@freebsd.org> Date: Fri, 09 Jan 2009 12:55:13 -0800 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081209) MIME-Version: 1.0 To: Jacques Fourie References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC-sonic.net-Metrics: ebb.errno.com; whitelist Cc: freebsd-arm@freebsd.org Subject: Re: hwpmc on xscale (pxa255) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2009 21:15:00 -0000 Jacques Fourie wrote: > I am interested in making hwpmc work on my gumstix (pxa255 cpu). Does > anyone have working examples of pmc_save_kernel_callchain() and > pmc_save_user_callchain() for arm platforms that they are willing to > share? > I'm very interested in pmc support for xscale. If you do anything I'll pitch in. Sam