From owner-freebsd-arm@FreeBSD.ORG Mon Jan 30 00:07:25 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E36A106566C for ; Mon, 30 Jan 2012 00:07:25 +0000 (UTC) (envelope-from freebsd_arm@myspectrum.nl) Received: from mail.virtualhost.nl (mail.virtualhost.nl [89.200.201.133]) by mx1.freebsd.org (Postfix) with ESMTP id A096C8FC0A for ; Mon, 30 Jan 2012 00:07:24 +0000 (UTC) Received: (qmail 62419 invoked by uid 1141); 30 Jan 2012 00:40:42 +0100 Received: from ip120-12-208-87.adsl2.static.versatel.nl (HELO [192.168.0.6]) (87.208.12.120) (smtp-auth username freebsd_arm@myspectrum.nl, mechanism plain) by mail.virtualhost.nl (qpsmtpd/0.84) with (CAMELLIA256-SHA encrypted) ESMTPSA; Mon, 30 Jan 2012 00:40:42 +0100 Message-ID: <4F25D8F9.8040108@myspectrum.nl> Date: Mon, 30 Jan 2012 00:40:41 +0100 From: Jeroen Hofstee User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:9.0) Gecko/20111229 Thunderbird/9.0 MIME-Version: 1.0 To: freebsd-arm@freebsd.org References: <23CB6C35-9450-40BA-9FA3-37C44B328CA8@freebsd.org> <8042D895-3B3D-431E-ADCC-A150BDC838ED@kientzle.com> <7E53112E-DE29-4145-B597-D39AA77252C5@kientzle.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: tim@kientzle.com Subject: Re: FreeBSD and BeagleBone X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2012 00:07:25 -0000 Hi Tim, On 01/21/12 07:34, Tim Kientzle wrote: > To actually get it to build, I've had to make two changes to U-Boot sources: > * In config.mk, remove "-nostdinc" from CPPFLAGS > * In Makefile, add /usr/arm-freebsd/usr/lib/libc.a to PLATFORM_LIBS > > The first fixes missing stddef.h, stdarg.h headers. (I'm not > sure why the -nostdinc works on Linux.) First of all, the standard c calls are excluded by u-boot on purpose since these might call the operating system which is not there yet... However some macro's, typedefs etc from the headers are of interest (like stdargs, types etc etc). Das U-boot resolves this by including gcc -print-file-name=include (when really needed?) The FreeBSD bootstrap gcc is simply too old to support this command, hence the missing includes. I don't know a workaround for it. > The second is needed to resolve __umodsi3, __udivsi3, etc, > functions. (These might be defined in libgcc on Linux, and > there's some logic here to try to locate libgcc.) I don't know the exact reason for this: - it could be as you suggested, it is in a different lib with GNU libc - or it could be that the FreeBSD bootstrap loader removes it on purpose, for the same reason as u-boot excludes it. FreeBSD provides libc.a afaik, it might therefore exclude it from linking against it from the gcc intended to build the kernel. But as with the includes, only use it to statically link to function which don't use system calls.. Hope this clears things up a bit, Regards, Jeroen patch closer to the GNU/Linux build diff --git a/Makefile b/Makefile index 36246b6..1ec3de3 100644 --- a/Makefile +++ b/Makefile @@ -326,7 +326,7 @@ else PLATFORM_LIBGCC = -L $(USE_PRIVATE_LIBGCC) -lgcc endif else -PLATFORM_LIBGCC := -L $(shell dirname `$(CC) $(CFLAGS) -print-libgcc-file-name`) -lgcc +PLATFORM_LIBGCC := -L $(shell dirname `$(CC) $(CFLAGS) -print-libgcc-file-name`) -lgcc -lc endif PLATFORM_LIBS += $(PLATFORM_LIBGCC) export PLATFORM_LIBS diff --git a/config.mk b/config.mk index ddaa477..d6fef21 100644 --- a/config.mk +++ b/config.mk @@ -183,7 +183,8 @@ OPTFLAGS= -Os #-fomit-frame-pointer OBJCFLAGS += --gap-fill=0xff -gccincdir := $(shell $(CC) -print-file-name=include) +# FreeBSD xdev compiler is too old to report include dir.. hardcode instead +gccincdir := "/usr/arm-freebsd/usr/include" CPPFLAGS := $(DBGFLAGS) $(OPTFLAGS) $(RELFLAGS) \ -D__KERNEL__ From owner-freebsd-arm@FreeBSD.ORG Mon Jan 30 02:54:30 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B8361065687 for ; Mon, 30 Jan 2012 02:54:30 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 5F4458FC16 for ; Mon, 30 Jan 2012 02:54:28 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q0U2ZJ5b044774; Mon, 30 Jan 2012 02:35:19 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.119] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 6ntkbtuhmtgmnx2vmrbp5ge78w; Mon, 30 Jan 2012 02:35:18 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Tim Kientzle In-Reply-To: <4F25D8F9.8040108@myspectrum.nl> Date: Sun, 29 Jan 2012 18:35:16 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <23CB6C35-9450-40BA-9FA3-37C44B328CA8@freebsd.org> <8042D895-3B3D-431E-ADCC-A150BDC838ED@kientzle.com> <7E53112E-DE29-4145-B597-D39AA77252C5@kientzle.com> <4F25D8F9.8040108@myspectrum.nl> To: Jeroen Hofstee X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-arm@freebsd.org Subject: Re: FreeBSD and BeagleBone X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2012 02:54:30 -0000 On Jan 29, 2012, at 3:40 PM, Jeroen Hofstee wrote: > On 01/21/12 07:34, Tim Kientzle wrote: >> To actually get it to build, I've had to make two changes to U-Boot = sources: >> * In config.mk, remove "-nostdinc" from CPPFLAGS >> * In Makefile, add /usr/arm-freebsd/usr/lib/libc.a to = PLATFORM_LIBS >>=20 >> The first fixes missing stddef.h, stdarg.h headers. (I'm not >> sure why the -nostdinc works on Linux.) > First of all, the standard c calls are excluded by u-boot on purpose = since > these might call the operating system which is not there yet... > However some macro's, typedefs etc from the headers are of interest = (like > stdargs, types etc etc). Das U-boot resolves this by including >=20 > gcc -print-file-name=3Dinclude >=20 > (when really needed?) The FreeBSD bootstrap gcc is simply too old to = support > this command, hence the missing includes. I don't know a workaround = for it. Thanks for the pointers, Jeroen. I might be able to come up with a workaround if I knew exactly what this option did. The most recent docs on gcc.gnu.org (for GCC 4.6.2) only mention the -print-file-name=3D form. Can you point me to docs for this option? Tim From owner-freebsd-arm@FreeBSD.ORG Mon Jan 30 05:35:45 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D25FD106567C; Mon, 30 Jan 2012 05:35:45 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [64.7.128.101]) by mx1.freebsd.org (Postfix) with ESMTP id 80FE98FC20; Mon, 30 Jan 2012 05:35:45 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id q0U5ZTw2011762; Mon, 30 Jan 2012 05:35:29 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id q0U5ZT0N011761; Mon, 30 Jan 2012 05:35:29 GMT (envelope-from tinderbox@freebsd.org) Date: Mon, 30 Jan 2012 05:35:29 GMT Message-Id: <201201300535.q0U5ZT0N011761@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [releng_8_2 tinderbox] failure on arm/arm X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2012 05:35:45 -0000 TB --- 2012-01-30 05:34:47 - tinderbox 2.8 running on freebsd-legacy2.sentex.ca TB --- 2012-01-30 05:34:47 - starting RELENG_8_2 tinderbox run for arm/arm TB --- 2012-01-30 05:34:47 - mkdir /usr/home/tinderbox/RELENG_8_2/arm TB --- 2012-01-30 05:34:47 - mkdir /usr/home/tinderbox/RELENG_8_2/arm/arm TB --- 2012-01-30 05:34:47 - cleaning the object tree TB --- 2012-01-30 05:34:47 - cvsupping the source tree TB --- 2012-01-30 05:34:47 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca -s /usr/home/tinderbox/RELENG_8_2/arm/arm/supfile TB --- 2012-01-30 05:35:29 - WARNING: /usr/bin/csup returned exit code 1 TB --- 2012-01-30 05:35:29 - ERROR: unable to cvsup the source tree TB --- 2012-01-30 05:35:29 - 0.11 user 0.30 system 41.35 real http://tinderbox.freebsd.org/tinderbox-releng_8-RELENG_8_2-arm-arm.full From owner-freebsd-arm@FreeBSD.ORG Mon Jan 30 11:07:27 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 454591065673 for ; Mon, 30 Jan 2012 11:07:27 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 329738FC20 for ; Mon, 30 Jan 2012 11:07:27 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0UB7RUd005336 for ; Mon, 30 Jan 2012 11:07:27 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0UB7Qic005334 for freebsd-arm@FreeBSD.org; Mon, 30 Jan 2012 11:07:26 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 30 Jan 2012 11:07:26 GMT Message-Id: <201201301107.q0UB7Qic005334@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-arm@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2012 11:07:27 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o arm/162159 arm [panic] USB errors leading to panic on DockStar 9.0-RC o arm/161110 arm /usr/src/sys/arm/include/signal.h is bad o arm/161044 arm devel/icu does not build on arm o arm/160431 arm [busdma] [patch] Disable interrupts during busdma cach o arm/158950 arm arm/sheevaplug fails fsx when mmap operations are enab o arm/156814 arm OpenRD Ultimate does not boot on DB-88F6XXX or SHEEVAP o arm/156496 arm [patch] Minor bugfixes and enhancements to mmc and mmc o arm/155894 arm [patch] Enable at91 booting from SDHC (high capacity) o arm/155214 arm [patch] MMC/SD IO slow on Atmel ARM with modern large o arm/154227 arm [geli] using GELI leads to panic on ARM o arm/154189 arm lang/perl5.12 doesn't build on arm o arm/153380 arm Panic / translation fault with wlan on ARM o arm/150581 arm [irq] Unknown error generates IRQ address decoding err o arm/149288 arm mail/dovecot causes panic during configure on Sheevapl o arm/134368 arm [patch] nslu2_led driver for the LEDs on the NSLU2 p arm/134338 arm [patch] Lock GPIO accesses on ixp425 16 problems total. From owner-freebsd-arm@FreeBSD.ORG Mon Jan 30 21:57:48 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9A091065676 for ; Mon, 30 Jan 2012 21:57:48 +0000 (UTC) (envelope-from freebsd_arm@myspectrum.nl) Received: from mail.virtualhost.nl (mail.virtualhost.nl [89.200.201.133]) by mx1.freebsd.org (Postfix) with ESMTP id 092338FC12 for ; Mon, 30 Jan 2012 21:57:47 +0000 (UTC) Received: (qmail 38902 invoked by uid 1141); 30 Jan 2012 22:57:45 +0100 Received: from ip120-12-208-87.adsl2.static.versatel.nl (HELO [192.168.0.6]) (87.208.12.120) (smtp-auth username freebsd_arm@myspectrum.nl, mechanism plain) by mail.virtualhost.nl (qpsmtpd/0.84) with (CAMELLIA256-SHA encrypted) ESMTPSA; Mon, 30 Jan 2012 22:57:45 +0100 Message-ID: <4F271258.2060203@myspectrum.nl> Date: Mon, 30 Jan 2012 22:57:44 +0100 From: Jeroen Hofstee User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:9.0) Gecko/20111229 Thunderbird/9.0 MIME-Version: 1.0 To: Tim Kientzle References: <23CB6C35-9450-40BA-9FA3-37C44B328CA8@freebsd.org> <8042D895-3B3D-431E-ADCC-A150BDC838ED@kientzle.com> <7E53112E-DE29-4145-B597-D39AA77252C5@kientzle.com> <4F25D8F9.8040108@myspectrum.nl> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org Subject: Re: FreeBSD and BeagleBone X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2012 21:57:48 -0000 On 01/30/12 03:35, Tim Kientzle wrote: > I might be able to come up with a workaround if I knew > exactly what this option did. The most recent > docs on gcc.gnu.org (for GCC 4.6.2) only mention > the -print-file-name= form. > > Can you point me to docs for this option? > > [jeroen@laptop_freebsd ~]$ gcc34 -print-file-name=include Hi Tim, I can't. I took a pragmatic approach, looked what the eldk compiler did and where it differed. Googled for it, and found some print-file-name related bugs fixes and simple _assumed_ those where not included in the FreeBSD gcc, and the cause of the problem. That was wrong however... Document or not, gcc does report the base include dir, where stdarg.h and friend live since at least 2006 on "common" gcc's. Below some output of FreeBSD gcc's from the ports. /usr/local/lib/gcc/i386-portbld-freebsd9.0/3.4.6/gcc/i386-portbld-freebsd9.0/3.4.6/include [jeroen@laptop_freebsd ~]$ gcc42 -print-file-name=include /usr/local/lib/gcc42/gcc/i386-portbld-freebsd9.0/4.2.5/include [jeroen@laptop_freebsd ~]$ gcc44 -print-file-name=include /usr/local/lib/gcc44/gcc/i386-portbld-freebsd9.0/4.4.7/include [jeroen@laptop_freebsd ~]$ gcc47 -print-file-name=include /usr/local/lib/gcc47/gcc/i386-portbld-freebsd9.0/4.7.0/include # and the base version [jeroen@laptop_freebsd ~]$ gcc -print-file-name=include include --- So I dug around a bit and came across: arm-freebsd-gcc -dumpspecs [port versions] *startfile_prefix_spec: [base gcc] *startfile_prefix_spec: /usr/lib/ [base arm-freebsd-gcc] *startfile_prefix_spec: /usr/arm-freebsd/usr/lib/ ---- So based on the assumption that limits searching to the lib dir, I hacked it in: cd /usr/arm-freebsd/usr/lib ln -s ../include/ [jeroen@laptop_freebsd ~/u-boot]$ arm-freebsd-gcc -print-file-name=include /usr/arm-freebsd/usr/lib/include Compiling works as well (besides the -lc). So the new question now is [afaic], what is startfile_prefix_spec and why is it set in the base gcc and empty in the port versions. I don't know the answer at this moment... Regards, Jeroen From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 03:31:47 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71FA41065670 for ; Tue, 31 Jan 2012 03:31:47 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id 4D0C28FC0A for ; Tue, 31 Jan 2012 03:31:46 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta15.emeryville.ca.mail.comcast.net with comcast id U3VE1i0030QkzPwAF3XmYo; Tue, 31 Jan 2012 03:31:46 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta02.emeryville.ca.mail.comcast.net with comcast id U3Xl1i00m4NgCEG8N3Xmii; Tue, 31 Jan 2012 03:31:46 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q0V3Vixs019711 for ; Mon, 30 Jan 2012 20:31:44 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: freebsd-arm@freebsd.org Content-Type: multipart/mixed; boundary="=-qVk9FMQEACnfJqjUCHNo" Date: Mon, 30 Jan 2012 20:31:43 -0700 Message-Id: <1327980703.1662.240.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.26.0 FreeBSD GNOME Team Port Subject: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 03:31:47 -0000 --=-qVk9FMQEACnfJqjUCHNo Content-Type: text/plain Content-Transfer-Encoding: 7bit I would like to revive and add to this old topic. I'm using the original subject line of the threads from long ago just to help out anyone searching for info in the future; we ran into the problem on Atmel at91rm9200 hardware rather than SheevaPlug. The original threads are archived here: http://lists.freebsd.org/pipermail/freebsd-arm/2010-March/002243.html http://lists.freebsd.org/pipermail/freebsd-arm/2010-November/002635.html To summarize them... Ever since 8.0, performance of userland code on arm systems using VIVT cache ranges from bad to unusable, with symptoms that tend to be hard to nail down definitively. Much of the evidence pointed to the instruction cache being disabled on some pages of apps and shared libraries, sometimes. Mark Tinguely explained about how and why it's necessary to disable caching on a page when there are multiple mappings and at least one is a writable mapping. There were some patches of pmap-layer code that had visible effects but never really fixed the problem. I don't think anybody ever definitively nailed down why some executable pages seem to permanently lose their icache enable bit. I tracked down the cause and developed a workaround (I'll post patches), but to really fix the problem I would need a lot of help from VM/VFS gurus. I apologize in advance for a bit of hand-waving in what follows here. It was months ago that I was immersed in this problem; now I'm working from a few notes and a fading memory. I figured I'd better just post before it fades completely, and hopefully some ensuing discussion will help me remember more details. I also still have a couple sandboxes built with instrumented code that I could dust off and run with, to help answer any questions that arise. One of the most confusing symptoms of the problem is that performance can change from run to run, and most especially it can change after rebooting. It turns out the run-to-run differences are based on what type of IO brought each executable page into memory. When portions of an executable file (including shared libs) are read or written using "normal IO" such as read(2), write(2), etc -- calls that end up in ffs_read() and ffs_write() -- a kernel-writable mapping for the pages is made before the physical IO is initiated, and that mapping stays in place and icache is disabled on those pages as long as the buffer remains in the cache (which for something like libc means forever). When pages are mapped as executable with mmap(2) and then the IO is done via demand paging when the pages are accessed, a temporary kernel writable mapping is made for the duration of the IO operation and then is removed again when the physical IO is completed (leaving just a read/execute mapping). When the last writable mapping is removed the icache bit is restored on the page. (Semi-germane aside: the aio routines appear to work like the pager IO, making a temporary writable kva mapping only for the duration of the physical IO.) The cause of the variability in symptoms is a race between the two types of IO that happens when shared libs are loaded. The race is kicked off by libexec/rtld-elf/map_object.c; it uses pread(2) to load the first 4K of a file to read the headers so that it can mmap() the file as needed. The pread() eventually lands in ffs_read() which decides to do a cluster read or normal read-ahead. Usually the read-ahead IO gets the blocks into the buffer cache (and thus disables icache on all those pages) before map_object() gets much work done, so the first part of a shared library usually ends up icache-disabled. If it's a small shared lib the whole library may end up icache-disabled due to read-ahead. Other times it appears that map_object() gets the pages mapped and triggers demand-paging IO which completes before the readahead IO initiated by the first 4K read, and in those cases the icache bit on the pages gets turned back on when the temporary kernel mappings are unmapped. So when cluster or read-ahead IO wins the race, app performance is bad until the next reboot or filesystem unmount or something else pushes those blocks out of the buffer cache (which never happens on our embedded systems). How badly the app performs depends on what shared libs it uses and the results of the races as each lib was loaded. When some demand-paging IO completes before the corresponding read-ahead IO for the blocks at the start of a library, it seems to cause any further read-ahead to stop as I remember it, so the app doesn't take such a big performance hit, sometimes hardly any hit at all. In addition to the races on loading shared libs, doing "normal IO things" to executable files and libs, such as compiling a new copy or using cp or even 'cat app >/dev/null' which I think came up in the original thread, will cause that app to execute without icache on its pages until its blocks are pushed out of the buffer cache. Here's where I have to be extra-hand-wavy... I think the right way to fix this is to make ffs_read/write (or maybe I guess all vop_read and vop_write implementations) work more like aio and pager io in the sense that they should make a temporary kva mapping that lasts only as long as it takes to do the physical IO and associated uio operations. I vaguely remember thinking that the place to make that happen was along the lines of doing the mapping in getblk() (or maybe breada()?) and unmapping it in bdone(), but I was quite frankly lost in that twisty maze of code and never felt like I understood it well enough to even make an experimental stab at such changes. I have two patches related to this stuff. They were generated from 8.2 sources but I've confirmed that they apply properly to -current. One patch modifies map_object.c to use mmap()+memcpy() instead of pread(). I think it's a useful enhancement even without its effect on this icache problem, because it seems to me that doing a readahead on a shared library will bring in pages that may never be referenced and wouldn't have required any physical memory or IO resources if the readahead hadn't happened. The other is a pure hack-workaround that's most helpful when you're developing code for an arm platform. It forces on the O_DIRECT flag in ffs_write() (and optionally ffs_read() but that's disabled by default) for executable files, to keep the blocks out of the buffer cache when doing normal IO stuff. It's ugly brute force, but it's good enough to let us develop and deploy embedded systems code using FreeBSD 8.2. This is not to be committed, this is just a workaround that let us start using 8.2 before finding a real fix to the root problem. Anyone else trying to work with 8.0 or later on VIVT-cache arm chips might find it useful until a proper fix is developed. -- Ian --=-qVk9FMQEACnfJqjUCHNo Content-Disposition: inline; filename="ffs_vnops_icache_hack.patch" Content-Type: text/x-patch; name="ffs_vnops_icache_hack.patch"; charset="us-ascii" Content-Transfer-Encoding: 7bit --- sys/ufs/ffs/ffs_vnops.c Thu Jun 16 14:43:20 2011 -0600 +++ sys/ufs/ffs/ffs_vnops.c Mon Jan 30 17:54:44 2012 -0700 @@ -467,6 +467,18 @@ ffs_read(ap) seqcount = ap->a_ioflag >> IO_SEQSHIFT; ip = VTOI(vp); + // This hack ensures that executable code never ends up in the buffer cache. + // It is currently disabled. + // It helps work around disabled-icache due to kernel-writable mappings. + // However, shell script files are executable and caching them is useful, so + // this is disabled for now. With the rtld-elf mmap() patch in place, + // nothing normally ever calls read on an executable file so this code + // doesn't buy us much. +#if 0 && defined(__arm__) + if (vp->v_type == VREG && ip->i_mode & IEXEC) + ioflag |= IO_DIRECT; +#endif + #ifdef INVARIANTS if (uio->uio_rw != UIO_READ) panic("ffs_read: mode"); @@ -670,6 +682,17 @@ ffs_write(ap) seqcount = ap->a_ioflag >> IO_SEQSHIFT; ip = VTOI(vp); + // This hack ensures that executable code never ends up in the buffer cache. + // It helps work around disabled-icache due to kernel-writable mappings. + // On a deployed production system, nothing normally ever calls write() on + // an executable file. This hack exists to allow development on the system + // (so that you can do things like copy a new executable onto the system + // without having that destroy performance on subsequent runs). +#if defined(__arm__) + if (vp->v_type == VREG && ip->i_mode & IEXEC) + ioflag |= IO_DIRECT; +#endif + #ifdef INVARIANTS if (uio->uio_rw != UIO_WRITE) panic("ffs_write: mode"); --=-qVk9FMQEACnfJqjUCHNo Content-Disposition: inline; filename="rtld-elf-map_object.patch" Content-Type: text/x-patch; name="rtld-elf-map_object.patch"; charset="us-ascii" Content-Transfer-Encoding: 7bit diff -r 0cb0be36b70f libexec/rtld-elf/map_object.c --- libexec/rtld-elf/map_object.c Thu Jun 16 14:43:20 2011 -0600 +++ libexec/rtld-elf/map_object.c Mon Jan 30 20:03:45 2012 -0700 @@ -272,11 +272,16 @@ get_elf_header (int fd, const char *path char buf[PAGE_SIZE]; } u; ssize_t nbytes; + void *mapped; - if ((nbytes = pread(fd, u.buf, PAGE_SIZE, 0)) == -1) { - _rtld_error("%s: read error: %s", path, strerror(errno)); + /* Use mmap() + memcpy() rather than [p]read() to avoid readahead. */ + nbytes = sizeof(u.buf); + if ((mapped = mmap(NULL, nbytes, PROT_READ, 0, fd, 0)) == (caddr_t) -1) { + _rtld_error("%s: mmap of header failed: %s", path, strerror(errno)); return NULL; } + memcpy(u.buf, mapped, nbytes); + munmap(mapped, nbytes); /* Make sure the file is valid */ if (nbytes < (ssize_t)sizeof(Elf_Ehdr) || !IS_ELF(u.hdr)) { --=-qVk9FMQEACnfJqjUCHNo-- From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 05:12:38 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 968CC106564A for ; Tue, 31 Jan 2012 05:12:38 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 4EDA88FC0C for ; Tue, 31 Jan 2012 05:12:37 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q0V5CZAc051923; Tue, 31 Jan 2012 05:12:35 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.119] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id jih5z7xrfwbgqc7ku2rvyesc8n; Tue, 31 Jan 2012 05:12:35 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Tim Kientzle In-Reply-To: <4F271258.2060203@myspectrum.nl> Date: Mon, 30 Jan 2012 21:12:33 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <23CB6C35-9450-40BA-9FA3-37C44B328CA8@freebsd.org> <8042D895-3B3D-431E-ADCC-A150BDC838ED@kientzle.com> <7E53112E-DE29-4145-B597-D39AA77252C5@kientzle.com> <4F25D8F9.8040108@myspectrum.nl> <4F271258.2060203@myspectrum.nl> To: Jeroen Hofstee X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-arm@freebsd.org Subject: Re: FreeBSD and BeagleBone X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 05:12:38 -0000 On Jan 30, 2012, at 1:57 PM, Jeroen Hofstee wrote: > On 01/30/12 03:35, Tim Kientzle wrote: >> I might be able to come up with a workaround if I knew >> exactly what this option did. The most recent >> docs on gcc.gnu.org (for GCC 4.6.2) only mention >> the -print-file-name=3D form. >>=20 >> Can you point me to docs for this option? >>=20 >> [jeroen@laptop_freebsd ~]$ gcc34 -print-file-name=3Dinclude > Hi Tim, I can't. I took a pragmatic approach, looked what the eldk = compiler did > and where it differed. Googled for it, and found some print-file-name = related > bugs fixes and simple _assumed_ those where not included in the = FreeBSD gcc, > and the cause of the problem. I traced through some of the code in gcc.c and managed to puzzle this out. I think this is a succinct definition of that option: -print-file-name=3Dinclude Prints the name of the directory containing stdarg.h and other basic ISO C header files. The current gcc.c implementation works on Linux because the directory in question is called 'include' and resides under one of the standard 'lib' directories. So it can be found via a search of the library paths. Of course, FreeBSD consolidates these headers with the rest of the system headers under /usr/include, which is not a subdirectory of any library path, so isn't found by the stock gcc.c implementation. That explains why the following symlink workaround works (which is the same one you found): $ sudo ln -s /usr/arm-freebsd/usr/include = /usr/arm-freebsd/usr/lib/include $ arm-freebsd-gcc -print-file-name=3Dinclude /usr/arm-freebsd/usr/lib/include The base system could do the same by creating a symlink /usr/lib/include =3D=3D> /usr/include, but I'm not convinced the FreeBSD folks would go for that. ;-) Long-term, I think the real answer is to implement -print-file-name=3Dinclude as a special option within gcc.c (separate from -print-file-name=3D). Tim From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 05:45:46 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13183106564A for ; Tue, 31 Jan 2012 05:45:46 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 8380E8FC14 for ; Tue, 31 Jan 2012 05:45:45 +0000 (UTC) Received: from 63.imp.bsdimp.com (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id q0V5dcrA013305 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Mon, 30 Jan 2012 22:39:38 -0700 (MST) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1327980703.1662.240.camel@revolution.hippie.lan> Date: Mon, 30 Jan 2012 22:39:37 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1327980703.1662.240.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Mon, 30 Jan 2012 22:39:39 -0700 (MST) Cc: freebsd-arm@FreeBSD.org Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 05:45:46 -0000 Hi Ian, Do you have any data on what 9.0 does? Warner On Jan 30, 2012, at 8:31 PM, Ian Lepore wrote: > I would like to revive and add to this old topic. I'm using the > original subject line of the threads from long ago just to help out > anyone searching for info in the future; we ran into the problem on > Atmel at91rm9200 hardware rather than SheevaPlug. The original = threads > are archived here: >=20 > http://lists.freebsd.org/pipermail/freebsd-arm/2010-March/002243.html > = http://lists.freebsd.org/pipermail/freebsd-arm/2010-November/002635.html >=20 > To summarize them... Ever since 8.0, performance of userland code on = arm > systems using VIVT cache ranges from bad to unusable, with symptoms = that > tend to be hard to nail down definitively. Much of the evidence = pointed > to the instruction cache being disabled on some pages of apps and = shared > libraries, sometimes. Mark Tinguely explained about how and why it's > necessary to disable caching on a page when there are multiple = mappings > and at least one is a writable mapping. There were some patches of > pmap-layer code that had visible effects but never really fixed the > problem. I don't think anybody ever definitively nailed down why some > executable pages seem to permanently lose their icache enable bit. >=20 > I tracked down the cause and developed a workaround (I'll post = patches), > but to really fix the problem I would need a lot of help from VM/VFS > gurus. =20 >=20 > I apologize in advance for a bit of hand-waving in what follows here. > It was months ago that I was immersed in this problem; now I'm working > from a few notes and a fading memory. I figured I'd better just post > before it fades completely, and hopefully some ensuing discussion will > help me remember more details. I also still have a couple sandboxes > built with instrumented code that I could dust off and run with, to = help > answer any questions that arise. >=20 > One of the most confusing symptoms of the problem is that performance > can change from run to run, and most especially it can change after > rebooting. It turns out the run-to-run differences are based on what > type of IO brought each executable page into memory. =20 >=20 > When portions of an executable file (including shared libs) are read = or > written using "normal IO" such as read(2), write(2), etc -- calls that > end up in ffs_read() and ffs_write() -- a kernel-writable mapping for > the pages is made before the physical IO is initiated, and that = mapping > stays in place and icache is disabled on those pages as long as the > buffer remains in the cache (which for something like libc means > forever). >=20 > When pages are mapped as executable with mmap(2) and then the IO is = done > via demand paging when the pages are accessed, a temporary kernel > writable mapping is made for the duration of the IO operation and then > is removed again when the physical IO is completed (leaving just a > read/execute mapping). When the last writable mapping is removed the > icache bit is restored on the page. >=20 > (Semi-germane aside: the aio routines appear to work like the pager = IO, > making a temporary writable kva mapping only for the duration of the > physical IO.) >=20 > The cause of the variability in symptoms is a race between the two = types > of IO that happens when shared libs are loaded. The race is kicked = off > by libexec/rtld-elf/map_object.c; it uses pread(2) to load the first = 4K > of a file to read the headers so that it can mmap() the file as = needed. > The pread() eventually lands in ffs_read() which decides to do a = cluster > read or normal read-ahead. Usually the read-ahead IO gets the blocks > into the buffer cache (and thus disables icache on all those pages) > before map_object() gets much work done, so the first part of a shared > library usually ends up icache-disabled. If it's a small shared lib = the > whole library may end up icache-disabled due to read-ahead. =20 >=20 > Other times it appears that map_object() gets the pages mapped and > triggers demand-paging IO which completes before the readahead IO > initiated by the first 4K read, and in those cases the icache bit on = the > pages gets turned back on when the temporary kernel mappings are > unmapped. >=20 > So when cluster or read-ahead IO wins the race, app performance is bad > until the next reboot or filesystem unmount or something else pushes > those blocks out of the buffer cache (which never happens on our > embedded systems). How badly the app performs depends on what shared > libs it uses and the results of the races as each lib was loaded. = When > some demand-paging IO completes before the corresponding read-ahead IO > for the blocks at the start of a library, it seems to cause any = further > read-ahead to stop as I remember it, so the app doesn't take such a = big > performance hit, sometimes hardly any hit at all. =20 >=20 > In addition to the races on loading shared libs, doing "normal IO > things" to executable files and libs, such as compiling a new copy or > using cp or even 'cat app >/dev/null' which I think came up in the > original thread, will cause that app to execute without icache on its > pages until its blocks are pushed out of the buffer cache. >=20 > Here's where I have to be extra-hand-wavy... I think the right way to > fix this is to make ffs_read/write (or maybe I guess all vop_read and > vop_write implementations) work more like aio and pager io in the = sense > that they should make a temporary kva mapping that lasts only as long = as > it takes to do the physical IO and associated uio operations. I = vaguely > remember thinking that the place to make that happen was along the = lines > of doing the mapping in getblk() (or maybe breada()?) and unmapping it > in bdone(), but I was quite frankly lost in that twisty maze of code = and > never felt like I understood it well enough to even make an = experimental > stab at such changes. >=20 > I have two patches related to this stuff. They were generated from = 8.2 > sources but I've confirmed that they apply properly to -current. >=20 > One patch modifies map_object.c to use mmap()+memcpy() instead of > pread(). I think it's a useful enhancement even without its effect on > this icache problem, because it seems to me that doing a readahead on = a > shared library will bring in pages that may never be referenced and > wouldn't have required any physical memory or IO resources if the > readahead hadn't happened. >=20 > The other is a pure hack-workaround that's most helpful when you're > developing code for an arm platform. It forces on the O_DIRECT flag = in > ffs_write() (and optionally ffs_read() but that's disabled by default) > for executable files, to keep the blocks out of the buffer cache when > doing normal IO stuff. It's ugly brute force, but it's good enough to > let us develop and deploy embedded systems code using FreeBSD 8.2. = This > is not to be committed, this is just a workaround that let us start > using 8.2 before finding a real fix to the root problem. Anyone else > trying to work with 8.0 or later on VIVT-cache arm chips might find it > useful until a proper fix is developed. >=20 > -- Ian >=20 > --- sys/ufs/ffs/ffs_vnops.c Thu Jun 16 14:43:20 2011 -0600 > +++ sys/ufs/ffs/ffs_vnops.c Mon Jan 30 17:54:44 2012 -0700 > @@ -467,6 +467,18 @@ ffs_read(ap) > seqcount =3D ap->a_ioflag >> IO_SEQSHIFT; > ip =3D VTOI(vp); >=20 > + // This hack ensures that executable code never ends up in the = buffer cache. > + // It is currently disabled. > + // It helps work around disabled-icache due to kernel-writable = mappings. > + // However, shell script files are executable and caching them = is useful, so > + // this is disabled for now. With the rtld-elf mmap() patch in = place, > + // nothing normally ever calls read on an executable file so = this code > + // doesn't buy us much. > +#if 0 && defined(__arm__) > + if (vp->v_type =3D=3D VREG && ip->i_mode & IEXEC) > + ioflag |=3D IO_DIRECT; > +#endif =20 > + > #ifdef INVARIANTS > if (uio->uio_rw !=3D UIO_READ) > panic("ffs_read: mode"); > @@ -670,6 +682,17 @@ ffs_write(ap) > seqcount =3D ap->a_ioflag >> IO_SEQSHIFT; > ip =3D VTOI(vp); >=20 > + // This hack ensures that executable code never ends up in the = buffer cache. > + // It helps work around disabled-icache due to kernel-writable = mappings. > + // On a deployed production system, nothing normally ever calls = write() on > + // an executable file. This hack exists to allow development on = the system > + // (so that you can do things like copy a new executable onto = the system > + // without having that destroy performance on subsequent runs). > +#if defined(__arm__) > + if (vp->v_type =3D=3D VREG && ip->i_mode & IEXEC) > + ioflag |=3D IO_DIRECT; > +#endif > + > #ifdef INVARIANTS > if (uio->uio_rw !=3D UIO_WRITE) > panic("ffs_write: mode"); > diff -r 0cb0be36b70f libexec/rtld-elf/map_object.c > --- libexec/rtld-elf/map_object.c Thu Jun 16 14:43:20 2011 -0600 > +++ libexec/rtld-elf/map_object.c Mon Jan 30 20:03:45 2012 -0700 > @@ -272,11 +272,16 @@ get_elf_header (int fd, const char *path > char buf[PAGE_SIZE]; > } u; > ssize_t nbytes; > + void *mapped; >=20 > - if ((nbytes =3D pread(fd, u.buf, PAGE_SIZE, 0)) =3D=3D -1) { > - _rtld_error("%s: read error: %s", path, strerror(errno)); > + /* Use mmap() + memcpy() rather than [p]read() to avoid = readahead. */ > + nbytes =3D sizeof(u.buf); > + if ((mapped =3D mmap(NULL, nbytes, PROT_READ, 0, fd, 0)) =3D=3D = (caddr_t) -1) { > + _rtld_error("%s: mmap of header failed: %s", path, = strerror(errno)); > return NULL; > } > + memcpy(u.buf, mapped, nbytes); > + munmap(mapped, nbytes); >=20 > /* Make sure the file is valid */ > if (nbytes < (ssize_t)sizeof(Elf_Ehdr) || !IS_ELF(u.hdr)) { > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 15:54:09 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0981B106566C for ; Tue, 31 Jan 2012 15:54:09 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id E2FCA8FC0A for ; Tue, 31 Jan 2012 15:54:08 +0000 (UTC) Received: from omta19.emeryville.ca.mail.comcast.net ([76.96.30.76]) by qmta11.emeryville.ca.mail.comcast.net with comcast id UFXi1i0021eYJf8ABFu8TV; Tue, 31 Jan 2012 15:54:08 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta19.emeryville.ca.mail.comcast.net with comcast id UFu71i00e4NgCEG01Fu88u; Tue, 31 Jan 2012 15:54:08 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q0VFs5jV020379; Tue, 31 Jan 2012 08:54:05 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1327980703.1662.240.camel@revolution.hippie.lan> Content-Type: text/plain Date: Tue, 31 Jan 2012 08:54:05 -0700 Message-Id: <1328025245.1662.289.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.26.0 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@FreeBSD.org Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 15:54:09 -0000 On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote: > Hi Ian, > > Do you have any data on what 9.0 does? > > Warner No. Do you have reason to believe it will be different than 8.x? It would be a major effort right now to get anything later than 8.2 built and running on one of our arm platforms. Maybe not as hard as the 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs from stock FreeBSD that have to be analyzed and merged by hand. Actually before that can even happen I'd have to grab a snapshot of 9.0 and do an svn->Hg conversion to even be able to start merging the diffs (and I'm hardly an Hg expert, but those in the company who are let me know last week that they're just as busy as me, and I'm on my own for this kind of work). It's work I want to do, but I suspect it's going to happen later rather than sooner because product deadlines are beginning to loom and my ability to spend most of my time working on the OS side of things is waning. If there are some specific changes you've got in mind that affect this problem I might be able to backport and test them faster than I could get a full 9.0 or -current build environment working, just point me at them. -- Ian From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 16:42:10 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE972106566B for ; Tue, 31 Jan 2012 16:42:10 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0D18FC12 for ; Tue, 31 Jan 2012 16:42:07 +0000 (UTC) Received: from 63.imp.bsdimp.com (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id q0VGbkVP021484 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Tue, 31 Jan 2012 09:37:47 -0700 (MST) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1328025245.1662.289.camel@revolution.hippie.lan> Date: Tue, 31 Jan 2012 09:37:46 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> References: <1327980703.1662.240.camel@revolution.hippie.lan> <1328025245.1662.289.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Tue, 31 Jan 2012 09:37:47 -0700 (MST) Cc: freebsd-arm@FreeBSD.ORG Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 16:42:10 -0000 On Jan 31, 2012, at 8:54 AM, Ian Lepore wrote: > On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote: >> Hi Ian, >>=20 >> Do you have any data on what 9.0 does? >>=20 >> Warner >=20 > No. Do you have reason to believe it will be different than 8.x? >=20 > It would be a major effort right now to get anything later than 8.2 > built and running on one of our arm platforms. Maybe not as hard as = the > 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs = from > stock FreeBSD that have to be analyzed and merged by hand. Actually > before that can even happen I'd have to grab a snapshot of 9.0 and do = an > svn->Hg conversion to even be able to start merging the diffs (and I'm > hardly an Hg expert, but those in the company who are let me know last > week that they're just as busy as me, and I'm on my own for this kind = of > work). It's work I want to do, but I suspect it's going to happen = later > rather than sooner because product deadlines are beginning to loom and > my ability to spend most of my time working on the OS side of things = is > waning. >=20 > If there are some specific changes you've got in mind that affect this > problem I might be able to backport and test them faster than I could > get a full 9.0 or -current build environment working, just point me at > them. I thought that we'd done a root cause of this and had put a fix into the = vm system. Lemme look... ------------------------------------------------------------------------ r224049 | marcel | 2011-07-14 20:11:26 -0600 (Thu, 14 Jul 2011) | 2 = lines In pmap_protect(), don't call vm_page_dirty() if the page is unmanaged. ------------------------------------------------------------------------ r221844 | cognet | 2011-05-13 09:54:12 -0600 (Fri, 13 May 2011) | 4 = lines In pmap_change_wiring(), use the right argument for pmap_modify_pv(). It only worked because the only consumer calls pmap_change_wiring() to = remove the wiring. ------------------------------------------------------------------------ r212507 | cognet | 2010-09-12 14:46:32 -0600 (Sun, 12 Sep 2010) | 5 = lines In pmap_remove_all(), do not decrease pm_stats.wired_count if the = mapping was wired, as it's been done later in pmap_nuke_pv(). Submitted by: Mark Tinguely ------------------------------------------------------------------------ r209223 | cognet | 2010-06-15 16:16:02 -0600 (Tue, 15 Jun 2010) | 4 = lines Turn off cache if there's more than one kernel mapping, and one is = writable. Submitted by: Mark Tinguely ------------------------------------------------------------------------ r205028 | raj | 2010-03-11 14:16:54 -0700 (Thu, 11 Mar 2010) | 12 lines Fix ARM cache handling yet more. 1) vm_machdep.c: remove the dangling allocations so they do not un-necessarily turn off the cache upon consecutive access. 2) busdma_machdep.c: remove the same amount than shadow mapped. Reported by: Maks Verver Submitted by: Mark Tinguely Reviewed by: Grzegorz Bernacki MFC after: 3 days ------------------------------------------------------------------------ r203637 | raj | 2010-02-07 13:48:57 -0700 (Sun, 07 Feb 2010) | 19 lines Improve checking whether an ARM VA has a valid mapping before performing = cache sync. VIPT/PIPT caches need valid VA-PA mapping in PTE for a cache operation = to succeed (unlike VIVT). Prior to this fix pmap was using l2pte_valid() = for that check, but this is not sufficient as the function merely checks if a PTE exists (there can be existing but _invalid_ entries in the table). A new pmap_has_valid_mapping() routine is introduced to do this job = right by checking proper PTE flags. Among other potential problems this cures coherency issues with L2 = caches on MV-78100. Submitted by: Grzegorz Bernacki, Piotr Ziecik Reviewed, tested by: marcel Obtained from: Semihalf MFC after: 1 week Only the last two have MFC, so you can start there and see which of = these changes are in... Just thought you might have a reference board that would be easy to = test... Warner= From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 17:30:12 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA0F51065673 for ; Tue, 31 Jan 2012 17:30:12 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id 368E58FC20 for ; Tue, 31 Jan 2012 17:30:02 +0000 (UTC) Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta15.emeryville.ca.mail.comcast.net with comcast id UHLx1i0050vp7WLAFHW2wZ; Tue, 31 Jan 2012 17:30:02 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta05.emeryville.ca.mail.comcast.net with comcast id UHW11i00n4NgCEG8RHW2t7; Tue, 31 Jan 2012 17:30:02 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q0VHTxcZ020488; Tue, 31 Jan 2012 10:29:59 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> References: <1327980703.1662.240.camel@revolution.hippie.lan> <1328025245.1662.289.camel@revolution.hippie.lan> <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> Content-Type: text/plain Date: Tue, 31 Jan 2012 10:29:59 -0700 Message-Id: <1328030999.1662.324.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.26.0 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@FreeBSD.ORG Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 17:30:12 -0000 On Tue, 2012-01-31 at 09:37 -0700, Warner Losh wrote: > On Jan 31, 2012, at 8:54 AM, Ian Lepore wrote: > > > On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote: > >> Hi Ian, > >> > >> Do you have any data on what 9.0 does? > >> > >> Warner > > > > No. Do you have reason to believe it will be different than 8.x? > > > > It would be a major effort right now to get anything later than 8.2 > > built and running on one of our arm platforms. Maybe not as hard as the > > 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs from > > stock FreeBSD that have to be analyzed and merged by hand. Actually > > before that can even happen I'd have to grab a snapshot of 9.0 and do an > > svn->Hg conversion to even be able to start merging the diffs (and I'm > > hardly an Hg expert, but those in the company who are let me know last > > week that they're just as busy as me, and I'm on my own for this kind of > > work). It's work I want to do, but I suspect it's going to happen later > > rather than sooner because product deadlines are beginning to loom and > > my ability to spend most of my time working on the OS side of things is > > waning. > > > > If there are some specific changes you've got in mind that affect this > > problem I might be able to backport and test them faster than I could > > get a full 9.0 or -current build environment working, just point me at > > them. > > I thought that we'd done a root cause of this and had put a fix into the vm system. Lemme look... > > ------------------------------------------------------------------------ > r224049 | marcel | 2011-07-14 20:11:26 -0600 (Thu, 14 Jul 2011) | 2 lines > > In pmap_protect(), don't call vm_page_dirty() if the page is unmanaged. > > > ------------------------------------------------------------------------ > r221844 | cognet | 2011-05-13 09:54:12 -0600 (Fri, 13 May 2011) | 4 lines > > In pmap_change_wiring(), use the right argument for pmap_modify_pv(). > It only worked because the only consumer calls pmap_change_wiring() to remove > the wiring. > > ------------------------------------------------------------------------ > r212507 | cognet | 2010-09-12 14:46:32 -0600 (Sun, 12 Sep 2010) | 5 lines > > In pmap_remove_all(), do not decrease pm_stats.wired_count if the mapping was > wired, as it's been done later in pmap_nuke_pv(). > > Submitted by: Mark Tinguely > > > ------------------------------------------------------------------------ > r209223 | cognet | 2010-06-15 16:16:02 -0600 (Tue, 15 Jun 2010) | 4 lines > > Turn off cache if there's more than one kernel mapping, and one is writable. > > Submitted by: Mark Tinguely > > ------------------------------------------------------------------------ > r205028 | raj | 2010-03-11 14:16:54 -0700 (Thu, 11 Mar 2010) | 12 lines > > Fix ARM cache handling yet more. > > 1) vm_machdep.c: remove the dangling allocations so they do not > un-necessarily turn off the cache upon consecutive access. > > 2) busdma_machdep.c: remove the same amount than shadow mapped. > > Reported by: Maks Verver > Submitted by: Mark Tinguely > Reviewed by: Grzegorz Bernacki > MFC after: 3 days > > ------------------------------------------------------------------------ > r203637 | raj | 2010-02-07 13:48:57 -0700 (Sun, 07 Feb 2010) | 19 lines > > Improve checking whether an ARM VA has a valid mapping before performing cache > sync. > > VIPT/PIPT caches need valid VA-PA mapping in PTE for a cache operation to > succeed (unlike VIVT). Prior to this fix pmap was using l2pte_valid() for that > check, but this is not sufficient as the function merely checks if a PTE > exists (there can be existing but _invalid_ entries in the table). > > A new pmap_has_valid_mapping() routine is introduced to do this job right by > checking proper PTE flags. > > Among other potential problems this cures coherency issues with L2 caches on > MV-78100. > > Submitted by: Grzegorz Bernacki, Piotr Ziecik > Reviewed, tested by: marcel > Obtained from: Semihalf > MFC after: 1 week > > > Only the last two have MFC, so you can start there and see which of these changes are in... > > Just thought you might have a reference board that would be easy to test... > > Warner I think we may have all those changes incorporated except perhaps r224049; I'll make sure of that. r209223 is the change that exposed this situation. I'm skeptical that any of the changes you cite (or any change at all in the pmap layer) will fix the problem, because the problem seems to be rooted in the fact that the vfs buffer cache establishes a kva mapping of the buffer pages with the protections set to READ|WRITE|EXEC and leaves that mapping in place as long as the buffer is in the cache, and r209223 says that as long as there are multiple mappings of a page with at least one writable, that page's i-cache and d-cache bits stay off. (The multiple mappings being the one for the buffer cache that includes write access and one or more READ|EXEC mappings made by pmap() when the executable or library is loaded/relocated.) If my analysis is correct (and I'm fairly sure, if not 100% positive, that it is), then it seems to me that the only fix available is going to be at the vfs layer, and it's going to involve dropping the write access to the pages in the buffer cache once any physical IO and/or uio operations needing write access are completed. Even if I could figure out a patchset to fix the problem, it's going to need a lot of input from the vm gurus to answer questions such as what the performance impact will be to non-VIVT platforms that don't need this extra work done. If the extra work is expensive enough (and I'm not sure I could evaluate that properly) it may need to be conditional on whether the platform needs it. I'm also vaguely uneasy with all this on a purely philosphical level, since this could end up basically infecting MI code with a platform-specific concept. -- Ian From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 20:35:41 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99ADF106566C for ; Tue, 31 Jan 2012 20:35:41 +0000 (UTC) (envelope-from raj@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id E8CD48FC08 for ; Tue, 31 Jan 2012 20:35:40 +0000 (UTC) Received: from localhost (unknown [213.17.239.109]) by smtp.semihalf.com (Postfix) with ESMTP id 05139C3847; Tue, 31 Jan 2012 21:20:12 +0100 (CET) X-Virus-Scanned: by amavisd-new at semihalf.com Received: from smtp.semihalf.com ([213.17.239.109]) by localhost (smtp.semihalf.com [213.17.239.109]) (amavisd-new, port 10024) with ESMTP id bZd8GUPGWW3S; Tue, 31 Jan 2012 21:20:10 +0100 (CET) Received: from [10.0.0.112] (nat3-133.ghnet.pl [91.150.222.133]) by smtp.semihalf.com (Postfix) with ESMTPSA id A0B86C3843; Tue, 31 Jan 2012 21:20:10 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Rafal Jaworowski In-Reply-To: <1328030999.1662.324.camel@revolution.hippie.lan> Date: Tue, 31 Jan 2012 21:20:09 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1327980703.1662.240.camel@revolution.hippie.lan> <1328025245.1662.289.camel@revolution.hippie.lan> <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> <1328030999.1662.324.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) Cc: freebsd-arm@FreeBSD.ORG Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 20:35:41 -0000 On 2012-01-31, at 18:29, Ian Lepore wrote: > On Tue, 2012-01-31 at 09:37 -0700, Warner Losh wrote: >> On Jan 31, 2012, at 8:54 AM, Ian Lepore wrote: >>=20 >>> On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote: >>>> Hi Ian, >>>>=20 >>>> Do you have any data on what 9.0 does? >>>>=20 >>>> Warner >>>=20 >>> No. Do you have reason to believe it will be different than 8.x? >>>=20 >>> It would be a major effort right now to get anything later than 8.2 >>> built and running on one of our arm platforms. Maybe not as hard as = the >>> 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs = from >>> stock FreeBSD that have to be analyzed and merged by hand. Actually >>> before that can even happen I'd have to grab a snapshot of 9.0 and = do an >>> svn->Hg conversion to even be able to start merging the diffs (and = I'm >>> hardly an Hg expert, but those in the company who are let me know = last >>> week that they're just as busy as me, and I'm on my own for this = kind of >>> work). It's work I want to do, but I suspect it's going to happen = later >>> rather than sooner because product deadlines are beginning to loom = and >>> my ability to spend most of my time working on the OS side of things = is >>> waning. >>>=20 >>> If there are some specific changes you've got in mind that affect = this >>> problem I might be able to backport and test them faster than I = could >>> get a full 9.0 or -current build environment working, just point me = at >>> them. >>=20 >> I thought that we'd done a root cause of this and had put a fix into = the vm system. Lemme look... >>=20 >> = ------------------------------------------------------------------------ >> r224049 | marcel | 2011-07-14 20:11:26 -0600 (Thu, 14 Jul 2011) | 2 = lines >>=20 >> In pmap_protect(), don't call vm_page_dirty() if the page is = unmanaged. >>=20 >>=20 >> = ------------------------------------------------------------------------ >> r221844 | cognet | 2011-05-13 09:54:12 -0600 (Fri, 13 May 2011) | 4 = lines >>=20 >> In pmap_change_wiring(), use the right argument for pmap_modify_pv(). >> It only worked because the only consumer calls pmap_change_wiring() = to remove >> the wiring. >>=20 >> = ------------------------------------------------------------------------ >> r212507 | cognet | 2010-09-12 14:46:32 -0600 (Sun, 12 Sep 2010) | 5 = lines >>=20 >> In pmap_remove_all(), do not decrease pm_stats.wired_count if the = mapping was >> wired, as it's been done later in pmap_nuke_pv(). >>=20 >> Submitted by: Mark Tinguely >>=20 >>=20 >> = ------------------------------------------------------------------------ >> r209223 | cognet | 2010-06-15 16:16:02 -0600 (Tue, 15 Jun 2010) | 4 = lines >>=20 >> Turn off cache if there's more than one kernel mapping, and one is = writable. >>=20 >> Submitted by: Mark Tinguely >>=20 >> = ------------------------------------------------------------------------ >> r205028 | raj | 2010-03-11 14:16:54 -0700 (Thu, 11 Mar 2010) | 12 = lines >>=20 >> Fix ARM cache handling yet more. >>=20 >> 1) vm_machdep.c: remove the dangling allocations so they do not >> un-necessarily turn off the cache upon consecutive access. >>=20 >> 2) busdma_machdep.c: remove the same amount than shadow mapped. >>=20 >> Reported by: Maks Verver >> Submitted by: Mark Tinguely >> Reviewed by: Grzegorz Bernacki >> MFC after: 3 days >>=20 >> = ------------------------------------------------------------------------ >> r203637 | raj | 2010-02-07 13:48:57 -0700 (Sun, 07 Feb 2010) | 19 = lines >>=20 >> Improve checking whether an ARM VA has a valid mapping before = performing cache >> sync. >>=20 >> VIPT/PIPT caches need valid VA-PA mapping in PTE for a cache = operation to >> succeed (unlike VIVT). Prior to this fix pmap was using l2pte_valid() = for that >> check, but this is not sufficient as the function merely checks if a = PTE >> exists (there can be existing but _invalid_ entries in the table). >>=20 >> A new pmap_has_valid_mapping() routine is introduced to do this job = right by >> checking proper PTE flags. >>=20 >> Among other potential problems this cures coherency issues with L2 = caches on >> MV-78100. >>=20 >> Submitted by: Grzegorz Bernacki, Piotr Ziecik >> Reviewed, tested by: marcel >> Obtained from: Semihalf >> MFC after: 1 week >>=20 >>=20 >> Only the last two have MFC, so you can start there and see which of = these changes are in... >>=20 >> Just thought you might have a reference board that would be easy to = test... >>=20 >> Warner >=20 > I think we may have all those changes incorporated except perhaps > r224049; I'll make sure of that. =20 >=20 > r209223 is the change that exposed this situation. =20 >=20 > I'm skeptical that any of the changes you cite (or any change at all = in > the pmap layer) will fix the problem, because the problem seems to be > rooted in the fact that the vfs buffer cache establishes a kva mapping > of the buffer pages with the protections set to READ|WRITE|EXEC and > leaves that mapping in place as long as the buffer is in the cache, = and > r209223 says that as long as there are multiple mappings of a page = with > at least one writable, that page's i-cache and d-cache bits stay off. > (The multiple mappings being the one for the buffer cache that = includes > write access and one or more READ|EXEC mappings made by pmap() when = the > executable or library is loaded/relocated.) >=20 > If my analysis is correct (and I'm fairly sure, if not 100% positive, > that it is), then it seems to me that the only fix available is going = to > be at the vfs layer, and it's going to involve dropping the write = access > to the pages in the buffer cache once any physical IO and/or uio > operations needing write access are completed. =20 >=20 > Even if I could figure out a patchset to fix the problem, it's going = to > need a lot of input from the vm gurus to answer questions such as what > the performance impact will be to non-VIVT platforms that don't need > this extra work done. If the extra work is expensive enough (and I'm > not sure I could evaluate that properly) it may need to be conditional > on whether the platform needs it. I'm also vaguely uneasy with all = this > on a purely philosphical level, since this could end up basically > infecting MI code with a platform-specific concept. Not sure if you've seen this, but we were discussing a long standing = problem with FFS clustering (actually with multiple mappings) here: http://lists.freebsd.org/pipermail/freebsd-arm/2008-December/001423.html The changes from Mark were supposed to somewhat mitigate this (by = introducing non-cached entries for consecutive mappings of the already = mapped address ranges), but the problem was still observed and we're = still using forced nocluster for stability. Rafal From owner-freebsd-arm@FreeBSD.ORG Tue Jan 31 20:49:26 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36029106564A for ; Tue, 31 Jan 2012 20:49:26 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta10.emeryville.ca.mail.comcast.net (qmta10.emeryville.ca.mail.comcast.net [76.96.30.17]) by mx1.freebsd.org (Postfix) with ESMTP id 15EDE8FC0C for ; Tue, 31 Jan 2012 20:49:25 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta10.emeryville.ca.mail.comcast.net with comcast id ULpD1i00617UAYkAALpRD3; Tue, 31 Jan 2012 20:49:25 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta13.emeryville.ca.mail.comcast.net with comcast id ULpQ1i00V4NgCEG8ZLpQ47; Tue, 31 Jan 2012 20:49:25 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q0VKnMYk020647; Tue, 31 Jan 2012 13:49:22 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Rafal Jaworowski In-Reply-To: References: <1327980703.1662.240.camel@revolution.hippie.lan> <1328025245.1662.289.camel@revolution.hippie.lan> <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> <1328030999.1662.324.camel@revolution.hippie.lan> Content-Type: text/plain Date: Tue, 31 Jan 2012 13:49:22 -0700 Message-Id: <1328042962.1662.398.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.26.0 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@FreeBSD.ORG Subject: Re: Performance of SheevaPlug on 8-stable X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2012 20:49:26 -0000 On Tue, 2012-01-31 at 21:20 +0100, Rafal Jaworowski wrote: > On 2012-01-31, at 18:29, Ian Lepore wrote: > > > On Tue, 2012-01-31 at 09:37 -0700, Warner Losh wrote: > >> On Jan 31, 2012, at 8:54 AM, Ian Lepore wrote: > >> > >>> On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote: > >>>> Hi Ian, > >>>> > >>>> Do you have any data on what 9.0 does? > >>>> > >>>> Warner > >>> > >>> No. Do you have reason to believe it will be different than 8.x? > >>> > >>> It would be a major effort right now to get anything later than 8.2 > >>> built and running on one of our arm platforms. Maybe not as hard as the > >>> 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs from > >>> stock FreeBSD that have to be analyzed and merged by hand. Actually > >>> before that can even happen I'd have to grab a snapshot of 9.0 and do an > >>> svn->Hg conversion to even be able to start merging the diffs (and I'm > >>> hardly an Hg expert, but those in the company who are let me know last > >>> week that they're just as busy as me, and I'm on my own for this kind of > >>> work). It's work I want to do, but I suspect it's going to happen later > >>> rather than sooner because product deadlines are beginning to loom and > >>> my ability to spend most of my time working on the OS side of things is > >>> waning. > >>> > >>> If there are some specific changes you've got in mind that affect this > >>> problem I might be able to backport and test them faster than I could > >>> get a full 9.0 or -current build environment working, just point me at > >>> them. > >> > >> I thought that we'd done a root cause of this and had put a fix into the vm system. Lemme look... > >> > >> ------------------------------------------------------------------------ > >> r224049 | marcel | 2011-07-14 20:11:26 -0600 (Thu, 14 Jul 2011) | 2 lines > >> > >> In pmap_protect(), don't call vm_page_dirty() if the page is unmanaged. > >> > >> > >> ------------------------------------------------------------------------ > >> r221844 | cognet | 2011-05-13 09:54:12 -0600 (Fri, 13 May 2011) | 4 lines > >> > >> In pmap_change_wiring(), use the right argument for pmap_modify_pv(). > >> It only worked because the only consumer calls pmap_change_wiring() to remove > >> the wiring. > >> > >> ------------------------------------------------------------------------ > >> r212507 | cognet | 2010-09-12 14:46:32 -0600 (Sun, 12 Sep 2010) | 5 lines > >> > >> In pmap_remove_all(), do not decrease pm_stats.wired_count if the mapping was > >> wired, as it's been done later in pmap_nuke_pv(). > >> > >> Submitted by: Mark Tinguely > >> > >> > >> ------------------------------------------------------------------------ > >> r209223 | cognet | 2010-06-15 16:16:02 -0600 (Tue, 15 Jun 2010) | 4 lines > >> > >> Turn off cache if there's more than one kernel mapping, and one is writable. > >> > >> Submitted by: Mark Tinguely > >> > >> ------------------------------------------------------------------------ > >> r205028 | raj | 2010-03-11 14:16:54 -0700 (Thu, 11 Mar 2010) | 12 lines > >> > >> Fix ARM cache handling yet more. > >> > >> 1) vm_machdep.c: remove the dangling allocations so they do not > >> un-necessarily turn off the cache upon consecutive access. > >> > >> 2) busdma_machdep.c: remove the same amount than shadow mapped. > >> > >> Reported by: Maks Verver > >> Submitted by: Mark Tinguely > >> Reviewed by: Grzegorz Bernacki > >> MFC after: 3 days > >> > >> ------------------------------------------------------------------------ > >> r203637 | raj | 2010-02-07 13:48:57 -0700 (Sun, 07 Feb 2010) | 19 lines > >> > >> Improve checking whether an ARM VA has a valid mapping before performing cache > >> sync. > >> > >> VIPT/PIPT caches need valid VA-PA mapping in PTE for a cache operation to > >> succeed (unlike VIVT). Prior to this fix pmap was using l2pte_valid() for that > >> check, but this is not sufficient as the function merely checks if a PTE > >> exists (there can be existing but _invalid_ entries in the table). > >> > >> A new pmap_has_valid_mapping() routine is introduced to do this job right by > >> checking proper PTE flags. > >> > >> Among other potential problems this cures coherency issues with L2 caches on > >> MV-78100. > >> > >> Submitted by: Grzegorz Bernacki, Piotr Ziecik > >> Reviewed, tested by: marcel > >> Obtained from: Semihalf > >> MFC after: 1 week > >> > >> > >> Only the last two have MFC, so you can start there and see which of these changes are in... > >> > >> Just thought you might have a reference board that would be easy to test... > >> > >> Warner > > > > I think we may have all those changes incorporated except perhaps > > r224049; I'll make sure of that. > > > > r209223 is the change that exposed this situation. > > > > I'm skeptical that any of the changes you cite (or any change at all in > > the pmap layer) will fix the problem, because the problem seems to be > > rooted in the fact that the vfs buffer cache establishes a kva mapping > > of the buffer pages with the protections set to READ|WRITE|EXEC and > > leaves that mapping in place as long as the buffer is in the cache, and > > r209223 says that as long as there are multiple mappings of a page with > > at least one writable, that page's i-cache and d-cache bits stay off. > > (The multiple mappings being the one for the buffer cache that includes > > write access and one or more READ|EXEC mappings made by pmap() when the > > executable or library is loaded/relocated.) > > > > If my analysis is correct (and I'm fairly sure, if not 100% positive, > > that it is), then it seems to me that the only fix available is going to > > be at the vfs layer, and it's going to involve dropping the write access > > to the pages in the buffer cache once any physical IO and/or uio > > operations needing write access are completed. > > > > Even if I could figure out a patchset to fix the problem, it's going to > > need a lot of input from the vm gurus to answer questions such as what > > the performance impact will be to non-VIVT platforms that don't need > > this extra work done. If the extra work is expensive enough (and I'm > > not sure I could evaluate that properly) it may need to be conditional > > on whether the platform needs it. I'm also vaguely uneasy with all this > > on a purely philosphical level, since this could end up basically > > infecting MI code with a platform-specific concept. > > Not sure if you've seen this, but we were discussing a long standing > problem with FFS clustering (actually with multiple mappings) here: > > http://lists.freebsd.org/pipermail/freebsd-arm/2008-December/001423.html > > The changes from Mark were supposed to somewhat mitigate this (by > introducing non-cached entries for consecutive mappings of the already > mapped address ranges), but the problem was still observed and we're > still using forced nocluster for stability. > > Rafal > I wasn't aware of that thread, thanks. Reading through it now, it appears that it may be the discussion that eventually led to the changes in r209223 and a few followup refinements that disable caching for pages with multiple mappings. I've never used nocluster, except briefly when first investigating the loss of icache on pages, to see if it made a difference. It made some difference, mainly by changing the readahead behavior from large cluster readaheads to the old readahead logic, so a smaller part of a shared lib became permanently cache-disabled. We also had a series of battles at Symmetricom with the 32 byte corruption during IO. Most of that trouble disappeared when we moved to the 8.2 code, and the last bit of it went away with the patch I posted in http://www.freebsd.org/cgi/query-pr.cgi?pr=160431 (which still isn't committed, but I now have a newer version of the patch that disables interrupts for a shorter time and still gets the job done). I also remember having to fix a few busdma sync calls in various drivers, but I think they were all our own private drivers except for the at91 mci and uart drivers. -- Ian