From owner-freebsd-sparc64@FreeBSD.ORG Sun Dec 11 01:06:57 2005 Return-Path: X-Original-To: sparc64@FreeBSD.org Delivered-To: freebsd-sparc64@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0330C16A41F; Sun, 11 Dec 2005 01:06:57 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id AEFD943D45; Sun, 11 Dec 2005 01:06:56 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 884421A3C26; Sat, 10 Dec 2005 17:06:56 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id C360353212; Sat, 10 Dec 2005 20:06:55 -0500 (EST) Date: Sat, 10 Dec 2005 20:06:55 -0500 From: Kris Kennaway To: alc@FreeBSD.org, sparc64@FreeBSD.org Message-ID: <20051211010655.GA84261@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Cc: Subject: Too many page faults on sparc64? X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2005 01:06:57 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I have been benchmarking jasone's new malloc implementation on a 14-cpu sparc64 (it performs pretty well, but that's not the point of this mail). Measuring MUTEX_PROFILING on a benchmark that does lots of malloc()/free() in a threaded application, I see the following acquisitions (column 3 is relevant): max total count avg cnt_hold cnt_lock name 646 13006535 2153037 6 15742 96076 kern/kern_condvar.c:135 (lockbuilder mtxpool) 330 5397096 2154156 2 950 1035 vm/vm_fault.c:849 (vm page queue mutex) 390 10672286 2154220 4 1068 1328 vm/vm_fault.c:344 (vm page queue mutex) 378 6064832 2154870 2 33658 33444 sparc64/sparc64/trap.c:439 (process lock) 423 7178990 2154870 3 638 591 sparc64/sparc64/trap.c:449 (process lock) 312 4746241 2154903 2 1135 1280 vm/vm_fault.c:907 (vm page queue mutex) 476 5494516 2154903 2 519 1194 vm/vm_fault.c:929 (process lock) 701 66305500 2154903 30 0 189 vm/vm_fault.c:295 (vm object) 735 21517443 2154903 9 0 0 vm/vm_fault.c:906 (vm object) 615 55351240 2155743 25 2219 2158 sparc64/sparc64/pmap.c:1288 (vm page queue mutex) 620 64873979 2155743 30 0 0 sparc64/sparc64/pmap.c:1289 (pmap) 206 3527815 2155838 1 0 0 vm/vm_object.c:452 (vm object) 365 8532728 2165256 3 33002 55736 kern/kern_sx.c:157 (lockbuilder mtxpool) 675 79063301 2165256 36 93683 16249 kern/kern_sx.c:245 (lockbuilder mtxpool) These seem to all be associated with processing of page faults. Measuring with /usr/bin/time -l shows: 246.48 real 176.50 user 419.00 sys 1536 maximum resident set size 7 average shared memory size 323710 average unshared data size 115 average unshared stack size 1795339 page reclaims 0 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 1794932 voluntary context switches 10759 involuntary context switches i.e. the number of mutex acquisitions correlates with the page reclaims and VCSs (verified with other numbers of threads). We're not sure why there are so many page faults being seen, though - there should be on the order of 100 pages touched on this workload, and indeed this is observed on i386. Also, time -l doesn't record any (is the counter broken?) I enabled KTR_TRAP and it shows that most of the traps are type T_DATA_MISS. Why should there be so many of these traps (and why aren't they measured by rusage)? Kris --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFDm3uvWry0BWjoQKURAsQcAKDp/si5mHHymUXoJWExIDbypb3+9gCgnFgY L8YGC6Kf1tUTwBNCkNGSyEw= =esbU -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh--