From owner-freebsd-amd64@FreeBSD.ORG Tue Jul 27 20:44:11 2010 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8117106567A; Tue, 27 Jul 2010 20:44:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by mx1.freebsd.org (Postfix) with ESMTP id 3C02A8FC17; Tue, 27 Jul 2010 20:44:10 +0000 (UTC) Received: from besplex.bde.org (c122-106-147-41.carlnfd1.nsw.optusnet.com.au [122.106.147.41]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o6RKi75Y024773 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 28 Jul 2010 06:44:08 +1000 Date: Wed, 28 Jul 2010 06:44:07 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: John Baldwin In-Reply-To: <201007271540.30774.jhb@freebsd.org> Message-ID: <20100728054627.T11937@besplex.bde.org> References: <201007270928.24959.jhb@freebsd.org> <20100728044151.Y1330@delplex.bde.org> <201007271540.30774.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: jkoshy@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: PERFMON isn't operational on amd64 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2010 20:44:11 -0000 On Tue, 27 Jul 2010, John Baldwin wrote: > On Tuesday, July 27, 2010 3:06:50 pm Bruce Evans wrote: >> hwpmc(4) is far too featureful for me, yet perfmon(4) still does things >> for me that hwpmc cannot do: >> - from userland, without using the library, try undocumented unnamed >> counters. There are a few useful ones for k7. E.g.: >> 0xc8 (k8-fr-retired-near-returns) and >> 0xc9 (k8-fr-retired-near-returns-mispredicted) >> are k8-only according to hwpmc and amd docs, but also work on my k7 >> (after a 4-line fix for perfmon on kx). > > I believe this is not too terribly hard to fix. Indeed. Just remove restrictions. > For the two above I think > this will suffice (manpage would also need to be updated for a full patch): > > (I think the misordering in pmc_events.h is required to preserve userland > ABI.) > > Index: hwpmc_amd.c > =================================================================== > --- hwpmc_amd.c (revision 210547) > +++ hwpmc_amd.c (working copy) > @@ -137,6 +137,8 @@ > { PMC_EV_K7_RETIRED_TAKEN_BRANCHES_MISPREDICTED, 0xC5, 0 }, > { PMC_EV_K7_RETIRED_FAR_CONTROL_TRANSFERS, 0xC6, 0 }, > { PMC_EV_K7_RETIRED_RESYNC_BRANCHES, 0xC7, 0 }, > + { PMC_EV_K7_RETIRED_NEAR_RETURNS, 0xC8, 0 }, > + { PMC_EV_K7_RETIRED_NEAR_RETURNS_MISPREDICTED, 0xC9, 0 }, > { PMC_EV_K7_INTERRUPTS_MASKED_CYCLES, 0xCD, 0 }, > { PMC_EV_K7_INTERRUPTS_MASKED_WHILE_PENDING_CYCLES, 0xCE, 0 }, > { PMC_EV_K7_HARDWARE_INTERRUPTS, 0xCF, 0 }, I think the undocumented ones should require a special flag or something to use. Maybe they depend on the stepping. Here is my complete list (54 items), from a case statement that is supposed to convert magic numbers to the hwpmc/amd names. This needs to be verified. I don't remember checking all of it. In fact, I probably didn't, but just duplicated the k8 names with s/k8/kx/ iff the counter is sometimes nonzero for both k7 and k8, then checked the more interesting ones. IIRC, checking consisted mainly of running normal utilites on k7 and k8 to see if the counts were similar. This wouldn't work if the CPU reasources were too different. % 0x21) src=kx-ls-microarchitectural-resync-by-self-mod-code;; % 0x23) src=kx-ls-buffer2-full;; I couldn't have verified the more arcane and/or lower-level ones like these if I tried. These are documented for k8, and nonzero for k7. The 202 cases not in this list are left out because they are either documented for k7 and identical with k8, or never observed to have nonzero counts. % 0x26) src=kx-ls-retired-cflush-instructions;; % 0x27) src=kx-ls-retired-cpuid-instructions;; Probably verifiable. % 0x2a) src=kx-ls-unknown-$ctr1;; % 0x2b) src=kx-ls-unknown-$ctr1;; % 0x2e) src=k7-ls-unknown-$ctr1;; % 0x2f) src=k7-ls-unknown-$ctr1;; % 0x32) src=kx-ls-unknown-$ctr1;; % 0x33) src=kx-ls-unknown-$ctr1;; % 0x36) src=k7-ls-unknown-$ctr1;; % 0x37) src=k7-ls-unknown-$ctr1;; % 0x3a) src=kx-ls-unknown-$ctr1;; % 0x3b) src=kx-ls-unknown-$ctr1;; % 0x3e) src=k7-ls-unknown-$ctr1;; % 0x3f) src=k7-ls-unknown-$ctr1;; Unknown means it's undocumented but observed to have a nonzero count (kx-*-unkown if nonzero on both, etc.). % 0x48) src=kx-dc-microarchitectural-late-cancel-of-an-access;; % 0x49) src=kx-dc-microarchitectural-early-cancel-of-an-access;; % 0x4d) src=k7-dc-unknown-$ctr1;; % 0x4e) src=k7-dc-unknown-$ctr1;; % 0x4f) src=k7-dc-unknown-$ctr1;; % 0x50) src=kx-dc-unknown-$ctr1;; % 0x51) src=kx-dc-unknown-$ctr1;; % 0x55) src=kx-dc-unknown-$ctr1;; % 0x56) src=kx-dc-unknown-$ctr1;; % 0x57) src=kx-dc-unknown-$ctr1;; % 0x58) src=kx-dc-unknown-$ctr1;; % 0x59) src=kx-dc-unknown-$ctr1;; % 0x5d) src=k7-dc-unknown-$ctr1;; % 0x5e) src=k7-dc-unknown-$ctr1;; % 0x5f) src=k7-dc-unknown-$ctr1;; % 0x64) src=kx-bu-unknown-$ctr1;; % 0x68) src=kx-bu-unknown-$ctr1;; % 0x69) src=kx-bu-unknown-$ctr1;; % 0x76) src=kx-bu-cpu-clk-unhalted;; % 0x82) src=kx-ic-refill-from-l2;; % 0x83) src=kx-ic-refill-from-system;; % 0x87) src=kx-ic-instruction-fetch-stall;; % 0x88) src=kx-ic-return-stack-hit;; % 0x89) src=kx-ic-return-stack-overflow;; Above 2 are even more interesting than the next 2. I should have checked these. % 0xc8) src=kx-fr-retired-near-returns;; % 0xc9) src=kx-fr-retired-near-returns-mispredicted;; % 0xca) src=kx-fr-retired-taken-branches-mispred-by-addr-miscompare;; % 0xd0) src=kx-fr-decoder-empty;; % 0xd1) src=kx-fr-dispatch-stalls;; % 0xd2) src=kx-fr-dispatch-stall-from-branch-abort-to-retire;; % 0xd3) src=kx-fr-dispatch-stall-for-serialization;; % 0xd4) src=kx-fr-dispatch-stall-for-segment-load;; % 0xd5) src=kx-fr-dispatch-stall-when-reorder-buffer-is-full;; % 0xd6) src=kx-fr-dispatch-stall-when-reservation-stations-are-full;; % 0xd7) src=kx-fr-dispatch-stall-when-fpu-is-full;; % 0xd8) src=kx-fr-dispatch-stall-when-ls-is-full;; % 0xd9) src=kx-fr-dispatch-stall-when-waiting-for-all-to-be-quiet;; % 0xda) src=kx-fr-dispatch-stall-when-far-xfer-or-resync-br-pending;; >> - in the kernel count, events at the level of individual functions, non- >> statistically using high resolution kernel profiling. This may result >> in more time spent counting than doing useful work, but provides >> high resolution. > > I would rather us provide this via hwpmc(4). perfmon(4) does not support > programmable events on modern Intel CPUs nor does it support non-x86 CPUs. Sure, but don't remove sort-of-working code until you have a replacement. Profiling just needs to call non-profiling-routine functions to read the counters (for perfmon, the whole module is hackishly configued as profiling-routine so that its normal functions can be called). The functions need to be callable in the "any" context without deadlocking or corrupting the context. This is nontrivial if the context is hwpmc or perfmon. This is especially nontrivial for hwpmc, since it uses NMIs. For perfmon, problems are supposed to be avoided by acquiring exclusive access everything in perfmon while profiling. I can only see it acquiring exclusive access to the counter being used. Bruce