From owner-freebsd-performance@FreeBSD.ORG Tue Nov 25 23:40:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 658C81065670; Tue, 25 Nov 2008 23:40:24 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from proxy.meer.net (proxy.meer.net [64.13.141.13]) by mx1.freebsd.org (Postfix) with ESMTP id 4B98D8FC14; Tue, 25 Nov 2008 23:40:24 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [64.13.141.3]) by proxy.meer.net (8.14.2/8.14.2) with ESMTP id mAPN0ZeL080221; Tue, 25 Nov 2008 15:00:48 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id mAPMvuiH033877; Tue, 25 Nov 2008 14:57:56 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (209.249.190.254.available.above.net [209.249.190.254] (may be forged)) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id mAPMvtZU025484; Tue, 25 Nov 2008 14:57:55 -0800 (PST) (envelope-from gnn@neville-neil.com) Date: Tue, 25 Nov 2008 17:57:54 -0500 Message-ID: From: gnn@freebsd.org To: "Attilio Rao" In-Reply-To: <3bbf2fe10811230502t3cc52809i6ac91082f780b730@mail.gmail.com> References: <3bbf2fe10811230502t3cc52809i6ac91082f780b730@mail.gmail.com> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.7 Emacs/22.3 (i386-apple-darwin9.5.0) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Canit-CHI2: 0.50 X-Bayes-Prob: 0.5 (Score 0, tokens from: ) X-Spam-Score: 0.10 () [Tag at 5.00] COMBINED_FROM X-CanItPRO-Stream: default X-Canit-Stats-ID: 2401205 - 6363b7530a94 X-Scanned-By: CanIt (www . roaringpenguin . com) on 64.13.141.13 Cc: FreeBSD Arch , freebsd-performance@freebsd.org, Joseph Koshy Subject: Re: [PATCH] pmcannotate tool X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2008 23:40:24 -0000 At Sun, 23 Nov 2008 14:02:22 +0100, Attilio Rao wrote: > > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. > > The tool usage is pretty simple: > pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj > > where samples.out is a pmcstat raw output and binaryobj is the binary > object that has been profiled and is accessible for (ELF) symbols > retrieving. > The options are better described in manpages but briefly: > - a: performs analysis on the assembly rather than the C source > - h: usage and informations > - k: specify a path for the kernel in order to locate correct objects for it > - l: specify a lower boundary (in total percentage time) after which > functions will be displayed nomore. > > A typical usage of pmcannotate can be some way of kernel annotation. > For example, you can follow the steps below: > 1) Generate a pmc raw output of system samples: > # pmcstat -S ipm-unhalted-core-cycles -O samples.out > 2) Copy the samples in the kernel building dir and cd there > # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd > /usr/src/sys/i386/compile/GENERIC/ > 3) Run pmcannotate > # pmcannotate -k . samples.out kernel.debug > kernel.ann > > In the example above please note that kernel.debug has to be used in > order to produce a C annotated source. This happens because in order > to get the binary sources we rely on the "objdump -S" command which > wants binary compiled with debugging options. > If not debugging options are present assembly analynsis is still > possible, but no C-backed one will be available. > objdump is not the only one tool on which pmcannotare rely. Infact, in > order to have it working, pmcstat needs to be present too because we > need to retrieve, from the pmcstat raw output, informations about the > sampled PCs (in particular the name of the function they live within, > its start and ending addresses). As long as currently pmcstat doesn't > return those informations, a new option has been added to the tool > (-m) which can extract (from a raw pmcstat output) all pc sampled, > name of the functions and symbol bundaries they live within. > > Also please note that pmcannotate suffers of 2 limitations. > Firstly, relying on objdump to dump the C source, with heavy > optimization levels and lots of inlines the code gets difficult to > read. Secondly, in particular on x86 but I guess it is not the only > one case, the sample is always attributed to the instruction directly > following the one that was interrupted. So in a C source view some > samples may be attributed to the line below the one you're interested > in. It's also important to keep in mind that if a line is a jump > target or the start of a function the sample really belongs elsewhere. > > The patch can be found here: > http://www.freebsd.org/~attilio/pmcannotate.diff/ > > where pmcannotate/ dir contains the code and needs to go under > /usr/src/usr.sbin/ and the patch has diffs against pmcstat and > Makefile. > > This work has been developed on the behalf of Nokia with important > feedbacks and directions from Jeff Roberson. > > Testing and feedbacks (before it hits the tree) are welcome. > Hi, First of all, this is excellent work. As soon as this and some other changes in PMC hit 7.x I'll be rolling this out to all the developers I work with. I've tested this on amd64 on HEAD, and with the changes we have talked about privately (%jx vs. %x) it works quite well. Secondly, I would like to request a feature. I would like to be able to get output in a more easily parsable format so I can write some Emacs code to highlight C code with the output. I'd like something along the lines of: path:function:line:percentage Keep up the good work! Later, George