From owner-freebsd-hackers@FreeBSD.ORG Fri Aug 3 19:57:53 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 94EA016A418 for ; Fri, 3 Aug 2007 19:57:53 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.238]) by mx1.freebsd.org (Postfix) with ESMTP id 6378913C47E for ; Fri, 3 Aug 2007 19:57:53 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so373185nzf for ; Fri, 03 Aug 2007 12:57:52 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=XOhBatTZHc+316begX2udUdAGDmbaAxonNwqcy86KOAzhaWuUMeVd/MgG22PubVmWzyJaqbt9d1LHiP2NSYHvcODS8gmiCpqqoSsYd0GaPabYiPmssPp3abPrCg3Z4Eq7SmN9/RfDGOohW0GLGR5/oK46MA1yy3D8skmCJyJfC4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=RV6VIsLCxysZEc5NO3eujtol4flBp0xfGyXYR76GPk/jgQofoNVmfjqZOnn+h5x9M+xbiuGRQ0d3nAe0KepYmSMMHT0R920izmLoFhP0NYcXWWNOeCYB06RTWxNJK8gSM6l7oD4cIf9lEnPk+da3Yg9xeA23AnfKmqtZ2R8g1X0= Received: by 10.142.240.9 with SMTP id n9mr147310wfh.1186169504285; Fri, 03 Aug 2007 12:31:44 -0700 (PDT) Received: by 10.143.158.6 with HTTP; Fri, 3 Aug 2007 12:31:44 -0700 (PDT) Message-ID: <84dead720708031231n45533e41uba5ffa6cbe252c1c@mail.gmail.com> Date: Fri, 3 Aug 2007 19:31:44 +0000 From: "Joseph Koshy" To: "freebsd-current@freebsd.org" , "FreeBSD Hackers" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: [CFT] Callchain capture in PmcTools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2007 19:57:53 -0000 I'm pleased to offer a patch implementing callchain capture for hwpmc(4) for review. Test reports, comments etc. are welcome. Regards, Koshy ============================================= >>>> Summary <<<< hwpmc(4): * hwpmc(4) can now walk kernel and user stacks and capture caller information for subsequent analysis by pmcstat(8). * An additional flag in the API `PMC_F_CALLCHAIN' controls whether a PMC will capture a callchain. * Tunable kern.hwpmc.callchaindepth controls the maximum depth to which stacks are walked. The default is `8'. pmcstat(8): * pmcstat(8) now defaults to allocating PMCs that capture callchains. Use the new `-N' option (a toggle) to turn this off. * The '-g' option (gmon.out generation) now writes call arc data for subsequent analysis by gprof(1). * The new '-G' option generates system-wide callchain summaries. The new '-z maxprintdepth' option restricts the depth of the the callchain summary. >>>> Patch Information <<<< 1) Download the patch % fetch \ http://people.freebsd.org/~jkoshy/download/pmctools-callgraph-patch.gz MD5 (pmctools-callgraph-patch.gz) = 632185755d1004e82c3d2bbc69827307 2) Apply using patch -p1 against a recent (Aug 3rd) -current: % cvs checkout -P src % cd src; gzip -dc pmctools-callgraph-patch.gz | patch -p1 3) Build and update the kernel+world. You will need to add 'options HWPMC_HOOKS' to your kernel config before you can use hwpmc(4). % (follow the usual procedure, see src/UPDATING) >>>> Using the patch <<<< 1) Load hwpmc into the kernel % kldload hwpmc 2) Collect measurements % pmcstat -S instructions -O logfile etc. 3) Use option -g to generate gprof(1) style 'gmon.out' files. % pmcstat -R logfile -g % gprof /boot/kernel/kernel /kernel.gmon 4) Use option -G to generate a callchain summary: % pmcstat -R logfile -G summaryfile % vi summaryfile >>>> Known Bugs with this patch <<<< 1) P4 HTT lockup Symptom: Lockup under load of Pentium 4 machines with HTT enabled. Workaround: Restrict sampling to one CPU using the '-c' option: # pmcstat -c 0 -S instructions -O logfile 2) pmcstat(8) stuck in an unkillable state. Symptom: When interrupted, pmcstat(8) gets stuck sleeping on wait channel "pmcctx". Workaround: Use the '-n' option to reduce sampling frequency: # pmcstat -n 1048576 -S instructions ...other options... Other (older) known bugs are listed at http://wiki.freebsd.org/PmcTools. >>>> Other Notes <<<< pmcstat(8) works best with unstripped executables (e.g. set "STRIP=" in /etc/make.conf). On the amd64 the heuristic used to determine the frame pointer given a sampled PC address is not very good and can at times result in the next to topmost frame being missed in the sampled callchain. >>>> Thanks <<<< - To the users of PmcTools in the FreeBSD community for their feedback, encouragement and support. - To the FreeBSD Foundation and Google Inc., for supporting this work. =============================================