Skip site navigation (1)Skip section navigation (2) 
Date:      Sat, 8 Jun 2002 00:34:56 -0400
From:      Amit Rao <arao@niksun.com>
To:        John Baldwin <jhb@FreeBSD.org>
Cc:        hackers@FreeBSD.org, Lars Eggert <larse@ISI.EDU>
Subject:   Re: Project: a benchmark utility
Message-ID:  <200206080434.AAA60442@arjun.niksun.com>
In-Reply-To: <bulk.91954.20020607154912@hub.freebsd.org>
References:  <bulk.91954.20020607154912@hub.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help 
On Friday 07 June 2002 06:49 pm, John Baldwin <jhb@FreeBSD.org>  wrote:
> Actually, if there's a Perl/Tcl/Python/C/C++/shell hacker running around I
> could use a decent benchmarking tool to compare stable and current.
>  Basically, what I would like is to be able to do the following:
>
> bench -n <number of trials> <command to run>
>
> So for example:
>
> bench -n 20 buildworld -j4
>
> To run my buildworld script 20 times (with -j4 passed in as an argument to
> buildworld).  I would like the program/script/whatever to collect time -l
> stats for each iteration.  It can simply spit the time -l output to a
> simple text file in a sensible format (one line per run, with a specific
> order of columns for example, just the numbers to make the file easier to
> parse).
>
> Once I have that, it would be nice to have a simple tool that would take
> one of these tabular files as input and spit out appropriate statistics
> about each column (mean, mode, median, stddev, highlight outliers, etc.).
>  If some sensible (i.e. meaningful) graphs can be generated from this data
> using gnuplot or some such that would be nice, too.  Any takers?

I'll bite. 
try the  following program:

---------------------------------
#!/usr/bin/env perl

# bench.pl

# benchmarking tool to collect /usr/bin/time -l stats for each
# iteration of the specified program. The output is in a form suitable
# for further processing by John Heidemann's JDB
# (http://www.isi.edu/~johnh/SOFTWARE/JDB/index.html)

#  Eg:  bench.pl -n 20 "buildworld -j4"

# Copyright 2002 Amit K. Rao
# arao@niksun.com

use strict;
use Getopt::Std;

sub usage {
    print <<END;
usage: $0 [-n <num_trials>] <command_to_run>
      Default no. of trials is 10.
      Eg:  $0 -n 20 "buildworld -j4"
END
  exit 1;
}

sub getrusage() {
  my $CMD = shift;
  my $TLOG;
  chomp($TLOG=`mktemp /usr/tmp/time.XXXXXX`);
  system "/usr/bin/time -l -o $TLOG $CMD 2>&1 > /dev/null";
  if (-e $TLOG) {
    my $out = `awk '{if (\$2==\"real\") {printf \"%s %s %s\", \$1, \$3, \$5} 
else { printf \" %s\", \$1}} END {printf \"\\n\"} ' $TLOG`;
    return $out;
    unlink($TLOG);
  }
  else {
    print "Error running command /usr/bin/time -l -o $TLOG $CMD 2>&1 > 
/dev/null\n";
  }
}


my $ntrials = 10;
my $cmd;
my %opts;

getopts('n:',\%opts);

$ntrials = $opts{'n'} if (defined $opts{'n'});
$cmd = join(" ",@ARGV);
if (!defined ($cmd)) {
  &usage();
}

print "#h real user sys ru_maxrss ru_ixrss ru_idrss ru_isrss ru_minflt 
ru_majflt ru_nswap ru_inblock ru_oublock ru_msgsnd ru_msgrcv ru_nsignals 
ru_nvcsw ru_nivcsw\n";

for (my $i = 0; $i < $ntrials; $i++) {
  print &getrusage("$cmd");
}

print "# $cmd\n"
-----------------------------------------

Eg: 
$ bench.pl tar tzvf somearchive.tgz 
#h real user sys ru_maxrss ru_ixrss ru_idrss ru_isrss ru_minflt ru_majflt 
ru_nswap ru_inblock ru_oublock ru_msgsnd ru_msgrcv ru_nsignals ru_nvcsw 
ru_nivcsw
0.72 0.63 0.09 576 92 365 126 130 0 0 0 0 0 0 0 1800 907
0.72 0.64 0.08 576 94 362 128 130 0 0 0 0 0 0 0 1801 906
0.72 0.62 0.10 576 96 359 129 130 0 0 0 0 0 0 0 1801 907
0.72 0.69 0.03 576 97 348 129 130 0 0 0 0 0 0 0 1801 908
0.72 0.65 0.08 576 96 367 132 130 0 0 0 0 0 0 0 1801 909
0.72 0.65 0.07 576 95 363 129 130 0 0 0 0 0 0 0 1800 907
0.72 0.67 0.05 576 96 370 130 130 0 0 0 0 0 0 0 1801 906
0.73 0.67 0.07 576 93 355 126 130 0 0 0 0 0 0 0 1801 907
0.72 0.65 0.07 576 94 358 128 130 0 0 0 0 0 0 0 1801 908
0.72 0.62 0.10 576 96 370 130 130 0 0 0 0 0 0 0 1764 909
# bench.pl tar tzvf somearchive.tgz

---
You can then run this output through any of the programs in JDB mentioned by 
Lars. Eg to get mean, std. dev, etc. over user time (the second column): 

$ bench.pl tar tzvf somearchive.tgz | dbstats user | dblistize
#L mean stddev pct_rsd conf_range conf_low conf_high conf_pct sum sum_squared 
min
max n
mean:        0.648
stddev:      0.026162
pct_rsd:     4.0373
conf_range:  0.018714
conf_low:    0.62929
conf_high:   0.66671
conf_pct:    0.95
sum:         6.48
sum_squared: 4.2052
min:         0.6
max:         0.7
n:           10

# bench.pl tar tzvf somearchive.tgz
#  | dbstats user
#               0.95 confidence intervals assume normal distribution and 
small n.
#  | dblistize


--------

-Amit

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200206080434.AAA60442>