Date: Tue, 11 Sep 2007 13:43:13 +0200 From: Borja Marcos <borjamar@sarenet.es> To: freebsd-performance@freebsd.org Subject: Announcement: Devilator 1.0a for FreeBSD Message-ID: <CAD4B228-9B25-4D7D-9982-1701746D978D@sarenet.es>
next in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello,
I'm happy to announce the first public release (consider it an alpha)
of Devilator, a performance
data collector for FreeBSD systems that generates Orca compatible
files. The README file follows.
************************************************************************
*******
WARNING!!!!
This is the first public evaluation release of a program developed
for internal
use. Please exercise extreme care when using it, and report any
issues to the
author to have them fixed in future releases.
************************************************************************
*******
==================================
DEVILATOR
==================================
"Why does my system run slowly?" "What can I do to make it run
faster?" "The
CPU seems to be idle. Why don't I get better performance?" "Will a
bigger CPU
make it run faster?" "Will I need a better CPU next year?" "Was the
system
slower last Thursday, or is it just my imagination?" These questions
are made probably hundreds or perhaps even thousands of times every
day, and the answer
is not always easy.
A system's performance is a function of its CPU power, bus bandwidth,
memory capacity, disk I/O bandwidth (often neglected) and software.
Nowadays,
with the usage of concurrent programs growing, software performance
problems
get increasingly important as well. We can have the fastest and
greatest hardware available, but a poor locking decision in a
concurrent program
can make it crawl, no matter what we do. A bad mix of workloads can
cause
these problems as well.
Performance tuning is often a black art, and historical data is reall
valuable, a need that the classical commands such as top, vmstat,
gstat, etc
don't address adequately (in my opinion).
Devilator is a system performance data collector for FreeBSD. It's
designed
to work with Blair Zajac's Orca system (http://www.orcaware.com/
orca/) and
it aims to be a FreeBSD equivalent of the "Orcallator" data collector
for
Solaris.
This program has been growing for a long time, supplying performance
data for
our in-house systems. As this program has been written as a series of
quick
hacks, it is actually quite ugly, but effective anyway. The main goal
when
writing it was to gather enough performance data from the OS, always
avoiding
typical scripts that launch many other utilities. This program reads
OS data
directly, either through the sysctl interface or from GEOM (in the
case of
disk I/O performance data). As the documentation on some of these
issues is
not great, I have borrowed code from the GEOM utility "gstat", from
"top",
and "orcallator.se", part of the Orca package.
The data collected by Devilator can be divided among the following
groups:
1- Processor/s usage. This includes the classical "system load"
values, and the percentage of CPU time spent on user processes,
system, interrupt and nice.
2- OS activity. Includes a graph of forks/s, system calls/s, system
processes and interrupt activity. The system processes and interrupt
information show
where the system is spending its share of CPU time. The system
processes shown
vary from version to version, with more information offered by the
most recent
FreeBSD releases. Hopefully this can be useful for system
administrators and
system developers alike, keeping historic data on the activity of tasks
like "vmdaemon", "syncer", etc.
3- Process activity. Includes the number of processes in the system,
how many
of them are in a given state (running, sleeping, stopped, zombies,
waiting or
locked), and a graph of sleep situations. How many processes are
sleeping on
select(), waiting for network I/O, waiting for the disk subsystem,
filesystem,
etc? This is really useful when evaluating the performance of
programs such
as MySQL, trying different filesystem configurations or threading
models. These
data will be hopefully useful to FreeBSD developers and porters as well.
4- Network activity, offering the classical information on
established connections, TCP and UDP traffic, network interface
traffic and errors, TCP
connection rate, etc.
5- Virtual memory, including swap and paging activity, memory usage,
and page
scanner activity.
6- Disk I/O data (available on FreeBSD 5+, as it depends on GEOM),
including
the percentage of busy time for disks, partitions and slices, together
with the volume of data transfers in MB/s and the average service
time in ms.
- ---------------
USING DEVILATOR
- ---------------
In order to use this program it's a good idea to be familiarised with
Orca
first. Orca (http://www.orcaware.com/orca/) is a general purpose
graphing
program based on RRDtool.
This program can be used to monitor a single machine, running both
Devilator
and Orca on it, or it can be installed at dozens of servers, where it
will just
gather performance data and write it to text files, while a different
server
downloads those performance data files periodically, running Orca and
generating the HTML files and RRD graphs for all of them.
Depending on the number of machines, a single machine can monitor
dozens of
servers, and the impact of Devilator is negligible. It just wakes up at
5 minute intervals, reads some OS statistics using sysctl(3) and
GEOM, and
writes a line to a text file.
In order to have Orca use the data provided by Devilator, the supplied
Orca configuration file, devilator.cfg, must be used. The relevant
variables to set up are:
- --------------
# base_dir is the directory where the RRD files will be stored.
base_dir /var/orca/rrd/orcallator
# html_dir is the directory where the web pages (HTML files and PNG
graphs)
# will be stored. This directory will be made available with a suitable
# web server such as Apache.
html_dir /var/www/orca
# find_files specifies the directory under which the text files with
# performance data reside. It is possible to monitor a set of
machines with a
# single Orca instance, with each machine having its own directory under
# /var/orca/orcallator, such as /var/orca/orcallator/server1,
# /var/orca/orcallator/server2...
find_files /var/orca/orcallator/(.*)/(?:(?:orcallator)|
(?:percol))-
\d{4}-\d{2}-\d{2}(?:-\d{3,})?(?:\.(?:Z|gz|bz2))?
- ---------------
Please read the Orca documentation and become familiar with it.
- -------------------
FUTURE WORK
- -------------------
The graphs are a bit messy right now, although they are easy to
customize
editing the .cfg file. This program generates a lot of information
that might
not be necessary for everyone. Unnecessary graphs can be turned off
just by
commenting out the relevant sections in the configuration file.
Some enhancements would probably be useful. Tracking resources usage by
process name would allow the administrator to know which CGI scripts are
using more resources, etc.
I will be happy to hear about other interesting datasets. Simple sysctl
fetches are trivial to do. But obviously it would be insane to include
all the available data.
- --------------------
PORTABILITY
- --------------------
This program has been developed under FreeBSD, and some of the bits
(like its reliance on GEOM to gather disk I/O statistics) are pretty
FreeBSD-specific.
So far I have tested it and used it on FreeBSD 4.9 (x86), FreeBSD 6.x on
sparc64 and x86, and FreeBSD 7 on amd64.
- --------------------
SUPPORT
- --------------------
This program is supplied as is, and there's absolutely no guarantee
of any kind, including suitability for any particular purpose.
There's no
guarantee of technical support either.
This program started its life as an internal development, and I'm
polishing
it for public distribution. Most of the development time has been
payed for
by my employer, Sarenet, and it's being released as a contribution to
the
FreeBSD community.
Please send bugs, ideas, flames, etc to the following address:
Borja Marcos.
Sarenet S.A.
borjamar@sarenet.es
borjam@gmail.com (Preferred for Devilator issues)
Devilator can be downloaded from:
http://homepage.mac.com/borjam/.Public/devilator-1.0a.tar.gz
MD5 (devilator-1.0a.tar.gz) = 5868761c2ce0f8dc3ff51167fd06400a
And the PGP signature for the file follows:
- -----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.3 (Build 5003)
qANQR1DEDQMAAhGlPtFtULJLjAHCPwMFAEbmbwOlPtFtULJLjBECYlEAnj1HejG2
rBJVYur8VvZ51MlSYVX1AJ9Aak05xRRkQ/BKn8Asl9nWsxD6rA==
=rKMg
- -----END PGP SIGNATURE-----
- ----------------
"The thing he realised about the windows was this: because they had
been converted into openable windows after they had first been
designed to be impregnable, they were, in fact, much less secure than
if they had been designed as openable windows in the first place."
Douglas Adams, "Mostly Harmless"
-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.3 (Build 5003)
wj8DBQFG5nfqpT7RbVCyS4wRAr7eAJ98PUckfBpE6nH+ouU9hpN/FhT0LACg03UH
ETLJnR4KJFgk02lNuTZanW4=
=ULQE
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD4B228-9B25-4D7D-9982-1701746D978D>