Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 17 Nov 2003 16:09:35 -0500
From:      Damian Gerow <damian@sentex.net>
To:        isp@freebsd.org
Subject:   Re: Daily/weekly/monthly output aggregation
Message-ID:  <20031117210935.GK98840@sentex.net>
In-Reply-To: <6.0.0.22.0.20031117154856.01b4eb58@pop.face2interface.com>
References:  <20031117203641.GG98840@sentex.net> <20031117204102.GI61630@complx.LF.net> <6.0.0.22.0.20031117154856.01b4eb58@pop.face2interface.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Thus spake Marty Landman (MLandman@face2interface.com) [17/11/03 15:57]:
> As a developer I'd like to throw my 2 cents in; although this stmt may come 
> as no news to anyone else imho the issue is what to parse out as 
> significant. With the underlined caveat that once you make (what's in 
> essence then) a policy decision about what system output is significant 
> enough to pass along to the admin as worthy of review the danger is in 
> everything that /isn't/ passed along.

Developer input is what I need at this point -- I have done development work
in the past, but I very quickly moved into sysadmin work.

> At least now you've got the gnawing feeling that you're behind in reading 
> the stuff; once you implement a system to decide what's worth reading 

I put 'read' in quotes, because I usually give each one a ten-second
once-over.  75% of the time, that's good enough, but I have missed more than
a couple of problems that I shouldn't have.

> you've gotten rid of that guilt pang. Should that evolve into a sense of 
> false security - well I can only speculate how many server crashes could've 
> been avoided if not for feelings of false security.

Being security-concious, this is a big concern.  Hence, my paper-napkin
draft of what needs to be done:

Everything gets stored in a SQL database, since it is the cure to any and
every computing problem that has ever been introduced.

Store a table of hostnames, and whether or not they are active.  When we run
the report generator, we can check to see if a hostname did *not* check in.
If not, we send an alert.

Each report is mailed to an address, that pipes the message to a program.
This program would break each report down into its already-labelled
sections, and store it *verbatim* in the database.  This makes looking up
past reports much, much easier.

The report generator would be run via a cron job.  The idea at this point is
to:

    - make sure all currently active servers have checked in, with the
      appropriate reports
    - detect any new servers that checked in
    - do, essentially, a diff against today and yesterday for each host
      (also do a diff against today and last week, when necessary)
    - if no changes, pring a 'Host OK' status
    - otherwise, print a line for every change.

The output of this would be one e-mail, that would be sent out however you
want it to be sent out.

I already have bigger ideas for this (i.e. paging if more that 'root/toor'
found with userid zero, paging if known hosts did not check in/unknown hosts
did check in, collision/error rate jumps too high, etc.), but I'd like to
avoid feature creap for now.

Any thoughts/suggestions/comments?



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031117210935.GK98840>