From owner-freebsd-isp@FreeBSD.ORG Mon Nov 17 13:09:53 2003 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2470816A4CE for ; Mon, 17 Nov 2003 13:09:53 -0800 (PST) Received: from smtp3.sentex.ca (smtp3.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 595FC43FBD for ; Mon, 17 Nov 2003 13:09:49 -0800 (PST) (envelope-from damian@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smtp3.sentex.ca (8.12.9p2/8.12.9) with ESMTP id hAHL9ix3066026 for ; Mon, 17 Nov 2003 16:09:44 -0500 (EST) (envelope-from damian@sentex.net) Received: from pegmatite.sentex.ca (pegmatite.sentex.ca [192.168.42.92]) by lava.sentex.ca (8.12.9p2/8.12.9) with ESMTP id hAHL9mUq001284 for ; Mon, 17 Nov 2003 16:09:48 -0500 (EST) (envelope-from damian@sentex.net) Received: by pegmatite.sentex.ca (Postfix, from userid 1001) id 5726D171D3; Mon, 17 Nov 2003 16:09:35 -0500 (EST) Date: Mon, 17 Nov 2003 16:09:35 -0500 From: Damian Gerow To: isp@freebsd.org Message-ID: <20031117210935.GK98840@sentex.net> Mail-Followup-To: isp@freebsd.org References: <20031117203641.GG98840@sentex.net> <20031117204102.GI61630@complx.LF.net> <6.0.0.22.0.20031117154856.01b4eb58@pop.face2interface.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6.0.0.22.0.20031117154856.01b4eb58@pop.face2interface.com> X-GPG-Key-Id: 0xB841F142 X-GPG-Fingerprint: C7C1 E1D1 EC06 7C86 AF7C 57E6 173D 9CF6 B841 F142 X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . User-Agent: Mutt/1.5.4i X-Virus-Scanned: by amavisd-new Subject: Re: Daily/weekly/monthly output aggregation X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2003 21:09:53 -0000 Thus spake Marty Landman (MLandman@face2interface.com) [17/11/03 15:57]: > As a developer I'd like to throw my 2 cents in; although this stmt may come > as no news to anyone else imho the issue is what to parse out as > significant. With the underlined caveat that once you make (what's in > essence then) a policy decision about what system output is significant > enough to pass along to the admin as worthy of review the danger is in > everything that /isn't/ passed along. Developer input is what I need at this point -- I have done development work in the past, but I very quickly moved into sysadmin work. > At least now you've got the gnawing feeling that you're behind in reading > the stuff; once you implement a system to decide what's worth reading I put 'read' in quotes, because I usually give each one a ten-second once-over. 75% of the time, that's good enough, but I have missed more than a couple of problems that I shouldn't have. > you've gotten rid of that guilt pang. Should that evolve into a sense of > false security - well I can only speculate how many server crashes could've > been avoided if not for feelings of false security. Being security-concious, this is a big concern. Hence, my paper-napkin draft of what needs to be done: Everything gets stored in a SQL database, since it is the cure to any and every computing problem that has ever been introduced. Store a table of hostnames, and whether or not they are active. When we run the report generator, we can check to see if a hostname did *not* check in. If not, we send an alert. Each report is mailed to an address, that pipes the message to a program. This program would break each report down into its already-labelled sections, and store it *verbatim* in the database. This makes looking up past reports much, much easier. The report generator would be run via a cron job. The idea at this point is to: - make sure all currently active servers have checked in, with the appropriate reports - detect any new servers that checked in - do, essentially, a diff against today and yesterday for each host (also do a diff against today and last week, when necessary) - if no changes, pring a 'Host OK' status - otherwise, print a line for every change. The output of this would be one e-mail, that would be sent out however you want it to be sent out. I already have bigger ideas for this (i.e. paging if more that 'root/toor' found with userid zero, paging if known hosts did not check in/unknown hosts did check in, collision/error rate jumps too high, etc.), but I'd like to avoid feature creap for now. Any thoughts/suggestions/comments?