From owner-freebsd-questions@FreeBSD.ORG Thu Feb 16 15:05:51 2006 Return-Path: X-Original-To: questions@freebsd.org Delivered-To: freebsd-questions@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4971F16A420 for ; Thu, 16 Feb 2006 15:05:51 +0000 (GMT) (envelope-from nvidican@wmptl.com) Received: from wmptl.net (mail.wmptl.com [216.8.159.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id EF1E543D58 for ; Thu, 16 Feb 2006 15:05:38 +0000 (GMT) (envelope-from nvidican@wmptl.com) Received: from [10.0.0.104] (r3140ca.wmptl.net [10.0.0.104]) by wmptl.net (8.13.4/8.13.4) with ESMTP id k1GF5Gh3084519; Thu, 16 Feb 2006 10:05:16 -0500 (EST) (envelope-from nvidican@wmptl.com) Message-ID: <43F4951E.5090203@wmptl.com> Date: Thu, 16 Feb 2006 10:07:10 -0500 From: Nathan Vidican User-Agent: Mozilla Thunderbird 1.0.6 (X11/20050716) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Ashley Moran References: <200602161418.32982.ashley.moran@codeweavers.net> In-Reply-To: <200602161418.32982.ashley.moran@codeweavers.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.54 on 10.0.0.80 Cc: questions@freebsd.org Subject: Re: Log analysis server suggestions? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2006 15:05:51 -0000 Ashley Moran wrote: > Until recently I had a server running syslog-ng set to archive all logs into > server/year/month/day/ directories. Now the server is running in amd64, > we've lost our hi-res scrolling display so I want to look at a better log > watching system. > > I've read about logging to a database. I quite like the idea of storing our > logs in PostgreSQL (I don't like MySQL and don't want to get involved in > administering a second database). I know I can log to a PG database quite > easily, but I don't know how I can get the data back out without writing > manual queries. > > Here is what I need: > > - Logs stored for the last 6 months or so, and easily searchable > - Live log watching > - Log analysis > > I might try swatch for the live log watching as this is not affected by the > choice of log storage and seems the best tool for the job. > > As for searching / analysis, I've seen php-syslog-ng > ( http://www.vermeer.org/projects/php-syslog-ng ), which looks very basic, > and phpLogCon ( http://www.phplogcon.com/ ), which does not support PG > anyway. Is there anything better GUI-wise? > > Maybe I am best keeping the logs in text files for now, and spending more time > on swatch. > > Any thoughts? > > Cheers > Ashley > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > > In my experience, logfiles are best NOT in SQL, flat-files are easy to deal with, and through a few simple Perl scripts you could accomplish all you need to. You can run a tail -f and dump output to stdout, or even pipe it to a socket and monitor remotely. Also, various programs have great open-source analysers, for specific logs, (ie Apache, sendmail, etc). I would advise against trying to log everything into SQL records, aside from the performance hit on translating log/write outputs to SQL inserts/queries then having the SQL server write to disk anyway, it just complicates things uneccessarily. My advice would be to take a step back and look at what's important to you. Decide which logs need to be monitored in real-time, are there certain criteria that require immediate attention? What about alhpa-numeric pager systems, or emailed warnings? Are customers going to require reports/information (ie web server stats, sendmail relay stats or spam logs, bandwidth usage, etc). It of course depends on your overall system, and your users more than anything... but in the end I find it's best to work with a mixture of things and hack your own scripts to fill in the gaps. Just my two cents, hope it helps. -- Nathan Vidican nvidican@wmptl.com Windsor Match Plate & Tool Ltd. http://www.wmptl.com/