Date: Thu, 16 Feb 2006 10:07:10 -0500 From: Nathan Vidican <nvidican@wmptl.com> To: Ashley Moran <ashley.moran@codeweavers.net> Cc: questions@freebsd.org Subject: Re: Log analysis server suggestions? Message-ID: <43F4951E.5090203@wmptl.com> In-Reply-To: <200602161418.32982.ashley.moran@codeweavers.net> References: <200602161418.32982.ashley.moran@codeweavers.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Ashley Moran wrote: > Until recently I had a server running syslog-ng set to archive all logs into > server/year/month/day/ directories. Now the server is running in amd64, > we've lost our hi-res scrolling display so I want to look at a better log > watching system. > > I've read about logging to a database. I quite like the idea of storing our > logs in PostgreSQL (I don't like MySQL and don't want to get involved in > administering a second database). I know I can log to a PG database quite > easily, but I don't know how I can get the data back out without writing > manual queries. > > Here is what I need: > > - Logs stored for the last 6 months or so, and easily searchable > - Live log watching > - Log analysis > > I might try swatch for the live log watching as this is not affected by the > choice of log storage and seems the best tool for the job. > > As for searching / analysis, I've seen php-syslog-ng > ( http://www.vermeer.org/projects/php-syslog-ng ), which looks very basic, > and phpLogCon ( http://www.phplogcon.com/ ), which does not support PG > anyway. Is there anything better GUI-wise? > > Maybe I am best keeping the logs in text files for now, and spending more time > on swatch. > > Any thoughts? > > Cheers > Ashley > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > > In my experience, logfiles are best NOT in SQL, flat-files are easy to deal with, and through a few simple Perl scripts you could accomplish all you need to. You can run a tail -f and dump output to stdout, or even pipe it to a socket and monitor remotely. Also, various programs have great open-source analysers, for specific logs, (ie Apache, sendmail, etc). I would advise against trying to log everything into SQL records, aside from the performance hit on translating log/write outputs to SQL inserts/queries then having the SQL server write to disk anyway, it just complicates things uneccessarily. My advice would be to take a step back and look at what's important to you. Decide which logs need to be monitored in real-time, are there certain criteria that require immediate attention? What about alhpa-numeric pager systems, or emailed warnings? Are customers going to require reports/information (ie web server stats, sendmail relay stats or spam logs, bandwidth usage, etc). It of course depends on your overall system, and your users more than anything... but in the end I find it's best to work with a mixture of things and hack your own scripts to fill in the gaps. Just my two cents, hope it helps. -- Nathan Vidican nvidican@wmptl.com Windsor Match Plate & Tool Ltd. http://www.wmptl.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43F4951E.5090203>