Date: Sat, 15 Sep 2001 16:41:41 +1000 From: Enno Davids <nconedd@webjump.national.com.au> To: Scott Lambert <lambert@lambertfam.org> Cc: freebsd-isp@FreeBSD.ORG Subject: Re: Web Server clustering Message-ID: <20010915164140.C28433@webjump.national.com.au> In-Reply-To: <20010914180334.A8148@laptop.lambertfam.org>; from lambert@lambertfam.org on Fri, Sep 14, 2001 at 06:03:34PM -0500 References: <20010914142134.A65BA37B408@hub.freebsd.org> <20010914180334.A8148@laptop.lambertfam.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Sep 14, 2001 at 06:03:34PM -0500, Scott Lambert wrote:
|On Fri, Sep 14, 2001 at 07:21:34AM -0700, brian.jackson@third-rail.net wrote:
|> Hi -
|>
|> We have a cluster of Apache web servers which are "balanced" using
|> round-robin DNS.
|>
|
|<snip losing syslog data due to less than optimal rsync script>
|
|> I could copy all of the logs to the primary server, and concatinate
|> them together before each sync. of the cluster, and then not copy that
|> directory to the other servers.....
|
|The first thing that jumps to my mind would be to set the secondary servers
|to log to the syslog daemon on your primary server.
I'm guessing we're talking Apache logs here rather than syslog logs.
There is a module for Apache called mod_spread I think which claims to be
able to do logging across a farm of servers to a central log repository. When
last I looked the licensing on its underlying transport prevented me from
using it though.
Here, we've taken the KISS approach. Content is uploaded to a master content
server (which isn't in the prod server pool) and the users log into a PHP
page where they do a PUSH operation to each of the prod servers at once using
rsync. Each server's logs are kept in a separate filesystem anyway (Apache has
no write permission in the content trees, only rsync does) and are amalgamated
in a separate process shortly after the log rotation happens in the wee small
hours. We use a custom log format with %{%Y%m%d%H%M%S}t at the front of it,
which gives us a field sort(1) can use to sort the entries, then its piped
through cut to remove the field, logresolve to change the IP addresses back
to names and finally a custom program to split out separate files for each
days worth of logging. If your logs are small, you can actually use grep in
the pipe of commands to extract the relevant days worth of output back before
you strip the extra field (which must be done as logresolve expects the first
field to be the IP address not the custom date/time field).
I'd be interested to hear other approaches though as our log files are now
getting quite big (> 1Gb on a busy day now) and juggling all that data can
be hard...
Enno.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010915164140.C28433>