From owner-freebsd-isp Fri Sep 14 23:41:59 2001 Delivered-To: freebsd-isp@freebsd.org Received: from orange.national.com.au (orange.national.com.au [203.57.240.81]) by hub.freebsd.org (Postfix) with ESMTP id 0391237B406 for ; Fri, 14 Sep 2001 23:41:55 -0700 (PDT) Received: by orange.national.com.au (Postfix, from userid 5) id 23F2D4E68A; Sat, 15 Sep 2001 16:41:53 +1000 (EST) Received: from peppermint.national.com.au(10.25.154.32) by orange.national.com.au via csmap (V4.1) id srcAAAR5aGqE; Sat, 15 Sep 01 16:41:52 +1000 Received: (from uucp@localhost) by peppermint.national.com.au (8.9.3+Sun/8.8.8) id QAA24508; Sat, 15 Sep 2001 16:41:51 +1000 (EST) Received: from UNKNOWN(164.53.27.38), claiming to be "webjump.national.com.au" via SMTP by peppermint, id smtpdAAA35aq3V; Sat Sep 15 16:41:48 2001 Received: (from nconedd@localhost) by webjump.national.com.au (8.10.2+Sun/8.10.2) id f8F6ffg13236; Sat, 15 Sep 2001 16:41:41 +1000 (EST) Date: Sat, 15 Sep 2001 16:41:41 +1000 From: Enno Davids To: Scott Lambert Cc: freebsd-isp@FreeBSD.ORG Subject: Re: Web Server clustering Message-ID: <20010915164140.C28433@webjump.national.com.au> Reply-To: enno.davids@metva.com.au References: <20010914142134.A65BA37B408@hub.freebsd.org> <20010914180334.A8148@laptop.lambertfam.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010914180334.A8148@laptop.lambertfam.org>; from lambert@lambertfam.org on Fri, Sep 14, 2001 at 06:03:34PM -0500 Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Sep 14, 2001 at 06:03:34PM -0500, Scott Lambert wrote: |On Fri, Sep 14, 2001 at 07:21:34AM -0700, brian.jackson@third-rail.net wrote: |> Hi - |> |> We have a cluster of Apache web servers which are "balanced" using |> round-robin DNS. |> | | | |> I could copy all of the logs to the primary server, and concatinate |> them together before each sync. of the cluster, and then not copy that |> directory to the other servers..... | |The first thing that jumps to my mind would be to set the secondary servers |to log to the syslog daemon on your primary server. I'm guessing we're talking Apache logs here rather than syslog logs. There is a module for Apache called mod_spread I think which claims to be able to do logging across a farm of servers to a central log repository. When last I looked the licensing on its underlying transport prevented me from using it though. Here, we've taken the KISS approach. Content is uploaded to a master content server (which isn't in the prod server pool) and the users log into a PHP page where they do a PUSH operation to each of the prod servers at once using rsync. Each server's logs are kept in a separate filesystem anyway (Apache has no write permission in the content trees, only rsync does) and are amalgamated in a separate process shortly after the log rotation happens in the wee small hours. We use a custom log format with %{%Y%m%d%H%M%S}t at the front of it, which gives us a field sort(1) can use to sort the entries, then its piped through cut to remove the field, logresolve to change the IP addresses back to names and finally a custom program to split out separate files for each days worth of logging. If your logs are small, you can actually use grep in the pipe of commands to extract the relevant days worth of output back before you strip the extra field (which must be done as logresolve expects the first field to be the IP address not the custom date/time field). I'd be interested to hear other approaches though as our log files are now getting quite big (> 1Gb on a busy day now) and juggling all that data can be hard... Enno. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message