From owner-freebsd-isp Thu Apr 16 22:51:53 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id WAA29222 for freebsd-isp-outgoing; Thu, 16 Apr 1998 22:51:53 -0700 (PDT) (envelope-from owner-freebsd-isp@FreeBSD.ORG) Received: from super-g.inch.com (super-g.com [207.240.140.161]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id FAA29140 for ; Fri, 17 Apr 1998 05:51:42 GMT (envelope-from spork@super-g.com) Received: from localhost (localhost [127.0.0.1]) by super-g.inch.com (8.8.8/8.8.5) with SMTP id BAA01919; Fri, 17 Apr 1998 01:50:06 -0400 (EDT) Date: Fri, 17 Apr 1998 01:50:06 -0400 (EDT) From: spork X-Sender: spork@super-g.inch.com To: "Jan L. Peterson" cc: Kevin Day , isp@FreeBSD.ORG Subject: Re: log to st0? In-Reply-To: <199804151618.KAA29846@loa.part.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org We're running into problems with archiving hits from some of the larger sites we host. We have been toying with the idea of a "log server" to collect and analyze the logs. Any suggestions? Thanks, Charles On Wed, 15 Apr 1998, Jan L. Peterson wrote: > > We're producing about 100M of http logs per day, append only... Is it at all > > possible to stream our httpd logs to tape, yet be able to rewind, read it > > all at random points, and pick up writing where I left off? > > When I was at iMALL, we were taking about 200MB of http logs per day. > We had an interesting system of batching the logs out to a log > processing machine that was separate from the front-line web servers > (four of them). The batching system was designed to be robust in the > event of full disks and/or network failures between the front-line and > the log processor. > > We found that a 200MB log file would compress down to about 20MB, and > by having a decent sized (4GB) disk available, we could keep 45 days > worth of logs on line fairly easily (took about a gig). We would also > expire these daily logs out to an archive directory when they were more > than 45 days old, and when the archive directory had about 650MB of > stuff in it, we would burn it onto a CD. > > Advantages were that log processing was fairly robust and logs were > kept in a way that we could grab a specific day's logs any time we > wanted to (if it wasn't in the active log directory that our log > summarizing system read, we could always find it either in the archive > directory or on a CD somewhere). > > -jan- > -- > Jan L. Peterson PartNET tel. +1 801 581 1118 > Senior Systems Admin 423 Wakara Way, Suite 216 fax +1 801 581 1785 > jlp@part.net Salt Lake City, UT 84108 http://www.part.net/ > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-isp" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message