Date: Fri, 17 Apr 1998 01:50:06 -0400 (EDT) From: spork <spork@super-g.com> To: "Jan L. Peterson" <jlp@Part.NET> Cc: Kevin Day <toasty@home.dragondata.com>, isp@FreeBSD.ORG Subject: Re: log to st0? Message-ID: <Pine.BSF.3.96.980417014813.399B-100000@super-g.inch.com> In-Reply-To: <199804151618.KAA29846@loa.part.net>
next in thread | previous in thread | raw e-mail | index | archive | help
We're running into problems with archiving hits from some of the larger sites we host. We have been toying with the idea of a "log server" to collect and analyze the logs. Any suggestions? Thanks, Charles On Wed, 15 Apr 1998, Jan L. Peterson wrote: > > We're producing about 100M of http logs per day, append only... Is it at all > > possible to stream our httpd logs to tape, yet be able to rewind, read it > > all at random points, and pick up writing where I left off? > > When I was at iMALL, we were taking about 200MB of http logs per day. > We had an interesting system of batching the logs out to a log > processing machine that was separate from the front-line web servers > (four of them). The batching system was designed to be robust in the > event of full disks and/or network failures between the front-line and > the log processor. > > We found that a 200MB log file would compress down to about 20MB, and > by having a decent sized (4GB) disk available, we could keep 45 days > worth of logs on line fairly easily (took about a gig). We would also > expire these daily logs out to an archive directory when they were more > than 45 days old, and when the archive directory had about 650MB of > stuff in it, we would burn it onto a CD. > > Advantages were that log processing was fairly robust and logs were > kept in a way that we could grab a specific day's logs any time we > wanted to (if it wasn't in the active log directory that our log > summarizing system read, we could always find it either in the archive > directory or on a CD somewhere). > > -jan- > -- > Jan L. Peterson PartNET tel. +1 801 581 1118 > Senior Systems Admin 423 Wakara Way, Suite 216 fax +1 801 581 1785 > jlp@part.net Salt Lake City, UT 84108 http://www.part.net/ > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-isp" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.96.980417014813.399B-100000>