Date: Wed, 15 Apr 1998 10:18:02 -0600 From: "Jan L. Peterson" <jlp@Part.NET> To: Kevin Day <toasty@home.dragondata.com> Cc: isp@FreeBSD.ORG Subject: Re: log to st0? Message-ID: <199804151618.KAA29846@loa.part.net> In-Reply-To: Your message of "Wed, 15 Apr 1998 00:33:15 CDT." <199804150533.AAA11780@home.dragondata.com> References: <199804150533.AAA11780@home.dragondata.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> We're producing about 100M of http logs per day, append only... Is it at all > possible to stream our httpd logs to tape, yet be able to rewind, read it > all at random points, and pick up writing where I left off? When I was at iMALL, we were taking about 200MB of http logs per day. We had an interesting system of batching the logs out to a log processing machine that was separate from the front-line web servers (four of them). The batching system was designed to be robust in the event of full disks and/or network failures between the front-line and the log processor. We found that a 200MB log file would compress down to about 20MB, and by having a decent sized (4GB) disk available, we could keep 45 days worth of logs on line fairly easily (took about a gig). We would also expire these daily logs out to an archive directory when they were more than 45 days old, and when the archive directory had about 650MB of stuff in it, we would burn it onto a CD. Advantages were that log processing was fairly robust and logs were kept in a way that we could grab a specific day's logs any time we wanted to (if it wasn't in the active log directory that our log summarizing system read, we could always find it either in the archive directory or on a CD somewhere). -jan- -- Jan L. Peterson PartNET tel. +1 801 581 1118 Senior Systems Admin 423 Wakara Way, Suite 216 fax +1 801 581 1785 jlp@part.net Salt Lake City, UT 84108 http://www.part.net/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199804151618.KAA29846>