From owner-freebsd-isp Wed Apr 15 09:19:15 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id JAA29831 for freebsd-isp-outgoing; Wed, 15 Apr 1998 09:19:15 -0700 (PDT) (envelope-from owner-freebsd-isp@FreeBSD.ORG) Received: from loa.part.net ([155.99.143.146]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id QAA29824 for ; Wed, 15 Apr 1998 16:19:08 GMT (envelope-from jlp@Part.NET) Received: from loa.part.net (localhost [127.0.0.1]) by loa.part.net (8.8.5/8.8.5) with ESMTP id KAA29846; Wed, 15 Apr 1998 10:18:02 -0600 (MDT) Message-Id: <199804151618.KAA29846@loa.part.net> X-Mailer: exmh version 2.0.2 2/24/98 To: Kevin Day cc: isp@FreeBSD.ORG Subject: Re: log to st0? X-face: p=61=y<.Il$z+k*y~"j>%c[8R~8{j3WTnaSd-'RyC>t.Ub>AAm\zYA#5JF +W=G?EI+|EI);]=fs_MOfKN0n9`OlmB[1^0;L^64K5][nOb&gv/n}p@mm06|J|WNa asp7mMEw0w)e_6T~7v-\]yHKvI^1}[2k)] References: <199804150533.AAA11780@home.dragondata.com> In-reply-to: Your message of "Wed, 15 Apr 1998 00:33:15 CDT." <199804150533.AAA11780@home.dragondata.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 15 Apr 1998 10:18:02 -0600 From: "Jan L. Peterson" Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > We're producing about 100M of http logs per day, append only... Is it at all > possible to stream our httpd logs to tape, yet be able to rewind, read it > all at random points, and pick up writing where I left off? When I was at iMALL, we were taking about 200MB of http logs per day. We had an interesting system of batching the logs out to a log processing machine that was separate from the front-line web servers (four of them). The batching system was designed to be robust in the event of full disks and/or network failures between the front-line and the log processor. We found that a 200MB log file would compress down to about 20MB, and by having a decent sized (4GB) disk available, we could keep 45 days worth of logs on line fairly easily (took about a gig). We would also expire these daily logs out to an archive directory when they were more than 45 days old, and when the archive directory had about 650MB of stuff in it, we would burn it onto a CD. Advantages were that log processing was fairly robust and logs were kept in a way that we could grab a specific day's logs any time we wanted to (if it wasn't in the active log directory that our log summarizing system read, we could always find it either in the archive directory or on a CD somewhere). -jan- -- Jan L. Peterson PartNET tel. +1 801 581 1118 Senior Systems Admin 423 Wakara Way, Suite 216 fax +1 801 581 1785 jlp@part.net Salt Lake City, UT 84108 http://www.part.net/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message