Date: Thu, 20 May 1999 15:25:49 -0500 (CDT) From: Jeff Lynch <jeff@mercury.jorsm.com> To: James Wyatt <jwyatt@RWSystems.net> Cc: Roger Marquis <marquis@roble.com>, freebsd-isp@FreeBSD.ORG Subject: Re: Web Statistics break up program. Message-ID: <Pine.BSF.3.95q.990520150142.21290C-100000@mercury.jorsm.com> In-Reply-To: <Pine.BSF.4.05.9905201115020.18069-100000@kasie.rwsystems.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 20 May 1999, James Wyatt wrote:
> On Tue, 18 May 1999, Roger Marquis wrote:
> > > > This will create an archived logfile (http.site.May_1999) and erase
> > > > the original without needing to kill -1 the httpd.
> > > >
> > > > #!/bin/sh -
> > > > LOGDIR=/var/log
> > > > ARCDIR=/var/log/oldlogs
> > > > DAY=`date | awk '{ OFS="_" ;print $2,$6}' `
> > > > for log in $LOGDIR/http* ; do
> > > > cp $log $ARCDIR/${log}.${DAY}
> > > > chmod 440 $ARCDIR/${log}.${DAY}
> > > > cp /dev/null $log
> > > > done
> > >
> > > Egads!!
> > > That's a pretty vicious race condition there, you'll lose records on busy
> > > servers.
> >
> > In theory perhaps, in reality it doesn't. I've never seen this algorithm
> > fail, even when used on log files that grow by several megabytes per day.
>
> Since you would quietly lose just a few lines once-in-a-while during a low
> traffic period, how would you *know*? The server I'm most concerned about
> handles eCommerce for transportation. It has logs of about 40-50MB/day.
>
> This looks like a noticable race condition on most platforms. I would
> like to know why it doesn't happen if it doesn't. Sounds like I have
> some playing to do some evening. - Jy@
>
Read my previous post :( If you run this daily, you are dealing with
40-50MB files. On FreeBSD, cp(1) mmaps the file with its size at the
instant determined by stat, but only if <= 8MB. There is a little room for
error due to rounding to the nearest multiple of the page size. If the
file is greater than 8MB, it goes into a while(read) loop until EOF. No
mmap, no fixed read of a specific number of bytes.
Meanwhile, apache keeps appending to the log file's open descriptor, and
while(read) keeps recieving data. As for the chown and cp /dev/null,
that's probably too fast relative to web access terms to really matter.
In summary, the race condition worsens as the file size grows up to
8MB. After passing 8MB in size, the race condition reduces to the
delay for chmod to modify the file's inode and copy a single null.
Look, I know it seems weird to have this 8MB thing happening, and it's
there to improve performance without trashing the system with extremely
large file sizes. Take a look, it's right there in
/usr/src/bin/cp/utils.c, copy_file(){}. Again, this is unchanged on
FreeBSD2.2.1 and 3.1R Take advantage of open source, RTFSC.
--jeff
============================================================================
Jeffrey A. Lynch | JORSM Internet, Regional Internet Services
email: jeff@jorsm.com | 7 Area Codes in Chicagoland and NW Indiana
Voice: (219)322-2180 | 100Mbps+ Connectivity, 56K-DS3, V.90, ISDN
Autoresponse: info@jorsm.com | Quality Service, Affordable Prices
http://www.jorsm.com | Serving Gov, Biz, Indivds Since 1995
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.95q.990520150142.21290C-100000>