From owner-freebsd-questions Wed May 1 22:22:21 2002 Delivered-To: freebsd-questions@freebsd.org Received: from buffnet4.buffnet.net (buffnet4.buffnet.net [205.246.19.13]) by hub.freebsd.org (Postfix) with ESMTP id 0231437B417 for ; Wed, 1 May 2002 22:22:15 -0700 (PDT) Received: from buffnet5.buffnet.net (buffnet5.buffnet.net [205.246.19.14]) by buffnet4.buffnet.net (8.9.3/8.8.7) with ESMTP id BAA60534; Thu, 2 May 2002 01:21:54 -0400 (EDT) (envelope-from mtech@buffnet.net) Date: Thu, 2 May 2002 01:21:54 -0400 (EDT) From: Mohsin Rahman To: Bill Moran Cc: RichardH , questions@FreeBSD.ORG Subject: Re: Parsing Log Files In-Reply-To: <3CD0B07C.6030207@potentialtech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 1 May 2002, Bill Moran wrote: > RichardH wrote: > > I have looked all in here and on the net for more specific info on > > parsing access logs for multiple user accounts. If I have 300 users on a > > server and want to have a "log" directory in each users home directory > > is there a script or a simple grep that will pull that users info from > > the raw access logs and split/parse the info out to the individual log > > directory in each users account so that it can then be viewed by the > > user with AWStats, Analog, etc. I know this has to be accomplished > > failry easily but cannot find any links or data giving the exact way of > > doing it. Any input would be greatly appreciated. OS is FreeBSD4.5 > > w/Apache Web Server 1.3.24. > > > > > The above post says 300 users as an example, actually looking at 3000+ > > users per server. > > > > Any input, tips, etc. would be greatly appreciated. > > Thanks,Richard Hutson > > Have you tried simply running a grep on the main log at a certain schedule > to extract the records that pertain to a specific host? It would work > around your open files concern. > Apache logs are pretty configurable, I don't remember details, but it should > be possible to ensure that each virtual host puts something in the log > file that is distinct, that can then be grepped out. > If you run the grep processes on a daily basis, you should probably stagger > them. 3000 greps trying to extract data from the same file at the same time > is going to be a HUGE load. Probably a script that simply ran all the required > greps, one after another in succession would work. > > > -- > Bill Moran > Potential Technology > http://www.potentialtech.com > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-questions" in the body of the message > I do not claim to be an apache expert, the config below works for me. Some more experienced users can probably suggest something better. Please also note that the REQUEST_URI method will match all instances of user1, user10, user110 and so on. Might not be accurate when the users grow to a large number. What you want to do for each user is to look at the SetEnvIf and Request_URI. For example, to filter out user1's log: In httpd.conf: SetEnvIf Request_URI \user1 user1_logs SetEnvIf Request_URI \~user1 user1_logs CustomLog /home/user1/user1-access_log combined env=user1_logs I picked this syntax from the Nimda virus days and adjusted it to log my personal webpage access logs (not a vertual host). If you mean you have 300+ virtual hosts, then add this in each ServerAdmin webmaster@buffnet.net DocumentRoot /home/user1 ServerName www.user1.com ErrorLog /home/user1/user1-error_log CustomLog /home/user1/user1-access_log combined Hope this helps. Mohsin AbdulRahman MTech@BuffNET.Net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message