Date: Tue, 07 May 2002 22:47:30 -0600 From: RichardH <richardh@wsonline.net> To: freebsd-questions@FreeBSD.ORG Subject: Re: Parsing Log Files Message-ID: <5.1.0.14.0.20020507224720.00ad6cc8@pop.wsonline.net>
next in thread | raw e-mail | index | archive | help
Thx for the parsing suggestions, we are working on a custom script for parsing access logs out as needed to a users home dir under a "log" dir for multiple users from 1 to 10000+. I don't know if this would help with getting this developed (we are working on it but any help would be appreciated and recognized as such). We run hashed user dir, like username is under /home/u/s/username, keeps dir structure more ?define-able?, do same with zone files, try it, you'll like it. Anyway, if we get a script together that can parse out the access logs without running massive processes (ie. transferlog directive) to do it we will post it in here so any help will be appreciated by all in the long run. Also, adding user logs to the script should be fairly painless, we are working on this part also in that the whole script would not have to be opened and edited for each user add, possible to write into the adduser??For more info on what we are wanting to do go to webhostingtalk.com and do a search for user storm2k and read the thread. Possibly at this link (may or may not work, if not do the search for user storm2k) http://www.webhostingtalk.com/showthread.php?s=0785248167d55ea6c36f39866be96f78&threadid=46871 now for a stoopid question, I have the large banner for FBSD on my site but I want a smaller button, where are those located (banners, buttons, linking stuff, etc.)? I cannot locate it for the life of me and I went through damn near the whole .org site. please send a link to that page asap, info on the other is appreciated but not expected asap :-) Thanks for input, Richard Hutson There are two major products that came out of Berkeley: LSD and BSD. We don't believe this to be a coincidence. —Jeremy S. Anderson At 07:48 PM 5/2/2002, RichardH wrote: >>Delivered-To: freebsd-questions@freebsd.org >>Date: Thu, 2 May 2002 09:24:35 -0700 >>To: questions@FreeBSD.ORG >>Subject: Re: Parsing Log Files >>X-Mailer: Sylpheed version 0.7.4 (GTK+ 1.2.10; i386-portbld-freebsd4.5) >>Sender: owner-freebsd-questions@FreeBSD.ORG >>List-ID: <freebsd-questions.FreeBSD.ORG> >>List-Archive: <http://docs.freebsd.org/mail/>; (Web Archive) >>List-Help: <mailto:majordomo@FreeBSD.ORG?subject=help> (List Instructions) >>List-Subscribe: >><mailto:majordomo@FreeBSD.ORG?subject=subscribe%20freebsd-questions> >>List-Unsubscribe: >><mailto:majordomo@FreeBSD.ORG?subject=unsubscribe%20freebsd-questions> >>X-Loop: FreeBSD.ORG >> >>On Thu, 2 May 2002 11:02:03 -0400 >>Rob Ellis wrote: >> >> > On Wed, May 01, 2002 at 07:29:29PM -0600, RichardH wrote: >> > > By parsing out the files with a script, it reduces overall server >> > > load AND permits the use of rewrite rules, that allow you to use a >> > > virtmap.txt type of setup for hosting entries (in which case the >> > > transferlog entry does not work at all). >> > >> > Assuming the domain name is the first thing on each log line, >> > you could do something like >> > >> > #! /usr/bin/perl -w >> > use FileCache; # opens/closes file descriptors as required >> > no strict "refs"; # FileCache generates "strict refs" warnings >> > $log = "/usr/local/apache/logs/access_log"; >> > $outdir = "/usr/local/var/weblogs"; >> > open(LOG, $log) || die $!; >> > while (<LOG>) { >> > if (/^([\w\.-]+)\s+/) { >> > $domain = $1; >> > $outfile = "$outdir/$domain/access_log"; >> > die $! unless (cacheout $outfile); >> > print $outfile $_; >> > } >> > # do something here with junk lines >> > } >> > close(LOG); >> > 1; >> >>Here are some snips from a small script that I put together to parse the >>apache log (/var/log/httpd-access.log) to find suspect log entries >>containing lame attempts to exploit IIS vulnerabilities. If found, it >>will try to send an email to "abuse" at whatever domain the user was at. >> It doesn't write anything to an output file, but it does selectively >>choose entries from the current date only. You could possibly modify >>this to append each days activities to each users log file. Again, the >>below doesn't necessarily speak to your particular problem, but maybe >>some tidbits of this could be a start, along with the post from Rob >>Ellis. >> >>#!/usr/bin/perl -w >> >>use strict; >>use Mail::Sendmail; >> >>my ($line, $host, $rcpt, $dstamp, $body); # some scalars >>my @date; # an array >>my (%mail, %offenders); # some hashes >> >>@date = split(" ", `date`); # get current date into >>an array$dstamp = "$date[2]/$date[1]/$date[5]"; # rearrange to >>match date in apache log file >> >> >>open (FILE, "/var/log/httpd-access.log"); # open log file for >>reading >> >>while ($line = <FILE>) { >> # find log entries from today that also contain mischevious keywords >> if ( (grep(/.*\[$dstamp:/, $line)) && >>(grep(/scripts|winnt|cmd\.exe|root\.exe|system32/, $line)) ) { >>$line =~ /^(\S+).*\[(.+)\].*GET\s(\S+)/; # parse interesting line >>$1=host $2=date/time $3=GET command push @{$offenders{$1}},"$2 >>$3\n"; # put values into a hash for later processing } >>} >> >>foreach $host (keys(%offenders)) { >> if ($host !~ /\.\d+$/) { # only act if $host is an actual host >>name to which we can construct an email $host =~ /^\S+\.(.*)$/; # >>get domain portion of $host $rcpt = $1; # assign >>$rcpt to value of previous regex $body = ( # create >>the email body "Email Body" >> ); >> %mail = ( # create some email headers >> 'Date' => Mail::Sendmail::time_to_date(), >> 'To' => "abuse\@$rcpt", >> 'From' => 'somebody@somewhere.org', >> 'Subject' => 'Notification of malicious user or system', >> 'Body' => "$body" >> ); >> sendmail(%mail); # send the mail >> } >>} >> >>close (FILE); # close the file log file >> >>To Unsubscribe: send mail to majordomo@FreeBSD.org >>with "unsubscribe freebsd-questions" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5.1.0.14.0.20020507224720.00ad6cc8>