From owner-freebsd-questions Tue May 7 21:43:47 2002 Delivered-To: freebsd-questions@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id 0CECE37B403 for ; Tue, 7 May 2002 21:43:37 -0700 (PDT) Received: from user-119a7q7.biz.mindspring.com ([66.149.31.71] helo=athlon.wsonline.net) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #2) id 175JIa-0001e0-00 for freebsd-questions@FreeBSD.ORG; Tue, 07 May 2002 21:43:32 -0700 Message-Id: <5.1.0.14.0.20020507224720.00ad6cc8@pop.wsonline.net> X-Sender: richardh@pop.wsonline.net X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 07 May 2002 22:47:30 -0600 To: freebsd-questions@FreeBSD.ORG From: RichardH Subject: Re: Parsing Log Files Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Thx for the parsing suggestions, we are working on a custom script for=20 parsing access logs out as needed to a users home dir under a "log" dir for= =20 multiple users from 1 to 10000+. I don't know if this would help with=20 getting this developed (we are working on it but any help would be=20 appreciated and recognized as such). We run hashed user dir, like username= =20 is under /home/u/s/username, keeps dir structure more ?define-able?, do=20 same with zone files, try it, you'll like it. Anyway, if we get a script=20 together that can parse out the access logs without running massive=20 processes (ie. transferlog directive) to do it we will post it in here so=20 any help will be appreciated by all in the long run. Also, adding user logs= =20 to the script should be fairly painless, we are working on this part also=20 in that the whole script would not have to be opened and edited for each=20 user add, possible to write into the adduser??For more info on what we are= =20 wanting to do go to webhostingtalk.com and do a search for user storm2k and= =20 read the thread. Possibly at this link (may or may not work, if not do the= =20 search for user storm2k) http://www.webhostingtalk.com/showthread.php?s=3D0785248167d55ea6c36f39866be= 96f78&threadid=3D46871 now for a stoopid question, I have the large banner for FBSD on my site but= =20 I want a smaller button, where are those located (banners, buttons, linking= =20 stuff, etc.)? I cannot locate it for the life of me and I went through damn= =20 near the whole .org site. please send a link to that page asap, info on the= =20 other is appreciated but not expected asap :-) Thanks for input, Richard Hutson There are two major products that came out of Berkeley: LSD and BSD. We=20 don't believe this to be a coincidence. =97Jeremy S. Anderson At 07:48 PM 5/2/2002, RichardH wrote: >>Delivered-To: freebsd-questions@freebsd.org >>Date: Thu, 2 May 2002 09:24:35 -0700 >>To: questions@FreeBSD.ORG >>Subject: Re: Parsing Log Files >>X-Mailer: Sylpheed version 0.7.4 (GTK+ 1.2.10; i386-portbld-freebsd4.5) >>Sender: owner-freebsd-questions@FreeBSD.ORG >>List-ID: >>List-Archive: (Web Archive) >>List-Help: (List= Instructions) >>List-Subscribe:=20 >> >>List-Unsubscribe:=20 >> >>X-Loop: FreeBSD.ORG >> >>On Thu, 2 May 2002 11:02:03 -0400 >>Rob Ellis wrote: >> >> > On Wed, May 01, 2002 at 07:29:29PM -0600, RichardH wrote: >> > > By parsing out the files with a script, it reduces overall server >> > > load AND permits the use of rewrite rules, that allow you to use a >> > > virtmap.txt type of setup for hosting entries (in which case the >> > > transferlog entry does not work at all). >> > >> > Assuming the domain name is the first thing on each log line, >> > you could do something like >> > >> > #! /usr/bin/perl -w >> > use FileCache; # opens/closes file descriptors as required >> > no strict "refs"; # FileCache generates "strict refs" warnings >> > $log =3D "/usr/local/apache/logs/access_log"; >> > $outdir =3D "/usr/local/var/weblogs"; >> > open(LOG, $log) || die $!; >> > while () { >> > if (/^([\w\.-]+)\s+/) { >> > $domain =3D $1; >> > $outfile =3D "$outdir/$domain/access_log"; >> > die $! unless (cacheout $outfile); >> > print $outfile $_; >> > } >> > # do something here with junk lines >> > } >> > close(LOG); >> > 1; >> >>Here are some snips from a small script that I put together to parse the >>apache log (/var/log/httpd-access.log) to find suspect log entries >>containing lame attempts to exploit IIS vulnerabilities. If found, it >>will try to send an email to "abuse" at whatever domain the user was at. >> It doesn't write anything to an output file, but it does selectively >>choose entries from the current date only. You could possibly modify >>this to append each days activities to each users log file. Again, the >>below doesn't necessarily speak to your particular problem, but maybe >>some tidbits of this could be a start, along with the post from Rob >>Ellis. >> >>#!/usr/bin/perl -w >> >>use strict; >>use Mail::Sendmail; >> >>my ($line, $host, $rcpt, $dstamp, $body); # some scalars >>my @date; # an array >>my (%mail, %offenders); # some hashes >> >>@date =3D split(" ", `date`); # get current date into >>an array$dstamp =3D "$date[2]/$date[1]/$date[5]"; # rearrange to >>match date in apache log file >> >> >>open (FILE, "/var/log/httpd-access.log"); # open log file for >>reading >> >>while ($line =3D ) { >> # find log entries from today that also contain mischevious keywords >> if ( (grep(/.*\[$dstamp:/, $line)) && >>(grep(/scripts|winnt|cmd\.exe|root\.exe|system32/, $line)) ) { >>$line =3D~ /^(\S+).*\[(.+)\].*GET\s(\S+)/; # parse interesting line >>$1=3Dhost $2=3Ddate/time $3=3DGET command push @{$offenders{$1}},"$2 >>$3\n"; # put values into a hash for later processing } >>} >> >>foreach $host (keys(%offenders)) { >> if ($host !~ /\.\d+$/) { # only act if $host is an actual host >>name to which we can construct an email $host =3D~ /^\S+\.(.*)$/; # >>get domain portion of $host $rcpt =3D $1; # assign >>$rcpt to value of previous regex $body =3D ( # create >>the email body "Email Body" >> ); >> %mail =3D ( # create some email headers >> 'Date' =3D> Mail::Sendmail::time_to_date(), >> 'To' =3D> "abuse\@$rcpt", >> 'From' =3D> 'somebody@somewhere.org', >> 'Subject' =3D> 'Notification of malicious user or system', >> 'Body' =3D> "$body" >> ); >> sendmail(%mail); # send the mail >> } >>} >> >>close (FILE); # close the file log file >> >>To Unsubscribe: send mail to majordomo@FreeBSD.org >>with "unsubscribe freebsd-questions" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message