From owner-freebsd-questions@FreeBSD.ORG Mon Mar 30 20:55:49 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0A5C1065731 for ; Mon, 30 Mar 2009 20:55:49 +0000 (UTC) (envelope-from chowse@charter.net) Received: from mta11.charter.net (mta11.charter.net [216.33.127.80]) by mx1.freebsd.org (Postfix) with ESMTP id 833688FC19 for ; Mon, 30 Mar 2009 20:55:49 +0000 (UTC) (envelope-from chowse@charter.net) Received: from imp09 ([10.20.200.9]) by mta11.charter.net (InterMail vM.7.09.01.00 201-2219-108-20080618) with ESMTP id <20090330205536.PHKP21265.mta11.charter.net@imp09>; Mon, 30 Mar 2009 16:55:36 -0400 Received: from [192.168.254.3] ([24.176.107.216]) by imp09 with charter.net id ZYvb1b00M4g9tMU05Yvckl; Mon, 30 Mar 2009 16:55:36 -0400 Message-Id: <82E8EACC-E6A3-48FA-9203-80A5F924870F@charter.net> From: Charles Howse To: Mel Flynn In-Reply-To: <200903302211.30208.mel.flynn+fbsd.questions@mailing.thruhere.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 30 Mar 2009 15:55:35 -0500 References: <5BCD53E0-5B9F-4349-8401-FBF0ACF369C2@charter.net> <4ad871310903291154j74c35b0p545157b848adf8b9@mail.gmail.com> <8BDE67B0-B6B1-4AAC-A0FD-0E519E74CBAF@charter.net> <200903302211.30208.mel.flynn+fbsd.questions@mailing.thruhere.net> X-Mailer: Apple Mail (2.930.3) Cc: FreeBSD Questions Subject: Re: analyzing httpd-error.log X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2009 20:55:51 -0000 On Mar 30, 2009, at 3:11 PM, Mel Flynn wrote: > On Sunday 29 March 2009 22:25:55 Charles Howse wrote: >> On Mar 29, 2009, at 1:54 PM, Glen Barber wrote: >>> On Sun, Mar 29, 2009 at 1:33 PM, Charles Howse >>> >>> wrote: >>>> On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: >>>>> Hi, >>>>> >>>>>> Webalizer is doing what it's supposed to with httpd-access.log, >>>>>> but >>>>>> when I give it the error log to process is coughs, spits and >>>>>> spills >>>>>> out errors with no data processed. My research hasn't turned >>>>>> up a >>>>>> good solution for webalizer and -error.log. >>> >>> What are the errors? >> >> Intrusion attempts, (a few) bad links in my website, also I use the >> error.log to troubleshoot cgi scripts. > > Nothing beats tail -f for debugging. > > 404's can be gathered from access log, by webalizer (or awstats > or ...) and > are summarized when enabled. If I remember correctly, there's also a > referer > top list, that specifies which pages link to invalid pages, but it > may have > been awstats that does this. > > Can't think of anything specific for apache error log, I roll my own > grok > rules if I'm really interested in a specific vulnerability. See > sysutils/grok. Yes, awstats does a fair job of summarizing 404's. I have a hardware router, not extremely interested in a packet filter for my webserver. I guess I could do some deny from rules in apache.conf if necessary...? ***** #!/bin/sh date=`date "+%b %e"` cd /var/log grep "$date" httpd-error.log | grep -v 192.168.254.254 | grep -v 192.168.254.3 > /root/err.log mail -s "httpd-error.log" charles < /root/err.log ***** I run this from cron @ 11:59 PM every night, and that gives me the error log for that day. Maybe I won't miss anything serious between 11:59 and midnight. Can anyone help with a grep command that will filter out all addresses beginning with 192.168.254? Something to replace the 2 piped commands with only 1. I tried: grep -v -regexp[^192\.168\.254\.] but that didn't match any records...?