From owner-freebsd-www  Wed Jan  8 09:33:23 1997
Return-Path: <owner-freebsd-www>
Received: (from root@localhost)
          by freefall.freebsd.org (8.8.4/8.8.4) id JAA19082
          for www-outgoing; Wed, 8 Jan 1997 09:33:23 -0800 (PST)
Received: from knight.cons.org (knight.cons.org [194.233.237.86])
          by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id JAA19073
          for <www@freebsd.org>; Wed, 8 Jan 1997 09:33:09 -0800 (PST)
Received: (from cracauer@localhost) by knight.cons.org (8.7.5/8.7.3) id SAA01754; Wed, 8 Jan 1997 18:30:04 +0100 (MET)
Date: Wed, 8 Jan 1997 18:30:04 +0100 (MET)
Message-Id: <199701081730.SAA01754@knight.cons.org>
From: Martin Cracauer <cracauer@cons.org>
To: www@freebsd.org
Subject: general URL testing tool availiable
Sender: owner-freebsd-www@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk


I wrote a little tool to automate testing WWW servers and traversing
mirrors. 

You can specify three values for each test:
- a list of hosts (you can also specify "all www servers of
  domain freebsd.org", see below)
- pathname on hosts i.e. /index.html
- a regular expression that should show up (This can be computed, for
  example, you can get a version string from a file on the main server
  and expect it on mirros).

The tool includes a nameserver walking funtion that returns every host
in a given domain, on request including all subdomains (it finds all
subdomains by itself, of course).

CGI-scripts using GET work flawlessly. Let me know if you need POST as
well.

First result: Out of 41 FreeBSD www-mirrors (counted by name, not IP
number) 25 were not able to serve /index.html up to the string "About
this server" :-(

You can use "urlcheck" on freefall, in /home/cracauer/urlcheck. See
examples.urlcheck for some kind of manual.

Examples of use:

testurl "www.bik-gmbh.de www.cons.org" \
    "/~cracauer/lisp.html" \
    "CMU" 

testurl "www-intern.bik-gmbh.de gilgamesch.bik-gmbh.de" \
    "/cgi-bin/html-zaehl0/zzz_mkhtml_caller?dir=burda-ma96&burda-gif=YES&titel1=Reichweitenvergleich+MA+96+Pressemedien+I+zu+MA+95+f%FCr+Zeitschriften%2C+Supplements+und+Wochenzeitungen+nach+ausgew%E4hlten+Verlagen*&titel4=*+Der+Berechnung+liegt+die+aktuelle+Titelliste+laut+MA+96+zugrunde.&titel2=+B.A.C.+Burda+Advertising+Center+GmbH+&zzzname=tmp.96.1&point.x=5&point.y=15" \
    "Der Berechnung liegt die aktuelle Titelliste laut MA 96
zugrunde."

testurl "a b c" /index.html `grep $\Date:\$ /usr/local/www/data/index.html`

testurl "`getns '^www' freebsd.org ns.freebsd.org`" \
	/index.html "About the FreeBSD WWW Server."

Let me know if you have further problems using it. Oh, yes, don't try
to run it using bash, /bin/sh is king. And no, I don't need CMU Common
Lisp on freefall for this task :-)

CGI interface is coming...

Martin
-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer
  cracauer@wavehh.hanse.de (batched, preferred for large mails)
  Tel.: (daytime) +4940 41478712 (sometimes hacker's daytime :-)
  Tel.: (private) +4940 5221829 Fax.: (private) +4940 5228536
  Paper: (private) Waldstrasse 200, 22846 Norderstedt, Germany