From owner-freebsd-ports@FreeBSD.ORG Tue Apr 4 09:45:53 2006 Return-Path: X-Original-To: freebsd-ports@FreeBSD.org Delivered-To: freebsd-ports@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9B8C416A41F for ; Tue, 4 Apr 2006 09:45:53 +0000 (UTC) (envelope-from shaun@inerd.com) Received: from dione.picobyte.net (host-212-158-207-124.bulldogdsl.com [212.158.207.124]) by mx1.FreeBSD.org (Postfix) with SMTP id C36AE43D55 for ; Tue, 4 Apr 2006 09:45:52 +0000 (GMT) (envelope-from shaun@inerd.com) Received: from charon.picobyte.net (charon.picobyte.net [IPv6:2001:4bd0:201e::fe03]) by dione.picobyte.net (Postfix) with ESMTP for ; Tue, 4 Apr 2006 10:45:50 +0100 (BST) Date: Tue, 4 Apr 2006 10:45:49 +0100 From: Shaun Amott To: freebsd-ports@FreeBSD.org Message-ID: <20060404094549.GA751@picobyte.net> Mail-Followup-To: freebsd-ports@FreeBSD.org References: <20060403190532.GA966@picobyte.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20060403190532.GA966@picobyte.net> User-Agent: Mutt/1.5.11 (FreeBSD i386) Cc: Subject: Re: portscout - new distfile scanner X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Apr 2006 09:45:53 -0000 Thanks for all the feedback everyone. It's been very useful. I'm slightly disappointed by all the false positives. The only explanation I have for portscout finding versions that don't exist is hosts lieing about files. Edwin's script dealt with this by using a host blacklist - portscout tries to avoid the problem by sending dummy files to the master site and looking for "ok" responses. I have a feeling that portscout is being too aggressive and causing connection failures by flooding the network. There seems to be another bug that has surfaced which affects certain files with version number components beginnings with "0". I'm looking into this. Someone noted that portscout found 0.811 as a update for 0.9 -- unfortunately, when I tried to account for weird versions like this before, it didn't work out well at all. So, for now, portscout assumes version numbers count up. I did another test run last night with improved results. Version string extraction has improved considerably; there are more results than before, and from what I can tell, less false positives. The results have been uploaded. Building the database from scratch takes a long time - I haven't timed it recently, but I think it finishes in around three hours. That is essentially converting all the information from the ports tree into a usable, easily-accessible format. Re-building after a cvsup/index takes a minute or two. The check last night took 2:40:51; the machine is a PIII 750MHz with 256MB RAM and very fast disks. It usually has a load average of 0 - 1; Running portscout with 30 parallel processes puts the load average up to 12.0. Average download bandwidth used was around 780Kbps. I will do another run tonight with less processes and see if the phantom updates disappear. -Shaun -- Shaun Amott [ PGP: 0x6B387A9A ] Scientia Est Potentia.