From owner-freebsd-ports@FreeBSD.ORG Fri Jul 23 22:29:36 2010 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A51F01065692; Fri, 23 Jul 2010 22:29:36 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 44EEA8FC1D; Fri, 23 Jul 2010 22:29:35 +0000 (UTC) Received: by yxe42 with SMTP id 42so4194431yxe.13 for ; Fri, 23 Jul 2010 15:29:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:openpgp:content-type:content-transfer-encoding; bh=idaFuuJE3+J4vuQ0OMFSfQMxLxoaa/liL4jYKFoBbaI=; b=cfT1nCBz0Z/Np5FH8tGzgV1EGLw3s6O6faRWIawTp589ZwPfuoAk19imM5GuKxNrSn kN8oP/DoPSZ2b7+ixMIeFVHVBiQISTnCQZLoLCTdbSS4dmUZBS0nJL9e5z0BlemGqQmR Zv5+prg4yjbCqJDhSLhLRIOEAJ5oTQg8MhJiU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:openpgp:content-type :content-transfer-encoding; b=lMQLBMqKhWjEPrQo/1W9y2hYrsunexLqfcMombpYvkNFOP56Wfk4wLdc/AA0zrV62K 8gPZvDzRF48k9MZ4CTz1Q2rUE69wsalu/KSz6YU0XMbmWO+g61J7XeA8iH/YaxhkMFiR z1zIHyB3MjR0YKiVL71O689hqa5chp+OjoDCk= Received: by 10.100.214.6 with SMTP id m6mr4562835ang.218.1279924175132; Fri, 23 Jul 2010 15:29:35 -0700 (PDT) Received: from centel.dataix.local ([99.181.133.55]) by mx.google.com with ESMTPS id t24sm1047244ano.32.2010.07.23.15.29.33 (version=SSLv3 cipher=RC4-MD5); Fri, 23 Jul 2010 15:29:34 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C4A17CC.8020707@dataix.net> Date: Fri, 23 Jul 2010 18:29:32 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.1.11) Gecko/20100722 Thunderbird MIME-Version: 1.0 To: Doug Barton References: <201007231421.o6NELsGa022354@monk.cnd.dundas.on.ca> In-Reply-To: X-Enigmail-Version: 1.0.1 OpenPGP: id=89D8547E Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-ports@freebsd.org Subject: Re: CFT: New version of portmaster in svn X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2010 22:29:36 -0000 On 07/23/2010 15:00, Doug Barton wrote: > The problem is that in order to accomplish that portmaster would have to > check every port in the tree. Assuming 22,000 ports, and that any given > port is equally likely to fall anywhere in the tree, on average you'd > have to search 11,000 ports for every distfile that is not related to a > port installed on that system. > > Now what I _could_ do is use the same technique I use in > --clean-distfiles (create a text file with the distfile information) but > instead of limiting it to installed ports, do all of them. I have never > even tested that to see how long it would take, but I suppose I could > take a look. Hi Doug Barton, Ports@, Here is a suggestion that I have just played around with involving locate(1) with environment LOCATE_PATH set in portmaster & ports-side locate.updatedb(8) with LOCATE_CONFIG before the port snapshots are made. Server side: ( at port snapshot creation ) LOCATE_CONFIG=/usr/ports/locate.rc /usr/libexec/locate.updatedb locate.rc: FCODES="/usr/ports/locate.database" SEARCHPATHS="/usr/ports" real 149.79 user 3.51 sys 27.53 2076 maximum resident set size 37 average shared memory size 3946 average unshared data size 134 average unshared stack size 2893 page reclaims 335 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 35875 voluntary context switches 38020 involuntary context switches In portmaster(8): LOCATE_PATH=/usr/ports/locate.database Generate a list of distinfo files and create a distfile index using ``SIZE''. for file in $(locate "/usr/ports/*/*/distinfo"); do grep "SIZE " $file >>DISTINDEX.8 done Use the output from that to judge if the file in /usr/ports/distfiles needs to exist or of the file does not meet the SIZE requirements & you can kill two birds with one stone. You could always use the MD5 or SHA256 sum that's recorded but that may be IMO a slight bit of overhead. This may not be an exact proccess right now but it sure would be nice to have a locate database in ports that could be updated by (make locatedb) or something similar. Just for creating the list of distfiles from the SIZE contents: $ du -sh DISTINDEX.8.1.gz 512B DISTINDEX.8.1.gz time -alhp ./createindex.sh real 490.99 user 28.03 sys 117.14 1820 maximum resident set size 228 average shared memory size 2297 average unshared data size 133 average unshared stack size 2353331 page reclaims 0 page faults 0 swaps 0 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 77884 voluntary context switches 165392 involuntary context switches Regards, -- jhell,v