From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 11:04:34 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A39B916A41F; Mon, 28 May 2007 11:04:34 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id 3E2AD13C480; Mon, 28 May 2007 11:04:34 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [129.247.12.6] ([129.247.12.6]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 May 2007 13:04:26 +0200 Message-ID: <465AB737.1030708@dlr.de> Date: Mon, 28 May 2007 13:04:23 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Matthew Seaman References: <4659EF80.70100@math.missouri.edu> <465A94C0.2080301@infracaninophile.co.uk> In-Reply-To: <465A94C0.2080301@infracaninophile.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 May 2007 11:04:26.0727 (UTC) FILETIME=[F4D50770:01C7A117] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 11:04:34 -0000 Matthew Seaman wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the >> value of the variable PKGNAME. > > pkg_version is one thing -- but to build the INDEX you need to extract > at least the values of the following variables: > > PKGNAME > .CURDIR > PREFIX > COMMENT > DESCR > MAINTAINER > CATEGORIES > EXTRACT_DEPENDS > PATCH_DEPENDS > FETCH_DEPENDS > BUILD_DEPENDS > RUN_DEPENDS > LIB_DEPENDS > > Plus you need to grep in the referenced pkg-descr file for any WWW > links. I also extract the values of: > > MASTER_PORT > .MAKEFILE_LIST > SUBDIR > > for my FreeBSD::Portindex stuff. > > Trouble is, by the time you've extracted all that lot, you have pretty > much done the same level of variable processing as you would were you > actually going to build the port. > > One thing that would speed up this process would be a make option > to just do parsing of the Makefile and variable expansion, without > calling stat(2) on all the various sources and dependencies involved. > > For instance: > > happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME |& grep stat | wc -l > 49 > > It is quite instructive to see what files make(1) touches while doing > that. At least half of them are irrelevant if all make(1) is going to > do is print out the values of some variables. Multiply that by 17,000 > and it adds up to a big waste of effort. Are you sure? A good deal of the stats() is make poking around for the make infrastructure. This should be in the cache. And there are a couple of stats for the *done* files that might be avoided by doing something in the ports infrastructure. But as I already said in my previous mail: numbers, please, no guessing. harti