From owner-freebsd-ports@FreeBSD.ORG Thu Feb 2 21:26:08 2012 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07E201065670 for ; Thu, 2 Feb 2012 21:26:08 +0000 (UTC) (envelope-from jhelfman@experts-exchange.com) Received: from mail.dw.redsrci.com (mail.pub.dw.redsrci.com [72.29.183.251]) by mx1.freebsd.org (Postfix) with ESMTP id D1E2D8FC1B for ; Thu, 2 Feb 2012 21:26:07 +0000 (UTC) Received: from mail.dw.redsrci.com (localhost [127.0.0.1]) by mail.dw.redsrci.com (Postfix) with ESMTP id 62E80701F3E; Thu, 2 Feb 2012 13:26:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=e-e.com; h= user-agent:organization:in-reply-to:content-disposition :content-type:content-type:mime-version:references:message-id :subject:subject:from:from:date:date:received:received:received; s=ee; t=1328217967; x=1330032367; bh=0Y1jj6Bb4cPzzhXCYt4EeQx9Rz PoaImfRiYzGENSwDU=; b=WrvagoudlIravS+TlJ6C1X0+jtADseAaY9YT4mCmMz YGisxrRrSp0L7YBclJlaS1YhJc6N7De87jHQU9Fv5rx7eFXXWNaTP1SFnHLvNWHO 7MXGTajMxSAposZ6QDEaYMbUIL+dpmfZHzoMY+AN8NJdAe3dGiP78wXjJKsW+8+p 0= X-Virus-Scanned: amavisd-new at experts-exchange.com Received: from mail.dw.redsrci.com ([127.0.0.1]) by mail.dw.redsrci.com (mail.dw.redsrci.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RVtzmo1aalbT; Thu, 2 Feb 2012 13:26:07 -0800 (PST) Received: from experts-exchange.com (unknown [192.168.103.120]) by mail.dw.redsrci.com (Postfix) with SMTP id 3BDB7701EDA; Thu, 2 Feb 2012 13:26:07 -0800 (PST) Received: (nullmailer pid 93725 invoked by uid 1001); Thu, 02 Feb 2012 21:25:14 -0000 Date: Thu, 2 Feb 2012 13:25:14 -0800 From: Jason Helfman To: "Conrad J. Sabatier" Message-ID: <20120202212512.GE19714@dormouse.experts-exchange.com> References: <20111203173149.224a64e6@cox.net> <20111214004838.GK1593@dormouse.experts-exchange.com> <20120112212905.GA78819@dormouse.experts-exchange.com> <20120127200325.66f36090@cox.net> <20120128143734.6166f5fc@gumby.homeunix.com> <20120128102831.51cf0f63@cox.net> <4F2427FD.2000900@infracaninophile.co.uk> <20120202152137.21b638d0@cox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20120202152137.21b638d0@cox.net> X-Operating-System: FreeBSD 8.2-RELEASE amd64 Organization: Experts-Exchange, LLC. http://www.experts-exchange.com X-Living-The-Dream: I love the SLO Life! User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-ports@freebsd.org Subject: Re: A new and better way to do "make readmes"? X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Feb 2012 21:26:08 -0000 On Thu, Feb 02, 2012 at 03:21:37PM -0600, Conrad J. Sabatier thus spake: >[ Sorry to be so late in following up on this; lost track for a while ] > >On Sat, 28 Jan 2012 16:53:17 +0000 >Matthew Seaman wrote: > >> On 28/01/2012 16:28, Conrad J. Sabatier wrote: >> > Am I understanding you correctly? Are >> > you saying you built 20,000+ port READMEs in only 9 seconds?! How >> > is that possible? Or do you mean 9 seconds for each one? >> >> 9 seconds sounds quite reasonable for generating 23000 or so files. > >It sounds incredible to me! :-) > >> >> > Selective updating isn't going to help because 99.9% of the time >> >> > is spent in the categories and it only takes a single port >> >> > update to make a category file obsolete. >> >> > This is the part I find troubling. It would seem that it should be >> > more work to create an individual port README, with its plucking the >> > appropriate line out of the INDEX-* file and then parsing it into >> > its respective pieces and filling in a template, than to simply >> > string together a list of references to a bunch of already built >> > port READMEs into a category README. >> > >> > What am I not getting here? >> >> No -- you're quite right. You could generate the category README.html >> files entirely from the data in the INDEX. It's not quite as easy as >> all that, because there aren't entries for each category separately, >> so you'll have to parse the structure out of all of the paths in the >> INDEX. > >Well, the idea I had in mind was that, if all of the individual ports' >README.html files already are in place, then it should be trivial to >just "ls" or "find" them under each category to fill in the category's >README.html. No need to reference the INDEX or anything else. Or??? > >The workaround method I've been running out of cron for the last month >or so is: > >1) Create a "sentinel" file under /tmp to use as a timestamp, just >before running "cvs update" on ports (I update my ports tree from a >local copy of the CVS repo maintained via csup) > >2) After cvs completes, look for any port directories containing >updates (check timestamps against the sentinel file) and do a "make >readme" for each one: > >find $PORTSDIR -type f ! -path "*/CVS/*" -newercm $SENTINEL -depth 3 | > xargs dirname | > sort -u | xargs -I@ /bin/sh -c "cd @ && make readme" > >3) Last, but not least, build the category README.html for any >categories with ports containing newly updated README.html files. > >I have noticed while doing this that, as you mentioned, the category >READMEs take considerably longer than the individual ports'. > >I don't even bother to rebuild the top-level file, since it's basically >unchanging anyway. > >> >> > I think the way to speed this up is to have the script generate >> >> > the category files too. There's no point in bringing in the >> >> > top-level README since that's already fast. >> >> > So what's making the category READMEs so slow then? >> >> The big problem with performance in all this INDEX and README.html >> building is that it takes quite a long time relatively to run make(1) >> within any port or category directory. make(1) has to read in a lot >> of other files and stat(2) many more[*] -- all of which involves a >> lot of random-access disk IO, and that's always going to take quite a >> lot of time. Now, doing 'make readme' in a category directory >> doesn't just run make in that directory, but also in every port in >> that category. Popular categories can contain many hundreds of ports. > >I'm a little rusty on the actual mechanics of make, but shouldn't it be >possible to run a single, over-arching make on each category that >wouldn't need to spawn a bunch of sub-makes? > >> Maybe I should add README.html generation to my FreeBSD::Portindex >> stuff. Should be pretty simple -- all the necessary bits are readily >> available and it is just a matter of formatting it as HTML and >> printing it out. > >"Maybe"? Whaddya mean, "maybe"? :-) Sounds like it would definitely >be worth doing! > >> Cheers, >> >> Matthew >> >> [*] Running 'make -dA' with maximum debug output is quite >> enlightening, as is running make under truss(1) > >Enlightening, perhaps. Sometimes overwhelming, is more like it. :-) > Not to fancy, but I used this when I was updating the readmes to not break. #!/bin/sh cd /usr/ports for i in `make -V SUBDIR |sed s/local//g`; do for p in `make -C $i -V SUBDIR`; do echo $i/$p && sudo make -C "$i/$p" readme ; done; done >> ~/readmes.log -jgh -- Jason Helfman System Administrator experts-exchange.com http://www.experts-exchange.com/M_4830110.html E4AD 7CF1 1396 27F6 79DD 4342 5E92 AD66 8C8C FBA5