Date: Thu, 2 Feb 2012 13:25:14 -0800 From: Jason Helfman <jhelfman@e-e.com> To: "Conrad J. Sabatier" <conrads@cox.net> Cc: freebsd-ports@freebsd.org Subject: Re: A new and better way to do "make readmes"? Message-ID: <20120202212512.GE19714@dormouse.experts-exchange.com> In-Reply-To: <20120202152137.21b638d0@cox.net> References: <20111203173149.224a64e6@cox.net> <20111214004838.GK1593@dormouse.experts-exchange.com> <20120112212905.GA78819@dormouse.experts-exchange.com> <20120127200325.66f36090@cox.net> <20120128143734.6166f5fc@gumby.homeunix.com> <20120128102831.51cf0f63@cox.net> <4F2427FD.2000900@infracaninophile.co.uk> <20120202152137.21b638d0@cox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Feb 02, 2012 at 03:21:37PM -0600, Conrad J. Sabatier thus spake: >[ Sorry to be so late in following up on this; lost track for a while ] > >On Sat, 28 Jan 2012 16:53:17 +0000 >Matthew Seaman <m.seaman@infracaninophile.co.uk> wrote: > >> On 28/01/2012 16:28, Conrad J. Sabatier wrote: >> > <rubbing eyes in disbelief> Am I understanding you correctly? Are >> > you saying you built 20,000+ port READMEs in only 9 seconds?! How >> > is that possible? Or do you mean 9 seconds for each one? >> >> 9 seconds sounds quite reasonable for generating 23000 or so files. > >It sounds incredible to me! :-) > >> >> > Selective updating isn't going to help because 99.9% of the time >> >> > is spent in the categories and it only takes a single port >> >> > update to make a category file obsolete. >> >> > This is the part I find troubling. It would seem that it should be >> > more work to create an individual port README, with its plucking the >> > appropriate line out of the INDEX-* file and then parsing it into >> > its respective pieces and filling in a template, than to simply >> > string together a list of references to a bunch of already built >> > port READMEs into a category README. >> > >> > What am I not getting here? >> >> No -- you're quite right. You could generate the category README.html >> files entirely from the data in the INDEX. It's not quite as easy as >> all that, because there aren't entries for each category separately, >> so you'll have to parse the structure out of all of the paths in the >> INDEX. > >Well, the idea I had in mind was that, if all of the individual ports' >README.html files already are in place, then it should be trivial to >just "ls" or "find" them under each category to fill in the category's >README.html. No need to reference the INDEX or anything else. Or??? > >The workaround method I've been running out of cron for the last month >or so is: > >1) Create a "sentinel" file under /tmp to use as a timestamp, just >before running "cvs update" on ports (I update my ports tree from a >local copy of the CVS repo maintained via csup) > >2) After cvs completes, look for any port directories containing >updates (check timestamps against the sentinel file) and do a "make >readme" for each one: > >find $PORTSDIR -type f ! -path "*/CVS/*" -newercm $SENTINEL -depth 3 | > xargs dirname | > sort -u | xargs -I@ /bin/sh -c "cd @ && make readme" > >3) Last, but not least, build the category README.html for any >categories with ports containing newly updated README.html files. > >I have noticed while doing this that, as you mentioned, the category >READMEs take considerably longer than the individual ports'. > >I don't even bother to rebuild the top-level file, since it's basically >unchanging anyway. > >> >> > I think the way to speed this up is to have the script generate >> >> > the category files too. There's no point in bringing in the >> >> > top-level README since that's already fast. >> >> > So what's making the category READMEs so slow then? >> >> The big problem with performance in all this INDEX and README.html >> building is that it takes quite a long time relatively to run make(1) >> within any port or category directory. make(1) has to read in a lot >> of other files and stat(2) many more[*] -- all of which involves a >> lot of random-access disk IO, and that's always going to take quite a >> lot of time. Now, doing 'make readme' in a category directory >> doesn't just run make in that directory, but also in every port in >> that category. Popular categories can contain many hundreds of ports. > >I'm a little rusty on the actual mechanics of make, but shouldn't it be >possible to run a single, over-arching make on each category that >wouldn't need to spawn a bunch of sub-makes? > >> Maybe I should add README.html generation to my FreeBSD::Portindex >> stuff. Should be pretty simple -- all the necessary bits are readily >> available and it is just a matter of formatting it as HTML and >> printing it out. > >"Maybe"? Whaddya mean, "maybe"? :-) Sounds like it would definitely >be worth doing! > >> Cheers, >> >> Matthew >> >> [*] Running 'make -dA' with maximum debug output is quite >> enlightening, as is running make under truss(1) > >Enlightening, perhaps. Sometimes overwhelming, is more like it. :-) > Not to fancy, but I used this when I was updating the readmes to not break. #!/bin/sh cd /usr/ports for i in `make -V SUBDIR |sed s/local//g`; do for p in `make -C $i -V SUBDIR`; do echo $i/$p && sudo make -C "$i/$p" readme ; done; done >> ~/readmes.log -jgh -- Jason Helfman System Administrator experts-exchange.com http://www.experts-exchange.com/M_4830110.html E4AD 7CF1 1396 27F6 79DD 4342 5E92 AD66 8C8C FBA5
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120202212512.GE19714>