Date: Sat, 28 Jan 2012 16:53:17 +0000 From: Matthew Seaman <m.seaman@infracaninophile.co.uk> To: freebsd-ports@freebsd.org Subject: Re: A new and better way to do "make readmes"? Message-ID: <4F2427FD.2000900@infracaninophile.co.uk> In-Reply-To: <20120128102831.51cf0f63@cox.net> References: <20111203173149.224a64e6@cox.net> <20111214004838.GK1593@dormouse.experts-exchange.com> <20120112212905.GA78819@dormouse.experts-exchange.com> <20120127200325.66f36090@cox.net> <20120128143734.6166f5fc@gumby.homeunix.com> <20120128102831.51cf0f63@cox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig100ACD4695C1313CE5922E24
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
On 28/01/2012 16:28, Conrad J. Sabatier wrote:
> <rubbing eyes in disbelief> Am I understanding you correctly? Are you=
> saying you built 20,000+ port READMEs in only 9 seconds?! How is that
> possible? Or do you mean 9 seconds for each one?
9 seconds sounds quite reasonable for generating 23000 or so files.
>> > Selective updating isn't going to help because 99.9% of the time is
>> > spent in the categories and it only takes a single port update to
>> > make a category file obsolete.
> This is the part I find troubling. It would seem that it should be
> more work to create an individual port README, with its plucking the
> appropriate line out of the INDEX-* file and then parsing it into its
> respective pieces and filling in a template, than to simply string
> together a list of references to a bunch of already built port READMEs
> into a category README.
>=20
> What am I not getting here?
No -- you're quite right. You could generate the category README.html
files entirely from the data in the INDEX. It's not quite as easy as
all that, because there aren't entries for each category separately, so
you'll have to parse the structure out of all of the paths in the INDEX.
>> > I think the way to speed this up is to have the script generate the
>> > category files too. There's no point in bringing in the top-level
>> > README since that's already fast.
> So what's making the category READMEs so slow then?
The big problem with performance in all this INDEX and README.html
building is that it takes quite a long time relatively to run make(1)
within any port or category directory. make(1) has to read in a lot of
other files and stat(2) many more[*] -- all of which involves a lot of
random-access disk IO, and that's always going to take quite a lot of
time. Now, doing 'make readme' in a category directory doesn't just run
make in that directory, but also in every port in that category.
Popular categories can contain many hundreds of ports.
Maybe I should add README.html generation to my FreeBSD::Portindex
stuff. Should be pretty simple -- all the necessary bits are readily
available and it is just a matter of formatting it as HTML and printing
it out.
Cheers,
Matthew
[*] Running 'make -dA' with maximum debug output is quite enlightening,
as is running make under truss(1)
--=20
Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
JID: matthew@infracaninophile.co.uk Kent, CT11 9PW
--------------enig100ACD4695C1313CE5922E24
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8kKAMACgkQ8Mjk52CukIyOZACgiwPKLRRKSQKo1twnBrEFo8Yg
lwYAni0yH6et5CwT4HQXSfXeHL5GSZxw
=H6T6
-----END PGP SIGNATURE-----
--------------enig100ACD4695C1313CE5922E24--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F2427FD.2000900>