From owner-freebsd-ports@FreeBSD.ORG Sat Jan 28 16:53:32 2012 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75A2F106564A for ; Sat, 28 Jan 2012 16:53:32 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id C613A8FC0C for ; Sat, 28 Jan 2012 16:53:31 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [IPv6:2001:8b0:151:1:fa1e:dfff:feda:c0bb]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id q0SGrNe7037777 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Sat, 28 Jan 2012 16:53:23 GMT (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.4.3 smtp.infracaninophile.co.uk q0SGrNe7037777 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1327769603; bh=kyyKx84DOFWkT4C2hVeMqDyOkYLmo0oyF55MXvqktX4=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Cc; b=qwD5RLG9upjr/wQCZkre8CMDokRfkCOC3mL0GJ30tPNnxRawvkFt75x42Oqw+pqs5 dIt/XQ26t4Q13RlJ3NsWT7EcJY/DQ9JRwLSQn3dceDdpfT/MrpCArwiF8RfZHBpwmo 01TkCUNDikHPomu9PlNK4BCd1pdGTL1NUBZQdwIU= Message-ID: <4F2427FD.2000900@infracaninophile.co.uk> Date: Sat, 28 Jan 2012 16:53:17 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: freebsd-ports@freebsd.org References: <20111203173149.224a64e6@cox.net> <20111214004838.GK1593@dormouse.experts-exchange.com> <20120112212905.GA78819@dormouse.experts-exchange.com> <20120127200325.66f36090@cox.net> <20120128143734.6166f5fc@gumby.homeunix.com> <20120128102831.51cf0f63@cox.net> In-Reply-To: <20120128102831.51cf0f63@cox.net> X-Enigmail-Version: 1.3.4 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig100ACD4695C1313CE5922E24" X-Virus-Scanned: clamav-milter 0.97.3 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.7 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: A new and better way to do "make readmes"? X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jan 2012 16:53:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig100ACD4695C1313CE5922E24 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 28/01/2012 16:28, Conrad J. Sabatier wrote: > Am I understanding you correctly? Are you= > saying you built 20,000+ port READMEs in only 9 seconds?! How is that > possible? Or do you mean 9 seconds for each one? 9 seconds sounds quite reasonable for generating 23000 or so files. >> > Selective updating isn't going to help because 99.9% of the time is >> > spent in the categories and it only takes a single port update to >> > make a category file obsolete. > This is the part I find troubling. It would seem that it should be > more work to create an individual port README, with its plucking the > appropriate line out of the INDEX-* file and then parsing it into its > respective pieces and filling in a template, than to simply string > together a list of references to a bunch of already built port READMEs > into a category README. >=20 > What am I not getting here? No -- you're quite right. You could generate the category README.html files entirely from the data in the INDEX. It's not quite as easy as all that, because there aren't entries for each category separately, so you'll have to parse the structure out of all of the paths in the INDEX. >> > I think the way to speed this up is to have the script generate the >> > category files too. There's no point in bringing in the top-level >> > README since that's already fast. > So what's making the category READMEs so slow then? The big problem with performance in all this INDEX and README.html building is that it takes quite a long time relatively to run make(1) within any port or category directory. make(1) has to read in a lot of other files and stat(2) many more[*] -- all of which involves a lot of random-access disk IO, and that's always going to take quite a lot of time. Now, doing 'make readme' in a category directory doesn't just run make in that directory, but also in every port in that category. Popular categories can contain many hundreds of ports. Maybe I should add README.html generation to my FreeBSD::Portindex stuff. Should be pretty simple -- all the necessary bits are readily available and it is just a matter of formatting it as HTML and printing it out. Cheers, Matthew [*] Running 'make -dA' with maximum debug output is quite enlightening, as is running make under truss(1) --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enig100ACD4695C1313CE5922E24 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8kKAMACgkQ8Mjk52CukIyOZACgiwPKLRRKSQKo1twnBrEFo8Yg lwYAni0yH6et5CwT4HQXSfXeHL5GSZxw =H6T6 -----END PGP SIGNATURE----- --------------enig100ACD4695C1313CE5922E24--