Date: Sun, 27 May 2007 15:30:48 -0700 From: Jeremy Chadwick <koitsu@FreeBSD.org> To: Stephen Montgomery-Smith <stephen@math.missouri.edu> Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports Message-ID: <20070527223048.GA37505@icarus.home.lan> In-Reply-To: <4659EF80.70100@math.missouri.edu> References: <4659EF80.70100@math.missouri.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
> I have been thinking a lot about looking for speed increases for "make
> index" and pkg_version and things like that. So for example, in
> pkg_version, it calls "make -V PKGNAME" for every installed package. Now
> "make -V PKGNAME" should be a speedy operation, but the make has to load in
> and analyze bsd.port.mk, a quite complicated file with about 200,000
> characters in it, when all it is needing to do is to figure out the value of
> the variable PKGNAME.
I have a related question, pertaining to "make all-depends-list" and the
utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't
know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
bsd.ports.mk.
I call it an atrocity because it's a mix of make variable expansion
combined with sh scripting, and it's nearly impossible to read. It's
not commented either, so folks like myself are left thinking "What IS
this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
throughout several places in bsd.port.mk.
I do not entirely understand what ALL-DEPENDS-LIST is about (that should
be apparent), but upon performing some of my benchmarks, I found this to
be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly
fast, but ALL-DEPENDS-LIST is not.
Does it need to be done this way? Can we just iterate through all of
the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I
suppose that depends on the operation (make vs. make clean vs.
others)...
The port I used for testing some of the benchmarks was net/gacxtool. It
seems to be a good example of a "hefty" port.
> I suggest rewriting "make" so that variables are only evaluated on a "need
> to know" basis. So, for example, if all we need to know is PKGNAME, there
> is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
> that particular port has done something like having PORTNAME depend on the
> value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is
> given, and only figure it out if it is needed to do so. This would include,
> for example, figuring out .for and .if directives on a need to know basis as
> well.
This sounds like a good solution. In fact, I'm lead to believe that
heavy reliance on /bin/sh is part of why the ports collection is slow.
No, it's not the sole reason, but it's one of many. I'm of the belief
that anything we can do to migrate portions into native make would be
benefitial.
That said, I'll ask this out in the open: am I the only one who sees the
benefit of GNU make in regards to this? There's a lot of built-in
functions in GNU make which could help in regards to ports. I have no
qualms with PMake per se, but if another tool gives us what we need,
then maybe we should consider the pros and cons of adapting that.
There's also CMake, which is incredibly fast.
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070527223048.GA37505>