Date: Mon, 28 May 2007 12:51:13 +0200 From: Hartmut Brandt <hartmut.brandt@dlr.de> To: Stephen Montgomery-Smith <stephen@math.missouri.edu> Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports Message-ID: <465AB421.10802@dlr.de> In-Reply-To: <4659EF80.70100@math.missouri.edu> References: <4659EF80.70100@math.missouri.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. Now > "make -V PKGNAME" should be a speedy operation, but the make has to load > in and analyze bsd.port.mk, a quite complicated file with about 200,000 > characters in it, when all it is needing to do is to figure out the > value of the variable PKGNAME. > > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. So, for example, if all we need to know is > PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, > unless the writer of that particular port has done something like having > PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should > analyze all the code it is given, and only figure it out if it is needed > to do so. This would include, for example, figuring out .for and .if > directives on a need to know basis as well. > > I have only poked around a little inside the source for make, but I have > a sense that this would be a major undertaking. I certainly have not > thought through what it entails in more than a cursory manner. However > I am quite excited about the possibility of doing this, albeit I may > well put off the whole thing for a year or two or even forever depending > upon other priorities in my life. > > However, in the mean time I want to throw this idea out there to get > some feedback, either of the form of "this won't work," or of the form > "I will do it," or "I have tried to do this." Having done a great deal of rewriting of make some two years ago I can tell you that even a small change to make is a tough job testing-wise: run all the combinations of !-j and -j <N> on all architectures and run the change through the port-building cluster. That's a warning to start with. Second I would start with careful profiling to find out where the problem actually is. You might be surprised. As an example: several times the idea came up to use a hash structure instead of linear lists for make variables. I got a patch for this and - it makes absolutely no difference performance-wise (well, there was some indication that performance gets worse, but that was around or below noise level). With careful I mean to find out who takes the time: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. 2. sub-shells executed for executing targets commands (note, that make optimizes the subshells away when there are no special shell symbol in the command line) 3. executed programs (find, sort, ...) Until you have numbers for this everything is rather moot. It might be a good idea to put some performance measurement hooks into make for this to do. If anybody wants to work on make, I would rather recommend to implement %-rules :-) And if anybody wants to recommend gmake over make(1) - look into the code, what mess that is :-/ Regards, harti
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?465AB421.10802>