Date: Sun, 27 May 2007 20:17:44 -0500 (CDT) From: Stephen Montgomery-Smith <stephen@math.missouri.edu> To: Michel Talon <talon@lpthe.jussieu.fr> Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" Message-ID: <20070527200954.C11340@math.missouri.edu> In-Reply-To: <20070527221528.GA19603@lpthe.jussieu.fr> References: <20070527221528.GA19603@lpthe.jussieu.fr>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 28 May 2007, Michel Talon wrote: > Stephen Montgomery-Smith said: > >> I suggest rewriting "make" so that variables are only evaluated on a >> "need to know" basis. .... >> or "I have tried to do this." > > Of course a lot of people have thinked about it, and quickly realized > that it was not going to work. In the bsd.ports.mk, evaluation of one > variable may be dependent on some conditional, which may itself be > dependant on some other variable, appearing as some target. This > constantly happens in bsd.ports.mk. > > If you think about that, you convince yourself that a "reduced make" > needs to understand targets, needs to understand conditionals, and needs > to evaluate not only the variable under consideration, but but possibly > any other. In other words, you need a full blown make. Actually I have thought quite a lot about this over the last couple of weeks. You are correct in that the "make" really does have to be a full blown make. What I am suggesting is something rather different - a rather sophisticated make that works hard to only that which it really has to do. And it certainly would require a lot of sophistication, precisely for the reasons you state. > > To gain some performance, a first idea would be to simplify > bsd.ports.mk. I am convinced that a substantial part of the 4000 lines > are historical crap which serve no useful purpose. There are tons of > variables who have probably purely anecdotical interest. There are > targets which could be happily suppressed. Of course, due to the > complexity of bsd.ports.mk, rewriting it is certainly not an easy task. > There is a freebsd port whose aim is to rewrite it, i don't know how > far they are. The NetBSD people have completely rewritten the system, i > have no idea if the new one is faster. The OpenBSD people have also > rewritten bsd.ports.mk, with apparently much faster makefile. > I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did was evalute variables. (I wrote a perl script that did this automatically.) The net effect was no speed gain. In other words, processing unnecessary targets is not the problem. I really have looked at bsd.ports.mk a lot, and given that it is written in the context of what make requires, it seems to be rather well and efficiently written. I did find one speed increase in bsd.gnome.mk, which I am kind of proud of, but even that only gained about a 5-10% speed increase in "make index." > A second idea would be to multithread make, since modern machines will > have a lot of cores. At present make -j <something> forks submakes > and waits for their completion. This doesn't help for the problem at > hand. However, multithreading the execution of a single makefile is > certainly not trivial due to the interdependencies. > I don't think multithreaded make will help in this situation. > Anyways, one of the things which cannot be a big factor is reading > bsd.ports.mk from hard disk. It is certainly cached in memory when you > run make index. On the other hand it is so big that it probably doesn't > fit in cache, or probably only fits in caches of machines generously > endowed. I have remarked that the difference of execution speed of make > index between machines of similar speed but very different cache size > (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less > than 10 minutes on the big cache machine versus 30 minutes on the small > cache one. If the cache effect is indeed dominant, then reducing the > size of bsd.ports.mk by all possible means would be very beneficial. > My suspicion is that the timne taken to read bsd.ports.mk from the hard disk is a rather small part of what takes it so long. Compare doing a "make" with a simple "cat" and the time differences are very great. I think make spends far more time processing the file than reading it. > By the way an alternative system would be to use something other than > make to do the job. This is what the Gentoo people are doing, using > first a python based system, and now a C++ one (paludis). Here also i > don't have any idea if it is faster. > It would be nice to find a solution that doesn't involve redesigning the whole ports process from scratch. > > > > -- > > Michel TALON > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070527200954.C11340>