Date: Mon, 28 May 2007 00:15:28 +0200 From: Michel Talon <talon@lpthe.jussieu.fr> To: freebsd-hackers@freebsd.org Cc: ports@freebsd.org Subject: Re: Looking for speed increases in "make index" Message-ID: <20070527221528.GA19603@lpthe.jussieu.fr>
next in thread | raw e-mail | index | archive | help
Stephen Montgomery-Smith said: > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. .... > or "I have tried to do this." Of course a lot of people have thinked about it, and quickly realized that it was not going to work. In the bsd.ports.mk, evaluation of one variable may be dependent on some conditional, which may itself be dependant on some other variable, appearing as some target. This constantly happens in bsd.ports.mk. If you think about that, you convince yourself that a "reduced make" needs to understand targets, needs to understand conditionals, and needs to evaluate not only the variable under consideration, but but possibly any other. In other words, you need a full blown make. To gain some performance, a first idea would be to simplify bsd.ports.mk. I am convinced that a substantial part of the 4000 lines are historical crap which serve no useful purpose. There are tons of variables who have probably purely anecdotical interest. There are targets which could be happily suppressed. Of course, due to the complexity of bsd.ports.mk, rewriting it is certainly not an easy task. There is a freebsd port whose aim is to rewrite it, i don't know how far they are. The NetBSD people have completely rewritten the system, i have no idea if the new one is faster. The OpenBSD people have also rewritten bsd.ports.mk, with apparently much faster makefile. A second idea would be to multithread make, since modern machines will have a lot of cores. At present make -j <something> forks submakes and waits for their completion. This doesn't help for the problem at hand. However, multithreading the execution of a single makefile is certainly not trivial due to the interdependencies. Anyways, one of the things which cannot be a big factor is reading bsd.ports.mk from hard disk. It is certainly cached in memory when you run make index. On the other hand it is so big that it probably doesn't fit in cache, or probably only fits in caches of machines generously endowed. I have remarked that the difference of execution speed of make index between machines of similar speed but very different cache size (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less than 10 minutes on the big cache machine versus 30 minutes on the small cache one. If the cache effect is indeed dominant, then reducing the size of bsd.ports.mk by all possible means would be very beneficial. By the way an alternative system would be to use something other than make to do the job. This is what the Gentoo people are doing, using first a python based system, and now a C++ one (paludis). Here also i don't have any idea if it is faster. -- Michel TALON
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070527221528.GA19603>