From owner-freebsd-ports@FreeBSD.ORG Mon May 28 01:17:45 2007 Return-Path: X-Original-To: ports@freebsd.org Delivered-To: freebsd-ports@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DB8BD16A400 for ; Mon, 28 May 2007 01:17:45 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (math.missouri.edu [128.206.184.200]) by mx1.freebsd.org (Postfix) with ESMTP id A61A413C44C for ; Mon, 28 May 2007 01:17:45 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (localhost [127.0.0.1]) by math.missouri.edu (8.13.1/8.13.1) with ESMTP id l4S1HjOn011712; Sun, 27 May 2007 20:17:45 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Received: from localhost (stephen@localhost) by math.missouri.edu (8.13.1/8.13.1/Submit) with ESMTP id l4S1HjgN011709; Sun, 27 May 2007 20:17:45 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Date: Sun, 27 May 2007 20:17:44 -0500 (CDT) From: Stephen Montgomery-Smith To: Michel Talon In-Reply-To: <20070527221528.GA19603@lpthe.jussieu.fr> Message-ID: <20070527200954.C11340@math.missouri.edu> References: <20070527221528.GA19603@lpthe.jussieu.fr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 01:17:45 -0000 On Mon, 28 May 2007, Michel Talon wrote: > Stephen Montgomery-Smith said: > >> I suggest rewriting "make" so that variables are only evaluated on a >> "need to know" basis. .... >> or "I have tried to do this." > > Of course a lot of people have thinked about it, and quickly realized > that it was not going to work. In the bsd.ports.mk, evaluation of one > variable may be dependent on some conditional, which may itself be > dependant on some other variable, appearing as some target. This > constantly happens in bsd.ports.mk. > > If you think about that, you convince yourself that a "reduced make" > needs to understand targets, needs to understand conditionals, and needs > to evaluate not only the variable under consideration, but but possibly > any other. In other words, you need a full blown make. Actually I have thought quite a lot about this over the last couple of weeks. You are correct in that the "make" really does have to be a full blown make. What I am suggesting is something rather different - a rather sophisticated make that works hard to only that which it really has to do. And it certainly would require a lot of sophistication, precisely for the reasons you state. > > To gain some performance, a first idea would be to simplify > bsd.ports.mk. I am convinced that a substantial part of the 4000 lines > are historical crap which serve no useful purpose. There are tons of > variables who have probably purely anecdotical interest. There are > targets which could be happily suppressed. Of course, due to the > complexity of bsd.ports.mk, rewriting it is certainly not an easy task. > There is a freebsd port whose aim is to rewrite it, i don't know how > far they are. The NetBSD people have completely rewritten the system, i > have no idea if the new one is faster. The OpenBSD people have also > rewritten bsd.ports.mk, with apparently much faster makefile. > I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did was evalute variables. (I wrote a perl script that did this automatically.) The net effect was no speed gain. In other words, processing unnecessary targets is not the problem. I really have looked at bsd.ports.mk a lot, and given that it is written in the context of what make requires, it seems to be rather well and efficiently written. I did find one speed increase in bsd.gnome.mk, which I am kind of proud of, but even that only gained about a 5-10% speed increase in "make index." > A second idea would be to multithread make, since modern machines will > have a lot of cores. At present make -j forks submakes > and waits for their completion. This doesn't help for the problem at > hand. However, multithreading the execution of a single makefile is > certainly not trivial due to the interdependencies. > I don't think multithreaded make will help in this situation. > Anyways, one of the things which cannot be a big factor is reading > bsd.ports.mk from hard disk. It is certainly cached in memory when you > run make index. On the other hand it is so big that it probably doesn't > fit in cache, or probably only fits in caches of machines generously > endowed. I have remarked that the difference of execution speed of make > index between machines of similar speed but very different cache size > (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less > than 10 minutes on the big cache machine versus 30 minutes on the small > cache one. If the cache effect is indeed dominant, then reducing the > size of bsd.ports.mk by all possible means would be very beneficial. > My suspicion is that the timne taken to read bsd.ports.mk from the hard disk is a rather small part of what takes it so long. Compare doing a "make" with a simple "cat" and the time differences are very great. I think make spends far more time processing the file than reading it. > By the way an alternative system would be to use something other than > make to do the job. This is what the Gentoo people are doing, using > first a python based system, and now a C++ one (paludis). Here also i > don't have any idea if it is faster. > It would be nice to find a solution that doesn't involve redesigning the whole ports process from scratch. > > > > -- > > Michel TALON > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >