From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 22:46:30 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D9D5A16A524 for ; Mon, 28 May 2007 22:46:30 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 7CA4F13C4B7 for ; Mon, 28 May 2007 22:46:30 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 27875 invoked by uid 1001); 28 May 2007 22:45:45 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Mon, 28 May 2007 18:45:44 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18011.23448.513335.199513@bhuda.mired.org> Date: Mon, 28 May 2007 18:45:44 -0400 To: Hartmut Brandt In-Reply-To: <465B4A84.6050407@dlr.de> References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> <465B4A84.6050407@dlr.de> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 22:46:31 -0000 In <465B4A84.6050407@dlr.de>, Hartmut Brandt typed: > Mike Meyer wrote: > > In <465AB421.10802@dlr.de>, Hartmut Brandt typed: > >> 1. make and its sub-makes for a) reading the file; b) parsing the file > >> (note that .if and .for processing is done while parsing); c) processing > >> targets. > > > > Make and submakes have been gone over already. See > http://miller.emu.id.au/pmiller/books/rmch/ >. > > > > I'm not sure it can be applied to the ports tree, though. I haven't > > looked into it, but recalled this paper when you mentioned measuring > > makes and sub-makes. > Unfortunately you deleted the sentence before, so I rephrase it: before > looking into optimizations find out where the time is actually spend - > how many seconds of the hours the process takes, are actually spent in > make and sub-makes. If the entire process takes 2 hours of which the > makes take 20 seconds then by enhancing performance of make by 50% you > win 10 seconds. This is probably not worth a single line of additional code. > > The paper you point to talks about something entirely different. It think we're talking about two different things. You're talking about the efficiency of make, whereas he's talking about the efficiency of make. Um, wait. You're talking about what I'll call the *internal* efficiency of make, defined as how fast it does the things it does. He's talking about what I'll call the *external* efficiency of make, which is how well it does at doing the minimum amount of work it needs to do. I hope you can see where the confusion comes from. In particular, he talks about how recursive makefiles screw up evaluating complex variables, causing them to be executed multiple times. So if you're running a makefile to pull some variables value, as opposed to do real commands, and your entire process takes 2 hours and the Makefile takes 20 seconds, but it evaluates all the variables twice, then by fixing your makefile you win at least 59 minutes and 50 seconds. I think cutting the run time by 50% is worth some work. Benchmarking can help you decide which things it pays to work on if all you're worried about is the internal efficiency. However, the goal is to make the process faster, so we need to worry about the external efficiency as well. The problem here is that the worse it is, the less it looks like you stand to gain by looking at your makefile when you look at the benchmarks. Given that the ports system has both highly complex variables and is very recursive, I believe that it warrants investigation if you're going to work on making make in the ports faster. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information.