From owner-freebsd-ports@FreeBSD.ORG Sun May 13 13:44:00 2007 Return-Path: X-Original-To: freebsd-ports@freebsd.org Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65AD916A404 for ; Sun, 13 May 2007 13:44:00 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id 2D9DF13C44C for ; Sun, 13 May 2007 13:44:00 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070513134357m9100rpfpse>; Sun, 13 May 2007 13:43:58 +0000 Message-ID: <4647161D.2040100@math.missouri.edu> Date: Sun, 13 May 2007 08:43:57 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070505 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Kris Kennaway References: <464597C6.3030406@gmx.de> <20070512174011.GA22526@xor.obsecurity.org> <4645FF71.60100@gmx.de> <20070512175824.GA23103@xor.obsecurity.org> <20070512133054.B5588@math.missouri.edu> <4646193E.5040503@gmx.de> <464665D5.1090509@math.missouri.edu> <4646AC48.1010107@gmx.de> <4646CFE3.1020105@gmx.de> <20070513084931.GA35696@xor.obsecurity.org> In-Reply-To: <20070513084931.GA35696@xor.obsecurity.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "\[LoN\]Kamikaze" , freebsd-ports@freebsd.org Subject: Re: Time to abandon recursive pulling of dependencies? X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 May 2007 13:44:00 -0000 Kris Kennaway wrote: > On Sun, May 13, 2007 at 10:44:19AM +0200, [LoN]Kamikaze wrote: >> [LoN]Kamikaze wrote: >>> Stephen Montgomery-Smith wrote: >>>> OK chaps, this is what I came up with. So for example, if I do "make >>>> install" on /usr/ports/x11/xorg (having made all the dependencies), on >>>> my computer it turns the pkg_create from taking about 4 minutes to the >>>> blink of an eye. Now people need to figure out how to speed up the >>>> "make package-depends" in bsd.ports.mk, but that is beyond my abilities. >>> I will give that a try, I've been experimenting with make a lot lately. Thank >>> you for your patch, the second attempt seems to work fine on my system. >> So far I've got a 20% performance increase, not really satisfying. The trouble >> is that for every dependency the whole bsd.port.mk and whatever else is pulled >> in by the dependency has to be parsed again. So the only way to make a >> significant improvement is to have a short bsd.port.mk. > > Again, what evidence do you have that *that* is really the problem? > Please try to be more scientific in your approach, it will help all of > us to figure this out. > > Kris I looked inside the bsd.ports.mk, and put in a few ${ECHO_MSG} statements here and there. I also put a few printf statements in pkg_create. I found out that there are two parts that take a long time: 1. Pulling in the dependencies. This is in effect doing a "make package-depends". You can do this for yourself, and see that it takes a long time. I honestly don't see how to make this faster, as presumably it involves calling "make" on all the dependency ports, and I would guess that each new invocation of make adds considerable overhead just loading the program. 2. Sorting the dependencies in pkg_create. My fix now cuts this out completely. Thus there is no need to change the structure of /var/db/pkg to use databases or anything like this. Stephen