From owner-freebsd-ports@FreeBSD.ORG Sat May 19 23:43:26 2007 Return-Path: X-Original-To: freebsd-ports@freebsd.org Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A5AA16A400 for ; Sat, 19 May 2007 23:43:26 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout1.cac.washington.edu (mxout1.cac.washington.edu [140.142.32.134]) by mx1.freebsd.org (Postfix) with ESMTP id E308D13C4B9 for ; Sat, 19 May 2007 23:43:25 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.7] (may be forged)) by mxout1.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4JNhPgs032449 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Sat, 19 May 2007 16:43:25 -0700 X-Auth-Received: from [192.168.10.45] (c-67-166-149-71.hsd1.ca.comcast.net [67.166.149.71]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4JNhOdh023832 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sat, 19 May 2007 16:43:24 -0700 Message-ID: <464F8B9C.9050607@u.washington.edu> Date: Sat, 19 May 2007 16:43:24 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: freebsd-ports@freebsd.org References: <464597C6.3030406@gmx.de> <20070512174011.GA22526@xor.obsecurity.org> <4645FF71.60100@gmx.de> <20070512175824.GA23103@xor.obsecurity.org> <20070512133054.B5588@math.missouri.edu> <4646193E.5040503@gmx.de> <464665D5.1090509@math.missouri.edu> <464676CC.2000203@math.missouri.edu> <46469BF5.9030603@math.missouri.edu> <1179058696.13237.3.camel@ikaros.oook.cz> <20070513101332.L82202@math.missouri.edu> <46479215.1080401@math.missouri.edu> <4647C4FB.9050607@math.missouri.edu> <4647F182.6000402@gmx.de> <46487491.6020806@math.missouri.edu> <46488168.3030300@math.missouri.edu> <464B155A.1080105@math.missouri.edu> <464B7144.2000600@math.missouri.edu> <1179353641.79329.11.camel@rnoland-ibm.acs.internap.com> <20070517114436.3fdd55b1@deskjail> <464C5338.2090300@math.missouri.edu> In-Reply-To: <464C5338.2090300@math.missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.19.162634 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Subject: Re: Time to abandon recursive pulling of dependencies? X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 May 2007 23:43:26 -0000 Stephen Montgomery-Smith wrote: > Alexander Leidinger wrote: >> Quoting Robert Noland (Wed, 16 May 2007 18:14:01 >> -0400): >> >>> On Wed, 2007-05-16 at 16:01 -0500, Stephen Montgomery-Smith wrote: >>>> Ok chaps, I think I have it. >>>> >>>> This involves no recursive calls of make. Furthermore the >>>> dependencies it creates are the real dependencies on your system, >>>> not what ports thinks it should be, because it gets all the >>>> information from /var/db/pkg. On my system it takes a second or two >>>> to register a port >>>> - it takes about the same amount of time whether it has few >>>> dependencies or many, except when there are zero dependencies, when >>>> it takes no >>>> time at all. >>>> >>>> If I get some positive feedback on this one, I'll submit a PR. You >>>> can try it out by typing "make actual-package-depends" as opposed to >>>> "make package-depends." >>> Ok, I'm really trying to make sense of these numbers... >>> >>> For gnome2-2.18.1_1 on my -current system... >>> >>> rnoland-ibm% make package-depends|wc -l >>> 362 >>> rnoland-ibm% make actual-package-depends|wc -l >>> 294 >>> >>> Registered the old way: >>> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l >>> 176 >>> >>> and the new way: >>> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l >>> 294 >>> >>> I put together an awk script yesterday which was very close to the one >>> Alexander posted and got 295... and the diff of the output of my awk and >>> actual-package-depends is much more than a single line, which I also >>> can't yet explain. >> >> I had a look at it. It is missing those dependencies which are not >> installed. Depending on when this target is used, this is ok or not. >> >> Some benchmark: >> ---snip--- >> [original] >> make package-depends 80.18s user 53.92s system 73% cpu 3:01.53 total >> make package-depends 80.70s user 52.94s system 87% cpu 2:32.25 total >> make package-depends > /tmp/list1 79.58s user 54.28s system 87% cpu >> 2:32.59 total >> >> [no AWK] >> make actual-package-depends 4.76s user 7.46s system 57% cpu 21.082 total >> make actual-package-depends 4.87s user 7.34s system 67% cpu 18.019 total >> make actual-package-depends > /tmp/list2 4.58s user 7.60s system 87% >> cpu 13.942 total >> >> [AWK] >> make actual-package-depends 3.15s user 7.45s system 85% cpu 12.420 total >> make actual-package-depends 3.13s user 7.49s system 84% cpu 12.524 total >> make actual-package-depends 3.11s user 7.49s system 88% cpu 11.995 total >> make actual-package-depends >| /tmp/list3 3.16s user 7.42s system 89% >> cpu 11.83 0 total >> >> [AWK + partly missing (only direct dependencies)] >> make actual-package-depends > /tmp/list4 4.60s user 7.63s system 90% >> cpu 13.479 total >> make actual-package-depends 4.68s user 7.56s system 87% cpu 13.985 total >> make actual-package-depends 4.58s user 7.64s system 89% cpu 13.594 total >> >> ...sorting list1 & list2 & list3 & list4... >> >> % wc -l /tmp/list*s >> 320 /tmp/list1s >> 308 /tmp/list2s >> 308 /tmp/list3s >> 310 /tmp/list4s >> 1246 total >> ---snip--- >> >> Note, there was trackerd running in the background while testing... >> >> For the difference between the redirected output case: I think the >> gnome terminal needs a lot of time to print all the lines. But still, >> the awk version takes around 3/4 of the time (interesting is the user >> time, not the total time). Stephen's version can be speed up some >> fractions by inserting a break into the first while-loop. >> >> Attached is my awk-version and the awk version which also includes the >> direct dependencies. >> >> Bye, >> Alexander. >> >> > > I think I like Alexander's awk version better than mine. Alexander - > can I ask you to submit the PR using send-pr? > > Stephen My lord.. now I see what everyone means in terms of taking a long time to update the ports / package databases. If you use portsnap, it doesn't take a long time. However, if you use csup/cvsup, it appears to take a long time running make and ruby (primarily disk I/O), to update all of the databases. It took approximately 3-5 minutes to complete on a P4 3.0 GHz, so I'd hate to see how long it takes to finish on a slower machine =\... Maybe this is where Stephen's revisions to pkg_add and bsd.ports.mk would come in handy. -Garrett