From owner-freebsd-ports@FreeBSD.ORG Sun Jun 22 09:21:21 2008 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E0D8106566C for ; Sun, 22 Jun 2008 09:21:21 +0000 (UTC) (envelope-from Vladimir.Chukharev@gmail.com) Received: from wr-out-0708.google.com (wr-out-0708.google.com [64.233.184.250]) by mx1.freebsd.org (Postfix) with ESMTP id 3B6518FC12 for ; Sun, 22 Jun 2008 09:21:20 +0000 (UTC) (envelope-from Vladimir.Chukharev@gmail.com) Received: by wr-out-0708.google.com with SMTP id c2so2620564wra.13 for ; Sun, 22 Jun 2008 02:21:20 -0700 (PDT) MIME-Version: 1.0 Path: k30g2000hse.googlegroups.com!not-for-mail Newsgroups: muc.lists.freebsd.ports Date: Sun, 22 Jun 2008 02:10:55 -0700 (PDT) Complaints-To: groups-abuse@google.com Injection-Info: k30g2000hse.googlegroups.com; posting-host=91.155.183.162; posting-account=65Y6kwoAAADir_VJEsCeuzkf5lvseO_H NNTP-Posting-Host: 91.155.183.162 User-Agent: G2/1.0 X-HTTP-UserAgent: Opera/9.27 (X11; FreeBSD 7 i386; U; en),gzip(gfe),gzip(gfe) Message-ID: <2cb3f13f-6ddd-427b-89be-850c3568f961@k30g2000hse.googlegroups.com> From: Vladimir.Chukharev@gmail.com To: freebsd-ports@freebsd.org Cc: freebsd-ports@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: INDEX build optimizations X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jun 2008 09:21:21 -0000 Since this subject has been discussed recently, I want to tell my observations. I played a bit with INDEX_JOBS environment variable, and I found that for my notebook the best value is 8. The limiting factor seems to be grep'ing for libssl through packages. I believe this is easy to reproduce, probably with a higher value for INDEX_JOBS. Below is a part of ps axww output, and it shows more than 8 greps. I hope if cashing the result of this command can be implemented, it will increase speed of make index quite considerably. Best regard, Vladimir 20384 p1 IN+ 0:00,00 sh -ev 20385 p1 IN+ 0:00,02 make -B _lang.describe 20391 p1 IN+ 0:00,00 [sh] 20393 p1 IN+ 0:00,09 make -B describe DIRPRFX=lang/ 20405 p1 SN+ 0:00,11 [sh] 25830 p1 IN+ 0:00,00 sh -ev 25831 p1 IN+ 0:00,02 make -B _mail.describe 25838 p1 IN+ 0:00,00 [sh] 25839 p1 IN+ 0:00,22 make -B describe DIRPRFX=mail/ 25845 p1 SN+ 0:00,01 [sh] 33352 p1 IWN+ 0:00,00 sh /home/chu/chu/bin/cvsup-all.sh 34293 p1 IWN+ 0:00,00 time nice make index 34294 p1 IWN+ 0:00,00 make index 34311 p1 IWN+ 0:00,00 /bin/sh -ec cd /usr/ports && make /usr/ports/ INDEX-7 34312 p1 IWN+ 0:00,00 make /usr/ports/INDEX-7 34318 p1 IWN+ 0:00,00 (sh) 34320 p1 SN+ 0:00,08 make -j8 INDEX_TMPDIR=/tmp/index.1fgs6hrt BUILDING_INDEX=1 ECHO_MSG=echo > /dev/null describe 35531 p1 IN+ 0:00,00 sh -ev 35532 p1 IN+ 0:00,02 make -B _misc.describe 35539 p1 IN+ 0:00,00 [sh] 35540 p1 IN+ 0:00,15 make -B describe DIRPRFX=misc/ 35551 p1 IN+ 0:00,12 [sh] 39146 p1 IN+ 0:00,00 sh -ev 39149 p1 IN+ 0:00,02 make -B _multimedia.describe 39213 p1 IN+ 0:00,00 [sh] 39215 p1 IN+ 0:00,07 make -B describe DIRPRFX=multimedia/ 39232 p1 IN+ 0:00,03 [sh] 40733 p1 IWN+ 0:00,00 sh -ev 40734 p1 IWN+ 0:00,00 make -B _databases.describe 40742 p1 IWN+ 0:00,00 (sh) 40743 p1 IWN+ 0:00,00 make -B describe DIRPRFX=databases/ 40772 p1 SN+ 0:00,16 [sh] 41039 p1 IN+ 0:00,00 make -B describe 41042 p1 IN+ 0:00,00 sh -c cd /usr/ports/devel/omniNotify/../ omniORB && make -V WRKSRC 41044 p1 IN+ 0:00,01 make -V WRKSRC 41051 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 41052 p1 DN+ 0:00,43 grep -l -r ^lib/libssl.so. /var/db/pkg 41053 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 41520 p1 IN+ 0:00,01 make -B describe 41526 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 41527 p1 DN+ 0:00,38 grep -l -r ^lib/libssl.so. /var/db/pkg 41528 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42412 p1 IN+ 0:00,01 make -B describe 42414 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42415 p1 DN+ 0:00,38 grep -l -r ^lib/libssl.so. /var/db/pkg 42416 p1 IN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42548 p1 SN+ 0:00,01 make -B describe 42555 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42560 p1 DN+ 0:00,05 grep -l -r ^lib/libssl.so. /var/db/pkg 42561 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42589 p1 SN+ 0:00,01 make -B describe 42599 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42600 p1 DN+ 0:00,05 grep -l -r ^lib/libssl.so. /var/db/pkg 42601 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42606 p1 SN+ 0:00,01 make -B describe 42611 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42613 p1 DN+ 0:00,05 grep -l -r ^lib/libssl.so. /var/db/pkg 42614 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42623 p1 SN+ 0:00,01 make -B describe 42626 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42627 p1 DN+ 0:00,05 grep -l -r ^lib/libssl.so. /var/db/pkg 42628 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42673 p1 SN+ 0:00,01 make -B describe 42675 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 42676 p1 DN+ 0:00,05 grep -l -r ^lib/libssl.so. /var/db/pkg 42677 p1 SN+ 0:00,00 sh -c grep -l -r "^lib/libssl.so." "/var/db/ pkg" | while read contents; do sslprefix=`grep "^@cwd " "$ {contents}" | /usr/bin/head -n 1`; if test "${sslprefix}" = "@cwd / usr/local" ; then echo "${contents}"; break; fi; done 54143 p1 IN+ 0:00,00 sh -ev 54144 p1 IN+ 0:00,02 make -B _devel.describe 54276 p1 IN+ 0:00,00 [sh] 54277 p1 IN+ 0:01,37 make -B describe DIRPRFX=devel/ 54361 p1 IN+ 0:00,35 [sh] 71092 p1 IN+ 0:00,00 sh -ev 71094 p1 IN+ 0:00,02 make -B _ftp.describe 71241 p1 IN+ 0:00,00 [sh] 71242 p1 IN+ 0:00,03 make -B describe DIRPRFX=ftp/ 71261 p1 SN+ 0:00,02 [sh] 76023 p1 IN+ 0:00,00 sh -ev 76024 p1 IN+ 0:00,02 make -B _irc.describe 76033 p1 IN+ 0:00,00 [sh] 76035 p1 IN+ 0:00,04 make -B describe DIRPRFX=irc/ 76080 p1 SN+ 0:00,02 [sh]