Date: Mon, 31 Mar 2008 10:36:43 +0200 From: Roman Divacky <rdivacky@FreeBSD.org> To: soralx@cydem.org Cc: pav@FreeBSD.org, freebsd-ports@FreeBSD.org Subject: Re: [patch] pkg_delete(1) speedup Message-ID: <20080331083643.GA39137@freebsd.org> In-Reply-To: <20080330234946.7fec88bf@soralx> References: <1206548309.78560.69.camel@pav.hide.vol.cz> <20080326201416.GA34807@freebsd.org> <20080330234946.7fec88bf@soralx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Mar 30, 2008 at 11:49:46PM -0700, soralx@cydem.org wrote: > > > > You might have noticed a thread on the mailing list called "ports system > > > woes". The submitter pointed out an inefficiency in pkg_delete routine, > > > that parses the whole /var/db/pkg over and over again for every > > > dependency of a package being removed. > > > > > > Attached is a patch by rdivacky that implements the idea of looking up > > > all the values in a single pass over /var/db/pkg content. > > > > I hacked a slightly better patch that coveres a part of pkg_add too.. > > > > please review/test on: > > > > www.vlakno.cz/~rdivacky/pkg_tools.patch > > > > comments, benchmarks more than welcome! > > All right, I've been gone to the Real World for a while, but I returned %-) > > First, allow me to note that it's rather impressing to see the level of > interest and responses my half-baked email about my little digs into pkg_* > tools produced. Before I even finished thinking whether I will have enough > time to fix the tools proper, patches started appearing on the horizon (the > same day, practically)! This is quite reassuring, as it shows that > developers still care about code and algorithm quality, even if things work > OK on modern hardware (just lack of developer time, that's all, I suppose). > For that I'm grateful -- way to go :) you provided excellent analysis... that couldnt go unpunished ;) > Now, here are the same tests on the same hardware, but > with pkg_tools.patch applied: I updated the patch in place several times.. please make sure you have the latest version or bad things can happen :( I'll respond to the rest of the mail later... thnx for caring about this! roman > [root@freen0de /usr/ports/x11/rxvt-unicode]# make > [root@freen0de /usr/ports/x11/rxvt-unicode]# time make install > <SNIP> > ===> Generating temporary packing list > ===> Checking if x11/rxvt-unicode already installed > load: 0.53 cmd: pkg_info 25799 [biord] 0.06u 0.07s 0% 1532k > /usr/bin/install -c -o root -g wheel -d /usr/local/bin > <SNIP> > ===> Documentation installed in /usr/local/share/doc/rxvt-unicode. > ===> Compressing manual pages for rxvt-unicode-9.02_1 > ===> Registering installation for rxvt-unicode-9.02_1 > load: 0.29 cmd: sed 26266 [biord] 0.00u 0.00s 0% 728k > load: 0.27 cmd: sh 26568 [runnable] 0.00u 0.00s 0% 164k > load: 0.24 cmd: sh 25951 [biord] 0.14u 0.09s 0% 1228k > load: 0.22 cmd: grep 27026 [runnable] 0.00u 0.00s 0% 256k > ===> SECURITY REPORT: > This port has installed the following binaries which execute with > <SNIP> > real 1m13.885s > user 0m3.903s > sys 0m4.870s > [root@freen0de /usr/ports/x11/rxvt-unicode]# s; sleep 300 && echo -e "<Several memory-intensive jobs performed to clean buffer>\n" > <Several memory-intensive jobs performed to clean buffer> > > [root@freen0de /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ > load: 0.36 cmd: pkg_delete 27480 [biord] 0.35u 0.40s 1% 972k > > real 0m37.218s > user 0m0.448s > sys 0m0.509s > [root@freen0de /usr/ports/x11/rxvt-unicode]# make reinstall > /dev/null; sync > [root@freen0de /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ > > real 0m20.261s > user 0m0.349s > sys 0m0.476s > [root@freen0de /usr/ports/x11/rxvt-unicode]# > > So, the time drops from over 7 minutes to 20 seconds -- sweet! :) > > Notice pkg_info in ^T output during "Checking if x11/rxvt-unicode already > installed" phase. This one takes awhile. The actual command is: > `/usr/sbin/pkg_info -q -O x11/rxvt-unicode` > real 0m37.697s > user 0m0.125s > sys 0m0.360s > > find_pkgs_by_origin() in info/perform.c uses the same matchbyorigin() > in lib/match.c. What's interesting here, however, is that simple > `time grep ORIGIN /var/db/pkg/*/+CONTENTS` takes ~7 sec (XXX re-test on > that same notebook XXX), while find_pkgs_by_origin() incarnation of > practically the same functionality takes over 30 sec. > > To my eye, it doesn't look like matchbyorigin() could be re-implemented > to be faster with little effort, but could somebody have a quick look > as well? Would doing mmap() instead of scanning file line-by-line be > any faster? (though I'm not saying it's a great idea) > > BTW, I have a feeling that the "Registering installation" should be made > more verbose. It takes more time that anything else now, and one's left > to wonder what exactly is going on (seems like quite a few different > things). > > Also, I found that during the "Checking if <*> already installed" step, > 'mtree' (XXX find out exact command here XXX) is called (from bsd.port.mk?), > which can be skipped by setting NO_MTREE. What effect does [not] calling > mtree have? > > And to conclude, here we have a benchmark from my faster machine (Core2 > Dual 2.72GHz, 2G RAM, MK2018GAS 4200RPM HDD with 2M buffer), > BEFORE patch was applied: > > [root@soralx /usr/ports/x11/rxvt-unicode]# time make install > /dev/null > real 0m23.097s > user 0m0.000s > sys 0m0.219s > [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02/ > real 0m2.243s > user 0m0.056s > sys 0m0.202s > [root@soralx /usr/ports/x11/rxvt-unicode]# time make reinstall > /dev/null > real 0m26.867s > user 0m0.641s > sys 0m0.936s > [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-2.6.4_3/ #very few depends > real 0m0.090s > user 0m0.009s > sys 0m0.001s > [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ > real 0m0.521s > user 0m0.073s > sys 0m0.443s > > After patching pkg_install/: > > [root@soralx /usr/ports/x11/rxvt-unicode]# time make reinstall > /dev/null > real 0m3.602s > user 0m0.534s > sys 0m0.905s > [root@soralx /usr/ports/x11/rxvt-unicode]# time pkg_delete /var/db/pkg/rxvt-unicode-9.02_1/ > real 0m0.079s > user 0m0.033s > sys 0m0.046s > > Not bad, we see improvements in both pkg_install and pkg_delete, as > expected. > > > roman > > [SorAlx] ridin' VN1500-B2
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080331083643.GA39137>