Date: Wed, 26 Mar 2008 02:01:31 -0700 From: soralx@cydem.org To: freebsd-hackers@freebsd.org Subject: ports system woes Message-ID: <20080326020131.589a3334@freen0de>
next in thread | raw e-mail | index | archive | help
Folks,
are there any plans to rewrite the ports/packages system? Maybe someone
started work on improving things in this area already?
The thought that pkg_* tools and Mk/* scripts might be somewhat
inefficient had crossed my mind before, when at last modular Xorg
exposed all the inefficiencies in these tools. Basically, pkg_* and
portupgrade seem to use very inefficient algorithms all over the place;
they've become nearly useless on my system these days.
Consider pkg_delete, for example:
`time pkg_delete /var/db/pkg/rxvt-unicode-9.02/`
real 7m4.207s
user 0m4.083s
sys 0m16.348s
This one was rather benign. Many of bigger packages take 15 minutes to
get removed. Now, all the test were conducted on Pentium 4-M 1.2GHz
notebook with 256M RAM and MHT2040AH hard drive (5400RPM, 8M buffer).
Notice the amount of RAM. Also:
[root@freen0de /var/db/pkg]# ll|wc -l
959
For the rxvt case, the offending function is matchbyorigin() inside
src/usr.sbin/pkg_install/lib/match.c, called from delete/perform.c
function pkg_do(). Here's a snippet from pkg_do():
for (p = Plist.head; p ; p = p->next) {
if (p->type != PLIST_PKGDEP)
continue;
deporigin = \
(p->next != NULL && p->next->type == PLIST_DEPORIGIN) ?p->next->name :
NULL;
if (Verbose) {
printf("Trying to remove dependency on package '%s'", p->name);
if (deporigin != NULL)
printf(" with '%s' origin", deporigin);
printf(".\n");
}
if (!Fake) {
depnames = (deporigin != NULL) ? matchbyorigin(deporigin, NULL) :
NULL;
if (depnames == NULL) {
depnames = alloca(sizeof(*depnames) * 2);
depnames[0] = p->name;
depnames[1] = NULL;
}
for (i = 0; depnames[i] != NULL; i++)
undepend(depnames[i], pkg);
}
}
What exactly "removing dependency on package" means? (i.e., what is
undepend() for?) I didn't figure it out yet (anyone?), but from what I
read from the code, it calls matchbyorigin(deporigin, NULL) -- which
is the super-slow part -- while processing each @pkgdep one-by-one.
matchbyorigin(), in turn, scans +CONTENTS of each entry in db/pkg/* to get
'@comment ORIGIN:' value. As a result, each operation (like "Trying to
remove dependency on package 'xineramaproto-1.1.2' with 'x11/xineramaproto'
origin.") takes 3-4 seconds (first one ~30s), and there are ~120(!)
dependencies for urxvt (I imagine them evil penguins are all too glad to
make everything complete chaos, but hopefully making things _so_ modular
has at least some benefits).
Replacing 'if (!Fake)' with 'if (FALSE)' makes pkg_delete go _really_ fast
-- almost instantaneous. I didn't notice any side effects of that hack so
far, but there sure are some. Anyway, I understand the desire to move
functions like matchbyorigin() to libinstall, but can't that one at least
be made to accept an array of strings (package names) and process them in
a single pass? The whole of src/usr.sbin/pkg_install needs rewriting, IMO,
as this is just one of many examples of slow code.
Here's another example. Checking if a package is already installed was
also quite slow. Much of the slowness, turns out, was caused by the
following in ports/Mk/bsd.openssl.mk:
.if !defined(OPENSSL_PORT) && \
exists(${LOCALBASE}/lib/libcrypto.so)
# find installed port and use it for dependency
PKG_DBDIR?= ${DESTDIR}/var/db/pkg
OPENSSL_INSTALLED!= grep -l -r "^lib/libssl.so." "${PKG_DBDIR}" | \
while read contents; do \
sslprefix=`grep "^@cwd " "$${contents}" | ${HEAD} -n 1`; \
if test "$${sslprefix}" = "@cwd ${LOCALBASE}" ; then \
echo "$${contents}"; break; fi; done
OPENSSL_PORT!= grep "^@comment ORIGIN:" "${OPENSSL_INSTALLED}" | ${CUT} -d : -f 2
OPENSSL_SHLIBFILE!= grep "^lib/libssl.so." "${OPENSSL_INSTALLED}"
OPENSSL_SHLIBVER?= ${OPENSSL_SHLIBFILE:E}
.endif
OPENSSL_PORT?= security/openssl
Simply defining 'OPENSSL_PORT=security/openssl' in make.conf helped a lot
(although really, we need to figure a better method of finding installed
openssl port -- perhaps a script that updates OPENSSL_PORT in make.conf after
port installs?). It is still fairly slow, though.
Another example:
[root@freen0de /usr/ports/x11/nvidia-driver-96xx]# make fetch
===> Vulnerability check disabled, database not found
===> Found saved configuration for nvidia-driver-96.43.01
=> NVIDIA-FreeBSD-x86-96.43.05.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch from http://jp.download.nvidia.com/freebsd/96.43.05/.
NVIDIA-FreeBSD-x86-96.43.05.tar.gz 100% of 9444 kB 71 kBps
[root@freen0de /usr/ports/x11/nvidia-driver-96xx]# time make fetch
===> Vulnerability check disabled, database not found
===> Found saved configuration for nvidia-driver-96.43.01
real 0m53.340s
user 0m0.299s
sys 0m0.886s
[root@freen0de /usr/ports/x11/nvidia-driver-96xx]# time make maintainer
<deleted>
real 0m22.817s
user 0m0.265s
sys 0m0.685s
[root@freen0de /usr/ports/x11/nvidia-driver-96xx]#
So all these examples are only tip of the iceberg that is currently making
pkg_* & ports sink, but of course no other bugs are as serious as the one with
pkg_delete (or so I hope).
And I'm not even talking about portupgrade. Dragging it's feet at a snail pace,
running sluggish 'pkgdb -Q' each tiny step... it's a rather useful tool. But,
I believe portupgrade is not really needed -- if FreeBSD's base ports/packaging
tools get fixed and extended, they'd do the job (yah, there's portmaster, but
with broken pkg_*, it's just as good as portupgrade).
BTW, to make `portupgrade -a` at all useful, a while ago I whipped up a bash
command that lists all dependencies of installed packages:
export PORTSBASE=/usr/ports; cd /var/db/pkg
for dir in `ls`; do tmp=`(grep '@comment ORIGIN' "${dir}/+CONTENTS"|awk -F ":" '{ print $2 }')`; echo ${tmp}; cd ${PORTSBASE}/${tmp} && (make package-depends-list|awk -F " " '{ print $3 }'); cd /var/db/pkg/; done|sort|uniq > /tmp/deps.lst
Allow it 3-5 hours per 1000 installed ports 8-) The good thing is that
theoretically this should get all the dependencies in fresh ports tree
right, when fed from old (not updated) /var/db/pkg, so then you could do:
for dir in `cat /tmp/deps.lst`; do cd ${PORTSBASE}/${dir} && make config-conditional; done
which in ~10 minutes should take care care of all the new unconfigured
(`make config`) ports in single attempt. Practically, there are always some
problems with the gazillion versions of tcl* and tk* (go to ports tree and
manually run `make config` for each and every version of tcl* and tk* that's
there), and any cases where 'dialog' coredumps (long lists of options).
/tmp/deps.lst can be usually reused. Now, just to convert these commands to
normal /bin/sh script...
[SorAlx] ridin' VN1500-B2
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080326020131.589a3334>