Date: Thu, 19 Oct 2006 23:24:25 -0700 From: Doug Barton <dougb@FreeBSD.org> To: Benjamin Lutz <mail@maxlor.com> Cc: freebsd-ports@freebsd.org Subject: Re: Parallel Builds Message-ID: <45386B99.8080501@FreeBSD.org> In-Reply-To: <200610191124.39379.mail@maxlor.com> References: <200610191124.39379.mail@maxlor.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Benjamin Lutz wrote:
> Hello,
>
> Since Multi-core processors are becoming popular (or, more
> egocentrically, since I've acquired one), I've become interested in
> parallel compilation. Unfortunately, it seems that parallel builds of
> any kind are completely unsupported by the ports framework at the
> moment.
Not for lack of interest, but for lack of person hours to do the work.
As you've already ascertained, the amount of work required is
substantial, and would amount to a constantly moving target, since new
ports are always being added, and "vendor code" is always changing.
> o Leave the ports framework as it is, and implement support for
> parallel building in add-on tool, eg., portupgrade. The tool would
> support automatic parallelism ("portupgrade -a" would automatically
> build ports in parallel where possible), or having several
> user-created instances running at the same time. I'll call this
> "tool-based macro-parallelism".
I specifically designed portmaster to be able to do this. Separate
instances do not share anything in common, other than the ports tree
and package database. Assuming that you were updating two (or more)
ports that you were positive did not share any dependencies in common,
you could run as many portmaster instances as you wanted to, and not
have any problems.
Where this gets really sticky is in (for example) the -a case. If you
run 'portmaster -l' on a typical system, you'll see that there are
some root ports (no dependencies, not depended on), some leaf ports
(not depended on), but most of the installed ports are in the
"middle," where they are depended on by something else, and most of
those have dependencies of their own. It would be a SMOP to have
portmaster track the ports that are "safe" to upgrade in parallel.
Then of course you have to keep track of your child processes, create
a queue for what ports to run next, etc. etc. It's all doable of
course, but I think the return on investment for this project would be
very small. Patches are always welcome however. :)
> Disadvantages:
> - Moderately difficult to implement.
*cough*
> Locking, build failures and interruptions would have to be taken
> care of. I don't see problems that can't be solved though.
Yeah, like I said, it's all about having the time to do it.
> - No speed gain when updating single large ports, eg. gcc. (To be
> fair, it must be said that some of the large ports, eg.
> OpenOffice.org, don't support micro-parallelism either. Macro-
> parallelism would at least allow the otherwise unused CPUs to
> do something sometimes.)
My gut feeling is that the cases where the gains outweigh the effort
required are very few, and far between. Maybe someday when 64-way
desktops are the norm, sure, but not now. :) I would of course not
ever discourage someone who wanted to work on this, it's just not on
my TODO list atm.
hth,
Doug
--
This .signature sanitized for your protection
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45386B99.8080501>