Date: Tue, 12 Jun 2012 18:06:52 +0400 From: Alexander Pronin <scher@FreeBSD.org> To: soc-status@FreeBSD.org Cc: Marcus von Appen <mva@FreeBSD.org> Subject: [ Week Report GSOC ] Parallelization in the ports collection Message-ID: <61953AC8-03BD-4E67-A008-B73A2F67A6D2@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
LOCK_DIR changed from ${PORTSDIR}/lock to /var/db/portlocks.
Make process (submake) is allowed to work in locked port's if and only =
if this port's dir was locked by parent make process. Hence all submakes =
can work in port's dir, locked by parent make.
First parallel non-blocking build of port's dependecies as background =
jobs in XXX-depends targets. Tested on fake ports.
_parv_PARALLEL_BUILDS_NUMBER variable controls number of parallel =
builds. May be set by user to choose degree of parallelization.
${_PROCESS_ACTIVE_BUILDS}: new script that covers most of non-blocking =
parallel dependency build magic.
Controls all spawned background jobs by PIDs which are stored in =
$${active_builds} array
Controls jobs exit codes.
Acts accordingly on various jobs exit codes (0, 1, =
${_parv_MAKE_LOCK_EXIT_STATUS}).
Recheck correct install of lib dependency.
Considers parallel dep builds limit as set by =
${_parv_PARALLEL_BUILDS_NUMBER}.
Waits for a while if it is impossible to spawn new dependency =
build.
Makes attempts to spawn another dependency build, if it is =
possible.
Changes in old targets (XXX-depends):
Reimplementation of loops in XXX-depends targets for =
non-blocking purpose:
Outer "while" loop replaced "for" loop.
Some kind of "round-robin" implementation is used to process =
port's deps.
If some dependency (dep) is locked then it is moved to the end =
of the deps queue.
If some dep is already installed then it is deleted from the =
deps queue.
If some dependency is not found - it's build starts as =
background process and
it's PID is added to $${active_builds} array to control it's =
evaluation.
All the above mentioned is actual for non-parallel builds, with the =
following exception: =20
${_parv_PARALLEL_BUILDS_NUMBER} is set to 1 in non-parallel case. This =
perevents from spawning more than one background job. Thus processing of =
port's deps will be blocked until this job will be ready. Hence we have =
one background job per time for non-parallel build.
Correct termination of the whole process tree with background processes =
starting from parent make process. To terminate the whole process tree, =
starting from some parent make process (pm_pid) one may just use =
keyboard interrupts in the right terminal (where the make process was =
started) or send signal to internal make script using pkill(1). e.g. =
pkill -P pm_pid
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?61953AC8-03BD-4E67-A008-B73A2F67A6D2>