Date: Tue, 28 Dec 2004 05:40:15 +0000 (UTC) From: Kris Kennaway <kris@FreeBSD.org> To: ports-committers@FreeBSD.org, cvs-ports@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: ports/Tools/portbuild/scripts pdispatch Message-ID: <200412280540.iBS5eFXC093424@repoman.freebsd.org>
next in thread | raw e-mail | index | archive | help
kris 2004-12-28 05:40:15 UTC
FreeBSD ports repository
Modified files:
Tools/portbuild/scripts pdispatch
Log:
Overhaul of the job scheduler. The new scheduler runs builds
synchronously instead of probabilistically scheduling jobs, which
means that the job load on a machine never exceeds a desired
threshold, and we can preferentially use faster machines when they are
available. This has a dramatic effect on package build throughput,
although I don't yet have precise measurements of the performance
improvements.
Specifically, the changes are:
* Introduce the new variable maxjobs in portbuild. This replaces the
build scheduling weights previously listed in the mlist file, which
now changes format to list the build machines only, ranked in order of
preference for job dispatches (i.e. faster machines first).
* The ${arch}/queue directory is used to list machines available for
jobs (file content is the number of jobs currently running on the
machine). Changes to files in this directory are serialized using
lockf on the .lock file.
* Claim a machine with the getmachine script, with the .lock held.
This picks the machine with the fewestnumber of jobs running, which is
listed highest in the mlist file in case of multiple machines with
equal load. The job counter is incremented, and the file removed if
the counter reaches ${maxjobs} for that machine. If all machines are
busy, sleep for 15 seconds and retry.
* After we have claimed a machine, we run claim-chroot on it to claim
an empty chroot, as before. If the claim fails, release the job from
the queue with the releasemachine script and retry after a 15 second
wait.
* When the build is finished, decrement the job counter with the
releasemachine script, with .lock held.
* The checkmachines script now exists only to poll the load averages
for admin convenience (every 2 minutes), and to ping for unreachable
machines. When a machine cannot be reached, remove the entry in the
queue directory to stop further job dispatches to it. This needs more
work to deal with reinitialization of machines after they become
available again.
Additional changes to this file:
* Exit if passed a null package name, to avoid badness later on
* Send a nag-mail if pkg-plist errors are detected in the build
Revision Changes Path
1.20 +32 -25 ports/Tools/portbuild/scripts/pdispatch
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200412280540.iBS5eFXC093424>