From owner-soc-status@FreeBSD.ORG Tue Jun 12 14:07:10 2012 Return-Path: Delivered-To: soc-status@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D6481065677; Tue, 12 Jun 2012 14:07:10 +0000 (UTC) (envelope-from scherfreebsd@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id C2E698FC1A; Tue, 12 Jun 2012 14:07:09 +0000 (UTC) Received: by lbon10 with SMTP id n10so513465lbo.13 for ; Tue, 12 Jun 2012 07:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:content-type:content-transfer-encoding:subject:date :message-id:cc:to:mime-version:x-mailer; bh=8sEsDF5XXtuJdE1bObJawgFqfRDBmn4sYgxphFroNik=; b=Eiujf3vWbrCs8IQf3sZUqhmRPeRt54PRhUAI1Qd8Gfs2/+Jcer2fAKcr3wiJdcFQUL W9XF2U9hqOXTasuYqv+qSOji0VSPz7cMjO3MJeIKDT6rPhnQD+XXCGl5fCTcC6ACMIQB EMFL0MBn0RCL9d3RwrM0ck13ftHDnMtBOxCs8GSZd+CQ2nNYpv5vCtS/oiIqYypNWHp6 /yj/bQ2BMLAlId+/bucHnD0MtdcaqXSO/HjZUqty8VHtxPQStCJ1BQi49+sFQZ7q5Ekd 8A70NWegwKL2zYjL20E3twVt5Rbqu5vZ2UTi/PlsCKAO+6Wc0kDU0L0KCcNs517FzBNJ K3xA== Received: by 10.152.102.137 with SMTP id fo9mr20932513lab.35.1339510023039; Tue, 12 Jun 2012 07:07:03 -0700 (PDT) Received: from [192.168.0.100] ([77.66.153.242]) by mx.google.com with ESMTPS id mo3sm29576399lab.2.2012.06.12.07.06.59 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 12 Jun 2012 07:07:02 -0700 (PDT) Sender: Alexander Pronin From: Alexander Pronin Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 12 Jun 2012 18:06:52 +0400 Message-Id: <61953AC8-03BD-4E67-A008-B73A2F67A6D2@FreeBSD.org> To: soc-status@FreeBSD.org Mime-Version: 1.0 (Apple Message framework v1278) X-Mailer: Apple Mail (2.1278) Cc: Marcus von Appen Subject: [ Week Report GSOC ] Parallelization in the ports collection X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 14:07:10 -0000 LOCK_DIR changed from ${PORTSDIR}/lock to /var/db/portlocks. Make process (submake) is allowed to work in locked port's if and only = if this port's dir was locked by parent make process. Hence all submakes = can work in port's dir, locked by parent make. First parallel non-blocking build of port's dependecies as background = jobs in XXX-depends targets. Tested on fake ports. _parv_PARALLEL_BUILDS_NUMBER variable controls number of parallel = builds. May be set by user to choose degree of parallelization. ${_PROCESS_ACTIVE_BUILDS}: new script that covers most of non-blocking = parallel dependency build magic. Controls all spawned background jobs by PIDs which are stored in = $${active_builds} array Controls jobs exit codes. Acts accordingly on various jobs exit codes (0, 1, = ${_parv_MAKE_LOCK_EXIT_STATUS}). Recheck correct install of lib dependency. Considers parallel dep builds limit as set by = ${_parv_PARALLEL_BUILDS_NUMBER}. Waits for a while if it is impossible to spawn new dependency = build. Makes attempts to spawn another dependency build, if it is = possible. Changes in old targets (XXX-depends): Reimplementation of loops in XXX-depends targets for = non-blocking purpose: Outer "while" loop replaced "for" loop. Some kind of "round-robin" implementation is used to process = port's deps. If some dependency (dep) is locked then it is moved to the end = of the deps queue. If some dep is already installed then it is deleted from the = deps queue. If some dependency is not found - it's build starts as = background process and it's PID is added to $${active_builds} array to control it's = evaluation. All the above mentioned is actual for non-parallel builds, with the = following exception: =20 ${_parv_PARALLEL_BUILDS_NUMBER} is set to 1 in non-parallel case. This = perevents from spawning more than one background job. Thus processing of = port's deps will be blocked until this job will be ready. Hence we have = one background job per time for non-parallel build. Correct termination of the whole process tree with background processes = starting from parent make process. To terminate the whole process tree, = starting from some parent make process (pm_pid) one may just use = keyboard interrupts in the right terminal (where the make process was = started) or send signal to internal make script using pkill(1). e.g. = pkill -P pm_pid