From owner-freebsd-toolchain@freebsd.org Sat Dec 9 23:16:00 2017 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D77EE9B548 for ; Sat, 9 Dec 2017 23:16:00 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-111.reflexion.net [208.70.210.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1AA7A79673 for ; Sat, 9 Dec 2017 23:15:59 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 25751 invoked from network); 9 Dec 2017 23:15:53 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 9 Dec 2017 23:15:53 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Sat, 09 Dec 2017 18:15:53 -0500 (EST) Received: (qmail 5465 invoked from network); 9 Dec 2017 23:15:53 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 9 Dec 2017 23:15:53 -0000 Received: from [192.168.1.25] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id AC08EEC91C1; Sat, 9 Dec 2017 15:15:52 -0800 (PST) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: parallel_run start_builder and clonefs for UFS in ports-mgmt/poudriere-devel: avoiding elapsed-time spent in the getblk wait channel, sleep a little Message-Id: <2F45D0AB-79E8-44B8-9B3C-25B0FF012298@dsl-only.net> Date: Sat, 9 Dec 2017 15:15:52 -0800 To: Bryan Drewery , FreeBSD Toolchain X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Dec 2017 23:16:00 -0000 I've been experimenting with using devel/poudriere-devel in the contexts: rpi2, bpi-m3 (Cortex-A7's, 4 core effective, 1 GiByte and 2 GiByte RAM) rpi3, Pine64+ 2GB (Cortex-A53's, 4 core, 1 GiByte and 2 GiByte RAM) powerpc64 (32-bit) powerpc (so-called Quad-Core PowerMacs, 16GiBYte and = 2 GiByte effective RAM) amd64 (Ryzen Threadripper 16-core/32-thread-total, 128 GiByte RAM) As part of this even the large configurations are currently set up for UFS so that aspect is uniform. One thing that I've noted (using the amd64 context as an example) is a non-linear relationship between builders and starting/cloning time: [00:00:26] Building 47 packages using 4 builders [00:00:26] Starting/Cloning builders [00:00:32] Hit CTRL+t at any time to see build progress and stats [00:00:22] Building 49 packages using 14 builders [00:00:22] Starting/Cloning builders [00:05:07] Hit CTRL+t at any time to see build progress and stats [00:00:33] Building 38 packages using 29 builders [00:00:33] Starting/Cloning builders [00:17:22] Hit CTRL+t at any time to see build progress and stats Monitoring the longer running ones I see lots of: getblk=20 (Waiting to access file data that is being accessed by another thread) So I tried a crude form of staggering the cloning by adding a 1 second sleep in start_builders ( /usr/local/share/poudriere/common.sh ): start_builders() { local arch=3D$(injail uname -p) msg "Starting/Cloning builders" bset status "starting_jobs:" run_hook start_builders start bset builders "${JOBS}" bset status "starting_builders:" parallel_start for j in ${JOBS}; do parallel_run start_builder ${j} ${arch} sleep 1 done parallel_stop run_hook start_builders stop } The result for 14 was: [00:00:14] Building 23 packages using 14 builders [00:00:14] Starting/Cloning builders [00:00:53] Hit CTRL+t at any time to see build progress and stats For 23 (I no longer had more to build) the result was: [00:00:12] Building 23 packages using 23 builders [00:00:12] Starting/Cloning builders [00:04:49] Hit CTRL+t at any time to see build progress and stats Trying 5 seconds instead of 1 second for the sleep time: The result for 14 was: [00:00:12] Building 23 packages using 14 builders [00:00:12] Starting/Cloning builders [00:01:25] Hit CTRL+t at any time to see build progress and stats For 23 the result was: [00:00:12] Building 23 packages using 23 builders [00:00:12] Starting/Cloning builders [00:02:11] Hit CTRL+t at any time to see build progress and stats It appears that a few second sleep is of significant help for having lots of builders. =3D=3D=3D Mark Millard markmi at dsl-only.net