From nobody Sat Jun 12 20:26:16 2021 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 376B85D28F7 for ; Sat, 12 Jun 2021 20:26:21 +0000 (UTC) (envelope-from freebsd@quinteiro.org) Received: from mx2.quinteiro.org (mx2.quinteiro.org [71.19.154.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4G2Tkh2Njpz3qHR for ; Sat, 12 Jun 2021 20:26:20 +0000 (UTC) (envelope-from freebsd@quinteiro.org) Received: from www.quinteiro.org (www.quinteiro.org [204.109.56.22]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx2.quinteiro.org (Postfix) with ESMTPS id 4722D21D9A6 for ; Sat, 12 Jun 2021 20:26:18 +0000 (UTC) (envelope-from freebsd@quinteiro.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=quinteiro.org; s=default; t=1623529578; bh=wyO7EWQmQxTEzBfnNbSofiM2GL0mentumcwwkgJJPVQ=; h=Subject:To:References:From:Date:In-Reply-To; b=eS7XUF7SKqMQNDzK/Ug+G4RommC1U+pjKBetrqxegeXF0viKFKbbfu/x76sXKLJFy 9TKUwUtNRNscwIY20lkbvmBpD2MTKv+LXIpFB4ReX7JTp3jMkf69+n4sdrx8YLjaKf gZyU7nqesoPj7zeRU5bNEMI7Rh3r0m/4lMzxCEtE= Received: from [172.16.1.157] (198-27-221-245.fiber.dynamic.sonic.net [198.27.221.245]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by www.quinteiro.org (Postfix) with ESMTPSA id D6C802E80A for ; Sat, 12 Jun 2021 20:26:16 +0000 (UTC) Subject: Re: Restraining poudriere To: ports@freebsd.org References: <20210612172957.GA71089@www.zefox.net> <20210612175704.GC71089@www.zefox.net> From: Jose Quinteiro Message-ID: <0e365b53-eb19-e751-f688-518d799e0f81@quinteiro.org> Date: Sat, 12 Jun 2021 13:26:16 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 In-Reply-To: <20210612175704.GC71089@www.zefox.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4G2Tkh2Njpz3qHR X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=quinteiro.org header.s=default header.b=eS7XUF7S; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@quinteiro.org designates 71.19.154.200 as permitted sender) smtp.mailfrom=freebsd@quinteiro.org X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[quinteiro.org:s=default]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[ports@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[71.19.154.200:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[quinteiro.org]; DKIM_TRACE(0.00)[quinteiro.org:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[71.19.154.200:from]; ASN(0.00)[asn:47066, ipnet:71.19.154.0/24, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[ports] X-ThisMailContainsUnwantedMimeParts: N On 6/12/21 10:57 AM, bob prohaska wrote: > On Sat, Jun 12, 2021 at 07:36:48PM +0200, Michael Gmelin wrote: >> >> >>> On 12. Jun 2021, at 19:31, bob prohaska wrote: >>> >>> ???In playing with poudriere on raspberry pi 3 and 4 it seems to >>> work well on the 8 GB Pi4 but is over-optimistic on the 1 GB Pi3. >>> >>> Can poudriere be configured to tackle packages one at a time? >> >> Yes, see poudriere.conf: >> >> # parallel build support. >> # >> # By default poudriere uses hw.ncpu to determine the number of builders. >> # You can override this default by changing PARALLEL_JOBS here, or >> # by specifying the -J flag to bulk/testport. >> # >> # Example to define PARALLEL_JOBS to one single job >> # PARALLEL_JOBS=1 >> >> -m >> > > I perhaps misunderstood what was meant by "builders", confusing it > with threads. Or maybe cores.... > > Trying it now, hoping to see parallel core use. You won't. Setting PARALLEL_JOBS=1 means exactly one Poudriere worker will run, and that make will not build in parallel. You have to decide whether you want multiple Poudriere jobs, each using a single core at most, one Poudriere job with make(1) potentially using several cores, or some combination of both. The three variables that control this are: PARALLEL_JOBS, as you already know. This is the maximum number of workers Poudriere will launch. It can be less than this if there are multiple dependent packages waiting for a big package to build. This happens a lot with Llvm and the like. Not having ALLOW_MAKE_JOBS set is very frustrating in this case, because the big pig of a build will use one core at most and take forever. ALLOW_MAKE_JOBS and MAKE_JOBS_NUMBER (make.conf). The first one allows make(1) to launch parallel jobs, and the second controls the maximum number of make jobs that will be launched. The problem here is that many builds have lengthy steps that are not concurrent (I'm looking at you, autotools!) A lot of times the actual compile and link steps take a tiny fraction of the total build time. I chose to use a combination of both, and had to experiment with different numbers for PARALLEL_JOBS mad MAKE_JOBS_NUMBER so that a majority of my cores were used for the build, but I did not run out of memory: https://forums.FreeBSD.org/threads/poudriere-never-completes-first-run-without-errors.78000/post-486707 > > Thanks for writing! > > bob prohaska > You're welcome! Jose