From owner-freebsd-ports@freebsd.org Fri Nov 30 17:08:46 2018 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5E26114A143 for ; Fri, 30 Nov 2018 17:08:46 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4229F6CF53 for ; Fri, 30 Nov 2018 17:08:46 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id wAUH8jUd027992 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 30 Nov 2018 17:08:45 GMT (envelope-from list1@gjunka.com) Subject: Re: Distributed poudriere To: freebsd-ports@freebsd.org References: <38edd5f9-4a84-a8a6-2439-d54fb826982d@gjunka.com> From: Grzegorz Junka Message-ID: Date: Fri, 30 Nov 2018 17:08:45 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <38edd5f9-4a84-a8a6-2439-d54fb826982d@gjunka.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-Rspamd-Queue-Id: 4229F6CF53 X-Spamd-Result: default: False [-2.31 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ports@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.97)[-0.973,0]; DMARC_NA(0.00)[gjunka.com]; MX_GOOD(-0.01)[cached: gjunka.com]; NEURAL_HAM_SHORT(-0.02)[-0.017,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-0.02)[country: GB(-0.10)]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2018 17:08:46 -0000 On 29/11/2018 19:12, Grzegorz Junka wrote: > Hello, > > Is it possible to run poudriere in an agent/server setup? Where one > central server maintains a distfiles folder for already compiled > packages and agents compile packages fetching and installing from the > central server dependencies? > Just to make it more clear, I am asking about the build process. Poudriere starts the specified amount of jails to build packages in parallel. In each jail it then install dependencies by fetching them from the distfiles folder. After the build is done a new package is created and uploaded to the distfiles folder to be used by builds that require it. This process is CPU-core-bound, i.e. with 32/64 threads cores it can't really build more than 64 packages in parallel. And systems with so many cores are really expensive. But the whole process is inherently distributed (as long as the amount of packages is big enough so that they can be build in parallel). Being able to build on multiple cheaper systems with less CPU cores per system over the network seems like a good idea, no? GrzegorzJ