From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 20 20:52:24 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 968CD16A41F for ; Fri, 20 Jan 2006 20:52:24 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from mail25.syd.optusnet.com.au (mail25.syd.optusnet.com.au [211.29.133.166]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB65E43D48 for ; Fri, 20 Jan 2006 20:52:23 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail25.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k0KKqKxa011561 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sat, 21 Jan 2006 07:52:21 +1100 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1]) by cirb503493.alcatel.com.au (8.12.10/8.12.10) with ESMTP id k0KKqKHh031784; Sat, 21 Jan 2006 07:52:20 +1100 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost) by cirb503493.alcatel.com.au (8.12.10/8.12.9/Submit) id k0KKqK9w031783; Sat, 21 Jan 2006 07:52:20 +1100 (EST) (envelope-from pjeremy) Date: Sat, 21 Jan 2006 07:52:20 +1100 From: Peter Jeremy To: Kris Kennaway Message-ID: <20060120205219.GK25397@cirb503493.alcatel.com.au> References: <79e2026f0601142345x1a9269bdl3153d1bb110be08d@mail.gmail.com> <86ek34d35f.fsf@xps.des.no> <17359.49899.183831.844670@bhuda.mired.org> <86k6cww445.fsf@xps.des.no> <20060119175224.GA71633@csh.rit.edu> <79e2026f0601191215p117d78ebjbee2c4b710075154@mail.gmail.com> <43D0139A.1040601@myrealbox.com> <20060119230604.GA98670@xor.obsecurity.org> <43D11499.8040808@myrealbox.com> <20060120194700.GB41039@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060120194700.GB41039@xor.obsecurity.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: freebsd-hackers@freebsd.org Subject: Re: speed up port compiling using RAM (tmpfs) ??? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2006 20:52:24 -0000 On Fri, 2006-Jan-20 14:47:00 -0500, Kris Kennaway wrote: >On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > >> >-j is not safe to use with port builds since many ported software >> >contain race conditions in the build. >> > >> >Kris >> >> This effectively means that you cannot take advantage of SMP to compile >> FreeBSD's ports collection. That sounds like a big >> limitation...especially for people trying to speed up bulk builds. > >Yeah, but what do you propose to do about it? We have 14000 ports >that need to be inspected for build race conditions and fixed before >you can turn on -j by default. There are some areas of the FreeBSD port building infrastructure that could be enhanced to make it possible to run multiple top-level makes in parallel, even if individual ports can't be build with '-j'. IMHO, the biggest problem (as des pointed out) is that there's nothing to prevent two makes attempting to build the same port (this can easily happen when both ports A and B depend on port C). One possible solution would be to create another status file in the work directory and hold a file lock (flock/lockf) on it whilst a make is in progress. Any parallel attempt to make that port would block. If the above was implemented, an enhancement would then be to process the port's dependency list in parallel, rather than serially. This would allow a make that was blocked on one dependency to continue with a different dependency. -- Peter Jeremy