From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 17:51:01 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70066B8A for ; Wed, 7 Jan 2015 17:51:01 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1.freebsd.org (Postfix) with ESMTP id 58F941CCA for ; Wed, 7 Jan 2015 17:51:01 +0000 (UTC) Received: from msam.nabble.com (unknown [162.253.133.85]) by mwork.nabble.com (Postfix) with ESMTP id C6DF8FCD0C6 for ; Wed, 7 Jan 2015 09:51:01 -0800 (PST) Date: Wed, 7 Jan 2015 10:51:01 -0700 (MST) From: sson To: freebsd-python@freebsd.org Message-ID: <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <54AD5BB9.2060701@ignoranthack.me> References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> Subject: Re: python waf bypasses _MAKE_JOBS number MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 17:51:01 -0000 Sean Bruno-6 wrote > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 01/07/15 00:21, Marcus von Appen wrote: >> On, Wed Jan 07, 2015, Sean Bruno wrote: >> >>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>> >>> Hey, so ... because qemu has a bug in it, we're trying to enforce >>> no SMP behaviour in builds. >>> >>> Turns out that python waf bypasses all of ports logic and probes >>> for the number of cpus and does its own thing. We noted this in >>> our builds as they locked up when using qemu due to a bug. >>> >>> Can this behaviour be investigated (configure behaviour) by some >>> python knowledgeable folks? >> >> Can you point us to the waf build logic for the qemu port? Looking >> at emulators/qemu, I do not see anything of relevance. >> >> Cheers Marcus >> > > Should have been a bit clearer, its qemu-bsd-user (via > qemu-user-static) that I'm talking about. > > I've added Stacey to the email to clarify a bit more. > > The behavior we are talking about isn't specific to qemu at all, its > just the configure script will probe the number of cpus and ignore my > override. > > sean > _______________________________________________ > freebsd-python@ > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-python > To unsubscribe, send any mail to " > freebsd-python-unsubscribe@ > " Hi all: The problem we are seeing is actually described in pretty good detail in an old a bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 It is hard to follow what was actually changed that fixed this problem on FreeBSD/amd64 but it seems the threading code for FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 port. In any case, if the "--jobs=1" flag is used then the ports will build without hanging. Python waf assumes, however, that the default jobs number should be the number of cores available on the system when the "--jobs=#" (or "-j#") is not explicitly given rather than assuming "--jobs=1". It seems, however, that many (if not all) the ports that use python waf make the assumption that it defaults to "--jobs=1". So, in summary: (1) It seems that described bug above wasn't fixed for all the FreeBSD arch's. It is unclear was the cause of the problem was and what was changed. If it is know what was changed it may be possible to figure out what needs to be changed to fix this problem in the other FreeBSD ports. (2) It seems to be assumed by many of the port maintainers that python waf defaults to "--jobs=1" when the argument is not given. This is problematic for poudriere, for example, given that it limits the MAKE_JOBS_NUMBER to 1 to better control the load on the build server. Of course, if #2 was fixed then it would solve two problems (for us anyway). :) Best Regards, -stacey. -- View this message in context: http://freebsd.1045724.n5.nabble.com/python-waf-bypasses-MAKE-JOBS-number-tp5978938p5979111.html Sent from the freebsd-python mailing list archive at Nabble.com.