From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 17:44:56 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7CD3B3A0; Fri, 9 Jan 2015 17:44:56 +0000 (UTC) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.29.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B4F664A; Fri, 9 Jan 2015 17:44:55 +0000 (UTC) Received: from [80.67.16.130] (helo=webmailfront01.ispgateway.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84) (envelope-from ) id 1Y9dRG-00022z-Sw; Fri, 09 Jan 2015 18:33:34 +0100 Received: from a89-182-73-111.net-htp.de (a89-182-73-111.net-htp.de [89.182.73.111]) by webmail.df.eu (Horde Framework) with HTTP; Fri, 09 Jan 2015 18:33:34 +0100 Date: Fri, 09 Jan 2015 18:33:34 +0100 Message-ID: <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> From: Marcus von Appen To: "sson [via FreeBSD]" Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <1420653060968-5979111.post@n5.nabble.com> Reply-to: mva@freebsd.org User-Agent: Internet Messaging Program (IMP) H5 (6.0.4) Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-Df-Sender: ZnJlZWJzZEBzeXNmYXVsdC5vcmc= Cc: sbruno@freebsd.org, python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 17:44:56 -0000 Quoting "sson [via FreeBSD]" : > Sean Bruno-6 wrote >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> On 01/07/15 00:21, Marcus von Appen wrote: >>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>> >>>> Hey, so ... because qemu has a bug in it, we're trying to enforce >>>> no SMP behaviour in builds. >>>> >>>> Turns out that python waf bypasses all of ports logic and probes >>>> for the number of cpus and does its own thing. We noted this in >>>> our builds as they locked up when using qemu due to a bug. >>>> >>>> Can this behaviour be investigated (configure behaviour) by some >>>> python knowledgeable folks? >>> >>> Can you point us to the waf build logic for the qemu port? Looking >>> at emulators/qemu, I do not see anything of relevance. >>> >>> Cheers Marcus >>> >> >> Should have been a bit clearer, its qemu-bsd-user (via >> qemu-user-static) that I'm talking about. >> >> I've added Stacey to the email to clarify a bit more. >> >> The behavior we are talking about isn't specific to qemu at all, its >> just the configure script will probe the number of cpus and ignore my >> override. >> >> sean >> _______________________________________________ > >> freebsd-python@ > >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-python >> To unsubscribe, send any mail to " > >> freebsd-python-unsubscribe@ > >> " > > Hi all: > > The problem we are seeing is actually described in pretty good detail in an > old a bug report: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 > > It is hard to follow what was actually changed that fixed this problem on > FreeBSD/amd64 but it seems the threading code for FreeBSD/{arm, mips, > mips64} may not be as mature as the amd64 port. In any case, if the So this is not a general issue, but a platform-specific one? > "--jobs=1" flag is used then the ports will build without hanging. Python > waf assumes, however, that the default jobs number should be the number of > cores available on the system when the "--jobs=#" (or "-j#") is not > explicitly given rather than assuming "--jobs=1". It seems, however, that > many (if not all) the ports that use python waf make the assumption that it > defaults to "--jobs=1". So, in summary: So, the waf build system does things, it should not do, but instead respect our wish for _MAKE_JOBS - but only on the platforms, on which it currently breaks? > (1) It seems that described bug above wasn't fixed for all the FreeBSD > arch's. It is unclear was the cause of the problem was and what was > changed. If it is know what was changed it may be possible to figure out > what needs to be changed to fix this problem in the other FreeBSD ports. > > (2) It seems to be assumed by many of the port maintainers that python waf > defaults to "--jobs=1" when the argument is not given. This is problematic > for poudriere, for example, given that it limits the MAKE_JOBS_NUMBER to 1 > to better control the load on the build server. I doubt this. Most port maintainers more likely assume: if the port builds and runs on $platform (amd64 and i386 for the majority), everything is good. > Of course, if #2 was fixed then it would solve two problems (for us anyway). > :) Point me to the correct script/location and I'll look through the code. Right now I do not understand enough of the qemu build infrastructure to find my way around it. Especially since I do not find the typical "waf" scripts within the extracted upstream sources. Cheers Marcus