From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 21:26:05 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2C5CDB5 for ; Wed, 7 Jan 2015 21:26:05 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C6461F5 for ; Wed, 7 Jan 2015 21:26:05 +0000 (UTC) Received: from [10.12.76.108] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id ECCEE192A3B for ; Wed, 7 Jan 2015 21:26:03 +0000 (UTC) Message-ID: <54ADA468.9070206@ignoranthack.me> Date: Wed, 07 Jan 2015 14:26:00 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-python@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <1420653060968-5979111.post@n5.nabble.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 21:26:06 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/07/15 10:51, sson [via FreeBSD] wrote: > Sean Bruno-6 wrote >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >> >> On 01/07/15 00:21, Marcus von Appen wrote: >>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>> >>>> Hey, so ... because qemu has a bug in it, we're trying to >>>> enforce no SMP behaviour in builds. >>>> >>>> Turns out that python waf bypasses all of ports logic and >>>> probes for the number of cpus and does its own thing. We >>>> noted this in our builds as they locked up when using qemu >>>> due to a bug. >>>> >>>> Can this behaviour be investigated (configure behaviour) by >>>> some python knowledgeable folks? >>> >>> Can you point us to the waf build logic for the qemu port? >>> Looking at emulators/qemu, I do not see anything of relevance. >>> >>> Cheers Marcus >>> >> >> Should have been a bit clearer, its qemu-bsd-user (via >> qemu-user-static) that I'm talking about. >> >> I've added Stacey to the email to clarify a bit more. >> >> The behavior we are talking about isn't specific to qemu at all, >> its just the configure script will probe the number of cpus and >> ignore my override. >> >> sean _______________________________________________ > >> freebsd-python@ > >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-python To >> unsubscribe, send any mail to " > >> freebsd-python-unsubscribe@ > >> " > > Hi all: > > The problem we are seeing is actually described in pretty good > detail in an old a bug report: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 > > It is hard to follow what was actually changed that fixed this > problem on FreeBSD/amd64 but it seems the threading code for > FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 port. > In any case, if the "--jobs=1" flag is used then the ports will > build without hanging. Python waf assumes, however, that the > default jobs number should be the number of cores available on the > system when the "--jobs=#" (or "-j#") is not explicitly given > rather than assuming "--jobs=1". It seems, however, that many (if > not all) the ports that use python waf make the assumption that it > defaults to "--jobs=1". So, in summary: > > (1) It seems that described bug above wasn't fixed for all the > FreeBSD arch's. It is unclear was the cause of the problem was and > what was changed. If it is know what was changed it may be > possible to figure out what needs to be changed to fix this problem > in the other FreeBSD ports. > > (2) It seems to be assumed by many of the port maintainers that > python waf defaults to "--jobs=1" when the argument is not given. > This is problematic for poudriere, for example, given that it > limits the MAKE_JOBS_NUMBER to 1 to better control the load on the > build server. > > Of course, if #2 was fixed then it would solve two problems (for us > anyway). :) > > Best Regards, > > -stacey. > > > I think I can work around python waf assuming it can figure out the number of cpus in the two critical ports tdb/talloc by enforcing a - --jobs=1 https://people.freebsd.org/~sbruno/disable_make_jobs.diff These two are the most obvious in the tree that fail for armv6 qemu builds. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUraRkXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kg6kIAMpyFOqqIxhbsDI1HBFAWIM6 j/ci3S1i7H0uVzoTmvGn3ITDY8zAPG3cDm1EaPPcn5vUNvAnk9uzo0ip33OZ22WM 8mJza2vGyoWLFGdrZPwm/+izWdXDoGqLX1ZSvzQlT8elyD2DlzduX++UJNdLgBkU assRj4HUFxe/1OQO911bZXfQVrsU9XbvssDrCvgZlQGo7siEDrQUpbQgqfDviQjK SXxJy8qM7l60309w13TmZpCriE3nTmNMdUazTV5mXS/NhQqkL5ckiv0uHGrn6/a7 KC/L+whcjuT4FF86n0qCyrBiSsEemxOywIIdVd+r2X75r0pmXCw6214CI6QuZ40= =f2RT -----END PGP SIGNATURE-----