Date: Fri, 09 Jan 2015 10:52:11 -0700 From: Sean Bruno <sbruno@ignoranthack.me> To: mva@freebsd.org, "sson [via FreeBSD]" <ml-node+s1045724n5979111h21@n5.nabble.com> Cc: sbruno@freebsd.org, python@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number Message-ID: <54B0154B.80801@ignoranthack.me> In-Reply-To: <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/09/15 10:33, Marcus von Appen wrote: > > Quoting "sson [via FreeBSD]" > <ml-node+s1045724n5979111h21@n5.nabble.com>: > >> Sean Bruno-6 wrote >>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>> >>> On 01/07/15 00:21, Marcus von Appen wrote: >>>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>>> >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>>> >>>>> Hey, so ... because qemu has a bug in it, we're trying to >>>>> enforce no SMP behaviour in builds. >>>>> >>>>> Turns out that python waf bypasses all of ports logic and >>>>> probes for the number of cpus and does its own thing. We >>>>> noted this in our builds as they locked up when using qemu >>>>> due to a bug. >>>>> >>>>> Can this behaviour be investigated (configure behaviour) by >>>>> some python knowledgeable folks? >>>> >>>> Can you point us to the waf build logic for the qemu port? >>>> Looking at emulators/qemu, I do not see anything of >>>> relevance. >>>> >>>> Cheers Marcus >>>> >>> >>> Should have been a bit clearer, its qemu-bsd-user (via >>> qemu-user-static) that I'm talking about. >>> >>> I've added Stacey to the email to clarify a bit more. >>> >>> The behavior we are talking about isn't specific to qemu at >>> all, its just the configure script will probe the number of >>> cpus and ignore my override. >>> >>> sean _______________________________________________ >> >>> freebsd-python@ >> >>> mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-python To >>> unsubscribe, send any mail to " >> >>> freebsd-python-unsubscribe@ >> >>> " >> >> Hi all: >> >> The problem we are seeing is actually described in pretty good >> detail in an old a bug report: >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 >> >> It is hard to follow what was actually changed that fixed this >> problem on FreeBSD/amd64 but it seems the threading code for >> FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 >> port. In any case, if the > > So this is not a general issue, but a platform-specific one? > >> "--jobs=1" flag is used then the ports will build without >> hanging. Python waf assumes, however, that the default jobs >> number should be the number of cores available on the system when >> the "--jobs=#" (or "-j#") is not explicitly given rather than >> assuming "--jobs=1". It seems, however, that many (if not all) >> the ports that use python waf make the assumption that it >> defaults to "--jobs=1". So, in summary: > > So, the waf build system does things, it should not do, but instead > respect our wish for _MAKE_JOBS - but only on the platforms, on > which it currently breaks? > >> (1) It seems that described bug above wasn't fixed for all the >> FreeBSD arch's. It is unclear was the cause of the problem was >> and what was changed. If it is know what was changed it may be >> possible to figure out what needs to be changed to fix this >> problem in the other FreeBSD ports. >> >> (2) It seems to be assumed by many of the port maintainers that >> python waf defaults to "--jobs=1" when the argument is not given. >> This is problematic for poudriere, for example, given that it >> limits the MAKE_JOBS_NUMBER to 1 to better control the load on >> the build server. > > I doubt this. Most port maintainers more likely assume: if the port > builds and runs on $platform (amd64 and i386 for the majority), > everything is good. > >> Of course, if #2 was fixed then it would solve two problems (for >> us anyway). :) > > Point me to the correct script/location and I'll look through the > code. Right now I do not understand enough of the qemu build > infrastructure to find my way around it. Especially since I do not > find the typical "waf" scripts within the extracted upstream > sources. > > Cheers Marcus > Marcus: The thing that I would like "fixed" is python waf ignoring the fact that it should not try and detect the number of CPUs on the system. QEMU user mode, used in poudriere building of arm/mips, has its own bug right now that causes SMP behaviour to fail. There's no need for you to go into QEMU right now, so don't worry about it(this way leads to madness). sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUsBVJXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5khFYH/08p1J0D0bsnJPs+EVSvfc/H vBxAUbetdA9HjRPey2RwM04C6Ww8rENw+28zwOFiUNL6a0vEPNRJHJFhji/IeaV/ 1HE+Jlk383WztcfEhWmyuw13OuaGVKMi9FbOue6gDjxZb7cV3EkOA7saWfeJmS9s 9QIKLGJWtS5GBz7zhV/stJ1562FPLmrskfmmLYhlgsZ2fvUwdTRwyZgWKdMrNwrX D9ypxXXB3lcmM0Kt6sS1OD7lX0TDR2aajauSzU2a0EoziYlFLzNg108aLZetUMmv 3+S9tntFCb8eqA5AP/8OuzpyY/4Z4m4zNTXCe9/wT1MWlUUjSEPVAYE/eOe9evo= =mHMp -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54B0154B.80801>