From owner-freebsd-python@FreeBSD.ORG Sun Jan 4 17:26:11 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 59C0E746 for ; Sun, 4 Jan 2015 17:26:11 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 40EC92B35 for ; Sun, 4 Jan 2015 17:26:11 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t04HQBBq055226 for ; Sun, 4 Jan 2015 17:26:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 195690] lang/python27 fails to package on i386 (nis.so) Date: Sun, 04 Jan 2015 17:26:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zaphod@berentweb.com X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: python@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jan 2015 17:26:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195690 zaphod@berentweb.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |FIXED --- Comment #2 from zaphod@berentweb.com --- SOLVED: This was probably an issue related to the i386 jail's world. I re-built i386 world with latest source and the jail with latest world builds the port without problem. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-python@FreeBSD.ORG Tue Jan 6 23:50:59 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D5965FA8 for ; Tue, 6 Jan 2015 23:50:59 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5A93644EA for ; Tue, 6 Jan 2015 23:50:59 +0000 (UTC) Received: from [10.12.76.127] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id B556C192A3B for ; Tue, 6 Jan 2015 23:50:58 +0000 (UTC) Message-ID: <54AC74E0.4050404@ignoranthack.me> Date: Tue, 06 Jan 2015 15:50:56 -0800 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-python@freebsd.org Subject: python waf bypasses _MAKE_JOBS number Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2015 23:50:59 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hey, so ... because qemu has a bug in it, we're trying to enforce no SMP behaviour in builds. Turns out that python waf bypasses all of ports logic and probes for the number of cpus and does its own thing. We noted this in our builds as they locked up when using qemu due to a bug. Can this behaviour be investigated (configure behaviour) by some python knowledgeable folks? sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUrHTgXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5ktAsH/3P9StdKr01WkEYjoQ8oG8S/ J6ZXmOsCAeG98rehng9+UuRFvD3ifFdFIxHdFRtVIFDe8JGBU9AGkusGHLzyFUnT M4CVR3mQh6qlxdE/d3Hp7gNOlnssA/BXwPnKSLmK2FZVacgeCtUA/d01B7KDTADz 6mhpGg5eMgKOXfnAExMI4MIfVVTlfzdcj62DuaQjTTx0PpcmTxNVs+5QnTOcRyBc qZFoorZr5x97L4PGlausDe8CVGTzOGs31bHgxCJZF28tQ0L49Qyq6tEo6DrK+ngB /7u0hL9ITxMN94m57uzZDp8T6Wkr8q5eHys7J7GMMRjm8QrtNEFvDl/X3M/Dweg= =nAQ4 -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 07:24:37 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF35A6EE; Wed, 7 Jan 2015 07:24:37 +0000 (UTC) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AF86E1BA3; Wed, 7 Jan 2015 07:24:37 +0000 (UTC) Received: from [89.182.174.158] (helo=localhost) by smtprelay06.ispgateway.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.84) (envelope-from ) id 1Y8kvr-0005Se-3l; Wed, 07 Jan 2015 08:21:31 +0100 Date: Wed, 7 Jan 2015 08:21:32 +0100 From: Marcus von Appen To: sbruno@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number Message-ID: <20150107072132.GA1050@medusa.sysfault.org> Reply-To: Marcus von Appen Mail-Followup-To: sbruno@freebsd.org, freebsd-python@freebsd.org References: <54AC74E0.4050404@ignoranthack.me> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WIyZ46R2i8wDzkSu" Content-Disposition: inline In-Reply-To: <54AC74E0.4050404@ignoranthack.me> User-Agent: Mutt/1.5.23 (2014-03-12) X-Df-Sender: MTEyNTc0Mg== Cc: freebsd-python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 07:24:38 -0000 --WIyZ46R2i8wDzkSu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On, Wed Jan 07, 2015, Sean Bruno wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hey, so ... because qemu has a bug in it, we're trying to enforce no > SMP behaviour in builds. > > Turns out that python waf bypasses all of ports logic and probes for > the number of cpus and does its own thing. We noted this in our > builds as they locked up when using qemu due to a bug. > > Can this behaviour be investigated (configure behaviour) by some > python knowledgeable folks? Can you point us to the waf build logic for the qemu port? Looking at emulators/qemu, I do not see anything of relevance. Cheers Marcus --WIyZ46R2i8wDzkSu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlSs3nwACgkQi68/ErJnpkf6SQCg0Txy+RMEPRvbds0ZnOTUskp0 fkUAoLv2VE0VTRVKtobNeCX+DWUkndwa =goLV -----END PGP SIGNATURE----- --WIyZ46R2i8wDzkSu-- From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 16:16:01 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D67715D; Wed, 7 Jan 2015 16:16:01 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C28E64BFD; Wed, 7 Jan 2015 16:16:00 +0000 (UTC) Received: from [10.12.76.108] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 98EDF192A3B; Wed, 7 Jan 2015 16:15:59 +0000 (UTC) Message-ID: <54AD5BB9.2060701@ignoranthack.me> Date: Wed, 07 Jan 2015 09:15:53 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: sbruno@freebsd.org, freebsd-python@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> In-Reply-To: <20150107072132.GA1050@medusa.sysfault.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: sson@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 16:16:01 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/07/15 00:21, Marcus von Appen wrote: > On, Wed Jan 07, 2015, Sean Bruno wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >> >> Hey, so ... because qemu has a bug in it, we're trying to enforce >> no SMP behaviour in builds. >> >> Turns out that python waf bypasses all of ports logic and probes >> for the number of cpus and does its own thing. We noted this in >> our builds as they locked up when using qemu due to a bug. >> >> Can this behaviour be investigated (configure behaviour) by some >> python knowledgeable folks? > > Can you point us to the waf build logic for the qemu port? Looking > at emulators/qemu, I do not see anything of relevance. > > Cheers Marcus > Should have been a bit clearer, its qemu-bsd-user (via qemu-user-static) that I'm talking about. I've added Stacey to the email to clarify a bit more. The behavior we are talking about isn't specific to qemu at all, its just the configure script will probe the number of cpus and ignore my override. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUrVu2XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kgCAIAIIbdKyhnLRgBqslQsK5iNgr djtRAxWTCckZpBAWvBPQunjaSBvPp9Xh4QlQE0FJ4/cApxGHk7Dcr9Sjjj8oyAjH RTL+XT9dgrba22Zo3FoP71UJtZHKYLUaB4vWss1HWLHbAe4c7VQ4Lzr/9+cWlTMR qrfuQpdanqN0H1czCZfJOHTF1Z9jHV6i4T7TxwDr/vt6Rs9BIa20ABD3B7dVLMo8 32LNyYBDBM5H8+oG7qE5tT4i788Y+qwTQaoE7JMjEbW0zfCI/Si8Z04jcXei2sXF KIEn2hqGBHCSddmEKZIviL/yrX/EeKtbTNWuQmt8W3ygnpNaOASdyGtjnphB8WA= =HJog -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 17:51:01 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70066B8A for ; Wed, 7 Jan 2015 17:51:01 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1.freebsd.org (Postfix) with ESMTP id 58F941CCA for ; Wed, 7 Jan 2015 17:51:01 +0000 (UTC) Received: from msam.nabble.com (unknown [162.253.133.85]) by mwork.nabble.com (Postfix) with ESMTP id C6DF8FCD0C6 for ; Wed, 7 Jan 2015 09:51:01 -0800 (PST) Date: Wed, 7 Jan 2015 10:51:01 -0700 (MST) From: sson To: freebsd-python@freebsd.org Message-ID: <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <54AD5BB9.2060701@ignoranthack.me> References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> Subject: Re: python waf bypasses _MAKE_JOBS number MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 17:51:01 -0000 Sean Bruno-6 wrote > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 01/07/15 00:21, Marcus von Appen wrote: >> On, Wed Jan 07, 2015, Sean Bruno wrote: >> >>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>> >>> Hey, so ... because qemu has a bug in it, we're trying to enforce >>> no SMP behaviour in builds. >>> >>> Turns out that python waf bypasses all of ports logic and probes >>> for the number of cpus and does its own thing. We noted this in >>> our builds as they locked up when using qemu due to a bug. >>> >>> Can this behaviour be investigated (configure behaviour) by some >>> python knowledgeable folks? >> >> Can you point us to the waf build logic for the qemu port? Looking >> at emulators/qemu, I do not see anything of relevance. >> >> Cheers Marcus >> > > Should have been a bit clearer, its qemu-bsd-user (via > qemu-user-static) that I'm talking about. > > I've added Stacey to the email to clarify a bit more. > > The behavior we are talking about isn't specific to qemu at all, its > just the configure script will probe the number of cpus and ignore my > override. > > sean > _______________________________________________ > freebsd-python@ > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-python > To unsubscribe, send any mail to " > freebsd-python-unsubscribe@ > " Hi all: The problem we are seeing is actually described in pretty good detail in an old a bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 It is hard to follow what was actually changed that fixed this problem on FreeBSD/amd64 but it seems the threading code for FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 port. In any case, if the "--jobs=1" flag is used then the ports will build without hanging. Python waf assumes, however, that the default jobs number should be the number of cores available on the system when the "--jobs=#" (or "-j#") is not explicitly given rather than assuming "--jobs=1". It seems, however, that many (if not all) the ports that use python waf make the assumption that it defaults to "--jobs=1". So, in summary: (1) It seems that described bug above wasn't fixed for all the FreeBSD arch's. It is unclear was the cause of the problem was and what was changed. If it is know what was changed it may be possible to figure out what needs to be changed to fix this problem in the other FreeBSD ports. (2) It seems to be assumed by many of the port maintainers that python waf defaults to "--jobs=1" when the argument is not given. This is problematic for poudriere, for example, given that it limits the MAKE_JOBS_NUMBER to 1 to better control the load on the build server. Of course, if #2 was fixed then it would solve two problems (for us anyway). :) Best Regards, -stacey. -- View this message in context: http://freebsd.1045724.n5.nabble.com/python-waf-bypasses-MAKE-JOBS-number-tp5978938p5979111.html Sent from the freebsd-python mailing list archive at Nabble.com. From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 21:26:05 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2C5CDB5 for ; Wed, 7 Jan 2015 21:26:05 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C6461F5 for ; Wed, 7 Jan 2015 21:26:05 +0000 (UTC) Received: from [10.12.76.108] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id ECCEE192A3B for ; Wed, 7 Jan 2015 21:26:03 +0000 (UTC) Message-ID: <54ADA468.9070206@ignoranthack.me> Date: Wed, 07 Jan 2015 14:26:00 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-python@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <1420653060968-5979111.post@n5.nabble.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 21:26:06 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/07/15 10:51, sson [via FreeBSD] wrote: > Sean Bruno-6 wrote >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >> >> On 01/07/15 00:21, Marcus von Appen wrote: >>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>> >>>> Hey, so ... because qemu has a bug in it, we're trying to >>>> enforce no SMP behaviour in builds. >>>> >>>> Turns out that python waf bypasses all of ports logic and >>>> probes for the number of cpus and does its own thing. We >>>> noted this in our builds as they locked up when using qemu >>>> due to a bug. >>>> >>>> Can this behaviour be investigated (configure behaviour) by >>>> some python knowledgeable folks? >>> >>> Can you point us to the waf build logic for the qemu port? >>> Looking at emulators/qemu, I do not see anything of relevance. >>> >>> Cheers Marcus >>> >> >> Should have been a bit clearer, its qemu-bsd-user (via >> qemu-user-static) that I'm talking about. >> >> I've added Stacey to the email to clarify a bit more. >> >> The behavior we are talking about isn't specific to qemu at all, >> its just the configure script will probe the number of cpus and >> ignore my override. >> >> sean _______________________________________________ > >> freebsd-python@ > >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-python To >> unsubscribe, send any mail to " > >> freebsd-python-unsubscribe@ > >> " > > Hi all: > > The problem we are seeing is actually described in pretty good > detail in an old a bug report: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 > > It is hard to follow what was actually changed that fixed this > problem on FreeBSD/amd64 but it seems the threading code for > FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 port. > In any case, if the "--jobs=1" flag is used then the ports will > build without hanging. Python waf assumes, however, that the > default jobs number should be the number of cores available on the > system when the "--jobs=#" (or "-j#") is not explicitly given > rather than assuming "--jobs=1". It seems, however, that many (if > not all) the ports that use python waf make the assumption that it > defaults to "--jobs=1". So, in summary: > > (1) It seems that described bug above wasn't fixed for all the > FreeBSD arch's. It is unclear was the cause of the problem was and > what was changed. If it is know what was changed it may be > possible to figure out what needs to be changed to fix this problem > in the other FreeBSD ports. > > (2) It seems to be assumed by many of the port maintainers that > python waf defaults to "--jobs=1" when the argument is not given. > This is problematic for poudriere, for example, given that it > limits the MAKE_JOBS_NUMBER to 1 to better control the load on the > build server. > > Of course, if #2 was fixed then it would solve two problems (for us > anyway). :) > > Best Regards, > > -stacey. > > > I think I can work around python waf assuming it can figure out the number of cpus in the two critical ports tdb/talloc by enforcing a - --jobs=1 https://people.freebsd.org/~sbruno/disable_make_jobs.diff These two are the most obvious in the tree that fail for armv6 qemu builds. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUraRkXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kg6kIAMpyFOqqIxhbsDI1HBFAWIM6 j/ci3S1i7H0uVzoTmvGn3ITDY8zAPG3cDm1EaPPcn5vUNvAnk9uzo0ip33OZ22WM 8mJza2vGyoWLFGdrZPwm/+izWdXDoGqLX1ZSvzQlT8elyD2DlzduX++UJNdLgBkU assRj4HUFxe/1OQO911bZXfQVrsU9XbvssDrCvgZlQGo7siEDrQUpbQgqfDviQjK SXxJy8qM7l60309w13TmZpCriE3nTmNMdUazTV5mXS/NhQqkL5ckiv0uHGrn6/a7 KC/L+whcjuT4FF86n0qCyrBiSsEemxOywIIdVd+r2X75r0pmXCw6214CI6QuZ40= =f2RT -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Wed Jan 7 22:28:40 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8FA5618F for ; Wed, 7 Jan 2015 22:28:40 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 781C2ADF for ; Wed, 7 Jan 2015 22:28:40 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t07MSeL8004341 for ; Wed, 7 Jan 2015 22:28:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 192511] lang/python27: Fix build with LibreSSL Date: Wed, 07 Jan 2015 22:28:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mozilla@tuxproject.de X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: python@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2015 22:28:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192511 --- Comment #25 from Jens K. Loewe --- Thanks, I missed the make.conf entries too. :) -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-python@FreeBSD.ORG Thu Jan 8 09:22:28 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C9FCE5B for ; Thu, 8 Jan 2015 09:22:28 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 057F0176 for ; Thu, 8 Jan 2015 09:22:28 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t089MRQF024965 for ; Thu, 8 Jan 2015 09:22:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 192511] lang/python27: Fix build with LibreSSL Date: Thu, 08 Jan 2015 09:22:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: spil.oss@gmail.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: python@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2015 09:22:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192511 --- Comment #26 from Bernard Spil --- This should now be fixed upstream (in 2.7.10?) See https://hg.python.org/cpython/rev/eddcb6671a48 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-python@FreeBSD.ORG Thu Jan 8 17:59:03 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFB82D9D for ; Thu, 8 Jan 2015 17:59:03 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6C2BC8 for ; Thu, 8 Jan 2015 17:59:03 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t08Hx31d063489 for ; Thu, 8 Jan 2015 17:59:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 196618] [exp-run]: changing python3 default version Date: Thu, 08 Jan 2015 17:59:03 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Ports Framework X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: antoine@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: antoine@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: exp-run? X-Bugzilla-Changed-Fields: cc assigned_to bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2015 17:59:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196618 Antoine Brodin changed: What |Removed |Added ---------------------------------------------------------------------------- CC|antoine@FreeBSD.org |python@FreeBSD.org Assignee|freebsd-ports-bugs@FreeBSD. |antoine@FreeBSD.org |org | Status|New |Open --- Comment #1 from Antoine Brodin --- Take for exp-run -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-python@FreeBSD.ORG Thu Jan 8 18:28:52 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3573A912 for ; Thu, 8 Jan 2015 18:28:52 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1D1DB68A for ; Thu, 8 Jan 2015 18:28:52 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t08ISp6V075066 for ; Thu, 8 Jan 2015 18:28:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 196618] [exp-run]: changing python3 default version Date: Thu, 08 Jan 2015 18:28:51 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Ports Framework X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: antoine@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: antoine@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: exp-run? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2015 18:28:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196618 Antoine Brodin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kwm@FreeBSD.org --- Comment #2 from Antoine Brodin --- The following plists will not work with python 3.4 as default (3.3 is hardcoded) accessibility/accerciser/pkg-plist accessibility/orca/pkg-plist accessibility/py3-atspi/pkg-plist audio/gnome-music/pkg-plist databases/monetdb/pkg-plist deskutils/alacarte/pkg-plist devel/libpeas/pkg-plist devel/py3-gobject3/pkg-plist editors/gedit/pkg-plist editors/gedit-plugins/pkg-plist -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-python@FreeBSD.ORG Thu Jan 8 21:52:46 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F15914 for ; Thu, 8 Jan 2015 21:52:46 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 76F5EE6D for ; Thu, 8 Jan 2015 21:52:46 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t08LqkDg010655 for ; Thu, 8 Jan 2015 21:52:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: python@FreeBSD.org Subject: [Bug 196618] [exp-run]: changing python3 default version Date: Thu, 08 Jan 2015 21:52:46 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Ports Framework X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: antoine@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: exp-run? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2015 21:52:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196618 --- Comment #3 from commit-hook@freebsd.org --- A commit references this bug: Author: kwm Date: Thu Jan 8 21:52:44 UTC 2015 New revision: 376571 URL: https://svnweb.freebsd.org/changeset/ports/376571 Log: Don't hardcode python version. While here: Add vala as build depend to gedit. Gedit-plugins: builds vte3 and gucharmap bindings. Register the depends and package the files. PR: 196618 Submitted by: python3 default to 3.4 exp-run (via antoine@) Changes: head/accessibility/accerciser/Makefile head/accessibility/accerciser/pkg-plist head/accessibility/orca/Makefile head/accessibility/orca/pkg-plist head/accessibility/py3-atspi/Makefile head/accessibility/py3-atspi/pkg-plist head/audio/gnome-music/Makefile head/audio/gnome-music/pkg-plist head/deskutils/alacarte/Makefile head/deskutils/alacarte/pkg-plist head/devel/libpeas/Makefile head/devel/libpeas/pkg-plist head/devel/py3-gobject3/pkg-plist head/editors/gedit/Makefile head/editors/gedit/pkg-plist head/editors/gedit-plugins/Makefile head/editors/gedit-plugins/pkg-plist -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 12:44:08 2015 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1AB1778B for ; Fri, 9 Jan 2015 12:44:08 +0000 (UTC) Received: from unitix.org (unitix.org [82.149.245.106]) by mx1.freebsd.org (Postfix) with SMTP id 76801FD1 for ; Fri, 9 Jan 2015 12:44:07 +0000 (UTC) Received: (qmail 37656 invoked from network); 9 Jan 2015 12:37:22 -0000 Received: from unknown (HELO hurx.thc) (88.68.203.142) by peer.unitix.org with SMTP; 9 Jan 2015 12:37:22 -0000 Received: (qmail 1574 invoked by uid 1001); 9 Jan 2015 12:37:21 -0000 Date: Fri, 9 Jan 2015 13:37:21 +0100 From: Christian Jachmann To: python@FreeBSD.org Subject: [notifications@github.com: Re: [youtube-dl] youtube-dl -U doesn't work anymore (#4666)] Message-ID: <20150109123721.GA1554@hurx.thc> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 12:44:08 -0000 Hi folks....what is necessary to get https working again ? So, what should I do ? Thanks a lot Christian Jachmann ----- Forwarded message from Philipp Hagemeister ----- Date: Thu, 08 Jan 2015 09:23:18 -0800 From: Philipp Hagemeister To: rg3/youtube-dl Cc: leooleo-habbib Subject: Re: [youtube-dl] youtube-dl -U doesn't work anymore (#4666) Most likely the result of upgrading to Python 2.7.9, which does include certificate verification, and a botched certificate setup on FreeBSD. Just for reference, can you visit https://yt-dl.org/ in a browser? What does `curl https://yt-dl.org/ >/dev/null` or `wget -O /dev/null https://yt-dl.org/` output? --- Reply to this email directly or view it on GitHub: https://github.com/rg3/youtube-dl/issues/4666#issuecomment-69214414 ----- End forwarded message ----- From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 13:09:47 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63CD4167 for ; Fri, 9 Jan 2015 13:09:47 +0000 (UTC) Received: from smtprelay05.ispgateway.de (smtprelay05.ispgateway.de [80.67.31.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2014423F for ; Fri, 9 Jan 2015 13:09:47 +0000 (UTC) Received: from [80.67.16.136] (helo=webmailfront01.ispgateway.de) by smtprelay05.ispgateway.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84) (envelope-from ) id 1Y9ZCD-00087V-3v; Fri, 09 Jan 2015 14:01:45 +0100 Received: from his1.his.de (his1.his.de [192.124.237.237]) by webmail.df.eu (Horde Framework) with HTTP; Fri, 09 Jan 2015 14:01:44 +0100 Date: Fri, 09 Jan 2015 14:01:44 +0100 Message-ID: <20150109140144.Horde.-Q8-xY6-DKrPLhgVcuQlqw1@webmail.df.eu> From: Marcus von Appen To: Christian Jachmann Subject: Re: [notifications@github.com: Re: [youtube-dl] youtube-dl -U doesn't work anymore (#4666)] References: <20150109123721.GA1554@hurx.thc> In-Reply-To: <20150109123721.GA1554@hurx.thc> Reply-to: mva@freebsd.org User-Agent: Internet Messaging Program (IMP) H5 (6.0.4) Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-Df-Sender: ZnJlZWJzZEBzeXNmYXVsdC5vcmc= Cc: python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 13:09:47 -0000 Christian Jachmann : > Hi folks....what is necessary to get https working again ? > > So, what should I do ? [youtube-dl not working due to certificate issues in Python 2.7.9] Check the advice from http://unix.stackexchange.com/questions/176294/what-should-i-do-about-python-2-7-9-not-looking-for-ssl-certificates-in-the-righ https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196431 (the root cause) currently waits for the certificate port maintainers to be evaluated. Cheers Marcus From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 17:44:56 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7CD3B3A0; Fri, 9 Jan 2015 17:44:56 +0000 (UTC) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.29.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B4F664A; Fri, 9 Jan 2015 17:44:55 +0000 (UTC) Received: from [80.67.16.130] (helo=webmailfront01.ispgateway.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84) (envelope-from ) id 1Y9dRG-00022z-Sw; Fri, 09 Jan 2015 18:33:34 +0100 Received: from a89-182-73-111.net-htp.de (a89-182-73-111.net-htp.de [89.182.73.111]) by webmail.df.eu (Horde Framework) with HTTP; Fri, 09 Jan 2015 18:33:34 +0100 Date: Fri, 09 Jan 2015 18:33:34 +0100 Message-ID: <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> From: Marcus von Appen To: "sson [via FreeBSD]" Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> In-Reply-To: <1420653060968-5979111.post@n5.nabble.com> Reply-to: mva@freebsd.org User-Agent: Internet Messaging Program (IMP) H5 (6.0.4) Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-Df-Sender: ZnJlZWJzZEBzeXNmYXVsdC5vcmc= Cc: sbruno@freebsd.org, python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 17:44:56 -0000 Quoting "sson [via FreeBSD]" : > Sean Bruno-6 wrote >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> On 01/07/15 00:21, Marcus von Appen wrote: >>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>> >>>> Hey, so ... because qemu has a bug in it, we're trying to enforce >>>> no SMP behaviour in builds. >>>> >>>> Turns out that python waf bypasses all of ports logic and probes >>>> for the number of cpus and does its own thing. We noted this in >>>> our builds as they locked up when using qemu due to a bug. >>>> >>>> Can this behaviour be investigated (configure behaviour) by some >>>> python knowledgeable folks? >>> >>> Can you point us to the waf build logic for the qemu port? Looking >>> at emulators/qemu, I do not see anything of relevance. >>> >>> Cheers Marcus >>> >> >> Should have been a bit clearer, its qemu-bsd-user (via >> qemu-user-static) that I'm talking about. >> >> I've added Stacey to the email to clarify a bit more. >> >> The behavior we are talking about isn't specific to qemu at all, its >> just the configure script will probe the number of cpus and ignore my >> override. >> >> sean >> _______________________________________________ > >> freebsd-python@ > >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-python >> To unsubscribe, send any mail to " > >> freebsd-python-unsubscribe@ > >> " > > Hi all: > > The problem we are seeing is actually described in pretty good detail in an > old a bug report: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 > > It is hard to follow what was actually changed that fixed this problem on > FreeBSD/amd64 but it seems the threading code for FreeBSD/{arm, mips, > mips64} may not be as mature as the amd64 port. In any case, if the So this is not a general issue, but a platform-specific one? > "--jobs=1" flag is used then the ports will build without hanging. Python > waf assumes, however, that the default jobs number should be the number of > cores available on the system when the "--jobs=#" (or "-j#") is not > explicitly given rather than assuming "--jobs=1". It seems, however, that > many (if not all) the ports that use python waf make the assumption that it > defaults to "--jobs=1". So, in summary: So, the waf build system does things, it should not do, but instead respect our wish for _MAKE_JOBS - but only on the platforms, on which it currently breaks? > (1) It seems that described bug above wasn't fixed for all the FreeBSD > arch's. It is unclear was the cause of the problem was and what was > changed. If it is know what was changed it may be possible to figure out > what needs to be changed to fix this problem in the other FreeBSD ports. > > (2) It seems to be assumed by many of the port maintainers that python waf > defaults to "--jobs=1" when the argument is not given. This is problematic > for poudriere, for example, given that it limits the MAKE_JOBS_NUMBER to 1 > to better control the load on the build server. I doubt this. Most port maintainers more likely assume: if the port builds and runs on $platform (amd64 and i386 for the majority), everything is good. > Of course, if #2 was fixed then it would solve two problems (for us anyway). > :) Point me to the correct script/location and I'll look through the code. Right now I do not understand enough of the qemu build infrastructure to find my way around it. Especially since I do not find the typical "waf" scripts within the extracted upstream sources. Cheers Marcus From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 17:52:15 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0B4EB540; Fri, 9 Jan 2015 17:52:15 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB21678A; Fri, 9 Jan 2015 17:52:14 +0000 (UTC) Received: from [10.12.76.105] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 534C8192A3B; Fri, 9 Jan 2015 17:52:13 +0000 (UTC) Message-ID: <54B0154B.80801@ignoranthack.me> Date: Fri, 09 Jan 2015 10:52:11 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: mva@freebsd.org, "sson [via FreeBSD]" Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> In-Reply-To: <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: sbruno@freebsd.org, python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 17:52:15 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/09/15 10:33, Marcus von Appen wrote: > > Quoting "sson [via FreeBSD]" > : > >> Sean Bruno-6 wrote >>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>> >>> On 01/07/15 00:21, Marcus von Appen wrote: >>>> On, Wed Jan 07, 2015, Sean Bruno wrote: >>>> >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >>>>> >>>>> Hey, so ... because qemu has a bug in it, we're trying to >>>>> enforce no SMP behaviour in builds. >>>>> >>>>> Turns out that python waf bypasses all of ports logic and >>>>> probes for the number of cpus and does its own thing. We >>>>> noted this in our builds as they locked up when using qemu >>>>> due to a bug. >>>>> >>>>> Can this behaviour be investigated (configure behaviour) by >>>>> some python knowledgeable folks? >>>> >>>> Can you point us to the waf build logic for the qemu port? >>>> Looking at emulators/qemu, I do not see anything of >>>> relevance. >>>> >>>> Cheers Marcus >>>> >>> >>> Should have been a bit clearer, its qemu-bsd-user (via >>> qemu-user-static) that I'm talking about. >>> >>> I've added Stacey to the email to clarify a bit more. >>> >>> The behavior we are talking about isn't specific to qemu at >>> all, its just the configure script will probe the number of >>> cpus and ignore my override. >>> >>> sean _______________________________________________ >> >>> freebsd-python@ >> >>> mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-python To >>> unsubscribe, send any mail to " >> >>> freebsd-python-unsubscribe@ >> >>> " >> >> Hi all: >> >> The problem we are seeing is actually described in pretty good >> detail in an old a bug report: >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717 >> >> It is hard to follow what was actually changed that fixed this >> problem on FreeBSD/amd64 but it seems the threading code for >> FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 >> port. In any case, if the > > So this is not a general issue, but a platform-specific one? > >> "--jobs=1" flag is used then the ports will build without >> hanging. Python waf assumes, however, that the default jobs >> number should be the number of cores available on the system when >> the "--jobs=#" (or "-j#") is not explicitly given rather than >> assuming "--jobs=1". It seems, however, that many (if not all) >> the ports that use python waf make the assumption that it >> defaults to "--jobs=1". So, in summary: > > So, the waf build system does things, it should not do, but instead > respect our wish for _MAKE_JOBS - but only on the platforms, on > which it currently breaks? > >> (1) It seems that described bug above wasn't fixed for all the >> FreeBSD arch's. It is unclear was the cause of the problem was >> and what was changed. If it is know what was changed it may be >> possible to figure out what needs to be changed to fix this >> problem in the other FreeBSD ports. >> >> (2) It seems to be assumed by many of the port maintainers that >> python waf defaults to "--jobs=1" when the argument is not given. >> This is problematic for poudriere, for example, given that it >> limits the MAKE_JOBS_NUMBER to 1 to better control the load on >> the build server. > > I doubt this. Most port maintainers more likely assume: if the port > builds and runs on $platform (amd64 and i386 for the majority), > everything is good. > >> Of course, if #2 was fixed then it would solve two problems (for >> us anyway). :) > > Point me to the correct script/location and I'll look through the > code. Right now I do not understand enough of the qemu build > infrastructure to find my way around it. Especially since I do not > find the typical "waf" scripts within the extracted upstream > sources. > > Cheers Marcus > Marcus: The thing that I would like "fixed" is python waf ignoring the fact that it should not try and detect the number of CPUs on the system. QEMU user mode, used in poudriere building of arm/mips, has its own bug right now that causes SMP behaviour to fail. There's no need for you to go into QEMU right now, so don't worry about it(this way leads to madness). sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUsBVJXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5khFYH/08p1J0D0bsnJPs+EVSvfc/H vBxAUbetdA9HjRPey2RwM04C6Ww8rENw+28zwOFiUNL6a0vEPNRJHJFhji/IeaV/ 1HE+Jlk383WztcfEhWmyuw13OuaGVKMi9FbOue6gDjxZb7cV3EkOA7saWfeJmS9s 9QIKLGJWtS5GBz7zhV/stJ1562FPLmrskfmmLYhlgsZ2fvUwdTRwyZgWKdMrNwrX D9ypxXXB3lcmM0Kt6sS1OD7lX0TDR2aajauSzU2a0EoziYlFLzNg108aLZetUMmv 3+S9tntFCb8eqA5AP/8OuzpyY/4Z4m4zNTXCe9/wT1MWlUUjSEPVAYE/eOe9evo= =mHMp -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 18:10:26 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04EFE855; Fri, 9 Jan 2015 18:10:26 +0000 (UTC) Received: from mail-yk0-x233.google.com (mail-yk0-x233.google.com [IPv6:2607:f8b0:4002:c07::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0E2994D; Fri, 9 Jan 2015 18:10:25 +0000 (UTC) Received: by mail-yk0-f179.google.com with SMTP id 19so4615893ykq.10; Fri, 09 Jan 2015 10:10:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NfsV5HWmmkI1qKq3s4xX3v/2BgUnKE630fmdgRFVOuo=; b=tOfQe9pJr08+WZypuJFS1y4V4Ac7oRQOleX3XLgjPO5SN200F6N0iToAx7IJHHBXdO uMlY7pMOkMtu/+aRYADYJXqrDpv6VuHb4ikWExwzmxrItBp0wxDPXVLTeHIG3FOyXnsk eiXt+ryNvtlOck/HtYXcYY2OMi42LoX/h65VThy3AH2qpVpk5bYQDQHckXQ8h9PTQ9Jf TNINScYnMdYeCnUCWXYagl9HD+SdzyjHRJh0v02gG61ye3y611HZkpJGCqao0sZpB67W zkWhjW5LyCOl8cRmTU4sFdYaOWAjAqw6oLGCJjDcL7pXJ/f5ktQvQpqvwIKvfGZW/aWt GWuQ== MIME-Version: 1.0 X-Received: by 10.236.105.210 with SMTP id k58mr12202378yhg.52.1420827024620; Fri, 09 Jan 2015 10:10:24 -0800 (PST) Sender: antoine.brodin.freebsd@gmail.com Received: by 10.170.142.198 with HTTP; Fri, 9 Jan 2015 10:10:24 -0800 (PST) In-Reply-To: <54B0154B.80801@ignoranthack.me> References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> Date: Fri, 9 Jan 2015 18:10:24 +0000 X-Google-Sender-Auth: 1MCMAnqTyju4Ou4qYhJfVJbViYI Message-ID: Subject: Re: python waf bypasses _MAKE_JOBS number From: Antoine Brodin To: Sean Bruno Content-Type: text/plain; charset=UTF-8 Cc: "sson \[via FreeBSD\]" , Marcus von Appen , python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 18:10:26 -0000 On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno wrote: > Marcus: > > The thing that I would like "fixed" is python waf ignoring the fact > that it should not try and detect the number of CPUs on the system. > Let me rephrase the problem, there are some problems in some individual ports using waf. Those ports do not respect ${MAKE_JOBS_NUMBER}, especially when MAKE_JOBS_NUMBER=1. This breaks qemu which is not multi job friendly (people using qemu have DISABLE_MAKE_JOBS=yes which sets MAKE_JOBS_NUMBER to 1). Cheers, Antoine From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 18:11:16 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81D618C4; Fri, 9 Jan 2015 18:11:16 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5BE65970; Fri, 9 Jan 2015 18:11:15 +0000 (UTC) Received: from [10.12.76.105] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 2A7E3192A3B; Fri, 9 Jan 2015 18:11:15 +0000 (UTC) Message-ID: <54B019C1.7090704@ignoranthack.me> Date: Fri, 09 Jan 2015 11:11:13 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Antoine Brodin Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: "sson \[via FreeBSD\]" , Marcus von Appen , python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 18:11:16 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/09/15 11:10, Antoine Brodin wrote: > On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno > wrote: >> Marcus: >> >> The thing that I would like "fixed" is python waf ignoring the >> fact that it should not try and detect the number of CPUs on the >> system. >> > > Let me rephrase the problem, there are some problems in some > individual ports using waf. Those ports do not respect > ${MAKE_JOBS_NUMBER}, especially when MAKE_JOBS_NUMBER=1. This > breaks qemu which is not multi job friendly (people using qemu > have DISABLE_MAKE_JOBS=yes which sets MAKE_JOBS_NUMBER to 1). > > Cheers, > > Antoine > YES. :-) Sorry for the confusion. sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUsBm+XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kRKQIAM6cUFJxc6LYQqUAIMXYuzjs /rtfp8qoeo1NFen026lh+z8LSQ7cYKdN87OYYH9wSfu7R2mwWyuQRn4Kdd9HrJJv pCnFRsQML7lbNS/h3kifoSX5C45CCgTVu1K134niRmI1/rQ6e1TilhRbl+IvQN9n m88GYZs61ZGulJlQRlKyj0VOLUL2qI52OHhzAoqQI4H1JJ2rgzctFmxmn+BBzrpT V6EK7zKis93sJ3AJEOZwxifQlo/Qd/lMKRsGVn3sqw9zRIml+URx+28d/AwdQ2K4 eNp32w37T1lrjNxG5/nZGNvyMyRzLwQ4Mr6xhAGE1wmcWOjLJbkNYZY/QkJXPVc= =XgZM -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 18:35:21 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 026E9B8C for ; Fri, 9 Jan 2015 18:35:21 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1.freebsd.org (Postfix) with ESMTP id DC9BBBE7 for ; Fri, 9 Jan 2015 18:35:20 +0000 (UTC) Received: from msam.nabble.com (unknown [162.253.133.85]) by mwork.nabble.com (Postfix) with ESMTP id B0C8F100655F for ; Fri, 9 Jan 2015 10:35:20 -0800 (PST) Date: Fri, 9 Jan 2015 11:35:20 -0700 (MST) From: sson To: freebsd-python@freebsd.org Message-ID: <1420828520061-5979566.post@n5.nabble.com> In-Reply-To: References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> Subject: Re: python waf bypasses _MAKE_JOBS number MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 18:35:21 -0000 Antoine Brodin-2 wrote > On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno < > sbruno@ > > wrote: >> Marcus: >> >> The thing that I would like "fixed" is python waf ignoring the fact >> that it should not try and detect the number of CPUs on the system. >> >=20 > Let me rephrase the problem, there are some problems in some > individual ports using waf. > Those ports do not respect ${MAKE_JOBS_NUMBER}, especially when > MAKE_JOBS_NUMBER=3D1. > This breaks qemu which is not multi job friendly (people using qemu > have DISABLE_MAKE_JOBS=3Dyes which sets MAKE_JOBS_NUMBER to 1). >=20 > Cheers, >=20 > Antoine > _______________________________________________ > freebsd-python@ > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-python > To unsubscribe, send any mail to " > freebsd-python-unsubscribe@ > " Actually Qemu (user-mode) is thread friendly (for the most part, anyway).= =20 Lots of other threaded programs will run just fine. There is a known problem with dealing with signals and threads but this is not the case with waf (and shows up usually with cmake instead). The problem with waf seem= s to be the old bug that was reported in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D160717 seems to still exist in the FreeBSD/arm, FreeBSD/mips, and maybe other threading implementations. The workaround is to only use "--jobs=3D1" with python waf. Of course, thi= s is what Proudriere wants to do anyway but it seems that all the port maintainers that use the waf build system assume if the "--job=3D..." argum= ent is not given then python waf will assume that it should be "--jobs=3D1" by default. Look at the Makefile for the devel/talloc port for example=E2=80=A6 Note t= hat Poudriere defines the DISABLE_MAKE_JOB variable. .if !defined(DISABLE_MAKE_JOBS) && !defined(MAKE_JOBS_UNSAFE) CONFIGURE_ARGS+=3D --jobs=3D${MAKE_JOBS_NUMBER} _MAKE_JOBS+=3D --jobs=3D${MAKE_JOBS_NUMBER} .endif Therefore, CONFIGURE_ARGS and _MAKE_JOBS never get set to =E2=80=9C=E2=80= =94jobs=3D1=E2=80=9D and Python waf then assumes it should be =E2=80=9C=E2=80=94jobs=3D=E2=80=9D. Maybe it should be?... .if !defined(DISABLE_MAKE_JOBS) && !defined(MAKE_JOBS_UNSAFE) CONFIGURE_ARGS+=3D --jobs=3D${MAKE_JOBS_NUMBER} _MAKE_JOBS+=3D --jobs=3D${MAKE_JOBS_NUMBER} .else CONFIGURE_ARGS+=3D =E2=80=9C=E2=80=94jobs=3D1=E2=80=9D _MAKE_JOBS+=3D =E2=80=9C=E2=80=94jobs=3D1" .endif So maybe it is more like a documentation bug with waf? Best Regards, -stacey. -- View this message in context: http://freebsd.1045724.n5.nabble.com/python-w= af-bypasses-MAKE-JOBS-number-tp5978938p5979566.html Sent from the freebsd-python mailing list archive at Nabble.com. From owner-freebsd-python@FreeBSD.ORG Fri Jan 9 23:29:18 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E96731D; Fri, 9 Jan 2015 23:29:18 +0000 (UTC) Received: from smtprelay02.ispgateway.de (smtprelay02.ispgateway.de [80.67.18.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 376D6E04; Fri, 9 Jan 2015 23:29:17 +0000 (UTC) Received: from [80.67.16.130] (helo=webmailfront01.ispgateway.de) by smtprelay02.ispgateway.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84) (envelope-from ) id 1Y9izM-0007Oo-Ll; Sat, 10 Jan 2015 00:29:08 +0100 Received: from a89-182-73-111.net-htp.de (a89-182-73-111.net-htp.de [89.182.73.111]) by webmail.df.eu (Horde Framework) with HTTP; Sat, 10 Jan 2015 00:29:08 +0100 Date: Sat, 10 Jan 2015 00:29:08 +0100 Message-ID: <20150110002908.Horde.QrB60C0_PExk9ypkaJhFiA1@webmail.df.eu> From: Marcus von Appen To: sbruno@freebsd.org, Sean Bruno , sson@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> <54B019C1.7090704@ignoranthack.me> In-Reply-To: <54B019C1.7090704@ignoranthack.me> Reply-to: mva@freebsd.org User-Agent: Internet Messaging Program (IMP) H5 (6.0.4) Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-Df-Sender: ZnJlZWJzZEBzeXNmYXVsdC5vcmc= Cc: Antoine Brodin , python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 23:29:18 -0000 Sean Bruno : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 01/09/15 11:10, Antoine Brodin wrote: >> On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno >> wrote: >>> Marcus: >>> >>> The thing that I would like "fixed" is python waf ignoring the >>> fact that it should not try and detect the number of CPUs on the >>> system. >>> >> >> Let me rephrase the problem, there are some problems in some >> individual ports using waf. Those ports do not respect >> ${MAKE_JOBS_NUMBER}, especially when MAKE_JOBS_NUMBER=1. This >> breaks qemu which is not multi job friendly (people using qemu >> have DISABLE_MAKE_JOBS=yes which sets MAKE_JOBS_NUMBER to 1). >> >> Cheers, >> >> Antoine >> > > YES. :-) Sorry for the confusion. Just to be a bit more specific: we are not talking about a waf port, but about some (still unknown) waf scripts in the upstream package, correct? Since this is the usual way, waf is used. Please point us to the correct scripts/call hierarchy/whatever that allows us to investigate, what the port actually does at the build phase. waf issues usually need to be fixed (since it's somewhat similar to a cmake, scons or autotools configuration/build file) on the problematic port or via injection, hence we need the information about how the build is done or at least a pointer, where to look at. I'm leaving the threading issue aside here, since (for now) there is not enough information (for me) to address this. Let's get the build issue sorted out first, then take a look at Python's threading support for non-amd64 and non-i386 platforms on FreeBSD. Cheers Marcus From owner-freebsd-python@FreeBSD.ORG Sat Jan 10 00:20:05 2015 Return-Path: Delivered-To: python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 75D61BF5; Sat, 10 Jan 2015 00:20:05 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D89D2C1; Sat, 10 Jan 2015 00:20:04 +0000 (UTC) Received: from [10.12.76.105] (llnw-corp-src.phx2.llnw.com [69.164.56.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 13F84192A3B; Sat, 10 Jan 2015 00:20:03 +0000 (UTC) Message-ID: <54B07031.8050706@ignoranthack.me> Date: Fri, 09 Jan 2015 17:20:01 -0700 From: Sean Bruno Reply-To: sbruno@freebsd.org User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: mva@freebsd.org, sson@freebsd.org Subject: Re: python waf bypasses _MAKE_JOBS number References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> <54B019C1.7090704@ignoranthack.me> <20150110002908.Horde.QrB60C0_PExk9ypkaJhFiA1@webmail.df.eu> In-Reply-To: <20150110002908.Horde.QrB60C0_PExk9ypkaJhFiA1@webmail.df.eu> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: Antoine Brodin , python@freebsd.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jan 2015 00:20:05 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01/09/15 16:29, Marcus von Appen wrote: > Sean Bruno : > >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 >> >> On 01/09/15 11:10, Antoine Brodin wrote: >>> On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno >>> wrote: >>>> Marcus: >>>> >>>> The thing that I would like "fixed" is python waf ignoring >>>> the fact that it should not try and detect the number of CPUs >>>> on the system. >>>> >>> >>> Let me rephrase the problem, there are some problems in some >>> individual ports using waf. Those ports do not respect >>> ${MAKE_JOBS_NUMBER}, especially when MAKE_JOBS_NUMBER=1. This >>> breaks qemu which is not multi job friendly (people using >>> qemu have DISABLE_MAKE_JOBS=yes which sets MAKE_JOBS_NUMBER to >>> 1). >>> >>> Cheers, >>> >>> Antoine >>> >> >> YES. :-) Sorry for the confusion. > > Just to be a bit more specific: we are not talking about a waf > port, but about some (still unknown) waf scripts in the upstream > package, correct? Since this is the usual way, waf is used. Please > point us to the correct scripts/call hierarchy/whatever that allows > us to investigate, what the port actually does at the build phase. > > waf issues usually need to be fixed (since it's somewhat similar > to a cmake, scons or autotools configuration/build file) on the > problematic port or via injection, hence we need the information > about how the build is done or at least a pointer, where to look > at. > > I'm leaving the threading issue aside here, since (for now) there > is not enough information (for me) to address this. Let's get the > build issue sorted out first, then take a look at Python's > threading support for non-amd64 and non-i386 platforms on FreeBSD. > > Cheers Marcus > > The most obvious 2 ports that we've been fighting with are databases/tdb and devel/talloc They have some Makefile instructions that manifest this issue. Specifically: 74 .if !defined(DISABLE_MAKE_JOBS) && !defined(MAKE_JOBS_UNSAFE) 75 CONFIGURE_ARGS+= --jobs=${MAKE_JOBS_NUMBER} 76 _MAKE_JOBS+= --jobs=${MAKE_JOBS_NUMBER} 77 .else 78 CONFIGURE_ARGS+= --jobs=1 79 _MAKE_JOBS+= --jobs=1 80 .endif sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJUsHAlXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kA4EH/2LS9xhdeAvFm7fsBqsywB99 8rJIOK//c4ZdoBA+OwbTvIAAaIB6tSGrtLp5jJ5Ae15E9/unYcoleLmk3nBci0Wa 155v/oD8YO/cYA8MuKcGCuKFiHAQoggEKDfI8prosL2dGeZw+C+hZOxi6/92JHY3 oc0G8BC4hosuZA4oT0aAdmX6tX3kzCCRuli9c1luW9in5Vdn6ifuiToKSpK15k/W u7+BKFQQFi6766ahIDTEoQAKofVm9M8SWJZG2HmfGanChEACV+T7I/Xxcw5ygSfK vGwevvtJcLwUqQuFFLcTxhk8JQkeCt41lRwgTUDCEsoMlielO1U0QO0h6YnHw4c= =d8Ie -----END PGP SIGNATURE----- From owner-freebsd-python@FreeBSD.ORG Sat Jan 10 03:27:08 2015 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B152CBAB for ; Sat, 10 Jan 2015 03:27:08 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1.freebsd.org (Postfix) with ESMTP id 84F26901 for ; Sat, 10 Jan 2015 03:27:08 +0000 (UTC) Received: from msam.nabble.com (unknown [162.253.133.85]) by mwork.nabble.com (Postfix) with ESMTP id 67384100FE33 for ; Fri, 9 Jan 2015 19:27:07 -0800 (PST) Date: Fri, 9 Jan 2015 20:27:06 -0700 (MST) From: sson To: freebsd-python@freebsd.org Message-ID: <1420860426447-5979662.post@n5.nabble.com> In-Reply-To: <20150110002908.Horde.QrB60C0_PExk9ypkaJhFiA1@webmail.df.eu> References: <54AC74E0.4050404@ignoranthack.me> <20150107072132.GA1050@medusa.sysfault.org> <54AD5BB9.2060701@ignoranthack.me> <1420653060968-5979111.post@n5.nabble.com> <20150109183334.Horde.DDLUOEFT2TrMFUcP63h_9Q1@webmail.df.eu> <54B0154B.80801@ignoranthack.me> <54B019C1.7090704@ignoranthack.me> <20150110002908.Horde.QrB60C0_PExk9ypkaJhFiA1@webmail.df.eu> Subject: Re: python waf bypasses _MAKE_JOBS number MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jan 2015 03:27:08 -0000 Marcus von Appen-2 wrote > Sean Bruno < > sbruno@ > >: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> On 01/09/15 11:10, Antoine Brodin wrote: >>> On Fri, Jan 9, 2015 at 5:52 PM, Sean Bruno < > sbruno@ > > >>> wrote: >>>> Marcus: >>>> >>>> The thing that I would like "fixed" is python waf ignoring the >>>> fact that it should not try and detect the number of CPUs on the >>>> system. >>>> >>> >>> Let me rephrase the problem, there are some problems in some >>> individual ports using waf. Those ports do not respect >>> ${MAKE_JOBS_NUMBER}, especially when MAKE_JOBS_NUMBER=1. This >>> breaks qemu which is not multi job friendly (people using qemu >>> have DISABLE_MAKE_JOBS=yes which sets MAKE_JOBS_NUMBER to 1). >>> >>> Cheers, >>> >>> Antoine >>> >> >> YES. :-) Sorry for the confusion. > > Just to be a bit more specific: we are not talking about a waf port, > but about some (still unknown) waf scripts in the upstream package, > correct? Since this is the usual way, waf is used. > Please point us to the correct scripts/call hierarchy/whatever that > allows us to investigate, what the port actually does at the build phase. > > waf issues usually need to be fixed (since it's somewhat similar to > a cmake, scons or autotools configuration/build file) on the problematic > port or via injection, hence we need the information about how the build > is done or at least a pointer, where to look at. > > I'm leaving the threading issue aside here, since (for now) there is not > enough information (for me) to address this. Let's get the build issue > sorted out first, then take a look at Python's threading support for > non-amd64 and non-i386 platforms on FreeBSD. > > Cheers > Marcus > > > _______________________________________________ > freebsd-python@ > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-python > To unsubscribe, send any mail to " > freebsd-python-unsubscribe@ > " Yes, in the case of devel/talloc, at least, it seems that the waf scripts are part of the talloc distribution. I am not familiar enough with waf to say for sure. Therefore, like I said before, this may be more of a documentation bug given the number of ports that don't correctly add the "--jobs=1" for waf when DISABLE_MAKE_JOBS is defined. It seems that maybe the port maintainers used the same Makefile template when creating these ports. The correct fix seems to be to make sure the waf documentation clearly describes the default for "--jobs=" and fixing the ports that don't add the "--jobs=1" when DISABLE_MAKE_JOBS is defined. As for the threading bug that one may take a bit of work to find the race condition. It seems to be the same bug described in the old bug report for x86. Some was changed in the underlining thread code to fix that. If we knew what the change was then it might give us a clue where to look for arm and mips. In short, the state of the deadlock is a bunch of threads blocked in the "usem" state (posix semaphore sleep) and one thread blocked doing a pipe read (or "piperd"). The bug report https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=163467 seems to show the same thread state as what we are seeing. -stacey. -- View this message in context: http://freebsd.1045724.n5.nabble.com/python-waf-bypasses-MAKE-JOBS-number-tp5978938p5979662.html Sent from the freebsd-python mailing list archive at Nabble.com.