From owner-freebsd-python@freebsd.org Sun May 7 15:20:39 2017 Return-Path: Delivered-To: freebsd-python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0202D616D2 for ; Sun, 7 May 2017 15:20:39 +0000 (UTC) (envelope-from portscout@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CD22D7F6 for ; Sun, 7 May 2017 15:20:39 +0000 (UTC) (envelope-from portscout@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id CC508D616D0; Sun, 7 May 2017 15:20:39 +0000 (UTC) Delivered-To: python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBE6FD616CF for ; Sun, 7 May 2017 15:20:39 +0000 (UTC) (envelope-from portscout@FreeBSD.org) Received: from portscout.ysv.freebsd.org (portscout.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B7A127F3 for ; Sun, 7 May 2017 15:20:39 +0000 (UTC) (envelope-from portscout@FreeBSD.org) Received: from portscout.ysv.freebsd.org ([127.0.1.123]) by portscout.ysv.freebsd.org (8.15.2/8.15.2) with ESMTP id v47FKddR059999 for ; Sun, 7 May 2017 15:20:39 GMT (envelope-from portscout@FreeBSD.org) Received: (from portscout@localhost) by portscout.ysv.freebsd.org (8.15.2/8.15.2/Submit) id v47FKdOJ059998; Sun, 7 May 2017 15:20:39 GMT (envelope-from portscout@FreeBSD.org) Message-Id: <201705071520.v47FKdOJ059998@portscout.ysv.freebsd.org> X-Authentication-Warning: portscout.ysv.freebsd.org: portscout set sender to portscout@FreeBSD.org using -f Content-Disposition: inline Content-Transfer-Encoding: 8bit Content-Type: text/plain MIME-Version: 1.0 Date: Sun, 7 May 2017 15:20:39 +0000 From: portscout@FreeBSD.org To: python@freebsd.org Subject: FreeBSD ports you maintain which are out of date X-Mailer: portscout/0.8.1 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2017 15:20:40 -0000 Dear port maintainer, The portscout new distfile checker has detected that one or more of your ports appears to be out of date. Please take the opportunity to check each of the ports listed below, and if possible and appropriate, submit/commit an update. If any ports have already been updated, you can safely ignore the entry. You will not be e-mailed again for any of the port/version combinations below. Full details can be found at the following URL: http://portscout.freebsd.org/python@freebsd.org.html Port | Current version | New version ------------------------------------------------+-----------------+------------ www/py-django | 1.8.7 | 1.11.1 ------------------------------------------------+-----------------+------------ If any of the above results are invalid, please check the following page for details on how to improve portscout's detection and selection of distfiles on a per-port basis: http://portscout.freebsd.org/info/portscout-portconfig.txt Thanks. From owner-freebsd-python@freebsd.org Sun May 7 21:00:09 2017 Return-Path: Delivered-To: freebsd-python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3915D53BA2 for ; Sun, 7 May 2017 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 254B3E for ; Sun, 7 May 2017 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 14525D53BA1; Sun, 7 May 2017 21:00:09 +0000 (UTC) Delivered-To: python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13D7ED53BA0 for ; Sun, 7 May 2017 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DC75FA for ; Sun, 7 May 2017 21:00:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v47L01uf049612 for ; Sun, 7 May 2017 21:00:08 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201705072100.v47L01uf049612@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: python@FreeBSD.org Subject: Problem reports for python@FreeBSD.org that need special attention Date: Sun, 07 May 2017 21:00:08 +0000 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2017 21:00:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 205308 | devel/py-pip and devel/py-virtualenv don't aggree Open | 211641 | graphics/py-mayavi mayavi2 runtime error Open | 217703 | lang/python36: Remove BROKEN_SSL for openssl-deve Open | 204982 | [NEW PORT] mail/py-alot: Terminal MUA using notmu 4 problems total for which you should take action. From owner-freebsd-python@freebsd.org Tue May 9 18:23:48 2017 Return-Path: Delivered-To: freebsd-python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B04A4D65545 for ; Tue, 9 May 2017 18:23:48 +0000 (UTC) (envelope-from pmhausen@me.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 94632A3F for ; Tue, 9 May 2017 18:23:48 +0000 (UTC) (envelope-from pmhausen@me.com) Received: by mailman.ysv.freebsd.org (Postfix) id 93AE8D65544; Tue, 9 May 2017 18:23:48 +0000 (UTC) Delivered-To: python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92D12D65543 for ; Tue, 9 May 2017 18:23:48 +0000 (UTC) (envelope-from pmhausen@me.com) Received: from pv38p43im-ztdg05061201.me.com (pv38p43im-ztdg05061201.me.com [17.133.183.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7972BA3E for ; Tue, 9 May 2017 18:23:48 +0000 (UTC) (envelope-from pmhausen@me.com) Received: from process-dkim-sign-daemon.pv38p43im-ztdg05061201.me.com by pv38p43im-ztdg05061201.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) id <0OPP00N0045YJ700@pv38p43im-ztdg05061201.me.com> for python@freebsd.org; Tue, 09 May 2017 17:23:42 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=4d515a; t=1494350622; bh=BF7AIP2YpI9L75Uu8FFR4SW4495E2dQjlEjMhTQTexU=; h=From:Content-type:MIME-version:Subject:Message-id:Date:To; b=dXeHRE/vtUFig5QBqHN2hO6W+4ZFSto1xU45pChNbyLf2UCAxY6ASThqZEe7TtdsU SSInezyrm3JSB5+2xqRXf2RItOxsu1wcpj1V7k6Ssts7IOzE8OFv88hFgQ+P84CvlJ t9LEUQ68J5BBKW+cgeqpUtl5w1K/EWu3cmemTU7e5ZNbNO7AimpC7z0WTaWchyVSQZ VbZQG7i9Y27ccodVrzeQh1qHOkGmKsWpTE0FbyJP2zLc6nPZotlcRzloIIG8UlCgK0 EA5xWsQbhy+apy7Ak0mqFrpeIS64sEOZkcvwYMW9m+EFHL6oADms2DOw/itru2bbmu XJxEfJ2eaCv6Q== Received: from icloud.com ([127.0.0.1]) by pv38p43im-ztdg05061201.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) with ESMTPSA id <0OPP007R94BFBW30@pv38p43im-ztdg05061201.me.com> for python@freebsd.org; Tue, 09 May 2017 17:23:42 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-09_14:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1034 suspectscore=14 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1701120000 definitions=main-1705090094 From: pmhausen Content-type: text/plain; charset=utf-8 Content-transfer-encoding: quoted-printable MIME-version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Proposal for pkg-message for python 3.x ports Message-id: <36BF7EEF-EE06-4A70-B0F2-69D1454CD1FF@me.com> Date: Tue, 09 May 2017 19:23:38 +0200 To: python@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2017 18:23:48 -0000 Dear python port maintainers, I discovered an odd performance problem with python 3.6 on servers with large amounts of memory. You can find all the details in a thread of the help@python.org mailinglist with subject "Python bug? Closing *all* file descriptors on fork() instead of only = open ones?" Unfortunately I cannot get at the list archives. Otherwise I would have = sent you a direct link. Short version: since python 3.3 (if my google-fu worked) python tries to close all open file descriptors in the child after fork(). Since there = is no standard POSIX compliant way to do this for *open* descriptors only, it tries to use /dev/fd on FreeBSD and if that is not available it = closes *all* descriptors (resulting in ERR#9 for the majority). On a 64 bit = FreeBSD 11 system with 128 GB of RAM the auto-tuned kern.maxfilesperproc is between 3 and 4 million. That's a heck of a lot of system calls for = each fork() ;-) I suggest adding a hint to mount fdescfs and add it to /etc/fstab in the pkg-message for the affected python ports. I'll attach my first message to the list below. The solution is to mount = /dev/fd. Kind regards, Patrick ----------------------- Hi, all, last week we encountered an odd behaviour of a particular python = application on a freshly deployed server. The application in question is: = https://github.com/iocage/iocage On our test system (SMALLBOX) everything seemed to be working fine while = on the new designated production system (BIGBOX) every invocation of iocage = was incredibly slow =E2=80=A6 This was suprising since BIGBOX features 20 cores, 128 GB of RAM, SSD = only while SMALLBOX is a run-off-the-mill server with 4 cores, 16 GB of RAM = and spinning drives we used for development. I finally found that the problem may not lie with iocage. To reproduce = with just python installed: SMALLBOX: $ uname -a FreeBSD testhost.proserver.punkt.de 11.0-RELEASE-p10 FreeBSD = 11.0-RELEASE-p10 #2 r317823: Fri May 5 11:41:45 CEST 2017 = root@testhost.proserver.punkt.de:/usr/obj/usr/src/sys/VIMAGE amd64 $ python3.6 -V Python 3.6.1 $ time pip3 list [=E2=80=A6] real 0m0.982s user 0m0.614s sys 0m0.370s BIGBOX: $ uname -a FreeBSD ph002.proserver.punkt.de 11.0-RELEASE-p10 FreeBSD = 11.0-RELEASE-p10 #2 r317823: Fri May 5 10:55:27 CEST 2017 = root@ph002.proserver.punkt.de:/usr/obj/usr/src/sys/VIMAGE amd64 $ python3.6 -V Python 3.6.1 $ time pip3 list real 0m4.140s user 0m1.242s sys 0m2.900s Well =E2=80=A6 something=E2=80=99s fishy. After double checking OS and = python versions, SSD configuration, CPU and I/O load, being a sysadmin and not really a developer I decided to = use truss to have a look at what the systems were actually *doing*. Result: on every fork of a python process we get: 77906: close(4) ERR#9 'Bad file = descriptor' 77906: close(5) ERR#9 'Bad file = descriptor' 77906: close(6) ERR#9 'Bad file = descriptor' 77906: close(7) ERR#9 'Bad file = descriptor' 77906: close(9) ERR#9 'Bad file = descriptor' 77906: close(10) ERR#9 'Bad file = descriptor' 77906: close(11) ERR#9 'Bad file = descriptor=E2=80=98 [=E2=80=A6 continued for literally millions of calls ...] I=E2=80=99m perfectly familiar with the concept of closing *open* file = descriptors on fork/exec but closing *all* of them regardless of whether they are in use or not means that on = the two systems we get this number of system calls: SMALLBOX: $ truss -f pip3 list 2>&1 | grep 'ERR#9' | wc -l 1877703 BIGBOX: $ truss -f pip3 list 2>&1 | grep 'ERR#9' | wc -l 15082323 Which rather nicely corresponds to the number of file descriptors per = process on each system multiplied by 4, since the initial python process seems to fork 4 times = to get the pip3 job done: SMALLBOX: $ sysctl kern.maxfilesperproc kern.maxfilesperproc: 469431 $ dc 469431 4 * p 1877724 BIGBOX: $ sysctl kern.maxfilesperproc kern.maxfilesperproc: 3770586 $ dc 3770586 4 * p 15082344 OK - obviously even on a system like BIGBOX 15 million invocations of = close() take a couple of seconds. And this will get worse with 256 or 512 GB of RAM since = kernel tables will dynamically increase accordingly. What now? Is this a bug? It should be sufficient to close descriptors 0 = to 3 after fork. Which is precisely what the invoking shell does before exec()ing python3.6 =E2=80= =A6 Thanks in advance for any hints Patrick= From owner-freebsd-python@freebsd.org Tue May 9 19:13:21 2017 Return-Path: Delivered-To: freebsd-python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D506DD666D0 for ; Tue, 9 May 2017 19:13:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BE0A71112 for ; Tue, 9 May 2017 19:13:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id BCCD4D666CF; Tue, 9 May 2017 19:13:21 +0000 (UTC) Delivered-To: python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC6F1D666CE for ; Tue, 9 May 2017 19:13:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3488D1111 for ; Tue, 9 May 2017 19:13:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v49JDFbA092113 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 9 May 2017 22:13:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v49JDFbA092113 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v49JDE9q092112; Tue, 9 May 2017 22:13:14 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 9 May 2017 22:13:14 +0300 From: Konstantin Belousov To: pmhausen Cc: python@freebsd.org Subject: Re: Proposal for pkg-message for python 3.x ports Message-ID: <20170509191314.GK1622@kib.kiev.ua> References: <36BF7EEF-EE06-4A70-B0F2-69D1454CD1FF@me.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <36BF7EEF-EE06-4A70-B0F2-69D1454CD1FF@me.com> User-Agent: Mutt/1.8.2 (2017-04-18) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2017 19:13:21 -0000 On Tue, May 09, 2017 at 07:23:38PM +0200, pmhausen wrote: > Dear python port maintainers, > > I discovered an odd performance problem with python 3.6 on servers with > large amounts of memory. > > You can find all the details in a thread of the help@python.org > mailinglist with subject > > "Python bug? Closing *all* file descriptors on fork() instead of only open ones?" > > Unfortunately I cannot get at the list archives. Otherwise I would have sent > you a direct link. > > Short version: since python 3.3 (if my google-fu worked) python tries to > close all open file descriptors in the child after fork(). Since there is > no standard POSIX compliant way to do this for *open* descriptors only, > it tries to use /dev/fd on FreeBSD and if that is not available it closes > *all* descriptors (resulting in ERR#9 for the majority). On a 64 bit FreeBSD > 11 system with 128 GB of RAM the auto-tuned kern.maxfilesperproc > is between 3 and 4 million. That's a heck of a lot of system calls for each > fork() ;-) > > I suggest adding a hint to mount fdescfs and add it to /etc/fstab in the > pkg-message for the affected python ports. We have closefrom(2), which is semi-portable, in the sense that it is available not only on FreeBSD.