From owner-svn-ports-all@freebsd.org Fri Nov 29 10:55:03 2019 Return-Path: Delivered-To: svn-ports-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 746481ABFFE; Fri, 29 Nov 2019 10:55:03 +0000 (UTC) (envelope-from koobs@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47PWcR34PXz3H9c; Fri, 29 Nov 2019 10:55:03 +0000 (UTC) (envelope-from koobs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 487A6D189; Fri, 29 Nov 2019 10:55:03 +0000 (UTC) (envelope-from koobs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xATAt3Io076655; Fri, 29 Nov 2019 10:55:03 GMT (envelope-from koobs@FreeBSD.org) Received: (from koobs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xATAt0YK076641; Fri, 29 Nov 2019 10:55:00 GMT (envelope-from koobs@FreeBSD.org) Message-Id: <201911291055.xATAt0YK076641@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: koobs set sender to koobs@FreeBSD.org using -f From: Kubilay Kocak Date: Fri, 29 Nov 2019 10:55:00 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r518640 - in head/lang: python27 python27/files python35 python35/files python36 python36/files python37 python37/files python38 python38/files X-SVN-Group: ports-head X-SVN-Commit-Author: koobs X-SVN-Commit-Paths: in head/lang: python27 python27/files python35 python35/files python36 python36/files python37 python37/files python38 python38/files X-SVN-Commit-Revision: 518640 X-SVN-Commit-Repository: ports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the ports tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 10:55:03 -0000 Author: koobs Date: Fri Nov 29 10:55:00 2019 New Revision: 518640 URL: https://svnweb.freebsd.org/changeset/ports/518640 Log: lang/python{27,35,36,37,38}: Add closefrom(2) support A single close(fd) syscall is cheap, but when MAXFDS (maximum file descriptor number) is high, the loop calling close(fd) on each file descriptor can take several milliseconds. The default value of subprocess.Popen "close_fds" parameter changed to True in Python 3. Compared to Python 2, close_fds=True can make Popen 10x slower: see bpo-37790 [1] The present workaround on FreeBSD to improve performance is to load and mount the fdescfs kernel module, but this is not enabled by default. This change adds minimum viable (and upstreamable) closefrom(2) syscall support to Python's subprocess and posix modules, improving performance significantly for loads that involve working with many processes, such as diffoscope, ansible, and many others. For additional optimizations, upstream recently (3.8) landed posix_spawn(2) support [3] and has stated that they will adopt close_range(2) after Linux merges it [4]. Linux/FreeBSD developers are already collaborating on ensuring compatible implementations, with FreeBSD's implementation pending in D21627. [5] Thank you emaste, cem, kevans for providing analysis, input, clarifications, comms/upstream support and patches. [1] https://bugs.python.org/issue37790 [2] https://bugs.python.org/issue38061 [3] https://bugs.python.org/issue35537 [4] https://lwn.net/Articles/789023/ [5] https://reviews.freebsd.org/D21627 Additional References: https://bugs.python.org/issue8052 https://bugs.python.org/issue11284 https://bugs.python.org/issue13788 https://bugs.python.org/issue1663329 https://www.python.org/dev/peps/pep-0446/ PR: 242274, 221700 Submitted by: kevans (emaste, cem) Approved by: koobs (python (maintainer), santa) Added: head/lang/python27/files/patch-Modules_posixmodule.c (contents, props changed) head/lang/python35/files/patch-Modules___posixsubprocess.c (contents, props changed) head/lang/python35/files/patch-Modules_posixmodule.c (contents, props changed) head/lang/python36/files/patch-Modules___posixsubprocess.c (contents, props changed) head/lang/python36/files/patch-Modules_posixmodule.c (contents, props changed) head/lang/python37/files/patch-Modules___posixsubprocess.c (contents, props changed) head/lang/python37/files/patch-Modules_posixmodule.c (contents, props changed) head/lang/python38/files/patch-Modules___posixsubprocess.c (contents, props changed) head/lang/python38/files/patch-Modules_posixmodule.c (contents, props changed) Modified: head/lang/python27/Makefile head/lang/python35/Makefile head/lang/python36/Makefile head/lang/python37/Makefile head/lang/python38/Makefile Modified: head/lang/python27/Makefile ============================================================================== --- head/lang/python27/Makefile Fri Nov 29 10:40:45 2019 (r518639) +++ head/lang/python27/Makefile Fri Nov 29 10:55:00 2019 (r518640) @@ -2,6 +2,7 @@ PORTNAME= python PORTVERSION= ${PYTHON_PORTVERSION} +PORTREVISION= 1 CATEGORIES= lang python MASTER_SITES= PYTHON/ftp/python/${PORTVERSION} PKGNAMESUFFIX= 27 Added: head/lang/python27/files/patch-Modules_posixmodule.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python27/files/patch-Modules_posixmodule.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,27 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/posixmodule.c.orig 2019-10-19 18:38:44 UTC ++++ Modules/posixmodule.c +@@ -6676,9 +6676,16 @@ posix_closerange(PyObject *self, PyObject *args) + if (!PyArg_ParseTuple(args, "ii:closerange", &fd_from, &fd_to)) + return NULL; + Py_BEGIN_ALLOW_THREADS +- for (i = fd_from; i < fd_to; i++) +- if (_PyVerify_fd(i)) +- close(i); ++#ifdef __FreeBSD__ ++ if (fd_to >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(fd_from); ++ } else ++#endif ++ { ++ for (i = fd_from; i < fd_to; i++) ++ if (_PyVerify_fd(i)) ++ close(i); ++ } + Py_END_ALLOW_THREADS + Py_RETURN_NONE; + } Modified: head/lang/python35/Makefile ============================================================================== --- head/lang/python35/Makefile Fri Nov 29 10:40:45 2019 (r518639) +++ head/lang/python35/Makefile Fri Nov 29 10:55:00 2019 (r518640) @@ -3,7 +3,7 @@ PORTNAME= python PORTVERSION= ${PYTHON_PORTVERSION} -PORTREVISION= 2 +PORTREVISION= 3 CATEGORIES= lang python MASTER_SITES= PYTHON/ftp/python/${PORTVERSION} PKGNAMESUFFIX= ${PYTHON_SUFFIX} Added: head/lang/python35/files/patch-Modules___posixsubprocess.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python35/files/patch-Modules___posixsubprocess.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,25 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/_posixsubprocess.c.orig 2019-11-01 23:02:34 UTC ++++ Modules/_posixsubprocess.c +@@ -233,8 +233,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_ + start_fd = keep_fd + 1; + } + if (start_fd <= end_fd) { +- for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { +- close(fd_num); ++#ifdef __FreeBSD__ ++ if (end_fd >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(start_fd); ++ } else ++#endif ++ { ++ for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { ++ close(fd_num); ++ } + } + } + } Added: head/lang/python35/files/patch-Modules_posixmodule.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python35/files/patch-Modules_posixmodule.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,27 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/posixmodule.c.orig 2019-11-01 23:02:34 UTC ++++ Modules/posixmodule.c +@@ -7853,9 +7853,16 @@ os_closerange_impl(PyObject *module, int fd_low, int f + int i; + Py_BEGIN_ALLOW_THREADS + _Py_BEGIN_SUPPRESS_IPH +- for (i = fd_low; i < fd_high; i++) +- if (_PyVerify_fd(i)) +- close(i); ++#ifdef __FreeBSD__ ++ if (fd_high >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(fd_low); ++ } else ++#endif ++ { ++ for (i = fd_low; i < fd_high; i++) ++ if (_PyVerify_fd(i)) ++ close(i); ++ } + _Py_END_SUPPRESS_IPH + Py_END_ALLOW_THREADS + Py_RETURN_NONE; Modified: head/lang/python36/Makefile ============================================================================== --- head/lang/python36/Makefile Fri Nov 29 10:40:45 2019 (r518639) +++ head/lang/python36/Makefile Fri Nov 29 10:55:00 2019 (r518640) @@ -3,6 +3,7 @@ PORTNAME= python PORTVERSION= ${PYTHON_PORTVERSION} +PORTREVISION= 1 CATEGORIES= lang python MASTER_SITES= PYTHON/ftp/python/${PORTVERSION} PKGNAMESUFFIX= ${PYTHON_SUFFIX} Added: head/lang/python36/files/patch-Modules___posixsubprocess.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python36/files/patch-Modules___posixsubprocess.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,25 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/_posixsubprocess.c.orig 2019-07-02 20:25:39 UTC ++++ Modules/_posixsubprocess.c +@@ -236,8 +236,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_ + start_fd = keep_fd + 1; + } + if (start_fd <= end_fd) { +- for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { +- close(fd_num); ++#ifdef __FreeBSD__ ++ if (end_fd >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(start_fd); ++ } else ++#endif ++ { ++ for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { ++ close(fd_num); ++ } + } + } + } Added: head/lang/python36/files/patch-Modules_posixmodule.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python36/files/patch-Modules_posixmodule.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,26 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/posixmodule.c.orig 2019-07-02 20:25:39 UTC ++++ Modules/posixmodule.c +@@ -7685,8 +7685,16 @@ os_closerange_impl(PyObject *module, int fd_low, int f + int i; + Py_BEGIN_ALLOW_THREADS + _Py_BEGIN_SUPPRESS_IPH +- for (i = Py_MAX(fd_low, 0); i < fd_high; i++) +- close(i); ++ fd_low = Py_MAX(fd_low, 0); ++#ifdef __FreeBSD__ ++ if (fd_high >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(fd_low); ++ } else ++#endif ++ { ++ for (i = fd_low; i < fd_high; i++) ++ close(i); ++ } + _Py_END_SUPPRESS_IPH + Py_END_ALLOW_THREADS + Py_RETURN_NONE; Modified: head/lang/python37/Makefile ============================================================================== --- head/lang/python37/Makefile Fri Nov 29 10:40:45 2019 (r518639) +++ head/lang/python37/Makefile Fri Nov 29 10:55:00 2019 (r518640) @@ -3,6 +3,7 @@ PORTNAME= python PORTVERSION= ${PYTHON_PORTVERSION} +PORTREVISION= 1 CATEGORIES= lang python MASTER_SITES= PYTHON/ftp/python/${PORTVERSION} PKGNAMESUFFIX= ${PYTHON_SUFFIX} Added: head/lang/python37/files/patch-Modules___posixsubprocess.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python37/files/patch-Modules___posixsubprocess.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,25 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/_posixsubprocess.c.orig 2019-10-14 22:32:36 UTC ++++ Modules/_posixsubprocess.c +@@ -236,8 +236,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_ + start_fd = keep_fd + 1; + } + if (start_fd <= end_fd) { +- for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { +- close(fd_num); ++#ifdef __FreeBSD__ ++ if (end_fd >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(start_fd); ++ } else ++#endif ++ { ++ for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { ++ close(fd_num); ++ } + } + } + } Added: head/lang/python37/files/patch-Modules_posixmodule.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python37/files/patch-Modules_posixmodule.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,26 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/posixmodule.c.orig 2019-10-14 22:32:36 UTC ++++ Modules/posixmodule.c +@@ -7810,8 +7810,16 @@ os_closerange_impl(PyObject *module, int fd_low, int f + int i; + Py_BEGIN_ALLOW_THREADS + _Py_BEGIN_SUPPRESS_IPH +- for (i = Py_MAX(fd_low, 0); i < fd_high; i++) +- close(i); ++ fd_low = Py_MAX(fd_low, 0); ++#ifdef __FreeBSD__ ++ if (fd_high >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(fd_low); ++ } else ++#endif ++ { ++ for (i = fd_low; i < fd_high; i++) ++ close(i); ++ } + _Py_END_SUPPRESS_IPH + Py_END_ALLOW_THREADS + Py_RETURN_NONE; Modified: head/lang/python38/Makefile ============================================================================== --- head/lang/python38/Makefile Fri Nov 29 10:40:45 2019 (r518639) +++ head/lang/python38/Makefile Fri Nov 29 10:55:00 2019 (r518640) @@ -3,7 +3,7 @@ PORTNAME= python PORTVERSION= ${PYTHON_PORTVERSION} -PORTREVISION= 1 +PORTREVISION= 2 CATEGORIES= lang python MASTER_SITES= PYTHON/ftp/python/${PORTVERSION} PKGNAMESUFFIX= ${PYTHON_SUFFIX} Added: head/lang/python38/files/patch-Modules___posixsubprocess.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python38/files/patch-Modules___posixsubprocess.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,25 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/_posixsubprocess.c.orig 2019-10-14 13:34:47 UTC ++++ Modules/_posixsubprocess.c +@@ -236,8 +236,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_ + start_fd = keep_fd + 1; + } + if (start_fd <= end_fd) { +- for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { +- close(fd_num); ++#if defined(__FreeBSD__) ++ if (end_fd >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(start_fd); ++ } else ++#endif ++ { ++ for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { ++ close(fd_num); ++ } + } + } + } Added: head/lang/python38/files/patch-Modules_posixmodule.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/lang/python38/files/patch-Modules_posixmodule.c Fri Nov 29 10:55:00 2019 (r518640) @@ -0,0 +1,26 @@ +# Add closefrom(2) support +# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274 +# https://bugs.python.org/issue38061 +# TODO: Upstream + +--- Modules/posixmodule.c.orig 2019-10-14 13:34:47 UTC ++++ Modules/posixmodule.c +@@ -8460,8 +8460,16 @@ os_closerange_impl(PyObject *module, int fd_low, int f + lohi[1] = fd_high; + fdwalk(_fdwalk_close_func, lohi); + #else +- for (i = Py_MAX(fd_low, 0); i < fd_high; i++) +- close(i); ++ fd_low = Py_MAX(fd_low, 0); ++#ifdef __FreeBSD__ ++ if (fd_high >= sysconf(_SC_OPEN_MAX)) { ++ closefrom(fd_low); ++ } else ++#endif ++ { ++ for (i = fd_low; i < fd_high; i++) ++ close(i); ++ } + #endif + _Py_END_SUPPRESS_IPH + Py_END_ALLOW_THREADS