From owner-freebsd-ports@freebsd.org Fri Apr 12 10:42:09 2019 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8F3D1578B79 for ; Fri, 12 Apr 2019 10:42:08 +0000 (UTC) (envelope-from dimpase@gmail.com) Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BD956E599 for ; Fri, 12 Apr 2019 10:42:08 +0000 (UTC) (envelope-from dimpase@gmail.com) Received: by mail-lj1-x242.google.com with SMTP id v13so8419368ljk.4 for ; Fri, 12 Apr 2019 03:42:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=px1kqs+0gcvUH5bvyDPwIkDyIww7wErM0ayKsS4H9v0=; b=iOUOcNiJAxvxc752JglNpVqo7PVKnocaqPU7HdDBzKDUvWiuZvdRUEOT9PeRzl2sym 9EyVmgGK360/nFdgb6HKqMWoL4Llez2WoqmK3pLPYUGyMdmHzQecFCY6dGxwoN0GDlpH x8I5S2H6Mzdfv28L+IDdUuRbmcp/zk9Q0REuWQdHSp4gG3ouQTTJ94qa7m4d098saedg ZRgyGHhr20+jQ6+uVKC0cejTMzLQpbE0SFxjwsKoP7tB3I7IPOi7pnMQIYBiXkn+qZep sLJ7W9fQBu9E9onWSb0IKLO0uoXtK8FHK6trMA7UbCNo9d/MMBw0Jza8UXj0Lw0WByQI UV6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=px1kqs+0gcvUH5bvyDPwIkDyIww7wErM0ayKsS4H9v0=; b=gY/Xu6o6udoTm5lQTtXf6i/r2RDNJeGEvQlnCcnpMpCanbvM5n5VgIvbfMSzt7rjSz hK6skSe4VHNDXHH1HcKeOomg1yS3GkgYXCepmtyT+5Khm+PNVzEHqpKP7N6IHTNrBEJ7 o+pTATIgng+VZIVW2qgdOh92Xx0+DArVsUxG/SAM8rUvzOUJPKAMIjfkRdfQnbW78N1D 1bOSiBqKKkOHKS4HBmc8niM0oN94zinlNqtb916uJ0Pkj714gErPkpVozhhw7ZPjENcb 8bluXj9C/OxuQ2naY/7rpTOWG7Q80E1T+loHe8fCBBWme2KEHYwYshLaF3ByC17dPeXG CM8w== X-Gm-Message-State: APjAAAUcW/q1/fG4MFegp7vPAXYznhN8wnfjutTbY9iOH8omEezzN+jo 4CjQRJbFGzQSO9h3R8L4rRMNEgmYi6XmPnp08XTsvxAPPxA= X-Google-Smtp-Source: APXvYqyp8ZtHJMX6UjfbZ7SZR6KoaE/35ZcCuij5NgTEg326EAZ+5R+A4zdDhsJev6kFMD9MiwTuUzQnmHM0NmaLIcM= X-Received: by 2002:a2e:9d12:: with SMTP id t18mr29548352lji.163.1555065725657; Fri, 12 Apr 2019 03:42:05 -0700 (PDT) MIME-Version: 1.0 References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru> <20190412104531.7b492a3c@vm2.home.zagrebin.ru> In-Reply-To: <20190412104531.7b492a3c@vm2.home.zagrebin.ru> From: Dima Pasechnik Date: Fri, 12 Apr 2019 12:41:53 +0200 Message-ID: Subject: Re: python 3 subprocess performance To: Alexander Zagrebin Cc: FreeBSD Ports Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4BD956E599 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 10:42:09 -0000 On Fri, Apr 12, 2019 at 9:46 AM Alexander Zagrebin wrote= : > > =D0=92 Fri, 12 Apr 2019 09:36:13 +0200 > Dima Pasechnik =D0=BF=D0=B8=D1=88=D0=B5=D1=82= : > > > On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin > > wrote: > > > > > > =D0=92 Thu, 11 Apr 2019 17:32:42 +0200 > > > Jan Bramkamp =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > > > > > The reason is that that python does something stupid (tm). It > > > > tries to close all file descriptors (except a few whitelisted > > > > ones) up to the maximum file descriptor number. It does this by > > > > asking the kernel for the maximum possible number and closing > > > > everything it doesn't want to keep. Some time later someone came > > > > up with an optimization (read the open file descriptors > > > > from /dev/fd). All of this pain and suffering is caused by good > > > > old Ulrich Drepper braindamage: > > > > https://sourceware.org/bugzilla/show_bug.cgi?id=3D10353. > > > > > > > > Most Linux distros have lower default file descriptor limits than > > > > FreeBSD making this workaround less painful. The correct solution > > > > would be to teach python3 about closefrom(2). > > > > > > Thank you for hint and testing! > > > > > > Indeed the problem is in closing more than 400,000 file descriptors > > > in loop. It seems that all current versions of Python are affected. > > > Python2 uses False as default value for the close_fds parameter of > > > the Popen constructor, so this issue is mostly not visible. > > > Python3 has changed this default to True. > > > > > > As Jan Bramkamp suggested, I've wrote simple patch to fix an issue > > > (see attached file). It seems the problem has gone. > > > > The attachment has been stripped out. Could you paste the diff into > > the message? > > Yes, sure. > > --- Modules/_posixsubprocess.c.orig 2018-12-24 00:37:14.000000000 > +0300 +++ Modules/_posixsubprocess.c 2019-04-12 > 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@ > _close_fds_by_brute_force(long start_fd, } > start_fd =3D keep_fd + 1; > } > +#if defined(__FreeBSD__) > + closefrom(start_fd); > +#else > if (start_fd <=3D end_fd) { > for (fd_num =3D start_fd; fd_num < end_fd; ++fd_num) { > close(fd_num); > } > } > +#endif > } > > > If this is a Python issue, shouldn't this be reported upstream, on > > https://bugs.python.org ? > > May be. Rather, it is a FreeBSD-specific optimization. Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)), OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is there, I just checked) Anyway, FreeBSD Python maintainers will ask for an upstream PR. I can do such a PR is noone else is willing to... Dima > > -- > Alexander Zagrebin > _______________________________________________ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org"