From owner-freebsd-ports@freebsd.org Fri Apr 12 11:11:47 2019 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAF9F15796FB for ; Fri, 12 Apr 2019 11:11:47 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (mail.zagrebin.ru [IPv6:2001:470:1f15:30e::1:2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF11F6F52A for ; Fri, 12 Apr 2019 11:11:46 +0000 (UTC) (envelope-from alex@zagrebin.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru ; s=mail; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=awj2mlNeSv3iLdLldRGIEed5PlU3WfLJxN9ub2SGSug=; b=A2c2Am3mOpLWODLaVm8yBfgd5g C1oWlw891PFfRetM7Qkn3aJJvmUkaeMHoUqornXFUzkP8esiS7DpF3kgbA95cqvlalqA0QO5Fqsfx Z8ZuJqly+ZZmCKLonI0sbgD4P7X3Wk1JmCkn/kGS1kHU6bmp6gygqJxe/0iZ3nHdDL5Yn+KicKcub RzzVjP7kvnTRoTwoOLn7dFsAnsfetvBNiuWNqrNkf2sjx15DmLNiCq8l+ouSSol3KIE7v8P2mK7qL mzmy7w+w5kmuzbqlhbROfoEfafhk9GBqH5bp/O1Fg1pfdI3ZMSIiQ4kdjtW5yZu7dfvE5TtpKibYX BOLJJbBg==; Received: from [2001:470:1f15:30e::2] (helo=vm2.home.zagrebin.ru) by mail.zagrebin.ru with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hEu5u-000K8D-E7; Fri, 12 Apr 2019 14:11:42 +0300 Date: Fri, 12 Apr 2019 14:11:41 +0300 From: Alexander Zagrebin To: Dima Pasechnik Cc: freebsd-ports@freebsd.org Subject: Re: python 3 subprocess performance Message-ID: <20190412141141.0fa85252@vm2.home.zagrebin.ru> In-Reply-To: References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru> <20190412104531.7b492a3c@vm2.home.zagrebin.ru> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 11:11:48 -0000 В Fri, 12 Apr 2019 12:41:53 +0200 Dima Pasechnik пишет: > > > > В Thu, 11 Apr 2019 17:32:42 +0200 > > > > Jan Bramkamp пишет: > > > > > > > > > The reason is that that python does something stupid (tm). It > > > > > tries to close all file descriptors (except a few whitelisted > > > > > ones) up to the maximum file descriptor number. It does this > > > > > by asking the kernel for the maximum possible number and > > > > > closing everything it doesn't want to keep. Some time later > > > > > someone came up with an optimization (read the open file > > > > > descriptors from /dev/fd). All of this pain and suffering is > > > > > caused by good old Ulrich Drepper braindamage: > > > > > https://sourceware.org/bugzilla/show_bug.cgi?id=10353. > > > > > > > > > > Most Linux distros have lower default file descriptor limits > > > > > than FreeBSD making this workaround less painful. The correct > > > > > solution would be to teach python3 about closefrom(2). > > > > > > > > Thank you for hint and testing! > > > > > > > > Indeed the problem is in closing more than 400,000 file > > > > descriptors in loop. It seems that all current versions of > > > > Python are affected. Python2 uses False as default value for > > > > the close_fds parameter of the Popen constructor, so this issue > > > > is mostly not visible. Python3 has changed this default to True. > > > > > > > > As Jan Bramkamp suggested, I've wrote simple patch to fix an > > > > issue (see attached file). It seems the problem has gone. > > > > > > The attachment has been stripped out. Could you paste the diff > > > into the message? > > > > Yes, sure. > > > > --- Modules/_posixsubprocess.c.orig 2018-12-24 > > 00:37:14.000000000 +0300 +++ Modules/_posixsubprocess.c > > 2019-04-12 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@ > > _close_fds_by_brute_force(long start_fd, } > > start_fd = keep_fd + 1; > > } > > +#if defined(__FreeBSD__) > > + closefrom(start_fd); > > +#else > > if (start_fd <= end_fd) { > > for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { > > close(fd_num); > > } > > } > > +#endif > > } > > > > > If this is a Python issue, shouldn't this be reported upstream, on > > > https://bugs.python.org ? > > > > May be. Rather, it is a FreeBSD-specific optimization. > > Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)), > OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is > there, I just checked) > Anyway, FreeBSD Python maintainers will ask for an upstream PR. > > I can do such a PR is noone else is willing to... This would be good. Thanks! -- Alexander Zagrebin