Date: Fri, 12 Apr 2019 14:11:41 +0300 From: Alexander Zagrebin <alex@zagrebin.ru> To: Dima Pasechnik <dimpase+freebsd@gmail.com> Cc: freebsd-ports@freebsd.org Subject: Re: python 3 subprocess performance Message-ID: <20190412141141.0fa85252@vm2.home.zagrebin.ru> In-Reply-To: <CAAWYfq16Y90-nXeV3TkT10uh=_tK=Vgjk_QEJo=DM-P8KGFNQg@mail.gmail.com> References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru> <CAAWYfq2P5QsSbc2bzQauApF6=ijQYpKCb2HQ1jy-4dNZCOB_=g@mail.gmail.com> <20190412104531.7b492a3c@vm2.home.zagrebin.ru> <CAAWYfq16Y90-nXeV3TkT10uh=_tK=Vgjk_QEJo=DM-P8KGFNQg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
В Fri, 12 Apr 2019 12:41:53 +0200 Dima Pasechnik <dimpase+freebsd@gmail.com> пишет: > > > > В Thu, 11 Apr 2019 17:32:42 +0200 > > > > Jan Bramkamp <crest@rlwinm.de> пишет: > > > > > > > > > The reason is that that python does something stupid (tm). It > > > > > tries to close all file descriptors (except a few whitelisted > > > > > ones) up to the maximum file descriptor number. It does this > > > > > by asking the kernel for the maximum possible number and > > > > > closing everything it doesn't want to keep. Some time later > > > > > someone came up with an optimization (read the open file > > > > > descriptors from /dev/fd). All of this pain and suffering is > > > > > caused by good old Ulrich Drepper braindamage: > > > > > https://sourceware.org/bugzilla/show_bug.cgi?id=10353. > > > > > > > > > > Most Linux distros have lower default file descriptor limits > > > > > than FreeBSD making this workaround less painful. The correct > > > > > solution would be to teach python3 about closefrom(2). > > > > > > > > Thank you for hint and testing! > > > > > > > > Indeed the problem is in closing more than 400,000 file > > > > descriptors in loop. It seems that all current versions of > > > > Python are affected. Python2 uses False as default value for > > > > the close_fds parameter of the Popen constructor, so this issue > > > > is mostly not visible. Python3 has changed this default to True. > > > > > > > > As Jan Bramkamp suggested, I've wrote simple patch to fix an > > > > issue (see attached file). It seems the problem has gone. > > > > > > The attachment has been stripped out. Could you paste the diff > > > into the message? > > > > Yes, sure. > > > > --- Modules/_posixsubprocess.c.orig 2018-12-24 > > 00:37:14.000000000 +0300 +++ Modules/_posixsubprocess.c > > 2019-04-12 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@ > > _close_fds_by_brute_force(long start_fd, } > > start_fd = keep_fd + 1; > > } > > +#if defined(__FreeBSD__) > > + closefrom(start_fd); > > +#else > > if (start_fd <= end_fd) { > > for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { > > close(fd_num); > > } > > } > > +#endif > > } > > > > > If this is a Python issue, shouldn't this be reported upstream, on > > > https://bugs.python.org ? > > > > May be. Rather, it is a FreeBSD-specific optimization. > > Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)), > OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is > there, I just checked) > Anyway, FreeBSD Python maintainers will ask for an upstream PR. > > I can do such a PR is noone else is willing to... This would be good. Thanks! -- Alexander Zagrebin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190412141141.0fa85252>