From owner-freebsd-current@FreeBSD.ORG Thu May 17 15:49:58 2012 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3188106564A; Thu, 17 May 2012 15:49:58 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) by mx1.freebsd.org (Postfix) with ESMTP id 7DDA98FC19; Thu, 17 May 2012 15:49:58 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id D38551DD4FB; Thu, 17 May 2012 17:49:56 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id BAC782847A; Thu, 17 May 2012 17:49:56 +0200 (CEST) Date: Thu, 17 May 2012 17:49:56 +0200 From: Jilles Tjoelker To: Dimitry Andric Message-ID: <20120517154956.GA63826@stack.nl> References: <4FB4EB74.5040009@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4FB4EB74.5040009@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: bf1783@gmail.com, freebsd-current@FreeBSD.org, Peter Jeremy , "b. f." Subject: Re: "make delete-old" performance. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2012 15:49:59 -0000 On Thu, May 17, 2012 at 02:13:40PM +0200, Dimitry Andric wrote: > On 2012-05-17 05:18, b. f. wrote:... > > The slowdown is probably due - at least in part - to two factors: > > - the list of files to be checked for removal has grown substantially, > > because missing entries for old knobs and new entries for new knobs > > have been added; and > > - a new (and slower) method of checking was added in: > > http://svnweb.FreeBSD.org/base?view=revision&revision=220255 > > because the old method broke down with the size of the new lists of files. > Hm, maybe it would have been better to fix make, so it can accept > arbitrarily long lists, without segfaulting? There's such a thing as > malloc(), and I simply don't believe any of those lists could be larger > than a few hundred kilobytes. Alternatively, make could be fixed so that the original code works. Although an invocation like sh -c 'for file in VERY_LONG_LIST; do something; done' will bump into {ARG_MAX}, the shell itself does not have a fixed limitation so longer command lines can be written to a temporary file and passed to sh that way. In some cases (such as with -j), make always uses a temporary file, slowing things down and obscuring ps output. At the cost of needing the temporary file named a bit longer, it is better to pass the pathname to sh rather than feeding the script on standard input: this avoids interfering with terminal input and is potentially faster. The code currently in Makefile.inc1 can probably be sped up by passing the output of the make -V command to something like xargs sh -c 'for file do rm -i "${DESTDIR}/${file}"; done' sh instead of the xargs -n1 | while read file; do ...; done loop. (Note the second "sh" at the end, which serves as a value for $0 so all strings from xargs become positional parameters.) -- Jilles Tjoelker