From owner-freebsd-ports@FreeBSD.ORG Tue May 28 20:26:18 2013 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5ED214C7 for ; Tue, 28 May 2013 20:26:18 +0000 (UTC) (envelope-from matthias.andree@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) by mx1.freebsd.org (Postfix) with ESMTP id E54A538B for ; Tue, 28 May 2013 20:26:17 +0000 (UTC) Received: from mailout-de.gmx.net ([10.1.76.29]) by mrigmx.server.lan (mrigmx001) with ESMTP (Nemesis) id 0M6hJq-1UTJrz147Q-00wUez for ; Tue, 28 May 2013 22:26:11 +0200 Received: (qmail invoked by alias); 28 May 2013 20:26:11 -0000 Received: from f049073108.adsl.alicedsl.de (EHLO mandree.no-ip.org) [78.49.73.108] by mail.gmx.net (mp029) with SMTP; 28 May 2013 22:26:11 +0200 X-Authenticated: #428038 X-Provags-ID: V01U2FsdGVkX19wT85ZXjZwU2+17TMXWypmkNnsvBDBcE+Zikrkj/ 5ZhHPhCJML4L6C Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id 4DCA623CEC5 for ; Tue, 28 May 2013 22:26:09 +0200 (CEST) Message-ID: <51A512E1.1060406@gmx.de> Date: Tue, 28 May 2013 22:26:09 +0200 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-ports@freebsd.org Subject: Re: The vim port needs a refresh References: <20130524212318.B967FE6739@smtp.hushmail.com> <51A4ADCC.4070204@marino.st> <20130528151600.4eb6f028@gumby.homeunix.com> <20130528195137.312c58d9@bsd64.grem.de> In-Reply-To: <20130528195137.312c58d9@bsd64.grem.de> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 20:26:18 -0000 Am 28.05.2013 19:51, schrieb Michael Gmelin: > On Tue, 28 May 2013 15:16:00 +0100 > RW wrote: > >> On Tue, 28 May 2013 15:14:52 +0200 >> John Marino wrote: >> >>> All >>> patches only take 74 seconds to download[2] so there is no sympathy >>> for your obviously single data point anecdote, >> >> Well at the point you provided one data-point there was only one data >> point. And it was like pulling teeth to get you to eliminate the >> alternative explanations. Was it really too much to ask that you >> provided some actual evidence. >> >>> you're clearly doing >>> something wrong. You need to stop complaining and start think about >>> folks with slow connections[3] who also rebuild Vim frequently. >> >> Don't make things up. I never said anything about frequent rebuilds, >> the patches all get redownloaded on the next rebuild. > > The real issue is not the number of patches, but the fact that every > single patch is downloaded by invoking the fetch(1) command, creating > lots of overhead not limited to the fetch command itself. The ports > system wasn't designed for such an amount of distfiles in a single > port I guess. > > I just timed fetching the patches through ports vs. fetching over > HTTP/1.1 using ftp/curl vs calling fetch directly. The VIM tarball was > already downloaded, so this is really just the patches (plus > downloading 6mb is barely noticeable on a fast line). It's a slow > machine on a fast line. > > Fetch: > [user@server /usr/ports/editors/vim]$ time sudo make fetch > .... > real 4m57.327s > user 0m17.010s > sys 0m39.588s > > Curl: > [user@server /tmp]$ longcurlcommandline > .... > real 0m15.291s > user 0m0.026s > sys 0m0.272s > > Fetch on the command line (after initial make fetch, so this is only > measuring transmission of the files): > cd /usr/ports/editors/distfiles > time for name in 7.3.*; do > fetch http://artfiles.org/vim.org/patches/7.3/$name > done > .... > real 1m25.329s > user 0m0.660s > sys 0m3.174s > > So just the fact we're invoking fetch for every file costs us about one > minute - I assume the time lost is much bigger on a slow line with > long latency. The remaining 3.5 minutes are spent somewhere in the > ports infrastructure and clearly depend on the performance of the > machine used. For comparison I timed "make fetch" on a reasonably fast > server (good IO, fast datacenter connection), make fetch still took > about 120 seconds(!). > > So the bottomline is: > - Using HTTP/1.1 and keepalive could safe a lot of time > - The ports infrastructure creates a lot of overhead per patch file > > Maybe there's something we can do to improve the situation. Probably. On the fetching side, we have: - /usr/src/usr.sbin/portsnap/phttpget/phttpget.c & /usr/libexec/phttpget - the possibility to specify multiple URLs on fetch(1)'s command line - the xargs command to assemble command lines with a decent amount of URL arguments Given that connection setup for FTP costs considerable amounts of time especially with FTP. On the URL list generation, we have excessive external command in shell scripts; try "make -nC /usr/ports/editors/vim fetch-url-list-int" to see the commands. I suppose fewer external commands and more make-internal processing could help quite a bit.