From owner-freebsd-ports@FreeBSD.ORG Tue May 28 18:18:22 2013 Return-Path: Delivered-To: ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7899AD22 for ; Tue, 28 May 2013 18:18:22 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.grem.de (outcast.grem.de [213.239.217.27]) by mx1.freebsd.org (Postfix) with SMTP id D9F14CFE for ; Tue, 28 May 2013 18:18:20 +0000 (UTC) Received: (qmail 17429 invoked by uid 89); 28 May 2013 17:51:37 -0000 Received: from unknown (HELO bsd64.grem.de) (mg@grem.de@194.97.158.66) by mail.grem.de with ESMTPA; 28 May 2013 17:51:37 -0000 Date: Tue, 28 May 2013 19:51:37 +0200 From: Michael Gmelin To: ports@freebsd.org Subject: Re: The vim port needs a refresh Message-ID: <20130528195137.312c58d9@bsd64.grem.de> In-Reply-To: <20130528151600.4eb6f028@gumby.homeunix.com> References: <20130524212318.B967FE6739@smtp.hushmail.com> <51A4ADCC.4070204@marino.st> <20130528151600.4eb6f028@gumby.homeunix.com> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: RW X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 18:18:22 -0000 On Tue, 28 May 2013 15:16:00 +0100 RW wrote: > On Tue, 28 May 2013 15:14:52 +0200 > John Marino wrote: > > > All > > patches only take 74 seconds to download[2] so there is no sympathy > > for your obviously single data point anecdote, > > Well at the point you provided one data-point there was only one data > point. And it was like pulling teeth to get you to eliminate the > alternative explanations. Was it really too much to ask that you > provided some actual evidence. > > > you're clearly doing > > something wrong. You need to stop complaining and start think about > > folks with slow connections[3] who also rebuild Vim frequently. > > Don't make things up. I never said anything about frequent rebuilds, > the patches all get redownloaded on the next rebuild. The real issue is not the number of patches, but the fact that every single patch is downloaded by invoking the fetch(1) command, creating lots of overhead not limited to the fetch command itself. The ports system wasn't designed for such an amount of distfiles in a single port I guess. I just timed fetching the patches through ports vs. fetching over HTTP/1.1 using ftp/curl vs calling fetch directly. The VIM tarball was already downloaded, so this is really just the patches (plus downloading 6mb is barely noticeable on a fast line). It's a slow machine on a fast line. Fetch: [user@server /usr/ports/editors/vim]$ time sudo make fetch .... real 4m57.327s user 0m17.010s sys 0m39.588s Curl: [user@server /tmp]$ longcurlcommandline .... real 0m15.291s user 0m0.026s sys 0m0.272s Fetch on the command line (after initial make fetch, so this is only measuring transmission of the files): cd /usr/ports/editors/distfiles time for name in 7.3.*; do fetch http://artfiles.org/vim.org/patches/7.3/$name done .... real 1m25.329s user 0m0.660s sys 0m3.174s So just the fact we're invoking fetch for every file costs us about one minute - I assume the time lost is much bigger on a slow line with long latency. The remaining 3.5 minutes are spent somewhere in the ports infrastructure and clearly depend on the performance of the machine used. For comparison I timed "make fetch" on a reasonably fast server (good IO, fast datacenter connection), make fetch still took about 120 seconds(!). So the bottomline is: - Using HTTP/1.1 and keepalive could safe a lot of time - The ports infrastructure creates a lot of overhead per patch file Maybe there's something we can do to improve the situation. Cheers, Michael PS: I don't use vim myself and have no stake in this discussion whatsoever. -- Michael Gmelin