From owner-freebsd-ports@FreeBSD.ORG Mon Oct 8 08:47:57 2007 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF42516A418 for ; Mon, 8 Oct 2007 08:47:57 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id B46FB13C458 for ; Mon, 8 Oct 2007 08:47:57 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 81D6F1CC065; Mon, 8 Oct 2007 01:47:57 -0700 (PDT) Date: Mon, 8 Oct 2007 01:47:57 -0700 From: Jeremy Chadwick To: obrien@freebsd.org Message-ID: <20071008084757.GA59389@eos.sc1.parodius.com> Mail-Followup-To: obrien@freebsd.org, freebsd-ports@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-ports@freebsd.org Subject: editors/vim -- long fetch times X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2007 08:47:57 -0000 This is a subject I've wanted to bring up for a few years now. Why I decided to bring it up now, I don't know; I guess I just felt it was time. Simply put: the time it takes for "make fetch" to complete in editors/vim gets worse and worse as more patches are released: icarus# cd /usr/ports/editors/vim-lite && make distclean icarus# time make fetch ... 10.801u 3.297s 6:16.33 3.7% 568+1347k 0+52io 0pf+0w That's over 6 minutes to fetch 116 patches and the base vim tarball, totalling about 7MBytes of data: icarus# du -sk /usr/ports/distfiles/vim 7248 /usr/ports/distfiles/vim True, it's only excessive when you have none of the patches in DISTFILES/vim (that is to say, if you already have 115 patches and there's 116 of them, the wait time isn't very long). The "wasted" time is caused by a few things. This opens somewhat of a can of worms (in regards to how to enhance fetch(1)), but the main two: * FTP protocol is used for each and every patch. FTP server may be doing things like forward and reverse DNS look-ups on client, as well as an RFC931/1413 ident check. * fetch(1) does not support the ability to fetch multiple files during the same FTP session. The idea would be to modify fetch(1) to support doing multiple RETRs during a single FTP session. Solutions as I see them: * Change the first site in MASTER_SITE_VIM to the HTTP version of the main distribution site: http://ftp.vim.org/pub/vim/unix/. This is already in the list, but the HTTP mirrors are *last*, not first. * Provide "snapshots" of the vim patches as a tgz-ball somewhere. Have "make fetch" download that and untar it into DISTFILES/vim/. Snapshots would need to be updated every time a new patch was added. * Modify fetch(1) as described above. * Get Bram to stop releasing hundreds of individual patches between minor releases. :-) Using the HTTP fetch method, the time drastically decreases to 1.5 minutes for me: icarus# cd /usr/ports/editors/vim-lite && make distclean icarus# time make MASTER_SITE_VIM=http://ftp.vim.org/pub/vim/unix/ fetch 9.282u 2.747s 1:28.83 13.5% 628+1444k 1+52io 0pf+0w Thoughts? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |