Date: Sat, 22 Nov 2014 22:48:23 -0600 From: Scot Hetzel <swhetzel@gmail.com> To: "Mikhail T." <mi+thun@aldan.algebra.com> Cc: "freebsd-ports@FreeBSD.ORG" <freebsd-ports@freebsd.org> Subject: Re: Downloading DISTFILES from multiple locations in parallel (torrent or ...)? Message-ID: <CACdU%2Bf_oeKRpm_NJsevF%2BugiK_4D2MKZt=9VjT9YoHTvihiWTw@mail.gmail.com> In-Reply-To: <54710832.8070709@aldan.algebra.com> References: <54710832.8070709@aldan.algebra.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Nov 22, 2014 at 4:03 PM, Mikhail T. <mi+thun@aldan.algebra.com> wrote: > Has anybody given the idea any thought? Though in addition to the > "upstream" MASTER_SITES, there is a (fast) collection of FreeBSD > mirrors, the downloads currently aren't happening in parallel -- they > are serial... > > One solution would be for the downloader (fetch(1) or whatever) to try > to open multiple connections -- to several possible sites -- and > download different ranges of the same tarball from different locations > in parallel thus affording the total download rate of the actually > fastest mirror (almost), rather than the one first on the list. > > The alternative would use one of the existing torrent-clients, but we'd > need to think of how to publish the torrent-information with the ports. > It would seem, this second plan can go further even if the first would > require fewer changes to the existing infrastructure. > > Any thoughts? > I found this article on using the ftp/axel port to perform multiple downloads for a file: http://scratching.psybermonkey.net/2009/09/freebsd-download-ports-simultaneously.html There is only one problem, the ports infrastructure only specifies one download location for each file to the FETCH_CMD, to make it useful. You would want to specify multiple download locations. According to the axel web site: http://axel.alioth.debian.org/ it can use multiple mirrors to get the file. Someone would have to re-code the the make fetch target so that multiple sites are specified when axel is used: axel -n 4 -a ftp://ftp1.freebsd.org/path/to/file.tgz ftp://ftp.mirror.org/path/to/file.tgz The above would open 4 connections, (2 connections to each site). -- DISCLAIMER: No electrons were maimed while sending this message. Only slightly bruised.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACdU%2Bf_oeKRpm_NJsevF%2BugiK_4D2MKZt=9VjT9YoHTvihiWTw>