From owner-freebsd-ports@FreeBSD.ORG Sun Nov 23 04:48:25 2014 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9A50A84C for ; Sun, 23 Nov 2014 04:48:25 +0000 (UTC) Received: from mail-wg0-x22b.google.com (mail-wg0-x22b.google.com [IPv6:2a00:1450:400c:c00::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2EA0E237 for ; Sun, 23 Nov 2014 04:48:25 +0000 (UTC) Received: by mail-wg0-f43.google.com with SMTP id l18so9723743wgh.2 for ; Sat, 22 Nov 2014 20:48:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=sDHwQl5NISMN3VMUqK3+6E0/3qWXofci+Ll11w8Pk7w=; b=Up78Mk0jxfCx1nwCtluYP9nFExa5+QBdjT1mDH11u/Fwn+YFPDnkI1ohHbabBh0J4/ ajCNgyhu5pYHXM+8QVOCjJoHws7drkHv5tmFFUR2xKNjDaPEgqZxzpSJLxq0RCEUlQ7a jD71YuP2aVTnv5xFFrz7xIF1g1n8/YEJq9ImhNUeaxxbYPRqbidgzyQ6+rhKWeuvZhzr OGEYE/+OPbCG2wq9TH1ZwgONS/xsABRW10/N38ryb28sod/UiSYx1xcsjct19A3pUoWp RDWWK19aJ8Q8XLTOR0+Tkvh5v1jtMbXrX/FNXDbxDRk6bxP8G6KI58uW0bg22fr1o9Sb PoJA== MIME-Version: 1.0 X-Received: by 10.194.172.4 with SMTP id ay4mr22493991wjc.13.1416718103581; Sat, 22 Nov 2014 20:48:23 -0800 (PST) Received: by 10.27.46.14 with HTTP; Sat, 22 Nov 2014 20:48:23 -0800 (PST) In-Reply-To: <54710832.8070709@aldan.algebra.com> References: <54710832.8070709@aldan.algebra.com> Date: Sat, 22 Nov 2014 22:48:23 -0600 Message-ID: Subject: Re: Downloading DISTFILES from multiple locations in parallel (torrent or ...)? From: Scot Hetzel To: "Mikhail T." Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-ports@FreeBSD.ORG" X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2014 04:48:25 -0000 On Sat, Nov 22, 2014 at 4:03 PM, Mikhail T. wrote: > Has anybody given the idea any thought? Though in addition to the > "upstream" MASTER_SITES, there is a (fast) collection of FreeBSD > mirrors, the downloads currently aren't happening in parallel -- they > are serial... > > One solution would be for the downloader (fetch(1) or whatever) to try > to open multiple connections -- to several possible sites -- and > download different ranges of the same tarball from different locations > in parallel thus affording the total download rate of the actually > fastest mirror (almost), rather than the one first on the list. > > The alternative would use one of the existing torrent-clients, but we'd > need to think of how to publish the torrent-information with the ports. > It would seem, this second plan can go further even if the first would > require fewer changes to the existing infrastructure. > > Any thoughts? > I found this article on using the ftp/axel port to perform multiple downloads for a file: http://scratching.psybermonkey.net/2009/09/freebsd-download-ports-simultaneously.html There is only one problem, the ports infrastructure only specifies one download location for each file to the FETCH_CMD, to make it useful. You would want to specify multiple download locations. According to the axel web site: http://axel.alioth.debian.org/ it can use multiple mirrors to get the file. Someone would have to re-code the the make fetch target so that multiple sites are specified when axel is used: axel -n 4 -a ftp://ftp1.freebsd.org/path/to/file.tgz ftp://ftp.mirror.org/path/to/file.tgz The above would open 4 connections, (2 connections to each site). -- DISCLAIMER: No electrons were maimed while sending this message. Only slightly bruised.