Date: Tue, 9 Nov 2010 11:23:18 -0400 From: Jason Harris <jharris@widomaker.com> To: Dominic Fandrey <kamikaze@bsdforen.de> Cc: freebsd-hubs@freebsd.org, Oliver Fromme <olli@lurza.secnetix.de> Subject: Re: /pub/FreeBSD or /FreeBSD Message-ID: <20101109152318.GA59055@laptop> In-Reply-To: <4CD94DB1.3030809@bsdforen.de> References: <201011091248.oA9CmFB9075923@lurza.secnetix.de> <4CD94DB1.3030809@bsdforen.de>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Tue, Nov 09, 2010 at 02:33:37PM +0100, Dominic Fandrey wrote: > On 09/11/2010 13:48, Oliver Fromme wrote: > > It depends very much on how the protocols are implemented > > by server and client. If you use HTTP and *both* server > > and client support keep-alive, all files can be downloaded > > with a single connection. Otherwise you need n connections > > (e.g. 77 connections for 77 files). Some clients also open > > several connections in parallel. I have been testing aria2c (ports/www/aria2) for parallel-fetching (1 connection per site, up to 32 sites) FreeBSD ports distfiles with the following ~/.aria2/aria2.conf: server-stat-if=server-stats server-stat-of=server-stats min-split-size=1M allow-overwrite=true uri-selector=adaptive max-concurrent-downloads=32 timeout=15 connect-timeout=15 lowest-speed-limit=40K remote-time=true server-stat-timeout=86313600 split=32 enable-http-pipelining=true use-head=true event-poll=kqueue file-allocation=none log-level=notice (NB: This creates ./server-stats, usually in /usr/ports/distfiles, with a few changes to /etc/make.conf and a patched ports/Mk/bsd.port.mk.) It works quite well, often saturating my 4Mbit (down) link. > My application tries to always have one running download per > mirror (if there are enough files to be downloaded, at least). > > I'd have to give up the single queue approach if I wanted to > use a keep-alive capability, if fetch supported it. And I > think single queue is the best approach to make the best out > of my internet connection. You can control instances of aria2c with XML-RPC (samples included in the man page). That would keep connections open and actively downloading for as long as possible. If you use one instance per mirror, use a different server-stats and you can check each mirror's overall performance when each aria2c finally exit()s. > I measured that it doesn't matter at all for downloads like > openoffice, this supports your assessment. OpenOffice uses MirrorBrain ( http://www.mirrorbrain.org/users/ ), which generates Metalink and BitTorrent files for built-in parallel downloading. Also, try "make fetch-url-list | xargs aria2c" in ports/www/firefox to see what is possible with a small patch to FreeBSD's existing ports infrastructure. -- Jason Harris | PGP: This _is_ PGP-signed, isn't it? jharris@widomaker.com _|_ Got photons? (TM), (C) 2004 [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iJ0EARECAF0FAkzZZ2ZWGGh0dHA6Ly9rZXlzZXJ2ZXIua2pzbC5jb206MTEzNzEv cGtzL2xvb2t1cD9vcD1nZXQmc2VhcmNoPTB4RDM5REEwRTMmd2VoYXZleW91bm93 PXRydWUACgkQSypIl9OdoON0QwCdFu/VizMoA6Q0aIo7BtTu8PiZ+54AoLC9pAmm ho75uRyCaC3GWRFVh7C5 =0v+w -----END PGP SIGNATURE-----help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20101109152318.GA59055>