From owner-freebsd-bugs@FreeBSD.ORG Mon Dec 31 18:30:02 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B742516A474 for ; Mon, 31 Dec 2007 18:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9527213C455 for ; Mon, 31 Dec 2007 18:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lBVIU2Dj034758 for ; Mon, 31 Dec 2007 18:30:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lBVIU2Sj034751; Mon, 31 Dec 2007 18:30:02 GMT (envelope-from gnats) Resent-Date: Mon, 31 Dec 2007 18:30:02 GMT Resent-Message-Id: <200712311830.lBVIU2Sj034751@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, "Mikhail T." Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBBDD16A417 for ; Mon, 31 Dec 2007 18:28:09 +0000 (UTC) (envelope-from mi@aldan.algebra.com) Received: from aldan.algebra.com (aldan.algebra.com [216.254.65.224]) by mx1.freebsd.org (Postfix) with ESMTP id 8414113C458 for ; Mon, 31 Dec 2007 18:28:09 +0000 (UTC) (envelope-from mi@aldan.algebra.com) Received: from aldan.algebra.com (localhost [127.0.0.1]) by aldan.algebra.com (8.14.1/8.14.1) with ESMTP id lBVI04Il004918 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 31 Dec 2007 13:00:04 -0500 (EST) (envelope-from mi@aldan.algebra.com) Received: (from mi@localhost) by aldan.algebra.com (8.14.1/8.14.1/Submit) id lBVI046T004917; Mon, 31 Dec 2007 13:00:04 -0500 (EST) (envelope-from mi) Message-Id: <200712311800.lBVI046T004917@aldan.algebra.com> Date: Mon, 31 Dec 2007 13:00:04 -0500 (EST) From: "Mikhail T." To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: bin/119196: [wish] parallel download from multiple servers by fetch X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 18:30:02 -0000 >Number: 119196 >Category: bin >Synopsis: [wish] parallel download from multiple servers by fetch >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Mon Dec 31 18:30:02 UTC 2007 >Closed-Date: >Last-Modified: >Originator: Mikhail T. >Release: FreeBSD 6.3-PRERELEASE amd64 >Organization: Virtual Estates, Inc. >Environment: FreeBSD >Description: In most cases, when fetch(1) is invoked (or libfetch called into), it is to either download sources for a port, or binaries for a package. In all of those cases multiple servers hosting the same files are known to the caller. Unfortunately, the caller must use the servers /in sequence/, because fetch does not allow downloading different ranges of the same file from different sources /in parallel/. This is, of course, suboptimal as some servers (such as ftp.freebsd.org) are hit by much higher loads. The users may also have a (much) wider download bandwidth, than an upload bandwidth, that a particular server has (or would allow a single client to use). It should be possible to implement parallel downloads (over different protocols -- http, ftp, etc.). The implementation should also be adaptive: if the download from one of the servers finishes faster, the program should proceed to automatically fetch a not-yet-downloaded sub-range of the range, that is being downloaded from another server. Various "p2p" downloading programs (torrent, gnuttella) implement this feature already -- out of necessity. We should do it, because we can. No, I don't have a patch. Not yet, anyway... Somebody looking for a cool project is most welcome to this idea. >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: