Date: Sat, 31 Mar 2012 22:16:48 -0400 From: Anthony Bryan <anthonybryan@gmail.com> To: ports@freebsd.org Subject: Re: Support for RfC 6249 - aka Metalinks - in ports? / Google Summer of Code Message-ID: <CANqTPeghqY4sYFWZ1ZF9yzcdWZOuqZV1TG6G1=0Z2Jj4H7%2BueQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
hi, I came across the following about metalink (a way to list mirrors & checksums)/ports: http://lists.freebsd.org/pipermail/freebsd-ports/2011-September/070607.html (follows after my sig) along with http://wiki.freebsd.org/Metalink we'd be interested in using one of our Google Summer of Code slots for this, if someone from your end was interested in mentoring. it's up on our ideas page, but we haven't had any nibbles yet :) http://www.metalinker.org/ideas.html --=20 (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] =A0 )) Easier, More Reliable, Self Healing Downloads ---- Raphael Eiselstein rabe at uugrn.org Hi, I wonder if there will be support for RfC 6249 / Metalink in the FreeBSD Ports infrastructure or someone is working / thinking about implementing this. >From http://wiki.freebsd.org/Metalink ------------------------------------------------------------- Metalink, an Open Standard, makes downloading easier by bundling the various ways (FTP/HTTP/P2P) to get files into one format for easier downloads. Simpler. Faster. More Reliable. Better. Metalink is a great concept which might be added to the ports infrastructure. Metalink files can be preprocessed in order to retain big mirror lists in bsd.sites.mk. Or an XML guru can convert bsd.sites.mk to an XML version and include it into metalinks in some way. Stub metalinks can be easily created for each port with a simple script. At this time only www/aria2 has some Metalink support. Aria2 has a few dependencies, but it's completely unnecessary to cram it into the base system. FETCH_DEPENDS would work very nice. Metalink support can be implemented simultaneously with P2P support, which is long overdue. Aria2 fully supports Metalink 3.0 and other downloaders will soon follow. Metalink (last edited 2008-06-17 21:37:49 by localhost) ------------------------------------------------------------- My approach would be some kind of http proxy which knows based on a table of URLs from the ports all the mirror information (if there are mirror sites) for given files or "content". We already have MD5 and SHA256 checksums and sizes in distinfo, so we might enhance the fetch routines not to request a specific URL and filename but for a given checksum and writing the downloaded content to a known filename. Instead of requesting ftp://ftp.freebsd.org/pub/FreeBSD/ports/distfiles/aria2-1.10.9.tar.bz2 (or mirror sites) the ports might fetch http://my-proxy.example.com/sha256/51319be3006545c442fbbedb476fa7ba2fd562e0= 468d9c573e7f5261ea610af5 and my-proxy.example.com would resolve this to a list of mirror URLs using an implementation of RfC 6249 and maybe could itself be an "dynamic ftp mirror" through some local caches (using squid-proxy or similar caches). +------------------+ | FreeBSD Ports | Our ports (already have distinfo) +----+-------+-----+ | fetch | using fetch(1) +-------+ | v +--------------------------+ | http://localhost:8080/ | requesting checksums | sha256/51319be300...0af5 | from a local RfC 6249 +--------------------------+ proxy/cache which might | | | serve this from local v v v cache or just will request +------+ +-----+ +------+ the file from multiple | A | | B | | C | sources which have a copy +------+ +-----+ +------+ (of a copy (of a copy)) \ | / of the requested file \ | / originally obtained from \ | / the master site or another +---------+ "primary source". | primary | | source | +---------+ All we need to change in ports is a little enhancement which will make fetch (optionally) requesting for the sha256 checksums instead of given filenames, enabled in /etc/make.conf So, basically this would be a endoscopic surgery (=3Dminimal impact) to the freebsd ports supporting optional requesting checksums instead of static URLs from a list. Requesting static files parallel from multiple proxy caches might be implemented by parallel requesting byte ranges from different sources using 64kByte..256kByte blocks for example. Beside this, this approach wouldn't be limited to distfiles but may work for every kind of static files (tarballs, packages, ... ) by any open source project which is distributed through ftp mirrors. And all you need for this is * a filename (we already have) * a sha256 checksum (we have) * a rfc6249 proxy server (3rd party component) * a traditional HTTP/1.0 or HTTP/1.1 compliant tool (like fetch(1)) Does anyone like this approach? Does anyone work on (knows about) a concept or is developing something implementing rfc6249? Regards Raphael Eiselstein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANqTPeghqY4sYFWZ1ZF9yzcdWZOuqZV1TG6G1=0Z2Jj4H7%2BueQ>