Date: Sat, 1 Oct 2011 00:34:08 +0200 From: Raphael Eiselstein <rabe@uugrn.org> To: freebsd-ports@freebsd.org Subject: Support for RfC 6249 - aka Metalinks - in ports? Message-ID: <20110930223408.GC3713@ma.sigsys.de>
next in thread | raw e-mail | index | archive | help
--m51xatjYGsM+13rf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I wonder if there will be support for RfC 6249 / Metalink in the FreeBSD Ports infrastructure or someone is working / thinking about implementing this. =46rom http://wiki.freebsd.org/Metalink ------------------------------------------------------------- Metalink, an Open Standard, makes downloading easier by bundling the various ways (FTP/HTTP/P2P) to get files into one format for easier downloads. Simpler. Faster. More Reliable. Better. Metalink is a great concept which might be added to the ports infrastructure. Metalink files can be preprocessed in order to retain big mirror lists in bsd.sites.mk. Or an XML guru can convert bsd.sites.mk to an XML version and include it into metalinks in some way. Stub metalinks can be easily created for each port with a simple script. At this time only www/aria2 has some Metalink support. Aria2 has a few dependencies, but it's completely unnecessary to cram it into the base system. FETCH_DEPENDS would work very nice. Metalink support can be implemented simultaneously with P2P support, which is long overdue. Aria2 fully supports Metalink 3.0 and other downloaders will soon follow. Metalink (last edited 2008-06-17 21:37:49 by localhost) ------------------------------------------------------------- My approach would be some kind of http proxy which knows based on a table of URLs from the ports all the mirror information (if there are mirror sites) for given files or "content". We already have MD5 and SHA256 checksums and sizes in distinfo, so we=20 might enhance the fetch routines not to request a specific URL and=20 filename but for a given checksum and writing the downloaded content=20 to a known filename. Instead of requesting ftp://ftp.freebsd.org/pub/FreeBSD/ports/distfiles/aria2-1.10.9.tar.bz2 (or mirror sites) the ports might fetch http://my-proxy.example.com/sha256/51319be3006545c442fbbedb476fa7ba2fd562e0= 468d9c573e7f5261ea610af5 and my-proxy.example.com would resolve this to a list of mirror URLs using an implementation of RfC 6249 and maybe could itself be an "dynamic ftp mirror" through some local caches (using squid-proxy or=20 similar caches).=20 +------------------+ | FreeBSD Ports | Our ports (already have distinfo) +----+-------+-----+ | fetch | using fetch(1) +-------+ | v +--------------------------+ | http://localhost:8080/ | requesting checksums | sha256/51319be300...0af5 | from a local RfC 6249 +--------------------------+ proxy/cache which might | | | serve this from local v v v cache or just will request +------+ +-----+ +------+ the file from multiple | A | | B | | C | sources which have a copy +------+ +-----+ +------+ (of a copy (of a copy)) \ | / of the requested file \ | / originally obtained from \ | / the master site or another +---------+ "primary source". | primary | | source | +---------+ All we need to change in ports is a little enhancement which will make fetch (optionally) requesting for the sha256 checksums instead of=20 given filenames, enabled in /etc/make.conf So, basically this would be a endoscopic surgery (=3Dminimal impact) to=20 the freebsd ports supporting optional requesting checksums instead of=20 static URLs from a list. Requesting static files parallel from multiple proxy caches might be=20 implemented by parallel requesting byte ranges from different sources=20 using 64kByte..256kByte blocks for example. =20 Beside this, this approach wouldn't be limited to distfiles but may=20 work for every kind of static files (tarballs, packages, ... ) by any=20 open source project which is distributed through ftp mirrors.=20 And all you need for this is * a filename (we already have) * a sha256 checksum (we have) * a rfc6249 proxy server (3rd party component) * a traditional HTTP/1.0 or HTTP/1.1 compliant tool (like fetch(1)) Does anyone like this approach? Does anyone work on (knows about) a=20 concept or is developing something implementing rfc6249? Regards Raphael Eiselstein --=20 Raphael Eiselstein <rabe@uugrn.org> http://rabe.uugrn.org/ xmpp:freibyter@gmx.de | https://www.xing.com/profile/Raphael_Eiselstein = =20 GnuPG: E7B2 1D66 3AF2 EDC7 9828 6D7A 9CDA 3E7B 10CA 9F2D =2E........|.........|.........|.........|.........|.........|.........|.. --m51xatjYGsM+13rf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk6GQ+AACgkQnNo+exDKny0C4QCgnkRhebR+bQzTpibmSbLDCpII AGUAn3SnLyXIctOEYWhOBzW2YDFJYjE6 =X7ME -----END PGP SIGNATURE----- --m51xatjYGsM+13rf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110930223408.GC3713>