Date: Mon, 07 Apr 2008 23:50:59 +0200 From: Pav Lucistnik <pav@FreeBSD.org> To: hubs@FreeBSD.org Subject: package distribution crisis - CDN needed Message-ID: <1207605059.1031.38.camel@ikaros.oook.cz>
next in thread | raw e-mail | index | archive | help
--=-a/9HlnXs3szxGInMrBFY
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Okay the situation recently was that the mirrors had no chance keeping
up with all the package sets I've been uploading to ftp-master.
We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does
not scale.
I do propose a creation of a CDN (Content Delivery Network), having
these features:
- no mirroring of a complete package set! (Also no directory listings.)
When client requests the file, and the file is not in the local cache,
the file is downloaded from the upstream server and while it's being
obtained, it's already being sent to the client. =EF=BB=BFThis is basical=
ly
squid.
- if the file is present in the local cache, it's returned from local
cache.
- local cache is invalidated when a new package set is available on
an upstream server. Invalidating mechanism:
option a) cronjob that polls upstream server every 5 minutes for a
file that gives current package set IDs (pull)
option b) master server sends notification to all mirrors to
invalidate a package set (push)
optimization: when package set was invalidated, don't delete old
files, instead on next hit, verify timestamp against upstream server
- atomic package set uploads to master from pointyhat (probably having
two directories that are switched over on master)
- everything runs over http
- default source of files for "pkg_add -r" command
The goal is to refresh a package set on a daily basis.
I don't know if we can use some existing software for this (Squid?
Apache mod_proxy?) or if we will need to put something new together.
Ideas?
--=20
Pav Lucistnik <pav@oook.cz>
<pav@FreeBSD.org>
East or west, ~ is best.
--=-a/9HlnXs3szxGInMrBFY
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?=
=?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?=
=?ISO-8859-1?Q?_zpr=E1vy?=
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (FreeBSD)
iEYEABECAAYFAkf6l0MACgkQntdYP8FOsoKR3ACeKpn/uUdFwsQujkbXQHtl5IF0
f8IAoMC/dd3VXEomKIdW0JnC8iks5KwZ
=8TVI
-----END PGP SIGNATURE-----
--=-a/9HlnXs3szxGInMrBFY--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1207605059.1031.38.camel>