From owner-freebsd-hubs@FreeBSD.ORG Mon Apr 7 22:18:02 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CFE0106566B for ; Mon, 7 Apr 2008 22:18:02 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id EDAA18FC14 for ; Mon, 7 Apr 2008 22:18:01 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m37Lox3R027044 for ; Mon, 7 Apr 2008 23:50:59 +0200 (CEST) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: hubs@FreeBSD.org Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-a/9HlnXs3szxGInMrBFY" Date: Mon, 07 Apr 2008 23:50:59 +0200 Message-Id: <1207605059.1031.38.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.22.0 FreeBSD GNOME Team Port X-Spam-Score: -2.544 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 Cc: Subject: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2008 22:18:02 -0000 --=-a/9HlnXs3szxGInMrBFY Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Okay the situation recently was that the mirrors had no chance keeping up with all the package sets I've been uploading to ftp-master. We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does not scale. I do propose a creation of a CDN (Content Delivery Network), having these features: - no mirroring of a complete package set! (Also no directory listings.) When client requests the file, and the file is not in the local cache, the file is downloaded from the upstream server and while it's being obtained, it's already being sent to the client. =EF=BB=BFThis is basical= ly squid. - if the file is present in the local cache, it's returned from local cache. - local cache is invalidated when a new package set is available on an upstream server. Invalidating mechanism: option a) cronjob that polls upstream server every 5 minutes for a file that gives current package set IDs (pull) option b) master server sends notification to all mirrors to invalidate a package set (push) optimization: when package set was invalidated, don't delete old files, instead on next hit, verify timestamp against upstream server - atomic package set uploads to master from pointyhat (probably having two directories that are switched over on master) - everything runs over http - default source of files for "pkg_add -r" command The goal is to refresh a package set on a daily basis. I don't know if we can use some existing software for this (Squid? Apache mod_proxy?) or if we will need to put something new together. Ideas? --=20 Pav Lucistnik East or west, ~ is best. --=-a/9HlnXs3szxGInMrBFY Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkf6l0MACgkQntdYP8FOsoKR3ACeKpn/uUdFwsQujkbXQHtl5IF0 f8IAoMC/dd3VXEomKIdW0JnC8iks5KwZ =8TVI -----END PGP SIGNATURE----- --=-a/9HlnXs3szxGInMrBFY--