From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 08:09:45 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1B821065671; Tue, 8 Apr 2008 08:09:45 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 672EA8FC1B; Tue, 8 Apr 2008 08:09:45 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m3889hCI081583; Tue, 8 Apr 2008 10:09:44 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m3889heJ081581; Tue, 8 Apr 2008 10:09:43 +0200 (CEST) (envelope-from olli) From: Oliver Fromme Message-Id: <200804080809.m3889heJ081581@lurza.secnetix.de> To: pav@freebsd.org Date: Tue, 8 Apr 2008 10:09:43 +0200 (CEST) In-Reply-To: <1207605059.1031.38.camel@ikaros.oook.cz> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 08 Apr 2008 10:09:44 +0200 (CEST) Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 08:09:46 -0000 Pav Lucistnik wrote: > Okay the situation recently was that the mirrors had no chance keeping > up with all the package sets I've been uploading to ftp-master. > > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does > not scale. > > I do propose a creation of a CDN (Content Delivery Network), having > these features: First of all I agree with Robert's concerns. We need to find out if such a CDN is actually suitable according to the download patterns of users. It could well be that it makes matters worse. And second: If a new solution is developed, it needs to be portable and easy to install and maintain. Remember that some FreeBSD mirrors are run by non-FreeBSD people on non-FreeBSD platforms. I don't think we want to lose those. I've also read about at least one case where server and client parts of a mirror ran on different machines, i.e. the releases and package sets are synced on one machine (which doesn't accept incoming connections for security reasons) and the files are served to clients from a different machine located in the DMZ of the site. The latter possibly can only accept incoming connections, but can't make outgoing connections (policy). We need to continue to support such setups. Also, multi-level topology should still work, e.g. like the following (4 levels): - ftpX.YY.freebsd.org sync from ftp.YY.freebsd.org - ftp.YY.freebsd.org syncs from ftp-master.eu.freebsd.org - ftp-master.eu.freebsd.org syncs from ftp-master.freebsd.org Furthermore, you should still be able to browse the packages with a standard FTP client. I know several people who are used to login to the nearest mirror, cd to the packages' "All" directory and do things like "dir gcc*", "mget xorg*" and so on. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Clear perl code is better than unclear awk code; but NOTHING comes close to unclear perl code" (taken from comp.lang.awk FAQ)