From owner-freebsd-hubs@FreeBSD.ORG Mon Apr 7 22:18:02 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CFE0106566B for ; Mon, 7 Apr 2008 22:18:02 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id EDAA18FC14 for ; Mon, 7 Apr 2008 22:18:01 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m37Lox3R027044 for ; Mon, 7 Apr 2008 23:50:59 +0200 (CEST) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: hubs@FreeBSD.org Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-a/9HlnXs3szxGInMrBFY" Date: Mon, 07 Apr 2008 23:50:59 +0200 Message-Id: <1207605059.1031.38.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.22.0 FreeBSD GNOME Team Port X-Spam-Score: -2.544 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 Cc: Subject: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2008 22:18:02 -0000 --=-a/9HlnXs3szxGInMrBFY Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Okay the situation recently was that the mirrors had no chance keeping up with all the package sets I've been uploading to ftp-master. We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does not scale. I do propose a creation of a CDN (Content Delivery Network), having these features: - no mirroring of a complete package set! (Also no directory listings.) When client requests the file, and the file is not in the local cache, the file is downloaded from the upstream server and while it's being obtained, it's already being sent to the client. =EF=BB=BFThis is basical= ly squid. - if the file is present in the local cache, it's returned from local cache. - local cache is invalidated when a new package set is available on an upstream server. Invalidating mechanism: option a) cronjob that polls upstream server every 5 minutes for a file that gives current package set IDs (pull) option b) master server sends notification to all mirrors to invalidate a package set (push) optimization: when package set was invalidated, don't delete old files, instead on next hit, verify timestamp against upstream server - atomic package set uploads to master from pointyhat (probably having two directories that are switched over on master) - everything runs over http - default source of files for "pkg_add -r" command The goal is to refresh a package set on a daily basis. I don't know if we can use some existing software for this (Squid? Apache mod_proxy?) or if we will need to put something new together. Ideas? --=20 Pav Lucistnik East or west, ~ is best. --=-a/9HlnXs3szxGInMrBFY Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkf6l0MACgkQntdYP8FOsoKR3ACeKpn/uUdFwsQujkbXQHtl5IF0 f8IAoMC/dd3VXEomKIdW0JnC8iks5KwZ =8TVI -----END PGP SIGNATURE----- --=-a/9HlnXs3szxGInMrBFY-- From owner-freebsd-hubs@FreeBSD.ORG Mon Apr 7 23:43:31 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2334A106566C for ; Mon, 7 Apr 2008 23:43:31 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id E59338FC0A for ; Mon, 7 Apr 2008 23:43:30 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id DF38946B5E; Mon, 7 Apr 2008 19:23:53 -0400 (EDT) Date: Tue, 8 Apr 2008 00:23:53 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Pav Lucistnik In-Reply-To: <1207605059.1031.38.camel@ikaros.oook.cz> Message-ID: <20080408002005.W15502@fledge.watson.org> References: <1207605059.1031.38.camel@ikaros.oook.cz> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: hubs@FreeBSD.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2008 23:43:31 -0000 On Mon, 7 Apr 2008, Pav Lucistnik wrote: > Okay the situation recently was that the mirrors had no chance keeping up > with all the package sets I've been uploading to ftp-master. > > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does not > scale. > > I do propose a creation of a CDN (Content Delivery Network), having these > features: The success of a CDN is premised on the working set of active files being [significantly] smaller than the set of available files. While I don't doubt that only a few packages are of interest to most users, for a CDN to provide a real benefit it needs to be the case that no users require most files. The reason being that, if you have to transfer all or even most files every time, a CDN may well be a significantly less efficient way to do it than rsync. Has a working set analysis for our main mirrors been done that demonstrates that the actual working set is sufficiently smaller than the total set that a CDN would be more, rather than less, efficient? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 00:14:10 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA0B51065675 for ; Tue, 8 Apr 2008 00:14:10 +0000 (UTC) (envelope-from jonny@jonny.eng.br) Received: from coe.ufrj.br (roma.coe.ufrj.br [146.164.53.65]) by mx1.freebsd.org (Postfix) with ESMTP id 9E5478FC24 for ; Tue, 8 Apr 2008 00:14:10 +0000 (UTC) (envelope-from jonny@jonny.eng.br) Received: from localhost (localhost [127.0.0.1]) by coe.ufrj.br (Postfix) with ESMTP id 7C6B312542F; Mon, 7 Apr 2008 20:46:21 -0300 (BRT) X-Virus-Scanned: amavisd-new at coe.ufrj.br Received: from coe.ufrj.br ([146.164.53.65]) by localhost (roma.coe.ufrj.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DG2T-DX-rc4H; Mon, 7 Apr 2008 20:46:15 -0300 (BRT) Received: from [201.19.165.115] (unknown [201.19.165.115]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by coe.ufrj.br (Postfix) with ESMTP id 9834A125421; Mon, 7 Apr 2008 20:46:15 -0300 (BRT) Message-ID: <47FAB249.2020001@jonny.eng.br> Date: Mon, 07 Apr 2008 20:46:17 -0300 From: =?UTF-8?B?Sm/Do28gQ2FybG9zIE1lbmRlcyBMdcOtcw==?= User-Agent: Thunderbird 2.0.0.12pre (Windows/20071214) MIME-Version: 1.0 To: pav@FreeBSD.org References: <1207605059.1031.38.camel@ikaros.oook.cz> In-Reply-To: <1207605059.1031.38.camel@ikaros.oook.cz> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: hubs@FreeBSD.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 00:14:11 -0000 Pav Lucistnik wrote: > Okay the situation recently was that the mirrors had no chance keeping > up with all the package sets I've been uploading to ftp-master. > > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does > not scale. > > I do propose a creation of a CDN (Content Delivery Network), having > these features: > > - no mirroring of a complete package set! (Also no directory listings.) > When client requests the file, and the file is not in the local cache, > the file is downloaded from the upstream server and while it's being > obtained, it's already being sent to the client. This is basically > squid. > > - if the file is present in the local cache, it's returned from local > cache. > > - local cache is invalidated when a new package set is available on > an upstream server. Invalidating mechanism: > option a) cronjob that polls upstream server every 5 minutes for a > file that gives current package set IDs (pull) > option b) master server sends notification to all mirrors to > invalidate a package set (push) > optimization: when package set was invalidated, don't delete old > files, instead on next hit, verify timestamp against upstream server > > - atomic package set uploads to master from pointyhat (probably having > two directories that are switched over on master) > > - everything runs over http > > - default source of files for "pkg_add -r" command > > The goal is to refresh a package set on a daily basis. > > > I don't know if we can use some existing software for this (Squid? > Apache mod_proxy?) or if we will need to put something new together. > Ideas? > I am not sure if this would solve anything, but if we go further in this direction, I'd like to see some architecture with prefetch capability. Note also that a real CDN would hide from the final user the real data location, and this would be selected by some sort of proximity and/or load information. Some CDNs indeed use proxy cache to central server as means of populating its own data, but proxy caching is only a small part of the solution. I did not follow whatever situation happened recently, but I had some trouble in the past with late announcements for mirror administrators. I had sometimes received the announce just like any other FreeBSD user. And even in that cases, packages were distributed much time earlier than final release. Jonny -- João Carlos Mendes Luís - Networking Engineer - jonny@jonny.eng.br From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 00:21:48 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8536106564A for ; Tue, 8 Apr 2008 00:21:48 +0000 (UTC) (envelope-from gad@FreeBSD.org) Received: from smtp6.server.rpi.edu (smtp6.server.rpi.edu [128.113.2.226]) by mx1.freebsd.org (Postfix) with ESMTP id 512EF8FC0A for ; Tue, 8 Apr 2008 00:21:48 +0000 (UTC) (envelope-from gad@FreeBSD.org) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp6.server.rpi.edu (8.13.1/8.13.1) with ESMTP id m37NKKNN019320; Mon, 7 Apr 2008 19:20:20 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <1207605059.1031.38.camel@ikaros.oook.cz> References: <1207605059.1031.38.camel@ikaros.oook.cz> Date: Mon, 7 Apr 2008 19:20:19 -0400 To: pav@FreeBSD.org, hubs@FreeBSD.org From: Garance A Drosehn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-RPI-SA-Score: undef - spam scanning disabled X-CanItPRO-Stream: default X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.226 Cc: Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 00:21:48 -0000 At 11:50 PM +0200 4/7/08, Pav Lucistnik wrote: >Okay the situation recently was that the mirrors had no chance keeping >up with all the package sets I've been uploading to ftp-master. > >We clearly need to move beyond rsync/cvsup synced ftp mirrors. This >does not scale. Maybe do something more "bit-torrent-y" ? That seems to scale quite well! -- Garance Alistair Drosehn = drosehn@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 00:41:16 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AF091065675 for ; Tue, 8 Apr 2008 00:41:16 +0000 (UTC) (envelope-from emoenke@gwdg.de) Received: from mailer.gwdg.de (mailer.gwdg.de [134.76.10.26]) by mx1.freebsd.org (Postfix) with ESMTP id 0A51F8FC19 for ; Tue, 8 Apr 2008 00:41:15 +0000 (UTC) (envelope-from emoenke@gwdg.de) Received: from gwdu05.gwdg.de ([134.76.8.5]) by mailer.gwdg.de with esmtps (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1Jj1Ah-0002Bd-O7 for hubs@FreeBSD.org; Tue, 08 Apr 2008 01:54:43 +0200 Date: Tue, 8 Apr 2008 01:54:43 +0200 (CEST) From: Eberhard Moenkeberg To: hubs@FreeBSD.org In-Reply-To: <20080408002005.W15502@fledge.watson.org> Message-ID: References: <1207605059.1031.38.camel@ikaros.oook.cz> <20080408002005.W15502@fledge.watson.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="1283851528-1837400255-1207612483=:10070" X-Spam-Level: - X-Virus-Scanned: (clean) by exiscan+sophie Cc: Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 00:41:16 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --1283851528-1837400255-1207612483=:10070 Content-Type: TEXT/PLAIN; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Hi, On Tue, 8 Apr 2008, Robert Watson wrote: > On Mon, 7 Apr 2008, Pav Lucistnik wrote: =20 > > Okay the situation recently was that the mirrors had no chance keeping = up > > with all the package sets I've been uploading to ftp-master. > > > > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This doe= s not > > scale. > > > > I do propose a creation of a CDN (Content Delivery Network), having the= se > > features: >=20 > The success of a CDN is premised on the working set of active files being > [significantly] smaller than the set of available files. While I don't d= oubt > that only a few packages are of interest to most users, for a CDN to prov= ide a > real benefit it needs to be the case that no users require most files. T= he > reason being that, if you have to transfer all or even most files every t= ime, > a CDN may well be a significantly less efficient way to do it than rsync.= Has > a working set analysis for our main mirrors been done that demonstrates t= hat > the actual working set is sufficiently smaller than the total set that a = CDN > would be more, rather than less, efficient? Maybe it is interesting for you to have a look at=20 http://www.poeml.de/~poeml/talks/redirector/ The SUSE team has a redirection mechanism at work on single file basis,=20 and they also have experienced that it would be "enough" for the users if= =20 there are lots of mirrors with the "most requested" files plus some=20 mirrors with all files. It is all GPL, so you (even you) should be able to use the software. Viele Gr=FC=DFe Eberhard M=F6nkeberg (emoenke@gwdg.de, em@kki.org) --1283851528-1837400255-1207612483=:10070-- From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 03:16:23 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5417F1065671; Tue, 8 Apr 2008 03:16:23 +0000 (UTC) (envelope-from jonny@jonny.eng.br) Received: from coe.ufrj.br (roma.coe.ufrj.br [146.164.53.65]) by mx1.freebsd.org (Postfix) with ESMTP id F2EC58FC16; Tue, 8 Apr 2008 03:16:22 +0000 (UTC) (envelope-from jonny@jonny.eng.br) Received: from localhost (localhost [127.0.0.1]) by coe.ufrj.br (Postfix) with ESMTP id 5F6D212542F; Tue, 8 Apr 2008 00:16:21 -0300 (BRT) X-Virus-Scanned: amavisd-new at coe.ufrj.br Received: from coe.ufrj.br ([146.164.53.65]) by localhost (roma.coe.ufrj.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c+e8fb96BVi5; Tue, 8 Apr 2008 00:16:15 -0300 (BRT) Received: from [201.19.165.115] (unknown [201.19.165.115]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by coe.ufrj.br (Postfix) with ESMTP id 8A90D125421; Tue, 8 Apr 2008 00:16:15 -0300 (BRT) Message-ID: <47FAE381.10305@jonny.eng.br> Date: Tue, 08 Apr 2008 00:16:17 -0300 From: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= User-Agent: Thunderbird 2.0.0.12pre (Windows/20071214) MIME-Version: 1.0 To: Garance A Drosehn References: <1207605059.1031.38.camel@ikaros.oook.cz> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: pav@FreeBSD.org, hubs@FreeBSD.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 03:16:23 -0000 Garance A Drosehn wrote: > At 11:50 PM +0200 4/7/08, Pav Lucistnik wrote: >> Okay the situation recently was that the mirrors had no chance keeping >> up with all the package sets I've been uploading to ftp-master. >> >> We clearly need to move beyond rsync/cvsup synced ftp mirrors. This >> does not scale. > > Maybe do something more "bit-torrent-y" ? That seems to scale quite > well! Surely a must for CD and DVD images, but not much useful for package installation, IMHO. Jonny -- João Carlos Mendes Luís - Networking Engineer - jonny@jonny.eng.br From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 06:12:09 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B8CE106567E for ; Tue, 8 Apr 2008 06:12:09 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id 8ACC08FC28 for ; Tue, 8 Apr 2008 06:12:08 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so1970344fgg.35 for ; Mon, 07 Apr 2008 23:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=goGUzBgX5dwINK0OTN/+WSAiqLU14R8KOWiixrb3Zo8=; b=ZSp/2BEvL3ZiBfNUzlchqRCsZNnSWHuPNYVFmZ2Gjhy6jgsbw6Tz4nik7Xky9tGmy8wmFBjlYIdyG/ntkFMi5+IznLqO4P0JpuvPUTHcf0Y/Q1m5bHfq79CTLh+pGJYmm4jYmlD9S5eDDOn4edIqOkuz8HyQnEhpWwFqwZSrCS8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=IaXF7TgovYA89WJA1P6ck/cKcgGpfLAXapIBwrGDd7vqf6Hfa5oJLsCFQ6P0MSN4lpH+lcpWWUzBpWCl9JVaZKKDE6STKaGd8de/I/ohvYl5b86zcH+Fe/ZiKnZYjc0ZJxPqGSAgjdj7oOYDmUDT4tuFEqWe8lqvNG+t12BAnc4= Received: by 10.82.161.19 with SMTP id j19mr9419019bue.12.1207635123664; Mon, 07 Apr 2008 23:12:03 -0700 (PDT) Received: by 10.82.174.5 with HTTP; Mon, 7 Apr 2008 23:12:03 -0700 (PDT) Message-ID: <59adc1a0804072312l192a98f8ncfc70e5687e45ef7@mail.gmail.com> Date: Tue, 8 Apr 2008 09:12:03 +0300 From: "Dimitar Vasilev" To: pav@freebsd.org In-Reply-To: <47FAE381.10305@jonny.eng.br> MIME-Version: 1.0 References: <1207605059.1031.38.camel@ikaros.oook.cz> <47FAE381.10305@jonny.eng.br> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 06:12:09 -0000 Pav, Segmented downloading also raises the need of packet signatures. Now if a package is broken on a mirror, user simply selects another one. I'm aware that for license reasons there is no x.509/opengpg-compatible solution that can assure you that you're getting the right package. Unfortunately I can't help there - got too late into computers to have time for learning C. Once upon a time a guy suggested that all FreeBSD mirrors join http://www.metalinker.org/, but someone from the French mirrors rejected the idea. Next thing is that all mirrors have different uplink/downlink speed to master servers and response may vary. This may cause a client to select couple of servers and at the end the master server could be hit badly. Also the need of additional hardware that may be purchased to sustain the model should be considered also. Best regardz/Nazdar, Dimitar Vassilev one of the bg.freebsd.org monkeys From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 08:09:45 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1B821065671; Tue, 8 Apr 2008 08:09:45 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 672EA8FC1B; Tue, 8 Apr 2008 08:09:45 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m3889hCI081583; Tue, 8 Apr 2008 10:09:44 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m3889heJ081581; Tue, 8 Apr 2008 10:09:43 +0200 (CEST) (envelope-from olli) From: Oliver Fromme Message-Id: <200804080809.m3889heJ081581@lurza.secnetix.de> To: pav@freebsd.org Date: Tue, 8 Apr 2008 10:09:43 +0200 (CEST) In-Reply-To: <1207605059.1031.38.camel@ikaros.oook.cz> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 08 Apr 2008 10:09:44 +0200 (CEST) Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 08:09:46 -0000 Pav Lucistnik wrote: > Okay the situation recently was that the mirrors had no chance keeping > up with all the package sets I've been uploading to ftp-master. > > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does > not scale. > > I do propose a creation of a CDN (Content Delivery Network), having > these features: First of all I agree with Robert's concerns. We need to find out if such a CDN is actually suitable according to the download patterns of users. It could well be that it makes matters worse. And second: If a new solution is developed, it needs to be portable and easy to install and maintain. Remember that some FreeBSD mirrors are run by non-FreeBSD people on non-FreeBSD platforms. I don't think we want to lose those. I've also read about at least one case where server and client parts of a mirror ran on different machines, i.e. the releases and package sets are synced on one machine (which doesn't accept incoming connections for security reasons) and the files are served to clients from a different machine located in the DMZ of the site. The latter possibly can only accept incoming connections, but can't make outgoing connections (policy). We need to continue to support such setups. Also, multi-level topology should still work, e.g. like the following (4 levels): - ftpX.YY.freebsd.org sync from ftp.YY.freebsd.org - ftp.YY.freebsd.org syncs from ftp-master.eu.freebsd.org - ftp-master.eu.freebsd.org syncs from ftp-master.freebsd.org Furthermore, you should still be able to browse the packages with a standard FTP client. I know several people who are used to login to the nearest mirror, cd to the packages' "All" directory and do things like "dir gcc*", "mget xorg*" and so on. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Clear perl code is better than unclear awk code; but NOTHING comes close to unclear perl code" (taken from comp.lang.awk FAQ) From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 08:49:58 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD8CA1065676 for ; Tue, 8 Apr 2008 08:49:58 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from nat-application.b1.lan.prg.vol.cz (nat-application.b1.lan.prg.vol.cz [195.122.204.152]) by mx1.freebsd.org (Postfix) with ESMTP id 2C34A8FC12 for ; Tue, 8 Apr 2008 08:49:57 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from pav.hide.vol.cz (localhost [127.0.0.1]) by nat-application.b1.lan.prg.vol.cz (8.14.2/8.14.2) with ESMTP id m388FmSq084461; Tue, 8 Apr 2008 10:15:48 +0200 (CEST) (envelope-from pav@FreeBSD.org) Received: (from pav@localhost) by pav.hide.vol.cz (8.14.2/8.14.2/Submit) id m388FmUE084460; Tue, 8 Apr 2008 10:15:48 +0200 (CEST) (envelope-from pav@FreeBSD.org) X-Authentication-Warning: pav.hide.vol.cz: pav set sender to pav@FreeBSD.org using -f From: Pav Lucistnik To: Eberhard Moenkeberg In-Reply-To: References: <1207605059.1031.38.camel@ikaros.oook.cz> <20080408002005.W15502@fledge.watson.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-qF1+FDhiuoVWGcf7/PI/" Date: Tue, 08 Apr 2008 10:15:48 +0200 Message-Id: <1207642548.84150.10.camel@pav.hide.vol.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Cc: hubs@FreeBSD.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 08:49:58 -0000 --=-qF1+FDhiuoVWGcf7/PI/ Content-Type: text/plain; charset=ISO8859-2 Content-Transfer-Encoding: quoted-printable Eberhard Moenkeberg p=ED=B9e v =FAt 08. 04. 2008 v 01:54 +0200: > > > I do propose a creation of a CDN (Content Delivery Network), having t= hese > > > features: > >=20 > > The success of a CDN is premised on the working set of active files bei= ng > > [significantly] smaller than the set of available files. While I don't= doubt > > that only a few packages are of interest to most users, for a CDN to pr= ovide a > > real benefit it needs to be the case that no users require most files. = The > > reason being that, if you have to transfer all or even most files every= time, > > a CDN may well be a significantly less efficient way to do it than rsyn= c. Has > > a working set analysis for our main mirrors been done that demonstrates= that > > the actual working set is sufficiently smaller than the total set that = a CDN > > would be more, rather than less, efficient? I don't have these statistics, but I'd like to see them very much. Can I ask some of the mirror admins to compile me statistics on downloads of files from /pub/FreeBSD/ports/*packages*, say for March 2008 ? I'd also like to have these numbers from ftp.FreeBSD.org so I have a comparision of volume. My suspicion is that 99% of users get their packages from ftp.FreeBSD.org, because that's what pkg_add -r uses without PACKAGEROOT specification. > Maybe it is interesting for you to have a look at=20 >=20 > http://www.poeml.de/~poeml/talks/redirector/ >=20 > The SUSE team has a redirection mechanism at work on single file basis,=20 > and they also have experienced that it would be "enough" for the users if= =20 > there are lots of mirrors with the "most requested" files plus some=20 > mirrors with all files. Now this looks so much better and easier than my CDN idea! We could use it practically as-is. We will need to patch pkg_install to use http://redirector.FreeBSD.org/ instead of ftp://ftp.FreeBSD.org/ --=20 Pav Lucistnik Logitech is infamous for providing 30 MB compressed driver software packages that install a bunch of crap applications on the system and round up the number of icons in the tray to the nearest power of two. -- Michael Stromberg --=-qF1+FDhiuoVWGcf7/PI/ Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkf7Ka4ACgkQntdYP8FOsoLkmACeNDmkKboZZemXwZN3Fb+gFICm Gi4AoKl7Dk8XM36bJAbRU1jnCl/eA2Y8 =Y101 -----END PGP SIGNATURE----- --=-qF1+FDhiuoVWGcf7/PI/-- From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 09:47:47 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20C1B1065675 for ; Tue, 8 Apr 2008 09:47:47 +0000 (UTC) (envelope-from ftpmaint@proxad.net) Received: from fanfan.staff.proxad.net (fanfan.staff.proxad.net [213.228.1.105]) by mx1.freebsd.org (Postfix) with ESMTP id DC20B8FC2C for ; Tue, 8 Apr 2008 09:47:46 +0000 (UTC) (envelope-from ftpmaint@proxad.net) Received: from [127.0.0.1] (localhost [127.0.0.1]) by fanfan.staff.proxad.net (Postfix) with ESMTP id 3E9F635A588; Tue, 8 Apr 2008 11:29:01 +0200 (CEST) Message-ID: <47FB3ADD.6070401@proxad.net> Date: Tue, 08 Apr 2008 11:29:01 +0200 From: Francois Petillon User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20070113 Debian/1.7.8-1sarge10 X-Accept-Language: en, fr MIME-Version: 1.0 To: Dimitar Vasilev References: <1207605059.1031.38.camel@ikaros.oook.cz> <47FAE381.10305@jonny.eng.br> <59adc1a0804072312l192a98f8ncfc70e5687e45ef7@mail.gmail.com> In-Reply-To: <59adc1a0804072312l192a98f8ncfc70e5687e45ef7@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 09:47:47 -0000 Dimitar Vasilev wrote: > Once upon a time a guy suggested that all FreeBSD mirrors join > http://www.metalinker.org/, but someone from the French mirrors rejected the > idea. The frenchy did not reject the idea, the frenchy just told segmented downloads is fucking up IOs on big servers. And metalinker is mainly aimed at multisources segmented downloads. François From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 12:15:00 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9077F1065679 for ; Tue, 8 Apr 2008 12:15:00 +0000 (UTC) (envelope-from contact@dann.ro) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.154]) by mx1.freebsd.org (Postfix) with ESMTP id A11D78FC2C for ; Tue, 8 Apr 2008 12:14:58 +0000 (UTC) (envelope-from contact@dann.ro) Received: by fg-out-1718.google.com with SMTP id 16so2107174fgg.35 for ; Tue, 08 Apr 2008 05:14:55 -0700 (PDT) Received: by 10.86.78.4 with SMTP id a4mr4454530fgb.3.1207655154213; Tue, 08 Apr 2008 04:45:54 -0700 (PDT) Received: from ?85.186.124.45? ( [85.186.124.45]) by mx.google.com with ESMTPS id e11sm8336500fga.5.2008.04.08.04.45.51 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 08 Apr 2008 04:45:53 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v752.3) Content-Transfer-Encoding: 7bit Message-Id: <64DB14CF-187E-46A5-945F-A80668444DAE@dann.ro> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: hubs@freebsd.org From: Daniel Niculescu Logofatu Date: Tue, 8 Apr 2008 14:45:56 +0300 X-Mailer: Apple Mail (2.752.3) Cc: Subject: Re: package distribution crisis X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 12:15:00 -0000 How about a RDBMS solution? Would it be posible to have the entire repository on a RDBMS? Like postgresql? From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 12:34:49 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99AE2106566C for ; Tue, 8 Apr 2008 12:34:49 +0000 (UTC) (envelope-from dan@langille.org) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) by mx1.freebsd.org (Postfix) with ESMTP id 55ADA8FC1F for ; Tue, 8 Apr 2008 12:34:49 +0000 (UTC) (envelope-from dan@langille.org) Received: from localhost (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 3DCD550841; Tue, 8 Apr 2008 13:17:26 +0100 (BST) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by localhost (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zpQyR22Gn9h5; Tue, 8 Apr 2008 13:17:23 +0100 (BST) Received: from [192.168.1.2] (bast.unixathome.org [72.94.192.69]) by nyi.unixathome.org (Postfix) with ESMTPSA id 06A805083E; Tue, 8 Apr 2008 13:17:22 +0100 (BST) In-Reply-To: <64DB14CF-187E-46A5-945F-A80668444DAE@dann.ro> References: <64DB14CF-187E-46A5-945F-A80668444DAE@dann.ro> Mime-Version: 1.0 (Apple Message framework v753) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <5108C9A7-C072-4F6D-AD36-4F5EF72ED86C@langille.org> Content-Transfer-Encoding: 7bit From: Dan Langille Date: Tue, 8 Apr 2008 08:16:44 -0400 To: Daniel Niculescu Logofatu X-Mailer: Apple Mail (2.753) Cc: hubs@freebsd.org Subject: Re: package distribution crisis X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 12:34:49 -0000 On Apr 8, 2008, at 7:45 AM, Daniel Niculescu Logofatu wrote: > How about a RDBMS solution? Would it be posible to have the entire > repository on a RDBMS? Like postgresql? Yes it is possible. But why would you want to do that? -- Dan Langille -- http://www.langille.org/ dan@langille.org From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 14:16:02 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 332031065741; Tue, 8 Apr 2008 14:16:02 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id A3C498FC2A; Tue, 8 Apr 2008 14:16:01 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.2/8.14.2) with ESMTP id m38EG4Un053821; Tue, 8 Apr 2008 09:16:04 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.2/8.14.2/Submit) id m38EG4Vi053820; Tue, 8 Apr 2008 09:16:04 -0500 (CDT) (envelope-from brooks) Date: Tue, 8 Apr 2008 09:16:04 -0500 From: Brooks Davis To: Pav Lucistnik Message-ID: <20080408141604.GA53370@lor.one-eyed-alien.net> References: <1207605059.1031.38.camel@ikaros.oook.cz> <20080408002005.W15502@fledge.watson.org> <1207642548.84150.10.camel@pav.hide.vol.cz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline In-Reply-To: <1207642548.84150.10.camel@pav.hide.vol.cz> User-Agent: Mutt/1.5.17 (2007-11-01) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Tue, 08 Apr 2008 09:16:04 -0500 (CDT) Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 14:16:02 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 08, 2008 at 10:15:48AM +0200, Pav Lucistnik wrote: > Eberhard Moenkeberg p??e v ?t 08. 04. 2008 v 01:54 +0200: >=20 > > > > I do propose a creation of a CDN (Content Delivery Network), having= these > > > > features: > > >=20 > > > The success of a CDN is premised on the working set of active files b= eing > > > [significantly] smaller than the set of available files. While I don= 't doubt > > > that only a few packages are of interest to most users, for a CDN to = provide a > > > real benefit it needs to be the case that no users require most files= =2E The > > > reason being that, if you have to transfer all or even most files eve= ry time, > > > a CDN may well be a significantly less efficient way to do it than rs= ync. Has > > > a working set analysis for our main mirrors been done that demonstrat= es that > > > the actual working set is sufficiently smaller than the total set tha= t a CDN > > > would be more, rather than less, efficient? >=20 > I don't have these statistics, but I'd like to see them very much. > Can I ask some of the mirror admins to compile me statistics on > downloads of files from /pub/FreeBSD/ports/*packages*, say for March > 2008 ? >=20 > I'd also like to have these numbers from ftp.FreeBSD.org so I have a > comparision of volume. >=20 > My suspicion is that 99% of users get their packages from > ftp.FreeBSD.org, because that's what pkg_add -r uses without PACKAGEROOT > specification. I suspect that like virtually everything else on the internet package access is heavy-tailed. If so, no from of cooperative caching will be of use. For example, this paper killed all research on cooperative caching web proxies by demostrating that its was fundamentally pointless: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=3D10.1.1.22.9583 -- Brooks --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iD8DBQFH+34jXY6L6fI4GtQRAhwDAJ9h3df8cg+ZxEsesdQIJJl4xJC5qACfTO8s eR4JbbyHsdmJTCF9Qr18jec= =kmyi -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF-- From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 14:21:28 2008 Return-Path: Delivered-To: hubs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A51A310656C4 for ; Tue, 8 Apr 2008 14:21:28 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from nat-application.b1.lan.prg.vol.cz (nat-application.b1.lan.prg.vol.cz [195.122.204.152]) by mx1.freebsd.org (Postfix) with ESMTP id 159DF8FC27 for ; Tue, 8 Apr 2008 14:21:27 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from pav.hide.vol.cz (localhost [127.0.0.1]) by nat-application.b1.lan.prg.vol.cz (8.14.2/8.14.2) with ESMTP id m38ELP24086364; Tue, 8 Apr 2008 16:21:25 +0200 (CEST) (envelope-from pav@FreeBSD.org) Received: (from pav@localhost) by pav.hide.vol.cz (8.14.2/8.14.2/Submit) id m38ELPGn086363; Tue, 8 Apr 2008 16:21:25 +0200 (CEST) (envelope-from pav@FreeBSD.org) X-Authentication-Warning: pav.hide.vol.cz: pav set sender to pav@FreeBSD.org using -f From: Pav Lucistnik To: Brooks Davis In-Reply-To: <20080408141604.GA53370@lor.one-eyed-alien.net> References: <1207605059.1031.38.camel@ikaros.oook.cz> <20080408002005.W15502@fledge.watson.org> <1207642548.84150.10.camel@pav.hide.vol.cz> <20080408141604.GA53370@lor.one-eyed-alien.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-aFnpnPzXnvLxueil2+Cv" Date: Tue, 08 Apr 2008 16:21:25 +0200 Message-Id: <1207664485.84150.31.camel@pav.hide.vol.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Cc: hubs@FreeBSD.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 14:21:28 -0000 --=-aFnpnPzXnvLxueil2+Cv Content-Type: text/plain; charset=ISO8859-2 Content-Transfer-Encoding: quoted-printable Brooks Davis p=ED=B9e v =FAt 08. 04. 2008 v 09:16 -0500: > I suspect that like virtually everything else on the internet package > access is heavy-tailed. If so, no from of cooperative caching will be > of use. >=20 > For example, this paper killed all research on cooperative caching > web proxies by demostrating that its was fundamentally pointless: >=20 > http://citeseerx.ist.psu.edu/viewdoc/summary?doi=3D10.1.1.22.9583 Ok forget the proxies idea. I like the partial mirrors approach much better. It will allow us to reuse existing synchronization infrastructure, too. --=20 Pav Lucistnik If God is perfect, why did He create discontinuous functions? --=-aFnpnPzXnvLxueil2+Cv Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkf7f2UACgkQntdYP8FOsoKKLACgo9psKENhPAyE5Syabf+4S8Zg BnsAnAkrb/C3S5jWj+tcKOYAr+rlgINl =wWAc -----END PGP SIGNATURE----- --=-aFnpnPzXnvLxueil2+Cv-- From owner-freebsd-hubs@FreeBSD.ORG Tue Apr 8 16:47:12 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E092B1065677 for ; Tue, 8 Apr 2008 16:47:11 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.185]) by mx1.freebsd.org (Postfix) with ESMTP id 50EDA8FC0A for ; Tue, 8 Apr 2008 16:47:10 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: by mu-out-0910.google.com with SMTP id w9so2962695mue.6 for ; Tue, 08 Apr 2008 09:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=MJnFk+yOkH8UW6ROQ7DkT1ubIfgm48r7INTBMAzqD9A=; b=ueH9uH6SGWW7C4qGMZsVPBhmLX0GsFX7L+G4qUk7z+ndExyu+2MGy+JOGhTSOXFIQNAcaP73mXAohoMJhjO4i6hieHvLYZZXZdQMNLw6qIw/v/1cn0Ph7PFYBAwZG9bYROY66uhF5A8GpkO7mJ+vQXHOpZNU3Sj5GbFFA09nalo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=OJFfalQ9fcdM+b8DHSqq2ifxwXCovvJNb5NlkpxNDhCMUaAGpw+Zd09jdjqFBNnUmhsT16u++iK6PwO+W+x9BzGaS+bCHS/xLr2zEw+ZkDjBA9Rrt0Pnog+ybOyfYbs5QgqvvUUedi3WYPqfxgnT/J1ZrC+9slLCb5lYfqoyrC4= Received: by 10.82.116.15 with SMTP id o15mr322028buc.74.1207673229594; Tue, 08 Apr 2008 09:47:09 -0700 (PDT) Received: by 10.82.174.5 with HTTP; Tue, 8 Apr 2008 09:47:09 -0700 (PDT) Message-ID: <59adc1a0804080947v3d252c51s4fa2cf2e7182bf54@mail.gmail.com> Date: Tue, 8 Apr 2008 19:47:09 +0300 From: "Dimitar Vasilev" To: "Francois Petillon" In-Reply-To: <47FB3ADD.6070401@proxad.net> MIME-Version: 1.0 References: <1207605059.1031.38.camel@ikaros.oook.cz> <47FAE381.10305@jonny.eng.br> <59adc1a0804072312l192a98f8ncfc70e5687e45ef7@mail.gmail.com> <47FB3ADD.6070401@proxad.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 16:47:12 -0000 RnJhbmNvaXMsCnBsZWFzZSBhY2NlcHQgbXkgYXBvbG9naWVzLgoKaHR0cDovL2xpc3RzLmZyZWVi c2Qub3JnL21haWxtYW4vaHRkaWcvZnJlZWJzZC1odWJzLzIwMDYtTm92ZW1iZXIvMDAxNTM5Lmh0 bWwKCgoyMDA4LzQvOCwgRnJhbmNvaXMgUGV0aWxsb24gPGZ0cG1haW50QHByb3hhZC5uZXQ+Ogo+ Cj4gRGltaXRhciBWYXNpbGV2IHdyb3RlOgo+Cj4gPiBPbmNlIHVwb24gYSB0aW1lIGEgZ3V5IHN1 Z2dlc3RlZCB0aGF0IGFsbCBGcmVlQlNEIG1pcnJvcnMgam9pbgo+ID4gaHR0cDovL3d3dy5tZXRh bGlua2VyLm9yZy8sIGJ1dCBzb21lb25lIGZyb20gdGhlIEZyZW5jaCBtaXJyb3JzIHJlamVjdGVk Cj4gPiB0aGUKPiA+IGlkZWEuCj4gPgo+Cj4gVGhlIGZyZW5jaHkgZGlkIG5vdCByZWplY3QgdGhl IGlkZWEsIHRoZSBmcmVuY2h5IGp1c3QgdG9sZCBzZWdtZW50ZWQKPiBkb3dubG9hZHMgaXMgZnVj a2luZyB1cCBJT3Mgb24gYmlnIHNlcnZlcnMuIEFuZCBtZXRhbGlua2VyIGlzIG1haW5seSBhaW1l ZAo+IGF0IG11bHRpc291cmNlcyBzZWdtZW50ZWQgZG93bmxvYWRzLgo+Cj4gRnJhbsOnb2lzCj4K From owner-freebsd-hubs@FreeBSD.ORG Wed Apr 9 07:35:30 2008 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35322106567A for ; Wed, 9 Apr 2008 07:35:30 +0000 (UTC) (envelope-from anthonybryan@gmail.com) Received: from hs-out-0708.google.com (hs-out-0708.google.com [64.233.178.240]) by mx1.freebsd.org (Postfix) with ESMTP id E49128FC1D for ; Wed, 9 Apr 2008 07:35:29 +0000 (UTC) (envelope-from anthonybryan@gmail.com) Received: by hs-out-0708.google.com with SMTP id m63so1820550hsc.11 for ; Wed, 09 Apr 2008 00:35:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=Id25dxhvZmO0+4XK0Z+R4mpuKelTLNbFQOfHr3SKO0A=; b=fsqClni/iHaPHukyNXrlRo72cz7JjBpsxdMAIemsjU/nS4WebiszM6OBdXS8+19QOO5Vq9vc8chZBNXzz0rRpKDU4TF7kFbuy665PsVcFfJKaODUTwY39n/igQ+IcD0aJhA/vQEzwH+nL27spF7p4o0jV97B7wkP6YG3H2iWCUM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=drP3I/j840VJ1m2/mPgwwJy5c5DZpz5vy+PEquE/03b4ntXs0ErzHt7394v3tdl7/72DOKflo11aWvD43FEfSy/yH9H85U1jUrOuynD9Fb4uGULpb8oyygtwdiuZ23zjkj3ECRTNfJ/VrooUYSMhdpwIRwibREXdOakyMZQ98Ok= Received: by 10.141.203.7 with SMTP id f7mr2527160rvq.7.1207724815628; Wed, 09 Apr 2008 00:06:55 -0700 (PDT) Received: by 10.141.48.20 with HTTP; Wed, 9 Apr 2008 00:06:55 -0700 (PDT) Message-ID: Date: Wed, 9 Apr 2008 03:06:55 -0400 From: "Anthony Bryan" To: freebsd-hubs@freebsd.org, dimitar.vassilev@gmail.com MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2008 07:35:30 -0000 I hope you will investigate the MirrorBrain mentioned earlier and in the presentation at http://www.poeml.de/~poeml/talks/redirector/ to see if it meets your needs. obviously, it works well for openSUSE but will function just as well for other projects in similar circumstances. the author will be receptive to FreeBSD (& other organization's) needs as he is actively improving it for everyone, not just openSUSE. http://www.mirrorbrain.org/ replies to other messages follow: Francois Petillon wrote: > Dimitar Vasilev wrote: > > Once upon a time a guy suggested that all FreeBSD mirrors join > > http://www.metalinker.org/, but someone from the French mirrors rejecte= d the > > idea. > > The frenchy did not reject the idea, the frenchy just told segmented > downloads is fucking up IOs on big servers. And metalinker is mainly > aimed at multisources segmented downloads. ahh, Fran=E7ois, my dear colleague, you wound me! Metalink is mainly aimed at providing mirror lists and checksums for automated integrity verification and error repair. If you want to completely disable segmented downloads in metalinks, there is that option in the .metalink. a 2006 reply https://lists.ubuntu.com/archives/ubuntu-mirrors/2007-February= /000042.html Thanks for bringing this up! These are all excellent reasons for using metalink - by putting your preferences into a machine readable format that courteous users can take advantage of. (Along with configuring a server correctly and to your liking). Metalink isn't focused on segmented downloading, although many of the clients do support that, some of them don't. The increased availability from multiple links, integrity/reliability from checksums, being able to add multiple files to a transfer queue and other automated features are also found useful by others. Ubuntu, OpenOffice.org, or other organizations can choose to limit (in the metalink, according to the spec) the number of connections used for downloading a file, or for specific servers. That is, you can say in the metalink that you don't want programs doing segmented downloading, or restrict them to a certain number of connections to a server. Of course, servers still need to be correctly configured because, as you say, other programs will not respect your wishes and use up resources. Dimitar Vassilev, I'm sorry I didn't reply to your message from Nov 2006 ( http://lists.freebsd.org/mailman/htdig/freebsd-hubs/2006-November/00= 1544.html ), I didn't see it until now or I would have replied. I believe some of the server side metalink generators do what you are asking about. --=20 (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] )) Easier, More Reliable, Self Healing Downloads From owner-freebsd-hubs@FreeBSD.ORG Thu Apr 10 20:14:43 2008 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 699D4106566B for ; Thu, 10 Apr 2008 20:14:43 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id E6B228FC0A for ; Thu, 10 Apr 2008 20:14:42 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id m3AJdmu1021230 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 10 Apr 2008 21:39:48 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id m3AJddLj026875 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 10 Apr 2008 21:39:40 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id m3AJdd2q060857; Thu, 10 Apr 2008 21:39:39 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id m3AJdddU060856; Thu, 10 Apr 2008 21:39:39 +0200 (CEST) (envelope-from ticso) Date: Thu, 10 Apr 2008 21:39:39 +0200 From: Bernd Walter To: Pav Lucistnik Message-ID: <20080410193938.GO55365@cicely12.cicely.de> References: <1207605059.1031.38.camel@ikaros.oook.cz> <20080408002005.W15502@fledge.watson.org> <1207642548.84150.10.camel@pav.hide.vol.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1207642548.84150.10.camel@pav.hide.vol.cz> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: hubs@freebsd.org Subject: Re: package distribution crisis - CDN needed X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2008 20:14:43 -0000 On Tue, Apr 08, 2008 at 10:15:48AM +0200, Pav Lucistnik wrote: > Eberhard Moenkeberg pí?e v út 08. 04. 2008 v 01:54 +0200: > Now this looks so much better and easier than my CDN idea! We could use > it practically as-is. > > We will need to patch pkg_install to use http://redirector.FreeBSD.org/ > instead of ftp://ftp.FreeBSD.org/ No need to patch - just setup an http server as your source. Our libfetch is flexible enough to hide the transfer details from any client. I use it within ports for source file fetching since a long time. And it would be possible to setup apache as transparent caching proxy service. No need to write new software or use GPL at all. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-hubs@FreeBSD.ORG Fri Apr 11 00:05:32 2008 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C715B1065670 for ; Fri, 11 Apr 2008 00:05:32 +0000 (UTC) (envelope-from zilla@antik.sk) Received: from mailserver.antik.sk (mailserver.antik.sk [88.212.10.6]) by mx1.freebsd.org (Postfix) with ESMTP id 13F3D8FC12 for ; Fri, 11 Apr 2008 00:05:31 +0000 (UTC) (envelope-from zilla@antik.sk) Received: (qmail 2058 invoked from network); 11 Apr 2008 01:38:49 +0200 Received: by simscan 1.4.0 ppid: 2052, pid: 2055, t: 0.0161s scanners: regex: 1.4.0 attach: 1.4.0 clamav: 0.92.1 /m: 46/d:6689 Received: from unknown (HELO ?172.16.121.100?) (zilla@antik.sk@172.16.121.100) by mailserver.antik.sk with AES256-SHA encrypted SMTP; 11 Apr 2008 01:38:49 +0200 Message-ID: <47FEA509.5010501@antik.sk> Date: Fri, 11 Apr 2008 01:38:49 +0200 From: Matus Zilla User-Agent: Mozilla-Thunderbird 2.0.0.9 (X11/20080110) MIME-Version: 1.0 To: freebsd-hubs@freebsd.org X-Enigmail-Version: 0.95.0 OpenPGP: id=2A99FD9A Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigC601D88079B332613BD99118" Subject: New mirror in Slovakia X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2008 00:05:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigC601D88079B332613BD99118 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi all, we are an major ISP in region with many FreeBSD servers in our network. I have succesfully created new FreeBSD mirror in Slovakia. It is accessible through FTP/HTTP/RSYNC://ftp.antik.sk or freebsd.antik.sk. I have also set up a CVSup mirror. All is mirrored from ftp.cz.freebsd.org like recommended, every day and CVSup every hour, because ftp.sk.freebsd.org has an IPv6 address, but it looks like only a tunnel from xs26 and the IPv6 doesn`t work. The server have a 1Gbit/s connection to backbone, with native IPv6 support for all services (http,ftp,rsync and also CVSup). Running on FreeBSD 7.0 with ZFS pool as data FS. You can checks the stats of the server on http://ftp.antik.sk/munin/ The server is also an mirror for Debian,Ubuntu and tucows. I would like to become second official mirror for FreeBSD in Slovakia. Sincerely Matus Zilla --------------enigC601D88079B332613BD99118 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH/qUJn+h45yqZ/ZoRAicIAKCRAjIP0i4TZxwvJPg0tU7gDCgcDQCeLfuD EFxdnBUgnPLVuXbHANZ3xO8= =fi38 -----END PGP SIGNATURE----- --------------enigC601D88079B332613BD99118--