From owner-freebsd-hubs Sun May 20 22:24: 9 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from aurora.sol.net (aurora.sol.net [206.55.65.76]) by hub.freebsd.org (Postfix) with ESMTP id 4D44C37B422 for ; Sun, 20 May 2001 22:24:07 -0700 (PDT) (envelope-from jgreco@aurora.sol.net) Received: (from jgreco@localhost) by aurora.sol.net (8.9.3/8.9.2/SNNS-1.02) id AAA71783 for hubs@freebsd.org; Mon, 21 May 2001 00:24:01 -0500 (CDT) From: Joe Greco Message-Id: <200105210524.AAA71783@aurora.sol.net> Subject: Re: We seriously need a cleanup on ftp-master To: hubs@freebsd.org Date: Mon, 21 May 2001 00:24:01 -0500 (CDT) X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > My guess is there won't be many signing up for the archive role. > Mirror operators typically need to justify their existence to someone. > *Usually* that justification will be having locally what it takes to > do a FreeBSD install and wind up with a current machine. That's not > what all end-users want to do but that's what the bulk of them want to > do. If a mirror operator is willing to store more than what you > propose ftp.freebsd.org would hold that's nice of them but I don't see > where being a site that has older stuff only is going to attract a lot > of folks. I could be wrong of course. As an archival site right now, I'll agree that there's not a lot of folks. However, the justification for such sites is the folks who do find it useful. I have FreeBSD releases dating back... well, way back. I didn't do the ports or packages, just the base OS. There's a fair number of accesses to these. I don't have 5000 people logged in on the server or anything, but it's nice for stuff to hang around where it can maybe be found. Try to remember that the purpose of all this is to serve the users... it is really irritating for a user to run across the need for something and then be told to upgrade to -stable to get it. Some stuff just doesn't work on newer FreeBSD's, or works better on older ones. -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Mon May 21 23:24:31 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from impatience.valueclick.com (impatience.valueclick.com [216.246.96.99]) by hub.freebsd.org (Postfix) with SMTP id 9A3DB37B422 for ; Mon, 21 May 2001 23:24:29 -0700 (PDT) (envelope-from ask@valueclick.com) Received: (qmail 24707 invoked by uid 500); 22 May 2001 06:24:28 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 22 May 2001 06:24:28 -0000 Date: Mon, 21 May 2001 23:24:28 -0700 (PDT) From: Ask Bjoern Hansen To: Gordon Tetlow Cc: Peter Wemm , hubs@FreeBSD.ORG Subject: Re: load distribution In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, 10 May 2001, Gordon Tetlow wrote: [...] > > If I in some automated way can get a list of mirrors with their > > country and (optionally) a weighting it's very easy to add support for > > ftp.freebsd.org. > > > > http://sourceforge.net/projects/dinamed/ > > Actually, I like the supersparrow project better, it pulls info from bgp > tables and goes from there. This approximates even better and you don't > have to figure out what IP block goes where. I'm not exactly sure how well > it works, but it looks kinda cool. yeah, I had a look at it. I didn't have a BGP feed available to test with and the "country / continent" load balancing works reasonably well for international users at least (or much much better than just having everyone go to (www|ftp).cpan.org (which they do)). - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do(); more than 100M impressions per day, http://valueclick.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Thu May 24 17:14:26 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from dns1.ahaza.com (maitai.ahaza.com [209.180.220.98]) by hub.freebsd.org (Postfix) with ESMTP id 32BE437B422 for ; Thu, 24 May 2001 17:14:24 -0700 (PDT) (envelope-from tim@ahaza.com) Received: from relay.ux.ahaza.com ([209.180.221.130]) by dns1.ahaza.com (8.11.1/8.11.1) with ESMTP id f4P0ENF75937 for ; Thu, 24 May 2001 17:14:23 -0700 (PDT) (envelope-from tim@ahaza.com) Received: from eunice.camelot.ahaza.com (eunice.camelot.ahaza.com [172.16.30.18]) by relay.ux.ahaza.com (8.11.1/8.11.1) with ESMTP id f4P0ENq56650 for ; Thu, 24 May 2001 17:14:23 -0700 (PDT) (envelope-from tim@ahaza.com) X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 content-class: urn:content-classes:message Subject: FreeBSD cvs/ftp mirror MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Thu, 24 May 2001 17:09:08 -0700 Message-ID: <60E8E68FF6165C43822B75708D64189CB682@eunice.camelot.ahaza.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FreeBSD cvs/ftp mirror Thread-Index: AcDkrurE02k7UVJaTHWMaOIu6B8/oQ== From: "Tim Flowers" To: Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I am interested in becoming an official FreeBSD cvs/ftp mirror site. I currently have an OC-3 internet connection. Please let us know what steps we need to take in order to become an official mirror. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Fri May 25 22:47:27 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from piglet.dstc.edu.au (piglet.dstc.edu.au [130.102.176.1]) by hub.freebsd.org (Postfix) with ESMTP id 6608437B423 for ; Fri, 25 May 2001 22:47:24 -0700 (PDT) (envelope-from jason@dstc.edu.au) Received: from azure.dstc.edu.au (azure.dstc.edu.au [130.102.176.27]) by piglet.dstc.edu.au (8.11.3/8.11.3) with ESMTP id f4Q5lIK23126 for ; Sat, 26 May 2001 15:47:18 +1000 (EST) Date: Sat, 26 May 2001 15:47:21 +1000 (EST) From: jason andrade To: hubs@freebsd.org Subject: final state of mirroring ? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Scanned-By: MIMEDefang 1.0 (http://www.roaringpenguin.com/mimedefang/) Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I've seen a lot of discussion - what is the final state of what mirrors should be doing ? Are people mirroring from ftp-master daily now ? Is it complete and being updated ? -jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Sat May 26 9:54:12 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from freesbee.wheel.dk (freesbee.wheel.dk [193.162.159.97]) by hub.freebsd.org (Postfix) with ESMTP id 1CDAE37B423 for ; Sat, 26 May 2001 09:54:10 -0700 (PDT) (envelope-from jesper@skriver.dk) Received: by freesbee.wheel.dk (Postfix, from userid 1001) id 863A65D3B; Sat, 26 May 2001 18:55:51 +0200 (CEST) Date: Sat, 26 May 2001 18:55:51 +0200 From: Jesper Skriver To: jason andrade Cc: hubs@freebsd.org Subject: Re: final state of mirroring ? Message-ID: <20010526185551.B94185@skriver.dk> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from jason@dstc.edu.au on Sat, May 26, 2001 at 03:47:21PM +1000 X-PGP-Fingerprint: 6B88 9CE8 66E9 E631 C9C5 5EB4 22AB F0EC F956 1C31 X-PGP-Public-Key: http://freesbee.wheel.dk/~jesper/gpgkey.pub Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sat, May 26, 2001 at 03:47:21PM +1000, jason andrade wrote: > > I've seen a lot of discussion - what is the final state of what > mirrors should be doing ? > > Are people mirroring from ftp-master daily now ? Is it complete > and being updated ? It is what is being used at ftp.FreeBSD.org, which sync from ftp-master via rsync once a day, same applied to ftp.dk.FreeBSD.org /Jesper -- Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456 Work: Network manager @ AS3292 (Tele Danmark DataNetworks) Private: FreeBSD committer @ AS2109 (A much smaller network ;-) One Unix to rule them all, One Resolver to find them, One IP to bring them all and in the zone to bind them. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Sat May 26 20:35: 2 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from eden.ispol.com (eden.ispol.com [206.239.103.254]) by hub.freebsd.org (Postfix) with ESMTP id 4981B37B424 for ; Sat, 26 May 2001 20:34:59 -0700 (PDT) (envelope-from grisha@verio.net) Received: from localhost (grisha@localhost) by eden.ispol.com (8.11.1/8.11.1) with ESMTP id f4R3YlM63526; Sat, 26 May 2001 23:34:48 -0400 (EDT) (envelope-from grisha@verio.net) X-Authentication-Warning: eden.ispol.com: grisha owned process doing -bs Date: Sat, 26 May 2001 23:34:47 -0400 (EDT) From: "Gregory (Grisha) Trubetskoy" X-X-Sender: To: Jesper Skriver Cc: jason andrade , Subject: Re: final state of mirroring ? In-Reply-To: <20010526185551.B94185@skriver.dk> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org and ftp2.freebsd.org :) Grisha On Sat, 26 May 2001, Jesper Skriver wrote: > On Sat, May 26, 2001 at 03:47:21PM +1000, jason andrade wrote: > > > > I've seen a lot of discussion - what is the final state of what > > mirrors should be doing ? > > > > Are people mirroring from ftp-master daily now ? Is it complete > > and being updated ? > > It is what is being used at ftp.FreeBSD.org, which sync from > ftp-master via rsync once a day, same applied to > ftp.dk.FreeBSD.org > > /Jesper > > -- > Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456 > Work: Network manager @ AS3292 (Tele Danmark DataNetworks) > Private: FreeBSD committer @ AS2109 (A much smaller network ;-) > > One Unix to rule them all, One Resolver to find them, > One IP to bring them all and in the zone to bind them. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-hubs" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Sat May 26 21:50:24 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from piglet.dstc.edu.au (piglet.dstc.edu.au [130.102.176.1]) by hub.freebsd.org (Postfix) with ESMTP id A3CD837B423 for ; Sat, 26 May 2001 21:50:18 -0700 (PDT) (envelope-from jason@dstc.edu.au) Received: from azure.dstc.edu.au (azure.dstc.edu.au [130.102.176.27]) by piglet.dstc.edu.au (8.11.3/8.11.3) with ESMTP id f4R4o3K05470; Sun, 27 May 2001 14:50:03 +1000 (EST) Date: Sun, 27 May 2001 14:50:05 +1000 (EST) From: jason andrade To: "Gregory (Grisha) Trubetskoy" Cc: Jesper Skriver , hubs@FreeBSD.ORG Subject: Re: final state of mirroring ? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Scanned-By: MIMEDefang 1.0 (http://www.roaringpenguin.com/mimedefang/) Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org This is somewhat long - apologies if i've brought up things already covered but the last discussion was a while ago and i don't feel consensus was reached (maybe i missed it :-) To summarise where things are: o ftp-master.freebsd.org is now the authoritative "master" site for tier 1 mirrors. where possible, tier 1 mirrors should switch to mirroring from ftp-master rather than ftp.freebsd.org o a tier 1 mirror is defined as the main mirror for a given country (e.g ftp.uk.freebsd.org). it may also include other mirrors within that country in some circumstances (politics, bandwidth issues, whatever) o a tier 2 mirror is defined as a non official mirror (not registered within the xx.freebsd.org namespace) or a mirror that updates from a tier 1 mirror. there is no "status" difference to the community that is served by a tier2 mirror except that they are *not* required to be a full mirror and may in fact carry a much more restricted subset of freebsd (e.g no alpha, or only the latest 2 releases) o ftp.freebsd.org and ftp2.freebsd.org are now updating from ftp-master.freebsd.org (side pedantic node, can we also cname ftp-master to ftp0 and ftp to ftp1). down the track this allows for ftp.freebsd.org to be a RR based system or even better, to use Horm's global content distribution project to provide a "work out where you are and redirect you to the closest tier 1 mirror" this is somewhat controversial and needs to be debated: o tier1 mirrors should carry a *complete* mirror of ftp-master? or create a tier0 category of only complete mirrors which are then part of the ftpX.freebsd.org hierarchy as well as ftpX.xx.freebsd.org. otherwise maybe there is a system which says it is mandatory for a tier1 to carry: releases/? ports/? branches/? CERT/ doc/ whatever.. Things i've noticed: o ftp-master no longer appears to cleanup versions of files in the ports/distfiles area. this is to some extent annoying because rather than no structure at all in distfiles, it's to the stage there is a non consistent structure. i really don't understand why there are 7 versions of one package, what LOCAL/ is for, vs local-distfiles/ in another area and why some packages get to have a separate directory and others don't. The growth in multiple versions i think will start to impact on mirrors soon - disk space and bandwidth. o the cleanup of the whole site has stalled a bit - parts have been cleaned up but there's still historical stuff lying around. o the questions of snapshots and archival status doesn't appear to have been resolved. my personal opinion is that i would like to see the "latest" snapshot being available via ftp-master which can be carried by tier-1 mirrors. however it is *NOT* implemented as something-snapshot-$DATE which results in a deletion and refetch of all files, using rsync. instead something like snapshots/i386/freebsd-4.X/ snapshots/i386/freebsd-4.X-DATE (symlink which is changed as it is updated) iso images should never be available in the snapshots area. o what is the mechanism for mirroring the website ? o what are the total services offered by / can be offered by the freebsd project to end users ? ftp access to the archive http access to the archive rsync access to the archive http access to the web site cvsupd read/updates access to the source trees cvsweb read access to the source trees sup (?) read access to the source trees those are the main ones i was thinking of - there are probably lots of "esoteric" or "non mainstream" things like NFS access, SMB access, IPv6 access, InterMezzo access.. which i didn't want to get into the scope of this discussion. regards, -jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message From owner-freebsd-hubs Sat May 26 23:25:11 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from winston.osd.bsdi.com (adsl-64-173-15-98.dsl.sntc01.pacbell.net [64.173.15.98]) by hub.freebsd.org (Postfix) with ESMTP id F2EC637B422 for ; Sat, 26 May 2001 23:25:05 -0700 (PDT) (envelope-from jkh@osd.bsdi.com) Received: from localhost (jkh@localhost [127.0.0.1]) by winston.osd.bsdi.com (8.11.3/8.11.3) with ESMTP id f4R6OnO19633; Sat, 26 May 2001 23:24:50 -0700 (PDT) (envelope-from jkh@osd.bsdi.com) To: jason@dstc.edu.au Cc: grisha@verio.net, jesper@skriver.dk, hubs@FreeBSD.ORG Subject: Re: final state of mirroring ? In-Reply-To: References: X-Mailer: Mew version 1.94.1 on Emacs 20.7 / Mule 4.0 (HANANOEN) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <20010526232449W.jkh@osd.bsdi.com> Date: Sat, 26 May 2001 23:24:49 -0700 From: Jordan Hubbard X-Dispatcher: imput version 20000228(IM140) Lines: 102 Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org From: jason andrade Subject: Re: final state of mirroring ? Date: Sun, 27 May 2001 14:50:05 +1000 (EST) > o ftp-master.freebsd.org is now the authoritative "master" > site for tier 1 mirrors. where possible, tier 1 mirrors > should switch to mirroring from ftp-master rather than > ftp.freebsd.org Correct. If they really want to continue to mirror from ftp.freebsd.org then then can compete for the scarcer resources, but there's no reason why a legitimate tier-1 mirror should have to. > o a tier 1 mirror is defined as the main mirror for a given > country (e.g ftp.uk.freebsd.org). it may also include > other mirrors within that country in some circumstances > (politics, bandwidth issues, whatever) That and a willingness to maintain a _full_ mirror of ftp-master, yes. > o a tier 2 mirror is defined as a non official mirror (not > registered within the xx.freebsd.org namespace) or a mirror > that updates from a tier 1 mirror. there is no "status" > difference to the community that is served by a tier2 mirror > except that they are *not* required to be a full mirror and > may in fact carry a much more restricted subset of freebsd > (e.g no alpha, or only the latest 2 releases) Correct. > o ftp.freebsd.org and ftp2.freebsd.org are now updating from > ftp-master.freebsd.org (side pedantic node, can we also > cname ftp-master to ftp0 and ftp to ftp1). We'll probably have at least 5 machines updating from ftp-master in the long run. I's also possible that "ftp.freebsd.org" will someday stop pointing to a single site and instead become a round-robin DNS or Inktomi/F5 load-balanced, least-cost routed mirror. We reserve the right to do that with "ftp.freebsd.org" at any time, though it's also something which would only happen if we could guarantee that "ftp.freebsd.org" would repeatedly resolve to a site containing the exact same bits each time. > o tier1 mirrors should carry a *complete* mirror of ftp-master? Yes. > o ftp-master no longer appears to cleanup versions of files in > the ports/distfiles area. this is to some extent annoying That's manpower constrained right now. If we can get people to dive in and start pruning/organizing this like it used to be, it will be. Satoshi used to do it but it appears that he's being progressively sidelined by Real Work(tm) nowadays. > o the cleanup of the whole site has stalled a bit - parts have > been cleaned up but there's still historical stuff lying around. This is an ongoing process as time permits and will probably always hold true for some value of "historical." > o the questions of snapshots and archival status doesn't appear to > have been resolved. Snapshots will continue to be made at stable.freebsd.org and current.freebsd.org for those respective branches. I won't be copying them to ftp-master.freebsd.org due to the increased cost to the tier-1 mirrors and keeping just one snapshot around for a *day* (or until it's outdated by the next successful snapshot) would be silly - it would be gone before some of the slower mirrors could get it. I also don't want to get into the argument of keeping n snapshots around since for a reasonable value of n, you might just as well go to ${branch}.freebsd.org and have access to the last 30 or so. As to archival stuff, nobody seems all that interested in hosting those bits, at least nobody we could put into some sort of meaningful and long-lived archival site taxonomy. One is currently just as well served by using ftpsearch and finding one of the many "unofficial archives" out there. > o what are the total services offered by / can be offered by the freebsd > project to end users ? ftp and rsync seem to be about the most likely services that we can offer and still have some form of security and sustainable "costs" in terms of running each service. Not that even just these two are not without cost. A running rsync invocation takes up a lot of memory, as a small snapshot of ftp-master shows: 88649 ftp 2 0 13492K 13144K select 11:55 0.00% 0.00% rsync 90604 ftp 2 0 19016K 18468K select 8:48 0.00% 0.00% rsync 95678 ftp 2 0 19032K 18488K select 4:31 0.00% 0.00% rsync 95775 ftp 2 0 19032K 18488K select 3:08 0.00% 0.00% rsync And that's about a third of the total rsync processes running. I edited the list for brevity's sake. http transfers are also pretty expensive, from what I hear, and the security people get the wobblies at the idea of running apache on the site as well. Who knows though - could happen. - Jordan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message