From owner-freebsd-hubs Sun Mar 9 17:25:29 2003 Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E2E0C37B401; Sun, 9 Mar 2003 17:25:27 -0800 (PST) Received: from luna.rtfmconsult.com (luna.rtfmconsult.com [202.83.72.190]) by mx1.FreeBSD.org (Postfix) with ESMTP id 150C443FBD; Sun, 9 Mar 2003 17:25:27 -0800 (PST) (envelope-from jason@rtfmconsult.com) Received: by luna.rtfmconsult.com (Postfix, from userid 42) id 8D2C848E6F; Mon, 10 Mar 2003 11:25:25 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by luna.rtfmconsult.com (Postfix) with ESMTP id 8562952D92; Mon, 10 Mar 2003 11:25:25 +1000 (EST) Date: Mon, 10 Mar 2003 11:25:25 +1000 (EST) From: jason andrade To: David O'Brien Cc: hubs@FreeBSD.ORG Subject: Re: Poor state of some top-level FTP mirrors In-Reply-To: <20030309215448.GB30033@dragon.nuxi.com> Message-ID: References: <20030309215448.GB30033@dragon.nuxi.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sun, 9 Mar 2003, David O'Brien wrote: > In my never ending quest to find an alternate ftp site for "pkg_add -r", > I've tried many of our ftpX.freebsd.org mirrors. The problem is the > mirrors only seem to be consistent for short periods of time. > > As a simple test, the documented mirrors was polled for the file > ftp://${SITE}/pub/FreeBSD/ports/i386/packages-4-stable/All/bonnie++-1.93.03.tgz > which was updated on ftp.freebsd.org 2-March-2003. the methodology of the way the packages tree is set out makes it very difficult to provide a stable and uptodate mirror. the freebsd archive is one of the few that has a system which causes an entire package tree to be rebuilt (and hence refetched) over and over again with no ability to only fetch updated or changed packages and the packages trees make up the bulk of the freebsd archive. keeping that in mind, as well as the load this puts on bandwidth (how many freebsd mirrors are there ? say 135 from below's stats) where at 6G for this tree, that is almost a Terabyte of data that is pushed out in various directions almost weekly for one tree. would anyone say that there is even half a terabyte of (compressed) code that is changing weekly ? or monthly.. i'd be happy to be proven wrong on this but i suspect it is one of the major issues behind the impossibility of keeping an up to date mirror of the packages tree, and this is made worse by having multiple architectures and releases now, e.g packages-4-stable, packages-5-current for i386, alpha, and eventually sparc64 and others. here at ftp.au.freebsd.org i try and keep the packages-4-stable tree updated once a week to once a fortnight. it still takes most of a day - sometimes 3 or 4 days, to update one tree. regards, -jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message