From owner-freebsd-hubs Sat May 29 0:24:30 1999 Delivered-To: freebsd-hubs@freebsd.org Received: from huset.math.ntnu.no (huset.math.ntnu.no [129.241.211.212]) by hub.freebsd.org (Postfix) with SMTP id 4067D14D8A for ; Sat, 29 May 1999 00:24:27 -0700 (PDT) (envelope-from arnej) Received: (qmail 14024 invoked from network); 29 May 1999 07:24:26 -0000 Received: from localhost (127.0.0.1) by localhost with SMTP; 29 May 1999 07:24:26 -0000 To: taavi@uninet.ee Cc: freebsd-hubs@freebsd.org Subject: Re: Best way to mirror FreeBSD ftp archive? In-Reply-To: Your message of "Mon, 24 May 1999 12:39:47 +0300 (EEST)" References: X-Mailer: Mew version 1.92.4 on Emacs 19.34 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <19990529092426C.arnej@unknown-domain> Date: Sat, 29 May 1999 09:24:26 +0200 From: Arne Henrik Juul X-Dispatcher: imput version 980905(IM100) Lines: 94 Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org From: Taavi Talvik > Hello ! > > I am running FreeBSD ftp mirror for Estonia. However I have > trouble trying to stay up to date. > > US site is quite far (in network terms) from us. Having to > refetch identical files just makes impossible for me stay > up to date. I have the same problems, and I've found that it's hopeless to get everything directly from the US. Particularly the packages tree is bad: Some of the trees there are updated regularly with more than a gigabyte result. Since you are in Estonia, you should have a pretty good link to Sweden, so I'll recommend fetching at least the big parts of the tree from there (ftp.sunet.se), they are very good at keeping up-to-date. > It's quite grazy to see gigabytes of files removed and aftewards > refetched.. Exactly :-) Currently I have an 8-gig disk for /pub/FreeBSD/ports, which means I have to include only 3 versions of the packages: 1019M /pub/FreeBSD/ports/i386/packages-2.2.8 1.2G /pub/FreeBSD/ports/i386/packages-3.2-release 1.1G /pub/FreeBSD/ports/i386/packages-4-current I probably should have at least one more, but I need to get more disk first. > Are there tools similiar to CVSup for mirroring ftp.freebsd.org > and allowing to specify parts of directory tree to include/exclude? There are various general FTP-mirroring tools, but they all are limited by the FTP protocol. I'm using "fmirror" which I find to be very effective and precise. ftp://ftp.guardian.no/pub/free/ftp/fmirror/ This is my configuration file for the 'ports' part of the tree: >>>> timefuzz: 200000 keep_newer: 1 username: anonymous password: arnej@math.ntnu.no host: ftp0.sunet.se remotedir: /pub/FreeBSD/ports localdir: /pub/FreeBSD/ports exclude: p /packages-2.2.7/ exclude: p /packages-2.2-stable/ exclude: p /packages-3-stable/ exclude: p /packages-3.1-release/ exclude: p /packages-3.0-aout/ exclude: p /packages-3.0/ exclude: f ^\.mutex_build$ exclude: f ^\.notar$ exclude: f ^\.nocompress$ exclude: f ^\.in\. exclude: f packages-3.0-stable exclude: f packages-3.1-stable exclude: f packages-3.2-stable exclude: f packages-stable exclude: f packages <<<< fmirror can also do things like get an ls-lR.gz file that was generated on the server, unzip and parse it and use it for deciding what to get. This speeds up things a lot. (I mirror everything that's not in "ports/" directly from ftp.freebsd.org with one single configuration file, which normally uses a couple of hours to run.) > Are there any documents describing which part of archive is > "stable", which parts is changing say "weekly" or changing "daily". I don't think this is documented, but some things that I have observed: development/FreeBSD-CVS is updated with cvsup (daily?), so you could do the same instead of FTP-mirroring it, or just skip it (since people will mostly use cvsup anyway.) The branches/* trees are mostly (?) just checked out from that CVS repository, and they are hard to FTP-mirror because it's lots and lots of small files. (There's a branches/-current/ports.tar.gz that you can get instead of trying to get all the small files, that helps a lot.) "releases" is a stable directory and "snapshots" is updated pretty often, just as one would guess. Skip snapshots or get them from ftp.sunet.se, is my recommendation. Another large directory is "development/3.1-express" (900M) and that one has also been completely updated at least once recently, so you may want to skip this one as well. - Arne H. J. (maintainer of ftp.pvv.ntnu.no:/pub/FreeBSD/) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message