Date: Sat, 29 May 1999 09:24:26 +0200 From: Arne Henrik Juul <arnej@FreeBSD.ORG> To: taavi@uninet.ee Cc: freebsd-hubs@freebsd.org Subject: Re: Best way to mirror FreeBSD ftp archive? Message-ID: <19990529092426C.arnej@unknown-domain> In-Reply-To: Your message of "Mon, 24 May 1999 12:39:47 %2B0300 (EEST)" <Pine.BSF.3.95.990524120005.11803A-100000@ns.uninet.ee> References: <Pine.BSF.3.95.990524120005.11803A-100000@ns.uninet.ee>
next in thread | previous in thread | raw e-mail | index | archive | help
From: Taavi Talvik <taavi@uninet.ee> > Hello ! > > I am running FreeBSD ftp mirror for Estonia. However I have > trouble trying to stay up to date. > > US site is quite far (in network terms) from us. Having to > refetch identical files just makes impossible for me stay > up to date. I have the same problems, and I've found that it's hopeless to get everything directly from the US. Particularly the packages tree is bad: Some of the trees there are updated regularly with more than a gigabyte result. Since you are in Estonia, you should have a pretty good link to Sweden, so I'll recommend fetching at least the big parts of the tree from there (ftp.sunet.se), they are very good at keeping up-to-date. > It's quite grazy to see gigabytes of files removed and aftewards > refetched.. Exactly :-) Currently I have an 8-gig disk for /pub/FreeBSD/ports, which means I have to include only 3 versions of the packages: 1019M /pub/FreeBSD/ports/i386/packages-2.2.8 1.2G /pub/FreeBSD/ports/i386/packages-3.2-release 1.1G /pub/FreeBSD/ports/i386/packages-4-current I probably should have at least one more, but I need to get more disk first. > Are there tools similiar to CVSup for mirroring ftp.freebsd.org > and allowing to specify parts of directory tree to include/exclude? There are various general FTP-mirroring tools, but they all are limited by the FTP protocol. I'm using "fmirror" which I find to be very effective and precise. ftp://ftp.guardian.no/pub/free/ftp/fmirror/ This is my configuration file for the 'ports' part of the tree: >>>> timefuzz: 200000 keep_newer: 1 username: anonymous password: arnej@math.ntnu.no host: ftp0.sunet.se remotedir: /pub/FreeBSD/ports localdir: /pub/FreeBSD/ports exclude: p /packages-2.2.7/ exclude: p /packages-2.2-stable/ exclude: p /packages-3-stable/ exclude: p /packages-3.1-release/ exclude: p /packages-3.0-aout/ exclude: p /packages-3.0/ exclude: f ^\.mutex_build$ exclude: f ^\.notar$ exclude: f ^\.nocompress$ exclude: f ^\.in\. exclude: f packages-3.0-stable exclude: f packages-3.1-stable exclude: f packages-3.2-stable exclude: f packages-stable exclude: f packages <<<< fmirror can also do things like get an ls-lR.gz file that was generated on the server, unzip and parse it and use it for deciding what to get. This speeds up things a lot. (I mirror everything that's not in "ports/" directly from ftp.freebsd.org with one single configuration file, which normally uses a couple of hours to run.) > Are there any documents describing which part of archive is > "stable", which parts is changing say "weekly" or changing "daily". I don't think this is documented, but some things that I have observed: development/FreeBSD-CVS is updated with cvsup (daily?), so you could do the same instead of FTP-mirroring it, or just skip it (since people will mostly use cvsup anyway.) The branches/* trees are mostly (?) just checked out from that CVS repository, and they are hard to FTP-mirror because it's lots and lots of small files. (There's a branches/-current/ports.tar.gz that you can get instead of trying to get all the small files, that helps a lot.) "releases" is a stable directory and "snapshots" is updated pretty often, just as one would guess. Skip snapshots or get them from ftp.sunet.se, is my recommendation. Another large directory is "development/3.1-express" (900M) and that one has also been completely updated at least once recently, so you may want to skip this one as well. - Arne H. J. (maintainer of ftp.pvv.ntnu.no:/pub/FreeBSD/) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19990529092426C.arnej>