Date: Fri, 04 Apr 1997 18:06:31 +0100 From: Lee McLoughlin <lmjm@doc.ic.ac.uk> To: Christoph Kukulies <kuku@gilberto.physik.rwth-aachen.de> Cc: "Jordan K. Hubbard" <jkh@time.cdrom.com>, hubs@freebsd.org Subject: Re: Things moving around on ftp://ftp.freebsd.org/pub/FreeBSD Message-ID: <33453517.2781E494@doc.ic.ac.uk> References: <17711.860119647@time.cdrom.com> <Mutt.19970404154517.kuku@gil.physik.rwth-aachen.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Christoph Kukulies wrote: > > I switched to mirroring single pieces although I have all /pub/FreeBSD > in a single (dual drive) 6GB ccd device so it wouldn't affect > ftp.de.freebsd.org. (Machine is a AmdK5/133 w 64MB and 128 MB swap). > > I switched because mirror 2.8 (with perl5.003) *never*, really *never*, > finished mirroring the whole tree at once. I suspect it a problem > deep in mirror and/or perl. One time I let the mirror process run for 5 days > with debug on and it turned out that it needed 5 minutes to decide > for the next Get (the actual file transfer time was negeglible). This > resulted in a predicted (at 40,000 files to fetch - give some worse case) > time of 158 days (theoretically). I was suspecting some 2^n effect in > searching or db looking up. Anyway, now breaking it into pieces again > seems to get through. Personally I blame perl :-> The next, perl, release of mirror will have an option to mirror directory by directory. My worry is that this will require an 'ls -l' to be run in each directory on the server - this could prove to be crippling to some servers. For FreeBSD it could mean replacing one 'get ls-lR' by 19,000 calls to 'ls -l'! Anyhow this will be discussed further on the mirror mailing list. > > So my question here - sorry that I jumped into this thread: > > Is *anyone* out there mirroring the whole tree in one piece using mirror > *and* is using a FreeBSD-x.x.x machine for this? I would be very > interested in knowing this. Some sites I know of use big Sun machines > and they don't seem to suffer from this problem. > > > since mirror should follow the symlink. If you're grabbing all > > Does it? I'm not sure. mirror has a variable: follow_local_symlink='' > ( define empty by default) and I'm not sure what this means, local from > the view of mirror would be the machine mirror runs on from what I > understand. I could use this on my system here in the UK to have a symlink called 'incoming' pointing elsewhere. But mirror would treat this like a real directory and NOT remove it to replace by a real directory as it normally would do. The trick that some people do is to add the 'L' flag to the flags_recursive so that when ls is run all the remote symlinks are expanded out. However if symlinks inside the area point to one of their parent directories this results in ls going into an infinite loop. -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8432 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?33453517.2781E494>