Date: Mon, 7 May 2001 12:04:20 +0200 From: Cejka Rudolf <cejkar@dcse.fee.vutbr.cz> To: Peter Wemm <peter@wemm.org> Cc: hubs@FreeBSD.ORG Subject: Re: ftp-master method Message-ID: <20010507120419.A42291@dcse.fee.vutbr.cz> In-Reply-To: <20010506213812.3BAFE380E@overcee.netplex.com.au>; from peter@wemm.org on Sun, May 06, 2001 at 02:38:12PM -0700 References: <200105041644.f44GiuO54477@vashon.polstra.com> <20010506213812.3BAFE380E@overcee.netplex.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Peter Wemm wrote (2001/05/06):
> One gripe I have with rsync itself is that it seems to build its file
> lists right from the start and holds them in memory.. It then streams
> the updates. This means that rsyncd consumes about 50-60MB of ram on the
> server at all times (for an update). It does its delete pass right at then
> end, if nothing went wrong.
Hmm, it seems that rsync is still a big memory killer :-( I have
tried a dry run from ftp-master and rsync took about 96 MB of RAM
(46 MB when syncing and the next 50 MB in deletion phase) for full
FreeBSD tree. If I exclude branches, FreeBSD-CVS and mailing-lists,
rsync takes about 44 MB of RAM (20 MB + 24 MB). I hope that second
rsync process with similar memory requirements just shares memory
pages with the first process. If I use mirror and algorithm=1, it
could not exceed 10 MB of RAM. It would be great to see similar
memory usage with rsync too...
However, the big advantage of rsync over ftp method is its speed
and I really want to switch to rsync or combine it with ftp.
* What does mean symbolic link
development/CTM/CTM -> usr/home/ftp/pub/FreeBSD/CTM
on rsync://ftp-master.FreeBSD.org/FreeBSD?
* It seems that it is not safe to use compression (-z). If I try to
rsync ISO-IMAGES directory where any *.iso is missing, it is ok.
However, if *.iso is already there, but with different timestamp,
time rsync -rltvz ends with this error:
receiving file list ... done
4.2-install.iso
deflate on token returned 0 (16384 bytes left)
unexpected EOF in read_timeout
unexpected EOF in read_timeout
real 0m42.533s
user 0m23.780s
sys 0m7.284s
rsync -rltv runs well, if I omit one full copy of *.iso file. Any
other experiences? Any comments? I can not suggest use of compression.
* Many timestamps on ftp-master are very different from timestamps
on usw.freebsd.org and on dead ftp.freebsd.org. Dry run rsync log
is 6900769 bytes with --modify-window=0 and 4623094 bytes with
--modify-window=60 (branches, FreeBSD-CVS and mailing-lists are
skipped).
What are your future plans about file timestamps on ftp-master?
Are you going to sync/change them with {usw,ftp}.freebsd.org or
I can rely on them and they are real "master timestams" for now and
for the future? I'm asking, because if I run full rsync now, it has
to generate big local traffic and I want to avoid these "updates"
as much as possible.
+ It seems to me that combination of ftp and rsync is possible.
Timestamps resolution for ftp is 1 minute HH:MM:00 (dir -T looks
as very non-standard and unreliable feature) and for rsync it is
1 second HH:MM:SS. Transition rsync -> ftp seems to be very
easy and fast if seconds are not taken into account by a mirroring
program. Transition ftp -> rsync would generate just small checksum
communication and non-critical number of one-time file copies with
timestamp fixes. rsync --modify-window=60 would reduce file copying
and timestamping at all.
+ I do not like any automatic deletions, so I have created patched
rsync. If --max-delete (hmm, there is an off by one error in
rsync...) is exceeded (think about -1), it prints deletion
commands only instead of skipping or removing them. After rsync run
I can convert log into a script and delete files by hand.
This looks well.
+ I really do not like how rsync works with file permissions, because
I can not say "use mode 644 for files and use mode 755 for directories".
If I use -p, I have to rely on server side. If I do not use -p, umask
is just mask for permissions from server side, so I can force to
remove some flags (w for others), but I can not add any permissions
(w for user) :-(
--
Rudolf Cejka (cejkar@dcse.fee.vutbr.cz; http://www.fee.vutbr.cz/~cejkar)
Brno University of Technology, Faculty of El. Engineering and Comp. Science
Bozetechova 2, 612 66 Brno, Czech Republic
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hubs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010507120419.A42291>