From owner-freebsd-hubs Mon May 7 3: 4:41 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from boco.fee.vutbr.cz (boco.fee.vutbr.cz [147.229.9.11]) by hub.freebsd.org (Postfix) with ESMTP id 0C1BD37B422 for ; Mon, 7 May 2001 03:04:34 -0700 (PDT) (envelope-from cejkar@dcse.fee.vutbr.cz) Received: from kazi.dcse.fee.vutbr.cz (kazi.dcse.fee.vutbr.cz [147.229.8.12]) by boco.fee.vutbr.cz (8.11.3/8.11.3) with ESMTP id f47A4Lc81635 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Mon, 7 May 2001 12:04:21 +0200 (CEST) Received: (from cejkar@localhost) by kazi.dcse.fee.vutbr.cz (8.11.2/8.11.2) id f47A4Ki53908; Mon, 7 May 2001 12:04:20 +0200 (CEST) Date: Mon, 7 May 2001 12:04:20 +0200 From: Cejka Rudolf To: Peter Wemm Cc: hubs@FreeBSD.ORG Subject: Re: ftp-master method Message-ID: <20010507120419.A42291@dcse.fee.vutbr.cz> References: <200105041644.f44GiuO54477@vashon.polstra.com> <20010506213812.3BAFE380E@overcee.netplex.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010506213812.3BAFE380E@overcee.netplex.com.au>; from peter@wemm.org on Sun, May 06, 2001 at 02:38:12PM -0700 Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Peter Wemm wrote (2001/05/06): > One gripe I have with rsync itself is that it seems to build its file > lists right from the start and holds them in memory.. It then streams > the updates. This means that rsyncd consumes about 50-60MB of ram on the > server at all times (for an update). It does its delete pass right at then > end, if nothing went wrong. Hmm, it seems that rsync is still a big memory killer :-( I have tried a dry run from ftp-master and rsync took about 96 MB of RAM (46 MB when syncing and the next 50 MB in deletion phase) for full FreeBSD tree. If I exclude branches, FreeBSD-CVS and mailing-lists, rsync takes about 44 MB of RAM (20 MB + 24 MB). I hope that second rsync process with similar memory requirements just shares memory pages with the first process. If I use mirror and algorithm=1, it could not exceed 10 MB of RAM. It would be great to see similar memory usage with rsync too... However, the big advantage of rsync over ftp method is its speed and I really want to switch to rsync or combine it with ftp. * What does mean symbolic link development/CTM/CTM -> usr/home/ftp/pub/FreeBSD/CTM on rsync://ftp-master.FreeBSD.org/FreeBSD? * It seems that it is not safe to use compression (-z). If I try to rsync ISO-IMAGES directory where any *.iso is missing, it is ok. However, if *.iso is already there, but with different timestamp, time rsync -rltvz ends with this error: receiving file list ... done 4.2-install.iso deflate on token returned 0 (16384 bytes left) unexpected EOF in read_timeout unexpected EOF in read_timeout real 0m42.533s user 0m23.780s sys 0m7.284s rsync -rltv runs well, if I omit one full copy of *.iso file. Any other experiences? Any comments? I can not suggest use of compression. * Many timestamps on ftp-master are very different from timestamps on usw.freebsd.org and on dead ftp.freebsd.org. Dry run rsync log is 6900769 bytes with --modify-window=0 and 4623094 bytes with --modify-window=60 (branches, FreeBSD-CVS and mailing-lists are skipped). What are your future plans about file timestamps on ftp-master? Are you going to sync/change them with {usw,ftp}.freebsd.org or I can rely on them and they are real "master timestams" for now and for the future? I'm asking, because if I run full rsync now, it has to generate big local traffic and I want to avoid these "updates" as much as possible. + It seems to me that combination of ftp and rsync is possible. Timestamps resolution for ftp is 1 minute HH:MM:00 (dir -T looks as very non-standard and unreliable feature) and for rsync it is 1 second HH:MM:SS. Transition rsync -> ftp seems to be very easy and fast if seconds are not taken into account by a mirroring program. Transition ftp -> rsync would generate just small checksum communication and non-critical number of one-time file copies with timestamp fixes. rsync --modify-window=60 would reduce file copying and timestamping at all. + I do not like any automatic deletions, so I have created patched rsync. If --max-delete (hmm, there is an off by one error in rsync...) is exceeded (think about -1), it prints deletion commands only instead of skipping or removing them. After rsync run I can convert log into a script and delete files by hand. This looks well. + I really do not like how rsync works with file permissions, because I can not say "use mode 644 for files and use mode 755 for directories". If I use -p, I have to rely on server side. If I do not use -p, umask is just mask for permissions from server side, so I can force to remove some flags (w for others), but I can not add any permissions (w for user) :-( -- Rudolf Cejka (cejkar@dcse.fee.vutbr.cz; http://www.fee.vutbr.cz/~cejkar) Brno University of Technology, Faculty of El. Engineering and Comp. Science Bozetechova 2, 612 66 Brno, Czech Republic To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message