From owner-freebsd-hubs Sun May 6 19:31:58 2001 Delivered-To: freebsd-hubs@freebsd.org Received: from wall.polstra.com (rtrwan160.accessone.com [206.213.115.74]) by hub.freebsd.org (Postfix) with ESMTP id 3961237B422 for ; Sun, 6 May 2001 19:31:55 -0700 (PDT) (envelope-from jdp@wall.polstra.com) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.1) with ESMTP id f472Vs045726; Sun, 6 May 2001 19:31:54 -0700 (PDT) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.3/8.11.0) id f472Vmq62912; Sun, 6 May 2001 19:31:48 -0700 (PDT) (envelope-from jdp) Date: Sun, 6 May 2001 19:31:48 -0700 (PDT) Message-Id: <200105070231.f472Vmq62912@vashon.polstra.com> To: peter@wemm.org From: John Polstra Cc: hubs@freebsd.org Subject: Re: ftp-master method In-Reply-To: <20010506213812.3BAFE380E@overcee.netplex.com.au> References: <20010506213812.3BAFE380E@overcee.netplex.com.au> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-hubs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In article <20010506213812.3BAFE380E@overcee.netplex.com.au>, Peter Wemm wrote: > > One gripe I have with rsync itself is that it seems to build its file > lists right from the start and holds them in memory.. It then streams > the updates. This means that rsyncd consumes about 50-60MB of ram on the > server at all times (for an update). It does its delete pass right at then > end, if nothing went wrong. > > Does cvsupd do better? (ie: transfer the file list incrementally?) Yes, it does. When doing rsync-style updates it still uses the streaming model shown in the diagram at http://www.polstra.com/projects/freeware/CVSup/howsofast.html Updates for the first files will be coming in while the file list is still streaming out in the other direction. > One thing does bother me though, is that cvsup tends to ignore files > that didn't exist. Having cvsup 'take over' maintainence of an > existing (out of date) tree seems likely to leave behind lots of > leftovers. You're right. I hadn't thought of that problem with respect to the FTP tree. We certainly don't want to make people repopulate their zillion-GB FTP trees. > You have script to clean this up, right? Yes, there is a script "cvsupchk" which does exactly that in the contrib subdirectory of the CVSup distribution (thanks to Joseph Koshy). > Or does cvsup have an even stronger "mirror" mode that will get all > this stuff? Not currently. But this makes it a higher priority. It's not that hard in principle. Hmm ... > The archive has got 5 or 6 checked out trees in it, including > several source checkouts, mailing list archives and the ncvs tree. > cvsupd can use the checkouts.cvs file to load the metadata for the > ,v files quickly, right? (pub/FreeBSD/development/FreeBSD-CVS/*, > including a sup/ dir). Yes, definitely. It will use its RCS file algorithm on RCS files and rsync on other files. John -- John Polstra jdp@polstra.com John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hubs" in the body of the message