From owner-freebsd-hackers@FreeBSD.ORG Mon Jul 10 21:38:54 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92DF316A4E0 for ; Mon, 10 Jul 2006 21:38:54 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id E207443D49 for ; Mon, 10 Jul 2006 21:38:53 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (yjwwo46bklxrhv61@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k6ALcgW1041478; Mon, 10 Jul 2006 14:38:42 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k6ALcfTu041477; Mon, 10 Jul 2006 14:38:41 -0700 (PDT) (envelope-from jmg) Date: Mon, 10 Jul 2006 14:38:41 -0700 From: John-Mark Gurney To: Brad Falzon Message-ID: <20060710213841.GP734@funkthat.com> Mail-Followup-To: Brad Falzon , freebsd-hackers@freebsd.org References: <44B1D0F6.1050809@adam.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44B1D0F6.1050809@adam.com.au> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-hackers@freebsd.org Subject: Re: File Mirror Software X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jul 2006 21:38:54 -0000 Brad Falzon wrote this message on Mon, Jul 10, 2006 at 13:30 +0930: > Currently, we're building our replacement file mirror box. (Its a beefy > 5TB Server) Anyway, currently, we use the application 'mirror' to handle > ftp mirrors and our own rsynctool which connects to a database for our > rsync mirrors. However, this is beginning to be a pain, ie no reports > are generated, hard to maintain etc. > > I was wondering if anybody has similar experiences and can recommend any > software that could handle ftp and rsync mirrors better than our home > grown scripts. > > I've spent along time researching this, but could never get a good > enough google query. > > Simba seems to have much potential, however, being version 0.8 has more > than its fair share of bugs at the moment. And the only other ftp mirror > utility (weex) can only handle ftp mirrors, not rsync. > > Does anyone have any recommended ports that we should be looking into > before making our final choice. If you're looking for software to do mirroring, I'd recommend you look at cvsup... Yes, this software is a general mirroring tool, and a few orders of a magnitude faster than rsync... It does have issues with really large files (like 2gb+) as the 10/15 minute timeout can be shorter than the time taken to generate a checksum for the file... Though w/ faster cpu's these days, it's probably no longer the case.. A number of years back, we used it to mirror 140GB of data between sites.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."