From owner-freebsd-isp@FreeBSD.ORG Sat Jun 21 11:29:27 2003 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5886C37B401 for ; Sat, 21 Jun 2003 11:29:27 -0700 (PDT) Received: from cobra.acceleratedweb.net (cobra-gw.acceleratedweb.net [207.99.79.37]) by mx1.FreeBSD.org (Postfix) with SMTP id 0A54143F3F for ; Sat, 21 Jun 2003 11:29:25 -0700 (PDT) (envelope-from simon@optinet.com) Received: (qmail 89830 invoked by uid 106); 21 Jun 2003 18:30:39 -0000 Received: from 24-90-127-16.nyc.rr.com (HELO win2kpc1) (24.90.127.16) by cobra.acceleratedweb.net with SMTP; 21 Jun 2003 18:30:39 -0000 From: "Simon" To: "Andy Dills" , "Chuck Swiger" Date: Sat, 21 Jun 2003 14:28:43 -0400 Priority: Normal X-Mailer: PMMail 2000 Professional (2.20.2661) For Windows 2000 (5.0.2195;3) In-Reply-To: MIME-Version: 1.0 Message-Id: <20030621182925.0A54143F3F@mx1.FreeBSD.org> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.1 cc: "freebsd-isp@freebsd.org" Subject: Re: What would be the best way to copy lots of files from oneserver to another. X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 18:29:27 -0000 I recently used rsync to copy ~35gb worth of data, 1,170,168 inodes, without any problem over 100mbps interface. Took 3 or so hours. You do need lots of RAM, if you swap, you're screwed. RAM usage would depend on number of files you need to copy at once. -Simon On Sat, 21 Jun 2003 11:26:28 -0400 (EDT), Andy Dills wrote: >On Fri, 20 Jun 2003, Chuck Swiger wrote: > >> Etienne Ledoux wrote: >> > I need to copy lots of directories/files from one server to another. Approx >> > 45Gb . These files are mailboxes stored in maildir format. What would be the >> > best/safest/quickest way to do this ? >> >> Putting the disks onto one machine and copying locally is going to be a win >> considering the amount of data you have, but if you have to copy over the >> network, consider using something like rsync. > >I love rsync and use it for lots of things. > >A one time move of 45 GB is not something I would use rsync for. > >Rsync is good for just what is says it's for: keeping thing in sync. > >Rsync will build metadata information about the ENTIRE data set before >moving a single bit. This is very costly, making rsync's utility plummet >as the data set size increases. You'll find it preferrable to write a >script to traverse and descend directories, spawning numerous rsync >processes for smaller sized directories. > > >Of course, considering that it's a one-time move, I would advise >physically moving the drive, to remove any sort of network involvement. > >I've been in your position before, and I can promise you that even using >NFS and spawning multiple moves (mv /var/mail/a* /mnt &; mv /var/mail/b* >/mnt &; etc.) will quickly outpace rsync. > >The number of concurrent processes involved in the move can really only be >determined by watching the aggregate traffic to your server. You'll see it >level off at a certain point (for me it was X=7), after which additional >processes don't yield any more optimization. > >Andy > >--- >Andy Dills >Xecunet, Inc. >www.xecu.net >301-682-9972 >--- > >_______________________________________________ >freebsd-isp@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-isp >To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" >