Date: Sat, 21 Jun 2003 14:28:43 -0400 From: "Simon" <simon@optinet.com> To: "Andy Dills" <andy@xecu.net>, "Chuck Swiger" <cswiger@mac.com> Cc: "freebsd-isp@freebsd.org" <freebsd-isp@freebsd.org> Subject: Re: What would be the best way to copy lots of files from oneserver to another. Message-ID: <20030621182925.0A54143F3F@mx1.FreeBSD.org> In-Reply-To: <Pine.BSF.4.44.0306211113110.54979-100000@thunder.xecu.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I recently used rsync to copy ~35gb worth of data, 1,170,168 inodes, without any problem over 100mbps interface. Took 3 or so hours. You do need lots of RAM, if you swap, you're screwed. RAM usage would depend on number of files you need to copy at once. -Simon On Sat, 21 Jun 2003 11:26:28 -0400 (EDT), Andy Dills wrote: >On Fri, 20 Jun 2003, Chuck Swiger wrote: > >> Etienne Ledoux wrote: >> > I need to copy lots of directories/files from one server to another. Approx >> > 45Gb . These files are mailboxes stored in maildir format. What would be the >> > best/safest/quickest way to do this ? >> >> Putting the disks onto one machine and copying locally is going to be a win >> considering the amount of data you have, but if you have to copy over the >> network, consider using something like rsync. > >I love rsync and use it for lots of things. > >A one time move of 45 GB is not something I would use rsync for. > >Rsync is good for just what is says it's for: keeping thing in sync. > >Rsync will build metadata information about the ENTIRE data set before >moving a single bit. This is very costly, making rsync's utility plummet >as the data set size increases. You'll find it preferrable to write a >script to traverse and descend directories, spawning numerous rsync >processes for smaller sized directories. > > >Of course, considering that it's a one-time move, I would advise >physically moving the drive, to remove any sort of network involvement. > >I've been in your position before, and I can promise you that even using >NFS and spawning multiple moves (mv /var/mail/a* /mnt &; mv /var/mail/b* >/mnt &; etc.) will quickly outpace rsync. > >The number of concurrent processes involved in the move can really only be >determined by watching the aggregate traffic to your server. You'll see it >level off at a certain point (for me it was X=7), after which additional >processes don't yield any more optimization. > >Andy > >--- >Andy Dills >Xecunet, Inc. >www.xecu.net >301-682-9972 >--- > >_______________________________________________ >freebsd-isp@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-isp >To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030621182925.0A54143F3F>