From owner-freebsd-isp@FreeBSD.ORG Tue Jun 24 21:56:36 2003 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8D06937B404 for ; Tue, 24 Jun 2003 21:56:36 -0700 (PDT) Received: from workofstone.net (workofstone.net [208.177.130.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 882F943FBD for ; Tue, 24 Jun 2003 21:56:35 -0700 (PDT) (envelope-from schluntz@workofstone.com) Received: from workofstone.com (belldandy.workofstone.net [172.20.1.9]) by workofstone.net (8.12.9/8.12.9) with ESMTP id h5P4uRbH002908; Tue, 24 Jun 2003 21:56:27 -0700 (PDT) Date: Tue, 24 Jun 2003 21:56:35 -0700 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) To: Tom Samplonius From: "Sean J. Schluntz" In-Reply-To: Message-Id: <657FE40E-A6C9-11D7-B680-000A95672454@workofstone.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) cc: "freebsd-isp@freebsd.org" Subject: Re: What would be the best way to copy lots of files from oneserver to another. X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jun 2003 04:56:37 -0000 On Saturday, June 21, 2003, at 01:08 PM, Tom Samplonius wrote: > > On Sat, 21 Jun 2003, Simon wrote: > >> I recently used rsync to copy ~35gb worth of data, 1,170,168 inodes, >> without any problem over 100mbps interface. Took 3 or so hours. >> You do need lots of RAM, if you swap, you're screwed. RAM >> usage would depend on number of files you need to copy at once. >> >> -Simon > > Yes, rsync uses large amounts of memory when the fileset is large. > Plus, depending on your CPU, you might want to avoid using ssh > encryption > too. > > Usually if it is a LAN to LAN copy, cpdup over NFS will be WAY faster > than rsync, and use little memory. I've used cpdup for moving mail to > a > new server. I synced before the cutover, shutdown the mail server, > synced > one more, and then brought up the mail server on the system. Very > tidy. > even faster than that: cd sourcedir ; find . -print | cpio -o | \ ssh user@host "cd destdir ; cpio -pdmuv" Drop the v to keep from cluttering your screen. You might need to add c to both ends for 'compatibility' if it's different arch or vers of cpio, but I have better luck with it. I have yet to find anything faster over long links, use rsh for local and it's even faster. -Sean