From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 11:09:20 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4183E37B404 for ; Mon, 2 Jun 2003 11:09:20 -0700 (PDT) Received: from sabre.velocet.net (sabre.velocet.net [216.138.209.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C94443F75 for ; Mon, 2 Jun 2003 11:09:18 -0700 (PDT) (envelope-from dgilbert@velocet.ca) Received: from trooper.velocet.ca (trooper.velocet.net [216.138.242.2]) by sabre.velocet.net (Postfix) with ESMTP id 03FED1396CF; Mon, 2 Jun 2003 13:25:26 -0400 (EDT) Received: by trooper.velocet.ca (Postfix, from userid 66) id 8DCA374CA7; Mon, 2 Jun 2003 13:25:26 -0400 (EDT) Received: by canoe.velocet.net (Postfix, from userid 101) id 1A04E56762E; Mon, 2 Jun 2003 13:25:24 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16091.34947.952653.880037@canoe.velocet.net> Date: Mon, 2 Jun 2003 13:25:23 -0400 To: damir@voljatel.si In-Reply-To: <200306021623.17927.damir@voljatel.si> References: <200306021623.17927.damir@voljatel.si> X-Mailer: VM 7.14 under 21.1 (patch 14) "Cuyahoga Valley" XEmacs Lucid cc: freebsd-performance@freebsd.org Subject: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 18:09:20 -0000 >>>>> "Damir" == Damir Horvat writes: Damir> Hi! I need to copy ~150Gbytes over switched 100Mbit network to Damir> new machine (in max 7-8 hours). Damir> Source machine (IDE RAID 5) is in production, iostat shows Damir> average transfer of 8-9Mbytes/sec. Destination box is FBC RAID Damir> 10. Damir> I've tryed rsync and cp over NFS. Both came down roughly to Damir> 2Gig's per hour, which is unacceptable. The only option left Damir> (as I see it) is to try with cross-over cable. Damir> I don't care for security as all transfers are in private Damir> network. Damir> Any ideas, how to approach this problem? Here are a few notes. As others have said, read and write speed are big factors. If you already have an 8 to 9 Mb/sec read rate on the old system, you may not be able to achieve another 12Mb/sec to fill the link. Also, many motherboards are going to have an overall limit to performance which is non linear (depends on number of cards and how much bandwidth each one does). NFS and rsync are not your best tools. They have no streaming. You want buffers at every point to eliminate bottlenecks. On one machine you might do: tar -cvf - . | team 1m 16 | nc two 6464 On the other: nc -l 6464 | team 1m 16 | tar -xvf - Someone also mentioned cpio being faster... I don't know if that will be true with enough buffering. Team (a port) is a good friend to have around. Softupdates will likely save your but. Make sure it's on for the target. You might even try the following sequence: touch FOOBAR do the backup wait for final day do the backup with files newer than FOOBAR most utilities will have a -newer switch. Dave. -- ============================================================================ |David Gilbert, Velocet Communications. | Two things can only be | |Mail: dgilbert@velocet.net | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================