From owner-freebsd-questions@FreeBSD.ORG Thu Mar 9 14:12:44 2006 Return-Path: X-Original-To: questions@freebsd.org Delivered-To: freebsd-questions@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 55E0D16A420 for ; Thu, 9 Mar 2006 14:12:44 +0000 (GMT) (envelope-from xfb52@dial.pipex.com) Received: from asmtp-out2.blueyonder.co.uk (asmtp-out2.blueyonder.co.uk [195.188.213.61]) by mx1.FreeBSD.org (Postfix) with ESMTP id BE2FE43D46 for ; Thu, 9 Mar 2006 14:12:43 +0000 (GMT) (envelope-from xfb52@dial.pipex.com) Received: from [62.31.10.79] (helo=[192.168.0.2]) by asmtp-out2.blueyonder.co.uk with esmtp (Exim 4.52) id 1FHLsd-0002mb-O3; Thu, 09 Mar 2006 14:12:39 +0000 Message-ID: <441037D7.9090107@dial.pipex.com> Date: Thu, 09 Mar 2006 14:12:39 +0000 From: Alex Zbyslaw User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-GB; rv:1.7.12) Gecko/20060305 X-Accept-Language: en MIME-Version: 1.0 To: Dave References: <010e01c642cb$0701ca70$0200a8c0@satellite> <44102F68.8020802@wmptl.com> In-Reply-To: <44102F68.8020802@wmptl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: questions@freebsd.org, Nathan Vidican Subject: Re: sending large amount of data securely X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2006 14:12:44 -0000 Nathan Vidican wrote: > Dave wrote: > >> Hello, >> I've got two freebsd6 boxes on one i have a large collection of >> files, like about 3.5 gb worth, that i want to send securely to the >> other box. I've tried: >> >> scp -24Cpr >> >> to do it, it transfered 1gb of data then quit. I'm suspecting it >> didn't get the sym links if any in this data. Any suggestions >> appreciated. >> Thanks. >> Dave. >> >> > We mirror (nightly) about 300GB worth of data accross one master and > three slave servers (three copies of 300GB of data nightly) using > rsync, we do not use ssh as it's a local-only gigabit ethernet link > between servers, but it's not overly difficult to accomplish in your > case. > > One thing to note; rsync will allow you to keep data synchronized by > only copying the difference from one directory tree to another, so in > our case the actual nightly transfer is only in the neighborhood of > about 30GB, and it takes about 1/2hr to complete to all slaves in > sequence. > > Hope it helps, but my two cents - use rsync for that. > If you do need to use ssh with rsync, then you could try out the latest openssh-portable port which now seems to support the HPN (high performance networking) patches. (There's also a ssh-hpn port, but I'm guessing that will become redundant). HPN improved my data transfer over gigabit by a factor of up to 4 - without them gigabit was actually slower than 100Mbit. The patches also support a "no encryption" option which is useful for internal networks as you get all the authentication without unnecessary overhead in the copying. I've seen reports that HPN patches also improve ssh performance between different OSes, but haven't tried that myself. Why those patches aren't yet part of ssh by default is beyond me. --Alex PS the only drawback of rsync is that it won't mirror "flags" (nodump, schg etc). But then neither would scp :-) PPS Using compression with SSH may actually make things slower depending on your CPU, load etc. You should benchmark with and without if you are considering using it. Over a modem it might be a no-brainer, but anything faster is much less obvious, especially if your data set doesn't compress well, which we have no way of knowing.