From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 07:23:21 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1ACD037B401 for ; Mon, 2 Jun 2003 07:23:21 -0700 (PDT) Received: from mail.voljatel.si (mail.voljatel.si [217.72.64.15]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33D1043F3F for ; Mon, 2 Jun 2003 07:23:20 -0700 (PDT) (envelope-from damir@voljatel.si) Received: from pxna.hide.voljatel.si (pehta.voljatel.si [217.72.64.8]) by mail.voljatel.si (Postfix) with ESMTP id 2C1B44A330; Mon, 2 Jun 2003 16:23:18 +0200 (CEST) From: Damir Horvat Organization: Voljatel telekomunikacije d.d. To: freebsd-performance@freebsd.org Date: Mon, 2 Jun 2003 16:23:17 +0200 User-Agent: KMail/1.5 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200306021623.17927.damir@voljatel.si> Subject: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: damir@voljatel.si List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 14:23:21 -0000 Hi! I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max 7-8 hours). Source machine (IDE RAID 5) is in production, iostat shows average transfer of 8-9Mbytes/sec. Destination box is FBC RAID 10. I've tryed rsync and cp over NFS. Both came down roughly to 2Gig's per hour, which is unacceptable. The only option left (as I see it) is to try with cross-over cable. I don't care for security as all transfers are in private network. Any ideas, how to approach this problem? Regards, Damir Horvat System Administrator Voljatel Telekomunikacije d.d. Smartinska 106 1000 Ljubljana SI-Slovenija From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 07:32:40 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 705B137B401 for ; Mon, 2 Jun 2003 07:32:40 -0700 (PDT) Received: from da.mailomat.net (mailomat.net [212.185.46.86]) by mx1.FreeBSD.org (Postfix) with ESMTP id DBB9F43FAF for ; Mon, 2 Jun 2003 07:32:34 -0700 (PDT) (envelope-from ap@bnc.net) Received: This line has been intentionally left blank. Received: from colossus.bnc.net ([212.202.5.188]) (user=bnc.mail mech=PLAIN bits=0) by da.mailomat.net (8.12.9/8.12.9) with ESMTP id h52EOs8J074347; Mon, 2 Jun 2003 16:24:56 +0200 (CEST) (envelope-from ap@bnc.net) Received: from [194.39.192.154] (hyperion.wlan.bnc.net [194.39.192.154]) by colossus.bnc.net (8.12.6/8.12.6) with ESMTP id h52EWLZ0047944; Mon, 2 Jun 2003 16:32:21 +0200 (CEST) (envelope-from ap@bnc.net) User-Agent: Microsoft-Entourage/10.1.0.2006 Date: Mon, 02 Jun 2003 16:32:20 +0200 From: Achim Patzner To: , Message-ID: In-Reply-To: <200306021623.17927.damir@voljatel.si> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-Virus-Scanned: by AMaViS perl-11 Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 14:32:40 -0000 Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter : > I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max > 7-8 hours). > > Source machine (IDE RAID 5) is in production, iostat shows average transfer of > 8-9Mbytes/sec. Destination box is FBC RAID 10. Excuse me, but what would you expect using a 100 M*bit*/s network connection? Achim From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 08:05:25 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D343E37B401 for ; Mon, 2 Jun 2003 08:05:25 -0700 (PDT) Received: from svaha.com (svaha.com [64.46.156.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id DC5A143F85 for ; Mon, 2 Jun 2003 08:05:24 -0700 (PDT) (envelope-from meconlen@obfuscated.net) Received: from obfuscated.net ([64.156.25.5]) (AUTH: LOGIN meconlen, TLS: TLSv1/SSLv3,128bits,RC4-MD5) by svaha.com with esmtp; Mon, 02 Jun 2003 11:05:23 -0400 Message-ID: <3EDB67B1.2040609@obfuscated.net> Date: Mon, 02 Jun 2003 11:05:21 -0400 From: Michael Conlen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Achim Patzner References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 15:05:26 -0000 12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour. Two problems, protocol overhead and disk speed. How fast can you read off that IDE system? Try tarring the filesystem and dumping it to /dev/null. If you can't get 12.5MBytes/sec then don't look to the network first.You should consider the size of your files. Sure your disks can move data fast, however if you have lots and lots of really small files your disks are going to be seeking all the time and reading very little of it. If your moving one large file there's less seek and more read. Also with lots of small files whatever protocol you use will have more overhead to eat more of your network bandwith. The difference can be an order of magnitude in disk throughput on reads. Over the network you will end up with all kinds of overhead for various protocols. NFS isn't the fastest thing in the world. Rsync has it's uses, but I don't think this is one of them. Consider using cpio over the network. Quick tests show that cpio gets better performance than tar by a long shot. Another consideration might be to move the entire filesystem. If you create identical sized partitions on each system, instead of mounting them just copy the whole filesystem across using ssh or rsh. I haven't tried this on FreeBSD but it shouldn't be to hard to get working. The idea being that you read from /dev/ad0s1f on one box and write to /dev/ad0s1f on the other. Something like dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f' You can try playing with the bs parameter to get optimal read/write speeds. A quick test on my system shows that the disk throughput increases as the block size increases up to about 8k from 4.5MBytes/sec at 512 to 12MBytes/sec at 8k. On a RAID setup it's likely to be larger. You may be able to dump to a larger partition on the destination system though the extra space would go unused. I don't have a good way to test this here. -- Michael Conlen Achim Patzner wrote: >Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter : > > >>I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max >>7-8 hours). >> >>Source machine (IDE RAID 5) is in production, iostat shows average transfer of >>8-9Mbytes/sec. Destination box is FBC RAID 10. >> >> > >Excuse me, but what would you expect using a 100 M*bit*/s network >connection? > > >Achim > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 08:08:40 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 160E237B401 for ; Mon, 2 Jun 2003 08:08:40 -0700 (PDT) Received: from wg.pu.ru (wg.pu.ru [193.124.85.219]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8862F43F85 for ; Mon, 2 Jun 2003 08:08:38 -0700 (PDT) (envelope-from ahk@spb.edu) Received: from ahk-work (ank.pu.ru [194.58.104.203]) by wg.pu.ru (8.9.1a/8.9.1) with ESMTP id PAA10784; Mon, 2 Jun 2003 15:08:33 GMT From: "Aleksey Kuznetsov" Organization: VTC SPbU To: Michael Conlen Date: Mon, 02 Jun 2003 19:08:25 +0400 MIME-Version: 1.0 Message-ID: <3EDBA0A9.21063.33B9D6B9@localhost> Priority: normal In-reply-to: <3EDB67B1.2040609@obfuscated.net> X-mailer: Pegasus Mail for Windows (v4.02a) Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Content-description: Mail message body Content-Transfer-Encoding: 7BIT cc: freebsd-performance@freebsd.org Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: ahk@spb.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 15:08:40 -0000 On 2 Jun 2003 at 11:05, Michael Conlen wrote: > 12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour. 12,5MBytes/sec = 750 MBytes/min = 45000 Mbytes/hour ==== > > Two problems, protocol overhead and disk speed. > > How fast can you read off that IDE system? Try tarring the filesystem > and dumping it to /dev/null. If you can't get 12.5MBytes/sec then > don't look to the network first.You should consider the size of your > files. Sure your disks can move data fast, however if you have lots > and lots of really small files your disks are going to be seeking all > the time and reading very little of it. If your moving one large file > there's less seek and more read. Also with lots of small files > whatever protocol you use will have more overhead to eat more of your > network bandwith. The difference can be an order of magnitude in disk > throughput on reads. > > Over the network you will end up with all kinds of overhead for > various protocols. NFS isn't the fastest thing in the world. Rsync has > it's uses, but I don't think this is one of them. > > Consider using cpio over the network. Quick tests show that cpio gets > better performance than tar by a long shot. > > Another consideration might be to move the entire filesystem. If you > create identical sized partitions on each system, instead of mounting > them just copy the whole filesystem across using ssh or rsh. I haven't > tried this on FreeBSD but it shouldn't be to hard to get working. The > idea being that you read from /dev/ad0s1f on one box and write to > /dev/ad0s1f on the other. Something like > > dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f' > > You can try playing with the bs parameter to get optimal read/write > speeds. A quick test on my system shows that the disk throughput > increases as the block size increases up to about 8k from > 4.5MBytes/sec at 512 to 12MBytes/sec at 8k. On a RAID setup it's > likely to be larger. > > You may be able to dump to a larger partition on the destination > system though the extra space would go unused. I don't have a good way > to test this here. > > -- > Michael Conlen > > > Achim Patzner wrote: > > >Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter > >: > > > > > >>I need to copy ~150Gbytes over switched 100Mbit network to new > >>machine (in max 7-8 hours). > >> > >>Source machine (IDE RAID 5) is in production, iostat shows average > >>transfer of 8-9Mbytes/sec. Destination box is FBC RAID 10. > >> > >> > > > >Excuse me, but what would you expect using a 100 M*bit*/s network > >connection? > > > > > >Achim > > > >_______________________________________________ > >freebsd-performance@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >To unsubscribe, send any mail to > >"freebsd-performance-unsubscribe@freebsd.org" > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 08:13:56 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 330C437B404 for ; Mon, 2 Jun 2003 08:13:54 -0700 (PDT) Received: from svaha.com (svaha.com [64.46.156.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A10443F75 for ; Mon, 2 Jun 2003 08:13:53 -0700 (PDT) (envelope-from meconlen@obfuscated.net) Received: from obfuscated.net ([64.156.25.5]) (AUTH: LOGIN meconlen, TLS: TLSv1/SSLv3,128bits,RC4-MD5) by svaha.com with esmtp; Mon, 02 Jun 2003 11:13:52 -0400 Message-ID: <3EDB69AF.2030104@obfuscated.net> Date: Mon, 02 Jun 2003 11:13:51 -0400 From: Michael Conlen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02 X-Accept-Language: en-us, en MIME-Version: 1.0 To: ahk@spb.edu References: <3EDBA0A9.21063.33B9D6B9@localhost> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 15:13:56 -0000 Thanks, All the math in the world is useless with bad typing. -- Michael Conlen Aleksey Kuznetsov wrote: >On 2 Jun 2003 at 11:05, Michael Conlen wrote: > > > >>12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour. >> >> > >12,5MBytes/sec = 750 MBytes/min = 45000 Mbytes/hour > > > From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 08:32:55 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 42CB537B404 for ; Mon, 2 Jun 2003 08:32:55 -0700 (PDT) Received: from mail.drunkencomputing.de (ratz.drunkencomputing.de [195.244.235.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id C87AB43F75 for ; Mon, 2 Jun 2003 08:32:53 -0700 (PDT) (envelope-from hscholz@raisdorf.net) Received: from localhost (localhost [127.0.0.1]) by mail.drunkencomputing.de (Postfix) with ESMTP id CDE108AF67 for ; Mon, 2 Jun 2003 17:33:51 +0200 (CEST) Received: from mail.drunkencomputing.de ([127.0.0.1])port 10024) with ESMTP id 41661-01 for ; Mon, 2 Jun 2003 17:33:50 +0200 (CEST) Received: from goanna.lan.raisdorf.net (pD9E3CAE0.dip.t-dialin.net [217.227.202.224]) by mail.drunkencomputing.de (Postfix) with ESMTP id 32DA28AF4D for ; Mon, 2 Jun 2003 17:33:49 +0200 (CEST) Date: Mon, 2 Jun 2003 17:32:42 +0200 From: Hendrik Scholz To: freebsd-performance@freebsd.org Message-Id: <20030602173242.716c89d1.hscholz@raisdorf.net> In-Reply-To: <200306021623.17927.damir@voljatel.si> References: <200306021623.17927.damir@voljatel.si> X-Mailer: Sylpheed version 0.8.10claws (GTK+ 1.2.10; i386-portbld-freebsd5.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at drunkencomputing.de Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 15:32:55 -0000 Hi! On Mon, 2 Jun 2003 16:23:17 +0200 Damir Horvat wrote: > I've tryed rsync and cp over NFS. Both came down roughly to 2Gig's per > hour, which is unacceptable. The only option left (as I see it) is to > try with cross-over cable. 'systat -vmstat 1' should show the bottleneck as long as it isn't the network itself. What kind of NICs do you have? Try enabling IRQ mitigation on them or other features if supported (ifconfig fxp0 link0 turns on IRQ mitigation on Intel cards). IRQ moderation/mitigation lowers the number of interrupts generated by the network card by bundling (in most cases up to 6 depending on the FIFO buffer size) interrupts generated by the card. Try 'bing' or another network performance measurement tool to determine the max. network speed possible without paying attention to your harddisks. Since you don't care about security you might want to try vsftpd from the ports. vsftpd supports sendfile() thus eliminating copying data between memory regions if you card supports the so called zero-copy features (i.e. can read packet header and body from different buffers). >From my point of view vsftpd is the fastest ftpd available. Hendrik P.S. I'm just gathering some useful information for a FreeBSD network performance tuning article. -- Hendrik Scholz - - http://raisdorf.net/ Forcast for tonight: Dark. From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 09:00:06 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 049E237B401 for ; Mon, 2 Jun 2003 09:00:06 -0700 (PDT) Received: from puffin.mail.pas.earthlink.net (puffin.mail.pas.earthlink.net [207.217.120.139]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7CA3143F75 for ; Mon, 2 Jun 2003 09:00:03 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38ldshk.dialup.mindspring.com ([209.86.242.52] helo=mindspring.com) by puffin.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Mrj6-0003Yy-00; Mon, 02 Jun 2003 09:00:00 -0700 Message-ID: <3EDB7425.95472506@mindspring.com> Date: Mon, 02 Jun 2003 08:58:29 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: damir@voljatel.si References: <200306021623.17927.damir@voljatel.si> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a491185bfa2acea75165bfa6dfb747899f666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 16:00:06 -0000 Damir Horvat wrote: > I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max > 7-8 hours). The best you can expect is ((150 * 8 / 100) * 1024) / 60 = ~3.4 hours. That assumes that your transfer rate off/on the disks on each end can keep up with 100Mbit... which they can't. > I've tryed rsync and cp over NFS. Both came down roughly to 2Gig's per hour, > which is unacceptable. The only option left (as I see it) is to try with > cross-over cable. Your disk I/O will be your limiting factor, not your network speed. -- Terry From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 11:09:20 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4183E37B404 for ; Mon, 2 Jun 2003 11:09:20 -0700 (PDT) Received: from sabre.velocet.net (sabre.velocet.net [216.138.209.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C94443F75 for ; Mon, 2 Jun 2003 11:09:18 -0700 (PDT) (envelope-from dgilbert@velocet.ca) Received: from trooper.velocet.ca (trooper.velocet.net [216.138.242.2]) by sabre.velocet.net (Postfix) with ESMTP id 03FED1396CF; Mon, 2 Jun 2003 13:25:26 -0400 (EDT) Received: by trooper.velocet.ca (Postfix, from userid 66) id 8DCA374CA7; Mon, 2 Jun 2003 13:25:26 -0400 (EDT) Received: by canoe.velocet.net (Postfix, from userid 101) id 1A04E56762E; Mon, 2 Jun 2003 13:25:24 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16091.34947.952653.880037@canoe.velocet.net> Date: Mon, 2 Jun 2003 13:25:23 -0400 To: damir@voljatel.si In-Reply-To: <200306021623.17927.damir@voljatel.si> References: <200306021623.17927.damir@voljatel.si> X-Mailer: VM 7.14 under 21.1 (patch 14) "Cuyahoga Valley" XEmacs Lucid cc: freebsd-performance@freebsd.org Subject: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 18:09:20 -0000 >>>>> "Damir" == Damir Horvat writes: Damir> Hi! I need to copy ~150Gbytes over switched 100Mbit network to Damir> new machine (in max 7-8 hours). Damir> Source machine (IDE RAID 5) is in production, iostat shows Damir> average transfer of 8-9Mbytes/sec. Destination box is FBC RAID Damir> 10. Damir> I've tryed rsync and cp over NFS. Both came down roughly to Damir> 2Gig's per hour, which is unacceptable. The only option left Damir> (as I see it) is to try with cross-over cable. Damir> I don't care for security as all transfers are in private Damir> network. Damir> Any ideas, how to approach this problem? Here are a few notes. As others have said, read and write speed are big factors. If you already have an 8 to 9 Mb/sec read rate on the old system, you may not be able to achieve another 12Mb/sec to fill the link. Also, many motherboards are going to have an overall limit to performance which is non linear (depends on number of cards and how much bandwidth each one does). NFS and rsync are not your best tools. They have no streaming. You want buffers at every point to eliminate bottlenecks. On one machine you might do: tar -cvf - . | team 1m 16 | nc two 6464 On the other: nc -l 6464 | team 1m 16 | tar -xvf - Someone also mentioned cpio being faster... I don't know if that will be true with enough buffering. Team (a port) is a good friend to have around. Softupdates will likely save your but. Make sure it's on for the target. You might even try the following sequence: touch FOOBAR do the backup wait for final day do the backup with files newer than FOOBAR most utilities will have a -newer switch. Dave. -- ============================================================================ |David Gilbert, Velocet Communications. | Two things can only be | |Mail: dgilbert@velocet.net | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 12:45:59 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 26BA037B401 for ; Mon, 2 Jun 2003 12:45:59 -0700 (PDT) Received: from dragon.nuxi.com (trang.nuxi.com [66.93.134.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E41A43FAF for ; Mon, 2 Jun 2003 12:45:58 -0700 (PDT) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (obrien@localhost [127.0.0.1]) by dragon.nuxi.com (8.12.9/8.12.9) with ESMTP id h52JjgVm014052; Mon, 2 Jun 2003 12:45:46 -0700 (PDT) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.12.9/8.12.9/Submit) id h52JjcP7014051; Mon, 2 Jun 2003 12:45:38 -0700 (PDT) Date: Mon, 2 Jun 2003 12:45:38 -0700 From: "David O'Brien" To: David Gilbert Message-ID: <20030602194538.GA13968@dragon.nuxi.com> References: <200306021623.17927.damir@voljatel.si> <16091.34947.952653.880037@canoe.velocet.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16091.34947.952653.880037@canoe.velocet.net> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 5.1-BETA Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 X-Mailman-Approved-At: Mon, 02 Jun 2003 12:50:06 -0700 cc: freebsd-performance@freebsd.org cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: freebsd-performance@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 19:45:59 -0000 On Mon, Jun 02, 2003 at 01:25:23PM -0400, David Gilbert wrote: > NFS and rsync are not your best tools. They have no streaming. You > want buffers at every point to eliminate bottlenecks. /usr/ports/archivers/star is a tar that claims to do streaming. From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 13:34:55 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4FE4C37B401 for ; Mon, 2 Jun 2003 13:34:55 -0700 (PDT) Received: from svaha.com (svaha.com [64.46.156.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 95A4F43F93 for ; Mon, 2 Jun 2003 13:34:52 -0700 (PDT) (envelope-from meconlen@obfuscated.net) Received: from obfuscated.net ([64.156.25.5]) (AUTH: LOGIN meconlen, TLS: TLSv1/SSLv3,128bits,RC4-MD5) by svaha.com with esmtp; Mon, 02 Jun 2003 16:34:51 -0400 Message-ID: <3EDBB4E9.6060004@obfuscated.net> Date: Mon, 02 Jun 2003 16:34:49 -0400 From: Michael Conlen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02 X-Accept-Language: en-us, en MIME-Version: 1.0 References: <200306021623.17927.damir@voljatel.si> <16091.34947.952653.880037@canoe.velocet.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 20:34:55 -0000 I don't know why it just hit me, but turn off atime updates on the filesystem your reading from. This won't likely solve the problem, but it should help reduce your per file time. -- Michael Conlen From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 14:35:54 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 57CD837B401 for ; Mon, 2 Jun 2003 14:35:54 -0700 (PDT) Received: from sabre.velocet.net (sabre.velocet.net [216.138.209.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id C1AC843F3F for ; Mon, 2 Jun 2003 14:35:53 -0700 (PDT) (envelope-from dgilbert@velocet.ca) Received: from trooper.velocet.ca (trooper.velocet.net [216.138.242.2]) by sabre.velocet.net (Postfix) with ESMTP id 0CB441380DC; Mon, 2 Jun 2003 17:35:33 -0400 (EDT) Received: by trooper.velocet.ca (Postfix, from userid 66) id 782CC74CA7; Mon, 2 Jun 2003 17:35:29 -0400 (EDT) Received: by canoe.velocet.net (Postfix, from userid 101) id CF42456762E; Mon, 2 Jun 2003 17:35:26 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16091.49950.769561.422125@canoe.velocet.net> Date: Mon, 2 Jun 2003 17:35:26 -0400 To: Michael Conlen In-Reply-To: <3EDBB4E9.6060004@obfuscated.net> References: <200306021623.17927.damir@voljatel.si> <16091.34947.952653.880037@canoe.velocet.net> <3EDBB4E9.6060004@obfuscated.net> X-Mailer: VM 7.14 under 21.1 (patch 14) "Cuyahoga Valley" XEmacs Lucid cc: freebsd-performance@freebsd.org cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2003 21:35:54 -0000 >>>>> "Michael" == Michael Conlen writes: Michael> I don't know why it just hit me, but turn off atime updates Michael> on the filesystem your reading from. This won't likely solve Michael> the problem, but it should help reduce your per file time. The read side of the equasion is not likely to be the bottleneck. This doesn't do any harm, but I don't think it will do much good. Dave. -- ============================================================================ |David Gilbert, Velocet Communications. | Two things can only be | |Mail: dgilbert@velocet.net | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 2 17:27:52 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C10937B401 for ; Mon, 2 Jun 2003 17:27:52 -0700 (PDT) Received: from skynet.stack.nl (skynet.stack.nl [131.155.140.225]) by mx1.FreeBSD.org (Postfix) with ESMTP id 910F943F93 for ; Mon, 2 Jun 2003 17:27:51 -0700 (PDT) (envelope-from marcolz@stack.nl) Received: by skynet.stack.nl (Postfix, from userid 65534) id 37E293E3F; Tue, 3 Jun 2003 02:28:18 +0200 (CEST) Received: from turtle.stack.nl (turtle.stack.nl [2001:610:1108:5010:2e0:81ff:fe22:51d8]) by skynet.stack.nl (Postfix) with ESMTP id 2D3063E2D; Tue, 3 Jun 2003 02:28:14 +0200 (CEST) Received: by turtle.stack.nl (Postfix, from userid 333) id 59A241CC2E; Tue, 3 Jun 2003 02:27:46 +0200 (CEST) Date: Tue, 3 Jun 2003 02:27:46 +0200 From: Marc Olzheim To: freebsd-performance@freebsd.org Message-ID: <20030603002746.GA26241@stack.nl> References: <200306021623.17927.damir@voljatel.si> <16091.34947.952653.880037@canoe.velocet.net> <20030602194538.GA13968@dragon.nuxi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030602194538.GA13968@dragon.nuxi.com> X-Operating-System: FreeBSD turtle.stack.nl 5.1-BETA FreeBSD 5.1-BETA X-URL: http://www.stack.nl/~marcolz/ User-Agent: Mutt/1.5.4i X-Spam-Status: No, hits=-32.5 required=5.0 tests=EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, REPLY_WITH_QUOTES,USER_AGENT_MUTT version=2.50 X-Spam-Level: X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-Mailman-Approved-At: Mon, 02 Jun 2003 21:03:34 -0700 cc: David Gilbert cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2003 00:27:52 -0000 On Mon, Jun 02, 2003 at 12:45:38PM -0700, David O'Brien wrote: > /usr/ports/archivers/star is a tar that claims to do streaming. Besides from being the only tar that can archive files of 8GB+ ... star is indeed the fastest unbuffered streaming tar, however, gnutar with some kind of buffering program is not that much slower... Zlo From owner-freebsd-performance@FreeBSD.ORG Tue Jun 3 03:08:19 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CF2737B401 for ; Tue, 3 Jun 2003 03:08:19 -0700 (PDT) Received: from mail.voljatel.si (mail.voljatel.si [217.72.64.15]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F12943F75 for ; Tue, 3 Jun 2003 03:08:18 -0700 (PDT) (envelope-from damir@voljatel.si) Received: from pxna.hide.voljatel.si (pehta.voljatel.si [217.72.64.8]) by mail.voljatel.si (Postfix) with ESMTP id 4CA144A3EE for ; Tue, 3 Jun 2003 12:08:16 +0200 (CEST) From: Damir Horvat Organization: Voljatel telekomunikacije d.d. To: freebsd-performance@freebsd.org Date: Tue, 3 Jun 2003 12:08:16 +0200 User-Agent: KMail/1.5 References: <200306021623.17927.damir@voljatel.si> In-Reply-To: <200306021623.17927.damir@voljatel.si> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200306031208.16021.damir@voljatel.si> Subject: Re: copy 150G over 100Mbit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: damir@voljatel.si List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2003 10:08:19 -0000 Hi again! Thanks for numerous answers. Interesting, how many solutions are there... Despite being almost the slowest one, we choose NFS mount, as there must be some pre and post data processing. For example, I can't manipulate stream in order to use it for per-user basis hashed $HOME creation, which is one of main priorities. Thanks again. -- Damir Horvat System Administrator Voljatel Telekomunikacije d.d. Smartinska 106 1000 Ljubljana SI-Slovenija