Date: Mon, 02 Jun 2003 11:05:21 -0400 From: Michael Conlen <meconlen@obfuscated.net> To: Achim Patzner <ap@bnc.net> Cc: damir@voljatel.si Subject: Re: copy 150G over 100Mbit Message-ID: <3EDB67B1.2040609@obfuscated.net> References: <BB012C94.66EC%ap@bnc.net>
next in thread | previous in thread | raw e-mail | index | archive | help
12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour. Two problems, protocol overhead and disk speed. How fast can you read off that IDE system? Try tarring the filesystem and dumping it to /dev/null. If you can't get 12.5MBytes/sec then don't look to the network first.You should consider the size of your files. Sure your disks can move data fast, however if you have lots and lots of really small files your disks are going to be seeking all the time and reading very little of it. If your moving one large file there's less seek and more read. Also with lots of small files whatever protocol you use will have more overhead to eat more of your network bandwith. The difference can be an order of magnitude in disk throughput on reads. Over the network you will end up with all kinds of overhead for various protocols. NFS isn't the fastest thing in the world. Rsync has it's uses, but I don't think this is one of them. Consider using cpio over the network. Quick tests show that cpio gets better performance than tar by a long shot. Another consideration might be to move the entire filesystem. If you create identical sized partitions on each system, instead of mounting them just copy the whole filesystem across using ssh or rsh. I haven't tried this on FreeBSD but it shouldn't be to hard to get working. The idea being that you read from /dev/ad0s1f on one box and write to /dev/ad0s1f on the other. Something like dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f' You can try playing with the bs parameter to get optimal read/write speeds. A quick test on my system shows that the disk throughput increases as the block size increases up to about 8k from 4.5MBytes/sec at 512 to 12MBytes/sec at 8k. On a RAID setup it's likely to be larger. You may be able to dump to a larger partition on the destination system though the extra space would go unused. I don't have a good way to test this here. -- Michael Conlen Achim Patzner wrote: >Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter <damir@voljatel.si>: > > >>I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max >>7-8 hours). >> >>Source machine (IDE RAID 5) is in production, iostat shows average transfer of >>8-9Mbytes/sec. Destination box is FBC RAID 10. >> >> > >Excuse me, but what would you expect using a 100 M*bit*/s network >connection? > > >Achim > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3EDB67B1.2040609>