Date: Mon, 02 Jun 2003 19:08:25 +0400 From: "Aleksey Kuznetsov" <ahk@spb.edu> To: Michael Conlen <meconlen@obfuscated.net> Cc: freebsd-performance@freebsd.org Subject: Re: copy 150G over 100Mbit Message-ID: <3EDBA0A9.21063.33B9D6B9@localhost> In-Reply-To: <3EDB67B1.2040609@obfuscated.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2 Jun 2003 at 11:05, Michael Conlen wrote:
> 12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour.
12,5MBytes/sec = 750 MBytes/min = 45000 Mbytes/hour
====
>
> Two problems, protocol overhead and disk speed.
>
> How fast can you read off that IDE system? Try tarring the filesystem
> and dumping it to /dev/null. If you can't get 12.5MBytes/sec then
> don't look to the network first.You should consider the size of your
> files. Sure your disks can move data fast, however if you have lots
> and lots of really small files your disks are going to be seeking all
> the time and reading very little of it. If your moving one large file
> there's less seek and more read. Also with lots of small files
> whatever protocol you use will have more overhead to eat more of your
> network bandwith. The difference can be an order of magnitude in disk
> throughput on reads.
>
> Over the network you will end up with all kinds of overhead for
> various protocols. NFS isn't the fastest thing in the world. Rsync has
> it's uses, but I don't think this is one of them.
>
> Consider using cpio over the network. Quick tests show that cpio gets
> better performance than tar by a long shot.
>
> Another consideration might be to move the entire filesystem. If you
> create identical sized partitions on each system, instead of mounting
> them just copy the whole filesystem across using ssh or rsh. I haven't
> tried this on FreeBSD but it shouldn't be to hard to get working. The
> idea being that you read from /dev/ad0s1f on one box and write to
> /dev/ad0s1f on the other. Something like
>
> dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f'
>
> You can try playing with the bs parameter to get optimal read/write
> speeds. A quick test on my system shows that the disk throughput
> increases as the block size increases up to about 8k from
> 4.5MBytes/sec at 512 to 12MBytes/sec at 8k. On a RAID setup it's
> likely to be larger.
>
> You may be able to dump to a larger partition on the destination
> system though the extra space would go unused. I don't have a good way
> to test this here.
>
> --
> Michael Conlen
>
>
> Achim Patzner wrote:
>
> >Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter
> ><damir@voljatel.si>:
> >
> >
> >>I need to copy ~150Gbytes over switched 100Mbit network to new
> >>machine (in max 7-8 hours).
> >>
> >>Source machine (IDE RAID 5) is in production, iostat shows average
> >>transfer of 8-9Mbytes/sec. Destination box is FBC RAID 10.
> >>
> >>
> >
> >Excuse me, but what would you expect using a 100 M*bit*/s network
> >connection?
> >
> >
> >Achim
> >
> >_______________________________________________
> >freebsd-performance@freebsd.org mailing list
> >http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> >To unsubscribe, send any mail to
> >"freebsd-performance-unsubscribe@freebsd.org"
> >
> >
>
>
> _______________________________________________
> freebsd-performance@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to
> "freebsd-performance-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3EDBA0A9.21063.33B9D6B9>