From owner-freebsd-performance@FreeBSD.ORG Sun Oct 11 21:06:53 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDED6106568F for ; Sun, 11 Oct 2009 21:06:53 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by mx1.freebsd.org (Postfix) with ESMTP id BC2CF8FC19 for ; Sun, 11 Oct 2009 21:06:53 +0000 (UTC) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.14.3/8.14.3) with ESMTP id n9BL6p7G074295; Sun, 11 Oct 2009 17:06:51 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200910112106.n9BL6p7G074295@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 11 Oct 2009 17:07:08 -0400 To: net.help@m2k.com.tw, "freebsd-performance" From: Mike Tancsa In-Reply-To: <1255189612.42461.net.help@m2k.com.tw> References: <1255189612.42461.net.help@m2k.com.tw> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: Subject: Re: Migrate large amount of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 21:06:54 -0000 At 11:46 AM 10/10/2009, Ckcheng wrote: >Hi all, >Currently, I have a directory with over 5M small files (1~32K). Now, >I want to transfer this directory to another machine and found >that it's extremely slow and painful process. I tried the following >method: It might help if you mount all -onoatime as well as bump up vfs.ufs.dirhash_maxmem to 4x the default size ---Mike >1. rsync >2. tar via ssh >3. tar via nc >(all take hours and hours to finish) > >None of them is able to give me a reasonable migration time. So, I'm >here for asking help. Any suggestion is extremely welcomed. Thank you. > >Btw, here is brief information of my server. (both machines are the same) > >OS: FreeBSD 6.4-Stable 64Bit >CPU: 2 x Xeon L5420 2.50GHz >RAM: 2 x 2G ECC DDR2-667 (full buffered) >DISK: Seagate Barracuda ES 16MB (SATA 300) >Network: 1Gbps (Broadcom BCM5708) >Filesystem: UFS2 > >Regards, >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike