From owner-freebsd-performance@FreeBSD.ORG Sun Oct 11 23:18:58 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B048E1065676 for ; Sun, 11 Oct 2009 23:18:58 +0000 (UTC) (envelope-from jguojun@sbcglobal.net) Received: from web82207.mail.mud.yahoo.com (web82207.mail.mud.yahoo.com [209.191.86.102]) by mx1.freebsd.org (Postfix) with SMTP id 7E8A58FC17 for ; Sun, 11 Oct 2009 23:18:58 +0000 (UTC) Received: (qmail 44538 invoked by uid 60001); 11 Oct 2009 22:52:17 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=sbcglobal.net; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=oP1nTVoqYAIXgq4gdY5eabOsgf1Y5PwqpM+d8TDEI2MUApCkaC8dikez3YItOtw3ZQ/Sl/klfdCkjign2l+SqCCtxslK0FFu4vV0fWPv9U3acB9+YZs88w5qIrBli3jRVg7I7U1AnWdz2A2RpcLhnwDyHmOZJDoHOb/LtCHdnds=; Message-ID: <598544.44269.qm@web82207.mail.mud.yahoo.com> X-YMail-OSG: OygjaHwVM1kn6at_QZiYj066zqxRdyYS8gsSAz2R8ecAPVBN1ZxyXluQCxPKBc9.O00K4QNdRxvxdtqF1BWbGkei4cvFoXJzowmPdRA6G2_bYwn6ZyWbGQu3mGkxVI8iDB.McaPDEiSS3Pe1ndq0DkNGo9sT6jxQt4F4CC5GnlJqCottMs8xVJkJo3859DNl7btTiV5YdtzINoVPzafbGwEilhXHEXNVyU6kuVhAIzE.xNckNWwUqQ08nHm7PcGFCkoPkZJAE.BswLF0EBDMSyGRG_9PESsno2t8QAaDlIG9nTm.6l6byGrn948xS9B8WqVhyIDU.BJD.WeCoviC06KOCf9yzCRgzb_chL6Epd.ulB.guOYei4.SX3cDMEGU8cw- Received: from [75.36.161.248] by web82207.mail.mud.yahoo.com via HTTP; Sun, 11 Oct 2009 15:52:17 PDT X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3 Date: Sun, 11 Oct 2009 15:52:17 -0700 (PDT) From: Jin Guojun To: net.help@m2k.com.tw, freebsd-performance , Mike Tancsa In-Reply-To: <200910112106.n9BL6p7G074295@lava.sentex.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Subject: Re: Migrate large amount of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 23:18:58 -0000 The major factor affects the performance the the number of files in "PER" directory. If I got impression correct, there are 5M files in JUST one directory. If this is true, then it is the problem. The best way to avoid performance penalty is to sort different files into differet named directories. Each directory should contain about 1K files or less. Carefully tuned FS may have better performance for a directory conataining a few thousand files, but no more. If files are created by some state machine automatically, and you do not have a good sorting algorithm for directory naming, use Year-Month-week for auto directory naming. --- On Sun, 10/11/09, Mike Tancsa wrote: > From: Mike Tancsa > Subject: Re: Migrate large amount of small files > To: net.help@m2k.com.tw, "freebsd-performance" > Date: Sunday, October 11, 2009, 9:07 PM > At 11:46 AM 10/10/2009, Ckcheng > wrote: > >Hi all, > >Currently, I have a directory with over 5M small files > (1~32K). Now, > >I want to transfer this directory to another machine > and found > >that it's extremely slow and painful process. I tried > the following > >method: > > It might help if you mount all -onoatime as well as bump up > > vfs.ufs.dirhash_maxmem to 4x the default size > > ---Mike > > >1. rsync > >2. tar via ssh > >3. tar via nc > >(all take hours and hours to finish) > > > >None of them is able to give me a reasonable migration > time. So, I'm > >here for asking help. Any suggestion is extremely > welcomed. Thank you. > > > >Btw, here is brief information of my server. (both > machines are the same) > > > >OS: FreeBSD 6.4-Stable 64Bit > >CPU: 2 x Xeon L5420 2.50GHz > >RAM: 2 x 2G ECC DDR2-667 (full buffered) > >DISK: Seagate Barracuda ES 16MB (SATA 300) > >Network: 1Gbps (Broadcom BCM5708) > >Filesystem: UFS2 > > > >Regards, > >_______________________________________________ > >freebsd-performance@freebsd.org > mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -------------------------------------------------------------------- > Mike Tancsa, > > tel +1 519 651 3400 > Sentex Communications, > > mike@sentex.net > Providing Internet since 1994 > www.sentex.net > Cambridge, Ontario Canada > > www.sentex.net/mike > > _______________________________________________ > freebsd-performance@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >