Date: Tue, 20 Aug 2013 08:32:48 +0100 From: krad <kraduk@gmail.com> To: aurfalien <aurfalien@gmail.com> Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: copying milllions of small files and millions of dirs Message-ID: <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com> In-Reply-To: <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com> References: <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com> <20130816064612.GH1190@petole.demisel.net> <1376934082.25499.11612497.1C73C726@webmail.messagingengine.com> <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
When i migrated a large mailspool in maildir format from the old nfs server to the new one in a previous job, I 1st generated a list of the top level maildirs. I then generated the rsync commands + plus a few other bits and pieces for each maildir to make a single transaction like function. I then pumped all this auto generated scripts into xjobs and ran them in parallel. This vastly speeded up the process as sequentially running the tree was far to slow. THis was for about 15 million maildirs in a hashed structure btw so a fair amount of files. eg find /maildir -type d -maxdepth 4 | while read d do r=$(($RANDOM*$RANDOM)) echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r echo some other stuff >> /tmp/scripts/$r done ls /tmp/scripts/| while read f echo /tmp/scripts/$f done | xjobs -j 20 On 19 August 2013 18:52, aurfalien <aurfalien@gmail.com> wrote: > > On Aug 19, 2013, at 10:41 AM, Mark Felder wrote: > > > On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: > >> On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: > >>> Is there a faster way to copy files over NFS? > >> > >> I would use find+cpio. This handles hard links, permissions, and in case > >> of later runs, will not copy files if they already exist on the > >> destination. > >> > >> # cd /source/dir > >> # find . | cpio -pvdm /destination/dir > >> > > > > I always found sysutils/cpdup to be faster than rsync. > > Ah, bookmarking this one. > > Many thanks. > > - aurf > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg>