From owner-freebsd-questions@FreeBSD.ORG Tue Aug 20 07:32:50 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E13724E8; Tue, 20 Aug 2013 07:32:50 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4E5592927; Tue, 20 Aug 2013 07:32:50 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id hi8so62258wib.3 for ; Tue, 20 Aug 2013 00:32:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=rzrqHsIJRIh1/77gEkxV9DOUM5ZmfGdcDLTKoolHHuA=; b=SSZJT3IhLPCExqC/vrDmbQ9qQxBU5k9vI0A2UjFO3KK0IXfj5i6+VWlq047in+Ulcy 4bZMdKs//V9J7FRpjN+ltMLane1O+RUmCBnJJonNmnoBhspLdn+kuonkjG5rCivVoJ3M 4fOdbAHl4g8/+BhIfkMPsZYsC5lgOITNAmPFeGhvk7+k75Aks+ePRqBpIltb8Cgz5sha TRto1TtUpCOXflShDADO7iV6Np6Q6k+LBQsI+8kyHFK2o2Mg1B6hh29qIv4JQI16Qpv0 kSSVqSvoHRX+hEfLUi9+d/V6dsz/GczoGoerd9gfThsQJaM12dxVqJmoNAMk7MKqeKv9 puig== MIME-Version: 1.0 X-Received: by 10.180.206.42 with SMTP id ll10mr11160108wic.50.1376983968560; Tue, 20 Aug 2013 00:32:48 -0700 (PDT) Received: by 10.216.67.196 with HTTP; Tue, 20 Aug 2013 00:32:48 -0700 (PDT) In-Reply-To: References: <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com> <20130816064612.GH1190@petole.demisel.net> <1376934082.25499.11612497.1C73C726@webmail.messagingengine.com> Date: Tue, 20 Aug 2013 08:32:48 +0100 Message-ID: Subject: Re: copying milllions of small files and millions of dirs From: krad To: aurfalien Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Aug 2013 07:32:50 -0000 When i migrated a large mailspool in maildir format from the old nfs server to the new one in a previous job, I 1st generated a list of the top level maildirs. I then generated the rsync commands + plus a few other bits and pieces for each maildir to make a single transaction like function. I then pumped all this auto generated scripts into xjobs and ran them in parallel. This vastly speeded up the process as sequentially running the tree was far to slow. THis was for about 15 million maildirs in a hashed structure btw so a fair amount of files. eg find /maildir -type d -maxdepth 4 | while read d do r=$(($RANDOM*$RANDOM)) echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r echo some other stuff >> /tmp/scripts/$r done ls /tmp/scripts/| while read f echo /tmp/scripts/$f done | xjobs -j 20 On 19 August 2013 18:52, aurfalien wrote: > > On Aug 19, 2013, at 10:41 AM, Mark Felder wrote: > > > On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: > >> On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: > >>> Is there a faster way to copy files over NFS? > >> > >> I would use find+cpio. This handles hard links, permissions, and in case > >> of later runs, will not copy files if they already exist on the > >> destination. > >> > >> # cd /source/dir > >> # find . | cpio -pvdm /destination/dir > >> > > > > I always found sysutils/cpdup to be faster than rsync. > > Ah, bookmarking this one. > > Many thanks. > > - aurf > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >