From owner-freebsd-questions@FreeBSD.ORG  Tue Aug 20 07:34:07 2013
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTP id 67501597;
 Tue, 20 Aug 2013 07:34:07 +0000 (UTC)
 (envelope-from kraduk@gmail.com)
Received: from mail-we0-x236.google.com (mail-we0-x236.google.com
 [IPv6:2a00:1450:400c:c03::236])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id B620F2940;
 Tue, 20 Aug 2013 07:34:06 +0000 (UTC)
Received: by mail-we0-f182.google.com with SMTP id q59so63441wes.13
 for <multiple recipients>; Tue, 20 Aug 2013 00:34:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc:content-type;
 bh=9n5VqNlnJnRoVh34eHISHiJjFjYxpIKirD2r8y4snJY=;
 b=k6BqsNEGTkUoOt2olJfNkynXpsj2wNQhTYvtU4BLak4BPzCusHxb1JAPdMjRzgLztR
 Y3wHFGzbSpmm7oqoqzHGUNwZqFEjOHnGuj38yZzMnpsVcED54BgF//Vmi/ko0oJgZRLf
 Yx/r90yYcH+2QXaQhowmAhw4I+uOv2b1I032Kmq95zguhmxJv3RU/MYAEPfRvlPSlqHl
 8/Ai6OQaVTEZbiCmafW+9AkcdbiP/j97bGHGwmwYw2tdDDdiiR8TrMUiBm7nZgoMFeH8
 N7HRwh0m3hxCqsr9CyA4gdlDQBELAr2AvIKa/+Noz5bX4lJYOFzvY9uyl5pjhhSi7z0w
 ytkQ==
MIME-Version: 1.0
X-Received: by 10.180.84.225 with SMTP id c1mr11151948wiz.53.1376984045001;
 Tue, 20 Aug 2013 00:34:05 -0700 (PDT)
Received: by 10.216.67.196 with HTTP; Tue, 20 Aug 2013 00:34:04 -0700 (PDT)
In-Reply-To: <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com>
References: <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com>
 <20130816064612.GH1190@petole.demisel.net>
 <1376934082.25499.11612497.1C73C726@webmail.messagingengine.com>
 <B629E9F8-C01A-4D09-8054-C63F69846F5C@gmail.com>
 <CALfReyeWxHjmqXhWiK4jbCvh3MktqKqnTBQjYgC0wDTgBcK5jg@mail.gmail.com>
Date: Tue, 20 Aug 2013 08:34:04 +0100
Message-ID: <CALfReydQUEmWdjEmh45ZRit61RJq_00PTtR7ej0LVOvqS=ayNw@mail.gmail.com>
Subject: Re: copying milllions of small files and millions of dirs
From: krad <kraduk@gmail.com>
To: aurfalien <aurfalien@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Content-Filtered-By: Mailman/MimeDel 2.1.14
Cc: FreeBSD Questions <freebsd-questions@freebsd.org>
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-questions>, 
 <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
 <mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Aug 2013 07:34:07 -0000

whops that should have been

ls /tmp/scripts/| while read f
echo sh /tmp/scripts/$f
done | xjobs -j 20


On 20 August 2013 08:32, krad <kraduk@gmail.com> wrote:

> When i migrated a large mailspool in maildir format from the old nfs
> server to the new one in a previous job, I 1st generated a list of the top
> level maildirs. I then generated the rsync commands + plus a few other bits
> and pieces for each maildir to make a single transaction like function. I
> then pumped all this auto generated scripts into xjobs and ran them in
> parallel. This vastly speeded up the process as sequentially running the
> tree was far to slow. THis was for about 15 million maildirs in a hashed
> structure btw so a fair amount of files.
>
>
> eg
>
> find /maildir -type d -maxdepth 4 | while read d
> do
> r=$(($RANDOM*$RANDOM))
> echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r
> echo some other stuff >> /tmp/scripts/$r
> done
>
> ls /tmp/scripts/| while read f
> echo /tmp/scripts/$f
> done | xjobs -j 20
>
>
>
>
>
>
>
>
>
>
> On 19 August 2013 18:52, aurfalien <aurfalien@gmail.com> wrote:
>
>>
>> On Aug 19, 2013, at 10:41 AM, Mark Felder wrote:
>>
>> > On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote:
>> >> On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote:
>> >>> Is there a faster way to copy files over NFS?
>> >>
>> >> I would use find+cpio. This handles hard links, permissions, and in
>> case
>> >> of later runs, will not copy files if they already exist on the
>> >> destination.
>> >>
>> >> # cd /source/dir
>> >> # find . | cpio -pvdm /destination/dir
>> >>
>> >
>> > I always found sysutils/cpdup to be faster than rsync.
>>
>> Ah, bookmarking this one.
>>
>> Many thanks.
>>
>> - aurf
>> _______________________________________________
>> freebsd-questions@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to "
>> freebsd-questions-unsubscribe@freebsd.org"
>>
>
>