From owner-freebsd-questions@FreeBSD.ORG Thu Aug 15 18:37:07 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 390C02CB for ; Thu, 15 Aug 2013 18:37:07 +0000 (UTC) (envelope-from aurfalien@gmail.com) Received: from mail-pb0-x22e.google.com (mail-pb0-x22e.google.com [IPv6:2607:f8b0:400e:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 10C5F21DC for ; Thu, 15 Aug 2013 18:37:07 +0000 (UTC) Received: by mail-pb0-f46.google.com with SMTP id rq2so1073485pbb.5 for ; Thu, 15 Aug 2013 11:37:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=HyYKJMEkVxmsQXH9HWqODH66XEinG1zlasrVwnGk0og=; b=NmGjppJeIWS4NaCW4Aot3uBhrHRZpsxxMj+zQtcGW/+dxpidZO5cB/ama/9sEV3+/2 s3ldf6n1d+96enliLoFA5yA2mjgQsGhnLSW8ihL/w/RB7yk/RaPeRj2kmwxvGfyLfKqc sFM/cnoGpW8/O/ZgOtXoREzAi7X/80hluZrSCDcEqsrH4Y6NKRDDfh0JnDjIxl+hvfd4 yQ6XvqcjKWvHYy8tPGH/fxFyfWe7pksDwFO1t5UtLIPHyvm+uf/wqqqzS3IuUElCGWrC 9bSDEtuvBnDtnfE3s+ymK5S4w39a3YeQetaJYEVhacKbYsByY3RtUxvLPNXDbNrv1tLb h8eA== X-Received: by 10.66.120.145 with SMTP id lc17mr3024917pab.182.1376591826761; Thu, 15 Aug 2013 11:37:06 -0700 (PDT) Received: from briankrusicw.logan.tv ([64.17.255.138]) by mx.google.com with ESMTPSA id xe9sm1484163pab.0.2013.08.15.11.37.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 Aug 2013 11:37:05 -0700 (PDT) Subject: Re: copying milllions of small files and millions of dirs Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: aurfalien In-Reply-To: Date: Thu, 15 Aug 2013 11:37:02 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <6483A298-6216-4306-913C-B3E0F4A3BC8D@gmail.com> References: <7E7AEB5A-7102-424E-8B1E-A33E0A2C8B2C@gmail.com> To: Charles Swiger X-Mailer: Apple Mail (2.1085) Cc: FreeBSD Questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Aug 2013 18:37:07 -0000 On Aug 15, 2013, at 11:26 AM, Charles Swiger wrote: > On Aug 15, 2013, at 11:13 AM, aurfalien wrote: >> Is there a faster way to copy files over NFS? >=20 > Probably. Ok, thanks for the specifics. >> Currently breaking up a simple rsync over 7 or so scripts which = copies 22 dirs having ~500,000 dirs or files each. >=20 > There's a maximum useful concurrency which depends on how many disk = spindles and what flavor of RAID is in use; exceeding it will result in = thrashing the disks and heavily reducing throughput due to competing I/O = requests. Try measuring aggregate performance when running fewer rsyncs = at once and see whether it improves. Its 35 disks broken into 7 striped RaidZ groups with an SLC based ZIL = and no atime, the server it self has 128GB ECC RAM. I didn't have time = to tune or really learn ZFS but at this point its only backing up the = data for emergency purposes. > Of course, putting half a million files into a single directory level = is also a bad idea, even with dirhash support. You'd do better to break = them up into subdirs containing fewer than ~10K files apiece. I can't, thats our job structure obviously developed by scrip kiddies = and not systems ppl, but I digress. >> Obviously reading all the meta data is a PITA. >=20 > Yes. >=20 >> Doin 10Gb/jumbos but in this case it don't make much of a hoot of a = diff. >=20 > Yeah, probably not-- you're almost certainly I/O bound, not network = bound. Actually it was network bound via 1 rsync process which is why I broke = up 154 dirs into 7 batches of 22 each. I'll have to acquaint myself with ZFS centric tools to help me determine = whats going on. But=20