From owner-freebsd-questions@FreeBSD.ORG Thu Sep 30 20:59:02 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8725516A4D0 for ; Thu, 30 Sep 2004 20:59:02 +0000 (GMT) Received: from o2.hostbaby.com (o2.hostbaby.com [208.187.29.121]) by mx1.FreeBSD.org (Postfix) with SMTP id 4865343D1D for ; Thu, 30 Sep 2004 20:59:02 +0000 (GMT) (envelope-from ceo@l-i-e.com) Received: (qmail 76712 invoked by uid 1001); 30 Sep 2004 20:59:05 -0000 Received: from 67.167.52.21 (SquirrelMail authenticated user ceo@l-i-e.com); by www.l-i-e.com with HTTP; Thu, 30 Sep 2004 13:59:05 -0700 (PDT) Message-ID: <2034.67.167.52.21.1096577945.squirrel@www.l-i-e.com> In-Reply-To: <2b5f066d04093013344d048003@mail.gmail.com> References: <2b5f066d04093013344d048003@mail.gmail.com> Date: Thu, 30 Sep 2004 13:59:05 -0700 (PDT) From: "Richard Lynch" To: "Brian McCann" User-Agent: Hostbaby Webmail X-Mailer: Hostbaby Webmail MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal cc: freebsd-questions@freebsd.org Subject: Re: Backup/Restore X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: ceo@l-i-e.com List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Sep 2004 20:59:02 -0000 Brian McCann wrote: > Hi all...I'm having a conceptual problem I can't get around and > was hoping someone can change my focus here. I've been backing up > roughly 6-8 million small files (roughly 2-4k each) using dump, but > restores take forever due to the huge number of files and directories. > Luckily, I haven't had to restore for an emergency yet...but if I > need to, I'm kinda stuck. I've looked at distributed file systems > like CODA, but the number of files I have to deal with will make it > choke. Can anyone offer any suggestions? I've pondered running > rsync, but am very worried about how long that will take... Do the files change a lot, or is it more like a few files added/changed every day, and the bulk don't change? If it's the latter, you could maybe get best performance from something like Subversion (a CVS derivative). Though I suspect rsync would also do well in that case. If a ton of those files are changing all the time, try doing a test on creating a tarball and then backing up the tarball. That may be a simple managable solution. There are probably other more complex solutions of which I am ignorant :-) -- Like Music? http://l-i-e.com/artists.htm