From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 16:04:15 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 385451065670 for ; Thu, 30 Oct 2008 16:04:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 0DB508FC1C for ; Thu, 30 Oct 2008 16:04:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id 6F3E51A000B34 for ; Thu, 30 Oct 2008 09:04:14 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 2IKHYLfGGEmi for ; Thu, 30 Oct 2008 09:04:08 -0700 (PDT) Received: from coal (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 176641A000B18 for ; Thu, 30 Oct 2008 09:04:08 -0700 (PDT) From: Freddie Cash To: freebsd-stable@freebsd.org Date: Thu, 30 Oct 2008 09:04:07 -0700 User-Agent: KMail/1.9.9 References: <20081029231926.GA35188@0lsen.net> <200810300804.31186.fjwcash@gmail.com> <444B4891-057B-4E09-99A1-A50A1187109E@gmail.com> In-Reply-To: <444B4891-057B-4E09-99A1-A50A1187109E@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810300904.07958.fjwcash@gmail.com> Subject: Re: Anyone used rsync scriptology for incremental backup? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2008 16:04:15 -0000 On October 30, 2008 08:18 am Nikolay Denev wrote: > On 30 Oct, 2008, at 17:04 , Freddie Cash wrote: > > On October 30, 2008 01:25 am Nikolay Denev wrote: > >> On 30 Oct, 2008, at 07:00 , Freddie Cash wrote: > >>> On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow > >>> > >>> wrote: > >>>> In this way, each day we generate a batch file that lets us step > >>>> back one > >>>> day. The diffs themselves, compressed with gzip, and extremely > >>>> space efficient. We can step back potentially hundreds of days, > >>>> though it seems > >>>> to throw errors sometimes when backing up Windows boxes, which I > >>>> haven't > >>>> tracked down yet. > >>>> > >>>> But to be honest, soon you can save yourself a lot of hassle by > >>>> simply using > >>>> ZFS and taking snapshots. It'll be faster, and with compression > >>>> very space > >>>> efficient. > >>> > >>> That's exactly what we do, use ZFS and RSync. We have a ZFS > >>> /storage/backup filesystem, with directories for each remote site, > >>> and sub-directories for each server to be backed up. > >>> > >>> Each night we snapshot the directory, then run rsync to backup each > >>> server. Snapshots are named with the current date. For 80 FreeBSD > >>> and Linux servers, we average 10 GB of changed data a night. > >>> > >>> No muss, no fuss. We've used it to restore entire servers (boot > >>> off Knoppix/Frenzy CD, format partitions, rsync back), individual > >>> files (no mounting required, just cd into the > >>> .zfs/snapshot/snapshotname directory and scp the file), and even > >>> once to restore the > >>> permissions > >>> on a pair of servers where a clueless admin "chmod -R user /home" > >>> and > >>> "chmod -R 777 /home". > >>> > >>> Our backup script is pretty much just a double-for loop that scans > >>> a set of site-name directories for server config files, and runs > >>> rsync in parallel (1 per remote site). > >>> > >>> We we looking into using variations on rsnapshot, custom > >>> squashfs/hardlink stuff, and other solutions, but once we started > >>> using ZFS, we stopped looking down those roads. We were able to do > >>> in 3 days of testing and scripting what we hadn't been able to do > >>> in almost a month of research and testing. > >> > >> Do you experience problems with the snapshots? > >> Last time I tried something similiar for backups the bachine > >> began to spit errors after a few days of snapshots. > >> > >> http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004413.h > >>tml > > > > We have 72 daily snapshots so far. Have had up to 30 of them mounted > > read-only while looking for the right version of a file to restore. > > > > These are ZFS snapshots, very different from UFS snapshots. > > > > -- > > Freddie Cash > > fjwcash@gmail.com > > Yes, > > Mine were zfs snapshots too, and I've never managed to create more > than a > few days worth of snapshots before the machine start to print "bad > file descriptor" errors > while trying to access the snapshot directory. > But I guess (hope) this problem does not exist anymore when you are > able to do 72 snapshots. Well, hopefully we're not just lucky. :) We're running 64-bit FreeBSD 7-STABLE from August (after the first round of ZFS patches hit the -STABLE tree). Took about 3 weeks-ish to get the kernel and ARC tuning set right. Since then, it's been smooth sailing. For posterity sake: uname -a: FreeBSD megadrive.sd73.bc.ca 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Aug 19 10:39:29 PDT 2008 root@megadrive.sd73.bc.ca:/usr/obj/usr/src/sys/ZFSHOST amd64 /boot/loader.conf: zfs_load="YES" hw.ata.ata_dma=0 kern.hz="100" vfs.zfs.arc_min="512M" vfs.zfs.arc_max="768M" vfs.zfs.prefetch_disable="1" vfs.zfs.zil_disable="0" vm.kmem_size="1596M" vm.kmem_size_max="1596M" The ata_dma=0 is needed as / is a gmirror of two 2 GB CompactFlash cards attached to IDE adapters, and they don't support DMA. And the zpool is a raidz2 of 12x400 GB SATA drives connected to a 3Ware 9550SXU-16ML, and 12x500 GB SATA drives connected to a 3Ware 9650SE-12ML, all configured as SingleDisk arrays (so 24 daXX devices). There's just over 9 TB of usable space in the zpool. zfs list -t filesystem: NAME USED AVAIL REFER MOUNTPOINT storage 3.32T 4.43T 117K /storage storage/Backup186 697M 4.43T 697M /storage/Backup186 storage/backup 3.25T 4.43T 2.21T /storage/backup storage/home 19.9G 4.43T 19.9G /home storage/tmp 27.2M 4.43T 27.2M /tmp storage/usr 2.71G 4.45T 223M /usr storage/usr/local 128M 4.45T 128M /usr/local storage/usr/obj 1.63G 4.45T 1.63G /usr/obj storage/usr/ports 540M 4.45T 449M /usr/ports storage/usr/ports/distfiles 91.2M 4.45T 91.2M /usr/ports/distfiles storage/usr/src 210M 4.45T 210M /usr/src storage/var 1.75G 4.45T 1.75G /var zfs list -t snapshot | wc -l 74 -- Freddie Cash fjwcash@gmail.com