From owner-freebsd-questions@FreeBSD.ORG Fri Apr 5 00:52:18 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 93BBA15E for ; Fri, 5 Apr 2013 00:52:18 +0000 (UTC) (envelope-from gobble.wa@gmail.com) Received: from mail-bk0-x22d.google.com (mail-bk0-x22d.google.com [IPv6:2a00:1450:4008:c01::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 26A127C4 for ; Fri, 5 Apr 2013 00:52:17 +0000 (UTC) Received: by mail-bk0-f45.google.com with SMTP id j10so1843429bkw.18 for ; Thu, 04 Apr 2013 17:52:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=nqIKBPq0HngygWLznW+zmq2jQNA+/B6iHk9EklLG6yE=; b=yQHulop3iGOymQj5zZMhjX6itYDOorlK8WxADNylsOsKSmAVqkaQ7wxAK3SOPqBe8M BgTIsTBKxIemQu6Bac3b53QyQVGS8CY8V7xvxEQnXNylziCWbaxEpSA6YcjsK/7mn9VO HJinmKze3q/6PJlX4X6EQ8D/nVeckdNJ1cT2BPTzzdKFUMDMoEIZJGQDKql5C/NhAZTq RCYvvsIZPBGIiLocro4Yy9sD0c3dj9+oHYtEUjOAek8hb0OmsyRhXajdoGkB0f7DgeMe mwlzSL4qNAKXOhJaKKmMm4Aflg+wxTz8NPsiUvchKu0Oebu2AoRq3t/slingERYvrk8M Fsdg== MIME-Version: 1.0 X-Received: by 10.205.40.72 with SMTP id tp8mr5930836bkb.31.1365123137011; Thu, 04 Apr 2013 17:52:17 -0700 (PDT) Received: by 10.204.25.207 with HTTP; Thu, 4 Apr 2013 17:52:16 -0700 (PDT) Received: by 10.204.25.207 with HTTP; Thu, 4 Apr 2013 17:52:16 -0700 (PDT) In-Reply-To: References: Date: Thu, 4 Apr 2013 17:52:16 -0700 Message-ID: Subject: Re: Regarding zfs send / receive From: Waitman Gobble To: Joar Jegleim Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Terje Elde , freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 00:52:18 -0000 Waitman Gobble San Jose California USA On Apr 4, 2013 2:07 PM, "Joar Jegleim" wrote: > > Hi Terje ! > sorry for late reply, I've been checking my mail, forgetting that all my > mailing list mail are sorted into their own folders skipping inbox :p > > the zfs sync setup is a huge advantage over rsync simply because > incremental rsync of the volume takes ~12 hours, while the zfs differential > snapshot's usually take less than a minute . Though it's only ~1TB of data, > it's more than 2 million jpegs which rsync have to stat ... > I'm guessing my predecessor who chose this setup, over for instance HAST, > didn't feel confident enough regarding HAST in production ( I'm looking > into that for a future solution) . > > There's no legacy stuff on the receiving end, old pools are deleted for > every sync. I haven't got my script here but google pointed me too > https://github.com/hoopty/zfs-sync/blob/master/zfs-sync which look like a > script very similar to the one I'm using . > In fact, I'm gonna take a closer look at that script and see what differs > from my script (apart from it being much prettier :p ) > I didn't know about zpool.cache, gonna check that tomorrow, thanks. > > > > -- > ---------------------- > Joar > Jegleim > Homepage: http://cosmicb.no > Linkedin: http://no.linkedin.com/in/joarjegleim > fb: http://www.facebook.com/joar.jegleim > AKA: CosmicB @Freenode > > ---------------------- > > On 2 April 2013 14:40, Terje Elde wrote: > > > On 2. apr. 2013, at 13.44, Joar Jegleim wrote: > > > So my question(s) to the list would be: > > > In my setup have I taken the use case for zfs send / receive too far > > > (?) as in, it's not meant for this kind of syncing and this often, so > > > there's actually nothing 'wrong'. > > > > I'm not sure if you've taken it too far, but I'm not entirely sure if > > you're getting any advantage over using rsync or similar for this kind of > > thing. > > > > First two things that spring to mind: > > > > Do you have any legacy stuff on the receiving machine? Things like > > physically removed old zpools, that are still in zpool.cache, seems to slow > > down various operations, including creation of new stuffs (such as the > > snapshots you receive). > > > > Also, you don't mention if you're deleting old snapshots on the receiving > > end? If you're doing an incremental run every 15 minutes, that's something > > like 3000 snapshots pr. month, pr. filesystem. > > > > Terje > > > > > hi, i have a similar situation. its better to only rsync new stuff in this case, because you should know when somebody ads something new. for example, a user uploads 200 new images, these are marked 'to sync' and are transferred to the other servers. letting rsync figure out what's new just isnt practical. an idea, works for me. hope it helps. Waitman Gobble San Jose California _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " freebsd-questions-unsubscribe@freebsd.org"