From owner-freebsd-questions@FreeBSD.ORG Fri Apr 5 10:11:38 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D20A3306 for ; Fri, 5 Apr 2013 10:11:38 +0000 (UTC) (envelope-from joar.jegleim@gmail.com) Received: from mail-wg0-f45.google.com (mail-wg0-f45.google.com [74.125.82.45]) by mx1.freebsd.org (Postfix) with ESMTP id 6B1C9CEB for ; Fri, 5 Apr 2013 10:11:38 +0000 (UTC) Received: by mail-wg0-f45.google.com with SMTP id x12so3675815wgg.12 for ; Fri, 05 Apr 2013 03:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=Q4UWu0xBWrE1EmgDeQDKI83WkOs+N8zNPkMj5omfQpk=; b=HDZ5Y7shcsT7reOHFzfxJ1fb1FX09Po/CY1xafA3XQlyFa69ursKOa3Vt1qhnJH5DZ NTPaxoQZ0q2YgoUssdY9U2u5zMxLBLdW67ldknDC7rvK3d3xKBp6m14ZRmR9e2S2CphL R4sDA7LYKNfvHar/nwYu9isd7ofKa4awVTe0hKP8+yJHQJrUP5R2AX0yAxiMpzxT/X3L A1UYD8YNlJMwxM3CS81bql6KvJNzI8FQZyULj8Imq7wyDfEL+lB/HoepZ1QsGToNp8ss mrCx/0leu4khLPkBuXUBpQrRubg7OmQ+3g9CgwYsnSFkMGHZjku6hfMtVYrEXfIJnCFs nLvg== MIME-Version: 1.0 X-Received: by 10.180.97.233 with SMTP id ed9mr2956147wib.32.1365156696926; Fri, 05 Apr 2013 03:11:36 -0700 (PDT) Received: by 10.216.34.9 with HTTP; Fri, 5 Apr 2013 03:11:36 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Apr 2013 12:11:36 +0200 Message-ID: Subject: Re: Regarding zfs send / receive From: Joar Jegleim To: Waitman Gobble Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Terje Elde , freebsd-questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 10:11:38 -0000 sounds like a good idea, I might look into that, thnx. Terje: zpool.cache is only 860 bytes, I don't think that should cause any problems (?) -- ---------------------- Joar Jegleim Homepage: http://cosmicb.no Linkedin: http://no.linkedin.com/in/joarjegleim fb: http://www.facebook.com/joar.jegleim AKA: CosmicB @Freenode ---------------------- On 5 April 2013 02:52, Waitman Gobble wrote: > Waitman Gobble > San Jose California USA > > On Apr 4, 2013 2:07 PM, "Joar Jegleim" wrote: > > > > Hi Terje ! > > sorry for late reply, I've been checking my mail, forgetting that all my > > mailing list mail are sorted into their own folders skipping inbox :p > > > > the zfs sync setup is a huge advantage over rsync simply because > > incremental rsync of the volume takes ~12 hours, while the zfs > differential > > snapshot's usually take less than a minute . Though it's only ~1TB of > data, > > it's more than 2 million jpegs which rsync have to stat ... > > I'm guessing my predecessor who chose this setup, over for instance HAST, > > didn't feel confident enough regarding HAST in production ( I'm looking > > into that for a future solution) . > > > > There's no legacy stuff on the receiving end, old pools are deleted for > > every sync. I haven't got my script here but google pointed me too > > https://github.com/hoopty/zfs-sync/blob/master/zfs-sync which look like > a > > script very similar to the one I'm using . > > In fact, I'm gonna take a closer look at that script and see what differs > > from my script (apart from it being much prettier :p ) > > I didn't know about zpool.cache, gonna check that tomorrow, thanks. > > > > > > > > -- > > ---------------------- > > Joar > > Jegleim > > Homepage: http://cosmicb.no > > Linkedin: http://no.linkedin.com/in/joarjegleim > > fb: http://www.facebook.com/joar.jegleim > > AKA: CosmicB @Freenode > > > > ---------------------- > > > > On 2 April 2013 14:40, Terje Elde wrote: > > > > > On 2. apr. 2013, at 13.44, Joar Jegleim wrote: > > > > So my question(s) to the list would be: > > > > In my setup have I taken the use case for zfs send / receive too far > > > > (?) as in, it's not meant for this kind of syncing and this often, so > > > > there's actually nothing 'wrong'. > > > > > > I'm not sure if you've taken it too far, but I'm not entirely sure if > > > you're getting any advantage over using rsync or similar for this kind > of > > > thing. > > > > > > First two things that spring to mind: > > > > > > Do you have any legacy stuff on the receiving machine? Things like > > > physically removed old zpools, that are still in zpool.cache, seems to > slow > > > down various operations, including creation of new stuffs (such as the > > > snapshots you receive). > > > > > > Also, you don't mention if you're deleting old snapshots on the > receiving > > > end? If you're doing an incremental run every 15 minutes, that's > something > > > like 3000 snapshots pr. month, pr. filesystem. > > > > > > Terje > > > > > > > > > > hi, > i have a similar situation. its better to only rsync new stuff in this > case, because you should know when somebody ads something new. > > for example, a user uploads 200 new images, these are marked 'to sync' and > are transferred to the other servers. letting rsync figure out what's new > just isnt practical. > > an idea, works for me. hope it helps. > > Waitman Gobble > San Jose California _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >