Date: Fri, 5 Apr 2013 12:11:36 +0200 From: Joar Jegleim <joar.jegleim@gmail.com> To: Waitman Gobble <gobble.wa@gmail.com> Cc: Terje Elde <terje@elde.net>, freebsd-questions <freebsd-questions@freebsd.org> Subject: Re: Regarding zfs send / receive Message-ID: <CAFfb-hojvD5n8_zkQ_X9VTqTi-%2BWXfK4qptKaJtu%2Bxk70fcsiw@mail.gmail.com> In-Reply-To: <CAFuo_fwtQnRFqpd%2BH=YNS8Ghk7nRf5mW7Ei9pz9c346PK0aOvQ@mail.gmail.com> References: <CAFfb-hqTFH0oK9rOpWHo6wrodzuOm5oRbetqY3RSXvF7Gsa6PA@mail.gmail.com> <E019EA56-708A-481D-9FD8-5EB66D6B98AA@elde.net> <CAFfb-hoDYEdZGo5gfv=PbyCUKuDC6N0ECn=27YzUYEW=C%2BeaLA@mail.gmail.com> <CAFuo_fwtQnRFqpd%2BH=YNS8Ghk7nRf5mW7Ei9pz9c346PK0aOvQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
sounds like a good idea, I might look into that, thnx. Terje: zpool.cache is only 860 bytes, I don't think that should cause any problems (?) -- ---------------------- Joar Jegleim Homepage: http://cosmicb.no Linkedin: http://no.linkedin.com/in/joarjegleim fb: http://www.facebook.com/joar.jegleim AKA: CosmicB @Freenode ---------------------- On 5 April 2013 02:52, Waitman Gobble <gobble.wa@gmail.com> wrote: > Waitman Gobble > San Jose California USA > > On Apr 4, 2013 2:07 PM, "Joar Jegleim" <joar.jegleim@gmail.com> wrote: > > > > Hi Terje ! > > sorry for late reply, I've been checking my mail, forgetting that all my > > mailing list mail are sorted into their own folders skipping inbox :p > > > > the zfs sync setup is a huge advantage over rsync simply because > > incremental rsync of the volume takes ~12 hours, while the zfs > differential > > snapshot's usually take less than a minute . Though it's only ~1TB of > data, > > it's more than 2 million jpegs which rsync have to stat ... > > I'm guessing my predecessor who chose this setup, over for instance HAST, > > didn't feel confident enough regarding HAST in production ( I'm looking > > into that for a future solution) . > > > > There's no legacy stuff on the receiving end, old pools are deleted for > > every sync. I haven't got my script here but google pointed me too > > https://github.com/hoopty/zfs-sync/blob/master/zfs-sync which look like > a > > script very similar to the one I'm using . > > In fact, I'm gonna take a closer look at that script and see what differs > > from my script (apart from it being much prettier :p ) > > I didn't know about zpool.cache, gonna check that tomorrow, thanks. > > > > > > > > -- > > ---------------------- > > Joar > > Jegleim > > Homepage: http://cosmicb.no > > Linkedin: http://no.linkedin.com/in/joarjegleim > > fb: http://www.facebook.com/joar.jegleim > > AKA: CosmicB @Freenode > > > > ---------------------- > > > > On 2 April 2013 14:40, Terje Elde <terje@elde.net> wrote: > > > > > On 2. apr. 2013, at 13.44, Joar Jegleim wrote: > > > > So my question(s) to the list would be: > > > > In my setup have I taken the use case for zfs send / receive too far > > > > (?) as in, it's not meant for this kind of syncing and this often, so > > > > there's actually nothing 'wrong'. > > > > > > I'm not sure if you've taken it too far, but I'm not entirely sure if > > > you're getting any advantage over using rsync or similar for this kind > of > > > thing. > > > > > > First two things that spring to mind: > > > > > > Do you have any legacy stuff on the receiving machine? Things like > > > physically removed old zpools, that are still in zpool.cache, seems to > slow > > > down various operations, including creation of new stuffs (such as the > > > snapshots you receive). > > > > > > Also, you don't mention if you're deleting old snapshots on the > receiving > > > end? If you're doing an incremental run every 15 minutes, that's > something > > > like 3000 snapshots pr. month, pr. filesystem. > > > > > > Terje > > > > > > > > > > hi, > i have a similar situation. its better to only rsync new stuff in this > case, because you should know when somebody ads something new. > > for example, a user uploads 200 new images, these are marked 'to sync' and > are transferred to the other servers. letting rsync figure out what's new > just isnt practical. > > an idea, works for me. hope it helps. > > Waitman Gobble > San Jose California _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFfb-hojvD5n8_zkQ_X9VTqTi-%2BWXfK4qptKaJtu%2Bxk70fcsiw>