Date: Wed, 3 Jul 2013 19:15:35 -0700 From: Jeremy Chadwick <jdc@koitsu.org> To: Berend de Boer <berend@pobox.com> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: EBS snapshot backups from a FreeBSD zfs file system: zpool freeze? Message-ID: <20130704021535.GA77546@icarus.home.lan> In-Reply-To: <8761wr3xxk.wl%berend@pobox.com> References: <A5A66641-5EF9-454E-A767-009480EE404E@dragondata.com> <871u7g57rl.wl%berend@pobox.com> <op.wznad7th34t2sn@tech304.office.supranet.net> <87mwq34emp.wl%berend@pobox.com> <20130703200241.GB60515@in-addr.com> <87k3l748gb.wl%berend@pobox.com> <20130703233631.GA74698@icarus.home.lan> <87d2qz42q4.wl%berend@pobox.com> <20130704010815.GB75529@icarus.home.lan> <8761wr3xxk.wl%berend@pobox.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 04, 2013 at 01:40:07PM +1200, Berend de Boer wrote: > >>>>> "Jeremy" == Jeremy Chadwick <jdc@koitsu.org> writes: > > > Jeremy> Also, because nobody seems to warn others of this: if > Jeremy> you go the ZFS route on FreeBSD, please do not use > Jeremy> features like dedup or compression. > > Exactly the two reasons why I'm experimenting with FreeBSD on AWs. > > Please tell me more. dedup has immense and crazy memory requirements; the commonly referenced model (which is in no way precise, it's just a general recommendation) is that for every 1TB of data you need 1GB of RAM just for the DDT (deduplication table)) -- understand that ZFS's ARC also eats lots of memory, so when I say 1GB of RAM, I'm talking about that being *purely dedicated* to DDT. But as I said the need varies depending on the type of data you have. When using dedup, the general attitude is "give ZFS as much memory as possible. Max your DIMM slots out with the biggest DIMMs the MCH can support". Many problems I have seen on the FreeBSD lists -- and one horror story on Solaris -- often pertain to people trying dedup. There have been reported issues with resilvering pools that use dedup, or even simply mounting filesystems using dedup. The situation when dedup is in use becomes significantly more complex in a "something is broken" scenario. The horror story I've heard and retell is this one, and this is me going off of memory: There was supposedly an Oracle customer who had been using dedup for some time, and they began to have problems (I don't remember what; if it was with ZFS, the controller, disks, or what). Anyway, the situation was such that the client needed to either resilver their pool, or just get their data -- but because they were using dedup, they could not. The system could not be upgraded to have more RAM (which would have alleviated the pains). The solution which was chosen was for Oracle to actually ship the customer an entire bare metal system with a gargantuan amount of RAM (hundreds of gigabytes; I often say 384GB because that's what sticks in my mind for some reason, maybe it was 192GB, doesn't matter), just to recover from the situation. compression is generally safe to use on FreeBSD, but there are often surprising changes to certain behaviours that people don't consider: the most common one I see reported is conflicting information between what "df", "du", and "zfs list" show. AFAIK this applies to Solaris/Illumos too, so it's just the nature of the beast. compression doesn't have the crazy memory requirements of dedup, obviously -- two separate things, don't confuse the two. :-) The final item is the one that, still to this day, keeps me from using either dedup or compression on FreeBSD (well actually I'd never consider dedup, only compression): system interactivity is destroyed when using either of these features. The system will regularly stall/lock up (depending on the I/O, for a few seconds) regularly, even at VGA console. This problem is specific to the FreeBSD port of ZFS as of this writing; Solaris/Illumos addressed this long ago. Rather than re-write it, I recommend you read my post from Feburary 2013 which references my convo with Bob Friesenhahn in October 2011 (please read all the quoted material too): http://lists.freebsd.org/pipermail/freebsd-stable/2013-February/072171.html Changing the compression scheme does not solve the issue; the less CPU-intensive schemes (ex. lzjb) help decrease the impact but do not solve it. All that said: there are people (often FreeNAS folks using their systems solely as a dedicated NAS, not as a shell server or desktop or other things) who do use these features happily and do not care about the last issue. Cool/great, I'm glad it works for them. But in my case it's not acceptable. If/when the above issue is addressed (putting the ZFS writer threads into their own priority/scheduling class), I look forward to using compression (but never dedup, I don't have the hardware/memory for that kind of thing). Otherwise please spend an afternoon looking through freebsd-fs and freebsd-stable lists over the past 2 years (see web archives) and reading about different stories/situations. I always, *always* advocate this. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130704021535.GA77546>