Date: Sun, 12 Jun 2011 00:45:57 -0400 From: David Magda <dmagda@ee.ryerson.ca> To: Bob Friesenhahn <bfriesen@simple.dallas.tx.us> Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: HEADS UP: ZFS v28 merged to 8-STABLE Message-ID: <C59BA6AA-3BCC-43F7-9EFE-55BF96CC1CAC@ee.ryerson.ca> In-Reply-To: <alpine.GSO.2.01.1106101619520.10265@freddy.simplesystems.org> References: <4DECB197.8020102@FreeBSD.org> <20110610211202.GA52253@icarus.home.lan> <alpine.GSO.2.01.1106101619520.10265@freddy.simplesystems.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jun 10, 2011, at 17:24, Bob Friesenhahn wrote: > Dedup can require a huge amount of RAM, or a dedicated L2ARC SSD, = depending on the size of your storage. You should not enable it unless = you are prepared for the consequences. Under OpenSolaris, each tracking entry for a deduped block (which can be = between 512B to 128KB) can be up to 376 bytes (struct ddt_entry): so for = one 1 GB (10^9) of deduped data (244140 blocks@4K), you would need ~91MB = of overhead to keep track of it; for 1 TB (10^12) of deduped data you = would need ~91 GB of space to keep track of all the blocks. And if you = can't fit the DDT in RAM, it will have to be saved to disk, which means = more I/O to fetch the data. = http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts= /common/fs/zfs/sys/ddt.h?rev=3D1.2 If your data is in blocks smaller than 4K you'll need more memory for = the DDT; if the data is broken up into blocks larger than 4K you'll = probably need less. Also remember that even though an L2ARC cache may save you from having = to go to spinning rust, you still need to use some RAM (struct = arc_buf_hdr; ~178B) to reference the DDT stuff in L2ARC: = http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts= /common/fs/zfs/arc.c A few threads on zfs-dicuss on this: = http://mail.opensolaris.org/pipermail/zfs-discuss/2011-April/thread.html#4= 8026 = http://mail.opensolaris.org/pipermail/zfs-discuss/2011-May/thread.html#481= 85 Also, the above numbers are for OpenSolaris: someone may want to check = the structure sizes for FreeBSD to be sure. They should get you in the = right ballpark though.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C59BA6AA-3BCC-43F7-9EFE-55BF96CC1CAC>