Date: Fri, 14 Oct 2011 07:50:37 -0700 From: Artem Belevich <art@freebsd.org> To: =?ISO-8859-2?Q?Radio_m=B3odych_bandyt=F3w?= <radiomlodychbandytow@o2.pl> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS/compression/performance Message-ID: <CAFqOu6gBiounMAzvrW8orNKRCJXQ34ujdABSMOk-tnBomPUHgQ@mail.gmail.com> In-Reply-To: <4E97D24C.4010606@o2.pl> References: <20111013120032.D6BA71065760@hub.freebsd.org> <4E97D24C.4010606@o2.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
2011/10/13 Radio m=B3odych bandyt=F3w <radiomlodychbandytow@o2.pl>: > On 2011-10-13 14:00, freebsd-fs-request@freebsd.org wrote: >> >> An option is not too compress with ZFS rather directly with gzip however= I >> would still need lots of temporary storage for manipulation, which is wh= at >> I am doing now (e.g., sort). Processing with zcat isn't always a good >> solution because some applications want files, but you have to do what y= ou >> have to do. > > It seems that with your data gzipping directly is a better option. Though= I > suggest that you experiment with codecs that support larger dictionary, i= .e. > 7zip, I expect that you would see huge strength improvement with somethin= g > like 7z a -mx=3D1 -md=3D26 out.7z in. You can use higher -md values if yo= u have > enough memory, compression mode 1 (mx=3D1) uses 4,5*2^md bytes of RAM, so= if > my maths is good, md=3D26 uses ~288 MB. If LZMA is too slow, you can at l= east > try 7-zip's deflate64. It's not great, but not as bad as gzip. Yup. Stand-alone archiver may work better. ZFS compression works on blocks. Subsequent blocks can't benefit from the data gathered compressing preceding block, so overall compression rate with ZFS would be lower than that of stand-alone gzip with the same compression level. On the other hand, ZFS will parallelize compression and on multi-core machine it would compress the same amount of data in less time than single instance of gzip would. --Artem
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFqOu6gBiounMAzvrW8orNKRCJXQ34ujdABSMOk-tnBomPUHgQ>