From owner-freebsd-fs@FreeBSD.ORG Wed Nov 2 12:45:32 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9839D106566B for ; Wed, 2 Nov 2011 12:45:32 +0000 (UTC) (envelope-from radiomlodychbandytow@o2.pl) Received: from moh1-ve2.go2.pl (moh1-ve2.go2.pl [193.17.41.132]) by mx1.freebsd.org (Postfix) with ESMTP id 29A5C8FC18 for ; Wed, 2 Nov 2011 12:45:31 +0000 (UTC) Received: from moh1-ve2.go2.pl (unknown [10.0.0.132]) by moh1-ve2.go2.pl (Postfix) with ESMTP id F2E1D1044099 for ; Wed, 2 Nov 2011 13:45:28 +0100 (CET) Received: from unknown (unknown [10.0.0.142]) by moh1-ve2.go2.pl (Postfix) with SMTP for ; Wed, 2 Nov 2011 13:45:28 +0100 (CET) Received: from host892524678.com-promis.3s.pl [89.25.246.78] by poczta.o2.pl with ESMTP id xQCQnb; Wed, 02 Nov 2011 13:45:28 +0100 Message-ID: <4EB13B65.20507@o2.pl> Date: Wed, 02 Nov 2011 13:45:25 +0100 From: =?UTF-8?B?UmFkaW8gbcWCb2R5Y2ggYmFuZHl0w7N3?= User-Agent: Mozilla/5.0 (Windows NT 5.2; WOW64; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <20111102120026.5F28D1065741@hub.freebsd.org> In-Reply-To: <20111102120026.5F28D1065741@hub.freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-O2-Trust: 2, 67 X-O2-SPF: neutral Cc: Subject: Re: freebsd-fs Digest, Vol 437, Issue 3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Nov 2011 12:45:32 -0000 On 2011-11-02 13:00, freebsd-fs-request@freebsd.org wrote: > Message: 1 > Date: Tue, 1 Nov 2011 22:55:36 +0530 > From: Shivaram Upadhyayula > Subject: Re: ZFS/compression/performance > To: Dennis Glatting > Cc:freebsd-fs@freebsd.org > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Thu, Oct 13, 2011 at 9:47 AM, Dennis Glatting wrote: >> > On Wed, 2011-10-12 at 08:59 +0100, Steven Hartland wrote: >> > ----- Original Message ----- >>> >> >>> >> From: "Dennis Glatting" >>> >> >>> >> > >>> >> Have you tried using the alternative compression algorithms >>> >> e.g. lzjb or gzip-[1-5] the default gzip = gzip-6 >>> >> >> > >> > I have tried lzjb and I am unimpressed. I have not tried different levels of >> > gzip on ZFS but I have tried it on documents with results I expected. >> > >> > As I mentioned, I have a lot of data. Two files were 26GB uncompressed but I >> > had to kill those data sets because I ran out of room (I have reorganized my >> > arrays since then). My ZFS compression ratio is 4.93x and I would require >> > more storage at different gzip levels or ljzb. >> > >> > An option is not too compress with ZFS rather directly with gzip however I >> > would still need lots of temporary storage for manipulation, which is what I >> > am doing now (e.g., sort). Processing with zcat isn't always a good solution >> > because some applications want files, but you have to do what you have to >> > do. >> > > A few years back there was a discussion of about the possibility of > other compression algorithms in ZFS > (http://mail.opensolaris.org/pipermail/zfs-discuss/2007-June/011952.html). > But it looks like there hasn't been anything much further on that. I > have recently started using ZFS and during that time i have tried out > ZFS with lzf (http://oldhome.schmorp.de/marc/liblzf.html) and it seems > to perform much better, both in speed and ratio over lzjb. > > Anyway my point is that, somewhere down the line other compression > algorithms should be evaluated. gzip seems slow and it looks like lzjb > may not be sufficient. For anyone interested, I have attached some of > the tests i had run and the diff for lzf support. > > Cheers, > Shivaram > > -- Reduce Storage expenditure with QUADStor Storage Virtualization > http://www.quadstor.com I did some synthetic tests and lzf was OK, but not great. LZJB was terrible though. http://encode.ru/threads/1266-In-memory-benchmark-with-fastest-LZSS-%28QuickLZ-Snappy%29-compressors?p=25913#post25913 https://extrememoderate.wordpress.com/2011/08/14/synthetic-test-of-filesystem-compression-part-1/ https://extrememoderate.wordpress.com/2011/08/20/synthetic-test-of-filesystem-compression-part-2/ I'm in the process of doing a more comprehensive part 3 and I think that LZ4 should be better. -- Twoje radio