From owner-freebsd-fs@FreeBSD.ORG Tue Nov 1 17:55:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C9BA0106566B for ; Tue, 1 Nov 2011 17:55:28 +0000 (UTC) (envelope-from shivaram.u@quadstor.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9A11E8FC08 for ; Tue, 1 Nov 2011 17:55:28 +0000 (UTC) Received: by iabz21 with SMTP id z21so3674360iab.13 for ; Tue, 01 Nov 2011 10:55:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.28.106 with SMTP id l42mr84581ibc.66.1320168337058; Tue, 01 Nov 2011 10:25:37 -0700 (PDT) Received: by 10.231.60.204 with HTTP; Tue, 1 Nov 2011 10:25:36 -0700 (PDT) X-Originating-IP: [117.192.117.147] In-Reply-To: References: Date: Tue, 1 Nov 2011 22:55:36 +0530 Message-ID: From: Shivaram Upadhyayula To: Dennis Glatting Content-Type: multipart/mixed; boundary=001517740adcc62d6f04b0afa4bf X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS/compression/performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2011 17:55:29 -0000 --001517740adcc62d6f04b0afa4bf Content-Type: text/plain; charset=ISO-8859-1 On Thu, Oct 13, 2011 at 9:47 AM, Dennis Glatting wrote: > On Wed, 2011-10-12 at 08:59 +0100, Steven Hartland wrote: > ----- Original Message ----- >> >> From: "Dennis Glatting" >> >> >> Have you tried using the alternative compression algorithms >> e.g. lzjb or gzip-[1-5] the default gzip = gzip-6 >> > > I have tried lzjb and I am unimpressed. I have not tried different levels of > gzip on ZFS but I have tried it on documents with results I expected. > > As I mentioned, I have a lot of data. Two files were 26GB uncompressed but I > had to kill those data sets because I ran out of room (I have reorganized my > arrays since then). My ZFS compression ratio is 4.93x and I would require > more storage at different gzip levels or ljzb. > > An option is not too compress with ZFS rather directly with gzip however I > would still need lots of temporary storage for manipulation, which is what I > am doing now (e.g., sort). Processing with zcat isn't always a good solution > because some applications want files, but you have to do what you have to > do. > A few years back there was a discussion of about the possibility of other compression algorithms in ZFS (http://mail.opensolaris.org/pipermail/zfs-discuss/2007-June/011952.html). But it looks like there hasn't been anything much further on that. I have recently started using ZFS and during that time i have tried out ZFS with lzf (http://oldhome.schmorp.de/marc/liblzf.html) and it seems to perform much better, both in speed and ratio over lzjb. Anyway my point is that, somewhere down the line other compression algorithms should be evaluated. gzip seems slow and it looks like lzjb may not be sufficient. For anyone interested, I have attached some of the tests i had run and the diff for lzf support. Cheers, Shivaram -- Reduce Storage expenditure with QUADStor Storage Virtualization http://www.quadstor.com --001517740adcc62d6f04b0afa4bf--