From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 14:50:38 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F843106567B for ; Fri, 14 Oct 2011 14:50:38 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0F0E28FC1A for ; Fri, 14 Oct 2011 14:50:37 +0000 (UTC) Received: by gyd8 with SMTP id 8so1533536gyd.13 for ; Fri, 14 Oct 2011 07:50:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=18D+nV6xM8IixPxl5ZCXus4ohDyRJy1f+Fa5QDAxEeo=; b=wMS2/zN5PvfXwEd2OD9HPKmIu7bptj9TRi6rGxVU+eC9RTsO0uMZsUp8bjW7Ju497Q NITmaJvd6LvXbagG5IVdRmutnT0P1jhk/TM9/0xnU66gfcMqi/SExvgkmYANdbVLy93I vJPlfIt1SV6ZG8kLtH3BIlimDzOz1+WlV/S4g= MIME-Version: 1.0 Received: by 10.236.187.101 with SMTP id x65mr12364384yhm.63.1318603837498; Fri, 14 Oct 2011 07:50:37 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.236.103.33 with HTTP; Fri, 14 Oct 2011 07:50:37 -0700 (PDT) In-Reply-To: <4E97D24C.4010606@o2.pl> References: <20111013120032.D6BA71065760@hub.freebsd.org> <4E97D24C.4010606@o2.pl> Date: Fri, 14 Oct 2011 07:50:37 -0700 X-Google-Sender-Auth: qqWSdnnf0BWaDCw5MwoSeNeNjWU Message-ID: From: Artem Belevich To: =?ISO-8859-2?Q?Radio_m=B3odych_bandyt=F3w?= Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: ZFS/compression/performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2011 14:50:38 -0000 2011/10/13 Radio m=B3odych bandyt=F3w : > On 2011-10-13 14:00, freebsd-fs-request@freebsd.org wrote: >> >> An option is not too compress with ZFS rather directly with gzip however= I >> would still need lots of temporary storage for manipulation, which is wh= at >> I am doing now (e.g., sort). Processing with zcat isn't always a good >> solution because some applications want files, but you have to do what y= ou >> have to do. > > It seems that with your data gzipping directly is a better option. Though= I > suggest that you experiment with codecs that support larger dictionary, i= .e. > 7zip, I expect that you would see huge strength improvement with somethin= g > like 7z a -mx=3D1 -md=3D26 out.7z in. You can use higher -md values if yo= u have > enough memory, compression mode 1 (mx=3D1) uses 4,5*2^md bytes of RAM, so= if > my maths is good, md=3D26 uses ~288 MB. If LZMA is too slow, you can at l= east > try 7-zip's deflate64. It's not great, but not as bad as gzip. Yup. Stand-alone archiver may work better. ZFS compression works on blocks. Subsequent blocks can't benefit from the data gathered compressing preceding block, so overall compression rate with ZFS would be lower than that of stand-alone gzip with the same compression level. On the other hand, ZFS will parallelize compression and on multi-core machine it would compress the same amount of data in less time than single instance of gzip would. --Artem