Date: Sun, 20 Jan 2013 08:26:10 +0000 (UTC) From: Xin LI <delphij@FreeBSD.org> To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r245686 - in user/delphij/zfs-lz4: cddl/contrib/opensolaris/cmd/zfs cddl/contrib/opensolaris/cmd/zpool sys/cddl/boot/zfs sys/cddl/compat/opensolaris/sys sys/cddl/contrib/opensolaris/com... Message-ID: <201301200826.r0K8QAiw090246@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: delphij Date: Sun Jan 20 08:26:09 2013 New Revision: 245686 URL: http://svnweb.freebsd.org/changeset/base/245686 Log: WIP checkpoint for LZ4 merge (MFV r245512). WARNING: there is known data corruption issue with this version and this is only intended for developers. Added: user/delphij/zfs-lz4/sys/cddl/boot/zfs/lz4.c (contents, props changed) user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4 - copied unchanged from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4 user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip - copied unchanged from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lz4.c - copied, changed from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c Modified: user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zfs/zfs.8 user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7 user/delphij/zfs-lz4/sys/cddl/boot/zfs/README user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfsimpl.h user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfssubr.c user/delphij/zfs-lz4/sys/cddl/compat/opensolaris/sys/byteorder.h user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfs_prop.c user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/Makefile.files user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Directory Properties: user/delphij/zfs-lz4/cddl/contrib/opensolaris/ (props changed) user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/ (props changed) Modified: user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zfs/zfs.8 ============================================================================== --- user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zfs/zfs.8 Sun Jan 20 08:26:09 2013 (r245686) @@ -24,6 +24,7 @@ .\" Copyright (c) 2011, Pawel Jakub Dawidek <pjd@FreeBSD.org> .\" Copyright (c) 2012, Glen Barber <gjb@FreeBSD.org> .\" Copyright (c) 2012, Bryan Drewery <bdrewery@FreeBSD.org> +.\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved. .\" .\" $FreeBSD$ .\" @@ -866,7 +867,7 @@ but this may change in future releases). disables integrity checking on user data. Disabling checksums is .Em NOT a recommended practice. -.It Sy compression Ns = Ns Cm on | off | lzjb | gzip | gzip- Ns Ar N | Cm zle +.It Sy compression Ns = Ns Cm on | off | lzjb | gzip | gzip- Ns Ar N | zle | Cm lz4 Controls the compression algorithm used for this dataset. The .Cm lzjb compression algorithm is optimized for performance while providing decent data @@ -894,6 +895,26 @@ The .Cm zle compression algorithm compresses runs of zeros. .Pp +The +.Sy lz4 +compression algorithm is a high-performance replacement +for the +.Sy lzjb +algorithm. It features significantly faster +compression and decompression, as well as a moderately higher +compression ratio than +.Sy lzjb , +but can only be used on pools with +the +.Sy lz4_compress +feature set to +.Sy enabled . +See +.Xr zpool-features 7 +for details on ZFS feature flags and the +.Sy lz4_compress +feature. +.Pp This property can also be referred to by its shortened column name .Cm compress . Changing this property affects only newly-written data. Modified: user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7 ============================================================================== --- user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7 Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/cddl/contrib/opensolaris/cmd/zpool/zpool-features.7 Sun Jan 20 08:26:09 2013 (r245686) @@ -18,6 +18,7 @@ .\" information: Portions Copyright [yyyy] [name of copyright owner] .\" .\" Copyright (c) 2012 by Delphix. All rights reserved. +.\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved. .\" .\" $FreeBSD$ .\" @@ -185,6 +186,49 @@ This feature is .Sy active while there are any filesystems, volumes, or snapshots which were created after enabling this feature. +.It Sy lz4_compress +.Bl -column "READ\-ONLY COMPATIBLE" "org.illumos:lz4_compress" +.It GUID Ta org.illumos:lz4_compress +.It READ\-ONLY COMPATIBLE Ta no +.It DEPENDENCIES Ta none +.El +.Pp +.Sy lz4 +is a high-performance real-time compression algorithm that +features significantly faster compression and decompression as well as a +higher compression ratio than the older +.Sy lzjb +compression. +Typically, +.Sy lz4 +compression is approximately 50% faster on +compressible data and 200% faster on incompressible data than +.Sy lzjb . +It is also approximately 80% faster on decompression, while +giving approximately 10% better compression ratio. +.Pp +When the +.Sy lz4_compress +feature is set to +.Sy enabled , +the +administrator can turn on +.Sy lz4 +compression on any dataset on the +pool using the +.Xr zfs 8 +command. Please note that doing so will +immediately activate the +.Sy lz4_compress +feature on the underlying +pool (even before any data is written). Since this feature is not +read-only compatible, this operation will render the pool unimportable +on systems without support for the +.Sy lz4_compress +feature. At the +moment, this operation cannot be reversed. Booting off of +.Sy lz4 +-compressed root pools is supported. .El .Sh SEE ALSO .Xr zpool 8 Modified: user/delphij/zfs-lz4/sys/cddl/boot/zfs/README ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/boot/zfs/README Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/boot/zfs/README Sun Jan 20 08:26:09 2013 (r245686) @@ -5,6 +5,7 @@ are used by the ZFS bootstrap: fletcher.c checksum support sha256.c checksum support + lz4.c compression support lzjb.c compression support zfssubr.c checksum, compression and raidz support zfsimpl.h mostly describing the physical layout Added: user/delphij/zfs-lz4/sys/cddl/boot/zfs/lz4.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/delphij/zfs-lz4/sys/cddl/boot/zfs/lz4.c Sun Jan 20 08:26:09 2013 (r245686) @@ -0,0 +1,310 @@ +/* + * LZ4 - Fast LZ compression algorithm + * Header File + * Copyright (C) 2011-2013, Yann Collet. + * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * You can contact the author at : + * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html + * - LZ4 source repository : http://code.google.com/p/lz4/ + */ + +static int LZ4_uncompress_unknownOutputSize(const char *source, char *dest, + int isize, int maxOutputSize); + +/* ARGSUSED */ +static int +lz4_decompress(void *s_start, void *d_start, size_t s_len, size_t d_len, int dummy __unused) +{ + const uint8_t *src = s_start; + uint32_t bufsiz = htonl(*(uint32_t *)src); + + /* invalid compressed buffer size encoded at start */ + if (bufsiz + 4 > s_len) + return (1); + + /* + * Returns 0 on success (decompression function returned non-negative) + * and non-zero on failure (decompression function returned negative). + */ + return (LZ4_uncompress_unknownOutputSize(s_start + 4, d_start, bufsiz, + d_len) < 0); +} + +/* + * CPU Feature Detection + */ + +/* 32 or 64 bits ? */ +#if (defined(__x86_64__) || defined(__x86_64) || defined(__amd64__) || \ + defined(__amd64) || defined(__ppc64__) || defined(_WIN64) || \ + defined(__LP64__) || defined(_LP64)) +#define LZ4_ARCH64 1 +#else +#define LZ4_ARCH64 0 +#endif + +/* + * Little Endian or Big Endian? + * Note: overwrite the below #define if you know your architecture endianess. + */ +#if (defined(__BIG_ENDIAN__) || defined(__BIG_ENDIAN) || \ + defined(_BIG_ENDIAN) || defined(_ARCH_PPC) || defined(__PPC__) || \ + defined(__PPC) || defined(PPC) || defined(__powerpc__) || \ + defined(__powerpc) || defined(powerpc) || \ + ((defined(__BYTE_ORDER__)&&(__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)))) +#define LZ4_BIG_ENDIAN 1 +#else + /* + * Little Endian assumed. PDP Endian and other very rare endian format + * are unsupported. + */ +#endif + +/* + * Compiler Options + */ +#if __STDC_VERSION__ >= 199901L /* C99 */ +/* "restrict" is a known keyword */ +#else +/* Disable restrict */ +#define restrict +#endif + +#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) + +#define lz4_bswap16(x) ((unsigned short int) ((((x) >> 8) & 0xffu) \ + | (((x) & 0xffu) << 8))) + +#if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) +#define expect(expr, value) (__builtin_expect((expr), (value))) +#else +#define expect(expr, value) (expr) +#endif + +#define likely(expr) expect((expr) != 0, 1) +#define unlikely(expr) expect((expr) != 0, 0) + +/* Basic types */ +#define BYTE uint8_t +#define U16 uint16_t +#define U32 uint32_t +#define S32 int32_t +#define U64 uint64_t + +typedef struct _U16_S { + U16 v; +} U16_S; +typedef struct _U32_S { + U32 v; +} U32_S; +typedef struct _U64_S { + U64 v; +} U64_S; + +#define A64(x) (((U64_S *)(x))->v) +#define A32(x) (((U32_S *)(x))->v) +#define A16(x) (((U16_S *)(x))->v) + +/* + * Constants + */ +#define MINMATCH 4 + +#define COPYLENGTH 8 +#define LASTLITERALS 5 + +#define ML_BITS 4 +#define ML_MASK ((1U<<ML_BITS)-1) +#define RUN_BITS (8-ML_BITS) +#define RUN_MASK ((1U<<RUN_BITS)-1) + +/* + * Architecture-specific macros + */ +#if LZ4_ARCH64 +#define STEPSIZE 8 +#define UARCH U64 +#define AARCH A64 +#define LZ4_COPYSTEP(s, d) A64(d) = A64(s); d += 8; s += 8; +#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d) +#define LZ4_SECURECOPY(s, d, e) if (d < e) LZ4_WILDCOPY(s, d, e) +#define HTYPE U32 +#define INITBASE(base) const BYTE* const base = ip +#else +#define STEPSIZE 4 +#define UARCH U32 +#define AARCH A32 +#define LZ4_COPYSTEP(s, d) A32(d) = A32(s); d += 4; s += 4; +#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d); LZ4_COPYSTEP(s, d); +#define LZ4_SECURECOPY LZ4_WILDCOPY +#define HTYPE const BYTE* +#define INITBASE(base) const int base = 0 +#endif + +#if (defined(LZ4_BIG_ENDIAN) && !defined(BIG_ENDIAN_NATIVE_BUT_INCOMPATIBLE)) +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \ + { U16 v = A16(p); v = lz4_bswap16(v); d = (s) - v; } +#define LZ4_WRITE_LITTLEENDIAN_16(p, i) \ + { U16 v = (U16)(i); v = lz4_bswap16(v); A16(p) = v; p += 2; } +#else +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) { d = (s) - A16(p); } +#define LZ4_WRITE_LITTLEENDIAN_16(p, v) { A16(p) = v; p += 2; } +#endif + +/* Macros */ +#define LZ4_WILDCOPY(s, d, e) do { LZ4_COPYPACKET(s, d) } while (d < e); + +/* Decompression functions */ + +static int +LZ4_uncompress_unknownOutputSize(const char *source, + char *dest, int isize, int maxOutputSize) +{ + /* Local Variables */ + const BYTE *restrict ip = (const BYTE *) source; + const BYTE *const iend = ip + isize; + const BYTE *restrict ref; + + BYTE *restrict op = (BYTE *) dest; + BYTE *const oend = op + maxOutputSize; + BYTE *cpy; + + size_t dec[] = { 0, 3, 2, 3, 0, 0, 0, 0 }; + + /* Main Loop */ + while (ip < iend) { + BYTE token; + int length; + + /* get runlength */ + token = *ip++; + if ((length = (token >> ML_BITS)) == RUN_MASK) { + int s = 255; + while ((ip < iend) && (s == 255)) { + s = *ip++; + length += s; + } + } + /* copy literals */ + cpy = op + length; + if ((cpy > oend - COPYLENGTH) || + (ip + length > iend - COPYLENGTH)) { + if (cpy > oend) + /* + * Error: request to write beyond destination + * buffer. + */ + goto _output_error; + if (ip + length > iend) + /* + * Error : request to read beyond source + * buffer. + */ + goto _output_error; + memcpy(op, ip, length); + op += length; + ip += length; + if (ip < iend) + /* Error : LZ4 format violation */ + goto _output_error; + /* Necessarily EOF, due to parsing restrictions. */ + break; + } + LZ4_WILDCOPY(ip, op, cpy); + ip -= (op - cpy); + op = cpy; + + /* get offset */ + LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip); + ip += 2; + if (ref < (BYTE * const) dest) + /* + * Error: offset creates reference outside of + * destination buffer. + */ + goto _output_error; + + /* get matchlength */ + if ((length = (token & ML_MASK)) == ML_MASK) { + while (ip < iend) { + int s = *ip++; + length += s; + if (s == 255) + continue; + break; + } + } + /* copy repeated sequence */ + if unlikely(op - ref < STEPSIZE) { +#if LZ4_ARCH64 + size_t dec2table[] = { 0, 0, 0, -1, 0, 1, 2, 3 }; + size_t dec2 = dec2table[op - ref]; +#else + const int dec2 = 0; +#endif + *op++ = *ref++; + *op++ = *ref++; + *op++ = *ref++; + *op++ = *ref++; + ref -= dec[op - ref]; + A32(op) = A32(ref); + op += STEPSIZE - 4; + ref -= dec2; + } else { + LZ4_COPYSTEP(ref, op); + } + cpy = op + length - (STEPSIZE - 4); + if (cpy > oend - COPYLENGTH) { + if (cpy > oend) + /* + * Error: request to write outside of + * destination buffer. + */ + goto _output_error; + LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH)); + while (op < cpy) + *op++ = *ref++; + op = cpy; + if (op == oend) + /* + * Check EOF (should never happen, since last + * 5 bytes are supposed to be literals). + */ + break; + continue; + } + LZ4_SECURECOPY(ref, op, cpy); + op = cpy; /* correction */ + } + + /* end of decoding */ + return (int)(((char *)op) - dest); + + /* write overflow error detected */ + _output_error: + return (int)(-(((char *)ip) - source)); +} Modified: user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfsimpl.h ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfsimpl.h Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfsimpl.h Sun Jan 20 08:26:09 2013 (r245686) @@ -52,6 +52,9 @@ * Copyright 2009 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ +/* + * Copyright 2013 by Saso Kiselkov. All rights reserved. + */ #define MAXNAMELEN 256 @@ -439,6 +442,7 @@ enum zio_compress { ZIO_COMPRESS_GZIP_8, ZIO_COMPRESS_GZIP_9, ZIO_COMPRESS_ZLE, + ZIO_COMPRESS_LZ4, ZIO_COMPRESS_FUNCTIONS }; Modified: user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfssubr.c ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfssubr.c Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/boot/zfs/zfssubr.c Sun Jan 20 08:26:09 2013 (r245686) @@ -119,6 +119,7 @@ typedef struct zio_compress_info { #include "lzjb.c" #include "zle.c" +#include "lz4.c" /* * Compression vectors. @@ -139,6 +140,7 @@ static zio_compress_info_t zio_compress_ {NULL, NULL, 8, "gzip-8"}, {NULL, NULL, 9, "gzip-9"}, {NULL, zle_decompress, 64, "zle"}, + {NULL, lz4_decompress, 0, "lz4"}, }; static void Modified: user/delphij/zfs-lz4/sys/cddl/compat/opensolaris/sys/byteorder.h ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/compat/opensolaris/sys/byteorder.h Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/compat/opensolaris/sys/byteorder.h Sun Jan 20 08:26:09 2013 (r245686) @@ -86,4 +86,6 @@ #define ntohll(x) BSWAP_64(x) #endif +#define BE_IN32(xa) htonl(*((uint32_t *)(void *)(xa))) + #endif /* _OPENSOLARIS_SYS_BYTEORDER_H_ */ Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.c Sun Jan 20 08:26:09 2013 (r245686) @@ -21,6 +21,7 @@ /* * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ #ifdef _KERNEL @@ -155,4 +156,7 @@ zpool_feature_init(void) zfeature_register(SPA_FEATURE_EMPTY_BPOBJ, "com.delphix:empty_bpobj", "empty_bpobj", "Snapshots use less space.", B_TRUE, B_FALSE, NULL); + zfeature_register(SPA_FEATURE_LZ4_COMPRESS, + "org.illumos:lz4_compress", "lz4_compress", + "LZ4 compression algorithm support.", B_FALSE, B_FALSE, NULL); } Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfeature_common.h Sun Jan 20 08:26:09 2013 (r245686) @@ -21,6 +21,7 @@ /* * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ #ifndef _ZFEATURE_COMMON_H @@ -51,6 +52,7 @@ typedef int (zfeature_func_t)(zfeature_i static enum spa_feature { SPA_FEATURE_ASYNC_DESTROY, SPA_FEATURE_EMPTY_BPOBJ, + SPA_FEATURE_LZ4_COMPRESS, SPA_FEATURES } spa_feature_t; Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfs_prop.c ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfs_prop.c Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/common/zfs/zfs_prop.c Sun Jan 20 08:26:09 2013 (r245686) @@ -21,6 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011 by Delphix. All rights reserved. + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ /* Portions Copyright 2010 Robert Milkowski */ @@ -96,6 +97,7 @@ zfs_prop_init(void) { "gzip-8", ZIO_COMPRESS_GZIP_8 }, { "gzip-9", ZIO_COMPRESS_GZIP_9 }, { "zle", ZIO_COMPRESS_ZLE }, + { "lz4", ZIO_COMPRESS_LZ4 }, { NULL } }; @@ -211,8 +213,8 @@ zfs_prop_init(void) zprop_register_index(ZFS_PROP_COMPRESSION, "compression", ZIO_COMPRESS_DEFAULT, PROP_INHERIT, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME, - "on | off | lzjb | gzip | gzip-[1-9] | zle", "COMPRESS", - compress_table); + "on | off | lzjb | gzip | gzip-[1-9] | zle | lz4", + "COMPRESS", compress_table); zprop_register_index(ZFS_PROP_SNAPDIR, "snapdir", ZFS_SNAPDIR_HIDDEN, PROP_INHERIT, ZFS_TYPE_FILESYSTEM, "hidden | visible", "SNAPDIR", snapdir_table); Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/Makefile.files ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/Makefile.files Sun Jan 20 08:26:09 2013 (r245686) @@ -22,6 +22,7 @@ # # Copyright (c) 1991, 2010, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2012 by Delphix. All rights reserved. +# Copyright (c) 2013 by Saso Kiselkov. All rights reserved. # # # This Makefile defines all file modules for the directory uts/common @@ -56,6 +57,7 @@ ZFS_COMMON_OBJS += \ dsl_scan.o \ zfeature.o \ gzip.o \ + lz4.o \ lzjb.o \ metaslab.o \ refcount.o \ Copied: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4 (from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4 Sun Jan 20 08:26:09 2013 (r245686, copy of r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4) @@ -0,0 +1,30 @@ +LZ4 - Fast LZ compression algorithm +Copyright (C) 2011-2013, Yann Collet. +BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +You can contact the author at : +- LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html +- LZ4 source repository : http://code.google.com/p/lz4/ Copied: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip (from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip Sun Jan 20 08:26:09 2013 (r245686, copy of r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/THIRDPARTYLICENSE.lz4.descrip) @@ -0,0 +1 @@ +LZ4 COMPRESSION FUNCTIONALITY IN ZFS Copied and modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lz4.c (from r245512, vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c) ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c Wed Jan 16 23:11:13 2013 (r245512, copy source) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lz4.c Sun Jan 20 08:26:09 2013 (r245686) @@ -201,8 +201,13 @@ lz4_decompress(void *s_start, void *d_st * Illumos: On amd64 we have 20k of stack and 24k on sun4u and sun4v, so we * can spend 16k on the algorithm */ +#if notyet #define STACKLIMIT 12 #else +/* XXX need larger stack size on FreeBSD for zio threads */ +#define STACKLIMIT 11 +#endif +#else #define LZ4_ARCH64 0 /* * Illumos: On i386 we only have 12k of stack, so in order to maintain the Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h Sun Jan 20 08:26:09 2013 (r245686) @@ -22,6 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ #ifndef _ZIO_H @@ -106,14 +107,17 @@ enum zio_compress { ZIO_COMPRESS_GZIP_8, ZIO_COMPRESS_GZIP_9, ZIO_COMPRESS_ZLE, + ZIO_COMPRESS_LZ4, ZIO_COMPRESS_FUNCTIONS }; +/* N.B. when altering this value, also change BOOTFS_COMPRESS_VALID below */ #define ZIO_COMPRESS_ON_VALUE ZIO_COMPRESS_LZJB #define ZIO_COMPRESS_DEFAULT ZIO_COMPRESS_OFF #define BOOTFS_COMPRESS_VALID(compress) \ ((compress) == ZIO_COMPRESS_LZJB || \ + (compress) == ZIO_COMPRESS_LZ4 || \ ((compress) == ZIO_COMPRESS_ON && \ ZIO_COMPRESS_ON_VALUE == ZIO_COMPRESS_LZJB) || \ (compress) == ZIO_COMPRESS_OFF) Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h Sun Jan 20 08:26:09 2013 (r245686) @@ -23,6 +23,9 @@ * Copyright 2009 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ +/* + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. + */ #ifndef _SYS_ZIO_COMPRESS_H #define _SYS_ZIO_COMPRESS_H @@ -68,6 +71,10 @@ extern size_t zle_compress(void *src, vo int level); extern int zle_decompress(void *src, void *dst, size_t s_len, size_t d_len, int level); +extern size_t lz4_compress(void *src, void *dst, size_t s_len, size_t d_len, + int level); +extern int lz4_decompress(void *src, void *dst, size_t s_len, size_t d_len, + int level); /* * Compress and decompress data if necessary. Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c Sun Jan 20 08:26:09 2013 (r245686) @@ -27,6 +27,7 @@ * Copyright 2011 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2012 by Delphix. All rights reserved. * Copyright (c) 2012, Joyent, Inc. All rights reserved. + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. */ #include <sys/types.h> @@ -75,6 +76,7 @@ #include <sys/zvol.h> #include <sys/dsl_scan.h> #include <sys/dmu_objset.h> +#include <sys/zfeature.h> #include "zfs_namecheck.h" #include "zfs_prop.h" @@ -131,6 +133,12 @@ int zfs_set_prop_nvlist(const char *, zp static void zfsdev_close(void *data); +static int zfs_prop_activate_feature(dsl_pool_t *dp, zfeature_info_t *feature); +static int zfs_prop_activate_feature_check(void *arg1, void *arg2, + dmu_tx_t *tx); +static void zfs_prop_activate_feature_sync(void *arg1, void *arg2, + dmu_tx_t *tx); + /* _NOTE(PRINTFLIKE(4)) - this is printf-like, but lint is too whiney */ void __dprintf(const char *file, const char *func, int line, const char *fmt, ...) @@ -2264,6 +2272,40 @@ zfs_prop_set_special(const char *dsname, } break; } + case ZFS_PROP_COMPRESSION: + { + if (intval == ZIO_COMPRESS_LZ4) { + zfeature_info_t *feature = + &spa_feature_table[SPA_FEATURE_LZ4_COMPRESS]; + spa_t *spa; + dsl_pool_t *dp; + + if ((err = spa_open(dsname, &spa, FTAG)) != 0) + return (err); + + dp = spa->spa_dsl_pool; + + /* + * Setting the LZ4 compression algorithm activates + * the feature. + */ + if (!spa_feature_is_active(spa, feature)) { + if ((err = zfs_prop_activate_feature(dp, + feature)) != 0) { + spa_close(spa, FTAG); + return (err); + } + } + + spa_close(spa, FTAG); + } + /* + * We still want the default set action to be performed in the + * caller, we only performed zfeature settings here. + */ + err = -1; + break; + } default: err = -1; @@ -3481,6 +3523,22 @@ zfs_check_settable(const char *dsname, n SPA_VERSION_ZLE_COMPRESSION)) return (ENOTSUP); + if (intval == ZIO_COMPRESS_LZ4) { + zfeature_info_t *feature = + &spa_feature_table[ + SPA_FEATURE_LZ4_COMPRESS]; + spa_t *spa; + + if ((err = spa_open(dsname, &spa, FTAG)) != 0) + return (err); + + if (!spa_feature_is_enabled(spa, feature)) { + spa_close(spa, FTAG); + return (ENOTSUP); + } + spa_close(spa, FTAG); + } + /* * If this is a bootable dataset then * verify that the compression algorithm @@ -3525,6 +3583,56 @@ zfs_check_settable(const char *dsname, n } /* + * Activates a feature on a pool in response to a property setting. This + * creates a new sync task which modifies the pool to reflect the feature + * as being active. + */ +static int +zfs_prop_activate_feature(dsl_pool_t *dp, zfeature_info_t *feature) +{ + int err; + + /* EBUSY here indicates that the feature is already active */ + err = dsl_sync_task_do(dp, zfs_prop_activate_feature_check, + zfs_prop_activate_feature_sync, dp->dp_spa, feature, 2); + + if (err != 0 && err != EBUSY) + return (err); + else + return (0); +} + +/* + * Checks for a race condition to make sure we don't increment a feature flag + * multiple times. + */ +/*ARGSUSED*/ +static int +zfs_prop_activate_feature_check(void *arg1, void *arg2, dmu_tx_t *tx) +{ + spa_t *spa = arg1; + zfeature_info_t *feature = arg2; + + if (!spa_feature_is_active(spa, feature)) + return (0); + else + return (EBUSY); +} + +/* + * The callback invoked on feature activation in the sync task caused by + * zfs_prop_activate_feature. + */ +static void +zfs_prop_activate_feature_sync(void *arg1, void *arg2, dmu_tx_t *tx) +{ + spa_t *spa = arg1; + zfeature_info_t *feature = arg2; + + spa_feature_incr(spa, feature, tx); +} + +/* * Removes properties from the given props list that fail permission checks * needed to clear them and to restore them in case of a receive error. For each * property, make sure we have both set and inherit permissions. Modified: user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c ============================================================================== --- user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Sun Jan 20 08:18:56 2013 (r245685) +++ user/delphij/zfs-lz4/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c Sun Jan 20 08:26:09 2013 (r245686) @@ -23,6 +23,9 @@ * Copyright 2009 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ +/* + * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. + */ #include <sys/zfs_context.h> #include <sys/compress.h> @@ -50,6 +53,7 @@ zio_compress_info_t zio_compress_table[Z {gzip_compress, gzip_decompress, 8, "gzip-8"}, {gzip_compress, gzip_decompress, 9, "gzip-9"}, {zle_compress, zle_decompress, 64, "zle"}, + {lz4_compress, lz4_decompress, 0, "lz4"}, }; enum zio_compress
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201301200826.r0K8QAiw090246>