Date: Sat, 29 Dec 2018 06:51:10 +0000 (UTC) From: Conrad Meyer <cem@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342589 - in vendor/zstd/dist: . contrib/adaptive-compression contrib/meson contrib/pzstd contrib/seekable_format contrib/seekable_format/examples doc lib lib/common lib/compress lib/de... Message-ID: <201812290651.wBT6pAxw006140@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: cem Date: Sat Dec 29 06:51:10 2018 New Revision: 342589 URL: https://svnweb.freebsd.org/changeset/base/342589 Log: import zstd 1.3.8 Added: vendor/zstd/dist/lib/decompress/zstd_ddict.c (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_ddict.h (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_block.c (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_block.h (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h (contents, props changed) vendor/zstd/dist/programs/benchfn.c (contents, props changed) vendor/zstd/dist/programs/benchfn.h (contents, props changed) vendor/zstd/dist/programs/benchzstd.c (contents, props changed) vendor/zstd/dist/programs/benchzstd.h (contents, props changed) vendor/zstd/dist/programs/util.c (contents, props changed) vendor/zstd/dist/tests/regression/ vendor/zstd/dist/tests/regression/Makefile (contents, props changed) vendor/zstd/dist/tests/regression/config.c (contents, props changed) vendor/zstd/dist/tests/regression/config.h (contents, props changed) vendor/zstd/dist/tests/regression/data.c (contents, props changed) vendor/zstd/dist/tests/regression/data.h (contents, props changed) vendor/zstd/dist/tests/regression/levels.h (contents, props changed) vendor/zstd/dist/tests/regression/method.c (contents, props changed) vendor/zstd/dist/tests/regression/method.h (contents, props changed) vendor/zstd/dist/tests/regression/result.c (contents, props changed) vendor/zstd/dist/tests/regression/result.h (contents, props changed) vendor/zstd/dist/tests/regression/results.csv vendor/zstd/dist/tests/regression/test.c (contents, props changed) Deleted: vendor/zstd/dist/NEWS vendor/zstd/dist/contrib/meson/ vendor/zstd/dist/programs/bench.c vendor/zstd/dist/programs/bench.h Modified: vendor/zstd/dist/Makefile vendor/zstd/dist/README.md vendor/zstd/dist/appveyor.yml vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/dist/contrib/adaptive-compression/datagencli.c vendor/zstd/dist/contrib/pzstd/Makefile vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c vendor/zstd/dist/doc/zstd_compression_format.md vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/dist/lib/BUCK vendor/zstd/dist/lib/Makefile vendor/zstd/dist/lib/README.md vendor/zstd/dist/lib/common/bitstream.h vendor/zstd/dist/lib/common/compiler.h vendor/zstd/dist/lib/common/cpu.h vendor/zstd/dist/lib/common/debug.h vendor/zstd/dist/lib/common/error_private.c vendor/zstd/dist/lib/common/fse.h vendor/zstd/dist/lib/common/huf.h vendor/zstd/dist/lib/common/pool.c vendor/zstd/dist/lib/common/zstd_common.c vendor/zstd/dist/lib/common/zstd_errors.h vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/dist/lib/compress/fse_compress.c vendor/zstd/dist/lib/compress/hist.c vendor/zstd/dist/lib/compress/hist.h vendor/zstd/dist/lib/compress/huf_compress.c vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/dist/lib/compress/zstd_double_fast.c vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/dist/lib/compress/zstd_lazy.c vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/dist/lib/compress/zstd_ldm.h vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/dist/lib/compress/zstd_opt.h vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/dist/lib/compress/zstdmt_compress.h vendor/zstd/dist/lib/decompress/huf_decompress.c vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/dist/lib/dictBuilder/fastcover.c vendor/zstd/dist/lib/dictBuilder/zdict.c vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/dist/lib/zstd.h vendor/zstd/dist/programs/Makefile vendor/zstd/dist/programs/README.md vendor/zstd/dist/programs/datagen.c vendor/zstd/dist/programs/dibio.c vendor/zstd/dist/programs/fileio.c vendor/zstd/dist/programs/fileio.h vendor/zstd/dist/programs/platform.h vendor/zstd/dist/programs/util.h vendor/zstd/dist/programs/zstd.1 vendor/zstd/dist/programs/zstd.1.md vendor/zstd/dist/programs/zstdcli.c vendor/zstd/dist/programs/zstdgrep vendor/zstd/dist/programs/zstdgrep.1 vendor/zstd/dist/programs/zstdless.1 vendor/zstd/dist/tests/Makefile vendor/zstd/dist/tests/README.md vendor/zstd/dist/tests/datagencli.c vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/dist/tests/fullbench.c vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/dist/tests/fuzzer.c vendor/zstd/dist/tests/libzstd_partial_builds.sh vendor/zstd/dist/tests/longmatch.c vendor/zstd/dist/tests/paramgrill.c vendor/zstd/dist/tests/playTests.sh vendor/zstd/dist/tests/poolTests.c vendor/zstd/dist/tests/roundTripCrash.c vendor/zstd/dist/tests/seqgen.c vendor/zstd/dist/tests/symbols.c vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/dist/zlibWrapper/zstd_zlibwrapper.c Modified: vendor/zstd/dist/Makefile ============================================================================== --- vendor/zstd/dist/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -64,7 +64,8 @@ zlibwrapper: lib ## test: run long-duration tests .PHONY: test -test: MOREFLAGS += -g -DDEBUGLEVEL=1 -Werror +DEBUGLEVEL ?= 1 +test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror test: MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants $(MAKE) -C $(TESTDIR) $@ @@ -129,7 +130,12 @@ ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFree HOST_OS = POSIX CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release -EGREP = egrep --color=never +HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0) +EGREP_OPTIONS ?= +ifeq ($HAVE_COLORNEVER, 1) +EGREP_OPTIONS += --color=never +endif +EGREP = egrep $(EGREP_OPTIONS) # Print a two column output of targets and their description. To add a target description, put a # comment in the Makefile with the format "## <TARGET>: <DESCRIPTION>". For example: Modified: vendor/zstd/dist/README.md ============================================================================== --- vendor/zstd/dist/README.md Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/README.md Sat Dec 29 06:51:10 2018 (r342589) @@ -9,8 +9,12 @@ and a command line utility producing and decoding `.zs Should your project require another programming language, a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages). -Development branch status : [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] +**Development branch status:** +[![Build Status][travisDevBadge]][travisLink] +[![Build status][AppveyorDevBadge]][AppveyorLink] +[![Build status][CircleDevBadge]][CircleLink] + [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" [travisLink]: https://travis-ci.org/facebook/zstd [AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite" @@ -18,7 +22,7 @@ Development branch status : [![Build Status][travisDev [CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" [CircleLink]: https://circleci.com/gh/facebook/zstd -### Benchmarks +## Benchmarks For reference, several fast compression algorithms were tested and compared on a server running Linux Debian (`Linux version 4.14.0-3-amd64`), @@ -42,7 +46,7 @@ on the [Silesia compression corpus]. | snappy 1.1.4 | 2.091 | 530 MB/s | 1800 MB/s | | lzf 3.6 -1 | 2.077 | 400 MB/s | 860 MB/s | -[zlib]:http://www.zlib.net/ +[zlib]: http://www.zlib.net/ [LZ4]: http://www.lz4.org/ Zstd can also offer stronger compression ratios at the cost of compression speed. @@ -65,7 +69,7 @@ A few other algorithms can produce higher compression For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png). -### The case for Small Data compression +## The case for Small Data compression Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives. @@ -89,24 +93,24 @@ Training works if there is some correlation in a famil Hence, deploying one dictionary per type of data will provide the greatest benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. -#### Dictionary compression How To: +### Dictionary compression How To: -1) Create the dictionary +1. Create the dictionary -`zstd --train FullPathToTrainingSet/* -o dictionaryName` + `zstd --train FullPathToTrainingSet/* -o dictionaryName` -2) Compress with dictionary +2. Compress with dictionary -`zstd -D dictionaryName FILE` + `zstd -D dictionaryName FILE` -3) Decompress with dictionary +3. Decompress with dictionary -`zstd -D dictionaryName --decompress FILE.zst` + `zstd -D dictionaryName --decompress FILE.zst` -### Build instructions +## Build instructions -#### Makefile +### Makefile If your system is compatible with standard `make` (or `gmake`), invoking `make` in root directory will generate `zstd` cli in root directory. @@ -115,7 +119,7 @@ Other available options include: - `make install` : create and install zstd cli, library and man pages - `make check` : create and run `zstd`, tests its behavior on local platform -#### cmake +### cmake A `cmake` project generator is provided within `build/cmake`. It can generate Makefiles or other build scripts @@ -123,29 +127,39 @@ to create `zstd` binary, and `libzstd` dynamic and sta By default, `CMAKE_BUILD_TYPE` is set to `Release`. -#### Meson +### Meson -A Meson project is provided within `contrib/meson`. +A Meson project is provided within [`build/meson`](build/meson). Follow +build instructions in that directory. -#### Visual Studio (Windows) +You can also take a look at [`.travis.yml`](.travis.yml) file for an +example about how Meson is used to build this project. +Note that default build type is **release**. + +### Visual Studio (Windows) + Going into `build` directory, you will find additional possibilities: - Projects for Visual Studio 2005, 2008 and 2010. + VS2010 project is compatible with VS2012, VS2013, VS2015 and VS2017. - Automated build scripts for Visual compiler by [@KrzysFR](https://github.com/KrzysFR), in `build/VS_scripts`, which will build `zstd` cli and `libzstd` library without any need to open Visual Studio solution. +### Buck -### Status +You can build the zstd binary via buck by executing: `buck build programs:zstd` from the root of the repo. +The output binary will be in `buck-out/gen/programs/`. +## Status + Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases. Zstandard is considered safe for production environments. -### License +## License Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING). -### Contributing +## Contributing The "dev" branch is the one where all contributions are merged before reaching "master". If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch. Modified: vendor/zstd/dist/appveyor.yml ============================================================================== --- vendor/zstd/dist/appveyor.yml Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/appveyor.yml Sat Dec 29 06:51:10 2018 (r342589) @@ -3,6 +3,8 @@ branches: only: - master + - appveyorTest + - /visual*/ environment: matrix: - COMPILER: "gcc" Modified: vendor/zstd/dist/contrib/adaptive-compression/Makefile ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -22,10 +22,10 @@ FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITH all: adapt datagen -adapt: $(ZSTD_FILES) adapt.c +adapt: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c $(CC) $(FLAGS) $^ -o $@ -adapt-debug: $(ZSTD_FILES) adapt.c +adapt-debug: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c $(CC) $(FLAGS) -DDEBUG_MODE=2 $^ -o adapt datagen : $(PRGDIR)/datagen.c datagencli.c Modified: vendor/zstd/dist/contrib/adaptive-compression/adapt.c ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/adapt.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/adapt.c Sat Dec 29 06:51:10 2018 (r342589) @@ -579,7 +579,7 @@ static void* compressionThread(void* arg) params.cParams.windowLog = 23; { size_t const initError = ZSTD_compressBegin_advanced(ctx->cctx, job->src.start + job->dictSize - useDictSize, useDictSize, params, 0); - size_t const windowSizeError = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_p_forceMaxWindow, 1); + size_t const windowSizeError = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_c_forceMaxWindow, 1); if (ZSTD_isError(initError) || ZSTD_isError(windowSizeError)) { DISPLAY("Error: something went wrong while starting compression\n"); signalErrorToThreads(ctx); Modified: vendor/zstd/dist/contrib/adaptive-compression/datagencli.c ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/datagencli.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/datagencli.c Sat Dec 29 06:51:10 2018 (r342589) @@ -120,7 +120,7 @@ int main(int argc, const char** argv) DISPLAYLEVEL(4, "Compressible data Generator \n"); if (probaU32!=COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", probaU32); - DISPLAYLEVEL(3, "Seed = %u \n", seed); + DISPLAYLEVEL(3, "Seed = %u \n", (unsigned)seed); RDG_genStdout(size, (double)probaU32/100, litProba, seed); DISPLAYLEVEL(1, "\n"); Modified: vendor/zstd/dist/contrib/pzstd/Makefile ============================================================================== --- vendor/zstd/dist/contrib/pzstd/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/pzstd/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -171,7 +171,7 @@ roundtripcheck: roundtrip check $(TESTPROG) ./test/RoundTripTest$(EXT) $(TESTFLAGS) # Build the main binary -pzstd$(EXT): main.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a +pzstd$(EXT): main.o $(PROGDIR)/util.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a $(LD_COMMAND) # Target that depends on all the tests Modified: vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c Sat Dec 29 06:51:10 2018 (r342589) @@ -148,20 +148,20 @@ static void sumFile_orDie(const char* fname, int nbThr size_t const initResult = ZSTD_seekable_initFile(seekable, fin); if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); } - size_t const numFrames = ZSTD_seekable_getNumFrames(seekable); + unsigned const numFrames = ZSTD_seekable_getNumFrames(seekable); struct sum_job* jobs = (struct sum_job*)malloc(numFrames * sizeof(struct sum_job)); - size_t i; - for (i = 0; i < numFrames; i++) { - jobs[i] = (struct sum_job){ fname, 0, i, 0 }; - POOL_add(pool, sumFrame, &jobs[i]); + unsigned fnb; + for (fnb = 0; fnb < numFrames; fnb++) { + jobs[fnb] = (struct sum_job){ fname, 0, fnb, 0 }; + POOL_add(pool, sumFrame, &jobs[fnb]); } unsigned long long total = 0; - for (i = 0; i < numFrames; i++) { - while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to check */ - total += jobs[i].sum; + for (fnb = 0; fnb < numFrames; fnb++) { + while (!jobs[fnb].done) SLEEP(5); /* wake up every 5 milliseconds to check */ + total += jobs[fnb].sum; } printf("Sum: %llu\n", total); Modified: vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c Sat Dec 29 06:51:10 2018 (r342589) @@ -8,6 +8,8 @@ */ #include <stdlib.h> /* malloc, free */ +#include <limits.h> /* UINT_MAX */ +#include <assert.h> #define XXH_STATIC_LINKING_ONLY #define XXH_NAMESPACE ZSTD_ @@ -139,7 +141,7 @@ size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs, int compressionLevel, int checksumFlag, - U32 maxFrameSize) + unsigned maxFrameSize) { zcs->framelog.size = 0; zcs->frameCSize = 0; @@ -167,9 +169,9 @@ size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream } size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, - unsigned compressedSize, - unsigned decompressedSize, - unsigned checksum) + unsigned compressedSize, + unsigned decompressedSize, + unsigned checksum) { if (fl->size == ZSTD_SEEKABLE_MAXFRAMES) return ERROR(frameIndex_tooLarge); @@ -184,7 +186,8 @@ size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, if (newEntries == NULL) return ERROR(memory_allocation); fl->entries = newEntries; - fl->capacity = newCapacity; + assert(newCapacity <= UINT_MAX); + fl->capacity = (U32)newCapacity; } fl->entries[fl->size] = (framelogEntry_t){ @@ -268,7 +271,7 @@ size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStr static inline size_t ZSTD_seekable_seekTableSize(const ZSTD_frameLog* fl) { size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0); - size_t const seekTableLen = ZSTD_skippableHeaderSize + + size_t const seekTableLen = ZSTD_SKIPPABLEHEADERSIZE + sizePerFrame * fl->size + ZSTD_seekTableFooterSize; @@ -307,32 +310,32 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, size_t const seekTableLen = ZSTD_seekable_seekTableSize(fl); CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_MAGIC_SKIPPABLE_START | 0xE, 0)); - CHECK_Z(ZSTD_stwrite32(fl, output, seekTableLen - ZSTD_skippableHeaderSize, - 4)); + assert(seekTableLen <= (size_t)UINT_MAX); + CHECK_Z(ZSTD_stwrite32(fl, output, (U32)seekTableLen - ZSTD_SKIPPABLEHEADERSIZE, 4)); while (fl->seekTableIndex < fl->size) { + unsigned long long const start = ZSTD_SKIPPABLEHEADERSIZE + sizePerFrame * fl->seekTableIndex; + assert(start + 8 <= UINT_MAX); CHECK_Z(ZSTD_stwrite32(fl, output, fl->entries[fl->seekTableIndex].cSize, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 0)); + (U32)start + 0)); CHECK_Z(ZSTD_stwrite32(fl, output, fl->entries[fl->seekTableIndex].dSize, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 4)); + (U32)start + 4)); if (fl->checksumFlag) { CHECK_Z(ZSTD_stwrite32( fl, output, fl->entries[fl->seekTableIndex].checksum, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 8)); + (U32)start + 8)); } fl->seekTableIndex++; } + assert(seekTableLen <= UINT_MAX); CHECK_Z(ZSTD_stwrite32(fl, output, fl->size, - seekTableLen - ZSTD_seekTableFooterSize)); + (U32)seekTableLen - ZSTD_seekTableFooterSize)); if (output->size - output->pos < 1) return seekTableLen - fl->seekTablePos; if (fl->seekTablePos < seekTableLen - 4) { @@ -345,7 +348,7 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, } CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_SEEKABLE_MAGICNUMBER, - seekTableLen - 4)); + (U32)seekTableLen - 4)); if (fl->seekTablePos != seekTableLen) return ERROR(GENERIC); return 0; Modified: vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c Sat Dec 29 06:51:10 2018 (r342589) @@ -54,8 +54,9 @@ # define LONG_SEEK fseek #endif -#include <stdlib.h> /* malloc, free */ -#include <stdio.h> /* FILE* */ +#include <stdlib.h> /* malloc, free */ +#include <stdio.h> /* FILE* */ +#include <limits.h> /* UNIT_MAX */ #include <assert.h> #define XXH_STATIC_LINKING_ONLY @@ -200,13 +201,14 @@ size_t ZSTD_seekable_free(ZSTD_seekable* zs) * Performs a binary search to find the last frame with a decompressed offset * <= pos * @return : the frame's index */ -U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long pos) +unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long pos) { U32 lo = 0; - U32 hi = zs->seekTable.tableLen; + U32 hi = (U32)zs->seekTable.tableLen; + assert(zs->seekTable.tableLen <= UINT_MAX); if (pos >= zs->seekTable.entries[zs->seekTable.tableLen].dOffset) { - return zs->seekTable.tableLen; + return (U32)zs->seekTable.tableLen; } while (lo + 1 < hi) { @@ -220,31 +222,32 @@ U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* co return lo; } -U32 ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs) +unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs) { - return zs->seekTable.tableLen; + assert(zs->seekTable.tableLen <= UINT_MAX); + return (unsigned)zs->seekTable.tableLen; } -unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, U32 frameIndex) +unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE; return zs->seekTable.entries[frameIndex].cOffset; } -unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, U32 frameIndex) +unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE; return zs->seekTable.entries[frameIndex].dOffset; } -size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, U32 frameIndex) +size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge); return zs->seekTable.entries[frameIndex + 1].cOffset - zs->seekTable.entries[frameIndex].cOffset; } -size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, U32 frameIndex) +size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex > zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge); return zs->seekTable.entries[frameIndex + 1].dOffset - @@ -275,7 +278,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl { U32 const numFrames = MEM_readLE32(zs->inBuff); U32 const sizePerEntry = 8 + (checksumFlag?4:0); U32 const tableSize = sizePerEntry * numFrames; - U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_skippableHeaderSize; + U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_SKIPPABLEHEADERSIZE; U32 remaining = frameSize - ZSTD_seekTableFooterSize; /* don't need to re-read footer */ { @@ -290,7 +293,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl if (MEM_readLE32(zs->inBuff) != (ZSTD_MAGIC_SKIPPABLE_START | 0xE)) { return ERROR(prefix_unknown); } - if (MEM_readLE32(zs->inBuff+4) + ZSTD_skippableHeaderSize != frameSize) { + if (MEM_readLE32(zs->inBuff+4) + ZSTD_SKIPPABLEHEADERSIZE != frameSize) { return ERROR(prefix_unknown); } @@ -444,7 +447,7 @@ size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, voi return len; } -size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, U32 frameIndex) +size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) { return ERROR(frameIndex_tooLarge); Modified: vendor/zstd/dist/doc/zstd_compression_format.md ============================================================================== --- vendor/zstd/dist/doc/zstd_compression_format.md Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/doc/zstd_compression_format.md Sat Dec 29 06:51:10 2018 (r342589) @@ -16,7 +16,7 @@ Distribution of this document is unlimited. ### Version -0.3.0 (25/09/18) +0.3.1 (25/10/18) Introduction @@ -913,13 +913,39 @@ Note that blocks which are not `Compressed_Block` are ###### Offset updates rules The newest offset takes the lead in offset history, -shifting others back (up to its previous place if it was already present). +shifting others back by one rank, +up to the previous rank of the new offset _if it was present in history_. -This means that when `Repeated_Offset1` (most recent) is used, history is unmodified. -When `Repeated_Offset2` is used, it's swapped with `Repeated_Offset1`. -If any other offset is used, it becomes `Repeated_Offset1` and the rest are shift back by one. +__Examples__ : +In the common case, when new offset is not part of history : +`Repeated_Offset3` = `Repeated_Offset2` +`Repeated_Offset2` = `Repeated_Offset1` +`Repeated_Offset1` = `NewOffset` +When the new offset _is_ part of history, there may be specific adjustments. + +When `NewOffset` == `Repeated_Offset1`, offset history remains actually unmodified. + +When `NewOffset` == `Repeated_Offset2`, +`Repeated_Offset1` and `Repeated_Offset2` ranks are swapped. +`Repeated_Offset3` is unmodified. + +When `NewOffset` == `Repeated_Offset3`, +there is actually no difference with the common case : +all offsets are shifted by one rank, +`NewOffset` (== `Repeated_Offset3`) becomes the new `Repeated_Offset1`. + +Also worth mentioning, the specific corner case when `offset_value` == 3, +and the literal length of the current sequence is zero. +In which case , `NewOffset` = `Repeated_Offset1` - 1_byte. +Here also, from an offset history update perspective, it's just a common case : +`Repeated_Offset3` = `Repeated_Offset2` +`Repeated_Offset2` = `Repeated_Offset1` +`Repeated_Offset1` = `NewOffset` ( == `Repeated_Offset1` - 1_byte ) + + + Skippable Frames ---------------- @@ -1629,6 +1655,7 @@ or at least provide a meaningful error code explaining Version changes --------------- +- 0.3.1 : minor clarification regarding offset history update rules - 0.3.0 : minor edits to match RFC8478 - 0.2.9 : clarifications for huffman weights direct representation, by Ulrich Kunitz - 0.2.8 : clarifications for IETF RFC discuss Modified: vendor/zstd/dist/doc/zstd_manual.html ============================================================================== --- vendor/zstd/dist/doc/zstd_manual.html Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/doc/zstd_manual.html Sat Dec 29 06:51:10 2018 (r342589) @@ -1,10 +1,10 @@ <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> -<title>zstd 1.3.7 Manual</title> +<title>zstd 1.3.8 Manual</title> </head> <body> -<h1>zstd 1.3.7 Manual</h1> +<h1>zstd 1.3.8 Manual</h1> <hr> <a name="Contents"></a><h2>Contents</h2> <ol> @@ -19,16 +19,19 @@ <li><a href="#Chapter9">Streaming compression - HowTo</a></li> <li><a href="#Chapter10">Streaming decompression - HowTo</a></li> <li><a href="#Chapter11">ADVANCED AND EXPERIMENTAL FUNCTIONS</a></li> -<li><a href="#Chapter12">Frame size functions</a></li> -<li><a href="#Chapter13">Memory management</a></li> -<li><a href="#Chapter14">Advanced compression functions</a></li> -<li><a href="#Chapter15">Advanced decompression functions</a></li> -<li><a href="#Chapter16">Advanced streaming functions</a></li> -<li><a href="#Chapter17">Buffer-less and synchronous inner streaming functions</a></li> -<li><a href="#Chapter18">Buffer-less streaming compression (synchronous mode)</a></li> -<li><a href="#Chapter19">Buffer-less streaming decompression (synchronous mode)</a></li> -<li><a href="#Chapter20">New advanced API (experimental)</a></li> -<li><a href="#Chapter21">Block level API</a></li> +<li><a href="#Chapter12">Candidate API for promotion to stable status</a></li> +<li><a href="#Chapter13">Advanced compression API</a></li> +<li><a href="#Chapter14">experimental API (static linking only)</a></li> +<li><a href="#Chapter15">Frame size functions</a></li> +<li><a href="#Chapter16">Memory management</a></li> +<li><a href="#Chapter17">Advanced compression functions</a></li> +<li><a href="#Chapter18">Advanced decompression functions</a></li> +<li><a href="#Chapter19">Advanced streaming functions</a></li> +<li><a href="#Chapter20">Buffer-less and synchronous inner streaming functions</a></li> +<li><a href="#Chapter21">Buffer-less streaming compression (synchronous mode)</a></li> +<li><a href="#Chapter22">Buffer-less streaming decompression (synchronous mode)</a></li> +<li><a href="#Chapter23">ZSTD_getFrameHeader() :</a></li> +<li><a href="#Chapter24">Block level API</a></li> </ol> <hr> <a name="Chapter1"></a><h2>Introduction</h2><pre> @@ -64,7 +67,7 @@ <a name="Chapter2"></a><h2>Version</h2><pre></pre> -<pre><b>unsigned ZSTD_versionNumber(void); </b>/**< useful to check dll version */<b> +<pre><b>unsigned ZSTD_versionNumber(void); </b>/**< to check runtime library version */<b> </b></pre><BR> <a name="Chapter3"></a><h2>Default constant</h2><pre></pre> @@ -139,11 +142,13 @@ int ZSTD_maxCLevel(void); </b>/* ZSTD_CCtx* ZSTD_createCCtx(void); size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); </pre></b><BR> -<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx, +<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); -</b><p> Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx()). +</b><p> Same as ZSTD_compress(), using an explicit ZSTD_CCtx + The function will compress at requested compression level, + ignoring any other parameter </p></pre><BR> <h3>Decompression context</h3><pre> When decompressing many times, @@ -155,10 +160,13 @@ size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); ZSTD_DCtx* ZSTD_createDCtx(void); size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); </pre></b><BR> -<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* ctx, +<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); -</b><p> Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx()) +</b><p> Same as ZSTD_decompress(), + requires an allocated ZSTD_DCtx. + Compatible with sticky parameters. + </p></pre><BR> <a name="Chapter6"></a><h2>Simple dictionary API</h2><pre></pre> @@ -168,18 +176,22 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); const void* src, size_t srcSize, const void* dict,size_t dictSize, int compressionLevel); -</b><p> Compression using a predefined Dictionary (see dictBuilder/zdict.h). +</b><p> Compression at an explicit compression level using a Dictionary. + A dictionary can be any arbitrary data segment (also called a prefix), + or a buffer with specified information (see dictBuilder/zdict.h). Note : This function loads the dictionary, resulting in significant startup delay. - Note : When `dict == NULL || dictSize < 8` no dictionary is used. + It's intended for a dictionary used only once. + Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. </p></pre><BR> <pre><b>size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize); -</b><p> Decompression using a predefined Dictionary (see dictBuilder/zdict.h). +</b><p> Decompression using a known Dictionary. Dictionary must be identical to the one used during compression. Note : This function loads the dictionary, resulting in significant startup delay. + It's intended for a dictionary used only once. Note : When `dict == NULL || dictSize < 8` no dictionary is used. </p></pre><BR> @@ -187,11 +199,12 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); <pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize, int compressionLevel); -</b><p> When compressing multiple messages / blocks with the same dictionary, it's recommended to load it just once. - ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup delay. +</b><p> When compressing multiple messages / blocks using the same dictionary, it's recommended to load it only once. + ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup cost. ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. - `dictBuffer` can be released after ZSTD_CDict creation, since its content is copied within CDict - Note : A ZSTD_CDict can be created with an empty dictionary, but it is inefficient for small data. + `dictBuffer` can be released after ZSTD_CDict creation, because its content is copied within CDict. + Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate `dictBuffer` content. + Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is inefficient when used to compress small data. </p></pre><BR> <pre><b>size_t ZSTD_freeCDict(ZSTD_CDict* CDict); @@ -203,16 +216,14 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); const void* src, size_t srcSize, const ZSTD_CDict* cdict); </b><p> Compression using a digested Dictionary. - Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. - Note that compression level is decided during dictionary creation. - Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) - Note : ZSTD_compress_usingCDict() can be used with a ZSTD_CDict created from an empty dictionary. - But it is inefficient for small data, and it is recommended to use ZSTD_compressCCtx(). + Recommended when same dictionary is used multiple times. + Note : compression level is _decided at dictionary creation time_, + and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) </p></pre><BR> <pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize); </b><p> Create a digested dictionary, ready to start decompression operation without startup delay. - dictBuffer can be released after DDict creation, as its content is copied inside DDict + dictBuffer can be released after DDict creation, as its content is copied inside DDict. </p></pre><BR> <pre><b>size_t ZSTD_freeDDict(ZSTD_DDict* ddict); @@ -224,7 +235,7 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); const void* src, size_t srcSize, const ZSTD_DDict* ddict); </b><p> Decompression using a digested Dictionary. - Faster startup than ZSTD_decompress_usingDict(), recommended when same dictionary is used multiple times. + Recommended when same dictionary is used multiple times. </p></pre><BR> <a name="Chapter8"></a><h2>Streaming</h2><pre></pre> @@ -245,14 +256,18 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); A ZSTD_CStream object is required to track streaming operation. Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources. ZSTD_CStream objects can be reused multiple times on consecutive compression operations. - It is recommended to re-use ZSTD_CStream in situations where many streaming operations will be achieved consecutively, - since it will play nicer with system's memory, by re-using already allocated memory. - Use one separate ZSTD_CStream per thread for parallel execution. + It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory. - Start a new compression by initializing ZSTD_CStream context. - Use ZSTD_initCStream() to start a new compression operation. - Use variants ZSTD_initCStream_usingDict() or ZSTD_initCStream_usingCDict() for streaming with dictionary (experimental section) + For parallel execution, use one separate ZSTD_CStream per thread. + note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing. + + Parameters are sticky : when starting a new compression on the same context, + it will re-use the same sticky parameters as previous compression session. + When in doubt, it's recommended to fully initialize the context before usage. + Use ZSTD_initCStream() to set the parameter to a selected compression level. + Use advanced API (ZSTD_CCtx_setParameter(), etc.) to set more specific parameters. + Use ZSTD_compressStream() as many times as necessary to consume input stream. The function will automatically update both `pos` fields within `input` and `output`. Note that the function may not consume the entire input, @@ -260,12 +275,11 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); in which case `input.pos < input.size`. The caller must check if input has been entirely consumed. If not, the caller must make some room to receive more compressed data, - typically by emptying output buffer, or allocating a new output buffer, and then present again remaining input data. - @return : a size hint, preferred nb of bytes to use as input for next function call - or an error code, which can be tested using ZSTD_isError(). - Note 1 : it's just a hint, to help latency a little, any other value will work fine. - Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize() + @return : a size hint, preferred nb of bytes to use as input for next function call + or an error code, which can be tested using ZSTD_isError(). + Note 1 : it's just a hint, to help latency a little, any value will work fine. + Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize() At any moment, it's possible to flush whatever data might remain stuck within internal buffer, using ZSTD_flushStream(). `output->pos` will be updated. @@ -305,25 +319,24 @@ size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffe Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources. ZSTD_DStream objects can be re-used multiple times. - Use ZSTD_initDStream() to start a new decompression operation, - or ZSTD_initDStream_usingDict() if decompression requires a dictionary. - @return : recommended first input size + Use ZSTD_initDStream() to start a new decompression operation. + @return : recommended first input size + Alternatively, use advanced API to set specific properties. Use ZSTD_decompressStream() repetitively to consume your input. The function will update both `pos` fields. If `input.pos < input.size`, some input has not been consumed. It's up to the caller to present again remaining data. - The function tries to flush all data decoded immediately, repecting buffer sizes. + The function tries to flush all data decoded immediately, respecting output buffer size. If `output.pos < output.size`, decoder has flushed everything it could. - But if `output.pos == output.size`, there is no such guarantee, - it's likely that some decoded data was not flushed and still remains within internal buffers. + But if `output.pos == output.size`, there might be some data left within internal buffers., In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer. - When no additional input is provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX. + Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX. @return : 0 when a frame is completely decoded and fully flushed, or an error code, which can be tested using ZSTD_isError(), or any other value > 0, which means there is still some decoding or flushing to do to complete current frame : - the return value is a suggested next input size (a hint for better latency) - that will never load more than the current frame. + the return value is a suggested next input size (just a hint for better latency) + that will never request more than the remaining frame size. <BR></pre> @@ -340,32 +353,477 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_o <pre><b>size_t ZSTD_DStreamOutSize(void); </b>/*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */<b> </b></pre><BR> <a name="Chapter11"></a><h2>ADVANCED AND EXPERIMENTAL FUNCTIONS</h2><pre> - The definitions in this section are considered experimental. - They should never be used with a dynamic library, as prototypes may change in the future. + The definitions in the following section are considered experimental. They are provided for advanced scenarios. + They should never be used with a dynamic library, as prototypes may change in the future. Use them only in association with static linking. <BR></pre> +<a name="Chapter12"></a><h2>Candidate API for promotion to stable status</h2><pre> + The following symbols and constants form the "staging area" : + they are considered to join "stable API" by v1.4.0. + The proposal is written so that it can be made stable "as is", + though it's still possible to suggest improvements. + Staging is in fact last chance for changes, + the API is locked once reaching "stable" status. + +<BR></pre> + <pre><b>int ZSTD_minCLevel(void); </b>/*!< minimum negative compression level allowed */<b> </b></pre><BR> -<pre><b>typedef enum { ZSTD_fast=1, ZSTD_dfast, ZSTD_greedy, ZSTD_lazy, ZSTD_lazy2, - ZSTD_btlazy2, ZSTD_btopt, ZSTD_btultra } ZSTD_strategy; </b>/* from faster to stronger */<b> +<pre><b>size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize); +</b><p> `src` should point to the start of a ZSTD frame or skippable frame. + `srcSize` must be >= first frame size + @return : the compressed size of the first frame starting at `src`, + suitable to pass as `srcSize` to `ZSTD_decompress` or similar, + or an error code if input is invalid +</p></pre><BR> + +<pre><b>size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx); +size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx); +size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); +size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds); +size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict); +size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); +</b><p> These functions give the _current_ memory usage of selected object. + Note that object memory usage can evolve (increase or decrease) over time. +</p></pre><BR> + +<a name="Chapter13"></a><h2>Advanced compression API</h2><pre></pre> + +<pre><b>typedef enum { ZSTD_fast=1, + ZSTD_dfast=2, + ZSTD_greedy=3, + ZSTD_lazy=4, + ZSTD_lazy2=5, + ZSTD_btlazy2=6, + ZSTD_btopt=7, + ZSTD_btultra=8, + ZSTD_btultra2=9 + </b>/* note : new strategies _might_ be added in the future.<b> + Only the order (from fast to strong) is guaranteed */ +} ZSTD_strategy; </b></pre><BR> +<pre><b>typedef enum { + + </b>/* compression parameters */<b> + ZSTD_c_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b> + * Default level is ZSTD_CLEVEL_DEFAULT==3. + * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. + * Note 1 : it's possible to pass a negative compression level. + * Note 2 : setting a level sets all default values of other compression parameters */ + ZSTD_c_windowLog=101, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<b> + * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. + * Special: value 0 means "use default windowLog". + * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT + * requires explicitly allowing such window size at decompression stage if using streaming. */ + ZSTD_c_hashLog=102, </b>/* Size of the initial probe table, as a power of 2.<b> + * Resulting memory usage is (1 << (hashLog+2)). + * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. + * Larger tables improve compression ratio of strategies <= dFast, + * and improve speed of strategies > dFast. + * Special: value 0 means "use default hashLog". */ + ZSTD_c_chainLog=103, </b>/* Size of the multi-probe search table, as a power of 2.<b> + * Resulting memory usage is (1 << (chainLog+2)). + * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. + * Larger tables result in better and slower compression. + * This parameter is useless when using "fast" strategy. + * It's still useful when using "dfast" strategy, + * in which case it defines a secondary probe table. + * Special: value 0 means "use default chainLog". */ + ZSTD_c_searchLog=104, </b>/* Number of search attempts, as a power of 2.<b> + * More attempts result in better and slower compression. + * This parameter is useless when using "fast" and "dFast" strategies. + * Special: value 0 means "use default searchLog". */ + ZSTD_c_minMatch=105, </b>/* Minimum size of searched matches.<b> + * Note that Zstandard can still find matches of smaller size, + * it just tweaks its search algorithm to look for this size and larger. + * Larger values increase compression and decompression speed, but decrease ratio. + * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX. + * Note that currently, for all strategies < btopt, effective minimum is 4. + * , for all strategies > fast, effective maximum is 6. + * Special: value 0 means "use default minMatchLength". */ + ZSTD_c_targetLength=106, </b>/* Impact of this field depends on strategy.<b> + * For strategies btopt, btultra & btultra2: + * Length of Match considered "good enough" to stop search. + * Larger values make compression stronger, and slower. + * For strategy fast: + * Distance between match sampling. + * Larger values make compression faster, and weaker. + * Special: value 0 means "use default targetLength". */ + ZSTD_c_strategy=107, </b>/* See ZSTD_strategy enum definition.<b> + * The higher the value of selected strategy, the more complex it is, + * resulting in stronger and slower compression. + * Special: value 0 means "use default strategy". */ + + </b>/* LDM mode parameters */<b> + ZSTD_c_enableLongDistanceMatching=160, </b>/* Enable long distance matching.<b> + * This parameter is designed to improve compression ratio + * for large inputs, by finding large matches at long distance. + * It increases memory usage and window size. + * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB + * except when expressly set to a different value. */ + ZSTD_c_ldmHashLog=161, </b>/* Size of the table for long distance matching, as a power of 2.<b> + * Larger values increase memory usage and compression ratio, + * but decrease compression speed. + * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX + * default: windowlog - 7. + * Special: value 0 means "automatically determine hashlog". */ + ZSTD_c_ldmMinMatch=162, </b>/* Minimum match size for long distance matcher.<b> + * Larger/too small values usually decrease compression ratio. + * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX. + * Special: value 0 means "use default value" (default: 64). */ + ZSTD_c_ldmBucketSizeLog=163, </b>/* Log size of each bucket in the LDM hash table for collision resolution.<b> + * Larger values improve collision resolution but decrease compression speed. + * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX. + * Special: value 0 means "use default value" (default: 3). */ + ZSTD_c_ldmHashRateLog=164, </b>/* Frequency of inserting/looking up entries into the LDM hash table.<b> + * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN). + * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage. + * Larger values improve compression speed. + * Deviating far from default value will likely result in a compression ratio decrease. + * Special: value 0 means "automatically determine hashRateLog". */ + + </b>/* frame parameters */<b> + ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame header _whenever known_ (default:1)<b> + * Content size must be known at the beginning of compression. + * This is automatically the case when using ZSTD_compress2(), + * For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ + ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written at end of frame (default:0) */<b> + ZSTD_c_dictIDFlag=202, </b>/* When applicable, dictionary's ID is written into frame header (default:1) */<b> + + </b>/* multi-threading parameters */<b> + </b>/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).<b> + * They return an error otherwise. */ + ZSTD_c_nbWorkers=400, </b>/* Select how many threads will be spawned to compress in parallel.<b> + * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() : + * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller, + * while compression work is performed in parallel, within worker threads. + * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end : + * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call). + * More workers improve speed, but also increase memory usage. + * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */ + ZSTD_c_jobSize=401, </b>/* Size of a compression job. This value is enforced only when nbWorkers >= 1.<b> + * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. + * 0 means default, which is dynamically determined based on compression parameters. + * Job size must be a minimum of overlap size, or 1 MB, whichever is largest. + * The minimum size is automatically and transparently enforced */ + ZSTD_c_overlapLog=402, </b>/* Control the overlap size, as a fraction of window size.<b> + * The overlap size is an amount of data reloaded from previous job at the beginning of a new job. + * It helps preserve compression ratio, while each job is compressed in parallel. + * This value is enforced only when nbWorkers >= 1. + * Larger values increase compression ratio, but decrease speed. + * Possible values range from 0 to 9 : + * - 0 means "default" : value will be determined by the library, depending on strategy + * - 1 means "no overlap" + * - 9 means "full overlap", using a full window size. + * Each intermediate rank increases/decreases load size by a factor 2 : + * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default + * default value varies between 6 and 9, depending on strategy */ + + </b>/* note : additional experimental parameters are also available<b> + * within the experimental section of the API. + * At the time of this writing, they include : + * ZSTD_c_rsyncable + * ZSTD_c_format + * ZSTD_c_forceMaxWindow + * ZSTD_c_forceAttachDict + * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. + * note : never ever use experimentalParam? names directly; + * also, the enums values themselves are unstable and can still change. + */ + ZSTD_c_experimentalParam1=500, + ZSTD_c_experimentalParam2=10, + ZSTD_c_experimentalParam3=1000, + ZSTD_c_experimentalParam4=1001 +} ZSTD_cParameter; +</b></pre><BR> <pre><b>typedef struct { - unsigned windowLog; </b>/**< largest match distance : larger == more compression, more memory needed during decompression */<b> - unsigned chainLog; </b>/**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */<b> - unsigned hashLog; </b>/**< dispatch table : larger == faster, more memory */<b> - unsigned searchLog; </b>/**< nb of searches : larger == more compression, slower */<b> - unsigned searchLength; </b>/**< match length searched : larger == faster decompression, sometimes less compression */<b> - unsigned targetLength; </b>/**< acceptable match size for optimal parser (only) : larger == more compression, slower */<b> - ZSTD_strategy strategy; + size_t error; + int lowerBound; + int upperBound; +} ZSTD_bounds; +</b></pre><BR> +<pre><b>ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam); +</b><p> All parameters must belong to an interval with lower and upper bounds, + otherwise they will either trigger an error or be automatically clamped. + @return : a structure, ZSTD_bounds, which contains + - an error status field, which must be tested using ZSTD_isError() + - lower and upper bounds, both inclusive + +</p></pre><BR> + +<pre><b>size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value); +</b><p> Set one compression parameter, selected by enum ZSTD_cParameter. + All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds(). + Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). + Setting a parameter is generally only possible during frame initialization (before starting compression). + Exception : when using multi-threading mode (nbWorkers >= 1), + the following parameters can be updated _during_ compression (within same frame): + => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy. + new parameters will be active for next job only (after a flush()). + @return : an error code (which can be tested using ZSTD_isError()). + +</p></pre><BR> + +<pre><b>size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize); +</b><p> Total input data size to be compressed as a single frame. + Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag. + This value will also be controlled at end of frame, and trigger an error if not respected. + @result : 0, or an error code (which can be tested with ZSTD_isError()). + Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame. + In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN. + ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame. + Note 2 : pledgedSrcSize is only valid once, for the next frame. + It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN. + Note 3 : Whenever all input data is provided and consumed in a single round, + for example with ZSTD_compress2(), + or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end), + this value is automatically overriden by srcSize instead. + +</p></pre><BR> + +<pre><b>size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); +</b><p> Create an internal CDict from `dict` buffer. + Decompression will have to use same dictionary. + @result : 0, or an error code (which can be tested with ZSTD_isError()). + Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary, + meaning "return to no-dictionary mode". + Note 1 : Dictionary is sticky, it will be used for all future compressed frames. *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201812290651.wBT6pAxw006140>