From owner-svn-src-all@freebsd.org Wed Mar 14 03:00:19 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B25F46D30; Wed, 14 Mar 2018 03:00:19 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E8C08472C; Wed, 14 Mar 2018 03:00:19 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0918F224F7; Wed, 14 Mar 2018 03:00:19 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w2E30IGh024762; Wed, 14 Mar 2018 03:00:18 GMT (envelope-from cem@FreeBSD.org) Received: (from cem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w2E30HmE024745; Wed, 14 Mar 2018 03:00:17 GMT (envelope-from cem@FreeBSD.org) Message-Id: <201803140300.w2E30HmE024745@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org using -f From: Conrad Meyer Date: Wed, 14 Mar 2018 03:00:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r330894 - in head/sys/contrib/zstd: . contrib/meson doc doc/images lib lib/common lib/compress lib/decompress lib/deprecated lib/dictBuilder lib/legacy programs tests zlibWrapper zlibWr... X-SVN-Group: head X-SVN-Commit-Author: cem X-SVN-Commit-Paths: in head/sys/contrib/zstd: . contrib/meson doc doc/images lib lib/common lib/compress lib/decompress lib/deprecated lib/dictBuilder lib/legacy programs tests zlibWrapper zlibWrapper/examples X-SVN-Commit-Revision: 330894 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Mar 2018 03:00:20 -0000 Author: cem Date: Wed Mar 14 03:00:17 2018 New Revision: 330894 URL: https://svnweb.freebsd.org/changeset/base/330894 Log: Update to Zstandard 1.3.3 Includes patch to conditionalize use of __builtin_clz(ll) on __has_builtin(). The issue is tracked upstream at https://github.com/facebook/zstd/pull/884 . Otherwise, these are vanilla Zstandard 1.3.3 files. Note that the 1.3.4 release should be due out soon. Sponsored by: Dell EMC Isilon Added: head/sys/contrib/zstd/doc/images/zstd_logo86.png (contents, props changed) head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h - copied, changed from r330893, head/sys/contrib/zstd/lib/compress/zstd_compress.h head/sys/contrib/zstd/tests/seqgen.c (contents, props changed) head/sys/contrib/zstd/tests/seqgen.h (contents, props changed) Deleted: head/sys/contrib/zstd/lib/compress/zstd_compress.h Modified: head/sys/contrib/zstd/Makefile head/sys/contrib/zstd/NEWS head/sys/contrib/zstd/README.md head/sys/contrib/zstd/circle.yml head/sys/contrib/zstd/contrib/meson/meson.build head/sys/contrib/zstd/doc/zstd_compression_format.md head/sys/contrib/zstd/doc/zstd_manual.html head/sys/contrib/zstd/lib/BUCK head/sys/contrib/zstd/lib/common/bitstream.h head/sys/contrib/zstd/lib/common/mem.h head/sys/contrib/zstd/lib/common/pool.c head/sys/contrib/zstd/lib/common/xxhash.c head/sys/contrib/zstd/lib/common/zstd_common.c head/sys/contrib/zstd/lib/common/zstd_internal.h head/sys/contrib/zstd/lib/compress/zstd_compress.c head/sys/contrib/zstd/lib/compress/zstd_double_fast.c head/sys/contrib/zstd/lib/compress/zstd_double_fast.h head/sys/contrib/zstd/lib/compress/zstd_fast.c head/sys/contrib/zstd/lib/compress/zstd_fast.h head/sys/contrib/zstd/lib/compress/zstd_lazy.c head/sys/contrib/zstd/lib/compress/zstd_lazy.h head/sys/contrib/zstd/lib/compress/zstd_ldm.h head/sys/contrib/zstd/lib/compress/zstd_opt.c head/sys/contrib/zstd/lib/compress/zstd_opt.h head/sys/contrib/zstd/lib/compress/zstdmt_compress.c head/sys/contrib/zstd/lib/compress/zstdmt_compress.h head/sys/contrib/zstd/lib/decompress/zstd_decompress.c head/sys/contrib/zstd/lib/deprecated/zbuff_compress.c head/sys/contrib/zstd/lib/dictBuilder/zdict.c head/sys/contrib/zstd/lib/legacy/zstd_v01.c head/sys/contrib/zstd/lib/legacy/zstd_v02.c head/sys/contrib/zstd/lib/legacy/zstd_v03.c head/sys/contrib/zstd/lib/legacy/zstd_v04.c head/sys/contrib/zstd/lib/legacy/zstd_v05.c head/sys/contrib/zstd/lib/legacy/zstd_v06.c head/sys/contrib/zstd/lib/legacy/zstd_v07.c head/sys/contrib/zstd/lib/zstd.h head/sys/contrib/zstd/programs/BUCK head/sys/contrib/zstd/programs/Makefile head/sys/contrib/zstd/programs/bench.c head/sys/contrib/zstd/programs/bench.h head/sys/contrib/zstd/programs/dibio.c head/sys/contrib/zstd/programs/fileio.c head/sys/contrib/zstd/programs/fileio.h head/sys/contrib/zstd/programs/platform.h head/sys/contrib/zstd/programs/util.h head/sys/contrib/zstd/programs/zstd.1 head/sys/contrib/zstd/programs/zstd.1.md head/sys/contrib/zstd/programs/zstdcli.c head/sys/contrib/zstd/tests/Makefile head/sys/contrib/zstd/tests/decodecorpus.c head/sys/contrib/zstd/tests/fullbench.c head/sys/contrib/zstd/tests/fuzzer.c head/sys/contrib/zstd/tests/paramgrill.c head/sys/contrib/zstd/tests/playTests.sh head/sys/contrib/zstd/tests/zbufftest.c head/sys/contrib/zstd/tests/zstreamtest.c head/sys/contrib/zstd/zlibWrapper/BUCK head/sys/contrib/zstd/zlibWrapper/examples/zwrapbench.c head/sys/contrib/zstd/zlibWrapper/zstd_zlibwrapper.c Modified: head/sys/contrib/zstd/Makefile ============================================================================== --- head/sys/contrib/zstd/Makefile Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/Makefile Wed Mar 14 03:00:17 2018 (r330894) @@ -72,9 +72,12 @@ zstdmt: zlibwrapper: $(MAKE) -C $(ZWRAPDIR) test +.PHONY: check +check: shortest + .PHONY: test shortest test shortest: - $(MAKE) -C $(PRGDIR) allVariants + $(MAKE) -C $(PRGDIR) allVariants MOREFLAGS="-g -DZSTD_DEBUG=1" $(MAKE) -C $(TESTDIR) $@ .PHONY: examples @@ -127,11 +130,6 @@ uninstall: travis-install: $(MAKE) install PREFIX=~/install_test_dir -.PHONY: gppbuild -gppbuild: clean - g++ -v - CC=g++ $(MAKE) -C programs all CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror" - .PHONY: gcc5build gcc5build: clean gcc-5 -v @@ -163,7 +161,7 @@ aarch64build: clean CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd ppcbuild: clean - CC=powerpc-linux-gnu-gcc CLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd + CC=powerpc-linux-gnu-gcc CFLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd ppc64build: clean CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd Modified: head/sys/contrib/zstd/NEWS ============================================================================== --- head/sys/contrib/zstd/NEWS Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/NEWS Wed Mar 14 03:00:17 2018 (r330894) @@ -1,3 +1,15 @@ +v1.3.3 +perf: faster zstd_opt strategy (levels 17-19) +fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen +cli : fix : content size written in header by default +cli : fix : improved LZ4 format support, by @felixhandte +cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file +api : fix : support large skippable frames, by @terrelln +api : fix : streaming interface was adding a useless 3-bytes null block to small frames +api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown" +build: fix : compilation under rhel6 and centos6, reported by @pixelb +build: added `check` target + v1.3.2 new : long range mode, using --long command, by Stella Lau (@stellamplau) new : ability to generate and decode magicless frames (#591) Modified: head/sys/contrib/zstd/README.md ============================================================================== --- head/sys/contrib/zstd/README.md Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/README.md Wed Mar 14 03:00:17 2018 (r330894) @@ -1,15 +1,16 @@ - __Zstandard__, or `zstd` as short version, is a fast lossless compression algorithm, - targeting real-time compression scenarios at zlib-level and better compression ratios. +

Zstandard

-It is provided as an open-source BSD-licensed **C** library, -and a command line utility producing and decoding `.zst` and `.gz` files. -For other programming languages, -you can consult a list of known ports on [Zstandard homepage](http://www.zstd.net/#other-languages). +__Zstandard__, or `zstd` as short version, is a fast lossless compression algorithm, +targeting real-time compression scenarios at zlib-level and better compression ratios. +It's backed by a very fast entropy stage, provided by [Huff0 and FSE library](https://github.com/Cyan4973/FiniteStateEntropy). -| dev branch status | -|-------------------| -| [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] +The project is provided as an open-source BSD-licensed **C** library, +and a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files. +Should your project require another programming language, +a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages). +Development branch status : [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] + [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" [travisLink]: https://travis-ci.org/facebook/zstd [AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite" @@ -17,8 +18,9 @@ you can consult a list of known ports on [Zstandard ho [CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" [CircleLink]: https://circleci.com/gh/facebook/zstd +### Benchmarks -As a reference, several fast compression algorithms were tested and compared +For reference, several fast compression algorithms were tested and compared on a server running Linux Debian (`Linux version 4.8.0-1-amd64`), with a Core i7-6700K CPU @ 4.0GHz, using [lzbench], an open-source in-memory benchmark by @inikep @@ -43,7 +45,9 @@ on the [Silesia compression corpus]. [LZ4]: http://www.lz4.org/ Zstd can also offer stronger compression ratios at the cost of compression speed. -Speed vs Compression trade-off is configurable by small increments. Decompression speed is preserved and remains roughly the same at all settings, a property shared by most LZ compression algorithms, such as [zlib] or lzma. +Speed vs Compression trade-off is configurable by small increments. +Decompression speed is preserved and remains roughly the same at all settings, +a property shared by most LZ compression algorithms, such as [zlib] or lzma. The following tests were run on a server running Linux Debian (`Linux version 4.8.0-1-amd64`) @@ -56,8 +60,8 @@ Compression Speed vs Ratio | Decompression Speed ---------------------------|-------------------- ![Compression Speed vs Ratio](doc/images/Cspeed4.png "Compression Speed vs Ratio") | ![Decompression Speed](doc/images/Dspeed4.png "Decompression Speed") -Several algorithms can produce higher compression ratios, but at slower speeds, falling outside of the graph. -For a larger picture including very slow modes, [click on this link](doc/images/DCspeed5.png) . +A few other algorithms can produce higher compression ratios at slower speeds, falling outside of the graph. +For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png). ### The case for Small Data compression @@ -84,7 +88,7 @@ Training works if there is some correlation in a famil Hence, deploying one dictionary per type of data will provide the greatest benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. -#### Dictionary compression How To : +#### Dictionary compression How To: 1) Create the dictionary @@ -99,19 +103,16 @@ Dictionary gains are mostly effective in the first few `zstd -D dictionaryName --decompress FILE.zst` -### Build +### Build instructions -Once you have the repository cloned, there are multiple ways provided to build Zstandard. - #### Makefile -If your system is compatible with a standard `make` (or `gmake`) binary generator, -you can simply run it at the root directory. -It will generate `zstd` within root directory. +If your system is compatible with standard `make` (or `gmake`), +invoking `make` in root directory will generate `zstd` cli in root directory. -Other available options include : -- `make install` : create and install zstd binary, library and man page -- `make test` : create and run `zstd` and test tools on local platform +Other available options include: +- `make install` : create and install zstd cli, library and man pages +- `make check` : create and run `zstd`, tests its behavior on local platform #### cmake @@ -125,9 +126,9 @@ A Meson project is provided within `contrib/meson`. #### Visual Studio (Windows) -Going into `build` directory, you will find additional possibilities : -- Projects for Visual Studio 2005, 2008 and 2010 - + VS2010 project is compatible with VS2012, VS2013 and VS2015 +Going into `build` directory, you will find additional possibilities: +- Projects for Visual Studio 2005, 2008 and 2010. + + VS2010 project is compatible with VS2012, VS2013 and VS2015. - Automated build scripts for Visual compiler by @KrzysFR , in `build/VS_scripts`, which will build `zstd` cli and `libzstd` library without any need to open Visual Studio solution. @@ -143,11 +144,7 @@ Zstandard is dual-licensed under [BSD](LICENSE) and [G ### Contributing -The "dev" branch is the one where all contributions will be merged before reaching "master". -If you plan to propose a patch, please commit into the "dev" branch or its own feature branch. +The "dev" branch is the one where all contributions are merged before reaching "master". +If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch. Direct commit to "master" are not permitted. For more information, please read [CONTRIBUTING](CONTRIBUTING.md). - -### Miscellaneous - -Zstd entropy stage is provided by [Huff0 and FSE, from Finite State Entropy library](https://github.com/Cyan4973/FiniteStateEntropy). Modified: head/sys/contrib/zstd/circle.yml ============================================================================== --- head/sys/contrib/zstd/circle.yml Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/circle.yml Wed Mar 14 03:00:17 2018 (r330894) @@ -3,13 +3,11 @@ dependencies: - sudo dpkg --add-architecture i386 - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test; sudo apt-get -y -qq update - sudo apt-get -y install gcc-powerpc-linux-gnu gcc-arm-linux-gnueabi libc6-dev-armel-cross gcc-aarch64-linux-gnu libc6-dev-arm64-cross - - sudo apt-get -y install libstdc++-7-dev clang gcc g++ gcc-5 gcc-6 gcc-7 zlib1g-dev liblzma-dev - - sudo apt-get -y install linux-libc-dev:i386 libc6-dev-i386 test: override: - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then cc -v; make all && make clean && make -C lib libzstd-nomt && make clean; fi && + if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then cc -v; CFLAGS="-O0 -Werror" make all && make clean; fi && if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gnu90build && make clean; fi : parallel: true @@ -20,32 +18,17 @@ test: parallel: true - ? | if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make c11build && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make cmakebuild && make clean; fi + if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make ppc64build && make clean; fi : parallel: true - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make gppbuild && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gcc5build && make clean; fi - : - parallel: true - - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make gcc6build && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make clangbuild && make clean; fi - : - parallel: true - - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make m32build && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make armbuild && make clean; fi - : - parallel: true - - ? | if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make aarch64build && make clean; fi && if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make ppcbuild && make clean; fi : parallel: true - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make ppc64build && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gcc7build && make clean; fi + if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make -j regressiontest && make clean; fi && + if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make armbuild && make clean; fi : parallel: true - ? | @@ -54,8 +37,8 @@ test: : parallel: true - ? | - if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make -j regressiontest && make clean; fi && - if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then true; fi # Could add another test here + if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make cxxtest && make clean; fi && + if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make -C lib libzstd-nomt && make clean; fi : parallel: true Modified: head/sys/contrib/zstd/contrib/meson/meson.build ============================================================================== --- head/sys/contrib/zstd/contrib/meson/meson.build Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/contrib/meson/meson.build Wed Mar 14 03:00:17 2018 (r330894) @@ -2,14 +2,39 @@ project('zstd', 'c', license: 'BSD') libm = meson.get_compiler('c').find_library('m', required: true) -lib_dir = join_paths(meson.source_root(), '..', '..', 'lib') +lib_dir = join_paths('..', '..', 'lib') common_dir = join_paths(lib_dir, 'common') compress_dir = join_paths(lib_dir, 'compress') decompress_dir = join_paths(lib_dir, 'decompress') dictbuilder_dir = join_paths(lib_dir, 'dictBuilder') deprecated_dir = join_paths(lib_dir, 'deprecated') -libzstd_srcs = [join_paths(common_dir, 'entropy_common.c'), join_paths(common_dir, 'fse_decompress.c'), join_paths(common_dir, 'threading.c'), join_paths(common_dir, 'pool.c'), join_paths(common_dir, 'zstd_common.c'), join_paths(common_dir, 'error_private.c'), join_paths(common_dir, 'xxhash.c'), join_paths(compress_dir, 'fse_compress.c'), join_paths(compress_dir, 'huf_compress.c'), join_paths(compress_dir, 'zstd_compress.c'), join_paths(compress_dir, 'zstdmt_compress.c'), join_paths(decompress_dir, 'huf_decompress.c'), join_paths(decompress_dir, 'zstd_decompress.c'), join_paths(dictbuilder_dir, 'cover.c'), join_paths(dictbuilder_dir, 'divsufsort.c'), join_paths(dictbuilder_dir, 'zdict.c'), join_paths(deprecated_dir, 'zbuff_common.c'), join_paths(deprecated_dir, 'zbuff_compress.c'), join_paths(deprecated_dir, 'zbuff_decompress.c')] +libzstd_srcs = [ + join_paths(common_dir, 'entropy_common.c'), + join_paths(common_dir, 'fse_decompress.c'), + join_paths(common_dir, 'threading.c'), + join_paths(common_dir, 'pool.c'), + join_paths(common_dir, 'zstd_common.c'), + join_paths(common_dir, 'error_private.c'), + join_paths(common_dir, 'xxhash.c'), + join_paths(compress_dir, 'fse_compress.c'), + join_paths(compress_dir, 'huf_compress.c'), + join_paths(compress_dir, 'zstd_compress.c'), + join_paths(compress_dir, 'zstd_fast.c'), + join_paths(compress_dir, 'zstd_double_fast.c'), + join_paths(compress_dir, 'zstd_lazy.c'), + join_paths(compress_dir, 'zstd_opt.c'), + join_paths(compress_dir, 'zstd_ldm.c'), + join_paths(compress_dir, 'zstdmt_compress.c'), + join_paths(decompress_dir, 'huf_decompress.c'), + join_paths(decompress_dir, 'zstd_decompress.c'), + join_paths(dictbuilder_dir, 'cover.c'), + join_paths(dictbuilder_dir, 'divsufsort.c'), + join_paths(dictbuilder_dir, 'zdict.c'), + join_paths(deprecated_dir, 'zbuff_common.c'), + join_paths(deprecated_dir, 'zbuff_compress.c'), + join_paths(deprecated_dir, 'zbuff_decompress.c') +] libzstd_includes = [include_directories(common_dir, dictbuilder_dir, compress_dir, lib_dir)] @@ -19,7 +44,15 @@ if get_option('legacy_support') legacy_dir = join_paths(lib_dir, 'legacy') libzstd_includes += [include_directories(legacy_dir)] - libzstd_srcs += [join_paths(legacy_dir, 'zstd_v01.c'), join_paths(legacy_dir, 'zstd_v02.c'), join_paths(legacy_dir, 'zstd_v03.c'), join_paths(legacy_dir, 'zstd_v04.c'), join_paths(legacy_dir, 'zstd_v05.c'), join_paths(legacy_dir, 'zstd_v06.c'), join_paths(legacy_dir, 'zstd_v07.c')] + libzstd_srcs += [ + join_paths(legacy_dir, 'zstd_v01.c'), + join_paths(legacy_dir, 'zstd_v02.c'), + join_paths(legacy_dir, 'zstd_v03.c'), + join_paths(legacy_dir, 'zstd_v04.c'), + join_paths(legacy_dir, 'zstd_v05.c'), + join_paths(legacy_dir, 'zstd_v06.c'), + join_paths(legacy_dir, 'zstd_v07.c') + ] else libzstd_cflags = [] endif @@ -39,16 +72,20 @@ libzstd = library('zstd', dependencies: libzstd_deps, install: true) -programs_dir = join_paths(meson.source_root(), '..', '..', 'programs') +programs_dir = join_paths('..', '..', 'programs') zstd = executable('zstd', - join_paths(programs_dir, 'bench.c'), join_paths(programs_dir, 'datagen.c'), join_paths(programs_dir, 'dibio.c'), join_paths(programs_dir, 'fileio.c'), join_paths(programs_dir, 'zstdcli.c'), + join_paths(programs_dir, 'bench.c'), + join_paths(programs_dir, 'datagen.c'), + join_paths(programs_dir, 'dibio.c'), + join_paths(programs_dir, 'fileio.c'), + join_paths(programs_dir, 'zstdcli.c'), include_directories: libzstd_includes, c_args: ['-DZSTD_NODICT', '-DZSTD_NOBENCH'], link_with: libzstd, install: true) -tests_dir = join_paths(meson.source_root(), '..', '..', 'tests') +tests_dir = join_paths('..', '..', 'tests') datagen_c = join_paths(programs_dir, 'datagen.c') test_includes = libzstd_includes + [include_directories(programs_dir)] Added: head/sys/contrib/zstd/doc/images/zstd_logo86.png ============================================================================== Binary file. No diff available. Modified: head/sys/contrib/zstd/doc/zstd_compression_format.md ============================================================================== --- head/sys/contrib/zstd/doc/zstd_compression_format.md Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/doc/zstd_compression_format.md Wed Mar 14 03:00:17 2018 (r330894) @@ -91,10 +91,10 @@ Overview Frames ------ -Zstandard compressed data is made of up one or more __frames__. +Zstandard compressed data is made of one or more __frames__. Each frame is independent and can be decompressed indepedently of other frames. The decompressed content of multiple concatenated frames is the concatenation of -each frames decompressed content. +each frame decompressed content. There are two frame formats defined by Zstandard: Zstandard frames and Skippable frames. @@ -182,7 +182,7 @@ data must be regenerated within a single continuous me In this case, `Window_Descriptor` byte is skipped, but `Frame_Content_Size` is necessarily present. As a consequence, the decoder must allocate a memory segment -of size equal or bigger than `Frame_Content_Size`. +of size equal or larger than `Frame_Content_Size`. In order to preserve the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame Modified: head/sys/contrib/zstd/doc/zstd_manual.html ============================================================================== --- head/sys/contrib/zstd/doc/zstd_manual.html Wed Mar 14 02:56:43 2018 (r330893) +++ head/sys/contrib/zstd/doc/zstd_manual.html Wed Mar 14 03:00:17 2018 (r330894) @@ -1,10 +1,10 @@ -zstd 1.3.2 Manual +zstd 1.3.3 Manual -

zstd 1.3.2 Manual

+

zstd 1.3.3 Manual

Contents

    @@ -19,16 +19,17 @@
  1. Streaming decompression - HowTo
  2. START OF ADVANCED AND EXPERIMENTAL FUNCTIONS
  3. Advanced types
    4. -
  4. Frame size functions
    5. -
  5. Context memory usage
    6. -
  6. Advanced compression functions
    7. -
  7. Advanced decompression functions
    8. -
  8. Advanced streaming functions
    9. -
  9. Buffer-less and synchronous inner streaming functions
    10. -
  10. Buffer-less streaming compression (synchronous mode)
    11. -
  11. Buffer-less streaming decompression (synchronous mode)
    12. -
  12. New advanced API (experimental)
    13. -
  13. Block level API
    14. +
  14. Custom memory allocation functions
    15. +
  15. Frame size functions
    16. +
  16. Context memory usage
    17. +
  17. Advanced compression functions
    18. +
  18. Advanced decompression functions
    19. +
  19. Advanced streaming functions
    20. +
  20. Buffer-less and synchronous inner streaming functions
    21. +
  21. Buffer-less streaming compression (synchronous mode)
    22. +
  22. Buffer-less streaming decompression (synchronous mode)
    23. +
  23. New advanced API (experimental)
    24. +
  24. Block level API

Introduction

@@ -111,7 +112,7 @@ unsigned long long ZSTD_getFrameContentSize(const void
   @return : content size to be decompressed, as a 64-bits value _if known and not empty_, 0 otherwise.

-

Helper functions

#define ZSTD_COMPRESSBOUND(srcSize)   ((srcSize) + ((srcSize)>>8) + (((srcSize) < 128 KB) ? ((128 KB - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0))  /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
+
Helper functions
#define ZSTD_COMPRESSBOUND(srcSize)   ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0))  /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
 size_t      ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case scenario */
 unsigned    ZSTD_isError(size_t code);          /*!< tells if a `size_t` function result is an error code */
 const char* ZSTD_getErrorName(size_t code);     /*!< provides readable string from an error code */
@@ -346,17 +347,15 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_o
     ZSTD_frameParameters fParams;
 } ZSTD_parameters;
 


-
Custom memory allocation functions
typedef void* (*ZSTD_allocFunction) (void* opaque, size_t size);
-typedef void  (*ZSTD_freeFunction) (void* opaque, void* address);
-typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem;
-/* use this constant to defer to stdlib's functions */
-static const ZSTD_customMem ZSTD_defaultCMem = { NULL, NULL, NULL };
-


-
Frame size functions

+
Custom memory allocation functions

 
+
typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem;
+


+
Frame size functions

+
 
size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
 
  `src` should point to the start of a ZSTD encoded frame or skippable frame
-  `srcSize` must be at least as large as the frame
+  `srcSize` must be >= first frame size
   @return : the compressed size of the first frame starting at `src`,
             suitable to pass to `ZSTD_decompress` or similar,
             or an error code if input is invalid 
@@ -391,7 +390,7 @@ static const ZSTD_customMem ZSTD_defaultCMem = { NULL,
    @return : size of the Frame Header 
 


 
-
Context memory usage

+
Context memory usage

 
 
size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
 size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
@@ -450,7 +449,7 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
  
 


 
-
Advanced compression functions

+
Advanced compression functions

 
 
ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
 
  Create a ZSTD compression context using external alloc and free functions 
@@ -462,7 +461,8 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
              It must outlive context usage.
   workspaceSize: Use ZSTD_estimateCCtxSize() or ZSTD_estimateCStreamSize()
                  to determine how large workspace must be to support scenario.
- @return : pointer to ZSTD_CCtx*, or NULL if error (size too small)
+ @return : pointer to ZSTD_CCtx* (same address as workspace, but different type),
+           or NULL if error (typically size too small)
   Note : zstd will never resize nor malloc() when using a static cctx.
          If it needs more memory than available, it will simply error out.
   Note 2 : there is no corresponding "free" function.
@@ -505,7 +505,8 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
                  to determine how large workspace must be.
   cParams : use ZSTD_getCParams() to transform a compression level
             into its relevants cParams.
- @return : pointer to ZSTD_CDict*, or NULL if error (size too small)
+ @return : pointer to ZSTD_CDict* (same address as workspace, but different type),
+           or NULL if error (typically, size too small).
   Note : there is no corresponding "free" function.
          Since workspace was allocated externally, it must be freed externally.
  
@@ -518,7 +519,7 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
 
 
ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
 
   same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`.
-   All fields of `ZSTD_frameParameters` are set to default (0) 
+   All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 
 


 
 
size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
@@ -545,7 +546,7 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
 
   Same as ZSTD_compress_usingCDict(), with fine-tune control over frame parameters 
 


 
-
Advanced decompression functions

+
Advanced decompression functions

 
 
unsigned ZSTD_isFrame(const void* buffer, size_t size);
 
  Tells if the content of `buffer` starts with a valid Frame Identifier.
@@ -564,7 +565,8 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
              It must outlive context usage.
   workspaceSize: Use ZSTD_estimateDCtxSize() or ZSTD_estimateDStreamSize()
                  to determine how large workspace must be to support scenario.
- @return : pointer to ZSTD_DCtx*, or NULL if error (size too small)
+ @return : pointer to ZSTD_DCtx* (same address as workspace, but different type),
+           or NULL if error (typically size too small)
   Note : zstd will never resize nor malloc() when using a static dctx.
          If it needs more memory than available, it will simply error out.
   Note 2 : static dctx is incompatible with legacy support
@@ -627,24 +629,26 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_di
   When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code. 
 


 
-
Advanced streaming functions

+
Advanced streaming functions

 
 
Advanced Streaming compression functions
ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
 ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize);    /**< same as ZSTD_initStaticCCtx() */
-size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize);   /**< pledgedSrcSize must be correct, a size of 0 means unknown.  for a frame size of 0 use initCStream_advanced */
+size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize);   /**< pledgedSrcSize must be correct. If it is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, "0" also disables frame content size field. It may be enabled in the future. */
 size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); /**< creates of an internal CDict (incompatible with static CCtx), except if dict == NULL or dictSize < 8, in which case no dict is used. Note: dict is loaded with ZSTD_dm_auto (treated as a full zstd dictionary if it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.*/
 size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize,
-                                             ZSTD_parameters params, unsigned long long pledgedSrcSize);  /**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0. dict is loaded with ZSTD_dm_auto and ZSTD_dlm_byCopy. */
+                                             ZSTD_parameters params, unsigned long long pledgedSrcSize);  /**< pledgedSrcSize must be correct. If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. dict is loaded with ZSTD_dm_auto and ZSTD_dlm_byCopy. */
 size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);  /**< note : cdict will just be referenced, and must outlive compression session */
-size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize);  /**< same as ZSTD_initCStream_usingCDict(), with control over frame parameters */
+size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize);  /**< same as ZSTD_initCStream_usingCDict(), with control over frame parameters. pledgedSrcSize must be correct. If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. */
 


 
size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
 
  start a new compression job, using same parameters from previous job.
   This is typically useful to skip dictionary loading stage, since it will re-use it in-place..
   Note that zcs must be init at least once before using ZSTD_resetCStream().
-  pledgedSrcSize==0 means "srcSize unknown".
+  If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN.
   If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end.
-  @return : 0, or an error code (which can be tested using ZSTD_isError()) 
+  For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs,
+  but it may change to mean "empty" in some future version, so prefer using macro ZSTD_CONTENTSIZE_UNKNOWN.
+ @return : 0, or an error code (which can be tested using ZSTD_isError()) 
 


 
 
Advanced Streaming decompression functions
ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
@@ -655,14 +659,14 @@ size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, c
 size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);  /**< note : ddict is referenced, it must outlive decompression session */
 size_t ZSTD_resetDStream(ZSTD_DStream* zds);  /**< re-use decompression parameters from previous init; saves dictionary loading */
 


-
Buffer-less and synchronous inner streaming functions
+
Buffer-less and synchronous inner streaming functions
   This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
   But it's also a complex one, with several restrictions, documented below.
   Prefer normal streaming API for an easier experience.
  
 

 
-
Buffer-less streaming compression (synchronous mode)
+
Buffer-less streaming compression (synchronous mode)
   A ZSTD_CCtx object is required to track streaming operations.
   Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource.
   ZSTD_CCtx object can be re-used multiple times within successive compression operations.
@@ -693,12 +697,12 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds);  /**<
 
 
Buffer-less streaming compression functions
size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
 size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
-size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0 */
+size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */
 size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /**< note: fails if cdict==NULL */
-size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize);   /* compression parameters are already set within cdict. pledgedSrcSize=0 means null-size */
-size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /**<  note: if pledgedSrcSize can be 0, indicating unknown size.  if it is non-zero, it must be accurate.  for 0 size frames, use compressBegin_advanced */
+size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize);   /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
+size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /**<  note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
 


-
Buffer-less streaming decompression (synchronous mode)
+
Buffer-less streaming decompression (synchronous mode)
   A ZSTD_DCtx object is required to track streaming operations.
   Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
   A ZSTD_DCtx object can be re-used multiple times.
@@ -784,7 +788,7 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long 
 


 
typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
 


-
New advanced API (experimental)

+
New advanced API (experimental)

 
 
typedef enum {
     /* Question : should we have a format ZSTD_f_auto ?
@@ -848,18 +852,19 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long 
                               * Special: value 0 means "do not change strategy". */
 
     /* frame parameters */
-    ZSTD_p_contentSizeFlag=200, /* Content size is written into frame header _whenever known_ (default:1)
-                              * note that content size must be known at the beginning,
-                              * it is sent using ZSTD_CCtx_setPledgedSrcSize() */
+    ZSTD_p_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
+                              * Content size must be known at the beginning of compression,
+                              * it is provided using ZSTD_CCtx_setPledgedSrcSize() */
     ZSTD_p_checksumFlag,     /* A 32-bits checksum of content is written at end of frame (default:0) */
-    ZSTD_p_dictIDFlag,       /* When applicable, dictID of dictionary is provided in frame header (default:1) */
+    ZSTD_p_dictIDFlag,       /* When applicable, dictionary's ID is written into frame header (default:1) */
 
     /* multi-threading parameters */
     ZSTD_p_nbThreads=400,    /* Select how many threads a compression job can spawn (default:1)
                               * More threads improve speed, but also increase memory usage.
                               * Can only receive a value > 1 if ZSTD_MULTITHREAD is enabled.
                               * Special: value 0 means "do not change nbThreads" */
-    ZSTD_p_jobSize,          /* Size of a compression job. Each compression job is completed in parallel.
+    ZSTD_p_jobSize,          /* Size of a compression job. This value is only enforced in streaming (non-blocking) mode.
+                              * Each compression job is completed in parallel, so indirectly controls the nb of active threads.
                               * 0 means default, which is dynamically determined based on compression parameters.
                               * Job size must be a minimum of overlapSize, or 1 KB, whichever is largest
                               * The minimum size is automatically and transparently enforced */
@@ -904,7 +909,8 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long 
 
size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, unsigned value);
 
  Set one compression parameter, selected by enum ZSTD_cParameter.
   Note : when `value` is an enum, cast it to unsigned for proper type checking.
-  @result : 0, or an error code (which can be tested with ZSTD_isError()). 
+  @result : informational value (typically, the one being set, possibly corrected),
+            or an error code (which can be tested with ZSTD_isError()). 
 


 
 
size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize);
@@ -913,7 +919,7 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long 
  @result : 0, or an error code (which can be tested with ZSTD_isError()).
   Note 1 : 0 means zero, empty.
            In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN.
-           Note that ZSTD_CONTENTSIZE_UNKNOWN is default value for new compression jobs.
+           ZSTD_CONTENTSIZE_UNKNOWN is default value for any new compression job.
   Note 2 : If all data is provided and consumed in a single round,
            this value is overriden by srcSize instead. 
 


@@ -985,13 +991,19 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, c
   - Compression parameters cannot be changed once compression is started.
   - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
   - outpot->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
-  - @return provides the minimum amount of data still to flush from internal buffers
+  - In single-thread mode (default), function is blocking : it completed its job before returning to caller.
+  - In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads,
+                                                     and then immediately returns, just indicating that there is some data remaining to be flushed.
+                                                     The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
+  - Exception : in multi-threading mode, if the first call requests a ZSTD_e_end directive, it is blocking : it will complete compression before giving back control to caller.
+  - @return provides the minimum amount of data remaining to be flushed from internal buffers
             or an error code, which can be tested using ZSTD_isError().
-            if @return != 0, flush is not fully completed, there is some data left within internal buffers.
-  - after a ZSTD_e_end directive, if internal buffer is not fully flushed,
+            if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
+            This is useful to determine if a ZSTD_e_flush or ZSTD_e_end directive is completed.
+  - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
             only ZSTD_e_end or ZSTD_e_flush operations are allowed.
-            It is necessary to fully flush internal buffers
-            before starting a new compression job, or changing compression parameters.
+            Before starting a new compression job, or changing compression parameters,
+            it is required to fully flush internal buffers.
  
 


 
@@ -1166,7 +1178,7 @@ size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, c
  
 


 
-
Block level API

+
Block level API

 
 
    Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
     User will have to take in charge required information to regenerate data, such as compressed and content sizes.

Modified: head/sys/contrib/zstd/lib/BUCK
==============================================================================
--- head/sys/contrib/zstd/lib/BUCK	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/BUCK	Wed Mar 14 03:00:17 2018	(r330894)
@@ -15,15 +15,9 @@ cxx_library(
     header_namespace='',
     visibility=['PUBLIC'],
     exported_headers=subdir_glob([
-        ('compress', 'zstdmt_compress.h'),
+        ('compress', 'zstd*.h'),
     ]),
-    headers=subdir_glob([
-        ('compress', 'zstd_opt.h'),
-    ]),
-    srcs=[
-        'compress/zstd_compress.c',
-        'compress/zstdmt_compress.c',
-    ],
+    srcs=glob(['compress/zstd*.c']),
     deps=[':common'],
 )
 
@@ -31,7 +25,7 @@ cxx_library(
     name='decompress',
     header_namespace='',
     visibility=['PUBLIC'],
-    srcs=['decompress/zstd_decompress.c'],
+    srcs=glob(['decompress/zstd*.c']),
     deps=[
         ':common',
         ':legacy',
@@ -58,6 +52,9 @@ cxx_library(
     ]),
     srcs=glob(['legacy/*.c']),
     deps=[':common'],
+    exported_preprocessor_flags=[
+        '-DZSTD_LEGACY_SUPPORT=4',
+    ],
 )
 
 cxx_library(
@@ -75,6 +72,15 @@ cxx_library(
 )
 
 cxx_library(
+    name='compiler',
+    header_namespace='',
+    visibility=['PUBLIC'],
+    exported_headers=subdir_glob([
+        ('common', 'compiler.h'),
+    ]),
+)
+
+cxx_library(
     name='bitstream',
     header_namespace='',
     visibility=['PUBLIC'],
@@ -100,6 +106,7 @@ cxx_library(
     ],
     deps=[
         ':bitstream',
+        ':compiler',
         ':errors',
         ':mem',
     ],
@@ -133,7 +140,10 @@ cxx_library(
         ('common', 'pool.h'),
     ]),
     srcs=['common/pool.c'],
-    deps=[':threading'],
+    deps=[
+        ':threading',
+        ':zstd_common',
+    ],
 )
 
 cxx_library(
@@ -144,6 +154,12 @@ cxx_library(
         ('common', 'threading.h'),
     ]),
     srcs=['common/threading.c'],
+    exported_preprocessor_flags=[
+        '-DZSTD_MULTITHREAD',
+    ],
+    exported_linker_flags=[
+        '-pthread',
+    ],
 )
 
 cxx_library(
@@ -154,6 +170,9 @@ cxx_library(
         ('common', 'xxhash.h'),
     ]),
     srcs=['common/xxhash.c'],
+    exported_preprocessor_flags=[
+        '-DXXH_NAMESPACE=ZSTD_',
+    ],
 )
 
 cxx_library(
@@ -166,6 +185,7 @@ cxx_library(
     ]),
     srcs=['common/zstd_common.c'],
     deps=[
+        ':compiler',
         ':errors',
         ':mem',
     ],
@@ -175,6 +195,7 @@ cxx_library(
     name='common',
     deps=[
         ':bitstream',
+        ':compiler',
         ':entropy',
         ':errors',
         ':mem',

Modified: head/sys/contrib/zstd/lib/common/bitstream.h
==============================================================================
--- head/sys/contrib/zstd/lib/common/bitstream.h	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/bitstream.h	Wed Mar 14 03:00:17 2018	(r330894)
@@ -167,7 +167,7 @@ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD
 /*-**************************************************************
 *  Internal functions
 ****************************************************************/
-MEM_STATIC unsigned BIT_highbit32 (register U32 val)
+MEM_STATIC unsigned BIT_highbit32 (U32 val)
 {
     assert(val != 0);
     {

Modified: head/sys/contrib/zstd/lib/common/mem.h
==============================================================================
--- head/sys/contrib/zstd/lib/common/mem.h	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/mem.h	Wed Mar 14 03:00:17 2018	(r330894)
@@ -56,8 +56,6 @@ MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((s
   typedef   int32_t S32;
   typedef  uint64_t U64;
   typedef   int64_t S64;
-  typedef  intptr_t iPtrDiff;
-  typedef uintptr_t uPtrDiff;
 #else
   typedef unsigned char      BYTE;
   typedef unsigned short      U16;
@@ -66,8 +64,6 @@ MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((s
   typedef   signed int        S32;
   typedef unsigned long long  U64;
   typedef   signed long long  S64;
-  typedef ptrdiff_t      iPtrDiff;
-  typedef size_t         uPtrDiff;
 #endif
 
 
@@ -123,20 +119,26 @@ MEM_STATIC void MEM_write64(void* memPtr, U64 value) {
 /* currently only defined for gcc and icc */
 #if defined(_MSC_VER) || (defined(__INTEL_COMPILER) && defined(WIN32))
     __pragma( pack(push, 1) )
-    typedef union { U16 u16; U32 u32; U64 u64; size_t st; } unalign;
+    typedef struct { U16 v; } unalign16;
+    typedef struct { U32 v; } unalign32;
+    typedef struct { U64 v; } unalign64;
+    typedef struct { size_t v; } unalignArch;
     __pragma( pack(pop) )
 #else
-    typedef union { U16 u16; U32 u32; U64 u64; size_t st; } __attribute__((packed)) unalign;
+    typedef struct { U16 v; } __attribute__((packed)) unalign16;
+    typedef struct { U32 v; } __attribute__((packed)) unalign32;
+    typedef struct { U64 v; } __attribute__((packed)) unalign64;
+    typedef struct { size_t v; } __attribute__((packed)) unalignArch;
 #endif
 
-MEM_STATIC U16 MEM_read16(const void* ptr) { return ((const unalign*)ptr)->u16; }
-MEM_STATIC U32 MEM_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
-MEM_STATIC U64 MEM_read64(const void* ptr) { return ((const unalign*)ptr)->u64; }
-MEM_STATIC size_t MEM_readST(const void* ptr) { return ((const unalign*)ptr)->st; }
+MEM_STATIC U16 MEM_read16(const void* ptr) { return ((const unalign16*)ptr)->v; }
+MEM_STATIC U32 MEM_read32(const void* ptr) { return ((const unalign32*)ptr)->v; }
+MEM_STATIC U64 MEM_read64(const void* ptr) { return ((const unalign64*)ptr)->v; }
+MEM_STATIC size_t MEM_readST(const void* ptr) { return ((const unalignArch*)ptr)->v; }
 
-MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ((unalign*)memPtr)->u16 = value; }
-MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ((unalign*)memPtr)->u32 = value; }
-MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ((unalign*)memPtr)->u64 = value; }
+MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ((unalign16*)memPtr)->v = value; }
+MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ((unalign32*)memPtr)->v = value; }
+MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ((unalign64*)memPtr)->v = value; }
 
 #else
 

Modified: head/sys/contrib/zstd/lib/common/pool.c
==============================================================================
--- head/sys/contrib/zstd/lib/common/pool.c	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/pool.c	Wed Mar 14 03:00:17 2018	(r330894)
@@ -11,7 +11,6 @@
 
 /* ======   Dependencies   ======= */
 #include   /* size_t */
-#include   /* malloc, calloc, free */
 #include "pool.h"
 
 /* ======   Compiler specifics   ====== */
@@ -115,7 +114,7 @@ POOL_ctx* POOL_create_advanced(size_t numThreads, size
      * and full queues.
      */
     ctx->queueSize = queueSize + 1;
-    ctx->queue = (POOL_job*) malloc(ctx->queueSize * sizeof(POOL_job));
+    ctx->queue = (POOL_job*)ZSTD_malloc(ctx->queueSize * sizeof(POOL_job), customMem);
     ctx->queueHead = 0;
     ctx->queueTail = 0;
     ctx->numThreadsBusy = 0;

Modified: head/sys/contrib/zstd/lib/common/xxhash.c
==============================================================================
--- head/sys/contrib/zstd/lib/common/xxhash.c	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/xxhash.c	Wed Mar 14 03:00:17 2018	(r330894)
@@ -212,7 +212,7 @@ static U64 XXH_read64(const void* memPtr)
 #if defined(_MSC_VER)     /* Visual Studio */
 #  define XXH_swap32 _byteswap_ulong
 #  define XXH_swap64 _byteswap_uint64
-#elif (GCC_VERSION >= 403 && !defined(__riscv))
+#elif GCC_VERSION >= 403
 #  define XXH_swap32 __builtin_bswap32
 #  define XXH_swap64 __builtin_bswap64
 #else

Modified: head/sys/contrib/zstd/lib/common/zstd_common.c
==============================================================================
--- head/sys/contrib/zstd/lib/common/zstd_common.c	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/zstd_common.c	Wed Mar 14 03:00:17 2018	(r330894)
@@ -31,20 +31,26 @@ const char* ZSTD_versionString(void) { return ZSTD_VER
 *  ZSTD Error Management
 ******************************************/
 /*! ZSTD_isError() :
-*   tells if a return value is an error code */
+ *  tells if a return value is an error code */
 unsigned ZSTD_isError(size_t code) { return ERR_isError(code); }
 
 /*! ZSTD_getErrorName() :
-*   provides error code string from function result (useful for debugging) */
+ *  provides error code string from function result (useful for debugging) */
 const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); }
 
 /*! ZSTD_getError() :
-*   convert a `size_t` function result into a proper ZSTD_errorCode enum */
+ *  convert a `size_t` function result into a proper ZSTD_errorCode enum */
 ZSTD_ErrorCode ZSTD_getErrorCode(size_t code) { return ERR_getErrorCode(code); }
 
 /*! ZSTD_getErrorString() :
-*   provides error code string from enum */
+ *  provides error code string from enum */
 const char* ZSTD_getErrorString(ZSTD_ErrorCode code) { return ERR_getErrorString(code); }
+
+/*! g_debuglog_enable :
+ *  turn on/off debug traces (global switch) */
+#if defined(ZSTD_DEBUG) && (ZSTD_DEBUG >= 2)
+int g_debuglog_enable = 1;
+#endif
 
 
 /*=**************************************************************

Modified: head/sys/contrib/zstd/lib/common/zstd_internal.h
==============================================================================
--- head/sys/contrib/zstd/lib/common/zstd_internal.h	Wed Mar 14 02:56:43 2018	(r330893)
+++ head/sys/contrib/zstd/lib/common/zstd_internal.h	Wed Mar 14 03:00:17 2018	(r330894)
@@ -11,6 +11,10 @@
 #ifndef ZSTD_CCOMMON_H_MODULE
 #define ZSTD_CCOMMON_H_MODULE
 
+/* this module contains definitions which must be identical
+ * across compression, decompression and dictBuilder.
+ * It also contains a few functions useful to at least 2 of them
+ * and which benefit from being inlined */
 
 /*-*************************************
 *  Dependencies
@@ -50,21 +54,26 @@ extern "C" {
 
 #if defined(ZSTD_DEBUG) && (ZSTD_DEBUG>=2)
 #  include 
+extern int g_debuglog_enable;
 /* recommended values for ZSTD_DEBUG display levels :
  * 1 : no display, enables assert() only
- * 2 : reserved for currently active debugging path
- * 3 : events once per object lifetime (CCtx, CDict)
+ * 2 : reserved for currently active debug path
+ * 3 : events once per object lifetime (CCtx, CDict, etc.)
  * 4 : events once per frame
  * 5 : events once per block
  * 6 : events once per sequence (*very* verbose) */
-#  define DEBUGLOG(l, ...) {                         \
-                if (l<=ZSTD_DEBUG) {                 \
-                    fprintf(stderr, __FILE__ ": ");  \
-                    fprintf(stderr, __VA_ARGS__);    \
-                    fprintf(stderr, " \n");          \
+#  define RAWLOG(l, ...) {                                      \
+                if ((g_debuglog_enable) & (l<=ZSTD_DEBUG)) {    \
+                    fprintf(stderr, __VA_ARGS__);               \
             }   }
+#  define DEBUGLOG(l, ...) {                                    \
+                if ((g_debuglog_enable) & (l<=ZSTD_DEBUG)) {    \
+                    fprintf(stderr, __FILE__ ": " __VA_ARGS__); \
+                    fprintf(stderr, " \n");                     \
+            }   }
 #else
-#  define DEBUGLOG(l, ...)      {}    /* disabled */
+#  define RAWLOG(l, ...)      {}    /* disabled */
+#  define DEBUGLOG(l, ...)    {}    /* disabled */
 #endif
 
 
@@ -85,9 +94,7 @@ extern "C" {
 #define ZSTD_OPT_NUM    (1<<12)
 
 #define ZSTD_REP_NUM      3                 /* number of repcodes */
-#define ZSTD_REP_CHECK    (ZSTD_REP_NUM)    /* number of repcodes to check by the optimal parser */
 #define ZSTD_REP_MOVE     (ZSTD_REP_NUM-1)
-#define ZSTD_REP_MOVE_OPT (ZSTD_REP_NUM)
 static const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };
 
 #define KB *(1 <<10)
@@ -134,28 +141,40 @@ typedef enum { set_basic, set_rle, set_compressed, set
 #define LLFSELog    9
 #define OffFSELog   8
 
-static const U32 LL_bits[MaxLL+1] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-                                      1, 1, 1, 1, 2, 2, 3, 3, 4, 6, 7, 8, 9,10,11,12,
+static const U32 LL_bits[MaxLL+1] = { 0, 0, 0, 0, 0, 0, 0, 0,
+                                      0, 0, 0, 0, 0, 0, 0, 0,
+                                      1, 1, 1, 1, 2, 2, 3, 3,
+                                      4, 6, 7, 8, 9,10,11,12,
                                      13,14,15,16 };
-static const S16 LL_defaultNorm[MaxLL+1] = { 4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1,
-                                             2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1,
+static const S16 LL_defaultNorm[MaxLL+1] = { 4, 3, 2, 2, 2, 2, 2, 2,
+                                             2, 2, 2, 2, 2, 1, 1, 1,
+                                             2, 2, 2, 2, 2, 2, 2, 2,
+                                             2, 3, 2, 1, 1, 1, 1, 1,
                                             -1,-1,-1,-1 };
 #define LL_DEFAULTNORMLOG 6  /* for static allocation */
 static const U32 LL_defaultNormLog = LL_DEFAULTNORMLOG;
 
-static const U32 ML_bits[MaxML+1] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-                                      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-                                      1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5, 7, 8, 9,10,11,
+static const U32 ML_bits[MaxML+1] = { 0, 0, 0, 0, 0, 0, 0, 0,
+                                      0, 0, 0, 0, 0, 0, 0, 0,
+                                      0, 0, 0, 0, 0, 0, 0, 0,
+                                      0, 0, 0, 0, 0, 0, 0, 0,
+                                      1, 1, 1, 1, 2, 2, 3, 3,
+                                      4, 4, 5, 7, 8, 9,10,11,
                                      12,13,14,15,16 };
-static const S16 ML_defaultNorm[MaxML+1] = { 1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
-                                             1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
-                                             1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,-1,-1,
+static const S16 ML_defaultNorm[MaxML+1] = { 1, 4, 3, 2, 2, 2, 2, 2,
+                                             2, 1, 1, 1, 1, 1, 1, 1,
+                                             1, 1, 1, 1, 1, 1, 1, 1,
+                                             1, 1, 1, 1, 1, 1, 1, 1,
+                                             1, 1, 1, 1, 1, 1, 1, 1,
+                                             1, 1, 1, 1, 1, 1,-1,-1,
                                             -1,-1,-1,-1,-1 };
 #define ML_DEFAULTNORMLOG 6  /* for static allocation */
 static const U32 ML_defaultNormLog = ML_DEFAULTNORMLOG;
 
-static const S16 OF_defaultNorm[DefaultMaxOff+1] = { 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
-                                                     1, 1, 1, 1, 1, 1, 1, 1,-1,-1,-1,-1,-1 };
+static const S16 OF_defaultNorm[DefaultMaxOff+1] = { 1, 1, 1, 1, 1, 1, 2, 2,
+                                                     2, 1, 1, 1, 1, 1, 1, 1,
+                                                     1, 1, 1, 1, 1, 1, 1, 1,
+                                                    -1,-1,-1,-1,-1 };
 #define OF_DEFAULTNORMLOG 5  /* for static allocation */

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***