From owner-svn-src-vendor@freebsd.org Tue Sep 3 14:07:06 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BB171DD4C6; Tue, 3 Sep 2019 14:07:06 +0000 (UTC) (envelope-from yuripv@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46N80B2KPyz4QF4; Tue, 3 Sep 2019 14:07:06 +0000 (UTC) (envelope-from yuripv@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1452) id 33A641B0AE; Tue, 3 Sep 2019 14:06:28 +0000 (UTC) X-Original-To: yuripv@localmail.freebsd.org Delivered-To: yuripv@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id BE98410133; Fri, 19 Apr 2019 00:31:10 +0000 (UTC) (envelope-from owner-src-committers@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CF0F86B87; Fri, 19 Apr 2019 00:31:10 +0000 (UTC) (envelope-from owner-src-committers@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 538) id 1371910119; Fri, 19 Apr 2019 00:31:10 +0000 (UTC) Delivered-To: src-committers@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [96.47.72.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id 0D82C10117 for ; Fri, 19 Apr 2019 00:31:07 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDBA286B7D; Fri, 19 Apr 2019 00:31:06 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 988AA26A2A; Fri, 19 Apr 2019 00:31:06 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x3J0V6jt089908; Fri, 19 Apr 2019 00:31:06 GMT (envelope-from cem@FreeBSD.org) Received: (from cem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x3J0V5em089899; Fri, 19 Apr 2019 00:31:05 GMT (envelope-from cem@FreeBSD.org) Message-Id: <201904190031.x3J0V5em089899@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org using -f From: Conrad Meyer To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r346361 - in vendor/zstd/dist: . contrib/adaptive-compression contrib/docker contrib/experimental_dict_builders contrib/experimental_dict_builders/benchmarkDictBuilder contrib/experimen... X-SVN-Group: vendor X-SVN-Commit-Author: cem X-SVN-Commit-Paths: in vendor/zstd/dist: . contrib/adaptive-compression contrib/docker contrib/experimental_dict_builders contrib/experimental_dict_builders/benchmarkDictBuilder contrib/experimental_dict_builders/fastCov... X-SVN-Commit-Revision: 346361 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk X-Loop: FreeBSD.org Sender: owner-src-committers@freebsd.org X-Rspamd-Queue-Id: 3CF0F86B87 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Status: O X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Tue, 03 Sep 2019 14:07:07 -0000 X-Original-Date: Fri, 19 Apr 2019 00:31:05 +0000 (UTC) X-List-Received-Date: Tue, 03 Sep 2019 14:07:07 -0000 Author: cem Date: Fri Apr 19 00:31:04 2019 New Revision: 346361 URL: https://svnweb.freebsd.org/changeset/base/346361 Log: import zstd 1.4.0 Added: vendor/zstd/dist/CHANGELOG vendor/zstd/dist/contrib/docker/ vendor/zstd/dist/contrib/docker/Dockerfile vendor/zstd/dist/contrib/docker/README.md vendor/zstd/dist/contrib/experimental_dict_builders/ vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/ vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/benchmark.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/dictBuilder.h (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/test.sh (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/ vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/Makefile (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/README.md vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/fastCover.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/fastCover.h (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/main.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/fastCover/test.sh (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/ vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/Makefile (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/README.md vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/io.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/io.h (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/main.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/random.c (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/random.h (contents, props changed) vendor/zstd/dist/contrib/experimental_dict_builders/randomDictBuilder/test.sh (contents, props changed) vendor/zstd/dist/contrib/largeNbDicts/ vendor/zstd/dist/contrib/largeNbDicts/Makefile (contents, props changed) vendor/zstd/dist/contrib/largeNbDicts/README.md vendor/zstd/dist/contrib/largeNbDicts/largeNbDicts.c (contents, props changed) vendor/zstd/dist/contrib/premake/ vendor/zstd/dist/contrib/premake/premake4.lua (contents, props changed) vendor/zstd/dist/contrib/premake/zstd.lua (contents, props changed) vendor/zstd/dist/contrib/snap/ vendor/zstd/dist/contrib/snap/snapcraft.yaml vendor/zstd/dist/examples/ vendor/zstd/dist/examples/.gitignore vendor/zstd/dist/examples/Makefile (contents, props changed) vendor/zstd/dist/examples/README.md vendor/zstd/dist/examples/common.h (contents, props changed) vendor/zstd/dist/examples/dictionary_compression.c (contents, props changed) vendor/zstd/dist/examples/dictionary_decompression.c (contents, props changed) vendor/zstd/dist/examples/multiple_simple_compression.c (contents, props changed) vendor/zstd/dist/examples/multiple_streaming_compression.c (contents, props changed) vendor/zstd/dist/examples/simple_compression.c (contents, props changed) vendor/zstd/dist/examples/simple_decompression.c (contents, props changed) vendor/zstd/dist/examples/streaming_compression.c (contents, props changed) vendor/zstd/dist/examples/streaming_decompression.c (contents, props changed) vendor/zstd/dist/examples/streaming_memory_usage.c (contents, props changed) vendor/zstd/dist/programs/timefn.c (contents, props changed) vendor/zstd/dist/programs/timefn.h (contents, props changed) vendor/zstd/dist/tests/fuzz/dictionary_decompress.c (contents, props changed) vendor/zstd/dist/tests/fuzz/dictionary_round_trip.c (contents, props changed) Modified: vendor/zstd/dist/Makefile vendor/zstd/dist/README.md vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/dist/contrib/pzstd/Makefile vendor/zstd/dist/contrib/pzstd/Pzstd.cpp vendor/zstd/dist/contrib/pzstd/utils/Range.h vendor/zstd/dist/contrib/pzstd/utils/ResourcePool.h vendor/zstd/dist/doc/README.md vendor/zstd/dist/doc/educational_decoder/Makefile vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/dist/lib/Makefile vendor/zstd/dist/lib/README.md vendor/zstd/dist/lib/common/compiler.h vendor/zstd/dist/lib/common/fse.h vendor/zstd/dist/lib/common/threading.c vendor/zstd/dist/lib/common/xxhash.c vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/dist/lib/compress/fse_compress.c vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/dist/lib/compress/zstd_lazy.h vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/dist/lib/compress/zstdmt_compress.h vendor/zstd/dist/lib/decompress/zstd_ddict.c vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/dist/lib/decompress/zstd_decompress_block.c vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/dist/lib/dictBuilder/cover.h vendor/zstd/dist/lib/dictBuilder/fastcover.c vendor/zstd/dist/lib/dictBuilder/zdict.h vendor/zstd/dist/lib/legacy/zstd_legacy.h vendor/zstd/dist/lib/legacy/zstd_v01.c vendor/zstd/dist/lib/legacy/zstd_v01.h vendor/zstd/dist/lib/legacy/zstd_v02.c vendor/zstd/dist/lib/legacy/zstd_v02.h vendor/zstd/dist/lib/legacy/zstd_v03.c vendor/zstd/dist/lib/legacy/zstd_v03.h vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/dist/lib/legacy/zstd_v04.h vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/dist/lib/legacy/zstd_v05.h vendor/zstd/dist/lib/legacy/zstd_v06.c vendor/zstd/dist/lib/legacy/zstd_v06.h vendor/zstd/dist/lib/legacy/zstd_v07.c vendor/zstd/dist/lib/legacy/zstd_v07.h vendor/zstd/dist/lib/zstd.h vendor/zstd/dist/programs/Makefile vendor/zstd/dist/programs/benchfn.c vendor/zstd/dist/programs/benchfn.h vendor/zstd/dist/programs/benchzstd.c vendor/zstd/dist/programs/benchzstd.h vendor/zstd/dist/programs/dibio.c vendor/zstd/dist/programs/fileio.c vendor/zstd/dist/programs/fileio.h vendor/zstd/dist/programs/platform.h vendor/zstd/dist/programs/util.c vendor/zstd/dist/programs/util.h vendor/zstd/dist/programs/zstd.1 vendor/zstd/dist/programs/zstd.1.md vendor/zstd/dist/programs/zstdcli.c vendor/zstd/dist/programs/zstdgrep vendor/zstd/dist/tests/Makefile vendor/zstd/dist/tests/README.md vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/dist/tests/fullbench.c vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/dist/tests/fuzz/README.md vendor/zstd/dist/tests/fuzz/fuzz.py vendor/zstd/dist/tests/fuzz/fuzz_helpers.h vendor/zstd/dist/tests/fuzz/regression_driver.c vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/dist/tests/fuzz/stream_decompress.c vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/dist/tests/fuzz/zstd_helpers.h vendor/zstd/dist/tests/fuzzer.c vendor/zstd/dist/tests/legacy.c vendor/zstd/dist/tests/paramgrill.c vendor/zstd/dist/tests/playTests.sh vendor/zstd/dist/tests/poolTests.c vendor/zstd/dist/tests/regression/config.c vendor/zstd/dist/tests/regression/method.c vendor/zstd/dist/tests/regression/results.csv vendor/zstd/dist/tests/regression/test.c vendor/zstd/dist/tests/roundTripCrash.c vendor/zstd/dist/tests/symbols.c vendor/zstd/dist/tests/zbufftest.c vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/dist/zlibWrapper/README.md vendor/zstd/dist/zlibWrapper/examples/fitblk.c vendor/zstd/dist/zlibWrapper/examples/fitblk_original.c vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/dist/zlibWrapper/gzguts.h Added: vendor/zstd/dist/CHANGELOG ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/dist/CHANGELOG Fri Apr 19 00:31:04 2019 (r346361) @@ -0,0 +1,461 @@ +v1.4.0 +perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln +api: Move the advanced API, including all functions in the staging section, to the stable section +api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress +api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter +api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter +api: Don't export ZSTDMT functions from the shared library by default +api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT +api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao +api: Fix ZSTD_decompressDCtx() corner cases with a dictionary +api: Move ZSTD_getDictID_*() functions to the stable section +api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln +api: Allow compression parameters to be set when a dictionary is used +api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called +api: Fix ZSTD_estimateCStreamSize_usingCCtxParams() +api: Setting ZSTD_d_maxWindowLog to 0 means use the default +cli: Ensure that a dictionary is not used to compress itself by @shakeelrao +cli: Add --[no-]compress-literals flag to enable or disable literal compression +doc: Update the examples to use the advanced API +doc: Explain how to transition from old streaming functions to the advanced API in the header +build: Improve the Windows release packages +build: Improve CMake build by @hjmjohnson +build: Build fixes for FreeBSD by @lwhsu +build: Remove redundant warnings by @thatsafunnyname +build: Fix tests on OpenBSD by @bket +build: Extend fuzzer build system to work with the new clang engine +build: CMake now creates the libzstd.so.1 symlink +build: Improve Menson build by @lzutao +misc: Fix symbolic link detection on FreeBSD +misc: Use physical core count for -T0 on FreeBSD by @cemeyer +misc: Fix zstd --list on truncated files by @kostmo +misc: Improve logging in debug mode by @felixhandte +misc: Add CirrusCI tests by @lwhsu +misc: Optimize dictionary memory usage in corner cases +misc: Improve the dictionary builder on small or homogeneous data +misc: Fix spelling across the repo by @jsoref + +v1.3.8 +perf: better decompression speed on large files (+7%) and cold dictionaries (+15%) +perf: slightly better compression ratio at high compression modes +api : finalized advanced API, last stage before "stable" status +api : new --rsyncable mode, by @terrelln +api : support decompression of empty frames into NULL (used to be an error) (#1385) +build: new set of macros to build a minimal size decoder, by @felixhandte +build: fix compilation on MIPS32, reported by @clbr (#1441) +build: fix compilation with multiple -arch flags, by @ryandesign +build: highly upgraded meson build, by @lzutao +build: improved buck support, by @obelisk +build: fix cmake script : can create debug build, by @pitrou +build: Makefile : grep works on both colored consoles and systems without color support +build: fixed zstd-pgo, by @bmwiedemann +cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423) +cli : --no-progress flag, preserving final summary (#1371), by @terrelln +cli : ensure destination file is not source file (#1422) +cli : clearer error messages, especially when input file not present +doc : clarified zstd_compression_format.md, by @ulikunitz +misc: fixed zstdgrep, returns 1 on failure, by @lzutao +misc: NEWS renamed as CHANGELOG, in accordance with fboss + +v1.3.7 +perf: slightly better decompression speed on clang (depending on hardware target) +fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10 +build: no longer build backtrace by default in release mode; restrict further automatic mode +build: control backtrace support through build macro BACKTRACE +misc: added man pages for zstdless and zstdgrep, by @samrussell + +v1.3.6 +perf: much faster dictionary builder, by @jenniferliu +perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte +perf: faster dictionary decompression when using a very large number of dictionaries simultaneously +cli : fix : does no longer overwrite destination when source does not exist (#1082) +cli : new command --adapt, for automatic compression level adaptation +api : fix : block api can be streamed with > 4 GB, reported by @catid +api : reduced ZSTD_DDict size by 2 KB +api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel(). +build: support Haiku target, by @korli +build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT. +doc : zstd_compression_format.md updated to match wording in IETF RFC 8478 +misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97 + +v1.3.5 +perf: much faster dictionary compression, by @felixhandte +perf: small quality improvement for dictionary generation, by @terrelln +perf: slightly improved high compression levels (notably level 19) +mem : automatic memory release for long duration contexts +cli : fix : overlapLog can be manually set +cli : fix : decoding invalid lz4 frames +api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln +api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln +build: select custom libzstd scope through control macros, by @GeorgeLu97 +build: OpenBSD patch, by @bket +build: make and make all are compatible with -j +doc : clarify zstd_compression_format.md, updated for IETF RFC process +misc: pzstd compatible with reproducible compilation, by @lamby + +v1.3.4 +perf: faster speed (especially decoding speed) on recent cpus (haswell+) +perf: much better performance associating --long with multi-threading, by @terrelln +perf: better compression at levels 13-15 +cli : asynchronous compression by default, for faster experience (use --single-thread for former behavior) +cli : smoother status report in multi-threading mode +cli : added command --fast=#, for faster compression modes +cli : fix crash when not overwriting existing files, by Pádraig Brady (@pixelb) +api : `nbThreads` becomes `nbWorkers` : 1 triggers asynchronous mode +api : compression levels can be negative, for even more speed +api : ZSTD_getFrameProgression() : get precise progress status of ZSTDMT anytime +api : ZSTDMT can accept new compression parameters during compression +api : implemented all advanced dictionary decompression prototypes +build: improved meson recipe, by Shawn Landden (@shawnl) +build: VS2017 scripts, by @HaydnTrigg +misc: all /contrib projects fixed +misc: added /contrib/docker script by @gyscos + +v1.3.3 +perf: faster zstd_opt strategy (levels 16-19) +fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen +cli : fix : content size written in header by default +cli : fix : improved LZ4 format support, by @felixhandte +cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file +api : fix : support large skippable frames, by @terrelln +api : fix : streaming interface was adding a useless 3-bytes null block to small frames +api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown" +build: fix : compilation under rhel6 and centos6, reported by @pixelb +build: added `check` target + +v1.3.2 +new : long range mode, using --long command, by Stella Lau (@stellamplau) +new : ability to generate and decode magicless frames (#591) +changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode +fix : multi-threading compression works with custom allocators +fix : ZSTD_sizeof_CStream() was over-evaluating memory usage +fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22) +fix : 32-bits build can now decode large offsets (levels 21+) +cli : added LZ4 frame support by default, by Felix Handte (@felixhandte) +cli : improved --list output +cli : new : can split input file for dictionary training, using command -B# +cli : new : clean operation artefact on Ctrl-C interruption +cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851) +cli : fix : write file size in header in multiple-files mode +api : added macro ZSTD_COMPRESSBOUND() for static allocation +api : experimental : new advanced decompression API +api : fix : sizeof_CCtx() used to over-estimate +build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819) +build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818) +example : added streaming_memory_usage +license : changed /examples license to BSD + GPLv2 +license : fix a few header files to reflect new license (#825) + +v1.3.1 +New license : BSD + GPLv2 +perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk) +perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760) +cli : improved and fixed --list command, by @ib (#772) +cli : command -vV to list supported formats, by @ib (#771) +build : fixed binary variants, reported by @svenha (#788) +build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718) +API exp : breaking change : ZSTD_getframeHeader() provides more information +API exp : breaking change : pinned down values of error codes +doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz) +new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74) +new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau) +updated : contrib/linux-kernel, by Nick Terrell (@terrelln) + +v1.3.0 +cli : new : `--list` command, by Paul Cruz +cli : changed : xz/lzma support enabled by default +cli : changed : `-t *` continue processing list after a decompression error +API : added : ZSTD_versionString() +API : promoted to stable status : ZSTD_getFrameContentSize(), by Sean Purcell +API exp : new advanced API : ZSTD_compress_generic(), ZSTD_CCtx_setParameter() +API exp : new : API for static or external allocation : ZSTD_initStatic?Ctx() +API exp : added : ZSTD_decompressBegin_usingDDict(), requested by Guy Riddle (#700) +API exp : clarified memory estimation / measurement functions. +API exp : changed : strongest strategy renamed ZSTD_btultra, fastest strategy ZSTD_fast set to 1 +tools : decodecorpus can generate random dictionary-compressed samples, by Paul Cruz +new : contrib/seekable_format, demo and API, by Sean Purcell +changed : contrib/linux-kernel, updated version and license, by Nick Terrell + +v1.2.0 +cli : changed : Multithreading enabled by default (use target zstd-nomt or HAVE_THREAD=0 to disable) +cli : new : command -T0 means "detect and use nb of cores", by Sean Purcell +cli : new : zstdmt symlink hardwired to `zstd -T0` +cli : new : command --threads=# (#671) +cli : changed : cover dictionary builder by default, for improved quality, by Nick Terrell +cli : new : commands --train-cover and --train-legacy, to select dictionary algorithm and parameters +cli : experimental targets `zstd4` and `xzstd4`, with support for lz4 format, by Sean Purcell +cli : fix : does not output compressed data on console +cli : fix : ignore symbolic links unless --force specified, +API : breaking change : ZSTD_createCDict_advanced(), only use compressionParameters as argument +API : added : prototypes ZSTD_*_usingCDict_advanced(), for direct control over frameParameters. +API : improved: ZSTDMT_compressCCtx() reduced memory usage +API : fix : ZSTDMT_compressCCtx() now provides srcSize in header (#634) +API : fix : src size stored in frame header is controlled at end of frame +API : fix : enforced consistent rules for pledgedSrcSize==0 (#641) +API : fix : error code "GENERIC" replaced by "dstSizeTooSmall" when appropriate +build: improved cmake script, by @Majlen +build: enabled Multi-threading support for *BSD, by Baptiste Daroussin +tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target. +new : contrib/linux-kernel version, by Nick Terrell + +v1.1.4 +cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski +cli : new : advanced benchmark command --priority=rt +cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77 +cli : fix : --rm remains silent when input is stdin +cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski +speed : improved decompression speed in streaming mode for single shot scenarios (+5%) +memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB +arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell +API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize() +API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value) +build : new: meson build system in contrib/meson, by Dima Krasner +build : improved cmake script, by @Majlen +build : added -Wformat-security flag, as recommended by Padraig Brady +doc : new : educational decoder, by Sean Purcell + +v1.1.3 +cli : zstd can decompress .gz files (can be disabled with `make zstd-nogz` or `make HAVE_ZLIB=0`) +cli : new : experimental target `make zstdmt`, with multi-threading support +cli : new : improved dictionary builder "cover" (experimental), by Nick Terrell, based on prior work by Giuseppe Ottaviano. +cli : new : advanced commands for detailed parameters, by Przemyslaw Skibinski +cli : fix zstdless on Mac OS-X, by Andrew Janke +cli : fix #232 "compress non-files" +dictBuilder : improved dictionary generation quality, thanks to Nick Terrell +API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental) +API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul +API : new : ZDICT_finalizeDictionary() +API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511) +API : fix : all symbols properly exposed in libzstd, by Nick Terrell +build : support for Solaris target, by Przemyslaw Skibinski +doc : clarified specification, by Sean Purcell + +v1.1.2 +API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init +API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize() +API : zbuff : changed : prototypes now generate deprecation warnings +lib : improved : faster decompression speed at ultra compression settings and 32-bits mode +lib : changed : only public ZSTD_ symbols are now exposed +lib : changed : reduced usage of stack memory +lib : fixed : several corner case bugs, by Nick Terrell +cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski +cli : new : preserve file attributes +cli : new : added zstdless and zstdgrep tools +cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd) +cli : fixed : zstdcat +zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski +install : better compatibility with FreeBSD, by Dimitry Andric +source tree : changed : zbuff source files moved to lib/deprecated + +v1.1.1 +New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption +New : doc/zstd_manual.html, by Przemyslaw Skibinski +Improved : slightly better compression ratio at --ultra levels (>= 20) +Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report +Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section) +Added : example/multiple_streaming_compression.c +Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h) +Updated man page +Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets + +v1.1.0 +New : contrib/pzstd, parallel version of zstd, by Nick Terrell +added : NetBSD install target (#338) +Improved : speed for batches of small files +Improved : speed of zlib wrapper, by Przemyslaw Skibinski +Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier +Fixed : CLI -d output to stdout by default when input is stdin (#322) +Fixed : CLI correctly detects console on Mac OS-X +Fixed : CLI supports recursive mode `-r` on Mac OS-X +Fixed : Legacy decoders use unified error codes, reported by benrg (#341), fixed by Przemyslaw Skibinski +Fixed : compatibility with OpenBSD, reported by Juan Francisco Cantero Hurtado (#319) +Fixed : compatibility with Hurd, by Przemyslaw Skibinski (#365) +Fixed : zstd-pgo, reported by octoploid (#329) + +v1.0.0 +Change Licensing, all project is now BSD, Copyright Facebook +Small decompression speed improvement +API : Streaming API supports legacy format +API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter() +CLI supports legacy formats v0.4+ +Fixed : compression fails on certain huge files, reported by Jesse McGrew +Enhanced documentation, by Przemyslaw Skibinski + +v0.8.1 +New streaming API +Changed : --ultra now enables levels beyond 19 +Changed : -i# now selects benchmark time in second +Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell +Fixed : speed regression on specific patterns (#272) +Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291) +Fixed : ICC compilation, by Przemyslaw Skibinski + +v0.8.0 +Improved : better speed on clang and gcc -O2, thanks to Eric Biggers +New : Build on FreeBSD and DragonFly, thanks to JrMarino +Changed : modified API : ZSTD_compressEnd() +Fixed : legacy mode with ZSTD_HEAPMODE=0, by Christopher Bergqvist +Fixed : premature end of frame when zero-sized raw block, reported by Eric Biggers +Fixed : large dictionaries (> 384 KB), reported by Ilona Papava +Fixed : checksum correctly checked in single-pass mode +Fixed : combined --test amd --rm, reported by Andreas M. Nilsson +Modified : minor compression level adaptations +Updated : compression format specification to v0.2.0 +changed : zstd.h moved to /lib directory + +v0.7.5 +Transition version, supporting decoding of v0.8.x + +v0.7.4 +Added : homebrew for Mac, by Daniel Cade +Added : more examples +Fixed : segfault when using small dictionaries, reported by Felix Handte +Modified : default compression level for CLI is now 3 +Updated : specification, to v0.1.1 + +v0.7.3 +New : compression format specification +New : `--` separator, stating that all following arguments are file names. Suggested by Chip Turner. +New : `ZSTD_getDecompressedSize()` +New : OpenBSD target, by Juan Francisco Cantero Hurtado +New : `examples` directory +fixed : dictBuilder using HC levels, reported by Bartosz Taudul +fixed : legacy support from ZSTD_decompress_usingDDict(), reported by Felix Handte +fixed : multi-blocks decoding with intermediate uncompressed blocks, reported by Greg Slazinski +modified : removed "mem.h" and "error_public.h" dependencies from "zstd.h" (experimental section) +modified : legacy functions no longer need magic number + +v0.7.2 +fixed : ZSTD_decompressBlock() using multiple consecutive blocks. Reported by Greg Slazinski. +fixed : potential segfault on very large files (many gigabytes). Reported by Chip Turner. +fixed : CLI displays system error message when destination file cannot be created (#231). Reported by Chip Turner. + +v0.7.1 +fixed : ZBUFF_compressEnd() called multiple times with too small `dst` buffer, reported by Christophe Chevalier +fixed : dictBuilder fails if first sample is too small, reported by Руслан Ковалёв +fixed : corruption issue, reported by cj +modified : checksum enabled by default in command line mode + +v0.7.0 +New : Support for directory compression, using `-r`, thanks to Przemyslaw Skibinski +New : Command `--rm`, to remove source file after successful de/compression +New : Visual build scripts, by Christophe Chevalier +New : Support for Sparse File-systems (do not use space for zero-filled sectors) +New : Frame checksum support +New : Support pass-through mode (when using `-df`) +API : more efficient Dictionary API : `ZSTD_compress_usingCDict()`, `ZSTD_decompress_usingDDict()` +API : create dictionary files from custom content, by Giuseppe Ottaviano +API : support for custom malloc/free functions +New : controllable Dictionary ID +New : Support for skippable frames + +v0.6.1 +New : zlib wrapper API, thanks to Przemyslaw Skibinski +New : Ability to compile compressor / decompressor separately +Changed : new lib directory structure +Fixed : Legacy codec v0.5 compatible with dictionary decompression +Fixed : Decoder corruption error (#173) +Fixed : null-string roundtrip (#176) +New : benchmark mode can select directory as input +Experimental : midipix support, VMS support + +v0.6.0 +Stronger high compression modes, thanks to Przemyslaw Skibinski +API : ZSTD_getFrameParams() provides size of decompressed content +New : highest compression modes require `--ultra` command to fully unleash their capacity +Fixed : zstd cli return error code > 0 and removes dst file artifact when decompression fails, thanks to Chip Turner + +v0.5.1 +New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski +Changed : Dictionary builder integrated into libzstd and zstd cli +Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`. +Fix : high compression modes for big-endian platforms +New : zstd cli : `-t` | `--test` command + +v0.5.0 +New : dictionary builder utility +Changed : streaming & dictionary API +Improved : better compression of small data + +v0.4.7 +Improved : small compression speed improvement in HC mode +Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default +fix : bt search bug + +v0.4.6 +fix : fast compression mode on Windows +New : cmake configuration file, thanks to Artyom Dymchenko +Improved : high compression mode on repetitive data +New : block-level API +New : ZSTD_duplicateCCtx() + +v0.4.5 +new : -m/--multiple : compress/decompress multiple files + +v0.4.4 +Fixed : high compression modes for Windows 32 bits +new : external dictionary API extended to buffered mode and accessible through command line +new : windows DLL project, thanks to Christophe Chevalier + +v0.4.3 : +new : external dictionary API +new : zstd-frugal + +v0.4.2 : +Generic minor improvements for small blocks +Fixed : big-endian compatibility, by Peter Harris (#85) + +v0.4.1 +Fixed : ZSTD_LEGACY_SUPPORT=0 build mode (reported by Luben) +removed `zstd.c` + +v0.4.0 +Command line utility compatible with high compression levels +Removed zstdhc => merged into zstd +Added : ZBUFF API (see zstd_buffered.h) +Rolling buffer support + +v0.3.6 +small blocks params + +v0.3.5 +minor generic compression improvements + +v0.3.4 +Faster fast cLevels + +v0.3.3 +Small compression ratio improvement + +v0.3.2 +Fixed Visual Studio + +v0.3.1 : +Small compression ratio improvement + +v0.3 +HC mode : compression levels 2-26 + +v0.2.2 +Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier + +v0.2.1 +Fix : Read errors, advanced fuzzer tests, by Hanno Böck + +v0.2.0 +**Breaking format change** +Faster decompression speed +Can still decode v0.1 format + +v0.1.3 +fix uninitialization warning, reported by Evan Nemerson + +v0.1.2 +frame concatenation support + +v0.1.1 +fix compression bug +detects write-flush errors + +v0.1.0 +first release Modified: vendor/zstd/dist/Makefile ============================================================================== --- vendor/zstd/dist/Makefile Thu Apr 18 23:21:26 2019 (r346360) +++ vendor/zstd/dist/Makefile Fri Apr 19 00:31:04 2019 (r346361) @@ -156,7 +156,7 @@ list: done \ } | column -t -s $$'\t' -.PHONY: install clangtest armtest usan asan uasan +.PHONY: install armtest usan asan uasan install: @$(MAKE) -C $(ZSTDDIR) $@ @$(MAKE) -C $(PRGDIR) $@ @@ -188,7 +188,7 @@ gcc7build: clean .PHONY: clangbuild clangbuild: clean clang -v - CXX=clang++ CC=clang $(MAKE) all MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" + CXX=clang++ CC=clang CFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" $(MAKE) all m32build: clean gcc -v @@ -231,10 +231,6 @@ gcc5test: clean gcc6test: clean gcc-6 -v $(MAKE) all CC=gcc-6 MOREFLAGS="-Werror" - -clangtest: clean - clang -v - $(MAKE) all CXX=clang++ CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" armtest: clean $(MAKE) -C $(TESTDIR) datagen # use native, faster Modified: vendor/zstd/dist/README.md ============================================================================== --- vendor/zstd/dist/README.md Thu Apr 18 23:21:26 2019 (r346360) +++ vendor/zstd/dist/README.md Fri Apr 19 00:31:04 2019 (r346361) @@ -14,6 +14,7 @@ a list of known ports and bindings is provided on [Zst [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] +[![Build status][CirrusDevBadge]][CirrusLink] [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" [travisLink]: https://travis-ci.org/facebook/zstd @@ -21,14 +22,16 @@ a list of known ports and bindings is provided on [Zst [AppveyorLink]: https://ci.appveyor.com/project/YannCollet/zstd-p0yf0 [CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" [CircleLink]: https://circleci.com/gh/facebook/zstd +[CirrusDevBadge]: https://api.cirrus-ci.com/github/facebook/zstd.svg?branch=dev +[CirrusLink]: https://cirrus-ci.com/github/facebook/zstd ## Benchmarks For reference, several fast compression algorithms were tested and compared -on a server running Linux Debian (`Linux version 4.14.0-3-amd64`), -with a Core i7-6700K CPU @ 4.0GHz, +on a server running Arch Linux (`Linux version 5.0.5-arch1-1`), +with a Core i9-9900K CPU @ 5.0GHz, using [lzbench], an open-source in-memory benchmark by @inikep -compiled with [gcc] 7.3.0, +compiled with [gcc] 8.2.1, on the [Silesia compression corpus]. [lzbench]: https://github.com/inikep/lzbench @@ -37,14 +40,14 @@ on the [Silesia compression corpus]. | Compressor name | Ratio | Compression| Decompress.| | --------------- | ------| -----------| ---------- | -| **zstd 1.3.4 -1** | 2.877 | 470 MB/s | 1380 MB/s | -| zlib 1.2.11 -1 | 2.743 | 110 MB/s | 400 MB/s | -| brotli 1.0.2 -0 | 2.701 | 410 MB/s | 430 MB/s | -| quicklz 1.5.0 -1 | 2.238 | 550 MB/s | 710 MB/s | -| lzo1x 2.09 -1 | 2.108 | 650 MB/s | 830 MB/s | -| lz4 1.8.1 | 2.101 | 750 MB/s | 3700 MB/s | -| snappy 1.1.4 | 2.091 | 530 MB/s | 1800 MB/s | -| lzf 3.6 -1 | 2.077 | 400 MB/s | 860 MB/s | +| **zstd 1.4.0 -1** | 2.884 | 530 MB/s | 1360 MB/s | +| zlib 1.2.11 -1 | 2.743 | 110 MB/s | 440 MB/s | +| brotli 1.0.7 -0 | 2.701 | 430 MB/s | 470 MB/s | +| quicklz 1.5.0 -1 | 2.238 | 600 MB/s | 800 MB/s | +| lzo1x 2.09 -1 | 2.106 | 680 MB/s | 950 MB/s | +| lz4 1.8.3 | 2.101 | 800 MB/s | 4220 MB/s | +| snappy 1.1.4 | 2.073 | 580 MB/s | 2020 MB/s | +| lzf 3.6 -1 | 2.077 | 440 MB/s | 930 MB/s | [zlib]: http://www.zlib.net/ [LZ4]: http://www.lz4.org/ Modified: vendor/zstd/dist/contrib/adaptive-compression/Makefile ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/Makefile Thu Apr 18 23:21:26 2019 (r346360) +++ vendor/zstd/dist/contrib/adaptive-compression/Makefile Fri Apr 19 00:31:04 2019 (r346361) @@ -13,7 +13,7 @@ CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDD CFLAGS ?= -O3 CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \ -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \ - -Wstrict-prototypes -Wundef -Wformat-security \ + -Wstrict-prototypes -Wundef \ -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \ -Wredundant-decls CFLAGS += $(DEBUGFLAGS) @@ -22,10 +22,10 @@ FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITH all: adapt datagen -adapt: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c +adapt: $(ZSTD_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c adapt.c $(CC) $(FLAGS) $^ -o $@ -adapt-debug: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c +adapt-debug: $(ZSTD_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c adapt.c $(CC) $(FLAGS) -DDEBUG_MODE=2 $^ -o adapt datagen : $(PRGDIR)/datagen.c datagencli.c Modified: vendor/zstd/dist/contrib/adaptive-compression/adapt.c ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/adapt.c Thu Apr 18 23:21:26 2019 (r346360) +++ vendor/zstd/dist/contrib/adaptive-compression/adapt.c Fri Apr 19 00:31:04 2019 (r346361) @@ -13,6 +13,7 @@ #include /* memset */ #include "zstd_internal.h" #include "util.h" +#include "timefn.h" /* UTIL_time_t, UTIL_getTime, UTIL_getSpanTimeMicro */ #define DISPLAY(...) fprintf(stderr, __VA_ARGS__) #define PRINT(...) fprintf(stdout, __VA_ARGS__) Added: vendor/zstd/dist/contrib/docker/Dockerfile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/dist/contrib/docker/Dockerfile Fri Apr 19 00:31:04 2019 (r346361) @@ -0,0 +1,20 @@ +# Dockerfile +# First image to build the binary +FROM alpine as builder + +RUN apk --no-cache add make gcc libc-dev +COPY . /src +RUN mkdir /pkg && cd /src && make && make DESTDIR=/pkg install + +# Second minimal image to only keep the built binary +FROM alpine + +# Copy the built files +COPY --from=builder /pkg / + +# Copy the license as well +RUN mkdir -p /usr/local/share/licenses/zstd +COPY --from=builder /src/LICENSE /usr/local/share/licences/zstd/ + +# Just run `zstd` if no other command is given +CMD ["/usr/local/bin/zstd"] Added: vendor/zstd/dist/contrib/docker/README.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/dist/contrib/docker/README.md Fri Apr 19 00:31:04 2019 (r346361) @@ -0,0 +1,20 @@ + +## Requirement + +The `Dockerfile` script requires a version of `docker` >= 17.05 + +## Installing docker + +The official docker install docs use a ppa with a modern version available: +https://docs.docker.com/install/linux/docker-ce/ubuntu/ + +## How to run + +`docker build -t zstd .` + +## test + +``` +echo foo | docker run -i --rm zstd | docker run -i --rm zstd zstdcat +foo +``` Added: vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile Fri Apr 19 00:31:04 2019 (r346361) @@ -0,0 +1,44 @@ +ARG := + +CC ?= gcc +CFLAGS ?= -O3 +INCLUDES := -I ../randomDictBuilder -I ../../../programs -I ../../../lib/common -I ../../../lib -I ../../../lib/dictBuilder + +RANDOM_FILE := ../randomDictBuilder/random.c +IO_FILE := ../randomDictBuilder/io.c + +all: run clean + +.PHONY: run +run: benchmark + echo "Benchmarking with $(ARG)" + ./benchmark $(ARG) + +.PHONY: test +test: benchmarkTest clean + +.PHONY: benchmarkTest +benchmarkTest: benchmark test.sh + sh test.sh + +benchmark: benchmark.o io.o random.o libzstd.a + $(CC) $(CFLAGS) benchmark.o io.o random.o libzstd.a -o benchmark + +benchmark.o: benchmark.c + $(CC) $(CFLAGS) $(INCLUDES) -c benchmark.c + +random.o: $(RANDOM_FILE) + $(CC) $(CFLAGS) $(INCLUDES) -c $(RANDOM_FILE) + +io.o: $(IO_FILE) + $(CC) $(CFLAGS) $(INCLUDES) -c $(IO_FILE) + +libzstd.a: + $(MAKE) -C ../../../lib libzstd.a + mv ../../../lib/libzstd.a . + +.PHONY: clean +clean: + rm -f *.o benchmark libzstd.a + $(MAKE) -C ../../../lib clean + echo "Cleaning is completed" Added: vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/dist/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md Fri Apr 19 00:31:04 2019 (r346361) @@ -0,0 +1,849 @@ +Benchmarking Dictionary Builder + +### Permitted Argument: +Input File/Directory (in=fileName): required; file/directory used to build dictionary; if directory, will operate recursively for files inside directory; can include multiple files/directories, each following "in=" + +###Running Test: +make test + +###Usage: +Benchmark given input files: make ARG= followed by permitted arguments + +### Examples: +make ARG="in=../../../lib/dictBuilder in=../../../lib/compress" + +###Benchmarking Result: +- First Cover is optimize cover, second Cover uses optimized d and k from first one. +- For every f value of fastCover, the first one is optimize fastCover and the second one uses optimized d and k from first one. This is run for accel values from 1 to 10. +- Fourth column is chosen d and fifth column is chosen k + +github: +NODICT 0.000004 2.999642 +RANDOM 0.024560 8.791189 +LEGACY 0.727109 8.173529 +COVER 40.565676 10.652243 8 1298 +COVER 3.608284 10.652243 8 1298 +FAST f=15 a=1 4.181024 10.570882 8 1154 +FAST f=15 a=1 0.040788 10.570882 8 1154 +FAST f=15 a=2 3.548352 10.574287 6 1970 +FAST f=15 a=2 0.035535 10.574287 6 1970 +FAST f=15 a=3 3.287364 10.613950 6 1010 +FAST f=15 a=3 0.032182 10.613950 6 1010 +FAST f=15 a=4 3.184976 10.573883 6 1058 +FAST f=15 a=4 0.029878 10.573883 6 1058 +FAST f=15 a=5 3.045513 10.580640 8 1154 +FAST f=15 a=5 0.022162 10.580640 8 1154 +FAST f=15 a=6 3.003296 10.583677 6 1010 +FAST f=15 a=6 0.028091 10.583677 6 1010 +FAST f=15 a=7 2.952655 10.622551 6 1106 +FAST f=15 a=7 0.02724 10.622551 6 1106 +FAST f=15 a=8 2.945674 10.614657 6 1010 +FAST f=15 a=8 0.027264 10.614657 6 1010 +FAST f=15 a=9 3.153439 10.564018 8 1154 +FAST f=15 a=9 0.020635 10.564018 8 1154 +FAST f=15 a=10 2.950416 10.511454 6 1010 +FAST f=15 a=10 0.026606 10.511454 6 1010 +FAST f=16 a=1 3.970029 10.681035 8 1154 +FAST f=16 a=1 0.038188 10.681035 8 1154 +FAST f=16 a=2 3.422892 10.484978 6 1874 +FAST f=16 a=2 0.034702 10.484978 6 1874 +FAST f=16 a=3 3.215836 10.632631 8 1154 +FAST f=16 a=3 0.026084 10.632631 8 1154 +FAST f=16 a=4 3.081353 10.626533 6 1106 +FAST f=16 a=4 0.030032 10.626533 6 1106 +FAST f=16 a=5 3.041241 10.545027 8 1922 +FAST f=16 a=5 0.022882 10.545027 8 1922 +FAST f=16 a=6 2.989390 10.638284 6 1874 +FAST f=16 a=6 0.028308 10.638284 6 1874 +FAST f=16 a=7 3.001581 10.797136 6 1106 +FAST f=16 a=7 0.027479 10.797136 6 1106 +FAST f=16 a=8 2.984107 10.658356 8 1058 +FAST f=16 a=8 0.021099 10.658356 8 1058 +FAST f=16 a=9 2.925788 10.523869 6 1010 +FAST f=16 a=9 0.026905 10.523869 6 1010 +FAST f=16 a=10 2.889605 10.745841 6 1874 +FAST f=16 a=10 0.026846 10.745841 6 1874 +FAST f=17 a=1 4.031953 10.672080 8 1202 +FAST f=17 a=1 0.040658 10.672080 8 1202 +FAST f=17 a=2 3.458107 10.589352 8 1106 +FAST f=17 a=2 0.02926 10.589352 8 1106 +FAST f=17 a=3 3.291189 10.662714 8 1154 +FAST f=17 a=3 0.026531 10.662714 8 1154 +FAST f=17 a=4 3.154950 10.549456 8 1346 +FAST f=17 a=4 0.024991 10.549456 8 1346 +FAST f=17 a=5 3.092271 10.541670 6 1202 +FAST f=17 a=5 0.038285 10.541670 6 1202 +FAST f=17 a=6 3.166146 10.729112 6 1874 +FAST f=17 a=6 0.038217 10.729112 6 1874 +FAST f=17 a=7 3.035467 10.810485 6 1106 +FAST f=17 a=7 0.036655 10.810485 6 1106 +FAST f=17 a=8 3.035668 10.530532 6 1058 +FAST f=17 a=8 0.037715 10.530532 6 1058 +FAST f=17 a=9 2.987917 10.589802 8 1922 +FAST f=17 a=9 0.02217 10.589802 8 1922 +FAST f=17 a=10 2.981647 10.722579 8 1106 +FAST f=17 a=10 0.021948 10.722579 8 1106 +FAST f=18 a=1 4.067144 10.634943 8 1154 +FAST f=18 a=1 0.041386 10.634943 8 1154 +FAST f=18 a=2 3.507377 10.546230 6 1970 +FAST f=18 a=2 0.037572 10.546230 6 1970 +FAST f=18 a=3 3.323015 10.648061 8 1154 +FAST f=18 a=3 0.028306 10.648061 8 1154 +FAST f=18 a=4 3.216735 10.705402 6 1010 +FAST f=18 a=4 0.030755 10.705402 6 1010 +FAST f=18 a=5 3.175794 10.588154 8 1874 +FAST f=18 a=5 0.025315 10.588154 8 1874 +FAST f=18 a=6 3.127459 10.751104 8 1106 +FAST f=18 a=6 0.023897 10.751104 8 1106 +FAST f=18 a=7 3.083017 10.780402 6 1106 +FAST f=18 a=7 0.029158 10.780402 6 1106 +FAST f=18 a=8 3.069700 10.547226 8 1346 +FAST f=18 a=8 0.024046 10.547226 8 1346 +FAST f=18 a=9 3.056591 10.674759 6 1010 +FAST f=18 a=9 0.028496 10.674759 6 1010 +FAST f=18 a=10 3.063588 10.737578 8 1106 +FAST f=18 a=10 0.023033 10.737578 8 1106 +FAST f=19 a=1 4.164041 10.650333 8 1154 +FAST f=19 a=1 0.042906 10.650333 8 1154 +FAST f=19 a=2 3.585409 10.577066 6 1058 +FAST f=19 a=2 0.038994 10.577066 6 1058 +FAST f=19 a=3 3.439643 10.639403 8 1154 +FAST f=19 a=3 0.028427 10.639403 8 1154 +FAST f=19 a=4 3.268869 10.554410 8 1298 +FAST f=19 a=4 0.026866 10.554410 8 1298 +FAST f=19 a=5 3.238225 10.615109 6 1010 +FAST f=19 a=5 0.03078 10.615109 6 1010 +FAST f=19 a=6 3.199558 10.609782 6 1874 +FAST f=19 a=6 0.030099 10.609782 6 1874 +FAST f=19 a=7 3.132395 10.794753 6 1106 +FAST f=19 a=7 0.028964 10.794753 6 1106 +FAST f=19 a=8 3.148446 10.554842 8 1298 +FAST f=19 a=8 0.024277 10.554842 8 1298 +FAST f=19 a=9 3.108324 10.668763 6 1010 +FAST f=19 a=9 0.02896 10.668763 6 1010 +FAST f=19 a=10 3.159863 10.757347 8 1106 +FAST f=19 a=10 0.023351 10.757347 8 1106 +FAST f=20 a=1 4.462698 10.661788 8 1154 +FAST f=20 a=1 0.047174 10.661788 8 1154 +FAST f=20 a=2 3.820269 10.678612 6 1106 +FAST f=20 a=2 0.040807 10.678612 6 1106 +FAST f=20 a=3 3.644955 10.648424 8 1154 +FAST f=20 a=3 0.031398 10.648424 8 1154 +FAST f=20 a=4 3.546257 10.559756 8 1298 +FAST f=20 a=4 0.029856 10.559756 8 1298 +FAST f=20 a=5 3.485248 10.646637 6 1010 +FAST f=20 a=5 0.033756 10.646637 6 1010 +FAST f=20 a=6 3.490438 10.775824 8 1106 +FAST f=20 a=6 0.028338 10.775824 8 1106 +FAST f=20 a=7 3.631289 10.801795 6 1106 +FAST f=20 a=7 0.035228 10.801795 6 1106 +FAST f=20 a=8 3.758936 10.545116 8 1346 +FAST f=20 a=8 0.027495 10.545116 8 1346 +FAST f=20 a=9 3.707024 10.677454 6 1010 +FAST f=20 a=9 0.031326 10.677454 6 1010 +FAST f=20 a=10 3.586593 10.756017 8 1106 +FAST f=20 a=10 0.027122 10.756017 8 1106 +FAST f=21 a=1 5.701396 10.655398 8 1154 +FAST f=21 a=1 0.067744 10.655398 8 1154 +FAST f=21 a=2 5.270542 10.650743 6 1106 +FAST f=21 a=2 0.052999 10.650743 6 1106 +FAST f=21 a=3 4.945294 10.652380 8 1154 +FAST f=21 a=3 0.052678 10.652380 8 1154 +FAST f=21 a=4 4.894079 10.543185 8 1298 +FAST f=21 a=4 0.04997 10.543185 8 1298 +FAST f=21 a=5 4.785417 10.630321 6 1010 +FAST f=21 a=5 0.045294 10.630321 6 1010 +FAST f=21 a=6 4.789381 10.664477 6 1874 +FAST f=21 a=6 0.046578 10.664477 6 1874 +FAST f=21 a=7 4.302955 10.805179 6 1106 +FAST f=21 a=7 0.041205 10.805179 6 1106 +FAST f=21 a=8 4.034630 10.551211 8 1298 +FAST f=21 a=8 0.040121 10.551211 8 1298 +FAST f=21 a=9 4.523868 10.799114 6 1010 +FAST f=21 a=9 0.043592 10.799114 6 1010 +FAST f=21 a=10 4.760736 10.750255 8 1106 +FAST f=21 a=10 0.043483 10.750255 8 1106 +FAST f=22 a=1 6.743064 10.640537 8 1154 +FAST f=22 a=1 0.086967 10.640537 8 1154 +FAST f=22 a=2 6.121739 10.626638 6 1970 +FAST f=22 a=2 0.066337 10.626638 6 1970 +FAST f=22 a=3 5.248851 10.640688 8 1154 +FAST f=22 a=3 0.054935 10.640688 8 1154 +FAST f=22 a=4 5.436579 10.588333 8 1298 +FAST f=22 a=4 0.064113 10.588333 8 1298 +FAST f=22 a=5 5.812815 10.652653 6 1010 +FAST f=22 a=5 0.058189 10.652653 6 1010 +FAST f=22 a=6 5.745472 10.666437 6 1874 +FAST f=22 a=6 0.057188 10.666437 6 1874 +FAST f=22 a=7 5.716393 10.806911 6 1106 +FAST f=22 a=7 0.056 10.806911 6 1106 +FAST f=22 a=8 5.698799 10.530784 8 1298 +FAST f=22 a=8 0.0583 10.530784 8 1298 +FAST f=22 a=9 5.710533 10.777391 6 1010 +FAST f=22 a=9 0.054945 10.777391 6 1010 +FAST f=22 a=10 5.685395 10.745023 8 1106 +FAST f=22 a=10 0.056526 10.745023 8 1106 +FAST f=23 a=1 7.836923 10.638828 8 1154 +FAST f=23 a=1 0.099522 10.638828 8 1154 +FAST f=23 a=2 6.627834 10.631061 6 1970 +FAST f=23 a=2 0.066769 10.631061 6 1970 +FAST f=23 a=3 5.602533 10.647288 8 1154 +FAST f=23 a=3 0.064513 10.647288 8 1154 +FAST f=23 a=4 6.005580 10.568747 8 1298 +FAST f=23 a=4 0.062022 10.568747 8 1298 +FAST f=23 a=5 5.481816 10.676921 6 1010 +FAST f=23 a=5 0.058959 10.676921 6 1010 +FAST f=23 a=6 5.460444 10.666194 6 1874 +FAST f=23 a=6 0.057687 10.666194 6 1874 +FAST f=23 a=7 5.659822 10.800377 6 1106 +FAST f=23 a=7 0.06783 10.800377 6 1106 +FAST f=23 a=8 6.826940 10.522167 8 1298 +FAST f=23 a=8 0.070533 10.522167 8 1298 +FAST f=23 a=9 6.804757 10.577799 8 1682 +FAST f=23 a=9 0.069949 10.577799 8 1682 +FAST f=23 a=10 6.774933 10.742093 8 1106 +FAST f=23 a=10 0.068395 10.742093 8 1106 +FAST f=24 a=1 8.444110 10.632783 8 1154 +FAST f=24 a=1 0.094357 10.632783 8 1154 +FAST f=24 a=2 7.289578 10.631061 6 1970 +FAST f=24 a=2 0.098515 10.631061 6 1970 +FAST f=24 a=3 8.619780 10.646289 8 1154 +FAST f=24 a=3 0.098041 10.646289 8 1154 +FAST f=24 a=4 8.508455 10.555199 8 1298 +FAST f=24 a=4 0.093885 10.555199 8 1298 +FAST f=24 a=5 8.471145 10.674363 6 1010 +FAST f=24 a=5 0.088676 10.674363 6 1010 +FAST f=24 a=6 8.426727 10.667228 6 1874 +FAST f=24 a=6 0.087247 10.667228 6 1874 +FAST f=24 a=7 8.356826 10.803027 6 1106 +FAST f=24 a=7 0.085835 10.803027 6 1106 +FAST f=24 a=8 6.756811 10.522049 8 1298 +FAST f=24 a=8 0.07107 10.522049 8 1298 +FAST f=24 a=9 6.548169 10.571882 8 1682 +FAST f=24 a=9 0.0713 10.571882 8 1682 +FAST f=24 a=10 8.238079 10.736453 8 1106 +FAST f=24 a=10 0.07004 10.736453 8 1106 + + +hg-commands: +NODICT 0.000005 2.425276 +RANDOM 0.046332 3.490331 +LEGACY 0.720351 3.911682 +COVER 45.507731 4.132653 8 386 +COVER 1.868810 4.132653 8 386 +FAST f=15 a=1 4.561427 3.866894 8 1202 +FAST f=15 a=1 0.048946 3.866894 8 1202 +FAST f=15 a=2 3.574462 3.892119 8 1538 +FAST f=15 a=2 0.033677 3.892119 8 1538 +FAST f=15 a=3 3.230227 3.888791 6 1346 +FAST f=15 a=3 0.034312 3.888791 6 1346 +FAST f=15 a=4 3.042388 3.899739 8 1010 +FAST f=15 a=4 0.024307 3.899739 8 1010 +FAST f=15 a=5 2.800148 3.896220 8 818 +FAST f=15 a=5 0.022331 3.896220 8 818 +FAST f=15 a=6 2.706518 3.882039 8 578 +FAST f=15 a=6 0.020955 3.882039 8 578 +FAST f=15 a=7 2.701820 3.885430 6 866 +FAST f=15 a=7 0.026074 3.885430 6 866 +FAST f=15 a=8 2.604445 3.906932 8 1826 +FAST f=15 a=8 0.021789 3.906932 8 1826 +FAST f=15 a=9 2.598568 3.870324 6 1682 +FAST f=15 a=9 0.026004 3.870324 6 1682 +FAST f=15 a=10 2.575920 3.920783 8 1442 +FAST f=15 a=10 0.020228 3.920783 8 1442 +FAST f=16 a=1 4.630623 4.001430 8 770 +FAST f=16 a=1 0.047497 4.001430 8 770 +FAST f=16 a=2 3.674721 3.974431 8 1874 +FAST f=16 a=2 0.035761 3.974431 8 1874 +FAST f=16 a=3 3.338384 3.978703 8 1010 +FAST f=16 a=3 0.029436 3.978703 8 1010 +FAST f=16 a=4 3.004412 3.983035 8 1010 +FAST f=16 a=4 0.025744 3.983035 8 1010 +FAST f=16 a=5 2.881892 3.987710 8 770 +FAST f=16 a=5 0.023211 3.987710 8 770 +FAST f=16 a=6 2.807410 3.952717 8 1298 +FAST f=16 a=6 0.023199 3.952717 8 1298 +FAST f=16 a=7 2.819623 3.994627 8 770 +FAST f=16 a=7 0.021806 3.994627 8 770 +FAST f=16 a=8 2.740092 3.954032 8 1826 +FAST f=16 a=8 0.0226 3.954032 8 1826 +FAST f=16 a=9 2.682564 3.969879 6 1442 +FAST f=16 a=9 0.026324 3.969879 6 1442 +FAST f=16 a=10 2.657959 3.969755 8 674 +FAST f=16 a=10 0.020413 3.969755 8 674 +FAST f=17 a=1 4.729228 4.046000 8 530 +FAST f=17 a=1 0.049703 4.046000 8 530 +FAST f=17 a=2 3.764510 3.991519 8 1970 +FAST f=17 a=2 0.038195 3.991519 8 1970 +FAST f=17 a=3 3.416992 4.006296 6 914 +FAST f=17 a=3 0.036244 4.006296 6 914 +FAST f=17 a=4 3.145626 3.979182 8 1970 +FAST f=17 a=4 0.028676 3.979182 8 1970 +FAST f=17 a=5 2.995070 4.050070 8 770 +FAST f=17 a=5 0.025707 4.050070 8 770 +FAST f=17 a=6 2.911833 4.040024 8 770 +FAST f=17 a=6 0.02453 4.040024 8 770 +FAST f=17 a=7 2.894796 4.015884 8 818 +FAST f=17 a=7 0.023956 4.015884 8 818 +FAST f=17 a=8 2.789962 4.039303 8 530 +FAST f=17 a=8 0.023219 4.039303 8 530 +FAST f=17 a=9 2.787625 3.996762 8 1634 +FAST f=17 a=9 0.023651 3.996762 8 1634 +FAST f=17 a=10 2.754796 4.005059 8 1058 +FAST f=17 a=10 0.022537 4.005059 8 1058 +FAST f=18 a=1 4.779117 4.038214 8 242 +FAST f=18 a=1 0.048814 4.038214 8 242 +FAST f=18 a=2 3.829753 4.045768 8 722 +FAST f=18 a=2 0.036541 4.045768 8 722 +FAST f=18 a=3 3.495053 4.021497 8 770 +FAST f=18 a=3 0.032648 4.021497 8 770 +FAST f=18 a=4 3.221395 4.039623 8 770 +FAST f=18 a=4 0.027818 4.039623 8 770 +FAST f=18 a=5 3.059369 4.050414 8 530 +FAST f=18 a=5 0.026296 4.050414 8 530 +FAST f=18 a=6 3.019292 4.010714 6 962 +FAST f=18 a=6 0.031104 4.010714 6 962 +FAST f=18 a=7 2.949322 4.031439 6 770 +FAST f=18 a=7 0.030745 4.031439 6 770 +FAST f=18 a=8 2.876425 4.032088 6 386 +FAST f=18 a=8 0.027407 4.032088 6 386 +FAST f=18 a=9 2.850958 4.053372 8 674 +FAST f=18 a=9 0.023799 4.053372 8 674 +FAST f=18 a=10 2.884352 4.020148 8 1730 +FAST f=18 a=10 0.024401 4.020148 8 1730 +FAST f=19 a=1 4.815669 4.061203 8 674 +FAST f=19 a=1 0.051425 4.061203 8 674 +FAST f=19 a=2 3.951356 4.013822 8 1442 +FAST f=19 a=2 0.039968 4.013822 8 1442 +FAST f=19 a=3 3.554682 4.050425 8 722 +FAST f=19 a=3 0.032725 4.050425 8 722 +FAST f=19 a=4 3.242585 4.054677 8 722 +FAST f=19 a=4 0.028194 4.054677 8 722 +FAST f=19 a=5 3.105909 4.064524 8 818 *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***