From owner-svn-src-vendor@freebsd.org Wed Nov 6 06:42:03 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1134C1AB1B9; Wed, 6 Nov 2019 06:42:03 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 477H570D0Qz4tnq; Wed, 6 Nov 2019 06:42:03 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DFD0421242; Wed, 6 Nov 2019 06:42:02 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA66g27k068331; Wed, 6 Nov 2019 06:42:02 GMT (envelope-from cem@FreeBSD.org) Received: (from cem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA66g0dn068321; Wed, 6 Nov 2019 06:42:00 GMT (envelope-from cem@FreeBSD.org) Message-Id: <201911060642.xA66g0dn068321@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org using -f From: Conrad Meyer Date: Wed, 6 Nov 2019 06:42:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354371 - in vendor/zstd/dist: . contrib/adaptive-compression contrib/gen_html contrib/pzstd contrib/seekable_format/examples doc doc/educational_decoder examples lib lib/common lib/com... X-SVN-Group: vendor X-SVN-Commit-Author: cem X-SVN-Commit-Paths: in vendor/zstd/dist: . contrib/adaptive-compression contrib/gen_html contrib/pzstd contrib/seekable_format/examples doc doc/educational_decoder examples lib lib/common lib/compress lib/decompress lib/... X-SVN-Commit-Revision: 354371 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2019 06:42:03 -0000 Author: cem Date: Wed Nov 6 06:42:00 2019 New Revision: 354371 URL: https://svnweb.freebsd.org/changeset/base/354371 Log: Import Zstd 1.4.4 Added: vendor/zstd/dist/lib/compress/zstd_cwksp.h (contents, props changed) vendor/zstd/dist/tests/fuzz/dictionary_loader.c (contents, props changed) vendor/zstd/dist/tests/fuzz/fuzz_data_producer.c (contents, props changed) vendor/zstd/dist/tests/fuzz/fuzz_data_producer.h (contents, props changed) vendor/zstd/dist/tests/golden-compression/ vendor/zstd/dist/tests/golden-compression/huffman-compressed-larger - copied unchanged from r354370, vendor/zstd/dist/tests/files/huffman-compressed-larger vendor/zstd/dist/tests/golden-decompression/ vendor/zstd/dist/tests/golden-decompression/rle-first-block.zst (contents, props changed) Deleted: vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/dist/contrib/adaptive-compression/README.md vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/dist/contrib/adaptive-compression/datagencli.c vendor/zstd/dist/contrib/adaptive-compression/test-correctness.sh vendor/zstd/dist/contrib/adaptive-compression/test-performance.sh vendor/zstd/dist/contrib/gen_html/.gitignore vendor/zstd/dist/contrib/pzstd/.gitignore vendor/zstd/dist/contrib/seekable_format/examples/.gitignore vendor/zstd/dist/examples/.gitignore vendor/zstd/dist/lib/.gitignore vendor/zstd/dist/programs/.gitignore vendor/zstd/dist/tests/.gitignore vendor/zstd/dist/tests/files/huffman-compressed-larger vendor/zstd/dist/zlibWrapper/.gitignore Modified: vendor/zstd/dist/CHANGELOG vendor/zstd/dist/Makefile vendor/zstd/dist/README.md vendor/zstd/dist/appveyor.yml vendor/zstd/dist/doc/educational_decoder/Makefile vendor/zstd/dist/doc/educational_decoder/harness.c vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c vendor/zstd/dist/doc/educational_decoder/zstd_decompress.h vendor/zstd/dist/doc/zstd_compression_format.md vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/dist/examples/streaming_compression.c vendor/zstd/dist/examples/streaming_decompression.c vendor/zstd/dist/lib/Makefile vendor/zstd/dist/lib/README.md vendor/zstd/dist/lib/common/bitstream.h vendor/zstd/dist/lib/common/compiler.h vendor/zstd/dist/lib/common/fse.h vendor/zstd/dist/lib/common/fse_decompress.c vendor/zstd/dist/lib/common/mem.h vendor/zstd/dist/lib/common/pool.c vendor/zstd/dist/lib/common/threading.c vendor/zstd/dist/lib/common/threading.h vendor/zstd/dist/lib/common/xxhash.c vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/dist/lib/compress/zstd_compress_literals.c vendor/zstd/dist/lib/compress/zstd_compress_literals.h vendor/zstd/dist/lib/compress/zstd_compress_sequences.c vendor/zstd/dist/lib/compress/zstd_compress_sequences.h vendor/zstd/dist/lib/compress/zstd_double_fast.c vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/dist/lib/compress/zstd_lazy.c vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/dist/lib/decompress/huf_decompress.c vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/dist/lib/decompress/zstd_decompress_block.c vendor/zstd/dist/lib/deprecated/zbuff.h vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/dist/lib/dictBuilder/zdict.c vendor/zstd/dist/lib/legacy/zstd_v01.c vendor/zstd/dist/lib/legacy/zstd_v02.c vendor/zstd/dist/lib/legacy/zstd_v03.c vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/dist/lib/legacy/zstd_v06.c vendor/zstd/dist/lib/legacy/zstd_v07.c vendor/zstd/dist/lib/libzstd.pc.in vendor/zstd/dist/lib/zstd.h vendor/zstd/dist/programs/README.md vendor/zstd/dist/programs/benchzstd.c vendor/zstd/dist/programs/benchzstd.h vendor/zstd/dist/programs/datagen.c vendor/zstd/dist/programs/dibio.c vendor/zstd/dist/programs/fileio.c vendor/zstd/dist/programs/fileio.h vendor/zstd/dist/programs/platform.h vendor/zstd/dist/programs/timefn.h vendor/zstd/dist/programs/util.c vendor/zstd/dist/programs/util.h vendor/zstd/dist/programs/zstd.1 vendor/zstd/dist/programs/zstd.1.md vendor/zstd/dist/programs/zstdcli.c vendor/zstd/dist/programs/zstdgrep.1 vendor/zstd/dist/programs/zstdless.1 vendor/zstd/dist/tests/Makefile vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/dist/tests/fullbench.c vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/dist/tests/fuzz/README.md vendor/zstd/dist/tests/fuzz/block_decompress.c vendor/zstd/dist/tests/fuzz/block_round_trip.c vendor/zstd/dist/tests/fuzz/dictionary_decompress.c vendor/zstd/dist/tests/fuzz/dictionary_round_trip.c vendor/zstd/dist/tests/fuzz/fuzz.h vendor/zstd/dist/tests/fuzz/fuzz.py vendor/zstd/dist/tests/fuzz/fuzz_helpers.h vendor/zstd/dist/tests/fuzz/regression_driver.c vendor/zstd/dist/tests/fuzz/simple_compress.c vendor/zstd/dist/tests/fuzz/simple_decompress.c vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/dist/tests/fuzz/stream_decompress.c vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/dist/tests/fuzz/zstd_frame_info.c vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/dist/tests/fuzz/zstd_helpers.h vendor/zstd/dist/tests/fuzzer.c vendor/zstd/dist/tests/playTests.sh vendor/zstd/dist/tests/poolTests.c vendor/zstd/dist/tests/regression/method.c vendor/zstd/dist/tests/regression/results.csv vendor/zstd/dist/tests/zbufftest.c vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/dist/zlibWrapper/examples/fitblk.c vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/dist/zlibWrapper/gzclose.c vendor/zstd/dist/zlibWrapper/gzlib.c vendor/zstd/dist/zlibWrapper/gzread.c vendor/zstd/dist/zlibWrapper/gzwrite.c vendor/zstd/dist/zlibWrapper/zstd_zlibwrapper.c Modified: vendor/zstd/dist/CHANGELOG ============================================================================== --- vendor/zstd/dist/CHANGELOG Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/CHANGELOG Wed Nov 6 06:42:00 2019 (r354371) @@ -1,3 +1,34 @@ +v1.4.4 +perf: Improved decompression speed, by > 10%, by @terrelln +perf: Better compression speed when re-using a context, by @felixhandte +perf: Fix compression ratio when compressing large files with small dictionary, by @senhuang42 +perf: zstd reference encoder can generate RLE blocks, by @bimbashrestha +perf: minor generic speed optimization, by @davidbolvansky +api: new ability to extract sequences from the parser for analysis, by @bimbashrestha +api: fixed decoding of magic-less frames, by @terrelln +api: fixed ZSTD_initCStream_advanced() performance with fast modes, reported by @QrczakMK +cli: Named pipes support, by @bimbashrestha +cli: short tar's extension support, by @stokito +cli: command --output-dir-flat= , generates target files into requested directory, by @senhuang42 +cli: commands --stream-size=# and --size-hint=#, by @nmagerko +cli: command --exclude-compressed, by @shashank0791 +cli: faster `-t` test mode +cli: improved some error messages, by @vangyzen +cli: rare deadlock condition within dictionary builder, by @terrelln +build: single-file decoder with emscripten compilation script, by @cwoffenden +build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive +build: fixed deprecation warning for certain gcc version, reported by @jasonma163 +build: fix compilation on old gcc versions, by @cemeyer +build: improved installation directories for cmake script, by Dmitri Shubin +pack: modified pkgconfig, for better integration into openwrt, requested by @neheb +misc: Improved documentation : ZSTD_CLEVEL, DYNAMIC_BMI2, ZSTD_CDict, function deprecation, zstd format +misc: fixed educational decoder : accept larger literals section, and removed UNALIGNED() macro + +v1.4.3 +bug: Fix Dictionary Compression Ratio Regression by @cyan4973 (#1709) +bug: Fix Buffer Overflow in legacy v0.3 decompression by @felixhandte (#1722) +build: Add support for IAR C/C++ Compiler for Arm by @joseph0918 (#1705) + v1.4.2 bug: Fix bug in zstd-0.5 decoder by @terrelln (#1696) bug: Fix seekable decompression in-memory API by @iburinoc (#1695) Modified: vendor/zstd/dist/Makefile ============================================================================== --- vendor/zstd/dist/Makefile Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/Makefile Wed Nov 6 06:42:00 2019 (r354371) @@ -69,6 +69,7 @@ test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werr test: MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants $(MAKE) -C $(TESTDIR) $@ + ZSTD=../../programs/zstd $(MAKE) -C doc/educational_decoder test ## shortest: same as `make check` .PHONY: shortest @@ -99,8 +100,8 @@ man: contrib: lib $(MAKE) -C contrib/pzstd all $(MAKE) -C contrib/seekable_format/examples all - $(MAKE) -C contrib/adaptive-compression all $(MAKE) -C contrib/largeNbDicts all + cd contrib/single_file_decoder/ ; ./build_test.sh .PHONY: cleanTabs cleanTabs: @@ -116,7 +117,6 @@ clean: @$(MAKE) -C contrib/gen_html $@ > $(VOID) @$(MAKE) -C contrib/pzstd $@ > $(VOID) @$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID) - @$(MAKE) -C contrib/adaptive-compression $@ > $(VOID) @$(MAKE) -C contrib/largeNbDicts $@ > $(VOID) @$(RM) zstd$(EXT) zstdmt$(EXT) tmp* @$(RM) -r lz4 Modified: vendor/zstd/dist/README.md ============================================================================== --- vendor/zstd/dist/README.md Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/README.md Wed Nov 6 06:42:00 2019 (r354371) @@ -15,6 +15,7 @@ a list of known ports and bindings is provided on [Zst [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] [![Build status][CirrusDevBadge]][CirrusLink] +[![Fuzzing Status][OSSFuzzBadge]][OSSFuzzLink] [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" [travisLink]: https://travis-ci.org/facebook/zstd @@ -24,6 +25,8 @@ a list of known ports and bindings is provided on [Zst [CircleLink]: https://circleci.com/gh/facebook/zstd [CirrusDevBadge]: https://api.cirrus-ci.com/github/facebook/zstd.svg?branch=dev [CirrusLink]: https://cirrus-ci.com/github/facebook/zstd +[OSSFuzzBadge]: https://oss-fuzz-build-logs.storage.googleapis.com/badges/zstd.svg +[OSSFuzzLink]: https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zstd ## Benchmarks Modified: vendor/zstd/dist/appveyor.yml ============================================================================== --- vendor/zstd/dist/appveyor.yml Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/appveyor.yml Wed Nov 6 06:42:00 2019 (r354371) @@ -1,3 +1,7 @@ +# Following tests are run _only_ on master branch +# To reproduce these tests, it's possible to push into a branch `appveyorTest` +# or a branch `visual*`, they will intentionnally trigger `master` tests + - version: 1.0.{build} branches: @@ -169,13 +173,16 @@ sh -e playTests.sh --test-large-data && fullbench.exe -i1 && fullbench.exe -i1 -P0 && - fuzzer_VS2008_%PLATFORM%_Release.exe %FUZZERTEST% && - fuzzer_VS2010_%PLATFORM%_Release.exe %FUZZERTEST% && fuzzer_VS2012_%PLATFORM%_Release.exe %FUZZERTEST% && fuzzer_VS2013_%PLATFORM%_Release.exe %FUZZERTEST% && fuzzer_VS2015_%PLATFORM%_Release.exe %FUZZERTEST% ) + +# The following tests are for regular pushes +# into `dev` or some feature branch +# There run less tests, for shorter feedback loop + - version: 1.0.{build} environment: @@ -248,4 +255,12 @@ MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2015_%PLATFORM%_%CONFIGURATION%.exe && COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe tests\ + ) + + + test_script: + - ECHO Testing %COMPILER% %PLATFORM% %CONFIGURATION% + - if [%HOST%]==[mingw] ( + set "CC=%COMPILER%" && + make check ) Modified: vendor/zstd/dist/doc/educational_decoder/Makefile ============================================================================== --- vendor/zstd/dist/doc/educational_decoder/Makefile Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/educational_decoder/Makefile Wed Nov 6 06:42:00 2019 (r354371) @@ -1,15 +1,33 @@ +# ################################################################ +# Copyright (c) 2016-present, Yann Collet, Facebook, Inc. +# All rights reserved. +# +# This source code is licensed under both the BSD-style license (found in the +# LICENSE file in the root directory of this source tree) and the GPLv2 (found +# in the COPYING file in the root directory of this source tree). +# ################################################################ + +ZSTD ?= zstd # note: requires zstd installation on local system + +UNAME?= $(shell uname) +ifeq ($(UNAME), SunOS) +DIFF ?= gdiff +else +DIFF ?= diff +endif + HARNESS_FILES=*.c MULTITHREAD_LDFLAGS = -pthread DEBUGFLAGS= -g -DZSTD_DEBUG=1 CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \ -I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR) -CFLAGS ?= -O3 +CFLAGS ?= -O2 CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \ - -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \ - -Wstrict-prototypes -Wundef \ + -Wstrict-aliasing=1 -Wswitch-enum \ + -Wredundant-decls -Wstrict-prototypes -Wundef \ -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \ - -Wredundant-decls + -std=c99 CFLAGS += $(DEBUGFLAGS) CFLAGS += $(MOREFLAGS) FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITHREAD_LDFLAGS) @@ -18,17 +36,27 @@ harness: $(HARNESS_FILES) $(CC) $(FLAGS) $^ -o $@ clean: - @$(RM) -f harness - @$(RM) -rf harness.dSYM + @$(RM) harness + @$(RM) -rf harness.dSYM # MacOS specific test: harness - @zstd README.md -o tmp.zst + # + # Testing single-file decompression with educational decoder + # + @$(ZSTD) -f README.md -o tmp.zst @./harness tmp.zst tmp - @diff -s tmp README.md - @$(RM) -f tmp* - @zstd --train harness.c zstd_decompress.c zstd_decompress.h README.md - @zstd -D dictionary README.md -o tmp.zst + @$(DIFF) -s tmp README.md + @$(RM) tmp* + # + # Testing dictionary decompression with education decoder + # + # note : files are presented multiple for training, to reach minimum threshold + @$(ZSTD) --train harness.c zstd_decompress.c zstd_decompress.h README.md \ + harness.c zstd_decompress.c zstd_decompress.h README.md \ + harness.c zstd_decompress.c zstd_decompress.h README.md \ + -o dictionary + @$(ZSTD) -f README.md -D dictionary -o tmp.zst @./harness tmp.zst tmp dictionary - @diff -s tmp README.md - @$(RM) -f tmp* dictionary - @make clean + @$(DIFF) -s tmp README.md + @$(RM) tmp* dictionary + @$(MAKE) clean Modified: vendor/zstd/dist/doc/educational_decoder/harness.c ============================================================================== --- vendor/zstd/dist/doc/educational_decoder/harness.c Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/educational_decoder/harness.c Wed Nov 6 06:42:00 2019 (r354371) @@ -21,88 +21,90 @@ typedef unsigned char u8; // Protect against allocating too much memory for output #define MAX_OUTPUT_SIZE ((size_t)1024 * 1024 * 1024) -u8 *input; -u8 *output; -u8 *dict; - -size_t read_file(const char *path, u8 **ptr) { - FILE *f = fopen(path, "rb"); +static size_t read_file(const char *path, u8 **ptr) +{ + FILE* const f = fopen(path, "rb"); if (!f) { - fprintf(stderr, "failed to open file %s\n", path); + fprintf(stderr, "failed to open file %s \n", path); exit(1); } fseek(f, 0L, SEEK_END); - size_t size = ftell(f); + size_t const size = (size_t)ftell(f); rewind(f); *ptr = malloc(size); if (!ptr) { - fprintf(stderr, "failed to allocate memory to hold %s\n", path); + fprintf(stderr, "failed to allocate memory to hold %s \n", path); exit(1); } - size_t pos = 0; - while (!feof(f)) { - size_t read = fread(&(*ptr)[pos], 1, size, f); - if (ferror(f)) { - fprintf(stderr, "error while reading file %s\n", path); - exit(1); - } - pos += read; + size_t const read = fread(*ptr, 1, size, f); + if (read != size) { /* must read everything in one pass */ + fprintf(stderr, "error while reading file %s \n", path); + exit(1); } fclose(f); - return pos; + return read; } -void write_file(const char *path, const u8 *ptr, size_t size) { - FILE *f = fopen(path, "wb"); +static void write_file(const char *path, const u8 *ptr, size_t size) +{ + FILE* const f = fopen(path, "wb"); + if (!f) { + fprintf(stderr, "failed to open file %s \n", path); + exit(1); + } size_t written = 0; while (written < size) { - written += fwrite(&ptr[written], 1, size, f); + written += fwrite(ptr+written, 1, size, f); if (ferror(f)) { fprintf(stderr, "error while writing file %s\n", path); exit(1); - } - } + } } fclose(f); } -int main(int argc, char **argv) { +int main(int argc, char **argv) +{ if (argc < 3) { - fprintf(stderr, "usage: %s [dictionary]\n", + fprintf(stderr, "usage: %s [dictionary] \n", argv[0]); return 1; } - size_t input_size = read_file(argv[1], &input); + u8* input; + size_t const input_size = read_file(argv[1], &input); + + u8* dict = NULL; size_t dict_size = 0; if (argc >= 4) { dict_size = read_file(argv[3], &dict); } - size_t decompressed_size = ZSTD_get_decompressed_size(input, input_size); - if (decompressed_size == (size_t)-1) { - decompressed_size = MAX_COMPRESSION_RATIO * input_size; + size_t out_capacity = ZSTD_get_decompressed_size(input, input_size); + if (out_capacity == (size_t)-1) { + out_capacity = MAX_COMPRESSION_RATIO * input_size; fprintf(stderr, "WARNING: Compressed data does not contain " "decompressed size, going to assume the compression " "ratio is at most %d (decompressed size of at most " - "%zu)\n", - MAX_COMPRESSION_RATIO, decompressed_size); + "%u) \n", + MAX_COMPRESSION_RATIO, (unsigned)out_capacity); } - if (decompressed_size > MAX_OUTPUT_SIZE) { + if (out_capacity > MAX_OUTPUT_SIZE) { fprintf(stderr, - "Required output size too large for this implementation\n"); + "Required output size too large for this implementation \n"); return 1; } - output = malloc(decompressed_size); + + u8* const output = malloc(out_capacity); if (!output) { - fprintf(stderr, "failed to allocate memory\n"); + fprintf(stderr, "failed to allocate memory \n"); return 1; } @@ -110,16 +112,17 @@ int main(int argc, char **argv) { if (dict) { parse_dictionary(parsed_dict, dict, dict_size); } - size_t decompressed = - ZSTD_decompress_with_dict(output, decompressed_size, - input, input_size, parsed_dict); + size_t const decompressed_size = + ZSTD_decompress_with_dict(output, out_capacity, + input, input_size, + parsed_dict); free_dictionary(parsed_dict); - write_file(argv[2], output, decompressed); + write_file(argv[2], output, decompressed_size); free(input); free(output); free(dict); - input = output = dict = NULL; + return 0; } Modified: vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c ============================================================================== --- vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c Wed Nov 6 06:42:00 2019 (r354371) @@ -395,7 +395,7 @@ size_t ZSTD_decompress_with_dict(void *const dst, cons /* this decoder assumes decompression of a single frame */ decode_frame(&out, &in, parsed_dict); - return out.ptr - (u8 *)dst; + return (size_t)(out.ptr - (u8 *)dst); } /******* FRAME DECODING ******************************************************/ @@ -416,7 +416,7 @@ static void decompress_data(frame_context_t *const ctx static void decode_frame(ostream_t *const out, istream_t *const in, const dictionary_t *const dict) { - const u32 magic_number = IO_read_bits(in, 32); + const u32 magic_number = (u32)IO_read_bits(in, 32); // Zstandard frame // // "Magic_Number @@ -497,7 +497,7 @@ static void parse_frame_header(frame_header_t *const h // 3 Reserved_bit // 2 Content_Checksum_flag // 1-0 Dictionary_ID_flag" - const u8 descriptor = IO_read_bits(in, 8); + const u8 descriptor = (u8)IO_read_bits(in, 8); // decode frame header descriptor into flags const u8 frame_content_size_flag = descriptor >> 6; @@ -521,7 +521,7 @@ static void parse_frame_header(frame_header_t *const h // // Bit numbers 7-3 2-0 // Field name Exponent Mantissa" - u8 window_descriptor = IO_read_bits(in, 8); + u8 window_descriptor = (u8)IO_read_bits(in, 8); u8 exponent = window_descriptor >> 3; u8 mantissa = window_descriptor & 7; @@ -541,7 +541,7 @@ static void parse_frame_header(frame_header_t *const h const int bytes_array[] = {0, 1, 2, 4}; const int bytes = bytes_array[dictionary_id_flag]; - header->dictionary_id = IO_read_bits(in, bytes * 8); + header->dictionary_id = (u32)IO_read_bits(in, bytes * 8); } else { header->dictionary_id = 0; } @@ -633,8 +633,8 @@ static void decompress_data(frame_context_t *const ctx // // The next 2 bits represent the Block_Type, while the remaining 21 bits // represent the Block_Size. Format is little-endian." - last_block = IO_read_bits(in, 1); - const int block_type = IO_read_bits(in, 2); + last_block = (int)IO_read_bits(in, 1); + const int block_type = (int)IO_read_bits(in, 2); const size_t block_len = IO_read_bits(in, 21); switch (block_type) { @@ -748,8 +748,8 @@ static size_t decode_literals(frame_context_t *const c // types" // // size_format takes between 1 and 2 bits - int block_type = IO_read_bits(in, 2); - int size_format = IO_read_bits(in, 2); + int block_type = (int)IO_read_bits(in, 2); + int size_format = (int)IO_read_bits(in, 2); if (block_type <= 1) { // Raw or RLE literals block @@ -833,6 +833,7 @@ static size_t decode_literals_compressed(frame_context // bits (0-1023)." num_streams = 1; // Fall through as it has the same size format + /* fallthrough */ case 1: // "4 streams. Both Compressed_Size and Regenerated_Size use 10 bits // (0-1023)." @@ -855,8 +856,7 @@ static size_t decode_literals_compressed(frame_context // Impossible IMPOSSIBLE(); } - if (regenerated_size > MAX_LITERALS_SIZE || - compressed_size >= regenerated_size) { + if (regenerated_size > MAX_LITERALS_SIZE) { CORRUPTION(); } @@ -1005,7 +1005,7 @@ static const i16 SEQ_MATCH_LENGTH_DEFAULT_DIST[53] = { static const u32 SEQ_LITERAL_LENGTH_BASELINES[36] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 24, 28, 32, 40, - 48, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65538}; + 48, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536}; static const u8 SEQ_LITERAL_LENGTH_EXTRA_BITS[36] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}; @@ -1021,7 +1021,7 @@ static const u8 SEQ_MATCH_LENGTH_EXTRA_BITS[53] = { 2, 2, 3, 3, 4, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}; /// Offset decoding is simpler so we just need a maximum code value -static const u8 SEQ_MAX_CODES[3] = {35, -1, 52}; +static const u8 SEQ_MAX_CODES[3] = {35, (u8)-1, 52}; static void decompress_sequences(frame_context_t *const ctx, istream_t *const in, @@ -1132,7 +1132,7 @@ static void decompress_sequences(frame_context_t *cons // a single 1-bit and then fills the byte with 0-7 0 bits of padding." const int padding = 8 - highest_set_bit(src[len - 1]); // The offset starts at the end because FSE streams are read backwards - i64 bit_offset = len * 8 - padding; + i64 bit_offset = (i64)(len * 8 - (size_t)padding); // "The bitstream starts with initial state values, each using the required // number of bits in their respective accuracy, decoded previously from @@ -1409,7 +1409,7 @@ size_t ZSTD_get_decompressed_size(const void *src, con // get decompressed size from ZSTD frame header { - const u32 magic_number = IO_read_bits(&in, 32); + const u32 magic_number = (u32)IO_read_bits(&in, 32); if (magic_number == 0xFD2FB528U) { // ZSTD frame @@ -1418,7 +1418,7 @@ size_t ZSTD_get_decompressed_size(const void *src, con if (header.frame_content_size == 0 && !header.single_segment_flag) { // Content size not provided, we can't tell - return -1; + return (size_t)-1; } return header.frame_content_size; @@ -1529,7 +1529,7 @@ void free_dictionary(dictionary_t *const dict) { /******* END DICTIONARY PARSING ***********************************************/ /******* IO STREAM OPERATIONS *************************************************/ -#define UNALIGNED() ERROR("Attempting to operate on a non-byte aligned stream") + /// Reads `num` bits from a bitstream, and updates the internal offset static inline u64 IO_read_bits(istream_t *const in, const int num_bits) { if (num_bits > 64 || num_bits <= 0) { @@ -1608,7 +1608,7 @@ static inline const u8 *IO_get_read_ptr(istream_t *con INP_SIZE(); } if (in->bit_offset != 0) { - UNALIGNED(); + ERROR("Attempting to operate on a non-byte aligned stream"); } const u8 *const ptr = in->ptr; in->ptr += len; @@ -1634,7 +1634,7 @@ static inline void IO_advance_input(istream_t *const i INP_SIZE(); } if (in->bit_offset != 0) { - UNALIGNED(); + ERROR("Attempting to operate on a non-byte aligned stream"); } in->ptr += len; Modified: vendor/zstd/dist/doc/educational_decoder/zstd_decompress.h ============================================================================== --- vendor/zstd/dist/doc/educational_decoder/zstd_decompress.h Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/educational_decoder/zstd_decompress.h Wed Nov 6 06:42:00 2019 (r354371) @@ -7,6 +7,8 @@ * in the COPYING file in the root directory of this source tree). */ +#include /* size_t */ + /******* EXPOSED TYPES ********************************************************/ /* * Contains the parsed contents of a dictionary @@ -39,7 +41,7 @@ size_t ZSTD_get_decompressed_size(const void *const sr * Return a valid dictionary_t pointer for use with dictionary initialization * or decompression */ -dictionary_t* create_dictionary(); +dictionary_t* create_dictionary(void); /* * Parse a provided dictionary blob for use in decompression Modified: vendor/zstd/dist/doc/zstd_compression_format.md ============================================================================== --- vendor/zstd/dist/doc/zstd_compression_format.md Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/zstd_compression_format.md Wed Nov 6 06:42:00 2019 (r354371) @@ -16,7 +16,7 @@ Distribution of this document is unlimited. ### Version -0.3.2 (17/07/19) +0.3.4 (16/08/19) Introduction @@ -358,6 +358,7 @@ It may be followed by an optional `Content_Checksum` __`Block_Type`__ The next 2 bits represent the `Block_Type`. +`Block_Type` influences the meaning of `Block_Size`. There are 4 block types : | Value | 0 | 1 | 2 | 3 | @@ -384,9 +385,12 @@ There are 4 block types : __`Block_Size`__ The upper 21 bits of `Block_Header` represent the `Block_Size`. -`Block_Size` is the size of the block excluding the header. -A block can contain any number of bytes (even zero), up to -`Block_Maximum_Decompressed_Size`, which is the smallest of: +When `Block_Type` is `Compressed_Block` or `Raw_Block`, +`Block_Size` is the size of `Block_Content`, hence excluding `Block_Header`. +When `Block_Type` is `RLE_Block`, `Block_Content`’s size is always 1, +and `Block_Size` represents the number of times this byte must be repeated. +A block can contain and decompress into any number of bytes (even zero), +up to `Block_Maximum_Decompressed_Size`, which is the smallest of: - Window_Size - 128 KB @@ -1103,18 +1107,18 @@ It follows the following build rule : The table has a size of `Table_Size = 1 << Accuracy_Log`. Each cell describes the symbol decoded, -and instructions to get the next state. +and instructions to get the next state (`Number_of_Bits` and `Baseline`). Symbols are scanned in their natural order for "less than 1" probabilities. Symbols with this probability are being attributed a single cell, starting from the end of the table and retreating. These symbols define a full state reset, reading `Accuracy_Log` bits. -All remaining symbols are allocated in their natural order. -Starting from symbol `0` and table position `0`, +Then, all remaining symbols, sorted in natural order, are allocated cells. +Starting from symbol `0` (if it exists), and table position `0`, each symbol gets allocated as many cells as its probability. Cell allocation is spreaded, not linear : -each successor position follow this rule : +each successor position follows this rule : ``` position += (tableSize>>1) + (tableSize>>3) + 3; @@ -1126,40 +1130,41 @@ A position is skipped if already occupied by a "less t each position in the table, switching to the next symbol when enough states have been allocated to the current one. -The result is a list of state values. -Each state will decode the current symbol. +The process guarantees that the table is entirely filled. +Each cell corresponds to a state value, which contains the symbol being decoded. -To get the `Number_of_Bits` and `Baseline` required for next state, -it's first necessary to sort all states in their natural order. -The lower states will need 1 more bit than higher ones. +To add the `Number_of_Bits` and `Baseline` required to retrieve next state, +it's first necessary to sort all occurrences of each symbol in state order. +Lower states will need 1 more bit than higher ones. The process is repeated for each symbol. __Example__ : -Presuming a symbol has a probability of 5. -It receives 5 state values. States are sorted in natural order. +Presuming a symbol has a probability of 5, +it receives 5 cells, corresponding to 5 state values. +These state values are then sorted in natural order. -Next power of 2 is 8. -Space of probabilities is divided into 8 equal parts. -Presuming the `Accuracy_Log` is 7, it defines 128 states. +Next power of 2 after 5 is 8. +Space of probabilities must be divided into 8 equal parts. +Presuming the `Accuracy_Log` is 7, it defines a space of 128 states. Divided by 8, each share is 16 large. -In order to reach 8, 8-5=3 lowest states will count "double", -doubling the number of shares (32 in width), -requiring one more bit in the process. +In order to reach 8 shares, 8-5=3 lowest states will count "double", +doubling their shares (32 in width), hence requiring one more bit. Baseline is assigned starting from the higher states using fewer bits, -and proceeding naturally, then resuming at the first state, -each takes its allocated width from Baseline. +increasing at each state, then resuming at the first state, +each state takes its allocated width from Baseline. -| state order | 0 | 1 | 2 | 3 | 4 | -| ---------------- | ----- | ----- | ------ | ---- | ----- | -| width | 32 | 32 | 32 | 16 | 16 | -| `Number_of_Bits` | 5 | 5 | 5 | 4 | 4 | -| range number | 2 | 4 | 6 | 0 | 1 | -| `Baseline` | 32 | 64 | 96 | 0 | 16 | -| range | 32-63 | 64-95 | 96-127 | 0-15 | 16-31 | +| state value | 1 | 39 | 77 | 84 | 122 | +| state order | 0 | 1 | 2 | 3 | 4 | +| ---------------- | ----- | ----- | ------ | ---- | ------ | +| width | 32 | 32 | 32 | 16 | 16 | +| `Number_of_Bits` | 5 | 5 | 5 | 4 | 4 | +| range number | 2 | 4 | 6 | 0 | 1 | +| `Baseline` | 32 | 64 | 96 | 0 | 16 | +| range | 32-63 | 64-95 | 96-127 | 0-15 | 16-31 | -The next state is determined from current state +During decoding, the next state value is determined from current state value, by reading the required `Number_of_Bits`, and adding the specified `Baseline`. See [Appendix A] for the results of this process applied to the default distributions. @@ -1653,6 +1658,8 @@ or at least provide a meaningful error code explaining Version changes --------------- +- 0.3.4 : clarifications for FSE decoding table +- 0.3.3 : clarifications for field Block_Size - 0.3.2 : remove additional block size restriction on compressed blocks - 0.3.1 : minor clarification regarding offset history update rules - 0.3.0 : minor edits to match RFC8478 Modified: vendor/zstd/dist/doc/zstd_manual.html ============================================================================== --- vendor/zstd/dist/doc/zstd_manual.html Wed Nov 6 06:38:34 2019 (r354370) +++ vendor/zstd/dist/doc/zstd_manual.html Wed Nov 6 06:42:00 2019 (r354371) @@ -1,10 +1,10 @@ -zstd 1.4.2 Manual +zstd 1.4.4 Manual -

zstd 1.4.2 Manual

+

zstd 1.4.4 Manual

Contents

    @@ -27,10 +27,16 @@
  1. Advanced compression functions
  2. Advanced decompression functions
  3. Advanced streaming functions
    4. -
  4. Buffer-less and synchronous inner streaming functions
    5. -
  5. Buffer-less streaming compression (synchronous mode)
    6. -
  6. Buffer-less streaming decompression (synchronous mode)
    7. -
  7. Block level API
    8. +
  8. ! ZSTD_initCStream_usingDict() :
    9. +
  9. ! ZSTD_initCStream_advanced() :
    10. +
  10. ! ZSTD_initCStream_usingCDict() :
    11. +
  11. ! ZSTD_initCStream_usingCDict_advanced() :
    12. +
  12. This function is deprecated, and is equivalent to:
    13. +
  13. This function is deprecated, and is equivalent to:
    14. +
  14. Buffer-less and synchronous inner streaming functions
    15. +
  15. Buffer-less streaming compression (synchronous mode)
    16. +
  16. Buffer-less streaming decompression (synchronous mode)
    17. +
  17. Block level API

Introduction

@@ -157,9 +163,13 @@ size_t     ZSTD_freeCCtx(ZSTD_CCtx* cctx);
                          void* dst, size_t dstCapacity,
                    const void* src, size_t srcSize,
                          int compressionLevel);
-
  Same as ZSTD_compress(), using an explicit ZSTD_CCtx
-  The function will compress at requested compression level,
-  ignoring any other parameter 
+
  Same as ZSTD_compress(), using an explicit ZSTD_CCtx.
+  Important : in order to behave similarly to `ZSTD_compress()`,
+  this function compresses at requested compression level,
+  __ignoring any other parameter__ .
+  If any advanced parameter was set using the advanced API,
+  they will all be reset. Only `compressionLevel` remains.
+

Decompression context

  When decompressing many times,
@@ -199,18 +209,26 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
 
     /* compression parameters
      * Note: When compressing with a ZSTD_CDict these parameters are superseded
-     * by the parameters used to construct the ZSTD_CDict. See ZSTD_CCtx_refCDict()
-     * for more info (superseded-by-cdict). */
-    ZSTD_c_compressionLevel=100, /* Update all compression parameters according to pre-defined cLevel table
+     * by the parameters used to construct the ZSTD_CDict.
+     * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
+    ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
+                              * Note that exact compression parameters are dynamically determined,
+                              * depending on both compression level and srcSize (when known).
                               * Default level is ZSTD_CLEVEL_DEFAULT==3.
                               * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
                               * Note 1 : it's possible to pass a negative compression level.
-                              * Note 2 : setting a level sets all default values of other compression parameters */
+                              * Note 2 : setting a level resets all other compression parameters to default */
+    /* Advanced compression parameters :
+     * It's possible to pin down compression parameters to some specific values.
+     * In which case, these values are no longer dynamically selected by the compressor */
     ZSTD_c_windowLog=101,    /* Maximum allowed back-reference distance, expressed as power of 2.
+                              * This will set a memory budget for streaming decompression,
+                              * with larger values requiring more memory
+                              * and typically compressing more.
                               * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
                               * Special: value 0 means "use default windowLog".
                               * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
-                              *       requires explicitly allowing such window size at decompression stage if using streaming. */
+                              *       requires explicitly allowing such size at streaming decompression stage. */
     ZSTD_c_hashLog=102,      /* Size of the initial probe table, as a power of 2.
                               * Resulting memory usage is (1 << (hashLog+2)).
                               * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
@@ -221,13 +239,13 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                               * Resulting memory usage is (1 << (chainLog+2)).
                               * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
                               * Larger tables result in better and slower compression.
-                              * This parameter is useless when using "fast" strategy.
+                              * This parameter is useless for "fast" strategy.
                               * It's still useful when using "dfast" strategy,
                               * in which case it defines a secondary probe table.
                               * Special: value 0 means "use default chainLog". */
     ZSTD_c_searchLog=104,    /* Number of search attempts, as a power of 2.
                               * More attempts result in better and slower compression.
-                              * This parameter is useless when using "fast" and "dFast" strategies.
+                              * This parameter is useless for "fast" and "dFast" strategies.
                               * Special: value 0 means "use default searchLog". */
     ZSTD_c_minMatch=105,     /* Minimum size of searched matches.
                               * Note that Zstandard can still find matches of smaller size,
@@ -282,7 +300,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
     ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
                               * Content size must be known at the beginning of compression.
                               * This is automatically the case when using ZSTD_compress2(),
-                              * For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
+                              * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
     ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
     ZSTD_c_dictIDFlag=202,   /* When applicable, dictionary's ID is written into frame header (default:1) */
 
@@ -301,7 +319,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                               * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
                               * 0 means default, which is dynamically determined based on compression parameters.
                               * Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
-                              * The minimum size is automatically and transparently enforced */
+                              * The minimum size is automatically and transparently enforced. */
     ZSTD_c_overlapLog=402,   /* Control the overlap size, as a fraction of window size.
                               * The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
                               * It helps preserve compression ratio, while each job is compressed in parallel.
@@ -324,6 +342,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
      * ZSTD_c_forceAttachDict
      * ZSTD_c_literalCompressionMode
      * ZSTD_c_targetCBlockSize
+     * ZSTD_c_srcSizeHint
      * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
      * note : never ever use experimentalParam? names directly;
      *        also, the enums values themselves are unstable and can still change.
@@ -334,6 +353,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
      ZSTD_c_experimentalParam4=1001,
      ZSTD_c_experimentalParam5=1002,
      ZSTD_c_experimentalParam6=1003,
+     ZSTD_c_experimentalParam7=1004
 } ZSTD_cParameter;
 

typedef struct {
@@ -672,12 +692,17 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds);
 
 
ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
                              int compressionLevel);
-
  When compressing multiple messages / blocks using the same dictionary, it's recommended to load it only once.
-  ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup cost.
+
  When compressing multiple messages or blocks using the same dictionary,
+  it's recommended to digest the dictionary only once, since it's a costly operation.
+  ZSTD_createCDict() will create a state from digesting a dictionary.
+  The resulting state can be used for future compression operations with very limited startup cost.
   ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
- `dictBuffer` can be released after ZSTD_CDict creation, because its content is copied within CDict.
-  Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate `dictBuffer` content.
-  Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is inefficient when used to compress small data. 
+ @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict.
+  Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content.
+  Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer,
+      in which case the only thing that it transports is the @compressionLevel.
+      This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively,
+      expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. 
 


 
 
size_t      ZSTD_freeCDict(ZSTD_CDict* CDict);
@@ -794,7 +819,7 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds);
   Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters.
            It's a CPU consuming operation, with non-negligible impact on latency.
            If there is a need to use the same prefix multiple times, consider loadDictionary instead.
-  Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dm_rawContent).
+  Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent).
            Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. 
 


 
@@ -838,7 +863,7 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds);
   Note 2 : Prefix buffer is referenced. It **must** outlive decompression.
            Prefix buffer must remain unmodified up to the end of frame,
            reached when ZSTD_decompressStream() returns 0.
-  Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent).
+  Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent).
            Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section)
   Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
            A full dictionary is more costly, as it requires building tables.
@@ -865,6 +890,24 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
 
typedef struct {
+    unsigned int matchPos; /* Match pos in dst */
+    /* If seqDef.offset > 3, then this is seqDef.offset - 3
+     * If seqDef.offset < 3, then this is the corresponding repeat offset
+     * But if seqDef.offset < 3 and litLength == 0, this is the
+     *   repeat offset before the corresponding repeat offset
+     * And if seqDef.offset == 3 and litLength == 0, this is the
+     *   most recent repeat offset - 1
+     */
+    unsigned int offset;
+    unsigned int litLength; /* Literal length */
+    unsigned int matchLength; /* Match length */
+    /* 0 when seq not rep and seqDef.offset otherwise
+     * when litLength == 0 this will be <= 4, otherwise <= 3 like normal
+     */
+    unsigned int rep;
+} ZSTD_Sequence;
+

+
typedef struct {
     unsigned windowLog;       /**< largest match distance : larger == more compression, more memory needed during decompression */
     unsigned chainLog;        /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */
     unsigned hashLog;         /**< dispatch table : larger == faster, more memory */
@@ -893,21 +936,12 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
 

typedef enum {
     ZSTD_dlm_byCopy = 0,  /**< Copy dictionary content internally */
-    ZSTD_dlm_byRef = 1,   /**< Reference dictionary content -- the dictionary buffer must outlive its users. */
+    ZSTD_dlm_byRef = 1    /**< Reference dictionary content -- the dictionary buffer must outlive its users. */
 } ZSTD_dictLoadMethod_e;
 

typedef enum {
-    /* Opened question : should we have a format ZSTD_f_auto ?
-     * Today, it would mean exactly the same as ZSTD_f_zstd1.
-     * But, in the future, should several formats become supported,
-     * on the compression side, it would mean "default format".
-     * On the decompression side, it would mean "automatic format detection",
-     * so that ZSTD_f_zstd1 would mean "accept *only* zstd frames".
-     * Since meaning is a little different, another option could be to define different enums for compression and decompression.
-     * This question could be kept for later, when there are actually multiple formats to support,
-     * but there is also the question of pinning enum values, and pinning value `0` is especially important */
     ZSTD_f_zstd1 = 0,           /* zstd frame format, specified in zstd_compression_format.md (default) */
-    ZSTD_f_zstd1_magicless = 1, /* Variant of zstd frame format, without initial 4-bytes magic number.
+    ZSTD_f_zstd1_magicless = 1  /* Variant of zstd frame format, without initial 4-bytes magic number.
                                  * Useful to save 4 bytes per generated frame.
                                  * Decoder cannot recognise automatically this format, requiring this instruction. */
 } ZSTD_format_e;
@@ -918,7 +952,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
      * to evolve and should be considered only in the context of extremely
      * advanced performance tuning.
      *
-     * Zstd currently supports the use of a CDict in two ways:
+     * Zstd currently supports the use of a CDict in three ways:
      *
      * - The contents of the CDict can be copied into the working context. This
      *   means that the compression can search both the dictionary and input
@@ -934,6 +968,12 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
      *   working context's tables can be reused). For small inputs, this can be
      *   faster than copying the CDict's tables.
      *
+     * - The CDict's tables are not used at all, and instead we use the working
+     *   context alone to reload the dictionary and use params based on the source
+     *   size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict().
+     *   This method is effective when the dictionary sizes are very small relative
+     *   to the input size, and the input size is fairly large to begin with.
+     *
      * Zstd has a simple internal heuristic that selects which strategy to use
      * at the beginning of a compression. However, if experimentation shows that
      * Zstd is making poor choices, it is possible to override that choice with
@@ -942,6 +982,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
     ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */
     ZSTD_dictForceAttach   = 1, /* Never copy the dictionary. */
     ZSTD_dictForceCopy     = 2, /* Always copy the dictionary. */
+    ZSTD_dictForceLoad     = 3  /* Always reload the dictionary */
 } ZSTD_dictAttachPref_e;
 

typedef enum {
@@ -950,7 +991,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
                                *   levels will be compressed. */
   ZSTD_lcm_huffman = 1,       /**< Always attempt Huffman compression. Uncompressed literals will still be
                                *   emitted if Huffman compression is not profitable. */
-  ZSTD_lcm_uncompressed = 2,  /**< Always emit uncompressed literals. */
+  ZSTD_lcm_uncompressed = 2   /**< Always emit uncompressed literals. */
 } ZSTD_literalCompressionMode_e;

Frame size functions


@@ -999,20 +1040,38 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
            or an error code (if srcSize is too small) 
 


 
+
size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
+    size_t outSeqsSize, const void* src, size_t srcSize);
+
 Extract sequences from the sequence store
+ zc can be used to insert custom compression params.
+ This function invokes ZSTD_compress2
+ @return : number of sequences extracted
+ 
+


+
 
Memory management

 
 
size_t ZSTD_estimateCCtxSize(int compressionLevel);
 size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
 size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
 size_t ZSTD_estimateDCtxSize(void);
-
  These functions make it possible to estimate memory usage
-  of a future {D,C}Ctx, before its creation.
-  ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one.
-  It will also consider src size to be arbitrarily "large", which is worst case.
-  If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
-  ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
-  ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1.
-  Note : CCtx size estimation is only correct for single-threaded compression. 
+
  These functions make it possible to estimate memory usage of a future
+  {D,C}Ctx, before its creation.
+
+  ZSTD_estimateCCtxSize() will provide a budget large enough for any
+  compression level up to selected one. Unlike ZSTD_estimateCStreamSize*(),
+  this estimate does not include space for a window buffer, so this estimate
+  is guaranteed to be enough for single-shot compressions, but not streaming
+  compressions. It will however assume the input may be arbitrarily large,
+  which is the worst case. If srcSize is known to always be small,
+  ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
+  ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with
+  ZSTD_getCParams() to create cParams from compressionLevel.
+  ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with
+  ZSTD_CCtxParams_setParameter().
+
+  Note: only single-threaded compression is supported. This function will
+  return an error code if ZSTD_c_nbWorkers is >= 1. 
 


 
 
size_t ZSTD_estimateCStreamSize(int compressionLevel);
@@ -1085,7 +1144,8 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL,
 
  Create a digested dictionary for compression
   Dictionary content is just referenced, not duplicated.
   As a consequence, `dictBuffer` **must** outlive CDict,
-  and its content must remain unmodified throughout the lifetime of CDict. 
+  and its content must remain unmodified throughout the lifetime of CDict.
+  note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef 
 


 
 
ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
@@ -1116,7 +1176,9 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL,
                         const void* src, size_t srcSize,
                         const void* dict,size_t dictSize,
                               ZSTD_parameters params);
-
  Same as ZSTD_compress_usingDict(), with fine-tune control over compression parameters (by structure) 
+
  Note : this function is now DEPRECATED.
+         It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
+  This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x 
 


 
 
size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
@@ -1124,7 +1186,9 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL,
                             const void* src, size_t srcSize,

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 06:43:14 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 30C961AB262;
 Wed,  6 Nov 2019 06:43:14 +0000 (UTC) (envelope-from cem@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477H6V1JTkz4txS;
 Wed,  6 Nov 2019 06:43:14 +0000 (UTC) (envelope-from cem@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 10F6221269;
 Wed,  6 Nov 2019 06:43:14 +0000 (UTC) (envelope-from cem@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA66hDmC069992;
 Wed, 6 Nov 2019 06:43:13 GMT (envelope-from cem@FreeBSD.org)
Received: (from cem@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA66hDZN069989;
 Wed, 6 Nov 2019 06:43:13 GMT (envelope-from cem@FreeBSD.org)
Message-Id: <201911060643.xA66hDZN069989@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org
 using -f
From: Conrad Meyer 
Date: Wed, 6 Nov 2019 06:43:13 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354372 - in vendor/zstd/1.4.4: .
 contrib/adaptive-compression contrib/gen_html contrib/pzstd
 contrib/seekable_format/examples doc doc/educational_decoder examples lib
 lib/common lib/co...
X-SVN-Group: vendor
X-SVN-Commit-Author: cem
X-SVN-Commit-Paths: in vendor/zstd/1.4.4: . contrib/adaptive-compression
 contrib/gen_html contrib/pzstd contrib/seekable_format/examples doc
 doc/educational_decoder examples lib lib/common lib/compress lib/decompress
 lib...
X-SVN-Commit-Revision: 354372
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 06:43:14 -0000

Author: cem
Date: Wed Nov  6 06:43:13 2019
New Revision: 354372
URL: https://svnweb.freebsd.org/changeset/base/354372

Log:
  Tag Zstd 1.4.4

Added:
  vendor/zstd/1.4.4/
     - copied from r354369, vendor/zstd/dist/
  vendor/zstd/1.4.4/lib/compress/zstd_cwksp.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_cwksp.h
  vendor/zstd/1.4.4/tests/fuzz/dictionary_loader.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/dictionary_loader.c
  vendor/zstd/1.4.4/tests/fuzz/fuzz_data_producer.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/fuzz_data_producer.c
  vendor/zstd/1.4.4/tests/fuzz/fuzz_data_producer.h
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/fuzz_data_producer.h
  vendor/zstd/1.4.4/tests/golden-compression/
     - copied from r354371, vendor/zstd/dist/tests/golden-compression/
  vendor/zstd/1.4.4/tests/golden-decompression/
     - copied from r354371, vendor/zstd/dist/tests/golden-decompression/
Replaced:
  vendor/zstd/1.4.4/CHANGELOG
     - copied unchanged from r354371, vendor/zstd/dist/CHANGELOG
  vendor/zstd/1.4.4/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/Makefile
  vendor/zstd/1.4.4/README.md
     - copied unchanged from r354371, vendor/zstd/dist/README.md
  vendor/zstd/1.4.4/appveyor.yml
     - copied unchanged from r354371, vendor/zstd/dist/appveyor.yml
  vendor/zstd/1.4.4/doc/educational_decoder/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/doc/educational_decoder/Makefile
  vendor/zstd/1.4.4/doc/educational_decoder/harness.c
     - copied unchanged from r354371, vendor/zstd/dist/doc/educational_decoder/harness.c
  vendor/zstd/1.4.4/doc/educational_decoder/zstd_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c
  vendor/zstd/1.4.4/doc/educational_decoder/zstd_decompress.h
     - copied unchanged from r354371, vendor/zstd/dist/doc/educational_decoder/zstd_decompress.h
  vendor/zstd/1.4.4/doc/zstd_compression_format.md
     - copied unchanged from r354371, vendor/zstd/dist/doc/zstd_compression_format.md
  vendor/zstd/1.4.4/doc/zstd_manual.html
     - copied unchanged from r354371, vendor/zstd/dist/doc/zstd_manual.html
  vendor/zstd/1.4.4/examples/streaming_compression.c
     - copied unchanged from r354371, vendor/zstd/dist/examples/streaming_compression.c
  vendor/zstd/1.4.4/examples/streaming_decompression.c
     - copied unchanged from r354371, vendor/zstd/dist/examples/streaming_decompression.c
  vendor/zstd/1.4.4/lib/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/lib/Makefile
  vendor/zstd/1.4.4/lib/README.md
     - copied unchanged from r354371, vendor/zstd/dist/lib/README.md
  vendor/zstd/1.4.4/lib/common/bitstream.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/bitstream.h
  vendor/zstd/1.4.4/lib/common/compiler.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/compiler.h
  vendor/zstd/1.4.4/lib/common/fse.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/fse.h
  vendor/zstd/1.4.4/lib/common/fse_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/fse_decompress.c
  vendor/zstd/1.4.4/lib/common/mem.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/mem.h
  vendor/zstd/1.4.4/lib/common/pool.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/pool.c
  vendor/zstd/1.4.4/lib/common/threading.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/threading.c
  vendor/zstd/1.4.4/lib/common/threading.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/threading.h
  vendor/zstd/1.4.4/lib/common/xxhash.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/xxhash.c
  vendor/zstd/1.4.4/lib/common/zstd_internal.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/common/zstd_internal.h
  vendor/zstd/1.4.4/lib/compress/zstd_compress.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress.c
  vendor/zstd/1.4.4/lib/compress/zstd_compress_internal.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress_internal.h
  vendor/zstd/1.4.4/lib/compress/zstd_compress_literals.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress_literals.c
  vendor/zstd/1.4.4/lib/compress/zstd_compress_literals.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress_literals.h
  vendor/zstd/1.4.4/lib/compress/zstd_compress_sequences.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress_sequences.c
  vendor/zstd/1.4.4/lib/compress/zstd_compress_sequences.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_compress_sequences.h
  vendor/zstd/1.4.4/lib/compress/zstd_double_fast.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_double_fast.c
  vendor/zstd/1.4.4/lib/compress/zstd_fast.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_fast.c
  vendor/zstd/1.4.4/lib/compress/zstd_lazy.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_lazy.c
  vendor/zstd/1.4.4/lib/compress/zstd_ldm.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_ldm.c
  vendor/zstd/1.4.4/lib/compress/zstd_opt.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstd_opt.c
  vendor/zstd/1.4.4/lib/compress/zstdmt_compress.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/compress/zstdmt_compress.c
  vendor/zstd/1.4.4/lib/decompress/huf_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/decompress/huf_decompress.c
  vendor/zstd/1.4.4/lib/decompress/zstd_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/decompress/zstd_decompress.c
  vendor/zstd/1.4.4/lib/decompress/zstd_decompress_block.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/decompress/zstd_decompress_block.c
  vendor/zstd/1.4.4/lib/deprecated/zbuff.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/deprecated/zbuff.h
  vendor/zstd/1.4.4/lib/dictBuilder/cover.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/dictBuilder/cover.c
  vendor/zstd/1.4.4/lib/dictBuilder/zdict.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/dictBuilder/zdict.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v01.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v01.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v02.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v02.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v03.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v03.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v04.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v04.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v05.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v05.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v06.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v06.c
  vendor/zstd/1.4.4/lib/legacy/zstd_v07.c
     - copied unchanged from r354371, vendor/zstd/dist/lib/legacy/zstd_v07.c
  vendor/zstd/1.4.4/lib/libzstd.pc.in
     - copied unchanged from r354371, vendor/zstd/dist/lib/libzstd.pc.in
  vendor/zstd/1.4.4/lib/zstd.h
     - copied unchanged from r354371, vendor/zstd/dist/lib/zstd.h
  vendor/zstd/1.4.4/programs/README.md
     - copied unchanged from r354371, vendor/zstd/dist/programs/README.md
  vendor/zstd/1.4.4/programs/benchzstd.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/benchzstd.c
  vendor/zstd/1.4.4/programs/benchzstd.h
     - copied unchanged from r354371, vendor/zstd/dist/programs/benchzstd.h
  vendor/zstd/1.4.4/programs/datagen.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/datagen.c
  vendor/zstd/1.4.4/programs/dibio.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/dibio.c
  vendor/zstd/1.4.4/programs/fileio.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/fileio.c
  vendor/zstd/1.4.4/programs/fileio.h
     - copied unchanged from r354371, vendor/zstd/dist/programs/fileio.h
  vendor/zstd/1.4.4/programs/platform.h
     - copied unchanged from r354371, vendor/zstd/dist/programs/platform.h
  vendor/zstd/1.4.4/programs/timefn.h
     - copied unchanged from r354371, vendor/zstd/dist/programs/timefn.h
  vendor/zstd/1.4.4/programs/util.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/util.c
  vendor/zstd/1.4.4/programs/util.h
     - copied unchanged from r354371, vendor/zstd/dist/programs/util.h
  vendor/zstd/1.4.4/programs/zstd.1
     - copied unchanged from r354371, vendor/zstd/dist/programs/zstd.1
  vendor/zstd/1.4.4/programs/zstd.1.md
     - copied unchanged from r354371, vendor/zstd/dist/programs/zstd.1.md
  vendor/zstd/1.4.4/programs/zstdcli.c
     - copied unchanged from r354371, vendor/zstd/dist/programs/zstdcli.c
  vendor/zstd/1.4.4/programs/zstdgrep.1
     - copied unchanged from r354371, vendor/zstd/dist/programs/zstdgrep.1
  vendor/zstd/1.4.4/programs/zstdless.1
     - copied unchanged from r354371, vendor/zstd/dist/programs/zstdless.1
  vendor/zstd/1.4.4/tests/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/tests/Makefile
  vendor/zstd/1.4.4/tests/decodecorpus.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/decodecorpus.c
  vendor/zstd/1.4.4/tests/fullbench.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fullbench.c
  vendor/zstd/1.4.4/tests/fuzz/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/Makefile
  vendor/zstd/1.4.4/tests/fuzz/README.md
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/README.md
  vendor/zstd/1.4.4/tests/fuzz/block_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/block_decompress.c
  vendor/zstd/1.4.4/tests/fuzz/block_round_trip.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/block_round_trip.c
  vendor/zstd/1.4.4/tests/fuzz/dictionary_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/dictionary_decompress.c
  vendor/zstd/1.4.4/tests/fuzz/dictionary_round_trip.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/dictionary_round_trip.c
  vendor/zstd/1.4.4/tests/fuzz/fuzz.h
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/fuzz.h
  vendor/zstd/1.4.4/tests/fuzz/fuzz.py
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/fuzz.py
  vendor/zstd/1.4.4/tests/fuzz/fuzz_helpers.h
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/fuzz_helpers.h
  vendor/zstd/1.4.4/tests/fuzz/regression_driver.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/regression_driver.c
  vendor/zstd/1.4.4/tests/fuzz/simple_compress.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/simple_compress.c
  vendor/zstd/1.4.4/tests/fuzz/simple_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/simple_decompress.c
  vendor/zstd/1.4.4/tests/fuzz/simple_round_trip.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/simple_round_trip.c
  vendor/zstd/1.4.4/tests/fuzz/stream_decompress.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/stream_decompress.c
  vendor/zstd/1.4.4/tests/fuzz/stream_round_trip.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/stream_round_trip.c
  vendor/zstd/1.4.4/tests/fuzz/zstd_frame_info.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/zstd_frame_info.c
  vendor/zstd/1.4.4/tests/fuzz/zstd_helpers.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/zstd_helpers.c
  vendor/zstd/1.4.4/tests/fuzz/zstd_helpers.h
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzz/zstd_helpers.h
  vendor/zstd/1.4.4/tests/fuzzer.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/fuzzer.c
  vendor/zstd/1.4.4/tests/playTests.sh
     - copied unchanged from r354371, vendor/zstd/dist/tests/playTests.sh
  vendor/zstd/1.4.4/tests/poolTests.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/poolTests.c
  vendor/zstd/1.4.4/tests/regression/method.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/regression/method.c
  vendor/zstd/1.4.4/tests/regression/results.csv
     - copied unchanged from r354371, vendor/zstd/dist/tests/regression/results.csv
  vendor/zstd/1.4.4/tests/zbufftest.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/zbufftest.c
  vendor/zstd/1.4.4/tests/zstreamtest.c
     - copied unchanged from r354371, vendor/zstd/dist/tests/zstreamtest.c
  vendor/zstd/1.4.4/zlibWrapper/Makefile
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/Makefile
  vendor/zstd/1.4.4/zlibWrapper/examples/fitblk.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/examples/fitblk.c
  vendor/zstd/1.4.4/zlibWrapper/examples/zwrapbench.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c
  vendor/zstd/1.4.4/zlibWrapper/gzclose.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/gzclose.c
  vendor/zstd/1.4.4/zlibWrapper/gzlib.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/gzlib.c
  vendor/zstd/1.4.4/zlibWrapper/gzread.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/gzread.c
  vendor/zstd/1.4.4/zlibWrapper/gzwrite.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/gzwrite.c
  vendor/zstd/1.4.4/zlibWrapper/zstd_zlibwrapper.c
     - copied unchanged from r354371, vendor/zstd/dist/zlibWrapper/zstd_zlibwrapper.c
Deleted:
  vendor/zstd/1.4.4/contrib/adaptive-compression/Makefile
  vendor/zstd/1.4.4/contrib/adaptive-compression/README.md
  vendor/zstd/1.4.4/contrib/adaptive-compression/adapt.c
  vendor/zstd/1.4.4/contrib/adaptive-compression/datagencli.c
  vendor/zstd/1.4.4/contrib/adaptive-compression/test-correctness.sh
  vendor/zstd/1.4.4/contrib/adaptive-compression/test-performance.sh
  vendor/zstd/1.4.4/contrib/gen_html/.gitignore
  vendor/zstd/1.4.4/contrib/pzstd/.gitignore
  vendor/zstd/1.4.4/contrib/seekable_format/examples/.gitignore
  vendor/zstd/1.4.4/examples/.gitignore
  vendor/zstd/1.4.4/lib/.gitignore
  vendor/zstd/1.4.4/programs/.gitignore
  vendor/zstd/1.4.4/tests/.gitignore
  vendor/zstd/1.4.4/tests/files/huffman-compressed-larger
  vendor/zstd/1.4.4/zlibWrapper/.gitignore

Copied: vendor/zstd/1.4.4/CHANGELOG (from r354371, vendor/zstd/dist/CHANGELOG)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/zstd/1.4.4/CHANGELOG	Wed Nov  6 06:43:13 2019	(r354372, copy of r354371, vendor/zstd/dist/CHANGELOG)
@@ -0,0 +1,528 @@
+v1.4.4
+perf: Improved decompression speed, by > 10%, by @terrelln
+perf: Better compression speed when re-using a context, by @felixhandte
+perf: Fix compression ratio when compressing large files with small dictionary, by @senhuang42
+perf: zstd reference encoder can generate RLE blocks, by @bimbashrestha
+perf: minor generic speed optimization, by @davidbolvansky
+api: new ability to extract sequences from the parser for analysis, by @bimbashrestha
+api: fixed decoding of magic-less frames, by @terrelln
+api: fixed ZSTD_initCStream_advanced() performance with fast modes, reported by @QrczakMK
+cli: Named pipes support, by @bimbashrestha
+cli: short tar's extension support, by @stokito
+cli: command --output-dir-flat= , generates target files into requested directory, by @senhuang42
+cli: commands --stream-size=# and --size-hint=#, by @nmagerko
+cli: command --exclude-compressed, by @shashank0791
+cli: faster `-t` test mode
+cli: improved some error messages, by @vangyzen
+cli: rare deadlock condition within dictionary builder, by @terrelln
+build: single-file decoder with emscripten compilation script, by @cwoffenden
+build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive
+build: fixed deprecation warning for certain gcc version, reported by @jasonma163
+build: fix compilation on old gcc versions, by @cemeyer
+build: improved installation directories for cmake script, by Dmitri Shubin
+pack: modified pkgconfig, for better integration into openwrt, requested by @neheb
+misc: Improved documentation : ZSTD_CLEVEL, DYNAMIC_BMI2, ZSTD_CDict, function deprecation, zstd format
+misc: fixed educational decoder : accept larger literals section, and removed UNALIGNED() macro
+
+v1.4.3
+bug: Fix Dictionary Compression Ratio Regression by @cyan4973 (#1709)
+bug: Fix Buffer Overflow in legacy v0.3 decompression by @felixhandte (#1722)
+build: Add support for IAR C/C++ Compiler for Arm by @joseph0918 (#1705)
+
+v1.4.2
+bug: Fix bug in zstd-0.5 decoder by @terrelln (#1696)
+bug: Fix seekable decompression in-memory API by @iburinoc (#1695)
+misc: Validate blocks are smaller than size limit by @vivekmg (#1685)
+misc: Restructure source files by @ephiepark (#1679)
+
+v1.4.1
+bug: Fix data corruption in niche use cases by @terrelln (#1659)
+bug: Fuzz legacy modes, fix uncovered bugs by @terrelln (#1593, #1594, #1595)
+bug: Fix out of bounds read by @terrelln (#1590)
+perf: Improve decode speed by ~7% @mgrice (#1668)
+perf: Slightly improved compression ratio of level 3 and 4 (ZSTD_dfast) by @cyan4973 (#1681)
+perf: Slightly faster compression speed when re-using a context by @cyan4973 (#1658)
+perf: Improve compression ratio for small windowLog by @cyan4973 (#1624)
+perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
+api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
+cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
+cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
+cli: Restrict read permissions on destination files by @chungy (#1644)
+cli: zstdgrep: handle -f flag by @felixhandte (#1618)
+cli: zstdcat: follow symlinks by @vejnar (#1604)
+doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
+doc: Fix typo by @yk-tanigawa (#1633)
+doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
+build: CMake: support building with LZ4 @leeyoung624 (#1626)
+build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
+build: CMake: respect existing uninstall target by @j301scott (#1619)
+build: Make: skip multithread tests when built without support by @michaelforney (#1620)
+build: Make: Fix examples/ test target by @sjnam (#1603)
+build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
+build: Meson: fix build by @lzutao (#1602)
+build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
+build: Visual Studio: fix linking by @absotively (#1639)
+build: Fix MinGW-W64 build by @myzhang1029 (#1600)
+misc: Expand decodecorpus coverage by @ephiepark (#1664)
+
+v1.4.0
+perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln
+api: Move the advanced API, including all functions in the staging section, to the stable section
+api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress
+api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter
+api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter
+api: Don't export ZSTDMT functions from the shared library by default
+api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT
+api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao
+api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
+api: Move ZSTD_getDictID_*() functions to the stable section
+api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
+api: Allow compression parameters to be set when a dictionary is used
+api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
+api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
+api: Setting ZSTD_d_maxWindowLog to 0 means use the default
+cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
+cli: Add --[no-]compress-literals flag to enable or disable literal compression
+doc: Update the examples to use the advanced API
+doc: Explain how to transition from old streaming functions to the advanced API in the header
+build: Improve the Windows release packages
+build: Improve CMake build by @hjmjohnson
+build: Build fixes for FreeBSD by @lwhsu
+build: Remove redundant warnings by @thatsafunnyname
+build: Fix tests on OpenBSD by @bket
+build: Extend fuzzer build system to work with the new clang engine
+build: CMake now creates the libzstd.so.1 symlink
+build: Improve Menson build by @lzutao
+misc: Fix symbolic link detection on FreeBSD
+misc: Use physical core count for -T0 on FreeBSD by @cemeyer
+misc: Fix zstd --list on truncated files by @kostmo
+misc: Improve logging in debug mode by @felixhandte
+misc: Add CirrusCI tests by @lwhsu
+misc: Optimize dictionary memory usage in corner cases
+misc: Improve the dictionary builder on small or homogeneous data
+misc: Fix spelling across the repo by @jsoref
+
+v1.3.8
+perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
+perf: slightly better compression ratio at high compression modes
+api : finalized advanced API, last stage before "stable" status
+api : new --rsyncable mode, by @terrelln
+api : support decompression of empty frames into NULL (used to be an error) (#1385)
+build: new set of macros to build a minimal size decoder, by @felixhandte
+build: fix compilation on MIPS32, reported by @clbr (#1441)
+build: fix compilation with multiple -arch flags, by @ryandesign
+build: highly upgraded meson build, by @lzutao
+build: improved buck support, by @obelisk
+build: fix cmake script : can create debug build, by @pitrou
+build: Makefile : grep works on both colored consoles and systems without color support
+build: fixed zstd-pgo, by @bmwiedemann
+cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423)
+cli : --no-progress flag, preserving final summary (#1371), by @terrelln
+cli : ensure destination file is not source file (#1422)
+cli : clearer error messages, especially when input file not present
+doc : clarified zstd_compression_format.md, by @ulikunitz
+misc: fixed zstdgrep, returns 1 on failure, by @lzutao
+misc: NEWS renamed as CHANGELOG, in accordance with fboss
+
+v1.3.7
+perf: slightly better decompression speed on clang (depending on hardware target)
+fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10
+build: no longer build backtrace by default in release mode; restrict further automatic mode
+build: control backtrace support through build macro BACKTRACE
+misc: added man pages for zstdless and zstdgrep, by @samrussell
+
+v1.3.6
+perf: much faster dictionary builder, by @jenniferliu
+perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte
+perf: faster dictionary decompression when using a very large number of dictionaries simultaneously
+cli : fix : does no longer overwrite destination when source does not exist (#1082)
+cli : new command --adapt, for automatic compression level adaptation
+api : fix : block api can be streamed with > 4 GB, reported by @catid
+api : reduced ZSTD_DDict size by 2 KB
+api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel().
+build: support Haiku target, by @korli
+build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT.
+doc : zstd_compression_format.md updated to match wording in IETF RFC 8478
+misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97
+
+v1.3.5
+perf: much faster dictionary compression, by @felixhandte
+perf: small quality improvement for dictionary generation, by @terrelln
+perf: slightly improved high compression levels (notably level 19)
+mem : automatic memory release for long duration contexts
+cli : fix : overlapLog can be manually set
+cli : fix : decoding invalid lz4 frames
+api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln
+api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln
+build: select custom libzstd scope through control macros, by @GeorgeLu97
+build: OpenBSD patch, by @bket
+build: make and make all are compatible with -j
+doc : clarify zstd_compression_format.md, updated for IETF RFC process
+misc: pzstd compatible with reproducible compilation, by @lamby
+
+v1.3.4
+perf: faster speed (especially decoding speed) on recent cpus (haswell+)
+perf: much better performance associating --long with multi-threading, by @terrelln
+perf: better compression at levels 13-15
+cli : asynchronous compression by default, for faster experience (use --single-thread for former behavior)
+cli : smoother status report in multi-threading mode
+cli : added command --fast=#, for faster compression modes
+cli : fix crash when not overwriting existing files, by Pádraig Brady (@pixelb)
+api : `nbThreads` becomes `nbWorkers` : 1 triggers asynchronous mode
+api : compression levels can be negative, for even more speed
+api : ZSTD_getFrameProgression() : get precise progress status of ZSTDMT anytime
+api : ZSTDMT can accept new compression parameters during compression
+api : implemented all advanced dictionary decompression prototypes
+build: improved meson recipe, by Shawn Landden (@shawnl)
+build: VS2017 scripts, by @HaydnTrigg
+misc: all /contrib projects fixed
+misc: added /contrib/docker script by @gyscos
+
+v1.3.3
+perf: faster zstd_opt strategy (levels 16-19)
+fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen
+cli : fix : content size written in header by default
+cli : fix : improved LZ4 format support, by @felixhandte
+cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file
+api : fix : support large skippable frames, by @terrelln
+api : fix : streaming interface was adding a useless 3-bytes null block to small frames
+api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown"
+build: fix : compilation under rhel6 and centos6, reported by @pixelb
+build: added `check` target
+
+v1.3.2
+new : long range mode, using --long command, by Stella Lau (@stellamplau)
+new : ability to generate and decode magicless frames (#591)
+changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
+fix : multi-threading compression works with custom allocators
+fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
+fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
+fix : 32-bits build can now decode large offsets (levels 21+)
+cli : added LZ4 frame support by default, by Felix Handte (@felixhandte)
+cli : improved --list output
+cli : new : can split input file for dictionary training, using command -B#
+cli : new : clean operation artefact on Ctrl-C interruption
+cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)
+cli : fix : write file size in header in multiple-files mode
+api : added macro ZSTD_COMPRESSBOUND() for static allocation
+api : experimental : new advanced decompression API
+api : fix : sizeof_CCtx() used to over-estimate
+build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819)
+build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
+example : added streaming_memory_usage
+license : changed /examples license to BSD + GPLv2
+license : fix a few header files to reflect new license (#825)
+
+v1.3.1
+New license : BSD + GPLv2
+perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
+perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760)
+cli : improved and fixed --list command, by @ib (#772)
+cli : command -vV to list supported formats, by @ib (#771)
+build : fixed binary variants, reported by @svenha (#788)
+build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718)
+API exp : breaking change : ZSTD_getframeHeader() provides more information
+API exp : breaking change : pinned down values of error codes
+doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz)
+new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74)
+new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau)
+updated : contrib/linux-kernel, by Nick Terrell (@terrelln)
+
+v1.3.0
+cli : new : `--list` command, by Paul Cruz
+cli : changed : xz/lzma support enabled by default
+cli : changed : `-t *` continue processing list after a decompression error
+API : added : ZSTD_versionString()
+API : promoted to stable status : ZSTD_getFrameContentSize(), by Sean Purcell
+API exp : new advanced API : ZSTD_compress_generic(), ZSTD_CCtx_setParameter()
+API exp : new : API for static or external allocation : ZSTD_initStatic?Ctx()
+API exp : added : ZSTD_decompressBegin_usingDDict(), requested by Guy Riddle (#700)
+API exp : clarified memory estimation / measurement functions.
+API exp : changed : strongest strategy renamed ZSTD_btultra, fastest strategy ZSTD_fast set to 1
+tools : decodecorpus can generate random dictionary-compressed samples, by Paul Cruz
+new : contrib/seekable_format, demo and API, by Sean Purcell
+changed : contrib/linux-kernel, updated version and license, by Nick Terrell
+
+v1.2.0
+cli : changed : Multithreading enabled by default (use target zstd-nomt or HAVE_THREAD=0 to disable)
+cli : new : command -T0 means "detect and use nb of cores", by Sean Purcell
+cli : new : zstdmt symlink hardwired to `zstd -T0`
+cli : new : command --threads=# (#671)
+cli : changed : cover dictionary builder by default, for improved quality, by Nick Terrell
+cli : new : commands --train-cover and --train-legacy, to select dictionary algorithm and parameters
+cli : experimental targets `zstd4` and `xzstd4`, with support for lz4 format, by Sean Purcell
+cli : fix : does not output compressed data on console
+cli : fix : ignore symbolic links unless --force specified,
+API : breaking change : ZSTD_createCDict_advanced(), only use compressionParameters as argument
+API : added : prototypes ZSTD_*_usingCDict_advanced(), for direct control over frameParameters.
+API : improved: ZSTDMT_compressCCtx() reduced memory usage
+API : fix : ZSTDMT_compressCCtx() now provides srcSize in header (#634)
+API : fix : src size stored in frame header is controlled at end of frame
+API : fix : enforced consistent rules for pledgedSrcSize==0 (#641)
+API : fix : error code "GENERIC" replaced by "dstSizeTooSmall" when appropriate
+build: improved cmake script, by @Majlen
+build: enabled Multi-threading support for *BSD, by Baptiste Daroussin
+tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target.
+new : contrib/linux-kernel version, by Nick Terrell
+
+v1.1.4
+cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski
+cli : new : advanced benchmark command --priority=rt
+cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77
+cli : fix : --rm remains silent when input is stdin
+cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski
+speed : improved decompression speed in streaming mode for single shot scenarios (+5%)
+memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB
+arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell
+API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize()
+API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value)
+build : new: meson build system in contrib/meson, by Dima Krasner
+build : improved cmake script, by @Majlen
+build : added -Wformat-security flag, as recommended by Padraig Brady
+doc : new : educational decoder, by Sean Purcell
+
+v1.1.3
+cli : zstd can decompress .gz files (can be disabled with `make zstd-nogz` or `make HAVE_ZLIB=0`)
+cli : new : experimental target `make zstdmt`, with multi-threading support
+cli : new : improved dictionary builder "cover" (experimental), by Nick Terrell, based on prior work by Giuseppe Ottaviano.
+cli : new : advanced commands for detailed parameters, by Przemyslaw Skibinski
+cli : fix zstdless on Mac OS-X, by Andrew Janke
+cli : fix #232 "compress non-files"
+dictBuilder : improved dictionary generation quality, thanks to Nick Terrell
+API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental)
+API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul
+API : new : ZDICT_finalizeDictionary()
+API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511)
+API : fix : all symbols properly exposed in libzstd, by Nick Terrell
+build : support for Solaris target, by Przemyslaw Skibinski
+doc : clarified specification, by Sean Purcell
+
+v1.1.2
+API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init
+API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize()
+API : zbuff : changed : prototypes now generate deprecation warnings
+lib : improved : faster decompression speed at ultra compression settings and 32-bits mode
+lib : changed : only public ZSTD_ symbols are now exposed
+lib : changed : reduced usage  of stack memory
+lib : fixed : several corner case bugs, by Nick Terrell
+cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski
+cli : new : preserve file attributes
+cli : new : added zstdless and zstdgrep tools
+cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd)
+cli : fixed : zstdcat
+zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski
+install : better compatibility with FreeBSD, by Dimitry Andric
+source tree : changed : zbuff source files moved to lib/deprecated
+
+v1.1.1
+New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption
+New : doc/zstd_manual.html, by Przemyslaw Skibinski
+Improved : slightly better compression ratio at --ultra levels (>= 20)
+Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report
+Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section)
+Added : example/multiple_streaming_compression.c
+Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h)
+Updated man page
+Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets
+
+v1.1.0
+New : contrib/pzstd, parallel version of zstd, by Nick Terrell
+added : NetBSD install target (#338)
+Improved : speed for batches of small files
+Improved : speed of zlib wrapper, by Przemyslaw Skibinski
+Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier
+Fixed : CLI -d output to stdout by default when input is stdin (#322)
+Fixed : CLI correctly detects console on Mac OS-X
+Fixed : CLI supports recursive mode `-r` on Mac OS-X
+Fixed : Legacy decoders use unified error codes, reported by benrg (#341), fixed by Przemyslaw Skibinski
+Fixed : compatibility with OpenBSD, reported by Juan Francisco Cantero Hurtado (#319)
+Fixed : compatibility with Hurd, by Przemyslaw Skibinski (#365)
+Fixed : zstd-pgo, reported by octoploid (#329)
+
+v1.0.0
+Change Licensing, all project is now BSD, Copyright Facebook
+Small decompression speed improvement
+API : Streaming API supports legacy format
+API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter()
+CLI supports legacy formats v0.4+
+Fixed : compression fails on certain huge files, reported by Jesse McGrew
+Enhanced documentation, by Przemyslaw Skibinski
+
+v0.8.1
+New streaming API
+Changed : --ultra now enables levels beyond 19
+Changed : -i# now selects benchmark time in second
+Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell
+Fixed : speed regression on specific patterns (#272)
+Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291)
+Fixed : ICC compilation, by Przemyslaw Skibinski
+
+v0.8.0
+Improved : better speed on clang and gcc -O2, thanks to Eric Biggers
+New : Build on FreeBSD and DragonFly, thanks to JrMarino
+Changed : modified API : ZSTD_compressEnd()
+Fixed : legacy mode with ZSTD_HEAPMODE=0, by Christopher Bergqvist
+Fixed : premature end of frame when zero-sized raw block, reported by Eric Biggers
+Fixed : large dictionaries (> 384 KB), reported by Ilona Papava
+Fixed : checksum correctly checked in single-pass mode
+Fixed : combined --test amd --rm, reported by Andreas M. Nilsson
+Modified : minor compression level adaptations
+Updated : compression format specification to v0.2.0
+changed : zstd.h moved to /lib directory
+
+v0.7.5
+Transition version, supporting decoding of v0.8.x
+
+v0.7.4
+Added : homebrew for Mac, by Daniel Cade
+Added : more examples
+Fixed : segfault when using small dictionaries, reported by Felix Handte
+Modified : default compression level for CLI is now 3
+Updated : specification, to v0.1.1
+
+v0.7.3
+New : compression format specification
+New : `--` separator, stating that all following arguments are file names. Suggested by Chip Turner.
+New : `ZSTD_getDecompressedSize()`
+New : OpenBSD target, by Juan Francisco Cantero Hurtado
+New : `examples` directory
+fixed : dictBuilder using HC levels, reported by Bartosz Taudul
+fixed : legacy support from ZSTD_decompress_usingDDict(), reported by Felix Handte
+fixed : multi-blocks decoding with intermediate uncompressed blocks, reported by Greg Slazinski
+modified : removed "mem.h" and "error_public.h" dependencies from "zstd.h" (experimental section)
+modified : legacy functions no longer need magic number
+
+v0.7.2
+fixed : ZSTD_decompressBlock() using multiple consecutive blocks. Reported by Greg Slazinski.
+fixed : potential segfault on very large files (many gigabytes). Reported by Chip Turner.
+fixed : CLI displays system error message when destination file cannot be created (#231). Reported by Chip Turner.
+
+v0.7.1
+fixed : ZBUFF_compressEnd() called multiple times with too small `dst` buffer, reported by Christophe Chevalier
+fixed : dictBuilder fails if first sample is too small, reported by Руслан Ковалёв
+fixed : corruption issue, reported by cj
+modified : checksum enabled by default in command line mode
+
+v0.7.0
+New : Support for directory compression, using `-r`, thanks to Przemyslaw Skibinski
+New : Command `--rm`, to remove source file after successful de/compression
+New : Visual build scripts, by Christophe Chevalier
+New : Support for Sparse File-systems (do not use space for zero-filled sectors)
+New : Frame checksum support
+New : Support pass-through mode (when using `-df`)
+API : more efficient Dictionary API : `ZSTD_compress_usingCDict()`, `ZSTD_decompress_usingDDict()`
+API : create dictionary files from custom content, by Giuseppe Ottaviano
+API : support for custom malloc/free functions
+New : controllable Dictionary ID
+New : Support for skippable frames
+
+v0.6.1
+New : zlib wrapper API, thanks to Przemyslaw Skibinski
+New : Ability to compile compressor / decompressor separately
+Changed : new lib directory structure
+Fixed : Legacy codec v0.5 compatible with dictionary decompression
+Fixed : Decoder corruption error (#173)
+Fixed : null-string roundtrip (#176)
+New : benchmark mode can select directory as input
+Experimental : midipix support, VMS support
+
+v0.6.0
+Stronger high compression modes, thanks to Przemyslaw Skibinski
+API : ZSTD_getFrameParams() provides size of decompressed content
+New : highest compression modes require `--ultra` command to fully unleash their capacity
+Fixed : zstd cli return error code > 0 and removes dst file artifact when decompression fails, thanks to Chip Turner
+
+v0.5.1
+New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski
+Changed : Dictionary builder integrated into libzstd and zstd cli
+Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`.
+Fix : high compression modes for big-endian platforms
+New : zstd cli : `-t` | `--test` command
+
+v0.5.0
+New : dictionary builder utility
+Changed : streaming & dictionary API
+Improved : better compression of small data
+
+v0.4.7
+Improved : small compression speed improvement in HC mode
+Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default
+fix : bt search bug
+
+v0.4.6
+fix : fast compression mode on Windows
+New : cmake configuration file, thanks to Artyom Dymchenko
+Improved : high compression mode on repetitive data
+New : block-level API
+New : ZSTD_duplicateCCtx()
+
+v0.4.5
+new : -m/--multiple : compress/decompress multiple files
+
+v0.4.4
+Fixed : high compression modes for Windows 32 bits
+new : external dictionary API extended to buffered mode and accessible through command line
+new : windows DLL project, thanks to Christophe Chevalier
+
+v0.4.3 :
+new : external dictionary API
+new : zstd-frugal
+
+v0.4.2 :
+Generic minor improvements for small blocks
+Fixed : big-endian compatibility, by Peter Harris (#85)
+
+v0.4.1
+Fixed : ZSTD_LEGACY_SUPPORT=0 build mode (reported by Luben)
+removed `zstd.c`
+
+v0.4.0
+Command line utility compatible with high compression levels
+Removed zstdhc => merged into zstd
+Added : ZBUFF API (see zstd_buffered.h)
+Rolling buffer support
+
+v0.3.6
+small blocks params
+
+v0.3.5
+minor generic compression improvements
+
+v0.3.4
+Faster fast cLevels
+
+v0.3.3
+Small compression ratio improvement
+
+v0.3.2
+Fixed Visual Studio
+
+v0.3.1 :
+Small compression ratio improvement
+
+v0.3
+HC mode : compression levels 2-26
+
+v0.2.2
+Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier
+
+v0.2.1
+Fix : Read errors, advanced fuzzer tests, by Hanno Böck
+
+v0.2.0
+**Breaking format change**
+Faster decompression speed
+Can still decode v0.1 format
+
+v0.1.3
+fix uninitialization warning, reported by Evan Nemerson
+
+v0.1.2
+frame concatenation support
+
+v0.1.1
+fix compression bug
+detects write-flush errors
+
+v0.1.0
+first release

Copied: vendor/zstd/1.4.4/Makefile (from r354371, vendor/zstd/dist/Makefile)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/zstd/1.4.4/Makefile	Wed Nov  6 06:43:13 2019	(r354372, copy of r354371, vendor/zstd/dist/Makefile)
@@ -0,0 +1,392 @@
+# ################################################################
+# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
+# All rights reserved.
+#
+# This source code is licensed under both the BSD-style license (found in the
+# LICENSE file in the root directory of this source tree) and the GPLv2 (found
+# in the COPYING file in the root directory of this source tree).
+# ################################################################
+
+PRGDIR   = programs
+ZSTDDIR  = lib
+BUILDIR  = build
+ZWRAPDIR = zlibWrapper
+TESTDIR  = tests
+FUZZDIR  = $(TESTDIR)/fuzz
+
+# Define nul output
+VOID = /dev/null
+
+ifneq (,$(filter Windows%,$(OS)))
+EXT =.exe
+else
+EXT =
+endif
+
+## default: Build lib-release and zstd-release
+.PHONY: default
+default: lib-release zstd-release
+
+.PHONY: all
+all: allmost examples manual contrib
+
+.PHONY: allmost
+allmost: allzstd zlibwrapper
+
+# skip zwrapper, can't build that on alternate architectures without the proper zlib installed
+.PHONY: allzstd
+allzstd: lib
+	$(MAKE) -C $(PRGDIR) all
+	$(MAKE) -C $(TESTDIR) all
+
+.PHONY: all32
+all32:
+	$(MAKE) -C $(PRGDIR) zstd32
+	$(MAKE) -C $(TESTDIR) all32
+
+.PHONY: lib lib-release libzstd.a
+lib lib-release :
+	@$(MAKE) -C $(ZSTDDIR) $@
+
+.PHONY: zstd zstd-release
+zstd zstd-release:
+	@$(MAKE) -C $(PRGDIR) $@
+	cp $(PRGDIR)/zstd$(EXT) .
+
+.PHONY: zstdmt
+zstdmt:
+	@$(MAKE) -C $(PRGDIR) $@
+	cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT)
+
+.PHONY: zlibwrapper
+zlibwrapper: lib
+	$(MAKE) -C $(ZWRAPDIR) all
+
+## test: run long-duration tests
+.PHONY: test
+DEBUGLEVEL ?= 1
+test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror
+test:
+	MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants
+	$(MAKE) -C $(TESTDIR) $@
+	ZSTD=../../programs/zstd $(MAKE) -C doc/educational_decoder test
+
+## shortest: same as `make check`
+.PHONY: shortest
+shortest:
+	$(MAKE) -C $(TESTDIR) $@
+
+## check: run basic tests for `zstd` cli
+.PHONY: check
+check: shortest
+
+## examples: build all examples in `/examples` directory
+.PHONY: examples
+examples: lib
+	CPPFLAGS=-I../lib LDFLAGS=-L../lib $(MAKE) -C examples/ all
+
+## manual: generate API documentation in html format
+.PHONY: manual
+manual:
+	$(MAKE) -C contrib/gen_html $@
+
+## man: generate man page
+.PHONY: man
+man:
+	$(MAKE) -C programs $@
+
+## contrib: build all supported projects in `/contrib` directory
+.PHONY: contrib
+contrib: lib
+	$(MAKE) -C contrib/pzstd all
+	$(MAKE) -C contrib/seekable_format/examples all
+	$(MAKE) -C contrib/largeNbDicts all
+	cd contrib/single_file_decoder/ ; ./build_test.sh
+
+.PHONY: cleanTabs
+cleanTabs:
+	cd contrib; ./cleanTabs
+
+.PHONY: clean
+clean:
+	@$(MAKE) -C $(ZSTDDIR) $@ > $(VOID)
+	@$(MAKE) -C $(PRGDIR) $@ > $(VOID)
+	@$(MAKE) -C $(TESTDIR) $@ > $(VOID)
+	@$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID)
+	@$(MAKE) -C examples/ $@ > $(VOID)
+	@$(MAKE) -C contrib/gen_html $@ > $(VOID)
+	@$(MAKE) -C contrib/pzstd $@ > $(VOID)
+	@$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID)
+	@$(MAKE) -C contrib/largeNbDicts $@ > $(VOID)
+	@$(RM) zstd$(EXT) zstdmt$(EXT) tmp*
+	@$(RM) -r lz4
+	@echo Cleaning completed
+
+#------------------------------------------------------------------------------
+# make install is validated only for Linux, macOS, Hurd and some BSD targets
+#------------------------------------------------------------------------------
+ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD DragonFly NetBSD MSYS_NT Haiku))
+
+HOST_OS = POSIX
+CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release
+
+HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0)
+EGREP_OPTIONS ?=
+ifeq ($HAVE_COLORNEVER, 1)
+EGREP_OPTIONS += --color=never
+endif
+EGREP = egrep $(EGREP_OPTIONS)
+
+# Print a two column output of targets and their description. To add a target description, put a
+# comment in the Makefile with the format "## : ".  For example:
+#
+## list: Print all targets and their descriptions (if provided)
+.PHONY: list
+list:
+	@TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \
+		| awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \
+		| $(EGREP) -v  -e '^[^[:alnum:]]' | sort); \
+	{ \
+	    printf "Target Name\tDescription\n"; \
+	    printf "%0.s-" {1..16}; printf "\t"; printf "%0.s-" {1..40}; printf "\n"; \
+	    for target in $$TARGETS; do \
+	        line=$$($(EGREP) "^##[[:space:]]+$$target:" $(lastword $(MAKEFILE_LIST))); \
+	        description=$$(echo $$line | awk '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \
+	        printf "$$target\t$$description\n"; \
+	    done \
+	} | column -t -s $$'\t'
+
+.PHONY: install armtest usan asan uasan
+install:
+	@$(MAKE) -C $(ZSTDDIR) $@
+	@$(MAKE) -C $(PRGDIR) $@
+
+.PHONY: uninstall
+uninstall:
+	@$(MAKE) -C $(ZSTDDIR) $@
+	@$(MAKE) -C $(PRGDIR) $@
+
+.PHONY: travis-install
+travis-install:
+	$(MAKE) install PREFIX=~/install_test_dir
+
+.PHONY: gcc5build
+gcc5build: clean
+	gcc-5 -v
+	CC=gcc-5 $(MAKE) all MOREFLAGS="-Werror"
+
+.PHONY: gcc6build
+gcc6build: clean
+	gcc-6 -v
+	CC=gcc-6 $(MAKE) all MOREFLAGS="-Werror"
+
+.PHONY: gcc7build
+gcc7build: clean
+	gcc-7 -v
+	CC=gcc-7 $(MAKE) all MOREFLAGS="-Werror"
+
+.PHONY: clangbuild
+clangbuild: clean
+	clang -v
+	CXX=clang++ CC=clang CFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" $(MAKE) all
+
+m32build: clean
+	gcc -v
+	$(MAKE) all32
+
+armbuild: clean
+	CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allzstd
+
+aarch64build: clean
+	CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd
+
+ppcbuild: clean
+	CC=powerpc-linux-gnu-gcc CFLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd
+
+ppc64build: clean
+	CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd
+
+armfuzz: clean
+	CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
+
+aarch64fuzz: clean
+	ld -v
+	CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
+
+ppcfuzz: clean
+	CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
+
+ppc64fuzz: clean
+	CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
+
+.PHONY: cxxtest
+cxxtest: CXXFLAGS += -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror
+cxxtest: clean
+	$(MAKE) -C $(PRGDIR) all CC="$(CXX) -Wno-deprecated" CFLAGS="$(CXXFLAGS)"   # adding -Wno-deprecated to avoid clang++ warning on dealing with C files directly
+
+gcc5test: clean
+	gcc-5 -v
+	$(MAKE) all CC=gcc-5 MOREFLAGS="-Werror"
+
+gcc6test: clean
+	gcc-6 -v
+	$(MAKE) all CC=gcc-6 MOREFLAGS="-Werror"
+
+armtest: clean
+	$(MAKE) -C $(TESTDIR) datagen   # use native, faster
+	$(MAKE) -C $(TESTDIR) test CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests
+
+aarch64test:
+	$(MAKE) -C $(TESTDIR) datagen   # use native, faster
+	$(MAKE) -C $(TESTDIR) test CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests
+
+ppctest: clean
+	$(MAKE) -C $(TESTDIR) datagen   # use native, faster
+	$(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static" FUZZER_FLAGS=--no-big-tests
+
+ppc64test: clean
+	$(MAKE) -C $(TESTDIR) datagen   # use native, faster
+	$(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests
+
+arm-ppc-compilation:
+	$(MAKE) -C $(PRGDIR) clean zstd CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static"
+	$(MAKE) -C $(PRGDIR) clean zstd CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static"
+	$(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static"
+	$(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static"
+
+regressiontest:
+	$(MAKE) -C $(FUZZDIR) regressiontest
+
+uasanregressiontest:
+	$(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=address,undefined" CXXFLAGS="-O3 -fsanitize=address,undefined"
+
+msanregressiontest:
+	$(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=memory" CXXFLAGS="-O3 -fsanitize=memory"
+
+# run UBsan with -fsanitize-recover=signed-integer-overflow
+# due to a bug in UBsan when doing pointer subtraction
+# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63303
+
+usan: clean
+	$(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=undefined -Werror"
+
+asan: clean
+	$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address -Werror"
+
+asan-%: clean
+	LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=address -Werror" $(MAKE) -C $(TESTDIR) $*
+
+msan: clean
+	$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer -Werror" HAVE_LZMA=0   # datagen.c fails this test for no obvious reason
+
+msan-%: clean
+	LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=memory -fno-omit-frame-pointer -Werror" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) HAVE_LZMA=0 $*
+
+asan32: clean
+	$(MAKE) -C $(TESTDIR) test32 CC=clang MOREFLAGS="-g -fsanitize=address"
+
+uasan: clean
+	$(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror"
+
+uasan-%: clean
+	LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror" $(MAKE) -C $(TESTDIR) $*
+
+tsan-%: clean
+	LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=thread -Werror" $(MAKE) -C $(TESTDIR) $* FUZZER_FLAGS=--no-big-tests
+
+apt-install:
+	sudo apt-get -yq --no-install-suggests --no-install-recommends --force-yes install $(APT_PACKAGES)
+
+apt-add-repo:
+	sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
+	sudo apt-get update -y -qq
+
+ppcinstall:
+	APT_PACKAGES="qemu-system-ppc qemu-user-static gcc-powerpc-linux-gnu" $(MAKE) apt-install
+
+arminstall:
+	APT_PACKAGES="qemu-system-arm qemu-user-static gcc-arm-linux-gnueabi libc6-dev-armel-cross gcc-aarch64-linux-gnu libc6-dev-arm64-cross" $(MAKE) apt-install
+
+valgrindinstall:
+	APT_PACKAGES="valgrind" $(MAKE) apt-install
+
+libc6install:
+	APT_PACKAGES="libc6-dev-i386 gcc-multilib" $(MAKE) apt-install
+
+gcc6install: apt-add-repo
+	APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-6 gcc-6-multilib" $(MAKE) apt-install
+
+gcc7install: apt-add-repo
+	APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-7 gcc-7-multilib" $(MAKE) apt-install
+
+gcc8install: apt-add-repo
+	APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-8 gcc-8-multilib" $(MAKE) apt-install
+
+gpp6install: apt-add-repo
+	APT_PACKAGES="libc6-dev-i386 g++-multilib gcc-6 g++-6 g++-6-multilib" $(MAKE) apt-install
+
+clang38install:
+	APT_PACKAGES="clang-3.8" $(MAKE) apt-install
+
+# Ubuntu 14.04 ships a too-old lz4
+lz4install:
+	[ -e lz4 ] || git clone https://github.com/lz4/lz4 && sudo $(MAKE) -C lz4 install
+
+endif
+
+
+ifneq (,$(filter MSYS%,$(shell uname)))
+HOST_OS = MSYS
+CMAKE_PARAMS = -G"MSYS Makefiles" -DZSTD_MULTITHREAD_SUPPORT:BOOL=OFF -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON
+endif
+
+
+#------------------------------------------------------------------------
+# target specific tests
+#------------------------------------------------------------------------
+ifneq (,$(filter $(HOST_OS),MSYS POSIX))
+cmakebuild:
+	cmake --version
+	$(RM) -r $(BUILDIR)/cmake/build
+	mkdir $(BUILDIR)/cmake/build
+	cd $(BUILDIR)/cmake/build ; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) .. ; $(MAKE) install ; $(MAKE) uninstall
+
+c90build: clean
+	$(CC) -v
+	CFLAGS="-std=c90 -Werror" $(MAKE) allmost  # will fail, due to missing support for `long long`
+
+gnu90build: clean
+	$(CC) -v
+	CFLAGS="-std=gnu90 -Werror" $(MAKE) allmost
+
+c99build: clean
+	$(CC) -v
+	CFLAGS="-std=c99 -Werror" $(MAKE) allmost
+
+gnu99build: clean
+	$(CC) -v
+	CFLAGS="-std=gnu99 -Werror" $(MAKE) allmost
+
+c11build: clean
+	$(CC) -v
+	CFLAGS="-std=c11 -Werror" $(MAKE) allmost
+
+bmix64build: clean
+	$(CC) -v
+	CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(TESTDIR) test
+
+bmix32build: clean
+	$(CC) -v
+	CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(TESTDIR) test
+
+bmi32build: clean
+	$(CC) -v
+	CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test
+
+# static analyzer test uses clang's scan-build
+# does not analyze zlibWrapper, due to detected issues in zlib source code
+staticAnalyze: SCANBUILD ?= scan-build
+staticAnalyze:
+	$(CC) -v
+	CC=$(CC) CPPFLAGS=-g $(SCANBUILD) --status-bugs -v $(MAKE) allzstd examples contrib
+endif

Copied: vendor/zstd/1.4.4/README.md (from r354371, vendor/zstd/dist/README.md)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/zstd/1.4.4/README.md	Wed Nov  6 06:43:13 2019	(r354372, copy of r354371, vendor/zstd/dist/README.md)
@@ -0,0 +1,173 @@
+
Zstandard

+
+__Zstandard__, or `zstd` as short version, is a fast lossless compression algorithm,
+targeting real-time compression scenarios at zlib-level and better compression ratios.
+It's backed by a very fast entropy stage, provided by [Huff0 and FSE library](https://github.com/Cyan4973/FiniteStateEntropy).
+
+The project is provided as an open-source dual [BSD](LICENSE) and [GPLv2](COPYING) licensed **C** library,
+and a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files.
+Should your project require another programming language,
+a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages).
+
+**Development branch status:**
+
+[![Build Status][travisDevBadge]][travisLink]
+[![Build status][AppveyorDevBadge]][AppveyorLink]
+[![Build status][CircleDevBadge]][CircleLink]
+[![Build status][CirrusDevBadge]][CirrusLink]
+[![Fuzzing Status][OSSFuzzBadge]][OSSFuzzLink]
+
+[travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite"
+[travisLink]: https://travis-ci.org/facebook/zstd
+[AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite"
+[AppveyorLink]: https://ci.appveyor.com/project/YannCollet/zstd-p0yf0
+[CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite"
+[CircleLink]: https://circleci.com/gh/facebook/zstd
+[CirrusDevBadge]: https://api.cirrus-ci.com/github/facebook/zstd.svg?branch=dev
+[CirrusLink]: https://cirrus-ci.com/github/facebook/zstd
+[OSSFuzzBadge]: https://oss-fuzz-build-logs.storage.googleapis.com/badges/zstd.svg
+[OSSFuzzLink]: https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zstd
+
+## Benchmarks
+
+For reference, several fast compression algorithms were tested and compared
+on a server running Arch Linux (`Linux version 5.0.5-arch1-1`),
+with a Core i9-9900K CPU @ 5.0GHz,
+using [lzbench], an open-source in-memory benchmark by @inikep
+compiled with [gcc] 8.2.1,
+on the [Silesia compression corpus].
+
+[lzbench]: https://github.com/inikep/lzbench
+[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
+[gcc]: https://gcc.gnu.org/
+
+| Compressor name         | Ratio | Compression| Decompress.|
+| ---------------         | ------| -----------| ---------- |
+| **zstd 1.4.0 -1**       | 2.884 |   530 MB/s |  1360 MB/s |
+| zlib 1.2.11 -1          | 2.743 |   110 MB/s |   440 MB/s |
+| brotli 1.0.7 -0         | 2.701 |   430 MB/s |   470 MB/s |
+| quicklz 1.5.0 -1        | 2.238 |   600 MB/s |   800 MB/s |
+| lzo1x 2.09 -1           | 2.106 |   680 MB/s |   950 MB/s |
+| lz4 1.8.3               | 2.101 |   800 MB/s |  4220 MB/s |
+| snappy 1.1.4            | 2.073 |   580 MB/s |  2020 MB/s |
+| lzf 3.6 -1              | 2.077 |   440 MB/s |   930 MB/s |
+
+[zlib]: http://www.zlib.net/
+[LZ4]: http://www.lz4.org/
+
+Zstd can also offer stronger compression ratios at the cost of compression speed.
+Speed vs Compression trade-off is configurable by small increments.
+Decompression speed is preserved and remains roughly the same at all settings,
+a property shared by most LZ compression algorithms, such as [zlib] or lzma.
+
+The following tests were run
+on a server running Linux Debian (`Linux version 4.14.0-3-amd64`)
+with a Core i7-6700K CPU @ 4.0GHz,
+using [lzbench], an open-source in-memory benchmark by @inikep
+compiled with [gcc] 7.3.0,
+on the [Silesia compression corpus].
+
+Compression Speed vs Ratio | Decompression Speed
+---------------------------|--------------------

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:44:37 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id AFD9D1AE703;
 Wed,  6 Nov 2019 08:44:37 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477KpY473lz3HZW;
 Wed,  6 Nov 2019 08:44:37 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 72725227F3;
 Wed,  6 Nov 2019 08:44:37 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68ibtS040509;
 Wed, 6 Nov 2019 08:44:37 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68ia9d040503;
 Wed, 6 Nov 2019 08:44:36 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060844.xA68ia9d040503@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:44:36 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354377 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool
 vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib...
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool
 vendor/illumos/dist/lib/libzfs/common
 vendor/illumos/dist/lib/libzfs_core/common vendo...
X-SVN-Commit-Revision: 354377
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:44:37 -0000

Author: avg
Date: Wed Nov  6 08:44:35 2019
New Revision: 354377
URL: https://svnweb.freebsd.org/changeset/base/354377

Log:
  10554 Implemented zpool sync command
  
  illumos/illumos-gate@9c2acf00e275b6b2125a306f33cdddcc58393220
  https://github.com/illumos/illumos-gate/commit/9c2acf00e275b6b2125a306f33cdddcc58393220
  
  https://www.illumos.org/issues/10554
    During the port of MMP (illumos bug 10499) from ZoL, I found this
    earlier ZoL project is a prerequisite. Here is the original
    description.  This addition will enable us to sync an open TXG to the
    main pool on demand. The functionality is similar to 'sync(2)' but
    'zpool sync' will return when data has hit the main storage instead of
    potentially just the ZIL as is the case with the 'sync(2)' cmd.
  
  Portions contributed by: Jerry Jelinek 
  Author: Alek Pinchuk 

Modified:
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c
  vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h
  vendor/illumos/dist/man/man1m/zpool.1m

Changes in other areas also in this revision:
Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c
==============================================================================
--- vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:44:35 2019	(r354377)
@@ -97,6 +97,8 @@ static int zpool_do_history(int, char **);
 static int zpool_do_get(int, char **);
 static int zpool_do_set(int, char **);
 
+static int zpool_do_sync(int, char **);
+
 /*
  * These libumem hooks provide a reasonable set of defaults for the allocator's
  * debugging facilities.
@@ -141,6 +143,7 @@ typedef enum {
 	HELP_GET,
 	HELP_SET,
 	HELP_SPLIT,
+	HELP_SYNC,
 	HELP_REGUID,
 	HELP_REOPEN
 } zpool_help_t;
@@ -197,6 +200,7 @@ static zpool_command_t command_table[] = {
 	{ "history",	zpool_do_history,	HELP_HISTORY		},
 	{ "get",	zpool_do_get,		HELP_GET		},
 	{ "set",	zpool_do_set,		HELP_SET		},
+	{ "sync",	zpool_do_sync,		HELP_SYNC		},
 };
 
 #define	NCOMMAND	(sizeof (command_table) / sizeof (command_table[0]))
@@ -283,6 +287,8 @@ get_usage(zpool_help_t idx)
 		    "[ ...]\n"));
 	case HELP_REGUID:
 		return (gettext("\treguid \n"));
+	case HELP_SYNC:
+		return (gettext("\tsync [pool] ...\n"));
 	}
 
 	abort();
@@ -2614,6 +2620,45 @@ error:
 	free(searchdirs);
 
 	return (err ? 1 : 0);
+}
+
+/*
+ * zpool sync [-f] [pool] ...
+ *
+ * -f (undocumented) force uberblock (and config including zpool cache file)
+ *    update.
+ *
+ * Sync the specified pool(s).
+ * Without arguments "zpool sync" will sync all pools.
+ * This command initiates TXG sync(s) and will return after the TXG(s) commit.
+ *
+ */
+static int
+zpool_do_sync(int argc, char **argv)
+{
+	int ret;
+	boolean_t force = B_FALSE;
+
+	/* check options */
+	while ((ret  = getopt(argc, argv, "f")) != -1) {
+		switch (ret) {
+		case 'f':
+			force = B_TRUE;
+			break;
+		case '?':
+			(void) fprintf(stderr, gettext("invalid option '%c'\n"),
+			    optopt);
+			usage(B_FALSE);
+		}
+	}
+
+	argc -= optind;
+	argv += optind;
+
+	/* if argc == 0 we will execute zpool_sync_one on all pools */
+	ret = for_each_pool(argc, argv, B_FALSE, NULL, zpool_sync_one, &force);
+
+	return (ret);
 }
 
 typedef struct iostat_cbdata {

Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h
==============================================================================
--- vendor/illumos/dist/lib/libzfs/common/libzfs.h	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/lib/libzfs/common/libzfs.h	Wed Nov  6 08:44:35 2019	(r354377)
@@ -269,6 +269,8 @@ extern int zpool_clear(zpool_handle_t *, const char *,
 extern int zpool_reguid(zpool_handle_t *);
 extern int zpool_reopen(zpool_handle_t *);
 
+extern int zpool_sync_one(zpool_handle_t *, void *);
+
 extern int zpool_vdev_online(zpool_handle_t *, const char *, int,
     vdev_state_t *);
 extern int zpool_vdev_offline(zpool_handle_t *, const char *, boolean_t);

Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
==============================================================================
--- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c	Wed Nov  6 08:44:35 2019	(r354377)
@@ -3504,6 +3504,27 @@ zpool_reopen(zpool_handle_t *zhp)
 	return (zpool_standard_error(hdl, errno, msg));
 }
 
+/* call into libzfs_core to execute the sync IOCTL per pool */
+int
+zpool_sync_one(zpool_handle_t *zhp, void *data)
+{
+	int ret;
+	libzfs_handle_t *hdl = zpool_get_handle(zhp);
+	const char *pool_name = zpool_get_name(zhp);
+	boolean_t *force = data;
+	nvlist_t *innvl = fnvlist_alloc();
+
+	fnvlist_add_boolean_value(innvl, "force", *force);
+	if ((ret = lzc_sync(pool_name, innvl, NULL)) != 0) {
+		nvlist_free(innvl);
+		return (zpool_standard_error_fmt(hdl, ret,
+		    dgettext(TEXT_DOMAIN, "sync '%s' failed"), pool_name));
+	}
+	nvlist_free(innvl);
+
+	return (0);
+}
+
 /*
  * Convert from a devid string to a path.
  */

Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c
==============================================================================
--- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c	Wed Nov  6 08:44:35 2019	(r354377)
@@ -24,6 +24,7 @@
  * Copyright (c) 2013 Steven Hartland. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2017 RackTop Systems.
+ * Copyright (c) 2017 Datto Inc.
  */
 
 /*
@@ -128,17 +129,20 @@ lzc_ioctl(zfs_ioc_t ioc, const char *name,
 {
 	zfs_cmd_t zc = { 0 };
 	int error = 0;
-	char *packed;
-	size_t size;
+	char *packed = NULL;
+	size_t size = 0;
 
 	ASSERT3S(g_refcount, >, 0);
 	VERIFY3S(g_fd, !=, -1);
 
-	(void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name));
+	if (name != NULL)
+		(void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name));
 
-	packed = fnvlist_pack(source, &size);
-	zc.zc_nvlist_src = (uint64_t)(uintptr_t)packed;
-	zc.zc_nvlist_src_size = size;
+	if (source != NULL) {
+		packed = fnvlist_pack(source, &size);
+		zc.zc_nvlist_src = (uint64_t)(uintptr_t)packed;
+		zc.zc_nvlist_src_size = size;
+	}
 
 	if (resultp != NULL) {
 		*resultp = NULL;
@@ -416,6 +420,18 @@ lzc_exists(const char *dataset)
 }
 
 /*
+ * outnvl is unused.
+ * It was added to preserve the function signature in case it is
+ * needed in the future.
+ */
+/*ARGSUSED*/
+int
+lzc_sync(const char *pool_name, nvlist_t *innvl, nvlist_t **outnvl)
+{
+	return (lzc_ioctl(ZFS_IOC_POOL_SYNC, pool_name, innvl, NULL));
+}
+
+/*
  * Create "user holds" on snapshots.  If there is a hold on a snapshot,
  * the snapshot can not be destroyed.  (However, it can be marked for deletion
  * by lzc_destroy_snaps(defer=B_TRUE).)
@@ -515,11 +531,7 @@ lzc_release(nvlist_t *holds, nvlist_t **errlist)
 int
 lzc_get_holds(const char *snapname, nvlist_t **holdsp)
 {
-	int error;
-	nvlist_t *innvl = fnvlist_alloc();
-	error = lzc_ioctl(ZFS_IOC_GET_HOLDS, snapname, innvl, holdsp);
-	fnvlist_free(innvl);
-	return (error);
+	return (lzc_ioctl(ZFS_IOC_GET_HOLDS, snapname, NULL, holdsp));
 }
 
 /*

Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h
==============================================================================
--- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h	Wed Nov  6 08:44:35 2019	(r354377)
@@ -23,6 +23,7 @@
  * Copyright (c) 2012, 2016 by Delphix. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2017 RackTop Systems.
+ * Copyright (c) 2017 Datto Inc.
  */
 
 #ifndef	_LIBZFS_CORE_H
@@ -90,6 +91,8 @@ boolean_t lzc_exists(const char *);
 
 int lzc_rollback(const char *, char *, int);
 int lzc_rollback_to(const char *, const char *);
+
+int lzc_sync(const char *, nvlist_t *, nvlist_t **);
 
 int lzc_rename(const char *, const char *);
 int lzc_destroy(const char *);

Modified: vendor/illumos/dist/man/man1m/zpool.1m
==============================================================================
--- vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:44:35 2019	(r354377)
@@ -174,6 +174,9 @@
 .Oo Ar pool Oc Ns ...
 .Op Ar interval Op Ar count
 .Nm
+.Cm sync
+.Oo Ar pool Oc Ns ...
+.Nm
 .Cm upgrade
 .Nm
 .Cm upgrade
@@ -1780,6 +1783,19 @@ Only display status for pools that are exhibiting erro
 unavailable.
 Warnings about pools not using the latest on-disk format will not be included.
 .El
+.It Xo
+.Nm
+.Cm sync
+.Oo Ar pool Oc Ns ...
+.Xc
+Forces all in-core dirty data to be written to the primary pool storage and
+not the ZIL.
+It will also update administrative information including quota reporting.
+Without arguments,
+.Nm zpool Cm sync
+will sync all pools on the system.
+Otherwise, it will only sync the specified
+.Ar pool .
 .It Xo
 .Nm
 .Cm upgrade

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:44:36 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 718721AE6FD;
 Wed,  6 Nov 2019 08:44:36 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477KpX2Mpzz3HZR;
 Wed,  6 Nov 2019 08:44:36 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 30748227F1;
 Wed,  6 Nov 2019 08:44:36 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68iavL040498;
 Wed, 6 Nov 2019 08:44:36 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68iZ0Z040496;
 Wed, 6 Nov 2019 08:44:35 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060844.xA68iZ0Z040496@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:44:35 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354377 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool
 vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib...
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool
 vendor/illumos/dist/lib/libzfs/common
 vendor/illumos/dist/lib/libzfs_core/common vendo...
X-SVN-Commit-Revision: 354377
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:44:36 -0000

Author: avg
Date: Wed Nov  6 08:44:35 2019
New Revision: 354377
URL: https://svnweb.freebsd.org/changeset/base/354377

Log:
  10554 Implemented zpool sync command
  
  illumos/illumos-gate@9c2acf00e275b6b2125a306f33cdddcc58393220
  https://github.com/illumos/illumos-gate/commit/9c2acf00e275b6b2125a306f33cdddcc58393220
  
  https://www.illumos.org/issues/10554
    During the port of MMP (illumos bug 10499) from ZoL, I found this
    earlier ZoL project is a prerequisite. Here is the original
    description.  This addition will enable us to sync an open TXG to the
    main pool on demand. The functionality is similar to 'sync(2)' but
    'zpool sync' will return when data has hit the main storage instead of
    potentially just the ZIL as is the case with the 'sync(2)' cmd.
  
  Portions contributed by: Jerry Jelinek 
  Author: Alek Pinchuk 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Changes in other areas also in this revision:
Modified:
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c
  vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h
  vendor/illumos/dist/man/man1m/zpool.1m

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c	Wed Nov  6 08:44:35 2019	(r354377)
@@ -5522,6 +5522,7 @@ zfs_ioc_hold(const char *pool, nvlist_t *args, nvlist_
 static int
 zfs_ioc_get_holds(const char *snapname, nvlist_t *args, nvlist_t *outnvl)
 {
+	ASSERT3P(args, ==, NULL);
 	return (dsl_dataset_get_holds(snapname, outnvl));
 }
 
@@ -5790,6 +5791,44 @@ out:
 	return (error);
 }
 
+/*
+ * Sync the currently open TXG to disk for the specified pool.
+ * This is somewhat similar to 'zfs_sync()'.
+ * For cases that do not result in error this ioctl will wait for
+ * the currently open TXG to commit before returning back to the caller.
+ *
+ * innvl: {
+ *  "force" -> when true, force uberblock update even if there is no dirty data.
+ *             In addition this will cause the vdev configuration to be written
+ *             out including updating the zpool cache file. (boolean_t)
+ * }
+ *
+ * onvl is unused
+ */
+/* ARGSUSED */
+static int
+zfs_ioc_pool_sync(const char *pool, nvlist_t *innvl, nvlist_t *onvl)
+{
+	int err;
+	boolean_t force;
+	spa_t *spa;
+
+	if ((err = spa_open(pool, &spa, FTAG)) != 0)
+		return (err);
+
+	force = fnvlist_lookup_boolean_value(innvl, "force");
+	if (force) {
+		spa_config_enter(spa, SCL_CONFIG, FTAG, RW_WRITER);
+		vdev_config_dirty(spa->spa_root_vdev);
+		spa_config_exit(spa, SCL_CONFIG, FTAG);
+	}
+	txg_wait_synced(spa_get_dsl(spa), 0);
+
+	spa_close(spa, FTAG);
+
+	return (err);
+}
+
 static zfs_ioc_vec_t zfs_ioc_vec[ZFS_IOC_LAST - ZFS_IOC_FIRST];
 
 static void
@@ -5979,6 +6018,10 @@ zfs_ioctl_init(void)
 	zfs_ioctl_register("initialize", ZFS_IOC_POOL_INITIALIZE,
 	    zfs_ioc_pool_initialize, zfs_secpolicy_config, POOL_NAME,
 	    POOL_CHECK_SUSPENDED | POOL_CHECK_READONLY, B_TRUE, B_TRUE);
+
+	zfs_ioctl_register("sync", ZFS_IOC_POOL_SYNC,
+	    zfs_ioc_pool_sync, zfs_secpolicy_none, POOL_NAME,
+	    POOL_CHECK_SUSPENDED | POOL_CHECK_READONLY, B_FALSE, B_FALSE);
 
 	/* IOCTLS that use the legacy function signature */
 

Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h	Wed Nov  6 08:37:55 2019	(r354376)
+++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h	Wed Nov  6 08:44:35 2019	(r354377)
@@ -988,6 +988,7 @@ typedef enum zfs_ioc {
 	ZFS_IOC_POOL_CHECKPOINT,
 	ZFS_IOC_POOL_DISCARD_CHECKPOINT,
 	ZFS_IOC_POOL_INITIALIZE,
+	ZFS_IOC_POOL_SYNC,
 	ZFS_IOC_LAST
 } zfs_ioc_t;
 

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:46:26 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2554E1AE8E7;
 Wed,  6 Nov 2019 08:46:26 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477Krf0kvXz3J5M;
 Wed,  6 Nov 2019 08:46:26 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F0677227F9;
 Wed,  6 Nov 2019 08:46:25 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68kPNh040873;
 Wed, 6 Nov 2019 08:46:25 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68kOqc040866;
 Wed, 6 Nov 2019 08:46:24 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060846.xA68kOqc040866@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:46:24 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354378 - vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumo...
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs ...
X-SVN-Commit-Revision: 354378
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:46:26 -0000

Author: avg
Date: Wed Nov  6 08:46:24 2019
New Revision: 354378
URL: https://svnweb.freebsd.org/changeset/base/354378

Log:
  10499 Multi-modifier protection (MMP)
  
  illumos/illumos-gate@e0f1c0afa46cc84d4b1e40124032a9a87310386e
  https://github.com/illumos/illumos-gate/commit/e0f1c0afa46cc84d4b1e40124032a9a87310386e
  
  https://www.illumos.org/issues/10499
    Port the following ZFS commits from ZoL to illumos.
    379ca9cf2 Multi-modifier protection (MMP)
    bbffb59ef Fix multihost stale cache file import
    0d398b256 Do not initiate MMP writes while pool is suspended
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Tim Chase 
  Portions contributed by: sanjeevbagewadi 
  Portions contributed by: John L. Hammond 
  Portions contributed by: Giuseppe Di Natale 
  Portions contributed by: Prakash Surya 
  Author: Olaf Faaland 

Added:
  vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c   (contents, props changed)
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h   (contents, props changed)
Modified:
  vendor-sys/illumos/dist/common/zfs/zfs_comutil.h
  vendor-sys/illumos/dist/common/zfs/zpool_prop.c
  vendor-sys/illumos/dist/uts/common/Makefile.files
  vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Changes in other areas also in this revision:
Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/cmd/zhack/zhack.c
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/cmd/ztest/ztest.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_import.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_status.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_util.c
  vendor/illumos/dist/lib/libzpool/common/kernel.c
  vendor/illumos/dist/man/man1m/zpool.1m

Modified: vendor-sys/illumos/dist/common/zfs/zfs_comutil.h
==============================================================================
--- vendor-sys/illumos/dist/common/zfs/zfs_comutil.h	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor-sys/illumos/dist/common/zfs/zfs_comutil.h	Wed Nov  6 08:46:24 2019	(r354378)
@@ -21,6 +21,7 @@
 /*
  * Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2012, 2017 by Delphix. All rights reserved.
+ * Copyright 2019 Joyent, Inc.
  */
 
 #ifndef	_ZFS_COMUTIL_H
@@ -32,6 +33,9 @@
 #ifdef	__cplusplus
 extern "C" {
 #endif
+
+/* Needed for ZoL errno usage in MMP kernel and user code */
+#define	EREMOTEIO EREMOTE
 
 extern boolean_t zfs_allocatable_devs(nvlist_t *);
 extern void zpool_get_load_policy(nvlist_t *, zpool_load_policy_t *);

Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c
==============================================================================
--- vendor-sys/illumos/dist/common/zfs/zpool_prop.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor-sys/illumos/dist/common/zfs/zpool_prop.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -125,6 +125,9 @@ zpool_prop_init(void)
 	    PROP_DEFAULT, ZFS_TYPE_POOL, "on | off", "EXPAND", boolean_table);
 	zprop_register_index(ZPOOL_PROP_READONLY, "readonly", 0,
 	    PROP_DEFAULT, ZFS_TYPE_POOL, "on | off", "RDONLY", boolean_table);
+	zprop_register_index(ZPOOL_PROP_MULTIHOST, "multihost", 0,
+	    PROP_DEFAULT, ZFS_TYPE_POOL, "on | off", "MULTIHOST",
+	    boolean_table);
 
 	/* default index properties */
 	zprop_register_index(ZPOOL_PROP_FAILUREMODE, "failmode",

Modified: vendor-sys/illumos/dist/uts/common/Makefile.files
==============================================================================
--- vendor-sys/illumos/dist/uts/common/Makefile.files	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor-sys/illumos/dist/uts/common/Makefile.files	Wed Nov  6 08:46:24 2019	(r354378)
@@ -1376,6 +1376,7 @@ ZFS_COMMON_OBJS +=		\
 	lz4.o			\
 	lzjb.o			\
 	metaslab.o		\
+	mmp.o			\
 	multilist.o		\
 	range_tree.o		\
 	refcount.o		\

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -50,6 +50,7 @@
 #include 
 #include 
 #include 
+#include 
 
 /*
  * ZFS Write Throttle
@@ -192,6 +193,7 @@ dsl_pool_open_impl(spa_t *spa, uint64_t txg)
 	dp->dp_meta_rootbp = *bp;
 	rrw_init(&dp->dp_config_rwlock, B_TRUE);
 	txg_init(dp, txg);
+	mmp_init(spa);
 
 	txg_list_create(&dp->dp_dirty_datasets, spa,
 	    offsetof(dsl_dataset_t, ds_dirty_link));
@@ -393,6 +395,7 @@ dsl_pool_close(dsl_pool_t *dp)
 	 */
 	arc_flush(dp->dp_spa, FALSE);
 
+	mmp_fini(dp->dp_spa);
 	txg_fini(dp);
 	dsl_scan_fini(dp);
 	dmu_buf_user_evict_wait();

Added: vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -0,0 +1,582 @@
+/*
+ * CDDL HEADER START
+ *
+ * The contents of this file are subject to the terms of the
+ * Common Development and Distribution License (the "License").
+ * You may not use this file except in compliance with the License.
+ *
+ * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ * or http://www.opensolaris.org/os/licensing.
+ * See the License for the specific language governing permissions
+ * and limitations under the License.
+ *
+ * When distributing Covered Code, include this CDDL HEADER in each
+ * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ * If applicable, add the following below this CDDL HEADER, with the
+ * fields enclosed by brackets "[]" replaced with your own identifying
+ * information: Portions Copyright [yyyy] [name of copyright owner]
+ *
+ * CDDL HEADER END
+ */
+/*
+ * Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+ * Copyright 2019 Joyent, Inc.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+/*
+ * Multi-Modifier Protection (MMP) attempts to prevent a user from importing
+ * or opening a pool on more than one host at a time.  In particular, it
+ * prevents "zpool import -f" on a host from succeeding while the pool is
+ * already imported on another host.  There are many other ways in which a
+ * device could be used by two hosts for different purposes at the same time
+ * resulting in pool damage.  This implementation does not attempt to detect
+ * those cases.
+ *
+ * MMP operates by ensuring there are frequent visible changes on disk (a
+ * "heartbeat") at all times.  And by altering the import process to check
+ * for these changes and failing the import when they are detected.  This
+ * functionality is enabled by setting the 'multihost' pool property to on.
+ *
+ * Uberblocks written by the txg_sync thread always go into the first
+ * (N-MMP_BLOCKS_PER_LABEL) slots, the remaining slots are reserved for MMP.
+ * They are used to hold uberblocks which are exactly the same as the last
+ * synced uberblock except that the ub_timestamp is frequently updated.
+ * Like all other uberblocks, the slot is written with an embedded checksum,
+ * and slots with invalid checksums are ignored.  This provides the
+ * "heartbeat", with no risk of overwriting good uberblocks that must be
+ * preserved, e.g. previous txgs and associated block pointers.
+ *
+ * Two optional fields are added to uberblock structure: ub_mmp_magic and
+ * ub_mmp_delay.  The magic field allows zfs to tell whether ub_mmp_delay is
+ * valid.  The delay field is a decaying average of the amount of time between
+ * completion of successive MMP writes, in nanoseconds.  It is used to predict
+ * how long the import must wait to detect activity in the pool, before
+ * concluding it is not in use.
+ *
+ * During import an activity test may now be performed to determine if
+ * the pool is in use.  The activity test is typically required if the
+ * ZPOOL_CONFIG_HOSTID does not match the system hostid, the pool state is
+ * POOL_STATE_ACTIVE, and the pool is not a root pool.
+ *
+ * The activity test finds the "best" uberblock (highest txg & timestamp),
+ * waits some time, and then finds the "best" uberblock again.  If the txg
+ * and timestamp in both "best" uberblocks do not match, the pool is in use
+ * by another host and the import fails.  Since the granularity of the
+ * timestamp is in seconds this activity test must take a bare minimum of one
+ * second.  In order to assure the accuracy of the activity test, the default
+ * values result in an activity test duration of 10x the mmp write interval.
+ *
+ * The "zpool import"  activity test can be expected to take a minimum time of
+ * zfs_multihost_import_intervals * zfs_multihost_interval milliseconds.  If the
+ * "best" uberblock has a valid ub_mmp_delay field, then the duration of the
+ * test may take longer if MMP writes were occurring less frequently than
+ * expected.  Additionally, the duration is then extended by a random 25% to
+ * attempt to to detect simultaneous imports.  For example, if both partner
+ * hosts are rebooted at the same time and automatically attempt to import the
+ * pool.
+ */
+
+/*
+ * Used to control the frequency of mmp writes which are performed when the
+ * 'multihost' pool property is on.  This is one factor used to determine the
+ * length of the activity check during import.
+ *
+ * The mmp write period is zfs_multihost_interval / leaf-vdevs milliseconds.
+ * This means that on average an mmp write will be issued for each leaf vdev
+ * every zfs_multihost_interval milliseconds.  In practice, the observed period
+ * can vary with the I/O load and this observed value is the delay which is
+ * stored in the uberblock.  The minimum allowed value is 100 ms.
+ */
+ulong_t zfs_multihost_interval = MMP_DEFAULT_INTERVAL;
+
+/*
+ * Used to control the duration of the activity test on import.  Smaller values
+ * of zfs_multihost_import_intervals will reduce the import time but increase
+ * the risk of failing to detect an active pool.  The total activity check time
+ * is never allowed to drop below one second.  A value of 0 is ignored and
+ * treated as if it was set to 1.
+ */
+uint_t zfs_multihost_import_intervals = MMP_DEFAULT_IMPORT_INTERVALS;
+
+/*
+ * Controls the behavior of the pool when mmp write failures are detected.
+ *
+ * When zfs_multihost_fail_intervals = 0 then mmp write failures are ignored.
+ * The failures will still be reported to the ZED which depending on its
+ * configuration may take action such as suspending the pool or taking a
+ * device offline.
+ *
+ * When zfs_multihost_fail_intervals > 0 then sequential mmp write failures will
+ * cause the pool to be suspended.  This occurs when
+ * zfs_multihost_fail_intervals * zfs_multihost_interval milliseconds have
+ * passed since the last successful mmp write.  This guarantees the activity
+ * test will see mmp writes if the
+ * pool is imported.
+ */
+uint_t zfs_multihost_fail_intervals = MMP_DEFAULT_FAIL_INTERVALS;
+
+char *mmp_tag = "mmp_write_uberblock";
+static void mmp_thread(void *arg);
+
+void
+mmp_init(spa_t *spa)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	mutex_init(&mmp->mmp_thread_lock, NULL, MUTEX_DEFAULT, NULL);
+	cv_init(&mmp->mmp_thread_cv, NULL, CV_DEFAULT, NULL);
+	mutex_init(&mmp->mmp_io_lock, NULL, MUTEX_DEFAULT, NULL);
+	mmp->mmp_kstat_id = 1;
+}
+
+void
+mmp_fini(spa_t *spa)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	mutex_destroy(&mmp->mmp_thread_lock);
+	cv_destroy(&mmp->mmp_thread_cv);
+	mutex_destroy(&mmp->mmp_io_lock);
+}
+
+static void
+mmp_thread_enter(mmp_thread_t *mmp, callb_cpr_t *cpr)
+{
+	CALLB_CPR_INIT(cpr, &mmp->mmp_thread_lock, callb_generic_cpr, FTAG);
+	mutex_enter(&mmp->mmp_thread_lock);
+}
+
+static void
+mmp_thread_exit(mmp_thread_t *mmp, kthread_t **mpp, callb_cpr_t *cpr)
+{
+	ASSERT(*mpp != NULL);
+	*mpp = NULL;
+	cv_broadcast(&mmp->mmp_thread_cv);
+	CALLB_CPR_EXIT(cpr);		/* drops &mmp->mmp_thread_lock */
+	thread_exit();
+}
+
+void
+mmp_thread_start(spa_t *spa)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	if (spa_writeable(spa)) {
+		mutex_enter(&mmp->mmp_thread_lock);
+		if (!mmp->mmp_thread) {
+			dprintf("mmp_thread_start pool %s\n",
+			    spa->spa_name);
+			mmp->mmp_thread = thread_create(NULL, 0, mmp_thread,
+			    spa, 0, &p0, TS_RUN, minclsyspri);
+		}
+		mutex_exit(&mmp->mmp_thread_lock);
+	}
+}
+
+void
+mmp_thread_stop(spa_t *spa)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	mutex_enter(&mmp->mmp_thread_lock);
+	mmp->mmp_thread_exiting = 1;
+	cv_broadcast(&mmp->mmp_thread_cv);
+
+	while (mmp->mmp_thread) {
+		cv_wait(&mmp->mmp_thread_cv, &mmp->mmp_thread_lock);
+	}
+	mutex_exit(&mmp->mmp_thread_lock);
+
+	ASSERT(mmp->mmp_thread == NULL);
+	mmp->mmp_thread_exiting = 0;
+}
+
+typedef enum mmp_vdev_state_flag {
+	MMP_FAIL_NOT_WRITABLE	= (1 << 0),
+	MMP_FAIL_WRITE_PENDING	= (1 << 1),
+} mmp_vdev_state_flag_t;
+
+/*
+ * Find a leaf vdev to write an MMP block to.  It must not have an outstanding
+ * mmp write (if so a new write will also likely block).  If there is no usable
+ * leaf, a nonzero error value is returned. The error value returned is a bit
+ * field.
+ *
+ * MMP_FAIL_WRITE_PENDING   One or more leaf vdevs are writeable, but have an
+ *                          outstanding MMP write.
+ * MMP_FAIL_NOT_WRITABLE    One or more leaf vdevs are not writeable.
+ */
+
+static int
+mmp_next_leaf(spa_t *spa)
+{
+	vdev_t *leaf;
+	vdev_t *starting_leaf;
+	int fail_mask = 0;
+
+	ASSERT(MUTEX_HELD(&spa->spa_mmp.mmp_io_lock));
+	ASSERT(spa_config_held(spa, SCL_STATE, RW_READER));
+	ASSERT(list_link_active(&spa->spa_leaf_list.list_head) == B_TRUE);
+	ASSERT(!list_is_empty(&spa->spa_leaf_list));
+
+	if (spa->spa_mmp.mmp_leaf_last_gen != spa->spa_leaf_list_gen) {
+		spa->spa_mmp.mmp_last_leaf = list_head(&spa->spa_leaf_list);
+		spa->spa_mmp.mmp_leaf_last_gen = spa->spa_leaf_list_gen;
+	}
+
+	leaf = spa->spa_mmp.mmp_last_leaf;
+	if (leaf == NULL)
+		leaf = list_head(&spa->spa_leaf_list);
+	starting_leaf = leaf;
+
+	do {
+		leaf = list_next(&spa->spa_leaf_list, leaf);
+		if (leaf == NULL)
+			leaf = list_head(&spa->spa_leaf_list);
+
+		if (!vdev_writeable(leaf)) {
+			fail_mask |= MMP_FAIL_NOT_WRITABLE;
+		} else if (leaf->vdev_mmp_pending != 0) {
+			fail_mask |= MMP_FAIL_WRITE_PENDING;
+		} else {
+			spa->spa_mmp.mmp_last_leaf = leaf;
+			return (0);
+		}
+	} while (leaf != starting_leaf);
+
+	ASSERT(fail_mask);
+
+	return (fail_mask);
+}
+
+/*
+ * MMP writes are issued on a fixed schedule, but may complete at variable,
+ * much longer, intervals.  The mmp_delay captures long periods between
+ * successful writes for any reason, including disk latency, scheduling delays,
+ * etc.
+ *
+ * The mmp_delay is usually calculated as a decaying average, but if the latest
+ * delay is higher we do not average it, so that we do not hide sudden spikes
+ * which the importing host must wait for.
+ *
+ * If writes are occurring frequently, such as due to a high rate of txg syncs,
+ * the mmp_delay could become very small.  Since those short delays depend on
+ * activity we cannot count on, we never allow mmp_delay to get lower than rate
+ * expected if only mmp_thread writes occur.
+ *
+ * If an mmp write was skipped or fails, and we have already waited longer than
+ * mmp_delay, we need to update it so the next write reflects the longer delay.
+ *
+ * Do not set mmp_delay if the multihost property is not on, so as not to
+ * trigger an activity check on import.
+ */
+static void
+mmp_delay_update(spa_t *spa, boolean_t write_completed)
+{
+	mmp_thread_t *mts = &spa->spa_mmp;
+	hrtime_t delay = gethrtime() - mts->mmp_last_write;
+
+	ASSERT(MUTEX_HELD(&mts->mmp_io_lock));
+
+	if (spa_multihost(spa) == B_FALSE) {
+		mts->mmp_delay = 0;
+		return;
+	}
+
+	if (delay > mts->mmp_delay)
+		mts->mmp_delay = delay;
+
+	if (write_completed == B_FALSE)
+		return;
+
+	mts->mmp_last_write = gethrtime();
+
+	/*
+	 * strictly less than, in case delay was changed above.
+	 */
+	if (delay < mts->mmp_delay) {
+		hrtime_t min_delay = MSEC2NSEC(zfs_multihost_interval) /
+		    MAX(1, vdev_count_leaves(spa));
+		mts->mmp_delay = MAX(((delay + mts->mmp_delay * 127) / 128),
+		    min_delay);
+	}
+}
+
+static void
+mmp_write_done(zio_t *zio)
+{
+	spa_t *spa = zio->io_spa;
+	vdev_t *vd = zio->io_vd;
+	mmp_thread_t *mts = zio->io_private;
+
+	mutex_enter(&mts->mmp_io_lock);
+	uint64_t mmp_kstat_id = vd->vdev_mmp_kstat_id;
+	hrtime_t mmp_write_duration = gethrtime() - vd->vdev_mmp_pending;
+
+	mmp_delay_update(spa, (zio->io_error == 0));
+
+	vd->vdev_mmp_pending = 0;
+	vd->vdev_mmp_kstat_id = 0;
+
+	mutex_exit(&mts->mmp_io_lock);
+	spa_config_exit(spa, SCL_STATE, mmp_tag);
+
+	abd_free(zio->io_abd);
+}
+
+/*
+ * When the uberblock on-disk is updated by a spa_sync,
+ * creating a new "best" uberblock, update the one stored
+ * in the mmp thread state, used for mmp writes.
+ */
+void
+mmp_update_uberblock(spa_t *spa, uberblock_t *ub)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	mutex_enter(&mmp->mmp_io_lock);
+	mmp->mmp_ub = *ub;
+	mmp->mmp_ub.ub_timestamp = gethrestime_sec();
+	mmp_delay_update(spa, B_TRUE);
+	mutex_exit(&mmp->mmp_io_lock);
+}
+
+/*
+ * Choose a random vdev, label, and MMP block, and write over it
+ * with a copy of the last-synced uberblock, whose timestamp
+ * has been updated to reflect that the pool is in use.
+ */
+static void
+mmp_write_uberblock(spa_t *spa)
+{
+	int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL;
+	mmp_thread_t *mmp = &spa->spa_mmp;
+	uberblock_t *ub;
+	vdev_t *vd = NULL;
+	int label, error;
+	uint64_t offset;
+
+	hrtime_t lock_acquire_time = gethrtime();
+	spa_config_enter(spa, SCL_STATE, mmp_tag, RW_READER);
+	lock_acquire_time = gethrtime() - lock_acquire_time;
+	if (lock_acquire_time > (MSEC2NSEC(MMP_MIN_INTERVAL) / 10))
+		zfs_dbgmsg("SCL_STATE acquisition took %llu ns\n",
+		    (u_longlong_t)lock_acquire_time);
+
+	mutex_enter(&mmp->mmp_io_lock);
+
+	error = mmp_next_leaf(spa);
+
+	/*
+	 * spa_mmp_history has two types of entries:
+	 * Issued MMP write: records time issued, error status, etc.
+	 * Skipped MMP write: an MMP write could not be issued because no
+	 * suitable leaf vdev was available.  See comment above struct
+	 * spa_mmp_history for details.
+	 */
+
+	if (error) {
+		mmp_delay_update(spa, B_FALSE);
+		if (mmp->mmp_skip_error == error) {
+			/*
+			 * ZoL porting note: the following is TBD
+			 * spa_mmp_history_set_skip(spa, mmp->mmp_kstat_id - 1);
+			 */
+		} else {
+			mmp->mmp_skip_error = error;
+			/*
+			 * ZoL porting note: the following is TBD
+			 * spa_mmp_history_add(spa, mmp->mmp_ub.ub_txg,
+			 * gethrestime_sec(), mmp->mmp_delay, NULL, 0,
+			 * mmp->mmp_kstat_id++, error);
+			 */
+		}
+		mutex_exit(&mmp->mmp_io_lock);
+		spa_config_exit(spa, SCL_STATE, mmp_tag);
+		return;
+	}
+
+	vd = spa->spa_mmp.mmp_last_leaf;
+	mmp->mmp_skip_error = 0;
+
+	if (mmp->mmp_zio_root == NULL)
+		mmp->mmp_zio_root = zio_root(spa, NULL, NULL,
+		    flags | ZIO_FLAG_GODFATHER);
+
+	ub = &mmp->mmp_ub;
+	ub->ub_timestamp = gethrestime_sec();
+	ub->ub_mmp_magic = MMP_MAGIC;
+	ub->ub_mmp_delay = mmp->mmp_delay;
+	vd->vdev_mmp_pending = gethrtime();
+	vd->vdev_mmp_kstat_id = mmp->mmp_kstat_id;
+
+	zio_t *zio  = zio_null(mmp->mmp_zio_root, spa, NULL, NULL, NULL, flags);
+	abd_t *ub_abd = abd_alloc_for_io(VDEV_UBERBLOCK_SIZE(vd), B_TRUE);
+	abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));
+	abd_copy_from_buf(ub_abd, ub, sizeof (uberblock_t));
+
+	mmp->mmp_kstat_id++;
+	mutex_exit(&mmp->mmp_io_lock);
+
+	offset = VDEV_UBERBLOCK_OFFSET(vd, VDEV_UBERBLOCK_COUNT(vd) -
+	    MMP_BLOCKS_PER_LABEL + spa_get_random(MMP_BLOCKS_PER_LABEL));
+
+	label = spa_get_random(VDEV_LABELS);
+	vdev_label_write(zio, vd, label, ub_abd, offset,
+	    VDEV_UBERBLOCK_SIZE(vd), mmp_write_done, mmp,
+	    flags | ZIO_FLAG_DONT_PROPAGATE);
+
+	/*
+	 * ZoL porting note: the following is TBD
+	 * (void) spa_mmp_history_add(spa, ub->ub_txg, ub->ub_timestamp,
+	 * ub->ub_mmp_delay, vd, label, vd->vdev_mmp_kstat_id, 0);
+	 */
+
+	zio_nowait(zio);
+}
+
+static void
+mmp_thread(void *arg)
+{
+	spa_t *spa = (spa_t *)arg;
+	mmp_thread_t *mmp = &spa->spa_mmp;
+	boolean_t last_spa_suspended = spa_suspended(spa);
+	boolean_t last_spa_multihost = spa_multihost(spa);
+	callb_cpr_t cpr;
+	hrtime_t max_fail_ns = zfs_multihost_fail_intervals *
+	    MSEC2NSEC(MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
+
+	mmp_thread_enter(mmp, &cpr);
+
+	/*
+	 * The mmp_write_done() function calculates mmp_delay based on the
+	 * prior value of mmp_delay and the elapsed time since the last write.
+	 * For the first mmp write, there is no "last write", so we start
+	 * with fake, but reasonable, default non-zero values.
+	 */
+	mmp->mmp_delay = MSEC2NSEC(MAX(zfs_multihost_interval,
+	    MMP_MIN_INTERVAL)) / MAX(vdev_count_leaves(spa), 1);
+	mmp->mmp_last_write = gethrtime() - mmp->mmp_delay;
+
+	while (!mmp->mmp_thread_exiting) {
+		uint64_t mmp_fail_intervals = zfs_multihost_fail_intervals;
+		uint64_t mmp_interval = MSEC2NSEC(
+		    MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
+		boolean_t suspended = spa_suspended(spa);
+		boolean_t multihost = spa_multihost(spa);
+		hrtime_t next_time;
+
+		if (multihost)
+			next_time = gethrtime() + mmp_interval /
+			    MAX(vdev_count_leaves(spa), 1);
+		else
+			next_time = gethrtime() +
+			    MSEC2NSEC(MMP_DEFAULT_INTERVAL);
+
+		/*
+		 * MMP off => on, or suspended => !suspended:
+		 * No writes occurred recently.  Update mmp_last_write to give
+		 * us some time to try.
+		 */
+		if ((!last_spa_multihost && multihost) ||
+		    (last_spa_suspended && !suspended)) {
+			mutex_enter(&mmp->mmp_io_lock);
+			mmp->mmp_last_write = gethrtime();
+			mutex_exit(&mmp->mmp_io_lock);
+		}
+
+		/*
+		 * MMP on => off:
+		 * mmp_delay == 0 tells importing node to skip activity check.
+		 */
+		if (last_spa_multihost && !multihost) {
+			mutex_enter(&mmp->mmp_io_lock);
+			mmp->mmp_delay = 0;
+			mutex_exit(&mmp->mmp_io_lock);
+		}
+		last_spa_multihost = multihost;
+		last_spa_suspended = suspended;
+
+		/*
+		 * Smooth max_fail_ns when its factors are decreased, because
+		 * making (max_fail_ns < mmp_interval) results in the pool being
+		 * immediately suspended before writes can occur at the new
+		 * higher frequency.
+		 */
+		if ((mmp_interval * mmp_fail_intervals) < max_fail_ns) {
+			max_fail_ns = ((31 * max_fail_ns) + (mmp_interval *
+			    mmp_fail_intervals)) / 32;
+		} else {
+			max_fail_ns = mmp_interval * mmp_fail_intervals;
+		}
+
+		/*
+		 * Suspend the pool if no MMP write has succeeded in over
+		 * mmp_interval * mmp_fail_intervals nanoseconds.
+		 */
+		if (!suspended && mmp_fail_intervals && multihost &&
+		    (gethrtime() - mmp->mmp_last_write) > max_fail_ns) {
+			cmn_err(CE_WARN, "MMP writes to pool '%s' have not "
+			    "succeeded in over %llus; suspending pool",
+			    spa_name(spa),
+			    NSEC2SEC(gethrtime() - mmp->mmp_last_write));
+			zio_suspend(spa, NULL, ZIO_SUSPEND_MMP);
+		}
+
+		if (multihost && !suspended)
+			mmp_write_uberblock(spa);
+
+		CALLB_CPR_SAFE_BEGIN(&cpr);
+		(void) cv_timedwait_sig_hrtime(&mmp->mmp_thread_cv,
+		    &mmp->mmp_thread_lock, next_time);
+		CALLB_CPR_SAFE_END(&cpr, &mmp->mmp_thread_lock);
+	}
+
+	/* Outstanding writes are allowed to complete. */
+	if (mmp->mmp_zio_root)
+		zio_wait(mmp->mmp_zio_root);
+
+	mmp->mmp_zio_root = NULL;
+	mmp_thread_exit(mmp, &mmp->mmp_thread, &cpr);
+}
+
+/*
+ * Signal the MMP thread to wake it, when it is sleeping on
+ * its cv.  Used when some module parameter has changed and
+ * we want the thread to know about it.
+ * Only signal if the pool is active and mmp thread is
+ * running, otherwise there is no thread to wake.
+ */
+static void
+mmp_signal_thread(spa_t *spa)
+{
+	mmp_thread_t *mmp = &spa->spa_mmp;
+
+	mutex_enter(&mmp->mmp_thread_lock);
+	if (mmp->mmp_thread)
+		cv_broadcast(&mmp->mmp_thread_cv);
+	mutex_exit(&mmp->mmp_thread_lock);
+}
+
+void
+mmp_signal_all_threads(void)
+{
+	spa_t *spa = NULL;
+
+	mutex_enter(&spa_namespace_lock);
+	while ((spa = spa_next(spa))) {
+		if (spa->spa_state == POOL_STATE_ACTIVE)
+			mmp_signal_thread(spa);
+	}
+	mutex_exit(&spa_namespace_lock);
+}

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -57,6 +57,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -546,6 +547,16 @@ spa_prop_validate(spa_t *spa, nvlist_t *props)
 				error = SET_ERROR(EINVAL);
 			break;
 
+		case ZPOOL_PROP_MULTIHOST:
+			error = nvpair_value_uint64(elem, &intval);
+			if (!error && intval > 1)
+				error = SET_ERROR(EINVAL);
+
+			if (!error && !spa_get_hostid())
+				error = SET_ERROR(ENOTSUP);
+
+			break;
+
 		case ZPOOL_PROP_BOOTFS:
 			/*
 			 * If the pool version is less than SPA_VERSION_BOOTFS,
@@ -1348,6 +1359,9 @@ spa_unload(spa_t *spa)
 		spa_config_exit(spa, SCL_ALL, spa);
 	}
 
+	if (spa->spa_mmp.mmp_thread)
+		mmp_thread_stop(spa);
+
 	/*
 	 * Wait for any outstanding async I/O to complete.
 	 */
@@ -2298,7 +2312,206 @@ vdev_count_verify_zaps(vdev_t *vd)
 	return (total);
 }
 
+/*
+ * Determine whether the activity check is required.
+ */
+static boolean_t
+spa_activity_check_required(spa_t *spa, uberblock_t *ub, nvlist_t *label,
+    nvlist_t *config)
+{
+	uint64_t state = 0;
+	uint64_t hostid = 0;
+	uint64_t tryconfig_txg = 0;
+	uint64_t tryconfig_timestamp = 0;
+	nvlist_t *nvinfo;
+
+	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
+		nvinfo = fnvlist_lookup_nvlist(config, ZPOOL_CONFIG_LOAD_INFO);
+		(void) nvlist_lookup_uint64(nvinfo, ZPOOL_CONFIG_MMP_TXG,
+		    &tryconfig_txg);
+		(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_TIMESTAMP,
+		    &tryconfig_timestamp);
+	}
+
+	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_STATE, &state);
+
+	/*
+	 * Disable the MMP activity check - This is used by zdb which
+	 * is intended to be used on potentially active pools.
+	 */
+	if (spa->spa_import_flags & ZFS_IMPORT_SKIP_MMP)
+		return (B_FALSE);
+
+	/*
+	 * Skip the activity check when the MMP feature is disabled.
+	 */
+	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay == 0)
+		return (B_FALSE);
+	/*
+	 * If the tryconfig_* values are nonzero, they are the results of an
+	 * earlier tryimport.  If they match the uberblock we just found, then
+	 * the pool has not changed and we return false so we do not test a
+	 * second time.
+	 */
+	if (tryconfig_txg && tryconfig_txg == ub->ub_txg &&
+	    tryconfig_timestamp && tryconfig_timestamp == ub->ub_timestamp)
+		return (B_FALSE);
+
+	/*
+	 * Allow the activity check to be skipped when importing the pool
+	 * on the same host which last imported it.  Since the hostid from
+	 * configuration may be stale use the one read from the label.
+	 */
+	if (nvlist_exists(label, ZPOOL_CONFIG_HOSTID))
+		hostid = fnvlist_lookup_uint64(label, ZPOOL_CONFIG_HOSTID);
+
+	if (hostid == spa_get_hostid())
+		return (B_FALSE);
+
+	/*
+	 * Skip the activity test when the pool was cleanly exported.
+	 */
+	if (state != POOL_STATE_ACTIVE)
+		return (B_FALSE);
+
+	return (B_TRUE);
+}
+
+/*
+ * Perform the import activity check.  If the user canceled the import or
+ * we detected activity then fail.
+ */
 static int
+spa_activity_check(spa_t *spa, uberblock_t *ub, nvlist_t *config)
+{
+	uint64_t import_intervals = MAX(zfs_multihost_import_intervals, 1);
+	uint64_t txg = ub->ub_txg;
+	uint64_t timestamp = ub->ub_timestamp;
+	uint64_t import_delay = NANOSEC;
+	hrtime_t import_expire;
+	nvlist_t *mmp_label = NULL;
+	vdev_t *rvd = spa->spa_root_vdev;
+	kcondvar_t cv;
+	kmutex_t mtx;
+	int error = 0;
+
+	cv_init(&cv, NULL, CV_DEFAULT, NULL);
+	mutex_init(&mtx, NULL, MUTEX_DEFAULT, NULL);
+	mutex_enter(&mtx);
+
+	/*
+	 * If ZPOOL_CONFIG_MMP_TXG is present an activity check was performed
+	 * during the earlier tryimport.  If the txg recorded there is 0 then
+	 * the pool is known to be active on another host.
+	 *
+	 * Otherwise, the pool might be in use on another node.  Check for
+	 * changes in the uberblocks on disk if necessary.
+	 */
+	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
+		nvlist_t *nvinfo = fnvlist_lookup_nvlist(config,
+		    ZPOOL_CONFIG_LOAD_INFO);
+
+		if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_TXG) &&
+		    fnvlist_lookup_uint64(nvinfo, ZPOOL_CONFIG_MMP_TXG) == 0) {
+			vdev_uberblock_load(rvd, ub, &mmp_label);
+			error = SET_ERROR(EREMOTEIO);
+			goto out;
+		}
+	}
+
+	/*
+	 * Preferentially use the zfs_multihost_interval from the node which
+	 * last imported the pool.  This value is stored in an MMP uberblock as.
+	 *
+	 * ub_mmp_delay * vdev_count_leaves() == zfs_multihost_interval
+	 */
+	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay)
+		import_delay = MAX(import_delay, import_intervals *
+		    ub->ub_mmp_delay * MAX(vdev_count_leaves(spa), 1));
+
+	/* Apply a floor using the local default values. */
+	import_delay = MAX(import_delay, import_intervals *
+	    MSEC2NSEC(MAX(zfs_multihost_interval, MMP_MIN_INTERVAL)));
+
+	zfs_dbgmsg("import_delay=%llu ub_mmp_delay=%llu import_intervals=%u "
+	    "leaves=%u", import_delay, ub->ub_mmp_delay, import_intervals,
+	    vdev_count_leaves(spa));
+
+	/* Add a small random factor in case of simultaneous imports (0-25%) */
+	import_expire = gethrtime() + import_delay +
+	    (import_delay * spa_get_random(250) / 1000);
+
+	while (gethrtime() < import_expire) {
+		vdev_uberblock_load(rvd, ub, &mmp_label);
+
+		if (txg != ub->ub_txg || timestamp != ub->ub_timestamp) {
+			error = SET_ERROR(EREMOTEIO);
+			break;
+		}
+
+		if (mmp_label) {
+			nvlist_free(mmp_label);
+			mmp_label = NULL;
+		}
+
+		error = cv_timedwait_sig(&cv, &mtx, ddi_get_lbolt() + hz);
+		if (error != -1) {
+			error = SET_ERROR(EINTR);
+			break;
+		}
+		error = 0;
+	}
+
+out:
+	mutex_exit(&mtx);
+	mutex_destroy(&mtx);
+	cv_destroy(&cv);
+
+	/*
+	 * If the pool is determined to be active store the status in the
+	 * spa->spa_load_info nvlist.  If the remote hostname or hostid are
+	 * available from configuration read from disk store them as well.
+	 * This allows 'zpool import' to generate a more useful message.
+	 *
+	 * ZPOOL_CONFIG_MMP_STATE    - observed pool status (mandatory)
+	 * ZPOOL_CONFIG_MMP_HOSTNAME - hostname from the active pool
+	 * ZPOOL_CONFIG_MMP_HOSTID   - hostid from the active pool
+	 */
+	if (error == EREMOTEIO) {
+		char *hostname = "";
+		uint64_t hostid = 0;
+
+		if (mmp_label) {
+			if (nvlist_exists(mmp_label, ZPOOL_CONFIG_HOSTNAME)) {
+				hostname = fnvlist_lookup_string(mmp_label,
+				    ZPOOL_CONFIG_HOSTNAME);
+				fnvlist_add_string(spa->spa_load_info,
+				    ZPOOL_CONFIG_MMP_HOSTNAME, hostname);
+			}
+
+			if (nvlist_exists(mmp_label, ZPOOL_CONFIG_HOSTID)) {
+				hostid = fnvlist_lookup_uint64(mmp_label,
+				    ZPOOL_CONFIG_HOSTID);
+				fnvlist_add_uint64(spa->spa_load_info,
+				    ZPOOL_CONFIG_MMP_HOSTID, hostid);
+			}
+		}
+
+		fnvlist_add_uint64(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_STATE, MMP_STATE_ACTIVE);
+		fnvlist_add_uint64(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_TXG, 0);
+
+		error = spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO);
+	}
+
+	if (mmp_label)
+		nvlist_free(mmp_label);
+
+	return (error);
+}
+
+static int
 spa_verify_host(spa_t *spa, nvlist_t *mos_config)
 {
 	uint64_t hostid;
@@ -2548,6 +2761,7 @@ spa_ld_select_uberblock(spa_t *spa, spa_import_type_t 
 	vdev_t *rvd = spa->spa_root_vdev;
 	nvlist_t *label;
 	uberblock_t *ub = &spa->spa_uberblock;
+	boolean_t activity_check = B_FALSE;
 
 	/*
 	 * If we are opening the checkpointed state of the pool by
@@ -2590,6 +2804,34 @@ spa_ld_select_uberblock(spa_t *spa, spa_import_type_t 
 	    (u_longlong_t)ub->ub_txg);
 
 	/*
+	 * For pools which have the multihost property on determine if the
+	 * pool is truly inactive and can be safely imported.  Prevent
+	 * hosts which don't have a hostid set from importing the pool.
+	 */
+	activity_check = spa_activity_check_required(spa, ub, label,
+	    spa->spa_config);
+	if (activity_check) {
+		if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay &&
+		    spa_get_hostid() == 0) {
+			nvlist_free(label);
+			fnvlist_add_uint64(spa->spa_load_info,
+			    ZPOOL_CONFIG_MMP_STATE, MMP_STATE_NO_HOSTID);
+			return (spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO));
+		}
+
+		int error = spa_activity_check(spa, ub, spa->spa_config);
+		if (error) {
+			nvlist_free(label);
+			return (error);
+		}
+
+		fnvlist_add_uint64(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_STATE, MMP_STATE_INACTIVE);
+		fnvlist_add_uint64(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_TXG, ub->ub_txg);
+	}
+
+	/*
 	 * If the pool has an unsupported version we can't open it.
 	 */
 	if (!SPA_VERSION_IS_SUPPORTED(ub->ub_version)) {
@@ -3144,6 +3386,7 @@ spa_ld_get_props(spa_t *spa)
 		spa_prop_find(spa, ZPOOL_PROP_DELEGATION, &spa->spa_delegation);
 		spa_prop_find(spa, ZPOOL_PROP_FAILUREMODE, &spa->spa_failmode);
 		spa_prop_find(spa, ZPOOL_PROP_AUTOEXPAND, &spa->spa_autoexpand);
+		spa_prop_find(spa, ZPOOL_PROP_MULTIHOST, &spa->spa_multihost);
 		spa_prop_find(spa, ZPOOL_PROP_DEDUPDITTO,
 		    &spa->spa_dedup_ditto);
 
@@ -3232,6 +3475,18 @@ spa_ld_load_vdev_metadata(spa_t *spa)
 	vdev_t *rvd = spa->spa_root_vdev;
 
 	/*
+	 * If the 'multihost' property is set, then never allow a pool to
+	 * be imported when the system hostid is zero.  The exception to
+	 * this rule is zdb which is always allowed to access pools.
+	 */
+	if (spa_multihost(spa) && spa_get_hostid() == 0 &&
+	    (spa->spa_import_flags & ZFS_IMPORT_SKIP_MMP) == 0) {
+		fnvlist_add_uint64(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_STATE, MMP_STATE_NO_HOSTID);
+		return (spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO));
+	}
+
+	/*
 	 * If the 'autoreplace' property is set, then post a resource notifying
 	 * the ZFS DE that it should not issue any faults for unopenable
 	 * devices.  We also iterate over the vdevs, and post a sysevent for any
@@ -3831,6 +4086,7 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char
 		 */
 		spa->spa_sync_on = B_TRUE;
 		txg_sync_start(spa->spa_dsl_pool);
+		mmp_thread_start(spa);
 
 		/*
 		 * Wait for all claims to sync.  We sync up to the highest
@@ -4354,10 +4610,14 @@ spa_get_stats(const char *name, nvlist_t **config,
 			    ZPOOL_CONFIG_ERRCOUNT,
 			    spa_get_errlog_size(spa)) == 0);
 
-			if (spa_suspended(spa))
+			if (spa_suspended(spa)) {
 				VERIFY(nvlist_add_uint64(*config,
 				    ZPOOL_CONFIG_SUSPENDED,
 				    spa->spa_failmode) == 0);
+				VERIFY(nvlist_add_uint64(*config,
+				    ZPOOL_CONFIG_SUSPENDED_REASON,
+				    spa->spa_suspended) == 0);
+			}
 
 			spa_add_spares(spa, *config);
 			spa_add_l2cache(spa, *config);
@@ -4444,18 +4704,6 @@ spa_validate_aux_devs(spa_t *spa, nvlist_t *nvroot, ui
 			goto out;
 		}
 
-		/*
-		 * The L2ARC currently only supports disk devices in
-		 * kernel context.  For user-level testing, we allow it.
-		 */
-#ifdef _KERNEL

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:46:27 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8194E1AE8EC;
 Wed,  6 Nov 2019 08:46:27 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477Krg3GDsz3J5Q;
 Wed,  6 Nov 2019 08:46:27 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 54100227FA;
 Wed,  6 Nov 2019 08:46:27 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68kRRX040885;
 Wed, 6 Nov 2019 08:46:27 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68kQ6O040878;
 Wed, 6 Nov 2019 08:46:26 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060846.xA68kQ6O040878@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:46:26 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354378 - vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumo...
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs ...
X-SVN-Commit-Revision: 354378
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:46:27 -0000

Author: avg
Date: Wed Nov  6 08:46:24 2019
New Revision: 354378
URL: https://svnweb.freebsd.org/changeset/base/354378

Log:
  10499 Multi-modifier protection (MMP)
  
  illumos/illumos-gate@e0f1c0afa46cc84d4b1e40124032a9a87310386e
  https://github.com/illumos/illumos-gate/commit/e0f1c0afa46cc84d4b1e40124032a9a87310386e
  
  https://www.illumos.org/issues/10499
    Port the following ZFS commits from ZoL to illumos.
    379ca9cf2 Multi-modifier protection (MMP)
    bbffb59ef Fix multihost stale cache file import
    0d398b256 Do not initiate MMP writes while pool is suspended
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Tim Chase 
  Portions contributed by: sanjeevbagewadi 
  Portions contributed by: John L. Hammond 
  Portions contributed by: Giuseppe Di Natale 
  Portions contributed by: Prakash Surya 
  Author: Olaf Faaland 

Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/cmd/zhack/zhack.c
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/cmd/ztest/ztest.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_import.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_status.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_util.c
  vendor/illumos/dist/lib/libzpool/common/kernel.c
  vendor/illumos/dist/man/man1m/zpool.1m

Changes in other areas also in this revision:
Added:
  vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c   (contents, props changed)
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h   (contents, props changed)
Modified:
  vendor-sys/illumos/dist/common/zfs/zfs_comutil.h
  vendor-sys/illumos/dist/common/zfs/zpool_prop.c
  vendor-sys/illumos/dist/uts/common/Makefile.files
  vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Modified: vendor/illumos/dist/cmd/zdb/zdb.c
==============================================================================
--- vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -24,6 +24,7 @@
  * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2017 Nexenta Systems, Inc.
+ * Copyright (c) 2017, 2018 Lawrence Livermore National Security, LLC.
  * Copyright 2017 RackTop Systems.
  */
 
@@ -2409,6 +2410,13 @@ dump_uberblock(uberblock_t *ub, const char *header, co
 	(void) printf("\tguid_sum = %llu\n", (u_longlong_t)ub->ub_guid_sum);
 	(void) printf("\ttimestamp = %llu UTC = %s",
 	    (u_longlong_t)ub->ub_timestamp, asctime(localtime(×tamp)));
+
+	(void) printf("\tmmp_magic = %016llx\n",
+	    (u_longlong_t)ub->ub_mmp_magic);
+	if (ub->ub_mmp_magic == MMP_MAGIC)
+		(void) printf("\tmmp_delay = %0llu\n",
+		    (u_longlong_t)ub->ub_mmp_delay);
+
 	if (dump_opt['u'] >= 3) {
 		char blkbuf[BP_SPRINTF_LEN];
 		snprintf_blkptr(blkbuf, sizeof (blkbuf), &ub->ub_rootbp);
@@ -2507,6 +2515,12 @@ dump_label_uberblocks(vdev_label_t *lbl, uint64_t ashi
 
 		if (uberblock_verify(ub))
 			continue;
+
+		if ((dump_opt['u'] < 4) &&
+		    (ub->ub_mmp_magic == MMP_MAGIC) && ub->ub_mmp_delay &&
+		    (i >= VDEV_UBERBLOCK_COUNT(&vd) - MMP_BLOCKS_PER_LABEL))
+			continue;
+
 		(void) snprintf(header, ZDB_MAX_UB_HEADER_SIZE,
 		    "Uberblock[%d]\n", i);
 		dump_uberblock(ub, header, "");
@@ -4142,6 +4156,22 @@ verify_device_removal_feature_counts(spa_t *spa)
 	return (ret);
 }
 
+static void
+zdb_set_skip_mmp(char *target)
+{
+	spa_t *spa;
+
+	/*
+	 * Disable the activity check to allow examination of
+	 * active pools.
+	 */
+	mutex_enter(&spa_namespace_lock);
+	if ((spa = spa_lookup(target)) != NULL) {
+		spa->spa_import_flags |= ZFS_IMPORT_SKIP_MMP;
+	}
+	mutex_exit(&spa_namespace_lock);
+}
+
 #define	BOGUS_SUFFIX "_CHECKPOINTED_UNIVERSE"
 /*
  * Import the checkpointed state of the pool specified by the target
@@ -4176,6 +4206,7 @@ import_checkpointed_state(char *target, nvlist_t *cfg,
 	}
 
 	if (cfg == NULL) {
+		zdb_set_skip_mmp(poolname);
 		error = spa_get_stats(poolname, &cfg, NULL, 0);
 		if (error != 0) {
 			fatal("Tried to read config of pool \"%s\" but "
@@ -4188,7 +4219,8 @@ import_checkpointed_state(char *target, nvlist_t *cfg,
 	fnvlist_add_string(cfg, ZPOOL_CONFIG_POOL_NAME, bogus_name);
 
 	error = spa_import(bogus_name, cfg, NULL,
-	    ZFS_IMPORT_MISSING_LOG | ZFS_IMPORT_CHECKPOINT);
+	    ZFS_IMPORT_MISSING_LOG | ZFS_IMPORT_CHECKPOINT |
+	    ZFS_IMPORT_SKIP_MMP);
 	if (error != 0) {
 		fatal("Tried to import pool \"%s\" but spa_import() failed "
 		    "with error %d\n", bogus_name, error);
@@ -5187,90 +5219,6 @@ zdb_embedded_block(char *thing)
 	free(buf);
 }
 
-static boolean_t
-pool_match(nvlist_t *cfg, char *tgt)
-{
-	uint64_t v, guid = strtoull(tgt, NULL, 0);
-	char *s;
-
-	if (guid != 0) {
-		if (nvlist_lookup_uint64(cfg, ZPOOL_CONFIG_POOL_GUID, &v) == 0)
-			return (v == guid);
-	} else {
-		if (nvlist_lookup_string(cfg, ZPOOL_CONFIG_POOL_NAME, &s) == 0)
-			return (strcmp(s, tgt) == 0);
-	}
-	return (B_FALSE);
-}
-
-static char *
-find_zpool(char **target, nvlist_t **configp, int dirc, char **dirv)
-{
-	nvlist_t *pools;
-	nvlist_t *match = NULL;
-	char *name = NULL;
-	char *sepp = NULL;
-	char sep = '\0';
-	int count = 0;
-	importargs_t args;
-
-	bzero(&args, sizeof (args));
-	args.paths = dirc;
-	args.path = dirv;
-	args.can_be_active = B_TRUE;
-
-	if ((sepp = strpbrk(*target, "/@")) != NULL) {
-		sep = *sepp;
-		*sepp = '\0';
-	}
-
-	pools = zpool_search_import(g_zfs, &args);
-
-	if (pools != NULL) {
-		nvpair_t *elem = NULL;
-		while ((elem = nvlist_next_nvpair(pools, elem)) != NULL) {
-			verify(nvpair_value_nvlist(elem, configp) == 0);
-			if (pool_match(*configp, *target)) {
-				count++;
-				if (match != NULL) {
-					/* print previously found config */
-					if (name != NULL) {
-						(void) printf("%s\n", name);
-						dump_nvlist(match, 8);
-						name = NULL;
-					}
-					(void) printf("%s\n",
-					    nvpair_name(elem));
-					dump_nvlist(*configp, 8);
-				} else {
-					match = *configp;
-					name = nvpair_name(elem);
-				}
-			}
-		}
-	}
-	if (count > 1)
-		(void) fatal("\tMatched %d pools - use pool GUID "
-		    "instead of pool name or \n"
-		    "\tpool name part of a dataset name to select pool", count);
-
-	if (sepp)
-		*sepp = sep;
-	/*
-	 * If pool GUID was specified for pool id, replace it with pool name
-	 */
-	if (name && (strstr(*target, name) != *target)) {
-		int sz = 1 + strlen(name) + ((sepp) ? strlen(sepp) : 0);
-
-		*target = umem_alloc(sz, UMEM_NOFAIL);
-		(void) snprintf(*target, sz, "%s%s", name, sepp ? sepp : "");
-	}
-
-	*configp = name ? match : NULL;
-
-	return (name);
-}
-
 int
 main(int argc, char **argv)
 {
@@ -5283,7 +5231,7 @@ main(int argc, char **argv)
 	int error = 0;
 	char **searchdirs = NULL;
 	int nsearch = 0;
-	char *target;
+	char *target, *target_pool;
 	nvlist_t *policy = NULL;
 	uint64_t max_txg = UINT64_MAX;
 	int flags = ZFS_IMPORT_MISSING_LOG;
@@ -5490,22 +5438,48 @@ main(int argc, char **argv)
 	error = 0;
 	target = argv[0];
 
+	if (strpbrk(target, "/@") != NULL) {
+		size_t targetlen;
+
+		target_pool = strdup(target);
+		*strpbrk(target_pool, "/@") = '\0';
+
+		target_is_spa = B_FALSE;
+		targetlen = strlen(target);
+		if (targetlen && target[targetlen - 1] == '/')
+			target[targetlen - 1] = '\0';
+	} else {
+		target_pool = target;
+	}
+
 	if (dump_opt['e']) {
-		char *name = find_zpool(&target, &cfg, nsearch, searchdirs);
+		importargs_t args = { 0 };
 
-		error = ENOENT;
-		if (name) {
-			if (dump_opt['C'] > 1) {
-				(void) printf("\nConfiguration for import:\n");
-				dump_nvlist(cfg, 8);
-			}
+		args.paths = nsearch;
+		args.path = searchdirs;
+		args.can_be_active = B_TRUE;
 
+		error = zpool_tryimport(g_zfs, target_pool, &cfg, &args);
+
+		if (error == 0) {
+
 			if (nvlist_add_nvlist(cfg,
 			    ZPOOL_LOAD_POLICY, policy) != 0) {
 				fatal("can't open '%s': %s",
 				    target, strerror(ENOMEM));
 			}
-			error = spa_import(name, cfg, NULL, flags);
+
+			if (dump_opt['C'] > 1) {
+				(void) printf("\nConfiguration for import:\n");
+				dump_nvlist(cfg, 8);
+			}
+
+			/*
+			 * Disable the activity check to allow examination of
+			 * active pools.
+			 */
+			error = spa_import(target_pool, cfg, NULL,
+			    flags | ZFS_IMPORT_SKIP_MMP);
 		}
 	}
 
@@ -5520,21 +5494,6 @@ main(int argc, char **argv)
 
 	}
 
-	if (strpbrk(target, "/@") != NULL) {
-		size_t targetlen;
-
-		target_is_spa = B_FALSE;
-		/*
-		 * Remove any trailing slash.  Later code would get confused
-		 * by it, but we want to allow it so that "pool/" can
-		 * indicate that we want to dump the topmost filesystem,
-		 * rather than the whole pool.
-		 */
-		targetlen = strlen(target);
-		if (targetlen != 0 && target[targetlen - 1] == '/')
-			target[targetlen - 1] = '\0';
-	}
-
 	if (error == 0) {
 		if (dump_opt['k'] && (target_is_spa || dump_opt['R'])) {
 			ASSERT(checkpoint_pool != NULL);
@@ -5548,6 +5507,7 @@ main(int argc, char **argv)
 			}
 
 		} else if (target_is_spa || dump_opt['R']) {
+			zdb_set_skip_mmp(target);
 			error = spa_open_rewind(target, &spa, FTAG, policy,
 			    NULL);
 			if (error) {
@@ -5570,6 +5530,7 @@ main(int argc, char **argv)
 				}
 			}
 		} else {
+			zdb_set_skip_mmp(target);
 			error = open_objset(target, DMU_OST_ANY, FTAG, &os);
 		}
 	}

Modified: vendor/illumos/dist/cmd/zhack/zhack.c
==============================================================================
--- vendor/illumos/dist/cmd/zhack/zhack.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/cmd/zhack/zhack.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -121,16 +121,11 @@ space_delta_cb(dmu_object_type_t bonustype, void *data
  * Target is the dataset whose pool we want to open.
  */
 static void
-import_pool(const char *target, boolean_t readonly)
+zhack_import(char *target, boolean_t readonly)
 {
 	nvlist_t *config;
-	nvlist_t *pools;
-	int error;
-	char *sepp;
-	spa_t *spa;
-	nvpair_t *elem;
 	nvlist_t *props;
-	const char *name;
+	int error;
 
 	kernel_init(readonly ? FREAD : (FREAD | FWRITE));
 	g_zfs = libzfs_init();
@@ -139,68 +134,40 @@ import_pool(const char *target, boolean_t readonly)
 	dmu_objset_register_type(DMU_OST_ZFS, space_delta_cb);
 
 	g_readonly = readonly;
-
-	/*
-	 * If we only want readonly access, it's OK if we find
-	 * a potentially-active (ie, imported into the kernel) pool from the
-	 * default cachefile.
-	 */
-	if (readonly && spa_open(target, &spa, FTAG) == 0) {
-		spa_close(spa, FTAG);
-		return;
-	}
-
 	g_importargs.unique = B_TRUE;
 	g_importargs.can_be_active = readonly;
 	g_pool = strdup(target);
-	if ((sepp = strpbrk(g_pool, "/@")) != NULL)
-		*sepp = '\0';
-	g_importargs.poolname = g_pool;
-	pools = zpool_search_import(g_zfs, &g_importargs);
 
-	if (nvlist_empty(pools)) {
-		if (!g_importargs.can_be_active) {
-			g_importargs.can_be_active = B_TRUE;
-			if (zpool_search_import(g_zfs, &g_importargs) != NULL ||
-			    spa_open(target, &spa, FTAG) == 0) {
-				fatal(spa, FTAG, "cannot import '%s': pool is "
-				    "active; run " "\"zpool export %s\" "
-				    "first\n", g_pool, g_pool);
-			}
-		}
+	error = zpool_tryimport(g_zfs, target, &config, &g_importargs);
+	if (error)
+		fatal(NULL, FTAG, "cannot import '%s': %s", target,
+		    libzfs_error_description(g_zfs));
 
-		fatal(NULL, FTAG, "cannot import '%s': no such pool "
-		    "available\n", g_pool);
-	}
-
-	elem = nvlist_next_nvpair(pools, NULL);
-	name = nvpair_name(elem);
-	verify(nvpair_value_nvlist(elem, &config) == 0);
-
 	props = NULL;
 	if (readonly) {
-		verify(nvlist_alloc(&props, NV_UNIQUE_NAME, 0) == 0);
-		verify(nvlist_add_uint64(props,
+		VERIFY(nvlist_alloc(&props, NV_UNIQUE_NAME, 0) == 0);
+		VERIFY(nvlist_add_uint64(props,
 		    zpool_prop_to_name(ZPOOL_PROP_READONLY), 1) == 0);
 	}
 
 	zfeature_checks_disable = B_TRUE;
-	error = spa_import(name, config, props, ZFS_IMPORT_NORMAL);
+	error = spa_import(target, config, props,
+	    (readonly ?  ZFS_IMPORT_SKIP_MMP : ZFS_IMPORT_NORMAL));
 	zfeature_checks_disable = B_FALSE;
 	if (error == EEXIST)
 		error = 0;
 
 	if (error)
-		fatal(NULL, FTAG, "can't import '%s': %s", name,
+		fatal(NULL, FTAG, "can't import '%s': %s", target,
 		    strerror(error));
 }
 
 static void
-zhack_spa_open(const char *target, boolean_t readonly, void *tag, spa_t **spa)
+zhack_spa_open(char *target, boolean_t readonly, void *tag, spa_t **spa)
 {
 	int err;
 
-	import_pool(target, readonly);
+	zhack_import(target, readonly);
 
 	zfeature_checks_disable = B_TRUE;
 	err = spa_open(target, spa, tag);

Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c
==============================================================================
--- vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -50,6 +50,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -1615,6 +1616,10 @@ print_status_config(zpool_handle_t *zhp, const char *n
 			(void) printf(gettext("split into new pool"));
 			break;
 
+		case VDEV_AUX_ACTIVE:
+			(void) printf(gettext("currently in use"));
+			break;
+
 		case VDEV_AUX_CHILDREN_OFFLINE:
 			(void) printf(gettext("all children offline"));
 			break;
@@ -1743,6 +1748,10 @@ print_import_config(const char *name, nvlist_t *nv, in
 			(void) printf(gettext("too many errors"));
 			break;
 
+		case VDEV_AUX_ACTIVE:
+			(void) printf(gettext("currently in use"));
+			break;
+
 		case VDEV_AUX_CHILDREN_OFFLINE:
 			(void) printf(gettext("all children offline"));
 			break;
@@ -1840,8 +1849,10 @@ show_import(nvlist_t *config)
 	vdev_stat_t *vs;
 	char *name;
 	uint64_t guid;
+	uint64_t hostid = 0;
 	char *msgid;
-	nvlist_t *nvroot;
+	char *hostname = "unknown";
+	nvlist_t *nvroot, *nvinfo;
 	int reason;
 	const char *health;
 	uint_t vsc;
@@ -1928,6 +1939,17 @@ show_import(nvlist_t *config)
 		zpool_print_unsup_feat(config);
 		break;
 
+	case ZPOOL_STATUS_HOSTID_ACTIVE:
+		(void) printf(gettext(" status: The pool is currently "
+		    "imported by another system.\n"));
+		break;
+
+	case ZPOOL_STATUS_HOSTID_REQUIRED:
+		(void) printf(gettext(" status: The pool has the "
+		    "multihost property on.  It cannot\n\tbe safely imported "
+		    "when the system hostid is not set.\n"));
+		break;
+
 	case ZPOOL_STATUS_HOSTID_MISMATCH:
 		(void) printf(gettext(" status: The pool was last accessed by "
 		    "another system.\n"));
@@ -2008,6 +2030,27 @@ show_import(nvlist_t *config)
 			    "imported. Attach the missing\n\tdevices and try "
 			    "again.\n"));
 			break;
+		case ZPOOL_STATUS_HOSTID_ACTIVE:
+			VERIFY0(nvlist_lookup_nvlist(config,
+			    ZPOOL_CONFIG_LOAD_INFO, &nvinfo));
+
+			if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_HOSTNAME))
+				hostname = fnvlist_lookup_string(nvinfo,
+				    ZPOOL_CONFIG_MMP_HOSTNAME);
+
+			if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_HOSTID))
+				hostid = fnvlist_lookup_uint64(nvinfo,
+				    ZPOOL_CONFIG_MMP_HOSTID);
+
+			(void) printf(gettext(" action: The pool must be "
+			    "exported from %s (hostid=%lx)\n\tbefore it "
+			    "can be safely imported.\n"), hostname,
+			    (unsigned long) hostid);
+			break;
+		case ZPOOL_STATUS_HOSTID_REQUIRED:
+			(void) printf(gettext(" action: Check the SMF "
+			    "svc:/system/hostid service.\n"));
+			break;
 		default:
 			(void) printf(gettext(" action: The pool cannot be "
 			    "imported due to damaged devices or data.\n"));
@@ -2055,6 +2098,31 @@ show_import(nvlist_t *config)
 	}
 }
 
+static boolean_t
+zfs_force_import_required(nvlist_t *config)
+{
+	uint64_t state;
+	uint64_t hostid = 0;
+	nvlist_t *nvinfo;
+
+	state = fnvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_STATE);
+	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_HOSTID, &hostid);
+
+	if (state != POOL_STATE_EXPORTED && hostid != get_system_hostid())
+		return (B_TRUE);
+
+	nvinfo = fnvlist_lookup_nvlist(config, ZPOOL_CONFIG_LOAD_INFO);
+	if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_STATE)) {
+		mmp_state_t mmp_state = fnvlist_lookup_uint64(nvinfo,
+		    ZPOOL_CONFIG_MMP_STATE);
+
+		if (mmp_state != MMP_STATE_INACTIVE)
+			return (B_TRUE);
+	}
+
+	return (B_FALSE);
+}
+
 /*
  * Perform the import for the given configuration.  This passes the heavy
  * lifting off to zpool_import_props(), and then mounts the datasets contained
@@ -2066,53 +2134,73 @@ do_import(nvlist_t *config, const char *newname, const
 {
 	zpool_handle_t *zhp;
 	char *name;
-	uint64_t state;
 	uint64_t version;
 
-	verify(nvlist_lookup_string(config, ZPOOL_CONFIG_POOL_NAME,
-	    &name) == 0);
+	name = fnvlist_lookup_string(config, ZPOOL_CONFIG_POOL_NAME);
+	version = fnvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION);
 
-	verify(nvlist_lookup_uint64(config,
-	    ZPOOL_CONFIG_POOL_STATE, &state) == 0);
-	verify(nvlist_lookup_uint64(config,
-	    ZPOOL_CONFIG_VERSION, &version) == 0);
 	if (!SPA_VERSION_IS_SUPPORTED(version)) {
 		(void) fprintf(stderr, gettext("cannot import '%s': pool "
 		    "is formatted using an unsupported ZFS version\n"), name);
 		return (1);
-	} else if (state != POOL_STATE_EXPORTED &&
+	} else if (zfs_force_import_required(config) &&
 	    !(flags & ZFS_IMPORT_ANY_HOST)) {
-		uint64_t hostid;
+		mmp_state_t mmp_state = MMP_STATE_INACTIVE;
+		nvlist_t *nvinfo;
 
-		if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_HOSTID,
-		    &hostid) == 0) {
-			if ((unsigned long)hostid != gethostid()) {
-				char *hostname;
-				uint64_t timestamp;
-				time_t t;
+		nvinfo = fnvlist_lookup_nvlist(config, ZPOOL_CONFIG_LOAD_INFO);
+		if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_STATE))
+			mmp_state = fnvlist_lookup_uint64(nvinfo,
+			    ZPOOL_CONFIG_MMP_STATE);
 
-				verify(nvlist_lookup_string(config,
-				    ZPOOL_CONFIG_HOSTNAME, &hostname) == 0);
-				verify(nvlist_lookup_uint64(config,
-				    ZPOOL_CONFIG_TIMESTAMP, ×tamp) == 0);
-				t = timestamp;
-				(void) fprintf(stderr, gettext("cannot import "
-				    "'%s': pool may be in use from other "
-				    "system, it was last accessed by %s "
-				    "(hostid: 0x%lx) on %s"), name, hostname,
-				    (unsigned long)hostid,
-				    asctime(localtime(&t)));
-				(void) fprintf(stderr, gettext("use '-f' to "
-				    "import anyway\n"));
-				return (1);
-			}
+		if (mmp_state == MMP_STATE_ACTIVE) {
+			char *hostname = "";
+			uint64_t hostid = 0;
+
+			if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_HOSTNAME))
+				hostname = fnvlist_lookup_string(nvinfo,
+				    ZPOOL_CONFIG_MMP_HOSTNAME);
+
+			if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_HOSTID))
+				hostid = fnvlist_lookup_uint64(nvinfo,
+				    ZPOOL_CONFIG_MMP_HOSTID);
+
+			(void) fprintf(stderr, gettext("cannot import '%s': "
+			    "pool is imported on %s (hostid: "
+			    "0x%lx)\nExport the pool on the other system, "
+			    "then run 'zpool import'.\n"),
+			    name, hostname, (unsigned long) hostid);
+		} else if (mmp_state == MMP_STATE_NO_HOSTID) {
+			(void) fprintf(stderr, gettext("Cannot import '%s': "
+			    "pool has the multihost property on and the\n"
+			    "system's hostid is not set.\n"), name);
 		} else {
+			char *hostname = "";
+			uint64_t timestamp = 0;
+			uint64_t hostid = 0;
+
+			if (nvlist_exists(config, ZPOOL_CONFIG_HOSTNAME))
+				hostname = fnvlist_lookup_string(config,
+				    ZPOOL_CONFIG_HOSTNAME);
+
+			if (nvlist_exists(config, ZPOOL_CONFIG_TIMESTAMP))
+				timestamp = fnvlist_lookup_uint64(config,
+				    ZPOOL_CONFIG_TIMESTAMP);
+
+			if (nvlist_exists(config, ZPOOL_CONFIG_HOSTID))
+				hostid = fnvlist_lookup_uint64(config,
+				    ZPOOL_CONFIG_HOSTID);
+
 			(void) fprintf(stderr, gettext("cannot import '%s': "
-			    "pool may be in use from other system\n"), name);
-			(void) fprintf(stderr, gettext("use '-f' to import "
-			    "anyway\n"));
-			return (1);
+			    "pool was previously in use from another system.\n"
+			    "Last accessed by %s (hostid=%lx) at %s"
+			    "The pool can be imported, use 'zpool import -f' "
+			    "to import the pool.\n"), name, hostname,
+			    (unsigned long)hostid, ctime((time_t *)×tamp));
+
 		}
+
+		return (1);
 	}
 
 	if (zpool_import_props(g_zfs, config, newname, props, flags) != 0)
@@ -5050,6 +5138,15 @@ status_callback(zpool_handle_t *zhp, void *data)
 		    "from a backup source.  Manually marking the device\n"
 		    "\trepaired using 'zpool clear' may allow some data "
 		    "to be recovered.\n"));
+		break;
+
+	case ZPOOL_STATUS_IO_FAILURE_MMP:
+		(void) printf(gettext("status: The pool is suspended because "
+		    "multihost writes failed or were delayed;\n\tanother "
+		    "system could import the pool undetected.\n"));
+		(void) printf(gettext("action: Make sure the pool's devices "
+		    "are connected, then reboot your system and\n\timport the "
+		    "pool.\n"));
 		break;
 
 	case ZPOOL_STATUS_IO_FAILURE_WAIT:

Modified: vendor/illumos/dist/cmd/ztest/ztest.c
==============================================================================
--- vendor/illumos/dist/cmd/ztest/ztest.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/cmd/ztest/ztest.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -126,6 +126,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 
 static int ztest_fd_data = -1;
@@ -164,6 +165,7 @@ typedef struct ztest_shared_opts {
 	uint64_t zo_time;
 	uint64_t zo_maxloops;
 	uint64_t zo_metaslab_force_ganging;
+	int zo_mmp_test;
 } ztest_shared_opts_t;
 
 static const ztest_shared_opts_t ztest_opts_defaults = {
@@ -182,6 +184,7 @@ static const ztest_shared_opts_t ztest_opts_defaults =
 	.zo_passtime = 60,		/* 60 seconds */
 	.zo_killrate = 70,		/* 70% kill rate */
 	.zo_verbose = 0,
+	.zo_mmp_test = 0,
 	.zo_init = 1,
 	.zo_time = 300,			/* 5 minutes */
 	.zo_maxloops = 50,		/* max loops during spa_freeze() */
@@ -341,6 +344,7 @@ ztest_func_t ztest_spa_create_destroy;
 ztest_func_t ztest_fault_inject;
 ztest_func_t ztest_ddt_repair;
 ztest_func_t ztest_dmu_snapshot_hold;
+ztest_func_t ztest_mmp_enable_disable;
 ztest_func_t ztest_scrub;
 ztest_func_t ztest_dsl_dataset_promote_busy;
 ztest_func_t ztest_vdev_attach_detach;
@@ -386,6 +390,7 @@ ztest_info_t ztest_info[] = {
 	{ ztest_fault_inject,			1,	&zopt_sometimes	},
 	{ ztest_ddt_repair,			1,	&zopt_sometimes	},
 	{ ztest_dmu_snapshot_hold,		1,	&zopt_sometimes	},
+	{ ztest_mmp_enable_disable,		1,	&zopt_sometimes	},
 	{ ztest_reguid,				1,	&zopt_rarely	},
 	{ ztest_scrub,				1,	&zopt_rarely	},
 	{ ztest_spa_upgrade,			1,	&zopt_rarely	},
@@ -599,6 +604,7 @@ usage(boolean_t requested)
 	    "\t[-k kill_percentage (default: %llu%%)]\n"
 	    "\t[-p pool_name (default: %s)]\n"
 	    "\t[-f dir (default: %s)] file directory for vdev files\n"
+	    "\t[-M] Multi-host simulate pool imported on remote host\n"
 	    "\t[-V] verbose (use multiple times for ever more blather)\n"
 	    "\t[-E] use existing pool instead of creating new one\n"
 	    "\t[-T time (default: %llu sec)] total run time\n"
@@ -642,7 +648,7 @@ process_options(int argc, char **argv)
 	bcopy(&ztest_opts_defaults, zo, sizeof (*zo));
 
 	while ((opt = getopt(argc, argv,
-	    "v:s:a:m:r:R:d:t:g:i:k:p:f:VET:P:hF:B:o:")) != EOF) {
+	    "v:s:a:m:r:R:d:t:g:i:k:p:f:MVET:P:hF:B:o:")) != EOF) {
 		value = 0;
 		switch (opt) {
 		case 'v':
@@ -711,6 +717,9 @@ process_options(int argc, char **argv)
 				    sizeof (zo->zo_dir));
 			}
 			break;
+		case 'M':
+			zo->zo_mmp_test = 1;
+			break;
 		case 'V':
 			zo->zo_verbose++;
 			break;
@@ -2478,6 +2487,9 @@ ztest_spa_create_destroy(ztest_ds_t *zd, uint64_t id)
 	spa_t *spa;
 	nvlist_t *nvroot;
 
+	if (zo->zo_mmp_test)
+		return;
+
 	/*
 	 * Attempt to create using a bad file.
 	 */
@@ -2509,8 +2521,58 @@ ztest_spa_create_destroy(ztest_ds_t *zd, uint64_t id)
 	rw_exit(&ztest_name_lock);
 }
 
+/*
+ * Start and then stop the MMP threads to ensure the startup and shutdown code
+ * works properly.  Actual protection and property-related code tested via ZTS.
+ */
 /* ARGSUSED */
 void
+ztest_mmp_enable_disable(ztest_ds_t *zd, uint64_t id)
+{
+	ztest_shared_opts_t *zo = &ztest_opts;
+	spa_t *spa = ztest_spa;
+
+	if (zo->zo_mmp_test)
+		return;
+
+	/*
+	 * Since enabling MMP involves setting a property, it could not be done
+	 * while the pool is suspended.
+	 */
+	if (spa_suspended(spa))
+		return;
+
+	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
+	mutex_enter(&spa->spa_props_lock);
+
+	zfs_multihost_fail_intervals = 0;
+
+	if (!spa_multihost(spa)) {
+		spa->spa_multihost = B_TRUE;
+		mmp_thread_start(spa);
+	}
+
+	mutex_exit(&spa->spa_props_lock);
+	spa_config_exit(spa, SCL_CONFIG, FTAG);
+
+	txg_wait_synced(spa_get_dsl(spa), 0);
+	mmp_signal_all_threads();
+	txg_wait_synced(spa_get_dsl(spa), 0);
+
+	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
+	mutex_enter(&spa->spa_props_lock);
+
+	if (spa_multihost(spa)) {
+		mmp_thread_stop(spa);
+		spa->spa_multihost = B_FALSE;
+	}
+
+	mutex_exit(&spa->spa_props_lock);
+	spa_config_exit(spa, SCL_CONFIG, FTAG);
+}
+
+/* ARGSUSED */
+void
 ztest_spa_upgrade(ztest_ds_t *zd, uint64_t id)
 {
 	spa_t *spa;
@@ -2519,6 +2581,9 @@ ztest_spa_upgrade(ztest_ds_t *zd, uint64_t id)
 	nvlist_t *nvroot, *props;
 	char *name;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	mutex_enter(&ztest_vdev_lock);
 	name = kmem_asprintf("%s_upgrade", ztest_opts.zo_pool);
 
@@ -2687,6 +2752,9 @@ ztest_vdev_add_remove(ztest_ds_t *zd, uint64_t id)
 	nvlist_t *nvroot;
 	int error;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	mutex_enter(&ztest_vdev_lock);
 	leaves = MAX(zs->zs_mirrors + zs->zs_splits, 1) * ztest_opts.zo_raidz;
 
@@ -2769,6 +2837,9 @@ ztest_vdev_aux_add_remove(ztest_ds_t *zd, uint64_t id)
 	uint64_t guid = 0;
 	int error;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	if (ztest_random(2) == 0) {
 		sav = &spa->spa_spares;
 		aux = ZPOOL_CONFIG_SPARES;
@@ -2864,6 +2935,9 @@ ztest_split_pool(ztest_ds_t *zd, uint64_t id)
 	uint_t c, children, schildren = 0, lastlogid = 0;
 	int error = 0;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	mutex_enter(&ztest_vdev_lock);
 
 	/* ensure we have a useable config; mirrors of raidz aren't supported */
@@ -2970,6 +3044,9 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id)
 	int oldvd_is_log;
 	int error, expected_error;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	mutex_enter(&ztest_vdev_lock);
 	leaves = MAX(zs->zs_mirrors, 1) * ztest_opts.zo_raidz;
 
@@ -5563,6 +5640,9 @@ ztest_reguid(ztest_ds_t *zd, uint64_t id)
 	uint64_t orig, load;
 	int error;
 
+	if (ztest_opts.zo_mmp_test)
+		return;
+
 	orig = spa_guid(spa);
 	load = spa_load_guid(spa);
 
@@ -6247,7 +6327,7 @@ ztest_run(ztest_shared_t *zs)
 	 * Verify that we can export the pool and reimport it under a
 	 * different name.
 	 */
-	if (ztest_random(2) == 0) {
+	if ((ztest_random(2) == 0) && !ztest_opts.zo_mmp_test) {
 		char name[ZFS_MAX_DATASET_NAME_LEN];
 		(void) snprintf(name, sizeof (name), "%s_import",
 		    ztest_opts.zo_pool);
@@ -6396,6 +6476,56 @@ make_random_props()
 }
 
 /*
+ * Import a storage pool with the given name.
+ */
+static void
+ztest_import(ztest_shared_t *zs)
+{
+	libzfs_handle_t *hdl;
+	importargs_t args = { 0 };
+	spa_t *spa;
+	nvlist_t *cfg = NULL;
+	int nsearch = 1;
+	char *searchdirs[nsearch];
+	char *name = ztest_opts.zo_pool;
+	int flags = ZFS_IMPORT_MISSING_LOG;
+	int error;
+
+	mutex_init(&ztest_vdev_lock, NULL, MUTEX_DEFAULT, NULL);
+	rw_init(&ztest_name_lock, NULL, USYNC_THREAD, NULL);
+
+	kernel_init(FREAD | FWRITE);
+	hdl = libzfs_init();
+
+	searchdirs[0] = ztest_opts.zo_dir;
+	args.paths = nsearch;
+	args.path = searchdirs;
+	args.can_be_active = B_FALSE;
+
+	error = zpool_tryimport(hdl, name, &cfg, &args);
+	if (error)
+		(void) fatal(0, "No pools found\n");
+
+	VERIFY0(spa_import(name, cfg, NULL, flags));
+	VERIFY0(spa_open(name, &spa, FTAG));
+	zs->zs_metaslab_sz =
+	    1ULL << spa->spa_root_vdev->vdev_child[0]->vdev_ms_shift;
+	spa_close(spa, FTAG);
+
+	libzfs_fini(hdl);
+	kernel_fini();
+
+	if (!ztest_opts.zo_mmp_test) {
+		ztest_run_zdb(ztest_opts.zo_pool);
+		ztest_freeze();
+		ztest_run_zdb(ztest_opts.zo_pool);
+	}
+
+	rw_destroy(&ztest_name_lock);
+	mutex_destroy(&ztest_vdev_lock);
+}
+
+/*
  * Create a storage pool with the given name and initial vdev size.
  * Then test spa_freeze() functionality.
  */
@@ -6438,12 +6568,12 @@ ztest_init(ztest_shared_t *zs)
 
 	kernel_fini();
 
-	ztest_run_zdb(ztest_opts.zo_pool);
+	if (!ztest_opts.zo_mmp_test) {
+		ztest_run_zdb(ztest_opts.zo_pool);
+		ztest_freeze();
+		ztest_run_zdb(ztest_opts.zo_pool);
+	}
 
-	ztest_freeze();
-
-	ztest_run_zdb(ztest_opts.zo_pool);
-
 	rw_destroy(&ztest_name_lock);
 	mutex_destroy(&ztest_vdev_lock);
 	mutex_destroy(&ztest_checkpoint_lock);
@@ -6603,13 +6733,19 @@ ztest_run_init(void)
 {
 	ztest_shared_t *zs = ztest_shared;
 
-	ASSERT(ztest_opts.zo_init != 0);
-
 	/*
 	 * Blow away any existing copy of zpool.cache
 	 */
 	(void) remove(spa_config_path);
 
+	if (ztest_opts.zo_init == 0) {
+		if (ztest_opts.zo_verbose >= 1)
+			(void) printf("Importing pool %s\n",
+			    ztest_opts.zo_pool);
+		ztest_import(zs);
+		return;
+	}
+
 	/*
 	 * Create and initialize our storage pool.
 	 */
@@ -6816,7 +6952,8 @@ main(int argc, char **argv)
 			(void) printf("\n");
 		}
 
-		ztest_run_zdb(ztest_opts.zo_pool);
+		if (!ztest_opts.zo_mmp_test)
+			ztest_run_zdb(ztest_opts.zo_pool);
 	}
 
 	if (ztest_opts.zo_verbose >= 1) {

Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h
==============================================================================
--- vendor/illumos/dist/lib/libzfs/common/libzfs.h	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/lib/libzfs/common/libzfs.h	Wed Nov  6 08:46:24 2019	(r354378)
@@ -23,7 +23,7 @@
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2011 Pawel Jakub Dawidek. All rights reserved.
  * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
- * Copyright (c) 2012, Joyent, Inc. All rights reserved.
+ * Copyright 2019 Joyent, Inc.
  * Copyright (c) 2013 Steven Hartland. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2016 Nexenta Systems, Inc.
@@ -130,6 +130,7 @@ typedef enum zfs_error {
 	EZFS_DIFFDATA,		/* bad zfs diff data */
 	EZFS_POOLREADONLY,	/* pool is in read-only mode */
 	EZFS_SCRUB_PAUSED,	/* scrub currently paused */
+	EZFS_ACTIVE_POOL,	/* pool is imported on a different system */
 	EZFS_NO_PENDING,	/* cannot cancel, no operation is pending */
 	EZFS_CHECKPOINT_EXISTS,	/* checkpoint exists */
 	EZFS_DISCARDING_CHECKPOINT,	/* currently discarding a checkpoint */
@@ -313,6 +314,8 @@ typedef enum {
 	/*
 	 * The following correspond to faults as defined in the (fault.fs.zfs.*)
 	 * event namespace.  Each is associated with a corresponding message ID.
+	 * This must be kept in sync with the zfs_msgid_table in
+	 * lib/libzfs/libzfs_status.c.
 	 */
 	ZPOOL_STATUS_CORRUPT_CACHE,	/* corrupt /kernel/drv/zpool.cache */
 	ZPOOL_STATUS_MISSING_DEV_R,	/* missing device with replicas */
@@ -325,8 +328,11 @@ typedef enum {
 	ZPOOL_STATUS_FAILING_DEV,	/* device experiencing errors */
 	ZPOOL_STATUS_VERSION_NEWER,	/* newer on-disk version */
 	ZPOOL_STATUS_HOSTID_MISMATCH,	/* last accessed by another system */
+	ZPOOL_STATUS_HOSTID_ACTIVE,	/* currently active on another system */
+	ZPOOL_STATUS_HOSTID_REQUIRED,	/* multihost=on and hostid=0 */
 	ZPOOL_STATUS_IO_FAILURE_WAIT,	/* failed I/O, failmode 'wait' */
 	ZPOOL_STATUS_IO_FAILURE_CONTINUE, /* failed I/O, failmode 'continue' */
+	ZPOOL_STATUS_IO_FAILURE_MMP,	/* failed MMP, failmode not 'panic' */
 	ZPOOL_STATUS_BAD_LOG,		/* cannot read log chain(s) */
 
 	/*
@@ -404,6 +410,8 @@ typedef struct importargs {
 } importargs_t;
 
 extern nvlist_t *zpool_search_import(libzfs_handle_t *, importargs_t *);
+extern int zpool_tryimport(libzfs_handle_t *hdl, char *target,
+    nvlist_t **configp, importargs_t *args);
 
 /* legacy pool search routines */
 extern nvlist_t *zpool_find_import(libzfs_handle_t *, int, char **);
@@ -723,6 +731,7 @@ extern boolean_t zfs_dataset_exists(libzfs_handle_t *,
     zfs_type_t);
 extern int zfs_spa_version(zfs_handle_t *, int *);
 extern boolean_t zfs_bookmark_exists(const char *path);
+extern ulong_t get_system_hostid(void);
 
 /*
  * Mount support functions.

Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c
==============================================================================
--- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c	Wed Nov  6 08:44:35 2019	(r354377)
+++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c	Wed Nov  6 08:46:24 2019	(r354378)
@@ -440,6 +440,8 @@ make_dataset_handle_common(zfs_handle_t *zhp, zfs_cmd_
 		zhp->zfs_head_type = ZFS_TYPE_VOLUME;
 	else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS)
 		zhp->zfs_head_type = ZFS_TYPE_FILESYSTEM;
+	else if (zhp->zfs_dmustats.dds_type == DMU_OST_OTHER)
+		return (-1);
 	else

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:47:10 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 761771AE9F2;
 Wed,  6 Nov 2019 08:47:10 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477KsV2ZdCz3JL0;
 Wed,  6 Nov 2019 08:47:10 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3C4EC227FC;
 Wed,  6 Nov 2019 08:47:10 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68lAkv040958;
 Wed, 6 Nov 2019 08:47:10 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68lA5v040957;
 Wed, 6 Nov 2019 08:47:10 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060847.xA68lA5v040957@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:47:10 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354379 - vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Commit-Revision: 354379
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:47:10 -0000

Author: avg
Date: Wed Nov  6 08:47:09 2019
New Revision: 354379
URL: https://svnweb.freebsd.org/changeset/base/354379

Log:
  10701 Correct lock ASSERTs in vdev_label_read/write
  
  illumos/illumos-gate@58447f688d5e308373ab16a3b129bc0ba0fbc154
  https://github.com/illumos/illumos-gate/commit/58447f688d5e308373ab16a3b129bc0ba0fbc154
  
  https://www.illumos.org/issues/10701
    Port of ZoL commit:
    0091d66f4e Correct lock ASSERTs in vdev_label_read/write
    At a minimum, this fixes a blown assert during an MMP test run when running on
    a DEBUG build.
  
  Portions contributed by: Jerry Jelinek 
  Author: Olaf Faaland 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c	Wed Nov  6 08:46:24 2019	(r354378)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c	Wed Nov  6 08:47:09 2019	(r354379)
@@ -184,8 +184,9 @@ static void
 vdev_label_read(zio_t *zio, vdev_t *vd, int l, abd_t *buf, uint64_t offset,
     uint64_t size, zio_done_func_t *done, void *private, int flags)
 {
-	ASSERT(spa_config_held(zio->io_spa, SCL_STATE_ALL, RW_WRITER) ==
-	    SCL_STATE_ALL);
+	ASSERT(
+	    spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE ||
+	    spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
 	ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);
 
 	zio_nowait(zio_read_phys(zio, vd,
@@ -198,17 +199,9 @@ void
 vdev_label_write(zio_t *zio, vdev_t *vd, int l, abd_t *buf, uint64_t offset,
     uint64_t size, zio_done_func_t *done, void *private, int flags)
 {
-#ifdef _KERNEL
-	/*
-	 * This assert is invalid in the user-level ztest MMP code because
-	 * the ztest thread is not in dsl_pool_sync_context. ZoL does not
-	 * build the user-level code with DEBUG so this is not an issue there.
-	 */
-	ASSERT(spa_config_held(zio->io_spa, SCL_ALL, RW_WRITER) == SCL_ALL ||
-	    (spa_config_held(zio->io_spa, SCL_CONFIG | SCL_STATE, RW_READER) ==
-	    (SCL_CONFIG | SCL_STATE) &&
-	    dsl_pool_sync_context(spa_get_dsl(zio->io_spa))));
-#endif
+	ASSERT(
+	    spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE ||
+	    spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
 	ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);
 
 	zio_nowait(zio_write_phys(zio, vd,

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:53:52 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1B7B1AEC63;
 Wed,  6 Nov 2019 08:53:52 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L1D59wHz3Jp6;
 Wed,  6 Nov 2019 08:53:52 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7BC7F229CA;
 Wed,  6 Nov 2019 08:53:52 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68rqgC046841;
 Wed, 6 Nov 2019 08:53:52 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68rqfc046840;
 Wed, 6 Nov 2019 08:53:52 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060853.xA68rqfc046840@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:53:52 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354380 - vendor/illumos/dist/man/man1m
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor/illumos/dist/man/man1m
X-SVN-Commit-Revision: 354380
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:53:52 -0000

Author: avg
Date: Wed Nov  6 08:53:52 2019
New Revision: 354380
URL: https://svnweb.freebsd.org/changeset/base/354380

Log:
  8899 zpool list property documentation doesn't match actual behaviour
  
  illumos/illumos-gate@b0e142e57dfda6f02bf1fa66973c610f5fefe8cb
  https://github.com/illumos/illumos-gate/commit/b0e142e57dfda6f02bf1fa66973c610f5fefe8cb
  
  https://www.illumos.org/issues/8899
    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221795:
    The `zpool list` section of the zpool(8) manpage says that the default list of
    displayed properties is as below:
    The default list is name, size, used, available, fragmentation, expandsize,
    capacity, health, altroot.
    This does not seem to be the case. In fact:
        # zpool list
        NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
        tank  10.9T  6.05T  4.83T         -    29%    55%  1.00x  ONLINE  -
    The Properties section of the same manpage includes the `used` property, but
    this seems not to be recognised:
        # zpool list -o used
        bad property list: invalid property 'used'
        usage:
        [...]
        #
    The usage message produced here indeed doesn't include `used`.
  
  Author: Yuri Pankov  (yuripv)

Modified:
  vendor/illumos/dist/man/man1m/zpool.1m

Modified: vendor/illumos/dist/man/man1m/zpool.1m
==============================================================================
--- vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:47:09 2019	(r354379)
+++ vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:53:52 2019	(r354380)
@@ -25,7 +25,7 @@
 .\" Copyright (c) 2017 Datto Inc.
 .\" Copyright (c) 2017 George Melikov. All Rights Reserved.
 .\"
-.Dd August 23, 2017
+.Dd December 6, 2017
 .Dt ZPOOL 1M
 .Os
 .Sh NAME
@@ -521,10 +521,8 @@ change the behavior of the pool.
 .Pp
 The following are read-only properties:
 .Bl -tag -width Ds
-.It Sy available
-Amount of storage available within the pool.
-This property can also be referred to by its shortened column name,
-.Sy avail .
+.It Cm allocated
+Amount of storage space used within the pool.
 .It Sy bootsize
 The size of the system boot partition.
 This property can only be set at pool creation time and is read-only once pool
@@ -572,8 +570,6 @@ Information about unsupported features that are enable
 See
 .Xr zpool-features 5
 for details.
-.It Sy used
-Amount of storage space used within the pool.
 .El
 .Pp
 The space usage properties report actual physical space available to the
@@ -1532,8 +1528,8 @@ See the
 .Sx Properties
 section for a list of valid properties.
 The default list is
-.Sy name , size , used , available , fragmentation , expandsize , capacity ,
-.Sy dedupratio , health , altroot .
+.Cm name , size , allocated , free , expandsize , fragmentation , capacity ,
+.Cm dedupratio , health , altroot .
 .It Fl p
 Display numbers in parsable
 .Pq exact

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:55:24 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id AADA41AED36;
 Wed,  6 Nov 2019 08:55:24 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L302wdGz3JyY;
 Wed,  6 Nov 2019 08:55:24 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 47C39229CF;
 Wed,  6 Nov 2019 08:55:24 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68tOsh046989;
 Wed, 6 Nov 2019 08:55:24 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68tO19046988;
 Wed, 6 Nov 2019 08:55:24 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060855.xA68tO19046988@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:55:24 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354381 - vendor/illumos/dist/man/man1m
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor/illumos/dist/man/man1m
X-SVN-Commit-Revision: 354381
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:55:24 -0000

Author: avg
Date: Wed Nov  6 08:55:23 2019
New Revision: 354381
URL: https://svnweb.freebsd.org/changeset/base/354381

Log:
  9521 Add checkpoint field in the default list of the zpool-list man page
  
  illumos/illumos-gate@f41179bd376293096297cdc1f32e610d44f65c8b
  https://github.com/illumos/illumos-gate/commit/f41179bd376293096297cdc1f32e610d44f65c8b
  
  https://www.illumos.org/issues/9521
    The default list of fields in the zpool list man page is missing checkpoint.
  
  Author: Eitan Adler  (eadler)

Modified:
  vendor/illumos/dist/man/man1m/zpool.1m

Modified: vendor/illumos/dist/man/man1m/zpool.1m
==============================================================================
--- vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:53:52 2019	(r354380)
+++ vendor/illumos/dist/man/man1m/zpool.1m	Wed Nov  6 08:55:23 2019	(r354381)
@@ -25,7 +25,7 @@
 .\" Copyright (c) 2017 Datto Inc.
 .\" Copyright (c) 2017 George Melikov. All Rights Reserved.
 .\"
-.Dd December 6, 2017
+.Dd April 27, 2018
 .Dt ZPOOL 1M
 .Os
 .Sh NAME
@@ -1528,7 +1528,7 @@ See the
 .Sx Properties
 section for a list of valid properties.
 The default list is
-.Cm name , size , allocated , free , expandsize , fragmentation , capacity ,
+.Cm name , size , allocated , free , checkpoint, expandsize , fragmentation , capacity ,
 .Cm dedupratio , health , altroot .
 .It Fl p
 Display numbers in parsable

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:58:08 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 196B91AEE9F;
 Wed,  6 Nov 2019 08:58:08 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L680cxNz3KF7;
 Wed,  6 Nov 2019 08:58:08 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EDD08229D2;
 Wed,  6 Nov 2019 08:58:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68w7cP047170;
 Wed, 6 Nov 2019 08:58:07 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68w787047168;
 Wed, 6 Nov 2019 08:58:07 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060858.xA68w787047168@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:58:07 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354382 - vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill...
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
 vendor/...
X-SVN-Commit-Revision: 354382
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:58:08 -0000

Author: avg
Date: Wed Nov  6 08:58:03 2019
New Revision: 354382
URL: https://svnweb.freebsd.org/changeset/base/354382

Log:
  10601 10757 Pool allocation classes
  
  illumos/illumos-gate@663207adb1669640c01c5ec6949ce78fd806efae
  https://github.com/illumos/illumos-gate/commit/663207adb1669640c01c5ec6949ce78fd806efae
  
  10601 Pool allocation classes
  https://www.illumos.org/issues/10601
    illumos port of ZoL Pool allocation classes. Includes at least these two
    commits:
    441709695 Pool allocation classes misplacing small file blocks
    cc99f275a Pool allocation classes
  
  10757 Add -gLp to zpool subcommands for alt vdev names
  https://www.illumos.org/issues/10757
    Port from ZoL of
    d2f3e292d Add -gLp to zpool subcommands for alt vdev names
    Note that a subsequent ZoL commit changed -p to -P
    a77f29f93 Change full path subcommand flag from -p to -P
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Håkan Johansson 
  Portions contributed by: Richard Yao 
  Portions contributed by: Chunwei Chen 
  Portions contributed by: loli10K 
  Author: Don Brady 

Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/cmd/zpool/zpool_vdev.c
  vendor/illumos/dist/cmd/ztest/ztest.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzpool/common/util.c
  vendor/illumos/dist/man/man1m/zfs.1m
  vendor/illumos/dist/man/man1m/zpool.1m
  vendor/illumos/dist/man/man5/zpool-features.5

Changes in other areas also in this revision:
Modified:
  vendor-sys/illumos/dist/common/zfs/zfeature_common.c
  vendor-sys/illumos/dist/common/zfs/zfeature_common.h
  vendor-sys/illumos/dist/common/zfs/zfs_prop.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Modified: vendor/illumos/dist/cmd/zdb/zdb.c
==============================================================================
--- vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -21,7 +21,7 @@
 
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2017 Nexenta Systems, Inc.
  * Copyright (c) 2017, 2018 Lawrence Livermore National Security, LLC.
@@ -931,13 +931,23 @@ dump_metaslab(metaslab_t *msp)
 static void
 print_vdev_metaslab_header(vdev_t *vd)
 {
-	(void) printf("\tvdev %10llu\n\t%-10s%5llu   %-19s   %-15s   %-10s\n",
-	    (u_longlong_t)vd->vdev_id,
+	vdev_alloc_bias_t alloc_bias = vd->vdev_alloc_bias;
+	const char *bias_str;
+
+	bias_str = (alloc_bias == VDEV_BIAS_LOG || vd->vdev_islog) ?
+	    VDEV_ALLOC_BIAS_LOG :
+	    (alloc_bias == VDEV_BIAS_SPECIAL) ? VDEV_ALLOC_BIAS_SPECIAL :
+	    (alloc_bias == VDEV_BIAS_DEDUP) ? VDEV_ALLOC_BIAS_DEDUP :
+	    vd->vdev_islog ? "log" : "";
+
+	(void) printf("\tvdev %10llu   %s\n"
+	    "\t%-10s%5llu   %-19s   %-15s   %-12s\n",
+	    (u_longlong_t)vd->vdev_id, bias_str,
 	    "metaslabs", (u_longlong_t)vd->vdev_ms_count,
 	    "offset", "spacemap", "free");
-	(void) printf("\t%15s   %19s   %15s   %10s\n",
+	(void) printf("\t%15s   %19s   %15s   %12s\n",
 	    "---------------", "-------------------",
-	    "---------------", "-------------");
+	    "---------------", "------------");
 }
 
 static void
@@ -953,7 +963,7 @@ dump_metaslab_groups(spa_t *spa)
 		vdev_t *tvd = rvd->vdev_child[c];
 		metaslab_group_t *mg = tvd->vdev_mg;
 
-		if (mg->mg_class != mc)
+		if (mg == NULL || mg->mg_class != mc)
 			continue;
 
 		metaslab_group_histogram_verify(mg);
@@ -2767,6 +2777,7 @@ typedef struct zdb_blkstats {
 	uint64_t zb_count;
 	uint64_t zb_gangs;
 	uint64_t zb_ditto_samevdev;
+	uint64_t zb_ditto_same_ms;
 	uint64_t zb_psize_histogram[PSIZE_HISTO_SIZE];
 } zdb_blkstats_t;
 
@@ -2806,6 +2817,16 @@ typedef struct zdb_cb {
 	uint32_t	**zcb_vd_obsolete_counts;
 } zdb_cb_t;
 
+/* test if two DVA offsets from same vdev are within the same metaslab */
+static boolean_t
+same_metaslab(spa_t *spa, uint64_t vdev, uint64_t off1, uint64_t off2)
+{
+	vdev_t *vd = vdev_lookup_top(spa, vdev);
+	uint64_t ms_shift = vd->vdev_ms_shift;
+
+	return ((off1 >> ms_shift) == (off2 >> ms_shift));
+}
+
 static void
 zdb_count_block(zdb_cb_t *zcb, zilog_t *zilog, const blkptr_t *bp,
     dmu_object_type_t type)
@@ -2817,6 +2838,8 @@ zdb_count_block(zdb_cb_t *zcb, zilog_t *zilog, const b
 	if (zilog && zil_bp_tree_add(zilog, bp) != 0)
 		return;
 
+	spa_config_enter(zcb->zcb_spa, SCL_CONFIG, FTAG, RW_READER);
+
 	for (int i = 0; i < 4; i++) {
 		int l = (i < 2) ? BP_GET_LEVEL(bp) : ZB_TOTAL;
 		int t = (i & 1) ? type : ZDB_OT_TOTAL;
@@ -2842,8 +2865,15 @@ zdb_count_block(zdb_cb_t *zcb, zilog_t *zilog, const b
 		switch (BP_GET_NDVAS(bp)) {
 		case 2:
 			if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
-			    DVA_GET_VDEV(&bp->blk_dva[1]))
+			    DVA_GET_VDEV(&bp->blk_dva[1])) {
 				zb->zb_ditto_samevdev++;
+
+				if (same_metaslab(zcb->zcb_spa,
+				    DVA_GET_VDEV(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[1])))
+					zb->zb_ditto_same_ms++;
+			}
 			break;
 		case 3:
 			equal = (DVA_GET_VDEV(&bp->blk_dva[0]) ==
@@ -2852,13 +2882,37 @@ zdb_count_block(zdb_cb_t *zcb, zilog_t *zilog, const b
 			    DVA_GET_VDEV(&bp->blk_dva[2])) +
 			    (DVA_GET_VDEV(&bp->blk_dva[1]) ==
 			    DVA_GET_VDEV(&bp->blk_dva[2]));
-			if (equal != 0)
+			if (equal != 0) {
 				zb->zb_ditto_samevdev++;
+
+				if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
+				    DVA_GET_VDEV(&bp->blk_dva[1]) &&
+				    same_metaslab(zcb->zcb_spa,
+				    DVA_GET_VDEV(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[1])))
+					zb->zb_ditto_same_ms++;
+				else if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
+				    DVA_GET_VDEV(&bp->blk_dva[2]) &&
+				    same_metaslab(zcb->zcb_spa,
+				    DVA_GET_VDEV(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[0]),
+				    DVA_GET_OFFSET(&bp->blk_dva[2])))
+					zb->zb_ditto_same_ms++;
+				else if (DVA_GET_VDEV(&bp->blk_dva[1]) ==
+				    DVA_GET_VDEV(&bp->blk_dva[2]) &&
+				    same_metaslab(zcb->zcb_spa,
+				    DVA_GET_VDEV(&bp->blk_dva[1]),
+				    DVA_GET_OFFSET(&bp->blk_dva[1]),
+				    DVA_GET_OFFSET(&bp->blk_dva[2])))
+					zb->zb_ditto_same_ms++;
+			}
 			break;
 		}
-
 	}
 
+	spa_config_exit(zcb->zcb_spa, SCL_CONFIG, FTAG);
+
 	if (BP_IS_EMBEDDED(bp)) {
 		zcb->zcb_embedded_blocks[BPE_GET_ETYPE(bp)]++;
 		zcb->zcb_embedded_histogram[BPE_GET_ETYPE(bp)]
@@ -3665,6 +3719,7 @@ dump_block_stats(spa_t *spa)
 	uint64_t norm_alloc, norm_space, total_alloc, total_found;
 	int flags = TRAVERSE_PRE | TRAVERSE_PREFETCH_METADATA | TRAVERSE_HARD;
 	boolean_t leaks = B_FALSE;
+	int err;
 
 	bzero(&zcb, sizeof (zcb));
 	(void) printf("\nTraversing all blocks %s%s%s%s%s...\n\n",
@@ -3707,8 +3762,10 @@ dump_block_stats(spa_t *spa)
 		flags |= TRAVERSE_PREFETCH_DATA;
 
 	zcb.zcb_totalasize = metaslab_class_get_alloc(spa_normal_class(spa));
+	zcb.zcb_totalasize += metaslab_class_get_alloc(spa_special_class(spa));
+	zcb.zcb_totalasize += metaslab_class_get_alloc(spa_dedup_class(spa));
 	zcb.zcb_start = zcb.zcb_lastprint = gethrtime();
-	zcb.zcb_haderrors |= traverse_pool(spa, 0, flags, zdb_blkptr_cb, &zcb);
+	err = traverse_pool(spa, 0, flags, zdb_blkptr_cb, &zcb);
 
 	/*
 	 * If we've traversed the data blocks then we need to wait for those
@@ -3724,6 +3781,12 @@ dump_block_stats(spa_t *spa)
 		}
 	}
 
+	/*
+	 * Done after zio_wait() since zcb_haderrors is modified in
+	 * zdb_blkptr_done()
+	 */
+	zcb.zcb_haderrors |= err;
+
 	if (zcb.zcb_haderrors) {
 		(void) printf("\nError counts:\n\n");
 		(void) printf("\t%5s  %s\n", "errno", "count");
@@ -3745,7 +3808,10 @@ dump_block_stats(spa_t *spa)
 	norm_alloc = metaslab_class_get_alloc(spa_normal_class(spa));
 	norm_space = metaslab_class_get_space(spa_normal_class(spa));
 
-	total_alloc = norm_alloc + metaslab_class_get_alloc(spa_log_class(spa));
+	total_alloc = norm_alloc +
+	    metaslab_class_get_alloc(spa_log_class(spa)) +
+	    metaslab_class_get_alloc(spa_special_class(spa)) +
+	    metaslab_class_get_alloc(spa_dedup_class(spa));
 	total_found = tzb->zb_asize - zcb.zcb_dedup_asize +
 	    zcb.zcb_removing_size + zcb.zcb_checkpoint_size;
 
@@ -3767,31 +3833,50 @@ dump_block_stats(spa_t *spa)
 		return (2);
 
 	(void) printf("\n");
-	(void) printf("\tbp count:      %10llu\n",
+	(void) printf("\t%-16s %14llu\n", "bp count:",
 	    (u_longlong_t)tzb->zb_count);
-	(void) printf("\tganged count:  %10llu\n",
+	(void) printf("\t%-16s %14llu\n", "ganged count:",
 	    (longlong_t)tzb->zb_gangs);
-	(void) printf("\tbp logical:    %10llu      avg: %6llu\n",
+	(void) printf("\t%-16s %14llu      avg: %6llu\n", "bp logical:",
 	    (u_longlong_t)tzb->zb_lsize,
 	    (u_longlong_t)(tzb->zb_lsize / tzb->zb_count));
-	(void) printf("\tbp physical:   %10llu      avg:"
-	    " %6llu     compression: %6.2f\n",
-	    (u_longlong_t)tzb->zb_psize,
+	(void) printf("\t%-16s %14llu      avg: %6llu     compression: %6.2f\n",
+	    "bp physical:", (u_longlong_t)tzb->zb_psize,
 	    (u_longlong_t)(tzb->zb_psize / tzb->zb_count),
 	    (double)tzb->zb_lsize / tzb->zb_psize);
-	(void) printf("\tbp allocated:  %10llu      avg:"
-	    " %6llu     compression: %6.2f\n",
-	    (u_longlong_t)tzb->zb_asize,
+	(void) printf("\t%-16s %14llu      avg: %6llu     compression: %6.2f\n",
+	    "bp allocated:", (u_longlong_t)tzb->zb_asize,
 	    (u_longlong_t)(tzb->zb_asize / tzb->zb_count),
 	    (double)tzb->zb_lsize / tzb->zb_asize);
-	(void) printf("\tbp deduped:    %10llu    ref>1:"
-	    " %6llu   deduplication: %6.2f\n",
-	    (u_longlong_t)zcb.zcb_dedup_asize,
+	(void) printf("\t%-16s %14llu    ref>1: %6llu   deduplication: %6.2f\n",
+	    "bp deduped:", (u_longlong_t)zcb.zcb_dedup_asize,
 	    (u_longlong_t)zcb.zcb_dedup_blocks,
 	    (double)zcb.zcb_dedup_asize / tzb->zb_asize + 1.0);
-	(void) printf("\tSPA allocated: %10llu     used: %5.2f%%\n",
+	(void) printf("\t%-16s %14llu     used: %5.2f%%\n", "Normal class:",
 	    (u_longlong_t)norm_alloc, 100.0 * norm_alloc / norm_space);
 
+	if (spa_special_class(spa)->mc_rotor != NULL) {
+		uint64_t alloc = metaslab_class_get_alloc(
+		    spa_special_class(spa));
+		uint64_t space = metaslab_class_get_space(
+		    spa_special_class(spa));
+
+		(void) printf("\t%-16s %14llu     used: %5.2f%%\n",
+		    "Special class", (u_longlong_t)alloc,
+		    100.0 * alloc / space);
+	}
+
+	if (spa_dedup_class(spa)->mc_rotor != NULL) {
+		uint64_t alloc = metaslab_class_get_alloc(
+		    spa_dedup_class(spa));
+		uint64_t space = metaslab_class_get_space(
+		    spa_dedup_class(spa));
+
+		(void) printf("\t%-16s %14llu     used: %5.2f%%\n",
+		    "Dedup class", (u_longlong_t)alloc,
+		    100.0 * alloc / space);
+	}
+
 	for (bp_embedded_type_t i = 0; i < NUM_BP_EMBEDDED_TYPES; i++) {
 		if (zcb.zcb_embedded_blocks[i] == 0)
 			continue;
@@ -3812,6 +3897,10 @@ dump_block_stats(spa_t *spa)
 	if (tzb->zb_ditto_samevdev != 0) {
 		(void) printf("\tDittoed blocks on same vdev: %llu\n",
 		    (longlong_t)tzb->zb_ditto_samevdev);
+	}
+	if (tzb->zb_ditto_same_ms != 0) {
+		(void) printf("\tDittoed blocks in same metaslab: %llu\n",
+		    (longlong_t)tzb->zb_ditto_same_ms);
 	}
 
 	for (uint64_t v = 0; v < spa->spa_root_vdev->vdev_children; v++) {

Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c
==============================================================================
--- vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor/illumos/dist/cmd/zpool/zpool_main.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -27,6 +27,7 @@
  * Copyright 2016 Igor Kozhukhov .
  * Copyright 2016 Nexenta Systems, Inc.
  * Copyright (c) 2017 Datto Inc.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #include 
@@ -206,6 +207,8 @@ static zpool_command_t command_table[] = {
 
 #define	NCOMMAND	(sizeof (command_table) / sizeof (command_table[0]))
 
+#define	VDEV_ALLOC_CLASS_LOGS	"logs"
+
 static zpool_command_t *current_command;
 static char history_str[HIS_MAX_RECORD_LEN];
 static boolean_t log_history = B_TRUE;
@@ -216,7 +219,7 @@ get_usage(zpool_help_t idx)
 {
 	switch (idx) {
 	case HELP_ADD:
-		return (gettext("\tadd [-fn]   ...\n"));
+		return (gettext("\tadd [-fgLnP]   ...\n"));
 	case HELP_ATTACH:
 		return (gettext("\tattach [-f]   "
 		    "\n"));
@@ -248,12 +251,12 @@ get_usage(zpool_help_t idx)
 		    "[-R root] [-F [-n]] [-t]\n"
 		    "\t    [--rewind-to-checkpoint]  [newpool]\n"));
 	case HELP_IOSTAT:
-		return (gettext("\tiostat [-v] [-T d|u] [pool] ... [interval "
-		    "[count]]\n"));
+		return (gettext("\tiostat [-gLPv] [-T d|u] [pool] ... "
+		    "[interval [count]]\n"));
 	case HELP_LABELCLEAR:
 		return (gettext("\tlabelclear [-f] \n"));
 	case HELP_LIST:
-		return (gettext("\tlist [-Hp] [-o property[,...]] "
+		return (gettext("\tlist [-gHLpPv] [-o property[,...]] "
 		    "[-T d|u] [pool] ... [interval [count]]\n"));
 	case HELP_OFFLINE:
 		return (gettext("\toffline [-t]   ...\n"));
@@ -271,8 +274,8 @@ get_usage(zpool_help_t idx)
 	case HELP_SCRUB:
 		return (gettext("\tscrub [-s | -p]  ...\n"));
 	case HELP_STATUS:
-		return (gettext("\tstatus [-vx] [-T d|u] [pool] ... [interval "
-		    "[count]]\n"));
+		return (gettext("\tstatus [-DgLPvx] [-T d|u] [pool] ... "
+		    "[interval [count]]\n"));
 	case HELP_UPGRADE:
 		return (gettext("\tupgrade\n"
 		    "\tupgrade -v\n"
@@ -283,7 +286,7 @@ get_usage(zpool_help_t idx)
 	case HELP_SET:
 		return (gettext("\tset   \n"));
 	case HELP_SPLIT:
-		return (gettext("\tsplit [-n] [-R altroot] [-o mntopts]\n"
+		return (gettext("\tsplit [-gLnP] [-R altroot] [-o mntopts]\n"
 		    "\t    [-o property=value]   "
 		    "[ ...]\n"));
 	case HELP_REGUID:
@@ -305,7 +308,7 @@ print_prop_cb(int prop, void *cb)
 {
 	FILE *fp = cb;
 
-	(void) fprintf(fp, "\t%-15s  ", zpool_prop_to_name(prop));
+	(void) fprintf(fp, "\t%-19s  ", zpool_prop_to_name(prop));
 
 	if (zpool_prop_readonly(prop))
 		(void) fprintf(fp, "  NO   ");
@@ -357,14 +360,14 @@ usage(boolean_t requested)
 		(void) fprintf(fp,
 		    gettext("\nthe following properties are supported:\n"));
 
-		(void) fprintf(fp, "\n\t%-15s  %s   %s\n\n",
+		(void) fprintf(fp, "\n\t%-19s  %s   %s\n\n",
 		    "PROPERTY", "EDIT", "VALUES");
 
 		/* Iterate over all properties */
 		(void) zprop_iter(print_prop_cb, fp, B_FALSE, B_TRUE,
 		    ZFS_TYPE_POOL);
 
-		(void) fprintf(fp, "\t%-15s   ", "feature@...");
+		(void) fprintf(fp, "\t%-19s   ", "feature@...");
 		(void) fprintf(fp, "YES   disabled | enabled | active\n");
 
 		(void) fprintf(fp, gettext("\nThe feature@ properties must be "
@@ -382,32 +385,45 @@ usage(boolean_t requested)
 	exit(requested ? 0 : 2);
 }
 
-void
+/*
+ * print a pool vdev config for dry runs
+ */
+static void
 print_vdev_tree(zpool_handle_t *zhp, const char *name, nvlist_t *nv, int indent,
-    boolean_t print_logs)
+    const char *match, int name_flags)
 {
 	nvlist_t **child;
 	uint_t c, children;
 	char *vname;
+	boolean_t printed = B_FALSE;
 
-	if (name != NULL)
-		(void) printf("\t%*s%s\n", indent, "", name);
-
 	if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN,
-	    &child, &children) != 0)
+	    &child, &children) != 0) {
+		if (name != NULL)
+			(void) printf("\t%*s%s\n", indent, "", name);
 		return;
+	}
 
 	for (c = 0; c < children; c++) {
 		uint64_t is_log = B_FALSE;
+		char *class = "";
 
 		(void) nvlist_lookup_uint64(child[c], ZPOOL_CONFIG_IS_LOG,
 		    &is_log);
-		if ((is_log && !print_logs) || (!is_log && print_logs))
+		if (is_log)
+			class = VDEV_ALLOC_BIAS_LOG;
+		(void) nvlist_lookup_string(child[c],
+		    ZPOOL_CONFIG_ALLOCATION_BIAS, &class);
+		if (strcmp(match, class) != 0)
 			continue;
 
-		vname = zpool_vdev_name(g_zfs, zhp, child[c], B_FALSE);
-		print_vdev_tree(zhp, vname, child[c], indent + 2,
-		    B_FALSE);
+		if (!printed && name != NULL) {
+			(void) printf("\t%*s%s\n", indent, "", name);
+			printed = B_TRUE;
+		}
+		vname = zpool_vdev_name(g_zfs, zhp, child[c], name_flags);
+		print_vdev_tree(zhp, vname, child[c], indent + 2, "",
+		    name_flags);
 		free(vname);
 	}
 }
@@ -515,11 +531,14 @@ add_prop_list_default(const char *propname, char *prop
 }
 
 /*
- * zpool add [-fn]   ...
+ * zpool add [-fgLnP] [-o property=value]   ...
  *
  *	-f	Force addition of devices, even if they appear in use
+ *	-g	Display guid for individual vdev name.
+ *	-L	Follow links when resolving vdev path name.
  *	-n	Do not add the devices, but display the resulting layout if
  *		they were to be added.
+ *	-P	Display full path for vdev name.
  *
  * Adds the given vdevs to 'pool'.  As with create, the bulk of this work is
  * handled by get_vdev_spec(), which constructs the nvlist needed to pass to
@@ -530,6 +549,7 @@ zpool_do_add(int argc, char **argv)
 {
 	boolean_t force = B_FALSE;
 	boolean_t dryrun = B_FALSE;
+	int name_flags = 0;
 	int c;
 	nvlist_t *nvroot;
 	char *poolname;
@@ -540,14 +560,23 @@ zpool_do_add(int argc, char **argv)
 	nvlist_t *config;
 
 	/* check options */
-	while ((c = getopt(argc, argv, "fn")) != -1) {
+	while ((c = getopt(argc, argv, "fgLnP")) != -1) {
 		switch (c) {
 		case 'f':
 			force = B_TRUE;
 			break;
+		case 'g':
+			name_flags |= VDEV_NAME_GUID;
+			break;
+		case 'L':
+			name_flags |= VDEV_NAME_FOLLOW_LINKS;
+			break;
 		case 'n':
 			dryrun = B_TRUE;
 			break;
+		case 'P':
+			name_flags |= VDEV_NAME_PATH;
+			break;
 		case '?':
 			(void) fprintf(stderr, gettext("invalid option '%c'\n"),
 			    optopt);
@@ -607,17 +636,26 @@ zpool_do_add(int argc, char **argv)
 		    "configuration:\n"), zpool_get_name(zhp));
 
 		/* print original main pool and new tree */
-		print_vdev_tree(zhp, poolname, poolnvroot, 0, B_FALSE);
-		print_vdev_tree(zhp, NULL, nvroot, 0, B_FALSE);
+		print_vdev_tree(zhp, poolname, poolnvroot, 0, "",
+		    name_flags | VDEV_NAME_TYPE_ID);
+		print_vdev_tree(zhp, NULL, nvroot, 0, "", name_flags);
 
-		/* Do the same for the logs */
-		if (num_logs(poolnvroot) > 0) {
-			print_vdev_tree(zhp, "logs", poolnvroot, 0, B_TRUE);
-			print_vdev_tree(zhp, NULL, nvroot, 0, B_TRUE);
-		} else if (num_logs(nvroot) > 0) {
-			print_vdev_tree(zhp, "logs", nvroot, 0, B_TRUE);
-		}
+		/* print other classes: 'dedup', 'special', and 'log' */
+		print_vdev_tree(zhp, "dedup", poolnvroot, 0,
+		    VDEV_ALLOC_BIAS_DEDUP, name_flags);
+		print_vdev_tree(zhp, NULL, nvroot, 0, VDEV_ALLOC_BIAS_DEDUP,
+		    name_flags);
 
+		print_vdev_tree(zhp, "special", poolnvroot, 0,
+		    VDEV_ALLOC_BIAS_SPECIAL, name_flags);
+		print_vdev_tree(zhp, NULL, nvroot, 0, VDEV_ALLOC_BIAS_SPECIAL,
+		    name_flags);
+
+		print_vdev_tree(zhp, "logs", poolnvroot, 0, VDEV_ALLOC_BIAS_LOG,
+		    name_flags);
+		print_vdev_tree(zhp, NULL, nvroot, 0, VDEV_ALLOC_BIAS_LOG,
+		    name_flags);
+
 		ret = 0;
 	} else {
 		ret = (zpool_add(zhp, nvroot) != 0);
@@ -1203,9 +1241,13 @@ zpool_do_create(int argc, char **argv)
 		(void) printf(gettext("would create '%s' with the "
 		    "following layout:\n\n"), poolname);
 
-		print_vdev_tree(NULL, poolname, nvroot, 0, B_FALSE);
-		if (num_logs(nvroot) > 0)
-			print_vdev_tree(NULL, "logs", nvroot, 0, B_TRUE);
+		print_vdev_tree(NULL, poolname, nvroot, 0, "", 0);
+		print_vdev_tree(NULL, "dedup", nvroot, 0,
+		    VDEV_ALLOC_BIAS_DEDUP, 0);
+		print_vdev_tree(NULL, "special", nvroot, 0,
+		    VDEV_ALLOC_BIAS_SPECIAL, 0);
+		print_vdev_tree(NULL, "logs", nvroot, 0,
+		    VDEV_ALLOC_BIAS_LOG, 0);
 
 		ret = 0;
 	} else {
@@ -1412,13 +1454,15 @@ zpool_do_export(int argc, char **argv)
  * name column.
  */
 static int
-max_width(zpool_handle_t *zhp, nvlist_t *nv, int depth, int max)
+max_width(zpool_handle_t *zhp, nvlist_t *nv, int depth, int max,
+    int name_flags)
 {
-	char *name = zpool_vdev_name(g_zfs, zhp, nv, B_TRUE);
+	char *name;
 	nvlist_t **child;
 	uint_t c, children;
 	int ret;
 
+	name = zpool_vdev_name(g_zfs, zhp, nv, name_flags | VDEV_NAME_TYPE_ID);
 	if (strlen(name) + depth > max)
 		max = strlen(name) + depth;
 
@@ -1428,7 +1472,7 @@ max_width(zpool_handle_t *zhp, nvlist_t *nv, int depth
 	    &child, &children) == 0) {
 		for (c = 0; c < children; c++)
 			if ((ret = max_width(zhp, child[c], depth + 2,
-			    max)) > max)
+			    max, name_flags)) > max)
 				max = ret;
 	}
 
@@ -1436,7 +1480,7 @@ max_width(zpool_handle_t *zhp, nvlist_t *nv, int depth
 	    &child, &children) == 0) {
 		for (c = 0; c < children; c++)
 			if ((ret = max_width(zhp, child[c], depth + 2,
-			    max)) > max)
+			    max, name_flags)) > max)
 				max = ret;
 	}
 
@@ -1444,11 +1488,10 @@ max_width(zpool_handle_t *zhp, nvlist_t *nv, int depth
 	    &child, &children) == 0) {
 		for (c = 0; c < children; c++)
 			if ((ret = max_width(zhp, child[c], depth + 2,
-			    max)) > max)
+			    max, name_flags)) > max)
 				max = ret;
 	}
 
-
 	return (max);
 }
 
@@ -1497,12 +1540,24 @@ find_spare(zpool_handle_t *zhp, void *data)
 	return (0);
 }
 
+typedef struct status_cbdata {
+	int		cb_count;
+	int		cb_name_flags;
+	int		cb_namewidth;
+	boolean_t	cb_allpools;
+	boolean_t	cb_verbose;
+	boolean_t	cb_explain;
+	boolean_t	cb_first;
+	boolean_t	cb_dedup_stats;
+	boolean_t	cb_print_status;
+} status_cbdata_t;
+
 /*
  * Print out configuration state as requested by status_callback.
  */
-void
-print_status_config(zpool_handle_t *zhp, const char *name, nvlist_t *nv,
-    int namewidth, int depth, boolean_t isspare)
+static void
+print_status_config(zpool_handle_t *zhp, status_cbdata_t *cb, const char *name,
+    nvlist_t *nv, int depth, boolean_t isspare)
 {
 	nvlist_t **child;
 	uint_t c, children;
@@ -1511,7 +1566,7 @@ print_status_config(zpool_handle_t *zhp, const char *n
 	char rbuf[6], wbuf[6], cbuf[6];
 	char *vname;
 	uint64_t notpresent;
-	spare_cbdata_t cb;
+	spare_cbdata_t spare_cb;
 	const char *state;
 	char *type;
 
@@ -1539,7 +1594,7 @@ print_status_config(zpool_handle_t *zhp, const char *n
 			state = "AVAIL";
 	}
 
-	(void) printf("\t%*s%-*s  %-8s", depth, "", namewidth - depth,
+	(void) printf("\t%*s%-*s  %-8s", depth, "", cb->cb_namewidth - depth,
 	    name, state);
 
 	if (!isspare) {
@@ -1580,17 +1635,17 @@ print_status_config(zpool_handle_t *zhp, const char *n
 
 		case VDEV_AUX_SPARED:
 			verify(nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID,
-			    &cb.cb_guid) == 0);
-			if (zpool_iter(g_zfs, find_spare, &cb) == 1) {
-				if (strcmp(zpool_get_name(cb.cb_zhp),
+			    &spare_cb.cb_guid) == 0);
+			if (zpool_iter(g_zfs, find_spare, &spare_cb) == 1) {
+				if (strcmp(zpool_get_name(spare_cb.cb_zhp),
 				    zpool_get_name(zhp)) == 0)
 					(void) printf(gettext("currently in "
 					    "use"));
 				else
 					(void) printf(gettext("in use by "
 					    "pool '%s'"),
-					    zpool_get_name(cb.cb_zhp));
-				zpool_close(cb.cb_zhp);
+					    zpool_get_name(spare_cb.cb_zhp));
+				zpool_close(spare_cb.cb_zhp);
 			} else {
 				(void) printf(gettext("currently in use"));
 			}
@@ -1689,20 +1744,25 @@ print_status_config(zpool_handle_t *zhp, const char *n
 		    &ishole);
 		if (islog || ishole)
 			continue;
-		vname = zpool_vdev_name(g_zfs, zhp, child[c], B_TRUE);
-		print_status_config(zhp, vname, child[c],
-		    namewidth, depth + 2, isspare);
+		/* Only print normal classes here */
+		if (nvlist_exists(child[c], ZPOOL_CONFIG_ALLOCATION_BIAS))
+			continue;
+
+		vname = zpool_vdev_name(g_zfs, zhp, child[c],
+		    cb->cb_name_flags | VDEV_NAME_TYPE_ID);
+		print_status_config(zhp, cb, vname, child[c], depth + 2,
+		    isspare);
 		free(vname);
 	}
 }
 
-
 /*
  * Print the configuration of an exported pool.  Iterate over all vdevs in the
  * pool, printing out the name and status for each one.
  */
-void
-print_import_config(const char *name, nvlist_t *nv, int namewidth, int depth)
+static void
+print_import_config(status_cbdata_t *cb, const char *name, nvlist_t *nv,
+    int depth)
 {
 	nvlist_t **child;
 	uint_t c, children;
@@ -1717,7 +1777,7 @@ print_import_config(const char *name, nvlist_t *nv, in
 	verify(nvlist_lookup_uint64_array(nv, ZPOOL_CONFIG_VDEV_STATS,
 	    (uint64_t **)&vs, &c) == 0);
 
-	(void) printf("\t%*s%-*s", depth, "", namewidth - depth, name);
+	(void) printf("\t%*s%-*s", depth, "", cb->cb_namewidth - depth, name);
 	(void) printf("  %s", zpool_state_to_name(vs->vs_state, vs->vs_aux));
 
 	if (vs->vs_aux != 0) {
@@ -1774,9 +1834,12 @@ print_import_config(const char *name, nvlist_t *nv, in
 		    &is_log);
 		if (is_log)
 			continue;
+		if (nvlist_exists(child[c], ZPOOL_CONFIG_ALLOCATION_BIAS))
+			continue;
 
-		vname = zpool_vdev_name(g_zfs, NULL, child[c], B_TRUE);
-		print_import_config(vname, child[c], namewidth, depth + 2);
+		vname = zpool_vdev_name(g_zfs, NULL, child[c],
+		    cb->cb_name_flags | VDEV_NAME_TYPE_ID);
+		print_import_config(cb, vname, child[c], depth + 2);
 		free(vname);
 	}
 
@@ -1784,7 +1847,8 @@ print_import_config(const char *name, nvlist_t *nv, in
 	    &child, &children) == 0) {
 		(void) printf(gettext("\tcache\n"));
 		for (c = 0; c < children; c++) {
-			vname = zpool_vdev_name(g_zfs, NULL, child[c], B_FALSE);
+			vname = zpool_vdev_name(g_zfs, NULL, child[c],
+			    cb->cb_name_flags);
 			(void) printf("\t  %s\n", vname);
 			free(vname);
 		}
@@ -1794,7 +1858,8 @@ print_import_config(const char *name, nvlist_t *nv, in
 	    &child, &children) == 0) {
 		(void) printf(gettext("\tspares\n"));
 		for (c = 0; c < children; c++) {
-			vname = zpool_vdev_name(g_zfs, NULL, child[c], B_FALSE);
+			vname = zpool_vdev_name(g_zfs, NULL, child[c],
+			    cb->cb_name_flags);
 			(void) printf("\t  %s\n", vname);
 			free(vname);
 		}
@@ -1802,39 +1867,62 @@ print_import_config(const char *name, nvlist_t *nv, in
 }
 
 /*
- * Print log vdevs.
- * Logs are recorded as top level vdevs in the main pool child array
- * but with "is_log" set to 1. We use either print_status_config() or
- * print_import_config() to print the top level logs then any log
- * children (eg mirrored slogs) are printed recursively - which
- * works because only the top level vdev is marked "is_log"
+ * Print specialized class vdevs.
+ *
+ * These are recorded as top level vdevs in the main pool child array
+ * but with "is_log" set to 1 or an "alloc_bias" string. We use either
+ * print_status_config() or print_import_config() to print the top level
+ * class vdevs then any of their children (eg mirrored slogs) are printed
+ * recursively - which works because only the top level vdev is marked.
  */
 static void
-print_logs(zpool_handle_t *zhp, nvlist_t *nv, int namewidth, boolean_t verbose)
+print_class_vdevs(zpool_handle_t *zhp, status_cbdata_t *cb, nvlist_t *nv,
+    const char *class)
 {
 	uint_t c, children;
 	nvlist_t **child;
+	boolean_t printed = B_FALSE;
 
+	assert(zhp != NULL || !cb->cb_verbose);
+
 	if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN, &child,
 	    &children) != 0)
 		return;
 
-	(void) printf(gettext("\tlogs\n"));
-
 	for (c = 0; c < children; c++) {
 		uint64_t is_log = B_FALSE;
-		char *name;
+		char *bias = NULL;
+		char *type = NULL;
 
 		(void) nvlist_lookup_uint64(child[c], ZPOOL_CONFIG_IS_LOG,
 		    &is_log);
-		if (!is_log)
+
+		if (is_log) {
+			bias = VDEV_ALLOC_CLASS_LOGS;
+		} else {
+			(void) nvlist_lookup_string(child[c],
+			    ZPOOL_CONFIG_ALLOCATION_BIAS, &bias);
+			(void) nvlist_lookup_string(child[c],
+			    ZPOOL_CONFIG_TYPE, &type);
+		}
+
+		if (bias == NULL || strcmp(bias, class) != 0)
 			continue;
-		name = zpool_vdev_name(g_zfs, zhp, child[c], B_TRUE);
-		if (verbose)
-			print_status_config(zhp, name, child[c], namewidth,
-			    2, B_FALSE);
+		if (!is_log && strcmp(type, VDEV_TYPE_INDIRECT) == 0)
+			continue;
+
+		if (!printed) {
+			(void) printf("\t%s\t\n", gettext(class));
+			printed = B_TRUE;
+		}
+
+		char *name = zpool_vdev_name(g_zfs, zhp, child[c],
+		    cb->cb_name_flags | VDEV_NAME_TYPE_ID);
+		if (cb->cb_print_status)
+			print_status_config(zhp, cb, name, child[c], 2,
+			    B_FALSE);
 		else
-			print_import_config(name, child[c], namewidth, 2);
+			print_import_config(cb, name, child[c], 2);
 		free(name);
 	}
 }
@@ -1856,8 +1944,8 @@ show_import(nvlist_t *config)
 	int reason;
 	const char *health;
 	uint_t vsc;
-	int namewidth;
 	char *comment;
+	status_cbdata_t cb = { 0 };
 
 	verify(nvlist_lookup_string(config, ZPOOL_CONFIG_POOL_NAME,
 	    &name) == 0);
@@ -2083,14 +2171,16 @@ show_import(nvlist_t *config)
 
 	(void) printf(gettext(" config:\n\n"));
 
-	namewidth = max_width(NULL, nvroot, 0, 0);
-	if (namewidth < 10)
-		namewidth = 10;
+	cb.cb_namewidth = max_width(NULL, nvroot, 0, 0, 0);
+	if (cb.cb_namewidth < 10)
+		cb.cb_namewidth = 10;
 
-	print_import_config(name, nvroot, namewidth, 0);
-	if (num_logs(nvroot) > 0)
-		print_logs(NULL, nvroot, namewidth, B_FALSE);
+	print_import_config(&cb, name, nvroot, 0);
 
+	print_class_vdevs(NULL, &cb, nvroot, VDEV_ALLOC_BIAS_DEDUP);
+	print_class_vdevs(NULL, &cb, nvroot, VDEV_ALLOC_BIAS_SPECIAL);
+	print_class_vdevs(NULL, &cb, nvroot, VDEV_ALLOC_CLASS_LOGS);
+
 	if (reason == ZPOOL_STATUS_BAD_GUID_SUM) {
 		(void) printf(gettext("\n\tAdditional devices are known to "
 		    "be part of this pool, though their\n\texact "
@@ -2751,8 +2841,10 @@ zpool_do_sync(int argc, char **argv)
 
 typedef struct iostat_cbdata {
 	boolean_t cb_verbose;
+	int cb_name_flags;
 	int cb_namewidth;
 	int cb_iteration;
+	boolean_t cb_scripted;
 	zpool_list_t *cb_list;
 } iostat_cbdata_t;
 
@@ -2788,12 +2880,20 @@ print_one_stat(uint64_t value)
 	(void) printf("  %5s", buf);
 }
 
+static const char *class_name[] = {
+	VDEV_ALLOC_BIAS_DEDUP,
+	VDEV_ALLOC_BIAS_SPECIAL,
+	VDEV_ALLOC_CLASS_LOGS
+};
+
 /*
  * Print out all the statistics for the given vdev.  This can either be the
  * toplevel configuration, or called recursively.  If 'name' is NULL, then this
  * is a verbose output, and we don't want to display the toplevel pool stats.
+ *
+ * Returns the number of stat lines printed.
  */
-void
+static unsigned int
 print_vdev_stats(zpool_handle_t *zhp, const char *name, nvlist_t *oldnv,
     nvlist_t *newnv, iostat_cbdata_t *cb, int depth)
 {
@@ -2801,12 +2901,13 @@ print_vdev_stats(zpool_handle_t *zhp, const char *name
 	uint_t c, children;
 	vdev_stat_t *oldvs, *newvs;
 	vdev_stat_t zerovs = { 0 };
+	char *vname;
+	int ret = 0;
 	uint64_t tdelta;
 	double scale;
-	char *vname;
 
 	if (strcmp(name, VDEV_TYPE_INDIRECT) == 0)
-		return;
+		return (ret);
 
 	if (oldnv != NULL) {
 		verify(nvlist_lookup_uint64_array(oldnv,
@@ -2854,16 +2955,19 @@ print_vdev_stats(zpool_handle_t *zhp, const char *name
 	(void) printf("\n");
 
 	if (!cb->cb_verbose)
-		return;
+		return (ret);
 
 	if (nvlist_lookup_nvlist_array(newnv, ZPOOL_CONFIG_CHILDREN,
 	    &newchild, &children) != 0)
-		return;
+		return (ret);
 
 	if (oldnv && nvlist_lookup_nvlist_array(oldnv, ZPOOL_CONFIG_CHILDREN,
 	    &oldchild, &c) != 0)
-		return;
+		return (ret);
 
+	/*
+	 * print normal top-level devices
+	 */
 	for (c = 0; c < children; c++) {
 		uint64_t ishole = B_FALSE, islog = B_FALSE;
 
@@ -2876,33 +2980,45 @@ print_vdev_stats(zpool_handle_t *zhp, const char *name
 		if (ishole || islog)
 			continue;
 
-		vname = zpool_vdev_name(g_zfs, zhp, newchild[c], B_FALSE);
+		if (nvlist_exists(newchild[c], ZPOOL_CONFIG_ALLOCATION_BIAS))
+			continue;
+
+		vname = zpool_vdev_name(g_zfs, zhp, newchild[c],
+		    cb->cb_name_flags);
 		print_vdev_stats(zhp, vname, oldnv ? oldchild[c] : NULL,
 		    newchild[c], cb, depth + 2);
 		free(vname);
 	}
 
 	/*
-	 * Log device section
+	 * print all other top-level devices
 	 */
-
-	if (num_logs(newnv) > 0) {
-		(void) printf("%-*s      -      -      -      -      -      "
-		    "-\n", cb->cb_namewidth, "logs");
-
+	for (uint_t n = 0; n < 3; n++) {
 		for (c = 0; c < children; c++) {
 			uint64_t islog = B_FALSE;
+			char *bias = NULL;
+			char *type = NULL;
+
 			(void) nvlist_lookup_uint64(newchild[c],
 			    ZPOOL_CONFIG_IS_LOG, &islog);
-
 			if (islog) {
-				vname = zpool_vdev_name(g_zfs, zhp, newchild[c],
-				    B_FALSE);
-				print_vdev_stats(zhp, vname, oldnv ?
-				    oldchild[c] : NULL, newchild[c],
-				    cb, depth + 2);
-				free(vname);
+				bias = VDEV_ALLOC_CLASS_LOGS;
+			} else {
+				(void) nvlist_lookup_string(newchild[c],
+				    ZPOOL_CONFIG_ALLOCATION_BIAS, &bias);
+				(void) nvlist_lookup_string(newchild[c],
+				    ZPOOL_CONFIG_TYPE, &type);
 			}
+			if (bias == NULL || strcmp(bias, class_name[n]) != 0)
+				continue;
+			if (!islog && strcmp(type, VDEV_TYPE_INDIRECT) == 0)
+				continue;
+
+			vname = zpool_vdev_name(g_zfs, zhp, newchild[c],
+			    cb->cb_name_flags);
+			ret += print_vdev_stats(zhp, vname, oldnv ?
+			    oldchild[c] : NULL, newchild[c], cb, depth + 2);
+			free(vname);
 		}
 
 	}
@@ -2912,23 +3028,25 @@ print_vdev_stats(zpool_handle_t *zhp, const char *name
 	 */
 	if (nvlist_lookup_nvlist_array(newnv, ZPOOL_CONFIG_L2CACHE,
 	    &newchild, &children) != 0)
-		return;
+		return (ret);
 
 	if (oldnv && nvlist_lookup_nvlist_array(oldnv, ZPOOL_CONFIG_L2CACHE,
 	    &oldchild, &c) != 0)
-		return;
+		return (ret);
 
 	if (children > 0) {
 		(void) printf("%-*s      -      -      -      -      -      "
 		    "-\n", cb->cb_namewidth, "cache");
 		for (c = 0; c < children; c++) {
 			vname = zpool_vdev_name(g_zfs, zhp, newchild[c],
-			    B_FALSE);
+			    cb->cb_name_flags);
 			print_vdev_stats(zhp, vname, oldnv ? oldchild[c] : NULL,
 			    newchild[c], cb, depth + 2);
 			free(vname);
 		}
 	}
+
+	return (ret);
 }
 
 static int
@@ -2997,7 +3115,7 @@ get_namewidth(zpool_handle_t *zhp, void *data)
 			cb->cb_namewidth = strlen(zpool_get_name(zhp));
 		else
 			cb->cb_namewidth = max_width(zhp, nvroot, 0,
-			    cb->cb_namewidth);
+			    cb->cb_namewidth, cb->cb_name_flags);
 	}
 
 	/*
@@ -3095,8 +3213,11 @@ get_timestamp_arg(char c)
 }
 
 /*
- * zpool iostat [-v] [-T d|u] [pool] ... [interval [count]]
+ * zpool iostat [-gLPv] [-T d|u] [pool] ... [interval [count]]
  *
+ *	-g	Display guid for individual vdev name.
+ *	-L	Follow links when resolving vdev path name.
+ *	-P	Display full path for vdev name.
  *	-v	Display statistics for individual vdevs
  *	-T	Display a timestamp in date(1) or Unix format
  *
@@ -3115,11 +3236,23 @@ zpool_do_iostat(int argc, char **argv)
 	unsigned long interval = 0, count = 0;
 	zpool_list_t *list;
 	boolean_t verbose = B_FALSE;

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 08:58:07 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9DD0E1AEE9C;
 Wed,  6 Nov 2019 08:58:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L674PxGz3KF6;
 Wed,  6 Nov 2019 08:58:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7A596229D1;
 Wed,  6 Nov 2019 08:58:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA68w74a047161;
 Wed, 6 Nov 2019 08:58:07 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA68w3xb047144;
 Wed, 6 Nov 2019 08:58:03 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060858.xA68w3xb047144@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 08:58:03 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354382 - vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill...
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
 vendor/...
X-SVN-Commit-Revision: 354382
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 08:58:07 -0000

Author: avg
Date: Wed Nov  6 08:58:03 2019
New Revision: 354382
URL: https://svnweb.freebsd.org/changeset/base/354382

Log:
  10601 10757 Pool allocation classes
  
  illumos/illumos-gate@663207adb1669640c01c5ec6949ce78fd806efae
  https://github.com/illumos/illumos-gate/commit/663207adb1669640c01c5ec6949ce78fd806efae
  
  10601 Pool allocation classes
  https://www.illumos.org/issues/10601
    illumos port of ZoL Pool allocation classes. Includes at least these two
    commits:
    441709695 Pool allocation classes misplacing small file blocks
    cc99f275a Pool allocation classes
  
  10757 Add -gLp to zpool subcommands for alt vdev names
  https://www.illumos.org/issues/10757
    Port from ZoL of
    d2f3e292d Add -gLp to zpool subcommands for alt vdev names
    Note that a subsequent ZoL commit changed -p to -P
    a77f29f93 Change full path subcommand flag from -p to -P
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Håkan Johansson 
  Portions contributed by: Richard Yao 
  Portions contributed by: Chunwei Chen 
  Portions contributed by: loli10K 
  Author: Don Brady 

Modified:
  vendor-sys/illumos/dist/common/zfs/zfeature_common.c
  vendor-sys/illumos/dist/common/zfs/zfeature_common.h
  vendor-sys/illumos/dist/common/zfs/zfs_prop.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Changes in other areas also in this revision:
Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/cmd/zpool/zpool_main.c
  vendor/illumos/dist/cmd/zpool/zpool_vdev.c
  vendor/illumos/dist/cmd/ztest/ztest.c
  vendor/illumos/dist/lib/libzfs/common/libzfs.h
  vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c
  vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c
  vendor/illumos/dist/lib/libzpool/common/util.c
  vendor/illumos/dist/man/man1m/zfs.1m
  vendor/illumos/dist/man/man1m/zpool.1m
  vendor/illumos/dist/man/man5/zpool-features.5

Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c
==============================================================================
--- vendor-sys/illumos/dist/common/zfs/zfeature_common.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -20,11 +20,12 @@
  */
 
 /*
- * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  * Copyright (c) 2013, Joyent, Inc. All rights reserved.
  * Copyright (c) 2014, Nexenta Systems, Inc. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifdef _KERNEL
@@ -298,4 +299,11 @@ zpool_feature_init(void)
 	    "Reduce memory used by removed devices when their blocks are "
 	    "freed or remapped.",
 	    ZFEATURE_FLAG_READONLY_COMPAT, obsolete_counts_deps);
+
+	{
+	zfeature_register(SPA_FEATURE_ALLOCATION_CLASSES,
+	    "org.zfsonlinux:allocation_classes", "allocation_classes",
+	    "Support for separate allocation classes.",
+	    ZFEATURE_FLAG_READONLY_COMPAT, NULL);
+	}
 }

Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.h
==============================================================================
--- vendor-sys/illumos/dist/common/zfs/zfeature_common.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/common/zfs/zfeature_common.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -24,6 +24,7 @@
  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  * Copyright (c) 2013, Joyent, Inc. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _ZFEATURE_COMMON_H
@@ -61,6 +62,7 @@ typedef enum spa_feature {
 	SPA_FEATURE_OBSOLETE_COUNTS,
 	SPA_FEATURE_POOL_CHECKPOINT,
 	SPA_FEATURE_SPACEMAP_V2,
+	SPA_FEATURE_ALLOCATION_CLASSES,
 	SPA_FEATURES
 } spa_feature_t;
 

Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c
==============================================================================
--- vendor-sys/illumos/dist/common/zfs/zfs_prop.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/common/zfs/zfs_prop.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -435,6 +435,9 @@ zfs_prop_init(void)
 	zprop_register_number(ZFS_PROP_RECORDSIZE, "recordsize",
 	    SPA_OLD_MAXBLOCKSIZE, PROP_INHERIT,
 	    ZFS_TYPE_FILESYSTEM, "512 to 1M, power of 2", "RECSIZE");
+	zprop_register_number(ZFS_PROP_SPECIAL_SMALL_BLOCKS,
+	    "special_small_blocks", 0, PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
+	    "zero or 512 to 128K, power of 2", "SPECIAL_SMALL_BLOCKS");
 
 	/* hidden properties */
 	zprop_register_hidden(ZFS_PROP_REMAPTXG, "remaptxg", PROP_TYPE_NUMBER,

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -2201,6 +2201,8 @@ dmu_write_policy(objset_t *os, dnode_t *dn, int level,
 	zp->zp_dedup = dedup;
 	zp->zp_dedup_verify = dedup && dedup_verify;
 	zp->zp_nopwrite = nopwrite;
+	zp->zp_zpl_smallblk = DMU_OT_IS_FILE(zp->zp_type) ?
+	    os->os_zpl_special_smallblock : 0;
 }
 
 int

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -304,6 +304,20 @@ dnodesize_changed_cb(void *arg, uint64_t newval)
 }
 
 static void
+smallblk_changed_cb(void *arg, uint64_t newval)
+{
+	objset_t *os = arg;
+
+	/*
+	 * Inheritance and range checking should have been done by now.
+	 */
+	ASSERT(newval <= SPA_OLD_MAXBLOCKSIZE);
+	ASSERT(ISP2(newval));
+
+	os->os_zpl_special_smallblock = newval;
+}
+
+static void
 logbias_changed_cb(void *arg, uint64_t newval)
 {
 	objset_t *os = arg;
@@ -515,6 +529,12 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl
 				err = dsl_prop_register(ds,
 				    zfs_prop_to_name(ZFS_PROP_DNODESIZE),
 				    dnodesize_changed_cb, os);
+			}
+			if (err == 0) {
+				err = dsl_prop_register(ds,
+				    zfs_prop_to_name(
+				    ZFS_PROP_SPECIAL_SMALL_BLOCKS),
+				    smallblk_changed_cb, os);
 			}
 		}
 		if (needlock)

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -23,6 +23,7 @@
  * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #include 
@@ -288,7 +289,7 @@ metaslab_class_validate(metaslab_class_t *mc)
 	return (0);
 }
 
-void
+static void
 metaslab_class_space_update(metaslab_class_t *mc, int64_t alloc_delta,
     int64_t defer_delta, int64_t space_delta, int64_t dspace_delta)
 {
@@ -325,7 +326,8 @@ metaslab_class_get_dspace(metaslab_class_t *mc)
 void
 metaslab_class_histogram_verify(metaslab_class_t *mc)
 {
-	vdev_t *rvd = mc->mc_spa->spa_root_vdev;
+	spa_t *spa = mc->mc_spa;
+	vdev_t *rvd = spa->spa_root_vdev;
 	uint64_t *mc_hist;
 	int i;
 
@@ -831,7 +833,8 @@ metaslab_group_histogram_verify(metaslab_group_t *mg)
 	for (int m = 0; m < vd->vdev_ms_count; m++) {
 		metaslab_t *msp = vd->vdev_ms[m];
 
-		if (msp->ms_sm == NULL)
+		/* skip if not active or not a member */
+		if (msp->ms_sm == NULL || msp->ms_group != mg)
 			continue;
 
 		for (i = 0; i < SPACE_MAP_HISTOGRAM_SIZE; i++)
@@ -964,12 +967,14 @@ metaslab_group_fragmentation(metaslab_group_t *mg)
 
 		if (msp->ms_fragmentation == ZFS_FRAG_INVALID)
 			continue;
+		if (msp->ms_group != mg)
+			continue;
 
 		valid_ms++;
 		fragmentation += msp->ms_fragmentation;
 	}
 
-	if (valid_ms <= vd->vdev_ms_count / 2)
+	if (valid_ms <= mg->mg_vd->vdev_ms_count / 2)
 		return (ZFS_FRAG_INVALID);
 
 	fragmentation /= valid_ms;
@@ -1000,7 +1005,10 @@ metaslab_group_allocatable(metaslab_group_t *mg, metas
 	 * groups to select from. Otherwise, we always consider it eligible
 	 * for allocations.
 	 */
-	if (mc != spa_normal_class(spa) || mc->mc_groups <= 1)
+	if ((mc != spa_normal_class(spa) &&
+	    mc != spa_special_class(spa) &&
+	    mc != spa_dedup_class(spa)) ||
+	    mc->mc_groups <= 1)
 		return (B_TRUE);
 
 	/*
@@ -1534,12 +1542,26 @@ metaslab_unload(metaslab_t *msp)
 	msp->ms_max_size = 0;
 }
 
+static void
+metaslab_space_update(vdev_t *vd, metaslab_class_t *mc, int64_t alloc_delta,
+    int64_t defer_delta, int64_t space_delta)
+{
+	vdev_space_update(vd, alloc_delta, defer_delta, space_delta);
+
+	ASSERT3P(vd->vdev_spa->spa_root_vdev, ==, vd->vdev_parent);
+	ASSERT(vd->vdev_ms_count != 0);
+
+	metaslab_class_space_update(mc, alloc_delta, defer_delta, space_delta,
+	    vdev_deflated_space(vd, space_delta));
+}
+
 int
 metaslab_init(metaslab_group_t *mg, uint64_t id, uint64_t object, uint64_t txg,
     metaslab_t **msp)
 {
 	vdev_t *vd = mg->mg_vd;
-	objset_t *mos = vd->vdev_spa->spa_meta_objset;
+	spa_t *spa = vd->vdev_spa;
+	objset_t *mos = spa->spa_meta_objset;
 	metaslab_t *ms;
 	int error;
 
@@ -1596,8 +1618,8 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6
 
 	/*
 	 * If metaslab_debug_load is set and we're initializing a metaslab
-	 * that has an allocated space map object then load the its space
-	 * map so that can verify frees.
+	 * that has an allocated space map object then load the space map
+	 * so that we can verify frees.
 	 */
 	if (metaslab_debug_load && ms->ms_sm != NULL) {
 		mutex_enter(&ms->ms_lock);
@@ -1619,16 +1641,19 @@ void
 metaslab_fini(metaslab_t *msp)
 {
 	metaslab_group_t *mg = msp->ms_group;
+	vdev_t *vd = mg->mg_vd;
 
 	metaslab_group_remove(mg, msp);
 
 	mutex_enter(&msp->ms_lock);
 	VERIFY(msp->ms_group == NULL);
-	vdev_space_update(mg->mg_vd, -space_map_allocated(msp->ms_sm),
-	    0, -msp->ms_size);
+	metaslab_space_update(vd, mg->mg_class,
+	    -space_map_allocated(msp->ms_sm), 0, -msp->ms_size);
+
 	space_map_close(msp->ms_sm);
 
 	metaslab_unload(msp);
+
 	range_tree_destroy(msp->ms_allocatable);
 	range_tree_destroy(msp->ms_freeing);
 	range_tree_destroy(msp->ms_freed);
@@ -2643,7 +2668,7 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 		ASSERT3P(msp->ms_checkpointing, ==, NULL);
 		msp->ms_checkpointing = range_tree_create(NULL, NULL);
 
-		vdev_space_update(vd, 0, 0, msp->ms_size);
+		metaslab_space_update(vd, mg->mg_class, 0, 0, msp->ms_size);
 	}
 	ASSERT0(range_tree_space(msp->ms_freeing));
 	ASSERT0(range_tree_space(msp->ms_checkpointing));
@@ -2665,7 +2690,8 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 		defer_delta -= range_tree_space(*defer_tree);
 	}
 
-	vdev_space_update(vd, alloc_delta + defer_delta, defer_delta, 0);
+	metaslab_space_update(vd, mg->mg_class, alloc_delta + defer_delta,
+	    defer_delta, 0);
 
 	/*
 	 * If there's a metaslab_load() in progress, wait for it to complete
@@ -2764,21 +2790,25 @@ metaslab_sync_reassess(metaslab_group_t *mg)
 	spa_config_exit(spa, SCL_ALLOC, FTAG);
 }
 
-static uint64_t
-metaslab_distance(metaslab_t *msp, dva_t *dva)
+/*
+ * When writing a ditto block (i.e. more than one DVA for a given BP) on
+ * the same vdev as an existing DVA of this BP, then try to allocate it
+ * on a different metaslab than existing DVAs (i.e. a unique metaslab).
+ */
+static boolean_t
+metaslab_is_unique(metaslab_t *msp, dva_t *dva)
 {
-	uint64_t ms_shift = msp->ms_group->mg_vd->vdev_ms_shift;
-	uint64_t offset = DVA_GET_OFFSET(dva) >> ms_shift;
-	uint64_t start = msp->ms_id;
+	uint64_t dva_ms_id;
 
+	if (DVA_GET_ASIZE(dva) == 0)
+		return (B_TRUE);
+
 	if (msp->ms_group->mg_vd->vdev_id != DVA_GET_VDEV(dva))
-		return (1ULL << 63);
+		return (B_TRUE);
 
-	if (offset < start)
-		return ((start - offset) << ms_shift);
-	if (offset > start)
-		return ((offset - start) << ms_shift);
-	return (0);
+	dva_ms_id = DVA_GET_OFFSET(dva) >> msp->ms_group->mg_vd->vdev_ms_shift;
+
+	return (msp->ms_id != dva_ms_id);
 }
 
 /*
@@ -3012,7 +3042,7 @@ metaslab_block_alloc(metaslab_t *msp, uint64_t size, u
  */
 static metaslab_t *
 find_valid_metaslab(metaslab_group_t *mg, uint64_t activation_weight,
-    dva_t *dva, int d, uint64_t min_distance, uint64_t asize, int allocator,
+    dva_t *dva, int d, boolean_t want_unique, uint64_t asize, int allocator,
     zio_alloc_list_t *zal, metaslab_t *search, boolean_t *was_active)
 {
 	avl_index_t idx;
@@ -3047,13 +3077,10 @@ find_valid_metaslab(metaslab_group_t *mg, uint64_t act
 		if (activation_weight == METASLAB_WEIGHT_PRIMARY || *was_active)
 			break;
 
-		uint64_t target_distance = min_distance
-		    + (space_map_allocated(msp->ms_sm) != 0 ? 0 :
-		    min_distance >> 1);
-
 		for (i = 0; i < d; i++) {
-			if (metaslab_distance(msp, &dva[i]) < target_distance)
-				break;
+			if (want_unique &&
+			    !metaslab_is_unique(msp, &dva[i]))
+				break;  /* try another metaslab */
 		}
 		if (i == d)
 			break;
@@ -3071,8 +3098,8 @@ find_valid_metaslab(metaslab_group_t *mg, uint64_t act
 /* ARGSUSED */
 static uint64_t
 metaslab_group_alloc_normal(metaslab_group_t *mg, zio_alloc_list_t *zal,
-    uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d,
-    int allocator)
+    uint64_t asize, uint64_t txg, boolean_t want_unique, dva_t *dva,
+    int d, int allocator)
 {
 	metaslab_t *msp = NULL;
 	uint64_t offset = -1ULL;
@@ -3126,7 +3153,7 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_
 			was_active = B_TRUE;
 		} else {
 			msp = find_valid_metaslab(mg, activation_weight, dva, d,
-			    min_distance, asize, allocator, zal, search,
+			    want_unique, asize, allocator, zal, search,
 			    &was_active);
 		}
 
@@ -3264,6 +3291,7 @@ next:
 		 * metaslab.
 		 */
 		ASSERT(!metaslab_should_allocate(msp, asize));
+
 		mutex_exit(&msp->ms_lock);
 	}
 	mutex_exit(&msp->ms_lock);
@@ -3273,14 +3301,14 @@ next:
 
 static uint64_t
 metaslab_group_alloc(metaslab_group_t *mg, zio_alloc_list_t *zal,
-    uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d,
-    int allocator)
+    uint64_t asize, uint64_t txg, boolean_t want_unique, dva_t *dva,
+    int d, int allocator)
 {
 	uint64_t offset;
 	ASSERT(mg->mg_initialized);
 
-	offset = metaslab_group_alloc_normal(mg, zal, asize, txg,
-	    min_distance, dva, d, allocator);
+	offset = metaslab_group_alloc_normal(mg, zal, asize, txg, want_unique,
+	    dva, d, allocator);
 
 	mutex_enter(&mg->mg_lock);
 	if (offset == -1ULL) {
@@ -3308,14 +3336,6 @@ metaslab_group_alloc(metaslab_group_t *mg, zio_alloc_l
 }
 
 /*
- * If we have to write a ditto block (i.e. more than one DVA for a given BP)
- * on the same vdev as an existing DVA of this BP, then try to allocate it
- * at least (vdev_asize / (2 ^ ditto_same_vdev_distance_shift)) away from the
- * existing DVAs.
- */
-int ditto_same_vdev_distance_shift = 3;
-
-/*
  * Allocate a block for the specified i/o.
  */
 int
@@ -3331,6 +3351,7 @@ metaslab_alloc_dva(spa_t *spa, metaslab_class_t *mc, u
 
 	/*
 	 * For testing, make some blocks above a certain size be gang blocks.
+	 * This will also test spilling from special to normal.
 	 */
 	if (psize >= metaslab_force_ganging && (ddi_get_lbolt() & 3) == 0) {
 		metaslab_trace_add(zal, NULL, NULL, psize, d, TRACE_FORCE_GANG,
@@ -3382,6 +3403,7 @@ metaslab_alloc_dva(spa_t *spa, metaslab_class_t *mc, u
 		vd = vdev_lookup_top(spa, DVA_GET_VDEV(&dva[d - 1]));
 		mg = vd->vdev_mg->mg_next;
 	} else {
+		ASSERT(mc->mc_rotor != NULL);
 		mg = mc->mc_rotor;
 	}
 
@@ -3446,25 +3468,17 @@ top:
 
 		ASSERT(mg->mg_class == mc);
 
-		/*
-		 * If we don't need to try hard, then require that the
-		 * block be 1/8th of the device away from any other DVAs
-		 * in this BP.  If we are trying hard, allow any offset
-		 * to be used (distance=0).
-		 */
-		uint64_t distance = 0;
-		if (!try_hard) {
-			distance = vd->vdev_asize >>
-			    ditto_same_vdev_distance_shift;
-			if (distance <= (1ULL << vd->vdev_ms_shift))
-				distance = 0;
-		}
-
 		uint64_t asize = vdev_psize_to_asize(vd, psize);
 		ASSERT(P2PHASE(asize, 1ULL << vd->vdev_ashift) == 0);
 
+		/*
+		 * If we don't need to try hard, then require that the
+		 * block be on an different metaslab from any other DVAs
+		 * in this BP (unique=true).  If we are trying hard, then
+		 * allow any metaslab to be used (unique=false).
+		 */
 		uint64_t offset = metaslab_group_alloc(mg, zal, asize, txg,
-		    distance, dva, d, allocator);
+		    !try_hard, dva, d, allocator);
 
 		if (offset != -1ULL) {
 			/*
@@ -3843,7 +3857,8 @@ metaslab_class_throttle_reserve(metaslab_class_t *mc, 
 	if (reserved_slots < max)
 		available_slots = max - reserved_slots;
 
-	if (slots <= available_slots || GANG_ALLOCATION(flags)) {
+	if (slots <= available_slots || GANG_ALLOCATION(flags) ||
+	    flags & METASLAB_MUST_RESERVE) {
 		/*
 		 * We reserve the slots individually so that we can unreserve
 		 * them individually when an I/O completes.
@@ -4126,9 +4141,11 @@ metaslab_claim(spa_t *spa, const blkptr_t *bp, uint64_
 
 	spa_config_enter(spa, SCL_ALLOC, FTAG, RW_READER);
 
-	for (int d = 0; d < ndvas; d++)
-		if ((error = metaslab_claim_dva(spa, &dva[d], txg)) != 0)
+	for (int d = 0; d < ndvas; d++) {
+		error = metaslab_claim_dva(spa, &dva[d], txg);
+		if (error != 0)
 			break;
+	}
 
 	spa_config_exit(spa, SCL_ALLOC, FTAG);
 

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -28,6 +28,7 @@
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2016 Toomas Soome 
  * Copyright 2018 Joyent, Inc.
+ * Copyright (c) 2017, Intel Corporation.
  * Copyright (c) 2017 Datto Inc.
  * Copyright 2018 OmniOS Community Edition (OmniOSce) Association.
  */
@@ -274,8 +275,14 @@ spa_prop_get_config(spa_t *spa, nvlist_t **nvp)
 	ASSERT(MUTEX_HELD(&spa->spa_props_lock));
 
 	if (rvd != NULL) {
-		alloc = metaslab_class_get_alloc(spa_normal_class(spa));
-		size = metaslab_class_get_space(spa_normal_class(spa));
+		alloc = metaslab_class_get_alloc(mc);
+		alloc += metaslab_class_get_alloc(spa_special_class(spa));
+		alloc += metaslab_class_get_alloc(spa_dedup_class(spa));
+
+		size = metaslab_class_get_space(mc);
+		size += metaslab_class_get_space(spa_special_class(spa));
+		size += metaslab_class_get_space(spa_dedup_class(spa));
+
 		spa_prop_add_list(*nvp, ZPOOL_PROP_NAME, spa_name(spa), 0, src);
 		spa_prop_add_list(*nvp, ZPOOL_PROP_SIZE, NULL, size, src);
 		spa_prop_add_list(*nvp, ZPOOL_PROP_ALLOCATED, NULL, alloc, src);
@@ -1137,6 +1144,8 @@ spa_activate(spa_t *spa, int mode)
 
 	spa->spa_normal_class = metaslab_class_create(spa, zfs_metaslab_ops);
 	spa->spa_log_class = metaslab_class_create(spa, zfs_metaslab_ops);
+	spa->spa_special_class = metaslab_class_create(spa, zfs_metaslab_ops);
+	spa->spa_dedup_class = metaslab_class_create(spa, zfs_metaslab_ops);
 
 	/* Try to create a covering process */
 	mutex_enter(&spa->spa_proc_lock);
@@ -1232,6 +1241,12 @@ spa_deactivate(spa_t *spa)
 	metaslab_class_destroy(spa->spa_log_class);
 	spa->spa_log_class = NULL;
 
+	metaslab_class_destroy(spa->spa_special_class);
+	spa->spa_special_class = NULL;
+
+	metaslab_class_destroy(spa->spa_dedup_class);
+	spa->spa_dedup_class = NULL;
+
 	/*
 	 * If this was part of an import or the open otherwise failed, we may
 	 * still have errors left in the queues.  Empty them just in case.
@@ -4834,7 +4849,8 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_
 	char *poolname;
 	nvlist_t *nvl;
 
-	if (nvlist_lookup_string(props,
+	if (props == NULL ||
+	    nvlist_lookup_string(props,
 	    zpool_prop_to_name(ZPOOL_PROP_TNAME), &poolname) != 0)
 		poolname = (char *)pool;
 
@@ -4922,9 +4938,15 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_
 	    (error = vdev_create(rvd, txg, B_FALSE)) == 0 &&
 	    (error = spa_validate_aux(spa, nvroot, txg,
 	    VDEV_ALLOC_ADD)) == 0) {
-		for (int c = 0; c < rvd->vdev_children; c++) {
-			vdev_metaslab_set_size(rvd->vdev_child[c]);
-			vdev_expand(rvd->vdev_child[c], txg);
+		/*
+		 * instantiate the metaslab groups (this will dirty the vdevs)
+		 * we can no longer error exit past this point
+		 */
+		for (int c = 0; error == 0 && c < rvd->vdev_children; c++) {
+			vdev_t *vd = rvd->vdev_child[c];
+
+			vdev_metaslab_set_size(vd);
+			vdev_expand(vd, txg);
 		}
 	}
 
@@ -7082,8 +7104,14 @@ spa_async_thread(void *arg)
 
 		mutex_enter(&spa_namespace_lock);
 		old_space = metaslab_class_get_space(spa_normal_class(spa));
+		old_space += metaslab_class_get_space(spa_special_class(spa));
+		old_space += metaslab_class_get_space(spa_dedup_class(spa));
+
 		spa_config_update(spa, SPA_CONFIG_UPDATE_POOL);
+
 		new_space = metaslab_class_get_space(spa_normal_class(spa));
+		new_space += metaslab_class_get_space(spa_special_class(spa));
+		new_space += metaslab_class_get_space(spa_dedup_class(spa));
 		mutex_exit(&spa_namespace_lock);
 
 		/*
@@ -7780,6 +7808,9 @@ spa_sync(spa_t *spa, uint64_t txg)
 	dsl_pool_t *dp = spa->spa_dsl_pool;
 	objset_t *mos = spa->spa_meta_objset;
 	bplist_t *free_bpl = &spa->spa_free_bplist[txg & TXG_MASK];
+	metaslab_class_t *normal = spa_normal_class(spa);
+	metaslab_class_t *special = spa_special_class(spa);
+	metaslab_class_t *dedup = spa_dedup_class(spa);
 	vdev_t *rvd = spa->spa_root_vdev;
 	vdev_t *vd;
 	dmu_tx_t *tx;
@@ -7872,11 +7903,15 @@ spa_sync(spa_t *spa, uint64_t txg)
 	for (int c = 0; c < rvd->vdev_children; c++) {
 		vdev_t *tvd = rvd->vdev_child[c];
 		metaslab_group_t *mg = tvd->vdev_mg;
+		metaslab_class_t *mc;
 
-		if (mg == NULL || mg->mg_class != spa_normal_class(spa) ||
-		    !metaslab_group_initialized(mg))
+		if (mg == NULL || !metaslab_group_initialized(mg))
 			continue;
 
+		mc = mg->mg_class;
+		if (mc != normal && mc != special && mc != dedup)
+			continue;
+
 		/*
 		 * It is safe to do a lock-free check here because only async
 		 * allocations look at mg_max_alloc_queue_depth, and async
@@ -7893,12 +7928,18 @@ spa_sync(spa_t *spa, uint64_t txg)
 		}
 		slots_per_allocator += zfs_vdev_def_queue_depth;
 	}
-	metaslab_class_t *mc = spa_normal_class(spa);
+
 	for (int i = 0; i < spa->spa_alloc_count; i++) {
-		ASSERT0(zfs_refcount_count(&mc->mc_alloc_slots[i]));
-		mc->mc_alloc_max_slots[i] = slots_per_allocator;
+		ASSERT0(zfs_refcount_count(&normal->mc_alloc_slots[i]));
+		ASSERT0(zfs_refcount_count(&special->mc_alloc_slots[i]));
+		ASSERT0(zfs_refcount_count(&dedup->mc_alloc_slots[i]));
+		normal->mc_alloc_max_slots[i] = slots_per_allocator;
+		special->mc_alloc_max_slots[i] = slots_per_allocator;
+		dedup->mc_alloc_max_slots[i] = slots_per_allocator;
 	}
-	mc->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
+	normal->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
+	special->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
+	dedup->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
 
 	for (int c = 0; c < rvd->vdev_children; c++) {
 		vdev_t *vd = rvd->vdev_child[c];

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -26,6 +26,7 @@
  * Copyright 2013 Saso Kiselkov. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright (c) 2017 Datto Inc.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #include 
@@ -388,6 +389,19 @@ spa_load_note(spa_t *spa, const char *fmt, ...)
 }
 
 /*
+ * By default dedup and user data indirects land in the special class
+ */
+int zfs_ddt_data_is_special = B_TRUE;
+int zfs_user_indirect_is_special = B_TRUE;
+
+/*
+ * The percentage of special class final space reserved for metadata only.
+ * Once we allocate 100 - zfs_special_class_metadata_reserve_pct we only
+ * let metadata into the class.
+ */
+int zfs_special_class_metadata_reserve_pct = 25;
+
+/*
  * ==========================================================================
  * SPA config locking
  * ==========================================================================
@@ -1173,6 +1187,8 @@ spa_vdev_config_exit(spa_t *spa, vdev_t *vd, uint64_t 
 	 */
 	ASSERT(metaslab_class_validate(spa_normal_class(spa)) == 0);
 	ASSERT(metaslab_class_validate(spa_log_class(spa)) == 0);
+	ASSERT(metaslab_class_validate(spa_special_class(spa)) == 0);
+	ASSERT(metaslab_class_validate(spa_dedup_class(spa)) == 0);
 
 	spa_config_exit(spa, SCL_ALL, spa);
 
@@ -1516,6 +1532,16 @@ zfs_strtonum(const char *str, char **nptr)
 	return (val);
 }
 
+void
+spa_activate_allocation_classes(spa_t *spa, dmu_tx_t *tx)
+{
+	/*
+	 * We bump the feature refcount for each special vdev added to the pool
+	 */
+	ASSERT(spa_feature_is_enabled(spa, SPA_FEATURE_ALLOCATION_CLASSES));
+	spa_feature_incr(spa, SPA_FEATURE_ALLOCATION_CLASSES, tx);
+}
+
 /*
  * ==========================================================================
  * Accessor functions
@@ -1763,6 +1789,79 @@ metaslab_class_t *
 spa_log_class(spa_t *spa)
 {
 	return (spa->spa_log_class);
+}
+
+metaslab_class_t *
+spa_special_class(spa_t *spa)
+{
+	return (spa->spa_special_class);
+}
+
+metaslab_class_t *
+spa_dedup_class(spa_t *spa)
+{
+	return (spa->spa_dedup_class);
+}
+
+/*
+ * Locate an appropriate allocation class
+ */
+metaslab_class_t *
+spa_preferred_class(spa_t *spa, uint64_t size, dmu_object_type_t objtype,
+    uint_t level, uint_t special_smallblk)
+{
+	if (DMU_OT_IS_ZIL(objtype)) {
+		if (spa->spa_log_class->mc_groups != 0)
+			return (spa_log_class(spa));
+		else
+			return (spa_normal_class(spa));
+	}
+
+	boolean_t has_special_class = spa->spa_special_class->mc_groups != 0;
+
+	if (DMU_OT_IS_DDT(objtype)) {
+		if (spa->spa_dedup_class->mc_groups != 0)
+			return (spa_dedup_class(spa));
+		else if (has_special_class && zfs_ddt_data_is_special)
+			return (spa_special_class(spa));
+		else
+			return (spa_normal_class(spa));
+	}
+
+	/* Indirect blocks for user data can land in special if allowed */
+	if (level > 0 && (DMU_OT_IS_FILE(objtype) || objtype == DMU_OT_ZVOL)) {
+		if (has_special_class && zfs_user_indirect_is_special)
+			return (spa_special_class(spa));
+		else
+			return (spa_normal_class(spa));
+	}
+
+	if (DMU_OT_IS_METADATA(objtype) || level > 0) {
+		if (has_special_class)
+			return (spa_special_class(spa));
+		else
+			return (spa_normal_class(spa));
+	}
+
+	/*
+	 * Allow small file blocks in special class in some cases (like
+	 * for the dRAID vdev feature). But always leave a reserve of
+	 * zfs_special_class_metadata_reserve_pct exclusively for metadata.
+	 */
+	if (DMU_OT_IS_FILE(objtype) &&
+	    has_special_class && size <= special_smallblk) {
+		metaslab_class_t *special = spa_special_class(spa);
+		uint64_t alloc = metaslab_class_get_alloc(special);
+		uint64_t space = metaslab_class_get_space(special);
+		uint64_t limit =
+		    (space * (100 - zfs_special_class_metadata_reserve_pct))
+		    / 100;
+
+		if (alloc < limit)
+			return (special);
+	}
+
+	return (spa_normal_class(spa));
 }
 
 void

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -21,13 +21,14 @@
 
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
  * Copyright 2011 Nexenta Systems, Inc. All rights reserved.
  * Copyright (c) 2012, Joyent, Inc. All rights reserved.
  * Copyright 2013 DEY Storage Systems, Inc.
  * Copyright 2014 HybridCluster. All rights reserved.
  * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
  * Copyright 2013 Saso Kiselkov. All rights reserved.
+ * Copyright (c) 2017, Intel Corporation.
  * Copyright (c) 2014 Integros [integros.com]
  */
 
@@ -126,6 +127,16 @@ typedef enum dmu_object_byteswap {
 	((ot) & DMU_OT_METADATA) : \
 	dmu_ot[(ot)].ot_metadata)
 
+#define	DMU_OT_IS_DDT(ot) \
+	((ot) == DMU_OT_DDT_ZAP)
+
+#define	DMU_OT_IS_ZIL(ot) \
+	((ot) == DMU_OT_INTENT_LOG)
+
+/* Note: ztest uses DMU_OT_UINT64_OTHER as a proxy for file blocks */
+#define	DMU_OT_IS_FILE(ot) \
+	((ot) == DMU_OT_PLAIN_FILE_CONTENTS || (ot) == DMU_OT_UINT64_OTHER)
+
 #define	DMU_OT_IS_METADATA_CACHED(ot) (((ot) & DMU_OT_NEWTYPE) ? \
 	B_TRUE : dmu_ot[(ot)].ot_dbuf_metadata_cache)
 
@@ -216,6 +227,7 @@ typedef enum dmu_object_type {
 	 *
 	 * The DMU_OTN_* types do not have entries in the dmu_ot table,
 	 * use the DMU_OT_IS_METDATA() and DMU_OT_BYTESWAP() macros instead
+	 * use the DMU_OT_IS_METADATA() and DMU_OT_BYTESWAP() macros instead
 	 * of indexing into dmu_ot directly (this works for both DMU_OT_* types
 	 * and DMU_OTN_* types).
 	 */

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -20,7 +20,7 @@
  */
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2012, 2018 by Delphix. All rights reserved.
  * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
@@ -113,6 +113,11 @@ struct objset {
 	uint64_t os_normalization;
 	uint64_t os_utf8only;
 	uint64_t os_casesensitivity;
+	/*
+	 * The largest zpl file block allowed in special class.
+	 * cached here instead of zfsvfs for easier access.
+	 */
+	int os_zpl_special_smallblock;
 
 	/*
 	 * Pointer is constant; the blkptr it points to is protected by

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -21,6 +21,7 @@
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _SYS_METASLAB_H
@@ -56,12 +57,17 @@ void metaslab_sync_done(metaslab_t *, uint64_t);
 void metaslab_sync_reassess(metaslab_group_t *);
 uint64_t metaslab_block_maxsize(metaslab_t *);
 
+/*
+ * metaslab alloc flags
+ */
 #define	METASLAB_HINTBP_FAVOR		0x0
 #define	METASLAB_HINTBP_AVOID		0x1
 #define	METASLAB_GANG_HEADER		0x2
 #define	METASLAB_GANG_CHILD		0x4
 #define	METASLAB_ASYNC_ALLOC		0x8
 #define	METASLAB_DONT_THROTTLE		0x10
+#define	METASLAB_MUST_RESERVE		0x20
+#define	METASLAB_FASTWRITE		0x40
 
 int metaslab_alloc(spa_t *, metaslab_class_t *, uint64_t,
     blkptr_t *, int, uint64_t, blkptr_t *, int, zio_alloc_list_t *, zio_t *,
@@ -92,8 +98,6 @@ boolean_t metaslab_class_throttle_reserve(metaslab_cla
     zio_t *, int);
 void metaslab_class_throttle_unreserve(metaslab_class_t *, int, int, zio_t *);
 
-void metaslab_class_space_update(metaslab_class_t *, int64_t, int64_t,
-    int64_t, int64_t);
 uint64_t metaslab_class_get_alloc(metaslab_class_t *);
 uint64_t metaslab_class_get_space(metaslab_class_t *);
 uint64_t metaslab_class_get_dspace(metaslab_class_t *);

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -27,6 +27,7 @@
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2017 Joyent, Inc.
  * Copyright (c) 2017 Datto Inc.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _SYS_SPA_H
@@ -801,6 +802,11 @@ extern uint64_t spa_version(spa_t *spa);
 extern boolean_t spa_deflate(spa_t *spa);
 extern metaslab_class_t *spa_normal_class(spa_t *spa);
 extern metaslab_class_t *spa_log_class(spa_t *spa);
+extern metaslab_class_t *spa_special_class(spa_t *spa);
+extern metaslab_class_t *spa_dedup_class(spa_t *spa);
+extern metaslab_class_t *spa_preferred_class(spa_t *spa, uint64_t size,
+    dmu_object_type_t objtype, uint_t level, uint_t special_smallblk);
+
 extern void spa_evicting_os_register(spa_t *, objset_t *os);
 extern void spa_evicting_os_deregister(spa_t *, objset_t *os);
 extern void spa_evicting_os_wait(spa_t *spa);
@@ -862,6 +868,7 @@ extern boolean_t spa_trust_config(spa_t *spa);
 extern uint64_t spa_missing_tvds_allowed(spa_t *spa);
 extern void spa_set_missing_tvds(spa_t *spa, uint64_t missing);
 extern boolean_t spa_top_vdevs_spacemap_addressable(spa_t *spa);
+extern void spa_activate_allocation_classes(spa_t *, dmu_tx_t *);
 
 extern int spa_mode(spa_t *spa);
 extern uint64_t zfs_strtonum(const char *str, char **nptr);

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -25,6 +25,7 @@
  * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
  * Copyright 2013 Saso Kiselkov. All rights reserved.
  * Copyright (c) 2017 Datto Inc.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _SYS_SPA_IMPL_H
@@ -218,6 +219,8 @@ struct spa {
 	boolean_t	spa_is_initializing;	/* true while opening pool */
 	metaslab_class_t *spa_normal_class;	/* normal data class */
 	metaslab_class_t *spa_log_class;	/* intent log data class */
+	metaslab_class_t *spa_special_class;	/* special allocation class */
+	metaslab_class_t *spa_dedup_class;	/* dedup allocation class */
 	uint64_t	spa_first_txg;		/* first txg after spa_open() */
 	uint64_t	spa_final_txg;		/* txg of export/destroy */
 	uint64_t	spa_freeze_txg;		/* freeze pool at this txg */

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -22,6 +22,7 @@
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _SYS_VDEV_H
@@ -106,6 +107,8 @@ extern boolean_t vdev_children_are_offline(vdev_t *vd)
 
 extern void vdev_space_update(vdev_t *vd,
     int64_t alloc_delta, int64_t defer_delta, int64_t space_delta);
+
+extern int64_t vdev_deflated_space(vdev_t *vd, int64_t space);
 
 extern uint64_t vdev_psize_to_asize(vdev_t *vd, uint64_t psize);
 

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -21,6 +21,7 @@
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #ifndef _SYS_VDEV_IMPL_H
@@ -146,6 +147,14 @@ struct vdev_queue {
 	kmutex_t	vq_lock;
 };
 
+typedef enum vdev_alloc_bias {
+	VDEV_BIAS_NONE,
+	VDEV_BIAS_LOG,		/* dedicated to ZIL data (SLOG) */
+	VDEV_BIAS_SPECIAL,	/* dedicated to ddt, metadata, and small blks */
+	VDEV_BIAS_DEDUP		/* dedicated to dedup metadata */
+} vdev_alloc_bias_t;
+
+
 /*
  * On-disk indirect vdev state.
  *
@@ -239,6 +248,7 @@ struct vdev {
 	boolean_t	vdev_ishole;	/* is a hole in the namespace	*/
 	kmutex_t	vdev_queue_lock; /* protects vdev_queue_depth	*/
 	uint64_t	vdev_top_zap;
+	vdev_alloc_bias_t vdev_alloc_bias; /* metaslab allocation bias	*/
 
 	/* pool checkpoint related */
 	space_map_t	*vdev_checkpoint_sm;	/* contains reserved blocks */

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h	Wed Nov  6 08:58:03 2019	(r354382)
@@ -313,6 +313,7 @@ typedef struct zio_prop {
 	boolean_t		zp_dedup;
 	boolean_t		zp_dedup_verify;
 	boolean_t		zp_nopwrite;
+	uint32_t		zp_zpl_smallblk;
 } zio_prop_t;
 
 typedef struct zio_cksum_report zio_cksum_report_t;
@@ -433,6 +434,7 @@ struct zio {
 	vdev_t		*io_vd;
 	void		*io_vsd;
 	const zio_vsd_ops_t *io_vsd_ops;
+	metaslab_class_t *io_metaslab_class;	/* dva throttle class */
 
 	uint64_t	io_offset;
 	hrtime_t	io_timestamp;

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c	Wed Nov  6 08:55:23 2019	(r354381)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c	Wed Nov  6 08:58:03 2019	(r354382)
@@ -26,6 +26,7 @@
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2016 Toomas Soome 
  * Copyright 2017 Joyent, Inc.
+ * Copyright (c) 2017, Intel Corporation.
  */
 
 #include 
@@ -192,6 +193,25 @@ vdev_getops(const char *type)
 	return (ops);
 }
 
+/*
+ * Derive the enumerated alloction bias from string input.
+ * String origin is either the per-vdev zap or zpool(1M).
+ */
+static vdev_alloc_bias_t
+vdev_derive_alloc_bias(const char *bias)
+{
+	vdev_alloc_bias_t alloc_bias = VDEV_BIAS_NONE;
+
+	if (strcmp(bias, VDEV_ALLOC_BIAS_LOG) == 0)
+		alloc_bias = VDEV_BIAS_LOG;
+	else if (strcmp(bias, VDEV_ALLOC_BIAS_SPECIAL) == 0)
+		alloc_bias = VDEV_BIAS_SPECIAL;
+	else if (strcmp(bias, VDEV_ALLOC_BIAS_DEDUP) == 0)
+		alloc_bias = VDEV_BIAS_DEDUP;
+
+	return (alloc_bias);
+}
+
 /* ARGSUSED */
 void
 vdev_default_xlate(vdev_t *vd, const range_seg_t *in, range_seg_t *res)
@@ -515,6 +535,8 @@ vdev_alloc(spa_t *spa, vdev_t **vdp, nvlist_t *nv, vde
 	uint64_t guid = 0, islog, nparity;
 	vdev_t *vd;
 	vdev_indirect_config_t *vic;
+	vdev_alloc_bias_t alloc_bias = VDEV_BIAS_NONE;
+	boolean_t top_level = (parent && !parent->vdev_parent);
 
 	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);
 
@@ -601,11 +623,32 @@ vdev_alloc(spa_t *spa, vdev_t **vdp, nvlist_t *nv, vde
 	}
 	ASSERT(nparity != -1ULL);

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:00:08 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1EA0A1AF13B;
 Wed,  6 Nov 2019 09:00:08 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L8S0lPQz3KWg;
 Wed,  6 Nov 2019 09:00:08 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F1DA3229DE;
 Wed,  6 Nov 2019 09:00:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6907wT047388;
 Wed, 6 Nov 2019 09:00:07 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA6907VL047386;
 Wed, 6 Nov 2019 09:00:07 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060900.xA6907VL047386@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:00:07 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354383 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb
 vendor/illumos/dist/man/man1m
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb
 vendor/illumos/dist/man/man1m
X-SVN-Commit-Revision: 354383
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:00:08 -0000

Author: avg
Date: Wed Nov  6 09:00:06 2019
New Revision: 354383
URL: https://svnweb.freebsd.org/changeset/base/354383

Log:
  10592 misc. metaslab and vdev related ZoL bug fixes
  
  illumos/illumos-gate@555d674d5d4b8191dc83723188349d28278b2431
  https://github.com/illumos/illumos-gate/commit/555d674d5d4b8191dc83723188349d28278b2431
  
  https://www.illumos.org/issues/10592
    This is a collection of recent fixes from ZoL:
    8eef997679b Error path in metaslab_load_impl() forgets to drop ms_sync_lock
    928e8ad47d3 Introduce auxiliary metaslab histograms
    425d3237ee8 Get rid of space_map_update() for ms_synced_length
    6c926f426a2 Simplify log vdev removal code
    21e7cf5da89 zdb -L should skip leak detection altogether
    df72b8bebe0 Rename range_tree_verify to range_tree_verify_not_present
    75058f33034 Remove unused vdev_t fields
  
  Portions contributed by: Jerry Jelinek 
  Author: Serapheim Dimitropoulos 

Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/man/man1m/zdb.1m

Changes in other areas also in this revision:
Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_initialize.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c

Modified: vendor/illumos/dist/cmd/zdb/zdb.c
==============================================================================
--- vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 09:00:06 2019	(r354383)
@@ -785,18 +785,21 @@ dump_spacemap(objset_t *os, space_map_t *sm)
 		return;
 
 	(void) printf("space map object %llu:\n",
-	    (longlong_t)sm->sm_phys->smp_object);
-	(void) printf("  smp_objsize = 0x%llx\n",
-	    (longlong_t)sm->sm_phys->smp_objsize);
+	    (longlong_t)sm->sm_object);
+	(void) printf("  smp_length = 0x%llx\n",
+	    (longlong_t)sm->sm_phys->smp_length);
 	(void) printf("  smp_alloc = 0x%llx\n",
 	    (longlong_t)sm->sm_phys->smp_alloc);
 
+	if (dump_opt['d'] < 6 && dump_opt['m'] < 4)
+		return;
+
 	/*
 	 * Print out the freelist entries in both encoded and decoded form.
 	 */
 	uint8_t mapshift = sm->sm_shift;
 	int64_t alloc = 0;
-	uint64_t word;
+	uint64_t word, entry_id = 0;
 	for (uint64_t offset = 0; offset < space_map_length(sm);
 	    offset += sizeof (word)) {
 
@@ -804,11 +807,12 @@ dump_spacemap(objset_t *os, space_map_t *sm)
 		    sizeof (word), &word, DMU_READ_PREFETCH));
 
 		if (sm_entry_is_debug(word)) {
-			(void) printf("\t    [%6llu] %s: txg %llu, pass %llu\n",
-			    (u_longlong_t)(offset / sizeof (word)),
+			(void) printf("\t    [%6llu] %s: txg %llu pass %llu\n",
+			    (u_longlong_t)entry_id,
 			    ddata[SM_DEBUG_ACTION_DECODE(word)],
 			    (u_longlong_t)SM_DEBUG_TXG_DECODE(word),
 			    (u_longlong_t)SM_DEBUG_SYNCPASS_DECODE(word));
+			entry_id++;
 			continue;
 		}
 
@@ -846,7 +850,7 @@ dump_spacemap(objset_t *os, space_map_t *sm)
 
 		(void) printf("\t    [%6llu]    %c  range:"
 		    " %010llx-%010llx  size: %06llx vdev: %06llu words: %u\n",
-		    (u_longlong_t)(offset / sizeof (word)),
+		    (u_longlong_t)entry_id,
 		    entry_type, (u_longlong_t)entry_off,
 		    (u_longlong_t)(entry_off + entry_run),
 		    (u_longlong_t)entry_run,
@@ -856,8 +860,9 @@ dump_spacemap(objset_t *os, space_map_t *sm)
 			alloc += entry_run;
 		else
 			alloc -= entry_run;
+		entry_id++;
 	}
-	if ((uint64_t)alloc != space_map_allocated(sm)) {
+	if (alloc != space_map_allocated(sm)) {
 		(void) printf("space_map_object alloc (%lld) INCONSISTENT "
 		    "with space map summary (%lld)\n",
 		    (longlong_t)space_map_allocated(sm), (longlong_t)alloc);
@@ -921,11 +926,8 @@ dump_metaslab(metaslab_t *msp)
 		    SPACE_MAP_HISTOGRAM_SIZE, sm->sm_shift);
 	}
 
-	if (dump_opt['d'] > 5 || dump_opt['m'] > 3) {
-		ASSERT(msp->ms_size == (1ULL << vd->vdev_ms_shift));
-
-		dump_spacemap(spa->spa_meta_objset, msp->ms_sm);
-	}
+	ASSERT(msp->ms_size == (1ULL << vd->vdev_ms_shift));
+	dump_spacemap(spa->spa_meta_objset, msp->ms_sm);
 }
 
 static void
@@ -3096,6 +3098,8 @@ zdb_ddt_leak_init(spa_t *spa, zdb_cb_t *zcb)
 	ddt_entry_t dde;
 	int error;
 
+	ASSERT(!dump_opt['L']);
+
 	bzero(&ddb, sizeof (ddb));
 	while ((error = ddt_walk(spa, &ddb, &dde)) == 0) {
 		blkptr_t blk;
@@ -3119,12 +3123,10 @@ zdb_ddt_leak_init(spa_t *spa, zdb_cb_t *zcb)
 				zcb->zcb_dedup_blocks++;
 			}
 		}
-		if (!dump_opt['L']) {
-			ddt_t *ddt = spa->spa_ddt[ddb.ddb_checksum];
-			ddt_enter(ddt);
-			VERIFY(ddt_lookup(ddt, &blk, B_TRUE) != NULL);
-			ddt_exit(ddt);
-		}
+		ddt_t *ddt = spa->spa_ddt[ddb.ddb_checksum];
+		ddt_enter(ddt);
+		VERIFY(ddt_lookup(ddt, &blk, B_TRUE) != NULL);
+		ddt_exit(ddt);
 	}
 
 	ASSERT(error == ENOENT);
@@ -3166,6 +3168,9 @@ claim_segment_cb(void *arg, uint64_t offset, uint64_t 
 static void
 zdb_claim_removing(spa_t *spa, zdb_cb_t *zcb)
 {
+	if (dump_opt['L'])
+		return;
+
 	if (spa->spa_vdev_removal == NULL)
 		return;
 
@@ -3257,7 +3262,6 @@ zdb_load_obsolete_counts(vdev_t *vd)
 		space_map_t *prev_obsolete_sm = NULL;
 		VERIFY0(space_map_open(&prev_obsolete_sm, spa->spa_meta_objset,
 		    scip->scip_prev_obsolete_sm_object, 0, vd->vdev_asize, 0));
-		space_map_update(prev_obsolete_sm);
 		vdev_indirect_mapping_load_obsolete_spacemap(vim, counts,
 		    prev_obsolete_sm);
 		space_map_close(prev_obsolete_sm);
@@ -3351,9 +3355,9 @@ zdb_leak_init_vdev_exclude_checkpoint(vdev_t *vd, zdb_
 
 	VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(spa),
 	    checkpoint_sm_obj, 0, vd->vdev_asize, vd->vdev_ashift));
-	space_map_update(checkpoint_sm);
 
 	VERIFY0(space_map_iterate(checkpoint_sm,
+	    space_map_length(checkpoint_sm),
 	    checkpoint_sm_exclude_entry_cb, &cseea));
 	space_map_close(checkpoint_sm);
 
@@ -3363,6 +3367,8 @@ zdb_leak_init_vdev_exclude_checkpoint(vdev_t *vd, zdb_
 static void
 zdb_leak_init_exclude_checkpoint(spa_t *spa, zdb_cb_t *zcb)
 {
+	ASSERT(!dump_opt['L']);
+
 	vdev_t *rvd = spa->spa_root_vdev;
 	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
 		ASSERT3U(c, ==, rvd->vdev_child[c]->vdev_id);
@@ -3459,6 +3465,8 @@ load_indirect_ms_allocatable_tree(vdev_t *vd, metaslab
 static void
 zdb_leak_init_prepare_indirect_vdevs(spa_t *spa, zdb_cb_t *zcb)
 {
+	ASSERT(!dump_opt['L']);
+
 	vdev_t *rvd = spa->spa_root_vdev;
 	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
 		vdev_t *vd = rvd->vdev_child[c];
@@ -3505,67 +3513,63 @@ zdb_leak_init(spa_t *spa, zdb_cb_t *zcb)
 {
 	zcb->zcb_spa = spa;
 
-	if (!dump_opt['L']) {
-		dsl_pool_t *dp = spa->spa_dsl_pool;
-		vdev_t *rvd = spa->spa_root_vdev;
+	if (dump_opt['L'])
+		return;
 
-		/*
-		 * We are going to be changing the meaning of the metaslab's
-		 * ms_allocatable.  Ensure that the allocator doesn't try to
-		 * use the tree.
-		 */
-		spa->spa_normal_class->mc_ops = &zdb_metaslab_ops;
-		spa->spa_log_class->mc_ops = &zdb_metaslab_ops;
+	dsl_pool_t *dp = spa->spa_dsl_pool;
+	vdev_t *rvd = spa->spa_root_vdev;
 
-		zcb->zcb_vd_obsolete_counts =
-		    umem_zalloc(rvd->vdev_children * sizeof (uint32_t *),
-		    UMEM_NOFAIL);
+	/*
+	 * We are going to be changing the meaning of the metaslab's
+	 * ms_allocatable.  Ensure that the allocator doesn't try to
+	 * use the tree.
+	 */
+	spa->spa_normal_class->mc_ops = &zdb_metaslab_ops;
+	spa->spa_log_class->mc_ops = &zdb_metaslab_ops;
 
-		/*
-		 * For leak detection, we overload the ms_allocatable trees
-		 * to contain allocated segments instead of free segments.
-		 * As a result, we can't use the normal metaslab_load/unload
-		 * interfaces.
-		 */
-		zdb_leak_init_prepare_indirect_vdevs(spa, zcb);
-		load_concrete_ms_allocatable_trees(spa, SM_ALLOC);
+	zcb->zcb_vd_obsolete_counts =
+	    umem_zalloc(rvd->vdev_children * sizeof (uint32_t *),
+	    UMEM_NOFAIL);
 
-		/*
-		 * On load_concrete_ms_allocatable_trees() we loaded all the
-		 * allocated entries from the ms_sm to the ms_allocatable for
-		 * each metaslab. If the pool has a checkpoint or is in the
-		 * middle of discarding a checkpoint, some of these blocks
-		 * may have been freed but their ms_sm may not have been
-		 * updated because they are referenced by the checkpoint. In
-		 * order to avoid false-positives during leak-detection, we
-		 * go through the vdev's checkpoint space map and exclude all
-		 * its entries from their relevant ms_allocatable.
-		 *
-		 * We also aggregate the space held by the checkpoint and add
-		 * it to zcb_checkpoint_size.
-		 *
-		 * Note that at this point we are also verifying that all the
-		 * entries on the checkpoint_sm are marked as allocated in
-		 * the ms_sm of their relevant metaslab.
-		 * [see comment in checkpoint_sm_exclude_entry_cb()]
-		 */
-		zdb_leak_init_exclude_checkpoint(spa, zcb);
+	/*
+	 * For leak detection, we overload the ms_allocatable trees
+	 * to contain allocated segments instead of free segments.
+	 * As a result, we can't use the normal metaslab_load/unload
+	 * interfaces.
+	 */
+	zdb_leak_init_prepare_indirect_vdevs(spa, zcb);
+	load_concrete_ms_allocatable_trees(spa, SM_ALLOC);
 
-		/* for cleaner progress output */
-		(void) fprintf(stderr, "\n");
+	/*
+	 * On load_concrete_ms_allocatable_trees() we loaded all the
+	 * allocated entries from the ms_sm to the ms_allocatable for
+	 * each metaslab. If the pool has a checkpoint or is in the
+	 * middle of discarding a checkpoint, some of these blocks
+	 * may have been freed but their ms_sm may not have been
+	 * updated because they are referenced by the checkpoint. In
+	 * order to avoid false-positives during leak-detection, we
+	 * go through the vdev's checkpoint space map and exclude all
+	 * its entries from their relevant ms_allocatable.
+	 *
+	 * We also aggregate the space held by the checkpoint and add
+	 * it to zcb_checkpoint_size.
+	 *
+	 * Note that at this point we are also verifying that all the
+	 * entries on the checkpoint_sm are marked as allocated in
+	 * the ms_sm of their relevant metaslab.
+	 * [see comment in checkpoint_sm_exclude_entry_cb()]
+	 */
+	zdb_leak_init_exclude_checkpoint(spa, zcb);
+	ASSERT3U(zcb->zcb_checkpoint_size, ==, spa_get_checkpoint_space(spa));
 
-		if (bpobj_is_open(&dp->dp_obsolete_bpobj)) {
-			ASSERT(spa_feature_is_enabled(spa,
-			    SPA_FEATURE_DEVICE_REMOVAL));
-			(void) bpobj_iterate_nofree(&dp->dp_obsolete_bpobj,
-			    increment_indirect_mapping_cb, zcb, NULL);
-		}
-	} else {
-		/*
-		 * If leak tracing is disabled, we still need to consider
-		 * any checkpointed space in our space verification.
-		 */
-		zcb->zcb_checkpoint_size += spa_get_checkpoint_space(spa);
+	/* for cleaner progress output */
+	(void) fprintf(stderr, "\n");
+
+	if (bpobj_is_open(&dp->dp_obsolete_bpobj)) {
+		ASSERT(spa_feature_is_enabled(spa,
+		    SPA_FEATURE_DEVICE_REMOVAL));
+		(void) bpobj_iterate_nofree(&dp->dp_obsolete_bpobj,
+		    increment_indirect_mapping_cb, zcb, NULL);
 	}
 
 	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
@@ -3646,52 +3650,58 @@ zdb_check_for_obsolete_leaks(vdev_t *vd, zdb_cb_t *zcb
 static boolean_t
 zdb_leak_fini(spa_t *spa, zdb_cb_t *zcb)
 {
+	if (dump_opt['L'])
+		return (B_FALSE);
+
 	boolean_t leaks = B_FALSE;
-	if (!dump_opt['L']) {
-		vdev_t *rvd = spa->spa_root_vdev;
-		for (unsigned c = 0; c < rvd->vdev_children; c++) {
-			vdev_t *vd = rvd->vdev_child[c];
-			metaslab_group_t *mg = vd->vdev_mg;
 
-			if (zcb->zcb_vd_obsolete_counts[c] != NULL) {
-				leaks |= zdb_check_for_obsolete_leaks(vd, zcb);
-			}
+	vdev_t *rvd = spa->spa_root_vdev;
+	for (unsigned c = 0; c < rvd->vdev_children; c++) {
+		vdev_t *vd = rvd->vdev_child[c];
+#if DEBUG
+		metaslab_group_t *mg = vd->vdev_mg;
+#endif
 
-			for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
-				metaslab_t *msp = vd->vdev_ms[m];
-				ASSERT3P(mg, ==, msp->ms_group);
+		if (zcb->zcb_vd_obsolete_counts[c] != NULL) {
+			leaks |= zdb_check_for_obsolete_leaks(vd, zcb);
+		}
 
-				/*
-				 * ms_allocatable has been overloaded
-				 * to contain allocated segments. Now that
-				 * we finished traversing all blocks, any
-				 * block that remains in the ms_allocatable
-				 * represents an allocated block that we
-				 * did not claim during the traversal.
-				 * Claimed blocks would have been removed
-				 * from the ms_allocatable.  For indirect
-				 * vdevs, space remaining in the tree
-				 * represents parts of the mapping that are
-				 * not referenced, which is not a bug.
-				 */
-				if (vd->vdev_ops == &vdev_indirect_ops) {
-					range_tree_vacate(msp->ms_allocatable,
-					    NULL, NULL);
-				} else {
-					range_tree_vacate(msp->ms_allocatable,
-					    zdb_leak, vd);
-				}
+		for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
+			metaslab_t *msp = vd->vdev_ms[m];
+			ASSERT3P(mg, ==, msp->ms_group);
 
-				if (msp->ms_loaded) {
-					msp->ms_loaded = B_FALSE;
-				}
+			/*
+			 * ms_allocatable has been overloaded
+			 * to contain allocated segments. Now that
+			 * we finished traversing all blocks, any
+			 * block that remains in the ms_allocatable
+			 * represents an allocated block that we
+			 * did not claim during the traversal.
+			 * Claimed blocks would have been removed
+			 * from the ms_allocatable.  For indirect
+			 * vdevs, space remaining in the tree
+			 * represents parts of the mapping that are
+			 * not referenced, which is not a bug.
+			 */
+			if (vd->vdev_ops == &vdev_indirect_ops) {
+				range_tree_vacate(msp->ms_allocatable,
+				    NULL, NULL);
+			} else {
+				range_tree_vacate(msp->ms_allocatable,
+				    zdb_leak, vd);
 			}
+
+			if (msp->ms_loaded) {
+				msp->ms_loaded = B_FALSE;
+			}
 		}
 
-		umem_free(zcb->zcb_vd_obsolete_counts,
-		    rvd->vdev_children * sizeof (uint32_t *));
-		zcb->zcb_vd_obsolete_counts = NULL;
 	}
+
+	umem_free(zcb->zcb_vd_obsolete_counts,
+	    rvd->vdev_children * sizeof (uint32_t *));
+	zcb->zcb_vd_obsolete_counts = NULL;
+
 	return (leaks);
 }
 
@@ -3730,13 +3740,18 @@ dump_block_stats(spa_t *spa)
 	    !dump_opt['L'] ? "nothing leaked " : "");
 
 	/*
-	 * Load all space maps as SM_ALLOC maps, then traverse the pool
-	 * claiming each block we discover.  If the pool is perfectly
-	 * consistent, the space maps will be empty when we're done.
-	 * Anything left over is a leak; any block we can't claim (because
-	 * it's not part of any space map) is a double allocation,
-	 * reference to a freed block, or an unclaimed log block.
+	 * When leak detection is enabled we load all space maps as SM_ALLOC
+	 * maps, then traverse the pool claiming each block we discover. If
+	 * the pool is perfectly consistent, the segment trees will be empty
+	 * when we're done. Anything left over is a leak; any block we can't
+	 * claim (because it's not part of any space map) is a double
+	 * allocation, reference to a freed block, or an unclaimed log block.
+	 *
+	 * When leak detection is disabled (-L option) we still traverse the
+	 * pool claiming each block we discover, but we skip opening any space
+	 * maps.
 	 */
+	bzero(&zcb, sizeof (zdb_cb_t));
 	zdb_leak_init(spa, &zcb);
 
 	/*
@@ -3815,11 +3830,10 @@ dump_block_stats(spa_t *spa)
 	total_found = tzb->zb_asize - zcb.zcb_dedup_asize +
 	    zcb.zcb_removing_size + zcb.zcb_checkpoint_size;
 
-	if (total_found == total_alloc) {
-		if (!dump_opt['L'])
-			(void) printf("\n\tNo leaks (block sum matches space"
-			    " maps exactly)\n");
-	} else {
+	if (total_found == total_alloc && !dump_opt['L']) {
+		(void) printf("\n\tNo leaks (block sum matches space"
+		    " maps exactly)\n");
+	} else if (!dump_opt['L']) {
 		(void) printf("block traversal size %llu != alloc %llu "
 		    "(%s %lld)\n",
 		    (u_longlong_t)total_found,
@@ -4159,7 +4173,6 @@ verify_device_removal_feature_counts(spa_t *spa)
 			    spa->spa_meta_objset,
 			    scip->scip_prev_obsolete_sm_object,
 			    0, vd->vdev_asize, 0));
-			space_map_update(prev_obsolete_sm);
 			dump_spacemap(spa->spa_meta_objset, prev_obsolete_sm);
 			(void) printf("\n");
 			space_map_close(prev_obsolete_sm);
@@ -4365,7 +4378,8 @@ verify_checkpoint_sm_entry_cb(space_map_entry_t *sme, 
 	 * their respective ms_allocateable trees should not contain them.
 	 */
 	mutex_enter(&ms->ms_lock);
-	range_tree_verify(ms->ms_allocatable, sme->sme_offset, sme->sme_run);
+	range_tree_verify_not_present(ms->ms_allocatable,
+	    sme->sme_offset, sme->sme_run);
 	mutex_exit(&ms->ms_lock);
 
 	return (0);
@@ -4428,7 +4442,6 @@ verify_checkpoint_vdev_spacemaps(spa_t *checkpoint, sp
 		VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(current),
 		    checkpoint_sm_obj, 0, current_vd->vdev_asize,
 		    current_vd->vdev_ashift));
-		space_map_update(checkpoint_sm);
 
 		verify_checkpoint_sm_entry_cb_arg_t vcsec;
 		vcsec.vcsec_vd = ckpoint_vd;
@@ -4436,6 +4449,7 @@ verify_checkpoint_vdev_spacemaps(spa_t *checkpoint, sp
 		vcsec.vcsec_num_entries =
 		    space_map_length(checkpoint_sm) / sizeof (uint64_t);
 		VERIFY0(space_map_iterate(checkpoint_sm,
+		    space_map_length(checkpoint_sm),
 		    verify_checkpoint_sm_entry_cb, &vcsec));
 		dump_spacemap(current->spa_meta_objset, checkpoint_sm);
 		space_map_close(checkpoint_sm);
@@ -4515,7 +4529,7 @@ verify_checkpoint_ms_spacemaps(spa_t *checkpoint, spa_
 			 * are part of the checkpoint were freed by mistake.
 			 */
 			range_tree_walk(ckpoint_msp->ms_allocatable,
-			    (range_tree_func_t *)range_tree_verify,
+			    (range_tree_func_t *)range_tree_verify_not_present,
 			    current_msp->ms_allocatable);
 		}
 	}
@@ -4527,6 +4541,8 @@ verify_checkpoint_ms_spacemaps(spa_t *checkpoint, spa_
 static void
 verify_checkpoint_blocks(spa_t *spa)
 {
+	ASSERT(!dump_opt['L']);
+
 	spa_t *checkpoint_spa;
 	char *checkpoint_pool;
 	nvlist_t *config = NULL;
@@ -4592,7 +4608,6 @@ dump_leftover_checkpoint_blocks(spa_t *spa)
 
 		VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(spa),
 		    checkpoint_sm_obj, 0, vd->vdev_asize, vd->vdev_ashift));
-		space_map_update(checkpoint_sm);
 		dump_spacemap(spa->spa_meta_objset, checkpoint_sm);
 		space_map_close(checkpoint_sm);
 	}

Modified: vendor/illumos/dist/man/man1m/zdb.1m
==============================================================================
--- vendor/illumos/dist/man/man1m/zdb.1m	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor/illumos/dist/man/man1m/zdb.1m	Wed Nov  6 09:00:06 2019	(r354383)
@@ -10,7 +10,7 @@
 .\"
 .\"
 .\" Copyright 2012, Richard Lowe.
-.\" Copyright (c) 2012, 2017 by Delphix. All rights reserved.
+.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\"
 .Dd April 14, 2017
@@ -187,7 +187,7 @@ If the
 .Fl u
 option is also specified, also display the uberblocks on this device.
 .It Fl L
-Disable leak tracing and the loading of space maps.
+Disable leak detection and the loading of space maps.
 By default,
 .Nm
 verifies that all non-free blocks are referenced, which can be very expensive.

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:00:07 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8C7311AF138;
 Wed,  6 Nov 2019 09:00:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477L8R3x5Yz3KWf;
 Wed,  6 Nov 2019 09:00:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6A078229DD;
 Wed,  6 Nov 2019 09:00:07 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6907An047381;
 Wed, 6 Nov 2019 09:00:07 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA69067g047376;
 Wed, 6 Nov 2019 09:00:06 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060900.xA69067g047376@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:00:06 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354383 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb
 vendor/illumos/dist/man/man1m
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb
 vendor/illumos/dist/man/man1m
X-SVN-Commit-Revision: 354383
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:00:07 -0000

Author: avg
Date: Wed Nov  6 09:00:06 2019
New Revision: 354383
URL: https://svnweb.freebsd.org/changeset/base/354383

Log:
  10592 misc. metaslab and vdev related ZoL bug fixes
  
  illumos/illumos-gate@555d674d5d4b8191dc83723188349d28278b2431
  https://github.com/illumos/illumos-gate/commit/555d674d5d4b8191dc83723188349d28278b2431
  
  https://www.illumos.org/issues/10592
    This is a collection of recent fixes from ZoL:
    8eef997679b Error path in metaslab_load_impl() forgets to drop ms_sync_lock
    928e8ad47d3 Introduce auxiliary metaslab histograms
    425d3237ee8 Get rid of space_map_update() for ms_synced_length
    6c926f426a2 Simplify log vdev removal code
    21e7cf5da89 zdb -L should skip leak detection altogether
    df72b8bebe0 Rename range_tree_verify to range_tree_verify_not_present
    75058f33034 Remove unused vdev_t fields
  
  Portions contributed by: Jerry Jelinek 
  Author: Serapheim Dimitropoulos 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_initialize.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c

Changes in other areas also in this revision:
Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c
  vendor/illumos/dist/man/man1m/zdb.1m

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c	Wed Nov  6 09:00:06 2019	(r354383)
@@ -489,45 +489,62 @@ metaslab_compare(const void *x1, const void *x2)
 	return (AVL_CMP(m1->ms_start, m2->ms_start));
 }
 
+uint64_t
+metaslab_allocated_space(metaslab_t *msp)
+{
+	return (msp->ms_allocated_space);
+}
+
 /*
  * Verify that the space accounting on disk matches the in-core range_trees.
  */
-void
+static void
 metaslab_verify_space(metaslab_t *msp, uint64_t txg)
 {
 	spa_t *spa = msp->ms_group->mg_vd->vdev_spa;
-	uint64_t allocated = 0;
+	uint64_t allocating = 0;
 	uint64_t sm_free_space, msp_free_space;
 
 	ASSERT(MUTEX_HELD(&msp->ms_lock));
+	ASSERT(!msp->ms_condensing);
 
 	if ((zfs_flags & ZFS_DEBUG_METASLAB_VERIFY) == 0)
 		return;
 
 	/*
 	 * We can only verify the metaslab space when we're called
-	 * from syncing context with a loaded metaslab that has an allocated
-	 * space map. Calling this in non-syncing context does not
-	 * provide a consistent view of the metaslab since we're performing
-	 * allocations in the future.
+	 * from syncing context with a loaded metaslab that has an
+	 * allocated space map. Calling this in non-syncing context
+	 * does not provide a consistent view of the metaslab since
+	 * we're performing allocations in the future.
 	 */
 	if (txg != spa_syncing_txg(spa) || msp->ms_sm == NULL ||
 	    !msp->ms_loaded)
 		return;
 
-	sm_free_space = msp->ms_size - space_map_allocated(msp->ms_sm) -
-	    space_map_alloc_delta(msp->ms_sm);
+	/*
+	 * Even though the smp_alloc field can get negative (e.g.
+	 * see vdev_checkpoint_sm), that should never be the case
+	 * when it come's to a metaslab's space map.
+	 */
+	ASSERT3S(space_map_allocated(msp->ms_sm), >=, 0);
 
+	sm_free_space = msp->ms_size - metaslab_allocated_space(msp);
+
 	/*
-	 * Account for future allocations since we would have already
-	 * deducted that space from the ms_freetree.
+	 * Account for future allocations since we would have
+	 * already deducted that space from the ms_allocatable.
 	 */
 	for (int t = 0; t < TXG_CONCURRENT_STATES; t++) {
-		allocated +=
+		allocating +=
 		    range_tree_space(msp->ms_allocating[(txg + t) & TXG_MASK]);
 	}
 
-	msp_free_space = range_tree_space(msp->ms_allocatable) + allocated +
+	ASSERT3U(msp->ms_deferspace, ==,
+	    range_tree_space(msp->ms_defer[0]) +
+	    range_tree_space(msp->ms_defer[1]));
+
+	msp_free_space = range_tree_space(msp->ms_allocatable) + allocating +
 	    msp->ms_deferspace + range_tree_space(msp->ms_freed);
 
 	VERIFY3U(sm_free_space, ==, msp_free_space);
@@ -832,6 +849,7 @@ metaslab_group_histogram_verify(metaslab_group_t *mg)
 
 	for (int m = 0; m < vd->vdev_ms_count; m++) {
 		metaslab_t *msp = vd->vdev_ms[m];
+		ASSERT(msp != NULL);
 
 		/* skip if not active or not a member */
 		if (msp->ms_sm == NULL || msp->ms_group != mg)
@@ -1445,7 +1463,204 @@ metaslab_ops_t *zfs_metaslab_ops = &metaslab_df_ops;
  * ==========================================================================
  */
 
+static void
+metaslab_aux_histograms_clear(metaslab_t *msp)
+{
+	/*
+	 * Auxiliary histograms are only cleared when resetting them,
+	 * which can only happen while the metaslab is loaded.
+	 */
+	ASSERT(msp->ms_loaded);
+
+	bzero(msp->ms_synchist, sizeof (msp->ms_synchist));
+	for (int t = 0; t < TXG_DEFER_SIZE; t++)
+		bzero(msp->ms_deferhist[t], sizeof (msp->ms_deferhist[t]));
+}
+
+static void
+metaslab_aux_histogram_add(uint64_t *histogram, uint64_t shift,
+    range_tree_t *rt)
+{
+	/*
+	 * This is modeled after space_map_histogram_add(), so refer to that
+	 * function for implementation details. We want this to work like
+	 * the space map histogram, and not the range tree histogram, as we
+	 * are essentially constructing a delta that will be later subtracted
+	 * from the space map histogram.
+	 */
+	int idx = 0;
+	for (int i = shift; i < RANGE_TREE_HISTOGRAM_SIZE; i++) {
+		ASSERT3U(i, >=, idx + shift);
+		histogram[idx] += rt->rt_histogram[i] << (i - idx - shift);
+
+		if (idx < SPACE_MAP_HISTOGRAM_SIZE - 1) {
+			ASSERT3U(idx + shift, ==, i);
+			idx++;
+			ASSERT3U(idx, <, SPACE_MAP_HISTOGRAM_SIZE);
+		}
+	}
+}
+
 /*
+ * Called at every sync pass that the metaslab gets synced.
+ *
+ * The reason is that we want our auxiliary histograms to be updated
+ * wherever the metaslab's space map histogram is updated. This way
+ * we stay consistent on which parts of the metaslab space map's
+ * histogram are currently not available for allocations (e.g because
+ * they are in the defer, freed, and freeing trees).
+ */
+static void
+metaslab_aux_histograms_update(metaslab_t *msp)
+{
+	space_map_t *sm = msp->ms_sm;
+	ASSERT(sm != NULL);
+
+	/*
+	 * This is similar to the metaslab's space map histogram updates
+	 * that take place in metaslab_sync(). The only difference is that
+	 * we only care about segments that haven't made it into the
+	 * ms_allocatable tree yet.
+	 */
+	if (msp->ms_loaded) {
+		metaslab_aux_histograms_clear(msp);
+
+		metaslab_aux_histogram_add(msp->ms_synchist,
+		    sm->sm_shift, msp->ms_freed);
+
+		for (int t = 0; t < TXG_DEFER_SIZE; t++) {
+			metaslab_aux_histogram_add(msp->ms_deferhist[t],
+			    sm->sm_shift, msp->ms_defer[t]);
+		}
+	}
+
+	metaslab_aux_histogram_add(msp->ms_synchist,
+	    sm->sm_shift, msp->ms_freeing);
+}
+
+/*
+ * Called every time we are done syncing (writing to) the metaslab,
+ * i.e. at the end of each sync pass.
+ * [see the comment in metaslab_impl.h for ms_synchist, ms_deferhist]
+ */
+static void
+metaslab_aux_histograms_update_done(metaslab_t *msp, boolean_t defer_allowed)
+{
+	spa_t *spa = msp->ms_group->mg_vd->vdev_spa;
+	space_map_t *sm = msp->ms_sm;
+
+	if (sm == NULL) {
+		/*
+		 * We came here from metaslab_init() when creating/opening a
+		 * pool, looking at a metaslab that hasn't had any allocations
+		 * yet.
+		 */
+		return;
+	}
+
+	/*
+	 * This is similar to the actions that we take for the ms_freed
+	 * and ms_defer trees in metaslab_sync_done().
+	 */
+	uint64_t hist_index = spa_syncing_txg(spa) % TXG_DEFER_SIZE;
+	if (defer_allowed) {
+		bcopy(msp->ms_synchist, msp->ms_deferhist[hist_index],
+		    sizeof (msp->ms_synchist));
+	} else {
+		bzero(msp->ms_deferhist[hist_index],
+		    sizeof (msp->ms_deferhist[hist_index]));
+	}
+	bzero(msp->ms_synchist, sizeof (msp->ms_synchist));
+}
+
+/*
+ * Ensure that the metaslab's weight and fragmentation are consistent
+ * with the contents of the histogram (either the range tree's histogram
+ * or the space map's depending whether the metaslab is loaded).
+ */
+static void
+metaslab_verify_weight_and_frag(metaslab_t *msp)
+{
+	ASSERT(MUTEX_HELD(&msp->ms_lock));
+
+	if ((zfs_flags & ZFS_DEBUG_METASLAB_VERIFY) == 0)
+		return;
+
+	/* see comment in metaslab_verify_unflushed_changes() */
+	if (msp->ms_group == NULL)
+		return;
+
+	/*
+	 * Devices being removed always return a weight of 0 and leave
+	 * fragmentation and ms_max_size as is - there is nothing for
+	 * us to verify here.
+	 */
+	vdev_t *vd = msp->ms_group->mg_vd;
+	if (vd->vdev_removing)
+		return;
+
+	/*
+	 * If the metaslab is dirty it probably means that we've done
+	 * some allocations or frees that have changed our histograms
+	 * and thus the weight.
+	 */
+	for (int t = 0; t < TXG_SIZE; t++) {
+		if (txg_list_member(&vd->vdev_ms_list, msp, t))
+			return;
+	}
+
+	/*
+	 * This verification checks that our in-memory state is consistent
+	 * with what's on disk. If the pool is read-only then there aren't
+	 * any changes and we just have the initially-loaded state.
+	 */
+	if (!spa_writeable(msp->ms_group->mg_vd->vdev_spa))
+		return;
+
+	/* some extra verification for in-core tree if you can */
+	if (msp->ms_loaded) {
+		range_tree_stat_verify(msp->ms_allocatable);
+		VERIFY(space_map_histogram_verify(msp->ms_sm,
+		    msp->ms_allocatable));
+	}
+
+	uint64_t weight = msp->ms_weight;
+	uint64_t was_active = msp->ms_weight & METASLAB_ACTIVE_MASK;
+	boolean_t space_based = WEIGHT_IS_SPACEBASED(msp->ms_weight);
+	uint64_t frag = msp->ms_fragmentation;
+	uint64_t max_segsize = msp->ms_max_size;
+
+	msp->ms_weight = 0;
+	msp->ms_fragmentation = 0;
+	msp->ms_max_size = 0;
+
+	/*
+	 * This function is used for verification purposes. Regardless of
+	 * whether metaslab_weight() thinks this metaslab should be active or
+	 * not, we want to ensure that the actual weight (and therefore the
+	 * value of ms_weight) would be the same if it was to be recalculated
+	 * at this point.
+	 */
+	msp->ms_weight = metaslab_weight(msp) | was_active;
+
+	VERIFY3U(max_segsize, ==, msp->ms_max_size);
+
+	/*
+	 * If the weight type changed then there is no point in doing
+	 * verification. Revert fields to their original values.
+	 */
+	if ((space_based && !WEIGHT_IS_SPACEBASED(msp->ms_weight)) ||
+	    (!space_based && WEIGHT_IS_SPACEBASED(msp->ms_weight))) {
+		msp->ms_fragmentation = frag;
+		msp->ms_weight = weight;
+		return;
+	}
+
+	VERIFY3U(msp->ms_fragmentation, ==, frag);
+	VERIFY3U(msp->ms_weight, ==, weight);
+}
+
+/*
  * Wait for any in-progress metaslab loads to complete.
  */
 static void
@@ -1466,47 +1681,94 @@ metaslab_load_impl(metaslab_t *msp)
 
 	ASSERT(MUTEX_HELD(&msp->ms_lock));
 	ASSERT(msp->ms_loading);
+	ASSERT(!msp->ms_condensing);
 
 	/*
-	 * Nobody else can manipulate a loading metaslab, so it's now safe
-	 * to drop the lock. This way we don't have to hold the lock while
-	 * reading the spacemap from disk.
+	 * We temporarily drop the lock to unblock other operations while we
+	 * are reading the space map. Therefore, metaslab_sync() and
+	 * metaslab_sync_done() can run at the same time as we do.
+	 *
+	 * metaslab_sync() can append to the space map while we are loading.
+	 * Therefore we load only entries that existed when we started the
+	 * load. Additionally, metaslab_sync_done() has to wait for the load
+	 * to complete because there are potential races like metaslab_load()
+	 * loading parts of the space map that are currently being appended
+	 * by metaslab_sync(). If we didn't, the ms_allocatable would have
+	 * entries that metaslab_sync_done() would try to re-add later.
+	 *
+	 * That's why before dropping the lock we remember the synced length
+	 * of the metaslab and read up to that point of the space map,
+	 * ignoring entries appended by metaslab_sync() that happen after we
+	 * drop the lock.
 	 */
+	uint64_t length = msp->ms_synced_length;
 	mutex_exit(&msp->ms_lock);
 
-	/*
-	 * If the space map has not been allocated yet, then treat
-	 * all the space in the metaslab as free and add it to ms_allocatable.
-	 */
 	if (msp->ms_sm != NULL) {
-		error = space_map_load(msp->ms_sm, msp->ms_allocatable,
-		    SM_FREE);
+		error = space_map_load_length(msp->ms_sm, msp->ms_allocatable,
+		    SM_FREE, length);
 	} else {
+		/*
+		 * The space map has not been allocated yet, so treat
+		 * all the space in the metaslab as free and add it to the
+		 * ms_allocatable tree.
+		 */
 		range_tree_add(msp->ms_allocatable,
 		    msp->ms_start, msp->ms_size);
 	}
 
+	/*
+	 * We need to grab the ms_sync_lock to prevent metaslab_sync() from
+	 * changing the ms_sm and the metaslab's range trees while we are
+	 * about to use them and populate the ms_allocatable. The ms_lock
+	 * is insufficient for this because metaslab_sync() doesn't hold
+	 * the ms_lock while writing the ms_checkpointing tree to disk.
+	 */
+	mutex_enter(&msp->ms_sync_lock);
 	mutex_enter(&msp->ms_lock);
+	ASSERT(!msp->ms_condensing);
 
-	if (error != 0)
+	if (error != 0) {
+		mutex_exit(&msp->ms_sync_lock);
 		return (error);
+	}
 
 	ASSERT3P(msp->ms_group, !=, NULL);
 	msp->ms_loaded = B_TRUE;
 
 	/*
-	 * If the metaslab already has a spacemap, then we need to
-	 * remove all segments from the defer tree; otherwise, the
-	 * metaslab is completely empty and we can skip this.
+	 * The ms_allocatable contains the segments that exist in the
+	 * ms_defer trees [see ms_synced_length]. Thus we need to remove
+	 * them from ms_allocatable as they will be added again in
+	 * metaslab_sync_done().
 	 */
-	if (msp->ms_sm != NULL) {
-		for (int t = 0; t < TXG_DEFER_SIZE; t++) {
-			range_tree_walk(msp->ms_defer[t],
-			    range_tree_remove, msp->ms_allocatable);
-		}
+	for (int t = 0; t < TXG_DEFER_SIZE; t++) {
+		range_tree_walk(msp->ms_defer[t],
+		    range_tree_remove, msp->ms_allocatable);
 	}
+
+	/*
+	 * Call metaslab_recalculate_weight_and_sort() now that the
+	 * metaslab is loaded so we get the metaslab's real weight.
+	 *
+	 * Unless this metaslab was created with older software and
+	 * has not yet been converted to use segment-based weight, we
+	 * expect the new weight to be better or equal to the weight
+	 * that the metaslab had while it was not loaded. This is
+	 * because the old weight does not take into account the
+	 * consolidation of adjacent segments between TXGs. [see
+	 * comment for ms_synchist and ms_deferhist[] for more info]
+	 */
+	uint64_t weight = msp->ms_weight;
+	metaslab_recalculate_weight_and_sort(msp);
+	if (!WEIGHT_IS_SPACEBASED(weight))
+		ASSERT3U(weight, <=, msp->ms_weight);
 	msp->ms_max_size = metaslab_block_maxsize(msp);
 
+	spa_t *spa = msp->ms_group->mg_vd->vdev_spa;
+	metaslab_verify_space(msp, spa_syncing_txg(spa));
+	mutex_exit(&msp->ms_sync_lock);
+
 	return (0);
 }
 
@@ -1523,6 +1785,7 @@ metaslab_load(metaslab_t *msp)
 	if (msp->ms_loaded)
 		return (0);
 	VERIFY(!msp->ms_loading);
+	ASSERT(!msp->ms_condensing);
 
 	msp->ms_loading = B_TRUE;
 	int error = metaslab_load_impl(msp);
@@ -1536,10 +1799,29 @@ void
 metaslab_unload(metaslab_t *msp)
 {
 	ASSERT(MUTEX_HELD(&msp->ms_lock));
+
+	metaslab_verify_weight_and_frag(msp);
+
 	range_tree_vacate(msp->ms_allocatable, NULL, NULL);
 	msp->ms_loaded = B_FALSE;
+
 	msp->ms_weight &= ~METASLAB_ACTIVE_MASK;
 	msp->ms_max_size = 0;
+
+	/*
+	 * We explicitly recalculate the metaslab's weight based on its space
+	 * map (as it is now not loaded). We want unload metaslabs to always
+	 * have their weights calculated from the space map histograms, while
+	 * loaded ones have it calculated from their in-core range tree
+	 * [see metaslab_load()]. This way, the weight reflects the information
+	 * available in-core, whether it is loaded or not
+	 *
+	 * If ms_group == NULL means that we came here from metaslab_fini(),
+	 * at which point it doesn't make sense for us to do the recalculation
+	 * and the sorting.
+	 */
+	if (msp->ms_group != NULL)
+		metaslab_recalculate_weight_and_sort(msp);
 }
 
 static void
@@ -1579,6 +1861,13 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6
 	/*
 	 * We only open space map objects that already exist. All others
 	 * will be opened when we finally allocate an object for it.
+	 *
+	 * Note:
+	 * When called from vdev_expand(), we can't call into the DMU as
+	 * we are holding the spa_config_lock as a writer and we would
+	 * deadlock [see relevant comment in vdev_metaslab_init()]. in
+	 * that case, the object parameter is zero though, so we won't
+	 * call into the DMU.
 	 */
 	if (object != 0) {
 		error = space_map_open(&ms->ms_sm, mos, object, ms->ms_start,
@@ -1590,14 +1879,17 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6
 		}
 
 		ASSERT(ms->ms_sm != NULL);
+		ASSERT3S(space_map_allocated(ms->ms_sm), >=, 0);
+		ms->ms_allocated_space = space_map_allocated(ms->ms_sm);
 	}
 
 	/*
-	 * We create the main range tree here, but we don't create the
+	 * We create the ms_allocatable here, but we don't create the
 	 * other range trees until metaslab_sync_done().  This serves
 	 * two purposes: it allows metaslab_sync_done() to detect the
-	 * addition of new space; and for debugging, it ensures that we'd
-	 * data fault on any attempt to use this metaslab before it's ready.
+	 * addition of new space; and for debugging, it ensures that
+	 * we'd data fault on any attempt to use this metaslab before
+	 * it's ready.
 	 */
 	ms->ms_allocatable = range_tree_create(&metaslab_rt_ops, ms);
 	metaslab_group_add(mg, ms);
@@ -1613,8 +1905,11 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6
 	 * out this txg. This ensures that we don't attempt to allocate
 	 * from it before we have initialized it completely.
 	 */
-	if (txg <= TXG_INITIAL)
+	if (txg <= TXG_INITIAL) {
 		metaslab_sync_done(ms, 0);
+		metaslab_space_update(vd, mg->mg_class,
+		    metaslab_allocated_space(ms), 0, 0);
+	}
 
 	/*
 	 * If metaslab_debug_load is set and we're initializing a metaslab
@@ -1648,7 +1943,7 @@ metaslab_fini(metaslab_t *msp)
 	mutex_enter(&msp->ms_lock);
 	VERIFY(msp->ms_group == NULL);
 	metaslab_space_update(vd, mg->mg_class,
-	    -space_map_allocated(msp->ms_sm), 0, -msp->ms_size);
+	    -metaslab_allocated_space(msp), 0, -msp->ms_size);
 
 	space_map_close(msp->ms_sm);
 
@@ -1669,6 +1964,9 @@ metaslab_fini(metaslab_t *msp)
 
 	range_tree_destroy(msp->ms_checkpointing);
 
+	for (int t = 0; t < TXG_SIZE; t++)
+		ASSERT(!txg_list_member(&vd->vdev_ms_list, msp, t));
+
 	mutex_exit(&msp->ms_lock);
 	cv_destroy(&msp->ms_load_cv);
 	mutex_destroy(&msp->ms_lock);
@@ -1684,7 +1982,7 @@ metaslab_fini(metaslab_t *msp)
  * This table defines a segment size based fragmentation metric that will
  * allow each metaslab to derive its own fragmentation value. This is done
  * by calculating the space in each bucket of the spacemap histogram and
- * multiplying that by the fragmetation metric in this table. Doing
+ * multiplying that by the fragmentation metric in this table. Doing
  * this for all buckets and dividing it by the total amount of free
  * space in this metaslab (i.e. the total free space in all buckets) gives
  * us the fragmentation metric. This means that a high fragmentation metric
@@ -1719,10 +2017,10 @@ int zfs_frag_table[FRAGMENTATION_TABLE_SIZE] = {
 };
 
 /*
- * Calclate the metaslab's fragmentation metric. A return value
- * of ZFS_FRAG_INVALID means that the metaslab has not been upgraded and does
- * not support this metric. Otherwise, the return value should be in the
- * range [0, 100].
+ * Calculate the metaslab's fragmentation metric and set ms_fragmentation.
+ * Setting this value to ZFS_FRAG_INVALID means that the metaslab has not
+ * been upgraded and does not support this metric. Otherwise, the return
+ * value should be in the range [0, 100].
  */
 static void
 metaslab_set_fragmentation(metaslab_t *msp)
@@ -1815,7 +2113,7 @@ metaslab_space_weight(metaslab_t *msp)
 	/*
 	 * The baseline weight is the metaslab's free space.
 	 */
-	space = msp->ms_size - space_map_allocated(msp->ms_sm);
+	space = msp->ms_size - metaslab_allocated_space(msp);
 
 	if (metaslab_fragmentation_factor_enabled &&
 	    msp->ms_fragmentation != ZFS_FRAG_INVALID) {
@@ -1919,14 +2217,38 @@ metaslab_weight_from_range_tree(metaslab_t *msp)
 static uint64_t
 metaslab_weight_from_spacemap(metaslab_t *msp)
 {
-	uint64_t weight = 0;
+	space_map_t *sm = msp->ms_sm;
+	ASSERT(!msp->ms_loaded);
+	ASSERT(sm != NULL);
+	ASSERT3U(space_map_object(sm), !=, 0);
+	ASSERT3U(sm->sm_dbuf->db_size, ==, sizeof (space_map_phys_t));
 
+	/*
+	 * Create a joint histogram from all the segments that have made
+	 * it to the metaslab's space map histogram, that are not yet
+	 * available for allocation because they are still in the freeing
+	 * pipeline (e.g. freeing, freed, and defer trees). Then subtract
+	 * these segments from the space map's histogram to get a more
+	 * accurate weight.
+	 */
+	uint64_t deferspace_histogram[SPACE_MAP_HISTOGRAM_SIZE] = {0};
+	for (int i = 0; i < SPACE_MAP_HISTOGRAM_SIZE; i++)
+		deferspace_histogram[i] += msp->ms_synchist[i];
+	for (int t = 0; t < TXG_DEFER_SIZE; t++) {
+		for (int i = 0; i < SPACE_MAP_HISTOGRAM_SIZE; i++) {
+			deferspace_histogram[i] += msp->ms_deferhist[t][i];
+		}
+	}
+
+	uint64_t weight = 0;
 	for (int i = SPACE_MAP_HISTOGRAM_SIZE - 1; i >= 0; i--) {
-		if (msp->ms_sm->sm_phys->smp_histogram[i] != 0) {
-			WEIGHT_SET_COUNT(weight,
-			    msp->ms_sm->sm_phys->smp_histogram[i]);
-			WEIGHT_SET_INDEX(weight, i +
-			    msp->ms_sm->sm_shift);
+		ASSERT3U(sm->sm_phys->smp_histogram[i], >=,
+		    deferspace_histogram[i]);
+		uint64_t count =
+		    sm->sm_phys->smp_histogram[i] - deferspace_histogram[i];
+		if (count != 0) {
+			WEIGHT_SET_COUNT(weight, count);
+			WEIGHT_SET_INDEX(weight, i + sm->sm_shift);
 			WEIGHT_SET_ACTIVE(weight, 0);
 			break;
 		}
@@ -1951,7 +2273,7 @@ metaslab_segment_weight(metaslab_t *msp)
 	/*
 	 * The metaslab is completely free.
 	 */
-	if (space_map_allocated(msp->ms_sm) == 0) {
+	if (metaslab_allocated_space(msp) == 0) {
 		int idx = highbit64(msp->ms_size) - 1;
 		int max_idx = SPACE_MAP_HISTOGRAM_SIZE + shift - 1;
 
@@ -1973,7 +2295,7 @@ metaslab_segment_weight(metaslab_t *msp)
 	/*
 	 * If the metaslab is fully allocated then just make the weight 0.
 	 */
-	if (space_map_allocated(msp->ms_sm) == msp->ms_size)
+	if (metaslab_allocated_space(msp) == msp->ms_size)
 		return (0);
 	/*
 	 * If the metaslab is already loaded, then use the range tree to
@@ -2054,6 +2376,8 @@ metaslab_weight(metaslab_t *msp)
 	 */
 	if (msp->ms_loaded)
 		msp->ms_max_size = metaslab_block_maxsize(msp);
+	else
+		ASSERT0(msp->ms_max_size);
 
 	/*
 	 * Segment-based weighting requires space map histogram support.
@@ -2069,6 +2393,15 @@ metaslab_weight(metaslab_t *msp)
 	return (weight);
 }
 
+void
+metaslab_recalculate_weight_and_sort(metaslab_t *msp)
+{
+	/* note: we preserve the mask (e.g. indication of primary, etc..) */
+	uint64_t was_active = msp->ms_weight & METASLAB_ACTIVE_MASK;
+	metaslab_group_sort(msp->ms_group, msp,
+	    metaslab_weight(msp) | was_active);
+}
+
 static int
 metaslab_activate_allocator(metaslab_group_t *mg, metaslab_t *msp,
     int allocator, uint64_t activation_weight)
@@ -2453,17 +2786,17 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 	VERIFY(txg <= spa_final_dirty_txg(spa));
 
 	/*
-	 * The only state that can actually be changing concurrently with
-	 * metaslab_sync() is the metaslab's ms_allocatable.  No other
-	 * thread can be modifying this txg's alloc, freeing,
+	 * The only state that can actually be changing concurrently
+	 * with metaslab_sync() is the metaslab's ms_allocatable. No
+	 * other thread can be modifying this txg's alloc, freeing,
 	 * freed, or space_map_phys_t.  We drop ms_lock whenever we
-	 * could call into the DMU, because the DMU can call down to us
-	 * (e.g. via zio_free()) at any time.
+	 * could call into the DMU, because the DMU can call down to
+	 * us (e.g. via zio_free()) at any time.
 	 *
 	 * The spa_vdev_remove_thread() can be reading metaslab state
-	 * concurrently, and it is locked out by the ms_sync_lock.  Note
-	 * that the ms_lock is insufficient for this, because it is dropped
-	 * by space_map_write().
+	 * concurrently, and it is locked out by the ms_sync_lock.
+	 * Note that the ms_lock is insufficient for this, because it
+	 * is dropped by space_map_write().
 	 */
 	tx = dmu_tx_create_assigned(spa_get_dsl(spa), txg);
 
@@ -2475,7 +2808,9 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 
 		VERIFY0(space_map_open(&msp->ms_sm, mos, new_object,
 		    msp->ms_start, msp->ms_size, vd->vdev_ashift));
+
 		ASSERT(msp->ms_sm != NULL);
+		ASSERT0(metaslab_allocated_space(msp));
 	}
 
 	if (!range_tree_is_empty(msp->ms_checkpointing) &&
@@ -2523,6 +2858,11 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 		mutex_enter(&msp->ms_lock);
 	}
 
+	msp->ms_allocated_space += range_tree_space(alloctree);
+	ASSERT3U(msp->ms_allocated_space, >=,
+	    range_tree_space(msp->ms_freeing));
+	msp->ms_allocated_space -= range_tree_space(msp->ms_freeing);
+
 	if (!range_tree_is_empty(msp->ms_checkpointing)) {
 		ASSERT(spa_has_checkpoint(spa));
 		ASSERT3P(vd->vdev_checkpoint_sm, !=, NULL);
@@ -2536,14 +2876,13 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 		space_map_write(vd->vdev_checkpoint_sm,
 		    msp->ms_checkpointing, SM_FREE, SM_NO_VDEVID, tx);
 		mutex_enter(&msp->ms_lock);
-		space_map_update(vd->vdev_checkpoint_sm);
 
 		spa->spa_checkpoint_info.sci_dspace +=
 		    range_tree_space(msp->ms_checkpointing);
 		vd->vdev_stat.vs_checkpoint_space +=
 		    range_tree_space(msp->ms_checkpointing);
 		ASSERT3U(vd->vdev_stat.vs_checkpoint_space, ==,
-		    -vd->vdev_checkpoint_sm->sm_alloc);
+		    -space_map_allocated(vd->vdev_checkpoint_sm));
 
 		range_tree_vacate(msp->ms_checkpointing, NULL, NULL);
 	}
@@ -2588,6 +2927,7 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 	 * time we load the space map.
 	 */
 	space_map_histogram_add(msp->ms_sm, msp->ms_freeing, tx);
+	metaslab_aux_histograms_update(msp);
 
 	metaslab_group_histogram_add(mg, msp);
 	metaslab_group_histogram_verify(mg);
@@ -2595,16 +2935,18 @@ metaslab_sync(metaslab_t *msp, uint64_t txg)
 
 	/*
 	 * For sync pass 1, we avoid traversing this txg's free range tree
-	 * and instead will just swap the pointers for freeing and
-	 * freed. We can safely do this since the freed_tree is
-	 * guaranteed to be empty on the initial pass.
+	 * and instead will just swap the pointers for freeing and freed.
+	 * We can safely do this since the freed_tree is guaranteed to be
+	 * empty on the initial pass.
 	 */
 	if (spa_sync_pass(spa) == 1) {
 		range_tree_swap(&msp->ms_freeing, &msp->ms_freed);
+		ASSERT0(msp->ms_allocated_this_txg);
 	} else {
 		range_tree_vacate(msp->ms_freeing,
 		    range_tree_add, msp->ms_freed);
 	}
+	msp->ms_allocated_this_txg += range_tree_space(alloctree);
 	range_tree_vacate(alloctree, NULL, NULL);
 
 	ASSERT0(range_tree_space(msp->ms_allocating[txg & TXG_MASK]));
@@ -2682,7 +3024,8 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 	}
 
 	defer_delta = 0;
-	alloc_delta = space_map_alloc_delta(msp->ms_sm);
+	alloc_delta = msp->ms_allocated_this_txg -
+	    range_tree_space(msp->ms_freed);
 	if (defer_allowed) {
 		defer_delta = range_tree_space(msp->ms_freed) -
 		    range_tree_space(*defer_tree);
@@ -2714,8 +3057,9 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 		    msp->ms_loaded ? range_tree_add : NULL,
 		    msp->ms_allocatable);
 	}
-	space_map_update(msp->ms_sm);
 
+	msp->ms_synced_length = space_map_length(msp->ms_sm);
+
 	msp->ms_deferspace += defer_delta;
 	ASSERT3S(msp->ms_deferspace, >=, 0);
 	ASSERT3S(msp->ms_deferspace, <=, msp->ms_size);
@@ -2726,6 +3070,7 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 		 */
 		vdev_dirty(vd, VDD_METASLAB, msp, txg + 1);
 	}
+	metaslab_aux_histograms_update_done(msp, defer_allowed);
 
 	if (msp->ms_new) {
 		msp->ms_new = B_FALSE;
@@ -2733,12 +3078,12 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 		mg->mg_ms_ready++;
 		mutex_exit(&mg->mg_lock);
 	}
+
 	/*
-	 * Calculate the new weights before unloading any metaslabs.
-	 * This will give us the most accurate weighting.
+	 * Re-sort metaslab within its group now that we've adjusted
+	 * its allocatable space.
 	 */
-	metaslab_group_sort(mg, msp, metaslab_weight(msp) |
-	    (msp->ms_weight & METASLAB_ACTIVE_MASK));
+	metaslab_recalculate_weight_and_sort(msp);
 
 	/*
 	 * If the metaslab is loaded and we've not tried to load or allocate
@@ -2765,6 +3110,7 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg)
 	ASSERT0(range_tree_space(msp->ms_freed));
 	ASSERT0(range_tree_space(msp->ms_checkpointing));
 
+	msp->ms_allocated_this_txg = 0;
 	mutex_exit(&msp->ms_lock);
 }
 
@@ -4020,7 +4366,7 @@ metaslab_alloc(spa_t *spa, metaslab_class_t *mc, uint6
     zio_alloc_list_t *zal, zio_t *zio, int allocator)
 {
 	dva_t *dva = bp->blk_dva;
-	dva_t *hintdva = hintbp->blk_dva;
+	dva_t *hintdva = (hintbp != NULL) ? hintbp->blk_dva : NULL;
 	int error = 0;
 
 	ASSERT(bp->blk_birth == 0);
@@ -4187,14 +4533,16 @@ metaslab_check_free_impl(vdev_t *vd, uint64_t offset, 
 	msp = vd->vdev_ms[offset >> vd->vdev_ms_shift];
 
 	mutex_enter(&msp->ms_lock);
-	if (msp->ms_loaded)
-		range_tree_verify(msp->ms_allocatable, offset, size);
+	if (msp->ms_loaded) {
+		range_tree_verify_not_present(msp->ms_allocatable,
+		    offset, size);
+	}
 
-	range_tree_verify(msp->ms_freeing, offset, size);
-	range_tree_verify(msp->ms_checkpointing, offset, size);
-	range_tree_verify(msp->ms_freed, offset, size);
+	range_tree_verify_not_present(msp->ms_freeing, offset, size);
+	range_tree_verify_not_present(msp->ms_checkpointing, offset, size);
+	range_tree_verify_not_present(msp->ms_freed, offset, size);
 	for (int j = 0; j < TXG_DEFER_SIZE; j++)
-		range_tree_verify(msp->ms_defer[j], offset, size);
+		range_tree_verify_not_present(msp->ms_defer[j], offset, size);
 	mutex_exit(&msp->ms_lock);
 }
 

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c	Wed Nov  6 09:00:06 2019	(r354383)
@@ -311,13 +311,11 @@ range_tree_find(range_tree_t *rt, uint64_t start, uint
 }
 
 void
-range_tree_verify(range_tree_t *rt, uint64_t off, uint64_t size)
+range_tree_verify_not_present(range_tree_t *rt, uint64_t off, uint64_t size)
 {
-	range_seg_t *rs;
-
-	rs = range_tree_find(rt, off, size);
+	range_seg_t *rs = range_tree_find(rt, off, size);
 	if (rs != NULL)
-		panic("freeing free block; rs=%p", (void *)rs);
+		panic("segment already in tree; rs=%p", (void *)rs);
 }
 
 boolean_t

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c	Wed Nov  6 09:00:06 2019	(r354383)
@@ -129,7 +129,7 @@
  *   uberblock would reference data in the removed device. For this reason
  *   and others of similar nature, we disallow the following operations that
  *   can change the config:
- *   	vdev removal and attach/detach, mirror splitting, and pool reguid.
+ *	vdev removal and attach/detach, mirror splitting, and pool reguid.
  *
  * - As most of the checkpoint logic is implemented in the SPA and doesn't
  *   distinguish datasets when it comes to space accounting, having a
@@ -262,7 +262,7 @@ spa_checkpoint_accounting_verify(spa_t *spa)
 
 		if (vd->vdev_checkpoint_sm != NULL) {
 			ckpoint_sm_space_sum +=
-			    -vd->vdev_checkpoint_sm->sm_alloc;
+			    -space_map_allocated(vd->vdev_checkpoint_sm);
 			vs_ckpoint_space_sum +=
 			    vd->vdev_stat.vs_checkpoint_space;
 			ASSERT3U(ckpoint_sm_space_sum, ==,
@@ -347,7 +347,7 @@ spa_checkpoint_discard_thread_sync(void *arg, dmu_tx_t
 			    error, vd->vdev_id);
 		}
 		ASSERT0(words_after);
-		ASSERT0(vd->vdev_checkpoint_sm->sm_alloc);
+		ASSERT0(space_map_allocated(vd->vdev_checkpoint_sm));
 		ASSERT0(space_map_length(vd->vdev_checkpoint_sm));
 
 		space_map_free(vd->vdev_checkpoint_sm, tx);

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c	Wed Nov  6 08:58:03 2019	(r354382)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c	Wed Nov  6 09:00:06 2019	(r354383)
@@ -23,7 +23,7 @@
  * Use is subject to license terms.
  */
 /*
- * Copyright (c) 2012, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2012, 2018 by Delphix. All rights reserved.
  */
 
 #include 
@@ -81,20 +81,22 @@ sm_entry_is_double_word(uint64_t e)
 
 /*
  * Iterate through the space map, invoking the callback on each (non-debug)
- * space map entry.
+ * space map entry. Stop after reading 'end' bytes of the space map.
  */
 int
-space_map_iterate(space_map_t *sm, sm_cb_t callback, void *arg)
+space_map_iterate(space_map_t *sm, uint64_t end, sm_cb_t callback, void *arg)
 {
-	uint64_t sm_len = space_map_length(sm);
-	ASSERT3U(sm->sm_blksz, !=, 0);
+	uint64_t blksz = sm->sm_blksz;
 
-	dmu_prefetch(sm->sm_os, space_map_object(sm), 0, 0, sm_len,
+	ASSERT3U(blksz, !=, 0);
+	ASSERT3U(end, <=, space_map_length(sm));
+	ASSERT0(P2PHASE(end, sizeof (uint64_t)));
+
+	dmu_prefetch(sm->sm_os, space_map_object(sm), 0, 0, end,
 	    ZIO_PRIORITY_SYNC_READ);
 
-	uint64_t blksz = sm->sm_blksz;
 	int error = 0;
-	for (uint64_t block_base = 0; block_base < sm_len && error == 0;
+	for (uint64_t block_base = 0; block_base < end && error == 0;
 	    block_base += blksz) {
 		dmu_buf_t *db;
 		error = dmu_buf_hold(sm->sm_os, space_map_object(sm),
@@ -103,7 +105,7 @@ space_map_iterate(space_map_t *sm, sm_cb_t callback, v
 			return (error);
 
 		uint64_t *block_start = db->db_data;
-		uint64_t block_length = MIN(sm_len - block_base, blksz);
+		uint64_t block_length = MIN(end - block_base, blksz);
 		uint64_t *block_end = block_start +
 		    (block_length / sizeof (uint64_t));
 
@@ -186,7 +188,7 @@ space_map_reversed_last_block_entries(space_map_t *sm,
 	 * dmu_buf_hold().
 	 */
 	uint64_t last_word_offset =
-	    sm->sm_phys->smp_objsize - sizeof (uint64_t);
+	    sm->sm_phys->smp_length - sizeof (uint64_t);
 	error = dmu_buf_hold(sm->sm_os, space_map_object(sm), last_word_offset,
 	    FTAG, &db, DMU_READ_NO_PREFETCH);
 	if (error != 0)
@@ -199,7 +201,7 @@ space_map_reversed_last_block_entries(space_map_t *sm,
 
 	uint64_t *words = db->db_data;
 	*nwords =
-	    (sm->sm_phys->smp_objsize - db->db_offset) / sizeof (uint64_t);
+	    (sm->sm_phys->smp_length - db->db_offset) / sizeof (uint64_t);
 
 	ASSERT3U(*nwords, <=, bufsz / sizeof (uint64_t));
 
@@ -298,8 +300,7 @@ space_map_incremental_destroy(space_map_t *sm, sm_cb_t
 			uint64_t e = buf[i];
 
 			if (sm_entry_is_debug(e)) {
-				sm->sm_phys->smp_objsize -= sizeof (uint64_t);
-				space_map_update(sm);
+				sm->sm_phys->smp_length -= sizeof (uint64_t);
 				continue;
 			}
 
@@ -354,15 +355,13 @@ space_map_incremental_destroy(space_map_t *sm, sm_cb_t
 				sm->sm_phys->smp_alloc -= entry_run;
 			else
 				sm->sm_phys->smp_alloc += entry_run;
-			sm->sm_phys->smp_objsize -= words * sizeof (uint64_t);
-			space_map_update(sm);
+			sm->sm_phys->smp_length -= words * sizeof (uint64_t);
 		}
 	}
 
 	if (space_map_length(sm) == 0) {
 		ASSERT0(error);
-		ASSERT0(sm->sm_phys->smp_objsize);
-		ASSERT0(sm->sm_alloc);
+		ASSERT0(space_map_allocated(sm));
 	}
 
 	zio_buf_free(buf, bufsz);
@@ -391,38 +390,42 @@ space_map_load_callback(space_map_entry_t *sme, void *
 }
 
 /*
- * Load the space map disk into the specified range tree. Segments of maptype
- * are added to the range tree, other segment types are removed.
+ * Load the spacemap into the rangetree, like space_map_load. But only
+ * read the first 'length' bytes of the spacemap.
  */
 int
-space_map_load(space_map_t *sm, range_tree_t *rt, maptype_t maptype)
+space_map_load_length(space_map_t *sm, range_tree_t *rt, maptype_t maptype,
+    uint64_t length)
 {
-	uint64_t space;
-	int err;
 	space_map_load_arg_t smla;
 
 	VERIFY0(range_tree_space(rt));
-	space = space_map_allocated(sm);
 
-	if (maptype == SM_FREE) {
+	if (maptype == SM_FREE)
 		range_tree_add(rt, sm->sm_start, sm->sm_size);
-		space = sm->sm_size - space;
-	}
 
 	smla.smla_rt = rt;
 	smla.smla_sm = sm;
 	smla.smla_type = maptype;
-	err = space_map_iterate(sm, space_map_load_callback, &smla);
+	int err = space_map_iterate(sm, length,
+	    space_map_load_callback, &smla);
 
-	if (err == 0) {
-		VERIFY3U(range_tree_space(rt), ==, space);
-	} else {
+	if (err != 0)
 		range_tree_vacate(rt, NULL, NULL);
-	}
 
 	return (err);
 }
 
+/*
+ * Load the space map disk into the specified range tree. Segments of maptype
+ * are added to the range tree, other segment types are removed.

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:04:19 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id F14E21AF3CA;
 Wed,  6 Nov 2019 09:04:19 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477LFH61VZz3L29;
 Wed,  6 Nov 2019 09:04:19 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ADF4522B91;
 Wed,  6 Nov 2019 09:04:19 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA694JCY052946;
 Wed, 6 Nov 2019 09:04:19 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA694JB5052945;
 Wed, 6 Nov 2019 09:04:19 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060904.xA694JB5052945@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:04:19 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354384 - vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Commit-Revision: 354384
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:04:20 -0000

Author: avg
Date: Wed Nov  6 09:04:19 2019
New Revision: 354384
URL: https://svnweb.freebsd.org/changeset/base/354384

Log:
  10570 Need workaround to EFI boot on AMI BIOS
  
  illumos/illumos-gate@fd7977362aae2eaa5dcb89671159f4fd82f22ca1
  https://github.com/illumos/illumos-gate/commit/fd7977362aae2eaa5dcb89671159f4fd82f22ca1
  
  https://www.illumos.org/issues/10570
    Some BIOS types are very pick about the GPT header size.
  
  Author: John Levon 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c	Wed Nov  6 09:00:06 2019	(r354383)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c	Wed Nov  6 09:04:19 2019	(r354384)
@@ -27,6 +27,7 @@
  * Copyright (c) 2012, 2017 by Delphix. All rights reserved.
  * Copyright (c) 2013, Joyent, Inc. All rights reserved.
  * Copyright (c) 2014 Integros [integros.com]
+ * Copyright (c) 2019, Joyent, Inc.
  */
 
 /*
@@ -1509,7 +1510,7 @@ zvol_getefi(void *arg, int flag, uint64_t vs, uint8_t 
 
 		gpt.efi_gpt_Signature = LE_64(EFI_SIGNATURE);
 		gpt.efi_gpt_Revision = LE_32(EFI_VERSION_CURRENT);
-		gpt.efi_gpt_HeaderSize = LE_32(sizeof (gpt));
+		gpt.efi_gpt_HeaderSize = LE_32(EFI_HEADER_SIZE);
 		gpt.efi_gpt_MyLBA = LE_64(1ULL);
 		gpt.efi_gpt_FirstUsableLBA = LE_64(34ULL);
 		gpt.efi_gpt_LastUsableLBA = LE_64((vs >> bs) - 1);
@@ -1519,7 +1520,7 @@ zvol_getefi(void *arg, int flag, uint64_t vs, uint8_t 
 		    LE_32(sizeof (efi_gpe_t));
 		CRC32(crc, &gpe, sizeof (gpe), -1U, crc32_table);
 		gpt.efi_gpt_PartitionEntryArrayCRC32 = LE_32(~crc);
-		CRC32(crc, &gpt, sizeof (gpt), -1U, crc32_table);
+		CRC32(crc, &gpt, EFI_HEADER_SIZE, -1U, crc32_table);
 		gpt.efi_gpt_HeaderCRC32 = LE_32(~crc);
 		if (ddi_copyout(&gpt, ptr, MIN(sizeof (gpt), length),
 		    flag))

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:05:40 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 50CF41AF463;
 Wed,  6 Nov 2019 09:05:40 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477LGr1Bp1z3L8Q;
 Wed,  6 Nov 2019 09:05:40 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0C74A22B94;
 Wed,  6 Nov 2019 09:05:40 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA695dpI053057;
 Wed, 6 Nov 2019 09:05:39 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA695dwH053056;
 Wed, 6 Nov 2019 09:05:39 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060905.xA695dwH053056@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:05:39 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354385 - vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
X-SVN-Commit-Revision: 354385
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:05:40 -0000

Author: avg
Date: Wed Nov  6 09:05:39 2019
New Revision: 354385
URL: https://svnweb.freebsd.org/changeset/base/354385

Log:
  11541 allocation_classes feature must be enabled to add log device
  
  illumos/illumos-gate@c1064fd7ce62fe763a4475e9988ffea3b22137de
  https://github.com/illumos/illumos-gate/commit/c1064fd7ce62fe763a4475e9988ffea3b22137de
  
  https://www.illumos.org/issues/11541
    After the allocation_classes feature was integrated, one can no longer add a
    log device to a pool unless that feature is enabled. There is an explicit check
    for this, but it is unnecessary in the case of log devices, so we should handle
    this better instead of forcing the feature to be enabled.
  
  Author: Jerry Jelinek 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c	Wed Nov  6 09:04:19 2019	(r354384)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c	Wed Nov  6 09:05:39 2019	(r354385)
@@ -25,7 +25,7 @@
  * Copyright 2017 Nexenta Systems, Inc.
  * Copyright (c) 2014 Integros [integros.com]
  * Copyright 2016 Toomas Soome 
- * Copyright 2017 Joyent, Inc.
+ * Copyright 2019 Joyent, Inc.
  * Copyright (c) 2017, Intel Corporation.
  */
 
@@ -633,7 +633,8 @@ vdev_alloc(spa_t *spa, vdev_t **vdp, nvlist_t *nv, vde
 			alloc_bias = vdev_derive_alloc_bias(bias);
 
 			/* spa_vdev_add() expects feature to be enabled */
-			if (spa->spa_load_state != SPA_LOAD_CREATE &&
+			if (alloc_bias != VDEV_BIAS_LOG &&
+			    spa->spa_load_state != SPA_LOAD_CREATE &&
 			    !spa_feature_is_enabled(spa,
 			    SPA_FEATURE_ALLOCATION_CLASSES)) {
 				return (SET_ERROR(ENOTSUP));

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:09:04 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id BAB2B1AF639;
 Wed,  6 Nov 2019 09:09:04 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477LLm5Fhzz3LMC;
 Wed,  6 Nov 2019 09:09:04 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9328722B9B;
 Wed,  6 Nov 2019 09:09:04 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6994Pq053253;
 Wed, 6 Nov 2019 09:09:04 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA6993qx053247;
 Wed, 6 Nov 2019 09:09:03 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060909.xA6993qx053247@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:09:03 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354386 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
X-SVN-Group: vendor-sys
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
X-SVN-Commit-Revision: 354386
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:09:04 -0000

Author: avg
Date: Wed Nov  6 09:09:03 2019
New Revision: 354386
URL: https://svnweb.freebsd.org/changeset/base/354386

Log:
  11770 additional mmp fixes
  
  illumos/illumos-gate@4348eb901228d2f8fa50bb132a34248e8662074e
  https://github.com/illumos/illumos-gate/commit/4348eb901228d2f8fa50bb132a34248e8662074e
  
  https://www.illumos.org/issues/11770
    Port a few additional MMP fixes from ZoL that came in after our
    initial MMP port.
    4ca457b065 ZTS: Fix mmp_interval failure
    ca95f70dff zpool import progress kstat
    (only minimal changes from above can be pulled in right now)
    060f0226e6 MMP interval and fail_intervals in uberblock
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Brian Behlendorf 
  Author: Olaf Faaland 

Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Changes in other areas also in this revision:
Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c	Wed Nov  6 09:09:03 2019	(r354386)
@@ -50,40 +50,101 @@
  * Uberblocks written by the txg_sync thread always go into the first
  * (N-MMP_BLOCKS_PER_LABEL) slots, the remaining slots are reserved for MMP.
  * They are used to hold uberblocks which are exactly the same as the last
- * synced uberblock except that the ub_timestamp is frequently updated.
- * Like all other uberblocks, the slot is written with an embedded checksum,
- * and slots with invalid checksums are ignored.  This provides the
+ * synced uberblock except that the ub_timestamp and mmp_config are frequently
+ * updated.  Like all other uberblocks, the slot is written with an embedded
+ * checksum, and slots with invalid checksums are ignored.  This provides the
  * "heartbeat", with no risk of overwriting good uberblocks that must be
  * preserved, e.g. previous txgs and associated block pointers.
  *
- * Two optional fields are added to uberblock structure: ub_mmp_magic and
- * ub_mmp_delay.  The magic field allows zfs to tell whether ub_mmp_delay is
- * valid.  The delay field is a decaying average of the amount of time between
- * completion of successive MMP writes, in nanoseconds.  It is used to predict
- * how long the import must wait to detect activity in the pool, before
- * concluding it is not in use.
+ * Three optional fields are added to uberblock structure; ub_mmp_magic,
+ * ub_mmp_config, and ub_mmp_delay.  The ub_mmp_magic value allows zfs to tell
+ * whether the other ub_mmp_* fields are valid.  The ub_mmp_config field tells
+ * the importing host the settings of zfs_multihost_interval and
+ * zfs_multihost_fail_intervals on the host which last had (or currently has)
+ * the pool imported.  These determine how long a host must wait to detect
+ * activity in the pool, before concluding the pool is not in use.  The
+ * mmp_delay field is a decaying average of the amount of time between
+ * completion of successive MMP writes, in nanoseconds.  It indicates whether
+ * MMP is enabled.
  *
  * During import an activity test may now be performed to determine if
  * the pool is in use.  The activity test is typically required if the
  * ZPOOL_CONFIG_HOSTID does not match the system hostid, the pool state is
  * POOL_STATE_ACTIVE, and the pool is not a root pool.
  *
- * The activity test finds the "best" uberblock (highest txg & timestamp),
- * waits some time, and then finds the "best" uberblock again.  If the txg
- * and timestamp in both "best" uberblocks do not match, the pool is in use
- * by another host and the import fails.  Since the granularity of the
- * timestamp is in seconds this activity test must take a bare minimum of one
- * second.  In order to assure the accuracy of the activity test, the default
- * values result in an activity test duration of 10x the mmp write interval.
+ * The activity test finds the "best" uberblock (highest txg, timestamp, and, if
+ * ub_mmp_magic is valid, sequence number from ub_mmp_config).  It then waits
+ * some time, and finds the "best" uberblock again.  If any of the mentioned
+ * fields have different values in the newly read uberblock, the pool is in use
+ * by another host and the import fails.  In order to assure the accuracy of the
+ * activity test, the default values result in an activity test duration of 20x
+ * the mmp write interval.
  *
- * The "zpool import"  activity test can be expected to take a minimum time of
- * zfs_multihost_import_intervals * zfs_multihost_interval milliseconds.  If the
- * "best" uberblock has a valid ub_mmp_delay field, then the duration of the
- * test may take longer if MMP writes were occurring less frequently than
- * expected.  Additionally, the duration is then extended by a random 25% to
- * attempt to to detect simultaneous imports.  For example, if both partner
- * hosts are rebooted at the same time and automatically attempt to import the
- * pool.
+ * The duration of the "zpool import" activity test depends on the information
+ * available in the "best" uberblock:
+ *
+ * 1) If uberblock was written by zfs-0.8 or newer and fail_intervals > 0:
+ *    ub_mmp_config.fail_intervals * ub_mmp_config.multihost_interval * 2
+ *
+ *    In this case, a weak guarantee is provided.  Since the host which last had
+ *    the pool imported will suspend the pool if no mmp writes land within
+ *    fail_intervals * multihost_interval ms, the absence of writes during that
+ *    time means either the pool is not imported, or it is imported but the pool
+ *    is suspended and no further writes will occur.
+ *
+ *    Note that resuming the suspended pool on the remote host would invalidate
+ *    this guarantee, and so it is not allowed.
+ *
+ *    The factor of 2 provides a conservative safety factor and derives from
+ *    MMP_IMPORT_SAFETY_FACTOR;
+ *
+ * 2) If uberblock was written by zfs-0.8 or newer and fail_intervals == 0:
+ *    (ub_mmp_config.multihost_interval + ub_mmp_delay) *
+ *        zfs_multihost_import_intervals
+ *
+ *    In this case no guarantee can provided.  However, as long as some devices
+ *    are healthy and connected, it is likely that at least one write will land
+ *    within (multihost_interval + mmp_delay) because multihost_interval is
+ *    enough time for a write to be attempted to each leaf vdev, and mmp_delay
+ *    is enough for one to land, based on past delays.  Multiplying by
+ *    zfs_multihost_import_intervals provides a conservative safety factor.
+ *
+ * 3) If uberblock was written by zfs-0.7:
+ *    (zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
+ *
+ *    The same logic as case #2 applies, but we do not know remote tunables.
+ *
+ *    We use the local value for zfs_multihost_interval because the original MMP
+ *    did not record this value in the uberblock.
+ *
+ *    ub_mmp_delay >= (zfs_multihost_interval / leaves), so if the other host
+ *    has a much larger zfs_multihost_interval set, ub_mmp_delay will reflect
+ *    that.  We will have waited enough time for zfs_multihost_import_intervals
+ *    writes to be issued and all but one to land.
+ *
+ *    single device pool example delays
+ *
+ *    import_delay = (1 + 1) * 20   =  40s #defaults, no I/O delay
+ *    import_delay = (1 + 10) * 20  = 220s #defaults, 10s I/O delay
+ *    import_delay = (10 + 10) * 20 = 400s #10s multihost_interval,
+ *                                          no I/O delay
+ *    100 device pool example delays
+ *
+ *    import_delay = (1 + .01) * 20 =  20s #defaults, no I/O delay
+ *    import_delay = (1 + 10) * 20  = 220s #defaults, 10s I/O delay
+ *    import_delay = (10 + .1) * 20 = 202s #10s multihost_interval,
+ *                                          no I/O delay
+ *
+ * 4) Otherwise, this uberblock was written by a pre-MMP zfs:
+ *    zfs_multihost_import_intervals * zfs_multihost_interval
+ *
+ *    In this case local tunables are used.  By default this product = 10s, long
+ *    enough for a pool with any activity at all to write at least one
+ *    uberblock.  No guarantee can be provided.
+ *
+ * Additionally, the duration is then extended by a random 25% to attempt to to
+ * detect simultaneous imports.  For example, if both partner hosts are rebooted
+ * at the same time and automatically attempt to import the pool.
  */
 
 /*
@@ -91,10 +152,9 @@
  * 'multihost' pool property is on.  This is one factor used to determine the
  * length of the activity check during import.
  *
- * The mmp write period is zfs_multihost_interval / leaf-vdevs milliseconds.
- * This means that on average an mmp write will be issued for each leaf vdev
- * every zfs_multihost_interval milliseconds.  In practice, the observed period
- * can vary with the I/O load and this observed value is the delay which is
+ * On average an mmp write will be issued for each leaf vdev every
+ * zfs_multihost_interval milliseconds.  In practice, the observed period can
+ * vary with the I/O load and this observed value is the ub_mmp_delay which is
  * stored in the uberblock.  The minimum allowed value is 100 ms.
  */
 ulong_t zfs_multihost_interval = MMP_DEFAULT_INTERVAL;
@@ -109,19 +169,21 @@ ulong_t zfs_multihost_interval = MMP_DEFAULT_INTERVAL;
 uint_t zfs_multihost_import_intervals = MMP_DEFAULT_IMPORT_INTERVALS;
 
 /*
- * Controls the behavior of the pool when mmp write failures are detected.
+ * Controls the behavior of the pool when mmp write failures or delays are
+ * detected.
  *
- * When zfs_multihost_fail_intervals = 0 then mmp write failures are ignored.
- * The failures will still be reported to the ZED which depending on its
- * configuration may take action such as suspending the pool or taking a
+ * When zfs_multihost_fail_intervals = 0, mmp write failures or delays are
+ * ignored.  The failures will still be reported to the ZED which depending on
+ * its configuration may take action such as suspending the pool or taking a
  * device offline.
  *
- * When zfs_multihost_fail_intervals > 0 then sequential mmp write failures will
- * cause the pool to be suspended.  This occurs when
- * zfs_multihost_fail_intervals * zfs_multihost_interval milliseconds have
- * passed since the last successful mmp write.  This guarantees the activity
- * test will see mmp writes if the
- * pool is imported.
+ * When zfs_multihost_fail_intervals > 0, the pool will be suspended if
+ * zfs_multihost_fail_intervals * zfs_multihost_interval milliseconds pass
+ * without a successful mmp write.  This guarantees the activity test will see
+ * mmp writes if the pool is imported.  A value of 1 is ignored and treated as
+ * if it was set to 2, because a single leaf vdev pool will issue a write once
+ * per multihost_interval and thus any variation in latency would cause the
+ * pool to be suspended.
  */
 uint_t zfs_multihost_fail_intervals = MMP_DEFAULT_FAIL_INTERVALS;
 
@@ -137,6 +199,14 @@ mmp_init(spa_t *spa)
 	cv_init(&mmp->mmp_thread_cv, NULL, CV_DEFAULT, NULL);
 	mutex_init(&mmp->mmp_io_lock, NULL, MUTEX_DEFAULT, NULL);
 	mmp->mmp_kstat_id = 1;
+
+	/*
+	 * mmp_write_done() calculates mmp_delay based on prior mmp_delay and
+	 * the elapsed time since the last write.  For the first mmp write,
+	 * there is no "last write", so we start with fake non-zero values.
+	 */
+	mmp->mmp_last_write = gethrtime();
+	mmp->mmp_delay = MSEC2NSEC(MMP_INTERVAL_OK(zfs_multihost_interval));
 }
 
 void
@@ -174,10 +244,10 @@ mmp_thread_start(spa_t *spa)
 	if (spa_writeable(spa)) {
 		mutex_enter(&mmp->mmp_thread_lock);
 		if (!mmp->mmp_thread) {
-			dprintf("mmp_thread_start pool %s\n",
-			    spa->spa_name);
 			mmp->mmp_thread = thread_create(NULL, 0, mmp_thread,
 			    spa, 0, &p0, TS_RUN, minclsyspri);
+			zfs_dbgmsg("MMP thread started pool '%s' "
+			    "gethrtime %llu", spa_name(spa), gethrtime());
 		}
 		mutex_exit(&mmp->mmp_thread_lock);
 	}
@@ -196,6 +266,8 @@ mmp_thread_stop(spa_t *spa)
 		cv_wait(&mmp->mmp_thread_cv, &mmp->mmp_thread_lock);
 	}
 	mutex_exit(&mmp->mmp_thread_lock);
+	zfs_dbgmsg("MMP thread stopped pool '%s' gethrtime %llu",
+	    spa_name(spa), gethrtime());
 
 	ASSERT(mmp->mmp_thread == NULL);
 	mmp->mmp_thread_exiting = 0;
@@ -305,7 +377,8 @@ mmp_delay_update(spa_t *spa, boolean_t write_completed
 	 * strictly less than, in case delay was changed above.
 	 */
 	if (delay < mts->mmp_delay) {
-		hrtime_t min_delay = MSEC2NSEC(zfs_multihost_interval) /
+		hrtime_t min_delay =
+		    MSEC2NSEC(MMP_INTERVAL_OK(zfs_multihost_interval)) /
 		    MAX(1, vdev_count_leaves(spa));
 		mts->mmp_delay = MAX(((delay + mts->mmp_delay * 127) / 128),
 		    min_delay);
@@ -346,6 +419,7 @@ mmp_update_uberblock(spa_t *spa, uberblock_t *ub)
 
 	mutex_enter(&mmp->mmp_io_lock);
 	mmp->mmp_ub = *ub;
+	mmp->mmp_seq = 1;
 	mmp->mmp_ub.ub_timestamp = gethrestime_sec();
 	mmp_delay_update(spa, B_TRUE);
 	mutex_exit(&mmp->mmp_io_lock);
@@ -370,8 +444,9 @@ mmp_write_uberblock(spa_t *spa)
 	spa_config_enter(spa, SCL_STATE, mmp_tag, RW_READER);
 	lock_acquire_time = gethrtime() - lock_acquire_time;
 	if (lock_acquire_time > (MSEC2NSEC(MMP_MIN_INTERVAL) / 10))
-		zfs_dbgmsg("SCL_STATE acquisition took %llu ns\n",
-		    (u_longlong_t)lock_acquire_time);
+		zfs_dbgmsg("MMP SCL_STATE acquisition pool '%s' took %llu ns "
+		    "gethrtime %llu", spa_name(spa), lock_acquire_time,
+		    gethrtime());
 
 	mutex_enter(&mmp->mmp_io_lock);
 
@@ -400,6 +475,9 @@ mmp_write_uberblock(spa_t *spa)
 			 * gethrestime_sec(), mmp->mmp_delay, NULL, 0,
 			 * mmp->mmp_kstat_id++, error);
 			 */
+			zfs_dbgmsg("MMP error choosing leaf pool '%s' "
+			    "gethrtime %llu fail_mask %#x", spa_name(spa),
+			    gethrtime(), error);
 		}
 		mutex_exit(&mmp->mmp_io_lock);
 		spa_config_exit(spa, SCL_STATE, mmp_tag);
@@ -407,16 +485,34 @@ mmp_write_uberblock(spa_t *spa)
 	}
 
 	vd = spa->spa_mmp.mmp_last_leaf;
-	mmp->mmp_skip_error = 0;
+	if (mmp->mmp_skip_error != 0) {
+		mmp->mmp_skip_error = 0;
+		zfs_dbgmsg("MMP write after skipping due to unavailable "
+		    "leaves, pool '%s' gethrtime %llu leaf %#llu",
+		    spa_name(spa), gethrtime(), vd->vdev_guid);
+	}
 
 	if (mmp->mmp_zio_root == NULL)
 		mmp->mmp_zio_root = zio_root(spa, NULL, NULL,
 		    flags | ZIO_FLAG_GODFATHER);
 
+	if (mmp->mmp_ub.ub_timestamp != gethrestime_sec()) {
+		/*
+		 * Want to reset mmp_seq when timestamp advances because after
+		 * an mmp_seq wrap new values will not be chosen by
+		 * uberblock_compare() as the "best".
+		 */
+		mmp->mmp_ub.ub_timestamp = gethrestime_sec();
+		mmp->mmp_seq = 1;
+	}
+
 	ub = &mmp->mmp_ub;
-	ub->ub_timestamp = gethrestime_sec();
 	ub->ub_mmp_magic = MMP_MAGIC;
 	ub->ub_mmp_delay = mmp->mmp_delay;
+	ub->ub_mmp_config = MMP_SEQ_SET(mmp->mmp_seq) |
+	    MMP_INTERVAL_SET(MMP_INTERVAL_OK(zfs_multihost_interval)) |
+	    MMP_FAIL_INT_SET(MMP_FAIL_INTVS_OK(
+	    zfs_multihost_fail_intervals));
 	vd->vdev_mmp_pending = gethrtime();
 	vd->vdev_mmp_kstat_id = mmp->mmp_kstat_id;
 
@@ -425,6 +521,7 @@ mmp_write_uberblock(spa_t *spa)
 	abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));
 	abd_copy_from_buf(ub_abd, ub, sizeof (uberblock_t));
 
+	mmp->mmp_seq++;
 	mmp->mmp_kstat_id++;
 	mutex_exit(&mmp->mmp_io_lock);
 
@@ -450,39 +547,76 @@ mmp_thread(void *arg)
 {
 	spa_t *spa = (spa_t *)arg;
 	mmp_thread_t *mmp = &spa->spa_mmp;
-	boolean_t last_spa_suspended = spa_suspended(spa);
-	boolean_t last_spa_multihost = spa_multihost(spa);
+	boolean_t suspended = spa_suspended(spa);
+	boolean_t multihost = spa_multihost(spa);
+	uint64_t mmp_interval = MSEC2NSEC(MMP_INTERVAL_OK(
+	    zfs_multihost_interval));
+	uint32_t mmp_fail_intervals = MMP_FAIL_INTVS_OK(
+	    zfs_multihost_fail_intervals);
+	hrtime_t mmp_fail_ns = mmp_fail_intervals * mmp_interval;
+	boolean_t last_spa_suspended = suspended;
+	boolean_t last_spa_multihost = multihost;
+	uint64_t last_mmp_interval = mmp_interval;
+	uint32_t last_mmp_fail_intervals = mmp_fail_intervals;
+	hrtime_t last_mmp_fail_ns = mmp_fail_ns;
 	callb_cpr_t cpr;
-	hrtime_t max_fail_ns = zfs_multihost_fail_intervals *
-	    MSEC2NSEC(MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
+	int skip_wait = 0;
 
 	mmp_thread_enter(mmp, &cpr);
 
-	/*
-	 * The mmp_write_done() function calculates mmp_delay based on the
-	 * prior value of mmp_delay and the elapsed time since the last write.
-	 * For the first mmp write, there is no "last write", so we start
-	 * with fake, but reasonable, default non-zero values.
-	 */
-	mmp->mmp_delay = MSEC2NSEC(MAX(zfs_multihost_interval,
-	    MMP_MIN_INTERVAL)) / MAX(vdev_count_leaves(spa), 1);
-	mmp->mmp_last_write = gethrtime() - mmp->mmp_delay;
-
 	while (!mmp->mmp_thread_exiting) {
-		uint64_t mmp_fail_intervals = zfs_multihost_fail_intervals;
-		uint64_t mmp_interval = MSEC2NSEC(
-		    MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
-		boolean_t suspended = spa_suspended(spa);
-		boolean_t multihost = spa_multihost(spa);
-		hrtime_t next_time;
+		hrtime_t next_time = gethrtime() +
+		    MSEC2NSEC(MMP_DEFAULT_INTERVAL);
+		int leaves = MAX(vdev_count_leaves(spa), 1);
 
+		/* Detect changes in tunables or state */
+
+		last_spa_suspended = suspended;
+		last_spa_multihost = multihost;
+		suspended = spa_suspended(spa);
+		multihost = spa_multihost(spa);
+
+		last_mmp_interval = mmp_interval;
+		last_mmp_fail_intervals = mmp_fail_intervals;
+		last_mmp_fail_ns = mmp_fail_ns;
+		mmp_interval = MSEC2NSEC(MMP_INTERVAL_OK(
+		    zfs_multihost_interval));
+		mmp_fail_intervals = MMP_FAIL_INTVS_OK(
+		    zfs_multihost_fail_intervals);
+
+		/* Smooth so pool is not suspended when reducing tunables */
+		if (mmp_fail_intervals * mmp_interval < mmp_fail_ns) {
+			mmp_fail_ns = (mmp_fail_ns * 31 +
+			    mmp_fail_intervals * mmp_interval) / 32;
+		} else {
+			mmp_fail_ns = mmp_fail_intervals *
+			    mmp_interval;
+		}
+
+		if (mmp_interval != last_mmp_interval ||
+		    mmp_fail_intervals != last_mmp_fail_intervals) {
+			/*
+			 * We want other hosts to see new tunables as quickly as
+			 * possible.  Write out at higher frequency than usual.
+			 */
+			skip_wait += leaves;
+		}
+
 		if (multihost)
-			next_time = gethrtime() + mmp_interval /
-			    MAX(vdev_count_leaves(spa), 1);
-		else
-			next_time = gethrtime() +
-			    MSEC2NSEC(MMP_DEFAULT_INTERVAL);
+			next_time = gethrtime() + mmp_interval / leaves;
 
+		if (mmp_fail_ns != last_mmp_fail_ns) {
+			zfs_dbgmsg("MMP interval change pool '%s' "
+			    "gethrtime %llu last_mmp_interval %llu "
+			    "mmp_interval %llu last_mmp_fail_intervals %u "
+			    "mmp_fail_intervals %u mmp_fail_ns %llu "
+			    "skip_wait %d leaves %d next_time %llu",
+			    spa_name(spa), gethrtime(), last_mmp_interval,
+			    mmp_interval, last_mmp_fail_intervals,
+			    mmp_fail_intervals, mmp_fail_ns, skip_wait, leaves,
+			    next_time);
+		}
+
 		/*
 		 * MMP off => on, or suspended => !suspended:
 		 * No writes occurred recently.  Update mmp_last_write to give
@@ -490,8 +624,14 @@ mmp_thread(void *arg)
 		 */
 		if ((!last_spa_multihost && multihost) ||
 		    (last_spa_suspended && !suspended)) {
+			zfs_dbgmsg("MMP state change pool '%s': gethrtime %llu "
+			    "last_spa_multihost %u multihost %u "
+			    "last_spa_suspended %u suspended %u",
+			    spa_name(spa), last_spa_multihost, multihost,
+			    last_spa_suspended, suspended);
 			mutex_enter(&mmp->mmp_io_lock);
 			mmp->mmp_last_write = gethrtime();
+			mmp->mmp_delay = mmp_interval;
 			mutex_exit(&mmp->mmp_io_lock);
 		}
 
@@ -504,37 +644,38 @@ mmp_thread(void *arg)
 			mmp->mmp_delay = 0;
 			mutex_exit(&mmp->mmp_io_lock);
 		}
-		last_spa_multihost = multihost;
-		last_spa_suspended = suspended;
 
 		/*
-		 * Smooth max_fail_ns when its factors are decreased, because
-		 * making (max_fail_ns < mmp_interval) results in the pool being
-		 * immediately suspended before writes can occur at the new
-		 * higher frequency.
-		 */
-		if ((mmp_interval * mmp_fail_intervals) < max_fail_ns) {
-			max_fail_ns = ((31 * max_fail_ns) + (mmp_interval *
-			    mmp_fail_intervals)) / 32;
-		} else {
-			max_fail_ns = mmp_interval * mmp_fail_intervals;
-		}
-
-		/*
 		 * Suspend the pool if no MMP write has succeeded in over
 		 * mmp_interval * mmp_fail_intervals nanoseconds.
 		 */
-		if (!suspended && mmp_fail_intervals && multihost &&
-		    (gethrtime() - mmp->mmp_last_write) > max_fail_ns) {
+		if (multihost && !suspended && mmp_fail_intervals &&
+		    (gethrtime() - mmp->mmp_last_write) > mmp_fail_ns) {
+			zfs_dbgmsg("MMP suspending pool '%s': gethrtime %llu "
+			    "mmp_last_write %llu mmp_interval %llu "
+			    "mmp_fail_intervals %llu mmp_fail_ns %llu",
+			    spa_name(spa), (u_longlong_t)gethrtime(),
+			    (u_longlong_t)mmp->mmp_last_write,
+			    (u_longlong_t)mmp_interval,
+			    (u_longlong_t)mmp_fail_intervals,
+			    (u_longlong_t)mmp_fail_ns);
 			cmn_err(CE_WARN, "MMP writes to pool '%s' have not "
-			    "succeeded in over %llus; suspending pool",
+			    "succeeded in over %llu ms; suspending pool. "
+			    "Hrtime %llu",
 			    spa_name(spa),
-			    NSEC2SEC(gethrtime() - mmp->mmp_last_write));
+			    NSEC2MSEC(gethrtime() - mmp->mmp_last_write),
+			    gethrtime());
 			zio_suspend(spa, NULL, ZIO_SUSPEND_MMP);
 		}
 
 		if (multihost && !suspended)
 			mmp_write_uberblock(spa);
+
+		if (skip_wait > 0) {
+			next_time = gethrtime() + MSEC2NSEC(MMP_MIN_INTERVAL) /
+			    leaves;
+			skip_wait--;
+		}
 
 		CALLB_CPR_SAFE_BEGIN(&cpr);
 		(void) cv_timedwait_sig_hrtime(&mmp->mmp_thread_cv,

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c	Wed Nov  6 09:09:03 2019	(r354386)
@@ -2338,6 +2338,7 @@ spa_activity_check_required(spa_t *spa, uberblock_t *u
 	uint64_t hostid = 0;
 	uint64_t tryconfig_txg = 0;
 	uint64_t tryconfig_timestamp = 0;
+	uint16_t tryconfig_mmp_seq = 0;
 	nvlist_t *nvinfo;
 
 	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
@@ -2346,6 +2347,8 @@ spa_activity_check_required(spa_t *spa, uberblock_t *u
 		    &tryconfig_txg);
 		(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_TIMESTAMP,
 		    &tryconfig_timestamp);
+		(void) nvlist_lookup_uint16(nvinfo, ZPOOL_CONFIG_MMP_SEQ,
+		    &tryconfig_mmp_seq);
 	}
 
 	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_STATE, &state);
@@ -2362,14 +2365,17 @@ spa_activity_check_required(spa_t *spa, uberblock_t *u
 	 */
 	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay == 0)
 		return (B_FALSE);
+
 	/*
-	 * If the tryconfig_* values are nonzero, they are the results of an
-	 * earlier tryimport.  If they match the uberblock we just found, then
-	 * the pool has not changed and we return false so we do not test a
-	 * second time.
+	 * If the tryconfig_ values are nonzero, they are the results of an
+	 * earlier tryimport.  If they all match the uberblock we just found,
+	 * then the pool has not changed and we return false so we do not test
+	 * a second time.
 	 */
 	if (tryconfig_txg && tryconfig_txg == ub->ub_txg &&
-	    tryconfig_timestamp && tryconfig_timestamp == ub->ub_timestamp)
+	    tryconfig_timestamp && tryconfig_timestamp == ub->ub_timestamp &&
+	    tryconfig_mmp_seq && tryconfig_mmp_seq ==
+	    (MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0))
 		return (B_FALSE);
 
 	/*
@@ -2393,16 +2399,87 @@ spa_activity_check_required(spa_t *spa, uberblock_t *u
 }
 
 /*
+ * Nanoseconds the activity check must watch for changes on-disk.
+ */
+static uint64_t
+spa_activity_check_duration(spa_t *spa, uberblock_t *ub)
+{
+	uint64_t import_intervals = MAX(zfs_multihost_import_intervals, 1);
+	uint64_t multihost_interval = MSEC2NSEC(
+	    MMP_INTERVAL_OK(zfs_multihost_interval));
+	uint64_t import_delay = MAX(NANOSEC, import_intervals *
+	    multihost_interval);
+
+	/*
+	 * Local tunables determine a minimum duration except for the case
+	 * where we know when the remote host will suspend the pool if MMP
+	 * writes do not land.
+	 *
+	 * See Big Theory comment at the top of mmp.c for the reasoning behind
+	 * these cases and times.
+	 */
+
+	ASSERT(MMP_IMPORT_SAFETY_FACTOR >= 100);
+
+	if (MMP_INTERVAL_VALID(ub) && MMP_FAIL_INT_VALID(ub) &&
+	    MMP_FAIL_INT(ub) > 0) {
+
+		/* MMP on remote host will suspend pool after failed writes */
+		import_delay = MMP_FAIL_INT(ub) * MSEC2NSEC(MMP_INTERVAL(ub)) *
+		    MMP_IMPORT_SAFETY_FACTOR / 100;
+
+		zfs_dbgmsg("fail_intvals>0 import_delay=%llu ub_mmp "
+		    "mmp_fails=%llu ub_mmp mmp_interval=%llu "
+		    "import_intervals=%u", import_delay, MMP_FAIL_INT(ub),
+		    MMP_INTERVAL(ub), import_intervals);
+
+	} else if (MMP_INTERVAL_VALID(ub) && MMP_FAIL_INT_VALID(ub) &&
+	    MMP_FAIL_INT(ub) == 0) {
+
+		/* MMP on remote host will never suspend pool */
+		import_delay = MAX(import_delay, (MSEC2NSEC(MMP_INTERVAL(ub)) +
+		    ub->ub_mmp_delay) * import_intervals);
+
+		zfs_dbgmsg("fail_intvals=0 import_delay=%llu ub_mmp "
+		    "mmp_interval=%llu ub_mmp_delay=%llu "
+		    "import_intervals=%u", import_delay, MMP_INTERVAL(ub),
+		    ub->ub_mmp_delay, import_intervals);
+
+	} else if (MMP_VALID(ub)) {
+		/*
+		 * zfs-0.7 compatability case
+		 */
+
+		import_delay = MAX(import_delay, (multihost_interval +
+		    ub->ub_mmp_delay) * import_intervals);
+
+		zfs_dbgmsg("import_delay=%llu ub_mmp_delay=%llu "
+		    "import_intervals=%u leaves=%u", import_delay,
+		    ub->ub_mmp_delay, import_intervals,
+		    vdev_count_leaves(spa));
+	} else {
+		/* Using local tunings is the only reasonable option */
+		zfs_dbgmsg("pool last imported on non-MMP aware "
+		    "host using import_delay=%llu multihost_interval=%llu "
+		    "import_intervals=%u", import_delay, multihost_interval,
+		    import_intervals);
+	}
+
+	return (import_delay);
+}
+
+/*
  * Perform the import activity check.  If the user canceled the import or
  * we detected activity then fail.
  */
 static int
 spa_activity_check(spa_t *spa, uberblock_t *ub, nvlist_t *config)
 {
-	uint64_t import_intervals = MAX(zfs_multihost_import_intervals, 1);
 	uint64_t txg = ub->ub_txg;
 	uint64_t timestamp = ub->ub_timestamp;
-	uint64_t import_delay = NANOSEC;
+	uint64_t mmp_config = ub->ub_mmp_config;
+	uint16_t mmp_seq = MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0;
+	uint64_t import_delay;
 	hrtime_t import_expire;
 	nvlist_t *mmp_label = NULL;
 	vdev_t *rvd = spa->spa_root_vdev;
@@ -2419,7 +2496,7 @@ spa_activity_check(spa_t *spa, uberblock_t *ub, nvlist
 	 * during the earlier tryimport.  If the txg recorded there is 0 then
 	 * the pool is known to be active on another host.
 	 *
-	 * Otherwise, the pool might be in use on another node.  Check for
+	 * Otherwise, the pool might be in use on another host.  Check for
 	 * changes in the uberblocks on disk if necessary.
 	 */
 	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
@@ -2434,32 +2511,25 @@ spa_activity_check(spa_t *spa, uberblock_t *ub, nvlist
 		}
 	}
 
-	/*
-	 * Preferentially use the zfs_multihost_interval from the node which
-	 * last imported the pool.  This value is stored in an MMP uberblock as.
-	 *
-	 * ub_mmp_delay * vdev_count_leaves() == zfs_multihost_interval
-	 */
-	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay)
-		import_delay = MAX(import_delay, import_intervals *
-		    ub->ub_mmp_delay * MAX(vdev_count_leaves(spa), 1));
+	import_delay = spa_activity_check_duration(spa, ub);
 
-	/* Apply a floor using the local default values. */
-	import_delay = MAX(import_delay, import_intervals *
-	    MSEC2NSEC(MAX(zfs_multihost_interval, MMP_MIN_INTERVAL)));
-
-	zfs_dbgmsg("import_delay=%llu ub_mmp_delay=%llu import_intervals=%u "
-	    "leaves=%u", import_delay, ub->ub_mmp_delay, import_intervals,
-	    vdev_count_leaves(spa));
-
 	/* Add a small random factor in case of simultaneous imports (0-25%) */
-	import_expire = gethrtime() + import_delay +
-	    (import_delay * spa_get_random(250) / 1000);
+	import_delay += import_delay * spa_get_random(250) / 1000;
 
+	import_expire = gethrtime() + import_delay;
+
 	while (gethrtime() < import_expire) {
 		vdev_uberblock_load(rvd, ub, &mmp_label);
 
-		if (txg != ub->ub_txg || timestamp != ub->ub_timestamp) {
+		if (txg != ub->ub_txg || timestamp != ub->ub_timestamp ||
+		    mmp_seq != (MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0)) {
+			zfs_dbgmsg("multihost activity detected "
+			    "txg %llu ub_txg  %llu "
+			    "timestamp %llu ub_timestamp  %llu "
+			    "mmp_config %#llx ub_mmp_config %#llx",
+			    txg, ub->ub_txg, timestamp, ub->ub_timestamp,
+			    mmp_config, ub->ub_mmp_config);
+
 			error = SET_ERROR(EREMOTEIO);
 			break;
 		}
@@ -2844,6 +2914,9 @@ spa_ld_select_uberblock(spa_t *spa, spa_import_type_t 
 		    ZPOOL_CONFIG_MMP_STATE, MMP_STATE_INACTIVE);
 		fnvlist_add_uint64(spa->spa_load_info,
 		    ZPOOL_CONFIG_MMP_TXG, ub->ub_txg);
+		fnvlist_add_uint16(spa->spa_load_info,
+		    ZPOOL_CONFIG_MMP_SEQ,
+		    (MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0));
 	}
 
 	/*

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h	Wed Nov  6 09:09:03 2019	(r354386)
@@ -29,8 +29,13 @@ extern "C" {
 
 #define	MMP_MIN_INTERVAL		100	/* ms */
 #define	MMP_DEFAULT_INTERVAL		1000	/* ms */
-#define	MMP_DEFAULT_IMPORT_INTERVALS	10
-#define	MMP_DEFAULT_FAIL_INTERVALS	5
+#define	MMP_DEFAULT_IMPORT_INTERVALS	20
+#define	MMP_DEFAULT_FAIL_INTERVALS	10
+#define	MMP_MIN_FAIL_INTERVALS		2	/* min if != 0 */
+#define	MMP_IMPORT_SAFETY_FACTOR	200	/* pct */
+#define	MMP_INTERVAL_OK(interval)	MAX(interval, MMP_MIN_INTERVAL)
+#define	MMP_FAIL_INTVS_OK(fails)	(fails == 0 ? 0 : MAX(fails, \
+					    MMP_MIN_FAIL_INTERVALS))
 
 typedef struct mmp_thread {
 	kmutex_t	mmp_thread_lock; /* protect thread mgmt fields */
@@ -46,6 +51,7 @@ typedef struct mmp_thread {
 	int		mmp_skip_error; /* reason for last skipped write */
 	vdev_t		*mmp_last_leaf;	/* last mmp write sent here */
 	uint64_t	mmp_leaf_last_gen;	/* last mmp write sent here */
+	uint32_t	mmp_seq;	/* intra-second update counter */
 } mmp_thread_t;
 
 

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h	Wed Nov  6 09:09:03 2019	(r354386)
@@ -44,8 +44,37 @@ extern "C" {
  */
 #define	UBERBLOCK_MAGIC		0x00bab10c		/* oo-ba-bloc!	*/
 #define	UBERBLOCK_SHIFT		10			/* up to 1K	*/
-#define	MMP_MAGIC		0xa11cea11		/* all-see-all  */
+#define	MMP_MAGIC		0xa11cea11		/* all-see-all	*/
 
+#define	MMP_INTERVAL_VALID_BIT	0x01
+#define	MMP_SEQ_VALID_BIT	0x02
+#define	MMP_FAIL_INT_VALID_BIT	0x04
+
+#define	MMP_VALID(ubp)		(ubp->ub_magic == UBERBLOCK_MAGIC && \
+				    ubp->ub_mmp_magic == MMP_MAGIC)
+#define	MMP_INTERVAL_VALID(ubp)	(MMP_VALID(ubp) && (ubp->ub_mmp_config & \
+				    MMP_INTERVAL_VALID_BIT))
+#define	MMP_SEQ_VALID(ubp)	(MMP_VALID(ubp) && (ubp->ub_mmp_config & \
+				    MMP_SEQ_VALID_BIT))
+#define	MMP_FAIL_INT_VALID(ubp)	(MMP_VALID(ubp) && (ubp->ub_mmp_config & \
+				    MMP_FAIL_INT_VALID_BIT))
+
+#define	MMP_INTERVAL(ubp)	((ubp->ub_mmp_config & 0x00000000FFFFFF00) \
+				    >> 8)
+#define	MMP_SEQ(ubp)		((ubp->ub_mmp_config & 0x0000FFFF00000000) \
+				    >> 32)
+#define	MMP_FAIL_INT(ubp)	((ubp->ub_mmp_config & 0xFFFF000000000000) \
+				    >> 48)
+
+#define	MMP_INTERVAL_SET(write) \
+	    (((uint64_t)(write & 0xFFFFFF) << 8) | MMP_INTERVAL_VALID_BIT)
+
+#define	MMP_SEQ_SET(seq) \
+	    (((uint64_t)(seq & 0xFFFF) << 32) | MMP_SEQ_VALID_BIT)
+
+#define	MMP_FAIL_INT_SET(fail) \
+	    (((uint64_t)(fail & 0xFFFF) << 48) | MMP_FAIL_INT_VALID_BIT)
+
 struct uberblock {
 	uint64_t	ub_magic;	/* UBERBLOCK_MAGIC		*/
 	uint64_t	ub_version;	/* SPA_VERSION			*/
@@ -59,8 +88,31 @@ struct uberblock {
 
 	/* Maybe missing in uberblocks we read, but always written */
 	uint64_t	ub_mmp_magic;
+	/*
+	 * If ub_mmp_delay == 0 and ub_mmp_magic is valid, MMP is off.
+	 * Otherwise, nanosec since last MMP write.
+	 */
 	uint64_t	ub_mmp_delay;
-	uint64_t	ub_mmp_seq;
+
+	/*
+	 * The ub_mmp_config contains the multihost write interval, multihost
+	 * fail intervals, sequence number for sub-second granularity, and
+	 * valid bit mask.  This layout is as follows:
+	 *
+	 *   64      56      48      40      32      24      16      8       0
+	 *   +-------+-------+-------+-------+-------+-------+-------+-------+
+	 * 0 | Fail Intervals|      Seq      |   Write Interval (ms) | VALID |
+	 *   +-------+-------+-------+-------+-------+-------+-------+-------+
+	 *
+	 * This allows a write_interval of (2^24/1000)s, over 4.5 hours
+	 *
+	 * VALID Bits:
+	 * - 0x01 - Write Interval (ms)
+	 * - 0x02 - Sequence number exists
+	 * - 0x04 - Fail Intervals
+	 * - 0xf8 - Reserved
+	 */
+	uint64_t	ub_mmp_config;
 
 	/*
 	 * ub_checkpoint_txg indicates two things about the current uberblock:

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c	Wed Nov  6 09:09:03 2019	(r354386)
@@ -26,6 +26,7 @@
 #include 
 #include 
 #include 
+#include 
 
 int
 uberblock_verify(uberblock_t *ub)
@@ -58,8 +59,15 @@ uberblock_update(uberblock_t *ub, vdev_t *rvd, uint64_
 	ub->ub_timestamp = gethrestime_sec();
 	ub->ub_software_version = SPA_VERSION;
 	ub->ub_mmp_magic = MMP_MAGIC;
-	ub->ub_mmp_delay = spa_multihost(rvd->vdev_spa) ? mmp_delay : 0;
-	ub->ub_mmp_seq = 0;
+	if (spa_multihost(rvd->vdev_spa)) {
+		ub->ub_mmp_delay = mmp_delay;
+		ub->ub_mmp_config = MMP_SEQ_SET(0) |
+		    MMP_INTERVAL_SET(zfs_multihost_interval) |
+		    MMP_FAIL_INT_SET(zfs_multihost_fail_intervals);
+	} else {
+		ub->ub_mmp_delay = 0;
+		ub->ub_mmp_config = 0;
+	}
 	ub->ub_checkpoint_txg = 0;
 
 	return (ub->ub_rootbp.blk_birth == txg);

Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
==============================================================================
--- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c	Wed Nov  6 09:09:03 2019	(r354386)
@@ -1016,10 +1016,35 @@ static int
 vdev_uberblock_compare(const uberblock_t *ub1, const uberblock_t *ub2)
 {
 	int cmp = AVL_CMP(ub1->ub_txg, ub2->ub_txg);
+
 	if (likely(cmp))
 		return (cmp);
 
-	return (AVL_CMP(ub1->ub_timestamp, ub2->ub_timestamp));
+	cmp = AVL_CMP(ub1->ub_timestamp, ub2->ub_timestamp);
+	if (likely(cmp))
+		return (cmp);
+
+	/*
+	 * If MMP_VALID(ub) && MMP_SEQ_VALID(ub) then the host has an MMP-aware
+	 * ZFS, e.g. zfsonlinux >= 0.7.
+	 *
+	 * If one ub has MMP and the other does not, they were written by
+	 * different hosts, which matters for MMP.  So we treat no MMP/no SEQ as
+	 * a 0 value.
+	 *
+	 * Since timestamp and txg are the same if we get this far, either is
+	 * acceptable for importing the pool.
+	 */
+	unsigned int seq1 = 0;
+	unsigned int seq2 = 0;
+
+	if (MMP_VALID(ub1) && MMP_SEQ_VALID(ub1))
+		seq1 = MMP_SEQ(ub1);
+
+	if (MMP_VALID(ub2) && MMP_SEQ_VALID(ub2))
+		seq2 = MMP_SEQ(ub2);
+
+	return (AVL_CMP(seq1, seq2));
 }
 
 struct ubl_cbdata {

Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h
==============================================================================
--- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h	Wed Nov  6 09:09:03 2019	(r354386)
@@ -600,6 +600,7 @@ typedef struct zpool_load_policy {
 #define	ZPOOL_CONFIG_CACHEFILE		"cachefile"	/* not stored on disk */
 #define	ZPOOL_CONFIG_MMP_STATE		"mmp_state"	/* not stored on disk */
 #define	ZPOOL_CONFIG_MMP_TXG		"mmp_txg"	/* not stored on disk */
+#define	ZPOOL_CONFIG_MMP_SEQ		"mmp_seq"	/* not stored on disk */
 #define	ZPOOL_CONFIG_MMP_HOSTNAME	"mmp_hostname"	/* not stored on disk */
 #define	ZPOOL_CONFIG_MMP_HOSTID		"mmp_hostid"	/* not stored on disk */
 #define	ZPOOL_CONFIG_ALLOCATION_BIAS	"alloc_bias"	/* not stored on disk */

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 09:09:05 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 202AD1AF63D;
 Wed,  6 Nov 2019 09:09:05 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477LLn06csz3LMD;
 Wed,  6 Nov 2019 09:09:05 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DC74E22B9C;
 Wed,  6 Nov 2019 09:09:04 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6994KO053259;
 Wed, 6 Nov 2019 09:09:04 GMT (envelope-from avg@FreeBSD.org)
Received: (from avg@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA6994ZX053258;
 Wed, 6 Nov 2019 09:09:04 GMT (envelope-from avg@FreeBSD.org)
Message-Id: <201911060909.xA6994ZX053258@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org
 using -f
From: Andriy Gapon 
Date: Wed, 6 Nov 2019 09:09:04 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354386 - vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
X-SVN-Group: vendor
X-SVN-Commit-Author: avg
X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs
 vendor-sys/illumos/dist/uts/common/fs/zfs/sys
 vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb
X-SVN-Commit-Revision: 354386
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 09:09:05 -0000

Author: avg
Date: Wed Nov  6 09:09:03 2019
New Revision: 354386
URL: https://svnweb.freebsd.org/changeset/base/354386

Log:
  11770 additional mmp fixes
  
  illumos/illumos-gate@4348eb901228d2f8fa50bb132a34248e8662074e
  https://github.com/illumos/illumos-gate/commit/4348eb901228d2f8fa50bb132a34248e8662074e
  
  https://www.illumos.org/issues/11770
    Port a few additional MMP fixes from ZoL that came in after our
    initial MMP port.
    4ca457b065 ZTS: Fix mmp_interval failure
    ca95f70dff zpool import progress kstat
    (only minimal changes from above can be pulled in right now)
    060f0226e6 MMP interval and fail_intervals in uberblock
  
  Portions contributed by: Jerry Jelinek 
  Portions contributed by: Brian Behlendorf 
  Author: Olaf Faaland 

Modified:
  vendor/illumos/dist/cmd/zdb/zdb.c

Changes in other areas also in this revision:
Modified:
  vendor-sys/illumos/dist/uts/common/fs/zfs/mmp.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/mmp.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/sys/uberblock_impl.h
  vendor-sys/illumos/dist/uts/common/fs/zfs/uberblock.c
  vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c
  vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h

Modified: vendor/illumos/dist/cmd/zdb/zdb.c
==============================================================================
--- vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 09:05:39 2019	(r354385)
+++ vendor/illumos/dist/cmd/zdb/zdb.c	Wed Nov  6 09:09:03 2019	(r354386)
@@ -2425,9 +2425,22 @@ dump_uberblock(uberblock_t *ub, const char *header, co
 
 	(void) printf("\tmmp_magic = %016llx\n",
 	    (u_longlong_t)ub->ub_mmp_magic);
-	if (ub->ub_mmp_magic == MMP_MAGIC)
+	if (MMP_VALID(ub)) {
 		(void) printf("\tmmp_delay = %0llu\n",
 		    (u_longlong_t)ub->ub_mmp_delay);
+		if (MMP_SEQ_VALID(ub))
+			(void) printf("\tmmp_seq = %u\n",
+			    (unsigned int) MMP_SEQ(ub));
+		if (MMP_FAIL_INT_VALID(ub))
+			(void) printf("\tmmp_fail = %u\n",
+			    (unsigned int) MMP_FAIL_INT(ub));
+		if (MMP_INTERVAL_VALID(ub))
+			(void) printf("\tmmp_write = %u\n",
+			    (unsigned int) MMP_INTERVAL(ub));
+		/* After MMP_* to make summarize_uberblock_mmp cleaner */
+		(void) printf("\tmmp_valid = %x\n",
+		    (unsigned int) ub->ub_mmp_config & 0xFF);
+	}
 
 	if (dump_opt['u'] >= 3) {
 		char blkbuf[BP_SPRINTF_LEN];

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 14:09:54 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 758101B7017;
 Wed,  6 Nov 2019 14:09:54 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477T1t3FNNz4939;
 Wed,  6 Nov 2019 14:09:54 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5348B2618D;
 Wed,  6 Nov 2019 14:09:54 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6E9sb4040776;
 Wed, 6 Nov 2019 14:09:54 GMT (envelope-from emaste@FreeBSD.org)
Received: (from emaste@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA6E9rgh040770;
 Wed, 6 Nov 2019 14:09:53 GMT (envelope-from emaste@FreeBSD.org)
Message-Id: <201911061409.xA6E9rgh040770@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: emaste set sender to
 emaste@FreeBSD.org using -f
From: Ed Maste 
Date: Wed, 6 Nov 2019 14:09:53 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354388 - in vendor/NetBSD/blacklist/dist: bin diff lib
X-SVN-Group: vendor
X-SVN-Commit-Author: emaste
X-SVN-Commit-Paths: in vendor/NetBSD/blacklist/dist: bin diff lib
X-SVN-Commit-Revision: 354388
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 14:09:54 -0000

Author: emaste
Date: Wed Nov  6 14:09:52 2019
New Revision: 354388
URL: https://svnweb.freebsd.org/changeset/base/354388

Log:
  Vendor import of NetBSD's external/bsd/blacklist @ 20191106
  
  Sponsored by:	The FreeBSD Foundation

Modified:
  vendor/NetBSD/blacklist/dist/bin/blacklistctl.c
  vendor/NetBSD/blacklist/dist/bin/blacklistd.c
  vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5
  vendor/NetBSD/blacklist/dist/bin/support.c
  vendor/NetBSD/blacklist/dist/diff/ssh.diff
  vendor/NetBSD/blacklist/dist/lib/Makefile
  vendor/NetBSD/blacklist/dist/lib/libblacklist.3

Modified: vendor/NetBSD/blacklist/dist/bin/blacklistctl.c
==============================================================================
--- vendor/NetBSD/blacklist/dist/bin/blacklistctl.c	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/bin/blacklistctl.c	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,4 +1,4 @@
-/*	$NetBSD: blacklistctl.c,v 1.21 2016/11/02 03:15:07 jnemeth Exp $	*/
+/*	$NetBSD: blacklistctl.c,v 1.23 2018/05/24 19:21:01 christos Exp $	*/
 
 /*-
  * Copyright (c) 2015 The NetBSD Foundation, Inc.
@@ -33,7 +33,7 @@
 #endif
 
 #include 
-__RCSID("$NetBSD: blacklistctl.c,v 1.21 2016/11/02 03:15:07 jnemeth Exp $");
+__RCSID("$NetBSD: blacklistctl.c,v 1.23 2018/05/24 19:21:01 christos Exp $");
 
 #include 
 #include 
@@ -67,6 +67,15 @@ usage(int c)
 	exit(EXIT_FAILURE);
 }
 
+static const char *
+star(char *buf, size_t len, int val)
+{
+	if (val == -1)
+		return "*";
+	snprintf(buf, len, "%d", val);
+	return buf;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -128,9 +137,10 @@ main(int argc, char *argv[])
 		    "address", remain ? "remaining time" : "last access");
 	for (i = 1; state_iterate(db, &c, &dbi, i) != 0; i = 0) {
 		char buf[BUFSIZ];
+		char mbuf[64], pbuf[64];
 		if (!all) {
 			if (blocked) {
-				if (dbi.count < c.c_nfail)
+				if (c.c_nfail == -1 || dbi.count < c.c_nfail)
 					continue;
 			} else {
 				if (dbi.count >= c.c_nfail)
@@ -138,13 +148,20 @@ main(int argc, char *argv[])
 			}
 		}
 		sockaddr_snprintf(buf, sizeof(buf), "%a", (void *)&c.c_ss);
-		printf("%*.*s/%d:%d\t", wide, wide, buf, c.c_lmask, c.c_port);
-		if (remain)
-			fmtydhms(buf, sizeof(buf),
-			    c.c_duration - (ts.tv_sec - dbi.last));
-		else
-			fmttime(buf, sizeof(buf), dbi.last);
-		printf("%s\t%d/%d\t%-s\n", dbi.id, dbi.count, c.c_nfail, buf);
+		printf("%*.*s/%s:%s\t", wide, wide, buf,
+		    star(mbuf, sizeof(mbuf), c.c_lmask),
+		    star(pbuf, sizeof(pbuf), c.c_port));
+		if (c.c_duration == -1) {
+			strlcpy(buf, "never", sizeof(buf));
+		} else {
+			if (remain)
+				fmtydhms(buf, sizeof(buf),
+				    c.c_duration - (ts.tv_sec - dbi.last));
+			else
+				fmttime(buf, sizeof(buf), dbi.last);
+		}
+		printf("%s\t%d/%s\t%-s\n", dbi.id, dbi.count,
+		    star(mbuf, sizeof(mbuf), c.c_nfail), buf);
 	}
 	state_close(db);
 	return EXIT_SUCCESS;

Modified: vendor/NetBSD/blacklist/dist/bin/blacklistd.c
==============================================================================
--- vendor/NetBSD/blacklist/dist/bin/blacklistd.c	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/bin/blacklistd.c	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,4 +1,4 @@
-/*	$NetBSD: blacklistd.c,v 1.37 2017/02/18 00:26:16 christos Exp $	*/
+/*	$NetBSD: blacklistd.c,v 1.38 2019/02/27 02:20:18 christos Exp $	*/
 
 /*-
  * Copyright (c) 2015 The NetBSD Foundation, Inc.
@@ -32,7 +32,7 @@
 #include "config.h"
 #endif
 #include 
-__RCSID("$NetBSD: blacklistd.c,v 1.37 2017/02/18 00:26:16 christos Exp $");
+__RCSID("$NetBSD: blacklistd.c,v 1.38 2019/02/27 02:20:18 christos Exp $");
 
 #include 
 #include 
@@ -394,7 +394,6 @@ rules_restore(void)
 	for (f = 1; state_iterate(state, &c, &dbi, f) == 1; f = 0) {
 		if (dbi.id[0] == '\0')
 			continue;
-		(void)run_change("rem", &c, dbi.id, 0);
 		(void)run_change("add", &c, dbi.id, sizeof(dbi.id));
 	}
 }
@@ -491,7 +490,8 @@ main(int argc, char *argv[])
 	conf_parse(configfile);
 	if (flush) {
 		rules_flush();
-		flags |= O_TRUNC;
+		if (!restore)
+			flags |= O_TRUNC;
 	}
 
 	struct pollfd *pfd = NULL;
@@ -522,8 +522,11 @@ main(int argc, char *argv[])
 	if (state == NULL)
 		return EXIT_FAILURE;
 
-	if (restore)
+	if (restore) {
+		if (!flush)
+			rules_flush();
 		rules_restore();
+	}
 
 	if (!debug) {
 		if (daemon(0, 0) == -1)

Modified: vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5
==============================================================================
--- vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,4 +1,4 @@
-.\" $NetBSD: blacklistd.conf.5,v 1.5 2016/06/08 12:48:37 wiz Exp $
+.\" $NetBSD: blacklistd.conf.5,v 1.7 2017/06/07 13:50:57 wiz Exp $
 .\"
 .\" Copyright (c) 2015 The NetBSD Foundation, Inc.
 .\" All rights reserved.
@@ -27,7 +27,7 @@
 .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 .\" POSSIBILITY OF SUCH DAMAGE.
 .\"
-.Dd June 7, 2016
+.Dd June 5, 2017
 .Dt BLACKLISTD.CONF 5
 .Os
 .Sh NAME
@@ -36,12 +36,13 @@
 .Sh DESCRIPTION
 The
 .Nm
-files contains configuration lines for
-.Xr blacklistd 8 .
-It contains one entry per line, and is similar to
+files contains configuration entries for
+.Xr blacklistd 8
+in a fashion similar to
 .Xr inetd.conf 5 .
-There must be an entry for each field of the configuration file, with
-entries for each field separated by a tab or a space.
+Only one entry per line is permitted.
+Every entry must have all fields populated.
+Each field can be separated by a tab or a space.
 Comments are denoted by a
 .Dq #
 at the beginning of a line.
@@ -109,7 +110,7 @@ The
 can be an IPv4 address in numeric format, an IPv6 address
 in numeric format and enclosed by square brackets, or an interface name.
 Mask modifiers are not allowed on interfaces because interfaces
-have multiple address in different protocols where the mask has a different
+can have multiple addresses in different protocols where the mask has a different
 size.
 .Pp
 The
@@ -150,8 +151,8 @@ If the
 contains a
 .Dq / ,
 the remaining portion of the name is interpreted as the mask to be
-applied to the address specified in the rule, so one can block whole
-subnets for a single rule violation.
+applied to the address specified in the rule, causing a single rule violation to
+block the entire subnet for the configured prefix.
 .Pp
 The
 .Va nfail
@@ -176,10 +177,11 @@ for days.
 .Pp
 Matching is done first by checking the
 .Va local
-rules one by one, from the most specific to the least specific.
+rules individually, in the order of the most specific to the least specific.
 If a match is found, then the
 .Va remote
-rules are applied, and if a match is found the
+rules are applied.
+The
 .Va name ,
 .Va nfail ,
 and
@@ -191,15 +193,15 @@ rule that matched.
 The
 .Va remote
 rules can be used for whitelisting specific addresses, changing the mask
-size, or the rule that the packet filter uses, the number of failed attempts,
-or the blocked duration.
+size, the rule that the packet filter uses, the number of failed attempts,
+or the block duration.
 .Sh FILES
 .Bl -tag -width /etc/blacklistd.conf -compact
 .It Pa /etc/blacklistd.conf
 Configuration file.
 .El
 .Sh EXAMPLES
-.Bd -literal -offset
+.Bd -literal -offset 8n
 # Block ssh, after 3 attempts for 6 hours on the bnx0 interface
 [local]
 # location	type	proto	owner	name	nfail	duration

Modified: vendor/NetBSD/blacklist/dist/bin/support.c
==============================================================================
--- vendor/NetBSD/blacklist/dist/bin/support.c	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/bin/support.c	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,4 +1,4 @@
-/*	$NetBSD: support.c,v 1.8 2016/04/04 15:52:56 christos Exp $	*/
+/*	$NetBSD: support.c,v 1.9 2018/09/18 22:12:19 christos Exp $	*/
 
 /*-
  * Copyright (c) 2015 The NetBSD Foundation, Inc.
@@ -33,7 +33,7 @@
 #endif
 
 #include 
-__RCSID("$NetBSD: support.c,v 1.8 2016/04/04 15:52:56 christos Exp $");
+__RCSID("$NetBSD: support.c,v 1.9 2018/09/18 22:12:19 christos Exp $");
 
 #include 
 #include 
@@ -105,12 +105,16 @@ fmtydhms(char *b, size_t l, time_t t)
 
 	s = t % 60;
 	t /= 60;
+
 	m = t % 60;
 	t /= 60;
-	h = t % 60;
+
+	h = t % 24;
 	t /= 24;
-	d = t % 24;
-	t /= 356;
+
+	d = t % 365;
+	t /= 365;
+
 	y = t;
 
 	z = 0;

Modified: vendor/NetBSD/blacklist/dist/diff/ssh.diff
==============================================================================
--- vendor/NetBSD/blacklist/dist/diff/ssh.diff	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/diff/ssh.diff	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,6 +1,6 @@
 --- /dev/null	2015-01-22 23:10:33.000000000 -0500
 +++ dist/pfilter.c	2015-01-22 23:46:03.000000000 -0500
-@@ -0,0 +1,28 @@
+@@ -0,0 +1,32 @@
 +#include "namespace.h"
 +#include "includes.h"
 +#include "ssh.h"
@@ -28,6 +28,10 @@
 +	// XXX: 3?
 + 	fd = packet_connection_is_on_socket() ? packet_get_connection_in() : 3;
 +	(void)blacklist_r(blstate, a, fd, "ssh");
++	if (a == 0) {
++		blacklist_close(blstate);
++		blstate = NULL;
++	}
 +}
 --- /dev/null	2015-01-20 21:14:44.000000000 -0500
 +++ dist/pfilter.h	2015-01-20 20:16:20.000000000 -0500
@@ -58,174 +62,89 @@ diff -u -u -r1.10 Makefile
 +
 +LDADD+=	-lblacklist
 +DPADD+=	${LIBBLACKLIST}
-Index: dist/auth.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth.c,v
-retrieving revision 1.10
-diff -u -u -r1.10 auth.c
---- dist/auth.c	19 Oct 2014 16:30:58 -0000	1.10
-+++ dist/auth.c	22 Jan 2015 21:39:22 -0000
-@@ -62,6 +62,7 @@
- #include "monitor_wrap.h"
- #include "krl.h"
- #include "compat.h"
-+#include "pfilter.h"
- 
- #ifdef HAVE_LOGIN_CAP
- #include 
-@@ -362,6 +363,8 @@
- 	    compat20 ? "ssh2" : "ssh1",
- 	    authctxt->info != NULL ? ": " : "",
- 	    authctxt->info != NULL ? authctxt->info : "");
-+	if (!authctxt->postponed)
-+		pfilter_notify(!authenticated);
- 	free(authctxt->info);
- 	authctxt->info = NULL;
- }
-Index: dist/sshd.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/sshd.c,v
-retrieving revision 1.15
-diff -u -u -r1.15 sshd.c
---- dist/sshd.c	28 Oct 2014 21:36:16 -0000	1.15
-+++ dist/sshd.c	22 Jan 2015 21:39:22 -0000
-@@ -109,6 +109,7 @@
- #include "roaming.h"
- #include "ssh-sandbox.h"
- #include "version.h"
-+#include "pfilter.h"
- 
- #ifdef LIBWRAP
- #include 
-@@ -364,6 +365,7 @@
- 		killpg(0, SIGTERM);
- 	}
- 
-+	pfilter_notify(1);
- 	/* Log error and exit. */
- 	sigdie("Timeout before authentication for %s", get_remote_ipaddr());
- }
-@@ -1160,6 +1162,7 @@
- 	for (i = 0; i < options.max_startups; i++)
- 		startup_pipes[i] = -1;
- 
-+	pfilter_init();
- 	/*
- 	 * Stay listening for connections until the system crashes or
- 	 * the daemon is killed with a signal.
-Index: auth1.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth1.c,v
-retrieving revision 1.9
-diff -u -u -r1.9 auth1.c
---- auth1.c	19 Oct 2014 16:30:58 -0000	1.9
-+++ auth1.c	14 Feb 2015 15:40:51 -0000
-@@ -41,6 +41,7 @@
+diff -ru openssh-7.7p1/auth-pam.c dist/auth-pam.c
+--- openssh-7.7p1/auth-pam.c	2018-04-02 01:38:28.000000000 -0400
++++ dist/auth-pam.c	2018-05-23 11:56:22.206661484 -0400
+@@ -103,6 +103,7 @@
+ #include "ssh-gss.h"
  #endif
  #include "monitor_wrap.h"
- #include "buffer.h"
 +#include "pfilter.h"
  
- /* import */
  extern ServerOptions options;
-@@ -445,6 +446,7 @@
- 	else {
- 		debug("do_authentication: invalid user %s", user);
- 		authctxt->pw = fakepw();
-+		pfilter_notify(1);
- 	}
+ extern Buffer loginmsg;
+@@ -526,6 +527,7 @@
+ 		ssh_msg_send(ctxt->pam_csock, PAM_MAXTRIES, &buffer);
+ 	else
+ 		ssh_msg_send(ctxt->pam_csock, PAM_AUTH_ERR, &buffer);
++	pfilter_notify(1);
+ 	buffer_free(&buffer);
+ 	pthread_exit(NULL);
  
- 	/* Configuration may have changed as a result of Match */
-Index: auth2.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth2.c,v
-retrieving revision 1.9
-diff -u -u -r1.9 auth2.c
---- auth2.c	19 Oct 2014 16:30:58 -0000	1.9
-+++ auth2.c	14 Feb 2015 15:40:51 -0000
-@@ -52,6 +52,7 @@
+@@ -804,6 +806,7 @@
+ 				free(msg);
+ 				return (0);
+ 			}
++			pfilter_notify(1);
+ 			error("PAM: %s for %s%.100s from %.100s", msg,
+ 			    sshpam_authctxt->valid ? "" : "illegal user ",
+ 			    sshpam_authctxt->user,
+diff -ru openssh-7.7p1/auth2.c dist/auth2.c
+--- openssh-7.7p1/auth2.c	2018-04-02 01:38:28.000000000 -0400
++++ dist/auth2.c	2018-05-23 11:57:31.022197317 -0400
+@@ -51,6 +51,7 @@
+ #include "dispatch.h"
  #include "pathnames.h"
  #include "buffer.h"
- #include "canohost.h"
 +#include "pfilter.h"
  
  #ifdef GSSAPI
  #include "ssh-gss.h"
-@@ -256,6 +257,7 @@
+@@ -242,6 +243,7 @@
  		} else {
- 			logit("input_userauth_request: invalid user %s", user);
+ 			/* Invalid user, fake password information */
  			authctxt->pw = fakepw();
 +			pfilter_notify(1);
- 		}
- #ifdef USE_PAM
- 		if (options.use_pam)
-Index: sshd.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/sshd.c,v
-retrieving revision 1.16
-diff -u -r1.16 sshd.c
---- sshd.c	25 Jan 2015 15:52:44 -0000	1.16
-+++ sshd.c	14 Feb 2015 09:55:06 -0000
-@@ -628,6 +628,8 @@
- 	explicit_bzero(pw->pw_passwd, strlen(pw->pw_passwd));
- 	endpwent();
- 
-+	pfilter_init();
-+
- 	/* Change our root directory */
- 	if (chroot(_PATH_PRIVSEP_CHROOT_DIR) == -1)
- 		fatal("chroot(\"%s\"): %s", _PATH_PRIVSEP_CHROOT_DIR,
-
-Index: auth-pam.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth-pam.c,v
-retrieving revision 1.7
-diff -u -u -r1.7 auth-pam.c
---- auth-pam.c	3 Jul 2015 00:59:59 -0000	1.7
-+++ auth-pam.c	23 Jan 2016 00:01:16 -0000
-@@ -114,6 +114,7 @@
- #include "ssh-gss.h"
+ #ifdef SSH_AUDIT_EVENTS
+ 			PRIVSEP(audit_event(SSH_INVALID_USER));
  #endif
- #include "monitor_wrap.h"
+Only in dist: pfilter.c
+Only in dist: pfilter.h
+diff -ru openssh-7.7p1/sshd.c dist/sshd.c
+--- openssh-7.7p1/sshd.c	2018-04-02 01:38:28.000000000 -0400
++++ dist/sshd.c	2018-05-23 11:59:39.573197347 -0400
+@@ -122,6 +122,7 @@
+ #include "auth-options.h"
+ #include "version.h"
+ #include "ssherr.h"
 +#include "pfilter.h"
  
- extern ServerOptions options;
- extern Buffer loginmsg;
-@@ -809,6 +810,7 @@
- 				free(msg);
- 				return (0);
- 			}
-+			pfilter_notify(1);
- 			error("PAM: %s for %s%.100s from %.100s", msg,
- 			    sshpam_authctxt->valid ? "" : "illegal user ",
- 			    sshpam_authctxt->user,
-Index: auth.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth.c,v
-retrieving revision 1.15
-diff -u -u -r1.15 auth.c
---- auth.c	21 Aug 2015 08:20:59 -0000	1.15
-+++ auth.c	23 Jan 2016 00:01:16 -0000
-@@ -656,6 +656,7 @@
+ /* Re-exec fds */
+ #define REEXEC_DEVCRYPTO_RESERVED_FD	(STDERR_FILENO + 1)
+@@ -346,6 +347,7 @@
+ static void
+ grace_alarm_handler(int sig)
+ {
++	pfilter_notify(1);
+ 	if (use_privsep && pmonitor != NULL && pmonitor->m_pid > 0)
+ 		kill(pmonitor->m_pid, SIGALRM);
  
- 	pw = getpwnam(user);
- 	if (pw == NULL) {
-+		pfilter_notify(1);
- 		logit("Invalid user %.100s from %.100s",
- 		    user, get_remote_ipaddr());
- 		return (NULL);
-Index: auth1.c
-===================================================================
-RCS file: /cvsroot/src/crypto/external/bsd/openssh/dist/auth1.c,v
-retrieving revision 1.12
-diff -u -u -r1.12 auth1.c
---- auth1.c	3 Jul 2015 00:59:59 -0000	1.12
-+++ auth1.c	23 Jan 2016 00:01:16 -0000
-@@ -376,6 +376,7 @@
- 			char *msg;
- 			size_t len;
+@@ -1835,6 +1837,8 @@
+ 	if (test_flag)
+ 		exit(0);
  
-+			pfilter_notify(1);
- 			error("Access denied for user %s by PAM account "
- 			    "configuration", authctxt->user);
- 			len = buffer_len(&loginmsg);
++	pfilter_init();
++
+ 	/*
+ 	 * Clear out any supplemental groups we may have inherited.  This
+ 	 * prevents inadvertent creation of files with bad modes (in the
+@@ -2280,6 +2284,9 @@
+ {
+ 	struct ssh *ssh = active_state; /* XXX */
+ 
++	if (i == 255)
++		pfilter_notify(1);
++
+ 	if (the_authctxt) {
+ 		do_cleanup(ssh, the_authctxt);
+ 		if (use_privsep && privsep_is_preauth &&

Modified: vendor/NetBSD/blacklist/dist/lib/Makefile
==============================================================================
--- vendor/NetBSD/blacklist/dist/lib/Makefile	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/lib/Makefile	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,11 +1,11 @@
-# $NetBSD: Makefile,v 1.6 2016/01/05 13:07:46 christos Exp $
+# $NetBSD: Makefile,v 1.7 2019/03/08 20:40:05 christos Exp $
 
 .include 
 
 USE_SHLIBDIR=   yes
 
 CPPFLAGS+=-D_REENTRANT
-LIBDPLIBS+=pthread ${NETBSDSRCDIR}/lib/libpthread
+#LIBDPLIBS+=pthread ${NETBSDSRCDIR}/lib/libpthread
 LIB=blacklist
 SRCS=bl.c blacklist.c
 MAN=libblacklist.3

Modified: vendor/NetBSD/blacklist/dist/lib/libblacklist.3
==============================================================================
--- vendor/NetBSD/blacklist/dist/lib/libblacklist.3	Wed Nov  6 13:45:24 2019	(r354387)
+++ vendor/NetBSD/blacklist/dist/lib/libblacklist.3	Wed Nov  6 14:09:52 2019	(r354388)
@@ -1,4 +1,4 @@
-.\" $NetBSD: libblacklist.3,v 1.7 2017/02/04 23:33:56 wiz Exp $
+.\" $NetBSD: libblacklist.3,v 1.8 2017/10/22 10:31:57 abhinav Exp $
 .\"
 .\" Copyright (c) 2015 The NetBSD Foundation, Inc.
 .\" All rights reserved.
@@ -35,7 +35,7 @@
 .Nm blacklist_close ,
 .Nm blacklist_r ,
 .Nm blacklist ,
-.Nm blacklist_sa
+.Nm blacklist_sa ,
 .Nm blacklist_sa_r
 .Nd Blacklistd notification library
 .Sh LIBRARY

From owner-svn-src-vendor@freebsd.org  Wed Nov  6 14:10:51 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 39B971B708D;
 Wed,  6 Nov 2019 14:10:51 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477T2z1W2xz49KF;
 Wed,  6 Nov 2019 14:10:51 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 16D01261B3;
 Wed,  6 Nov 2019 14:10:51 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA6EApdR043162;
 Wed, 6 Nov 2019 14:10:51 GMT (envelope-from emaste@FreeBSD.org)
Received: (from emaste@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA6EAoK3043157;
 Wed, 6 Nov 2019 14:10:50 GMT (envelope-from emaste@FreeBSD.org)
Message-Id: <201911061410.xA6EAoK3043157@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: emaste set sender to
 emaste@FreeBSD.org using -f
From: Ed Maste 
Date: Wed, 6 Nov 2019 14:10:50 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354389 - in vendor/NetBSD/blacklist/20191106: . bin diff
 lib
X-SVN-Group: vendor
X-SVN-Commit-Author: emaste
X-SVN-Commit-Paths: in vendor/NetBSD/blacklist/20191106: . bin diff lib
X-SVN-Commit-Revision: 354389
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Wed, 06 Nov 2019 14:10:51 -0000

Author: emaste
Date: Wed Nov  6 14:10:50 2019
New Revision: 354389
URL: https://svnweb.freebsd.org/changeset/base/354389

Log:
  Tag import of NetBSD's external/bsd/blacklist @ 20191106
  
  Sponsored by:	The FreeBSD Foundation

Added:
  vendor/NetBSD/blacklist/20191106/
     - copied from r354387, vendor/NetBSD/blacklist/dist/
Replaced:
  vendor/NetBSD/blacklist/20191106/bin/blacklistctl.c
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistctl.c
  vendor/NetBSD/blacklist/20191106/bin/blacklistd.c
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.c
  vendor/NetBSD/blacklist/20191106/bin/blacklistd.conf.5
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5
  vendor/NetBSD/blacklist/20191106/bin/support.c
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/bin/support.c
  vendor/NetBSD/blacklist/20191106/diff/ssh.diff
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/diff/ssh.diff
  vendor/NetBSD/blacklist/20191106/lib/Makefile
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/lib/Makefile
  vendor/NetBSD/blacklist/20191106/lib/libblacklist.3
     - copied unchanged from r354388, vendor/NetBSD/blacklist/dist/lib/libblacklist.3

Copied: vendor/NetBSD/blacklist/20191106/bin/blacklistctl.c (from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistctl.c)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/NetBSD/blacklist/20191106/bin/blacklistctl.c	Wed Nov  6 14:10:50 2019	(r354389, copy of r354388, vendor/NetBSD/blacklist/dist/bin/blacklistctl.c)
@@ -0,0 +1,168 @@
+/*	$NetBSD: blacklistctl.c,v 1.23 2018/05/24 19:21:01 christos Exp $	*/
+
+/*-
+ * Copyright (c) 2015 The NetBSD Foundation, Inc.
+ * All rights reserved.
+ *
+ * This code is derived from software contributed to The NetBSD Foundation
+ * by Christos Zoulas.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif
+
+#include 
+__RCSID("$NetBSD: blacklistctl.c,v 1.23 2018/05/24 19:21:01 christos Exp $");
+
+#include 
+#include 
+#ifdef HAVE_LIBUTIL_H
+#include 
+#endif
+#ifdef HAVE_UTIL_H
+#include 
+#endif
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "conf.h"
+#include "state.h"
+#include "internal.h"
+#include "support.h"
+
+static __dead void
+usage(int c)
+{
+	if (c == 0)
+		warnx("Missing/unknown command");
+	else
+		warnx("Unknown option `%c'", (char)c);
+	fprintf(stderr, "Usage: %s dump [-abdnrw]\n", getprogname());
+	exit(EXIT_FAILURE);
+}
+
+static const char *
+star(char *buf, size_t len, int val)
+{
+	if (val == -1)
+		return "*";
+	snprintf(buf, len, "%d", val);
+	return buf;
+}
+
+int
+main(int argc, char *argv[])
+{
+	const char *dbname = _PATH_BLSTATE;
+	DB *db;
+	struct conf c;
+	struct dbinfo dbi;
+	unsigned int i;
+	struct timespec ts;
+	int all, blocked, remain, wide, noheader;
+	int o;
+
+	noheader = wide = blocked = all = remain = 0;
+	lfun = dlog;
+
+	if (argc == 1 || strcmp(argv[1], "dump") != 0)
+		usage(0);
+
+	argc--;
+	argv++;
+
+	while ((o = getopt(argc, argv, "abD:dnrw")) != -1)
+		switch (o) {
+		case 'a':
+			all = 1;
+			blocked = 0;
+			break;
+		case 'b':
+			blocked = 1;
+			break;
+		case 'D':
+			dbname = optarg;
+			break;
+		case 'd':
+			debug++;
+			break;
+		case 'n':
+			noheader = 1;
+			break;
+		case 'r':
+			remain = 1;
+			break;
+		case 'w':
+			wide = 1;
+			break;
+		default:
+			usage(o);
+			break;
+		}
+
+	db = state_open(dbname, O_RDONLY, 0);
+	if (db == NULL)
+		err(EXIT_FAILURE, "Can't open `%s'", dbname);
+
+	clock_gettime(CLOCK_REALTIME, &ts);
+	wide = wide ? 8 * 4 + 7 : 4 * 3 + 3;
+	if (!noheader)
+		printf("%*.*s/ma:port\tid\tnfail\t%s\n", wide, wide,
+		    "address", remain ? "remaining time" : "last access");
+	for (i = 1; state_iterate(db, &c, &dbi, i) != 0; i = 0) {
+		char buf[BUFSIZ];
+		char mbuf[64], pbuf[64];
+		if (!all) {
+			if (blocked) {
+				if (c.c_nfail == -1 || dbi.count < c.c_nfail)
+					continue;
+			} else {
+				if (dbi.count >= c.c_nfail)
+					continue;
+			}
+		}
+		sockaddr_snprintf(buf, sizeof(buf), "%a", (void *)&c.c_ss);
+		printf("%*.*s/%s:%s\t", wide, wide, buf,
+		    star(mbuf, sizeof(mbuf), c.c_lmask),
+		    star(pbuf, sizeof(pbuf), c.c_port));
+		if (c.c_duration == -1) {
+			strlcpy(buf, "never", sizeof(buf));
+		} else {
+			if (remain)
+				fmtydhms(buf, sizeof(buf),
+				    c.c_duration - (ts.tv_sec - dbi.last));
+			else
+				fmttime(buf, sizeof(buf), dbi.last);
+		}
+		printf("%s\t%d/%s\t%-s\n", dbi.id, dbi.count,
+		    star(mbuf, sizeof(mbuf), c.c_nfail), buf);
+	}
+	state_close(db);
+	return EXIT_SUCCESS;
+}

Copied: vendor/NetBSD/blacklist/20191106/bin/blacklistd.c (from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.c)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/NetBSD/blacklist/20191106/bin/blacklistd.c	Wed Nov  6 14:10:50 2019	(r354389, copy of r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.c)
@@ -0,0 +1,568 @@
+/*	$NetBSD: blacklistd.c,v 1.38 2019/02/27 02:20:18 christos Exp $	*/
+
+/*-
+ * Copyright (c) 2015 The NetBSD Foundation, Inc.
+ * All rights reserved.
+ *
+ * This code is derived from software contributed to The NetBSD Foundation
+ * by Christos Zoulas.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif
+#include 
+__RCSID("$NetBSD: blacklistd.c,v 1.38 2019/02/27 02:20:18 christos Exp $");
+
+#include 
+#include 
+#include 
+
+#ifdef HAVE_LIBUTIL_H
+#include 
+#endif
+#ifdef HAVE_UTIL_H
+#include 
+#endif
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "bl.h"
+#include "internal.h"
+#include "conf.h"
+#include "run.h"
+#include "state.h"
+#include "support.h"
+
+static const char *configfile = _PATH_BLCONF;
+static DB *state;
+static const char *dbfile = _PATH_BLSTATE;
+static sig_atomic_t readconf;
+static sig_atomic_t done;
+static int vflag;
+
+static void
+sigusr1(int n __unused)
+{
+	debug++;
+}
+
+static void
+sigusr2(int n __unused)
+{
+	debug--;
+}
+
+static void
+sighup(int n __unused)
+{
+	readconf++;
+}
+
+static void
+sigdone(int n __unused)
+{
+	done++;
+}
+
+static __dead void
+usage(int c)
+{
+	if (c)
+		warnx("Unknown option `%c'", (char)c);
+	fprintf(stderr, "Usage: %s [-vdfr] [-c ] [-R ] "
+	    "[-P ] [-C ] [-D ] "
+	    "[-s ] [-t ]\n", getprogname());
+	exit(EXIT_FAILURE);
+}
+
+static int
+getremoteaddress(bl_info_t *bi, struct sockaddr_storage *rss, socklen_t *rsl)
+{
+	*rsl = sizeof(*rss);
+	memset(rss, 0, *rsl);
+
+	if (getpeername(bi->bi_fd, (void *)rss, rsl) != -1)
+		return 0;
+
+	if (errno != ENOTCONN) {
+		(*lfun)(LOG_ERR, "getpeername failed (%m)"); 
+		return -1;
+	}
+
+	if (bi->bi_slen == 0) {
+		(*lfun)(LOG_ERR, "unconnected socket with no peer in message");
+		return -1;
+	}
+
+	switch (bi->bi_ss.ss_family) {
+	case AF_INET:
+		*rsl = sizeof(struct sockaddr_in);
+		break;
+	case AF_INET6:
+		*rsl = sizeof(struct sockaddr_in6);
+		break;
+	default:
+		(*lfun)(LOG_ERR, "bad client passed socket family %u",
+		    (unsigned)bi->bi_ss.ss_family); 
+		return -1;
+	}
+
+	if (*rsl != bi->bi_slen) {
+		(*lfun)(LOG_ERR, "bad client passed socket length %u != %u",
+		    (unsigned)*rsl, (unsigned)bi->bi_slen); 
+		return -1;
+	}
+
+	memcpy(rss, &bi->bi_ss, *rsl);
+
+#ifdef HAVE_STRUCT_SOCKADDR_SA_LEN
+	if (*rsl != rss->ss_len) {
+		(*lfun)(LOG_ERR,
+		    "bad client passed socket internal length %u != %u",
+		    (unsigned)*rsl, (unsigned)rss->ss_len); 
+		return -1;
+	}
+#endif
+	return 0;
+}
+
+static void
+process(bl_t bl)
+{
+	struct sockaddr_storage rss;
+	socklen_t rsl;
+	char rbuf[BUFSIZ];
+	bl_info_t *bi;
+	struct conf c;
+	struct dbinfo dbi;
+	struct timespec ts;
+
+	if (clock_gettime(CLOCK_REALTIME, &ts) == -1) {
+		(*lfun)(LOG_ERR, "clock_gettime failed (%m)"); 
+		return;
+	}
+
+	if ((bi = bl_recv(bl)) == NULL) {
+		(*lfun)(LOG_ERR, "no message (%m)"); 
+		return;
+	}
+
+	if (getremoteaddress(bi, &rss, &rsl) == -1)
+		goto out;
+
+	if (debug) {
+		sockaddr_snprintf(rbuf, sizeof(rbuf), "%a:%p", (void *)&rss);
+		(*lfun)(LOG_DEBUG, "processing type=%d fd=%d remote=%s msg=%s"
+		    " uid=%lu gid=%lu", bi->bi_type, bi->bi_fd, rbuf,
+		    bi->bi_msg, (unsigned long)bi->bi_uid,
+		    (unsigned long)bi->bi_gid);
+	}
+
+	if (conf_find(bi->bi_fd, bi->bi_uid, &rss, &c) == NULL) {
+		(*lfun)(LOG_DEBUG, "no rule matched");
+		goto out;
+	}
+
+
+	if (state_get(state, &c, &dbi) == -1)
+		goto out;
+
+	if (debug) {
+		char b1[128], b2[128];
+		(*lfun)(LOG_DEBUG, "%s: initial db state for %s: count=%d/%d "
+		    "last=%s now=%s", __func__, rbuf, dbi.count, c.c_nfail,
+		    fmttime(b1, sizeof(b1), dbi.last),
+		    fmttime(b2, sizeof(b2), ts.tv_sec));
+	}
+
+	switch (bi->bi_type) {
+	case BL_ADD:
+		dbi.count++;
+		dbi.last = ts.tv_sec;
+		if (dbi.id[0]) {
+			/*
+			 * We should not be getting this since the rule
+			 * should have blocked the address. A possible
+			 * explanation is that someone removed that rule,
+			 * and another would be that we got another attempt
+			 * before we added the rule. In anycase, we remove
+			 * and re-add the rule because we don't want to add
+			 * it twice, because then we'd lose track of it.
+			 */
+			(*lfun)(LOG_DEBUG, "rule exists %s", dbi.id);
+			(void)run_change("rem", &c, dbi.id, 0);
+			dbi.id[0] = '\0';
+		}
+		if (c.c_nfail != -1 && dbi.count >= c.c_nfail) {
+			int res = run_change("add", &c, dbi.id, sizeof(dbi.id));
+			if (res == -1)
+				goto out;
+			sockaddr_snprintf(rbuf, sizeof(rbuf), "%a",
+			    (void *)&rss);
+			(*lfun)(LOG_INFO,
+			    "blocked %s/%d:%d for %d seconds",
+			    rbuf, c.c_lmask, c.c_port, c.c_duration);
+				
+		}
+		break;
+	case BL_DELETE:
+		if (dbi.last == 0)
+			goto out;
+		dbi.count = 0;
+		dbi.last = 0;
+		break;
+	default:
+		(*lfun)(LOG_ERR, "unknown message %d", bi->bi_type); 
+	}
+	state_put(state, &c, &dbi);
+
+out:
+	close(bi->bi_fd);
+
+	if (debug) {
+		char b1[128], b2[128];
+		(*lfun)(LOG_DEBUG, "%s: final db state for %s: count=%d/%d "
+		    "last=%s now=%s", __func__, rbuf, dbi.count, c.c_nfail,
+		    fmttime(b1, sizeof(b1), dbi.last),
+		    fmttime(b2, sizeof(b2), ts.tv_sec));
+	}
+}
+
+static void
+update_interfaces(void)
+{
+	struct ifaddrs *oifas, *nifas;
+
+	if (getifaddrs(&nifas) == -1)
+		return;
+
+	oifas = ifas;
+	ifas = nifas;
+
+	if (oifas)
+		freeifaddrs(oifas);
+}
+
+static void
+update(void)
+{
+	struct timespec ts;
+	struct conf c;
+	struct dbinfo dbi;
+	unsigned int f, n;
+	char buf[128];
+	void *ss = &c.c_ss;
+
+	if (clock_gettime(CLOCK_REALTIME, &ts) == -1) {
+		(*lfun)(LOG_ERR, "clock_gettime failed (%m)"); 
+		return;
+	}
+
+again:
+	for (n = 0, f = 1; state_iterate(state, &c, &dbi, f) == 1;
+	    f = 0, n++)
+	{
+		time_t when = c.c_duration + dbi.last;
+		if (debug > 1) {
+			char b1[64], b2[64];
+			sockaddr_snprintf(buf, sizeof(buf), "%a:%p", ss);
+			(*lfun)(LOG_DEBUG, "%s:[%u] %s count=%d duration=%d "
+			    "last=%s " "now=%s", __func__, n, buf, dbi.count,
+			    c.c_duration, fmttime(b1, sizeof(b1), dbi.last),
+			    fmttime(b2, sizeof(b2), ts.tv_sec));
+		}
+		if (c.c_duration == -1 || when >= ts.tv_sec)
+			continue;
+		if (dbi.id[0]) {
+			run_change("rem", &c, dbi.id, 0);
+			sockaddr_snprintf(buf, sizeof(buf), "%a", ss);
+			syslog(LOG_INFO, "released %s/%d:%d after %d seconds",
+			    buf, c.c_lmask, c.c_port, c.c_duration);
+		}
+		state_del(state, &c);
+		goto again;
+	}
+}
+
+static void
+addfd(struct pollfd **pfdp, bl_t **blp, size_t *nfd, size_t *maxfd,
+    const char *path)
+{
+	bl_t bl = bl_create(true, path, vflag ? vdlog : vsyslog);
+	if (bl == NULL || !bl_isconnected(bl))
+		exit(EXIT_FAILURE);
+	if (*nfd >= *maxfd) {
+		*maxfd += 10;
+		*blp = realloc(*blp, sizeof(**blp) * *maxfd);
+		if (*blp == NULL)
+			err(EXIT_FAILURE, "malloc");
+		*pfdp = realloc(*pfdp, sizeof(**pfdp) * *maxfd);
+		if (*pfdp == NULL)
+			err(EXIT_FAILURE, "malloc");
+	}
+
+	(*pfdp)[*nfd].fd = bl_getfd(bl);
+	(*pfdp)[*nfd].events = POLLIN;
+	(*blp)[*nfd] = bl;
+	*nfd += 1;
+}
+
+static void
+uniqueadd(struct conf ***listp, size_t *nlist, size_t *mlist, struct conf *c)
+{
+	struct conf **list = *listp;
+
+	if (c->c_name[0] == '\0')
+		return;
+	for (size_t i = 0; i < *nlist; i++) {
+		if (strcmp(list[i]->c_name, c->c_name) == 0)
+			return;
+	}
+	if (*nlist == *mlist) {
+		*mlist += 10;
+		void *p = realloc(*listp, *mlist * sizeof(*list));
+		if (p == NULL)
+			err(EXIT_FAILURE, "Can't allocate for rule list");
+		list = *listp = p;
+	}
+	list[(*nlist)++] = c;
+}
+
+static void
+rules_flush(void)
+{
+	struct conf **list;
+	size_t nlist, mlist;
+
+	list = NULL;
+	mlist = nlist = 0;
+	for (size_t i = 0; i < rconf.cs_n; i++)
+		uniqueadd(&list, &nlist, &mlist, &rconf.cs_c[i]);
+	for (size_t i = 0; i < lconf.cs_n; i++)
+		uniqueadd(&list, &nlist, &mlist, &lconf.cs_c[i]);
+
+	for (size_t i = 0; i < nlist; i++)
+		run_flush(list[i]);
+	free(list);
+}
+
+static void
+rules_restore(void)
+{
+	struct conf c;
+	struct dbinfo dbi;
+	unsigned int f;
+
+	for (f = 1; state_iterate(state, &c, &dbi, f) == 1; f = 0) {
+		if (dbi.id[0] == '\0')
+			continue;
+		(void)run_change("add", &c, dbi.id, sizeof(dbi.id));
+	}
+}
+
+int
+main(int argc, char *argv[])
+{
+	int c, tout, flags, flush, restore, ret;
+	const char *spath, **blsock;
+	size_t nblsock, maxblsock;
+
+	setprogname(argv[0]);
+
+	spath = NULL;
+	blsock = NULL;
+	maxblsock = nblsock = 0;
+	flush = 0;
+	restore = 0;
+	tout = 0;
+	flags = O_RDWR|O_EXCL|O_CLOEXEC;
+	while ((c = getopt(argc, argv, "C:c:D:dfP:rR:s:t:v")) != -1) {
+		switch (c) {
+		case 'C':
+			controlprog = optarg;
+			break;
+		case 'c':
+			configfile = optarg;
+			break;
+		case 'D':
+			dbfile = optarg;
+			break;
+		case 'd':
+			debug++;
+			break;
+		case 'f':
+			flush++;
+			break;
+		case 'P':
+			spath = optarg;
+			break;
+		case 'R':
+			rulename = optarg;
+			break;
+		case 'r':
+			restore++;
+			break;
+		case 's':
+			if (nblsock >= maxblsock) {
+				maxblsock += 10;
+				void *p = realloc(blsock,
+				    sizeof(*blsock) * maxblsock);
+				if (p == NULL)
+				    err(EXIT_FAILURE,
+					"Can't allocate memory for %zu sockets",
+					maxblsock);
+				blsock = p;
+			}
+			blsock[nblsock++] = optarg;
+			break;
+		case 't':
+			tout = atoi(optarg) * 1000;
+			break;
+		case 'v':
+			vflag++;
+			break;
+		default:
+			usage(c);
+		}
+	}
+
+	argc -= optind;
+	if (argc)
+		usage(0);
+
+	signal(SIGHUP, sighup);
+	signal(SIGINT, sigdone);
+	signal(SIGQUIT, sigdone);
+	signal(SIGTERM, sigdone);
+	signal(SIGUSR1, sigusr1);
+	signal(SIGUSR2, sigusr2);
+
+	openlog(getprogname(), LOG_PID, LOG_DAEMON);
+
+	if (debug) {
+		lfun = dlog;
+		if (tout == 0)
+			tout = 5000;
+	} else {
+		if (tout == 0)
+			tout = 15000;
+	}
+
+	update_interfaces();
+	conf_parse(configfile);
+	if (flush) {
+		rules_flush();
+		if (!restore)
+			flags |= O_TRUNC;
+	}
+
+	struct pollfd *pfd = NULL;
+	bl_t *bl = NULL;
+	size_t nfd = 0;
+	size_t maxfd = 0;
+
+	for (size_t i = 0; i < nblsock; i++)
+		addfd(&pfd, &bl, &nfd, &maxfd, blsock[i]);
+	free(blsock);
+
+	if (spath) {
+		FILE *fp = fopen(spath, "r");
+		char *line;
+		if (fp == NULL)
+			err(EXIT_FAILURE, "Can't open `%s'", spath);
+		for (; (line = fparseln(fp, NULL, NULL, NULL, 0)) != NULL;
+		    free(line))
+			addfd(&pfd, &bl, &nfd, &maxfd, line);
+		fclose(fp);
+	}
+	if (nfd == 0)
+		addfd(&pfd, &bl, &nfd, &maxfd, _PATH_BLSOCK);
+
+	state = state_open(dbfile, flags, 0600);
+	if (state == NULL)
+		state = state_open(dbfile,  flags | O_CREAT, 0600);
+	if (state == NULL)
+		return EXIT_FAILURE;
+
+	if (restore) {
+		if (!flush)
+			rules_flush();
+		rules_restore();
+	}
+
+	if (!debug) {
+		if (daemon(0, 0) == -1)
+			err(EXIT_FAILURE, "daemon failed");
+		if (pidfile(NULL) == -1)
+			err(EXIT_FAILURE, "Can't create pidfile");
+	}
+
+	for (size_t t = 0; !done; t++) {
+		if (readconf) {
+			readconf = 0;
+			conf_parse(configfile);
+		}
+		ret = poll(pfd, (nfds_t)nfd, tout);
+		if (debug)
+			(*lfun)(LOG_DEBUG, "received %d from poll()", ret);
+		switch (ret) {
+		case -1:
+			if (errno == EINTR)
+				continue;
+			(*lfun)(LOG_ERR, "poll (%m)");
+			return EXIT_FAILURE;
+		case 0:
+			state_sync(state);
+			break;
+		default:
+			for (size_t i = 0; i < nfd; i++)
+				if (pfd[i].revents & POLLIN)
+					process(bl[i]);
+		}
+		if (t % 100 == 0)
+			state_sync(state);
+		if (t % 10000 == 0)
+			update_interfaces();
+		update();
+	}
+	state_close(state);
+	return 0;
+}

Copied: vendor/NetBSD/blacklist/20191106/bin/blacklistd.conf.5 (from r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/NetBSD/blacklist/20191106/bin/blacklistd.conf.5	Wed Nov  6 14:10:50 2019	(r354389, copy of r354388, vendor/NetBSD/blacklist/dist/bin/blacklistd.conf.5)
@@ -0,0 +1,229 @@
+.\" $NetBSD: blacklistd.conf.5,v 1.7 2017/06/07 13:50:57 wiz Exp $
+.\"
+.\" Copyright (c) 2015 The NetBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This code is derived from software contributed to The NetBSD Foundation
+.\" by Christos Zoulas.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+.\" PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+.\" POSSIBILITY OF SUCH DAMAGE.
+.\"
+.Dd June 5, 2017
+.Dt BLACKLISTD.CONF 5
+.Os
+.Sh NAME
+.Nm blacklistd.conf
+.Nd configuration file format for blacklistd
+.Sh DESCRIPTION
+The
+.Nm
+files contains configuration entries for
+.Xr blacklistd 8
+in a fashion similar to
+.Xr inetd.conf 5 .
+Only one entry per line is permitted.
+Every entry must have all fields populated.
+Each field can be separated by a tab or a space.
+Comments are denoted by a
+.Dq #
+at the beginning of a line.
+.Pp
+There are two kinds of configuration lines,
+.Va local
+and
+.Va remote .
+By default, configuration lines are
+.Va local ,
+i.e. the address specified refers to the addresses on the local machine.
+To switch to between
+.Va local
+and
+.Va remote
+configuration lines you can specify the stanzas:
+.Dq [local]
+and
+.Dq [remote] .
+.Pp
+On
+.Va local
+and
+.Va remote
+lines
+.Dq *
+means use the default, or wildcard match.
+In addition, for
+.Va remote
+lines
+.Dq =
+means use the values from the matched
+.Va local
+configuration line.
+.Pp
+The first four fields,
+.Va location ,
+.Va type ,
+.Va proto ,
+and
+.Va owner
+are used to match the
+.Va local
+or
+.Va remote
+addresses, whereas the last 3 fields
+.Va name ,
+.Va nfail ,
+and
+.Va disable
+are used to modify the filtering action.
+.Pp
+The first field denotes the
+.Va location
+as an address, mask, and port.
+The syntax for the
+.Va location
+is:
+.Bd -literal -offset indent
+	[
|][/][:]
+.Ed
+.Pp
+The
+.Dv address
+can be an IPv4 address in numeric format, an IPv6 address
+in numeric format and enclosed by square brackets, or an interface name.
+Mask modifiers are not allowed on interfaces because interfaces
+can have multiple addresses in different protocols where the mask has a different
+size.
+.Pp
+The
+.Dv mask
+is always numeric, but the
+.Dv port
+can be either numeric or symbolic.
+.Pp
+The second field is the socket
+.Va type :
+.Dv stream ,
+.Dv dgram ,
+or numeric.
+The third field is the
+.Va prococol :
+.Dv tcp ,
+.Dv udp ,
+.Dv tcp6 ,
+.Dv udp6 ,
+or numeric.
+The fourth file is the effective user
+.Va ( owner )
+of the daemon process reporting the event,
+either as a username or a userid.
+.Pp
+The rest of the fields are controlling the behavior of the filter.
+.Pp
+The
+.Va name
+field, is the name of the packet filter rule to be used.
+If the
+.Va name
+starts with a
+.Dq - ,
+then the default rulename is prepended to the given name.
+If the
+.Dv name
+contains a
+.Dq / ,
+the remaining portion of the name is interpreted as the mask to be
+applied to the address specified in the rule, causing a single rule violation to
+block the entire subnet for the configured prefix.
+.Pp
+The
+.Va nfail
+field contains the number of failed attempts before access is blocked,
+defaulting to
+.Dq *
+meaning never, and the last field
+.Va disable
+specifies the amount of time since the last access that the blocking
+rule should be active, defaulting to
+.Dq *
+meaning forever.
+The default unit for
+.Va disable
+is seconds, but one can specify suffixes for different units, such as
+.Dq m
+for minutes
+.Dq h
+for hours and
+.Dq d
+for days.
+.Pp
+Matching is done first by checking the
+.Va local
+rules individually, in the order of the most specific to the least specific.
+If a match is found, then the
+.Va remote
+rules are applied.
+The
+.Va name ,
+.Va nfail ,
+and
+.Va disable
+fields can be altered by the
+.Va remote
+rule that matched.
+.Pp
+The
+.Va remote
+rules can be used for whitelisting specific addresses, changing the mask
+size, the rule that the packet filter uses, the number of failed attempts,
+or the block duration.
+.Sh FILES
+.Bl -tag -width /etc/blacklistd.conf -compact
+.It Pa /etc/blacklistd.conf
+Configuration file.
+.El
+.Sh EXAMPLES
+.Bd -literal -offset 8n
+# Block ssh, after 3 attempts for 6 hours on the bnx0 interface
+[local]
+# location	type	proto	owner	name	nfail	duration
+bnx0:ssh	*	*	*	*	3	6h
+[remote]
+# Never block 1.2.3.4
+1.2.3.4:ssh	*	*	*	*	*	*
+# For addresses coming from 8.8.0.0/16 block class C networks instead
+# individual hosts, but keep the rest of the blocking parameters the same.
+8.8.0.0/16:ssh	*	*	*	/24	=	=
+.Ed
+.Sh SEE ALSO
+.Xr blacklistctl 8 ,
+.Xr blacklistd 8
+.Sh HISTORY
+.Nm
+first appeared in
+.Nx 7 .
+.Fx
+support for
+.Nm
+was implemented in
+.Fx 11 .
+.Sh AUTHORS
+.An Christos Zoulas

Copied: vendor/NetBSD/blacklist/20191106/bin/support.c (from r354388, vendor/NetBSD/blacklist/dist/bin/support.c)
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/NetBSD/blacklist/20191106/bin/support.c	Wed Nov  6 14:10:50 2019	(r354389, copy of r354388, vendor/NetBSD/blacklist/dist/bin/support.c)
@@ -0,0 +1,161 @@
+/*	$NetBSD: support.c,v 1.9 2018/09/18 22:12:19 christos Exp $	*/
+
+/*-
+ * Copyright (c) 2015 The NetBSD Foundation, Inc.
+ * All rights reserved.
+ *
+ * This code is derived from software contributed to The NetBSD Foundation
+ * by Christos Zoulas.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Thu Nov  7 03:46:20 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id D464A1A5F72;
 Thu,  7 Nov 2019 03:46:20 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477q7w5tRhz43Md;
 Thu,  7 Nov 2019 03:46:20 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ADE357BD3;
 Thu,  7 Nov 2019 03:46:20 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA73kKfC042751;
 Thu, 7 Nov 2019 03:46:20 GMT (envelope-from phil@FreeBSD.org)
Received: (from phil@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA73kIS5042739;
 Thu, 7 Nov 2019 03:46:18 GMT (envelope-from phil@FreeBSD.org)
Message-Id: <201911070346.xA73kIS5042739@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: phil set sender to
 phil@FreeBSD.org using -f
From: Phil Shafer 
Date: Thu, 7 Nov 2019 03:46:18 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354425 - in vendor/Juniper/libxo/dist: . doc encoder
 encoder/cbor encoder/csv encoder/test libxo tests/core tests/core/saved
 tests/gettext tests/gettext/saved tests/xo/saved xo xolint
X-SVN-Group: vendor
X-SVN-Commit-Author: phil
X-SVN-Commit-Paths: in vendor/Juniper/libxo/dist: . doc encoder encoder/cbor
 encoder/csv encoder/test libxo tests/core tests/core/saved tests/gettext
 tests/gettext/saved tests/xo/saved xo xolint
X-SVN-Commit-Revision: 354425
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Thu, 07 Nov 2019 03:46:20 -0000

Author: phil
Date: Thu Nov  7 03:46:17 2019
New Revision: 354425
URL: https://svnweb.freebsd.org/changeset/base/354425

Log:
  Import libxo 1.3.0

Added:
  vendor/Juniper/libxo/dist/doc/encoders.rst
  vendor/Juniper/libxo/dist/doc/top-link.html.in   (contents, props changed)
  vendor/Juniper/libxo/dist/encoder/csv/
  vendor/Juniper/libxo/dist/encoder/csv/Makefile.am   (contents, props changed)
  vendor/Juniper/libxo/dist/encoder/csv/enc_csv.c   (contents, props changed)
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv1.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv1.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv2.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv2.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv3.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.Ecsv3.out
Deleted:
  vendor/Juniper/libxo/dist/doc/libxo-manual.html
  vendor/Juniper/libxo/dist/doc/libxo.txt
Modified:
  vendor/Juniper/libxo/dist/Makefile.am
  vendor/Juniper/libxo/dist/README.md
  vendor/Juniper/libxo/dist/configure.ac
  vendor/Juniper/libxo/dist/doc/Makefile.am
  vendor/Juniper/libxo/dist/doc/api.rst
  vendor/Juniper/libxo/dist/doc/conf.py
  vendor/Juniper/libxo/dist/doc/faq.rst
  vendor/Juniper/libxo/dist/doc/field-formatting.rst
  vendor/Juniper/libxo/dist/doc/field-modifiers.rst
  vendor/Juniper/libxo/dist/doc/field-roles.rst
  vendor/Juniper/libxo/dist/doc/index.rst
  vendor/Juniper/libxo/dist/doc/options.rst
  vendor/Juniper/libxo/dist/doc/xo.rst
  vendor/Juniper/libxo/dist/doc/xohtml.rst
  vendor/Juniper/libxo/dist/doc/xolint.rst
  vendor/Juniper/libxo/dist/doc/xopo.rst
  vendor/Juniper/libxo/dist/encoder/Makefile.am
  vendor/Juniper/libxo/dist/encoder/cbor/enc_cbor.c
  vendor/Juniper/libxo/dist/encoder/test/enc_test.c
  vendor/Juniper/libxo/dist/libxo/add.man.in
  vendor/Juniper/libxo/dist/libxo/libxo.c
  vendor/Juniper/libxo/dist/libxo/xo_buf.h
  vendor/Juniper/libxo/dist/libxo/xo_encoder.c
  vendor/Juniper/libxo/dist/libxo/xo_encoder.h
  vendor/Juniper/libxo/dist/tests/core/Makefile.am
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.H.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.HIPx.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.HP.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.J.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.JP.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.T.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.X.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_01.XP.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.E.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.H.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.HIPx.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.HP.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.J.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.JP.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.T.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.X.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_02.XP.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_03.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_04.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_05.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_06.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_07.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_08.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_09.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_10.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_11.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.E.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.E.out
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.H.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.HIPx.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.HP.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.J.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.JP.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.T.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.X.err
  vendor/Juniper/libxo/dist/tests/core/saved/test_12.XP.err
  vendor/Juniper/libxo/dist/tests/core/test_01.c
  vendor/Juniper/libxo/dist/tests/core/test_02.c
  vendor/Juniper/libxo/dist/tests/core/test_03.c
  vendor/Juniper/libxo/dist/tests/core/test_08.c
  vendor/Juniper/libxo/dist/tests/core/test_09.c
  vendor/Juniper/libxo/dist/tests/core/test_10.c
  vendor/Juniper/libxo/dist/tests/core/test_11.c
  vendor/Juniper/libxo/dist/tests/core/test_12.c
  vendor/Juniper/libxo/dist/tests/gettext/gt_01.c
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.H.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.HIPx.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.HP.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.J.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.JP.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.T.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.X.out
  vendor/Juniper/libxo/dist/tests/gettext/saved/gt_01.XP.out
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.H.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.HIPx.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.HP.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.J.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.JP.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.T.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.X.err
  vendor/Juniper/libxo/dist/tests/xo/saved/xo_02.XP.err
  vendor/Juniper/libxo/dist/xo/xo.c
  vendor/Juniper/libxo/dist/xolint/xolint.pl

Modified: vendor/Juniper/libxo/dist/Makefile.am
==============================================================================
--- vendor/Juniper/libxo/dist/Makefile.am	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/Makefile.am	Thu Nov  7 03:46:17 2019	(r354425)
@@ -32,7 +32,6 @@ errors:
 docs:
 	@(cd doc ; ${MAKE} docs)
 
-
 DIST_FILES_DIR = ~/Dropbox/dist-files/
 GH_PAGES_DIR = gh-pages/
 GH_PAGES_DIR_VER = gh-pages/${PACKAGE_VERSION}
@@ -49,18 +48,20 @@ upload: dist upload-docs upload-xohtml-files
 	@echo "Remember to run:"
 	@echo "        gt tag ${PACKAGE_VERSION}"
 
-upload-docs: docs
-	@echo "Uploading libxo-manual.html ... "
-	@-[ -d ${GH_PAGES_DIR} ] \
-		&& echo "Updating manual on gh-pages ..." \
-		&& mkdir -p ${GH_PAGES_DIR_VER} \
-		&& cp doc/libxo-manual.html ${GH_PAGES_DIR} \
-		&& cp doc/libxo-manual.html ${GH_PAGES_DIR_VER} \
+upload-docs: docs upload-html
+
+upload-html:
+	@echo "Uploading html ... "
+	@-[ -d ${GH_PAGES_DIR} -a -d doc/html ] \
+		&& echo "Updating html on gh-pages ..." \
+		&& mkdir -p ${GH_PAGES_DIR_VER}/html \
+		&& cp doc/top-link.html ${GH_PAGES_DIR}/libxo.html \
+		&& cp -r doc/html/* ${GH_PAGES_DIR_VER}/html/ \
 		&& (cd ${GH_PAGES_DIR} \
-			&& git add ${PACKAGE_VERSION} \
-			&& git add libxo-manual.html \
+			&& git add libxo.html \
+			&& git add ${PACKAGE_VERSION}/html \
 			&& git commit -m 'new docs' \
-                               libxo-manual.html ${PACKAGE_VERSION} \
+                               libxo.html ${PACKAGE_VERSION}/html \
 			&& git push origin gh-pages ) ; true
 
 upload-xohtml-files:

Modified: vendor/Juniper/libxo/dist/README.md
==============================================================================
--- vendor/Juniper/libxo/dist/README.md	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/README.md	Thu Nov  7 03:46:17 2019	(r354425)
@@ -10,6 +10,9 @@ application calls a function "xo_emit" to product outp
 described in a format string.  A "field descriptor" tells libxo what
 the field is and what it means.
 
+Imagine a simplified ``wc`` that emits its output fields in a single
+xo_emit call:
+
 ```
     xo_emit(" {:lines/%7ju/%ju} {:words/%7ju/%ju} "
             "{:characters/%7ju/%ju}{d:filename/%s}\n",

Modified: vendor/Juniper/libxo/dist/configure.ac
==============================================================================
--- vendor/Juniper/libxo/dist/configure.ac	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/configure.ac	Thu Nov  7 03:46:17 2019	(r354425)
@@ -12,7 +12,7 @@
 #
 
 AC_PREREQ(2.2)
-AC_INIT([libxo], [1.0.4], [phil@juniper.net])
+AC_INIT([libxo], [1.3.0], [phil@juniper.net])
 AM_INIT_AUTOMAKE([-Wall -Werror foreign -Wno-portability])
 
 # Support silent build rules.  Requires at least automake-1.11.
@@ -452,6 +452,7 @@ AC_CONFIG_FILES([
   libxo/add.man
   encoder/Makefile
   encoder/cbor/Makefile
+  encoder/csv/Makefile
   encoder/test/Makefile
   xo/Makefile
   xolint/Makefile
@@ -459,6 +460,7 @@ AC_CONFIG_FILES([
   xopo/Makefile
   packaging/libxo.pc
   doc/Makefile
+  doc/top-link.html
   tests/Makefile
   tests/core/Makefile
   tests/gettext/Makefile

Modified: vendor/Juniper/libxo/dist/doc/Makefile.am
==============================================================================
--- vendor/Juniper/libxo/dist/doc/Makefile.am	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/Makefile.am	Thu Nov  7 03:46:17 2019	(r354425)
@@ -8,68 +8,22 @@
 # using the SOFTWARE, you agree to be bound by the terms of that
 # LICENSE.
 
-if HAVE_OXTRADOC
-OXTRADOC_DIR = ${SLAX_OXTRADOCDIR}
-OXTRADOC_PREFIX = ${OXTRADOC_DIR}
-OXTRADOC = ${OXTRADOC_DIR}/oxtradoc
-SLAXPROC_BINDIR = ${SLAX_BINDIR}
+doc docs: xolint.rst html
 
-XML2RFC = ${OXTRADOC_DIR}/xml2rfc.tcl
-XML2HTMLDIR = ${OXTRADOC_DIR}
-XML2HTMLBIN = ${XML2HTMLDIR}/rfc2629-to-html.slax
-SLAXPROC = ${SLAX_BINDIR}/slaxproc
+#
+# The contents of xolint.rst is generated based on xolint.pl, since we
+# really want this to be self-documenting.  But readthedocs.org needs this
+# data to be _in_ repo.  So we generate this file on command only, and
+# the developer needs to commit any changes.
+#
 
-SLAXPROC_ARGS = \
-    -a oxtradoc-dir ${OXTRADOC_DIR} \
-    -a oxtradoc-install-dir ${OXTRADOC_DIR} \
-    -a anchor-prefix docs
+xolint.rst: ${top_srcdir}/xolint/xolint.pl
+	perl ${top_srcdir}/xolint/xolint.pl -D > ${top_srcdir}/doc/xolint.rst
 
-SLAXPROC_ARGS_INLINE = \
-    -a oxtradoc-inline yes
+SPHINX = python3 -msphinx
 
-SLAXPROC_ARGS += ${SLAXPROC_ARGS_INLINE}
-
-XML2HTML = \
-    ${SLAXPROC} -g -e -I ${OXTRADOC_DIR} -I . \
-    ${SLAXPROC_ARGS} \
-    ${XML2HTMLBIN}
-
-OX_ARGS = -P ${OXTRADOC_PREFIX} -L ${OXTRADOC_PREFIX}
-OX_ARGS += -S ${SLAXPROC} -p doc
-OX_CMD = ${PERL} ${PERLOPTS} ${OXTRADOC} ${OX_ARGS}
-OXTRADOC_CMD = ${OX_CMD}
-
-OUTPUT = libxo-manual
-INPUT = libxo
-
-EXTRA_DIST = \
-    ${INPUT}.txt \
-    ${OUTPUT}.html \
-    ${OUTPUT}.txt
-
-doc docs: ${OUTPUT}.txt ${OUTPUT}.html
-
-${OUTPUT}.txt: ${INPUT}.txt ${OXTRADOC} xolint.txt
-	${OXTRADOC_CMD} -m text -o $@ $<
-
-${OUTPUT}.html: ${INPUT}.txt ${OXTRADOC} ${XML2HTMLBIN} xolint.txt
-	${OXTRADOC_CMD} -m html -o $@ $<
-
-xolint.txt: ${top_srcdir}/xolint/xolint.pl
-	perl ${top_srcdir}/xolint/xolint.pl -D > xolint.txt
-
-CLEANFILES = \
-xolint.txt \
-${INPUT}.xml \
-${INPUT}.fxml \
-${OUTPUT}.txt \
-${OUTPUT}.html
-else
-doc docs:
-	@${ECHO} "The 'oxtradoc' tool is not installed; see libslax.org"
-endif
-
-SPHINX = python3.4 -msphinx
-
 html sphinx sphinx-html:
-	${SPHINX} -M html ${srcdir} .
+	${SPHINX} -M html ${srcdir} . -N -E
+
+singlehtml:
+	${SPHINX} -M singlehtml ${srcdir} . -N -E

Modified: vendor/Juniper/libxo/dist/doc/api.rst
==============================================================================
--- vendor/Juniper/libxo/dist/doc/api.rst	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/api.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -1,4 +1,4 @@
-.. index: API
+.. index:: API
 
 The libxo API
 =============
@@ -155,14 +155,14 @@ Output Styles (XO_STYLE\_\*)
 
 The libxo functions accept a set of output styles:
 
-=============== =========================
- Flag            Description
-=============== =========================
- XO_STYLE_TEXT   Traditional text output
- XO_STYLE_XML    XML encoded data
- XO_STYLE_JSON   JSON encoded data
- XO_STYLE_HTML   HTML encoded data
-=============== =========================
+  =============== =========================
+   Flag            Description
+  =============== =========================
+   XO_STYLE_TEXT   Traditional text output
+   XO_STYLE_XML    XML encoded data
+   XO_STYLE_JSON   JSON encoded data
+   XO_STYLE_HTML   HTML encoded data
+  =============== =========================
 
 The "XML", "JSON", and "HTML" output styles all use the UTF-8
 character encoding.  "TEXT" using locale-based encoding.
@@ -256,26 +256,26 @@ Flags (XOF\_\*)
 
 The set of valid flags include:
 
-=================== =========================================
- Flag                Description
-=================== =========================================
- XOF_CLOSE_FP        Close file pointer on `xo_destroy`
- XOF_COLOR           Enable color and effects in output
- XOF_COLOR_ALLOWED   Allow color/effect for terminal output
- XOF_DTRT            Enable "do the right thing" mode
- XOF_INFO            Display info data attributes (HTML)
- XOF_KEYS            Emit the key attribute (XML)
- XOF_NO_ENV          Do not use the :ref:`libxo-options` env var
- XOF_NO_HUMANIZE     Display humanization (TEXT, HTML)
- XOF_PRETTY          Make "pretty printed" output
- XOF_UNDERSCORES     Replaces hyphens with underscores
- XOF_UNITS           Display units (XML, HMTL)
- XOF_WARN            Generate warnings for broken calls
- XOF_WARN_XML        Generate warnings in XML on stdout
- XOF_XPATH           Emit XPath expressions (HTML)
- XOF_COLUMNS         Force xo_emit to return columns used
- XOF_FLUSH           Flush output after each `xo_emit` call
-=================== =========================================
+  =================== =========================================
+   Flag                Description
+  =================== =========================================
+   XOF_CLOSE_FP        Close file pointer on `xo_destroy`
+   XOF_COLOR           Enable color and effects in output
+   XOF_COLOR_ALLOWED   Allow color/effect for terminal output
+   XOF_DTRT            Enable "do the right thing" mode
+   XOF_INFO            Display info data attributes (HTML)
+   XOF_KEYS            Emit the key attribute (XML)
+   XOF_NO_ENV          Do not use the :ref:`libxo-options` env var
+   XOF_NO_HUMANIZE     Display humanization (TEXT, HTML)
+   XOF_PRETTY          Make "pretty printed" output
+   XOF_UNDERSCORES     Replaces hyphens with underscores
+   XOF_UNITS           Display units (XML, HMTL)
+   XOF_WARN            Generate warnings for broken calls
+   XOF_WARN_XML        Generate warnings in XML on stdout
+   XOF_XPATH           Emit XPath expressions (HTML)
+   XOF_COLUMNS         Force xo_emit to return columns used
+   XOF_FLUSH           Flush output after each `xo_emit` call
+  =================== =========================================
 
 The `XOF_CLOSE_FP` flag will trigger the call of the *close_func*
 (provided via `xo_set_writer`) when the handle is destroyed.
@@ -300,12 +300,12 @@ regardless of whether warnings are enabled.
 If the style is `XO_STYLE_HTML`, the following additional flags can be
 used:
 
-=============== =========================================
- Flag            Description
-=============== =========================================
- XOF_XPATH       Emit "data-xpath" attributes
- XOF_INFO        Emit additional info fields
-=============== =========================================
+  =============== =========================================
+   Flag            Description
+  =============== =========================================
+   XOF_XPATH       Emit "data-xpath" attributes
+   XOF_INFO        Emit additional info fields
+  =============== =========================================
 
 The `XOF_XPATH` flag enables the emission of XPath expressions detailing
 the hierarchy of XML elements used to encode the data field, if the
@@ -317,11 +317,11 @@ output.  See :ref:`field-information` for details.
 If the style is `XO_STYLE_XML`, the following additional flags can be
 used:
 
-=============== =========================================
- Flag            Description
-=============== =========================================
- XOF_KEYS        Flag "key" fields for XML
-=============== =========================================
+  =============== =========================================
+   Flag            Description
+  =============== =========================================
+   XOF_KEYS        Flag "key" fields for XML
+  =============== =========================================
 
 The `XOF_KEYS` flag adds "key" attribute to the XML encoding for
 field definitions that use the "k" modifier.  The key attribute has
@@ -1308,52 +1308,52 @@ These values are defined in .
 The priority value indicates the importance and potential impact of
 each message:
 
-============= =======================================================
- Priority      Description
-============= =======================================================
- LOG_EMERG     A panic condition, normally broadcast to all users
- LOG_ALERT     A condition that should be corrected immediately
- LOG_CRIT      Critical conditions
- LOG_ERR       Generic errors
- LOG_WARNING   Warning messages
- LOG_NOTICE    Non-error conditions that might need special handling
- LOG_INFO      Informational messages
- LOG_DEBUG     Developer-oriented messages
-============= =======================================================
+  ============= =======================================================
+   Priority      Description
+  ============= =======================================================
+   LOG_EMERG     A panic condition, normally broadcast to all users
+   LOG_ALERT     A condition that should be corrected immediately
+   LOG_CRIT      Critical conditions
+   LOG_ERR       Generic errors
+   LOG_WARNING   Warning messages
+   LOG_NOTICE    Non-error conditions that might need special handling
+   LOG_INFO      Informational messages
+   LOG_DEBUG     Developer-oriented messages
+  ============= =======================================================
 
 The facility value indicates the source of message, in fairly generic
 terms:
 
-=============== =======================================================
- Facility        Description
-=============== =======================================================
- LOG_AUTH        The authorization system (e.g. :manpage:`login(1)`)
- LOG_AUTHPRIV    As LOG_AUTH, but logged to a privileged file
- LOG_CRON        The cron daemon: :manpage:`cron(8)`
- LOG_DAEMON      System daemons, not otherwise explicitly listed
- LOG_FTP         The file transfer protocol daemons
- LOG_KERN        Messages generated by the kernel
- LOG_LPR         The line printer spooling system
- LOG_MAIL        The mail system
- LOG_NEWS        The network news system
- LOG_SECURITY    Security subsystems, such as :manpage:`ipfw(4)`
- LOG_SYSLOG      Messages generated internally by :manpage:`syslogd(8)`
- LOG_USER        Messages generated by user processes (default)
- LOG_UUCP        The uucp system
- LOG_LOCAL0..7   Reserved for local use
-=============== =======================================================
+  =============== =======================================================
+   Facility        Description
+  =============== =======================================================
+   LOG_AUTH        The authorization system (e.g. :manpage:`login(1)`)
+   LOG_AUTHPRIV    As LOG_AUTH, but logged to a privileged file
+   LOG_CRON        The cron daemon: :manpage:`cron(8)`
+   LOG_DAEMON      System daemons, not otherwise explicitly listed
+   LOG_FTP         The file transfer protocol daemons
+   LOG_KERN        Messages generated by the kernel
+   LOG_LPR         The line printer spooling system
+   LOG_MAIL        The mail system
+   LOG_NEWS        The network news system
+   LOG_SECURITY    Security subsystems, such as :manpage:`ipfw(4)`
+   LOG_SYSLOG      Messages generated internally by :manpage:`syslogd(8)`
+   LOG_USER        Messages generated by user processes (default)
+   LOG_UUCP        The uucp system
+   LOG_LOCAL0..7   Reserved for local use
+  =============== =======================================================
 
 In addition to the values listed above, xo_open_log accepts a set of
 addition flags requesting specific logging behaviors:
 
-============ ====================================================
- Flag         Description
-============ ====================================================
- LOG_CONS     If syslogd fails, attempt to write to /dev/console
- LOG_NDELAY   Open the connection to :manpage:`syslogd(8)` immediately
- LOG_PERROR   Write the message also to standard error output
- LOG_PID      Log the process id with each message
-============ ====================================================
+  ============ ====================================================
+   Flag         Description
+  ============ ====================================================
+   LOG_CONS     If syslogd fails, attempt to write to /dev/console
+   LOG_NDELAY   Open the connection to :manpage:`syslogd(8)` immediately
+   LOG_PERROR   Write the message also to standard error output
+   LOG_PID      Log the process id with each message
+  ============ ====================================================
 
 .. index:: xo_syslog
 
@@ -1588,26 +1588,26 @@ processing model of libxo.  Content is formatted withi
 callbacks are made to the encoder's handler function when data is
 ready to be processed:
 
-======================= =======================================
- Operation               Meaning  (Base function)
-======================= =======================================
- XO_OP_CREATE            Called when the handle is created
- XO_OP_OPEN_CONTAINER    Container opened (xo_open_container)
- XO_OP_CLOSE_CONTAINER   Container closed (xo_close_container)
- XO_OP_OPEN_LIST         List opened (xo_open_list)
- XO_OP_CLOSE_LIST        List closed (xo_close_list)
- XO_OP_OPEN_LEAF_LIST    Leaf list opened (xo_open_leaf_list)
- XO_OP_CLOSE_LEAF_LIST   Leaf list closed (xo_close_leaf_list)
- XO_OP_OPEN_INSTANCE     Instance opened (xo_open_instance)
- XO_OP_CLOSE_INSTANCE    Instance closed (xo_close_instance)
- XO_OP_STRING            Field with Quoted UTF-8 string
- XO_OP_CONTENT           Field with content
- XO_OP_FINISH            Finish any pending output
- XO_OP_FLUSH             Flush any buffered output
- XO_OP_DESTROY           Clean up resources
- XO_OP_ATTRIBUTE         An attribute name/value pair
- XO_OP_VERSION           A version string
-======================= =======================================
+  ======================= =======================================
+   Operation               Meaning  (Base function)
+  ======================= =======================================
+   XO_OP_CREATE            Called when the handle is created
+   XO_OP_OPEN_CONTAINER    Container opened (xo_open_container)
+   XO_OP_CLOSE_CONTAINER   Container closed (xo_close_container)
+   XO_OP_OPEN_LIST         List opened (xo_open_list)
+   XO_OP_CLOSE_LIST        List closed (xo_close_list)
+   XO_OP_OPEN_LEAF_LIST    Leaf list opened (xo_open_leaf_list)
+   XO_OP_CLOSE_LEAF_LIST   Leaf list closed (xo_close_leaf_list)
+   XO_OP_OPEN_INSTANCE     Instance opened (xo_open_instance)
+   XO_OP_CLOSE_INSTANCE    Instance closed (xo_close_instance)
+   XO_OP_STRING            Field with Quoted UTF-8 string
+   XO_OP_CONTENT           Field with content
+   XO_OP_FINISH            Finish any pending output
+   XO_OP_FLUSH             Flush any buffered output
+   XO_OP_DESTROY           Clean up resources
+   XO_OP_ATTRIBUTE         An attribute name/value pair
+   XO_OP_VERSION           A version string
+  ======================= =======================================
 
 For all the open and close operations, the name parameter holds the
 name of the construct.  For string, content, and attribute operations,

Modified: vendor/Juniper/libxo/dist/doc/conf.py
==============================================================================
--- vendor/Juniper/libxo/dist/doc/conf.py	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/conf.py	Thu Nov  7 03:46:17 2019	(r354425)
@@ -21,7 +21,15 @@
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
 
+import subprocess
 
+#
+# Instead of hardcoding the version number here, we read it from the
+# project's configure script
+#
+vers_cmd = "grep AC_INIT ../configure.ac | awk '{ print substr($2, 2, length($2) - 3);}'"
+version = subprocess.check_output(vers_cmd, shell=True).decode("utf-8") 
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -47,7 +55,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'libxo'
-copyright = '2017, Juniper Networks'
+copyright = '2017-2019, Juniper Networks Inc'
 author = 'Phil Shafer'
 default_role = 'code'
 primary_domain = 'c'
@@ -58,9 +66,9 @@ smart_quotes = False
 # built documents.
 #
 # The short X.Y version.
-version = '0.8.4'
+#version = 'develop'
 # The full version, including alpha/beta/rc tags.
-release = '0.8.4'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

Added: vendor/Juniper/libxo/dist/doc/encoders.rst
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ vendor/Juniper/libxo/dist/doc/encoders.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -0,0 +1,269 @@
+.. index:: encoder
+
+Encoders
+========
+
+This section gives an overview of encoders, details on the encoders
+that ship with libxo, and documentation for developers of future
+encoders.
+
+Overview
+--------
+
+The libxo library contains software to generate four "built-in"
+formats: text, XML, JSON, and HTML.  These formats are common and
+useful, but there are other common and useful formats that users will
+want, and including them all in the libxo software would be difficult
+and cumbersome.
+
+To allow support for additional encodings, libxo includes a
+"pluggable" extension mechanism for dynamically loading new encoders.
+libxo-based applications can automatically use any installed encoder.
+
+Use the "encoder=XXX" option to access encoders.  The following
+example uses the "cbor" encoder, saving the output into a file::
+
+    df --libxo encoder=cbor > df-output.cbor
+
+Encoders can support specific options that can be accessed by
+following the encoder name with a colon (':') and one of more options,
+separated by a plus sign "+"::
+
+    df --libxo encoder=csv:path=filesystem+leaf=name+no-header
+
+This example instructs libxo to load the "csv" encoder and pass the
+following options::
+
+   path=filesystem
+   leaf=name
+   no-header
+
+Each of these option is interpreted by the encoder, and all such
+options names and semantics are specific to the particular encoder.
+Refer to the intended encoder for documentation on its options.
+
+.. _csv_encoder:
+
+CSV - Comma Separated Values
+----------------------------
+
+libxo ships with a custom encoder for "CSV" files, a common format for
+comma separated values.  The output of the CSV encoder can be loaded
+directly into spreadsheets or similar applications.
+
+A standard for CSV files is provided in :RFC:`4180`, but since the
+format predates that standard by decades, there are many minor
+differences in CSV file consumers and their expectations.  The CSV
+encoder has a number of options to tailor output to those
+expectations.
+
+Consider the following XML::
+
+  % list-items --libxo xml,pretty
+  
+    
+      
+        GRO-000-415
+        gum
+        1412
+        54
+        10
+      
+      
+        HRD-000-212
+        rope
+        85
+        4
+        2
+      
+      
+        HRD-000-517
+        ladder
+        0
+        2
+        1
+      
+    
+  
+
+This output is a list of `instances` (named "item"), each containing a
+set of `leafs` ("sku", "name", etc).
+
+The CSV encoder will emit the leaf values in this output as `fields`
+inside a CSV `record`, which is a line containing a set of
+comma-separated values::
+
+  % list-items --libxo encoder=csv
+  sku,name,sold,in-stock,on-order
+  GRO-000-415,gum,1412,54,10
+  HRD-000-212,rope,85,4,2
+  HRD-000-517,ladder,0,2,1
+
+Be aware that since the CSV encoder looks for data instances, when
+used with :ref:`xo`, the `--instance` option will be needed::
+
+  % xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route"
+  product,status
+  stereo,in route
+
+.. _csv_path:
+
+The `path` Option
+~~~~~~~~~~~~~~~~~
+
+By default, the CSV encoder will attempt to emit any list instance
+generated by the application.  In some cases, this may be
+unacceptable, and a specific list may be desired.
+
+Use the "path" option to limit the processing of output to a specific
+hierarchy.  The path should be one or more names of containers or
+lists.
+
+For example, if the "list-items" application generates other lists,
+the user can give "path=top/data/item" as a path::
+
+  % list-items --libxo encoder=csv:path=top/data/item
+  sku,name,sold,in-stock,on-order
+  GRO-000-415,gum,1412,54,10
+  HRD-000-212,rope,85,4,2
+  HRD-000-517,ladder,0,2,1
+
+Paths are "relative", meaning they need not be a complete set
+of names to the list.  This means that "path=item" may be sufficient
+for the above example.
+
+.. _csv_leafs:
+
+The `leafs` Option
+~~~~~~~~~~~~~~~~~~
+
+The CSV encoding requires that all lines of output have the same
+number of fields with the same order.  In contrast, XML and JSON allow
+any order (though libxo forces key leafs to appear before other
+leafs).
+
+To maintain a consistent set of fields inside the CSV file, the same
+set of leafs must be selected from each list item.  By default, the
+CSV encoder records the set of leafs that appear in the first list
+instance it processes, and extract only those leafs from future
+instances.  If the first instance is missing a leaf that is desired by
+the consumer, the "leaf" option can be used to ensure that an empty
+value is recorded for instances that lack a particular leaf.
+
+The "leafs" option can also be used to exclude leafs, limiting the
+output to only those leafs provided.
+
+In addition, the order of the output fields follows the order in which
+the leafs are listed.  "leafs=one.two" and "leafs=two.one" give
+distinct output.
+
+So the "leafs" option can be used to expand, limit, and order the set
+of leafs.
+
+The value of the leafs option should be one or more leaf names,
+separated by a period (".")::
+
+  % list-items --libxo encoder=csv:leafs=sku.on-order
+  sku,on-order
+  GRO-000-415,10
+  HRD-000-212,2
+  HRD-000-517,1
+  % list-items -libxo encoder=csv:leafs=on-order.sku
+  on-order,sku
+  10,GRO-000-415
+  2,HRD-000-212
+  1,HRD-000-517
+
+Note that since libxo uses terminology from YANG (:RFC:`7950`), the
+data modeling language for NETCONF (:RFC:`6241`), which uses "leafs"
+as the plural form of "leaf".  libxo follows that convention.
+
+.. _csv_no_header:
+
+The `no-header` Option
+~~~~~~~~~~~~~~~~~~~~~~
+
+CSV files typical begin with a line that defines the fields included
+in that file, in an attempt to make the contents self-defining::
+
+    sku,name,sold,in-stock,on-order
+    GRO-000-415,gum,1412,54,10
+    HRD-000-212,rope,85,4,2
+    HRD-000-517,ladder,0,2,1
+
+There is no reliable mechanism for determining whether this header
+line is included, so the consumer must make an assumption.
+
+The csv encoder defaults to producing the header line, but the
+"no-header" option can be included to avoid the header line.
+
+.. _csv_no_quotes:
+
+The `no-quotes` Option
+~~~~~~~~~~~~~~~~~~~~~~
+
+:RFC:`4180` specifies that fields containing spaces should be quoted, but
+many CSV consumers do not handle quotes.  The "no-quotes" option
+instruct the CSV encoder to avoid the use of quotes.
+
+.. _csv_dos:
+
+The `dos` Option
+~~~~~~~~~~~~~~~~
+
+:RFC:`4180` defines the end-of-line marker as a carriage return
+followed by a newline.  This `CRLF` convention dates from the distant
+past, but its use was anchored in the 1980s by the `DOS` operating
+system.
+
+The CSV encoder defaults to using the standard Unix end-of-line
+marker, a simple newline.  Use the "dos" option to use the `CRLF`
+convention.
+
+The Encoder API
+---------------
+
+The encoder API consists of three distinct phases:
+
+- loading the encoder
+- initializing the encoder
+- feeding operations to the encoder
+
+To load the encoder, libxo will open a shared library named:
+
+   ${prefix}/lib/libxo/encoder/${name}.enc
+
+This file is typically a symbolic link to a dynamic library, suitable
+for `dlopen`().  libxo looks for a symbol called
+`xo_encoder_library_init` inside that library and calls it with the
+arguments defined in the header file "xo_encoder.h".  This function
+should look as follows::
+
+  int
+  xo_encoder_library_init (XO_ENCODER_INIT_ARGS)
+  {
+      arg->xei_version = XO_ENCODER_VERSION;
+      arg->xei_handler = test_handler;
+  
+      return 0;
+  }
+
+Several features here allow for future compatibility: the macro
+XO_ENCODER_INIT_ARGS allows the arguments to this function change over
+time, and the XO_ENCODER_VERSION allows the library to tell libxo
+which version of the API it was compiled with.
+
+The function places in xei_handler should be have the signature::
+
+  static int
+  test_handler (XO_ENCODER_HANDLER_ARGS)
+  {
+       ...
+
+This function will be called with the "op" codes defined in
+"xo_encoder.h".  Each op code represents a distinct event in the libxo
+processing model.  For example OP_OPEN_CONTAINER tells the encoder
+that a new container has been opened, and the encoder can behave in an
+appropriate manner.
+
+

Modified: vendor/Juniper/libxo/dist/doc/faq.rst
==============================================================================
--- vendor/Juniper/libxo/dist/doc/faq.rst	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/faq.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -202,8 +202,7 @@ will lead users to ask the difference between the two 
 there is no difference, use only one of the field names.  If there is
 a difference, change the names to make that difference more obvious.
 
-.. ignore for now, since we want can't have generated content
-  What does this message mean?
-  ----------------------------
+What does this message mean?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-  !!include-file xolint.txt
+.. include:: xolint.rst

Modified: vendor/Juniper/libxo/dist/doc/field-formatting.rst
==============================================================================
--- vendor/Juniper/libxo/dist/doc/field-formatting.rst	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/field-formatting.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -53,48 +53,48 @@ removed eventually.
 
 The format character is described in the following table:
 
-===== ================= ======================
- Ltr   Argument Type     Format
-===== ================= ======================
- d     int               base 10 (decimal)
- i     int               base 10 (decimal)
- o     int               base 8 (octal)
- u     unsigned          base 10 (decimal)
- x     unsigned          base 16 (hex)
- X     unsigned long     base 16 (hex)
- D     long              base 10 (decimal)
- O     unsigned long     base 8 (octal)
- U     unsigned long     base 10 (decimal)
- e     double            [-]d.ddde+-dd
- E     double            [-]d.dddE+-dd
- f     double            [-]ddd.ddd
- F     double            [-]ddd.ddd
- g     double            as 'e' or 'f'
- G     double            as 'E' or 'F'
- a     double            [-]0xh.hhhp[+-]d
- A     double            [-]0Xh.hhhp[+-]d
- c     unsigned char     a character
- C     wint_t            a character
- s     char \*           a UTF-8 string
- S     wchar_t \*        a unicode/WCS string
- p     void \*           '%#lx'
-===== ================= ======================
+  ===== ================= ======================
+   Ltr   Argument Type     Format
+  ===== ================= ======================
+   d     int               base 10 (decimal)
+   i     int               base 10 (decimal)
+   o     int               base 8 (octal)
+   u     unsigned          base 10 (decimal)
+   x     unsigned          base 16 (hex)
+   X     unsigned long     base 16 (hex)
+   D     long              base 10 (decimal)
+   O     unsigned long     base 8 (octal)
+   U     unsigned long     base 10 (decimal)
+   e     double            [-]d.ddde+-dd
+   E     double            [-]d.dddE+-dd
+   f     double            [-]ddd.ddd
+   F     double            [-]ddd.ddd
+   g     double            as 'e' or 'f'
+   G     double            as 'E' or 'F'
+   a     double            [-]0xh.hhhp[+-]d
+   A     double            [-]0Xh.hhhp[+-]d
+   c     unsigned char     a character
+   C     wint_t            a character
+   s     char \*           a UTF-8 string
+   S     wchar_t \*        a unicode/WCS string
+   p     void \*           '%#lx'
+  ===== ================= ======================
 
 The 'h' and 'l' modifiers affect the size and treatment of the
 argument:
 
-===== ============= ====================
- Mod   d, i          o, u, x, X
-===== ============= ====================
- hh    signed char   unsigned char
- h     short         unsigned short
- l     long          unsigned long
- ll    long long     unsigned long long
- j     intmax_t      uintmax_t
- t     ptrdiff_t     ptrdiff_t
- z     size_t        size_t
- q     quad_t        u_quad_t
-===== ============= ====================
+  ===== ============= ====================
+   Mod   d, i          o, u, x, X
+  ===== ============= ====================
+   hh    signed char   unsigned char
+   h     short         unsigned short
+   l     long          unsigned long
+   ll    long long     unsigned long long
+   j     intmax_t      uintmax_t
+   t     ptrdiff_t     ptrdiff_t
+   z     size_t        size_t
+   q     quad_t        u_quad_t
+  ===== ============= ====================
 
 .. index:: UTF-8
 .. index:: Locale
@@ -122,14 +122,14 @@ style::
    xo_emit("All strings are utf-8 content {:tag/%ls}",
            L"except for wide strings");
 
-======== ================== ===============================
- Format   Argument Type      Argument Contents
-======== ================== ===============================
- %s       const char \*      UTF-8 string
- %S       const char \*      UTF-8 string (alias for '%ls')
- %ls      const wchar_t \*   Wide character UNICODE string
- %hs      const char *       locale-based string
-======== ================== ===============================
+  ======== ================== ===============================
+   Format   Argument Type      Argument Contents
+  ======== ================== ===============================
+   %s       const char \*      UTF-8 string
+   %S       const char \*      UTF-8 string (alias for '%ls')
+   %ls      const wchar_t \*   Wide character UNICODE string
+   %hs      const char *       locale-based string
+  ======== ================== ===============================
 
 .. admonition:: "Long", not "locale"
 
@@ -266,21 +266,21 @@ incompatible with printf-like testing:
 If none of these features are in use by your code, then using the "_p"
 variants might be wise:
 
-================== ========================
- Function           printf-like Equivalent
-================== ========================
- xo_emit_hv         xo_emit_hvp
- xo_emit_h          xo_emit_hp
- xo_emit            xo_emit_p
- xo_emit_warn_hcv   xo_emit_warn_hcvp
- xo_emit_warn_hc    xo_emit_warn_hcp
- xo_emit_warn_c     xo_emit_warn_cp
- xo_emit_warn       xo_emit_warn_p
- xo_emit_warnx      xo_emit_warnx_p
- xo_emit_err        xo_emit_err_p
- xo_emit_errx       xo_emit_errx_p
- xo_emit_errc       xo_emit_errc_p
-================== ========================
+  ================== ========================
+   Function           printf-like Equivalent
+  ================== ========================
+   xo_emit_hv         xo_emit_hvp
+   xo_emit_h          xo_emit_hp
+   xo_emit            xo_emit_p
+   xo_emit_warn_hcv   xo_emit_warn_hcvp
+   xo_emit_warn_hc    xo_emit_warn_hcp
+   xo_emit_warn_c     xo_emit_warn_cp
+   xo_emit_warn       xo_emit_warn_p
+   xo_emit_warnx      xo_emit_warnx_p
+   xo_emit_err        xo_emit_err_p
+   xo_emit_errx       xo_emit_errx_p
+   xo_emit_errc       xo_emit_errc_p
+  ================== ========================
 
 .. index:: performance
 .. index:: XOEF_RETAIN
@@ -305,16 +305,16 @@ xo_emit_f() function.  A complete set of xo_emit_f fun
 match all the xo_emit function signatures (with handles, varadic
 argument, and printf-like flags):
 
-================== ========================
- Function           Flags Equivalent
-================== ========================
- xo_emit_hv         xo_emit_hvf
- xo_emit_h          xo_emit_hf
- xo_emit            xo_emit_f
- xo_emit_hvp        xo_emit_hvfp
- xo_emit_hp         xo_emit_hfp
- xo_emit_p          xo_emit_fp
-================== ========================
+  ================== ========================
+   Function           Flags Equivalent
+  ================== ========================
+   xo_emit_hv         xo_emit_hvf
+   xo_emit_h          xo_emit_hf
+   xo_emit            xo_emit_f
+   xo_emit_hvp        xo_emit_hvfp
+   xo_emit_hp         xo_emit_hfp
+   xo_emit_p          xo_emit_fp
+  ================== ========================
 
 The format string must be immutable across multiple calls to xo_emit_f(),
 since the library retains the string.  Typically this is done by using

Modified: vendor/Juniper/libxo/dist/doc/field-modifiers.rst
==============================================================================
--- vendor/Juniper/libxo/dist/doc/field-modifiers.rst	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/field-modifiers.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -8,26 +8,26 @@ Field Modifiers
 Field modifiers are flags which modify the way content emitted for
 particular output styles:
 
-=== =============== ===================================================
- M   Name            Description
-=== =============== ===================================================
- a   argument        The content appears as a 'const char \*' argument
- c   colon           A colon (":") is appended after the label
- d   display         Only emit field for display styles (text/HTML)
- e   encoding        Only emit for encoding styles (XML/JSON)
- g   gettext         Call gettext on field's render content
- h   humanize (hn)   Format large numbers in human-readable style
-\    hn-space        Humanize: Place space between numeric and unit
-\    hn-decimal      Humanize: Add a decimal digit, if number < 10
-\    hn-1000         Humanize: Use 1000 as divisor instead of 1024
- k   key             Field is a key, suitable for XPath predicates
- l   leaf-list       Field is a leaf-list
- n   no-quotes       Do not quote the field when using JSON style
- p   plural          Gettext: Use comma-separated plural form
- q   quotes          Quote the field when using JSON style
- t   trim            Trim leading and trailing whitespace
- w   white           A blank (" ") is appended after the label
-=== =============== ===================================================
+  === =============== ===================================================
+   M   Name            Description
+  === =============== ===================================================
+   a   argument        The content appears as a 'const char \*' argument
+   c   colon           A colon (":") is appended after the label
+   d   display         Only emit field for display styles (text/HTML)
+   e   encoding        Only emit for encoding styles (XML/JSON)
+   g   gettext         Call gettext on field's render content
+   h   humanize (hn)   Format large numbers in human-readable style
+  \    hn-space        Humanize: Place space between numeric and unit
+  \    hn-decimal      Humanize: Add a decimal digit, if number < 10
+  \    hn-1000         Humanize: Use 1000 as divisor instead of 1024
+   k   key             Field is a key, suitable for XPath predicates
+   l   leaf-list       Field is a leaf-list
+   n   no-quotes       Do not quote the field when using JSON style
+   p   plural          Gettext: Use comma-separated plural form
+   q   quotes          Quote the field when using JSON style
+   t   trim            Trim leading and trailing whitespace
+   w   white           A blank (" ") is appended after the label
+  === =============== ===================================================
 
 Roles and modifiers can also use more verbose names, when preceded by
 a comma.  For example, the modifier string "Lwc" (or "L,white,colon")

Modified: vendor/Juniper/libxo/dist/doc/field-roles.rst
==============================================================================
--- vendor/Juniper/libxo/dist/doc/field-roles.rst	Thu Nov  7 00:28:50 2019	(r354424)
+++ vendor/Juniper/libxo/dist/doc/field-roles.rst	Thu Nov  7 03:46:17 2019	(r354425)
@@ -8,24 +8,26 @@ Field Roles
 Field roles are optional, and indicate the role and formatting of the
 content.  The roles are listed below; only one role is permitted:
 
-=== ============== =================================================
-R   Name           Description
-=== ============== =================================================
-C   color          Field has color and effect controls
-D   decoration     Field is non-text (e.g., colon, comma)
-E   error          Field is an error message
-G   gettext        Call gettext(3) on the format string
-L   label          Field is text that prefixes a value
-N   note           Field is text that follows a value
-P   padding        Field is spaces needed for vertical alignment
-T   title          Field is a title value for headings
-U   units          Field is the units for the previous value field
-V   value          Field is the name of field (the default)
-W   warning        Field is a warning message
-[   start-anchor   Begin a section of anchored variable-width text
-]   stop-anchor    End a section of anchored variable-width text
-=== ============== =================================================
+  === ============== =================================================
+  R   Name           Description
+  === ============== =================================================
+  C   color          Field has color and effect controls
+  D   decoration     Field is non-text (e.g., colon, comma)
+  E   error          Field is an error message
+  G   gettext        Call gettext(3) on the format string
+  L   label          Field is text that prefixes a value
+  N   note           Field is text that follows a value
+  P   padding        Field is spaces needed for vertical alignment

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***

From owner-svn-src-vendor@freebsd.org  Thu Nov  7 03:47:14 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0DB9F1A601B;
 Thu,  7 Nov 2019 03:47:14 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 477q8x5LFVz43VS;
 Thu,  7 Nov 2019 03:47:13 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 830D17BD7;
 Thu,  7 Nov 2019 03:47:13 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA73lDDE042837;
 Thu, 7 Nov 2019 03:47:13 GMT (envelope-from phil@FreeBSD.org)
Received: (from phil@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA73lDnx042836;
 Thu, 7 Nov 2019 03:47:13 GMT (envelope-from phil@FreeBSD.org)
Message-Id: <201911070347.xA73lDnx042836@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: phil set sender to
 phil@FreeBSD.org using -f
From: Phil Shafer 
Date: Thu, 7 Nov 2019 03:47:13 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354426 - vendor/Juniper/libxo/1.3.0
X-SVN-Group: vendor
X-SVN-Commit-Author: phil
X-SVN-Commit-Paths: vendor/Juniper/libxo/1.3.0
X-SVN-Commit-Revision: 354426
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Thu, 07 Nov 2019 03:47:14 -0000

Author: phil
Date: Thu Nov  7 03:47:13 2019
New Revision: 354426
URL: https://svnweb.freebsd.org/changeset/base/354426

Log:
  Tag libxo 1.3.0

Added:
     - copied from r354425, vendor/Juniper/libxo/dist/
Directory Properties:
  vendor/Juniper/libxo/1.3.0/   (props changed)

From owner-svn-src-vendor@freebsd.org  Thu Nov  7 18:05:28 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0C43C1BB600;
 Thu,  7 Nov 2019 18:05:28 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 478BCC6Xp5z42yC;
 Thu,  7 Nov 2019 18:05:27 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C52C119A75;
 Thu,  7 Nov 2019 18:05:27 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA7I5RfR061872;
 Thu, 7 Nov 2019 18:05:27 GMT (envelope-from phil@FreeBSD.org)
Received: (from phil@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA7I5R2t061869;
 Thu, 7 Nov 2019 18:05:27 GMT (envelope-from phil@FreeBSD.org)
Message-Id: <201911071805.xA7I5R2t061869@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: phil set sender to
 phil@FreeBSD.org using -f
From: Phil Shafer 
Date: Thu, 7 Nov 2019 18:05:27 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354453 - in vendor/Juniper/libxo/dist: . libxo tests/core
X-SVN-Group: vendor
X-SVN-Commit-Author: phil
X-SVN-Commit-Paths: in vendor/Juniper/libxo/dist: . libxo tests/core
X-SVN-Commit-Revision: 354453
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Thu, 07 Nov 2019 18:05:28 -0000

Author: phil
Date: Thu Nov  7 18:05:26 2019
New Revision: 354453
URL: https://svnweb.freebsd.org/changeset/base/354453

Log:
  Import libxo 1.3.1

Modified:
  vendor/Juniper/libxo/dist/configure.ac
  vendor/Juniper/libxo/dist/libxo/libxo.c
  vendor/Juniper/libxo/dist/tests/core/test_01.c

Modified: vendor/Juniper/libxo/dist/configure.ac
==============================================================================
--- vendor/Juniper/libxo/dist/configure.ac	Thu Nov  7 17:34:44 2019	(r354452)
+++ vendor/Juniper/libxo/dist/configure.ac	Thu Nov  7 18:05:26 2019	(r354453)
@@ -12,7 +12,7 @@
 #
 
 AC_PREREQ(2.2)
-AC_INIT([libxo], [1.3.0], [phil@juniper.net])
+AC_INIT([libxo], [1.3.1], [phil@juniper.net])
 AM_INIT_AUTOMAKE([-Wall -Werror foreign -Wno-portability])
 
 # Support silent build rules.  Requires at least automake-1.11.

Modified: vendor/Juniper/libxo/dist/libxo/libxo.c
==============================================================================
--- vendor/Juniper/libxo/dist/libxo/libxo.c	Thu Nov  7 17:34:44 2019	(r354452)
+++ vendor/Juniper/libxo/dist/libxo/libxo.c	Thu Nov  7 18:05:26 2019	(r354453)
@@ -8104,12 +8104,14 @@ xo_parse_args (int argc, char **argv)
     cp = strrchr(xo_program, '/');
     if (cp)
 	xo_program = ++cp;
+    else
+	cp = argv[0];		/* Reset to front of string */
 
     /* GNU tools add an annoying ".test" as the program extension; remove it */
     size_t len = strlen(xo_program);
     static const char gnu_ext[] = ".test";
     if (len >= sizeof(gnu_ext)) {
-	cp = &cp[len + 1 - sizeof(gnu_ext)];
+	cp += len + 1 - sizeof(gnu_ext);
 	if (xo_streq(cp, gnu_ext))
 	    *cp = '\0';
     }

Modified: vendor/Juniper/libxo/dist/tests/core/test_01.c
==============================================================================
--- vendor/Juniper/libxo/dist/tests/core/test_01.c	Thu Nov  7 17:34:44 2019	(r354452)
+++ vendor/Juniper/libxo/dist/tests/core/test_01.c	Thu Nov  7 18:05:26 2019	(r354453)
@@ -50,6 +50,9 @@ main (int argc, char **argv)
 	{ "sold", "number", "Number of items sold" },
 	{ XO_INFO_NULL },
     };
+
+    char name[] = "test_01.test";  /* test trimming of xo_program */
+    argv[0] = name;
     
     argc = xo_parse_args(argc, argv);
     if (argc < 0)

From owner-svn-src-vendor@freebsd.org  Thu Nov  7 18:05:39 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 676261BB62D;
 Thu,  7 Nov 2019 18:05:39 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 478BCR2Crlz433v;
 Thu,  7 Nov 2019 18:05:39 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1737019A76;
 Thu,  7 Nov 2019 18:05:39 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA7I5ckE061922;
 Thu, 7 Nov 2019 18:05:38 GMT (envelope-from phil@FreeBSD.org)
Received: (from phil@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA7I5cHX061921;
 Thu, 7 Nov 2019 18:05:38 GMT (envelope-from phil@FreeBSD.org)
Message-Id: <201911071805.xA7I5cHX061921@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: phil set sender to
 phil@FreeBSD.org using -f
From: Phil Shafer 
Date: Thu, 7 Nov 2019 18:05:38 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354454 - vendor/Juniper/libxo/1.3.1
X-SVN-Group: vendor
X-SVN-Commit-Author: phil
X-SVN-Commit-Paths: vendor/Juniper/libxo/1.3.1
X-SVN-Commit-Revision: 354454
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Thu, 07 Nov 2019 18:05:39 -0000

Author: phil
Date: Thu Nov  7 18:05:38 2019
New Revision: 354454
URL: https://svnweb.freebsd.org/changeset/base/354454

Log:
  Tag libxo 1.3.1

Added:
     - copied from r354453, vendor/Juniper/libxo/dist/
Directory Properties:
  vendor/Juniper/libxo/1.3.1/   (props changed)

From owner-svn-src-vendor@freebsd.org  Thu Nov  7 18:07:34 2019
Return-Path: 
Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id E2F5F1BB83F;
 Thu,  7 Nov 2019 18:07:34 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 478BFf5ldnz43Xv;
 Thu,  7 Nov 2019 18:07:34 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A1AB019A7E;
 Thu,  7 Nov 2019 18:07:34 +0000 (UTC)
 (envelope-from phil@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xA7I7Yv5063860;
 Thu, 7 Nov 2019 18:07:34 GMT (envelope-from phil@FreeBSD.org)
Received: (from phil@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id xA7I7Ywq063859;
 Thu, 7 Nov 2019 18:07:34 GMT (envelope-from phil@FreeBSD.org)
Message-Id: <201911071807.xA7I7Ywq063859@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: phil set sender to
 phil@FreeBSD.org using -f
From: Phil Shafer 
Date: Thu, 7 Nov 2019 18:07:34 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-vendor@freebsd.org
Subject: svn commit: r354456 - vendor/Juniper/libxo
X-SVN-Group: vendor
X-SVN-Commit-Author: phil
X-SVN-Commit-Paths: vendor/Juniper/libxo
X-SVN-Commit-Revision: 354456
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-vendor@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for the vendor work area tree
 
List-Unsubscribe: , 
 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: ,
 
X-List-Received-Date: Thu, 07 Nov 2019 18:07:35 -0000

Author: phil
Date: Thu Nov  7 18:07:34 2019
New Revision: 354456
URL: https://svnweb.freebsd.org/changeset/base/354456

Log:
  Remove oxtradoc version of documentation; move to RST/Sphinx

Modified:
  vendor/Juniper/libxo/import.sh

Modified: vendor/Juniper/libxo/import.sh
==============================================================================
--- vendor/Juniper/libxo/import.sh	Thu Nov  7 18:06:44 2019	(r354455)
+++ vendor/Juniper/libxo/import.sh	Thu Nov  7 18:07:34 2019	(r354456)
@@ -26,14 +26,19 @@ GMAKE=${GMAKE:-gmake}
 
 # For consistency...
 Error() {
-	echo ERROR: ${1+"$@"} >&2
-	exit 1
+    echo ERROR: ${1+"$@"} >&2
+    exit 1
 }
 
 Cd() {
-	[ $# -eq 1 ] || Error "Cd() takes a single parameter."
-	cd $1 || Error "cannot \"cd $1\" from $PWD"
-        info "Directory =" `pwd`
+    [ $# -eq 1 ] || Error "Cd() takes a single parameter."
+    cd $1 || Error "cannot \"cd $1\" from $PWD"
+    info "Directory =" `pwd`
+         
+    if [ "$DOC" = doc ]; then
+       echo "    cd $1"
+       echo "  "
+    fi
 }
 
 siginfo() {
@@ -51,8 +56,8 @@ run() {
     CMD="$2"
 
     if [ "$DOC" = doc ]; then
-        echo " == $desc"
-        echo "     - $cmd"
+        echo "## $desc"
+        echo "     $cmd"
         echo " "
     else
         echo "===="
@@ -66,7 +71,7 @@ run() {
 }
 
 info() {
-    echo " -- " "$@"
+    echo "## -- " "$@"
 }
 
 okay() {
@@ -182,13 +187,7 @@ fi
 
 # We need the release tar ball for the HTML docs, nothing more
 DOCURL=https://github.com/Juniper/libxo/releases/download/$VERS/libxo-$VERS.tar.gz
-DOCBALL=~/tars/doc-$BASEURL
 
-if [ "$FETCH" = "yes" -o ! -f $DOCBALL ]; then
-    run "fetching doc tarball" "fetch -o $DOCBALL $DOCURL"
-    test -s ${DOCBALL} || Error need DOCBALL
-fi
-
 # BASE should match what the TARBALL contains
 BASE=`basename $TARBALL .tar.gz`
 VERSION=`echo $BASE | sed 's/libxo-//'`
@@ -196,8 +195,6 @@ VERSION=`echo $BASE | sed 's/libxo-//'`
 TF=$BASE/info
 
 run "untarring source files TARBALL" "tar zxf $TARBALL"
-run "untarring html manual" \
-    "tar zxf $DOCBALL libxo-$VERS/doc/libxo-manual.html"
 
 # List of top-level files we want to ignore
 TOPJUNKFILES="\