From owner-svn-src-vendor@freebsd.org Sun Nov 17 20:56:27 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6FAF41AA7EE; Sun, 17 Nov 2019 20:56:27 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47GPWt6czqz3Q7F; Sun, 17 Nov 2019 20:56:26 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6E65F1F78A; Sun, 17 Nov 2019 20:56:26 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAHKuQd6078707; Sun, 17 Nov 2019 20:56:26 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAHKuQMo078706; Sun, 17 Nov 2019 20:56:26 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201911172056.xAHKuQMo078706@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Sun, 17 Nov 2019 20:56:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354798 - vendor/file/dist/src X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/file/dist/src X-SVN-Commit-Revision: 354798 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Nov 2019 20:56:27 -0000 Author: delphij Date: Sun Nov 17 20:56:25 2019 New Revision: 354798 URL: https://svnweb.freebsd.org/changeset/base/354798 Log: Apply vendor fixes: 06de62c Detect multiplication overflow when computing sector position 46a8443 Limit the number of elements in a vector (found by oss-fuzz) Modified: vendor/file/dist/src/cdf.c vendor/file/dist/src/cdf.h Modified: vendor/file/dist/src/cdf.c ============================================================================== --- vendor/file/dist/src/cdf.c Sun Nov 17 20:49:24 2019 (r354797) +++ vendor/file/dist/src/cdf.c Sun Nov 17 20:56:25 2019 (r354798) @@ -35,7 +35,7 @@ #include "file.h" #ifndef lint -FILE_RCSID("@(#)$File: cdf.c,v 1.114 2019/02/20 02:35:27 christos Exp $") +FILE_RCSID("@(#)$File: cdf.c,v 1.116 2019/08/26 14:31:39 christos Exp $") #endif #include @@ -53,6 +53,10 @@ FILE_RCSID("@(#)$File: cdf.c,v 1.114 2019/02/20 02:35: #define EFTYPE EINVAL #endif +#ifndef SIZE_T_MAX +#define SIZE_T_MAX CAST(size_t, ~0ULL) +#endif + #include "cdf.h" #ifdef CDF_DEBUG @@ -405,7 +409,12 @@ cdf_read_sector(const cdf_info_t *info, void *buf, siz const cdf_header_t *h, cdf_secid_t id) { size_t ss = CDF_SEC_SIZE(h); - size_t pos = CDF_SEC_POS(h, id); + size_t pos; + + if (SIZE_T_MAX / ss < CAST(size_t, id)) + return -1; + + pos = CDF_SEC_POS(h, id); assert(ss == len); return cdf_read(info, CAST(off_t, pos), RCAST(char *, buf) + offs, len); } @@ -415,7 +424,12 @@ cdf_read_short_sector(const cdf_stream_t *sst, void *b size_t len, const cdf_header_t *h, cdf_secid_t id) { size_t ss = CDF_SHORT_SEC_SIZE(h); - size_t pos = CDF_SHORT_SEC_POS(h, id); + size_t pos; + + if (SIZE_T_MAX / ss < CAST(size_t, id)) + return -1; + + pos = CDF_SHORT_SEC_POS(h, id); assert(ss == len); if (pos + len > CDF_SEC_SIZE(h) * sst->sst_len) { DPRINTF(("Out of bounds read %" SIZE_T_FORMAT "u > %" @@ -1013,8 +1027,9 @@ cdf_read_property_info(const cdf_stream_t *sst, const goto out; } nelements = CDF_GETUINT32(q, 1); - if (nelements == 0) { - DPRINTF(("CDF_VECTOR with nelements == 0\n")); + if (nelements > CDF_ELEMENT_LIMIT || nelements == 0) { + DPRINTF(("CDF_VECTOR with nelements == %" + SIZE_T_FORMAT "u\n", nelements)); goto out; } slen = 2; @@ -1056,8 +1071,6 @@ cdf_read_property_info(const cdf_stream_t *sst, const goto out; inp += nelem; } - DPRINTF(("nelements = %" SIZE_T_FORMAT "u\n", - nelements)); for (j = 0; j < nelements && i < sh.sh_properties; j++, i++) { Modified: vendor/file/dist/src/cdf.h ============================================================================== --- vendor/file/dist/src/cdf.h Sun Nov 17 20:49:24 2019 (r354797) +++ vendor/file/dist/src/cdf.h Sun Nov 17 20:56:25 2019 (r354798) @@ -48,6 +48,7 @@ typedef int32_t cdf_secid_t; #define CDF_LOOP_LIMIT 10000 +#define CDF_ELEMENT_LIMIT 100000 #define CDF_SECID_NULL 0 #define CDF_SECID_FREE -1 From owner-svn-src-vendor@freebsd.org Wed Nov 20 22:13:17 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 347D41C30BA; Wed, 20 Nov 2019 22:13:17 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JH591JQ1z4QNK; Wed, 20 Nov 2019 22:13:17 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 10404162A; Wed, 20 Nov 2019 22:13:17 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAKMDHAl085572; Wed, 20 Nov 2019 22:13:17 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAKMDEF8085559; Wed, 20 Nov 2019 22:13:14 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201911202213.xAKMDEF8085559@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 20 Nov 2019 22:13:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354917 - vendor/openresolv/dist X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: vendor/openresolv/dist X-SVN-Commit-Revision: 354917 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Nov 2019 22:13:17 -0000 Author: pfg Date: Wed Nov 20 22:13:14 2019 New Revision: 354917 URL: https://svnweb.freebsd.org/changeset/base/354917 Log: Import openresolv 3.9.0 More information at: https://roy.marples.name/projects/openresolv Added: vendor/openresolv/dist/LICENSE vendor/openresolv/dist/README.md Deleted: vendor/openresolv/dist/README Modified: vendor/openresolv/dist/Makefile vendor/openresolv/dist/configure vendor/openresolv/dist/dnsmasq.in vendor/openresolv/dist/libc.in vendor/openresolv/dist/named.in vendor/openresolv/dist/pdns_recursor.in vendor/openresolv/dist/pdnsd.in vendor/openresolv/dist/resolvconf.conf vendor/openresolv/dist/resolvconf.conf.5.in vendor/openresolv/dist/resolvconf.in vendor/openresolv/dist/unbound.in Added: vendor/openresolv/dist/LICENSE ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/dist/LICENSE Wed Nov 20 22:13:14 2019 (r354917) @@ -0,0 +1,23 @@ +Copyright (c) 2007-2019 Roy Marples +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. Modified: vendor/openresolv/dist/Makefile ============================================================================== --- vendor/openresolv/dist/Makefile Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/Makefile Wed Nov 20 22:13:14 2019 (r354917) @@ -10,6 +10,7 @@ SYSCONFDIR?= /etc LIBEXECDIR?= /libexec/resolvconf VARDIR?= /var/run/resolvconf +ECHO?= echo INSTALL?= install SED?= sed @@ -20,7 +21,7 @@ DOCMODE?= 0644 MANMODE?= 0444 RESOLVCONF= resolvconf resolvconf.8 resolvconf.conf.5 -SUBSCRIBERS= libc dnsmasq named pdnsd unbound +SUBSCRIBERS= libc dnsmasq named pdnsd pdns_recursor unbound TARGET= ${RESOLVCONF} ${SUBSCRIBERS} SRCS= ${TARGET:C,$,.in,} # pmake SRCS:= ${TARGET:=.in} # gmake @@ -42,7 +43,7 @@ DISTINFOSIGN= ${DISTINFO}.asc CKSUM?= cksum -a SHA256 PGP?= netpgp -FOSSILID?= current +GITREF?= HEAD .SUFFIXES: .in @@ -79,15 +80,17 @@ maninstall: install: proginstall maninstall -import: +dist-git: + git archive --prefix=${DISTPREFIX}/ ${GITREF} | xz >${DISTFILE} + +dist-inst: + mkdir /tmp/${DISTPREFIX} + cp -RPp * /tmp/${DISTPREFIX} + (cd /tmp/${DISTPREFIX}; make clean) + tar -cvjpf ${DISTFILE} -C /tmp ${DISTPREFIX} rm -rf /tmp/${DISTPREFIX} - ${INSTALL} -d /tmp/${DISTPREFIX} - cp README ${SRCS} /tmp/${DISTPREFIX} -dist: - fossil tarball --name ${DISTPREFIX} ${FOSSILID} ${DISTFILEGZ} - gunzip -c ${DISTFILEGZ} | xz >${DISTFILE} - rm ${DISTFILEGZ} +dist: dist-git distinfo: dist rm -f ${DISTINFO} ${DISTINFOSIGN} @@ -96,3 +99,20 @@ distinfo: dist ${PGP} --clearsign --output=${DISTINFOSIGN} ${DISTINFO} chmod 644 ${DISTINFOSIGN} ls -l ${DISTFILE} ${DISTINFO} ${DISTINFOSIGN} + +import: dist + rm -rf /tmp/${DISTPREFIX} + ${INSTALL} -d /tmp/${DISTPREFIX} + tar xvJpf ${DISTFILE} -C /tmp + +_import-src: + rm -rf ${DESTDIR}/* + ${INSTALL} -d ${DESTDIR} + cp LICENSE README.md ${SRCS} resolvconf.conf ${DESTDIR}; + cp resolvconf.8.in resolvconf.conf.5.in ${DESTDIR}; + @${ECHO} + @${ECHO} "=============================================================" + @${ECHO} "openresolv-${VERSION} imported to ${DESTDIR}" + +import-src: + ${MAKE} _import-src DESTDIR=`if [ -n "${DESTDIR}" ]; then echo "${DESTDIR}"; else echo /tmp/${DISTPREFIX}; fi` Added: vendor/openresolv/dist/README.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/dist/README.md Wed Nov 20 22:13:14 2019 (r354917) @@ -0,0 +1,64 @@ +# openresolv + +openresolv is a [resolvconf](https://en.wikipedia.org/wiki/Resolvconf) +implementation which manages `/etc/resolv.conf`. + +`/etc/resolv.conf` is a file that holds the configuration for the local +resolution of domain names. +Normally this file is either static or maintained by a local daemon, +normally a DHCP daemon. But what happens if more than one thing wants to +control the file? +Say you have wired and wireless interfaces to different subnets and run a VPN +or two on top of that, how do you say which one controls the file? +It's also not as easy as just adding and removing the nameservers each client +knows about as different clients could add the same nameservers. + +Enter resolvconf, the middleman between the network configuration services and +`/etc/resolv.conf`. +resolvconf itself is just a script that stores, removes and lists a full +`resolv.conf` generated for the interface. It then calls all the helper scripts +it knows about so it can configure the real `/etc/resolv.conf` and optionally +any local nameservers other than libc. + +## Reasons for using openresolv + +Why openresolv over the +[Debian implementation](http://qref.sourceforge.net/Debian/reference/ch-gateway.en.html#s-dns-resolvconf)? +Here's some reasons: + * Works with + [POSIX shell and userland](http://www.opengroup.org/onlinepubs/009695399) + * Does not need awk, grep or sed which means we can work without `/usr` + mounted + * Works with other init systems than Debians' out of the box + * Available as a 2 clause + [BSD license](http://www.freebsd.org/copyright/freebsd-license.html) + * Prefer configs via IF_METRIC for dynamic ordering + * Configures zones for local resolvers other than libc + +The last point is quite important, especially when running VPN systems. +Take the following resolv.conf files which have been generated by a +[DHCP client](../dhcpcd) and sent to resolvconf: + +``` +# resolv.conf from bge0 +search foo.com +nameserver 1.2.3.4 + +# resolv.conf from tap0 +domain bar.org +nameserver 5.6.7.8 +``` + +In this instance, queries for foo.com will go to 1.2.3.4 and queries for +bar.org will go to 5.6.7.8. +This does require the resolvers to be configured to pickup the resolvconf +generated configuration for them though. +openresolv ships with helpers for: + * [unbound](http://www.unbound.net/) + * [dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) + * [ISC BIND](http://www.isc.org/software/bind) + * [PowerDNS Recursor](http://wiki.powerdns.com/trac) + +See the +[configuration section](https://roy.marples.name/projects/openresolv/config) +for more details. Modified: vendor/openresolv/dist/configure ============================================================================== --- vendor/openresolv/dist/configure Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/configure Wed Nov 20 22:13:14 2019 (r354917) @@ -44,42 +44,8 @@ for x do esac done -if [ -z "$LIBEXECDIR" ]; then - printf "Checking for directory /libexec ... " - if [ -d /libexec ]; then - echo "yes" - LIBEXECDIR=$PREFIX/libexec/resolvconf - else - echo "no" - LIBEXECDIR=$PREFIX/lib/resolvconf - fi -fi -if [ -z "$RUNDIR" ]; then - printf "Checking for directory /run ... " - if [ -d /run ]; then - echo "yes" - RUNDIR=/run - else - echo "no" - RUNDIR=/var/run - fi -fi - : ${SED:=sed} -: ${SYSCONFDIR:=$PREFIX/etc} -: ${SBINDIR:=$PREFIX/sbin} -: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} -: ${STATEDIR:=/var} -: ${RUNDIR:=$STATEDIR/run} -: ${MANDIR:=${PREFIX:-/usr}/share/man} - -eval SYSCONFDIR="$SYSCONFDIR" -eval SBINDIR="$SBINDIR" -eval LIBEXECDIR="$LIBEXECDIR" -eval VARDIR="$RUNDIR/resolvconf" -eval MANDIR="$MANDIR" - CONFIG_MK=config.mk if [ -z "$BUILD" ]; then @@ -121,7 +87,19 @@ rm -rf $CONFIG_MK echo "# $OS" >$CONFIG_MK case "$OS" in -freebsd*) +dragonfly*) + # This means /usr HAS to be mounted not via dhcpcd + : ${LIBEXECDIR:=${PREFIX:-/usr}/libexec/resolvconf} + ;; +linux*) + # cksum does't support -a and netpgp is rare + echo "CKSUM= sha256sum --tag" >>$CONFIG_MK + echo "PGP= gpg2" >>$CONFIG_MK + ;; +esac + +case "$OS" in +dragonfly*|freebsd*) # On FreeBSD, /etc/init.d/foo status returns 0 if foo is not enabled # regardless of if it's not running. # So we force onestatus to work around this silly bug. @@ -129,12 +107,42 @@ freebsd*) STATUSARG="onestatus" fi ;; -linux*) - # cksum does't support -a and netpgp is rare - echo "CKSUM= sha256sum --tag" >>$CONFIG_MK - echo "PGP= gpg2" >>$CONFIG_MK - ;; esac + + +if [ -z "$LIBEXECDIR" ]; then + printf "Checking for directory /libexec ... " + if [ -d /libexec ]; then + echo "yes" + LIBEXECDIR=$PREFIX/libexec/resolvconf + else + echo "no" + LIBEXECDIR=$PREFIX/lib/resolvconf + fi +fi +if [ -z "$RUNDIR" ]; then + printf "Checking for directory /run ... " + if [ -d /run ]; then + echo "yes" + RUNDIR=/run + else + echo "no" + RUNDIR=/var/run + fi +fi + +: ${SYSCONFDIR:=$PREFIX/etc} +: ${SBINDIR:=$PREFIX/sbin} +: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} +: ${STATEDIR:=/var} +: ${RUNDIR:=$STATEDIR/run} +: ${MANDIR:=${PREFIX:-/usr}/share/man} + +eval SYSCONFDIR="$SYSCONFDIR" +eval SBINDIR="$SBINDIR" +eval LIBEXECDIR="$LIBEXECDIR" +eval VARDIR="$RUNDIR/resolvconf" +eval MANDIR="$MANDIR" for x in SYSCONFDIR SBINDIR LIBEXECDIR VARDIR MANDIR RESTARTCMD RCDIR STATUSARG do Modified: vendor/openresolv/dist/dnsmasq.in ============================================================================== --- vendor/openresolv/dist/dnsmasq.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/dnsmasq.in Wed Nov 20 22:13:14 2019 (r354917) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # dnsmasq subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$dnsmasq_conf" -a -z "$dnsmasq_resolv" ] && exit 0 +[ -z "${dnsmasq_conf}${dnsmasq_resolv}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -98,7 +98,7 @@ for d in $DOMAINS; do empty=false i=0 IFS=: set -- $n - while [ -n "$1" -o -n "$2" ]; do + while [ -n "$1" ] || [ -n "$2" ]; do addr="$1" shift if [ -z "$addr" ]; then @@ -184,7 +184,7 @@ if $changed; then eval $dnsmasq_restart elif [ -n "$RESTARTCMD" ]; then set -- ${dnsmasq_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${dnsmasq_service} fi @@ -206,4 +206,6 @@ if $dbus; then dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ /uk/org/thekelleys/dnsmasq uk.org.thekelleys.$method \ $dbusdest + dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ + /uk/org/thekelleys/dnsmasq uk.org.thekelleys.ClearCache fi Modified: vendor/openresolv/dist/libc.in ============================================================================== --- vendor/openresolv/dist/libc.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/libc.in Wed Nov 20 22:13:14 2019 (r354917) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # libc subscriber for resolvconf @@ -36,9 +36,9 @@ NL=" # sed may not be available, and this is faster on small files key_get_value() { - local key="$1" x= line= - + key="$1" shift + if [ $# -eq 0 ]; then while read -r line; do case "$line" in @@ -58,8 +58,6 @@ key_get_value() keys_remove() { - local key x line found - while read -r line; do found=false for key do @@ -79,7 +77,7 @@ local_nameservers="127.* 0.0.0.0 255.255.255.255 ::1" if [ -f "$SYSCONFDIR"/resolvconf.conf ]; then . "$SYSCONFDIR"/resolvconf.conf elif [ -d "$SYSCONFDIR"/resolvconf ]; then - SYSCONFDIR="$SYSCONFDIR/resolvconf/resolv.conf.d" + SYSCONFDIR="$SYSCONFDIR/resolvconf" base="$SYSCONFDIR/resolv.conf.d/base" if [ -f "$base" ]; then prepend_nameservers="$(key_get_value "nameserver " "$base")" @@ -98,10 +96,12 @@ fi : ${resolv_conf:=/etc/resolv.conf} : ${libc_service:=nscd} : ${list_resolv:=@SBINDIR@/resolvconf -l} -if [ "${resolv_conf_head-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.head ]; then +if [ "${resolv_conf_head-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.head ] +then resolv_conf_head="$(cat "${SYSCONFDIR}"/resolv.conf.head)" fi -if [ "${resolv_conf_tail-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.tail ]; then +if [ "${resolv_conf_tail-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.tail ] +then resolv_conf_tail="$(cat "$SYSCONFDIR"/resolv.conf.tail)" fi @@ -110,7 +110,7 @@ signature="# Generated by resolvconf" uniqify() { - local result= + result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -126,7 +126,7 @@ case "${resolv_conf_passthrough:-NO}" in backup=false newest= for conf in "$IFACEDIR"/*; do - if [ -z "$newest" -o "$conf" -nt "$newest" ]; then + if [ -z "$newest" ] || [ "$conf" -nt "$newest" ]; then newest="$conf" fi done @@ -178,7 +178,7 @@ case "${resolv_conf_passthrough:-NO}" in fi [ -n "$domain" ] && newconf="${newconf}domain $domain$NL" - if [ -n "$newsearch" -a "$newsearch" != "$domain" ]; then + if [ -n "$newsearch" ] && [ "$newsearch" != "$domain" ]; then newconf="${newconf}search $newsearch$NL" fi for n in $newns; do @@ -232,7 +232,7 @@ if [ -n "$libc_restart" ]; then eval $libc_restart elif [ -n "$RESTARTCMD" ]; then set -- ${libc_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${libc_service} fi Modified: vendor/openresolv/dist/named.in ============================================================================== --- vendor/openresolv/dist/named.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/named.in Wed Nov 20 22:13:14 2019 (r354917) @@ -28,14 +28,14 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$named_zones" -a -z "$named_options" ] && exit 0 +[ -z "${named_zones}${named_options}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " # Platform specific kludges -if [ -z "$named_service" -a -z "$named_restart" -a \ - -d "$RCDIR" -a ! -x "$RCDIR"/named ] +if [ -z "${named_service}${named_restart}" ] && + [ -d "$RCDIR" ] && ! [ -x "$RCDIR"/named ] then if [ -x "$RCDIR"/bind9 ]; then # Debian and derivatives @@ -111,7 +111,7 @@ if $changed; then eval $named_restart elif [ -n "$RESTARTCMD" ]; then set -- ${named_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${named_service} fi Modified: vendor/openresolv/dist/pdns_recursor.in ============================================================================== --- vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 22:13:14 2019 (r354917) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2009-2011 Roy Marples +# Copyright (c) 2009-2019 Roy Marples # All rights reserved # PowerDNS Recursor subscriber for resolvconf @@ -33,17 +33,14 @@ NL=" " -: ${pdns_service:=pdns_recursor} +: ${pdns_service:=pdns-recursor} newzones= -# pds_recursor does not present support global forward servers, which -# does limit it's usefulness somewhat. -# If it did, the below code can be enabled, or something like it. -#for n in $NAMESERVERS; do -# newzones="$newzones${newzones:+,}$n" -#done -#[ -n "$newzones" ] && newzones=".=$newzones$NL" +for n in $NAMESERVERS; do + newzones="$newzones${newzones:+,}$n" +done +[ -n "$newzones" ] && newzones="+.=$newzones$NL" for d in $DOMAINS; do newns= @@ -71,7 +68,7 @@ then eval $pdns_restart elif [ -n "$RESTARTCMD" ]; then set -- ${pdns_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${pdns_service} fi Modified: vendor/openresolv/dist/pdnsd.in ============================================================================== --- vendor/openresolv/dist/pdnsd.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/pdnsd.in Wed Nov 20 22:13:14 2019 (r354917) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2010-2013 Roy Marples +# Copyright (c) 2010-2018 Roy Marples # All rights reserved # pdnsd subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$pdnsd_conf" -a -z "$pdnsd_resolv" ] && exit 0 +[ -z "${pdnsd_conf}${pdnsd_resolv}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -41,14 +41,16 @@ signature_end="# End of resolvconf" # but sed may not always be available at the time. remove_markers() { - local m1="$1" m2="$2" x= line= in_marker=0 + m1="$1" + m2="$2" + in_marker=0 shift; shift if type sed >/dev/null 2>&1; then sed "/^$m1/,/^$m2/d" $@ else - for x; do - while read -r line; do + for x do + while read line; do case "$line" in "$m1"*) in_marker=1;; "$m2"*) in_marker=0;; Modified: vendor/openresolv/dist/resolvconf.conf ============================================================================== --- vendor/openresolv/dist/resolvconf.conf Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/resolvconf.conf Wed Nov 20 22:13:14 2019 (r354917) @@ -4,4 +4,4 @@ resolv_conf=/etc/resolv.conf # If you run a local name server, you should uncomment the below line and # configure your subscribers configuration files below. -#name_servers=127.0.0.1 \ No newline at end of file +#name_servers=127.0.0.1 Modified: vendor/openresolv/dist/resolvconf.conf.5.in ============================================================================== --- vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 22:13:14 2019 (r354917) @@ -22,7 +22,7 @@ .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" -.Dd December 29, 2016 +.Dd September 8, 2019 .Dt RESOLVCONF.CONF 5 .Os .Sh NAME @@ -64,19 +64,25 @@ Defaults to YES. .It Sy interface_order These interfaces will always be processed first. If unset, defaults to the following:- -.D1 lo lo[0-9]* +.Bd -compact -literal -offset indent +lo lo[0-9]* +.Ed .It Sy dynamic_order These interfaces will be processed next, unless they have a metric. If unset, defaults to the following:- -.D1 tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* +.Bd -compact -literal -offset indent +tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* +.Ed .It Sy inclusive_interfaces -Ignore any exlcusive marking for these interfaces. +Ignore any exclusive marking for these interfaces. This is handy when 3rd party integrations force the .Nm resolvconf -x option and you want to disable it easily. .It Sy local_nameservers If unset, defaults to the following:- -.D1 127.* 0.0.0.0 255.255.255.255 ::1 +.Bd -compact -literal -offset indent +127.* 0.0.0.0 255.255.255.255 ::1 +.Ed .It Sy search_domains Prepend search domains to the dynamically generated list. .It Sy search_domains_append @@ -118,16 +124,24 @@ The syntax is this: .Va $keyword Ns / Ns Va $match Ns / Ns Va $replacement .Pp Example, given this resolv.conf: -.D1 domain foo.org -.D1 search foo.org dead.beef -.D1 nameserver 1.2.3.4 -.D1 nameserver 2.3.4.5 +.Bd -compact -literal -offset indent +domain foo.org +search foo.org dead.beef +nameserver 1.2.3.4 +nameserver 2.3.4.5 +.Ed and this configuaration: -.D1 replace="search/foo*/bar.com nameserver/1.2.3.4/5.6.7.8 nameserver/2.3.4.5/" +.Bd -compact -literal -offset indent +replace="search/foo*/bar.com" +replace="$replace nameserver/1.2.3.4/5.6.7.8" +replace="$replace nameserver/2.3.4.5/" +.Ed you would get this resolv.conf instead: -.D1 domain foo.org -.D1 search bar.com -.D1 nameserver 5.6.7.8 +.Bd -compact -literal -offset indent +domain foo.org +search bar.com +nameserver 5.6.7.8 +.Ed .It Sy replace_sub Works the same way as .Sy replace @@ -138,9 +152,11 @@ Using the same example resolv.conf and changing to .Sy replace_sub , you would get this resolv.conf instead: -.D1 domain foo.org -.D1 search bar.com dead.beef -.D1 nameserver 5.6.7.8 +.Bd -compact -literal -offset indent +domain foo.org +search bar.com dead.beef +nameserver 5.6.7.8 +.Ed .It Sy state_dir Override the default state directory of .Pa @VARDIR@ . @@ -195,7 +211,8 @@ Prepend search domains to the dynamically generated li openresolv ships with subscribers for the name servers .Xr dnsmasq 8 , .Xr named 8 , -.Xr pdnsd 8 +.Xr pdnsd 8 , +.Xr pdns_recursor 8 , and .Xr unbound 8 . Each subscriber can create configuration files which should be included in @@ -203,7 +220,9 @@ in the subscribers main configuration file. .Pp To disable a subscriber, simply set it's name to NO. For example, to disable the libc subscriber you would set: -.D1 libc=NO +.Bd -compact -literal -offset indent +libc=NO +.Ed .Bl -tag -width indent .It Sy dnsmasq_conf This file tells dnsmasq which name servers to use for specific domains. @@ -211,17 +230,21 @@ This file tells dnsmasq which name servers to use for This file tells dnsmasq which name servers to use for global lookups. .Pp Example resolvconf.conf for dnsmasq: -.D1 name_servers=127.0.0.1 -.D1 dnsmasq_conf=/etc/dnsmasq-conf.conf -.D1 dnsmasq_resolv=/etc/dnsmasq-resolv.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +dnsmasq_conf=/etc/dnsmasq-conf.conf +dnsmasq_resolv=/etc/dnsmasq-resolv.conf +.Ed .Pp Example dnsmasq.conf: -.D1 listen-address=127.0.0.1 -.D1 # If dnsmasq is compiled for DBus then we can take -.D1 # advantage of not having to restart dnsmasq. -.D1 enable-dbus -.D1 conf-file=/etc/dnsmasq-conf.conf -.D1 resolv-file=/etc/dnsmasq-resolv.conf +.Bd -compact -literal -offset indent +listen-address=127.0.0.1 +# If dnsmasq is compiled for DBus then we can take +# advantage of not having to restart dnsmasq. +enable-dbus +conf-file=/etc/dnsmasq-conf.conf +resolv-file=/etc/dnsmasq-resolv.conf +.Ed .It Sy named_options Include this file in the named options block. This file tells named which name servers to use for global lookups. @@ -230,16 +253,21 @@ Include this file in the named global scope, after the This file tells named which name servers to use for specific domains. .Pp Example resolvconf.conf for named: -.D1 name_servers=127.0.0.1 -.D1 named_options=/etc/named-options.conf -.D1 named_zones=/etc/named-zones.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +named_options=/etc/named-options.conf +named_zones=/etc/named-zones.conf +.Ed .Pp Example named.conf: -.D1 options { -.D1 listen-on { 127.0.0.1; }; -.D1 include "/etc/named-options.conf"; -.D1 }; -.D1 include "/etc/named-zones.conf"; +.Bd -compact -literal -offset indent +options { + listen-on { 127.0.0.1; }; + include "/etc/named-options.conf"; +}; + +include "/etc/named-zones.conf"; +.Ed .It Sy pdnsd_conf This is the main pdnsd configuration file which we modify to add our forward domains to. @@ -253,32 +281,54 @@ If this variable is not set then it's written to .Pa pdnsd_conf . .Pp Example resolvconf.conf for pdnsd: -.D1 name_servers=127.0.0.1 -.D1 pdnsd_conf=/etc/pdnsd.conf -.D1 # pdnsd_resolv=/etc/pdnsd-resolv.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +pdnsd_conf=/etc/pdnsd.conf +# pdnsd_resolv=/etc/pdnsd-resolv.conf +.Ed .Pp Example pdnsd.conf: -.D1 global { -.D1 server_ip = 127.0.0.1; -.D1 status_ctl = on; -.D1 } -.D1 server { -.D1 # A server definition is required, even if emtpy. -.D1 label="empty"; -.D1 proxy_only=on; -.D1 # file="/etc/pdnsd-resolv.conf"; -.D1 } +.Bd -compact -literal -offset indent +global { + server_ip = 127.0.0.1; + status_ctl = on; +} +server { + # A server definition is required, even if empty. + label="empty"; + proxy_only=on; + # file="/etc/pdnsd-resolv.conf"; +} +.Ed +.It Sy pdns_zones +This file tells pdns_recursor about specific and global name servers. +.Pp +Example resolvconf.conf for pdns_recursor: +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +pdns_zones=/etc/pdns/recursor-zones.conf +.Ed +.Pp +Example recursor.conf: +.Bd -compact -literal -offset indent +allow-from=127.0.0.0/8, ::1/128 +forward-zones-file=/etc/pdns/recursor-zones.conf +.Ed .It Sy unbound_conf This file tells unbound about specific and global name servers. .It Sy unbound_insecure When set to YES, unbound marks the domains as insecure, thus ignoring DNSSEC. .Pp Example resolvconf.conf for unbound: -.D1 name_servers=127.0.0.1 -.D1 unbound_conf=/etc/unbound-resolvconf.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +unbound_conf=/etc/unbound-resolvconf.conf +.Ed .Pp Example unbound.conf: -.D1 include: /etc/unbound-resolvconf.conf +.Bd -compact -literal -offset indent +include: /etc/unbound-resolvconf.conf +.Ed .El .Sh SUBSCRIBER INTEGRATION Not all distributions store the files the subscribers need in the same @@ -292,7 +342,6 @@ Also, users could equally want to use a different vers installed by default, such as bind8 and bind9. To accommodate this, the subscribers have these files in configurable variables, documented below. -.Pp .Bl -tag -width indent .It Sy dnsmasq_service Name of the dnsmasq service. @@ -310,6 +359,10 @@ Name of the named service. Command to restart the named service. .It Sy pdnsd_restart Command to restart the pdnsd service. +.It Sy pdns_service +Command to restart the pdns_recursor service. +.It Sy pdns_restart +Command to restart the pdns_recursor service. .It Sy unbound_service Name of the unbound service. .It Sy unbound_restart Modified: vendor/openresolv/dist/resolvconf.in ============================================================================== --- vendor/openresolv/dist/resolvconf.in Wed Nov 20 21:48:27 2019 (r354916) +++ vendor/openresolv/dist/resolvconf.in Wed Nov 20 22:13:14 2019 (r354917) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # Redistribution and use in source and binary forms, with or without @@ -25,7 +25,7 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. RESOLVCONF="$0" -OPENRESOLV_VERSION="3.9.0" +OPENRESOLV_VERSION="3.9.2" SYSCONFDIR=@SYSCONFDIR@ LIBEXECDIR=@LIBEXECDIR@ VARDIR=@VARDIR@ @@ -125,21 +125,22 @@ usage() # If you think otherwise, capture a DNS trace and you'll see libc # will strip it regardless. # This also solves setting up duplicate zones in our subscribers. -strip_trailing_dots() +# Also strip any comments denoted by #. +resolv_strip() { - local n= d= - - for n; do - printf "$d%s" "${n%.}" - d=" " + space= + for word; do + case "$word" in + \#*) break;; + esac + printf "%s%s" "$space${word%.}" + space=" " done printf "\n" } private_iface() { - local p - # Allow expansion cd "$IFACEDIR" @@ -168,12 +169,15 @@ private_iface() # for domain name servers, search name servers and global nameservers parse_resolv() { - local line= ns= ds= search= d= n= newns= - local new=true iface= private=false p= domain= l= islocal= - + domain= + new=true newns= + ns= + private=false + search= while read -r line; do + stripped_line="$(resolv_strip ${line#* })" case "$line" in "# resolv.conf from "*) if ${new}; then @@ -189,29 +193,32 @@ parse_resolv() "nameserver "*) islocal=false for l in $local_nameservers; do - case "${line#* }" in + case "$stripped_line" in $l) islocal=true - echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS ${line#* }\"" break ;; esac done - $islocal || ns="$ns${line#* } " + if $islocal; then + echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS $stripped_line\"" + else + ns="$ns$stripped_line " + fi ;; "domain "*) - search="$(strip_trailing_dots ${line#* })" + search="$stripped_line" if [ -z "$domain" ]; then domain="$search" echo "DOMAIN=\"$domain\"" fi ;; "search "*) - search="$(strip_trailing_dots ${line#* })" + search="$stripped_line" ;; *) [ -n "$line" ] && continue - if [ -n "$ns" -a -n "$search" ]; then + if [ -n "$ns" ] && [ -n "$search" ]; then newns= for n in $ns; do newns="$newns${newns:+,}$n" @@ -236,7 +243,7 @@ parse_resolv() uniqify() { - local result= + result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -249,8 +256,8 @@ uniqify() dirname() { - local dir= OIFS="$IFS" - local IFS=/ + OIFS="$IFS" + IFS=/ set -- $@ IFS="$OIFS" if [ -n "$1" ]; then @@ -267,7 +274,7 @@ dirname() config_mkdirs() { - local e=0 f d + e=0 for f; do [ -n "$f" ] || continue d="$(dirname "$f")" @@ -295,66 +302,86 @@ detect_init() # Detect the running init system. # As systemd and OpenRC can be installed on top of legacy init # systems we try to detect them first. - local status="@STATUSARG@" + status="@STATUSARG@" : ${status:=status} - if [ -x /bin/systemctl -a -S /run/systemd/private ]; then - RESTARTCMD="if /bin/systemctl --quiet is-active \$1.service; then - /bin/systemctl restart \$1.service; -fi" - elif [ -x /usr/bin/systemctl -a -S /run/systemd/private ]; then - RESTARTCMD="if /usr/bin/systemctl --quiet is-active \$1.service; then - /usr/bin/systemctl restart \$1.service; -fi" - elif [ -x /sbin/rc-service -a \ - -s /libexec/rc/init.d/softlevel -o -s /run/openrc/softlevel ] + if [ -x /bin/systemctl ] && [ -S /run/systemd/private ]; then + RESTARTCMD=' + if /bin/systemctl --quiet is-active $1.service + then + /bin/systemctl restart $1.service + fi' + elif [ -x /usr/bin/systemctl ] && [ -S /run/systemd/private ]; then + RESTARTCMD=' + if /usr/bin/systemctl --quiet is-active $1.service + then + /usr/bin/systemctl restart $1.service + fi' + elif [ -x /sbin/rc-service ] && + { [ -s /libexec/rc/init.d/softlevel ] || + [ -s /run/openrc/softlevel ]; } then - RESTARTCMD="/sbin/rc-service -i \$1 -- -Ds restart" + RESTARTCMD='/sbin/rc-service -i $1 -- -Ds restart' elif [ -x /usr/sbin/invoke-rc.d ]; then RCDIR=/etc/init.d - RESTARTCMD="if /usr/sbin/invoke-rc.d --quiet \$1 status 1>/dev/null 2>&1; then - /usr/sbin/invoke-rc.d \$1 restart; -fi" + RESTARTCMD=' + if /usr/sbin/invoke-rc.d --quiet $1 status >/dev/null 2>&1 + then + /usr/sbin/invoke-rc.d $1 restart + fi' elif [ -x /sbin/service ]; then # Old RedHat RCDIR=/etc/init.d - RESTARTCMD="if /sbin/service \$1; then - /sbin/service \$1 restart; -fi" + RESTARTCMD=' + if /sbin/service $1; then + /sbin/service $1 restart + fi' elif [ -x /usr/sbin/service ]; then # Could be FreeBSD - RESTARTCMD="if /usr/sbin/service \$1 $status 1>/dev/null 2>&1; then - /usr/sbin/service \$1 restart; -fi" + RESTARTCMD=" + if /usr/sbin/service \$1 $status >/dev/null 2>&1 + then + /usr/sbin/service \$1 restart + fi" elif [ -x /bin/sv ]; then - RESTARTCMD="/bin/sv status \$1 >/dev/null 2>&1 && /bin/sv try-restart \$1" + RESTARTCMD='/bin/sv status $1 >/dev/null 2>&1 && + /bin/sv try-restart $1' elif [ -x /usr/bin/sv ]; then - RESTARTCMD="/usr/bin/sv status \$1 >/dev/null 2>&1 && /usr/bin/sv try-restart \$1" - elif [ -e /etc/arch-release -a -d /etc/rc.d ]; then *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Nov 20 22:20:14 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 495741C331A; Wed, 20 Nov 2019 22:20:14 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JHFB1dXVz4QkM; Wed, 20 Nov 2019 22:20:14 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F3D10163F; Wed, 20 Nov 2019 22:20:13 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAKMKDoF085975; Wed, 20 Nov 2019 22:20:13 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAKMKBIU085964; Wed, 20 Nov 2019 22:20:11 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201911202220.xAKMKBIU085964@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 20 Nov 2019 22:20:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354918 - vendor/openresolv/dist X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: vendor/openresolv/dist X-SVN-Commit-Revision: 354918 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Nov 2019 22:20:14 -0000 Author: pfg Date: Wed Nov 20 22:20:11 2019 New Revision: 354918 URL: https://svnweb.freebsd.org/changeset/base/354918 Log: Undo r354917 to correct the log: it was actually version 3.9.2 Added: vendor/openresolv/dist/README - copied unchanged from r354916, vendor/openresolv/dist/README Deleted: vendor/openresolv/dist/LICENSE vendor/openresolv/dist/README.md Modified: vendor/openresolv/dist/Makefile vendor/openresolv/dist/configure vendor/openresolv/dist/dnsmasq.in vendor/openresolv/dist/libc.in vendor/openresolv/dist/named.in vendor/openresolv/dist/pdns_recursor.in vendor/openresolv/dist/pdnsd.in vendor/openresolv/dist/resolvconf.conf vendor/openresolv/dist/resolvconf.conf.5.in vendor/openresolv/dist/resolvconf.in vendor/openresolv/dist/unbound.in Modified: vendor/openresolv/dist/Makefile ============================================================================== --- vendor/openresolv/dist/Makefile Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/Makefile Wed Nov 20 22:20:11 2019 (r354918) @@ -10,7 +10,6 @@ SYSCONFDIR?= /etc LIBEXECDIR?= /libexec/resolvconf VARDIR?= /var/run/resolvconf -ECHO?= echo INSTALL?= install SED?= sed @@ -21,7 +20,7 @@ DOCMODE?= 0644 MANMODE?= 0444 RESOLVCONF= resolvconf resolvconf.8 resolvconf.conf.5 -SUBSCRIBERS= libc dnsmasq named pdnsd pdns_recursor unbound +SUBSCRIBERS= libc dnsmasq named pdnsd unbound TARGET= ${RESOLVCONF} ${SUBSCRIBERS} SRCS= ${TARGET:C,$,.in,} # pmake SRCS:= ${TARGET:=.in} # gmake @@ -43,7 +42,7 @@ DISTINFOSIGN= ${DISTINFO}.asc CKSUM?= cksum -a SHA256 PGP?= netpgp -GITREF?= HEAD +FOSSILID?= current .SUFFIXES: .in @@ -80,17 +79,15 @@ maninstall: install: proginstall maninstall -dist-git: - git archive --prefix=${DISTPREFIX}/ ${GITREF} | xz >${DISTFILE} - -dist-inst: - mkdir /tmp/${DISTPREFIX} - cp -RPp * /tmp/${DISTPREFIX} - (cd /tmp/${DISTPREFIX}; make clean) - tar -cvjpf ${DISTFILE} -C /tmp ${DISTPREFIX} +import: rm -rf /tmp/${DISTPREFIX} + ${INSTALL} -d /tmp/${DISTPREFIX} + cp README ${SRCS} /tmp/${DISTPREFIX} -dist: dist-git +dist: + fossil tarball --name ${DISTPREFIX} ${FOSSILID} ${DISTFILEGZ} + gunzip -c ${DISTFILEGZ} | xz >${DISTFILE} + rm ${DISTFILEGZ} distinfo: dist rm -f ${DISTINFO} ${DISTINFOSIGN} @@ -99,20 +96,3 @@ distinfo: dist ${PGP} --clearsign --output=${DISTINFOSIGN} ${DISTINFO} chmod 644 ${DISTINFOSIGN} ls -l ${DISTFILE} ${DISTINFO} ${DISTINFOSIGN} - -import: dist - rm -rf /tmp/${DISTPREFIX} - ${INSTALL} -d /tmp/${DISTPREFIX} - tar xvJpf ${DISTFILE} -C /tmp - -_import-src: - rm -rf ${DESTDIR}/* - ${INSTALL} -d ${DESTDIR} - cp LICENSE README.md ${SRCS} resolvconf.conf ${DESTDIR}; - cp resolvconf.8.in resolvconf.conf.5.in ${DESTDIR}; - @${ECHO} - @${ECHO} "=============================================================" - @${ECHO} "openresolv-${VERSION} imported to ${DESTDIR}" - -import-src: - ${MAKE} _import-src DESTDIR=`if [ -n "${DESTDIR}" ]; then echo "${DESTDIR}"; else echo /tmp/${DISTPREFIX}; fi` Copied: vendor/openresolv/dist/README (from r354916, vendor/openresolv/dist/README) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/dist/README Wed Nov 20 22:20:11 2019 (r354918, copy of r354916, vendor/openresolv/dist/README) @@ -0,0 +1,11 @@ +openresolv is a resolvconf implementation which manages resolv.conf +You can find the latest version at http://roy.marples.name/projects/openresolv +It is written and maintained by Roy Marples + +This resolvconf implementation, along with its subscribers, work with a +POSIX compliant shell and userland utilities. It is designed to work without +tools such as sed as it *has* to work without /usr being available. + +On systems where resolvconf is expected to be used before /var/run is available +for writing, you can configure openresolv to write somewhere else, like say a +ramdisk. Modified: vendor/openresolv/dist/configure ============================================================================== --- vendor/openresolv/dist/configure Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/configure Wed Nov 20 22:20:11 2019 (r354918) @@ -44,8 +44,42 @@ for x do esac done +if [ -z "$LIBEXECDIR" ]; then + printf "Checking for directory /libexec ... " + if [ -d /libexec ]; then + echo "yes" + LIBEXECDIR=$PREFIX/libexec/resolvconf + else + echo "no" + LIBEXECDIR=$PREFIX/lib/resolvconf + fi +fi +if [ -z "$RUNDIR" ]; then + printf "Checking for directory /run ... " + if [ -d /run ]; then + echo "yes" + RUNDIR=/run + else + echo "no" + RUNDIR=/var/run + fi +fi + : ${SED:=sed} +: ${SYSCONFDIR:=$PREFIX/etc} +: ${SBINDIR:=$PREFIX/sbin} +: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} +: ${STATEDIR:=/var} +: ${RUNDIR:=$STATEDIR/run} +: ${MANDIR:=${PREFIX:-/usr}/share/man} + +eval SYSCONFDIR="$SYSCONFDIR" +eval SBINDIR="$SBINDIR" +eval LIBEXECDIR="$LIBEXECDIR" +eval VARDIR="$RUNDIR/resolvconf" +eval MANDIR="$MANDIR" + CONFIG_MK=config.mk if [ -z "$BUILD" ]; then @@ -87,19 +121,7 @@ rm -rf $CONFIG_MK echo "# $OS" >$CONFIG_MK case "$OS" in -dragonfly*) - # This means /usr HAS to be mounted not via dhcpcd - : ${LIBEXECDIR:=${PREFIX:-/usr}/libexec/resolvconf} - ;; -linux*) - # cksum does't support -a and netpgp is rare - echo "CKSUM= sha256sum --tag" >>$CONFIG_MK - echo "PGP= gpg2" >>$CONFIG_MK - ;; -esac - -case "$OS" in -dragonfly*|freebsd*) +freebsd*) # On FreeBSD, /etc/init.d/foo status returns 0 if foo is not enabled # regardless of if it's not running. # So we force onestatus to work around this silly bug. @@ -107,42 +129,12 @@ dragonfly*|freebsd*) STATUSARG="onestatus" fi ;; +linux*) + # cksum does't support -a and netpgp is rare + echo "CKSUM= sha256sum --tag" >>$CONFIG_MK + echo "PGP= gpg2" >>$CONFIG_MK + ;; esac - - -if [ -z "$LIBEXECDIR" ]; then - printf "Checking for directory /libexec ... " - if [ -d /libexec ]; then - echo "yes" - LIBEXECDIR=$PREFIX/libexec/resolvconf - else - echo "no" - LIBEXECDIR=$PREFIX/lib/resolvconf - fi -fi -if [ -z "$RUNDIR" ]; then - printf "Checking for directory /run ... " - if [ -d /run ]; then - echo "yes" - RUNDIR=/run - else - echo "no" - RUNDIR=/var/run - fi -fi - -: ${SYSCONFDIR:=$PREFIX/etc} -: ${SBINDIR:=$PREFIX/sbin} -: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} -: ${STATEDIR:=/var} -: ${RUNDIR:=$STATEDIR/run} -: ${MANDIR:=${PREFIX:-/usr}/share/man} - -eval SYSCONFDIR="$SYSCONFDIR" -eval SBINDIR="$SBINDIR" -eval LIBEXECDIR="$LIBEXECDIR" -eval VARDIR="$RUNDIR/resolvconf" -eval MANDIR="$MANDIR" for x in SYSCONFDIR SBINDIR LIBEXECDIR VARDIR MANDIR RESTARTCMD RCDIR STATUSARG do Modified: vendor/openresolv/dist/dnsmasq.in ============================================================================== --- vendor/openresolv/dist/dnsmasq.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/dnsmasq.in Wed Nov 20 22:20:11 2019 (r354918) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2019 Roy Marples +# Copyright (c) 2007-2016 Roy Marples # All rights reserved # dnsmasq subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "${dnsmasq_conf}${dnsmasq_resolv}" ] && exit 0 +[ -z "$dnsmasq_conf" -a -z "$dnsmasq_resolv" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -98,7 +98,7 @@ for d in $DOMAINS; do empty=false i=0 IFS=: set -- $n - while [ -n "$1" ] || [ -n "$2" ]; do + while [ -n "$1" -o -n "$2" ]; do addr="$1" shift if [ -z "$addr" ]; then @@ -184,7 +184,7 @@ if $changed; then eval $dnsmasq_restart elif [ -n "$RESTARTCMD" ]; then set -- ${dnsmasq_service} - eval "$RESTARTCMD" + eval $RESTARTCMD else @SBINDIR@/resolvconf -r ${dnsmasq_service} fi @@ -206,6 +206,4 @@ if $dbus; then dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ /uk/org/thekelleys/dnsmasq uk.org.thekelleys.$method \ $dbusdest - dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ - /uk/org/thekelleys/dnsmasq uk.org.thekelleys.ClearCache fi Modified: vendor/openresolv/dist/libc.in ============================================================================== --- vendor/openresolv/dist/libc.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/libc.in Wed Nov 20 22:20:11 2019 (r354918) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2019 Roy Marples +# Copyright (c) 2007-2016 Roy Marples # All rights reserved # libc subscriber for resolvconf @@ -36,9 +36,9 @@ NL=" # sed may not be available, and this is faster on small files key_get_value() { - key="$1" - shift + local key="$1" x= line= + shift if [ $# -eq 0 ]; then while read -r line; do case "$line" in @@ -58,6 +58,8 @@ key_get_value() keys_remove() { + local key x line found + while read -r line; do found=false for key do @@ -77,7 +79,7 @@ local_nameservers="127.* 0.0.0.0 255.255.255.255 ::1" if [ -f "$SYSCONFDIR"/resolvconf.conf ]; then . "$SYSCONFDIR"/resolvconf.conf elif [ -d "$SYSCONFDIR"/resolvconf ]; then - SYSCONFDIR="$SYSCONFDIR/resolvconf" + SYSCONFDIR="$SYSCONFDIR/resolvconf/resolv.conf.d" base="$SYSCONFDIR/resolv.conf.d/base" if [ -f "$base" ]; then prepend_nameservers="$(key_get_value "nameserver " "$base")" @@ -96,12 +98,10 @@ fi : ${resolv_conf:=/etc/resolv.conf} : ${libc_service:=nscd} : ${list_resolv:=@SBINDIR@/resolvconf -l} -if [ "${resolv_conf_head-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.head ] -then +if [ "${resolv_conf_head-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.head ]; then resolv_conf_head="$(cat "${SYSCONFDIR}"/resolv.conf.head)" fi -if [ "${resolv_conf_tail-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.tail ] -then +if [ "${resolv_conf_tail-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.tail ]; then resolv_conf_tail="$(cat "$SYSCONFDIR"/resolv.conf.tail)" fi @@ -110,7 +110,7 @@ signature="# Generated by resolvconf" uniqify() { - result= + local result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -126,7 +126,7 @@ case "${resolv_conf_passthrough:-NO}" in backup=false newest= for conf in "$IFACEDIR"/*; do - if [ -z "$newest" ] || [ "$conf" -nt "$newest" ]; then + if [ -z "$newest" -o "$conf" -nt "$newest" ]; then newest="$conf" fi done @@ -178,7 +178,7 @@ case "${resolv_conf_passthrough:-NO}" in fi [ -n "$domain" ] && newconf="${newconf}domain $domain$NL" - if [ -n "$newsearch" ] && [ "$newsearch" != "$domain" ]; then + if [ -n "$newsearch" -a "$newsearch" != "$domain" ]; then newconf="${newconf}search $newsearch$NL" fi for n in $newns; do @@ -232,7 +232,7 @@ if [ -n "$libc_restart" ]; then eval $libc_restart elif [ -n "$RESTARTCMD" ]; then set -- ${libc_service} - eval "$RESTARTCMD" + eval $RESTARTCMD else @SBINDIR@/resolvconf -r ${libc_service} fi Modified: vendor/openresolv/dist/named.in ============================================================================== --- vendor/openresolv/dist/named.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/named.in Wed Nov 20 22:20:11 2019 (r354918) @@ -28,14 +28,14 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "${named_zones}${named_options}" ] && exit 0 +[ -z "$named_zones" -a -z "$named_options" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " # Platform specific kludges -if [ -z "${named_service}${named_restart}" ] && - [ -d "$RCDIR" ] && ! [ -x "$RCDIR"/named ] +if [ -z "$named_service" -a -z "$named_restart" -a \ + -d "$RCDIR" -a ! -x "$RCDIR"/named ] then if [ -x "$RCDIR"/bind9 ]; then # Debian and derivatives @@ -111,7 +111,7 @@ if $changed; then eval $named_restart elif [ -n "$RESTARTCMD" ]; then set -- ${named_service} - eval "$RESTARTCMD" + eval $RESTARTCMD else @SBINDIR@/resolvconf -r ${named_service} fi Modified: vendor/openresolv/dist/pdns_recursor.in ============================================================================== --- vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 22:20:11 2019 (r354918) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2009-2019 Roy Marples +# Copyright (c) 2009-2011 Roy Marples # All rights reserved # PowerDNS Recursor subscriber for resolvconf @@ -33,14 +33,17 @@ NL=" " -: ${pdns_service:=pdns-recursor} +: ${pdns_service:=pdns_recursor} newzones= -for n in $NAMESERVERS; do - newzones="$newzones${newzones:+,}$n" -done -[ -n "$newzones" ] && newzones="+.=$newzones$NL" +# pds_recursor does not present support global forward servers, which +# does limit it's usefulness somewhat. +# If it did, the below code can be enabled, or something like it. +#for n in $NAMESERVERS; do +# newzones="$newzones${newzones:+,}$n" +#done +#[ -n "$newzones" ] && newzones=".=$newzones$NL" for d in $DOMAINS; do newns= @@ -68,7 +71,7 @@ then eval $pdns_restart elif [ -n "$RESTARTCMD" ]; then set -- ${pdns_service} - eval "$RESTARTCMD" + eval $RESTARTCMD else @SBINDIR@/resolvconf -r ${pdns_service} fi Modified: vendor/openresolv/dist/pdnsd.in ============================================================================== --- vendor/openresolv/dist/pdnsd.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/pdnsd.in Wed Nov 20 22:20:11 2019 (r354918) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2010-2018 Roy Marples +# Copyright (c) 2010-2013 Roy Marples # All rights reserved # pdnsd subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "${pdnsd_conf}${pdnsd_resolv}" ] && exit 0 +[ -z "$pdnsd_conf" -a -z "$pdnsd_resolv" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -41,16 +41,14 @@ signature_end="# End of resolvconf" # but sed may not always be available at the time. remove_markers() { - m1="$1" - m2="$2" - in_marker=0 + local m1="$1" m2="$2" x= line= in_marker=0 shift; shift if type sed >/dev/null 2>&1; then sed "/^$m1/,/^$m2/d" $@ else - for x do - while read line; do + for x; do + while read -r line; do case "$line" in "$m1"*) in_marker=1;; "$m2"*) in_marker=0;; Modified: vendor/openresolv/dist/resolvconf.conf ============================================================================== --- vendor/openresolv/dist/resolvconf.conf Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/resolvconf.conf Wed Nov 20 22:20:11 2019 (r354918) @@ -4,4 +4,4 @@ resolv_conf=/etc/resolv.conf # If you run a local name server, you should uncomment the below line and # configure your subscribers configuration files below. -#name_servers=127.0.0.1 +#name_servers=127.0.0.1 \ No newline at end of file Modified: vendor/openresolv/dist/resolvconf.conf.5.in ============================================================================== --- vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 22:20:11 2019 (r354918) @@ -22,7 +22,7 @@ .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" -.Dd September 8, 2019 +.Dd December 29, 2016 .Dt RESOLVCONF.CONF 5 .Os .Sh NAME @@ -64,25 +64,19 @@ Defaults to YES. .It Sy interface_order These interfaces will always be processed first. If unset, defaults to the following:- -.Bd -compact -literal -offset indent -lo lo[0-9]* -.Ed +.D1 lo lo[0-9]* .It Sy dynamic_order These interfaces will be processed next, unless they have a metric. If unset, defaults to the following:- -.Bd -compact -literal -offset indent -tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* -.Ed +.D1 tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* .It Sy inclusive_interfaces -Ignore any exclusive marking for these interfaces. +Ignore any exlcusive marking for these interfaces. This is handy when 3rd party integrations force the .Nm resolvconf -x option and you want to disable it easily. .It Sy local_nameservers If unset, defaults to the following:- -.Bd -compact -literal -offset indent -127.* 0.0.0.0 255.255.255.255 ::1 -.Ed +.D1 127.* 0.0.0.0 255.255.255.255 ::1 .It Sy search_domains Prepend search domains to the dynamically generated list. .It Sy search_domains_append @@ -124,24 +118,16 @@ The syntax is this: .Va $keyword Ns / Ns Va $match Ns / Ns Va $replacement .Pp Example, given this resolv.conf: -.Bd -compact -literal -offset indent -domain foo.org -search foo.org dead.beef -nameserver 1.2.3.4 -nameserver 2.3.4.5 -.Ed +.D1 domain foo.org +.D1 search foo.org dead.beef +.D1 nameserver 1.2.3.4 +.D1 nameserver 2.3.4.5 and this configuaration: -.Bd -compact -literal -offset indent -replace="search/foo*/bar.com" -replace="$replace nameserver/1.2.3.4/5.6.7.8" -replace="$replace nameserver/2.3.4.5/" -.Ed +.D1 replace="search/foo*/bar.com nameserver/1.2.3.4/5.6.7.8 nameserver/2.3.4.5/" you would get this resolv.conf instead: -.Bd -compact -literal -offset indent -domain foo.org -search bar.com -nameserver 5.6.7.8 -.Ed +.D1 domain foo.org +.D1 search bar.com +.D1 nameserver 5.6.7.8 .It Sy replace_sub Works the same way as .Sy replace @@ -152,11 +138,9 @@ Using the same example resolv.conf and changing to .Sy replace_sub , you would get this resolv.conf instead: -.Bd -compact -literal -offset indent -domain foo.org -search bar.com dead.beef -nameserver 5.6.7.8 -.Ed +.D1 domain foo.org +.D1 search bar.com dead.beef +.D1 nameserver 5.6.7.8 .It Sy state_dir Override the default state directory of .Pa @VARDIR@ . @@ -211,8 +195,7 @@ Prepend search domains to the dynamically generated li openresolv ships with subscribers for the name servers .Xr dnsmasq 8 , .Xr named 8 , -.Xr pdnsd 8 , -.Xr pdns_recursor 8 , +.Xr pdnsd 8 and .Xr unbound 8 . Each subscriber can create configuration files which should be included in @@ -220,9 +203,7 @@ in the subscribers main configuration file. .Pp To disable a subscriber, simply set it's name to NO. For example, to disable the libc subscriber you would set: -.Bd -compact -literal -offset indent -libc=NO -.Ed +.D1 libc=NO .Bl -tag -width indent .It Sy dnsmasq_conf This file tells dnsmasq which name servers to use for specific domains. @@ -230,21 +211,17 @@ This file tells dnsmasq which name servers to use for This file tells dnsmasq which name servers to use for global lookups. .Pp Example resolvconf.conf for dnsmasq: -.Bd -compact -literal -offset indent -name_servers=127.0.0.1 -dnsmasq_conf=/etc/dnsmasq-conf.conf -dnsmasq_resolv=/etc/dnsmasq-resolv.conf -.Ed +.D1 name_servers=127.0.0.1 +.D1 dnsmasq_conf=/etc/dnsmasq-conf.conf +.D1 dnsmasq_resolv=/etc/dnsmasq-resolv.conf .Pp Example dnsmasq.conf: -.Bd -compact -literal -offset indent -listen-address=127.0.0.1 -# If dnsmasq is compiled for DBus then we can take -# advantage of not having to restart dnsmasq. -enable-dbus -conf-file=/etc/dnsmasq-conf.conf -resolv-file=/etc/dnsmasq-resolv.conf -.Ed +.D1 listen-address=127.0.0.1 +.D1 # If dnsmasq is compiled for DBus then we can take +.D1 # advantage of not having to restart dnsmasq. +.D1 enable-dbus +.D1 conf-file=/etc/dnsmasq-conf.conf +.D1 resolv-file=/etc/dnsmasq-resolv.conf .It Sy named_options Include this file in the named options block. This file tells named which name servers to use for global lookups. @@ -253,21 +230,16 @@ Include this file in the named global scope, after the This file tells named which name servers to use for specific domains. .Pp Example resolvconf.conf for named: -.Bd -compact -literal -offset indent -name_servers=127.0.0.1 -named_options=/etc/named-options.conf -named_zones=/etc/named-zones.conf -.Ed +.D1 name_servers=127.0.0.1 +.D1 named_options=/etc/named-options.conf +.D1 named_zones=/etc/named-zones.conf .Pp Example named.conf: -.Bd -compact -literal -offset indent -options { - listen-on { 127.0.0.1; }; - include "/etc/named-options.conf"; -}; - -include "/etc/named-zones.conf"; -.Ed +.D1 options { +.D1 listen-on { 127.0.0.1; }; +.D1 include "/etc/named-options.conf"; +.D1 }; +.D1 include "/etc/named-zones.conf"; .It Sy pdnsd_conf This is the main pdnsd configuration file which we modify to add our forward domains to. @@ -281,54 +253,32 @@ If this variable is not set then it's written to .Pa pdnsd_conf . .Pp Example resolvconf.conf for pdnsd: -.Bd -compact -literal -offset indent -name_servers=127.0.0.1 -pdnsd_conf=/etc/pdnsd.conf -# pdnsd_resolv=/etc/pdnsd-resolv.conf -.Ed +.D1 name_servers=127.0.0.1 +.D1 pdnsd_conf=/etc/pdnsd.conf +.D1 # pdnsd_resolv=/etc/pdnsd-resolv.conf .Pp Example pdnsd.conf: -.Bd -compact -literal -offset indent -global { - server_ip = 127.0.0.1; - status_ctl = on; -} -server { - # A server definition is required, even if empty. - label="empty"; - proxy_only=on; - # file="/etc/pdnsd-resolv.conf"; -} -.Ed -.It Sy pdns_zones -This file tells pdns_recursor about specific and global name servers. -.Pp -Example resolvconf.conf for pdns_recursor: -.Bd -compact -literal -offset indent -name_servers=127.0.0.1 -pdns_zones=/etc/pdns/recursor-zones.conf -.Ed -.Pp -Example recursor.conf: -.Bd -compact -literal -offset indent -allow-from=127.0.0.0/8, ::1/128 -forward-zones-file=/etc/pdns/recursor-zones.conf -.Ed +.D1 global { +.D1 server_ip = 127.0.0.1; +.D1 status_ctl = on; +.D1 } +.D1 server { +.D1 # A server definition is required, even if emtpy. +.D1 label="empty"; +.D1 proxy_only=on; +.D1 # file="/etc/pdnsd-resolv.conf"; +.D1 } .It Sy unbound_conf This file tells unbound about specific and global name servers. .It Sy unbound_insecure When set to YES, unbound marks the domains as insecure, thus ignoring DNSSEC. .Pp Example resolvconf.conf for unbound: -.Bd -compact -literal -offset indent -name_servers=127.0.0.1 -unbound_conf=/etc/unbound-resolvconf.conf -.Ed +.D1 name_servers=127.0.0.1 +.D1 unbound_conf=/etc/unbound-resolvconf.conf .Pp Example unbound.conf: -.Bd -compact -literal -offset indent -include: /etc/unbound-resolvconf.conf -.Ed +.D1 include: /etc/unbound-resolvconf.conf .El .Sh SUBSCRIBER INTEGRATION Not all distributions store the files the subscribers need in the same @@ -342,6 +292,7 @@ Also, users could equally want to use a different vers installed by default, such as bind8 and bind9. To accommodate this, the subscribers have these files in configurable variables, documented below. +.Pp .Bl -tag -width indent .It Sy dnsmasq_service Name of the dnsmasq service. @@ -359,10 +310,6 @@ Name of the named service. Command to restart the named service. .It Sy pdnsd_restart Command to restart the pdnsd service. -.It Sy pdns_service -Command to restart the pdns_recursor service. -.It Sy pdns_restart -Command to restart the pdns_recursor service. .It Sy unbound_service Name of the unbound service. .It Sy unbound_restart Modified: vendor/openresolv/dist/resolvconf.in ============================================================================== --- vendor/openresolv/dist/resolvconf.in Wed Nov 20 22:13:14 2019 (r354917) +++ vendor/openresolv/dist/resolvconf.in Wed Nov 20 22:20:11 2019 (r354918) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2019 Roy Marples +# Copyright (c) 2007-2016 Roy Marples # All rights reserved # Redistribution and use in source and binary forms, with or without @@ -25,7 +25,7 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. RESOLVCONF="$0" -OPENRESOLV_VERSION="3.9.2" +OPENRESOLV_VERSION="3.9.0" SYSCONFDIR=@SYSCONFDIR@ LIBEXECDIR=@LIBEXECDIR@ VARDIR=@VARDIR@ @@ -125,22 +125,21 @@ usage() # If you think otherwise, capture a DNS trace and you'll see libc # will strip it regardless. # This also solves setting up duplicate zones in our subscribers. -# Also strip any comments denoted by #. -resolv_strip() +strip_trailing_dots() { - space= - for word; do - case "$word" in - \#*) break;; - esac - printf "%s%s" "$space${word%.}" - space=" " + local n= d= + + for n; do + printf "$d%s" "${n%.}" + d=" " done printf "\n" } private_iface() { + local p + # Allow expansion cd "$IFACEDIR" @@ -169,15 +168,12 @@ private_iface() # for domain name servers, search name servers and global nameservers parse_resolv() { - domain= - new=true + local line= ns= ds= search= d= n= newns= + local new=true iface= private=false p= domain= l= islocal= + newns= - ns= - private=false - search= while read -r line; do - stripped_line="$(resolv_strip ${line#* })" case "$line" in "# resolv.conf from "*) if ${new}; then @@ -193,32 +189,29 @@ parse_resolv() "nameserver "*) islocal=false for l in $local_nameservers; do - case "$stripped_line" in + case "${line#* }" in $l) islocal=true + echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS ${line#* }\"" break ;; esac done - if $islocal; then - echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS $stripped_line\"" - else - ns="$ns$stripped_line " - fi + $islocal || ns="$ns${line#* } " ;; "domain "*) - search="$stripped_line" + search="$(strip_trailing_dots ${line#* })" if [ -z "$domain" ]; then domain="$search" echo "DOMAIN=\"$domain\"" fi ;; "search "*) - search="$stripped_line" + search="$(strip_trailing_dots ${line#* })" ;; *) [ -n "$line" ] && continue - if [ -n "$ns" ] && [ -n "$search" ]; then + if [ -n "$ns" -a -n "$search" ]; then newns= for n in $ns; do newns="$newns${newns:+,}$n" @@ -243,7 +236,7 @@ parse_resolv() uniqify() { - result= + local result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -256,8 +249,8 @@ uniqify() dirname() { - OIFS="$IFS" - IFS=/ + local dir= OIFS="$IFS" + local IFS=/ set -- $@ IFS="$OIFS" if [ -n "$1" ]; then @@ -274,7 +267,7 @@ dirname() config_mkdirs() { - e=0 + local e=0 f d for f; do [ -n "$f" ] || continue d="$(dirname "$f")" @@ -302,86 +295,66 @@ detect_init() # Detect the running init system. # As systemd and OpenRC can be installed on top of legacy init # systems we try to detect them first. - status="@STATUSARG@" + local status="@STATUSARG@" : ${status:=status} - if [ -x /bin/systemctl ] && [ -S /run/systemd/private ]; then - RESTARTCMD=' - if /bin/systemctl --quiet is-active $1.service - then - /bin/systemctl restart $1.service - fi' - elif [ -x /usr/bin/systemctl ] && [ -S /run/systemd/private ]; then - RESTARTCMD=' - if /usr/bin/systemctl --quiet is-active $1.service - then - /usr/bin/systemctl restart $1.service - fi' - elif [ -x /sbin/rc-service ] && - { [ -s /libexec/rc/init.d/softlevel ] || - [ -s /run/openrc/softlevel ]; } + if [ -x /bin/systemctl -a -S /run/systemd/private ]; then + RESTARTCMD="if /bin/systemctl --quiet is-active \$1.service; then + /bin/systemctl restart \$1.service; +fi" + elif [ -x /usr/bin/systemctl -a -S /run/systemd/private ]; then + RESTARTCMD="if /usr/bin/systemctl --quiet is-active \$1.service; then + /usr/bin/systemctl restart \$1.service; +fi" + elif [ -x /sbin/rc-service -a \ + -s /libexec/rc/init.d/softlevel -o -s /run/openrc/softlevel ] then - RESTARTCMD='/sbin/rc-service -i $1 -- -Ds restart' + RESTARTCMD="/sbin/rc-service -i \$1 -- -Ds restart" elif [ -x /usr/sbin/invoke-rc.d ]; then RCDIR=/etc/init.d - RESTARTCMD=' - if /usr/sbin/invoke-rc.d --quiet $1 status >/dev/null 2>&1 - then - /usr/sbin/invoke-rc.d $1 restart - fi' + RESTARTCMD="if /usr/sbin/invoke-rc.d --quiet \$1 status 1>/dev/null 2>&1; then + /usr/sbin/invoke-rc.d \$1 restart; +fi" elif [ -x /sbin/service ]; then # Old RedHat RCDIR=/etc/init.d - RESTARTCMD=' - if /sbin/service $1; then - /sbin/service $1 restart - fi' + RESTARTCMD="if /sbin/service \$1; then + /sbin/service \$1 restart; +fi" elif [ -x /usr/sbin/service ]; then # Could be FreeBSD - RESTARTCMD=" - if /usr/sbin/service \$1 $status >/dev/null 2>&1 - then - /usr/sbin/service \$1 restart - fi" + RESTARTCMD="if /usr/sbin/service \$1 $status 1>/dev/null 2>&1; then + /usr/sbin/service \$1 restart; +fi" elif [ -x /bin/sv ]; then - RESTARTCMD='/bin/sv status $1 >/dev/null 2>&1 && - /bin/sv try-restart $1' + RESTARTCMD="/bin/sv status \$1 >/dev/null 2>&1 && /bin/sv try-restart \$1" elif [ -x /usr/bin/sv ]; then - RESTARTCMD='/usr/bin/sv status $1 >/dev/null 2>&1 && - /usr/bin/sv try-restart $1' - elif [ -e /etc/arch-release ] && [ -d /etc/rc.d ]; then + RESTARTCMD="/usr/bin/sv status \$1 >/dev/null 2>&1 && /usr/bin/sv try-restart \$1" + elif [ -e /etc/arch-release -a -d /etc/rc.d ]; then RCDIR=/etc/rc.d - RESTARTCMD=' - if [ -e /var/run/daemons/$1 ] - then - /etc/rc.d/$1 restart - fi' - elif [ -e /etc/slackware-version ] && [ -d /etc/rc.d ]; then - RESTARTCMD=' - if /etc/rc.d/rc.$1 status >/dev/null 2>&1 - then - /etc/rc.d/rc.$1 restart - fi' - elif [ -e /etc/rc.d/rc.subr ] && [ -d /etc/rc.d ]; then + RESTARTCMD="if [ -e /var/run/daemons/\$1 ]; then + /etc/rc.d/\$1 restart; +fi" + elif [ -e /etc/slackware-version -a -d /etc/rc.d ]; then + RESTARTCMD="if /etc/rc.d/rc.\$1 status 1>/dev/null 2>&1; then + /etc/rc.d/rc.\$1 restart; +fi" + elif [ -e /etc/rc.d/rc.subr -a -d /etc/rc.d ]; then # OpenBSD - RESTARTCMD=' - if /etc/rc.d/$1 check >/dev/null 2>&1 - then - /etc/rc.d/$1 restart - fi' + RESTARTCMD="if /etc/rc.d/\$1 check 1>/dev/null 2>&1; then + /etc/rc.d/\$1 restart; +fi" else for x in /etc/init.d/rc.d /etc/rc.d /etc/init.d; do [ -d $x ] || continue - RESTARTCMD=" - if $x/\$1 $status >/dev/null 2>&1 - then - $x/\$1 restart - fi" + RESTARTCMD="if $x/\$1 $status 1>/dev/null 2>&1; then + $x/\$1 restart; +fi" break done fi if [ -z "$RESTARTCMD" ]; then - if [ "$_NOINIT_WARNED" != true ]; then + if [ "$NOINIT_WARNED" != true ]; then warn "could not detect a useable init system" _NOINIT_WARNED=true fi @@ -393,9 +366,9 @@ detect_init() echo_resolv() { - OIFS="$IFS" + local line= OIFS="$IFS" - [ -n "$1" ] && [ -f "$IFACEDIR/$1" ] || return 1 + [ -n "$1" -a -f "$IFACEDIR/$1" ] || return 1 echo "# resolv.conf from $1" # Our variable maker works of the fact each resolv.conf per interface # is separated by blank lines. @@ -415,16 +388,11 @@ list_resolv() { [ -d "$IFACEDIR" ] || return 0 - cmd="$1" + local report=false list= retval=0 cmd="$1" excl= shift - excl=false - list= - report=false - retval=0 case "$IF_EXCLUSIVE" in *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Nov 20 22:23:53 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E9AB1C3535; Wed, 20 Nov 2019 22:23:53 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JHKP1fFNz4R7S; Wed, 20 Nov 2019 22:23:53 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1BAEC17F3; Wed, 20 Nov 2019 22:23:53 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAKMNqbF091518; Wed, 20 Nov 2019 22:23:52 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAKMNo3Y091505; Wed, 20 Nov 2019 22:23:50 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201911202223.xAKMNo3Y091505@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 20 Nov 2019 22:23:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354919 - vendor/openresolv/dist X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: vendor/openresolv/dist X-SVN-Commit-Revision: 354919 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Nov 2019 22:23:53 -0000 Author: pfg Date: Wed Nov 20 22:23:50 2019 New Revision: 354919 URL: https://svnweb.freebsd.org/changeset/base/354919 Log: Import openresolv 3.9.2 More information at: https://roy.marples.name/projects/openresolv Added: vendor/openresolv/dist/LICENSE - copied unchanged from r354917, vendor/openresolv/dist/LICENSE vendor/openresolv/dist/README.md - copied unchanged from r354917, vendor/openresolv/dist/README.md Deleted: vendor/openresolv/dist/README Modified: vendor/openresolv/dist/Makefile vendor/openresolv/dist/configure vendor/openresolv/dist/dnsmasq.in vendor/openresolv/dist/libc.in vendor/openresolv/dist/named.in vendor/openresolv/dist/pdns_recursor.in vendor/openresolv/dist/pdnsd.in vendor/openresolv/dist/resolvconf.conf vendor/openresolv/dist/resolvconf.conf.5.in vendor/openresolv/dist/resolvconf.in vendor/openresolv/dist/unbound.in Copied: vendor/openresolv/dist/LICENSE (from r354917, vendor/openresolv/dist/LICENSE) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/dist/LICENSE Wed Nov 20 22:23:50 2019 (r354919, copy of r354917, vendor/openresolv/dist/LICENSE) @@ -0,0 +1,23 @@ +Copyright (c) 2007-2019 Roy Marples +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. Modified: vendor/openresolv/dist/Makefile ============================================================================== --- vendor/openresolv/dist/Makefile Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/Makefile Wed Nov 20 22:23:50 2019 (r354919) @@ -10,6 +10,7 @@ SYSCONFDIR?= /etc LIBEXECDIR?= /libexec/resolvconf VARDIR?= /var/run/resolvconf +ECHO?= echo INSTALL?= install SED?= sed @@ -20,7 +21,7 @@ DOCMODE?= 0644 MANMODE?= 0444 RESOLVCONF= resolvconf resolvconf.8 resolvconf.conf.5 -SUBSCRIBERS= libc dnsmasq named pdnsd unbound +SUBSCRIBERS= libc dnsmasq named pdnsd pdns_recursor unbound TARGET= ${RESOLVCONF} ${SUBSCRIBERS} SRCS= ${TARGET:C,$,.in,} # pmake SRCS:= ${TARGET:=.in} # gmake @@ -42,7 +43,7 @@ DISTINFOSIGN= ${DISTINFO}.asc CKSUM?= cksum -a SHA256 PGP?= netpgp -FOSSILID?= current +GITREF?= HEAD .SUFFIXES: .in @@ -79,15 +80,17 @@ maninstall: install: proginstall maninstall -import: +dist-git: + git archive --prefix=${DISTPREFIX}/ ${GITREF} | xz >${DISTFILE} + +dist-inst: + mkdir /tmp/${DISTPREFIX} + cp -RPp * /tmp/${DISTPREFIX} + (cd /tmp/${DISTPREFIX}; make clean) + tar -cvjpf ${DISTFILE} -C /tmp ${DISTPREFIX} rm -rf /tmp/${DISTPREFIX} - ${INSTALL} -d /tmp/${DISTPREFIX} - cp README ${SRCS} /tmp/${DISTPREFIX} -dist: - fossil tarball --name ${DISTPREFIX} ${FOSSILID} ${DISTFILEGZ} - gunzip -c ${DISTFILEGZ} | xz >${DISTFILE} - rm ${DISTFILEGZ} +dist: dist-git distinfo: dist rm -f ${DISTINFO} ${DISTINFOSIGN} @@ -96,3 +99,20 @@ distinfo: dist ${PGP} --clearsign --output=${DISTINFOSIGN} ${DISTINFO} chmod 644 ${DISTINFOSIGN} ls -l ${DISTFILE} ${DISTINFO} ${DISTINFOSIGN} + +import: dist + rm -rf /tmp/${DISTPREFIX} + ${INSTALL} -d /tmp/${DISTPREFIX} + tar xvJpf ${DISTFILE} -C /tmp + +_import-src: + rm -rf ${DESTDIR}/* + ${INSTALL} -d ${DESTDIR} + cp LICENSE README.md ${SRCS} resolvconf.conf ${DESTDIR}; + cp resolvconf.8.in resolvconf.conf.5.in ${DESTDIR}; + @${ECHO} + @${ECHO} "=============================================================" + @${ECHO} "openresolv-${VERSION} imported to ${DESTDIR}" + +import-src: + ${MAKE} _import-src DESTDIR=`if [ -n "${DESTDIR}" ]; then echo "${DESTDIR}"; else echo /tmp/${DISTPREFIX}; fi` Copied: vendor/openresolv/dist/README.md (from r354917, vendor/openresolv/dist/README.md) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/dist/README.md Wed Nov 20 22:23:50 2019 (r354919, copy of r354917, vendor/openresolv/dist/README.md) @@ -0,0 +1,64 @@ +# openresolv + +openresolv is a [resolvconf](https://en.wikipedia.org/wiki/Resolvconf) +implementation which manages `/etc/resolv.conf`. + +`/etc/resolv.conf` is a file that holds the configuration for the local +resolution of domain names. +Normally this file is either static or maintained by a local daemon, +normally a DHCP daemon. But what happens if more than one thing wants to +control the file? +Say you have wired and wireless interfaces to different subnets and run a VPN +or two on top of that, how do you say which one controls the file? +It's also not as easy as just adding and removing the nameservers each client +knows about as different clients could add the same nameservers. + +Enter resolvconf, the middleman between the network configuration services and +`/etc/resolv.conf`. +resolvconf itself is just a script that stores, removes and lists a full +`resolv.conf` generated for the interface. It then calls all the helper scripts +it knows about so it can configure the real `/etc/resolv.conf` and optionally +any local nameservers other than libc. + +## Reasons for using openresolv + +Why openresolv over the +[Debian implementation](http://qref.sourceforge.net/Debian/reference/ch-gateway.en.html#s-dns-resolvconf)? +Here's some reasons: + * Works with + [POSIX shell and userland](http://www.opengroup.org/onlinepubs/009695399) + * Does not need awk, grep or sed which means we can work without `/usr` + mounted + * Works with other init systems than Debians' out of the box + * Available as a 2 clause + [BSD license](http://www.freebsd.org/copyright/freebsd-license.html) + * Prefer configs via IF_METRIC for dynamic ordering + * Configures zones for local resolvers other than libc + +The last point is quite important, especially when running VPN systems. +Take the following resolv.conf files which have been generated by a +[DHCP client](../dhcpcd) and sent to resolvconf: + +``` +# resolv.conf from bge0 +search foo.com +nameserver 1.2.3.4 + +# resolv.conf from tap0 +domain bar.org +nameserver 5.6.7.8 +``` + +In this instance, queries for foo.com will go to 1.2.3.4 and queries for +bar.org will go to 5.6.7.8. +This does require the resolvers to be configured to pickup the resolvconf +generated configuration for them though. +openresolv ships with helpers for: + * [unbound](http://www.unbound.net/) + * [dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) + * [ISC BIND](http://www.isc.org/software/bind) + * [PowerDNS Recursor](http://wiki.powerdns.com/trac) + +See the +[configuration section](https://roy.marples.name/projects/openresolv/config) +for more details. Modified: vendor/openresolv/dist/configure ============================================================================== --- vendor/openresolv/dist/configure Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/configure Wed Nov 20 22:23:50 2019 (r354919) @@ -44,42 +44,8 @@ for x do esac done -if [ -z "$LIBEXECDIR" ]; then - printf "Checking for directory /libexec ... " - if [ -d /libexec ]; then - echo "yes" - LIBEXECDIR=$PREFIX/libexec/resolvconf - else - echo "no" - LIBEXECDIR=$PREFIX/lib/resolvconf - fi -fi -if [ -z "$RUNDIR" ]; then - printf "Checking for directory /run ... " - if [ -d /run ]; then - echo "yes" - RUNDIR=/run - else - echo "no" - RUNDIR=/var/run - fi -fi - : ${SED:=sed} -: ${SYSCONFDIR:=$PREFIX/etc} -: ${SBINDIR:=$PREFIX/sbin} -: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} -: ${STATEDIR:=/var} -: ${RUNDIR:=$STATEDIR/run} -: ${MANDIR:=${PREFIX:-/usr}/share/man} - -eval SYSCONFDIR="$SYSCONFDIR" -eval SBINDIR="$SBINDIR" -eval LIBEXECDIR="$LIBEXECDIR" -eval VARDIR="$RUNDIR/resolvconf" -eval MANDIR="$MANDIR" - CONFIG_MK=config.mk if [ -z "$BUILD" ]; then @@ -121,7 +87,19 @@ rm -rf $CONFIG_MK echo "# $OS" >$CONFIG_MK case "$OS" in -freebsd*) +dragonfly*) + # This means /usr HAS to be mounted not via dhcpcd + : ${LIBEXECDIR:=${PREFIX:-/usr}/libexec/resolvconf} + ;; +linux*) + # cksum does't support -a and netpgp is rare + echo "CKSUM= sha256sum --tag" >>$CONFIG_MK + echo "PGP= gpg2" >>$CONFIG_MK + ;; +esac + +case "$OS" in +dragonfly*|freebsd*) # On FreeBSD, /etc/init.d/foo status returns 0 if foo is not enabled # regardless of if it's not running. # So we force onestatus to work around this silly bug. @@ -129,12 +107,42 @@ freebsd*) STATUSARG="onestatus" fi ;; -linux*) - # cksum does't support -a and netpgp is rare - echo "CKSUM= sha256sum --tag" >>$CONFIG_MK - echo "PGP= gpg2" >>$CONFIG_MK - ;; esac + + +if [ -z "$LIBEXECDIR" ]; then + printf "Checking for directory /libexec ... " + if [ -d /libexec ]; then + echo "yes" + LIBEXECDIR=$PREFIX/libexec/resolvconf + else + echo "no" + LIBEXECDIR=$PREFIX/lib/resolvconf + fi +fi +if [ -z "$RUNDIR" ]; then + printf "Checking for directory /run ... " + if [ -d /run ]; then + echo "yes" + RUNDIR=/run + else + echo "no" + RUNDIR=/var/run + fi +fi + +: ${SYSCONFDIR:=$PREFIX/etc} +: ${SBINDIR:=$PREFIX/sbin} +: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} +: ${STATEDIR:=/var} +: ${RUNDIR:=$STATEDIR/run} +: ${MANDIR:=${PREFIX:-/usr}/share/man} + +eval SYSCONFDIR="$SYSCONFDIR" +eval SBINDIR="$SBINDIR" +eval LIBEXECDIR="$LIBEXECDIR" +eval VARDIR="$RUNDIR/resolvconf" +eval MANDIR="$MANDIR" for x in SYSCONFDIR SBINDIR LIBEXECDIR VARDIR MANDIR RESTARTCMD RCDIR STATUSARG do Modified: vendor/openresolv/dist/dnsmasq.in ============================================================================== --- vendor/openresolv/dist/dnsmasq.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/dnsmasq.in Wed Nov 20 22:23:50 2019 (r354919) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # dnsmasq subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$dnsmasq_conf" -a -z "$dnsmasq_resolv" ] && exit 0 +[ -z "${dnsmasq_conf}${dnsmasq_resolv}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -98,7 +98,7 @@ for d in $DOMAINS; do empty=false i=0 IFS=: set -- $n - while [ -n "$1" -o -n "$2" ]; do + while [ -n "$1" ] || [ -n "$2" ]; do addr="$1" shift if [ -z "$addr" ]; then @@ -184,7 +184,7 @@ if $changed; then eval $dnsmasq_restart elif [ -n "$RESTARTCMD" ]; then set -- ${dnsmasq_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${dnsmasq_service} fi @@ -206,4 +206,6 @@ if $dbus; then dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ /uk/org/thekelleys/dnsmasq uk.org.thekelleys.$method \ $dbusdest + dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ + /uk/org/thekelleys/dnsmasq uk.org.thekelleys.ClearCache fi Modified: vendor/openresolv/dist/libc.in ============================================================================== --- vendor/openresolv/dist/libc.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/libc.in Wed Nov 20 22:23:50 2019 (r354919) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # libc subscriber for resolvconf @@ -36,9 +36,9 @@ NL=" # sed may not be available, and this is faster on small files key_get_value() { - local key="$1" x= line= - + key="$1" shift + if [ $# -eq 0 ]; then while read -r line; do case "$line" in @@ -58,8 +58,6 @@ key_get_value() keys_remove() { - local key x line found - while read -r line; do found=false for key do @@ -79,7 +77,7 @@ local_nameservers="127.* 0.0.0.0 255.255.255.255 ::1" if [ -f "$SYSCONFDIR"/resolvconf.conf ]; then . "$SYSCONFDIR"/resolvconf.conf elif [ -d "$SYSCONFDIR"/resolvconf ]; then - SYSCONFDIR="$SYSCONFDIR/resolvconf/resolv.conf.d" + SYSCONFDIR="$SYSCONFDIR/resolvconf" base="$SYSCONFDIR/resolv.conf.d/base" if [ -f "$base" ]; then prepend_nameservers="$(key_get_value "nameserver " "$base")" @@ -98,10 +96,12 @@ fi : ${resolv_conf:=/etc/resolv.conf} : ${libc_service:=nscd} : ${list_resolv:=@SBINDIR@/resolvconf -l} -if [ "${resolv_conf_head-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.head ]; then +if [ "${resolv_conf_head-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.head ] +then resolv_conf_head="$(cat "${SYSCONFDIR}"/resolv.conf.head)" fi -if [ "${resolv_conf_tail-x}" = x -a -f "$SYSCONFDIR"/resolv.conf.tail ]; then +if [ "${resolv_conf_tail-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.tail ] +then resolv_conf_tail="$(cat "$SYSCONFDIR"/resolv.conf.tail)" fi @@ -110,7 +110,7 @@ signature="# Generated by resolvconf" uniqify() { - local result= + result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -126,7 +126,7 @@ case "${resolv_conf_passthrough:-NO}" in backup=false newest= for conf in "$IFACEDIR"/*; do - if [ -z "$newest" -o "$conf" -nt "$newest" ]; then + if [ -z "$newest" ] || [ "$conf" -nt "$newest" ]; then newest="$conf" fi done @@ -178,7 +178,7 @@ case "${resolv_conf_passthrough:-NO}" in fi [ -n "$domain" ] && newconf="${newconf}domain $domain$NL" - if [ -n "$newsearch" -a "$newsearch" != "$domain" ]; then + if [ -n "$newsearch" ] && [ "$newsearch" != "$domain" ]; then newconf="${newconf}search $newsearch$NL" fi for n in $newns; do @@ -232,7 +232,7 @@ if [ -n "$libc_restart" ]; then eval $libc_restart elif [ -n "$RESTARTCMD" ]; then set -- ${libc_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${libc_service} fi Modified: vendor/openresolv/dist/named.in ============================================================================== --- vendor/openresolv/dist/named.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/named.in Wed Nov 20 22:23:50 2019 (r354919) @@ -28,14 +28,14 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$named_zones" -a -z "$named_options" ] && exit 0 +[ -z "${named_zones}${named_options}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " # Platform specific kludges -if [ -z "$named_service" -a -z "$named_restart" -a \ - -d "$RCDIR" -a ! -x "$RCDIR"/named ] +if [ -z "${named_service}${named_restart}" ] && + [ -d "$RCDIR" ] && ! [ -x "$RCDIR"/named ] then if [ -x "$RCDIR"/bind9 ]; then # Debian and derivatives @@ -111,7 +111,7 @@ if $changed; then eval $named_restart elif [ -n "$RESTARTCMD" ]; then set -- ${named_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${named_service} fi Modified: vendor/openresolv/dist/pdns_recursor.in ============================================================================== --- vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/pdns_recursor.in Wed Nov 20 22:23:50 2019 (r354919) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2009-2011 Roy Marples +# Copyright (c) 2009-2019 Roy Marples # All rights reserved # PowerDNS Recursor subscriber for resolvconf @@ -33,17 +33,14 @@ NL=" " -: ${pdns_service:=pdns_recursor} +: ${pdns_service:=pdns-recursor} newzones= -# pds_recursor does not present support global forward servers, which -# does limit it's usefulness somewhat. -# If it did, the below code can be enabled, or something like it. -#for n in $NAMESERVERS; do -# newzones="$newzones${newzones:+,}$n" -#done -#[ -n "$newzones" ] && newzones=".=$newzones$NL" +for n in $NAMESERVERS; do + newzones="$newzones${newzones:+,}$n" +done +[ -n "$newzones" ] && newzones="+.=$newzones$NL" for d in $DOMAINS; do newns= @@ -71,7 +68,7 @@ then eval $pdns_restart elif [ -n "$RESTARTCMD" ]; then set -- ${pdns_service} - eval $RESTARTCMD + eval "$RESTARTCMD" else @SBINDIR@/resolvconf -r ${pdns_service} fi Modified: vendor/openresolv/dist/pdnsd.in ============================================================================== --- vendor/openresolv/dist/pdnsd.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/pdnsd.in Wed Nov 20 22:23:50 2019 (r354919) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2010-2013 Roy Marples +# Copyright (c) 2010-2018 Roy Marples # All rights reserved # pdnsd subscriber for resolvconf @@ -28,7 +28,7 @@ [ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 . "@SYSCONFDIR@/resolvconf.conf" || exit 1 -[ -z "$pdnsd_conf" -a -z "$pdnsd_resolv" ] && exit 0 +[ -z "${pdnsd_conf}${pdnsd_resolv}" ] && exit 0 [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" NL=" " @@ -41,14 +41,16 @@ signature_end="# End of resolvconf" # but sed may not always be available at the time. remove_markers() { - local m1="$1" m2="$2" x= line= in_marker=0 + m1="$1" + m2="$2" + in_marker=0 shift; shift if type sed >/dev/null 2>&1; then sed "/^$m1/,/^$m2/d" $@ else - for x; do - while read -r line; do + for x do + while read line; do case "$line" in "$m1"*) in_marker=1;; "$m2"*) in_marker=0;; Modified: vendor/openresolv/dist/resolvconf.conf ============================================================================== --- vendor/openresolv/dist/resolvconf.conf Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/resolvconf.conf Wed Nov 20 22:23:50 2019 (r354919) @@ -4,4 +4,4 @@ resolv_conf=/etc/resolv.conf # If you run a local name server, you should uncomment the below line and # configure your subscribers configuration files below. -#name_servers=127.0.0.1 \ No newline at end of file +#name_servers=127.0.0.1 Modified: vendor/openresolv/dist/resolvconf.conf.5.in ============================================================================== --- vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/resolvconf.conf.5.in Wed Nov 20 22:23:50 2019 (r354919) @@ -22,7 +22,7 @@ .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" -.Dd December 29, 2016 +.Dd September 8, 2019 .Dt RESOLVCONF.CONF 5 .Os .Sh NAME @@ -64,19 +64,25 @@ Defaults to YES. .It Sy interface_order These interfaces will always be processed first. If unset, defaults to the following:- -.D1 lo lo[0-9]* +.Bd -compact -literal -offset indent +lo lo[0-9]* +.Ed .It Sy dynamic_order These interfaces will be processed next, unless they have a metric. If unset, defaults to the following:- -.D1 tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* +.Bd -compact -literal -offset indent +tap[0-9]* tun[0-9]* vpn vpn[0-9]* ppp[0-9]* ippp[0-9]* +.Ed .It Sy inclusive_interfaces -Ignore any exlcusive marking for these interfaces. +Ignore any exclusive marking for these interfaces. This is handy when 3rd party integrations force the .Nm resolvconf -x option and you want to disable it easily. .It Sy local_nameservers If unset, defaults to the following:- -.D1 127.* 0.0.0.0 255.255.255.255 ::1 +.Bd -compact -literal -offset indent +127.* 0.0.0.0 255.255.255.255 ::1 +.Ed .It Sy search_domains Prepend search domains to the dynamically generated list. .It Sy search_domains_append @@ -118,16 +124,24 @@ The syntax is this: .Va $keyword Ns / Ns Va $match Ns / Ns Va $replacement .Pp Example, given this resolv.conf: -.D1 domain foo.org -.D1 search foo.org dead.beef -.D1 nameserver 1.2.3.4 -.D1 nameserver 2.3.4.5 +.Bd -compact -literal -offset indent +domain foo.org +search foo.org dead.beef +nameserver 1.2.3.4 +nameserver 2.3.4.5 +.Ed and this configuaration: -.D1 replace="search/foo*/bar.com nameserver/1.2.3.4/5.6.7.8 nameserver/2.3.4.5/" +.Bd -compact -literal -offset indent +replace="search/foo*/bar.com" +replace="$replace nameserver/1.2.3.4/5.6.7.8" +replace="$replace nameserver/2.3.4.5/" +.Ed you would get this resolv.conf instead: -.D1 domain foo.org -.D1 search bar.com -.D1 nameserver 5.6.7.8 +.Bd -compact -literal -offset indent +domain foo.org +search bar.com +nameserver 5.6.7.8 +.Ed .It Sy replace_sub Works the same way as .Sy replace @@ -138,9 +152,11 @@ Using the same example resolv.conf and changing to .Sy replace_sub , you would get this resolv.conf instead: -.D1 domain foo.org -.D1 search bar.com dead.beef -.D1 nameserver 5.6.7.8 +.Bd -compact -literal -offset indent +domain foo.org +search bar.com dead.beef +nameserver 5.6.7.8 +.Ed .It Sy state_dir Override the default state directory of .Pa @VARDIR@ . @@ -195,7 +211,8 @@ Prepend search domains to the dynamically generated li openresolv ships with subscribers for the name servers .Xr dnsmasq 8 , .Xr named 8 , -.Xr pdnsd 8 +.Xr pdnsd 8 , +.Xr pdns_recursor 8 , and .Xr unbound 8 . Each subscriber can create configuration files which should be included in @@ -203,7 +220,9 @@ in the subscribers main configuration file. .Pp To disable a subscriber, simply set it's name to NO. For example, to disable the libc subscriber you would set: -.D1 libc=NO +.Bd -compact -literal -offset indent +libc=NO +.Ed .Bl -tag -width indent .It Sy dnsmasq_conf This file tells dnsmasq which name servers to use for specific domains. @@ -211,17 +230,21 @@ This file tells dnsmasq which name servers to use for This file tells dnsmasq which name servers to use for global lookups. .Pp Example resolvconf.conf for dnsmasq: -.D1 name_servers=127.0.0.1 -.D1 dnsmasq_conf=/etc/dnsmasq-conf.conf -.D1 dnsmasq_resolv=/etc/dnsmasq-resolv.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +dnsmasq_conf=/etc/dnsmasq-conf.conf +dnsmasq_resolv=/etc/dnsmasq-resolv.conf +.Ed .Pp Example dnsmasq.conf: -.D1 listen-address=127.0.0.1 -.D1 # If dnsmasq is compiled for DBus then we can take -.D1 # advantage of not having to restart dnsmasq. -.D1 enable-dbus -.D1 conf-file=/etc/dnsmasq-conf.conf -.D1 resolv-file=/etc/dnsmasq-resolv.conf +.Bd -compact -literal -offset indent +listen-address=127.0.0.1 +# If dnsmasq is compiled for DBus then we can take +# advantage of not having to restart dnsmasq. +enable-dbus +conf-file=/etc/dnsmasq-conf.conf +resolv-file=/etc/dnsmasq-resolv.conf +.Ed .It Sy named_options Include this file in the named options block. This file tells named which name servers to use for global lookups. @@ -230,16 +253,21 @@ Include this file in the named global scope, after the This file tells named which name servers to use for specific domains. .Pp Example resolvconf.conf for named: -.D1 name_servers=127.0.0.1 -.D1 named_options=/etc/named-options.conf -.D1 named_zones=/etc/named-zones.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +named_options=/etc/named-options.conf +named_zones=/etc/named-zones.conf +.Ed .Pp Example named.conf: -.D1 options { -.D1 listen-on { 127.0.0.1; }; -.D1 include "/etc/named-options.conf"; -.D1 }; -.D1 include "/etc/named-zones.conf"; +.Bd -compact -literal -offset indent +options { + listen-on { 127.0.0.1; }; + include "/etc/named-options.conf"; +}; + +include "/etc/named-zones.conf"; +.Ed .It Sy pdnsd_conf This is the main pdnsd configuration file which we modify to add our forward domains to. @@ -253,32 +281,54 @@ If this variable is not set then it's written to .Pa pdnsd_conf . .Pp Example resolvconf.conf for pdnsd: -.D1 name_servers=127.0.0.1 -.D1 pdnsd_conf=/etc/pdnsd.conf -.D1 # pdnsd_resolv=/etc/pdnsd-resolv.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +pdnsd_conf=/etc/pdnsd.conf +# pdnsd_resolv=/etc/pdnsd-resolv.conf +.Ed .Pp Example pdnsd.conf: -.D1 global { -.D1 server_ip = 127.0.0.1; -.D1 status_ctl = on; -.D1 } -.D1 server { -.D1 # A server definition is required, even if emtpy. -.D1 label="empty"; -.D1 proxy_only=on; -.D1 # file="/etc/pdnsd-resolv.conf"; -.D1 } +.Bd -compact -literal -offset indent +global { + server_ip = 127.0.0.1; + status_ctl = on; +} +server { + # A server definition is required, even if empty. + label="empty"; + proxy_only=on; + # file="/etc/pdnsd-resolv.conf"; +} +.Ed +.It Sy pdns_zones +This file tells pdns_recursor about specific and global name servers. +.Pp +Example resolvconf.conf for pdns_recursor: +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +pdns_zones=/etc/pdns/recursor-zones.conf +.Ed +.Pp +Example recursor.conf: +.Bd -compact -literal -offset indent +allow-from=127.0.0.0/8, ::1/128 +forward-zones-file=/etc/pdns/recursor-zones.conf +.Ed .It Sy unbound_conf This file tells unbound about specific and global name servers. .It Sy unbound_insecure When set to YES, unbound marks the domains as insecure, thus ignoring DNSSEC. .Pp Example resolvconf.conf for unbound: -.D1 name_servers=127.0.0.1 -.D1 unbound_conf=/etc/unbound-resolvconf.conf +.Bd -compact -literal -offset indent +name_servers=127.0.0.1 +unbound_conf=/etc/unbound-resolvconf.conf +.Ed .Pp Example unbound.conf: -.D1 include: /etc/unbound-resolvconf.conf +.Bd -compact -literal -offset indent +include: /etc/unbound-resolvconf.conf +.Ed .El .Sh SUBSCRIBER INTEGRATION Not all distributions store the files the subscribers need in the same @@ -292,7 +342,6 @@ Also, users could equally want to use a different vers installed by default, such as bind8 and bind9. To accommodate this, the subscribers have these files in configurable variables, documented below. -.Pp .Bl -tag -width indent .It Sy dnsmasq_service Name of the dnsmasq service. @@ -310,6 +359,10 @@ Name of the named service. Command to restart the named service. .It Sy pdnsd_restart Command to restart the pdnsd service. +.It Sy pdns_service +Command to restart the pdns_recursor service. +.It Sy pdns_restart +Command to restart the pdns_recursor service. .It Sy unbound_service Name of the unbound service. .It Sy unbound_restart Modified: vendor/openresolv/dist/resolvconf.in ============================================================================== --- vendor/openresolv/dist/resolvconf.in Wed Nov 20 22:20:11 2019 (r354918) +++ vendor/openresolv/dist/resolvconf.in Wed Nov 20 22:23:50 2019 (r354919) @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2007-2016 Roy Marples +# Copyright (c) 2007-2019 Roy Marples # All rights reserved # Redistribution and use in source and binary forms, with or without @@ -25,7 +25,7 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. RESOLVCONF="$0" -OPENRESOLV_VERSION="3.9.0" +OPENRESOLV_VERSION="3.9.2" SYSCONFDIR=@SYSCONFDIR@ LIBEXECDIR=@LIBEXECDIR@ VARDIR=@VARDIR@ @@ -125,21 +125,22 @@ usage() # If you think otherwise, capture a DNS trace and you'll see libc # will strip it regardless. # This also solves setting up duplicate zones in our subscribers. -strip_trailing_dots() +# Also strip any comments denoted by #. +resolv_strip() { - local n= d= - - for n; do - printf "$d%s" "${n%.}" - d=" " + space= + for word; do + case "$word" in + \#*) break;; + esac + printf "%s%s" "$space${word%.}" + space=" " done printf "\n" } private_iface() { - local p - # Allow expansion cd "$IFACEDIR" @@ -168,12 +169,15 @@ private_iface() # for domain name servers, search name servers and global nameservers parse_resolv() { - local line= ns= ds= search= d= n= newns= - local new=true iface= private=false p= domain= l= islocal= - + domain= + new=true newns= + ns= + private=false + search= while read -r line; do + stripped_line="$(resolv_strip ${line#* })" case "$line" in "# resolv.conf from "*) if ${new}; then @@ -189,29 +193,32 @@ parse_resolv() "nameserver "*) islocal=false for l in $local_nameservers; do - case "${line#* }" in + case "$stripped_line" in $l) islocal=true - echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS ${line#* }\"" break ;; esac done - $islocal || ns="$ns${line#* } " + if $islocal; then + echo "LOCALNAMESERVERS=\"\$LOCALNAMESERVERS $stripped_line\"" + else + ns="$ns$stripped_line " + fi ;; "domain "*) - search="$(strip_trailing_dots ${line#* })" + search="$stripped_line" if [ -z "$domain" ]; then domain="$search" echo "DOMAIN=\"$domain\"" fi ;; "search "*) - search="$(strip_trailing_dots ${line#* })" + search="$stripped_line" ;; *) [ -n "$line" ] && continue - if [ -n "$ns" -a -n "$search" ]; then + if [ -n "$ns" ] && [ -n "$search" ]; then newns= for n in $ns; do newns="$newns${newns:+,}$n" @@ -236,7 +243,7 @@ parse_resolv() uniqify() { - local result= + result= while [ -n "$1" ]; do case " $result " in *" $1 "*);; @@ -249,8 +256,8 @@ uniqify() dirname() { - local dir= OIFS="$IFS" - local IFS=/ + OIFS="$IFS" + IFS=/ set -- $@ IFS="$OIFS" if [ -n "$1" ]; then @@ -267,7 +274,7 @@ dirname() config_mkdirs() { - local e=0 f d + e=0 for f; do [ -n "$f" ] || continue d="$(dirname "$f")" @@ -295,66 +302,86 @@ detect_init() # Detect the running init system. # As systemd and OpenRC can be installed on top of legacy init # systems we try to detect them first. - local status="@STATUSARG@" + status="@STATUSARG@" : ${status:=status} - if [ -x /bin/systemctl -a -S /run/systemd/private ]; then - RESTARTCMD="if /bin/systemctl --quiet is-active \$1.service; then - /bin/systemctl restart \$1.service; -fi" - elif [ -x /usr/bin/systemctl -a -S /run/systemd/private ]; then - RESTARTCMD="if /usr/bin/systemctl --quiet is-active \$1.service; then - /usr/bin/systemctl restart \$1.service; -fi" - elif [ -x /sbin/rc-service -a \ - -s /libexec/rc/init.d/softlevel -o -s /run/openrc/softlevel ] + if [ -x /bin/systemctl ] && [ -S /run/systemd/private ]; then + RESTARTCMD=' + if /bin/systemctl --quiet is-active $1.service + then + /bin/systemctl restart $1.service + fi' + elif [ -x /usr/bin/systemctl ] && [ -S /run/systemd/private ]; then + RESTARTCMD=' + if /usr/bin/systemctl --quiet is-active $1.service + then + /usr/bin/systemctl restart $1.service + fi' + elif [ -x /sbin/rc-service ] && + { [ -s /libexec/rc/init.d/softlevel ] || + [ -s /run/openrc/softlevel ]; } then - RESTARTCMD="/sbin/rc-service -i \$1 -- -Ds restart" + RESTARTCMD='/sbin/rc-service -i $1 -- -Ds restart' elif [ -x /usr/sbin/invoke-rc.d ]; then RCDIR=/etc/init.d - RESTARTCMD="if /usr/sbin/invoke-rc.d --quiet \$1 status 1>/dev/null 2>&1; then - /usr/sbin/invoke-rc.d \$1 restart; -fi" + RESTARTCMD=' + if /usr/sbin/invoke-rc.d --quiet $1 status >/dev/null 2>&1 + then + /usr/sbin/invoke-rc.d $1 restart + fi' elif [ -x /sbin/service ]; then # Old RedHat RCDIR=/etc/init.d - RESTARTCMD="if /sbin/service \$1; then - /sbin/service \$1 restart; -fi" + RESTARTCMD=' + if /sbin/service $1; then + /sbin/service $1 restart + fi' elif [ -x /usr/sbin/service ]; then # Could be FreeBSD - RESTARTCMD="if /usr/sbin/service \$1 $status 1>/dev/null 2>&1; then - /usr/sbin/service \$1 restart; -fi" + RESTARTCMD=" + if /usr/sbin/service \$1 $status >/dev/null 2>&1 + then + /usr/sbin/service \$1 restart + fi" elif [ -x /bin/sv ]; then - RESTARTCMD="/bin/sv status \$1 >/dev/null 2>&1 && /bin/sv try-restart \$1" + RESTARTCMD='/bin/sv status $1 >/dev/null 2>&1 && + /bin/sv try-restart $1' elif [ -x /usr/bin/sv ]; then - RESTARTCMD="/usr/bin/sv status \$1 >/dev/null 2>&1 && /usr/bin/sv try-restart \$1" - elif [ -e /etc/arch-release -a -d /etc/rc.d ]; then *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Nov 20 22:25:51 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 345151C35C0; Wed, 20 Nov 2019 22:25:51 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JHMg1H2kz4RG9; Wed, 20 Nov 2019 22:25:51 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0E1CB17FD; Wed, 20 Nov 2019 22:25:51 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAKMPpg3091663; Wed, 20 Nov 2019 22:25:51 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAKMPn6u091655; Wed, 20 Nov 2019 22:25:49 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201911202225.xAKMPn6u091655@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 20 Nov 2019 22:25:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354920 - vendor/openresolv/3.9.2 X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: vendor/openresolv/3.9.2 X-SVN-Commit-Revision: 354920 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Nov 2019 22:25:51 -0000 Author: pfg Date: Wed Nov 20 22:25:49 2019 New Revision: 354920 URL: https://svnweb.freebsd.org/changeset/base/354920 Log: Tag openresolv 3.9.2. More information at: https://roy.marples.name/projects/openresolv Added: vendor/openresolv/3.9.2/ - copied from r354918, vendor/openresolv/dist/ vendor/openresolv/3.9.2/LICENSE - copied unchanged from r354919, vendor/openresolv/dist/LICENSE vendor/openresolv/3.9.2/README.md - copied unchanged from r354919, vendor/openresolv/dist/README.md Replaced: vendor/openresolv/3.9.2/Makefile - copied unchanged from r354919, vendor/openresolv/dist/Makefile vendor/openresolv/3.9.2/configure - copied unchanged from r354919, vendor/openresolv/dist/configure vendor/openresolv/3.9.2/dnsmasq.in - copied unchanged from r354919, vendor/openresolv/dist/dnsmasq.in vendor/openresolv/3.9.2/libc.in - copied unchanged from r354919, vendor/openresolv/dist/libc.in vendor/openresolv/3.9.2/named.in - copied unchanged from r354919, vendor/openresolv/dist/named.in vendor/openresolv/3.9.2/pdns_recursor.in - copied unchanged from r354919, vendor/openresolv/dist/pdns_recursor.in vendor/openresolv/3.9.2/pdnsd.in - copied unchanged from r354919, vendor/openresolv/dist/pdnsd.in vendor/openresolv/3.9.2/resolvconf.conf - copied unchanged from r354919, vendor/openresolv/dist/resolvconf.conf vendor/openresolv/3.9.2/resolvconf.conf.5.in - copied unchanged from r354919, vendor/openresolv/dist/resolvconf.conf.5.in vendor/openresolv/3.9.2/resolvconf.in - copied unchanged from r354919, vendor/openresolv/dist/resolvconf.in vendor/openresolv/3.9.2/unbound.in - copied unchanged from r354919, vendor/openresolv/dist/unbound.in Deleted: vendor/openresolv/3.9.2/README Copied: vendor/openresolv/3.9.2/LICENSE (from r354919, vendor/openresolv/dist/LICENSE) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/LICENSE Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/LICENSE) @@ -0,0 +1,23 @@ +Copyright (c) 2007-2019 Roy Marples +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. Copied: vendor/openresolv/3.9.2/Makefile (from r354919, vendor/openresolv/dist/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/Makefile Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/Makefile) @@ -0,0 +1,118 @@ +PKG= openresolv + +# Nasty hack so that make clean works without configure being run +_CONFIG_MK!= test -e config.mk && echo config.mk || echo config-null.mk +CONFIG_MK?= ${_CONFIG_MK} +include ${CONFIG_MK} + +SBINDIR?= /sbin +SYSCONFDIR?= /etc +LIBEXECDIR?= /libexec/resolvconf +VARDIR?= /var/run/resolvconf + +ECHO?= echo +INSTALL?= install +SED?= sed + +VERSION!= ${SED} -n 's/OPENRESOLV_VERSION="\(.*\)".*/\1/p' resolvconf.in + +BINMODE?= 0755 +DOCMODE?= 0644 +MANMODE?= 0444 + +RESOLVCONF= resolvconf resolvconf.8 resolvconf.conf.5 +SUBSCRIBERS= libc dnsmasq named pdnsd pdns_recursor unbound +TARGET= ${RESOLVCONF} ${SUBSCRIBERS} +SRCS= ${TARGET:C,$,.in,} # pmake +SRCS:= ${TARGET:=.in} # gmake + +SED_SBINDIR= -e 's:@SBINDIR@:${SBINDIR}:g' +SED_SYSCONFDIR= -e 's:@SYSCONFDIR@:${SYSCONFDIR}:g' +SED_LIBEXECDIR= -e 's:@LIBEXECDIR@:${LIBEXECDIR}:g' +SED_VARDIR= -e 's:@VARDIR@:${VARDIR}:g' +SED_RCDIR= -e 's:@RCDIR@:${RCDIR}:g' +SED_RESTARTCMD= -e 's:@RESTARTCMD@:${RESTARTCMD}:g' +SED_RCDIR= -e 's:@RCDIR@:${RCDIR}:g' +SED_STATUSARG= -e 's:@STATUSARG@:${STATUSARG}:g' + +DISTPREFIX?= ${PKG}-${VERSION} +DISTFILEGZ?= ${DISTPREFIX}.tar.gz +DISTFILE?= ${DISTPREFIX}.tar.xz +DISTINFO= ${DISTFILE}.distinfo +DISTINFOSIGN= ${DISTINFO}.asc +CKSUM?= cksum -a SHA256 +PGP?= netpgp + +GITREF?= HEAD + +.SUFFIXES: .in + +all: ${TARGET} + +.in: Makefile ${CONFIG_MK} + ${SED} ${SED_SBINDIR} ${SED_SYSCONFDIR} ${SED_LIBEXECDIR} \ + ${SED_VARDIR} \ + ${SED_RCDIR} ${SED_RESTARTCMD} ${SED_RCDIR} ${SED_STATUSARG} \ + $< > $@ + +clean: + rm -f ${TARGET} + +distclean: clean + rm -f config.mk ${DISTFILE} ${DISTINFO} ${DISTINFOSIGN} + +installdirs: + +proginstall: ${TARGET} + ${INSTALL} -d ${DESTDIR}${SBINDIR} + ${INSTALL} -m ${BINMODE} resolvconf ${DESTDIR}${SBINDIR} + ${INSTALL} -d ${DESTDIR}${SYSCONFDIR} + test -e ${DESTDIR}${SYSCONFDIR}/resolvconf.conf || \ + ${INSTALL} -m ${DOCMODE} resolvconf.conf ${DESTDIR}${SYSCONFDIR} + ${INSTALL} -d ${DESTDIR}${LIBEXECDIR} + ${INSTALL} -m ${DOCMODE} ${SUBSCRIBERS} ${DESTDIR}${LIBEXECDIR} + +maninstall: + ${INSTALL} -d ${DESTDIR}${MANDIR}/man8 + ${INSTALL} -m ${MANMODE} resolvconf.8 ${DESTDIR}${MANDIR}/man8 + ${INSTALL} -d ${DESTDIR}${MANDIR}/man5 + ${INSTALL} -m ${MANMODE} resolvconf.conf.5 ${DESTDIR}${MANDIR}/man5 + +install: proginstall maninstall + +dist-git: + git archive --prefix=${DISTPREFIX}/ ${GITREF} | xz >${DISTFILE} + +dist-inst: + mkdir /tmp/${DISTPREFIX} + cp -RPp * /tmp/${DISTPREFIX} + (cd /tmp/${DISTPREFIX}; make clean) + tar -cvjpf ${DISTFILE} -C /tmp ${DISTPREFIX} + rm -rf /tmp/${DISTPREFIX} + +dist: dist-git + +distinfo: dist + rm -f ${DISTINFO} ${DISTINFOSIGN} + ${CKSUM} ${DISTFILE} >${DISTINFO} + #printf "SIZE (${DISTFILE}) = %s\n" $$(wc -c <${DISTFILE}) >>${DISTINFO} + ${PGP} --clearsign --output=${DISTINFOSIGN} ${DISTINFO} + chmod 644 ${DISTINFOSIGN} + ls -l ${DISTFILE} ${DISTINFO} ${DISTINFOSIGN} + +import: dist + rm -rf /tmp/${DISTPREFIX} + ${INSTALL} -d /tmp/${DISTPREFIX} + tar xvJpf ${DISTFILE} -C /tmp + +_import-src: + rm -rf ${DESTDIR}/* + ${INSTALL} -d ${DESTDIR} + cp LICENSE README.md ${SRCS} resolvconf.conf ${DESTDIR}; + cp resolvconf.8.in resolvconf.conf.5.in ${DESTDIR}; + @${ECHO} + @${ECHO} "=============================================================" + @${ECHO} "openresolv-${VERSION} imported to ${DESTDIR}" + +import-src: + ${MAKE} _import-src DESTDIR=`if [ -n "${DESTDIR}" ]; then echo "${DESTDIR}"; else echo /tmp/${DISTPREFIX}; fi` Copied: vendor/openresolv/3.9.2/README.md (from r354919, vendor/openresolv/dist/README.md) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/README.md Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/README.md) @@ -0,0 +1,64 @@ +# openresolv + +openresolv is a [resolvconf](https://en.wikipedia.org/wiki/Resolvconf) +implementation which manages `/etc/resolv.conf`. + +`/etc/resolv.conf` is a file that holds the configuration for the local +resolution of domain names. +Normally this file is either static or maintained by a local daemon, +normally a DHCP daemon. But what happens if more than one thing wants to +control the file? +Say you have wired and wireless interfaces to different subnets and run a VPN +or two on top of that, how do you say which one controls the file? +It's also not as easy as just adding and removing the nameservers each client +knows about as different clients could add the same nameservers. + +Enter resolvconf, the middleman between the network configuration services and +`/etc/resolv.conf`. +resolvconf itself is just a script that stores, removes and lists a full +`resolv.conf` generated for the interface. It then calls all the helper scripts +it knows about so it can configure the real `/etc/resolv.conf` and optionally +any local nameservers other than libc. + +## Reasons for using openresolv + +Why openresolv over the +[Debian implementation](http://qref.sourceforge.net/Debian/reference/ch-gateway.en.html#s-dns-resolvconf)? +Here's some reasons: + * Works with + [POSIX shell and userland](http://www.opengroup.org/onlinepubs/009695399) + * Does not need awk, grep or sed which means we can work without `/usr` + mounted + * Works with other init systems than Debians' out of the box + * Available as a 2 clause + [BSD license](http://www.freebsd.org/copyright/freebsd-license.html) + * Prefer configs via IF_METRIC for dynamic ordering + * Configures zones for local resolvers other than libc + +The last point is quite important, especially when running VPN systems. +Take the following resolv.conf files which have been generated by a +[DHCP client](../dhcpcd) and sent to resolvconf: + +``` +# resolv.conf from bge0 +search foo.com +nameserver 1.2.3.4 + +# resolv.conf from tap0 +domain bar.org +nameserver 5.6.7.8 +``` + +In this instance, queries for foo.com will go to 1.2.3.4 and queries for +bar.org will go to 5.6.7.8. +This does require the resolvers to be configured to pickup the resolvconf +generated configuration for them though. +openresolv ships with helpers for: + * [unbound](http://www.unbound.net/) + * [dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) + * [ISC BIND](http://www.isc.org/software/bind) + * [PowerDNS Recursor](http://wiki.powerdns.com/trac) + +See the +[configuration section](https://roy.marples.name/projects/openresolv/config) +for more details. Copied: vendor/openresolv/3.9.2/configure (from r354919, vendor/openresolv/dist/configure) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/configure Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/configure) @@ -0,0 +1,167 @@ +#!/bin/sh +# Try and be like autotools configure, but without autotools + +# Ensure that we do not inherit these from env +OS= +BUILD= +HOST= +TARGET= +RESTARTCMD= +RCDIR= +STATUSARG= + +for x do + opt=${x%%=*} + var=${x#*=} + case "$opt" in + --os|OS) OS=$var;; + --with-cc|CC) CC=$var;; + --debug) DEBUG=$var;; + --disable-debug) DEBUG=no;; + --enable-debug) DEBUG=yes;; + --prefix) PREFIX=$var;; + --sysconfdir) SYSCONFDIR=$var;; + --bindir|--sbindir) SBINDIR=$var;; + --libexecdir) LIBEXECDIR=$var;; + --statedir|--localstatedir) STATEDIR=$var;; + --dbdir) DBDIR=$var;; + --rundir) RUNDIR=$var;; + --mandir) MANDIR=$var;; + --with-ccopts|CFLAGS) CFLAGS=$var;; + CPPFLAGS) CPPFLAGS=$var;; + --build) BUILD=$var;; + --host) HOST=$var;; + --target) TARGET=$var;; + --libdir) LIBDIR=$var;; + --restartcmd) RESTARTCMD=$var;; + --rcdir) RCDIR=$var;; + --statusarg) STATUSARG=$var;; + --includedir) eval INCLUDEDIR="$INCLUDEDIR${INCLUDEDIR:+ }$var";; + --datadir|--infodir) ;; # ignore autotools + --disable-maintainer-mode|--disable-dependency-tracking) ;; + --help) echo "See the README file for available options"; exit 0;; + *) echo "$0: WARNING: unknown option $opt" >&2;; + esac +done + +: ${SED:=sed} + +CONFIG_MK=config.mk + +if [ -z "$BUILD" ]; then + # autoconf target triplet: cpu-vendor-os + BUILD=$(uname -m)-unknown-$(uname -s | tr '[:upper:]' '[:lower:]') +fi +: ${HOST:=$BUILD} + +if [ -z "$OS" ]; then + echo "Deriving operating system from ... $HOST" + # Derive OS from cpu-vendor-[kernel-]os + CPU=${HOST%%-*} + REST=${HOST#*-} + if [ "$CPU" != "$REST" ]; then + VENDOR=${REST%%-*} + REST=${REST#*-} + if [ "$VENDOR" != "$REST" ]; then + # Use kernel if given, otherwise os + OS=${REST%%-*} + else + # 2 tupple + OS=$VENDOR + VENDOR= + fi + fi + + # Work with cpu-kernel-os, ie Debian + case "$VENDOR" in + linux*|kfreebsd*) OS=$VENDOR; VENDOR= ;; + esac + # Special case + case "$OS" in + gnu*) OS=hurd;; # No HURD support as yet + esac +fi + +echo "Configuring openresolv for ... $OS" +rm -rf $CONFIG_MK +echo "# $OS" >$CONFIG_MK + +case "$OS" in +dragonfly*) + # This means /usr HAS to be mounted not via dhcpcd + : ${LIBEXECDIR:=${PREFIX:-/usr}/libexec/resolvconf} + ;; +linux*) + # cksum does't support -a and netpgp is rare + echo "CKSUM= sha256sum --tag" >>$CONFIG_MK + echo "PGP= gpg2" >>$CONFIG_MK + ;; +esac + +case "$OS" in +dragonfly*|freebsd*) + # On FreeBSD, /etc/init.d/foo status returns 0 if foo is not enabled + # regardless of if it's not running. + # So we force onestatus to work around this silly bug. + if [ -z "$STATUSARG" ]; then + STATUSARG="onestatus" + fi + ;; +esac + + +if [ -z "$LIBEXECDIR" ]; then + printf "Checking for directory /libexec ... " + if [ -d /libexec ]; then + echo "yes" + LIBEXECDIR=$PREFIX/libexec/resolvconf + else + echo "no" + LIBEXECDIR=$PREFIX/lib/resolvconf + fi +fi +if [ -z "$RUNDIR" ]; then + printf "Checking for directory /run ... " + if [ -d /run ]; then + echo "yes" + RUNDIR=/run + else + echo "no" + RUNDIR=/var/run + fi +fi + +: ${SYSCONFDIR:=$PREFIX/etc} +: ${SBINDIR:=$PREFIX/sbin} +: ${LIBEXECDIR:=$PREFIX/libexec/resolvconf} +: ${STATEDIR:=/var} +: ${RUNDIR:=$STATEDIR/run} +: ${MANDIR:=${PREFIX:-/usr}/share/man} + +eval SYSCONFDIR="$SYSCONFDIR" +eval SBINDIR="$SBINDIR" +eval LIBEXECDIR="$LIBEXECDIR" +eval VARDIR="$RUNDIR/resolvconf" +eval MANDIR="$MANDIR" + +for x in SYSCONFDIR SBINDIR LIBEXECDIR VARDIR MANDIR RESTARTCMD RCDIR STATUSARG +do + eval v=\$$x + # Make files look nice for import + l=$((10 - ${#x})) + unset t + [ $l -gt 3 ] && t=" " + echo "$x=$t $v" >>$CONFIG_MK +done + +echo +echo " SYSCONFDIR = $SYSCONFDIR" +echo " SBINDIR = $SBINDIR" +echo " LIBEXECDIR = $LIBEXECDIR" +echo " VARDIR = $RUNDIR" +echo " MANDIR = $MANDIR" +echo +echo " RESTARTCMD = $RESTARTCMD" +echo " RCDIR = $RCDIR" +echo " STATUSARG = $STATUSARG" +echo Copied: vendor/openresolv/3.9.2/dnsmasq.in (from r354919, vendor/openresolv/dist/dnsmasq.in) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/dnsmasq.in Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/dnsmasq.in) @@ -0,0 +1,211 @@ +#!/bin/sh +# Copyright (c) 2007-2019 Roy Marples +# All rights reserved + +# dnsmasq subscriber for resolvconf + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials provided +# with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +[ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 +. "@SYSCONFDIR@/resolvconf.conf" || exit 1 +[ -z "${dnsmasq_conf}${dnsmasq_resolv}" ] && exit 0 +[ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" +NL=" +" + +: ${dnsmasq_pid:=/var/run/dnsmasq.pid} +[ -s "$dnsmasq_pid" ] || dnsmasq_pid=/var/run/dnsmasq/dnsmasq.pid +[ -s "$dnsmasq_pid" ] || unset dnsmasq_pid +: ${dnsmasq_service:=dnsmasq} +newconf="# Generated by resolvconf$NL" +newresolv="$newconf" + +# Using dbus means that we never have to restart the daemon +# This is important as it means we should not drop DNS queries +# whilst changing DNS options around. However, dbus support is optional +# so we need to validate a few things first. +# Check for DBus support in the binary +dbus=false +dbus_ex=false +dbus_introspect=$(dbus-send --print-reply --system \ + --dest=uk.org.thekelleys.dnsmasq \ + /uk/org/thekelleys/dnsmasq \ + org.freedesktop.DBus.Introspectable.Introspect \ + 2>/dev/null) +if [ $? = 0 ]; then + dbus=true + if printf %s "$dbus_introspect" | \ + grep -q '' + then + dbus_ex=true + fi +fi + +for n in $NAMESERVERS; do + newresolv="${newresolv}nameserver $n$NL" +done + +dbusdest= +dbusdest_ex= +conf= +for d in $DOMAINS; do + dn="${d%%:*}" + ns="${d#*:}" + while [ -n "$ns" ]; do + n="${ns%%,*}" + if $dbus && ! $dbus_ex; then + case "$n" in + *.*.*.*) + SIFS=${IFS-y} OIFS=$IFS + IFS=. + set -- $n + num="0x$(printf %02x $1 $2 $3 $4)" + if [ "$SIFS" = y ]; then + unset IFS + else + IFS=$OIFS + fi + dbusdest="$dbusdest uint32:$(printf %u $num)" + dbusdest="$dbusdest string:$dn" + ;; + *:*%*) + # This version of dnsmasq won't accept + # scoped IPv6 addresses + dbus=false + ;; + *:*) + SIFS=${IFS-y} OIFS=$IFS bytes= front= back= + empty=false i=0 + IFS=: + set -- $n + while [ -n "$1" ] || [ -n "$2" ]; do + addr="$1" + shift + if [ -z "$addr" ]; then + empty=true + continue + fi + i=$(($i + 1)) + while [ ${#addr} -lt 4 ]; do + addr="0${addr}" + done + byte1="$(printf %d 0x${addr%??})" + byte2="$(printf %d 0x${addr#??})" + if $empty; then + back="$back byte:$byte1 byte:$byte2" + else + front="$front byte:$byte1 byte:$byte2" + fi + done + while [ $i != 8 ]; do + i=$(($i + 1)) + front="$front byte:0 byte:0" + done + front="${front}$back" + if [ "$SIFS" = y ]; then + unset IFS + else + IFS=$OIFS + fi + dbusdest="${dbusdest}$front string:$dn" + ;; + *) + if ! $dbus_ex; then + dbus=false + fi + ;; + esac + fi + dbusdest_ex="$dbusdest_ex${dbusdest_ex:+,}/$dn/$n" + conf="${conf}server=/$dn/$n$NL" + [ "$ns" = "${ns#*,}" ] && break + ns="${ns#*,}" + done +done + +if $dbus; then + newconf="$newconf$NL# Domain specific servers will" + newconf="$newconf be sent over dbus${NL}" +else + newconf="$newconf$conf" +fi + +# Try to ensure that config dirs exist +if type config_mkdirs >/dev/null 2>&1; then + config_mkdirs "$dnsmasq_conf" "$dnsmasq_resolv" +else + @SBINDIR@/resolvconf -D "$dnsmasq_conf" "$dnsmasq_resolv" +fi + +changed=false +if [ -n "$dnsmasq_conf" ]; then + if [ ! -f "$dnsmasq_conf" ] || \ + [ "$(cat "$dnsmasq_conf")" != "$(printf %s "$newconf")" ] + then + changed=true + printf %s "$newconf" >"$dnsmasq_conf" + fi +fi +if [ -n "$dnsmasq_resolv" ]; then + # dnsmasq polls this file so no need to set changed=true + if [ -f "$dnsmasq_resolv" ]; then + if [ "$(cat "$dnsmasq_resolv")" != "$(printf %s "$newresolv")" ] + then + printf %s "$newresolv" >"$dnsmasq_resolv" + fi + else + printf %s "$newresolv" >"$dnsmasq_resolv" + fi +fi + +if $changed; then + # dnsmasq does not re-read the configuration file on SIGHUP + if [ -n "$dnsmasq_restart" ]; then + eval $dnsmasq_restart + elif [ -n "$RESTARTCMD" ]; then + set -- ${dnsmasq_service} + eval "$RESTARTCMD" + else + @SBINDIR@/resolvconf -r ${dnsmasq_service} + fi +fi +if $dbus; then + if [ -s "$dnsmasq_pid" ]; then + $changed || kill -HUP $(cat "$dnsmasq_pid") + fi + # Send even if empty so old servers are cleared + if $dbus_ex; then + method=SetDomainServers + if [ -n "$dbusdest_ex" ]; then + dbusdest_ex="array:string:$dbusdest_ex" + fi + dbusdest="$dbusdest_ex" + else + method=SetServers + fi + dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ + /uk/org/thekelleys/dnsmasq uk.org.thekelleys.$method \ + $dbusdest + dbus-send --system --dest=uk.org.thekelleys.dnsmasq \ + /uk/org/thekelleys/dnsmasq uk.org.thekelleys.ClearCache +fi Copied: vendor/openresolv/3.9.2/libc.in (from r354919, vendor/openresolv/dist/libc.in) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/libc.in Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/libc.in) @@ -0,0 +1,252 @@ +#!/bin/sh +# Copyright (c) 2007-2019 Roy Marples +# All rights reserved + +# libc subscriber for resolvconf + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials provided +# with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +SYSCONFDIR=@SYSCONFDIR@ +LIBEXECDIR=@LIBEXECDIR@ +VARDIR=@VARDIR@ +IFACEDIR="$VARDIR/interfaces" +NL=" +" + +# sed may not be available, and this is faster on small files +key_get_value() +{ + key="$1" + shift + + if [ $# -eq 0 ]; then + while read -r line; do + case "$line" in + "$key"*) echo "${line##$key}";; + esac + done + else + for x do + while read -r line; do + case "$line" in + "$key"*) echo "${line##$key}";; + esac + done < "$x" + done + fi +} + +keys_remove() +{ + while read -r line; do + found=false + for key do + case "$line" in + "$key"*|"#"*|" "*|" "*|"") found=true;; + esac + $found && break + done + $found || echo "$line" + done +} + +local_nameservers="127.* 0.0.0.0 255.255.255.255 ::1" + +# Support original resolvconf configuration layout +# as well as the openresolv config file +if [ -f "$SYSCONFDIR"/resolvconf.conf ]; then + . "$SYSCONFDIR"/resolvconf.conf +elif [ -d "$SYSCONFDIR"/resolvconf ]; then + SYSCONFDIR="$SYSCONFDIR/resolvconf" + base="$SYSCONFDIR/resolv.conf.d/base" + if [ -f "$base" ]; then + prepend_nameservers="$(key_get_value "nameserver " "$base")" + domain="$(key_get_value "domain " "$base")" + prepend_search="$(key_get_value "search " "$base")" + resolv_conf_options="$(key_get_value "options " "$base")" + resolv_conf_sortlist="$(key_get_value "sortlist " "$base")" + fi + if [ -f "$SYSCONFDIR"/resolv.conf.d/head ]; then + resolv_conf_head="$(cat "${SYSCONFDIR}"/resolv.conf.d/head)" + fi + if [ -f "$SYSCONFDIR"/resolv.conf.d/tail ]; then + resolv_conf_tail="$(cat "$SYSCONFDIR"/resolv.conf.d/tail)" + fi +fi +: ${resolv_conf:=/etc/resolv.conf} +: ${libc_service:=nscd} +: ${list_resolv:=@SBINDIR@/resolvconf -l} +if [ "${resolv_conf_head-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.head ] +then + resolv_conf_head="$(cat "${SYSCONFDIR}"/resolv.conf.head)" +fi +if [ "${resolv_conf_tail-x}" = x ] && [ -f "$SYSCONFDIR"/resolv.conf.tail ] +then + resolv_conf_tail="$(cat "$SYSCONFDIR"/resolv.conf.tail)" +fi + +backup=true +signature="# Generated by resolvconf" + +uniqify() +{ + result= + while [ -n "$1" ]; do + case " $result " in + *" $1 "*);; + *) result="$result $1";; + esac + shift + done + echo "${result# *}" +} + +case "${resolv_conf_passthrough:-NO}" in +[Yy][Ee][Ss]|[Tt][Rr][Uu][Ee]|[Oo][Nn]|1) + backup=false + newest= + for conf in "$IFACEDIR"/*; do + if [ -z "$newest" ] || [ "$conf" -nt "$newest" ]; then + newest="$conf" + fi + done + [ -z "$newest" ] && exit 0 + newconf="$(cat "$newest")$NL" + ;; +/dev/null|[Nn][Uu][Ll][Ll]) + : ${resolv_conf_local_only:=NO} + if [ "$local_nameservers" = "127.* 0.0.0.0 255.255.255.255 ::1" ]; then + local_nameservers= + fi + # Need to overwrite our variables. + eval "$(@SBINDIR@/resolvconf -V)" + ;; + +*) + [ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" + ;; +esac +case "${resolv_conf_passthrough:-NO}" in +[Yy][Ee][Ss]|[Tt][Rr][Uu][Ee]|[Oo][Nn]|1) ;; +*) + : ${domain:=$DOMAIN} + newsearch="$(uniqify $prepend_search $SEARCH $append_search)" + NS="$LOCALNAMESERVERS $NAMESERVERS" + newns= + gotlocal=false + for n in $(uniqify $prepend_nameservers $NS $append_nameservers); do + add=true + islocal=false + for l in $local_nameservers; do + case "$n" in + $l) islocal=true; gotlocal=true; break;; + esac + done + if ! $islocal; then + case "${resolv_conf_local_only:-YES}" in + [Yy][Ee][Ss]|[Tt][Rr][Uu][Ee]|[Oo][Nn]|1) + $gotlocal && add=false;; + esac + fi + $add && newns="$newns $n" + done + + # Hold our new resolv.conf in a variable to save on temporary files + newconf="$signature$NL" + if [ -n "$resolv_conf_head" ]; then + newconf="$newconf$resolv_conf_head$NL" + fi + + [ -n "$domain" ] && newconf="${newconf}domain $domain$NL" + if [ -n "$newsearch" ] && [ "$newsearch" != "$domain" ]; then + newconf="${newconf}search $newsearch$NL" + fi + for n in $newns; do + newconf="${newconf}nameserver $n$NL" + done + + # Now add anything we don't care about such as sortlist and options + stuff="$($list_resolv | keys_remove nameserver domain search)" + if [ -n "$stuff" ]; then + newconf="$newconf$stuff$NL" + fi + + # Append any user defined ones + if [ -n "$resolv_conf_options" ]; then + newconf="${newconf}options $resolv_conf_options$NL" + fi + if [ -n "$resolv_conf_sortlist" ]; then + newconf="${newconf}sortlist $resolv_conf_sortlist$NL" + fi + + if [ -n "$resolv_conf_tail" ]; then + newconf="$newconf$resolv_conf_tail$NL" + fi + ;; +esac + +# Check if the file has actually changed or not +if [ -e "$resolv_conf" ]; then + [ "$(cat "$resolv_conf")" = "$(printf %s "$newconf")" ] && exit 0 +fi + +# Change is good. +# If the old file does not have our signature, back it up. +# If the new file just has our signature, restore the backup. +if $backup; then + if [ "$newconf" = "$signature$NL" ]; then + if [ -e "$resolv_conf.bak" ]; then + newconf="$(cat "$resolv_conf.bak")$NL" + fi + elif [ -e "$resolv_conf" ]; then + read line <"$resolv_conf" + if [ "$line" != "$signature" ]; then + cp "$resolv_conf" "$resolv_conf.bak" + fi + fi +fi + +# Create our resolv.conf now +(umask 022; printf %s "$newconf" >"$resolv_conf") +if [ -n "$libc_restart" ]; then + eval $libc_restart +elif [ -n "$RESTARTCMD" ]; then + set -- ${libc_service} + eval "$RESTARTCMD" +else + @SBINDIR@/resolvconf -r ${libc_service} +fi + +retval=0 +# Notify users of the resolver +for script in "$LIBEXECDIR"/libc.d/*; do + if [ -f "$script" ]; then + if [ -x "$script" ]; then + "$script" "$@" + else + (. "$script") + fi + retval=$(($retval + $?)) + fi +done +exit $retval Copied: vendor/openresolv/3.9.2/named.in (from r354919, vendor/openresolv/dist/named.in) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/named.in Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/named.in) @@ -0,0 +1,118 @@ +#!/bin/sh +# Copyright (c) 2007-2016 Roy Marples +# All rights reserved + +# named subscriber for resolvconf + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials provided +# with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +[ -f "@SYSCONFDIR@"/resolvconf.conf ] || exit 0 +. "@SYSCONFDIR@/resolvconf.conf" || exit 1 +[ -z "${named_zones}${named_options}" ] && exit 0 +[ -z "$RESOLVCONF" ] && eval "$(@SBINDIR@/resolvconf -v)" +NL=" +" + +# Platform specific kludges +if [ -z "${named_service}${named_restart}" ] && + [ -d "$RCDIR" ] && ! [ -x "$RCDIR"/named ] +then + if [ -x "$RCDIR"/bind9 ]; then + # Debian and derivatives + named_service=bind9 + elif [ -x "$RCDIR"/rc.bind ]; then + # Slackware + named_service=rc.bind + fi +fi +: ${named_service:=named} + +: ${named_pid:=/var/run/$named_service.pid} +[ -s "$named_pid" ] || named_pid=/var/run/$named_service/$named_service.pid +[ -s "$named_pid" ] || unset named_pid + +newoptions="# Generated by resolvconf$NL" +newzones="$newoptions" + +forward= +for n in $NAMESERVERS; do + case "$forward" in + *"$NL $n;"*);; + *) forward="$forward$NL $n;";; + esac +done +if [ -n "$forward" ]; then + newoptions="${newoptions}forward first;${NL}forwarders {$forward${NL}};$NL" +fi + +for d in $DOMAINS; do + newzones="${newzones}zone \"${d%%:*}\" {$NL" + newzones="$newzones type forward;$NL" + newzones="$newzones forward first;$NL forwarders {$NL" + ns="${d#*:}" + while [ -n "$ns" ]; do + newzones="$newzones ${ns%%,*};$NL" + [ "$ns" = "${ns#*,}" ] && break + ns="${ns#*,}" + done + newzones="$newzones };$NL};$NL" +done + +# Try to ensure that config dirs exist +if type config_mkdirs >/dev/null 2>&1; then + config_mkdirs "$named_options" "$named_zones" +else + @SBINDIR@/resolvconf -D "$named_options" "$named_zones" +fi + +# No point in changing files or reloading bind if the end result has not +# changed +changed=false +if [ -n "$named_options" ]; then + if [ ! -f "$named_options" ] || \ + [ "$(cat "$named_options")" != "$(printf %s "$newoptions")" ] + then + printf %s "$newoptions" >"$named_options" + changed=true + fi +fi +if [ -n "$named_zones" ]; then + if [ ! -f "$named_zones" ] || \ + [ "$(cat "$named_zones")" != "$(printf %s "$newzones")" ] + then + printf %s "$newzones" >"$named_zones" + changed=true + fi +fi + +# named does not seem to work with SIGHUP which is a same +if $changed; then + if [ -n "$named_restart" ]; then + eval $named_restart + elif [ -n "$RESTARTCMD" ]; then + set -- ${named_service} + eval "$RESTARTCMD" + else + @SBINDIR@/resolvconf -r ${named_service} + fi +fi Copied: vendor/openresolv/3.9.2/pdns_recursor.in (from r354919, vendor/openresolv/dist/pdns_recursor.in) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/openresolv/3.9.2/pdns_recursor.in Wed Nov 20 22:25:49 2019 (r354920, copy of r354919, vendor/openresolv/dist/pdns_recursor.in) @@ -0,0 +1,75 @@ +#!/bin/sh +# Copyright (c) 2009-2019 Roy Marples +# All rights reserved + +# PowerDNS Recursor subscriber for resolvconf + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials provided +# with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Nov 21 13:46:18 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9C5571BF169; Thu, 21 Nov 2019 13:46:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jgnk4RBWz4Pdf; Thu, 21 Nov 2019 13:46:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7C82CC27B; Thu, 21 Nov 2019 13:46:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALDkI8H035767; Thu, 21 Nov 2019 13:46:18 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALDkHZ8035760; Thu, 21 Nov 2019 13:46:17 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211346.xALDkHZ8035760@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 13:46:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354949 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb ven... X-SVN-Commit-Revision: 354949 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 13:46:18 -0000 Author: avg Date: Thu Nov 21 13:46:16 2019 New Revision: 354949 URL: https://svnweb.freebsd.org/changeset/base/354949 Log: 10405 Implement ZFS sorted scans illumos/illumos-gate@a3874b8b1fe5103fc1f961609557c0587435fec0 https://github.com/illumos/illumos-gate/commit/a3874b8b1fe5103fc1f961609557c0587435fec0 https://www.illumos.org/issues/10405 The original implementation is: https://github.com/zfsonlinux/zfs/commit/d4a72f23863382bdf6d0ae33196f5b5decbc48fd Author: Toomas Soome Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_scan.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h vendor-sys/illumos/dist/uts/common/sys/taskq.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_status.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Thu Nov 21 13:46:16 2019 (r354949) @@ -348,7 +348,8 @@ int arc_no_grow_shift = 5; * minimum lifespan of a prefetch block in clock ticks * (initialized in arc_init()) */ -static int arc_min_prefetch_lifespan; +static int zfs_arc_min_prefetch_ms = 1; +static int zfs_arc_min_prescient_prefetch_ms = 6; /* * If this percent of memory is free, don't throttle. @@ -690,8 +691,9 @@ typedef struct arc_stats { kstat_named_t arcstat_meta_limit; kstat_named_t arcstat_meta_max; kstat_named_t arcstat_meta_min; - kstat_named_t arcstat_sync_wait_for_async; + kstat_named_t arcstat_async_upgrade_sync; kstat_named_t arcstat_demand_hit_predictive_prefetch; + kstat_named_t arcstat_demand_hit_prescient_prefetch; } arc_stats_t; static arc_stats_t arc_stats = { @@ -774,8 +776,9 @@ static arc_stats_t arc_stats = { { "arc_meta_limit", KSTAT_DATA_UINT64 }, { "arc_meta_max", KSTAT_DATA_UINT64 }, { "arc_meta_min", KSTAT_DATA_UINT64 }, - { "sync_wait_for_async", KSTAT_DATA_UINT64 }, + { "async_upgrade_sync", KSTAT_DATA_UINT64 }, { "demand_hit_predictive_prefetch", KSTAT_DATA_UINT64 }, + { "demand_hit_prescient_prefetch", KSTAT_DATA_UINT64 }, }; #define ARCSTAT(stat) (arc_stats.stat.value.ui64) @@ -872,22 +875,23 @@ typedef struct arc_callback arc_callback_t; struct arc_callback { void *acb_private; - arc_done_func_t *acb_done; + arc_read_done_func_t *acb_done; arc_buf_t *acb_buf; boolean_t acb_compressed; zio_t *acb_zio_dummy; + zio_t *acb_zio_head; arc_callback_t *acb_next; }; typedef struct arc_write_callback arc_write_callback_t; struct arc_write_callback { - void *awcb_private; - arc_done_func_t *awcb_ready; - arc_done_func_t *awcb_children_ready; - arc_done_func_t *awcb_physdone; - arc_done_func_t *awcb_done; - arc_buf_t *awcb_buf; + void *awcb_private; + arc_write_done_func_t *awcb_ready; + arc_write_done_func_t *awcb_children_ready; + arc_write_done_func_t *awcb_physdone; + arc_write_done_func_t *awcb_done; + arc_buf_t *awcb_buf; }; /* @@ -1013,6 +1017,8 @@ struct arc_buf_hdr { #define HDR_IO_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) #define HDR_IO_ERROR(hdr) ((hdr)->b_flags & ARC_FLAG_IO_ERROR) #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_FLAG_PREFETCH) +#define HDR_PRESCIENT_PREFETCH(hdr) \ + ((hdr)->b_flags & ARC_FLAG_PRESCIENT_PREFETCH) #define HDR_COMPRESSION_ENABLED(hdr) \ ((hdr)->b_flags & ARC_FLAG_COMPRESSED_ARC) @@ -3243,6 +3249,8 @@ arc_evict_hdr(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) { arc_state_t *evicted_state, *state; int64_t bytes_evicted = 0; + int min_lifetime = HDR_PRESCIENT_PREFETCH(hdr) ? + zfs_arc_min_prescient_prefetch_ms : zfs_arc_min_prefetch_ms; ASSERT(MUTEX_HELD(hash_lock)); ASSERT(HDR_HAS_L1HDR(hdr)); @@ -3295,8 +3303,7 @@ arc_evict_hdr(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) /* prefetch buffers have a minimum lifespan */ if (HDR_IO_IN_PROGRESS(hdr) || ((hdr->b_flags & (ARC_FLAG_PREFETCH | ARC_FLAG_INDIRECT)) && - ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < - arc_min_prefetch_lifespan)) { + ddi_get_lbolt() - hdr->b_l1hdr.b_arc_access < min_lifetime * hz)) { ARCSTAT_BUMP(arcstat_evict_skip); return (bytes_evicted); } @@ -4607,13 +4614,15 @@ arc_access(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) * - move the buffer to the head of the list if this is * another prefetch (to make it less likely to be evicted). */ - if (HDR_PREFETCH(hdr)) { + if (HDR_PREFETCH(hdr) || HDR_PRESCIENT_PREFETCH(hdr)) { if (zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) == 0) { /* link protected by hash lock */ ASSERT(multilist_link_active( &hdr->b_l1hdr.b_arc_node)); } else { - arc_hdr_clear_flags(hdr, ARC_FLAG_PREFETCH); + arc_hdr_clear_flags(hdr, + ARC_FLAG_PREFETCH | + ARC_FLAG_PRESCIENT_PREFETCH); ARCSTAT_BUMP(arcstat_mru_hits); } hdr->b_l1hdr.b_arc_access = now; @@ -4644,10 +4653,13 @@ arc_access(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) * MFU state. */ - if (HDR_PREFETCH(hdr)) { + if (HDR_PREFETCH(hdr) || HDR_PRESCIENT_PREFETCH(hdr)) { new_state = arc_mru; - if (zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) > 0) - arc_hdr_clear_flags(hdr, ARC_FLAG_PREFETCH); + if (zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) > 0) { + arc_hdr_clear_flags(hdr, + ARC_FLAG_PREFETCH | + ARC_FLAG_PRESCIENT_PREFETCH); + } DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, hdr); } else { new_state = arc_mfu; @@ -4668,11 +4680,6 @@ arc_access(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) * If it was a prefetch, we will explicitly move it to * the head of the list now. */ - if ((HDR_PREFETCH(hdr)) != 0) { - ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt)); - /* link protected by hash_lock */ - ASSERT(multilist_link_active(&hdr->b_l1hdr.b_arc_node)); - } ARCSTAT_BUMP(arcstat_mfu_hits); hdr->b_l1hdr.b_arc_access = ddi_get_lbolt(); } else if (hdr->b_l1hdr.b_state == arc_mfu_ghost) { @@ -4683,12 +4690,11 @@ arc_access(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) * MFU state. */ - if (HDR_PREFETCH(hdr)) { + if (HDR_PREFETCH(hdr) || HDR_PRESCIENT_PREFETCH(hdr)) { /* * This is a prefetch access... * move this block back to the MRU state. */ - ASSERT0(zfs_refcount_count(&hdr->b_l1hdr.b_refcnt)); new_state = arc_mru; } @@ -4710,21 +4716,26 @@ arc_access(arc_buf_hdr_t *hdr, kmutex_t *hash_lock) } } -/* a generic arc_done_func_t which you can use */ +/* a generic arc_read_done_func_t which you can use */ /* ARGSUSED */ void -arc_bcopy_func(zio_t *zio, arc_buf_t *buf, void *arg) +arc_bcopy_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, + arc_buf_t *buf, void *arg) { - if (zio == NULL || zio->io_error == 0) - bcopy(buf->b_data, arg, arc_buf_size(buf)); + if (buf == NULL) + return; + + bcopy(buf->b_data, arg, arc_buf_size(buf)); arc_buf_destroy(buf, arg); } -/* a generic arc_done_func_t */ +/* a generic arc_read_done_func_t */ void -arc_getbuf_func(zio_t *zio, arc_buf_t *buf, void *arg) +arc_getbuf_func(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, + arc_buf_t *buf, void *arg) { arc_buf_t **bufp = arg; + if (buf == NULL) { ASSERT(zio == NULL || zio->io_error != 0); *bufp = NULL; @@ -4759,7 +4770,6 @@ arc_read_done(zio_t *zio) arc_callback_t *callback_list; arc_callback_t *acb; boolean_t freeable = B_FALSE; - boolean_t no_zio_error = (zio->io_error == 0); /* * The hdr was inserted into hash-table and removed from lists @@ -4785,7 +4795,7 @@ arc_read_done(zio_t *zio) ASSERT3P(hash_lock, !=, NULL); } - if (no_zio_error) { + if (zio->io_error == 0) { /* byteswap if necessary */ if (BP_SHOULD_BYTESWAP(zio->io_bp)) { if (BP_GET_LEVEL(zio->io_bp) > 0) { @@ -4806,7 +4816,8 @@ arc_read_done(zio_t *zio) callback_list = hdr->b_l1hdr.b_acb; ASSERT3P(callback_list, !=, NULL); - if (hash_lock && no_zio_error && hdr->b_l1hdr.b_state == arc_anon) { + if (hash_lock && zio->io_error == 0 && + hdr->b_l1hdr.b_state == arc_anon) { /* * Only call arc_access on anonymous buffers. This is because * if we've issued an I/O for an evicted buffer, we've already @@ -4827,30 +4838,29 @@ arc_read_done(zio_t *zio) if (!acb->acb_done) continue; - /* This is a demand read since prefetches don't use callbacks */ callback_cnt++; - if (no_zio_error) { - int error = arc_buf_alloc_impl(hdr, acb->acb_private, - acb->acb_compressed, zio->io_error == 0, - &acb->acb_buf); - if (error != 0) { - /* - * Decompression failed. Set io_error - * so that when we call acb_done (below), - * we will indicate that the read failed. - * Note that in the unusual case where one - * callback is compressed and another - * uncompressed, we will mark all of them - * as failed, even though the uncompressed - * one can't actually fail. In this case, - * the hdr will not be anonymous, because - * if there are multiple callbacks, it's - * because multiple threads found the same - * arc buf in the hash table. - */ - zio->io_error = error; - } + if (zio->io_error != 0) + continue; + + int error = arc_buf_alloc_impl(hdr, acb->acb_private, + acb->acb_compressed, B_TRUE, &acb->acb_buf); + if (error != 0) { + /* + * Decompression failed. Set io_error + * so that when we call acb_done (below), + * we will indicate that the read failed. + * Note that in the unusual case where one + * callback is compressed and another + * uncompressed, we will mark all of them + * as failed, even though the uncompressed + * one can't actually fail. In this case, + * the hdr will not be anonymous, because + * if there are multiple callbacks, it's + * because multiple threads found the same + * arc buf in the hash table. + */ + zio->io_error = error; } } /* @@ -4873,7 +4883,7 @@ arc_read_done(zio_t *zio) ASSERT(zfs_refcount_is_zero(&hdr->b_l1hdr.b_refcnt) || callback_list != NULL); - if (no_zio_error) { + if (zio->io_error == 0) { arc_hdr_verify(hdr, zio->io_bp); } else { arc_hdr_set_flags(hdr, ARC_FLAG_IO_ERROR); @@ -4916,7 +4926,8 @@ arc_read_done(zio_t *zio) arc_buf_destroy(acb->acb_buf, acb->acb_private); acb->acb_buf = NULL; } - acb->acb_done(zio, acb->acb_buf, acb->acb_private); + acb->acb_done(zio, &zio->io_bookmark, zio->io_bp, + acb->acb_buf, acb->acb_private); } if (acb->acb_zio_dummy != NULL) { @@ -4951,7 +4962,7 @@ arc_read_done(zio_t *zio) * for readers of this block. */ int -arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, arc_done_func_t *done, +arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, arc_read_done_func_t *done, void *private, zio_priority_t priority, int zio_flags, arc_flags_t *arc_flags, const zbookmark_phys_t *zb) { @@ -4960,6 +4971,7 @@ arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, a zio_t *rzio; uint64_t guid = spa_load_guid(spa); boolean_t compressed_read = (zio_flags & ZIO_FLAG_RAW) != 0; + int rc = 0; ASSERT(!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA); @@ -4978,32 +4990,20 @@ top: *arc_flags |= ARC_FLAG_CACHED; if (HDR_IO_IN_PROGRESS(hdr)) { + zio_t *head_zio = hdr->b_l1hdr.b_acb->acb_zio_head; + ASSERT3P(head_zio, !=, NULL); if ((hdr->b_flags & ARC_FLAG_PRIO_ASYNC_READ) && priority == ZIO_PRIORITY_SYNC_READ) { /* - * This sync read must wait for an - * in-progress async read (e.g. a predictive - * prefetch). Async reads are queued - * separately at the vdev_queue layer, so - * this is a form of priority inversion. - * Ideally, we would "inherit" the demand - * i/o's priority by moving the i/o from - * the async queue to the synchronous queue, - * but there is currently no mechanism to do - * so. Track this so that we can evaluate - * the magnitude of this potential performance - * problem. - * - * Note that if the prefetch i/o is already - * active (has been issued to the device), - * the prefetch improved performance, because - * we issued it sooner than we would have - * without the prefetch. + * This is a sync read that needs to wait for + * an in-flight async read. Request that the + * zio have its priority upgraded. */ - DTRACE_PROBE1(arc__sync__wait__for__async, + zio_change_priority(head_zio, priority); + DTRACE_PROBE1(arc__async__upgrade__sync, arc_buf_hdr_t *, hdr); - ARCSTAT_BUMP(arcstat_sync_wait_for_async); + ARCSTAT_BUMP(arcstat_async_upgrade_sync); } if (hdr->b_flags & ARC_FLAG_PREDICTIVE_PREFETCH) { arc_hdr_clear_flags(hdr, @@ -5030,6 +5030,7 @@ top: spa, NULL, NULL, NULL, zio_flags); ASSERT3P(acb->acb_done, !=, NULL); + acb->acb_zio_head = head_zio; acb->acb_next = hdr->b_l1hdr.b_acb; hdr->b_l1hdr.b_acb = acb; mutex_exit(hash_lock); @@ -5057,17 +5058,33 @@ top: arc_hdr_clear_flags(hdr, ARC_FLAG_PREDICTIVE_PREFETCH); } + + if (hdr->b_flags & ARC_FLAG_PRESCIENT_PREFETCH) { + ARCSTAT_BUMP( + arcstat_demand_hit_prescient_prefetch); + arc_hdr_clear_flags(hdr, + ARC_FLAG_PRESCIENT_PREFETCH); + } + ASSERT(!BP_IS_EMBEDDED(bp) || !BP_IS_HOLE(bp)); /* Get a buf with the desired data in it. */ - VERIFY0(arc_buf_alloc_impl(hdr, private, - compressed_read, B_TRUE, &buf)); + rc = arc_buf_alloc_impl(hdr, private, + compressed_read, B_TRUE, &buf); + if (rc != 0) { + arc_buf_destroy(buf, private); + buf = NULL; + } + ASSERT((zio_flags & ZIO_FLAG_SPECULATIVE) || + rc == 0 || rc != ENOENT); } else if (*arc_flags & ARC_FLAG_PREFETCH && zfs_refcount_count(&hdr->b_l1hdr.b_refcnt) == 0) { arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); } DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); arc_access(hdr, hash_lock); + if (*arc_flags & ARC_FLAG_PRESCIENT_PREFETCH) + arc_hdr_set_flags(hdr, ARC_FLAG_PRESCIENT_PREFETCH); if (*arc_flags & ARC_FLAG_L2CACHE) arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); mutex_exit(hash_lock); @@ -5077,7 +5094,7 @@ top: data, metadata, hits); if (done) - done(NULL, buf, private); + done(NULL, zb, bp, buf, private); } else { uint64_t lsize = BP_GET_LSIZE(bp); uint64_t psize = BP_GET_PSIZE(bp); @@ -5151,6 +5168,9 @@ top: if (*arc_flags & ARC_FLAG_PREFETCH) arc_hdr_set_flags(hdr, ARC_FLAG_PREFETCH); + if (*arc_flags & ARC_FLAG_PRESCIENT_PREFETCH) + arc_hdr_set_flags(hdr, ARC_FLAG_PRESCIENT_PREFETCH); + if (*arc_flags & ARC_FLAG_L2CACHE) arc_hdr_set_flags(hdr, ARC_FLAG_L2CACHE); if (BP_GET_LEVEL(bp) > 0) @@ -5180,14 +5200,17 @@ top: vd = NULL; } - if (priority == ZIO_PRIORITY_ASYNC_READ) + /* + * We count both async reads and scrub IOs as asynchronous so + * that both can be upgraded in the event of a cache hit while + * the read IO is still in-flight. + */ + if (priority == ZIO_PRIORITY_ASYNC_READ || + priority == ZIO_PRIORITY_SCRUB) arc_hdr_set_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); else arc_hdr_clear_flags(hdr, ARC_FLAG_PRIO_ASYNC_READ); - if (hash_lock != NULL) - mutex_exit(hash_lock); - /* * At this point, we have a level 1 cache miss. Try again in * L2ARC if possible. @@ -5257,6 +5280,11 @@ top: ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY, B_FALSE); + acb->acb_zio_head = rzio; + + if (hash_lock != NULL) + mutex_exit(hash_lock); + DTRACE_PROBE2(l2arc__read, vdev_t *, vd, zio_t *, rzio); ARCSTAT_INCR(arcstat_l2_read_bytes, size); @@ -5271,6 +5299,8 @@ top: return (0); /* l2arc read error; goto zio_read() */ + if (hash_lock != NULL) + mutex_enter(hash_lock); } else { DTRACE_PROBE1(l2arc__miss, arc_buf_hdr_t *, hdr); @@ -5291,7 +5321,11 @@ top: rzio = zio_read(pio, spa, bp, hdr->b_l1hdr.b_pabd, size, arc_read_done, hdr, priority, zio_flags, zb); + acb->acb_zio_head = rzio; + if (hash_lock != NULL) + mutex_exit(hash_lock); + if (*arc_flags & ARC_FLAG_WAIT) return (zio_wait(rzio)); @@ -5778,9 +5812,9 @@ arc_write_done(zio_t *zio) zio_t * arc_write(zio_t *pio, spa_t *spa, uint64_t txg, blkptr_t *bp, arc_buf_t *buf, - boolean_t l2arc, const zio_prop_t *zp, arc_done_func_t *ready, - arc_done_func_t *children_ready, arc_done_func_t *physdone, - arc_done_func_t *done, void *private, zio_priority_t priority, + boolean_t l2arc, const zio_prop_t *zp, arc_write_done_func_t *ready, + arc_write_done_func_t *children_ready, arc_write_done_func_t *physdone, + arc_write_done_func_t *done, void *private, zio_priority_t priority, int zio_flags, const zbookmark_phys_t *zb) { arc_buf_hdr_t *hdr = buf->b_hdr; @@ -6191,9 +6225,6 @@ arc_init(void) #endif mutex_init(&arc_adjust_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&arc_adjust_waiters_cv, NULL, CV_DEFAULT, NULL); - - /* Convert seconds to clock ticks */ - arc_min_prefetch_lifespan = 1 * hz; /* set min cache to 1/32 of all memory, or 64MB, whichever is more */ arc_c_min = MAX(allmem / 32, 64 << 20); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Thu Nov 21 13:46:16 2019 (r354949) @@ -942,7 +942,8 @@ dbuf_whichblock(dnode_t *dn, int64_t level, uint64_t o } static void -dbuf_read_done(zio_t *zio, arc_buf_t *buf, void *vdb) +dbuf_read_done(zio_t *zio, const zbookmark_phys_t *zb, const blkptr_t *bp, + arc_buf_t *buf, void *vdb) { dmu_buf_impl_t *db = vdb; @@ -961,15 +962,19 @@ dbuf_read_done(zio_t *zio, arc_buf_t *buf, void *vdb) ASSERT3P(db->db_buf, ==, NULL); db->db_state = DB_UNCACHED; } else if (db->db_level == 0 && db->db_freed_in_flight) { - /* freed in flight */ + /* we were freed in flight; disregard any error */ ASSERT(zio == NULL || zio->io_error == 0); + if (buf == NULL) { + buf = arc_alloc_buf(db->db_objset->os_spa, + db, DBUF_GET_BUFC_TYPE(db), db->db.db_size); + } arc_release(buf, db); bzero(buf->b_data, db->db.db_size); arc_buf_freeze(buf); db->db_freed_in_flight = FALSE; dbuf_set_data(db, buf); db->db_state = DB_CACHED; - } else { + } else if (buf != NULL) { /* success */ ASSERT(zio == NULL || zio->io_error == 0); dbuf_set_data(db, buf); @@ -2395,7 +2400,8 @@ dbuf_issue_final_prefetch(dbuf_prefetch_arg_t *dpa, bl * prefetch if the next block down is our target. */ static void -dbuf_prefetch_indirect_done(zio_t *zio, arc_buf_t *abuf, void *private) +dbuf_prefetch_indirect_done(zio_t *zio, const zbookmark_phys_t *zb, + const blkptr_t *iobp, arc_buf_t *abuf, void *private) { dbuf_prefetch_arg_t *dpa = private; @@ -2442,11 +2448,11 @@ dbuf_prefetch_indirect_done(zio_t *zio, arc_buf_t *abu } dpa->dpa_curlevel--; - uint64_t nextblkid = dpa->dpa_zb.zb_blkid >> (dpa->dpa_epbs * (dpa->dpa_curlevel - dpa->dpa_zb.zb_level)); blkptr_t *bp = ((blkptr_t *)abuf->b_data) + P2PHASE(nextblkid, 1ULL << dpa->dpa_epbs); + if (BP_IS_HOLE(bp)) { kmem_free(dpa, sizeof (*dpa)); } else if (dpa->dpa_curlevel == dpa->dpa_zb.zb_level) { @@ -3852,7 +3858,7 @@ dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, d * ready callback so that we can properly handle an indirect * block that only contains holes. */ - arc_done_func_t *children_ready_cb = NULL; + arc_write_done_func_t *children_ready_cb = NULL; if (db->db_level != 0) children_ready_cb = dbuf_write_children_ready; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Thu Nov 21 13:46:16 2019 (r354949) @@ -1106,14 +1106,26 @@ ddt_sync_table(ddt_t *ddt, dmu_tx_t *tx, uint64_t txg) void ddt_sync(spa_t *spa, uint64_t txg) { + dsl_scan_t *scn = spa->spa_dsl_pool->dp_scan; dmu_tx_t *tx; - zio_t *rio = zio_root(spa, NULL, NULL, - ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE | ZIO_FLAG_SELF_HEAL); + zio_t *rio; ASSERT(spa_syncing_txg(spa) == txg); tx = dmu_tx_create_assigned(spa->spa_dsl_pool, txg); + rio = zio_root(spa, NULL, NULL, + ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE | ZIO_FLAG_SELF_HEAL); + + /* + * This function may cause an immediate scan of ddt blocks (see + * the comment above dsl_scan_ddt() for details). We set the + * scan's root zio here so that we can wait for any scan IOs in + * addition to the regular ddt IOs. + */ + ASSERT3P(scn->scn_zio_root, ==, NULL); + scn->scn_zio_root = rio; + for (enum zio_checksum c = 0; c < ZIO_CHECKSUM_FUNCTIONS; c++) { ddt_t *ddt = spa->spa_ddt[c]; if (ddt == NULL) @@ -1123,6 +1135,7 @@ ddt_sync(spa_t *spa, uint64_t txg) } (void) zio_wait(rio); + scn->scn_zio_root = NULL; dmu_tx_commit(tx); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Thu Nov 21 13:46:16 2019 (r354949) @@ -398,6 +398,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl ASSERT(ds == NULL || MUTEX_HELD(&ds->ds_opening_lock)); +#if 0 /* * The $ORIGIN dataset (if it exists) doesn't have an associated * objset, so there's no reason to open it. The $ORIGIN dataset @@ -408,6 +409,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl ASSERT3P(ds->ds_dir, !=, spa_get_dsl(spa)->dp_origin_snap->ds_dir); } +#endif os = kmem_zalloc(sizeof (objset_t), KM_SLEEP); os->os_dsl_dataset = ds; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Thu Nov 21 13:46:16 2019 (r354949) @@ -492,7 +492,8 @@ traverse_prefetcher(spa_t *spa, zilog_t *zilog, const const zbookmark_phys_t *zb, const dnode_phys_t *dnp, void *arg) { prefetch_data_t *pfd = arg; - arc_flags_t aflags = ARC_FLAG_NOWAIT | ARC_FLAG_PREFETCH; + arc_flags_t aflags = ARC_FLAG_NOWAIT | ARC_FLAG_PREFETCH | + ARC_FLAG_PRESCIENT_PREFETCH; ASSERT(pfd->pd_bytes_fetched >= 0); if (bp == NULL) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 13:46:16 2019 (r354949) @@ -51,34 +51,157 @@ #include #include #include +#include #ifdef _KERNEL #include #endif +/* + * Grand theory statement on scan queue sorting + * + * Scanning is implemented by recursively traversing all indirection levels + * in an object and reading all blocks referenced from said objects. This + * results in us approximately traversing the object from lowest logical + * offset to the highest. For best performance, we would want the logical + * blocks to be physically contiguous. However, this is frequently not the + * case with pools given the allocation patterns of copy-on-write filesystems. + * So instead, we put the I/Os into a reordering queue and issue them in a + * way that will most benefit physical disks (LBA-order). + * + * Queue management: + * + * Ideally, we would want to scan all metadata and queue up all block I/O + * prior to starting to issue it, because that allows us to do an optimal + * sorting job. This can however consume large amounts of memory. Therefore + * we continuously monitor the size of the queues and constrain them to 5% + * (zfs_scan_mem_lim_fact) of physmem. If the queues grow larger than this + * limit, we clear out a few of the largest extents at the head of the queues + * to make room for more scanning. Hopefully, these extents will be fairly + * large and contiguous, allowing us to approach sequential I/O throughput + * even without a fully sorted tree. + * + * Metadata scanning takes place in dsl_scan_visit(), which is called from + * dsl_scan_sync() every spa_sync(). If we have either fully scanned all + * metadata on the pool, or we need to make room in memory because our + * queues are too large, dsl_scan_visit() is postponed and + * scan_io_queues_run() is called from dsl_scan_sync() instead. This implies + * that metadata scanning and queued I/O issuing are mutually exclusive. This + * allows us to provide maximum sequential I/O throughput for the majority of + * I/O's issued since sequential I/O performance is significantly negatively + * impacted if it is interleaved with random I/O. + * + * Implementation Notes + * + * One side effect of the queued scanning algorithm is that the scanning code + * needs to be notified whenever a block is freed. This is needed to allow + * the scanning code to remove these I/Os from the issuing queue. Additionally, + * we do not attempt to queue gang blocks to be issued sequentially since this + * is very hard to do and would have an extremely limited performance benefit. + * Instead, we simply issue gang I/Os as soon as we find them using the legacy + * algorithm. + * + * Backwards compatibility + * + * This new algorithm is backwards compatible with the legacy on-disk data + * structures (and therefore does not require a new feature flag). + * Periodically during scanning (see zfs_scan_checkpoint_intval), the scan + * will stop scanning metadata (in logical order) and wait for all outstanding + * sorted I/O to complete. Once this is done, we write out a checkpoint + * bookmark, indicating that we have scanned everything logically before it. + * If the pool is imported on a machine without the new sorting algorithm, + * the scan simply resumes from the last checkpoint using the legacy algorithm. + */ + typedef int (scan_cb_t)(dsl_pool_t *, const blkptr_t *, const zbookmark_phys_t *); static scan_cb_t dsl_scan_scrub_cb; -static void dsl_scan_cancel_sync(void *, dmu_tx_t *); -static void dsl_scan_sync_state(dsl_scan_t *, dmu_tx_t *); -static boolean_t dsl_scan_restarting(dsl_scan_t *, dmu_tx_t *); +static int scan_ds_queue_compare(const void *a, const void *b); +static int scan_prefetch_queue_compare(const void *a, const void *b); +static void scan_ds_queue_clear(dsl_scan_t *scn); +static boolean_t scan_ds_queue_contains(dsl_scan_t *scn, uint64_t dsobj, + uint64_t *txg); +static void scan_ds_queue_insert(dsl_scan_t *scn, uint64_t dsobj, uint64_t txg); +static void scan_ds_queue_remove(dsl_scan_t *scn, uint64_t dsobj); +static void scan_ds_queue_sync(dsl_scan_t *scn, dmu_tx_t *tx); + +extern int zfs_vdev_async_write_active_min_dirty_percent; + +/* + * By default zfs will check to ensure it is not over the hard memory + * limit before each txg. If finer-grained control of this is needed + * this value can be set to 1 to enable checking before scanning each + * block. + */ +int zfs_scan_strict_mem_lim = B_FALSE; + +/* + * Maximum number of parallelly executing I/Os per top-level vdev. + * Tune with care. Very high settings (hundreds) are known to trigger + * some firmware bugs and resets on certain SSDs. + */ int zfs_top_maxinflight = 32; /* maximum I/Os per top-level */ -int zfs_resilver_delay = 2; /* number of ticks to delay resilver */ -int zfs_scrub_delay = 4; /* number of ticks to delay scrub */ -int zfs_scan_idle = 50; /* idle window in clock ticks */ +unsigned int zfs_resilver_delay = 2; /* number of ticks to delay resilver */ +unsigned int zfs_scrub_delay = 4; /* number of ticks to delay scrub */ +unsigned int zfs_scan_idle = 50; /* idle window in clock ticks */ -int zfs_scan_min_time_ms = 1000; /* min millisecs to scrub per txg */ -int zfs_free_min_time_ms = 1000; /* min millisecs to free per txg */ -int zfs_obsolete_min_time_ms = 500; /* min millisecs to obsolete per txg */ -int zfs_resilver_min_time_ms = 3000; /* min millisecs to resilver per txg */ +/* + * Maximum number of parallelly executed bytes per leaf vdev. We attempt + * to strike a balance here between keeping the vdev queues full of I/Os + * at all times and not overflowing the queues to cause long latency, + * which would cause long txg sync times. No matter what, we will not + * overload the drives with I/O, since that is protected by + * zfs_vdev_scrub_max_active. + */ +unsigned long zfs_scan_vdev_limit = 4 << 20; + +int zfs_scan_issue_strategy = 0; +int zfs_scan_legacy = B_FALSE; /* don't queue & sort zios, go direct */ +uint64_t zfs_scan_max_ext_gap = 2 << 20; /* in bytes */ + +unsigned int zfs_scan_checkpoint_intval = 7200; /* seconds */ +#define ZFS_SCAN_CHECKPOINT_INTVAL SEC_TO_TICK(zfs_scan_checkpoint_intval) + +/* + * fill_weight is non-tunable at runtime, so we copy it at module init from + * zfs_scan_fill_weight. Runtime adjustments to zfs_scan_fill_weight would + * break queue sorting. + */ +uint64_t zfs_scan_fill_weight = 3; +static uint64_t fill_weight; + +/* See dsl_scan_should_clear() for details on the memory limit tunables */ +uint64_t zfs_scan_mem_lim_min = 16 << 20; /* bytes */ +uint64_t zfs_scan_mem_lim_soft_max = 128 << 20; /* bytes */ +int zfs_scan_mem_lim_fact = 20; /* fraction of physmem */ +int zfs_scan_mem_lim_soft_fact = 20; /* fraction of mem lim above */ + +unsigned int zfs_scrub_min_time_ms = 1000; /* min millisecs to scrub per txg */ +unsigned int zfs_free_min_time_ms = 1000; /* min millisecs to free per txg */ +/* min millisecs to obsolete per txg */ +unsigned int zfs_obsolete_min_time_ms = 500; +/* min millisecs to resilver per txg */ +unsigned int zfs_resilver_min_time_ms = 3000; boolean_t zfs_no_scrub_io = B_FALSE; /* set to disable scrub i/o */ boolean_t zfs_no_scrub_prefetch = B_FALSE; /* set to disable scrub prefetch */ enum ddt_class zfs_scrub_ddt_class_max = DDT_CLASS_DUPLICATE; -int dsl_scan_delay_completion = B_FALSE; /* set to delay scan completion */ /* max number of blocks to free in a single TXG */ uint64_t zfs_async_block_max_blocks = UINT64_MAX; +/* + * We wait a few txgs after importing a pool to begin scanning so that + * the import / mounting code isn't held up by scrub / resilver IO. + * Unfortunately, it is a bit difficult to determine exactly how long + * this will take since userspace will trigger fs mounts asynchronously + * and the kernel will create zvol minors asynchronously. As a result, + * the value provided here is a bit arbitrary, but represents a + * reasonable estimate of how many txgs it will take to finish fully + * importing a pool + */ +#define SCAN_IMPORT_WAIT_TXGS 5 + + #define DSL_SCAN_IS_SCRUB_RESILVER(scn) \ ((scn)->scn_phys.scn_func == POOL_SCAN_SCRUB || \ (scn)->scn_phys.scn_func == POOL_SCAN_RESILVER) @@ -97,6 +220,163 @@ static scan_cb_t *scan_funcs[POOL_SCAN_FUNCS] = { dsl_scan_scrub_cb, /* POOL_SCAN_RESILVER */ }; +/* In core node for the scn->scn_queue. Represents a dataset to be scanned */ +typedef struct { + uint64_t sds_dsobj; + uint64_t sds_txg; + avl_node_t sds_node; +} scan_ds_t; + +/* + * This controls what conditions are placed on dsl_scan_sync_state(): + * SYNC_OPTIONAL) write out scn_phys iff scn_bytes_pending == 0 + * SYNC_MANDATORY) write out scn_phys always. scn_bytes_pending must be 0. + * SYNC_CACHED) if scn_bytes_pending == 0, write out scn_phys. Otherwise + * write out the scn_phys_cached version. + * See dsl_scan_sync_state for details. + */ +typedef enum { + SYNC_OPTIONAL, + SYNC_MANDATORY, + SYNC_CACHED +} state_sync_type_t; + +/* + * This struct represents the minimum information needed to reconstruct a + * zio for sequential scanning. This is useful because many of these will + * accumulate in the sequential IO queues before being issued, so saving + * memory matters here. + */ +typedef struct scan_io { + /* fields from blkptr_t */ + uint64_t sio_offset; + uint64_t sio_blk_prop; + uint64_t sio_phys_birth; + uint64_t sio_birth; + zio_cksum_t sio_cksum; + uint32_t sio_asize; + + /* fields from zio_t */ + int sio_flags; + zbookmark_phys_t sio_zb; + + /* members for queue sorting */ + union { + avl_node_t sio_addr_node; /* link into issueing queue */ + list_node_t sio_list_node; /* link for issuing to disk */ + } sio_nodes; +} scan_io_t; + +struct dsl_scan_io_queue { + dsl_scan_t *q_scn; /* associated dsl_scan_t */ + vdev_t *q_vd; /* top-level vdev that this queue represents */ + + /* trees used for sorting I/Os and extents of I/Os */ + range_tree_t *q_exts_by_addr; + avl_tree_t q_exts_by_size; + avl_tree_t q_sios_by_addr; + + /* members for zio rate limiting */ + uint64_t q_maxinflight_bytes; + uint64_t q_inflight_bytes; + kcondvar_t q_zio_cv; /* used under vd->vdev_scan_io_queue_lock */ + + /* per txg statistics */ + uint64_t q_total_seg_size_this_txg; + uint64_t q_segs_this_txg; + uint64_t q_total_zio_size_this_txg; + uint64_t q_zios_this_txg; +}; + +/* private data for dsl_scan_prefetch_cb() */ +typedef struct scan_prefetch_ctx { + zfs_refcount_t spc_refcnt; /* refcount for memory management */ + dsl_scan_t *spc_scn; /* dsl_scan_t for the pool */ + boolean_t spc_root; /* is this prefetch for an objset? */ + uint8_t spc_indblkshift; /* dn_indblkshift of current dnode */ + uint16_t spc_datablkszsec; /* dn_idatablkszsec of current dnode */ +} scan_prefetch_ctx_t; + +/* private data for dsl_scan_prefetch() */ +typedef struct scan_prefetch_issue_ctx { + avl_node_t spic_avl_node; /* link into scn->scn_prefetch_queue */ + scan_prefetch_ctx_t *spic_spc; /* spc for the callback */ + blkptr_t spic_bp; /* bp to prefetch */ + zbookmark_phys_t spic_zb; /* bookmark to prefetch */ +} scan_prefetch_issue_ctx_t; + +static void scan_exec_io(dsl_pool_t *dp, const blkptr_t *bp, int zio_flags, + const zbookmark_phys_t *zb, dsl_scan_io_queue_t *queue); +static void scan_io_queue_insert_impl(dsl_scan_io_queue_t *queue, + scan_io_t *sio); + +static dsl_scan_io_queue_t *scan_io_queue_create(vdev_t *vd); +static void scan_io_queues_destroy(dsl_scan_t *scn); + +static kmem_cache_t *sio_cache; + +void +scan_init(void) +{ + /* + * This is used in ext_size_compare() to weight segments + * based on how sparse they are. This cannot be changed + * mid-scan and the tree comparison functions don't currently + * have a mechansim for passing additional context to the + * compare functions. Thus we store this value globally and + * we only allow it to be set at module intiailization time + */ + fill_weight = zfs_scan_fill_weight; + + sio_cache = kmem_cache_create("sio_cache", + sizeof (scan_io_t), 0, NULL, NULL, NULL, NULL, NULL, 0); +} + +void +scan_fini(void) +{ + kmem_cache_destroy(sio_cache); +} + +static inline boolean_t +dsl_scan_is_running(const dsl_scan_t *scn) +{ + return (scn->scn_phys.scn_state == DSS_SCANNING); +} + +boolean_t +dsl_scan_resilvering(dsl_pool_t *dp) +{ + return (dsl_scan_is_running(dp->dp_scan) && + dp->dp_scan->scn_phys.scn_func == POOL_SCAN_RESILVER); +} + +static inline void +sio2bp(const scan_io_t *sio, blkptr_t *bp, uint64_t vdev_id) +{ + bzero(bp, sizeof (*bp)); + DVA_SET_ASIZE(&bp->blk_dva[0], sio->sio_asize); + DVA_SET_VDEV(&bp->blk_dva[0], vdev_id); + DVA_SET_OFFSET(&bp->blk_dva[0], sio->sio_offset); + bp->blk_prop = sio->sio_blk_prop; + bp->blk_phys_birth = sio->sio_phys_birth; + bp->blk_birth = sio->sio_birth; + bp->blk_fill = 1; /* we always only work with data pointers */ + bp->blk_cksum = sio->sio_cksum; +} + +static inline void +bp2sio(const blkptr_t *bp, scan_io_t *sio, int dva_i) +{ + /* we discard the vdev id, since we can deduce it from the queue */ + sio->sio_offset = DVA_GET_OFFSET(&bp->blk_dva[dva_i]); + sio->sio_asize = DVA_GET_ASIZE(&bp->blk_dva[dva_i]); + sio->sio_blk_prop = bp->blk_prop; + sio->sio_phys_birth = bp->blk_phys_birth; + sio->sio_birth = bp->blk_birth; + sio->sio_cksum = bp->blk_cksum; +} + int dsl_scan_init(dsl_pool_t *dp, uint64_t txg) { @@ -117,6 +397,13 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) scn->scn_async_destroying = spa_feature_is_active(dp->dp_spa, SPA_FEATURE_ASYNC_DESTROY); + bcopy(&scn->scn_phys, &scn->scn_phys_cached, sizeof (scn->scn_phys)); + avl_create(&scn->scn_queue, scan_ds_queue_compare, sizeof (scan_ds_t), + offsetof(scan_ds_t, sds_node)); + avl_create(&scn->scn_prefetch_queue, scan_prefetch_queue_compare, + sizeof (scan_prefetch_issue_ctx_t), + offsetof(scan_prefetch_issue_ctx_t, spic_avl_node)); + err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, "scrub_func", sizeof (uint64_t), 1, &f); if (err == 0) { @@ -127,7 +414,7 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) scn->scn_restart_txg = txg; zfs_dbgmsg("old-style scrub was in progress; " "restarting new-style scrub in txg %llu", - scn->scn_restart_txg); + (longlong_t)scn->scn_restart_txg); /* * Load the queue obj from the old location so that it @@ -145,7 +432,14 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) else if (err) return (err); - if (scn->scn_phys.scn_state == DSS_SCANNING && + /* + * We might be restarting after a reboot, so jump the issued + * counter to how far we've scanned. We know we're consistent + * up to here. + */ + scn->scn_issued_before_pass = scn->scn_phys.scn_examined; + + if (dsl_scan_is_running(scn) && spa_prev_software_version(dp->dp_spa) < SPA_VERSION_SCAN) { /* * A new-type scrub was in progress on an old @@ -157,10 +451,26 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) scn->scn_restart_txg = txg; zfs_dbgmsg("new-style scrub was modified " "by old software; restarting in txg %llu", - scn->scn_restart_txg); + (longlong_t)scn->scn_restart_txg); } *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Nov 21 13:46:19 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D621F1BF174; Thu, 21 Nov 2019 13:46:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jgnl5GgKz4Pdg; Thu, 21 Nov 2019 13:46:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 80FFCC27C; Thu, 21 Nov 2019 13:46:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALDkJRJ035776; Thu, 21 Nov 2019 13:46:19 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALDkI5P035772; Thu, 21 Nov 2019 13:46:18 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211346.xALDkI5P035772@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 13:46:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354949 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor... X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb ven... X-SVN-Commit-Revision: 354949 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 13:46:19 -0000 Author: avg Date: Thu Nov 21 13:46:16 2019 New Revision: 354949 URL: https://svnweb.freebsd.org/changeset/base/354949 Log: 10405 Implement ZFS sorted scans illumos/illumos-gate@a3874b8b1fe5103fc1f961609557c0587435fec0 https://github.com/illumos/illumos-gate/commit/a3874b8b1fe5103fc1f961609557c0587435fec0 https://www.illumos.org/issues/10405 The original implementation is: https://github.com/zfsonlinux/zfs/commit/d4a72f23863382bdf6d0ae33196f5b5decbc48fd Author: Toomas Soome Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_status.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_scan.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h vendor-sys/illumos/dist/uts/common/sys/taskq.h Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor/illumos/dist/cmd/zdb/zdb.c Thu Nov 21 13:46:16 2019 (r354949) @@ -2384,8 +2384,6 @@ dump_dir(objset_t *os) max_slot_used = object + dnode_slots - 1; } - ASSERT3U(object_count, ==, usedobjs); - (void) printf("\n"); (void) printf(" Dnode slots:\n"); @@ -2408,6 +2406,8 @@ dump_dir(objset_t *os) leaked_objects); leaked_objects = 0; } + + ASSERT3U(object_count, ==, usedobjs); } static void @@ -2975,7 +2975,7 @@ zdb_blkptr_done(zio_t *zio) abd_free(zio->io_abd); mutex_enter(&spa->spa_scrub_lock); - spa->spa_scrub_inflight--; + spa->spa_load_verify_ios--; cv_broadcast(&spa->spa_scrub_io_cv); if (ioerr && !(zio->io_flags & ZIO_FLAG_SPECULATIVE)) { @@ -3046,9 +3046,9 @@ zdb_blkptr_cb(spa_t *spa, zilog_t *zilog, const blkptr flags |= ZIO_FLAG_SPECULATIVE; mutex_enter(&spa->spa_scrub_lock); - while (spa->spa_scrub_inflight > max_inflight) + while (spa->spa_load_verify_ios > max_inflight) cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock); - spa->spa_scrub_inflight++; + spa->spa_load_verify_ios++; mutex_exit(&spa->spa_scrub_lock); zio_nowait(zio_read(NULL, spa, bp, abd, size, Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 13:46:16 2019 (r354949) @@ -1688,7 +1688,7 @@ print_status_config(zpool_handle_t *zhp, status_cbdata (void) nvlist_lookup_uint64_array(nv, ZPOOL_CONFIG_SCAN_STATS, (uint64_t **)&ps, &c); - if (ps && ps->pss_state == DSS_SCANNING && + if (ps != NULL && ps->pss_state == DSS_SCANNING && vs->vs_scan_processed != 0 && children == 0) { (void) printf(gettext(" (%s)"), (ps->pss_func == POOL_SCAN_RESILVER) ? @@ -4707,11 +4707,13 @@ static void print_scan_status(pool_scan_stat_t *ps) { time_t start, end, pause; - uint64_t elapsed, mins_left, hours_left; - uint64_t pass_exam, examined, total; - uint_t rate; + uint64_t total_secs_left; + uint64_t elapsed, secs_left, mins_left, hours_left, days_left; + uint64_t pass_scanned, scanned, pass_issued, issued, total; + uint_t scan_rate, issue_rate; double fraction_done; - char processed_buf[7], examined_buf[7], total_buf[7], rate_buf[7]; + char processed_buf[7], scanned_buf[7], issued_buf[7], total_buf[7]; + char srate_buf[7], irate_buf[7]; (void) printf(gettext(" scan: ")); @@ -4725,30 +4727,37 @@ print_scan_status(pool_scan_stat_t *ps) start = ps->pss_start_time; end = ps->pss_end_time; pause = ps->pss_pass_scrub_pause; + zfs_nicenum(ps->pss_processed, processed_buf, sizeof (processed_buf)); assert(ps->pss_func == POOL_SCAN_SCRUB || ps->pss_func == POOL_SCAN_RESILVER); + /* * Scan is finished or canceled. */ if (ps->pss_state == DSS_FINISHED) { - uint64_t minutes_taken = (end - start) / 60; - char *fmt = NULL; + total_secs_left = end - start; + days_left = total_secs_left / 60 / 60 / 24; + hours_left = (total_secs_left / 60 / 60) % 24; + mins_left = (total_secs_left / 60) % 60; + secs_left = (total_secs_left % 60); if (ps->pss_func == POOL_SCAN_SCRUB) { - fmt = gettext("scrub repaired %s in %lluh%um with " - "%llu errors on %s"); + (void) printf(gettext("scrub repaired %s " + "in %llu days %02llu:%02llu:%02llu " + "with %llu errors on %s"), processed_buf, + (u_longlong_t)days_left, (u_longlong_t)hours_left, + (u_longlong_t)mins_left, (u_longlong_t)secs_left, + (u_longlong_t)ps->pss_errors, ctime(&end)); } else if (ps->pss_func == POOL_SCAN_RESILVER) { - fmt = gettext("resilvered %s in %lluh%um with " - "%llu errors on %s"); + (void) printf(gettext("resilvered %s " + "in %llu days %02llu:%02llu:%02llu " + "with %llu errors on %s"), processed_buf, + (u_longlong_t)days_left, (u_longlong_t)hours_left, + (u_longlong_t)mins_left, (u_longlong_t)secs_left, + (u_longlong_t)ps->pss_errors, ctime(&end)); } - /* LINTED */ - (void) printf(fmt, processed_buf, - (u_longlong_t)(minutes_taken / 60), - (uint_t)(minutes_taken % 60), - (u_longlong_t)ps->pss_errors, - ctime((time_t *)&end)); return; } else if (ps->pss_state == DSS_CANCELED) { if (ps->pss_func == POOL_SCAN_SCRUB) { @@ -4763,19 +4772,15 @@ print_scan_status(pool_scan_stat_t *ps) assert(ps->pss_state == DSS_SCANNING); - /* - * Scan is in progress. - */ + /* Scan is in progress. Resilvers can't be paused. */ if (ps->pss_func == POOL_SCAN_SCRUB) { if (pause == 0) { (void) printf(gettext("scrub in progress since %s"), ctime(&start)); } else { - char buf[32]; - struct tm *p = localtime(&pause); - (void) strftime(buf, sizeof (buf), "%a %b %e %T %Y", p); - (void) printf(gettext("scrub paused since %s\n"), buf); - (void) printf(gettext("\tscrub started on %s"), + (void) printf(gettext("scrub paused since %s"), + ctime(&pause)); + (void) printf(gettext("\tscrub started on %s"), ctime(&start)); } } else if (ps->pss_func == POOL_SCAN_RESILVER) { @@ -4783,49 +4788,67 @@ print_scan_status(pool_scan_stat_t *ps) ctime(&start)); } - examined = ps->pss_examined ? ps->pss_examined : 1; + scanned = ps->pss_examined; + pass_scanned = ps->pss_pass_exam; + issued = ps->pss_issued; + pass_issued = ps->pss_pass_issued; total = ps->pss_to_examine; - fraction_done = (double)examined / total; - /* elapsed time for this pass */ + /* we are only done with a block once we have issued the IO for it */ + fraction_done = (double)issued / total; + + /* elapsed time for this pass, rounding up to 1 if it's 0 */ elapsed = time(NULL) - ps->pss_pass_start; elapsed -= ps->pss_pass_scrub_spent_paused; - elapsed = elapsed ? elapsed : 1; - pass_exam = ps->pss_pass_exam ? ps->pss_pass_exam : 1; - rate = pass_exam / elapsed; - rate = rate ? rate : 1; - mins_left = ((total - examined) / rate) / 60; - hours_left = mins_left / 60; + elapsed = (elapsed != 0) ? elapsed : 1; - zfs_nicenum(examined, examined_buf, sizeof (examined_buf)); + scan_rate = pass_scanned / elapsed; + issue_rate = pass_issued / elapsed; + total_secs_left = (issue_rate != 0) ? + ((total - issued) / issue_rate) : UINT64_MAX; + + days_left = total_secs_left / 60 / 60 / 24; + hours_left = (total_secs_left / 60 / 60) % 24; + mins_left = (total_secs_left / 60) % 60; + secs_left = (total_secs_left % 60); + + /* format all of the numbers we will be reporting */ + zfs_nicenum(scanned, scanned_buf, sizeof (scanned_buf)); + zfs_nicenum(issued, issued_buf, sizeof (issued_buf)); zfs_nicenum(total, total_buf, sizeof (total_buf)); + zfs_nicenum(scan_rate, srate_buf, sizeof (srate_buf)); + zfs_nicenum(issue_rate, irate_buf, sizeof (irate_buf)); - /* - * do not print estimated time if hours_left is more than 30 days - * or we have a paused scrub - */ + /* do not print estimated time if we have a paused scrub */ if (pause == 0) { - zfs_nicenum(rate, rate_buf, sizeof (rate_buf)); - (void) printf(gettext("\t%s scanned out of %s at %s/s"), - examined_buf, total_buf, rate_buf); - if (hours_left < (30 * 24)) { - (void) printf(gettext(", %lluh%um to go\n"), - (u_longlong_t)hours_left, (uint_t)(mins_left % 60)); - } else { - (void) printf(gettext( - ", (scan is slow, no estimated time)\n")); - } + (void) printf(gettext("\t%s scanned at %s/s, " + "%s issued at %s/s, %s total\n"), + scanned_buf, srate_buf, issued_buf, irate_buf, total_buf); } else { - (void) printf(gettext("\t%s scanned out of %s\n"), - examined_buf, total_buf); + (void) printf(gettext("\t%s scanned, %s issued, %s total\n"), + scanned_buf, issued_buf, total_buf); } if (ps->pss_func == POOL_SCAN_RESILVER) { - (void) printf(gettext(" %s resilvered, %.2f%% done\n"), + (void) printf(gettext("\t%s resilvered, %.2f%% done"), processed_buf, 100 * fraction_done); } else if (ps->pss_func == POOL_SCAN_SCRUB) { - (void) printf(gettext(" %s repaired, %.2f%% done\n"), + (void) printf(gettext("\t%s repaired, %.2f%% done"), processed_buf, 100 * fraction_done); + } + + if (pause == 0) { + if (issue_rate >= 10 * 1024 * 1024) { + (void) printf(gettext(", %llu days " + "%02llu:%02llu:%02llu to go\n"), + (u_longlong_t)days_left, (u_longlong_t)hours_left, + (u_longlong_t)mins_left, (u_longlong_t)secs_left); + } else { + (void) printf(gettext(", no estimated " + "completion time\n")); + } + } else { + (void) printf(gettext("\n")); } } Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor/illumos/dist/cmd/ztest/ztest.c Thu Nov 21 13:46:16 2019 (r354949) @@ -397,15 +397,15 @@ ztest_info_t ztest_info[] = { { ztest_fzap, 1, &zopt_sometimes }, { ztest_dmu_snapshot_create_destroy, 1, &zopt_sometimes }, { ztest_spa_create_destroy, 1, &zopt_sometimes }, - { ztest_fault_inject, 1, &zopt_sometimes }, + { ztest_fault_inject, 1, &zopt_incessant }, { ztest_ddt_repair, 1, &zopt_sometimes }, { ztest_dmu_snapshot_hold, 1, &zopt_sometimes }, { ztest_mmp_enable_disable, 1, &zopt_sometimes }, { ztest_reguid, 1, &zopt_rarely }, - { ztest_scrub, 1, &zopt_rarely }, + { ztest_scrub, 1, &zopt_often }, { ztest_spa_upgrade, 1, &zopt_rarely }, { ztest_dsl_dataset_promote_busy, 1, &zopt_rarely }, - { ztest_vdev_attach_detach, 1, &zopt_sometimes }, + { ztest_vdev_attach_detach, 1, &zopt_incessant }, { ztest_vdev_LUN_growth, 1, &zopt_rarely }, { ztest_vdev_add_remove, 1, &ztest_opts.zo_vdevtime }, Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_status.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_status.c Thu Nov 21 13:35:43 2019 (r354948) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_status.c Thu Nov 21 13:46:16 2019 (r354949) @@ -225,7 +225,7 @@ check_status(nvlist_t *config, boolean_t isimport) */ (void) nvlist_lookup_uint64_array(nvroot, ZPOOL_CONFIG_SCAN_STATS, (uint64_t **)&ps, &psc); - if (ps && ps->pss_func == POOL_SCAN_RESILVER && + if (ps != NULL && ps->pss_func == POOL_SCAN_RESILVER && ps->pss_state == DSS_SCANNING) return (ZPOOL_STATUS_RESILVERING); From owner-svn-src-vendor@freebsd.org Thu Nov 21 13:59:08 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8CFA31BFA0D; Thu, 21 Nov 2019 13:59:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jh4X1qtkz4QqL; Thu, 21 Nov 2019 13:59:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CE709C442; Thu, 21 Nov 2019 13:59:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALDx7Me041771; Thu, 21 Nov 2019 13:59:07 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALDx6X7041764; Thu, 21 Nov 2019 13:59:06 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211359.xALDx6X7041764@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 13:59:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354953 - in vendor-sys/illumos/dist/uts/common: fs fs/zfs sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common: fs fs/zfs sys X-SVN-Commit-Revision: 354953 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 13:59:08 -0000 Author: avg Date: Thu Nov 21 13:59:06 2019 New Revision: 354953 URL: https://svnweb.freebsd.org/changeset/base/354953 Log: 10573 define TASKQID_INVALID as (taskq_id)0 illumos/illumos-gate@fc8ae2ec4282de7ec96f48e11078345f3dc0ac3d https://github.com/illumos/illumos-gate/commit/fc8ae2ec4282de7ec96f48e11078345f3dc0ac3d https://www.illumos.org/issues/10573 We do have taskqid_t taskq_dispatch() and the result of this function is compared with 0, NULL and combined with type casts. Define TASKQID_INVALID (taskq_id)0 and use TASKQID_INVALID everywhere. Author: Toomas Soome Modified: vendor-sys/illumos/dist/uts/common/fs/vnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/taskq.h Modified: vendor-sys/illumos/dist/uts/common/fs/vnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/vnode.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/vnode.c Thu Nov 21 13:59:06 2019 (r354953) @@ -915,7 +915,7 @@ vn_rele_async(vnode_t *vp, taskq_t *taskq) if (vp->v_count == 1) { mutex_exit(&vp->v_lock); VERIFY(taskq_dispatch(taskq, (task_func_t *)vn_rele_inactive, - vp, TQ_SLEEP) != NULL); + vp, TQ_SLEEP) != TASKQID_INVALID); return; } VN_RELE_LOCKED(vp); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Thu Nov 21 13:59:06 2019 (r354953) @@ -599,8 +599,8 @@ traverse_impl(spa_t *spa, dsl_dataset_t *ds, uint64_t } if (!(flags & TRAVERSE_PREFETCH_DATA) || - 0 == taskq_dispatch(system_taskq, traverse_prefetch_thread, - &td, TQ_NOQUEUE)) + taskq_dispatch(system_taskq, traverse_prefetch_thread, + &td, TQ_NOQUEUE) == TASKQID_INVALID) pd.pd_exited = B_TRUE; SET_BOOKMARK(&czb, td.td_objset, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Thu Nov 21 13:59:06 2019 (r354953) @@ -2511,7 +2511,7 @@ metaslab_group_preload(metaslab_group_t *mg) } VERIFY(taskq_dispatch(mg->mg_taskq, metaslab_preload, - msp, TQ_SLEEP) != NULL); + msp, TQ_SLEEP) != TASKQID_INVALID); } mutex_exit(&mg->mg_lock); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 13:59:06 2019 (r354953) @@ -1489,7 +1489,7 @@ vdev_open_children(vdev_t *vd) for (int c = 0; c < children; c++) VERIFY(taskq_dispatch(tq, vdev_open_child, vd->vdev_child[c], - TQ_SLEEP) != NULL); + TQ_SLEEP) != TASKQID_INVALID); taskq_destroy(tq); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Thu Nov 21 13:59:06 2019 (r354953) @@ -245,7 +245,7 @@ vdev_file_io_start(zio_t *zio) bp->b_iodone = vdev_file_io_intr; VERIFY3U(taskq_dispatch(system_taskq, vdev_file_io_strategy, bp, - TQ_SLEEP), !=, 0); + TQ_SLEEP), !=, TASKQID_INVALID); } /* ARGSUSED */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Nov 21 13:59:06 2019 (r354953) @@ -1891,7 +1891,8 @@ zil_clean(zilog_t *zilog, uint64_t synced_txg) ASSERT3P(zilog->zl_dmu_pool, !=, NULL); ASSERT3P(zilog->zl_dmu_pool->dp_zil_clean_taskq, !=, NULL); if (taskq_dispatch(zilog->zl_dmu_pool->dp_zil_clean_taskq, - (void (*)(void *))zil_itxg_clean, clean_me, TQ_NOSLEEP) == NULL) + (void (*)(void *))zil_itxg_clean, clean_me, TQ_NOSLEEP) == + TASKQID_INVALID) zil_itxg_clean(clean_me); } Modified: vendor-sys/illumos/dist/uts/common/sys/taskq.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/taskq.h Thu Nov 21 13:59:01 2019 (r354952) +++ vendor-sys/illumos/dist/uts/common/sys/taskq.h Thu Nov 21 13:59:06 2019 (r354953) @@ -60,6 +60,8 @@ struct proc; #define TQ_NOALLOC 0x04 /* cannot allocate memory; may fail */ #define TQ_FRONT 0x08 /* Put task at the front of the queue */ +#define TASKQID_INVALID ((taskqid_t)0) + #ifdef _KERNEL extern taskq_t *system_taskq; From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:00:12 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DE2701BFB21; Thu, 21 Nov 2019 14:00:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jh5m5TCPz4R2v; Thu, 21 Nov 2019 14:00:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9F7EEC44B; Thu, 21 Nov 2019 14:00:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE0Cbi041939; Thu, 21 Nov 2019 14:00:12 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE0AtL041927; Thu, 21 Nov 2019 14:00:10 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211400.xALE0AtL041927@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:00:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354954 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zinject X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zinject X-SVN-Commit-Revision: 354954 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:00:12 -0000 Author: avg Date: Thu Nov 21 14:00:09 2019 New Revision: 354954 URL: https://svnweb.freebsd.org/changeset/base/354954 Log: 10566 Multiple DVA Scrubbing Fix illumos/illumos-gate@12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://github.com/illumos/illumos-gate/commit/12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://www.illumos.org/issues/10566 ZoL PR_8453 Author: Tom Caputi Date: Fri Mar 15 17:14:31 2019 -0400 Multiple DVA Scrubbing Fix Currently, there is an issue in the sequential scrub code which prevents self healing from working in some cases. The scrub code will split up all DVA copies of a bp and issue each of them separately. The problem is that, since each of the DVAs is no longer associated with the others, the self healing code doesn't have the opportunity to repair problems that show up in one of the DVAs with the data from the others. This patch fixes this issue by ensuring that all IOs issued by the sequential scrub code include all DVAs. Initially, only the first DVA of each is attempted. If an issue arises, the IO is retried with all available copies, giving the self healing code a chance to correct the issue. To test this change, this patch also adds the ability for zinject to specify individual DVAs to inject read errors into. We then add a new test case that utilizes this functionality to ensure scrubs and self-healing reads can handle and transparently fix issues with individual copies of blocks. This update is followup on #10405 While attempting to port this update, the following ZoL updates are included: 551905dd4 vdev_mirror: kstat observables for preferred vdev d6c6590c5 vdev_mirror: load balancing fixes 9f500936c FreeBSD r256956: Improve ZFS N-way mirror read performance by Portions contributed by: Toomas Soome Portions contributed by: Jerry Jelinek Author: Tom Caputi Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_inject.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zinject/zinject.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:00:09 2019 (r354954) @@ -249,24 +249,43 @@ typedef enum { */ typedef struct scan_io { /* fields from blkptr_t */ - uint64_t sio_offset; uint64_t sio_blk_prop; uint64_t sio_phys_birth; uint64_t sio_birth; zio_cksum_t sio_cksum; - uint32_t sio_asize; + uint32_t sio_nr_dvas; /* fields from zio_t */ - int sio_flags; + uint32_t sio_flags; zbookmark_phys_t sio_zb; /* members for queue sorting */ union { - avl_node_t sio_addr_node; /* link into issueing queue */ + avl_node_t sio_addr_node; /* link into issuing queue */ list_node_t sio_list_node; /* link for issuing to disk */ } sio_nodes; + + /* + * There may be up to SPA_DVAS_PER_BP DVAs here from the bp, + * depending on how many were in the original bp. Only the + * first DVA is really used for sorting and issuing purposes. + * The other DVAs (if provided) simply exist so that the zio + * layer can find additional copies to repair from in the + * event of an error. This array must go at the end of the + * struct to allow this for the variable number of elements. + */ + dva_t sio_dva[0]; } scan_io_t; +#define SIO_SET_OFFSET(sio, x) DVA_SET_OFFSET(&(sio)->sio_dva[0], x) +#define SIO_SET_ASIZE(sio, x) DVA_SET_ASIZE(&(sio)->sio_dva[0], x) +#define SIO_GET_OFFSET(sio) DVA_GET_OFFSET(&(sio)->sio_dva[0]) +#define SIO_GET_ASIZE(sio) DVA_GET_ASIZE(&(sio)->sio_dva[0]) +#define SIO_GET_END_OFFSET(sio) \ + (SIO_GET_OFFSET(sio) + SIO_GET_ASIZE(sio)) +#define SIO_GET_MUSED(sio) \ + (sizeof (scan_io_t) + ((sio)->sio_nr_dvas * sizeof (dva_t))) + struct dsl_scan_io_queue { dsl_scan_t *q_scn; /* associated dsl_scan_t */ vdev_t *q_vd; /* top-level vdev that this queue represents */ @@ -275,6 +294,7 @@ struct dsl_scan_io_queue { range_tree_t *q_exts_by_addr; avl_tree_t q_exts_by_size; avl_tree_t q_sios_by_addr; + uint64_t q_sio_memused; /* members for zio rate limiting */ uint64_t q_maxinflight_bytes; @@ -313,8 +333,28 @@ static void scan_io_queue_insert_impl(dsl_scan_io_queu static dsl_scan_io_queue_t *scan_io_queue_create(vdev_t *vd); static void scan_io_queues_destroy(dsl_scan_t *scn); -static kmem_cache_t *sio_cache; +static kmem_cache_t *sio_cache[SPA_DVAS_PER_BP]; +/* sio->sio_nr_dvas must be set so we know which cache to free from */ +static void +sio_free(scan_io_t *sio) +{ + ASSERT3U(sio->sio_nr_dvas, >, 0); + ASSERT3U(sio->sio_nr_dvas, <=, SPA_DVAS_PER_BP); + + kmem_cache_free(sio_cache[sio->sio_nr_dvas - 1], sio); +} + +/* It is up to the caller to set sio->sio_nr_dvas for freeing */ +static scan_io_t * +sio_alloc(unsigned short nr_dvas) +{ + ASSERT3U(nr_dvas, >, 0); + ASSERT3U(nr_dvas, <=, SPA_DVAS_PER_BP); + + return (kmem_cache_alloc(sio_cache[nr_dvas - 1], KM_SLEEP)); +} + void scan_init(void) { @@ -328,14 +368,22 @@ scan_init(void) */ fill_weight = zfs_scan_fill_weight; - sio_cache = kmem_cache_create("sio_cache", - sizeof (scan_io_t), 0, NULL, NULL, NULL, NULL, NULL, 0); + for (int i = 0; i < SPA_DVAS_PER_BP; i++) { + char name[36]; + + (void) sprintf(name, "sio_cache_%d", i); + sio_cache[i] = kmem_cache_create(name, + (sizeof (scan_io_t) + ((i + 1) * sizeof (dva_t))), + 0, NULL, NULL, NULL, NULL, NULL, 0); + } } void scan_fini(void) { - kmem_cache_destroy(sio_cache); + for (int i = 0; i < SPA_DVAS_PER_BP; i++) { + kmem_cache_destroy(sio_cache[i]); + } } static inline boolean_t @@ -352,29 +400,39 @@ dsl_scan_resilvering(dsl_pool_t *dp) } static inline void -sio2bp(const scan_io_t *sio, blkptr_t *bp, uint64_t vdev_id) +sio2bp(const scan_io_t *sio, blkptr_t *bp) { bzero(bp, sizeof (*bp)); - DVA_SET_ASIZE(&bp->blk_dva[0], sio->sio_asize); - DVA_SET_VDEV(&bp->blk_dva[0], vdev_id); - DVA_SET_OFFSET(&bp->blk_dva[0], sio->sio_offset); bp->blk_prop = sio->sio_blk_prop; bp->blk_phys_birth = sio->sio_phys_birth; bp->blk_birth = sio->sio_birth; bp->blk_fill = 1; /* we always only work with data pointers */ bp->blk_cksum = sio->sio_cksum; + + ASSERT3U(sio->sio_nr_dvas, >, 0); + ASSERT3U(sio->sio_nr_dvas, <=, SPA_DVAS_PER_BP); + + bcopy(sio->sio_dva, bp->blk_dva, sio->sio_nr_dvas * sizeof (dva_t)); } static inline void bp2sio(const blkptr_t *bp, scan_io_t *sio, int dva_i) { - /* we discard the vdev id, since we can deduce it from the queue */ - sio->sio_offset = DVA_GET_OFFSET(&bp->blk_dva[dva_i]); - sio->sio_asize = DVA_GET_ASIZE(&bp->blk_dva[dva_i]); sio->sio_blk_prop = bp->blk_prop; sio->sio_phys_birth = bp->blk_phys_birth; sio->sio_birth = bp->blk_birth; sio->sio_cksum = bp->blk_cksum; + sio->sio_nr_dvas = BP_GET_NDVAS(bp); + + /* + * Copy the DVAs to the sio. We need all copies of the block so + * that the self healing code can use the alternate copies if the + * first is corrupted. We want the DVA at index dva_i to be first + * in the sio since this is the primary one that we want to issue. + */ + for (int i = 0, j = dva_i; i < sio->sio_nr_dvas; i++, j++) { + sio->sio_dva[i] = bp->blk_dva[j % sio->sio_nr_dvas]; + } } int @@ -1076,11 +1134,9 @@ dsl_scan_should_clear(dsl_scan_t *scn) mutex_enter(&tvd->vdev_scan_io_queue_lock); queue = tvd->vdev_scan_io_queue; if (queue != NULL) { - /* #extents in exts_by_size = # in exts_by_addr */ + /* # extents in exts_by_size = # in exts_by_addr */ mused += avl_numnodes(&queue->q_exts_by_size) * - sizeof (range_seg_t) + - avl_numnodes(&queue->q_sios_by_addr) * - sizeof (scan_io_t); + sizeof (range_seg_t) + queue->q_sio_memused; } mutex_exit(&tvd->vdev_scan_io_queue_lock); } @@ -2546,13 +2602,13 @@ scan_io_queue_issue(dsl_scan_io_queue_t *queue, list_t break; } - sio2bp(sio, &bp, queue->q_vd->vdev_id); - bytes_issued += sio->sio_asize; + sio2bp(sio, &bp); + bytes_issued += SIO_GET_ASIZE(sio); scan_exec_io(scn->scn_dp, &bp, sio->sio_flags, &sio->sio_zb, queue); (void) list_remove_head(io_list); scan_io_queues_update_zio_stats(queue, &bp); - kmem_free(sio, sizeof (*sio)); + sio_free(sio); } atomic_add_64(&scn->scn_bytes_pending, -bytes_issued); @@ -2569,7 +2625,7 @@ scan_io_queue_issue(dsl_scan_io_queue_t *queue, list_t static boolean_t scan_io_queue_gather(dsl_scan_io_queue_t *queue, range_seg_t *rs, list_t *list) { - scan_io_t srch_sio, *sio, *next_sio; + scan_io_t *srch_sio, *sio, *next_sio; avl_index_t idx; uint_t num_sios = 0; int64_t bytes_issued = 0; @@ -2577,24 +2633,30 @@ scan_io_queue_gather(dsl_scan_io_queue_t *queue, range ASSERT(rs != NULL); ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock)); - srch_sio.sio_offset = rs->rs_start; + srch_sio = sio_alloc(1); + srch_sio->sio_nr_dvas = 1; + SIO_SET_OFFSET(srch_sio, rs->rs_start); /* * The exact start of the extent might not contain any matching zios, * so if that's the case, examine the next one in the tree. */ - sio = avl_find(&queue->q_sios_by_addr, &srch_sio, &idx); + sio = avl_find(&queue->q_sios_by_addr, srch_sio, &idx); + sio_free(srch_sio); + if (sio == NULL) sio = avl_nearest(&queue->q_sios_by_addr, idx, AVL_AFTER); - while (sio != NULL && sio->sio_offset < rs->rs_end && num_sios <= 32) { - ASSERT3U(sio->sio_offset, >=, rs->rs_start); - ASSERT3U(sio->sio_offset + sio->sio_asize, <=, rs->rs_end); + while (sio != NULL && + SIO_GET_OFFSET(sio) < rs->rs_end && num_sios <= 32) { + ASSERT3U(SIO_GET_OFFSET(sio), >=, rs->rs_start); + ASSERT3U(SIO_GET_END_OFFSET(sio), <=, rs->rs_end); next_sio = AVL_NEXT(&queue->q_sios_by_addr, sio); avl_remove(&queue->q_sios_by_addr, sio); + queue->q_sio_memused -= SIO_GET_MUSED(sio); - bytes_issued += sio->sio_asize; + bytes_issued += SIO_GET_ASIZE(sio); num_sios++; list_insert_tail(list, sio); sio = next_sio; @@ -2606,11 +2668,11 @@ scan_io_queue_gather(dsl_scan_io_queue_t *queue, range * in the segment we update it to reflect the work we were able to * complete. Otherwise, we remove it from the range tree entirely. */ - if (sio != NULL && sio->sio_offset < rs->rs_end) { + if (sio != NULL && SIO_GET_OFFSET(sio) < rs->rs_end) { range_tree_adjust_fill(queue->q_exts_by_addr, rs, -bytes_issued); range_tree_resize_segment(queue->q_exts_by_addr, rs, - sio->sio_offset, rs->rs_end - sio->sio_offset); + SIO_GET_OFFSET(sio), rs->rs_end - SIO_GET_OFFSET(sio)); return (B_TRUE); } else { @@ -2715,9 +2777,9 @@ scan_io_queues_run_one(void *arg) first_sio = list_head(&sio_list); last_sio = list_tail(&sio_list); - seg_end = last_sio->sio_offset + last_sio->sio_asize; + seg_end = SIO_GET_END_OFFSET(last_sio); if (seg_start == 0) - seg_start = first_sio->sio_offset; + seg_start = SIO_GET_OFFSET(first_sio); /* * Issuing sios can take a long time so drop the @@ -3369,10 +3431,23 @@ count_block(dsl_scan_t *scn, zfs_all_blkstats_t *zab, { int i; - /* update the spa's stats on how many bytes we have issued */ - for (i = 0; i < BP_GET_NDVAS(bp); i++) { + /* + * Update the spa's stats on how many bytes we have issued. + * Sequential scrubs create a zio for each DVA of the bp. Each + * of these will include all DVAs for repair purposes, but the + * zio code will only try the first one unless there is an issue. + * Therefore, we should only count the first DVA for these IOs. + */ + if (scn->scn_is_sorted) { atomic_add_64(&scn->scn_dp->dp_spa->spa_scan_pass_issued, - DVA_GET_ASIZE(&bp->blk_dva[i])); + DVA_GET_ASIZE(&bp->blk_dva[0])); + } else { + spa_t *spa = scn->scn_dp->dp_spa; + + for (i = 0; i < BP_GET_NDVAS(bp); i++) { + atomic_add_64(&spa->spa_scan_pass_issued, + DVA_GET_ASIZE(&bp->blk_dva[i])); + } } /* @@ -3426,7 +3501,7 @@ static void scan_io_queue_insert_impl(dsl_scan_io_queue_t *queue, scan_io_t *sio) { avl_index_t idx; - int64_t asize = sio->sio_asize; + int64_t asize = SIO_GET_ASIZE(sio); dsl_scan_t *scn = queue->q_scn; ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock)); @@ -3434,11 +3509,12 @@ scan_io_queue_insert_impl(dsl_scan_io_queue_t *queue, if (avl_find(&queue->q_sios_by_addr, sio, &idx) != NULL) { /* block is already scheduled for reading */ atomic_add_64(&scn->scn_bytes_pending, -asize); - kmem_free(sio, sizeof (*sio)); + sio_free(sio); return; } avl_insert(&queue->q_sios_by_addr, sio, idx); - range_tree_add(queue->q_exts_by_addr, sio->sio_offset, asize); + queue->q_sio_memused += SIO_GET_MUSED(sio); + range_tree_add(queue->q_exts_by_addr, SIO_GET_OFFSET(sio), asize); } /* @@ -3452,7 +3528,7 @@ scan_io_queue_insert(dsl_scan_io_queue_t *queue, const int zio_flags, const zbookmark_phys_t *zb) { dsl_scan_t *scn = queue->q_scn; - scan_io_t *sio = kmem_zalloc(sizeof (*sio), KM_SLEEP); + scan_io_t *sio = sio_alloc(BP_GET_NDVAS(bp)); ASSERT0(BP_IS_GANG(bp)); ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock)); @@ -3466,7 +3542,7 @@ scan_io_queue_insert(dsl_scan_io_queue_t *queue, const * get an integer underflow in case the worker processes the * zio before we get to incrementing this counter. */ - atomic_add_64(&scn->scn_bytes_pending, sio->sio_asize); + atomic_add_64(&scn->scn_bytes_pending, SIO_GET_ASIZE(sio)); scan_io_queue_insert_impl(queue, sio); } @@ -3699,15 +3775,11 @@ ext_size_compare(const void *x, const void *y) * based on LBA-order (from lowest to highest). */ static int -io_addr_compare(const void *x, const void *y) +sio_addr_compare(const void *x, const void *y) { const scan_io_t *a = x, *b = y; - if (a->sio_offset < b->sio_offset) - return (-1); - if (a->sio_offset == b->sio_offset) - return (0); - return (1); + return (AVL_CMP(SIO_GET_OFFSET(a), SIO_GET_OFFSET(b))); } /* IO queues are created on demand when they are needed. */ @@ -3719,10 +3791,11 @@ scan_io_queue_create(vdev_t *vd) q->q_scn = scn; q->q_vd = vd; + q->q_sio_memused = 0; cv_init(&q->q_zio_cv, NULL, CV_DEFAULT, NULL); q->q_exts_by_addr = range_tree_create_impl(&rt_avl_ops, &q->q_exts_by_size, ext_size_compare, zfs_scan_max_ext_gap); - avl_create(&q->q_sios_by_addr, io_addr_compare, + avl_create(&q->q_sios_by_addr, sio_addr_compare, sizeof (scan_io_t), offsetof(scan_io_t, sio_nodes.sio_addr_node)); return (q); @@ -3746,11 +3819,13 @@ dsl_scan_io_queue_destroy(dsl_scan_io_queue_t *queue) while ((sio = avl_destroy_nodes(&queue->q_sios_by_addr, &cookie)) != NULL) { ASSERT(range_tree_contains(queue->q_exts_by_addr, - sio->sio_offset, sio->sio_asize)); - bytes_dequeued += sio->sio_asize; - kmem_free(sio, sizeof (*sio)); + SIO_GET_OFFSET(sio), SIO_GET_ASIZE(sio))); + bytes_dequeued += SIO_GET_ASIZE(sio); + queue->q_sio_memused -= SIO_GET_MUSED(sio); + sio_free(sio); } + ASSERT0(queue->q_sio_memused); atomic_add_64(&scn->scn_bytes_pending, -bytes_dequeued); range_tree_vacate(queue->q_exts_by_addr, NULL, queue); range_tree_destroy(queue->q_exts_by_addr); @@ -3805,7 +3880,7 @@ dsl_scan_freed_dva(spa_t *spa, const blkptr_t *bp, int vdev_t *vdev; kmutex_t *q_lock; dsl_scan_io_queue_t *queue; - scan_io_t srch, *sio; + scan_io_t *srch_sio, *sio; avl_index_t idx; uint64_t start, size; @@ -3820,9 +3895,10 @@ dsl_scan_freed_dva(spa_t *spa, const blkptr_t *bp, int return; } - bp2sio(bp, &srch, dva_i); - start = srch.sio_offset; - size = srch.sio_asize; + srch_sio = sio_alloc(BP_GET_NDVAS(bp)); + bp2sio(bp, srch_sio, dva_i); + start = SIO_GET_OFFSET(srch_sio); + size = SIO_GET_ASIZE(srch_sio); /* * We can find the zio in two states: @@ -3842,15 +3918,18 @@ dsl_scan_freed_dva(spa_t *spa, const blkptr_t *bp, int * be done with issuing the zio's it gathered and will * signal us. */ - sio = avl_find(&queue->q_sios_by_addr, &srch, &idx); + sio = avl_find(&queue->q_sios_by_addr, srch_sio, &idx); + sio_free(srch_sio); + if (sio != NULL) { - int64_t asize = sio->sio_asize; + int64_t asize = SIO_GET_ASIZE(sio); blkptr_t tmpbp; /* Got it while it was cold in the queue */ - ASSERT3U(start, ==, sio->sio_offset); + ASSERT3U(start, ==, SIO_GET_OFFSET(sio)); ASSERT3U(size, ==, asize); avl_remove(&queue->q_sios_by_addr, sio); + queue->q_sio_memused -= SIO_GET_MUSED(sio); ASSERT(range_tree_contains(queue->q_exts_by_addr, start, size)); range_tree_remove_fill(queue->q_exts_by_addr, start, size); @@ -3863,10 +3942,10 @@ dsl_scan_freed_dva(spa_t *spa, const blkptr_t *bp, int atomic_add_64(&scn->scn_bytes_pending, -asize); /* count the block as though we issued it */ - sio2bp(sio, &tmpbp, dva_i); + sio2bp(sio, &tmpbp); count_block(scn, dp->dp_blkstats, &tmpbp); - kmem_free(sio, sizeof (*sio)); + sio_free(sio); } mutex_exit(q_lock); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Thu Nov 21 14:00:09 2019 (r354954) @@ -2069,7 +2069,7 @@ metaslab_space_weight(metaslab_t *msp) * In effect, this means that we'll select the metaslab with the most * free bandwidth rather than simply the one with the most free space. */ - if (metaslab_lba_weighting_enabled) { + if (!vd->vdev_nonrot && metaslab_lba_weighting_enabled) { weight = 2 * weight - (msp->ms_id * weight) / vd->vdev_ms_count; ASSERT(weight >= space && weight <= 2 * space); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Thu Nov 21 14:00:09 2019 (r354954) @@ -2036,6 +2036,7 @@ spa_init(int mode) dmu_init(); zil_init(); vdev_cache_stat_init(); + vdev_mirror_stat_init(); zfs_prop_init(); zpool_prop_init(); zpool_feature_init(); @@ -2052,6 +2053,7 @@ spa_fini(void) spa_evict_all(); vdev_cache_stat_fini(); + vdev_mirror_stat_fini(); zil_fini(); dmu_fini(); zio_fini(); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Thu Nov 21 14:00:09 2019 (r354954) @@ -907,6 +907,10 @@ extern void spa_get_errlists(spa_t *spa, avl_tree_t *l extern void vdev_cache_stat_init(void); extern void vdev_cache_stat_fini(void); +/* vdev mirror */ +extern void vdev_mirror_stat_init(void); +extern void vdev_mirror_stat_fini(void); + /* Initialization and termination */ extern void spa_init(int flags); extern void spa_fini(void); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Thu Nov 21 14:00:09 2019 (r354954) @@ -139,6 +139,9 @@ extern zio_t *vdev_queue_io(zio_t *zio); extern void vdev_queue_io_done(zio_t *zio); extern void vdev_queue_change_io_priority(zio_t *zio, zio_priority_t priority); +extern int vdev_queue_length(vdev_t *vd); +extern uint64_t vdev_queue_last_offset(vdev_t *vd); + extern void vdev_config_dirty(vdev_t *vd); extern void vdev_config_clean(vdev_t *vd); extern int vdev_config_sync(vdev_t **svd, int svdcount, uint64_t txg); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Thu Nov 21 14:00:09 2019 (r354954) @@ -225,6 +225,7 @@ struct vdev { vdev_stat_t vdev_stat; /* virtual device statistics */ boolean_t vdev_expanding; /* expand the vdev? */ boolean_t vdev_reopening; /* reopen in progress? */ + boolean_t vdev_nonrot; /* true if solid state */ int vdev_open_error; /* error on last open */ kthread_t *vdev_open_thread; /* thread opening children */ uint64_t vdev_crtxg; /* txg when top-level was added */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h Thu Nov 21 14:00:09 2019 (r354954) @@ -318,12 +318,14 @@ typedef struct zinject_record { uint64_t zi_timer; uint64_t zi_nlanes; uint32_t zi_cmd; - uint32_t zi_pad; + uint32_t zi_dvas; } zinject_record_t; #define ZINJECT_NULL 0x1 #define ZINJECT_FLUSH_ARC 0x2 #define ZINJECT_UNLOAD_SPA 0x4 + +#define ZI_NO_DVA (-1) typedef enum zinject_type { ZINJECT_UNINITIALIZED, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 14:00:09 2019 (r354954) @@ -1479,19 +1479,27 @@ vdev_open_children(vdev_t *vd) * spa_namespace_lock */ if (vdev_uses_zvols(vd)) { +retry_sync: for (int c = 0; c < children; c++) vd->vdev_child[c]->vdev_open_error = vdev_open(vd->vdev_child[c]); - return; + } else { + tq = taskq_create("vdev_open", children, minclsyspri, + children, children, TASKQ_PREPOPULATE); + if (tq == NULL) + goto retry_sync; + + for (int c = 0; c < children; c++) + VERIFY(taskq_dispatch(tq, vdev_open_child, + vd->vdev_child[c], TQ_SLEEP) != TASKQID_INVALID); + + taskq_destroy(tq); } - tq = taskq_create("vdev_open", children, minclsyspri, - children, children, TASKQ_PREPOPULATE); - for (int c = 0; c < children; c++) - VERIFY(taskq_dispatch(tq, vdev_open_child, vd->vdev_child[c], - TQ_SLEEP) != TASKQID_INVALID); + vd->vdev_nonrot = B_TRUE; - taskq_destroy(tq); + for (int c = 0; c < children; c++) + vd->vdev_nonrot &= vd->vdev_child[c]->vdev_nonrot; } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Thu Nov 21 14:00:09 2019 (r354954) @@ -606,6 +606,16 @@ skip_open: */ vd->vdev_nowritecache = B_FALSE; + /* Inform the ZIO pipeline that we are non-rotational */ + vd->vdev_nonrot = B_FALSE; + if (ldi_prop_exists(dvd->vd_lh, DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, + "device-solid-state")) { + if (ldi_prop_get_int(dvd->vd_lh, + LDI_DEV_T_ANY | DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, + "device-solid-state", B_FALSE) != 0) + vd->vdev_nonrot = B_TRUE; + } + return (0); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Thu Nov 21 14:00:09 2019 (r354954) @@ -58,6 +58,9 @@ vdev_file_open(vdev_t *vd, uint64_t *psize, uint64_t * vattr_t vattr; int error; + /* Rotational optimizations only make sense on block devices */ + vd->vdev_nonrot = B_TRUE; + /* * We must have a pathname, and it must be absolute. */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Thu Nov 21 14:00:09 2019 (r354954) @@ -38,6 +38,65 @@ #include /* + * Vdev mirror kstats + */ +static kstat_t *mirror_ksp = NULL; + +typedef struct mirror_stats { + kstat_named_t vdev_mirror_stat_rotating_linear; + kstat_named_t vdev_mirror_stat_rotating_offset; + kstat_named_t vdev_mirror_stat_rotating_seek; + kstat_named_t vdev_mirror_stat_non_rotating_linear; + kstat_named_t vdev_mirror_stat_non_rotating_seek; + + kstat_named_t vdev_mirror_stat_preferred_found; + kstat_named_t vdev_mirror_stat_preferred_not_found; +} mirror_stats_t; + +static mirror_stats_t mirror_stats = { + /* New I/O follows directly the last I/O */ + { "rotating_linear", KSTAT_DATA_UINT64 }, + /* New I/O is within zfs_vdev_mirror_rotating_seek_offset of the last */ + { "rotating_offset", KSTAT_DATA_UINT64 }, + /* New I/O requires random seek */ + { "rotating_seek", KSTAT_DATA_UINT64 }, + /* New I/O follows directly the last I/O (nonrot) */ + { "non_rotating_linear", KSTAT_DATA_UINT64 }, + /* New I/O requires random seek (nonrot) */ + { "non_rotating_seek", KSTAT_DATA_UINT64 }, + /* Preferred child vdev found */ + { "preferred_found", KSTAT_DATA_UINT64 }, + /* Preferred child vdev not found or equal load */ + { "preferred_not_found", KSTAT_DATA_UINT64 }, + +}; + +#define MIRROR_STAT(stat) (mirror_stats.stat.value.ui64) +#define MIRROR_INCR(stat, val) atomic_add_64(&MIRROR_STAT(stat), val) +#define MIRROR_BUMP(stat) MIRROR_INCR(stat, 1) + +void +vdev_mirror_stat_init(void) +{ + mirror_ksp = kstat_create("zfs", 0, "vdev_mirror_stats", + "misc", KSTAT_TYPE_NAMED, + sizeof (mirror_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); + if (mirror_ksp != NULL) { + mirror_ksp->ks_data = &mirror_stats; + kstat_install(mirror_ksp); + } +} + +void +vdev_mirror_stat_fini(void) +{ + if (mirror_ksp != NULL) { + kstat_delete(mirror_ksp); + mirror_ksp = NULL; + } +} + +/* * Virtual device vector for mirroring. */ @@ -45,48 +104,182 @@ typedef struct mirror_child { vdev_t *mc_vd; uint64_t mc_offset; int mc_error; + int mc_load; uint8_t mc_tried; uint8_t mc_skipped; uint8_t mc_speculative; } mirror_child_t; typedef struct mirror_map { + int *mm_preferred; + int mm_preferred_cnt; int mm_children; int mm_resilvering; - int mm_preferred; int mm_root; - mirror_child_t mm_child[1]; + mirror_child_t mm_child[]; } mirror_map_t; int vdev_mirror_shift = 21; +/* + * The load configuration settings below are tuned by default for + * the case where all devices are of the same rotational type. + * + * If there is a mixture of rotating and non-rotating media, setting + * zfs_vdev_mirror_non_rotating_seek_inc to 0 may well provide better results + * as it will direct more reads to the non-rotating vdevs which are more likely + * to have a higher performance. + */ + +/* Rotating media load calculation configuration. */ +static int zfs_vdev_mirror_rotating_inc = 0; +static int zfs_vdev_mirror_rotating_seek_inc = 5; +static int zfs_vdev_mirror_rotating_seek_offset = 1 * 1024 * 1024; + +/* Non-rotating media load calculation configuration. */ +static int zfs_vdev_mirror_non_rotating_inc = 0; +static int zfs_vdev_mirror_non_rotating_seek_inc = 1; + +static inline size_t +vdev_mirror_map_size(int children) +{ + return (offsetof(mirror_map_t, mm_child[children]) + + sizeof (int) * children); +} + +static inline mirror_map_t * +vdev_mirror_map_alloc(int children, boolean_t resilvering, boolean_t root) +{ + mirror_map_t *mm; + + mm = kmem_zalloc(vdev_mirror_map_size(children), KM_SLEEP); + mm->mm_children = children; + mm->mm_resilvering = resilvering; + mm->mm_root = root; + mm->mm_preferred = (int *)((uintptr_t)mm + + offsetof(mirror_map_t, mm_child[children])); + + return (mm); +} + static void vdev_mirror_map_free(zio_t *zio) { mirror_map_t *mm = zio->io_vsd; - kmem_free(mm, offsetof(mirror_map_t, mm_child[mm->mm_children])); + kmem_free(mm, vdev_mirror_map_size(mm->mm_children)); } static const zio_vsd_ops_t vdev_mirror_vsd_ops = { - vdev_mirror_map_free, - zio_vsd_default_cksum_report + .vsd_free = vdev_mirror_map_free, + .vsd_cksum_report = zio_vsd_default_cksum_report }; +static int +vdev_mirror_load(mirror_map_t *mm, vdev_t *vd, uint64_t zio_offset) +{ + uint64_t last_offset; + int64_t offset_diff; + int load; + + /* All DVAs have equal weight at the root. */ + if (mm->mm_root) + return (INT_MAX); + + /* + * We don't return INT_MAX if the device is resilvering i.e. + * vdev_resilver_txg != 0 as when tested performance was slightly + * worse overall when resilvering with compared to without. + */ + + /* Fix zio_offset for leaf vdevs */ + if (vd->vdev_ops->vdev_op_leaf) + zio_offset += VDEV_LABEL_START_SIZE; + + /* Standard load based on pending queue length. */ + load = vdev_queue_length(vd); + last_offset = vdev_queue_last_offset(vd); + + if (vd->vdev_nonrot) { + /* Non-rotating media. */ + if (last_offset == zio_offset) { + MIRROR_BUMP(vdev_mirror_stat_non_rotating_linear); + return (load + zfs_vdev_mirror_non_rotating_inc); + } + + /* + * Apply a seek penalty even for non-rotating devices as + * sequential I/O's can be aggregated into fewer operations on + * the device, thus avoiding unnecessary per-command overhead + * and boosting performance. + */ + MIRROR_BUMP(vdev_mirror_stat_non_rotating_seek); + return (load + zfs_vdev_mirror_non_rotating_seek_inc); + } + + /* Rotating media I/O's which directly follow the last I/O. */ + if (last_offset == zio_offset) { + MIRROR_BUMP(vdev_mirror_stat_rotating_linear); + return (load + zfs_vdev_mirror_rotating_inc); + } + + /* + * Apply half the seek increment to I/O's within seek offset + * of the last I/O issued to this vdev as they should incur less + * of a seek increment. + */ + offset_diff = (int64_t)(last_offset - zio_offset); + if (ABS(offset_diff) < zfs_vdev_mirror_rotating_seek_offset) { + MIRROR_BUMP(vdev_mirror_stat_rotating_offset); + return (load + (zfs_vdev_mirror_rotating_seek_inc / 2)); + } + + /* Apply the full seek increment to all other I/O's. */ + MIRROR_BUMP(vdev_mirror_stat_rotating_seek); + return (load + zfs_vdev_mirror_rotating_seek_inc); +} + static mirror_map_t * -vdev_mirror_map_alloc(zio_t *zio) +vdev_mirror_map_init(zio_t *zio) { mirror_map_t *mm = NULL; mirror_child_t *mc; vdev_t *vd = zio->io_vd; - int c, d; + int c; if (vd == NULL) { dva_t *dva = zio->io_bp->blk_dva; spa_t *spa = zio->io_spa; + dsl_scan_t *scn = NULL; dva_t dva_copy[SPA_DVAS_PER_BP]; - c = BP_GET_NDVAS(zio->io_bp); + if (spa->spa_dsl_pool != NULL) { + scn = spa->spa_dsl_pool->dp_scan; + } + /* + * The sequential scrub code sorts and issues all DVAs + * of a bp separately. Each of these IOs includes all + * original DVA copies so that repairs can be performed + * in the event of an error, but we only actually want + * to check the first DVA since the others will be + * checked by their respective sorted IOs. Only if we + * hit an error will we try all DVAs upon retrying. + * + * Note: This check is safe even if the user switches + * from a legacy scrub to a sequential one in the middle + * of processing, since scn_is_sorted isn't updated until + * all outstanding IOs from the previous scrub pass + * complete. + */ + if ((zio->io_flags & ZIO_FLAG_SCRUB) && + !(zio->io_flags & ZIO_FLAG_IO_RETRY) && + scn != NULL && + scn->scn_is_sorted && + dsl_scan_scrubbing(spa->spa_dsl_pool)) { + c = 1; + } else { + c = BP_GET_NDVAS(zio->io_bp); + } /* * If we do not trust the pool config, some DVAs might be @@ -110,24 +303,7 @@ vdev_mirror_map_alloc(zio_t *zio) } } - mm = kmem_zalloc(offsetof(mirror_map_t, mm_child[c]), KM_SLEEP); - mm->mm_children = c; - mm->mm_resilvering = B_FALSE; - mm->mm_preferred = spa_get_random(c); - mm->mm_root = B_TRUE; - - /* - * Check the other, lower-index DVAs to see if they're on - * the same vdev as the child we picked. If they are, use - * them since they are likely to have been allocated from - * the primary metaslab in use at the time, and hence are - * more likely to have locality with single-copy data. - */ - for (c = mm->mm_preferred, d = c - 1; d >= 0; d--) { - if (DVA_GET_VDEV(&dva[d]) == DVA_GET_VDEV(&dva[c])) - mm->mm_preferred = d; - } - + mm = vdev_mirror_map_alloc(c, B_FALSE, B_TRUE); for (c = 0; c < mm->mm_children; c++) { mc = &mm->mm_child[c]; @@ -135,12 +311,6 @@ vdev_mirror_map_alloc(zio_t *zio) mc->mc_offset = DVA_GET_OFFSET(&dva[c]); } } else { - int replacing; - - c = vd->vdev_children; - - mm = kmem_zalloc(offsetof(mirror_map_t, mm_child[c]), KM_SLEEP); - mm->mm_children = c; /* * If we are resilvering, then we should handle scrub reads * differently; we shouldn't issue them to the resilvering @@ -164,25 +334,12 @@ vdev_mirror_map_alloc(zio_t *zio) * automatically removed from the pool after the user replaces * the device that originally failed. */ - replacing = (vd->vdev_ops == &vdev_replacing_ops || - vd->vdev_ops == &vdev_spare_ops); - /* - * If a spa load is in progress, then spa_dsl_pool may be - * uninitialized. But we shouldn't be resilvering during a spa - * load anyway. - */ - if (replacing && - (spa_load_state(vd->vdev_spa) == SPA_LOAD_NONE) && - dsl_scan_resilvering(vd->vdev_spa->spa_dsl_pool)) { - mm->mm_resilvering = B_TRUE; - } else { - mm->mm_resilvering = B_FALSE; - } - - mm->mm_preferred = mm->mm_resilvering ? 0 : - (zio->io_offset >> vdev_mirror_shift) % c; - mm->mm_root = B_FALSE; - + boolean_t replacing = (vd->vdev_ops == &vdev_replacing_ops || + vd->vdev_ops == &vdev_spare_ops) && + spa_load_state(vd->vdev_spa) == SPA_LOAD_NONE && + dsl_scan_resilvering(vd->vdev_spa->spa_dsl_pool); + mm = vdev_mirror_map_alloc(vd->vdev_children, replacing, + B_FALSE); for (c = 0; c < mm->mm_children; c++) { mc = &mm->mm_child[c]; mc->mc_vd = vd->vdev_child[c]; @@ -269,6 +426,7 @@ vdev_mirror_scrub_done(zio_t *zio) } mutex_exit(&zio->io_lock); } + abd_free(zio->io_abd); mc->mc_error = zio->io_error; @@ -277,6 +435,54 @@ vdev_mirror_scrub_done(zio_t *zio) } /* + * Check the other, lower-index DVAs to see if they're on the same + * vdev as the child we picked. If they are, use them since they + * are likely to have been allocated from the primary metaslab in + * use at the time, and hence are more likely to have locality with + * single-copy data. + */ +static int +vdev_mirror_dva_select(zio_t *zio, int p) +{ + dva_t *dva = zio->io_bp->blk_dva; + mirror_map_t *mm = zio->io_vsd; + int preferred; + int c; + + preferred = mm->mm_preferred[p]; + for (p--; p >= 0; p--) { + c = mm->mm_preferred[p]; + if (DVA_GET_VDEV(&dva[c]) == DVA_GET_VDEV(&dva[preferred])) + preferred = c; + } + return (preferred); +} + +static int +vdev_mirror_preferred_child_randomize(zio_t *zio) +{ + mirror_map_t *mm = zio->io_vsd; + int p; + + if (mm->mm_root) { + p = spa_get_random(mm->mm_preferred_cnt); + return (vdev_mirror_dva_select(zio, p)); + } + + /* + * To ensure we don't always favour the first matching vdev, + * which could lead to wear leveling issues on SSD's, we + * use the I/O offset as a pseudo random seed into the vdevs + * which have the lowest load. + */ + p = (zio->io_offset >> vdev_mirror_shift) % mm->mm_preferred_cnt; + return (mm->mm_preferred[p]); +} + +/* + * Try to find a vdev whose DTL doesn't contain the block we want to read + * prefering vdevs based on determined load. + * * Try to find a child whose DTL doesn't contain the block we want to read. * If we can't, try the read on any vdev we haven't already tried. */ @@ -284,43 +490,64 @@ static int vdev_mirror_child_select(zio_t *zio) { mirror_map_t *mm = zio->io_vsd; - mirror_child_t *mc; uint64_t txg = zio->io_txg; - int i, c; + int c, lowest_load; ASSERT(zio->io_bp == NULL || BP_PHYSICAL_BIRTH(zio->io_bp) == txg); - /* - * Try to find a child whose DTL doesn't contain the block to read. - * If a child is known to be completely inaccessible (indicated by - * vdev_readable() returning B_FALSE), don't even try. - */ - for (i = 0, c = mm->mm_preferred; i < mm->mm_children; i++, c++) { - if (c >= mm->mm_children) *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:00:13 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 315551BFB24; Thu, 21 Nov 2019 14:00:13 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jh5n0TjLz4R2w; Thu, 21 Nov 2019 14:00:13 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CF085C44C; Thu, 21 Nov 2019 14:00:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE0CBG041945; Thu, 21 Nov 2019 14:00:12 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE0CVE041944; Thu, 21 Nov 2019 14:00:12 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211400.xALE0CVE041944@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:00:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354954 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zinject X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zinject X-SVN-Commit-Revision: 354954 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:00:13 -0000 Author: avg Date: Thu Nov 21 14:00:09 2019 New Revision: 354954 URL: https://svnweb.freebsd.org/changeset/base/354954 Log: 10566 Multiple DVA Scrubbing Fix illumos/illumos-gate@12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://github.com/illumos/illumos-gate/commit/12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://www.illumos.org/issues/10566 ZoL PR_8453 Author: Tom Caputi Date: Fri Mar 15 17:14:31 2019 -0400 Multiple DVA Scrubbing Fix Currently, there is an issue in the sequential scrub code which prevents self healing from working in some cases. The scrub code will split up all DVA copies of a bp and issue each of them separately. The problem is that, since each of the DVAs is no longer associated with the others, the self healing code doesn't have the opportunity to repair problems that show up in one of the DVAs with the data from the others. This patch fixes this issue by ensuring that all IOs issued by the sequential scrub code include all DVAs. Initially, only the first DVA of each is attempted. If an issue arises, the IO is retried with all available copies, giving the self healing code a chance to correct the issue. To test this change, this patch also adds the ability for zinject to specify individual DVAs to inject read errors into. We then add a new test case that utilizes this functionality to ensure scrubs and self-healing reads can handle and transparently fix issues with individual copies of blocks. This update is followup on #10405 While attempting to port this update, the following ZoL updates are included: 551905dd4 vdev_mirror: kstat observables for preferred vdev d6c6590c5 vdev_mirror: load balancing fixes 9f500936c FreeBSD r256956: Improve ZFS N-way mirror read performance by Portions contributed by: Toomas Soome Portions contributed by: Jerry Jelinek Author: Tom Caputi Modified: vendor/illumos/dist/cmd/zinject/zinject.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_inject.c Modified: vendor/illumos/dist/cmd/zinject/zinject.c ============================================================================== --- vendor/illumos/dist/cmd/zinject/zinject.c Thu Nov 21 13:59:06 2019 (r354953) +++ vendor/illumos/dist/cmd/zinject/zinject.c Thu Nov 21 14:00:09 2019 (r354954) @@ -47,48 +47,48 @@ * * This form of the command looks like: * - * zinject -d device [-e errno] [-L ] pool + * zinject -d device [-e errno] [-L ] pool * * * DATA FAULTS * * We begin with a tuple of the form: * - * + * * - * type A string describing the type of data to target. Each type - * implicitly describes how to interpret 'object'. Currently, - * the following values are supported: + * type A string describing the type of data to target. Each type + * implicitly describes how to interpret 'object'. Currently, + * the following values are supported: * - * data User data for a file - * dnode Dnode for a file or directory + * data User data for a file + * dnode Dnode for a file or directory * * The following MOS objects are special. Instead of injecting * errors on a particular object or blkid, we inject errors across * all objects of the given type. * - * mos Any data in the MOS - * mosdir object directory - * config pool configuration - * bpobj blkptr list - * spacemap spacemap - * metaslab metaslab - * errlog persistent error log + * mos Any data in the MOS + * mosdir object directory + * config pool configuration + * bpobj blkptr list + * spacemap spacemap + * metaslab metaslab + * errlog persistent error log * - * level Object level. Defaults to '0', not applicable to all types. If - * a range is given, this corresponds to the indirect block - * corresponding to the specific range. + * level Object level. Defaults to '0', not applicable to all types. If + * a range is given, this corresponds to the indirect block + * corresponding to the specific range. * * range A numerical range [start,end) within the object. Defaults to * the full size of the file. * - * object A string describing the logical location of the object. For - * files and directories (currently the only supported types), - * this is the path of the object on disk. + * object A string describing the logical location of the object. For + * files and directories (currently the only supported types), + * this is the path of the object on disk. * * This is translated, via libzpool, into the following internal representation: * - * + * * * These types should be self-explanatory. This tuple is then passed to the * kernel via a special ioctl() to initiate fault injection for the given @@ -98,12 +98,12 @@ * * The command itself takes one of the forms: * - * zinject - * zinject <-a | -u pool> - * zinject -c - * zinject [-q] <-t type> [-f freq] [-u] [-a] [-m] [-e errno] [-l level] + * zinject + * zinject <-a | -u pool> + * zinject -c + * zinject [-q] <-t type> [-f freq] [-u] [-a] [-m] [-e errno] [-l level] * [-r range] - * zinject [-f freq] [-a] [-m] [-u] -b objset:object:level:start:end pool + * zinject [-f freq] [-a] [-m] [-u] -b objset:object:level:start:end pool * * With no arguments, the command prints all currently registered injection * handlers, with their numeric identifiers. @@ -288,8 +288,8 @@ usage(void) "\t\tspecified by the remaining tuple. Each number is in\n" "\t\thexidecimal, and only one block can be specified.\n" "\n" - "\tzinject [-q] <-t type> [-e errno] [-l level] [-r range]\n" - "\t [-a] [-m] [-u] [-f freq] \n" + "\tzinject [-q] <-t type> [-C dvas] [-e errno] [-l level]\n" + "\t\t[-r range] [-a] [-m] [-u] [-f freq] \n" "\n" "\t\tInject an error into the object specified by the '-t' option\n" "\t\tand the object descriptor. The 'object' parameter is\n" @@ -297,7 +297,10 @@ usage(void) "\n" "\t\t-q\tQuiet mode. Only print out the handler number added.\n" "\t\t-e\tInject a specific error. Must be either 'io' or\n" - "\t\t\t'checksum'. Default is 'io'.\n" + "\t\t\t'checksum', or 'decompress'. Default is 'io'.\n" + "\t\t-C\tInject the given error only into specific DVAs. The\n" + "\t\t\tDVAs should be specified as a list of 0-indexed DVAs\n" + "\t\t\tseparated by commas (ex. '0,2').\n" "\t\t-l\tInject error at a particular block level. Default is " "0.\n" "\t\t-m\tAutomatically remount underlying filesystem.\n" @@ -358,17 +361,19 @@ print_data_handler(int id, const char *pool, zinject_r return (0); if (*count == 0) { - (void) printf("%3s %-15s %-6s %-6s %-8s %3s %-15s\n", - "ID", "POOL", "OBJSET", "OBJECT", "TYPE", "LVL", "RANGE"); + (void) printf("%3s %-15s %-6s %-6s %-8s %3s %-4s ", + "%-15s\n", "ID", "POOL", "OBJSET", "OBJECT", "TYPE", + "LVL", "DVAs", "RANGE"); (void) printf("--- --------------- ------ " - "------ -------- --- ---------------\n"); + "------ -------- --- ---- ----------------\n"); } *count += 1; - (void) printf("%3d %-15s %-6llu %-6llu %-8s %3d ", id, pool, - (u_longlong_t)record->zi_objset, (u_longlong_t)record->zi_object, - type_to_name(record->zi_type), record->zi_level); + (void) printf("%3d %-15s %-6llu %-6llu %-8s %-3d 0x%02x ", + id, pool, (u_longlong_t)record->zi_objset, + (u_longlong_t)record->zi_object, type_to_name(record->zi_type), + record->zi_level, record->zi_dvas); if (record->zi_start == 0 && record->zi_end == -1ULL) @@ -598,6 +603,7 @@ register_handler(const char *pool, int flags, zinject_ (void) printf(" range: [%llu, %llu)\n", (u_longlong_t)record->zi_start, (u_longlong_t)record->zi_end); + (void) printf(" dvas: 0x%x\n", record->zi_dvas); } } @@ -649,6 +655,59 @@ parse_delay(char *str, uint64_t *delay, uint64_t *nlan return (0); } +/* + * This function converts a string specifier for DVAs into a bit mask. + * The dva's provided by the user should be 0 indexed and separated by + * a comma. For example: + * "1" -> 0b0010 (0x2) + * "0,1" -> 0b0011 (0x3) + * "0,1,2" -> 0b0111 (0x7) + */ +static int +parse_dvas(const char *str, uint32_t *dvas_out) +{ + const char *c = str; + uint32_t mask = 0; + boolean_t need_delim = B_FALSE; + + /* max string length is 5 ("0,1,2") */ + if (strlen(str) > 5 || strlen(str) == 0) + return (EINVAL); + + while (*c != '\0') { + switch (*c) { + case '0': + case '1': + case '2': + /* check for pipe between DVAs */ + if (need_delim) + return (EINVAL); + + /* check if this DVA has been set already */ + if (mask & (1 << ((*c) - '0'))) + return (EINVAL); + + mask |= (1 << ((*c) - '0')); + need_delim = B_TRUE; + break; + case ',': + need_delim = B_FALSE; + break; + default: + /* check for invalid character */ + return (EINVAL); + } + c++; + } + + /* check for dangling delimiter */ + if (!need_delim) + return (EINVAL); + + *dvas_out = mask; + return (0); +} + int main(int argc, char **argv) { @@ -675,6 +734,7 @@ main(int argc, char **argv) int dur_secs = 0; int ret; int flags = 0; + uint32_t dvas = 0; if ((g_zfs = libzfs_init()) == NULL) { (void) fprintf(stderr, "internal error: failed to " @@ -705,7 +765,7 @@ main(int argc, char **argv) } while ((c = getopt(argc, argv, - ":aA:b:d:D:f:Fg:qhIc:t:T:l:mr:s:e:uL:p:")) != -1) { + ":aA:b:C:d:D:f:Fg:qhIc:t:T:l:mr:s:e:uL:p:")) != -1) { switch (c) { case 'a': flags |= ZINJECT_FLUSH_ARC; @@ -728,6 +788,17 @@ main(int argc, char **argv) case 'c': cancel = optarg; break; + case 'C': + ret = parse_dvas(optarg, &dvas); + if (ret != 0) { + (void) fprintf(stderr, "invalid DVA list '%s': " + "DVAs should be 0 indexed and separated by " + "commas.\n", optarg); + usage(); + libzfs_fini(g_zfs); + return (1); + } + break; case 'd': device = optarg; break; @@ -887,7 +958,8 @@ main(int argc, char **argv) * '-c' is invalid with any other options. */ if (raw != NULL || range != NULL || type != TYPE_INVAL || - level != 0 || record.zi_cmd != ZINJECT_UNINITIALIZED) { + level != 0 || record.zi_cmd != ZINJECT_UNINITIALIZED || + record.zi_freq > 0 || dvas != 0) { (void) fprintf(stderr, "cancel (-c) incompatible with " "any other options\n"); usage(); @@ -919,7 +991,8 @@ main(int argc, char **argv) * for doing injection, so handle it separately here. */ if (raw != NULL || range != NULL || type != TYPE_INVAL || - level != 0 || record.zi_cmd != ZINJECT_UNINITIALIZED) { + level != 0 || record.zi_cmd != ZINJECT_UNINITIALIZED || + dvas != 0) { (void) fprintf(stderr, "device (-d) incompatible with " "data error injection\n"); usage(); @@ -953,7 +1026,8 @@ main(int argc, char **argv) } else if (raw != NULL) { if (range != NULL || type != TYPE_INVAL || level != 0 || - record.zi_cmd != ZINJECT_UNINITIALIZED) { + record.zi_cmd != ZINJECT_UNINITIALIZED || + record.zi_freq > 0 || dvas != 0) { (void) fprintf(stderr, "raw (-b) format with " "any other options\n"); usage(); @@ -983,7 +1057,8 @@ main(int argc, char **argv) error = EIO; } else if (record.zi_cmd == ZINJECT_PANIC) { if (raw != NULL || range != NULL || type != TYPE_INVAL || - level != 0 || device != NULL) { + level != 0 || device != NULL || record.zi_freq > 0 || + dvas != 0) { (void) fprintf(stderr, "panic (-p) incompatible with " "other options\n"); usage(); @@ -1002,6 +1077,15 @@ main(int argc, char **argv) record.zi_type = atoi(argv[1]); dataset[0] = '\0'; } else if (record.zi_cmd == ZINJECT_IGNORED_WRITES) { + if (raw != NULL || range != NULL || type != TYPE_INVAL || + level != 0 || record.zi_freq > 0 || dvas != 0) { + (void) fprintf(stderr, "hardware failure (-I) " + "incompatible with other options\n"); + usage(); + libzfs_fini(g_zfs); + return (2); + } + if (nowrites == 0) { (void) fprintf(stderr, "-s or -g meaningless " "without -I (ignore writes)\n"); @@ -1053,6 +1137,18 @@ main(int argc, char **argv) (void) fprintf(stderr, "data error type must be " "'checksum' or 'io'\n"); return (1); + } + + if (dvas != 0) { + if (error == EACCES || error == EINVAL) { + (void) fprintf(stderr, "the '-C' option may " + "not be used with logical data errors " + "'decrypt' and 'decompress'\n"); + libzfs_fini(g_zfs); + return (1); + } + + record.zi_dvas = dvas; } record.zi_cmd = ZINJECT_DATA_FAULT; From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:01:00 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3C3021BFBF9; Thu, 21 Nov 2019 14:01:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jh6h0tXFz4RPV; Thu, 21 Nov 2019 14:01:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 028F9C478; Thu, 21 Nov 2019 14:01:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE0xOg042020; Thu, 21 Nov 2019 14:00:59 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE0xl7042019; Thu, 21 Nov 2019 14:00:59 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211400.xALE0xl7042019@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:00:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354955 - in vendor/illumos/dist: cmd/zpool man/man1m X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: in vendor/illumos/dist: cmd/zpool man/man1m X-SVN-Commit-Revision: 354955 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:01:00 -0000 Author: avg Date: Thu Nov 21 14:00:59 2019 New Revision: 354955 URL: https://svnweb.freebsd.org/changeset/base/354955 Log: 10900 Fix estimated scrub completion time illumos/illumos-gate@3db6627c6730f7ec4426ac0be0e9338235aed2d8 https://github.com/illumos/illumos-gate/commit/3db6627c6730f7ec4426ac0be0e9338235aed2d8 https://www.illumos.org/issues/10900 ZoL update: Author: Tom Caputi Date: Wed May 1 20:34:24 2019 -0400 Fix estimated scrub completion time Currently, it is possible for the 'zpool scrub' command to progress slightly beyond 100% due to concurrent changes happening on the live pool. This behavior is expected, but the userspace code for 'zpool status' would subtract the expected amount of data from the amount of data already scrubbed, resulting in a negative integer being casted to a large positive one. This number was then used to calculate the estimated completion time, resulting in wildly wrong results. This code changes the behavior so that 'zpool status' does not attempt to report an estimate during this period. Reviewed by: Brian Behlendorf Reviewed-by: Igor Kozhukhov Reviewed-by: George Melikov Signed-off-by: Tom Caputi Closes #8611 Closes #8687 After this fix, the zpool status output does display more reasonable output. Author: Tom Caputi Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 14:00:09 2019 (r354954) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 14:00:59 2019 (r354955) @@ -4804,7 +4804,7 @@ print_scan_status(pool_scan_stat_t *ps) scan_rate = pass_scanned / elapsed; issue_rate = pass_issued / elapsed; - total_secs_left = (issue_rate != 0) ? + total_secs_left = (issue_rate != 0 && total >= issued) ? ((total - issued) / issue_rate) : UINT64_MAX; days_left = total_secs_left / 60 / 60 / 24; @@ -4838,7 +4838,8 @@ print_scan_status(pool_scan_stat_t *ps) } if (pause == 0) { - if (issue_rate >= 10 * 1024 * 1024) { + if (total_secs_left != UINT64_MAX && + issue_rate >= 10 * 1024 * 1024) { (void) printf(gettext(", %llu days " "%02llu:%02llu:%02llu to go\n"), (u_longlong_t)days_left, (u_longlong_t)hours_left, Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Thu Nov 21 14:00:09 2019 (r354954) +++ vendor/illumos/dist/man/man1m/zpool.1m Thu Nov 21 14:00:59 2019 (r354955) @@ -1808,6 +1808,10 @@ If a scrub is paused, the resumes it. If a resilver is in progress, ZFS does not allow a scrub to be started until the resilver completes. +.Pp +Note that, due to changes in pool data on a live system, it is possible for +scrubs to progress slightly beyond 100% completion. +During this period, no completion time estimate will be provided. .Bl -tag -width Ds .It Fl s Stop scrubbing. From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:02:55 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 99A741BFE93; Thu, 21 Nov 2019 14:02:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Jh8v4ML6z4Rl4; Thu, 21 Nov 2019 14:02:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 79782C616; Thu, 21 Nov 2019 14:02:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE2t7n047282; Thu, 21 Nov 2019 14:02:55 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE2sCR047278; Thu, 21 Nov 2019 14:02:54 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211402.xALE2sCR047278@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:02:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354957 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 354957 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:02:55 -0000 Author: avg Date: Thu Nov 21 14:02:54 2019 New Revision: 354957 URL: https://svnweb.freebsd.org/changeset/base/354957 Log: 10853 spa_sync, vs_alloc can underflow and checkpoint test fixes illumos/illumos-gate@9740f25f0360eb7d9131fa15fabebf958bf19126 https://github.com/illumos/illumos-gate/commit/9740f25f0360eb7d9131fa15fabebf958bf19126 https://www.illumos.org/issues/10853 From ZoL 7558997 vs_alloc can underflow in L2ARC vdevs 8dc2197 Simplify spa_sync by breaking it up to smaller functions db58794 Make zdb results for checkpoint tests consistent Here are the commit msg summaries from ZoL: vs_alloc can underflow in L2ARC vdevs The current L2 ARC device code consistently uses psize to increment vs_alloc but varies between psize and lsize when decrementing it. The result of this behavior is that vs_alloc can be decremented more that it is incremented and underflow. This patch changes the code so asize is used anywhere. In addition, it ensures that vs_alloc gets incremented by the L2 ARC device code as buffers are written and not at the end of the l2arc_write_buffers() routine. The latter (and old) way would temporarily underflow vs_alloc as buffers that were just written, would be destroyed while l2arc_write_buffers() was still looping. Simplify spa_sync by breaking it up to smaller functions The point of this refactoring is to break the high-level conceptual steps of spa_sync() to their own helper functions. In general large functions can enhance readability if structured well, but in this case the amount of conceptual steps taken could use the help of helper functions. Make zdb results for checkpoint tests consistent This patch exports and re-imports the pool when these tests are analyzed with zdb to get consistent results. Portions contributed by: Jerry Jelinek Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Thu Nov 21 14:01:44 2019 (r354956) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Thu Nov 21 14:02:54 2019 (r354957) @@ -3124,7 +3124,8 @@ arc_hdr_l2hdr_destroy(arc_buf_hdr_t *hdr) { l2arc_buf_hdr_t *l2hdr = &hdr->b_l2hdr; l2arc_dev_t *dev = l2hdr->b_dev; - uint64_t psize = arc_hdr_size(hdr); + uint64_t psize = HDR_GET_PSIZE(hdr); + uint64_t asize = vdev_psize_to_asize(dev->l2ad_vdev, psize); ASSERT(MUTEX_HELD(&dev->l2ad_mtx)); ASSERT(HDR_HAS_L2HDR(hdr)); @@ -3134,9 +3135,10 @@ arc_hdr_l2hdr_destroy(arc_buf_hdr_t *hdr) ARCSTAT_INCR(arcstat_l2_psize, -psize); ARCSTAT_INCR(arcstat_l2_lsize, -HDR_GET_LSIZE(hdr)); - vdev_space_update(dev->l2ad_vdev, -psize, 0, 0); + vdev_space_update(dev->l2ad_vdev, -asize, 0, 0); - (void) zfs_refcount_remove_many(&dev->l2ad_alloc, psize, hdr); + (void) zfs_refcount_remove_many(&dev->l2ad_alloc, arc_hdr_size(hdr), + hdr); arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); } @@ -6759,10 +6761,12 @@ top: list_remove(buflist, hdr); arc_hdr_clear_flags(hdr, ARC_FLAG_HAS_L2HDR); - ARCSTAT_INCR(arcstat_l2_psize, -arc_hdr_size(hdr)); + uint64_t psize = HDR_GET_PSIZE(hdr); + ARCSTAT_INCR(arcstat_l2_psize, -psize); ARCSTAT_INCR(arcstat_l2_lsize, -HDR_GET_LSIZE(hdr)); - bytes_dropped += arc_hdr_size(hdr); + bytes_dropped += + vdev_psize_to_asize(dev->l2ad_vdev, psize); (void) zfs_refcount_remove_many(&dev->l2ad_alloc, arc_hdr_size(hdr), hdr); } @@ -7213,6 +7217,7 @@ l2arc_write_buffers(spa_t *spa, l2arc_dev_t *dev, uint write_psize += psize; write_asize += asize; dev->l2ad_hand += asize; + vdev_space_update(dev->l2ad_vdev, asize, 0, 0); mutex_exit(hash_lock); @@ -7238,7 +7243,6 @@ l2arc_write_buffers(spa_t *spa, l2arc_dev_t *dev, uint ARCSTAT_INCR(arcstat_l2_write_bytes, write_psize); ARCSTAT_INCR(arcstat_l2_lsize, write_lsize); ARCSTAT_INCR(arcstat_l2_psize, write_psize); - vdev_space_update(dev->l2ad_vdev, write_psize, 0, 0); /* * Bump device hand to the device start if it is approaching the end. Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Nov 21 14:01:44 2019 (r354956) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Nov 21 14:02:54 2019 (r354957) @@ -7379,6 +7379,9 @@ spa_sync_frees(spa_t *spa, bplist_t *bpl, dmu_tx_t *tx static void spa_sync_deferred_frees(spa_t *spa, dmu_tx_t *tx) { + if (spa_sync_pass(spa) != 1) + return; + zio_t *zio = zio_root(spa, NULL, NULL, 0); VERIFY3U(bpobj_iterate(&spa->spa_deferred_bpobj, spa_free_sync_cb, zio, tx), ==, 0); @@ -7774,10 +7777,10 @@ spa_sync_props(void *arg, dmu_tx_t *tx) static void spa_sync_upgrades(spa_t *spa, dmu_tx_t *tx) { - dsl_pool_t *dp = spa->spa_dsl_pool; + if (spa_sync_pass(spa) != 1) + return; - ASSERT(spa->spa_sync_pass == 1); - + dsl_pool_t *dp = spa->spa_dsl_pool; rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG); if (spa->spa_ubsync.ub_version < SPA_VERSION_ORIGIN && @@ -7872,116 +7875,31 @@ vdev_indirect_state_sync_verify(vdev_t *vd) } /* - * Sync the specified transaction group. New blocks may be dirtied as - * part of the process, so we iterate until it converges. + * Set the top-level vdev's max queue depth. Evaluate each top-level's + * async write queue depth in case it changed. The max queue depth will + * not change in the middle of syncing out this txg. */ -void -spa_sync(spa_t *spa, uint64_t txg) +static void +spa_sync_adjust_vdev_max_queue_depth(spa_t *spa) { - dsl_pool_t *dp = spa->spa_dsl_pool; - objset_t *mos = spa->spa_meta_objset; - bplist_t *free_bpl = &spa->spa_free_bplist[txg & TXG_MASK]; - metaslab_class_t *normal = spa_normal_class(spa); - metaslab_class_t *special = spa_special_class(spa); - metaslab_class_t *dedup = spa_dedup_class(spa); + ASSERT(spa_writeable(spa)); + vdev_t *rvd = spa->spa_root_vdev; - vdev_t *vd; - dmu_tx_t *tx; - int error; uint32_t max_queue_depth = zfs_vdev_async_write_max_active * zfs_vdev_queue_depth_pct / 100; + metaslab_class_t *normal = spa_normal_class(spa); + metaslab_class_t *special = spa_special_class(spa); + metaslab_class_t *dedup = spa_dedup_class(spa); - VERIFY(spa_writeable(spa)); - - /* - * Wait for i/os issued in open context that need to complete - * before this txg syncs. - */ - (void) zio_wait(spa->spa_txg_zio[txg & TXG_MASK]); - spa->spa_txg_zio[txg & TXG_MASK] = zio_root(spa, NULL, NULL, - ZIO_FLAG_CANFAIL); - - /* - * Lock out configuration changes. - */ - spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); - - spa->spa_syncing_txg = txg; - spa->spa_sync_pass = 0; - - for (int i = 0; i < spa->spa_alloc_count; i++) { - mutex_enter(&spa->spa_alloc_locks[i]); - VERIFY0(avl_numnodes(&spa->spa_alloc_trees[i])); - mutex_exit(&spa->spa_alloc_locks[i]); - } - - /* - * If there are any pending vdev state changes, convert them - * into config changes that go out with this transaction group. - */ - spa_config_enter(spa, SCL_STATE, FTAG, RW_READER); - while (list_head(&spa->spa_state_dirty_list) != NULL) { - /* - * We need the write lock here because, for aux vdevs, - * calling vdev_config_dirty() modifies sav_config. - * This is ugly and will become unnecessary when we - * eliminate the aux vdev wart by integrating all vdevs - * into the root vdev tree. - */ - spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); - spa_config_enter(spa, SCL_CONFIG | SCL_STATE, FTAG, RW_WRITER); - while ((vd = list_head(&spa->spa_state_dirty_list)) != NULL) { - vdev_state_clean(vd); - vdev_config_dirty(vd); - } - spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); - spa_config_enter(spa, SCL_CONFIG | SCL_STATE, FTAG, RW_READER); - } - spa_config_exit(spa, SCL_STATE, FTAG); - - tx = dmu_tx_create_assigned(dp, txg); - - spa->spa_sync_starttime = gethrtime(); - VERIFY(cyclic_reprogram(spa->spa_deadman_cycid, - spa->spa_sync_starttime + spa->spa_deadman_synctime)); - - /* - * If we are upgrading to SPA_VERSION_RAIDZ_DEFLATE this txg, - * set spa_deflate if we have no raid-z vdevs. - */ - if (spa->spa_ubsync.ub_version < SPA_VERSION_RAIDZ_DEFLATE && - spa->spa_uberblock.ub_version >= SPA_VERSION_RAIDZ_DEFLATE) { - int i; - - for (i = 0; i < rvd->vdev_children; i++) { - vd = rvd->vdev_child[i]; - if (vd->vdev_deflate_ratio != SPA_MINBLOCKSIZE) - break; - } - if (i == rvd->vdev_children) { - spa->spa_deflate = TRUE; - VERIFY(0 == zap_add(spa->spa_meta_objset, - DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_DEFLATE, - sizeof (uint64_t), 1, &spa->spa_deflate, tx)); - } - } - - /* - * Set the top-level vdev's max queue depth. Evaluate each - * top-level's async write queue depth in case it changed. - * The max queue depth will not change in the middle of syncing - * out this txg. - */ uint64_t slots_per_allocator = 0; for (int c = 0; c < rvd->vdev_children; c++) { vdev_t *tvd = rvd->vdev_child[c]; - metaslab_group_t *mg = tvd->vdev_mg; - metaslab_class_t *mc; + metaslab_group_t *mg = tvd->vdev_mg; if (mg == NULL || !metaslab_group_initialized(mg)) continue; - mc = mg->mg_class; + metaslab_class_t *mc = mg->mg_class; if (mc != normal && mc != special && mc != dedup) continue; @@ -8013,7 +7931,14 @@ spa_sync(spa_t *spa, uint64_t txg) normal->mc_alloc_throttle_enabled = zio_dva_throttle_enabled; special->mc_alloc_throttle_enabled = zio_dva_throttle_enabled; dedup->mc_alloc_throttle_enabled = zio_dva_throttle_enabled; +} +static void +spa_sync_condense_indirect(spa_t *spa, dmu_tx_t *tx) +{ + ASSERT(spa_writeable(spa)); + + vdev_t *rvd = spa->spa_root_vdev; for (int c = 0; c < rvd->vdev_children; c++) { vdev_t *vd = rvd->vdev_child[c]; vdev_indirect_state_sync_verify(vd); @@ -8023,10 +7948,16 @@ spa_sync(spa_t *spa, uint64_t txg) break; } } +} - /* - * Iterate to convergence. - */ +static void +spa_sync_iterate_to_convergence(spa_t *spa, dmu_tx_t *tx) +{ + objset_t *mos = spa->spa_meta_objset; + dsl_pool_t *dp = spa->spa_dsl_pool; + uint64_t txg = tx->tx_txg; + bplist_t *free_bpl = &spa->spa_free_bplist[txg & TXG_MASK]; + do { int pass = ++spa->spa_sync_pass; @@ -8052,79 +7983,60 @@ spa_sync(spa_t *spa, uint64_t txg) ddt_sync(spa, txg); dsl_scan_sync(dp, tx); + svr_sync(spa, tx); + spa_sync_upgrades(spa, tx); - if (spa->spa_vdev_removal != NULL) - svr_sync(spa, tx); - + vdev_t *vd = NULL; while ((vd = txg_list_remove(&spa->spa_vdev_txg_list, txg)) != NULL) vdev_sync(vd, txg); - if (pass == 1) { - spa_sync_upgrades(spa, tx); - ASSERT3U(txg, >=, - spa->spa_uberblock.ub_rootbp.blk_birth); + /* + * Note: We need to check if the MOS is dirty because we could + * have marked the MOS dirty without updating the uberblock + * (e.g. if we have sync tasks but no dirty user data). We need + * to check the uberblock's rootbp because it is updated if we + * have synced out dirty data (though in this case the MOS will + * most likely also be dirty due to second order effects, we + * don't want to rely on that here). + */ + if (pass == 1 && + spa->spa_uberblock.ub_rootbp.blk_birth < txg && + !dmu_objset_is_dirty(mos, txg)) { /* - * Note: We need to check if the MOS is dirty - * because we could have marked the MOS dirty - * without updating the uberblock (e.g. if we - * have sync tasks but no dirty user data). We - * need to check the uberblock's rootbp because - * it is updated if we have synced out dirty - * data (though in this case the MOS will most - * likely also be dirty due to second order - * effects, we don't want to rely on that here). + * Nothing changed on the first pass, therefore this + * TXG is a no-op. Avoid syncing deferred frees, so + * that we can keep this TXG as a no-op. */ - if (spa->spa_uberblock.ub_rootbp.blk_birth < txg && - !dmu_objset_is_dirty(mos, txg)) { - /* - * Nothing changed on the first pass, - * therefore this TXG is a no-op. Avoid - * syncing deferred frees, so that we - * can keep this TXG as a no-op. - */ - ASSERT(txg_list_empty(&dp->dp_dirty_datasets, - txg)); - ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg)); - ASSERT(txg_list_empty(&dp->dp_sync_tasks, txg)); - ASSERT(txg_list_empty(&dp->dp_early_sync_tasks, - txg)); - break; - } - spa_sync_deferred_frees(spa, tx); + ASSERT(txg_list_empty(&dp->dp_dirty_datasets, txg)); + ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg)); + ASSERT(txg_list_empty(&dp->dp_sync_tasks, txg)); + ASSERT(txg_list_empty(&dp->dp_early_sync_tasks, txg)); + break; } + spa_sync_deferred_frees(spa, tx); } while (dmu_objset_is_dirty(mos, txg)); +} - if (!list_is_empty(&spa->spa_config_dirty_list)) { - /* - * Make sure that the number of ZAPs for all the vdevs matches - * the number of ZAPs in the per-vdev ZAP list. This only gets - * called if the config is dirty; otherwise there may be - * outstanding AVZ operations that weren't completed in - * spa_sync_config_object. - */ - uint64_t all_vdev_zap_entry_count; - ASSERT0(zap_count(spa->spa_meta_objset, - spa->spa_all_vdev_zaps, &all_vdev_zap_entry_count)); - ASSERT3U(vdev_count_verify_zaps(spa->spa_root_vdev), ==, - all_vdev_zap_entry_count); - } +/* + * Rewrite the vdev configuration (which includes the uberblock) to + * commit the transaction group. + * + * If there are no dirty vdevs, we sync the uberblock to a few random + * top-level vdevs that are known to be visible in the config cache + * (see spa_vdev_add() for a complete description). If there *are* dirty + * vdevs, sync the uberblock to all vdevs. + */ +static void +spa_sync_rewrite_vdev_config(spa_t *spa, dmu_tx_t *tx) +{ + vdev_t *rvd = spa->spa_root_vdev; + uint64_t txg = tx->tx_txg; - if (spa->spa_vdev_removal != NULL) { - ASSERT0(spa->spa_vdev_removal->svr_bytes_done[txg & TXG_MASK]); - } - - /* - * Rewrite the vdev configuration (which includes the uberblock) - * to commit the transaction group. - * - * If there are no dirty vdevs, we sync the uberblock to a few - * random top-level vdevs that are known to be visible in the - * config cache (see spa_vdev_add() for a complete description). - * If there *are* dirty vdevs, sync the uberblock to all vdevs. - */ for (;;) { + int error = 0; + /* * We hold SCL_STATE to prevent vdev open/close/etc. * while we're attempting to write the vdev labels. @@ -8138,13 +8050,15 @@ spa_sync(spa_t *spa, uint64_t txg) int c0 = spa_get_random(children); for (int c = 0; c < children; c++) { - vd = rvd->vdev_child[(c0 + c) % children]; + vdev_t *vd = + rvd->vdev_child[(c0 + c) % children]; /* Stop when revisiting the first vdev */ if (c > 0 && svd[0] == vd) break; - if (vd->vdev_ms_array == 0 || vd->vdev_islog || + if (vd->vdev_ms_array == 0 || + vd->vdev_islog || !vdev_is_concrete(vd)) continue; @@ -8168,6 +8082,122 @@ spa_sync(spa_t *spa, uint64_t txg) zio_suspend(spa, NULL, ZIO_SUSPEND_IOERR); zio_resume_wait(spa); } +} + +/* + * Sync the specified transaction group. New blocks may be dirtied as + * part of the process, so we iterate until it converges. + */ +void +spa_sync(spa_t *spa, uint64_t txg) +{ + vdev_t *vd = NULL; + + VERIFY(spa_writeable(spa)); + + /* + * Wait for i/os issued in open context that need to complete + * before this txg syncs. + */ + (void) zio_wait(spa->spa_txg_zio[txg & TXG_MASK]); + spa->spa_txg_zio[txg & TXG_MASK] = zio_root(spa, NULL, NULL, + ZIO_FLAG_CANFAIL); + + /* + * Lock out configuration changes. + */ + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + + spa->spa_syncing_txg = txg; + spa->spa_sync_pass = 0; + + for (int i = 0; i < spa->spa_alloc_count; i++) { + mutex_enter(&spa->spa_alloc_locks[i]); + VERIFY0(avl_numnodes(&spa->spa_alloc_trees[i])); + mutex_exit(&spa->spa_alloc_locks[i]); + } + + /* + * If there are any pending vdev state changes, convert them + * into config changes that go out with this transaction group. + */ + spa_config_enter(spa, SCL_STATE, FTAG, RW_READER); + while (list_head(&spa->spa_state_dirty_list) != NULL) { + /* + * We need the write lock here because, for aux vdevs, + * calling vdev_config_dirty() modifies sav_config. + * This is ugly and will become unnecessary when we + * eliminate the aux vdev wart by integrating all vdevs + * into the root vdev tree. + */ + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + spa_config_enter(spa, SCL_CONFIG | SCL_STATE, FTAG, RW_WRITER); + while ((vd = list_head(&spa->spa_state_dirty_list)) != NULL) { + vdev_state_clean(vd); + vdev_config_dirty(vd); + } + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + spa_config_enter(spa, SCL_CONFIG | SCL_STATE, FTAG, RW_READER); + } + spa_config_exit(spa, SCL_STATE, FTAG); + + dsl_pool_t *dp = spa->spa_dsl_pool; + dmu_tx_t *tx = dmu_tx_create_assigned(dp, txg); + + spa->spa_sync_starttime = gethrtime(); + VERIFY(cyclic_reprogram(spa->spa_deadman_cycid, + spa->spa_sync_starttime + spa->spa_deadman_synctime)); + + /* + * If we are upgrading to SPA_VERSION_RAIDZ_DEFLATE this txg, + * set spa_deflate if we have no raid-z vdevs. + */ + if (spa->spa_ubsync.ub_version < SPA_VERSION_RAIDZ_DEFLATE && + spa->spa_uberblock.ub_version >= SPA_VERSION_RAIDZ_DEFLATE) { + vdev_t *rvd = spa->spa_root_vdev; + + int i; + for (i = 0; i < rvd->vdev_children; i++) { + vd = rvd->vdev_child[i]; + if (vd->vdev_deflate_ratio != SPA_MINBLOCKSIZE) + break; + } + if (i == rvd->vdev_children) { + spa->spa_deflate = TRUE; + VERIFY0(zap_add(spa->spa_meta_objset, + DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_DEFLATE, + sizeof (uint64_t), 1, &spa->spa_deflate, tx)); + } + } + + spa_sync_adjust_vdev_max_queue_depth(spa); + + spa_sync_condense_indirect(spa, tx); + + spa_sync_iterate_to_convergence(spa, tx); + +#ifdef ZFS_DEBUG + if (!list_is_empty(&spa->spa_config_dirty_list)) { + /* + * Make sure that the number of ZAPs for all the vdevs matches + * the number of ZAPs in the per-vdev ZAP list. This only gets + * called if the config is dirty; otherwise there may be + * outstanding AVZ operations that weren't completed in + * spa_sync_config_object. + */ + uint64_t all_vdev_zap_entry_count; + ASSERT0(zap_count(spa->spa_meta_objset, + spa->spa_all_vdev_zaps, &all_vdev_zap_entry_count)); + ASSERT3U(vdev_count_verify_zaps(spa->spa_root_vdev), ==, + all_vdev_zap_entry_count); + } +#endif + + if (spa->spa_vdev_removal != NULL) { + ASSERT0(spa->spa_vdev_removal->svr_bytes_done[txg & TXG_MASK]); + } + + spa_sync_rewrite_vdev_config(spa, tx); dmu_tx_commit(tx); VERIFY(cyclic_reprogram(spa->spa_deadman_cycid, CY_INFINITY)); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 14:01:44 2019 (r354956) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 14:02:54 2019 (r354957) @@ -3833,6 +3833,11 @@ vdev_space_update(vdev_t *vd, int64_t alloc_delta, int dspace_delta = vdev_deflated_space(vd, space_delta); mutex_enter(&vd->vdev_stat_lock); + /* ensure we won't underflow */ + if (alloc_delta < 0) { + ASSERT3U(vd->vdev_stat.vs_alloc, >=, -alloc_delta); + } + vd->vdev_stat.vs_alloc += alloc_delta; vd->vdev_stat.vs_space += space_delta; vd->vdev_stat.vs_dspace += dspace_delta; @@ -3840,6 +3845,7 @@ vdev_space_update(vdev_t *vd, int64_t alloc_delta, int /* every class but log contributes to root space stats */ if (vd->vdev_mg != NULL && !vd->vdev_islog) { + ASSERT(!vd->vdev_isl2cache); mutex_enter(&rvd->vdev_stat_lock); rvd->vdev_stat.vs_alloc += alloc_delta; rvd->vdev_stat.vs_space += space_delta; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Nov 21 14:01:44 2019 (r354956) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Nov 21 14:02:54 2019 (r354957) @@ -1688,6 +1688,9 @@ svr_sync(spa_t *spa, dmu_tx_t *tx) spa_vdev_removal_t *svr = spa->spa_vdev_removal; int txgoff = dmu_tx_get_txg(tx) & TXG_MASK; + if (svr == NULL) + return; + /* * This check is necessary so that we do not dirty the * DIRECTORY_OBJECT via spa_sync_removing_state() when there From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:04:07 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 16C9F1BFF26; Thu, 21 Nov 2019 14:04:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhBG6YTGz4Rs9; Thu, 21 Nov 2019 14:04:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C4384C618; Thu, 21 Nov 2019 14:04:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE46pf047380; Thu, 21 Nov 2019 14:04:06 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE469R047379; Thu, 21 Nov 2019 14:04:06 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211404.xALE469R047379@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:04:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354958 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 354958 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:04:07 -0000 Author: avg Date: Thu Nov 21 14:04:06 2019 New Revision: 354958 URL: https://svnweb.freebsd.org/changeset/base/354958 Log: 11051 zfs miscounts BP_IS_EMBEDDED blocks during scan. illumos/illumos-gate@ee2f9ca4ea24f72b05598c92aad7f42fb77b1345 https://github.com/illumos/illumos-gate/commit/ee2f9ca4ea24f72b05598c92aad7f42fb77b1345 https://www.illumos.org/issues/11051 Author: Bill Sommerfeld Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:02:54 2019 (r354957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:04:06 2019 (r354958) @@ -3432,6 +3432,13 @@ count_block(dsl_scan_t *scn, zfs_all_blkstats_t *zab, int i; /* + * Don't count embedded bp's, since we already did the work of + * scanning these when we scanned the containing block. + */ + if (BP_IS_EMBEDDED(bp)) + return; + + /* * Update the spa's stats on how many bytes we have issued. * Sequential scrubs create a zio for each DVA of the bp. Each * of these will include all DVAs for repair purposes, but the From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:05:16 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 86C2B1BFFD2; Thu, 21 Nov 2019 14:05:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhCc32Cjz4S0W; Thu, 21 Nov 2019 14:05:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4B685C61A; Thu, 21 Nov 2019 14:05:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE5GKG047482; Thu, 21 Nov 2019 14:05:16 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE5GHL047481; Thu, 21 Nov 2019 14:05:16 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211405.xALE5GHL047481@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:05:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354959 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 354959 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:05:16 -0000 Author: avg Date: Thu Nov 21 14:05:15 2019 New Revision: 354959 URL: https://svnweb.freebsd.org/changeset/base/354959 Log: 11056 vdev_disk_io_start will panic the system if ldi_strategy returns an error. illumos/illumos-gate@fa88c70fa54c01be933cc49fd271e5c4468f7eb8 https://github.com/illumos/illumos-gate/commit/fa88c70fa54c01be933cc49fd271e5c4468f7eb8 https://www.illumos.org/issues/11056 Author: Jerry Jelinek Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Thu Nov 21 14:04:06 2019 (r354958) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Thu Nov 21 14:05:15 2019 (r354959) @@ -22,7 +22,7 @@ * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright 2016 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2013 Joyent, Inc. All rights reserved. + * Copyright 2019 Joyent, Inc. */ #include @@ -857,8 +857,15 @@ vdev_disk_io_start(zio_t *zio) bp->b_bufsize = zio->io_size; bp->b_iodone = vdev_disk_io_intr; - /* ldi_strategy() will return non-zero only on programming errors */ - VERIFY(ldi_strategy(dvd->vd_lh, bp) == 0); + /* + * In general we would expect ldi_strategy() to return non-zero only + * because of programming errors, but we've also seen this fail shortly + * after a disk dies. + */ + if (ldi_strategy(dvd->vd_lh, bp) != 0) { + zio->io_error = ENXIO; + zio_interrupt(zio); + } } static void From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:09:47 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 879811C0142; Thu, 21 Nov 2019 14:09:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhJq30bDz4SC9; Thu, 21 Nov 2019 14:09:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4AB77C62E; Thu, 21 Nov 2019 14:09:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALE9lTW047715; Thu, 21 Nov 2019 14:09:47 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALE9lrF047714; Thu, 21 Nov 2019 14:09:47 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211409.xALE9lrF047714@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:09:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354960 - vendor/illumos/dist/man/man5 X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor/illumos/dist/man/man5 X-SVN-Commit-Revision: 354960 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:09:47 -0000 Author: avg Date: Thu Nov 21 14:09:46 2019 New Revision: 354960 URL: https://svnweb.freebsd.org/changeset/base/354960 Log: 10829 zpool-features.5 add missing .RE illumos/illumos-gate@1d775d5906fd85b9ab1ac2251af8d38b5567d9a1 https://github.com/illumos/illumos-gate/commit/1d775d5906fd85b9ab1ac2251af8d38b5567d9a1 https://www.illumos.org/issues/10829 zpool-features.5 is missing .RE before allocation_classes. Author: Toomas Soome Modified: vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Thu Nov 21 14:05:15 2019 (r354959) +++ vendor/illumos/dist/man/man5/zpool-features.5 Thu Nov 21 14:09:46 2019 (r354960) @@ -15,7 +15,7 @@ .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your .\" own identifying information: .\" Portions Copyright [yyyy] [name of copyright owner] -.TH ZPOOL-FEATURES 5 "Jun 8, 2018" +.TH ZPOOL-FEATURES 5 "Apr 19, 2019" .SH NAME zpool\-features \- ZFS pool feature descriptions .SH DESCRIPTION @@ -661,6 +661,8 @@ ever had their checksum set to \fBedonr\fR are destroy Booting off of pools using \fBedonr\fR is supported. +.RE + .sp .ne 2 .na @@ -680,8 +682,6 @@ This feature becomes \fBactive\fR when a dedicated all (dedup or special) is created with zpool create or zpool add. With device removal, it can be returned to the \fBenabled\fR state if all the top-level vdevs from an allocation class are removed. - -.RE .SH "SEE ALSO" \fBzpool\fR(1M) From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:10:56 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7B9291C01D7; Thu, 21 Nov 2019 14:10:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhL83QcSz4SLS; Thu, 21 Nov 2019 14:10:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 58D67C657; Thu, 21 Nov 2019 14:10:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALEAuUn050099; Thu, 21 Nov 2019 14:10:56 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALEArvv050085; Thu, 21 Nov 2019 14:10:53 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211410.xALEArvv050085@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:10:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354961 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendo... X-SVN-Commit-Revision: 354961 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:10:56 -0000 Author: avg Date: Thu Nov 21 14:10:53 2019 New Revision: 354961 URL: https://svnweb.freebsd.org/changeset/base/354961 Log: 10952 defer new resilvers and misc. resilver-related fixes illumos/illumos-gate@e4c795beb33bf59dd4ad2e3f88f493111484b890 https://github.com/illumos/illumos-gate/commit/e4c795beb33bf59dd4ad2e3f88f493111484b890 https://www.illumos.org/issues/10952 From ZoL 612c4930dd2 Fix the spelling of deferred ??? cef48f14da6 Remove races from scrub / resilver tests 4021ba4cfaa Make vdev_set_deferred_resilver() recursive 8cb119e3dc0 Fix 2 small bugs with cached dsl_scan_phys_t 5e0bd0ae056 Fix issue with scanning dedup blocks as scan ends b3d7725c943 Remove zfs_gitrev.h (this shouldn't be part of 80a91e74696) 80a91e74696 Defer new resilvers until the current one ends Portions contributed by: Jerry Jelinek Portions contributed by: Brian Behlendorf Portions contributed by: Arkadiusz Bubała Author: Tom Caputi Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/man/man1m/zpool.1m vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c Thu Nov 21 14:10:53 2019 (r354961) @@ -300,10 +300,13 @@ zpool_feature_init(void) "freed or remapped.", ZFEATURE_FLAG_READONLY_COMPAT, obsolete_counts_deps); - { zfeature_register(SPA_FEATURE_ALLOCATION_CLASSES, "org.zfsonlinux:allocation_classes", "allocation_classes", "Support for separate allocation classes.", ZFEATURE_FLAG_READONLY_COMPAT, NULL); - } + + zfeature_register(SPA_FEATURE_RESILVER_DEFER, + "com.datto:resilver_defer", "resilver_defer", + "Support for defering new resilvers when one is already running.", + ZFEATURE_FLAG_READONLY_COMPAT, NULL); } Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.h Thu Nov 21 14:10:53 2019 (r354961) @@ -63,6 +63,7 @@ typedef enum spa_feature { SPA_FEATURE_POOL_CHECKPOINT, SPA_FEATURE_SPACEMAP_V2, SPA_FEATURE_ALLOCATION_CLASSES, + SPA_FEATURE_RESILVER_DEFER, SPA_FEATURES } spa_feature_t; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:10:53 2019 (r354961) @@ -183,12 +183,15 @@ unsigned int zfs_free_min_time_ms = 1000; /* min milli unsigned int zfs_obsolete_min_time_ms = 500; /* min millisecs to resilver per txg */ unsigned int zfs_resilver_min_time_ms = 3000; +int zfs_scan_suspend_progress = 0; /* set to prevent scans from progressing */ boolean_t zfs_no_scrub_io = B_FALSE; /* set to disable scrub i/o */ boolean_t zfs_no_scrub_prefetch = B_FALSE; /* set to disable scrub prefetch */ enum ddt_class zfs_scrub_ddt_class_max = DDT_CLASS_DUPLICATE; /* max number of blocks to free in a single TXG */ uint64_t zfs_async_block_max_blocks = UINT64_MAX; +int zfs_resilver_disable_defer = 0; /* set to disable resilver deferring */ + /* * We wait a few txgs after importing a pool to begin scanning so that * the import / mounting code isn't held up by scrub / resilver IO. @@ -455,7 +458,6 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) scn->scn_async_destroying = spa_feature_is_active(dp->dp_spa, SPA_FEATURE_ASYNC_DESTROY); - bcopy(&scn->scn_phys, &scn->scn_phys_cached, sizeof (scn->scn_phys)); avl_create(&scn->scn_queue, scan_ds_queue_compare, sizeof (scan_ds_t), offsetof(scan_ds_t, sds_node)); avl_create(&scn->scn_prefetch_queue, scan_prefetch_queue_compare, @@ -513,6 +515,8 @@ dsl_scan_init(dsl_pool_t *dp, uint64_t txg) } } + bcopy(&scn->scn_phys, &scn->scn_phys_cached, sizeof (scn->scn_phys)); + /* reload the queue into the in-core state */ if (scn->scn_phys.scn_queue_obj != 0) { zap_cursor_t zc; @@ -751,6 +755,11 @@ dsl_scan(dsl_pool_t *dp, pool_scan_func_t func) spa->spa_scrub_reopen = B_FALSE; (void) spa_vdev_state_exit(spa, NULL, 0); + if (func == POOL_SCAN_RESILVER) { + dsl_resilver_restart(spa->spa_dsl_pool, 0); + return (0); + } + if (func == POOL_SCAN_SCRUB && dsl_scan_is_paused_scrub(scn)) { /* got scrub start cmd, resume paused scrub */ int err = dsl_scrub_set_pause_resume(scn->scn_dp, @@ -766,6 +775,41 @@ dsl_scan(dsl_pool_t *dp, pool_scan_func_t func) dsl_scan_setup_sync, &func, 0, ZFS_SPACE_CHECK_EXTRA_RESERVED)); } +/* + * Sets the resilver defer flag to B_FALSE on all leaf devs under vd. Returns + * B_TRUE if we have devices that need to be resilvered and are available to + * accept resilver I/Os. + */ +static boolean_t +dsl_scan_clear_deferred(vdev_t *vd, dmu_tx_t *tx) +{ + boolean_t resilver_needed = B_FALSE; + spa_t *spa = vd->vdev_spa; + + for (int c = 0; c < vd->vdev_children; c++) { + resilver_needed |= + dsl_scan_clear_deferred(vd->vdev_child[c], tx); + } + + if (vd == spa->spa_root_vdev && + spa_feature_is_active(spa, SPA_FEATURE_RESILVER_DEFER)) { + spa_feature_decr(spa, SPA_FEATURE_RESILVER_DEFER, tx); + vdev_config_dirty(vd); + spa->spa_resilver_deferred = B_FALSE; + return (resilver_needed); + } + + if (!vdev_is_concrete(vd) || vd->vdev_aux || + !vd->vdev_ops->vdev_op_leaf) + return (resilver_needed); + + if (vd->vdev_resilver_deferred) + vd->vdev_resilver_deferred = B_FALSE; + + return (!vdev_is_dead(vd) && !vd->vdev_offline && + vdev_resilver_needed(vd, NULL, NULL)); +} + /* ARGSUSED */ static void dsl_scan_done(dsl_scan_t *scn, boolean_t complete, dmu_tx_t *tx) @@ -865,6 +909,25 @@ dsl_scan_done(dsl_scan_t *scn, boolean_t complete, dmu * Let the async thread assess this and handle the detach. */ spa_async_request(spa, SPA_ASYNC_RESILVER_DONE); + + /* + * Clear any deferred_resilver flags in the config. + * If there are drives that need resilvering, kick + * off an asynchronous request to start resilver. + * dsl_scan_clear_deferred() may update the config + * before the resilver can restart. In the event of + * a crash during this period, the spa loading code + * will find the drives that need to be resilvered + * when the machine reboots and start the resilver then. + */ + boolean_t resilver_needed = + dsl_scan_clear_deferred(spa->spa_root_vdev, tx); + if (resilver_needed) { + spa_history_log_internal(spa, + "starting deferred resilver", tx, + "errors=%llu", spa_get_errlog_size(spa)); + spa_async_request(spa, SPA_ASYNC_RESILVER); + } } scn->scn_phys.scn_end_time = gethrestime_sec(); @@ -935,6 +998,7 @@ dsl_scrub_pause_resume_sync(void *arg, dmu_tx_t *tx) /* can't pause a scrub when there is no in-progress scrub */ spa->spa_scan_pass_scrub_pause = gethrestime_sec(); scn->scn_phys.scn_flags |= DSF_SCRUB_PAUSED; + scn->scn_phys_cached.scn_flags |= DSF_SCRUB_PAUSED; dsl_scan_sync_state(scn, tx, SYNC_CACHED); spa_event_notify(spa, NULL, NULL, ESC_ZFS_SCRUB_PAUSED); } else { @@ -949,6 +1013,7 @@ dsl_scrub_pause_resume_sync(void *arg, dmu_tx_t *tx) gethrestime_sec() - spa->spa_scan_pass_scrub_pause; spa->spa_scan_pass_scrub_pause = 0; scn->scn_phys.scn_flags &= ~DSF_SCRUB_PAUSED; + scn->scn_phys_cached.scn_flags &= ~DSF_SCRUB_PAUSED; dsl_scan_sync_state(scn, tx, SYNC_CACHED); } } @@ -2335,6 +2400,20 @@ dsl_scan_ddt_entry(dsl_scan_t *scn, enum zio_checksum if (scn->scn_phys.scn_state != DSS_SCANNING) return; + /* + * This function is special because it is the only thing + * that can add scan_io_t's to the vdev scan queues from + * outside dsl_scan_sync(). For the most part this is ok + * as long as it is called from within syncing context. + * However, dsl_scan_sync() expects that no new sio's will + * be added between when all the work for a scan is done + * and the next txg when the scan is actually marked as + * completed. This check ensures we do not issue new sio's + * during this period. + */ + if (scn->scn_done_txg != 0) + return; + for (p = 0; p < DDT_PHYS_TYPES; p++, ddp++) { if (ddp->ddp_phys_birth == 0 || ddp->ddp_phys_birth > scn->scn_phys.scn_max_txg) @@ -2986,6 +3065,26 @@ dsl_scan_active(dsl_scan_t *scn) } static boolean_t +dsl_scan_check_deferred(vdev_t *vd) +{ + boolean_t need_resilver = B_FALSE; + + for (int c = 0; c < vd->vdev_children; c++) { + need_resilver |= + dsl_scan_check_deferred(vd->vdev_child[c]); + } + + if (!vdev_is_concrete(vd) || vd->vdev_aux || + !vd->vdev_ops->vdev_op_leaf) + return (need_resilver); + + if (!vd->vdev_resilver_deferred) + need_resilver = B_TRUE; + + return (need_resilver); +} + +static boolean_t dsl_scan_need_resilver(spa_t *spa, const dva_t *dva, size_t psize, uint64_t phys_birth) { @@ -3032,6 +3131,13 @@ dsl_scan_need_resilver(spa_t *spa, const dva_t *dva, s if (!vdev_dtl_need_resilver(vd, DVA_GET_OFFSET(dva), psize)) return (B_FALSE); + /* + * Check that this top-level vdev has a device under it which + * is resilvering and is not deferred. + */ + if (!dsl_scan_check_deferred(vd)) + return (B_FALSE); + return (B_TRUE); } @@ -3193,12 +3299,19 @@ dsl_scan_sync(dsl_pool_t *dp, dmu_tx_t *tx) int err = 0; state_sync_type_t sync_type = SYNC_OPTIONAL; + if (spa->spa_resilver_deferred && + !spa_feature_is_active(dp->dp_spa, SPA_FEATURE_RESILVER_DEFER)) + spa_feature_incr(spa, SPA_FEATURE_RESILVER_DEFER, tx); + /* * Check for scn_restart_txg before checking spa_load_state, so * that we can restart an old-style scan while the pool is being - * imported (see dsl_scan_init). + * imported (see dsl_scan_init). We also restart scans if there + * is a deferred resilver and the user has manually disabled + * deferred resilvers via the tunable. */ - if (dsl_scan_restarting(scn, tx)) { + if (dsl_scan_restarting(scn, tx) || + (spa->spa_resilver_deferred && zfs_resilver_disable_defer)) { pool_scan_func_t func = POOL_SCAN_SCRUB; dsl_scan_done(scn, B_FALSE, tx); if (vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL)) @@ -3265,6 +3378,27 @@ dsl_scan_sync(dsl_pool_t *dp, dmu_tx_t *tx) return; /* + * zfs_scan_suspend_progress can be set to disable scan progress. + * We don't want to spin the txg_sync thread, so we add a delay + * here to simulate the time spent doing a scan. This is mostly + * useful for testing and debugging. + */ + if (zfs_scan_suspend_progress) { + uint64_t scan_time_ns = gethrtime() - scn->scn_sync_start_time; + int mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ? + zfs_resilver_min_time_ms : zfs_scrub_min_time_ms; + + while (zfs_scan_suspend_progress && + !txg_sync_waiting(scn->scn_dp) && + !spa_shutting_down(scn->scn_dp->dp_spa) && + NSEC2MSEC(scan_time_ns) < mintime) { + delay(hz); + scan_time_ns = gethrtime() - scn->scn_sync_start_time; + } + return; + } + + /* * It is possible to switch from unsorted to sorted at any time, * but afterwards the scan will remain sorted unless reloaded from * a checkpoint after a reboot. @@ -3393,6 +3527,8 @@ dsl_scan_sync(dsl_pool_t *dp, dmu_tx_t *tx) (longlong_t)tx->tx_txg); } } else if (scn->scn_is_sorted && scn->scn_bytes_pending != 0) { + ASSERT(scn->scn_clearing); + /* need to issue scrubbing IOs from per-vdev queues */ scn->scn_zio_root = zio_root(dp->dp_spa, NULL, NULL, ZIO_FLAG_CANFAIL); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Nov 21 14:10:53 2019 (r354961) @@ -6176,9 +6176,14 @@ spa_vdev_attach(spa_t *spa, uint64_t guid, nvlist_t *n /* * Schedule the resilver to restart in the future. We do this to * ensure that dmu_sync-ed blocks have been stitched into the - * respective datasets. + * respective datasets. We do not do this if resilvers have been + * deferred. */ - dsl_resilver_restart(spa->spa_dsl_pool, dtl_max_txg); + if (dsl_scan_resilvering(spa_get_dsl(spa)) && + spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER)) + vdev_set_deferred_resilver(spa, newvd); + else + dsl_resilver_restart(spa->spa_dsl_pool, dtl_max_txg); if (spa->spa_bootfs) spa_event_notify(spa, newvd, NULL, ESC_ZFS_BOOTFS_VDEV_ATTACH); @@ -7069,6 +7074,10 @@ spa_scan(spa_t *spa, pool_scan_func_t func) if (func >= POOL_SCAN_FUNCS || func == POOL_SCAN_NONE) return (SET_ERROR(ENOTSUP)); + if (func == POOL_SCAN_RESILVER && + !spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER)) + return (SET_ERROR(ENOTSUP)); + /* * If a resilver was requested, but there is no DTL on a * writeable leaf device, we have nothing to do. @@ -7160,6 +7169,7 @@ static void spa_async_thread(void *arg) { spa_t *spa = (spa_t *)arg; + dsl_pool_t *dp = spa->spa_dsl_pool; int tasks; ASSERT(spa->spa_sync_on); @@ -7235,8 +7245,10 @@ spa_async_thread(void *arg) /* * Kick off a resilver. */ - if (tasks & SPA_ASYNC_RESILVER) - dsl_resilver_restart(spa->spa_dsl_pool, 0); + if (tasks & SPA_ASYNC_RESILVER && + (!dsl_scan_resilvering(dp) || + !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_RESILVER_DEFER))) + dsl_resilver_restart(dp, 0); if (tasks & SPA_ASYNC_INITIALIZE_RESTART) { mutex_enter(&spa_namespace_lock); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Thu Nov 21 14:10:53 2019 (r354961) @@ -279,6 +279,13 @@ struct spa { uint64_t spa_scan_pass_scrub_spent_paused; /* total paused */ uint64_t spa_scan_pass_exam; /* examined bytes per pass */ uint64_t spa_scan_pass_issued; /* issued bytes per pass */ + + /* + * We are in the middle of a resilver, and another resilver + * is needed once this one completes. This is set iff any + * vdev_resilver_deferred is set. + */ + boolean_t spa_resilver_deferred; kmutex_t spa_async_lock; /* protect async state */ kthread_t *spa_async_thread; /* thread doing async task */ int spa_async_suspended; /* async tasks suspended */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Thu Nov 21 14:10:53 2019 (r354961) @@ -149,6 +149,8 @@ extern int vdev_config_sync(vdev_t **svd, int svdcount extern void vdev_state_dirty(vdev_t *vd); extern void vdev_state_clean(vdev_t *vd); +extern void vdev_set_deferred_resilver(spa_t *spa, vdev_t *vd); + typedef enum vdev_config_flag { VDEV_CONFIG_SPARE = 1 << 0, VDEV_CONFIG_L2CACHE = 1 << 1, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Thu Nov 21 14:10:53 2019 (r354961) @@ -346,6 +346,7 @@ struct vdev { boolean_t vdev_cant_write; /* vdev is failing all writes */ boolean_t vdev_isspare; /* was a hot spare */ boolean_t vdev_isl2cache; /* was a l2cache device */ + boolean_t vdev_resilver_deferred; /* resilver deferred */ vdev_queue_t vdev_queue; /* I/O deadline schedule queue */ vdev_cache_t vdev_cache; /* physical block cache */ spa_aux_vdev_t *vdev_aux; /* for l2cache and spares vdevs */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Nov 21 14:10:53 2019 (r354961) @@ -760,6 +760,9 @@ vdev_alloc(spa_t *spa, vdev_t **vdp, nvlist_t *nv, vde (void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_RESILVER_TXG, &vd->vdev_resilver_txg); + if (nvlist_exists(nv, ZPOOL_CONFIG_RESILVER_DEFER)) + vdev_set_deferred_resilver(spa, vd); + /* * When importing a pool, we want to ignore the persistent fault * state, as the diagnosis made on another system may not be @@ -1733,8 +1736,13 @@ vdev_open(vdev_t *vd) * since this would just restart the scrub we are already doing. */ if (vd->vdev_ops->vdev_op_leaf && !spa->spa_scrub_reopen && - vdev_resilver_needed(vd, NULL, NULL)) - spa_async_request(spa, SPA_ASYNC_RESILVER); + vdev_resilver_needed(vd, NULL, NULL)) { + if (dsl_scan_resilvering(spa->spa_dsl_pool) && + spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER)) + vdev_set_deferred_resilver(spa, vd); + else + spa_async_request(spa, SPA_ASYNC_RESILVER); + } return (0); } @@ -2441,6 +2449,9 @@ vdev_dtl_should_excise(vdev_t *vd) if (vd->vdev_state < VDEV_STATE_DEGRADED) return (B_FALSE); + if (vd->vdev_resilver_deferred) + return (B_FALSE); + if (vd->vdev_resilver_txg == 0 || range_tree_is_empty(vd->vdev_dtl[DTL_MISSING])) return (B_TRUE); @@ -3474,8 +3485,14 @@ vdev_clear(spa_t *spa, vdev_t *vd) if (vd != rvd && vdev_writeable(vd->vdev_top)) vdev_state_dirty(vd->vdev_top); - if (vd->vdev_aux == NULL && !vdev_is_dead(vd)) - spa_async_request(spa, SPA_ASYNC_RESILVER); + if (vd->vdev_aux == NULL && !vdev_is_dead(vd)) { + if (dsl_scan_resilvering(spa->spa_dsl_pool) && + spa_feature_is_enabled(spa, + SPA_FEATURE_RESILVER_DEFER)) + vdev_set_deferred_resilver(spa, vd); + else + spa_async_request(spa, SPA_ASYNC_RESILVER); + } spa_event_notify(spa, vd, NULL, ESC_ZFS_VDEV_CLEAR); } @@ -3618,6 +3635,8 @@ vdev_get_stats(vdev_t *vd, vdev_stat_t *vs) vs->vs_fragmentation = (vd->vdev_mg != NULL) ? vd->vdev_mg->mg_fragmentation : 0; } + if (vd->vdev_ops->vdev_op_leaf) + vs->vs_resilver_deferred = vd->vdev_resilver_deferred; /* * If we're getting stats on the root vdev, aggregate the I/O counts @@ -4330,4 +4349,19 @@ vdev_deadman(vdev_t *vd) } mutex_exit(&vq->vq_lock); } +} + +void +vdev_set_deferred_resilver(spa_t *spa, vdev_t *vd) +{ + for (uint64_t i = 0; i < vd->vdev_children; i++) + vdev_set_deferred_resilver(spa, vd->vdev_child[i]); + + if (!vd->vdev_ops->vdev_op_leaf || !vdev_writeable(vd) || + range_tree_is_empty(vd->vdev_dtl[DTL_MISSING])) { + return; + } + + vd->vdev_resilver_deferred = B_TRUE; + spa->spa_resilver_deferred = B_TRUE; } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Nov 21 14:10:53 2019 (r354961) @@ -1239,6 +1239,8 @@ vdev_indirect_read_all(zio_t *zio) { indirect_vsd_t *iv = zio->io_vsd; + ASSERT3U(zio->io_type, ==, ZIO_TYPE_READ); + for (indirect_split_t *is = list_head(&iv->iv_splits); is != NULL; is = list_next(&iv->iv_splits, is)) { for (int i = 0; i < is->is_children; i++) { @@ -1321,7 +1323,8 @@ vdev_indirect_io_start(zio_t *zio) vdev_indirect_child_io_done, zio)); } else { iv->iv_split_block = B_TRUE; - if (zio->io_flags & (ZIO_FLAG_SCRUB | ZIO_FLAG_RESILVER)) { + if (zio->io_type == ZIO_TYPE_READ && + zio->io_flags & (ZIO_FLAG_SCRUB | ZIO_FLAG_RESILVER)) { /* * Read all copies. Note that for simplicity, * we don't bother consulting the DTL in the @@ -1330,13 +1333,17 @@ vdev_indirect_io_start(zio_t *zio) vdev_indirect_read_all(zio); } else { /* - * Read one copy of each split segment, from the - * top-level vdev. Since we don't know the - * checksum of each split individually, the child - * zio can't ensure that we get the right data. - * E.g. if it's a mirror, it will just read from a - * random (healthy) leaf vdev. We have to verify - * the checksum in vdev_indirect_io_done(). + * If this is a read zio, we read one copy of each + * split segment, from the top-level vdev. Since + * we don't know the checksum of each split + * individually, the child zio can't ensure that + * we get the right data. E.g. if it's a mirror, + * it will just read from a random (healthy) leaf + * vdev. We have to verify the checksum in + * vdev_indirect_io_done(). + * + * For write zios, the vdev code will ensure we write + * to all children. */ for (indirect_split_t *is = list_head(&iv->iv_splits); is != NULL; is = list_next(&iv->iv_splits, is)) { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Thu Nov 21 14:10:53 2019 (r354961) @@ -377,6 +377,12 @@ vdev_config_generate(spa_t *spa, vdev_t *vd, boolean_t fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_TOP_ZAP, vd->vdev_top_zap); } + + if (vd->vdev_resilver_deferred) { + ASSERT(vd->vdev_ops->vdev_op_leaf); + ASSERT(spa->spa_resilver_deferred); + fnvlist_add_boolean(nv, ZPOOL_CONFIG_RESILVER_DEFER); + } } if (getstats) { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Nov 21 14:10:53 2019 (r354961) @@ -127,7 +127,7 @@ int vdev_removal_max_span = 32 * 1024; * This is used by the test suite so that it can ensure that certain * actions happen while in the middle of a removal. */ -uint64_t zfs_remove_max_bytes_pause = UINT64_MAX; +int zfs_removal_suspend_progress = 0; #define VDEV_REMOVAL_ZAP_OBJS "lzap" @@ -1433,14 +1433,14 @@ spa_vdev_remove_thread(void *arg) /* * This delay will pause the removal around the point - * specified by zfs_remove_max_bytes_pause. We do this + * specified by zfs_removal_suspend_progress. We do this * solely from the test suite or during debugging. */ uint64_t bytes_copied = spa->spa_removing_phys.sr_copied; for (int i = 0; i < TXG_SIZE; i++) bytes_copied += svr->svr_bytes_done[i]; - while (zfs_remove_max_bytes_pause <= bytes_copied && + while (zfs_removal_suspend_progress && !svr->svr_thread_exit) delay(hz); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Nov 21 14:10:53 2019 (r354961) @@ -1252,7 +1252,7 @@ zil_lwb_set_zio_dependency(zilog_t *zilog, lwb_t *lwb) * root zios). This is required because of how we can * defer the DKIOCFLUSHWRITECACHE commands for each lwb. * - * When the DKIOCFLUSHWRITECACHE commands are defered, + * When the DKIOCFLUSHWRITECACHE commands are deferred, * the previous lwb will rely on this lwb to flush the * vdevs written to by that previous lwb. Thus, we need * to ensure this lwb doesn't issue the flush until Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Thu Nov 21 14:10:53 2019 (r354961) @@ -597,6 +597,7 @@ typedef struct zpool_load_policy { #define ZPOOL_CONFIG_VDEV_TOP_ZAP "com.delphix:vdev_zap_top" #define ZPOOL_CONFIG_VDEV_LEAF_ZAP "com.delphix:vdev_zap_leaf" #define ZPOOL_CONFIG_HAS_PER_VDEV_ZAPS "com.delphix:has_per_vdev_zaps" +#define ZPOOL_CONFIG_RESILVER_DEFER "com.datto:resilver_defer" #define ZPOOL_CONFIG_CACHEFILE "cachefile" /* not stored on disk */ #define ZPOOL_CONFIG_MMP_STATE "mmp_state" /* not stored on disk */ #define ZPOOL_CONFIG_MMP_TXG "mmp_txg" /* not stored on disk */ @@ -896,6 +897,7 @@ typedef struct vdev_stat { uint64_t vs_initialize_state; /* vdev_initialzing_state_t */ uint64_t vs_initialize_action_time; /* time_t */ uint64_t vs_checkpoint_space; /* checkpoint-consumed space */ + uint64_t vs_resilver_deferred; /* resilver deferred */ } vdev_stat_t; /* From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:10:58 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 178AF1C01DD; Thu, 21 Nov 2019 14:10:58 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhL96rHDz4SLX; Thu, 21 Nov 2019 14:10:57 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CF53AC65B; Thu, 21 Nov 2019 14:10:57 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALEAvw7050112; Thu, 21 Nov 2019 14:10:57 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALEAuL1050104; Thu, 21 Nov 2019 14:10:56 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211410.xALEAuL1050104@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:10:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354961 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendo... X-SVN-Commit-Revision: 354961 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:10:58 -0000 Author: avg Date: Thu Nov 21 14:10:53 2019 New Revision: 354961 URL: https://svnweb.freebsd.org/changeset/base/354961 Log: 10952 defer new resilvers and misc. resilver-related fixes illumos/illumos-gate@e4c795beb33bf59dd4ad2e3f88f493111484b890 https://github.com/illumos/illumos-gate/commit/e4c795beb33bf59dd4ad2e3f88f493111484b890 https://www.illumos.org/issues/10952 From ZoL 612c4930dd2 Fix the spelling of deferred ??? cef48f14da6 Remove races from scrub / resilver tests 4021ba4cfaa Make vdev_set_deferred_resilver() recursive 8cb119e3dc0 Fix 2 small bugs with cached dsl_scan_phys_t 5e0bd0ae056 Fix issue with scanning dedup blocks as scan ends b3d7725c943 Remove zfs_gitrev.h (this shouldn't be part of 80a91e74696) 80a91e74696 Defer new resilvers until the current one ends Portions contributed by: Jerry Jelinek Portions contributed by: Brian Behlendorf Portions contributed by: Arkadiusz Bubała Author: Tom Caputi Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/man/man1m/zpool.1m vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Nov 21 14:10:53 2019 (r354961) @@ -88,6 +88,7 @@ static int zpool_do_split(int, char **); static int zpool_do_initialize(int, char **); static int zpool_do_scrub(int, char **); +static int zpool_do_resilver(int, char **); static int zpool_do_import(int, char **); static int zpool_do_export(int, char **); @@ -140,6 +141,7 @@ typedef enum { HELP_REMOVE, HELP_INITIALIZE, HELP_SCRUB, + HELP_RESILVER, HELP_STATUS, HELP_UPGRADE, HELP_GET, @@ -192,6 +194,7 @@ static zpool_command_t command_table[] = { { "split", zpool_do_split, HELP_SPLIT }, { NULL }, { "initialize", zpool_do_initialize, HELP_INITIALIZE }, + { "resilver", zpool_do_resilver, HELP_RESILVER }, { "scrub", zpool_do_scrub, HELP_SCRUB }, { NULL }, { "import", zpool_do_import, HELP_IMPORT }, @@ -273,6 +276,8 @@ get_usage(zpool_help_t idx) return (gettext("\tinitialize [-cs] [ ...]\n")); case HELP_SCRUB: return (gettext("\tscrub [-s | -p] ...\n")); + case HELP_RESILVER: + return (gettext("\tresilver ...\n")); case HELP_STATUS: return (gettext("\tstatus [-DgLPvx] [-T d|u] [pool] ... " "[interval [count]]\n")); @@ -1688,11 +1693,14 @@ print_status_config(zpool_handle_t *zhp, status_cbdata (void) nvlist_lookup_uint64_array(nv, ZPOOL_CONFIG_SCAN_STATS, (uint64_t **)&ps, &c); - if (ps != NULL && ps->pss_state == DSS_SCANNING && - vs->vs_scan_processed != 0 && children == 0) { - (void) printf(gettext(" (%s)"), - (ps->pss_func == POOL_SCAN_RESILVER) ? - "resilvering" : "repairing"); + if (ps != NULL && ps->pss_state == DSS_SCANNING && children == 0) { + if (vs->vs_scan_processed != 0) { + (void) printf(gettext(" (%s)"), + (ps->pss_func == POOL_SCAN_RESILVER) ? + "resilvering" : "repairing"); + } else if (vs->vs_resilver_deferred) { + (void) printf(gettext(" (awaiting resilver)")); + } } if ((vs->vs_initialize_state == VDEV_INITIALIZE_ACTIVE || @@ -4519,7 +4527,7 @@ scrub_callback(zpool_handle_t *zhp, void *data) * Ignore faulted pools. */ if (zpool_get_state(zhp) == POOL_STATE_UNAVAIL) { - (void) fprintf(stderr, gettext("cannot scrub '%s': pool is " + (void) fprintf(stderr, gettext("cannot scan '%s': pool is " "currently unavailable\n"), zpool_get_name(zhp)); return (1); } @@ -4576,6 +4584,43 @@ zpool_do_scrub(int argc, char **argv) cb.cb_argc = argc; cb.cb_argv = argv; + argc -= optind; + argv += optind; + + if (argc < 1) { + (void) fprintf(stderr, gettext("missing pool name argument\n")); + usage(B_FALSE); + } + + return (for_each_pool(argc, argv, B_TRUE, NULL, scrub_callback, &cb)); +} + +/* + * zpool resilver ... + * + * Restarts any in-progress resilver + */ +int +zpool_do_resilver(int argc, char **argv) +{ + int c; + scrub_cbdata_t cb; + + cb.cb_type = POOL_SCAN_RESILVER; + cb.cb_scrub_cmd = POOL_SCRUB_NORMAL; + cb.cb_argc = argc; + cb.cb_argv = argv; + + /* check options */ + while ((c = getopt(argc, argv, "")) != -1) { + switch (c) { + case '?': + (void) fprintf(stderr, gettext("invalid option '%c'\n"), + optopt); + usage(B_FALSE); + } + } + argc -= optind; argv += optind; Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs.h Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/lib/libzfs/common/libzfs.h Thu Nov 21 14:10:53 2019 (r354961) @@ -140,6 +140,7 @@ typedef enum zfs_error { EZFS_TOOMANY, /* argument list too long */ EZFS_INITIALIZING, /* currently initializing */ EZFS_NO_INITIALIZE, /* no active initialize */ + EZFS_NO_RESILVER_DEFER, /* pool doesn't support resilver_defer */ EZFS_UNKNOWN } zfs_error_t; Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Thu Nov 21 14:10:53 2019 (r354961) @@ -2034,6 +2034,10 @@ zpool_scan(zpool_handle_t *zhp, pool_scan_func_t func, (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN, "cannot scrub %s"), zc.zc_name); } + } else if (func == POOL_SCAN_RESILVER) { + assert(cmd == POOL_SCRUB_NORMAL); + (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN, + "cannot restart resilver on %s"), zc.zc_name); } else if (func == POOL_SCAN_NONE) { (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN, "cannot cancel scrubbing %s"), @@ -2061,6 +2065,8 @@ zpool_scan(zpool_handle_t *zhp, pool_scan_func_t func, } } else if (err == ENOENT) { return (zfs_error(hdl, EZFS_NO_SCRUB, msg)); + } else if (err == ENOTSUP && func == POOL_SCAN_RESILVER) { + return (zfs_error(hdl, EZFS_NO_RESILVER_DEFER, msg)); } else { return (zpool_standard_error(hdl, err, msg)); } Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_util.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Thu Nov 21 14:10:53 2019 (r354961) @@ -260,6 +260,9 @@ libzfs_error_description(libzfs_handle_t *hdl) case EZFS_NO_INITIALIZE: return (dgettext(TEXT_DOMAIN, "there is no active " "initialization")); + case EZFS_NO_RESILVER_DEFER: + return (dgettext(TEXT_DOMAIN, "this action requires the " + "resilver_defer feature")); case EZFS_UNKNOWN: return (dgettext(TEXT_DOMAIN, "unknown error")); default: Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/man/man1m/zpool.1m Thu Nov 21 14:10:53 2019 (r354961) @@ -26,7 +26,7 @@ .\" Copyright (c) 2017 George Melikov. All Rights Reserved. .\" Copyright 2019 Joyent, Inc. .\" -.Dd April 27, 2018 +.Dd May 15, 2019 .Dt ZPOOL 1M .Os .Sh NAME @@ -156,6 +156,9 @@ .Op Fl f .Ar pool Ar device Op Ar new_device .Nm +.Cm resilver +.Ar pool Ns ... +.Nm .Cm scrub .Op Fl s | Fl p .Ar pool Ns ... @@ -1774,6 +1777,19 @@ Forces use of even if its appears to be in use. Not all devices can be overridden in this manner. .El +.It Xo +.Nm +.Cm resilver +.Ar pool Ns ... +.Xc +Starts a resilver. +If an existing resilver is already running it will be restarted from the +beginning. +Any drives that were scheduled for a deferred resilver will be added to the +new one. +This requires the +.Sy resilver_defer +feature. .It Xo .Nm .Cm scrub Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Thu Nov 21 14:09:46 2019 (r354960) +++ vendor/illumos/dist/man/man5/zpool-features.5 Thu Nov 21 14:10:53 2019 (r354961) @@ -15,7 +15,7 @@ .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your .\" own identifying information: .\" Portions Copyright [yyyy] [name of copyright owner] -.TH ZPOOL-FEATURES 5 "Apr 19, 2019" +.TH ZPOOL-FEATURES 5 "May 15, 2019" .SH NAME zpool\-features \- ZFS pool feature descriptions .SH DESCRIPTION @@ -682,6 +682,27 @@ This feature becomes \fBactive\fR when a dedicated all (dedup or special) is created with zpool create or zpool add. With device removal, it can be returned to the \fBenabled\fR state if all the top-level vdevs from an allocation class are removed. + +.RE +.sp +.ne 2 +.na +\fB\fBresilver_defer\fR\fR +.ad +.RS 4n +.TS +l l . +GUID com.datto:resilver_defer +READ\-ONLY COMPATIBLE yes +DEPENDENCIES none +.TE + +This feature allows zfs to postpone new resilvers if an existing one is already +in progress. Without this feature, any new resilvers will cause the currently +running one to be immediately restarted from the beginning. + +This feature becomes \fBactive\fR once a resilver has been deferred, and +returns to being \fBenabled\fR when the deferred resilver begins. .SH "SEE ALSO" \fBzpool\fR(1M) From owner-svn-src-vendor@freebsd.org Thu Nov 21 14:14:08 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EFC801C04C5; Thu, 21 Nov 2019 14:14:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JhPr623Lz4Sqs; Thu, 21 Nov 2019 14:14:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B1BADC7E8; Thu, 21 Nov 2019 14:14:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xALEE81Z053469; Thu, 21 Nov 2019 14:14:08 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xALEE8hY053468; Thu, 21 Nov 2019 14:14:08 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201911211414.xALEE8hY053468@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 21 Nov 2019 14:14:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r354962 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 354962 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 14:14:09 -0000 Author: avg Date: Thu Nov 21 14:14:08 2019 New Revision: 354962 URL: https://svnweb.freebsd.org/changeset/base/354962 Log: disabled resilver_defer feature leads to looping resilvers illumos/illumos-gate@233f6c49954dadfb21fa0809febd15e2160e0ff5 https://github.com/illumos/illumos-gate/commit/233f6c49954dadfb21fa0809febd15e2160e0ff5 Note: unusually for illumos this commit does not have a bug ID. Author: Kody Kantor Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:10:53 2019 (r354961) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Nov 21 14:14:08 2019 (r354962) @@ -23,7 +23,7 @@ * Copyright (c) 2011, 2018 by Delphix. All rights reserved. * Copyright 2016 Gary Mills * Copyright (c) 2011, 2017 by Delphix. All rights reserved. - * Copyright 2017 Joyent, Inc. + * Copyright 2019 Joyent, Inc. * Copyright (c) 2017 Datto Inc. */ @@ -920,13 +920,15 @@ dsl_scan_done(dsl_scan_t *scn, boolean_t complete, dmu * will find the drives that need to be resilvered * when the machine reboots and start the resilver then. */ - boolean_t resilver_needed = - dsl_scan_clear_deferred(spa->spa_root_vdev, tx); - if (resilver_needed) { - spa_history_log_internal(spa, - "starting deferred resilver", tx, - "errors=%llu", spa_get_errlog_size(spa)); - spa_async_request(spa, SPA_ASYNC_RESILVER); + if (spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER)) { + boolean_t resilver_needed = + dsl_scan_clear_deferred(spa->spa_root_vdev, tx); + if (resilver_needed) { + spa_history_log_internal(spa, + "starting deferred resilver", tx, + "errors=%llu", spa_get_errlog_size(spa)); + spa_async_request(spa, SPA_ASYNC_RESILVER); + } } }