From owner-svn-src-vendor@freebsd.org Sun Feb 18 01:21:55 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9C77F25624; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3D8F37928D; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3843624A15; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1I1LsCV091354; Sun, 18 Feb 2018 01:21:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1I1LqQd091337; Sun, 18 Feb 2018 01:21:52 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802180121.w1I1LqQd091337@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sun, 18 Feb 2018 01:21:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329502 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb vendor/... X-SVN-Commit-Revision: 329502 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Feb 2018 01:21:55 -0000 Author: mav Date: Sun Feb 18 01:21:52 2018 New Revision: 329502 URL: https://svnweb.freebsd.org/changeset/base/329502 Log: 7614 zfs device evacuation/removal illumos/illumos-gate@5cabbc6b49070407fb9610cfe73d4c0e0dea3e77 https://www.illumos.org/issues/7614: This project allows top-level vdevs to be removed from the storage pool with “zpool remove”, reducing the total amount of storage in the pool. This operation copies all allocated regions of the device to be removed onto other devices, recording the mapping from old to new location. After the removal is complete, read and free operations to the removed (now “indirect”) vdev must be remapped and performed at the new location on disk. The indirect mapping table is kept in memory whenever the pool is loaded, so there is minimal performance overhead when doing operations on the indirect vdev. The size of the in-memory mapping table will be reduced when its entries become “obsolete” because they are no longer used by any block pointers in the pool. An entry becomes obsolete when all the blocks that use it are freed. An entry can also become obsolete when all the snapshots that reference it are deleted, and the block pointers that reference it have been “remapped” in all filesystems/zvols (and clones). Whenever an indirect block is written, all the block pointers in it will be “remapped” to their new (concrete) locations if possible. This process can be accelerated by using the “zfs remap” command to proactively rewrite all indirect blocks that reference indirect (removed) vdevs. Note that when a device is removed, we do not verify the checksum of the data that is copied. This makes the process much faster, but if it were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be possible to copy the wrong data, when we have the correct data on e.g. the other side of the mirror. Therefore, mirror and raidz devices can not be removed. Reviewed by: Alex Reece Reviewed by: George Wilson Reviewed by: John Kennedy Reviewed by: Prakash Surya Reviewed by: Matthew Ahrens Reviewed by: Richard Laager Reviewed by: Tim Chase Approved by: Garrett D'Amore Author: Prashanth Sreenivasa Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_indirect_births.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_indirect_mapping.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_births.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/common/zfs/zfs_deleg.c vendor-sys/illumos/dist/common/zfs/zfs_deleg.h vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/bpobj.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_zfetch.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/bpobj.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_deadlist.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_deleg.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dir.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_scan.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/man/man1m/zfs.1m vendor/illumos/dist/man/man1m/zpool.1m vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c Sun Feb 18 01:21:52 2018 (r329502) @@ -246,4 +246,20 @@ zpool_feature_init(void) "org.illumos:edonr", "edonr", "Edon-R hash algorithm.", ZFEATURE_FLAG_PER_DATASET, NULL); + + zfeature_register(SPA_FEATURE_DEVICE_REMOVAL, + "com.delphix:device_removal", "device_removal", + "Top-level vdevs can be removed, reducing logical pool size.", + ZFEATURE_FLAG_MOS, NULL); + + static const spa_feature_t obsolete_counts_deps[] = { + SPA_FEATURE_EXTENSIBLE_DATASET, + SPA_FEATURE_DEVICE_REMOVAL, + SPA_FEATURE_NONE + }; + zfeature_register(SPA_FEATURE_OBSOLETE_COUNTS, + "com.delphix:obsolete_counts", "obsolete_counts", + "Reduce memory used by removed devices when their blocks are " + "freed or remapped.", + ZFEATURE_FLAG_READONLY_COMPAT, obsolete_counts_deps); } Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.h Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.h Sun Feb 18 01:21:52 2018 (r329502) @@ -56,6 +56,8 @@ typedef enum spa_feature { SPA_FEATURE_SHA512, SPA_FEATURE_SKEIN, SPA_FEATURE_EDONR, + SPA_FEATURE_DEVICE_REMOVAL, + SPA_FEATURE_OBSOLETE_COUNTS, SPA_FEATURES } spa_feature_t; Modified: vendor-sys/illumos/dist/common/zfs/zfs_deleg.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_deleg.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/common/zfs/zfs_deleg.c Sun Feb 18 01:21:52 2018 (r329502) @@ -21,7 +21,7 @@ /* * Copyright (c) 2007, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2010 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2015 by Delphix. All rights reserved. * Copyright 2016 Igor Kozhukhov */ @@ -53,6 +53,7 @@ zfs_deleg_perm_tab_t zfs_deleg_perm_tab[] = { {ZFS_DELEG_PERM_MOUNT}, {ZFS_DELEG_PERM_PROMOTE}, {ZFS_DELEG_PERM_RECEIVE}, + {ZFS_DELEG_PERM_REMAP}, {ZFS_DELEG_PERM_RENAME}, {ZFS_DELEG_PERM_ROLLBACK}, {ZFS_DELEG_PERM_SNAPSHOT}, Modified: vendor-sys/illumos/dist/common/zfs/zfs_deleg.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_deleg.h Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/common/zfs/zfs_deleg.h Sun Feb 18 01:21:52 2018 (r329502) @@ -21,7 +21,7 @@ /* * Copyright (c) 2007, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2010 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2015 by Delphix. All rights reserved. */ #ifndef _ZFS_DELEG_H @@ -67,6 +67,7 @@ typedef enum { ZFS_DELEG_NOTE_RELEASE, ZFS_DELEG_NOTE_DIFF, ZFS_DELEG_NOTE_BOOKMARK, + ZFS_DELEG_NOTE_REMAP, ZFS_DELEG_NOTE_NONE } zfs_deleg_note_t; Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_prop.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/common/zfs/zfs_prop.c Sun Feb 18 01:21:52 2018 (r329502) @@ -420,6 +420,8 @@ zfs_prop_init(void) /* hidden properties */ zprop_register_hidden(ZFS_PROP_CREATETXG, "createtxg", PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_DATASET | ZFS_TYPE_BOOKMARK, "CREATETXG"); + zprop_register_hidden(ZFS_PROP_REMAPTXG, "remaptxg", PROP_TYPE_NUMBER, + PROP_READONLY, ZFS_TYPE_DATASET, "REMAPTXG"); zprop_register_hidden(ZFS_PROP_NUMCLONES, "numclones", PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_SNAPSHOT, "NUMCLONES"); zprop_register_hidden(ZFS_PROP_NAME, "name", PROP_TYPE_STRING, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Sun Feb 18 01:21:52 2018 (r329502) @@ -5009,7 +5009,7 @@ top: devw = hdr->b_l2hdr.b_dev->l2ad_writing; addr = hdr->b_l2hdr.b_daddr; /* - * Lock out device removal. + * Lock out L2ARC device removal. */ if (vdev_is_dead(vd) || !spa_config_tryenter(spa, SCL_L2ARC, vd, RW_READER)) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/bpobj.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/bpobj.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/bpobj.c Sun Feb 18 01:21:52 2018 (r329502) @@ -176,6 +176,12 @@ bpobj_open(bpobj_t *bpo, objset_t *os, uint64_t object return (0); } +boolean_t +bpobj_is_open(const bpobj_t *bpo) +{ + return (bpo->bpo_object != 0); +} + void bpobj_close(bpobj_t *bpo) { @@ -194,11 +200,11 @@ bpobj_close(bpobj_t *bpo) mutex_destroy(&bpo->bpo_lock); } -static boolean_t -bpobj_hasentries(bpobj_t *bpo) +boolean_t +bpobj_is_empty(bpobj_t *bpo) { - return (bpo->bpo_phys->bpo_num_blkptrs != 0 || - (bpo->bpo_havesubobj && bpo->bpo_phys->bpo_num_subobjs != 0)); + return (bpo->bpo_phys->bpo_num_blkptrs == 0 && + (!bpo->bpo_havesubobj || bpo->bpo_phys->bpo_num_subobjs == 0)); } static int @@ -211,11 +217,9 @@ bpobj_iterate_impl(bpobj_t *bpo, bpobj_itor_t func, vo int err = 0; dmu_buf_t *dbuf = NULL; + ASSERT(bpobj_is_open(bpo)); mutex_enter(&bpo->bpo_lock); - if (!bpobj_hasentries(bpo)) - goto out; - if (free) dmu_buf_will_dirty(bpo->bpo_dbuf, tx); @@ -345,7 +349,7 @@ bpobj_iterate_impl(bpobj_t *bpo, bpobj_itor_t func, vo out: /* If there are no entries, there should be no bytes. */ - if (!bpobj_hasentries(bpo)) { + if (bpobj_is_empty(bpo)) { ASSERT0(bpo->bpo_phys->bpo_bytes); ASSERT0(bpo->bpo_phys->bpo_comp); ASSERT0(bpo->bpo_phys->bpo_uncomp); @@ -380,6 +384,8 @@ bpobj_enqueue_subobj(bpobj_t *bpo, uint64_t subobj, dm bpobj_t subbpo; uint64_t used, comp, uncomp, subsubobjs; + ASSERT(bpobj_is_open(bpo)); + ASSERT(subobj != 0); ASSERT(bpo->bpo_havesubobj); ASSERT(bpo->bpo_havecomp); ASSERT(bpo->bpo_object != dmu_objset_pool(bpo->bpo_os)->dp_empty_bpobj); @@ -392,7 +398,7 @@ bpobj_enqueue_subobj(bpobj_t *bpo, uint64_t subobj, dm VERIFY3U(0, ==, bpobj_open(&subbpo, bpo->bpo_os, subobj)); VERIFY3U(0, ==, bpobj_space(&subbpo, &used, &comp, &uncomp)); - if (!bpobj_hasentries(&subbpo)) { + if (bpobj_is_empty(&subbpo)) { /* No point in having an empty subobj. */ bpobj_close(&subbpo); bpobj_free(bpo->bpo_os, subobj, tx); @@ -466,6 +472,7 @@ bpobj_enqueue(bpobj_t *bpo, const blkptr_t *bp, dmu_tx int blkoff; blkptr_t *bparray; + ASSERT(bpobj_is_open(bpo)); ASSERT(!BP_IS_HOLE(bp)); ASSERT(bpo->bpo_object != dmu_objset_pool(bpo->bpo_os)->dp_empty_bpobj); @@ -551,6 +558,7 @@ space_range_cb(void *arg, const blkptr_t *bp, dmu_tx_t int bpobj_space(bpobj_t *bpo, uint64_t *usedp, uint64_t *compp, uint64_t *uncompp) { + ASSERT(bpobj_is_open(bpo)); mutex_enter(&bpo->bpo_lock); *usedp = bpo->bpo_phys->bpo_bytes; @@ -576,6 +584,8 @@ bpobj_space_range(bpobj_t *bpo, uint64_t mintxg, uint6 { struct space_range_arg sra = { 0 }; int err; + + ASSERT(bpobj_is_open(bpo)); /* * As an optimization, if they want the whole txg range, just Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Sun Feb 18 01:21:52 2018 (r329502) @@ -47,6 +47,7 @@ #include #include #include +#include uint_t zfs_dbuf_evict_key; @@ -3007,6 +3008,7 @@ dbuf_sync_indirect(dbuf_dirty_record_t *dr, dmu_tx_t * db->db_data_pending = dr; mutex_exit(&db->db_mtx); + dbuf_write(dr, db->db_buf, tx); zio = dr->dr_zio; @@ -3478,6 +3480,141 @@ dbuf_write_override_done(zio_t *zio) abd_put(zio->io_abd); } +typedef struct dbuf_remap_impl_callback_arg { + objset_t *drica_os; + uint64_t drica_blk_birth; + dmu_tx_t *drica_tx; +} dbuf_remap_impl_callback_arg_t; + +static void +dbuf_remap_impl_callback(uint64_t vdev, uint64_t offset, uint64_t size, + void *arg) +{ + dbuf_remap_impl_callback_arg_t *drica = arg; + objset_t *os = drica->drica_os; + spa_t *spa = dmu_objset_spa(os); + dmu_tx_t *tx = drica->drica_tx; + + ASSERT(dsl_pool_sync_context(spa_get_dsl(spa))); + + if (os == spa_meta_objset(spa)) { + spa_vdev_indirect_mark_obsolete(spa, vdev, offset, size, tx); + } else { + dsl_dataset_block_remapped(dmu_objset_ds(os), vdev, offset, + size, drica->drica_blk_birth, tx); + } +} + +static void +dbuf_remap_impl(dnode_t *dn, blkptr_t *bp, dmu_tx_t *tx) +{ + blkptr_t bp_copy = *bp; + spa_t *spa = dmu_objset_spa(dn->dn_objset); + dbuf_remap_impl_callback_arg_t drica; + + ASSERT(dsl_pool_sync_context(spa_get_dsl(spa))); + + drica.drica_os = dn->dn_objset; + drica.drica_blk_birth = bp->blk_birth; + drica.drica_tx = tx; + if (spa_remap_blkptr(spa, &bp_copy, dbuf_remap_impl_callback, + &drica)) { + /* + * The struct_rwlock prevents dbuf_read_impl() from + * dereferencing the BP while we are changing it. To + * avoid lock contention, only grab it when we are actually + * changing the BP. + */ + rw_enter(&dn->dn_struct_rwlock, RW_WRITER); + *bp = bp_copy; + rw_exit(&dn->dn_struct_rwlock); + } +} + +/* + * Returns true if a dbuf_remap would modify the dbuf. We do this by attempting + * to remap a copy of every bp in the dbuf. + */ +boolean_t +dbuf_can_remap(const dmu_buf_impl_t *db) +{ + spa_t *spa = dmu_objset_spa(db->db_objset); + blkptr_t *bp = db->db.db_data; + boolean_t ret = B_FALSE; + + ASSERT3U(db->db_level, >, 0); + ASSERT3S(db->db_state, ==, DB_CACHED); + + ASSERT(spa_feature_is_active(spa, SPA_FEATURE_DEVICE_REMOVAL)); + + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + for (int i = 0; i < db->db.db_size >> SPA_BLKPTRSHIFT; i++) { + blkptr_t bp_copy = bp[i]; + if (spa_remap_blkptr(spa, &bp_copy, NULL, NULL)) { + ret = B_TRUE; + break; + } + } + spa_config_exit(spa, SCL_VDEV, FTAG); + + return (ret); +} + +boolean_t +dnode_needs_remap(const dnode_t *dn) +{ + spa_t *spa = dmu_objset_spa(dn->dn_objset); + boolean_t ret = B_FALSE; + + if (dn->dn_phys->dn_nlevels == 0) { + return (B_FALSE); + } + + ASSERT(spa_feature_is_active(spa, SPA_FEATURE_DEVICE_REMOVAL)); + + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + for (int j = 0; j < dn->dn_phys->dn_nblkptr; j++) { + blkptr_t bp_copy = dn->dn_phys->dn_blkptr[j]; + if (spa_remap_blkptr(spa, &bp_copy, NULL, NULL)) { + ret = B_TRUE; + break; + } + } + spa_config_exit(spa, SCL_VDEV, FTAG); + + return (ret); +} + +/* + * Remap any existing BP's to concrete vdevs, if possible. + */ +static void +dbuf_remap(dnode_t *dn, dmu_buf_impl_t *db, dmu_tx_t *tx) +{ + spa_t *spa = dmu_objset_spa(db->db_objset); + ASSERT(dsl_pool_sync_context(spa_get_dsl(spa))); + + if (!spa_feature_is_active(spa, SPA_FEATURE_DEVICE_REMOVAL)) + return; + + if (db->db_level > 0) { + blkptr_t *bp = db->db.db_data; + for (int i = 0; i < db->db.db_size >> SPA_BLKPTRSHIFT; i++) { + dbuf_remap_impl(dn, &bp[i], tx); + } + } else if (db->db.db_object == DMU_META_DNODE_OBJECT) { + dnode_phys_t *dnp = db->db.db_data; + ASSERT3U(db->db_dnode_handle->dnh_dnode->dn_type, ==, + DMU_OT_DNODE); + for (int i = 0; i < db->db.db_size >> DNODE_SHIFT; i++) { + for (int j = 0; j < dnp[i].dn_nblkptr; j++) { + dbuf_remap_impl(dn, &dnp[i].dn_blkptr[j], tx); + } + } + } +} + + /* Issue I/O to commit a dirty buffer to disk. */ static void dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, dmu_tx_t *tx) @@ -3511,6 +3648,7 @@ dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, d } else { dbuf_release_bp(db); } + dbuf_remap(dn, db, tx); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Sun Feb 18 01:21:52 2018 (r329502) @@ -704,15 +704,15 @@ ddt_lookup(ddt_t *ddt, const blkptr_t *bp, boolean_t a for (type = 0; type < DDT_TYPES; type++) { for (class = 0; class < DDT_CLASSES; class++) { error = ddt_object_lookup(ddt, type, class, dde); - if (error != ENOENT) + if (error != ENOENT) { + ASSERT0(error); break; + } } if (error != ENOENT) break; } - ASSERT(error == 0 || error == ENOENT); - ddt_enter(ddt); ASSERT(dde->dde_loaded == B_FALSE); @@ -1099,7 +1099,7 @@ ddt_sync(spa_t *spa, uint64_t txg) { dmu_tx_t *tx; zio_t *rio = zio_root(spa, NULL, NULL, - ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE); + ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE | ZIO_FLAG_SELF_HEAL); ASSERT(spa_syncing_txg(spa) == txg); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Sun Feb 18 01:21:52 2018 (r329502) @@ -65,6 +65,13 @@ int zfs_nopwrite_enabled = 1; */ uint32_t zfs_per_txg_dirty_frees_percent = 30; +/* + * This can be used for testing, to ensure that certain actions happen + * while in the middle of a remap (which might otherwise complete too + * quickly). + */ +int zfs_object_remap_one_indirect_delay_ticks = 0; + const dmu_object_type_info_t dmu_ot[DMU_OT_NUMTYPES] = { { DMU_BSWAP_UINT8, TRUE, "unallocated" }, { DMU_BSWAP_ZAP, TRUE, "object directory" }, @@ -1012,6 +1019,123 @@ dmu_write_by_dnode(dnode_t *dn, uint64_t offset, uint6 FALSE, FTAG, &numbufs, &dbp, DMU_READ_PREFETCH)); dmu_write_impl(dbp, numbufs, offset, size, buf, tx); dmu_buf_rele_array(dbp, numbufs, FTAG); +} + +static int +dmu_object_remap_one_indirect(objset_t *os, dnode_t *dn, + uint64_t last_removal_txg, uint64_t offset) +{ + uint64_t l1blkid = dbuf_whichblock(dn, 1, offset); + int err = 0; + + rw_enter(&dn->dn_struct_rwlock, RW_READER); + dmu_buf_impl_t *dbuf = dbuf_hold_level(dn, 1, l1blkid, FTAG); + ASSERT3P(dbuf, !=, NULL); + + /* + * If the block hasn't been written yet, this default will ensure + * we don't try to remap it. + */ + uint64_t birth = UINT64_MAX; + ASSERT3U(last_removal_txg, !=, UINT64_MAX); + if (dbuf->db_blkptr != NULL) + birth = dbuf->db_blkptr->blk_birth; + rw_exit(&dn->dn_struct_rwlock); + + /* + * If this L1 was already written after the last removal, then we've + * already tried to remap it. + */ + if (birth <= last_removal_txg && + dbuf_read(dbuf, NULL, DB_RF_MUST_SUCCEED) == 0 && + dbuf_can_remap(dbuf)) { + dmu_tx_t *tx = dmu_tx_create(os); + dmu_tx_hold_remap_l1indirect(tx, dn->dn_object); + err = dmu_tx_assign(tx, TXG_WAIT); + if (err == 0) { + (void) dbuf_dirty(dbuf, tx); + dmu_tx_commit(tx); + } else { + dmu_tx_abort(tx); + } + } + + dbuf_rele(dbuf, FTAG); + + delay(zfs_object_remap_one_indirect_delay_ticks); + + return (err); +} + +/* + * Remap all blockpointers in the object, if possible, so that they reference + * only concrete vdevs. + * + * To do this, iterate over the L0 blockpointers and remap any that reference + * an indirect vdev. Note that we only examine L0 blockpointers; since we + * cannot guarantee that we can remap all blockpointer anyways (due to split + * blocks), we do not want to make the code unnecessarily complicated to + * catch the unlikely case that there is an L1 block on an indirect vdev that + * contains no indirect blockpointers. + */ +int +dmu_object_remap_indirects(objset_t *os, uint64_t object, + uint64_t last_removal_txg) +{ + uint64_t offset, l1span; + int err; + dnode_t *dn; + + err = dnode_hold(os, object, FTAG, &dn); + if (err != 0) { + return (err); + } + + if (dn->dn_nlevels <= 1) { + if (issig(JUSTLOOKING) && issig(FORREAL)) { + err = SET_ERROR(EINTR); + } + + /* + * If the dnode has no indirect blocks, we cannot dirty them. + * We still want to remap the blkptr(s) in the dnode if + * appropriate, so mark it as dirty. + */ + if (err == 0 && dnode_needs_remap(dn)) { + dmu_tx_t *tx = dmu_tx_create(os); + dmu_tx_hold_bonus(tx, dn->dn_object); + if ((err = dmu_tx_assign(tx, TXG_WAIT)) == 0) { + dnode_setdirty(dn, tx); + dmu_tx_commit(tx); + } else { + dmu_tx_abort(tx); + } + } + + dnode_rele(dn, FTAG); + return (err); + } + + offset = 0; + l1span = 1ULL << (dn->dn_indblkshift - SPA_BLKPTRSHIFT + + dn->dn_datablkshift); + /* + * Find the next L1 indirect that is not a hole. + */ + while (dnode_next_offset(dn, 0, &offset, 2, 1, 0) == 0) { + if (issig(JUSTLOOKING) && issig(FORREAL)) { + err = SET_ERROR(EINTR); + break; + } + if ((err = dmu_object_remap_one_indirect(os, dn, + last_removal_txg, offset)) != 0) { + break; + } + offset += l1span; + } + + dnode_rele(dn, FTAG); + return (err); } void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Sun Feb 18 01:21:52 2018 (r329502) @@ -53,6 +53,7 @@ #include #include #include +#include /* * Needed to close a window in dnode_move() that allows the objset to be freed @@ -348,6 +349,17 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl ASSERT(ds == NULL || MUTEX_HELD(&ds->ds_opening_lock)); + /* + * The $ORIGIN dataset (if it exists) doesn't have an associated + * objset, so there's no reason to open it. The $ORIGIN dataset + * will not exist on pools older than SPA_VERSION_ORIGIN. + */ + if (ds != NULL && spa_get_dsl(spa) != NULL && + spa_get_dsl(spa)->dp_origin_snap != NULL) { + ASSERT3P(ds->ds_dir, !=, + spa_get_dsl(spa)->dp_origin_snap->ds_dir); + } + os = kmem_zalloc(sizeof (objset_t), KM_SLEEP); os->os_dsl_dataset = ds; os->os_spa = spa; @@ -1050,6 +1062,101 @@ dmu_objset_clone(const char *clone, const char *origin return (dsl_sync_task(clone, dmu_objset_clone_check, dmu_objset_clone_sync, &doca, 5, ZFS_SPACE_CHECK_NORMAL)); +} + +static int +dmu_objset_remap_indirects_impl(objset_t *os, uint64_t last_removed_txg) +{ + int error = 0; + uint64_t object = 0; + while ((error = dmu_object_next(os, &object, B_FALSE, 0)) == 0) { + error = dmu_object_remap_indirects(os, object, + last_removed_txg); + /* + * If the ZPL removed the object before we managed to dnode_hold + * it, we would get an ENOENT. If the ZPL declares its intent + * to remove the object (dnode_free) before we manage to + * dnode_hold it, we would get an EEXIST. In either case, we + * want to continue remapping the other objects in the objset; + * in all other cases, we want to break early. + */ + if (error != 0 && error != ENOENT && error != EEXIST) { + break; + } + } + if (error == ESRCH) { + error = 0; + } + return (error); +} + +int +dmu_objset_remap_indirects(const char *fsname) +{ + int error = 0; + objset_t *os = NULL; + uint64_t last_removed_txg; + uint64_t remap_start_txg; + dsl_dir_t *dd; + + error = dmu_objset_hold(fsname, FTAG, &os); + if (error != 0) { + return (error); + } + dd = dmu_objset_ds(os)->ds_dir; + + if (!spa_feature_is_enabled(dmu_objset_spa(os), + SPA_FEATURE_OBSOLETE_COUNTS)) { + dmu_objset_rele(os, FTAG); + return (SET_ERROR(ENOTSUP)); + } + + if (dsl_dataset_is_snapshot(dmu_objset_ds(os))) { + dmu_objset_rele(os, FTAG); + return (SET_ERROR(EINVAL)); + } + + /* + * If there has not been a removal, we're done. + */ + last_removed_txg = spa_get_last_removal_txg(dmu_objset_spa(os)); + if (last_removed_txg == -1ULL) { + dmu_objset_rele(os, FTAG); + return (0); + } + + /* + * If we have remapped since the last removal, we're done. + */ + if (dsl_dir_is_zapified(dd)) { + uint64_t last_remap_txg; + if (zap_lookup(spa_meta_objset(dmu_objset_spa(os)), + dd->dd_object, DD_FIELD_LAST_REMAP_TXG, + sizeof (last_remap_txg), 1, &last_remap_txg) == 0 && + last_remap_txg > last_removed_txg) { + dmu_objset_rele(os, FTAG); + return (0); + } + } + + dsl_dataset_long_hold(dmu_objset_ds(os), FTAG); + dsl_pool_rele(dmu_objset_pool(os), FTAG); + + remap_start_txg = spa_last_synced_txg(dmu_objset_spa(os)); + error = dmu_objset_remap_indirects_impl(os, last_removed_txg); + if (error == 0) { + /* + * We update the last_remap_txg to be the start txg so that + * we can guarantee that every block older than last_remap_txg + * that can be remapped has been remapped. + */ + error = dsl_dir_update_last_remap_txg(dd, remap_start_txg); + } + + dsl_dataset_long_rele(dmu_objset_ds(os), FTAG); + dsl_dataset_rele(dmu_objset_ds(os), FTAG); + + return (error); } int Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Sun Feb 18 01:21:52 2018 (r329502) @@ -301,6 +301,23 @@ dmu_tx_hold_write(dmu_tx_t *tx, uint64_t object, uint6 } void +dmu_tx_hold_remap_l1indirect(dmu_tx_t *tx, uint64_t object) +{ + dmu_tx_hold_t *txh; + + ASSERT(tx->tx_txg == 0); + txh = dmu_tx_hold_object_impl(tx, tx->tx_objset, + object, THT_WRITE, 0, 0); + if (txh == NULL) + return; + + dnode_t *dn = txh->txh_dnode; + (void) refcount_add_many(&txh->txh_space_towrite, + 1ULL << dn->dn_indblkshift, FTAG); + dmu_tx_count_dnode(txh); +} + +void dmu_tx_hold_write_by_dnode(dmu_tx_t *tx, dnode_t *dn, uint64_t off, int len) { dmu_tx_hold_t *txh; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_zfetch.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_zfetch.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_zfetch.c Sun Feb 18 01:21:52 2018 (r329502) @@ -211,8 +211,19 @@ dmu_zfetch(zfetch_t *zf, uint64_t blkid, uint64_t nblk int64_t pf_ahead_blks, max_blks; int epbs, max_dist_blks, pf_nblks, ipf_nblks; uint64_t end_of_access_blkid = blkid + nblks; + spa_t *spa = zf->zf_dnode->dn_objset->os_spa; if (zfs_prefetch_disable) + return; + + /* + * If we haven't yet loaded the indirect vdevs' mappings, we + * can only read from blocks that we carefully ensure are on + * concrete vdevs (or previously-loaded indirect vdevs). So we + * can't allow the predictive prefetcher to attempt reads of other + * blocks (e.g. of the MOS's dnode obejct). + */ + if (!spa_indirect_vdevs_loaded(spa)) return; /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Sun Feb 18 01:21:52 2018 (r329502) @@ -1690,8 +1690,7 @@ done: mutex_enter(&dn->dn_mtx); int txgoff = tx->tx_txg & TXG_MASK; if (dn->dn_free_ranges[txgoff] == NULL) { - dn->dn_free_ranges[txgoff] = - range_tree_create(NULL, NULL, &dn->dn_mtx); + dn->dn_free_ranges[txgoff] = range_tree_create(NULL, NULL); } range_tree_clear(dn->dn_free_ranges[txgoff], blkid, nblks); range_tree_add(dn->dn_free_ranges[txgoff], blkid, nblks); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Sun Feb 18 01:21:52 2018 (r329502) @@ -46,6 +46,7 @@ #include #include #include +#include #include #include #include @@ -81,6 +82,11 @@ int zfs_max_recordsize = 1 * 1024 * 1024; extern inline dsl_dataset_phys_t *dsl_dataset_phys(dsl_dataset_t *ds); +static void dsl_dataset_set_remap_deadlist_object(dsl_dataset_t *ds, + uint64_t obj, dmu_tx_t *tx); +static void dsl_dataset_unset_remap_deadlist_object(dsl_dataset_t *ds, + dmu_tx_t *tx); + extern int spa_asize_inflation; static zil_header_t zero_zil; @@ -154,6 +160,47 @@ dsl_dataset_block_born(dsl_dataset_t *ds, const blkptr DD_USED_REFRSRV, DD_USED_HEAD, tx); } +/* + * Called when the specified segment has been remapped, and is thus no + * longer referenced in the head dataset. The vdev must be indirect. + * + * If the segment is referenced by a snapshot, put it on the remap deadlist. + * Otherwise, add this segment to the obsolete spacemap. + */ +void +dsl_dataset_block_remapped(dsl_dataset_t *ds, uint64_t vdev, uint64_t offset, + uint64_t size, uint64_t birth, dmu_tx_t *tx) +{ + spa_t *spa = ds->ds_dir->dd_pool->dp_spa; + + ASSERT(dmu_tx_is_syncing(tx)); + ASSERT(birth <= tx->tx_txg); + ASSERT(!ds->ds_is_snapshot); + + if (birth > dsl_dataset_phys(ds)->ds_prev_snap_txg) { + spa_vdev_indirect_mark_obsolete(spa, vdev, offset, size, tx); + } else { + blkptr_t fakebp; + dva_t *dva = &fakebp.blk_dva[0]; + + ASSERT(ds != NULL); + + mutex_enter(&ds->ds_remap_deadlist_lock); + if (!dsl_dataset_remap_deadlist_exists(ds)) { + dsl_dataset_create_remap_deadlist(ds, tx); + } + mutex_exit(&ds->ds_remap_deadlist_lock); + + BP_ZERO(&fakebp); + fakebp.blk_birth = birth; + DVA_SET_VDEV(dva, vdev); + DVA_SET_OFFSET(dva, offset); + DVA_SET_ASIZE(dva, size); + + dsl_deadlist_insert(&ds->ds_remap_deadlist, &fakebp, tx); + } +} + int dsl_dataset_block_kill(dsl_dataset_t *ds, const blkptr_t *bp, dmu_tx_t *tx, boolean_t async) @@ -272,8 +319,10 @@ dsl_dataset_evict_async(void *dbu) } bplist_destroy(&ds->ds_pending_deadlist); - if (ds->ds_deadlist.dl_os != NULL) + if (dsl_deadlist_is_open(&ds->ds_deadlist)) dsl_deadlist_close(&ds->ds_deadlist); + if (dsl_deadlist_is_open(&ds->ds_remap_deadlist)) + dsl_deadlist_close(&ds->ds_remap_deadlist); if (ds->ds_dir) dsl_dir_async_rele(ds->ds_dir, ds); @@ -283,6 +332,7 @@ dsl_dataset_evict_async(void *dbu) mutex_destroy(&ds->ds_lock); mutex_destroy(&ds->ds_opening_lock); mutex_destroy(&ds->ds_sendstream_lock); + mutex_destroy(&ds->ds_remap_deadlist_lock); refcount_destroy(&ds->ds_longholds); rrw_destroy(&ds->ds_bp_rwlock); @@ -407,15 +457,23 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, v ds->ds_object = dsobj; ds->ds_is_snapshot = dsl_dataset_phys(ds)->ds_num_children != 0; + err = dsl_dir_hold_obj(dp, dsl_dataset_phys(ds)->ds_dir_obj, + NULL, ds, &ds->ds_dir); + if (err != 0) { + kmem_free(ds, sizeof (dsl_dataset_t)); + dmu_buf_rele(dbuf, tag); + return (err); + } + mutex_init(&ds->ds_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ds->ds_opening_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ds->ds_sendstream_lock, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&ds->ds_remap_deadlist_lock, + NULL, MUTEX_DEFAULT, NULL); rrw_init(&ds->ds_bp_rwlock, B_FALSE); refcount_create(&ds->ds_longholds); bplist_create(&ds->ds_pending_deadlist); - dsl_deadlist_open(&ds->ds_deadlist, - mos, dsl_dataset_phys(ds)->ds_deadlist_obj); list_create(&ds->ds_sendstreams, sizeof (dmu_sendarg_t), offsetof(dmu_sendarg_t, dsa_link)); @@ -439,20 +497,6 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, v } } - err = dsl_dir_hold_obj(dp, - dsl_dataset_phys(ds)->ds_dir_obj, NULL, ds, &ds->ds_dir); - if (err != 0) { - mutex_destroy(&ds->ds_lock); - mutex_destroy(&ds->ds_opening_lock); - mutex_destroy(&ds->ds_sendstream_lock); - refcount_destroy(&ds->ds_longholds); - bplist_destroy(&ds->ds_pending_deadlist); - dsl_deadlist_close(&ds->ds_deadlist); - kmem_free(ds, sizeof (dsl_dataset_t)); - dmu_buf_rele(dbuf, tag); - return (err); - } - if (!ds->ds_is_snapshot) { ds->ds_snapname[0] = '\0'; if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) { @@ -493,6 +537,15 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, v ds->ds_reserved = ds->ds_quota = 0; } + dsl_deadlist_open(&ds->ds_deadlist, + mos, dsl_dataset_phys(ds)->ds_deadlist_obj); + uint64_t remap_deadlist_obj = + dsl_dataset_get_remap_deadlist_object(ds); + if (remap_deadlist_obj != 0) { + dsl_deadlist_open(&ds->ds_remap_deadlist, mos, + remap_deadlist_obj); + } + dmu_buf_init_user(&ds->ds_dbu, dsl_dataset_evict_sync, dsl_dataset_evict_async, &ds->ds_dbuf); if (err == 0) @@ -501,6 +554,8 @@ dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, v if (err != 0 || winner != NULL) { bplist_destroy(&ds->ds_pending_deadlist); dsl_deadlist_close(&ds->ds_deadlist); + if (dsl_deadlist_is_open(&ds->ds_remap_deadlist)) + dsl_deadlist_close(&ds->ds_remap_deadlist); if (ds->ds_prev) dsl_dataset_rele(ds->ds_prev, ds); dsl_dir_rele(ds->ds_dir, ds); @@ -1393,10 +1448,27 @@ dsl_dataset_snapshot_sync_impl(dsl_dataset_t *ds, cons dsl_deadlist_add_key(&ds->ds_deadlist, dsl_dataset_phys(ds)->ds_prev_snap_txg, tx); + if (dsl_dataset_remap_deadlist_exists(ds)) { + uint64_t remap_deadlist_obj = + dsl_dataset_get_remap_deadlist_object(ds); + /* + * Move the remap_deadlist to the snapshot. The head + * will create a new remap deadlist on demand, from + * dsl_dataset_block_remapped(). + */ + dsl_dataset_unset_remap_deadlist_object(ds, tx); + dsl_deadlist_close(&ds->ds_remap_deadlist); + + dmu_object_zapify(mos, dsobj, DMU_OT_DSL_DATASET, tx); + VERIFY0(zap_add(mos, dsobj, DS_FIELD_REMAP_DEADLIST, + sizeof (remap_deadlist_obj), 1, &remap_deadlist_obj, tx)); + } + ASSERT3U(dsl_dataset_phys(ds)->ds_prev_snap_txg, <, tx->tx_txg); dsl_dataset_phys(ds)->ds_prev_snap_obj = dsobj; dsl_dataset_phys(ds)->ds_prev_snap_txg = crtxg; dsl_dataset_phys(ds)->ds_unique_bytes = 0; + if (spa_version(dp->dp_spa) >= SPA_VERSION_UNIQUE_ACCURATE) dsl_dataset_phys(ds)->ds_flags |= DS_FLAG_UNIQUE_ACCURATE; @@ -3233,6 +3305,41 @@ dsl_dataset_clone_swap_check_impl(dsl_dataset_t *clone return (0); } +static void +dsl_dataset_swap_remap_deadlists(dsl_dataset_t *clone, + dsl_dataset_t *origin, dmu_tx_t *tx) +{ + uint64_t clone_remap_dl_obj, origin_remap_dl_obj; + dsl_pool_t *dp = dmu_tx_pool(tx); + + ASSERT(dsl_pool_sync_context(dp)); + + clone_remap_dl_obj = dsl_dataset_get_remap_deadlist_object(clone); + origin_remap_dl_obj = dsl_dataset_get_remap_deadlist_object(origin); + + if (clone_remap_dl_obj != 0) { + dsl_deadlist_close(&clone->ds_remap_deadlist); + dsl_dataset_unset_remap_deadlist_object(clone, tx); + } + if (origin_remap_dl_obj != 0) { + dsl_deadlist_close(&origin->ds_remap_deadlist); + dsl_dataset_unset_remap_deadlist_object(origin, tx); + } + + if (clone_remap_dl_obj != 0) { + dsl_dataset_set_remap_deadlist_object(origin, + clone_remap_dl_obj, tx); + dsl_deadlist_open(&origin->ds_remap_deadlist, + dp->dp_meta_objset, clone_remap_dl_obj); + } + if (origin_remap_dl_obj != 0) { + dsl_dataset_set_remap_deadlist_object(clone, + origin_remap_dl_obj, tx); + dsl_deadlist_open(&clone->ds_remap_deadlist, + dp->dp_meta_objset, origin_remap_dl_obj); + } +} + void dsl_dataset_clone_swap_sync_impl(dsl_dataset_t *clone, dsl_dataset_t *origin_head, dmu_tx_t *tx) @@ -3402,6 +3509,7 @@ dsl_dataset_clone_swap_sync_impl(dsl_dataset_t *clone, dsl_dataset_phys(clone)->ds_deadlist_obj); dsl_deadlist_open(&origin_head->ds_deadlist, dp->dp_meta_objset, dsl_dataset_phys(origin_head)->ds_deadlist_obj); + dsl_dataset_swap_remap_deadlists(clone, origin_head, tx); dsl_scan_ds_clone_swapped(origin_head, clone, tx); @@ -3908,4 +4016,91 @@ dsl_dataset_has_resume_receive_state(dsl_dataset_t *ds return (dsl_dataset_is_zapified(ds) && zap_contains(ds->ds_dir->dd_pool->dp_meta_objset, ds->ds_object, DS_FIELD_RESUME_TOGUID) == 0); +} + +uint64_t +dsl_dataset_get_remap_deadlist_object(dsl_dataset_t *ds) +{ + uint64_t remap_deadlist_obj; + int err; + + if (!dsl_dataset_is_zapified(ds)) + return (0); + + err = zap_lookup(ds->ds_dir->dd_pool->dp_meta_objset, ds->ds_object, + DS_FIELD_REMAP_DEADLIST, sizeof (remap_deadlist_obj), 1, + &remap_deadlist_obj); + + if (err != 0) { + VERIFY3S(err, ==, ENOENT); + return (0); + } + + ASSERT(remap_deadlist_obj != 0); + return (remap_deadlist_obj); +} + +boolean_t +dsl_dataset_remap_deadlist_exists(dsl_dataset_t *ds) +{ + EQUIV(dsl_deadlist_is_open(&ds->ds_remap_deadlist), + dsl_dataset_get_remap_deadlist_object(ds) != 0); + return (dsl_deadlist_is_open(&ds->ds_remap_deadlist)); +} + +static void +dsl_dataset_set_remap_deadlist_object(dsl_dataset_t *ds, uint64_t obj, + dmu_tx_t *tx) +{ + ASSERT(obj != 0); + dsl_dataset_zapify(ds, tx); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sun Feb 18 01:21:55 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E43D9F25625; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96E7679290; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 919B324A16; Sun, 18 Feb 2018 01:21:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1I1Ls13091362; Sun, 18 Feb 2018 01:21:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1I1LsnP091359; Sun, 18 Feb 2018 01:21:54 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802180121.w1I1LsnP091359@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Sun, 18 Feb 2018 01:21:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329502 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb vendor/... X-SVN-Commit-Revision: 329502 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Feb 2018 01:21:55 -0000 Author: mav Date: Sun Feb 18 01:21:52 2018 New Revision: 329502 URL: https://svnweb.freebsd.org/changeset/base/329502 Log: 7614 zfs device evacuation/removal illumos/illumos-gate@5cabbc6b49070407fb9610cfe73d4c0e0dea3e77 https://www.illumos.org/issues/7614: This project allows top-level vdevs to be removed from the storage pool with “zpool remove”, reducing the total amount of storage in the pool. This operation copies all allocated regions of the device to be removed onto other devices, recording the mapping from old to new location. After the removal is complete, read and free operations to the removed (now “indirect”) vdev must be remapped and performed at the new location on disk. The indirect mapping table is kept in memory whenever the pool is loaded, so there is minimal performance overhead when doing operations on the indirect vdev. The size of the in-memory mapping table will be reduced when its entries become “obsolete” because they are no longer used by any block pointers in the pool. An entry becomes obsolete when all the blocks that use it are freed. An entry can also become obsolete when all the snapshots that reference it are deleted, and the block pointers that reference it have been “remapped” in all filesystems/zvols (and clones). Whenever an indirect block is written, all the block pointers in it will be “remapped” to their new (concrete) locations if possible. This process can be accelerated by using the “zfs remap” command to proactively rewrite all indirect blocks that reference indirect (removed) vdevs. Note that when a device is removed, we do not verify the checksum of the data that is copied. This makes the process much faster, but if it were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be possible to copy the wrong data, when we have the correct data on e.g. the other side of the mirror. Therefore, mirror and raidz devices can not be removed. Reviewed by: Alex Reece Reviewed by: George Wilson Reviewed by: John Kennedy Reviewed by: Prakash Surya Reviewed by: Matthew Ahrens Reviewed by: Richard Laager Reviewed by: Tim Chase Approved by: Garrett D'Amore Author: Prashanth Sreenivasa Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/man/man1m/zfs.1m vendor/illumos/dist/man/man1m/zpool.1m vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_indirect_births.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_indirect_mapping.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_births.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/common/zfs/zfs_deleg.c vendor-sys/illumos/dist/common/zfs/zfs_deleg.h vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/bpobj.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_zfetch.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/bpobj.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_deadlist.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_deleg.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dir.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_scan.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor/illumos/dist/cmd/zdb/zdb.c Sun Feb 18 01:21:52 2018 (r329502) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2016 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 Nexenta Systems, Inc. */ @@ -77,8 +77,10 @@ dmu_ot[(idx)].ot_name : DMU_OT_IS_VALID(idx) ? \ dmu_ot_byteswap[DMU_OT_BYTESWAP(idx)].ob_name : "UNKNOWN") #define ZDB_OT_TYPE(idx) ((idx) < DMU_OT_NUMTYPES ? (idx) : \ - (((idx) == DMU_OTN_ZAP_DATA || (idx) == DMU_OTN_ZAP_METADATA) ? \ - DMU_OT_ZAP_OTHER : DMU_OT_NUMTYPES)) + (idx) == DMU_OTN_ZAP_DATA || (idx) == DMU_OTN_ZAP_METADATA ? \ + DMU_OT_ZAP_OTHER : \ + (idx) == DMU_OTN_UINT64_DATA || (idx) == DMU_OTN_UINT64_METADATA ? \ + DMU_OT_UINT64_OTHER : DMU_OT_NUMTYPES) #ifndef lint extern int reference_tracking_enable; @@ -667,8 +669,8 @@ get_metaslab_refcount(vdev_t *vd) { int refcount = 0; - if (vd->vdev_top == vd && !vd->vdev_removing) { - for (unsigned m = 0; m < vd->vdev_ms_count; m++) { + if (vd->vdev_top == vd) { + for (uint64_t m = 0; m < vd->vdev_ms_count; m++) { space_map_t *sm = vd->vdev_ms[m]->ms_sm; if (sm != NULL && @@ -683,6 +685,45 @@ get_metaslab_refcount(vdev_t *vd) } static int +get_obsolete_refcount(vdev_t *vd) +{ + int refcount = 0; + + uint64_t obsolete_sm_obj = vdev_obsolete_sm_object(vd); + if (vd->vdev_top == vd && obsolete_sm_obj != 0) { + dmu_object_info_t doi; + VERIFY0(dmu_object_info(vd->vdev_spa->spa_meta_objset, + obsolete_sm_obj, &doi)); + if (doi.doi_bonus_size == sizeof (space_map_phys_t)) { + refcount++; + } + } else { + ASSERT3P(vd->vdev_obsolete_sm, ==, NULL); + ASSERT3U(obsolete_sm_obj, ==, 0); + } + for (unsigned c = 0; c < vd->vdev_children; c++) { + refcount += get_obsolete_refcount(vd->vdev_child[c]); + } + + return (refcount); +} + +static int +get_prev_obsolete_spacemap_refcount(spa_t *spa) +{ + uint64_t prev_obj = + spa->spa_condensing_indirect_phys.scip_prev_obsolete_sm_object; + if (prev_obj != 0) { + dmu_object_info_t doi; + VERIFY0(dmu_object_info(spa->spa_meta_objset, prev_obj, &doi)); + if (doi.doi_bonus_size == sizeof (space_map_phys_t)) { + return (1); + } + } + return (0); +} + +static int verify_spacemap_refcounts(spa_t *spa) { uint64_t expected_refcount = 0; @@ -693,6 +734,8 @@ verify_spacemap_refcounts(spa_t *spa) &expected_refcount); actual_refcount = get_dtl_refcount(spa->spa_root_vdev); actual_refcount += get_metaslab_refcount(spa->spa_root_vdev); + actual_refcount += get_obsolete_refcount(spa->spa_root_vdev); + actual_refcount += get_prev_obsolete_spacemap_refcount(spa); if (expected_refcount != actual_refcount) { (void) printf("space map refcount mismatch: expected %lld != " @@ -708,12 +751,19 @@ static void dump_spacemap(objset_t *os, space_map_t *sm) { uint64_t alloc, offset, entry; - const char *ddata[] = { "ALLOC", "FREE", "CONDENSE", "INVALID", - "INVALID", "INVALID", "INVALID", "INVALID" }; + char *ddata[] = { "ALLOC", "FREE", "CONDENSE", "INVALID", + "INVALID", "INVALID", "INVALID", "INVALID" }; if (sm == NULL) return; + (void) printf("space map object %llu:\n", + (longlong_t)sm->sm_phys->smp_object); + (void) printf(" smp_objsize = 0x%llx\n", + (longlong_t)sm->sm_phys->smp_objsize); + (void) printf(" smp_alloc = 0x%llx\n", + (longlong_t)sm->sm_phys->smp_alloc); + /* * Print out the freelist entries in both encoded and decoded form. */ @@ -818,9 +868,7 @@ dump_metaslab(metaslab_t *msp) if (dump_opt['d'] > 5 || dump_opt['m'] > 3) { ASSERT(msp->ms_size == (1ULL << vd->vdev_ms_shift)); - mutex_enter(&msp->ms_lock); dump_spacemap(spa->spa_meta_objset, msp->ms_sm); - mutex_exit(&msp->ms_lock); } } @@ -878,6 +926,78 @@ dump_metaslab_groups(spa_t *spa) } static void +print_vdev_indirect(vdev_t *vd) +{ + vdev_indirect_config_t *vic = &vd->vdev_indirect_config; + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + vdev_indirect_births_t *vib = vd->vdev_indirect_births; + + if (vim == NULL) { + ASSERT3P(vib, ==, NULL); + return; + } + + ASSERT3U(vdev_indirect_mapping_object(vim), ==, + vic->vic_mapping_object); + ASSERT3U(vdev_indirect_births_object(vib), ==, + vic->vic_births_object); + + (void) printf("indirect births obj %llu:\n", + (longlong_t)vic->vic_births_object); + (void) printf(" vib_count = %llu\n", + (longlong_t)vdev_indirect_births_count(vib)); + for (uint64_t i = 0; i < vdev_indirect_births_count(vib); i++) { + vdev_indirect_birth_entry_phys_t *cur_vibe = + &vib->vib_entries[i]; + (void) printf("\toffset %llx -> txg %llu\n", + (longlong_t)cur_vibe->vibe_offset, + (longlong_t)cur_vibe->vibe_phys_birth_txg); + } + (void) printf("\n"); + + (void) printf("indirect mapping obj %llu:\n", + (longlong_t)vic->vic_mapping_object); + (void) printf(" vim_max_offset = 0x%llx\n", + (longlong_t)vdev_indirect_mapping_max_offset(vim)); + (void) printf(" vim_bytes_mapped = 0x%llx\n", + (longlong_t)vdev_indirect_mapping_bytes_mapped(vim)); + (void) printf(" vim_count = %llu\n", + (longlong_t)vdev_indirect_mapping_num_entries(vim)); + + if (dump_opt['d'] <= 5 && dump_opt['m'] <= 3) + return; + + uint32_t *counts = vdev_indirect_mapping_load_obsolete_counts(vim); + + for (uint64_t i = 0; i < vdev_indirect_mapping_num_entries(vim); i++) { + vdev_indirect_mapping_entry_phys_t *vimep = + &vim->vim_entries[i]; + (void) printf("\t<%llx:%llx:%llx> -> " + "<%llx:%llx:%llx> (%x obsolete)\n", + (longlong_t)vd->vdev_id, + (longlong_t)DVA_MAPPING_GET_SRC_OFFSET(vimep), + (longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst), + (longlong_t)DVA_GET_VDEV(&vimep->vimep_dst), + (longlong_t)DVA_GET_OFFSET(&vimep->vimep_dst), + (longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst), + counts[i]); + } + (void) printf("\n"); + + uint64_t obsolete_sm_object = vdev_obsolete_sm_object(vd); + if (obsolete_sm_object != 0) { + objset_t *mos = vd->vdev_spa->spa_meta_objset; + (void) printf("obsolete space map object %llu:\n", + (u_longlong_t)obsolete_sm_object); + ASSERT(vd->vdev_obsolete_sm != NULL); + ASSERT3U(space_map_object(vd->vdev_obsolete_sm), ==, + obsolete_sm_object); + dump_spacemap(mos, vd->vdev_obsolete_sm); + (void) printf("\n"); + } +} + +static void dump_metaslabs(spa_t *spa) { vdev_t *vd, *rvd = spa->spa_root_vdev; @@ -913,6 +1033,8 @@ dump_metaslabs(spa_t *spa) vd = rvd->vdev_child[c]; print_vdev_metaslab_header(vd); + print_vdev_indirect(vd); + for (m = 0; m < vd->vdev_ms_count; m++) dump_metaslab(vd->vdev_ms[m]); (void) printf("\n"); @@ -1088,9 +1210,7 @@ dump_dtl(vdev_t *vd, int indent) continue; (void) snprintf(prefix, sizeof (prefix), "\t%*s%s", indent + 2, "", name[t]); - mutex_enter(rt->rt_lock); range_tree_walk(rt, dump_dtl_seg, prefix); - mutex_exit(rt->rt_lock); if (dump_opt['d'] > 5 && vd->vdev_children == 0) dump_spacemap(spa->spa_meta_objset, vd->vdev_dtl_sm); } @@ -2082,9 +2202,16 @@ dump_dir(objset_t *os) if (dump_opt['i'] != 0 || verbosity >= 2) dump_intent_log(dmu_objset_zil(os)); - if (dmu_objset_ds(os) != NULL) - dump_deadlist(&dmu_objset_ds(os)->ds_deadlist); + if (dmu_objset_ds(os) != NULL) { + dsl_dataset_t *ds = dmu_objset_ds(os); + dump_deadlist(&ds->ds_deadlist); + if (dsl_dataset_remap_deadlist_exists(ds)) { + (void) printf("ds_remap_deadlist:\n"); + dump_deadlist(&ds->ds_remap_deadlist); + } + } + if (verbosity < 2) return; @@ -2427,6 +2554,7 @@ dump_label(const char *dev) } static uint64_t dataset_feature_count[SPA_FEATURES]; +static uint64_t remap_deadlist_count = 0; /*ARGSUSED*/ static int @@ -2447,6 +2575,10 @@ dump_one_dir(const char *dsname, void *arg) dataset_feature_count[f]++; } + if (dsl_dataset_remap_deadlist_exists(dmu_objset_ds(os))) { + remap_deadlist_count++; + } + dump_dir(os); close_objset(os, FTAG); fuid_table_destroy(); @@ -2486,6 +2618,7 @@ static const char *zdb_ot_extname[] = { typedef struct zdb_cb { zdb_blkstats_t zcb_type[ZB_TOTAL + 1][ZDB_OT_TOTAL + 1]; + uint64_t zcb_removing_size; uint64_t zcb_dedup_asize; uint64_t zcb_dedup_blocks; uint64_t zcb_embedded_blocks[NUM_BP_EMBEDDED_TYPES]; @@ -2498,6 +2631,7 @@ typedef struct zdb_cb { int zcb_readfails; int zcb_haderrors; spa_t *zcb_spa; + uint32_t **zcb_vd_obsolete_counts; } zdb_cb_t; static void @@ -2770,12 +2904,208 @@ zdb_ddt_leak_init(spa_t *spa, zdb_cb_t *zcb) ASSERT(error == ENOENT); } +/* ARGSUSED */ static void +claim_segment_impl_cb(uint64_t inner_offset, vdev_t *vd, uint64_t offset, + uint64_t size, void *arg) +{ + /* + * This callback was called through a remap from + * a device being removed. Therefore, the vdev that + * this callback is applied to is a concrete + * vdev. + */ + ASSERT(vdev_is_concrete(vd)); + + VERIFY0(metaslab_claim_impl(vd, offset, size, + spa_first_txg(vd->vdev_spa))); +} + +static void +claim_segment_cb(void *arg, uint64_t offset, uint64_t size) +{ + vdev_t *vd = arg; + + vdev_indirect_ops.vdev_op_remap(vd, offset, size, + claim_segment_impl_cb, NULL); +} + +/* + * After accounting for all allocated blocks that are directly referenced, + * we might have missed a reference to a block from a partially complete + * (and thus unused) indirect mapping object. We perform a secondary pass + * through the metaslabs we have already mapped and claim the destination + * blocks. + */ +static void +zdb_claim_removing(spa_t *spa, zdb_cb_t *zcb) +{ + if (spa->spa_vdev_removal == NULL) + return; + + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + + spa_vdev_removal_t *svr = spa->spa_vdev_removal; + vdev_t *vd = svr->svr_vdev; + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + + for (uint64_t msi = 0; msi < vd->vdev_ms_count; msi++) { + metaslab_t *msp = vd->vdev_ms[msi]; + + if (msp->ms_start >= vdev_indirect_mapping_max_offset(vim)) + break; + + ASSERT0(range_tree_space(svr->svr_allocd_segs)); + + if (msp->ms_sm != NULL) { + VERIFY0(space_map_load(msp->ms_sm, + svr->svr_allocd_segs, SM_ALLOC)); + + /* + * Clear everything past what has been synced, + * because we have not allocated mappings for it yet. + */ + range_tree_clear(svr->svr_allocd_segs, + vdev_indirect_mapping_max_offset(vim), + msp->ms_sm->sm_start + msp->ms_sm->sm_size - + vdev_indirect_mapping_max_offset(vim)); + } + + zcb->zcb_removing_size += + range_tree_space(svr->svr_allocd_segs); + range_tree_vacate(svr->svr_allocd_segs, claim_segment_cb, vd); + } + + spa_config_exit(spa, SCL_CONFIG, FTAG); +} + +/* + * vm_idxp is an in-out parameter which (for indirect vdevs) is the + * index in vim_entries that has the first entry in this metaslab. On + * return, it will be set to the first entry after this metaslab. + */ +static void +zdb_leak_init_ms(metaslab_t *msp, uint64_t *vim_idxp) +{ + metaslab_group_t *mg = msp->ms_group; + vdev_t *vd = mg->mg_vd; + vdev_t *rvd = vd->vdev_spa->spa_root_vdev; + + mutex_enter(&msp->ms_lock); + metaslab_unload(msp); + + /* + * We don't want to spend the CPU manipulating the size-ordered + * tree, so clear the range_tree ops. + */ + msp->ms_tree->rt_ops = NULL; + + (void) fprintf(stderr, + "\rloading vdev %llu of %llu, metaslab %llu of %llu ...", + (longlong_t)vd->vdev_id, + (longlong_t)rvd->vdev_children, + (longlong_t)msp->ms_id, + (longlong_t)vd->vdev_ms_count); + + /* + * For leak detection, we overload the metaslab ms_tree to + * contain allocated segments instead of free segments. As a + * result, we can't use the normal metaslab_load/unload + * interfaces. + */ + if (vd->vdev_ops == &vdev_indirect_ops) { + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + for (; *vim_idxp < vdev_indirect_mapping_num_entries(vim); + (*vim_idxp)++) { + vdev_indirect_mapping_entry_phys_t *vimep = + &vim->vim_entries[*vim_idxp]; + uint64_t ent_offset = DVA_MAPPING_GET_SRC_OFFSET(vimep); + uint64_t ent_len = DVA_GET_ASIZE(&vimep->vimep_dst); + ASSERT3U(ent_offset, >=, msp->ms_start); + if (ent_offset >= msp->ms_start + msp->ms_size) + break; + + /* + * Mappings do not cross metaslab boundaries, + * because we create them by walking the metaslabs. + */ + ASSERT3U(ent_offset + ent_len, <=, + msp->ms_start + msp->ms_size); + range_tree_add(msp->ms_tree, ent_offset, ent_len); + } + } else if (msp->ms_sm != NULL) { + VERIFY0(space_map_load(msp->ms_sm, msp->ms_tree, SM_ALLOC)); + } + + if (!msp->ms_loaded) { + msp->ms_loaded = B_TRUE; + } + mutex_exit(&msp->ms_lock); +} + +/* ARGSUSED */ +static int +increment_indirect_mapping_cb(void *arg, const blkptr_t *bp, dmu_tx_t *tx) +{ + zdb_cb_t *zcb = arg; + spa_t *spa = zcb->zcb_spa; + vdev_t *vd; + const dva_t *dva = &bp->blk_dva[0]; + + ASSERT(!dump_opt['L']); + ASSERT3U(BP_GET_NDVAS(bp), ==, 1); + + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + vd = vdev_lookup_top(zcb->zcb_spa, DVA_GET_VDEV(dva)); + ASSERT3P(vd, !=, NULL); + spa_config_exit(spa, SCL_VDEV, FTAG); + + ASSERT(vd->vdev_indirect_config.vic_mapping_object != 0); + ASSERT3P(zcb->zcb_vd_obsolete_counts[vd->vdev_id], !=, NULL); + + vdev_indirect_mapping_increment_obsolete_count( + vd->vdev_indirect_mapping, + DVA_GET_OFFSET(dva), DVA_GET_ASIZE(dva), + zcb->zcb_vd_obsolete_counts[vd->vdev_id]); + + return (0); +} + +static uint32_t * +zdb_load_obsolete_counts(vdev_t *vd) +{ + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + spa_t *spa = vd->vdev_spa; + spa_condensing_indirect_phys_t *scip = + &spa->spa_condensing_indirect_phys; + uint32_t *counts; + + EQUIV(vdev_obsolete_sm_object(vd) != 0, vd->vdev_obsolete_sm != NULL); + counts = vdev_indirect_mapping_load_obsolete_counts(vim); + if (vd->vdev_obsolete_sm != NULL) { + vdev_indirect_mapping_load_obsolete_spacemap(vim, counts, + vd->vdev_obsolete_sm); + } + if (scip->scip_vdev == vd->vdev_id && + scip->scip_prev_obsolete_sm_object != 0) { + space_map_t *prev_obsolete_sm = NULL; + VERIFY0(space_map_open(&prev_obsolete_sm, spa->spa_meta_objset, + scip->scip_prev_obsolete_sm_object, 0, vd->vdev_asize, 0)); + space_map_update(prev_obsolete_sm); + vdev_indirect_mapping_load_obsolete_spacemap(vim, counts, + prev_obsolete_sm); + space_map_close(prev_obsolete_sm); + } + return (counts); +} + +static void zdb_leak_init(spa_t *spa, zdb_cb_t *zcb) { zcb->zcb_spa = spa; if (!dump_opt['L']) { + dsl_pool_t *dp = spa->spa_dsl_pool; vdev_t *rvd = spa->spa_root_vdev; /* @@ -2786,50 +3116,51 @@ zdb_leak_init(spa_t *spa, zdb_cb_t *zcb) spa->spa_normal_class->mc_ops = &zdb_metaslab_ops; spa->spa_log_class->mc_ops = &zdb_metaslab_ops; + zcb->zcb_vd_obsolete_counts = + umem_zalloc(rvd->vdev_children * sizeof (uint32_t *), + UMEM_NOFAIL); + + for (uint64_t c = 0; c < rvd->vdev_children; c++) { vdev_t *vd = rvd->vdev_child[c]; - metaslab_group_t *mg = vd->vdev_mg; - for (uint64_t m = 0; m < vd->vdev_ms_count; m++) { - metaslab_t *msp = vd->vdev_ms[m]; - ASSERT3P(msp->ms_group, ==, mg); - mutex_enter(&msp->ms_lock); - metaslab_unload(msp); + uint64_t vim_idx = 0; + ASSERT3U(c, ==, vd->vdev_id); + + /* + * Note: we don't check for mapping leaks on + * removing vdevs because their ms_tree's are + * used to look for leaks in allocated space. + */ + if (vd->vdev_ops == &vdev_indirect_ops) { + zcb->zcb_vd_obsolete_counts[c] = + zdb_load_obsolete_counts(vd); + /* - * For leak detection, we overload the metaslab - * ms_tree to contain allocated segments - * instead of free segments. As a result, - * we can't use the normal metaslab_load/unload - * interfaces. + * Normally, indirect vdevs don't have any + * metaslabs. We want to set them up for + * zio_claim(). */ - if (msp->ms_sm != NULL) { - (void) fprintf(stderr, - "\rloading space map for " - "vdev %llu of %llu, " - "metaslab %llu of %llu ...", - (longlong_t)c, - (longlong_t)rvd->vdev_children, - (longlong_t)m, - (longlong_t)vd->vdev_ms_count); + VERIFY0(vdev_metaslab_init(vd, 0)); + } - /* - * We don't want to spend the CPU - * manipulating the size-ordered - * tree, so clear the range_tree - * ops. - */ - msp->ms_tree->rt_ops = NULL; - VERIFY0(space_map_load(msp->ms_sm, - msp->ms_tree, SM_ALLOC)); - - if (!msp->ms_loaded) { - msp->ms_loaded = B_TRUE; - } - } - mutex_exit(&msp->ms_lock); + for (uint64_t m = 0; m < vd->vdev_ms_count; m++) { + zdb_leak_init_ms(vd->vdev_ms[m], &vim_idx); } + if (vd->vdev_ops == &vdev_indirect_ops) { + ASSERT3U(vim_idx, ==, + vdev_indirect_mapping_num_entries( + vd->vdev_indirect_mapping)); + } } (void) fprintf(stderr, "\n"); + + if (bpobj_is_open(&dp->dp_obsolete_bpobj)) { + ASSERT(spa_feature_is_enabled(spa, + SPA_FEATURE_DEVICE_REMOVAL)); + (void) bpobj_iterate_nofree(&dp->dp_obsolete_bpobj, + increment_indirect_mapping_cb, zcb, NULL); + } } spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); @@ -2839,18 +3170,93 @@ zdb_leak_init(spa_t *spa, zdb_cb_t *zcb) spa_config_exit(spa, SCL_CONFIG, FTAG); } -static void -zdb_leak_fini(spa_t *spa) +static boolean_t +zdb_check_for_obsolete_leaks(vdev_t *vd, zdb_cb_t *zcb) { + boolean_t leaks = B_FALSE; + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + uint64_t total_leaked = 0; + + ASSERT(vim != NULL); + + for (uint64_t i = 0; i < vdev_indirect_mapping_num_entries(vim); i++) { + vdev_indirect_mapping_entry_phys_t *vimep = + &vim->vim_entries[i]; + uint64_t obsolete_bytes = 0; + uint64_t offset = DVA_MAPPING_GET_SRC_OFFSET(vimep); + metaslab_t *msp = vd->vdev_ms[offset >> vd->vdev_ms_shift]; + + /* + * This is not very efficient but it's easy to + * verify correctness. + */ + for (uint64_t inner_offset = 0; + inner_offset < DVA_GET_ASIZE(&vimep->vimep_dst); + inner_offset += 1 << vd->vdev_ashift) { + if (range_tree_contains(msp->ms_tree, + offset + inner_offset, 1 << vd->vdev_ashift)) { + obsolete_bytes += 1 << vd->vdev_ashift; + } + } + + int64_t bytes_leaked = obsolete_bytes - + zcb->zcb_vd_obsolete_counts[vd->vdev_id][i]; + ASSERT3U(DVA_GET_ASIZE(&vimep->vimep_dst), >=, + zcb->zcb_vd_obsolete_counts[vd->vdev_id][i]); + if (bytes_leaked != 0 && + (vdev_obsolete_counts_are_precise(vd) || + dump_opt['d'] >= 5)) { + (void) printf("obsolete indirect mapping count " + "mismatch on %llu:%llx:%llx : %llx bytes leaked\n", + (u_longlong_t)vd->vdev_id, + (u_longlong_t)DVA_MAPPING_GET_SRC_OFFSET(vimep), + (u_longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst), + (u_longlong_t)bytes_leaked); + } + total_leaked += ABS(bytes_leaked); + } + + if (!vdev_obsolete_counts_are_precise(vd) && total_leaked > 0) { + int pct_leaked = total_leaked * 100 / + vdev_indirect_mapping_bytes_mapped(vim); + (void) printf("cannot verify obsolete indirect mapping " + "counts of vdev %llu because precise feature was not " + "enabled when it was removed: %d%% (%llx bytes) of mapping" + "unreferenced\n", + (u_longlong_t)vd->vdev_id, pct_leaked, + (u_longlong_t)total_leaked); + } else if (total_leaked > 0) { + (void) printf("obsolete indirect mapping count mismatch " + "for vdev %llu -- %llx total bytes mismatched\n", + (u_longlong_t)vd->vdev_id, + (u_longlong_t)total_leaked); + leaks |= B_TRUE; + } + + vdev_indirect_mapping_free_obsolete_counts(vim, + zcb->zcb_vd_obsolete_counts[vd->vdev_id]); + zcb->zcb_vd_obsolete_counts[vd->vdev_id] = NULL; + + return (leaks); +} + +static boolean_t +zdb_leak_fini(spa_t *spa, zdb_cb_t *zcb) +{ + boolean_t leaks = B_FALSE; if (!dump_opt['L']) { vdev_t *rvd = spa->spa_root_vdev; for (unsigned c = 0; c < rvd->vdev_children; c++) { vdev_t *vd = rvd->vdev_child[c]; metaslab_group_t *mg = vd->vdev_mg; - for (unsigned m = 0; m < vd->vdev_ms_count; m++) { + + if (zcb->zcb_vd_obsolete_counts[c] != NULL) { + leaks |= zdb_check_for_obsolete_leaks(vd, zcb); + } + + for (uint64_t m = 0; m < vd->vdev_ms_count; m++) { metaslab_t *msp = vd->vdev_ms[m]; ASSERT3P(mg, ==, msp->ms_group); - mutex_enter(&msp->ms_lock); /* * The ms_tree has been overloaded to @@ -2860,18 +3266,30 @@ zdb_leak_fini(spa_t *spa) * represents an allocated block that we * did not claim during the traversal. * Claimed blocks would have been removed - * from the ms_tree. + * from the ms_tree. For indirect vdevs, + * space remaining in the tree represents + * parts of the mapping that are not + * referenced, which is not a bug. */ - range_tree_vacate(msp->ms_tree, zdb_leak, vd); + if (vd->vdev_ops == &vdev_indirect_ops) { + range_tree_vacate(msp->ms_tree, + NULL, NULL); + } else { + range_tree_vacate(msp->ms_tree, + zdb_leak, vd); + } if (msp->ms_loaded) { msp->ms_loaded = B_FALSE; } - - mutex_exit(&msp->ms_lock); } } + + umem_free(zcb->zcb_vd_obsolete_counts, + rvd->vdev_children * sizeof (uint32_t *)); + zcb->zcb_vd_obsolete_counts = NULL; } + return (leaks); } /* ARGSUSED */ @@ -2922,10 +3340,14 @@ dump_block_stats(spa_t *spa) */ (void) bpobj_iterate_nofree(&spa->spa_deferred_bpobj, count_block_cb, &zcb, NULL); + if (spa_version(spa) >= SPA_VERSION_DEADLISTS) { (void) bpobj_iterate_nofree(&spa->spa_dsl_pool->dp_free_bpobj, count_block_cb, &zcb, NULL); } + + zdb_claim_removing(spa, &zcb); + if (spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)) { VERIFY3U(0, ==, bptree_iterate(spa->spa_meta_objset, spa->spa_dsl_pool->dp_bptree_obj, B_FALSE, count_block_cb, @@ -2967,7 +3389,7 @@ dump_block_stats(spa_t *spa) /* * Report any leaked segments. */ - zdb_leak_fini(spa); + leaks |= zdb_leak_fini(spa, &zcb); tzb = &zcb.zcb_type[ZB_TOTAL][ZDB_OT_TOTAL]; @@ -2975,7 +3397,8 @@ dump_block_stats(spa_t *spa) norm_space = metaslab_class_get_space(spa_normal_class(spa)); total_alloc = norm_alloc + metaslab_class_get_alloc(spa_log_class(spa)); - total_found = tzb->zb_asize - zcb.zcb_dedup_asize; + total_found = tzb->zb_asize - zcb.zcb_dedup_asize + + zcb.zcb_removing_size; if (total_found == total_alloc) { if (!dump_opt['L']) @@ -3042,6 +3465,24 @@ dump_block_stats(spa_t *spa) (longlong_t)tzb->zb_ditto_samevdev); } + for (uint64_t v = 0; v < spa->spa_root_vdev->vdev_children; v++) { + vdev_t *vd = spa->spa_root_vdev->vdev_child[v]; + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + + if (vim == NULL) { + continue; + } + + char mem[32]; + zdb_nicenum(vdev_indirect_mapping_num_entries(vim), + mem, vdev_indirect_mapping_size(vim)); + + (void) printf("\tindirect vdev id %llu has %llu segments " + "(%s in memory)\n", + (longlong_t)vd->vdev_id, + (longlong_t)vdev_indirect_mapping_num_entries(vim), mem); + } + if (dump_opt['b'] >= 2) { int l, t, level; (void) printf("\nBlocks\tLSIZE\tPSIZE\tASIZE" @@ -3248,6 +3689,124 @@ dump_simulated_ddt(spa_t *spa) dump_dedup_ratio(&dds_total); } +static int +verify_device_removal_feature_counts(spa_t *spa) +{ + uint64_t dr_feature_refcount = 0; + uint64_t oc_feature_refcount = 0; + uint64_t indirect_vdev_count = 0; + uint64_t precise_vdev_count = 0; + uint64_t obsolete_counts_object_count = 0; + uint64_t obsolete_sm_count = 0; + uint64_t obsolete_counts_count = 0; + uint64_t scip_count = 0; + uint64_t obsolete_bpobj_count = 0; + int ret = 0; + + spa_condensing_indirect_phys_t *scip = + &spa->spa_condensing_indirect_phys; + if (scip->scip_next_mapping_object != 0) { + vdev_t *vd = spa->spa_root_vdev->vdev_child[scip->scip_vdev]; + ASSERT(scip->scip_prev_obsolete_sm_object != 0); + ASSERT3P(vd->vdev_ops, ==, &vdev_indirect_ops); + + (void) printf("Condensing indirect vdev %llu: new mapping " + "object %llu, prev obsolete sm %llu\n", + (u_longlong_t)scip->scip_vdev, + (u_longlong_t)scip->scip_next_mapping_object, + (u_longlong_t)scip->scip_prev_obsolete_sm_object); + if (scip->scip_prev_obsolete_sm_object != 0) { + space_map_t *prev_obsolete_sm = NULL; + VERIFY0(space_map_open(&prev_obsolete_sm, + spa->spa_meta_objset, + scip->scip_prev_obsolete_sm_object, + 0, vd->vdev_asize, 0)); + space_map_update(prev_obsolete_sm); + dump_spacemap(spa->spa_meta_objset, prev_obsolete_sm); + (void) printf("\n"); + space_map_close(prev_obsolete_sm); + } + + scip_count += 2; + } + + for (uint64_t i = 0; i < spa->spa_root_vdev->vdev_children; i++) { + vdev_t *vd = spa->spa_root_vdev->vdev_child[i]; + vdev_indirect_config_t *vic = &vd->vdev_indirect_config; + + if (vic->vic_mapping_object != 0) { + ASSERT(vd->vdev_ops == &vdev_indirect_ops || + vd->vdev_removing); + indirect_vdev_count++; + + if (vd->vdev_indirect_mapping->vim_havecounts) { + obsolete_counts_count++; + } + } + if (vdev_obsolete_counts_are_precise(vd)) { + ASSERT(vic->vic_mapping_object != 0); + precise_vdev_count++; + } + if (vdev_obsolete_sm_object(vd) != 0) { + ASSERT(vic->vic_mapping_object != 0); + obsolete_sm_count++; + } + } + + (void) feature_get_refcount(spa, + &spa_feature_table[SPA_FEATURE_DEVICE_REMOVAL], + &dr_feature_refcount); + (void) feature_get_refcount(spa, + &spa_feature_table[SPA_FEATURE_OBSOLETE_COUNTS], + &oc_feature_refcount); + + if (dr_feature_refcount != indirect_vdev_count) { + ret = 1; + (void) printf("Number of indirect vdevs (%llu) " \ + "does not match feature count (%llu)\n", + (u_longlong_t)indirect_vdev_count, + (u_longlong_t)dr_feature_refcount); + } else { + (void) printf("Verified device_removal feature refcount " \ + "of %llu is correct\n", + (u_longlong_t)dr_feature_refcount); + } + + if (zap_contains(spa_meta_objset(spa), DMU_POOL_DIRECTORY_OBJECT, + DMU_POOL_OBSOLETE_BPOBJ) == 0) { + obsolete_bpobj_count++; + } + + + obsolete_counts_object_count = precise_vdev_count; + obsolete_counts_object_count += obsolete_sm_count; + obsolete_counts_object_count += obsolete_counts_count; + obsolete_counts_object_count += scip_count; + obsolete_counts_object_count += obsolete_bpobj_count; + obsolete_counts_object_count += remap_deadlist_count; + + if (oc_feature_refcount != obsolete_counts_object_count) { + ret = 1; + (void) printf("Number of obsolete counts objects (%llu) " \ + "does not match feature count (%llu)\n", + (u_longlong_t)obsolete_counts_object_count, + (u_longlong_t)oc_feature_refcount); + (void) printf("pv:%llu os:%llu oc:%llu sc:%llu " + "ob:%llu rd:%llu\n", + (u_longlong_t)precise_vdev_count, + (u_longlong_t)obsolete_sm_count, + (u_longlong_t)obsolete_counts_count, + (u_longlong_t)scip_count, + (u_longlong_t)obsolete_bpobj_count, + (u_longlong_t)remap_deadlist_count); + } else { + (void) printf("Verified indirect_refcount feature refcount " \ + "of %llu is correct\n", + (u_longlong_t)oc_feature_refcount); + } + return (ret); +} + static void dump_zpool(spa_t *spa) { @@ -3281,18 +3840,24 @@ dump_zpool(spa_t *spa) if (dump_opt['d'] || dump_opt['i']) { dump_dir(dp->dp_meta_objset); if (dump_opt['d'] >= 3) { + dsl_pool_t *dp = spa->spa_dsl_pool; dump_full_bpobj(&spa->spa_deferred_bpobj, "Deferred frees", 0); if (spa_version(spa) >= SPA_VERSION_DEADLISTS) { - dump_full_bpobj( - &spa->spa_dsl_pool->dp_free_bpobj, + dump_full_bpobj(&dp->dp_free_bpobj, "Pool snapshot frees", 0); } + if (bpobj_is_open(&dp->dp_obsolete_bpobj)) { + ASSERT(spa_feature_is_enabled(spa, + SPA_FEATURE_DEVICE_REMOVAL)); + dump_full_bpobj(&dp->dp_obsolete_bpobj, + "Pool obsolete blocks", 0); + } if (spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)) { dump_bptree(spa->spa_meta_objset, - spa->spa_dsl_pool->dp_bptree_obj, + dp->dp_bptree_obj, "Pool dataset frees"); } dump_dtl(spa->spa_root_vdev, 0); @@ -3324,6 +3889,10 @@ dump_zpool(spa_t *spa) (longlong_t)refcount); } } + + if (rc == 0) { + rc = verify_device_removal_feature_counts(spa); + } } if (rc == 0 && (dump_opt['b'] || dump_opt['c'])) rc = dump_block_stats(spa); @@ -3631,7 +4200,8 @@ zdb_read_block(char *thing, spa_t *spa) psize, ZIO_TYPE_READ, ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_DONT_CACHE | ZIO_FLAG_DONT_QUEUE | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY | - ZIO_FLAG_CANFAIL | ZIO_FLAG_RAW, NULL, NULL)); + ZIO_FLAG_CANFAIL | ZIO_FLAG_RAW | ZIO_FLAG_OPTIONAL, + NULL, NULL)); } error = zio_wait(zio); Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Sun Feb 18 01:16:37 2018 (r329501) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Sun Feb 18 01:21:52 2018 (r329502) @@ -107,6 +107,7 @@ static int zfs_do_holds(int argc, char **argv); static int zfs_do_release(int argc, char **argv); static int zfs_do_diff(int argc, char **argv); static int zfs_do_bookmark(int argc, char **argv); +static int zfs_do_remap(int argc, char **argv); static int zfs_do_channel_program(int argc, char **argv); /* @@ -154,6 +155,7 @@ typedef enum { HELP_HOLDS, HELP_RELEASE, HELP_DIFF, + HELP_REMAP, HELP_BOOKMARK, HELP_CHANNEL_PROGRAM, } zfs_help_t; @@ -210,6 +212,7 @@ static zfs_command_t command_table[] = { { "holds", zfs_do_holds, HELP_HOLDS }, { "release", zfs_do_release, HELP_RELEASE }, { "diff", zfs_do_diff, HELP_DIFF }, + { "remap", zfs_do_remap, HELP_REMAP }, }; #define NCOMMAND (sizeof (command_table) / sizeof (command_table[0])) @@ -326,6 +329,8 @@ get_usage(zfs_help_t idx) case HELP_DIFF: return (gettext("\tdiff [-FHt] " "[snapshot|filesystem]\n")); + case HELP_REMAP: + return (gettext("\tremap \n")); case HELP_BOOKMARK: return (gettext("\tbookmark \n")); case HELP_CHANNEL_PROGRAM: @@ -4104,6 +4109,7 @@ zfs_do_receive(int argc, char **argv) #define ZFS_DELEG_PERM_RELEASE "release" #define ZFS_DELEG_PERM_DIFF "diff" #define ZFS_DELEG_PERM_BOOKMARK "bookmark" +#define ZFS_DELEG_PERM_REMAP "remap" #define ZFS_NUM_DELEG_NOTES ZFS_DELEG_NOTE_NONE @@ -4124,6 +4130,7 @@ static zfs_deleg_perm_tab_t zfs_deleg_perm_tbl[] = { { ZFS_DELEG_PERM_SHARE, ZFS_DELEG_NOTE_SHARE }, { ZFS_DELEG_PERM_SNAPSHOT, ZFS_DELEG_NOTE_SNAPSHOT }, { ZFS_DELEG_PERM_BOOKMARK, ZFS_DELEG_NOTE_BOOKMARK }, + { ZFS_DELEG_PERM_REMAP, ZFS_DELEG_NOTE_REMAP }, { ZFS_DELEG_PERM_GROUPQUOTA, ZFS_DELEG_NOTE_GROUPQUOTA }, { ZFS_DELEG_PERM_GROUPUSED, ZFS_DELEG_NOTE_GROUPUSED }, @@ -6829,7 +6836,7 @@ zfs_do_diff(int argc, char **argv) if (argc < 1) { (void) fprintf(stderr, - gettext("must provide at least one snapshot name\n")); + gettext("must provide at least one snapshot name\n")); usage(B_FALSE); } @@ -6868,6 +6875,22 @@ zfs_do_diff(int argc, char **argv) zfs_close(zhp); return (err != 0); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon Feb 19 04:47:33 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDBF2F008AF; Mon, 19 Feb 2018 04:47:33 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 669B87F833; Mon, 19 Feb 2018 04:47:33 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5648B15EB2; Mon, 19 Feb 2018 04:47:33 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1J4lXEW015783; Mon, 19 Feb 2018 04:47:33 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1J4lWto015772; Mon, 19 Feb 2018 04:47:32 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201802190447.w1J4lWto015772@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Mon, 19 Feb 2018 04:47:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329552 - vendor/less/dist X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/less/dist X-SVN-Commit-Revision: 329552 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Feb 2018 04:47:34 -0000 Author: delphij Date: Mon Feb 19 04:47:31 2018 New Revision: 329552 URL: https://svnweb.freebsd.org/changeset/base/329552 Log: Vendor import of less v530. Modified: vendor/less/dist/NEWS vendor/less/dist/README vendor/less/dist/command.c vendor/less/dist/help.c vendor/less/dist/less.man vendor/less/dist/less.nro vendor/less/dist/lessecho.man vendor/less/dist/lessecho.nro vendor/less/dist/lesskey.man vendor/less/dist/lesskey.nro vendor/less/dist/version.c Modified: vendor/less/dist/NEWS ============================================================================== --- vendor/less/dist/NEWS Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/NEWS Mon Feb 19 04:47:31 2018 (r329552) @@ -11,7 +11,7 @@ ====================================================================== - Major changes between "less" versions 487 and 529 + Major changes between "less" versions 487 and 530 * Don't output terminal init sequence if using -F and file fits on one screen. Modified: vendor/less/dist/README ============================================================================== --- vendor/less/dist/README Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/README Mon Feb 19 04:47:31 2018 (r329552) @@ -1,7 +1,7 @@ - Less, version 529 + Less, version 530 - This is the distribution of less, version 529, released 13 Nov 2017. + This is the distribution of less, version 530, released 05 Dec 2017. This program is part of the GNU project (http://www.gnu.org). This program is free software. You may redistribute it and/or Modified: vendor/less/dist/command.c ============================================================================== --- vendor/less/dist/command.c Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/command.c Mon Feb 19 04:47:31 2018 (r329552) @@ -806,7 +806,7 @@ getcc_end_command() * (characters previously given to ungetcc or ungetsc). */ static LWCHAR -getccu() +getccu(VOID_PARAM) { LWCHAR c; if (ungot == NULL) Modified: vendor/less/dist/help.c ============================================================================== --- vendor/less/dist/help.c Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/help.c Mon Feb 19 04:47:31 2018 (r329552) @@ -1,4 +1,4 @@ -/* This file was generated by mkhelp.pl from less.hlp at 17:31 on 2017/11/13 */ +/* This file was generated by mkhelp.pl from less.hlp at 22:56 on 2017/12/5 */ #include "less.h" constant char helpdata[] = { '\n', Modified: vendor/less/dist/less.man ============================================================================== --- vendor/less/dist/less.man Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/less.man Mon Feb 19 04:47:31 2018 (r329552) @@ -1106,7 +1106,9 @@ LESS(1) General Commands Manual ment file and less uses the original file, as normal. To use an input pipe, make the first character in the LESSOPEN environment variable a vertical bar (|) to signify that the input preprocessor is an input - pipe. + pipe. As with non-pipe input preprocessors, the command string must + contain one occurrence of %s, which is replaced with the filename of + the input file. For example, on many Unix systems, this script will work like the pre- vious example scripts: @@ -1664,4 +1666,4 @@ LESS(1) General Commands Manual - Version 529: 13 Nov 2017 LESS(1) + Version 530: 05 Dec 2017 LESS(1) Modified: vendor/less/dist/less.nro ============================================================================== --- vendor/less/dist/less.nro Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/less.nro Mon Feb 19 04:47:31 2018 (r329552) @@ -1,4 +1,4 @@ -.TH LESS 1 "Version 529: 13 Nov 2017" +.TH LESS 1 "Version 530: 05 Dec 2017" .SH NAME less \- opposite of more .SH SYNOPSIS @@ -1199,6 +1199,8 @@ uses the original file, as normal. To use an input pipe, make the first character in the LESSOPEN environment variable a vertical bar (|) to signify that the input preprocessor is an input pipe. +As with non-pipe input preprocessors, the command string must contain one +occurrence of %s, which is replaced with the filename of the input file. .PP For example, on many Unix systems, this script will work like the previous example scripts: Modified: vendor/less/dist/lessecho.man ============================================================================== --- vendor/less/dist/lessecho.man Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/lessecho.man Mon Feb 19 04:47:31 2018 (r329552) @@ -51,4 +51,4 @@ LESSECHO(1) General Commands Manual - Version 529: 13 Nov 2017 LESSECHO(1) + Version 530: 05 Dec 2017 LESSECHO(1) Modified: vendor/less/dist/lessecho.nro ============================================================================== --- vendor/less/dist/lessecho.nro Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/lessecho.nro Mon Feb 19 04:47:31 2018 (r329552) @@ -1,4 +1,4 @@ -.TH LESSECHO 1 "Version 529: 13 Nov 2017" +.TH LESSECHO 1 "Version 530: 05 Dec 2017" .SH NAME lessecho \- expand metacharacters .SH SYNOPSIS Modified: vendor/less/dist/lesskey.man ============================================================================== --- vendor/less/dist/lesskey.man Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/lesskey.man Mon Feb 19 04:47:31 2018 (r329552) @@ -361,4 +361,4 @@ LESSKEY(1) General Commands Manual - Version 529: 13 Nov 2017 LESSKEY(1) + Version 530: 05 Dec 2017 LESSKEY(1) Modified: vendor/less/dist/lesskey.nro ============================================================================== --- vendor/less/dist/lesskey.nro Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/lesskey.nro Mon Feb 19 04:47:31 2018 (r329552) @@ -1,4 +1,4 @@ -.TH LESSKEY 1 "Version 529: 13 Nov 2017" +.TH LESSKEY 1 "Version 530: 05 Dec 2017" .SH NAME lesskey \- specify key bindings for less .SH SYNOPSIS Modified: vendor/less/dist/version.c ============================================================================== --- vendor/less/dist/version.c Mon Feb 19 03:59:26 2018 (r329551) +++ vendor/less/dist/version.c Mon Feb 19 04:47:31 2018 (r329552) @@ -852,6 +852,7 @@ v526 10/25/17 Fix search hilite bug introduced in v5 v527 10/30/17 Fix search hilite bug on last page with -a. v528 11/3/17 Make second ESC-u clear status column. v529 11/12/17 Display Unicode formatting chars in hex if -U is set. +v530 12/2/17 Minor doc change and add missing VOID_PARAM. */ -char version[] = "529"; +char version[] = "530"; From owner-svn-src-vendor@freebsd.org Mon Feb 19 04:56:23 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DD6AF0123B; Mon, 19 Feb 2018 04:56:23 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D28E7FDAC; Mon, 19 Feb 2018 04:56:23 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 293ED16043; Mon, 19 Feb 2018 04:56:23 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1J4uNxE020509; Mon, 19 Feb 2018 04:56:23 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1J4uMns020508; Mon, 19 Feb 2018 04:56:23 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201802190456.w1J4uMns020508@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Mon, 19 Feb 2018 04:56:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329553 - vendor/less/v530 X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/less/v530 X-SVN-Commit-Revision: 329553 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Feb 2018 04:56:23 -0000 Author: delphij Date: Mon Feb 19 04:56:22 2018 New Revision: 329553 URL: https://svnweb.freebsd.org/changeset/base/329553 Log: Tag less v530. Added: vendor/less/v530/ - copied from r329552, vendor/less/dist/ From owner-svn-src-vendor@freebsd.org Wed Feb 21 14:12:30 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 550FBF04FEC; Wed, 21 Feb 2018 14:12:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 072A97CCAE; Wed, 21 Feb 2018 14:12:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0217719953; Wed, 21 Feb 2018 14:12:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LECTFW092247; Wed, 21 Feb 2018 14:12:29 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LECTSv092246; Wed, 21 Feb 2018 14:12:29 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201802211412.w1LECTSv092246@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 21 Feb 2018 14:12:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329710 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329710 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 14:12:30 -0000 Author: avg Date: Wed Feb 21 14:12:29 2018 New Revision: 329710 URL: https://svnweb.freebsd.org/changeset/base/329710 Log: 8966 Source file zfs_acl.c, function zfs_aclset_common contains a use after end of the lifetime of a local variable illumos/illumos-gate@82693e09cc02331fa1b3b73b54b1060e73507a8d https://github.com/illumos/illumos-gate/commit/82693e09cc02331fa1b3b73b54b1060e73507a8d https://www.illumos.org/issues/8966 Reviewed by: Matt Ahrens Reviewed by: Andriy Gapon Approved by: Richard Lowe Author: WHR Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c Wed Feb 21 14:07:53 2018 (r329709) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c Wed Feb 21 14:12:29 2018 (r329710) @@ -1188,6 +1188,7 @@ zfs_aclset_common(znode_t *zp, zfs_acl_t *aclp, cred_t sa_bulk_attr_t bulk[5]; uint64_t ctime[2]; int count = 0; + zfs_acl_phys_t acl_phys; mode = zp->z_mode; @@ -1234,7 +1235,6 @@ zfs_aclset_common(znode_t *zp, zfs_acl_t *aclp, cred_t } else { /* Painful legacy way */ zfs_acl_node_t *aclnode; uint64_t off = 0; - zfs_acl_phys_t acl_phys; uint64_t aoid; if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zfsvfs), From owner-svn-src-vendor@freebsd.org Wed Feb 21 14:30:35 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27FB6F06A3D; Wed, 21 Feb 2018 14:30:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCCC27E008; Wed, 21 Feb 2018 14:30:34 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AD94F19B04; Wed, 21 Feb 2018 14:30:34 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LEUYVn098048; Wed, 21 Feb 2018 14:30:34 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LEUYRm098047; Wed, 21 Feb 2018 14:30:34 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201802211430.w1LEUYRm098047@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 21 Feb 2018 14:30:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329713 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329713 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 14:30:35 -0000 Author: avg Date: Wed Feb 21 14:30:34 2018 New Revision: 329713 URL: https://svnweb.freebsd.org/changeset/base/329713 Log: 8731 ASSERT3U(nui64s, <=, UINT16_MAX) fails for large blocks illumos/illumos-gate@a6c1eb3c08094a6db69aa1dc6315bc814e82e79c https://github.com/illumos/illumos-gate/commit/a6c1eb3c08094a6db69aa1dc6315bc814e82e79c https://www.illumos.org/issues/8731 annotate_ecksum() asserts that nui64s, calculated as nui64s = size / sizeof (uint64_t), is not greater than UINT16_MAX. This restriction is needed because histograms of incorrectly set and cleared bits have 16 bit counters and if the buffer consists of too many 64-bit words, then a counter can potentially overflow producing an incorrect result. When the largest buffer size was 128KB the greatest value of nui64s was 16K, well within the limit. But now we have support for large buffers and for buffer sizes of 512KB and above the restriction is violated. Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fm.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fm.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fm.c Wed Feb 21 14:28:40 2018 (r329712) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fm.c Wed Feb 21 14:30:34 2018 (r329713) @@ -360,8 +360,8 @@ zfs_ereport_start(nvlist_t **ereport_out, nvlist_t **d typedef struct zfs_ecksum_info { /* histograms of set and cleared bits by bit number in a 64-bit word */ - uint16_t zei_histogram_set[sizeof (uint64_t) * NBBY]; - uint16_t zei_histogram_cleared[sizeof (uint64_t) * NBBY]; + uint32_t zei_histogram_set[sizeof (uint64_t) * NBBY]; + uint32_t zei_histogram_cleared[sizeof (uint64_t) * NBBY]; /* inline arrays of bits set and cleared. */ uint64_t zei_bits_set[ZFM_MAX_INLINE]; @@ -386,7 +386,7 @@ typedef struct zfs_ecksum_info { } zfs_ecksum_info_t; static void -update_histogram(uint64_t value_arg, uint16_t *hist, uint32_t *count) +update_histogram(uint64_t value_arg, uint32_t *hist, uint32_t *count) { size_t i; size_t bits = 0; @@ -552,7 +552,7 @@ annotate_ecksum(nvlist_t *ereport, zio_bad_cksum_t *in if (badbuf == NULL || goodbuf == NULL) return (eip); - ASSERT3U(nui64s, <=, UINT16_MAX); + ASSERT3U(nui64s, <=, UINT32_MAX); ASSERT3U(size, ==, nui64s * sizeof (uint64_t)); ASSERT3U(size, <=, SPA_MAXBLOCKSIZE); ASSERT3U(size, <=, UINT32_MAX); @@ -654,10 +654,10 @@ annotate_ecksum(nvlist_t *ereport, zio_bad_cksum_t *in } else { fm_payload_set(ereport, FM_EREPORT_PAYLOAD_ZFS_BAD_SET_HISTOGRAM, - DATA_TYPE_UINT16_ARRAY, + DATA_TYPE_UINT32_ARRAY, NBBY * sizeof (uint64_t), eip->zei_histogram_set, FM_EREPORT_PAYLOAD_ZFS_BAD_CLEARED_HISTOGRAM, - DATA_TYPE_UINT16_ARRAY, + DATA_TYPE_UINT32_ARRAY, NBBY * sizeof (uint64_t), eip->zei_histogram_cleared, NULL); } From owner-svn-src-vendor@freebsd.org Wed Feb 21 14:33:01 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F5F7F06F0C; Wed, 21 Feb 2018 14:33:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D59F47E55E; Wed, 21 Feb 2018 14:33:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D077219C8C; Wed, 21 Feb 2018 14:33:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LEX0mG002748; Wed, 21 Feb 2018 14:33:00 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LEX0aZ002743; Wed, 21 Feb 2018 14:33:00 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201802211433.w1LEX0aZ002743@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 21 Feb 2018 14:33:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329715 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 329715 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 14:33:01 -0000 Author: avg Date: Wed Feb 21 14:33:00 2018 New Revision: 329715 URL: https://svnweb.freebsd.org/changeset/base/329715 Log: 8997 ztest assertion failure in zil_lwb_write_issue illumos/illumos-gate@f864f99efe57685e1762590c1a880dd16bca6da9 https://github.com/illumos/illumos-gate/commit/f864f99efe57685e1762590c1a880dd16bca6da9 https://www.illumos.org/issues/8997 When dmu_tx_assign is called from zil_lwb_write_issue, it's possible for either ERESTART or EIO to be returned. If ERESTART is returned, this will cause an assertion to fail directly in zil_lwb_write_issue, where the code assumes the return value is EIO if dmu_tx_assign returns a non-zero value. This can occur if the SPA is suspended when dmu_tx_assign is called, and most often occurs when running zloop. If EIO is returned, this can cause assertions to fail elsewhere in the ZIL code. For example, zil_commit_waiter_timeout contains the following logic: lwb_t *nlwb = zil_lwb_write_issue(zilog, lwb); ASSERT3S(lwb->lwb_state, !=, LWB_STATE_OPENED); In this case, if dmu_tx_assign returned EIO from within zil_lwb_write_issue, the lwb variable passed in will not be issued to disk. Thus, it's lwb_state field will remain LWB_STATE_OPENED and this assertion will fail. zil_commit_waiter_timeout assumes that after it calls zil_lwb_write_issue, the lwb will be issued to disk, and doesn't handle the case where this is not true; i.e. it doesn't handle the case where dmu_tx_assign returns EIO. Reviewed by: Matt Ahrens Reviewed by: Andriy Gapon Approved by: Robert Mustacchi Author: Prakash Surya Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_tx.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Wed Feb 21 14:31:48 2018 (r329714) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Wed Feb 21 14:33:00 2018 (r329715) @@ -869,7 +869,7 @@ dmu_tx_delay(dmu_tx_t *tx, uint64_t dirty) * decreasing performance. */ static int -dmu_tx_try_assign(dmu_tx_t *tx, txg_how_t txg_how) +dmu_tx_try_assign(dmu_tx_t *tx, uint64_t txg_how) { spa_t *spa = tx->tx_pool->dp_spa; @@ -889,13 +889,13 @@ dmu_tx_try_assign(dmu_tx_t *tx, txg_how_t txg_how) * of the failuremode setting. */ if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_CONTINUE && - txg_how != TXG_WAIT) + !(txg_how & TXG_WAIT)) return (SET_ERROR(EIO)); return (SET_ERROR(ERESTART)); } - if (!tx->tx_waited && + if (!tx->tx_dirty_delayed && dsl_pool_need_dirty_delay(tx->tx_pool)) { tx->tx_wait_dirty = B_TRUE; return (SET_ERROR(ERESTART)); @@ -983,41 +983,44 @@ dmu_tx_unassign(dmu_tx_t *tx) } /* - * Assign tx to a transaction group. txg_how can be one of: + * Assign tx to a transaction group; txg_how is a bitmask: * - * (1) TXG_WAIT. If the current open txg is full, waits until there's - * a new one. This should be used when you're not holding locks. - * It will only fail if we're truly out of space (or over quota). + * If TXG_WAIT is set and the currently open txg is full, this function + * will wait until there's a new txg. This should be used when no locks + * are being held. With this bit set, this function will only fail if + * we're truly out of space (or over quota). * - * (2) TXG_NOWAIT. If we can't assign into the current open txg without - * blocking, returns immediately with ERESTART. This should be used - * whenever you're holding locks. On an ERESTART error, the caller - * should drop locks, do a dmu_tx_wait(tx), and try again. + * If TXG_WAIT is *not* set and we can't assign into the currently open + * txg without blocking, this function will return immediately with + * ERESTART. This should be used whenever locks are being held. On an + * ERESTART error, the caller should drop all locks, call dmu_tx_wait(), + * and try again. * - * (3) TXG_WAITED. Like TXG_NOWAIT, but indicates that dmu_tx_wait() - * has already been called on behalf of this operation (though - * most likely on a different tx). + * If TXG_NOTHROTTLE is set, this indicates that this tx should not be + * delayed due on the ZFS Write Throttle (see comments in dsl_pool.c for + * details on the throttle). This is used by the VFS operations, after + * they have already called dmu_tx_wait() (though most likely on a + * different tx). */ int -dmu_tx_assign(dmu_tx_t *tx, txg_how_t txg_how) +dmu_tx_assign(dmu_tx_t *tx, uint64_t txg_how) { int err; ASSERT(tx->tx_txg == 0); - ASSERT(txg_how == TXG_WAIT || txg_how == TXG_NOWAIT || - txg_how == TXG_WAITED); + ASSERT0(txg_how & ~(TXG_WAIT | TXG_NOTHROTTLE)); ASSERT(!dsl_pool_sync_context(tx->tx_pool)); /* If we might wait, we must not hold the config lock. */ - ASSERT(txg_how != TXG_WAIT || !dsl_pool_config_held(tx->tx_pool)); + IMPLY((txg_how & TXG_WAIT), !dsl_pool_config_held(tx->tx_pool)); - if (txg_how == TXG_WAITED) - tx->tx_waited = B_TRUE; + if ((txg_how & TXG_NOTHROTTLE)) + tx->tx_dirty_delayed = B_TRUE; while ((err = dmu_tx_try_assign(tx, txg_how)) != 0) { dmu_tx_unassign(tx); - if (err != ERESTART || txg_how != TXG_WAIT) + if (err != ERESTART || !(txg_how & TXG_WAIT)) return (err); dmu_tx_wait(tx); @@ -1054,12 +1057,12 @@ dmu_tx_wait(dmu_tx_t *tx) tx->tx_wait_dirty = B_FALSE; /* - * Note: setting tx_waited only has effect if the caller - * used TX_WAIT. Otherwise they are going to destroy - * this tx and try again. The common case, zfs_write(), - * uses TX_WAIT. + * Note: setting tx_dirty_delayed only has effect if the + * caller used TX_WAIT. Otherwise they are going to + * destroy this tx and try again. The common case, + * zfs_write(), uses TX_WAIT. */ - tx->tx_waited = B_TRUE; + tx->tx_dirty_delayed = B_TRUE; } else if (spa_suspended(spa) || tx->tx_lasttried_txg == 0) { /* * If the pool is suspended we need to wait until it Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Wed Feb 21 14:31:48 2018 (r329714) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Wed Feb 21 14:33:00 2018 (r329715) @@ -231,11 +231,14 @@ typedef enum dmu_object_type { DMU_OTN_ZAP_METADATA = DMU_OT(DMU_BSWAP_ZAP, B_TRUE), } dmu_object_type_t; -typedef enum txg_how { - TXG_WAIT = 1, - TXG_NOWAIT, - TXG_WAITED, -} txg_how_t; +/* + * These flags are intended to be used to specify the "txg_how" + * parameter when calling the dmu_tx_assign() function. See the comment + * above dmu_tx_assign() for more details on the meaning of these flags. + */ +#define TXG_NOWAIT (0ULL) +#define TXG_WAIT (1ULL<<0) +#define TXG_NOTHROTTLE (1ULL<<1) void byteswap_uint64_array(void *buf, size_t size); void byteswap_uint32_array(void *buf, size_t size); @@ -689,7 +692,7 @@ void dmu_tx_hold_spill(dmu_tx_t *tx, uint64_t object); void dmu_tx_hold_sa(dmu_tx_t *tx, struct sa_handle *hdl, boolean_t may_grow); void dmu_tx_hold_sa_create(dmu_tx_t *tx, int total_size); void dmu_tx_abort(dmu_tx_t *tx); -int dmu_tx_assign(dmu_tx_t *tx, enum txg_how txg_how); +int dmu_tx_assign(dmu_tx_t *tx, uint64_t txg_how); void dmu_tx_wait(dmu_tx_t *tx); void dmu_tx_commit(dmu_tx_t *tx); void dmu_tx_mark_netfree(dmu_tx_t *tx); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_tx.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_tx.h Wed Feb 21 14:31:48 2018 (r329714) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_tx.h Wed Feb 21 14:33:00 2018 (r329715) @@ -67,9 +67,6 @@ struct dmu_tx { /* placeholder for syncing context, doesn't need specific holds */ boolean_t tx_anyobj; - /* has this transaction already been delayed? */ - boolean_t tx_waited; - /* transaction is marked as being a "net free" of space */ boolean_t tx_netfree; @@ -79,6 +76,9 @@ struct dmu_tx { /* need to wait for sufficient dirty space */ boolean_t tx_wait_dirty; + /* has this transaction already been delayed? */ + boolean_t tx_dirty_delayed; + int tx_err; }; @@ -114,7 +114,7 @@ typedef struct dmu_tx_callback { * These routines are defined in dmu.h, and are called by the user. */ dmu_tx_t *dmu_tx_create(objset_t *dd); -int dmu_tx_assign(dmu_tx_t *tx, txg_how_t txg_how); +int dmu_tx_assign(dmu_tx_t *tx, uint64_t txg_how); void dmu_tx_commit(dmu_tx_t *tx); void dmu_tx_abort(dmu_tx_t *tx); uint64_t dmu_tx_get_txg(dmu_tx_t *tx); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Wed Feb 21 14:31:48 2018 (r329714) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Wed Feb 21 14:33:00 2018 (r329715) @@ -135,7 +135,7 @@ * * If dmu_tx_assign() returns ERESTART and zfsvfs->z_assign is TXG_NOWAIT, * then drop all locks, call dmu_tx_wait(), and try again. On subsequent - * calls to dmu_tx_assign(), pass TXG_WAITED rather than TXG_NOWAIT, + * calls to dmu_tx_assign(), pass TXG_NOTHROTTLE in addition to TXG_NOWAIT, * to indicate that this operation has already called dmu_tx_wait(). * This will ensure that we don't retry forever, waiting a short bit * each time. @@ -160,7 +160,7 @@ * rw_enter(...); // grab any other locks you need * tx = dmu_tx_create(...); // get DMU tx * dmu_tx_hold_*(); // hold each object you might modify - * error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + * error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); * if (error) { * rw_exit(...); // drop locks * zfs_dirent_unlock(dl); // unlock directory entry @@ -1532,7 +1532,8 @@ top: dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, acl_ids.z_aclp->z_acl_bytes); } - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, + (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART) { @@ -1762,7 +1763,7 @@ top: */ dmu_tx_mark_netfree(tx); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { zfs_dirent_unlock(dl); VN_RELE(vp); @@ -1999,7 +2000,7 @@ top: dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes + ZFS_SA_BASE_ATTR_SIZE); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART) { @@ -2136,7 +2137,7 @@ top: zfs_sa_upgrade_txholds(tx, zp); zfs_sa_upgrade_txholds(tx, dzp); dmu_tx_mark_netfree(tx); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { rw_exit(&zp->z_parent_lock); rw_exit(&zp->z_name_lock); @@ -3709,7 +3710,7 @@ top: zfs_sa_upgrade_txholds(tx, szp); dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { if (zl != NULL) zfs_rename_unlock(&zl); @@ -3902,7 +3903,7 @@ top: } if (fuid_dirtied) zfs_fuid_txhold(zfsvfs, tx); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART) { @@ -4123,7 +4124,7 @@ top: dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name); zfs_sa_upgrade_txholds(tx, szp); zfs_sa_upgrade_txholds(tx, dzp); - error = dmu_tx_assign(tx, waited ? TXG_WAITED : TXG_NOWAIT); + error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) | TXG_NOWAIT); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART) { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Feb 21 14:31:48 2018 (r329714) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Feb 21 14:33:00 2018 (r329715) @@ -1208,22 +1208,13 @@ zil_lwb_write_issue(zilog_t *zilog, lwb_t *lwb) tx = dmu_tx_create(zilog->zl_os); /* - * Since we are not going to create any new dirty data and we can even - * help with clearing the existing dirty data, we should not be subject - * to the dirty data based delays. - * We (ab)use TXG_WAITED to bypass the delay mechanism. - * One side effect from using TXG_WAITED is that dmu_tx_assign() can - * fail if the pool is suspended. Those are dramatic circumstances, - * so we return NULL to signal that the normal ZIL processing is not - * possible and txg_wait_synced() should be used to ensure that the data - * is on disk. + * Since we are not going to create any new dirty data, and we + * can even help with clearing the existing dirty data, we + * should not be subject to the dirty data based delays. We + * use TXG_NOTHROTTLE to bypass the delay mechanism. */ - error = dmu_tx_assign(tx, TXG_WAITED); - if (error != 0) { - ASSERT3S(error, ==, EIO); - dmu_tx_abort(tx); - return (NULL); - } + VERIFY0(dmu_tx_assign(tx, TXG_WAIT | TXG_NOTHROTTLE)); + dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx); txg = dmu_tx_get_txg(tx); From owner-svn-src-vendor@freebsd.org Wed Feb 21 15:10:34 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CDC2F0A6D9; Wed, 21 Feb 2018 15:10:34 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA8DE8030D; Wed, 21 Feb 2018 15:10:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C59091A172; Wed, 21 Feb 2018 15:10:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LFAXOv020324; Wed, 21 Feb 2018 15:10:33 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LFAXq3020204; Wed, 21 Feb 2018 15:10:33 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201802211510.w1LFAXq3020204@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 21 Feb 2018 15:10:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329718 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 329718 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 15:10:34 -0000 Author: avg Date: Wed Feb 21 15:10:33 2018 New Revision: 329718 URL: https://svnweb.freebsd.org/changeset/base/329718 Log: 8520 7198 lzc_rollback_to should support rolling back to origin illumos/illumos-gate@95643f75d23914a3e332adc9661ed51749e9858d https://github.com/illumos/illumos-gate/commit/95643f75d23914a3e332adc9661ed51749e9858d https://www.illumos.org/issues/8520 lzc_rollback_to() should support rolling back to a clone's origin. The current checks in zfs_ioc_rollback() would not allow that because the origin snapshot belongs to a different filesystem. The overly restrictive check was introduced in 7600, but it was not a regression as none of the existing tools provided a way to rollback to the origin. https://www.illumos.org/issues/7198 EINVAL is returned when a dataset does not have any snapshots, so there is nothing to roll back to. Although the code in zfs_do_rollback checks for that condition in advance, it's still possible that the snapshot(s) gets removed after the check and before the rollback sync task is executed. At the moment zfs command would crash when that happens. Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Feb 21 15:07:49 2018 (r329717) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Feb 21 15:10:33 2018 (r329718) @@ -2539,7 +2539,7 @@ dsl_dataset_rollback_check(void *arg, dmu_tx_t *tx) /* must have a most recent snapshot */ if (dsl_dataset_phys(ds)->ds_prev_snap_txg < TXG_INITIAL) { dsl_dataset_rele(ds, FTAG); - return (SET_ERROR(EINVAL)); + return (SET_ERROR(ESRCH)); } /* @@ -2559,11 +2559,46 @@ dsl_dataset_rollback_check(void *arg, dmu_tx_t *tx) * the latest snapshot is it. */ if (ddra->ddra_tosnap != NULL) { - char namebuf[ZFS_MAX_DATASET_NAME_LEN]; + dsl_dataset_t *snapds; - dsl_dataset_name(ds->ds_prev, namebuf); - if (strcmp(namebuf, ddra->ddra_tosnap) != 0) - return (SET_ERROR(EXDEV)); + /* Check if the target snapshot exists at all. */ + error = dsl_dataset_hold(dp, ddra->ddra_tosnap, FTAG, &snapds); + if (error != 0) { + /* + * ESRCH is used to signal that the target snapshot does + * not exist, while ENOENT is used to report that + * the rolled back dataset does not exist. + * ESRCH is also used to cover other cases where the + * target snapshot is not related to the dataset being + * rolled back such as being in a different pool. + */ + if (error == ENOENT || error == EXDEV) + error = SET_ERROR(ESRCH); + dsl_dataset_rele(ds, FTAG); + return (error); + } + ASSERT(snapds->ds_is_snapshot); + + /* Check if the snapshot is the latest snapshot indeed. */ + if (snapds != ds->ds_prev) { + /* + * Distinguish between the case where the only problem + * is intervening snapshots (EEXIST) vs the snapshot + * not being a valid target for rollback (ESRCH). + */ + if (snapds->ds_dir == ds->ds_dir || + (dsl_dir_is_clone(ds->ds_dir) && + dsl_dir_phys(ds->ds_dir)->dd_origin_obj == + snapds->ds_object)) { + error = SET_ERROR(EEXIST); + } else { + error = SET_ERROR(ESRCH); + } + dsl_dataset_rele(snapds, FTAG); + dsl_dataset_rele(ds, FTAG); + return (error); + } + dsl_dataset_rele(snapds, FTAG); } /* must not have any bookmarks after the most recent snapshot */ @@ -2572,8 +2607,10 @@ dsl_dataset_rollback_check(void *arg, dmu_tx_t *tx) nvlist_t *bookmarks = fnvlist_alloc(); error = dsl_get_bookmarks_impl(ds, proprequest, bookmarks); fnvlist_free(proprequest); - if (error != 0) + if (error != 0) { + dsl_dataset_rele(ds, FTAG); return (error); + } for (nvpair_t *pair = nvlist_next_nvpair(bookmarks, NULL); pair != NULL; pair = nvlist_next_nvpair(bookmarks, pair)) { nvlist_t *valuenv = Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Feb 21 15:07:49 2018 (r329717) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Feb 21 15:10:33 2018 (r329718) @@ -3701,11 +3701,14 @@ zfs_ioc_rollback(const char *fsname, nvlist_t *innvl, (void) nvlist_lookup_string(innvl, "target", &target); if (target != NULL) { - int fslen = strlen(fsname); + const char *cp = strchr(target, '@'); - if (strncmp(fsname, target, fslen) != 0) - return (SET_ERROR(EINVAL)); - if (target[fslen] != '@') + /* + * The snap name must contain an @, and the part after it must + * contain only valid characters. + */ + if (cp == NULL || + zfs_component_namecheck(cp + 1, NULL, NULL) != 0) return (SET_ERROR(EINVAL)); } From owner-svn-src-vendor@freebsd.org Wed Feb 21 15:10:34 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 481D4F0A6DB; Wed, 21 Feb 2018 15:10:34 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ECE6B80310; Wed, 21 Feb 2018 15:10:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E3C531A174; Wed, 21 Feb 2018 15:10:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LFAXaH020447; Wed, 21 Feb 2018 15:10:33 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LFAXWh020437; Wed, 21 Feb 2018 15:10:33 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201802211510.w1LFAXWh020437@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 21 Feb 2018 15:10:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329718 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 329718 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 15:10:34 -0000 Author: avg Date: Wed Feb 21 15:10:33 2018 New Revision: 329718 URL: https://svnweb.freebsd.org/changeset/base/329718 Log: 8520 7198 lzc_rollback_to should support rolling back to origin illumos/illumos-gate@95643f75d23914a3e332adc9661ed51749e9858d https://github.com/illumos/illumos-gate/commit/95643f75d23914a3e332adc9661ed51749e9858d https://www.illumos.org/issues/8520 lzc_rollback_to() should support rolling back to a clone's origin. The current checks in zfs_ioc_rollback() would not allow that because the origin snapshot belongs to a different filesystem. The overly restrictive check was introduced in 7600, but it was not a regression as none of the existing tools provided a way to rollback to the origin. https://www.illumos.org/issues/7198 EINVAL is returned when a dataset does not have any snapshots, so there is nothing to roll back to. Although the code in zfs_do_rollback checks for that condition in advance, it's still possible that the snapshot(s) gets removed after the check and before the rollback sync task is executed. At the moment zfs command would crash when that happens. Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Feb 21 15:07:49 2018 (r329717) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Feb 21 15:10:33 2018 (r329718) @@ -4023,17 +4023,31 @@ zfs_rollback(zfs_handle_t *zhp, zfs_handle_t *snap, bo * a new snapshot is created before this request is processed. */ err = lzc_rollback_to(zhp->zfs_name, snap->zfs_name); - if (err == EXDEV) { - zfs_error_aux(zhp->zfs_hdl, dgettext(TEXT_DOMAIN, - "'%s' is not the latest snapshot"), snap->zfs_name); - (void) zfs_error_fmt(zhp->zfs_hdl, EZFS_BUSY, + if (err != 0) { + char errbuf[1024]; + + (void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN, "cannot rollback '%s'"), zhp->zfs_name); - return (err); - } else if (err != 0) { - (void) zfs_standard_error_fmt(zhp->zfs_hdl, errno, - dgettext(TEXT_DOMAIN, "cannot rollback '%s'"), - zhp->zfs_name); + switch (err) { + case EEXIST: + zfs_error_aux(zhp->zfs_hdl, dgettext(TEXT_DOMAIN, + "there is a snapshot or bookmark more recent " + "than '%s'"), snap->zfs_name); + (void) zfs_error(zhp->zfs_hdl, EZFS_EXISTS, errbuf); + break; + case ESRCH: + zfs_error_aux(zhp->zfs_hdl, dgettext(TEXT_DOMAIN, + "'%s' is not found among snapshots of '%s'"), + snap->zfs_name, zhp->zfs_name); + (void) zfs_error(zhp->zfs_hdl, EZFS_NOENT, errbuf); + break; + case EINVAL: + (void) zfs_error(zhp->zfs_hdl, EZFS_BADTYPE, errbuf); + break; + default: + (void) zfs_standard_error(zhp->zfs_hdl, err, errbuf); + } return (err); } From owner-svn-src-vendor@freebsd.org Wed Feb 21 18:09:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1D14F1AA51; Wed, 21 Feb 2018 18:09:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9333E693E5; Wed, 21 Feb 2018 18:09:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8E21F1BE82; Wed, 21 Feb 2018 18:09:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LI97x5011698; Wed, 21 Feb 2018 18:09:07 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LI97iE011697; Wed, 21 Feb 2018 18:09:07 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802211809.w1LI97iE011697@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 18:09:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329736 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329736 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 18:09:08 -0000 Author: mav Date: Wed Feb 21 18:09:07 2018 New Revision: 329736 URL: https://svnweb.freebsd.org/changeset/base/329736 Log: 8969 Cannot boot from RAIDZ with parity > 1 illumos/illumos-gate@0fb055e81fd0cda5221da8ddd98b2f8d1fc6bdbe At present it is possible to boot from a root pool that is on RAIDZ but not one that is on RAIDZ2 or RAIDZ3. This is because, at the time the pool version is checked to ensure support for dual/triple parity, the uberblock has not yet been loaded into the SPA and therefore the code determines that the pool version is too old and returns ENOTSUP. Reviewed by: Igor Kozhukhov Reviewed by: Andriy Gapon Reviewed by: Pavel Zakharov Reviewed by: Andy Stormont Reviewed by: Toomas Soome Approved by: Gordon Ross Author: Andy Fiddaman Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 17:46:13 2018 (r329735) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 18:09:07 2018 (r329736) @@ -29,6 +29,7 @@ * Copyright 2016 Toomas Soome * Copyright 2017 Joyent, Inc. * Copyright (c) 2017 Datto Inc. + * Copyright 2018 OmniOS Community Edition (OmniOSce) Association. */ /* @@ -4051,6 +4052,9 @@ spa_import_rootpool(char *devpath, char *devid) spa = spa_add(pname, config, NULL); spa->spa_is_root = B_TRUE; spa->spa_import_flags = ZFS_IMPORT_VERBATIM; + if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION, + &spa->spa_ubsync.ub_version) != 0) + spa->spa_ubsync.ub_version = SPA_VERSION_INITIAL; /* * Build up a vdev tree based on the boot device's label config. From owner-svn-src-vendor@freebsd.org Wed Feb 21 21:04:48 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B800F001A0; Wed, 21 Feb 2018 21:04:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF09F7305E; Wed, 21 Feb 2018 21:04:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B5A561DBE1; Wed, 21 Feb 2018 21:04:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LL4lJY006678; Wed, 21 Feb 2018 21:04:47 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LL4k9F006671; Wed, 21 Feb 2018 21:04:46 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802212104.w1LL4k9F006671@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 21:04:46 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329753 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fm vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/... X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fm vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist... X-SVN-Commit-Revision: 329753 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 21:04:48 -0000 Author: mav Date: Wed Feb 21 21:04:46 2018 New Revision: 329753 URL: https://svnweb.freebsd.org/changeset/base/329753 Log: 8809 libzpool should leverage work done in libfakekernel illumos/illumos-gate@f06dce2c1f0f3af78581e7574f65bfba843ddb6e Reviewed by: Sebastien Roy Reviewed by: Prakash Surya Reviewed by: Gordon Ross Approved by: Richard Lowe Author: Andrew Stormont Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/sys/acl.h vendor-sys/illumos/dist/uts/common/sys/bitmap.h vendor-sys/illumos/dist/uts/common/sys/cpupart.h vendor-sys/illumos/dist/uts/common/sys/cpuvar.h vendor-sys/illumos/dist/uts/common/sys/fm/util.h vendor-sys/illumos/dist/uts/common/sys/vnode.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h vendor/illumos/dist/lib/libzpool/common/taskq.c vendor/illumos/dist/lib/libzpool/common/util.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Wed Feb 21 21:04:46 2018 (r329753) @@ -23,6 +23,7 @@ * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. * Copyright (c) 2014 Integros [integros.com] + * Copyright 2017 RackTop Systems. */ #include @@ -59,7 +60,9 @@ static dnode_phys_t dnode_phys_zero; int zfs_default_bs = SPA_MINBLOCKSHIFT; int zfs_default_ibs = DN_MAX_INDBLKSHIFT; +#ifdef _KERNEL static kmem_cbrc_t dnode_move(void *, void *, size_t, void *); +#endif /* _KERNEL */ static int dbuf_compare(const void *x1, const void *x2) @@ -213,7 +216,9 @@ dnode_init(void) dnode_cache = kmem_cache_create("dnode_t", sizeof (dnode_t), 0, dnode_cons, dnode_dest, NULL, NULL, NULL, 0); +#ifdef _KERNEL kmem_cache_set_move(dnode_cache, dnode_move); +#endif /* _KERNEL */ } void @@ -405,7 +410,9 @@ dnode_create(objset_t *os, dnode_phys_t *dnp, dmu_buf_ dnode_t *dn; dn = kmem_cache_alloc(dnode_cache, KM_SLEEP); +#ifdef _KERNEL ASSERT(!POINTER_IS_VALID(dn->dn_objset)); +#endif /* _KERNEL */ dn->dn_moved = 0; /* @@ -697,6 +704,7 @@ static struct { } dnode_move_stats; #endif /* DNODE_STATS */ +#ifdef _KERNEL static void dnode_move_impl(dnode_t *odn, dnode_t *ndn) { @@ -833,7 +841,6 @@ dnode_move_impl(dnode_t *odn, dnode_t *ndn) odn->dn_moved = (uint8_t)-1; } -#ifdef _KERNEL /*ARGSUSED*/ static kmem_cbrc_t dnode_move(void *buf, void *newbuf, size_t size, void *arg) Modified: vendor-sys/illumos/dist/uts/common/sys/acl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/acl.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/acl.h Wed Feb 21 21:04:46 2018 (r329753) @@ -25,6 +25,7 @@ * Use is subject to license terms. * * Copyright 2014 Nexenta Systems, Inc. All rights reserved. + * Copyright 2017 RackTop Systems. */ #ifndef _SYS_ACL_H @@ -138,7 +139,7 @@ typedef struct acl_info acl_t; #define ACL_FLAGS_ALL (ACL_AUTO_INHERIT|ACL_PROTECTED| \ ACL_DEFAULTED) -#ifdef _KERNEL +#if defined(_KERNEL) || defined(_FAKE_KERNEL) /* * These are only applicable in a CIFS context. Modified: vendor-sys/illumos/dist/uts/common/sys/bitmap.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/bitmap.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/bitmap.h Wed Feb 21 21:04:46 2018 (r329753) @@ -26,6 +26,7 @@ /* * Copyright (c) 2014 by Delphix. All rights reserved. + * Copyright 2017 RackTop Systems. */ /* Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T */ @@ -134,7 +135,7 @@ extern "C" { #define BIT_ONLYONESET(u) \ ((((u) == 0) ? 0 : ((u) & ((u) - 1)) == 0)) -#if defined(_KERNEL) && !defined(_ASM) +#if (defined(_KERNEL) || defined(_FAKE_KERNEL)) && !defined(_ASM) #include /* @@ -188,7 +189,7 @@ extern int odd_parity(ulong_t); */ #define BITX(u, h, l) (((u) >> (l)) & ((1LU << ((h) - (l) + 1LU)) - 1LU)) -#endif /* _KERNEL && !_ASM */ +#endif /* (_KERNEL || _FAKE_KERNEL) && !_ASM */ #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/sys/cpupart.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/cpupart.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/cpupart.h Wed Feb 21 21:04:46 2018 (r329753) @@ -20,6 +20,7 @@ */ /* * Copyright (c) 1996, 2010, Oracle and/or its affiliates. All rights reserved. + * Copyright 2017 RackTop Systems. */ #ifndef _SYS_CPUPART_H @@ -40,7 +41,7 @@ extern "C" { #endif -#ifdef _KERNEL +#if defined(_KERNEL) || defined(_FAKE_KERNEL) typedef int cpupartid_t; @@ -148,7 +149,7 @@ extern uint_t cpupart_list(psetid_t *, uint_t, int); extern int cpupart_setattr(psetid_t, uint_t); extern int cpupart_getattr(psetid_t, uint_t *); -#endif /* _KERNEL */ +#endif /* _KERNEL || _FAKE_KERNEL */ #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/sys/cpuvar.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/cpuvar.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/cpuvar.h Wed Feb 21 21:04:46 2018 (r329753) @@ -23,6 +23,7 @@ * Copyright (c) 1989, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012 by Delphix. All rights reserved. * Copyright 2014 Igor Kozhukhov . + * Copyright 2017 RackTop Systems. */ #ifndef _SYS_CPUVAR_H @@ -32,6 +33,7 @@ #include /* has cpu_stat_t definition */ #include #include +#include /* has kcpc_ctx_t definition */ #include #if (defined(_KERNEL) || defined(_KMEMUSER)) && defined(_MACHDEP) @@ -614,9 +616,9 @@ extern struct cpu *curcpup(void); #endif /* _KERNEL || _KMEMUSER */ /* - * CPU support routines. + * CPU support routines (not for genassym.c) */ -#if defined(_KERNEL) && defined(__STDC__) /* not for genassym.c */ +#if (defined(_KERNEL) || defined(_FAKE_KERNEL)) && defined(__STDC__) struct zone; @@ -823,7 +825,7 @@ extern void populate_idstr(struct cpu *); extern void cpu_vm_data_init(struct cpu *); extern void cpu_vm_data_destroy(struct cpu *); -#endif /* _KERNEL */ +#endif /* _KERNEL || _FAKE_KERNEL */ #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/sys/fm/util.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/fm/util.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/fm/util.h Wed Feb 21 21:04:46 2018 (r329753) @@ -21,6 +21,7 @@ /* * Copyright (c) 2004, 2010, Oracle and/or its affiliates. All rights reserved. + * Copyright 2017 RackTop Systems. */ #ifndef _SYS_FM_UTIL_H @@ -70,7 +71,7 @@ typedef struct erpt_dump { } ed_tod_base; } erpt_dump_t; -#ifdef _KERNEL +#if defined(_KERNEL) || defined(_FAKE_KERNEL) #include #define FM_STK_DEPTH 20 /* maximum stack depth */ @@ -94,7 +95,7 @@ extern void fm_ereport_post(nvlist_t *, int); extern void fm_payload_stack_add(nvlist_t *, const pc_t *, int); extern int is_fm_panic(); -#endif /* _KERNEL */ +#endif /* _KERNEL || _FAKE_KERNEL */ #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/sys/vnode.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/vnode.h Wed Feb 21 20:56:28 2018 (r329752) +++ vendor-sys/illumos/dist/uts/common/sys/vnode.h Wed Feb 21 21:04:46 2018 (r329753) @@ -23,6 +23,7 @@ * Copyright (c) 1988, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2017, Joyent, Inc. * Copyright (c) 2011, 2017 by Delphix. All rights reserved. + * Copyright 2017 RackTop Systems. */ /* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */ @@ -340,6 +341,7 @@ typedef struct vnode { #define IS_SWAPVP(vp) (((vp)->v_flag & (VISSWAP | VSWAPLIKE)) != 0) +#ifdef _KERNEL typedef struct vn_vfslocks_entry { rwstlock_t ve_lock; void *ve_vpvfs; @@ -348,6 +350,7 @@ typedef struct vn_vfslocks_entry { char pad[64 - sizeof (rwstlock_t) - 2 * sizeof (void *) - \ sizeof (uint32_t)]; } vn_vfslocks_entry_t; +#endif /* * The following two flags are used to lock the v_vfsmountedhere field From owner-svn-src-vendor@freebsd.org Wed Feb 21 21:04:48 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85727F001AC; Wed, 21 Feb 2018 21:04:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F2157305F; Wed, 21 Feb 2018 21:04:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0FD3A1DBE2; Wed, 21 Feb 2018 21:04:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LL4mEq006687; Wed, 21 Feb 2018 21:04:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LL4lQO006683; Wed, 21 Feb 2018 21:04:47 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802212104.w1LL4lQO006683@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 21:04:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329753 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fm vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/... X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys vendor-sys/illumos/dist/uts/common/sys/fm vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist... X-SVN-Commit-Revision: 329753 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 21:04:48 -0000 Author: mav Date: Wed Feb 21 21:04:46 2018 New Revision: 329753 URL: https://svnweb.freebsd.org/changeset/base/329753 Log: 8809 libzpool should leverage work done in libfakekernel illumos/illumos-gate@f06dce2c1f0f3af78581e7574f65bfba843ddb6e Reviewed by: Sebastien Roy Reviewed by: Prakash Surya Reviewed by: Gordon Ross Approved by: Richard Lowe Author: Andrew Stormont Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h vendor/illumos/dist/lib/libzpool/common/taskq.c vendor/illumos/dist/lib/libzpool/common/util.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/sys/acl.h vendor-sys/illumos/dist/uts/common/sys/bitmap.h vendor-sys/illumos/dist/uts/common/sys/cpupart.h vendor-sys/illumos/dist/uts/common/sys/cpuvar.h vendor-sys/illumos/dist/uts/common/sys/fm/util.h vendor-sys/illumos/dist/uts/common/sys/vnode.h Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Wed Feb 21 20:56:28 2018 (r329752) +++ vendor/illumos/dist/cmd/zdb/zdb.c Wed Feb 21 21:04:46 2018 (r329753) @@ -24,6 +24,7 @@ * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 Nexenta Systems, Inc. + * Copyright 2017 RackTop Systems. */ #include @@ -87,11 +88,13 @@ extern int reference_tracking_enable; extern boolean_t zfs_recover; extern uint64_t zfs_arc_max, zfs_arc_meta_limit; extern int zfs_vdev_async_read_max_active; +extern int aok; #else int reference_tracking_enable; boolean_t zfs_recover; uint64_t zfs_arc_max, zfs_arc_meta_limit; int zfs_vdev_async_read_max_active; +int aok; #endif static const char cmdname[] = "zdb"; Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Wed Feb 21 20:56:28 2018 (r329752) +++ vendor/illumos/dist/cmd/ztest/ztest.c Wed Feb 21 21:04:46 2018 (r329753) @@ -25,6 +25,7 @@ * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 Joyent, Inc. + * Copyright 2017 RackTop Systems. */ /* @@ -245,8 +246,8 @@ typedef enum { typedef struct rll { void *rll_writer; int rll_readers; - mutex_t rll_lock; - cond_t rll_cv; + kmutex_t rll_lock; + kcondvar_t rll_cv; } rll_t; typedef struct rl { @@ -280,11 +281,11 @@ typedef struct ztest_od { typedef struct ztest_ds { ztest_shared_ds_t *zd_shared; objset_t *zd_os; - rwlock_t zd_zilog_lock; + krwlock_t zd_zilog_lock; zilog_t *zd_zilog; ztest_od_t *zd_od; /* debugging aid */ char zd_name[ZFS_MAX_DATASET_NAME_LEN]; - mutex_t zd_dirobj_lock; + kmutex_t zd_dirobj_lock; rll_t zd_object_lock[ZTEST_OBJECT_LOCKS]; rll_t zd_range_lock[ZTEST_RANGE_LOCKS]; } ztest_ds_t; @@ -395,7 +396,7 @@ ztest_info_t ztest_info[] = { * The callbacks are ordered by txg number. */ typedef struct ztest_cb_list { - mutex_t zcl_callbacks_lock; + kmutex_t zcl_callbacks_lock; list_t zcl_callbacks; } ztest_cb_list_t; @@ -430,7 +431,7 @@ ztest_shared_t *ztest_shared; static spa_t *ztest_spa = NULL; static ztest_ds_t *ztest_ds; -static mutex_t ztest_vdev_lock; +static kmutex_t ztest_vdev_lock; /* * The ztest_name_lock protects the pool and dataset namespace used by @@ -438,7 +439,7 @@ static mutex_t ztest_vdev_lock; * this lock as writer. Grabbing the lock as reader will ensure that the * namespace does not change while the lock is held. */ -static rwlock_t ztest_name_lock; +static krwlock_t ztest_name_lock; static boolean_t ztest_dump_core = B_TRUE; static boolean_t ztest_exiting; @@ -1094,8 +1095,8 @@ ztest_rll_init(rll_t *rll) { rll->rll_writer = NULL; rll->rll_readers = 0; - VERIFY(_mutex_init(&rll->rll_lock, USYNC_THREAD, NULL) == 0); - VERIFY(cond_init(&rll->rll_cv, USYNC_THREAD, NULL) == 0); + mutex_init(&rll->rll_lock, NULL, USYNC_THREAD, NULL); + cv_init(&rll->rll_cv, NULL, USYNC_THREAD, NULL); } static void @@ -1103,32 +1104,32 @@ ztest_rll_destroy(rll_t *rll) { ASSERT(rll->rll_writer == NULL); ASSERT(rll->rll_readers == 0); - VERIFY(_mutex_destroy(&rll->rll_lock) == 0); - VERIFY(cond_destroy(&rll->rll_cv) == 0); + mutex_destroy(&rll->rll_lock); + cv_destroy(&rll->rll_cv); } static void ztest_rll_lock(rll_t *rll, rl_type_t type) { - VERIFY(mutex_lock(&rll->rll_lock) == 0); + mutex_enter(&rll->rll_lock); if (type == RL_READER) { while (rll->rll_writer != NULL) - (void) cond_wait(&rll->rll_cv, &rll->rll_lock); + cv_wait(&rll->rll_cv, &rll->rll_lock); rll->rll_readers++; } else { while (rll->rll_writer != NULL || rll->rll_readers) - (void) cond_wait(&rll->rll_cv, &rll->rll_lock); + cv_wait(&rll->rll_cv, &rll->rll_lock); rll->rll_writer = curthread; } - VERIFY(mutex_unlock(&rll->rll_lock) == 0); + mutex_exit(&rll->rll_lock); } static void ztest_rll_unlock(rll_t *rll) { - VERIFY(mutex_lock(&rll->rll_lock) == 0); + mutex_enter(&rll->rll_lock); if (rll->rll_writer) { ASSERT(rll->rll_readers == 0); @@ -1140,9 +1141,9 @@ ztest_rll_unlock(rll_t *rll) } if (rll->rll_writer == NULL && rll->rll_readers == 0) - VERIFY(cond_broadcast(&rll->rll_cv) == 0); + cv_broadcast(&rll->rll_cv); - VERIFY(mutex_unlock(&rll->rll_lock) == 0); + mutex_exit(&rll->rll_lock); } static void @@ -1201,8 +1202,8 @@ ztest_zd_init(ztest_ds_t *zd, ztest_shared_ds_t *szd, if (zd->zd_shared != NULL) zd->zd_shared->zd_seq = 0; - VERIFY(rwlock_init(&zd->zd_zilog_lock, USYNC_THREAD, NULL) == 0); - VERIFY(_mutex_init(&zd->zd_dirobj_lock, USYNC_THREAD, NULL) == 0); + rw_init(&zd->zd_zilog_lock, NULL, USYNC_THREAD, NULL); + mutex_init(&zd->zd_dirobj_lock, NULL, USYNC_THREAD, NULL); for (int l = 0; l < ZTEST_OBJECT_LOCKS; l++) ztest_rll_init(&zd->zd_object_lock[l]); @@ -1214,7 +1215,7 @@ ztest_zd_init(ztest_ds_t *zd, ztest_shared_ds_t *szd, static void ztest_zd_fini(ztest_ds_t *zd) { - VERIFY(_mutex_destroy(&zd->zd_dirobj_lock) == 0); + mutex_destroy(&zd->zd_dirobj_lock); for (int l = 0; l < ZTEST_OBJECT_LOCKS; l++) ztest_rll_destroy(&zd->zd_object_lock[l]); @@ -1969,7 +1970,7 @@ ztest_lookup(ztest_ds_t *zd, ztest_od_t *od, int count int missing = 0; int error; - ASSERT(_mutex_held(&zd->zd_dirobj_lock)); + ASSERT(MUTEX_HELD(&zd->zd_dirobj_lock)); for (int i = 0; i < count; i++, od++) { od->od_object = 0; @@ -2009,7 +2010,7 @@ ztest_create(ztest_ds_t *zd, ztest_od_t *od, int count { int missing = 0; - ASSERT(_mutex_held(&zd->zd_dirobj_lock)); + ASSERT(MUTEX_HELD(&zd->zd_dirobj_lock)); for (int i = 0; i < count; i++, od++) { if (missing) { @@ -2054,7 +2055,7 @@ ztest_remove(ztest_ds_t *zd, ztest_od_t *od, int count int missing = 0; int error; - ASSERT(_mutex_held(&zd->zd_dirobj_lock)); + ASSERT(MUTEX_HELD(&zd->zd_dirobj_lock)); od += count - 1; @@ -2200,7 +2201,7 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off if (ztest_random(2) == 0) io_type = ZTEST_IO_WRITE_TAG; - (void) rw_rdlock(&zd->zd_zilog_lock); + rw_enter(&zd->zd_zilog_lock, RW_READER); switch (io_type) { @@ -2237,7 +2238,7 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off break; case ZTEST_IO_REWRITE: - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); err = ztest_dsl_prop_set_uint64(zd->zd_name, ZFS_PROP_CHECKSUM, spa_dedup_checksum(ztest_spa), B_FALSE); @@ -2247,7 +2248,7 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off ztest_random_dsl_prop(ZFS_PROP_COMPRESSION), B_FALSE); VERIFY(err == 0 || err == ENOSPC); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); VERIFY0(dmu_read(zd->zd_os, object, offset, blocksize, data, DMU_READ_NO_PREFETCH)); @@ -2256,7 +2257,7 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off break; } - (void) rw_unlock(&zd->zd_zilog_lock); + rw_exit(&zd->zd_zilog_lock); umem_free(data, blocksize); } @@ -2295,13 +2296,13 @@ ztest_object_init(ztest_ds_t *zd, ztest_od_t *od, size int count = size / sizeof (*od); int rv = 0; - VERIFY(mutex_lock(&zd->zd_dirobj_lock) == 0); + mutex_enter(&zd->zd_dirobj_lock); if ((ztest_lookup(zd, od, count) != 0 || remove) && (ztest_remove(zd, od, count) != 0 || ztest_create(zd, od, count) != 0)) rv = -1; zd->zd_od = od; - VERIFY(mutex_unlock(&zd->zd_dirobj_lock) == 0); + mutex_exit(&zd->zd_dirobj_lock); return (rv); } @@ -2312,7 +2313,7 @@ ztest_zil_commit(ztest_ds_t *zd, uint64_t id) { zilog_t *zilog = zd->zd_zilog; - (void) rw_rdlock(&zd->zd_zilog_lock); + rw_enter(&zd->zd_zilog_lock, RW_READER); zil_commit(zilog, ztest_random(ZTEST_OBJECTS)); @@ -2327,7 +2328,7 @@ ztest_zil_commit(ztest_ds_t *zd, uint64_t id) zd->zd_shared->zd_seq = zilog->zl_commit_lr_seq; mutex_exit(&zilog->zl_lock); - (void) rw_unlock(&zd->zd_zilog_lock); + rw_exit(&zd->zd_zilog_lock); } /* @@ -2346,8 +2347,8 @@ ztest_zil_remount(ztest_ds_t *zd, uint64_t id) * updating the zil (i.e. adding in-memory log records) and the * zd_zilog_lock to block any I/O. */ - VERIFY0(mutex_lock(&zd->zd_dirobj_lock)); - (void) rw_wrlock(&zd->zd_zilog_lock); + mutex_enter(&zd->zd_dirobj_lock); + rw_enter(&zd->zd_zilog_lock, RW_WRITER); /* zfsvfs_teardown() */ zil_close(zd->zd_zilog); @@ -2356,8 +2357,8 @@ ztest_zil_remount(ztest_ds_t *zd, uint64_t id) VERIFY(zil_open(os, ztest_get_data) == zd->zd_zilog); zil_replay(os, zd, ztest_replay_vector); - (void) rw_unlock(&zd->zd_zilog_lock); - VERIFY(mutex_unlock(&zd->zd_dirobj_lock) == 0); + rw_exit(&zd->zd_zilog_lock); + mutex_exit(&zd->zd_dirobj_lock); } /* @@ -2392,7 +2393,7 @@ ztest_spa_create_destroy(ztest_ds_t *zd, uint64_t id) * Attempt to create an existing pool. It shouldn't matter * what's in the nvroot; we should fail with EEXIST. */ - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); nvroot = make_vdev_root("/dev/bogus", NULL, NULL, 0, 0, 0, 0, 0, 1); VERIFY3U(EEXIST, ==, spa_create(zo->zo_pool, nvroot, NULL, NULL)); nvlist_free(nvroot); @@ -2400,7 +2401,7 @@ ztest_spa_create_destroy(ztest_ds_t *zd, uint64_t id) VERIFY3U(EBUSY, ==, spa_destroy(zo->zo_pool)); spa_close(spa, FTAG); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* ARGSUSED */ @@ -2413,7 +2414,7 @@ ztest_spa_upgrade(ztest_ds_t *zd, uint64_t id) nvlist_t *nvroot, *props; char *name; - VERIFY0(mutex_lock(&ztest_vdev_lock)); + mutex_enter(&ztest_vdev_lock); name = kmem_asprintf("%s_upgrade", ztest_opts.zo_pool); /* @@ -2472,7 +2473,7 @@ ztest_spa_upgrade(ztest_ds_t *zd, uint64_t id) spa_close(spa, FTAG); strfree(name); - VERIFY0(mutex_unlock(&ztest_vdev_lock)); + mutex_exit(&ztest_vdev_lock); } static vdev_t * @@ -2525,7 +2526,7 @@ ztest_vdev_add_remove(ztest_ds_t *zd, uint64_t id) nvlist_t *nvroot; int error; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); leaves = MAX(zs->zs_mirrors + zs->zs_splits, 1) * ztest_opts.zo_raidz; spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); @@ -2551,9 +2552,9 @@ ztest_vdev_add_remove(ztest_ds_t *zd, uint64_t id) * dmu_objset_destroy() to fail with EBUSY thus * leaving the dataset in an inconsistent state. */ - VERIFY(rw_wrlock(&ztest_name_lock) == 0); + rw_enter(&ztest_name_lock, RW_WRITER); error = spa_vdev_remove(spa, guid, B_FALSE); - VERIFY(rw_unlock(&ztest_name_lock) == 0); + rw_exit(&ztest_name_lock); if (error && error != EEXIST) fatal(0, "spa_vdev_remove() = %d", error); @@ -2577,7 +2578,7 @@ ztest_vdev_add_remove(ztest_ds_t *zd, uint64_t id) fatal(0, "spa_vdev_add() = %d", error); } - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } /* @@ -2603,7 +2604,7 @@ ztest_vdev_aux_add_remove(ztest_ds_t *zd, uint64_t id) aux = ZPOOL_CONFIG_L2CACHE; } - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); @@ -2660,7 +2661,7 @@ ztest_vdev_aux_add_remove(ztest_ds_t *zd, uint64_t id) fatal(0, "spa_vdev_remove(%llu) = %d", guid, error); } - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } /* @@ -2677,11 +2678,11 @@ ztest_split_pool(ztest_ds_t *zd, uint64_t id) uint_t c, children, schildren = 0, lastlogid = 0; int error = 0; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); /* ensure we have a useable config; mirrors of raidz aren't supported */ if (zs->zs_mirrors < 3 || ztest_opts.zo_raidz > 1) { - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -2740,9 +2741,9 @@ ztest_split_pool(ztest_ds_t *zd, uint64_t id) spa_config_exit(spa, SCL_VDEV, FTAG); - (void) rw_wrlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_WRITER); error = spa_vdev_split_mirror(spa, "splitp", config, NULL, B_FALSE); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); nvlist_free(config); @@ -2755,7 +2756,7 @@ ztest_split_pool(ztest_ds_t *zd, uint64_t id) ++zs->zs_splits; --zs->zs_mirrors; } - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } @@ -2784,7 +2785,7 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id) int oldvd_is_log; int error, expected_error; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); leaves = MAX(zs->zs_mirrors, 1) * ztest_opts.zo_raidz; spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); @@ -2797,7 +2798,7 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id) */ if (spa->spa_vdev_removal != NULL) { spa_config_exit(spa, SCL_ALL, FTAG); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -2857,7 +2858,7 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id) if (error != 0 && error != ENODEV && error != EBUSY && error != ENOTSUP) fatal(0, "detach (%s) returned %d", oldpath, error); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -2955,7 +2956,7 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id) newsize, replacing, error, expected_error); } - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } /* ARGSUSED */ @@ -2966,7 +2967,7 @@ ztest_device_removal(ztest_ds_t *zd, uint64_t id) vdev_t *vd; uint64_t guid; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); vd = vdev_lookup_top(spa, ztest_random_vdev_top(spa, B_FALSE)); @@ -2975,7 +2976,7 @@ ztest_device_removal(ztest_ds_t *zd, uint64_t id) (void) spa_vdev_remove(spa, guid, B_FALSE); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } /* @@ -3103,7 +3104,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) uint64_t top; uint64_t old_class_space, new_class_space, old_ms_count, new_ms_count; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); spa_config_enter(spa, SCL_STATE, spa, RW_READER); /* @@ -3114,7 +3115,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) */ if (spa->spa_vdev_removal != NULL) { spa_config_exit(spa, SCL_STATE, FTAG); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -3143,7 +3144,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) if (tvd->vdev_state != VDEV_STATE_HEALTHY || psize == 0 || psize >= 4 * ztest_opts.zo_vdev_size) { spa_config_exit(spa, SCL_STATE, spa); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } ASSERT(psize > 0); @@ -3168,7 +3169,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) "the vdev configuration changed.\n"); } spa_config_exit(spa, SCL_STATE, spa); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -3202,7 +3203,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) "intervening vdev offline or remove.\n"); } spa_config_exit(spa, SCL_STATE, spa); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); return; } @@ -3232,7 +3233,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) } spa_config_exit(spa, SCL_STATE, spa); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } /* @@ -3343,7 +3344,7 @@ ztest_dmu_objset_create_destroy(ztest_ds_t *zd, uint64 char name[ZFS_MAX_DATASET_NAME_LEN]; zilog_t *zilog; - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); (void) snprintf(name, sizeof (name), "%s/temp_%llu", ztest_opts.zo_pool, (u_longlong_t)id); @@ -3382,7 +3383,7 @@ ztest_dmu_objset_create_destroy(ztest_ds_t *zd, uint64 if (error) { if (error == ENOSPC) { ztest_record_enospc(FTAG); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); return; } fatal(0, "dmu_objset_create(%s) = %d", name, error); @@ -3430,7 +3431,7 @@ ztest_dmu_objset_create_destroy(ztest_ds_t *zd, uint64 dmu_objset_disown(os, FTAG); ztest_zd_fini(&zdtmp); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -3439,10 +3440,10 @@ ztest_dmu_objset_create_destroy(ztest_ds_t *zd, uint64 void ztest_dmu_snapshot_create_destroy(ztest_ds_t *zd, uint64_t id) { - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); (void) ztest_snapshot_destroy(zd->zd_name, id); (void) ztest_snapshot_create(zd->zd_name, id); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -3501,7 +3502,7 @@ ztest_dsl_dataset_promote_busy(ztest_ds_t *zd, uint64_ char *osname = zd->zd_name; int error; - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); ztest_dsl_dataset_cleanup(osname, id); @@ -3578,7 +3579,7 @@ ztest_dsl_dataset_promote_busy(ztest_ds_t *zd, uint64_ out: ztest_dsl_dataset_cleanup(osname, id); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -4512,9 +4513,9 @@ ztest_commit_callback(void *arg, int error) ASSERT3U(data->zcd_txg, !=, 0); /* Remove our callback from the list */ - (void) mutex_lock(&zcl.zcl_callbacks_lock); + mutex_enter(&zcl.zcl_callbacks_lock); list_remove(&zcl.zcl_callbacks, data); - (void) mutex_unlock(&zcl.zcl_callbacks_lock); + mutex_exit(&zcl.zcl_callbacks_lock); out: umem_free(data, sizeof (ztest_cb_data_t)); @@ -4616,7 +4617,7 @@ ztest_dmu_commit_callbacks(ztest_ds_t *zd, uint64_t id dmu_write(os, od[0].od_object, 0, sizeof (uint64_t), &txg, tx); - (void) mutex_lock(&zcl.zcl_callbacks_lock); + mutex_enter(&zcl.zcl_callbacks_lock); /* * Since commit callbacks don't have any ordering requirement and since @@ -4663,7 +4664,7 @@ ztest_dmu_commit_callbacks(ztest_ds_t *zd, uint64_t id tmp_cb = cb_data[i]; } - (void) mutex_unlock(&zcl.zcl_callbacks_lock); + mutex_exit(&zcl.zcl_callbacks_lock); dmu_tx_commit(tx); } @@ -4679,27 +4680,27 @@ ztest_dsl_prop_get_set(ztest_ds_t *zd, uint64_t id) ZFS_PROP_DEDUP }; - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); for (int p = 0; p < sizeof (proplist) / sizeof (proplist[0]); p++) (void) ztest_dsl_prop_set_uint64(zd->zd_name, proplist[p], ztest_random_dsl_prop(proplist[p]), (int)ztest_random(2)); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* ARGSUSED */ void ztest_remap_blocks(ztest_ds_t *zd, uint64_t id) { - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); int error = dmu_objset_remap_indirects(zd->zd_name); if (error == ENOSPC) error = 0; ASSERT0(error); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* ARGSUSED */ @@ -4708,7 +4709,7 @@ ztest_spa_prop_get_set(ztest_ds_t *zd, uint64_t id) { nvlist_t *props = NULL; - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); (void) ztest_spa_prop_set_uint64(ZPOOL_PROP_DEDUPDITTO, ZIO_DEDUPDITTO_MIN + ztest_random(ZIO_DEDUPDITTO_MIN)); @@ -4720,7 +4721,7 @@ ztest_spa_prop_get_set(ztest_ds_t *zd, uint64_t id) nvlist_free(props); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } static int @@ -4755,7 +4756,7 @@ ztest_dmu_snapshot_hold(ztest_ds_t *zd, uint64_t id) char osname[ZFS_MAX_DATASET_NAME_LEN]; nvlist_t *holds; - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); dmu_objset_name(os, osname); @@ -4860,7 +4861,7 @@ ztest_dmu_snapshot_hold(ztest_ds_t *zd, uint64_t id) VERIFY3U(dmu_objset_hold(fullname, FTAG, &origin), ==, ENOENT); out: - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -4888,11 +4889,11 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) uint64_t guid0 = 0; boolean_t islog = B_FALSE; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); maxfaults = MAXFAULTS(); leaves = MAX(zs->zs_mirrors, 1) * ztest_opts.zo_raidz; mirror_save = zs->zs_mirrors; - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); ASSERT(leaves >= 1); @@ -4902,7 +4903,7 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) * they are in progress (i.e. spa_change_guid). Those * operations will have grabbed the name lock as writer. */ - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); /* * We need SCL_STATE here because we're going to look at vd0->vdev_tsd. @@ -4974,7 +4975,7 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) if (sav->sav_count == 0) { spa_config_exit(spa, SCL_STATE, FTAG); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); return; } vd0 = sav->sav_vdevs[ztest_random(sav->sav_count)]; @@ -4988,7 +4989,7 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) } spa_config_exit(spa, SCL_STATE, FTAG); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); /* * If we can tolerate two or more faults, or we're dealing @@ -5008,12 +5009,12 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) * leaving the dataset in an inconsistent state. */ if (islog) - (void) rw_wrlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_WRITER); VERIFY(vdev_offline(spa, guid0, flags) != EBUSY); if (islog) - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } else { /* * Ideally we would like to be able to randomly @@ -5024,9 +5025,9 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) * prevent a race between injection testing and * aux_vdev removal. */ - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); (void) vdev_online(spa, guid0, 0, NULL); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); } } @@ -5098,9 +5099,9 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) offset + sizeof (bad) > psize - VDEV_LABEL_END_SIZE) continue; - VERIFY(mutex_lock(&ztest_vdev_lock) == 0); + mutex_enter(&ztest_vdev_lock); if (mirror_save != zs->zs_mirrors) { - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); (void) close(fd); return; } @@ -5109,7 +5110,7 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) fatal(1, "can't inject bad word at 0x%llx in %s", offset, pathrand); - VERIFY(mutex_unlock(&ztest_vdev_lock) == 0); + mutex_exit(&ztest_vdev_lock); if (ztest_opts.zo_verbose >= 7) (void) printf("injected bad word into %s," @@ -5149,13 +5150,13 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_t id) * Take the name lock as writer to prevent anyone else from changing * the pool and dataset properies we need to maintain during this test. */ - (void) rw_wrlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_WRITER); if (ztest_dsl_prop_set_uint64(zd->zd_name, ZFS_PROP_DEDUP, checksum, B_FALSE) != 0 || ztest_dsl_prop_set_uint64(zd->zd_name, ZFS_PROP_COPIES, 1, B_FALSE) != 0) { - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); return; } @@ -5174,7 +5175,7 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_t id) dmu_tx_hold_write(tx, object, 0, copies * blocksize); txg = ztest_tx_assign(tx, TXG_WAIT, FTAG); if (txg == 0) { - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); return; } @@ -5222,7 +5223,7 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_t id) abd_free(abd); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -5253,9 +5254,9 @@ ztest_reguid(ztest_ds_t *zd, uint64_t id) orig = spa_guid(spa); load = spa_load_guid(spa); - (void) rw_wrlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_WRITER); error = spa_change_guid(spa); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); if (error != 0) return; @@ -5279,7 +5280,7 @@ ztest_spa_rename(ztest_ds_t *zd, uint64_t id) char *oldname, *newname; spa_t *spa; - (void) rw_wrlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_WRITER); oldname = ztest_opts.zo_pool; newname = umem_alloc(strlen(oldname) + 5, UMEM_NOFAIL); @@ -5319,7 +5320,7 @@ ztest_spa_rename(ztest_ds_t *zd, uint64_t id) umem_free(newname, strlen(newname) + 1); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); } /* @@ -5674,18 +5675,18 @@ ztest_dataset_open(int d) ztest_dataset_name(name, ztest_opts.zo_pool, d); - (void) rw_rdlock(&ztest_name_lock); + rw_enter(&ztest_name_lock, RW_READER); error = ztest_dataset_create(name); if (error == ENOSPC) { - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); ztest_record_enospc(FTAG); return (error); } ASSERT(error == 0 || error == EEXIST); VERIFY0(dmu_objset_own(name, DMU_OST_OTHER, B_FALSE, zd, &os)); - (void) rw_unlock(&ztest_name_lock); + rw_exit(&ztest_name_lock); ztest_zd_init(zd, ZTEST_GET_SHARED_DS(d), os); @@ -5747,8 +5748,8 @@ ztest_run(ztest_shared_t *zs) /* * Initialize parent/child shared state. */ - VERIFY(_mutex_init(&ztest_vdev_lock, USYNC_THREAD, NULL) == 0); - VERIFY(rwlock_init(&ztest_name_lock, USYNC_THREAD, NULL) == 0); + mutex_init(&ztest_vdev_lock, NULL, USYNC_THREAD, NULL); + rw_init(&ztest_name_lock, NULL, USYNC_THREAD, NULL); zs->zs_thread_start = gethrtime(); zs->zs_thread_stop = @@ -5760,7 +5761,7 @@ ztest_run(ztest_shared_t *zs) ztest_random(ztest_opts.zo_passtime * NANOSEC); } - (void) _mutex_init(&zcl.zcl_callbacks_lock, USYNC_THREAD, NULL); + mutex_init(&zcl.zcl_callbacks_lock, NULL, USYNC_THREAD, NULL); list_create(&zcl.zcl_callbacks, sizeof (ztest_cb_data_t), offsetof(ztest_cb_data_t, zcd_node)); @@ -5907,10 +5908,10 @@ ztest_run(ztest_shared_t *zs) list_destroy(&zcl.zcl_callbacks); - (void) _mutex_destroy(&zcl.zcl_callbacks_lock); + mutex_destroy(&zcl.zcl_callbacks_lock); - (void) rwlock_destroy(&ztest_name_lock); - (void) _mutex_destroy(&ztest_vdev_lock); + rw_destroy(&ztest_name_lock); + mutex_destroy(&ztest_vdev_lock); } static void @@ -6054,8 +6055,8 @@ ztest_init(ztest_shared_t *zs) spa_t *spa; nvlist_t *nvroot, *props; - VERIFY(_mutex_init(&ztest_vdev_lock, USYNC_THREAD, NULL) == 0); - VERIFY(rwlock_init(&ztest_name_lock, USYNC_THREAD, NULL) == 0); + mutex_init(&ztest_vdev_lock, NULL, USYNC_THREAD, NULL); + rw_init(&ztest_name_lock, NULL, USYNC_THREAD, NULL); kernel_init(FREAD | FWRITE); @@ -6092,8 +6093,8 @@ ztest_init(ztest_shared_t *zs) ztest_run_zdb(ztest_opts.zo_pool); - (void) rwlock_destroy(&ztest_name_lock); - (void) _mutex_destroy(&ztest_vdev_lock); + rw_destroy(&ztest_name_lock); + mutex_destroy(&ztest_vdev_lock); } static void Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Feb 21 20:56:28 2018 (r329752) +++ vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Feb 21 21:04:46 2018 (r329753) @@ -22,6 +22,7 @@ * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2015 by Delphix. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright 2017 RackTop Systems. */ #include @@ -41,12 +42,14 @@ #include #include +extern void system_taskq_init(void); +extern void system_taskq_fini(void); + /* * Emulation of kernel services in userland. */ -int aok; -uint64_t physmem; +pgcnt_t physmem; vnode_t *rootdir = (vnode_t *)0xabcd1234; char hw_serial[HW_HOSTID_LEN]; kmutex_t cpu_lock; @@ -59,338 +62,8 @@ struct utsname utsname = { "userland", "libzpool", "1", "1", "na" }; -/* this only exists to have its address taken */ -struct proc p0; - /* * ========================================================================= - * threads - * ========================================================================= - */ -/*ARGSUSED*/ -kthread_t * -zk_thread_create(void (*func)(), void *arg) -{ - thread_t tid; - - VERIFY(thr_create(0, 0, (void *(*)(void *))func, arg, THR_DETACHED, - &tid) == 0); - - return ((void *)(uintptr_t)tid); -} - -/* - * ========================================================================= - * kstats - * ========================================================================= - */ -/*ARGSUSED*/ -kstat_t * -kstat_create(const char *module, int instance, const char *name, - const char *class, uchar_t type, ulong_t ndata, uchar_t ks_flag) -{ - return (NULL); -} - -/*ARGSUSED*/ -void -kstat_named_init(kstat_named_t *knp, const char *name, uchar_t type) -{} - -/*ARGSUSED*/ -void -kstat_install(kstat_t *ksp) -{} - -/*ARGSUSED*/ -void -kstat_delete(kstat_t *ksp) -{} - -/*ARGSUSED*/ -void -kstat_waitq_enter(kstat_io_t *kiop) -{} - -/*ARGSUSED*/ -void -kstat_waitq_exit(kstat_io_t *kiop) -{} - -/*ARGSUSED*/ -void -kstat_runq_enter(kstat_io_t *kiop) -{} - -/*ARGSUSED*/ -void -kstat_runq_exit(kstat_io_t *kiop) -{} *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Feb 21 22:14:20 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95878F0599F; Wed, 21 Feb 2018 22:14:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44C9E76201; Wed, 21 Feb 2018 22:14:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3B9901E72E; Wed, 21 Feb 2018 22:14:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LMEKRf041702; Wed, 21 Feb 2018 22:14:20 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LMEK5u041701; Wed, 21 Feb 2018 22:14:20 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802212214.w1LMEK5u041701@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 22:14:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329757 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329757 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 22:14:20 -0000 Author: mav Date: Wed Feb 21 22:14:19 2018 New Revision: 329757 URL: https://svnweb.freebsd.org/changeset/base/329757 Log: 9018 Replace kmem_cache_reap_now() with kmem_cache_reap_soon() illumos/illumos-gate@36a64e62848b51ac5a9a5216e894ec723cfef14e To prevent kmem_cache reaping from blocking other system resources, turn kmem_cache_reap_now() (which blocks) into kmem_cache_reap_soon(). Callers to kmem_cache_reap_soon() should use kmem_cache_reap_active(), which exploits #9017's new taskq_empty(). Reviewed by: Bryan Cantrill Reviewed by: Dan McDonald Reviewed by: Matthew Ahrens Reviewed by: Yuri Pankov Author: Tim Kordas Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Wed Feb 21 21:39:47 2018 (r329756) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Wed Feb 21 22:14:19 2018 (r329757) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, Joyent, Inc. All rights reserved. + * Copyright (c) 2018, Joyent, Inc. * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. * Copyright 2017 Nexenta Systems, Inc. All rights reserved. @@ -301,6 +301,9 @@ int zfs_arc_evict_batch_limit = 10; /* number of seconds before growing cache again */ static int arc_grow_retry = 60; +/* number of milliseconds before attempting a kmem-cache-reap */ +static int arc_kmem_cache_reap_retry_ms = 1000; + /* shift of arc_c for calculating overflow limit in arc_get_data_impl */ int zfs_arc_overflow_shift = 8; @@ -4047,21 +4050,31 @@ arc_kmem_reap_now(void) #endif #endif + /* + * If a kmem reap is already active, don't schedule more. We must + * check for this because kmem_cache_reap_soon() won't actually + * block on the cache being reaped (this is to prevent callers from + * becoming implicitly blocked by a system-wide kmem reap -- which, + * on a system with many, many full magazines, can take minutes). + */ + if (kmem_cache_reap_active()) + return; + for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { if (zio_buf_cache[i] != prev_cache) { prev_cache = zio_buf_cache[i]; - kmem_cache_reap_now(zio_buf_cache[i]); + kmem_cache_reap_soon(zio_buf_cache[i]); } if (zio_data_buf_cache[i] != prev_data_cache) { prev_data_cache = zio_data_buf_cache[i]; - kmem_cache_reap_now(zio_data_buf_cache[i]); + kmem_cache_reap_soon(zio_data_buf_cache[i]); } } - kmem_cache_reap_now(abd_chunk_cache); - kmem_cache_reap_now(buf_cache); - kmem_cache_reap_now(hdr_full_cache); - kmem_cache_reap_now(hdr_l2only_cache); - kmem_cache_reap_now(range_seg_cache); + kmem_cache_reap_soon(abd_chunk_cache); + kmem_cache_reap_soon(buf_cache); + kmem_cache_reap_soon(hdr_full_cache); + kmem_cache_reap_soon(hdr_l2only_cache); + kmem_cache_reap_soon(range_seg_cache); if (zio_arena != NULL) { /* @@ -4093,6 +4106,7 @@ static void arc_reclaim_thread(void *unused) { hrtime_t growtime = 0; + hrtime_t kmem_reap_time = 0; callb_cpr_t cpr; CALLB_CPR_INIT(&cpr, &arc_reclaim_lock, callb_generic_cpr, FTAG); @@ -4126,7 +4140,7 @@ arc_reclaim_thread(void *unused) int64_t free_memory = arc_available_memory(); if (free_memory < 0) { - + hrtime_t curtime = gethrtime(); arc_no_grow = B_TRUE; arc_warm = B_TRUE; @@ -4134,9 +4148,20 @@ arc_reclaim_thread(void *unused) * Wait at least zfs_grow_retry (default 60) seconds * before considering growing. */ - growtime = gethrtime() + SEC2NSEC(arc_grow_retry); + growtime = curtime + SEC2NSEC(arc_grow_retry); - arc_kmem_reap_now(); + /* + * Wait at least arc_kmem_cache_reap_retry_ms + * between arc_kmem_reap_now() calls. Without + * this check it is possible to end up in a + * situation where we spend lots of time + * reaping caches, while we're near arc_c_min. + */ + if (curtime >= kmem_reap_time) { + arc_kmem_reap_now(); + kmem_reap_time = gethrtime() + + MSEC2NSEC(arc_kmem_cache_reap_retry_ms); + } /* * If we are still low on memory, shrink the ARC From owner-svn-src-vendor@freebsd.org Wed Feb 21 23:25:12 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94978F0A9FE; Wed, 21 Feb 2018 23:25:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4764D78B28; Wed, 21 Feb 2018 23:25:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3E1E71F26F; Wed, 21 Feb 2018 23:25:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LNPCNs076083; Wed, 21 Feb 2018 23:25:12 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LNPC8f076082; Wed, 21 Feb 2018 23:25:12 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802212325.w1LNPC8f076082@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 23:25:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329760 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329760 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 23:25:12 -0000 Author: mav Date: Wed Feb 21 23:25:11 2018 New Revision: 329760 URL: https://svnweb.freebsd.org/changeset/base/329760 Log: 7638 Refactor spa_load_impl into several functions illumos/illumos-gate@1fd3785ff6601d3e391378c2dcbf4c5f27e1fe32 spa_load_impl has grown out of proportions. It is currently over 700 lines long and makes it very hard to follow or debug the import process even for experienced ZFS developers. The objective is to split it up in a series of well commented functions. Reviewed by: Paul Dagnelie Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andrew Stormont Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 23:15:06 2018 (r329759) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 23:25:11 2018 (r329760) @@ -2144,7 +2144,7 @@ spa_try_repair(spa_t *spa, nvlist_t *config) static int spa_load(spa_t *spa, spa_load_state_t state, spa_import_type_t type, - boolean_t mosconfig) + boolean_t trust_config) { nvlist_t *config = spa->spa_config; char *ereport = FM_EREPORT_ZFS_POOL; @@ -2188,7 +2188,7 @@ spa_load(spa_t *spa, spa_load_state_t state, spa_impor gethrestime(&spa->spa_loaded_ts); error = spa_load_impl(spa, pool_guid, config, state, type, - mosconfig, &ereport); + trust_config, &ereport); } /* @@ -2240,38 +2240,20 @@ vdev_count_verify_zaps(vdev_t *vd) return (total); } -/* - * Load an existing storage pool, using the pool's builtin spa_config as a - * source of configuration information. - */ static int -spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t *config, - spa_load_state_t state, spa_import_type_t type, boolean_t trust_config, - char **ereport) +spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nvlist_t *config, + spa_load_state_t state, spa_import_type_t type) { int error = 0; - nvlist_t *nvroot = NULL; - nvlist_t *label; - vdev_t *rvd; - uberblock_t *ub = &spa->spa_uberblock; - uint64_t children, config_cache_txg = spa->spa_config_txg; - int orig_mode = spa->spa_mode; + nvlist_t *nvtree = NULL; int parse; - uint64_t obj; - boolean_t missing_feat_write = B_FALSE; + vdev_t *rvd; - /* - * If this is an untrusted config, access the pool in read-only mode. - * This prevents things like resilvering recently removed devices. - */ - if (!trust_config) - spa->spa_mode = FREAD; - ASSERT(MUTEX_HELD(&spa_namespace_lock)); spa->spa_load_state = state; - if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvroot)) + if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvtree)) return (SET_ERROR(EINVAL)); parse = (type == SPA_IMPORT_EXISTING ? @@ -2294,7 +2276,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * configuration requires knowing the version number. */ spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - error = spa_config_parse(spa, &rvd, nvroot, NULL, 0, parse); + error = spa_config_parse(spa, &rvd, nvtree, NULL, 0, parse); spa_config_exit(spa, SCL_ALL, FTAG); if (error != 0) @@ -2308,22 +2290,35 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t ASSERT(spa_guid(spa) == pool_guid); } - /* - * Try to open all vdevs, loading each label in the process. - */ + return (0); +} + +static int +spa_ld_open_vdevs(spa_t *spa) +{ + int error = 0; + spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - error = vdev_open(rvd); + error = vdev_open(spa->spa_root_vdev); spa_config_exit(spa, SCL_ALL, FTAG); - if (error != 0) - return (error); + return (error); +} + +static int +spa_ld_validate_vdevs(spa_t *spa, spa_import_type_t type, + boolean_t trust_config) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + /* * We need to validate the vdev labels against the configuration that - * we have in hand, which is dependent on the setting of mosconfig. If - * mosconfig is true then we're validating the vdev labels based on - * that config. Otherwise, we're validating against the cached config - * (zpool.cache) that was read when we loaded the zfs module, and then - * later we will recursively call spa_load() and validate against + * we have in hand, which is dependent on the setting of trust_config. + * If trust_config is true then we're validating the vdev labels based + * on that config. Otherwise, we're validating against the cached + * config (zpool.cache) that was read when we loaded the zfs module, and + * then later we will recursively call spa_load() and validate against * the vdev config. * * If we're assembling a new pool that's been split off from an @@ -2342,6 +2337,18 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t return (SET_ERROR(ENXIO)); } + return (0); +} + +static int +spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, spa_import_type_t type, + boolean_t trust_config) +{ + vdev_t *rvd = spa->spa_root_vdev; + nvlist_t *label; + uberblock_t *ub = &spa->spa_uberblock; + uint64_t children; + /* * Find the best uberblock. */ @@ -2450,18 +2457,28 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t spa->spa_claim_max_txg = spa->spa_first_txg; spa->spa_prev_software_version = ub->ub_software_version; - /* - * Everything that we read before we do spa_remove_init() must - * have been rewritten after the last device removal was initiated. - * Otherwise we could be reading from indirect vdevs before - * we have loaded their mappings. - */ + return (0); +} +static int +spa_ld_open_rootbp(spa_t *spa) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + error = dsl_pool_init(spa, spa->spa_first_txg, &spa->spa_dsl_pool); if (error) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); spa->spa_meta_objset = spa->spa_dsl_pool->dp_meta_objset; + return (0); +} + +static int +spa_ld_validate_config(spa_t *spa, spa_import_type_t type) +{ + vdev_t *rvd = spa->spa_root_vdev; + if (spa_dir_prop(spa, DMU_POOL_CONFIG, &spa->spa_config_object) != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); @@ -2493,6 +2510,15 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t return (SET_ERROR(ENXIO)); } + return (0); +} + +static int +spa_ld_open_indirect_vdev_metadata(spa_t *spa) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + /* * Everything that we read before spa_remove_init() must be stored * on concreted vdevs. Therefore we do this as early as possible. @@ -2500,6 +2526,24 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t if (spa_remove_init(spa) != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + /* + * Retrieve information needed to condense indirect vdev mappings. + */ + error = spa_condense_init(spa); + if (error != 0) { + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); + } + + return (0); +} + +static int +spa_ld_check_features(spa_t *spa, spa_load_state_t state, + boolean_t *missing_feat_writep) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + if (spa_version(spa) >= SPA_VERSION_FEATURES) { boolean_t missing_feat_read = B_FALSE; nvlist_t *unsup_feat, *enabled_feat; @@ -2529,7 +2573,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t if (spa_writeable(spa) || state == SPA_LOAD_TRYIMPORT) { if (!spa_features_check(spa, B_TRUE, unsup_feat, enabled_feat)) { - missing_feat_write = B_TRUE; + *missing_feat_writep = B_TRUE; } } @@ -2568,7 +2612,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * userland in order to know whether to display the * abovementioned note. */ - if (missing_feat_read || (missing_feat_write && + if (missing_feat_read || (*missing_feat_writep && spa_writeable(spa))) { return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT, ENOTSUP)); @@ -2601,62 +2645,85 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } + return (0); +} + +static int +spa_ld_load_special_directories(spa_t *spa) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + spa->spa_is_initializing = B_TRUE; error = dsl_pool_open(spa->spa_dsl_pool); spa->spa_is_initializing = B_FALSE; if (error != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); - if (!trust_config) { - uint64_t hostid; - nvlist_t *policy = NULL; - nvlist_t *mos_config; + return (0); +} - if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); +static int +spa_ld_prepare_for_reload(spa_t *spa, int orig_mode) +{ + vdev_t *rvd = spa->spa_root_vdev; - if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config, - ZPOOL_CONFIG_HOSTID, &hostid) == 0) { - char *hostname; - unsigned long myhostid = 0; + uint64_t hostid; + nvlist_t *policy = NULL; + nvlist_t *mos_config; - VERIFY(nvlist_lookup_string(mos_config, - ZPOOL_CONFIG_HOSTNAME, &hostname) == 0); + if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config, + ZPOOL_CONFIG_HOSTID, &hostid) == 0) { + char *hostname; + unsigned long myhostid = 0; + + VERIFY(nvlist_lookup_string(mos_config, + ZPOOL_CONFIG_HOSTNAME, &hostname) == 0); + #ifdef _KERNEL - myhostid = zone_get_hostid(NULL); + myhostid = zone_get_hostid(NULL); #else /* _KERNEL */ - /* - * We're emulating the system's hostid in userland, so - * we can't use zone_get_hostid(). - */ - (void) ddi_strtoul(hw_serial, NULL, 10, &myhostid); + /* + * We're emulating the system's hostid in userland, so + * we can't use zone_get_hostid(). + */ + (void) ddi_strtoul(hw_serial, NULL, 10, &myhostid); #endif /* _KERNEL */ - if (hostid != 0 && myhostid != 0 && - hostid != myhostid) { - nvlist_free(mos_config); - cmn_err(CE_WARN, "pool '%s' could not be " - "loaded as it was last accessed by " - "another system (host: %s hostid: 0x%lx). " - "See: http://illumos.org/msg/ZFS-8000-EY", - spa_name(spa), hostname, - (unsigned long)hostid); - return (SET_ERROR(EBADF)); - } + if (hostid != 0 && myhostid != 0 && + hostid != myhostid) { + nvlist_free(mos_config); + cmn_err(CE_WARN, "pool '%s' could not be " + "loaded as it was last accessed by " + "another system (host: %s hostid: 0x%lx). " + "See: http://illumos.org/msg/ZFS-8000-EY", + spa_name(spa), hostname, + (unsigned long)hostid); + return (SET_ERROR(EBADF)); } - if (nvlist_lookup_nvlist(spa->spa_config, - ZPOOL_REWIND_POLICY, &policy) == 0) - VERIFY(nvlist_add_nvlist(mos_config, - ZPOOL_REWIND_POLICY, policy) == 0); + } + if (nvlist_lookup_nvlist(spa->spa_config, + ZPOOL_REWIND_POLICY, &policy) == 0) + VERIFY(nvlist_add_nvlist(mos_config, + ZPOOL_REWIND_POLICY, policy) == 0); - spa_config_set(spa, mos_config); - spa_unload(spa); - spa_deactivate(spa); - spa_activate(spa, orig_mode); + spa_config_set(spa, mos_config); + spa_unload(spa); + spa_deactivate(spa); + spa_activate(spa, orig_mode); - return (spa_load(spa, state, SPA_IMPORT_EXISTING, B_TRUE)); - } + return (0); +} +static int +spa_ld_get_props(spa_t *spa) +{ + int error = 0; + uint64_t obj; + vdev_t *rvd = spa->spa_root_vdev; + /* Grab the secret checksum salt from the MOS. */ error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_CHECKSUM_SALT, 1, @@ -2748,6 +2815,35 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t } nvlist_free(mos_config); + spa->spa_delegation = zpool_prop_default_numeric(ZPOOL_PROP_DELEGATION); + + error = spa_dir_prop(spa, DMU_POOL_PROPS, &spa->spa_pool_props_object); + if (error && error != ENOENT) + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + + if (error == 0) { + uint64_t autoreplace; + + spa_prop_find(spa, ZPOOL_PROP_BOOTFS, &spa->spa_bootfs); + spa_prop_find(spa, ZPOOL_PROP_AUTOREPLACE, &autoreplace); + spa_prop_find(spa, ZPOOL_PROP_DELEGATION, &spa->spa_delegation); + spa_prop_find(spa, ZPOOL_PROP_FAILUREMODE, &spa->spa_failmode); + spa_prop_find(spa, ZPOOL_PROP_AUTOEXPAND, &spa->spa_autoexpand); + spa_prop_find(spa, ZPOOL_PROP_DEDUPDITTO, + &spa->spa_dedup_ditto); + + spa->spa_autoreplace = (autoreplace != 0); + } + + return (0); +} + +static int +spa_ld_open_aux_vdevs(spa_t *spa, spa_import_type_t type) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + /* * If we're assembling the pool from the split-off vdevs of * an existing pool, we don't want to attach the spares & cache @@ -2793,27 +2889,15 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t spa->spa_l2cache.sav_sync = B_TRUE; } - spa->spa_delegation = zpool_prop_default_numeric(ZPOOL_PROP_DELEGATION); + return (0); +} - error = spa_dir_prop(spa, DMU_POOL_PROPS, &spa->spa_pool_props_object); - if (error && error != ENOENT) - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); +static int +spa_ld_load_vdev_metadata(spa_t *spa, spa_load_state_t state) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; - if (error == 0) { - uint64_t autoreplace; - - spa_prop_find(spa, ZPOOL_PROP_BOOTFS, &spa->spa_bootfs); - spa_prop_find(spa, ZPOOL_PROP_AUTOREPLACE, &autoreplace); - spa_prop_find(spa, ZPOOL_PROP_DELEGATION, &spa->spa_delegation); - spa_prop_find(spa, ZPOOL_PROP_FAILUREMODE, &spa->spa_failmode); - spa_prop_find(spa, ZPOOL_PROP_AUTOEXPAND, &spa->spa_autoexpand); - spa_prop_find(spa, ZPOOL_PROP_BOOTSIZE, &spa->spa_bootsize); - spa_prop_find(spa, ZPOOL_PROP_DEDUPDITTO, - &spa->spa_dedup_ditto); - - spa->spa_autoreplace = (autoreplace != 0); - } - /* * If the 'autoreplace' property is set, then post a resource notifying * the ZFS DE that it should not issue any faults for unopenable @@ -2835,40 +2919,281 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t } /* - * Load the vdev state for all toplevel vdevs. + * Load the vdev metadata such as metaslabs, DTLs, spacemap object, etc. */ error = vdev_load(rvd); if (error != 0) { return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); } - error = spa_condense_init(spa); - if (error != 0) { - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); - } - /* - * Propagate the leaf DTLs we just loaded all the way up the tree. + * Propagate the leaf DTLs we just loaded all the way up the vdev tree. */ spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); vdev_dtl_reassess(rvd, 0, 0, B_FALSE); spa_config_exit(spa, SCL_ALL, FTAG); - /* - * Load the DDTs (dedup tables). - */ + return (0); +} + +static int +spa_ld_load_dedup_tables(spa_t *spa) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + error = ddt_load(spa); if (error != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); - spa_update_dspace(spa); + return (0); +} +static int +spa_ld_verify_logs(spa_t *spa, spa_import_type_t type, char **ereport) +{ + vdev_t *rvd = spa->spa_root_vdev; + if (type != SPA_IMPORT_ASSEMBLE && spa_writeable(spa) && spa_check_logs(spa)) { *ereport = FM_EREPORT_ZFS_LOG_REPLAY; return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, ENXIO)); } + return (0); +} + +static int +spa_ld_verify_pool_data(spa_t *spa, spa_load_state_t state) +{ + int error = 0; + vdev_t *rvd = spa->spa_root_vdev; + + /* + * We've successfully opened the pool, verify that we're ready + * to start pushing transactions. + */ + if (state != SPA_LOAD_TRYIMPORT) { + error = spa_load_verify(spa); + if (error != 0) { + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, + error)); + } + } + + return (0); +} + +static void +spa_ld_claim_log_blocks(spa_t *spa) +{ + dmu_tx_t *tx; + dsl_pool_t *dp = spa_get_dsl(spa); + + /* + * Claim log blocks that haven't been committed yet. + * This must all happen in a single txg. + * Note: spa_claim_max_txg is updated by spa_claim_notify(), + * invoked from zil_claim_log_block()'s i/o done callback. + * Price of rollback is that we abandon the log. + */ + spa->spa_claiming = B_TRUE; + + tx = dmu_tx_create_assigned(dp, spa_first_txg(spa)); + (void) dmu_objset_find_dp(dp, dp->dp_root_dir_obj, + zil_claim, tx, DS_FIND_CHILDREN); + dmu_tx_commit(tx); + + spa->spa_claiming = B_FALSE; + + spa_set_log_state(spa, SPA_LOG_GOOD); +} + +static void +spa_ld_check_for_config_update(spa_t *spa, spa_load_state_t state, + int64_t config_cache_txg) +{ + vdev_t *rvd = spa->spa_root_vdev; + int need_update = B_FALSE; + + /* + * If the config cache is stale, or we have uninitialized + * metaslabs (see spa_vdev_add()), then update the config. + * + * If this is a verbatim import, trust the current + * in-core spa_config and update the disk labels. + */ + if (config_cache_txg != spa->spa_config_txg || + state == SPA_LOAD_IMPORT || + state == SPA_LOAD_RECOVER || + (spa->spa_import_flags & ZFS_IMPORT_VERBATIM)) + need_update = B_TRUE; + + for (int c = 0; c < rvd->vdev_children; c++) + if (rvd->vdev_child[c]->vdev_ms_array == 0) + need_update = B_TRUE; + + /* + * Update the config cache asychronously in case we're the + * root pool, in which case the config cache isn't writable yet. + */ + if (need_update) + spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE); +} + +/* + * Load an existing storage pool, using the config provided. This config + * describes which vdevs are part of the pool and is later validated against + * partial configs present in each vdev's label and an entire copy of the + * config stored in the MOS. + */ +static int +spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t *config, + spa_load_state_t state, spa_import_type_t type, boolean_t trust_config, + char **ereport) +{ + int error = 0; + uint64_t config_cache_txg = spa->spa_config_txg; + int orig_mode = spa->spa_mode; + boolean_t missing_feat_write = B_FALSE; + + /* + * If this is an untrusted config, first access the pool in read-only + * mode. We will then retrieve a trusted copy of the config from the MOS + * and use it to reopen the pool in read-write mode. + */ + if (!trust_config) + spa->spa_mode = FREAD; + + /* + * Parse the config provided to create a vdev tree. + */ + error = spa_ld_parse_config(spa, pool_guid, config, state, type); + if (error != 0) + return (error); + + /* + * Now that we have the vdev tree, try to open each vdev. This involves + * opening the underlying physical device, retrieving its geometry and + * probing the vdev with a dummy I/O. The state of each vdev will be set + * based on the success of those operations. After this we'll be ready + * to read from the vdevs. + */ + error = spa_ld_open_vdevs(spa); + if (error != 0) + return (error); + + /* + * Read the label of each vdev and make sure that the GUIDs stored + * there match the GUIDs in the config provided. + */ + error = spa_ld_validate_vdevs(spa, type, trust_config); + if (error != 0) + return (error); + + /* + * Read vdev labels to find the best uberblock (i.e. latest, unless + * spa_load_max_txg is set) and store it in spa_uberblock. We get the + * list of features required to read blkptrs in the MOS from the vdev + * label with the best uberblock and verify that our version of zfs + * supports them all. + */ + error = spa_ld_select_uberblock(spa, config, type, trust_config); + if (error != 0) + return (error); + + /* + * Pass that uberblock to the dsl_pool layer which will open the root + * blkptr. This blkptr points to the latest version of the MOS and will + * allow us to read its contents. + */ + error = spa_ld_open_rootbp(spa); + if (error != 0) + return (error); + + /* + * Retrieve the config stored in the MOS and use it to validate the + * config provided. Also extract some information from the MOS config + * to update our vdev tree. + */ + error = spa_ld_validate_config(spa, type); + if (error != 0) + return (error); + + /* + * Retrieve the mapping of indirect vdevs. Those vdevs were removed + * from the pool and their contents were re-mapped to other vdevs. Note + * that everything that we read before this step must have been + * rewritten on concrete vdevs after the last device removal was + * initiated. Otherwise we could be reading from indirect vdevs before + * we have loaded their mappings. + */ + error = spa_ld_open_indirect_vdev_metadata(spa); + if (error != 0) + return (error); + + /* + * Retrieve the full list of active features from the MOS and check if + * they are all supported. + */ + error = spa_ld_check_features(spa, state, &missing_feat_write); + if (error != 0) + return (error); + + /* + * Load several special directories from the MOS needed by the dsl_pool + * layer. + */ + error = spa_ld_load_special_directories(spa); + if (error != 0) + return (error); + + /* + * If the config provided is not trusted, discard it and use the config + * from the MOS to reload the pool. + */ + if (!trust_config) { + error = spa_ld_prepare_for_reload(spa, orig_mode); + if (error != 0) + return (error); + return (spa_load(spa, state, SPA_IMPORT_EXISTING, B_TRUE)); + } + + /* + * Retrieve pool properties from the MOS. + */ + error = spa_ld_get_props(spa); + if (error != 0) + return (error); + + /* + * Retrieve the list of auxiliary devices - cache devices and spares - + * and open them. + */ + error = spa_ld_open_aux_vdevs(spa, type); + if (error != 0) + return (error); + + /* + * Load the metadata for all vdevs. Also check if unopenable devices + * should be autoreplaced. + */ + error = spa_ld_load_vdev_metadata(spa, state); + if (error != 0) + return (error); + + error = spa_ld_load_dedup_tables(spa); + if (error != 0) + return (error); + + /* + * Verify the logs now to make sure we don't have any unexpected errors + * when we claim log blocks later. + */ + error = spa_ld_verify_logs(spa, type, ereport); + if (error != 0) + return (error); + if (missing_feat_write) { ASSERT(state == SPA_LOAD_TRYIMPORT); @@ -2877,24 +3202,34 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * read-only mode but not read-write mode. We now have enough * information and can return to userland. */ - return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT, ENOTSUP)); + return (spa_vdev_err(spa->spa_root_vdev, VDEV_AUX_UNSUP_FEAT, + ENOTSUP)); } /* - * We've successfully opened the pool, verify that we're ready - * to start pushing transactions. + * Traverse the last txgs to make sure the pool was left off in a safe + * state. When performing an extreme rewind, we verify the whole pool, + * which can take a very long time. */ - if (state != SPA_LOAD_TRYIMPORT) { - if (error = spa_load_verify(spa)) - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, - error)); - } + error = spa_ld_verify_pool_data(spa, state); + if (error != 0) + return (error); + /* + * Calculate the deflated space for the pool. This must be done before + * we write anything to the pool because we'd need to update the space + * accounting using the deflated sizes. + */ + spa_update_dspace(spa); + + /* + * We have now retrieved all the information we needed to open the + * pool. If we are importing the pool in read-write mode, a few + * additional steps must be performed to finish the import. + */ if (spa_writeable(spa) && (state == SPA_LOAD_RECOVER || spa->spa_load_max_txg == UINT64_MAX)) { - dmu_tx_t *tx; - int need_update = B_FALSE; - dsl_pool_t *dp = spa_get_dsl(spa); + ASSERT(state != SPA_LOAD_TRYIMPORT); /* * We must check this before we start the sync thread, because @@ -2908,25 +3243,14 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t boolean_t condense_in_progress = (spa->spa_condensing_indirect != NULL); - ASSERT(state != SPA_LOAD_TRYIMPORT); - /* - * Claim log blocks that haven't been committed yet. - * This must all happen in a single txg. - * Note: spa_claim_max_txg is updated by spa_claim_notify(), - * invoked from zil_claim_log_block()'s i/o done callback. - * Price of rollback is that we abandon the log. + * Traverse the ZIL and claim all blocks. */ - spa->spa_claiming = B_TRUE; + spa_ld_claim_log_blocks(spa); - tx = dmu_tx_create_assigned(dp, spa_first_txg(spa)); - (void) dmu_objset_find_dp(dp, dp->dp_root_dir_obj, - zil_claim, tx, DS_FIND_CHILDREN); - dmu_tx_commit(tx); - - spa->spa_claiming = B_FALSE; - - spa_set_log_state(spa, SPA_LOG_GOOD); + /* + * Kick-off the syncing thread. + */ spa->spa_sync_on = B_TRUE; txg_sync_start(spa->spa_dsl_pool); @@ -2934,40 +3258,23 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * Wait for all claims to sync. We sync up to the highest * claimed log block birth time so that claimed log blocks * don't appear to be from the future. spa_claim_max_txg - * will have been set for us by either zil_check_log_chain() - * (invoked from spa_check_logs()) or zil_claim() above. + * will have been set for us by ZIL traversal operations + * performed above. */ txg_wait_synced(spa->spa_dsl_pool, spa->spa_claim_max_txg); /* - * If the config cache is stale, or we have uninitialized - * metaslabs (see spa_vdev_add()), then update the config. - * - * If this is a verbatim import, trust the current - * in-core spa_config and update the disk labels. + * Check if we need to request an update of the config. On the + * next sync, we would update the config stored in vdev labels + * and the cachefile (by default /etc/zfs/zpool.cache). */ - if (config_cache_txg != spa->spa_config_txg || - state == SPA_LOAD_IMPORT || - state == SPA_LOAD_RECOVER || - (spa->spa_import_flags & ZFS_IMPORT_VERBATIM)) - need_update = B_TRUE; + spa_ld_check_for_config_update(spa, state, config_cache_txg); - for (int c = 0; c < rvd->vdev_children; c++) - if (rvd->vdev_child[c]->vdev_ms_array == 0) - need_update = B_TRUE; - /* - * Update the config cache asychronously in case we're the - * root pool, in which case the config cache isn't writable yet. - */ - if (need_update) - spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE); - - /* * Check all DTLs to see if anything needs resilvering. */ if (!dsl_scan_resilvering(spa->spa_dsl_pool) && - vdev_resilver_needed(rvd, NULL, NULL)) + vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL)) spa_async_request(spa, SPA_ASYNC_RESILVER); /* @@ -3002,7 +3309,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t } static int -spa_load_retry(spa_t *spa, spa_load_state_t state, int mosconfig) +spa_load_retry(spa_t *spa, spa_load_state_t state, int trust_config) { int mode = spa->spa_mode; @@ -3014,7 +3321,7 @@ spa_load_retry(spa_t *spa, spa_load_state_t state, int spa_activate(spa, mode); spa_async_suspend(spa); - return (spa_load(spa, state, SPA_IMPORT_EXISTING, mosconfig)); + return (spa_load(spa, state, SPA_IMPORT_EXISTING, trust_config)); } /* @@ -3025,7 +3332,7 @@ spa_load_retry(spa_t *spa, spa_load_state_t state, int * spa_load(). */ static int -spa_load_best(spa_t *spa, spa_load_state_t state, int mosconfig, +spa_load_best(spa_t *spa, spa_load_state_t state, int trust_config, uint64_t max_request, int rewind_flags) { nvlist_t *loadinfo = NULL; @@ -3044,7 +3351,7 @@ spa_load_best(spa_t *spa, spa_load_state_t state, int } load_error = rewind_error = spa_load(spa, state, SPA_IMPORT_EXISTING, - mosconfig); + trust_config); if (load_error == 0) return (0); @@ -3085,7 +3392,7 @@ spa_load_best(spa_t *spa, spa_load_state_t state, int spa->spa_uberblock.ub_txg <= spa->spa_load_max_txg) { if (spa->spa_load_max_txg < safe_rewind_txg) spa->spa_extreme_rewind = B_TRUE; - rewind_error = spa_load_retry(spa, state, mosconfig); + rewind_error = spa_load_retry(spa, state, trust_config); } spa->spa_extreme_rewind = B_FALSE; @@ -4187,7 +4494,7 @@ spa_import(const char *pool, nvlist_t *config, nvlist_ state = SPA_LOAD_RECOVER; /* - * Pass off the heavy lifting to spa_load(). Pass TRUE for mosconfig + * Pass off the heavy lifting to spa_load(). Pass TRUE for trust_config * because the user-supplied config is actually the one to trust when * doing an import. */ @@ -4333,7 +4640,7 @@ spa_tryimport(nvlist_t *tryconfig) /* * Pass off the heavy lifting to spa_load(). - * Pass TRUE for mosconfig because the user-supplied config + * Pass TRUE for trust_config because the user-supplied config * is actually the one to trust when doing an import. */ error = spa_load(spa, SPA_LOAD_TRYIMPORT, SPA_IMPORT_EXISTING, B_TRUE); From owner-svn-src-vendor@freebsd.org Wed Feb 21 23:42:32 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E30FF0BD12; Wed, 21 Feb 2018 23:42:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B52BA7966A; Wed, 21 Feb 2018 23:42:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AF7161F5A3; Wed, 21 Feb 2018 23:42:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LNgV6v086185; Wed, 21 Feb 2018 23:42:31 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LNgUUW086175; Wed, 21 Feb 2018 23:42:30 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802212342.w1LNgUUW086175@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 21 Feb 2018 23:42:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329762 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 329762 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 23:42:32 -0000 Author: mav Date: Wed Feb 21 23:42:30 2018 New Revision: 329762 URL: https://svnweb.freebsd.org/changeset/base/329762 Log: 8961 SPA load/import should tell us why it failed illumos/illumos-gate@3ee8c80c747c4aa3f83351a6920f30c411236e1b When we fail to open or import a storage pool, we typically don't get any additional diagnostic information, just "no pool found" or "can not import". While there may be no additional user-consumable information, we should at least make this situation easier to debug/diagnose for developers and support. For example, we could start by using `zfs_dbgmsg()` to log each thing that we try when importing, and which things failed. E.g. "tried uberblock of txg X from label Y of device Z". Also, we could log each of the stages that we go through in `spa_load_impl()`. Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andrew Stormont Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Feb 21 23:42:30 2018 (r329762) @@ -1242,6 +1242,8 @@ spa_unload(spa_t *spa) ASSERT(MUTEX_HELD(&spa_namespace_lock)); + spa_load_note(spa, "UNLOADING"); + /* * Stop async tasks. */ @@ -1995,6 +1997,13 @@ spa_load_verify(spa_t *spa) ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE); if (spa_load_verify_metadata) { + if (spa->spa_extreme_rewind) { + spa_load_note(spa, "performing a complete scan of the " + "pool since extreme rewind is on. This may take " + "a very long time.\n (spa_load_verify_data=%u, " + "spa_load_verify_metadata=%u)", + spa_load_verify_data, spa_load_verify_metadata); + } error = traverse_pool(spa, spa->spa_verify_min_txg, TRAVERSE_PRE | TRAVERSE_PREFETCH_METADATA, spa_load_verify_cb, rio); @@ -2047,10 +2056,17 @@ spa_prop_find(spa_t *spa, zpool_prop_t prop, uint64_t * Find a value in the pool directory object. */ static int -spa_dir_prop(spa_t *spa, const char *name, uint64_t *val) +spa_dir_prop(spa_t *spa, const char *name, uint64_t *val, boolean_t log_enoent) { - return (zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT, - name, sizeof (uint64_t), 1, val)); + int error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT, + name, sizeof (uint64_t), 1, val); + + if (error != 0 && (error != ENOENT || log_enoent)) { + spa_load_failed(spa, "couldn't get '%s' value in MOS directory " + "[error=%d]", name, error); + } + + return (error); } static int @@ -2242,19 +2258,18 @@ vdev_count_verify_zaps(vdev_t *vd) static int spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nvlist_t *config, - spa_load_state_t state, spa_import_type_t type) + spa_import_type_t type) { int error = 0; nvlist_t *nvtree = NULL; int parse; vdev_t *rvd; - ASSERT(MUTEX_HELD(&spa_namespace_lock)); - - spa->spa_load_state = state; - - if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvtree)) + if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvtree)) { + spa_load_failed(spa, "invalid config provided: '%s' missing", + ZPOOL_CONFIG_VDEV_TREE); return (SET_ERROR(EINVAL)); + } parse = (type == SPA_IMPORT_EXISTING ? VDEV_ALLOC_LOAD : VDEV_ALLOC_SPLIT); @@ -2279,8 +2294,11 @@ spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nv error = spa_config_parse(spa, &rvd, nvtree, NULL, 0, parse); spa_config_exit(spa, SCL_ALL, FTAG); - if (error != 0) + if (error != 0) { + spa_load_failed(spa, "unable to parse config [error=%d]", + error); return (error); + } ASSERT(spa->spa_root_vdev == rvd); ASSERT3U(spa->spa_min_ashift, >=, SPA_MINBLOCKSHIFT); @@ -2301,6 +2319,10 @@ spa_ld_open_vdevs(spa_t *spa) spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); error = vdev_open(spa->spa_root_vdev); spa_config_exit(spa, SCL_ALL, FTAG); + if (error != 0) { + spa_load_failed(spa, "unable to open vdev tree [error=%d]", + error); + } return (error); } @@ -2330,11 +2352,17 @@ spa_ld_validate_vdevs(spa_t *spa, spa_import_type_t ty error = vdev_validate(rvd, trust_config); spa_config_exit(spa, SCL_ALL, FTAG); - if (error != 0) + if (error != 0) { + spa_load_failed(spa, "vdev_validate failed [error=%d]", + error); return (error); + } - if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) + if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) { + spa_load_failed(spa, "cannot open vdev tree after " + "invalidating some vdevs"); return (SET_ERROR(ENXIO)); + } } return (0); @@ -2359,14 +2387,20 @@ spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, */ if (ub->ub_txg == 0) { nvlist_free(label); + spa_load_failed(spa, "no valid uberblock found"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO)); } + spa_load_note(spa, "using uberblock with txg=%llu", + (u_longlong_t)ub->ub_txg); + /* * If the pool has an unsupported version we can't open it. */ if (!SPA_VERSION_IS_SUPPORTED(ub->ub_version)) { nvlist_free(label); + spa_load_failed(spa, "version %llu is not supported", + (u_longlong_t)ub->ub_version); return (spa_vdev_err(rvd, VDEV_AUX_VERSION_NEWER, ENOTSUP)); } @@ -2377,9 +2411,17 @@ spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, * If we weren't able to find what's necessary for reading the * MOS in the label, return failure. */ - if (label == NULL || nvlist_lookup_nvlist(label, - ZPOOL_CONFIG_FEATURES_FOR_READ, &features) != 0) { + if (label == NULL) { + spa_load_failed(spa, "label config unavailable"); + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, + ENXIO)); + } + + if (nvlist_lookup_nvlist(label, ZPOOL_CONFIG_FEATURES_FOR_READ, + &features) != 0) { nvlist_free(label); + spa_load_failed(spa, "invalid label: '%s' missing", + ZPOOL_CONFIG_FEATURES_FOR_READ); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO)); } @@ -2418,6 +2460,7 @@ spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, VERIFY(nvlist_add_nvlist(spa->spa_load_info, ZPOOL_CONFIG_UNSUP_FEAT, unsup_feat) == 0); nvlist_free(unsup_feat); + spa_load_failed(spa, "some features are unsupported"); return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT, ENOTSUP)); } @@ -2434,8 +2477,13 @@ spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, */ if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VDEV_CHILDREN, &children) != 0 && trust_config && type != SPA_IMPORT_ASSEMBLE && - rvd->vdev_guid_sum != ub->ub_guid_sum) + rvd->vdev_guid_sum != ub->ub_guid_sum) { + spa_load_failed(spa, "guid sum in config doesn't match guid " + "sum in uberblock (%llu != %llu)", + (u_longlong_t)rvd->vdev_guid_sum, + (u_longlong_t)ub->ub_guid_sum); return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); + } if (type != SPA_IMPORT_ASSEMBLE && spa->spa_config_splitting) { spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); @@ -2467,8 +2515,11 @@ spa_ld_open_rootbp(spa_t *spa) vdev_t *rvd = spa->spa_root_vdev; error = dsl_pool_init(spa, spa->spa_first_txg, &spa->spa_dsl_pool); - if (error) + if (error != 0) { + spa_load_failed(spa, "unable to open rootbp in dsl_pool_init " + "[error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } spa->spa_meta_objset = spa->spa_dsl_pool->dp_meta_objset; return (0); @@ -2479,7 +2530,8 @@ spa_ld_validate_config(spa_t *spa, spa_import_type_t t { vdev_t *rvd = spa->spa_root_vdev; - if (spa_dir_prop(spa, DMU_POOL_CONFIG, &spa->spa_config_object) != 0) + if (spa_dir_prop(spa, DMU_POOL_CONFIG, &spa->spa_config_object, B_TRUE) + != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); /* @@ -2491,11 +2543,16 @@ spa_ld_validate_config(spa_t *spa, spa_import_type_t t */ if (type != SPA_IMPORT_ASSEMBLE) { nvlist_t *mos_config; - if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) + if (load_nvlist(spa, spa->spa_config_object, &mos_config) + != 0) { + spa_load_failed(spa, "unable to retrieve MOS config"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } if (!spa_config_valid(spa, mos_config)) { nvlist_free(mos_config); + spa_load_failed(spa, "mismatch between config provided " + "and config stored in MOS"); return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); } @@ -2506,8 +2563,10 @@ spa_ld_validate_config(spa_t *spa, spa_import_type_t t * root vdev. If it can't be opened, it indicates one or * more toplevel vdevs are faulted. */ - if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) + if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) { + spa_load_failed(spa, "some top vdevs are unavailable"); return (SET_ERROR(ENXIO)); + } } return (0); @@ -2523,14 +2582,20 @@ spa_ld_open_indirect_vdev_metadata(spa_t *spa) * Everything that we read before spa_remove_init() must be stored * on concreted vdevs. Therefore we do this as early as possible. */ - if (spa_remove_init(spa) != 0) + error = spa_remove_init(spa); + if (error != 0) { + spa_load_failed(spa, "spa_remove_init failed [error=%d]", + error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } /* * Retrieve information needed to condense indirect vdev mappings. */ error = spa_condense_init(spa); if (error != 0) { + spa_load_failed(spa, "spa_condense_init failed [error=%d]", + error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); } @@ -2538,8 +2603,7 @@ spa_ld_open_indirect_vdev_metadata(spa_t *spa) } static int -spa_ld_check_features(spa_t *spa, spa_load_state_t state, - boolean_t *missing_feat_writep) +spa_ld_check_features(spa_t *spa, boolean_t *missing_feat_writep) { int error = 0; vdev_t *rvd = spa->spa_root_vdev; @@ -2549,17 +2613,17 @@ spa_ld_check_features(spa_t *spa, spa_load_state_t sta nvlist_t *unsup_feat, *enabled_feat; if (spa_dir_prop(spa, DMU_POOL_FEATURES_FOR_READ, - &spa->spa_feat_for_read_obj) != 0) { + &spa->spa_feat_for_read_obj, B_TRUE) != 0) { return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } if (spa_dir_prop(spa, DMU_POOL_FEATURES_FOR_WRITE, - &spa->spa_feat_for_write_obj) != 0) { + &spa->spa_feat_for_write_obj, B_TRUE) != 0) { return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } if (spa_dir_prop(spa, DMU_POOL_FEATURE_DESCRIPTIONS, - &spa->spa_feat_desc_obj) != 0) { + &spa->spa_feat_desc_obj, B_TRUE) != 0) { return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } @@ -2570,7 +2634,8 @@ spa_ld_check_features(spa_t *spa, spa_load_state_t sta unsup_feat, enabled_feat)) missing_feat_read = B_TRUE; - if (spa_writeable(spa) || state == SPA_LOAD_TRYIMPORT) { + if (spa_writeable(spa) || + spa->spa_load_state == SPA_LOAD_TRYIMPORT) { if (!spa_features_check(spa, B_TRUE, unsup_feat, enabled_feat)) { *missing_feat_writep = B_TRUE; @@ -2614,6 +2679,7 @@ spa_ld_check_features(spa_t *spa, spa_load_state_t sta */ if (missing_feat_read || (*missing_feat_writep && spa_writeable(spa))) { + spa_load_failed(spa, "pool uses unsupported features"); return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT, ENOTSUP)); } @@ -2633,6 +2699,9 @@ spa_ld_check_features(spa_t *spa, spa_load_state_t sta spa->spa_feat_refcount_cache[i] = SPA_FEATURE_DISABLED; } else { + spa_load_failed(spa, "error getting refcount " + "for feature %s [error=%d]", + spa_feature_table[i].fi_guid, error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } @@ -2641,7 +2710,7 @@ spa_ld_check_features(spa_t *spa, spa_load_state_t sta if (spa_feature_is_active(spa, SPA_FEATURE_ENABLED_TXG)) { if (spa_dir_prop(spa, DMU_POOL_FEATURE_ENABLED_TXG, - &spa->spa_feat_enabled_txg_obj) != 0) + &spa->spa_feat_enabled_txg_obj, B_TRUE) != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } @@ -2657,8 +2726,10 @@ spa_ld_load_special_directories(spa_t *spa) spa->spa_is_initializing = B_TRUE; error = dsl_pool_open(spa->spa_dsl_pool); spa->spa_is_initializing = B_FALSE; - if (error != 0) + if (error != 0) { + spa_load_failed(spa, "dsl_pool_open failed [error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } return (0); } @@ -2672,8 +2743,10 @@ spa_ld_prepare_for_reload(spa_t *spa, int orig_mode) nvlist_t *policy = NULL; nvlist_t *mos_config; - if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) + if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) { + spa_load_failed(spa, "unable to retrieve MOS config"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config, ZPOOL_CONFIG_HOSTID, &hostid) == 0) { @@ -2734,26 +2807,31 @@ spa_ld_get_props(spa_t *spa) (void) random_get_pseudo_bytes(spa->spa_cksum_salt.zcs_bytes, sizeof (spa->spa_cksum_salt.zcs_bytes)); } else if (error != 0) { + spa_load_failed(spa, "unable to retrieve checksum salt from " + "MOS [error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); } - if (spa_dir_prop(spa, DMU_POOL_SYNC_BPOBJ, &obj) != 0) + if (spa_dir_prop(spa, DMU_POOL_SYNC_BPOBJ, &obj, B_TRUE) != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); error = bpobj_open(&spa->spa_deferred_bpobj, spa->spa_meta_objset, obj); - if (error != 0) + if (error != 0) { + spa_load_failed(spa, "error opening deferred-frees bpobj " + "[error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } /* * Load the bit that tells us to use the new accounting function * (raid-z deflation). If we have an older pool, this will not * be present. */ - error = spa_dir_prop(spa, DMU_POOL_DEFLATE, &spa->spa_deflate); + error = spa_dir_prop(spa, DMU_POOL_DEFLATE, &spa->spa_deflate, B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); error = spa_dir_prop(spa, DMU_POOL_CREATION_VERSION, - &spa->spa_creation_version); + &spa->spa_creation_version, B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); @@ -2761,12 +2839,13 @@ spa_ld_get_props(spa_t *spa) * Load the persistent error log. If we have an older pool, this will * not be present. */ - error = spa_dir_prop(spa, DMU_POOL_ERRLOG_LAST, &spa->spa_errlog_last); + error = spa_dir_prop(spa, DMU_POOL_ERRLOG_LAST, &spa->spa_errlog_last, + B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); error = spa_dir_prop(spa, DMU_POOL_ERRLOG_SCRUB, - &spa->spa_errlog_scrub); + &spa->spa_errlog_scrub, B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); @@ -2774,7 +2853,7 @@ spa_ld_get_props(spa_t *spa) * Load the history object. If we have an older pool, this * will not be present. */ - error = spa_dir_prop(spa, DMU_POOL_HISTORY, &spa->spa_history); + error = spa_dir_prop(spa, DMU_POOL_HISTORY, &spa->spa_history, B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); @@ -2787,11 +2866,13 @@ spa_ld_get_props(spa_t *spa) /* The sentinel is only available in the MOS config. */ nvlist_t *mos_config; - if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) + if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) { + spa_load_failed(spa, "unable to retrieve MOS config"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } error = spa_dir_prop(spa, DMU_POOL_VDEV_ZAP_MAP, - &spa->spa_all_vdev_zaps); + &spa->spa_all_vdev_zaps, B_FALSE); if (error == ENOENT) { VERIFY(!nvlist_exists(mos_config, @@ -2817,7 +2898,8 @@ spa_ld_get_props(spa_t *spa) spa->spa_delegation = zpool_prop_default_numeric(ZPOOL_PROP_DELEGATION); - error = spa_dir_prop(spa, DMU_POOL_PROPS, &spa->spa_pool_props_object); + error = spa_dir_prop(spa, DMU_POOL_PROPS, &spa->spa_pool_props_object, + B_FALSE); if (error && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); @@ -2853,14 +2935,17 @@ spa_ld_open_aux_vdevs(spa_t *spa, spa_import_type_t ty /* * Load any hot spares for this pool. */ - error = spa_dir_prop(spa, DMU_POOL_SPARES, &spa->spa_spares.sav_object); + error = spa_dir_prop(spa, DMU_POOL_SPARES, &spa->spa_spares.sav_object, + B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); if (error == 0 && type != SPA_IMPORT_ASSEMBLE) { ASSERT(spa_version(spa) >= SPA_VERSION_SPARES); if (load_nvlist(spa, spa->spa_spares.sav_object, - &spa->spa_spares.sav_config) != 0) + &spa->spa_spares.sav_config) != 0) { + spa_load_failed(spa, "error loading spares nvlist"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); spa_load_spares(spa); @@ -2873,14 +2958,16 @@ spa_ld_open_aux_vdevs(spa_t *spa, spa_import_type_t ty * Load any level 2 ARC devices for this pool. */ error = spa_dir_prop(spa, DMU_POOL_L2CACHE, - &spa->spa_l2cache.sav_object); + &spa->spa_l2cache.sav_object, B_FALSE); if (error != 0 && error != ENOENT) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); if (error == 0 && type != SPA_IMPORT_ASSEMBLE) { ASSERT(spa_version(spa) >= SPA_VERSION_L2CACHE); if (load_nvlist(spa, spa->spa_l2cache.sav_object, - &spa->spa_l2cache.sav_config) != 0) + &spa->spa_l2cache.sav_config) != 0) { + spa_load_failed(spa, "error loading l2cache nvlist"); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); spa_load_l2cache(spa); @@ -2893,7 +2980,7 @@ spa_ld_open_aux_vdevs(spa_t *spa, spa_import_type_t ty } static int -spa_ld_load_vdev_metadata(spa_t *spa, spa_load_state_t state) +spa_ld_load_vdev_metadata(spa_t *spa) { int error = 0; vdev_t *rvd = spa->spa_root_vdev; @@ -2905,14 +2992,14 @@ spa_ld_load_vdev_metadata(spa_t *spa, spa_load_state_t * unopenable vdevs so that the normal autoreplace handler can take * over. */ - if (spa->spa_autoreplace && state != SPA_LOAD_TRYIMPORT) { + if (spa->spa_autoreplace && spa->spa_load_state != SPA_LOAD_TRYIMPORT) { spa_check_removed(spa->spa_root_vdev); /* * For the import case, this is done in spa_import(), because * at this point we're using the spare definitions from * the MOS config, not necessarily from the userland config. */ - if (state != SPA_LOAD_IMPORT) { + if (spa->spa_load_state != SPA_LOAD_IMPORT) { spa_aux_check_removed(&spa->spa_spares); spa_aux_check_removed(&spa->spa_l2cache); } @@ -2923,6 +3010,7 @@ spa_ld_load_vdev_metadata(spa_t *spa, spa_load_state_t */ error = vdev_load(rvd); if (error != 0) { + spa_load_failed(spa, "vdev_load failed [error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); } @@ -2943,8 +3031,10 @@ spa_ld_load_dedup_tables(spa_t *spa) vdev_t *rvd = spa->spa_root_vdev; error = ddt_load(spa); - if (error != 0) + if (error != 0) { + spa_load_failed(spa, "ddt_load failed [error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } return (0); } @@ -2954,17 +3044,20 @@ spa_ld_verify_logs(spa_t *spa, spa_import_type_t type, { vdev_t *rvd = spa->spa_root_vdev; - if (type != SPA_IMPORT_ASSEMBLE && spa_writeable(spa) && - spa_check_logs(spa)) { - *ereport = FM_EREPORT_ZFS_LOG_REPLAY; - return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, ENXIO)); + if (type != SPA_IMPORT_ASSEMBLE && spa_writeable(spa)) { + boolean_t missing = spa_check_logs(spa); + if (missing) { + *ereport = FM_EREPORT_ZFS_LOG_REPLAY; + spa_load_failed(spa, "spa_check_logs failed"); + return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, ENXIO)); + } } return (0); } static int -spa_ld_verify_pool_data(spa_t *spa, spa_load_state_t state) +spa_ld_verify_pool_data(spa_t *spa) { int error = 0; vdev_t *rvd = spa->spa_root_vdev; @@ -2973,9 +3066,11 @@ spa_ld_verify_pool_data(spa_t *spa, spa_load_state_t s * We've successfully opened the pool, verify that we're ready * to start pushing transactions. */ - if (state != SPA_LOAD_TRYIMPORT) { + if (spa->spa_load_state != SPA_LOAD_TRYIMPORT) { error = spa_load_verify(spa); if (error != 0) { + spa_load_failed(spa, "spa_load_verify failed " + "[error=%d]", error); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error)); } @@ -3010,8 +3105,7 @@ spa_ld_claim_log_blocks(spa_t *spa) } static void -spa_ld_check_for_config_update(spa_t *spa, spa_load_state_t state, - int64_t config_cache_txg) +spa_ld_check_for_config_update(spa_t *spa, uint64_t config_cache_txg) { vdev_t *rvd = spa->spa_root_vdev; int need_update = B_FALSE; @@ -3024,8 +3118,8 @@ spa_ld_check_for_config_update(spa_t *spa, spa_load_st * in-core spa_config and update the disk labels. */ if (config_cache_txg != spa->spa_config_txg || - state == SPA_LOAD_IMPORT || - state == SPA_LOAD_RECOVER || + spa->spa_load_state == SPA_LOAD_IMPORT || + spa->spa_load_state == SPA_LOAD_RECOVER || (spa->spa_import_flags & ZFS_IMPORT_VERBATIM)) need_update = B_TRUE; @@ -3057,6 +3151,11 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t int orig_mode = spa->spa_mode; boolean_t missing_feat_write = B_FALSE; + ASSERT(MUTEX_HELD(&spa_namespace_lock)); + + spa->spa_load_state = state; + spa_load_note(spa, "LOADING"); + /* * If this is an untrusted config, first access the pool in read-only * mode. We will then retrieve a trusted copy of the config from the MOS @@ -3068,7 +3167,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t /* * Parse the config provided to create a vdev tree. */ - error = spa_ld_parse_config(spa, pool_guid, config, state, type); + error = spa_ld_parse_config(spa, pool_guid, config, type); if (error != 0) return (error); @@ -3136,7 +3235,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * Retrieve the full list of active features from the MOS and check if * they are all supported. */ - error = spa_ld_check_features(spa, state, &missing_feat_write); + error = spa_ld_check_features(spa, &missing_feat_write); if (error != 0) return (error); @@ -3156,6 +3255,8 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t error = spa_ld_prepare_for_reload(spa, orig_mode); if (error != 0) return (error); + + spa_load_note(spa, "RELOADING"); return (spa_load(spa, state, SPA_IMPORT_EXISTING, B_TRUE)); } @@ -3178,7 +3279,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * Load the metadata for all vdevs. Also check if unopenable devices * should be autoreplaced. */ - error = spa_ld_load_vdev_metadata(spa, state); + error = spa_ld_load_vdev_metadata(spa); if (error != 0) return (error); @@ -3211,7 +3312,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * state. When performing an extreme rewind, we verify the whole pool, * which can take a very long time. */ - error = spa_ld_verify_pool_data(spa, state); + error = spa_ld_verify_pool_data(spa); if (error != 0) return (error); @@ -3268,7 +3369,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t * next sync, we would update the config stored in vdev labels * and the cachefile (by default /etc/zfs/zpool.cache). */ - spa_ld_check_for_config_update(spa, state, config_cache_txg); + spa_ld_check_for_config_update(spa, config_cache_txg); /* * Check all DTLs to see if anything needs resilvering. @@ -3305,6 +3406,8 @@ spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t spa_condense_indirect_restart(spa); } + spa_load_note(spa, "LOADED"); + return (0); } @@ -3321,6 +3424,9 @@ spa_load_retry(spa_t *spa, spa_load_state_t state, int spa_activate(spa, mode); spa_async_suspend(spa); + spa_load_note(spa, "spa_load_retry: rewind, max txg: %llu", + (u_longlong_t)spa->spa_load_max_txg); + return (spa_load(spa, state, SPA_IMPORT_EXISTING, trust_config)); } @@ -3472,6 +3578,7 @@ spa_open_common(const char *pool, spa_t **spapp, void if (state != SPA_LOAD_RECOVER) spa->spa_last_ubsync_txg = spa->spa_load_txg = 0; + zfs_dbgmsg("spa_open_common: opening %s", pool); error = spa_load_best(spa, state, B_FALSE, policy.zrp_txg, policy.zrp_request); @@ -4477,7 +4584,7 @@ spa_import(const char *pool, nvlist_t *config, nvlist_ spa_write_cachefile(spa, B_FALSE, B_TRUE); spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_IMPORT); - + zfs_dbgmsg("spa_import: verbatim import of %s", pool); mutex_exit(&spa_namespace_lock); return (0); } @@ -4501,6 +4608,8 @@ spa_import(const char *pool, nvlist_t *config, nvlist_ if (state != SPA_LOAD_RECOVER) spa->spa_last_ubsync_txg = spa->spa_load_txg = 0; + zfs_dbgmsg("spa_import: importing %s%s", pool, + (state == SPA_LOAD_RECOVER) ? " (RECOVERY MODE)" : ""); error = spa_load_best(spa, state, B_TRUE, policy.zrp_txg, policy.zrp_request); @@ -4637,6 +4746,8 @@ spa_tryimport(nvlist_t *tryconfig) mutex_enter(&spa_namespace_lock); spa = spa_add(TRYIMPORT_NAME, tryconfig, NULL); spa_activate(spa, FREAD); + + zfs_dbgmsg("spa_tryimport: importing %s", poolname); /* * Pass off the heavy lifting to spa_load(). Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Wed Feb 21 23:42:30 2018 (r329762) @@ -354,6 +354,34 @@ int spa_asize_inflation = 24; int spa_slop_shift = 5; uint64_t spa_min_slop = 128 * 1024 * 1024; +/*PRINTFLIKE2*/ +void +spa_load_failed(spa_t *spa, const char *fmt, ...) +{ + va_list adx; + char buf[256]; + + va_start(adx, fmt); + (void) vsnprintf(buf, sizeof (buf), fmt, adx); + va_end(adx); + + zfs_dbgmsg("spa_load(%s): FAILED: %s", spa->spa_name, buf); +} + +/*PRINTFLIKE2*/ +void +spa_load_note(spa_t *spa, const char *fmt, ...) +{ + va_list adx; + char buf[256]; + + va_start(adx, fmt); + (void) vsnprintf(buf, sizeof (buf), fmt, adx); + va_end(adx); + + zfs_dbgmsg("spa_load(%s): %s", spa->spa_name, buf); +} + /* * ========================================================================== * SPA config locking Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Wed Feb 21 23:42:30 2018 (r329762) @@ -807,6 +807,8 @@ extern objset_t *spa_meta_objset(spa_t *spa); extern uint64_t spa_deadman_synctime(spa_t *spa); /* Miscellaneous support routines */ +extern void spa_load_failed(spa_t *spa, const char *fmt, ...); +extern void spa_load_note(spa_t *spa, const char *fmt, ...); extern void spa_activate_mos_feature(spa_t *spa, const char *feature, dmu_tx_t *tx); extern void spa_deactivate_mos_feature(spa_t *spa, const char *feature); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h Wed Feb 21 23:42:30 2018 (r329762) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. */ #ifndef _SYS_VDEV_H @@ -47,6 +47,7 @@ typedef enum vdev_dtl_type { extern boolean_t zfs_nocacheflush; +extern void vdev_dbgmsg(vdev_t *vd, const char *fmt, ...); extern int vdev_open(vdev_t *); extern void vdev_open_children(vdev_t *); extern boolean_t vdev_uses_zvols(vdev_t *); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Wed Feb 21 23:42:30 2018 (r329762) @@ -77,6 +77,28 @@ int zfs_scrub_limit = 10; */ int metaslabs_per_vdev = 200; +/*PRINTFLIKE2*/ +void +vdev_dbgmsg(vdev_t *vd, const char *fmt, ...) +{ + va_list adx; + char buf[256]; + + va_start(adx, fmt); + (void) vsnprintf(buf, sizeof (buf), fmt, adx); + va_end(adx); + + if (vd->vdev_path != NULL) { + zfs_dbgmsg("%s vdev '%s': %s", vd->vdev_ops->vdev_op_type, + vd->vdev_path, buf); + } else { + zfs_dbgmsg("%s-%llu vdev (guid %llu): %s", + vd->vdev_ops->vdev_op_type, + (u_longlong_t)vd->vdev_id, + (u_longlong_t)vd->vdev_guid, buf); + } +} + /* * Given a vdev type, return the appropriate ops vector. */ @@ -932,14 +954,20 @@ vdev_metaslab_init(vdev_t *vd, uint64_t txg) error = dmu_read(mos, vd->vdev_ms_array, m * sizeof (uint64_t), sizeof (uint64_t), &object, DMU_READ_PREFETCH); - if (error) + if (error != 0) { + vdev_dbgmsg(vd, "unable to read the metaslab " + "array [error=%d]", error); return (error); + } } error = metaslab_init(vd->vdev_mg, m, object, txg, &(vd->vdev_ms[m])); - if (error) + if (error != 0) { + vdev_dbgmsg(vd, "metaslab_init failed [error=%d]", + error); return (error); + } } if (txg == 0) @@ -1021,8 +1049,7 @@ vdev_probe_done(zio_t *zio) zio->io_error = 0; } else { ASSERT(zio->io_error != 0); - zfs_dbgmsg("failed probe on vdev %llu", - (longlong_t)vd->vdev_id); + vdev_dbgmsg(vd, "failed probe"); zfs_ereport_post(FM_EREPORT_ZFS_PROBE_FAILURE, spa, vd, NULL, 0, 0); zio->io_error = SET_ERROR(ENXIO); @@ -1465,6 +1492,7 @@ vdev_validate(vdev_t *vd, boolean_t strict) if ((label = vdev_label_read_config(vd, txg)) == NULL) { vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN, VDEV_AUX_BAD_LABEL); + vdev_dbgmsg(vd, "vdev_validate: failed reading config"); return (0); } @@ -1477,6 +1505,8 @@ vdev_validate(vdev_t *vd, boolean_t strict) vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_SPLIT_POOL); nvlist_free(label); + vdev_dbgmsg(vd, "vdev_validate: vdev split into other " + "pool"); return (0); } @@ -1486,6 +1516,10 @@ vdev_validate(vdev_t *vd, boolean_t strict) vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); nvlist_free(label); + vdev_dbgmsg(vd, "vdev_validate: vdev label pool_guid " + "doesn't match config (%llu != %llu)", + (u_longlong_t)guid, + (u_longlong_t)spa_guid(spa)); return (0); } @@ -1515,6 +1549,9 @@ vdev_validate(vdev_t *vd, boolean_t strict) vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); nvlist_free(label); + vdev_dbgmsg(vd, "vdev_validate: config guid doesn't " + "match label guid (%llu != %llu)", + (u_longlong_t)vd->vdev_guid, (u_longlong_t)guid); return (0); } @@ -1523,6 +1560,8 @@ vdev_validate(vdev_t *vd, boolean_t strict) vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); nvlist_free(label); + vdev_dbgmsg(vd, "vdev_validate: '%s' missing", + ZPOOL_CONFIG_POOL_STATE); return (0); } @@ -1534,8 +1573,12 @@ vdev_validate(vdev_t *vd, boolean_t strict) */ if (!(spa->spa_import_flags & ZFS_IMPORT_VERBATIM) && spa_load_state(spa) == SPA_LOAD_OPEN && - state != POOL_STATE_ACTIVE) + state != POOL_STATE_ACTIVE) { + vdev_dbgmsg(vd, "vdev_validate: invalid pool state " + "(%llu) for spa %s", (u_longlong_t)state, + spa->spa_name); return (SET_ERROR(EBADF)); + } /* * If we were able to open and validate a vdev that was @@ -2149,9 +2192,10 @@ vdev_dtl_sync(vdev_t *vd, uint64_t txg) * the top level so that we update the config. */ if (object != space_map_object(vd->vdev_dtl_sm)) { - zfs_dbgmsg("txg %llu, spa %s, DTL old object %llu, " - "new object %llu", txg, spa_name(spa), object, - space_map_object(vd->vdev_dtl_sm)); + vdev_dbgmsg(vd, "txg %llu, spa %s, DTL old object %llu, " + "new object %llu", (u_longlong_t)txg, spa_name(spa), + (u_longlong_t)object, + (u_longlong_t)space_map_object(vd->vdev_dtl_sm)); vdev_config_dirty(vd->vdev_top); } @@ -2259,8 +2303,13 @@ vdev_load(vdev_t *vd) if (vd->vdev_ashift == 0 || vd->vdev_asize == 0) { vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); + vdev_dbgmsg(vd, "vdev_load: invalid size. ashift=%llu, " + "asize=%llu", (u_longlong_t)vd->vdev_ashift, + (u_longlong_t)vd->vdev_asize); return (SET_ERROR(ENXIO)); } else if ((error = vdev_metaslab_init(vd, 0)) != 0) { + vdev_dbgmsg(vd, "vdev_load: metaslab_init failed " + "[error=%d]", error); vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); return (error); @@ -2273,6 +2322,8 @@ vdev_load(vdev_t *vd) if (vd->vdev_ops->vdev_op_leaf && (error = vdev_dtl_load(vd)) != 0) { vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); + vdev_dbgmsg(vd, "vdev_load: vdev_dtl_load failed " + "[error=%d]", error); return (error); } @@ -2286,6 +2337,9 @@ vdev_load(vdev_t *vd) obsolete_sm_object, 0, vd->vdev_asize, 0))) { vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN, VDEV_AUX_CORRUPT_DATA); + vdev_dbgmsg(vd, "vdev_load: space_map_open failed for " + "obsolete spacemap (obj %llu) [error=%d]", + (u_longlong_t)obsolete_sm_object, error); return (error); } space_map_update(vd->vdev_obsolete_sm); @@ -3647,9 +3701,9 @@ vdev_deadman(vdev_t *vd) fio = avl_first(&vq->vq_active_tree); delta = gethrtime() - fio->io_timestamp; if (delta > spa_deadman_synctime(spa)) { - zfs_dbgmsg("SLOW IO: zio timestamp %lluns, " - "delta %lluns, last io %lluns", - fio->io_timestamp, delta, + vdev_dbgmsg(vd, "SLOW IO: zio timestamp " + "%lluns, delta %lluns, last io %lluns", + fio->io_timestamp, (u_longlong_t)delta, vq->vq_io_complete_ts); fm_panic("I/O to pool '%s' appears to be " "hung.", spa_name(spa)); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Wed Feb 21 23:42:30 2018 (r329762) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright 2016 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2013 Joyent, Inc. All rights reserved. */ @@ -325,6 +325,8 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * if (ddi_devid_str_decode(vd->vdev_devid, &dvd->vd_devid, &dvd->vd_minor) != 0) { vd->vdev_stat.vs_aux = VDEV_AUX_BAD_LABEL; + vdev_dbgmsg(vd, "vdev_disk_open: invalid " + "vdev_devid '%s'", vd->vdev_devid); return (SET_ERROR(EINVAL)); } } @@ -417,6 +419,8 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * if (error) { vd->vdev_stat.vs_aux = VDEV_AUX_OPEN_FAILED; + vdev_dbgmsg(vd, "vdev_disk_open: failed to open [error=%d]", + error); return (error); } @@ -430,8 +434,8 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * char *vd_devid; vd_devid = ddi_devid_str_encode(devid, dvd->vd_minor); - zfs_dbgmsg("vdev %s: update devid from %s, " - "to %s", vd->vdev_path, vd->vdev_devid, vd_devid); + vdev_dbgmsg(vd, "vdev_disk_open: update devid from " + "'%s' to '%s'", vd->vdev_devid, vd_devid); spa_strfree(vd->vdev_devid); vd->vdev_devid = spa_strdup(vd_devid); ddi_devid_str_free(vd_devid); @@ -491,6 +495,7 @@ skip_open: */ if (ldi_get_size(dvd->vd_lh, psize) != 0) { vd->vdev_stat.vs_aux = VDEV_AUX_OPEN_FAILED; + vdev_dbgmsg(vd, "vdev_disk_open: failed to get size"); return (SET_ERROR(EINVAL)); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Wed Feb 21 23:38:30 2018 (r329761) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Wed Feb 21 23:42:30 2018 (r329762) *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:09:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97237F0F166; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4559C7AB21; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3FCB41F8D9; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M09GTS096722; Thu, 22 Feb 2018 00:09:16 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M09GZ0096721; Thu, 22 Feb 2018 00:09:16 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220009.w1M09GZ0096721@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:09:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329766 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 329766 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:09:16 -0000 Author: mav Date: Thu Feb 22 00:09:15 2018 New Revision: 329766 URL: https://svnweb.freebsd.org/changeset/base/329766 Log: 8962 zdb should work on non-idle pools illumos/illumos-gate@e144c4e6c90e7d4dccaad6db660ee42b6e7ba04f Currently `zdb` consistently fails to examine non-idle pools as it fails during the `spa_load()` process. The main problem seems to be that `spa_load_verify()` fails as can be seen below: $ sudo zdb -d -G dcenter zdb: can't open 'dcenter': I/O error ZFS_DBGMSG(zdb): spa_open_common: opening dcenter spa_load(dcenter): LOADING disk vdev '/dev/dsk/c4t11d0s0': best uberblock found for spa dcenter. txg 40824950 spa_load(dcenter): using uberblock with txg=40824950 spa_load(dcenter): UNLOADING spa_load(dcenter): RELOADING spa_load(dcenter): LOADING disk vdev '/dev/dsk/c3t10d0s0': best uberblock found for spa dcenter. txg 40824952 spa_load(dcenter): using uberblock with txg=40824952 spa_load(dcenter): FAILED: spa_load_verify failed [error=5] spa_load(dcenter): UNLOADING This change makes `spa_load_verify()` a dryrun when ran from `zdb`. This is done by creating a global flag in zfs and then setting it in `zdb`. Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andy Stormont Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor/illumos/dist/cmd/zdb/zdb.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Thu Feb 22 00:03:14 2018 (r329765) +++ vendor/illumos/dist/cmd/zdb/zdb.c Thu Feb 22 00:09:15 2018 (r329766) @@ -89,12 +89,14 @@ extern boolean_t zfs_recover; extern uint64_t zfs_arc_max, zfs_arc_meta_limit; extern int zfs_vdev_async_read_max_active; extern int aok; +extern boolean_t spa_load_verify_dryrun; #else int reference_tracking_enable; boolean_t zfs_recover; uint64_t zfs_arc_max, zfs_arc_meta_limit; int zfs_vdev_async_read_max_active; int aok; +boolean_t spa_load_verify_dryrun; #endif static const char cmdname[] = "zdb"; @@ -4542,6 +4544,12 @@ main(int argc, char **argv) * Disable reference tracking for better performance. */ reference_tracking_enable = B_FALSE; + + /* + * Do not fail spa_load when spa_load_verify fails. This is needed + * to load non-idle pools. + */ + spa_load_verify_dryrun = B_TRUE; kernel_init(FREAD); g_zfs = libzfs_init(); From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:09:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67362F0F165; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19F497AB20; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 10B441F8D8; Thu, 22 Feb 2018 00:09:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M09Fwd096716; Thu, 22 Feb 2018 00:09:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M09FkI096715; Thu, 22 Feb 2018 00:09:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220009.w1M09FkI096715@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:09:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329766 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 329766 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:09:16 -0000 Author: mav Date: Thu Feb 22 00:09:15 2018 New Revision: 329766 URL: https://svnweb.freebsd.org/changeset/base/329766 Log: 8962 zdb should work on non-idle pools illumos/illumos-gate@e144c4e6c90e7d4dccaad6db660ee42b6e7ba04f Currently `zdb` consistently fails to examine non-idle pools as it fails during the `spa_load()` process. The main problem seems to be that `spa_load_verify()` fails as can be seen below: $ sudo zdb -d -G dcenter zdb: can't open 'dcenter': I/O error ZFS_DBGMSG(zdb): spa_open_common: opening dcenter spa_load(dcenter): LOADING disk vdev '/dev/dsk/c4t11d0s0': best uberblock found for spa dcenter. txg 40824950 spa_load(dcenter): using uberblock with txg=40824950 spa_load(dcenter): UNLOADING spa_load(dcenter): RELOADING spa_load(dcenter): LOADING disk vdev '/dev/dsk/c3t10d0s0': best uberblock found for spa dcenter. txg 40824952 spa_load(dcenter): using uberblock with txg=40824952 spa_load(dcenter): FAILED: spa_load_verify failed [error=5] spa_load(dcenter): UNLOADING This change makes `spa_load_verify()` a dryrun when ran from `zdb`. This is done by creating a global flag in zfs and then setting it in `zdb`. Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andy Stormont Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 00:03:14 2018 (r329765) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 00:09:15 2018 (r329766) @@ -165,6 +165,12 @@ boolean_t spa_create_process = B_TRUE; /* no process = extern int zfs_sync_pass_deferred_free; /* + * Report any spa_load_verify errors found, but do not fail spa_load. + * This is used by zdb to analyze non-idle pools. + */ +boolean_t spa_load_verify_dryrun = B_FALSE; + +/* * This (illegal) pool name is used when temporarily importing a spa_t in order * to get the vdev stats associated with the imported devices. */ @@ -2014,8 +2020,15 @@ spa_load_verify(spa_t *spa) spa->spa_load_meta_errors = sle.sle_meta_count; spa->spa_load_data_errors = sle.sle_data_count; - if (!error && sle.sle_meta_count <= policy.zrp_maxmeta && - sle.sle_data_count <= policy.zrp_maxdata) { + if (sle.sle_meta_count != 0 || sle.sle_data_count != 0) { + spa_load_note(spa, "spa_load_verify found %llu metadata errors " + "and %llu data errors", (u_longlong_t)sle.sle_meta_count, + (u_longlong_t)sle.sle_data_count); + } + + if (spa_load_verify_dryrun || + (!error && sle.sle_meta_count <= policy.zrp_maxmeta && + sle.sle_data_count <= policy.zrp_maxdata)) { int64_t loss = 0; verify_ok = B_TRUE; @@ -2032,6 +2045,9 @@ spa_load_verify(spa_t *spa) } else { spa->spa_load_max_txg = spa->spa_uberblock.ub_txg; } + + if (spa_load_verify_dryrun) + return (0); if (error) { if (error != ENXIO && error != EIO) From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:46:25 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9273F11C74; Thu, 22 Feb 2018 00:46:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 593737C34C; Thu, 22 Feb 2018 00:46:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4B7A41FF5D; Thu, 22 Feb 2018 00:46:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M0kOZ9016761; Thu, 22 Feb 2018 00:46:24 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M0kOJw016757; Thu, 22 Feb 2018 00:46:24 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220046.w1M0kOJw016757@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:46:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329770 - vendor-sys/illumos/dist/uts/common/fs/zfs/lua X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs/lua X-SVN-Commit-Revision: 329770 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:46:26 -0000 Author: mav Date: Thu Feb 22 00:46:24 2018 New Revision: 329770 URL: https://svnweb.freebsd.org/changeset/base/329770 Log: 9035 zfs: this statement may fall through illumos/illumos-gate@46ac8fdfc5a1f9d8240c79a6ae5b2889cbe83553 Reviewed by: Yuri Pankov Reviewed by: Andy Fiddaman Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Toomas Soome Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:42:12 2018 (r329769) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:46:24 2018 (r329770) @@ -677,7 +677,7 @@ static void freeobj (lua_State *L, GCObject *o) { case LUA_TUSERDATA: luaM_freemem(L, o, sizeudata(gco2u(o))); break; case LUA_TSHRSTR: G(L)->strt.nuse--; - /* go through */ + /* FALLTHROUGH */ case LUA_TLNGSTR: { luaM_freemem(L, o, sizestring(gco2ts(o))); break; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:42:12 2018 (r329769) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:46:24 2018 (r329770) @@ -475,6 +475,7 @@ static int llex (LexState *ls, SemInfo *seminfo) { else if (!lisdigit(ls->current)) return '.'; /* else go through */ } + /* FALLTHROUGH */ case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': { read_numeral(ls, seminfo); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Thu Feb 22 00:42:12 2018 (r329769) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Thu Feb 22 00:46:24 2018 (r329770) @@ -514,7 +514,7 @@ static const char *match (MatchState *ms, const char * } case '+': /* 1 or more repetitions */ s++; /* 1 match already done */ - /* go through */ + /* FALLTHROUGH */ case '*': /* 0 or more repetitions */ s = max_expand(ms, s, p, ep); break; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c Thu Feb 22 00:42:12 2018 (r329769) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c Thu Feb 22 00:46:24 2018 (r329770) @@ -490,6 +490,7 @@ const TValue *luaH_get (Table *t, const TValue *key) { return luaH_getint(t, k); /* use specialized version */ /* else go through */ } + /* FALLTHROUGH */ default: { Node *n = mainposition(t, key); do { /* check whether `key' is somewhere in the chain */ From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:49:00 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 279FDF11FD9; Thu, 22 Feb 2018 00:49:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D01057C654; Thu, 22 Feb 2018 00:48:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CAF421FF63; Thu, 22 Feb 2018 00:48:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M0mxr1016940; Thu, 22 Feb 2018 00:48:59 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M0mxgq016939; Thu, 22 Feb 2018 00:48:59 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220048.w1M0mxgq016939@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:48:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329772 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329772 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:49:00 -0000 Author: mav Date: Thu Feb 22 00:48:59 2018 New Revision: 329772 URL: https://svnweb.freebsd.org/changeset/base/329772 Log: 9036 zfs: duplicate 'const' declaration specifier illumos/illumos-gate@f02c28e434fb4d81d95122bab187fb43d4f19c2a Reviewed by: Yuri Pankov Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Toomas Soome Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Thu Feb 22 00:47:38 2018 (r329771) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Thu Feb 22 00:48:59 2018 (r329772) @@ -133,8 +133,8 @@ vdev_indirect_mapping_size(vdev_indirect_mapping_t *vi static int dva_mapping_overlap_compare(const void *v_key, const void *v_array_elem) { - const uint64_t const *key = v_key; - const vdev_indirect_mapping_entry_phys_t const *array_elem = + const uint64_t *key = v_key; + const vdev_indirect_mapping_entry_phys_t *array_elem = v_array_elem; uint64_t src_offset = DVA_MAPPING_GET_SRC_OFFSET(array_elem); From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:54:00 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E157F12746; Thu, 22 Feb 2018 00:54:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4093D7CC07; Thu, 22 Feb 2018 00:54:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3BA3A20104; Thu, 22 Feb 2018 00:54:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M0s02l021744; Thu, 22 Feb 2018 00:54:00 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M0s0H8021743; Thu, 22 Feb 2018 00:54:00 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220054.w1M0s0H8021743@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:54:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329774 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329774 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:54:00 -0000 Author: mav Date: Thu Feb 22 00:53:59 2018 New Revision: 329774 URL: https://svnweb.freebsd.org/changeset/base/329774 Log: 8408 dsl_props_set_sync_impl() does not handle nested nvlists correctly illumos/illumos-gate@85723e5eec42f46dbfdb4c09b9e1ed66501d1ccf When iterating over the input nvlist in dsl_props_set_sync_impl() when we don't preserve the nvpair name before looking up ZPROP_VALUE, so when we later go to process it nvpair_name() is always "value" instead of the actual property name. This results in a couple of bugs in the recv code: - received properties are not restored correctly when failing to receive an incremental send stream - received properties are not completely replaced by the new ones when successfully receiving an incremental send stream This was discovered on ZFS on Linux (fixed in https://github.com/zfsonlinux/zfs/commit/5f1346c29997dd4e02acf4c19c875d5484f33b1e) Reviewed by: Paul Dagnelie Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: loli10K Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_prop.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_prop.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_prop.c Thu Feb 22 00:53:05 2018 (r329773) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_prop.c Thu Feb 22 00:53:59 2018 (r329774) @@ -855,11 +855,15 @@ dsl_props_set_sync_impl(dsl_dataset_t *ds, zprop_sourc while ((elem = nvlist_next_nvpair(props, elem)) != NULL) { nvpair_t *pair = elem; + const char *name = nvpair_name(pair); if (nvpair_type(pair) == DATA_TYPE_NVLIST) { /* - * dsl_prop_get_all_impl() returns properties in this - * format. + * This usually happens when we reuse the nvlist_t data + * returned by the counterpart dsl_prop_get_all_impl(). + * For instance we do this to restore the original + * received properties when an error occurs in the + * zfs_ioc_recv() codepath. */ nvlist_t *attrs = fnvpair_value_nvlist(pair); pair = fnvlist_lookup_nvpair(attrs, ZPROP_VALUE); @@ -867,14 +871,14 @@ dsl_props_set_sync_impl(dsl_dataset_t *ds, zprop_sourc if (nvpair_type(pair) == DATA_TYPE_STRING) { const char *value = fnvpair_value_string(pair); - dsl_prop_set_sync_impl(ds, nvpair_name(pair), + dsl_prop_set_sync_impl(ds, name, source, 1, strlen(value) + 1, value, tx); } else if (nvpair_type(pair) == DATA_TYPE_UINT64) { uint64_t intval = fnvpair_value_uint64(pair); - dsl_prop_set_sync_impl(ds, nvpair_name(pair), + dsl_prop_set_sync_impl(ds, name, source, sizeof (intval), 1, &intval, tx); } else if (nvpair_type(pair) == DATA_TYPE_BOOLEAN) { - dsl_prop_set_sync_impl(ds, nvpair_name(pair), + dsl_prop_set_sync_impl(ds, name, source, 0, 0, NULL, tx); } else { panic("invalid nvpair type"); From owner-svn-src-vendor@freebsd.org Thu Feb 22 00:59:51 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E37A8F12F25; Thu, 22 Feb 2018 00:59:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 97CDA7D2A5; Thu, 22 Feb 2018 00:59:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6463620109; Thu, 22 Feb 2018 00:59:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M0xngN022532; Thu, 22 Feb 2018 00:59:49 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M0xnwE022531; Thu, 22 Feb 2018 00:59:49 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220059.w1M0xnwE022531@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 00:59:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329776 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329776 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 00:59:51 -0000 Author: mav Date: Thu Feb 22 00:59:49 2018 New Revision: 329776 URL: https://svnweb.freebsd.org/changeset/base/329776 Log: 8477 Assertion failed in vdev_state_dirty(): spa_writeable(spa) illumos/illumos-gate@f4c1745bd6c9829a05ecec15759ede7757100ab5 Illumos 4080 allows "zpool clear" to work on readonly pools: i don't think this is the intended behaviour, we shouldn't be allowed to clear readonly pools. Probably. A fix is already in the ZFS on Linux repository to addess this issue: https://github.com/zfsonlinux/zfs/commit/92e43c17188d47f47b69318e4884096dec380e36 Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: loli10K Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Feb 22 00:55:25 2018 (r329775) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Feb 22 00:59:49 2018 (r329776) @@ -5900,7 +5900,7 @@ zfs_ioctl_init(void) zfs_secpolicy_config, B_TRUE, POOL_CHECK_NONE); zfs_ioctl_register_pool(ZFS_IOC_CLEAR, zfs_ioc_clear, - zfs_secpolicy_config, B_TRUE, POOL_CHECK_NONE); + zfs_secpolicy_config, B_TRUE, POOL_CHECK_READONLY); zfs_ioctl_register_pool(ZFS_IOC_POOL_REOPEN, zfs_ioc_pool_reopen, zfs_secpolicy_config, B_TRUE, POOL_CHECK_SUSPENDED); From owner-svn-src-vendor@freebsd.org Thu Feb 22 01:04:59 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7484FF13671; Thu, 22 Feb 2018 01:04:59 +0000 (UTC) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB7AA7DBA1; Thu, 22 Feb 2018 01:04:58 +0000 (UTC) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w1M14v6w088615; Wed, 21 Feb 2018 17:04:57 -0800 (PST) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w1M14uHQ088614; Wed, 21 Feb 2018 17:04:56 -0800 (PST) (envelope-from freebsd) From: "Rodney W. Grimes" Message-Id: <201802220104.w1M14uHQ088614@pdx.rh.CN85.dnsmgr.net> Subject: Re: svn commit: r329770 - vendor-sys/illumos/dist/uts/common/fs/zfs/lua In-Reply-To: <201802220046.w1M0kOJw016757@repo.freebsd.org> To: Alexander Motin Date: Wed, 21 Feb 2018 17:04:56 -0800 (PST) CC: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Reply-To: rgrimes@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 01:04:59 -0000 [ Charset UTF-8 unsupported, converting... ] > Author: mav > Date: Thu Feb 22 00:46:24 2018 > New Revision: 329770 > URL: https://svnweb.freebsd.org/changeset/base/329770 > > Log: > 9035 zfs: this statement may fall through > > illumos/illumos-gate@46ac8fdfc5a1f9d8240c79a6ae5b2889cbe83553 > > Reviewed by: Yuri Pankov > Reviewed by: Andy Fiddaman > Reviewed by: Matthew Ahrens > Approved by: Dan McDonald > Author: Toomas Soome > > Modified: > vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c > vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c > vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c > vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c > > Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c > ============================================================================== > --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:42:12 2018 (r329769) > +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:46:24 2018 (r329770) > @@ -677,7 +677,7 @@ static void freeobj (lua_State *L, GCObject *o) { > case LUA_TUSERDATA: luaM_freemem(L, o, sizeudata(gco2u(o))); break; > case LUA_TSHRSTR: > G(L)->strt.nuse--; > - /* go through */ > + /* FALLTHROUGH */ > case LUA_TLNGSTR: { > luaM_freemem(L, o, sizestring(gco2ts(o))); > break; > > Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c > ============================================================================== > --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:42:12 2018 (r329769) > +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:46:24 2018 (r329770) > @@ -475,6 +475,7 @@ static int llex (LexState *ls, SemInfo *seminfo) { > else if (!lisdigit(ls->current)) return '.'; > /* else go through */ ^^^^^^^^^^^^^^^^^^^^^^^ Wasnt this already marked, perhaps in the wrong place with the wrong string? > } > + /* FALLTHROUGH */ > case '0': case '1': case '2': case '3': case '4': > case '5': case '6': case '7': case '8': case '9': { > read_numeral(ls, seminfo); > > Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c > ============================================================================== > --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Thu Feb 22 00:42:12 2018 (r329769) > +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Thu Feb 22 00:46:24 2018 (r329770) > @@ -514,7 +514,7 @@ static const char *match (MatchState *ms, const char * > } > case '+': /* 1 or more repetitions */ > s++; /* 1 match already done */ > - /* go through */ > + /* FALLTHROUGH */ > case '*': /* 0 or more repetitions */ > s = max_expand(ms, s, p, ep); > break; > > Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c > ============================================================================== > --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c Thu Feb 22 00:42:12 2018 (r329769) > +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c Thu Feb 22 00:46:24 2018 (r329770) > @@ -490,6 +490,7 @@ const TValue *luaH_get (Table *t, const TValue *key) { > return luaH_getint(t, k); /* use specialized version */ > /* else go through */ And again? > } > + /* FALLTHROUGH */ > default: { > Node *n = mainposition(t, key); > do { /* check whether `key' is somewhere in the chain */ -- Rod Grimes rgrimes@freebsd.org From owner-svn-src-vendor@freebsd.org Thu Feb 22 01:17:32 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1ACBF148B4; Thu, 22 Feb 2018 01:17:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 74FA37E4C7; Thu, 22 Feb 2018 01:17:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6FD80204C6; Thu, 22 Feb 2018 01:17:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M1HWei033010; Thu, 22 Feb 2018 01:17:32 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M1HWb1033009; Thu, 22 Feb 2018 01:17:32 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220117.w1M1HWb1033009@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 01:17:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329778 - vendor/illumos/dist/cmd/zpool X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/cmd/zpool X-SVN-Commit-Revision: 329778 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 01:17:32 -0000 Author: mav Date: Thu Feb 22 01:17:32 2018 New Revision: 329778 URL: https://svnweb.freebsd.org/changeset/base/329778 Log: 8941 zpool add: assertion failed in get_replication() with nested interior VDEVs illumos/illumos-gate@ac0215f4d618163d117a40fbf77a3f944852cb7b When replacing a faulted device which was previously handled by a spare multiple levels of nested interior VDEVs will be present in the pool configuration: get_replication() needs to handle this situation gracefully to let zpool add new devices to the pool Reviewed by: Matthew Ahrens Reviewed by: Andrew Stormont Approved by: Dan McDonald Author: loli10K Modified: vendor/illumos/dist/cmd/zpool/zpool_vdev.c Modified: vendor/illumos/dist/cmd/zpool/zpool_vdev.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_vdev.c Thu Feb 22 01:00:46 2018 (r329777) +++ vendor/illumos/dist/cmd/zpool/zpool_vdev.c Thu Feb 22 01:17:32 2018 (r329778) @@ -628,9 +628,11 @@ get_replication(nvlist_t *nvroot, boolean_t fatal) /* * If this is a replacing or spare vdev, then - * get the real first child of the vdev. + * get the real first child of the vdev: do this + * in a loop because replacing and spare vdevs + * can be nested. */ - if (strcmp(childtype, + while (strcmp(childtype, VDEV_TYPE_REPLACING) == 0 || strcmp(childtype, VDEV_TYPE_SPARE) == 0) { nvlist_t **rchild; From owner-svn-src-vendor@freebsd.org Thu Feb 22 01:30:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F076F15ACA; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 201937F0DA; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1B0D9206C1; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M1U3bW037991; Thu, 22 Feb 2018 01:30:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M1U3K6037990; Thu, 22 Feb 2018 01:30:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220130.w1M1U3K6037990@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 01:30:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329781 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 329781 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 01:30:04 -0000 Author: mav Date: Thu Feb 22 01:30:03 2018 New Revision: 329781 URL: https://svnweb.freebsd.org/changeset/base/329781 Log: 8942 zfs promote .../%recv should be an error illumos/illumos-gate@add927f8c8d101e16c23eb9cd270be4fd7edf7d5 Reported on the ZFSonLinux https://github.com/zfsonlinux/zfs/issues/4843, fixed by https://github.com/zfsonlinux/zfs/pull/6339: If we are in the middle of an incremental zfs receive, the child .../%recv will exist. If you concurrently run zfs promote .../%recv, it will "work", but then zfs gets confused. For example, there's no obvious way to destroy the containing filesystem (because it is now a clone of its invisible child). Attempting to do this promote should be an error. We could fix this by having zfs_ioc_promote() check if zc_name contains a %, similar to zfs_ioc_rename(). Reviewed by: Paul Dagnelie Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: loli10K Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Feb 22 01:25:28 2018 (r329780) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Feb 22 01:30:03 2018 (r329781) @@ -3758,9 +3758,12 @@ zfs_ioc_rename(zfs_cmd_t *zc) boolean_t recursive = zc->zc_cookie & 1; char *at; + /* "zfs rename" from and to ...%recv datasets should both fail */ + zc->zc_name[sizeof (zc->zc_name) - 1] = '\0'; zc->zc_value[sizeof (zc->zc_value) - 1] = '\0'; - if (dataset_namecheck(zc->zc_value, NULL, NULL) != 0 || - strchr(zc->zc_value, '%')) + if (dataset_namecheck(zc->zc_name, NULL, NULL) != 0 || + dataset_namecheck(zc->zc_value, NULL, NULL) != 0 || + strchr(zc->zc_name, '%') || strchr(zc->zc_value, '%')) return (SET_ERROR(EINVAL)); at = strchr(zc->zc_name, '@'); @@ -4782,6 +4785,11 @@ zfs_ioc_promote(zfs_cmd_t *zc) char origin[ZFS_MAX_DATASET_NAME_LEN]; char *cp; int error; + + zc->zc_name[sizeof (zc->zc_name) - 1] = '\0'; + if (dataset_namecheck(zc->zc_name, NULL, NULL) != 0 || + strchr(zc->zc_name, '%')) + return (SET_ERROR(EINVAL)); error = dsl_pool_hold(zc->zc_name, FTAG, &dp); if (error != 0) From owner-svn-src-vendor@freebsd.org Thu Feb 22 01:30:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC85AF15ACD; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DFCF7F0DC; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 68FE8206C2; Thu, 22 Feb 2018 01:30:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M1U4uO037997; Thu, 22 Feb 2018 01:30:04 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M1U4Fg037996; Thu, 22 Feb 2018 01:30:04 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220130.w1M1U4Fg037996@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 01:30:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329781 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 329781 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 01:30:05 -0000 Author: mav Date: Thu Feb 22 01:30:03 2018 New Revision: 329781 URL: https://svnweb.freebsd.org/changeset/base/329781 Log: 8942 zfs promote .../%recv should be an error illumos/illumos-gate@add927f8c8d101e16c23eb9cd270be4fd7edf7d5 Reported on the ZFSonLinux https://github.com/zfsonlinux/zfs/issues/4843, fixed by https://github.com/zfsonlinux/zfs/pull/6339: If we are in the middle of an incremental zfs receive, the child .../%recv will exist. If you concurrently run zfs promote .../%recv, it will "work", but then zfs gets confused. For example, there's no obvious way to destroy the containing filesystem (because it is now a clone of its invisible child). Attempting to do this promote should be an error. We could fix this by having zfs_ioc_promote() check if zc_name contains a %, similar to zfs_ioc_rename(). Reviewed by: Paul Dagnelie Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: loli10K Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Feb 22 01:25:28 2018 (r329780) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Feb 22 01:30:03 2018 (r329781) @@ -3737,6 +3737,9 @@ zfs_promote(zfs_handle_t *zhp) return (zfs_error(hdl, EZFS_BADTYPE, errbuf)); } + if (!zfs_validate_name(hdl, zhp->zfs_name, zhp->zfs_type, B_TRUE)) + return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); + ret = lzc_promote(zhp->zfs_name, snapname, sizeof (snapname)); if (ret != 0) { @@ -4093,6 +4096,10 @@ zfs_rename(zfs_handle_t *zhp, const char *target, bool (void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN, "cannot rename to '%s'"), target); + + /* make sure source name is valid */ + if (!zfs_validate_name(hdl, zhp->zfs_name, zhp->zfs_type, B_TRUE)) + return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); /* * Make sure the target name is valid From owner-svn-src-vendor@freebsd.org Thu Feb 22 02:21:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6E65F1A5DC; Thu, 22 Feb 2018 02:21:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4881E8246D; Thu, 22 Feb 2018 02:21:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 436EF21046; Thu, 22 Feb 2018 02:21:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M2L4Jq065452; Thu, 22 Feb 2018 02:21:04 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M2L3dj065445; Thu, 22 Feb 2018 02:21:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220221.w1M2L3dj065445@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 02:21:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329793 - in vendor/illumos/dist: cmd/zpool lib/libzfs/common lib/libzpool/common lib/libzpool/common/sys X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor/illumos/dist: cmd/zpool lib/libzfs/common lib/libzpool/common lib/libzpool/common/sys X-SVN-Commit-Revision: 329793 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 02:21:04 -0000 Author: mav Date: Thu Feb 22 02:21:03 2018 New Revision: 329793 URL: https://svnweb.freebsd.org/changeset/base/329793 Log: 9075 Improve ZFS pool import/load process and corrupted pool recovery illumos/illumos-gate@6f7938128a2c5e23f4b970ea101137eadd1470a1 Some work has been done lately to improve the debugability of the ZFS pool load (and import) process. This includes: https://www.illumos.org/issues/7638: Refactor spa_load_impl into several functions https://www.illumos.org/issues/8961: SPA load/import should tell us why it failed https://www.illumos.org/issues/7277: zdb should be able to print zfs_dbgmsg's To iterate on top of that, there's a few changes that were made to make the import process more resilient and crash free. One of the first tasks during the pool load process is to parse a config provided from userland that describes what devices the pool is composed of. A vdev tree is generated from that config, and then all the vdevs are opened. The Meta Object Set (MOS) of the pool is accessed, and several metadata objects that are necessary to load the pool are read. The exact configuration of the pool is also stored inside the MOS. Since the configuration provided from userland is external and might not accurately describe the vdev tree of the pool at the txg that is being loaded, it cannot be relied upon to safely operate the pool. For that reason, the configuration in the MOS is read early on. In the past, the two configurations were compared together and if there was a mismatch then the load process was aborted and an error was returned. The latter was a good way to ensure a pool does not get corrupted, however it made the pool load process needlessly fragile in cases where the vdev configuration changed or the userland configuration was outdated. Since the MOS is stored in 3 copies, the configuration provided by userland doesn't have to be perfect in order to read its contents. Hence, a new approach has been adopted: The pool is first opened with the untrusted userland configuration just so that the real configuration can be read from the MOS. The trusted MOS configuration is then used to generate a new vdev tree and the pool is re-opened. When the pool is opened with an untrusted configuration, writes are disabled to avoid accidentally damaging it. During reads, some sanity checks are performed on block pointers to see if each DVA points to a known vdev; when the configuration is untrusted, instead of panicking the system if those checks fail we simply avoid issuing reads to the invalid DVAs. This new two-step pool load process now allows rewinding pools accross vdev tree changes such as device replacement, addition, etc. Loading a pool from an external config file in a clustering environment also becomes much safer now since the pool will import even if the config is outdated and didn't, for instance, register a recent device addition. With this code in place, it became relatively easy to implement a long-sought-after feature: the ability to import a pool with missing top level (i.e. non-redundant) devices. Note that since this almost guarantees some loss Of data, this feature is for now restricted to a read-only import. Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andrew Stormont Approved by: Hans Rosenfeld Author: Pavel Zakharov Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_import.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Thu Feb 22 02:21:03 2018 (r329793) @@ -1562,6 +1562,10 @@ print_status_config(zpool_handle_t *zhp, const char *n (void) printf(gettext("split into new pool")); break; + case VDEV_AUX_CHILDREN_OFFLINE: + (void) printf(gettext("all children offline")); + break; + default: (void) printf(gettext("corrupted data")); break; @@ -1649,6 +1653,10 @@ print_import_config(const char *name, nvlist_t *nv, in (void) printf(gettext("too many errors")); break; + case VDEV_AUX_CHILDREN_OFFLINE: + (void) printf(gettext("all children offline")); + break; + default: (void) printf(gettext("corrupted data")); break; @@ -2296,6 +2304,7 @@ zpool_do_import(int argc, char **argv) idata.poolname = searchname; idata.guid = searchguid; idata.cachefile = cachefile; + idata.policy = policy; pools = zpool_search_import(g_zfs, &idata); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs.h Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/lib/libzfs/common/libzfs.h Thu Feb 22 02:21:03 2018 (r329793) @@ -388,6 +388,7 @@ typedef struct importargs { int can_be_active : 1; /* can the pool be active? */ int unique : 1; /* does 'poolname' already exist? */ int exists : 1; /* set on return if pool already exists */ + nvlist_t *policy; /* rewind policy (rewind txg, etc.) */ } importargs_t; extern nvlist_t *zpool_search_import(libzfs_handle_t *, importargs_t *); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_import.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Thu Feb 22 02:21:03 2018 (r329793) @@ -412,7 +412,8 @@ vdev_is_hole(uint64_t *hole_array, uint_t holes, uint_ * return to the user. */ static nvlist_t * -get_configs(libzfs_handle_t *hdl, pool_list_t *pl, boolean_t active_ok) +get_configs(libzfs_handle_t *hdl, pool_list_t *pl, boolean_t active_ok, + nvlist_t *policy) { pool_entry_t *pe; vdev_entry_t *ve; @@ -746,6 +747,12 @@ get_configs(libzfs_handle_t *hdl, pool_list_t *pl, boo continue; } + if (policy != NULL) { + if (nvlist_add_nvlist(config, ZPOOL_REWIND_POLICY, + policy) != 0) + goto nomem; + } + if ((nvl = refresh_config(hdl, config)) == NULL) { nvlist_free(config); config = NULL; @@ -1251,7 +1258,7 @@ zpool_find_import_impl(libzfs_handle_t *hdl, importarg goto error; } - ret = get_configs(hdl, &pools, iarg->can_be_active); + ret = get_configs(hdl, &pools, iarg->can_be_active, iarg->policy); error: for (pe = pools.pools; pe != NULL; pe = penext) { @@ -1380,6 +1387,14 @@ zpool_find_import_cached(libzfs_handle_t *hdl, const c if (active) continue; + + if (nvlist_add_string(src, ZPOOL_CONFIG_CACHEFILE, + cachefile) != 0) { + (void) no_memory(hdl); + nvlist_free(raw); + nvlist_free(pools); + return (NULL); + } if ((dst = refresh_config(hdl, src)) == NULL) { nvlist_free(raw); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Thu Feb 22 02:21:03 2018 (r329793) @@ -1808,8 +1808,9 @@ zpool_import_props(libzfs_handle_t *hdl, nvlist_t *con nvlist_lookup_nvlist(nvinfo, ZPOOL_CONFIG_MISSING_DEVICES, &missing) == 0) { (void) printf(dgettext(TEXT_DOMAIN, - "The devices below are missing, use " - "'-m' to import the pool anyway:\n")); + "The devices below are missing or " + "corrupted, use '-m' to import the pool " + "anyway:\n")); print_vdev_tree(hdl, NULL, missing, 2); (void) printf("\n"); } Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/kernel.c Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/lib/libzpool/common/kernel.c Thu Feb 22 02:21:03 2018 (r329793) @@ -461,6 +461,16 @@ kernel_fini(void) system_taskq_fini(); } +/* ARGSUSED */ +uint32_t +zone_get_hostid(void *zonep) +{ + /* + * We're emulating the system's hostid in userland. + */ + return (strtoul(hw_serial, NULL, 10)); +} + int z_uncompress(void *dst, size_t *dstlen, const void *src, size_t srclen) { Modified: vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Thu Feb 22 02:16:44 2018 (r329792) +++ vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Thu Feb 22 02:21:03 2018 (r329793) @@ -317,6 +317,7 @@ typedef struct callb_cpr { #define zone_dataset_visible(x, y) (1) #define INGLOBALZONE(z) (1) +extern uint32_t zone_get_hostid(void *zonep); extern int zfs_secpolicy_snapshot_perms(const char *name, cred_t *cr); extern int zfs_secpolicy_rename_perms(const char *from, const char *to, From owner-svn-src-vendor@freebsd.org Thu Feb 22 02:25:10 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 590E0F1AD9C; Thu, 22 Feb 2018 02:25:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A612828B8; Thu, 22 Feb 2018 02:25:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DF04D211E7; Thu, 22 Feb 2018 02:25:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M2P9ss068352; Thu, 22 Feb 2018 02:25:09 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M2P9Un068350; Thu, 22 Feb 2018 02:25:09 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220225.w1M2P9Un068350@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 02:25:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329795 - in vendor-sys/illumos/dist/uts/common: fs/zfs fs/zfs/sys sys/fs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common: fs/zfs fs/zfs/sys sys/fs X-SVN-Commit-Revision: 329795 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 02:25:10 -0000 Author: mav Date: Thu Feb 22 02:25:09 2018 New Revision: 329795 URL: https://svnweb.freebsd.org/changeset/base/329795 Log: r329793 | mav | 2018-02-22 04:21:03 +0200 (чт, 22 февр. 2018) | 58 lines 9075 Improve ZFS pool import/load process and corrupted pool recovery illumos/illumos-gate@6f7938128a2c5e23f4b970ea101137eadd1470a1 Some work has been done lately to improve the debugability of the ZFS pool load (and import) process. This includes: https://www.illumos.org/issues/7638: Refactor spa_load_impl into several functions https://www.illumos.org/issues/8961: SPA load/import should tell us why it failed https://www.illumos.org/issues/7277: zdb should be able to print zfs_dbgmsg's To iterate on top of that, there's a few changes that were made to make the import process more resilient and crash free. One of the first tasks during the pool load process is to parse a config provided from userland that describes what devices the pool is composed of. A vdev tree is generated from that config, and then all the vdevs are opened. The Meta Object Set (MOS) of the pool is accessed, and several metadata objects that are necessary to load the pool are read. The exact configuration of the pool is also stored inside the MOS. Since the configuration provided from userland is external and might not accurately describe the vdev tree of the pool at the txg that is being loaded, it cannot be relied upon to safely operate the pool. For that reason, the configuration in the MOS is read early on. In the past, the two configurations were compared together and if there was a mismatch then the load process was aborted and an error was returned. The latter was a good way to ensure a pool does not get corrupted, however it made the pool load process needlessly fragile in cases where the vdev configuration changed or the userland configuration was outdated. Since the MOS is stored in 3 copies, the configuration provided by userland doesn't have to be perfect in order to read its contents. Hence, a new approach has been adopted: The pool is first opened with the untrusted userland configuration just so that the real configuration can be read from the MOS. The trusted MOS configuration is then used to generate a new vdev tree and the pool is re-opened. When the pool is opened with an untrusted configuration, writes are disabled to avoid accidentally damaging it. During reads, some sanity checks are performed on block pointers to see if each DVA points to a known vdev; when the configuration is untrusted, instead of panicking the system if those checks fail we simply avoid issuing reads to the invalid DVAs. This new two-step pool load process now allows rewinding pools accross vdev tree changes such as device replacement, addition, etc. Loading a pool from an external config file in a clustering environment also becomes much safer now since the pool will import even if the config is outdated and didn't, for instance, register a recent device addition. With this code in place, it became relatively easy to implement a long-sought-after feature: the ability to import a pool with missing top level (i.e. non-redundant) devices. Note that since this almost guarantees some loss Of data, this feature is for now restricted to a read-only import. Reviewed by: George Wilson Reviewed by: Matthew Ahrens Reviewed by: Andrew Stormont Approved by: Hans Rosenfeld Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 02:24:17 2018 (r329794) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 02:25:09 2018 (r329795) @@ -151,9 +151,8 @@ const zio_taskq_info_t zio_taskqs[ZIO_TYPES][ZIO_TASKQ static void spa_sync_version(void *arg, dmu_tx_t *tx); static void spa_sync_props(void *arg, dmu_tx_t *tx); static boolean_t spa_has_active_shared_spare(spa_t *spa); -static int spa_load_impl(spa_t *spa, uint64_t, nvlist_t *config, - spa_load_state_t state, spa_import_type_t type, boolean_t trust_config, - char **ereport); +static int spa_load_impl(spa_t *spa, spa_import_type_t type, char **ereport, + boolean_t reloading); static void spa_vdev_resilver_done(spa_t *spa); uint_t zio_taskq_batch_pct = 75; /* 1 thread per cpu in pset */ @@ -177,6 +176,54 @@ boolean_t spa_load_verify_dryrun = B_FALSE; #define TRYIMPORT_NAME "$import" /* + * For debugging purposes: print out vdev tree during pool import. + */ +boolean_t spa_load_print_vdev_tree = B_FALSE; + +/* + * A non-zero value for zfs_max_missing_tvds means that we allow importing + * pools with missing top-level vdevs. This is strictly intended for advanced + * pool recovery cases since missing data is almost inevitable. Pools with + * missing devices can only be imported read-only for safety reasons, and their + * fail-mode will be automatically set to "continue". + * + * With 1 missing vdev we should be able to import the pool and mount all + * datasets. User data that was not modified after the missing device has been + * added should be recoverable. This means that snapshots created prior to the + * addition of that device should be completely intact. + * + * With 2 missing vdevs, some datasets may fail to mount since there are + * dataset statistics that are stored as regular metadata. Some data might be + * recoverable if those vdevs were added recently. + * + * With 3 or more missing vdevs, the pool is severely damaged and MOS entries + * may be missing entirely. Chances of data recovery are very low. Note that + * there are also risks of performing an inadvertent rewind as we might be + * missing all the vdevs with the latest uberblocks. + */ +uint64_t zfs_max_missing_tvds = 0; + +/* + * The parameters below are similar to zfs_max_missing_tvds but are only + * intended for a preliminary open of the pool with an untrusted config which + * might be incomplete or out-dated. + * + * We are more tolerant for pools opened from a cachefile since we could have + * an out-dated cachefile where a device removal was not registered. + * We could have set the limit arbitrarily high but in the case where devices + * are really missing we would want to return the proper error codes; we chose + * SPA_DVAS_PER_BP - 1 so that some copies of the MOS would still be available + * and we get a chance to retrieve the trusted config. + */ +uint64_t zfs_max_missing_tvds_cachefile = SPA_DVAS_PER_BP - 1; +/* + * In the case where config was assembled by scanning device paths (/dev/dsks + * by default) we are less tolerant since all the existing devices should have + * been detected and we want spa_load to return the right error codes. + */ +uint64_t zfs_max_missing_tvds_scan = 0; + +/* * ========================================================================== * SPA properties routines * ========================================================================== @@ -1633,13 +1680,34 @@ load_nvlist(spa_t *spa, uint64_t obj, nvlist_t **value } /* + * Concrete top-level vdevs that are not missing and are not logs. At every + * spa_sync we write new uberblocks to at least SPA_SYNC_MIN_VDEVS core tvds. + */ +static uint64_t +spa_healthy_core_tvds(spa_t *spa) +{ + vdev_t *rvd = spa->spa_root_vdev; + uint64_t tvds = 0; + + for (uint64_t i = 0; i < rvd->vdev_children; i++) { + vdev_t *vd = rvd->vdev_child[i]; + if (vd->vdev_islog) + continue; + if (vdev_is_concrete(vd) && !vdev_is_dead(vd)) + tvds++; + } + + return (tvds); +} + +/* * Checks to see if the given vdev could not be opened, in which case we post a * sysevent to notify the autoreplace code that the device has been removed. */ static void spa_check_removed(vdev_t *vd) { - for (int c = 0; c < vd->vdev_children; c++) + for (uint64_t c = 0; c < vd->vdev_children; c++) spa_check_removed(vd->vdev_child[c]); if (vd->vdev_ops->vdev_op_leaf && vdev_is_dead(vd) && @@ -1649,38 +1717,14 @@ spa_check_removed(vdev_t *vd) } } -static void -spa_config_valid_zaps(vdev_t *vd, vdev_t *mvd) +static int +spa_check_for_missing_logs(spa_t *spa) { - ASSERT3U(vd->vdev_children, ==, mvd->vdev_children); + vdev_t *rvd = spa->spa_root_vdev; - vd->vdev_top_zap = mvd->vdev_top_zap; - vd->vdev_leaf_zap = mvd->vdev_leaf_zap; - - for (uint64_t i = 0; i < vd->vdev_children; i++) { - spa_config_valid_zaps(vd->vdev_child[i], mvd->vdev_child[i]); - } -} - -/* - * Validate the current config against the MOS config - */ -static boolean_t -spa_config_valid(spa_t *spa, nvlist_t *config) -{ - vdev_t *mrvd, *rvd = spa->spa_root_vdev; - nvlist_t *nv; - - VERIFY(nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nv) == 0); - - spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - VERIFY(spa_config_parse(spa, &mrvd, nv, NULL, 0, VDEV_ALLOC_LOAD) == 0); - - ASSERT3U(rvd->vdev_children, ==, mrvd->vdev_children); - /* * If we're doing a normal import, then build up any additional - * diagnostic information about missing devices in this config. + * diagnostic information about missing log devices. * We'll pass this up to the user for further processing. */ if (!(spa->spa_import_flags & ZFS_IMPORT_MISSING_LOG)) { @@ -1691,109 +1735,52 @@ spa_config_valid(spa_t *spa, nvlist_t *config) KM_SLEEP); VERIFY(nvlist_alloc(&nv, NV_UNIQUE_NAME, KM_SLEEP) == 0); - for (int c = 0; c < rvd->vdev_children; c++) { + for (uint64_t c = 0; c < rvd->vdev_children; c++) { vdev_t *tvd = rvd->vdev_child[c]; - vdev_t *mtvd = mrvd->vdev_child[c]; - if (tvd->vdev_ops == &vdev_missing_ops && - mtvd->vdev_ops != &vdev_missing_ops && - mtvd->vdev_islog) - child[idx++] = vdev_config_generate(spa, mtvd, - B_FALSE, 0); + /* + * We consider a device as missing only if it failed + * to open (i.e. offline or faulted is not considered + * as missing). + */ + if (tvd->vdev_islog && + tvd->vdev_state == VDEV_STATE_CANT_OPEN) { + child[idx++] = vdev_config_generate(spa, tvd, + B_FALSE, VDEV_CONFIG_MISSING); + } } - if (idx) { - VERIFY(nvlist_add_nvlist_array(nv, - ZPOOL_CONFIG_CHILDREN, child, idx) == 0); - VERIFY(nvlist_add_nvlist(spa->spa_load_info, - ZPOOL_CONFIG_MISSING_DEVICES, nv) == 0); + if (idx > 0) { + fnvlist_add_nvlist_array(nv, + ZPOOL_CONFIG_CHILDREN, child, idx); + fnvlist_add_nvlist(spa->spa_load_info, + ZPOOL_CONFIG_MISSING_DEVICES, nv); - for (int i = 0; i < idx; i++) + for (uint64_t i = 0; i < idx; i++) nvlist_free(child[i]); } nvlist_free(nv); kmem_free(child, rvd->vdev_children * sizeof (char **)); - } - /* - * Compare the root vdev tree with the information we have - * from the MOS config (mrvd). Check each top-level vdev - * with the corresponding MOS config top-level (mtvd). - */ - for (int c = 0; c < rvd->vdev_children; c++) { - vdev_t *tvd = rvd->vdev_child[c]; - vdev_t *mtvd = mrvd->vdev_child[c]; + if (idx > 0) { + spa_load_failed(spa, "some log devices are missing"); + return (SET_ERROR(ENXIO)); + } + } else { + for (uint64_t c = 0; c < rvd->vdev_children; c++) { + vdev_t *tvd = rvd->vdev_child[c]; - /* - * Resolve any "missing" vdevs in the current configuration. - * Also trust the MOS config about any "indirect" vdevs. - * If we find that the MOS config has more accurate information - * about the top-level vdev then use that vdev instead. - */ - if ((tvd->vdev_ops == &vdev_missing_ops && - mtvd->vdev_ops != &vdev_missing_ops) || - (mtvd->vdev_ops == &vdev_indirect_ops && - tvd->vdev_ops != &vdev_indirect_ops)) { - - /* - * Device specific actions. - */ - if (mtvd->vdev_islog) { - if (!(spa->spa_import_flags & - ZFS_IMPORT_MISSING_LOG)) { - continue; - } - + if (tvd->vdev_islog && + tvd->vdev_state == VDEV_STATE_CANT_OPEN) { spa_set_log_state(spa, SPA_LOG_CLEAR); - } else if (mtvd->vdev_ops != &vdev_indirect_ops) { - continue; + spa_load_note(spa, "some log devices are " + "missing, ZIL is dropped."); + break; } - - /* - * Swap the missing vdev with the data we were - * able to obtain from the MOS config. - */ - vdev_remove_child(rvd, tvd); - vdev_remove_child(mrvd, mtvd); - - vdev_add_child(rvd, mtvd); - vdev_add_child(mrvd, tvd); - - vdev_reopen(rvd); - } else { - if (mtvd->vdev_islog) { - /* - * Load the slog device's state from the MOS - * config since it's possible that the label - * does not contain the most up-to-date - * information. - */ - vdev_load_log_state(tvd, mtvd); - vdev_reopen(tvd); - } - - /* - * Per-vdev ZAP info is stored exclusively in the MOS. - */ - spa_config_valid_zaps(tvd, mtvd); } - - /* - * Never trust this info from userland; always use what's - * in the MOS. This prevents it from getting out of sync - * with the rest of the info in the MOS. - */ - tvd->vdev_removing = mtvd->vdev_removing; - tvd->vdev_indirect_config = mtvd->vdev_indirect_config; } - vdev_free(mrvd); - spa_config_exit(spa, SCL_ALL, FTAG); - - /* - * Ensure we were able to validate the config. - */ - return (rvd->vdev_guid_sum == spa->spa_uberblock.ub_guid_sum); + return (0); } /* @@ -2175,55 +2162,17 @@ spa_try_repair(spa_t *spa, nvlist_t *config) } static int -spa_load(spa_t *spa, spa_load_state_t state, spa_import_type_t type, - boolean_t trust_config) +spa_load(spa_t *spa, spa_load_state_t state, spa_import_type_t type) { - nvlist_t *config = spa->spa_config; char *ereport = FM_EREPORT_ZFS_POOL; - char *comment; int error; - uint64_t pool_guid; - nvlist_t *nvl; - if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID, &pool_guid)) - return (SET_ERROR(EINVAL)); + spa->spa_load_state = state; - ASSERT(spa->spa_comment == NULL); - if (nvlist_lookup_string(config, ZPOOL_CONFIG_COMMENT, &comment) == 0) - spa->spa_comment = spa_strdup(comment); + gethrestime(&spa->spa_loaded_ts); + error = spa_load_impl(spa, type, &ereport, B_FALSE); /* - * Versioning wasn't explicitly added to the label until later, so if - * it's not present treat it as the initial version. - */ - if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION, - &spa->spa_ubsync.ub_version) != 0) - spa->spa_ubsync.ub_version = SPA_VERSION_INITIAL; - - (void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, - &spa->spa_config_txg); - - if ((state == SPA_LOAD_IMPORT || state == SPA_LOAD_TRYIMPORT) && - spa_guid_exists(pool_guid, 0)) { - error = SET_ERROR(EEXIST); - } else { - spa->spa_config_guid = pool_guid; - - if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_SPLIT, - &nvl) == 0) { - VERIFY(nvlist_dup(nvl, &spa->spa_config_splitting, - KM_SLEEP) == 0); - } - - nvlist_free(spa->spa_load_info); - spa->spa_load_info = fnvlist_alloc(); - - gethrestime(&spa->spa_loaded_ts); - error = spa_load_impl(spa, pool_guid, config, state, type, - trust_config, &ereport); - } - - /* * Don't count references from objsets that are already closed * and are making their way through the eviction process. */ @@ -2273,23 +2222,87 @@ vdev_count_verify_zaps(vdev_t *vd) } static int -spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nvlist_t *config, - spa_import_type_t type) +spa_verify_host(spa_t *spa, nvlist_t *mos_config) { + uint64_t hostid; + char *hostname; + uint64_t myhostid = 0; + + if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config, + ZPOOL_CONFIG_HOSTID, &hostid) == 0) { + hostname = fnvlist_lookup_string(mos_config, + ZPOOL_CONFIG_HOSTNAME); + + myhostid = zone_get_hostid(NULL); + + if (hostid != 0 && myhostid != 0 && hostid != myhostid) { + cmn_err(CE_WARN, "pool '%s' could not be " + "loaded as it was last accessed by " + "another system (host: %s hostid: 0x%llx). " + "See: http://illumos.org/msg/ZFS-8000-EY", + spa_name(spa), hostname, (u_longlong_t)hostid); + spa_load_failed(spa, "hostid verification failed: pool " + "last accessed by host: %s (hostid: 0x%llx)", + hostname, (u_longlong_t)hostid); + return (SET_ERROR(EBADF)); + } + } + + return (0); +} + +static int +spa_ld_parse_config(spa_t *spa, spa_import_type_t type) +{ int error = 0; - nvlist_t *nvtree = NULL; + nvlist_t *nvtree, *nvl, *config = spa->spa_config; int parse; vdev_t *rvd; + uint64_t pool_guid; + char *comment; + /* + * Versioning wasn't explicitly added to the label until later, so if + * it's not present treat it as the initial version. + */ + if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION, + &spa->spa_ubsync.ub_version) != 0) + spa->spa_ubsync.ub_version = SPA_VERSION_INITIAL; + + if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID, &pool_guid)) { + spa_load_failed(spa, "invalid config provided: '%s' missing", + ZPOOL_CONFIG_POOL_GUID); + return (SET_ERROR(EINVAL)); + } + + if ((spa->spa_load_state == SPA_LOAD_IMPORT || spa->spa_load_state == + SPA_LOAD_TRYIMPORT) && spa_guid_exists(pool_guid, 0)) { + spa_load_failed(spa, "a pool with guid %llu is already open", + (u_longlong_t)pool_guid); + return (SET_ERROR(EEXIST)); + } + + spa->spa_config_guid = pool_guid; + + nvlist_free(spa->spa_load_info); + spa->spa_load_info = fnvlist_alloc(); + + ASSERT(spa->spa_comment == NULL); + if (nvlist_lookup_string(config, ZPOOL_CONFIG_COMMENT, &comment) == 0) + spa->spa_comment = spa_strdup(comment); + + (void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, + &spa->spa_config_txg); + + if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_SPLIT, &nvl) == 0) + spa->spa_config_splitting = fnvlist_dup(nvl); + if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvtree)) { spa_load_failed(spa, "invalid config provided: '%s' missing", ZPOOL_CONFIG_VDEV_TREE); return (SET_ERROR(EINVAL)); } - parse = (type == SPA_IMPORT_EXISTING ? - VDEV_ALLOC_LOAD : VDEV_ALLOC_SPLIT); - /* * Create "The Godfather" zio to hold all async IOs */ @@ -2307,6 +2320,8 @@ spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nv * configuration requires knowing the version number. */ spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); + parse = (type == SPA_IMPORT_EXISTING ? + VDEV_ALLOC_LOAD : VDEV_ALLOC_SPLIT); error = spa_config_parse(spa, &rvd, nvtree, NULL, 0, parse); spa_config_exit(spa, SCL_ALL, FTAG); @@ -2327,71 +2342,105 @@ spa_ld_parse_config(spa_t *spa, uint64_t pool_guid, nv return (0); } +/* + * Recursively open all vdevs in the vdev tree. This function is called twice: + * first with the untrusted config, then with the trusted config. + */ static int spa_ld_open_vdevs(spa_t *spa) { int error = 0; + /* + * spa_missing_tvds_allowed defines how many top-level vdevs can be + * missing/unopenable for the root vdev to be still considered openable. + */ + if (spa->spa_trust_config) { + spa->spa_missing_tvds_allowed = zfs_max_missing_tvds; + } else if (spa->spa_config_source == SPA_CONFIG_SRC_CACHEFILE) { + spa->spa_missing_tvds_allowed = zfs_max_missing_tvds_cachefile; + } else if (spa->spa_config_source == SPA_CONFIG_SRC_SCAN) { + spa->spa_missing_tvds_allowed = zfs_max_missing_tvds_scan; + } else { + spa->spa_missing_tvds_allowed = 0; + } + + spa->spa_missing_tvds_allowed = + MAX(zfs_max_missing_tvds, spa->spa_missing_tvds_allowed); + spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); error = vdev_open(spa->spa_root_vdev); spa_config_exit(spa, SCL_ALL, FTAG); + + if (spa->spa_missing_tvds != 0) { + spa_load_note(spa, "vdev tree has %lld missing top-level " + "vdevs.", (u_longlong_t)spa->spa_missing_tvds); + if (spa->spa_trust_config && (spa->spa_mode & FWRITE)) { + /* + * Although theoretically we could allow users to open + * incomplete pools in RW mode, we'd need to add a lot + * of extra logic (e.g. adjust pool space to account + * for missing vdevs). + * This limitation also prevents users from accidentally + * opening the pool in RW mode during data recovery and + * damaging it further. + */ + spa_load_note(spa, "pools with missing top-level " + "vdevs can only be opened in read-only mode."); + error = SET_ERROR(ENXIO); + } else { + spa_load_note(spa, "current settings allow for maximum " + "%lld missing top-level vdevs at this stage.", + (u_longlong_t)spa->spa_missing_tvds_allowed); + } + } if (error != 0) { spa_load_failed(spa, "unable to open vdev tree [error=%d]", error); } + if (spa->spa_missing_tvds != 0 || error != 0) + vdev_dbgmsg_print_tree(spa->spa_root_vdev, 2); return (error); } +/* + * We need to validate the vdev labels against the configuration that + * we have in hand. This function is called twice: first with an untrusted + * config, then with a trusted config. The validation is more strict when the + * config is trusted. + */ static int -spa_ld_validate_vdevs(spa_t *spa, spa_import_type_t type, - boolean_t trust_config) +spa_ld_validate_vdevs(spa_t *spa) { int error = 0; vdev_t *rvd = spa->spa_root_vdev; - /* - * We need to validate the vdev labels against the configuration that - * we have in hand, which is dependent on the setting of trust_config. - * If trust_config is true then we're validating the vdev labels based - * on that config. Otherwise, we're validating against the cached - * config (zpool.cache) that was read when we loaded the zfs module, and - * then later we will recursively call spa_load() and validate against - * the vdev config. - * - * If we're assembling a new pool that's been split off from an - * existing pool, the labels haven't yet been updated so we skip - * validation for now. - */ - if (type != SPA_IMPORT_ASSEMBLE) { - spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - error = vdev_validate(rvd, trust_config); - spa_config_exit(spa, SCL_ALL, FTAG); + spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); + error = vdev_validate(rvd); + spa_config_exit(spa, SCL_ALL, FTAG); - if (error != 0) { - spa_load_failed(spa, "vdev_validate failed [error=%d]", - error); - return (error); - } + if (error != 0) { + spa_load_failed(spa, "vdev_validate failed [error=%d]", error); + return (error); + } - if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) { - spa_load_failed(spa, "cannot open vdev tree after " - "invalidating some vdevs"); - return (SET_ERROR(ENXIO)); - } + if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) { + spa_load_failed(spa, "cannot open vdev tree after invalidating " + "some vdevs"); + vdev_dbgmsg_print_tree(rvd, 2); + return (SET_ERROR(ENXIO)); } return (0); } static int -spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, spa_import_type_t type, - boolean_t trust_config) +spa_ld_select_uberblock(spa_t *spa, spa_import_type_t type) { vdev_t *rvd = spa->spa_root_vdev; nvlist_t *label; uberblock_t *ub = &spa->spa_uberblock; - uint64_t children; /* * Find the best uberblock. @@ -2484,26 +2533,9 @@ spa_ld_select_uberblock(spa_t *spa, nvlist_t *config, nvlist_free(unsup_feat); } - /* - * If the vdev guid sum doesn't match the uberblock, we have an - * incomplete configuration. We first check to see if the pool - * is aware of the complete config (i.e ZPOOL_CONFIG_VDEV_CHILDREN). - * If it is, defer the vdev_guid_sum check till later so we - * can handle missing vdevs. - */ - if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VDEV_CHILDREN, - &children) != 0 && trust_config && type != SPA_IMPORT_ASSEMBLE && - rvd->vdev_guid_sum != ub->ub_guid_sum) { - spa_load_failed(spa, "guid sum in config doesn't match guid " - "sum in uberblock (%llu != %llu)", - (u_longlong_t)rvd->vdev_guid_sum, - (u_longlong_t)ub->ub_guid_sum); - return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); - } - if (type != SPA_IMPORT_ASSEMBLE && spa->spa_config_splitting) { spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - spa_try_repair(spa, config); + spa_try_repair(spa, spa->spa_config); spa_config_exit(spa, SCL_ALL, FTAG); nvlist_free(spa->spa_config_splitting); spa->spa_config_splitting = NULL; @@ -2542,49 +2574,167 @@ spa_ld_open_rootbp(spa_t *spa) } static int -spa_ld_validate_config(spa_t *spa, spa_import_type_t type) +spa_ld_load_trusted_config(spa_t *spa, spa_import_type_t type, + boolean_t reloading) { - vdev_t *rvd = spa->spa_root_vdev; + vdev_t *mrvd, *rvd = spa->spa_root_vdev; + nvlist_t *nv, *mos_config, *policy; + int error = 0, copy_error; + uint64_t healthy_tvds, healthy_tvds_mos; + uint64_t mos_config_txg; if (spa_dir_prop(spa, DMU_POOL_CONFIG, &spa->spa_config_object, B_TRUE) != 0) return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); /* - * Validate the config, using the MOS config to fill in any - * information which might be missing. If we fail to validate - * the config then declare the pool unfit for use. If we're - * assembling a pool from a split, the log is not transferred - * over. + * If we're assembling a pool from a split, the config provided is + * already trusted so there is nothing to do. */ - if (type != SPA_IMPORT_ASSEMBLE) { - nvlist_t *mos_config; - if (load_nvlist(spa, spa->spa_config_object, &mos_config) - != 0) { - spa_load_failed(spa, "unable to retrieve MOS config"); - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); - } + if (type == SPA_IMPORT_ASSEMBLE) + return (0); - if (!spa_config_valid(spa, mos_config)) { + healthy_tvds = spa_healthy_core_tvds(spa); + + if (load_nvlist(spa, spa->spa_config_object, &mos_config) + != 0) { + spa_load_failed(spa, "unable to retrieve MOS config"); + return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); + } + + /* + * If we are doing an open, pool owner wasn't verified yet, thus do + * the verification here. + */ + if (spa->spa_load_state == SPA_LOAD_OPEN) { + error = spa_verify_host(spa, mos_config); + if (error != 0) { nvlist_free(mos_config); - spa_load_failed(spa, "mismatch between config provided " - "and config stored in MOS"); - return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, - ENXIO)); + return (error); } - nvlist_free(mos_config); + } + nv = fnvlist_lookup_nvlist(mos_config, ZPOOL_CONFIG_VDEV_TREE); + + spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); + + /* + * Build a new vdev tree from the trusted config + */ + VERIFY(spa_config_parse(spa, &mrvd, nv, NULL, 0, VDEV_ALLOC_LOAD) == 0); + + /* + * Vdev paths in the MOS may be obsolete. If the untrusted config was + * obtained by scanning /dev/dsk, then it will have the right vdev + * paths. We update the trusted MOS config with this information. + * We first try to copy the paths with vdev_copy_path_strict, which + * succeeds only when both configs have exactly the same vdev tree. + * If that fails, we fall back to a more flexible method that has a + * best effort policy. + */ + copy_error = vdev_copy_path_strict(rvd, mrvd); + if (copy_error != 0 || spa_load_print_vdev_tree) { + spa_load_note(spa, "provided vdev tree:"); + vdev_dbgmsg_print_tree(rvd, 2); + spa_load_note(spa, "MOS vdev tree:"); + vdev_dbgmsg_print_tree(mrvd, 2); + } + if (copy_error != 0) { + spa_load_note(spa, "vdev_copy_path_strict failed, falling " + "back to vdev_copy_path_relaxed"); + vdev_copy_path_relaxed(rvd, mrvd); + } + + vdev_close(rvd); + vdev_free(rvd); + spa->spa_root_vdev = mrvd; + rvd = mrvd; + spa_config_exit(spa, SCL_ALL, FTAG); + + /* + * We will use spa_config if we decide to reload the spa or if spa_load + * fails and we rewind. We must thus regenerate the config using the + * MOS information with the updated paths. Rewind policy is an import + * setting and is not in the MOS. We copy it over to our new, trusted + * config. + */ + mos_config_txg = fnvlist_lookup_uint64(mos_config, + ZPOOL_CONFIG_POOL_TXG); + nvlist_free(mos_config); + mos_config = spa_config_generate(spa, NULL, mos_config_txg, B_FALSE); + if (nvlist_lookup_nvlist(spa->spa_config, ZPOOL_REWIND_POLICY, + &policy) == 0) + fnvlist_add_nvlist(mos_config, ZPOOL_REWIND_POLICY, policy); + spa_config_set(spa, mos_config); + spa->spa_config_source = SPA_CONFIG_SRC_MOS; + + /* + * Now that we got the config from the MOS, we should be more strict + * in checking blkptrs and can make assumptions about the consistency + * of the vdev tree. spa_trust_config must be set to true before opening + * vdevs in order for them to be writeable. + */ + spa->spa_trust_config = B_TRUE; + + /* + * Open and validate the new vdev tree + */ + error = spa_ld_open_vdevs(spa); + if (error != 0) + return (error); + + error = spa_ld_validate_vdevs(spa); + if (error != 0) + return (error); + + if (copy_error != 0 || spa_load_print_vdev_tree) { + spa_load_note(spa, "final vdev tree:"); + vdev_dbgmsg_print_tree(rvd, 2); + } + + if (spa->spa_load_state != SPA_LOAD_TRYIMPORT && + !spa->spa_extreme_rewind && zfs_max_missing_tvds == 0) { /* - * Now that we've validated the config, check the state of the - * root vdev. If it can't be opened, it indicates one or - * more toplevel vdevs are faulted. + * Sanity check to make sure that we are indeed loading the + * latest uberblock. If we missed SPA_SYNC_MIN_VDEVS tvds + * in the config provided and they happened to be the only ones + * to have the latest uberblock, we could involuntarily perform + * an extreme rewind. */ - if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) { - spa_load_failed(spa, "some top vdevs are unavailable"); - return (SET_ERROR(ENXIO)); + healthy_tvds_mos = spa_healthy_core_tvds(spa); + if (healthy_tvds_mos - healthy_tvds >= + SPA_SYNC_MIN_VDEVS) { + spa_load_note(spa, "config provided misses too many " + "top-level vdevs compared to MOS (%lld vs %lld). ", + (u_longlong_t)healthy_tvds, + (u_longlong_t)healthy_tvds_mos); + spa_load_note(spa, "vdev tree:"); + vdev_dbgmsg_print_tree(rvd, 2); + if (reloading) { + spa_load_failed(spa, "config was already " + "provided from MOS. Aborting."); + return (spa_vdev_err(rvd, + VDEV_AUX_CORRUPT_DATA, EIO)); + } + spa_load_note(spa, "spa must be reloaded using MOS " + "config"); + return (SET_ERROR(EAGAIN)); } } + error = spa_check_for_missing_logs(spa); + if (error != 0) + return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO)); + + if (rvd->vdev_guid_sum != spa->spa_uberblock.ub_guid_sum) { + spa_load_failed(spa, "uberblock guid sum doesn't match MOS " + "guid sum (%llu != %llu)", + (u_longlong_t)spa->spa_uberblock.ub_guid_sum, + (u_longlong_t)rvd->vdev_guid_sum); + return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, + ENXIO)); + } + return (0); } @@ -2751,62 +2901,6 @@ spa_ld_load_special_directories(spa_t *spa) } static int -spa_ld_prepare_for_reload(spa_t *spa, int orig_mode) -{ - vdev_t *rvd = spa->spa_root_vdev; - - uint64_t hostid; - nvlist_t *policy = NULL; - nvlist_t *mos_config; - - if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) { - spa_load_failed(spa, "unable to retrieve MOS config"); - return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO)); - } - - if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config, - ZPOOL_CONFIG_HOSTID, &hostid) == 0) { - char *hostname; - unsigned long myhostid = 0; - - VERIFY(nvlist_lookup_string(mos_config, - ZPOOL_CONFIG_HOSTNAME, &hostname) == 0); - -#ifdef _KERNEL - myhostid = zone_get_hostid(NULL); -#else /* _KERNEL */ - /* - * We're emulating the system's hostid in userland, so - * we can't use zone_get_hostid(). - */ - (void) ddi_strtoul(hw_serial, NULL, 10, &myhostid); -#endif /* _KERNEL */ - if (hostid != 0 && myhostid != 0 && - hostid != myhostid) { - nvlist_free(mos_config); - cmn_err(CE_WARN, "pool '%s' could not be " - "loaded as it was last accessed by " - "another system (host: %s hostid: 0x%lx). " - "See: http://illumos.org/msg/ZFS-8000-EY", - spa_name(spa), hostname, - (unsigned long)hostid); - return (SET_ERROR(EBADF)); - } - } - if (nvlist_lookup_nvlist(spa->spa_config, - ZPOOL_REWIND_POLICY, &policy) == 0) - VERIFY(nvlist_add_nvlist(mos_config, - ZPOOL_REWIND_POLICY, policy) == 0); - - spa_config_set(spa, mos_config); - spa_unload(spa); - spa_deactivate(spa); - spa_activate(spa, orig_mode); - - return (0); -} - -static int spa_ld_get_props(spa_t *spa) { int error = 0; @@ -2933,6 +3027,19 @@ spa_ld_get_props(spa_t *spa) spa->spa_autoreplace = (autoreplace != 0); } + /* + * If we are importing a pool with missing top-level vdevs, + * we enforce that the pool doesn't panic or get suspended on + * error since the likelihood of missing data is extremely high. + */ + if (spa->spa_missing_tvds > 0 && + spa->spa_failmode != ZIO_FAILURE_MODE_CONTINUE && + spa->spa_load_state != SPA_LOAD_TRYIMPORT) { + spa_load_note(spa, "forcing failmode to 'continue' " + "as some top level vdevs are missing"); + spa->spa_failmode = ZIO_FAILURE_MODE_CONTINUE; + } + return (0); } @@ -3063,9 +3170,15 @@ spa_ld_verify_logs(spa_t *spa, spa_import_type_t type, if (type != SPA_IMPORT_ASSEMBLE && spa_writeable(spa)) { boolean_t missing = spa_check_logs(spa); if (missing) { - *ereport = FM_EREPORT_ZFS_LOG_REPLAY; - spa_load_failed(spa, "spa_check_logs failed"); - return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, ENXIO)); + if (spa->spa_missing_tvds != 0) { + spa_load_note(spa, "spa_check_logs failed " + "so dropping the logs"); + } else { + *ereport = FM_EREPORT_ZFS_LOG_REPLAY; + spa_load_failed(spa, "spa_check_logs failed"); + return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG, + ENXIO)); + } } } @@ -3121,7 +3234,8 @@ spa_ld_claim_log_blocks(spa_t *spa) } static void -spa_ld_check_for_config_update(spa_t *spa, uint64_t config_cache_txg) +spa_ld_check_for_config_update(spa_t *spa, uint64_t config_cache_txg, + boolean_t reloading) { vdev_t *rvd = spa->spa_root_vdev; int need_update = B_FALSE; @@ -3133,7 +3247,7 @@ spa_ld_check_for_config_update(spa_t *spa, uint64_t co * If this is a verbatim import, trust the current * in-core spa_config and update the disk labels. */ - if (config_cache_txg != spa->spa_config_txg || + if (reloading || config_cache_txg != spa->spa_config_txg || spa->spa_load_state == SPA_LOAD_IMPORT || spa->spa_load_state == SPA_LOAD_RECOVER || (spa->spa_import_flags & ZFS_IMPORT_VERBATIM)) @@ -3151,6 +3265,24 @@ spa_ld_check_for_config_update(spa_t *spa, uint64_t co spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE); } +static void +spa_ld_prepare_for_reload(spa_t *spa) +{ + int mode = spa->spa_mode; + int async_suspended = spa->spa_async_suspended; + + spa_unload(spa); + spa_deactivate(spa); + spa_activate(spa, mode); + + /* + * We save the value of spa_async_suspended as it gets reset to 0 by + * spa_unload(). We want to restore it back to the original value before + * returning as we might be calling spa_async_resume() later. + */ + spa->spa_async_suspended = async_suspended; +} + /* * Load an existing storage pool, using the config provided. This config * describes which vdevs are part of the pool and is later validated against @@ -3158,32 +3290,35 @@ spa_ld_check_for_config_update(spa_t *spa, uint64_t co * config stored in the MOS. */ static int -spa_load_impl(spa_t *spa, uint64_t pool_guid, nvlist_t *config, - spa_load_state_t state, spa_import_type_t type, boolean_t trust_config, - char **ereport) +spa_load_impl(spa_t *spa, spa_import_type_t type, char **ereport, + boolean_t reloading) { int error = 0; - uint64_t config_cache_txg = spa->spa_config_txg; - int orig_mode = spa->spa_mode; boolean_t missing_feat_write = B_FALSE; ASSERT(MUTEX_HELD(&spa_namespace_lock)); + ASSERT(spa->spa_config_source != SPA_CONFIG_SRC_NONE); - spa->spa_load_state = state; - spa_load_note(spa, "LOADING"); - /* - * If this is an untrusted config, first access the pool in read-only - * mode. We will then retrieve a trusted copy of the config from the MOS - * and use it to reopen the pool in read-write mode. + * Never trust the config that is provided unless we are assembling + * a pool following a split. + * This means don't trust blkptrs and the vdev tree in general. This + * also effectively puts the spa in read-only mode since + * spa_writeable() checks for spa_trust_config to be true. + * We will later load a trusted config from the MOS. */ - if (!trust_config) - spa->spa_mode = FREAD; + if (type != SPA_IMPORT_ASSEMBLE) + spa->spa_trust_config = B_FALSE; + if (reloading) + spa_load_note(spa, "RELOADING"); + else + spa_load_note(spa, "LOADING"); + /* *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Feb 22 03:03:49 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A70A4F1FDC3; Thu, 22 Feb 2018 03:03:49 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-it0-x230.google.com (mail-it0-x230.google.com [IPv6:2607:f8b0:4001:c0b::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3DCBE84FEA; Thu, 22 Feb 2018 03:03:49 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-it0-x230.google.com with SMTP id o13so4592906ito.2; Wed, 21 Feb 2018 19:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=TXNGTqA4AMIeBH8S6sRW7XS9QpkKNIgxk+L9x/04q24=; b=sYga6ICiIXoK3X3HKfLoCHvtmmoaVuUr1d8odDCX0xmBADWr/GCICYVZDXrdSAA3S8 zG7T1/6sfrJ0isML+6Ax3gDST+VvWjwbwcsnKvV6kegUIraZ25Urf7BUE36OtPTEIKxq 57XJ+MlDS2Pf8MiZ7lCssxyK8YPnD0SnZRhE4Pz2UROZCetCYJPlJn4Bijif7fC+uhCl VEeJ+b3FrFuGrGrVb+HlfaVBpszXOjd/8tB4n2vLZK4ev0bAdcNN7YGaWv10p0GngCRZ 22CMiW5Q2lYo061p5OJ+3kE8FWlnZJSHDeQK3P2hai8MD5+KF/BUKTY+vn28f+z9fMyT eSZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=TXNGTqA4AMIeBH8S6sRW7XS9QpkKNIgxk+L9x/04q24=; b=F0hAuLgyRlTc3pVD9SwdsRulPQXj9rDXPJ+G1prt4Oy9+1ACQa0qPE9PaGt7DfmSPO 4NCqOCR2eunC4EnpoqcB3Q5dMilaHr+eYyGu3B0s25HN6Cf8EioeB9myMxmdsOPMPB3Y 2/6Q+Vk+7mc1AEM2QCskAEmwKxtAlcB2ckv3I6/WNjB5QtoMIC5CuSZSOOyYWmhI93lD sBrEWmBlUKJMLrwPqoNms4jgjQCk813/D+yUVslo2m6TAMV5/0m3jByv0tMR34Ym3Q/R P/HI+gRpa4+bC8u1/ESAgz2i3yNaIOmQ0QD6KhDHSnEeQygzP+a5qhnsyiAXKq9U2DUK QaLQ== X-Gm-Message-State: APf1xPCJUC4beem2u1yJwTatGIWQFG7lmzmlpqYMfDh//bBgH/+KeL68 IMt504evWD4d0DYrLwW3m+ciZlZb X-Google-Smtp-Source: AG47ELtlJu7xqbUzr1JISXuPu2tdbBlCxQLY23cwhtIaGKfx5C9AjgC0UN7xkFjl8aUZlJIQepIDIA== X-Received: by 10.36.151.197 with SMTP id k188mr6444362ite.45.1519268628256; Wed, 21 Feb 2018 19:03:48 -0800 (PST) Received: from spectre.mavhome.dp.ua ([2600:1700:3580:3560:205:1bff:fee0:411b]) by smtp.gmail.com with ESMTPSA id j2sm10446356iob.43.2018.02.21.19.03.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 19:03:47 -0800 (PST) Sender: Alexander Motin Subject: Re: svn commit: r329770 - vendor-sys/illumos/dist/uts/common/fs/zfs/lua To: rgrimes@freebsd.org Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org References: <201802220104.w1M14uHQ088614@pdx.rh.CN85.dnsmgr.net> From: Alexander Motin Message-ID: <56dda9e2-e9ea-83b9-1e54-6569d478615a@FreeBSD.org> Date: Wed, 21 Feb 2018 22:03:47 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201802220104.w1M14uHQ088614@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 03:03:50 -0000 On 21.02.2018 20:04, Rodney W. Grimes wrote: > [ Charset UTF-8 unsupported, converting... ] >> Author: mav >> Date: Thu Feb 22 00:46:24 2018 >> New Revision: 329770 >> URL: https://svnweb.freebsd.org/changeset/base/329770 >> >> Log: >> 9035 zfs: this statement may fall through >> >> illumos/illumos-gate@46ac8fdfc5a1f9d8240c79a6ae5b2889cbe83553 >> >> Reviewed by: Yuri Pankov >> Reviewed by: Andy Fiddaman >> Reviewed by: Matthew Ahrens >> Approved by: Dan McDonald >> Author: Toomas Soome >> >> Modified: >> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c >> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c >> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c >> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c >> >> Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c >> ============================================================================== >> --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:42:12 2018 (r329769) >> +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb 22 00:46:24 2018 (r329770) >> @@ -677,7 +677,7 @@ static void freeobj (lua_State *L, GCObject *o) { >> case LUA_TUSERDATA: luaM_freemem(L, o, sizeudata(gco2u(o))); break; >> case LUA_TSHRSTR: >> G(L)->strt.nuse--; >> - /* go through */ >> + /* FALLTHROUGH */ >> case LUA_TLNGSTR: { >> luaM_freemem(L, o, sizestring(gco2ts(o))); >> break; >> >> Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c >> ============================================================================== >> --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:42:12 2018 (r329769) >> +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb 22 00:46:24 2018 (r329770) >> @@ -475,6 +475,7 @@ static int llex (LexState *ls, SemInfo *seminfo) { >> else if (!lisdigit(ls->current)) return '.'; >> /* else go through */ > ^^^^^^^^^^^^^^^^^^^^^^^ Wasnt this already marked, perhaps in the > wrong place with the wrong string? As I understand GCC 7 parses comments like that to decide whether generate warnings or not, but this informal text is out of his understanding. -- Alexander Motin From owner-svn-src-vendor@freebsd.org Thu Feb 22 03:23:44 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80180F21F1B; Thu, 22 Feb 2018 03:23:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 306E0865C3; Thu, 22 Feb 2018 03:23:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 298CA21D4A; Thu, 22 Feb 2018 03:23:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M3NiXb098918; Thu, 22 Feb 2018 03:23:44 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M3Nhjg098916; Thu, 22 Feb 2018 03:23:43 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220323.w1M3Nhjg098916@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 03:23:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329800 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 329800 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 03:23:44 -0000 Author: mav Date: Thu Feb 22 03:23:43 2018 New Revision: 329800 URL: https://svnweb.freebsd.org/changeset/base/329800 Log: Missed pieces of r329799. Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c (contents, props changed) Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h Thu Feb 22 03:23:43 2018 (r329800) @@ -0,0 +1,52 @@ +/* + * CDDL HEADER START + * + * This file and its contents are supplied under the terms of the + * Common Development and Distribution License ("CDDL"), version 1.0. + * You may only use this file in accordance with the terms of version + * 1.0 of the CDDL. + * + * A full copy of the text of the CDDL should have accompanied this + * source. A copy of the CDDL is also available via the Internet at + * http://www.illumos.org/license/CDDL. + * + * CDDL HEADER END + */ + + +/* + * Copyright (c) 2017 by Delphix. All rights reserved. + */ + +#ifndef _SYS_ZTHR_H +#define _SYS_ZTHR_H + +typedef struct zthr zthr_t; +typedef int (zthr_func_t)(void *, zthr_t *); +typedef boolean_t (zthr_checkfunc_t)(void *, zthr_t *); + +struct zthr { + kthread_t *zthr_thread; + kmutex_t zthr_lock; + kcondvar_t zthr_cv; + boolean_t zthr_cancel; + + zthr_checkfunc_t *zthr_checkfunc; + zthr_func_t *zthr_func; + void *zthr_arg; + int zthr_rc; +}; + +extern zthr_t *zthr_create(zthr_checkfunc_t checkfunc, + zthr_func_t *func, void *arg); +extern void zthr_exit(zthr_t *t, int rc); +extern void zthr_destroy(zthr_t *t); + +extern void zthr_wakeup(zthr_t *t); +extern int zthr_cancel(zthr_t *t); +extern void zthr_resume(zthr_t *t); + +extern boolean_t zthr_iscancelled(zthr_t *t); +extern boolean_t zthr_isrunning(zthr_t *t); + +#endif /* _SYS_ZTHR_H */ Added: vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c Thu Feb 22 03:23:43 2018 (r329800) @@ -0,0 +1,319 @@ +/* + * CDDL HEADER START + * + * This file and its contents are supplied under the terms of the + * Common Development and Distribution License ("CDDL"), version 1.0. + * You may only use this file in accordance with the terms of version + * 1.0 of the CDDL. + * + * A full copy of the text of the CDDL should have accompanied this + * source. A copy of the CDDL is also available via the Internet at + * http://www.illumos.org/license/CDDL. + * + * CDDL HEADER END + */ + +/* + * Copyright (c) 2017 by Delphix. All rights reserved. + */ + +/* + * ZTHR Infrastructure + * =================== + * + * ZTHR threads are used for isolated operations that span multiple txgs + * within a SPA. They generally exist from SPA creation/loading and until + * the SPA is exported/destroyed. The ideal requirements for an operation + * to be modeled with a zthr are the following: + * + * 1] The operation needs to run over multiple txgs. + * 2] There is be a single point of reference in memory or on disk that + * indicates whether the operation should run/is running or is + * stopped. + * + * If the operation satisfies the above then the following rules guarantee + * a certain level of correctness: + * + * 1] Any thread EXCEPT the zthr changes the work indicator from stopped + * to running but not the opposite. + * 2] Only the zthr can change the work indicator from running to stopped + * (e.g. when it is done) but not the opposite. + * + * This way a normal zthr cycle should go like this: + * + * 1] An external thread changes the work indicator from stopped to + * running and wakes up the zthr. + * 2] The zthr wakes up, checks the indicator and starts working. + * 3] When the zthr is done, it changes the indicator to stopped, allowing + * a new cycle to start. + * + * == ZTHR creation + * + * Every zthr needs three inputs to start running: + * + * 1] A user-defined checker function (checkfunc) that decides whether + * the zthr should start working or go to sleep. The function should + * return TRUE when the zthr needs to work or FALSE to let it sleep, + * and should adhere to the following signature: + * boolean_t checkfunc_name(void *args, zthr_t *t); + * + * 2] A user-defined ZTHR function (func) which the zthr executes when + * it is not sleeping. The function should adhere to the following + * signature type: + * int func_name(void *args, zthr_t *t); + * + * 3] A void args pointer that will be passed to checkfunc and func + * implicitly by the infrastructure. + * + * The reason why the above API needs two different functions, + * instead of one that both checks and does the work, has to do with + * the zthr's internal lock (zthr_lock) and the allowed cancellation + * windows. We want to hold the zthr_lock while running checkfunc + * but not while running func. This way the zthr can be cancelled + * while doing work and not while checking for work. + * + * To start a zthr: + * zthr_t *zthr_pointer = zthr_create(checkfunc, func, args); + * + * After that you should be able to wakeup, cancel, and resume the + * zthr from another thread using zthr_pointer. + * + * NOTE: ZTHR threads could potentially wake up spuriously and the + * user should take this into account when writing a checkfunc. + * [see ZTHR state transitions] + * + * == ZTHR cancellation + * + * ZTHR threads must be cancelled when their SPA is being exported + * or when they need to be paused so they don't interfere with other + * operations. + * + * To cancel a zthr: + * zthr_cancel(zthr_pointer); + * + * To resume it: + * zthr_resume(zthr_pointer); + * + * A zthr will implicitly check if it has received a cancellation + * signal every time func returns and everytime it wakes up [see ZTHR + * state transitions below]. + * + * At times, waiting for the zthr's func to finish its job may take + * time. This may be very time-consuming for some operations that + * need to cancel the SPA's zthrs (e.g spa_export). For this scenario + * the user can explicitly make their ZTHR function aware of incoming + * cancellation signals using zthr_iscancelled(). A common pattern for + * that looks like this: + * + * int + * func_name(void *args, zthr_t *t) + * { + * ... ... + * while (!work_done && !zthr_iscancelled(t)) { + * ... ... + * } + * return (0); + * } + * + * == ZTHR exit + * + * For the rare cases where the zthr wants to stop running voluntarily + * while running its ZTHR function (func), we provide zthr_exit(). + * When a zthr has voluntarily stopped running, it can be resumed with + * zthr_resume(), just like it would if it was cancelled by some other + * thread. + * + * == ZTHR cleanup + * + * Cancelling a zthr doesn't clean up its metadata (internal locks, + * function pointers to func and checkfunc, etc..). This is because + * we want to keep them around in case we want to resume the execution + * of the zthr later. Similarly for zthrs that exit themselves. + * + * To completely cleanup a zthr, cancel it first to ensure that it + * is not running and then use zthr_destroy(). + * + * == ZTHR state transitions + * + * zthr creation + * + + * | + * | woke up + * | +--------------+ sleep + * | | ^ + * | | | + * | | | FALSE + * | | | + * v v FALSE + + * cancelled? +---------> checkfunc? + * + ^ + + * | | | + * | | | TRUE + * | | | + * | | func returned v + * | +---------------+ func + * | + * | TRUE + * | + * v + * zthr stopped running + * + */ + +#include +#include + +void +zthr_exit(zthr_t *t, int rc) +{ + ASSERT3P(t->zthr_thread, ==, curthread); + mutex_enter(&t->zthr_lock); + t->zthr_thread = NULL; + t->zthr_rc = rc; + cv_broadcast(&t->zthr_cv); + mutex_exit(&t->zthr_lock); + thread_exit(); +} + +static void +zthr_procedure(void *arg) +{ + zthr_t *t = arg; + int rc = 0; + + mutex_enter(&t->zthr_lock); + while (!t->zthr_cancel) { + if (t->zthr_checkfunc(t->zthr_arg, t)) { + mutex_exit(&t->zthr_lock); + rc = t->zthr_func(t->zthr_arg, t); + mutex_enter(&t->zthr_lock); + } else { + /* go to sleep */ + cv_wait(&t->zthr_cv, &t->zthr_lock); + } + } + mutex_exit(&t->zthr_lock); + + zthr_exit(t, rc); +} + +zthr_t * +zthr_create(zthr_checkfunc_t *checkfunc, zthr_func_t *func, void *arg) +{ + zthr_t *t = kmem_zalloc(sizeof (*t), KM_SLEEP); + mutex_init(&t->zthr_lock, NULL, MUTEX_DEFAULT, NULL); + cv_init(&t->zthr_cv, NULL, CV_DEFAULT, NULL); + + mutex_enter(&t->zthr_lock); + t->zthr_checkfunc = checkfunc; + t->zthr_func = func; + t->zthr_arg = arg; + + t->zthr_thread = thread_create(NULL, 0, zthr_procedure, t, + 0, &p0, TS_RUN, minclsyspri); + mutex_exit(&t->zthr_lock); + + return (t); +} + +void +zthr_destroy(zthr_t *t) +{ + VERIFY3P(t->zthr_thread, ==, NULL); + mutex_destroy(&t->zthr_lock); + cv_destroy(&t->zthr_cv); + kmem_free(t, sizeof (*t)); +} + +/* + * Note: If the zthr is not sleeping and misses the wakeup + * (e.g it is running its ZTHR function), it will check if + * there is work to do before going to sleep using its checker + * function [see ZTHR state transition in ZTHR block comment]. + * Thus, missing the wakeup still yields the expected behavior. + */ +void +zthr_wakeup(zthr_t *t) +{ + ASSERT3P(t->zthr_thread, !=, NULL); + + mutex_enter(&t->zthr_lock); + cv_broadcast(&t->zthr_cv); + mutex_exit(&t->zthr_lock); +} + +/* + * Note: If the zthr is not running (e.g. has been cancelled + * already), this is a no-op. + */ +int +zthr_cancel(zthr_t *t) +{ + int rc = 0; + + mutex_enter(&t->zthr_lock); + + /* broadcast in case the zthr is sleeping */ + cv_broadcast(&t->zthr_cv); + + t->zthr_cancel = B_TRUE; + while (t->zthr_thread != NULL) + cv_wait(&t->zthr_cv, &t->zthr_lock); + t->zthr_cancel = B_FALSE; + rc = t->zthr_rc; + mutex_exit(&t->zthr_lock); + + return (rc); +} + +void +zthr_resume(zthr_t *t) +{ + ASSERT3P(t->zthr_thread, ==, NULL); + + mutex_enter(&t->zthr_lock); + + ASSERT3P(&t->zthr_checkfunc, !=, NULL); + ASSERT3P(&t->zthr_func, !=, NULL); + ASSERT(!t->zthr_cancel); + + t->zthr_thread = thread_create(NULL, 0, zthr_procedure, t, + 0, &p0, TS_RUN, minclsyspri); + + mutex_exit(&t->zthr_lock); +} + +/* + * This function is intended to be used by the zthr itself + * to check if another thread has signal it to stop running. + * + * returns TRUE if we are in the middle of trying to cancel + * this thread. + * + * returns FALSE otherwise. + */ +boolean_t +zthr_iscancelled(zthr_t *t) +{ + boolean_t cancelled; + + ASSERT3P(t->zthr_thread, ==, curthread); + + mutex_enter(&t->zthr_lock); + cancelled = t->zthr_cancel; + mutex_exit(&t->zthr_lock); + + return (cancelled); +} + +boolean_t +zthr_isrunning(zthr_t *t) +{ + boolean_t running; + + mutex_enter(&t->zthr_lock); + running = (t->zthr_thread != NULL); + mutex_exit(&t->zthr_lock); + + return (running); +} From owner-svn-src-vendor@freebsd.org Thu Feb 22 04:01:06 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D638F2534D; Thu, 22 Feb 2018 04:01:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1896168460; Thu, 22 Feb 2018 04:01:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1380122347; Thu, 22 Feb 2018 04:01:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M415fR016390; Thu, 22 Feb 2018 04:01:05 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M415FS016388; Thu, 22 Feb 2018 04:01:05 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220401.w1M415FS016388@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 04:01:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329807 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 329807 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 04:01:06 -0000 Author: mav Date: Thu Feb 22 04:01:05 2018 New Revision: 329807 URL: https://svnweb.freebsd.org/changeset/base/329807 Log: 8940 Sending an intra-pool resumable send stream may result in EXDEV illumos/illumos-gate@544132fce3fa6583f01318f9559adc46614343a7 "zfs send -t " for an incremental send should be able to resume successfully when sending to the same pool: a subtle issue in zfs_iter_children() doesn't currently allow this. Because resuming from a token requires "guid" -> "dataset" mapping (guid_to_name()), we have to walk the whole hierarchy to find the right snapshots to send. When resuming an incremental send both source and destination live in the same pool and have the same guid: this is where zfs_iter_children() gets confused and picks up the wrong snapshot, so we end up trying to send an incremental "destination@snap1 -> source@snap2" stream instead of "source@snap1 -> source@snap2": this fails with an "Invalid cross-device link" (EXDEV) error. Reviewed by: Paul Dagnelie Reviewed by: Matthew Ahrens Approved by: Hans Rosenfeld Author: loli10K Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_iter.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_iter.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_iter.c Thu Feb 22 03:55:02 2018 (r329806) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_iter.c Thu Feb 22 04:01:05 2018 (r329807) @@ -424,16 +424,20 @@ zfs_iter_snapspec(zfs_handle_t *fs_zhp, const char *sp /* * Iterate over all children, snapshots and filesystems + * Process snapshots before filesystems because they are nearer the input + * handle: this is extremely important when used with zfs_iter_f functions + * looking for data, following the logic that we would like to find it as soon + * and as close as possible. */ int zfs_iter_children(zfs_handle_t *zhp, zfs_iter_f func, void *data) { int ret; - if ((ret = zfs_iter_filesystems(zhp, func, data)) != 0) + if ((ret = zfs_iter_snapshots(zhp, B_FALSE, func, data)) != 0) return (ret); - return (zfs_iter_snapshots(zhp, B_FALSE, func, data)); + return (zfs_iter_filesystems(zhp, func, data)); } Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Thu Feb 22 03:55:02 2018 (r329806) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Thu Feb 22 04:01:05 2018 (r329807) @@ -1577,6 +1577,7 @@ zfs_send_resume(libzfs_handle_t *hdl, sendflags_t *fla int error = 0; char name[ZFS_MAX_DATASET_NAME_LEN]; enum lzc_send_flags lzc_flags = 0; + FILE *fout = (flags->verbose && flags->dryrun) ? stdout : stderr; (void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN, "cannot resume send")); @@ -1591,9 +1592,9 @@ zfs_send_resume(libzfs_handle_t *hdl, sendflags_t *fla return (zfs_error(hdl, EZFS_FAULT, errbuf)); } if (flags->verbose) { - (void) fprintf(stderr, dgettext(TEXT_DOMAIN, + (void) fprintf(fout, dgettext(TEXT_DOMAIN, "resume token contents:\n")); - nvlist_print(stderr, resume_nvl); + nvlist_print(fout, resume_nvl); } if (nvlist_lookup_string(resume_nvl, "toname", &toname) != 0 || @@ -1650,7 +1651,7 @@ zfs_send_resume(libzfs_handle_t *hdl, sendflags_t *fla lzc_flags, &size); if (error == 0) size = MAX(0, (int64_t)(size - bytes)); - send_print_verbose(stderr, zhp->zfs_name, fromname, + send_print_verbose(fout, zhp->zfs_name, fromname, size, flags->parsable); } From owner-svn-src-vendor@freebsd.org Thu Feb 22 03:22:28 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67757F21DA7; Thu, 22 Feb 2018 03:22:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18D228643C; Thu, 22 Feb 2018 03:22:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 13C2B21D3B; Thu, 22 Feb 2018 03:22:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M3MR8a098829; Thu, 22 Feb 2018 03:22:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M3MR7t098824; Thu, 22 Feb 2018 03:22:27 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220322.w1M3MR7t098824@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 03:22:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329799 - in vendor-sys/illumos/dist/uts/common: . fs/zfs fs/zfs/sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common: . fs/zfs fs/zfs/sys X-SVN-Commit-Revision: 329799 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 03:22:28 -0000 Author: mav Date: Thu Feb 22 03:22:27 2018 New Revision: 329799 URL: https://svnweb.freebsd.org/changeset/base/329799 Log: 9079 race condition in starting and ending condesing thread for indirect vdevs illumos/illumos-gate@667ec66f1b4f491d5e839644e0912cad1c9e7122 The timeline of the race condition is the following: [1] Thread A is about to finish condesing the first vdev in spa_condense_indirect_thread(), so it calls the spa_condense_indirect_complete_sync() sync task which sets the spa_condensing_indirect field to NULL. Waiting for the sync task to finish, thread A sleeps until the txg is done. When this happens, thread A will acquire spa_async_lock and set spa_condense_thread to NULL. [2] While thread A waits for the txg to finish, thread B which is running spa_sync() checks whether it should condense the second vdev in vdev_indirect_should_condense() by checking the spa_condensing_indirect field which was set to NULL by spa_condense_indirect_thread() from thread A. So it goes on and tries to spawn a new condensing thread in spa_condense_indirect_start_sync() and the aforementioned assertions fails because thread A has not set spa_condense_thread to NULL (which is basically the last thing it does before returning). The main issue here is that we rely on both spa_condensing_indirect and spa_condense_thread to signify whether a condensing thread is running. Ideally we would only use one throughout the codebase. In addition, for managing spa_condense_thread we currently use spa_async_lock which basically tights condensing to scrubing when it comes to pausing and resuming those actions during spa export. Reviewed by: Matt Ahrens Reviewed by: Pavel Zakharov Approved by: Hans Rosenfeld Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Modified: vendor-sys/illumos/dist/uts/common/Makefile.files ============================================================================== --- vendor-sys/illumos/dist/uts/common/Makefile.files Thu Feb 22 03:15:35 2018 (r329798) +++ vendor-sys/illumos/dist/uts/common/Makefile.files Thu Feb 22 03:22:27 2018 (r329799) @@ -1421,7 +1421,8 @@ ZFS_COMMON_OBJS += \ zio_compress.o \ zio_inject.o \ zle.o \ - zrlock.o + zrlock.o \ + zthr.o ZFS_SHARED_OBJS += \ zfeature_common.o \ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 03:15:35 2018 (r329798) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Thu Feb 22 03:22:27 2018 (r329799) @@ -1338,6 +1338,12 @@ spa_unload(spa_t *spa) spa->spa_vdev_removal = NULL; } + if (spa->spa_condense_zthr != NULL) { + ASSERT(!zthr_isrunning(spa->spa_condense_zthr)); + zthr_destroy(spa->spa_condense_zthr); + spa->spa_condense_zthr = NULL; + } + spa_condense_fini(spa); bpobj_close(&spa->spa_deferred_bpobj); @@ -2079,6 +2085,16 @@ spa_vdev_err(vdev_t *vdev, vdev_aux_t aux, int err) return (SET_ERROR(err)); } +static void +spa_spawn_aux_threads(spa_t *spa) +{ + ASSERT(spa_writeable(spa)); + + ASSERT(MUTEX_HELD(&spa_namespace_lock)); + + spa_start_indirect_condensing_thread(spa); +} + /* * Fix up config after a partly-completed split. This is done with the * ZPOOL_CONFIG_SPLIT nvlist. Both the splitting pool and the split-off @@ -3486,18 +3502,6 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char ASSERT(spa->spa_load_state != SPA_LOAD_TRYIMPORT); /* - * We must check this before we start the sync thread, because - * we only want to start a condense thread for condense - * operations that were in progress when the pool was - * imported. Once we start syncing, spa_sync() could - * initiate a condense (and start a thread for it). In - * that case it would be wrong to start a second - * condense thread. - */ - boolean_t condense_in_progress = - (spa->spa_condensing_indirect != NULL); - - /* * Traverse the ZIL and claim all blocks. */ spa_ld_claim_log_blocks(spa); @@ -3549,15 +3553,9 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char */ dsl_pool_clean_tmp_userrefs(spa->spa_dsl_pool); - /* - * Note: unlike condensing, we don't need an analogous - * "removal_in_progress" dance because no other thread - * can start a removal while we hold the spa_namespace_lock. - */ spa_restart_removal(spa); - if (condense_in_progress) - spa_condense_indirect_restart(spa); + spa_spawn_aux_threads(spa); } spa_load_note(spa, "LOADED"); @@ -4466,6 +4464,8 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_ */ txg_wait_synced(spa->spa_dsl_pool, txg); + spa_spawn_aux_threads(spa); + spa_write_cachefile(spa, B_FALSE, B_TRUE); spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_CREATE); @@ -6405,12 +6405,15 @@ spa_async_suspend(spa_t *spa) { mutex_enter(&spa->spa_async_lock); spa->spa_async_suspended++; - while (spa->spa_async_thread != NULL || - spa->spa_condense_thread != NULL) + while (spa->spa_async_thread != NULL) cv_wait(&spa->spa_async_cv, &spa->spa_async_lock); mutex_exit(&spa->spa_async_lock); spa_vdev_remove_suspend(spa); + + zthr_t *condense_thread = spa->spa_condense_zthr; + if (condense_thread != NULL && zthr_isrunning(condense_thread)) + VERIFY0(zthr_cancel(condense_thread)); } void @@ -6421,6 +6424,10 @@ spa_async_resume(spa_t *spa) spa->spa_async_suspended--; mutex_exit(&spa->spa_async_lock); spa_restart_removal(spa); + + zthr_t *condense_thread = spa->spa_condense_zthr; + if (condense_thread != NULL && !zthr_isrunning(condense_thread)) + zthr_resume(condense_thread); } static boolean_t Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Thu Feb 22 03:15:35 2018 (r329798) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Thu Feb 22 03:22:27 2018 (r329799) @@ -43,6 +43,7 @@ #include #include #include +#include #include #ifdef __cplusplus @@ -278,7 +279,7 @@ struct spa { spa_condensing_indirect_phys_t spa_condensing_indirect_phys; spa_condensing_indirect_t *spa_condensing_indirect; - kthread_t *spa_condense_thread; /* thread doing condense. */ + zthr_t *spa_condense_zthr; /* zthr doing condense. */ char *spa_root; /* alternate root directory */ uint64_t spa_ena; /* spa-wide ereport ENA */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Thu Feb 22 03:15:35 2018 (r329798) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Thu Feb 22 03:22:27 2018 (r329799) @@ -76,7 +76,7 @@ extern int spa_remove_init(spa_t *); extern void spa_restart_removal(spa_t *); extern int spa_condense_init(spa_t *); extern void spa_condense_fini(spa_t *); -extern void spa_condense_indirect_restart(spa_t *); +extern void spa_start_indirect_condensing_thread(spa_t *); extern void spa_vdev_condense_suspend(spa_t *); extern int spa_vdev_remove(spa_t *, uint64_t, boolean_t); extern void free_from_removing_vdev(vdev_t *, uint64_t, uint64_t, uint64_t); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Feb 22 03:15:35 2018 (r329798) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Feb 22 03:22:27 2018 (r329799) @@ -30,6 +30,8 @@ #include #include #include +#include +#include /* * An indirect vdev corresponds to a vdev that has been removed. Since @@ -475,7 +477,7 @@ spa_condense_indirect_commit_entry(spa_t *spa, static void spa_condense_indirect_generate_new_mapping(vdev_t *vd, - uint32_t *obsolete_counts, uint64_t start_index) + uint32_t *obsolete_counts, uint64_t start_index, zthr_t *zthr) { spa_t *spa = vd->vdev_spa; uint64_t mapi = start_index; @@ -490,7 +492,15 @@ spa_condense_indirect_generate_new_mapping(vdev_t *vd, (u_longlong_t)vd->vdev_id, (u_longlong_t)mapi); - while (mapi < old_num_entries && !spa_shutting_down(spa)) { + while (mapi < old_num_entries) { + + if (zthr_iscancelled(zthr)) { + zfs_dbgmsg("pausing condense of vdev %llu " + "at index %llu", (u_longlong_t)vd->vdev_id, + (u_longlong_t)mapi); + break; + } + vdev_indirect_mapping_entry_phys_t *entry = &old_mapping->vim_entries[mapi]; uint64_t entry_size = DVA_GET_ASIZE(&entry->vimep_dst); @@ -508,18 +518,30 @@ spa_condense_indirect_generate_new_mapping(vdev_t *vd, mapi++; } - if (spa_shutting_down(spa)) { - zfs_dbgmsg("pausing condense of vdev %llu at index %llu", - (u_longlong_t)vd->vdev_id, - (u_longlong_t)mapi); - } } -static void -spa_condense_indirect_thread(void *arg) +/* ARGSUSED */ +static boolean_t +spa_condense_indirect_thread_check(void *arg, zthr_t *zthr) { - vdev_t *vd = arg; - spa_t *spa = vd->vdev_spa; + spa_t *spa = arg; + + return (spa->spa_condensing_indirect != NULL); +} + +/* ARGSUSED */ +static int +spa_condense_indirect_thread(void *arg, zthr_t *zthr) +{ + spa_t *spa = arg; + vdev_t *vd; + + ASSERT3P(spa->spa_condensing_indirect, !=, NULL); + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + vd = vdev_lookup_top(spa, spa->spa_condensing_indirect_phys.scip_vdev); + ASSERT3P(vd, !=, NULL); + spa_config_exit(spa, SCL_VDEV, FTAG); + spa_condensing_indirect_t *sci = spa->spa_condensing_indirect; spa_condensing_indirect_phys_t *scip = &spa->spa_condensing_indirect_phys; @@ -593,25 +615,24 @@ spa_condense_indirect_thread(void *arg) } } - spa_condense_indirect_generate_new_mapping(vd, counts, start_index); + spa_condense_indirect_generate_new_mapping(vd, counts, + start_index, zthr); vdev_indirect_mapping_free_obsolete_counts(old_mapping, counts); /* - * We may have bailed early from generate_new_mapping(), if - * the spa is shutting down. In this case, do not complete - * the condense. + * If the zthr has received a cancellation signal while running + * in generate_new_mapping() or at any point after that, then bail + * early. We don't want to complete the condense if the spa is + * shutting down. */ - if (!spa_shutting_down(spa)) { - VERIFY0(dsl_sync_task(spa_name(spa), NULL, - spa_condense_indirect_complete_sync, sci, 0, - ZFS_SPACE_CHECK_NONE)); - } + if (zthr_iscancelled(zthr)) + return (0); - mutex_enter(&spa->spa_async_lock); - spa->spa_condense_thread = NULL; - cv_broadcast(&spa->spa_async_cv); - mutex_exit(&spa->spa_async_lock); + VERIFY0(dsl_sync_task(spa_name(spa), NULL, + spa_condense_indirect_complete_sync, sci, 0, ZFS_SPACE_CHECK_NONE)); + + return (0); } /* @@ -664,9 +685,7 @@ spa_condense_indirect_start_sync(vdev_t *vd, dmu_tx_t (u_longlong_t)scip->scip_prev_obsolete_sm_object, (u_longlong_t)scip->scip_next_mapping_object); - ASSERT3P(spa->spa_condense_thread, ==, NULL); - spa->spa_condense_thread = thread_create(NULL, 0, - spa_condense_indirect_thread, vd, 0, &p0, TS_RUN, minclsyspri); + zthr_wakeup(spa->spa_condense_zthr); } /* @@ -743,24 +762,12 @@ spa_condense_fini(spa_t *spa) } } -/* - * Restart the condense - called when the pool is opened. - */ void -spa_condense_indirect_restart(spa_t *spa) +spa_start_indirect_condensing_thread(spa_t *spa) { - vdev_t *vd; - ASSERT(spa->spa_condensing_indirect != NULL); - spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); - vd = vdev_lookup_top(spa, - spa->spa_condensing_indirect_phys.scip_vdev); - ASSERT(vd != NULL); - spa_config_exit(spa, SCL_VDEV, FTAG); - - ASSERT3P(spa->spa_condense_thread, ==, NULL); - spa->spa_condense_thread = thread_create(NULL, 0, - spa_condense_indirect_thread, vd, 0, &p0, TS_RUN, - minclsyspri); + ASSERT3P(spa->spa_condense_zthr, ==, NULL); + spa->spa_condense_zthr = zthr_create(spa_condense_indirect_thread_check, + spa_condense_indirect_thread, spa); } /* From owner-svn-src-vendor@freebsd.org Thu Feb 22 03:52:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F593F24434; Thu, 22 Feb 2018 03:52:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E5C587A7B; Thu, 22 Feb 2018 03:52:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3945E222B9; Thu, 22 Feb 2018 03:52:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M3q4Vs011960; Thu, 22 Feb 2018 03:52:04 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1M3q4kN011959; Thu, 22 Feb 2018 03:52:04 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201802220352.w1M3q4kN011959@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 22 Feb 2018 03:52:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329803 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 329803 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 03:52:04 -0000 Author: mav Date: Thu Feb 22 03:52:03 2018 New Revision: 329803 URL: https://svnweb.freebsd.org/changeset/base/329803 Log: 9080 recursive enter of vdev_indirect_rwlock from vdev_indirect_remap() illumos/illumos-gate@bdfded42e66b9fc1395ff2401aa2952f7c44ae34 A scenario came up where a callback executed by vdev_indirect_remap() on a vdev, calls vdev_indirect_remap() on the same vdev and tries to reacquire vdev_indirect_rwlock that was already acquired from the first call to vdev_indirect_remap(). The specific scenario, is that we want to remap a block pointer that is snapshoted but its dataset's remap_deadlist is not cached. So in order to add it we issue a read through a vdev_indirect_remap() on the same vdev, which brings up the aforementioned issue. Reviewed by: Matthew Ahrens Reviewed by: George Wilson Approved by: Hans Rosenfeld Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Feb 22 03:49:06 2018 (r329802) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Thu Feb 22 03:52:03 2018 (r329803) @@ -14,7 +14,7 @@ */ /* - * Copyright (c) 2014, 2015 by Delphix. All rights reserved. + * Copyright (c) 2014, 2017 by Delphix. All rights reserved. */ #include @@ -852,6 +852,57 @@ rs_alloc(vdev_t *vd, uint64_t offset, uint64_t asize, } /* + * Given an indirect vdev and an extent on that vdev, it duplicates the + * physical entries of the indirect mapping that correspond to the extent + * to a new array and returns a pointer to it. In addition, copied_entries + * is populated with the number of mapping entries that were duplicated. + * + * Note that the function assumes that the caller holds vdev_indirect_rwlock. + * This ensures that the mapping won't change due to condensing as we + * copy over its contents. + * + * Finally, since we are doing an allocation, it is up to the caller to + * free the array allocated in this function. + */ +vdev_indirect_mapping_entry_phys_t * +vdev_indirect_mapping_duplicate_adjacent_entries(vdev_t *vd, uint64_t offset, + uint64_t asize, uint64_t *copied_entries) +{ + vdev_indirect_mapping_entry_phys_t *duplicate_mappings = NULL; + vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; + uint64_t entries = 0; + + ASSERT(RW_READ_HELD(&vd->vdev_indirect_rwlock)); + + vdev_indirect_mapping_entry_phys_t *first_mapping = + vdev_indirect_mapping_entry_for_offset(vim, offset); + ASSERT3P(first_mapping, !=, NULL); + + vdev_indirect_mapping_entry_phys_t *m = first_mapping; + while (asize > 0) { + uint64_t size = DVA_GET_ASIZE(&m->vimep_dst); + + ASSERT3U(offset, >=, DVA_MAPPING_GET_SRC_OFFSET(m)); + ASSERT3U(offset, <, DVA_MAPPING_GET_SRC_OFFSET(m) + size); + + uint64_t inner_offset = offset - DVA_MAPPING_GET_SRC_OFFSET(m); + uint64_t inner_size = MIN(asize, size - inner_offset); + + offset += inner_size; + asize -= inner_size; + entries++; + m++; + } + + size_t copy_length = entries * sizeof (*first_mapping); + duplicate_mappings = kmem_alloc(copy_length, KM_SLEEP); + bcopy(first_mapping, duplicate_mappings, copy_length); + *copied_entries = entries; + + return (duplicate_mappings); +} + +/* * Goes through the relevant indirect mappings until it hits a concrete vdev * and issues the callback. On the way to the concrete vdev, if any other * indirect vdevs are encountered, then the callback will also be called on @@ -891,24 +942,42 @@ vdev_indirect_remap(vdev_t *vd, uint64_t offset, uint6 for (remap_segment_t *rs = rs_alloc(vd, offset, asize, 0); rs != NULL; rs = list_remove_head(&stack)) { vdev_t *v = rs->rs_vd; + uint64_t num_entries = 0; + ASSERT(spa_config_held(spa, SCL_ALL, RW_READER) != 0); + ASSERT(rs->rs_asize > 0); + /* - * Note: this can be called from open context - * (eg. zio_read()), so we need the rwlock to prevent - * the mapping from being changed by condensing. + * Note: As this function can be called from open context + * (e.g. zio_read()), we need the following rwlock to + * prevent the mapping from being changed by condensing. + * + * So we grab the lock and we make a copy of the entries + * that are relevant to the extent that we are working on. + * Once that is done, we drop the lock and iterate over + * our copy of the mapping. Once we are done with the with + * the remap segment and we free it, we also free our copy + * of the indirect mapping entries that are relevant to it. + * + * This way we don't need to wait until the function is + * finished with a segment, to condense it. In addition, we + * don't need a recursive rwlock for the case that a call to + * vdev_indirect_remap() needs to call itself (through the + * codepath of its callback) for the same vdev in the middle + * of its execution. */ rw_enter(&v->vdev_indirect_rwlock, RW_READER); vdev_indirect_mapping_t *vim = v->vdev_indirect_mapping; ASSERT3P(vim, !=, NULL); - ASSERT(spa_config_held(spa, SCL_ALL, RW_READER) != 0); - ASSERT(rs->rs_asize > 0); - vdev_indirect_mapping_entry_phys_t *mapping = - vdev_indirect_mapping_entry_for_offset(vim, rs->rs_offset); + vdev_indirect_mapping_duplicate_adjacent_entries(v, + rs->rs_offset, rs->rs_asize, &num_entries); ASSERT3P(mapping, !=, NULL); + ASSERT3U(num_entries, >, 0); + rw_exit(&v->vdev_indirect_rwlock); - while (rs->rs_asize > 0) { + for (uint64_t i = 0; i < num_entries; i++) { /* * Note: the vdev_indirect_mapping can not change * while we are running. It only changes while the @@ -917,20 +986,23 @@ vdev_indirect_remap(vdev_t *vd, uint64_t offset, uint6 * function is only called for frees, which also only * happen from syncing context. */ + vdev_indirect_mapping_entry_phys_t *m = &mapping[i]; - uint64_t size = DVA_GET_ASIZE(&mapping->vimep_dst); - uint64_t dst_offset = - DVA_GET_OFFSET(&mapping->vimep_dst); - uint64_t dst_vdev = DVA_GET_VDEV(&mapping->vimep_dst); + ASSERT3P(m, !=, NULL); + ASSERT3U(rs->rs_asize, >, 0); + uint64_t size = DVA_GET_ASIZE(&m->vimep_dst); + uint64_t dst_offset = DVA_GET_OFFSET(&m->vimep_dst); + uint64_t dst_vdev = DVA_GET_VDEV(&m->vimep_dst); + ASSERT3U(rs->rs_offset, >=, - DVA_MAPPING_GET_SRC_OFFSET(mapping)); + DVA_MAPPING_GET_SRC_OFFSET(m)); ASSERT3U(rs->rs_offset, <, - DVA_MAPPING_GET_SRC_OFFSET(mapping) + size); + DVA_MAPPING_GET_SRC_OFFSET(m) + size); ASSERT3U(dst_vdev, !=, v->vdev_id); uint64_t inner_offset = rs->rs_offset - - DVA_MAPPING_GET_SRC_OFFSET(mapping); + DVA_MAPPING_GET_SRC_OFFSET(m); uint64_t inner_size = MIN(rs->rs_asize, size - inner_offset); @@ -971,10 +1043,10 @@ vdev_indirect_remap(vdev_t *vd, uint64_t offset, uint6 rs->rs_offset += inner_size; rs->rs_asize -= inner_size; rs->rs_split_offset += inner_size; - mapping++; } + VERIFY0(rs->rs_asize); - rw_exit(&v->vdev_indirect_rwlock); + kmem_free(mapping, num_entries * sizeof (*mapping)); kmem_free(rs, sizeof (remap_segment_t)); } list_destroy(&stack); From owner-svn-src-vendor@freebsd.org Thu Feb 22 07:16:15 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 339ADF0A947; Thu, 22 Feb 2018 07:16:15 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D47C71556; Thu, 22 Feb 2018 07:16:14 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f44.google.com with SMTP id q69so5937710lfi.10; Wed, 21 Feb 2018 23:16:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=1okof+2PZjJH77r3RlIdwEB4e/dQGkeffkbuTkHWv1c=; b=fwyzx3z8ljpCEGQoCg4/IvEbyYZ7BiTIq9EFvu4DMQprNdbtgV/4PVhN3eZdvTZgvY 8p804sVdbyBJz2dU0FhTqmRDiXa1JqucFI9RmZRaCTFdydp9A0vM/olBOPDFVBjLWdfm 7tAHNUhgzN5M3wZOgIApISH+HcrDKi47WN/SvZ/98rs2pu2WVwag09VUIpd0pbi9qm80 yjJxuDhVK6nOJmZkBKWoIyAOyPf3ykNdN/7GZ7Y+2H5H6OfOBh32s1/Txb/vKFR2yPMN OBYHMqq/TgQ1s8XQLstzLVCTp3n5Q7H56BJEl37M+7lq3ucri2VVt2BXxEO0T1rVczSt tltg== X-Gm-Message-State: APf1xPA+8Ethe+kt85TFAYRdc72gw0tmHga5EaJOdmA11H7lLMXLtSre SPyOAushL4D5leKIEWF1+zP1LKvz X-Google-Smtp-Source: AH8x2269oKQ1MwFwdjhf6/gT+8nP2B6UVCFybjn1fk9et/mxI4MCUqFT6uFvm/Yr55PvnqZoMNtmag== X-Received: by 10.46.41.157 with SMTP id p29mr3933724ljp.137.1519283767324; Wed, 21 Feb 2018 23:16:07 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id c74sm1233978lfe.96.2018.02.21.23.16.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 23:16:06 -0800 (PST) Subject: Re: svn commit: r329795 - in vendor-sys/illumos/dist/uts/common: fs/zfs fs/zfs/sys sys/fs To: Alexander Motin , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org References: <201802220225.w1M2P9Un068350@repo.freebsd.org> From: Andriy Gapon Message-ID: <68dd6e31-9616-2190-544a-82b400d81370@FreeBSD.org> Date: Thu, 22 Feb 2018 09:16:05 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201802220225.w1M2P9Un068350@repo.freebsd.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 07:16:15 -0000 On 22/02/2018 04:25, Alexander Motin wrote: > Author: mav > Date: Thu Feb 22 02:25:09 2018 > New Revision: 329795 > URL: https://svnweb.freebsd.org/changeset/base/329795 > > Log: > r329793 | mav | 2018-02-22 04:21:03 +0200 (чт, 22 февр. 2018) | 58 lines > > 9075 Improve ZFS pool import/load process and corrupted pool recovery > > illumos/illumos-gate@6f7938128a2c5e23f4b970ea101137eadd1470a1 This commit message looks strange. -- Andriy Gapon From owner-svn-src-vendor@freebsd.org Thu Feb 22 06:57:10 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3333F08EA7; Thu, 22 Feb 2018 06:57:09 +0000 (UTC) (envelope-from tsoome@me.com) Received: from st13p35im-asmtp002.me.com (st13p35im-asmtp002.me.com [17.164.199.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EEBD702C7; Thu, 22 Feb 2018 06:57:09 +0000 (UTC) (envelope-from tsoome@me.com) Received: from process-dkim-sign-daemon.st13p35im-asmtp002.me.com by st13p35im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P4J00D00EYFNE00@st13p35im-asmtp002.me.com>; Thu, 22 Feb 2018 05:56:20 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=04042017; t=1519278980; bh=SFohAlVngPoNg11ANPeb5yykEhOc1jAensrflQ3beWY=; h=From:Message-id:Content-type:MIME-version:Subject:Date:To; b=F7O3M1jUG+lPmkG7l+8sUkT0cJHjcQkS5EZ5b/bKGvWR3wXFq5JrGRlZkQ+jgS+FJ qyl9fjo9TOCeDTsFOKtPnWO9mMg0DesuV3DIG9OmF6cMCtqgu0CISINgNverXYxzGY n/KjSeF6JEABgqdT3tcd6iAlX7ndzwQHHdbd5bYQAnXhP/6dbEBE4+tgVBkO2RlQXK pdqq+SJRP3fvAPeJEjx2yh52kip6RxwVTHfbjP9eCUJdWfMSnXUYp42HjsYLvop7XA JXkwTY4uDnZDNObGdS5Z+Vp4wWMWJ0H0Ar3N4p/fI6G835B9H8JyfiW5lh6jlV5ZfU lv+lPdPLj/FYQ== Received: from icloud.com ([127.0.0.1]) by st13p35im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P4J00OSCF5SDY10@st13p35im-asmtp002.me.com>; Thu, 22 Feb 2018 05:56:19 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-22_03:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1011 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1802220076 From: Toomas Soome Message-id: <14ECCB6A-7AEE-4831-AF14-6AAF79CF4102@me.com> MIME-version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: svn commit: r329770 - vendor-sys/illumos/dist/uts/common/fs/zfs/lua Date: Thu, 22 Feb 2018 07:56:16 +0200 In-reply-to: <56dda9e2-e9ea-83b9-1e54-6569d478615a@FreeBSD.org> Cc: rgrimes@freebsd.org, src-committers , svn-src-all@freebsd.org, svn-src-vendor@freebsd.org To: Alexander Motin References: <201802220104.w1M14uHQ088614@pdx.rh.CN85.dnsmgr.net> <56dda9e2-e9ea-83b9-1e54-6569d478615a@FreeBSD.org> X-Mailer: Apple Mail (2.3445.5.20) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 06:57:10 -0000 > On 22 Feb 2018, at 05:03, Alexander Motin wrote: >=20 > On 21.02.2018 20:04, Rodney W. Grimes wrote: >> [ Charset UTF-8 unsupported, converting... ] >>> Author: mav >>> Date: Thu Feb 22 00:46:24 2018 >>> New Revision: 329770 >>> URL: https://svnweb.freebsd.org/changeset/base/329770 >>>=20 >>> Log: >>> 9035 zfs: this statement may fall through >>>=20 >>> illumos/illumos-gate@46ac8fdfc5a1f9d8240c79a6ae5b2889cbe83553 >>>=20 >>> Reviewed by: Yuri Pankov >>> Reviewed by: Andy Fiddaman >>> Reviewed by: Matthew Ahrens >>> Approved by: Dan McDonald >>> Author: Toomas Soome >>>=20 >>> Modified: >>> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c >>> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c >>> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c >>> vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c >>>=20 >>> Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c >>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D >>> --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb = 22 00:42:12 2018 (r329769) >>> +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c Thu Feb = 22 00:46:24 2018 (r329770) >>> @@ -677,7 +677,7 @@ static void freeobj (lua_State *L, GCObject *o) = { >>> case LUA_TUSERDATA: luaM_freemem(L, o, sizeudata(gco2u(o))); = break; >>> case LUA_TSHRSTR: >>> G(L)->strt.nuse--; >>> - /* go through */ >>> + /* FALLTHROUGH */ >>> case LUA_TLNGSTR: { >>> luaM_freemem(L, o, sizestring(gco2ts(o))); >>> break; >>>=20 >>> Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c >>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D >>> --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb = 22 00:42:12 2018 (r329769) >>> +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c Thu Feb = 22 00:46:24 2018 (r329770) >>> @@ -475,6 +475,7 @@ static int llex (LexState *ls, SemInfo *seminfo) = { >>> else if (!lisdigit(ls->current)) return '.'; >>> /* else go through */ >> ^^^^^^^^^^^^^^^^^^^^^^^ Wasnt this already marked, perhaps = in the >> wrong place with the wrong string? > As I understand GCC 7 parses comments like that to decide whether > generate warnings or not, but this informal text is out of his > understanding. >=20 >=20 Exactly. Also in some cases it is desirable to have more text than just = FALLTHROUGH, And in some cases we may just have missed the original = comment which should be removed:) rgds, toomas From owner-svn-src-vendor@freebsd.org Thu Feb 22 15:16:28 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03A09F0DE06; Thu, 22 Feb 2018 15:16:28 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yb0-x230.google.com (mail-yb0-x230.google.com [IPv6:2607:f8b0:4002:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 86D92683C1; Thu, 22 Feb 2018 15:16:27 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-yb0-x230.google.com with SMTP id z90-v6so1825066ybh.11; Thu, 22 Feb 2018 07:16:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=vBy4nccERfeaVPANgnaPwQuFqw38H3iVW5T3oGgre9I=; b=UyllqCL6lrJfjc6AEIrTyJhyJbjudaLSy/F8IZWK8yjtb3ALkQem9S9jZUBWO3yXLN Cs1kxMtVOZlsAnwn2GwIpTJI9HmX/bokrxnC17zvRsSVSDdy18b04wTctcU4IU5aL94l RtCHMzWzDEDkHLM6zaK5H/Lux6D/uv0WXEAJBazzqP67SZisTJfAyRmJeRNZ6TmAe/6t H6e+dfm9svpXNtqDLun85QEnzjdmAavF5vV8lFOLLSllmgHJ6bO+oIDU9GvC+1MxNVmG 5kqidU602pVZo4p2TncZ/6vAUK3Nd2NfPe9Qb4+MBgkrYfZadNjiRCZ3HrZOP+nm13sO /LoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=vBy4nccERfeaVPANgnaPwQuFqw38H3iVW5T3oGgre9I=; b=b/uoNz3qFee4s/fdK5tj4l3kAg7TbzbdJh7bVeU5Sibmc0HfITEJMjj9HlP/l3SmSB xICr/Se53YarnbSDx1CZIV+sPGUSc1ypX7F3CMJRJCkLDdrrUqUfVLdhuPQIgLH/0ubS T51uNzqf/Zz9AZflyiIfLcN12Frw0X0oGn8UdHhxK0lADPkx2ZT58LrS28YFZH3OtTFX cHPgjzb6o0BzGsim+U6PSQIUQdHRcxwq3PIQcs7XiF8rWKE2yd2z+/nqNvYHaCIcv75Y 7rFo/z4Kf1OF16iOf2KXcImeK07HYz99GVsSFCaw4IWyV/PDXdQd5j4wjQbXYUUWoiTN g66w== X-Gm-Message-State: APf1xPCZGPwi9glVJb86pVAwbaWj6dWRKM66wn9SssdQiYBxKy9S6tsJ q8NHzAi99CAYDhtJetwJYyLqxB4N X-Google-Smtp-Source: AH8x226y3GJoM7ITfVWSVyxvp3N5S4cdVNxx6NT61B4NMyx/zfshcMVtjqEnQuFwsdoXrb6wcJIPGA== X-Received: by 2002:a25:1907:: with SMTP id 7-v6mr4909911ybz.129.1519312586761; Thu, 22 Feb 2018 07:16:26 -0800 (PST) Received: from mavoffice.ixsystems.com ([12.164.17.129]) by smtp.gmail.com with ESMTPSA id c207sm111861ywa.96.2018.02.22.07.16.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Feb 2018 07:16:26 -0800 (PST) Sender: Alexander Motin Subject: Re: svn commit: r329795 - in vendor-sys/illumos/dist/uts/common: fs/zfs fs/zfs/sys sys/fs To: Andriy Gapon , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org References: <201802220225.w1M2P9Un068350@repo.freebsd.org> <68dd6e31-9616-2190-544a-82b400d81370@FreeBSD.org> From: Alexander Motin Message-ID: <796c1f47-76e3-ac2d-ba32-e5544c1e40a5@FreeBSD.org> Date: Thu, 22 Feb 2018 10:16:24 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <68dd6e31-9616-2190-544a-82b400d81370@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 15:16:28 -0000 On 22.02.2018 02:16, Andriy Gapon wrote: > On 22/02/2018 04:25, Alexander Motin wrote: >> Author: mav >> Date: Thu Feb 22 02:25:09 2018 >> New Revision: 329795 >> URL: https://svnweb.freebsd.org/changeset/base/329795 >> >> Log: >> r329793 | mav | 2018-02-22 04:21:03 +0200 (чт, 22 февр. 2018) | 58 lines >> >> 9075 Improve ZFS pool import/load process and corrupted pool recovery >> >> illumos/illumos-gate@6f7938128a2c5e23f4b970ea101137eadd1470a1 > > This commit message looks strange. It does. I'm sorry. r329793 and r329795 were supposed to be a single commit into two vendor repos, but then I made mistake, committing to only one of them, and then made one more in copy/paste while fixing the first. Into the head the change went properly as one commit. -- Alexander Motin From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:33 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAE92F23E60; Sat, 24 Feb 2018 21:27:32 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E65E77AAE; Sat, 24 Feb 2018 21:27:32 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 791202B05E; Sat, 24 Feb 2018 21:27:32 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRWGD008871; Sat, 24 Feb 2018 21:27:32 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRUsV008854; Sat, 24 Feb 2018 21:27:30 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRUsV008854@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329931 - in vendor/llvm/dist-release_60: docs include/llvm/Bitcode include/llvm/MC include/llvm/Transforms/Utils lib/Analysis lib/Bitcode/Reader lib/Bitcode/Writer lib/Support lib/Targ... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/llvm/dist-release_60: docs include/llvm/Bitcode include/llvm/MC include/llvm/Transforms/Utils lib/Analysis lib/Bitcode/Reader lib/Bitcode/Writer lib/Support lib/Target/AArch64 lib/Target/AMD... X-SVN-Commit-Revision: 329931 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:33 -0000 Author: dim Date: Sat Feb 24 21:27:30 2018 New Revision: 329931 URL: https://svnweb.freebsd.org/changeset/base/329931 Log: Vendor import of llvm release_60 branch r325932: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325932 Added: vendor/llvm/dist-release_60/test/CodeGen/AArch64/GlobalISel/fp16-copy-gpr.mir vendor/llvm/dist-release_60/test/CodeGen/PowerPC/pr36292.ll vendor/llvm/dist-release_60/test/Transforms/InstCombine/pr36362.ll Modified: vendor/llvm/dist-release_60/docs/ReleaseNotes.rst vendor/llvm/dist-release_60/docs/index.rst vendor/llvm/dist-release_60/include/llvm/Bitcode/LLVMBitCodes.h vendor/llvm/dist-release_60/include/llvm/MC/MCAsmMacro.h vendor/llvm/dist-release_60/include/llvm/Transforms/Utils/LoopUtils.h vendor/llvm/dist-release_60/lib/Analysis/ScalarEvolution.cpp vendor/llvm/dist-release_60/lib/Bitcode/Reader/BitcodeReader.cpp vendor/llvm/dist-release_60/lib/Bitcode/Writer/BitcodeWriter.cpp vendor/llvm/dist-release_60/lib/Support/CMakeLists.txt vendor/llvm/dist-release_60/lib/Target/AArch64/AArch64InstructionSelector.cpp vendor/llvm/dist-release_60/lib/Target/AMDGPU/SIInstrInfo.cpp vendor/llvm/dist-release_60/lib/Target/PowerPC/PPCCTRLoops.cpp vendor/llvm/dist-release_60/lib/Target/X86/X86.td vendor/llvm/dist-release_60/lib/Transforms/InstCombine/InstCombineSelect.cpp vendor/llvm/dist-release_60/lib/Transforms/Scalar/LICM.cpp vendor/llvm/dist-release_60/lib/Transforms/Utils/LoopUtils.cpp vendor/llvm/dist-release_60/lib/Transforms/Vectorize/LoopVectorize.cpp vendor/llvm/dist-release_60/test/Bitcode/compatibility-3.6.ll vendor/llvm/dist-release_60/test/Bitcode/compatibility-3.7.ll vendor/llvm/dist-release_60/test/Bitcode/compatibility-3.8.ll vendor/llvm/dist-release_60/test/Bitcode/compatibility-3.9.ll vendor/llvm/dist-release_60/test/Bitcode/compatibility-4.0.ll vendor/llvm/dist-release_60/test/Bitcode/compatibility-5.0.ll vendor/llvm/dist-release_60/test/CodeGen/AArch64/GlobalISel/select-insert-extract.mir vendor/llvm/dist-release_60/test/CodeGen/AMDGPU/smrd.ll vendor/llvm/dist-release_60/test/CodeGen/X86/clwb.ll vendor/llvm/dist-release_60/test/Transforms/LICM/sinking.ll vendor/llvm/dist-release_60/test/Transforms/LoopVectorize/pr30654-phiscev-sext-trunc.ll vendor/llvm/dist-release_60/test/Transforms/LoopVectorize/pr35773.ll vendor/llvm/dist-release_60/test/Transforms/LoopVectorize/reduction-small-size.ll vendor/llvm/dist-release_60/test/Transforms/LoopVectorize/vect-phiscev-sext-trunc.ll vendor/llvm/dist-release_60/test/tools/llvm-config/system-libs.windows.test Modified: vendor/llvm/dist-release_60/docs/ReleaseNotes.rst ============================================================================== --- vendor/llvm/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:27:30 2018 (r329931) @@ -5,12 +5,6 @@ LLVM 6.0.0 Release Notes .. contents:: :local: -.. warning:: - These are in-progress notes for the upcoming LLVM 6 release. - Release notes for previous releases can be found on - `the Download Page `_. - - Introduction ============ @@ -26,11 +20,6 @@ have questions or comments, the `LLVM Developer's Mail `_ is a good place to send them. -Note that if you are reading this file from a Subversion checkout or the main -LLVM web page, this document applies to the *next* release, not the current -one. To see the release notes for a specific release, please see the `releases -page `_. - Non-comprehensive list of changes in this release ================================================= .. NOTE @@ -56,6 +45,9 @@ Non-comprehensive list of changes in this release * Significantly improved quality of CodeView debug info for Windows. +* Preliminary support for Sanitizers and sibling features on X86(_64) NetBSD + (ASan, UBsan, TSan, MSan, SafeStack, libFuzzer). + * Note.. .. NOTE @@ -71,6 +63,15 @@ Non-comprehensive list of changes in this release Changes to the LLVM IR ---------------------- +* The fast-math-flags (FMF) have been updated. Previously, the 'fast' flag + indicated that floating-point reassociation was allowed and all other flags + were set too. The 'fast' flag still exists, but there is a new flag called + 'reassoc' to indicate specifically that reassociation is allowed. A new bit + called 'afn' was also added to selectively allow approximations for common + mathlib functions like square-root. The new flags provide more flexibility + to enable/disable specific floating-point optimizations. Making the + optimizer respond appropriately to these flags is an ongoing effort. + Changes to the AArch64 Target ----------------------------- @@ -112,9 +113,45 @@ Changes to the Hexagon Target Changes to the MIPS Target -------------------------- - During this release ... +Fixed numerous bugs: +* fpowi on MIPS64 giving incorrect results when used with a negative integer. +* Usage of the asm 'c' constraint with the wrong datatype causing an + assert/crash. +* Fixed a conversion bug when using the DSP ASE. +* Fixed an inconsistency where objects were not marked as using the microMIPS as + when the micromips function attribute or the ".set micromips" directive was + used. +* Reordered the MIPSR6 specific hazard scheduler pass to after the delay slot + filler, fixing a class of rare edge case bugs where the delay slot filler + would violate ISA restrictions. +* Fixed a crash when using a type of unknown size with gp relative addressing. +* Corrected the j macro for microMIPS. +* Corrected the encoding of movep for microMIPS32r6. +* Fixed an issue with the usage of insert instructions having an invalid set of + operands. +* Fixed an issue where TLS symbols where not marked as such. +* Enabled the usage of register scavanging with MSA, due to its' shorter offsets + for loads and stores. +* Corrected the ELF headers when using the DSP ASE. +New features: + +* The long branch pass now generates some R6 specific instructions when + targeting MIPSR6. +* The delay slot filler now performs more branch conversions if delay slots + cannot be filled. +* The MIPS MT ASE is now fully supported. +* Added support for the ``lapc`` pseudo instruction. +* Improved the selection of multiple instructions (``dext``, ``nmadd``, + ``nmsub``). +* Further improved microMIPS codesize reduction. + +Deprecation notices: + +* microMIPS64R6 support was been deprecated since 5.0, and has now been + completely removed. + Changes to the PowerPC Target ----------------------------- @@ -132,10 +169,42 @@ During this release the SystemZ target has: Changes to the X86 Target ------------------------- -During this release ... +During this release the X86 target has: -* Got support for enabling SjLj exception handling on platforms where it +* Added support for enabling SjLj exception handling on platforms where it isn't the default. + +* Added intrinsics for Intel Extensions: VAES, GFNI, VPCLMULQDQ, AVX512VBMI2, AVX512BITALG, AVX512VNNI. + +* Added support for Intel Icelake CPU. + +* Fixed some X87 codegen bugs. + +* Added instruction scheduling information for Intel Sandy Bridge, Ivy Bridge, Haswell, Broadwell, and Skylake CPUs. + +* Improved scheduler model for AMD Jaguar CPUs. + +* Improved llvm-mc's disassembler for some EVEX encoded instructions. + +* Add support for i8 and i16 vector signed/unsigned min/max horizontal reductions. + +* Improved codegen for memory comparisons + +* Improved codegen for i32 vector multiplies + +* Improved codegen for scalar integer absolute values + +* Improved codegen for vector integer rotations (XOP and AVX512) + +* Improved codegen of data being transferred between GPRs and K-registers. + +* Improved codegen for vector truncations. + +* Improved folding of address computations into gather/scatter instructions. + +* Gained initial support recognizing variable shuffles from vector element extracts and inserts. + +* Improved documentation for SSE/AVX intrinsics in *intrin.h header files. Changes to the AMDGPU Target ----------------------------- Modified: vendor/llvm/dist-release_60/docs/index.rst ============================================================================== --- vendor/llvm/dist-release_60/docs/index.rst Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/docs/index.rst Sat Feb 24 21:27:30 2018 (r329931) @@ -1,11 +1,6 @@ Overview ======== -.. warning:: - - If you are using a released version of LLVM, see `the download page - `_ to find your documentation. - The LLVM compiler infrastructure supports a wide range of projects, from industrial strength compilers to specialized JIT applications to small research projects. Modified: vendor/llvm/dist-release_60/include/llvm/Bitcode/LLVMBitCodes.h ============================================================================== --- vendor/llvm/dist-release_60/include/llvm/Bitcode/LLVMBitCodes.h Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/include/llvm/Bitcode/LLVMBitCodes.h Sat Feb 24 21:27:30 2018 (r329931) @@ -395,6 +395,20 @@ enum OverflowingBinaryOperatorOptionalFlags { OBO_NO_SIGNED_WRAP = 1 }; +/// FastMath Flags +/// This is a fixed layout derived from the bitcode emitted by LLVM 5.0 +/// intended to decouple the in-memory representation from the serialization. +enum FastMathMap { + UnsafeAlgebra = (1 << 0), // Legacy + NoNaNs = (1 << 1), + NoInfs = (1 << 2), + NoSignedZeros = (1 << 3), + AllowReciprocal = (1 << 4), + AllowContract = (1 << 5), + ApproxFunc = (1 << 6), + AllowReassoc = (1 << 7) +}; + /// PossiblyExactOperatorOptionalFlags - Flags for serializing /// PossiblyExactOperator's SubclassOptionalData contents. enum PossiblyExactOperatorOptionalFlags { PEO_EXACT = 0 }; Modified: vendor/llvm/dist-release_60/include/llvm/MC/MCAsmMacro.h ============================================================================== --- vendor/llvm/dist-release_60/include/llvm/MC/MCAsmMacro.h Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/include/llvm/MC/MCAsmMacro.h Sat Feb 24 21:27:30 2018 (r329931) @@ -33,6 +33,6 @@ struct MCAsmMacro { (public) MCAsmMacro(StringRef N, StringRef B, MCAsmMacroParameters P) : Name(N), Body(B), Parameters(std::move(P)) {} }; -}; // namespace llvm +} // namespace llvm #endif Modified: vendor/llvm/dist-release_60/include/llvm/Transforms/Utils/LoopUtils.h ============================================================================== --- vendor/llvm/dist-release_60/include/llvm/Transforms/Utils/LoopUtils.h Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/include/llvm/Transforms/Utils/LoopUtils.h Sat Feb 24 21:27:30 2018 (r329931) @@ -21,6 +21,7 @@ #include "llvm/ADT/SmallVector.h" #include "llvm/ADT/StringRef.h" #include "llvm/Analysis/AliasAnalysis.h" +#include "llvm/Analysis/DemandedBits.h" #include "llvm/Analysis/EHPersonalities.h" #include "llvm/Analysis/TargetTransformInfo.h" #include "llvm/IR/Dominators.h" @@ -172,15 +173,25 @@ class RecurrenceDescriptor { (public) Value *Left, Value *Right); /// Returns true if Phi is a reduction of type Kind and adds it to the - /// RecurrenceDescriptor. + /// RecurrenceDescriptor. If either \p DB is non-null or \p AC and \p DT are + /// non-null, the minimal bit width needed to compute the reduction will be + /// computed. static bool AddReductionVar(PHINode *Phi, RecurrenceKind Kind, Loop *TheLoop, bool HasFunNoNaNAttr, - RecurrenceDescriptor &RedDes); + RecurrenceDescriptor &RedDes, + DemandedBits *DB = nullptr, + AssumptionCache *AC = nullptr, + DominatorTree *DT = nullptr); - /// Returns true if Phi is a reduction in TheLoop. The RecurrenceDescriptor is - /// returned in RedDes. + /// Returns true if Phi is a reduction in TheLoop. The RecurrenceDescriptor + /// is returned in RedDes. If either \p DB is non-null or \p AC and \p DT are + /// non-null, the minimal bit width needed to compute the reduction will be + /// computed. static bool isReductionPHI(PHINode *Phi, Loop *TheLoop, - RecurrenceDescriptor &RedDes); + RecurrenceDescriptor &RedDes, + DemandedBits *DB = nullptr, + AssumptionCache *AC = nullptr, + DominatorTree *DT = nullptr); /// Returns true if Phi is a first-order recurrence. A first-order recurrence /// is a non-reduction recurrence relation in which the value of the @@ -217,24 +228,6 @@ class RecurrenceDescriptor { (public) /// Returns true if the recurrence kind is an arithmetic kind. static bool isArithmeticRecurrenceKind(RecurrenceKind Kind); - - /// Determines if Phi may have been type-promoted. If Phi has a single user - /// that ANDs the Phi with a type mask, return the user. RT is updated to - /// account for the narrower bit width represented by the mask, and the AND - /// instruction is added to CI. - static Instruction *lookThroughAnd(PHINode *Phi, Type *&RT, - SmallPtrSetImpl &Visited, - SmallPtrSetImpl &CI); - - /// Returns true if all the source operands of a recurrence are either - /// SExtInsts or ZExtInsts. This function is intended to be used with - /// lookThroughAnd to determine if the recurrence has been type-promoted. The - /// source operands are added to CI, and IsSigned is updated to indicate if - /// all source operands are SExtInsts. - static bool getSourceExtensionKind(Instruction *Start, Instruction *Exit, - Type *RT, bool &IsSigned, - SmallPtrSetImpl &Visited, - SmallPtrSetImpl &CI); /// Returns the type of the recurrence. This type can be narrower than the /// actual type of the Phi if the recurrence has been type-promoted. Modified: vendor/llvm/dist-release_60/lib/Analysis/ScalarEvolution.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Analysis/ScalarEvolution.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Analysis/ScalarEvolution.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -205,6 +205,11 @@ static cl::opt cl::desc("Max coefficients in AddRec during evolving"), cl::init(16)); +static cl::opt VersionUnknown( + "scev-version-unknown", cl::Hidden, + cl::desc("Use predicated scalar evolution to version SCEVUnknowns"), + cl::init(false)); + //===----------------------------------------------------------------------===// // SCEV class definitions //===----------------------------------------------------------------------===// @@ -11467,6 +11472,8 @@ class SCEVPredicateRewriter : public SCEVRewriteVisito // couldn't create an AddRec for it, or couldn't add the predicate), we just // return \p Expr. const SCEV *convertToAddRecWithPreds(const SCEVUnknown *Expr) { + if (!VersionUnknown) + return Expr; if (!isa(Expr->getValue())) return Expr; Optional>> Modified: vendor/llvm/dist-release_60/lib/Bitcode/Reader/BitcodeReader.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Bitcode/Reader/BitcodeReader.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Bitcode/Reader/BitcodeReader.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -1046,19 +1046,21 @@ static Comdat::SelectionKind getDecodedComdatSelection static FastMathFlags getDecodedFastMathFlags(unsigned Val) { FastMathFlags FMF; - if (0 != (Val & FastMathFlags::AllowReassoc)) + if (0 != (Val & bitc::UnsafeAlgebra)) + FMF.setFast(); + if (0 != (Val & bitc::AllowReassoc)) FMF.setAllowReassoc(); - if (0 != (Val & FastMathFlags::NoNaNs)) + if (0 != (Val & bitc::NoNaNs)) FMF.setNoNaNs(); - if (0 != (Val & FastMathFlags::NoInfs)) + if (0 != (Val & bitc::NoInfs)) FMF.setNoInfs(); - if (0 != (Val & FastMathFlags::NoSignedZeros)) + if (0 != (Val & bitc::NoSignedZeros)) FMF.setNoSignedZeros(); - if (0 != (Val & FastMathFlags::AllowReciprocal)) + if (0 != (Val & bitc::AllowReciprocal)) FMF.setAllowReciprocal(); - if (0 != (Val & FastMathFlags::AllowContract)) + if (0 != (Val & bitc::AllowContract)) FMF.setAllowContract(true); - if (0 != (Val & FastMathFlags::ApproxFunc)) + if (0 != (Val & bitc::ApproxFunc)) FMF.setApproxFunc(); return FMF; } Modified: vendor/llvm/dist-release_60/lib/Bitcode/Writer/BitcodeWriter.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Bitcode/Writer/BitcodeWriter.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Bitcode/Writer/BitcodeWriter.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -1330,19 +1330,19 @@ static uint64_t getOptimizationFlags(const Value *V) { Flags |= 1 << bitc::PEO_EXACT; } else if (const auto *FPMO = dyn_cast(V)) { if (FPMO->hasAllowReassoc()) - Flags |= FastMathFlags::AllowReassoc; + Flags |= bitc::AllowReassoc; if (FPMO->hasNoNaNs()) - Flags |= FastMathFlags::NoNaNs; + Flags |= bitc::NoNaNs; if (FPMO->hasNoInfs()) - Flags |= FastMathFlags::NoInfs; + Flags |= bitc::NoInfs; if (FPMO->hasNoSignedZeros()) - Flags |= FastMathFlags::NoSignedZeros; + Flags |= bitc::NoSignedZeros; if (FPMO->hasAllowReciprocal()) - Flags |= FastMathFlags::AllowReciprocal; + Flags |= bitc::AllowReciprocal; if (FPMO->hasAllowContract()) - Flags |= FastMathFlags::AllowContract; + Flags |= bitc::AllowContract; if (FPMO->hasApproxFunc()) - Flags |= FastMathFlags::ApproxFunc; + Flags |= bitc::ApproxFunc; } return Flags; @@ -3183,7 +3183,7 @@ void ModuleBitcodeWriter::writeBlockInfo() { Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // LHS Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // RHS Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 4)); // opc - Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 7)); // flags + Abbv->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 8)); // flags if (Stream.EmitBlockInfoAbbrev(bitc::FUNCTION_BLOCK_ID, Abbv) != FUNCTION_INST_BINOP_FLAGS_ABBREV) llvm_unreachable("Unexpected abbrev ordering!"); Modified: vendor/llvm/dist-release_60/lib/Support/CMakeLists.txt ============================================================================== --- vendor/llvm/dist-release_60/lib/Support/CMakeLists.txt Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Support/CMakeLists.txt Sat Feb 24 21:27:30 2018 (r329931) @@ -4,7 +4,8 @@ if ( LLVM_ENABLE_ZLIB AND HAVE_LIBZ ) endif() if( MSVC OR MINGW ) # libuuid required for FOLDERID_Profile usage in lib/Support/Windows/Path.inc. - set(system_libs ${system_libs} psapi shell32 ole32 uuid) + # advapi32 required for CryptAcquireContextW in lib/Support/Windows/Path.inc. + set(system_libs ${system_libs} psapi shell32 ole32 uuid advapi32) elseif( CMAKE_HOST_UNIX ) if( HAVE_LIBRT ) set(system_libs ${system_libs} rt) Modified: vendor/llvm/dist-release_60/lib/Target/AArch64/AArch64InstructionSelector.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Target/AArch64/AArch64InstructionSelector.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Target/AArch64/AArch64InstructionSelector.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -133,16 +133,21 @@ AArch64InstructionSelector::AArch64InstructionSelector // for each class in the bank. static const TargetRegisterClass * getRegClassForTypeOnBank(LLT Ty, const RegisterBank &RB, - const RegisterBankInfo &RBI) { + const RegisterBankInfo &RBI, + bool GetAllRegSet = false) { if (RB.getID() == AArch64::GPRRegBankID) { if (Ty.getSizeInBits() <= 32) - return &AArch64::GPR32RegClass; + return GetAllRegSet ? &AArch64::GPR32allRegClass + : &AArch64::GPR32RegClass; if (Ty.getSizeInBits() == 64) - return &AArch64::GPR64RegClass; + return GetAllRegSet ? &AArch64::GPR64allRegClass + : &AArch64::GPR64RegClass; return nullptr; } if (RB.getID() == AArch64::FPRRegBankID) { + if (Ty.getSizeInBits() <= 16) + return &AArch64::FPR16RegClass; if (Ty.getSizeInBits() == 32) return &AArch64::FPR32RegClass; if (Ty.getSizeInBits() == 64) @@ -310,19 +315,46 @@ static unsigned selectLoadStoreUIOp(unsigned GenericOp return GenericOpc; } +static bool selectFP16CopyFromGPR32(MachineInstr &I, const TargetInstrInfo &TII, + MachineRegisterInfo &MRI, unsigned SrcReg) { + // Copies from gpr32 to fpr16 need to use a sub-register copy. + unsigned CopyReg = MRI.createVirtualRegister(&AArch64::FPR32RegClass); + BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(AArch64::COPY)) + .addDef(CopyReg) + .addUse(SrcReg); + unsigned SubRegCopy = MRI.createVirtualRegister(&AArch64::FPR16RegClass); + BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(TargetOpcode::COPY)) + .addDef(SubRegCopy) + .addUse(CopyReg, 0, AArch64::hsub); + + MachineOperand &RegOp = I.getOperand(1); + RegOp.setReg(SubRegCopy); + return true; +} + static bool selectCopy(MachineInstr &I, const TargetInstrInfo &TII, MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI, const RegisterBankInfo &RBI) { unsigned DstReg = I.getOperand(0).getReg(); + unsigned SrcReg = I.getOperand(1).getReg(); + if (TargetRegisterInfo::isPhysicalRegister(DstReg)) { + if (TRI.getRegClass(AArch64::FPR16RegClassID)->contains(DstReg) && + !TargetRegisterInfo::isPhysicalRegister(SrcReg)) { + const RegisterBank &RegBank = *RBI.getRegBank(SrcReg, MRI, TRI); + const TargetRegisterClass *SrcRC = getRegClassForTypeOnBank( + MRI.getType(SrcReg), RegBank, RBI, /* GetAllRegSet */ true); + if (SrcRC == &AArch64::GPR32allRegClass) + return selectFP16CopyFromGPR32(I, TII, MRI, SrcReg); + } assert(I.isCopy() && "Generic operators do not allow physical registers"); return true; } const RegisterBank &RegBank = *RBI.getRegBank(DstReg, MRI, TRI); const unsigned DstSize = MRI.getType(DstReg).getSizeInBits(); - unsigned SrcReg = I.getOperand(1).getReg(); + (void)DstSize; const unsigned SrcSize = RBI.getSizeInBits(SrcReg, MRI, TRI); (void)SrcSize; assert((!TargetRegisterInfo::isPhysicalRegister(SrcReg) || I.isCopy()) && @@ -340,26 +372,38 @@ static bool selectCopy(MachineInstr &I, const TargetIn "Copy with different width?!"); assert((DstSize <= 64 || RegBank.getID() == AArch64::FPRRegBankID) && "GPRs cannot get more than 64-bit width values"); - const TargetRegisterClass *RC = nullptr; - if (RegBank.getID() == AArch64::FPRRegBankID) { - if (DstSize <= 16) - RC = &AArch64::FPR16RegClass; - else if (DstSize <= 32) - RC = &AArch64::FPR32RegClass; - else if (DstSize <= 64) - RC = &AArch64::FPR64RegClass; - else if (DstSize <= 128) - RC = &AArch64::FPR128RegClass; - else { - DEBUG(dbgs() << "Unexpected bitcast size " << DstSize << '\n'); - return false; + const TargetRegisterClass *RC = getRegClassForTypeOnBank( + MRI.getType(DstReg), RegBank, RBI, /* GetAllRegSet */ true); + if (!RC) { + DEBUG(dbgs() << "Unexpected bitcast size " << DstSize << '\n'); + return false; + } + + if (!TargetRegisterInfo::isPhysicalRegister(SrcReg)) { + const RegClassOrRegBank &RegClassOrBank = MRI.getRegClassOrRegBank(SrcReg); + const TargetRegisterClass *SrcRC = + RegClassOrBank.dyn_cast(); + const RegisterBank *RB = nullptr; + if (!SrcRC) { + RB = RegClassOrBank.get(); + SrcRC = getRegClassForTypeOnBank(MRI.getType(SrcReg), *RB, RBI, true); } - } else { - assert(RegBank.getID() == AArch64::GPRRegBankID && - "Bitcast for the flags?"); - RC = - DstSize <= 32 ? &AArch64::GPR32allRegClass : &AArch64::GPR64allRegClass; + // Copies from fpr16 to gpr32 need to use SUBREG_TO_REG. + if (RC == &AArch64::GPR32allRegClass && SrcRC == &AArch64::FPR16RegClass) { + unsigned PromoteReg = MRI.createVirtualRegister(&AArch64::FPR32RegClass); + BuildMI(*I.getParent(), I, I.getDebugLoc(), + TII.get(AArch64::SUBREG_TO_REG)) + .addDef(PromoteReg) + .addImm(0) + .addUse(SrcReg) + .addImm(AArch64::hsub); + MachineOperand &RegOp = I.getOperand(1); + RegOp.setReg(PromoteReg); + } else if (RC == &AArch64::FPR16RegClass && + SrcRC == &AArch64::GPR32allRegClass) { + selectFP16CopyFromGPR32(I, TII, MRI, SrcReg); + } } // No need to constrain SrcReg. It will get constrained when @@ -795,15 +839,23 @@ bool AArch64InstructionSelector::select(MachineInstr & } case TargetOpcode::G_EXTRACT: { LLT SrcTy = MRI.getType(I.getOperand(1).getReg()); + LLT DstTy = MRI.getType(I.getOperand(0).getReg()); + unsigned SrcSize = SrcTy.getSizeInBits(); // Larger extracts are vectors, same-size extracts should be something else // by now (either split up or simplified to a COPY). if (SrcTy.getSizeInBits() > 64 || Ty.getSizeInBits() > 32) return false; - I.setDesc(TII.get(AArch64::UBFMXri)); + I.setDesc(TII.get(SrcSize == 64 ? AArch64::UBFMXri : AArch64::UBFMWri)); MachineInstrBuilder(MF, I).addImm(I.getOperand(2).getImm() + Ty.getSizeInBits() - 1); + if (SrcSize < 64) { + assert(SrcSize == 32 && DstTy.getSizeInBits() == 16 && + "unexpected G_EXTRACT types"); + return constrainSelectedInstRegOperands(I, TII, TRI, RBI); + } + unsigned DstReg = MRI.createGenericVirtualRegister(LLT::scalar(64)); BuildMI(MBB, std::next(I.getIterator()), I.getDebugLoc(), TII.get(AArch64::COPY)) @@ -818,16 +870,25 @@ bool AArch64InstructionSelector::select(MachineInstr & case TargetOpcode::G_INSERT: { LLT SrcTy = MRI.getType(I.getOperand(2).getReg()); + LLT DstTy = MRI.getType(I.getOperand(0).getReg()); + unsigned DstSize = DstTy.getSizeInBits(); + (void)DstSize; // Larger inserts are vectors, same-size ones should be something else by // now (split up or turned into COPYs). if (Ty.getSizeInBits() > 64 || SrcTy.getSizeInBits() > 32) return false; - I.setDesc(TII.get(AArch64::BFMXri)); + I.setDesc(TII.get(DstSize == 64 ? AArch64::BFMXri : AArch64::BFMWri)); unsigned LSB = I.getOperand(3).getImm(); unsigned Width = MRI.getType(I.getOperand(2).getReg()).getSizeInBits(); - I.getOperand(3).setImm((64 - LSB) % 64); + I.getOperand(3).setImm((DstSize - LSB) % DstSize); MachineInstrBuilder(MF, I).addImm(Width - 1); + + if (DstSize < 64) { + assert(DstSize == 32 && SrcTy.getSizeInBits() == 16 && + "unexpected G_INSERT types"); + return constrainSelectedInstRegOperands(I, TII, TRI, RBI); + } unsigned SrcReg = MRI.createGenericVirtualRegister(LLT::scalar(64)); BuildMI(MBB, I.getIterator(), I.getDebugLoc(), Modified: vendor/llvm/dist-release_60/lib/Target/AMDGPU/SIInstrInfo.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Target/AMDGPU/SIInstrInfo.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Target/AMDGPU/SIInstrInfo.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -3797,7 +3797,8 @@ void SIInstrInfo::moveToVALU(MachineInstr &TopInst) co } } - BuildMI(*MBB, Inst, Inst.getDebugLoc(), + MachineInstr *NewInstr = + BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(AMDGPU::BUFFER_LOAD_DWORD_OFFEN), VDst) .add(*VAddr) // vaddr .add(*getNamedOperand(Inst, AMDGPU::OpName::sbase)) // srsrc @@ -3806,12 +3807,17 @@ void SIInstrInfo::moveToVALU(MachineInstr &TopInst) co .addImm(getNamedOperand(Inst, AMDGPU::OpName::glc)->getImm()) .addImm(0) // slc .addImm(0) // tfe - .setMemRefs(Inst.memoperands_begin(), Inst.memoperands_end()); + .setMemRefs(Inst.memoperands_begin(), Inst.memoperands_end()) + .getInstr(); MRI.replaceRegWith(getNamedOperand(Inst, AMDGPU::OpName::sdst)->getReg(), VDst); addUsersToMoveToVALUWorklist(VDst, MRI, Worklist); Inst.eraseFromParent(); + + // Legalize all operands other than the offset. Notably, convert the srsrc + // into SGPRs using v_readfirstlane if needed. + legalizeOperands(*NewInstr); continue; } } Modified: vendor/llvm/dist-release_60/lib/Target/PowerPC/PPCCTRLoops.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Target/PowerPC/PPCCTRLoops.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Target/PowerPC/PPCCTRLoops.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -454,13 +454,16 @@ bool PPCCTRLoops::mightUseCTR(BasicBlock *BB) { return true; } + // FREM is always a call. + if (J->getOpcode() == Instruction::FRem) + return true; + if (STI->useSoftFloat()) { switch(J->getOpcode()) { case Instruction::FAdd: case Instruction::FSub: case Instruction::FMul: case Instruction::FDiv: - case Instruction::FRem: case Instruction::FPTrunc: case Instruction::FPExt: case Instruction::FPToUI: Modified: vendor/llvm/dist-release_60/lib/Target/X86/X86.td ============================================================================== --- vendor/llvm/dist-release_60/lib/Target/X86/X86.td Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Target/X86/X86.td Sat Feb 24 21:27:30 2018 (r329931) @@ -740,7 +740,13 @@ class SkylakeServerProc : ProcModel; def : SkylakeServerProc<"skx">; // Legacy alias. -def CNLFeatures : ProcessorFeaturesgetOpcode()) { + default: + return true; + case Instruction::SRem: + case Instruction::URem: + case Instruction::SDiv: + case Instruction::UDiv: + return false; + } + }; + // Try to simplify a binop sandwiched between 2 selects with the same // condition. // select(C, binop(select(C, X, Y), W), Z) -> select(C, binop(X, W), Z) BinaryOperator *TrueBO; - if (match(TrueVal, m_OneUse(m_BinOp(TrueBO)))) { + if (match(TrueVal, m_OneUse(m_BinOp(TrueBO))) && + canMergeSelectThroughBinop(TrueBO)) { if (auto *TrueBOSI = dyn_cast(TrueBO->getOperand(0))) { if (TrueBOSI->getCondition() == CondVal) { TrueBO->setOperand(0, TrueBOSI->getTrueValue()); @@ -1666,7 +1680,8 @@ Instruction *InstCombiner::visitSelectInst(SelectInst // select(C, Z, binop(select(C, X, Y), W)) -> select(C, Z, binop(Y, W)) BinaryOperator *FalseBO; - if (match(FalseVal, m_OneUse(m_BinOp(FalseBO)))) { + if (match(FalseVal, m_OneUse(m_BinOp(FalseBO))) && + canMergeSelectThroughBinop(FalseBO)) { if (auto *FalseBOSI = dyn_cast(FalseBO->getOperand(0))) { if (FalseBOSI->getCondition() == CondVal) { FalseBO->setOperand(0, FalseBOSI->getFalseValue()); Modified: vendor/llvm/dist-release_60/lib/Transforms/Scalar/LICM.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Transforms/Scalar/LICM.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Transforms/Scalar/LICM.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -97,7 +97,7 @@ static bool hoist(Instruction &I, const DominatorTree const LoopSafetyInfo *SafetyInfo, OptimizationRemarkEmitter *ORE); static bool sink(Instruction &I, LoopInfo *LI, DominatorTree *DT, - const Loop *CurLoop, const LoopSafetyInfo *SafetyInfo, + const Loop *CurLoop, LoopSafetyInfo *SafetyInfo, OptimizationRemarkEmitter *ORE, bool FreeInLoop); static bool isSafeToExecuteUnconditionally(Instruction &Inst, const DominatorTree *DT, @@ -855,10 +855,16 @@ static Instruction *sinkThroughTriviallyReplacablePHI( return New; } -static bool canSplitPredecessors(PHINode *PN) { +static bool canSplitPredecessors(PHINode *PN, LoopSafetyInfo *SafetyInfo) { BasicBlock *BB = PN->getParent(); if (!BB->canSplitPredecessors()) return false; + // It's not impossible to split EHPad blocks, but if BlockColors already exist + // it require updating BlockColors for all offspring blocks accordingly. By + // skipping such corner case, we can make updating BlockColors after splitting + // predecessor fairly simple. + if (!SafetyInfo->BlockColors.empty() && BB->getFirstNonPHI()->isEHPad()) + return false; for (pred_iterator PI = pred_begin(BB), E = pred_end(BB); PI != E; ++PI) { BasicBlock *BBPred = *PI; if (isa(BBPred->getTerminator())) @@ -868,7 +874,8 @@ static bool canSplitPredecessors(PHINode *PN) { } static void splitPredecessorsOfLoopExit(PHINode *PN, DominatorTree *DT, - LoopInfo *LI, const Loop *CurLoop) { + LoopInfo *LI, const Loop *CurLoop, + LoopSafetyInfo *SafetyInfo) { #ifndef NDEBUG SmallVector ExitBlocks; CurLoop->getUniqueExitBlocks(ExitBlocks); @@ -910,13 +917,21 @@ static void splitPredecessorsOfLoopExit(PHINode *PN, D // LE: // %p = phi [%p1, %LE.split], [%p2, %LE.split2] // + auto &BlockColors = SafetyInfo->BlockColors; SmallSetVector PredBBs(pred_begin(ExitBB), pred_end(ExitBB)); while (!PredBBs.empty()) { BasicBlock *PredBB = *PredBBs.begin(); assert(CurLoop->contains(PredBB) && "Expect all predecessors are in the loop"); - if (PN->getBasicBlockIndex(PredBB) >= 0) - SplitBlockPredecessors(ExitBB, PredBB, ".split.loop.exit", DT, LI, true); + if (PN->getBasicBlockIndex(PredBB) >= 0) { + BasicBlock *NewPred = SplitBlockPredecessors( + ExitBB, PredBB, ".split.loop.exit", DT, LI, true); + // Since we do not allow splitting EH-block with BlockColors in + // canSplitPredecessors(), we can simply assign predecessor's color to + // the new block. + if (!BlockColors.empty()) + BlockColors[NewPred] = BlockColors[PredBB]; + } PredBBs.remove(PredBB); } } @@ -927,7 +942,7 @@ static void splitPredecessorsOfLoopExit(PHINode *PN, D /// position, and may either delete it or move it to outside of the loop. /// static bool sink(Instruction &I, LoopInfo *LI, DominatorTree *DT, - const Loop *CurLoop, const LoopSafetyInfo *SafetyInfo, + const Loop *CurLoop, LoopSafetyInfo *SafetyInfo, OptimizationRemarkEmitter *ORE, bool FreeInLoop) { DEBUG(dbgs() << "LICM sinking instruction: " << I << "\n"); ORE->emit([&]() { @@ -975,12 +990,12 @@ static bool sink(Instruction &I, LoopInfo *LI, Dominat if (isTriviallyReplacablePHI(*PN, I)) continue; - if (!canSplitPredecessors(PN)) + if (!canSplitPredecessors(PN, SafetyInfo)) return Changed; // Split predecessors of the PHI so that we can make users trivially // replacable. - splitPredecessorsOfLoopExit(PN, DT, LI, CurLoop); + splitPredecessorsOfLoopExit(PN, DT, LI, CurLoop, SafetyInfo); // Should rebuild the iterators, as they may be invalidated by // splitPredecessorsOfLoopExit(). Modified: vendor/llvm/dist-release_60/lib/Transforms/Utils/LoopUtils.cpp ============================================================================== --- vendor/llvm/dist-release_60/lib/Transforms/Utils/LoopUtils.cpp Sat Feb 24 21:25:56 2018 (r329930) +++ vendor/llvm/dist-release_60/lib/Transforms/Utils/LoopUtils.cpp Sat Feb 24 21:27:30 2018 (r329931) @@ -23,6 +23,7 @@ #include "llvm/Analysis/ScalarEvolutionExpander.h" #include "llvm/Analysis/ScalarEvolutionExpressions.h" #include "llvm/Analysis/TargetTransformInfo.h" +#include "llvm/Analysis/ValueTracking.h" #include "llvm/IR/Dominators.h" #include "llvm/IR/Instructions.h" #include "llvm/IR/Module.h" @@ -30,6 +31,7 @@ #include "llvm/IR/ValueHandle.h" #include "llvm/Pass.h" #include "llvm/Support/Debug.h" +#include "llvm/Support/KnownBits.h" #include "llvm/Transforms/Utils/BasicBlockUtils.h" using namespace llvm; @@ -77,10 +79,13 @@ bool RecurrenceDescriptor::isArithmeticRecurrenceKind( return false; } -Instruction * -RecurrenceDescriptor::lookThroughAnd(PHINode *Phi, Type *&RT, - SmallPtrSetImpl &Visited, - SmallPtrSetImpl &CI) { +/// Determines if Phi may have been type-promoted. If Phi has a single user +/// that ANDs the Phi with a type mask, return the user. RT is updated to +/// account for the narrower bit width represented by the mask, and the AND +/// instruction is added to CI. +static Instruction *lookThroughAnd(PHINode *Phi, Type *&RT, + SmallPtrSetImpl &Visited, + SmallPtrSetImpl &CI) { if (!Phi->hasOneUse()) return Phi; @@ -101,70 +106,92 @@ RecurrenceDescriptor::lookThroughAnd(PHINode *Phi, Typ return Phi; } -bool RecurrenceDescriptor::getSourceExtensionKind( - Instruction *Start, Instruction *Exit, Type *RT, bool &IsSigned, - SmallPtrSetImpl &Visited, - SmallPtrSetImpl &CI) { +/// Compute the minimal bit width needed to represent a reduction whose exit +/// instruction is given by Exit. +static std::pair computeRecurrenceType(Instruction *Exit, + DemandedBits *DB, + AssumptionCache *AC, + DominatorTree *DT) { + bool IsSigned = false; + const DataLayout &DL = Exit->getModule()->getDataLayout(); + uint64_t MaxBitWidth = DL.getTypeSizeInBits(Exit->getType()); + if (DB) { + // Use the demanded bits analysis to determine the bits that are live out + // of the exit instruction, rounding up to the nearest power of two. If the + // use of demanded bits results in a smaller bit width, we know the value + // must be positive (i.e., IsSigned = false), because if this were not the + // case, the sign bit would have been demanded. + auto Mask = DB->getDemandedBits(Exit); + MaxBitWidth = Mask.getBitWidth() - Mask.countLeadingZeros(); + } + + if (MaxBitWidth == DL.getTypeSizeInBits(Exit->getType()) && AC && DT) { + // If demanded bits wasn't able to limit the bit width, we can try to use + // value tracking instead. This can be the case, for example, if the value + // may be negative. + auto NumSignBits = ComputeNumSignBits(Exit, DL, 0, AC, nullptr, DT); + auto NumTypeBits = DL.getTypeSizeInBits(Exit->getType()); + MaxBitWidth = NumTypeBits - NumSignBits; + KnownBits Bits = computeKnownBits(Exit, DL); + if (!Bits.isNonNegative()) { + // If the value is not known to be non-negative, we set IsSigned to true, + // meaning that we will use sext instructions instead of zext + // instructions to restore the original type. + IsSigned = true; + if (!Bits.isNegative()) + // If the value is not known to be negative, we don't known what the + // upper bit is, and therefore, we don't know what kind of extend we + // will need. In this case, just increase the bit width by one bit and + // use sext. + ++MaxBitWidth; + } + } + if (!isPowerOf2_64(MaxBitWidth)) + MaxBitWidth = NextPowerOf2(MaxBitWidth); + + return std::make_pair(Type::getIntNTy(Exit->getContext(), MaxBitWidth), + IsSigned); +} + +/// Collect cast instructions that can be ignored in the vectorizer's cost +/// model, given a reduction exit value and the minimal type in which the +/// reduction can be represented. +static void collectCastsToIgnore(Loop *TheLoop, Instruction *Exit, + Type *RecurrenceType, + SmallPtrSetImpl &Casts) { + SmallVector Worklist; - bool FoundOneOperand = false; - unsigned DstSize = RT->getPrimitiveSizeInBits(); + SmallPtrSet Visited; Worklist.push_back(Exit); - // Traverse the instructions in the reduction expression, beginning with the - // exit value. while (!Worklist.empty()) { - Instruction *I = Worklist.pop_back_val(); - for (Use &U : I->operands()) { - - // Terminate the traversal if the operand is not an instruction, or we - // reach the starting value. - Instruction *J = dyn_cast(U.get()); - if (!J || J == Start) + Instruction *Val = Worklist.pop_back_val(); + Visited.insert(Val); + if (auto *Cast = dyn_cast(Val)) + if (Cast->getSrcTy() == RecurrenceType) { + // If the source type of a cast instruction is equal to the recurrence + // type, it will be eliminated, and should be ignored in the vectorizer + // cost model. + Casts.insert(Cast); continue; - - // Otherwise, investigate the operation if it is also in the expression. - if (Visited.count(J)) { - Worklist.push_back(J); - continue; } - // If the operand is not in Visited, it is not a reduction operation, but - // it does feed into one. Make sure it is either a single-use sign- or - // zero-extend instruction. - CastInst *Cast = dyn_cast(J); - bool IsSExtInst = isa(J); - if (!Cast || !Cast->hasOneUse() || !(isa(J) || IsSExtInst)) - return false; - - // Ensure the source type of the extend is no larger than the reduction - // type. It is not necessary for the types to be identical. - unsigned SrcSize = Cast->getSrcTy()->getPrimitiveSizeInBits(); - if (SrcSize > DstSize) - return false; - - // Furthermore, ensure that all such extends are of the same kind. - if (FoundOneOperand) { - if (IsSigned != IsSExtInst) - return false; - } else { - FoundOneOperand = true; - IsSigned = IsSExtInst; - } - - // Lastly, if the source type of the extend matches the reduction type, - // add the extend to CI so that we can avoid accounting for it in the - // cost model. - if (SrcSize == DstSize) - CI.insert(Cast); - } + // Add all operands to the work list if they are loop-varying values that + // we haven't yet visited. + for (Value *O : cast(Val)->operands()) + if (auto *I = dyn_cast(O)) + if (TheLoop->contains(I) && !Visited.count(I)) + Worklist.push_back(I); } - return true; } bool RecurrenceDescriptor::AddReductionVar(PHINode *Phi, RecurrenceKind Kind, Loop *TheLoop, bool HasFunNoNaNAttr, - RecurrenceDescriptor &RedDes) { + RecurrenceDescriptor &RedDes, + DemandedBits *DB, + AssumptionCache *AC, + DominatorTree *DT) { if (Phi->getNumIncomingValues() != 2) return false; @@ -353,14 +380,49 @@ bool RecurrenceDescriptor::AddReductionVar(PHINode *Ph if (!FoundStartPHI || !FoundReduxOp || !ExitInstruction) return false; - // If we think Phi may have been type-promoted, we also need to ensure that - // all source operands of the reduction are either SExtInsts or ZEstInsts. If - // so, we will be able to evaluate the reduction in the narrower bit width. - if (Start != Phi) - if (!getSourceExtensionKind(Start, ExitInstruction, RecurrenceType, - IsSigned, VisitedInsts, CastInsts)) + if (Start != Phi) { + // If the starting value is not the same as the phi node, we speculatively + // looked through an 'and' instruction when evaluating a potential + // arithmetic reduction to determine if it may have been type-promoted. + // + // We now compute the minimal bit width that is required to represent the + // reduction. If this is the same width that was indicated by the 'and', we + // can represent the reduction in the smaller type. The 'and' instruction + // will be eliminated since it will essentially be a cast instruction that + // can be ignore in the cost model. If we compute a different type than we + // did when evaluating the 'and', the 'and' will not be eliminated, and we + // will end up with different kinds of operations in the recurrence + // expression (e.g., RK_IntegerAND, RK_IntegerADD). We give up if this is + // the case. + // + // The vectorizer relies on InstCombine to perform the actual + // type-shrinking. It does this by inserting instructions to truncate the + // exit value of the reduction to the width indicated by RecurrenceType and + // then extend this value back to the original width. If IsSigned is false, + // a 'zext' instruction will be generated; otherwise, a 'sext' will be + // used. + // + // TODO: We should not rely on InstCombine to rewrite the reduction in the + // smaller type. We should just generate a correctly typed expression + // to begin with. + Type *ComputedType; + std::tie(ComputedType, IsSigned) = + computeRecurrenceType(ExitInstruction, DB, AC, DT); + if (ComputedType != RecurrenceType) return false; + // The recurrence expression will be represented in a narrower type. If + // there are any cast instructions that will be unnecessary, collect them + // in CastInsts. Note that the 'and' instruction was already included in + // this list. + // + // TODO: A better way to represent this may be to tag in some way all the + // instructions that are a part of the reduction. The vectorizer cost + // model could then apply the recurrence type to these instructions, + // without needing a white list of instructions to ignore. + collectCastsToIgnore(TheLoop, ExitInstruction, RecurrenceType, CastInsts); + } + // We found a reduction var if we have reached the original phi node and we // only have a single instruction with out-of-loop users. @@ -480,47 +542,57 @@ bool RecurrenceDescriptor::hasMultipleUsesOf( return false; } bool RecurrenceDescriptor::isReductionPHI(PHINode *Phi, Loop *TheLoop, - RecurrenceDescriptor &RedDes) { + RecurrenceDescriptor &RedDes, + DemandedBits *DB, AssumptionCache *AC, + DominatorTree *DT) { BasicBlock *Header = TheLoop->getHeader(); Function &F = *Header->getParent(); bool HasFunNoNaNAttr = F.getFnAttribute("no-nans-fp-math").getValueAsString() == "true"; - if (AddReductionVar(Phi, RK_IntegerAdd, TheLoop, HasFunNoNaNAttr, RedDes)) { + if (AddReductionVar(Phi, RK_IntegerAdd, TheLoop, HasFunNoNaNAttr, RedDes, DB, + AC, DT)) { DEBUG(dbgs() << "Found an ADD reduction PHI." << *Phi << "\n"); return true; } - if (AddReductionVar(Phi, RK_IntegerMult, TheLoop, HasFunNoNaNAttr, RedDes)) { + if (AddReductionVar(Phi, RK_IntegerMult, TheLoop, HasFunNoNaNAttr, RedDes, DB, + AC, DT)) { *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:39 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE9F1F23E9B; Sat, 24 Feb 2018 21:27:39 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0584477AFA; Sat, 24 Feb 2018 21:27:36 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 57E692B05F; Sat, 24 Feb 2018 21:27:36 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRagA008919; Sat, 24 Feb 2018 21:27:36 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRaGt008918; Sat, 24 Feb 2018 21:27:36 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRaGt008918@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329932 - vendor/llvm/llvm-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/llvm/llvm-release_60-r325932 X-SVN-Commit-Revision: 329932 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:39 -0000 Author: dim Date: Sat Feb 24 21:27:35 2018 New Revision: 329932 URL: https://svnweb.freebsd.org/changeset/base/329932 Log: Tag llvm release_60 branch r325932. Added: vendor/llvm/llvm-release_60-r325932/ - copied from r329931, vendor/llvm/dist-release_60/ From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:41 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4472CF23EAD; Sat, 24 Feb 2018 21:27:41 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED4E677B90; Sat, 24 Feb 2018 21:27:40 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E7FF12B060; Sat, 24 Feb 2018 21:27:40 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLReCZ008973; Sat, 24 Feb 2018 21:27:40 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRe0D008966; Sat, 24 Feb 2018 21:27:40 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRe0D008966@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:40 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329933 - in vendor/clang/dist-release_60: docs lib/Basic/Targets lib/Sema test/Preprocessor test/SemaCXX X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/clang/dist-release_60: docs lib/Basic/Targets lib/Sema test/Preprocessor test/SemaCXX X-SVN-Commit-Revision: 329933 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:41 -0000 Author: dim Date: Sat Feb 24 21:27:39 2018 New Revision: 329933 URL: https://svnweb.freebsd.org/changeset/base/329933 Log: Vendor import of clang release_60 branch r325932: https://llvm.org/svn/llvm-project/cfe/branches/release_60@325932 Modified: vendor/clang/dist-release_60/docs/ClangCommandLineReference.rst vendor/clang/dist-release_60/docs/DiagnosticsReference.rst vendor/clang/dist-release_60/docs/ReleaseNotes.rst vendor/clang/dist-release_60/lib/Basic/Targets/X86.cpp vendor/clang/dist-release_60/lib/Sema/SemaExpr.cpp vendor/clang/dist-release_60/test/Preprocessor/predefined-arch-macros.c vendor/clang/dist-release_60/test/SemaCXX/lambda-expressions.cpp Modified: vendor/clang/dist-release_60/docs/ClangCommandLineReference.rst ============================================================================== --- vendor/clang/dist-release_60/docs/ClangCommandLineReference.rst Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/docs/ClangCommandLineReference.rst Sat Feb 24 21:27:39 2018 (r329933) @@ -706,6 +706,10 @@ Print source range spans in numeric form .. option:: -fdiagnostics-show-category= +.. option:: -fexperimental-isel, -fno-experimental-isel + +Enables the experimental global instruction selector + .. option:: -fexperimental-new-pass-manager, -fno-experimental-new-pass-manager Enables an experimental new pass manager in LLVM. @@ -2413,6 +2417,10 @@ X86 .. option:: -mrdrnd, -mno-rdrnd .. option:: -mrdseed, -mno-rdseed + +.. option:: -mretpoline, -mno-retpoline + +.. option:: -mretpoline-external-thunk, -mno-retpoline-external-thunk .. option:: -mrtm, -mno-rtm Modified: vendor/clang/dist-release_60/docs/DiagnosticsReference.rst ============================================================================== --- vendor/clang/dist-release_60/docs/DiagnosticsReference.rst Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/docs/DiagnosticsReference.rst Sat Feb 24 21:27:39 2018 (r329933) @@ -806,6 +806,17 @@ Also controls `-Wc++98-compat-bind-to-temporary-copy`_ +--------------------------------------------------------------------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------+ +-Wbinding-in-condition +---------------------- +This diagnostic is enabled by default. + +**Diagnostic text:** + ++--------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`ISO C++17 does not permit structured binding declaration in a condition`| ++--------------------------------------------------------------------------------------------------------------+ + + -Wbitfield-constant-conversion ------------------------------ This diagnostic is enabled by default. @@ -1402,6 +1413,10 @@ Some of the diagnostics controlled by this flag are en **Diagnostic text:** ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`ISO C++ standards before C++17 do not allow new expression for type` |nbsp| :placeholder:`A` |nbsp| :diagtext:`to use list-initialization`| ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + +------------------------------------------------------------------------+ |:warning:`warning:` |nbsp| :diagtext:`constexpr if is a C++17 extension`| +------------------------------------------------------------------------+ @@ -1501,8 +1516,14 @@ Synonym for `-Wc++17-extensions`_. -Wc++2a-compat -------------- +Some of the diagnostics controlled by this flag are enabled by default. + **Diagnostic text:** ++------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`'<=>' is a single token in C++2a; add a space to avoid a change in behavior`| ++------------------------------------------------------------------------------------------------------------------+ + +-------------------------------------------------------------------------------------------+ |:warning:`warning:` |nbsp| :diagtext:`'`:placeholder:`A`:diagtext:`' is a keyword in C++2a`| +-------------------------------------------------------------------------------------------+ @@ -1544,7 +1565,11 @@ Some of the diagnostics controlled by this flag are en |:warning:`warning:` |nbsp| :diagtext:`explicit capture of 'this' with a capture default of '=' is incompatible with C++ standards before C++2a`| +-----------------------------------------------------------------------------------------------------------------------------------------------+ ++-----------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`'<=>' operator is incompatible with C++ standards before C++2a`| ++-----------------------------------------------------------------------------------------------------+ + -Wc++98-c++11-c++14-c++17-compat-pedantic ----------------------------------------- Also controls `-Wc++98-c++11-c++14-c++17-compat`_. @@ -3540,6 +3565,21 @@ Some of the diagnostics controlled by this flag are en +-------------------------------------------------------------------------------------------------+ +-Wexperimental-isel +------------------- +This diagnostic is enabled by default. + +**Diagnostic text:** + ++------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`-fexperimental-isel support for the '`:placeholder:`A`:diagtext:`' architecture is incomplete`| ++------------------------------------------------------------------------------------------------------------------------------------+ + ++----------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`-fexperimental-isel support is incomplete for this architecture at the current optimization level`| ++----------------------------------------------------------------------------------------------------------------------------------------+ + + -Wexplicit-initialize-call -------------------------- This diagnostic is enabled by default. @@ -3564,15 +3604,6 @@ This diagnostic is enabled by default. +-------------------------------------------------------------------------------------------------------------------------------------+ --Wextended-offsetof -------------------- -**Diagnostic text:** - -+--------------------------------------------------------------------------------------+ -|:warning:`warning:` |nbsp| :diagtext:`using extended field designator is an extension`| -+--------------------------------------------------------------------------------------+ - - -Wextern-c-compat ----------------- This diagnostic is enabled by default. @@ -4565,96 +4596,34 @@ This diagnostic is enabled by default. | |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables and functions` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions and global variables` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, variables, and Objective-C interfaces` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`functions and methods` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`parameters` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`functions, methods and blocks` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, methods, and classes` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`functions, methods, and parameters` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, methods, and global variables` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`classes` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`enums` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`methods` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`fields and global variables` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`structs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`parameters and typedefs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`variables and typedefs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`thread-local variables` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables and fields` || | |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables, data members and tag types` || | |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`types and namespaces` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`Objective-C interfaces` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`methods and properties` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, methods, and properties` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`struct or union` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`struct, union or class` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`types` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`Objective-C instance methods` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`init methods of interface or class extension declarations` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables, functions and classes` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, variables, classes, and Objective-C interfaces` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`Objective-C protocols` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`variables with static or thread storage duration` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, methods, properties, and global variables` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`structs, unions, and typedefs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`structs and typedefs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`interface or protocol declarations` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`kernel functions` || | |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`non-K&R-style functions` || | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`variables, enums, fields and typedefs` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`functions, methods, enums, and classes` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`structs, classes, variables, functions, and inline namespaces` || -| |+----------------------------------------------------------------------------------------------------------------+| | ||:diagtext:`variables, functions, methods, types, enumerations, enumerators, labels, and non-static data members`|| | |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`classes and enumerations` || -| |+----------------------------------------------------------------------------------------------------------------+| -| ||:diagtext:`named declarations` || -| |+----------------------------------------------------------------------------------------------------------------+| +------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+ ++----------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :placeholder:`A` |nbsp| :diagtext:`attribute only applies to` |nbsp| :placeholder:`B`| ++----------------------------------------------------------------------------------------------------------------+ + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ |:warning:`warning:` |nbsp| :diagtext:`attribute` |nbsp| :placeholder:`A` |nbsp| :diagtext:`ignored, because it cannot be applied to omitted return type`| +--------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -6333,6 +6302,17 @@ This diagnostic is enabled by default. +----------------------------------------------------------------------------------------------------------+ +-Wmicrosoft-inaccessible-base +----------------------------- +This diagnostic is enabled by default. + +**Diagnostic text:** + ++-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`accessing inaccessible direct base` |nbsp| :placeholder:`A` |nbsp| :diagtext:`of` |nbsp| :placeholder:`B` |nbsp| :diagtext:`is a Microsoft extension`| ++-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + + -Wmicrosoft-include ------------------- This diagnostic is enabled by default. @@ -8075,7 +8055,7 @@ This diagnostic is enabled by default. -Wpedantic ---------- -Also controls `-Wc++11-extra-semi`_, `-Wc++11-long-long`_, `-Wc++14-binary-literal`_, `-Wc11-extensions`_, `-Wcomplex-component-init`_, `-Wdeclaration-after-statement`_, `-Wdollar-in-identifier-extension`_, `-Wembedded-directive`_, `-Wempty-translation-unit`_, `-Wextended-offsetof`_, `-Wflexible-array-extensions`_, `-Wformat-pedantic`_, `-Wfour-char-constants`_, `-Wgnu-anonymous-struct`_, `-Wgnu-auto-type`_, `-Wgnu-binary-literal`_, `-Wgnu-case-range`_, `-Wgnu-complex-integer`_, `-Wgnu-compound-literal-initializer`_, `-Wgnu-conditional-omitted-operand`_, `-Wgnu-empty-initializer`_, `-Wgnu-empty-struct`_, `-Wgnu-flexible-array-initializer`_, `-Wgnu-flexible-array-union-member`_, `-Wgnu-folding-constant`_, `-Wgnu-imaginary-constant`_, `-Wgnu-include-next`_, `-Wgnu-label-as-value`_, `-Wgnu-redeclared-enum`_, `-Wgnu-statement-expression`_, `-Wgnu-union-cast`_, `-Wgnu-zero-line-directive`_, `-Wgnu-zero-variadic-macro-arguments`_, `-Wimport-preprocessor-directive-pedantic`_, `-Wkeyword-ma cro`_, `-Wlanguage-extension-token`_, `-Wlong-long`_, `-Wmicrosoft-charize`_, `-Wmicrosoft-comment-paste`_, `-Wmicrosoft-cpp-macro`_, `-Wmicrosoft-end-of-file`_, `-Wmicrosoft-enum-value`_, `-Wmicrosoft-fixed-enum`_, `-Wmicrosoft-flexible-array`_, `-Wmicrosoft-redeclare-static`_, `-Wnested-anon-types`_, `-Wnullability-extension`_, `-Woverlength-strings`_, `-Wretained-language-linkage`_, `-Wundefined-internal-type`_, `-Wvla-extension`_, `-Wzero-length-array`_. +Also controls `-Wc++11-extra-semi`_, `-Wc++11-long-long`_, `-Wc++14-binary-literal`_, `-Wc11-extensions`_, `-Wcomplex-component-init`_, `-Wdeclaration-after-statement`_, `-Wdollar-in-identifier-extension`_, `-Wembedded-directive`_, `-Wempty-translation-unit`_, `-Wflexible-array-extensions`_, `-Wformat-pedantic`_, `-Wfour-char-constants`_, `-Wgnu-anonymous-struct`_, `-Wgnu-auto-type`_, `-Wgnu-binary-literal`_, `-Wgnu-case-range`_, `-Wgnu-complex-integer`_, `-Wgnu-compound-literal-initializer`_, `-Wgnu-conditional-omitted-operand`_, `-Wgnu-empty-initializer`_, `-Wgnu-empty-struct`_, `-Wgnu-flexible-array-initializer`_, `-Wgnu-flexible-array-union-member`_, `-Wgnu-folding-constant`_, `-Wgnu-imaginary-constant`_, `-Wgnu-include-next`_, `-Wgnu-label-as-value`_, `-Wgnu-redeclared-enum`_, `-Wgnu-statement-expression`_, `-Wgnu-union-cast`_, `-Wgnu-zero-line-directive`_, `-Wgnu-zero-variadic-macro-arguments`_, `-Wimport-preprocessor-directive-pedantic`_, `-Wkeyword-macro`_, `-Wlanguage-exten sion-token`_, `-Wlong-long`_, `-Wmicrosoft-charize`_, `-Wmicrosoft-comment-paste`_, `-Wmicrosoft-cpp-macro`_, `-Wmicrosoft-end-of-file`_, `-Wmicrosoft-enum-value`_, `-Wmicrosoft-fixed-enum`_, `-Wmicrosoft-flexible-array`_, `-Wmicrosoft-redeclare-static`_, `-Wnested-anon-types`_, `-Wnullability-extension`_, `-Woverlength-strings`_, `-Wretained-language-linkage`_, `-Wundefined-internal-type`_, `-Wvla-extension`_, `-Wzero-length-array`_. **Diagnostic text:** @@ -8137,6 +8117,10 @@ Also controls `-Wc++11-extra-semi`_, `-Wc++11-long-lon | |+---------------------------------------+| | +--------------------------------------------------------------------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`ISO C++ standards before C++17 do not allow new expression for type` |nbsp| :placeholder:`A` |nbsp| :diagtext:`to use list-initialization`| ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + +--------------------------------------------------------------------------------------------------------------------------------------+ |:warning:`warning:` |nbsp| :diagtext:`parameter` |nbsp| :placeholder:`A` |nbsp| :diagtext:`was not declared, defaulting to type 'int'`| +--------------------------------------------------------------------------------------------------------------------------------------+ @@ -8722,15 +8706,23 @@ This diagnostic is enabled by default. **Diagnostic text:** -+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -|:warning:`warning:` |nbsp| :diagtext:`top-level module '`:placeholder:`A`:diagtext:`' in private module map, expected a submodule of '`:placeholder:`B`:diagtext:`'`| -+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ++----------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`expected canonical name for private module '`:placeholder:`A`:diagtext:`'`| ++----------------------------------------------------------------------------------------------------------------+ ++----------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`private submodule '`:placeholder:`A`:diagtext:`' in private module map, expected top-level module`| ++----------------------------------------------------------------------------------------------------------------------------------------+ + +----------------------------------------------------------------------------------------------------------------------------------+ |:warning:`warning:` |nbsp| :diagtext:`module '`:placeholder:`A`:diagtext:`' already re-exported as '`:placeholder:`B`:diagtext:`'`| +----------------------------------------------------------------------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`no submodule named` |nbsp| :placeholder:`A` |nbsp| :diagtext:`in module '`:placeholder:`B`:diagtext:`'; using top level '`:placeholder:`C`:diagtext:`'`| ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + -Wprofile-instr-missing ----------------------- **Diagnostic text:** @@ -10013,36 +10005,45 @@ Also controls `-Wtautological-constant-compare`_, `-Wt ------------------------------- This diagnostic is enabled by default. -Also controls `-Wtautological-constant-out-of-range-compare`_, `-Wtautological-unsigned-enum-zero-compare`_, `-Wtautological-unsigned-zero-compare`_. +Also controls `-Wtautological-constant-out-of-range-compare`_. **Diagnostic text:** -+---------------------------------------------------------+------------------+--------------------------------+------------------+-------------------------------------+-------------------+ -|:warning:`warning:` |nbsp| :diagtext:`comparison` |nbsp| |+----------------+| |nbsp| :placeholder:`C` |nbsp| |+----------------+| |nbsp| :diagtext:`is always` |nbsp| |+-----------------+| -| ||:placeholder:`D`|| ||:placeholder:`B`|| ||:diagtext:`false`|| -| |+----------------+| |+----------------+| |+-----------------+| -| ||:placeholder:`B`|| ||:placeholder:`D`|| ||:diagtext:`true` || -| |+----------------+| |+----------------+| |+-----------------+| -+---------------------------------------------------------+------------------+--------------------------------+------------------+-------------------------------------+-------------------+ ++----------------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-----------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`result of comparison of` |nbsp| |+----------------------------------------------+| |nbsp| :diagtext:`with` |nbsp| |+--------------------------------------------------------+| |nbsp| :diagtext:`is always` |nbsp| :placeholder:`E`| +| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| | +| |||:diagtext:`constant` |nbsp| :placeholder:`A`||| |||:diagtext:`expression of type` |nbsp| :placeholder:`C`||| | +| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| | +| |+----------------------------------------------+| |+--------------------------------------------------------+| | +| ||:diagtext:`true` || ||:diagtext:`boolean expression` || | +| |+----------------------------------------------+| |+--------------------------------------------------------+| | +| ||:diagtext:`false` || | | | +| |+----------------------------------------------+| | | | ++----------------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-----------------------------------------------------+ +-Wtautological-constant-in-range-compare +---------------------------------------- +Controls `-Wtautological-type-limit-compare`_, `-Wtautological-unsigned-enum-zero-compare`_, `-Wtautological-unsigned-zero-compare`_. + + -Wtautological-constant-out-of-range-compare -------------------------------------------- This diagnostic is enabled by default. **Diagnostic text:** -+------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-------------------------------------+-------------------+ -|:warning:`warning:` |nbsp| :diagtext:`comparison of` |nbsp| |+----------------------------------------------+| |nbsp| :diagtext:`with` |nbsp| |+--------------------------------------------------------+| |nbsp| :diagtext:`is always` |nbsp| |+-----------------+| -| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| ||:diagtext:`false`|| -| |||:diagtext:`constant` |nbsp| :placeholder:`A`||| |||:diagtext:`expression of type` |nbsp| :placeholder:`C`||| |+-----------------+| -| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| ||:diagtext:`true` || -| |+----------------------------------------------+| |+--------------------------------------------------------+| |+-----------------+| -| ||:diagtext:`true` || ||:diagtext:`boolean expression` || | | -| |+----------------------------------------------+| |+--------------------------------------------------------+| | | -| ||:diagtext:`false` || | | | | -| |+----------------------------------------------+| | | | | -+------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-------------------------------------+-------------------+ ++----------------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-----------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`result of comparison of` |nbsp| |+----------------------------------------------+| |nbsp| :diagtext:`with` |nbsp| |+--------------------------------------------------------+| |nbsp| :diagtext:`is always` |nbsp| :placeholder:`E`| +| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| | +| |||:diagtext:`constant` |nbsp| :placeholder:`A`||| |||:diagtext:`expression of type` |nbsp| :placeholder:`C`||| | +| ||+--------------------------------------------+|| ||+------------------------------------------------------+|| | +| |+----------------------------------------------+| |+--------------------------------------------------------+| | +| ||:diagtext:`true` || ||:diagtext:`boolean expression` || | +| |+----------------------------------------------+| |+--------------------------------------------------------+| | +| ||:diagtext:`false` || | | | +| |+----------------------------------------------+| | | | ++----------------------------------------------------------------------+------------------------------------------------+--------------------------------+----------------------------------------------------------+-----------------------------------------------------+ -Wtautological-overlap-compare @@ -10083,6 +10084,19 @@ This diagnostic is enabled by default. +------------------------------------------------------------+------------------------+----------------------------------------------------------+-------------------------+-----------------------------------------------------+-------------------+ +-Wtautological-type-limit-compare +--------------------------------- +**Diagnostic text:** + ++-------------------------------------------------------------------+------------------+--------------------------------+------------------+-----------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`result of comparison` |nbsp| |+----------------+| |nbsp| :placeholder:`C` |nbsp| |+----------------+| |nbsp| :diagtext:`is always` |nbsp| :placeholder:`E`| +| ||:placeholder:`D`|| ||:placeholder:`B`|| | +| |+----------------+| |+----------------+| | +| ||:placeholder:`B`|| ||:placeholder:`D`|| | +| |+----------------+| |+----------------+| | ++-------------------------------------------------------------------+------------------+--------------------------------+------------------+-----------------------------------------------------+ + + -Wtautological-undefined-compare -------------------------------- This diagnostic is enabled by default. @@ -10108,32 +10122,28 @@ This diagnostic is enabled by default. -Wtautological-unsigned-enum-zero-compare ----------------------------------------- -This diagnostic is enabled by default. - **Diagnostic text:** -+------------------------------------------------------------+--------------------------------------+--------------------------------+--------------------------------------+-------------------------------------+-------------------+ -|:warning:`warning:` |nbsp| :diagtext:`comparison of` |nbsp| |+------------------------------------+| |nbsp| :placeholder:`C` |nbsp| |+------------------------------------+| |nbsp| :diagtext:`is always` |nbsp| |+-----------------+| -| ||:placeholder:`D` || ||:diagtext:`unsigned enum expression`|| ||:diagtext:`false`|| -| |+------------------------------------+| |+------------------------------------+| |+-----------------+| -| ||:diagtext:`unsigned enum expression`|| ||:placeholder:`D` || ||:diagtext:`true` || -| |+------------------------------------+| |+------------------------------------+| |+-----------------+| -+------------------------------------------------------------+--------------------------------------+--------------------------------+--------------------------------------+-------------------------------------+-------------------+ ++----------------------------------------------------------------------+--------------------------------------+--------------------------------+--------------------------------------+-----------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`result of comparison of` |nbsp| |+------------------------------------+| |nbsp| :placeholder:`C` |nbsp| |+------------------------------------+| |nbsp| :diagtext:`is always` |nbsp| :placeholder:`E`| +| ||:placeholder:`D` || ||:diagtext:`unsigned enum expression`|| | +| |+------------------------------------+| |+------------------------------------+| | +| ||:diagtext:`unsigned enum expression`|| ||:placeholder:`D` || | +| |+------------------------------------+| |+------------------------------------+| | ++----------------------------------------------------------------------+--------------------------------------+--------------------------------+--------------------------------------+-----------------------------------------------------+ -Wtautological-unsigned-zero-compare ------------------------------------ -This diagnostic is enabled by default. - **Diagnostic text:** -+------------------------------------------------------------+---------------------------------+--------------------------------+---------------------------------+-------------------------------------+-------------------+ -|:warning:`warning:` |nbsp| :diagtext:`comparison of` |nbsp| |+-------------------------------+| |nbsp| :placeholder:`C` |nbsp| |+-------------------------------+| |nbsp| :diagtext:`is always` |nbsp| |+-----------------+| -| ||:placeholder:`D` || ||:diagtext:`unsigned expression`|| ||:diagtext:`false`|| -| |+-------------------------------+| |+-------------------------------+| |+-----------------+| -| ||:diagtext:`unsigned expression`|| ||:placeholder:`D` || ||:diagtext:`true` || -| |+-------------------------------+| |+-------------------------------+| |+-----------------+| -+------------------------------------------------------------+---------------------------------+--------------------------------+---------------------------------+-------------------------------------+-------------------+ ++----------------------------------------------------------------------+---------------------------------+--------------------------------+---------------------------------+-----------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`result of comparison of` |nbsp| |+-------------------------------+| |nbsp| :placeholder:`C` |nbsp| |+-------------------------------+| |nbsp| :diagtext:`is always` |nbsp| :placeholder:`E`| +| ||:placeholder:`D` || ||:diagtext:`unsigned expression`|| | +| |+-------------------------------+| |+-------------------------------+| | +| ||:diagtext:`unsigned expression`|| ||:placeholder:`D` || | +| |+-------------------------------+| |+-------------------------------+| | ++----------------------------------------------------------------------+---------------------------------+--------------------------------+---------------------------------+-----------------------------------------------------+ -Wtentative-definition-incomplete-type @@ -10652,6 +10662,17 @@ This diagnostic is enabled by default. +---------------------------------------------------------------------------------------------+ +-Wunicode-homoglyph +------------------- +This diagnostic is enabled by default. + +**Diagnostic text:** + ++-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +|:warning:`warning:` |nbsp| :diagtext:`treating Unicode character as identifier character rather than as '`:placeholder:`B`:diagtext:`' symbol`| ++-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + + -Wunicode-whitespace -------------------- This diagnostic is enabled by default. @@ -11134,19 +11155,17 @@ This diagnostic is enabled by default. **Diagnostic text:** -+---------------------------+-----------------------------------------+--------------------------------------------+ -|:warning:`warning:` |nbsp| |+---------------------------------------+| |nbsp| :diagtext:`comparison result unused`| -| ||+----------------+--------------------+|| | -| |||+--------------+|:diagtext:`equality`||| | -| |||| || ||| | -| |||+--------------+| ||| | -| ||||:diagtext:`in`|| ||| | -| |||+--------------+| ||| | -| ||+----------------+--------------------+|| | -| |+---------------------------------------+| | -| ||:diagtext:`relational` || | -| |+---------------------------------------+| | -+---------------------------+-----------------------------------------+--------------------------------------------+ ++---------------------------+------------------------+--------------------------------------------+ +|:warning:`warning:` |nbsp| |+----------------------+| |nbsp| :diagtext:`comparison result unused`| +| ||:diagtext:`equality` || | +| |+----------------------+| | +| ||:diagtext:`inequality`|| | +| |+----------------------+| | +| ||:diagtext:`relational`|| | +| |+----------------------+| | +| ||:diagtext:`three-way` || | +| |+----------------------+| | ++---------------------------+------------------------+--------------------------------------------+ -Wunused-const-variable Modified: vendor/clang/dist-release_60/docs/ReleaseNotes.rst ============================================================================== --- vendor/clang/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:27:39 2018 (r329933) @@ -1,6 +1,6 @@ -======================================= -Clang 6.0.0 (In-Progress) Release Notes -======================================= +========================= +Clang 6.0.0 Release Notes +========================= .. contents:: :local: @@ -8,12 +8,6 @@ Clang 6.0.0 (In-Progress) Release Notes Written by the `LLVM Team `_ -.. warning:: - - These are in-progress notes for the upcoming Clang 6 release. - Release notes for previous releases can be found on - `the Download Page `_. - Introduction ============ @@ -30,11 +24,6 @@ For more information about Clang or LLVM, including in latest release, please see the `Clang Web Site `_ or the `LLVM Web Site `_. -Note that if you are reading this file from a Subversion checkout or the -main Clang web page, this document applies to the *next* release, not -the current one. To see the release notes for a specific release, please -see the `releases page `_. - What's New in Clang 6.0.0? ========================== @@ -95,6 +84,12 @@ Improvements to Clang's diagnostics - ``-Wzero-as-null-pointer-constant`` was adjusted not to warn on null pointer constants that originate from system macros, except ``NULL`` macro. +- ``-Wdelete-non-virtual-dtor`` can now fire in system headers, so that + ``std::unique_ptr<>`` deleting through a non-virtual dtor is now diagnosed. + +- ``-Wunreachable-code`` can now reason about ``__try``, ``__except`` and + ``__leave``. + Non-comprehensive list of changes in this release ------------------------------------------------- @@ -137,6 +132,9 @@ New Compiler Flags feature is enabled by default for AArch64 at the ``-O0`` optimization level. Support for other targets or optimization levels is currently incomplete. +- New ``-nostdlib++`` flag to disable linking the C++ standard library. Similar + to using ``clang`` instead of ``clang++`` but doesn't disable ``-lm``. + Deprecated Compiler Flags ------------------------- @@ -184,6 +182,8 @@ Windows Support - Clang now has initial, preliminary support for targeting Windows on ARM64. + +- clang-cl now exposes the ``--version`` flag. C Language Changes in Clang --------------------------- Modified: vendor/clang/dist-release_60/lib/Basic/Targets/X86.cpp ============================================================================== --- vendor/clang/dist-release_60/lib/Basic/Targets/X86.cpp Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/lib/Basic/Targets/X86.cpp Sat Feb 24 21:27:39 2018 (r329933) @@ -152,7 +152,8 @@ bool X86TargetInfo::initFeatureMap( setFeatureEnabledImpl(Features, "avx512bw", true); setFeatureEnabledImpl(Features, "avx512vl", true); setFeatureEnabledImpl(Features, "pku", true); - setFeatureEnabledImpl(Features, "clwb", true); + if (Kind != CK_Cannonlake) // CNL inherits all SKX features, except CLWB + setFeatureEnabledImpl(Features, "clwb", true); LLVM_FALLTHROUGH; case CK_SkylakeClient: setFeatureEnabledImpl(Features, "xsavec", true); Modified: vendor/clang/dist-release_60/lib/Sema/SemaExpr.cpp ============================================================================== --- vendor/clang/dist-release_60/lib/Sema/SemaExpr.cpp Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/lib/Sema/SemaExpr.cpp Sat Feb 24 21:27:39 2018 (r329933) @@ -14926,7 +14926,8 @@ static void DoMarkVarDeclReferenced(Sema &SemaRef, Sou if (RefersToEnclosingScope) { LambdaScopeInfo *const LSI = SemaRef.getCurLambda(/*IgnoreNonLambdaCapturingScope=*/true); - if (LSI && !LSI->CallOperator->Encloses(Var->getDeclContext())) { + if (LSI && (!LSI->CallOperator || + !LSI->CallOperator->Encloses(Var->getDeclContext()))) { // If a variable could potentially be odr-used, defer marking it so // until we finish analyzing the full expression for any // lvalue-to-rvalue Modified: vendor/clang/dist-release_60/test/Preprocessor/predefined-arch-macros.c ============================================================================== --- vendor/clang/dist-release_60/test/Preprocessor/predefined-arch-macros.c Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/test/Preprocessor/predefined-arch-macros.c Sat Feb 24 21:27:39 2018 (r329933) @@ -972,7 +972,7 @@ // CHECK_CNL_M32: #define __BMI2__ 1 // CHECK_CNL_M32: #define __BMI__ 1 // CHECK_CNL_M32: #define __CLFLUSHOPT__ 1 -// CHECK_CNL_M32: #define __CLWB__ 1 +// CHECK_CNL_M32-NOT: #define __CLWB__ 1 // CHECK_CNL_M32: #define __F16C__ 1 // CHECK_CNL_M32: #define __FMA__ 1 // CHECK_CNL_M32: #define __LZCNT__ 1 @@ -1019,7 +1019,7 @@ // CHECK_CNL_M64: #define __BMI2__ 1 // CHECK_CNL_M64: #define __BMI__ 1 // CHECK_CNL_M64: #define __CLFLUSHOPT__ 1 -// CHECK_CNL_M64: #define __CLWB__ 1 +// CHECK_CNL_M64-NOT: #define __CLWB__ 1 // CHECK_CNL_M64: #define __F16C__ 1 // CHECK_CNL_M64: #define __FMA__ 1 // CHECK_CNL_M64: #define __LZCNT__ 1 Modified: vendor/clang/dist-release_60/test/SemaCXX/lambda-expressions.cpp ============================================================================== --- vendor/clang/dist-release_60/test/SemaCXX/lambda-expressions.cpp Sat Feb 24 21:27:35 2018 (r329932) +++ vendor/clang/dist-release_60/test/SemaCXX/lambda-expressions.cpp Sat Feb 24 21:27:39 2018 (r329933) @@ -608,3 +608,18 @@ namespace ConversionOperatorDoesNotHaveDeducedReturnTy // This used to crash in return type deduction for the conversion opreator. struct A { int n; void f() { +[](decltype(n)) {}; } }; } + +namespace TypoCorrection { +template struct X {}; +// expected-note@-1 {{template parameter is declared here}} + +template +void Run(const int& points) { +// expected-note@-1 {{'points' declared here}} + auto outer_lambda = []() { + auto inner_lambda = [](const X&) {}; + // expected-error@-1 {{use of undeclared identifier 'Points'; did you mean 'points'?}} + // expected-error@-2 {{template argument for template type parameter must be a type}} + }; +} +} From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:51 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74415F23F62; Sat, 24 Feb 2018 21:27:51 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C53DB77D1B; Sat, 24 Feb 2018 21:27:48 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 801FC2B062; Sat, 24 Feb 2018 21:27:48 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRm3U009068; Sat, 24 Feb 2018 21:27:48 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRm5F009067; Sat, 24 Feb 2018 21:27:48 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRm5F009067@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329935 - vendor/compiler-rt/compiler-rt-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/compiler-rt/compiler-rt-release_60-r325932 X-SVN-Commit-Revision: 329935 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:51 -0000 Author: dim Date: Sat Feb 24 21:27:48 2018 New Revision: 329935 URL: https://svnweb.freebsd.org/changeset/base/329935 Log: Tag compiler-rt release_60 branch r325932. Added: vendor/compiler-rt/compiler-rt-release_60-r325932/ - copied from r329934, vendor/compiler-rt/dist-release_60/ From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:47 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C392F23EFD; Sat, 24 Feb 2018 21:27:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 95C9B77C81; Sat, 24 Feb 2018 21:27:46 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AB80C2B061; Sat, 24 Feb 2018 21:27:45 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRjjg009021; Sat, 24 Feb 2018 21:27:45 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRjt2009020; Sat, 24 Feb 2018 21:27:45 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRjt2009020@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329934 - vendor/clang/clang-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/clang/clang-release_60-r325932 X-SVN-Commit-Revision: 329934 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:47 -0000 Author: dim Date: Sat Feb 24 21:27:45 2018 New Revision: 329934 URL: https://svnweb.freebsd.org/changeset/base/329934 Log: Tag clang release_60 branch r325932. Added: vendor/clang/clang-release_60-r325932/ - copied from r329933, vendor/clang/dist-release_60/ From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:56 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6C3BF23FB8; Sat, 24 Feb 2018 21:27:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBBFC77E22; Sat, 24 Feb 2018 21:27:55 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A172F2B064; Sat, 24 Feb 2018 21:27:55 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRteY009167; Sat, 24 Feb 2018 21:27:55 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRtXQ009162; Sat, 24 Feb 2018 21:27:55 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRtXQ009162@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329937 - in vendor/lld/dist-release_60: ELF docs test/ELF X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/lld/dist-release_60: ELF docs test/ELF X-SVN-Commit-Revision: 329937 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:57 -0000 Author: dim Date: Sat Feb 24 21:27:54 2018 New Revision: 329937 URL: https://svnweb.freebsd.org/changeset/base/329937 Log: Vendor import of lld release_60 branch r325932: https://llvm.org/svn/llvm-project/lld/branches/release_60@325932 Added: vendor/lld/dist-release_60/test/ELF/relocatable-versioned.s (contents, props changed) Modified: vendor/lld/dist-release_60/ELF/Driver.cpp vendor/lld/dist-release_60/ELF/Options.td vendor/lld/dist-release_60/docs/ReleaseNotes.rst vendor/lld/dist-release_60/test/ELF/pie.s Modified: vendor/lld/dist-release_60/ELF/Driver.cpp ============================================================================== --- vendor/lld/dist-release_60/ELF/Driver.cpp Sat Feb 24 21:27:52 2018 (r329936) +++ vendor/lld/dist-release_60/ELF/Driver.cpp Sat Feb 24 21:27:54 2018 (r329937) @@ -638,7 +638,7 @@ void LinkerDriver::readConfigs(opt::InputArgList &Args Config->Optimize = args::getInteger(Args, OPT_O, 1); Config->OrphanHandling = getOrphanHandling(Args); Config->OutputFile = Args.getLastArgValue(OPT_o); - Config->Pie = Args.hasFlag(OPT_pie, OPT_no_pie, false); + Config->Pie = Args.hasFlag(OPT_pie, OPT_nopie, false); Config->PrintGcSections = Args.hasFlag(OPT_print_gc_sections, OPT_no_print_gc_sections, false); Config->Rpath = getRpath(Args); @@ -1061,7 +1061,12 @@ template void LinkerDriver::link(opt::Inp addReservedSymbols(); // Apply version scripts. - Symtab->scanVersionScript(); + // + // For a relocatable output, version scripts don't make sense, and + // parsing a symbol version string (e.g. dropping "@ver1" from a symbol + // name "foo@ver1") rather do harm, so we don't call this if -r is given. + if (!Config->Relocatable) + Symtab->scanVersionScript(); // Create wrapped symbols for -wrap option. for (auto *Arg : Args.filtered(OPT_wrap)) Modified: vendor/lld/dist-release_60/ELF/Options.td ============================================================================== --- vendor/lld/dist-release_60/ELF/Options.td Sat Feb 24 21:27:52 2018 (r329936) +++ vendor/lld/dist-release_60/ELF/Options.td Sat Feb 24 21:27:54 2018 (r329937) @@ -202,8 +202,6 @@ def no_gnu_unique: F<"no-gnu-unique">, def no_merge_exidx_entries: F<"no-merge-exidx-entries">, HelpText<"Disable merging .ARM.exidx entries">; -def no_pie: F<"no-pie">, HelpText<"Do not create a position independent executable">; - def no_threads: F<"no-threads">, HelpText<"Do not run the linker multi-threaded">; @@ -212,6 +210,8 @@ def no_whole_archive: F<"no-whole-archive">, def noinhibit_exec: F<"noinhibit-exec">, HelpText<"Retain the executable output file whenever it is still usable">; + +def nopie: F<"nopie">, HelpText<"Do not create a position independent executable">; def no_omagic: Flag<["--"], "no-omagic">, MetaVarName<"">, HelpText<"Do not set the text data sections to be writable">; Modified: vendor/lld/dist-release_60/docs/ReleaseNotes.rst ============================================================================== --- vendor/lld/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:27:52 2018 (r329936) +++ vendor/lld/dist-release_60/docs/ReleaseNotes.rst Sat Feb 24 21:27:54 2018 (r329937) @@ -5,17 +5,12 @@ LLD 6.0.0 Release Notes .. contents:: :local: -.. warning:: - These are in-progress notes for the upcoming LLVM 6.0.0 release. - Release notes for previous releases can be found on - `the Download Page `_. - Introduction ============ -This document contains the release notes for the LLD linker, release 6.0.0. -Here we describe the status of LLD, including major improvements -from the previous release. All LLD releases may be downloaded +This document contains the release notes for the lld linker, release 6.0.0. +Here we describe the status of lld, including major improvements +from the previous release. All lld releases may be downloaded from the `LLVM releases web site `_. Non-comprehensive list of changes in this release @@ -24,20 +19,81 @@ Non-comprehensive list of changes in this release ELF Improvements ---------------- -* Item 1. +* A lot of bugs and compatibility issues have been identified and fixed as a + result of people using lld 5.0 as a standard system linker. In particular, + linker script and version script support has significantly improved that + it should be able to handle almost all scripts. +* A mitigation for Spectre v2 has been implemented. If you pass ``-z + retpolineplt``, lld uses RET instruction instead of JMP instruction in PLT. + The option is available for x86 and x86-64. + +* Identical Code Folding (ICF) now de-duplicates .eh_frame entries, so lld now + generates slightly smaller outputs than before when you pass ``--icf=all``. + +* Analysis for ``--as-needed`` is now done after garbage collection. If garbage + collector eliminates all sections that use some library, that library is + eliminated from DT_NEEDED tags. Previously, the analysis ran before garbage + collection. + +* Size of code segment is now always rounded up to page size to make sure that + unused bytes at end of code segment is filled with trap instructions (such + as INT3) instead of zeros. + +* lld is now able to generate Android-style compact dynamic relocation table. + You can turn on the feature by passing ``--pack-dyn-relocs=android``. + +* Debug information is used in more cases when reporting errors. + +* ``--gdb-index`` gets faster than before. + +* String merging is now multi-threaded, which makes ``-O2`` faster. + +* ``--hash-style=both`` is now default instead of ``--hash-style=sysv`` to + match the behavior of recent versions of GNU linkers. + +* ARM PLT entries automatically use short or long variants. + +* lld can now identify and patch a code sequence that triggers AArch64 errata 843419. + Add ``--fix-cortex-a53-843419`` to enable the feature. + +* lld can now generate thunks for out of range thunks. + +* MIPS port now generates all output dynamic relocations using Elf_Rel format only. + +* Added handling of the R_MIPS_26 relocation in case of N32/N64 ABIs and + generating proper PLT entries. + +* The following options have been added: ``--icf=none`` ``-z muldefs`` + ``--plugin-opt`` ``--no-eh-frame-hdr`` ``--no-gdb-index`` + ``--orphan-handling={place,discard,warn,error}`` + ``--pack-dyn-relocs={none,android}`` ``--no-omagic`` + ``--no-print-gc-sections`` ``--ignore-function-address-equality`` ``-z + retpolineplt`` ``--print-icf-sections`` ``--no-pie`` + COFF Improvements ----------------- * A GNU ld style frontend for the COFF linker has been added for MinGW. In MinGW environments, the linker is invoked with GNU ld style parameters; - which LLD previously only supported when used as an ELF linker. When + which lld previously only supported when used as an ELF linker. When a PE/COFF target is chosen, those parameters are rewritten into the lld-link style parameters and the COFF linker is invoked instead. * Initial support for the ARM64 architecture has been added. -MachO Improvements ------------------- +* New ``--version`` flag. -* Item 1. +* Significantly improved support for writing PDB Files. + +* New ``--rsp-quoting`` flag, like ``clang-cl``. + +* ``/manifestuac:no`` no longer incorrectly disables ``/manifestdependency:``. + +* Only write ``.manifest`` files if ``/manifest`` is passed. + +WebAssembly Improvements +------------------------ + +* Initial version of WebAssembly support has landed. You can invoke the + WebAssembly linker by ``wasm-ld``. Modified: vendor/lld/dist-release_60/test/ELF/pie.s ============================================================================== --- vendor/lld/dist-release_60/test/ELF/pie.s Sat Feb 24 21:27:52 2018 (r329936) +++ vendor/lld/dist-release_60/test/ELF/pie.s Sat Feb 24 21:27:54 2018 (r329937) @@ -48,7 +48,7 @@ # CHECK: Type: PT_DYNAMIC ## Check -nopie -# RUN: ld.lld -no-pie %t1.o -o %t2 +# RUN: ld.lld -nopie %t1.o -o %t2 # RUN: llvm-readobj -file-headers -r %t2 | FileCheck %s --check-prefix=NOPIE # NOPIE-NOT: Type: SharedObject Added: vendor/lld/dist-release_60/test/ELF/relocatable-versioned.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist-release_60/test/ELF/relocatable-versioned.s Sat Feb 24 21:27:54 2018 (r329937) @@ -0,0 +1,9 @@ +# REQUIRES: x86 +# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t1.o +# RUN: ld.lld -o %t2.o -r %t1.o +# RUN: llvm-nm %t2.o | FileCheck %s +# CHECK: foo@VERSION + +.global "foo@VERSION" +"foo@VERSION": + ret From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:27:56 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B240F23FB7; Sat, 24 Feb 2018 21:27:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0540A77DF2; Sat, 24 Feb 2018 21:27:53 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E56F52B063; Sat, 24 Feb 2018 21:27:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRqmN009116; Sat, 24 Feb 2018 21:27:52 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRqjJ009115; Sat, 24 Feb 2018 21:27:52 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRqjJ009115@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329936 - vendor/libc++/libc++-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libc++/libc++-release_60-r325932 X-SVN-Commit-Revision: 329936 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:27:56 -0000 Author: dim Date: Sat Feb 24 21:27:52 2018 New Revision: 329936 URL: https://svnweb.freebsd.org/changeset/base/329936 Log: Tag libc++ release_60 branch r325932. Added: vendor/libc++/libc++-release_60-r325932/ - copied from r329935, vendor/libc++/dist-release_60/ From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:28:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 118B0F24015; Sat, 24 Feb 2018 21:28:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B365C77F0E; Sat, 24 Feb 2018 21:28:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0D9132B065; Sat, 24 Feb 2018 21:28:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLRxJQ009216; Sat, 24 Feb 2018 21:27:59 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLRx5B009215; Sat, 24 Feb 2018 21:27:59 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242127.w1OLRx5B009215@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:27:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329938 - vendor/lld/lld-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lld/lld-release_60-r325932 X-SVN-Commit-Revision: 329938 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:28:08 -0000 Author: dim Date: Sat Feb 24 21:27:59 2018 New Revision: 329938 URL: https://svnweb.freebsd.org/changeset/base/329938 Log: Tag lld release_60 branch r325932. Added: vendor/lld/lld-release_60-r325932/ - copied from r329937, vendor/lld/dist-release_60/ From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:28:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E771F24018; Sat, 24 Feb 2018 21:28:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 68E6677F62; Sat, 24 Feb 2018 21:28:03 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F07472B068; Sat, 24 Feb 2018 21:28:02 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLS2RN009271; Sat, 24 Feb 2018 21:28:02 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLS2QO009266; Sat, 24 Feb 2018 21:28:02 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242128.w1OLS2QO009266@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:28:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329939 - vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests X-SVN-Commit-Revision: 329939 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:28:08 -0000 Author: dim Date: Sat Feb 24 21:28:02 2018 New Revision: 329939 URL: https://svnweb.freebsd.org/changeset/base/329939 Log: Vendor import of lldb release_60 branch r325932: https://llvm.org/svn/llvm-project/lldb/branches/release_60@325932 Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.cpp vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.h vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.cpp vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.h vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/ThreadIdsInJstopinfoTest.cpp Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.cpp ============================================================================== --- vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.cpp Sat Feb 24 21:27:59 2018 (r329938) +++ vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.cpp Sat Feb 24 21:28:02 2018 (r329939) @@ -8,16 +8,17 @@ //===----------------------------------------------------------------------===// #include "MessageObjects.h" -#include "lldb/Utility/StructuredData.h" +#include "lldb/Interpreter/Args.h" +#include "lldb/Utility/StringExtractor.h" #include "llvm/ADT/StringExtras.h" #include "gtest/gtest.h" using namespace lldb_private; +using namespace lldb; using namespace llvm; -using namespace llvm::support; namespace llgs_tests { -Expected ProcessInfo::Create(StringRef response) { +Expected ProcessInfo::create(StringRef response) { ProcessInfo process_info; auto elements_or_error = SplitUniquePairList("ProcessInfo", response); if (!elements_or_error) @@ -53,36 +54,50 @@ Expected ProcessInfo::Create(StringRef re lldb::pid_t ProcessInfo::GetPid() const { return m_pid; } -endianness ProcessInfo::GetEndian() const { return m_endian; } +support::endianness ProcessInfo::GetEndian() const { return m_endian; } //====== ThreadInfo ============================================================ -ThreadInfo::ThreadInfo(StringRef name, StringRef reason, - const RegisterMap ®isters, unsigned int signal) - : m_name(name.str()), m_reason(reason.str()), m_registers(registers), - m_signal(signal) {} +ThreadInfo::ThreadInfo(StringRef name, StringRef reason, RegisterMap registers, + unsigned int signal) + : m_name(name.str()), m_reason(reason.str()), + m_registers(std::move(registers)), m_signal(signal) {} -StringRef ThreadInfo::ReadRegister(unsigned int register_id) const { - return m_registers.lookup(register_id); +const RegisterValue *ThreadInfo::ReadRegister(unsigned int Id) const { + auto Iter = m_registers.find(Id); + return Iter == m_registers.end() ? nullptr : &Iter->getSecond(); } -Expected -ThreadInfo::ReadRegisterAsUint64(unsigned int register_id) const { - uint64_t value; - std::string value_str(m_registers.lookup(register_id)); - if (!llvm::to_integer(value_str, value, 16)) - return make_parsing_error("ThreadInfo value for register {0}: {1}", - register_id, value_str); +//====== JThreadsInfo ========================================================== - sys::swapByteOrder(value); - return value; +Expected +JThreadsInfo::parseRegisters(const StructuredData::Dictionary &Dict, + ArrayRef RegInfos) { + RegisterMap Result; + + auto KeysObj = Dict.GetKeys(); + auto Keys = KeysObj->GetAsArray(); + for (size_t i = 0; i < Keys->GetSize(); i++) { + StringRef KeyStr, ValueStr; + Keys->GetItemAtIndexAsString(i, KeyStr); + Dict.GetValueForKeyAsString(KeyStr, ValueStr); + unsigned int Register; + if (!llvm::to_integer(KeyStr, Register, 10)) + return make_parsing_error("JThreadsInfo: register key[{0}]", i); + + auto RegValOr = + parseRegisterValue(RegInfos[Register], ValueStr, support::big); + if (!RegValOr) + return RegValOr.takeError(); + Result[Register] = std::move(*RegValOr); + } + return std::move(Result); } -//====== JThreadsInfo ========================================================== -Expected JThreadsInfo::Create(StringRef response, - endianness endian) { +Expected JThreadsInfo::create(StringRef Response, + ArrayRef RegInfos) { JThreadsInfo jthreads_info; - StructuredData::ObjectSP json = StructuredData::ParseJSON(response); + StructuredData::ObjectSP json = StructuredData::ParseJSON(Response); StructuredData::Array *array = json->GetAsArray(); if (!array) return make_parsing_error("JThreadsInfo: JSON array"); @@ -106,23 +121,11 @@ Expected JThreadsInfo::Create(StringRef if (!register_dict) return make_parsing_error("JThreadsInfo: registers JSON obj"); - RegisterMap registers; - - auto keys_obj = register_dict->GetKeys(); - auto keys = keys_obj->GetAsArray(); - for (size_t i = 0; i < keys->GetSize(); i++) { - StringRef key_str, value_str; - keys->GetItemAtIndexAsString(i, key_str); - register_dict->GetValueForKeyAsString(key_str, value_str); - unsigned int register_id; - if (key_str.getAsInteger(10, register_id)) - return make_parsing_error("JThreadsInfo: register key[{0}]", i); - - registers[register_id] = value_str.str(); - } - + auto RegsOr = parseRegisters(*register_dict, RegInfos); + if (!RegsOr) + return RegsOr.takeError(); jthreads_info.m_thread_infos[tid] = - ThreadInfo(name, reason, registers, signal); + ThreadInfo(name, reason, std::move(*RegsOr), signal); } return jthreads_info; @@ -132,20 +135,130 @@ const ThreadInfoMap &JThreadsInfo::GetThreadInfos() co return m_thread_infos; } +Expected RegisterInfoParser::create(StringRef Response) { + auto ElementsOr = SplitUniquePairList("RegisterInfoParser", Response); + if (!ElementsOr) + return ElementsOr.takeError(); + auto &Elements = *ElementsOr; + + RegisterInfo Info = { + nullptr, // Name + nullptr, // Alt name + 0, // byte size + 0, // offset + eEncodingUint, // encoding + eFormatHex, // format + { + LLDB_INVALID_REGNUM, // eh_frame reg num + LLDB_INVALID_REGNUM, // DWARF reg num + LLDB_INVALID_REGNUM, // generic reg num + LLDB_INVALID_REGNUM, // process plugin reg num + LLDB_INVALID_REGNUM // native register number + }, + NULL, + NULL, + NULL, // Dwarf expression opcode bytes pointer + 0 // Dwarf expression opcode bytes length + }; + Info.name = ConstString(Elements["name"]).GetCString(); + if (!Info.name) + return make_parsing_error("qRegisterInfo: name"); + + Info.alt_name = ConstString(Elements["alt-name"]).GetCString(); + + if (!to_integer(Elements["bitsize"], Info.byte_size, 10)) + return make_parsing_error("qRegisterInfo: bit-size"); + Info.byte_size /= CHAR_BIT; + + if (!to_integer(Elements["offset"], Info.byte_offset, 10)) + return make_parsing_error("qRegisterInfo: offset"); + + Info.encoding = Args::StringToEncoding(Elements["encoding"]); + if (Info.encoding == eEncodingInvalid) + return make_parsing_error("qRegisterInfo: encoding"); + + Info.format = StringSwitch(Elements["format"]) + .Case("binary", eFormatBinary) + .Case("decimal", eFormatDecimal) + .Case("hex", eFormatHex) + .Case("float", eFormatFloat) + .Case("vector-sint8", eFormatVectorOfSInt8) + .Case("vector-uint8", eFormatVectorOfUInt8) + .Case("vector-sint16", eFormatVectorOfSInt16) + .Case("vector-uint16", eFormatVectorOfUInt16) + .Case("vector-sint32", eFormatVectorOfSInt32) + .Case("vector-uint32", eFormatVectorOfUInt32) + .Case("vector-float32", eFormatVectorOfFloat32) + .Case("vector-uint64", eFormatVectorOfUInt64) + .Case("vector-uint128", eFormatVectorOfUInt128) + .Default(eFormatInvalid); + if (Info.format == eFormatInvalid) + return make_parsing_error("qRegisterInfo: format"); + + Info.kinds[eRegisterKindGeneric] = + Args::StringToGenericRegister(Elements["generic"]); + + return std::move(Info); +} + +Expected parseRegisterValue(const RegisterInfo &Info, + StringRef HexValue, + llvm::support::endianness Endian) { + SmallVector Bytes(HexValue.size() / 2); + StringExtractor(HexValue).GetHexBytes(Bytes, '\xcc'); + RegisterValue Value; + Status ST; + Value.SetFromMemoryData( + &Info, Bytes.data(), Bytes.size(), + Endian == support::little ? eByteOrderLittle : eByteOrderBig, ST); + if (ST.Fail()) + return ST.ToError(); + return Value; +} + //====== StopReply ============================================================= Expected> -StopReply::create(StringRef Response, llvm::support::endianness Endian) { +StopReply::create(StringRef Response, llvm::support::endianness Endian, + ArrayRef RegInfos) { if (Response.size() < 3) return make_parsing_error("StopReply: Invalid packet"); if (Response.consume_front("T")) - return StopReplyStop::create(Response, Endian); + return StopReplyStop::create(Response, Endian, RegInfos); if (Response.consume_front("W")) return StopReplyExit::create(Response); return make_parsing_error("StopReply: Invalid packet"); } +Expected StopReplyStop::parseRegisters( + const StringMap> &Elements, + support::endianness Endian, ArrayRef RegInfos) { + + RegisterMap Result; + for (const auto &E : Elements) { + StringRef Key = E.getKey(); + const auto &Val = E.getValue(); + if (Key.size() != 2) + continue; + + unsigned int Reg; + if (!to_integer(Key, Reg, 16)) + continue; + + if (Val.size() != 1) + return make_parsing_error( + "StopReplyStop: multiple entries for register field [{0:x}]", Reg); + + auto RegValOr = parseRegisterValue(RegInfos[Reg], Val[0], Endian); + if (!RegValOr) + return RegValOr.takeError(); + Result[Reg] = std::move(*RegValOr); + } + return std::move(Result); +} + Expected> -StopReplyStop::create(StringRef Response, llvm::support::endianness Endian) { +StopReplyStop::create(StringRef Response, support::endianness Endian, + ArrayRef RegInfos) { unsigned int Signal; StringRef SignalStr = Response.take_front(2); Response = Response.drop_front(2); @@ -176,40 +289,31 @@ StopReplyStop::create(StringRef Response, llvm::suppor if (Threads.size() != Pcs.size()) return make_parsing_error("StopReply: thread/PC count mismatch"); - U64Map ThreadPcs; + RegisterMap ThreadPcs; + const RegisterInfo *PcInfo = find_if(RegInfos, [](const RegisterInfo &Info) { + return Info.kinds[eRegisterKindGeneric] == LLDB_REGNUM_GENERIC_PC; + }); + assert(PcInfo); + for (auto ThreadPc : zip(Threads, Pcs)) { lldb::tid_t Id; - uint64_t Pc; if (!to_integer(std::get<0>(ThreadPc), Id, 16)) return make_parsing_error("StopReply: Thread id '{0}'", std::get<0>(ThreadPc)); - if (!to_integer(std::get<1>(ThreadPc), Pc, 16)) - return make_parsing_error("StopReply Thread Pc '{0}'", - std::get<1>(ThreadPc)); - ThreadPcs[Id] = Pc; + auto PcOr = parseRegisterValue(*PcInfo, std::get<1>(ThreadPc), Endian); + if (!PcOr) + return PcOr.takeError(); + ThreadPcs[Id] = std::move(*PcOr); } - RegisterMap Registers; - for (const auto &E : Elements) { - StringRef Key = E.getKey(); - const auto &Val = E.getValue(); - if (Key.size() != 2) - continue; + auto RegistersOr = parseRegisters(Elements, Endian, RegInfos); + if (!RegistersOr) + return RegistersOr.takeError(); - unsigned int Reg; - if (!to_integer(Key, Reg, 16)) - continue; - - if (Val.size() != 1) - return make_parsing_error( - "StopReply: multiple entries for register field [{0:x}]", Reg); - - Registers[Reg] = Val[0].str(); - } - - return llvm::make_unique(Signal, Thread, Name, ThreadPcs, - Registers, Reason); + return llvm::make_unique(Signal, Thread, Name, + std::move(ThreadPcs), + std::move(*RegistersOr), Reason); } Expected> @@ -251,3 +355,12 @@ StringMap> SplitPairList(Str return pairs; } } // namespace llgs_tests + +std::ostream &lldb_private::operator<<(std::ostream &OS, + const RegisterValue &RegVal) { + ArrayRef Bytes(static_cast(RegVal.GetBytes()), + RegVal.GetByteSize()); + return OS << formatv("RegisterValue[{0}]: {1:@[x-2]}", RegVal.GetByteSize(), + make_range(Bytes.begin(), Bytes.end())) + .str(); +} Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.h ============================================================================== --- vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.h Sat Feb 24 21:27:59 2018 (r329938) +++ vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/MessageObjects.h Sat Feb 24 21:28:02 2018 (r329939) @@ -10,7 +10,9 @@ #ifndef LLDB_SERVER_TESTS_MESSAGEOBJECTS_H #define LLDB_SERVER_TESTS_MESSAGEOBJECTS_H +#include "lldb/Core/RegisterValue.h" #include "lldb/Host/Host.h" +#include "lldb/Utility/StructuredData.h" #include "lldb/lldb-types.h" #include "llvm/ADT/DenseMap.h" #include "llvm/ADT/SmallString.h" @@ -22,12 +24,13 @@ namespace llgs_tests { class ThreadInfo; typedef llvm::DenseMap ThreadInfoMap; -typedef llvm::DenseMap U64Map; -typedef llvm::DenseMap RegisterMap; +typedef llvm::DenseMap RegisterMap; -class ProcessInfo { +template struct Parser { using result_type = T; }; + +class ProcessInfo : public Parser { public: - static llvm::Expected Create(llvm::StringRef response); + static llvm::Expected create(llvm::StringRef response); lldb::pid_t GetPid() const; llvm::support::endianness GetEndian() const; @@ -49,10 +52,9 @@ class ThreadInfo { public: ThreadInfo() = default; ThreadInfo(llvm::StringRef name, llvm::StringRef reason, - const RegisterMap ®isters, unsigned int signal); + RegisterMap registers, unsigned int signal); - llvm::StringRef ReadRegister(unsigned int register_id) const; - llvm::Expected ReadRegisterAsUint64(unsigned int register_id) const; + const lldb_private::RegisterValue *ReadRegister(unsigned int Id) const; private: std::string m_name; @@ -61,25 +63,40 @@ class ThreadInfo { unsigned int m_signal; }; -class JThreadsInfo { +class JThreadsInfo : public Parser { public: - static llvm::Expected Create(llvm::StringRef response, - llvm::support::endianness endian); + static llvm::Expected + create(llvm::StringRef Response, + llvm::ArrayRef RegInfos); const ThreadInfoMap &GetThreadInfos() const; private: + static llvm::Expected + parseRegisters(const lldb_private::StructuredData::Dictionary &Dict, + llvm::ArrayRef RegInfos); + JThreadsInfo() = default; ThreadInfoMap m_thread_infos; }; +struct RegisterInfoParser : public Parser { + static llvm::Expected + create(llvm::StringRef Response); +}; + +llvm::Expected +parseRegisterValue(const lldb_private::RegisterInfo &Info, + llvm::StringRef HexValue, llvm::support::endianness Endian); + class StopReply { public: StopReply() = default; virtual ~StopReply() = default; static llvm::Expected> - create(llvm::StringRef response, llvm::support::endianness endian); + create(llvm::StringRef Response, llvm::support::endianness Endian, + llvm::ArrayRef RegInfos); // for llvm::cast<> virtual lldb_private::WaitStatus getKind() const = 0; @@ -91,15 +108,17 @@ class StopReply { (public) class StopReplyStop : public StopReply { public: StopReplyStop(uint8_t Signal, lldb::tid_t ThreadId, llvm::StringRef Name, - U64Map ThreadPcs, RegisterMap Registers, llvm::StringRef Reason) + RegisterMap ThreadPcs, RegisterMap Registers, + llvm::StringRef Reason) : Signal(Signal), ThreadId(ThreadId), Name(Name), ThreadPcs(std::move(ThreadPcs)), Registers(std::move(Registers)), Reason(Reason) {} static llvm::Expected> - create(llvm::StringRef response, llvm::support::endianness endian); + create(llvm::StringRef Response, llvm::support::endianness Endian, + llvm::ArrayRef RegInfos); - const U64Map &getThreadPcs() const { return ThreadPcs; } + const RegisterMap &getThreadPcs() const { return ThreadPcs; } lldb::tid_t getThreadId() const { return ThreadId; } // for llvm::cast<> @@ -111,10 +130,15 @@ class StopReplyStop : public StopReply { (public) } private: + static llvm::Expected parseRegisters( + const llvm::StringMap> &Elements, + llvm::support::endianness Endian, + llvm::ArrayRef RegInfos); + uint8_t Signal; lldb::tid_t ThreadId; std::string Name; - U64Map ThreadPcs; + RegisterMap ThreadPcs; RegisterMap Registers; std::string Reason; }; @@ -155,5 +179,9 @@ llvm::Error make_parsing_error(llvm::StringRef format, } } // namespace llgs_tests + +namespace lldb_private { +std::ostream &operator<<(std::ostream &OS, const RegisterValue &RegVal); +} #endif // LLDB_SERVER_TESTS_MESSAGEOBJECTS_H Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.cpp ============================================================================== --- vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.cpp Sat Feb 24 21:27:59 2018 (r329938) +++ vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.cpp Sat Feb 24 21:28:02 2018 (r329939) @@ -25,9 +25,8 @@ using namespace lldb; using namespace lldb_private; using namespace llvm; +using namespace llgs_tests; -namespace llgs_tests { - TestClient::TestClient(std::unique_ptr Conn) { SetConnection(Conn.release()); @@ -106,7 +105,7 @@ Expected> TestClient::laun auto Client = std::unique_ptr(new TestClient(std::move(Conn))); if (!InferiorArgs.empty()) { - if (Error E = Client->QueryProcessInfo()) + if (Error E = Client->queryProcess()) return std::move(E); } @@ -136,7 +135,7 @@ Error TestClient::SetInferior(llvm::ArrayRef TestClient::GetJThreadsInfo() { - std::string response; - if (SendMessage("jThreadsInfo", response)) - return llvm::None; - auto creation = JThreadsInfo::Create(response, m_process_info->GetEndian()); - if (auto create_error = creation.takeError()) { - GTEST_LOG_(ERROR) << toString(std::move(create_error)); - return llvm::None; - } - - return std::move(*creation); +Expected TestClient::GetJThreadsInfo() { + return SendMessage("jThreadsInfo", m_register_infos); } const StopReply &TestClient::GetLatestStopReply() { @@ -209,42 +201,42 @@ Error TestClient::SendMessage(StringRef message, std:: } unsigned int TestClient::GetPcRegisterId() { - if (m_pc_register != UINT_MAX) - return m_pc_register; + assert(m_pc_register != LLDB_INVALID_REGNUM); + return m_pc_register; +} - for (unsigned int register_id = 0;; register_id++) { - std::string message = formatv("qRegisterInfo{0:x-}", register_id).str(); - std::string response; - if (SendMessage(message, response)) { - GTEST_LOG_(ERROR) << "Unable to query register ID for PC register."; - return UINT_MAX; - } +Error TestClient::qProcessInfo() { + m_process_info = None; + auto InfoOr = SendMessage("qProcessInfo"); + if (!InfoOr) + return InfoOr.takeError(); + m_process_info = std::move(*InfoOr); + return Error::success(); +} - auto elements_or_error = SplitUniquePairList("GetPcRegisterId", response); - if (auto split_error = elements_or_error.takeError()) { - GTEST_LOG_(ERROR) << "GetPcRegisterId: Error splitting response: " - << response; - return UINT_MAX; - } - - auto elements = *elements_or_error; - if (elements["alt-name"] == "pc" || elements["generic"] == "pc") { - m_pc_register = register_id; +Error TestClient::qRegisterInfos() { + for (unsigned int Reg = 0;; ++Reg) { + std::string Message = formatv("qRegisterInfo{0:x-}", Reg).str(); + Expected InfoOr = SendMessage(Message); + if (!InfoOr) { + consumeError(InfoOr.takeError()); break; } + m_register_infos.emplace_back(std::move(*InfoOr)); + if (m_register_infos[Reg].kinds[eRegisterKindGeneric] == + LLDB_REGNUM_GENERIC_PC) + m_pc_register = Reg; } - - return m_pc_register; + if (m_pc_register == LLDB_INVALID_REGNUM) + return make_parsing_error("qRegisterInfo: generic"); + return Error::success(); } -llvm::Error TestClient::QueryProcessInfo() { - std::string response; - if (Error E = SendMessage("qProcessInfo", response)) +Error TestClient::queryProcess() { + if (Error E = qProcessInfo()) return E; - auto create_or_error = ProcessInfo::Create(response); - if (!create_or_error) - return create_or_error.takeError(); - m_process_info = *create_or_error; + if (Error E = qRegisterInfos()) + return E; return Error::success(); } @@ -254,7 +246,8 @@ Error TestClient::Continue(StringRef message) { std::string response; if (Error E = SendMessage(message, response)) return E; - auto creation = StopReply::create(response, m_process_info->GetEndian()); + auto creation = StopReply::create(response, m_process_info->GetEndian(), + m_register_infos); if (Error E = creation.takeError()) return E; @@ -273,5 +266,3 @@ Error TestClient::Continue(StringRef message) { } return Error::success(); } - -} // namespace llgs_tests Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.h ============================================================================== --- vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.h Sat Feb 24 21:27:59 2018 (r329938) +++ vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/TestClient.h Sat Feb 24 21:28:02 2018 (r329939) @@ -59,7 +59,7 @@ class TestClient (public) llvm::Error ContinueAll(); llvm::Error ContinueThread(unsigned long thread_id); const ProcessInfo &GetProcessInfo(); - llvm::Optional GetJThreadsInfo(); + llvm::Expected GetJThreadsInfo(); const StopReply &GetLatestStopReply(); template llvm::Expected GetLatestStopReplyAs() { assert(m_stop_reply); @@ -74,12 +74,18 @@ class TestClient (public) std::string &response_string); llvm::Error SendMessage(llvm::StringRef message, std::string &response_string, PacketResult expected_result); + + template + llvm::Expected SendMessage(llvm::StringRef Message, + CreateArgs &&... Args); unsigned int GetPcRegisterId(); private: TestClient(std::unique_ptr Conn); - llvm::Error QueryProcessInfo(); + llvm::Error qProcessInfo(); + llvm::Error qRegisterInfos(); + llvm::Error queryProcess(); llvm::Error Continue(llvm::StringRef message); std::string FormatFailedResult( const std::string &message, @@ -88,8 +94,18 @@ class TestClient (public) llvm::Optional m_process_info; std::unique_ptr m_stop_reply; - unsigned int m_pc_register = UINT_MAX; + std::vector m_register_infos; + unsigned int m_pc_register = LLDB_INVALID_REGNUM; }; + +template +llvm::Expected +TestClient::SendMessage(llvm::StringRef Message, CreateArgs &&... Args) { + std::string ResponseText; + if (llvm::Error E = SendMessage(Message, ResponseText)) + return std::move(E); + return P::create(ResponseText, std::forward(Args)...); +} } // namespace llgs_tests Modified: vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/ThreadIdsInJstopinfoTest.cpp ============================================================================== --- vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/ThreadIdsInJstopinfoTest.cpp Sat Feb 24 21:27:59 2018 (r329938) +++ vendor/lldb/dist-release_60/unittests/tools/lldb-server/tests/ThreadIdsInJstopinfoTest.cpp Sat Feb 24 21:28:02 2018 (r329939) @@ -9,13 +9,19 @@ #include "TestBase.h" #include "TestClient.h" +#include "lldb/Utility/DataExtractor.h" +#include "llvm/Support/FormatVariadic.h" #include "llvm/Support/Path.h" #include "llvm/Testing/Support/Error.h" +#include "gmock/gmock.h" #include "gtest/gtest.h" #include using namespace llgs_tests; +using namespace lldb_private; using namespace llvm; +using namespace lldb; +using namespace testing; TEST_F(StandardStartupTest, TestStopReplyContainsThreadPcs) { // This inferior spawns 4 threads, then forces a break. @@ -29,7 +35,7 @@ TEST_F(StandardStartupTest, TestStopReplyContainsThrea ASSERT_NE(pc_reg, UINT_MAX); auto jthreads_info = Client->GetJThreadsInfo(); - ASSERT_TRUE(jthreads_info); + ASSERT_THAT_EXPECTED(jthreads_info, Succeeded()); auto stop_reply = Client->GetLatestStopReplyAs(); ASSERT_THAT_EXPECTED(stop_reply, Succeeded()); @@ -42,9 +48,7 @@ TEST_F(StandardStartupTest, TestStopReplyContainsThrea unsigned long tid = stop_reply_pc.first; ASSERT_TRUE(thread_infos.find(tid) != thread_infos.end()) << "Thread ID: " << tid << " not in JThreadsInfo."; - auto pc_value = thread_infos[tid].ReadRegisterAsUint64(pc_reg); - ASSERT_THAT_EXPECTED(pc_value, Succeeded()); - ASSERT_EQ(stop_reply_pcs[tid], *pc_value) - << "Mismatched PC for thread: " << tid; + EXPECT_THAT(thread_infos[tid].ReadRegister(pc_reg), + Pointee(Eq(stop_reply_pc.second))); } } From owner-svn-src-vendor@freebsd.org Sat Feb 24 21:28:09 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86322F24029; Sat, 24 Feb 2018 21:28:09 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4134377FAD; Sat, 24 Feb 2018 21:28:05 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A7EB52B069; Sat, 24 Feb 2018 21:28:05 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1OLS5eK009319; Sat, 24 Feb 2018 21:28:05 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1OLS510009318; Sat, 24 Feb 2018 21:28:05 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201802242128.w1OLS510009318@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 24 Feb 2018 21:28:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r329940 - vendor/lldb/lldb-release_60-r325932 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lldb/lldb-release_60-r325932 X-SVN-Commit-Revision: 329940 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2018 21:28:09 -0000 Author: dim Date: Sat Feb 24 21:28:05 2018 New Revision: 329940 URL: https://svnweb.freebsd.org/changeset/base/329940 Log: Tag lldb release_60 branch r325932. Added: vendor/lldb/lldb-release_60-r325932/ - copied from r329939, vendor/lldb/dist-release_60/