From owner-dev-commits-src-all@freebsd.org Wed Mar 3 00:45:24 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2515255742D for ; Wed, 3 Mar 2021 00:45:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DqwJh0ZLKz3sLh; Wed, 3 Mar 2021 00:45:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 066902080F; Wed, 3 Mar 2021 00:45:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1230jOFN094390; Wed, 3 Mar 2021 00:45:24 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1230jN9n094389; Wed, 3 Mar 2021 00:45:23 GMT (envelope-from git) Date: Wed, 3 Mar 2021 00:45:23 GMT Message-Id: <202103030045.1230jN9n094389@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org From: Martin Matuska Subject: git: 154ce66101db - vendor/openzfs - Update vendor/openzfs to master-bedbc13da MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mm X-Git-Repository: src X-Git-Refname: refs/heads/vendor/openzfs X-Git-Reftype: branch X-Git-Commit: 154ce66101db0e1a9c90a94e391612b2ab8c048f Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2021 00:45:24 -0000 The branch vendor/openzfs has been updated by mm: URL: https://cgit.FreeBSD.org/src/commit/?id=154ce66101db0e1a9c90a94e391612b2ab8c048f commit 154ce66101db0e1a9c90a94e391612b2ab8c048f Author: Martin Matuska AuthorDate: 2021-03-03 00:40:13 +0000 Commit: Martin Matuska CommitDate: 2021-03-03 00:40:13 +0000 Update vendor/openzfs to master-bedbc13da Notable upstream commits: 8e43fa12c Fix vdev_rebuild_thread deadlock 03ef8f09e Add missing checks for unsupported features 2e160dee9 Fix assert in FreeBSD-specific dmu_read_pages bedbc13da Cancel TRIM / initialize on FAULTED non-writeable vdevs --- cmd/vdev_id/vdev_id | 12 ++- cmd/zpool/Makefile.am | 2 +- cmd/zpool/zpool_main.c | 2 + cmd/ztest/ztest.c | 3 + config/kernel-bio.m4 | 29 ++++++++ config/kernel-generic_io_acct.m4 | 69 +++++++++++------ configure.ac | 1 + include/os/linux/kernel/linux/blkdev_compat.h | 8 +- include/sys/dsl_synctask.h | 9 ++- lib/libzfs/libzfs_sendrecv.c | 9 +++ lib/libzfs/libzfs_status.c | 2 + man/man5/zfs-module-parameters.5 | 2 +- module/os/freebsd/zfs/dmu_os.c | 2 +- module/os/linux/zfs/vdev_disk.c | 4 + module/os/linux/zfs/zfs_ioctl_os.c | 2 +- module/os/linux/zfs/zvol_os.c | 4 + module/zcommon/zfeature_common.c | 2 + module/zfs/spa_misc.c | 31 +++++--- module/zfs/vdev_initialize.c | 10 ++- module/zfs/vdev_rebuild.c | 2 +- module/zfs/vdev_trim.c | 15 +++- module/zfs/zfs_ioctl.c | 4 +- tests/runfiles/common.run | 4 +- tests/zfs-tests/cmd/Makefile.am | 1 + tests/zfs-tests/cmd/send_doall/.gitignore | 1 + tests/zfs-tests/cmd/send_doall/Makefile.am | 11 +++ tests/zfs-tests/cmd/send_doall/send_doall.c | 87 ++++++++++++++++++++++ tests/zfs-tests/include/commands.cfg | 1 + .../cli_root/zpool_initialize/Makefile.am | 1 + ...zpool_initialize_fault_export_import_online.ksh | 59 +++++++++++++++ .../functional/cli_root/zpool_trim/Makefile.am | 1 + .../zpool_trim_fault_export_import_online.ksh | 62 +++++++++++++++ .../zpool_trim/zpool_trim_start_and_cancel_pos.ksh | 22 +++--- tests/zfs-tests/tests/functional/rsend/Makefile.am | 4 +- .../tests/functional/rsend/send_doall.ksh | 67 +++++++++++++++++ 35 files changed, 476 insertions(+), 69 deletions(-) diff --git a/cmd/vdev_id/vdev_id b/cmd/vdev_id/vdev_id index 95a4e483b876..8a379a72690e 100755 --- a/cmd/vdev_id/vdev_id +++ b/cmd/vdev_id/vdev_id @@ -285,7 +285,9 @@ sas_handler() { # we have to append the -part suffix directly in the # helper. if [ "$DEVTYPE" != "partition" ] ; then - PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}') + # Match p[number], remove the 'p' and prepend "-part" + PART=$(echo "$DM_NAME" | + awk 'match($0,/p[0-9]+$/) {print "-part"substr($0,RSTART+1,RLENGTH-1)}') fi # Strip off partition information. @@ -499,7 +501,9 @@ scsi_handler() { # we have to append the -part suffix directly in the # helper. if [ "$DEVTYPE" != "partition" ] ; then - PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}') + # Match p[number], remove the 'p' and prepend "-part" + PART=$(echo "$DM_NAME" | + awk 'match($0,/p[0-9]+$/) {print "-part"substr($0,RSTART+1,RLENGTH-1)}') fi # Strip off partition information. @@ -648,7 +652,9 @@ alias_handler () { DM_PART= if echo "$DM_NAME" | grep -q -E 'p[0-9][0-9]*$' ; then if [ "$DEVTYPE" != "partition" ] ; then - DM_PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}') + # Match p[number], remove the 'p' and prepend "-part" + DM_PART=$(echo "$DM_NAME" | + awk 'match($0,/p[0-9]+$/) {print "-part"substr($0,RSTART+1,RLENGTH-1)}') fi fi diff --git a/cmd/zpool/Makefile.am b/cmd/zpool/Makefile.am index d47051e4fe50..abfa940c3d72 100644 --- a/cmd/zpool/Makefile.am +++ b/cmd/zpool/Makefile.am @@ -179,5 +179,5 @@ install-data-hook: ln -s "$(zpoolexecdir)/$${f}" "$(DESTDIR)$(zpoolconfdir)"; \ done for l in $(zpoolcompatlinks); do \ - (cd "$(DESTDIR)$(zpoolcompatdir)"; ln -s $${l} ); \ + (cd "$(DESTDIR)$(zpoolcompatdir)"; ln -sf $${l} ); \ done diff --git a/cmd/zpool/zpool_main.c b/cmd/zpool/zpool_main.c index 29252e6a24f4..e89eb3bea770 100644 --- a/cmd/zpool/zpool_main.c +++ b/cmd/zpool/zpool_main.c @@ -9030,6 +9030,8 @@ zpool_do_upgrade(int argc, char **argv) "---------------\n"); for (i = 0; i < SPA_FEATURES; i++) { zfeature_info_t *fi = &spa_feature_table[i]; + if (!fi->fi_zfs_mod_supported) + continue; const char *ro = (fi->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ? " (read-only compatible)" : ""; diff --git a/cmd/ztest/ztest.c b/cmd/ztest/ztest.c index cfa1290d78d1..1a030280704a 100644 --- a/cmd/ztest/ztest.c +++ b/cmd/ztest/ztest.c @@ -7592,6 +7592,9 @@ ztest_init(ztest_shared_t *zs) for (i = 0; i < SPA_FEATURES; i++) { char *buf; + if (!spa_feature_table[i].fi_zfs_mod_supported) + continue; + /* * 75% chance of using the log space map feature. We want ztest * to exercise both the code paths that use the log space map diff --git a/config/kernel-bio.m4 b/config/kernel-bio.m4 index 0c533531dceb..d8692bd39999 100644 --- a/config/kernel-bio.m4 +++ b/config/kernel-bio.m4 @@ -369,6 +369,33 @@ AC_DEFUN([ZFS_AC_KERNEL_BLKG_TRYGET], [ ]) ]) +dnl # +dnl # Linux 5.12 API, +dnl # +dnl # The Linux 5.12 kernel updated struct bio to create a new bi_bdev member +dnl # and bio->bi_disk was moved to bio->bi_bdev->bd_disk +dnl # +AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_BDEV_DISK], [ + ZFS_LINUX_TEST_SRC([bio_bdev_disk], [ + #include + #include + ],[ + struct bio *b = NULL; + struct gendisk *d = b->bi_bdev->bd_disk; + blk_register_queue(d); + ]) +]) + +AC_DEFUN([ZFS_AC_KERNEL_BIO_BDEV_DISK], [ + AC_MSG_CHECKING([whether bio->bi_bdev->bd_disk exists]) + ZFS_LINUX_TEST_RESULT([bio_bdev_disk], [ + AC_MSG_RESULT(yes) + AC_DEFINE(HAVE_BIO_BDEV_DISK, 1, [bio->bi_bdev->bd_disk exists]) + ],[ + AC_MSG_RESULT(no) + ]) +]) + AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO], [ ZFS_AC_KERNEL_SRC_REQ ZFS_AC_KERNEL_SRC_BIO_OPS @@ -379,6 +406,7 @@ AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO], [ ZFS_AC_KERNEL_SRC_BIO_SUBMIT_BIO ZFS_AC_KERNEL_SRC_BIO_CURRENT_BIO_LIST ZFS_AC_KERNEL_SRC_BLKG_TRYGET + ZFS_AC_KERNEL_SRC_BIO_BDEV_DISK ]) AC_DEFUN([ZFS_AC_KERNEL_BIO], [ @@ -400,4 +428,5 @@ AC_DEFUN([ZFS_AC_KERNEL_BIO], [ ZFS_AC_KERNEL_BIO_SUBMIT_BIO ZFS_AC_KERNEL_BIO_CURRENT_BIO_LIST ZFS_AC_KERNEL_BLKG_TRYGET + ZFS_AC_KERNEL_BIO_BDEV_DISK ]) diff --git a/config/kernel-generic_io_acct.m4 b/config/kernel-generic_io_acct.m4 index e4ab503d5e1c..0f4381db4c5e 100644 --- a/config/kernel-generic_io_acct.m4 +++ b/config/kernel-generic_io_acct.m4 @@ -2,6 +2,17 @@ dnl # dnl # Check for generic io accounting interface. dnl # AC_DEFUN([ZFS_AC_KERNEL_SRC_GENERIC_IO_ACCT], [ + ZFS_LINUX_TEST_SRC([disk_io_acct], [ + #include + ], [ + struct gendisk *disk = NULL; + struct bio *bio = NULL; + unsigned long start_time; + + start_time = disk_start_io_acct(disk, bio_sectors(bio), bio_op(bio)); + disk_end_io_acct(disk, bio_op(bio), start_time); + ]) + ZFS_LINUX_TEST_SRC([bio_io_acct], [ #include ], [ @@ -39,48 +50,62 @@ AC_DEFUN([ZFS_AC_KERNEL_SRC_GENERIC_IO_ACCT], [ AC_DEFUN([ZFS_AC_KERNEL_GENERIC_IO_ACCT], [ dnl # - dnl # 5.7 API, + dnl # 5.12 API, dnl # - dnl # Added bio_start_io_acct() and bio_end_io_acct() helpers. + dnl # bio_start_io_acct() and bio_end_io_acct() became GPL-exported + dnl # so use disk_start_io_acct() and disk_end_io_acct() instead dnl # - AC_MSG_CHECKING([whether generic bio_*_io_acct() are available]) - ZFS_LINUX_TEST_RESULT([bio_io_acct], [ + AC_MSG_CHECKING([whether generic disk_*_io_acct() are available]) + ZFS_LINUX_TEST_RESULT([disk_io_acct], [ AC_MSG_RESULT(yes) - AC_DEFINE(HAVE_BIO_IO_ACCT, 1, [bio_*_io_acct() available]) + AC_DEFINE(HAVE_DISK_IO_ACCT, 1, [disk_*_io_acct() available]) ], [ AC_MSG_RESULT(no) dnl # - dnl # 4.14 API, + dnl # 5.7 API, dnl # - dnl # generic_start_io_acct/generic_end_io_acct now require - dnl # request_queue to be provided. No functional changes, - dnl # but preparation for inflight accounting. + dnl # Added bio_start_io_acct() and bio_end_io_acct() helpers. dnl # - AC_MSG_CHECKING([whether generic_*_io_acct wants 4 args]) - ZFS_LINUX_TEST_RESULT_SYMBOL([generic_acct_4args], - [generic_start_io_acct], [block/bio.c], [ + AC_MSG_CHECKING([whether generic bio_*_io_acct() are available]) + ZFS_LINUX_TEST_RESULT([bio_io_acct], [ AC_MSG_RESULT(yes) - AC_DEFINE(HAVE_GENERIC_IO_ACCT_4ARG, 1, - [generic_*_io_acct() 4 arg available]) + AC_DEFINE(HAVE_BIO_IO_ACCT, 1, [bio_*_io_acct() available]) ], [ AC_MSG_RESULT(no) dnl # - dnl # 3.19 API addition + dnl # 4.14 API, dnl # - dnl # torvalds/linux@394ffa50 allows us to increment - dnl # iostat counters without generic_make_request(). + dnl # generic_start_io_acct/generic_end_io_acct now require + dnl # request_queue to be provided. No functional changes, + dnl # but preparation for inflight accounting. dnl # - AC_MSG_CHECKING( - [whether generic_*_io_acct wants 3 args]) - ZFS_LINUX_TEST_RESULT_SYMBOL([generic_acct_3args], + AC_MSG_CHECKING([whether generic_*_io_acct wants 4 args]) + ZFS_LINUX_TEST_RESULT_SYMBOL([generic_acct_4args], [generic_start_io_acct], [block/bio.c], [ AC_MSG_RESULT(yes) - AC_DEFINE(HAVE_GENERIC_IO_ACCT_3ARG, 1, - [generic_*_io_acct() 3 arg available]) + AC_DEFINE(HAVE_GENERIC_IO_ACCT_4ARG, 1, + [generic_*_io_acct() 4 arg available]) ], [ AC_MSG_RESULT(no) + + dnl # + dnl # 3.19 API addition + dnl # + dnl # torvalds/linux@394ffa50 allows us to increment + dnl # iostat counters without generic_make_request(). + dnl # + AC_MSG_CHECKING( + [whether generic_*_io_acct wants 3 args]) + ZFS_LINUX_TEST_RESULT_SYMBOL([generic_acct_3args], + [generic_start_io_acct], [block/bio.c], [ + AC_MSG_RESULT(yes) + AC_DEFINE(HAVE_GENERIC_IO_ACCT_3ARG, 1, + [generic_*_io_acct() 3 arg available]) + ], [ + AC_MSG_RESULT(no) + ]) ]) ]) ]) diff --git a/configure.ac b/configure.ac index 4520a290a9a5..b2d88554ed7d 100644 --- a/configure.ac +++ b/configure.ac @@ -230,6 +230,7 @@ AC_CONFIG_FILES([ tests/zfs-tests/cmd/readmmap/Makefile tests/zfs-tests/cmd/rename_dir/Makefile tests/zfs-tests/cmd/rm_lnkcnt_zero_file/Makefile + tests/zfs-tests/cmd/send_doall/Makefile tests/zfs-tests/cmd/stride_dd/Makefile tests/zfs-tests/cmd/threadsappend/Makefile tests/zfs-tests/cmd/user_ns_exec/Makefile diff --git a/include/os/linux/kernel/linux/blkdev_compat.h b/include/os/linux/kernel/linux/blkdev_compat.h index e41b248b0405..ee066537b900 100644 --- a/include/os/linux/kernel/linux/blkdev_compat.h +++ b/include/os/linux/kernel/linux/blkdev_compat.h @@ -520,7 +520,9 @@ blk_generic_start_io_acct(struct request_queue *q __attribute__((unused)), struct gendisk *disk __attribute__((unused)), int rw __attribute__((unused)), struct bio *bio) { -#if defined(HAVE_BIO_IO_ACCT) +#if defined(HAVE_DISK_IO_ACCT) + return (disk_start_io_acct(disk, bio_sectors(bio), bio_op(bio))); +#elif defined(HAVE_BIO_IO_ACCT) return (bio_start_io_acct(bio)); #elif defined(HAVE_GENERIC_IO_ACCT_3ARG) unsigned long start_time = jiffies; @@ -541,7 +543,9 @@ blk_generic_end_io_acct(struct request_queue *q __attribute__((unused)), struct gendisk *disk __attribute__((unused)), int rw __attribute__((unused)), struct bio *bio, unsigned long start_time) { -#if defined(HAVE_BIO_IO_ACCT) +#if defined(HAVE_DISK_IO_ACCT) + disk_end_io_acct(disk, bio_op(bio), start_time); +#elif defined(HAVE_BIO_IO_ACCT) bio_end_io_acct(bio, start_time); #elif defined(HAVE_GENERIC_IO_ACCT_3ARG) generic_end_io_acct(rw, &disk->part0, start_time); diff --git a/include/sys/dsl_synctask.h b/include/sys/dsl_synctask.h index 0bb602e8f7ff..5a5b306419f1 100644 --- a/include/sys/dsl_synctask.h +++ b/include/sys/dsl_synctask.h @@ -41,10 +41,11 @@ typedef void (dsl_sigfunc_t)(void *, dmu_tx_t *); typedef enum zfs_space_check { /* - * Normal space check: if there is less than 3.2% free space, - * the operation will fail. Operations which are logically - * creating things should use this (e.g. "zfs create", "zfs snapshot"). - * User writes (via the ZPL / ZVOL) also fail at this point. + * Normal space check: if there is less than 3.2% free space (bounded + * by spa_max_slop), the operation will fail. Operations which are + * logically creating things should use this (e.g. "zfs create", "zfs + * snapshot"). User writes (via the ZPL / ZVOL) also fail at this + * point. */ ZFS_SPACE_CHECK_NORMAL, diff --git a/lib/libzfs/libzfs_sendrecv.c b/lib/libzfs/libzfs_sendrecv.c index 62a94264494f..1e3a0bf5618a 100644 --- a/lib/libzfs/libzfs_sendrecv.c +++ b/lib/libzfs/libzfs_sendrecv.c @@ -321,6 +321,15 @@ send_iterate_snap(zfs_handle_t *zhp, void *arg) } if (!sd->recursive) { + + /* + * To allow a doall stream to work properly + * with a NULL fromsnap + */ + if (sd->doall && sd->fromsnap == NULL && !sd->seenfrom) { + sd->seenfrom = B_TRUE; + } + if (!sd->seenfrom && isfromsnap) { sd->seenfrom = B_TRUE; zfs_close(zhp); diff --git a/lib/libzfs/libzfs_status.c b/lib/libzfs/libzfs_status.c index fadae9388ac1..5e5cb5f5d440 100644 --- a/lib/libzfs/libzfs_status.c +++ b/lib/libzfs/libzfs_status.c @@ -482,6 +482,8 @@ check_status(nvlist_t *config, boolean_t isimport, return (ZPOOL_STATUS_COMPATIBILITY_ERR); for (i = 0; i < SPA_FEATURES; i++) { zfeature_info_t *fi = &spa_feature_table[i]; + if (!fi->fi_zfs_mod_supported) + continue; if (pool_features[i] && !nvlist_exists(feat, fi->fi_guid)) return (ZPOOL_STATUS_FEAT_DISABLED); diff --git a/man/man5/zfs-module-parameters.5 b/man/man5/zfs-module-parameters.5 index 8fec44dd37e5..d68e85fa078d 100644 --- a/man/man5/zfs-module-parameters.5 +++ b/man/man5/zfs-module-parameters.5 @@ -1238,7 +1238,7 @@ amount of memory. When the limit is exceeded, the ioctl fails with EINVAL and a description of the error is sent to the zfs-dbgmsg log. This parameter should not need to be touched under normal circumstances. On FreeBSD, the default is based on the system limit on user wired memory. On Linux, the default is -\fBKMALLOC_MAX_SIZE\fR . +\fB128MB\fR. .sp Default value: \fB0\fR (kernel decides) .RE diff --git a/module/os/freebsd/zfs/dmu_os.c b/module/os/freebsd/zfs/dmu_os.c index 8e412d9c1359..fb8f560316ea 100644 --- a/module/os/freebsd/zfs/dmu_os.c +++ b/module/os/freebsd/zfs/dmu_os.c @@ -319,7 +319,7 @@ dmu_read_pages(objset_t *os, uint64_t object, vm_page_t *ma, int count, break; } ASSERT(m->dirty == 0); - ASSERT(!pmap_page_is_mapped(m)); + ASSERT(!pmap_page_is_write_mapped(m)); ASSERT(db->db_size > PAGE_SIZE); bufoff = IDX_TO_OFF(m->pindex) % db->db_size; diff --git a/module/os/linux/zfs/vdev_disk.c b/module/os/linux/zfs/vdev_disk.c index b373f2c2e83c..ff71ef4cd065 100644 --- a/module/os/linux/zfs/vdev_disk.c +++ b/module/os/linux/zfs/vdev_disk.c @@ -494,7 +494,11 @@ vdev_blkg_tryget(struct blkcg_gq *blkg) static inline void vdev_bio_associate_blkg(struct bio *bio) { +#if defined(HAVE_BIO_BDEV_DISK) + struct request_queue *q = bio->bi_bdev->bd_disk->queue; +#else struct request_queue *q = bio->bi_disk->queue; +#endif ASSERT3P(q, !=, NULL); ASSERT3P(bio->bi_blkg, ==, NULL); diff --git a/module/os/linux/zfs/zfs_ioctl_os.c b/module/os/linux/zfs/zfs_ioctl_os.c index b88e0497d000..6f5cff1770e1 100644 --- a/module/os/linux/zfs/zfs_ioctl_os.c +++ b/module/os/linux/zfs/zfs_ioctl_os.c @@ -209,7 +209,7 @@ zfs_max_nvlist_src_size_os(void) if (zfs_max_nvlist_src_size != 0) return (zfs_max_nvlist_src_size); - return (KMALLOC_MAX_SIZE); + return (MIN(ptob(zfs_totalram_pages) / 4, 128 * 1024 * 1024)); } void diff --git a/module/os/linux/zfs/zvol_os.c b/module/os/linux/zfs/zvol_os.c index 0caf31307718..0d62b1490702 100644 --- a/module/os/linux/zfs/zvol_os.c +++ b/module/os/linux/zfs/zvol_os.c @@ -307,7 +307,11 @@ zvol_request(struct request_queue *q, struct bio *bio) #endif { #ifdef HAVE_SUBMIT_BIO_IN_BLOCK_DEVICE_OPERATIONS +#if defined(HAVE_BIO_BDEV_DISK) + struct request_queue *q = bio->bi_bdev->bd_disk->queue; +#else struct request_queue *q = bio->bi_disk->queue; +#endif #endif zvol_state_t *zv = q->queuedata; fstrans_cookie_t cookie = spl_fstrans_mark(); diff --git a/module/zcommon/zfeature_common.c b/module/zcommon/zfeature_common.c index e95a85e89ba2..fc0e09605eef 100644 --- a/module/zcommon/zfeature_common.c +++ b/module/zcommon/zfeature_common.c @@ -100,6 +100,8 @@ zfeature_is_supported(const char *guid) for (spa_feature_t i = 0; i < SPA_FEATURES; i++) { zfeature_info_t *feature = &spa_feature_table[i]; + if (!feature->fi_zfs_mod_supported) + continue; if (strcmp(guid, feature->fi_guid) == 0) return (B_TRUE); } diff --git a/module/zfs/spa_misc.c b/module/zfs/spa_misc.c index b4c73f58d3bc..0dacf9027b27 100644 --- a/module/zfs/spa_misc.c +++ b/module/zfs/spa_misc.c @@ -347,13 +347,14 @@ int spa_asize_inflation = 24; /* * Normally, we don't allow the last 3.2% (1/(2^spa_slop_shift)) of space in - * the pool to be consumed. This ensures that we don't run the pool - * completely out of space, due to unaccounted changes (e.g. to the MOS). - * It also limits the worst-case time to allocate space. If we have less than - * this amount of free space, most ZPL operations (e.g. write, create) will - * return ENOSPC. The ZIL metaslabs (spa_embedded_log_class) are also part of - * this 3.2% of space which can't be consumed by normal writes; the slop space - * "proper" (spa_get_slop_space()) is decreased by the embedded log space. + * the pool to be consumed (bounded by spa_max_slop). This ensures that we + * don't run the pool completely out of space, due to unaccounted changes (e.g. + * to the MOS). It also limits the worst-case time to allocate space. If we + * have less than this amount of free space, most ZPL operations (e.g. write, + * create) will return ENOSPC. The ZIL metaslabs (spa_embedded_log_class) are + * also part of this 3.2% of space which can't be consumed by normal writes; + * the slop space "proper" (spa_get_slop_space()) is decreased by the embedded + * log space. * * Certain operations (e.g. file removal, most administrative actions) can * use half the slop space. They will only return ENOSPC if less than half @@ -376,10 +377,15 @@ int spa_asize_inflation = 24; * 3.2%, in an effort to have it be at least spa_min_slop (128MB), * but we never allow it to be more than half the pool size. * + * Further, on very large pools, the slop space will be smaller than + * 3.2%, to avoid reserving much more space than we actually need; bounded + * by spa_max_slop (128GB). + * * See also the comments in zfs_space_check_t. */ int spa_slop_shift = 5; -uint64_t spa_min_slop = 128 * 1024 * 1024; +uint64_t spa_min_slop = 128ULL * 1024 * 1024; +uint64_t spa_max_slop = 128ULL * 1024 * 1024 * 1024; int spa_allocators = 4; @@ -1278,9 +1284,9 @@ spa_vdev_config_exit(spa_t *spa, vdev_t *vd, uint64_t txg, int error, char *tag) */ vdev_autotrim_stop_wait(vd); - spa_config_enter(spa, SCL_ALL, spa, RW_WRITER); + spa_config_enter(spa, SCL_STATE_ALL, spa, RW_WRITER); vdev_free(vd); - spa_config_exit(spa, SCL_ALL, spa); + spa_config_exit(spa, SCL_STATE_ALL, spa); } /* @@ -1781,7 +1787,8 @@ spa_get_worst_case_asize(spa_t *spa, uint64_t lsize) /* * Return the amount of slop space in bytes. It is typically 1/32 of the pool * (3.2%), minus the embedded log space. On very small pools, it may be - * slightly larger than this. The embedded log space is not included in + * slightly larger than this. On very large pools, it will be capped to + * the value of spa_max_slop. The embedded log space is not included in * spa_dspace. By subtracting it, the usable space (per "zfs list") is a * constant 97% of the total space, regardless of metaslab size (assuming the * default spa_slop_shift=5 and a non-tiny pool). @@ -1792,7 +1799,7 @@ uint64_t spa_get_slop_space(spa_t *spa) { uint64_t space = spa_get_dspace(spa); - uint64_t slop = space >> spa_slop_shift; + uint64_t slop = MIN(space >> spa_slop_shift, spa_max_slop); /* * Subtract the embedded log space, but no more than half the (3.2%) diff --git a/module/zfs/vdev_initialize.c b/module/zfs/vdev_initialize.c index 083ad2861b5b..e9156c32f384 100644 --- a/module/zfs/vdev_initialize.c +++ b/module/zfs/vdev_initialize.c @@ -553,8 +553,14 @@ vdev_initialize_thread(void *arg) vd->vdev_initialize_tree = NULL; mutex_enter(&vd->vdev_initialize_lock); - if (!vd->vdev_initialize_exit_wanted && vdev_writeable(vd)) { - vdev_initialize_change_state(vd, VDEV_INITIALIZE_COMPLETE); + if (!vd->vdev_initialize_exit_wanted) { + if (vdev_writeable(vd)) { + vdev_initialize_change_state(vd, + VDEV_INITIALIZE_COMPLETE); + } else if (vd->vdev_faulted) { + vdev_initialize_change_state(vd, + VDEV_INITIALIZE_CANCELED); + } } ASSERT(vd->vdev_initialize_thread != NULL || vd->vdev_initialize_inflight == 0); diff --git a/module/zfs/vdev_rebuild.c b/module/zfs/vdev_rebuild.c index 037abc01f7a7..a77ff99faa92 100644 --- a/module/zfs/vdev_rebuild.c +++ b/module/zfs/vdev_rebuild.c @@ -807,8 +807,8 @@ vdev_rebuild_thread(void *arg) ASSERT0(range_tree_space(vr->vr_scan_tree)); /* Disable any new allocations to this metaslab */ - metaslab_disable(msp); spa_config_exit(spa, SCL_CONFIG, FTAG); + metaslab_disable(msp); mutex_enter(&msp->ms_sync_lock); mutex_enter(&msp->ms_lock); diff --git a/module/zfs/vdev_trim.c b/module/zfs/vdev_trim.c index 895957bda195..deea7fedd770 100644 --- a/module/zfs/vdev_trim.c +++ b/module/zfs/vdev_trim.c @@ -22,6 +22,7 @@ /* * Copyright (c) 2016 by Delphix. All rights reserved. * Copyright (c) 2019 by Lawrence Livermore National Security, LLC. + * Copyright (c) 2021 Hewlett Packard Enterprise Development LP */ #include @@ -930,10 +931,16 @@ vdev_trim_thread(void *arg) range_tree_destroy(ta.trim_tree); mutex_enter(&vd->vdev_trim_lock); - if (!vd->vdev_trim_exit_wanted && vdev_writeable(vd)) { - vdev_trim_change_state(vd, VDEV_TRIM_COMPLETE, - vd->vdev_trim_rate, vd->vdev_trim_partial, - vd->vdev_trim_secure); + if (!vd->vdev_trim_exit_wanted) { + if (vdev_writeable(vd)) { + vdev_trim_change_state(vd, VDEV_TRIM_COMPLETE, + vd->vdev_trim_rate, vd->vdev_trim_partial, + vd->vdev_trim_secure); + } else if (vd->vdev_faulted) { + vdev_trim_change_state(vd, VDEV_TRIM_CANCELED, + vd->vdev_trim_rate, vd->vdev_trim_partial, + vd->vdev_trim_secure); + } } ASSERT(vd->vdev_trim_thread != NULL || vd->vdev_trim_inflight[0] == 0); diff --git a/module/zfs/zfs_ioctl.c b/module/zfs/zfs_ioctl.c index 4e64563388f1..db02f2ea0987 100644 --- a/module/zfs/zfs_ioctl.c +++ b/module/zfs/zfs_ioctl.c @@ -3986,7 +3986,7 @@ zfs_ioc_pool_initialize(const char *poolname, nvlist_t *innvl, nvlist_t *outnvl) fnvlist_free(vdev_errlist); spa_close(spa, FTAG); - return (total_errors > 0 ? EINVAL : 0); + return (total_errors > 0 ? SET_ERROR(EINVAL) : 0); } /* @@ -4071,7 +4071,7 @@ zfs_ioc_pool_trim(const char *poolname, nvlist_t *innvl, nvlist_t *outnvl) fnvlist_free(vdev_errlist); spa_close(spa, FTAG); - return (total_errors > 0 ? EINVAL : 0); + return (total_errors > 0 ? SET_ERROR(EINVAL) : 0); } /* diff --git a/tests/runfiles/common.run b/tests/runfiles/common.run index e656785d95d8..fcb0fa6cd24f 100644 --- a/tests/runfiles/common.run +++ b/tests/runfiles/common.run @@ -415,6 +415,7 @@ tags = ['functional', 'cli_root', 'zpool_labelclear'] [tests/functional/cli_root/zpool_initialize] tests = ['zpool_initialize_attach_detach_add_remove', + 'zpool_initialize_fault_export_import_online', 'zpool_initialize_import_export', 'zpool_initialize_offline_export_import_online', 'zpool_initialize_online_offline', @@ -480,6 +481,7 @@ tags = ['functional', 'cli_root', 'zpool_sync'] [tests/functional/cli_root/zpool_trim] tests = ['zpool_trim_attach_detach_add_remove', + 'zpool_trim_fault_export_import_online', 'zpool_trim_import_export', 'zpool_trim_multiple', 'zpool_trim_neg', 'zpool_trim_offline_export_import_online', 'zpool_trim_online_offline', 'zpool_trim_partial', 'zpool_trim_rate', 'zpool_trim_rate_neg', @@ -807,7 +809,7 @@ tests = ['recv_dedup', 'recv_dedup_encrypted_zvol', 'rsend_001_pos', 'send_freeobjects', 'send_realloc_files', 'send_realloc_encrypted_files', 'send_spill_block', 'send_holds', 'send_hole_birth', 'send_mixed_raw', 'send-wR_encrypted_zvol', - 'send_partial_dataset', 'send_invalid'] + 'send_partial_dataset', 'send_invalid', 'send_doall'] tags = ['functional', 'rsend'] [tests/functional/scrub_mirror] diff --git a/tests/zfs-tests/cmd/Makefile.am b/tests/zfs-tests/cmd/Makefile.am index 7fe9a2c571f8..2b965ca70009 100644 --- a/tests/zfs-tests/cmd/Makefile.am +++ b/tests/zfs-tests/cmd/Makefile.am @@ -25,6 +25,7 @@ SUBDIRS = \ readmmap \ rename_dir \ rm_lnkcnt_zero_file \ + send_doall \ stride_dd \ threadsappend diff --git a/tests/zfs-tests/cmd/send_doall/.gitignore b/tests/zfs-tests/cmd/send_doall/.gitignore new file mode 100644 index 000000000000..6ba2e603f744 --- /dev/null +++ b/tests/zfs-tests/cmd/send_doall/.gitignore @@ -0,0 +1 @@ +/send_doall diff --git a/tests/zfs-tests/cmd/send_doall/Makefile.am b/tests/zfs-tests/cmd/send_doall/Makefile.am new file mode 100644 index 000000000000..33a6b83122b8 --- /dev/null +++ b/tests/zfs-tests/cmd/send_doall/Makefile.am @@ -0,0 +1,11 @@ +include $(top_srcdir)/config/Rules.am + +pkgexecdir = $(datadir)/@PACKAGE@/zfs-tests/bin + +pkgexec_PROGRAMS = send_doall + +send_doall_SOURCES = send_doall.c +send_doall_LDADD = \ + $(abs_top_builddir)/lib/libzfs_core/libzfs_core.la \ + $(abs_top_builddir)/lib/libzfs/libzfs.la \ + $(abs_top_builddir)/lib/libnvpair/libnvpair.la diff --git a/tests/zfs-tests/cmd/send_doall/send_doall.c b/tests/zfs-tests/cmd/send_doall/send_doall.c new file mode 100644 index 000000000000..6f47df047478 --- /dev/null +++ b/tests/zfs-tests/cmd/send_doall/send_doall.c @@ -0,0 +1,87 @@ +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ + +/* + * Portions Copyright 2020 iXsystems, Inc. + */ + +/* + * Test a corner case : a "doall" send without children datasets. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include + +static void +usage(const char *name) +{ + fprintf(stderr, "usage: %s snap\n", name); + exit(EX_USAGE); +} + +int +main(int argc, char const * const argv[]) +{ + sendflags_t flags = { 0 }; + libzfs_handle_t *zhdl; + zfs_handle_t *zhp; + const char *tofull, *fsname, *tosnap, *p; + int error; + + if (argc != 2) + usage(argv[0]); + + tofull = argv[1]; + + p = strchr(tofull, '@'); + if (p == NULL) + usage(argv[0]); + tosnap = p + 1; + + fsname = strndup(tofull, p - tofull); + + zhdl = libzfs_init(); + if (zhdl == NULL) + errx(EX_OSERR, "libzfs_init(): %s", libzfs_error_init(errno)); + + zhp = zfs_open(zhdl, fsname, ZFS_TYPE_FILESYSTEM); + if (zhp == NULL) + err(EX_OSERR, "zfs_open(\"%s\")", fsname); + + flags.doall = B_TRUE; + + error = zfs_send(zhp, NULL, tosnap, &flags, + STDOUT_FILENO, NULL, NULL, NULL); + + zfs_close(zhp); + + libzfs_fini(zhdl); + free((void *)fsname); + + return (error); +} diff --git a/tests/zfs-tests/include/commands.cfg b/tests/zfs-tests/include/commands.cfg index 299653547759..a43ddd016fde 100644 --- a/tests/zfs-tests/include/commands.cfg +++ b/tests/zfs-tests/include/commands.cfg @@ -217,6 +217,7 @@ export ZFSTEST_FILES='badsend readmmap rename_dir rm_lnkcnt_zero_file + send_doall threadsappend user_ns_exec xattrtest diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile.am b/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile.am index 2ebc376d9cb9..3968902ec36d 100644 --- a/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile.am +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile.am @@ -2,6 +2,7 @@ pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/cli_root/zpool_init dist_pkgdata_SCRIPTS = \ cleanup.ksh \ zpool_initialize_attach_detach_add_remove.ksh \ + zpool_initialize_fault_export_import_online.ksh \ zpool_initialize_import_export.ksh \ zpool_initialize_offline_export_import_online.ksh \ zpool_initialize_online_offline.ksh \ diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_fault_export_import_online.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_fault_export_import_online.ksh new file mode 100755 index 000000000000..11b8a483e662 --- /dev/null +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_fault_export_import_online.ksh @@ -0,0 +1,59 @@ +#!/bin/ksh -p +# +# CDDL HEADER START +# +# This file and its contents are supplied under the terms of the +# Common Development and Distribution License ("CDDL"), version 1.0. +# You may only use this file in accordance with the terms of version +# 1.0 of the CDDL. +# +# A full copy of the text of the CDDL should have accompanied this +# source. A copy of the CDDL is also available via the Internet at +# http://www.illumos.org/license/CDDL. +# +# CDDL HEADER END +# + +# +# Copyright (c) 2021 Lawrence Livermore National Security, LLC. +# + +. $STF_SUITE/include/libtest.shlib +. $STF_SUITE/tests/functional/cli_root/zpool_initialize/zpool_initialize.kshlib + +# +# DESCRIPTION: +# Miscellaneous complex sequences of operations function as expected. +# +# STRATEGY: +# 1. Create a pool with a two-way mirror. +# 2. Start initializing, fault, export, import, online and verify along +# the way that the initializing was cancelled and not restarted. +# + +DISK1="$(echo $DISKS | cut -d' ' -f1)" +DISK2="$(echo $DISKS | cut -d' ' -f2)" + +log_must zpool create -f $TESTPOOL mirror $DISK1 $DISK2 + +log_must zpool initialize $TESTPOOL $DISK1 +progress="$(initialize_progress $TESTPOOL $DISK1)" +[[ -z "$progress" ]] && log_fail "Initializing did not start" + +log_must zpool offline -f $TESTPOOL $DISK1 +log_must check_vdev_state $TESTPOOL $DISK1 "FAULTED" +log_must eval "zpool status -i $TESTPOOL | grep $DISK1 | grep uninitialized" + +log_must zpool export $TESTPOOL +log_must zpool import $TESTPOOL + +log_must check_vdev_state $TESTPOOL $DISK1 "FAULTED" +log_must eval "zpool status -i $TESTPOOL | grep $DISK1 | grep uninitialized" + +log_must zpool online $TESTPOOL $DISK1 +log_must zpool clear $TESTPOOL $DISK1 +log_must check_vdev_state $TESTPOOL $DISK1 "ONLINE" +log_must eval "zpool status -i $TESTPOOL | grep $DISK1 | grep uninitialized" + +log_pass "Initializing behaves as expected at each step of:" \ + "initialize + fault + export + import + online" diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile.am b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile.am index d2d3b4ae88bb..0411ab4e0070 100644 --- a/tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile.am +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile.am @@ -3,6 +3,7 @@ dist_pkgdata_SCRIPTS = \ setup.ksh \ cleanup.ksh \ zpool_trim_attach_detach_add_remove.ksh \ + zpool_trim_fault_export_import_online.ksh \ zpool_trim_import_export.ksh \ zpool_trim_multiple.ksh \ zpool_trim_neg.ksh \ diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_fault_export_import_online.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_fault_export_import_online.ksh new file mode 100755 index 000000000000..6bb9fc346daf --- /dev/null +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_fault_export_import_online.ksh @@ -0,0 +1,62 @@ +#!/bin/ksh -p +# +# CDDL HEADER START +# +# This file and its contents are supplied under the terms of the +# Common Development and Distribution License ("CDDL"), version 1.0. +# You may only use this file in accordance with the terms of version +# 1.0 of the CDDL. +# +# A full copy of the text of the CDDL should have accompanied this +# source. A copy of the CDDL is also available via the Internet at +# http://www.illumos.org/license/CDDL. +# +# CDDL HEADER END +# + +# +# Copyright (c) 2019 by Tim Chase. All rights reserved. +# Copyright (c) 2021 Lawrence Livermore National Security, LLC. +# + +. $STF_SUITE/include/libtest.shlib +. $STF_SUITE/tests/functional/cli_root/zpool_trim/zpool_trim.kshlib + +# +# DESCRIPTION: +# Miscellaneous complex sequences of operations function as expected. +# +# STRATEGY: +# 1. Create a pool with a two-way mirror. +# 2. Start trimming, fault, export, import, online and verify along +# the way that the trim was cancelled and not restarted. +# + +DISK1="$(echo $DISKS | cut -d' ' -f1)" +DISK2="$(echo $DISKS | cut -d' ' -f2)" + +log_must zpool create -f $TESTPOOL mirror $DISK1 $DISK2 + +log_must zpool trim -r 128M $TESTPOOL $DISK1 +progress="$(trim_progress $TESTPOOL $DISK1)" +[[ -z "$progress" ]] && log_fail "Trimming did not start" + +log_must zpool offline -f $TESTPOOL $DISK1 +log_must check_vdev_state $TESTPOOL $DISK1 "FAULTED" +log_must eval "zpool status -t $TESTPOOL | grep $DISK1 | grep untrimmed" + +log_must zpool export $TESTPOOL +log_must zpool import $TESTPOOL + +# Note: the expected state here is unsupported since the faulted device +# cannot be checked to determine if it supports TRIM. +log_must check_vdev_state $TESTPOOL $DISK1 "FAULTED" +log_must eval "zpool status -t $TESTPOOL | grep $DISK1 | grep unsupported" + +log_must zpool online $TESTPOOL $DISK1 +log_must zpool clear $TESTPOOL $DISK1 +log_must check_vdev_state $TESTPOOL $DISK1 "ONLINE" +log_must eval "zpool status -t $TESTPOOL | grep $DISK1 | grep untrimmed" + +log_pass "Trimming behaves as expected at each step of:" \ + "trim + fault + export + import + online" diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_start_and_cancel_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_start_and_cancel_pos.ksh index eaa4d90444b6..fbb0c291046c 100755 --- a/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_start_and_cancel_pos.ksh +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_trim/zpool_trim_start_and_cancel_pos.ksh @@ -20,29 +20,29 @@ # . $STF_SUITE/include/libtest.shlib -. $STF_SUITE/tests/functional/cli_root/zpool_initialize/zpool_initialize.kshlib +. $STF_SUITE/tests/functional/cli_root/zpool_trim/zpool_trim.kshlib # # DESCRIPTION: -# Starting and stopping an initialize works. +# Starting and stopping a trim works. # # STRATEGY: # 1. Create a one-disk pool. -# 2. Start initializing and verify that initializing is active. -# 3. Cancel initializing and verify that initializing is not active. +# 2. Start trimming and verify that trimming is active. +# 3. Cancel trimming and verify that trimming is not active. # DISK1=${DISKS%% *} log_must zpool create -f $TESTPOOL $DISK1 -log_must zpool initialize $TESTPOOL +log_must zpool trim $TESTPOOL -[[ -z "$(initialize_progress $TESTPOOL $DISK1)" ]] && \ - log_fail "Initialize did not start" *** 105 LINES SKIPPED ***