From owner-freebsd-stable@freebsd.org Sun Jul 19 14:17:04 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8609536046E for ; Sun, 19 Jul 2020 14:17:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8n3z6Ccdz4cqP for ; Sun, 19 Jul 2020 14:17:03 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:cd26:ee8a:a97e:89a8] ([IPv6:2607:f3e0:0:4:cd26:ee8a:a97e:89a8]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06JEH24w058796 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Sun, 19 Jul 2020 10:17:02 -0400 (EDT) (envelope-from mike@sentex.net) To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: zfs meta data slowness Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: Date: Sun, 19 Jul 2020 10:17:02 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4B8n3z6Ccdz4cqP X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.32 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.58)[-0.577]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.65)[-0.646]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.09)[-0.093]; DMARC_NA(0.00)[sentex.net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 14:17:04 -0000 Are there any tweaks that can be done to speed up or improve zfs metadata performance ? I have a backup server with a lot of snapshots (40,000)  and just doing a listing can take a great deal of time.  Best case scenario is about 24 seconds, worst case, I have seen it up to 15 minutes.  (FreeBSD 12.1-STABLE r363078) ARC Efficiency:                                 79.33b         Cache Hit Ratio:                92.81%  73.62b         Cache Miss Ratio:               7.19%   5.71b         Actual Hit Ratio:               92.78%  73.60b         Data Demand Efficiency:         96.47%  461.91m         Data Prefetch Efficiency:       1.00%   262.73m         CACHE HITS BY CACHE LIST:           Anonymously Used:             0.01%   3.86m           Most Recently Used:           3.91%   2.88b           Most Frequently Used:         96.06%  70.72b           Most Recently Used Ghost:     0.01%   5.31m           Most Frequently Used Ghost:   0.01%   10.47m         CACHE HITS BY DATA TYPE:           Demand Data:                  0.61%   445.60m           Prefetch Data:                0.00%   2.63m           Demand Metadata:              99.36%  73.15b           Prefetch Metadata:            0.03%   21.00m         CACHE MISSES BY DATA TYPE:           Demand Data:                  0.29%   16.31m           Prefetch Data:                4.56%   260.10m           Demand Metadata:              95.02%  5.42b           Prefetch Metadata:            0.14%   7.75m Other than increase the metadata max, I havent really changed any tuneables ZFS Tunables (sysctl):         kern.maxusers                           4416         vm.kmem_size                            66691842048         vm.kmem_size_scale                      1         vm.kmem_size_min                        0         vm.kmem_size_max                        1319413950874         vfs.zfs.trim.max_interval               1         vfs.zfs.trim.timeout                    30         vfs.zfs.trim.txg_delay                  32         vfs.zfs.trim.enabled                    1         vfs.zfs.vol.immediate_write_sz          32768         vfs.zfs.vol.unmap_sync_enabled          0         vfs.zfs.vol.unmap_enabled               1         vfs.zfs.vol.recursive                   0         vfs.zfs.vol.mode                        1         vfs.zfs.version.zpl                     5         vfs.zfs.version.spa                     5000         vfs.zfs.version.acl                     1         vfs.zfs.version.ioctl                   7         vfs.zfs.debug                           0         vfs.zfs.super_owner                     0         vfs.zfs.immediate_write_sz              32768         vfs.zfs.sync_pass_rewrite               2         vfs.zfs.sync_pass_dont_compress         5         vfs.zfs.sync_pass_deferred_free         2         vfs.zfs.zio.dva_throttle_enabled        1         vfs.zfs.zio.exclude_metadata            0         vfs.zfs.zio.use_uma                     1         vfs.zfs.zio.taskq_batch_pct             75         vfs.zfs.zil_maxblocksize                131072         vfs.zfs.zil_slog_bulk                   786432         vfs.zfs.zil_nocacheflush                0         vfs.zfs.zil_replay_disable              0         vfs.zfs.cache_flush_disable             0         vfs.zfs.standard_sm_blksz               131072         vfs.zfs.dtl_sm_blksz                    4096         vfs.zfs.min_auto_ashift                 9         vfs.zfs.max_auto_ashift                 13         vfs.zfs.vdev.trim_max_pending           10000         vfs.zfs.vdev.bio_delete_disable         0         vfs.zfs.vdev.bio_flush_disable          0         vfs.zfs.vdev.def_queue_depth            32         vfs.zfs.vdev.queue_depth_pct            1000         vfs.zfs.vdev.write_gap_limit            4096         vfs.zfs.vdev.read_gap_limit             32768         vfs.zfs.vdev.aggregation_limit_non_rotating131072         vfs.zfs.vdev.aggregation_limit          1048576         vfs.zfs.vdev.initializing_max_active    1         vfs.zfs.vdev.initializing_min_active    1         vfs.zfs.vdev.removal_max_active         2         vfs.zfs.vdev.removal_min_active         1         vfs.zfs.vdev.trim_max_active            64         vfs.zfs.vdev.trim_min_active            1         vfs.zfs.vdev.scrub_max_active           2         vfs.zfs.vdev.scrub_min_active           1         vfs.zfs.vdev.async_write_max_active     10         vfs.zfs.vdev.async_write_min_active     1         vfs.zfs.vdev.async_read_max_active      3         vfs.zfs.vdev.async_read_min_active      1         vfs.zfs.vdev.sync_write_max_active      10         vfs.zfs.vdev.sync_write_min_active      10         vfs.zfs.vdev.sync_read_max_active       10         vfs.zfs.vdev.sync_read_min_active       10         vfs.zfs.vdev.max_active                 1000         vfs.zfs.vdev.async_write_active_max_dirty_percent60         vfs.zfs.vdev.async_write_active_min_dirty_percent30         vfs.zfs.vdev.mirror.non_rotating_seek_inc1         vfs.zfs.vdev.mirror.non_rotating_inc    0         vfs.zfs.vdev.mirror.rotating_seek_offset1048576         vfs.zfs.vdev.mirror.rotating_seek_inc   5         vfs.zfs.vdev.mirror.rotating_inc        0         vfs.zfs.vdev.trim_on_init               1         vfs.zfs.vdev.cache.bshift               16         vfs.zfs.vdev.cache.size                 0         vfs.zfs.vdev.cache.max                  16384         vfs.zfs.vdev.validate_skip              0         vfs.zfs.vdev.max_ms_shift               34         vfs.zfs.vdev.default_ms_shift           29         vfs.zfs.vdev.max_ms_count_limit         131072         vfs.zfs.vdev.min_ms_count               16         vfs.zfs.vdev.default_ms_count           200         vfs.zfs.txg.timeout                     5         vfs.zfs.space_map_ibs                   14         vfs.zfs.special_class_metadata_reserve_pct25         vfs.zfs.user_indirect_is_special        1         vfs.zfs.ddt_data_is_special             1         vfs.zfs.spa_allocators                  4         vfs.zfs.spa_min_slop                    134217728         vfs.zfs.spa_slop_shift                  5         vfs.zfs.spa_asize_inflation             24         vfs.zfs.deadman_enabled                 1         vfs.zfs.deadman_checktime_ms            5000         vfs.zfs.deadman_synctime_ms             1000000         vfs.zfs.debugflags                      0         vfs.zfs.recover                         0         vfs.zfs.spa_load_verify_data            1         vfs.zfs.spa_load_verify_metadata        1         vfs.zfs.spa_load_verify_maxinflight     10000         vfs.zfs.max_missing_tvds_scan           0         vfs.zfs.max_missing_tvds_cachefile      2         vfs.zfs.max_missing_tvds                0         vfs.zfs.spa_load_print_vdev_tree        0         vfs.zfs.ccw_retry_interval              300         vfs.zfs.check_hostid                    1         vfs.zfs.multihost_fail_intervals        10         vfs.zfs.multihost_import_intervals      20         vfs.zfs.multihost_interval              1000         vfs.zfs.mg_fragmentation_threshold      85         vfs.zfs.mg_noalloc_threshold            0         vfs.zfs.condense_pct                    200         vfs.zfs.metaslab_sm_blksz               4096         vfs.zfs.metaslab.bias_enabled           1         vfs.zfs.metaslab.lba_weighting_enabled  1         vfs.zfs.metaslab.fragmentation_factor_enabled1         vfs.zfs.metaslab.preload_enabled        1         vfs.zfs.metaslab.preload_limit          3         vfs.zfs.metaslab.unload_delay           8         vfs.zfs.metaslab.load_pct               50         vfs.zfs.metaslab.min_alloc_size         33554432         vfs.zfs.metaslab.df_free_pct            4         vfs.zfs.metaslab.df_alloc_threshold     131072         vfs.zfs.metaslab.debug_unload           0         vfs.zfs.metaslab.debug_load             0         vfs.zfs.metaslab.fragmentation_threshold70         vfs.zfs.metaslab.force_ganging          16777217         vfs.zfs.free_bpobj_enabled              1         vfs.zfs.free_max_blocks                 -1         vfs.zfs.zfs_scan_checkpoint_interval    7200         vfs.zfs.zfs_scan_legacy                 0         vfs.zfs.no_scrub_prefetch               0         vfs.zfs.no_scrub_io                     0         vfs.zfs.resilver_min_time_ms            3000         vfs.zfs.free_min_time_ms                1000         vfs.zfs.scan_min_time_ms                1000         vfs.zfs.scan_idle                       50         vfs.zfs.scrub_delay                     4         vfs.zfs.resilver_delay                  2         vfs.zfs.zfetch.array_rd_sz              1048576         vfs.zfs.zfetch.max_idistance            67108864         vfs.zfs.zfetch.max_distance             8388608         vfs.zfs.zfetch.min_sec_reap             2         vfs.zfs.zfetch.max_streams              8         vfs.zfs.prefetch_disable                0         vfs.zfs.delay_scale                     500000         vfs.zfs.delay_min_dirty_percent         60         vfs.zfs.dirty_data_sync_pct             20         vfs.zfs.dirty_data_max_percent          10         vfs.zfs.dirty_data_max_max              4294967296         vfs.zfs.dirty_data_max                  4294967296         vfs.zfs.max_recordsize                  1048576         vfs.zfs.default_ibs                     17         vfs.zfs.default_bs                      9         vfs.zfs.send_holes_without_birth_time   1         vfs.zfs.mdcomp_disable                  0         vfs.zfs.per_txg_dirty_frees_percent     5         vfs.zfs.nopwrite_enabled                1         vfs.zfs.dedup.prefetch                  1         vfs.zfs.dbuf_cache_lowater_pct          10         vfs.zfs.dbuf_cache_hiwater_pct          10         vfs.zfs.dbuf_metadata_cache_overflow    0         vfs.zfs.dbuf_metadata_cache_shift       6         vfs.zfs.dbuf_cache_shift                5         vfs.zfs.dbuf_metadata_cache_max_bytes   1025282816         vfs.zfs.dbuf_cache_max_bytes            2050565632         vfs.zfs.arc_min_prescient_prefetch_ms   6         vfs.zfs.arc_min_prefetch_ms             1         vfs.zfs.l2c_only_size                   0         vfs.zfs.mfu_ghost_data_esize            7778263552         vfs.zfs.mfu_ghost_metadata_esize        16851792896         vfs.zfs.mfu_ghost_size                  24630056448         vfs.zfs.mfu_data_esize                  3059418112         vfs.zfs.mfu_metadata_esize              28641792         vfs.zfs.mfu_size                        6399023104         vfs.zfs.mru_ghost_data_esize            2199812096         vfs.zfs.mru_ghost_metadata_esize        6289682432         vfs.zfs.mru_ghost_size                  8489494528         vfs.zfs.mru_data_esize                  22781456384         vfs.zfs.mru_metadata_esize              309155840         vfs.zfs.mru_size                        23847875584         vfs.zfs.anon_data_esize                 0         vfs.zfs.anon_metadata_esize             0         vfs.zfs.anon_size                       8556544         vfs.zfs.l2arc_norw                      1         vfs.zfs.l2arc_feed_again                1         vfs.zfs.l2arc_noprefetch                1         vfs.zfs.l2arc_feed_min_ms               200         vfs.zfs.l2arc_feed_secs                 1         vfs.zfs.l2arc_headroom                  2         vfs.zfs.l2arc_write_boost               8388608         vfs.zfs.l2arc_write_max                 8388608         vfs.zfs.arc_meta_strategy               1         vfs.zfs.arc_meta_limit                  15833624576         vfs.zfs.arc_free_target                 346902         vfs.zfs.arc_kmem_cache_reap_retry_ms    1000         vfs.zfs.compressed_arc_enabled          1         vfs.zfs.arc_grow_retry                  60         vfs.zfs.arc_shrink_shift                7         vfs.zfs.arc_average_blocksize           8192         vfs.zfs.arc_no_grow_shift               5         vfs.zfs.arc_min                         8202262528         vfs.zfs.arc_max                         39334498304         vfs.zfs.abd_chunk_size                  4096         vfs.zfs.abd_scatter_enabled             1