Date: Thu, 12 Mar 2026 00:26:25 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 293750] mv process hangs at 100% CPU (State R) in cache_fplookup / zap_lookup on ZFS Message-ID: <bug-293750-227@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293750 Bug ID: 293750 Summary: mv process hangs at 100% CPU (State R) in cache_fplookup / zap_lookup on ZFS Product: Base System Version: 15.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: chris@ctgameinfo.com Created attachment 268738 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268738&action=edit procstat -kka A mv command involving a ZFS dataset has entered an uninterruptible state, consuming 100% of a single CPU core. The process state is reported as R+ in ps and shows a specific CPU (e.g., CPU8) in top, indicating it is spinning in kernel space rather than sleeping on I/O. The process does not respond to SIGKILL (-9). After running these commands mv is unkillable and stuck at 100% # cd /home/sites/x/ # mv www/www . Additional info ------------ # uname -a FreeBSD Akane 15.0-RELEASE-p4 FreeBSD 15.0-RELEASE-p4 GENERIC amd64 # zfs list NAME USED AVAIL REFER MOUNTPOINT <snip> zroot/home/sites/x 836M 764G 554M /home/sites/x zroot/home/sites/x/logs 6.35M 764G 6.35M /home/sites/x/logs zroot/home/sites/x/www 276M 764G 276M /home/sites/x/www <snip> # ps -lp 84123 UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 84123 76510 7 0 0 14072 2424 - R+ 1 23:36.51 mv www/www . # truss -p 84123 <hangs with no output> # ktrace -p 84123 # # procstat -k 84123 PID TID COMM TDNAME KSTACK 84123 100542 mv - dsl_dir_tempreserve_space dmu_tx_assign zfs_write zfs_freebsd_write VOP_WRITE_APV ktr_writerequest ktr_submitrequest namei kern_renameat amd64_syscall fast_syscall_common #top last pid: 84552; load averages: 1.76, 1.77, 1.47 up 4+15:07:18 17:13:56 54 processes: 2 running, 52 sleeping CPU: 3.3% user, 0.0% nice, 0.5% system, 0.0% interrupt, 96.2% idle Mem: 877M Active, 5358M Inact, 24G Wired, 104K Buf, 909M Free ARC: 18G Total, 9819M MFU, 7578M MRU, 74M Anon, 206M Header, 1035M Other 16G Compressed, 40G Uncompressed, 2.51:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 84123 root 1 0 0 14M 2424K CPU9 9 25:11 99.02% mv <snip> # zpool status -v pool: zroot state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0p4 ONLINE 0 0 0 nda1p4 ONLINE 0 0 0 errors: No known data errors # zfs get all zroot/home/sites/x/www | grep -v default NAME PROPERTY VALUE SOURCE zroot/home/sites/x/www type filesystem - zroot/home/sites/x/www creation Fri Oct 3 15:19 2025 - zroot/home/sites/x/www used 276M - zroot/home/sites/x/www available 763G - zroot/home/sites/x/www referenced 276M - zroot/home/sites/x/www compressratio 1.19x - zroot/home/sites/x/www mounted yes - zroot/home/sites/x/www mountpoint /home/sites/x/www inherited from zroot/home zroot/home/sites/x/www compression lz4 inherited from zroot zroot/home/sites/x/www atime off inherited from zroot zroot/home/sites/x/www exec off inherited from zroot/home/sites zroot/home/sites/x/www setuid off inherited from zroot/home/sites zroot/home/sites/x/www createtxg 14688 - zroot/home/sites/x/www version 5 - zroot/home/sites/x/www utf8only off - zroot/home/sites/x/www normalization none - zroot/home/sites/x/www casesensitivity sensitive - zroot/home/sites/x/www guid 8959091402359551840 - zroot/home/sites/x/www usedbysnapshots 0B - zroot/home/sites/x/www usedbydataset 276M - zroot/home/sites/x/www usedbychildren 0B - zroot/home/sites/x/www usedbyrefreservation 0B - zroot/home/sites/x/www objsetid 863 - zroot/home/sites/x/www refcompressratio 1.19x - zroot/home/sites/x/www written 276M - zroot/home/sites/x/www logicalused 163M - zroot/home/sites/x/www logicalreferenced 163M - # zfs get all zroot/home/sites/x | grep -v default NAME PROPERTY VALUE SOURCE zroot/home/sites/x type filesystem - zroot/home/sites/x creation Thu Oct 2 22:28 2025 - zroot/home/sites/x used 2.14G - zroot/home/sites/x available 763G - zroot/home/sites/x referenced 1.86G - zroot/home/sites/x compressratio 6.00x - zroot/home/sites/x mounted yes - zroot/home/sites/x mountpoint /home/sites/x inherited from zroot/home zroot/home/sites/x compression lz4 inherited from zroot zroot/home/sites/x atime off inherited from zroot zroot/home/sites/x exec off inherited from zroot/home/sites zroot/home/sites/x setuid off inherited from zroot/home/sites zroot/home/sites/x createtxg 2832 - zroot/home/sites/x version 5 - zroot/home/sites/x utf8only off - zroot/home/sites/x normalization none - zroot/home/sites/x casesensitivity sensitive - zroot/home/sites/x guid 7802420260360558524 - zroot/home/sites/x usedbysnapshots 0B - zroot/home/sites/x usedbydataset 1.86G - zroot/home/sites/x usedbychildren 282M - zroot/home/sites/x usedbyrefreservation 0B - zroot/home/sites/x objsetid 939 - zroot/home/sites/x refcompressratio 6.35x - zroot/home/sites/x written 1.86G - zroot/home/sites/x logicalused 12.0G - zroot/home/sites/x logicalreferenced 11.8G - # sysctl kstat.zfs.misc.arcstats kstat.zfs.misc.arcstats.abd_chunk_waste_size: 27941888 kstat.zfs.misc.arcstats.cached_only_in_progress: 0 kstat.zfs.misc.arcstats.arc_raw_size: 0 kstat.zfs.misc.arcstats.arc_sys_free: 0 kstat.zfs.misc.arcstats.arc_need_free: 0 kstat.zfs.misc.arcstats.demand_iohit_prescient_prefetch: 4 kstat.zfs.misc.arcstats.demand_hit_prescient_prefetch: 16 kstat.zfs.misc.arcstats.prescient_prefetch: 20 kstat.zfs.misc.arcstats.demand_iohit_predictive_prefetch: 4102 kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 2861087 kstat.zfs.misc.arcstats.predictive_prefetch: 2906774 kstat.zfs.misc.arcstats.async_upgrade_sync: 1340 kstat.zfs.misc.arcstats.arc_dnode_limit: 3282970214 kstat.zfs.misc.arcstats.arc_meta_used: 2282039376 kstat.zfs.misc.arcstats.arc_prune: 0 kstat.zfs.misc.arcstats.arc_loaned_bytes: 0 kstat.zfs.misc.arcstats.arc_tempreserve: 0 kstat.zfs.misc.arcstats.arc_no_grow: 1 kstat.zfs.misc.arcstats.memory_available_bytes: 985264128 kstat.zfs.misc.arcstats.memory_free_bytes: 1691168768 kstat.zfs.misc.arcstats.memory_all_bytes: 33903443968 kstat.zfs.misc.arcstats.memory_indirect_count: 74 kstat.zfs.misc.arcstats.memory_direct_count: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 0 kstat.zfs.misc.arcstats.l2_rebuild_log_blks: 0 kstat.zfs.misc.arcstats.l2_rebuild_bufs_precached: 0 kstat.zfs.misc.arcstats.l2_rebuild_bufs: 0 kstat.zfs.misc.arcstats.l2_rebuild_asize: 0 kstat.zfs.misc.arcstats.l2_rebuild_size: 0 kstat.zfs.misc.arcstats.l2_rebuild_lowmem: 0 kstat.zfs.misc.arcstats.l2_rebuild_cksum_lb_errors: 0 kstat.zfs.misc.arcstats.l2_rebuild_dh_errors: 0 kstat.zfs.misc.arcstats.l2_rebuild_io_errors: 0 kstat.zfs.misc.arcstats.l2_rebuild_unsupported: 0 kstat.zfs.misc.arcstats.l2_rebuild_success: 0 kstat.zfs.misc.arcstats.l2_data_to_meta_ratio: 0 kstat.zfs.misc.arcstats.l2_log_blk_count: 0 kstat.zfs.misc.arcstats.l2_log_blk_asize: 0 kstat.zfs.misc.arcstats.l2_log_blk_avg_asize: 0 kstat.zfs.misc.arcstats.l2_log_blk_writes: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.l2_asize: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_evict_l1cached: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_writes_lock_retry: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_bufc_metadata_asize: 0 kstat.zfs.misc.arcstats.l2_bufc_data_asize: 0 kstat.zfs.misc.arcstats.l2_mfu_asize: 0 kstat.zfs.misc.arcstats.l2_mru_asize: 0 kstat.zfs.misc.arcstats.l2_prefetch_asize: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.uncached_evictable_metadata: 0 kstat.zfs.misc.arcstats.uncached_evictable_data: 0 kstat.zfs.misc.arcstats.uncached_metadata: 0 kstat.zfs.misc.arcstats.uncached_data: 0 kstat.zfs.misc.arcstats.uncached_size: 0 kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 0 kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 4881907712 kstat.zfs.misc.arcstats.mfu_ghost_metadata: 0 kstat.zfs.misc.arcstats.mfu_ghost_data: 4881907712 kstat.zfs.misc.arcstats.mfu_ghost_size: 4881907712 kstat.zfs.misc.arcstats.mfu_evictable_metadata: 143363584 kstat.zfs.misc.arcstats.mfu_evictable_data: 9197573632 kstat.zfs.misc.arcstats.mfu_metadata: 361143808 kstat.zfs.misc.arcstats.mfu_data: 9354936832 kstat.zfs.misc.arcstats.mfu_size: 9716080640 kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 0 kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 5151820800 kstat.zfs.misc.arcstats.mru_ghost_metadata: 0 kstat.zfs.misc.arcstats.mru_ghost_data: 5151820800 kstat.zfs.misc.arcstats.mru_ghost_size: 5151820800 kstat.zfs.misc.arcstats.mru_evictable_metadata: 63363072 kstat.zfs.misc.arcstats.mru_evictable_data: 8079086592 kstat.zfs.misc.arcstats.mru_metadata: 567136768 kstat.zfs.misc.arcstats.mru_data: 8619606528 kstat.zfs.misc.arcstats.mru_size: 9186743296 kstat.zfs.misc.arcstats.anon_evictable_metadata: 0 kstat.zfs.misc.arcstats.anon_evictable_data: 0 kstat.zfs.misc.arcstats.anon_metadata: 9278976 kstat.zfs.misc.arcstats.anon_data: 119400960 kstat.zfs.misc.arcstats.anon_size: 128679936 kstat.zfs.misc.arcstats.other_size: 1100486960 kstat.zfs.misc.arcstats.bonus_size: 194260160 kstat.zfs.misc.arcstats.dnode_size: 709538280 kstat.zfs.misc.arcstats.dbuf_size: 196688520 kstat.zfs.misc.arcstats.metadata_size: 937559552 kstat.zfs.misc.arcstats.data_size: 18093944320 kstat.zfs.misc.arcstats.hdr_size: 243992864 kstat.zfs.misc.arcstats.overhead_size: 1214676992 kstat.zfs.misc.arcstats.uncompressed_size: 53352812544 kstat.zfs.misc.arcstats.compressed_size: 17816826880 kstat.zfs.misc.arcstats.size: 20403925584 kstat.zfs.misc.arcstats.c_max: 32829702144 kstat.zfs.misc.arcstats.c_min: 1059482624 kstat.zfs.misc.arcstats.c: 20487444490 kstat.zfs.misc.arcstats.pm: 2147483648 kstat.zfs.misc.arcstats.pd: 2083816569 kstat.zfs.misc.arcstats.meta: 970340663 kstat.zfs.misc.arcstats.hash_chain_max: 6 kstat.zfs.misc.arcstats.hash_chains: 104985 kstat.zfs.misc.arcstats.hash_collisions: 9094239 kstat.zfs.misc.arcstats.hash_elements_max: 1017607 kstat.zfs.misc.arcstats.hash_elements: 1017607 kstat.zfs.misc.arcstats.evict_l2_skip: 0 kstat.zfs.misc.arcstats.evict_l2_ineligible: 99614720 kstat.zfs.misc.arcstats.evict_l2_eligible_mru: 10961854464 kstat.zfs.misc.arcstats.evict_l2_eligible_mfu: 162220245504 kstat.zfs.misc.arcstats.evict_l2_eligible: 173182099968 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_not_enough: 0 kstat.zfs.misc.arcstats.evict_skip: 88612 kstat.zfs.misc.arcstats.access_skip: 23 kstat.zfs.misc.arcstats.mutex_miss: 93 kstat.zfs.misc.arcstats.deleted: 1282149 kstat.zfs.misc.arcstats.uncached_hits: 380244 kstat.zfs.misc.arcstats.mfu_ghost_hits: 65466 kstat.zfs.misc.arcstats.mfu_hits: 18867017351 kstat.zfs.misc.arcstats.mru_ghost_hits: 92342 kstat.zfs.misc.arcstats.mru_hits: 46794883 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 25067 kstat.zfs.misc.arcstats.prefetch_metadata_iohits: 26586 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 187438 kstat.zfs.misc.arcstats.prefetch_data_misses: 1160285 kstat.zfs.misc.arcstats.prefetch_data_iohits: 0 kstat.zfs.misc.arcstats.prefetch_data_hits: 1507418 kstat.zfs.misc.arcstats.demand_metadata_misses: 42409 kstat.zfs.misc.arcstats.demand_metadata_iohits: 3145 kstat.zfs.misc.arcstats.demand_metadata_hits: 12904666598 kstat.zfs.misc.arcstats.demand_data_misses: 531109 kstat.zfs.misc.arcstats.demand_data_iohits: 1178 kstat.zfs.misc.arcstats.demand_data_hits: 6007831233 kstat.zfs.misc.arcstats.misses: 1758870 kstat.zfs.misc.arcstats.iohits: 30909 kstat.zfs.misc.arcstats.hits: 18914192788 # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XDOM kstack_cache: 16384, 0, 673, 22, 8227, 0, 0, 0 buffer arena-40: 4096, 0, 0, 0, 0, 0, 0, 0 buffer arena-81: 8192, 0, 0, 0, 0, 0, 0, 0 buffer arena-12: 12288, 0, 0, 0, 0, 0, 0, 0 buffer arena-16: 16384, 0, 10, 0, 10, 0, 0, 0 buffer arena-20: 20480, 0, 0, 0, 0, 0, 0, 0 buffer arena-24: 24576, 0, 0, 0, 0, 0, 0, 0 buffer arena-28: 28672, 0, 0, 0, 0, 0, 0, 0 buffer arena-32: 32768, 0, 4, 0, 4, 0, 0, 0 buffer arena-36: 36864, 0, 0, 0, 0, 0, 0, 0 buffer arena-40: 40960, 0, 0, 0, 0, 0, 0, 0 buffer arena-45: 45056, 0, 0, 0, 0, 0, 0, 0 buffer arena-49: 49152, 0, 0, 0, 0, 0, 0, 0 buffer arena-53: 53248, 0, 0, 0, 0, 0, 0, 0 buffer arena-57: 57344, 0, 0, 0, 0, 0, 0, 0 buffer arena-61: 61440, 0, 0, 0, 0, 0, 0, 0 buffer arena-65: 65536, 0, 0, 0, 0, 0, 0, 0 buf free cache: 432, 0, 14, 196, 14, 0, 0, 0 vm pgcache: 4096, 0, 844303, 3296,1004668630,3061, 0, 0 vm pgcache: 4096, 0, 5644515, 718,32595219, 610, 0, 0 UMA Kegs: 384, 0, 157, 6, 157, 0, 0, 0 UMA Zones: 2048, 0, 183, 7, 183, 0, 0, 0 UMA Slabs 0: 80, 0, 4973917, 372927,19957826, 0, 0, 0 UMA Slabs 1: 112, 0, 11, 24, 11, 0, 0, 0 UMA Hash: 256, 0, 0, 0, 0, 0, 0, 0 2 Bucket: 32, 0, 935, 8389, 3046611, 0, 0, 0 4 Bucket: 48, 0, 36, 4668, 343338, 0, 0, 0 8 Bucket: 80, 0, 4346, 2004, 3539030, 0, 0, 0 16 Bucket: 144, 0, 1336, 2696, 2918946, 0, 0, 0 32 Bucket: 256, 0, 7401, 504, 7134107, 1, 0, 0 64 Bucket: 512, 0, 6764, 324,62634999, 185, 0, 0 128 Bucket: 1024, 0, 1600, 175,11047021, 211, 0, 0 256 Bucket: 2048, 0, 8416, 4900,29106739,2102558, 0, 0 SMR SHARED: 24, 0, 8, 759, 8, 0, 0, 0 SMR CPU: 32, 0, 8, 759, 8, 0, 0, 0 vmem: 1856, 0, 2, 7, 2, 0, 0, 0 vmem btag: 56, 0, 132182, 42334, 1138834,1214, 0, 0 VM OBJECT: 248, 0, 238509, 93395,51578461, 0, 0, 0 RADIX NODE: 152, 0, 154419, 21208,82398204, 0, 0, 0 KMAP ENTRY: 96, 0, 33, 6, 35, 0, 0, 0 MAP ENTRY: 96, 0, 4709, 7387,17333200, 0, 0, 0 VMSPACE: 584, 0, 55, 267, 88176, 0, 0, 0 fakepg: 88, 0, 0, 138, 4, 0, 0, 0 pcpu-4: 4, 0, 0, 0, 0, 0, 0, 0 pcpu-8: 8, 0, 4144, 2000, 4146, 0, 0, 0 pcpu-16: 16, 0, 44, 2772, 44, 0, 0, 0 pcpu-32: 32, 0, 0, 0, 0, 0, 0, 0 pcpu-64: 64, 0, 467, 813, 467, 0, 0, 0 malloc-16: 16, 0, 19064, 4120,8363168925, 0, 0, 0 malloc-32: 32, 0, 271616, 96808,4697673642, 0, 0, 0 malloc-64: 64, 0, 281350, 181889,14155325777, 0, 0, 0 malloc-128: 128, 0, 1050510, 116454,14491626167, 0, 0, 0 malloc-256: 256, 0, 125059, 91541,1197604864, 0, 0, 0 malloc-384: 384, 0, 565691, 106349,310312929, 0, 0, 0 malloc-512: 512, 0, 240, 1432,156315676, 0, 0, 0 malloc-1024: 1024, 0, 2005, 68367,119633993, 0, 0, 0 malloc-2048: 2048, 0, 677, 213,102092015, 0, 0, 0 malloc-4096: 4096, 0, 33684, 69,262040425, 0, 0, 0 malloc-8192: 8192, 0, 86, 33, 8038, 0, 0, 0 malloc-16384: 16384, 0, 29, 30,56428258, 0, 0, 0 malloc-32768: 32768, 0, 30, 11, 812, 0, 0, 0 malloc-65536: 65536, 0, 14, 13, 81454, 0, 0, 0 SLEEPQUEUE: 80, 0, 809, 1111, 809, 0, 0, 0 kenv: 258, 0, 18, 342, 10063, 0, 0, 0 Files: 80, 0, 866, 2434,2179224078, 0, 0, 0 filedesc0: 1072, 0, 77, 161, 95038, 0, 0, 0 PWD: 40, 0, 24, 4319, 2967629, 0, 0, 0 rl_entry: 40, 0, 20428, 4721,2127216242, 0, 0, 0 rangeset pctrie nodes: 152, 0, 0, 0, 0, 0, 0, 0 TURNSTILE: 136, 0, 809, 388, 809, 0, 0, 0 lkpicurr: 168, 0, 2, 3142, 3, 0, 0, 0 lkpimm: 56, 0, 1, 3167, 2, 0, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0, 0 umtx_shm: 88, 0, 0, 0, 0, 0, 0, 0 swrbuf: 2360, 192, 0, 0, 0, 0, 0, 0 swwbuf: 2360, 96, 0, 0, 0, 0, 0, 0 nfspbuf: 2360, 192, 0, 0, 0, 0, 0, 0 vnpbuf: 2360, 768, 0, 0, 0, 0, 0, 0 clpbuf: 2360, 192, 0, 0, 0, 0, 0, 0 pbuf: 2360, 16, 0, 1170, 0, 0, 0, 0 FPU_save_area: 2440, 0, 675, 75, 444110, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0, 0 PROC: 1368, 0, 76, 133, 95037, 0, 0, 0 PGRP: 120, 0, 37, 1481, 12981, 0, 0, 0 THREAD: 1860, 0, 673, 135, 8227, 0, 0, 0 cpuset: 200, 0, 39, 713, 39, 0, 0, 0 domainset: 40, 0, 0, 0, 0, 0, 0, 0 pkru ranges: 24, 0, 0, 0, 0, 0, 0, 0 audit_record: 1280, 0, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 20, 4298,1221253241, 0, 0, 0 mbuf: 256,12939731, 3211, 9886,3808837764, 0, 0, 0 mbuf_cluster: 2048,2021833, 7389, 3917, 6878252, 0, 0, 0 mbuf_jumbo_page: 4096,1010916, 46, 4549,1083988772, 0, 0, 0 mbuf_jumbo_9k: 9216, 299530, 0, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 168486, 0, 0, 0, 0, 0, 0 epoch_record pcpu: 256, 0, 4, 60, 4, 0, 0, 0 ttyinq: 160, 0, 420, 530, 795, 0, 0, 0 ttyoutq: 256, 0, 219, 531, 417, 0, 0, 0 skbuff: 1808, 0, 0, 0, 0, 0, 0, 0 IOMMU_MAP_ENTRY: 112, 0, 0, 0, 0, 0, 0, 0 DEVCTL: 1024, 1000, 0, 100, 517, 0, 0, 0 g_bio: 376, 0, 0, 1080,106457875, 0, 0, 0 linux_dma_pctrie: 152, 0, 0, 0, 0, 0, 0, 0 linux_dma_object: 32, 0, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 168, 0, 0, 0, 0 cryptop: 280, 0, 0, 0, 0, 0, 0, 0 taskq_zone: 160, 0, 0, 2184,13319951, 0, 0, 0 VNODE: 440, 0, 540562, 90824,3649099928, 0, 0, 0 BUF TRIE: 152, 0, 5, 105945, 5, 0, 0, 0 S VFS Cache: 104, 0, 544663, 184559,964582813, 0, 0, 0 STS VFS Cache: 144, 0, 0, 0, 0, 0, 0, 0 L VFS Cache: 320, 0, 58432, 620, 61650, 0, 0, 0 LTS VFS Cache: 360, 0, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 2, 234,9577136004, 0, 0, 0 NCLNODE: 608, 0, 0, 0, 0, 0, 0, 0 TMPFS node: 240, 0, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0, 0 pipe: 736, 0, 6, 247, 50894, 0, 0, 0 AIO: 208, 0, 3, 510, 15, 0, 0, 0 AIOCB: 552, 0, 0, 385,11454075, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0, 0 Mountpoints: 2816, 0, 24, 20, 24, 0, 0, 0 zfs_btree_leaf_cache: 4096, 0, 420, 68222,100219137, 0, 0, 0 metaslab_alloc_trace_cache:72, 0, 0, 0, 0, 0, 0, 0 brt_entry_cache: 168, 0, 0, 2112, 9566, 0, 0, 0 ddt_cache: 29240, 0, 5, 1, 5, 0, 0, 0 ddt_entry_flat_cache: 184, 0, 0, 0, 0, 0, 0, 0 ddt_entry_trad_cache: 368, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_flat_cache:144, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_trad_cache:328, 0, 0, 0, 0, 0, 0, 0 zio_cache: 1152, 0, 16, 78300,8481397489, 0, 0, 0 zio_link_cache: 48, 0, 0, 78792,767079482, 0, 0, 0 zio_buf_comb_512: 512, 0, 1366012, 1578060,1420577179, 0, 0, 0 zio_buf_comb_1024: 1024, 0, 9512, 6544, 9438728, 0, 0, 0 zio_buf_comb_1536: 1536, 0, 6475, 5150, 5112094, 0, 0, 0 zio_buf_comb_2048: 2048, 0, 4294, 3254, 3461483, 0, 0, 0 zio_buf_comb_3072: 3072, 0, 14741, 25079,26768368, 0, 0, 0 zio_buf_comb_4096: 4096, 0, 72352, 11097,36431683, 0, 0, 0 zio_buf_comb_6144: 6144, 0, 9, 721, 6483480, 0, 0, 0 zio_buf_comb_8192: 8192, 0, 4, 379,15771599, 0, 0, 0 zio_buf_comb_12288: 12288, 0, 4, 81, 4786468, 0, 0, 0 zio_buf_comb_16384: 16384, 0, 34230, 2904,32898875, 0, 0, 0 zio_buf_comb_24576: 24576, 0, 4, 58, 1654310, 0, 0, 0 zio_buf_comb_32768: 32768, 0, 6, 31, 928036, 0, 0, 0 zio_buf_comb_49152: 49152, 0, 19, 34, 1788802, 0, 0, 0 zio_buf_comb_65536: 65536, 0, 30, 44, 891995, 0, 0, 0 zio_buf_comb_98304: 98304, 0, 129, 48, 1447082, 0, 0, 0 zio_buf_comb_131072: 131072, 0, 5177, 2171,15216190, 0, 0, 0 zio_buf_comb_196608: 196608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_262144: 262144, 0, 0, 1, 17, 0, 0, 0 zio_buf_comb_393216: 393216, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_524288: 524288, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_786432: 786432, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_1048576:1048576, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_1572864:1572864, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_2097152:2097152, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_3145728:3145728, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_4194304:4194304, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_6291456:6291456, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_8388608:8388608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_12582912:12582912, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_16777216:16777216, 0, 0, 0, 0, 0, 0, 0 abd_chunk: 4096, 0, 4318036, 632,53939212, 0, 0, 0 sa_cache: 264, 0, 540406, 82604,49065995, 0, 0, 0 dnode_t: 760, 0, 932459, 55136,12137113, 0, 0, 0 arc_buf_hdr_t_full: 232, 0, 997020, 50180,421922938, 0, 0, 0 arc_buf_hdr_t_l2only: 96, 0, 0, 0, 0, 0, 0, 0 arc_buf_t: 32, 0, 40429, 81665,446777460, 0, 0, 0 dmu_buf_impl_t: 280, 0, 581171, 165449,486635758, 0, 0, 0 dbuf_dirty_record_t: 376, 0, 1643, 96397,158527418, 0, 0, 0 zil_lwb_cache: 376, 0, 58, 1182, 4083369, 0, 0, 0 zil_zcw_cache: 80, 0, 0, 1900, 3735150, 0, 0, 0 sio_cache_0: 136, 0, 0, 0, 0, 0, 0, 0 sio_cache_1: 152, 0, 0, 0, 0, 0, 0, 0 sio_cache_2: 168, 0, 0, 0, 0, 0, 0, 0 zap_name: 328, 0, 0, 408,4147714242, 0, 0, 0 zap_attr_cache: 288, 0, 0, 364, 27382, 0, 0, 0 zap_name_long: 1096, 0, 0, 0, 0, 0, 0, 0 zap_attr_long_cache: 1056, 0, 0, 133,260549647, 0, 0, 0 zfs_znode_cache: 456, 0, 540406, 96737,49065995, 0, 0, 0 ksiginfo: 112, 0, 203, 2533, 420371, 0, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0, 0 KNOTE: 160, 0, 160, 1340,67699396, 0, 0, 0 socket: 1024,1034651, 275, 3769,10474914, 0, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0, 0 unpcb: 320,1034651, 45, 3795, 8255784, 0, 0, 0 ripcb: 376, 0, 1, 369, 345, 0, 0, 0 tcp_inpcb: 1304, 0, 208, 203, 929640, 0, 0, 0 udp_inpcb: 408, 0, 18, 531, 1289098, 0, 0, 0 udplite_inpcb: 408, 0, 0, 0, 0, 0, 0, 0 routing nhops: 256, 0, 8, 187, 13, 0, 0, 0 tcpreass: 48, 126364, 0, 1596, 316394, 0, 0, 0 tcp_log: 408,5000000, 0, 0, 0, 0, 0, 0 tcp_log_id_bucket: 176, 0, 0, 0, 0, 0, 0, 0 tcp_log_id_node: 120, 0, 0, 0, 0, 0, 0, 0 rtentry: 168, 0, 11, 181, 13, 0, 0, 0 ipq: 56, 51200, 0, 576, 7, 0, 0, 0 syncache: 176, 15360, 1, 4392, 1182096, 0, 0, 0 hostcache: 64, 15360, 3085, 11972, 287572,6101, 0, 0 sackhole: 32, 0, 0, 2394, 1017279, 0, 0, 0 tfo: 4, 0, 0, 0, 0, 0, 0, 0 tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0, 0 ada_ccb: 272, 0, 0, 0, 0, 0, 0, 0 da_ccb: 536, 0, 0, 0, 0, 0, 0, 0 swpctrie: 152,4043676, 0, 0, 0, 0, 0, 0 swblk: 136,4043673, 0, 0, 0, 0, 0, 0 IPFW counters: 16, 0, 60, 2756, 71, 0, 0, 0 IPFW dynamic states data:88, 16384, 10, 2980, 630566, 0, 0, 0 IPFW parent dynamic states:32, 4096, 0, 0, 0, 0, 0, 0 IPFW IPv4 dynamic states:48, 0, 10, 3350, 630566, 0, 0, 0 IPFW IPv6 dynamic states:72, 0, 0, 0, 0, 0, 0, 0 # iostat -x 1 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 2 43 38.5 1362.2 0 0 0 0 0 0 nda1 2 43 38.7 1362.2 0 0 0 0 0 0 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 0 4 0.0 35.5 0 0 0 0 0 0 nda1 0 4 0.0 35.5 0 0 0 0 0 0 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 0 388 0.0 33440.7 0 0 0 0 0 8 nda1 0 376 0.0 33440.7 0 0 0 0 0 9 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 0 0 0.0 0.0 0 0 0 0 0 0 nda1 0 0 0.0 0.0 0 0 0 0 0 0 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 0 30 0.0 308.0 0 0 0 0 0 0 nda1 0 30 0.0 308.0 0 0 0 0 0 0 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 ^C extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b nda0 0 7 0.0 64.8 0 0 0 0 0 0 nda1 0 7 0.0 64.8 0 0 0 0 0 0 pass0 0 0 0.0 0.0 0 0 0 0 0 0 pass1 0 0 0.0 0.0 0 0 0 0 0 0 # dtrace -n 'fbt::zap_lookup:entry /execname == "mv"/ { self->ts = timestamp; } fbt::zap_lookup:return /self->ts/ { printf("lat: %d", timestamp - self->ts); }' | more dtrace: description 'fbt::zap_lookup:entry ' matched 2 probes CPU ID FUNCTION:NAME 5 76354 zap_lookup:return lat: 1342 5 76354 zap_lookup:return lat: 601 5 76354 zap_lookup:return lat: 311 5 76354 zap_lookup:return lat: 330 5 76354 zap_lookup:return lat: 471 5 76354 zap_lookup:return lat: 320 5 76354 zap_lookup:return lat: 321 5 76354 zap_lookup:return lat: 321 5 76354 zap_lookup:return lat: 421 5 76354 zap_lookup:return lat: 350 5 76354 zap_lookup:return lat: 291 5 76354 zap_lookup:return lat: 330 5 76354 zap_lookup:return lat: 411 5 76354 zap_lookup:return lat: 320 5 76354 zap_lookup:return lat: 310 5 76354 zap_lookup:return lat: 340 5 76354 zap_lookup:return lat: 410 5 76354 zap_lookup:return lat: 341 5 76354 zap_lookup:return lat: 290 5 76354 zap_lookup:return lat: 351 5 76354 zap_lookup:return lat: 421 5 76354 zap_lookup:return lat: 351 5 76354 zap_lookup:return lat: 311 5 76354 zap_lookup:return lat: 371 5 76354 zap_lookup:return lat: 431 5 76354 zap_lookup:return lat: 321 5 76354 zap_lookup:return lat: 291 5 76354 zap_lookup:return lat: 321 5 76354 zap_lookup:return lat: 441 5 76354 zap_lookup:return lat: 381 5 76354 zap_lookup:return lat: 301 5 76354 zap_lookup:return lat: 331 5 76354 zap_lookup:return lat: 411 5 76354 zap_lookup:return lat: 330 5 76354 zap_lookup:return lat: 301 5 76354 zap_lookup:return lat: 692 5 76354 zap_lookup:return lat: 410 5 76354 zap_lookup:return lat: 321 5 76354 zap_lookup:return lat: 291 5 76354 zap_lookup:return lat: 320 5 76354 zap_lookup:return lat: 411 5 76354 zap_lookup:return lat: 350 5 76354 zap_lookup:return lat: 301 5 76354 zap_lookup:return lat: 320 dtrace: failed to write to <stdout>: Broken pipe Sorry about the long report. Trying to provide anything that could be useful before I have to reboot. -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-293750-227>