From owner-freebsd-fs@freebsd.org Sun Jun 19 19:38:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44F04A7ADFB for ; Sun, 19 Jun 2016 19:38:10 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B81B52726 for ; Sun, 19 Jun 2016 19:38:09 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm0-x229.google.com with SMTP id a66so52085911wme.0 for ; Sun, 19 Jun 2016 12:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=HU0kcL6RSAjOe42a8yzDDEC3w20eR8/JO8Y1J1WG8oI=; b=X1KmXEU8SplTvvv1tcmBr9aEud4vmYzYXaq6iPYrZGrQ0VpgoTSxeiRdBW5kx+wDzJ /QYx6Ap3cfGScYst5NMdLP497QSrXCFOQU6LQDKDXT3UDhqeKezYxtjGqzYEZseUw0BY xvm/fyFu0x8iDwXlE3B7jITpwnJ7+6OqBp+CV9IgWxnC8MPnG91BHEAokuECg54c4G0A 1wPUdZKCEfGp7lwXnJRfJmYvJCem1Fq3IYFDYFSYk3uboSeVYVYqskc2ehVzTl5L22/E Z17nlgnUuc2um5YMH/2ZN8PDHoM4vHmup8DgO5Z3/IwuZnyY+rSKERm5r6oFpOalbEzn 36pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=HU0kcL6RSAjOe42a8yzDDEC3w20eR8/JO8Y1J1WG8oI=; b=PboFbD6V41HYusd+4pyk4q6gB2zTyrAqbY8OUKlMWJMD1ElGZcac+hkhQXAkyu44jE y8F5DqEphQkTYuFe0dsFUZdAOmQFcTEgs1kUMWURh4Mj889/6QhyNKj+7S0X6wiYWkgT OXIjzpIeN/nFYTW0q9Km4IPvvE4EcZ/XyAaXMtTOn5rGE7ZFyq8Q/+PaXEOWg5TNs8v/ sZqSdCf/aoj+M9kxyhDQw7JUKWbvyXTFRa+/FDYpsFE/ZuhU7V+/i4qT3mN3rs1q+BSw FVpc3+oaf6Pn0dknfLcLGkRdLitXriBAAC1YT3+YQ0FbFSQoCuzDR0+SXzljw/Hs/Yw9 cUlA== X-Gm-Message-State: ALyK8tIlFrd1oIi8KwgS6bnZyU2GUqNfnWpGZhynxOOnPG4kWZJhUmMqJMBCRsdh6qlCdQ== X-Received: by 10.28.141.4 with SMTP id p4mr8325567wmd.46.1466365087968; Sun, 19 Jun 2016 12:38:07 -0700 (PDT) Received: from [192.168.20.30] (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.googlemail.com with ESMTPSA id z5sm4727158wme.5.2016.06.19.12.38.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 Jun 2016 12:38:07 -0700 (PDT) To: FreeBSD Filesystems From: Kaya Saman Subject: High CPU Interrupt using ZFS Message-ID: <57cfcda4-6ff7-0c2e-4f58-ad09ce7cab28@gmail.com> Date: Sun, 19 Jun 2016 20:38:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jun 2016 19:38:10 -0000 Hi, I have a strange problem and I'm not sure if anyone has ever experienced this to help give me some advice on how to tackle it. Basically I run ZFS as root FS mirrored over two drives which are directly connected to the SATA connectors on a SuperMicro Xeon E5 server based MB. Then I have an LSI HBA connected to the remaining disks with various ZPOOLs. The main pool has ZIL and L2ARC enabled. As the majority of data is A/V content I disabled Prefetch as instructed in the FreeBSD tuning tips guide. https://www.freebsd.org/doc/handbook/zfs.html For some reason after a period of time the CPU interrupt will just got sky high and the system will totally bog down. My home drive is running off the "Main Pool" too and when this happens it becomes inaccessible. The system runs FBSD 10.3: 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016 ZPOOL List output: # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ZPOOL_2 27.2T 26.3T 884G - 41% 96% 1.00x ONLINE - ZPOOL_3 298G 248G 50.2G - 34% 83% 1.00x ONLINE - ZPOOL_4 1.81T 1.75T 66.4G - 25% 96% 1.00x ONLINE - ZPOOL_5 186G 171G 14.9G - 62% 92% 1.00x ONLINE - workspaces 119G 77.7G 41.3G - 56% 65% 1.00x ONLINE - zroot 111G 88.9G 22.1G - 70% 80% 1.00x ONLINE - The system has a Xeon E5 with 24GB RAM and 16GB of Swap space. I also run 5x jails on this box. 1x for network based monitoring (munin, zabbix etc) 1x DB jail which runs Postgresql and Mysql + some others; they are all run off the ZRoot Boot Loader info: zfs_load="YES" kern.ipc.semmni=6000000 kern.ipc.semmns=6000000 kern.ipc.semmnu=256 net.isr.numthreads=4 net.isr.maxthreads=4 net.isr.bindthreads=1 vfs.zfs.l2arc_noprefetch=1 Other information: # camcontrol devlist at scbus0 target 8 lun 0 (pass0,da0) at scbus0 target 10 lun 0 (pass1,da1) at scbus0 target 11 lun 0 (pass2,da2) at scbus0 target 12 lun 0 (pass3,da3) at scbus0 target 13 lun 0 (pass4,ses0) at scbus0 target 14 lun 0 (pass5,da4) at scbus0 target 15 lun 0 (pass6,da5) at scbus0 target 17 lun 0 (pass7,da6) at scbus0 target 18 lun 0 (pass8,da7) at scbus0 target 19 lun 0 (pass9,da8) at scbus0 target 20 lun 0 (pass10,da9) at scbus0 target 21 lun 0 (pass11,da10) at scbus0 target 22 lun 0 (pass12,da11) at scbus0 target 29 lun 0 (pass13,da12) at scbus0 target 30 lun 0 (pass14,da13) at scbus0 target 31 lun 0 (pass15,da14) at scbus0 target 34 lun 0 (pass16,da15) at scbus0 target 35 lun 0 (pass17,da16) at scbus0 target 36 lun 0 (pass18,da17) at scbus0 target 37 lun 0 (pass19,da18) at scbus0 target 38 lun 0 (pass20,da19) at scbus0 target 40 lun 0 (pass21,da20) at scbus0 target 41 lun 0 (pass22,da21) at scbus2 target 0 lun 0 (pass23,ada0) at scbus3 target 0 lun 0 (pass24,ada1) at scbus8 target 0 lun 0 (pass25,ses1) Sysctl output for ZFS: # sysctl -a |grep zfs 2 PART diskid/DISK-1350790500009986007Fp2 229319956992 512 i 2 o 10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b 2 PART diskid/DISK-1350790500009986007Fp1 10737418240 512 i 1 o 17408 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b 2 PART diskid/DISK-13507905000099860071p2 229319956992 512 i 2 o 10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b 2 PART diskid/DISK-13507905000099860071p1 10735321088 512 i 1 o 2097152 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b 2 PART diskid/DISK-14067903000097960BD7p3 119445590016 512 i 3 o 8590065664 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b 1 PART ada0p3 119445590016 512 i 3 o 8590065664 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b z0xfffff80012422d00 [shape=box,label="ZFS::VDEV\nzfs::vdev\nr#4"]; zfs::vdev freebsd-zfs freebsd-zfs freebsd-zfs freebsd-zfs freebsd-zfs freebsd-zfs vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 1 vfs.zfs.vol.unmap_enabled: 1 vfs.zfs.vol.mode: 1 vfs.zfs.version.zpl: 5 vfs.zfs.version.spa: 5000 vfs.zfs.version.acl: 1 vfs.zfs.version.ioctl: 5 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 vfs.zfs.sync_pass_rewrite: 2 vfs.zfs.sync_pass_dont_compress: 5 vfs.zfs.sync_pass_deferred_free: 2 vfs.zfs.zio.exclude_metadata: 0 vfs.zfs.zio.use_uma: 1 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_replay_disable: 0 vfs.zfs.min_auto_ashift: 9 vfs.zfs.max_auto_ashift: 13 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.bio_delete_disable: 0 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.scrub_max_active: 2 vfs.zfs.vdev.scrub_min_active: 1 vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_write_min_active: 1 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.async_read_min_active: 1 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_write_min_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 vfs.zfs.vdev.sync_read_min_active: 10 vfs.zfs.vdev.max_active: 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent: 60 vfs.zfs.vdev.async_write_active_min_dirty_percent: 30 vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 vfs.zfs.vdev.mirror.non_rotating_inc: 0 vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 vfs.zfs.vdev.mirror.rotating_seek_inc: 5 vfs.zfs.vdev.mirror.rotating_inc: 0 vfs.zfs.vdev.trim_on_init: 1 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.metaslabs_per_vdev: 200 vfs.zfs.txg.timeout: 5 vfs.zfs.space_map_blksz: 4096 vfs.zfs.spa_slop_shift: 5 vfs.zfs.spa_asize_inflation: 24 vfs.zfs.deadman_enabled: 1 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_synctime_ms: 1000000 vfs.zfs.recover: 0 vfs.zfs.spa_load_verify_data: 1 vfs.zfs.spa_load_verify_metadata: 1 vfs.zfs.spa_load_verify_maxinflight: 10000 vfs.zfs.check_hostid: 1 vfs.zfs.mg_fragmentation_threshold: 85 vfs.zfs.mg_noalloc_threshold: 0 vfs.zfs.condense_pct: 200 vfs.zfs.metaslab.bias_enabled: 1 vfs.zfs.metaslab.lba_weighting_enabled: 1 vfs.zfs.metaslab.fragmentation_factor_enabled: 1 vfs.zfs.metaslab.preload_enabled: 1 vfs.zfs.metaslab.preload_limit: 3 vfs.zfs.metaslab.unload_delay: 8 vfs.zfs.metaslab.load_pct: 50 vfs.zfs.metaslab.min_alloc_size: 33554432 vfs.zfs.metaslab.df_free_pct: 4 vfs.zfs.metaslab.df_alloc_threshold: 131072 vfs.zfs.metaslab.debug_unload: 0 vfs.zfs.metaslab.debug_load: 0 vfs.zfs.metaslab.fragmentation_threshold: 70 vfs.zfs.metaslab.gang_bang: 16777217 vfs.zfs.free_bpobj_enabled: 1 vfs.zfs.free_max_blocks: 18446744073709551615 vfs.zfs.no_scrub_prefetch: 0 vfs.zfs.no_scrub_io: 0 vfs.zfs.resilver_min_time_ms: 3000 vfs.zfs.free_min_time_ms: 1000 vfs.zfs.scan_min_time_ms: 1000 vfs.zfs.scan_idle: 50 vfs.zfs.scrub_delay: 4 vfs.zfs.resilver_delay: 2 vfs.zfs.top_maxinflight: 32 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.max_distance: 8388608 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 0 vfs.zfs.delay_scale: 500000 vfs.zfs.delay_min_dirty_percent: 60 vfs.zfs.dirty_data_sync: 67108864 vfs.zfs.dirty_data_max_percent: 10 vfs.zfs.dirty_data_max_max: 4294967296 vfs.zfs.dirty_data_max: 2570453401 vfs.zfs.max_recordsize: 1048576 vfs.zfs.mdcomp_disable: 0 vfs.zfs.nopwrite_enabled: 1 vfs.zfs.dedup.prefetch: 1 vfs.zfs.l2c_only_size: 0 vfs.zfs.mfu_ghost_data_lsize: 3288968704 vfs.zfs.mfu_ghost_metadata_lsize: 5136092672 vfs.zfs.mfu_ghost_size: 8425061376 vfs.zfs.mfu_data_lsize: 8574981632 vfs.zfs.mfu_metadata_lsize: 68123648 vfs.zfs.mfu_size: 8745474560 vfs.zfs.mru_ghost_data_lsize: 5324684800 vfs.zfs.mru_ghost_metadata_lsize: 923847680 vfs.zfs.mru_ghost_size: 6248532480 vfs.zfs.mru_data_lsize: 1456756224 vfs.zfs.mru_metadata_lsize: 1278004224 vfs.zfs.mru_size: 2862586368 vfs.zfs.anon_data_lsize: 0 vfs.zfs.anon_metadata_lsize: 0 vfs.zfs.anon_size: 2841088 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 134217728 vfs.zfs.l2arc_write_max: 67108864 vfs.zfs.arc_meta_limit: 5979973632 vfs.zfs.arc_free_target: 42350 vfs.zfs.arc_shrink_shift: 7 vfs.zfs.arc_average_blocksize: 8192 vfs.zfs.arc_min: 2989986816 vfs.zfs.arc_max: 23919894528 debug.zfs_flags: 0 kstat.zfs.misc.vdev_cache_stats.misses: 0 kstat.zfs.misc.vdev_cache_stats.hits: 0 kstat.zfs.misc.vdev_cache_stats.delegations: 0 kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 3034069 kstat.zfs.misc.arcstats.sync_wait_for_async: 1779944 kstat.zfs.misc.arcstats.arc_meta_min: 1494993408 kstat.zfs.misc.arcstats.arc_meta_max: 12233249160 kstat.zfs.misc.arcstats.arc_meta_limit: 5979973632 kstat.zfs.misc.arcstats.arc_meta_used: 4638138472 kstat.zfs.misc.arcstats.duplicate_reads: 1709068 kstat.zfs.misc.arcstats.duplicate_buffers_size: 0 kstat.zfs.misc.arcstats.duplicate_buffers: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 2200 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 4772510 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 561679501832704 kstat.zfs.misc.arcstats.l2_write_pios: 377935 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 1193136 kstat.zfs.misc.arcstats.l2_write_full: 148 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 264598116 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 83 kstat.zfs.misc.arcstats.l2_write_in_l2: 5284665382 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 14626890890 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 3318575 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 6328431 kstat.zfs.misc.arcstats.l2_compress_failures: 655251 kstat.zfs.misc.arcstats.l2_compress_zeros: 0 kstat.zfs.misc.arcstats.l2_compress_successes: 1205377 kstat.zfs.misc.arcstats.l2_hdr_size: 63556704 kstat.zfs.misc.arcstats.l2_asize: 84595239936 kstat.zfs.misc.arcstats.l2_size: 93178570752 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 42 kstat.zfs.misc.arcstats.l2_cdata_free_on_write: 41 kstat.zfs.misc.arcstats.l2_free_on_write: 722 kstat.zfs.misc.arcstats.l2_evict_l1cached: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_writes_lock_retry: 63 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_done: 377935 kstat.zfs.misc.arcstats.l2_writes_sent: 377935 kstat.zfs.misc.arcstats.l2_write_bytes: 101118255104 kstat.zfs.misc.arcstats.l2_read_bytes: 59571878912 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_feeds: 1193136 kstat.zfs.misc.arcstats.l2_misses: 137818470 kstat.zfs.misc.arcstats.l2_hits: 3613135 kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 5136092672 kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 3722561024 kstat.zfs.misc.arcstats.mfu_ghost_size: 8858653696 kstat.zfs.misc.arcstats.mfu_evictable_metadata: 68123648 kstat.zfs.misc.arcstats.mfu_evictable_data: 8575112704 kstat.zfs.misc.arcstats.mfu_size: 8745605632 kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 923847680 kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 5324684800 kstat.zfs.misc.arcstats.mru_ghost_size: 6248532480 kstat.zfs.misc.arcstats.mru_evictable_metadata: 1278004224 kstat.zfs.misc.arcstats.mru_evictable_data: 1457411584 kstat.zfs.misc.arcstats.mru_size: 2863241728 kstat.zfs.misc.arcstats.anon_evictable_metadata: 0 kstat.zfs.misc.arcstats.anon_evictable_data: 0 kstat.zfs.misc.arcstats.anon_size: 2038272 kstat.zfs.misc.arcstats.other_size: 2797801256 kstat.zfs.misc.arcstats.metadata_size: 1576374272 kstat.zfs.misc.arcstats.data_size: 10034527744 kstat.zfs.misc.arcstats.hdr_size: 200406240 kstat.zfs.misc.arcstats.size: 14672666216 kstat.zfs.misc.arcstats.c_max: 23919894528 kstat.zfs.misc.arcstats.c_min: 2989986816 kstat.zfs.misc.arcstats.c: 14673666683 kstat.zfs.misc.arcstats.p: 8668447917 kstat.zfs.misc.arcstats.hash_chain_max: 7 kstat.zfs.misc.arcstats.hash_chains: 219061 kstat.zfs.misc.arcstats.hash_collisions: 33107789 kstat.zfs.misc.arcstats.hash_elements_max: 1529284 kstat.zfs.misc.arcstats.hash_elements: 1529163 kstat.zfs.misc.arcstats.evict_l2_skip: 0 kstat.zfs.misc.arcstats.evict_l2_ineligible: 353901531136 kstat.zfs.misc.arcstats.evict_l2_eligible: 611148992512 kstat.zfs.misc.arcstats.evict_l2_cached: 471776311808 kstat.zfs.misc.arcstats.evict_not_enough: 2164 kstat.zfs.misc.arcstats.evict_skip: 232562 kstat.zfs.misc.arcstats.mutex_miss: 17547 kstat.zfs.misc.arcstats.deleted: 10350064 kstat.zfs.misc.arcstats.allocated: 172235521 kstat.zfs.misc.arcstats.mfu_ghost_hits: 8494679 kstat.zfs.misc.arcstats.mfu_hits: 1457647309 kstat.zfs.misc.arcstats.mru_ghost_hits: 5765227 kstat.zfs.misc.arcstats.mru_hits: 90829356 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 4657105 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 14515029 kstat.zfs.misc.arcstats.prefetch_data_misses: 6610395 kstat.zfs.misc.arcstats.prefetch_data_hits: 6837739 kstat.zfs.misc.arcstats.demand_metadata_misses: 127929204 kstat.zfs.misc.arcstats.demand_metadata_hits: 404655615 kstat.zfs.misc.arcstats.demand_data_misses: 2235496 kstat.zfs.misc.arcstats.demand_data_hits: 1138662563 kstat.zfs.misc.arcstats.misses: 141432200 kstat.zfs.misc.arcstats.hits: 1564670946 kstat.zfs.misc.zcompstats.skipped_insufficient_gain: 4581339 kstat.zfs.misc.zcompstats.empty: 842987 kstat.zfs.misc.zcompstats.attempts: 121463608 kstat.zfs.misc.zfetchstats.max_streams: 2029717049 kstat.zfs.misc.zfetchstats.misses: 2043239863 kstat.zfs.misc.zfetchstats.hits: 15544425 kstat.zfs.misc.xuio_stats.write_buf_nocopy: 1453761 kstat.zfs.misc.xuio_stats.write_buf_copied: 0 kstat.zfs.misc.xuio_stats.read_buf_nocopy: 0 kstat.zfs.misc.xuio_stats.read_buf_copied: 0 kstat.zfs.misc.xuio_stats.onloan_write_buf: 0 kstat.zfs.misc.xuio_stats.onloan_read_buf: 0 kstat.zfs.misc.zio_trim.failed: 0 kstat.zfs.misc.zio_trim.unsupported: 867 kstat.zfs.misc.zio_trim.success: 165333478 kstat.zfs.misc.zio_trim.bytes: 9734294003712 security.jail.param.allow.mount.zfs: 0 security.jail.mount_zfs_allowed: 0 I really don't know but could it be a conflict between the MB SATA ports and LSI HBA?? As upon startup there do seem to be some ATA Error messages in dmesg... So more of a physical HW issue then FS based? Or is it due to the term "bursty IO" that happens with ZFS... either way I have been looking at this for months trying to figure things out but other then a reboot nothing I do makes things better! Turning off my monitoring jail does help on occasion but outside of that I'm lost. I have another NAS based system with UFS root on SSD also yet ZPOOLs over the various large mechanical drives but never run into this particular issue! Would anyone be able to help??? Many thanks. Kaya