From owner-freebsd-fs@FreeBSD.ORG Wed Jan 30 15:19:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8559B308 for ; Wed, 30 Jan 2013 15:19:59 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ie0-x22c.google.com (ie-in-x022c.1e100.net [IPv6:2607:f8b0:4001:c03::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 515C7FDA for ; Wed, 30 Jan 2013 15:19:59 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id c10so1369847ieb.3 for ; Wed, 30 Jan 2013 07:19:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=r8f3Yu1HhqeZ9wHXx0xyu9zShuD73vpJQwFBGe3oISw=; b=qUbIKur38wWMrpJww/gOOCIYLFt90vPmxsMEGUiEsZPXgDgEuytOkpc3a7oGj3bnhM 1/hP4ihpYbVWlSAMv5Y5q5CiG+Xj+HbL5zVmEM+WmqAxVO22TuxlPkxz2boseY6fjr5I 9QHeX464fAAbOcFCpiWySrA/03PMOICKl5Uek= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=r8f3Yu1HhqeZ9wHXx0xyu9zShuD73vpJQwFBGe3oISw=; b=PavDKmiLf/Sn1qOmSrsbmGHPgeoQS5LROkHLQLEEjBaGTAxDahWGqnkF6JvnkAVgGI S7A/gdGsI00skwd2VmVjZapgNlTywx5JdduWCrDWqcBtLzZJ/G+gkXhWhoU3iwb+JHUH H5NbctKDY+9xSqer5YcyiL3jVuHfHK3Z46NalGaZ6+T/op10FLh5hiHesi0bSen2z+Jv HaMmepWyojXOeiqWLXbz11gfqwfutPI8JIrcCIlwrWJhygizB9RON3NgI//Trq65MUOy +DxoEr0Yh6ZsfNIqUnmU9hdjjVWg7GLBBZSR9AAcQbx9kg5IAy9DcJ8k8Z+kLBcNJJMp 3e+w== X-Received: by 10.50.13.208 with SMTP id j16mr3837750igc.73.1359559199008; Wed, 30 Jan 2013 07:19:59 -0800 (PST) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id fb10sm2077564igb.1.2013.01.30.07.19.55 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 30 Jan 2013 07:19:57 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Improving ZFS performance for large directories From: Kevin Day In-Reply-To: <5267B97C-ED47-4AAB-8415-12D6987E9371@gmail.com> Date: Wed, 30 Jan 2013 09:19:52 -0600 Content-Transfer-Encoding: 7bit Message-Id: <47975CEB-EA50-4F6C-8C47-6F32312F34C4@dragondata.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <5267B97C-ED47-4AAB-8415-12D6987E9371@gmail.com> To: Nikolay Denev X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQkYnbcKjzrRuMXTbOa1TnXlK7LV6CYRTKJE5zTGGBvs7cMBaMH2lTMQXKldb04O4ZSDB54d Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2013 15:19:59 -0000 On Jan 30, 2013, at 4:36 AM, Nikolay Denev wrote: > > > What are your : vfs.zfs.arc_meta_limit and vfs.zfs.arc_meta_used sysctls? > Maybe increasing the limit can help? vfs.zfs.arc_meta_limit: 8199079936 vfs.zfs.arc_meta_used: 13965744408 Full output of zfs-stats: ------------------------------------------------------------------------ ZFS Subsystem Report Wed Jan 30 15:16:54 2013 ------------------------------------------------------------------------ System Information: Kernel Version: 901000 (osreldate) Hardware Platform: amd64 Processor Architecture: amd64 ZFS Storage pool Version: 28 ZFS Filesystem Version: 5 FreeBSD 9.1-RC2 #1: Tue Oct 30 20:37:38 UTC 2012 root 3:16PM up 19 days, 19:44, 2 users, load averages: 0.91, 0.80, 0.68 ------------------------------------------------------------------------ System Memory: 12.44% 7.72 GiB Active, 6.04% 3.75 GiB Inact 77.33% 48.01 GiB Wired, 2.25% 1.40 GiB Cache 1.94% 1.21 GiB Free, 0.00% 1.21 MiB Gap Real Installed: 64.00 GiB Real Available: 99.97% 63.98 GiB Real Managed: 97.04% 62.08 GiB Logical Total: 64.00 GiB Logical Used: 90.07% 57.65 GiB Logical Free: 9.93% 6.35 GiB Kernel Memory: 22.62 GiB Data: 99.91% 22.60 GiB Text: 0.09% 21.27 MiB Kernel Memory Map: 54.28 GiB Size: 34.75% 18.86 GiB Free: 65.25% 35.42 GiB ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 430.91m Recycle Misses: 111.27m Mutex Misses: 2.49m Evict Skips: 647.25m ARC Size: 87.63% 26.77 GiB Target Size: (Adaptive) 87.64% 26.77 GiB Min Size (Hard Limit): 12.50% 3.82 GiB Max Size (High Water): 8:1 30.54 GiB ARC Size Breakdown: Recently Used Cache Size: 58.64% 15.70 GiB Frequently Used Cache Size: 41.36% 11.07 GiB ARC Hash Breakdown: Elements Max: 2.19m Elements Current: 86.15% 1.89m Collisions: 344.47m Chain Max: 17 Chains: 552.47k ------------------------------------------------------------------------ ARC Efficiency: 21.94b Cache Hit Ratio: 97.00% 21.28b Cache Miss Ratio: 3.00% 657.23m Actual Hit Ratio: 73.15% 16.05b Data Demand Efficiency: 98.94% 1.32b Data Prefetch Efficiency: 14.83% 299.44m CACHE HITS BY CACHE LIST: Anonymously Used: 23.03% 4.90b Most Recently Used: 6.12% 1.30b Most Frequently Used: 69.29% 14.75b Most Recently Used Ghost: 0.50% 105.94m Most Frequently Used Ghost: 1.07% 226.92m CACHE HITS BY DATA TYPE: Demand Data: 6.11% 1.30b Prefetch Data: 0.21% 44.42m Demand Metadata: 69.29% 14.75b Prefetch Metadata: 24.38% 5.19b CACHE MISSES BY DATA TYPE: Demand Data: 2.12% 13.90m Prefetch Data: 38.80% 255.02m Demand Metadata: 30.97% 203.56m Prefetch Metadata: 28.11% 184.75m ------------------------------------------------------------------------ L2ARC is disabled ------------------------------------------------------------------------ File-Level Prefetch: (HEALTHY) DMU Efficiency: 24.08b Hit Ratio: 66.02% 15.90b Miss Ratio: 33.98% 8.18b Colinear: 8.18b Hit Ratio: 0.01% 560.82k Miss Ratio: 99.99% 8.18b Stride: 15.23b Hit Ratio: 99.98% 15.23b Miss Ratio: 0.02% 2.62m DMU Misc: Reclaim: 8.18b Successes: 0.08% 6.31m Failures: 99.92% 8.17b Streams: 663.44m +Resets: 0.06% 397.18k -Resets: 99.94% 663.04m Bogus: 0 ------------------------------------------------------------------------ VDEV cache is disabled ------------------------------------------------------------------------ ZFS Tunables (sysctl): kern.maxusers 384 vm.kmem_size 66662760448 vm.kmem_size_scale 1 vm.kmem_size_min 0 vm.kmem_size_max 329853485875 vfs.zfs.l2c_only_size 0 vfs.zfs.mfu_ghost_data_lsize 2121007104 vfs.zfs.mfu_ghost_metadata_lsize 7876605440 vfs.zfs.mfu_ghost_size 9997612544 vfs.zfs.mfu_data_lsize 10160539648 vfs.zfs.mfu_metadata_lsize 17161216 vfs.zfs.mfu_size 11163991040 vfs.zfs.mru_ghost_data_lsize 7235079680 vfs.zfs.mru_ghost_metadata_lsize 11107812352 vfs.zfs.mru_ghost_size 18342892032 vfs.zfs.mru_data_lsize 4406255616 vfs.zfs.mru_metadata_lsize 3924364288 vfs.zfs.mru_size 8893582336 vfs.zfs.anon_data_lsize 0 vfs.zfs.anon_metadata_lsize 0 vfs.zfs.anon_size 999424 vfs.zfs.l2arc_norw 1 vfs.zfs.l2arc_feed_again 1 vfs.zfs.l2arc_noprefetch 1 vfs.zfs.l2arc_feed_min_ms 200 vfs.zfs.l2arc_feed_secs 1 vfs.zfs.l2arc_headroom 2 vfs.zfs.l2arc_write_boost 8388608 vfs.zfs.l2arc_write_max 8388608 vfs.zfs.arc_meta_limit 8199079936 vfs.zfs.arc_meta_used 14161977912 vfs.zfs.arc_min 4099539968 vfs.zfs.arc_max 32796319744 vfs.zfs.dedup.prefetch 1 vfs.zfs.mdcomp_disable 0 vfs.zfs.write_limit_override 0 vfs.zfs.write_limit_inflated 206088929280 vfs.zfs.write_limit_max 8587038720 vfs.zfs.write_limit_min 33554432 vfs.zfs.write_limit_shift 3 vfs.zfs.no_write_throttle 0 vfs.zfs.zfetch.array_rd_sz 1048576 vfs.zfs.zfetch.block_cap 256 vfs.zfs.zfetch.min_sec_reap 2 vfs.zfs.zfetch.max_streams 8 vfs.zfs.prefetch_disable 0 vfs.zfs.mg_alloc_failures 12 vfs.zfs.check_hostid 1 vfs.zfs.recover 0 vfs.zfs.txg.synctime_ms 1000 vfs.zfs.txg.timeout 5 vfs.zfs.vdev.cache.bshift 16 vfs.zfs.vdev.cache.size 0 vfs.zfs.vdev.cache.max 16384 vfs.zfs.vdev.write_gap_limit 4096 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.aggregation_limit 131072 vfs.zfs.vdev.ramp_rate 2 vfs.zfs.vdev.time_shift 6 vfs.zfs.vdev.min_pending 4 vfs.zfs.vdev.max_pending 10 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.cache_flush_disable 0 vfs.zfs.zil_replay_disable 0 vfs.zfs.zio.use_uma 0 vfs.zfs.snapshot_list_prefetch 0 vfs.zfs.version.zpl 5 vfs.zfs.version.spa 28 vfs.zfs.version.acl 1 vfs.zfs.debug 0 vfs.zfs.super_owner 0 ------------------------------------------------------------------------