From owner-freebsd-stable@FreeBSD.ORG Sun Jan 24 11:32:22 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9C721065672 for ; Sun, 24 Jan 2010 11:32:22 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 64CE08FC1C for ; Sun, 24 Jan 2010 11:32:22 +0000 (UTC) Received: from elsa.codelab.cz (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id C47F319E019; Sun, 24 Jan 2010 12:32:20 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2375F19E023; Sun, 24 Jan 2010 12:32:18 +0100 (CET) Message-ID: <4B5C2FC1.9070001@quip.cz> Date: Sun, 24 Jan 2010 12:32:17 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.7) Gecko/20100104 SeaMonkey/2.0.2 MIME-Version: 1.0 To: Garrett Moore References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <7346c5c61001091706m45a3a2a5k3ca8bb0c4bec5ea8@mail.gmail.com> <7346c5c61001171521w1ca4738w98e8fcca24643cda@mail.gmail.com> <201001180829.48126.npapke@acm.org> <7346c5c61001190840k31466754i32b2ae833390b79b@mail.gmail.com> In-Reply-To: <7346c5c61001190840k31466754i32b2ae833390b79b@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: ZFS performance degradation over time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2010 11:32:23 -0000 Garrett Moore wrote: > I've been watching my memory usage and I have no idea what is consuming > memory as 'Active'. > > Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB > buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was > again slow (<1MB/s). I tried killing rTorrent and no significant amount of > memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any > significant amount of memory, and I was definitely nowhere near 6500MB > usage. > > I tried running the perl oneliner again to hog a bunch of memory, and almost > all of the Active memory was IMMEDIATELY marked as Free, and my performance > was excellent again. > > I'm not sure what in userland could be causing the issue. The only things > I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash, > Python, Perl, and SABNZBd. There is nothing that *should* be consuming any > serious amount of memory. Last night I tried ZFS with pool on iSCSI connected Dell MD3000i and I was suprised by too low speed of simple cp -a command (copying from UFS partition to ZFS) The write speed was about 2MB/s only. After looking in to ARC stuff, I realized some weird values: ARC Size: Current Size: 1 MB (arcsize) Target Size (Adaptive): 205 MB (c) Min Size (Hard Limit): 205 MB (zfs_arc_min) Max Size (Hard Limit): 1647 MB (zfs_arc_max) (stats from script http://cuddletech.com/arc_summary/ freebsd version http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/ ) I don't know why it shows Current Size 1MB. I tried the perl oneliner from this thread. Then I got about 5GB free merory and Target Size and Current Size growed to 1647MB. Write speed increase to about 8MB/s and after few minuts slowly dropped to 2MB/s and ARC Current Size dropped to 1MB again. This server is not in production and was idle. Just copying the data from one partition to another. Today I tried serving the data by Lighttpd. There is impressive iSCSI read performance - because of ZFS prefetch, it can achieve 880Mbits of read from iSCSI, but serving by Lighttpd only about 66Mbits bce0 - internet bce1 - iSCSI to storage MD3000i bce0 bce1 Kbps in Kbps out Kbps in Kbps out 2423.22 65481.56 855970.7 4348.73 2355.26 63911.74 820561.3 4846.08 2424.87 65998.62 848937.1 4312.37 2442.78 66544.95 858019.0 4356.64 iostat -x extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da1 1596.8 3.6 102196.7 22.2 13 7.4 97 da1 1650.2 2.9 105612.7 55.7 16 7.4 103 da1 1647.3 0.0 105422.9 0.0 13 7.2 100 da1 1636.5 2.3 104735.4 20.0 16 7.3 100 da1 1642.9 0.0 105141.1 0.0 13 7.3 100 ~/bin/arcstat.pl -f Time,read,hits,Hit%,miss,miss%,dmis,dm%,mmis,mm%,arcsz,c 30 Time read hits Hit% miss miss% dmis dm% mmis mm% arcsz c 12:18:05 16K 15K 94 838 5 570 3 1 0 16933296 215902720 12:18:36 16K 15K 94 839 5 571 3 0 0 21488288 215902720 12:19:06 16K 15K 94 836 5 569 3 1 0 17228688 215902720 12:19:37 16K 15K 94 839 5 572 3 4 1 22002672 215902720 12:20:07 16K 15K 94 841 5 570 3 1 0 27784960 215902720 12:20:38 16K 15K 94 838 5 569 3 0 0 21839472 215902720 12:21:08 16K 15K 94 837 5 568 3 0 0 28244992 215902720 12:21:39 16K 15K 94 833 5 565 3 1 0 28744416 215902720 12:22:09 16K 15K 94 842 5 576 3 4 1 28646656 215902720 12:22:39 16K 15K 94 840 5 575 3 3 0 28903696 215902720 12:23:10 15K 15K 94 821 5 561 3 0 0 28765904 215902720 12:23:40 16K 15K 94 828 5 566 3 0 0 28395840 215902720 12:24:11 16K 15K 94 828 5 568 3 0 0 32063408 215902720 12:24:41 16K 15K 94 834 5 570 3 0 0 29800976 215902720 12:25:12 15K 15K 94 820 5 562 3 1 0 29066512 215902720 # ~/bin/arc_summary.pl System Memory: Physical RAM: 8169 MB Free Memory : 0 MB ARC Size: Current Size: 22 MB (arcsize) Target Size (Adaptive): 205 MB (c) Min Size (Hard Limit): 205 MB (zfs_arc_min) Max Size (Hard Limit): 1647 MB (zfs_arc_max) ARC Size Breakdown: Most Recently Used Cache Size: 5% 11 MB (p) Most Frequently Used Cache Size: 94% 194 MB (c-p) ARC Efficency: Cache Access Total: 81843958 Cache Hit Ratio: 95% 78525502 [Defined State for buffer] Cache Miss Ratio: 4% 3318456 [Undefined State for Buffer] REAL Hit Ratio: 95% 78418994 [MRU/MFU Hits Only] Data Demand Efficiency: 97% Data Prefetch Efficiency: 10% CACHE HITS BY CACHE LIST: Anon: --% Counter Rolled. Most Recently Used: 2% 2209869 (mru) [ Return Customer ] Most Frequently Used: 97% 76209125 (mfu) [ Frequent Customer ] Most Recently Used Ghost: 1% 965711 (mru_ghost) [ Return Customer Evicted, Now Back ] Most Frequently Used Ghost: 0% 176871 (mfu_ghost) [ Frequent Customer Evicted, Now Back ] CACHE HITS BY DATA TYPE: Demand Data: 97% 76770304 Prefetch Data: 0% 126644 Demand Metadata: 2% 1628528 Prefetch Metadata: 0% 26 CACHE MISSES BY DATA TYPE: Demand Data: 63% 2122089 Prefetch Data: 32% 1063449 Demand Metadata: 4% 132894 Prefetch Metadata: 0% 24 --------------------------------------------- # sysctl kstat.zfs.misc.arcstats kstat.zfs.misc.arcstats.hits: 75409326 kstat.zfs.misc.arcstats.misses: 3144748 kstat.zfs.misc.arcstats.demand_data_hits: 73731356 kstat.zfs.misc.arcstats.demand_data_misses: 2003526 kstat.zfs.misc.arcstats.demand_metadata_hits: 1551917 kstat.zfs.misc.arcstats.demand_metadata_misses: 132730 kstat.zfs.misc.arcstats.prefetch_data_hits: 126027 kstat.zfs.misc.arcstats.prefetch_data_misses: 1008468 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 26 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 24 kstat.zfs.misc.arcstats.mru_hits: 2105758 kstat.zfs.misc.arcstats.mru_ghost_hits: 914887 kstat.zfs.misc.arcstats.mfu_hits: 73197609 kstat.zfs.misc.arcstats.mfu_ghost_hits: 171171 kstat.zfs.misc.arcstats.deleted: 2367973 kstat.zfs.misc.arcstats.recycle_miss: 412788 kstat.zfs.misc.arcstats.mutex_miss: 2865 kstat.zfs.misc.arcstats.evict_skip: 17459 kstat.zfs.misc.arcstats.hash_elements: 2478 kstat.zfs.misc.arcstats.hash_elements_max: 28921 kstat.zfs.misc.arcstats.hash_collisions: 86135 kstat.zfs.misc.arcstats.hash_chains: 25 kstat.zfs.misc.arcstats.hash_chain_max: 3 kstat.zfs.misc.arcstats.p: 14908416 kstat.zfs.misc.arcstats.c: 215902720 kstat.zfs.misc.arcstats.c_min: 215902720 kstat.zfs.misc.arcstats.c_max: 1727221760 kstat.zfs.misc.arcstats.size: 30430560 kstat.zfs.misc.arcstats.hdr_size: 555072 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 135489 This is on FreeBSD 7.2-STABLE #0: Sun Dec 6 23:21:17 CET 2009 root@dust.hrej.cz:/usr/obj/usr/src/sys/GENERIC amd64 Can somebody tell me, why ARC Current Size is dropping too low? (1-20MB if arc_min is 205MB) The system have 8GB of memory and 8 CPU cores: last pid: 83605; load averages: 0.17, 0.15, 0.10 up 36+10:34:34 12:29:05 58 processes: 1 running, 56 sleeping, 1 zombie CPU: 0.1% user, 0.0% nice, 2.3% system, 1.7% interrupt, 95.8% idle Mem: 237M Active, 6259M Inact, 1154M Wired, 138M Cache, 827M Buf, 117M Free Swap: 8192M Total, 96K Used, 8192M Free I have no loader.conf tunning on this machine. Miroslav Lachman