From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 17:13:49 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7D631961 for ; Tue, 15 Oct 2013 17:13:49 +0000 (UTC) (envelope-from jlh@FreeBSD.org) Received: from caravan.chchile.org (caravan.chchile.org [178.32.125.136]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E294B27FE for ; Tue, 15 Oct 2013 17:13:48 +0000 (UTC) Received: by caravan.chchile.org (Postfix, from userid 1000) id 27ACEC22A2; Tue, 15 Oct 2013 17:13:41 +0000 (UTC) Date: Tue, 15 Oct 2013 19:13:41 +0200 From: Jeremie Le Hen To: freebsd-fs@FreeBSD.org Subject: Re: ARC_SPACE_OTHER exceeds arc_max Message-ID: <20131015171340.GA30906@caravan.chchile.org> Mail-Followup-To: freebsd-fs@FreeBSD.org References: <20131011184206.GA60057@caravan.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131011184206.GA60057@caravan.chchile.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 17:13:49 -0000 On Fri, Oct 11, 2013 at 08:42:06PM +0200, Jeremie Le Hen wrote: > Hi, > > (Please Cc: me on reply, as I'm not subscribed.) > > On my FreeBSD 9.1 machine, roughly 2/3 of the times the daily scripts > are run, my ARC size is outgrowing like crazy the vfs.zfs.arc_max being > set to 536870912 (512 MB). > > The consquence of this is that userland processes are killed (!). OK > this box has no swap space and should have but still, this sounds really > crazy that a filesystem cache is able to "reclaim" memory for running > processes :). This has hit me again three times since this email. Any idea? > > I've used top -b every 30 seconds to get an idea of the system's memory, > logs below. Here is also a snippet of my sysctls related to zfs: > > vfs.zfs.l2arc_norw: 1 > vfs.zfs.l2arc_feed_again: 1 > vfs.zfs.l2arc_noprefetch: 1 > vfs.zfs.l2arc_feed_min_ms: 200 > vfs.zfs.l2arc_feed_secs: 1 > vfs.zfs.l2arc_headroom: 2 > vfs.zfs.l2arc_write_boost: 8388608 > vfs.zfs.l2arc_write_max: 8388608 > vfs.zfs.arc_meta_limit: 134217728 > vfs.zfs.arc_meta_used: 236528568 > vfs.zfs.arc_min: 67108864 > vfs.zfs.arc_max: 536870912 > debug.adaptive_machine_arch: 1 > hw.machine_arch: amd64 > kstat.zfs.misc.arcstats.hits: 77149964 > kstat.zfs.misc.arcstats.misses: 13690743 > kstat.zfs.misc.arcstats.demand_data_hits: 8073317 > kstat.zfs.misc.arcstats.demand_data_misses: 274103 > kstat.zfs.misc.arcstats.demand_metadata_hits: 69076639 > kstat.zfs.misc.arcstats.demand_metadata_misses: 13416617 > kstat.zfs.misc.arcstats.prefetch_data_hits: 0 > kstat.zfs.misc.arcstats.prefetch_data_misses: 0 > kstat.zfs.misc.arcstats.prefetch_metadata_hits: 8 > kstat.zfs.misc.arcstats.prefetch_metadata_misses: 23 > kstat.zfs.misc.arcstats.mru_hits: 33260238 > kstat.zfs.misc.arcstats.mru_ghost_hits: 2830869 > kstat.zfs.misc.arcstats.mfu_hits: 43889719 > kstat.zfs.misc.arcstats.mfu_ghost_hits: 3452884 > kstat.zfs.misc.arcstats.allocated: 14361097 > kstat.zfs.misc.arcstats.deleted: 7860017 > kstat.zfs.misc.arcstats.stolen: 6994054 > kstat.zfs.misc.arcstats.recycle_miss: 7051205 > kstat.zfs.misc.arcstats.mutex_miss: 4479 > kstat.zfs.misc.arcstats.evict_skip: 4052583 > kstat.zfs.misc.arcstats.evict_l2_cached: 0 > kstat.zfs.misc.arcstats.evict_l2_eligible: 129211340800 > kstat.zfs.misc.arcstats.evict_l2_ineligible: 14336 > kstat.zfs.misc.arcstats.hash_elements: 88387 > kstat.zfs.misc.arcstats.hash_elements_max: 133658 > kstat.zfs.misc.arcstats.hash_collisions: 14197499 > kstat.zfs.misc.arcstats.hash_chains: 16044 > kstat.zfs.misc.arcstats.hash_chain_max: 27 > kstat.zfs.misc.arcstats.p: 188161024 > kstat.zfs.misc.arcstats.c: 536870912 > kstat.zfs.misc.arcstats.c_min: 67108864 > kstat.zfs.misc.arcstats.c_max: 536870912 > kstat.zfs.misc.arcstats.size: 534264760 > kstat.zfs.misc.arcstats.hdr_size: 20228688 > kstat.zfs.misc.arcstats.data_size: 406874112 > kstat.zfs.misc.arcstats.other_size: 107161960 > kstat.zfs.misc.arcstats.l2_hits: 0 > kstat.zfs.misc.arcstats.l2_misses: 0 > kstat.zfs.misc.arcstats.l2_feeds: 0 > kstat.zfs.misc.arcstats.l2_rw_clash: 0 > kstat.zfs.misc.arcstats.l2_read_bytes: 0 > kstat.zfs.misc.arcstats.l2_write_bytes: 0 > kstat.zfs.misc.arcstats.l2_writes_sent: 0 > kstat.zfs.misc.arcstats.l2_writes_done: 0 > kstat.zfs.misc.arcstats.l2_writes_error: 0 > kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 > kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 > kstat.zfs.misc.arcstats.l2_evict_reading: 0 > kstat.zfs.misc.arcstats.l2_free_on_write: 0 > kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 > kstat.zfs.misc.arcstats.l2_cksum_bad: 0 > kstat.zfs.misc.arcstats.l2_io_error: 0 > kstat.zfs.misc.arcstats.l2_size: 0 > kstat.zfs.misc.arcstats.l2_hdr_size: 0 > kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 > kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 > kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 > kstat.zfs.misc.arcstats.l2_write_in_l2: 0 > kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 > kstat.zfs.misc.arcstats.l2_write_not_cacheable: 5 > kstat.zfs.misc.arcstats.l2_write_full: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 > kstat.zfs.misc.arcstats.l2_write_pios: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 > kstat.zfs.misc.arcstats.memory_throttle_count: 0 > kstat.zfs.misc.arcstats.duplicate_buffers: 0 > kstat.zfs.misc.arcstats.duplicate_buffers_size: 0 > kstat.zfs.misc.arcstats.duplicate_reads: 0 > > > > > > > > > ct 8 03:00:11 CEST 2013 > Mem: 97M Active, 112M Inact, 1258M Wired, 552K Cache, 482M Free > ARC: 310M Total, 42M MFU, 147M MRU, 912K Anon, 20M Header, 101M Other > > Tue Oct 8 03:00:41 CEST 2013 > Mem: 97M Active, 112M Inact, 1258M Wired, 552K Cache, 482M Free > ARC: 310M Total, 42M MFU, 147M MRU, 912K Anon, 20M Header, 101M Other > > Tue Oct 8 03:01:11 CEST 2013 > Mem: 105M Active, 114M Inact, 1274M Wired, 552K Cache, 457M Free > ARC: 330M Total, 42M MFU, 159M MRU, 2241K Anon, 20M Header, 107M Other > > Tue Oct 8 03:01:41 CEST 2013 > Mem: 105M Active, 115M Inact, 1241M Wired, 552K Cache, 490M Free > ARC: 332M Total, 32M MFU, 137M MRU, 912K Anon, 21M Header, 141M Other > > Tue Oct 8 03:02:11 CEST 2013 > Mem: 105M Active, 121M Inact, 1251M Wired, 552K Cache, 473M Free > ARC: 326M Total, 34M MFU, 145M MRU, 912K Anon, 22M Header, 124M Other > > Tue Oct 8 03:02:42 CEST 2013 > Mem: 105M Active, 121M Inact, 1247M Wired, 552K Cache, 478M Free > ARC: 350M Total, 32M MFU, 143M MRU, 1265K Anon, 23M Header, 150M Other > > Tue Oct 8 03:03:12 CEST 2013 > Mem: 104M Active, 121M Inact, 1232M Wired, 552K Cache, 493M Free > ARC: 293M Total, 35M MFU, 126M MRU, 928K Anon, 24M Header, 107M Other > > Tue Oct 8 03:03:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1253M Wired, 552K Cache, 475M Free > ARC: 367M Total, 37M MFU, 145M MRU, 928K Anon, 24M Header, 160M Other > > Tue Oct 8 03:04:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1284M Wired, 552K Cache, 443M Free > ARC: 316M Total, 55M MFU, 157M MRU, 912K Anon, 24M Header, 78M Other > > Tue Oct 8 03:04:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1379M Wired, 552K Cache, 348M Free > ARC: 413M Total, 84M MFU, 223M MRU, 1743K Anon, 24M Header, 81M Other > > Tue Oct 8 03:05:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 484M Total, 106M MFU, 274M MRU, 928K Anon, 23M Header, 80M Other > > Tue Oct 8 03:05:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 494M Total, 119M MFU, 261M MRU, 930K Anon, 23M Header, 90M Other > > Tue Oct 8 03:06:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 501M Total, 151M MFU, 228M MRU, 1226K Anon, 24M Header, 96M Other > > Tue Oct 8 03:06:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1450M Wired, 552K Cache, 277M Free > ARC: 502M Total, 191M MFU, 188M MRU, 944K Anon, 24M Header, 99M Other > > Tue Oct 8 03:07:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 507M Total, 226M MFU, 153M MRU, 1094K Anon, 24M Header, 103M Other > > Tue Oct 8 03:07:42 CEST 2013 > Mem: 104M Active, 118M Inact, 1407M Wired, 552K Cache, 321M Free > ARC: 479M Total, 175M MFU, 161M MRU, 915K Anon, 25M Header, 117M Other > > Tue Oct 8 03:08:13 CEST 2013 > Mem: 94M Active, 116M Inact, 1387M Wired, 552K Cache, 353M Free > ARC: 506M Total, 90M MFU, 231M MRU, 912K Anon, 26M Header, 158M Other > > Tue Oct 8 03:08:43 CEST 2013 > Mem: 96M Active, 117M Inact, 1351M Wired, 552K Cache, 386M Free > ARC: 436M Total, 52M MFU, 231M MRU, 1322K Anon, 23M Header, 129M Other > > Tue Oct 8 03:09:13 CEST 2013 > Mem: 95M Active, 117M Inact, 1314M Wired, 552K Cache, 423M Free > ARC: 440M Total, 73M MFU, 173M MRU, 941K Anon, 23M Header, 170M Other > > Tue Oct 8 03:09:43 CEST 2013 > Mem: 94M Active, 117M Inact, 1327M Wired, 552K Cache, 412M Free > ARC: 510M Total, 80M MFU, 182M MRU, 912K Anon, 23M Header, 223M Other > > Tue Oct 8 03:10:13 CEST 2013 > Mem: 94M Active, 117M Inact, 1276M Wired, 552K Cache, 463M Free > ARC: 578M Total, 65M MFU, 150M MRU, 912K Anon, 27M Header, 336M Other > > Tue Oct 8 03:10:43 CEST 2013 > Mem: 95M Active, 117M Inact, 1262M Wired, 552K Cache, 476M Free > ARC: 585M Total, 56M MFU, 144M MRU, 913K Anon, 26M Header, 358M Other > > Tue Oct 8 03:11:13 CEST 2013 > Mem: 103M Active, 114M Inact, 1269M Wired, 552K Cache, 464M Free > ARC: 590M Total, 66M MFU, 140M MRU, 912K Anon, 26M Header, 357M Other > > Tue Oct 8 03:11:44 CEST 2013 > Mem: 103M Active, 114M Inact, 1294M Wired, 552K Cache, 439M Free > ARC: 667M Total, 70M MFU, 155M MRU, 1056K Anon, 27M Header, 415M Other > > Tue Oct 8 03:12:14 CEST 2013 > Mem: 103M Active, 114M Inact, 1389M Wired, 552K Cache, 343M Free > ARC: 792M Total, 73M MFU, 185M MRU, 1040K Anon, 27M Header, 507M Other > > Tue Oct 8 03:12:44 CEST 2013 > Mem: 94M Active, 119M Inact, 1472M Wired, 552K Cache, 265M Free > ARC: 891M Total, 82M MFU, 212M MRU, 1056K Anon, 27M Header, 568M Other > > Tue Oct 8 03:13:15 CEST 2013 > Mem: 94M Active, 119M Inact, 1538M Wired, 544K Cache, 199M Free > ARC: 951M Total, 85M MFU, 220M MRU, 928K Anon, 28M Header, 618M Other > > Tue Oct 8 03:13:45 CEST 2013 > Mem: 136M Active, 22M Inact, 1682M Wired, 46M Cache, 65M Free > ARC: 1113M Total, 90M MFU, 259M MRU, 912K Anon, 26M Header, 738M Other > > Tue Oct 8 03:14:15 CEST 2013 > Mem: 153M Active, 3936K Inact, 1736M Wired, 36M Cache, 21M Free > ARC: 1159M Total, 95M MFU, 271M MRU, 1040K Anon, 25M Header, 767M Other > > Tue Oct 8 03:14:46 CEST 2013 > Mem: 62M Active, 15M Inact, 1808M Wired, 35M Cache, 30M Free > ARC: 1213M Total, 80M MFU, 294M MRU, 819K Anon, 19M Header, 819M Other > > Tue Oct 8 03:15:17 CEST 2013 > Mem: 61M Active, 6488K Inact, 1816M Wired, 33M Cache, 34M Free > ARC: 1194M Total, 73M MFU, 293M MRU, 1040K Anon, 19M Header, 808M Other > > Tue Oct 8 03:15:47 CEST 2013 > Mem: 75M Active, 2548K Inact, 1808M Wired, 25M Cache, 41M Free > ARC: 1189M Total, 72M MFU, 292M MRU, 1475K Anon, 19M Header, 804M Other > > Tue Oct 8 03:16:17 CEST 2013 > Mem: 75M Active, 2928K Inact, 1806M Wired, 24M Cache, 43M Free > ARC: 1183M Total, 72M MFU, 291M MRU, 912K Anon, 18M Header, 801M Other > > Tue Oct 8 03:16:47 CEST 2013 > Mem: 64M Active, 14M Inact, 1805M Wired, 24M Cache, 44M Free > ARC: 1179M Total, 71M MFU, 290M MRU, 912K Anon, 18M Header, 798M Other > > Tue Oct 8 03:17:17 CEST 2013 > Mem: 21M Active, 57M Inact, 1802M Wired, 24M Cache, 47M Free > ARC: 1174M Total, 71M MFU, 289M MRU, 912K Anon, 18M Header, 795M Other > > Tue Oct 8 03:17:48 CEST 2013 > Mem: 16M Active, 61M Inact, 1801M Wired, 24M Cache, 49M Free > ARC: 1170M Total, 70M MFU, 288M MRU, 912K Anon, 18M Header, 792M Other > > Tue Oct 8 03:18:18 CEST 2013 > Mem: 16M Active, 61M Inact, 1800M Wired, 24M Cache, 50M Free > ARC: 1165M Total, 70M MFU, 287M MRU, 912K Anon, 18M Header, 789M Other > > Tue Oct 8 03:18:48 CEST 2013 > Mem: 17M Active, 62M Inact, 1799M Wired, 23M Cache, 50M Free > ARC: 1162M Total, 70M MFU, 286M MRU, 912K Anon, 18M Header, 787M Other > > Tue Oct 8 03:19:18 CEST 2013 > Mem: 14M Active, 57M Inact, 1797M Wired, 23M Cache, 60M Free > ARC: 1157M Total, 69M MFU, 285M MRU, 912K Anon, 18M Header, 784M Other > > Tue Oct 8 03:19:48 CEST 2013 > Mem: 14M Active, 57M Inact, 1796M Wired, 23M Cache, 61M Free > ARC: 1153M Total, 69M MFU, 284M MRU, 912K Anon, 18M Header, 781M Other > > Tue Oct 8 03:20:18 CEST 2013 > Mem: 15M Active, 56M Inact, 1794M Wired, 22M Cache, 63M Free > ARC: 1148M Total, 69M MFU, 283M MRU, 912K Anon, 18M Header, 777M Other > > Tue Oct 8 03:20:48 CEST 2013 > Mem: 14M Active, 56M Inact, 1793M Wired, 22M Cache, 65M Free > ARC: 1145M Total, 69M MFU, 282M MRU, 912K Anon, 18M Header, 775M Other > > > -- > Jeremie Le Hen > > Scientists say the world is made up of Protons, Neutrons and Electrons. > They forgot to mention Morons. > -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons.