Date: Fri, 14 Mar 2014 21:11:40 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Two odd problems with STABLE-10 r262921 [ZFS ARC memory allocation / "freeze" problems -- RESOLVED] Message-ID: <5323B6DC.7020205@denninger.net> In-Reply-To: <531F14A7.9090103@denninger.net> References: <CAHCMRk_=s%2B2LYr-pLkt7LJK3LcWSiomtLb_HhfUrj4VMUHjQVQ@mail.gmail.com> <531F14A7.9090103@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 3/11/2014 8:50 AM, Karl Denninger wrote: > > The other issue is that I am running into bizarre wired memory behavior. I > have ZFS on the box and it was wiring down ~20GB, driving the system into > paging. I dropped arc_max to 16GB as a hard limit, it's currently showing > 15GB, but... > > 20531152 wire > > zfs-stats -A shows: > > > ------------------------------------------------------------------------ > ZFS Subsystem Report Tue Mar 11 08:27:55 2014 > ------------------------------------------------------------------------ > > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > > ARC Misc: > Deleted: 4.22m > Recycle Misses: 57.10k > Mutex Misses: 320 > Evict Skips: 2.35m > > ARC Size: 95.96% 15.35 GiB > Target Size: (Adaptive) 100.00% 16.00 GiB > Min Size (Hard Limit): 12.50% 2.00 GiB > Max Size (High Water): 8:1 16.00 GiB > > ARC Size Breakdown: > Recently Used Cache Size: 75.49% 12.08 GiB > Frequently Used Cache Size: 24.51% 3.92 GiB > > ARC Hash Breakdown: > Elements Max: 1.59m > Elements Current: 55.53% 883.80k > Collisions: 5.84m > Chain Max: 16 > Chains: 261.48k > > ------------------------------------------------------------------------ > > Ok, where's the rest coming from? There's nothing in the shared memory > segment list that appears to be related. > > Is there a memory leak in here somewhere? As posted to the fs mailing list I have found and corrected this issue; the ARC eviction code has issues. Please pardon the apparently-reversed patch I submitted; it's Friday, I didn't have enough coffee and Murphy got me twice while putting that together (the first time when I pulled the diff off without committing the last change I had made back to my local tree first!) If you're having issues with your system stalling while running ZFS filesystems and it appears you're being driven into paging when you shouldn't be (in other words, the ARC cache is consuming all of your RAM and not releasing it as other things demand memory) this should fix it. It also includes a run-time tunable via sysctl (defaults to 25%, which was the original intent of the code that I replaced) for the percentage of memory to target as free, below which the ARC will be asked to release RAM. See http://www.freebsd.org/cgi/query-pr.cgi?pr=187594 -- -- Karl karl@denninger.net [-- Attachment #2 --] 0 *H 01 0 + 0 *H O0K030 *H 01 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 130824190344Z 180823190344Z0[1 0 UUS10UFlorida10UKarl Denninger1!0 *H karl@denninger.net0"0 *H 0 bi՞] MNԿawx?`)'ҴcWgR@BlWh+ u}ApdCF JVй~FOL }EW^bچYp3K& ׂ(R lxڝ.xz?6 &nsJ +1v9 v/( kqĪp[vjcK%fϻe?iq]z lyzFO'ppdX//Lw(3 JI A*S#՟H[f|CGqJKoo y.oEuOw$ /섀$삻J9b|A P~8]D1YI<"""Y^T2iQ2 b yH)] Ƶ0y$ _N 6XqMC 9 XgώjGTP"#nˋ"Bk1 00 U 0 0 `HB0 U 0, `HB OpenSSL Generated Certificate0 U |8 ˴d[20 U #0]Af4 U3x&^"408 `HB+)https://cudasystems.net:11443/revoked.crl0 *H gBwH] j\x`( &gW32"Uf^. ^Iϱ k!DQA g{(w /)\N'[ oRW @C HO>)XrTNɘ!u`xt5(=f\-l3<@C6mnhv##1ŃbH͍_Nq aʷ?rk$^9TIa!kh,D -ct1 0001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 + ;0 *H 1 *H 0 *H 1 140315021140Z0# *H 1B8٨K݈v HFR0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H 101 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H >4hCEz|B_(`Z! ٬˺hu e<)ѕXdVNZp3<괤4ke܊1Jt*4CO(!dF ak?]\Q'X <Ke`T-1ڤng >-y=EZW{o:_aK ~ 9qu(^st:hMmz. Bҋ/L\d02q$7n: ˑ M Yf4aKkL~qVZ &cUڵT p] 8^9dwP$vCfxMNIL$H9ʣ'vnr1C\zjXA Rܺ jhE+\3 /sh4 qOrmEg%XD;bH_[ Tr,"_7% R; s {6< Iam aBn5g)^}\)iǁQ
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5323B6DC.7020205>