Date: Fri, 28 Feb 2014 22:04:41 +0200 From: Anton Sayetsky <vsjcfm@gmail.com> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Fwd: ZFS and Wired memory, again Message-ID: <CAFG2KCLJpcS_eSHHh0%2BRqzO0gAsMSAtnaN%2BHS0xdtvFmhUtUYA@mail.gmail.com> In-Reply-To: <CAFG2KC%2BZSHEVFbpPD9e1QHRdY=Sd6EuAD80vyDLDDQcpgCQNhA@mail.gmail.com> References: <CAFG2KC%2BZSHEVFbpPD9e1QHRdY=Sd6EuAD80vyDLDDQcpgCQNhA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] 2014-02-28 21:58 GMT+02:00 Mark Felder <feld@freebsd.org>: > > > On Fri, Feb 28, 2014, at 13:43, Adam Vande More wrote: >> On Fri, Feb 28, 2014 at 1:35 PM, Mark Felder <feld@freebsd.org> wrote: >> >> > >> > > >> > > >> > > Without question, cron could do it. >> > > >> > >> > I can't see cron using kernel memory; that just doesn't make sense to >> > me. Not even the periodic scripts that cron executes should be able to >> > balloon kernel like that. >> > >> > I think I know what meant to infer though -- that some nonstandard cron >> > script is doing something ugly. >> >> >> He's running 150 TB on 3 GB of mem. Periodic I think could consume that >> alone. >> > > Which periodic script are you referring to? The daily security check? I > suppose I could see resource usage increase after checking for new > setuid bits... assuming he had a lot of files to crawl through. But I > don't actually know if that would do much to wired memory. Ok, I'll repost my first message. I repeat: this in only _test_ system, with _single disk_, with _only_ system, ports & src on ZFS pool. No services are running except those which are in base system. ---------- Forwarded message ---------- From: Anton Sayetsky <vsjcfm@gmail.com> Date: 2013-11-22 21:53 GMT+02:00 Subject: ZFS and Wired memory, again To: freebsd-fs@freebsd.org Hello, I'm planning to deploy a ~150 TiB ZFS pool and when playing with ZFS noticed that amount of wired memory is MUCH bigger than ARC size (in absence of other hungry memory consumers, of course). I'm afraid that this strange behavior may become even worse on a machine with big pool and some hundreds gibibytes of RAM. So let me explain what happened. Immediately after booting system top says the following: ===== Mem: 14M Active, 13M Inact, 117M Wired, 2947M Free ARC: 24M Total, 5360K MFU, 18M MRU, 16K Anon, 328K Header, 1096K Other ===== Ok, wired mem - arc = 92 MiB Then I started to read pool (tar cpf /dev/null /). Memory usage when ARC size is ~1GiB ===== Mem: 16M Active, 15M Inact, 1410M Wired, 1649M Free ARC: 1114M Total, 29M MFU, 972M MRU, 21K Anon, 18M Header, 95M Other ===== 1410-1114=296 MiB Memory usage when ARC size reaches it's maximum of 2 GiB ===== Mem: 16M Active, 16M Inact, 2523M Wired, 536M Free ARC: 2067M Total, 3255K MFU, 1821M MRU, 35K Anon, 38M Header, 204M Other ===== 2523-2067=456 MiB Memory usage a few minutes later ===== Mem: 10M Active, 27M Inact, 2721M Wired, 333M Free ARC: 2002M Total, 22M MFU, 1655M MRU, 21K Anon, 36M Header, 289M Other ===== 2721-2002=719 MiB So why the wired ram on a machine with only minimal amount of services has grown from 92 to 719 MiB? Sometimes I can even see about a gig! I'm using 9.2-RELEASE-p1 amd64. Test machine has a T5450 C2D CPU and 4 G RAM (actual available amount is 3 G). ZFS pool is configured on a GPT partition of a single 1 TB HDD. Disabling/enabling prefetch does't helps. Limiting ARC to 1 gig doesn't helps. When reading a pool, evict skips can increment very fast and sometimes arc metadata exceeds limit (2x-5x). I've attached logs with system configuration, outputs from top, ps, zfs-stats and vmstat. conf.log = system configuration, also uploaded to http://pastebin.com/NYBcJPeT top_ps_zfs-stats_vmstat_afterboot = memory stats immediately after booting system, http://pastebin.com/mudmEyG5 top_ps_zfs-stats_vmstat_1g-arc = after ARC grown to 1 gig, http://pastebin.com/4AC8dn5C top_ps_zfs-stats_vmstat_fullmem = when ARC reached limit of 2 gigs, http://pastebin.com/bx7svEP0 top_ps_zfs-stats_vmstat_fullmem_2 = few minutes later, http://pastebin.com/qYWFaNeA What should I do next? [-- Attachment #2 --] 7zXZ ִF ! X79] 1<GbMH32`ɯrO'N@+ũe E(fΒR<@j;Q2 N\W "[#.8ΉT WSXTô^K'ΐN(!T5Upra2Gg1FdQ_ȣd'M&p6M_b<[q|tǺTaٔ目wJoSgǎzbX5 MBA/,S9+}@dd:ZW[GzEPSL +6$̑KDLQ_b~֨Sib,oPS\n>^jG@58#H;eNG;\}W!=+-P1ya6+ysQ2Z96snԊpePhl%^d,jm`b'[# J[^):ŀ9O$$̘KTNɾo=ƿYKWp8VdV +\H](?$?3Z rDgJfŦBTod4pŇN+&,DVhCڐEnj |Gs&ǽIFtf^fÏ+TkXΧYw[Drj.+6#0&C,i#vY.N]# UNkI_c=I+1V +Aě=)$k "l?ThOʧ.W[3>@K :ίXֱ2D'Z1V_biOY3U#B"⡉*xl4.:\܅PGw'#@zgw5;# w0*2>ѓ8UfѤ_o OCٱ Luv, 8[-jMzÜmSCH` z9>FUg-8!XPy<~=V;\N+:@D~HK:%LJ@D*|k"?ŝg{WӌPaK*P Y<RZY_R%5M1f`,XOjgxCa&Xڕ7RٟTmAԜm2rhZc9x<I(ݑ@gO&㻶 BDu9Gܔ)CzZȄ{l* %T0x PFDlEed_=zG4li9HkKXԖ=9"%F}xU0D&+eG"X_?X[{)"QV_),'`4GnV=KQ= hG,b&\SƁt43)R 9_W-_]DxDуD:֨>4&rXqvLm}s3geNwZSX6sx|T<\Lͭ2#!ࠆ\9{tK,H>ˀ^"({q8l)k6GR#w\ ˝gȔ@_F 'k*`S[Cd|6i]LPRlS} 8,hD&xÈ~ҘZ*n5W6.si8k&69eۈ+c[U 9+LU6;k[r ۠"o8/Pɜ8(xP8܌Y\b<rgId;g{7b4@!=C)\AO9J?, )%,-I}Aj kU&@DO?"*ʧ)eħNˍ/[TS.+t*a%GPxoHmxvL.MXl])1g>pMZ+G^~۹o(4˝gkn=Y 0E돖%""PwZ5ٶYT㥆^{Ó.5h0nWTkVLbxFA$Ρ`R}QFIeG~7e+J̈́~|`OQ54^+xK_%Z8 q-xJK4c>4D~?vV]6?~"vFt;;R<Tc}yR kCx#9aȼt]kӶ0h"ii)\{k^[CfflE ![RCkZ3$ U}Ud #r哇LQ+~Ê|o谀-oj]&ף-30Fz H|MǜC" ;ӑB }R,Zd7LyR%c@"|<n:!8 bb;7`>q?_ C:~*;'*tdb`[ j {.9zq_\[͜A" TG$/1Ma `k34ԡ}I5k<Hgd 6ozg3Idy::o%txTK.S\VrP9|Lm]4S&-`fH 'l(*^m4IgC3DdcM:Uˈghvֹ(T˥38ͮ51`l :onHA5v {>*mD[)À&*,u\~'ץl}Vzzڃ#ύAH`IEp-j۽n ~f#W.#噐 *kc$xԏ3--D%Vd=D:צ$HkTyj# Rp(1E_8̈́yfiǮ]M' (.bXFS'-Bm*Q-=:j\E?*у}@81[,AC_>yN0nm}~Nv46=vhKnOvlhmVua䂷1hb["x[/ADZȀL $,/ b7yɲTٺZfl&hU⧟"c$79/,Ѹoܶ?t7hABf1WΙI3)<cNbFU&/d,&"!X5#Th^D@)VH1rw=}>I8M4qQn13g>Ak@j/],XKiUrdOL!wJLHۄ$j1 a M]@`fk~u8kPoFS^q`p<>&[y̗?ǚd 4.. g\ڡ~. ^0)NQ\uQ-=A5,0hM(Nh?Q1cf{GVxv +74byMk&2\lUAީN`V ۗlcoK]vTS8&tUljn 5&FG|MVÕ+InI!tgw:L tԓS倧&L@O*Rۂdxs}OTm-E"z3SMviݏ8%yn7=|-">{j,XqGc@s!2{[ڿK'+&)қQ)|D]PZ|](!dp Cn2@{D%^FŒis`;Ed93%hw~Y\rH~[Pz4] p@JЏ.&۶LT3'1%IZn20^^?Z*ay4\$ۺ@f?)E a=5ђVaj:C2ɸVbS#u!E2k#CV/j#Ba'Xp LBSԔ^p41ErޤfIQf|+LA:|Z-aLSUk+۠+@匶@Gl%Y(Mڄ랺IΪ⢉(D;57RDTαqEvuC oX&[iN,ABR*_oۄ<aeW`7eJ~AĽIx1&뇲p]g}LIwحzXN=avAPPC@ZQJxpP%ygXmf8BSu$P_)11)z62aɀâ;aRU5<βܑיv93؈almt)^c՟}&JQZ{Iqx|P-Af"4ƭrDDO.%mHӪ^"]GÜV-oQ":v*W*8-4xlwV&q̪i_ 8_R;Ns\Dx+2XdoHЦ^Mbo7,3b'Q>ϲD8wvjk9-ΈK ܦ }# __fdx'فR;͞=^Th!BIDϭuO"LJ'ed`(=σP$yR ~b_ac7S](d @˦,N0(Ykx9\ (Xe4)?thp5 | 6hAQZ ص2ڸ'</y@X*`i8Y$Jr7ZJz, MH\hP&b3߃0v ^ũ?rBǓJgm*,;v^tڟ 12ÜaH手Z2{eKTUE@phc#Qrc\?J J{ a᪾XRᴾnLezߖ03 /O_74j6 biZS cvtQi9=VdvH 0~BSӼ`nDZdz&TЄ`I>xϲT^t"7mQSx*&,Yz ˛nRZ0?i ~*Ь]k( )b:}m~&C\*SԪp#u%AWӜih fAއ+Ibh ,X1/22Y_?J:tPֲWܡچ(d\I+4N*L"֥h0!g3=dK5P[rU(\$wVF܁|[kxuR[tHb1{0WZ:uVOmјNef(̃N>o2W8?3X&{RtO6\nlØu1rM0J0)dL6-:ᳺÏȵQ!$fY.h ujBh}N, _& R^>Xm7|i^|~P '=)t?\3c Zηh<t71j3e]ƪֲ ۆM\" r|!;JRXVӣ@CSPTIy\K3XO? x&<4H#aSvnSaŅLa=&(6tH}ddֹ@9:^;}96=6>ԛ)sDY~mc KPm:썸,K4:RYK@.@cra{aȹ<'=O*W"PT0(Є]ÜEߓkuv#SQ_ _rQ!p3D)9rEϕ)VjyIke%㰂,3WwGQb$UMu=Bp6-vC^p$bՃdQ7MkP*j:ElE߆Rl<yzzdV|m%D7P,mgQ#gn%;n= >cOXP%<dcKDY/g8:h?A-*vݡBY16}܈,,fA5)C/_ մnQOҥ>_T9)'-] GJy~$ <l !o~'4 } @H\V[ehP&6PO+VM~S/ ~K{iaGjMU+H3G=2ųq4ZzC?m5ogD`:<7$ut^YnXxE5Kv2Okll &b\mEqa?/Ft]Cmˌ&VԲ28Wʸq}qS|3*ZIvN1+Xy܊(@ZQ`Nn"AuiW5^XQ9k]C5tz~âQtdmu?6֯sN Yes͆mҾ|BV#b\`T!9hce[,̈́HvC#&uh(C4 ;D JQ%UYu}x16kFd%)[(¿MRw{[ +^wx*)X53WkyTU")V?$:A;i9%$Ϯ^Շ&4@SHvTJU r)m]+־'3_P _˂Aٸܞ+MYm]trR/'?eo%`B~{/O+Q}Ofw݃΅4/]zzɂ%ͳ#d'ӝM5/f8!S #YL!ɟ0t>*@.S %٠s͒_QքXNxϿҜq_M0ꃓ"kp)%_љ5 J, a^|i [e =6<PKʅ {^8/wv"$Jōeoma4JWW+h|LSҐN7u_tUae~",U1'3qsQL Aø` rՅo!Lm-jAXVk@Ɲ<6>njцd5϶=u9.$.oP]<?lk/yEJ8NbRv,}vfw'2GV{P( 6؍`<Cތt0mwxke i1Rr;TD(ImX菎zP U0>ei魛ѣݦ8ZWJN,u^Trs0`%{@:BtL xyi+udzMEj^2-13Ŕ5" ZMrLl%vlεz8:|x[#BZGcΰSB婋[bӸsjا[>ܚnFwn`G0Tb +؋tg~/eOIGhg4+7H0zNU{wqpc$M ,p ɣg/iBLu7L+T=ʐ\邮]>"2Np*xO6? Pm%yuB@|ǃXIӮleaO\x:#3ɬ i'IԫoVr JHz?%v->.qDeT!dM8n+ 9/Z#E209\"D*1Md:Πp ku0^˘?_DfSKo ;m-#|ϱq̻dFzQEa;j ~mPaxh/DrdTH:ܸ6i2mef sBu&*yFI!$e3ގ0$4~ b稵CR$S8^qLt%ZK ,& :^: aVEfPDmѮl%L~){ j/jun)7|nqYǨ>U.YbD EoԷ2G\@ÁT'b!x 3Fn7 )@u).ĦFq>@e 5܋Ϟ-rQ!}!~_|c:}{>` 8EG*gS ^NvvN.IkAX\yT]CFPrh.=gV1"s?4-%?r%7r><k*Cq|T R|T;H_X=ښfC~v[)K~~cvo +.J~1sA(ܚ<GSr%kT!`~U0r 4!RQZKNԛ;X}?{Ȉ$+hE[>L`͂oF ,~ :9KFZ!ՖM wa :3h1)5e=ٕka*hMs;!A") 5ad/pdQGXi觰=i3yj2opT<}ѝ'Vv[|(Ranc[f yLjb͡lG]@L .}S*P]`^W{odjBsR@gZ@bF*cvtM.zۛ1 ]56]?8 ~-n>9"Iz5tq'>ȁ:t9;{ӝM~]! y dZEt 9oݬ$,Ej xvDf ^F qM߹,maB9PdKcٻfh8-*THlx Uei-eCWT(h̩2_?C3鿹W41Z>.jnGuz-1_S(O*S @Zٲ da _&&lՈu&:ajB[NVI\l=w4 Gd6{%צ?50[UGUOÉH+ۍVT1và2H<eѵz_*qޭn,pɟY7s n-)-ej}P?"Qәm5n8XQj£20&.', !8(;ޘ ~F'?5&ɽ9`Y*7}Y^^`#ښ O+`pb(`+(8͗E_,j$*=<ml/xq5g %Za,yL̢tA!ZK숕*c+ + VH f7A|j :HcXi-N}( pOebIMRPz $E[C ]'T=դlicGuX I'W۞'ʌh$c[?M?;B<!!=BMf{)<QBmQ:N m|QO)lURfaZIoI8ءL*:9$G!6~_iqjsD?mCofsjj@g'I\|7#pg3XaQK@ lfN%CV߅c}=ikRD̠v_;9~5gzKOb~Sf cp+"dDhoy&f-ABNu1hN;˱!u&`j>mrcΊyYܣ`grsg^AКk?[ }-&w$-X[Jn5#gQp&F6Ƭ> бv 3G`,t=v3g=@?'"M)-$HR%Q)oyhw.@=״x<8X% $ZHcG= 즤cD]U DkVv>o2.PpLVe/hٽe^4=BE{ۉb.lUe~>= #1`# Y58&p8j8$3uRci!(5dZ6q9C3;ԸK)G)'m$s,8O]rhLP}| 4;ep.f$U>uKo935 䭒 9/ g.9Sq[4-W/Ԍ&[=ݰ3ܣ[?⒆Vr*ToIbUP+"&s[(DÊ<~jf%{Jalz©te:(]kn#xSӪ vq1FdE,v\⏴<j3%T^&\Scgۨ]4q/ ;\pgLZR &uqьAx]T5O4$6/`\vXoz5kZ(ܸ;$=8n 7 B`hqwF?c[-f?o'.Ս=?S¡P'3嬃yW{x$J/Qu`5xH@cs"e 2yڏ+#穒ZWsDv .J]RkԔqFGѝ @hƒɪ|j~%2:}!Bʭ t/ƗjqI^ ®⎠.5\eZ";a6QpYESQ ޙRрa j8#r{/hdS.\%`*Fj:AaH8;PrQ lxQkךkd<ȾW>mϥ>
