Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 28 Feb 2014 22:04:41 +0200
From:      Anton Sayetsky <vsjcfm@gmail.com>
To:        freebsd-fs <freebsd-fs@freebsd.org>
Subject:   Fwd: ZFS and Wired memory, again
Message-ID:  <CAFG2KCLJpcS_eSHHh0%2BRqzO0gAsMSAtnaN%2BHS0xdtvFmhUtUYA@mail.gmail.com>
In-Reply-To: <CAFG2KC%2BZSHEVFbpPD9e1QHRdY=Sd6EuAD80vyDLDDQcpgCQNhA@mail.gmail.com>
References:  <CAFG2KC%2BZSHEVFbpPD9e1QHRdY=Sd6EuAD80vyDLDDQcpgCQNhA@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

[-- Attachment #1 --]
2014-02-28 21:58 GMT+02:00 Mark Felder <feld@freebsd.org>:
>
>
> On Fri, Feb 28, 2014, at 13:43, Adam Vande More wrote:
>> On Fri, Feb 28, 2014 at 1:35 PM, Mark Felder <feld@freebsd.org> wrote:
>>
>> >
>> > >
>> > >
>> > > Without question, cron could do it.
>> > >
>> >
>> > I can't see cron using kernel memory; that just doesn't make sense to
>> > me. Not even the periodic scripts that cron executes should be able to
>> > balloon kernel like that.
>> >
>> > I think I know what meant to infer though -- that some nonstandard cron
>> > script is doing something ugly.
>>
>>
>>  He's running 150 TB on 3 GB of mem.  Periodic I think could consume that
>> alone.
>>
>
> Which periodic script are you referring to? The daily security check? I
> suppose I could see resource usage increase after checking for new
> setuid bits... assuming he had a lot of files to crawl through. But I
> don't actually know if that would do much to wired memory.
Ok, I'll repost my first message. I repeat: this in only _test_
system, with _single disk_, with _only_ system, ports & src on ZFS
pool. No services are running except those which are in base system.


---------- Forwarded message ----------
From: Anton Sayetsky <vsjcfm@gmail.com>
Date: 2013-11-22 21:53 GMT+02:00
Subject: ZFS and Wired memory, again
To: freebsd-fs@freebsd.org


Hello,

I'm planning to deploy a ~150 TiB ZFS pool and when playing with ZFS
noticed that amount of wired memory is MUCH bigger than ARC size (in
absence of other hungry memory consumers, of course). I'm afraid that
this strange behavior may become even worse on a machine with big pool
and some hundreds gibibytes of RAM.

So let me explain what happened.

Immediately after booting system top says the following:
=====
Mem: 14M Active, 13M Inact, 117M Wired, 2947M Free
ARC: 24M Total, 5360K MFU, 18M MRU, 16K Anon, 328K Header, 1096K Other
=====
Ok, wired mem - arc = 92 MiB

Then I started to read pool (tar cpf /dev/null /).
Memory usage when ARC size is ~1GiB
=====
Mem: 16M Active, 15M Inact, 1410M Wired, 1649M Free
ARC: 1114M Total, 29M MFU, 972M MRU, 21K Anon, 18M Header, 95M Other
=====
1410-1114=296 MiB

Memory usage when ARC size reaches it's maximum of 2 GiB
=====
Mem: 16M Active, 16M Inact, 2523M Wired, 536M Free
ARC: 2067M Total, 3255K MFU, 1821M MRU, 35K Anon, 38M Header, 204M Other
=====
2523-2067=456 MiB

Memory usage a few minutes later
=====
Mem: 10M Active, 27M Inact, 2721M Wired, 333M Free
ARC: 2002M Total, 22M MFU, 1655M MRU, 21K Anon, 36M Header, 289M Other
=====
2721-2002=719 MiB

So why the wired ram on a machine with only minimal amount of services
has grown from 92 to 719 MiB? Sometimes I can even see about a gig!
I'm using 9.2-RELEASE-p1 amd64. Test machine has a T5450 C2D CPU and 4
G RAM (actual available amount is 3 G). ZFS pool is configured on a
GPT partition of a single 1 TB HDD.
Disabling/enabling prefetch does't helps. Limiting ARC to 1 gig doesn't helps.
When reading a pool, evict skips can increment very fast and sometimes
arc metadata exceeds limit (2x-5x).

I've attached logs with system configuration, outputs from top, ps,
zfs-stats and vmstat.
conf.log = system configuration, also uploaded to http://pastebin.com/NYBcJPeT
top_ps_zfs-stats_vmstat_afterboot = memory stats immediately after
booting system, http://pastebin.com/mudmEyG5
top_ps_zfs-stats_vmstat_1g-arc = after ARC grown to 1 gig,
http://pastebin.com/4AC8dn5C
top_ps_zfs-stats_vmstat_fullmem = when ARC reached limit of 2 gigs,
http://pastebin.com/bx7svEP0
top_ps_zfs-stats_vmstat_fullmem_2 = few minutes later,
http://pastebin.com/qYWFaNeA

What should I do next?

[-- Attachment #2 --]
7zXZִF!X79]1<G׹bMH32`ɯrO'N@+ũe
E(fΒR<@j;Q2
N\W	"[#.8ΉT
WSXTô^K'ΐN(!T5Upra2Gg1FdQ_ȣd'M&p6M_b<[q|tǺTaٔ目wJoSgǎzbX5MBA/,S9+}@dd:ZW[߼GzEPSL
+6$̑KDLQ_b~֨Sižb,oPS\n>^jG@58#H;eNG;\}W!=+-P1ya6+ysQ2Z96snԊpePhl%^d,jm`b'[#	J[^):ŀ9O$$̘KTNɾo=ƿYKWp8VdV+\H](?$?3Z	rDgJfŦBTod4pŇN+&,DVhCڐEnj|Gs&ǽIFtf^fÏ+TkXΧYw[Drj.+6#0&C,i#vY.N]#
UNkI_c=I+1V
+Aě=)$k
"l?ThOʧ.W[3>@K	:ίXֱ2D'Z1V_biOY3U#B"⡉*xl4.:\܅PGw'#@zgw5;#
w0*2>ѓ8UfѤ_o
OCٱ
Luv,8[-jMzÜmSCH`
z9>FUg-8!XPy<~=V;\N+:@D~HK:%LJ@D*|k"?ŝg{WӌPaK*P	Y<RZY_R%5M1f`,XOjgxCa&Xڕ7RٟTmAԜm2rhZc9x<I(ݑ@gO&㻶
BDu9Gܔ)CzZȄ{l*
%T0x PFDlEed_=zG4li9HkKXԖ=9"%F}xU0D&+eG"X_?X[{)"QV_),'`4GnV=KQ=	hG,b&\SƁt43)R	9_W-_]DxDуD:֨>4&rXqvLm}s3geNwZSX6sx|ŽT<\Lͭ2#!ࠆ\9{tK,H>ˀ^"({q8l)k6GR#w\
˝gȔ@_F
'k*`S[Cd|6i]LPRlS}8,hD&xÈ~ҘZ*n5W6.si8k&69eۈ+c[U9+LU6;k[r
۠"o8/Pɜ8(xP8܌Y\b<rgId;g{՘7b4@!=C)\AO9J?,
)%,-I}Aj
kU&@DO?"*ʧ)eħNˍ/[TS.+t*a%GPxoHmxvL.MXl])1g>pMZ+G^~۹o(4˝gkn=Y 0E돖%""PwZ5ٶYT㥆^{Ó.5h0nWTkVLbxFA$Ρ`R}QFIeG~7e+J̈́~|`OQ54^+xK_%Z8q-xJK4c>4D~?vV]6?~"vFt;;R<Tc}yR
kCx#9aȼt]kӶ0h"ii)\{k^[CfflE![RCkZ3$
U}Ud
#r哇LQ+~Ê|o谀-oj]&ף-30Fz
H|MǜC•"	;ӑB }R,Zd7LyR%c@"|<n:!8bb;7`>q?_C:~*;'*tdb`[
j	{.9zq_\[͜A"TG$/1Ma`k34ԡ}I5k<Hgd
6ozg3Idy::o%txTK.S\VrP9|Lm]4S&-`fH 'l(*^m4IgC3DdcM:Uˈghvֹ(T˥38ͮ51`l
:onHA5v {>*mD[)À&*,u\~'ץl}Vzzڃ#ύAH`IEp-j۽n
~f#W.#噐
*kc$xԏ3--D%Vd=D:צ$HkTyj#	Rp(1E_8̈́yfiǮ]M'(.bXFS'-Bm*Q-=:j\E?*у}@81[,AC_>yN0nm}~Nv46=vhKnOvlhmVua䂷1hb["x[/ADZȀL$,/b7yɲTٺZfl&hU⧟"c$79/,Ѹoܶ?t7hABf1WΙI3)<cNbFU&/d,&"!X5#Th^D@)VH1rw=}>I8M4qQn13g>Ak@j/],XKiUrdOL!wJLHۄ$j1
a
M]@`fk~u8kPoFS^q`p<>&[y̗?ǚd
4..g\ڡ~.	^0)NQ\uQ-=A5,0hM(Nh?Q1cf{GVxv+74byMk&2\lUAީN`VۗlcoK]vTS8&tUljn	5&FG|MVÕ+InI!tgw:L
tԓS倧&L@O*Rۂdxs}OTm-E"z3SMviݏ8%yn7=|-">{j,XqGc@s!2{[ڿK'+&)қQ)|D]PZ|](!dp	Cn2@{D%^FŒis`;Ed93%hw~Y\rH~[Pz4]
p@JЏ.&۶LT3'1%IZn20^^?Z*ay4\$ۺ@f?)Ea=5ђVaj:C2ɸVbS#u!E2k#CV/j#Ba'Xp
LBSԔ^p41ErޤfIQf|+LA:|Z-aLSUk+۠+@؁匶@Gl%Y(Mڄ랺IΪ⢉(D;57RDTαqEvuC
oX&[iN,ABR*_oۄ<aeW`7eJ~AĽIx1&뇲p]g}LIwحzXN=avAPPC@ZQJxpP%ygXmf8BSu$P_)11)z62aɀâ;aRU5<βܑיv93؈almt)^c՟}&JQZ{Iqx|P-Af"4ƭrDDO.%mHӪ^"]GÜV-oQ":v*W*8-4xlwV&q̪i_	8_R;Ns\Dx+2XdoHЦ^Mbo7,3b'Q>ϲD8wvjk9-ΈK ܦ }#	__fdx'فR;͞=^Th!BIDϭuO"LJ'ed`(=σP$–yR
~b_ac7S](d
@˦,N0(Ykx9\	(Xe4)?thp5|
6hAQZص2ڸ'</y@X*`i8Y$Jr7ZJz,MH\hP&b3߃0v
^ũ?rBǓJgm*,;v^tڟ
12ÜaH手Z2{eKTUE@phc#Qrc\?J J{
a᪾XRᴾnLezߖ03

/׈O_74j6	biZS cvtQi9=VdvH	0~BSӼ`nDZdz&TЄ`I>܏xϲT^t"7mQSx*&,Yz˛nRZ0?i ~*Ь]k( ޳)b:}m~&C\*SԪp#u%AWӜih 
fAއ+Ibh ,X1/22Y_?J:􄕁tPֲWܡچ(d\I+4N*L"֥h0!g3=dK5P[rU(\$wVF܁|[kxuR[tHb1{0WZ:uVOmјNef(̃N>o2W8?3X&{RtO6\nlØu1rM0J0)dL6-:ᳺÏȵQ!$fY.h
ujBh}N, _&
R^>Xm7|i^|~P
'=)t?\3c
Zηh<t71j3e]ƪֲ
ۆM\"
žr|!;JRXVӣ@CSPTIy€\K3XO?x&<4H#aSvnSaŅLa=&(6tH}ddֹ@9:^;}96=6>ԛ)sDY~mc
KPm:썸,K4:RYK@.@cra{aȹ<'=O*W"PT0(Є]ÜEߓkuv#SQ_	_rQ!p3D)9rEϕ)VjyIke%㰂,3WwGQb$UMu=Bp6-vC^p$bՃdQ7MkP*j:ElE߆Rl<yzzdV|m%D7P,mgQ#gn%;n=>cOXP%<dcKDY/g8:h?A-*vݡBY16}܈,,fA5)C/_	մnQOҥ>_T9)'-]
GJy~$	<l	!o~'4}
@H\V[ehP&6PO+VM~S/	~K{iaGjMU+H3G=2ųq4ZzC?m5ogD`:<7$ut^YnXxE5Kv2Okll
&b\mEqa?/Ft]Cmˌ&VԲ28Wʸq}qS|3*ZIvN1+Xy܊(@ZQ`Nn"AuiW5^XQ9k]C5tz~âQtdmu?6֯sN Yes͆mҾ|BV#b\`T!9hce[,̈́HvC#&uh(C4;DJQ%UYu}x16kFd%)[(¿MRw{[
+^wx*)X53WkyTU")V?$:A;i9%$Ϯ^Շ&4@SHvTJU
r)m]+־'3_P_˂Aٸܞ+MYm]trR/'?eo%`B~{/O+Q}Ofw݃΅4/]zzɂ%ͳ#d'ӝM5/f8!S
#YL!ɟ0t>*@.S%٠s͒_QքXNxϿҜq_M0ꃓ"kp)%_љ5
J,a^|i [e
=6<PKʅ {^8/wv"$Jōeoma4JWW+h|LSҐN7u_tUae~",U1'3qsQL	Aø`
rՅo!Lm-jAXVk@Ɲ<6>njцd5϶=u9.$.oP]<?lk/yEJ׹8NbRv,}vfw'2GV{P(
6؍`<Cތt0mwxke i1Rr;TD(ImX菎zPU0>ei魛ѣݦ8ZWJN,u^Trs0`%{@:BtLxyi+udzMEj^2-13Ŕ5" ZMrLl%vlεz8:|x[#BZGcΰSB婋[bӸsjا[>ܚnFwn`G0Tb +؋tg~/eOIGhg4+7H0zNU{wqpc$M
,p	
ɣg/iBLu7L+T=ʐ\邮]>"2Np*xO6?
Pm%yuB@|ǃXIӮleaO\x:#3ɬ
i'IԫoVr JHz?%v->.qDeT!dM8n+
9/Z#E209\"D*1Md:Πp ku0^˘?_DfSKo	;m-#|ϱq̻dFzQEa;j	~mPaxh/DrdTH:ܸ6i2mefsBu&*yFI!$e3ގ0$4~	b稵CR$S8^qLt%ZK
,&
:^:
aVEfPDmѮl%L~){
j/jun)7|nqYǨ>U.YbD
EoԷ2G\@ÁT'b!x
3Fn7
)@u).ĦFq>@e
5܋Ϟ-rQ!}!~_|c:}{>`
8EG*gS ^NvvN.IkAX\yT]CFPrh.=gV1"s?4-%?r%7r><k*Cq|T
R|T;H_X=ښfC~v[)K~~cvo
+.J񋴹~1sA(ܚ<GSr%kT!`~U0r
4!RQZKNԛ;X}?{Ȉ$+hE[>L`͂oF
,~
:9KFZ!ՖM	wa :3h1)5e=ٕka*hMs;!A")
5ad/pdQGXi觰=i3yj2opT<}ѝ'Vv[|(Ranc[f yLjb͡lG]@L
 .}S*P]`^W{odjBsR@gZ@bF*cvtM޻.zۛ1	]56]?8
~-n>9"Iz5tq'>ȁ:t9;{ӝM~]!
y	dZEt	9oݬ$,Ej xvDf ^F	qM߹,maB9PdKcٻfh8-*THlx Uei-eCWT(h̩2_?C3鿹W41Z>.jnGuz-1_S(O*S@Zٲ
da
_&&lՈu&:ajB[NVI\l=w4	Gd6{%צ?50[UGUOÉH+ۍVT1và2H<eѵz_*qޭn,pɟY7s
n-)-ej}P?"Qәm5n8XQj£20&.',	!8(;ޘ ~F'?5&ɽ9`Y*7}Y^^`#ښ
O+`pb(`+(8͗E_,j$*=<ml/xq5g
%Za,yL̢tA!ZK숕*c+	+؁	VH
f7A|j:HcXi-N}(pOebIMRPz	$E[C
]'T=դlicGuX I'W۞'ʌh$c[?M?;B<!!=BMf{)<QBmQ:N m|QO)lURfaZIoI8ءL*:9$G!6~_iqjsD?mCofsjj@g'I\|7#pg3XaQK@lfN%CV߅c}=ikRD̠v_;9~5gzKOb~Sfcp+"dDhoy&f-ABNu1hN;˱!u&`j>mrcΊyYܣ`grsg^AКk?[
}-&w$-X[Jn5#gQp&F6Ƭ>
бv
3G`,t=v3g=@?'"M)-$HR%Q)oyhw.@=״x<8X%
$ZHcG=즤cD]U	DkVv>o2.PpLVe/hٽe^4=BE{ۉb.lUe~>=
#1`# Y58&p8j8$3uRci!(5dZ6q9C3;ԸK)G)'m$s,8O]rhLP}| 4;ep.f$U>uKo935 䭒
9/g.9Sq[4-W/Ԍ&[=ݰ3ܣ[?⒆Vr*ToIbUP+"&s[(DÊ<~jf%{Jalz©te:(]kn#xSӪvq1FdE,v\⏴<j3%T^&\Scgۨ]4q/	;\pgLZR	&uqьAx]T؃5O4$6/`\vXoz5kZ(ܸ;$=8n7B`hqwF?c[-f?o'.Ս=?S¡P'3嬃yW{x$J/Qu`5xH@cs"e
2yڏ+#穒ZWsDv .J]RkԔqFGѝ	@hƒɪ|j~%2:}!Bʭ
t/ƗjqI^ ®⎠.5\eZ";a6QpYESQ
ޙRрaj8#r{/hdS.\%`*Fj:AaH8;PrQ lxQkךkd<ȾW>mϥ>܋c'$DB?ț=r#o`tK-T+~}POn
A'UJV:;
F=>Cn_e92/Ќ.H}DƋ6\3?Ix|QB2nhVj_hկyjKss2XSER8zіbx"OdB;>HЫ#̧ʜh\a\Vm$P
&Ќbne}=dRU3@@jt"hy:o4ZSPsR)*щF	]:OH|^`ySn~2_XM!%qNdP?>Q	LƗR<$2ml" loiW.Ix
i:\C9AW~yH!l?f#kwoixk҆SZjo-
0DYQ06|I:0끨mK
[~P+ɜ,˃9i36.wj1(AhJf]=/Y;A33ί#|
`3OAB"m]nt_6mJUFUv7aD=/2<94prOɖ#esۀ+Bd]>y5ro^7a?aYagbHdЯ((WF؜;['؈*fDt:i
I-R=fe]%Wi.n".F4ރxLQ	
{K!U=V&Áb(^0ZAlA=E$vMJnK7֔/
#s*JY	P;5\
JAxyij{
8'hvh	!.<mq<'`J#_ꜜΫ)1vN_('8n)"*kn2I\^+={-p5/\1E-zxo`FNM'l&Y҇nGF.ζSlWrt`-BZP)Ы7PmJ(
G۫Ij*d-?gUVGQhV|IC}RQu$h8Ay<֏*wZ
n=D›%)ĦRĂvN2c{E
*Dh(x7S
'c|-P1,'S}R7Ɔ<	ÈuBlB8Xrnbw&L>z]W/;TőiJ)b#Z)DNasb>Ձ	U3ߢ꠹;iζl-
_wO媥^@ġ
|@Hvw<:M4)IvT0ظ4dvE\֛$T@6VVy?zFA
#0`Ǡ:/w4=e'ZLG9(߃J\Po'8\UҖAߋxqUijcs&}4yyZZ_;ٕiH'B
<-U\`\~[4vK+Ш4Ef+C4myfg#>"Lhʛ֤YH.okibc~I[sk$rGHvUssDUH38fbv0jDt=!LCSNb~^zQ?M4],T®F^m#Gw3[bvZLLI{X	gkJCS ][椪!X-,k$d'Pb3fw6o,*~юs|;EҌ.⮁,c̆qʎpgr攊b^UD48`x?jYxd&d7T[˜CcvIZCЌC"'>L+J2ׅ	"no,`QNuXrv.)fZ?d&_ٵiɧZ7xWEJN|ehXxn!ka3Y`V
IC?&“L$aDaY]ηG~pT]L*jk7@D&ߟ
+Lk`ō7sI7h3YGL+{stQtRGd2}‡7U:3S(p{nb
t*}X>BMbE^㼿Mr
L Re^'7ں)H(K;aөXJ(ҏHUaP/\J{ʐJyS{]%Qx)l'N1wl,βOp^l?ÞOa"iM^uM̈20Z{{9, fs!1\	8H";и6gBLG<Qv! ŗJa>5
,WBP 2zZFx`tPc%єSsFn\Ay'o9\`@cHm{vśw-o1q!{hT26fR TRYŲL	VJE(ǸS7xF5"8G(ˢsPbٛZ7#Peu.#Zh&ܔOSڝ(*v6Ԋר=Us
5{jG:9҄X.Ge[SДRsϮ(m|4wyI
e<us\Cs.Upcִ%$jzˉ&{\3 Os}Rw
Zf.%UOcXv%':b9%(b*V/` |OYxQ)ؼO'W'R|#\]jyCtGi|ߎ'͐Zuel-y+IS"6	R:63D^y^63?ŹkzٝtmkNm	;TpTLqV/Çz;Eoe%LbI*$Nwmi\)2Nt^5 Fb7Nf8ndS/&t܆V{f1=ZVH{DVj¢
t]HD
؜GaGl#HBP6\J["m2T׻H;YJLTۉ<=!$-KxZt+7]ٲ ,2ufg}H=@{ޥ|!p1E2H#bO_%'B?_׮n^.7#"jtgYZ

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFG2KCLJpcS_eSHHh0%2BRqzO0gAsMSAtnaN%2BHS0xdtvFmhUtUYA>