Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 15 Jul 2024 13:41:15 -0700
From:      Jim Long <freebsd-questions@umpquanet.com>
To:        Dan Langille <dan@langille.org>
Cc:        freebsd-questions@freebsd.org
Subject:   Re: Unable to limit memory consumption with vfs.zfs.arc_max
Message-ID:  <ZpWJa%2BVLW1apLA%2BD@sfo.umpquanet.com>
In-Reply-To: <ZpWFhbZFukxWyzMP@sfo.umpquanet.com>
References:  <ZjUYzdiz01GgnbX/@sfo.umpquanet.com> <4299f196-1fbe-4590-8668-d023d7044e8d@sentex.net> <ZjVLxKH5LDBZjakR@sfo.umpquanet.com> <3d2f7913-b91d-4d00-a9b4-f37cf1032fc0@app.fastmail.com> <ZpWFhbZFukxWyzMP@sfo.umpquanet.com>
index | next in thread | previous in thread | raw e-mail 

[-- Attachment #1 --]
As Bugs Bunny often said, "What a maroon!"

Here's the attached MRTG graph.



On Mon, Jul 15, 2024 at 01:24:37PM -0700, Jim Long wrote:
> Picking up this old thread since it's still vexing me....
> 
> On Sat, May 04, 2024 at 07:56:39AM -0400, Dan Langille wrote:
> > 
> > This is from FreeBSD 14 on an Dell R730 in the basement (primary purpose, poudriere, and PostgreSQL, and running four FreshPorts nodes):
> > 
> > >From top:
> > 
> > ARC: 34G Total, 14G MFU, 9963M MRU, 22M Anon, 1043M Header, 9268M Other
> >      18G Compressed, 41G Uncompressed, 2.28:1 Ratio
> > 
> > % grep arc /boot/loader.conf
> > vfs.zfs.arc_max="36000M"
> > 
> > Looks like the value to set is:
> > 
> > % sysctl -a vfs.zfs.arc | grep max
> > vfs.zfs.arc.max: 37748736000
> > 
> > Perhaps not a good example, but this might be more appropriate:
> > 
> > % grep vfs.zfs.arc.max /boot/loader.conf
> > vfs.zfs.arc_max="1200M"
> > 
> > with top showing:
> > 
> > ARC: 1198M Total, 664M MFU, 117M MRU, 3141K Anon, 36M Header, 371M Other
> >      550M Compressed, 1855M Uncompressed, 3.37:1 Ratio
> 
> Thank you, Dan, I appreciate you chiming in.
> 
> Unfortunately, I think I have those bases covered, although I'm open to
> anything I may have missed:
> 
> # grep -i arc /boot/loader.conf /etc/sysctl.conf 
> /boot/loader.conf:vfs.zfs.arc.max=4294967296
> /boot/loader.conf:vfs.zfs.arc_max=4294967296
> /boot/loader.conf:vfs.zfs.arc.min=2147483648
> /etc/sysctl.conf:vfs.zfs.arc_max=4294967296
> /etc/sysctl.conf:vfs.zfs.arc.max=4294967296
> /etc/sysctl.conf:vfs.zfs.arc.min=2147483648
> 
> # top -b
> last pid: 16257;  load averages:  0.80,  1.15,  1.18  up 0+02:03:34    12:05:06
> 55 processes:  2 running, 53 sleeping
> CPU: 11.7% user,  0.0% nice, 18.4% system,  0.1% interrupt, 69.9% idle
> Mem: 32M Active, 141M Inact, 11G Wired, 3958M Free
> ARC: 10G Total, 5143M MFU, 4679M MRU, 2304K Anon, 44M Header, 219M Other
>      421M Compressed, 4744M Uncompressed, 11.28:1 Ratio
> 
>   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
> 11057 root          1 127    0    59M    33M CPU0     0  60:16  82.28% ssh
> 11056 root          5  24    0    22M    12M pipewr   3   6:00   6.25% zfs
>  1619 snmpd         1  20    0    34M    14M select   0   0:06   0.00% snmpd
>  1344 root          1  20    0    14M  3884K select   3   0:03   0.00% devd
>  1544 root          1  20    0    13M  2776K select   3   0:01   0.00% syslogd
>  1661 root          1  68    0    22M  9996K select   0   0:01   0.00% sshd
>  1587 ntpd          1  20    0    23M  5876K select   1   0:00   0.00% ntpd
> 14391 root          1  20    0    22M    11M select   3   0:00   0.00% sshd
>  2098 root          1  20    0    24M    11M select   1   0:00   0.00% httpd
>  1904 root          1  20    0    24M    11M select   2   0:00   0.00% httpd
>  1870 root          1  20    0    19M  8688K select   2   0:00   0.00% sendmail
>  2067 root          1  20    0    19M  8688K select   1   0:00   0.00% sendmail
>  2066  65529        1  20    0    13M  4564K select   2   0:00   0.00% mathlm
>  1883  65529        1  20    0    11M  2772K select   3   0:00   0.00% mathlm
> 14397 root          1  20    0    14M  4568K wait     1   0:00   0.00% bash
>  1636 root          1  20    0    13M  2608K nanslp   0   0:00   0.00% cron
>  2082 root          1  20    0    13M  2560K nanslp   3   0:00   0.00% cron
>  1887 root          1  20    0    13M  2568K nanslp   2   0:00   0.00% cron
> 
> # sysctl -a | grep m.u_evictable
> kstat.zfs.misc.arcstats.mfu_evictable_metadata: 0
> kstat.zfs.misc.arcstats.mfu_evictable_data: 0
> kstat.zfs.misc.arcstats.mru_evictable_metadata: 0
> kstat.zfs.misc.arcstats.mru_evictable_data: 0
> 
> An mrtg graph is attached showing ARC bytes used
> (kstat.zfs.misc.arcstats.size) in green, vs. ARC bytes max
> (vfs.zfs.arc.max) in blue.  We can see that daily, the ARC bytes used
> blows right past the 4G limit.  Most days, it is brought under control
> by two reboots in /etc/crontab ("shutdown -r now" at 02:55, 05:35),
> although some days the system is too far gone by the time the cron job
> rolls around, and the system stays hung until I can get to the data
> center and power cycle it.
> 
> I'm not very skilled at kernel debugging, but is a kernel PR in order?
> This has happened with a GENERIC kernel across at least two builds of
> 14-STABLE:
> 
> FreeBSD 14.0-STABLE #0 stable/14-n267062-77205dbc1397: Thu Mar 28 12:12:02 PDT 2024
> FreeBSD 14.1-STABLE #0 stable/14-n267886-4987c12cb878: Thu Jun  6 12:24:06 PDT 2024
> 
> Would it help to reproduce this with a -RELEASE version?
> 
> 
> Thank you again, everyone.
> 
> Jim

[-- Attachment #2 --]
PNG


IHDRH-PLTEdddfOq	pHYs+MIDATx]˒$)Lp5f"cnx´PzI |QS?*OPa<,8"2_h>DspA~}8<ܨ\J.nnMzSgjNj.9lv
ѐ1Bb1|;N¹8p2\a0\΍gQrn\yzX,/0LgyjFnVX0')/}<`%"$.g.ok,oM*bηiՇ)2?Xx&rB|c7:p^O:Trد=\\4W+:O.x?d2u21Uu3]*'aD_0M$;GS]_2r(̈́lמL{_X$'`\]',"ƗI_<p"pV^`n3pXR]n6ƇC
X]Wl`iK`IGppj2/mW=j^Xƫ5,_E'"ǿIXTUV
b66
Fq;eiIMZ:¬>gBe*VW
|-M[O6->/+O[4W{$R (/><m||wiiuےU"%.DyE||||||_󽚾}W$S_mpw'r#k\vrbϿ'仛~|s<|Z|_o1?/ɌKN\-$_	߶9|ۭO|'6,jngݎd0˷ed]]ﲗYw+#%8||obW]tW9jw#Ӟ|̷ډfgoB+ic;m3i{{W
߷Gߩ
׻	y%~/x-׻?kU_j7eo\ߢP:Z _S/ȭҸn`+juUzv7R?d@GKr@Vԝ|?X\Q'䀂;?OM#SOmyZ(וoZ?|3A|aF^L?8%LTY||iK̥<O5~(u	W={}L~8~/懙
h~k|1C|C[l?C^oT9o>Qw(Ԣ=?mJUM~(/|o÷ZkA^?𥟞ksv)@/O.߳*p{v?㇏+>;d~:k16mLR/b^_2\KAzC>D1_~~~~~~~x*as!	pt}i0X/D>-R['}McrXQgst}IKaQaw
?\b{ۼs?DY@~~~{"?|~~WsNu}%PחԿb.eO6r}%|R/~8~]=~.~(.y1?p|}W6&sA{?={	pl~C~3Az}vv+8|@;|{*?v/\!|a>c+|üE>x|
?4};Wt+D9L×wj|閙oϮW:E.Rm,ǁ?L{^a|pHARMruqTKh;Bm?y/Ja__uP~ Y^+Ա4{Ed{ۦ+j#}v+jAlȍө|nV6R赣#/2v8l4&:I6_׽ne7'Þ^뤟GZJWv3MZ7iAޤE+Į;|*Bl{s9aFv$k_=د }ǜi-_wLWZ\gSț=0T"yS
a,B8xa.xppwp@ ^!< B8,G]9~fFoKsYpX?//#*,yiSmYȶ~F8Dw#
użh87qS=W8wv8spDfC{"yi{ɻȢyq`6^?*{R8<@M=q8ȋppphW~p8sp+7>v#ƒݯs6yMߗըna_!KfDVymU_9ɖMK8pfپ`&wskȫ! OMwl^Ң9S\y8ҏ=4Խeסo#a~AF
??q8dٽyjGJM3<D6mF"$aC^1{}77F /a,HHq)B8$v/a94#wCh^-pyq`":!zG8قp(g&!#wCh^-pyq`":!zG8قp(g&!#wCh^-pyq`":!zG8قp(g&!7Y
 Y!$pB8 ¡^ٗ	
pӍ2Wb	aÙ+oAw#	yn$4 ufy𭍦	~Ӿ.W	μE85[+m8w6B8$B8d(g&!#wCh^-pyq`":!zG8قp(g&!#wCh^-pyq`"5!W*/!)E8xȋpH*/6aU8ȥ;{I0ʽ@]E8 B8 #CG8pxp+Fy);9hw#ͯ_9߻_s/pn$a=J8ލD,KvI+9e@ާ
m"pxY|c?{}YOzC$yvpX˛<vE8 0sKC[ R6tpD8ˋp@+A";_^ʽT8Ē+
nȫhyʆwxvwԣ7Ζ~pxwA'~䥡xk.[]F3ly!K^EUáKb;[Qngg<MC[8'5p,o$P<6Z8_,dp<tw¡;?Vp(l{\5^EA?G㸗#o1s>h\;:iR]x]d)k򖫙Aṥ3m۰},#Zf"\e27c=vv5o$CWel3_̿XMk
[ju'&qwwmY͔N{<)@۾c4oUQz5y֧͗7h;(c2zeo_/?3>g5"I' &-AԓZ=jy75
Qk`{+椨k.N:`f\#M++FBX"i;i=@w۽WI#13LL{ؑטq	dO
i󕷪\M{LZ/a6yn+%-ț>e}BAY7MIENDB`
help

Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ZpWJa%2BVLW1apLA%2BD>