Date: Tue, 14 Jul 2015 07:01:47 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion Message-ID: <55A4FA2B.5050903@denninger.net> In-Reply-To: <55A4BF06.4060505@ShaneWare.Biz> References: <CAB2_NwCngPqFH4q-YZk00RO_aVF9JraeSsVX3xS0z5EV3YGa1Q@mail.gmail.com> <CAJ-Vmom58SjgOG7HYPE4MVaB=XPaEkx_OTYgvOTHxwqGnTxtug@mail.gmail.com> <55A3F9E1.9090901@denninger.net> <55A4BF06.4060505@ShaneWare.Biz>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms020808060602090806000809
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 7/14/2015 02:49, Shane Ambler wrote:
> On 14/07/2015 03:18, Karl Denninger wrote:
>
>> The ARC is supposed to auto-size and use all available free memory. T=
he
>> problem is that the VM system and ARC system both make assumptions tha=
t
>> under certain load patterns fight with one another, and when this
>> happens and ARC wins the system gets in trouble FAST. The pattern is=
>> that the system will start to page RSS out rather than evict ARC, ARC
>> will fill the freed space, it pages more RSS out..... you see where th=
is
>> winds up heading yes?
>>
>
> Something I noticed was that vfs.zfs.arc_free_target is smaller
> than vm.v_free_target
>
> on my desktop with 8GB I get
> vfs.zfs.arc_free_target: 14091
> vm.v_free_target: 43195
>
> Doesn't that cause arc allocation to trigger swapping leaving space
> for arc allocation....
>
Yes and no.
On my system with the patch:
vm.v_free_target: 130312
vm.stats.vm.v_free_target: 130312
vfs.zfs.arc_free_target: 86375
and...
[karl@NewFS ~]$ pstat -s
Device 1K-blocks Used Avail Capacity
/dev/mirror/sw.eli 67108860 0 67108860 0%
No swapping :-)
It's not busy right now, but this is what the system looks like at the
moment...
1 users Load 0.22 0.28 0.32 Jul 14 06:57
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out =20
in out
Act 2009856 39980 7884504 92820 937732 count
All 17499k 52212 8727248 381980 pages
Proc: Interrup=
ts
r p d s w Csw Trp Sys Int Sof Flt ioflt 2638 tot=
al
2 251 1 9264 3332 3982 1134 181 2174 1134 cow 11
uart0 4
830 zfod =20
pcm0 17
0.4%Sys 0.1%Intr 0.8%User 0.0%Nice 98.8%Idle ozfod =20
ehci0 uhci
| | | | | | | | | | %ozfod =20
uhci1 21
> daefr 508
uhci3 ehci
dtbuf 1612 prcfr 991
cpu0:timer
Namei Name-cache Dir-cache 485859 desvn 3105 totfr 139
mps0 256
Calls hits % hits % 161014 numvn react 43
em0:rx 0
7109 7026 99 121460 frevn pdwak 77
em0:tx 0
459 pdpgs =20
em0:link
Disks da1 da2 da3 da4 da5 da6 da7 intrn 192
em1:rx 0
KB/t 0.00 11.41 10.84 11.68 11.60 0.00 0.00 21089128 wire 165
em1:tx 0
tps 0 21 24 22 21 0 0 1153712 act =20
em1:link
MB/s 0.00 0.23 0.25 0.25 0.24 0.00 0.00 1281556 inact 32
cpu1:timer
%busy 0 4 5 5 4 0 0 20372 cache 25
cpu9:timer
916480 free 39
cpu4:timer
buf 32
cpu13:time
22
cpu2:timer
33
cpu11:time
28
cpu3:timer
30
cpu14:time
35
cpu5:timer
37
cpu12:time
71
cpu7:timer
26
cpu10:time
26
cpu6:timer
28
cpu8:timer
48
cpu15:time
Most of that wired memory is in ARC...
------------------------------------------------------------------------
ZFS Subsystem Report Tue Jul 14 07:00:29 2015
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 53.54m
Recycle Misses: 15.12m
Mutex Misses: 6.63k
Evict Skips: 275.51m
ARC Size: 75.59% 16.88 GiB
Target Size: (Adaptive) 75.73% 16.91 GiB
Min Size (Hard Limit): 12.50% 2.79 GiB
Max Size (High Water): 8:1 22.33 GiB
ARC Size Breakdown:
Recently Used Cache Size: 58.52% 9.89 GiB
Frequently Used Cache Size: 41.48% 7.01 GiB
ARC Hash Breakdown:
Elements Max: 1.72m
Elements Current: 58.40% 1.00m
Collisions: 50.07m
Chain Max: 8
Chains: 119.31k
------------------------------------------------------------------------
ARC Efficiency: 2.01b
Cache Hit Ratio: 81.50% 1.64b
Cache Miss Ratio: 18.50% 371.70m
Actual Hit Ratio: 79.46% 1.60b
Data Demand Efficiency: 83.00% 1.60b
Data Prefetch Efficiency: 15.11% 21.33m
CACHE HITS BY CACHE LIST:
Anonymously Used: 1.79% 29.34m
Most Recently Used: 6.36% 104.08m
Most Frequently Used: 91.14% 1.49b
Most Recently Used Ghost: 0.09% 1.40m
Most Frequently Used Ghost: 0.62% 10.17m
CACHE HITS BY DATA TYPE:
Demand Data: 81.12% 1.33b
Prefetch Data: 0.20% 3.22m
Demand Metadata: 16.06% 262.92m
Prefetch Metadata: 2.62% 42.89m
CACHE MISSES BY DATA TYPE:
Demand Data: 73.17% 271.97m
Prefetch Data: 4.87% 18.11m
Demand Metadata: 17.75% 65.97m
Prefetch Metadata: 4.21% 15.65m
------------------------------------------------------------------------
--=20
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
--------------ms020808060602090806000809
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC
BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg
U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD
VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe
MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A
MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9
TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE
hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc
rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv
/e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi
S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL
J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z
OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK
eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4
k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF
BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV
HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W
HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1
o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr
YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm
7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+
fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+
ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv
liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG
MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko
3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8
2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa
NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx
h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg
xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y
aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa
BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt
cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc
BgkqhkiG9w0BCQUxDxcNMTUwNzE0MTIwMTQ3WjAjBgkqhkiG9w0BCQQxFgQUJe2Y5XNR2F6F
3zKJHMtdho0AYy8wbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC
MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN
BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ
Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG
SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgAdMo90
sXH9XwJL6eC+hve+Ztn60ucDQLqjD4zjrvYF5qQG6Z5qeer5SmqDd8IbhjwSj62uK3YhubFl
VnpAgeNk9Fk4XkFv4lGawgbEKPE7PKkNti+6q3K3Zq+URqWEsAAOj9n0Ohx4BNjcLNdugZIb
eXXRPHHWHsEv61Ftp5AmkTm1jMsnm35azlMe8jB1g5hoKOBa+qnl8hvMDKyzL9z7ewWRqdeg
1LjZ2n5WakFKSghsGiwj2oYeRfjprmIgTpfaMuSqn2lwXm5iETvd/KVS5fC7n2aoXDfRQkab
Ezf0MioI7pXUHTKEDOS0gfJs4SGrcOVp5hsWVFz4yLtXmb+RLsiWuizv5OTUn/TaU4nL/OA4
EW3kWhnhu+AnWhxUibd9zVeArjHWP8viB2+0lbWfmM0GlhYb1lBHZocQNniDNihpbNZuGc6q
FqsurPiRtifTX6On3rMdCvn2sN4fARcHolVUf8yA4fecpobUapUaYj1N22QPLUfyILkQxOPz
FtRpXKhP3a8xeirjPijAxF0wZ7+5RsWppn9/L9JTkj6bnRmWoRvVJOjVDJvpKaXBpwZRSCPn
YEHbvEQVVH2/n5OjK5v2pICumHPuJd9FOmItXSfcl6BjyC7C88fYnaY/RuvO7h2wTvaaKcbJ
H9oFF9U1Pp99FxVi8+aJXx304htidgAAAAAAAA==
--------------ms020808060602090806000809--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55A4FA2B.5050903>