Date: Sun, 16 Mar 2014 15:36:06 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-hackers@freebsd.org Subject: Tracking down what has inact pages locked up Message-ID: <53260B36.2070409@denninger.net>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms010005010301020203050001
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Given the following:
3 users Load 2.66 2.46 2.37 Mar 16 15:26
Mem:KB REAL VIRTUAL VN PAGER SWAP P=
AGER
Tot Share Tot Share Free in out in =
out
Act 879332 19864 8732852 54112 472560 count =
5
All 1681232 26376 9557840 218308 pages =
13
Proc: Interrup=
ts
r p d s w Csw Trp Sys Int Sof Flt ioflt 25470 to=
tal
3 206 48 71k 28k 153k 15k 279 7015 1202 cow 11 ua=
rt0 4
1487 zfod 1647 uh=
ci0 16
6.7%Sys 1.9%Intr 4.0%User 0.0%Nice 87.3%Idle 122 ozfod pc=
m0 17
| | | | | | | | | | 8%ozfod ehc=
i0 uhci
=3D=3D=3D+>> daefr =
uhci1 21
73 dtbuf 2063 prcfr 523 uh=
ci3 ehci
Namei Name-cache Dir-cache 485946 desvn 6426 totfr 800 arc=
msr0 30
Calls hits % hits % 163373 numvn react 1065 cp=
u0:timer
8636 8554 99 2 0 121487 frevn pdwak 220 mp=
s0 256
833329 pdpgs 6090 em=
0:rx 0
Disks da0 da1 da2 da3 da4 da5 da6 intrn 5759 em0=
:tx 0
KB/t 11.07 63.91 14.88 20.64 16.49 14.03 0.00 4801680 wire em0=
:link
tps 6 1222 25 81 73 26 0 608344 act 67 em1=
:rx 0
MB/s 0.06 76.26 0.36 1.63 1.17 0.35 0.00 18579164 inact 69 em1=
:tx 0
%busy 0 20 7 38 35 8 0 472216 cache em1=
:link
3224 free ah=
ci0:ch0
1694896 buf ah=
ci0:ch2
541 cp=
u1:timer
744 cp=
u11:time
515 cp=
u2:timer
673 cp=
u10:time
811 cp=
u5:timer
955 cp=
u13:time
473 cp=
u4:timer
442 cp=
u12:time
542 cp=
u3:timer
524 cp=
u8:timer
530 cp=
u6:timer
546 cp=
u9:timer
540 cp=
u7:timer
940 cp=
u14:time
443 cp=
u15:time
Note the "free" and "inact" counts.
The system ought, if I'm reading the sysctl variables correctly, be=20
attempting to aggressively reclaim that memory.
vm.v_free_min: 38595
vm.v_free_target: 130338
vm.v_free_reserved: 8014
vm.v_inactive_target: 195507
vm.v_cache_min: 0
vm.v_cache_max: 0
vm.v_pageout_free_min: 34
vm.swap_enabled: 1
It's not.
In fact, what's happening is that it is (slowly) filling the swap!
There is nothing in the RSS or size of the processes that give=20
indication of a rogue process that has run away with all the memory,=20
never mind that it obviously isn't being touched or it would be in the=20
"active" bucket. Shutting down the user processes (this is a pretty=20
busy box) does not clear any of it out.
Is there a reasonable way to determine who or what has that memory=20
locked up -- and thus why the vm system is not demoting that space into=20
the cache bucket so it can be freed (which, if my understanding is=20
correct, should be happening long before now!)
The system has both ZFS and UFS filesystems on it. I did put the PR on=20
there I submitted to change the ARC cache sizing to dynamic, and it is=20
(correctly) sizing down the ARC cache over time to the point that it is=20
now pinned on the minimum due to low RAM.
This is what I have loaded as modules:
Id Refs Address Size Name
1 36 0xffffffff80200000 166de08 kernel
2 1 0xffffffff8186e000 20400 geom_eli.ko
3 1 0xffffffff8188f000 6dc8 aesni.ko
4 1 0xffffffff81a12000 15af3a zfs.ko
5 1 0xffffffff81b6d000 3847 opensolaris.ko
6 1 0xffffffff81b71000 34d3 ums.ko
7 1 0xffffffff81b75000 7c18 uftdi.ko
8 1 0xffffffff81b7d000 5134 ucom.ko
9 1 0xffffffff81b83000 2a44 uhid.ko
10 2 0xffffffff81b86000 10a42 ipfw.ko
11 1 0xffffffff81b97000 478d ipfw_nat.ko
12 1 0xffffffff81b9c000 daf2 libalias.ko
13 1 0xffffffff81baa000 1f4b daemon_saver.ko
FreeBSD 10.0-STABLE #13 r263037M: Fri Mar 14 14:58:11 CDT 2014=20
karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
--=20
-- Karl
karl@denninger.net
--------------ms010005010301020203050001
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC
BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv
bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5
MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg
RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th
d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W
6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV
jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5
SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY
5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8
Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4
GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci
WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN
nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB
o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg
hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4
+LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG
CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy
bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO
31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c
L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1
YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD
pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE
f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk
YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD
VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2
aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ
KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTYyMDM2MDZaMCMGCSqGSIb3DQEJBDEW
BBQHP6rKNc+kGSFEZftraSONh3Uz4zBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL
BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA
MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV
BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT
EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq
hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG
9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV
BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk
YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh
c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAlyW1O69A1a9vGdIkbUQfeEtZpsIe
RQo1/jOrP0joYTJEoUe/9Bwyj6y/5Q+UunD7Mv46KO4t8oEZwQjWx0CwXtDGiQKeaFyubxDL
zQ+YVKxhBXa21Y2ulDvz7BD/vb5ruNeYEOjiZtncLNM0ZDnIt/8CACdN46CKgZHE5H9JsFhZ
/jSmUarmAPtH4ePzF4lswC+6QR5dUGd2ussr8UMq8AR3u9o3DMIZNCT2yHSWLGpypOdQ7TWt
R0h/vfrVbn8vh7OERzkn1SeS/yRwxjbW8Bfs74uRv4h+m+6Rl6ynqcFkEQglblFI9OHLWQuO
K2pMhXhWvzDdWeCDdkOF72gDt3IkKcD7YOvImpGKbUHIcVQ0OjS3g3UiCbmPmut8nQRTHTeI
JwDN92kQxWibkpLLlc82GplmIerl3QYNXTOIPTcyD6Hh2RRV86rgXzg9ujUSmKZ5BK3J2Z9v
a3b8IbxEI7hH7Ytm1M+CZLpVNwJVD6v57f9mBaDm72bB6yujEkfTdk85lpohRrcEm9zCB3nL
nKT63LAxpCLx+SKpqpPrbpZla6CvxWxN01nS0SA0rrPpsVKAZawaiR5zr68EUuew0gZnESOq
pMjpe09A+YxS9I1r3Mo3YKz9EXDmjBNkPCd70jZMRNeLPeMDVwShAj9BE96NMWPJWjVMa4ix
rcAnhbsAAAAAAAA=
--------------ms010005010301020203050001--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53260B36.2070409>