From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 18 19:36:11 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C5975DA2 for ; Tue, 18 Mar 2014 19:36:11 +0000 (UTC) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6E1512AA for ; Tue, 18 Mar 2014 19:36:11 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by fs.denninger.net (8.14.8/8.14.8) with ESMTP id s2IJa9Ex025340 for ; Tue, 18 Mar 2014 14:36:10 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [127.0.0.1] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Tue Mar 18 14:36:09 2014 Message-ID: <5328A024.6050901@denninger.net> Date: Tue, 18 Mar 2014 14:36:04 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: John Baldwin , freebsd-hackers@freebsd.org Subject: Re: Tracking down what has inact pages locked up References: <53260B36.2070409@denninger.net> <201403181505.47349.jhb@freebsd.org> In-Reply-To: <201403181505.47349.jhb@freebsd.org> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms010604070004050106070803" X-Antivirus: avast! (VPS 140318-1, 03/18/2014), Outbound message X-Antivirus-Status: Clean Cc: Alan Cox X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 19:36:11 -0000 This is a cryptographically signed message in MIME format. --------------ms010604070004050106070803 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable On 3/18/2014 2:05 PM, John Baldwin wrote: > On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote: >> Is there a reasonable way to determine who or what has that memory >> locked up -- and thus why the vm system is not demoting that space int= o >> the cache bucket so it can be freed (which, if my understanding is >> correct, should be happening long before now!) > I have a hackish thing (for 8.x, might work on 10.x) to let you figure = out > what is using up RAM. This should perhaps go into the base system at s= ome > point. > > Grab the bits at http://people.freebsd.org/~jhb/vm_objects/ > > You will want to build the kld first and use 'make load' to load it. I= t adds > a new sysctl that dumps info about all the VM objects in the system. Y= ou can > then build the 'vm_objects' tool and run it. It can take a while to ru= n if > you have NFS mounts, so I typically save its output to a file first and= then > use sort on the results. sort -n will show you the largest consumer of= RAM, > sort -n -k 3 will show you the largest consumer of inactive pages. Not= e > that 'df' and 'ph' objects are anonymous, and that filename paths aren'= t > always reliable, but this can still be useful. > Thanks. I suspect the cause of the huge inact consumption is a RAM leak in the=20 NAT code in IPFW. It was not occurring in 9.2-STABLE, but is on=20 10.0-STABLE, and reverting to natd in userland stops it -- which=20 pretty-well isolates where it's coming from. --=20 -- Karl karl@denninger.net --------------ms010604070004050106070803 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTgxOTM2MDRaMCMGCSqGSIb3DQEJBDEW BBSEr5bX0fuIwLemVox1b8Rgprf5xDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAZSQNkaXYQd20+UBvbMrU3lHCmbjm dtfDJBGgSHQpeQfJqZ/v8WgmMMgn5nh7PjIwjMkRXBelHhIvxltajuxpLQ56sY2YlI/L0Lxf OHfe3QzI35tyNzxFab7TcGxkYSlOeVE8+zMzIs/7EBdRgz4+6l/BVzhsEWBGMJ4lSb23RAq8 fKdqkItTCJFPIH3laktu8/KtKm7BuaspMqGxkC1CLURUmgNuTNKBifI2pMQfm8UR+FVw0jak FNQXQVdxsjDFuP4AoEZhv1sd9JvYHuewMKfU1iWcDr72Ubo3VB4CTLeIIzrLv3K+Rp9ZXVAc MqfoVVhJTUmfK5Wf5p1tjOriJgW2MkWWqR6fHWIhJjbuBaB7VTfi4Z1FQxOXtsYawwyeRLQ2 sE+7F1uuAX9122GgNlFgBPZ5C1mzUOZ2oYXcQXe8t7shmmDXqH5OioBzi9Rn3h4YNEzOwALA nqqcst956LZqNestHOL17kTETHSDGxO9EDlRzjVOy9YOBbv6Q9FJ0R97qkNKV/2B0KTDyfh5 Hjs8iUvffTcVfnAnDVpbwjXALHFBJpZHx6l7OuigxnxrS8I2Vdx1zam3abOKsFsB/OABYGFA 8o2/CrcDGYDImC0M39NwqqeMdEgRhazJA2kKRTCsq1/jyjy7lPm7aujfr2zMvqOeZ1O/hOgk 91F2w04AAAAAAAA= --------------ms010604070004050106070803--