Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 19 Mar 2014 15:05:18 -0500
From:      Karl Denninger <karl@denninger.net>
To:        freebsd-hackers@freebsd.org
Subject:   Re: Tracking down what has inact pages locked up
Message-ID:  <5329F87E.1010203@denninger.net>
In-Reply-To: <201403191529.06567.jhb@freebsd.org>
References:  <53260B36.2070409@denninger.net> <201403181730.02471.jhb@freebsd.org> <532910D1.3010704@denninger.net> <201403191529.06567.jhb@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.

--------------ms050104020404010300040109
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable


On 3/19/2014 2:29 PM, John Baldwin wrote:
> On Tuesday, March 18, 2014 11:36:49 pm Karl Denninger wrote:
>> On 3/18/2014 4:30 PM, John Baldwin wrote:
>>> On Tuesday, March 18, 2014 3:36:04 pm Karl Denninger wrote:
>>>> On 3/18/2014 2:05 PM, John Baldwin wrote:
>>>>> On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote:
>>>>>> Is there a reasonable way to determine who or what has that memory=

>>>>>> locked up -- and thus why the vm system is not demoting that space=
 into
>>>>>> the cache bucket so it can be freed (which, if my understanding is=

>>>>>> correct, should be happening long before now!)
>>>>> I have a hackish thing (for 8.x, might work on 10.x) to let you fig=
ure out
>>>>> what is using up RAM.  This should perhaps go into the base system =
at some
>>>>> point.
>>>>>
>>>>> Grab the bits at http://people.freebsd.org/~jhb/vm_objects/
>>>>>
>>>>> You will want to build the kld first and use 'make load' to load it=
=2E  It adds
>>>>> a new sysctl that dumps info about all the VM objects in the system=
=2E  You can
>>>>> then build the 'vm_objects' tool and run it.  It can take a while t=
o run if
>>>>> you have NFS mounts, so I typically save its output to a file first=
 and then
>>>>> use sort on the results.  sort -n will show you the largest consume=
r of RAM,
>>>>> sort -n -k 3 will show you the largest consumer of inactive pages. =
 Note
>>>>> that 'df' and 'ph' objects are anonymous, and that filename paths a=
ren't
>>>>> always reliable, but this can still be useful.
>>>>>
>>>> Thanks.
>>>>
>>>> I suspect the cause of the huge inact consumption is a RAM leak in t=
he
>>>> NAT code in IPFW.  It was not occurring in 9.2-STABLE, but is on
>>>> 10.0-STABLE, and reverting to natd in userland stops it -- which
>>>> pretty-well isolates where it's coming from.
>>> Memory for in-kernel NAT should be wired pages, not inactive.
>> Yeah, should be. :-)
>>
>> But..... it managed to lock up 19GB of the 24GB the system has in inac=
t
>> pages over 12 hours, and dropping the system to single user and
>> unloading the modules did not release the RAM...... which is why the
>> question (on how to track down what the hell is going on.)
>>
>> Changing the config back to natd as opposed to in-kernel NAT, however,=

>> made the problem disappear.
> It would be useful to run the program I posted above to see what is tyi=
ng
> up in active pages.  It is not going to be wired kernel memory.  You ma=
y
> simply be seeing a different aritfact that natd causes additional memor=
y
> pressure so pagedaemon purges inactive pages more often.  If you aren't=

> using memory, then having a lot of inactive pages isn't a problem, it
> means the system will be able to satisfy potential future reads without=

> needing to go to disk.
>
> What I have done in places where I want to limit inactive memory is to
> write a simple program that invokes posix_fadvise(POSIX_FADV_DONTNEED)
> on each file to flush it from inactive to cache.  You may also need an
> fsync() on each file to flush any dirty pages before the fadvise.
What caught my attention originally was that swap was rising in=20
consumption at the same time inact was basically pegged (and=20
performance, as expected, was in the trashcan.)

I'll see what I can track down.

--=20
-- Karl
karl@denninger.net



--------------ms050104020404010300040109
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC
BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv
bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5
MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg
RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th
d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W
6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV
jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5
SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY
5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8
Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4
GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci
WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN
nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB
o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg
hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4
+LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG
CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy
bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO
31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c
L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1
YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD
pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE
f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk
YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD
VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2
aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ
KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTkyMDA1MThaMCMGCSqGSIb3DQEJBDEW
BBTfgwgsxdCQdy5L/zSZuFfH5re0SjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL
BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA
MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV
BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT
EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq
hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG
9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV
BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk
YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh
c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAM2TYYaoO8DnBppiVZEPW3wBibegI
TG/UN+f7CTuiH1sPdzm0rlvD4Sdffb6wDbeF2MjX5k9bkr51AntJqBHY1gMcduDCPWnqtv2x
ODrcxBpI+l5qS2nMgyoyTjbubjGrGHh/6a9AUOBDAzHur1YcwGQeVr0H1WZDINbANgZNcgvW
MunexwOHd3V+9Xh/WaZHbvPd6UfgLaQpQosKWFhno00FOKjWMIJ0lUKyO2pQEvhwMdGyRb9N
o79pKBzWVnq2YROJ0ih4BhQPMS4WGxqHfJBPeVNHM5e5Q6dfOWJuQjPEaAq2/mz8xBWodGBm
CVpZYXx90s9RPXZmtxckGB/ONOs20exJdQ88BK9XUZqSuNOYosFpYtMsCiyOX8gdu5W8kL2f
svk5E9NQY1i8di3YPN52r/HHqr57WzVvltaBXItNVhQvD1UVPq4lTp6/AWY2JhxaYD7jXEz1
mXTK2yqomS+VaZuGkbFl2DLnW+cOSqGb7qX+OQYEL02v4hHoK7iaDc2T9ku/Fi6MXhwND0LM
2dkHgeNh3S7a4CgBwJYZZFXsi+h2cNIBSVHCDymY/F/wyikrVcTOyT7xjOSQiTRK68DQrI15
SyW7xSeANVn3A9iZsO5GsZvNvNRVj/moK2mQrxxNy7QUQfWy3e8Y4DJSv6Vuz/0es9xrJYgz
V6UDEfMAAAAAAAA=
--------------ms050104020404010300040109--





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5329F87E.1010203>