From owner-freebsd-stable@FreeBSD.ORG Sat Mar 15 02:11:53 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 31C36DCF for ; Sat, 15 Mar 2014 02:11:53 +0000 (UTC) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EE139BC2 for ; Sat, 15 Mar 2014 02:11:51 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by fs.denninger.net (8.14.8/8.14.8) with ESMTP id s2F2BjbQ023142 for ; Fri, 14 Mar 2014 21:11:45 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [127.0.0.1] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Fri Mar 14 21:11:45 2014 Message-ID: <5323B6DC.7020205@denninger.net> Date: Fri, 14 Mar 2014 21:11:40 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Two odd problems with STABLE-10 r262921 [ZFS ARC memory allocation / "freeze" problems -- RESOLVED] References: <531F14A7.9090103@denninger.net> In-Reply-To: <531F14A7.9090103@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms050600020105040909000403" X-Antivirus: avast! (VPS 140314-1, 03/14/2014), Outbound message X-Antivirus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2014 02:11:53 -0000 This is a cryptographically signed message in MIME format. --------------ms050600020105040909000403 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable On 3/11/2014 8:50 AM, Karl Denninger wrote: > > The other issue is that I am running into bizarre wired memory behavior= =2E I > have ZFS on the box and it was wiring down ~20GB, driving the system in= to > paging. I dropped arc_max to 16GB as a hard limit, it's currently show= ing > 15GB, but... > > 20531152 wire > > zfs-stats -A shows: > > > -----------------------------------------------------------------------= - > ZFS Subsystem Report Tue Mar 11 08:27:55 201= 4 > -----------------------------------------------------------------------= - > > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > > ARC Misc: > Deleted: 4.22m > Recycle Misses: 57.10k > Mutex Misses: 320 > Evict Skips: 2.35m > > ARC Size: 95.96% 15.35 GiB > Target Size: (Adaptive) 100.00% 16.00 GiB > Min Size (Hard Limit): 12.50% 2.00 GiB > Max Size (High Water): 8:1 16.00 GiB > > ARC Size Breakdown: > Recently Used Cache Size: 75.49% 12.08 GiB > Frequently Used Cache Size: 24.51% 3.92 GiB > > ARC Hash Breakdown: > Elements Max: 1.59m > Elements Current: 55.53% 883.80k > Collisions: 5.84m > Chain Max: 16 > Chains: 261.48k > > -----------------------------------------------------------------------= - > > Ok, where's the rest coming from? There's nothing in the shared memory= > segment list that appears to be related. > > Is there a memory leak in here somewhere? As posted to the fs mailing list I have found and corrected this issue;=20 the ARC eviction code has issues. Please pardon the apparently-reversed patch I submitted; it's Friday, I=20 didn't have enough coffee and Murphy got me twice while putting that=20 together (the first time when I pulled the diff off without committing=20 the last change I had made back to my local tree first!) If you're having issues with your system stalling while running ZFS=20 filesystems and it appears you're being driven into paging when you=20 shouldn't be (in other words, the ARC cache is consuming all of your RAM = and not releasing it as other things demand memory) this should fix it. = It also includes a run-time tunable via sysctl (defaults to 25%, which=20 was the original intent of the code that I replaced) for the percentage=20 of memory to target as free, below which the ARC will be asked to=20 release RAM. See http://www.freebsd.org/cgi/query-pr.cgi?pr=3D187594 --=20 -- Karl karl@denninger.net --------------ms050600020105040909000403 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTUwMjExNDBaMCMGCSqGSIb3DQEJBDEW BBRCppqEOPTZqEvdiHYgl+VI9UbqUjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAPjRoQ0XyenxCXyig02BakSGLkdEN l5jZrMu6aHW/DMjsZRgEtjwpA9GVlvrTG1im32RWTp5a8nAz09k8wKvqtKT9NLNrz2Xcip8B iDFKzBd0Kvg0pkNPKMkh3GT4RhEA+2EGoms/XZLD9lxRJ6ewWBzG5zyw5cdLq+oC+WXQYFQC Bi0xmeXapG5nIOY+LZKmedc9wkXFWo8D0fLSFFcEuntvOl/8YUsJfgOA4g8AOdNxv+11KMde c4x0OmhNbaAPei4L3fAdoLKy00Lw0ouY7xOeyC9M2g9cAmSx0zAycST8N8bE326puDogy5Ed zvwExx/CkchNC8nZ0hDVC3+vmVlm5b3FNGEavUv/hGtMGKJ+ce9Wltf4WgujBM4m/OBjyFXa tdxUwx2xcF0gOPy6XvGHOWSVvncCUBYkdkNmqtrf1nhNTqUE/LQOvUnm60zmJNP7kMbm30gG rTkTyqMntH/Z93ZuqazV/nIxgLJDl1x/eg5q6+pYQawAyRpS3LoLasJoRdormVzlMwIcLxZz oGg0AONxT4hyBIyHbRBFZyXkxxlYiLv9RDti3YxIX1uR8Lq6gewfVHKU+Swiv1/Dy/Q3JR1S jzveIK1z9wm2Hns2PAmNg5rR0+vKCEnGYYttDWEEQshunsA1Z8H995wp7JfwXuB95Vwb9ylp 88eBUbIAAAAAAAA= --------------ms050600020105040909000403--