From owner-freebsd-stable@FreeBSD.ORG Sun Mar 16 15:56:45 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D714DFB for ; Sun, 16 Mar 2014 15:56:45 +0000 (UTC) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DE9ED801 for ; Sun, 16 Mar 2014 15:56:43 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by fs.denninger.net (8.14.8/8.14.8) with ESMTP id s2GFuXbO065582 for ; Sun, 16 Mar 2014 10:56:34 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [127.0.0.1] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Sun Mar 16 10:56:34 2014 Message-ID: <5325C9AC.3020106@denninger.net> Date: Sun, 16 Mar 2014 10:56:28 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: OK, so the buffer allocation problem with ZFS is fixed, but now I got this.... (VM management issues) Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms030907030401050008070005" X-Antivirus: avast! (VPS 140316-0, 03/16/2014), Outbound message X-Antivirus-Status: Clean X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 15:56:45 -0000 This is a cryptographically signed message in MIME format. --------------ms030907030401050008070005 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable From this morning..... 3 users Load 2.08 2.40 2.33 Mar 16 10:41 Mem:KB REAL VIRTUAL VN PAGER SWAP PAG= ER Tot Share Tot Share Free in out in = out Act 2341416 20052 9279140 56272 436064 count 44 = 32 All 3144676 25544 10111364 255624 pages 112 43 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 43 ioflt 29689 to= tal 3 40 200 45 80k 40k 186k 17k 438 14k 1348 cow 11 uart0 4= 1890 zfod 2085=20 uhci0 16 8.0%Sys 2.3%Intr 5.6%User 0.0%Nice 84.1%Idle 48 ozfod =20 pcm0 17 | | | | | | | | | | 2%ozfod ehci0 uhci =3D=3D=3D=3D+>>> daefr uhci1 21 55 dtbuf 2123 prcfr 520=20 uhci3 ehci Namei Name-cache Dir-cache 485946 desvn 8196 totfr 1045=20 arcmsr0 30 Calls hits % hits % 164802 numvn 22 react 1085=20 cpu0:timer 14254 14202 100 121388 frevn pdwak 77 mps0 = 256 1663632 pdpgs 7020=20 em0:rx 0 Disks da0 da1 da2 da3 da4 da5 da6 49 intrn 6864=20 em0:tx 0 KB/t 8.20 63.89 11.91 29.66 34.00 17.36 0.00 4829120 wire =20 em0:link tps 77 1497 9 19 15 9 0 2117724 act 86 em1:rx 0= MB/s 0.61 93.43 0.11 0.54 0.51 0.15 0.00 17078072 inact 87 em1:tx= 0 %busy 26 26 1 3 2 1 0 431968 cache =20 em1:link 7832 free =20 ahci0:ch0 1694896 buf =20 ahci0:ch2 655 cpu1:timer 898 cpu11:time 627 cpu2:timer 784 cpu10:time 938 cpu5:timer 1054 cpu13:time 636 cpu4:timer 476 cpu12:time 579 cpu3:timer 702 cpu8:timer 646 cpu6:timer 573 cpu9:timer 670 cpu7:timer 1056 cpu14:time 515 cpu15:time This is a rather busy (read: extreme demands on the system) time during=20 which I have managed to provoke some really awful behavior, including=20 filesystem stalls. The system in question has both ufs and zfs=20 filesystems (but won't for much longer) along with running both SMB=20 service (samba) and Postgres. Of note is that nasty "inact" page count. It has driven the adaptive=20 ARC code patch (which is on this box) to trim the ARC cache down to the=20 minimum, where it remains pinned. My reading of the "inact" page count is that pages shouldn't stay in=20 that state on an indefinite basis - - they should either be reactivated=20 (if they're re-used) or invalidated and moved to the "cache" bucket=20 where the VM code can free them. Buuuuut.... neither is happening over the space of several hours and a=20 look at the RSS of the working processes doesn't show anything=20 interesting -- or different than normal activity in that regard. 17=20 _*gigabytes*_ of inactive pages (out of 24 GB of RAM in total) and=20 they're not being reclaimed? Time for me to dig into the vm code? FreeBSD 10.0-STABLE #13 r263037M: Fri Mar 14 14:58:11 CDT 2014=20 karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP --=20 -- Karl karl@denninger.net --------------ms030907030401050008070005 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTYxNTU2MjhaMCMGCSqGSIb3DQEJBDEW BBS1GLDmVkM5De90TrlBijuI3XtVzDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAd5yyl8dNfMiH03AiZ0t2MurhEyrk dBikvzH8O1zuOxUYg1DxK4pqk2Ijf9u6VWUzwv/Qpkd4KmNaTjSuyCCrIgoZXGaBfmuC0H7W +EMjUIaVc+ZSxH6tEpCtaePQQG/fp//drdFtF1NQacbqLurnclAd5uaFT/fqYHM4PwnfbZha ySmZu38Qr8ftKTK9FF8xEKl4qT99tvddQgdRe/Q4Yl0B53W/1PC9g/QZ5Y0yOyqQV366h2o2 /pAaWJVfTvSq4iJZEiCGKbIL01lbU+uLNClKFWDAjj3Fm+1T7hTT8PirnxaOHW1eCLM4PqZV Nd5HWu71BolnnJTbPzLMFoV2DeT9oLdXU2ItaKWlFENxhkAJ0Tl3iocKKjMc04/2rwSt3D13 QkRuDYbiRqeWSaysxdcacg3oWx0QlblfnvsPd7l7wnrP5t5fBzwwMaRW+Vjlw/xUah8VGCQW 77H9aJss38G6JDywes0ZTGZeFJqlBCpMoEl3KAT7jN5yLs2JQmM5AXzhgpe4DWasXlWN/LIQ m6O46UFfthRucvLRXIGd5dFJnrhEnZKCbsmOqLDrhIa2LODYQvQr17JN9wDqEEh490IJxTHo DR6Hn1DrYJwcDwqCYigxWWoI98KNlWObROpgvSg+5yrpQKve4jICs1P225OI4eiB7r4YXwJx 05//kaoAAAAAAAA= --------------ms030907030401050008070005--