Date: Sun, 16 Mar 2014 10:56:28 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: OK, so the buffer allocation problem with ZFS is fixed, but now I got this.... (VM management issues) Message-ID: <5325C9AC.3020106@denninger.net>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms030907030401050008070005
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
From this morning.....
3 users Load 2.08 2.40 2.33 Mar 16 10:41
Mem:KB REAL VIRTUAL VN PAGER SWAP PAG=
ER
Tot Share Tot Share Free in out in =
out
Act 2341416 20052 9279140 56272 436064 count 44 =
32
All 3144676 25544 10111364 255624 pages 112 43
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 43 ioflt 29689 to=
tal
3 40 200 45 80k 40k 186k 17k 438 14k 1348 cow 11 uart0 4=
1890 zfod 2085=20
uhci0 16
8.0%Sys 2.3%Intr 5.6%User 0.0%Nice 84.1%Idle 48 ozfod =20
pcm0 17
| | | | | | | | | | 2%ozfod ehci0 uhci
=3D=3D=3D=3D+>>> daefr uhci1 21
55 dtbuf 2123 prcfr 520=20
uhci3 ehci
Namei Name-cache Dir-cache 485946 desvn 8196 totfr 1045=20
arcmsr0 30
Calls hits % hits % 164802 numvn 22 react 1085=20
cpu0:timer
14254 14202 100 121388 frevn pdwak 77 mps0 =
256
1663632 pdpgs 7020=20
em0:rx 0
Disks da0 da1 da2 da3 da4 da5 da6 49 intrn 6864=20
em0:tx 0
KB/t 8.20 63.89 11.91 29.66 34.00 17.36 0.00 4829120 wire =20
em0:link
tps 77 1497 9 19 15 9 0 2117724 act 86 em1:rx 0=
MB/s 0.61 93.43 0.11 0.54 0.51 0.15 0.00 17078072 inact 87 em1:tx=
0
%busy 26 26 1 3 2 1 0 431968 cache =20
em1:link
7832 free =20
ahci0:ch0
1694896 buf =20
ahci0:ch2
655 cpu1:timer
898 cpu11:time
627 cpu2:timer
784 cpu10:time
938 cpu5:timer
1054 cpu13:time
636 cpu4:timer
476 cpu12:time
579 cpu3:timer
702 cpu8:timer
646 cpu6:timer
573 cpu9:timer
670 cpu7:timer
1056 cpu14:time
515 cpu15:time
This is a rather busy (read: extreme demands on the system) time during=20
which I have managed to provoke some really awful behavior, including=20
filesystem stalls. The system in question has both ufs and zfs=20
filesystems (but won't for much longer) along with running both SMB=20
service (samba) and Postgres.
Of note is that nasty "inact" page count. It has driven the adaptive=20
ARC code patch (which is on this box) to trim the ARC cache down to the=20
minimum, where it remains pinned.
My reading of the "inact" page count is that pages shouldn't stay in=20
that state on an indefinite basis - - they should either be reactivated=20
(if they're re-used) or invalidated and moved to the "cache" bucket=20
where the VM code can free them.
Buuuuut.... neither is happening over the space of several hours and a=20
look at the RSS of the working processes doesn't show anything=20
interesting -- or different than normal activity in that regard. 17=20
_*gigabytes*_ of inactive pages (out of 24 GB of RAM in total) and=20
they're not being reclaimed?
Time for me to dig into the vm code?
FreeBSD 10.0-STABLE #13 r263037M: Fri Mar 14 14:58:11 CDT 2014=20
karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
--=20
-- Karl
karl@denninger.net
--------------ms030907030401050008070005
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC
BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv
bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5
MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg
RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th
d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W
6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV
jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5
SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY
5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8
Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4
GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci
WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN
nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB
o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg
hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4
+LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG
CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy
bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO
31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c
L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1
YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD
pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE
f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk
YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD
VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2
aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ
KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTYxNTU2MjhaMCMGCSqGSIb3DQEJBDEW
BBS1GLDmVkM5De90TrlBijuI3XtVzDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL
BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA
MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV
BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT
EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq
hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG
9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV
BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk
YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh
c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAd5yyl8dNfMiH03AiZ0t2MurhEyrk
dBikvzH8O1zuOxUYg1DxK4pqk2Ijf9u6VWUzwv/Qpkd4KmNaTjSuyCCrIgoZXGaBfmuC0H7W
+EMjUIaVc+ZSxH6tEpCtaePQQG/fp//drdFtF1NQacbqLurnclAd5uaFT/fqYHM4PwnfbZha
ySmZu38Qr8ftKTK9FF8xEKl4qT99tvddQgdRe/Q4Yl0B53W/1PC9g/QZ5Y0yOyqQV366h2o2
/pAaWJVfTvSq4iJZEiCGKbIL01lbU+uLNClKFWDAjj3Fm+1T7hTT8PirnxaOHW1eCLM4PqZV
Nd5HWu71BolnnJTbPzLMFoV2DeT9oLdXU2ItaKWlFENxhkAJ0Tl3iocKKjMc04/2rwSt3D13
QkRuDYbiRqeWSaysxdcacg3oWx0QlblfnvsPd7l7wnrP5t5fBzwwMaRW+Vjlw/xUah8VGCQW
77H9aJss38G6JDywes0ZTGZeFJqlBCpMoEl3KAT7jN5yLs2JQmM5AXzhgpe4DWasXlWN/LIQ
m6O46UFfthRucvLRXIGd5dFJnrhEnZKCbsmOqLDrhIa2LODYQvQr17JN9wDqEEh490IJxTHo
DR6Hn1DrYJwcDwqCYigxWWoI98KNlWObROpgvSg+5yrpQKve4jICs1P225OI4eiB7r4YXwJx
05//kaoAAAAAAAA=
--------------ms030907030401050008070005--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5325C9AC.3020106>