Date: Thu, 18 Aug 2016 15:31:26 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: ZFS ARC under memory pressure Message-ID: <c3bc6c5a-961c-e3a4-2302-f0f7417bc34f@denninger.net> In-Reply-To: <20160818202657.GS8192@zxy.spb.ru> References: <20160816193416.GM8192@zxy.spb.ru> <8dbf2a3a-da64-f7f8-5463-bfa23462446e@FreeBSD.org> <20160818202657.GS8192@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms020806060003040608060704 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 8/18/2016 15:26, Slawa Olhovchenkov wrote: > On Thu, Aug 18, 2016 at 11:00:28PM +0300, Andriy Gapon wrote: > >> On 16/08/2016 22:34, Slawa Olhovchenkov wrote: >>> I see issuses with ZFS ARC inder memory pressure. >>> ZFS ARC size can be dramaticaly reduced, up to arc_min. >>> >>> As I see memory pressure event cause call arc_lowmem and set needfree= : >>> >>> arc.c:arc_lowmem >>> >>> needfree =3D btoc(arc_c >> arc_shrink_shift); >>> >>> After this, arc_available_memory return negative vaules (PAGESIZE * >>> (-needfree)) until needfree is zero. Independent how too much memory >>> freed. needfree set to 0 in arc_reclaim_thread(), when arc_size <=3D >>> arc_c. Until arc_size don't drop below arc_c (arc_c deceased at every= >>> loop interation). >>> >>> arc_c droped to minimum value if arc_size fast enough droped. >>> >>> No control current to initial memory allocation. >>> >>> As result, I can see needless arc reclaim, from 10x to 100x times. >>> >>> Can some one check me and comment this? >> You might have found a real problem here, but I am short of time right= now to >> properly analyze the issue. I think that on illumos 'needfree' is a v= ariable >> that's managed by the virtual memory system and it is akin to our >> vm_pageout_deficit. But during the porting it became an artificial va= lue and >> its handling might be sub-optimal. > As I see, totaly not optimal. > I am create some patch for sub-optimal handling and now test it. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" You might want to look at the code contained in here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 There are some ugly interactions with the VM system you can run into if you're not careful; I've chased this issue before and while I haven't yet done the work to integrate it into 11.x (and the underlying code *has* changed since the 10.x patches I developed) if you wind up driving the VM system to evict pages to swap rather than pare back ARC you're probably making the wrong choice. In addition UMA can come into the picture too and (at least previously) was a severe contributor to pathological behavior. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020806060003040608060704 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA4MTgyMDMxMjZaME8GCSqGSIb3DQEJBDFCBEAv nKKhxLndyPUzBHmLmYJfFxZ6B48eRagCuHF7H9YjfaCFQq0ZxSIvaajnjktqNcWyqQbfWIvd fEewlEUa35snMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIALszJzrwg 0ll1oeKkSiMgZsap90O6BhYKSp2DflnUELzShod/ya7uo1yqQmavyfSHk+Wdm0soqRUfbdcs pfvTInNpmKIhShrURrLCFNIQyXc5x+gvrtrsXZbTleUYAIZqU38b6ipCshOIBcDXtzWKT7yn zojAu8wi01V3rcB9L/NdcPHNDtemEq0n5mbCGuukGNnKUzLKOfBIT5OaCKFJIq61Sh60uz+v TPKudH3l7+04fiS3WBSN8z3LP9J3MjRckUszvH+abFi5X4d9NKn1gfDXiGZ0ketn68x3fKu+ f2LMCXNIXbIGWL6iSNiavPR7h5Bx172LsN/PJMtcCU7rT0Ctd9I41lOCCayEtZ24AewBL3uU kMzv/24LMEqpgXqQ6N0Uw9qC0R4KGL4LijO06uop0ZBZcOUECTwEdj6UYPPRLkGHYdIOzqla zYD13MTpdou58qn5KANHq8VS5iJ4A2axET7Niddj4Eu4K72FLS2DjIrrH+1KXtEcPgUxKm8X 1yGhALgdgvIU8Pbs/+YccPmCPX5KUDBOkrW/wcHPytNYKBFWnLlM3cvmSOLnTSVjeNnbpyKQ jxCvGpV6UfQBNBmBvHw6mh/lpnmwt55Djs2xA8oEnZa37Vg86RpXuzy9IZI0s6Rfz6B7ODuO hvplmtx7ZtTHCMe+I4HjXQMVxnsAAAAAAAA= --------------ms020806060003040608060704--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c3bc6c5a-961c-e3a4-2302-f0f7417bc34f>