From owner-freebsd-virtualization@freebsd.org Fri Dec 1 17:43:02 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01BD5E65F26 for ; Fri, 1 Dec 2017 17:43:02 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CB2C569D69 for ; Fri, 1 Dec 2017 17:43:01 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 799C4843D18E for ; Fri, 1 Dec 2017 11:42:59 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRaDdGSJZPUu for ; Fri, 1 Dec 2017 11:42:58 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id EC1F0843D181 for ; Fri, 1 Dec 2017 11:42:58 -0600 (CST) From: Dustin Wenz Content-Type: multipart/signed; boundary="Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: bhyve uses all available memory during IO-intensive operations Date: Fri, 1 Dec 2017 11:42:58 -0600 References: To: freebsd-virtualization@freebsd.org In-Reply-To: Message-Id: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 17:43:02 -0000 --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Here's the top -uS output from a test this morning: last pid: 57375; load averages: 8.29, 7.02, 4.05 = = = = up 38+22:19:14 = 11:28:25 68 processes: 2 running, 65 sleeping, 1 waiting CPU: 0.1% user, 0.0% nice, 40.4% system, 0.4% interrupt, 59.1% idle Mem: 2188K Active, 4K Inact, 62G Wired, 449M Free ARC: 7947M Total, 58M MFU, 3364M MRU, 1000M Anon, 2620M Header, 904M = Other 4070M Compressed, 4658M Uncompressed, 1.14:1 Ratio Swap: 112G Total, 78M Used, 112G Free, 4K In, 12K Out PID UID THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 11 0 24 155 ki31 0K 384K RUN 0 ??? 1446.82% = idle 0 0 644 -16 - 0K 10304K swapin 21 554:59 492.45% = kernel 57333 0 30 20 0 17445M 1325M kqread 9 16:38 357.42% = bhyve 15 0 10 -8 - 0K 192K arc_re 20 80:54 81.55% = zfskern 5 0 6 -16 - 0K 96K - 5 12:35 11.50% cam 12 0 53 -60 - 0K 848K WAIT 21 74:35 9.40% intr 41094 0 30 20 0 17445M 14587M kqread 17 301:29 0.39% = bhyve Dec 1 11:29:31 service014 kernel: pid 57333 (bhyve), uid 0, = was killed: out of swap space Dec 1 11:29:31 service014 kernel: pid 69549 (bhyve), uid 0, = was killed: out of swap space Dec 1 11:29:31 service014 kernel: pid 41094 (bhyve), uid 0, = was killed: out of swap space This was with three VMs running, but only one of them was doing any IO. = Note that the whole machine hung for about about 60 seconds before the = VMs were shut down and memory recovered. That's why the top output is = over a minute older than the kill messages (top had stopped refreshing). What I'm suspicious of is that almost all of the physical memory is = wired. If that is bhyve memory, why did it not page out? - .Dustin > On Nov 30, 2017, at 5:15 PM, Dustin Wenz = wrote: >=20 > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS = is also FreeBSD 11.1). Their sole purpose is to house some medium-sized = Postgres databases (100-200GB). The host system has 64GB of real memory = and 112GB of swap. I have configured each guest to only use 16GB of = memory, yet while doing my initial database imports in the VMs, bhyve = will quickly grow to use all available system memory and then be killed = by the kernel: >=20 > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size = 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size = 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size = 4096, error 12 > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >=20 > The OOM condition seems related to doing moderate IO within the VM, = though nothing within the VM itself shows high memory usage. This is the = chyves config for one of them: >=20 > bargs -A -H -P -S > bhyve_disk_type virtio-blk > bhyve_net_type virtio-net > bhyveload_flags > chyves_guest_version 0300 > cpu 4 > creation Created on Mon Oct 23 16:17:04 CDT = 2017 by chyves v0.2.0 2016/09/11 using __create() > loader bhyveload > net_ifaces tap51 > os default > ram 16G > rcboot 0 > revert_to_snapshot > revert_to_snapshot_method off > serial nmdm51 > template no > uuid 8495a130-b837-11e7-b092-0025909a8b56 >=20 >=20 > I've also tried using different bhyve_disk_types, with no improvement. = How is it that bhyve can use far more memory that I'm specifying? >=20 > - .Dustin --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDExNzQyNThaMCMGCSqGSIb3DQEJBDEW BBSQmDVwsaLi/1A4nK4vJezoM3o3/zCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQCC585LUU2Fuhug7DjxhEWllg0NzDYIUP7o2gEPckgOiT5o0kc3peE6cgiMMZu+Is/wIE1PqzCn lyhOLOje49R7YADBFG+YgFLI5MlHn4P+iEVigja0Ksyu3idyQlPjtN/Fhv/LR6s4I1VTOX2HOhZQ UE5j/yPLKLs9bps6OAg5u+jk1G9bROsqAOFXeSgwZTSfYOGGgS+ijNQ868RT39Mea0RcOMYEQwJY qZUHPWIL+8dSUrrMgkgeCSs9fwiNVwAk35sLiz4YnYdnRyQPqbiA0J9g9aAHtAhoyBDn1UpOQo/X PE/ePATSAOyn1OMwmiQ+5OKoSnohnWQ0DfFvdDhTAAAAAAAA --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761--