Date: Fri, 1 Dec 2017 11:42:58 -0600 From: Dustin Wenz <dustinwenz@ebureau.com> To: freebsd-virtualization@freebsd.org Subject: Re: bhyve uses all available memory during IO-intensive operations Message-ID: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> In-Reply-To: <F4E35CB9-30F9-4C63-B4CC-F8ADC9947E3C@ebureau.com> References: <F4E35CB9-30F9-4C63-B4CC-F8ADC9947E3C@ebureau.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Here's the top -uS output from a test this morning:
last pid: 57375; load averages: 8.29, 7.02, 4.05 =
=
=
=
up 38+22:19:14 =
11:28:25
68 processes: 2 running, 65 sleeping, 1 waiting
CPU: 0.1% user, 0.0% nice, 40.4% system, 0.4% interrupt, 59.1% idle
Mem: 2188K Active, 4K Inact, 62G Wired, 449M Free
ARC: 7947M Total, 58M MFU, 3364M MRU, 1000M Anon, 2620M Header, 904M =
Other
4070M Compressed, 4658M Uncompressed, 1.14:1 Ratio
Swap: 112G Total, 78M Used, 112G Free, 4K In, 12K Out
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU =
COMMAND
11 0 24 155 ki31 0K 384K RUN 0 ??? 1446.82% =
idle
0 0 644 -16 - 0K 10304K swapin 21 554:59 492.45% =
kernel
57333 0 30 20 0 17445M 1325M kqread 9 16:38 357.42% =
bhyve
15 0 10 -8 - 0K 192K arc_re 20 80:54 81.55% =
zfskern
5 0 6 -16 - 0K 96K - 5 12:35 11.50% cam
12 0 53 -60 - 0K 848K WAIT 21 74:35 9.40% intr
41094 0 30 20 0 17445M 14587M kqread 17 301:29 0.39% =
bhyve
Dec 1 11:29:31 <kern.err> service014 kernel: pid 57333 (bhyve), uid 0, =
was killed: out of swap space
Dec 1 11:29:31 <kern.err> service014 kernel: pid 69549 (bhyve), uid 0, =
was killed: out of swap space
Dec 1 11:29:31 <kern.err> service014 kernel: pid 41094 (bhyve), uid 0, =
was killed: out of swap space
This was with three VMs running, but only one of them was doing any IO. =
Note that the whole machine hung for about about 60 seconds before the =
VMs were shut down and memory recovered. That's why the top output is =
over a minute older than the kill messages (top had stopped refreshing).
What I'm suspicious of is that almost all of the physical memory is =
wired. If that is bhyve memory, why did it not page out?
- .Dustin
> On Nov 30, 2017, at 5:15 PM, Dustin Wenz <dustinwenz@ebureau.com> =
wrote:
>=20
> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS =
is also FreeBSD 11.1). Their sole purpose is to house some medium-sized =
Postgres databases (100-200GB). The host system has 64GB of real memory =
and 112GB of swap. I have configured each guest to only use 16GB of =
memory, yet while doing my initial database imports in the VMs, bhyve =
will quickly grow to use all available system memory and then be killed =
by the kernel:
>=20
> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size =
4096, error 12
> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size =
4096, error 12
> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size =
4096, error 12
> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space
>=20
> The OOM condition seems related to doing moderate IO within the VM, =
though nothing within the VM itself shows high memory usage. This is the =
chyves config for one of them:
>=20
> bargs -A -H -P -S
> bhyve_disk_type virtio-blk
> bhyve_net_type virtio-net
> bhyveload_flags
> chyves_guest_version 0300
> cpu 4
> creation Created on Mon Oct 23 16:17:04 CDT =
2017 by chyves v0.2.0 2016/09/11 using __create()
> loader bhyveload
> net_ifaces tap51
> os default
> ram 16G
> rcboot 0
> revert_to_snapshot
> revert_to_snapshot_method off
> serial nmdm51
> template no
> uuid 8495a130-b837-11e7-b092-0025909a8b56
>=20
>=20
> I've also tried using different bhyve_disk_types, with no improvement. =
How is it that bhyve can use far more memory that I'm specifying?
>=20
> - .Dustin
--Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761
Content-Disposition: attachment;
filename=smime.p7s
Content-Type: application/pkcs7-signature;
name=smime.p7s
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw
ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u
ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ
bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA
ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV
UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh
dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue
sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU
xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq
fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k
Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had
sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe
BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax
AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE
BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT
B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG
CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj
FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO
qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4
LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ
nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE
xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND
EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx
EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl
YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3
DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx
CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDExNzQyNThaMCMGCSqGSIb3DQEJBDEW
BBSQmDVwsaLi/1A4nK4vJezoM3o3/zCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV
UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1
cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI
hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL
MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO
BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv
bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC
AQCC585LUU2Fuhug7DjxhEWllg0NzDYIUP7o2gEPckgOiT5o0kc3peE6cgiMMZu+Is/wIE1PqzCn
lyhOLOje49R7YADBFG+YgFLI5MlHn4P+iEVigja0Ksyu3idyQlPjtN/Fhv/LR6s4I1VTOX2HOhZQ
UE5j/yPLKLs9bps6OAg5u+jk1G9bROsqAOFXeSgwZTSfYOGGgS+ijNQ868RT39Mea0RcOMYEQwJY
qZUHPWIL+8dSUrrMgkgeCSs9fwiNVwAk35sLiz4YnYdnRyQPqbiA0J9g9aAHtAhoyBDn1UpOQo/X
PE/ePATSAOyn1OMwmiQ+5OKoSnohnWQ0DfFvdDhTAAAAAAAA
--Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9C202C72-EF74-4DB7-9B2C-736C049A9F7A>