From owner-freebsd-virtualization@freebsd.org Fri Dec 1 11:23:19 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09B8ADFCCB9 for ; Fri, 1 Dec 2017 11:23:19 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lf0-x231.google.com (mail-lf0-x231.google.com [IPv6:2a00:1450:4010:c07::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E5397D46A; Fri, 1 Dec 2017 11:23:18 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lf0-x231.google.com with SMTP id t197so11323075lfe.7; Fri, 01 Dec 2017 03:23:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=oobhfC0mhEzq3bPOHrPgc3CODSRtORePKAlM3n9DeGY=; b=NjpdjgJApQByaHsu71HAcl6N3+4yxLJhil6INLOJglMrXcGbd/mjwCNRYQxV0slYX+ UbTuCm5vyVAhnETrJbNzfE+c9lPyDG6wejJ2QxnZ2mqgYrIB3diFuY9uuRaVvyr+pisp HiCn8oFZwlRhU+KcBET43F//FIdKidx6hiRce6naV0MdZAoHbhC9706WFGG9BWSym6pD 31gmMoHleoBkyZ8wHlhspNElc0YojKTuGGpMGuiFUM/szMigAzb4PbKU5xyOKH7zBdLx xxT1tzznWRgh/8V/7Uar6/Sr6Ji2+r/wTQNzzc4II447clo9/Rzg/Y5crQzBox/jhWd1 dl+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=oobhfC0mhEzq3bPOHrPgc3CODSRtORePKAlM3n9DeGY=; b=JMKUvyRjD7I9nJOR7dfuu2iCOk7I6bsWGgEBjQz3AcSHGyr1ghJ2ClntGkYkkXNZWw mcE0hcSVv+OZRw8CmW7YITMBkUwMSZHi7tTi90vu+WhsURWp0MfHKv+rIB2bpouxOffj 2g5wpELAuevSG2zdj24TO4nPAikb4+Gt6cv4HM0lKO0kKNTZ7KlilMsgpm/wBBzo2cZc b9EJEoXs0D2g46urrkoeVTf+3GQb17hDUVQXlM+u9xcvijoFcsUZEXDTbbb+U5k5ea7J 08VDXr/1EFzDaUWOj+Xie9fCf8gkUV6yFa6crkam35LosC0qthxXCxecoz+IjpTLNTA+ dt3g== X-Gm-Message-State: AJaThX5uomYR8Ae5WYrbvJq1uB+P0P1/lXJ3CCFXtxcAmiYzPV/qpVux qg9GZIRFwvn/wbPvfoAogn9TBzswYOW9TkXk8xcv8Q== X-Google-Smtp-Source: AGs4zMbt4d11wu2hNjM3mSu2Tnqo5PYT8BplR/POrs8Cbb3vFNQOvvdZ4pCndq1J/BCKMS/j5kk0s15eac6oSfGWDdI= X-Received: by 10.25.79.67 with SMTP id a3mr3810174lfk.162.1512127396300; Fri, 01 Dec 2017 03:23:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.168.197 with HTTP; Fri, 1 Dec 2017 03:23:15 -0800 (PST) Reply-To: araujo@freebsd.org In-Reply-To: References: From: Marcelo Araujo Date: Fri, 1 Dec 2017 19:23:15 +0800 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Shane Ambler Cc: Allan Jude , Dustin Wenz , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 11:23:19 -0000 2017-12-01 17:53 GMT+08:00 Shane Ambler : > On 01/12/2017 13:43, Allan Jude wrote: > > On 2017-11-30 22:10, Dustin Wenz wrote: > >> I am using a zvol as the storage for the VM, and I do not have any ARC > >> limits set. However, the bhyve process itself ends up grabbing the vas= t > >> majority of memory.=C3=82 > >> > >> I=C3=A2=E2=82=AC=E2=84=A2ll run a test tomorrow to get the exact outpu= t from top. > >> > >> =C3=82 =C3=82 - .Dustin > >> > >> On Nov 30, 2017, at 5:28 PM, Allan Jude >> > wrote: > >> > >>> On 11/30/2017 18:15, Dustin Wenz wrote: > >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest > >>>> OS is also FreeBSD 11.1). Their sole purpose is to house some > >>>> medium-sized Postgres databases (100-200GB). The host system has 64G= B > >>>> of real memory and 112GB of swap. I have configured each guest to > >>>> only use 16GB of memory, yet while doing my initial database imports > >>>> in the VMs, bhyve will quickly grow to use all available system > >>>> memory and then be killed by the kernel: > >>>> > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1735,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1610,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1763,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: pid 41123 (bhyve), uid 0, was killed: out of = swap space > > That's the type of errors I see when wired jumps high. I'm not seeing > this from bhyve but when your watching top, keep an eye on the wired > amount. > > -- > FreeBSD - the place to B...Sharing Devices > > Shane Ambler > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > As Allan said, it is likely to be your ARC cache holding memory and unfortunately right now all these swap out doesn't play well with the combination of zfs + bhyve. Try to tune your vfs.zfs.arc_max to a minimum where you give memory space enough to your VM. What I'm doing now is, launch a VM get the amount of memory and remove it from vfs.zfs.arc_max, as soon as the VM stops, I give the memory back to vfs.zfs.arc_max. Best, --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_)