From nobody Mon Jun 26 11:49:55 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QqR3j5BB0z4jGgf; Mon, 26 Jun 2023 11:50:09 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QqR3h3pTTz3GwN; Mon, 26 Jun 2023 11:50:08 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=CYLLj2Ev; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::233 as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2b5c231c23aso26441891fa.0; Mon, 26 Jun 2023 04:50:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687780206; x=1690372206; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=papG/2UPNAqsDItiGfX6Pws5dLghqjiR/FzqJmikTyg=; b=CYLLj2EvRBMkzwyhdwXk+c5LlCixPckRUbQyJQdvBfQ8m/2DPmC/Mqc1XvsxUdfVCR yvK8Kt7rG8/ZR5vX/cMm6sAQS1fOoR3u5z5jVsP0co/hMkJOwuJg/5v6HAcFbKuYQywA ajcftujS1IvGX9oQ2jGggOmI1PyxQZrU/eoUEsRnwudhnAhqt1uLSPTlaAUptKY2IEqd 2NqIN/1tk1Ix0ZGW4G+XmPX3IQaHp+WXAl2y9MuhmBkCZP1VoBGAFhlwbvT8ufXZ4udG 1bCYCJdVx8Et5yxR5ar340DTjUShRzF7/q5AppD1ygUkIw5tW3lIXZo7t7FWExm7lgh3 skHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687780206; x=1690372206; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=papG/2UPNAqsDItiGfX6Pws5dLghqjiR/FzqJmikTyg=; b=ilnUhUuNIIGZiWu31VrQb4yQILa+HE74v/mO+gfBvEh7Ypm1eS4zQTqqm2/PncCPZB axwizz7bnmj16DZnfWowbf5NYmLA/uVN8vsE8gX5elPHHbAQXN+8l9E6VR7bhSWv1swB DfRNTfIfXmRlPyZF7sH81Ufp7SUNSQpxZd68Whk0gGK33+jg95L1fNSaG5smI/vAkPdU suezOx030Ia3HiI/Hnr9UWm1/CYJLOm5f6nn+Mx1XA5a9QodAFcoMU/wim+2AJASTJrY 1uMZHmLYpfHOD/UixgXzxvE5YjNHYZZGIE/exMw2OTbvpLC0iug6bCq7U4+ISjY4nSpY P8bw== X-Gm-Message-State: AC+VfDyWMWi4SewAah+fRv0AnB1bJ2RX0bSB44kaWBCaKZ0SQzP3Q37m JwKCtQviODOi26pRrhEmLZU= X-Google-Smtp-Source: ACHHUZ4L5KisfJ43D9oaayM3XTWlmSvozI1PY89EEv93oPkQW1LHl6W7jjAQzCxwkyv3UEPGy6g/RQ== X-Received: by 2002:a2e:83d0:0:b0:2b4:792d:a4b5 with SMTP id s16-20020a2e83d0000000b002b4792da4b5mr12568472ljh.33.1687780206176; Mon, 26 Jun 2023 04:50:06 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id m18-20020a2eb6d2000000b002b2044a4d10sm1190278ljo.57.2023.06.26.04.50.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Jun 2023 04:50:05 -0700 (PDT) From: Vitaliy Gusev Message-Id: <48D9FE17-B885-40AC-8958-EA90EEB9D8B6@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_F9A90480-54B2-4065-8807-AE4CB16B967D" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: bhyve: how to keep the host from starving the guest Date: Mon, 26 Jun 2023 14:49:55 +0300 In-Reply-To: Cc: FreeBSD virtualization , FreeBSD Mailing List To: Aryeh Friedman References: X-Mailer: Apple Mail (2.3731.600.7) X-Spamd-Result: default: False [-2.21 / 15.00]; NEURAL_HAM_SHORT(-0.96)[-0.962]; NEURAL_HAM_MEDIUM(-0.96)[-0.962]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; NEURAL_SPAM_LONG(0.22)[0.215]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::233:from]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-questions@freebsd.org] X-Rspamd-Queue-Id: 4QqR3h3pTTz3GwN X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_F9A90480-54B2-4065-8807-AE4CB16B967D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Aryeh, Have you wired a guest memory with bhyve's -S option? -S Wire guest memory Anyway, OS does not have another choice than kill a process to free some = memory when RAM+swap is fully used (assume kernel already scanned Inactive = memory). As recommendation: Look at an another memory consumers like ZFS, another processes Increase swap Tune vm.overcommit sysctl. See tuning(7) for details. So in short, there is no good way to run applications that fully use 10 = GB memory on a systems with just 1 GB RAM + 1 GB swap. You should have enough resources to do that. And it would be nice, if you provide more information and metrics for = your system, total RAM, memory assigned for VM, additional statistics for = all intensive processes (SIZE, RES), etc. =E2=80=94 Vitaliy Gusev > On 26 Jun 2023, at 13:44, Aryeh Friedman = wrote: >=20 > I am a 12 core machine that I want allocate only 4 CPU's to the host > and 8 to a VM (the host is my desktop FreeBSD machine and the guest is > debian 11 used for playing around with learning AI model making).... I > have passed my GeForce 1030 (bottom of the line GPU for AI work it > seems) but since it appears that no one can get tensorflow, pytorch or > anything else that runs ANN's on a GPU to work on FreeBSD (I have > tracked down to the fact nvidia never ported CUDA to FreeBSD)... the > problem is sometimes during heavy work on the guest then the host > slows down and if the the host is doing heavy work (especially > resource intensive things like compiling lang/rust) that the host will > kill the guest if it runs out of available memory+swap (speaking of > that does wiring the memory at least prevent this?) >=20 > --=20 > Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org >=20 --Apple-Mail=_F9A90480-54B2-4065-8807-AE4CB16B967D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Aryeh,

Have you wired a guest memory with bhyve's = -S option?

-S  =         Wire guest = memory

Anyway, OS does not = have another choice than kill a process to free some = memory
when RAM+swap is fully used (assume kernel already = scanned Inactive memory).

As = recommendation:
  • Look at an = another memory consumers like ZFS, another processes
  • Increase = swap
  • Tune vm.overcommit sysctl. See tuning(7) = for details.

So in short, there is no good = way to run applications that fully use 10 GB
memory on a = systems with just 1 GB RAM + 1 GB swap. You should have
enough = resources to do that.

And it would be = nice, if you provide more information and metrics for = your
system, total RAM, memory assigned for VM, =  additional statistics for all
intensive processes (SIZE, = RES), etc.

=E2=80=94
Vitaliy = Gusev

On 26 Jun = 2023, at 13:44, Aryeh Friedman <aryeh.friedman@gmail.com> = wrote:

I am a 12 = core machine that I want allocate only 4 CPU's to the host
and 8 to a = VM (the host is my desktop FreeBSD machine and the guest is
debian 11 = used for playing around with learning AI model making).... I
have = passed my GeForce 1030 (bottom of the line GPU for AI work it
seems) = but since it appears that no one can get tensorflow, pytorch = or
anything else that runs ANN's on a GPU to work on FreeBSD (I = have
tracked down to the fact nvidia never ported CUDA to FreeBSD)... = the
problem is sometimes during heavy work on the guest then the = host
slows down and if the the host is doing heavy work = (especially
resource intensive things like compiling lang/rust) that = the host will
kill the guest if it runs out of available memory+swap = (speaking of
that does wiring the memory at least prevent = this?)

--
Aryeh M. Friedman, Lead Developer, = http://www.PetiteCloud.org


= --Apple-Mail=_F9A90480-54B2-4065-8807-AE4CB16B967D--