From owner-freebsd-stable@freebsd.org Tue Aug 13 05:28:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ECE93CC01C for ; Tue, 13 Aug 2019 05:28:42 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4671Tk5m8Xz4RVs for ; Tue, 13 Aug 2019 05:28:42 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id C3E21CC01B; Tue, 13 Aug 2019 05:28:42 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C27E6CC01A for ; Tue, 13 Aug 2019 05:28:42 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4671Tj6swMz4RVq; Tue, 13 Aug 2019 05:28:41 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-ot1-x32f.google.com with SMTP id g17so17047997otl.2; Mon, 12 Aug 2019 22:28:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ayryILj0QLCoPVcZxeEn/KLXHMbL8ZHSDeVvBUoNz/g=; b=TbMsnTi/SclTD15XM7Jr1m+E3MgKnQeEZy75ifCIXDOCd9IrdhYWUZYawqvH8BOQJ3 YXzUGSdLL2yUSaAqyXPkR0S/Dqgf56kDULTDEaLIWFGo1jNFtreI2sGCX6LAUejcnCM7 /7mf0NK5g8s69LJgv++KtEjLI4cHt7COrO/cxwTEgKsNZ1J7muZ4uogy+3qFW2kO4n5w 0TfoL4eLW1f5HlKpuceUYCBTYCwbLGOXJ2M2PhD0Y30vzfqINX7EZzW80rDujyYJq4kj ahbN43T8qjaOB7uFYE12rsp/ikTRDLHn1HY9jx/ulOaFn78PY3YYJic1SuwqoKaDZThF tusw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ayryILj0QLCoPVcZxeEn/KLXHMbL8ZHSDeVvBUoNz/g=; b=PE8/XF4CaTnUSRZg8rZiW0yPi1hR+4Xf15OQeeubsSTG6eEpSu1BkwFTauGMSEkZyY 3upPujxYp/WzmvU6GAMHoE9RXFW30NW8JXCUHjBkG+D20YGYn5OY+tjyb6ZxkPPF0VUL JoPiQhxwIw+hbrAzpwbh/AexrRIQYaDEK8Owd0S/A8d5fB9MvFoeX2nO3IU292Ke2tfd TSQiaToxP13dl1pFr34XA/ayePv0gmqR02Xm/SNQFZkL5N87JOu/7+uicjCT/6+WidOf +IUY51DHY7sMao7fog8gnSsuTgOGN6idCTWIT2CesqDPJG7nq7QjnczqtioNBczaZvNq cvIg== X-Gm-Message-State: APjAAAU5urkRl3dUplvVTNZrV6qyrY6ybfinWsNqtR4S+dRxk2wMlOKJ vs6YFsvJvT6y4HJCu3qBK4gnDMue6rDQgmM8iJfwbOAiS1U= X-Google-Smtp-Source: APXvYqx6o+wcUiIga8pqV2y1vRqjDX3olWH5wexdj81lxgtCHInv43srp9cUEe1PpcWe1NE8UZ7PgeLAl2H2xMQreqw= X-Received: by 2002:a9d:460c:: with SMTP id y12mr31325004ote.358.1565674120295; Mon, 12 Aug 2019 22:28:40 -0700 (PDT) MIME-Version: 1.0 References: <20190809183547.GE86532@raichu> <20190809211612.GF86532@raichu> In-Reply-To: From: Kevin Oberman Date: Mon, 12 Aug 2019 22:28:23 -0700 Message-ID: Subject: Re: Memory management changes after kernel update on 6-Aug To: Mark Johnston Cc: FreeBSD Stable ML X-Rspamd-Queue-Id: 4671Tj6swMz4RVq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=TbMsnTi/; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::32f as permitted sender) smtp.mailfrom=kob6558@gmail.com X-Spamd-Result: default: False [-3.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-9.19), ipnet: 2607:f8b0::/32(-2.97), asn: 15169(-2.39), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[f.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Aug 2019 05:28:43 -0000 On Fri, Aug 9, 2019 at 5:34 PM Kevin Oberman wrote: > > > On Fri, Aug 9, 2019 at 2:16 PM Mark Johnston wrote: > >> On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote: >> > On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston >> wrote: >> > >> > > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote: >> > > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing >> issues >> > > > resuming my Win7 VM on VirtualBox. My prior kernel was built on >> 24-Jul. >> > > If >> > > > there is not sufficient memory available to reload the system (4 >> Meg.), >> > > the >> > > >> > > Where does this number come from? What memory usage stats do you see >> in >> > > top(1) when the error occurs? >> > > >> > >> > I am monitoring memory usage with gkrellm. It appears to define "Free" >> as >> > the sum of "Inactive" and "Free". If you are referring to size of the >> VM, >> > was supposed to be the memory specified when I created the VM, but my >> > fingers got ahead of my brain and it should have been 4G, not 4M. Hey! >> > What's a few orders of magnitude? >> > >> > Oddly, when I watch memory space closely I note that, as the VM loads, I >> > started seeing swap utilization increase as free space was exhausted at >> > about 80% loaded. Loading continued to 98%. at that point loading >> stopped >> > and swap use continued to grow for a bit. Then free space started to >> > increase from about 300M to about 700M before the error window popped >> up. >> > >> > >> > > > resume fails with a message that memory was exhausted. Usually I >> can try >> > > > resuming again and it will work. Sometimes I get the error two or >> three >> > > > times before the system resumes. >> > > >> > > What exactly is the error message? >> > > >> > Failed to open a session for the virtual machine Win7. >> > >> > Failed to load unit 'pgm' (VERR_EM_NO_MEMORY). >> > >> > Result Code: NS_ERROR_FAILURE (0x80004005) >> > Component: ConsoleWrap >> > Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed} >> > >> > >> > > >> > > > Since I have not touched VirtualBox other than to rebuild the kmod >> after >> > > > the kernel build, it looks like something in the OS triggered this. >> Since >> > > > the system frees up some memory each time so that the VM eventually >> > > > resumes, it looks like the memory request is made to the OS, but VB >> is >> > > not >> > > > waiting or not enough memory is freed to allow the VB to complete >> the >> > > > resume. >> > > > >> > > > Any clue what might have changed over those 13 days? I am running >> GENERIC >> > > > except that I run the 4BSD scheduler. >> > > >> > > Possible culprits are r350374 and r350375, but I can't really see how. >> > > >> > >> > This started after the 6-Aug build (r350664). My prior build was >> r350292, >> > so just before these two commits. >> > >> > Can I try just reverting these two? Once I do, it will need to run for a >> > while or do something to tie up a lot of memory before the error will >> > recur. In normal use it is a matter of firefox increasing resident >> memory >> > until there is not enough free memory to load the VM without swapping. >> > (These days I often see the sum of all firefox process resident memory >> > exceeding 3G after it's been up for a day or two. Still, not worse than >> > chromium.) >> >> Those commits can simply be reverted, but I am skeptical that they will >> help. You should also verify that these same conditions don't lead to >> errors on your prior build, if you haven't already. >> > > OK. Running identical kernel except for 350374-5. > > Yes, I am sure that it was not happening with the r350292 kernel. I hit > this quite consistently when firefox has been running for a while. > > Firefox has rss of just under 3G on startup and will slowly grow until I > don't have the resources to run the Win7 VM without the error. Right now > the VM completes loading with no swapping and about 800M of memory free > after it is running. I'll let you know when it gets big enough to cause a > problem and whether it fails. Probably won't happen until tomorrow. > For three days I have been trying to repeat the failure with the to changed backed out and the problem does not occur with 350374-5 reverted. I have no idea how these changes trigger the issue, but it sure looks like they do. It happens when a 4G file is being loaded into memory. I have not looked at the VB code to see if anything unusual is being done. I did noticed that the problem only seems to show up when the VM is almost fully loaded... usually 98%, but I have seen it all the way to 100%. That was an odd "failure" where the error was not fatal. The VM was in a "Paused" state. I was able to resume it and it ran normally. Let me know what else, if anything, you would like me to try or test. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683