Date: Mon, 12 Aug 2019 22:28:23 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Mark Johnston <markj@freebsd.org> Cc: FreeBSD Stable ML <stable@freebsd.org> Subject: Re: Memory management changes after kernel update on 6-Aug Message-ID: <CAN6yY1tMqD2bw8%2BhKiKq9aFwTAXw-VvoAiJVFDNqPNcQ1=QAEw@mail.gmail.com> In-Reply-To: <CAN6yY1t8D3Qa-ha2biLDPh58KBJZwdxZjRYkhXKDjKBuE_yYHA@mail.gmail.com> References: <CAN6yY1uLDBJc_We4N1BHi3s9f5R0pCTvVsKV_E8oj0YnWo6pQQ@mail.gmail.com> <20190809183547.GE86532@raichu> <CAN6yY1vitZejeqAWc9ATLCv_UHOQNNJ2uDgByWfvgoxfsCM7CQ@mail.gmail.com> <20190809211612.GF86532@raichu> <CAN6yY1t8D3Qa-ha2biLDPh58KBJZwdxZjRYkhXKDjKBuE_yYHA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Aug 9, 2019 at 5:34 PM Kevin Oberman <rkoberman@gmail.com> wrote:
>
>
> On Fri, Aug 9, 2019 at 2:16 PM Mark Johnston <markj@freebsd.org> wrote:
>
>> On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
>> > On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston <markj@freebsd.org>
>> wrote:
>> >
>> > > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
>> > > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing
>> issues
>> > > > resuming my Win7 VM on VirtualBox. My prior kernel was built on
>> 24-Jul.
>> > > If
>> > > > there is not sufficient memory available to reload the system (4
>> Meg.),
>> > > the
>> > >
>> > > Where does this number come from? What memory usage stats do you see
>> in
>> > > top(1) when the error occurs?
>> > >
>> >
>> > I am monitoring memory usage with gkrellm. It appears to define "Free"
>> as
>> > the sum of "Inactive" and "Free". If you are referring to size of the
>> VM,
>> > was supposed to be the memory specified when I created the VM, but my
>> > fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
>> > What's a few orders of magnitude?
>> >
>> > Oddly, when I watch memory space closely I note that, as the VM loads, I
>> > started seeing swap utilization increase as free space was exhausted at
>> > about 80% loaded. Loading continued to 98%. at that point loading
>> stopped
>> > and swap use continued to grow for a bit. Then free space started to
>> > increase from about 300M to about 700M before the error window popped
>> up.
>> >
>> >
>> > > > resume fails with a message that memory was exhausted. Usually I
>> can try
>> > > > resuming again and it will work. Sometimes I get the error two or
>> three
>> > > > times before the system resumes.
>> > >
>> > > What exactly is the error message?
>> > >
>> > Failed to open a session for the virtual machine Win7.
>> >
>> > Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
>> >
>> > Result Code: NS_ERROR_FAILURE (0x80004005)
>> > Component: ConsoleWrap
>> > Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>> >
>> >
>> > >
>> > > > Since I have not touched VirtualBox other than to rebuild the kmod
>> after
>> > > > the kernel build, it looks like something in the OS triggered this.
>> Since
>> > > > the system frees up some memory each time so that the VM eventually
>> > > > resumes, it looks like the memory request is made to the OS, but VB
>> is
>> > > not
>> > > > waiting or not enough memory is freed to allow the VB to complete
>> the
>> > > > resume.
>> > > >
>> > > > Any clue what might have changed over those 13 days? I am running
>> GENERIC
>> > > > except that I run the 4BSD scheduler.
>> > >
>> > > Possible culprits are r350374 and r350375, but I can't really see how.
>> > >
>> >
>> > This started after the 6-Aug build (r350664). My prior build was
>> r350292,
>> > so just before these two commits.
>> >
>> > Can I try just reverting these two? Once I do, it will need to run for a
>> > while or do something to tie up a lot of memory before the error will
>> > recur. In normal use it is a matter of firefox increasing resident
>> memory
>> > until there is not enough free memory to load the VM without swapping.
>> > (These days I often see the sum of all firefox process resident memory
>> > exceeding 3G after it's been up for a day or two. Still, not worse than
>> > chromium.)
>>
>> Those commits can simply be reverted, but I am skeptical that they will
>> help. You should also verify that these same conditions don't lead to
>> errors on your prior build, if you haven't already.
>>
>
> OK. Running identical kernel except for 350374-5.
>
> Yes, I am sure that it was not happening with the r350292 kernel. I hit
> this quite consistently when firefox has been running for a while.
>
> Firefox has rss of just under 3G on startup and will slowly grow until I
> don't have the resources to run the Win7 VM without the error. Right now
> the VM completes loading with no swapping and about 800M of memory free
> after it is running. I'll let you know when it gets big enough to cause a
> problem and whether it fails. Probably won't happen until tomorrow.
>
For three days I have been trying to repeat the failure with the to changed
backed out and the problem does not occur with 350374-5 reverted.
I have no idea how these changes trigger the issue, but it sure looks like
they do. It happens when a 4G file is being loaded into memory. I have not
looked at the VB code to see if anything unusual is being done. I did
noticed that the problem only seems to show up when the VM is almost fully
loaded... usually 98%, but I have seen it all the way to 100%. That was an
odd "failure" where the error was not fatal. The VM was in a "Paused"
state. I was able to resume it and it ran normally.
Let me know what else, if anything, you would like me to try or test.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1tMqD2bw8%2BhKiKq9aFwTAXw-VvoAiJVFDNqPNcQ1=QAEw>