Date: Sun, 12 Jul 2020 01:28:41 -0500 From: Scott Bennett <bennett@sdf.org> To: freebsd-stable@freebsd.org Cc: "Peter Jeremy <peter@rulingia.com> Donald Wilde" <dwilde1@gmail.com> Subject: Re: swap space issues Message-ID: <202007120628.06C6SfNB015907@sdf.org>
next in thread | raw e-mail | index | archive | help
I have read this entire thread to date with growing dismay, and I
thank Donald Wilde for reporting his ongoing troubles, although they
spoil my hopes that the kernel's memory management bugs that first became
apparent in 11.2-RELEASE (and -STABLE around the same time) were not
propagated into 12.x. A recent update to stable/12 source tree made it
finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I
was just about to install the upgrade when this thread appeared.
On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde <dwilde1@gmail.com>
wrote:
>On 6/26/20, Peter Jeremy <peter@rulingia.com> wrote:
>> On 2020-Jun-25 11:30:31 -0700, Donald Wilde <dwilde1@gmail.com> wrote:
>>>Here's 'pstat -s' on the i3 (which registers as cpu HAMMER):
>>>
>>>Device 1K-blocks Used Avail Capacity
>>>/dev/ada0s1b 33554432 0 33554432 0%
>>>/dev/ada0s1d 33554432 0 33554432 0%
>>>Total 67108864 0 67108864 0%
>>
>> I strongly suggest you don't have more than one swap device on spinning
>> rust - the VM system will stripe I/O across the available devices and
>> that will give particularly poor results when it has to seek between the
>> partitions.
True. The only reason I can think of to use more than one swapping/
paging area on the same device for the same OS instance is for emergencies
or highly unusual, temporary situations in which more space is needed until
those situations conclude. and even in such situations, if the space can be
found on another device, it should be placed there. Interleaving of swap
space across multiple devices is intended as a performance enhancement
akin to striping (a.k.a. RAID0), although the virtual memory isn't
necessarily always actually striped across those devices. Adding a paging
area on the same device as an existing one is an abhorrent situation, as
Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as
the extraordinary situation has passed. N.B. the GENERIC kernel sets a
limit of four swap devices, although it can be rebuilt with a different
limit.
>
>My intent is to make this machine function -- getting the bear
>dancing. How deftly she dances is less important than that she dances
>at all. My for-real boxen will have real HP and real cores and RAM.
>
>>
>> Also, you can't actually use 64GB swap with 4GB RAM. If you look back
>> through your boot messages, I expect you'll find messages like:
>> warning: total configured swap (524288 pages) exceeds maximum recommended
>> amount (498848 pages).
>> warning: increase kern.maxswzone or reduce amount of swap.
Also true. Unfortunately, no guidance whatsoever is provided to advise
system administrators who need more space as to how to increase the relevant
table sizes and limits. However, that is a documentation bug, not a code
bug.
>
>Yes, as I posted, those were part of the failure stream from the synth
>program. When I had kern.maxswzone increased, it got through boot
>without complaining.
>
>> or maybe:
>> WARNING: reducing swap size to maximum of xxxxMB per unit
>
>The warnings were there, in the as-it-failed complaints.
>
>> The absolute limit on swap space is vm.swap_maxpages pages but the
>> realistic
>> limit is about half that. By default the realistic limit is about 4?RAM
>> (on
>> 64-bit architectures), but this can be adjusted via kern.maxswzone (which
>> defines the #bytes of RAM to allocate to swzone structures - the actual
>> space allocated is vm.swzone).
>>
>> As a further piece of arcana, vm.pageout_oom_seq is a count that controls
>> the number of passes before the pageout daemon gives up and starts killing
>> processes when it can't free up enough RAM. "out of swap space" messages
>> generally mean that this number is too low, rather than there being a
>> shortage of swap - particularly if your swap device is rather slow.
>>
>Thanks, Peter!
A second round of thanks to Peter Jeremy for pointing out this sysctl
variable (vm.pageout_oom_seq), although thus far I have yet to see that it is
actually effective in working around the memory management bugs. I have added
the following lines to /etc/sysctl.conf.
# Because FreeBSD 11.{2,3,4} tie up page frames unnecessarily, set value high
#vm.pageout_wakeup_thresh=14124 # Default value
vm.pageout_wakeup_thresh=112640 # 410 MB
Between the two changes, the pagedaemon *seems* to have stopped killing important
processes (or others) for now, which is a huge improvement and relief. Too bad
FreeBSD needs the changes to be made to keep the system usable somewhat longer.
My system has 8 GB of real memory. The kernel apparently refuses to swap
in *any* process, even one as small as /bin/sh, when the free page frame list
has less that ~410 MB of page frames on it. Setting the vm.pageout_wakeup_thresh
to at least 410 MB *seems* to help reduce the number of times a process that
has been marked as swapped out when the system has been under some form of
memory pressure, but it doesn't stop it from happening when the kernel has
pagefixed far too many page frames and hasn't pagefreed them when no longer
really requiring them to remain in real memory. I don't know whether the bug
is one of illegitimately pagefixing or failure to pagefree, but eventually
the number of page frames tied up becomes so high that too few frames remain
available for the kernel to be willing to page processes back in to resume
them. Increasing vm.pageout_wakeup_thresh drastically from its default value
is the primary way I have found to extend the time that my system remains
usable before I am forced to reboot.
As mentioned previously, I am dismayed that 12.1 appears to contain at
least some of 11.2+'s bugs. Given that 11.1 went EOL a long time ago, that
means that there is presently *NO PRODUCTION RELEASE OF FREEBSD AVAILABLE*
since 11.1-RELEASE, the FreeBSD project web site's erroneous claims
notwithstanding. This is an awful situation and probably calls for some
corrective action by the FreeBSD core team. A production release does not
force a reboot every few days or even every month or two in order to remain
usable. That's one of the reasons Windows through XP and VISTA were never
production systems, even though Mico$lop gave its users no production
alternatives. FreeBSD used to do better and it should be doing better now.
It is worth noting that a few years ago, FreeBSD project staff realized
that an elderly Pentium II(?) machine running FreeBSD 2.something and still
doing some simple, but necessary, task for the project had an uptime in excess
of 19 *years*. That is a reliability record for any OS to strive for. It is
a shame that FreeBSD's quality control no longer results in anything close to
that.
Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet: bennett at sdf.org *xor* bennett at freeshell.org *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good *
* objection to the introduction of that bane of all free governments *
* -- a standing army." *
* -- Gov. John Hancock, New York Journal, 28 January 1790 *
**********************************************************************
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202007120628.06C6SfNB015907>