From owner-freebsd-stable@freebsd.org Sun Jul 12 06:28:50 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DECC13597BE for ; Sun, 12 Jul 2020 06:28:50 +0000 (UTC) (envelope-from bennett@sdf.org) Received: from mx.sdf.org (mx.sdf.org [205.166.94.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx.sdf.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B4H0x1k3dz4BDl for ; Sun, 12 Jul 2020 06:28:49 +0000 (UTC) (envelope-from bennett@sdf.org) Received: from sdf.org (IDENT:bennett@miku.sdf.org [205.166.94.6]) by mx.sdf.org (8.15.2/8.14.5) with ESMTPS id 06C6SfV5017848 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits) verified NO); Sun, 12 Jul 2020 06:28:42 GMT Received: (from bennett@localhost) by sdf.org (8.15.2/8.12.8/Submit) id 06C6SfNB015907; Sun, 12 Jul 2020 01:28:41 -0500 (CDT) From: Scott Bennett Message-Id: <202007120628.06C6SfNB015907@sdf.org> Date: Sun, 12 Jul 2020 01:28:41 -0500 To: freebsd-stable@freebsd.org Subject: Re: swap space issues Cc: "Peter Jeremy Donald Wilde" User-Agent: Heirloom mailx 12.5 6/20/10 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4B4H0x1k3dz4BDl X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.25 / 15.00]; FAKE_REPLY(1.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:205.166.94.0/24]; NEURAL_HAM_LONG(-1.01)[-1.011]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_MEDIUM(-1.03)[-1.025]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.42)[-0.418]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[sdf.org,quarantine]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14361, ipnet:205.166.94.0/24, country:US]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2020 06:28:50 -0000 I have read this entire thread to date with growing dismay, and I thank Donald Wilde for reporting his ongoing troubles, although they spoil my hopes that the kernel's memory management bugs that first became apparent in 11.2-RELEASE (and -STABLE around the same time) were not propagated into 12.x. A recent update to stable/12 source tree made it finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I was just about to install the upgrade when this thread appeared. On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde wrote: >On 6/26/20, Peter Jeremy wrote: >> On 2020-Jun-25 11:30:31 -0700, Donald Wilde wrote: >>>Here's 'pstat -s' on the i3 (which registers as cpu HAMMER): >>> >>>Device 1K-blocks Used Avail Capacity >>>/dev/ada0s1b 33554432 0 33554432 0% >>>/dev/ada0s1d 33554432 0 33554432 0% >>>Total 67108864 0 67108864 0% >> >> I strongly suggest you don't have more than one swap device on spinning >> rust - the VM system will stripe I/O across the available devices and >> that will give particularly poor results when it has to seek between the >> partitions. True. The only reason I can think of to use more than one swapping/ paging area on the same device for the same OS instance is for emergencies or highly unusual, temporary situations in which more space is needed until those situations conclude. and even in such situations, if the space can be found on another device, it should be placed there. Interleaving of swap space across multiple devices is intended as a performance enhancement akin to striping (a.k.a. RAID0), although the virtual memory isn't necessarily always actually striped across those devices. Adding a paging area on the same device as an existing one is an abhorrent situation, as Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as the extraordinary situation has passed. N.B. the GENERIC kernel sets a limit of four swap devices, although it can be rebuilt with a different limit. > >My intent is to make this machine function -- getting the bear >dancing. How deftly she dances is less important than that she dances >at all. My for-real boxen will have real HP and real cores and RAM. > >> >> Also, you can't actually use 64GB swap with 4GB RAM. If you look back >> through your boot messages, I expect you'll find messages like: >> warning: total configured swap (524288 pages) exceeds maximum recommended >> amount (498848 pages). >> warning: increase kern.maxswzone or reduce amount of swap. Also true. Unfortunately, no guidance whatsoever is provided to advise system administrators who need more space as to how to increase the relevant table sizes and limits. However, that is a documentation bug, not a code bug. > >Yes, as I posted, those were part of the failure stream from the synth >program. When I had kern.maxswzone increased, it got through boot >without complaining. > >> or maybe: >> WARNING: reducing swap size to maximum of xxxxMB per unit > >The warnings were there, in the as-it-failed complaints. > >> The absolute limit on swap space is vm.swap_maxpages pages but the >> realistic >> limit is about half that. By default the realistic limit is about 4?RAM >> (on >> 64-bit architectures), but this can be adjusted via kern.maxswzone (which >> defines the #bytes of RAM to allocate to swzone structures - the actual >> space allocated is vm.swzone). >> >> As a further piece of arcana, vm.pageout_oom_seq is a count that controls >> the number of passes before the pageout daemon gives up and starts killing >> processes when it can't free up enough RAM. "out of swap space" messages >> generally mean that this number is too low, rather than there being a >> shortage of swap - particularly if your swap device is rather slow. >> >Thanks, Peter! A second round of thanks to Peter Jeremy for pointing out this sysctl variable (vm.pageout_oom_seq), although thus far I have yet to see that it is actually effective in working around the memory management bugs. I have added the following lines to /etc/sysctl.conf. # Because FreeBSD 11.{2,3,4} tie up page frames unnecessarily, set value high #vm.pageout_wakeup_thresh=14124 # Default value vm.pageout_wakeup_thresh=112640 # 410 MB Between the two changes, the pagedaemon *seems* to have stopped killing important processes (or others) for now, which is a huge improvement and relief. Too bad FreeBSD needs the changes to be made to keep the system usable somewhat longer. My system has 8 GB of real memory. The kernel apparently refuses to swap in *any* process, even one as small as /bin/sh, when the free page frame list has less that ~410 MB of page frames on it. Setting the vm.pageout_wakeup_thresh to at least 410 MB *seems* to help reduce the number of times a process that has been marked as swapped out when the system has been under some form of memory pressure, but it doesn't stop it from happening when the kernel has pagefixed far too many page frames and hasn't pagefreed them when no longer really requiring them to remain in real memory. I don't know whether the bug is one of illegitimately pagefixing or failure to pagefree, but eventually the number of page frames tied up becomes so high that too few frames remain available for the kernel to be willing to page processes back in to resume them. Increasing vm.pageout_wakeup_thresh drastically from its default value is the primary way I have found to extend the time that my system remains usable before I am forced to reboot. As mentioned previously, I am dismayed that 12.1 appears to contain at least some of 11.2+'s bugs. Given that 11.1 went EOL a long time ago, that means that there is presently *NO PRODUCTION RELEASE OF FREEBSD AVAILABLE* since 11.1-RELEASE, the FreeBSD project web site's erroneous claims notwithstanding. This is an awful situation and probably calls for some corrective action by the FreeBSD core team. A production release does not force a reboot every few days or even every month or two in order to remain usable. That's one of the reasons Windows through XP and VISTA were never production systems, even though Mico$lop gave its users no production alternatives. FreeBSD used to do better and it should be doing better now. It is worth noting that a few years ago, FreeBSD project staff realized that an elderly Pentium II(?) machine running FreeBSD 2.something and still doing some simple, but necessary, task for the project had an uptime in excess of 19 *years*. That is a reliability record for any OS to strive for. It is a shame that FreeBSD's quality control no longer results in anything close to that. Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************