From owner-freebsd-stable@freebsd.org Sun Jul 12 14:24:04 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4AD1C364C29 for ; Sun, 12 Jul 2020 14:24:04 +0000 (UTC) (envelope-from dwilde1@gmail.com) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B4TYH24KRz4bJw for ; Sun, 12 Jul 2020 14:24:03 +0000 (UTC) (envelope-from dwilde1@gmail.com) Received: by mail-pl1-x62f.google.com with SMTP id x8so4346032plm.10 for ; Sun, 12 Jul 2020 07:24:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=RMuDBp+GGkoxrt3Sv9nknH47ITShFpJXhcA4ZUXzfXE=; b=daOibBevqrLDlDIVwIqW9ZzJmng+Lo98lAJbQafgMeUEeBr75Stg6T2Hy9/sjs2lnH oDCPFpRIjmb6MGb/dYZzqy9uibn1Rny2tve5hkw7gyonFJRJdViFh2PZMePs96BKcobg bIi9IRE+GWSnnSnqpPV5mYzXbt61RfF3Mh/mkGVVFA7tuTr6B5Lb3TlZC5uEbp8N7Y9C gzbY3Jg4SDsHEaih0/jYkC5lZFwvlTLi3hzIkUCc1L9DfRfQodAX7nGBjz3botp9L4Fv VgST44KE02zuji8L1Uw7ToqzAUPBJEJysm1HnjWOPX9kbKGrIgyQMpzMXOZeuNV0Urqz 36Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=RMuDBp+GGkoxrt3Sv9nknH47ITShFpJXhcA4ZUXzfXE=; b=ZbzOr5eP1YwayI9vj7ig2Br5D2JRgm9j+FFAtbNoYDjFQxgBknfJPKz6maNGisd/ZZ JmIptH6BaHSlz9updfADkMCPeLnGDoo4bCoU28TK10vWxv0sQRCWMQ8CQiOPyEa01FMu TuWPX+z4hhV3VTKYw8ZUOik8WHd3fpZtQhSQC0R7kASByzWxyhnOrhmHBa3lvUyGaaBG cuJo6G5gK5j6erYO4IbfPIbpCGXNcnv0Ot6+XWtY3DyLNSoaLcT5dIi++dZMCqYaPNIF ICYcGExSuSj7IsgaTNAzHGr13f6kZdqVIIHChqdbejoYMKhcsfhzwkuqMFbgbZIdW5E9 iGSg== X-Gm-Message-State: AOAM533zZlFvUfFh/XhzzCgbJbnTjkn5HG+F+c0P/RQPYoPo9XPq0+UI TyrGZ9BheQviMOZAsCDnL14UocuY X-Google-Smtp-Source: ABdhPJx0tjH32PpVJLaJlCEdYV8asPoDyEPvYqj9LvHdL820lrqI4cr8++domyNKQY0Kx8ehf65VyQ== X-Received: by 2002:a17:90b:3842:: with SMTP id nl2mr14889869pjb.111.1594563841024; Sun, 12 Jul 2020 07:24:01 -0700 (PDT) Received: from [192.168.0.4] (174-26-193-115.phnx.qwest.net. [174.26.193.115]) by smtp.gmail.com with ESMTPSA id k2sm10771260pgm.11.2020.07.12.07.23.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Jul 2020 07:24:00 -0700 (PDT) Subject: Re: swap space issues To: Scott Bennett , freebsd-stable@freebsd.org References: <202007120628.06C6SfNB015907@sdf.org> From: Don Wilde Message-ID: <3afe4d2f-3b1e-a1c5-f947-5f57800317a6@gmail.com> Date: Sun, 12 Jul 2020 07:23:58 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <202007120628.06C6SfNB015907@sdf.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4B4TYH24KRz4bJw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=daOibBev; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of dwilde1@gmail.com designates 2607:f8b0:4864:20::62f as permitted sender) smtp.mailfrom=dwilde1@gmail.com X-Spamd-Result: default: False [-3.55 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.48)[-0.478]; RECEIVED_SPAMHAUS_PBL(0.00)[174.26.193.115:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.048]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.019]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62f:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2020 14:24:04 -0000 On 7/11/20 11:28 PM, Scott Bennett via freebsd-stable wrote: > I have read this entire thread to date with growing dismay, and I > thank Donald Wilde for reporting his ongoing troubles, although they > spoil my hopes that the kernel's memory management bugs that first became > apparent in 11.2-RELEASE (and -STABLE around the same time) were not > propagated into 12.x. A recent update to stable/12 source tree made it > finally possible for me to build 12.1-STABLE under 11.4-PRERELEASE, and I > was just about to install the upgrade when this thread appeared. Spoiler alert. Since I gave up on Synth, I haven't had a single swap issue. It does appear to be one particular port that drove it nuts (apparently, one of the 'Google performance' bits, with a mismatched-brackets problem). I have rebuilt the machine several times, but that's more for my sense of tidiness than anything. I've got a little Crystal script that walks the installed packages and ports and updates them with system() calls. The machine is very slow, but it's not swapping at all. It is quite usable now with 12-STABLE. > > On Fri, 26 Jun 2020 03:55:04 -0700 : Donald Wilde > wrote: > >> On 6/26/20, Peter Jeremy wrote: >>> [snip] >>> I strongly suggest you don't have more than one swap device on spinning >>> rust - the VM system will stripe I/O across the available devices and >>> that will give particularly poor results when it has to seek between the >>> partitions. > True. The only reason I can think of to use more than one swapping/ > paging area on the same device for the same OS instance is for emergencies > or highly unusual, temporary situations in which more space is needed until > those situations conclude. and even in such situations, if the space can be > found on another device, it should be placed there. Interleaving of swap > space across multiple devices is intended as a performance enhancement > akin to striping (a.k.a. RAID0), although the virtual memory isn't > necessarily always actually striped across those devices. Adding a paging > area on the same device as an existing one is an abhorrent situation, as > Peter Jeremy noted, and it should be eliminated via swapoff(8) as soon as > the extraordinary situation has passed. N.B. the GENERIC kernel sets a > limit of four swap devices, although it can be rebuilt with a different > limit. That's good data, Scott, thanks! The only reason I got into that situation of trying to add another swap device was that it was crashing with OO swap messages. >> My intent is to make this machine function -- getting the bear >> dancing. How deftly she dances is less important than that she dances >> at all. My for-real boxen will have real HP and real cores and RAM. >> >>> Also, you can't actually use 64GB swap with 4GB RAM. If you look back >>> through your boot messages, I expect you'll find messages like: >>> warning: total configured swap (524288 pages) exceeds maximum recommended >>> amount (498848 pages). >>> warning: increase kern.maxswzone or reduce amount of swap. > Also true. Unfortunately, no guidance whatsoever is provided to advise > system administrators who need more space as to how to increase the relevant > table sizes and limits. However, that is a documentation bug, not a code > bug. I've got both my kern.max* and CCACHE set up mostly correctly. Everything builds and runs well, although I've found that it's helpful to only use -j3 while building, not -j4 which would be appropriate for my HAMMER i3. I'd much rather have the bear *dancing* than running into walls. :D >> Yes, as I posted, those were part of the failure stream from the synth >> program. When I had kern.maxswzone increased, it got through boot >> without complaining. >> >>> or maybe: >>> WARNING: reducing swap size to maximum of xxxxMB per unit >> The warnings were there, in the as-it-failed complaints. >> >>> The absolute limit on swap space is vm.swap_maxpages pages but the >>> realistic >>> limit is about half that. By default the realistic limit is about 4?RAM >>> (on >>> 64-bit architectures), but this can be adjusted via kern.maxswzone (which >>> defines the #bytes of RAM to allocate to swzone structures - the actual >>> space allocated is vm.swzone). >>> >>> As a further piece of arcana, vm.pageout_oom_seq is a count that controls >>> the number of passes before the pageout daemon gives up and starts killing >>> processes when it can't free up enough RAM. "out of swap space" messages >>> generally mean that this number is too low, rather than there being a >>> shortage of swap - particularly if your swap device is rather slow. >>> >> Thanks, Peter! > A second round of thanks to Peter Jeremy for pointing out this sysctl > variable (vm.pageout_oom_seq), although thus far I have yet to see that it is > actually effective in working around the memory management bugs. I have added > the following lines to /etc/sysctl.conf. > > # Because FreeBSD 11.{2,3,4} tie up page frames unnecessarily, set value high > #vm.pageout_wakeup_thresh=14124 # Default value > vm.pageout_wakeup_thresh=112640 # 410 MB [snip] I do totally agree that these are crucial issues for both operation and documentation, although my issues stemmed from bad _userland_ stack control. Those who live on -CURRENT are used to OOPS, but the rest of us get paid not to have them. I am happy with what the Core Team gives us, AND of course we want ['more','better','faster','STABLE']. :D