From owner-freebsd-questions@freebsd.org Fri Oct 16 17:54:24 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA6E443C22B for ; Fri, 16 Oct 2020 17:54:24 +0000 (UTC) (envelope-from freebsd@jschneider.net) Received: from slave.jschneider.net (slave.jschneider.net [46.235.226.89]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CCYgg6nk5z41M5 for ; Fri, 16 Oct 2020 17:54:23 +0000 (UTC) (envelope-from freebsd@jschneider.net) X-Disposition: Mostly harlmess; Fri, 16 Oct 2020 18:54:08 +0100 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Jon Schneider Mime-Version: 1.0 (1.0) Subject: Re: FreeBSD using swap even though there's a lot of free memory Message-Id: <05399132-C7A7-4B96-B6E4-D469CCDC70D5@jschneider.net> Date: Fri, 16 Oct 2020 18:54:06 +0100 Cc: freebsd-questions@freebsd.org, Twingly Customer Support To: doug@safeport.com X-Mailer: iPad Mail (18A393) X-Rspamd-Queue-Id: 4CCYgg6nk5z41M5 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@jschneider.net designates 46.235.226.89 as permitted sender) smtp.mailfrom=freebsd@jschneider.net X-Spamd-Result: default: False [-1.73 / 15.00]; FAKE_REPLY(1.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.991]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:46.235.226.89]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[jschneider.net]; NEURAL_HAM_LONG(-0.99)[-0.987]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.955]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:44684, ipnet:46.235.224.0/21, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2020 17:54:24 -0000 =EF=BB=BFSunOS 4 used to (speculatively) page stuff out. Probably just indic= ates relatively unused pages. Jon > On 16 Oct 2020, at 18:50, doug wrote: >=20 > =EF=BB=BF >=20 >> On Thu, 15 Oct 2020, Jon Schneider wrote: >> top -w -oswap >> seems to report the right thing. Would be interesting if it's _not_ MySQL= . >> Jon >>> On 15/10/2020 15:23, Twingly Customer Support wrote: >>> Hi, >>> We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a p= roblem where FreeBSD starting to swap when running ZFS scrub, even though we= have ~70G of free memory. >>> This did not happen before when running FreeBSD 11.3 for example. It sta= rted happening at approximately the time we upgraded from 12.1-RELEASE-p5 to= 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear,= though FreeBSD never swapped for us before that. "Laundry" memory was not s= omething we saw before either, it started to appear at the same time as Free= BSD started swapping. >>> Eventually, after scrubbing a few times, the swap becomes full and we st= art seeing "swap_pager_getswapspace(24): failed" etc. in dmesg. >>> This is the memory usage a while after scrubbing, note the values for Me= m/Free and Swap: >>> ``` >>> % top | head -n 7 >>> last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 10:5= 3:48 >>> 35 processes: 1 running, 34 sleeping >>> CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle >>> Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free >>> ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other >>> 30G Compressed, 53G Uncompressed, 1.77:1 Ratio >>> Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse >>> ``` >>> We are running MySQL, which has been configured to use ~50% of the total= amount memory (using innodb_buffer_pool_size=3D127748M) >>> ZFS ARC has been configured to use 25% of the total memory (using vfs.zf= s.arc_max=3D"63874M") >>> We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, b= ut this did not make any change to the total memory usage, the free memory s= tays at around 70G and FreeBSD still started swapping. >>> It's as if the memory is capped at around 180G for some reason. >>> Are there any configuration values that could cause FreeBSD to swap even= though there's free memory? Are there any config values one could try to ch= ange in order to get FreeBSD to use the remaining ~70G of free memory instea= d of swapping? >>> Let me know if there's any more details you want me to provide and I'll a= ttach those. >>> Thanks! >>> // Mattias >=20 > I see similar things. The Jails in question are 11.1. The systems updated t= o 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo= is shown below: >=20 > Device 1K-blocks Used Avail Capacity > /dev/aacd0p3 4194304 1776000 2418304 42% >=20 > These numbers are developed from top on the base system >=20 > [ 0 50861 ] root > [ 20 281903 ] camden squirellmail/roundcube/postfix/mysql > [ 21 322759 ] bassharbor wordpress/php56 > [ 19 343522 ] monhegan wordpress/php56 > [ 18 369139 ] newharbor apache24/sendmail > [ 17 587332 ] pemaquid wordpress/php74 >=20 > Jails: 1904655 > total: 1955516 >=20 > I read somewhere that the virtual memory system pre-pages modified pages a= s a just-in-case measure. If this is correct, 12.1 does not do this on a non= -paging system. The system shown above uses about 10% swapspace after a rebo= ot and works its way to the 42% shown above in a day or so.