From owner-freebsd-hackers@freebsd.org Mon Nov 19 17:41:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BCCE110F1F5 for ; Mon, 19 Nov 2018 17:41:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-9.consmr.mail.gq1.yahoo.com (sonic307-9.consmr.mail.gq1.yahoo.com [98.137.64.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A69481742 for ; Mon, 19 Nov 2018 17:41:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: wsEe1cwVM1mzvd37YjOoie0DKeN0C8.IomPi6w4mtuzzto52VHTPHEv2zsCvfqH MjfXp3um1TXN8NXvWjvfy1mhkMQSxMbxZn4A0SCh7NppZ8cd9XTpe6xq1Qyv5e.WfrbP5j7hg5tD MEx3qqcnnFNxsKyksvpX1sSZAXxTJnDIrcrmph63lKoEKRWq7bH7SWB9pUWMdEbeb7rBiO2b9SxJ VmETovH14axOjmYxLCxtEW3lgSW7b8B_ysRhh6QX35E4D2vOC58tmqlTilvVWdSGRkaUVOHTN_Td XuR58KLiE49n2dj4JKBPZPyxOWoiPZI_qxtFRwFNvCrf0RnHzCGLmRYhq72Pg64xfLoztiA9z64L vGvhFgS2doNQd2_U_mhyKB3kmEJaUzUbpx7z94xM.xP8cHW8ADq3aO1W04t9Pvbe2C.C2TfbzmON vlnHB4jofTJfZ.MwXCZFhdGbOKtIYcTU0OqVvpWsMe5FHaC_3rlMajSgEeXV1mQrHbS1paBmp6Q0 WUWRrbFKEk97DH0a_XFePZdvKN_CZJu3tvGqBUcPkYoqwhO4lUGimNtHEJuIAa3VTN25z4kRrxpa hbVlOT9AnYRKFyP9mb1OAywAcqtgZH2aTTPYKVHLaVbGWkd4wMBX_lyKCtcEFjl42QVInRXM37WZ RnBP3RQoEoSTMN.oYueI9GCxm.H_FrWVyL1RHVu1vf_IhNloPCunQhbVi7.E85SdwogWnNREEwnu QM6KolB1J_DyHJcIEpZlkuZwUcCGlijB.DKOSxQWeTPPbrn8SjkXbXVfiIYfTkjq8uCq55l5W7Th sW600uYTDCQJ3POZeMsOIeIM8wO.t_WS9PZK8UCGUxHU.sbiQkcyf9sb9QoK5gJtQRCGk2xQofDR uWSjQrxIpwe83n55lLowTIC5cmCT28b_8JBOWX._CyPdvk4CNlscX2DIkzkqgeflv7LJo6lYK5KR c8zlO2cmfZDbrKRJUtBC98LgIlTVsH7lUyIOHZ8ldjqTni4HR8G6qCMwNKEP_wb2PMpr46cbUPl6 bb2EFcc8JHd9C2vk8RbXoa3KVCrakf8HMDaJ9L6mjmMwDNrd0GKixOT0- Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Mon, 19 Nov 2018 17:41:09 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp415.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 18d4660213a4acdc0052b74c7158b9f7; Mon, 19 Nov 2018 17:41:07 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Mark Millard In-Reply-To: Date: Mon, 19 Nov 2018 09:41:06 -0800 Cc: Wojciech Puchar , Cy Schubert , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore Content-Transfer-Encoding: quoted-printable Message-Id: References: <201811182205.wAIM543W036241@slippy.cwsent.com> To: Stefan Blachmann X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: 0A69481742 X-Spamd-Result: default: False [-0.85 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com]; NEURAL_HAM_MEDIUM(-0.89)[-0.892,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[yahoo.com]; NEURAL_SPAM_SHORT(0.31)[0.313,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCVD_IN_DNSWL_NONE(0.00)[33.64.137.98.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[7]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; IP_SCORE(0.24)[ipnet: 98.137.64.0/21(0.72), asn: 36647(0.58), country: US(-0.09)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 17:41:18 -0000 On 2018-Nov-19, at 06:40, Stefan Blachmann = wrote: > On 11/19/18, Wojciech Puchar wrote: >>> can go a long way. Having said that, it's been a while since I've = had >>> to do this. The updates made to ZFS ARC and NUMA have allowed me to >>> rely on the algorithms baked into FreeBSD. My 8 GB systems have >>> performed rather well. >>=20 >> 8GB is LOT LOT LOT of memory. >>=20 >=20 > This might indeed be a cue. > Actually, I *never* had such annoying swap lag issues, until I > replaced my old computers (8GB) with newer ones (24-48GB). >=20 > With this increased memory, the ZFS ARC issue became apparent, causing > massive problems when ARC used more than 40 of the 48GB, starving > programs from memory, but this can be tuned easily. >=20 > And the "preemptive swapping" issue became apparent, too. You mentioned ZFS. Have any reports of "preemptive swapping" been for a context where ZFS was not involved? This might always be part of the context, sort of like you are saying larger amounts of RAM are. My contexts do not use ZFS but I have one context with 96 GiBytes of RAM. It has no problem with pageouts/swapouts during idle, even when idle for days. Such activity is always well explained by process RAM use and dirty page handling that follows. No evidence of "preemptive swapping". But idle here means very little running: only running the processes the OS starts plus up to a few ssh sessions, nfs client/server included, ntpd included, ssh allowed, a couple of network protocol daemons enabled, sendmail enables set to "NO". (There are 32 "cpus".) The machines are not outward accessible and do not run X11 or other such. > Letting > programs idle for some time caused swapping out. Is there any evidence of growing RAM use during such generally idle times? Might there be some form of RAM leak in your workload or the OS (for how you are using it)? FreeBSD waits to start paging/swapping until the free RAM is low --by its criteria for low. The pageout/swapout is just a a step in the means of gaining more free RAM. Have you observed what is going on (RAM use) just before and during when pages-outs/swap-outs are first initiated? A description of that would be handy for folks trying to help you. > This led to a very > bad user experience... often when one changes to another > desktop/virtual screen to another program, there is a delay of > swapping in, which in some cases could last for seconds when hundreds > of megabytes were swapped in again, in spite of some tens of gigabytes > of memory that never have been used since system boot. >=20 >> and my servers that have LOTS of httpd servers each for one webpage = which are usually rarely visited. >=20 > I wondered why my test server had a delayed response time the first > few times I call a page from it after having it idled for some time. > To me it looks like that the threads apache keeps cached in memory for > fast response times got swapped out and the server response is slow > until all these have become "memory resident" again. >=20 > As the bad response times due to unnecessary preemptive swapping would > induce Google downranking for sites that aren't visited sufficiently > frequently enough to keep them from being swapped out, I consider this > a very very serious problem. >=20 >=20 > Disabling swap (either totally or for the affected > programs/users/jails using rctl) and taking care that memory never > runs out was the only way I was able to fix these issues. >=20 > And I find very interesting that this seems to occur *only* on > machines with *much* memory. > The complaints about this swapping behavior I saw on the forums > *always* regarded machines with at least 48GB of RAM. (I am not sure > whether I saw reports regarding 32GB machines, so I would suggest > investigating the issue on machines >=3D48GB) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)