Date: Mon, 19 Nov 2018 09:41:06 -0800 From: Mark Millard <marklmi@yahoo.com> To: Stefan Blachmann <sblachmann@gmail.com> Cc: Wojciech Puchar <wojtek@puchar.net>, Cy Schubert <Cy.Schubert@cschubert.com>, Rebecca Cran <rebecca@bluestop.org>, freebsd-hackers Hackers <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org>, Ian Lepore <ian@freebsd.org> Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM Message-ID: <FCBC1231-56D2-4ED6-882A-84E94DAB28CD@yahoo.com> In-Reply-To: <CACc-My0LcjESEvzW%2Bwe_95Gkv4PbD=uUGZ8y_1=0bO0XWqGhxQ@mail.gmail.com> References: <201811182205.wAIM543W036241@slippy.cwsent.com> <alpine.BSF.2.20.1811191124260.52689@puchar.net> <CACc-My0LcjESEvzW%2Bwe_95Gkv4PbD=uUGZ8y_1=0bO0XWqGhxQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2018-Nov-19, at 06:40, Stefan Blachmann <sblachmann a tgmail.com> = wrote: > On 11/19/18, Wojciech Puchar <wojtek@puchar.net> wrote: >>> can go a long way. Having said that, it's been a while since I've = had >>> to do this. The updates made to ZFS ARC and NUMA have allowed me to >>> rely on the algorithms baked into FreeBSD. My 8 GB systems have >>> performed rather well. >>=20 >> 8GB is LOT LOT LOT of memory. >>=20 >=20 > This might indeed be a cue. > Actually, I *never* had such annoying swap lag issues, until I > replaced my old computers (8GB) with newer ones (24-48GB). >=20 > With this increased memory, the ZFS ARC issue became apparent, causing > massive problems when ARC used more than 40 of the 48GB, starving > programs from memory, but this can be tuned easily. >=20 > And the "preemptive swapping" issue became apparent, too. You mentioned ZFS. Have any reports of "preemptive swapping" been for a context where ZFS was not involved? This might always be part of the context, sort of like you are saying larger amounts of RAM are. My contexts do not use ZFS but I have one context with 96 GiBytes of RAM. It has no problem with pageouts/swapouts during idle, even when idle for days. Such activity is always well explained by process RAM use and dirty page handling that follows. No evidence of "preemptive swapping". But idle here means very little running: only running the processes the OS starts plus up to a few ssh sessions, nfs client/server included, ntpd included, ssh allowed, a couple of network protocol daemons enabled, sendmail enables set to "NO". (There are 32 "cpus".) The machines are not outward accessible and do not run X11 or other such. > Letting > programs idle for some time caused swapping out. Is there any evidence of growing RAM use during such generally idle times? Might there be some form of RAM leak in your workload or the OS (for how you are using it)? FreeBSD waits to start paging/swapping until the free RAM is low --by its criteria for low. The pageout/swapout is just a a step in the means of gaining more free RAM. Have you observed what is going on (RAM use) just before and during when pages-outs/swap-outs are first initiated? A description of that would be handy for folks trying to help you. > This led to a very > bad user experience... often when one changes to another > desktop/virtual screen to another program, there is a delay of > swapping in, which in some cases could last for seconds when hundreds > of megabytes were swapped in again, in spite of some tens of gigabytes > of memory that never have been used since system boot. >=20 >> and my servers that have LOTS of httpd servers each for one webpage = which are usually rarely visited. >=20 > I wondered why my test server had a delayed response time the first > few times I call a page from it after having it idled for some time. > To me it looks like that the threads apache keeps cached in memory for > fast response times got swapped out and the server response is slow > until all these have become "memory resident" again. >=20 > As the bad response times due to unnecessary preemptive swapping would > induce Google downranking for sites that aren't visited sufficiently > frequently enough to keep them from being swapped out, I consider this > a very very serious problem. >=20 >=20 > Disabling swap (either totally or for the affected > programs/users/jails using rctl) and taking care that memory never > runs out was the only way I was able to fix these issues. >=20 > And I find very interesting that this seems to occur *only* on > machines with *much* memory. > The complaints about this swapping behavior I saw on the forums > *always* regarded machines with at least 48GB of RAM. (I am not sure > whether I saw reports regarding 32GB machines, so I would suggest > investigating the issue on machines >=3D48GB) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FCBC1231-56D2-4ED6-882A-84E94DAB28CD>