Date: Tue, 20 Nov 2018 12:22:31 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= <Trond.Endrestol@fagskolen.gjovik.no> To: FreeBSD stable <freebsd-stable@freebsd.org> Subject: Re: plenty of memory, but system us intensively swapping Message-ID: <alpine.BSF.2.21.9999.1811201206110.90197@mail.fig.ol.no> In-Reply-To: <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in> References: <1ddb53ff-23a8-b364-0213-d8df3c7dcd3c@zhegan.in> <alpine.BSF.2.21.9999.1811201108060.90197@mail.fig.ol.no> <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 20 Nov 2018 15:22+0500, Eugene M. Zheganin wrote: > Hello, > > On 20.11.2018 15:12, Trond Endrestøl wrote: > > On freebsd-hackers the other day, > > https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html, > > it was suggested to set vm.pageout_update_period=0. This sysctl is at > > 600 initially. > > > > ZFS' ARC needs to be capped, otherwise it will eat most, if not all, > > of your memory. > Well, as you can see, ARC ate only half, and the other half is eaten by the > kernel. So far I suppose that if I will cap the ARC, the kernel will simply > eat the rest. I know others have created a daemon that observe the ARC and the amount of wired and free memory, and when these values exceed some threshold, the daemon will allocate a number of gigabytes, writing zero to the first byte or word of every page, and then freeing the allocated memory before going back to sleep. The ARC will release most of its allocations and the kernel will also release some but not all of its wired memory, and some user pages are likely to be thrown onto the swap device, turning the user experience to a mild nightmare while waiting for applications to be paged back into memory. ZFS seems to be the common factor in most, if not all, of these cases. I created my own and not so sophisticated C program that I run every now and then: #include <stdio.h> #include <stdlib.h> #include <unistd.h> int main(int argc, char **argv) { const size_t pagesize = (size_t)getpagesize(); const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL; size_t amount, n = 1ULL; char *p, *offset; if (argc > 1) { sscanf(argv[1], "%zu", &n); } amount = n * gigabyte; if (amount > 0ULL) { if ( (p = malloc(amount)) != NULL) { for (offset = p; offset < p + amount; offset += pagesize) { *offset = '\0'; } free(p); } else { fprintf(stderr, "%s:%s:%d: unable to allocate %zu gigabyte%s\n", argv[0], __FILE__, __LINE__, n, (n == 1ULL) ? "" : "s"); return 2; } } else { return 1; } return 0; } // main() // allocate_gigabytes.c -- Trond. From owner-freebsd-stable@freebsd.org Tue Nov 20 11:38:54 2018 Return-Path: <owner-freebsd-stable@freebsd.org> Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31C0B1122433 for <freebsd-stable@mailman.ysv.freebsd.org>; Tue, 20 Nov 2018 11:38:54 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48AE46A08E for <freebsd-stable@freebsd.org>; Tue, 20 Nov 2018 11:38:53 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from bsdrookie.norma.com. (asterisk.enaza.ru [91.237.76.254] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id wAKBckOD046799 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for <freebsd-stable@freebsd.org>; Tue, 20 Nov 2018 16:38:47 +0500 (+05) (envelope-from eugene@zhegan.in) Subject: Re: plenty of memory, but system us intensively swapping To: freebsd-stable@freebsd.org References: <1ddb53ff-23a8-b364-0213-d8df3c7dcd3c@zhegan.in> <alpine.BSF.2.21.9999.1811201108060.90197@mail.fig.ol.no> <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in> <alpine.BSF.2.21.9999.1811201206110.90197@mail.fig.ol.no> From: "Eugene M. Zheganin" <eugene@zhegan.in> Message-ID: <3efa0dd1-11d6-d424-9e37-a96cf7a6cf47@zhegan.in> Date: Tue, 20 Nov 2018 16:38:46 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <alpine.BSF.2.21.9999.1811201206110.90197@mail.fig.ol.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Scan-Time: 0.64 X-Rspamd-Queue-Id: 48AE46A08E X-Spamd-Result: default: False [0.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.14)[-0.143,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[zhegan.in]; NEURAL_SPAM_SHORT(0.46)[0.456,0]; MX_GOOD(-0.01)[cached: mail-backup.zhegan.in]; NEURAL_SPAM_LONG(0.64)[0.641,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(0.00)[country: RU(0.01)]; ASN(0.00)[asn:57401, ipnet:2a00:7540:1::/48, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>; List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 20 Nov 2018 11:38:54 -0000 Hello, On 20.11.2018 16:22, Trond Endrestøl wrote: > > I know others have created a daemon that observe the ARC and the > amount of wired and free memory, and when these values exceed some > threshold, the daemon will allocate a number of gigabytes, writing > zero to the first byte or word of every page, and then freeing the > allocated memory before going back to sleep. > > The ARC will release most of its allocations and the kernel will also > release some but not all of its wired memory, and some user pages are > likely to be thrown onto the swap device, turning the user experience > to a mild nightmare while waiting for applications to be paged back > into memory. > > ZFS seems to be the common factor in most, if not all, of these cases. > > I created my own and not so sophisticated C program that I run every > now and then: > > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > > int main(int argc, char **argv) > { > const size_t pagesize = (size_t)getpagesize(); > const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL; > > size_t amount, n = 1ULL; > char *p, *offset; > > if (argc > 1) { > sscanf(argv[1], "%zu", &n); > } > > amount = n * gigabyte; > > if (amount > 0ULL) { > if ( (p = malloc(amount)) != NULL) { > for (offset = p; offset < p + amount; offset += pagesize) { > *offset = '\0'; > } > > free(p); > } > else { > fprintf(stderr, > "%s:%s:%d: unable to allocate %zu gigabyte%s\n", > argv[0], __FILE__, __LINE__, > n, (n == 1ULL) ? "" : "s"); > return 2; > } > } > else { > return 1; > } > > return 0; > } // main() > > // allocate_gigabytes.c > Jeez, thanks a lot, this stuff is working. Now the system has 8 Gigs of free memory and stopped swapping. Well, the next question is addressed to the core team which I suppose reads this ML eventually - why we don't have something similar as a watchdog in the base system ? I understand that this solution is architecturally ugly, but it's now worse not to have any, and it still works. At least I'm about to run this periodically. Trond, thanks again. Eugene.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.21.9999.1811201206110.90197>