From owner-freebsd-stable@freebsd.org Tue Nov 20 11:22:36 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49F7C1121B53 for ; Tue, 20 Nov 2018 11:22:36 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48DE687550 for ; Tue, 20 Nov 2018 11:22:35 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id wAKBMVil049909 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 20 Nov 2018 12:22:31 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id wAKBMVBs049906 for ; Tue, 20 Nov 2018 12:22:31 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Tue, 20 Nov 2018 12:22:31 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: Re: plenty of memory, but system us intensively swapping In-Reply-To: <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in> Message-ID: References: <1ddb53ff-23a8-b364-0213-d8df3c7dcd3c@zhegan.in> <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no X-Rspamd-Queue-Id: 48DE687550 X-Spamd-Result: default: False [1.32 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; HAS_XAW(0.00)[]; NEURAL_SPAM_MEDIUM(0.15)[0.149,0]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[fagskolen.gjovik.no]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: smtp.fagskolen.gjovik.no]; CTYPE_MIXED_BOGUS(1.00)[]; NEURAL_SPAM_LONG(0.16)[0.155,0]; NEURAL_SPAM_SHORT(0.06)[0.056,0]; IP_SCORE(-0.03)[ipnet: 2001:700::/32(-0.01), asn: 224(-0.14), country: NO(-0.01)]; FORGED_SENDER(0.30)[Trond.Endrestol@fagskolen.gjovik.no,trond@fagskolen.gjovik.no]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:224, ipnet:2001:700::/32, country:NO]; FROM_NEQ_ENVFROM(0.00)[Trond.Endrestol@fagskolen.gjovik.no, trond@fagskolen.gjovik.no] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 11:22:36 -0000 On Tue, 20 Nov 2018 15:22+0500, Eugene M. Zheganin wrote: > Hello, > > On 20.11.2018 15:12, Trond Endrestøl wrote: > > On freebsd-hackers the other day, > > https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html, > > it was suggested to set vm.pageout_update_period=0. This sysctl is at > > 600 initially. > > > > ZFS' ARC needs to be capped, otherwise it will eat most, if not all, > > of your memory. > Well, as you can see, ARC ate only half, and the other half is eaten by the > kernel. So far I suppose that if I will cap the ARC, the kernel will simply > eat the rest. I know others have created a daemon that observe the ARC and the amount of wired and free memory, and when these values exceed some threshold, the daemon will allocate a number of gigabytes, writing zero to the first byte or word of every page, and then freeing the allocated memory before going back to sleep. The ARC will release most of its allocations and the kernel will also release some but not all of its wired memory, and some user pages are likely to be thrown onto the swap device, turning the user experience to a mild nightmare while waiting for applications to be paged back into memory. ZFS seems to be the common factor in most, if not all, of these cases. I created my own and not so sophisticated C program that I run every now and then: #include #include #include int main(int argc, char **argv) { const size_t pagesize = (size_t)getpagesize(); const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL; size_t amount, n = 1ULL; char *p, *offset; if (argc > 1) { sscanf(argv[1], "%zu", &n); } amount = n * gigabyte; if (amount > 0ULL) { if ( (p = malloc(amount)) != NULL) { for (offset = p; offset < p + amount; offset += pagesize) { *offset = '\0'; } free(p); } else { fprintf(stderr, "%s:%s:%d: unable to allocate %zu gigabyte%s\n", argv[0], __FILE__, __LINE__, n, (n == 1ULL) ? "" : "s"); return 2; } } else { return 1; } return 0; } // main() // allocate_gigabytes.c -- Trond. From owner-freebsd-stable@freebsd.org Tue Nov 20 11:38:54 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31C0B1122433 for ; Tue, 20 Nov 2018 11:38:54 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48AE46A08E for ; Tue, 20 Nov 2018 11:38:53 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from bsdrookie.norma.com. (asterisk.enaza.ru [91.237.76.254] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id wAKBckOD046799 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 20 Nov 2018 16:38:47 +0500 (+05) (envelope-from eugene@zhegan.in) Subject: Re: plenty of memory, but system us intensively swapping To: freebsd-stable@freebsd.org References: <1ddb53ff-23a8-b364-0213-d8df3c7dcd3c@zhegan.in> <192ecadf-4e66-297c-576f-f09e41ad7e29@zhegan.in> From: "Eugene M. Zheganin" Message-ID: <3efa0dd1-11d6-d424-9e37-a96cf7a6cf47@zhegan.in> Date: Tue, 20 Nov 2018 16:38:46 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Scan-Time: 0.64 X-Rspamd-Queue-Id: 48AE46A08E X-Spamd-Result: default: False [0.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.14)[-0.143,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[zhegan.in]; NEURAL_SPAM_SHORT(0.46)[0.456,0]; MX_GOOD(-0.01)[cached: mail-backup.zhegan.in]; NEURAL_SPAM_LONG(0.64)[0.641,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(0.00)[country: RU(0.01)]; ASN(0.00)[asn:57401, ipnet:2a00:7540:1::/48, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 11:38:54 -0000 Hello, On 20.11.2018 16:22, Trond Endrestøl wrote: > > I know others have created a daemon that observe the ARC and the > amount of wired and free memory, and when these values exceed some > threshold, the daemon will allocate a number of gigabytes, writing > zero to the first byte or word of every page, and then freeing the > allocated memory before going back to sleep. > > The ARC will release most of its allocations and the kernel will also > release some but not all of its wired memory, and some user pages are > likely to be thrown onto the swap device, turning the user experience > to a mild nightmare while waiting for applications to be paged back > into memory. > > ZFS seems to be the common factor in most, if not all, of these cases. > > I created my own and not so sophisticated C program that I run every > now and then: > > #include > #include > #include > > int main(int argc, char **argv) > { > const size_t pagesize = (size_t)getpagesize(); > const size_t gigabyte = 1024ULL * 1024ULL * 1024ULL; > > size_t amount, n = 1ULL; > char *p, *offset; > > if (argc > 1) { > sscanf(argv[1], "%zu", &n); > } > > amount = n * gigabyte; > > if (amount > 0ULL) { > if ( (p = malloc(amount)) != NULL) { > for (offset = p; offset < p + amount; offset += pagesize) { > *offset = '\0'; > } > > free(p); > } > else { > fprintf(stderr, > "%s:%s:%d: unable to allocate %zu gigabyte%s\n", > argv[0], __FILE__, __LINE__, > n, (n == 1ULL) ? "" : "s"); > return 2; > } > } > else { > return 1; > } > > return 0; > } // main() > > // allocate_gigabytes.c > Jeez, thanks a lot, this stuff is working. Now the system has 8 Gigs of free memory and stopped swapping. Well, the next question is addressed to the core team which I suppose reads this ML eventually - why we don't have something similar as a watchdog in the base system ? I understand that this solution is architecturally ugly, but it's now worse not to have any, and it still works. At least I'm about to run this periodically. Trond, thanks again. Eugene.