Date: Wed, 30 Dec 2020 01:03:05 +0000 (UTC) From: doug@safeport.com To: Pete Wright <pete@nomadlogic.org> Cc: Michael Schuster <michaelsprivate@gmail.com>, freeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: Observations on virtual memory operations Message-ID: <c5cbc8d0-dccd-cb66-a35c-93bcb6cd676@safeport.com> In-Reply-To: <03553c65-c1c2-db2c-6ab9-a0f4d09c3e2d@nomadlogic.org> References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> <CADqw_gKWdMaMpCq=%2By=_pvd6wp2bJvDsWkv5GYTwrznhCSpfeQ@mail.gmail.com> <af8ceb5-8ac-57f5-24f8-72b114fa8a@safeport.com> <03553c65-c1c2-db2c-6ab9-a0f4d09c3e2d@nomadlogic.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 29 Dec 2020, Pete Wright wrote: > > > On 12/29/20 9:40 AM, doug@safeport.com wrote: >> On Tue, 29 Dec 2020, Michael Schuster wrote: >> >>> On Tue, Dec 29, 2020, 00:37 Pete Wright <pete@nomadlogic.org> wrote: >>> >>>> >>>> >>>> On 12/28/20 3:25 PM, doug wrote: >>>>> I have two servers running jails that "routinely" run out of >>>>> swapspace with no demand paging activity. To try and get a handle on >>>>> VM/swapspace management I have been tracking swapinfo vs memory use >>>>> as measured by top. The numbers do not exactly add up but I assume >>>>> that is not involved in my issue. >>>>> <snip> >> Thank you all for the information and thoughts. If vmstat produces >> correct infomation there is no demand paging. The limiting condition >> on these systems is swapfile space rather than real memory. There are >> 69 sysctl elements dealing with paging and swapfile. If there is >> documentation (other than in C) on these that would be helpful >> perhaps. Most are totals, demand paging rates may be in this set, but >> not so as I can tell. >> >> The one time I caught the system dying the limiting resource was >> swapspace. There was no paging (last vmstat) and about 670MB left in >> the swapfile. In this state I could recover by restarting apache. > > I wouldn't go down that rabbit hole just yet. If the issue is with > apache-httpd causing your memory to run away I would instead focus on > trying to determine *why* httpd is doing that. Generally a well behaved > process should not need to page out to disk if the system is > appropriately sized and configured. As such I would suggest starting at > the application layer before trying to tweak how FreeBSD manages paging > out to disk. > > For example, I remember issues back in the day where httpd would consume > tons of memory if people were uploading files. We were able to address > this by being more aggressive in how we wrote files to disk in chunks > during the upload process. I do not seem to be able to say clearly enough, there is no memory problem, the is a problem with the paging subsystem filling up swapspace. I have monitored memory with vmstat, systat, top, a perl program I found that makes a report using sysctl, and a python program I wrote to see how swap space is allocated among the jails. As far as I can see other than forking, there is no paging and assuming 100% of active memory is backed up, there is ample swap space. All that said there are so many data elements that sorta match that making definitive statements with a pretty good understanding of the code is risky. when the system starts with the 'out of swap space' message being logged, the console is totally locked up and pretty much the rest of these system along with it. I keep an xterm with an apache shutdown command handy as one enter key will get thourgh quicker than I can enlist the data center personel to do a manual reboot. It would appear at this point this issue is unique to me. In reading Dillon's writeup on VM processing it seems possbile that space allocated to the inactive and/or laundry queues is not being cleared. I found one guy suggesting that directing all console output to a file at least prevents losing the serial console. I wonder if anyone has experience with that. If there is a program that will fill up the swap file I could test this. I am somewhat reluctant to try this on a production system without having any experience with the fall out of doing this. From owner-freebsd-questions@freebsd.org Wed Dec 30 06:06:53 2020 Return-Path: <owner-freebsd-questions@freebsd.org> Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8E3494D7C79 for <freebsd-questions@mailman.nyi.freebsd.org>; Wed, 30 Dec 2020 06:06:53 +0000 (UTC) (envelope-from 4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com) Received: from s1-b0c6.socketlabs.email-od.com (s1-b0c6.socketlabs.email-od.com [142.0.176.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D5LQg6fwpz4h2l for <freebsd-questions@freebsd.org>; Wed, 30 Dec 2020 06:06:51 +0000 (UTC) (envelope-from 4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com) DKIM-Signature: v=1; a=rsa-sha256; d=email-od.com;i=@email-od.com;s=dkim; c=relaxed/relaxed; q=dns/txt; t=1609308412; x=1611900412; h=content-transfer-encoding:content-type:mime-version:references:in-reply-to:message-id:subject:cc:to:from:date:x-thread-info; bh=wT1fPkAHIU072b6GkrWdf3n2eNI5mnnt8jKmJvZ8JAE=; b=tFw1zRFOag5aO1OqtNHuAa3qaNaqdm9GzV21QTbxQaBgEPmUwEeYcsGvrBlEawu2W3xXt+j7tmMh03jSWQNQtLhEN7/51F4P+HUMcyoDA8TZvcerig2/Vr0QbXlZTqbvWzjGzF0VdUrNoXvKl2f6LWRDvaDHkzCt/NvILOE9UFE= X-Thread-Info: NDI1MC45Mi4xZDRjNzAwMDIxYzk3MDIuZnJlZWJzZC1xdWVzdGlvbnM9ZnJlZWJzZC5vcmc= Received: from r3.us-east-1.aws.in.socketlabs.com (r3.us-east-1.aws.in.socketlabs.com [142.0.191.3]) by mxsg2.email-od.com with ESMTP(version=Tls12 cipher=Aes256 bits=256); Wed, 30 Dec 2020 01:06:49 -0500 Received: from smtp.lan.sohara.org (EMTPY [185.202.17.215]) by r3.us-east-1.aws.in.socketlabs.com with ESMTP(version=Tls12 cipher=Aes256 bits=256); Wed, 30 Dec 2020 01:06:49 -0500 Received: from [192.168.63.1] (helo=steve.lan.sohara.org) by smtp.lan.sohara.org with smtp (Exim 4.94 (FreeBSD)) (envelope-from <steve@sohara.org>) id 1kuUdD-000NKm-K6; Wed, 30 Dec 2020 06:06:47 +0000 Date: Wed, 30 Dec 2020 06:06:47 +0000 From: Steve O'Hara-Smith <steve@sohara.org> To: freebsd-questions@freebsd.org Cc: doug@fledge.watson.org Subject: Re: Observations on virtual memory operations Message-Id: <20201230060647.38938a75f69e6c045802f655@sohara.org> In-Reply-To: <c5cbc8d0-dccd-cb66-a35c-93bcb6cd676@safeport.com> References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> <CADqw_gKWdMaMpCq=+y=_pvd6wp2bJvDsWkv5GYTwrznhCSpfeQ@mail.gmail.com> <af8ceb5-8ac-57f5-24f8-72b114fa8a@safeport.com> <03553c65-c1c2-db2c-6ab9-a0f4d09c3e2d@nomadlogic.org> <c5cbc8d0-dccd-cb66-a35c-93bcb6cd676@safeport.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.1) X-Clacks-Overhead: "GNU Terry Pratchett" Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4D5LQg6fwpz4h2l X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=email-od.com header.s=dkim header.b=tFw1zRFO; dmarc=none; spf=pass (mx1.freebsd.org: domain of 4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com designates 142.0.176.198 as permitted sender) smtp.mailfrom=4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com X-Spamd-Result: default: False [-2.70 / 15.00]; RWL_MAILSPIKE_GOOD(0.00)[142.0.176.198:from]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:142.0.176.0/20]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[email-od.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-0.995]; FORGED_SENDER(0.30)[steve@sohara.org,4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[142.0.176.198:from]; ASN(0.00)[asn:7381, ipnet:142.0.176.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[steve@sohara.org,4250.82.1d4c700021c9702.c32295275e4ccd9c507105269619081a@email-od.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[email-od.com:s=dkim]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sohara.org]; SPAMHAUS_ZRD(0.00)[142.0.176.198:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[142.0.176.198:from]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>; List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 30 Dec 2020 06:06:53 -0000 On Wed, 30 Dec 2020 01:03:05 +0000 (UTC) doug@safeport.com wrote: > I do not seem to be able to say clearly enough, there is no memory > problem, the is a problem with the paging subsystem filling up swapspace. Are you using tmpfs or swap backed md ? -- Steve O'Hara-Smith <steve@sohara.org>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c5cbc8d0-dccd-cb66-a35c-93bcb6cd676>