From owner-freebsd-questions@freebsd.org Tue Dec 29 17:49:51 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 15A994C6805 for ; Tue, 29 Dec 2020 17:49:51 +0000 (UTC) (envelope-from doug@safeport.com) Received: from cyrus.watson.org (cyrus.watson.org [204.107.128.30]) by mx1.freebsd.org (Postfix) with ESMTP id 4D524G0wwGz4t0l for ; Tue, 29 Dec 2020 17:49:49 +0000 (UTC) (envelope-from doug@safeport.com) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 6601E8C564; Tue, 29 Dec 2020 17:40:10 +0000 (UTC) Received: from fledge.watson.org (doug@localhost [127.0.0.1]) by fledge.watson.org (8.16.1/8.16.1) with ESMTPS id 0BTHeA2M097175 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 29 Dec 2020 17:40:10 GMT (envelope-from doug@safeport.com) Received: from localhost (doug@localhost) by fledge.watson.org (8.16.1/8.16.1/Submit) with ESMTP id 0BTHe9UB097172; Tue, 29 Dec 2020 17:40:09 GMT (envelope-from doug@safeport.com) X-Authentication-Warning: fledge.watson.org: doug owned process doing -bs Date: Tue, 29 Dec 2020 17:40:09 +0000 (UTC) From: doug@safeport.com Reply-To: doug@fledge.watson.org To: Michael Schuster cc: Pete Wright , freeBSD Mailing List Subject: Re: Observations on virtual memory operations In-Reply-To: Message-ID: References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 4D524G0wwGz4t0l X-Spamd-Bar: +++++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=fail (mx1.freebsd.org: domain of doug@safeport.com does not designate 204.107.128.30 as permitted sender) smtp.mailfrom=doug@safeport.com X-Spamd-Result: default: False [7.50 / 15.00]; HAS_REPLYTO(0.00)[doug@fledge.watson.org]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; HAS_XAW(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_ALL(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[204.107.128.30:from]; ASN(0.00)[asn:11288, ipnet:204.107.128.0/24, country:US]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all:c]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(1.00)[1.000]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[safeport.com]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[204.107.128.30:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; VIOLATED_DIRECT_SPF(3.50)[]; NEURAL_SPAM_LONG(1.00)[1.000]; FROM_NO_DN(0.00)[]; GREYLIST(0.00)[pass,body]; MAILMAN_DEST(0.00)[freebsd-questions] X-Spam: Yes X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2020 17:49:51 -0000 On Tue, 29 Dec 2020, Michael Schuster wrote: > On Tue, Dec 29, 2020, 00:37 Pete Wright wrote: > >> >> >> On 12/28/20 3:25 PM, doug wrote: >>> I have two servers running jails that "routinely" run out of swapspace >>> with >>> no demand paging activity. To try and get a handle on VM/swapspace >>> management I have been tracking swapinfo vs memory use as measured by >>> top. >>> The numbers do not exactly add up but I assume that is not involved in my >>> issue. >>> >> >>> >>> The other day I caught the system at 73% swapspace used. At this level >>> the >>> system was in a near thrashing state in that typing a key got it >>> echoed in >>> 10 <--> 30 seconds. There was about 600MB of swapspace at this point. I >>> would think there is no way to debug this except as a thought experiment. >> >> The first thing that comes to mind is do you have the ability to hook >> any metrics/monitoring onto this system. For example, I use collectd on >> my systems to report overall CPU/memory metrics as well as per-process >> memory metrics. >> >> Alternatively you could write a simple shell script that run's "ps" and >> parses the output of memory utilization on a per-process basis. >> >> either of the above approaches should give you some insight into where >> the memory leak is coming from (assuming you already do not know). >> >> one trick i use is to invoke a process with "limits" to ensure it does >> not exceed a certain amount of memory that I allocate to it. for example >> with firefox i do this: >> $ limits -m 6g -v 6g /usr/local/bin/firefox >> >> that should at least buy you enough time to investigate why the process >> needs so much memory and see what you can do about it. > Thank you all for the information and thoughts. If vmstat produces correct infomation there is no demand paging. The limiting condition on these systems is swapfile space rather than real memory. There are 69 sysctl elements dealing with paging and swapfile. If there is documentation (other than in C) on these that would be helpful perhaps. Most are totals, demand paging rates may be in this set, but not so as I can tell. The one time I caught the system dying the limiting resource was swapspace. There was no paging (last vmstat) and about 670MB left in the swapfile. In this state I could recover by restarting apache.