From owner-freebsd-questions@freebsd.org Mon Dec 28 23:37:13 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 53CA14C4D68 for ; Mon, 28 Dec 2020 23:37:13 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D4YqX3CWyz4skR for ; Mon, 28 Dec 2020 23:37:12 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.160] (cpe-24-24-163-126.socal.res.rr.com [24.24.163.126]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id a758e86b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 28 Dec 2020 23:37:05 +0000 (UTC) Subject: Re: Observations on virtual memory operations To: doug@safeport.com, freebsd-questions@FreeBSD.org References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> From: Pete Wright Message-ID: <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> Date: Mon, 28 Dec 2020 15:36:59 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4D4YqX3CWyz4skR X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-3.27 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[174.136.98.114:from]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[nomadlogic.org]; SPAMHAUS_ZRD(0.00)[174.136.98.114:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.971]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 23:37:13 -0000 On 12/28/20 3:25 PM, doug wrote: > I have two servers running jails that "routinely" run out of swapspace > with > no demand paging activity. To try and get a handle on VM/swapspace > management I have been tracking swapinfo vs memory use as measured by > top. > The numbers do not exactly add up but I assume that is not involved in my > issue. > > > The other day I caught the system at 73% swapspace used. At this level > the > system was in a near thrashing state in that typing a key got it > echoed in > 10 <--> 30 seconds. There was about 600MB of swapspace at this point. I > would think there is no way to debug this except as a thought experiment. The first thing that comes to mind is do you have the ability to hook any metrics/monitoring onto this system.  For example, I use collectd on my systems to report overall CPU/memory metrics as well as per-process memory metrics. Alternatively you could write a simple shell script that run's "ps" and parses the output of memory utilization on a per-process basis. either of the above approaches should give you some insight into where the memory leak is coming from (assuming you already do not know). one trick i use is to invoke a process with "limits" to ensure it does not exceed a certain amount of memory that I allocate to it. for example with firefox i do this: $ limits -m 6g -v 6g /usr/local/bin/firefox that should at least buy you enough time to investigate why the process needs so much memory and see what you can do about it. -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA