From owner-freebsd-questions@freebsd.org Mon Dec 28 23:25:50 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6511F4C4B88 for ; Mon, 28 Dec 2020 23:25:50 +0000 (UTC) (envelope-from doug@fledge.watson.org) Received: from cyrus.watson.org (cyrus.watson.org [204.107.128.30]) by mx1.freebsd.org (Postfix) with ESMTP id 4D4YZP3p1Cz4rhT for ; Mon, 28 Dec 2020 23:25:49 +0000 (UTC) (envelope-from doug@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 4D1D792DD0 for ; Mon, 28 Dec 2020 23:25:48 +0000 (UTC) Received: from fledge.watson.org (doug@localhost [127.0.0.1]) by fledge.watson.org (8.16.1/8.16.1) with ESMTPS id 0BSNPm2E080509 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Mon, 28 Dec 2020 23:25:48 GMT (envelope-from doug@fledge.watson.org) Received: from localhost (doug@localhost) by fledge.watson.org (8.16.1/8.16.1/Submit) with ESMTP id 0BSNPmpC080506 for ; Mon, 28 Dec 2020 23:25:48 GMT (envelope-from doug@fledge.watson.org) Date: Mon, 28 Dec 2020 23:25:48 +0000 (UTC) From: doug Reply-To: doug@safeport.com To: freebsd-questions@FreeBSD.org Subject: Observations on virtual memory operations Message-ID: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 4D4YZP3p1Cz4rhT X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of doug@fledge.watson.org has no SPF policy when checking 204.107.128.30) smtp.mailfrom=doug@fledge.watson.org X-Spamd-Result: default: False [1.90 / 15.00]; HAS_REPLYTO(0.00)[doug@safeport.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[204.107.128.30:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[204.107.128.30:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_LONG(0.90)[0.901]; DMARC_NA(0.00)[watson.org]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11288, ipnet:204.107.128.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 23:25:50 -0000 I have two servers running jails that "routinely" run out of swapspace with no demand paging activity. To try and get a handle on VM/swapspace management I have been tracking swapinfo vs memory use as measured by top. The numbers do not exactly add up but I assume that is not involved in my issue. The system: real memory = 8589934592 (8192 MB) avail memory = 8248696832 (7866 MB) In top the sum of Active, Inactive, Laundry, Wired, Buf and Free is almost constant between 8704 and 8706 MB. Wired+Buf is fairly constant around 2200MB. Free has no relation to anything as long as the system is working (e.g., what you would expect). The other day I caught the system at 73% swapspace used. At this level the system was in a near thrashing state in that typing a key got it echoed in 10 <--> 30 seconds. There was about 600MB of swapspace at this point. I would think there is no way to debug this except as a thought experiment. My other thought is it would be nice if after perhaps 5,000 -> 10,000 'out of swap space message' logging could stop. I read a google post suggesting this can be handled by logging the console output to a file. Does anyone have experience with this? Right now we are surviving by email warnings when the system gets to 50% swap used and monitoring,