From owner-freebsd-questions@freebsd.org Tue Dec 29 07:42:14 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A1554D3C7A for ; Tue, 29 Dec 2020 07:42:14 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D4mb95BKGz3QZB for ; Tue, 29 Dec 2020 07:42:13 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: by mail-io1-xd31.google.com with SMTP id i18so11455493ioa.1 for ; Mon, 28 Dec 2020 23:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qmYzWHRLFjACZLSF3FwlMn8BGv5oJE3Fu/89sT8vSHI=; b=K9+vXXtBkvovgv2oDKvuTVJohUqz8wU6iWOiTsJ5mxJllIq+AVxc7I25AbSSqHusRT hi0uDrVRelNxIpmJ3DpYeNpb8A8kFAzTLpbEZ47s8gyfpFw7asA5JUYr2Aw1fgtdyKwP AtEH6NDbleKPoIH7I6n/NexgPHPJokv5oS5SWwBhpH45AoO0+hqaeM2TIhK0MM98OjFQ LL1SRdHzEEzyLbVwOlLG3Dq7UZQ29E4x5A+sROOkVALob6cjFJ3AZuAypfHeBqPTNa7L /ssTYI4zFZXVf+AMc1XApP91SY7uYOlxew3zaLh+IF30Hz1JZEN4cyczTMvfjbt0AFXr vMqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qmYzWHRLFjACZLSF3FwlMn8BGv5oJE3Fu/89sT8vSHI=; b=PfIsCl9JUQDXGtGLKl/z4ehlkEAmfBqYlkyQcFtFc16Zj3Lchk7qQzW7f3/tmZsxzG wTEhL/aVaohrVttbwrOmqZX2fsNR+rgpH9FZEhY4dyB7rkRNBmz6PWrkgs/loZqlgNds PSIUPHmuiy71alj/8+QYqmdLRTQMfxESvzjWYrQNQkrdfK1RF9+DbaRH14UpzaANxEGv g+YonpsDqtnpBTLn0GmVGZzse0fM5djx2xeqlzqafb5NUyYXntHZ0Xt171PUNCSsUO0Z YRHBjvTXn8sYikZZLMaPlIS5te4GwelZeHsOo9OPzEDiL6jB/BvGJaHSwUZNn9VG/o7v TPxw== X-Gm-Message-State: AOAM530WhaBhXw/yHDqyWHjKGs6WOuVwu5e9tr/H4W07ZbW5DodF39qY ezXsiwYS/CFhYnbvr8yGMyQKTvLUHRG5WkSHeJP15aBr X-Google-Smtp-Source: ABdhPJx0A0EMZA+WNDwyYbl910ltWpt+/MoLXylNvxk1UTBN3beNMLGwzQ0pjtE5dQ0j2R1r9aLZaAF+EWZelv8cWMo= X-Received: by 2002:a05:6602:1608:: with SMTP id x8mr39100450iow.72.1609227732818; Mon, 28 Dec 2020 23:42:12 -0800 (PST) MIME-Version: 1.0 References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> In-Reply-To: <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> From: Michael Schuster Date: Tue, 29 Dec 2020 08:42:01 +0100 Message-ID: Subject: Re: Observations on virtual memory operations To: Pete Wright Cc: doug@safeport.com, freeBSD Mailing List X-Rspamd-Queue-Id: 4D4mb95BKGz3QZB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=K9+vXXtB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of michaelsprivate@gmail.com designates 2607:f8b0:4864:20::d31 as permitted sender) smtp.mailfrom=michaelsprivate@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d31:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d31:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d31:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2020 07:42:14 -0000 On Tue, Dec 29, 2020, 00:37 Pete Wright wrote: > > > On 12/28/20 3:25 PM, doug wrote: > > I have two servers running jails that "routinely" run out of swapspace > > with > > no demand paging activity. To try and get a handle on VM/swapspace > > management I have been tracking swapinfo vs memory use as measured by > > top. > > The numbers do not exactly add up but I assume that is not involved in my > > issue. > > > > > > > The other day I caught the system at 73% swapspace used. At this level > > the > > system was in a near thrashing state in that typing a key got it > > echoed in > > 10 <--> 30 seconds. There was about 600MB of swapspace at this point. I > > would think there is no way to debug this except as a thought experiment. > > The first thing that comes to mind is do you have the ability to hook > any metrics/monitoring onto this system. For example, I use collectd on > my systems to report overall CPU/memory metrics as well as per-process > memory metrics. > > Alternatively you could write a simple shell script that run's "ps" and > parses the output of memory utilization on a per-process basis. > > either of the above approaches should give you some insight into where > the memory leak is coming from (assuming you already do not know). > > one trick i use is to invoke a process with "limits" to ensure it does > not exceed a certain amount of memory that I allocate to it. for example > with firefox i do this: > $ limits -m 6g -v 6g /usr/local/bin/firefox > > that should at least buy you enough time to investigate why the process > needs so much memory and see what you can do about it. > If the usual observation tools (and please note, I don't have too much specific knowledge here) don't tell you what you need to know, have a look at DTrace. Again, I have no (active) specific knowledge; personally, I would start with Brendan Gregg's tools (ask your favourite search engine)... Regards Michael