From owner-freebsd-hackers@freebsd.org Fri Oct 26 22:07:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D00E310C973A for ; Fri, 26 Oct 2018 22:07:12 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 38E626D294; Fri, 26 Oct 2018 22:07:12 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pl1-x62f.google.com with SMTP id 30-v6so1087255plb.10; Fri, 26 Oct 2018 15:07:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=jbjE8CdogO6iBzRNmWLN9ZkQlhu+jpKarMAv7prj3WQ=; b=HDuLhiHNvmhxmJQcLCCA+IfJPQy9XYWnlhZOI4QGZrEoOr+H09aQkVVv/tZV+HLdLB opB8mYY/HM0RB1+e0mIikVxqvVbd/Fu3nk/6TLaoyvKwlkU0v+AghNQ3bifp/XLgSDgt LAjvHZfWs/8UXtIxXLU+AR8Oh7aZujgyeoXaupEmy4+c1XW3VZ7e9TbAV6EMiY2GPbsp DkISKRcqSyWWoC2uK/fPG9hgf2eZueQbvRrqCQ5HYkuLhB7GSvFFr/WSrajpwSrDc46G wzFwUmbb7TpL0pdPzcYPGjvrsNZd0XqWwWWxEixMdr019CIWwULHKmTXW6wqX4wlFspm 2nWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=jbjE8CdogO6iBzRNmWLN9ZkQlhu+jpKarMAv7prj3WQ=; b=AYoTFLdfzv7dmv4qBg7tnqKTqTuhzlkOMvyuKUHEZxDY2pXNhZPw57KU8m+R4AFu4v wg3CyaRx8l8Yt/BwF4R+nHQN1YXHCcjmP1GVlmk+Vm/ynT4ulaCW+SvEdK9oYjpTfYyE E8aAqDaNhncdrvv8qTXG//mfAF2S7KC0lC+0LKNVoHGKNsNeO8nJmRZTxI1bNhuuGUKE TAmwyp0adYY2EsgQhdc3pf9p4vHZHVErnGIRT5MOOcJ+t2HWbNlfB3/sQhuVAt2rr/RX gcvtCfXs5ed8VrkK9nxru9PVYcaPSyuiQM7iB5Jv0DLDeMfZXjtJfYNpQuwywuSE9kot Ewbg== X-Gm-Message-State: AGRZ1gLXCRh7cx+Lmu8dsNgzEqSHyWYGGjrWXju4tqBg/zvrSoxip+cD 00pLkqnd/gr60Xl7MiLd384ZtKM= X-Google-Smtp-Source: AJdET5dRHYMR6c9lXJoKHttrcO84w9CRx0cbFG2IMjmvrwcxkHZQHKDLNYVkm0/5o1VjHl7QVC/f0w== X-Received: by 2002:a17:902:e101:: with SMTP id cc1-v6mr5224088plb.165.1540591629513; Fri, 26 Oct 2018 15:07:09 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id p63-v6sm1946329pfg.79.2018.10.26.15.07.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Oct 2018 15:07:08 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Millard Cc: Rozhuk Ivan , FreeBSD , Mark Johnston References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> From: Robert Message-ID: <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> Date: Fri, 26 Oct 2018 15:07:07 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2018 22:07:13 -0000 Sorry, let me be more specific. Please look into: https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing (wait until charts fully loaded). These are all memory states and mmap\munmap stats collected. Y axis is in MBs, X is a timeline. It's not a problem to understand which process produces allocations and is being swapped. I know this for sure. The issue is: I strongly believe that by some reason FreeBSD kernel fails to reuse deallocated memory properly. Looking into graphs we can see following: 1. When process allocates memory (mmap), "Active" memory increases, "Free" memory decreases (that's expected). 2. When process deallocates memory (munmap), "Inactive" memory increases, "Active" memory decreases. Memory never returns into "Free" state. That's kind of expected as well. 3. At some point, when sum of "Active" and "Inactive" memory exceeds some upper memory limits, OS starts to push "Inactive" memory into "Laundry" and "Swap". This happens very quick and unexpectedly. Now why OS doesn't reuse huge amounts of "Inactive" memory when calling mmap? Or my assumption about availability of "Inactive" memory is wrong? Which one is free for allocations then? Thanks. On 10/24/18 11:58 AM, Mark Millard wrote: > On 2018-Oct-24, at 1:25 PM, Robert wrote: > >> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: > No screen shot made it through the list back out to those that > get messages from the freebsd-hackers at freebsd.org reference > in the CC. The list limits itself to text as I understand. > >> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free >> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other >> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse > Relative to my limited point: I do not know the status of > mutually-exclusive categorizations vs. not for ZFS ARC and > Mem. > > Unfortunately, as I understand things, it is questionable if > adding -S to the top command gives you swap information that > can point to what makes up the 15G swapped out by totaling > the sizes listed. But you might at least be able to infer > what processes became swapped out even if you can not get > a good size for the swap space use for each. > > Using -ores does seem like it puts the top users of resident > memory at the top of top's process list. > > Sufficient Active RAM use by processes that stay active will > tend to cause inactive processes to be swapped out. FreeBSD > does not swap out processes that stay active: it pages those > as needed instead of swapping. > > So using top -Sores might allow watching what active > process(es) grow and stay active and what inactive processes > are swapped out at the time of the activity. > > I do infer that the 15G Used for Swap is tied to processes > that were not active when swapped out. > >> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", >> >> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. >> >> >> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? >> >> Thanks. >> >> >> On 10/24/18 9:34 AM, Mark Millard wrote: >>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: >>> >>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>> Robert wrote: >>>> >>>>> So the issue is still happening. Please check attached screenshot. >>>>> The green area is "inactive + cached + free". >>>>> >>>>> . . . >>>> +1 >>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free >>>> Swap: 112G Total, 19M Used, 112G Free >>> Just a limited point based on my understanding of "Buf" in >>> top's display . . . >>> >>> If "cached" means "Buf" in top's output, my understanding of Buf >>> is that it is not a distinct memory area. Instead it totals the >>> buffer space that is spread across multiple states: Active, >>> Inactive, Laundry, and possibly Wired(?). >>> >>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. >>> If Buf is added to that then there is double counting of >>> everything included in Buf and the total will be larger >>> than the TotalMemory. >>> >>> Also Inact+Buf+Free may double count some of the Inact space, >>> the space that happens to be inactive buffer space. >>> >>> I may be wrong, but that is my understanding. >>> > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) >