From owner-freebsd-hackers@freebsd.org Mon Oct 29 22:45:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B93B010EC881 for ; Mon, 29 Oct 2018 22:45:30 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 381AB894D5; Mon, 29 Oct 2018 22:45:30 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pf1-x432.google.com with SMTP id h4-v6so4747480pfi.10; Mon, 29 Oct 2018 15:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=j2PTZBPryYKgKYdVZ0bILYsrxqd2a9MjLrCTs8re2Ak=; b=c9PTGzA3fIMm8mu6b+ui4QOlgsh0IhjlvMn05HK6HA7Whhv5i/DuP8pM8Xb7CBdZo8 YsJQRSDrWo3DNVaAG51txTH6HJuNFhSACBbSAbUEhyV1zerVrlOyPzx/yU8IB7eOTuCW tKLbN4xltou16KcyErhtzwuKaDZI22dvRKcpaRVKHQGVs0eCUCT4Ca4CrDIY0HBNRSsg nmAuOCYaAqeT4h1AM59Mffe8kDEIV/7KYXN/5bru+ZBZesQSts8EnrBk2hn81jTR2X2d L7y73qarQ+VDTyvL8YU6hz4l+sBUQLvmR+6lxE58kNSvBmtWOA454cBkAmFJCxPstb3h AuWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=j2PTZBPryYKgKYdVZ0bILYsrxqd2a9MjLrCTs8re2Ak=; b=AK+UGBUVYKZ4+5gCz6d2cLEz/sjKdty/eTQco4Coy2hGccNzc7H1Y9QSCQbFD9UXWG gf4YJ8mJVUOKTK7sKx/Y1rdsQgg+e3nkz6K3uCq1JkqK4wbcv1L3el4SkLKrG3gXU1oI jEqP7G6oImYDdU6kAfx8S0SDxJJGjzAg8ZxgTj6CE9Ha6IizKuWToKvJj18uq6kGnoIM hhXaHZHvRm9h4lMIZw1S8fvXvd8RE/C9iUB6vz3CAW4WzxAd6oHovZ3+i7RrnZz0ep+K 9sYGMq1/JbjgBC/Kt3612gor+ym0fT7OjxVrqS56dqye/bIwIQQ38tBliyubjW2aCBIS ytbA== X-Gm-Message-State: AGRZ1gLyTPEVSFX5oGdnFKUinlx+qnNdMv5GNbODb2Sn0LZDl+CRVJFV pjdk1sCv4Ts/7DZ0Pxu4eQ== X-Google-Smtp-Source: AJdET5cJeKpi0fw/rXStIDA+6UWBtJm4bNZ3OSCxNlaiyh5MndIcYsryI0cvHDaH39lkXftFVHfDNg== X-Received: by 2002:a62:8145:: with SMTP id t66-v6mr290886pfd.246.1540853128622; Mon, 29 Oct 2018 15:45:28 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id f22-v6sm27451190pff.29.2018.10.29.15.45.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 15:45:26 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Konstantin Belousov Cc: Mark Millard , FreeBSD , Mark Johnston , Rozhuk Ivan References: <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> <20181027043819.GX5335@kib.kiev.ua> From: Robert Message-ID: Date: Mon, 29 Oct 2018 15:45:25 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20181027043819.GX5335@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 22:45:31 -0000 Hi Konstantin, thanks for your reply. As per your response, both active and inactive regions are actually allocated to the user app, means my app allocated (and haven't freed) more than available amount of RAM... Does that mean there is basically a memory leak in the app code? Could you recommend a good instrument to find and catch a memory leak in a FreeBSD? Thanks. On 10/26/18 9:38 PM, Konstantin Belousov wrote: > On Fri, Oct 26, 2018 at 03:07:07PM -0700, Robert wrote: >> Sorry, let me be more specific. >> >> Please look into: >> https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing >> (wait until charts fully loaded). >> >> These are all memory states and mmap\munmap stats collected. Y axis is >> in MBs, X is a timeline. >> >> It's not a problem to understand which process produces allocations and >> is being swapped. I know this for sure. >> >> The issue is: I strongly believe that by some reason FreeBSD kernel >> fails to reuse deallocated memory properly. >> >> Looking into graphs we can see following: >> >> 1. When process allocates memory (mmap), "Active" memory increases, >> "Free" memory decreases (that's expected). > No, mmap(2) call only causes some small amount of the kernel memory > allocation, to track the mmaped region. > > Pages are allocated on the first touch, lets ignore the prefaulting > mechanism which is not too relevant for the discussion. Mapped page > can be either active or inactive, this is determined by the usage > history. > >> 2. When process deallocates memory (munmap), "Inactive" memory >> increases, "Active" memory decreases. > Again no. Unmapped pages does not get active references from userspace > which touch them, so eventually pagedaemon moves active unmapped memory > to inactive. > >> Memory never returns into "Free" state. That's kind of expected as well. > When the last mapping of the anonymous swap-backed page is destroyed, > there is no point in retaining the page content, so it is freed. > For the pages backed by the file, there is no point to drop content > which we already paid for by read. > >> 3. At some point, when sum of "Active" and "Inactive" memory exceeds >> some upper memory limits, >> >> OS starts to push "Inactive" memory into "Laundry" and "Swap". This >> happens very quick and unexpectedly. > So this is the pattern of behaviour of your applications. > >> Now why OS doesn't reuse huge amounts of "Inactive" memory when calling >> mmap? > Because typically active and inactive pages carry some useful content, > which cannot be lost. Before reusing the page, pagedaemon must ensure > that the page is written to file for file-backed mappings, or to swap > for anonymous mappings, so its data can be restored when needed again. > > If the page content is synced with the permanent storage, it is called > clean, otherwise dirty. Reuse of dirty page requires changing its state > to clean by write-out. Laundry is the queue where the dirty pages > sit waiting for write, this is done to not clog inactive queue since > the page was already processed by the page daemon and decided for laundry. > >> Or my assumption about availability of "Inactive" memory is wrong? Which >> one is free for allocations then? > Yes, you assumptions are wrong. Active/inactive represent the > usage/reference history, not the reusability (clean/dirty) state. > >> Thanks. >> >> >> On 10/24/18 11:58 AM, Mark Millard wrote: >>> On 2018-Oct-24, at 1:25 PM, Robert wrote: >>> >>>> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: >>> No screen shot made it through the list back out to those that >>> get messages from the freebsd-hackers at freebsd.org reference >>> in the CC. The list limits itself to text as I understand. >>> >>>> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free >>>> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other >>>> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >>>> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse >>> Relative to my limited point: I do not know the status of >>> mutually-exclusive categorizations vs. not for ZFS ARC and >>> Mem. >>> >>> Unfortunately, as I understand things, it is questionable if >>> adding -S to the top command gives you swap information that >>> can point to what makes up the 15G swapped out by totaling >>> the sizes listed. But you might at least be able to infer >>> what processes became swapped out even if you can not get >>> a good size for the swap space use for each. >>> >>> Using -ores does seem like it puts the top users of resident >>> memory at the top of top's process list. >>> >>> Sufficient Active RAM use by processes that stay active will >>> tend to cause inactive processes to be swapped out. FreeBSD >>> does not swap out processes that stay active: it pages those >>> as needed instead of swapping. >>> >>> So using top -Sores might allow watching what active >>> process(es) grow and stay active and what inactive processes >>> are swapped out at the time of the activity. >>> >>> I do infer that the 15G Used for Swap is tied to processes >>> that were not active when swapped out. >>> >>>> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", >>>> >>>> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. >>>> >>>> >>>> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? >>>> >>>> Thanks. >>>> >>>> >>>> On 10/24/18 9:34 AM, Mark Millard wrote: >>>>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: >>>>> >>>>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>>>> Robert wrote: >>>>>> >>>>>>> So the issue is still happening. Please check attached screenshot. >>>>>>> The green area is "inactive + cached + free". >>>>>>> >>>>>>> . . . >>>>>> +1 >>>>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free >>>>>> Swap: 112G Total, 19M Used, 112G Free >>>>> Just a limited point based on my understanding of "Buf" in >>>>> top's display . . . >>>>> >>>>> If "cached" means "Buf" in top's output, my understanding of Buf >>>>> is that it is not a distinct memory area. Instead it totals the >>>>> buffer space that is spread across multiple states: Active, >>>>> Inactive, Laundry, and possibly Wired(?). >>>>> >>>>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. >>>>> If Buf is added to that then there is double counting of >>>>> everything included in Buf and the total will be larger >>>>> than the TotalMemory. >>>>> >>>>> Also Inact+Buf+Free may double count some of the Inact space, >>>>> the space that happens to be inactive buffer space. >>>>> >>>>> I may be wrong, but that is my understanding. >>>>> >>> === >>> Mark Millard >>> marklmi at yahoo.com >>> ( dsl-only.net went >>> away in early 2018-Mar) >>> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"