From owner-freebsd-hackers@freebsd.org Wed Oct 24 17:19:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7192AFFDA15 for ; Wed, 24 Oct 2018 17:19:25 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9A036F4B2; Wed, 24 Oct 2018 17:19:24 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pg1-x532.google.com with SMTP id g12-v6so2647626pgs.1; Wed, 24 Oct 2018 10:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=uSIZHFG8hcuWuTMTiaQmi5GUWmlMmXvgqDoOWrUZLNk=; b=lokqwz8EUEiArQMc6Ww5Btrq3OG9OTjYXPul8tsamaGdmPKIS8Cp+p07a8YyEVYqeI lyjNRmyLmNNV8ck+0gyy6TMl6Uh4kCtRnLHGQ+J1aQ+9GptdVJG+Xr6F9z0nJNLJTvs4 t0faLspr9/gZOoNnfZfXwdLlNQaqCclsbpsuwW7fQ+bUl6A57vOw/DjMNoFZ9/wPvLHD 9/AvjpOFN17Q3wK5SzaJZ8y0XZA7sCQAUDQc8vFSpVeE9Ljq4AYs/7xXlfWVRazU6ano 7XWQ35fkdd3IXbkKmZKUbDCgi+ucZWHdGPIMQo/R09Fg7zv7exmRwAG8nMrPNbEQb9ct T1DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=uSIZHFG8hcuWuTMTiaQmi5GUWmlMmXvgqDoOWrUZLNk=; b=qSBmu9r/oDECpyQM5R9PsdT50QEuYAmaXdZXWQcf+spyxu63T10PtqI2acgpkkbLVA b90pGF1IgiOmvMwmX6gEBid76eDRGzkE6+kBi/4uNQdRIcujpSW5mnHcU6glGDub23sO AiQnxdHZuDlm6ypl26nusB0qI96Wc0OGs3/dIpfHuDCvFyXaW6YURLgZjhpFhRGWGpYJ lg8UxZS/UNiwNNk6X3aFl0WxA7ICRes633EMVaS4w+CV8xpOyLfJ2s4TwkAZL9iTe7Rx IbbAjiVzFNoXbCJrGcbscYtUw3nUVQ2WEB2CgRWLa8O2XPoUG3M5XrHxp/bReEcEH0tD vnWA== X-Gm-Message-State: AGRZ1gLoUlXTrzPz/5Jld5IhtzS5PlmiLr2bryC/ip8u/TehNv4hFDaj DNXPmxa3nBWCbwcRMXpo0c09x10= X-Google-Smtp-Source: AJdET5dkOlp/hF/2l5XHvvztoRVD4C4ACSTeqCVcGS9tpjbpoPjVDdPE0PGXgGqWDLhtcNalzrxLtg== X-Received: by 2002:a62:418a:: with SMTP id g10-v6mr1005449pfd.44.1540401563129; Wed, 24 Oct 2018 10:19:23 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id r23-v6sm7007328pfh.79.2018.10.24.10.19.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 10:19:21 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Johnston Cc: freebsd-hackers@freebsd.org References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> From: Robert Message-ID: Date: Wed, 24 Oct 2018 10:19:20 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20180928152550.GA3609@raichu> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 17:19:26 -0000 So the issue is still happening. Please check attached screenshot. The green area is "inactive + cached + free". At some point massive amount of "Free" memory is moving to Swap and Laundry. Then slowly returns back to inactive. My concern is that I've tried all kind of monitoring for mem allocations, including dtrace-ing of all mmap calls. It shows no any allocations of such a huge size, so I believe this is something related to the kernel's mem management. Could you advice which other places should be checked\monitored to find a root cause? Thanks. On 9/28/18 6:25 AM, Mark Johnston wrote: > On Thu, Sep 27, 2018 at 04:04:15PM -0700, Robert wrote: >> Is there a way to force move pages back from laundry to Free or Inactive? > There's no real way to do that from outside of the application. The > application itself can free anonymous memory by unmapping it. It can > also force memory to be marked clean (thus moving it back to the > inactive queue) using madvise(MADV_FREE). This will cause the memory's > contents to be discarded, though. > >> Also, what's the best way to identify addresses of these pages and >> "look" inside of them? > There's no convenient way to do that that I'm aware of. On amd64, you > could in principle use kgdb to dump individual pages in the queue via > the direct map: > > # kgdb82 -q > Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. > done. > sched_switch (td=0xfffff8002c560000, newtd=0xfffff800025ca000, flags=) at /home/mark/src/freebsd-dev/sys/kern/sched_ule.c:2112 > 2112 cpuid = PCPU_GET(cpuid); > (kgdb) p vm_dom[0].vmd_pagequeues[2] > $1 = { > pq_mutex = { > lock_object = { > lo_name = 0xffffffff80c34c0d "vm laundry pagequeue", > lo_flags = 21168128, > lo_data = 0, > lo_witness = 0xfffff8041ed6cb00 > }, > mtx_lock = 0 > }, > pq_pl = { > tqh_first = 0xfffff8040f9ef980, > tqh_last = 0xfffff8041227f448 > }, > pq_cnt = 20087, > pq_name = 0xffffffff80c34c0d "vm laundry pagequeue", > pq_pdpages = 0 > } > (kgdb) x/512gx vm_dom[0].vmd_pagequeues[2].pq_pl.tqh_first->phys_addr + (vm_offset_t)&dmapbase > 0xfffff801aea00000: 0x0000000800000000 0x000000082298c400 > 0xfffff801aea00010: 0x00000009be801000 0x0000000000000006 > ...